Let's start with the wrong times; it's much easier that way.

High noon on a cloudless day is probably wrong. Although your subjects (flowers) won't be suffering from racoon eyes (and you can shoot many of them from near-overhead), shadows will be at their deepest, highlights will be at their brightest, and the contrast will likely overwhelm whatever dynamic range you have to work with. Besides which, it's just plain uncomfortable to work in most of the time.

The slices of time around sunrise and sunset that we usually refer to as "the golden hour" are also probably wrong. Of course, it depends on exactly which flowers, etc., you are shooting, but you may find that many of the reddest or orangest (yeah, I know, it's not a "real" word, so I don't particularly need the wavy red underline, thank you very much) blossoms need some amount of ultraviolet in order to be as bright as they are. The reflective and diffractive components alone aren't enough, and the results can be disappointing without the fluorescence. And anything that tends toward the blue end of the spectrum will be somewhat muddy at best. There is nothing wrong with a bit of residual warmth from a recent sunrise or some incipient warmth from an impending sunset, but you do want to be far enough from either to have at least some amount of UV in play.

So that gets us to a half-hour or so after sunrise to a half-hour or so before sunset, with two to four hours chopped out of the middle of the day (depending on your latitude and the time of year). And to optimize that, you'd like to have some cloud without being thoroughly overcast (you want the fill, but don't want to lose too much at the red end or the balance between reflective/diffractive and fluorescent colours will be off in an unfixable way). Both reflected light (light coming from within 90 degrees of the same side of the subject as yourself) and transmitted light (looking through the flower, etc., like a stained glass window) work well. Having some fill available (wearing a white shirt works wonders, and a white golf towel is good for more than mopping your brow) will extend your range a lot. And never, ever, underestimate the evocative power of the dew. If you can't be there for it in the morning, an atomizer works well in the evenings as a substitute. (Try not to use an atomizer before noon, or at least no later than an hour or so after sunrise. The droplets become magnifying glasses for the noonday sun, and can damage the plants.)