To make the most stunning sunset/sunrise photos you need at least some post production. Be this in camera via picture styles, during raw conversion or in Photoshop (or its cousins). You also need the most stunning scenery/lighting - that should go without saying.

However the thing most often missing when I see people compare their work to others online is post production to push the image to it's limits, people often don't realise how much goes into to a typical sunset image. Check out the 28 most "interesting" sunset images on Flickr. Pretty much all of them display signs of editing that go beyond what you can achieve in camera.

Taking multiple exposures to maximise the dynamic range of your camera, and give you the best raw materials to work with when producing a sunset/sunrise image. Also it doesn't even have to be sunset or sunrise, this was shot in the middle of the afternoon, just after a particularly heavy storm. Lots of post processing in Photoshop yeided this "sunset":

Besides that, all the standard rules of composition apply, and there's a lot to be said for picking the right subject. Atmospheric conditions play a large part but that's not really in your hands.

There's no secret to it, really, you just press the shutter button and provided you have composed and set the exposure correctly there's nothing more you could have done once you've maximised the scenery/lighting. Post production is where you have the greatest input. Yes luck is involved (with the weather etc.) but as a wise man said, the harder I work the luckier I get! Be tenacious and stick at it!

edit:

To answer your second question, yes there are indications in the weather, basically you want the opposite conditions to when you're trying to avoid haze. You want a lot of particulates in the sky to shift the colours as much as possible, so you want the end of a high pressure system with low wind. Thin cloud low in the sky also looks very good in sunset/sunrise shots. Any extreme or unusual conditions are worth exploring, see the following link:

Volcano's Eruption Colors World's Sunsets