Different light sources have different color temperature and when you want natural colors, you need to correct them for that particular light source. Basically white balance says what is rendered as neutral gray. You can find a more thorough explanation here: http://www.cambridgeincolour.com/tutorials/white-balance.htm. Wikipedia lists different light sources with their approximate color temperatures in Kelvins.

Try setting your camera to different white balances and take some shots - when you use Cloudy under incandescent light, you'll get yellow results and when you use Incandescent under daylight, you'll get very blue results.

Whenever you need correct white balance, you should use gray card as a reference when shooting and apply WB corrections based on that. Usually just believable white balance will do and might be even useful for artistic purposes (when you want to make your sunset just a bit warmer).

When shooting with digital camera, the easiest approach is to shoot RAW and use AWB (automatic white balance). This gives pretty usable results in most of the cameras and you can always fine tune your WB during RAW conversion. When shooting RAW, this correction is lossless. Most of the RAW converters have a tool called WB picker, which you can use to set neutral gray, meaning correction curves will be applied to the whole picture so that whatever you pick will appear as gray. This tool is very useful in mixed lightning situations or whenever the light source has strong tint either towards green or purple.

When you shoot and correct large batches of pictures, you want consistent WB and it's best to use the same non-auto white balance setting for all of them (even if it's slightly off), so you can apply the same correction to whole set. (Thanks, Matt)

When shooting film, you have to mess with color correction filters.

White balance affects exposure, but the effects are quite small to care in most common situations. There is a technique called UniWB, whci is used to minimize white balance effects on exposure, but this is already an advanced topic.

Don't underestimate the effect of correct white balance to the photo. Here is an example:

In-camera auto white balance:

Software auto white balance:

Hand-picked white balance (final image with some other adjustments):