This is down to the way digital sensors work. This tutorial is a good introductory article on the way digital sensors work.

For the purposes of illustration, think of the light cavity as a bucket. Let's assume that a well exposed image at the sensor's base ISO of 100 will fully fill the light cavity of a photosite. Therefore, going from ISO 100 to ISO 200 will halve the exposure time and will only half fill the light cavity. Each stop of increment in the ISO speed will result in the corresponding halving of the amount of light filling the cavity. This is why you have a usable ISO range on digital cameras, e.g. the light cavity is full at base ISO and it is "almost empty" or filled to the minimal amount that is usable at maximum ISO.

With most cameras, you get a usable ISO range of about 6-7 stops which corresponds to an ISO range of ISO 100 - ISO 6400. Camera manufacturers are free to shift the sensitivity window of the sensor to any value they like. The reason most cameras choose to have ISO 100 - ISO 6400 is because it's the most useful range for a wide variety of photography as opposed to a camera that went from ISO 6 - ISO 400. It's also doesn't hurt that having a bigger number always impresses!

While you may be able to push the camera below the base ISO, e.g. setting your camera to ISO 100 and overexposing by +2EV. This is equivalent to going to ISO 25 on a sensor with a base ISO of 100. You'll end up with an image that clips the highlights as the light cavities have overflowed and you won't be able to recover them. This is similar to what you get at the opposite end, where if you push a camera beyond it's maximum rated ISO you will end up with clipped shadows. For more information about these techniques, please look for the term "Push Processing".