Durability, weather proofing and build tolerance are probably the main three. A well constructed lens will deal with more rigorous use and last longer without losing it's precision.

The build tolerances are also far tighter on more expensive lenses. You are talking about very small amounts of difference, but you will notice it if you set your depth of field really shallow and try to adjust the focus. There is significantly more play on cheaper lenses than on expensive ones. This extends not only to the focus and zoom gears, but also to internal alignment tracks of the lenses. The more play they have, the less precise the lens can be.

Finally, weather sealing keeps moisture and dust out of the interior of lenses when used with weather sealed bodies. This is important for use in harsher conditions as well as for avoiding mold and fungus on the inside of the lens.

Owning a couple of cheaper lenses as well as several high end lenses, I can tell you there is an immense difference. If I compare the kit lens that came with my xTi with my 24-70 f/2.8L II that I use on my 5D Mark iii or even between the $600 70-300mm f/4-5.6 IS lens and my 70-200 f/2.8L IS II, there is a huge difference in terms of how tightly the zoom and focus hold and how precisely they are able to be focused. When I reverse directions of focus on the 70-300 it takes a bit of movement before the lens elements start moving. On either of the L's though, the adjustments are immediate as soon as I move in either direction with no slack.

Similarly, if I tilt the lens down, the 70-300 will sometimes stretch out and not hold the focal length due to the weight of the lens pulling it down. The 70-200 and 24-70's are solidly enough built however that they hold their position.