What does the 'Prop' setting do on a studio strobe's modeling lamp?

The "Prop" is short for "Proportional", meaning that the intensity of the modelling lamp changes as you change the output power of the flash. It's mostly useful for multi-light setups using the same model of flash, and gives you a visual idea of what the lighting ratio (the difference between key and fill, etc) is going to look like. It's very approximate, though, but still useful.

I generally used it to pick up potentially troublesome reflections. You want your main (key) modelling light to dominate so that you can see the shape and position of the main shadows, so you want the modelling lights on any fill lights to be much lower—but if you turn them off, you'll miss things like extra highlights on noses and so forth. It can give you some idea of the intensity of kickers and hair lights and that sort of thing, but you really need to see what the actual flash is doing. Since the modelling lights are tungsten, they'll change colour temperature as well as intensity, and that can fool the eye quite a bit. And being tungsten, they're going to change in colour and intensity as they are used, so bulbs that matched when you bought them may not match very closely a few months down the road. Proportional is good, but take what it tells you with a grain of salt.