Edit: before I can accept an answer I need a clarification and maybe someone should tell me if this should be a different question. Light coming from the scene is constant regardless of the focal length, but apertures are represented as a function of focal length and we usually just refer to the denominator. So if F/4 at 24mm is 24/4 versus F/4 at 200mm is 200/4, does that mean that the amount of light actually touching the sensor is different or did I screw up math again?

The whole reason to state aperture as a fraction of focal length is to remove focal length from the computation of exposure.

A white wall is reflecting a certain amount of light per unit area. A 50mm f/4.0 lens captures the entire area of the wall, about 12 meters by 8 meters.

A 200mm f/4.0 lens captures a 3m by 2m section of the wall. Now only 1/16th of the light given off by the wall is actually recorded by the camera. So you might expect the image to be darker. However the physical aperture (actually it's the entrance pupil but that's another story) of the 50mm lens is 50/4 = 12.5mm across, whereas the physical aperture of the 200mm lens is 200/4 = 50mm across. The 200mm makes more use of the light coming from the wall due to having a bigger hole, so the exposure ends up the same for the same shutter speed, as both lenses were f/4.