A lens is designed to project an image circle of a certain size in order to cover an image area (sensor) of a certain size... and for that image circle to be in focus at that distance (flange distance). If you arbitrarily change the distance you change the crop factor (size relative to the image circle) and adversely affect the lens' ability to focus at all distances.

What you are describing is more commonly known as "bellows factor." If you attach a lens to a bellows, when you move the lens father away from the image area/sensor the projected image circle spreads out more over the increased distance (although the additional area/size created may be masked out). This causes an increase in the effective magnification (increased crop factor); and simultaneously causes a reduction in the light density at the image plane (resulting in a smaller effective aperture). It also prevents the lens from focusing at longer distances (incorrect/long flange distance).

And the opposite is also true... less distance = less projected image spread, less magnification/crop factor, greater light density, and an inability to focus as close.