You can calculate that distance given the formula for the pinhole camera model (which is an approximation, but should work well in the cases of interest). You need to know the focal distance (which by assumption, you should know — in practice, you can usually extract it from the EXIF data of the image) and the size of each pixel.

For the size of a pixel, you don't need to obtain it "experimentally" from sample images to determine the ratio. If you know the size of the sensor and the resolution, you're done. For example, on a 24Mpix full-frame sensor camera, the size of the sensor is 36mm by 24mm, and resolution is 6000 by 4000 (approximately — for example, my camera is actually 6064 by 4040, and I think it is 35.9mm by 23.9mm or something like that). That means that the pixel to mm conversion factor is 166.67 pixels per mm, or 0.006 mm per pixel (pixels should normally be perfectly square, meaning that the factor should be the same in both axes).

So, given a subject of size s pixels in the image when shot at a distance d from the camera, your formula for the size of the actual object, S given a focal length f would be:

S / d = s / f (this is because the triangle between the pinhole and the image plane, with sides f (distance from the pinhole to the image plane) and s is proportional to the triangle from the pinhole to the subject (sides d and S.

Applying the conversion factor to express s in mm instead of in pixels, you get:

S = d (any units) · s (px) · 0.006 (mm/px) / f (mm)

This gives you the size of the subject in the same units that you used for the distance of the subject to the camera (since the mm of the pix to mm factor cancels out with the mm of the focal distance, and the pixels just cancel out)

EDIT:
As you do not have information on the geometry of the camera (focal length or sensor/pixel size), you would be left with what you suggested: knowing that the actual size of the object is proportional to the size in the image (in pixels) and also to the distance, by a factor that depends on the focal length and pixel size, then you determine this factor experimentally: take an image of an object of a known size, at a known distance, and obtain the factor. This factor would correspond to the 0.006 / f in the equation above:

S = d (any units) · s (px) · camera_scaling_factor

Therefore:

camera_scaling_factor = S / ( d · s )

(notice that the actual size of your reference object and its distance to the camera need to be in the same units)