The only other factor you need is the height of the object in real life (otherwise you could be photographing a model which is much closer to the camera).

The maths isn't actually that complex, the ratio of the size of the object on the sensor and the size of the object in real life is the same as the ratio between the focal length and distance to the object.

To work out the size of the object on the sensor, work out it's height in pixels, divide by the image height in pixels and multiply by the physical height of the sensor.

So the whole sum is:

$$ \textrm{Distance to object}(mm) = \frac{f(mm)\,\times\, \textrm{real height}(mm)\,\times\, \textrm{image height}(pixels)} {\textrm{object height}(pixels)\,\times\, \textrm{sensor height}(mm)} $$

Let's sanity check this equation.

If we keep everything else constant and increase the focal length then the distance increases (as focal length is on the numerator). This is what you would expect, if you have to zoom your lens to make one object the size another equally sized object used to be, the first object must be further away.

If we keep everything else constant and increase the real height of the object then again the distance increases as if two objects of different real heights appear the same height in the image the taller one must be further away.

If we keep everything else constant and increase the image height, then the distance increases, as if two objects (of the same size, remember we're keeping everything else constant) appear the same pixel size in a cropped and uncropped image then the object in the uncropped image must be further away.

If we keep everything else constant and increase the object height in pixels then the distance decreases (we're on the denominator now): two equally sized objects, one takes up more pixels, it must be closer.

Finally if we keep everything else constant and increase sensor size, then distance decreases: two equally sized objects have the same height in pixels when shot with a compact (small sensor, where 20mm is a long lens) and shot with a DSLR (large sensor where 20mm is a wide lens), then the object in the DSLR image must be further away (because it appeared the same size but with a wide lens).