Have a DOG sniff out blur in the photos.

If you're going to be penalizing for digitally enlarged photos, you might as well penalize for out-of-focus photos too. The blurred edges and details in both cause the same bad experience for viewers, regardless of whether it is caused by a small original or poor focus. What you want to do is detect blur, which is an absence of high spatial frequencies.

Try taking the difference between an image and a blurred copy of itself. If an image is already blurry, a 1-pixel Gaussian blur isn't going to change the image as much as if the image were sharp. So there will be more difference between a sharp image and a blurred version than there is between a blurry image and a further blurred version. In computer vision, this technique is called the "difference of Gaussians" (DOG).

1. Open the image in GIMP or another layered photo editor.
2. Duplicate the layer.
3. Apply a Gaussian Blur with a radius of 1 pixel to this new layer.
4. Change the layer mode to "Difference". The image will go black except for the edges.
5. Repeat steps 1-4 for a known sharp image of similar subject matter, composition, and size.
6. Compare the intensity of the edges in the two difference images. You can eyeball this or use a histogram.

I just tried this on a 400x480 pixel photo and on the same thing that had been reduced to 200x240 (50%) and then enlarged back to 400x480 (200%), and the edges in the upscaled photo were quite noticeably fainter. It won't be conclusive on a mild enlargement such as 140%, but it will catch blatant cases.

Several computer vision libraries include means to calculate difference of Gaussians on an image. So do many graphical image editors. Recent versions of GIMP, for example. include a DOG macro that automates steps 2 through 4: Filters > Edge-Detect > Difference of Gaussians, then set the radii to 1.0 and 0.0.

Related questions on other Stack Exchange sites:

• Difference of Gaussian, Laplace of Gaussian, and Mexican Hat wavelet on Signal Processing Stack Exchange
• OpenCV: Efficient Difference-of-Gaussian on Stack Overflow

DOG won't catch nearest neighbor, but you can do that by looking for a pattern of rows and columns that are identical to their immediate neighbor toward the top or left.

1. Open the image.
2. Duplicate the layer.
3. Offset the new layer one pixel up or to the left.
4. Change the layer mode to "Difference".
5. Look for a pattern of blank lines.