Sensor size matters more than pixel size. Diffraction becomes an issue the smaller your aperture becomes and that means that as you stop down, a camera reaches a point where it is "diffraction limited". Basically, the area of light made by diffraction becomes larger than the size of a pixel and thus the resolution can not exceed that point.

The larger the sensor, the more pixels you can fit before getting too small, but if you are, for example, diffraction limited at 15MP and have a 24MP sensor, you will still get the equivalent of a 15MP photo.

There are also some minor trade offs because the light has to maneuver down a small tunnel to get to each photosite on the sensor. The smaller each pixel becomes, the harder it is for light to make it down that tunnel, so the smaller the pixels become, the more light sensitivity is lost, but this is often compensated for by making a sensor with lower noise levels, so it isn't necessarily a disadvantage in all cases.

So yes, pixel size does make a difference in multiple ways, but looking at just the pixel size doesn't give you the entire picture without looking at a number of other factors about the sensor performance. In this particular case, I would expect that the D3200 would have a newer sensor and likely be better in other ways other than just the resolution, so it's probably worth the increase in cost unless you don't need the features it is capable of providing.