A higher f-number (technically a smaller aperture) contributes to sharpness in two ways. Firstly the depth of field is increased, thus objects which would appear blurry are now rendered sharp. Secondly a smaller aperture reduces aberrations which cause the image to appear soft even at the plane of focus.

In a perfect lens light coming from an object spreads out, passes through the aperture and then is focussed into a dot on the film/sensor plane. However real lenses suffer from aberrations, such as spherical aberration whereby the light passing through the aperture isn't all focussed at the correct distance, light passing through the edges of the aperture might come into focus in front of or behind the sensor, and thus not form a precise dot but a smear. Closing the aperture simply blocks the light from the edges thus it can't have a softening effect on the image.

image by Lookang

If you make the aperture too small then diffraction occurs, whereby light spreads out, again causing a softening of the image. So for every lens there is a crossover point where the increase in sharpness from reducing aberrations is balanced by the decrease in sharpness from diffraction. This is the "sweet spot"