They affect quality in different ways.

Shrinking an image discards pixel information. Once those pixels are removed, that detail is gone and enlarging the smaller file later will not restore the original. In that sense, downscaling is more destructive to the underlying data.

Enlarging does not create new real detail. The software adds pixels by interpolation, estimating values between existing pixels. So you keep the original information, but spread it over more pixels, which can make the image look softer or blurrier.

Why does enlarging often look worse? Because it makes flaws easier to see: blur, noise, compression artifacts, and jagged edges all become more visible. Shrinking often hides those same problems and can even make a marginal image look better because noise and blur are less noticeable at smaller sizes.

So: downscaling loses information permanently, while upscaling usually hurts perceived sharpness because it cannot invent detail. Which looks “worse” depends on viewing size, distance, and the resampling method used.