So I first shoot with ISO 1600 and shutter speed set to 1/125 second and then I shoot with ISO 3200 and shutter speed set to 1/250 second. The amount of light should be identical and indeed both shots look properly exposed and exposed the same way.

The amount of light is not identical. You let twice as much light into the camera at 1/125 second than at 1/250 second. You then amplified the image with half as much light to match the brightness of the other image. In so doing, you amplified the noise in the image by a factor of two as well. To let the same amount of light into the camera at 1/250 second, you would need to also open up the camera's aperture by one stop compared to where it was set at 1/125 second.

When we say noise in the context of an image, what we often really mean is the Signal-to-Noise Ratio (SNR).

In the context of digital photography, the 'signal level' is determined by the amount of light in the scene and how much of that light we allow to enter the camera by selecting a particular aperture size and shutter time.

The amount of actual noise is divided into two main types: Read noise and photon shot noise. Read noise is fairly constant for a given shooting condition. It can be affected by the camera's internal temperature, but it won't usually change from one shot to the next under the same general environmental conditions. Photon noise, often called 'shot' noise or 'Poisson distribution' noise, is variable and increases with the amount of light, but increases at a lower rate than the increase in light.

Adjusting ISO affects how much the signal and noise combined are amplified. Increasing the ISO setting multiplies both the signal and noise equally, so the SNR remains the same. However, as the image is amplified more it becomes brighter and we may be able to see more of the noise in the image that was previously masked in the darker shadows of the image.

The end result is that the more light that is allowed into the camera, the less influence the noise will have on an image because the SNR is increased by adding more additional signal (light) than additional noise.

The advice in the answer referenced at the beginning of the question is to be understood as "... let enough light into the camera and then use the highest ISO value that doesn't result in blown highlights. If enough light is let into the camera, that ideal ISO could well be the camera's minimum ISO setting. Only when we are limited by the amount of available light, the camera's maximum aperture width or the aperture setting needed to get a desired depth of field, or the shutter time needed to prevent blurring a moving subject does the advice to increase ISO to the highest setting that does not overexpose the highlights become applicable to an ISO setting higher than the camera's baseline ISO.