What am I doing wrong?

You are holding down the shutter button half-press while moving the 'AF/MF' switch on the lens. It seems somewhere between the "AF" and "MF" position some contacts are being made in the switch that is providing current to the AF motor in your lens, causing it to move.

Assuming you are using 'One Shot' AF mode (which is what you should be using to do what it seems you are attempting to do), all you need to do is:

• Half press the shutter button (or better yet, remap AF to the 'AF ON' button on the back of the camera so that the shutter button half-press only affects metering)
• Wait until the camera confirms focus by displaying the green dot in the lower right corner of the viewfinder
• Release the shutter button half-press (or the AF ON button)
• Move the 'AF/MF' switch on the lens from 'Autofocus' to 'Manual Focus'.

If you remove AF from the shutter button half-press and assign it only to the 'AF ON' button, you can do the same thing when in 'AI Servo' AF mode. As soon as you release the 'AF ON' button the camera will stop focus tracking.

There is a difference if your lens is the EF-S 24mm STM, rather than any of the EF 24mm f/2.8 lenses with USM AF motors. STM lenses have a 'focus-by-wire' system with no mechanical connection between the focus ring and the focus elements of the lens. When the manual focus ring is moved it sends electrical signals back to the camera which then uses the STM motor to actually move the lens' focusing elements, even when the 'AF/MF' switch is set to "MF." The focusing elements of an STM lens can not be moved when metering is not active. (Metering is active any time the exposure information is lit up in the viewfinder. Metering is inactive when the exposure information is not lit up in the viewfinder.)

Although there is no current 24mm 'nano USM' lens, for the purpose of completeness for others that might find this question and are using a 'nano USM' lens: Nano USM lenses are also all 'focus-by-wire'. They use a linear, rather than ring, USM motor to move the lens' focusing elements.