Out-of-focus highlights show the shape of the lens aperture, not the shape of the subject. A point of light that is not in focus is rendered as a blurred disk called a circle of confusion. That blur is effectively an image of the lens opening as seen through the optics.

If you put a heart-shaped mask over the lens so it acts as the effective aperture, each defocused point light is blurred into that heart shape. In-focus parts of the scene do not become hearts because rays from each subject point are still brought together at a single image point, so the aperture shape is not directly visible there.

This effect is most obvious with bright specular highlights and shallow depth of field, because those produce larger blur disks. If the mask is too large, off-center, or causes heavy vignetting, the effect can be weak or uneven. In practice, a custom bokeh mask works by changing the entrance pupil/aperture shape the lens presents to out-of-focus light.