This look is commonly called swirly bokeh. It’s mainly caused by lens characteristics rather than a camera setting.

According to the answers, the classic way to get it is with a Petzval-type design or certain vintage lenses known for strong field curvature and vignetting. Those optical traits make out-of-focus highlights near the frame edges appear to curve into a swirl. Helios lenses are often mentioned for this look.

To get it in-camera:

• Use a lens known for swirly bokeh, often a fast vintage lens.
• Shoot wide open or close to it.
• Place your subject near the center of the frame.
• Use a busy, detailed background, especially foliage or small highlights.
• Keep some distance between subject and background so the background goes well out of focus.

One answer also notes that using a circular cover/mask over a fast lens can encourage a swirl-like background effect.

So: this is not usually a standard shooting technique by itself—it’s mostly an optical rendering style produced by specific lenses and emphasized by the right background and aperture choice.