Usually there’s no direct image-quality penalty just from leaving continuous burst enabled. For portraits or general shooting, the main downsides are practical:

• You may shoot more frames than needed, which means more time culling and processing later.
• Extra frames add shutter actuations, which some people prefer to minimize.
• If you fill the camera’s burst buffer on less important shots, the camera can slow down while it clears, and you may miss a better moment.
• On some cameras, burst behavior can affect usability: slower review, limited preview, or lag in live view while shooting.
• Autofocus behavior matters: with focus priority, there’s usually little downside; with release priority, continuous shooting can increase the chance of out-of-focus frames.

So: leaving burst mode on generally won’t hurt your photos by itself, but it can create workflow, buffer, and sometimes focus-tracking issues depending on the camera. Many photographers still use short bursts even for non-action because one of the later frames may be the sharpest or have the best expression.