Usually, if either the camera body or the lens is set to manual focus, autofocus is effectively disabled, so you often only need to switch one. Many photographers simply leave the body in AF and use the lens switch.

The main caution is manual focusing a lens while the lens itself is still in AF mode. On some lenses, the focus ring may not engage at all, and on others forcing it can drive or fight the AF mechanism, which may cause extra wear. Some ultrasonic/full-time manual-focus lenses are designed to allow this safely, but not all lenses are.

So: switching only one control generally will not harm the camera or lens, as long as the lens is actually designed to be manually focused in that state. If you want the safest rule, use the lens switch when going to manual focus unless your lens specifically supports full-time manual override. Check the camera and lens manuals for the exact behavior of your combination.