I wouldn't clean lens elements with used sensor swabs. Any unseen dust that had been removed by the swab could scratch the coatings/glass of the lens element. Other unseen contaminants could also wind up on the lens element.

I would consider using the plastic part of the swab with a clean Pec-Pad attached to the tip of the swab. Pec-Pads are actually made for cleaning optical lenses and are quite affordable. I even know some folks who use Pec-Pads attached to sensor swab stems to clean camera sensors. The same company that makes one of the most popular sensor cleaning swabs also makes Pec-Pads, and many have theorized that it is the same material.

As far as the cleaning fluid goes, most sensor swabs come with fluid that is a mixture of distilled water and highly purified denatured methyl alcohol. Many lens cleaning fluids are less pure versions of the same stuff. It's just as safe to use as any other alcohol based lens cleaning fluid would be.