Usually yes: if the lens was designed with a rear drop-in filter slot, the manufacturer often expects a clear or other filter to be installed. Manuals for some Nikon telephotos explicitly say to always use a filter.

Optically, a flat parallel filter does affect the light path, even though it has no optical power like a lens element. Its presence slightly shifts the focus position (roughly on the order of about one-third of the filter thickness, depending on material). In some lenses, refocusing may compensate; in others, especially where internal focusing groups are involved, removing it could move the lens away from its intended design condition.

A good filter is an optical flat, so it should not significantly add aberrations by itself, but adding or removing it can alter spacing enough to matter slightly. Filters also add two reflective surfaces, which can increase flare and reduce contrast, though coatings minimize this.

So the practical answer is: if the lens manual says to keep a rear filter installed, use one—typically a clear/neutral drop-in filter—if you want the lens to match the maker’s intended optical performance.