Most likely this is from the lab/printing workflow, not stray light or your yellow filter.

Ilford XP2 is a black-and-white film that is developed in C-41 chemistry, but if it’s printed on standard color paper or handled in a color minilab workflow, color casts can appear. Labs that mainly process color film often run XP2 through their normal C-41 system and print it as if it were color, which can produce arbitrary tinting rather than a neutral black-and-white result. A proper black-and-white printing workflow is more likely to give a neutral result.

Since you see the effect on both the print and the scan, and you confirmed it appears in the negative, the issue is most consistent with how the film was processed/scanned/printed by the lab rather than sunlight hitting the lens. Stray light would more typically show as flare or uneven loss of contrast, not a consistent blue cast from a black-and-white negative.

If you want cleaner neutral tones from XP2, try a lab experienced with black-and-white film and ask for neutral B&W scans/prints.