A likely cause is not the time shift itself, but how the GPS position is being mapped from the recorded 3D coordinates to the 2D map used for geotagging.

Your airport shots geotag correctly, which suggests the basic clock offset is close enough. The bigger errors appear during flight, when both altitude and speed were much higher. GPS records latitude, longitude, and altitude, but the map display is only 2D. If the software, track log, or map datum/projection is mishandling that conversion, it can create a systematic offset—exactly the kind of parallel displacement you describe.

Because the error seems consistent rather than random, it points more to a coordinate/projection issue than to simple GPS drift. High speed might worsen position accuracy, but the pattern you describe sounds more like a mapping/conversion mismatch.

Check the coordinate system, datum, and projection used by the GPX file, GeoSetter, and the map source, and verify that altitude handling isn’t confusing the plotted positions.