Cut film holders have a bit of "play" to allow the film to slide into the holder with the minimum of force. They are engineered this way.

After the holder is loaded, the end-flap on the holder is closed which holds the film securely along the short edge.

The dark slide is then inserted which holds everything securely in place.

When the film holder is put into the camera back and the slide is withdrawn for the exposure, the end-flap is then held closed by the camera back which, in-turn, clamps the sheet of film in place.

After the exposure is made and the dark-slide is re-inserted, the holder can be removed from the camera back with the dark-slide holding the end-flap tightly closed.

This is normally sufficient. The clearance is slightly less than the thickness of the film. A vice-like grip isn't necessary.

What you might try is a consistent practice of loading your holders.

For example: Very lightly tap the holder to ensure the film has smoothly dropped into the holder. When the sheet is squarely seated, close the end-flap to hold the film sheet in place.

When it is more of an issue, such as for technical requirements, a vacuum-back is used which sucks the air out of the space pulling the sheet of film flush with the film holder back so there is no gap.

EDIT: Here's how to diagnose the source of the alignment problem:
Examine a problem negative.

If the clear film holder guide marks are parallel to the cut edge of the film as shown in example 1, it indicates that the film was in the holder correctly. It also implies that the holder, camera back, camera, etc. was not parallel.
Your set-up was problematic rather than the film position due to film-holder tolerance.
BUT:
If the camera was set up level and perpendicular to the horizon, the film holder marks will be parallel to the horizon in the image because the film holder and the camera back were also level and perpendicular. The clear edges along the edge of the film will not be parallel to the cut edge of the film as shown in example 2.

Consequently, the problem is as you suspected and despite the alignment of the film-holder marks along the edges of the image itself, the horizon will appear slightly tilted by the same amount due to misalignment of the film in the holder.

My opinion is that it will not be so severe that it will compromise your product adversely.