If you almost never change the diopter, the simplest fix is to set it correctly and secure it with a small piece of gaffer tape. That’s a common, removable way to stop a control from being bumped.

You can also mark the correct position with a small paint dot or similar reference mark so you can quickly see whether it has moved and reset it fast.

Your carry method may be part of the issue. A strap attached to the tripod socket carries the camera in a way the body wasn’t primarily designed for, and hanging it upside down against your body can make controls rub or get knocked. Using the camera’s regular strap lugs may reduce how often the diopter gets bumped.

So: tape it if you want it fixed in place, mark it if you want a quick visual reference, and consider switching to the standard strap attachment points if the current setup is causing repeated contact with the diopter.