A likely cause is a small mechanical interference at the F-mount. On the extension tube there is a little tab at the edge, and on the camera mount there is another spring-loaded tab on the rim that should be pushed down during mounting. If those two tabs catch against each other, the tube will insert but won’t rotate far enough to lock, so the camera won’t detect it.

Since your lenses still mount and the tubes still fit another Nikon body, the issue is probably not the whole mount, but that specific tab interaction on the D5100. Carefully inspect the mount edge and the tube’s tab for anything misaligned or stuck, and make sure the spring-loaded tab on the camera moves freely. Do not force the tube into place, as that could damage the mount.