This can happen because your D5300 uses two different autofocus systems.

Through the viewfinder, it uses phase-detect AF, which is faster and generally more capable. In Live View, it uses contrast-detect AF, which is slower and often hunts more before locking focus. So a lens can appear to focus fine through the viewfinder but struggle in Live View.

Based on the answers provided, the most likely explanation is the normal difference between Nikon’s viewfinder AF and Live View AF performance, rather than dirt in the viewfinder or a camera body fault. The dust you see in the viewfinder would not affect the recorded image and is not likely the cause.

If the problem became noticeably worse only recently, try testing in good light and on a subject with strong contrast, since contrast-detect AF is more sensitive to those conditions. If it still consistently fails only with this lens, the lens may simply perform poorly with Live View AF on this body.