Scanning software performs a good deal of "magic" in the background -- color correction, auto exposure, and so forth. What's probably happening here is that your Epson Scan is getting the "magic" wrong because it's not set up for cross-processed E-6 film.

At the very least, the orange mask the software expects in color negatives isn't present, which will cause color casts and crossover (where the contrast curve for one color channel doesn't match that of another). That's the "cross processed look", so it's not necessarily a bad thing, but it means the scanner software can't do what it would usually do.

You're getting a result more like you expect by photographing the negatives and manually inverting the image, so what I'd suggest is to scan them as color positives (transparencies), and be sure to include some of the rebate (film between frames) in the scans; then in your photo editor set the white balance to the unexposed film you included in the scan before inverting the image. That should get you a color positive that you can then adjust, and in the process of editing you can also adjust brightness and curves to restore a normal-looking exposure (if you choose).