Fortunately, it's not actually voodoo magic. You have set a fixed ISO of 400, and you're in aperture priority mode, which means you're choosing the aperture and not giving the camera control of that.

That means the one variable the exposure program can change is shutter speed. When you tell it you want the exposure to be two stops darker, the only thing it can do to meet what you're asking is to shorten the exposure duration.

Two stops should mean halving the shutter time twice, so one would expect it to go from ¼ to ¹⁄₁₆ rather than ¹⁄₂₀, but there's probably some rounding going on, and maybe slight changes in the exposure in the scene.

You ask how to solve the problem, but I don't think it's clear what the problem is here. It might help to read What is exposure compensation?, but it might also help if you can explain more clearly what you're aiming to do, and what you expect to happen when you change the EV compensation.

Assuming you can't change the lighting, the only way you can get a brighter exposure is by increasing one of shutter speed, aperture, or ISO. If you've fixed two of those, there's nothing to do but change the third.

If you need a faster shutter for your exposure, you're going to need to raise the ISO or open the aperture more. If neither of those are possible, you need to increase the light. If that's not possible, you are out of luck.