Your intuition is essentially correct but there are a few important points.

1. When the lens is stopped right down, only light heading for the centre of the front element will make it into the picture, so the whole front element isn't used for every point of light hitting the sensor (though all of it is used for some point of light).

2. Even when the aperture fully open and light is passing through the whole front element for every pixel, the filter will be a small but significant distance in front of the lens.

Time for some crudely handdrawn diagrams. For the first case, it's easy to see that if the filter is placed anywhere except dead level with the iris, then it's going to block light coming from the top of the scene but not the bottom:

Now consider the second case. The lens is wide open, and bar the effects of vignetting, each point of light from an object in focus spreads out, hitting the whole front element before being focussed down to a dot. This suggests that light from the top of the object and bottom of the object both pass through the same amount of ND filter, meaning there is no graduated effect. However the filter wont be mounted flush with the front element:

When slightly in front there will be a angle at which light just skirts under the filter when coming from the bottom of the scene, whilst light from the top is partially filtered out, leading to a difference in brightness between top and bottom.

However if the lens is very fast, and the filter is mounted quite close to the lens then the grad ND really wont work, as suggested by your question. To prove this I've taped a black card over a 50mm f/1.4 lens so it's just above the centre, covering almost half of the front element:

That's wide open, pointing at the sky (focus distance 3m). There is a slight vignetting visible at the top, but nowhere near the effect you'd expect from what is essentially a graduated filter that goes abruptly from 0 to 100% filtration! Here's the same shot at f/11: