Essentially yes, light gathering ability of a lens is determined by its maximum aperture. Transmission rates of the materials used also has an effect but it is very small.

You intuition is correct in that you would expect a large aperture lens to have a large barrel, however the aperture is specified as a ratio of the *apparent** size of lens opening divided by the focal length. So a 200mm f/2.0 lens must have a front element large enough to see a 200/2.0 = 100mm aperture, so the barrel must be at least 10cm. However a 20mm f/2.0 only appears to have a 10mm aperture, which is small is comparison to most lens sizes.

To complicate matters wide angle lenses need larger front elements than dictated by their aperture to prevent vignetting across the frame. For focal lengths shorter than about 50mm lens sizes increase as focal length decreases despite apertures, and thus light gathering ability, also decreasing.

Here's nice example, this Nikon lens is only f/2.8:

but is absolutely huge, due to its extreme wide angle nature.

* note that 100mm f/2.0 doesn't mean the physical opening in the middle of the lens is actually 50mm diameter, only that the image of said opening when viewed through the front of the lens appears to be 50mm in diameter. The actually opening is often smaller, but the lens front element has to be large enough to accommodate its theoretical size.