Depends what sort of AP you're talking about. And whether you're using a standard photo tripod or a proper astronomical tracking mount.

For a standard tripod, you're limited in exposure by star trailing; as the earth rotates, the stars appear to move across the sky. Unless you're intentionally after star trails, with a fixed tripod the length of time you can expose for is determined by focal length (and whether you're aiming near the celestial equator - stars there appear to move faster than ones near the celestial pole). Longer focal lengths have a larger image scale, so stars will drift across the field of view faster. as a rough guide, something like 400/focal length is a ballpark limit for exposure times (people use different numbers depending on how picky they are).

For longer exposures, you need a decent equatorial tracking mount (especially with longer lenses and telescopes) although a "Barn Door" DIY mount can (with practive) give you decent results at shorter focal length.

The problem with quite a few lenses - especially very fast ones - is that they're not optimised for infinity focus - so you may need to stop down by a few stops to reduce lens aberrations. Some lenses perform well wide open and are good for AP as a result. Others - including older 50mm fast lenses from the film days - have problems wide open for AP, so that although they're OK for normal use, there's not much point in buying one specifically for AP since by the time you stop them down enough to get decent quality for AP you might as well have gone for the cheaper, slower versions.

So what you want is not so much a fast lens in terms of maximum aperture, but one that's still fast when stopped down enough to give good quality results.

For long exposure astrophotography of deep sky objects with a telescope, the most important thing is a good equatorial mount (There's a saying that the three most important things for AP are the mount, the mount, and the mount :) ).

It doesn't matter how good your optics are if the mount can't track the target accurately for the whole exposure time.

Once you can do that, how fast your optics are becomes less of an issue - though since it can take hours of total exposures for a good image, fast scopes still have an advantage (though slower ones give you a larger image scale), so you see a wide range of telescope f-ratios used - mostly in the f4 to f8 region, but going down to around f2 (Celestron SCTs with Starizona's hyperstar) and up to f10 (native SCTs) or f13 or so (Maks).

Planetary and lunar imaging is different - they're bright enough that you can capture images at video rates so a long focal length (for the bigger image scale) and large physical aperture (i.e. lens/mirror diameter) for higher resolution are more important (top planetary imagers use large (9-14") SCTs with 2000mm+ focal lengths).

To reduce thermal noise, some DSO images use specially cooled astronomical CCD cameras - reducing the temperature of the sensor (down to well below freezing point, in some cases) dramatically reduces the thermal noise.

Combining ("stacking") multiple images in free software like Deep Sky Stacker can also dramatically reduce overall noise - random noise tends to average out while wanted detail remains present.