If you backup the library without the new images you would lose all of the new working copies of the images. That is how referenced images in iPhoto works. I would recommend against doing anything but backing up the entire 20GB library file unless you want to lose your changes.

A better option is really to attack the actual issue here. A single iPhoto library of that size gets unwieldy because iPhoto is simply not designed to work with that large of data sets. Apple has the program Aperture that does handle larger collections of photos, and I would also consider Adobe Lightroom as well. Not only will the program performance be significantly better, but then you extend your options to natively work with multiple libraries and also you will have options to export the changes to sidecar files(XMP in the case of lightroom).

A workaround would be to break up your library into a few smaller libraries using something like this: iPhoto Library Manager