Unfortunately I had the exact opposite criteria as you :) I looked at every software I could which did NOT touch the files and stored the information in a central database. The one constant is that having a central database makes it much more efficient to perform searches and filter your images.

I do have hundreds of gigs of photos and the most efficient software to do this is Bibble Pro. They claim to be 20X faster than the competition and I measured at least 5X for most (not all, most softwares have perform well at least something) except for PicaJet FX which was close in speed.

PicaJet actually stores EXIF/IPTC edits in its database AND the files, so it may suit you. You can get a free evaluation copy. Read about PicaJet here.

Unfortunately I did not end up using Bibble Pro because it would not import my very large panoramas and PicaJet because it was destructive. I consider my files sacred :)

So, I finally went with Lightroom which is highly regarded. It is reasonably fast (still 5X slower than Bibble), supports non-destructive editing (so no need to duplicate storage to crop to different paper sizes!) and good support options and online resources. For about 280GB of images (44K) it starts instantly and finishes its 'refresh' (what the hell is it doing? I don't know) in less than 2 mins. Most simple (2-3 criteria) queries complete in under 1 min.