I own both lenses. I can't imagine the price difference between them being "small". So I just looked, $700 vs $1100... Both new, from reputable vendors. I wonder therefore if we are talking about the same glass.

As with any two lenses, there are pros and cons to each, off the top of my head, here's what I would say for each:

EF 24-105 f/4 L Pros:

• razor sharp
• constant f/4 all the way from 24 to 105
• fast auto focus, all the way out to 105, due to the constant f/4
• well understood lens, supported by lots of correction software
• minimal distortion, easily corrected
• weather sealed, better construction

Cons:

• heavy
• bigger
• needs bigger, more expensive filters
• expensive

EF-S 15-85: Pros:

• cheaper
• smaller
• lighter weight

Cons:

• can only be used with an aps-c body, if you ever want to move up to a full frame body down the road, you won't be able to use this lens.
• the CA is pretty bad
• as a less prevalent lens, especially in the pro space, the correction maps for it aren't as smooth as other lenses. If you are at one end or the other they are solid, if you were somewhere in the middle... They seem to just approximate it, leaving kinda less than stellar or consistent results.
• variable max aperture
• slower autofocus on most bodies, and progressively slower as you zoom in and the max aperture opens up.

Now, you say you are doing mostly portraits with this lens, so you won't be using the extreme ends of the zoom capacity. The 105 end of the 24-105 is a magnificent portrait lens, and being able to stay stopped down to f/4 at that end is a wonderful thing for shooting people. (That just never sounds right.)