We can't tell you whether the difference in image quality is worth the price, because that's a personal decision based on your circumstances.

What we can tell you is the advantages the EF-S 24mm will give you over your kit lens:

• A stop of aperture (the 18-55 has a maximum aperture of f/4.0 at 24mm). That's never something to be sniffed at.
• Physically much smaller and lighter.
• Non-trivially better (but by no means top of the range) image quality.
• STM autofocusing as opposed to the micromotors in your 18-55. The main advantage here is going to be they're much quieter - a significant advantage if you ever want to do autofocus with video.
• No rotation of the front element when focusing. If you ever shoot with polarised or graduated filters, this is a big advantage.

[ The last two points would go away if the comparision were with the latest STM kit lens ]

And one potentially big disadvantage:

• No image stabilization. If you're used to relying on that for your shots, this lens isn't for you.

The other point to note is that 24mm on an APS-C camera isn't very wide - equivalent to 38mm in full-frame terms. However, as you're already shooting with a lens that can manage 24mm, you probably know what you'll be getting here.