A greater zoom range means a more complex design at greatly increased cost. There are some excellent lenses make for broadcast with incredible zoom ranges, like the Fujinon 8-832mm (yes that's not a typo!), but you don't want to know the price.

Designing a lens with a larger zoom range at a lower price does lead to compromises on quality.

Finally lens technology moves very slowly compared to say camera sensors or processing chips, the nature of the game means you can't just take the same pixel design and shrink it for a quick win in performance. Advances come from better design tools more experience and improvements in manufacturing/quality control, all of which are gradual.

There have been advances recently but that's being driven by photographers becoming more demanding in terms of resolution, and those advances come at a price - as lenses are replaced with new designs the price is almost always higher.

Now you can get a "good" 28-300mm telephoto lens from Canon, but it's £2000, and not as good optically as the £1000 300mm f/4, which is a prime lens with no zoom. Canon could make a 28-300mm lens that was as good or better than the 300 f/4 prime, but it would be ten times the price.

Another good example is the Canon 70-200 f/2.8 zoom and the Canon 200 f/2.8 prime. The zoom is actually a little bit better than the prime at 200mm. But it's 3x the price and has a modest 3x zoom range. The wider the zoom range the more you'll have to pay to match the performance of a prime or smaller zoom range lens.