Aesthetic considerations of the image aside, it's just mechanically obvious that a square fits inside of the image-circle produced by the camera's lens a lot better than a rectangle does.

The short answer to all of this is that this premise seems logical, but is actually incorrect. It's true that a square is the most area, but that's only the best use of space if you want a square in end.

To put it in concrete terms, a 35mm frame of 36×24mm has a diagonal of about 43.3mm — so we need a circle of that diameter. With our handy high school geometry, we can figure out that the largest square that can fit in that same circle is about 30.6mm on a side. (See Would it be possible to make a 36×36 mm "full frame" sensor? for some example diagrams showing how this works.)

So, if a wider frame is desired, 36mm is reduced to 30.6mm — almost 20% less. That's not "a lot better" at all for that case. If you look at What historic reasons are there for common aspect ratios? — or other aspect-ratio questions, you'll generally see that while square is interesting and popular, there's a historic preference for somewhat-off-square composition.

For almost all of these other ratios, a 3:2 sensor is better — even with almost-square 5:4, that comes to 30×24mm, which is close enough to not make a big difference.

Of course, this could be solved by making a much larger sensor, exceeding the size of the image circle — the corners would vignette but you could crop however you want. But this would be a lot more expensive — the sensor would have to be bigger and so would everything around it. In the case of DSLRs, there would need to be a lot more clearance for the mirror. The shutter would have farther to travel. Overall, the market has basically judged this to be not worth it.