None of these lenses are going to approach L series lenses in terms of build quality and the ability to take punishment and just keep working as they should. There are reasons they offer near the same optical quality at 1/5 to 1/15 the price of an "L" prime lens.

I've had an EF 50mm f/1.8 II "nifty fifty" since 1997 or so and it still works fine for what it is. I used it quite a bit for about the first decade or so I owned it. Most of the "falling apart" stories are from folks who didn't take proper care to protect it from damage or try to force manual focus when the AF motor is connected to the focus ring. That being said, it is more fragile than most of its counterparts. But treated properly it is a perfectly usable AF lens.

Where it falls short is in the usability of the manual focus. There is no true focus ring, you must turn the very front of the barrel to manually focus it, and care must be taken lest your fingers wind up in the field of view! Since it does not have Full Time Manual Focus, one must also be sure to slide the AF/M to M (for Manual) before moving the focus ring manually. Attempting to force the ring to move when the AF motor is engaged can damage the gearing or the focus motor, even when the lens is not mounted on a camera. There are plenty of warnings regarding this in the documentation supplied with the lens.

I use an EF 50mm f/1.4 a lot now. In addition to the wider maximum aperture, it offers Full Time Manual Focus and has a usable AF ring. It is a much better lens, in my opinion, than the EF 50mm f/1.8 II. Better build quality, a faster, quieter, more precise micro-USM focus motor, a truly usable manual focus ring, and better bokeh from the 8 blade aperture (compared to the harsh bokeh from the 5 bladed f/1.8 at any aperture other than wide open) are some of the reasons why I recommend the EF 50mm f/1.4 over the f/1.8 II. I own both of them. I carried the f/1.8 version around in my bag for many years and used it somewhat frequently. Since upgrading to the f/1.4 I have found I pull it out more often due to the increased usability of the faster focus and the usable manual focus ring. The known mechanical weakness of the focus collar can be well managed by remembering to park the focus setting at ∞ when not actively shooting with the lens. When the barrel of the focus mechanism is fully retracted in that position it is not subject to the deformation that a drop or hard bump can do to it when the focusing barrel is extended. If the focus collar helicoids do get bent, it is a fairly easy fix for the DIY type.

Having said all of that, if I were looking for an economical 50mm prime today I would carefully consider the new EF 50mm f/1.8 STM. It is just as good optically as the f/1.4 and f/1.8 II. A lot of the build quality and usability problems of the f/1.8 II have been well addressed by the design of the F/1.8 STM. It sells for a fraction of the price of the f/1.4. The one thing that might be a downside for some users is the way the manual focusing ring works with all STM lenses. The focusing of the lens, even when manually focusing, is strictly fly-by-wire. Turning the focus ring on the barrel of the lens does nothing mechanically to move the focus elements of the lens. Instead, an electrical signal is sent to the camera body which in turn sends a command to the focus motor in the lens to move. This limits the smallest increment of focus adjustment to one "step" of the stepping motor in the lens. It also eliminates the ability to focus the lens when using "dumb" extension tubes or freelensing. You can't move the focus elements to park them when the lens is not attached to a powered up EOS camera body either.

In the end you must weigh the advantage of the wider Angle of View provided by the EF 40mm f/2.8 STM to the wider maximum aperture of the EF 50mm f/1.8 STM to the true manual focus (not focus-by-wire) capability of the EF 50mm f/1.4 USM.