What's the point in high quality rather expensive cameras with permanently mounted prime lens?

To take pictures.

By creating a camera intended for only one focal length a lot of things cans be optimized for that particular lens. The camera can also be simpler with no need to adapt to a variety of different lenses and focal lengths. A through-the-lens viewfinder is not necessary since the field of view is constant. Each location on the sensor can be optimized for light from an exact angle. The various connections between the main body and the lens can be stronger and more permanent than on an interchangeable lens camera.

There's also a bit of marketing going on with such cameras. By using traditional high quality external materials and by making them seem exclusive and "not for everyone" they attempt to give them the same cachet that similar film cameras once had. In the case of Leica, it's even the same company that has made many such niche cameras for seven decades. Leica has been making cameras for a century. Following increased competitions from Soviet and Japanese cameras following WWII, they established themselves as the leader in the high-end rangefinder market space.