How small can a full-frame camera get without sacrificing image quality?

The Sony Cybershot DSC-RX1 is a demonstration of a full-frame camera in a very compact body. They could made the lenses interchangeable with some effort but it would not add much bulk. If you exclude the lens, the body is less than 1.5" thick.

A camera like the RX1 saves bulk compared to a full-frame DSLR because it can mount the lens much closer to the sensor since there is no mirror in the optical path and no pentaprism to direct an optical image to the viewfinder. So, if all you care about is image quality and neither ergonomics nor an optical viewfinder matters to you, they can make it much smaller than a DSLR, as small as an RX1 at least!

Lenses for a camera like the RX1 are only marginally smaller because the full-frame image circle must still be covered. For this reason, the size advantage is most significant for normal or moderate wide-angle lenses, my guess is between 28 and 75mm. Beyond that, the lens will dominate the camera size.

Now you did not specify a time-frame but technology evolves and while larger sensor always have an advantage based on the laws of physics, smaller sensors keep improving. So today we have APS-C sensors that match the performance of the previous generation of full-frame ones. In a few yours, APS-C sensors can match today's top full-frame models but - of course - by then newer full-frame models will be even better. In other words, within 3 years, I would expect a camera like the Fuji X-E1 to match today's best full-frame cameras.