Y' is designed to approximate brightness by definition: the RGB channels are combined with weighted coefficients chosen to match human visual sensitivity, which is greater for green and lower for blue and red. That is why the Y' channel, by itself, looks like a reasonable grayscale image.

However, Y' in Y'CbCr is luma, not true luminance. It is derived from gamma-corrected color signals, so it is only an approximation of perceived brightness. This causes known errors, especially for strongly saturated red or blue areas, sometimes called constant-luminance error.

Historically, this design was also practical for television: it provided a brightness-like signal compatible with black-and-white display while carrying color separately in the chroma channels.

So the “magic numbers” are not arbitrary. They were standardized from human vision experiments and engineering requirements, with green weighted most heavily, then red, then blue. The result is useful and intentional, but not physically exact luminance.