The answer depends on the purpose of the photos. As you've already discovered, each color has its own effect and quirks. Adequate lighting that does not create unwanted shadows is going to be essential with any kind of background, because those shadows will also manifest on subject.

Background and lighting for passport or other document pictures is usually regulated and light gray is usually a safe bet. Refer to guidelines provided by the receiving authority.

For more beauty-oriented memorabilia, I would look for complementary color of the subject. Most often, the subject is a face, with tone ranging somewhere between dark brown and yellowish pink; complementary colors then fall into some bluish shade, from teal to purple. For ladies with bright red lips, dark green would be a nice complement. I would also prefer a darker background, as light areas steal attention and show shadows more clearly.

But if you aim for the true meaning of portraiture, really portraying the subject, there is no universal answer. By definition, a portrait should underline the personality of subject, and background has to be selected accordingly. The natural environment for subject without any backdrop might work the best.