No, it's not all that bad. On most shots, you won't be able to tell by eye if the curvilinear distortions have been corrected for or not - you'll need straight lines adjacently parallel to each other or to edge to tell. In nature, you won't have those lines. In portraiture, it will exaggerate some parts and diminish others, but perspective distortion and lighting has much more effect on result.

The reason you've heard so much about them is that you've read many lens reviews, and distortion is rather easy to measure and write about.

Possible downsides of shooting distorted and correcting in post:

• you'll spend more time setting up and verifying your workflow;
• in your heart, you might want to do as much as possible in camera (many cameras can correct JPEGs for manufacturer's brand lenses);
• some contests have quite strict rules against manipulation in digital post-processing;
• as Jerry Coffin already commented, slight loss of sharpness;
• as you mentioned in your question, it's easy to correct if you're shooting RAW and your software knows the lens. These conditions might not be true for various reasons.