Your best chance is to start from the original source, not from an existing print. If these were shot on film, the original negatives are far better for enlargement than scanning a print. If they were digital, the original camera files are the best option.

A print is already a “final output,” so enlarging it a lot usually reveals softness, grain, and dust. A lab’s warning about insufficient resolution is worth taking seriously.

If all you have is the print, ask the lab whether they offer high-resolution scanning. Make sure the print is very clean before scanning, because dust and defects will enlarge too. But going all the way to 30×40" from a small print is often unrealistic. Roughly speaking, a 2× enlargement from a print might be acceptable; 3× often becomes noticeably worse; 4× is generally not recommended.

Enlargement software may help a little, but it cannot truly restore detail that isn’t there. So before buying software, first try to locate the negatives or original digital files. If those can’t be found, consider making a smaller print size, which is much more likely to look good.