Don't use Photoshop's Photomerge to stitch; use a more sophisticated panorama stitching package that allows for perspective correction and a variety of mappings. You also need to work on your shooting technique.

The main issue here is the perspective, and the panorama's mapping. Some of the recommendations here, though, don't make a lot of sense. A tilt-shift lens might help with the extreme sort of keystoning you have here, but TS lenses are typically limited to only about 8° of tilt and 10mm of shift, and being designed mostly for full frame, they won't go ultrawide on a crop body, and shifting only gives you so much coverage. You really need a view camera where you can manipulate both the lens and image plane to correct for keystoning that extreme. Unless you have the correct working distance and focal length, a $2000+ 17mm TS lens isn't a practical solution for a lot of shooters.

However. You can correct for it in post with software, such as Hugin or PTGui. You can do this while you're stitching using vertical and horizontal control points. And you may be able to correct most of it on a stitched pano with simple dragging in the preview window (dragging horizontally corrects yaw, dragging vertically corrects pitch). But you also made some shooting errors.

Your camera is shifting position between shots, as well as varying in pitch and that makes for some weird distortions when you get around to stitching. My recommendation is to stop handholding and use a tripod, preferably with a head where you can rotate around the no-parallax point--although that's probably overkill unless you're within 10' of the building, or at least consider using a plumbline. If you must go completely freehand, in a pinch, consider balancing the camera's tripod hole on your thumb, to try and hold the camera in place, as you rotate it, to minimize movement.

In addition, you need to consider using projections other than equirectangular or cylindrical (the only two Photomerge can handle), and think possibly about spreading out to projections like architectural, Panini, or stereographic. Again, stitching applications like Hugin and PTGui allow you to select from a number of different projections.

You also need to consider shooting coverage in time as well as space, so that you'll have enough "clean plate" background so you can mask out ghosts and clones when subjects move through your member shots.