If you're going to be moving around Manhattan all day, you'll probably want to bring some sort of bag with you to carry things like a light jacket (it can get chilly in the shade, or on the water, and certainly in the evening), a bottle of water, a map, a guide book, things you might buy along the way, etc. I'd recommend a backpack, but one that doesn't scream "camera bag!" If it's a little beat up, so much the better. Having a bag with you will give you a place to stash your camera when you want to blend in a bit better, like when taking busses or the subway, when when you want it out of the way, perhaps when you're in a restaurant.

Now, you certainly don't need to drag a big kit full of photo gear around the city with you, but you do want to have some flexibility. I'd recommend taking your zoom lens if you're only taking one lens. There are lots of interesting things in NYC that you simply can't get very close to: the ground, from tops of tall buildings; the tops of buildings from the ground; the Statue of Liberty from almost anywhere; etc. And there are also some great big things that you can't far enough away from to get in a single frame at 35mm: various façades; the craziness of Times Square; and the grandeur of the Brooklyn Bridge to name a few. Your 18-140mm zoom will give you a lot more options than will a single prime lens at any focal length.

But your 35mm lens probably weighs less than half a pound. Since you'll have a bag with you anyway, just toss your extra lens in there with the other things you'll need. If you're concerned about protection, buy a padded lens pouch big enough to hold either of your lenses. If you don't have time to get one before your trip, you can pick one up at B&H or any of the many camera stores in the city.