In the US, taking pictures on private property without the permission of the property owner is illegal. This is why security guards can and will stop you from taking pictures at, say, the Westfield chain of malls. First Amendment rights only apply if you are on public property and your subjects have no reasonable right to privacy.

A person in their own home has their right to privacy. And by taking pictures of them, their home, or their belongings, is invading that privacy. To quote from The Photographer's Right (bold mine):

... Property owners may legally prohibit photography on their premises but have no right to prohibit others from photographing their property from other locations. ... In any case, when a property owner tells you not to take photographs while on the premises, you are legally obligated to honor the request. ...

and:

... Members of the public have a very limited scope of privacy rights when they are in public places. Basically, anyone can be photographed without their consent except when they have secluded themselves in places where they have a reasonable expectation of privacy such as dressing rooms, restrooms, medical facilities, and inside their homes.