The sensor is made of of photosites. Each one is a finite physical element which cannot change size. When a camera or phone takes a photo, each photosite is normally turned into a pixel. Other than on Foveon sensors (and a few others no longer in production), the camera uses information of adjacent photosites to interpolate full color information. So, while you get a mapping from each photosite to one pixel, the relationship is not as simple.

When you reduce the output size, the sensor still used the same photosites to capture the image. The photosites are therefore the same size. The difference is that now that a lower resolution image needs to be created, it can use more photosites for each pixel. Most commonly you see this as 4 photosites combined to produce one pixel which reduces noise and makes it easier to get full color output, just take the red, blue and average of two greens and you have a pixel. This is known as pixel binning. It happens quite fast, often at the sensor level which allows many cameras to actually shoot faster in those modes. If it does take longer, it is because additional image processing is being done.

While using a portion of the sensor for video is common, it is rarely done for images since it reduces the field of view. In other words, if only part of the sensor is used, you would instantly see the framing preview change and become more zoomed-in.