You've got the problem space pretty much pegged. Incident light metering measures what's falling on the subject independent of its characteristic reflectivity, etc., while reflected light metering measures what's being reflected from the subject, independent of the characteristics of the incident light. The recording medium in the camera, as you pointed out, records what's reflected from (or, if a light source is included in the image, what it transmitted by) the subject. No matter which type of metering is used, the object of the game is to record the light coming from the subject appropriately for the image.

It's easy to picture what's going on with a reflected light meter, whether that's the in-camera meter or a separate hand-held spot or average meter. Incident light metering is just a little more difficult to picture.

Most incident light metering is done with a dome-type invercone. Essentially, you have a hemispherical diffuser over the light-sensitive element that accepts light from 180 degrees (90 degrees off of the axis in all directions), and transmits 18 percent of that light to the light-sensitive element. (The element can usually be used as a reflected light meter as well, sometimes requiring accessories for the meter.) You can measure light sources separately, either by shading the dome from individual sources, or by selectively turning the sources on and off (as with studio lighting) to determine lighting ratios (the difference in illumination between areas that will be highlights and shadows on the subject).

While the method of working is different, what you are accomplishing is almost identical to placing a grey card at the subject position, angled to be tangent to the subject, and talking a spot meter reading of that grey card. It tells you nothing about the subject -- but that can be a good thing. If your metering depends on the subject, then you need to know what the subject's reflectivity is (or, in Zone System terminology, where to place the subject on the exposure scale). Without that bit of knowledge, you won't be able to tell the truth about the subject using only the meter reading -- you may be able to capture the whole range of tonalities perfectly, but that may mean, say, severely under- or over-exposing the one thing you were really trying to capture.

That's why there's such a divide between incident and reflected light metering. You'll see incident light metering used most often in the studio, but often in location portraiture or fashion/product work as well. There is usually something in the image that needs to be absolutely bang on, and that's best achieved by leaving it out of the equation altogether when metering, then letting its actual reflectance and color characteristics fall where they may on the sensor. The rest of the scene is atmospherics -- you can play with them to your heart's content, and as long as the subject is correctly lit and exposed, the picture will be what you want it to be.

Working in less controlled environments, especially when shooting things like landscapes, you generally want to get it all in (so to speak) without worrying too much about the "truth". In these cases, the "truth" is more subjective, and you are looking to collect the data that support your version of the truth. That may mean making that painfully bright highlight a citizen of Zone VII or forcing the inky blackness up into a barely dark lower midtone in order to preserve the detail that's important to you. But you have to know where things go in your version of the world. Or you can let the camera's scene-recognition algorithms make that determination for you.

In either case, though, the meter is just the starting point. It gets you into the ballpark, then your experience and vision will let you adjust the objective reality to meet your subjective requirements.