The article in rfusca's answer includes some references: The Aero-Ektars, by NASA scientist Michael Briggs; Radioactive Materials in Camera Lenses, from the Health Physics Society (an organization focused on radiation safety); and Thoriated Camera Lens (ca. 1970s), from Oak Ridge Associated Universities's professional training on radiation safety.

From the ORAU PTP article:

Measurements have indicated that the exposure rate at a depth of 10 cm in the body of an individual carrying a camera containing 0.36 uCi of thorium would be approximately 0.01 mrem/hr. Based on this value, NUREG-1717 calculated that a serious photographer might receive an annual exposure of 2 mrem. This assumed that the photographer carried the camera 30 days per year and for 6 hours per day. They also estimated an exposure of 0.7 mrem per year for an average photographer. If the camera lens contained the maximum permitted concentration of thorium (30%), NUREG-1717 estimated that the aforementioned annual doses could triple.

This puts the "6hrs/day for a month" usage at about the same as getting a chest X-ray — or, one little green square on the xkcd chart. Or to put it another way, using the lens six hours a day for a year would be the same as taking three round-trip flights from one US coast to the other in that year. Not completely trivial, but not something people normally stress about. And that'd be really heavy usage.

The articles indicate that exposure to the eye might be a greater concern than overall dosage, particularly if you happen to have thorium in an eyepiece (unlikely for general photo equipment). So you might decide to spend a little less time holding the camera right to your eye than you might otherwise.

Assuming (based on the reading) that looking through the viewfinder is very roughly an order of magnitude greater exposure than the general usage, looking through the viewfinder for an hour is about 1µSv — equivalent to getting an arm x-ray.