There are tags that affect how the image can be interpreted by software (think about orientation), but no, the image bits should not be touched.

Now, lore says that there are (bad) editors that open the image (decode it) to show it, and even if you just change EXIF data, they re-encode the JPEG data instead of copying over the original. In that case, you can lose quality. I am quite sure that, for example, exiftool is safe (as will a lot of other tools be).

Proof: I start with the image blue_boniface.jpg, and I copy the same image to blue_boniface_bare.jpg and blue_boniface_comment.jpg:

% cp blue_boniface.jpg blue_boniface_bare.jpg
% cp blue_boniface.jpg blue_boniface_comment.jpg

Now I remove all EXIF from the first, and add a Comment field to the second:

% exiftool -all= blue_boniface_bare.jpg
% exiftool -Comment='This is a new comment' blue_boniface_comment.jpg

And after that I convert all the three images into PNM format (which has no ancillary data and is the barest picture format I know):

% jpegtopnm blue_boniface_bare.jpg > one.pnm
% jpegtopnm blue_boniface.jpg > two.pnm
% jpegtopnm blue_boniface_comment.jpg > three.pnm
% ls -l *.pnm
-rw-rw-r-- 1 romano romano 24307892 Sep 13 17:57 one.pnm
-rw-rw-r-- 1 romano romano 24307892 Sep 13 17:57 three.pnm
-rw-rw-r-- 1 romano romano 24307892 Sep 13 17:57 two.pnm

The three files seems the same, let's check:

% diff one.pnm two.pnm
% diff one.pnm three.pnm 

(diff is silent if the files are the same). So the image data is exactly the same after the EXIF file manipulation. QED