A camera lens is made up of a number of individual glass elements (lenses) which work together to focus an image onto the plane of a piece of film or a digital sensor. Each element in the 'stack' of a lens is either there specifically to make the image hitting that sensor more accurate, or to correct for some inherent deficiency that has been introduced as a result of an element within the overall lens. In general, the more expensive the camera lens, the more elements are in the overall lens and/or the better it has been engineered to produce accurate images and reduce deficiencies of the lens design. This is true even if you're using a camera that doesn't have the ability to change lenses.

Does adding a modifier- generally speaking a modifier of significantly lower quality than any other element likely to be used- to the front of the 'stack' of elements that makes up a lens detract from the quality of a lenses optics? Yes. Absolutely. Your picture will never be as sharp or accurate as if you hadn't added the modifier to the front of the lens. In fact, it will be significantly degraded. Period.

Is it a worthwhile purchase? That depends on what you're looking to accomplish with your photography, and what (in general) you want your pictures to look like. If you like the 'lo-fi' photography look that can be achieved through the use of things like Lensbabys, and Lomo cameras, then you're likely to think the look that comes from one of these modifiers is a great addition to your bag. If, on the other hand, you're hoping for performance that in any way rivals an actual fisheye or wide-angle lens, it would be better to avoid such modifiers because the quality of the images you'll be able to make won't even be in the same city as a dedicated lens would, let alone the same ballpark.