JPEG

JPEG is lossy, which means it compresses the image (in part) by discarding data. The data it discards is (normally) chosen to minimize the affect on the quality of the image, but it (virtually) always loses at least a little quality -- and depending on the quality level you choose, it can lose quite a bit. For most photographs it should be considered a display-only format -- once you've converted something to JPEG, you don't want to do any more editing on it. If you need to make changes, you re-start from some other format, make the changes, and do another JPEG conversion.

JPEG 2000, JPEG XR

There are newer versions of the JPEG specification. They define new forms of image compression that can generally give a better trade off between file size and image quality--your choice of the same quality with a smaller file, or better quality with roughly the same file size. They also support higher color resolution (e.g., 16 bits per channel and floating point formats to support high dynamic range). From a technical perspective, they're extremely attractive. The big disadvantage is that not nearly as many programs know how to read, display, manipulate, or write them.

HEIF

Like TIFF, HEIF is really a container format, which can contain images encoded with various methods (primarily h.265, but also h.264 and JPEG). It provides a better ratio of quality to file size than the original JPEG. Like TIFF (or GIF) you can package a whole sequence of pictures into a single file. Although there was considerable fanfare when HEIF was introduced in 2014, with many proclamations about how it was going to finally be the format that killed JPEG, most of the excitement seems to have fizzled without its displacing JPEG to any significant degree.

BPG

BPG is a format designed by the ever-prolific programmer Fabrice Bellard. It's similar to HEIF in basically being a container for an image encoded with h.265. The wrapper is a bit different, however, so the two aren't compatible with each other. From a photographic viewpoint, however, BPG has a fairly significant advantage: it directly supports embedding EXIF data into the image file.

Lossless JPEG

Although what we normally think of as JPEG is lossy, the JPEG specifications define file formats that use lossless compression as well. Since the compression is lossless, they don't usually produce nearly as small of files as normal JPEG compression can, but they actually do really well for lossless compression--much better than general-purpose compression like LZW or Huffman encoding normally even hope for on photographs. Like JPEG 2000 and JPEG XR, these work well, but suffer from lack of support.

GIF

GIF uses only lossless compression, but is limited to 8-bit (256) colors, which is quite limiting for photographs.

PNG

PNG was designed as a replacement for GIF, and mostly succeeds. It supports 24-bit color (8 bits each for red, green, and blue) and uses lossless compression. It has the color resolution necessary for photographs, but the compression it uses tends to be quite ineffective for most photographs, so files end up pretty large. The other big disadvantage of PNG is that it doesn't define a way to store EXIF (or similar) data, so if you use it to store photographs, you have to store the metadata separately. That can be fine for your own use, but means it'll generally be lost if you use it on a web page, or anything like that.

TIFF

TIFF is really a container format that allows you to insert various kinds of data into the container. While it's used primarily for images, it's really almost like a file system, so you could theoretically use it for almost any kind of data. This has a couple of consequences. One is that even if a program supports TIFF files, it may not support all TIFF files -- e.g., many don't support LZW-compressed images. In fact, few programs support all possible TIFF files. Another consequence is that TIFF tends to have a fair amount of overhead, and writing code to support TIFF (at all well) is a pain (which is why so many programs support it only incompletely).

BMP

BMP is basically just a Windows device independent bitmap written out to disk. It has only extremely limited support for compression (and most BMPs aren't compressed at all). Programs written for Windows can read/write BMP really easily, but there's not much else to recommend it (in particular, BMP files tend to be quite large for the amount of data being stored). BMP lacks any way to store EXIF (or similar) metadata. BMP is kind of like PNG, but more specific to Windows.

Conclusion

JPEG is useful as an output format (e.g., to display things on web pages it's good because it's compact, and virtually everybody can read it).

TIFF is frequently used as an intermediate format to (for example) store a file that might later be edited.

The 256-color limitation makes GIF pretty close to useless for photographs. BMP and PNG are basically harmless to the photograph itself, but can't store metadata, and the compression they use is rarely very effective for photographs (though storage prices are now low enough that quite a few people may not care much about that).