Photographing nail art is essentially product photography. The key to good product photography has less to do with which camera you use and everything to do with lighting the subject properly, so your first concern should be some of the resources mentioned in this question and this one.

Having said that, I would recommend the Nikon D5100 over the Panasonic Lumix FZ45 and here's why.

• The lens. There's nothing wrong with using a fixed lens bridge camera for product photography, but the FZ45 is a super zoom with a 24X zoom ratio. The disadvantage of such a wide range of focal lengths is that a lot of design compromises are made to create such a lens that ranges from 4.5-108mm (field of view equivalent to a 25-600mm lens on a 35mm format camera). They have their uses when size, weight, and the need for a wide variety of focal lengths in one lens is a big factor, but for shooting product photos or nail art you don't need to be able to go from 25mm to 600mm without changing a lens, so why settle for the compromise in image quality? Depending on how tightly you want to frame things, you might consider the AF-S DX Micro-NIKKOR 40mm f/2.8G as your lens instead of the kit zoom.
• The sensor. The D5100 has an APS-C sensor that is 23.6X15.6mm. The FZ45 has a sensor that measures about 7.7X6.2mm. That means you could fit almost 8 of the FZ45's sensors on a single D5100 sensor. This makes a huge difference in image quality. Even though both cameras have basically the same number of pixels (16.2 vs. 14.1) the pixels on the Panasonic are packed in so much tighter that it affects the amount of image noise. Not to mention that the Nikon sensor can catch almost eight times as much light due to the larger surface area.
• The light control. As already mentioned, managing the light is everything in product photography. With the Nikon D5100, you will be able to control an external flash via the hot shoe connection. A simple i-TTL (Nikon's name for automatic flash control) with a head that will rotate up to allow you to bounce the light off a white ceiling may be all you need. The Nikon SB-400 or a third party product such as the Yongnuo YN-468 II will allow you a great deal of control over your light. Anything other than the pop up flash on the FZ45 will be more complicated. You could choose to use continuous lights instead, which leads us to...
• Color control. Although both can save files in RAW format, which allows much more control of WB, color saturation, contrast, and exposure in post processing, I would venture to guess that the Nikon .nef format is more universally supported by a larger spectrum of image processing software than the Panasonic .rw2 format.
• The view. The Nikon D5100 has an optical viewfinder, the Panasonic FZ45 has an electronic viewfinder. For a discussion of the differences, see this question. Most of the advantages of an electronic viewfinder can be realized using the LCD screen on the back of the Nikon. None of the advantages of an optical viewfinder can be realized using the FZ45.