You should focus on taking pictures!

Don't worry too much about the gear. If you're really interested in this as a hobby, I suggest budgeting a significant chunk right off (See this slightly-tongue-in-cheek article) and buying some decent mid-range gear.

I don't mean you need to buy a ton of stuff without knowing what you need, but if you jump in at higher than the entry-level, you'll have more flexibility, room to grow, and not be as distracted by lusting after better gear (you'll know that what you have is good enough even though there's always something better). Most crucially, you'll have a camera designed for direct access to the essential controls, rather than one designed to produce snapshots without any thought. (See Are there disadvantages to a prosumer camera for a beginner, aside from cost? for more on this.)

You almost certainly want a camera with interchangeable lenses, but it's not a necessity. You could go for a high-end point and shoot like the Canon G12, or something like the lovely Fujifilm X100, which has a large sensor and a built-in lens tailored to go with it. Interchangeable lenses offer a lot more flexibility, but unlimited flexibility isn't necessarily all good.

A dSLR with a prime lens or two is a good choice too, or one of the newer mirrorless options (like Micro 4/3rds) with one of the tiny "pancake" lenses. Some people will suggest a basic zoom lens; some will recommend the prime-lens route, as I am doing — this is basically a choice of personal style. (While prime lenses offer some image quality advantages and usually wider max apertures, this can also be seen as a place where less flexibility helps learning by providing structure.)

So anyway, get something, and start taking a lot of pictures. Malcolm Gladwell puts forth the idea that to be great at anything, you need to put in about 10,000 hours of working at it. So, the most important thing is to not obsess about how to start and to just do it.

Take a lot of pictures, and then review each one with a critical eye. What are your favorites? What worked, what didn't? Narrow down your favorites from each batch, and make prints from those — there's really no substitute for the physical artifact. Look at other photographs you like. What makes them work? How do yours compare?

By repeating this process of taking pictures and narrowing them down, you'll become good.