First thing to come in mind is availability. Canon 5D mark II supply was in shortage for quite a while due to the camera's success among videographers. You can't take pictures with a camera you can't get, so good video capabilities reduced its potential by causing excess demand for the new model.

Another issue is cost - whether you need the capabilities or not, you must put your share in the pot for developing those features (and possibly dedicated encoding chips).

A related issue is allocation of resources. Each feature takes money to develop, and there's only so much that can be spent without exceeding target price. So some esoteric features to support specific genres of photography will be skipped (e.g. why do I need separate sound detectors with triggering box for high-speed photography while my camera could do it in software?). Some compromises could be pretty major, like leaving the poor AF system already criticized in 5D also in Canon 5DmkII - very important for still photographers, irrelevant for video work.

For those who pick up camera only as a tool to get some pictures, not to immerse in the process of making them, video-related options and buttons make the camera more complicated.

While you only asked about negative aspects, I'll also point out a positive one. Development in video has allowed Live View in dSLR-s, which often helps to nail precise manual focus or composition in awkward positions.