DSLR or mirrorless for sharp photos on a doors-off helicopter flight?

Mirrorless or DSLR doesn't really matter. Use whichever lens-body combination gives you the best reduction of vibrations that are relatively slow in frequency but very deep in amplitude. Smaller bodies/lenses will catch less wind, but larger bodies have more mass which helps with the vibration coming from the helicopter body to which you are strapped.

One of the cardinal rules of shooting "open door" is to never, ever change lenses in flight,¹ so it is not inconceivable to take two cameras: one with a wide to very wide angle of view and another with anywhere from a medium zoom to as much focal length as you are willing to haul the required size/weight along with you. Just keep in mind that if it is a full tourist flight, there won't be a lot of extra room in the cabin of many helicopters. Most of the Hawaii tours have a reputation for getting you in close enough that you probably won't need the long focal length the way you might if taking a cityscape tour over a major city where you might want details of certain buildings or landmarks that the helicopter can't approach as closely.

Longer focal lengths require faster shutter speeds and the accompanying ISOs. Larger lenses catch more wind. This is the one time you should leave the hood in your camera bag (so the wind doesn't rip it off the lens).

Wider lenses risk getting rotors and skids in the frame, so with those you'll probably want to shoot each composition with multiple frames in high speed bursts to try and catch one without a rotor fouling the shot.

Keep everything strapped to your body or to a hard point on the helicopter, preferably with redundancy. Smaller items must be secured as well. Any items that get loose (memory cards, lens caps, cleaning cloths, etc.) are a risk to be sucked out the door and possibly into the tail rotor or even the engine's intake, depending on the helicopter's design.

Don't get so caught up in getting so many shots that you forget to pause and enjoy the ride every now and then, too. That pause might just allow you to see a shot you otherwise might have missed, too!

Here are a few extensive articles that discuss shooting from helicopters. Some are written from the viewpoint of a relatively experienced photographer taking their first helicopter photo flight, others are written by those who have done more shoots through the open door of a helicopter. Several specifically mention flights in Hawaii and how to pick the best operator to match your photographic requirements.

Helicopter Aerial Photography
How to Take Photographs From a Helicopter
Tips on shooting from a helicopter
7 Tips for Shooting from a Helicopter
5 Tips for Photographing From A Helicopter

¹ Apart from the obvious risk of an unattached lens getting loose in the cabin or rolling out the door, there's also the consideration that a lot of dirt and dust will be blowing around. Think convertible going 70 mph x 10. You wouldn't want to expose the insides of your cameras and lenses to that even if you were allowed by the flight operator to do so (which you very likely will not be if it is open door).