A great deal here depends on when you (generally) take pictures.

In particular, with bright light, a smaller sensor makes little or no difference in quality. As the light level drops, however, a large sensor (generally) gains a greater advantage.

So, if you're mostly taking pictures of the view from a mountain top in broad daylight, chances are that the Olympus will work beautifully (and even the much smaller sensors in most cell phones will also work quite nicely).

On the other hand, if you were to take some pictures at night by the light of a camp-fire (to give only one obvious example) the differences due to sensor size will be much more noticeable--quite possibly to the point that you want to think hard about the larger sensor.

If it were up to me, I'd think hard about a mirrorless camera with an APS-C sensor. For one example, the Sony A6300 has an APS-C sensor, but is still very close to the size and weight of the Olympus (12.7 ounces vs. 12.4 ounces for the Olympus).

As a disclaimer: no, I'm not really trying to push Sony in particular--as it happens I own a Sony camera, so I'm a little more familiar with what they offer than the other brands. Canon and Nikon (for the two most obvious possibilities) offer mirrorless cameras as well. I believe the Canons use an APS-C sensor (though Canon's version of APS-C is a bit smaller than everybody else's) and the Nikons use one that's substantially smaller (even smaller than four thirds).

That does translate to smaller size and weight though--for example, the Nikon 1 series bodies are around 11 ounces apiece. If you shoot (at least primarily) in situations where a smaller sensor will work well (and given that size and weight are at an extreme premium for you), it might be worth considering something even smaller than four thirds.