People are clever to notice patterns. Actually, a human mind instinctively searches for patterns. Use this to your advantage, even when you first think there will not be a pattern to follow. Maybe there is no pattern throughout the whole book, but at least for groups of photos.

An example: three photos that you've shot during your trips onto a certain mountain. One macro-shot of a single insect, one portrait of an old man and one landscape from the mountainside. No pattern? Yes, they are all shot at the same mountain, that's a pattern. But, this is a pattern that has a meaning to you, not necessarily easy to see by your audience. They might see a pattern in field of vision or scale of living space. You know the photos are from the same mountain, they don't. They'll find a pattern in the order the photos from small to large or large to small, while in your mind the mountain is the pattern and you might just place them in chronological order: man, insect, landscape - practically ruining the pattern for your audience.

So to help you with the organizing task, forget your own knowledge of your photos, and step in the shoes of your audience. Try to see patterns the way they'll probably see them.

Personally, I like the idea of using colors for grouping of photos. With the key color of a photo I'd place them in pairs or triplets, mixing colors, opposite colors, groups of same color, etc. And if there is no key color in a photo, then leave it to another photobook. Not sure if I could make a pattern with colors, but at least I'd have fun organizing them.