What I'm finding as I go through and stamp all of the index cards for the stored-on-panels sets is that I have WAY TOO MANY OF EVERYTHING!!! I'm going to do a "favorites" binder or box or whatever - a few favorite butterflies, a few favorite hearts, a few favorite "Happy Birthdays", etc. After I use it for a while, I'll know if I miss the others or if I can safely sell them or give them away. I'd really like to down-size even more than I already have, but I'm hesitant to go all crazy with it. (I've already sold/given away close to half of the stamps I owned)
As far as the OP's dilemma - go for one consistent system for ALL of your stamps! You'll be able to see what you have (which makes it easier to actually USE it!) and you won't have to have multiple types of storage units (which makes me crazy). I think you'll be very happy...
Happy for no reason...