I totally know what you mean! I broke all of my sets apart - put flowers with flowers, hearts with hearts, etc. - and HATED it! My brain thinks in "sets", apparently... After spending a billion hours separating them, I was surprised how quickly I could put them back together again (I like puzzles - maybe that's why...), but it still was hours wasted. I do like having an index with all of the flowers with the flowers and the hearts with the hearts, but I don't know any way around writing the names of the sets next to each image, if that's important to you.
I have all of my stamps (sets, singles, rubber, clear) in binders organized alphabetically by manufacturer, and also alphabetically within that manufacturer. I have a "master" list of all of the manufacturers and gave each of them a 2- or 3-letter abbreviation that I note next to each stamped image. I'm familiar enough with most of my stamps that I can usually tell which set a stamp is from once I've identified who made it. If not, a quick flip through that binder and I know (and have the stamp in hand)! I don't know if that's helpful for you, but it works for me. Side note: I was shocked to realize that I have stamps from 66 different manufacturers! Crazy, that...
Happy for no reason...