Don't be tempted to use 'glue remover' products on any residue, most of them contain oil of one sort or another, and oil and rubber are a bad combination.
Usually, if you can just get an edge started, it will peel away. I often found that using a sharp bladed knife like a scalpel, on edge, so it's flat to the rubber, can get you started.
Sue, I have a few stamps that did stubbornly refuse to peel, so I'm going to give that a try.