Have you tried basic dish soap like original Dawn? I don't recommend anything with added antibacterial properties, but the original blue dawn (watered down) washes most everything well.
Another option (although I'm sure it has an aroma of some kind) is Simple Green. Many people have stated that they use this product, also watered down a good bit, as their preferred method of cleaning stamps.
Aside from this, I remember seeing a video by Jennifer McGuire where she shares some of her favorite cleaners and why she uses them. The shammy in particular is something she uses without a spray, though I'm not sure if it will help you with ink that has dried on.
Here's the link
That link will take you to the portion of the video where she starts discussing the different cleaning tools.