I am a Distress Ink addict, I love the stuff, but I do not stamp with it unless I am going for a watercolor-y look. Like everyone has said, it is not meant for stamping a crisp image.
Your ink can be too juicy and you can over-ink a stamp. Sometimes in an effort to get what looks like total coverage I will over-ink a stamp and get a blotchy image. It's very tempting to look through a clear stamp and try to evaluate how it's inked. We can't do that with rubber stamps. If I ink a clear stamp, it will always looks unevenly inked, even if I've inked it perfectly. When you press down, the ink spreads out. All I'm saying is, I wonder if you are looking at the "beading" on the stamp and then inking it more (too much) to compensate.
Versafine is an excellent ink for clears. It doesn't come in nearly enough colors, though, or I'd probably use nothing else but Versafine and Distress.
Esther is right about conditioning with Ranger Archival - Tim Holtz does it with all his stamps and just wipes them with water to clean. But you have to commit to having black stamps. I have not been able to work myself up to trying it. Maybe I will, eventually, starting with a set I'm not very attached to.
Definitely use a pad. I buy the inexpensive burgundy Darice pads and cut them in half.
Someone said to use Stazon cleaner to condition them? Huh. From what I have read, Stazon cleaner can actually damage polymers.
Don't hesitate to post pictures of what's going on, with info on what paper and ink you used. I've done that here when I was having problems with image quality and got lots of good help.