Remember to sharpen a regular graphite pencil in between your Prismacolors, every third or fourth time. I keep one by the sharpener just for this. The waxy Prismacolor led builds up on the blades then starts to "grip" the next pencil. Graphite will knock all that off and lube the blade, ready to sharpen again.
You will also have better luck with any handheld sharpener if you hold the pencil still and rotate the sharpener. You tend to have less inward pressure and more rotational energy this way, and you will not have near as much pencil waggle!
I use the Prismacolor sharpener, and as long as I clean it in between with a graphite pencil, I get excellent results!