For those not wanted to be affiliated with content- I think of it this way.
It's not so much that by being listed they are affiliated with my content, it's just that my blog has my list of blogs I click on each day so I can read them. It's really not there for anyone else to use but me. As long as they have a public webpage, I can go visit.
Now if they specifically ask for me to remove it, I probably would- BUT I still probably wouldn't ask permission- it's the easiest way for me to keep track of every site I visit each day. If my "readers" click them too, that's secondary.