I don't think it is necessary to ask permission to link to someones blog or website. That's what the internet is, a collection of links that lead us to other things.
However, I think it is good manners to take it down if asked. I wouldn't take the others down though.
On my livejournal I occasionally have people post they "friended" me, and I always wonder why- I really don't care, except maybe it gives me the opportunity to check out their site too.