Woke up this morning to find one of my servers was under a DDoS attack. Not just any attack, but one that was originating from Microsoft. There were hundreds of requests per second coming from a lot of very very large blocks of IPs from Microsoft. All the requests had one thing in common: they all claimed to be Bing. It created a major load on the server trying to serve that many requests and my only solution was to block all of the IP ranges at the firewall.

I'm not sure what Microsoft is doing and why they don't throttle their spider. Having all of their computers hit one site at the same time is not very good manners. I've never seen any other spider do that. Google is polite and only requests one page every minute or two. The other spider I see a lot is Baidu and is no faster than Google. But when Bing decides to spider you, they bring down the full force of hundreds - maybe thousands - of computers.

This isn't the first time I've had problems with Bing hitting my servers too hard. I've done some searching and people have mentioned that you need to put some stuff in your robots.txt to tell Bing to slow down. But why? Why can't Bing be polite in the first place?