|
Googlebot has taken down my servers before when I was running stuff I don't talk about. Pretty much same scenario as yours, a few dozen domains and it was crawling pages like 40 - 50 per second for 3 days straight.
Running tail -f on the logs and shit was just flying off the screen.
|