GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Y'all ever had problems with google crawl rate?

rowan · 11-26-2019, 04:48 AM

I've had the same problem.

Basically, Google are arrogant cunts that refuse to follow the "non-standard" Crawl-Rate robots.txt directive, even though it's a de-facto standard, and a pretty clear indication by the webmaster that a crawler should slow down.

Since Google ignores such a directive, you have to log into webmaster tools to manually configure the rate to something lower. Furthering their arrogance, the setting expires it after 90 days, reverting back to normal behaviour. You have to log in and manually configure the crawl rate again to stop them beating the shit out of your server.

Fuck Google.

11-26-2019, 04:48 AM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I've had the same problem. Basically, Google are arrogant cunts that refuse to follow the "non-standard" Crawl-Rate robots.txt directive, even though it's a de-facto standard, and a pretty clear indication by the webmaster that a crawler should slow down. Since Google ignores such a directive, you have to log into webmaster tools to manually configure the rate to something lower. Furthering their arrogance, the setting expires it after 90 days, reverting back to normal behaviour. You have to log in and manually configure the crawl rate again to stop them beating the shit out of your server. Fuck Google.