Tech DotBot from moz.com is not obeying robots.txt directives - GoFuckYourself.com

Cyber Fucker · 04-27-2018, 06:25 AM

It was pounding like crazy 1000s of my pages daily, I had not choice but to ban on webserver level.

Besides what good it does? It's just spying on your keywords and links.

Well, even after ban it is still trying to access my sites. (I allowed only robots.txt for it. It reads it and does not give a fuck.) So their shit is obviously broken.

What I’m trying to say is that, perhaps you should review your access logs from time to time and see what is wasting your bandwidth and server resources.

For example, my other observation is that within last 3 years I have received from 500 000 to 1000 000 hacking and exploiting attempts per each live website. While all of them were not successful, it must have some impact on server performance.

Well configured webserver can reduce all this bad stuff by 90-99%.

rowan · 04-27-2018, 09:55 AM

What directive(s) was the bot ignoring?

Cyber Fucker · 04-27-2018, 09:58 AM

Code:

User-agent: dotbot
Disallow: /

freecartoonporn · 04-27-2018, 10:02 AM

disallow all

allow google.

or contact the dotbot guys

Bladewire · 04-28-2018, 10:55 AM

Have you blocked it by IP? If so , has it come back under other IP's?

TFCash · 04-28-2018, 11:21 AM

Try this little script out, it seems to grab most of the bad bots.

Bot Black Hole

Cyber Fucker · 04-28-2018, 12:10 PM

Nah, I've blocked it by user-agent header, I got no problem with it now since it’s blocked.
This thread was more like an educational one and a suggestion for everyone to monitor webserver logs, at least from time to time.
I'm fine.

Less resources eaten by bad stuff = more resources for good traffic = speed and speed = better SEO.

rowan · 04-28-2018, 09:18 PM

Even "good" bots can cause issues.

At one point GoogleBot was fetching 150k+ pages per day from one of my sites. The site is heavily database driven so fetching two pages per second continuously did cause some server load issues.

You can dial back the crawl rate in webmaster tools, but that setting expires after 90 days, and then Googlebot just starts pounding away again. They deliberately ignore the Crawl-Delay robots.txt directive.

04-27-2018, 06:25 AM	#1
Cyber Fucker Hmm Industry Role: Join Date: Sep 2005 Location: On an endless road around the world for rock and roll. Posts: 12,642	DotBot from moz.com is not obeying robots.txt directives It was pounding like crazy 1000s of my pages daily, I had not choice but to ban on webserver level. Besides what good it does? It's just spying on your keywords and links. Well, even after ban it is still trying to access my sites. (I allowed only robots.txt for it. It reads it and does not give a fuck.) So their shit is obviously broken. What I’m trying to say is that, perhaps you should review your access logs from time to time and see what is wasting your bandwidth and server resources. For example, my other observation is that within last 3 years I have received from 500 000 to 1000 000 hacking and exploiting attempts per each live website. While all of them were not successful, it must have some impact on server performance. Well configured webserver can reduce all this bad stuff by 90-99%. __________________

04-27-2018, 09:58 AM	#3
Cyber Fucker Hmm Industry Role: Join Date: Sep 2005 Location: On an endless road around the world for rock and roll. Posts: 12,642	Code: User-agent: dotbot Disallow: / __________________

04-27-2018, 10:02 AM	#4
freecartoonporn Confirmed User Industry Role: Join Date: Jan 2012 Location: NC Posts: 7,683	disallow all allow google. or contact the dotbot guys __________________ SSD Cloud Server, VPS Server, Simple Cloud Hosting \| DigitalOcean

04-28-2018, 11:21 AM	#6
TFCash Confirmed User Industry Role: Join Date: Apr 2001 Posts: 1,738	Try this little script out, it seems to grab most of the bad bots. Bot Black Hole __________________ TeenFlood.com Online since 1998. TFCash KissMeGirl VirginRiches MondoBucks tim at tfcash.com or submit a ticket at our HelpDesk

04-28-2018, 12:10 PM	#7
Cyber Fucker Hmm Industry Role: Join Date: Sep 2005 Location: On an endless road around the world for rock and roll. Posts: 12,642	Nah, I've blocked it by user-agent header, I got no problem with it now since it’s blocked. This thread was more like an educational one and a suggestion for everyone to monitor webserver logs, at least from time to time. I'm fine. Less resources eaten by bad stuff = more resources for good traffic = speed and speed = better SEO. __________________

04-27-2018, 09:55 AM	#2
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	What directive(s) was the bot ignoring?

04-28-2018, 10:55 AM	#5
Bladewire StraightBro Industry Role: Join Date: Aug 2003 Location: Monarch Beach, CA USA Posts: 56,229	Have you blocked it by IP? If so , has it come back under other IP's?

04-28-2018, 09:18 PM	#8
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Even "good" bots can cause issues. At one point GoogleBot was fetching 150k+ pages per day from one of my sites. The site is heavily database driven so fetching two pages per second continuously did cause some server load issues. You can dial back the crawl rate in webmaster tools, but that setting expires after 90 days, and then Googlebot just starts pounding away again. They deliberately ignore the Crawl-Delay robots.txt directive.