View Single Post
Old 10-17-2017, 04:23 PM  
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 380
So, I downloaded the CIDR of the engines that I want to block



and launched this:
Code:
while read line; do sudo ufw insert 1 deny from $line to any; done < cdir.txt
And he have added about 650 new rules in UFW

But in the access.log the ones I see most often are:
Opensiteexplorer.org/dotbot, [email protected]
semrush.com/bot.html
bing.com/bingbot.htm
ahrefs.com/robot/

Apart from Bing, the rest seem to be marketing tools, some more or less connected to Google or moz.com

I don't use them, but mostly I don't need them if they first destroyed my server...

Can I block them? Always via IP-UFW?

And in case, which IP should I block?
Their Ip in my access.log changes, eg:

Code:
46.229.168.76 - - [17/Oct/2017:23:04:53 +0000] "GET /search-busty%20mom%20loves%20to%20suck%20cock/ HTTP/1.1" 200 24789 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
46.229.168.69 - - [17/Oct/2017:22:57:38 +0000] "GET /search-big%20brest%20sex%20photo/random/ HTTP/1.1" 200 21170 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
46.229.168.67 - - [17/Oct/2017:22:55:13 +0000] "GET /desi%20girls%20boobs%20suckers%20bees%20photos2015/ HTTP/1.1" 200 31832 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
How do I find all their IP?

Or is it better in this case to use robots.txt? Is there a serious list (robots.txt or IP) of bad bots to block?
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote