So, I downloaded the CIDR of the engines that I want to block
and launched this:
Code:
while read line; do sudo ufw insert 1 deny from $line to any; done < cdir.txt
And he have added about 650 new rules in UFW
But in the access.log the ones I see most often are:
Opensiteexplorer.org/dotbot,
[email protected]
semrush.com/bot.html
bing.com/bingbot.htm
ahrefs.com/robot/
Apart from Bing, the rest seem to be marketing tools, some more or less connected to Google or moz.com
I don't use them, but mostly I don't need them if they first destroyed my server...
Can I block them? Always via IP-UFW?
And in case, which IP should I block?
Their Ip in my access.log changes, eg:
Code:
46.229.168.76 - - [17/Oct/2017:23:04:53 +0000] "GET /search-busty%20mom%20loves%20to%20suck%20cock/ HTTP/1.1" 200 24789 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
46.229.168.69 - - [17/Oct/2017:22:57:38 +0000] "GET /search-big%20brest%20sex%20photo/random/ HTTP/1.1" 200 21170 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
46.229.168.67 - - [17/Oct/2017:22:55:13 +0000] "GET /desi%20girls%20boobs%20suckers%20bees%20photos2015/ HTTP/1.1" 200 31832 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
How do I find all their IP?
Or is it better in this case to use robots.txt? Is there a serious list (robots.txt or IP) of bad bots to block?