View Single Post
Old 11-26-2019, 05:07 AM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by Klen View Post
I banned their ip ranges on firewall level instead.
Better to do it via robots.txt, that way you don't need to keep up with any future changes to their IP ranges.

Code:
User-agent: SemrushBot
Disallow: /
(This assumes their crawler can understand these directives. )
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote