Quote:
Originally Posted by Klen
I banned their ip ranges on firewall level instead.
|
Better to do it via robots.txt, that way you don't need to keep up with any future changes to their IP ranges.
Code:
User-agent: SemrushBot
Disallow: /
(This assumes their crawler can understand these directives.

)