Update: Crawler Protection Added
WiredGuy reached out privately with a valid concern about ASN blocking - Google's AS15169 covers both Googlebot AND Google Cloud. Same with Microsoft's AS8075 covering Bingbot and Azure.
Block those ASNs to stop bot traffic and you accidentally kill your SEO.
He was right. Fixed it within minutes.
The solution:
The plugin now pulls the official crawler IP lists directly from Google and Bing:
https://developers.google.com/static...googlebot.json
https://www.bing.com/toolbox/bingbot.json
These get cached and refreshed weekly. Before any ASN block is applied, it checks if the IP is a known search crawler. If so, it's automatically allowed through.
There's a checkbox in settings: "Protect Search Engine Crawlers" - ON by default with a warning about what happens if you turn it off.
You can now safely block all of AWS, Google Cloud, Azure, Hetzner, OVH, etc. without worrying about blocking the crawlers that share those ASNs.
Thanks to WiredGuy for catching this before it became a problem for someone.
This is why I post here before pushing things wider - you guys know your shit.
Even if you dont want to try a plugin for this, there is the information on how to fix this yourselves
