Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn’t entail handing over your website to Cloudflare?
Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn’t entail handing over your website to Cloudflare?
The ultimate bad bot blocker (https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker) does the heavy lifting for me, it updates multiple times per day to add and remove IP addreses and bot referers. It does need some monitoring though, some of the rules wildcard a bit hard and will catch mastadon servers with unusual names for example.