Yeah, that’s a problem. So even if you want to search your local newspaper, you won’t be able to because of bot protection. Your crawler will be blocked. You also can’t put the whole of Reddit or even Lemmy in your search index.