Skip to main content
Cloudflare’s bot protection features are powerful, but their default settings can block legitimate AI crawlers — the same bots that Klove needs to intercept and serve your agentic pages to. This guide explains what to check and how to fix it.

​
Why this matters

Cloudflare’s Bot Fight Mode and AI Scrapers and Crawlers block setting are designed to stop unwanted automated traffic. However, they don’t distinguish between malicious scrapers and legitimate AI assistants like ChatGPT or Claude that are trying to read your content in order to cite you. If these features are enabled, bots like GPTBot and ClaudeBot will receive a block response before the Klove Worker ever sees the request. The result: Klove cannot serve your agentic pages, AI crawl visits go untracked, and your content is invisible to these platforms.
Once you allow the bots listed below, Klove’s Cloudflare Worker takes over routing automatically. You don’t need to create additional firewall rules or custom routing logic — the Worker handles detection and delivery for you.

​
AI bot user agents to allow

Make sure the following user agents are permitted through Cloudflare’s bot protection:

​
Steps to allow AI bots in Cloudflare

1

Open your Cloudflare dashboard

Log in to dash.cloudflare.com and select the domain where Klove is deployed.
2

Navigate to Security → Bots

In the left sidebar, click Security, then select Bots from the submenu.You’ll see the bot protection configuration for your zone, including Bot Fight Mode and any active bot-blocking rules.
3

Disable 'Block AI Scrapers and Crawlers' (if enabled)

If you see a toggle labelled Block AI Scrapers and Crawlers, turn it off.This setting blocks a broad category of AI-related user agents, including the legitimate crawlers that Klove relies on. Disabling it allows these bots to reach your site, where the Klove Worker will intercept and route them appropriately.
4

Check Bot Fight Mode

If Bot Fight Mode is enabled under Security → Bots, review whether it is challenging or blocking traffic that matches AI bot user agents.For most Klove users, disabling Block AI Scrapers and Crawlers in the previous step is sufficient. However, if you have custom firewall rules that explicitly block specific user agents (GPTBot, ClaudeBot, etc.), remove or update those rules to allow the bots listed above.
5

Add an allow rule for AI bots (optional — for stricter configurations)

If you have a strict security setup and want to explicitly allow AI crawlers while keeping other bot protections in place, create a Cloudflare WAF rule:
  1. Go to Security → WAF → Custom Rules.
  2. Click Create Rule.
  3. Set the rule to Allow requests where the User Agent contains any of the bot strings listed above (e.g. GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, anthropic-ai, Google-Extended).
  4. Set the action to Skip (bypass other security rules).
  5. Save and deploy the rule.
This explicitly whitelists known AI crawlers before Cloudflare’s bot scoring evaluates them.
6

Verify in Klove

After saving your Cloudflare changes, check the Agentic Pages → Bot Visits section in Klove. Within the next crawl cycle, you should start seeing visits from AI bots like GPTBot, ClaudeBot, and PerplexityBot appearing in the log.If bot visits are still not appearing after 24 hours, contact Klove support and include your domain and a screenshot of your Cloudflare bot settings.

​
Frequently asked questions

No. Legitimate AI crawlers like GPTBot and ClaudeBot are read-only — they fetch pages but do not submit forms, attempt logins, or interact with your application. The Klove Worker intercepts their requests at the edge and serves them a lightweight static page, so your origin server sees minimal additional load.
No. Googlebot (the standard SEO crawler) is separate from Google-Extended (used by Google AI Overviews and Gemini). Cloudflare’s bot protection settings for standard Googlebot are unaffected by the changes described in this guide.
Super Bot Fight Mode on paid Cloudflare plans has more granular controls. Look for a Verified Bots setting and ensure it is set to Allow. Verified bots include most of the AI crawlers listed above. If you need to allow specific unverified bots, use a custom WAF rule as described in Step 5.