> ## Documentation Index
> Fetch the complete documentation index at: https://docs.klove.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Stop Cloudflare from blocking AI crawlers on your site

> How to configure Cloudflare's bot protection settings to allow AI crawler traffic so Klove can serve agentic pages and track visits.

Cloudflare's bot protection features are powerful, but their default settings can block legitimate AI crawlers — the same bots that Klove needs to intercept and serve your agentic pages to. This guide explains what to check and how to fix it.

## Why this matters

Cloudflare's **Bot Fight Mode** and **AI Scrapers and Crawlers** block setting are designed to stop unwanted automated traffic. However, they don't distinguish between malicious scrapers and legitimate AI assistants like ChatGPT or Claude that are trying to read your content in order to cite you.

If these features are enabled, bots like GPTBot and ClaudeBot will receive a block response before the Klove Worker ever sees the request. The result: Klove cannot serve your agentic pages, AI crawl visits go untracked, and your content is invisible to these platforms.

<Tip>
  Once you allow the bots listed below, Klove's Cloudflare Worker takes over routing automatically. You don't need to create additional firewall rules or custom routing logic — the Worker handles detection and delivery for you.
</Tip>

## AI bot user agents to allow

Make sure the following user agents are permitted through Cloudflare's bot protection:

| Bot               | Platform                           |
| ----------------- | ---------------------------------- |
| `GPTBot`          | ChatGPT (training and browsing)    |
| `ChatGPT-User`    | ChatGPT (real-time browsing agent) |
| `OAI-SearchBot`   | ChatGPT Search                     |
| `PerplexityBot`   | Perplexity                         |
| `ClaudeBot`       | Claude (Anthropic)                 |
| `anthropic-ai`    | Anthropic crawlers                 |
| `Google-Extended` | Google AI Overviews and Gemini     |
| `bingbot`         | Bing / Microsoft AI features       |

## Steps to allow AI bots in Cloudflare

<Steps>
  <Step title="Open your Cloudflare dashboard">
    Log in to [dash.cloudflare.com](https://dash.cloudflare.com) and select the domain where Klove is deployed.
  </Step>

  <Step title="Navigate to Security → Bots">
    In the left sidebar, click **Security**, then select **Bots** from the submenu.

    You'll see the bot protection configuration for your zone, including Bot Fight Mode and any active bot-blocking rules.
  </Step>

  <Step title="Disable 'Block AI Scrapers and Crawlers' (if enabled)">
    If you see a toggle labelled **Block AI Scrapers and Crawlers**, turn it off.

    This setting blocks a broad category of AI-related user agents, including the legitimate crawlers that Klove relies on. Disabling it allows these bots to reach your site, where the Klove Worker will intercept and route them appropriately.
  </Step>

  <Step title="Check Bot Fight Mode">
    If **Bot Fight Mode** is enabled under **Security → Bots**, review whether it is challenging or blocking traffic that matches AI bot user agents.

    For most Klove users, disabling **Block AI Scrapers and Crawlers** in the previous step is sufficient. However, if you have custom firewall rules that explicitly block specific user agents (GPTBot, ClaudeBot, etc.), remove or update those rules to allow the bots listed above.
  </Step>

  <Step title="Add an allow rule for AI bots (optional — for stricter configurations)">
    If you have a strict security setup and want to explicitly allow AI crawlers while keeping other bot protections in place, create a Cloudflare WAF rule:

    1. Go to **Security → WAF → Custom Rules**.
    2. Click **Create Rule**.
    3. Set the rule to **Allow** requests where the **User Agent** contains any of the bot strings listed above (e.g. `GPTBot`, `ChatGPT-User`, `PerplexityBot`, `ClaudeBot`, `anthropic-ai`, `Google-Extended`).
    4. Set the action to **Skip** (bypass other security rules).
    5. Save and deploy the rule.

    This explicitly whitelists known AI crawlers before Cloudflare's bot scoring evaluates them.
  </Step>

  <Step title="Verify in Klove">
    After saving your Cloudflare changes, check the **Agentic Pages → Bot Visits** section in Klove. Within the next crawl cycle, you should start seeing visits from AI bots like GPTBot, ClaudeBot, and PerplexityBot appearing in the log.

    If bot visits are still not appearing after 24 hours, contact [Klove support](https://klove.ai/contact) and include your domain and a screenshot of your Cloudflare bot settings.
  </Step>
</Steps>

## Frequently asked questions

<AccordionGroup>
  <Accordion title="Will allowing AI bots affect my site's security?">
    No. Legitimate AI crawlers like GPTBot and ClaudeBot are read-only — they fetch pages but do not submit forms, attempt logins, or interact with your application. The Klove Worker intercepts their requests at the edge and serves them a lightweight static page, so your origin server sees minimal additional load.
  </Accordion>

  <Accordion title="Does this affect Google's regular search crawler?">
    No. Googlebot (the standard SEO crawler) is separate from Google-Extended (used by Google AI Overviews and Gemini). Cloudflare's bot protection settings for standard Googlebot are unaffected by the changes described in this guide.
  </Accordion>

  <Accordion title="What if I use Cloudflare's Super Bot Fight Mode (Pro/Business plans)?">
    Super Bot Fight Mode on paid Cloudflare plans has more granular controls. Look for a **Verified Bots** setting and ensure it is set to **Allow**. Verified bots include most of the AI crawlers listed above. If you need to allow specific unverified bots, use a custom WAF rule as described in Step 5.
  </Accordion>
</AccordionGroup>