> ## Documentation Index
> Fetch the complete documentation index at: https://docs.klove.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloudflare is blocking AI crawlers — how to fix it

> Troubleshoot why Klove agentic pages are not being served to AI crawlers when Cloudflare bot protection is enabled, and how to fix it step by step.

Klove's Agentic Pages feature works by intercepting requests from AI crawlers at the edge and serving them a clean, AI-optimised version of your pages. For this to work, those crawlers need to actually reach your Cloudflare Worker. If Cloudflare is blocking AI bots before they get that far, Klove never has a chance to intercept — and your agentic pages are never served.

***

## Symptoms

You may be experiencing this issue if:

* Your **AI traffic analytics** dashboard shows no visits from AI bots, even several days after deploying the Cloudflare Worker.
* Your **agentic pages** are set to *Published* but the visit counter never increments.
* Your **prompt monitoring visibility score** seems low despite having relevant content, and the cited sources panel shows no visits from bots like GPTBot or ClaudeBot.
* Checking your Cloudflare analytics shows a high number of blocked bot requests.

***

## Cause

Cloudflare offers several bot protection features that are aggressive enough to block legitimate AI crawlers. The two most common culprits are:

**Bot Fight Mode** — Cloudflare's built-in bot mitigation challenges or blocks requests from known bots, including many AI crawlers.

**"Block AI Scrapers and Crawlers"** — A dedicated toggle (introduced in 2024) that blocks a defined list of AI user agents including GPTBot, ClaudeBot, PerplexityBot, and others.

When either of these is enabled, AI bots are turned away at Cloudflare's edge before your Worker ever runs — so Klove cannot intercept or serve them.

***

## Fix

<Steps>
  <Step title="Log in to the Cloudflare dashboard">
    Go to [dash.cloudflare.com](https://dash.cloudflare.com) and select the zone (domain) where your Klove Worker is deployed.
  </Step>

  <Step title="Open Security settings">
    In the left sidebar, click **Security**, then select **Bots**.
  </Step>

  <Step title="Disable 'Block AI Scrapers and Crawlers'">
    If you see the **Block AI Scrapers and Crawlers** toggle, turn it **off**. This immediately stops Cloudflare from blanket-blocking the AI user agents on that list.
  </Step>

  <Step title="Check Bot Fight Mode">
    On the same **Bots** page, review the **Bot Fight Mode** setting. If it is enabled and you are seeing AI crawlers blocked, consider switching from *Bot Fight Mode* to **Super Bot Fight Mode** (available on Pro plans and above), which allows you to configure more granular rules — or disable it entirely if you do not need active bot mitigation.
  </Step>

  <Step title="Add WAF allow rules for specific AI user agents">
    If you need to keep bot protection enabled for other bots but want to allow AI crawlers through, add a **WAF custom rule** to skip challenges for the specific user agents Klove needs. Navigate to **Security → WAF → Custom Rules** and create a rule with:

    * **Field:** User Agent
    * **Operator:** contains (one rule per agent, or use *matches regex* to combine them)
    * **Action:** Skip → All remaining custom rules

    Add rules for each of the following user agents:

    | Bot                               | User Agent String |
    | --------------------------------- | ----------------- |
    | ChatGPT crawler                   | `GPTBot`          |
    | ChatGPT user browsing agent       | `ChatGPT-User`    |
    | Perplexity                        | `PerplexityBot`   |
    | Claude                            | `ClaudeBot`       |
    | Anthropic general                 | `anthropic-ai`    |
    | Google AI (Gemini / AI Overviews) | `Google-Extended` |
  </Step>

  <Step title="Save and deploy">
    Save your WAF rules. Changes propagate to Cloudflare's edge within a few seconds.
  </Step>
</Steps>

***

## Verifying the fix

After making the changes above, AI crawlers should begin reaching your Cloudflare Worker within hours. However, because AI platforms crawl on their own schedules, allow up to **24 hours** before expecting to see visits in your Klove dashboard.

To verify:

1. Go to your **Klove dashboard → AI Traffic Analytics**.
2. Check the **Recent Bot Visits** table for entries from GPTBot, ClaudeBot, PerplexityBot, or other crawlers.
3. If visits appear, your agentic pages are now being served correctly.

If you have waited 24 hours and still see no bot visits, check your Cloudflare **Security → Events** log to confirm whether bot requests are still being blocked. If they are, the WAF allow rules may need to be reviewed — ensure the user agent strings match exactly and the rule action is set to *Skip*.

<Note>
  The user agents listed above are the most common AI crawlers Klove supports. For a full list of the 200+ AI bots Klove can detect and serve agentic pages to, refer to the Agentic Pages documentation.
</Note>