Skip to main content
Klove’s Agentic Pages feature works by intercepting requests from AI crawlers at the edge and serving them a clean, AI-optimised version of your pages. For this to work, those crawlers need to actually reach your Cloudflare Worker. If Cloudflare is blocking AI bots before they get that far, Klove never has a chance to intercept — and your agentic pages are never served.

​
Symptoms

You may be experiencing this issue if:
  • Your AI traffic analytics dashboard shows no visits from AI bots, even several days after deploying the Cloudflare Worker.
  • Your agentic pages are set to Published but the visit counter never increments.
  • Your prompt monitoring visibility score seems low despite having relevant content, and the cited sources panel shows no visits from bots like GPTBot or ClaudeBot.
  • Checking your Cloudflare analytics shows a high number of blocked bot requests.

​
Cause

Cloudflare offers several bot protection features that are aggressive enough to block legitimate AI crawlers. The two most common culprits are: Bot Fight Mode — Cloudflare’s built-in bot mitigation challenges or blocks requests from known bots, including many AI crawlers. “Block AI Scrapers and Crawlers” — A dedicated toggle (introduced in 2024) that blocks a defined list of AI user agents including GPTBot, ClaudeBot, PerplexityBot, and others. When either of these is enabled, AI bots are turned away at Cloudflare’s edge before your Worker ever runs — so Klove cannot intercept or serve them.

​
Fix

1

Log in to the Cloudflare dashboard

Go to dash.cloudflare.com and select the zone (domain) where your Klove Worker is deployed.
2

Open Security settings

In the left sidebar, click Security, then select Bots.
3

Disable 'Block AI Scrapers and Crawlers'

If you see the Block AI Scrapers and Crawlers toggle, turn it off. This immediately stops Cloudflare from blanket-blocking the AI user agents on that list.
4

Check Bot Fight Mode

On the same Bots page, review the Bot Fight Mode setting. If it is enabled and you are seeing AI crawlers blocked, consider switching from Bot Fight Mode to Super Bot Fight Mode (available on Pro plans and above), which allows you to configure more granular rules — or disable it entirely if you do not need active bot mitigation.
5

Add WAF allow rules for specific AI user agents

If you need to keep bot protection enabled for other bots but want to allow AI crawlers through, add a WAF custom rule to skip challenges for the specific user agents Klove needs. Navigate to Security → WAF → Custom Rules and create a rule with:
  • Field: User Agent
  • Operator: contains (one rule per agent, or use matches regex to combine them)
  • Action: Skip → All remaining custom rules
Add rules for each of the following user agents:
6

Save and deploy

Save your WAF rules. Changes propagate to Cloudflare’s edge within a few seconds.

​
Verifying the fix

After making the changes above, AI crawlers should begin reaching your Cloudflare Worker within hours. However, because AI platforms crawl on their own schedules, allow up to 24 hours before expecting to see visits in your Klove dashboard. To verify:
  1. Go to your Klove dashboard → AI Traffic Analytics.
  2. Check the Recent Bot Visits table for entries from GPTBot, ClaudeBot, PerplexityBot, or other crawlers.
  3. If visits appear, your agentic pages are now being served correctly.
If you have waited 24 hours and still see no bot visits, check your Cloudflare Security → Events log to confirm whether bot requests are still being blocked. If they are, the WAF allow rules may need to be reviewed — ensure the user agent strings match exactly and the rule action is set to Skip.
The user agents listed above are the most common AI crawlers Klove supports. For a full list of the 200+ AI bots Klove can detect and serve agentic pages to, refer to the Agentic Pages documentation.