Integration docs

Cloudflare

Enhance your website's AI SEO with our Cloudflare Agent integration. Monitor AI interactions and optimize your content for better visibility in AI search results.

Overview

Our Cloudflare Agent integration provides a seamless way to track AI interactions with your website. Deploy through Cloudflare Workers for efficient, edge-based tracking of how AI systems interact with your content. This solution works with any website hosted on Cloudflare and requires minimal configuration. The edge-based approach ensures fast performance and comprehensive tracking across all pages.

Features

Deploy via Cloudflare Workers
Edge-based AI tracking
Zero impact on site performance
Global distribution
Automatic updates and maintenance
Works with any website on Cloudflare
No code changes to your actual website
Identifies all major AI bots and crawlers

Requirements

Cloudflare account
Website hosted on Cloudflare or using Cloudflare DNS
API key from xseek
Website ID from your xseek dashboard

Setup process

1
Ensure AI bots are allowed through Cloudflare's WAF and security settings
2
Create a Cloudflare Worker in your Cloudflare dashboard
3
Add our AI bot detection code to the worker
4
Set up environment variables with your API key and website ID
5
Configure a route pattern to direct traffic through the worker
6
Deploy the worker to start tracking AI visits

Integration setup

Environment Variables Required

Add these environment variables to your project:

XSEEK_API_KEY=your_api_keyXSEEK_WEBSITE_ID=your_website_id

Your API key can be found in your account settings. Make sure it has the ai_visits:push privilege.

1Create a Cloudflare Worker

// Cloudflare Worker - AI Bot Detection with patterns from xSeek's bot database
// Add secrets: wrangler secret put XSEEK_API_KEY && wrangler secret put XSEEK_WEBSITE_ID

const AI_BOTS = [
  { name: 'anthropic-ai', pattern: /anthropic-ai/i },
  { name: 'claudebot', pattern: /ClaudeBot/i },
  { name: 'claude-web', pattern: /claude-web/i },
  { name: 'claude-user', pattern: /Claude-User/i },
  { name: 'claude-searchbot', pattern: /Claude-SearchBot/i },
  { name: 'perplexitybot', pattern: /PerplexityBot/i },
  { name: 'perplexity-user', pattern: /Perplexity-User/i },
  { name: 'grokbot', pattern: /GrokBot(?!.*DeepSearch)/i },
  { name: 'grok-search', pattern: /xAI-Grok/i },
  { name: 'grok-deepsearch', pattern: /Grok-DeepSearch/i },
  { name: 'GPTBot', pattern: /GPTBot/i },
  { name: 'chatgpt-user', pattern: /ChatGPT-User/i },
  { name: 'oai-searchbot', pattern: /OAI-SearchBot/i },
  { name: 'google-extended', pattern: /Google-Extended/i },
  { name: 'applebot', pattern: /Applebot(?!-Extended)/i },
  { name: 'applebot-extended', pattern: /Applebot-Extended/i },
  { name: 'meta-external', pattern: /meta-externalagent/i },
  { name: 'meta-externalfetcher', pattern: /meta-externalfetcher/i },
  { name: 'bingbot', pattern: /Bingbot(?!.*AI)/i },
  { name: 'bingpreview', pattern: /bingbot.*Chrome/i },
  { name: 'microsoftpreview', pattern: /MicrosoftPreview/i },
  { name: 'cohere-ai', pattern: /cohere-ai/i },
  { name: 'cohere-training-data-crawler', pattern: /cohere-training-data-crawler/i },
  { name: 'youbot', pattern: /YouBot/i },
  { name: 'duckassistbot', pattern: /DuckAssistBot/i },
  { name: 'semanticscholarbot', pattern: /SemanticScholarBot/i },
  { name: 'ccbot', pattern: /CCBot/i },
  { name: 'ai2bot', pattern: /AI2Bot/i },
  { name: 'ai2bot-dolma', pattern: /AI2Bot-Dolma/i },
  { name: 'aihitbot', pattern: /aiHitBot/i },
  { name: 'amazonbot', pattern: /Amazonbot/i },
  { name: 'novaact', pattern: /NovaAct/i },
  { name: 'brightbot', pattern: /Brightbot/i },
  { name: 'bytespider', pattern: /Bytespider/i },
  { name: 'tiktokspider', pattern: /TikTokSpider/i },
  { name: 'cotoyogi', pattern: /Cotoyogi/i },
  { name: 'crawlspace', pattern: /Crawlspace/i },
  { name: 'pangubot', pattern: /PanguBot/i },
  { name: 'petalbot', pattern: /PetalBot/i },
  { name: 'sidetrade-indexer', pattern: /Sidetrade indexer bot/i },
  { name: 'timpibot', pattern: /Timpibot/i },
  { name: 'omgili', pattern: /omgili/i },
  { name: 'omgilibot', pattern: /omgilibot/i },
  { name: 'webzio-extended', pattern: /Webzio-Extended/i },
  { name: 'baiduspider', pattern: /Baiduspider/i },
  { name: 'mistralai-user', pattern: /MistralAI-User/i }
];

export default {
  async fetch(request, env) {
    const userAgent = request.headers.get('user-agent') || '';
    const ip = request.headers.get('cf-connecting-ip') || '';
    
    let detectedBot = null;
    for (const bot of AI_BOTS) {
      if (bot.pattern.test(userAgent)) {
        detectedBot = bot.name;
        break;
      }
    }

    const response = await fetch(request);

    if (detectedBot && env.XSEEK_API_KEY && env.XSEEK_WEBSITE_ID) {
      try {
        const res = await fetch('https://www.xseek.io/api/track-ai-bot', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'x-api-key': env.XSEEK_API_KEY,
          },
          body: JSON.stringify({
            botName: detectedBot,
            userAgent,
            url: request.url,
            ip: ip || undefined,
            referer: request.headers.get('referer') || undefined,
            websiteId: env.XSEEK_WEBSITE_ID,
          }),
        });

        if (!res.ok) {
          const text = await res.text();
          console.error('[xSeek] Tracking failed', {
            status: res.status,
            statusText: res.statusText,
            body: text,
          });
        }
      } catch (err) {
        console.error('[xSeek] Tracking request error', {
          error: err?.message || err,
          botName: detectedBot,
          url: request.url,
        });
      }
    }

    return response;
  },
};

2Deploy your Worker

After saving your worker, set up a route to direct traffic through it. Go to the "Triggers" tab and add a route pattern:

yourdomain.com/*

This will process all requests through your worker, enabling AI bot detection across your entire site.

All set!

Your Cloudflare-powered website is now tracking AI crawler visits. Data will appear in the xseek dashboard soon.

Need more help?

We can walk you through the setup or review your configuration.

Contact support