Skip to main content

What this does

The Worker sits at Cloudflare’s edge in front of your origin. For every inbound request it:
  1. Lets the request flow through to your origin unchanged (no added latency for users)
  2. Checks the user agent against Searchable’s AI-bot registry
  3. If it matches, fires a fire-and-forget POST to Searchable with the request metadata
The bot registry is fetched from Searchable and refreshed hourly, so new AI crawlers are picked up automatically — you never need to redeploy.
Works on any Cloudflare plan, including Free. Most sites stay well within the Workers free tier (100k requests/day). High-traffic sites can upgrade to Workers Paid for a few dollars a month.

Prerequisites

A Cloudflare account with your domain on it (the domain doesn’t have to be on a paid plan)
Permission to deploy Workers and add Worker routes to your zone
A Searchable project with your domain confirmed

Setup

1

Generate the Worker script in Searchable

  1. Open your Searchable dashboard
  2. Go to LLM Analytics → Setup
  3. Pick Cloudflare Worker as your crawler source
  4. Click Generate token
Searchable templates the token directly into the Worker script in the next panel — no find-and-replace needed. Copy the whole script.
The Searchable UI is the easiest way to get the script — the token is auto-filled and you avoid typos. The reference script is below if you’d rather build it yourself.
2

Create a new Worker in Cloudflare

Open dash.cloudflare.com → Workers & Pages → Create.
  1. Pick the Hello World template
  2. Name it (e.g. searchable-tracker)
  3. Click Deploy
3

Paste the Worker script

From the new Worker’s overview page:
  1. Click Edit code
  2. Replace the entire file contents with the script you copied from Searchable
  3. Click Save and deploy
4

Bind the Worker to your domain

A Worker by itself isn’t on your domain — it has its own *.workers.dev URL. To make it observe your real traffic, add a route.From the Worker’s Settings tab:
  1. Open Domains & Routes (older UIs label this Triggers)
  2. Click Add → Route
  3. Enter your route, e.g. *yourdomain.com/* (the leading * matches all subdomains)
  4. Pick your zone and save
5

Verify in Searchable

Return to LLM Analytics → Setup in your Searchable dashboard. The status strip should show Connected within a few minutes once an AI bot hits your site.

Reference: Worker script

If you’d rather not copy the script from the Searchable UI, here’s the reference. Replace INTEGRATION_TOKEN with the token you generated.
worker.js
A few things worth knowing about this script:
  • It never blocks the response. The forward to Searchable is ctx.waitUntil(...), so even if Searchable is unreachable, your users see no slowdown.
  • It pauses while Searchable is failing. After five forwards in a row fail (an error response or no response), the Worker stops forwarding for a while, then sends one request to check. If that succeeds, forwarding resumes; if not, it waits longer: 1, 2, 5 and 10 minutes, then every 10 minutes. Your site is served as usual throughout. AI-bot visits during a pause are not recorded.
  • It fails open by default, and you can change that. While the bot registry is loading (cold start), or if it can’t be fetched, the Worker forwards every request and lets Searchable’s server-side classifier do the filtering, so no bot traffic is missed. Once the registry loads, it filters at the edge. Set FAIL_OPEN to false in the script to filter against a built-in list of major AI crawlers during that window instead. The next section explains the trade-off.
  • Static assets are skipped. CSS, JS, images, fonts, and video bypass the Worker entirely — those almost never come from AI crawlers, and there’s no point burning a Worker invocation on each.

What gets sent to Searchable

For each request the Worker forwards, it sends:
  • HTTP method, host, scheme and path (the query string is dropped before sending)
  • User agent
  • Referer, as the browser or bot sent it (a query string in the referring URL is not removed)
  • Client IP address from Cloudflare’s CF-Connecting-IP header, anonymised by default: the last octet of an IPv4 address is zeroed and an IPv6 address keeps only its first 48 bits
  • Country code and Cloudflare data-center code
  • Cloudflare Ray ID and a timestamp
Request and response bodies, cookies and all other headers are never sent. Which requests are forwarded. Once the bot registry has loaded, only requests whose user agent matches the registry are forwarded: AI crawlers and assistants, plus the search crawlers Searchable tracks. What happens before it has loaded once (the first requests after each cold start, and for as long as the registry can’t be fetched; a failed fetch is retried after 1, 2, 5, 10 and 30 minutes, then every 60 minutes, so after a long outage a Worker can stay in this state for up to an hour after the registry is back) is set by the FAIL_OPEN constant near the top of the script:
  • FAIL_OPEN = true (default). The Worker forwards every non-static request, human visitors included, so no bot traffic is missed. It also keeps forwarding requests with no user agent after the registry has loaded. Searchable discards requests that don’t match its bot registry on arrival and does not store their path or IP address, with one exception: requests to UCP (Universal Commerce Protocol) endpoints (/.well-known/ucp, /ucp, /api/ucp/* and any others Searchable recognises) are kept whatever their user agent. Either way, the path, IP address and country of every forwarded request do leave your Cloudflare zone.
  • FAIL_OPEN = false. The Worker forwards only requests whose user agent matches the list of major crawlers built into the script, and never a request with no user agent, even after the registry has loaded. The trade-off is that a newer or less common bot can be missed for those first requests.
To switch, change the line to const FAIL_OPEN = false; and redeploy. If your script has no FAIL_OPEN line it predates this option: copy the current script from LLM Analytics → Setup. IP addresses. By default the Worker anonymises the client IP address before sending it (ANONYMIZE_IP = true near the top of the script), so the full address never leaves your Cloudflare zone: 203.0.113.7 is sent as 203.0.113.0, and 2001:db8:85a3:8d3::1 as 2001:db8:85a3::. Set ANONYMIZE_IP to false to send full addresses. To send no address at all, replace the ClientIP line with ClientIP: "0.0.0.0",. Bot detection and country data are unaffected either way, because neither uses the IP. What changes is the count of distinct IP addresses per bot, which counts networks rather than addresses when anonymised (a project that also sends full addresses from another integration, such as Logpush, gets one count mixing the two), and the IP stored on each request record. A Worker script copied or deployed before anonymisation existed keeps sending full addresses until it is replaced: copy the current script from LLM Analytics → Setup, or redeploy from there.
Both of these are edits to your copy of the script. If Searchable auto-deployed your Worker, redeploying from LLM Analytics → Setup (for example after revoking a token) uploads a fresh script: FAIL_OPEN and ANONYMIZE_IP go back to true and any ClientIP edit is removed. Re-apply your edits after every redeploy.
For tighter security, store the token as a Worker secret instead of leaving it inline.
  1. In your Worker, open Settings → Variables and Secrets
  2. Add a secret named INTEGRATION_TOKEN with your sa_… token as the value
  3. In the script, blank the inline INTEGRATION_TOKEN constant: const INTEGRATION_TOKEN = "";
  4. Redeploy
The Worker reads the secret if present and falls back to the constant otherwise. Same script, no other edits. This means you can rotate the token in Cloudflare’s UI without ever touching the Worker code.

Cost

Each AI-bot request costs you one extra Worker invocation. The Workers Free plan includes 100,000 requests per day — that’s about 3 million crawls per month, which covers nearly all sites. If you exceed it, the Workers Paid plan is $5/month for 10 million requests. The Worker itself does no CPU-intensive work (a regex check and a fetch call), so CPU time is negligible.

Troubleshooting

Most likely the route isn’t bound correctly.
  • In Cloudflare, open the Worker → Settings → Domains & Routes
  • Confirm the route pattern matches your live domain (e.g. *yourdomain.com/*, not *yourdomain.dev/*)
  • Confirm the route’s Zone is the zone that’s actually serving traffic for the domain
If the route is right, hit your site with a curl using a known AI user agent:
Then click Check in Searchable. If that doesn’t appear in the dashboard, the Worker’s logs (Cloudflare → Worker → Logs) will show whether forwardToSearchable is running and whether Searchable returned a non-2xx status.
The token is missing or wrong.
  • Confirm INTEGRATION_TOKEN (inline) or the secret named INTEGRATION_TOKEN is set to a sa_… value
  • If you’ve recently revoked the token in Searchable, generate a new one and update the Worker
  • The token must have no quotes around it in the secret value
You need permission to add Worker routes on the zone. If you don’t have it:
  • Ask the zone admin to add the route, or
  • Use the custom REST API path from your application server
If you also have Cloudflare Logpush sending traffic to Searchable, you’ll get the same request from both sources. Searchable de-duplicates server-side using the Cloudflare Ray ID, so the dashboard shows each request once. If you’d rather not run both, pick one and disable the other.

Removing the integration

Revoking the token stops the Worker from sending within a minute; redeploy from Setup to mint a new one (auto-deployed Workers), or paste a new token into the Worker (manual installs).
  1. Cloudflare → Worker → Settings → Domains & Routes → remove the route (the Worker stops observing traffic)
  2. Optionally delete the Worker itself
  3. Searchable → LLM Analytics → Setup → Tokens → revoke the token

Next steps

Cloudflare Logpush

On the Enterprise plan? See the native Logpush path.

See the data

Open LLM Analytics to see which assistants are crawling your site.