What this does
The Worker sits at Cloudflare’s edge in front of your origin. For every inbound request it:- Lets the request flow through to your origin unchanged (no added latency for users)
- Checks the user agent against Searchable’s AI-bot registry
- If it matches, fires a fire-and-forget POST to Searchable with the request metadata
Works on any Cloudflare plan, including Free. Most sites stay well within the Workers free tier (100k requests/day). High-traffic sites can upgrade to Workers Paid for a few dollars a month.
Prerequisites
A Cloudflare account with your domain on it (the domain doesn’t have to be on a paid plan)
Permission to deploy Workers and add Worker routes to your zone
A Searchable project with your domain confirmed
Setup
1
Generate the Worker script in Searchable
- Open your Searchable dashboard
- Go to LLM Analytics → Setup
- Pick Cloudflare Worker as your crawler source
- Click Generate token
2
Create a new Worker in Cloudflare
Open dash.cloudflare.com → Workers & Pages → Create.
- Pick the Hello World template
- Name it (e.g.
searchable-tracker) - Click Deploy
3
Paste the Worker script
From the new Worker’s overview page:
- Click Edit code
- Replace the entire file contents with the script you copied from Searchable
- Click Save and deploy
4
Bind the Worker to your domain
A Worker by itself isn’t on your domain — it has its own
*.workers.dev URL. To make it observe your real traffic, add a route.From the Worker’s Settings tab:- Open Domains & Routes (older UIs label this Triggers)
- Click Add → Route
- Enter your route, e.g.
*yourdomain.com/*(the leading*matches all subdomains) - Pick your zone and save
5
Verify in Searchable
Return to LLM Analytics → Setup in your Searchable dashboard. The status strip should show Connected within a few minutes once an AI bot hits your site.
Reference: Worker script
If you’d rather not copy the script from the Searchable UI, here’s the reference. ReplaceINTEGRATION_TOKEN with the token you generated.
worker.js
- It never blocks the response. The forward to Searchable is
ctx.waitUntil(...), so even if Searchable is unreachable, your users see no slowdown. - It pauses while Searchable is failing. After five forwards in a row fail (an error response or no response), the Worker stops forwarding for a while, then sends one request to check. If that succeeds, forwarding resumes; if not, it waits longer: 1, 2, 5 and 10 minutes, then every 10 minutes. Your site is served as usual throughout. AI-bot visits during a pause are not recorded.
- It fails open by default, and you can change that. While the bot registry is loading (cold start), or if it can’t be fetched, the Worker forwards every request and lets Searchable’s server-side classifier do the filtering, so no bot traffic is missed. Once the registry loads, it filters at the edge. Set
FAIL_OPENtofalsein the script to filter against a built-in list of major AI crawlers during that window instead. The next section explains the trade-off. - Static assets are skipped. CSS, JS, images, fonts, and video bypass the Worker entirely — those almost never come from AI crawlers, and there’s no point burning a Worker invocation on each.
What gets sent to Searchable
For each request the Worker forwards, it sends:- HTTP method, host, scheme and path (the query string is dropped before sending)
- User agent
- Referer, as the browser or bot sent it (a query string in the referring URL is not removed)
- Client IP address from Cloudflare’s
CF-Connecting-IPheader, anonymised by default: the last octet of an IPv4 address is zeroed and an IPv6 address keeps only its first 48 bits - Country code and Cloudflare data-center code
- Cloudflare Ray ID and a timestamp
FAIL_OPEN constant near the top of the script:
FAIL_OPEN = true(default). The Worker forwards every non-static request, human visitors included, so no bot traffic is missed. It also keeps forwarding requests with no user agent after the registry has loaded. Searchable discards requests that don’t match its bot registry on arrival and does not store their path or IP address, with one exception: requests to UCP (Universal Commerce Protocol) endpoints (/.well-known/ucp,/ucp,/api/ucp/*and any others Searchable recognises) are kept whatever their user agent. Either way, the path, IP address and country of every forwarded request do leave your Cloudflare zone.FAIL_OPEN = false. The Worker forwards only requests whose user agent matches the list of major crawlers built into the script, and never a request with no user agent, even after the registry has loaded. The trade-off is that a newer or less common bot can be missed for those first requests.
const FAIL_OPEN = false; and redeploy. If your script has no FAIL_OPEN line it predates this option: copy the current script from LLM Analytics → Setup.
IP addresses. By default the Worker anonymises the client IP address before sending it (ANONYMIZE_IP = true near the top of the script), so the full address never leaves your Cloudflare zone: 203.0.113.7 is sent as 203.0.113.0, and 2001:db8:85a3:8d3::1 as 2001:db8:85a3::. Set ANONYMIZE_IP to false to send full addresses. To send no address at all, replace the ClientIP line with ClientIP: "0.0.0.0",. Bot detection and country data are unaffected either way, because neither uses the IP. What changes is the count of distinct IP addresses per bot, which counts networks rather than addresses when anonymised (a project that also sends full addresses from another integration, such as Logpush, gets one count mixing the two), and the IP stored on each request record. A Worker script copied or deployed before anonymisation existed keeps sending full addresses until it is replaced: copy the current script from LLM Analytics → Setup, or redeploy from there.
Storing the token as a secret (recommended)
For tighter security, store the token as a Worker secret instead of leaving it inline.- In your Worker, open Settings → Variables and Secrets
- Add a secret named
INTEGRATION_TOKENwith yoursa_…token as the value - In the script, blank the inline
INTEGRATION_TOKENconstant:const INTEGRATION_TOKEN = ""; - Redeploy
Cost
Each AI-bot request costs you one extra Worker invocation. The Workers Free plan includes 100,000 requests per day — that’s about 3 million crawls per month, which covers nearly all sites. If you exceed it, the Workers Paid plan is $5/month for 10 million requests. The Worker itself does no CPU-intensive work (a regex check and afetch call), so CPU time is negligible.
Troubleshooting
Status stays on 'Waiting for first event'
Status stays on 'Waiting for first event'
Most likely the route isn’t bound correctly.Then click Check in Searchable. If that doesn’t appear in the dashboard, the Worker’s logs (Cloudflare → Worker → Logs) will show whether
- In Cloudflare, open the Worker → Settings → Domains & Routes
- Confirm the route pattern matches your live domain (e.g.
*yourdomain.com/*, not*yourdomain.dev/*) - Confirm the route’s Zone is the zone that’s actually serving traffic for the domain
forwardToSearchable is running and whether Searchable returned a non-2xx status.Worker logs show 401 from Searchable
Worker logs show 401 from Searchable
The token is missing or wrong.
- Confirm
INTEGRATION_TOKEN(inline) or the secret namedINTEGRATION_TOKENis set to asa_…value - If you’ve recently revoked the token in Searchable, generate a new one and update the Worker
- The token must have no quotes around it in the secret value
My site is fronted by Cloudflare but I don't manage the zone
My site is fronted by Cloudflare but I don't manage the zone
You need permission to add Worker routes on the zone. If you don’t have it:
- Ask the zone admin to add the route, or
- Use the custom REST API path from your application server
I see double-counted events
I see double-counted events
If you also have Cloudflare Logpush sending traffic to Searchable, you’ll get the same request from both sources. Searchable de-duplicates server-side using the Cloudflare Ray ID, so the dashboard shows each request once. If you’d rather not run both, pick one and disable the other.
Removing the integration
Revoking the token stops the Worker from sending within a minute; redeploy from Setup to mint a new one (auto-deployed Workers), or paste a new token into the Worker (manual installs).- Cloudflare → Worker → Settings → Domains & Routes → remove the route (the Worker stops observing traffic)
- Optionally delete the Worker itself
- Searchable → LLM Analytics → Setup → Tokens → revoke the token
Next steps
Cloudflare Logpush
On the Enterprise plan? See the native Logpush path.
See the data
Open LLM Analytics to see which assistants are crawling your site.