Skip to main content

​
What this is

A small must-use plugin that runs inside WordPress, inspects the user agent of every request that reaches PHP, and forwards the ones that match Searchable’s canonical AI-crawler list to LLM Analytics. It feeds the same server-side pipeline as the Vercel, Cloudflare, and Netlify connectors — same classifier, same dashboards. The difference is where it runs: in your application, not at a CDN edge.
Must-use plugins load automatically and can’t be deactivated from the admin UI, so the forwarder survives plugin and theme updates.

​
First, check this is the right path

If your site sits behind Cloudflare, use the Cloudflare Worker instead. It runs at the edge, so it sees every AI-crawler request — including the ones a full-page cache serves without ever booting PHP. It works on every Cloudflare plan including Free, and it’s less to maintain.
Use the WordPress plugin when your site isn’t behind a CDN we support natively, or when you specifically want the forwarder in the application layer. The same caveat applies to any full-page cache. See Troubleshooting below.

​
Before you start

Self-hosted WordPress (this does not work on WordPress.com’s hosted plans, which don’t allow mu-plugins or wp-config.php edits)
SFTP, SSH, or a file manager that can write to wp-content/ and wp-config.php
Your project’s domain confirmed in LLM Analytics → Setup → Confirm your domain
LLM Analytics is available on every plan — no upgrade required.

​
Step 1: Open the WordPress connector

1

Go to the Setup tab

In Searchable, open LLM Analytics (left nav, under Analytics) → Setup, then pick WordPress under “Connect your crawler source”.
2

Generate a tracker API key

Click Generate key. Searchable mints a key with the log_events permission, scoped to this project, and shows it once — copy it now. Keys start with sk_live_….
The key is a live write credential. Treat it like a password. If you lose it, revoke and regenerate from the same panel or Settings → API Keys.
3

Copy the wp-config snippet

The panel shows a define() block with your project’s site token already filled in. Copy it verbatim — you’ll paste it in Step 3.

​
Step 2: Install the plugin

1

Download the plugin file

Click Download plugin in the WordPress panel. It downloads searchable-ai-traffic.php, served from your project’s tracker host.
Download from the button rather than reusing a copy from a colleague. The artifact is refreshed in place on every release, so the button always gives you the current plugin — and the download is pinned to the tracker host your project actually posts to.
2

Upload it to wp-content/mu-plugins/

Create wp-content/mu-plugins/ if it doesn’t already exist, then copy searchable-ai-traffic.php directly into it.
WordPress does not load must-use plugins from subdirectories. The file has to sit at wp-content/mu-plugins/searchable-ai-traffic.php, not wp-content/mu-plugins/searchable/searchable-ai-traffic.php.
There is nothing to activate. Must-use plugins load on every request automatically, and appear under the Must-Use filter on the admin Plugins screen (not in the main list, and with no activate/deactivate controls).

​
Step 3: Add your credentials

Open wp-config.php and paste the snippet from Step 1 above the line that reads /* That's all, stop editing! Happy publishing. */.
wp-config.php
Replace sk_live_YOUR_API_KEY with the key you generated. The site token is already filled in for you in the in-app snippet.
The plugin deliberately stays inert — it sends nothing at all — while either constant still contains YOUR_. If nothing arrives in your dashboard, this is the first thing to check.
Projects on a dedicated tracker hostname see two extra define() lines in the in-app snippet (SEARCHABLE_API_URL and SEARCHABLE_BOTS_URL). Don’t drop them — without them the plugin falls back to the shared host, and you’d end up downloading the plugin from one endpoint while posting events to another. Most projects use the shared host and won’t see these lines.

​
Why wp-config.php and not the plugin file

wp-config.php sits outside the plugin directory, is excluded from most deployment pipelines by default, and won’t leak the key if mu-plugins/ is ever served as plain text.

​
Optional constants

​
Step 4: Verify

Send a request to your site with a known crawler user agent:
Expected result: the request appears in LLM Analytics within roughly 30 seconds.
  1. Check the Logs tab first — it shows individual bot requests as they land.
  2. On the Setup tab, the WordPress card flips from Waiting for the first event from your WordPress site… to Connected — the plugin is sending events.
Real crawler traffic follows on its own schedule. AI bots typically hit any indexed site within a few hours, so don’t read an empty dashboard an hour after install as a failure — the curl above is the definitive test that the wiring works.

​
How it works

  • Bot list. The plugin fetches Searchable’s canonical bot list from /v1/bots.json, validates each pattern, and caches the compiled list in a transient for one hour. If the fetch fails it falls back to the last good list persisted in wp_options, so a network blip doesn’t silently disarm the filter. Failures are cached for five minutes, so an outage never adds a fetch attempt to every request.
  • Matching. On the shutdown hook the plugin matches the request’s user agent against that list. Non-matching requests do nothing — no outbound call, no work beyond the regex pass. WP-CLI, cron, and CLI requests are skipped entirely.
  • Latency. On PHP-FPM the response is flushed to the crawler via fastcgi_finish_request() before the outbound POST, so forwarding adds no latency. On other SAPIs the POST is blocking with a 2-second timeout, and only ever fires for matched crawler requests.
  • IP handling. The client IP is resolved proxy-aware (CF-Connecting-IP → left-most X-Forwarded-For → X-Real-IP → REMOTE_ADDR, each validated) and anonymized on your server before sending — the last IPv4 octet or IPv6 group is zeroed, so the full IP never leaves the site.

​
What gets sent

Only requests from user agents matching Searchable’s canonical AI-crawler list are forwarded. Human traffic is never sent. Each matched request sends:
  • HTTP method, path, and full URL (query strings are stripped server-side before storage)
  • Response status code and server response time
  • User agent and referrer
  • An anonymized client IP — last octet zeroed, or omitted entirely with SEARCHABLE_SEND_IP
  • A source tag of wordpress, so these events are distinguishable from Middleware SDK or edge-connector events
Cookies, request and response bodies, other headers, and full IP addresses are never sent.

​
Optional: enhance the data

The plugin is a crawler-log source. It powers the Overview, Pages, AI Crawlers and Logs tabs. To populate the Visitors tab — humans arriving on your site from AI answers — connect Google Analytics 4, Search Console, or Bing in step 3 of the Setup wizard. These are additive; they don’t replace the crawler-log source.

​
Troubleshooting

Work through these in order:
  1. Placeholders still in place. The plugin no-ops while SEARCHABLE_API_KEY or SEARCHABLE_SITE_TOKEN contains YOUR_. Confirm both were replaced with real values.
  2. Wrong directory. The file must be directly in wp-content/mu-plugins/ — not plugins/, and not a subdirectory. Confirm it appears under the Must-Use filter on the admin Plugins screen. If that filter isn’t shown at all, WordPress isn’t seeing any mu-plugin — check the path over SFTP.
  3. Constant names. They’re case-sensitive and must be spelled exactly as in the snippet.
  4. Outbound HTTP blocked. Some managed hosts restrict wp_remote_post to an allowlist. Ask your host to allow tracker.searchableanalytics.com (or your dedicated tracker hostname).
Full-page caches — WP Rocket, WP Super Cache, W3 Total Cache, Varnish, Cloudflare APO, or host-level caches on WP Engine and Kinsta — serve cached HTML without booting PHP. When that happens the shutdown hook never fires and the crawler visit is never recorded.Two options:
  • Exclude known crawler user agents from the cache, so their requests always reach PHP.
  • Move to an edge-level integration — the Cloudflare Worker is the usual answer, since it runs in front of the cache.
If you do neither, you’ll still capture the origin-level hits that pass through on cache misses and TTL expiry — an undercount, not a zero.
You’ve connected Google Analytics or Search Console but no crawler-log source. The Overview, Pages, AI Crawlers and Logs tabs need this plugin (or a Cloudflare, Vercel, or other edge source) working.
Searchable’s canonical bot list is regenerated once daily at 03:00 UTC, and the plugin caches its own copy for up to another hour. A newly added crawler can therefore take up to ~25 hours to start being captured.Deleting the searchable_bot_patterns transient forces a refetch, but that only helps once the daily upstream publish has picked the new bot up. To see what the plugin is working from, fetch the list yourself:
On PHP-FPM they shouldn’t be — the response is flushed before the POST. If your host runs mod_php or another non-FPM SAPI, matched crawler requests carry a blocking POST bounded at 2 seconds. Human traffic is unaffected either way, since non-matching requests never make an outbound call.

​
Next steps

See the data

Open LLM Analytics to see which assistants are crawling your site.

Connect Search Console

Layer in keyword data so you can correlate AI crawls with search demand.