Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.searchable.com/llms.txt

Use this file to discover all available pages before exploring further.

​
What this does

Fastly’s Real-Time Log Streaming ships request log records to a destination of your choosing in near-real-time. We point that stream at Searchable’s tracker endpoint, classify the AI bots, and drop everything else. No code changes — all configuration happens inside your Fastly service.
No code changes. All configuration is done inside the Fastly UI.

​
Prerequisites

A Fastly account on any paid plan (HTTPS Logging is available on all paid plans)
Admin or Engineer-level access to the Fastly service you want to instrument
A Searchable project with your domain confirmed

​
Setup

1

Generate an integration token in Searchable

  1. Open your Searchable dashboard
  2. Go to Agent Analytics → Setup
  3. Pick Fastly as your crawler source
  4. Click Generate token
Copy the token now — it starts with sa_… and won’t be shown again. You can always generate a new one if you lose it.
2

Create an HTTPS logging endpoint in Fastly

In the Fastly UI, navigate to your service:Service → Logging → HTTPS → Create endpointThis creates a new draft version of your service config; you’ll activate it at the end.
3

Configure the endpoint

Fill in these fields:
FieldValue
NameSearchable Agent Analytics (or any label you prefer)
URLhttps://searchable-tracker.searchable.workers.dev/v1/fastly-logs
MethodPOST
Content typeapplication/json
Log format version2
Custom header nameAuthorization
Custom header valueBearer <your-sa_-token>
Log format(paste the template below)
The custom header value must be Bearer followed by the full sa_… token, with one space and no quotes. The header name must be exactly Authorization.
4

Paste the log format template verbatim

Paste this into the Log format field exactly as-is. Do not edit any of the %{...}V tokens — they are Fastly VCL substitutions that we depend on for correct request attribution and deduplication:
{
  "timestamp": "%{strftime({"%Y-%m-%dT%H:%M:%S"}, time.start)}V",
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "method": "%{req.method}V",
  "url": "%{req.url}V",
  "host": "%{req.http.host}V",
  "status": %{resp.status}V,
  "response_time": %{time.elapsed.msec}V,
  "user_agent": "%{json.escape(req.http.User-Agent)}V",
  "referer": "%{json.escape(req.http.Referer)}V",
  "fastly_request_id": "%{req.xid}V",
  "country": "%{client.geo.country_code}V",
  "region": "%{client.geo.region}V",
  "city": "%{json.escape(client.geo.city)}V"
}
fastly_request_id must use %{req.xid}V — Fastly’s canonical per-request unique identifier. Substituting any other token (e.g. Fastly-SOA-ID or a custom X-Request-ID) will silently break deduplication on our side and events will appear to be lost.
5

Save and activate the new service version

Click Create to save the endpoint, then Activate the new service version. Fastly batches log records and ships them within a few seconds; expect events to appear in Searchable within a couple of minutes once an AI bot hits your site.

​
Verifying the connection

In Searchable:
  1. Go to Agent Analytics → Setup
  2. Look at the Fastly card status
  3. Click Check if it still shows “Waiting for first event”
StatusWhat it means
Waiting for first eventThe endpoint is configured but no AI bot has hit your site yet. Typical wait is a few hours for sites that are already indexed.
ConnectedEvents are arriving. The card shows the count from the last 24 hours.

​
Geo enrichment

country, region, and city arrive from Fastly’s edge geo lookups (client.geo.*). If geo lookups aren’t enabled on your service (the default is on for paid plans), those fields will arrive empty — everything else still works, and the integration counts as healthy. You can enable geo at Service → Settings → Geolocation.

​
Troubleshooting

The Authorization header is missing or wrong.
  • Make sure you added a custom header named exactly Authorization
  • The value must be Bearer followed by the full sa_… token, with one space and no quotes
  • If you’ve recently revoked the token in Searchable, generate a new one and update the endpoint
The log format is likely wrong.
  • Confirm Log format version is set to 2, not 1 — version 1 wraps each record in a different shape that our endpoint doesn’t accept
  • Confirm the log format template was pasted exactly, including all %{...}V tokens. Do not remove any token, even if it looks redundant
  • Confirm the Content type is application/json
fastly_request_id is being populated with a non-unique value.
  • Confirm the template uses %{req.xid}V (not %{req.http.Fastly-SOA-ID}V or any other header)
  • req.xid is auto-populated by Fastly on every request and requires no special VCL setup — if it’s empty in your logs, contact Fastly support
A few possible causes:
  • The new service version was never activated — go to Service → Versions and confirm the version containing the HTTPS endpoint is currently Active
  • Your domain in Searchable doesn’t match the site served by Fastly (check Agent Analytics → Setup → Confirm your domain)
  • The endpoint is configured but no AI bot has visited yet — try visiting your site with a known AI user agent (e.g. Mozilla/5.0 (compatible; GPTBot/1.0)) to trigger a test event
If deliveries are succeeding in Fastly’s logs view but nothing appears in Searchable, that points to a domain mismatch or an Authorization-header issue.

​
Removing the integration

To stop sending traffic to Searchable:
  1. Fastly → Service → Logging → HTTPS → delete the endpoint and activate a new service version
  2. Searchable → Agent Analytics → Setup → Tokens → revoke the token
Both sides are independent — revoking the token alone is enough to stop ingestion immediately, even if the endpoint stays configured in Fastly (its deliveries will start returning 401).

​
Next steps

See the data

Open Agent Analytics to see which assistants are crawling your site.

Add Search Console

Correlate AI crawls with search demand.