Skip to main content

​
What this does

Akamai DataStream 2 ships near-real-time request logs from your Akamai properties to a destination of your choosing. We point that stream at Searchable’s tracker endpoint, classify the AI bots at the edge, and drop everything else. No code changes — all configuration happens inside Akamai Control Center.
No code changes. All configuration is done inside Akamai Control Center.

​
Prerequisites

An Akamai account with DataStream 2 entitled on the contract that owns your delivery property
The Edit role on that property, or an Akamai admin who can author and activate a stream
A Searchable project with your domain confirmed
DataStream 2 is a per-contract product. If the DataStream entry doesn’t appear in Control Center, your Akamai contract doesn’t include it yet — your Akamai account team can add it. The legacy DataStream 1 product is not supported by this integration; you need DataStream 2 (the JSON-format successor).

​
Setup

1

Generate an integration token in Searchable

  1. Open your Searchable dashboard
  2. Go to LLM Analytics → Setup
  3. Pick Akamai as your crawler source
  4. Click Generate token
Copy the token now — it starts with sa_… and won’t be shown again. You can always generate a new one if you lose it.The endpoint URL is fixed:
Sanity check before pointing DataStream at the endpoint: curl https://tracker.searchableanalytics.com/v1/akamai-logs should return 200 ok. That’s the same health check Control Center’s Validate and save step runs against the URL when you finish the wizard.
2

Open the DataStream 2 wizard

In Akamai Control Center, navigate to:CDN → DataStream → Create new streamGive the stream a name like Searchable LLM Analytics and assign it to the group that contains the property fronting your domain. Pick the property that serves the traffic you want Searchable to see — DataStream is scoped per-property, not per-account.
If your domain is fronted by more than one property (for example, www and api on separate properties), create one stream per property and point them all at the same endpoint with the same token. Searchable tags events by host, so they still arrive split by domain in the dashboard.
3

Set the log file format to Structured JSON

Before picking fields, set the log format. On the Data sets step, look near the top for Log file format and set it to Structured JSON.
“JSON” and “Structured JSON” look almost identical in the dropdown — pick Structured JSON. Only Structured JSON emits one record per line (NDJSON), which is what our endpoint ingests. Choosing plain JSON is the single most common cause of a stream that delivers successfully on Akamai’s side but produces no events in Searchable.
4

Pick the data sets (log format fields)

Still on the Data sets step, select the fields below. Field names are case-sensitive — note UA is capitalised (it is not reqUserAgent), and the sub-country admin region is state (not region).Select all of these:
  • reqID
  • reqMethod
  • reqHost
  • reqPath
  • UA (capital — not reqUserAgent)
  • statusCode
  • cliIP
  • reqTimeSec
  • reqEndTimeMSec
Of those, the endpoint drops any record missing reqID, reqMethod, reqHost, reqPath, UA, statusCode, cliIP, or reqTimeSec. reqEndTimeMSec is optional in the strict sense — a record without it is still ingested — but tick it anyway: it’s what populates response time in your dashboard.Recommended additions (used for enrichment + debugging; missing fields don’t break the integration):
  • queryStr
  • referer (HTTP-standard single-r spelling)
  • rspContentLen
  • turnAroundTimeMSec (time-to-first-byte in ms)
  • cacheStatus
  • country
  • state
  • city
  • edgeIP
  • tlsVersion
The exact list, comma-separated for quick copy/paste, is also shown in the Searchable setup card.
In Structured JSON, Akamai emits the request-ID field as reqId (lowercase d) and all numeric values as JSON strings. Searchable accepts either casing (reqID or reqId) and coerces the strings, so you don’t need to do anything special on your end — just tick the field as Control Center labels it.
After changing the format or the field selection, confirm your fields are still ticked and save — switching the log file format can reset the selection.
reqID is load-bearing and required. DataStream 2 has at-least-once delivery semantics, so the same record can arrive in multiple batches; Searchable derives a stable event ID from reqID to deduplicate them. Records without it are dropped.
reqTimeSec vs reqEndTimeMSec: despite the suffix, reqEndTimeMSec is a duration (how long the request took, in ms), not a wall-clock value. The actual epoch timestamp is reqTimeSec (epoch seconds). Searchable uses reqTimeSec for the event time and reqEndTimeMSec for response_time_ms — both fields are useful and we capture both when present.
5

Configure the HTTPS delivery

On the Delivery step, choose HTTPS and fill in:
Use X-Searchable-Token, not Authorization — and do not prefix the value with Bearer . Paste the raw sa_… token as the header value, with no prefix, no space, and no quotes.
Set Authentication to None and put the token in a custom header. Akamai’s built-in Authentication choices (Basic, mTLS) are for upstreams that expect those formats — Searchable verifies its own signed token, so the right slot is a plain custom header.
DataStream 2’s HTTPS delivery sends batched, line-delimited JSON. Searchable’s endpoint accepts both uncompressed and gzip-compressed bodies. If you turn on Compress files, leave the rest of the settings unchanged — Akamai sets Content-Encoding: gzip automatically and our endpoint decompresses on the way in.
6

Set the upload frequency

On the Frequency step, choose 30 seconds (the default) or 60 seconds. Either is fine — both keep dashboard latency under a minute.Don’t choose the largest “files per group” / longest interval option. Larger Akamai batches are still well under our 5 MB payload cap on any realistic traffic level, but tighter intervals give you faster feedback in the dashboard while debugging.
7

Validate and activate the stream

On the Review step, click Validate and save. Akamai sends a GET to the endpoint URL and expects a 200 response — Searchable’s tracker responds with a literal ok. If validation fails, see Troubleshooting below.Once validation succeeds, click Activate. Akamai activation typically completes within a couple of minutes on Staging and 5–10 minutes on Production. Once active, expect events in Searchable as soon as an AI bot hits your site.

​
Verifying the connection

In Searchable:
  1. Go to LLM Analytics → Setup
  2. Look at the Akamai card status
  3. Click Check if it still shows “Waiting for first event”
You can also confirm in Akamai: CDN → DataStream → your stream → Monitor. The stream’s metrics show outgoing batches and the HTTP response codes from our endpoint. Healthy delivery looks like a steady stream of 204 responses.

​
What Searchable receives

For each request that matches an AI-bot user agent, Searchable receives:
  • HTTP method, path, and host (query strings stripped before storage)
  • User agent
  • Referer
  • Country, state, and city (from country / state / city)
  • Response status and bytes out (rspContentLen)
  • Edge turnaround time (turnAroundTimeMSec)
  • Akamai edge metadata (edgeIP, tlsVersion, cacheStatus) — preserved as custom_properties for debugging
  • The DataStream 2 reqID — used as the dedup key on our side
Bodies, headers other than User-Agent / Referer, cookies, and full IPs are never sent or stored. The DataStream 2 reqID is hashed into a deterministic event ID so retried batches collapse into a single event server-side.

​
Troubleshooting

Akamai’s validator does a GET against your endpoint URL and expects 200. The two common reasons it fails:
  • Typo in the URL — the correct value is exactly https://tracker.searchableanalytics.com/v1/akamai-logs. No trailing slash, no path suffix.
  • A custom Akamai delivery network policy is blocking egress to tracker.searchableanalytics.com. In strict environments this surfaces as a connection error rather than a 4xx. If you suspect this, ask your Akamai admin to allow-list that hostname for the delivery property.
The custom header isn’t validated at this step — Akamai’s Validate and save only verifies that the URL is reachable. Auth issues surface in the Monitor tab after the stream activates.
The token header is missing or wrong.
  • Make sure you added a custom HTTP header named exactly X-Searchable-Token
  • The value must be the raw sa_… token with no Bearer prefix, no quotes, and no leading or trailing space. A value of Bearer sa_… will fail — paste the token on its own
  • Confirm the Authentication dropdown is set to None — picking Basic there overrides the custom header
  • If you’ve recently revoked the token in Searchable, generate a new one and update the stream’s custom header (no need to recreate the stream)
In Akamai’s Monitor tab, repeated 401 responses are the visible symptom — fix the header and the next batch will succeed. A 403 instead means the header arrived but the token failed signature verification: it’s truncated, from a different environment, or has been revoked.
Map the response code to the cause:
  • 401 Unauthorized — the token header is missing or malformed; see the section above.
  • 403 Forbidden — the header arrived but the token failed verification (truncated, revoked, or from a different environment).
  • 413 Payload Too Large — the batch exceeded Searchable’s 5 MB payload cap. This is rare on default frequency settings (30s / 60s); if you see it, lower the upload interval rather than raising file sizes.
While 4xx errors are occurring, Akamai retries each batch a few times and then drops it. Fix the root cause and the stream catches up automatically with the next batch.
The endpoint silently drops records it can’t parse or that are missing required fields, rather than rejecting the whole batch — so a misconfigured Data sets step shows up as healthy delivery on Akamai’s side and empty cards on Searchable’s side. Check these two things, in order:1. Log file format is Structured JSON. Open Data sets and check Log file format near the top. It must be Structured JSON, not JSON — the two look nearly identical in the dropdown, but only Structured JSON emits one record per line, which is what our endpoint parses. This is the most common cause of a stream that delivers 204s and produces nothing.2. Every required field is selected:
  • reqID, reqMethod, reqHost, reqPath, UA, statusCode, cliIP, reqTimeSec
Field names are case-sensitive — UA is capitalised and is not reqUserAgent. After missing UA, the most common single-field cause is missing reqTimeSec: without it the record is dropped outright.If the format is right and all required fields are present, the next most common cause is a domain mismatch — see “Status stays on ‘Waiting for first event’” below.
These two are not interchangeable, despite the similar names:
  • reqTimeSec is the wall-clock timestamp of the request (epoch seconds). This is what Searchable uses to place the event in time, and it is required — a record without it is dropped.
  • reqEndTimeMSec is a duration — how long the request took to process, in ms — not a wall-clock value despite the name. Searchable uses it for response time only.
Tick both. If you ticked only reqEndTimeMSec, no events reach your dashboard at all, because every record fails the required-field check.Fix: open Data sets, tick reqTimeSec, and save. The stream re-activates in place and the next batch arrives correctly.
reqID is populated with a non-unique value, or reqID was omitted from the field selection altogether.
  • Open the stream’s Data sets step and confirm reqID (capital ID) is selected — Akamai’s UI lists it alphabetically near the bottom of the request-section fields
  • reqID is Akamai’s per-request unique identifier and is auto-populated; no property-level VCL or PMUSER variables required
  • If reqID is selected and you still see this symptom, contact support and include a few sample records — most often this turns out to be a property-level override blanking the value
Without reqID, Searchable falls back to a (timestamp, path, user-agent) heuristic that’s much coarser. The fix is non-destructive — add reqID and republish.
The wall-clock field is reqTimeSec — epoch seconds, which Searchable converts to milliseconds on ingest. reqEndTimeMSec is a duration, not a timestamp, and is never used to place an event in time.If timestamps look wrong, the usual cause is a property-level override rewriting reqTimeSec, or a stream config copied from a much older DataStream 1 export that maps a different field into that slot. Confirm reqTimeSec is ticked in Data sets, then contact support with a few sample records if it persists.
A few possible causes, in order of likelihood:
  • The stream isn’t activated — open the stream in Control Center and confirm the status badge reads Active, not Inactive or Pending activation. Activations on Production take 5–10 minutes
  • Your domain in Searchable doesn’t match the host the Akamai property serves (check LLM Analytics → Setup → Confirm your domain)
  • The Akamai property fronts a different host than the one you’re testing with — DataStream is scoped per-property, so traffic going through a different property won’t appear
  • No AI bot has visited yet — try visiting your site with a known AI user agent (e.g. Mozilla/5.0 (compatible; GPTBot/1.0)) to trigger a test event
If Akamai’s Monitor tab shows successful 204 responses but Searchable still says no events, the issue is almost always a domain mismatch on the Searchable side.
Akamai’s standard staging-network workflow works fine — activate the stream on Staging first, then point a few test requests at your staging hostname (Akamai exposes the staging network at <your-host>.edgesuite-staging.net or via a Pragma: akamai-x-cache-on header trick from your origin). Successful staging delivery is a strong signal the production activation will work as well.You’ll see events arrive in the same Searchable project — there’s no separate staging endpoint to worry about.

​
Multi-site and region-level properties (domain mapping)

A single Akamai property sometimes serves traffic for many different hostnames — for example, a region-level property that fronts example.co.uk, example.de, and example.fr. With a standard integration token those requests would all land on one Searchable project. Domain mapping lets you route each hostname to the right Searchable project from a single DataStream stream, without touching the Akamai config or reissuing the token.
Gated rollout. The domain-mapping manager is available on request. If you don’t see a Domain mappings section on the Setup tab, reach out to support to have it enabled for your account.

​
How it works

  1. In Searchable, go to LLM Analytics → Setup and open Domain mappings.
  2. Create a mapping and add host rules — one rule per hostname your Akamai property serves (e.g. example.co.uk → the UK project, example.de → the German project, and so on).
  3. Generate a domain-mapping token from the same manager. The token looks the same as a standard sa_… integration token and goes in the same X-Searchable-Token header, raw and with no Bearer prefix.
  4. Paste the token into your DataStream’s Custom HTTP header field (replacing any existing token).
Searchable inspects the reqHost field of each incoming log record and routes it to the matching project. Hostnames that have no rule are silently discarded — they are never stored. Rule changes propagate within about a minute; you don’t need to touch the Akamai stream or generate a new token.
Any hostname your Akamai property serves that is not covered by a domain-mapping rule will be dropped at ingestion and will not appear in any Searchable dashboard. Double-check your rules cover every hostname you want tracked.

​
Diagnostic events

DataStream diagnostic envelopes from a mapping token are not attributable to a single project and are dropped in the current release. If you need per-batch ingestion diagnostics, use a standard (single-project) integration token instead.

​
Removing the integration

To stop sending traffic to Searchable:
  1. Akamai → CDN → DataStream → your stream → Deactivate (and optionally delete the stream once it’s deactivated)
  2. Searchable → LLM Analytics → Setup → Tokens → revoke the token
Both sides are independent — revoking the token alone is enough to stop ingestion immediately, even if the stream stays configured in Akamai (its deliveries will start returning 401, which Akamai will retry and then drop).

​
Next steps

See the data

Open LLM Analytics to see which assistants are crawling your site.

Add Search Console

Correlate AI crawls with search demand.