> ## Documentation Index > Fetch the complete documentation index at: https://docs.searchable.com/llms.txt > Use this file to discover all available pages before exploring further. # Send Akamai traffic to Searchable (DataStream 2) > Configure an Akamai DataStream 2 stream to push request logs to Searchable over HTTPS. No code changes — all configuration is done inside Akamai Control Center. ## What this does Akamai [DataStream 2](https://techdocs.akamai.com/datastream2/docs/welcome-datastream) ships near-real-time request logs from your Akamai properties to a destination of your choosing. We point that stream at Searchable's tracker endpoint, classify the AI bots at the edge, and drop everything else. No code changes — all configuration happens inside Akamai Control Center. **No code changes.** All configuration is done inside Akamai Control Center. ## Prerequisites An Akamai account with **DataStream 2** entitled on the contract that owns your delivery property The **Edit** role on that property, or an Akamai admin who can author and activate a stream A Searchable project with your domain confirmed DataStream 2 is a per-contract product. If the **DataStream** entry doesn't appear in Control Center, your Akamai contract doesn't include it yet — your Akamai account team can add it. The legacy DataStream 1 product is not supported by this integration; you need DataStream 2 (the JSON-format successor). ## Setup 1. Open your Searchable dashboard 2. Go to **LLM Analytics → Setup** 3. Pick **Akamai** as your crawler source 4. Click **Generate token** Copy the token now — it starts with `sa_…` and won't be shown again. You can always generate a new one if you lose it. The endpoint URL is fixed: ``` https://tracker.searchableanalytics.com/v1/akamai-logs ``` Sanity check before pointing DataStream at the endpoint: `curl https://tracker.searchableanalytics.com/v1/akamai-logs` should return `200 ok`. That's the same health check Control Center's **Validate and save** step runs against the URL when you finish the wizard. In Akamai Control Center, navigate to: **CDN → DataStream → Create new stream** Give the stream a name like `Searchable LLM Analytics` and assign it to the group that contains the property fronting your domain. Pick the **property** that serves the traffic you want Searchable to see — DataStream is scoped per-property, not per-account. If your domain is fronted by more than one property (for example, `www` and `api` on separate properties), create one stream per property and point them all at the same endpoint with the same token. Searchable tags events by host, so they still arrive split by domain in the dashboard. On the **Data sets** step, switch the log format to **Structured JSON**, then select the fields below. The sub-country admin region is `state` (not `region`). In Structured JSON, Akamai emits the request-ID field as `reqId` (lowercase `d`) and all numeric values as JSON strings. Searchable accepts both shapes, so you don't need to do anything special on your end. Required (the endpoint drops records missing any of these): * `reqId` * `reqMethod` * `reqHost` * `reqPath` * `UA` * `statusCode` * `cliIP` * `reqTimeSec` Recommended (used for enrichment + debugging; missing fields don't break the integration): * `queryStr` * `referer` *(HTTP-standard single-r spelling)* * `rspContentLen` * `reqEndTimeMSec` *(total request duration in ms, despite the name)* * `turnAroundTimeMSec` *(time-to-first-byte in ms)* * `cacheStatus` * `country` * `state` * `city` * `edgeIP` * `tlsVersion` The exact list, comma-separated for quick copy/paste, is also shown in the Searchable setup card. `reqId` is load-bearing. DataStream 2 has at-least-once delivery semantics, so the same record can arrive in multiple batches; Searchable derives a stable event ID from `reqId` to deduplicate them. If you skip `reqId`, dedup falls back to a heuristic that's noticeably less precise — your dashboard will show inflated counts after stream retries. **`reqTimeSec` vs `reqEndTimeMSec`**: despite the suffix, `reqEndTimeMSec` is a *duration* (how long the request took, in ms), not a wall-clock value. The actual epoch timestamp is `reqTimeSec` (epoch seconds). Searchable uses `reqTimeSec` for the event time and `reqEndTimeMSec` for `response_time_ms` — both fields are useful and we capture both when present. On the **Delivery** step, choose **HTTPS** and fill in: | Field | Value | | ------------------------------ | -------------------------------------------------------- | | **Endpoint URL** | `https://tracker.searchableanalytics.com/v1/akamai-logs` | | **Authentication** | None | | **Custom HTTP header — name** | `Authorization` | | **Custom HTTP header — value** | `Bearer ` | | **Compress files** | Off *(see note below)* | Set **Authentication** to **None** and put the token in a **custom header**. Akamai's built-in Authentication choices (Basic, mTLS) are for upstreams that expect those formats — Searchable verifies a signed Bearer token, so the right slot is a plain custom header. The header value must be `Bearer ` followed by the full `sa_…` token, with one space and no quotes. The header name must be exactly `Authorization`. DataStream 2's HTTPS delivery sends batched, line-delimited JSON. Searchable's endpoint accepts both uncompressed and gzip-compressed bodies. If you turn on **Compress files**, leave the rest of the settings unchanged — Akamai sets `Content-Encoding: gzip` automatically and our endpoint decompresses on the way in. On the **Frequency** step, choose **30 seconds** (the default) or **60 seconds**. Either is fine — both keep dashboard latency under a minute. Don't choose the largest "files per group" / longest interval option. Larger Akamai batches are still well under our 5 MB payload cap on any realistic traffic level, but tighter intervals give you faster feedback in the dashboard while debugging. On the **Review** step, click **Validate and save**. Akamai sends a `GET` to the endpoint URL and expects a `200` response — Searchable's tracker responds with a literal `ok`. If validation fails, see [Troubleshooting](#troubleshooting) below. Once validation succeeds, click **Activate**. Akamai activation typically completes within a couple of minutes on Staging and 5–10 minutes on Production. Once active, expect events in Searchable as soon as an AI bot hits your site. ## Verifying the connection In Searchable: 1. Go to **LLM Analytics → Setup** 2. Look at the Akamai card status 3. Click **Check** if it still shows "Waiting for first event" | Status | What it means | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Waiting for first event** | The stream is configured but no AI bot has hit your site yet. Typical wait is a few hours for sites that are already indexed. | | **Connected** | Events are arriving. The card shows the count from the last 24 hours. | You can also confirm in Akamai: **CDN → DataStream → your stream → Monitor**. The stream's metrics show outgoing batches and the HTTP response codes from our endpoint. Healthy delivery looks like a steady stream of `204` responses. ## What Searchable receives For each request that matches an AI-bot user agent, Searchable receives: * HTTP method, path, and host (query strings stripped before storage) * User agent * Referer * Country, state, and city (from `country` / `state` / `city`) * Response status and bytes out (`rspContentLen`) * Edge turnaround time (`turnAroundTimeMSec`) * Akamai edge metadata (`edgeIP`, `tlsVersion`, `cacheStatus`) — preserved as `custom_properties` for debugging * The DataStream 2 `reqId` — used as the dedup key on our side Bodies, headers other than `User-Agent` / `Referer`, cookies, and full IPs are never sent or stored. The DataStream 2 `reqId` is hashed into a deterministic event ID so retried batches collapse into a single event server-side. ## Troubleshooting Akamai's validator does a `GET` against your endpoint URL and expects `200`. The two common reasons it fails: * **Typo in the URL** — the correct value is exactly `https://tracker.searchableanalytics.com/v1/akamai-logs`. No trailing slash, no path suffix. * **A custom Akamai delivery network policy** is blocking egress to `tracker.searchableanalytics.com`. In strict environments this surfaces as a connection error rather than a 4xx. If you suspect this, ask your Akamai admin to allow-list that hostname for the delivery property. The custom header isn't validated at this step — Akamai's `Validate and save` only verifies that the URL is reachable. Auth issues surface in the **Monitor** tab after the stream activates. The Authorization header is missing or wrong. * Make sure you added a **custom HTTP header** named exactly `Authorization` (not `X-Authorization`, not `Bearer`) * The value must be `Bearer ` followed by the full `sa_…` token, with one space and no quotes * Confirm the **Authentication** dropdown is set to **None** — picking **Basic** there overrides the custom header * If you've recently revoked the token in Searchable, generate a new one and update the stream's custom header (no need to recreate the stream) In Akamai's **Monitor** tab, repeated `401` responses are the visible symptom — fix the header and the next batch will succeed. Map the response code to the cause: * **`401 Unauthorized`** — see the Authorization-header section above. * **`413 Payload Too Large`** — the batch exceeded Searchable's 5 MB payload cap. This is rare on default frequency settings (30s / 60s); if you see it, lower the upload interval rather than raising file sizes. While 4xx errors are occurring, Akamai retries each batch a few times and then drops it. Fix the root cause and the stream catches up automatically with the next batch. The endpoint silently drops records that are missing required fields rather than rejecting the whole batch, so a misconfigured **Data sets** step shows up as healthy delivery on Akamai's side and empty cards on Searchable's side. Open the stream's **Data sets** step and confirm every required field is selected: * `reqId`, `reqMethod`, `reqHost`, `reqPath`, `UA`, `statusCode`, `cliIP`, `reqTimeSec` Field names are case-sensitive. The single most common cause of an empty dashboard is missing `reqTimeSec` — without it, events land at the wrong timestamp (the year 1970) and fall outside the dashboard's 24-hour window even though they're technically stored. If all required fields are present, the next most common cause is a domain mismatch — see "Status stays on 'Waiting for first event'" below. This is almost always the `reqTimeSec` issue described above. `reqEndTimeMSec` is a *duration* (request processing time in ms), not a wall-clock timestamp despite its name — if you only ticked `reqEndTimeMSec` and not `reqTimeSec`, every event ends up with a timestamp a few milliseconds past `1970-01-01`. Fix: open **Data sets** and tick `reqTimeSec`. The stream re-activates in place; the next batch will arrive with correct timestamps. `reqID` is populated with a non-unique value, or `reqID` was omitted from the field selection altogether. * Open the stream's **Data sets** step and confirm `reqID` (capital `ID`) is selected — Akamai's UI lists it alphabetically near the bottom of the request-section fields * `reqID` is Akamai's per-request unique identifier and is auto-populated; no property-level VCL or PMUSER variables required * If `reqID` is selected and you still see this symptom, contact support and include a few sample records — most often this turns out to be a property-level override blanking the value Without `reqID`, Searchable falls back to a (timestamp, path, user-agent) heuristic that's much coarser. The fix is non-destructive — add `reqID` and republish. DataStream 2's documented epoch field is `reqEndTimeMSec` — milliseconds since the Unix epoch. Some legacy stream configurations use `reqTimeSec` (seconds) by mistake. Searchable handles this defensively: if the value looks like seconds (below \~year-2001-in-ms), we scale it up to milliseconds before storage. So you shouldn't see this in practice — but if you've copied a stream config from a much older DataStream 1 export, double-check that the field is `reqEndTimeMSec` and not `reqTimeSec`. A few possible causes, in order of likelihood: * The stream isn't activated — open the stream in Control Center and confirm the status badge reads **Active**, not **Inactive** or **Pending activation**. Activations on Production take 5–10 minutes * Your domain in Searchable doesn't match the host the Akamai property serves (check **LLM Analytics → Setup → Confirm your domain**) * The Akamai property fronts a different host than the one you're testing with — DataStream is scoped per-property, so traffic going through a different property won't appear * No AI bot has visited yet — try visiting your site with a known AI user agent (e.g. `Mozilla/5.0 (compatible; GPTBot/1.0)`) to trigger a test event If Akamai's **Monitor** tab shows successful `204` responses but Searchable still says no events, the issue is almost always a domain mismatch on the Searchable side. Akamai's standard staging-network workflow works fine — activate the stream on **Staging** first, then point a few test requests at your staging hostname (Akamai exposes the staging network at `.edgesuite-staging.net` or via a `Pragma: akamai-x-cache-on` header trick from your origin). Successful staging delivery is a strong signal the production activation will work as well. You'll see events arrive in the same Searchable project — there's no separate staging endpoint to worry about. ## Multi-site and region-level properties (domain mapping) A single Akamai property sometimes serves traffic for many different hostnames — for example, a region-level property that fronts `example.co.uk`, `example.de`, and `example.fr`. With a standard integration token those requests would all land on one Searchable project. Domain mapping lets you route each hostname to the right Searchable project from a single DataStream stream, without touching the Akamai config or reissuing the token. **Gated rollout.** The domain-mapping manager is available on request. If you don't see a **Domain mappings** section on the Setup tab, reach out to support to have it enabled for your account. ### How it works 1. In Searchable, go to **LLM Analytics → Setup** and open **Domain mappings**. 2. Create a mapping and add host rules — one rule per hostname your Akamai property serves (e.g. `example.co.uk` → the UK project, `example.de` → the German project, and so on). 3. Generate a **domain-mapping token** from the same manager. The token looks the same as a standard `sa_…` integration token and is used in the same `Authorization: Bearer …` header. 4. Paste the token into your DataStream's **Custom HTTP header** field (replacing any existing token). Searchable inspects the `reqHost` field of each incoming log record and routes it to the matching project. **Hostnames that have no rule are silently discarded** — they are never stored. Rule changes propagate within about a minute; you don't need to touch the Akamai stream or generate a new token. Any hostname your Akamai property serves that is **not** covered by a domain-mapping rule will be dropped at ingestion and will not appear in any Searchable dashboard. Double-check your rules cover every hostname you want tracked. ### Diagnostic events DataStream diagnostic envelopes from a mapping token are not attributable to a single project and are dropped in the current release. If you need per-batch ingestion diagnostics, use a standard (single-project) integration token instead. ## Removing the integration To stop sending traffic to Searchable: 1. Akamai → **CDN → DataStream → your stream → Deactivate** (and optionally delete the stream once it's deactivated) 2. Searchable → **LLM Analytics → Setup → Tokens** → revoke the token Both sides are independent — revoking the token alone is enough to stop ingestion immediately, even if the stream stays configured in Akamai (its deliveries will start returning `401`, which Akamai will retry and then drop). ## Next steps Open LLM Analytics to see which assistants are crawling your site. Correlate AI crawls with search demand.