# Metrics API

> Read request rates, error counts, latency percentiles, and byte volumes for your project through typed time-series and snapshot endpoints, or scrape a self-hosted gateway pod with Prometheus.

The metrics API serves traffic measurements for a project from the project API server, backed by aggregated telemetry. It has two shapes: a **snapshot** on an object (current totals for one Gateway, route, Proxy, Service, VPC network, or tunnel) and a **series** query (one or more lines over time). Both read the same measurements, so a tile and the chart under it agree.

The rule for picking one: one object and you want its current numbers, read `/metrics` on that object. Many objects, or any view over time, use `metrics/<name>/series`.

This API measures traffic. For the health of the proxy process itself, read the `envoy.*` recipes: see [Proxy health in the metrics API](#proxy-health-in-the-metrics-api). A gateway you run yourself publishes the same signals as Prometheus series on the pod as well, covered in [Gateway pod metrics (self-hosted)](#gateway-pod-metrics-self-hosted) at the bottom of this page.

## Endpoint and authentication

All requests go to your project API server:

```
https://<project-id>.api.apoxy.dev
```

Authenticate with an API key in the `X-Apoxy-API-Key` header, the same as every other call to this API:

```
X-Apoxy-API-Key: <api-key>
```

`Authorization: Bearer <api-key>` is also accepted. To create a key, see [Authentication](/docs/reference/mcp.md#authentication) in the MCP server reference. Requests without a valid key receive `401 Unauthorized`.

## Resources

The group is `metrics.apoxy.dev/v1alpha1`. All resources are cluster-scoped, and no project identifier appears in any path or parameter because the API server serves exactly one project.

| Resource | Kind | Verbs |
|---|---|---|
| `metrics` | `Metric` | get, list, watch, create, update, delete |
| `metricsources` | `MetricSource` | get, list |
| `metrics/<name>/series` | `MetricSeriesSet` | get |

A `Metric` is a named recipe: a query fragment plus display preferences. The built-in recipes are managed for you; you can also save your own. A `MetricSource` describes what a recipe can read and group by.

## Built-in metrics

Every project starts with these recipes. They carry the `metrics.apoxy.dev/managed: "true"` label.

| Metric | Type | Measures | Unit |
|---|---|---|---|
| `http.requests` | counter | `total`, `status_2xx`, `status_4xx`, `status_5xx` | - |
| `http.errors` | counter | `errors_4xx`, `errors_5xx` | - |
| `http.latency` | histogram | `p50`, `p95`, `p99` | `ms` |
| `http.bytes` | counter | `bytes_in`, `bytes_out` | `By` |
| `log.severity` | counter | `total`, `errors`, `warnings` | - |
| `upstream.connections` | gauge | `active`, `peak` | - |
| `tls.handshakes` | counter | `handshakes` | - |
| `envoy.bandwidth` | counter | `downstream_in`, `downstream_out`, `upstream_out`, `upstream_in` | `By` |
| `network.bytes` | counter | `bytes` | `By` |
| `network.packets` | counter | `packets` | - |
| `network.drops` | counter | `drops` | - |
| `network.keepalives` | counter | `keepalives` | - |
| `network.tunnels` | gauge | `active`, `peak` | - |
| `tunnel.rtt` | gauge | `rtt`, `rtt_peak` | `s` |

`http.requests` and `http.errors` are plain counts and carry no unit, so they contribute nothing to the `units` map in a response. `upstream.connections` reports live upstream connections, and `tls.handshakes` counts downstream TLS connections established in the window. `http.latency` percentiles come from fixed latency buckets, and the largest bound is 60 s, so a percentile that falls past that bound reads as 60000 ms.

The collector stores metrics whose names start with `envoy.` or `apoxy.` when it can identify their project. Raw samples remain in ClickHouse for 72 hours, including new metric names under either prefix.

`upstream.connections` requires a recognized route or Backend. `tls.handshakes` requires a Gateway and listener. Proxy health metrics require both a Proxy name and a replica name. Samples without this identity remain in raw storage and do not contribute to the API results.

`envoy.bandwidth` counts the bytes Envoy moved on the wire, which is why it carries traffic that `http.bytes` cannot see: `http.bytes` counts request and response bodies at the end of a request, so a WebSocket or another long-lived stream stays at zero until it closes.

The catalog also holds the other `envoy.*` recipes, which measure the proxy process rather than the traffic through it. See [Proxy health in the metrics API](#proxy-health-in-the-metrics-api).

The `network.*` and `tunnel.*` recipes measure the tunnels of a VPC network rather than HTTP traffic. `network.bytes` and `network.packets` count every packet of payload a tunnel carries, so east-west traffic and non-HTTP protocols show up in them and not in `http.*`. `network.keepalives` counts the keep-alive frames that keep the tunnel path active; they carry no payload, so they are counted apart and a tunnel that carries no traffic reports keep-alives with zero bytes and zero packets. `network.tunnels` reports how many tunnels a network had connected, counted across every relay that serves it; its `peak` adds the largest count each relay saw, so it is an upper bound. `tunnel.rtt` reports the round trip time between an agent and the relay it connects to. `network.tunnels` is reported on the network alone, so it is read with `scopeKind=VPCNetwork`; each of the others is readable with `scopeKind=VPCNetwork` or `scopeKind=Tunnel`.

List the catalog with the resolved source and unit for each recipe:

```bash title="terminal"
kubectl get metrics
```

You should see `NAME`, `TYPE`, `UNIT`, and `SOURCE` columns. A `GET` on one name returns the full recipe, including the measures and groupable keys the server derived from it.

<Callout label="Two catalogs">
CLRK has its own fleet metrics catalog under `metrics.clrk.apoxy.dev`, with different recipe names and its own scope kinds. It is a separate product surface. See [Query fleet metrics](/docs/clrk/guides/query-fleet-metrics.md) for that one.
</Callout>

## Proxy health in the metrics API

The `envoy.*` recipes measure the proxy process rather than the traffic through it, so a series query and a snapshot answer for its health the way they do for traffic. They are in the catalog of every project on the managed platform. On a gateway you run yourself they come with the `apoxy-gateway-ee` chart, which installs the metrics pipeline and stores measurements in ClickHouse inside your cluster. Apoxy gives you a pull key. It reads the chart and its images from `oci://us-west1-docker.pkg.dev/apoxy-internal/ee`.

Two values decide the install: `projectID` names the project the measurements belong to, and the ClickHouse choice is either the single node the chart brings up or an external DSN pointing at one you already run. The images run on `amd64` and `arm64` nodes. The single node keeps its own query and metric logs for `clickhouse.systemLogs.retentionDays` days, 7 by default, and its trace and text logs are off.

| Recipe | Measures | Reports |
|---|---|---|
| `envoy.exits` | `exits`, `total` | Envoy process exits, by reason and exit status: `exits` counts the exits inside the window, `total` is the count since the replica started. |
| `envoy.connections` | `active`, `peak`, `limit` | Downstream connections open, their peak, and the ceiling they are measured against. |
| `envoy.pending` | `active`, `peak`, `overflow` | Requests waiting for an upstream connection, their peak, and the ones a full queue rejected. |
| `envoy.latency` | `p50_ms`, `p99_ms`, `upstream_p50_ms`, `upstream_p99_ms` | Request latency through the proxy and the share the upstream owns, p50 and p99. Only the gateway listeners and the route clusters count, so the admin listener and the proxy's own internal clusters do not move the numbers. |
| `envoy.bandwidth` | `downstream_in`, `downstream_out`, `upstream_out`, `upstream_in` | Bytes on the wire in the window, in `By`. The downstream pair is the client side of the gateway listeners, so a Gateway scope narrows it to that Gateway. The upstream pair is the route clusters, which only a Proxy or a replica names, so a Gateway scope reports zero for it. The admin listener and the proxy's own internal clusters do not move the numbers. |
| `envoy.watchdog` | `misses`, `mega_misses` | Threads that missed the watchdog deadline. A mega miss is the long one and says the process was blocked. |
| `envoy.overload` | `heap_pressure`, `cx_pressure`, `stop_accepting` | Heap and connection pressure, and whether the proxy stopped accepting requests. |
| `envoy.dns_cache` | `hosts`, `limit`, `overflow` | Hosts in a dynamic proxy DNS cache, its limit, and the hosts it could not hold. |
| `envoy.fds` | `open`, `limit` | Open file descriptors and the soft limit. |

All nine read `envoy_1m` and group by replica by default, so a single replica in trouble stays visible in a fleet reading. `envoy.dns_cache` groups by cache instead, because a project can run several.

`envoy.bandwidth` counts connection bytes rather than request bodies, so a long-lived stream shows up while it runs instead of only when it closes. Read it with `scopeKind=Gateway` for the client side and with `scopeKind=Proxy` for the upstream side.

`ProxyMetrics` reports the same recipes per replica in `perReplica`, next to the `replicas` counts it takes from the Proxy status. Each entry names one replica and carries that replica's measures alone. A replica that reported nothing in the window is absent from the list rather than present with zeros, and the list is empty until measurements arrive.

### In the console

The console reads these recipes on the **Proxy** tab of a gateway, next to Telemetry. The tab shows how many replicas are connected, the Envoy exits in the selected window, the p99 request latency through the proxy, the active downstream connections against their summed ceiling, the open file descriptors against their summed limit, and the highest heap use of a replica. Under that, one row per replica gives its status, which reads Gone for a replica the Proxy no longer lists, how long it has been connected, its exits in the selected window, its queue overflows, its file descriptors, its DNS cache and its watchdog misses. Telemetry and Proxy share one window, so the range you pick on one tab opens on the other.

The Telemetry tab of the same gateway draws a Bandwidth chart from `envoy.bandwidth`: the client side for the gateway, the upstream side for its proxy.

Four charts draw the window: active downstream connections against the ceiling they are measured against, request latency with the downstream and the upstream distributions drawn apart, heap pressure as a percent, and pending upstream requests. Every Envoy exit is marked on all four, so a jump in one number lines up with the restart behind it. A filter narrows the charts to the replicas you pick, matching on name, locality, address or status, with presets for the replicas that need attention, the ones that restarted Envoy in the window, and each locality. An empty selection means every replica, and the table always lists them all.

When the numbers are missing or old, the tab says why:

- **Proxy health is not available** - this apiserver does not serve the metrics API, so there is no per-replica health to read. Read the replicas from the Proxy object instead.
- **No Proxy attached** - the gateway names no Proxy, so there is no data plane to report on.
- **Shared data plane** - the gateway runs on the shared data plane. Per-replica health is reported for dedicated Proxies and for gateways you run yourself.
- **No replicas reported** - no replica is connected, or no replica reported a measurement in the selected window.
- **Metrics backend not configured** - the API is served but no backend is behind it, and the card names what the install is missing. Replica status keeps coming from the Proxy object.
- **Could not read** - one of the reads the page needs came back with an error, or the network did. Nothing is drawn under the message, because a failed read says nothing about the replicas. Select Retry to read again.
- **Stale data** - the newest measurement is old, so the numbers on the page have stopped moving. Select Retry to read again.

## Metric sources

A source is what a recipe reads. `kubectl get metricsources` lists them with their granularity and retention.

| Source | Granularity | Retention | Notes |
|---|---|---|---|
| `http_1m` | 1m | 7d | HTTP traffic aggregated into one-minute buckets. |
| `http_1h` | 1h | 400d | The same measurements in one-hour buckets. |
| `otel_logs` | row | 90d | Individual log records. |
| `envoy_1m` | 1m | 30d | Proxy connection and TLS stats in one-minute buckets. |
| `relay_1m` | 1m | 30d | Tunnel traffic in one-minute buckets, per network, per relay, and per tunnel. |

The `http.*` recipes read `http_1m`. Series and snapshot reads over longer windows are served at coarser granularity automatically, so history stays available past the finer source's retention without any change to your request.

A snapshot evaluates recipes from several sources, and the sources keep different amounts of history. A window wider than a source's retention drops that source's recipes from the snapshot and reports the rest: `vpcnetworks/{name}/metrics?window=1440h` answers with the `http.*` recipes, which the 400 day rollup serves, and without the `network.*` and `tunnel.*` recipes, which are kept for 30 days. A window no evaluated source can reach is a `400` instead - `tunnels/{name}/metrics` reads `relay_1m` alone, so a 60 day window there names the source and its maximum. Naming such a recipe explicitly with `metric=` is a `400` as well, rather than a quiet drop.

`log.severity` reads raw log records, which is why it is **series-only**: snapshots evaluate the aggregated recipes and skip raw-log ones. A grouped read over raw records is limited to a 24 hour window.

`MetricSource.status.fields` lists each field with a `role`:

- `time` - the bucket column.
- `key` - a groupable dimension. These are exactly the values `groupBy` accepts.
- `measure` - a value column a recipe aggregates.

Each field also reports `discovered`, which is `true` for a field found by sampling log attributes (so it can disappear) and `false` for a fixed column.

## Series

<APIEndpoint method="GET" path="/apis/metrics.apoxy.dev/v1alpha1/metrics/{name}/series" />

Runs one recipe over a window and returns a `MetricSeriesSet`.

### Parameters

| Parameter | Default | Description |
|---|---|---|
| `scopeKind` | `Project` | Owner kind to scope to: `Project`, `Gateway`, `HTTPRoute`, `Proxy`, `Service`, `VPCNetwork`, or `Tunnel`. Takes a kind, not a resource name (`Gateway`, not `gateways`). |
| `scopeName` | - | Name of the owner object. Omit it for `Project`, which reads the whole project. |
| `scopeListener` | - | Narrows a `Gateway` scope to one listener. |
| `since` | `-1h` | Start of the window. An RFC3339 instant or a relative duration. |
| `until` | end of the last complete bucket | End of the window. Same formats as `since`. |
| `window` | - | Shorthand for `since=-<duration>`. Rejected together with `since`. |
| `step` | source granularity | Bucket width. Rounded **up** to the source granularity and echoed in the response. |
| `groupBy` | - | One key field to split the result by. |
| `orderBy` | the recipe's first default column | Measure to rank groups by. |
| `top` | `50` | Maximum number of series to return. `50` is also the ceiling. |

The window is half-open, `[since, until)`. Relative durations accept `h`, `m`, `s`, and `d`: `-6h`, `-7d`, `-1.5d`, and `+30m` all parse. A bare `6h` with no sign does not, and neither does `-1w`.

Buckets count whole steps forward from `since` rather than from the clock, so the first point carries `since`, and a `step` equal to the window returns exactly one point per series.

### Group-by keys

`groupBy` takes exactly one key:

| Key | Description |
|---|---|
| `gateway` | Gateway that served the request. |
| `listener` | Gateway listener that served the request. |
| `route_kind` | Route object kind. |
| `route_name` | Route object name. |
| `route_rule` | Rule index inside the route. |
| `backend_kind` | Backend object kind. |
| `backend_name` | Backend object name. |
| `backend_revision` | Compute revision that served the request. Empty for traffic that is not served by a revision. |
| `status_class` | Response status class (`2`, `4`, `5`). |
| `method` | HTTP request method. |

The `network.*` and `tunnel.*` recipes read a different source and take their own keys:

| Key | Description |
|---|---|
| `network` | VPC network the traffic crossed. |
| `tunnel` | Tunnel that carried it. One tunnel is one agent connection. |
| `relay` | Relay that carried it. A network is served by more than one relay. Empty on a metric that is not counted per relay. |
| `direction` | `rx` or `tx`. Empty on a metric that counts no traffic. |

A key that the recipe's source does not carry returns `400` with the valid keys listed. The authoritative list for any recipe is its `status.keys`.

### Response

```json
{
  "kind": "MetricSeriesSet",
  "apiVersion": "metrics.apoxy.dev/v1alpha1",
  "metric": "http.requests",
  "scopeKind": "Gateway",
  "scopeName": "prod",
  "since": "2026-08-20T15:00:00Z",
  "until": "2026-08-20T21:00:00Z",
  "step": "5m0s",
  "dataUpTo": "2026-08-20T21:00:00Z",
  "truncated": false,
  "totalCount": 7,
  "units": {},
  "series": [
    {
      "labels": { "route_name": "api" },
      "points": [
        { "timestamp": "2026-08-20T15:00:00Z", "values": { "total": 5210, "status_4xx": 140, "status_5xx": 12 } },
        { "timestamp": "2026-08-20T15:05:00Z", "values": { "total": 5188, "status_4xx": 131, "status_5xx": 9 } }
      ]
    }
  ]
}
```

| Field | Description |
|---|---|
| `metric`, `scopeKind`, `scopeName` | Echo the resolved query. `scopeName` is empty for a project-wide read. |
| `since`, `until` | Resolved window bounds. |
| `step` | Applied bucket width, serialized as a duration string (`"5m0s"`). |
| `dataUpTo` | End of the last complete bucket. |
| `truncated` | `true` when more groups matched than `top` returned. |
| `totalCount` | How many groups had data in the window. |
| `units` | Measure name to display unit, echoed from the catalog. Empty when no measure carries a unit. |
| `series[].labels` | Group key and value. Empty for an ungrouped read, which returns exactly one series. |
| `series[].points[]` | Buckets in timestamp order, each with the recipe's measures. |

Points are **sparse**: a bucket with no traffic is omitted rather than returned as zero. Compare a missing bucket against `dataUpTo` to tell missing data from a partial trailing bucket.

### Fleet views

A one-bucket series is how you get one row per object. Set `step` equal to the window and group by the owner key:

```
metrics/http.requests/series?scopeKind=Project&groupBy=gateway&since=-1h&step=1h
```

That returns one point per Gateway, ranked by `orderBy` and bounded by `top`. The same shape with `scopeKind=Gateway&groupBy=route_name` gives one row per route.

## Snapshots

A snapshot is every applicable recipe evaluated over one window for one object, in a single call. It mounts as a `metrics` subresource on the owner.

| Owner | Path | Response kind | `include` tokens |
|---|---|---|---|
| Gateway | `/apis/gateway.apoxy.dev/v1/gateways/{name}/metrics` | `GatewayMetrics` | `routes` |
| HTTPRoute | `/apis/gateway.apoxy.dev/v1/httproutes/{name}/metrics` | `HTTPRouteMetrics` | `rules`, `backends` |
| Proxy | `/apis/core.apoxy.dev/v1alpha2/proxies/{name}/metrics` | `ProxyMetrics` | none |
| Service | `/apis/compute.apoxy.dev/v1alpha1/services/{name}/metrics` | `ServiceMetrics` | `revisions` |
| VPCNetwork | `/apis/vpc.apoxy.dev/v1alpha1/vpcnetworks/{name}/metrics` | `VPCNetworkMetrics` | `services`, `tunnels` |
| Tunnel | `/apis/vpc.apoxy.dev/v1alpha1/tunnels/{name}/metrics` | `TunnelMetrics` | none |

Every snapshot carries `timestamp`, `window`, `since`, `until`, `dataUpTo`, a `metrics` map keyed by recipe name, and a `units` map. The window parameters are the same as for series.

```json
{
  "kind": "GatewayMetrics",
  "apiVersion": "metrics.apoxy.dev/v1alpha1",
  "metadata": { "name": "prod" },
  "timestamp": "2026-08-20T21:00:00Z",
  "window": "24h0m0s",
  "since": "2026-08-19T21:00:00Z",
  "until": "2026-08-20T21:00:00Z",
  "dataUpTo": "2026-08-20T21:00:00Z",
  "metrics": {
    "http.requests": { "total": 1842113, "status_2xx": 1790220, "status_4xx": 48120, "status_5xx": 3773 },
    "http.latency": { "p50": 38, "p95": 190, "p99": 412 }
  },
  "units": { "p50": "ms", "p95": "ms", "p99": "ms" },
  "listeners": [
    {
      "name": "https",
      "metrics": { "http.requests": { "total": 1839900, "status_5xx": 3773 } },
      "truncated": true,
      "totalCount": 37
    }
  ]
}
```

### Nesting

The default response is the owner totals plus the first nesting level, with no leaf rows. Add them with `include`:

- `include=routes` on a Gateway adds `routes[]` under each listener.
- `include=rules`, `include=backends` on an HTTPRoute add `rules[]` and `backends[]`.
- `include=revisions` on a Service.
- `include=services`, `include=tunnels` on a VPC network.
- `include=all` means every token for that kind.

`include` may repeat or take a comma-separated list. An unknown token returns `400`.

`truncated` and `totalCount` sit on whichever container was cut. A Gateway cuts routes per listener, so they appear on each listener. `HTTPRouteMetrics`, `ServiceMetrics`, and `VPCNetworkMetrics` cut their own leaf lists, so they appear on the object itself and count every list the read asked for.

A VPC network reports its services and its tunnels from different measurements: `services[]` covers gateway traffic to the network's VPC services, and `tunnels[]` covers everything the tunnels carried. The two do not add up to each other. `network.tunnels` counts the network rather than one connection, so it is reported in the network's own `metrics` map and never as an entry of `tunnels[]`; every entry of `tunnels[]` names a connection, and `totalCount` counts those entries alone.

For complete per-route coverage instead of the top slice, use a one-bucket series grouped by `route_name`.

`orderBy` names the measure a nested list is ranked by. A snapshot can nest by levels that come from different sources, and a level is ranked by the named measure only when a recipe of that level reports it; a level whose recipes do not report it keeps its own default order, which is the request count where there is one. `include=all&orderBy=bytes` on a VPC network therefore ranks `tunnels[]` by bytes and leaves `services[]` in its own order. A measure no evaluated recipe reports anywhere is a `400`.

### Restricting recipes

`metric=<name>` limits which recipes are evaluated. The parameter repeats and does **not** take a comma-separated list:

```
?metric=http.requests&metric=http.latency
```

Snapshots evaluate the managed recipes by default. Naming a recipe of your own with `metric=` evaluates that one too.

`ProxyMetrics` additionally carries `replicas`, which reports `connected` replica counts from the Proxy's own status rather than from traffic. Proxy metrics cover traffic on the Gateways attached to that Proxy.

`ServiceMetrics` revision rows appear as traffic arrives for each revision. Traffic recorded before revision attribution existed is grouped under an empty revision name.

`TunnelMetrics` covers one agent connection. A tunnel's history belongs to that connection: when an agent reconnects it gets a new `Tunnel` with a new name, and its metrics start over with it. Read `vpcnetworks/{name}/metrics?include=tunnels` for a view that spans reconnects, and `tunnels/{name}/metrics` for one connection.

The `Tunnel` object is removed when the agent disconnects, but its measurements are kept for the retention of `relay_1m`. `tunnels/{name}/metrics` keeps answering for a disconnected connection for that long, so the history of a connection outlives the connection itself. A name with no measurements left, and one that never existed, are both a `404`.

## Custom metrics

A `Metric` you create is a saved query. It appears in `kubectl get metrics`, works with `series`, and can be pulled into a snapshot with `metric=`.

```yaml title="metric.yaml"
apiVersion: metrics.apoxy.dev/v1alpha1
kind: Metric
metadata:
  name: v2.errors
spec:
  source: otel_logs
  type: counter
  description: Server errors on the /v2 path prefix.
  prql: |
    filter (url.path | text.starts_with "/v2/")
    filter http.response.status_code >= 500
    aggregate { n = count this }
```

Apply it with `apoxy apply -f metric.yaml`.

The fragment is **aggregate-only**: zero or more `filter` and `derive` steps followed by one `aggregate`. It states no `from`, no `group`, and no `time_bucket`, because the source comes from `spec.source`, the grouping from `groupBy`, the bucket from `step`, and the scope from the scope parameters. A fragment carrying any of those steps is rejected.

The server compiles the fragment when you write it and fills in `status`: the resolved `source`, the output `measures` with their types and units, the groupable `keys`, and a `Compiled` condition. You never write `status` yourself. A fragment that does not compile is never stored, so a broken recipe cannot surface later as a snapshot silently missing a measure.

Leave `spec.source` unset to have the server resolve it from the fields the fragment uses.

<Callout label="Reserved names" variant="warn">
The prefixes `http.`, `log.`, `upstream.`, `tls.`, `network.`, and `tunnel.` are reserved for built-in recipes, as is the exact name `series`. A write to a reserved name is rejected with `403`, so a recipe of yours can never shadow a built-in. The `envoy_1m` and `relay_1m` sources are reserved the same way: their schema moves with the built-in recipes, so a recipe of yours cannot read them.
</Callout>

## Limits

| Limit | Value |
|---|---|
| Series per response (`top`) | 50 |
| Buckets per series | 1500 |
| Total points (buckets x series x measures) | 20000 |
| Minimum `step` | the source granularity, rounded up |
| Maximum lookback on raw records | 31d |
| Maximum window for a grouped raw-record read | 24h |

Exceeding the point budget returns `400` naming all three factors, so you can lower `top`, widen `step`, or ask for fewer measures with `metric=`. Exceeding `top` is not an error: the response returns the top slice and sets `truncated` with `totalCount`.

## Errors

Failures are standard Kubernetes `Status` objects.

| Status | Meaning |
|---|---|
| `400` | Guardrail violation, or an unknown `groupBy` key, `include` token, or `scopeKind`. The message lists the valid values. |
| `403` | Write to a reserved recipe name. |
| `404` | The owner object does not exist. |
| `422` | The recipe does not compile against the current schema. |
| `429` | Concurrent query limit reached. Retry after the interval in the `Retry-After` header. |
| `503` | The metrics backend is unavailable. |

A project with no data yet returns `200` with an empty window rather than an error, so an empty result is not a failure.

## Caching and polling

Responses carry:

```
Cache-Control: max-age=60
```

Poll on the interval the response states rather than faster. Window bounds are aligned to `step` before a response is cached, so two reads a few seconds apart over the same window return the same result.

Treat `dataUpTo` as the edge of settled data. A bucket after it is still filling, and a client that charts it will show an apparent dip that recovers on the next poll.

## Gateway pod metrics (self-hosted)

A gateway you run in your own cluster exposes Prometheus metrics on the gateway pod. They describe the Envoy process rather than the traffic through it, so they answer what the API above cannot: whether Envoy restarted, how close it is to a limit, and where a request spent its time. The series carry the `apoxy_backplane_` prefix, after the process that supervises Envoy in the pod.

### Endpoint

| | |
|---|---|
| Container port | `8888` (named `metrics`) |
| Protocol | HTTP, cluster-internal |
| `/envoy/metrics` | Envoy's own stats and the `apoxy_backplane_*` series in one scrape. |
| `/metrics` | The `apoxy_backplane_*` series alone. |

`/envoy/metrics` answers `200` while Envoy is restarting. Envoy's own stats are missing from that response and `apoxy_backplane_envoy_up` reads `0`, so alert on that gauge rather than on a failed scrape.

Chart values:

- `backplane.envoyPodMonitor` - creates the `PodMonitor` for `/envoy/metrics`.
- `backplane.prometheusRule` - creates the `PrometheusRule` holding the alerts below.
- `backplane.grafanaDashboard` - creates the dashboard ConfigMap.
- `backplane.autoscaling.podsMetrics` - adds `type: Pods` metrics to the gateway HPA, so you can scale on a series such as `envoy_http_downstream_cx_active`. Your cluster needs prometheus-adapter to publish the named series as a pods metric.

### Envoy restarts

| Metric | Type | Description |
|---|---|---|
| `apoxy_backplane_envoy_up` | Gauge | `1` while the Envoy process runs, `0` between an exit and the next start. |
| `apoxy_backplane_envoy_info` | Gauge (`1`) | Build info, carried on the `release` and `envoy_version` labels. |
| `apoxy_backplane_envoy_start_time_seconds` | Gauge | Unix seconds of the current start. |
| `apoxy_backplane_envoy_restarts_total` | Counter | Starts that followed an exit. Resets to `0` when the pod restarts. Alert on this one rather than on `exits_total`, because it is published from `0`. |
| `apoxy_backplane_envoy_exits_total` | Counter | Exits by `reason` and `code`. A pair appears only once an exit carries it, so its first value is already `1`. |
| `apoxy_backplane_envoy_last_exit_timestamp_seconds` | Gauge | Unix seconds of the last exit. Absent until the first one. |
| `apoxy_backplane_envoy_last_exit_requests_in_flight` | Gauge | Requests Envoy was serving at the last exit. None of them reach an access log. |
| `apoxy_backplane_envoy_last_exit_connections` | Gauge | Downstream connections open at the last exit. |
| `apoxy_backplane_envoy_last_exit_connections_aborted` | Gauge | Established connections reset at the last exit. |
| `apoxy_backplane_envoy_last_exit_connections_refused` | Gauge | Connection attempts answered with a reset while Envoy was down. |
| `apoxy_backplane_envoy_admin_scrape_errors_total` | Counter | Scrapes that could not read Envoy's own stats. A rising count while `apoxy_backplane_envoy_up` reads `1` means Envoy is alive but not answering. |

`reason` takes four values, and `code` qualifies it:

- `exit` - the process returned an exit status, which `code` carries. `exit` with `code="0"` is a clean shutdown, and `code="1"` is usually a configuration Envoy refused at start.
- `signal` - a signal ended the process, and `code` carries the signal name.
- `oom_kill` - the container reached its memory limit and the kernel killed the process.
- `start_failed` - the process never started, and `code` is empty.

The same exit appears in two other places. `Proxy.status.replicas[]` carries `envoyRestarts` and `lastEnvoyExit` for each connected replica:

```bash title="terminal"
apoxy proxy get my-proxy -o yaml
```

And the gateway logs one structured line per exit, `Envoy exited`, with the reason, the exit status or signal name, and how long the process ran.

### Limits and the series that show pressure

Every limit below has a ceiling you can read, a current value, and usually a pressure signal that moves before the limit bites.

| Limit | Ceiling | Current use | Pressure |
|---|---|---|---|
| Active downstream connections | `apoxy_backplane_envoy_limit_max_active_downstream_connections` | `envoy_server_total_connections` | `envoy_overload_envoy_resource_monitors_global_downstream_max_connections_pressure` |
| Envoy heap | `apoxy_backplane_envoy_limit_max_heap_bytes` | `envoy_server_memory_allocated` | `envoy_overload_envoy_resource_monitors_fixed_heap_pressure` |
| Container memory | `apoxy_backplane_envoy_limit_memory_bytes` | `apoxy_backplane_envoy_rss_bytes` | - |
| File descriptors | `apoxy_backplane_envoy_max_fds` | `apoxy_backplane_envoy_open_fds` | - |
| Pending requests per cluster | unlimited unless the Backend sets one | `envoy_cluster_upstream_rq_pending_active` | `envoy_cluster_circuit_breakers_default_remaining_pending`, `envoy_cluster_upstream_rq_pending_overflow` |
| Circuit breakers per cluster | set per Backend | `envoy_cluster_circuit_breakers_default_remaining_cx`, `_remaining_rq`, `_remaining_pending` hold the headroom left | `envoy_cluster_circuit_breakers_default_cx_open`, `_rq_open`, `_rq_pending_open` |
| Dynamic proxy DNS cache | `apoxy_backplane_envoy_limit_dns_cache_max_hosts`, label `cache` | `envoy_dns_cache_<name>_num_hosts` | `envoy_dns_cache_<name>_host_overflow` |
| Worker thread progress | - | - | `envoy_server_watchdog_miss`, `envoy_server_watchdog_mega_miss` |
| Kernel accept queue | - | - | `apoxy_backplane_net_tcp_listen_overflows_total`, `apoxy_backplane_net_tcp_listen_drops_total` |
| Connections the kernel cut | - | - | `apoxy_backplane_net_tcp_out_rsts_total`, `apoxy_backplane_net_tcp_estab_resets_total`, `apoxy_backplane_net_tcp_abort_on_close_total`, `apoxy_backplane_net_tcp_abort_on_data_total` |

Reading that table:

- The two Envoy pressure gauges are percentages from `0` to `100`, not ratios. Compare against `80`, not `0.8`.
- The heap and container memory rows need a memory limit on the gateway container. Without one there is no ceiling to publish, `apoxy_backplane_envoy_limit_memory_bytes` is absent, and Envoy's heap monitor has no threshold to measure pressure against.
- Heap and resident memory are different numbers, and neither covers the other. `envoy_server_memory_allocated` counts Envoy's own allocator and misses every Edge Function library loaded into the same process, so resident memory can climb while heap pressure stays flat. `apoxy_backplane_envoy_rss_bytes` is the Envoy process alone, while `apoxy_backplane_envoy_limit_memory_bytes` is the container limit for the whole pod, so the headroom that ratio shows is optimistic. The container limit is the one the kubelet kills against.
- Pending requests are unbounded unless a Backend sets a limit, so a slow upstream grows the queue instead of overflowing it. Watch the level with `envoy_cluster_upstream_rq_pending_active`, which is what `EnvoyPendingQueueHigh` alerts on; `upstream_rq_pending_overflow` and `rq_pending_open` stay at zero until a Backend sets a finite limit.
- `<name>` in the DNS cache series is the Backend the dynamic proxy serves, so a Backend named `dynamic-proxy` gives `envoy_dns_cache_dynamic_proxy_num_hosts`.
- The `apoxy_backplane_net_tcp_*` counters cover connections the kernel dropped or reset before Envoy saw them, which is what a client experiences during a restart or a worker stall. No Envoy stat records those.
- The watchdog series are the totals over all threads. A per-thread breakdown is published beside them, either on an `envoy_thread_name` label or in the metric name.

### Time inside the proxy

Three histograms split where a request spends its time:

- `envoy_http_downstream_rq_time` - the whole request, from the first downstream byte to the last response byte.
- `envoy_cluster_upstream_rq_time` - the part your upstream owns.
- `envoy_cluster_upstream_cx_connect_ms` - connection setup to the upstream, which `upstream_rq_time` does not include.

The difference between the first two is the proxy's own share: filters, Edge Functions, and time waiting in the pending queue. Three access-log fields split that share per request, so a slow tail can be attributed to a single call rather than to a percentile:

- `filter_duration` - from the first downstream byte to the first byte sent upstream.
- `request_tx_duration` - from the first downstream byte to the last byte sent upstream.
- `response_duration` - from the first downstream byte to the first upstream response byte.

In the hosted logs the same values are the attributes `http.request.filter_duration_ms`, `http.request.tx_duration_ms` and `http.response.duration_ms`, next to `http.request.duration_ms`.

### Recommended alerts

The chart's `PrometheusRule` ships these. Tune the thresholds to your traffic.

| Alert | Fires when |
|---|---|
| `EnvoyExited` | Envoy exited at least once in the last 10 minutes. |
| `EnvoyDown` | `apoxy_backplane_envoy_up` has read `0` for 2 minutes. |
| `EnvoyCrashLoop` | Envoy restarted three times in 30 minutes. |
| `EnvoyPendingQueueHigh` | More than 512 requests wait for a connection to one cluster. This is the one that fires on a default configuration. |
| `EnvoyPendingOverflow` | A cluster rejects requests with a 503 because its pending queue is full. Needs a Backend that sets a pending limit. |
| `EnvoyCircuitBreakerOpen` | A default-priority circuit breaker has been open for 2 minutes. |
| `EnvoyWatchdogMegaMiss` | A worker thread stalled past the watchdog mega-miss timeout. |
| `EnvoyOverloadStopAcceptingRequests` | The overload manager is refusing new requests. |
| `EnvoyDownstreamCxNearLimit` | Downstream connections are above 80% of the configured ceiling. |
| `EnvoyFdNearLimit` | Open file descriptors are above 80% of the soft limit. |
| `EnvoyRssNearLimit` | Resident memory is above 85% of the container memory limit. |
| `DfpDnsCacheNearLimit` | The dynamic proxy DNS cache is near `max_hosts`, or it is evicting hosts. |
| `EnvoyConfigRejected` | Envoy rejected a cluster, listener, or route update and keeps serving the previous one. |
| `EnvoyReloadStorm` | Listeners reload repeatedly while older ones are still draining. |
| `EnvoyListenOverflow` | The kernel drops new connections because the accept queue is full. |

`EnvoyConfigRejected` is the one with no other symptom. Envoy keeps the last good configuration when it rejects an update, so the pod stays healthy and serves stale routes until someone notices.

The same health signals are readable through the metrics API as the `envoy.*` recipes. See [Proxy health in the metrics API](#proxy-health-in-the-metrics-api).

### Scraping the pipeline itself

The `apoxy-gateway-ee` chart also watches the pipeline that stores those measurements. With `monitoring.enabled` (the default) it creates three `ServiceMonitor` objects, `<release>-apoxy-ee-apiserver`, `<release>-apoxy-ee-collector` and `<release>-apoxy-ee-clickhouse`, and a `PrometheusRule` named `<release>-apoxy-ee-pipeline` holding the `apoxy-gateway-ee.pipeline` alert group. Its alerts fire when a rollup falls behind, when the collector refuses or drops datapoints, and when ClickHouse is down or low on disk. All four render only on a cluster that serves `monitoring.coreos.com/v1`, so an install without the Prometheus operator skips them.

## Where to next

- Work through the calls end to end in [Query gateway metrics](/docs/guides/query-gateway-metrics.md).
- Ask free-form questions over raw log records with [PRQL through the MCP server](/docs/reference/mcp.md).
- Diagnose a restart with [Envoy restarted inside the gateway](/docs/guides/troubleshooting.md#envoy-restarted-inside-the-gateway).

---

**Navigation** (Reference)

- Previous: [MCP server](/docs/reference/mcp.md)
- Next: [HTTP APIs](/docs/reference/http-apis.md)
- All pages: [index](/docs/llms.txt)