Skip to content

feat(accesslogs): fetch access logs of add-ons - #1127

Open
miton18 wants to merge 4 commits into
masterfrom
feat/accesslogs-addons
Open

miton18 wants to merge 4 commits into
masterfrom
feat/accesslogs-addons

Conversation

@miton18

@miton18 miton18 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Closes #1126

What

clever accesslogs --addon <id> now works. The option was already declared and documented, but the handler threw Access Logs are not available for add-ons yet behind a // TODO: drop when add-ons are supported in API.

Why the TODO was stale

The v4 endpoint reads /v4/accesslogs/organisations/{ownerId}/applications/{id}/accesslogs, and that applications segment is a misnomer. Server side it is parsed as a LoggableId, a union accepting every add-on prefix (redis_, postgresql_, cellar_, pulsar_, keycloak_, otoroshi_…), and the handler then reads the Pulsar topic named after that ID. Nothing along that path is application specific — the API has been able to serve add-on access logs all along.

The real ID is what matters: it is the topic name. An addon_ ID matches no topic and would silently stream nothing, so add-ons go through resolveAddon first, exactly like clever logs --addon already does. Both ID forms work for the user.

TCP access logs were being dropped

Add-ons exposed over raw TCP emit access logs with no http section (so do TCP redirections on applications, and connections cut before the request line is parsed). Both the human and clf formatters skipped any log where log.http == null — meaning that without this second fix, the command would have connected, received events, and printed nothing.

The human format now renders them, keeping the exact same columns as HTTP lines so a mixed stream stays aligned: TCP where the status code sits, connection details where the request line sits.

2026-08-13T05:31:10.793Z  91.231.89.104    FR/Gravelines            TCP  :53171  0B↑ 0B↓
2026-08-13T05:31:20.979Z  91.231.89.107    FR/Gravelines            TCP  :53749  0B↑ 0B↓
2026-08-13T11:50:09.802Z  185.177.72.22    (unknown)                TCP  :49700  0B↑ 0B↓
2026-08-13T05:31:44.201Z  54.36.148.12     FR/Roubaix               200  GET  /api/v1/items?page=2

The destination is dropped, it is the load balancer rather than the resource. The target VM is printed only when the API resolved it. clf keeps skipping these on purpose: it describes an HTTP request, and a line with neither request line nor status is not parsable by grok or GoAccess.

Two platform-side gaps, not addressed here

Observed in production across four add-ons: bytesIn/bytesOut are always 0, and instanceId is the nil UUID on Redis/PostgreSQL — but a real VM UUID on Elasticsearch. The CLI renders both correctly whenever the platform provides them; nothing more can be done client side.

  • Target VM. The value does reach the API. It carries HAProxy's server_name, which is a bare UUID for some providers (c958d0a5-…) and <real_id>_<index> for others (postgresql_bde3942f-…_0) — see the access-logs-forwarder parser fixtures. The API parses that string as a UUID-typed InstanceID and silently substitutes the nil UUID when it fails, so the information is dropped for a whole class of providers. Fix belongs in the API's AccessLogView (or in a uniform load balancer server naming).
  • Byte counters. Every other field of the record survives, so the loss is confined to the metrics envelope rather than the record. The v2→v3 conversion in the common SDK only builds the TCP metrics envelope when protocol equals the literal string "TCP", and the v3→customer-avro conversion then defaults the counters to 0 when that envelope is absent. Needs confirmation by reading the raw customer topic (the source and proxy_hostname fields, which the API drops, identify the emitter).

Also for the record: the API hardcodes region/zone to "" and tls to null in its mapping — dead fields in the payload.

Also

Drops the /!\ This feature is in Beta testing phase banner: the command shipped in 2.1.0 and the endpoint is stable.

Testing

npm run validate passes (lint, prettier, tsc, docs). Manually verified against production on Redis, PostgreSQL and Elasticsearch add-ons, with both addon_ and real ID forms, across human, json, json-stream and clf formats.

`--addon` was declared and documented on the command, but the handler threw
`Access Logs are not available for add-ons yet` right away.

The API has been able to serve them all along: the v4 endpoint reads
`/v4/accesslogs/organisations/{ownerId}/applications/{id}/accesslogs`, but its
`applications` path segment is a misnomer. Server side that segment is parsed as
a `LoggableId`, a union accepting every add-on prefix (`redis_`, `postgresql_`,
`cellar_`, `pulsar_`…), and the handler then just reads the Pulsar topic named
after it. Nothing along that path is application specific.

Add-ons are therefore resolved with `resolveAddon` and their real ID is passed
as the stream resource. The real ID matters: it is the topic name, an `addon_`
ID matches no topic at all and would silently stream nothing.

Verified against production on a Redis add-on.
Access logs carry no `http` section when the connection is cut before the
request line is parsed, when it goes through a TCP redirection, and for every
add-on exposed over raw TCP (Redis, PostgreSQL, MongoDB…). Both human and CLF
formatters silently skipped those, so `clever accesslogs` on such an add-on
printed nothing at all while the stream was in fact delivering events.

The human format now renders them on their own line, keeping the exact same
columns as HTTP lines so a mixed stream stays aligned: `TCP` where the status
code sits, connection details where the request line sits.

    2026-08-13T05:31:10.793Z  91.231.89.104  FR/Gravelines  TCP  :53171  0B↑ 0B↓

The destination is dropped, it is the load balancer rather than the resource.
The target VM is printed only when the platform actually filled it, add-ons
leave it at the nil UUID today.

CLF keeps skipping them on purpose: it describes an HTTP request, a line with
neither request line nor status is not parsable by grok or GoAccess.
The command shipped in 2.1.0 and the endpoint it consumes is stable, the banner
printed on every human run is just noise now.
@miton18
miton18 requested a review from a team as a code owner August 20, 2026 10:15
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

🔎 A preview has been automatically published!

If you created the alias to the preview script, you can run this command to download and install this preview:

clever-preview update feat-accesslogs-addons

You can also run it from your local repository:

./scripts/preview.js update feat-accesslogs-addons
OS SHA256 checksum
🐧 linux b3331a7d6866e1771e7c38ace570e721fcde27a972e6b52b0c0777b8c10b2a9f
🍏 macos bd17385f52c7f2fd0168b66a74b25d8067309a313d28e212182f79e13503e465

This preview will be deleted once this PR is closed.

@miton18 miton18 self-assigned this Aug 20, 2026
@hsablonniere hsablonniere modified the milestone: 5.0.0 Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fetch access logs of add-ons with clever accesslogs --addon

2 participants