Overview
Add a top-level datumctl logs command so users can read project-scoped telemetry logs from the CLI.
This is the datumctl-side half of the telemetry read path. The server-side half is milo-os/telemetry#77 (Query API), which explicitly scopes itself to "a service allowing project-scoped log and metrics query for use by datumctl and the cloud portal" — this issue tracks the client that calls it.
What we're doing
datumctl logs '{service_name="checkout"} |= "error"' --since 1h
- Query the queryapi aggregated APIService at
o11y.miloapis.com/v1alpha1/logs, hung off the active project's control-plane host, matching the server URL declared in queryapi's openapi.yaml.
- Accept the LogQL subset queryapi accepts — label matchers and line filters.
- Rely on server-side project scoping. The client never supplies a tenant identifier; there is no
X-Scope-OrgID-style header on this path by design.
- Window selection via
--since, or --start/--end, with --limit and --direction.
- Render returned streams with deterministic label ordering and a single global time ordering across streams.
What we're not doing
- Live tail. queryapi does not implement
/loki/api/v1/tail yet, and the transport (chunked HTTP vs. WebSocket) is still under evaluation upstream. Tracked as a follow-up rather than blocking.
- Label/series discovery subcommands. queryapi serves
/labels, /label/{name}/values and /series, but this issue does not wrap them.
- Metrics. queryapi's Prometheus-shaped endpoints return 501 today.
- Raw query passthrough. The accepted syntax is whatever queryapi accepts; datumctl adds no query language of its own.
What done looks like
datumctl logs QUERY reaches queryapi through the kube-aggregator proxy chain with no client-side routing logic beyond the path prefix.- The group, version and
logs signal segment match the APIService registration, since queryapi derives both its served paths and the permission it reviews (logs.query) from the same constants.
- Failures are actionable: the response body is surfaced (unwrapping either the Loki envelope or a Kubernetes
Status) so a rejected query, a permission denial and a missing APIService are distinguishable.
- A query argument is required, since queryapi rejects a selector with no label matchers.
- Output is deterministic run to run, and ordered by time across all returned streams rather than per stream.
go build, go vet and the existing test suite stay green.
Known blocker, outside this repo
An end-to-end query returns zero rows in staging today. Every row in o11y.logs carries ProjectId = 'internal':
SELECT DISTINCT ProjectId FROM o11y.logs -- → internal (only value)
ProjectId is MATERIALIZED ResourceAttributes['milo.project.id'] and the row policy is ProjectId = getSetting('telemetry_project_id'), so no customer project can match until the edge pipeline stamps that resource attribute with the owning project. Confirmed by generating real traffic through an HTTPProxy in a test project — the Envoy access logs were ingested correctly but tagged internal.
This does not block the CLI work, but it does mean the command cannot be demonstrated against real project data yet.
Overview
Add a top-level
datumctl logscommand so users can read project-scoped telemetry logs from the CLI.This is the datumctl-side half of the telemetry read path. The server-side half is milo-os/telemetry#77 (Query API), which explicitly scopes itself to "a service allowing project-scoped log and metrics query for use by datumctl and the cloud portal" — this issue tracks the client that calls it.
What we're doing
o11y.miloapis.com/v1alpha1/logs, hung off the active project's control-plane host, matching the server URL declared in queryapi'sopenapi.yaml.X-Scope-OrgID-style header on this path by design.--since, or--start/--end, with--limitand--direction.What we're not doing
/loki/api/v1/tailyet, and the transport (chunked HTTP vs. WebSocket) is still under evaluation upstream. Tracked as a follow-up rather than blocking./labels,/label/{name}/valuesand/series, but this issue does not wrap them.What done looks like
datumctl logs QUERYreaches queryapi through the kube-aggregator proxy chain with no client-side routing logic beyond the path prefix.logssignal segment match theAPIServiceregistration, since queryapi derives both its served paths and the permission it reviews (logs.query) from the same constants.Status) so a rejected query, a permission denial and a missing APIService are distinguishable.go build,go vetand the existing test suite stay green.Known blocker, outside this repo
An end-to-end query returns zero rows in staging today. Every row in
o11y.logscarriesProjectId = 'internal':ProjectIdisMATERIALIZED ResourceAttributes['milo.project.id']and the row policy isProjectId = getSetting('telemetry_project_id'), so no customer project can match until the edge pipeline stamps that resource attribute with the owning project. Confirmed by generating real traffic through anHTTPProxyin a test project — the Envoy access logs were ingested correctly but taggedinternal.This does not block the CLI work, but it does mean the command cannot be demonstrated against real project data yet.