[feature]: Make Kubernetes API-server audit logs searchable in Coroot #2318

🤖 Generated by the Daily AI Assistant

Summary

Make Kubernetes API-server audit logs searchable in Coroot, alongside the container logs and traces Coroot already ingests.

This is the one genuine residual of #1507 (the kube-prometheus-stack / Loki / Alloy log-aggregation epic), which is now superseded by #1742 ("replace kube-prometheus-stack with self-hosted Coroot"). Coroot already delivers the bulk of #1507's goals — container-log search across all namespaces, metric/log correlation, and a single UI — so this issue carves out only the part Coroot does not cover today.

Problem

• Since feat: replace kube-prometheus-stack with self-hosted Coroot #1742, Coroot is the single observability stack. Its eBPF node-agent collects per-pod container logs (stdout/stderr) and traces, stored in ClickHouse (logsTTL, see k8s/bases/infrastructure/coroot/coroot.yaml). docs/dr/alerting.md documents Coroot as the replacement for "the log store and the log shipper".
• Kubernetes API-server audit logs are a different stream. Per talos/cluster/audit-logging.yaml (added in feat: add Talos security hardening patches #1506), the API server writes audit events to a host file — /var/log/audit/kube/audit.log on the ephemeral partition of each control-plane node (rotation: 30 days, 3 backups, 100 MB/file).
• Coroot's eBPF agent captures container stdout/stderr and L7 traffic — it does not tail arbitrary host log files, so audit events never reach Coroot. They remain accessible only via talosctl on individual nodes, with no central search, no correlation, and no retention beyond the node's ephemeral disk.

Proposed direction (maintainer decides)

1. Accept file-only audit access (won't-do). Audit logs persist on-node for ~30 days and are retrievable via talosctl for forensics. If on-demand per-node retrieval is sufficient, close this — no component added.
2. Forward the audit file into Coroot via OTLP (recommended if centralization is wanted). A small log-forwarder DaemonSet on control-plane nodes (e.g. OpenTelemetry Collector / Vector / Fluent Bit) hostPath-mounts /var/log/audit/kube/ read-only, parses the JSON audit lines, and ships them to Coroot's OTLP logs endpoint — making audit events searchable alongside everything else. Coroot CE supports OTLP log ingestion.
3. API-server audit webhook backend. Add audit-webhook-* flags to talos/cluster/audit-logging.yaml streaming to an in-cluster receiver → Coroot. More moving parts than option 2 and couples audit delivery to a live receiver; the file backend (option 2's source) is more resilient.

Rough size

Medium — option 2 is one DaemonSet + a read-only hostPath on control planes + a CiliumNetworkPolicy egress to the Coroot OTLP endpoint + retention/parse config. The gating decision is whether centralizing audit logs is worth the added collector vs. accepting file-only access (option 1).

Acceptance criteria (if a forwarding option is chosen)

• Kubernetes API-server audit events are searchable in the Coroot UI
• Audit retention in Coroot is documented and aligned with the platform's needs
• The on-node file backend (audit-logging.yaml) is retained as the resilient primary
• NetworkPolicy allows the forwarder → Coroot OTLP endpoint
• docs/dr/alerting.md notes audit-log searchability

Context

• Superseded epic: [feature]: Add log aggregation to the kube-prometheus-stack observability setup #1507 · replacement: feat: replace kube-prometheus-stack with self-hosted Coroot #1742 · audit logging: feat: add Talos security hardening patches #1506
• Relevant files: talos/cluster/audit-logging.yaml, k8s/bases/infrastructure/coroot/coroot.yaml, docs/dr/alerting.md