Repository navigation
[feature]: Add log aggregation to the kube-prometheus-stack observability setup #1507
Description
Summary
Extend the existing kube-prometheus-stack deployment to include centralized log aggregation, enabling Kubernetes audit logs and container logs to be queryable in a self-hosted Grafana instance.
Motivation
- The monitoring stack is currently metrics-only (Prometheus + Alertmanager). Grafana is disabled (
grafana.enabled: false). - Kubernetes audit logs (added in feat: add Talos security hardening patches #1506) are written to files on the EPHEMERAL partition — only accessible via
talosctlon individual nodes. - No centralized log search or correlation between logs and metrics exists.
- A self-hosted Grafana with Loki would enable audit log investigation, pod log search, and metric/log correlation — all from one UI.
Architecture
kube-prometheus-stack does not include Loki or Alloy as sub-charts. However, Grafana is a sub-chart that can be enabled. The approach is:
- Enable Grafana in kube-prometheus-stack via
grafana.enabled: true+ datasource configuration - Deploy Loki as a separate HelmRelease (Grafana Helm repo) for log storage/query
- Deploy Alloy as a separate HelmRelease (Grafana Helm repo) as the log collector
Note: Promtail is deprecated (EOL March 2026). Grafana Alloy is the official successor — an OpenTelemetry-based collector for logs, metrics, and traces.
┌─────────────────────────────────────────────────┐
│ kube-prometheus-stack │
│ ┌───────────┐ ┌─────────────┐ ┌───────────┐ │
│ │ Prometheus │ │ Alertmanager│ │ Grafana │ │
│ │ (metrics) │ │ (webhooks) │ │ (UI/viz) │ │
│ └─────┬─────┘ └─────────────┘ └──┬────┬───┘ │
│ │ │ │ │
└────────│─────────────────────────────│────│─────┘
│ ┌──────────────────────────┘ │
│ │ datasource: prometheus │ datasource: loki
│ │ │
┌─────┴──┴───┐ ┌──────┴──────┐
│ Prometheus │◄── scrape ──┐ │ Loki │
│ (metrics) │ │ │ (logs) │
└─────────────┘ │ └──────▲──────┘
│ │ push
┌─────┴────────────┴─────┐
│ Grafana Alloy │
│ (DaemonSet collector) │
│ • container logs │
│ • audit log files │
└─────────────────────────┘
Proposed Solution
1. Enable Grafana in kube-prometheus-stack
Update helm-release.yaml values:
grafana:
enabled: true
additionalDataSources:
- name: Loki
type: loki
url: http://loki-gateway.monitoring.svc.cluster.local
access: proxy2. Deploy Loki (separate HelmRelease)
Deploy Loki in monitoring namespace as a companion to kube-prometheus-stack. Use the grafana/loki Helm chart in single-binary or simple-scalable mode (homelab scale).
3. Deploy Alloy (separate HelmRelease)
Deploy Grafana Alloy as a DaemonSet to collect:
- Container logs from all pods (standard Kubernetes log collection)
- Audit log files from control-plane nodes by mounting
/var/log/kubernetes/audit/(host path from Talos audit-logging patch)
4. Optional: Audit webhook backend
Add audit-webhook-* API server flags to talos/cluster/audit-logging.yaml for real-time streaming to Loki (in addition to the file backend). This is optional — Alloy tailing the audit log files achieves the same result with better resilience.
Acceptance Criteria
- Grafana enabled in kube-prometheus-stack with Prometheus + Loki datasources
- Loki deployed as a separate HelmRelease in
monitoringnamespace - Alloy deployed as a DaemonSet HelmRelease collecting container logs + audit logs
- Kubernetes audit logs queryable via LogQL in Grafana
- Container logs from all namespaces searchable in Grafana
- Existing Prometheus metrics remain accessible in Grafana
- CiliumNetworkPolicy allows Alloy → Loki and Grafana → Loki traffic
Additional Context
- Current kube-prometheus-stack chart: v84.5.0 (prometheus-community)
- Current audit logging setup:
talos/cluster/audit-logging.yaml(file backend, 30-day rotation) - Helm charts needed: grafana/loki, grafana/alloy
- The file-based audit backend should be kept as the reliable primary — Loki provides the search/query layer on top
- Reference: Grafana Alloy docs, Loki docs
Activity
[-][feature]: Deploy Loki + Alloy for centralized log aggregation and Grafana integration[/-][+][feature]: Add log aggregation to the kube-prometheus-stack observability setup[/+]devantler commentedon May 23, 2026
Thoughtful, well-scoped proposal. A few notes from checking the current manifests, in case they help when implementing:
kube-prometheus-stackis now85.1.1(the issue referencesv84.5.0) —k8s/bases/infrastructure/controllers/kube-prometheus-stack/helm-release.yaml:14. The "enable Grafana" approach is unaffected.grafana.enabled: false— "alerting-only … operators look at logs / kubectl directly" — and runs Prometheus with no persistence (storageSpec: {}, emptyDir, 14d/5GiB). This is a deliberate "no UI, no persistence" stance, so the PR (anddocs/dr/alerting.md) should note it's intentionally superseded..github/instructions/talos-patches.instructions.md, local Talos disk is ephemeral while prod uses Hetzner CSI volumes, so the storage backend differs per environment — single-binaryfilesystemon emptyDir would drop logs on restart (same trade-off Prometheus accepts, but audit-log retention may warrant a PVC or object store).allow-monitoringCiliumNetworkPolicy(networkpolicy.yaml,endpointSelector: {}) already permits intra-namespace ingress + egress, so Alloy→Loki and Grafana→Loki are allowed if all three land in themonitoringnamespace. The net-new networking is mainly any Loki egress to external object storage; Alloy's access to audit/container log files is ahostPathconcern, not a network one.k8s/bases/infrastructure/controllers/<component>/(infrastructure-controllers Flux layer), alongside kube-prometheus-stack. There's no GrafanaHelmRepositoryin the repo yet, so adding one (https://grafana.github.io/helm-charts, shared by both charts) is a prerequisite.devantler commentedon May 25, 2026
Labelling this
roadmap— in a first-ever platform strategy review, this is the observability epic. Both prior milestones (Platform – Reimagined, Kubestronaut Journey) are 100% complete, so the platform's two live roadmap themes going forward are:grafana.enabled: false); enabling Grafana + Loki + Alloy gives audit-log search, pod-log search, and metric/log correlation in one UI. The 3-phase plan in the description (enable Grafana + Loki datasource → Loki → Alloy) is a clean, independently-shippable decomposition — each phase can land as its own draft PR.No scope change proposed here — just elevating this well-specified, maintainer-filed request to the roadmap so it's tracked as the observability north star. The
roadmaplabel + a tight 2-epic set keeps the roadmap current rather than long and stale.devantler commentedon Jun 28, 2026
Superseded by #1742 — closing
This epic proposed extending kube-prometheus-stack with Loki + Alloy + Grafana for centralized log aggregation. Since it was filed (2026-05-10), the observability stack was replaced wholesale by #1742 ("replace kube-prometheus-stack with self-hosted Coroot", merged 2026-06-04), so the approach here no longer maps to the codebase: there is no
kube-prometheus-stack, Grafana, Loki, or Alloy in the tree to extend.docs/dr/alerting.mddocuments the pivot directly:The original goals are already delivered by Coroot
logsTTL: 3d,k8s/bases/infrastructure/coroot/coroot.yaml), searchable in the Coroot UIobservability-namespace policy; no Loki/Alloy/Grafana to wireThe "no UI / no persistence" stance the old kube-prometheus-stack HelmRelease documented is intentionally superseded by Coroot (durable
hcloudPVCs + replicated ClickHouse).One residual carved out → #2318
The only #1507 criterion Coroot does not cover is Kubernetes API-server audit-log searchability. Audit events are written to a host file (
/var/log/audit/kube/audit.log,talos/cluster/audit-logging.yaml, from #1506), which the eBPF node-agent does not tail — so they stay file-only on nodes, retrievable viatalosctl. That narrow, still-relevant piece is tracked as a fresh, correctly-scoped issue: #2318.Closing this as superseded; the live observability north-star is Coroot, and the remaining audit-log decision lives in #2318. Reopen if the kube-prometheus-stack framing is still wanted for any reason.