Repository navigation
Sync observability stack to Coroot (+ audit-log-forwarder) like the reference platform #65
Description
🤖 Generated by the Daily AI Engineer
Problem. The template still scaffolds the retired observability set (kube-prometheus-stack, Loki, Grafana Alloy, OpenCost) while the reference platform has fully migrated to Coroot plus the audit-log-forwarder. New platforms instantiated today start on a stack the suite no longer develops or dogfoods — the template's single largest drift.
Proposed direction. Replace the Prometheus/Loki/Alloy/OpenCost bases with the platform's Coroot + audit-log-forwarder manifests (genericized: no org-specific endpoints/secrets), update the clusters' kustomizations and any dashboards/alerts wiring, and drop the retired bases. Prove the bootstrap flow still converges on both providers.
Rough size. L
Part of #24
Activity
devantler commentedon Jul 18, 2026
Progress: #82 merged as
b7d144ad1314ec9bec360f1dc7580d7729509c4e,closing child #81. The explicit Docker/Hetzner Coroot fixtures now model complete
OpenCost retirement across workload objects, auth-proxy routing, network policy,
ReferenceGrant, and security exemptions while every real provider render remains
byte-for-byte unchanged by default.
The direction changed after exact-head review prompted read-only live validation:
the reference Platform's OpenCost returns non-zero requested-resource costs but
zero CPU/RAM usage for all 34 namespace allocations because Coroot Prometheus
lacks the standard node/cAdvisor usage series. That live defect is tracked in
devantler-tech/platform#2689; it must not be copied into this template.The child passed 86-kustomization / 258-file validation for both local and prod,
negative controls for future references and array drift, exact-head CI, Codex
review, and post-merge TODO/CodeQL workflows. No cluster was mutated. Parent #65
remains open for real-provider activation and the remaining rollout slices.
devantler commentedon Jul 18, 2026
Delivered the first production-profile deduplication slice in #91 (closing #90): the Hetzner Coroot controller payload no longer renders the legacy Alloy audit collector, while default profiles, Docker Coroot, and the main Alloy pod-log shipper remain unchanged. The replacement audit forwarder is now guarded end to end—file receiver → logs pipeline → Coroot OTLP endpoint with the expected API-key header—and three negative controls cover disconnection, endpoint drift, and auth-header drift. All CI and exact-head Codex review gates passed before merge.
devantler commentedon Jul 20, 2026
Delivery child #96 is complete via #97 (
8f53f316466684619f748d1d67b10d57c8f1f09c). Both selectable Coroot profiles now retire Loki, the pod-log and audit Alloy collectors, their dead Grafana datasource, and the now-unconsumed Grafana chart source, while default Docker and Hetzner controller renders remain byte-for-byte unchanged.The focused gate covers both providers and rejected four ablations: either provider dropping the component, default-stack drift, and retention of the dead source. Both KSail validations covered 97 kustomizations / 258 files; exact-head CI and Codex re-review were green after its Grafana-source finding was fixed. Parent #65 remains open for the separate kube-prometheus-stack metrics/alerting migration.
devantler commentedon Nov 4, 2026
Status: the outcome is not met yet, so this stays open. All seven earlier sub-issues are closed, but they built Coroot as an opt-in profile (
clusters/local-coroot,clusters/prod-coroot). Onmainat98ff4b1, the default cluster paths still render Loki, Alloy, OpenCost and the full kube-prometheus-stack. Even the Coroot profile still runs kube-prometheus-stack, because theplatform-criticalPrometheusRule, the Alertmanager Watchdog heartbeat and the ServiceMonitor CRDs depend on it. The reference platform runs none of these.I split the remaining work into two sub-issues, in order:
*-corootpaths as aliases for instances that opted in.Alert, the Watchdog gives way tocluster-heartbeat, and the chart ServiceMonitors and themonitoringnamespace allowances go.The two can't be merged into one small change. The 1,400-line
validate-observability-option.sh, the heartbeat and SSO tests, the bootstrap workflow and the alerting docs all encode the current opt-in boundary, so each step needs its own reviewed PR.🤖 Generated with Claude Code