Skip to content

feat(observability): stage OpenCost retirement with Coroot - #82

Merged
devantler merged 3 commits into
mainfrom
claude/coroot-opencost-layer-81
Jul 18, 2026
Merged

devantler merged 3 commits into
mainfrom
claude/coroot-opencost-layer-81

Conversation

@devantler

@devantler devantler commented Jul 18, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Generated by the Daily AI Engineer

Why

The default-off Coroot migration must not copy the reference Platform's false-green OpenCost setup. Live verification found that Coroot Prometheus lacks the standard usage series OpenCost needs: allocations return requested-resource costs while CPU and RAM usage remain zero. The reference defect is tracked in devantler-tech/platform#2689.

What

  • stage complete OpenCost retirement only in the explicit Coroot opt-in fixtures
  • keep every real Docker and Hetzner provider payload byte-for-byte unchanged by default
  • preserve the Coroot controller/custom-resource ordering and Hetzner-only audit forwarder
  • remove cross-layer routes, network rules, grants, and security exemptions with
    the retired workload
  • extend the executable contract to reject any OpenCost object or reference in
    either opt-in layer and fail closed when indexed policy entries drift

Fixes #81
Part of #65
Part of #24

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

User evaluation at head b6ded35cdf220db380718dedfa1804bbd2777058: rendered the template as a consumer. All four real Docker/Hetzner controller and infrastructure payloads are byte-for-byte identical to origin/main, so the staged option changes no provider by default. Both explicit opt-in infrastructure fixtures render exactly one OpenCost release, depend only on coroot-operator, query coroot-prometheus.observability.svc.cluster.local:9090, disable ServiceMonitor output, and contain no legacy monitoring-namespace ingress or egress.

The executable contract was proved RED before implementation and again for the review-discovered stale ingress grant, then GREEN after each fix. Both KSail validations pass (86 kustomizations / 258 files each), together with the observability both-state validator, naming gate, ShellCheck, shfmt, YAML hygiene, and diff checks. An independent manifest review found no remaining blocker.

No cluster was started or mutated: this repository's user surface for a default-off template slice is its rendered provider output. CI and exact-head automated review are still pending, so the PR remains draft.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6ded35cdf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler devantler changed the title feat(observability): stage Coroot-backed OpenCost handoff feat(observability): stage OpenCost retirement with Coroot Jul 18, 2026
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

Updated user evaluation at head 97af39fe71ccae64fd0cebf4e935b8d7c19af768 (supersedes the initial handoff evaluation): rendered the template as a consumer after correcting the design. All four real Docker/Hetzner controller and infrastructure payloads remain byte-for-byte identical to origin/main. Both explicit Coroot opt-in pairs render zero resources in the opencost namespace, while each retains one Coroot operator and one later Coroot resource; only Hetzner retains the audit-log-forwarder.

The retirement contract was mutation-tested twice. Removing the component brought Namespace/opencost back and failed. Adding a synthetic future ConfigMap in the OpenCost namespace first exposed a fail-open kind list, then failed after the assertion was made namespace-wide; removing the mutation returned GREEN. Both KSail validations pass (85 kustomizations / 258 files each), together with the observability contract, naming, ShellCheck, shfmt, YAML hygiene, and diff checks. A fresh independent cumulative review found no remaining issue.

Read-only production evidence explains the pivot: OpenCost returned 34 namespace allocations with non-zero requested-resource costs but zero CPU/RAM usage for all 34 because Coroot Prometheus lacks the standard node/cAdvisor usage series. That reference defect is captured in devantler-tech/platform#2689; no production resource was mutated. CI and exact-head automated review are pending, so this PR remains draft.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97af39fe71

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

Final user evaluation at head 67da63606b714e15974883abf90f871ce8ea3414
(supersedes both earlier evaluations):
rendered the template as a consumer
after the review-driven hardening. All four real Docker/Hetzner controller and
infrastructure payloads remain byte-for-byte identical to origin/main. The
four explicit Coroot opt-in payloads contain zero OpenCost objects or references;
each provider retains one Coroot operator and one later Coroot resource, and only
Hetzner retains the audit-log-forwarder.

The contract rejected three negative controls: a future resource in the OpenCost
namespace, drift in the copied auth-proxy configuration, and a mismatched indexed
policy entry. It also rejected a surviving security exemption. Every indexed
deletion now tests the exact OpenCost object/value before removal. Both local and
production KSail validations pass (86 kustomizations / 258 files), together with
the observability contract, naming, ShellCheck, shfmt, YAML lint, and diff checks.
A fresh independent cumulative review found no remaining issue.

Read-only production evidence explains the retirement direction: OpenCost
returned 34 namespace allocations with non-zero requested-resource costs but zero
CPU/RAM usage for all 34 because Coroot Prometheus lacks the standard node/cAdvisor
usage series. That reference defect is captured in devantler-tech/platform#2689;
no production resource was mutated. CI and exact-head automated review are still
pending, so this PR remains draft.

@devantler

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 67da63606b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

Readiness decision for 67da63606b714e15974883abf90f871ce8ea3414:

  • programmatic: focused RED/GREEN retirement contract, ShellCheck, shfmt, YAML
    lint, naming, and both KSail scaffold validations pass (86 kustomizations /
    258 files); exact-head GitHub CI is 12 passed / 0 failed
  • review: independent cumulative review is clean; Codex reviewed this exact
    commit and found no major issues; zero unresolved threads or review-body findings
  • user evaluation: all four real provider renders are byte-identical to
    origin/main; explicit Coroot fixtures contain zero OpenCost objects/references
    while preserving Coroot ordering and the Hetzner-only audit forwarder
  • merge hygiene: current head matches the reviewed/tested commit, base is current,
    mergeability is CLEAN, and the CodeRabbit status is green

No cluster was created or mutated; the production investigation was read-only.
All three readiness gates are satisfied, so this routine-owned draft is ready for
self-promotion and squash merge.

@devantler
devantler marked this pull request as ready for review July 18, 2026 11:05
@devantler
devantler requested a review from a team as a code owner July 18, 2026 11:05
@devantler
devantler merged commit b7d144a into main Jul 18, 2026
16 checks passed
@devantler
devantler deleted the claude/coroot-opencost-layer-81 branch July 18, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stage OpenCost retirement behind the Coroot option

1 participant