Skip to content

OpenCost allocations report zero usage against Coroot Prometheus #2689

Closed

Description

@devantler

🤖 Generated by the Daily AI Engineer

Reproduction and evidence

Read-only production verification on 2026-07-18 found the OpenCost release Ready with two healthy primary replicas, but its Coroot-backed metrics input is incomplete:

  • Coroot Prometheus returns 0 series for node_cpu_seconds_total, container_cpu_usage_seconds_total, and container_memory_working_set_bytes.
  • It does return kube-state data (kube_pod_container_resource_requests: 677 current samples; kube_node_status_capacity: 54).
  • OpenCost GET /allocation/compute?window=24h&aggregate=namespace returns HTTP 200 and 34 namespaces with non-zero requests/costs, but 0/34 namespaces have non-zero cpuCoreUsageAverage or ramByteUsageAverage.

The release being healthy therefore masks a functional degradation: allocation cost is derived from requested resources, while actual usage and efficiency are absent.

Audience and impact

Operators using OpenCost for utilization, idle-spend, or request-versus-usage decisions receive plausible-looking costs with zero actual CPU/RAM usage. This can misstate efficiency and makes the current OpenCost view unsafe as a usage-aware source of record.

The same latent design was about to be copied into the Platform template; platform-template#82 stopped that rollout after exact-head review prompted this live check.

Expected behavior

Choose and prove one coherent cost path:

  1. keep OpenCost and provide a Prometheus source with the standard node/cAdvisor/kube-state series its allocation model needs; or
  2. retire OpenCost and use the independently live-proven Coroot node/application cost rollups instead.

Do not treat Helm readiness or a successful allocation HTTP response as sufficient proof.

Acceptance criteria

  • Record the chosen OpenCost-versus-Coroot cost architecture and its trade-off.
  • If OpenCost stays, its Prometheus query surface contains the required standard usage series and a 24-hour allocation reports non-zero CPU/RAM usage for active namespaces.
  • If OpenCost retires, remove its release, route, policy, and related ownership cleanly, and update user-facing cost guidance to Coroot.
  • Add an executable or operational check that detects the current zero-usage false-green state.
  • Validate statically and verify the resulting cost view read-only in production before closing.

Rough size: M

Related: #1784, #2043, devantler-tech/platform-template#65

Activity

  1. coderabbitai commented on Jul 18, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #2507 - fix(apps): make actual-budget openid schema offline-resolvable [merged]
    #2527 - feat: install the OpenFeature Operator (runtime feature-flag layer) [merged]
    #2554 - chore(deps): update helm release opencost to v2.5.26 [merged]
    #2555 - fix(opencost): patch ui probe timeouts [merged]
    #2578 - fix(policy-reporter): run all components at the HA replica floor [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false
    

    💬 Have feedback or questions? Drop into our discord!

  2. moved this to 🫴 Ready in 🌊 Project Boardon Jul 18, 2026
  3. added theissue type on Jul 18, 2026
  4. moved this from 🫴 Ready to 📥 Backlog in 🌊 Project Boardon Jul 18, 2026
  5. devantler commented on Jul 22, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Read-only revalidation on 2026-07-22 narrows the failure to an incompatible metrics input, not OpenCost-to-Prometheus reachability:

    • The 3-day allocation still returns 34 namespaces, but cpuCoreUsageAverage sums to zero across all of them.
    • Coroot Prometheus returns 339 kube_pod_container_resource_requests{resource="cpu"} series, but zero container_cpu_usage_seconds_total series and no 5-minute container CPU rate.
    • The live coroot-prometheus deployment config only self-scrapes 127.0.0.1:9090; its other data arrives through remote write. The required cAdvisor/container-usage series are not present in that TSDB.
    • OpenCost is healthy and explicitly points PROMETHEUS_SERVER_ENDPOINT at this Coroot Prometheus service.

    So the current root cause is the metrics contract: the configured Prometheus source has kube-state request data but does not ingest the container usage series OpenCost needs. Rightsizing remains correctly fail-closed until a compatible usage source is supplied and the proposed regression check proves non-zero usage for active namespaces.

  6. self-assigned this
    on Jul 23, 2026
  7. moved this from 📥 Backlog to 👀 In Review in 🌊 Project Boardon Jul 23, 2026
  8. added a commit that references this issue on Jul 27, 2026
  9. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions