Repository navigation
Align spending charts with forecast JSON schema - #76621
Conversation
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review. PR Code Quality ReviewerOutcome: The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Specification Reviewer completed successfully! PR Specification ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the forecast chart data-extraction and unavailable-data handling changes.
No open issues 🎉
What changed in this PR
Aligns the daily spending forecast charts with the current forecast JSON schema, addressing #76439 and avoiding misleading zero-valued projections when data is unavailable.
Changes:
- Flattens per-workflow run samples using each sample’s
dateand its parentworkflow_id. - Reads forecast percentiles from
weekly_monte_carloand displays unavailable states when no usable data exists. - Updates generated workflow hashes to match the revised source.
| File | Description |
|---|---|
.github/workflows/daily-spending-forecast.md |
Corrects chart data extraction and adds unavailable-data handling. |
.github/workflows/daily-spending-forecast.lock.yml |
Refreshes body and admission hashes for the updated workflow. |
Review effort: Balanced (auto)
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
Found two data-quality issues in the chart transformations: malformed historical samples and incomplete Monte Carlo workflows are silently omitted, which can make partial results look complete. Inline comments explain the impact and expected handling.
Verification
Static review only; repository tests were not run per review constraints.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again
| elif not df.empty: | ||
| df["date"] = pd.to_datetime(df["date"], errors="coerce") | ||
| df["aic"] = pd.to_numeric(df["aic"], errors="coerce") | ||
| df = df.dropna(subset=["date", "aic", "workflow_id"]) |
There was a problem hiding this comment.
Dropping every sample with an invalid or missing date silently makes the chart and top-five ranking represent only a partial subset of observed spend.
💡 Preserve data-quality visibility
dropna removes undated samples without recording how many were discarded, so the resulting chart can understate spending and change the selected workflows while still looking complete. Track the dropped count and surface it in the chart or report (or mark the chart unavailable when required historical data is incomplete).
| ] | ||
| if not all(isinstance(value, (int, float)) and math.isfinite(value) | ||
| for value in percentiles): | ||
| continue |
There was a problem hiding this comment.
Skipping workflows with incomplete Monte Carlo percentiles makes a partially populated chart look like a complete forecast.
💡 Surface omitted workflows
The continue path removes any workflow missing one percentile, and the empty-state message appears only when all workflows are removed. A report with 3 of 10 workflows therefore gives no indication that seven forecasts are unavailable. Track skipped workflow IDs and render an explicit unavailable note or expose the omission in the report.
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /diagnosing-bugs to the two chart scripts. Overall the PR moves the charts onto the real forecast.json schema (run_samples with parent workflow_id, and weekly_monte_carlo P10/P50/P90), which is correct and matches pkg/cli/forecast_types.go. Two inline suggestions are posted.
📋 Key Themes & Highlights
Key Themes
- Rolling window semantics:
rolling(7)is per observation, not per calendar day, so gaps in run dates distort the 7-day overlay. - Percentile guard: booleans pass the numeric check, and inverted P10/P50/P90 values are not rejected.
Positive Highlights
- ✅
sort_index()before the rolling mean is required and present. - ✅
weekly_monte_carlois treated as optional (omitempty/null) and the chart falls back to an explicit unavailable state. - ✅ Flattening
run_sampleswith the parentworkflow_idmatches the Go producer.
Limitations
- The chart scripts could not be executed in this sandbox (PyPI TLS failure), so the findings come from reading the code, not from running it.
- Lock-file
body_hashconsistency was not verified. Please confirmmake recompileproduces no diff.
@copilot please address the review comments above.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~7 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again
| daily = grp.groupby("date")["aic"].mean().sort_index() | ||
| ax.plot(daily.index, daily.values, marker="o", linewidth=1.5, | ||
| label=name, color=palette[idx], alpha=0.8) | ||
| rolling = daily.rolling(7, min_periods=1).mean() |
There was a problem hiding this comment.
[/diagnosing-bugs] rolling(7) counts observations, not calendar days. Days without sampled runs are skipped, so the "7-day" overlay can span far more than 7 days. Use a time-based window instead.
💡 Suggested fix
rolling = daily.rolling("7D", min_periods=1).mean()daily is indexed by the datetime date column, so "7D" gives a true 7-calendar-day window. The sort_index() added in this PR is required for a time-based window to be valid.
@copilot please address this.
| weekly.get("p50_projected_aic"), | ||
| weekly.get("p90_projected_aic"), | ||
| ] | ||
| if not all(isinstance(value, (int, float)) and math.isfinite(value) |
There was a problem hiding this comment.
[/tdd] This guard accepts bool values (a subclass of int) and does not check ordering. An inverted set such as P10 > P50 would draw misleading bars. Reject booleans and require p10 <= p50 <= p90, then add a fixture for the inverted case.
💡 Suggested check
def _is_number(v):
return isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v)
if not (_is_number(p10) and _is_number(p50) and _is_number(p90) and p10 <= p50 <= p90):
continueA regression fixture with inverted percentiles would lock this in.
@copilot please address this.
|
🎉 This pull request is included in a new release. Release: |
The spending charts used fields that don’t match the current forecast output, risking incorrect dates, workflow grouping, and zero-valued projections when data is absent.
run_samples, using each sample’sdateand its parentworkflow_id.weekly_monte_carlo; show an explicit unavailable state when forecast data is missing or incomplete.{ "workflow_id": "daily-report", "run_samples": [{"date": "2026-11-10", "aic": 12.5}], "weekly_monte_carlo": { "p10_projected_aic": 20, "p50_projected_aic": 35, "p90_projected_aic": 60 } }