Skip to content

Align spending charts with forecast JSON schema - #76621

Merged
pelikhan merged 2 commits into
mainfrom
copilot/aw-top-10-align-spending-charts
Nov 11, 2026
Merged

pelikhan merged 2 commits into
mainfrom
copilot/aw-top-10-align-spending-charts

Conversation

Copilot AI commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The spending charts used fields that don’t match the current forecast output, risking incorrect dates, workflow grouping, and zero-valued projections when data is absent.

  • Trend chart: Flatten per-workflow run_samples, using each sample’s date and its parent workflow_id.
  • Forecast distribution: Read P10/P50/P90 from weekly_monte_carlo; show an explicit unavailable state when forecast data is missing or incomplete.
{
  "workflow_id": "daily-report",
  "run_samples": [{"date": "2026-11-10", "aic": 12.5}],
  "weekly_monte_carlo": {
    "p10_projected_aic": 20,
    "p50_projected_aic": 35,
    "p90_projected_aic": 60
  }
}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Align spending charts with forecast JSON Align spending charts with forecast JSON schema Nov 10, 2026
Copilot AI requested a review from gh-aw-bot November 10, 2026 22:35
@pelikhan
pelikhan marked this pull request as ready for review November 10, 2026 23:21
Copilot AI dynamic review requested due to automatic review settings November 10, 2026 23:21
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~3 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~2 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #76621

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~4 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76621

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the forecast chart data-extraction and unavailable-data handling changes.

No open issues 🎉

What changed in this PR

Aligns the daily spending forecast charts with the current forecast JSON schema, addressing #76439 and avoiding misleading zero-valued projections when data is unavailable.

Changes:

  • Flattens per-workflow run samples using each sample’s date and its parent workflow_id.
  • Reads forecast percentiles from weekly_monte_carlo and displays unavailable states when no usable data exists.
  • Updates generated workflow hashes to match the revised source.
File Description
.github/​workflows/​daily-spending-forecast.md Corrects chart data extraction and adds unavailable-data handling.
.github/​workflows/​daily-spending-forecast.lock.yml Refreshes body and admission hashes for the updated workflow.

Review effort: Balanced (auto)

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-10T23:24:29Z
review_event: COMMENT
top_themes: [silent omission of incomplete historical samples, partial Monte Carlo forecasts presented as complete]
files_reviewed: [.github/workflows/daily-spending-forecast.md]
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two data-quality issues in the chart transformations: malformed historical samples and incomplete Monte Carlo workflows are silently omitted, which can make partial results look complete. Inline comments explain the impact and expected handling.

Verification

Static review only; repository tests were not run per review constraints.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

elif not df.empty:
df["date"] = pd.to_datetime(df["date"], errors="coerce")
df["aic"] = pd.to_numeric(df["aic"], errors="coerce")
df = df.dropna(subset=["date", "aic", "workflow_id"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping every sample with an invalid or missing date silently makes the chart and top-five ranking represent only a partial subset of observed spend.

💡 Preserve data-quality visibility

dropna removes undated samples without recording how many were discarded, so the resulting chart can understate spending and change the selected workflows while still looking complete. Track the dropped count and surface it in the chart or report (or mark the chart unavailable when required historical data is incomplete).

]
if not all(isinstance(value, (int, float)) and math.isfinite(value)
for value in percentiles):
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping workflows with incomplete Monte Carlo percentiles makes a partially populated chart look like a complete forecast.

💡 Surface omitted workflows

The continue path removes any workflow missing one percentile, and the empty-state message appears only when all workflows are removed. A report with 3 of 10 workflows therefore gives no indication that seven forecasts are unavailable. Track skipped workflow IDs and render an explicit unavailable note or expose the omission in the report.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd and /diagnosing-bugs to the two chart scripts. Overall the PR moves the charts onto the real forecast.json schema (run_samples with parent workflow_id, and weekly_monte_carlo P10/P50/P90), which is correct and matches pkg/cli/forecast_types.go. Two inline suggestions are posted.

📋 Key Themes & Highlights

Key Themes

  • Rolling window semantics: rolling(7) is per observation, not per calendar day, so gaps in run dates distort the 7-day overlay.
  • Percentile guard: booleans pass the numeric check, and inverted P10/P50/P90 values are not rejected.

Positive Highlights

  • ✅ sort_index() before the rolling mean is required and present.
  • ✅ weekly_monte_carlo is treated as optional (omitempty/null) and the chart falls back to an explicit unavailable state.
  • ✅ Flattening run_samples with the parent workflow_id matches the Go producer.

Limitations

  • The chart scripts could not be executed in this sandbox (PyPI TLS failure), so the findings come from reading the code, not from running it.
  • Lock-file body_hash consistency was not verified. Please confirm make recompile produces no diff.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~7 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

daily = grp.groupby("date")["aic"].mean().sort_index()
ax.plot(daily.index, daily.values, marker="o", linewidth=1.5,
label=name, color=palette[idx], alpha=0.8)
rolling = daily.rolling(7, min_periods=1).mean()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] rolling(7) counts observations, not calendar days. Days without sampled runs are skipped, so the "7-day" overlay can span far more than 7 days. Use a time-based window instead.

💡 Suggested fix
rolling = daily.rolling("7D", min_periods=1).mean()

daily is indexed by the datetime date column, so "7D" gives a true 7-calendar-day window. The sort_index() added in this PR is required for a time-based window to be valid.

@copilot please address this.

weekly.get("p50_projected_aic"),
weekly.get("p90_projected_aic"),
]
if not all(isinstance(value, (int, float)) and math.isfinite(value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] This guard accepts bool values (a subclass of int) and does not check ordering. An inverted set such as P10 > P50 would draw misleading bars. Reject booleans and require p10 <= p50 <= p90, then add a fixture for the inverted case.

💡 Suggested check
def _is_number(v):
    return isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v)

if not (_is_number(p10) and _is_number(p50) and _is_number(p90) and p10 <= p50 <= p90):
    continue

A regression fixture with inverted percentiles would lock this in.

@copilot please address this.

@pelikhan
pelikhan merged commit e80177a into main Nov 11, 2026
80 checks passed
@pelikhan
pelikhan deleted the copilot/aw-top-10-align-spending-charts branch November 11, 2026 00:17
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AW Top 10] 09 Align spending charts with forecast JSON

4 participants