Skip to content

Add forecast operation to generated agentic maintenance workflow - #31422

Merged
pelikhan merged 15 commits into
mainfrom
copilot/add-forecast-command-agentic-job
May 11, 2026
Merged

pelikhan merged 15 commits into
mainfrom
copilot/add-forecast-command-agentic-job

Conversation

Copilot AI commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

✨ Enhancement

What does this improve?
This PR adds and hardens the generated forecast_report maintenance operation so it produces clearer, more actionable forecast issues and is more reliable in CI.

Why is this valuable?
Forecast issues are now easier to read, link back to their source run, provide better diagnostics when projected ET values are zero, and are easier to maintain/test by moving issue-generation logic into a dedicated script file.

Implementation approach:

  • Added forecast_report to generated maintenance workflow output for forecast operations.
  • Set forecast_report timeout to 60 minutes.
  • Filtered the known experimental forecast warning while preserving other stderr output.
  • Warmed forecast cache with broader history before forecasting:
    • gh aw logs --repo "${{ github.repository }}" --start-date -30d --count 1500
  • Added forecast logs cache restore/save around warm-up so existing .github/aw/logs cache can be reused when available before calling gh aw logs, then persisted for subsequent runs.
  • Kept fail-fast behavior when run summary cache is missing after warm-up.
  • Updated forecast issue rendering to:
    • show markdown table output,
    • pretty-format ET values (human-readable numeric formatting),
    • include a footnote with the source workflow run HTML link,
    • include targeted diagnostics when workflows have sampled runs but forecast ET remains zero.
  • Moved forecast issue creation logic from inline workflow YAML into actions/setup/js/create_forecast_issue.cjs and updated generated workflow/script wiring to call the file.
  • Updated targeted tests to validate both workflow generator invocation of the new script and the extracted script behavior/content.
  • Updated workflow generator assertions to validate forecast logs cache restore/save steps.

Testing

  • Ran targeted workflow generator test coverage for forecast-related assertions.
  • Ran targeted JS tests for create_forecast_issue.cjs.
  • Built the project successfully.
  • Ran parallel validation (code review feedback addressed; CodeQL scan timed out in this environment).

Copilot AI and others added 2 commits May 11, 2026 05:03
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title Add forecast operation to agentic maintenance workflow Add forecast operation to generated agentic maintenance workflow May 11, 2026
Copilot AI requested a review from pelikhan May 11, 2026 05:09
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot use markdown render format for forecast. Validate why all predications are 0

See #31429

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot use markdown render format for forecast. Validate why all predications are 0

See #31429

Implemented in 56ae9d9.

Forecast issue publishing now renders a markdown table (workflow / sampled runs / P50 ET) instead of embedding raw JSON.

I also validated the zero-projection behavior: in this maintenance job, ET values come from local gh aw logs run-summary cache, and the runner starts clean, so projected ET can be 0 across workflows. The generated issue now includes an explicit note explaining that and how to warm cache (gh aw logs --repo owner/repo --all) before running forecast.

@github-actions github-actions Bot mentioned this pull request May 11, 2026
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot add call to logs to warm cache. Fail command if cache is missing

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot add call to logs to warm cache. Fail command if cache is missing

Addressed in f86f945. The forecast job now warms cache with gh aw logs --repo "${{ github.repository }}" --all before running forecast, and it fails fast if no .github/aw/logs/run-*/run_summary.json files are present after warm-up.

@pelikhan

Copy link
Copy Markdown
Collaborator

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot review crash https://github.com/github/gh-aw/actions/runs/25665958684/job/75338343701#step:7:1

Root cause confirmed from the run logs: gh aw logs does not support --all, which caused step 7 to fail with unknown flag: --all. Fixed in 5ffde12 by switching warm-up to gh aw logs --repo "${{ github.repository }}" --count 100 and keeping the missing-cache fail-fast check.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot increase forecasts time out to 60min

Fix warning
https://github.com/github/gh-aw/actions/runs/25668041568/job/75345643389#step:8:1

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI and others added 2 commits May 11, 2026 12:10
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot increase forecasts time out to 60min

Fix warning...

Addressed in 208b61c and 66488b0. forecast_report now sets timeout-minutes: 60, and the forecast command filters the known experimental warning while preserving other stderr output. I also tightened the workflow generator test to validate the full stderr-filter command and explicit bash shell usage.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('node:fs');

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot move script to file, add tests

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0087c0a.

  • Moved the inline forecast issue-generation script into actions/setup/js/create_forecast_issue.cjs.
  • Updated generated maintenance workflow to call that script via setup_globals.cjs + main().
  • Added tests in actions/setup/js/create_forecast_issue.test.cjs and updated workflow generator assertions accordingly.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot

  • add footnote with run html link
  • pretty render ET...

Addressed in 4f50e3d.

  • Added a forecast issue footnote linking back to the maintenance workflow run HTML URL.
  • ET values now render in a pretty formatted style (thousands separators) in the markdown table.
  • Investigated zero forecasts and updated warm-up to fetch a broader 30-day cache (gh aw logs --start-date -30d --count 1500), plus added a targeted diagnostic note when workflows still have sampled runs but ET=0 due to missing token usage in cached summaries.

Copilot AI and others added 2 commits May 11, 2026 12:33
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot downloads the index cache memory if available to accelerate logs gathering before actually calling logs

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot downloads the index cache memory if available to accelerate logs gathering before actually calling

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented May 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot downloads the index cache memory if available to accelerate logs gathering before actually calling

Implemented in d7a4912. The forecast job now restores .github/aw/logs from Actions cache before running gh aw logs, then saves the refreshed logs cache after forecast generation so subsequent runs can warm faster when cache is available.

@pelikhan
pelikhan marked this pull request as ready for review May 11, 2026 16:59
Copilot AI review requested due to automatic review settings May 11, 2026 16:59
@pelikhan
pelikhan merged commit 36536f7 into main May 11, 2026
29 of 34 checks passed
@pelikhan
pelikhan deleted the copilot/add-forecast-command-agentic-job branch May 11, 2026 17:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new forecast maintenance operation to the generated agentic maintenance workflow and extracts forecast issue rendering into a dedicated script for clearer output and better CI reliability.

Changes:

  • Add forecast_report job (60m timeout) and wire it into the generated and checked-in maintenance workflow.
  • Warm forecast inputs via gh aw logs with cache restore/save around .github/aw/logs, then run gh aw forecast --json with filtered experimental-warning stderr.
  • Extract forecast issue creation into actions/setup/js/create_forecast_issue.cjs and add targeted Vitest coverage.
Show a summary per file
File Description
pkg/workflow/maintenance_workflow_yaml.go Generates the new forecast_report maintenance job and adds forecast as a selectable operation.
pkg/workflow/maintenance_workflow_test.go Extends generator tests to assert forecast_report wiring, conditions, caching steps, and setup-go pin counts.
.github/workflows/agentics-maintenance.yml Updates the checked-in workflow to include the new forecast operation/job and caching/warm-up steps.
actions/setup/js/create_forecast_issue.cjs New script that reads forecast JSON and creates a formatted issue body (table + diagnostics + source run link).
actions/setup/js/create_forecast_issue.test.cjs New Vitest coverage for formatting/diagnostics and missing-report behavior.
docs/src/content/docs/guides/ephemerals.md Documents the new forecast operation in the ephemerals maintenance operations list/details.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

docs/src/content/docs/guides/ephemerals.md:120

  • The forecast operation description says the issue will include the JSON payload in a fenced block, but the implementation now renders a markdown table with formatted ET values, diagnostics, and a source-run footnote (no embedded JSON block). Please update this doc line to match the new issue format.
- **`validate`**: Runs `gh aw compile --validate --no-emit --zizmor --actionlint --poutine --verbose`. If errors or warnings are found, creates or updates a GitHub issue titled `[aw] workflow validation findings` with the full output.
- **`activity_report`**: Runs `gh aw logs --format markdown` for the last 24 hours, 7 days, and 30 days (up to 1000 runs each), then creates an issue titled `[aw] agentic status report` with all three time-range sections as collapsible `<details>` blocks. Downloaded logs are cached under `./.cache/gh-aw/activity-report-logs`. The job has a 2-hour timeout and skips the 30-day query when the GitHub API is rate-limited.
- **`forecast`**: Runs `gh aw forecast --repo <owner/repo> --json`, writes the output to `./.cache/gh-aw/forecast/report.json`, then creates an issue titled `[aw] workflow forecast report` with the JSON payload embedded in a fenced block.
  • Files reviewed: 6/6 changed files
  • Comments generated: 2

Comment on lines +37 to +38
const p50 = workflow?.monte_carlo?.p50_projected_effective_tokens ?? workflow?.projected_effective_tokens ?? 0;
return [escapeCell(workflow.workflow_id), workflow.sampled_runs ?? 0, Number(p50)];
Comment on lines 108 to 112
| `clean_cache_memories` | Clean up outdated cache-memory entries (same as the automated scheduled cleanup) |
| `validate` | Run full workflow validation with all linters and file an issue if findings are detected |
| `activity_report` | Generate a repository activity report for the last 24 hours, week, and month, and create an issue with the results |
| `forecast` | Run a workflow token-usage forecast and create an issue with the JSON results |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants