Repository navigation
Closed
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupenhancementFeature requestsFeature requestsinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Part of epic #581 (self-improving skills). Follow-up to #584 / #741.
Why
skill-eval-report.yml (#584) runs the deterministic holdout scorer daily (07:00 UTC) and on dispatch, but its result is effectively invisible:
- It is intentionally non-blocking (
continue-on-error: true) — a held-out regression or a scorer hard-error never fails the workflow. - It has
permissions: contents: readonly, so it writes no issue and no PR comment. Output lives solely in the workflow run's step summary, an artifact (eval-report.json), and a::notice::/::warning::annotation.
Net effect: a real skill regression (outcome=regression) — or, worse, a blind scorer failure (outcome=error, hidden because the workflow still goes green) — is only discoverable by manually opening the Actions run. Continuous skill-quality "visibility" that nobody sees defeats the purpose of Phase 1.
What
Surface the daily eval outcome where a human will actually see it, without making it blocking:
- On
outcome=regressionoroutcome=error: open or update a single tracking issue (idempotent — update the same issue in place, never spam a new one per day), labelled e.g.eval-health, containing the score table + regressed-case detail and a link to the run. - On recovery (
outcome=passafter a prior open tracking issue): comment + close the tracking issue. - Alternative/complementary: fold the latest score + outcome into the existing
daily-pr-review-healthdigest so the org has one daily health surface instead of two silent crons.
Acceptance criteria
- A
regressionorerrorin the scheduled run produces a visible, de-duplicated notification (tracking issue or digest line) — not just an Actions annotation. - Healthy (
pass) runs create no noise (no new issue/comment when nothing is wrong). - The workflow stays non-blocking —
continue-on-errorpreserved; this story adds visibility only, no gating. - Least-privilege: add only the
issues: writescope required for the notify step; keepcontents: readfor checkout. - An offline test (stub, no network) covers the open/update/close + de-dupe logic, wired into
lint.yml(mirrors the#583/#584offline-test pattern).
Out of scope
- Making the eval blocking / gating merges (a later phase decision).
- Growing the holdout case set or fixing per-trigger coverage (separate follow-up — note
workflow-secretsandunresolved-threadscurrently live only indev/, so they are not scored). - Reducing scorer/model nondeterminism (noise-floor) — separate.
Metadata
Metadata
Assignees
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupenhancementFeature requestsFeature requestsinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity
github-actions commentedon Jun 15, 2026
Thanks for the thorough write-up and for connecting this back to the epic — the visibility gap is well-articulated! Before we start on implementation: for the tracking-issue approach, should the issue be opened in this same repo (
petry-projects/.github-private) or in a separate ops/observability repo, and is there an existingeval-healthlabel to reuse or should the workflow create it on first use?don-petry commentedon Jun 15, 2026
Dev-Lead Implementation Plan
Issue: #747 — [Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent)
Scope
Add a non-blocking visibility layer to
skill-eval-report.yml: onoutcome=regression|error, open-or-update a single de-duplicatedeval-healthtracking issue; on recovery (passwith a prior open issue), comment + close it. Healthy runs stay silent. Logic lives in a stub-testable script with offline bats coverage.Implementation Checklist
scripts/evals/notify-eval-health.sh— new: find-or-create/update/close the singleeval-healthtracking issue (idempotent, de-duped by stable title+label), rendering the score table + regressed-case detail + run link. Modeled onscripts/idea-triage/upsert-queue.sh. Honors DRY_RUN; usesghfrom PATH (stub-injectable)..github/workflows/skill-eval-report.yml— addissues: writeto the job permissions (keepcontents: read); add anif: always()"Notify eval health" step that invokes the script with OUTCOME / SKILL / REPORT_PATH / RUN_URL.continue-on-erroron the eval step is preserved (no gating added).tests/test_eval_health_notify.bats— new offline test (stubghon PATH, no network) covering open / update / close / de-dupe / no-noise-on-pass..github/workflows/lint.yml— add the new bats file to the repo-specific test list.tests/test_skill_eval_report.bats— extend with workflow-YAML assertions for the newissues: writescope and the notify step.Tests to Write
eval-health)issues: writeand wires the notify stepTest Command
bats tests/test_eval_health_notify.bats tests/test_skill_eval_report.bats tests/test_skill_evals.bats(full suite: thebatslist in.github/workflows/lint.yml)don-petry commentedon Jun 15, 2026
Confirming the defaults so you're unblocked (matches your 00:29 plan):
petry-projects/.github-private, next to the workflow and CODEOWNERS. No separate ops/observability repo.eval-healthlabel; have the workflow create-it-on-first-use (idempotent find-or-create) so there's no manual setup step. Single issue, de-duped by stable title + theeval-healthlabel, updated in place.Proceed on that basis. One real-world note to fold in: the first baseline dispatch of
skill-eval-report.ymljust came backoutcome=regression(run 27516878017) — so this notify path will light up immediately on the first scheduled run. Please make sure the issue body renders the per-case expected-vs-actual detail for regressed cases (your checklist covers it) — that's the first thing we'll need to read.Generated by Claude Code
don-petry commentedon Jun 15, 2026
Dev-Lead: Implementation Complete
Plan Execution
scripts/evals/notify-eval-health.sh— idempotent open/update/close of a single de-duplicatedeval-healthtracking issue (modeled onidea-triage/upsert-queue.sh).github/workflows/skill-eval-report.yml— addedissues: write(keptcontents: read) and anif: always()"Notify eval health" step;continue-on-erroron the eval step preserved (no gating)tests/test_eval_health_notify.bats— offline stub-ghtests for open / update / close / de-dupe / no-noise + guardstests/test_skill_eval_report.bats— extended with workflow-YAML assertions for the new scope + notify step.github/workflows/lint.yml— added the new bats file to the repo-specific test listBehavior (visibility only, non-blocking)
outcome=regression|error→ opens or updates in place a singleeval-healthtracking issue (score table + regressed-case detail + run link). Never spams a new issue per day.outcome=passwith a prior open tracking issue → comments Recovered and closes it.outcome=passwith no open issue → zero noise (no create/comment/close).issues: writeadded;contents: readretained for checkout.Test Results
Files Changed
scripts/evals/notify-eval-health.sh(new): notifier logic — find-or-create/update/close..github/workflows/skill-eval-report.yml:issues: write+ notify step (always()).tests/test_eval_health_notify.bats(new): offline notifier tests.tests/test_skill_eval_report.bats: +4 YAML assertions for [Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent) #747..github/workflows/lint.yml: registered new bats file.Notes
daily-pr-review-healthdigest fold-in was explicitly "alternative/complementary" and is left out to keep scope minimal.feat: implement issue #747 — [Phase 1] Make the Skill Eval Report vis…