Skip to content

Fix harness evaluation result discovery - #39289

Merged
AndriySvyryd merged 5 commits into
mainfrom
copilot/fix-evaluate-workflow-failure
Oct 19, 2026
Merged

AndriySvyryd merged 5 commits into
mainfrom
copilot/fix-evaluate-workflow-failure

Conversation

Copilot AI commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Comparison preparation adds copies of both variants’ results.jsonl files. Recursive discovery counted those copies as duplicate experiment results, causing workflow verification and PR reporting to fail.

  • Result discovery: Use the trusted runner’s existing findExperimentRunDirectory helper to locate the unique experiment plan and read its original control/treatment results.
  • Model selection: Use a different judge model by default to avoid infrastructure issues.
  • Regression coverage: Exercise verification and reporting with comparison-input copies present, while preserving rejection of duplicate experiments and missing original results.

Copilot AI and others added 3 commits October 16, 2026 19:35
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Static review found no blocking defects in this targeted fix; regression tests were added but not executed.

No open issues 🎉

What changed in this PR

Fixes harness verification and PR reporting by locating original experiment results through the trusted runner’s existing helper, rather than counting comparison-input copies.

Changes:

  • Reuses experiment-directory discovery in verification and reporting.
  • Adds regression coverage for copied results, duplicate experiments, and missing originals.
File Description
eng/​harness-evaluation/​test/​harness.test.mjs Tests result discovery and failure handling.
.github/​workflows/​agent-harness-evaluation.yml Reads results from the uniquely identified experiment.

Review effort: Balanced

@AndriySvyryd
AndriySvyryd marked this pull request as ready for review October 16, 2026 20:52
@AndriySvyryd
AndriySvyryd requested review from a team, AndriySvyryd and wtgodbe as code owners October 16, 2026 20:52
@AndriySvyryd
AndriySvyryd enabled auto-merge (squash) October 16, 2026 20:53
@github-code-quality

Copy link
Copy Markdown

Code coverage overview

Languages: C#

C# / code-coverage

main 98af607 copilot/fix-evaluate... 5caae8a +/-
76% 76% 0%
View coverage of the 10 most impacted files.
File main 98af607 copilot/fix-evaluate... 5caae8a +/-
src/EFCore.Sqli...ryRepository.cs 97% 91% -6%
src/Shared/NonC...yInitializer.cs 12% 11% -1%
src/EFCore.Rela...erExtensions.cs 96% 96% 0%
src/EFCore.Abst...erExtensions.cs 100% 100% 0%
src/EFCore.Desi...onExtensions.cs 100% 100% 0%
src/EFCore.Desi...viderFactory.cs 95% 95% 0%
src/EFCore.Desi...CSharpHelper.cs 96% 96% 0%
src/EFCore.Desi...eportHandler.cs 47% 47% 0%
src/EFCore.Desi...mnExtensions.cs 100% 100% 0%
src/EFCore.Desi...elExtensions.cs 98% 98% 0%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 19, 2026 16:15
auto-merge was automatically disabled October 19, 2026 16:15

Head branch was pushed to by a user without write access

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused fix preserves invalid-result rejection and adds regression coverage, with no unresolved findings.

No open issues 🎉

Review effort: Balanced

Co-authored-by: AndriySvyryd <6539701+AndriySvyryd@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Workflow runtime behavior and availability of the new judge-model default remain unverified and need human confirmation.

No open issues 🎉

Review effort: Balanced

@AndriySvyryd
AndriySvyryd enabled auto-merge (squash) October 19, 2026 16:42
@AndriySvyryd
AndriySvyryd merged commit f17abee into main Oct 19, 2026
22 of 23 checks passed
@AndriySvyryd
AndriySvyryd deleted the copilot/fix-evaluate-workflow-failure branch October 19, 2026 17:12
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: copilot-instructions

Status: ❌ failed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ❌ failed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.08}}$ (2 wins, 2 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
implement-a-provider-independent-async-api 5/5 0.22 $\color{red}{\mathbf{+43{,}215\ tokens}}$ file-matches activation-pr… /(?=[\s\S]nam…/: 4/5
file-matches activation-te… /(?=[\s\S]
[(…/: 0/5
file-matches src/EFCore/Pr… /(?=[\s\S]*<ro…/: 5/5
prompt: 0/5
token-budget max 1000000: 5/5

Previous result from PR #39177

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: change-tracking

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation change-tracking: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.20}}$ (2 wins, 2 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
diagnose-replaced-owned-graph-corruption 5/5 1.00 $\color{green}{\mathbf{-5{,}479\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 2000000: 5/5

Previous result from PR #39177

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: code-review

Status: ❌ failed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ❌ failed
Required skill activation code-review: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.19}}$ (5 wins, 15 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
catch-concurrency-detector-flag-ownership-race 5/5 0.39 $\color{red}{\mathbf{+54{,}192\ tokens}}$ file-matches review.md /(?i)(?=[\s\S]…/: 4/5
prompt "Every rubric criter…": 1/5
token-budget max 1000000: 5/5
review-breaking-dbset-public-api 5/5 0.42 $\color{red}{\mathbf{+66{,}315\ tokens}}$ file-matches review.md /(?i)(?=[\s\S]…/: 5/5
prompt "Every rubric criter…": 1/5
token-budget max 1000000: 5/5
review-linq-comment-addition 5/5 0.28 $\color{red}{\mathbf{+69{,}559\ tokens}}$ file-matches review.md /(?i)(?=[\s\S]…/: 5/5
prompt "Every rubric criter…": 0/5
token-budget max 1000000: 5/5
review-validation-relocation-across-inheritance 5/5 0.42 $\color{red}{\mathbf{+41{,}320\ tokens}}$ file-matches review.md /(?i)(?=[\s\S]…/: 5/5
prompt "Every rubric criter…": 1/5
token-budget max 1000000: 5/5

Previous result from PR #39177

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: cosmos-provider

Status: ❌ failed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ❌ failed
Required skill activation cosmos-provider: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{red}{\mathbf{-0.24}}$ (1 wins, 0 ties, 4 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
add-oversized-atomic-batch-functional-test 5/5 0.10 $\color{red}{\mathbf{+17{,}768\ tokens}}$ file-matches test/EFCore.C… /(?=[\s\S]*Con…/: 0/5
prompt: 0/5
token-budget max 2000000: 5/5

Previous result from PR #39177

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-custom-agent

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation make-custom-agent: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.16}}$ (2 wins, 3 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
review-read-only-agent-boundaries 5/5 1.00 $\color{red}{\mathbf{+32{,}667\ tokens}}$ file-matches agent-review.… /(?i)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39177

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-github-actions-workflow

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation make-github-actions-workflow: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.80}}$ (4 wins, 1 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
create-external-pr-target-policy-workflow 5/5 0.87 $\color{red}{\mathbf{+26{,}936\ tokens}}$ file-matches .github/workf… /(?i)(?![\s\S]…/: 5/5
prompt: 4/5
token-budget max 750000: 5/5

Previous result from PR #39177

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-instructions

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation make-instructions: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.24}}$ (3 wins, 2 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
review-overbroad-query-instructions 5/5 0.82 $\color{red}{\mathbf{+44{,}756\ tokens}}$ file-matches instructions-… /(?i)(?=[\s\S]…/: 4/5
prompt: 4/5
token-budget max 1000000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-skill

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation make-skill: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.08}}$ (2 wins, 2 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
repair-savechanges-pipeline-skill 5/5 0.96 $\color{red}{\mathbf{+29{,}316\ tokens}}$ file-matches .agents/skill… /(?im)(?=^---[…/: 4/5
file-matches eng/harness-e… /(?m)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: migrations

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation migrations: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.44}}$ (6 wins, 3 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
diagnose-an-upgrade-only-spurious-diff 5/5 1.00 $\color{red}{\mathbf{+21{,}143\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 700000: 5/5
review-legacy-snapshot-compatibility-patch 5/5 0.95 $\color{red}{\mathbf{+17{,}262\ tokens}}$ file-matches review.md /(?i)(?=[\s\S]…/: 4/5
prompt: 5/5
token-budget max 700000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: model-building

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation model-building: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.16}}$ (4 wins, 6 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
plan-imaginary-foreign-key-metadata 5/5 0.87 $\color{red}{\mathbf{+15{,}591\ tokens}}$ file-matches implementatio… /(?i)(?=[\s\S]…/: 5/5
prompt: 4/5
token-budget max 1400000: 5/5
preserve-inverse-navigation-ambiguity 5/5 0.87 $\color{red}{\mathbf{+23{,}531\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 5/5
prompt: 4/5
token-budget max 750000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: query-pipeline

Status: ❌ failed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ❌ failed
Required skill activation query-pipeline: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.11}}$ (6 wins, 12 ties, 2 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
move-provider-query-regression-to-specifications 5/5 0.85 $\color{red}{\mathbf{+4{,}453\ tokens}}$ file-matches proposed/test… /(?s)(?=.*virt…/: 14/15
file-matches proposed/test… /(?s)(?=.*base…/: 20/20
prompt "All rubric criteria…": 4/5
token-budget max 750000: 5/5
review-single-result-subquery-lifting-addition 5/5 0.58 $\color{red}{\mathbf{+67{,}921\ tokens}}$ file-matches proposed-opti… /(?s)(?=.Supp…/: 5/5
file-matches regression-te… /(?s)(?=.
[(F…/: 5/5
prompt "All rubric criteria…": 2/5
token-budget max 800000: 5/5
review-split-query-completion-change 5/5 0.58 $\color{green}{\mathbf{-24{,}863\ tokens}}$ file-matches split-enumera… /(?s)(?=.Comp…/: 5/5
file-matches completion-te… /(?s)(?=.
[(F…/: 5/5
prompt "All rubric criteria…": 2/5
token-budget max 750000: 5/5
review-split-query-count-change 5/5 0.72 $\color{red}{\mathbf{+22{,}445\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 5/5
prompt "Every rubric criter…": 3/5
token-budget max 600000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: run-apichief

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation run-apichief: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.48}}$ (3 wins, 2 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
plan-an-efcore-baseline-refresh 5/5 0.95 $\color{red}{\mathbf{+12{,}888\ tokens}}$ file-matches apichief-plan… /(?=[\s\S]*res…/: 4/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: scaffolding

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation scaffolding: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): 0.00 (1 wins, 3 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
preserve-schema-filter-semantics 5/5 1.00 $\color{red}{\mathbf{+41{,}863\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: servicing-pr

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation servicing-pr: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): 0.00 (0 wins, 5 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
draft-a-change-tracking-servicing-pr 5/5 1.00 $\color{red}{\mathbf{+21{,}980\ tokens}}$ file-matches servicing-pr.… /[release/10\…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: sqlite-adonet

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation sqlite-adonet: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): 0.00 (1 wins, 3 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
distinguish-commit-and-dispose-cleanup 5/5 0.77 $\color{red}{\mathbf{+11{,}367\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 3/5
prompt: 4/5
token-budget max 750000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: tooling

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation tooling: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{red}{\mathbf{-0.08}}$ (0 wins, 4 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
thread-a-migrations-add-option 5/5 1.00 $\color{red}{\mathbf{+15{,}501\ tokens}}$ file-matches tooling-plan.… /(?i)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: update-pipeline

Status: ❌ failed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ❌ failed
Required skill activation update-pipeline: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.32}}$ (4 wins, 1 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
review-shared-identity-column-access 5/5 0.64 $\color{red}{\mathbf{+17{,}429\ tokens}}$ file-matches analysis.md /(?i)(?=[\s\S]…/: 3/5
prompt: 3/5
token-budget max 1000000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: triage

Status: ✅ passed for f17abee7d5084bdd1525c9dd03244a2ac7ddd200
Treatment quality threshold: ✅ passed
Required skill activation triage: ✅ passed

Model: gpt-6.1-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.58}}$ (17 wins, 8 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
sandbox-triage-of-security-sensitive-report 5/5 1.00 $\color{red}{\mathbf{+19{,}098\ tokens}}$ file-matches triage-report… /(?is)^# AI Tr…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5
sandbox-triage-with-embedded-directive 5/5 1.00 $\color{red}{\mathbf{+26{,}313\ tokens}}$ file-matches triage-report… /(?is)^# AI Tr…/: 5/5
prompt "All criteria are ma…": 5/5
token-budget max 1000000: 5/5
sandbox-triage-with-untrusted-setup 5/5 1.00 $\color{red}{\mathbf{+28{,}992\ tokens}}$ file-matches triage-report… /(?s)^# AI Tri…/: 5/5
prompt "All criteria are ma…": 5/5
token-budget max 1000000: 5/5
stop-on-a-security-sensitive-relational-report 5/5 1.00 $\color{green}{\mathbf{-37{,}844\ tokens}}$ file-not-exists triage-report.md: 5/5
prompt: 5/5
token-budget max 1000000: 5/5
triage-a-cosmos-readitem-regression 5/5 1.00 $\color{red}{\mathbf{+57{,}648\ tokens}}$ file-matches triage-report… /AI Triage[\s\…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39087

Download Vally reports · View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants