Repository navigation
feat(metric-benchmark): execute one published agent on Helix - #162
Draft
mstaats-nvidia wants to merge 26 commits into
Draft
mstaats-nvidia wants to merge 26 commits into
mstaats-nvidia wants to merge 26 commits into
Conversation
asutermo
reviewed
Oct 30, 2026
| See the [`run_evals.py` command guide](run-evals.md) for arguments, the output | ||
| layout, completed-job reuse, interrupted-job recovery, and the Python API. | ||
|
|
||
| ### Helix exploration |
Collaborator
There was a problem hiding this comment.
I would link the Helix repo if not done already (either here or somewhere higher level). Maybe longer-term a link to compass/eval-author inside helix, etc
asutermo
approved these changes
Oct 30, 2026
mstaats-nvidia
force-pushed
the
explore/metric-benchmark-helix-runner
branch
2 times, most recently
from
November 3, 2026 19:48
7c00a7c to
19f66d2
Compare
mstaats-nvidia
force-pushed
the
explore/metric-benchmark-helix-runner
branch
from
November 4, 2026 18:12
19f66d2 to
a54bac9
Compare
mstaats-nvidia
changed the base branch from
main
to
explore/metric-benchmark-helix-prepare
November 4, 2026 18:13
mstaats-nvidia
force-pushed
the
explore/metric-benchmark-helix-runner
branch
2 times, most recently
from
November 6, 2026 10:27
21ec144 to
ffc0a66
Compare
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
…sion Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
explore/metric-benchmark-helix-runner
branch
from
November 9, 2026 11:51
4304b2f to
907daee
Compare
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
explore/metric-benchmark-helix-runner
branch
from
November 9, 2026 12:15
907daee to
10996cf
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements one-agent execution from #185 using the portable references published by #173.
helix_execute.pyselects one prepared agent and case taskset, requests missing image builds, verifies registered contents, and saves its retry key and exact request before submission. Retrying the same receipt reuses the accepted run; an intentional new run requires a new receipt.Execution works without the original local agent/task directories. Run receipts retain the actual resolved configuration and source hashes. The internal native collector preserves full artifacts and job specifications, checks complete matrix accounting, keeps failed trials, and publishes evidence atomically. #174 adds the public command that collects and verifies directly into the standard results format.
Stack: #185 → #173 → #162 → #174. Shared client settings control the deployment; execution settings control dispatcher, concurrency and sandbox resources. Agent behavior and task selection remain in the prepared experiment.
Validation: full CI passed on Python 3.12/3.13 (4,496 tests per version, 110 optional/environment skips). Live per-agent submissions worked without original upload directories; identical replay left receipts unchanged and server listing showed exactly one run per retry key. Pending collection returned exit 2 without a final output directory. Completed runs retained recursion-limit, agent-timeout and sandbox-start errors as unavailable observations. A serial Oracle retry reproduced sandbox startup failure before agent execution; this is not a clean performance qualification.