Skip to content

feat(metric-benchmark): execute one published agent on Helix - #162

Draft
mstaats-nvidia wants to merge 26 commits into
explore/metric-benchmark-helix-preparefrom
explore/metric-benchmark-helix-runner
Draft

mstaats-nvidia wants to merge 26 commits into
explore/metric-benchmark-helix-preparefrom
explore/metric-benchmark-helix-runner

Conversation

@mstaats-nvidia

@mstaats-nvidia mstaats-nvidia commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Implements one-agent execution from #185 using the portable references published by #173. helix_execute.py selects one prepared agent and case taskset, requests missing image builds, verifies registered contents, and saves its retry key and exact request before submission. Retrying the same receipt reuses the accepted run; an intentional new run requires a new receipt.

Execution works without the original local agent/task directories. Run receipts retain the actual resolved configuration and source hashes. The internal native collector preserves full artifacts and job specifications, checks complete matrix accounting, keeps failed trials, and publishes evidence atomically. #174 adds the public command that collects and verifies directly into the standard results format.

Stack: #185 → #173 → #162 → #174. Shared client settings control the deployment; execution settings control dispatcher, concurrency and sandbox resources. Agent behavior and task selection remain in the prepared experiment.

Validation: full CI passed on Python 3.12/3.13 (4,496 tests per version, 110 optional/environment skips). Live per-agent submissions worked without original upload directories; identical replay left receipts unchanged and server listing showed exactly one run per retry key. Pending collection returned exit 2 without a final output directory. Completed runs retained recursion-limit, agent-timeout and sandbox-start errors as unavailable observations. A serial Oracle retry reproduced sandbox startup failure before agent execution; this is not a clean performance qualification.

See the [`run_evals.py` command guide](run-evals.md) for arguments, the output
layout, completed-job reuse, interrupted-job recovery, and the Python API.

### Helix exploration

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would link the Helix repo if not done already (either here or somewhere higher level). Maybe longer-term a link to compass/eval-author inside helix, etc

Base automatically changed from feat/metric-benchmark-stage-config to main October 30, 2026 21:36
@mstaats-nvidia
mstaats-nvidia force-pushed the explore/metric-benchmark-helix-runner branch 2 times, most recently from 7c00a7c to 19f66d2 Compare November 3, 2026 19:48
@mstaats-nvidia
mstaats-nvidia force-pushed the explore/metric-benchmark-helix-runner branch from 19f66d2 to a54bac9 Compare November 4, 2026 18:12
@mstaats-nvidia mstaats-nvidia changed the title feat(metric-benchmark): prototype Helix submission and evidence collection feat(metric-benchmark): execute and collect native Helix experiments Nov 4, 2026
@mstaats-nvidia
mstaats-nvidia changed the base branch from main to explore/metric-benchmark-helix-prepare November 4, 2026 18:13
@mstaats-nvidia
mstaats-nvidia force-pushed the explore/metric-benchmark-helix-runner branch 2 times, most recently from 21ec144 to ffc0a66 Compare November 6, 2026 10:27
@mstaats-nvidia
mstaats-nvidia added this pull request to stack #178 November 6, 2026 11:35
Signed-off-by: mstaats <mstaats@nvidia.com>
@mstaats-nvidia mstaats-nvidia changed the title feat(metric-benchmark): execute and collect native Helix experiments feat(metric-benchmark): run registered agents against Helix tasksets Nov 8, 2026
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
@mstaats-nvidia
mstaats-nvidia force-pushed the explore/metric-benchmark-helix-runner branch from 4304b2f to 907daee Compare November 9, 2026 11:51
@mstaats-nvidia mstaats-nvidia changed the title feat(metric-benchmark): run registered agents against Helix tasksets feat(metric-benchmark): execute one published agent on Helix Nov 9, 2026
@mstaats-nvidia
mstaats-nvidia force-pushed the explore/metric-benchmark-helix-runner branch from 907daee to 10996cf Compare November 9, 2026 12:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants