Skip to content

docs(metric-benchmark): design registered-agent Helix execution - #185

Merged
mstaats-nvidia merged 24 commits into
mainfrom
explore/metric-benchmark-helix-architecture
Nov 13, 2026
Merged

mstaats-nvidia merged 24 commits into
mainfrom
explore/metric-benchmark-helix-architecture

Conversation

@mstaats-nvidia

@mstaats-nvidia mstaats-nvidia commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Define a reversible Helix benchmark workflow: publish agents and unchanged tasks separately, execute one registered agent against a pinned taskset, and download standard results and exact sources.

The architecture specifies four proposed tools: publication, execution, collection with verification/import, and source download. It includes an SVG resource flow, configuration filenames and ownership, and command examples. Tasks use pinned revisions; agent registrations are mutable, so the design requires content-derived agent names, content checks and a separately retained remote source copy. That retention mechanism remains a downstream requirement to verify. Execution waits for image builds before submitting and saves a retry key before the submission request.

The example directory contains only illustrative YAML configurations and a README. Experiment, case, agent/task manifest, analysis, publication, execution and dummy client files show how the configuration fits together. Referenced agent bundles, tasks and analysis data are deliberately absent. The README explains how case/agent/task names map to registered Helix IDs through references.json, how execution and downloads use that mapping, and what changes from local experiments. No real secrets or generated run evidence are included.

This PR contains architecture documentation and YAML examples only. Proposed commands and registered-agent/signing support belong downstream. The example is intentionally unsigned and not runnable. No merge or downstream restacking is included.

Validation: YAML syntax, YAML-only example inventory, local documentation links, copyright headers, whitespace and DCO hook passed. For the latest documentation revision, copyright headers, whitespace, SVG XML parsing and DCO checks passed; the SVG layout is unchanged from its previous visual inspection. No live Helix, model or container execution. Full tests and broad runtime/dependency checks were not run under the requested focused-check scope; dependencies are unchanged.

Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
…sion

Signed-off-by: mstaats <mstaats@nvidia.com>
@mstaats-nvidia
mstaats-nvidia marked this pull request as ready for review November 10, 2026 13:03
@mstaats-nvidia
mstaats-nvidia requested a review from a team as a code owner November 10, 2026 13:03
@mstaats-nvidia
mstaats-nvidia merged commit 3093ba6 into main Nov 13, 2026
23 checks passed
@mstaats-nvidia
mstaats-nvidia deleted the explore/metric-benchmark-helix-architecture branch November 13, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants