Skip to content

[5/5] fix(health): include invocation-owned token usage - #3872

Merged
giuliolovisotto merged 2 commits into
glovisotto/opencode-child-session-usagefrom
glovisotto/health-compaction-accounting
Oct 6, 2026
Merged

giuliolovisotto merged 2 commits into
glovisotto/opencode-child-session-usagefrom
glovisotto/health-compaction-accounting

Conversation

@giuliolovisotto

@giuliolovisotto giuliolovisotto commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Compaction summaries can contribute to response usage without becoming policy turns. The rollout token-total check previously summed only turn-bound calls and incorrectly reported those summaries as discrepancies. Compare response totals against the existing deduplicated invocation-or-turn-owned call view, which includes exactly owned compaction calls and still excludes unowned auxiliary calls.

This intentionally advances only token-total accounting beyond the turn-only boundary retained in #2885. Successful-task-call checks and structural turn checks keep their existing scope. Missing or ambiguous references, missing token counts, and the existing incomplete-scope gap remain conservative; real total mismatches still produce findings. The documentation describes the revised contract. The change affects health accounting, not generated answers or benchmark rewards.

Stacked on #3871, above #3856 / #3764 / #3711. This focused follow-up does not need a separate issue.

Validation:

  • 96 focused tests pass: 71 health tests and 25 sandbox-agent tests.
  • New regressions cover invocation-only compaction accounting, duplicate references, unowned-call exclusion, missing captures and genuinely incorrect totals. The compaction regression fails against the parent implementation.
  • pre-commit run --all-files passes; Fern validation reports zero errors (authenticated redirect validation was skipped).
  • Ran the actual Gym health-check CLI at the prior tip bbe2d7eb8d3d9bd702ae5f1416b1c6f335b9f550 over 2,193 retained records, saving new reports. On untouched records, all three compaction warnings clear. The broader ownership view also exposes 16 pre-existing unfinished-call usage discrepancies that turn-only accounting could not see.
  • A separately saved producer-usage replay for parent [4/5] fix(opencode): include child-session usage #3871 clears all 140 child-session mismatches. Final combined result: 2,175 healthy, 17 unhealthy, 1 unobserved; zero trajectory/capture mismatch findings and zero check execution errors. Remaining findings are the 16 unfinished native/captured-usage discrepancies and one recovered server-error incident; one sandbox failure lacks canonical evidence.
  • The child replay used retained native root exports plus capture-derived child token metadata passed through the production reader/converter: original child SQLite snapshots were unavailable. Original records/reports remain intact, no checks are ignored, and the parent PR separately has a live parent/child model smoke. This is not a new benchmark rollout or a score-parity claim.
  • Enabled GitHub checks pass at this head. Draft-gated automated review/integration checks were skipped; no unresolved review threads were present when checked.

Parent header-property update

Restacked on #3711, which now obtains the optional capture header from a harness-owned property. The previous run-level header option is removed. OpenCode declares its existing binary header in its own package; no binary rebuild or selection change is required. The focused property/startup/correlation suite passes 397 tests and all-files pre-commit passes. A fresh live attempt retained IDs but failed in the upstream proxy, so successful new deployment validation is not claimed. Historical deployment/replay results above remain tied to their original commits.

The restacked accounting tip a94028067d4861a14ccceb135e72fda0d554addd passes 730 selected tests, including both adapters, collection, shared correlation and health. No saved-run health replay was repeated for this wiring-only change.

Rebase validation (2026-10-02)

Rebased the five-PR stack onto main at 4f95540cc61c2ad783312d0cdb48d8a7c95c30e4, preserving the new capture execution metadata and failed-call health checks. Each PR targets its immediate parent.

At the final tip 00f19f01a18097cdbe7504de691090505effcbd9: 918 selected tests pass, all-files pre-commit passes, and Fern validation reports zero errors and one warning. Range-diff confirms only conflict-context adaptations; all nine commits retain DCO sign-off and SSH signatures. Historical deployment and saved-run replay results above remain tied to their original commits; no new model run or full saved-run health replay was performed for this rebase.

Stack: main → #3711 → #3764 → #3856 → #3871 → #3872

@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/health-compaction-accounting branch from bbe2d7e to 5259407 Compare October 1, 2026 13:47
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/opencode-child-session-usage branch from 1c126ef to 73dd2ca Compare October 1, 2026 13:48
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/opencode-child-session-usage branch from 73dd2ca to 76bc108 Compare October 1, 2026 14:31
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/health-compaction-accounting branch 2 times, most recently from a940280 to 00f19f0 Compare October 2, 2026 12:13
@giuliolovisotto giuliolovisotto changed the title fix(health): include invocation-owned token usage [5/5] fix(health): include invocation-owned token usage Oct 2, 2026
@giuliolovisotto
giuliolovisotto marked this pull request as ready for review October 2, 2026 12:33
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 2, 2026 12:33
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

@yaoyu-33 yaoyu-33 added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 2, 2026
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/health-compaction-accounting branch from 00f19f0 to fb9a2f3 Compare October 5, 2026 08:14
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/opencode-child-session-usage branch from f086638 to fcf08f6 Compare October 5, 2026 08:14
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 5, 2026 08:14
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 5, 2026
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 5, 2026
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/health-compaction-accounting branch from c80d74c to ccf9f99 Compare October 5, 2026 16:43
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/opencode-child-session-usage branch from 0d2dcac to edd7861 Compare October 5, 2026 16:43
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 5, 2026 16:43
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/opencode-child-session-usage branch from edd7861 to 6823ead Compare October 6, 2026 07:24
@giuliolovisotto
giuliolovisotto force-pushed the glovisotto/health-compaction-accounting branch from ccf9f99 to 043e643 Compare October 6, 2026 07:24
@giuliolovisotto
giuliolovisotto merged commit ab0f376 into glovisotto/opencode-child-session-usage Oct 6, 2026
47 checks passed
@giuliolovisotto
giuliolovisotto deleted the glovisotto/health-compaction-accounting branch October 6, 2026 09:10
giuliolovisotto added a commit that referenced this pull request Oct 6, 2026
OpenCode sandboxed runs can spawn child sessions, but response usage
previously summed only the root export. Include persisted descendant
message usage once, including nested children and reasoning details,
while excluding the separately counted root and unrelated sessions.
Response content remains the root export.

The adapter reuses its consistent SQLite snapshot and also collects
usage when detailed observability is disabled. Failed database reads
preserve the root-export fallback and diagnostics. This adds database
snapshot work for runs with observability disabled; no model prompts,
binaries, or benchmark scoring change.

Stacked on #3856; this narrowly scoped accounting follow-up does not
need a separate issue. Compaction health-check scope is a separate
stacked change.

Validation:
- 25 sandbox-agent tests pass, including children/grandchildren,
unrelated-session exclusion, observability enabled/disabled, live WAL
snapshots, custom database paths and failure fallback. The new usage
regression assertion fails against the parent implementation.
- `pre-commit run --all-files` passes.
- A small live-model smoke through the actual adapter, with local
exec/download transport, made three calls across parent and child
sessions. Aggregated prompt/output usage exactly matches captured
provider usage and reasoning detail is preserved. This does not validate
a new sandbox-provider deployment.
- At stacked tip `bbe2d7eb8d3d9bd702ae5f1416b1c6f335b9f550` (#3872), the
actual Gym health-check CLI checked 2,193 retained records and a
separately saved accounting replay. All 140 child-session mismatches
clear in the replay (162 child sessions, 938 calls), with every
corrected record healthy. Original records/reports remain intact.
- Replay limitation: original child SQLite snapshots had been deleted.
Root usage was verified against retained native exports; child usage was
reconstructed from captured token metadata and exact identities, then
passed through the production SQLite reader/converter. This is
capture-derived accounting replay, not independent recovery of original
child SQLite. The live-model smoke above separately validates the
producer.
- Combined result with #3872: 2,175 healthy, 17 unhealthy, 1 unobserved.
Remaining findings are 16 pre-existing unfinished native/captured-usage
discrepancies and one recovered server-error incident; one sandbox
failure lacks canonical evidence. These failures are preserved, not
suppressed.
- Enabled GitHub checks pass at this head. Draft-gated automated
review/integration checks were skipped; no unresolved review threads
were present when checked.

## Parent header-property update

Restacked on #3711, which now obtains the optional capture header from a
harness-owned property. The previous run-level header option is removed.
OpenCode declares its existing binary header in its own package; no
binary rebuild or selection change is required. The focused
property/startup/correlation suite passes 397 tests and all-files
pre-commit passes. A fresh live attempt retained IDs but failed in the
upstream proxy, so successful new deployment validation is not claimed.
Historical deployment/replay results above remain tied to their original
commits.

The restacked accounting tip `a94028067d4861a14ccceb135e72fda0d554addd`
passes **730 selected tests**, including both adapters, collection,
shared correlation and health. No saved-run health replay was repeated
for this wiring-only change.

## Rebase validation (2026-10-02)

Rebased the five-PR stack onto `main` at
`4f95540cc61c2ad783312d0cdb48d8a7c95c30e4`, preserving the new capture
execution metadata and failed-call health checks. Each PR targets its
immediate parent.

At the final tip `00f19f01a18097cdbe7504de691090505effcbd9`: **918
selected tests pass**, all-files pre-commit passes, and Fern validation
reports zero errors and one warning. Range-diff confirms only
conflict-context adaptations; all nine commits retain DCO sign-off and
SSH signatures. Historical deployment and saved-run replay results above
remain tied to their original commits; no new model run or full
saved-run health replay was performed for this rebase.

Stack: main → #3711 → #3764 → #3856 → #3871 → #3872

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
giuliolovisotto added a commit that referenced this pull request Oct 6, 2026
OpenCode saves assistant replies separately from Gym's captured model
calls. Ship a JavaScript plugin that uses stock OpenCode 1.17.11's event
and request-header hooks to send each persisted assistant-message ID to
Gym, allowing the parent correlation change to join the two records
without a custom binary.

The sandboxed adapter uploads the bundled plugin outside the task
repository and appends its file URL to OpenCode's plugin list when
model-call capture is enabled. Existing user plugins are preserved. The
plugin matches session, parent message, agent and model, retains IDs
across retries, includes compaction replies, and omits the header for
title calls or ambiguous ownership.

Restore the existing stock-binary paths and remove this PR's
patched-build manifest. The configured OpenCode version remains 1.17.11.
No new configuration switch or binary rebuild is required. This targets
the classic runner; V2 and benchmark-score parity are outside the
validation scope.

Stack: main → #3711 → #3764 → **#3856** → #3871 → #3872. A separate
issue is unnecessary for this producer follow-up to #3711.

Validation:
- 21 sandboxed-adapter tests pass, including automatic upload/config
wiring and preserving existing plugins; four JavaScript hook tests also
pass.
- 721 selected tests pass at the restacked accounting tip, covering both
adapters, collection, trajectory construction, shared server wiring and
health. Six previously stale accounting assertions were reproduced on
the previous tip and corrected separately in #3872.
- `pre-commit run --all-files` passes. Commits carry DCO sign-off and
SSH signatures.
- Fresh Linux x86-64 glibc and musl Docker checks use the actual adapter
to upload/load the plugin with official stock 1.17.11 binaries. Each
scripted case links all three HTTP attempts to two saved replies,
including a retry and tool continuation; output/reasoning totals
reconcile.
- One fresh live-model call through the adapter and capture middleware
preserves the exact reply ID, response text and token totals; finalized
Gym health is clean. The model did not follow the literal-response
instruction, so this is plumbing validation, not a task-success or
benchmark-score claim.
- Earlier Linux stress validation of the same plugin logic covers 42
requests, child sessions, concurrent retries, two compactions and
failure/abort recovery. Production Gym replay retains compaction IDs and
exact links. Two pre-stream failure/abort records retain explicit
unmatched-turn gaps; that behavior is unchanged.

These checks use Docker locally, with x86-64 emulation on an ARM64 host.
The stress replay is distinct from live Gym HTTP capture. No full
benchmark or cluster/S3 deployment of the plugin is claimed. Historical
validation of the patched binaries is not validation of this revision.

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
giuliolovisotto added a commit that referenced this pull request Oct 6, 2026
Stacked on #4041. Validate each piece of evidence once at its designated
saved location, with JSON Schema for fields and explicit semantic checks
for relationships. Missing collections block dependent checks without
invalidating unrelated evidence.

- Keep call, step, tool, grading and sandbox requirements; resolve
invocation/call references against canonical targets. All supplied
reference identifiers must agree, and ambiguous targets fail.
- Replace blanket Python-model validation and cross-copy reconciliation.
Do not repeat Gym-owned uniqueness, clock-order or provider-usage
normalization checks.
- Retain independent comparisons with scripted exchanges, response
metadata, token counts and tool outcomes. A behavioral failure does not
erase artifact passes.
- Emit one table per harness, with individual check rows grouped as
Schema, Semantic and Behavioral, scenario columns, and compact status
symbols. Process status stays separate. Missing normal rollouts leave
artifact checks unassessed.

The profile is `gym-p0/v3`. Check IDs, report format and authoritative
locations change; regenerate existing reports. TE labels only group
results. `ng_trajectory` supplies calls, steps, invocations and tool
executions; sandbox/compaction observations retain their explicitly
designated source. Capture sidecars are provenance only. Benchmark
execution and scoring code are unchanged.

Validation:
- Focused checker/runner suite: **253 passed, 2 skipped** (`jj`
unavailable).
- `pre-commit run --all-files`: passed.
- Fern check: zero errors, one warning.
- Fresh scripted runs on main `b0e3fa6cf` plus this checker stack for
simple_agent, SciCode, OpenCode, Terminus2, mini-SWE, Hermes, NOOA,
Codex, Pi and Stirrup. OpenCode is also tested separately with
observability stack tip `ccf9f997b` (#3872). Failure sidecars leave
artifact checks unassessed.
- Local additional-driver/checker suite: **272 passed, 2 skipped**.
Combined current OpenCode stack, producer/capture and checker tests:
**625 passed, 2 skipped**.

Controlled-endpoint tests execute shell commands and inspect actual
model/tool/verifier exchanges. These scripted runs do not qualify
production benchmarks. Stirrup also ran all seven scenarios in
Lima/Apptainer with a small shell-test image: six normal rollouts and
one terminal-error failure record. The six normal rollouts retain
model-call evidence but lack invocation, step and tool records. This
uses a local native-tool driver; it does not qualify the full GDPval
environment. Full server CI was not run locally.

Follow-up to #3766 and #4041; no separate issue is needed for this
focused integration.

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
sylvesterkaczmarek pushed a commit to sylvesterkaczmarek/Gym that referenced this pull request Oct 6, 2026
…NVIDIA-NeMo#3711)

## What does this PR do?

OpenCode can make several HTTP attempts for one saved assistant message,
including retries; compaction creates a separate summary message. Gym
records these calls but needs an exact link to the saved messages. This
focused follow-up to NVIDIA-NeMo#3363 supplies that link; a separate issue is
unnecessary.

- Retain the client's saved assistant-message ID alongside the session
ID. Each physical HTTP attempt still receives its own Gym
`model_call_id`.
- Match calls to turns or compaction observations by saved message,
session, model server and unique invocation ownership. The shared
matcher does not branch on the harness source or guess from
content/timing. Ambiguous or incomplete evidence remains unbound with
explicit gaps.
- Keep native artifact parsing and summary identification in the
OpenCode adapters. Compaction calls retain ownership outside policy
turns. Requests without a saved assistant ID remain invocation-owned.
- Discover the sandbox database with `opencode db path` before taking
its consistent SQLite snapshot, including release-channel-specific
filenames; preserve lookup/snapshot errors.

## Header ownership

Capture accepts an optional `assistant_message_header: bytes | None`,
defaulting to `None`. OpenCode declares its header on its own package;
the sandboxed adapter shares that declaration. Model-server startup
reads the property from the configured harnesses referencing that server
and passes it to capture, without importing the agent app. Harnesses
sharing a model server must agree on any declared non-None header.

There is no run-level header setting, regex validation, extractor plugin
or entry-point registry. Existing patched binaries work unchanged.
Harnesses using response IDs for correlation, such as `simple_agent`, do
not need this header. The generic model module contains no OpenCode
header literal or OpenCode-specific matching branch.

## Validation

- At `f300b0da8b0ca454f74148c3ae7f42165aadcd61`: **397 focused tests
pass**, including actual model-server startup for both OpenCode
adapters, custom headers, None, scope/ownership ambiguity, retries,
compactions and streaming/error capture. All-files pre-commit passes.
The removed documentation section is no longer in the diff against the
original base.
- A fresh bounded live-model attempt using the patched producer reached
capture with both IDs, but ended in an upstream-proxy error. Failed-call
IDs were retained; a successful fresh live completion is not claimed for
this revision.
- Earlier producer/deployment validation remains tied to its historical
commits, as documented in the dependent PRs. It covered native
reconstruction, retries, child sessions and compaction. This
header-wiring change does not rerun those deployments or establish score
parity.

The restacked accounting tip `a94028067d4861a14ccceb135e72fda0d554addd`
passes **730 selected tests**, including both adapters, collection,
shared correlation and health. No saved-run health replay was repeated
for this wiring-only change.

## Compatibility

New fields are optional. Binaries that do not emit the saved-message
header remain usable with invocation-level attribution. Existing
response-ID correlation remains intact. This changes evidence
capture/linking, not prompts, generated answers, rewards, default binary
selection or health rules. Richer evidence can make existing checks
evaluable. Classic OpenCode is covered; a V2 reader remains outside
scope.

Compaction evidence remains on `ng_agent_observations`. New diagnostic
codes use `assistant_message_call_*`; older artifacts remain readable.
No end-user configuration is introduced; the optional harness property
is described above. Commits have DCO sign-off and cryptographic
signatures. The PR remains a draft; skipped or absent integration/review
CI is not reported as passing.

## Checklist

- [x] I have read the [contributing
guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md).
- [x] The change is focused; unrelated "drive-by" edits are tracked as
separate issues/PRs.
- [x] Tests added or updated and pass locally, or N/A for docs-only /
non-code changes (so CI unit/server checks pass when applicable).
- [x] Documentation added or updated, or N/A with justification.
- [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so
CI lint/format checks pass).
- [x] New source files include the required Apache-2.0 SPDX header.
- [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check
passes).

## Rebase validation (2026-10-02)

Rebased the five-PR stack onto `main` at
`4f95540cc61c2ad783312d0cdb48d8a7c95c30e4`, preserving the new capture
execution metadata and failed-call health checks. Each PR targets its
immediate parent.

At the final tip `00f19f01a18097cdbe7504de691090505effcbd9`: **918
selected tests pass**, all-files pre-commit passes, and Fern validation
reports zero errors and one warning. Range-diff confirms only
conflict-context adaptations; all nine commits retain DCO sign-off and
SSH signatures. Historical deployment and saved-run replay results above
remain tied to their original commits; no new model run or full
saved-run health replay was performed for this rebase.

Stack: main → NVIDIA-NeMo#3711 → NVIDIA-NeMo#3764 → NVIDIA-NeMo#3856 → NVIDIA-NeMo#3871 → NVIDIA-NeMo#3872

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
yuhezhang-ai pushed a commit to larroy/Gym that referenced this pull request Oct 6, 2026
…ction calls token accounting, binary patch for observability (NVIDIA-NeMo#3764)

## Consolidated scope

**This PR consolidates PRs 2–5 of the original five-PR stack.** The
author squash-merged the dependent PRs into this branch, preserving the
complete implementation previously reviewed at the stack tip. NVIDIA-NeMo#3711
(1/5: assistant-message/model-call correlation) has landed separately;
this PR now targets `main`.

| Original stack entry | Included change |
| --- | --- |
| 2/5 — NVIDIA-NeMo#3764 | Include reasoning tokens in OpenCode output usage |
| 3/5 — NVIDIA-NeMo#3856 | Enable the assistant-message correlation plugin with
stock OpenCode |
| 4/5 — NVIDIA-NeMo#3871 | Include child-session usage and move child-message
SQLite reads off the event loop |
| 5/5 — NVIDIA-NeMo#3872 | Include invocation-owned calls in token accounting,
covering compaction calls |

## Behavior

- **Reasoning usage:** OpenCode reports visible output and reasoning
separately. Both Gym OpenCode adapters include both in `output_tokens`
and preserve the reasoning breakdown. For example, 3,509 visible tokens
plus 4,396 reasoning tokens yields 7,905 output tokens, including 4,396
reasoning tokens in the detail field.
- **Assistant correlation:** the sandboxed adapter uploads and
configures the bundled JavaScript plugin, which supplies persisted
assistant-message IDs for the capture/linking support in NVIDIA-NeMo#3711. This
uses the stock OpenCode binary; no patched binary or rebuild is
required. Existing runtime plugins remain supported.
- **Child-session usage:** response totals include descendant sessions
even when detailed observation collection is disabled. Child-message
SQLite reads run through `asyncio.to_thread`.
- **Compaction accounting:** the health checker reconciles usage against
deduplicated policy calls owned by invocations as well as turns, so
compaction calls without a saved visible turn are included.

These changes correct recorded usage, correlation evidence and health
accounting; they do not change the model sampling policy or reward
calculation. No separate issue is needed for these focused integration
fixes found during validation.

## Validation

At consolidated head `f9ea16597a9263a7f3717e875e78098cba0cbd2b`:

- 1,187 selected Python tests passed across capture, configuration,
server/client, rollout collection, observability, health and both
OpenCode adapters.
- All four JavaScript correlation-plugin tests and `pre-commit run
--all-files` passed.
- [Full GitHub CI
passed](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37444862493),
including core tests and all eight server-test shards. The merge commit
has DCO sign-off and a verified SSH signature.
- The consolidated source tree matched the previously validated five-PR
tip exactly before integrating current `main`. Conflict resolution
preserved the OpenCode implementation and health changes unchanged.

Prior runtime validation remains applicable to the unchanged
implementation: a focused live sandbox smoke with stock OpenCode 1.17.11
and both runtime plugins produced three successful calls, three exact
assistant links and two completed invocations (parent and child). Usage
reconciled, the SQLite read ran off the event loop, and Gym health
reported one healthy rollout with zero findings. That smoke did not
exercise compaction, retries or nonzero reasoning. Earlier
reasoning-accounting validation separately reconciled 119 captured calls
across two healthy SWE-Pro rollouts; it did not exercise child sessions
or compaction. Compaction accounting is covered by retained-artifact
validation and regression tests.

No new live benchmark was run for the consolidation/conflict resolution,
and these checks do not establish benchmark score parity.

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
giuliolovisotto added a commit that referenced this pull request Oct 7, 2026
Stacked on #4041. Validate each piece of evidence once at its designated
saved location, with JSON Schema for fields and explicit semantic checks
for relationships. Missing collections block dependent checks without
invalidating unrelated evidence.

- Keep call, step, tool, grading and sandbox requirements; resolve
invocation/call references against canonical targets. All supplied
reference identifiers must agree, and ambiguous targets fail.
- Replace blanket Python-model validation and cross-copy reconciliation.
Do not repeat Gym-owned uniqueness, clock-order or provider-usage
normalization checks.
- Retain independent comparisons with scripted exchanges, response
metadata, token counts and tool outcomes. A behavioral failure does not
erase artifact passes.
- Emit one table per harness, with individual check rows grouped as
Schema, Semantic and Behavioral, scenario columns, and compact status
symbols. Process status stays separate. Missing normal rollouts leave
artifact checks unassessed.

The profile is `gym-p0/v3`. Check IDs, report format and authoritative
locations change; regenerate existing reports. TE labels only group
results. `ng_trajectory` supplies calls, steps, invocations and tool
executions; sandbox/compaction observations retain their explicitly
designated source. Capture sidecars are provenance only. Benchmark
execution and scoring code are unchanged.

Validation:
- Focused checker/runner suite: **253 passed, 2 skipped** (`jj`
unavailable).
- `pre-commit run --all-files`: passed.
- Fern check: zero errors, one warning.
- Fresh scripted runs on main `b0e3fa6cf` plus this checker stack for
simple_agent, SciCode, OpenCode, Terminus2, mini-SWE, Hermes, NOOA,
Codex, Pi and Stirrup. OpenCode is also tested separately with
observability stack tip `ccf9f997b` (#3872). Failure sidecars leave
artifact checks unassessed.
- Local additional-driver/checker suite: **272 passed, 2 skipped**.
Combined current OpenCode stack, producer/capture and checker tests:
**625 passed, 2 skipped**.

Controlled-endpoint tests execute shell commands and inspect actual
model/tool/verifier exchanges. These scripted runs do not qualify
production benchmarks. Stirrup also ran all seven scenarios in
Lima/Apptainer with a small shell-test image: six normal rollouts and
one terminal-error failure record. The six normal rollouts retain
model-call evidence but lack invocation, step and tool records. This
uses a local native-tool driver; it does not qualify the full GDPval
environment. Full server CI was not run locally.

Follow-up to #3766 and #4041; no separate issue is needed for this
focused integration.

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 043e6439 Deployed Oct 6, 2026 by copy-pr-bot[bot] via release / finalize / notify #4395
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants