[jessbuhl]

Saved audits need to retain which original trace records belong to selected suite runs. Add an optional selection-manifest reference to the existing lifecycle so changes to original sources, selectors, or mappings reopen check review. Prepared traces, conversion receipts, outcomes, and native results remain separate measurement evidence. Existing lifecycle records without a manifest still validate.

The guided flow uses the existing trace preparation and native measurement/report commands. On resume, it reuses the manifest path saved in the lifecycle. New manifests use .eval-author/.working/audit/selection-manifest.json, outside source bounds, candidates, and exclusions. Missing or ambiguous suite/task/run mappings stay pending until the user resolves them. Measurement receives explicit task IDs, run IDs, and evidence classes, and aggregation uses only the selected coverage files. Failed preparation stays visible in retained outcomes and keeps measurement partial.

Human-facing documentation calls the evidence traces. Agent instructions retain the format-specific preparation commands required by the current measurement CLI. No scripts or dependencies are added.

Related: ASE-1292.

Validation

• Full local suite on Python 3.13.15 at 99353c4: 3,286 passed, 14 skipped.

• Ten synthetic chain cases cover ATIF, Gym, and MLflow sources, unresolved mappings across restarts, explicit run IDs, unrelated measurements in a shared output directory, changed inputs, and a selected Gym row that fails conversion. They exercise the real converters and native measurement/report commands.

• Ruff lint and formatting, Linux type checks for Python 3.12 and 3.13, runtime policy, copyright headers, licenses with OSV-Scanner 2.3.3, whitespace, and DCO checks passed.

• Optional native Gym, Docker, Ratchet, platform-specific, and live-execution checks were skipped. Two command-isolation tests also skip when an installed Gym command would shadow their fixture. Those skips do not establish compatibility for the affected integrations.

• Deterministic manifest-path follow-up: all ten trace-chain cases passed with the existing working-path and atomic-write helpers. This follow-up changes guidance and fixture storage only.

[aleckhoury]

Approving to unblock the stack; the items below should still be addressed (here or in a follow-up).

Most of the ASE-1292 verification list is covered at the lifecycle level by #154 (exclusions, partial status, missing evidence) or rendered by #156. Two cases have no coverage in #153–#156: ambiguous suite/run mappings staying unresolved instead of being inferred, and unrelated measurements in a shared output directory being kept out of aggregation. Could those be added to the parametrized chain test here? Optionally, one preparation-level failure/exclusion case (e.g., a selected JSONL row that fails conversion leaves measurement partial) would connect #154's lifecycle tests to real conversion output.

[rrhyne]

In response to the coverage review, 99353c4 expands the chain to ten cases across ATIF, Gym, and MLflow.

• Ambiguous mappings survive a state reload as a pending trace decision. Existing native results don’t settle that decision. The test records an explicit user mapping before continuing.
• A real unrelated measurement shares the output directory but stays out of the selected aggregate and retained evidence. Changing that unrelated file leaves aggregate verification and audit completion intact.
• A selected Gym row fails conversion, retains the converter error, and keeps measurement partial. Partial findings can be reviewed, but audit completion is rejected.

The full local suite passed with 3,286 tests passed and 14 skipped. The PR description lists the skips and their limits. I’ve also replied separately to each inline comment.