feat(audit): bind selected trace preparation to saved audits

docs/results.md

@@ -159,6 +159,14 @@ The Audit coverage report explains what those results mean. Journey state saves
 a measurement summary and references its evidence; failed inputs and missing
 methods keep measurement partial.
 
+Saved audit evidence links original traces and selected suite runs to prepared
+traces, conversion limits, and measurements. Journey state references the saved
+selection manifest. New manifests use
+`.eval-author/.working/audit/selection-manifest.json`, outside the source
+locations. Resumed audits keep their saved manifest path. Changing the original
+source or its selection reopens check review even if the prepared trace is
+unchanged. Changed prepared evidence reopens measurement.
+
 Without usable traces, coverage is **unmeasured**, not 0%; the specification can
 still be completed and the report prepared while measurement remains
 pending. The unchecked **Measure coverage** step states the reason inline,

skills/eval-author-audit/references/guided-audit.md

@@ -60,8 +60,9 @@ when the user selects it. Findings remain in `audit-coverage-report.md`; link it
 whenever it is created or updated.
 
 Treat changed inputs conservatively: changes to the agent, confirmed Ethos,
-selected scope, specification bytes, or selected case, verifier, or trace bytes
-reopen check review and all dependent measurement and findings. A change only to
+selected scope, specification or selection manifest bytes, or selected case,
+verifier, or trace bytes reopen check review and all dependent measurement and
+findings. A change only to
 the findings report reopens findings review. Record findings acceptance in state
 without rewriting the report merely to mark it accepted. Acceptance can complete
 a fully measured audit while the next action remains undecided; improvement
@@ -145,12 +146,39 @@ first; do not batch a broad inventory alongside loading these instructions.
 If the location is unknown, ask the next focused source question rather than searching the checkout, home
 directory, platform accounts, or unrelated jobs for candidate traces.
 
-Once selected, inspect only that source to establish which records are accessible
-and usable. Audit measurement accepts local ATIF files or Harbor trial directories
-containing ATIF; selecting a remote source does not grant access or make it ATIF.
-For other formats or inaccessible services, explain the needed local export or
-appropriate supported conversion. Do not query Intake or start platform discovery
-from this skill. Preserve the source selection and describe the limitation.
+Once selected, inspect only that source and prepare its traces with the existing
+loaders and converters. Use local ATIF directly, including ATIF in Harbor trials.
+Prepare Gym JSON or explicitly selected JSONL rows with
+[`gym-to-atif`](../../gym-to-atif/SKILL.md), preserving its original-ATIF preference,
+receipts, and losses. Use [`mlflow-to-atif`](../../mlflow-to-atif/SKILL.md) for complete
+local `Trace.to_dict()` exports. For a selected MLflow record, check the complete
+export for pagination and unique trace IDs before passing that record through
+`--input -`; retain its original export and exact ID. Do not fetch remote traces,
+query Intake, or generate runs to fill gaps.
+
+For selected local traces, reuse the saved `lifecycle.selection_manifest` on
+resume. For a new manifest, use
+`.eval-author/.working/audit/selection-manifest.json`. Resolve it with
+[`working_paths.working_path(repo, "audit", "selection-manifest.json")`](../../eval-author/scripts/working_paths.py)
+and save it with `working_paths.atomic_write_json`. Keep the manifest outside
+every suite or trace source path, candidate directory, and exclusion. Do this
+before check approval and set lifecycle `selection_manifest` to its
+repository-relative path. Leave it null when traces are unavailable.
+Record each original source path and digest,
+format, exact record selector, explicit suite/task/run mapping with its provenance,
+evidence class, required methods, conversion options, and exclusions with reasons.
+Keep original files in `trace_files`. Never infer mappings from filenames or
+trajectory IDs. When a suite/task/run mapping is missing or ambiguous, retain
+the candidates and leave trace selection pending until the user resolves it.
+Keep this manifest stable during execution. Changed selections must reopen check review.
+
+Retain preparation outcomes separately: for each selection link its source and
+record to prepared ATIF, conversion metadata/losses, and measurement artifacts, or
+record the missing, failed, or excluded outcome and reason. Include this outcome
+record and the prepared files, receipts, and native results as `{path, digest}`
+references in `measurement.evidence`. Before measurement or reuse, verify the
+original bytes and selected record, mapping, and retained references. Changed
+originals require preparation again even when prepared ATIF is unchanged.
 
 When the selected run folder is missing, explain that measurement is unavailable.
 If the confirmed evaluation definitions are accessible, offer the remaining
@@ -257,6 +285,12 @@ arguments to aggregation, or use a fresh output directory containing only that
 set. Do not recursively aggregate a shared measurements directory containing
 other audits' results. Reused coverage must belong to the current selected scope
 as well as match the current specification and trace inputs.
+Pass the saved task, run, and evidence class explicitly to native measurement.
+Only mapped agent-under-test runs from selected suites can establish suite
+coverage; production runs and task controls cannot substitute for them. Retain
+their evidence roles and unresolved selections in the report. Use only applicable
+coverage files from the retained outcomes and verify the aggregate before saving
+measurement; conversion success alone does not complete measurement.
 
 Complete when the agreed measurement scope has been processed and aggregated
 successfully, even when coverage is low. If some inputs fail, report only the

skills/eval-author/scripts/audit_state.py

@@ -23,6 +23,7 @@ def new_record() -> dict:
         "traces": copy.deepcopy(selection),
         "suite_files": [],
         "trace_files": [],
+        "selection_manifest": None,
         "specification": None,
         "checks_approval": None,
         "measurement": None,
@@ -90,6 +91,7 @@ def _decision(value) -> None:
 
 def validate(value: dict) -> None:
     try:
+        value = {"selection_manifest": None, **value}
         _fields(value, " ".join(new_record()))
         if type(value["schema_version"]) is not int or value["schema_version"] != 1:
             raise ValueError("Unsupported audit lifecycle version")
@@ -120,8 +122,9 @@ def validate(value: dict) -> None:
                 raise ValueError(f"{field} requires a selected {key} source")
             if any(not _within(path, value[key]["paths"]) for path in value[field]):
                 raise ValueError(f"{field} must stay within the selected {key} paths")
-        if value["specification"] is not None:
-            _path(value["specification"])
+        for key in ("specification", "selection_manifest"):
+            if value[key] is not None:
+                _path(value[key])
         for key in ("checks_approval", "findings_review"):
             if value[key] is not None:
                 _decision(value[key])
@@ -175,6 +178,12 @@ def _sources(state: dict) -> set[str]:
     for key, field in (("suites", "suite_files"), ("traces", "trace_files")):
         if value[key]["status"] == "selected" and (key == "suites" or value["suites"]["status"] != "pending"):
             files.update(value[field])
+    if (
+        value.get("selection_manifest")
+        and value["suites"]["status"] != "pending"
+        and value["traces"]["status"] == "selected"
+    ):
+        files.add(value["selection_manifest"])
     if value["findings"]:
         files.add(value["findings"]["report"]["path"])
     if value["measurement"] and value["suites"]["status"] != "pending" and value["traces"]["status"] == "selected":
@@ -230,6 +239,8 @@ def derive(state: dict, observed: dict) -> dict | None:
     files = {path: digest for path, digest in observed["files"].items() if path in allowed}
     sources_ready = all(value[key]["status"] != "pending" for key in ("suites", "traces"))
     inputs = value["suite_files"] + value["trace_files"] + ([value["specification"]] if value["specification"] else [])
+    if value.get("selection_manifest"):
+        inputs.append(value["selection_manifest"])
     checks_digest = None
     if (
         sources_ready

tests/test_audit_binding.py

@@ -17,10 +17,11 @@
 )
 
 
-def measured(tmp_path, evidence_class="agent_under_test"):
+def measured(tmp_path, evidence_class="agent_under_test", *, trace=None, run_id="fixture-run", include_unrelated=False):
     audit = _write_audit(tmp_path)
-    trace = tmp_path / "trace.json"
-    _write_atif_trace(trace, tool_calls=["customer.lookup"])
+    if trace is None:
+        trace = tmp_path / "trace.json"
+        _write_atif_trace(trace, tool_calls=["customer.lookup"])
     judgments = tmp_path / "judgments.json"
     _write_capability_judgments(judgments, trace=trace)
     out = tmp_path / "measurements"
@@ -31,6 +32,8 @@ def measured(tmp_path, evidence_class="agent_under_test"):
         str(trace),
         "--task-id",
         "fixture",
+        "--run-id",
+        run_id,
         "--out-dir",
         str(out),
         "--measure",
@@ -45,13 +48,38 @@ def measured(tmp_path, evidence_class="agent_under_test"):
     for _ in range(2):
         code, result, stderr = _run_json_script(_AUDIT_MEASURE, *command)
         assert code == 0, stderr or result
+        assert result["run_id"] == run_id
         contents = {path.relative_to(out): path.read_bytes() for path in out.rglob("*.json")}
         if _ == 0:
             original = contents
         else:
             assert contents == original
+    measurements = result["measurements"]
+    if include_unrelated:
+        unrelated = tmp_path / "unrelated.json"
+        _write_atif_trace(unrelated, tool_calls=[])
+        code, result, stderr = _run_json_script(
+            _AUDIT_MEASURE,
+            "--audit",
+            str(audit),
+            "--trace",
+            str(unrelated),
+            "--task-id",
+            "unrelated",
+            "--run-id",
+            "unrelated-run",
+            "--evidence-class",
+            "agent_under_test",
+            "--measure",
+            "tool_calls",
+            "--out-dir",
+            str(out),
+        )
+        assert code == 0, stderr or result
     report = tmp_path / "aggregate.json"
-    command = ["--audit", str(audit), "--coverage-dir", str(out), "--out", str(report)]
+    command = ["--audit", str(audit), "--out", str(report)]
+    for measurement in measurements:
+        command.extend(["--coverage", measurement["coverage"]])
     code, result, stderr = _run_json_script(_AUDIT_REPORT, *command)
     assert code == 0, stderr or result
     before = report.read_bytes()