diff --git a/AGENTS.md b/AGENTS.md index 8d1ce46dbf48..70bff240393f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -193,6 +193,21 @@ For a full list of up-to-date bot commands, post `/bot help` as a PR comment and - Use `TLLM_LOG_LEVEL_BY_MODULE` to enable per-module log filtering (e.g., `"debug:_torch,runtime;info:serve"`); see [Module-Level Logging](docs/source/developer-guide/overview.md#module-level-logging) for details. +**Environment failures that read as model failures.** Each of these presents as a broken +model or a broken build, and none of them is. Check them before debugging modeling code: + +| symptom | cause | fix | +|---|---|---| +| `MPI_ERR_SPAWN: could not spawn` | the container runtime forbids `MPI_Comm_spawn`, which the default LLM API path uses to start workers (normal under pyxis/enroot) | start one task per rank under `trtllm-llmapi-launch` so the ranks exist before the engine; `TLLM_WORKER_USE_SINGLE_PROCESS=1` only at world size 1 | +| `Impossible to fit in any sequence in kvCache` | the engine inherited the checkpoint's `max_position_embeddings` (262144 is common) and sized the cache for it | pass `--max_seq_len` / `--max_batch_size` / `--kv_cache_fraction` explicitly | +| `NVRTC ... could not open source file "cuda.h"` | usually the venv's `bin` missing from `PATH`; separately, the generation FMHA kernel takes the NVRTC path unless the KV dtype is `E2m1` (grep `options.mDtypeKv != tg::Dtype::E2m1`) | check `PATH` first; `--kv_cache_dtype fp8` takes the non-NVRTC branch | +| the engine hangs at startup and never returns | a wrong launch substrate hangs rather than raises — typically an MPI connect-back on an interface that does not carry it | probe with a deadline rather than waiting | +| the import resolves to a different checkout | `/root/.local` is shared across container runs on some clusters | `PYTHONNOUSERSITE=1` at runtime — but **not** during the native build, which needs the user site for `cutlass_library` | + +A longer list, with the wrong diagnoses recorded alongside the right ones, is in +[`agent-flow/agent_flow/workflows/trtllm_modeling_v2/TROUBLESHOOTING.md`](agent-flow/agent_flow/workflows/trtllm_modeling_v2/TROUBLESHOOTING.md). +It is keyed on verbatim error text, so search it for the string you saw. + ## Key Documentation | Topic | Path | diff --git a/agent-flow/README.md b/agent-flow/README.md index 279b00ce9d83..c9e7c9d4c1d5 100644 --- a/agent-flow/README.md +++ b/agent-flow/README.md @@ -297,4 +297,5 @@ block, before `run`: it writes each role's composed system prompt verbatim to handed in, since that is what the agents are built from (`AgentTeamWorkflow` appends its MCP-tool protocol to the bundle it is given). The `perf-analyze` and `perf-optimize` CLIs use it to snapshot `/prompts/` at every -launch. +launch; `trtllm-modeling-v2` does the same from preflight, once the launch +environment its prompts carry is resolved. diff --git a/agent-flow/agent_flow/backends/claude_code.py b/agent-flow/agent_flow/backends/claude_code.py index 2203a6e6c326..213907574a17 100644 --- a/agent-flow/agent_flow/backends/claude_code.py +++ b/agent-flow/agent_flow/backends/claude_code.py @@ -6,11 +6,12 @@ import subprocess import sys import tempfile -from contextlib import asynccontextmanager +from contextlib import asynccontextmanager, contextmanager +from contextvars import ContextVar from importlib.metadata import PackageNotFoundError from importlib.metadata import version as _pkg_version from pathlib import Path -from typing import Any, AsyncIterator +from typing import Any, AsyncIterator, Callable, Iterator import claude_agent_sdk from claude_agent_sdk import ( @@ -370,13 +371,114 @@ def _claude_sdk_version() -> str: return "" +#: A ``claude`` binary every layer launches instead of the SDK's own choice. +CLI_PATH_ENV = "AGENT_FLOW_CLAUDE_CLI" + + +def _version_of(cli_path: str) -> tuple[int, ...]: + """``claude --version`` of ``cli_path`` as a comparable tuple; empty when unreadable.""" + try: + out = subprocess.run( + [cli_path, "--version"], capture_output=True, text=True, timeout=_CLI_VERSION_TIMEOUT_S + ) + except (OSError, subprocess.SubprocessError): + return () + token = out.stdout.strip().split()[0] if out.returncode == 0 and out.stdout.strip() else "" + try: + return tuple(int(part) for part in token.split(".")) + except ValueError: + return () + + +def _cli_override() -> str | None: + """The ``claude`` binary layers launch, when it is not the SDK's bundled one. + + ``$AGENT_FLOW_CLAUDE_CLI`` names one outright. Otherwise the ``claude`` + on ``PATH`` is used when it is newer than the bundled binary: an + organization can require a minimum CLI version, and the bundled one + only moves when the SDK is upgraded, so it is refused while the + user's own CLI runs. ``None`` keeps the SDK's choice. + """ + named = os.environ.get(CLI_PATH_ENV, "").strip() + if named: + return named + global _CLI_OVERRIDE_CACHE + if _CLI_OVERRIDE_CACHE is None: + bundled = Path(claude_agent_sdk.__file__).parent / "_bundled" / "claude" + on_path = shutil.which("claude") + choice = "" + if on_path and bundled.is_file(): + path_version = _version_of(on_path) + if path_version and path_version > _version_of(str(bundled)): + choice = on_path + _CLI_OVERRIDE_CACHE = choice + return _CLI_OVERRIDE_CACHE or None + + +_CLI_OVERRIDE_CACHE: str | None = None + + +def launched_cli() -> tuple[str, str]: + """The ``claude`` binary a layer launches here, and its version; ``""`` for what is unknown.""" + return _find_claude_cli() or "", _claude_cli_version() + + +#: Given each line a CLI started in this context writes to stderr; see :func:`cli_stderr_to`. +_CLI_STDERR_LISTENER: ContextVar[Callable[[str], None] | None] = ContextVar( + "claude_cli_stderr_listener", default=None +) + + +@contextmanager +def cli_stderr_to(listener: Callable[[str], None]) -> Iterator[None]: + """Hand ``listener`` every stderr line of a CLI started inside the block, too. + + The lines still reach this process's stderr, and a listener set by an + enclosing block still gets them. A layer called in the block's thread + starts its CLI in the block's context, so the lines of one turn reach the + listener of that turn and of no other running beside it. A listener that + raises does not stop the CLI's stderr from being read. + """ + outer = _CLI_STDERR_LISTENER.get() + + def both(line: str) -> None: + if outer is not None: + _tell(outer, line) + _tell(listener, line) + + token = _CLI_STDERR_LISTENER.set(both) + try: + yield + finally: + _CLI_STDERR_LISTENER.reset(token) + + +def _tell(listener: Callable[[str], None], line: str) -> None: + try: + listener(line) + except Exception: # noqa: BLE001 - an observer cannot fail the CLI it reads + pass + + +def _forward_stderr(line: str) -> None: + """Print a CLI stderr line, so a start-up failure is diagnosable, and pass it to the listener.""" + print(f"[claude-cli stderr] {line}", file=sys.stderr, flush=True) + listener = _CLI_STDERR_LISTENER.get() + if listener is not None: + _tell(listener, line) + + def _find_claude_cli() -> str | None: - """Locate the ``claude`` CLI binary the SDK would actually launch. + """Locate the ``claude`` CLI binary the SDK will actually launch. - Mirrors the SDK's lookup order — bundled binary first, then ``PATH``. - Returns ``None`` when nothing is found so the caller can simply omit - the CLI version from the rendered string. + The override (:func:`_cli_override`) first, then the SDK's own lookup + order — bundled binary first, then ``PATH``. Returns ``None`` when + nothing is found so the caller can simply omit the CLI version from + the rendered string. """ + override = _cli_override() + if override: + return override bundled = Path(claude_agent_sdk.__file__).parent / "_bundled" / "claude" if bundled.is_file(): return str(bundled) @@ -576,6 +678,7 @@ async def create_client( cwd=work_dir, sandbox={"enabled": False}, permission_mode="bypassPermissions", + cli_path=_cli_override(), hooks=hooks, disallowed_tools=[ *(disallowed_tools or []), @@ -583,7 +686,7 @@ async def create_client( ], # Forward the CLI's stderr so startup failures are diagnosable # instead of vanishing with the subprocess. - stderr=lambda line: print(f"[claude-cli stderr] {line}", file=sys.stderr, flush=True), + stderr=_forward_stderr, ) async with ClaudeSDKClient(options=options) as sdk_client: diff --git a/agent-flow/agent_flow/workflows/trtllm_modeling_v2/.gitignore b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/.gitignore new file mode 100644 index 000000000000..fdaa0d14ee36 --- /dev/null +++ b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/.gitignore @@ -0,0 +1,5 @@ +# Placeholders a sandboxed agent session leaves in its working directory. +# Nothing in this package is configuration for a session that runs here: a run +# materializes its skills into its own workspace. +.mcp.json +.claude/ diff --git a/agent-flow/agent_flow/workflows/trtllm_modeling_v2/BUILD_AND_TEST.md b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/BUILD_AND_TEST.md new file mode 100644 index 000000000000..ec312088133a --- /dev/null +++ b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/BUILD_AND_TEST.md @@ -0,0 +1,302 @@ +# Build, rebuild, and test at each scale + +A companion to [TROUBLESHOOTING.md](TROUBLESHOOTING.md). That file is keyed on +symptoms — *something broke, what is it*. This one is keyed on tasks — *I need +to build / rebuild / run something at N GPUs, how*. + +Most of it is not this workflow's knowledge. It is collected from the places +in TensorRT-LLM that already do these things correctly, each named so you can +read the current version rather than trust this page: + +| where | what it owns | +|---|---| +| `scripts/build_wheel.py` | the build itself, and every flag below | +| `.claude/skills/exec-slurm-compile/` | building on a Slurm cluster, and `enroot import` | +| `.claude/skills/trtllm-test-script-builder/` | generating Slurm test scripts — **the launch rules** | +| `.claude/skills/exec-local-slurm/` | persistent allocations, container reuse, step cancellation | +| `tests/integration/test_lists/test-db/*.yml` | which tests run at which GPU count | +| `scripts/test_to_stage_mapping.py` | which CI stage a given test lands in | + +--- + +## 0. First ask: do you need to build at all? + +**For a modeling_v2 onboard, almost certainly not.** This is worth checking +before spending an hour, because the answer is usually no: + +| what you changed | rebuild? | +|---|---| +| a target's `modeling.py` / `weights.py` | **no** — pure Python, editable install picks it up | +| a catalog entry `.py`, shared or in a target's `ops/` | **no** | +| a FlashInfer call (sourcing tier 3) | **no** — the package is installed, its kernels JIT | +| a Triton kernel (sourcing tier 5) | **no** — Triton is JIT | +| routing, `_router_index.py` | **no** | +| anything under `cpp/` | yes | +| a new `torch.ops.trtllm.*` op | yes | +| pulling a trunk that touched `cpp/` | yes | + +The five sourcing tiers this workflow uses — an exported op, the kernel plane, +FlashInfer's public API, native torch, Triton — **none of them require a native +rebuild**. That is why +the orchestrator never builds anything and `environment.python` just points at +an install someone else made. + +If you are rebuilding during an onboard, stop and confirm why. + +### Rebuilding while compute sessions are running + +`build_wheel.py` installs `tensorrt_llm/libs/*.so` by rewriting each file in +place. Any process, on any node, that has the old file mapped then faults on +its next page-in, and when that process is attached to its allocation's MPI it +takes the whole allocation with it. Measured: a rebuild on one session killed +another session thirty seconds later. + +So rebuild through the workflow's `incremental_build.sh` (in the package's +`scripts/` directory), as a whole-node spool job: + +```bash +spoolctl.py submit --gpus all --wall 3h build- \ + 'bash /scripts/incremental_build.sh --repo --spool --spool ' +``` + +It takes a per-checkout build lock, announces the rebuild on every listed +spool (each session's scheduler then holds new GPU jobs against that checkout), +gives every installed library a fresh inode by copy-and-rename so running +processes keep the file they mapped, runs the build with ccache, checks the +checkout still imports, and retracts the announcement — also when the build +fails. Spools default to `$MV2FLOW_SESSION_SPOOLS`, which every rendered +session exports. + +## 1. Build + +```bash +python3 ./scripts/build_wheel.py \ + --use_ccache \ + --skip_building_wheel \ + --cuda_architectures "100-real;103-real" +``` + +| flag | when | +|---|---| +| `--cuda_architectures` | **always set it.** `100-real` Blackwell B200/GB200, `103-real` GB300, `90-real` Hopper, `89-real` L40S, `80-real` A100. Building all of them is the slowest thing you can do | +| `--use_ccache` | always. A no-op change rebuilds in minutes instead of an hour | +| `--skip_building_wheel` | development — builds in place, no wheel artifact | +| `--fast_build` / `-f` | skips some kernels. Development only — **do not gate on a fast build** | +| `--clean` / `-c` | wipes the build directory. The answer to "weird compile issues" | +| `--configure_cmake` | force a CMake re-configure. Needed when a CMake input changed but the build did not notice | +| `-j ` | lower it when the build OOMs | + +`build_wheel.py` creates its own venv and runs `pip install -e .[devel]` into +it, so the interpreter it produces is the one to put in `environment.python`. + +### ccache does not carry across checkouts + +Measured: pointing `CCACHE_DIR` at another checkout's cache produced no +useful hit rate. Absolute paths are baked into the cache keys. Give each +checkout its own, and expect the first build there to be cold. + +### The one that wastes an afternoon + +A rebuild is *not* the fix for `NVRTC ... could not open source file "cuda.h"`. +Measured: rebuilding with `--nvrtc_dynamic_linking` (98% ccache hits) did not +fix it; the cause was the venv's `bin` missing from `PATH`. See +TROUBLESHOOTING.md before you rebuild for that symptom. + +## 2. Build on Slurm + +`.claude/skills/exec-slurm-compile/` owns this end to end — `submit_compile.sh` +submits, `compile.slurm` allocates, `compile.sh` runs the build inside the +container, `enroot-import` fetches the image. + +Two failure modes from its own troubleshooting table that are not obvious: + +- `enroot import` **auth** failure → `~/.config/enroot/.credentials`; +- `enroot import` producing an **empty or corrupt** `.sqsh` → the image URL + probably has an `https://` prefix it should not have. + +### The environment hazard on a shared cluster + +`/root/.local` and `/root/.cache` may be symlinked into a shared home, so +another checkout's develop-install is on the path of every container run. + +- **At runtime: `PYTHONNOUSERSITE=1`.** Without it an import can resolve to a + different tree than the one you are measuring. +- **During the native build: leave the user site enabled.** The CUTLASS kernel + CMake runs `setup_library.py develop --user` and then imports + `cutlass_library`. Isolate the `pip install -e` step and everything after + it, not the compile itself. + +## 3. Test — the scale ladder + +Scale is declared by `system_gpu_count` in each list's `condition` block, and +the file name follows it: + +| scale | list | `system_gpu_count` | what belongs here | +|---|---|---|---| +| **CPU** | `l0_cpu.yml` | `0–0` | anything that needs no device. `backend: generic` | +| **1–4 GPU** | `l0_gb300.yml`, `l0_b200.yml`, `l0_h100.yml`, … | `1–4` | the ordinary pre-merge matrix | +| **multi-GPU** | `l0_*_multi_gpus.yml` | `4–4`, `8–8` | collectives, TP/EP topologies | +| **multi-node** | `l0_*_multi_nodes*.yml` | `8–8`+ | anything spanning nodes | + +To find where a test you care about runs: + +```bash +python scripts/test_to_stage_mapping.py --tests "test_modeling_v2_allgather" +``` + +### CPU + +`agent-flow`'s own suite is the clearest example: pure CPU, its own stage +(`CPU-AgentFlow-UnitTest`), and `AgentFlowRule` narrows CI so an +agent-flow-only change runs exactly that stage and no GPU stage at all. + +For the engine, CPU tests are the schema / list / prose audits — including the +modeling_v2 claims audits, which are text scans and need no device. + +### Single GPU + +```bash +pytest tests/unittest/_torch/modeling_v2/norm/test_modeling_v2_flashinfer_rmsnorm.py +LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/accuracy/test_modeling_v2_gpt_oss.py +``` + +Unit tests need no models. Integration tests need `LLM_MODELS_ROOT` **and** a +GPU; forgetting the variable produces a failure that reads like a missing +model rather than a missing setting. + +### Multi-GPU: two mechanisms, and they are not interchangeable + +**`mpi_pool_executor`** — a fixture, worker count by indirect parametrization: + +```python +@pytest.mark.parametrize("mpi_pool_executor", [2], indirect=True) +def test_something(mpi_pool_executor): ... +``` + +**A launcher in a fresh interpreter** — what the modeling_v2 collective +entries use, and the reason is stated in +`tests/unittest/_torch/modeling_v2/comm/_rank_job.py`: + +> A collective that breaks \*hangs\* rather than raising, so the +> deadline is load-bearing and the ``mpi_pool_executor`` fixture has none. + +Two consequences worth carrying into any new multi-rank test: + +- **A hang is the normal failure.** Whatever you write needs a deadline that + kills the process group, or one wedged rank takes the whole run with it. +- **You cannot start an MPI launcher from a pytest process that has already + imported `tensorrt_llm`** — that process has initialized MPI. The launcher + must go in a fresh interpreter, started *by file path*. + +### Multi-node + +The launch rules are the whole content here, and they come from +`.claude/skills/trtllm-test-script-builder/`. **The rule is a function of the +task count and nothing else:** + +| | `ntasks == 1` | `ntasks > 1` | +|---|---|---| +| `--mpi=` | `none` | `pmix` (or the cluster's variant — ask `srun --mpi=list`) | +| payload | clear the MPI environment, then exec | wrap in `trtllm-llmapi-launch` | + +```bash +srun -N 2 --ntasks 8 --ntasks-per-node 4 --mpi=pmix --gres=gpu:4 \ + --container-image=... --container-name=trtllm-run \ + bash -c "trtllm-llmapi-launch python3 -m tensorrt_llm.commands.serve $MODEL ..." +``` + +Four things that are each a whole afternoon if you get them wrong: + +1. **`--ntasks` must match the world size.** Without it `srun` launches one + task, the MPI world size is 1, and the LLM API concludes it must create the + other ranks itself. `need_spawn_mpi_workers` in `llmapi/mpi_session.py` is + exactly `get_mpi_world_size() == 1 and model_world_size > 1`. Under pyxis + that dynamic spawn is forbidden → `MPI_ERR_SPAWN`. +2. **`--gres` is per node**, not per world. They diverge the moment you span + nodes. +3. **At one task, clear the MPI environment.** mpi4py auto-initializes MPI + whenever PMI variables are present, and inside a PMIx-managed step they + always are — so a single-task job initializes MPI for no reason and then + dies spawning a worker. Keep `SLURM_PROCID` and `SLURM_LOCALID`; they + identify the rank rather than bootstrap MPI. +4. **`TLLM_WORKER_USE_SINGLE_PROCESS=1` is not the general fix.** The function + behind it is named `enable_worker_single_process_for_tp1` and the source + calls it a WAR for TP1. It does not scale past one rank. + +Inside a multi-node compute session there is no `srun` and no sshd in the +container. A session declared with `nodes: 2` runs one scheduler per node, and +the workflow's `mn_mpirun.sh` (in the package's scripts directory, under +`mn/`, with its own README) starts one `mpirun` across them, launching each remote daemon through +that node's scheduler. + +Inside a spool session the workflow emits the `mpirun` form of the same rule +from `environment.py` (`mpirun -n ... trtllm-llmapi-launch`); nothing +here is an agent's decision. The `srun` form is written out for a human +launching a test by hand, who needs to recognize a correct line. + +## 4. Iterating without paying full price + +| cost | how to stop paying it | +|---|---| +| container import, 26 GB / 4–8 min **per job** | `--container-name=` — pyxis skips the import when that name already exists on the node | +| queue wait per command | a persistent allocation: `salloc` once, then `srun --jobid=` steps into it (see `exec-local-slurm`) | +| both, plus a transport that cannot reach Slurm at all | a compute session — one long-lived job whose **own main process** is the spool scheduler; work is a file | +| cold native build | `--use_ccache`, and one cache per checkout | + +On the spool: work started inside a `slurm_exec_in_job` step does **not** +survive the step's teardown, `nohup` included — measured. The long-running +process has to be the job's own main process. If your cluster's Slurm CLI is +reachable, prefer the persistent-allocation pattern; the spool exists for the +case where it is not. + +A session is declared in `task.yaml` (`environment.sessions`) and its entry +script is rendered, never written by hand: + +```bash +trtllm-modeling-v2 session render --task task.yaml --name a # /session_entry.sh +trtllm-modeling-v2 session submission --task task.yaml --name a # the broker call that starts it +trtllm-modeling-v2 session status --task task.yaml # heartbeat, end, queue +``` + +Inside a session, size each job instead of picking devices: + +| you run | submit it as | +|---|---| +| a single-GPU unit test or microbenchmark | `--gpus 1` | +| a 4-rank collective test | `--gpus 4` | +| anything that loads the full model, a gate, a rebuild | `--gpus all` | +| a worktree's code | add `--tree ` — sets the working directory **and** `PYTHONPATH`, and refuses a result whose ranks imported another checkout | +| the orchestrator, or anything GPU-free and long-lived | a `free-` name, no `--gpus` | + +`--wall` counts from the job's start, so time queued behind other work is +free. `spoolctl.py status` says why each waiting job waits; `spoolctl.py cancel +` is the only way to stop one (see the safe-kill rule in +TROUBLESHOOTING.md). + +To abort one run without losing the allocation: + +```bash +squeue -s -j # list steps +scancel .0 # cancel one step +``` + +## 5. Where a new test belongs + +1. **No device?** → a CPU list, and check whether a CBTS rule already narrows + CI for your subtree (`jenkins/scripts/cbts/rules/`). +2. **One device?** → the ordinary `l0_.yml` for the architecture you + certified on. Certification is per-architecture (`ARCHS`); a pass elsewhere + is not a pass here. +3. **Collectives or a TP/EP topology?** → `l0_*_multi_gpus.yml`, and give it a + deadline. +4. **Spans nodes?** → `l0_*_multi_nodes*.yml`, and it needs the launch rules + in §3. + +Then confirm with `scripts/test_to_stage_mapping.py` that it actually lands in +a stage. A test on no list is a test nobody runs. + +**One trap for documentation-shaped changes:** `ModelingV2Rule` deliberately +excludes `*.md` so a documentation-only edit does not pull in multi-GPU GB300 +stages. That is the right trade, but it means a check that *scans* `.md` files +gets no CI trigger from the PRs most likely to break it. Put text audits in a +pre-commit hook, not in a GPU test stage. diff --git a/agent-flow/agent_flow/workflows/trtllm_modeling_v2/INTAKE.md b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/INTAKE.md new file mode 100644 index 000000000000..d372f91ff360 --- /dev/null +++ b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/INTAKE.md @@ -0,0 +1,680 @@ + + +# Intake: from a request to a confirmed task + +Status: the request schema, the fast path, the feasibility checks, +confirmation and compilation are implemented (`spec/intake/`); the +natural-language request parser and its interactive loop (section 7), the +memory and compatibility-matrix checks, budgets and `nice_to_have` are not. +Step 2 (the pipeline described in [README.md](README.md)) reads only the +compiled spec. + +## The fast path + +```bash +trtllm-modeling-v2 --checkpoint /models/newmodel --site site.yaml --workspace W \ + [--goal "mtp disagg@tp1/tep4 tp1 tep4 dep4 isl=8192 osl=1024 tune"] [--confirm] +``` + +derives the request in seconds (`spec/intake/derive.py`), writes it to +`W/task.intake.yaml` beside `W/intake.report.md` (what was read, what was +defaulted, what is still open), compiles it to `W/task.resolved.yaml` and +starts the run. There is no separate intake step: the request's checks run in +phase 0, first, beside the survey and the substrate gate, together with the +file-reading checks of the compiled spec (each unit's and each variant's +accuracy anchor, the speculation each launch path is asked for, the capture +lists), and a block stops the run there with every block listed. + +| value | from | +|---|---| +| architecture, MoE or dense, weight and KV-cache quantization | `config.json`, `hf_quant_config.json` | +| speculative decoding (`mtp` or `dspark`) and its depth; multimodal | the checkpoint's draft layers and towers: built unless the goal says `-` (`text_only` is `-multimodal`) | +| other features (`disagg`, `kv_quant`, ...) | the goal only; `@tp1/tep4` narrows one to members | +| `device.arch`, `gpus_per_node`, `nodes` | `arch=` in the goal, else `site.yaml device` | +| members | the goal's `tp1 tep4 ...`, else `site.yaml defaults.members`, else `tp1` | +| workload | the goal's `isl= osl= max_seq_len= max_batch_size= concurrency=1/16`, else `defaults.workload`, else isl 4096, osl 1024, max batch 64 (fitted to `max_position_embeddings`) | +| core switches | `cuda_graph`, `overlap_scheduler`, `chunked_prefill`, `prefix_reuse` required unless the goal says `-`; a hybrid checkpoint gets a 2048-token state snapshot interval for block reuse (`snapshot=`) | +| accuracy anchor | `benchmark= model_key= reference=`, else `defaults.accuracy`, else gsm8k keyed by the config's `_name_or_path` | +| boot prompts | `defaults.boot`, else a checked `The capital of France is` -> `Paris` and one open prompt | +| witness declarations | `defaults.witness.`, and a replay window from `sliding_window`; what is still missing is an open question | +| environment, sessions, agents, commit policy, perf spools | `site.yaml`, as for any intake task | + +`site.yaml` takes two optional blocks for this: `device: {arch, gpus_per_node, +nodes}` and `defaults: {members, workload, accuracy, boot, witness}`. + +Confirmation is recorded, not awaited. `--confirm [BY]` on the command, or +`trtllm-modeling-v2 intake confirm W/task.intake.yaml --by NAME` at any time, +writes `W/task.intake.yaml.confirmed`; the request itself is never rewritten. +Until the run is past preflight the request may be edited and the same command +re-run. After that, `W/task.intake.yaml.started` holds the digest it started +from, and an edit is refused until it is confirmed; a request already +confirmed is refused once it changes, as a hand-written one is. Re-running the +command resumes the request on disk; another `--goal` or `--checkpoint` is +refused, and `--clean` derives it again. + +## 1. Problem + +A run today starts from `task.example.yaml` (810 lines) validated by +`task_schema.py` (2,400 lines), plus a natural-language prompt. Three things +go wrong with that: + +- **The user reviews the wrong thing.** About a dozen fields express what the + user wants. The rest — `environment` (~200 lines), accuracy floors, witness + workloads, agent backends, commit policy — say how to run it. Nobody can + check a requirement buried in that. +- **Requirements are inferred by agents, mid-run.** Whether a checkpoint's + `mtp.*` weights are MTP heads or a DSpark drafter, whether `max_seq_len` + has to be set explicitly, which benchmark protocol a reasoning model needs: + these are taught to agents through prompts and skills. When an agent gets + one wrong, the run spends hours before a gate notices. +- **Infeasible requests are discovered late.** Preflight + (`check_checkpoint`, `check_speculation_support`, `check_accuracy_anchors`, + …) runs after the user has committed to a run, and it only checks what the + spec already says. It cannot say "this model does not fit TP4 on this GPU" + or "this checkpoint has no MTP weights". + +## 2. Goal + +Add a **Step 1, intake**. It turns a natural-language request into a short +`task.yaml` that the user reads, corrects and confirms before any GPU or agent +time is spent. After that, Step 2 runs as it does today. + +```text +request (natural language) + │ + ▼ +Step 1 intake ──► task.yaml (requirements; the user confirms this) + │ intake.report.md (what was checked, what was inferred) + │ + site.yaml (per cluster, reused) + ▼ +compile ──► task.resolved.yaml (today's schema, unchanged) + │ + ▼ +Step 2 trtllm-modeling-v2 (preflight → plan → smith → assemble → deliver → [tune]) +``` + +Non-goals: + +- changing Step 2's stages, gates or schema; +- letting intake write modeling code or launch GPU jobs. Intake may read + checkpoints, the repository and the site description, nothing more. + +## 3. Three files + +| file | written by | reviewed by user | holds | +|---|---|---|---| +| `task.yaml` | intake, edited by the user | **yes**, target 50–80 lines | requirements: model, device, parallelism, workload, features, accuracy, perf, constraints, notes | +| `site.yaml` | once per cluster | no | `environment` (`local` or `session_spool`, with its sessions), `trtllm_repo_path`, perf spools and `env_script`, `agents` defaults | +| `task.resolved.yaml` | `compile`, never edited | no (audit only) | today's full schema, the only file Step 2 reads | + +Splitting `site.yaml` out removes the largest block from what the user reads, +and it is the one block that does not change between tasks on the same +cluster. + +## 4. `task.yaml` + +### 4.1 Example + +```yaml +version: 1 +goals: [onboard, tune] # any of: onboard | add_features | tune; see 4.2 + +model: + checkpoint: /models/Qwen3-30B-A3B-NVFP4 + architecture: Qwen3MoeForCausalLM # inferred: config.json + quantization: {weights: nvfp4, kv_cache: bf16} # inferred: hf_quant_config.json + kind: moe # inferred: num_experts=128 + in_tree: tensorrt_llm/_torch/models/modeling_qwen3_moe.py # inferred + reference_impl: {engine: vllm, path: /src/vllm} # also the accuracy oracle + base: null # required when goals has no `onboard`: the existing target and branch + +device: + arch: sm_100 # B200 + gpus_per_node: 8 + nodes: 1 + +parallelism: # one entry = one config; several = `unified` + - {kind: tep, world: 4} + - {kind: dep, world: 4} + +workload: + isl: 8192 + osl: 1024 + max_seq_len: 9216 # inferred: isl + osl, never the checkpoint's 262144 + max_batch_size: 64 + concurrency: [1, 16, 64] + +features: # section 5 + cuda_graph: required + overlap_scheduler: required + chunked_prefill: {status: required, params: {chunk_tokens: 8192}} + prefix_reuse: required + kv_cache: {status: nice_to_have, params: {dtype: fp8}} + long_context: out_of_scope + speculative: {status: required, params: {algo: mtp, draft_len: 3}} # inferred: num_nextn_predict_layers=1 + host_offload: {status: nice_to_have, params: {tensors: [per_layer_embed]}, members: [dep4]} + disaggregated: out_of_scope + multimodal: out_of_scope # becomes scope: text_only + +accuracy: + benchmark: gsm8k + protocol: {chat_template: true, reasoning: off, shots: 0} + reference: {source: reference_yaml} # or: {source: measure, engine: vllm} + +perf: # present exactly when goals contains `tune`; section 5.8 + metric: output_tps_per_gpu + baseline: stock # stock | vllm | sglang | + goal: best_effort # or: {ratio_to_baseline: 1.1} and/or {pct_of_sol: 0.7} + configs: + - name: low_latency + parallel: tep4 + features: {cuda_graph: on, speculative: on} + workload: {isl: 8192, osl: 1024, concurrency: [1, 4, 16]} + constraint: {tpot_ms_max: 30} + +constraints: + embargo: [tensorrt_llm/_torch/models/modeling_qwen3_moe.py, /src/vllm] + base_ref: origin/main + budget: {gpu_hours: 200, wall_days: 3} + +deliverable: {branch: auto, pr: false, tests: [unit, accuracy_reference]} + +notes: | + Follow modeling_v2 naming. Reuse the NVFP4 MoE kernels in + tensorrt_llm/_torch/modules/fused_moe; do not add new CUDA. + +provenance: # maintained by intake; see 4.3 + model.architecture: inferred + workload.max_seq_len: inferred + features.speculative.algo: inferred + perf.constraint: user +open_questions: [] # must be empty to confirm +# confirmed: {by, at, sha256} is written to task.yaml.confirmed, beside this file +# derived: {checkpoint, site, goal, at}, when the fast path wrote this file +``` + +### 4.2 Fields + +**`goals`** is a set, not one mode. A request often asks for more than one +thing ("onboard it, then tune it"), so intake records every goal it heard and +the run executes them in a fixed order: build first, then tune. + +| goal | what it means | requires | +|---|---|---| +| `onboard` | build a new target for a checkpoint with none | `model`, `features`, `accuracy` | +| `add_features` | add features to an existing target | `model.base`, the new features with `status: required` | +| `tune` | improve the performance of a target | the `perf` block | + +Legal combinations and how Step 2 runs them: + +| `goals` | Step 2 | +|---|---| +| `[onboard]` | `trtllm-modeling-v2 --task` | +| `[onboard, tune]` | the same run with `perf.enabled: true`: tune is chained after delivery, on the delivered tree | +| `[add_features]` | `--task` starting from `model.base` (a gap today, section 8) | +| `[add_features, tune]` | as above, then tune on the delivered tree | +| `[tune]` | `trtllm-modeling-v2 tune` on `model.base` | + +Rules the checks enforce: + +- `onboard` and `add_features` are exclusive: one creates the target the other + presupposes. A request that reads as both ("onboard X with MTP") is + `onboard` with MTP in its features. +- Without `onboard`, `model.base` is required and must exist. +- With `tune` after a build goal, the tune stage measures the tree the build + delivered, so its accuracy anchor is that build's accepted score unless + `perf` names another. +- An empty set is an open question, not a default. + +**`features`** lists capabilities with a status, parameters and the parallel +configs they must work on; section 5 defines them. + +**`perf`** gains a metric, a goal and per-configuration constraints +(section 5.8). Today the tune stage keeps whatever beats noise and has no +definition of done. + +**`notes`** is free text for code style, reference implementations, +reference kernels and infrastructure. It is passed to every role verbatim. +It may not change requirements; anything that would belongs in a field. + +### 4.3 Provenance + +Every value records where it came from: `user` (stated), `inferred` (read from +the checkpoint, the repository or the site) or `default`. The review summary +lists `inferred` and `default` values first: those are the ones the user did +not say and has to check. + +`open_questions` holds what intake could not decide. Confirmation is refused +while it is non-empty. + +## 5. Features + +### 5.1 Capabilities and configurations + +A feature appears in two places, and the two must not be confused: + +- **Capability** (`features:`): the target must support it, and a build goal + must prove that with a gate. "The target supports MTP." +- **Configuration** (`perf.configs[*].features`, and the switches the gates run + with): a concrete serving setup that turns capabilities on or off. "Measure + TEP4 with MTP on, CUDA graph on, at concurrency 1–16." + +A configuration may only turn on what is a capability: one the build goal +delivers with `status: required`, or one `model.base` already has. Turning on +anything else is a `block` (section 6). + +### 5.2 Entry shape + +Every feature has the same shape, so intake, the checks and compile treat +them uniformly: + +```yaml +features: + : + status: required # required | nice_to_have | out_of_scope + params: {...} # per feature, section 5.3 + members: [tep4] # optional: the parallel configs it must work on + accept: {...} # optional: overrides the default acceptance rule +``` + +The short form `: required` means `{status: required}` with default +params. + +**Status** + +| status | build goal | delivery | prompts | +|---|---|---|---| +| `required` | built and gated | blocked if its gate fails | listed as work | +| `nice_to_have` | built on idle resources within its budget share, never ahead of `required` work | not blocked; on failure the attempt moves to a side branch (section 5.7) | listed as work, marked optional | +| `out_of_scope` | not built | not checked | stated as out of scope, so an absent path is not read as a defect | + +A feature the request did not mention: + +- the checkpoint carries it (MTP weights, a vision tower, …) → an open + question with a recommended answer. Intake never silently builds or drops a + capability the weights have; +- the checkpoint does not carry it → `out_of_scope`, provenance `default`. + +**`members`** defaults to every parallel config the feature applies to +(section 5.6). Narrowing it is how a user says "host offload only on DEP8". It compiles +to today's per-unit `members` mapping. + +### 5.3 Catalog + +One row per feature: its parameters, what intake reads from the checkpoint, +and how Step 2 proves it works. Step 2's existing feature names are in +brackets. + +**Core.** These are runtime behaviours every target has. They need no +increment of their own, but the gates must run with them on, or the target is +never tested in the configuration it will serve. + +| feature | params | inferred from | proof | +|---|---|---|---| +| `cuda_graph` | `batch_sizes` (default: powers of two up to `workload.max_batch_size`), `padding` | — | every gate runs with it; the captured sizes are the serving envelope the reviewer checks against | +| `overlap_scheduler` | — | — | every gate runs with it | +| `chunked_prefill` | `chunk_tokens` (`max_num_tokens`) | — | every gate runs with it. **Gap**: nothing ensures a gate prompt is longer than `chunk_tokens`, so the chunked path may never run; intake should add such a prompt | +| `kv_cache` | `dtype` (`auto` \| `fp8` \| `nvfp4`), `tokens_per_block`, `memory_fraction`, `host_cache_gb` | `hf_quant_config.kv_cache_quant_algo` | accuracy gate with the matching `discriminators` row | +| `prefix_reuse` | `workload` (a shared-prefix set) | sliding-window layers (`layer_types`, `sliding_window`) | **gap**: a hit-rate witness, plus outputs equal with reuse on and off | +| `long_context` | `length`, `benchmark` (`ruler` \| `longbench` \| `needle`), `rope_scaling` | `max_position_embeddings`, `rope_scaling` | **gap**: a long-context accuracy gate at `length` | + +`kv_cache.dtype` is the engine's KV storage type. It is not `[kv_quant]`: that +feature moves the target to the checkpoint's own native KV formats (FP4 +compressed KV, MXFP4 indexer keys, …) and is listed below. + +**Speculative decoding.** At most one algorithm per configuration. + +| feature | params | inferred from | proof | +|---|---|---|---| +| `speculative` | `algo` (`mtp` \| `eagle3` \| `dflash` \| `dspark`), `draft_len`, `drafter` (path, eagle3/dflash), `tree` (`linear` \| `dynamic`) | `num_nextn_predict_layers`, `mtp.*` tensors, `dspark_*` config keys, a drafter checkpoint next to the target | acceptance gate (acceptance length vs `=off`, or an absolute floor for a new architecture) | + +`algo` is inferred, not taken from the request's wording. A request for "MTP" +on a checkpoint whose `mtp.*` stages are a DSpark drafter becomes `dspark`, +with the reason shown to the user. + +**Model-structure features.** These exist only for some architectures. +Intake offers them when the checkpoint has the structure. + +| feature | params | inferred from | proof | +|---|---|---|---| +| `kv_quant` [`kv_quant`] | `formats` per KV pool, `bytes_per_token` | checkpoint KV scales and formats | bytes-per-token witness | +| `decoder_replay` / `encoder_replay` [same] | `window` | SWA layers | work witness (accuracy alone passes an empty implementation) | +| `host_offload` [`host_offload`] | `tensors`, predicted `memory_delta` | a tensor too large to keep resident | residency witness | + +**Serving topology** + +| feature | params | inferred from | proof | +|---|---|---|---| +| `disaggregated` [`disagg`] | `ctx: {parallel, count}`, `gen: {parallel, count}`, `transfer` (`nixl` \| `ucx`) | per-sequence state beyond attention KV (mamba/linear-attention state) | equivalence witness: the same outputs as one engine | + +**Multimodality** + +| feature | params | inferred from | proof | +|---|---|---|---| +| `multimodal` [`multimodal`] | `modalities` (`image` \| `video` \| `audio`), `max_items_per_request`, `max_resolution` | `vision_config`, `audio_config`, `preprocessor_config.json` | MMMU-Pro (or the modality's benchmark), or the logit reference made with image prompts (`onboard/logit_reference.py --image`): one of them judges the vision path, or preflight stops on the uncovered cells; plus an image boot prompt | + +`multimodal: out_of_scope` on a multimodal checkpoint compiles to +`scope: text_only`. + +**Serving-API features.** Guided decoding, logprobs, logits post-processors, +reasoning and tool-call parsers, LoRA. These are runtime features. They need +no modeling increment, but they can still break on a new target. + +| feature | params | proof | +|---|---|---| +| `guided_decoding` | `backend` | **gap**: a serve smoke test with a schema | +| `logprobs` | `top_k` | **gap**: a serve smoke test | +| `reasoning_parser`, `tool_parser` | parser name | **gap**: a serve smoke test | +| `lora` | `max_rank`, `adapters` | **gap**: needs a modeling increment (LoRA layers), no skill exists | + +**Custom features.** Anything else the user asks for: + +```yaml +features: + custom: + - name: fused_qk_norm_rope + description: fuse QK-norm and RoPE into one kernel in attention + status: nice_to_have + accept: {kind: perf, metric: tpot_ms, max_regression: 0, min_gain: 0.02} +``` + +A custom feature without an `accept` rule is an open question. "Done" has to +be checkable, otherwise an agent decides it. Allowed `accept.kind` values: +`accuracy` (a benchmark and floor), `perf` (a metric and gain), `test` +(a pytest node id that must pass), `review` (a human signs off at delivery). +Today custom features compile to `notes` plus a gap. + +### 5.4 From the request to features + +The words in a request do not map one-to-one onto features. Intake normalizes +them and records the mapping in provenance, so the user can see each +translation: + +| the request says | becomes | watch out for | +|---|---|---| +| "MTP", "投机解码", "speculative" | `speculative.algo` from the checkpoint | `mtp.*` may be a DSpark drafter; no algo → ask if several are possible | +| "Eagle", "DFlash" | `speculative.algo` + `drafter` path | the drafter checkpoint must exist | +| "FP8 KV", "KV 量化" | `kv_cache.dtype: fp8` | not `kv_quant`, unless the checkpoint ships native KV formats | +| "PD 分离", "disagg" | `disaggregated` | ctx/gen parallelism are separate questions | +| "EP", "attention DP", "TEP/DEP" | `parallelism`, not a feature | — | +| "EPLB", "DWDP" | not a feature: no target exercises them, so there is nothing to build from | reported as out of scope | +| "128K 长上下文" | `long_context.length: 131072` | compare against `max_position_embeddings` and RoPE scaling | +| "前缀缓存", "prefix caching" | `prefix_reuse` | SWA models need `encoder_replay` for reuse to help | +| "VLM", "多模态", "图片输入" | `multimodal.modalities` from the config | "text only" → `out_of_scope` | +| "CUDA graph", "overlap" | core switches | — | +| "像 vLLM 那样实现" | `notes` + `model.reference_impl` | the reference may also be embargoed | + +### 5.5 Compatibility + +Rules, with their source. Each is a check in section 6. Breaking a rule +between two `required` features is a `block`. If either side is +`nice_to_have`, it is a `warn` and the `nice_to_have` side is dropped. + +| rule | source | +|---|---| +| `speculative.algo: mtp` and `dspark` never both | `task_schema.MTP_NAMESPACE_FEATURES` | +| speculative × pipeline or helix parallelism: No | feature-combination-matrix | +| logits post-processor × disaggregated, × speculative: No | feature-combination-matrix | +| `multimodal: required` needs a vision/audio config in the checkpoint | definition | +| `long_context.length` ≤ `max_position_embeddings` × RoPE scaling factor | config | +| `prefix_reuse` on an SWA model without `encoder_replay`: reuse limited to the window | `trtllm-modeling-v2-bounded-replay` skill (warn) | +| an "Untested" cell in the matrix | warn; the user decides | +| architecture-specific limits (e.g. the CSA2 table at the top of the matrix) | applied when `model.architecture` matches | + +The matrix is read from the document at intake time, not copied into code, so +the rules follow the doc. A small parser turns its table into `{(a, b): Yes | +No | Untested | Known issues}`. + +### 5.6 Feature × parallel config + +Not every feature applies to every member. Intake builds this table, shows it +in the report and compiles it to per-unit `members`: + +```text + tp4 tep4 dep4 +core ✓ ✓ ✓ +speculative ✓ ✓ ✓ +host_offload – – ✓ (user narrowed) +disaggregated ctx: tep4 ×1, gen: dep4 ×1 +``` + +A `required` feature with no applicable member is a `block`, for example +`disaggregated` naming a member the run does not have. + +### 5.7 Ordering and `nice_to_have` + +Intake does not fix a build order. Step 2's planner decomposes and orders the +work freely ([docs/design.md, section 6](docs/design.md#6-the-task-graph-and-the-planner)). +Intake hands it only the hard dependencies that follow from the features +themselves: + +- every feature needs a booting core on at least one member; +- a feature's witness compares against the target without it, so that + target must exist on the same tree; +- `disaggregated`'s equivalence witness compares against the single-engine + target with every other delivered feature on. + +A `nice_to_have` feature: + +- **runs only on idle resources**, within a budget share (default 20% of the + run's budget). The orchestrator never gives it a device a ready `required` + task could use; +- **is abandoned** when its share is spent, when one of its checks fails N + times with no new hypothesis from the planner, or as soon as its change + turns a passing `required` check red; +- **is moved to a side branch when abandoned.** The delivered tree does not + contain it. Its commits go to `--wip`, with the failing + check's evidence, and the delivery report names the branch; +- **is reported as one of**: delivered; not delivered (reason, evidence, side + branch); not started (budget). + +**Gap**: Step 2 has no `nice_to_have` today. Every listed feature is required. + +### 5.8 Perf configurations + +`perf.configs` names the serving setups to tune. Each one picks a parallel +config, the features it turns on and a workload: + +```yaml +perf: + metric: output_tps_per_gpu + baseline: stock + configs: + - name: low_latency + parallel: tep4 + features: {cuda_graph: on, speculative: on} + workload: {isl: 8192, osl: 1024, concurrency: [1, 4, 16]} + constraint: {tpot_ms_max: 30} + - name: max_throughput + parallel: dep4 + features: {cuda_graph: on, speculative: off} + workload: {isl: 8192, osl: 1024, concurrency: [128, 256]} +``` + +Checks: + +- every `on` is a capability (section 5.1); +- the parallel config is a member; +- the workload fits in memory with these features (spec decoding adds draft + KV; disaggregated splits memory per role); +- a multimodal workload names a dataset with media. + +Each config compiles to one `perf.configs` entry: its workload, its +constraint, and its feature switches as `llm_api_options` over the serving +config derived from the workload. `perf.goal` compiles to +`perf.goal.ratio_to_reference` and/or `perf.goal.pct_of_sol` (a fraction of +the speed-of-light ceiling; README "Tuning"). **Gap**: the tune stage has no +disaggregated or multimodal measurement path, and measures every config at +the run's parallel configuration. Configs that need another are recorded +and warned. + +## 6. Feasibility checks + +Each check returns `pass`, `warn` or `block`, a one-line reason and a +suggested fix. A `block` turns into an open question. + +| area | check | example `block` | +|---|---|---| +| checkpoint | `config.json` parses; architecture known; quantization readable | not a HF checkpoint | +| weights present | MTP layers (`num_nextn_predict_layers`), `dspark_*` keys, EAGLE3/DFlash drafter, vision tower | "MTP requested; checkpoint has no `mtp.*` weights" | +| MTP vs DSpark | `mtp.*` namespace read as one or the other, never both | "the `mtp.*` stages are a DSpark drafter; use `dspark`" | +| memory | weights / (tp·ep) + KV for `max_batch_size × max_seq_len` + activations ≤ HBM × fraction | "TP2 needs 212 GB/GPU; B200 has 180 GB — use TEP4 or FP8 KV" | +| divisibility | heads, kv heads % tp; experts % ep; world ≤ gpus_per_node × nodes | "8 KV heads cannot be split 16 ways" | +| quant × arch | NVFP4 needs sm_100+; KV dtype has a kernel on this arch | "NVFP4 on sm_90" | +| length | isl + osl and `long_context.length` ≤ `max_position_embeddings` and RoPE scaling | "131072 requested; config caps at 32768 without YaRN" | +| features | the section 5.5 rules; section 5.6 applicability; every unmentioned capability the checkpoint carries; every custom feature has `accept` | `mtp` and `dspark` both required; speculative with pipeline parallelism | +| perf configs | section 5.8: only capabilities turned on, member exists, fits in memory | `speculative: on` in a config, but speculative is `out_of_scope` | +| goals | `onboard` with a modeling_v2 target already present → ask `add_features`; `onboard` with an in-tree stock implementation → ask about embargo; no `onboard` and no `model.base`; `onboard` + `add_features` together | base target missing | +| accuracy | a reference row matches the discriminators, or a reference engine is available; protocol matches a reasoning model | "no gsm8k row for nvfp4 + mtp; measure with vllm?" | +| site | the cluster has this arch and enough GPUs; the container and python resolve | "site has sm_90 only" | +| budget | rough GPU-hours for the requested units and members vs `budget` | warn only | + +**These checks share code with `preflight.py`.** Where a check exists there +(`check_checkpoint`, `check_speculation_support`, `check_accuracy_anchors`, +`check_multirank_substrate`), intake calls it; new checks go next to it so that +Step 2's preflight runs them too. Two copies of these rules would drift. + +## 7. Interaction loop + +```text +1. parse the request → draft task.yaml, provenance = user +2. inspect checkpoint/repo/site → fill inferred fields +3. run feasibility checks → blocks become open_questions +4. show intake.report.md: requirements table, check table, questions +5. user answers / edits → back to 2 +6. open_questions empty and the user confirms + → write confirmed {by, at, sha256} +``` + +Rules: + +- Ask only blocking questions, at most a handful per round, each with a + recommended answer. +- Confirmation is refused while an open question or a block stands. +- Step 2 refuses a hand-written `task.yaml` that is unconfirmed, and any + request whose content no longer matches its confirmation + (`.confirmed`, or a `confirmed` block in the file). A derived request + runs unconfirmed (the fast path, above). +- If Step 2 finds a requirement it cannot meet (a feature that does not work + on this checkpoint), it stops and hands back to intake. Agents may not + downgrade a requirement on their own. +- A natural-language prompt given to Step 2 is an implementation hint only. + +## 8. Compile: `task.yaml` + `site.yaml` → today's schema + +| `task.yaml` | today's key | +|---|---| +| `goals` contains `tune` after a build goal | `perf.enabled: true` (tune chained after delivery) | +| `goals: [tune]` | the `tune` subcommand on `model.base` | +| `model.checkpoint` | `checkpoint_path` | +| `model.architecture` / segment override | `target.family`, `target.checkpoint_segment` | +| `device.arch` | `arch` | +| `parallelism` (one entry) | `parallel: ` | +| `parallelism` (several) | `parallel: unified`, `members: [...]` | +| `workload.max_seq_len`, `max_batch_size` | `accuracy.max_seq_len`, `accuracy.max_batch_size`, `runtime.extra_flags` | +| `features.{cuda_graph, overlap_scheduler, chunked_prefill}` | `runtime.*`; `chunk_tokens`, `batch_sizes` → `runtime.extra_flags` and `accuracy.llm_api_options` | +| `features.prefix_reuse` | `runtime.kv_cache_reuse` | +| `features.kv_cache` | `accuracy.llm_api_options.kv_cache_config`, `accuracy.discriminators` | +| `features.speculative` | `features: [mtp \| eagle3 \| dflash \| dspark]`; the unit's (and every later unit's) `speculative_config` (`draft_len`, `drafter`) in its options file and `spec_dec_algo` in its discriminators | +| `features.{kv_quant, decoder_replay, encoder_replay, host_offload}` | `features: []` + `accuracy.witness` where the feature has one | +| `features.disaggregated` | `features: [disagg]` + `accuracy.witness` | +| `features.multimodal` required / out of scope | `features: [multimodal]` / `scope: text_only` | +| `features.*.members` | per-unit `members` (`{default: [...], : [...]}`) | +| `model.quantization.weights`, `features.kv_cache.params.dtype` | `accuracy.discriminators` (`quant_algo`, `kv_cache_quant_algo`), unless `accuracy.discriminators` is stated | +| every unit | `accuracy.llm_api_options.`: an options file under `llm_api/` beside the compiled spec, with the runtime switches, `kv_cache_config` and `cuda_graph_config` the quickstart gates also take | +| `accuracy.boot`, `accuracy.witness` | the same keys, as they are | +| `perf.configs`, `perf.goal` | `perf.configs` (workload, constraint, `llm_api_options` with the config's `parallel` as `tensor_parallel_size`/`moe_expert_parallel_size`/`enable_attention_dp` and `speculative: on` as the run's `speculative_config`), `perf.goal`; without configs, one per member at the run's workload, with the draft | +| `accuracy.benchmark`, reference | `accuracy.benchmark`, `reference_yaml`, `discriminators`, `model_key` | +| `accuracy.protocol` | `accuracy.benchmark_args`, `accuracy.llm_api_options` | +| `perf` | `perf.enabled`, `perf.workload`, `perf.reference`, `perf.identity_configs` | +| `constraints.embargo` | `source_embargo`, `perf.source_embargo`, `perf.sister_engines` | +| `constraints.base_ref` | `commit_policy.base_ref` | +| `notes` | `notes`, appended verbatim to every role prompt | +| `constraints.budget` | `perf.campaign` limits (partial) | +| `site.yaml` | `environment`, `trtllm_repo_path`, `agents`, `perf.spools`, `perf.dev_spools`, `perf.env_script` | + +Fields with **no consumer in Step 2 today**. Compile records them and warns, +by name (as it does for any feature parameter it does not compile, +`accuracy.reference`, `accuracy.protocol` keys other than `chat_template`, +`perf.metric`, a `perf.baseline` other than `stock`, `deliverable`, +`model.in_tree` and `model.reference_impl`); each needs a Step 2 change before +it is enforced: + +- `features.long_context` — needs a long-context gate; +- `features.prefix_reuse` as a verified feature — needs a hit-rate witness; +- `chunked_prefill` — no gate prompt is guaranteed to exceed `chunk_tokens`; +- `status: nice_to_have` — Step 2 treats every feature as required; +- serving-API features and `custom` features — no gate; +- `perf.configs` that are disaggregated, multimodal, or at another parallel + configuration — aggregate text-only measurement at the run's; +- `perf.metric` — the keep rule judges both headline metrics; +- `constraints.budget.gpu_hours` / `wall_days` — no run-wide budget exists; +- `goals: [add_features, ...]` with `model.base` — Step 2 has no "start from an + existing target" entry; today a feature increment is built in the same run as + core. `[onboard, tune]` and `[tune]` are already supported. + +## 9. What this removes from prompts + +Facts that are now decided in Step 1 and handed to agents rather than taught +to them: + +- which speculator the checkpoint actually carries (MTP vs DSpark); +- `max_seq_len` / `max_batch_size` and why they must be explicit; +- the accuracy protocol for a reasoning model; +- which features are out of scope and why that is not a defect; +- the workload shapes that the serving envelope is judged against. + +Once intake ships, the corresponding passages in `agents/prompts/` and `agents/skills/` +can be cut to a statement of the confirmed value. That list is to be built by +reading the prompts against section 8, as a follow-up. + +## 10. Interface + +```bash +trtllm-modeling-v2 --checkpoint C --site site.yaml --workspace W [--goal G] [--confirm [BY]] + # the fast path: derive, check in phase 0, run +trtllm-modeling-v2 intake derive --checkpoint C --site site.yaml [--goal G] -o task.yaml + # derive only; writes intake.report.md beside it +trtllm-modeling-v2 intake check task.yaml # the feasibility checks; exit 1 on a block +trtllm-modeling-v2 intake confirm task.yaml --by NAME # writes task.yaml.confirmed +trtllm-modeling-v2 intake compile task.yaml --site site.yaml -o task.resolved.yaml +trtllm-modeling-v2 --task task.yaml --site site.yaml --workspace W # compiles, verifies, runs +``` + +`--task task.resolved.yaml` keeps working, so existing specs and runs are +unaffected. + +Parsing a natural-language request is the only step that needs a model, and +it is not built: the goal line of the fast path stands in for it. Everything +else (inspection, checks, compile, confirmation) is deterministic code with +unit tests. + +## 11. Implementation + +Done: the `task.yaml` and `site.yaml` schemas; the fast path (`derive.py`); +compile with the section 8 table; the section 6 checks for the checkpoint, +weights present, MTP vs DSpark, divisibility and device capacity, quant x +arch, length, feature members, a hybrid checkpoint's block reuse, perf configs +and goals, run again in phase 0 with the compiled spec's anchor, speculation +and capture-list checks; confirmation beside the request. + +Not done: the request parser and its interactive loop; the memory and +compatibility-matrix checks; the site and budget checks; Step 2 consumers for +the section 8 gaps, one at a time, each in its own change. + +## 12. Open questions + +- Is `task.yaml` per target, or may one request expand into several targets + (one checkpoint, two archs)? +- Does `add_features` build on a delivered branch, or on a target already + merged into `tensorrt_llm/_torch/_experimental/modeling_v2/`? +- Who owns `site.yaml`: one shared file per cluster in the repo, or a user + file? +- Is the perf goal a pass/fail condition for delivery, or a stopping rule for + the tune campaign only? +- With `[onboard, tune]`, does a perf goal the build cannot reach (the delivered + target is far below baseline) stop the run before tune, or is tune expected + to close any gap? diff --git a/agent-flow/agent_flow/workflows/trtllm_modeling_v2/LESSONS.md b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/LESSONS.md new file mode 100644 index 000000000000..6474f48a5934 --- /dev/null +++ b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/LESSONS.md @@ -0,0 +1,589 @@ + + +# Lessons from real runs + +What full onboarding runs of large MoE checkpoints with features, on +multi-GPU Blackwell nodes, taught this workflow. Each lesson gives the +**symptom** an operator or agent saw, the **cause**, and the mechanism that +now **prevents** it. Where the prevention is not in the workflow, **open** +names the gap. + +Most of these presented as a model or target defect while they were +happening, and none of them were. Read this before debugging a target. For +verbatim error text, use [TROUBLESHOOTING.md](TROUBLESHOOTING.md) instead. + +## Substrate and preflight + +### 1. Prove the host before any agent exists +- **Symptom.** Hours into a run, a turn or gate fails with `MPI_ERR_SPAWN`, + a kvCache that cannot fit a sequence, or NVRTC missing `cuda.h`, and the + failure reads as a target defect. +- **Cause.** The substrate was first exercised by the target itself. +- **Prevented by.** + - `preflight.run_preflight` claims a local run's devices (`claim_devices`, + `probe_device_alone`) and checks the checkpoint and the accuracy anchors + (`check_checkpoint`, `check_accuracy_anchors`). + - It also checks `TMPDIR` (`check_tmpdir`) and the multi-rank substrate + (`check_multirank_substrate`). + - `substrate_gate` cold-starts the stock path under the run's own runtime + block. +- **Open.** Nothing proves an agent turn can start before the first one is + dispatched (see lesson 24). + +### 2. A new architecture has no stock path, so the substrate check needs a proxy +- **Symptom.** Preflight, or a stock-path gate leg, stops on "unknown + architecture". That is the condition onboarding exists for. +- **Cause.** The check booted the run's own checkpoint. "Unsupported" was + matched by a marker list that missed refusals from other layers: the + registry, `transformers`' unknown `model_type`, custom code, and + registration shims. +- **Prevented by.** + - `environment.substrate_checkpoint` boots a proxy checkpoint instead. + - `UNSUPPORTED_ARCHITECTURE_MARKERS` / `reports_unsupported_architecture` + cover each refusal path. + - The acceptance gate falls back to a declared `min_absolute` floor and + records the weaker claim. + +### 3. A multi-rank launch inside someone else's allocation has rules; generate it in one place +- **Symptom.** "Not enough slots"; N ranks that each believe they are the + leader and hang without an error; a ZeroMQ IPC path longer than 107 + characters. +- **Cause.** A session is one Slurm task. The launcher prefers `SLURM_*` over + `OMPI_*`. The RPC socket lives under `TMPDIR`, and `sun_path` caps at 107. +- **Prevented by.** + - `environment.resolve_environment` emits the one launch line every prompt + carries, with `--oversubscribe` and the `MULTI_TASK_SLURM_SCRUB` / + `SINGLE_TASK_MPI_SCRUB` scrubs. + - `check_tmpdir` bounds `TMPDIR` at `MAX_TMPDIR_LEN` (61) and shows the + arithmetic. + +### 4. Work that is not an engine must not inherit the engine's launcher +- **Symptom.** A probe exits 127 with no output. A leg that is itself a + launcher runs only on rank 0, and its child engines hang on the outer proxy. +- **Cause.** Every command was wrapped for multi-rank. A spec `PATH` + replaced the inherited one. Launcher variables leaked into child engines. +- **Prevented by.** + - Single-process commands for the survey probe and for witness contrast + legs (`single_process=True`). + - Launcher variables unset in contrast legs (`witness.LAUNCHER_EXPORTS`). + - A session's `PATH` composed with the inherited value + (`infra/substrate/sessions.py`, `MERGED_ENV_VARS`). + +### 5. A skipped or failed measurement must never print like an empty or passing one +- **Symptom.** "0/0 entries certified ✔" against a populated catalog; + "preflight passed" when the substrate gate never ran; a supplied + vocabulary that dispatched nothing, read as "no catalog work". +- **Cause.** Summaries did not separate "measured zero" from "did not + measure", and test fixtures used a different file shape from the real + index. +- **Prevented by.** + - The survey's inventory probe runs on a dry run too, states its error in + the block the plan drafter reads, and renders whatever did land. + - A supplied `--vocabulary` is validated, and its queue rows are + dispatched exactly as a drafted one's; a malformed one stops the run. +- **Open.** The dry run should print that the substrate gate was skipped. + +### 6. Environment knowledge is an artifact, not something passed by hand +- **Symptom.** Every agent re-derives the same host facts (pytest's exit + code, the `TMPDIR` bound, which log line is noise). Hand-edited entry + scripts point at the wrong spool. The container image is imported again + for every allocation. +- **Prevented by.** + - `host_facts.md`, seeded by preflight and appended by roles, which + survives resumes. + - Lessons (`agents/lessons.py`): what a failed task taught, shown to every + later dispatch in its scope, in this run and later ones. A lesson that + works around the workflow's own code (`workflow_defect`) also files a + flow issue in the workspace's `flow_issues.yaml`, the operator's fix + list, and is retired, shown to no one, once the commit that fixed it + (`fixed_by`) is in the running code. + - Session entry scripts rendered from `environment.sessions` + (`trtllm-modeling-v2 session render`, `sessions.render_session_entry`). + - `image_import_warning` and `shares_allocation_with_spool` notes. + +## Plan and survey + +### 7. Every structural config key is a demand, and the demand map is closed +- **Symptom.** The plan omits obligations that surface hours later as + `BLOCKED: vocabulary_gap`, or describes a drafter as the wrong kind. +- **Cause.** The survey had no probe for hybrid layer patterns, attention + sinks, asymmetric head dimensions or routing groups. +- **Prevented by.** + - `survey.read_checkpoint_surface`, with probes such as `attention.hybrid` + and `attention.sink`; unrecognized keys are reported, not dropped. + - `runtime_index.yaml` and a test that every demand an area declares is one + some probe can produce. + +### 8. Anchors are phase zero +- **Symptom.** A unit stalls at its accuracy gate because the reference row + does not exist, or anchor jobs spend hours rediscovering the reference + engine's recipe. +- **Prevented by.** `preflight.check_accuracy_anchors` checks each unit's + `reference_yaml` row for its model key and discriminators before the plan + phase. + +### 9. An independent plan review is worth its cost, and must stay independent +- **Symptom.** The review was run on the drafter's own model family and + recorded as cross-model. +- **Cause.** The reviewer's backend did not travel with an externally + dispatched turn. +- **Prevented by.** + - Per-role routing (`roles.py`; `plan_reviewer` defaults to Codex). + - The CLI warns when the reviewer resolves to the drafter's model. + - Each `dispatch/.request.json` names the role's backend and model. +- **Open.** Record in the run's own record when the answering model cannot + be verified. + +## Smith and catalog + +### 10. Schedule a smith row by the unit that needs it +- **Symptom.** The run closes before core over an entry only a later feature + needs. +- **Prevented by.** `unit:` on queue rows; a feature-only failure is + `deferred`, retried once before its unit, and closes the run as + `smith-blocked:` only if the retry fails too. + +### 11. Certify against the serving envelope, never against observed values +- **Symptom.** Review loops that never converge. UNCOVERED rows found only + at the final audit. Crashes or silent corruption beyond the configuration + the gates ran. +- **Cause.** + - Coverage was keyed on values the gates happened to dump. + - Capture sets were derived per entry rather than per call site. + - Target capacity constants and kernel integer widths were not tied to + engine limits. +- **Prevented by.** + - The serving envelope is what `task.yaml`'s runtime block declares + (`max_batch_size`, `max_num_tokens`, `max_seq_len`, the CUDA-graph + capture batches). A smith reads its step's `call_sites` in + `vocabulary.yaml`, multiplies them by every capture batch, makes every + resulting row a cell, and records the integer-width boundaries the + entry's index arguments cross as hard limits (the smith prompt). + - The reviewer checks each capacity constant against the envelope. +- **Not worth its cost: a delivery audit of runtime values.** An auditor + compared every value the gates dumped against the certified cells, and a + remediation cycle sent what it found uncovered to verify smiths. Across + four runs it handled 85 uncovered rows and changed no target, kernel or + engine behaviour: every fix was a widened cell, a note or a test. It also + cannot see what never reaches a gate. The crashes that mattered (top-k + above 8192, an int32 indexer overflow, a token count above + `max_num_tokens`) were found by running at the envelope's limits by hand, + and by the smiths' own boundary probes at build time. + - The envelope boot (`envelope_boot.py`) serves the delivered target at + those limits on every member. +- **Open.** No check ties a target's capacity constants to engine limits; + that rests on the reviewer. + +### 12. Prove a test can fail +- **Symptom.** Tests and floors pass a deliberately wrong implementation. A + hypothesis is "confirmed" by a count that both hypotheses predict. +- **Cause.** No negative control, and thresholds chosen without measurement. +- **Open.** The kernel skills should refuse a receipt without a negative + control, and floors should sit above the best wrong control. + +### 13. Recurring outcomes get typed decisions, not prose +- **Symptom.** The operator picks redirects, verify requests and spec fixes + out of prose and dispatches them by hand. +- **Prevented by.** Planner events: a finding reaches the planner's next turn, + which turns it into a task. +- **Open.** A typed `spec_change` the orchestrator applies and records. + +## Gates + +### 14. Gate every configuration a target claims +- **Symptom.** A target serving several parallel configurations is green, + but its expert-parallel and attention-DP branches never ran. +- **Cause.** The gate folded a unified target to tensor parallelism, and the + member set was prose. +- **Prevented by.** `members:` with `members_for`; each verdict is named + `@`; boot, accuracy, the witness, acceptance and the envelope + boot run at every member, cheapest first (`order_members_by_cost`). + +### 15. A gate records, and owns, the code it measured +- **Symptom.** A number is about to be credited to a commit it was not + measured on. Rank 0 runs one checkout while the workers run another. +- **Cause.** Gates import whatever is on disk, and MPI workers resolve the + package from their working directory ahead of `PYTHONPATH`. +- **Prevented by.** + - `gates.inspect_tree` records the tree in every verdict and refuses a + dirty target path unless `MV2FLOW_ALLOW_DIRTY_GATE` is set, which marks + the verdict as such. A member's overlay is never written into the + source tree. + - `trtllm-modeling-v2 gate --tree ` measures another tree. The + scheduler's resolve hook records which checkout each process imported, + and refuses a result whose ranks disagree or that did not import the + declared tree (`reason=mixed_checkout`). + +### 16. Gates run with `modeling_v2=require`, and each unit's gates take its own path +- **Symptom.** A feature unit's boot never exercises the feature. The + serving path returns errors no gate saw. A number measured through + fallback reads as the target's. +- **Cause.** Target selection is the LLM-API argument `modeling_v2` + (default `"off"`); the engine reads no environment variable for it, so + exporting one selects nothing and every gate serves the built-in model. +- **Prevented by.** + - The workflow carries the mode to its own harness as + `TRTLLM_MODELING_V2`, and the harness turns it into the argument: + `eval_command` merges `modeling_v2: ` into the member's options + YAML, and the engine driver passes it to the quickstart's `LLM(...)` + (test `test_the_accuracy_command_selects_the_target_through_the_llm_api`). + - The speculator depth comes from the unit's own options YAML + (`resolve_max_draft_len`), and the unit's own LLM-API options file + (`accuracy.llm_api_options`) is used. + - Each unit has its own boot workload (`gates.resolve_boot_spec`); a + multimodal unit is sent an image, and keywords are matched on generated + text only. + - `preflight.check_speculation_support` refuses a speculation algorithm the + gates' launcher has no branch for, instead of letting it run without a + speculator. + +### 17. Read failures from the whole log, and results from explicit markers +- **Symptom.** The gate shows only a generic re-raise. A keyword matches a + log line rather than the answer. Reviewers hand-parse interleaved + multi-rank stdout. +- **Prevented by.** + - Witness token ids and text printed on marked lines + (`TRTLLM_WITNESS_FULL_OUTPUT`). +- **Open.** Classify from the first traceback in the full output, not from + the display tail. + +### 18. An acceptance gate needs a workload that drafts, and a floor that means something +- **Symptom.** Acceptance pinned at 1.00 for every drafter, working or + broken. Floors that pass wrong drafters. Regressions flagged on noise. +- **Cause.** + - The workload's answers ended in 2–3 tokens. + - The floor was taken from the ceiling. + - A 4-prompt comparison used a fixed delta. +- **Prevented by.** + - `accuracy.acceptance` declares the workload (`num_prompts`, default 32, + from the accuracy dataset; or `prompts`/`prompts_file`), `max_tokens` + and `min_draft_iterations`. Fewer than `min_prompts` prompts, or too few + draft iterations, fail as such, not as a low rate. + - Each run is compared prompt by prompt with the previous verdict through + a paired test (`accuracy.acceptance.regression`). + - A floor with no measured source (`floor_source` or + `measured_reference`) is warned about. + - Every unit whose target keeps speculation on is gated, not only the unit + that adds it. + +### 19. Accuracy floors are statistical, and generation steps need a golden check +- **Symptom.** A defect present in every member passes every gate. +- **Cause.** The floor sat several standard errors below the anchor, and + golden checks covered context steps only. +- **Prevented by.** + - `accuracy.floor` with `rule: statistical` (the default): the floor sits + `k` (default 2) standard errors below the anchor. + - A unit whose every member scores more than 1 SE below the anchor is + recorded as a finding (`gates.note_systematic_shift`), and delivery + refuses it (`delivery-failed:findings`) until + `accuracy.floor.acknowledge.` gives the reason. +- **Not worth its cost: two gates.** A scheduler-equivalence gate (overlap + scheduler on vs off, token-identical at batch 1) and a launch-path smoke + through `trtllm-serve` ran 17 and 16 times across the historical runs, + caught no target defect, and each failed once falsely (a Triton autotune + difference, a scorer reading the last number), costing about 2.7 h of + stale-verdict waves. Both are removed. The defects such a comparison did + find were found by agents comparing legs while developing. + +### 20. Add a cheap performance gate +- **Symptom.** An accuracy-neutral change regresses the serving path by + double digits, and only a profile catches it. +- **Open.** One prefill chunk plus one decode step, compared with the + previous tree; fail on more than 5% unless the commit declares a + trade-off. The engine driver's timing mode exists; no gate uses it. + +## Witnesses + +### 21. A feature the standing gates cannot see needs its own kind, skill and witness +- **Symptom.** An output-identical feature, or one trained to stay inside + the accuracy tolerance, passes every gate whether or not it was built. The + nearest existing kind builds the wrong thing. +- **Prevented by.** + - A closed `features:` list, each feature with a skill. + - `WITNESS_GATED_UNITS`: residency, equivalence, replay and kv_quant + witnesses. + - Counters the target must print (`MODELING_V2_WITNESS` under + `TRTLLM_MODELING_V2_WITNESS=1`), where a missing counter is a failure. + - The residency reading is the target's to print (`__DEVICE_MEMORY_MIB__`, + before the KV pool is allocated); a leg without it fails the witness. + - Conflicting kinds refused by the schema (`MTP_NAMESPACE_FEATURES`). + +### 22. A contrast must be real and differ in exactly one variable +- **Symptom.** A witness compares a run with itself. Desynchronized legs + read as transfer defects. Identity fails by construction. +- **Prevented by.** + - The equivalence witness's leg B is a real disaggregated hop: the flow's + own `checks/harness/disagg_leg_b.py` unless the spec names a + `contrast_command`, which then needs its `contrast_env`. + - Leg B gets leg A's workload through `MV2FLOW_WITNESS_*` variables, with + absolute paths. + - `legs_env` pins what both legs share. + - The assembler's turn prompt names the workflow's own entry for + reproducing a gate or witness (`gate_cli.invocation`: + `trtllm-modeling-v2 gate ...`), never a reconstruction of its command. + +### 23. A token comparison must be able to see a difference, and say where +- **Symptom.** Witnesses on 2–3-token continuations; prefix-only text + matches; near-tie flips caused by batching; guessed timeouts. An exact + scheduler-equivalence pair failed, and the verdict blamed stale state, + while the same configuration gave two outputs across processes because + each process timed its own `@triton.autotune` configs. +- **Prevented by.** + - `witness.compare_continuations` compares token ids, then text, then + prefix, and quotes the first difference from both legs. + - Only token ids can pass (`witness.missing_token_ids`), so + `min_generated_tokens` holds whatever the legs print. + - A witness block's `max_batch_size` applies to both legs; at 1, requests + go one at a time. + - A per-leg `timeout`. + - `gates.pin_tuning` makes every leg of a gate run share the AutoTuner's + tactics and Triton's autotuning, emptied at the start of each gate run. + The token-comparing witnesses use it, so a task-level Triton cache is + not needed. +- **Open.** A concurrent, batch > 1 leg beside the batch-1 one. + +## Transport, resume and concurrency + +### 24. When a host cannot authenticate an agent, move the turn and nothing else +- **Symptom.** Every turn fails with zero tokens, as `EROFS ... + credentials.lock` or `Not logged in` after about forty minutes, which + reads as a model refusal. +- **Cause.** A read-only `$HOME`, then a single-use refresh token shared + between copies. +- **Prevented by.** + - `--agent-transport external`: an already-authenticated session runs the + turns, and the stage machine, decision validation and gates stay in the + orchestrator. Staging a writable home with a copied credential store + lasted minutes, for the token reason above. + - Decisions travel as files on both transports, so no turn depends on a + session to host a tool in. + +### 25. Resume and re-entry must not cost work or budget +- **Symptom.** A restart re-dispatches finished or in-flight turns, charges + turns nobody answered, or closes a run a later repair could reopen. A gate + whose compute session died closes the run as if the target had failed. +- **Prevented by.** + - A resumed process adopts a smith's decision file, and waits on an + unanswered twin request instead of asking again. On the external + transport it also adopts every answered turn of a task that had no + outcome yet, and collects the decision where that turn wrote it; + `driver status` marks those turns `ADOPTED`. A turn is consumed once its + task is done, failed or abandoned, so a task reopened later is asked + afresh. + - Verdicts are kept in `contract_status.jsonl` under the fingerprint of + the tree they measured (lesson 32), and a check the contract already + counts as passed on the current tree is not run again. + - A tune task run again, after a restart or as a retry, takes its campaign + back from `state.tune` and asks no turn whose answer is on disk. It never + adopts a failed job receipt, and from its second attempt queues its jobs + under names of its own. + - A unit whose `UNIT_READY` still describes the target + (`_unit_work_unchanged`, git asked from the orchestrator's own process, + never through the gate runner) goes straight to its gates. + - `REMEDIABLE_VERDICTS` (including `assembler-blocked:knowledge_gap` / + `:environment` and `session-lost`) re-enter with the turn budget + restored. + - A gate lost to its session is re-run once a session heart-beats again, + bounded by `environment.session_wait` and `SESSION_LOSS_RERUNS`; past + either bound the run closes `session-lost:`. + - A `smith-blocked` run whose failures are feature-only re-enters. + - A delivered run whose spec grew re-enters at the new unit. + - The spec is re-read on every launch (`_refresh_workspace_spec`). + +### 26. Liveness is per turn, timeouts are per role, and the driver must see the run end +- **Symptom.** Blocking for hours on a turn that committed and then died. A + long role times out while a short one hangs. The driver idles after the + orchestrator exits. +- **Prevented by.** + - A stale `driver.heartbeat` fails the turn fast. + - Each role has its own turn timeout (`DEFAULT_ROLE_TURN_TIMEOUTS`); + `--agent-turn-timeout` overrides them all. + - The orchestrator beats `orchestrator.heartbeat`, and `driver wait` + returns `ORCHESTRATOR_EXITED` when it stops. +- **Open.** Liveness is per driver, not per turn. + +### 27. Cap concurrency by what binds, and let one scheduler own the devices +- **Symptom.** Agent streams stall at high parallelism. GPU jobs collide, + serialize or starve while devices sit idle. Tasks wait for devices that + only gate jobs hold. +- **Prevented by.** + - `driver wait` hands out at most 16 worker turns at once + (`DEFAULT_MAX_INFLIGHT`), whatever `--max-parallel-tasks` (default 16) + allows, the planner's and the critic's beside them, and concurrent + dispatch turns never share a request id. + - Tasks are admitted by capacity: the live sessions' devices less the + `gpus` the running tasks declare, not the devices free at that moment. + An agent turn holds no device; its gate jobs queue in the spools. + - Checks run on one bounded pool shared by every task, and the planner's + turn and the critic's read on threads of their own, so none of them + holds tasks from starting. + - The spool scheduler (`infra/substrate/scheduler.py`): one priority queue, + device slots exported as `CUDA_VISIBLE_DEVICES`, and a waiting + multi-device job holding the devices it needs. + +### 28. No single job, kill or build may take down the allocation or another session +- **Symptom.** One crashed, killed or out-of-memory job takes the whole + session, and every queued job, with it. A rebuild segfaults processes in + another allocation. +- **Cause.** + - Jobs inherited the allocation's PMI context. + - Jobs had no memory or time limit. + - Libraries were rewritten in place. +- **Prevented by.** + - The scheduler scrubs the PMI and Slurm environment from every job + (`SCRUB_PATTERN`), and gives each a wall limit and a resident-memory + limit. It never signals a process that still carries the allocation's + MPI context (the safe-kill rule). + - `infra/scripts/incremental_build.sh` announces the rebuild to every session + and gives every installed library a fresh inode. + - Preflight warns when the orchestrator shares the allocation it + dispatches to (`shares_allocation_with_spool`). + +## Commits, delivery and records + +### 29. The parallelism boundary is the file or the worktree, never a shared index +- **Symptom.** Concurrent smiths' commits cross-attribute. A gate is queued + against a file another agent is rewriting. Unrelated tasks run one after + another because their `touches` name a shared directory. +- **Prevented by.** + - Each task works on its own branch in its own worktree and is rebased + onto the checkout when it finishes; a rebase that conflicts goes back to + its worker to resolve (`integration.rebase_block`). So two tasks are + kept apart only when both name the very same file in `touches`, or + either is `exclusive`. + - `bin/commit.sh` (`committer.install_committer`): commits explicit paths + under a lock and verifies the file list. + - A gate refuses a dirty target path (`gates.inspect_tree`). + - The catalog is its entry files: modeling_v2 keeps no catalog index, so a + smith's commit is one entry and its test, and no file two smiths share. + - Each tune candidate is its own worktree (`tune/worktrees/`). + +### 30. Commits meet the target repository's policy before the first one +- **Symptom.** Dozens of unsigned commits found at delivery. A re-sign under + the wrong identity. Sandbox placeholder files committed. +- **Prevented by.** + - `commit.sh` authors, commits and signs off as the repository's + configured user. + - It refuses untracked symlinks, sandbox placeholders, co-author trailers + and paths outside `commit_policy.allowed_paths`. + - The CLI runs `committer.trial_commit` before the run starts, and + delivery audits every commit since the merge base + (`delivery.check_commit_hygiene`). + +### 31. Run every delivery check that needs no finished target early +- **Symptom.** Every unit is green, then delivery fails on test naming, a + missing accuracy test and a missing test-db entry. +- **Open.** The delivery checks that need no finished target (test naming, + the accuracy test and its test-db listing) run only at delivery. + +### 32. Evidence goes stale; key it to the tree and settings it was measured on +- **Symptom.** Delivery accepts a verdict older than the code and settings + under it, and `HEAD` never receives a full regression. The opposite costs + too: a commit that only splits a test-list entry makes every verdict + stale, and checks re-run on a tree no gate sees differently. +- **Prevented by.** + - Every verdict records a fingerprint of the tree the gates measure + (`workflow._target_marker`): the commit the checkout is at, less + `tests/`, `docs/`, `.github/`, `examples/` and `*.md` + (`gate_relevant`), but with the files a gate reads there + (`gate_inputs`), and with the native libraries' sizes and modification + times. A catalog entry or a native rebuild makes every verdict stale; a + test list does not. + - A check measures a read-only snapshot of that commit, never the + checkout integrations move while it runs (`run/snapshots.py`), and its + verdict records what each command imported: one whose legs imported two + trees, or a checkout no commit pins, is void. A two-leg witness taken on + the live checkout once ran its legs on two commits and passed. + - Delivery evaluates the contract against the fingerprint of the tree as + it is, and a verdict under another is missing + (`delivery-failed:contract`). The delivery tier runs once, on the final + tree. + - Boot and accuracy checks that run one command on one engine against one + criterion share a pass (`contract.equivalence_key`); failures are never + shared. +- **Open.** Nothing invalidates a verdict when a value the spec's + `environment` block sets (a `TRTLLM_*` variable, say) changes. + +### 33. Records are never overwritten, and every claim is traceable +- **Symptom.** Earlier units' decisions overwritten by later ones; duplicate + dispatch numbers; commit hashes in records that no longer resolve after a + re-sign. +- **Prevented by.** + - Decision files are keyed by unit, role and turn + (`decisions/--.yaml`, `progress.decision_path`); a + reused key gets `.2`, `.3` and so on, claimed atomically. + - Journal dispatch ordinals are claimed atomically + (`journal.next_ordinal`). +- **Open.** Commit hashes in the journal and reports are not re-resolved + after a history rewrite. + +### 34. An embargo must be able to name every reachable source of the answer +- **Symptom.** Another engine's implementation of the checkpoint, or this + workflow's own worked example for it, is one read away from every agent, + and the spec cannot name it. +- **Prevented by.** + - `source_embargo` takes repo-relative paths, absolute paths outside the + repo (as trees), and `package:` inside this package; the schema + refuses an entry that does not exist (`task_schema.py`). + - Every role prompt restates the list + (`prompts._common.source_embargo_block`). + - Running another implementation as an oracle stays allowed. +- **Open.** The embargo is a rule in the role prompts only: no tool hook + refuses a read of an embargoed path, and no read is logged. + +## Performance (the tune stage) + +### 35. Hosts differ by more than knobs do; judge on the same host +- **Symptom.** Six unrelated config rungs (a GC threshold, early first + token, stream interval, a token budget, graph sizes, ...) all measured 3% + to 4% worse at concurrency 1, and the ladder rejected a +4.8% rung as a + loss. +- **Cause.** Each rung had run on one of the nodes that were slower for + identical configs; the noise floor was the cross-host spread. +- **Prevented by.** `campaign.Campaign._control_for(host)`: every measurement is judged + against a control of the same config measured on the same host; the noise + floor is the worst same-host spread, never the cross-host one. + +### 36. A profiler that cannot finish is worse than none +- **Symptom.** nsys captures that never wrote a report; the server idle at + 0% GPU forever after `Profiling started at iteration N`. +- **Cause.** The in-process `torch.cuda.cudart().cudaProfilerStop()` holds + the GIL while CUPTI waits for GPU work that other Python threads must + complete (gdb: every other thread in `PyGILState_Ensure`). A driver-level + `cuProfilerStop` did not return on that server either, with node- and + graph-level graph tracing alike. +- **Prevented by.** `bench.py --nsys` defaults to session mode: `nsys launch` + injects, `nsys start`/`stop` from outside the server time the window, and + nothing in the server calls the profiler API. + +### 37. A job's processes include the ones that left its session +- **Symptom.** A server left by a cancelled capture held most of a GPU's + memory during the next measurement. +- **Cause.** `nsys profile` and a server started with `start_new_session` + are re-parented to init when the job's leader exits, and the scheduler + found a job's processes by session and ancestry only. +- **Prevented by.** + - `scheduler.processes_with_marker`: the job's environment markers + (spool, watcher and job) survive `setsid`, so a job's leftovers are + found and stopped with it. + - `bench.py` turns SIGTERM into its teardown and refuses a device that + already has memory in use. + +### 38. Module knobs never reach a target +- **Symptom.** A config rung that selects a kernel (`moe_config.backend`, + `attn_backend`, `use_cute_dsl_*`) measures as noise on the target. +- **Cause.** A modeling_v2 target pins its kernels through the catalog. +- **Prevented by.** The serving config is derived from the workload and + carries no module knob; a kernel choice for a target is a catalog change, + made by a tuner candidate. + +### 39. Measure candidates stacked on the current best +- **Symptom.** Candidates written in parallel from one base, each measured + against the base, would be judged against a control that no longer + exists once the first keep moves the best tree. +- **Prevented by.** `campaign.Campaign._stack`: an `eval-` worktree is + the best tree plus the candidate's commits; a keep that moves the best + tree re-stacks and re-measures the others before their verdict. diff --git a/agent-flow/agent_flow/workflows/trtllm_modeling_v2/README.md b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/README.md new file mode 100644 index 000000000000..14751d85528a --- /dev/null +++ b/agent-flow/agent_flow/workflows/trtllm_modeling_v2/README.md @@ -0,0 +1,1937 @@ +# trtllm-modeling-v2 + +Brings one HuggingFace checkpoint to a gated **ModelingV2** target: a flat, +self-contained forward assembled only from catalog entries, trusted through +accuracy gates rather than through shared abstractions. + +This workflow is the agent half of modeling_v2. +The code half is already in tree at +`tensorrt_llm/_torch/_experimental/modeling_v2/` — the catalog, the targets, +the router and their tests. What lives here is what builds them: the +orchestrator, the agent roles, and the skills those agents load. + +```bash +trtllm-modeling-v2 --task task.yaml --workspace workspace/trtllm-modeling-v2 +``` + +Or let the run derive its task from the checkpoint (the fast path, +[INTAKE.md](INTAKE.md#the-fast-path)): the device, sessions and defaults come +from the cluster's `site.yaml`, the features the checkpoint carries are built +unless the goal leaves them out, and the request is written to +`/task.intake.yaml` and run at once. Its intake checks run in phase +0 and stop the run with every block listed; a confirmation is recorded +(`--confirm`), not awaited. + +```bash +trtllm-modeling-v2 --checkpoint /models/newmodel --site site.yaml --workspace W \ + --goal "mtp disagg@tp1/tep4 tp1 tep4 dep4 isl=8192 osl=1024" +``` + +Start with `--dry-run`. It runs preflight, resolves the launch environment and +stops before the first agent turn — the cheapest way to find out whether this +host can host this run. `task.example.yaml` documents every key of the spec. + +Three companions to this file: **[TROUBLESHOOTING.md](TROUBLESHOOTING.md)** is keyed on +symptoms — the verbatim error you saw. **[BUILD_AND_TEST.md](BUILD_AND_TEST.md)** is +keyed on tasks — building, rebuilding, and running at CPU / 1 GPU / multi-GPU / +multi-node scale. **[LESSONS.md](LESSONS.md)** is keyed on what real runs got +wrong — each lesson's symptom, its cause, and the mechanism in this workflow +that now prevents it, or the gap that is still open. + +**When something breaks, read [TROUBLESHOOTING.md](TROUBLESHOOTING.md) before +you debug the target.** Every entry there read as a model or target defect +while it was happening — `MPI_ERR_SPAWN`, a kvCache that cannot fit a +sequence, NVRTC missing `cuda.h`, a 27-point accuracy regression that was a +protocol mismatch — and every one was the environment. + +## The pipeline + +``` +preflight -> planned run -> delivery + Python planner + workers, checks run by the orchestrator Python + (a `tune` task joins once the core is correct) + | + improver: lessons, flow issues and patch drafts, beside the run +``` + +| step | who | what it produces | +|---|---|---| +| `preflight` | **nobody**: pure Python | the slug, the branch, the resolved launch prefix, the device claim (local runs), the substrate gate, the surface survey, `contract.yaml` (every check the run owes) and `coverage.yaml` (what those checks prove of the request, feature by member; a requested cell left uncovered stops the run unless `coverage.accept` names it) | +| planned run | the planner, the workers, the orchestrator | the planner writes the task graph (`graph.yaml`); the orchestrator starts its tasks as capacity and their files allow, runs each finished task's checks itself, and wakes the planner when something happens. It ends when every required check passes on one tree | + +The first two overlap (phase 0). Preflight's checks take seconds; the +substrate gate, its GPU boot, runs beside the planned run's first turns rather +than before them, and its failure stops the run there, which a re-run of the +same command resumes: a stopped run is left open, not closed, and wakes no +improver. A run started with `--checkpoint` runs its intake checks +first: the request's feasibility checks and the anchor, speculation and +capture-list checks of the spec compiled from it, every block reported at once +before the host is touched. A new run starts with the survey (task `survey`) beside +the planner's first turn, and when the survey lands the orchestrator adds one +kernel task per vocabulary queue row (`k-`, touching the row's +entry and its test), which start at once; the planner wires the rest of the +graph to them and may reshape those not yet running. A row on a shared entry +touches its local variant in the target, not the catalog file (see +[Develop in the target, then promote](#develop-in-the-target-then-promote)). +A shared entry has one row, every call site's shapes as its cells. +| delivery | Python | the gate findings, the trees on disk, and the contract on the target as it is | + +The planner orders the work freely: nothing fixes stages, units or their +order. Each task has a kind, and each kind runs as one role: + +| kind | role | does | +|---|---|---| +| `survey` | `plan_drafter` | maps the checkpoint's computation steps onto the catalog: `plan.md`, `vocabulary.yaml` | +| `kernel` | `smith` | one catalog entry, from a `vocabulary.yaml` queue row or `:`; or a `promote:[,...]` of local variants into the catalog | +| `assemble`, `feature`, `integrate`, `investigate` | `assembler` | the target's core, one feature on it, a merge of parallel branches, a diagnosis | +| `review` | `reviewer` (`plan_reviewer` for `ref: plan`) | reads a diff (or the plan) against its evidence, for what no check sees | +| `tune` | `tuner` turns under Python | the tune campaign; see [Tuning](#tuning) | +| `check` | none: the orchestrator | runs the contract checks in its `satisfies`, with no agent turn | + +Each `kernel`, `assemble`, `feature` and `integrate` task works in its own +worktree (`worktrees/`, branch `mv2//`) cut from the +checkout's head, and commits there through `bin/commit.sh` +(`MV2FLOW_COMMIT_TREE=`). When it finishes, its branch is rebased +onto the head and fast-forwarded in (`run/integration.py`), so the checkout +only ever moves by whole tasks and carries no merge commits. A rebase that +conflicts goes back to the task's worker, up to twice, naming the files and +the head to rebase onto: the worker resolves in its worktree, re-runs its +tests and hands back, and its branch is integrated again. Only then does the +task end `INTEGRATION_FAILED`. An `investigate` task gets a scratch worktree +that is never integrated. + +A task worktree is a sparse checkout: the package, the tests and examples a +worker runs, the directories its task `touches`, and the files at the top +level; a task that touches `cpp/` or `3rdparty/`, or runs `exclusive`, gets +the whole tree. The worker widens its own with the `git sparse-checkout add` +command its prompt gives. No config file is written for it: the cone is kept +in the worktree's own git directory, so a repository whose shared +`.git/config` is read-only or locked gets sparse worktrees too. Each +checkout writes 16 files at once (`checkout.workers`), and how each worktree +was made (sparse, or whole and why) is a journal `integration` event. + +Ready tasks start together, whatever directories they share, unless both name +the very same file in `touches` (a path whose last part has a suffix) or +either is `exclusive: true`; an exclusive task that waits holds back every +task ranked after it, so the run drains to it. A `nice_to_have` task only +gets devices no waiting required task can use. Tasks are admitted by +capacity: the devices of the live spool schedulers (`status.json`, +quarantined devices left out) less the `gpus` the running tasks declare, or +the site's device count without a live session. An agent turn holds no +device, so gate jobs that fill the nodes keep no task waiting. A review, a +`verify` kernel row and a `check` task reserve none (`resources.gpus: 0`; +the orchestrator's own verify rows get none): their tests and gates are jobs +that take their own devices as each starts. + +A `check` task is contract checks and nothing else (re-certifying a member +after a change lands, a stale check run again). Once its dependencies are +done the orchestrator sends its checks to the check pool itself, with no +agent turn; it takes no task slot, its `touches` are empty and its `gpus` 0. +It ends done when they pass, with a receipt naming them; only a failure +wakes the planner, which sends an agent (an `investigate` task) or retries it. + +A task may soft-depend on another (`depends_on_soft`; both land on the +target): it starts once the other has started rather than once it is done, +and is told where each soft dependency stands. When its worker finishes, its +integration waits for each of them to land or end, so its branch is rebased +onto a head that holds what landed and its checks run there. One that ended +without landing is named in a line after the task's decision, which reaches +its receipt and the planner's event, and the planner hears at once which +tasks go on without it. + +When the run binds the cross-model rule (a new run does; one with work done +before it keeps the rules it started under), every task that lands a change +outside the target's own paths needs a required `review` with +`regression: true` after it, and such reviews are batched. A regression +review covers, as it starts, every change to shared code that landed and no +earlier review covered: its batch, recorded as `covers` (`@`) +in `graph.yaml`, whatever its `depends_on`. While a task that changes shared +code runs, a ready regression review waits up to 30 minutes after its +batch's first landing for that change to join; one runs at a time, and what +lands meanwhile joins the next batch; one whose batch is empty when it would +start closes with no turn. A failed review covers nothing, a retry that +lands again is a new change, and so are the guard commits a failed receipt +lands. The planner's request says what the next batch holds, a refused +revision names the open review its new shared tasks should be added to, and +the run is not delivered while a landed change to shared code is in no +review's batch. + +A kernel task's smith is dispatched with its task's goal beside the queue +row's surface and note, and with the sourcing tier on record for its entry: +its own row's, else the `tier` the last smith that landed the entry wrote in +its decision file, else the vocabulary step that maps onto the entry. With +none on record it is asked to name the tier, not told tier 1. + +### Develop in the target, then promote + +Every certification of a shared entry used to write the one catalog file, and +two tasks that declare one file never run at once, so certifications of one +entry ran one after another. A kernel task on a shared entry +(`/`) now lands a **local variant** in the target's own +`ops/`: `ops//__.py` and its test +`test_modeling_v2___.py` beside the target's other op tests +(`vocabulary.entry_paths`). The site is the row's `site` (lowercase words +joined by single underscores), else `r`; a `verify:@` task +names it after `@`, else its task id does, and a task whose `touches` declare +a variant's files lands there. A variant starts as a copy of the entry, +keeps its class and instance name, and the target imports it. Variants touch +only the run's own paths, so no cross-model regression review follows them. +A task that declares the catalog file instead lands in place, as before. + +Before delivery one `promote:[,...]` kernel task +(`onboard/promotion.py`) merges each entry's variants into the catalog: one +entry with the union of their cells, guards and `ARCHS`, one test file, the +target's imports pointed at the catalog, the variants removed, and the merged +tests with the claims gates run once in one process (the verify skill's +`references/promote.md`). `promotion.plan` adds one per entry as soon as it +is due, beside the feature work: no kernel task still to land a variant of it +is pending or running, and the target already imports each of its variants +(or nothing is left to change the target). Every entry due at once goes in +one promote. It declares every file it rewrites (the catalog entries and +tests, the variants, the target files that import them, `modeling.py`), so a +task declaring one of them never runs beside it. An entry with an open +promote waits for it; after one fails the planner retries or reshapes it. An entry `task.yaml` keeps +local (`target.keep_local`) is never promoted, and delivery accepts its +variants. + +A task that finishes leaves a receipt, `tasks/.receipt.yaml` +(`run/receipts.py`): its decision, the first paragraph of its summary, the +commits its branch integrated, a smith's entry and test result, the commit a +cross-model regression review found nothing regressed at (`cleared`), its +decision and result files, and the notes its worker addressed to the +orchestrator or the planner, each bounded. A task that starts carries the receipt of each +task it depends on in its envelope and its prompt, under `## Receipts of the +tasks this one depends on`, so no planner turn goes to restating them in a +goal. + +A worker reports that it finished; it never reports that a check passed. +The orchestrator runs the checks a task claims (`satisfies`) once its branch +is in, all at once on one pool every task shares, as many at a time as the +sessions' free devices can start (at least 2, at most 32; 8 when no session +can be read), so one long gate never holds up the rest of the graph; the task keeps its +slot until its checks are in. A claimed delivery-tier check waits for the +delivery tier. + +A check never measures the checkout, which integrations move while it runs. +It measures a snapshot of the commit the checkout is at when it starts: a +detached worktree of that commit under `/snapshots/`, sparse with +a task worktree's cone, the checkout's native build linked in, its files +read-only. Every check of one commit shares that snapshot; a check leases it +while it runs, and once the checkout has moved past its commit a snapshot no +check holds is removed. Every command of the check runs from inside the +snapshot with it first on `PYTHONPATH` and, on a spool, names it as the job's +tree. The verdict names the commit and, in `contract_status.jsonl`, what each +command (each leg, each engine job) imported the package from, as the spool's +resolve hook and the engine driver print it. A verdict whose commands +imported more than one tree, or a checkout no commit pins, is `VOID`: it is +measured once more, and if it stays void it is recorded as not passing. A +verdict is recorded under the state of the target it measured, so one taken +before another task's integration is stale and runs again once nothing but a +tune campaign is running (it measures only its own worktrees), and a check +the contract already counts as passed on the current state is not run again. + +That state is a fingerprint of the tree the gates measure: the commit the +checkout is at, less what no gate runs or reads +(`tests/`, `docs/`, `.github/`, `examples/`, `*.md` anywhere, and the run's +workspace), but with the files a gate does read there (the quickstart, the +default boot image, and every file in the checkout that the spec's +`accuracy`, `runtime`, `variants` or `environment` block names). Of an anchor +table a unit or variant names as `reference_yaml`, only the entries of the +model keys the run reads count, unless the spec names the table elsewhere +too. The native libraries built into `tensorrt_llm/` and `tensorrt_llm/libs/` +are outside git, so their sizes and modification times are part of it. A +commit that only edits a test list or another model's anchor leaves every +verdict standing; a commit that changes any code, or a native rebuild, makes +every verdict on the target stale. An uncommitted change in the checkout is in +no snapshot: it changes no verdict, and each check it would have reached +warns that it did not measure it. What a check reads from outside the +checkout (the spec as it applies to its unit, the options' content, an +anchor, a prompts file, the logit reference) is its `inputs` digest, recorded +on every verdict: a run relaunched with other inputs runs again the checks +they reach. Boot, accuracy and acceptance checks of different units that run +one command on one engine against one criterion (the same runtime block, +speculator, options content and workload; the same accuracy or acceptance +inputs) share a verdict: a pass of one on the current tree holds for the +others (`SHARED` in the standing). A failure is never shared. + +The boot, logit, acceptance and accuracy checks of one unit at one member +that are ready together start one engine between them (`checks/bundle.py`). +They join a bundle as they are admitted: the first leads it and holds the +one admission slot the bundle takes, the others wait in it holding none. +After a short gathering window (2 s) the leader captures the command each +member's gate would launch, on one snapshot of the checkout's head, and the +quickstart commands whose launch, quickstart and argv agree (but for +`--max_tokens`) form one group. An accuracy check joins that group only when +its `trtllm-eval` argv and options file state every engine setting either +launch path can state the way the quickstart argv does (`max_batch_size`, +`max_seq_len`, the KV-cache settings, the capture list, the speculator, the +topology, the attention and MoE backends) and its options file sets nothing +else; the acceptance gate's `=off` leg and a check without the speculator are +other engines. The group runs as one job (`harness/bundle_driver.py`): the +engine is built once, then each member's workload runs in a section of its +own, the evaluation last, in process. Each gate then judges its own section +as the output of its own command, so every verdict is its gate's own, every +member's carries the one engine record they share, and the verdict names the +bundle (`[bundle : ]` in the journal, `bundle` in +`contract_status.jsonl`). A section that failed, ran past its own time or +never ran (and every section after it) runs again singly, admitted on its +own; so does an evaluation whose engine, once up, was not the one its options +file declares. When the engine could not be built, each quickstart gate takes +that failure as its own job's. The journal says which checks were judged from +each bundle and why any ran singly (`bundle` lines). + +The planner's turn runs on a thread of its own, one turn at a time: tasks +keep finishing and starting from the current graph while it thinks, and its +revision is applied when the turn returns. A failed task or check, a worker's +report, a critique, a failed re-check, or nothing left to run or start wakes +it at once. A task that finished with its checks green waits up to +`--event-window` seconds while anything is in flight, so successes reach the +planner together. A green task that satisfies no check and that an open task +depends on wakes no turn at all: the graph already says what follows it, and +it waits for the next turn something else wakes. What arrives during a turn +goes to the next one, and stays +in `planner_events.json` until it is read, unless the revision that turn +wrote already answered it: a task it retried or dropped, or one it wrote +done at the attempt that finished. The first graph, and any revision that +changes more than half of the open tasks, gets one advisory read by the plan +reviewer (`--no-graph-critic` skips it) on a thread of its own: nothing waits +for it, and its words reach the planner's next turn. + +A failed task runs again under its own id when the planner writes it back as +`pending`. Each such retry is the task's next attempt (`attempts` in +`graph.yaml`, the orchestrator's count), and a revision that retries a task +whose third attempt failed is refused. A retried kernel task's smith is +dispatched afresh rather than handed the failed attempt's decision, and a +retried tune task keeps its measurements apart (see [Tuning](#tuning)). A +retry works in its task's worktree, whose branch is first rebased onto the +checkout's head (`Integration.refresh`): it starts from what landed since, +with its earlier attempts' commits on top. A worktree with uncommitted +changes, or a branch that does not rebase cleanly, is left as it was, and +its worker is told how far behind the head it is. A kernel task that ends +`FAILED_RECEIPT` has its commits integrated when they change only the +entry's own files (the entry or its local variant, its test, a new +`ops//__init__.py`): +the refusal and its guard test reach the target whether or not the task runs +again; anything else stays on the branch for the next attempt. A +task the run's infrastructure failed (its turn raised, it ended +`INTEGRATION_FAILED`, or its tune baseline could not be measured) starts +again once by itself, as the same attempt, and the planner hears why; the +count is kept in `planner_events.json` (`retries`), so a restart grants no +second one. + +Every step checkpoints, so a crash resumes where it was: the graph, the +verdicts and the tune campaign are on disk (`graph.yaml` and +`planner_events.json` are replaced whole, never half-written), and a task +that was running when the process died starts again. Re-running the same +command resumes; `--clean` starts over. + +The bring-up comes first: the first state of the target on which the first +unit's boot and accuracy hold at its narrowest member (tp1) is named the same +way, once, with the branch `-bringup`, the journal line +`BRINGUP DELIVERED ` and a `bringup` line in `correctness.jsonl`. It is a +usable one-rank target hours before every member, feature and variant is +certified. + +Correctness does not wait for performance. The tune campaign is background +work: no task depends on it, and its best branch is reported as an artifact of +the run, never integrated in it. The delivery tier runs once every other +required check holds and no required task is open but the campaign. Full +support is one predicate: every required check holds on the target as it is, +the delivery tier's included, the delivery files on disk check out, every +landed change to shared code has had its regression review, and no local +variant waits for promotion. The first state on which it holds is the +correctness milestone, recorded with no planner turn: the branch +`-correct` (the branch checked out in `trtllm_repo_path`; the +run's own branch name when it is detached) is set to that commit, the journal says +`CORRECTNESS DELIVERED ` with the hours since the run opened, and +`correctness.jsonl` gets a line. Later work that turns a check red or stale +leaves it where it is: each new failure on a later state is journaled as +`correctness stays delivered at : ...`, and the planner's standing and +the run's closing journal sections say whether the target has moved since. +The next state full support holds on moves it forward. Once it holds, a +running campaign is told to wrap up: it dispatches nothing new, drops what it +has not started measuring, and ends with its best kept tree; then the run is +delivered. + +The run closes with one verdict: `delivered`, `tuned` (delivered, and the +tune task finished), `escalated` or `ceiling` (the planner's), `planner-stalled` +(`--revision-cap` revisions without a check turning green), `planner-refused`, +`planner-failed` (five planner turns in a row raised), or a `delivery-failed:*`. + +## Why it is shaped this way + +The predecessor workflow for this job reliably failed the same way: runs spent +their budget deciding *how to launch a job* and re-litigating a plan, and never +reached model onboarding. Four things here exist to prevent that, and they are +the parts worth not "simplifying" later. + +**Preflight is Python, not an agent turn.** The checkpoint parses, the arch +matches the host, `TMPDIR` is sane, a local run's devices are claimed by +`memory.free` and a lone-process probe, a multi-rank run's NCCL/MPI substrate +answers, and the stock path generates a token under the run's runtime block — +all before any model is invoked. + +**The launch environment is declared once and handed to every agent.** +`task.yaml`'s `environment` block resolves into a single command prefix that +every role's prompt carries verbatim. Agents never construct a launcher +command, never probe for one, and never call a test-runner skill. A prefix +that does not fit is a *symptom to report*, not a problem to solve. + +**Rules constrain outcomes, not process.** The contract is the only +definition of done, and only the orchestrator runs it. How the work is split +and ordered is the planner's call; the planner cannot mark a check, relax a +requirement or write code. + +**Gates are exit codes, not opinions.** There is no `--min-score` and no agent +scoring an artifact out of ten. `trtllm-eval` prints a number, the anchor comes +from TensorRT-LLM's own accuracy references, and the verdict is +`measured >= floor`. + +## The gates + +| gate | when | what | +|---|---|---| +| **boot** | inner tier: after every task that claims it | minutes. The declared boot workload through the engine driver; keywords are matched on generated text only, and a multimodal target is sent an image. With a logit reference, the reference's first text prompts go as token ids and the engine's first greedy tokens (4 prompts × 24 by default, `accuracy.boot.greedy`) must be the reference's, up to a near-tie no wider than the logit gate's largest \|Δ log-prob\| limit; a mismatch fails the boot before any benchmark runs | +| **accuracy** | inner tier: after every task that claims it, at every member, or with the logit gate on at the narrowest, the widest and every attention-DP member (and every member of a speculator unit whose boot compares no tokens); `accuracy.benchmark_members: every` keeps every member. `contract.yaml`'s `notes` say where and why | the release criterion. `trtllm-eval` against the anchor, with a statistical floor | +| **witness** | proof tier: each feature boot and accuracy cannot see, every member | two legs that differ in exactly one variable; see [Feature increments](#feature-increments) | +| **acceptance** | proof tier: every unit that keeps a speculator on, every member | `acceptance_length` on ≥ 32 natural-text prompts, paired against the previous verdict | +| **envelope** | delivery tier: once no required task but a tune campaign and what waits for it is open, every member | the finished target at its declared limits: `max_batch_size` in flight, prompts at `max_seq_len`, every capture batch replayed; probes must keep their first tokens from a batch-1 run (`checks/envelope_boot.py`) | +| **serve** | on wherever the spec bounds the server's sequence length (`accuracy.max_seq_len`, or `max_seq_len` in the run-wide options file); `enabled: false` turns it off. Inner tier: every unit, at its widest member; delivery tier: the last unit at one rank | `trtllm-serve` starts the target with the accuracy gate's options file and answers one `/v1/completions` request, then the boot workload's prompts the boot gate judges (each with keywords; an image prompt as a `/v1/chat/completions` request carrying its image); every answer must have text, and hold the keywords the boot gate holds the engine's to. The server and its ranks are stopped either way (`checks/serve_gate.py`) | +| **logits** | wherever `accuracy.logits.reference` names a file, or `logits.reference.json.gz` sits beside the spec; `enabled: false` turns it off. Inner tier: every unit, every member | the reference's fixed prompts through the boot gate's engine, speculator off, with prompt and generation log-probs; at every position whose target is not an image placeholder, the reference's top-1 token is compared: top-1 agreement (pooled), mean \|Δ log-prob\| (per prompt) and the largest \|Δ log-prob\|, each held to the reference's own bf16-vs-fp32 spread times `threshold_factor`; the verdict quotes the worst position (`checks/logit_agreement.py`) | + +All of them run with **`modeling_v2=require`** (the gates export +`TRTLLM_MODELING_V2`, and the harness hands it to the LLM API). Under `auto` a +configuration that misses a target's routing criteria silently gets the +built-in implementation, and a number measured that way reads as +modeling_v2's — the single most expensive mistake available in this package. + +**Every gate attests the engine it ran.** A gate's argv and handed-over +KV-cache keys are what it *asks*; the LLM API and the executor's worker can +change them (a hybrid model with no snapshot policy runs without block reuse, +and its gates still generate the expected text). Once the engine is up, the harness prints one +`MV2FLOW_ENGINE` record (`checks/harness/engine_facts.py`): `modeling_v2`, +block reuse, the Mamba snapshot policy, the KV and SSM cache dtypes (as asked +and as resolved), the CUDA-graph batch list and padding, the speculator and +its depth, `max_num_tokens`, `max_batch_size`, chunked prefill, TP, EP and +attention DP. + +- **Where each value comes from.** Every rank's own worker: the gates put + `checks/harness/rankhook` first on `PYTHONPATH` and set + `MV2FLOW_RANK_FACTS=1`, and its `sitecustomize` has each executor worker, in + whatever process it runs, print one `MV2FLOW_RANK_ENGINE` record as its + engine is set up (the same reading as below, and its rank). A field every + reporting rank carries is held to the declaration on each rank, and a + difference names the ranks. Without those records: rank 0's worker when it + sets its engine up in the harness process (`TLLM_WORKER_USE_SINGLE_PROCESS=1` + at one rank), otherwise `llm.args`, without block reuse and the overlap + scheduler, which only the worker decides; `llm.args` is the request, and it + agrees with any declaration. Block reuse then comes from the log, when a + rank logged turning it off, or from what the engine reused (below), and the + KV pool from rank 0's executor through `LLM.get_kv_cache_capacity`. +- **What a check is about must be observed.** `checks/observables.py` + registers, per feature, the evidence that it took effect and the oracle it + is judged by: a cache dtype as each rank's worker resolved it; chunked + prefill as a request whose prompt, past what the cache resumed, exceeds + `max_num_tokens` (the boot of a chunked variant sends the reference's long + prompt, or a chunk probe of twice `max_num_tokens`); block reuse as the + reuse probe; a speculator as acceptance per prompt. A configuration + variant is about the fields its overlay sets, and its boot fails when one + of them is not observed: read only from `llm.args`, or with a rank that + printed no record. A variant that changes only lossless settings (a chunk + size, block reuse, the capture list) is held to the HF reference's greedy + tokens as its unit is. +- **The comparison.** Boot, both acceptance legs, both envelope legs and every + witness leg the quickstart shim runs compare each field with what the + unit's resolved runtime declares (`gates.engine_declaration`: the gate argv, + the options file's snapshot policy and cache dtypes). Any difference fails + the gate with the fields named. A field the record lacks is reported as + `NOT ATTESTED`, never as agreeing. The accuracy gate runs `trtllm-eval` + under the eval shim (`checks/harness/eval_shim.py`), which prints the same + record; of its configuration block reuse and the cache dtypes are compared, + with the options file's `enable_block_reuse` and `kv_cache_config` dtypes. Leg B of the equivalence witness builds its + engines outside the shim, so their configuration is not attested. A + witness leg the shim runs that asks for block reuse and reused nothing, + although it had a prefix to reuse, fails the witness as it fails boot. +- **What the engine reused.** Every response carries `cached_tokens`, the + prompt tokens rank 0's executor resumed from the KV cache, at every world + size. After the workload the harness prints one `MV2FLOW_REUSE` record + (`checks/harness/engine_reuse.py`): the requests that resumed, the tokens + and positions they resumed at, and how many could have (their prompt shared + more than the engine's shortest resumable prefix with an earlier one: the + first snapshot point of its policy, else one KV block). For a hybrid model + each resume restored a recurrent-state snapshot, and the record counts them + as `snapshot_restores`. The boot, acceptance and envelope drivers then send + the reuse probe (the envelope's in each leg, so also once the engine has + served at its limits): one prompt two KV blocks past that prefix, twice, the + second after the first, on attention-DP rank 0. The probe attests block reuse where no + worker was read, so every member attests all its fields. A unit that asks + for reuse fails a gate where the engine reused nothing although it could + have (`gates.reuse_shortfall`): the probe's second prompt computed from token + 0, or more requests than the engine runs at once could have resumed and + none did. +- **Pools and memory.** Each verdict carries the recurrent slots every rank + logged (`bound #N: recurrent slots`), rank 0's KV-cache + tokens and the free memory of each device once the engine was up (NVML). + The journal line shows them as `[engine 12/12 fields as declared; recurrent + slots 4096 x4; KV 262144 tokens; reuse probe resumed at 256 of 320 + (snapshot restored); min free 23.97 of 276.62 GiB]`, so pool sizes compare + across runs, and an accuracy line as `[engine 1/1 fields as declared; reuse + 1318/1319 requests, 1012224 tokens (recurrent state restored at 768 x1318); + ...]`. +- **Margin.** A device under the larger of `runtime.min_free_memory_gib` + (default 2) and `runtime.min_free_memory_fraction` of the device (default + 0.02) still passes, with `[WARNING: LOW MARGIN: ...]` in its summary. Both + keys work per unit in the runtime block. +- **Runtime facts.** Each gate that passes adds its engines to + `runtime_facts.yaml` (`run/runtime_facts.py`), the latest per gate and + engine leg for its unit and member: the CUDA-graph batches captured and + their padding, the recurrent slots per rank, the KV pool, the free memory, + the speculation depth, TP, EP, attention DP, `max_batch_size` and + `max_num_tokens`. A member's facts span its engines on the target as it is + (or the latest, marked `stale`); the envelope spans every member. A gate + whose own numbers moved since its last pass (not free memory or the KV + pool, which follow the device's other tenants) is a `runtime facts + changed` journal event and a line in the file's `changes`. The planner's + request carries the key numbers and the latest changes; every smith's and + worker's prompt carries the key numbers and the path, and the smith is + told to certify against them. +- **Serving geometry.** Before any kernel is certified, the geometry probe + (`run/geometry.py`, harness `checks/harness/geometry_probe.py`) boots the + engine once per width under the run's runtime block: the narrowest member, + the widest one that is not attention-DP, the widest attention-DP member, + and the narrowest again under the first unit that turns speculation on. It + records under `geometry` in `runtime_facts.yaml` what each serves: the + capture list and padding, the KV pool and its page view (tokens per block, + pages, pages per sequence), the limits, the snapshot policy, and the + context chunks prompts of each probe length ran in (read from the + iteration statistics; the lengths cross the token budget, the snapshot + interval and offsets, and the witness's prompt lengths), plus each + speculative unit's declared draft depth. It runs the stock path; for an + architecture the stock path does not know it boots + `environment.substrate_checkpoint` and keeps only what the configuration + decides (capture list, limits, page size, widths). Every prompt's runtime + facts block carries it with the values a verify row's `cells` state; a + gate's first pass that serves different numbers is a `changes` line. It + never stops the run, a resumed run whose file holds a probe of the same + inputs does not repeat it, and a smith dispatch waits for a running probe + at most 15 minutes. The first time the first unit boots on the target at a + width, that width's legs are probed again on the target (`require`), + beside the other legs; widths whose core boots meanwhile go in the next + probe together. The values the target serves that no earlier probe had + are one planner event naming the entries certified so far, so + re-certification is planned once, as a batch + (`geometry.recertification_note`). +- **Covered certifications.** A `verify` dispatch that states its required + cells (the queue row's `cells`, or a ```` ```cells ```` YAML list in its + task's goal; each cell a mapping of the entry's own spec keys) is closed + `LANDED` without a smith turn when the entry and its test are at the + checkout's `HEAD` and every required cell is a cell of its `CELLS` on the + run's architecture in its `ARCHS` (`run/covered.py`, read statically by + `onboard/coverage.py`). Its decision file and receipt name the commit and + the covering cells. A missing cell, an uncertified architecture, a value + the source does not state readably, or a task branch with commits of its + own dispatches the smith as before. + +The **acceptance** gate exists because boot and accuracy are blind to +speculative decoding: rejection sampling holds the emitted distribution to the +target model's, so a *miscomputed* draft path produces correct text more +slowly. Only speed moves. + +Under `parallel: unified` — the default — the gates run **once per member**, +where the members are the concrete configurations the one codebase serves +(`members:` in `task.yaml`, default `[tp, tep, dep]`), cheapest first. +A unified target gated at a single topology leaves the expert-parallel and +attention-DP branches unexecuted while its Coverage claims them. Every verdict +names its member (`accuracy[gsm8k]@dep4`), and the contract holds only when +all of them pass. `--dry-run` prints the member set and the resulting gate count +before anything is spent. + +### Gate workloads, floors and findings + +Every gate's workload is declared in `task.yaml` under `accuracy`, never sized +from the run it judges. Each key also works inside one unit's entry, for that +unit only. + +- **Accuracy floor** (`accuracy.floor`). + - `rule: statistical` (the default) puts the floor `k` standard errors + below the anchor; `k` defaults to 2, and the SE comes from the + benchmark's sample count (`samples` when the gate cannot read it). + - `rule: tolerance` keeps the fixed `tolerance` width instead. + - If *every* member of a unit scores more than 1 SE below the anchor, the + unit passes but the run records a **finding**. Delivery refuses it + (`delivery-failed:findings`) until `floor.acknowledge.` gives the + reason. +- **Acceptance** (`accuracy.acceptance`). + - The workload is `num_prompts` (default 32) questions from the accuracy + dataset (`dataset`, read offline), formatted with `prompt_template`. A + run may instead declare `prompts` or `prompts_file`. + - Fewer than `min_prompts` prompts fail as "workload too small". A workload + that reaches fewer than `min_draft_iterations` draft iterations fails as + "too short". + - Each unit's run is compared prompt by prompt with the previous verdict + (`regression: {test, alpha, min_effect}`). + - A stock leg that cannot load the checkpoint is recorded per checkpoint + and GPU architecture (`gate_results.jsonl`), and skipped from then on + where a floor or a shallower depth stands in. + - Where the stock path cannot load the checkpoint, a draft depth above 1 + of a speculator whose depth is a setting (MTP, Eagle3; not a DSpark + drafter, whose block is the checkpoint's) is compared with the same + target one depth shallower, on the same + prompts, in the same job: the deeper must accept significantly more + (one-sided paired test at the regression `alpha`). A draft position that + is never accepted adds nothing, so a first verdict is checked rather than + taken as the baseline. + - Per unit, set `min_absolute` with a `floor_source`, or set + `measured_reference`; at depth 1 without a stock path it is the only + comparison. A floor with no measured source is warned about. +- **Serve** (`accuracy.serve`). + - On by default wherever the spec bounds the sequence length the server + sizes its cache for (`accuracy.max_seq_len`, or `max_seq_len` in the + run-wide options file); off where it does not, since the server would + size its cache for the checkpoint's own maximum. The loader writes the + value it chose into the workspace spec. `enabled: false` turns it off; a + run resumed with a spec that does not mention the block owes it from then + on when it can be derived. + - `prompt` (default "The capital of France is") and `max_tokens` (default + 32) are the completion request; `timeout` (default 1800) is how long the + server may take to answer `/health`. +- **Logits** (`accuracy.logits`). + - On wherever `reference` names a reference file. A spec that names none, + and does not say `enabled: false`, gets `logits.reference.json.gz` beside + the spec when that file is there. With it the accuracy benchmark is + sampled (`benchmark_members`, below) and the boot compares the + reference's greedy tokens. The file is made once, in the anchors phase, on + the HF implementation: + `python /agent_flow/workflows/trtllm_modeling_v2/onboard/logit_reference.py + --model --out logits.reference.json.gz [--trust-remote-code] + [--image --image-root ] [--prompts prompts.yaml]`. + It loads the checkpoint twice (bf16, then fp32 for the spread), so give it + room for both, one after the other (`--device-map auto` spreads them). + - The default prompts are three short raw-text prompts and a long prompt of + 8704 tokens, one context chunk past the default `max_num_tokens`; make it + longer than the unit's own `max_num_tokens` (`--long-tokens`) so it + crosses a chunk, and the gate warns when nothing does. The long prompt's + context logits are held in device memory (8704 × vocabulary × 4 bytes), + which the KV-cache fraction must leave room for. + - `prompts` picks reference prompt ids; by default every one the unit can + take (an image prompt only to a multimodal unit; a prompt past + `max_seq_len`, or past `max_num_tokens` without chunked prefill, is left + out with a warning). `threshold_factor` (default 3) scales every limit; + `timeout` (default 1800) bounds the job. + - Lossy configurations (a quantized KV cache, a narrower state dtype) move + the logits by design, so the configuration variants owe no logits check, + their boots compare no greedy tokens unless their own + `accuracy.boot..greedy` turns it on, and their benchmark runs at + every member they name. +- **Where the benchmark runs** (`accuracy.benchmark_members`). + - `sampled` (default): with the logit gate owed, a unit's benchmark runs at + its narrowest member, its widest (the cheapest to start among equals) and + every member whose served engine the logit gate does not run whole: an + attention-DP member (the gate sends one prompt at a time, which one rank + serves), and every member of a unit with a speculator when its boot + compares no greedy tokens (the gate builds the engine without it). The + logit gate at every member proves the rest equivalent to the reference. + Without the logit gate the benchmark runs at every member. + - `every`: at every member regardless. +- **Boot greedy tokens** (`accuracy.boot.greedy`, run-wide or a unit's own). + - On wherever the logit reference is named, logit gate on or off; `enabled: + false` compares none. `prompts` (default 4) of the reference's text + prompts that fit the engine, `tokens` (default 24, at most the + reference's continuation) each. Every boot request then generates its + whole budget past any end token. A level holding only `greedy` declares no + workload, so the workspace `boot.yaml` still applies. + - An image prompt needs a target whose input processor keeps the image + placeholders' vocabulary ids: TensorRT-LLM looks up each prompt token's + log-prob, and an id past the vocabulary fails the request. Leave image + prompts out with `prompts` for such a target. +- **Boot** (`accuracy.boot`, else the workspace `boot.yaml`). + - `prompts` may carry their own `keywords` and `images`, plus `max_tokens`. + - `multimodal.prompts` gives the image prompts. Without one, the + repository's `merlion.png` is sent; it is a git-lfs file, so pull it in + the tree under test. +- **Witness workloads** (`accuracy.witness.`). + - Token-comparing witnesses declare `prompts`/`prompts_file`, `max_tokens` + and `min_generated_tokens` (default 32). A continuation too short to + compare is refused, not passed. + - Both legs print token ids (`[i] Witness token ids: [...]` under + `TRTLLM_WITNESS_FULL_OUTPUT=1`). Text that agrees without them is + refused, not passed. + - The equivalence witness's `prompt_tokens` prompts are prose + (`checks/witness_text.py`): that many words, so at least that many + tokens, and about 16% more under the one tokenizer measured, so leave + `max_seq_len` room. A counting pattern is continued from the attention + pages alone and cannot show a hop that drops the recurrent state. + - **The equivalence self-test.** Before a pass, leg B runs once more on the + longest prompt with `MV2FLOW_WITNESS_ABLATE=state`, which destroys the + state the generation engine received. That continuation must differ from + leg A's. If it does not, the verdict is `INSENSITIVE, not a pass`. + - **Attention DP.** At a `dep` member, leg B is handed + `MV2FLOW_WITNESS_RANK_PAIRS`, one (context rank, generation rank) pair + per prompt covering every pair (`0:0,1:1,0:1,1:0` at dep2). The prompts + repeat on both legs until there is one per pair. Unpinned, every hop + goes from rank 0 to rank 0. + - **Leg B.** Without a `contrast_command` the witness runs the flow's own + (`checks/harness/disagg_leg_b.py`), so no run writes one. It splits the + node's devices into a context and a generation engine of the member's + width (and refuses when there are fewer than twice the member's ranks), + builds both from leg A's quickstart argv through the tree's own + `quickstart_advanced.py`, with the block's `cache_transceiver_config` + (default `{backend: DEFAULT, transceiver_runtime: auto}`) and no overlap + scheduler on the context engine, and sends each prompt context-only, then + generation-only with the `DisaggregatedParams` it got back. It honours + every variable below. Its ablation reaches the generation engine's + processes through a `sitecustomize` first on their `PYTHONPATH` (which + runs the next one, such as a session's resolve hook), and acts where the + executor takes up a request whose transfer completed + (`PyExecutor._prepare_disagg_gen_transmission_complete`). An engine whose + caches it cannot reach (no recurrent layers with conv and SSM state, no + KV manager that names a request's pages) is refused when it is built: + `Witness ablation refused: ...`. + - **What a `contrast_command` honours**, beyond reading leg A's workload + from the `MV2FLOW_WITNESS_*` variables (see `checks/witness.py`), when a + run names its own launcher (then `contrast_env` is required too): + - `MV2FLOW_WITNESS_RANK_PAIRS`: pin prompt `i`'s context-only and + generation-only requests to the `i`-th pair + (`SchedulingParams(attention_dp_rank=..., attention_dp_relax=False)`), + and print `[i] Witness ranks: ctx= gen=`. A hop without that line + fails the witness. + - `MV2FLOW_WITNESS_ABLATE=state`: on the generation engine, after each + generation-only request's transfer completes and before its first + decode step, zero the recurrent state (conv window and SSM state) of + every layer that has one, else the request's KV pages. Print + `Witness ablated: ` for each such request, from any process. + Without that line the self-test cannot run and the witness fails. +- **Pinned tuning.** The token-comparing witnesses run each leg as its own + process sharing the AutoTuner's tactics and Triton's autotuning + (`checks/gates.py`, `pin_tuning`), emptied at the start of each gate run, so two legs of + one configuration do not differ in the last bit. + +Per-prompt baselines and findings are kept in `/gate_results.jsonl`. + +A gate whose compute session was lost under it measured nothing. It is not +recorded and costs no assembler turn: the orchestrator waits up to +`environment.session_wait` (default 1 h) for a session to heart-beat again and +re-runs it, at most three times in a row. Past either bound the run closes +`session-lost:`, and re-running the same command resumes at that gate. + +**Running one gate by hand.** `trtllm-modeling-v2 gate +{boot,accuracy,acceptance,envelope,witness,serve,logits} --task task.yaml +[--unit U] [--member M] [--commit C | --tree T] [--results F] [--workdir W]` +runs a single gate or witness exactly as the orchestrator does: on a snapshot +of the commit the checkout is at, or of `--commit`, kept with the run's own +snapshots when `--task` is the workspace's `task.yaml` (`--snapshots` puts +them elsewhere). With `--tree` it measures that tree as it is (a worktree +with uncommitted work, another checkout), which is how a historical defect is +replayed on the trees before and after its fix; its verdict names that tree +(`git describe`, `-dirty` when the target's path is uncommitted), a witness's +included. Assemblers are pointed at the same command. + +**Not built yet:** +- a performance gate: one prefill chunk plus one decode step against the + previous tree, failing on more than 5% (the engine driver's timing mode + exists); +- a concurrent, batch > 1 leg beside the batch-1 witness legs. + +### Tests after a shared change + +A change to code other models run is followed by a cross-model regression +review, which runs those models' unit tests. It runs only the ones the +change can affect, and reuses every other result from the run's test cache, +`/test_cache/`: + +``` +trtllm-modeling-v2 tests affected --workspace W --head --paths + checks/testcache_driver.py run --cache W/test_cache --repo \ + --selection -- -p no:cacheprovider -rA +trtllm-modeling-v2 tests compare --workspace W --head --paths +``` + +The driver runs one pytest process per test file and records, for each test, +its outcome and everything the process ran on before it ended, each input +with the test it was first seen in (`checks/testcache_driver.py`): the +tree's modules it loaded, the functions that started (`sys.monitoring`), the +files it read, the directories it listed, the paths whose existence it +asked, the module names an import searched for and did not find in the +tree, and the tree's shared libraries. A recorded result holds at another +commit when the test ran there, or when none of that differs: a changed +module counts only through a function body the process ran, unless the +change is outside function bodies (module-level code, a class body, a +signature, a decorator) or in a function a decorator compiles or traces; a +native source change counts for every test of a process that loaded the +tree's library. A test from the first process it starts that can run Python +or build code, or from the first frame `torch.compile` compiled, is reused +only at its own commit, and results hold only in the environment they were +recorded in (interpreter, packages, GPUs, the settings variables). +Cumulative inputs are what make a memoized result safe: a test that only +reads back what an earlier one computed still depends on what computed it. + +`tests affected` writes the tests no result holds for to a selection file (a +whole file where what its collection ran changed, so the test ids may have) +and prints the command. `tests compare` then reads the head's results +against the base's (`--base`, else the commit the last regression review +`cleared`, else the parent of the run's first integrated commit): +`REGRESSED` names each test that passed at the base and fails now, or is new +and fails; `INCOMPLETE` lists what still has to run, at the head or, for a +failure with no base result, at the base; `CLEAN` lets the review approve. +The base tree is run only for a test that fails at the head and has no +result that holds at the base. + +A smith's certifying run of its entry's test goes through the same cache, so +the result is recorded with its commit. A certification whose cells the +entry already lists is closed without a turn only when the cache holds a +pass of that test file that still holds at the head (`run/covered.py`), and +delivery requires the delivered tests to pass at the head (below). A process +counts as running code by its program (`checks/testcache_driver.starts_code`): +`file -b /usr/bin/python3.12`, which `platform.architecture()` starts, names +an interpreter and runs none. + +## The serving envelope + +The runtime block's limits — `max_batch_size`, `max_num_tokens`, +`max_seq_len` and the CUDA-graph capture batches — are the serving envelope: +what the target is claimed to serve. The roles judge against it, never +against the shapes one gate happens to launch: + +- the plan roles mark a vocabulary row `CERT`, not `OK`, when its entry's + contract stops at the gate workloads; +- each vocabulary step's `call_sites` give the rows per request at each + place the target calls the entry, and the smith multiplies them by every + captured batch size to get the surface it certifies; +- the reviewer checks every capacity constant in the target against the + envelope (a constant is derived from an engine argument or carries a + `# capacity:` binding on its line). + +The capture batches are one list per unit (`gates.resolve_capture_list`): +the options file's `cuda_graph_config` when it sets one, else the LLM API's +`CudaGraphConfig` rule over the runtime block's flags. The accuracy gate's +`trtllm-eval` reads it from the options file; boot, acceptance and envelope +get it as `--cuda_graph_padding_enabled` / `--cuda_graph_batch_sizes`, as +they get the draft depth as `--spec_decode_max_draft_len`. Preflight +(`check_capture_lists`) refuses a unit whose runtime flags would still make +the two launch paths capture different lists, naming both. The survey turn +is given each unit's list beside the LLM API default and tune's lists, and +the plan's envelope section states them. + +The KV-cache configuration is one per unit too. The quickstart has flags for +only some `KvCacheConfig` keys; the rest of the options file's +`kv_cache_config` (`gates.resolve_kv_cache_config`), `mamba_state_config` +above all, goes to the boot and acceptance engine driver and to every witness +leg as `MV2FLOW_KV_CACHE_CONFIG`, and the harness merges it into the +`KvCacheConfig` the quickstart builds; a key a quickstart flag sets keeps the +flag's value. Without a Mamba state snapshot policy the engine turns block +reuse off for a hybrid model, so a boot, acceptance or accuracy gate whose log +says it did fails, naming the setting (`gates.reuse_disabled_detail`), and +preflight notes each unit that asks for reuse on a hybrid checkpoint without +one (`check_kv_cache_configs`). + +## Delivery + +Once the planned run's contract holds, delivery verifies, in order: + +1. **Findings.** An unacknowledged accuracy finding closes + `delivery-failed:findings`. +2. **Disk** (`onboard/delivery.py`). The six trees a finished onboarding + touches: the catalog entries the target imports (shared ones under + `catalog/`, its own under `ops/`), their GPU tests, the target, its routing + entry, the accuracy anchor and the test-list registration; `ops:promoted` + (`promotion.check_promoted`): no local variant left in the target or + imported by it, but those `target.keep_local` names. Then + `git:target-committed` and `git:commit-hygiene`, and `tests`: the + target's own tests and those of the catalog entries it is delivered with, + by the newest result in the run's test cache that holds at the head + (`testcache.head_results`): none failing, none still to run. A failure + closes `delivery-failed`. +3. **The contract** (`run/contract.py`) on the target as it is now. Each + verdict records the fingerprint of the tree its check measured (see + [The pipeline](#the-pipeline)); one recorded against another state counts + as missing, so a target that changed after it was gated (its own files, a + catalog entry it imports, the native build) is not delivered on old + evidence. A failure closes `delivery-failed:contract`. Its journal + section also says where correctness stands delivered + (`correctness.jsonl`). + +Records are never overwritten. Decisions are +`decisions/--.yaml`, and a reused key gets `.2`, `.3` and so +on; an external driver must write the decision path its request names, +verbatim. Journal ordinals are claimed atomically. + +The catalog is its entry files. A smith commits one entry file and its test +(a shared entry's local variant, until the promote), and nothing else; there +is no index to regenerate. The survey lists the shared +entries from `catalog//*.py` themselves, and a target's own ops from +its `ops/`. + +## The roles + +| role | runs | one turn does | +|---|---|---| +| `planner` | at the start and on every event | writes the next task graph; `GRAPH_UPDATED`, `DONE`, `ESCALATE` or `CEILING` | +| `plan_drafter` | a `survey` task | the checkpoint→catalog mapping and the plan | +| `plan_reviewer` | a `review` task with `ref: plan` | a paper review of both | +| `smith` | a `kernel` task | exactly one catalog entry | +| `assembler` | an `assemble`, `feature`, `integrate` or `investigate` task | the core, one feature, a merge or a diagnosis | +| `reviewer` | a `review` task | reads one unit's diff against the gate evidence | +| `tuner` | inside a `tune` task | one tune hypothesis, or a roadmap | +| `improver` | when the run closes, on what cost it time (a failed task, a re-certification, a ready task waiting, a no-op planner revision, a reported flow issue, a void verdict); beside the run too in one that measures the workflow (`knowledge: measurement`), on idle agent slots | lessons, flow issues and patch drafts; see [Lessons](#lessons-and-the-improver) | + +Every role is stateless: tasks run concurrently, and what a turn needs from +earlier ones is on disk. Every role runs on the agent that drives the run, at +its model and reasoning effort, unless `task.yaml` routes the role elsewhere +(below). + +### Choosing each role's backend and model + +An optional `agents` block in `task.yaml` routes a role away from the driving +agent: the backend (`claude-code` or `codex`), model, reasoning effort and +extra MCP servers, for every role (`defaults`) or for one (`roles.`). +Use it for a specific need -- a plan reviewer on another provider for a +cross-model review, say -- not by default. It is the shared per-role routing of +`agent_flow.agent_runtime`, the same block `perf-analyze` and `perf-optimize` +read, with the role list and defaults in `agents/roles.py`: + +```yaml +agents: + roles: + plan_reviewer: {backend: codex, model: gpt-6-astra, reasoning_effort: high} +``` + +- A role on Codex gets no Stop hook, because Codex refuses Claude SDK hooks; + its decision file is validated when it is read, like every role's. +- Under `--agent-transport external` each `dispatch/.request.json` carries + the role's configured backend and model as `agent`, and running the turn on + a matching model is the driver's job. +- The spec is re-read on every resume (`_refresh_workspace_spec`), so a + changed `agents` block applies on the next resume. + +Above `AGENT_FLOW_SYSTEM_PROMPT_FILE_THRESHOLD` (64 KiB) the Claude backend +writes a system prompt to `/.agent-flow/system-prompt-.md` and +passes a pointer, because Linux caps one argv string at 128 KiB. The large +per-run blocks (survey, runtime areas, kernel sources) travel in the turn +prompt, which is not on argv. + +## Workspace layout + +``` +/ +├── task.yaml the resolved spec, defaults filled in +├── prompts/.md each role's composed system prompt, snapshotted at every launch +├── contract.yaml every check the run owes (preflight) +├── coverage.yaml what the checks prove of the request, feature x member (preflight) +├── contract_status.jsonl every verdict recorded against it, with the commit it measured and what each command imported +├── graph.yaml the task graph the orchestrator adopted, with every revision's reason +├── graph.proposed.yaml the planner's last proposal, before validation +├── planner_events.json what happened since the planner's last turn, and each task's infrastructure retries +├── correctness.jsonl each commit correctness was delivered at (every required check held there), and the bring-up's (tier `bringup`) +├── tasks/.json each task's envelope: goal, touches, satisfies, priority, its dependencies' receipts +├── tasks/.receipt.yaml a finished task's receipt, handed to the tasks that depend on it +├── runtime_facts.yaml what the engines of passed gates ran, per unit and member; the probe's serving geometry +├── gates/geometry/ the geometry probe's leg specs +├── plan.md a `survey` task's plan +├── vocabulary.yaml the typed step→catalog mapping and dispatch queue +├── surface.md the surface survey, unabridged (preflight) +├── host_facts.md host facts every dispatch is pointed at; roles append below the marker +├── worktrees// each tree-changing task's own branch of the checkout (removed once integrated) +├── snapshots// the read-only snapshot of a commit the checks measure (removed once no check needs it) +├── progress.yaml every agent's typed decision +├── decisions/--.yaml typed decisions, one file per turn +├── journal.md the run's timeline, written as events happen +├── reports/NN--.md each dispatch's prompt and result +├── gates// gate working files (on the shared filesystem, read by the job) +├── gates//bundles/ each gate bundle's spec, `.bundle.json` +├── gates/agent-probe/ preflight's agent probe: each agent's turn and file under /, the last pass in pass.json +├── turns/.* each in-process turn: request.json, system.md, prompt.md, activity.log, result.md, done +├── witness// witness working files +├── gate_results.jsonl per-prompt gate baselines and open findings +├── test_cache/sessions/ every recorded pytest process: each test's result and what it ran on (`tests affected`) +├── tune/ the tune campaign: worktrees, measurements, perf_ledger.jsonl, summary.md +├── lessons.proposed.yaml the improver's last proposal, before validation +├── flow_issues.yaml the workflow defects the improver filed: the operator's fix list +├── flow_issues.reported.jsonl flow issues the driver and the workers reported, one JSON object per line +├── improve/wakes.json what woke the improver and no turn has read yet, and what woke it before +├── improve/patches/.diff the improver's patch drafts, each a fix and its test; never applied +├── boot.yaml the boot gate's prompts and keywords (optional) +├── bin/commit.sh the locking, self-verifying commit helper +├── .commit.lock its lock file +├── .claude/skills/ this run's copy of the packaged skills +├── .agent-flow/ system prompts too long for argv, when there are any +├── dispatch/ the orchestrator's heartbeat and exit note; under --agent-transport external also the turns, each with what the orchestrator did with it (.read, .adopted, .consumed) +├── TURN_INSTRUCTIONS.md the driver's standing instructions, under --agent-transport external +├── inflight.json what the orchestrator holds that a stop would cut short, and how it last stopped +├── restarts.jsonl one line per orchestrator start: what the stop before it cost +├── reload.request.json a reload the driver asked for and the orchestrator has not answered yet +├── reload.json the latest reload: draining, re-executed, reloaded, cancelled or failed +└── .trtllm_modeling_v2_state.json +``` + +The journal and the reports stay here rather than in the repository: they are +evidence *about a run*, and a product tree is the wrong home for it. + +`vocabulary.yaml` is the one artifact with a schema, because the orchestrator +walks it. `onboard/vocabulary.py` validates it and rejects a malformed one: +`OK`/`CERT` rows quote the cell, `note` line or `is_valid` assertion that +decides coverage, a queue `target` is `/` (shared catalog) or +`ops//` (the target's own `ops/`) and has no `__` in its name, +a `site` is a shared row's and no two rows land on one variant, `GAP` rows +name a candidate, `verify` dispatches name the surface, the two chapters do +not overlap, and a `ceiling` verdict carries its evidence. A `survey` task whose +`vocabulary.yaml` does not validate has failed. + +A `GAP` row names its rung on the sourcing ladder, cheapest first: + +| tier | source | the row names | +|---|---|---| +| 1 | a `torch.ops.trtllm.*` op | the op | +| 2 | an in-tree callable under `tensorrt_llm/_torch/modules/**` | the callable | +| 3 | FlashInfer's public API, for a kernel no trtllm op wraps | `flashinfer..` | +| 4 | native torch, composed by this run | what tiers 1-3 offered (`tried`) | +| 5 | a Triton kernel, written by this run | what tiers 1-3 offered (`tried`) | + +The op inventory sees tier 1 only. The surface survey also reads the installed +FlashInfer's public API off disk and records which callables the engine +already references; `surface.md` lists them. A ceiling accounts for all five +tiers. + +Each queue row becomes a `kernel` task when the planner decides; the row's +`unit` and `depends_on` tell the planner which assembly needs it first. A smith +that answers `MACRO` names its constituents, and the planner adds them as +`kernel` tasks of their own (`ref: :`). + +## Package layout + +``` +trtllm_modeling_v2/ +├── cli.py the command line and its subcommands +├── spec/ what the user asks: task_schema, schema_keys, intake/ (Step 1) +├── run/ the run: contract, graph, orchestrator, workflow, state, journal, +│ progress, preflight, receipts, runtime_facts, geometry (the +│ probe), covered (a certification the tree holds), stages/ (one +│ mixin per concern) +├── onboard/ what onboarding knows: survey, vocabulary, runtime_map, delivery, +│ promotion (local variants and their promote), embargo, +│ logit_reference (the logit gate's HF reference, made in the +│ anchors phase), coverage (an entry's CELLS read statically), +│ runtime_index.yaml, kernel_sources.yaml +├── checks/ the contract's checks: gates, gate_records, witness, envelope_boot, +│ serve_gate (and serve_driver, the job it launches), logit_agreement +│ (and logit_driver), bundle (one engine for a member's checks), stats, +│ gate_cli, harness/ (the engine driver the gates launch, and the +│ bundle driver), testcache (and testcache_driver, the recorded pytest run) +├── tune/ the tune campaign: stage, campaign, keep_gate, measure, records, inert, serving_config +├── agents/ roles, prompts/, skills/, lessons (the improver's store) +├── infra/ not modeling: environment, transport, execution/, driver/, +│ substrate/ (the spool scheduler), scripts/, committer, vcs +└── examples/ worked examples for operators; never handed to a run +``` + +## Flags + +| flag | default | | +|---|---|---| +| `--task` | `--task` or `--checkpoint` | the spec. Validated before any agent is constructed | +| `--checkpoint` | `--task` or `--checkpoint` | a HuggingFace checkpoint: derive the task from its `config.json`, `--site` and `--goal` into `/task.intake.yaml` and run it; its intake checks run in phase 0. Re-running the same command resumes; `--clean` derives again | +| `--site` | none | the cluster's `site.yaml`: required with `--checkpoint`; with an intake `task.yaml` (see INTAKE.md) the two compile into `/task.resolved.yaml` | +| `--goal` | none | with `--checkpoint`: features, members and workload in one line, e.g. `"mtp disagg@tp1/tep4 tp1 tep4 isl=8192 osl=1024 tune"`; `-` leaves out one the checkpoint carries (write `--goal=-mtp ...` when the line starts with `-`) | +| `--confirm` [BY] | not confirmed | with `--checkpoint`: record that BY (default `$USER`) confirms the derived task. The run does not wait for it; an edit after the run started is refused until confirmed | +| `--workspace` | `workspace/trtllm-modeling-v2` | | +| `--dry-run` | off | preflight, then stop. Leaves the checkpoint at `preflight`, so the real run re-runs it | +| `--max-parallel-tasks` | `8` | how many agents work at once, shared by priority (`infra/execution/agent_gate.py`): the planner owns one slot and goes first, then tasks on the critical path, other required tasks, the critic and improver, and last a tune campaign's turns, which use only the slots the open feature work leaves idle; a task holds a slot only while its agent works, not while it waits on a soft dependency or runs its checks, and checks (8 at a time) take no agent. Under `--agent-transport external`, `driver wait` keeps at most 8 turns in flight | +| `--revision-cap` | `6` | graph revisions without a contract check turning green before the run closes `planner-stalled` | +| `--no-task-worktrees` | off | tasks edit the checkout directly instead of each in its own worktree; see [The pipeline](#the-pipeline) | +| `--no-graph-critic` | off | skip the advisory read of the first task graph and of reshaping revisions | +| `--event-window` SECONDS | `60` | while tasks, checks or a planner turn are in flight, how long a task that finished with its checks green waits to reach the planner with the events after it; a failure, a worker's report, a critique, a failed re-check, or nothing left to run or start wakes the planner at once | +| `--lessons-dir` | `$MV2FLOW_LESSONS_DIR`, else `~/.trtllm_modeling_v2/lessons` | where the improver keeps lessons across runs; see [Lessons](#lessons-and-the-improver) | +| `--no-lessons` | off | neither show lessons nor learn new ones: an evaluation run that starts from what the workflow ships | +| `--wait-for-spool` MINUTES | 0 | with a `session_spool` environment, wait up to MINUTES for the spool to appear (its session is still importing the image) before validating the task | +| `--skip-substrate-gate` | off | skip the stock-path cold start in preflight. An escape hatch, not a speed-up — see the flag's own help | +| `--agent-transport` | `inprocess` | how a role turn is delivered. `inprocess` runs it here — see [Running the turns in process](#running-the-turns-in-process); `external` writes it to `/dispatch/` and waits — see below | +| `--agent-turn-timeout` | per role | seconds one role turn may take, in process or dispatched, for every role. Without it each role has its own bound: assembler 16 h, tuner 8 h, smith 6 h, plan drafter and reviewer 4 h, plan reviewer 3 h, planner 2 h | +| `--stage` | — | `tune`: tune a target onboarded elsewhere, or re-enter a closed run to tune it further. Same as the `tune` subcommand | +| `--clean` | off | wipe the checkpoint and managed files | + +Subcommands: + +| command | does | +|---|---| +| `trtllm-modeling-v2 tune --task task.yaml --workspace W` | tune a target on its own — see [Tuning](#tuning) | +| `trtllm-modeling-v2 gate --task task.yaml [--unit U] [--member M] [--commit C \| --tree T]` | one gate or witness run standalone, exactly as the orchestrator runs it | +| `trtllm-modeling-v2 perf {campaign,table} ...` | perf tooling outside a run: inspect a campaign, print a curve table | +| `trtllm-modeling-v2 tests {affected,compare} ...` | the test-result cache: the tests a change can affect, and the regressions against the base — see [Tests after a shared change](#tests-after-a-shared-change) | +| `trtllm-modeling-v2 spool {submit,launch,wait,status,cancel} ...` | jobs in a compute session's spool (the same program as `infra/substrate/spoolctl.py`, which a shell without the CLI runs by path) | +| `trtllm-modeling-v2 intake {derive,check,confirm,compile} ...` | Step 1 (INTAKE.md): derive a user-facing `task.yaml` from a checkpoint, check it against its checkpoint, confirm it (recorded in `.confirmed`), compile it with a `site.yaml` | +| `trtllm-modeling-v2 session {render,status,requests,submission,allocate,ack,watchdog} ...` | compute sessions — see [Execution substrate](#execution-substrate) | +| `trtllm-modeling-v2 driver --workspace W {wait,pending,status,claim,release,prompt,answer,instructions,reload}` | the driver side of `--agent-transport external`, and with `wait --sessions-only` the operator loop of an in-process run (see [Execution substrate](#execution-substrate)); `reload` loads changed flow code into the running orchestrator — see [Reloading and relaunching](#reloading-and-relaunching) | + +## Running the turns in process + +By default (`--agent-transport inprocess`) the orchestrator runs every role +turn itself, on an `AgentLayer` (`agent_flow`, Claude Agent SDK), which starts +one `claude` CLI per turn in the workspace. No driver session is needed. The +orchestrator still writes `dispatch/orchestrator.heartbeat`, and +`orchestrator.exit` when it exits, so a driver watching the run's sessions +reads a crash as an exit. + +On a Slurm cluster, launch it from the login node: + +- **From a plain terminal or a tmux session, not from inside another agent's + session.** A turn started there inherits that session's sandbox, and its own + Bash tool cannot start the sandbox the managed policy requires + (`... srt-mux-*.sock ... EPERM`): the model then *reports* that Bash failed, + and the turn ends as if it had worked. +- **Logged in.** The CLI authenticates from the launching shell's `HOME` (and + `CLAUDE_CONFIG_DIR`); run `claude` once there. +- **With a CLI the organization accepts.** The SDK bundles its own `claude`, + which can be older than the organization's minimum version; then every turn + is refused at start-up. `AGENT_FLOW_CLAUDE_CLI` names the binary every layer + launches. Without it a `claude` on `PATH` newer than the bundled one is used. +- **On the model and effort you mean.** Without an `agents` block every role + runs on Claude Code at its default model (`claude-opus-5-5`) and the + backend's maximum effort. Name them for every role: + +```yaml +agents: + defaults: {backend: claude-code, model: claude-opus-5-5, reasoning_effort: xhigh} +``` + +```bash +tmux new -s mv2 # on the login node +export AGENT_FLOW_CLAUDE_CLI="$(command -v claude)" +trtllm-modeling-v2 --task task.yaml --site site.yaml --workspace "$WS" --dry-run +trtllm-modeling-v2 --task task.yaml --site site.yaml --workspace "$WS" 2>&1 | tee -a "$WS.log" +``` + +**Preflight proves the turns work before the run spends anything.** Under the +in-process transport, preflight (a `--dry-run` too) runs one tiny turn on every +distinct agent the roles resolve to (`run/agent_probe.py`): the turn must run +one command through its Bash tool that writes a nonce, the nonce's SHA-256 and +`git -C rev-parse HEAD` into +`gates/agent-probe//probe.out`. Preflight judges the file, never the model's +words: a model that only says it ran the command cannot write a hash it never +computed. A failure stops the run before any planner turn and says, per agent, +what the turn raised or said, the CLI's stderr, where the turn's files are, and +what to do: `AGENT_FLOW_CLAUDE_CLI` for a refused CLI, a plain terminal for a +sandbox `EPERM`, a login for an authentication failure, or +`--agent-transport external`. A pass is kept in `gates/agent-probe/pass.json`, +and a resumed run (a reload, a relaunch, the run after a `--dry-run`) stands on +it for 12 h while the agents, the CLI and its version, the host, the home it +authenticates from, whether this process runs inside another agent's session, +and the checkout path are what they were; the journal says `agent probe passed` +or `agent probe not run`, and why. `MV2FLOW_SKIP_AGENT_PROBE=1` skips it, for +when the probe itself is what is broken. + +**Watching a turn.** Each in-process turn leaves `turns/.*`, written as it +goes: `request.json` (role, task, agent, process, start, bound), `system.md`, +`prompt.md`, `activity.log` (every tool call with its input, text and thinking +block as it streams, and the CLI's stderr), then `result.md` and, last, +`done` (`answered`, `raised`, `timed out` or `interrupted`). `tail -f +"$WS"/turns/*.activity.log` follows them all. During the planned run, +`driver status` lists the turns in flight as +`orchestrator holds: turn `. + +**Bounds.** An in-process turn has its role's timeout (see `--agent-turn-timeout` +above). Past it the turn is stopped, its CLI with it, and it raises +`TurnTimeout`, as an external turn nobody answered does: a task's turn is +retried once as an infrastructure failure, then fails to the planner. + +## Running where the orchestrator cannot authenticate an agent + +On a sandboxed host the spawned CLI cannot write its credential lock, and the +turn fails with `EROFS ... credentials.lock` reported as +`stop_reason='stop_sequence'` and **zero tokens** — shaped exactly like a model +refusal when in fact no agent started. + +Copying the credential store to a writable home does not fix it for long: the +copy shares a single-use refresh token with the parent session, and one of +the two loses within minutes. Use `--agent-transport external`, so an +already-authenticated session runs the turns. + +External transport (`infra/transport.py`) moves the **turn** and nothing else. The +orchestrator writes `dispatch/.system.md`, `dispatch/.prompt.md` and +`dispatch/.request.json`, then blocks until a driver writes +`dispatch/.result.md` followed by `dispatch/.done`. The driver can be a +human, another session or a service; it is trusted with nothing, because every +*decision* still arrives through `collect_decision_file` — validated against the +role's closed enum — and every *verdict* still +comes from a gate the orchestrator ran itself. + +The one thing a driver must get right: **the role's system prompt travels with +the turn**. A driver that delivers only the user prompt is running a different +agent, and its answer still looks plausible. When `task.yaml` routes a role to +a particular agent, the request names it (`agent`); otherwise the driver runs +the turn on its own model and effort. + +## Execution substrate + +`environment.kind` is `local` (this host, the devices preflight claims) or +`session_spool`. On a Slurm/pyxis cluster a run stands on one or more **compute sessions**: +allocations whose main process is the spool scheduler +(`infra/substrate/scheduler.py`, started by `spool_watcher.sh`). Work is a script +renamed into `/pending/`; the receipt is `done/.rc`, written last, +with the verdict beside it in `done/.json`. The orchestrator submits +through `infra/substrate/spoolctl.py`; agents and operators through +`infra/substrate/spoolctl.py submit`. + +**Declare the sessions, render their entries.** `environment.sessions` in +`task.yaml` names each session's spool, role (`gates`, `work`, `any`), node +count, GPUs and time limit. `trtllm-modeling-v2 session render --task +task.yaml --name a` writes the session's entry script from that block, so no +entry script is hand-edited; `session submission` prints the broker call that +starts it, `session status` shows every session. + +**The scheduler** gives each job what its header asks for +(`spoolctl.py submit --gpus N|all --wall 2h --tree ''`): + +- one priority queue; whole-node jobs go before slot jobs, and a waiting + multi-device job holds the devices it needs, so nothing is starved. Within + those, jobs run in the order they were queued: by the pending file's + modification time, and within one second of it by the `#spool-queued:` line + the submitter stamped to the microsecond (a shared filesystem may keep + modification times to the second), never by name; +- device slots, exported as `CUDA_VISIBLE_DEVICES`. Under any scheduler + preflight claims no devices and no prompt names one, and the files a gate's + job reads are written under the workspace, on the shared filesystem; +- a wall limit counted from the job's start, a resident-memory limit, and an + optional host-memory pressure floor; +- a GPU job starts only on devices whose memory the last job released: one + `nvidia-smi` reading as it is about to start hands out devices with at most + `device_max_used_mib` (1024) in use first and holds the job for the rest; a + device still over it after `device_release_wait` (5 min) is quarantined until + a reading finds it released; +- the allocation's MPI context (`PMI*`, `PMIX*`, `OMPI*`, `SLURM*`, ...) removed + from every job, core dumps off. Once a job is over, processes that escaped + its session are found by their marker (spool, watcher and job) and stopped; + exited orphans are collected only when the scheduler adopts them (PID 1 or + a child subreaper); +- names checked against the requester's prefix (each external turn gets one), + and earlier results of a reused name archived rather than deleted; +- **the safe-kill rule**: it stops its own jobs (limits, `spoolctl.py cancel`) + but never signals a process that still carries the allocation's MPI context; + such a job is marked failed and its devices are quarantined until it exits. + +**`--tree `** runs a job with that checkout as its working directory +and first `PYTHONPATH` entry. Every job carries a resolve hook that records +which checkout each process imported; a result whose ranks disagree, or that +did not import the declared tree, is refused (`reason=mixed_checkout`). Gates +declare `trtllm_repo_path` as their tree. Questions about the target's git +state are asked of the checkout from the orchestrator's own process, never +through a spool job (`infra/vcs.py`). + +**Native rebuilds** go through `infra/scripts/incremental_build.sh`: it announces the +rebuild on every session (their schedulers hold new GPU jobs against that +tree, and against every tree whose native build resolves into it: the +snapshots and the task and tune worktrees link `tensorrt_llm/libs` and the +bindings to the checkout's), gives every installed library a fresh inode so +mapped copies are never rewritten, builds, checks the import and retracts the +announcement. + +**No hang needs a human.** A gate's deadline counts from its start; a gate +whose session is lost is withdrawn and re-run once a session is back (see +[the gates](#gate-workloads-floors-and-findings)); a job that prints a +traceback and then goes silent is marked `stalled`; an engine waits for the +host memory and `/dev/shm` it declares (the VMM release lag) and is refused +with the reason if it never comes back; `trtllm-modeling-v2 session watchdog` returns the +moment a session is stale, ending without a successor, or quarantined. + +**A lost allocation.** When Slurm requeues a session onto another node, or an +allocation ends or fails with jobs running, those jobs go with it. The next +scheduler on the spool (the requeued start, or a successor) finds them in +`running/` and queues each again with its place in the queue, once unless the +job says otherwise (`spoolctl.py submit --requeue N`, header `requeue=N`; the +scheduler's `--requeue` sets the session's default). A job that may not run +again (`requeue=0`, or no re-runs left) or whose cancel was requested is +closed out `reason=node_lost` (rc 218). The receipt's `attempts` lists every +allocation the job ran on; `spoolctl.py submit` and `wait` say on stderr when +a job is queued again and wait for that run, and `spoolctl.py status` lists +such jobs. A job whose processes outlived only their watcher, on this host, is +closed out `orphaned` rather than run beside itself. + +**Hand-over.** A session with a `time_limit` writes +`/handover/request.json` `handover_lead` before its end and stops +admitting jobs that would not finish, so they wait in the spool for the +successor that watches the same spool; once the successor is up the old +scheduler retires. The successor is acquired by the driver; the orchestrator +never calls the cluster's scheduler: + +1. `driver wait` returns `SESSION_REQUEST` (or `SESSION_ENDING` when no request + was written); +2. `trtllm-modeling-v2 session allocate --task --name ` + prints the broker call, with `dependency=afterany:` on the allocation + the session's **live** watcher holds (the one that ends last, if two + overlap), and says why. With no live watcher it prints the call without a + dependency and says so; when that allocation already has a successor + acknowledged it says that too, since a second one would start beside it; +3. submit it, then run the `session ack ... --job --after ` it + printed. + +The hand-over state is keyed by the allocation it is about. An ack is +`/handover/acks/.json`: written before the request, it +answers the request when the scheduler writes it, and an early ack for the next +allocation is a file of its own, so it never replaces the one that answered +this request. A request stands only while the watcher that wrote it +heart-beats and no successor is up: when the successor starts it moves its +predecessor's request and acks to `handover/archive/` (its `watcher.status` +says so), and `driver wait` archives any request no live watcher stands +behind (an ended allocation's, or an earlier start of a requeued one) and says +so on a `note:` line after its event; `session requests` lists those as +ignored. A watcher whose request left the disk while it still needs a +successor writes it again. + +A job sent to a spool that will not start it soon (in hand-over, ending before +the job's wall limit, or with no live watcher) gets a `warning:` from +`spoolctl.py submit` naming the spools of the same kind (`gates-*`, `work-*`, +...: same directory, name without its trailing number) that can take it now; +`spoolctl.py submit --pick` sends it to the one that can start it soonest, and +the orchestrator's gate pool passes over such a session the same way +(`infra/substrate/placement.py`). + +**The operator loop of an in-process run.** With `--agent-transport inprocess` +no turn reaches a driver, but the sessions still need their hand-overs and a +watchdog, and someone must notice the orchestrator stop. Run, in a session that +holds the broker tools and on the orchestrator's host: + +```bash +trtllm-modeling-v2 driver --workspace wait --sessions-only +``` + +It never returns `NEW` and returns everything else: `ORCHESTRATOR_EXITED` +(read its log; relaunch it unless the run closed), `RELOADED`, +`RELOAD_OVERDUE`, `RELOAD_FAILED`, `SESSION_REQUEST` (the hand-over above), +`SESSION_STALE`, `SESSION_ENDING`, `SESSION_QUARANTINED` (look at that +session), or `TIMEOUT` (call it again). Its status block shows the turns in +flight as what the orchestrator holds (`turn `). Such an +orchestrator heart-beats like an external one, so a crash reads as +`ORCHESTRATOR_EXITED` from any host. Without a fresh heartbeat (an +orchestrator from before heartbeats ran in process), `wait` reads the run's +ledger (`inflight.json`) and the workspace lock (`.orchestrator.lock`, held for +the whole run): a held lock is a running orchestrator, a run that closed says +so in the ledger, and a free lock on the ledger's own host is one that stopped +or crashed; on another host `driver status` says `unknown` with the host to run +it on. + +**Multi-node.** A session with `nodes: 2` runs one scheduler per node; +`infra/scripts/mn/mn_mpirun.sh` launches one MPI job across them through the +schedulers, with no sshd or srun in the container (`infra/scripts/mn/README.md`). + +**Executors.** `infra/execution/` turns a command into a result under the resolved +environment: `commands.py` (local), `spool.py` (a spool job), and `turns.py` +(a role turn, in process or through `infra/transport.py`). A gate goes to the +gate sessions; one none of them would start at once goes to another session +that would (devices free, nothing queued), and failing that to a tune dev +session (`perf.dev_spools`, less any in `perf.spools`) while nothing runs or +waits there. `check_capacity` counts by the same rules how many more checks +would start at once (file reads only; never under 2 in flight nor over 32, and +8 when no session can be read), and `capacity.CheckAdmission` holds a check +until one would. + +**The external driver.** With `--agent-transport external`, the driver session +runs `trtllm-modeling-v2 driver --workspace wait`, which returns `NEW ` +(a worker's within an in-flight cap of 16, the orchestrator's default task concurrency; the planner's turn and the critic's read at once, ahead of any worker's and beside the cap), `ORCHESTRATOR_EXITED`, `RELOADED`, +`RELOAD_OVERDUE`, `RELOAD_FAILED`, `SESSION_REQUEST` or a +session alert; `claim`, `prompt `, `answer` and `status` complete the loop, +`instructions` renders the standing `TURN_INSTRUCTIONS.md` from `task.yaml`, and +`reload` asks the orchestrator to reload itself (next section). +A relaunched orchestrator adopts every unanswered turn rather than asking again, +and every answered turn of a task that had no outcome yet: the task's first +dispatch in the new process collects the decision where that turn wrote it, +and `driver status` marks the turn `ADOPTED`. A turn is consumed once its task is done, failed or +abandoned, so a task the planner reopens is asked afresh; the planner's, +critic's and improver's answered turns are never adopted. + +## Reloading and relaunching + +The orchestrator imports the flow's code once, when it starts: a fix to the +loop, a stage, a prompt or the contract reaches a running run only through a +new process. What an agent runs (the gate CLI, `spoolctl.py`) and the spool +jobs a tune campaign submits are processes of their own, and read the code as +it is when they start. + +**Reload, not kill.** `trtllm-modeling-v2 driver --workspace reload` writes +`reload.request.json`; the orchestrator notices within its poll interval +(5 s) and drains (`run/drain.py`): + +- it starts no task, no integration, no contract check and no planner turn; +- what is in flight runs to its end: an integration (and a worktree being + made), a contract check that started, and the planner's turn or the critic's + read. A check that would start meanwhile waits for the new process; +- worker turns on the external transport go on: the new process adopts every + turn the old one asked, answered or not; +- on the in-process transport a turn dies with its process, so the drain also + waits for every turn in flight, a tune campaign's included. + +Once nothing is left, the orchestrator checkpoints, journals a `reload` event +with what it waited for and for how long, and re-executes itself +(`os.execve` of its own interpreter command line, under the environment it was +started with): same pid, same log, same arguments but for `--clean`. The new +process resumes from the checkpoint; when its loop runs, `reload.json` says +`reloaded` and `driver wait` returns `RELOADED`. Its preflight, like that of +any resumed run, does not run the substrate gate again when the last pass +(`gates/substrate/pass.json`) is under 12 h old and was recorded for the same +resolved environment, checkpoint, runtime block, options file and checkout +path; the journal says `substrate gate not run` or `substrate gate passed`, +and why. Delete that file to have the next start prove the substrate again. + +The drain waits `--timeout` seconds (default 1800). Past it, it journals what +it still waits for, `driver wait` returns `RELOAD_OVERDUE` once, and it keeps +waiting; `driver reload --force` re-executes at the deadline without what is +left (a cut-short check runs again in the new process), and `driver reload +--cancel` withdraws the request. `driver status` shows a request not yet +taken up, a drain and what it waits for, and what the orchestrator holds right +now. A re-exec that fails (`RELOAD_FAILED`) leaves the run going on in the old +process, on the old code. The planned loop is the only place a reload is +answered: a request made during preflight waits for it, and one left when the +process exits is answered by the next start. The tune stage's own measurements +are not drained: a campaign resumes from its ledger. + +**What a stop cost.** Every start reads `inflight.json`, the ledger the previous +process kept of what it held, and journals `restart ()`: `reload` (nothing +cut short), `hard-stop` (the checks, integrations, worktrees and in-process +turns it cut short; the check jobs it left queued or running in the spools, +which nothing reads any more while their checks run again; the tasks set back +to pending; and on the external transport the turns it left: still with the +driver, answered and never read, or answered for a task with no outcome yet), +`closed` (a run re-entered after it closed) or `unrecorded` (a flow from before +the ledger). Each start is a line in `restarts.jsonl`. + +**Relaunch** only when there is no orchestrator to reload (`driver status` says +`exited`): run the same command again. It adopts every turn its predecessor +asked, the checks a hard stop cut short run again, and an integration a stop +cut short is integrated again from the task's branch. + +## Commits + +Agents commit only through `/bin/commit.sh` (`infra/committer.py`). It +commits and signs off as the repository's configured user — so the DCO +sign-off email always equals the author email — takes explicit repo-relative +file paths only, and refuses untracked symlinks (a worktree's link to the +native build), sandbox placeholders (`.mcp.json`, `.claude/`), the run's own +workspace and anything outside `commit_policy.allowed_paths`. The CLI refuses +to start without an identity, after one trial commit in a throwaway repository +proves this environment's git writes what the rules require. + +The identity is the target repository's configured `user.name` / +`user.email`, or `MV2FLOW_GIT_NAME` and `MV2FLOW_GIT_EMAIL` when both are +set. The optional `commit_policy` block in `task.yaml` sets: +- `allowed_paths` (default `tensorrt_llm/ tests/ cpp/ examples/`); +- `allow_co_authors` (default false); +- `base_ref`, the merge base delivery audits from (default `origin/main`, + `upstream/main` or `main`). + +A refused commit prints `commit.sh: REFUSED` and the reason; see +TROUBLESHOOTING.md. Delivery runs the same rules over the branch: every commit +since the merge base that touches the run's allowed paths must be signed off by +its author, with no co-author, symlink or placeholder (`git:commit-hygiene`). + +## Source embargo + +`source_embargo` in `task.yaml` lists paths no role may read. Entries take +three forms: +- a repo-relative path, for an implementation of this exact checkpoint in the + tree; +- an absolute path outside the repo, for a sibling engine's checkout (a + directory embargoes its whole tree); +- `package:`, inside this workflow package, for its own worked + examples, e.g. `package:examples/.md`. + +The schema refuses an entry that does not exist (`spec/task_schema.py`). The embargo is +a rule, not a tool hook: every role prompt states it and lists the paths +(`agents/prompts/_common.py`, `source_embargo_block`), and the driver's turn instructions +repeat it. Running the other implementation, as an accuracy oracle, stays +allowed. + +No blanket rule keeps agents out of TensorRT-LLM's per-model definitions: the +zoo is an ordinary read. What carries the duty instead is the `modeling-core` +invariants — a copied forward cannot pass the closed-vocabulary audit, is not +flat, and leans on shared helpers a target may not have — plus the rule kept +verbatim in all three writing roles: *read to understand an interface, never +to copy a structure*. + +## Tuning + +Onboarding makes a target *correct*; tuning makes it *fast* without making it +less correct. Perf is measured, never a release gate — but every kept change +passes the accuracy gate, and every number traces to a label dir. + +With `perf.enabled: true` the planner must put one `tune` task in the graph. +The orchestrator holds it back until every inner check of the first unit +(the core's boot and accuracy) has passed, on the current target or an +earlier one, with no failure since, then starts it on whatever devices the +build work leaves idle. The task pins the commit the target is at when it +starts and works in its own worktrees, so feature tasks landing meanwhile +never reach a measurement, and an integration that leaves the core's passes +stale does not hold it back. When it ends, its report names the best branch; +merging that branch into a target that has moved since is an `integrate` +task the planner adds. A failed tune task is reported to the planner with +what failed and where its job log is, not as a failed run: delivery does not +wait on performance. Nor does correctness: final reviews and the tasks that +claim delivery-tier checks never depend on the tune task or on integrating +its branch, the delivery tier runs beside the campaign, and integrating its +best branch is a later task whose checks run again on the merged tree (see +[The pipeline](#the-pipeline)). + +```bash +trtllm-modeling-v2 tune --task task.yaml --workspace W # a target onboarded elsewhere, or re-enter a closed run +trtllm-modeling-v2 perf campaign status --spec tune.yaml # tooling: candidates and verdicts +trtllm-modeling-v2 perf table --baseline A A B C # a curve table of label dirs +trtllm-modeling-v2 perf ab --a A --b B --a A2 # judge an A/B as the campaign does +``` + +The campaign has two layers (`tune/`): + +| layer | who | what moves | judged by | +|---|---|---|---| +| **ground truth** | Python | a perf smoke, then the baseline (`modeling_v2: require`), its identity repeats and the stock reference (`"off"`) at once over the perf nodes; the repeats' same-host spread = the noise floor | -- | +| **campaign** (`campaign.py`, `stage.py`) | `tuner` turns | the forward: fusion, ordering, overlap, kernel source; one candidate = one worktree = one hypothesis | e2e A/B + the keep gate, in Python | + +Both measure at one serving config. Without `perf.identity_configs` it is +derived from the workload (`serving_config.py`): `max_batch_size` is the +highest concurrency, `max_seq_len` is ISL + OSL (+ the draft length), +`max_num_tokens` holds a whole prompt plus one token per other request +(a power of two in [8192, 32768]), chunked prefill is on only when a prompt +does not fit it, and CUDA graphs capture every batch size up to the maximum, +with padding. A target's own switches are tuner hypotheses, not defaults. + +- **The keep rule** (`records.compare`): a WIN is at least `max(min_gain, + noise_factor x spread)` better on some headline metric (tok/s/user at the + lowest concurrency, tok/s/GPU at the highest) and no worse than + `max(1%, noise_factor x spread)` on any. The spread is measured on the tree + under test: the same-host controls of the current best tree (for the + baseline tree, also the ground runs of a member that one host ran twice), + per site, the widest any host's controls read, one spread for both metrics + of a site. A floor measured on an older + best is never used: a keep can make a point far noisier than it was on the + tree before. Every measurement and decision is appended to + `perf_ledger.jsonl`. +- **tok/s/user** (`tune/bench.py`). A point is one closed loop: the client + sends `concurrency` requests at once, then one more as each finishes. How + the server admits that first wave sets how every later one arrives, all at + once again or staggered, and the two patterns trade TTFT against TPOT at the + same throughput and the same mean end-to-end latency. `tok_s_user` is the + output tokens over the end-to-end latency, both summed over every request + of the measured run, which Little's law ties to the point's throughput + whichever pattern a run falls into; at concurrency 1 it agrees with the + mean of each request's rate. Each summary row names that basis + (`tok_s_user_basis`) and keeps that mean, benchmark_serving's figure, as + `tok_s_user_per_request`: it grows with the spread of latencies a pattern + sets, and at an attention-DP point reads a blocked run a few percent below + a staggered one. A discarded warm-up wave would not settle the pattern (the + measured run starts its own wave), nor would a window past the first wave + (the pattern lasts through every wave). A summary whose rows name no basis + is restated on this one when it is read (`records.py`). +- **Ground truth over every perf node.** The baseline, its + `perf.noise_repeats` repeats and the stock reference are measured at once. + Each perf config (member) is a run of its own: every idle node takes its + share of the runs left as one job, never more runs than one label has, and + first the members its host has not run, so a member's repeats run on other + hosts than its baseline, beside it. While the repeats have members a node + has not run, it takes one run of each: the runs one host made of every + member are a control of that host (the label itself, or `t-ground--`, + links to the runs it takes), and the campaign adopts each as that host's + first control. A label measured across nodes records the host of each + member (`hosts` in its `meta.json`) and is no host's control. The noise + floor a keep is judged under stays a same-host one: what a member's runs on + one host spread by; with no member run twice on one node, each node's + controls (A, A') set it. The spread across hosts is recorded beside it + (`cross_host` in the ledger's `noise_floor`, `state.tune.cross_host_spread`, + and the journal's `[tune] noise floor` line) and judges nothing. The jobs in + flight are checkpointed with their runs (`state.tune.queued_runs`), so a + restart adopts them; a run that fails is measured again from where its job + stopped. The wrap-up's `t-final` spreads the same way. +- **A, B, A' on one idle node.** Every measurement starts on an idle node: + nothing runs on it, whoever's job, and the last job there ended + `perf.campaign.idle_settle_s` (default 120 s) ago: a point measured right + after another job reads low by more than the floor. Per best tree, each + perf node measures a control (A) before the first candidate (B) it measures + and a second (A') right after; a candidate is judged against the mean of its + node's controls. The measurement of a keep is the new best's first control + on its node. +- **Measured again before it counts.** A LOSS whose every loss is within twice + its loss floor is what one disturbed run looks like: the candidate is + measured once more on the same node, then a control (B', A''), and judged + on the means of all of them. A node that leaves before its controls are in + sets the candidate's measurement aside (`