Restructure OPD package for the thinking-on MOPD experiment

.gitignore

@@ -7,6 +7,7 @@ __pycache__/
 .ruff_cache/
 .venv/
 site.local.json
+config/*.local.json
 work/
 checkpoints/
 *.log
@@ -15,3 +16,4 @@ checkpoints/
 *.safetensors
 *.whl
 .env*
+article/.backups/

README.md

@@ -1,64 +1,63 @@
-# Multi-teacher on-policy distillation
+# On-policy distillation
 
-This repository preserves the scripts for three completed experiments with Miles, Prime-RL, and Slime.
-Each experiment trains Qwen3.6-35B-A3B with frozen Countdown and Graph Coloring teachers.
+On-policy distillation (OPD) experiments across five training frameworks: Miles, NeMo-RL, Prime-RL, Slime and verl.
 
-**The original experiments completed. The portable package has passed local checks, but it has not been rerun on GPUs.**
-The runtime image and converted model checkpoints are external prerequisites. This repository is not a self-contained container distribution.
+The current experiment is multi-teacher OPD (MOPD): Qwen3.6-35B-A3B learns from frozen reasoning_gym GRPO teachers for `caesar_cipher` and `simple_geometry`. Thinking is on everywhere (training, teachers and evaluation), with a 32k context and up to 30,720 response tokens.
 
-## Start here
+## Status
 
-1. Read [the setup guide](docs/SETUP.md), including the runtime and checkpoint requirements.
-2. Copy `site.example.json` to `site.local.json` and supply your node names, addresses, and paths.
-3. Run `python3 tools/prepare.py slime --site site.local.json --output /shared/opd-runs/slime-01` on the shared filesystem.
-4. Complete the framework setup and inspect the command with `python3 tools/submit.py /shared/opd-runs/slime-01 --partition YOUR_PARTITION`.
-5. Add `--submit` only when you intend to allocate 16 GPUs. The command remains attached and writes `allocation.out`.
+- The single-teacher `caesar_cipher` variant (`--domains caesar_cipher`) has run on GPUs in Miles, Prime-RL, Slime and verl. NeMo-RL deadlocked before its first update. See the [benchmark report](docs/CAESAR-OPD-2026-09-27.md).
+- The two-teacher MOPD experiment ran in Miles, Prime-RL and verl on 16 B200 GPUs (benchmarks and telemetry in [`docs/mopd-2026-09-29/data`](docs/mopd-2026-09-29/data)). Miles also ran it on 24 H100 80 GB GPUs with the `h100x3` layout: after 20 updates the student scores 0.773 avg@8 on caesar_cipher and 0.998 on simple_geometry (teachers 0.703 and 0.996). See the [H100 MOPD report](docs/MOPD-H100-2026-10-07.md); benchmarks and telemetry are in [`docs/mopd-h100-2026-10-07/data`](docs/mopd-h100-2026-10-07/data).
+- A previous experiment (Countdown and Graph Coloring, thinking off) completed in Miles, Prime-RL and Slime on two nodes of eight B200 GPUs. See [provenance](docs/PROVENANCE.md).
 
-Replace `slime` with `miles` or `prime-rl` to prepare another framework. Use a new campaign directory for each experiment.
-Preparation clones source and writes files, but does not submit a GPU job or download model weights.
-An existing campaign directory is never overwritten.
+Results are single observations and do not rank the frameworks. The implementations differ in objective, scheduling, precision and checkpoint content, and maximum useful GPU utilization was not demonstrated.
 
-## What is included
+## Quick start
 
-| Directory | Contents |
-|---|---|
-| `frameworks/miles/` | The fully async candidate-based objective, domain-balanced buffer, scoring retries, and instrumentation. |
-| `frameworks/prime-rl/` | Native per-source OPD, configuration generation, task environment, and the frozen-vision compatibility patch. |
-| `frameworks/slime/` | Native batched async OPD, teacher routing, baseline evaluation, timing, and runtime compatibility patches. |
-| `data/` | The exact compressed puzzle files and hashes of their uncompressed contents. |
-| `tools/` | Preparation, guarded Slurm submission, and source reconstruction checks. |
+The runtime image and converted checkpoints are external prerequisites; the [setup guide](docs/SETUP.md) covers them.
+
+To rerun the whole `caesar_cipher` benchmark (every framework, checkpoint benchmarks and the telemetry report), fill in `config/site.local.json` and run this on a Slurm login node:
 
-The files in `campaign/` are templates. Values such as `@BASE_MODEL@` are replaced during preparation.
-Run the prepared copies rather than the templates. Each manifest records the upstream revision, patch digest, and original archive digest.
-The [provenance document](docs/PROVENANCE.md) identifies packaging changes and scientific differences.
+```bash
+PARTITION=YOUR_PARTITION tools/reproduce.sh all
+```
 
-The separate [Tinker campaign](frameworks/tinker/README.md) trains hosted puzzle
-teachers and a 40-update student through the Tinker SDK. It retains remote
-checkpoints and local evidence without downloading model weights. See its own
-protocol for LoRA, objective, and teacher-comparability differences; it does not
-use the Slurm preparation or submission tools below.
+See [One command: `tools/reproduce.sh`](docs/SETUP.md#one-command-toolsreproducesh) for its settings.
 
-## Fixed experiment
+To run one framework by hand:
 
-The experiments use 40 optimizer updates, 128 prompts per update, one response per prompt, learning rate 1e-6, and a 256-token response limit.
-The GPU allocation contains eight trainer GPUs, six policy GPUs, and one GPU per frozen teacher.
-Each experiment uses two nodes with eight NVIDIA B200 GPUs each. Evaluation uses 512 development prompts per domain with thinking disabled.
+1. Read the [framework overview](frameworks/README.md) and the [setup guide](docs/SETUP.md).
+2. Copy `config/site.example.json` to `config/site.local.json` and fill in your nodes, addresses and paths.
+3. Prepare a campaign:
+   ```bash
+   python3 tools/prepare.py slime --site config/site.local.json --output /shared/opd-runs/slime-01
+   ```
+4. Print the allocation, then add `--submit` to allocate 16 GPUs:
+   ```bash
+   python3 tools/submit.py /shared/opd-runs/slime-01 --partition YOUR_PARTITION
+   ```
 
-The implementations have different objectives, scheduling, precision choices, and checkpoint content.
-These experiments are individual observations and do not establish a framework ranking.
-The sampled utilization did not demonstrate maximum useful utilization. No GPU profiler was enabled.
+Replace `slime` with `miles`, `prime-rl`, `verl` or `nemo-rl`. NeMo-RL needs three nodes and `config/site-nemo-rl.example.json`. Preparation never overwrites a directory, submits a job or downloads weights.
+
+## Layout
+
+| Path | Contents |
+|---|---|
+| `shared/` | Recipe, data checks, verifier, Slurm and container runtime, telemetry |
+| `frameworks/` | Per-framework code, each with its pinned `upstream/` revision and patch, if any |
+| `hosted/` | Tinker and Fireworks campaigns, each with its own protocol, and the hosted OPD/MOPD loop ([README](hosted/opd/README.md)) |
+| `data/` | Compressed task rows and their hashes |
+| `config/` | Example site and hosted-account files ([README](config/README.md)) |
+| `tools/` | Preparation, teacher fusing, guarded submission, source verification, benchmarking, reporting, plots (`plot.py`), `reproduce.sh` |
+| `docs/` | [Setup](docs/SETUP.md), [provenance](docs/PROVENANCE.md), [validation](docs/VALIDATION.md), [caesar_cipher benchmark](docs/CAESAR-OPD-2026-09-27.md), [H100 MOPD report](docs/MOPD-H100-2026-10-07.md), benchmark data under `docs/*/data` and `docs/mopd-h100-2026-10-07` |
 
 ## Checks
 
-Python 3.12 is required for the preparation tools and archived campaign code.
-The following checks need no GPU and do not contact the cluster.
+Python 3.12 is required. These checks need no GPU:
 
 ```bash
 python3 -m unittest discover -s tests -v
 git diff --check
 ```
 
-Raw logs, model weights, generated checkpoints, local credentials, and site configuration are excluded from Git.
-Keep run artifacts outside this checkout and retain their source snapshot, resolved configuration, exit status, and checksums.
-The old repository contents remain in Git history at `a1bf83efd1a5cf11ef72b3ea18eb98132d83c92e`.
-This change does not remove old data from Git history or change repository visibility.
+Raw logs, weights, checkpoints, credentials and local config files are kept out of Git. Earlier repository contents remain in history at `a1bf83efd1a5cf11ef72b3ea18eb98132d83c92e`.

config/README.md

@@ -0,0 +1,18 @@
+# Local configuration
+
+Copy an example to `*.local.json` and edit it. Local files are gitignored, so cluster addresses, account IDs and account-guard values stay out of the repository.
+
+| File | Used by | Required |
+|---|---|---|
+| `site.local.json` | `tools/prepare.py` | Yes, for Miles, Prime-RL, Slime and verl |
+| `site-nemo-rl.local.json` | `tools/prepare.py nemo-rl` | Yes, for NeMo-RL. It is the site file plus `teacher_node`, `teacher_ip` and `nemo_rl_container` for the third node. |
+| `tinker.local.json` | `hosted/tinker/launch.py` | No. Without it, `TINKER_API_KEY` is used with no account guard. |
+| `fireworks.local.json` | `hosted/fireworks/campaign.py` | No. `--account` or `FIREWORKS_ACCOUNT_ID` also work. |
+
+## API keys
+
+API keys never go in these files. The hosted launchers read `TINKER_API_KEY` or `FIREWORKS_API_KEY` from the environment. To read a `LABEL=value` line from `~/.zprofile` instead, set `api_key_profile_label`. This helps when one machine holds keys for several accounts.
+
+## Tinker account guard
+
+Set `expected_email` and `expected_org` to make the Tinker launcher refuse any other account. On a shared organization account, keep these values in your local file.

config/fireworks.example.json

@@ -0,0 +1,4 @@
+{
+  "account": "your-fireworks-account-id",
+  "api_key_profile_label": null
+}

config/site-nemo-rl.example.json

@@ -0,0 +1,16 @@
+{
+  "assets": "/shared/opd-assets",
+  "base_model": "/shared/opd-assets/models/Qwen3.6-35B-A3B",
+  "megatron_model": "/shared/opd-assets/models/Qwen3.6-35B-A3B_torch_dist",
+  "teachers": "/shared/opd-assets/teachers",
+  "container_image": "/shared/opd-runtime/miles-amd64.sqsh",
+  "megatron_source": "/root/Megatron-LM",
+  "uv": "/usr/local/bin/uv",
+  "generation_node": "generation-node",
+  "trainer_node": "trainer-node",
+  "generation_ip": "192.0.2.20",
+  "trainer_ip": "192.0.2.10",
+  "teacher_node": "teacher-node",
+  "teacher_ip": "192.0.2.30",
+  "nemo_rl_container": "/shared/opd-runtime/nemo-rl-v0.7.0.sqsh"
+}

site.example.json → config/site.example.json

@@ -3,7 +3,7 @@
   "base_model": "/shared/opd-assets/models/Qwen3.6-35B-A3B",
   "megatron_model": "/shared/opd-assets/models/Qwen3.6-35B-A3B_torch_dist",
   "teachers": "/shared/opd-assets/teachers",
-  "container_image": "/shared/opd-runtime/miles-dev-202609050049.sqsh",
+  "container_image": "/shared/opd-runtime/miles-amd64.sqsh",
   "megatron_source": "/root/Megatron-LM",
   "uv": "/usr/local/bin/uv",
   "generation_node": "generation-node",

config/tinker.example.json

@@ -0,0 +1,5 @@
+{
+  "api_key_profile_label": null,
+  "expected_email": null,
+  "expected_org": null
+}

data/manifest.json

@@ -1,37 +1,27 @@
 {
-  "countdown4-dev.jsonl": {
-    "sha256": "ba7497391529339026f0db7bc90e1b26aa25751dad4a19a3ca90992b417bbda7",
-    "bytes": 319918,
-    "rows": 512
+  "caesar_cipher-train.jsonl": {
+    "bytes": 11291100,
+    "rows": 7788,
+    "sha256": "41beb35347fd0774d702464466e3467b243535b594658161f2adaa2a7e4758c2"
   },
-  "countdown4-test.jsonl": {
-    "sha256": "142bf2ff50c30afa81bd30dc36f63f1fd31de07c0e4b5b8134f957df6d4dcf7e",
-    "bytes": 641017,
-    "rows": 1024
+  "caesar_cipher-dev.jsonl": {
+    "bytes": 183027,
+    "rows": 126,
+    "sha256": "c2f97f3f252640c9652faebac523b1f5669bd952ffb92f6890a74e5765297d41"
   },
-  "countdown4-train.jsonl": {
-    "sha256": "2a48909ab0a876585fc007dcc021203dd2adc0bd5a6db5296b4ea09736702913",
-    "bytes": 6279290,
-    "rows": 10000
+  "simple_geometry-train.jsonl": {
+    "bytes": 11946723,
+    "rows": 7788,
+    "sha256": "bb7fcfbda69a3b6ac7db9c5c545e0e84fa78d8eeb432d37608573b5464c8a02f"
   },
-  "graph12-dev.jsonl": {
-    "sha256": "fe9cd47d24baf3fbc16e68b119adf6c55405bfaffa13e2539a79f224ebdcfff3",
-    "bytes": 557290,
-    "rows": 512
-  },
-  "graph12-test.jsonl": {
-    "sha256": "274531fe90a56a67acd7e2f4336bead075bd2e7d891bce70ac7323ace1b76246",
-    "bytes": 1115638,
-    "rows": 1024
-  },
-  "graph12-train.jsonl": {
-    "sha256": "7229bc6d4d7a37d8cf3aaab449ab35d9efb5bf63aaceac64fdd0334554f48066",
-    "bytes": 10908616,
-    "rows": 10000
+  "simple_geometry-dev.jsonl": {
+    "bytes": 195111,
+    "rows": 128,
+    "sha256": "8dc5fc2aa9c3355d8ee3d4b521a997c7863f471723663a9889e9e4db5c5e6b8d"
   },
   "mixed-train.jsonl": {
-    "sha256": "bafdaa11f181f55606eefa7227583d80783e7d781281a2b0136dcdf9d307a70e",
-    "bytes": 17187906,
-    "rows": 20000
+    "bytes": 23237823,
+    "rows": 15576,
+    "sha256": "d8c46c8218174820779690e92535a16dea9244558315c4eb3d5c700812c5c278"
   }
 }