Repository navigation
Conversation
Replace the Countdown/Graph Coloring package with the caesar_cipher and simple_geometry experiment (thinking on, 32k context): shared recipe and runtime, per-framework code for Miles, Prime-RL, Slime, verl and NeMo-RL, hosted Tinker and Fireworks campaigns, new datasets, the Sep 27 caesar OPD benchmark write-up, and updated tools, docs and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rewrite READMEs, docs and code comments to be shorter and plainer, fix statements that contradicted the code (default time limit, verl trainer backend, framework lists, reproduce.sh step order, stale sft.py paths), and drop references to internal research campaigns. Code changes are limited to comments, docstrings and argparse help text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add an h100x3 layout (Miles only): one trainer node with Megatron TP2/SP, EP8 and CPU Adam, and two generation nodes, each serving one TP2 teacher and three TP2 policy engines (12 policy GPUs instead of 6). Throughput fixes after comparing with Miles' fully async examples: - Submit prompts from the task with the fewest pending samples and keep surplus finished samples instead of evicting them after generation and teacher scoring. - Cap in-flight samples at 160, since at policy lag 1 a deeper pipeline mostly produces stale samples. - Evaluate only after the last update; benchmark checkpoints offline. - Give policy engines 80% of GPU memory, and each CPU-Adam trainer rank 16 OpenMP threads instead of Ray's 1. - Use per-role kernel caches, so NFS does not hand out stale handles. Log every trained sample's task, score, lengths and status, add tools/plot.py for a per-run dashboard, a tensor-parallel option for tools/benchmark.py, and the in-progress H100 report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 533 ran out of memory at update 15 in the backward pass of a full 32k-token sequence. The h100x3 trainer now uses context parallelism 2 (16,384 tokens per GPU, 32k per micro-batch); run 535 finished all 20 updates and also trained faster. Benchmarks (avg@8, 20 updates): caesar_cipher 0.773 (teacher 0.703, base 0.270), simple_geometry 0.998 (teacher 0.996, base 0.553), level with the B200 run. Add the report, benchmark summaries, telemetry and dashboards, and build benchmark venvs on uv's managed Python, which ships the headers Triton needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Named teacher entries start from RolloutConfig, whose load_format is "dummy", so multi-teacher runs scored against random weights (job 915). Set auto, eager and the default entry's lengths. Copy the trainer's Ray worker log (verl's per-step metrics) into the result and dump each trained batch to rollouts/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- h100x3 layout for verl: one TP2 teacher and three TP2 policy engines on each generation node, FSDP2 CPUOffloadPolicy (CPU Adam) with Ulysses SP2 on the trainer, 160 samples in flight and staleness 1 (Miles' policy lag 1), evaluation after the last update only. - Node-local compiled-kernel caches on h100x3: a shared NFS cache hit "Stale file handle" and hung two TP engines (job 542). - vLLM request statistics on; shared/capture.py --vllm scrapes every policy and teacher server on its node. - tools/schema_verl.py: per-update step/train/wait, MFU, prompt and response P50, rollout and teacher latency and throughput, stale drops. Run 547: median update 120 s (Miles 535: 123 s); step-10 avg@8 caesar 0.741, geometry 0.998 (Miles: 0.732, 0.996). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schema-verl-547.json (matched run) and schema-verl-544.json (staleness 0.25, trainer waited on rollouts) from tools/schema_verl.py, schema-miles-535.json in the same columns, and run 547's step-10 benchmark. The step-20 benchmark follows when it finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
avg@8 caesar 0.765, geometry 0.997 (Miles 535: 0.773, 0.998). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs One definition per column for both frameworks: rollout and teacher latency, throughput and teacher MFU from the servers' Prometheus metrics (SGLang or vLLM), trainer MFU at 6 x 3B active parameters per trained token, cost per update at $6.16 per H100-hour for every GPU of the allocation, and the trained batch's verifier score per task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tools/case_study.py output for the three H100x3 runs. verl 555 is single-teacher OPD (caesar_cipher) with the same layout and settings as verl 547. Fix: read the campaign's uncompressed training data so verl samples get their task and train score. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
avg@8 caesar 0.765 (teacher 0.703, base 0.270); simple_geometry, not trained, 0.695. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
avg@8 caesar 0.763 (step 10: 0.765; teacher 0.703); simple_geometry, not trained, 0.664. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| rows = [json.loads(l) for l in (root/'billing.jsonl').read_text().splitlines()] | ||
| row = billing(root, read_json(root/'resources.json')['account'], 'later') | ||
| base = next(r for r in rows if r['label'] == 'baseline') | ||
| return print(json.dumps({'today_usd': row.get('total_usd'), 'lease_usd': round(row['total_usd'] - base['total_usd'], 2)})) |
| # Fail before loading models if a campaign port is occupied. | ||
| for port in config.GENERATION_NODE_PORTS if self.role == 'generation' else config.TRAINER_NODE_PORTS: | ||
| with socket.socket() as probe: | ||
| probe.bind(('0.0.0.0', port)) |
Single-teacher OPD results get their own folder, separate from the MOPD runs, with copies of the base and caesar-teacher benchmarks they compare against. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 549ce20. Configure here.
Same recipe and verl settings as run 555 on three nodes, with 6 policy GPUs instead of 12: median update 203 s vs 182 s, run cost $139 vs $183. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
avg@8 caesar 0.750 (three-node run 555: 0.765; teacher 0.703); simple_geometry, not trained, 0.648. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
avg@8 caesar 0.778 (step 10: 0.750; three-node run 555: 0.763; teacher 0.703); simple_geometry, not trained, 0.658. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the Countdown/Graph Coloring package with the caesar_cipher and
simple_geometry experiment (thinking on, 32k context): shared recipe and
runtime, per-framework code for Miles, Prime-RL, Slime, verl and NeMo-RL,
hosted Tinker and Fireworks campaigns, new datasets, the Sep 27 caesar
OPD benchmark write-up, and updated tools, docs and tests.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Note
Medium Risk
Large packaging and workflow changes affect how Slurm campaigns are prepared and submitted across five RL stacks; mistakes in site config or framework wiring could waste GPU allocations, but changes are research tooling rather than production services.
Overview
Replaces the old Countdown/Graph Coloring (thinking off) package with a thinking-on on-policy distillation stack centered on reasoning_gym
caesar_cipherandsimple_geometry, 20 updates, 32k context, and multi-teacher MOPD.Experiment and data: Swaps datasets and
data/manifest.jsonto the new task splits; updates provenance and validation docs to reflect GPU runs (Sep 27 caesar benchmark across Miles, Prime-RL, Slime, verl; Oct H100 MOPD for Miles) and adds long-form reports plus bundled benchmark/telemetry JSON and chart scripts underdocs/.Layout and ops: Repositions site config under
config/(including NeMo-RL three-node and hosted Tinker/Fireworks examples), documentsshared/+ per-frameworkframeworks/+hosted/, and drives end-to-end reproduction viatools/reproduce.shwith updatedSETUP.md(H100h100x3layout, teacher fusing, FlashInfer, verl/NeMo-RL runtimes).Framework coverage: Extends beyond Miles/Prime-RL/Slime to verl (unpatched) and NeMo-RL; Prime-RL’s task package moves from
mopd_puzzlestorg_tasksfor reasoning_gym scoring.Reviewed by Cursor Bugbot for commit 6aff598. Bugbot is set up for automated code reviews on this repo. Configure here.