From 7da2309c6eebb9e042b0e245ffdd4a34471f684d Mon Sep 17 00:00:00 2001 From: agent Date: Thu, 5 Nov 2026 09:56:25 -0400 Subject: [PATCH 1/2] determinism: batching does not add cross-engine noise, measured (1.00x at width 4) The page said a batched cell's log-probabilities are a ladder mixture, but never answered the question a 'we win the batched column' claim rests on: does concurrency change the delta between two engines? Two routes at --parallel 4, each decoding 4 identical copies simultaneously, 8 prompts x 64 tokens x 3 rounds, scored over the prompts whose greedy streams agree at both widths: cpu vs hrx, width 1 3.670e-02 cpu vs hrx, both w4 3.685e-02 (1.00x) route vs its own batch 1.054e-02 (0.29x) So batching moves a route by less than a third of the between-engine difference and leaves the cross-route figure where it was. Two caveats decide whether such a number means anything: deltas are only comparable over positions where both routes chose the same token (4 of the 8 prompts did not), and the batched value is a dispersion over rounds because the cell is not itself repeatable. This corrects an inference I had put in the scoreboard annotation ('indicative only' because batching noise was assumed to swamp the signal) -- the conclusion that a batched cell cannot be gated bitwise stands, the reason was wrong. --- docs/determinism.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/determinism.md b/docs/determinism.md index 2bb2f8fc4..49b0d93ac 100644 --- a/docs/determinism.md +++ b/docs/determinism.md @@ -211,6 +211,33 @@ Consequence for a correctness gate: a batched cell cannot be gated on one number are a mixture over that ladder, the displaced rungs are 1-2 nats away on `cpu`, and a single run can be any rung — so the cell has to report the dispersion over its own rounds, or pin the schedule. +**Cross-engine, both batched** — the question the tables above do not answer, and the one a scoreboard's +"we win the batched column" claim rests on. Two routes at `--parallel 4`, each decoding 4 byte-identical +copies *at the same time*, 8 prompts × 64 tokens × 3 rounds, shipped c24 payload, scored over the prompts +where both routes choose the same greedy stream at width 1 **and** at width 4 (4 of the 8): + +| measured on | median \|Δlogp\| | against the width-1 delta | +|---|---:|---:| +| `cpu` vs `hrx`, width 1 | 3.670e-02 | — | +| `cpu` vs `hrx`, both at width 4 | **3.685e-02** | **1.00×** | +| a route vs its own batched answer | 1.054e-02 | 0.29× | + +So batching does **not** add noise on top of the between-engine difference: the cross-route delta is the +same figure at width 1 and at width 4, and batching moves a single route by only ~0.3× of it. Two caveats +decide whether a given cross-route number means anything at all: + +1. **Only positions where both routes chose the same token are comparable.** Four of those eight prompts +differed at width 1 (token match 0.267-0.923), and there a per-position delta compares *different* tokens — +it measures the divergence, not the arithmetic. `tools/logprob_agree.py` already restricts deltas to common +positions; `tools/batched_repeat.py --vs-solo` reports `same_token` beside every delta for the same reason. +2. **The batched value is a dispersion, not a number**: the cell's own run is not repeatable (6/12 identical +copies at width 4 on `cpu`), so what reproduces is the distribution over rounds. + +Consequence: a cross-engine comparison *does* survive concurrency — it is the same ~0.037 nats it was at +width 1 — so a batched cell can carry a log-prob column. It cannot carry a bitwise one, and the column is +only defined over agreed tokens. (Scratch probe, not shipped: `cross_concurrent.py` imports this repo's +comparators and `logprob_agree.py`'s prompts, so it measures what this page documents.) + ### What the `hrx` width gate is, by name `GGML_HRX_LOG_DISPATCH=1` prints each dispatch registration the first time it matches From b4069addf374c50103a77efce7248584cf1e6936 Mon Sep 17 00:00:00 2001 From: agent Date: Thu, 5 Nov 2026 10:22:54 -0400 Subject: [PATCH 2/2] determinism: how much the agreed-token restriction is worth (2.4x), from the same run The caveat said deltas are only comparable over tokens both routes chose, but not how much that matters. Same measurement, width 1: 3.670e-02 over the four agreeing prompts, 1.468e-01 over the four mismatching ones, 8.960e-02 over all eight -- so a figure quoted over every prompt is mostly a statement about how often the two routes pick the same token, not about arithmetic agreement. --- docs/determinism.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/determinism.md b/docs/determinism.md index 49b0d93ac..56a774055 100644 --- a/docs/determinism.md +++ b/docs/determinism.md @@ -228,8 +228,12 @@ decide whether a given cross-route number means anything at all: 1. **Only positions where both routes chose the same token are comparable.** Four of those eight prompts differed at width 1 (token match 0.267-0.923), and there a per-position delta compares *different* tokens — -it measures the divergence, not the arithmetic. `tools/logprob_agree.py` already restricts deltas to common -positions; `tools/batched_repeat.py --vs-solo` reports `same_token` beside every delta for the same reason. +it measures the divergence, not the arithmetic. The size of that effect is measurable in the same run: at +width 1 the cross-route delta is **3.670e-02** over the four agreeing prompts and **1.468e-01** over the four +mismatching ones, so a figure quoted over all eight (**8.960e-02**, 2.4× the comparable subset) is mostly a +statement about how often the two routes pick the same token. `tools/logprob_agree.py` already restricts deltas +to common positions; `tools/batched_repeat.py --vs-solo` reports `same_token` beside every delta for the same +reason. 2. **The batched value is a dispersion, not a number**: the cell's own run is not repeatable (6/12 identical copies at width 4 on `cpu`), so what reproduces is the distribution over rounds.