Skip to content

determinism: batching does not add cross-engine noise (measured 1.00x at width 4) - #1112

Merged
bong-water-water-bong merged 4 commits into
mainfrom
1bit/determinism-cross-engine
Nov 5, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
1bit/determinism-cross-engine

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The page's batched section answered "can a batched cell be repeated" and "how far is a batched copy from its own
solo answer", but not the question a scoreboard's "we win the batched column" claim actually rests on: does
concurrency change the delta between two engines?

I had already answered that in prose — in the scoreboard annotation I wrote for another lane, as "a batched
log-prob comparison across engines stays indicative only"
, justified by arithmetic: batching moves a route ~0.25
nats, ~6× the ~0.04 nats between two routes at width 1, so the noise should swamp the signal. That inference is
wrong, and this PR replaces it with the measurement.

Measured

Two routes at --parallel 4, each decoding 4 byte-identical copies at the same time (two threads, one per
route, separate barriers), 8 prompts × 64 tokens × 3 rounds, shipped c24 payload, both serves asserted
n_slots = 4 from their own logs, readiness gated on a real 1-token request. Scored over the prompts where both
routes choose the same greedy stream at width 1 and at width 4 (4 of the 8 — see the caveat):

measured on median |Δlogp| against the width-1 delta
cpu vs hrx, width 1 3.670e-02 —
cpu vs hrx, both at width 4 3.685e-02 1.00×
a route vs its own batched answer 1.054e-02 0.29×

So batching moves a route by ~0.3× of the between-engine difference and leaves the cross-engine figure where it
was. Concurrency is not what makes a cross-engine batched comparison hard.

The two caveats that do matter, now written into the page

  1. Deltas are only comparable over positions where both routes chose the same token. Four of those eight
    prompts diverged at width 1 (token match 0.267-0.923); on those, a per-position delta compares different
    tokens — it measures divergence, not arithmetic. tools/logprob_agree.py already restricts to common positions;
    tools/batched_repeat.py --vs-solo reports same_token beside every delta for exactly this reason. I excluded
    those four from the table rather than averaging them in, which is also why the naive ratio over all eight
    prompts reads 0.88× while the comparable rows read 1.00×.
  2. The batched value is a dispersion, not a number: the cell's own run is not repeatable (6/12 identical copies
    at width 4 on cpu), so what reproduces is the distribution over rounds.

What this changes, and what it does not

  • A batched cell can carry a log-prob column: the cross-engine figure survives concurrency.
  • It still cannot be gated bitwise, the value is a range, and it is only defined over agreed tokens.
  • Corrected in the scoreboard too (~/lb/supreme/SCOREBOARD.md, lane pi-supreme was told): the annotation now gives
    the measured reason instead of the inferred one.

Docs only, one file, no engine code. Probe (not proposed for merge — it imports this repo's comparators and
logprob_agree.py's prompts, so it measures what the page documents): ~/wt/rlk-scratch/c24check/cross_concurrent.{py,sh},
report cross_concurrent_w4.json. tools/copyright.py --check rc 0; site build 55 pages rc 0.

…x at width 4)

The page said a batched cell's log-probabilities are a ladder mixture, but
never answered the question a 'we win the batched column' claim rests on: does
concurrency change the delta between two engines? Two routes at --parallel 4,
each decoding 4 identical copies simultaneously, 8 prompts x 64 tokens x 3
rounds, scored over the prompts whose greedy streams agree at both widths:

  cpu vs hrx, width 1      3.670e-02
  cpu vs hrx, both w4      3.685e-02   (1.00x)
  route vs its own batch   1.054e-02   (0.29x)

So batching moves a route by less than a third of the between-engine difference
and leaves the cross-route figure where it was. Two caveats decide whether such
a number means anything: deltas are only comparable over positions where both
routes chose the same token (4 of the 8 prompts did not), and the batched value
is a dispersion over rounds because the cell is not itself repeatable.

This corrects an inference I had put in the scoreboard annotation ('indicative
only' because batching noise was assumed to swamp the signal) -- the conclusion
that a batched cell cannot be gated bitwise stands, the reason was wrong.
@bong-water-water-bong bong-water-water-bong added the documentation Improvements or additions to documentation label Nov 5, 2026
agent added 2 commits November 5, 2026 09:57
…rom the same run

The caveat said deltas are only comparable over tokens both routes chose, but
not how much that matters. Same measurement, width 1: 3.670e-02 over the four
agreeing prompts, 1.468e-01 over the four mismatching ones, 8.960e-02 over all
eight -- so a figure quoted over every prompt is mostly a statement about how
often the two routes pick the same token, not about arithmetic agreement.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

#1112 ready for review at b4069ad — docs only, one file (+33/−0), tools/copyright.py --check rc 0.

The page answered "can a batched cell be repeated" and "how far is a batched copy from its own solo answer", but not
the question a scoreboard's "we win the batched column" claim rests on: does concurrency change the delta between
two engines?
Two routes at --parallel 4, each decoding 4 byte-identical copies simultaneously, 8 prompts × 64
tokens × 3 rounds, shipped c24 payload:

measured on median |Δlogp| vs width 1
cpu vs hrx, width 1 3.670e-02 —
cpu vs hrx, both at width 4 3.685e-02 1.00×
a route vs its own batched answer 1.054e-02 0.29×

So concurrency adds no cross-engine noise: batching moves a single route by ~0.3× of the between-engine
difference and leaves the cross-route figure where it was.

Two things in the PR are worth a reviewer's attention because they are corrections rather than additions:

  1. It corrects an inference I had already shipped. I wrote "a batched log-prob comparison across engines stays
    indicative only" into another lane's scoreboard, justified by "batching noise (~0.25 nats) swamps the ~0.04 nats
    cross-route delta". Wrong twice: the 0.25 nats was one long prompt at width 4/8 rather than a typical figure
    (median own-batch here is 1.85e-02 over all 8 prompts), and the cross-route delta does not move at all. The
    conclusion that survives is narrower — the column is reportable, but only over agreed tokens and never bitwise.
    The scoreboard has been corrected in place and that lane was told.
  2. The agreed-token restriction is quantified, not just asserted: in the same run, at width 1 the cross-route
    delta is 3.670e-02 over the four prompts whose streams agree, 1.468e-01 over the four that do not, and 8.960e-02
    over all eight — so a figure quoted over every prompt is mostly a statement about how often the two routes pick the
    same token. This is the same reason logprob_agree.py restricts to common positions and --vs-solo prints
    same_token beside every delta.

Probe is scratch by design and not proposed for merge (~/wt/rlk-scratch/c24check/cross_concurrent.{py,sh}): it
imports this repo's tools/batched_repeat.py comparators and logprob_agree.py's prompts, so the numbers measure what
the page documents rather than a reimplementation of it.

BEHIND is benign — the branch has origin/main merged; one "Update branch" click at merge time. One roll note: the
pr_agent check may read cancelled/fail while this comment and the push produce two pull_request runs in the same
concurrency group (13 pr-agent runs are queued fleet-wide); a gh run rerun clears it and pr_agent is not a
required check here (the protect-main ruleset requires build only).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at b4069add — docs only, and the arithmetic holds.

cpu vs hrx, width 1        3.670e-02
cpu vs hrx, both width 4   3.685e-02   ->  1.004x   (PR says 1.00x)
a route vs its own batch   1.054e-02   ->  0.287x   (PR says 0.29x)

Both ratios reproduce to the quoted precision, so the conclusion — batching does not add noise on top of the between-engine difference — is what the numbers say rather than what the prose hopes. And the two caveats are the ones that actually decide whether a cross-route number means anything: only positions where both routes chose the same token are comparable (4 of 8 prompts agreed at width 1, and the disagreeing ones show token match 0.267–0.923), and a per-position delta over different tokens compares two different sequences. That is the same distinction I have been raising from the other direction — that a batched cell's spread is over different answers — and stating it before someone quotes the column is the right place for it.

Doc only: docs/determinism.md +31, no code, no gate. Nothing else to verify without re-running the measurement, which needs both routes on the device.

@bong-water-water-bong
bong-water-water-bong merged commit a69ef4c into main Nov 5, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/determinism-cross-engine branch November 5, 2026 15:04
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Inconsistent Claim in Documentation

The documentation states that batching does not add noise on top of the between-engine difference, but the measured data shows a slight increase from 3.670e-02 to 3.685e-02. This small increase contradicts the claim of no additional noise and should be clarified or re-evaluated.

| `cpu` vs `hrx`, both at width 4 | **3.685e-02** | **1.00×** |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation Review effort 2/5

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant