From c814783ea1ce11e03fbbdc562242cbd2544e0d41 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Tue, 20 Oct 2026 06:49:39 -0300 Subject: [PATCH 1/2] Release blockers: docs drift, NPU status, hrx-system pin note, weekly.sh preflight - README and docs: --moe-slots dropped from the feature list (serve refuses it; serve.md marks it), current hrx-system/llama.cpp and Lemonade pins, no "packages every Sunday" promise (no release is published yet), the empty config/recipes.json said as such, pm.md examples on Lemonade's port 13305 and serve.md on the --lemonade-url default. - The #315 debugging notes and campaign move from docs/ to dev/315/, so the site no longer publishes them; the site build failed on them (and on three '#' line starts in hrx.md, hw-gate.md and serve.md, also fixed) since 2026-10-05. - releases.md and README: the release is HRX with our HIP kernels and the NPU engine inside Lemonade; the NPU route is marked in progress (kernels from source not merged yet). npu.md no longer describes how the runtime talks to the driver. - hrx.md and .gitmodules: why the hrx-system pin 82804197 is off 1bit/main (diverged, not a fast-forward). - scripts/weekly.sh: a preflight before the first stage checks the HRX toolchain, hipcc for DwarfStar, patchelf and a readable kernel image; the package stage checks the run's own checkout has the Lemonade submodule. Nothing changes when they are present. Co-Authored-By: Claude Opus 5.5 --- .gitmodules | 1 + .pi/agents/hrx315.md | 4 +- 315-COORDINATION.md | 20 ++++---- KNOWN-ISSUES.md | 50 +++++++++---------- README.md | 18 ++++--- {docs => dev/315}/315-ENDOGAME.md | 12 ++--- {docs => dev/315}/315-H1-FALSIFIED.md | 0 {docs => dev/315}/315-STATUS.md | 34 ++++++------- .../315}/315-campaign/315-HANDOFF-box.md | 2 +- {docs => dev/315}/315-campaign/315-analyse.py | 2 +- {docs => dev/315}/315-campaign/315-attrib.sh | 0 .../315}/315-campaign/315-campaign.sh | 2 +- .../315}/315-campaign/315-cell-build.sh | 0 {docs => dev/315}/315-campaign/315-chain.sh | 0 {docs => dev/315}/315-campaign/315-chain2.sh | 0 .../315}/315-campaign/315-cheap-probe.sh | 0 .../315}/315-campaign/315-classify.py | 0 .../315}/315-campaign/315-cwsr-control.sh | 0 .../315}/315-campaign/315-detector.py | 0 .../315}/315-campaign/315-differential.sh | 0 {docs => dev/315}/315-campaign/315-dose.sh | 0 .../315}/315-campaign/315-evict-trace.sh | 0 {docs => dev/315}/315-campaign/315-faceb.sh | 0 .../315}/315-campaign/315-fix-build.sh | 0 .../315}/315-campaign/315-forcebarrier.sh | 0 {docs => dev/315}/315-campaign/315-fp-hunt.sh | 0 {docs => dev/315}/315-campaign/315-fp2.sh | 0 .../315}/315-campaign/315-graphsync.sh | 0 {docs => dev/315}/315-campaign/315-idprobe.sh | 2 +- .../315}/315-campaign/315-instr-build.sh | 0 .../315}/315-campaign/315-instr-capture.sh | 0 {docs => dev/315}/315-campaign/315-janitor.sh | 0 {docs => dev/315}/315-campaign/315-mode.sh | 0 {docs => dev/315}/315-campaign/315-paired.sh | 8 +-- {docs => dev/315}/315-campaign/315-regime.sh | 0 .../315}/315-campaign/315-regression.sh | 2 +- .../315}/315-campaign/315-rowprobe.sh | 0 .../315}/315-campaign/315-trace-premise.sh | 0 .../315}/315-campaign/315-wbfp-read.py | 4 +- .../315}/315-campaign/315-when-free.sh | 0 .../315}/315-campaign/EVIDENCE-INDEX.md | 4 +- .../ISSUE-COMMENT-2-2026-10-10.md | 0 .../315-campaign/ISSUE-COMMENT-2026-10-10.md | 0 .../ISSUE-COMMENT-3-2026-10-10.md | 0 .../315-campaign/ISSUE-COMMENT-4-DRAFT.md | 0 .../ISSUE-COMMENT-5-2026-10-10.md | 6 +-- .../315-0of70-has-no-eviction-cell.md | 0 ...315-1984-reconciles-the-two-populations.md | 0 ...SM-CANDIDATE-pointer-upload-refill-race.md | 0 ...verges-at-first-32-entry-attention-step.md | 0 .../315-MECHANISM-kv-not-visible-to-decode.md | 2 +- ...NISM-staging-handoff-not-memory-visible.md | 0 ...15-MECHANISM-unwaited-async-logits-read.md | 0 ...ce-revises-mechanism-to-device-ordering.md | 0 .../315-PREREG-argsort-precondition.md | 0 .../evidence/315-PREREG-sync-between-reads.md | 0 ...GISTRATION-f32-readpath-and-graph-input.md | 2 +- .../315-PROPOSED-RESCOPED-OBJECTIVE.md | 0 ...nter-upload-race-replays-are-serialized.md | 0 .../315-RESCOPE-PROPOSAL-2026-10-15.md | 2 +- ...5-RETRACTION-no-fixed-32-entry-boundary.md | 0 .../evidence/315-ab-preregistration.md | 0 .../evidence/315-ab1-arm-A-result.md | 2 +- .../evidence/315-ab1-arm-B-interim.md | 0 .../evidence/315-ab1-cell-c-fault-drift.md | 0 .../evidence/315-ab1-fix-path-is-coherent.md | 0 .../evidence/315-ab1-two-drivers-incident.md | 2 +- .../315-acceptance-harness-faceb-defects.md | 4 +- .../315-align-frozen-preregistration.md | 0 .../evidence/315-align-frozen-result.md | 0 .../evidence/315-align-preregistration.md | 2 +- .../evidence/315-align-run1-uninformative.md | 2 +- .../evidence/315-align-run2-result.md | 0 .../evidence/315-aql-publish-vs-reorder.md | 0 ...argsort-nonbijection-verified-both-ends.md | 0 .../315-campaign/evidence/315-arm-ledger.csv | 0 .../315-campaign/evidence/315-arm-ledger.md | 0 .../evidence/315-asan-preregistration.md | 0 .../evidence/315-asan-wiring-verified.md | 0 .../evidence/315-both-faces-one-trigger.md | 2 +- .../evidence/315-campaign-preregistration.md | 0 ...ate-deferred-host-writeback-publication.md | 0 ...-candidate-deferred-writeback-FALSIFIED.md | 0 .../315-candidate-host-staging-coherence.md | 0 .../evidence/315-cause-boundary-final.md | 0 ...5-cellA-measured-noncoherent-divergence.md | 0 ...B-divergence-independent-of-memory-type.md | 0 .../315-chain-plus-early-preregistration.md | 0 .../evidence/315-chaining-guard-defect.md | 0 .../315-campaign/evidence/315-cheap-repro.md | 0 ...coherence-candidate-falsified-by-decomp.md | 0 ...coherence-closed-from-source-both-paths.md | 0 .../315-corruption-is-one-row-sized.md | 0 .../315-corruption-is-wave-granular.md | 0 .../evidence/315-crit4-control-result.md | 0 .../evidence/315-deferred-writeback-gap.md | 4 +- .../315-device-buffer-probe-proposal.md | 2 +- .../evidence/315-device-hunt-boundary.md | 0 .../evidence/315-differential-artifact.md | 0 .../315-dossier-spt-claim-corrected.md | 0 .../evidence/315-engine-ci-baseline.md | 0 .../evidence/315-evict-trace-instrument.md | 0 .../evidence/315-evict-trace-status.md | 0 .../evidence/315-eviction-hits-the-arm.md | 2 +- ...15-eviction-replication-preregistration.md | 0 .../315-exclusions-vs-falsified-levers.md | 0 .../evidence/315-extra-env-comma-trap.md | 0 .../315-f32-sync-twin-preregistration.md | 0 ...n-result-divergence-on-the-faulting-arm.md | 0 .../evidence/315-fa-masked-v-regime.md | 0 .../evidence/315-fa-race-pin-sweep.md | 0 .../evidence/315-face-landscape.md | 0 .../evidence/315-faceb-amplifier.md | 0 .../315-faceb-has-no-capable-instrument.md | 0 .../evidence/315-faceb-invariant-signature.md | 0 ...-two-populations-and-faces-are-disjoint.md | 0 ...ault-is-a-per-step-hazard-after-prefill.md | 0 ...e-token-row-and-unifies-the-granularity.md | 0 .../evidence/315-faults-track-gtt-drift.md | 2 +- .../evidence/315-ffn-inp-0-carries-nan.md | 0 ...es-not-measure-per-node-buffer-aliasing.md | 0 .../315-first-row-site-is-fixed-layer-41.md | 0 .../evidence/315-fp2-ordered-fingerprint.md | 0 .../315-fresh-arena-preregistration.md | 0 ...15-galign-CORRECTION-my-parse-was-wrong.md | 0 .../evidence/315-galign-label-discrepancy.md | 0 .../evidence/315-galign-peer-result.md | 0 .../evidence/315-glm-align-preregistration.md | 0 .../evidence/315-glm-dispatch-inventory.md | 0 ...glm-never-dispatches-the-changed-kernel.md | 0 .../evidence/315-gord1-arm-A-result.md | 0 .../evidence/315-gord1-arm-B-interim.md | 0 .../315-campaign/evidence/315-gord1-final.md | 0 .../evidence/315-gord1-preregistration.md | 0 .../evidence/315-graph-reorder-lever.md | 0 .../315-gtt-start-is-the-discriminator.md | 0 .../evidence/315-gttstep-preregistration.md | 2 +- ...n-decode-not-prefill-index0-never-fails.md | 0 .../evidence/315-hrx-mem-pool-quarantine.md | 0 ...rx-re-decomp-gaps-are-the-315-functions.md | 0 .../315-hrx-system-dossier-by-peer-agent.md | 0 .../evidence/315-hrx-tests-baseline.md | 0 .../evidence/315-id-producing-kernel.md | 0 .../evidence/315-idprobe-readpath-refuted.md | 0 .../evidence/315-idprobe-result.md | 0 .../evidence/315-idprobe-was-unrunnable.md | 0 ...315-if-staging-is-cold-wave64-signature.md | 0 .../315-instrument-independent-summary.md | 0 .../evidence/315-instrumented-arm-hunt.md | 0 .../315-invalid-utf8-never-observed.md | 0 .../315-layer0-tail-preregistration.md | 0 .../315-layer41-site-by-graph-arithmetic.md | 0 .../315-lifetime-is-the-faithfulness-rule.md | 0 .../evidence/315-logits-wait-FALSIFIED.md | 0 .../315-lout-sweep-preregistration.md | 0 .../evidence/315-lout-sweep-result.md | 0 ...futed-on-ids-family-and-the-precise-gap.md | 0 ...15-moe-id-instrument-never-observed-ids.md | 0 .../315-moe-routing-routed-ffn-static.md | 0 .../evidence/315-nan-origin-is-layer-0.md | 2 +- ...nagement-so-platform-provides-coherence.md | 2 +- .../evidence/315-no-fix-available.md | 0 ...ode-2948-is-an-unnamed-moe-accumulation.md | 0 .../315-one-row-at-the-front-of-the-buffer.md | 2 +- ...at-layer-41-ffn-and-node-2948-confirmed.md | 0 ...onset-is-set-by-read-pattern-not-probes.md | 0 .../315-onset-layer-is-config-dependent.md | 0 .../315-origin-is-at-or-before-layer-4.md | 0 .../evidence/315-origin-varies-across-arms.md | 0 .../evidence/315-origin-varies-reinstated.md | 0 .../315-orphaned-forcers-contaminate-cells.md | 0 .../evidence/315-orphaned-neighbour-defect.md | 0 .../evidence/315-peer-rig-replication.md | 0 .../315-persistent-mapping-correction.md | 0 .../315-campaign/evidence/315-pid-capture.md | 0 .../evidence/315-pid-capture.patch | 0 .../evidence/315-pooled-rig-verification.md | 0 .../315-producer-probe-preregistration.md | 4 +- .../evidence/315-queue-profile.md | 0 .../evidence/315-quiet-vs-forcer-by-level.md | 0 ...ce-result-buffer-moves-under-the-reader.md | 0 .../evidence/315-refuted-coherent-read.patch | 0 .../evidence/315-regime-overshoot-oor.md | 0 .../evidence/315-rescope-ground-truth.md | 0 .../evidence/315-row-index-preregistration.md | 0 .../315-row-p1c-inregime-mechanism.md | 0 ...15-row-size-replication-preregistration.md | 0 .../315-row-size-scored-from-archive.md | 0 .../315-row-size-test-preregistration.md | 0 .../315-runtime-restore-fidelity-proposal.md | 0 ...seed-is-an-all-nan-moe-router-per-token.md | 0 .../evidence/315-sentinel-preregistration.md | 0 ...ubgraphs-are-9-nodes-so-tail-covers-all.md | 0 .../315-tail16-run-what-it-settled.md | 0 .../315-task1-writer-not-stream-ordered.md | 0 ...nded-negative-precondition-unmeasurable.md | 0 .../315-task2-instrument-hard-negative.md | 6 +-- ...-task3-dense-control-clean-at-moved-pin.md | 0 ...15-task3-op-test-unchanged-at-moved-pin.md | 0 .../315-task9-session-state-2026-10-12.md | 30 +++++------ .../evidence/315-thread-composition.md | 0 ...15-two-cell-memory-type-preregistration.md | 0 ...15-two-gathers-bound-the-id-differently.md | 0 .../evidence/315-wb-probe-first-run-null.md | 0 .../315-what-the-merge-and-the-gates-found.md | 4 +- .../315-what-the-synchronise-drains.md | 0 .../evidence/315-which-binary-is-which.md | 0 .../315-which-readings-are-faithful.md | 0 .../evidence/315-write-side-unaddressed.md | 0 .../315-campaign/evidence/315nan-console.md | 0 .../evidence/NPU-LOCK-WEDGED-2026-10-12.md | 4 +- .../315}/315-campaign/evidence/amd_cwsr.cfg | 0 .../patches/315-buffer-trace-diag.patch | 0 .../315-campaign/patches/315-coherence-fix.md | 0 .../patches/315-coherent-read-fix.patch | 0 .../patches/315-download-fingerprint.md | 0 .../patches/315-fa-masked-v-nonfinite.md | 0 .../patches/315-instrumentation.md | 0 .../315}/315-campaign/patches/315-nan-dump.md | 0 .../315-campaign/tools/315-evict-per-arm.py | 0 .../315-campaign/tools/315-lever-check.py | 2 +- .../315}/315-campaign/tools/315-rate.py | 0 .../tools/315restore-fidelity.hip | 2 +- .../315}/315-campaign/tools/arm-pid.py | 0 .../tools/audit-binding-access.py | 0 .../315}/315-campaign/tools/fa-sweep.sh | 0 .../315-campaign/tools/fa-variant-profile.py | 0 .../315-campaign/tools/fix-dense-count.py | 2 +- .../tools/gen-fa-masked-nonfinite.py | 0 .../315}/315-campaign/tools/gen-nan-dump.py | 0 .../315}/315-campaign/tools/glm-mla-race.cpp | 0 .../315}/315-campaign/tools/gord-analysis.py | 4 +- .../315}/315-campaign/tools/lever-census.py | 0 .../tools/patch-device-producer-probe.py | 2 +- .../tools/patch-f32-read-twice-control.py | 0 .../315}/315-campaign/tools/patch-fp-sync.py | 2 +- .../315-campaign/tools/patch-fp-wavemask.py | 2 +- .../tools/patch-id-instrument-sync.py | 0 .../315-campaign/tools/patch-id-instrument.py | 0 .../tools/patch-input-f32-sync-twin.py | 0 .../tools/patch-staging-coherence-ab.py | 2 +- .../tools/patch-sync-between-reads.py | 2 +- .../315-campaign/tools/patch-sync-dose.py | 2 +- .../tools/patch-writeback-read-probe.py | 0 .../tools/producer-probe-readout.py | 2 +- .../tools/row-position-census-by-family.py | 0 .../315-campaign/tools/row-position-census.py | 0 .../315}/315-campaign/tools/row-readout.py | 6 +-- .../tools/scan-global-handoffs.py | 0 .../315}/315-campaign/tools/test-analyse.py | 2 +- .../315-campaign/tools/verify-evidence.sh | 0 {docs => dev/315}/315-cwsr-hypothesis.md | 0 .../315}/315-cwsr-refined-hypothesis.md | 0 {docs => dev/315}/315-cwsr-result.md | 0 .../315}/315-cwsr-size-excluded-PROOF.md | 0 {docs => dev/315}/315-cwsr-sizes-excluded.md | 0 {docs => dev/315}/315-instrumentation-plan.md | 2 +- {docs => dev/315}/315-mechanism.md | 12 ++--- {docs => dev/315}/315-preregistration.md | 6 +-- docs/PORTING.md | 2 +- docs/hrx.md | 16 +++++- docs/hw-gate.md | 5 +- docs/lemonade.md | 6 +-- docs/npu.md | 6 +-- docs/pm.md | 6 +-- docs/releases.md | 27 ++++++++-- docs/serve.md | 21 ++++---- scripts/weekly.sh | 44 +++++++++++++++- 268 files changed, 258 insertions(+), 184 deletions(-) rename {docs => dev/315}/315-ENDOGAME.md (97%) rename {docs => dev/315}/315-H1-FALSIFIED.md (100%) rename {docs => dev/315}/315-STATUS.md (98%) rename {docs => dev/315}/315-campaign/315-HANDOFF-box.md (98%) rename {docs => dev/315}/315-campaign/315-analyse.py (98%) rename {docs => dev/315}/315-campaign/315-attrib.sh (100%) rename {docs => dev/315}/315-campaign/315-campaign.sh (99%) rename {docs => dev/315}/315-campaign/315-cell-build.sh (100%) rename {docs => dev/315}/315-campaign/315-chain.sh (100%) rename {docs => dev/315}/315-campaign/315-chain2.sh (100%) rename {docs => dev/315}/315-campaign/315-cheap-probe.sh (100%) rename {docs => dev/315}/315-campaign/315-classify.py (100%) rename {docs => dev/315}/315-campaign/315-cwsr-control.sh (100%) rename {docs => dev/315}/315-campaign/315-detector.py (100%) rename {docs => dev/315}/315-campaign/315-differential.sh (100%) rename {docs => dev/315}/315-campaign/315-dose.sh (100%) rename {docs => dev/315}/315-campaign/315-evict-trace.sh (100%) rename {docs => dev/315}/315-campaign/315-faceb.sh (100%) rename {docs => dev/315}/315-campaign/315-fix-build.sh (100%) rename {docs => dev/315}/315-campaign/315-forcebarrier.sh (100%) rename {docs => dev/315}/315-campaign/315-fp-hunt.sh (100%) rename {docs => dev/315}/315-campaign/315-fp2.sh (100%) rename {docs => dev/315}/315-campaign/315-graphsync.sh (100%) rename {docs => dev/315}/315-campaign/315-idprobe.sh (98%) rename {docs => dev/315}/315-campaign/315-instr-build.sh (100%) rename {docs => dev/315}/315-campaign/315-instr-capture.sh (100%) rename {docs => dev/315}/315-campaign/315-janitor.sh (100%) rename {docs => dev/315}/315-campaign/315-mode.sh (100%) rename {docs => dev/315}/315-campaign/315-paired.sh (96%) rename {docs => dev/315}/315-campaign/315-regime.sh (100%) rename {docs => dev/315}/315-campaign/315-regression.sh (98%) rename {docs => dev/315}/315-campaign/315-rowprobe.sh (100%) rename {docs => dev/315}/315-campaign/315-trace-premise.sh (100%) rename {docs => dev/315}/315-campaign/315-wbfp-read.py (97%) rename {docs => dev/315}/315-campaign/315-when-free.sh (100%) rename {docs => dev/315}/315-campaign/EVIDENCE-INDEX.md (98%) rename {docs => dev/315}/315-campaign/ISSUE-COMMENT-2-2026-10-10.md (100%) rename {docs => dev/315}/315-campaign/ISSUE-COMMENT-2026-10-10.md (100%) rename {docs => dev/315}/315-campaign/ISSUE-COMMENT-3-2026-10-10.md (100%) rename {docs => dev/315}/315-campaign/ISSUE-COMMENT-4-DRAFT.md (100%) rename {docs => dev/315}/315-campaign/ISSUE-COMMENT-5-2026-10-10.md (96%) rename {docs => dev/315}/315-campaign/evidence/315-0of70-has-no-eviction-cell.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-1984-reconciles-the-two-populations.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-PREREG-argsort-precondition.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-PREREG-sync-between-reads.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md (99%) rename {docs => dev/315}/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-ab-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-ab1-arm-A-result.md (97%) rename {docs => dev/315}/315-campaign/evidence/315-ab1-arm-B-interim.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-ab1-cell-c-fault-drift.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-ab1-fix-path-is-coherent.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-ab1-two-drivers-incident.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-acceptance-harness-faceb-defects.md (97%) rename {docs => dev/315}/315-campaign/evidence/315-align-frozen-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-align-frozen-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-align-preregistration.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-align-run1-uninformative.md (97%) rename {docs => dev/315}/315-campaign/evidence/315-align-run2-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-aql-publish-vs-reorder.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-arm-ledger.csv (100%) rename {docs => dev/315}/315-campaign/evidence/315-arm-ledger.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-asan-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-asan-wiring-verified.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-both-faces-one-trigger.md (96%) rename {docs => dev/315}/315-campaign/evidence/315-campaign-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-candidate-host-staging-coherence.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-cause-boundary-final.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-chain-plus-early-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-chaining-guard-defect.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-cheap-repro.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-corruption-is-one-row-sized.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-corruption-is-wave-granular.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-crit4-control-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-deferred-writeback-gap.md (99%) rename {docs => dev/315}/315-campaign/evidence/315-device-buffer-probe-proposal.md (99%) rename {docs => dev/315}/315-campaign/evidence/315-device-hunt-boundary.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-differential-artifact.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-dossier-spt-claim-corrected.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-engine-ci-baseline.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-evict-trace-instrument.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-evict-trace-status.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-eviction-hits-the-arm.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-eviction-replication-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-exclusions-vs-falsified-levers.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-extra-env-comma-trap.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-f32-sync-twin-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fa-masked-v-regime.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fa-race-pin-sweep.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-face-landscape.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-faceb-amplifier.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-faceb-has-no-capable-instrument.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-faceb-invariant-signature.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-faceb-is-two-populations-and-faces-are-disjoint.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fault-is-a-per-step-hazard-after-prefill.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fault-is-one-token-row-and-unifies-the-granularity.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-faults-track-gtt-drift.md (97%) rename {docs => dev/315}/315-campaign/evidence/315-ffn-inp-0-carries-nan.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fingerprint-does-not-measure-per-node-buffer-aliasing.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-first-row-site-is-fixed-layer-41.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fp2-ordered-fingerprint.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-fresh-arena-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-galign-CORRECTION-my-parse-was-wrong.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-galign-label-discrepancy.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-galign-peer-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-glm-align-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-glm-dispatch-inventory.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-glm-never-dispatches-the-changed-kernel.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-gord1-arm-A-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-gord1-arm-B-interim.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-gord1-final.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-gord1-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-graph-reorder-lever.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-gtt-start-is-the-discriminator.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-gttstep-preregistration.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-hazard-is-in-decode-not-prefill-index0-never-fails.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-hrx-mem-pool-quarantine.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-hrx-system-dossier-by-peer-agent.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-hrx-tests-baseline.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-id-producing-kernel.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-idprobe-readpath-refuted.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-idprobe-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-idprobe-was-unrunnable.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-if-staging-is-cold-wave64-signature.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-instrument-independent-summary.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-instrumented-arm-hunt.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-invalid-utf8-never-observed.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-layer0-tail-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-lifetime-is-the-faithfulness-rule.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-logits-wait-FALSIFIED.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-lout-sweep-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-lout-sweep-result.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-mechanism-refuted-on-ids-family-and-the-precise-gap.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-moe-id-instrument-never-observed-ids.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-moe-routing-routed-ffn-static.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-nan-origin-is-layer-0.md (99%) rename {docs => dev/315}/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-no-fix-available.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-node-2948-is-an-unnamed-moe-accumulation.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-onset-is-a-full-row-at-layer-41-ffn-and-node-2948-confirmed.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-onset-layer-is-config-dependent.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-origin-is-at-or-before-layer-4.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-origin-varies-across-arms.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-origin-varies-reinstated.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-orphaned-forcers-contaminate-cells.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-orphaned-neighbour-defect.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-peer-rig-replication.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-persistent-mapping-correction.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-pid-capture.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-pid-capture.patch (100%) rename {docs => dev/315}/315-campaign/evidence/315-pooled-rig-verification.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-producer-probe-preregistration.md (98%) rename {docs => dev/315}/315-campaign/evidence/315-queue-profile.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-quiet-vs-forcer-by-level.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-read-twice-result-buffer-moves-under-the-reader.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-refuted-coherent-read.patch (100%) rename {docs => dev/315}/315-campaign/evidence/315-regime-overshoot-oor.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-rescope-ground-truth.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-row-index-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-row-p1c-inregime-mechanism.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-row-size-replication-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-row-size-scored-from-archive.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-row-size-test-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-seed-is-an-all-nan-moe-router-per-token.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-sentinel-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-subgraphs-are-9-nodes-so-tail-covers-all.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-tail16-run-what-it-settled.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-task1-writer-not-stream-ordered.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-task2-instrument-hard-negative.md (95%) rename {docs => dev/315}/315-campaign/evidence/315-task3-dense-control-clean-at-moved-pin.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-task9-session-state-2026-10-12.md (99%) rename {docs => dev/315}/315-campaign/evidence/315-thread-composition.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-two-cell-memory-type-preregistration.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-two-gathers-bound-the-id-differently.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-wb-probe-first-run-null.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md (95%) rename {docs => dev/315}/315-campaign/evidence/315-what-the-synchronise-drains.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-which-binary-is-which.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-which-readings-are-faithful.md (100%) rename {docs => dev/315}/315-campaign/evidence/315-write-side-unaddressed.md (100%) rename {docs => dev/315}/315-campaign/evidence/315nan-console.md (100%) rename {docs => dev/315}/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md (96%) rename {docs => dev/315}/315-campaign/evidence/amd_cwsr.cfg (100%) rename {docs => dev/315}/315-campaign/patches/315-buffer-trace-diag.patch (100%) rename {docs => dev/315}/315-campaign/patches/315-coherence-fix.md (100%) rename {docs => dev/315}/315-campaign/patches/315-coherent-read-fix.patch (100%) rename {docs => dev/315}/315-campaign/patches/315-download-fingerprint.md (100%) rename {docs => dev/315}/315-campaign/patches/315-fa-masked-v-nonfinite.md (100%) rename {docs => dev/315}/315-campaign/patches/315-instrumentation.md (100%) rename {docs => dev/315}/315-campaign/patches/315-nan-dump.md (100%) rename {docs => dev/315}/315-campaign/tools/315-evict-per-arm.py (100%) rename {docs => dev/315}/315-campaign/tools/315-lever-check.py (98%) rename {docs => dev/315}/315-campaign/tools/315-rate.py (100%) rename {docs => dev/315}/315-campaign/tools/315restore-fidelity.hip (98%) rename {docs => dev/315}/315-campaign/tools/arm-pid.py (100%) rename {docs => dev/315}/315-campaign/tools/audit-binding-access.py (100%) rename {docs => dev/315}/315-campaign/tools/fa-sweep.sh (100%) rename {docs => dev/315}/315-campaign/tools/fa-variant-profile.py (100%) rename {docs => dev/315}/315-campaign/tools/fix-dense-count.py (96%) rename {docs => dev/315}/315-campaign/tools/gen-fa-masked-nonfinite.py (100%) rename {docs => dev/315}/315-campaign/tools/gen-nan-dump.py (100%) rename {docs => dev/315}/315-campaign/tools/glm-mla-race.cpp (100%) rename {docs => dev/315}/315-campaign/tools/gord-analysis.py (94%) rename {docs => dev/315}/315-campaign/tools/lever-census.py (100%) rename {docs => dev/315}/315-campaign/tools/patch-device-producer-probe.py (99%) rename {docs => dev/315}/315-campaign/tools/patch-f32-read-twice-control.py (100%) rename {docs => dev/315}/315-campaign/tools/patch-fp-sync.py (98%) rename {docs => dev/315}/315-campaign/tools/patch-fp-wavemask.py (98%) rename {docs => dev/315}/315-campaign/tools/patch-id-instrument-sync.py (100%) rename {docs => dev/315}/315-campaign/tools/patch-id-instrument.py (100%) rename {docs => dev/315}/315-campaign/tools/patch-input-f32-sync-twin.py (100%) rename {docs => dev/315}/315-campaign/tools/patch-staging-coherence-ab.py (98%) rename {docs => dev/315}/315-campaign/tools/patch-sync-between-reads.py (98%) rename {docs => dev/315}/315-campaign/tools/patch-sync-dose.py (97%) rename {docs => dev/315}/315-campaign/tools/patch-writeback-read-probe.py (100%) rename {docs => dev/315}/315-campaign/tools/producer-probe-readout.py (98%) rename {docs => dev/315}/315-campaign/tools/row-position-census-by-family.py (100%) rename {docs => dev/315}/315-campaign/tools/row-position-census.py (100%) rename {docs => dev/315}/315-campaign/tools/row-readout.py (95%) rename {docs => dev/315}/315-campaign/tools/scan-global-handoffs.py (100%) rename {docs => dev/315}/315-campaign/tools/test-analyse.py (98%) rename {docs => dev/315}/315-campaign/tools/verify-evidence.sh (100%) rename {docs => dev/315}/315-cwsr-hypothesis.md (100%) rename {docs => dev/315}/315-cwsr-refined-hypothesis.md (100%) rename {docs => dev/315}/315-cwsr-result.md (100%) rename {docs => dev/315}/315-cwsr-size-excluded-PROOF.md (100%) rename {docs => dev/315}/315-cwsr-sizes-excluded.md (100%) rename {docs => dev/315}/315-instrumentation-plan.md (99%) rename {docs => dev/315}/315-mechanism.md (99%) rename {docs => dev/315}/315-preregistration.md (98%) diff --git a/.gitmodules b/.gitmodules index f4f121072..e46f5ece7 100644 --- a/.gitmodules +++ b/.gitmodules @@ -16,6 +16,7 @@ # HRX (docs/hrx.md), the engine's GPU route; also the tree of the CPU llama-server (Windows). # Needed with -DONEBIT_HRX=ON. Both trees are AMD's tested pair (ROCm/ggml-staging-automation) plus # our commits: llama.cpp on our fork's 1bit/hrx-vulkan-patched, hrx-system on our fork's 1bit/main +# (the hrx-system pin is on 1bit/e3e6a976-pin-6, off 1bit/main for now: docs/hrx.md, "The pin is off 1bit/main") # (Loom and libhrx fixes, docs/hrx.md "Our hrx-system commits"). .github/workflows/bump-hrx.yml moves # the llama.cpp pin when AMD moves its pair; hrx-system is rebased by hand. The branch name still says "vulkan": AMD's pair # carried a Vulkan backend too, which the engine no longer builds (RFC #213 stage 3). diff --git a/.pi/agents/hrx315.md b/.pi/agents/hrx315.md index 4361c2d95..b57c5accc 100644 --- a/.pi/agents/hrx315.md +++ b/.pi/agents/hrx315.md @@ -27,7 +27,7 @@ You work on `1bit-MONSTER/engine` issue #315: **intermittent in-regime HRX GLM-4 NaN logits / `HSA_STATUS_ERROR_MEMORY_FAULT`; Face B is non-JSON ("Content-only") output answered with HTTP 500. The work lives in the worktree `~/wt315-prereg` (branch `1bit/315-preregistration`, PR #414) and runs on the **strixhalo** box over `ssh -o BatchMode=yes`. **Read `~/wt/315-HANDOFF.md` on that box and -`docs/315-STATUS.md` before doing anything**, plus `315-COORDINATION.md`, which is the inter-session channel. +`dev/315/315-STATUS.md` before doing anything**, plus `315-COORDINATION.md`, which is the inter-session channel. ## The standing state — do not re-derive it, and do not contradict it without new evidence @@ -103,5 +103,5 @@ earned or a fault that did not happen. Assume any harness you are shown has thes ## How to report Report what you **measured**, tied to a re-runnable command at a named revision, with evidence under -`docs/315-campaign/evidence/`. Separate **measured** from **inferred** every time. If a reading is +`dev/315/315-campaign/evidence/`. Separate **measured** from **inferred** every time. If a reading is uninformative, say so — an absent record is not a zero. If you did not check something, say that too. diff --git a/315-COORDINATION.md b/315-COORDINATION.md index 9a73f3be9..858dc612d 100644 --- a/315-COORDINATION.md +++ b/315-COORDINATION.md @@ -47,7 +47,7 @@ arms**, across nine sequence lengths — one `n_embd`-wide row = 8192 bytes. If arm e OK (98G)**, in-regime with `side=1/1`, then was superseded by the amendment. Two clean arms, no Face-A. Consistent with `0ed67c2`'s finding that the `tail:16` + probes yield did not reproduce. -Evidence: `docs/315-campaign/evidence/315-row-size-scored-from-archive.md`, +Evidence: `dev/315/315-campaign/evidence/315-row-size-scored-from-archive.md`, `315-layer41-site-by-graph-arithmetic.md`. --- @@ -93,14 +93,14 @@ only as a formal pre-registered check. Its interesting half is `A vs C`, which t is still thermally blocked when you read this, stopping it loses little — and note it holds `box.lock`, so it is the one thing blocking the box right now. -Full evidence: `docs/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md`. +Full evidence: `dev/315/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md`. --- ## 2026-10-14 05:39 — `315prod` is QUEUED and holds `box.lock`; the box is load-blocked by foreign work `315prod` (the producer-side sentinel/payload test, pre-registered in -`docs/315-campaign/evidence/315-producer-probe-preregistration.md` with its two amendments) took `box.lock` +`dev/315/315-campaign/evidence/315-producer-probe-preregistration.md` with its two amendments) took `box.lock` at 05:37:34 and is in its **gate wait**: `tctl = 96 °C` against a required < 80, and `load1 = 30` against a required < 4. @@ -176,9 +176,9 @@ exactly the functions that *did* decompile — `hrx_allocator_allocate_buffer`, `HOST_COHERENT = 0x04`) and `hrx_buffer_create_from_hal`. That closed "the coherence bit cannot change the **pool**". Whether it can change anything **downstream** is unresolved, and `hrx_allocator_virtual_memory_map` is the function that would settle it. See -`docs/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md`. +`dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md`. -Full list and reasoning: `docs/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. +Full list and reasoning: `dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. ### 06:30 — priority REVISED for the hrx-re lane: `iree_hal_streaming_memory_register_host` is now #1 @@ -202,7 +202,7 @@ either, so it was never captured rather than captured-and-failed. Reasons it out So: `register_host` first, then `virtual_memory_map` / `_protect` / `_unmap`, then the `hrx_fence_*` and `hrx_*_synchronize` families already listed. The whole list is still in -`docs/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. +`dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. **Also useful if it is cheap:** `flags` values for `hipHostRegister` as the HIP headers define them (`Portable = 0x1`, `Mapped = 0x2`, `IoMemory = 0x4`, `ReadOnly = 0x8`), so the `(flags & 8)` branch in the @@ -244,7 +244,7 @@ the others from their callers. `hipHostRegister` yields coherent or non-coherent memory — and `flags = 0` is exactly what the #315 trigger passes. Nothing here is a mechanism claim; it is the one change that makes the question answerable. -Detail: `docs/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. +Detail: `dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. ### 06:50 — WITHDRAW the `register_host` ask: answered from source, and it closes the coherence family @@ -274,7 +274,7 @@ recorded negatives at once: the `GGML_HRX_USE_UNIFIED_MEMORY` A/B could only cha and the never-attempted set needs the **DWARF-built queue** described in the previous note. The coherence question is no longer among the reasons. -Evidence: `docs/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md`. +Evidence: `dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md`. --- @@ -306,7 +306,7 @@ under the reader" observation is real and is *consistent* with `mark_stream_sync writebacks, but this run cannot separate that from a writer racing during the wait — do not read it as proof of the writeback path. -Evidence: `docs/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` (`b7d6b46`, `68c430c`). +Evidence: `dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` (`b7d6b46`, `68c430c`). Pre-registration: `…/315-PREREG-sync-between-reads.md` (`b51f883`). ### 2. What #315 now needs, and therefore what to ask your lane for @@ -343,7 +343,7 @@ silent no-op, not a clean result. If any arm of yours keyed on that name, its re `ffn_moe_probs` reading in the §50 chain needs re-establishing by whatever anchored mechanism produced it, and that is recorded as an **open discrepancy**, not smoothed over. -Evidence: `docs/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`). +Evidence: `dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`). ### 3. Four harness traps that have now cost four readings in a single session diff --git a/KNOWN-ISSUES.md b/KNOWN-ISSUES.md index 3a915f2ae..e73cd7670 100644 --- a/KNOWN-ISSUES.md +++ b/KNOWN-ISSUES.md @@ -63,7 +63,7 @@ allocation with. The earlier reading here, that "the logits buffer is never writ it was unsafe from the start: the sentinel only ever fills the 619520-byte logits allocation (`ggml-hrx.cpp`, `size == 619520`), so it says nothing about the tensors where the corruption is born. The guard fires on the consumer side and reports the shape, not the producer — the producer is upstream of the final norm, which the -nan-dump fingerprint (docs/315-campaign/evidence/315-instrumented-arm-hunt.md) established by finding the first +nan-dump fingerprint (dev/315/315-campaign/evidence/315-instrumented-arm-hunt.md) established by finding the first NaN-bearing tensors in the FFN path. ### Both failure faces @@ -77,7 +77,7 @@ depending on what the corrupted memory holds (NaN, or stale finite values): **2048 in fifteen of them** — exactly one `n_embd` row — while the batch size varies four-fold (T of 87, 98, 135, 140, 172, 306, 342), and the first affected node is an unnamed partial expert sum (`build_moe_ffn` names only the final aggregate). Mechanism, evidence and the discriminating experiment: - docs/315-mechanism.md sections 40, 44, 47. + dev/315/315-mechanism.md sections 40, 44, 47. **Correction, 2026-10-10.** That "one row" figure is a property of the instrument that measured it, not yet a property of the fault. Those arms used `GGML_HRX_TENSOR_FP=tail:16` (a window over the last sixteen graph @@ -87,13 +87,13 @@ depending on what the corrupted memory holds (NaN, or stale finite values): `ffn_moe_weights-4` — the `ggml_get_rows(ffn_moe_probs, ffn_moe_topk)` route-weight gather — while `ffn_moe_probs-4` carried none. Until the same instrument is run over the same arms, the safe statement is that the corruption involves the MoE routing tensors and that the row extent is instrument-dependent. - Evidence: docs/315-campaign/evidence/315-row-p1c-inregime-mechanism.md. + Evidence: dev/315/315-campaign/evidence/315-row-p1c-inregime-mechanism.md. 2. **Format-500 (Face B).** The same event with *finite* contents. A faulting request whose 60-name tensor fingerprint included `norm`, `result_norm` and `result_output` logged **no NaN at all**, and the server rejected its output as `unparsed Content-only output: the}:}:}甚;` — garbage; the paired control arm produced a repetition loop, `a gaps gaps gaps ...`. The sampler is drawing from a corrupted distribution, the text is degenerate, and the failure surfaces at the server's chat-template parser instead of at the sampler guard. - Evidence: docs/315-campaign/evidence/315-face-landscape.md. + Evidence: dev/315/315-campaign/evidence/315-face-landscape.md. A NaN-only count records the second face as a clean run, which is what makes it dangerous as a metric. Scoring must be: an arm is valid only if it completes the request phase in full (66 of 66 requests); an arm that aborts @@ -101,11 +101,11 @@ early has reduced exposure and is excluded, not counted as clean. Both faces mus guard, format failures through HTTP status, UTF-8 and JSON parse failures — and a third signal is needed for finite corruption that still parses: the detector's eight identical greedy requests must agree (`identical_distinct == 1`), because no clean arm among the 142 recorded has ever been non-deterministic while all seven non-deterministic arms -were Face-A faults (`docs/315-preregistration.md` section 8, freeze note 2026-10-10). +were Face-A faults (`dev/315/315-preregistration.md` section 8, freeze note 2026-10-10). ### State as of 2026-10-10 (this section supersedes the mechanism text above wherever they differ) -Diagnostic record: `docs/315-mechanism.md` sections 42-51; evidence under `docs/315-campaign/evidence/`. +Diagnostic record: `dev/315/315-mechanism.md` sections 42-51; evidence under `dev/315/315-campaign/evidence/`. **Where the fault sits now.** The routed FFN's *routing* tensors, in the MoE path: `ffn_moe_argsort` / `ffn_moe_topk` (the top-k ids, `src/llama-graph.cpp:1930-1935`) feed @@ -124,7 +124,7 @@ consume it with `index.assume` *ranges* (`mul_mat_id_swiglu_f32_f32_wmma.loom:47 `evidence/315-id-producing-kernel.md`. **Falsified in this round** (each with its own evidence path). Arm counts below are **re-derived from the -run consoles** (`docs/315-campaign/tools/lever-census.py`, all arms at `gtt_start` 97-98 GiB, i.e. inside +run consoles** (`dev/315/315-campaign/tools/lever-census.py`, all arms at `gtt_start` 97-98 GiB, i.e. inside the frozen `>= 95 GiB` gate), not from prose: | candidate | run | control cell | fix cell | reading | evidence | @@ -148,7 +148,7 @@ control. The comment's command section cites these same consoles as its source, **Withdrawn (kept for its own record) - was "still untested, and next".** The ids themselves: instrumented as `i32` with the name match anchored (the previous filter was `std::strstr(list, name)`, so `-41` also -matched `-4` and only layers 4 and 41 were covered). `docs/315-campaign/315-idprobe.sh` builds that probe +matched `-4` and only layers 4 and 41 were covered). `dev/315/315-campaign/315-idprobe.sh` builds that probe and runs four in-regime arms, reporting per row the first eight ids, the count outside `[0, n_expert)` and whether the row's ids repeat. Duplicate or out-of-range ids make the argsort kernel the site; a clean permutation kills it. @@ -361,7 +361,7 @@ What is supported: the forcer causes faults (0/22 -> 7/10 pooled, arm A 4/4, arm on); and individual arms have drifted into a high band and faulted while doing so (arm A quiet r2 97->102G; arm B cell `c` 97->106G). Isolated observations, not rates. The level question is open and is what the pre-registered `315gttstep` run exists to test. See -`docs/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, whose correction section carries the +`dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, whose correction section carries the within-console numbers. This reconciles four observations that previously did not fit together: the 0/70 campaign (run at 97G, @@ -371,7 +371,7 @@ through the high band; the quiet- and CPU-cell faults occur at 98G starts or by falsified per-arm eviction-*count* correlation (faulting mean 109.5 vs clean 254.5) failed because a count of evictions is not a measure of band occupancy. -Evidence: `docs/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, +Evidence: `dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, `315-faults-track-gtt-drift.md`, `315-ab1-cell-c-fault-drift.md`, `315-0of70-has-no-eviction-cell.md`. ### The forced-eviction intervention, and the fix it motivated @@ -381,7 +381,7 @@ two-sided p = 3.6e-5), producing **both** faces under one intervention — the t The candidate repair that intervention motivated — routing `buffer_get`'s bare `memcpy` of the non-coherent `0x52` logits mapping through `hrx_synchronous_d2h` — has been **REFUTED by its own -pre-registered A/B** (`fc7ffd5`, `docs/315-campaign/evidence/315-ab-preregistration.md`): +pre-registered A/B** (`fc7ffd5`, `dev/315/315-campaign/evidence/315-ab-preregistration.md`): ``` cell e (forced eviction, the test cell) arm A (knob OFF): 0 OK / 4 FAULT @@ -395,7 +395,7 @@ created `HOST_LOCAL | HOST_COHERENT | DEVICE_VISIBLE` (`host_queue_staging.c:427`, `315-ab1-fix-path-is-coherent.md`). **The prepared patch must NOT be landed.** It is preserved at -`docs/315-campaign/patches/315-coherent-read-fix.patch` for the record only. +`dev/315/315-campaign/patches/315-coherent-read-fix.patch` for the record only. ### Consequence for this entry's acceptance criterion @@ -473,13 +473,13 @@ documented limitation is acceptable only *after* the mechanism is named, so no l recorded. The controlled confirmation of the `gtt_start` boundary (cell fixed, start stepped 97/98/99) is -pre-registered at `docs/315-campaign/evidence/315-gttstep-preregistration.md`, not yet run. +pre-registered at `dev/315/315-campaign/evidence/315-gttstep-preregistration.md`, not yet run. --- ## UPDATE 2026-10-14 — the site is instrument-relative, the size is not, and Face B was never measurable -Session evidence: `docs/315-campaign/evidence/315-instrument-independent-summary.md` and the nine documents it +Session evidence: `dev/315/315-campaign/evidence/315-instrument-independent-summary.md` and the nine documents it cites; all read from the archived arm logs at pin `e44c9d01a`, no new arms. Nothing above is edited. ### The condition is named (as UPDATE 2026-10-13 says) but the SITE is not a property of the defect @@ -544,7 +544,7 @@ instrument-relative unless the read pattern is fixed. - **The acceptance harness could not attribute Face B** — and it failed twice over: its condition was `parse > 0 or utf8 > 0`, which are **0 in every Face-B arm**, and `315-paired.sh` computed `http`/`transport`/`s500`/`unparsed` and used them in its **verdict** but never **printed** them. Both fixed - in `docs/315-campaign/315-campaign.sh` and `315-paired.sh`, verified end-to-end on both branches (a real + in `dev/315/315-campaign/315-campaign.sh` and `315-paired.sh`, verified end-to-end on both branches (a real Face-B signature now attributes; an all-zero fault still does not). **Any future campaign must detect Content-only/HTTP 500 — a UTF-8-validity check would never fire and would pass a faulting arm as clean.** - **The "rule of three" bound prints probabilities above 100 %** for `n < 3` (`300 %` at n=1). Correct at the @@ -562,8 +562,8 @@ instrument-relative unless the read pattern is fixed. admits exactly one tensor, the **logits** region; layer residuals print `host=0` and can never be armed. A **device-buffer** variant is proposed, pre-registered, and its patch written and anchor-verified against a real source copy (two defects caught in review; **compile NOT claimed**) at - `docs/315-campaign/evidence/315-device-buffer-probe-proposal.md` and - `docs/315-campaign/tools/patch-device-producer-probe.py`. + `dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md` and + `dev/315/315-campaign/tools/patch-device-producer-probe.py`. ### On the fix — unchanged conclusion, one new reason @@ -616,13 +616,13 @@ work ordered the instrument's reads rather than the engine's. `git merge-base --is-ancestor e44c9d01a 47a8fe929` true. It is not a revert. **The campaign.** Pre-registered before it ran -(`docs/315-campaign/evidence/315-campaign-preregistration.md`): 60 in-regime **fix** arms plus 20 +(`dev/315/315-campaign/evidence/315-campaign-preregistration.md`): 60 in-regime **fix** arms plus 20 interleaved **control** arms on the unfixed build, same harness, same regime gate `[95,110] GiB`, one window, instruments off, stopping at the first fix-cell fault. Zero faults in 60 arms gives a one-sided 95 % upper bound of `1 − 0.05^(1/60) ≈ 4.9 %`. A clean fix cell with a control cell that never faults is reported as **under-powered, not passed**. -**Evidence paths:** `docs/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md` (the mechanism +**Evidence paths:** `dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md` (the mechanism and its falsifier), `…-campaign-preregistration.md`, `…-which-binary-is-which.md` (cell F = `~/wt/315logwait-build` at `47a8fe929`; cell C = `~/wt/315fixlog-build` at `e44c9d01a`; both with `hrx-f32-sync` and `hrx-wb-read` = 0). @@ -630,7 +630,7 @@ and its falsifier), `…-campaign-preregistration.md`, `…-which-binary-is-whic ### 2026-10-15 (later) — the logits-wait mechanism is FALSIFIED by its own campaign; the fix stays landed The mechanism recorded above was tested by the pre-registered campaign it named, and it failed on the first -fix arm. The full record is `docs/315-campaign/evidence/315-logits-wait-FALSIFIED.md`. +fix arm. The full record is `dev/315/315-campaign/evidence/315-logits-wait-FALSIFIED.md`. ``` pair 1 arm c (control, e44c9d01a): OK req=66 gtt_start=98G @@ -658,7 +658,7 @@ change is satisfied by it. What is not satisfied is the mechanism and the campai removed by its own pre-registered experiment and one has been shown unmeasurable by the current instrument. * **The stream-ordering explanation for the `result_norm` divergence is REMOVED.** - `docs/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` ran the pre-registered + `dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` ran the pre-registered `315-PREREG-sync-between-reads.md` on a faulting arm (Face A, 312 records) and landed on its **second** pre-declared row: of 82 records where `result_norm` differed between two consecutive plain reads, a fresh `ggml_backend_synchronize` stopped the difference in **0** and did not stop it in **82** — and changed the @@ -666,13 +666,13 @@ removed by its own pre-registered experiment and one has been shown unmeasurable `GraphExecutor::execute` returning without a wait (with `backend_synchronize` covering only `context->stream`) as the explanation. Recorded at `b7d6b46`. * **§51's rank-sort precondition is UNMEASURED and not measurable by the by-name fingerprint.** - `docs/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`) — the + `dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`) — the faulting arm produced zero records because `ffn_moe_probs-*` is not among the 946 node names this build's fingerprint walks. This is a negative about the instrument, not about the site. **Surviving family, stated as surviving:** an open writer of the `result_norm` slot that is *not* ordered by `context->stream` — the reused-arena-slot family in -`docs/315-campaign/evidence/315-what-the-synchronise-drains.md` — plus the unmeasured §51 site. The measured, +`dev/315/315-campaign/evidence/315-what-the-synchronise-drains.md` — plus the unmeasured §51 site. The measured, unexplained fact handed forward is that `result_norm` changes between two consecutive identical plain reads in a faulting arm, at two different addresses, with no pending host writeback at the engine's logits read, and with a fresh synchronise failing to stop it. @@ -682,12 +682,12 @@ with a fresh synchronise failing to stop it. the `MUL_MAT_ID` op-test are **unchanged** (`8 pass, 0 skip, 0 fail`, `mxfp4` nmse **1.34e-05 … 1.61e-05**, an exact match to the recorded baseline, same two kernels); and the dense-model control (Qwen3-8B, dense, not MoE) is **clean and unchanged** (50 responses, 1 distinct content, zero fault signatures). Evidence: -`docs/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md` and +`dev/315/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md` and `315-task3-dense-control-clean-at-moved-pin.md`. **CI did not pass on arrival.** It required fixing two pre-existing defects — a half-move (`95238dc` moved the `llama.cpp` gitlink without `registry/architectures.json`) and 46 files missing the full copyright notice — -recorded in `docs/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md`, with no gate loosened. +recorded in `dev/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md`, with no gate loosened. ### 2026-10-18 — candidate 1: the CWSR save area in GTT (CWSR off stays the shipped default) diff --git a/README.md b/README.md index b29cdb81a..85b3728b2 100644 --- a/README.md +++ b/README.md @@ -25,16 +25,15 @@ the server you talk to (its catalog, downloads, router and UI), and for the mode Lemonade runs the engine as one of its backends, the same way it runs `llama-server`. The engine serves each model behind an OpenAI-compatible API (`1bit serve`), whatever device runs it: -- the XDNA 2 NPU engine, on our own NPU runtime (no XRT; [docs/npu.md](docs/npu.md#the-npu-runtime)) +- the XDNA 2 NPU engine, on our own NPU runtime (no XRT; [docs/npu.md](docs/npu.md#the-npu-runtime)); in progress: the release carries the runtime, and the NPU kernels built from source are not merged yet - HRX on the Radeon iGPU (AMD's ggml-hrx with our Loom kernels), the engine's one GPU route ([docs/hrx.md](docs/hrx.md)), and the CPU from the same llama.cpp build -- MoE experts streamed from the drive (`1bit serve --moe-slots N`), for MoE models larger than memory: kept, but not in this build until it moves to HRX with Qwen3.8-Flash-Next ([docs/moe-streaming.md](docs/moe-streaming.md)) - DwarfStar, for DeepSeek V4 Flash, GLM 5.x and Qwen3.8-Flash-Next in its own GGUFs, with SSD expert streaming - ONNX Runtime GenAI models (Lemonade's ONNX format) on the CPU ([docs/onnx.md](docs/onnx.md)) - Laya, which decides where each request runs - every Hugging Face model architecture, kept current by a daily census -Every tuned setting `1bit serve` gives a backend is a recipe with its measurement attached -([docs/recipes.md](docs/recipes.md)), and the serve numbers in these docs come from +A tuned setting `1bit serve` gives a backend is a recipe with its measurement attached +([docs/recipes.md](docs/recipes.md); the built-in set is empty until there are HRX measurements), and the serve numbers in these docs come from `tools/bench.py`, which A/B-measures configurations against a baseline in the same run ([docs/bench.md](docs/bench.md)). HRX pin bumps and PRs labelled `hw-gate` are checked on the Strix Halo GPU before merge by the hardware gate: kernel tests, known answers on three models, @@ -44,7 +43,9 @@ On the GPU the engine is pure HRX with HIP kernels; no Vulkan, ROCm backend, Met Every other backend and platform (Vulkan, ROCm, CUDA, Apple's MLX and Metal, Windows) is Lemonade's, through its own backends and builds. -**Install:** through Lemonade. Packages ship every Sunday, rebuilt at that week's upstream pins: +**Install:** through Lemonade. The release is HRX with our HIP kernels and the NPU engine, all inside +Lemonade. The weekly release run (`scripts/weekly.sh`, on Sunday) rebuilds the packages at that week's +upstream pins; the first release is not published yet. The packages: Lemonade with the onebit recipe and the engine (unpack it and run `lemond-onebit`), and 1bit OS, a Lemonade appliance with the engine on a bootable USB stick ([docs/releases.md](docs/releases.md)). The engine has no standalone package of its own. @@ -56,12 +57,13 @@ The engine has no standalone package of its own. > > **Status:** the engine runs inside Lemonade through `1bit serve` ([docs/lemonade.md](docs/lemonade.md), > [docs/serve.md](docs/serve.md)); the NPU and HRX each pass its end-to-end test on -> Strix Halo, and the Lemonade recipe that runs it (`onebit`, in our fork +> Strix Halo (the NPU route is still in progress for the release, above), and the Lemonade recipe that runs it (`onebit`, in our fork > [1bit-MONSTER/lemonade](https://github.com/1bit-MONSTER/lemonade)) passes Lemonade's own LLM test > suite on HRX. Following a review,[^geramyl] the engine no longer > vendors Lemonade: Lemonade is the host, 1bit is the engine inside it. Ported so far: HRX on AMD's live ggml-hrx, -> pinned since 2026-10-18 to AMD's `e3e6a976` hrx-system line with llama.cpp `8d9ce2c1a` (our four fixes -> replayed plus our long-context live-out fix for ROCm/hrx-system#1535; against the old pins on Strix Halo: +> on AMD's `e3e6a976` hrx-system line since 2026-10-18, now hrx-system `82804197` with llama.cpp `00f2a68de` +> (2026-10-19, #458, #461; our four fixes replayed plus our long-context live-out fix for ROCm/hrx-system#1535; +> measured when the line moved on 2026-10-18 with llama.cpp `8d9ce2c1a`, #441, against the old pins on Strix Halo: > test-backend-ops 1141/1141, long-prompt needles 20/20 + 5/5 + 5/5, KLD 0 and identical output on two > models, pp512 +4.4% / +7.8%, tg128 +2.5% / +2.8%) > ([docs/hrx.md](docs/hrx.md); its decode-split race, #123/#140, is fixed and the kernel is on by default; Q2_K, IQ2 and IQ3_XXS GGUFs run on HRX instead of the CPU, and `--mtp` on Qwen3.8-27B runs NaN-free since #257), the NPU engine on full ELFs and on our own diff --git a/docs/315-ENDOGAME.md b/dev/315/315-ENDOGAME.md similarity index 97% rename from docs/315-ENDOGAME.md rename to dev/315/315-ENDOGAME.md index ea9bbf58f..707e761b2 100644 --- a/docs/315-ENDOGAME.md +++ b/dev/315/315-ENDOGAME.md @@ -33,9 +33,9 @@ Everything below is committed to `1bit/315-preregistration`. Nothing is only in | campaign of record | `315cwsr70` — 70 pairs → ~70 fix-cell arms, `amdgpu.cwsr_enable=0` | | campaign started | 2026-10-12 ~16:54 box time; box booted 15:15:40 | | expected duration | ~8 h at ~4.7–5.3 min/arm (measure, don't assume — see §1) | -| H1 | **FALSIFIED** — run 7 ran on a fresh boot too (two independent clocks; `docs/315-H1-FALSIFIED.md`) | +| H1 | **FALSIFIED** — run 7 ran on a fresh boot too (two independent clocks; `dev/315/315-H1-FALSIFIED.md`) | | H2 | **the leading and only surviving candidate** — mid-wave preemption under memory pressure | -| H3/H4 | dead (the GPU-MV sizing defect is proven fixed on this box; `docs/315-cwsr-size-excluded-PROOF.md`) | +| H3/H4 | dead (the GPU-MV sizing defect is proven fixed on this box; `dev/315/315-cwsr-size-excluded-PROOF.md`) | Criterion status at the time of writing: **1 open** (needs the control), **2 open** (no repair landed yet), **3 in progress** (campaign running), **4 not started**, **5 not started**. @@ -86,7 +86,7 @@ change the criteria — §8 forbids both. Re-run only for a **recorded invalid** ## 2. Then the control (this is criterion 1's decisive test, and H2's only faulting arm) -`docs/315-campaign/315-cwsr-control.sh` removes the `cwsr_enable=0` drop-in, reboots, and prints the +`dev/315/315-campaign/315-cwsr-control.sh` removes the `cwsr_enable=0` drop-in, reboots, and prints the command to run the identical campaign at `cwsr_enable=1`. It is now sized at **20 pairs** — at the 26% baseline that gives ~5.2 expected faults and `P(0) = 0.002`, so it is decisive for H1, and it supplies the few faulting arms H2 needs. @@ -96,7 +96,7 @@ campaign of record is finished, because the reboot ends it: ```bash # on ryzen, from ~/wt315-prereg -scp docs/315-campaign/315-cwsr-control.sh strixhalo:~/wt/ +scp dev/315/315-campaign/315-cwsr-control.sh strixhalo:~/wt/ ssh strixhalo 'cd ~/wt && TAG=315ctl1 PAIRS=20 bash 315-cwsr-control.sh' ``` @@ -174,8 +174,8 @@ The candidate patch branch `1bit/315-coherence-fwd` was **deleted** as falsified ```bash # both baselines are committed; compare against them, do not re-derive -docs/315-campaign/evidence/315-engine-ci-baseline.md -docs/315-campaign/evidence/315-hrx-tests-baseline.md +dev/315/315-campaign/evidence/315-engine-ci-baseline.md +dev/315/315-campaign/evidence/315-hrx-tests-baseline.md ``` Two gates: (a) `1bit-MONSTER/engine` CI green; (b) the **dense-model control** and the diff --git a/docs/315-H1-FALSIFIED.md b/dev/315/315-H1-FALSIFIED.md similarity index 100% rename from docs/315-H1-FALSIFIED.md rename to dev/315/315-H1-FALSIFIED.md diff --git a/docs/315-STATUS.md b/dev/315/315-STATUS.md similarity index 98% rename from docs/315-STATUS.md rename to dev/315/315-STATUS.md index 4c9304e51..82929aa30 100644 --- a/docs/315-STATUS.md +++ b/dev/315/315-STATUS.md @@ -25,7 +25,7 @@ negative" below is the resume-here state, and it supersedes the criterion tables | task | state | |---|---| | task-1..3 | complete (earlier sessions) | -| task-4 | completed against a FALSIFIED candidate; retracted in the record — read `docs/315-campaign/evidence/ISSUE-COMMENT-5-2026-10-10.md` addendum | +| task-4 | completed against a FALSIFIED candidate; retracted in the record — read `dev/315/315-campaign/evidence/ISSUE-COMMENT-5-2026-10-10.md` addendum | | task-5 | complete — falsified-lever table rebuilt from the run consoles | | task-6 | complete — closed families cited at source with verified file:line | | task-7 | complete — the preemption-window lead recorded as a lead, with its two unbuilt instruments | @@ -64,7 +64,7 @@ no prior campaigns, cool box. Run 7 ran on a long-uptime box. So a clean `cwsr=0 class of error ("the rate followed memory-pressure settings"). **H1 CONTROL** — fresh boot with `cwsr_enable=1`, identical campaign. Scripted and ready: -`docs/315-campaign/315-cwsr-control.sh`. Sized to **20 arms, not 60**: at the 26% baseline, +`dev/315/315-campaign/315-cwsr-control.sh`. Sized to **20 arms, not 60**: at the 26% baseline, 20 arms give ~5.2 expected faults and P(0 faults) = 0.002, which is decisive; 60 would cost ~3 extra hours for no added discriminating power. @@ -98,10 +98,10 @@ SDK ships. | campaign console | strixhalo `~/wt/315cwsr60-console.log` | | eviction stream | strixhalo `~/wt/315cwsr60-evict/kfd_smi.tsv` | | run 7 (the faulting baseline) | `~/wt/315evict7-console.log`, `~/wt/315evict7-evict/correlated-p3l.tsv` | -| harness | `docs/315-campaign/315-evict-trace.sh` (+ `tools/arm-pid.py`, `tools/lever-census.py`) | -| harness defects found+fixed | `docs/315-campaign/evidence/315-evict-trace-instrument.md` §6a–§6h | -| regression baselines | `docs/315-campaign/evidence/315-hrx-tests-baseline.md`, `315-engine-ci-baseline.md` | -| box-wide lock finding | `docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` | +| harness | `dev/315/315-campaign/315-evict-trace.sh` (+ `tools/arm-pid.py`, `tools/lever-census.py`) | +| harness defects found+fixed | `dev/315/315-campaign/evidence/315-evict-trace-instrument.md` §6a–§6h | +| regression baselines | `dev/315/315-campaign/evidence/315-hrx-tests-baseline.md`, `315-engine-ci-baseline.md` | +| box-wide lock finding | `dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` | ## Honest bottom line @@ -155,7 +155,7 @@ Recorded in advance so it cannot be read post hoc. `~/wt/315cwsr60-console.log`, tag `315cwsr60`, 48 pairs (~66 arms), `cwsr=0`, ~5.5 min/arm. **Next actions after it completes, in order:** (1) read its rate; (2) **H1 control** — fresh -boot, `cwsr=1`, 20 arms (`docs/315-campaign/315-cwsr-control.sh`); (3) only then H2 +boot, `cwsr=1`, 20 arms (`dev/315/315-campaign/315-cwsr-control.sh`); (3) only then H2 instrumentation. --- @@ -226,8 +226,8 @@ reason. | source | says | |---|---| | the objective (user-provided) | "≥60 arms, every arm inside the declared regime gate, zero faults across both faces" | -| `docs/315-preregistration.md` §6 | "Target: **≥60 in-regime clean arms in the fix cell**, with paired control arms running in the same session… Hard cap: **80 scheduled pairs** (160 arms)" | -| `docs/315-preregistration.md` §7 | "The campaign passes only if, **in the fix cell**, ≥60 in-regime arms are clean…" | +| `dev/315/315-preregistration.md` §6 | "Target: **≥60 in-regime clean arms in the fix cell**, with paired control arms running in the same session… Hard cap: **80 scheduled pairs** (160 arms)" | +| `dev/315/315-preregistration.md` §7 | "The campaign passes only if, **in the fix cell**, ≥60 in-regime arms are clean…" | The pre-registration counts **fix-cell arms**, not total arms. And the driver emits exactly **one fix-cell (`l`) arm per pair** — measured, not assumed: the 8-pair run produced 8 `l` arms and 3 @@ -333,7 +333,7 @@ Proven on a fixture where it changes the answer (`0/3 = 0%` without it → `1/2 difference being a real HTTP-500 fault). Re-checked on run 7: still 3/11, no missed arms, so it is a guard and not a revision. -**2. The endgame runbook.** (`f3ac0a5`) `docs/315-ENDOGAME.md` — exact commands, the decision table +**2. The endgame runbook.** (`f3ac0a5`) `dev/315/315-ENDOGAME.md` — exact commands, the decision table for the control, and the standing constraints plus seven traps already paid for. Written because the session that started the campaign will not outlast it. @@ -589,7 +589,7 @@ extensions of this one. - **The cwsr=0 drop-in's bytes were never preserved.** The ledger cited `~/work/amd_cwsr.cfg.backup` md5 `a8b25738...`, but the file does not exist anywhere under `$HOME`, and the repo had recorded only the path and md5. A rollback that lives only at a path is - not a rollback. The content is now committed at `docs/315-campaign/evidence/amd_cwsr.cfg`, labelled + not a rollback. The content is now committed at `dev/315/315-campaign/evidence/amd_cwsr.cfg`, labelled as a reconstruction from documented semantics, and the box's boot config is restored (`cwsr_enable=0` in 3 grub entries, new md5 **e20f05bf91d50b645c0fbc8d11998e86**; the old md5 is uncitable). No finding depends on the missing bytes - every arm's cwsr state was read live. @@ -736,7 +736,7 @@ This is the pre-registration's OWN refutation branch (`fc7ffd5`): The one objection that would have made it vacuous — that `hrx_synchronous_d2h` might itself end in an incoherent read — was checked at source and **disproved** -(`docs/315-campaign/evidence/315-ab1-fix-path-is-coherent.md`): the runtime's download routes to the +(`dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md`): the runtime's download routes to the amdgpu staging path, whose pool is created `HOST_LOCAL | HOST_COHERENT | DEVICE_VISIBLE` (`host_queue_staging.c:427`). The fix genuinely supplies a coherent read; it simply is not where the corruption enters. @@ -769,7 +769,7 @@ in-flight work**, not the forcer process and not the read path. | criterion | state | |---|---| | 1 — named mechanism + discriminating experiment | **trigger MET** (forced eviction, replicated, both faces); cause **NOT** located | -| 2 — forward-only fix landed | **NOT ACHIEVED.** The prepared read-path fix is refuted by this A/B. Patch preserved at `docs/315-campaign/patches/315-coherent-read-fix.patch`, NOT to be landed. | +| 2 — forward-only fix landed | **NOT ACHIEVED.** The prepared read-path fix is refuted by this A/B. Patch preserved at `dev/315/315-campaign/patches/315-coherent-read-fix.patch`, NOT to be landed. | | 3 — >=60 in-regime arms, zero faults | not met in the sense that matters: the 0/70 run has no eviction cell, and on a fixed build the bar needs arms that reach the band | | 4 — no regression | not reached (nothing landed to regress) | @@ -933,7 +933,7 @@ Three specific bounded checks are named, each of which could silently drop a pro hazard behaviour for larger spans is unread. The full revised objective is written out at -`docs/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md`, with the established-facts list, the +`dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md`, with the established-facts list, the re-scoped criteria, the revised boundaries and the added constraints. ## PUSH IS BLOCKED — credentials expired mid-session @@ -1355,7 +1355,7 @@ Recording a constraint violation rather than continuing past it. ## The constraint and the arithmetic The objective sets **≤12 hours of strixhalo box time in total**. The record shows this goal turn had already -spent **~10–11.1 h of that ceiling** before today's runs (`docs/315-STATUS.md` budget table: ~2 h "before the +spent **~10–11.1 h of that ceiling** before today's runs (`dev/315/315-STATUS.md` budget table: ~2 h "before the relaunch" + ~7.8 h "campaign of record" ≈ 10 h, with a later note putting it at 11.1 h). Today's GPU-active work on my side: @@ -1782,7 +1782,7 @@ recorded 16/17 did not reproduce here. ### Evidence -`docs/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md`, +`dev/315/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md`, `315-row-size-scored-from-archive.md`, `315-node-2948-is-an-unnamed-moe-accumulation.md` (with its withdrawal note), and the coordination note in `315-COORDINATION.md`. Every number above is reproduced by a command in one of those files; none required GPU time, all were read from the archived arm logs at pin `e44c9d01a`. @@ -1817,7 +1817,7 @@ set. That decides whether "layer 41" is the defect or the instrument. `nan=1`, `side=0/0`), so its cells are not a clean contrast. Its arms were run by the peer session, read here only. -Evidence: `docs/315-campaign/evidence/315-onset-layer-is-config-dependent.md`. +Evidence: `dev/315/315-campaign/evidence/315-onset-layer-is-config-dependent.md`. --- diff --git a/docs/315-campaign/315-HANDOFF-box.md b/dev/315/315-campaign/315-HANDOFF-box.md similarity index 98% rename from docs/315-campaign/315-HANDOFF-box.md rename to dev/315/315-campaign/315-HANDOFF-box.md index c8535028c..c5403c528 100644 --- a/docs/315-campaign/315-HANDOFF-box.md +++ b/dev/315/315-campaign/315-HANDOFF-box.md @@ -19,7 +19,7 @@ limitations under the License. Written 2026-10-15 by the #315 goal session when it closed out. **Read this before launching anything here.** The full record is in the repo worktree `~/wt315-prereg` (branch `1bit/315-preregistration`, PR #414 on -`1bit-MONSTER/engine`); `docs/315-STATUS.md` is its single entry point and `315-COORDINATION.md` is the +`1bit-MONSTER/engine`); `dev/315/315-STATUS.md` is its single entry point and `315-COORDINATION.md` is the inter-session channel. ## 1. Box state, handed over clean diff --git a/docs/315-campaign/315-analyse.py b/dev/315/315-campaign/315-analyse.py similarity index 98% rename from docs/315-campaign/315-analyse.py rename to dev/315/315-campaign/315-analyse.py index 9734b4f5d..4158d0977 100755 --- a/docs/315-campaign/315-analyse.py +++ b/dev/315/315-campaign/315-analyse.py @@ -16,7 +16,7 @@ """engine#315 pre-registered analyser. Reads the campaign console produced by `315-paired.sh` and applies the classification fixed in -`docs/315-preregistration.md` — which differs from `issue-315-analysis.py` in one decisive way: +`dev/315/315-preregistration.md` — which differs from `issue-315-analysis.py` in one decisive way: Face B (invalid UTF-8 / JSON parse failure) is a FAULT face, not "harness-side truncation". The issue's own record characterises invalid-UTF-8 output as the second real defect, and diff --git a/docs/315-campaign/315-attrib.sh b/dev/315/315-campaign/315-attrib.sh similarity index 100% rename from docs/315-campaign/315-attrib.sh rename to dev/315/315-campaign/315-attrib.sh diff --git a/docs/315-campaign/315-campaign.sh b/dev/315/315-campaign/315-campaign.sh similarity index 99% rename from docs/315-campaign/315-campaign.sh rename to dev/315/315-campaign/315-campaign.sh index 45c0d0c73..c848d9661 100644 --- a/docs/315-campaign/315-campaign.sh +++ b/dev/315/315-campaign/315-campaign.sh @@ -15,7 +15,7 @@ # limitations under the License. # engine#315: in-regime acceptance campaign runner + verdict summary. # -# Pre-registered settings (docs/315-preregistration.md sections 3, 6, 7, 18b): +# Pre-registered settings (dev/315/315-preregistration.md sections 3, 6, 7, 18b): # * regime: REGIME_LO/HI tuned so the arm lands in [95, 110) GiB with the resident inventory; the arm's # own gtt_start is what decides membership (gate >= 95 GiB). # * EXTRA_ENV is EMPTY: the campaign measures the shipped configuration, so the sentinel fill, the copy diff --git a/docs/315-campaign/315-cell-build.sh b/dev/315/315-campaign/315-cell-build.sh similarity index 100% rename from docs/315-campaign/315-cell-build.sh rename to dev/315/315-campaign/315-cell-build.sh diff --git a/docs/315-campaign/315-chain.sh b/dev/315/315-campaign/315-chain.sh similarity index 100% rename from docs/315-campaign/315-chain.sh rename to dev/315/315-campaign/315-chain.sh diff --git a/docs/315-campaign/315-chain2.sh b/dev/315/315-campaign/315-chain2.sh similarity index 100% rename from docs/315-campaign/315-chain2.sh rename to dev/315/315-campaign/315-chain2.sh diff --git a/docs/315-campaign/315-cheap-probe.sh b/dev/315/315-campaign/315-cheap-probe.sh similarity index 100% rename from docs/315-campaign/315-cheap-probe.sh rename to dev/315/315-campaign/315-cheap-probe.sh diff --git a/docs/315-campaign/315-classify.py b/dev/315/315-campaign/315-classify.py similarity index 100% rename from docs/315-campaign/315-classify.py rename to dev/315/315-campaign/315-classify.py diff --git a/docs/315-campaign/315-cwsr-control.sh b/dev/315/315-campaign/315-cwsr-control.sh similarity index 100% rename from docs/315-campaign/315-cwsr-control.sh rename to dev/315/315-campaign/315-cwsr-control.sh diff --git a/docs/315-campaign/315-detector.py b/dev/315/315-campaign/315-detector.py similarity index 100% rename from docs/315-campaign/315-detector.py rename to dev/315/315-campaign/315-detector.py diff --git a/docs/315-campaign/315-differential.sh b/dev/315/315-campaign/315-differential.sh similarity index 100% rename from docs/315-campaign/315-differential.sh rename to dev/315/315-campaign/315-differential.sh diff --git a/docs/315-campaign/315-dose.sh b/dev/315/315-campaign/315-dose.sh similarity index 100% rename from docs/315-campaign/315-dose.sh rename to dev/315/315-campaign/315-dose.sh diff --git a/docs/315-campaign/315-evict-trace.sh b/dev/315/315-campaign/315-evict-trace.sh similarity index 100% rename from docs/315-campaign/315-evict-trace.sh rename to dev/315/315-campaign/315-evict-trace.sh diff --git a/docs/315-campaign/315-faceb.sh b/dev/315/315-campaign/315-faceb.sh similarity index 100% rename from docs/315-campaign/315-faceb.sh rename to dev/315/315-campaign/315-faceb.sh diff --git a/docs/315-campaign/315-fix-build.sh b/dev/315/315-campaign/315-fix-build.sh similarity index 100% rename from docs/315-campaign/315-fix-build.sh rename to dev/315/315-campaign/315-fix-build.sh diff --git a/docs/315-campaign/315-forcebarrier.sh b/dev/315/315-campaign/315-forcebarrier.sh similarity index 100% rename from docs/315-campaign/315-forcebarrier.sh rename to dev/315/315-campaign/315-forcebarrier.sh diff --git a/docs/315-campaign/315-fp-hunt.sh b/dev/315/315-campaign/315-fp-hunt.sh similarity index 100% rename from docs/315-campaign/315-fp-hunt.sh rename to dev/315/315-campaign/315-fp-hunt.sh diff --git a/docs/315-campaign/315-fp2.sh b/dev/315/315-campaign/315-fp2.sh similarity index 100% rename from docs/315-campaign/315-fp2.sh rename to dev/315/315-campaign/315-fp2.sh diff --git a/docs/315-campaign/315-graphsync.sh b/dev/315/315-campaign/315-graphsync.sh similarity index 100% rename from docs/315-campaign/315-graphsync.sh rename to dev/315/315-campaign/315-graphsync.sh diff --git a/docs/315-campaign/315-idprobe.sh b/dev/315/315-campaign/315-idprobe.sh similarity index 98% rename from docs/315-campaign/315-idprobe.sh rename to dev/315/315-campaign/315-idprobe.sh index 1d8696d74..75562d9b2 100755 --- a/docs/315-campaign/315-idprobe.sh +++ b/dev/315/315-campaign/315-idprobe.sh @@ -41,7 +41,7 @@ # Usage: 315-idprobe.sh [arms] [tag] # # Note on the gate: the composition target defaults to the module's 80 GiB ("State" section of -# docs/315-mechanism.md: the probe wrappers used to override it *down* to 70 and under-shot the gate). For the +# dev/315/315-mechanism.md: the probe wrappers used to override it *down* to 70 and under-shot the gate). For the # strictest reading of the in-regime requirement - task-2's "gtt_start >= 101 GiB" - export REGIME_LO=85 # REGIME_HI=95: neighbours at 85 GiB put the detector's ~19-21 GiB at ~101-106, inside the 110 GiB upper bound. # The arms are labelled by their own gtt_start either way, so nothing is inferred from the setting. diff --git a/docs/315-campaign/315-instr-build.sh b/dev/315/315-campaign/315-instr-build.sh similarity index 100% rename from docs/315-campaign/315-instr-build.sh rename to dev/315/315-campaign/315-instr-build.sh diff --git a/docs/315-campaign/315-instr-capture.sh b/dev/315/315-campaign/315-instr-capture.sh similarity index 100% rename from docs/315-campaign/315-instr-capture.sh rename to dev/315/315-campaign/315-instr-capture.sh diff --git a/docs/315-campaign/315-janitor.sh b/dev/315/315-campaign/315-janitor.sh similarity index 100% rename from docs/315-campaign/315-janitor.sh rename to dev/315/315-campaign/315-janitor.sh diff --git a/docs/315-campaign/315-mode.sh b/dev/315/315-campaign/315-mode.sh similarity index 100% rename from docs/315-campaign/315-mode.sh rename to dev/315/315-campaign/315-mode.sh diff --git a/docs/315-campaign/315-paired.sh b/dev/315/315-campaign/315-paired.sh similarity index 96% rename from docs/315-campaign/315-paired.sh rename to dev/315/315-campaign/315-paired.sh index 34ea3e124..88fd91711 100755 --- a/docs/315-campaign/315-paired.sh +++ b/dev/315/315-campaign/315-paired.sh @@ -21,7 +21,7 @@ # matches; they carry the Face-B count and the startup-allocation failures that are classified # invalid rather than faulted.) # -# Classification, fixed in the pre-registration (docs/315-preregistration.md): +# Classification, fixed in the pre-registration (dev/315/315-preregistration.md): # invalid startup allocation failure, server died before the request phase, or no gtt.log # out-of-regime gtt_start < 95 GiB (reported, not counted) # fault nan>0 or hsa>0 or parse>0 or utf8>0 or idn>1 or verdict==TRUNCATED or req<66 @@ -122,7 +122,7 @@ arm() { # arm hv52=$(grep -c "memory_type=0x52 size=619520" "$O/server.log" 2>/dev/null || true); hv52=${hv52:-0} hv56=$(grep -c "memory_type=0x56 size=619520" "$O/server.log" 2>/dev/null || true); hv56=${hv56:-0} local g1; g1=$(gtt) - # The gate is two-sided (docs/315-preregistration.md 5): above the upper bound the detector fails to + # The gate is two-sided (dev/315/315-preregistration.md 5): above the upper bound the detector fails to # allocate and the arm is invalid, so it must never be recorded as clean. The analyser already treats it # as out-of-regime; recording it as clean here was a gap. if [ "${g0:-0}" -lt 95 ] || [ "${g0:-0}" -gt 110 ]; then verdict=OOR @@ -145,13 +145,13 @@ arm() { # arm for P in $(pgrep -f "port $port" 2>/dev/null); do kill -TERM $P 2>/dev/null; done } -# Pre-registered stopping rule (docs/315-preregistration.md §6): run pairs until the fix cell has +# Pre-registered stopping rule (dev/315/315-preregistration.md §6): run pairs until the fix cell has # MINCLEAN (default 60) clean in-regime arms, stop immediately if any fix-cell arm faults (acceptance # requires zero), and never exceed PAIRS pairs. The count is read back from the console, i.e. from the # same pre-registered classification the analyser uses. clean_fix() { grep -c 'arm l: OK' "$CONSOLE" 2>/dev/null || true; } fault_fix() { grep -c 'arm l: \(FAULT\|INVALID\)' "$CONSOLE" 2>/dev/null || true; } -# Control cadence. The acceptance criterion (docs/315-preregistration.md §7) is on the FIX cell, and a +# Control cadence. The acceptance criterion (dev/315/315-preregistration.md §7) is on the FIX cell, and a # control arm costs the same box time as a fix arm, so controls run every CONTROL_EVERY-th pair (default # 3): a live positive control spread across the whole session instead of 1:1 pairing, which would double # the run and leave no budget for the regression cell. §7 does not constrain the ratio; the analyser diff --git a/docs/315-campaign/315-regime.sh b/dev/315/315-campaign/315-regime.sh similarity index 100% rename from docs/315-campaign/315-regime.sh rename to dev/315/315-campaign/315-regime.sh diff --git a/docs/315-campaign/315-regression.sh b/dev/315/315-campaign/315-regression.sh similarity index 98% rename from docs/315-campaign/315-regression.sh rename to dev/315/315-campaign/315-regression.sh index dbc6179a7..3f2bc50b2 100755 --- a/docs/315-campaign/315-regression.sh +++ b/dev/315/315-campaign/315-regression.sh @@ -15,7 +15,7 @@ # limitations under the License. # engine#315 regression cell (task-6). # -# Reproduces the recorded baseline from docs/315-preregistration.md 9b, from the SAME source revision as +# Reproduces the recorded baseline from dev/315/315-preregistration.md 9b, from the SAME source revision as # the fix cell (~/wt/llama-315fix), because test-backend-ops is absent from the study's builds (their # recipe sets -DLLAMA_BUILD_TESTS=OFF). Two checks: # diff --git a/docs/315-campaign/315-rowprobe.sh b/dev/315/315-campaign/315-rowprobe.sh similarity index 100% rename from docs/315-campaign/315-rowprobe.sh rename to dev/315/315-campaign/315-rowprobe.sh diff --git a/docs/315-campaign/315-trace-premise.sh b/dev/315/315-campaign/315-trace-premise.sh similarity index 100% rename from docs/315-campaign/315-trace-premise.sh rename to dev/315/315-campaign/315-trace-premise.sh diff --git a/docs/315-campaign/315-wbfp-read.py b/dev/315/315-campaign/315-wbfp-read.py similarity index 97% rename from docs/315-campaign/315-wbfp-read.py rename to dev/315/315-campaign/315-wbfp-read.py index 929f431a2..2927e12e2 100755 --- a/docs/315-campaign/315-wbfp-read.py +++ b/dev/315/315-campaign/315-wbfp-read.py @@ -15,7 +15,7 @@ # limitations under the License. """engine#315 — read [hrx-wb-fp] records and apply the pre-registered outcome table. -The instrumentation (docs/315-instrumentation.patch, enabled by +The instrumentation (dev/315/315-instrumentation.patch, enabled by GGML_HRX_WRITEBACK_FINGERPRINT=1) logs one line per deferred host writeback: [hrx-wb-fp] value= size= barrier=<0|1> rec_hash= sync_hash= dst_hash= @@ -25,7 +25,7 @@ `rec_class` is the state of the mapped staging buffer **when the writeback was queued**, `sync_class` immediately **before** the host memcpy, `dst_class` the host destination **after** it. -Pre-registered reading (docs/315-instrumentation-plan.md): +Pre-registered reading (dev/315/315-instrumentation-plan.md): sync=NONFINITE, dst=NONFINITE -> producer wrote non-finite values: H1 excluded, H2 stands sync=ZERO/stale, dst=NONFINITE -> host read staging before the device copy landed: H1 diff --git a/docs/315-campaign/315-when-free.sh b/dev/315/315-campaign/315-when-free.sh similarity index 100% rename from docs/315-campaign/315-when-free.sh rename to dev/315/315-campaign/315-when-free.sh diff --git a/docs/315-campaign/EVIDENCE-INDEX.md b/dev/315/315-campaign/EVIDENCE-INDEX.md similarity index 98% rename from docs/315-campaign/EVIDENCE-INDEX.md rename to dev/315/315-campaign/EVIDENCE-INDEX.md index fb2752b5b..d6400f7bf 100644 --- a/docs/315-campaign/EVIDENCE-INDEX.md +++ b/dev/315/315-campaign/EVIDENCE-INDEX.md @@ -20,7 +20,7 @@ limitations under the License. engine#315: evidence index. Maps every objective criterion, every claim the mechanism document makes, and every instrument or tool, to the path that backs it - so the completion audit (and task-7's write-back) can check evidence rather than prose. Sections refer to - docs/315-mechanism.md. --> + dev/315/315-mechanism.md. --> # engine#315 evidence index @@ -106,7 +106,7 @@ Note the glob must be `*-*l-*`, not `*-l-*`. Campaign arms would be `315fix-*`; Every number quoted across these evidence files is re-checked from the box by one command: ```sh -bash docs/315-campaign/tools/verify-evidence.sh # 30 checks, all PASS; ~25 s per check +bash dev/315/315-campaign/tools/verify-evidence.sh # 30 checks, all PASS; ~25 s per check # ----- PASS=30 FAIL=0 ``` diff --git a/docs/315-campaign/ISSUE-COMMENT-2-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-2-2026-10-10.md similarity index 100% rename from docs/315-campaign/ISSUE-COMMENT-2-2026-10-10.md rename to dev/315/315-campaign/ISSUE-COMMENT-2-2026-10-10.md diff --git a/docs/315-campaign/ISSUE-COMMENT-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-2026-10-10.md similarity index 100% rename from docs/315-campaign/ISSUE-COMMENT-2026-10-10.md rename to dev/315/315-campaign/ISSUE-COMMENT-2026-10-10.md diff --git a/docs/315-campaign/ISSUE-COMMENT-3-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-3-2026-10-10.md similarity index 100% rename from docs/315-campaign/ISSUE-COMMENT-3-2026-10-10.md rename to dev/315/315-campaign/ISSUE-COMMENT-3-2026-10-10.md diff --git a/docs/315-campaign/ISSUE-COMMENT-4-DRAFT.md b/dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md similarity index 100% rename from docs/315-campaign/ISSUE-COMMENT-4-DRAFT.md rename to dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md diff --git a/docs/315-campaign/ISSUE-COMMENT-5-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md similarity index 96% rename from docs/315-campaign/ISSUE-COMMENT-5-2026-10-10.md rename to dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md index 19531be5e..078d6d274 100644 --- a/docs/315-campaign/ISSUE-COMMENT-5-2026-10-10.md +++ b/dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md @@ -114,8 +114,8 @@ the first produced 8 clean arms, so the faulting half of that trace is still owe ssh strixhalo 'grep -E "^ pair" ~/wt/{315row,315gsync2,315fbar,315nofa,315pm4}-console.log' # the toggles above ssh strixhalo 'grep -m1 "HRX returned NaN logits" ~/wt/rt-det/315fp2-p2l-*/server.log' # the guard line ssh strixhalo 'grep -m1 "size=44048640" ~/wt/rt-det/315accept-p1c-*/server.log' # the host-visible arena -bash docs/315-campaign/tools/verify-evidence.sh # 30 checks, all PASS -bash docs/315-campaign/315-forcebarrier.sh # the barrier toggle (re-runnable) +bash dev/315/315-campaign/tools/verify-evidence.sh # 30 checks, all PASS +bash dev/315/315-campaign/315-forcebarrier.sh # the barrier toggle (re-runnable) ``` --- @@ -133,7 +133,7 @@ insertions / 3 deletions, and it touches no arena or reuse knob. **Why it is not the fix, and must not be read as progress.** It is the same change as `fb4dff7bb` (`engine-315/host-visible-coherence`), which this document records as falsified in its own pre-registered -campaign - section 13 of `docs/315-mechanism.md`, and the first row of the "Falsified in this round" table +campaign - section 13 of `dev/315/315-mechanism.md`, and the first row of the "Falsified in this round" table above: *"memory coherence is the carrier | `HOST_COHERENT` patch live (`hv52=0 hv56=1`), still 9 clean + 1 fault in 10"*. The mechanism note warns about exactly this trap: *"The fork has a forward-only engine#315 change, and it is not the answer."* diff --git a/docs/315-campaign/evidence/315-0of70-has-no-eviction-cell.md b/dev/315/315-campaign/evidence/315-0of70-has-no-eviction-cell.md similarity index 100% rename from docs/315-campaign/evidence/315-0of70-has-no-eviction-cell.md rename to dev/315/315-campaign/evidence/315-0of70-has-no-eviction-cell.md diff --git a/docs/315-campaign/evidence/315-1984-reconciles-the-two-populations.md b/dev/315/315-campaign/evidence/315-1984-reconciles-the-two-populations.md similarity index 100% rename from docs/315-campaign/evidence/315-1984-reconciles-the-two-populations.md rename to dev/315/315-campaign/evidence/315-1984-reconciles-the-two-populations.md diff --git a/docs/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md b/dev/315/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md similarity index 100% rename from docs/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md rename to dev/315/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md diff --git a/docs/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md b/dev/315/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md similarity index 100% rename from docs/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md rename to dev/315/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md diff --git a/docs/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md b/dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md similarity index 98% rename from docs/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md rename to dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md index 7a4184246..ad964fa55 100644 --- a/docs/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md +++ b/dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md @@ -92,7 +92,7 @@ through the plain host memcpy and through `hrx_synchronous_d2h` should differ on refuted. The record identifies the arm, request and decode step for all 18 events for free, and -`docs/315-campaign/tools/patch-input-f32-sync-twin.py` (verified: applies cleanly, idempotent, compiles with the +`dev/315/315-campaign/tools/patch-input-f32-sync-twin.py` (verified: applies cleanly, idempotent, compiles with the build directory's exact flags) reads exactly that pair, with a content hash so an interior divergence is visible. **One arm, on a step the archive already names.** diff --git a/docs/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md b/dev/315/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md similarity index 100% rename from docs/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md rename to dev/315/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md diff --git a/docs/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md b/dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md similarity index 100% rename from docs/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md rename to dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md diff --git a/docs/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md b/dev/315/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md similarity index 100% rename from docs/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md rename to dev/315/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md diff --git a/docs/315-campaign/evidence/315-PREREG-argsort-precondition.md b/dev/315/315-campaign/evidence/315-PREREG-argsort-precondition.md similarity index 100% rename from docs/315-campaign/evidence/315-PREREG-argsort-precondition.md rename to dev/315/315-campaign/evidence/315-PREREG-argsort-precondition.md diff --git a/docs/315-campaign/evidence/315-PREREG-sync-between-reads.md b/dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md similarity index 100% rename from docs/315-campaign/evidence/315-PREREG-sync-between-reads.md rename to dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md diff --git a/docs/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md b/dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md similarity index 98% rename from docs/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md rename to dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md index 401f6d2de..b465dafca 100644 --- a/docs/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md +++ b/dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md @@ -51,7 +51,7 @@ computes from stale input"*, and the record's own PR #90 was an **input-path** d ## The instrumentation -`docs/315-campaign/tools/patch-input-f32-sync-twin.py` — verified: applies cleanly to +`dev/315/315-campaign/tools/patch-input-f32-sync-twin.py` — verified: applies cleanly to `ggml-hrx.cpp` (4 anchors, exactly one occurrence each), is idempotent, and the patched translation unit **compiles with the exact build flags of the real build directory** (`-fsyntax-only`, exit 0). diff --git a/docs/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md b/dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md similarity index 100% rename from docs/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md rename to dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md diff --git a/docs/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md b/dev/315/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md similarity index 100% rename from docs/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md rename to dev/315/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md diff --git a/docs/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md b/dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md similarity index 99% rename from docs/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md rename to dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md index b7bde744e..25f983ab0 100644 --- a/docs/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md +++ b/dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md @@ -74,7 +74,7 @@ has been shown to remove the defect. ## The bounded negative, as it would be reported under the new criterion 1 **Eliminated by direct measurement, each with its command and output:** the eleven families listed above, -all in `docs/315-campaign/evidence/`. +all in `dev/315/315-campaign/evidence/`. **Surviving families, stated explicitly:** diff --git a/docs/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md b/dev/315/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md similarity index 100% rename from docs/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md rename to dev/315/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md diff --git a/docs/315-campaign/evidence/315-ab-preregistration.md b/dev/315/315-campaign/evidence/315-ab-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-ab-preregistration.md rename to dev/315/315-campaign/evidence/315-ab-preregistration.md diff --git a/docs/315-campaign/evidence/315-ab1-arm-A-result.md b/dev/315/315-campaign/evidence/315-ab1-arm-A-result.md similarity index 97% rename from docs/315-campaign/evidence/315-ab1-arm-A-result.md rename to dev/315/315-campaign/evidence/315-ab1-arm-A-result.md index fb488af93..4281a4a4b 100644 --- a/docs/315-campaign/evidence/315-ab1-arm-A-result.md +++ b/dev/315/315-campaign/evidence/315-ab1-arm-A-result.md @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0 # A/B arm A — the control that must fault (315ab1a), 4 complete rounds -Pre-registered at `fc7ffd5` (`docs/315-campaign/evidence/315-ab-preregistration.md`), which fixed the +Pre-registered at `fc7ffd5` (`dev/315/315-campaign/evidence/315-ab-preregistration.md`), which fixed the binary, the rig, the four reading branches and the cell-`e` comparison BEFORE the run. This file records the arm-A result only, written after the arm finished and before arm B's result is known. diff --git a/docs/315-campaign/evidence/315-ab1-arm-B-interim.md b/dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md similarity index 100% rename from docs/315-campaign/evidence/315-ab1-arm-B-interim.md rename to dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md diff --git a/docs/315-campaign/evidence/315-ab1-cell-c-fault-drift.md b/dev/315/315-campaign/evidence/315-ab1-cell-c-fault-drift.md similarity index 100% rename from docs/315-campaign/evidence/315-ab1-cell-c-fault-drift.md rename to dev/315/315-campaign/evidence/315-ab1-cell-c-fault-drift.md diff --git a/docs/315-campaign/evidence/315-ab1-fix-path-is-coherent.md b/dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md similarity index 100% rename from docs/315-campaign/evidence/315-ab1-fix-path-is-coherent.md rename to dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md diff --git a/docs/315-campaign/evidence/315-ab1-two-drivers-incident.md b/dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md similarity index 98% rename from docs/315-campaign/evidence/315-ab1-two-drivers-incident.md rename to dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md index f1771c41b..c2581a8de 100644 --- a/docs/315-campaign/evidence/315-ab1-two-drivers-incident.md +++ b/dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md @@ -62,7 +62,7 @@ its own neighbour/regime lines, and was stopped before any `round N arm X:` line - no arm was scored under the contention; - **arm A (`315ab1a`) is untouched** — it completed at 10:20:23, before the collision, under a single - driver, and its result (`docs/315-campaign/evidence/315-ab1-arm-A-result.md`) stands; + driver, and its result (`dev/315/315-campaign/evidence/315-ab1-arm-A-result.md`) stands; - the patched binary and the patch file are unaffected. ## Disposition diff --git a/docs/315-campaign/evidence/315-acceptance-harness-faceb-defects.md b/dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md similarity index 97% rename from docs/315-campaign/evidence/315-acceptance-harness-faceb-defects.md rename to dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md index 7193b7361..d9c717d44 100644 --- a/docs/315-campaign/evidence/315-acceptance-harness-faceb-defects.md +++ b/dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md @@ -22,7 +22,7 @@ SPDX-License-Identifier: Apache-2.0 # Two defects in the acceptance harness found by fixing Face B: an unattributable face, and a bound over 100% Found while implementing the user's ruling that Face B is *Content-only / HTTP 500 **or** a genuine -non-UTF-8 body, whichever occurs*. Both defects are in `docs/315-campaign/315-campaign.sh`, the acceptance +non-UTF-8 body, whichever occurs*. Both defects are in `dev/315/315-campaign/315-campaign.sh`, the acceptance campaign summary. Only the first is fixed here; the second is recorded because it is a smaller change to **how a criterion reads** and should not be made silently. @@ -120,7 +120,7 @@ campaign should *say* is a criterion question, and the objective is immutable. ## Evidence - sed -n '52,152p' docs/315-campaign/315-campaign.sh > /tmp/sum.py # the extracted summariser + sed -n '52,152p' dev/315/315-campaign/315-campaign.sh > /tmp/sum.py # the extracted summariser python3 /tmp/sum.py /tmp/t_paired.txt # test A python3 /tmp/sum.py /tmp/t_disc.txt # test B diff --git a/docs/315-campaign/evidence/315-align-frozen-preregistration.md b/dev/315/315-campaign/evidence/315-align-frozen-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-align-frozen-preregistration.md rename to dev/315/315-campaign/evidence/315-align-frozen-preregistration.md diff --git a/docs/315-campaign/evidence/315-align-frozen-result.md b/dev/315/315-campaign/evidence/315-align-frozen-result.md similarity index 100% rename from docs/315-campaign/evidence/315-align-frozen-result.md rename to dev/315/315-campaign/evidence/315-align-frozen-result.md diff --git a/docs/315-campaign/evidence/315-align-preregistration.md b/dev/315/315-campaign/evidence/315-align-preregistration.md similarity index 98% rename from docs/315-campaign/evidence/315-align-preregistration.md rename to dev/315/315-campaign/evidence/315-align-preregistration.md index 9f4877056..5deda97fa 100644 --- a/docs/315-campaign/evidence/315-align-preregistration.md +++ b/dev/315/315-campaign/evidence/315-align-preregistration.md @@ -52,7 +52,7 @@ alignment AND kernel selection — and any result must be read with that in mind ## What the prior work already established (do not re-derive) -From `docs/315-mechanism.md`, the `align=256` + compaction recipe on **Qwen3-0.6B**: +From `dev/315/315-mechanism.md`, the `align=256` + compaction recipe on **Qwen3-0.6B**: default align, quiet 1 run clean 180/180 default align + compaction 2 runs clean 180/180 both diff --git a/docs/315-campaign/evidence/315-align-run1-uninformative.md b/dev/315/315-campaign/evidence/315-align-run1-uninformative.md similarity index 97% rename from docs/315-campaign/evidence/315-align-run1-uninformative.md rename to dev/315/315-campaign/evidence/315-align-run1-uninformative.md index c8460e647..ecd744415 100644 --- a/docs/315-campaign/evidence/315-align-run1-uninformative.md +++ b/dev/315/315-campaign/evidence/315-align-run1-uninformative.md @@ -42,7 +42,7 @@ The pre-registered branch for "align=256 ~= align=4096, both clean" is "the #140 mechanism; leave the FA path". **That branch does not apply, because the arms could not have faulted.** 1. **No regime, no neighbours, no compaction.** GTT was **47 GiB of 120 GiB** — low pressure by - construction. The prior #140 recipe (`docs/315-mechanism.md`) required `align=256` **plus** compaction + construction. The prior #140 recipe (`dev/315/315-mechanism.md`) required `align=256` **plus** compaction (`tee /proc/sys/vm/compact_memory`) and still fired only once in five runs, and the record notes that the one faulty run coincided with foreign load 8-12 while the four clean repeats ran at 4-7. 2. **38 seconds is too short.** Four arms in 38 s gives ~10 s per arm. The eviction window this class needs diff --git a/docs/315-campaign/evidence/315-align-run2-result.md b/dev/315/315-campaign/evidence/315-align-run2-result.md similarity index 100% rename from docs/315-campaign/evidence/315-align-run2-result.md rename to dev/315/315-campaign/evidence/315-align-run2-result.md diff --git a/docs/315-campaign/evidence/315-aql-publish-vs-reorder.md b/dev/315/315-campaign/evidence/315-aql-publish-vs-reorder.md similarity index 100% rename from docs/315-campaign/evidence/315-aql-publish-vs-reorder.md rename to dev/315/315-campaign/evidence/315-aql-publish-vs-reorder.md diff --git a/docs/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md b/dev/315/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md similarity index 100% rename from docs/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md rename to dev/315/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md diff --git a/docs/315-campaign/evidence/315-arm-ledger.csv b/dev/315/315-campaign/evidence/315-arm-ledger.csv similarity index 100% rename from docs/315-campaign/evidence/315-arm-ledger.csv rename to dev/315/315-campaign/evidence/315-arm-ledger.csv diff --git a/docs/315-campaign/evidence/315-arm-ledger.md b/dev/315/315-campaign/evidence/315-arm-ledger.md similarity index 100% rename from docs/315-campaign/evidence/315-arm-ledger.md rename to dev/315/315-campaign/evidence/315-arm-ledger.md diff --git a/docs/315-campaign/evidence/315-asan-preregistration.md b/dev/315/315-campaign/evidence/315-asan-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-asan-preregistration.md rename to dev/315/315-campaign/evidence/315-asan-preregistration.md diff --git a/docs/315-campaign/evidence/315-asan-wiring-verified.md b/dev/315/315-campaign/evidence/315-asan-wiring-verified.md similarity index 100% rename from docs/315-campaign/evidence/315-asan-wiring-verified.md rename to dev/315/315-campaign/evidence/315-asan-wiring-verified.md diff --git a/docs/315-campaign/evidence/315-both-faces-one-trigger.md b/dev/315/315-campaign/evidence/315-both-faces-one-trigger.md similarity index 96% rename from docs/315-campaign/evidence/315-both-faces-one-trigger.md rename to dev/315/315-campaign/evidence/315-both-faces-one-trigger.md index a87ee5ff2..3e43b5978 100644 --- a/docs/315-campaign/evidence/315-both-faces-one-trigger.md +++ b/dev/315/315-campaign/evidence/315-both-faces-one-trigger.md @@ -52,7 +52,7 @@ So **11 forced-eviction faults split 7 Face A / 4 Face B**, and both faces occur cause itself. - **The face that appears is a property of the session, not the event.** An earlier session faulted 16 of 17 arms with `nan=1` each while another faulted half its arms with zero NaN anywhere - (`docs/315-campaign/evidence/315-face-landscape.md`). So the two faces are not two mechanisms so much as + (`dev/315/315-campaign/evidence/315-face-landscape.md`). So the two faces are not two mechanisms so much as two observables of one event, selected by whether the corrupted bytes happen to decode as NaN or as parseable-but-wrong text. diff --git a/docs/315-campaign/evidence/315-campaign-preregistration.md b/dev/315/315-campaign/evidence/315-campaign-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-campaign-preregistration.md rename to dev/315/315-campaign/evidence/315-campaign-preregistration.md diff --git a/docs/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md b/dev/315/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md similarity index 100% rename from docs/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md rename to dev/315/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md diff --git a/docs/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md b/dev/315/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md similarity index 100% rename from docs/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md rename to dev/315/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md diff --git a/docs/315-campaign/evidence/315-candidate-host-staging-coherence.md b/dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md similarity index 100% rename from docs/315-campaign/evidence/315-candidate-host-staging-coherence.md rename to dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md diff --git a/docs/315-campaign/evidence/315-cause-boundary-final.md b/dev/315/315-campaign/evidence/315-cause-boundary-final.md similarity index 100% rename from docs/315-campaign/evidence/315-cause-boundary-final.md rename to dev/315/315-campaign/evidence/315-cause-boundary-final.md diff --git a/docs/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md b/dev/315/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md similarity index 100% rename from docs/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md rename to dev/315/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md diff --git a/docs/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md b/dev/315/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md similarity index 100% rename from docs/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md rename to dev/315/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md diff --git a/docs/315-campaign/evidence/315-chain-plus-early-preregistration.md b/dev/315/315-campaign/evidence/315-chain-plus-early-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-chain-plus-early-preregistration.md rename to dev/315/315-campaign/evidence/315-chain-plus-early-preregistration.md diff --git a/docs/315-campaign/evidence/315-chaining-guard-defect.md b/dev/315/315-campaign/evidence/315-chaining-guard-defect.md similarity index 100% rename from docs/315-campaign/evidence/315-chaining-guard-defect.md rename to dev/315/315-campaign/evidence/315-chaining-guard-defect.md diff --git a/docs/315-campaign/evidence/315-cheap-repro.md b/dev/315/315-campaign/evidence/315-cheap-repro.md similarity index 100% rename from docs/315-campaign/evidence/315-cheap-repro.md rename to dev/315/315-campaign/evidence/315-cheap-repro.md diff --git a/docs/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md b/dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md similarity index 100% rename from docs/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md rename to dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md diff --git a/docs/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md b/dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md similarity index 100% rename from docs/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md rename to dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md diff --git a/docs/315-campaign/evidence/315-corruption-is-one-row-sized.md b/dev/315/315-campaign/evidence/315-corruption-is-one-row-sized.md similarity index 100% rename from docs/315-campaign/evidence/315-corruption-is-one-row-sized.md rename to dev/315/315-campaign/evidence/315-corruption-is-one-row-sized.md diff --git a/docs/315-campaign/evidence/315-corruption-is-wave-granular.md b/dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md similarity index 100% rename from docs/315-campaign/evidence/315-corruption-is-wave-granular.md rename to dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md diff --git a/docs/315-campaign/evidence/315-crit4-control-result.md b/dev/315/315-campaign/evidence/315-crit4-control-result.md similarity index 100% rename from docs/315-campaign/evidence/315-crit4-control-result.md rename to dev/315/315-campaign/evidence/315-crit4-control-result.md diff --git a/docs/315-campaign/evidence/315-deferred-writeback-gap.md b/dev/315/315-campaign/evidence/315-deferred-writeback-gap.md similarity index 99% rename from docs/315-campaign/evidence/315-deferred-writeback-gap.md rename to dev/315/315-campaign/evidence/315-deferred-writeback-gap.md index 7060db774..69ea6d129 100644 --- a/docs/315-campaign/evidence/315-deferred-writeback-gap.md +++ b/dev/315/315-campaign/evidence/315-deferred-writeback-gap.md @@ -128,7 +128,7 @@ That is purpose-built for exactly the question this lead poses. Per writeback it record time than at sync time; - **`rec_nf16/32`, `sync_nf16/32`** — non-finite counts on each side, which is the link to Face A. -And there is a driver already written for it: `docs/315-campaign/315-fp-hunt.sh` runs instrumented +And there is a driver already written for it: `dev/315/315-campaign/315-fp-hunt.sh` runs instrumented in-regime arms and reports `wb_records=$(grep -c "\[hrx-wb-fp\]" ...)` per arm. **Neither has ever produced output.** A search of the whole record (`docs/`, `KNOWN-ISSUES.md`) and of the @@ -216,7 +216,7 @@ bindings reach it. A host-visible binding that the graph **writes** therefore qu ## The logits are exactly that, and not by the direct path -The recorded trace (`docs/315-mechanism.md:34`, and `315-task9-session-state-2026-10-12.md:3304`) shows the +The recorded trace (`dev/315/315-mechanism.md:34`, and `315-task9-session-state-2026-10-12.md:3304`) shows the logits allocation as: type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=619520 diff --git a/docs/315-campaign/evidence/315-device-buffer-probe-proposal.md b/dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md similarity index 99% rename from docs/315-campaign/evidence/315-device-buffer-probe-proposal.md rename to dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md index 764603b31..68969ca2f 100644 --- a/docs/315-campaign/evidence/315-device-buffer-probe-proposal.md +++ b/dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md @@ -167,7 +167,7 @@ box is already carrying. ### The patch is now written and its anchors verified — but it is NOT compile-tested -`docs/315-campaign/tools/patch-device-producer-probe.py` applies the change in the record's own patch style +`dev/315/315-campaign/tools/patch-device-producer-probe.py` applies the change in the record's own patch style (every anchor asserted to occur exactly once, a pre-check over all anchors before any write, an idempotency marker, and a brace-balance check). Verified against a **real copy** of the source, not from memory: diff --git a/docs/315-campaign/evidence/315-device-hunt-boundary.md b/dev/315/315-campaign/evidence/315-device-hunt-boundary.md similarity index 100% rename from docs/315-campaign/evidence/315-device-hunt-boundary.md rename to dev/315/315-campaign/evidence/315-device-hunt-boundary.md diff --git a/docs/315-campaign/evidence/315-differential-artifact.md b/dev/315/315-campaign/evidence/315-differential-artifact.md similarity index 100% rename from docs/315-campaign/evidence/315-differential-artifact.md rename to dev/315/315-campaign/evidence/315-differential-artifact.md diff --git a/docs/315-campaign/evidence/315-dossier-spt-claim-corrected.md b/dev/315/315-campaign/evidence/315-dossier-spt-claim-corrected.md similarity index 100% rename from docs/315-campaign/evidence/315-dossier-spt-claim-corrected.md rename to dev/315/315-campaign/evidence/315-dossier-spt-claim-corrected.md diff --git a/docs/315-campaign/evidence/315-engine-ci-baseline.md b/dev/315/315-campaign/evidence/315-engine-ci-baseline.md similarity index 100% rename from docs/315-campaign/evidence/315-engine-ci-baseline.md rename to dev/315/315-campaign/evidence/315-engine-ci-baseline.md diff --git a/docs/315-campaign/evidence/315-evict-trace-instrument.md b/dev/315/315-campaign/evidence/315-evict-trace-instrument.md similarity index 100% rename from docs/315-campaign/evidence/315-evict-trace-instrument.md rename to dev/315/315-campaign/evidence/315-evict-trace-instrument.md diff --git a/docs/315-campaign/evidence/315-evict-trace-status.md b/dev/315/315-campaign/evidence/315-evict-trace-status.md similarity index 100% rename from docs/315-campaign/evidence/315-evict-trace-status.md rename to dev/315/315-campaign/evidence/315-evict-trace-status.md diff --git a/docs/315-campaign/evidence/315-eviction-hits-the-arm.md b/dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md similarity index 98% rename from docs/315-campaign/evidence/315-eviction-hits-the-arm.md rename to dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md index 0a823bf77..1c2d65603 100644 --- a/docs/315-campaign/evidence/315-eviction-hits-the-arm.md +++ b/dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md @@ -96,7 +96,7 @@ The section above promises to "read the re-run's per-arm SMI stream for arm `315 ## Why not The eviction counters come from a separate privileged watcher, launched by -`docs/315-campaign/315-evict-trace.sh`, not by the campaign driver: +`dev/315/315-campaign/315-evict-trace.sh`, not by the campaign driver: WATCH=$HOME/amd-oss/tools/315-trace/kfd_smi_watch # 315-evict-trace.sh:25 sudo -n true || { echo "ABORT: the watcher needs passwordless sudo"; exit 2; } # :40 diff --git a/docs/315-campaign/evidence/315-eviction-replication-preregistration.md b/dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-eviction-replication-preregistration.md rename to dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md diff --git a/docs/315-campaign/evidence/315-exclusions-vs-falsified-levers.md b/dev/315/315-campaign/evidence/315-exclusions-vs-falsified-levers.md similarity index 100% rename from docs/315-campaign/evidence/315-exclusions-vs-falsified-levers.md rename to dev/315/315-campaign/evidence/315-exclusions-vs-falsified-levers.md diff --git a/docs/315-campaign/evidence/315-extra-env-comma-trap.md b/dev/315/315-campaign/evidence/315-extra-env-comma-trap.md similarity index 100% rename from docs/315-campaign/evidence/315-extra-env-comma-trap.md rename to dev/315/315-campaign/evidence/315-extra-env-comma-trap.md diff --git a/docs/315-campaign/evidence/315-f32-sync-twin-preregistration.md b/dev/315/315-campaign/evidence/315-f32-sync-twin-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-f32-sync-twin-preregistration.md rename to dev/315/315-campaign/evidence/315-f32-sync-twin-preregistration.md diff --git a/docs/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md b/dev/315/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md similarity index 100% rename from docs/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md rename to dev/315/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md diff --git a/docs/315-campaign/evidence/315-fa-masked-v-regime.md b/dev/315/315-campaign/evidence/315-fa-masked-v-regime.md similarity index 100% rename from docs/315-campaign/evidence/315-fa-masked-v-regime.md rename to dev/315/315-campaign/evidence/315-fa-masked-v-regime.md diff --git a/docs/315-campaign/evidence/315-fa-race-pin-sweep.md b/dev/315/315-campaign/evidence/315-fa-race-pin-sweep.md similarity index 100% rename from docs/315-campaign/evidence/315-fa-race-pin-sweep.md rename to dev/315/315-campaign/evidence/315-fa-race-pin-sweep.md diff --git a/docs/315-campaign/evidence/315-face-landscape.md b/dev/315/315-campaign/evidence/315-face-landscape.md similarity index 100% rename from docs/315-campaign/evidence/315-face-landscape.md rename to dev/315/315-campaign/evidence/315-face-landscape.md diff --git a/docs/315-campaign/evidence/315-faceb-amplifier.md b/dev/315/315-campaign/evidence/315-faceb-amplifier.md similarity index 100% rename from docs/315-campaign/evidence/315-faceb-amplifier.md rename to dev/315/315-campaign/evidence/315-faceb-amplifier.md diff --git a/docs/315-campaign/evidence/315-faceb-has-no-capable-instrument.md b/dev/315/315-campaign/evidence/315-faceb-has-no-capable-instrument.md similarity index 100% rename from docs/315-campaign/evidence/315-faceb-has-no-capable-instrument.md rename to dev/315/315-campaign/evidence/315-faceb-has-no-capable-instrument.md diff --git a/docs/315-campaign/evidence/315-faceb-invariant-signature.md b/dev/315/315-campaign/evidence/315-faceb-invariant-signature.md similarity index 100% rename from docs/315-campaign/evidence/315-faceb-invariant-signature.md rename to dev/315/315-campaign/evidence/315-faceb-invariant-signature.md diff --git a/docs/315-campaign/evidence/315-faceb-is-two-populations-and-faces-are-disjoint.md b/dev/315/315-campaign/evidence/315-faceb-is-two-populations-and-faces-are-disjoint.md similarity index 100% rename from docs/315-campaign/evidence/315-faceb-is-two-populations-and-faces-are-disjoint.md rename to dev/315/315-campaign/evidence/315-faceb-is-two-populations-and-faces-are-disjoint.md diff --git a/docs/315-campaign/evidence/315-fault-is-a-per-step-hazard-after-prefill.md b/dev/315/315-campaign/evidence/315-fault-is-a-per-step-hazard-after-prefill.md similarity index 100% rename from docs/315-campaign/evidence/315-fault-is-a-per-step-hazard-after-prefill.md rename to dev/315/315-campaign/evidence/315-fault-is-a-per-step-hazard-after-prefill.md diff --git a/docs/315-campaign/evidence/315-fault-is-one-token-row-and-unifies-the-granularity.md b/dev/315/315-campaign/evidence/315-fault-is-one-token-row-and-unifies-the-granularity.md similarity index 100% rename from docs/315-campaign/evidence/315-fault-is-one-token-row-and-unifies-the-granularity.md rename to dev/315/315-campaign/evidence/315-fault-is-one-token-row-and-unifies-the-granularity.md diff --git a/docs/315-campaign/evidence/315-faults-track-gtt-drift.md b/dev/315/315-campaign/evidence/315-faults-track-gtt-drift.md similarity index 97% rename from docs/315-campaign/evidence/315-faults-track-gtt-drift.md rename to dev/315/315-campaign/evidence/315-faults-track-gtt-drift.md index 23bfef9b9..66003c1ff 100644 --- a/docs/315-campaign/evidence/315-faults-track-gtt-drift.md +++ b/dev/315/315-campaign/evidence/315-faults-track-gtt-drift.md @@ -28,7 +28,7 @@ contradiction in the corpus. ## The tallies, extracted from every recorded arm -Cell x verdict x `gtt_start` (all arms in `docs/315-campaign/evidence/*.md` that carry the fields): +Cell x verdict x `gtt_start` (all arms in `dev/315/315-campaign/evidence/*.md` that carry the fields): 21 l FAULT 98 12 l OK 98 8 l FAULT 97 2 l OK 97 6 c FAULT 98 5 c OK 98 1 c OK 97 diff --git a/docs/315-campaign/evidence/315-ffn-inp-0-carries-nan.md b/dev/315/315-campaign/evidence/315-ffn-inp-0-carries-nan.md similarity index 100% rename from docs/315-campaign/evidence/315-ffn-inp-0-carries-nan.md rename to dev/315/315-campaign/evidence/315-ffn-inp-0-carries-nan.md diff --git a/docs/315-campaign/evidence/315-fingerprint-does-not-measure-per-node-buffer-aliasing.md b/dev/315/315-campaign/evidence/315-fingerprint-does-not-measure-per-node-buffer-aliasing.md similarity index 100% rename from docs/315-campaign/evidence/315-fingerprint-does-not-measure-per-node-buffer-aliasing.md rename to dev/315/315-campaign/evidence/315-fingerprint-does-not-measure-per-node-buffer-aliasing.md diff --git a/docs/315-campaign/evidence/315-first-row-site-is-fixed-layer-41.md b/dev/315/315-campaign/evidence/315-first-row-site-is-fixed-layer-41.md similarity index 100% rename from docs/315-campaign/evidence/315-first-row-site-is-fixed-layer-41.md rename to dev/315/315-campaign/evidence/315-first-row-site-is-fixed-layer-41.md diff --git a/docs/315-campaign/evidence/315-fp2-ordered-fingerprint.md b/dev/315/315-campaign/evidence/315-fp2-ordered-fingerprint.md similarity index 100% rename from docs/315-campaign/evidence/315-fp2-ordered-fingerprint.md rename to dev/315/315-campaign/evidence/315-fp2-ordered-fingerprint.md diff --git a/docs/315-campaign/evidence/315-fresh-arena-preregistration.md b/dev/315/315-campaign/evidence/315-fresh-arena-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-fresh-arena-preregistration.md rename to dev/315/315-campaign/evidence/315-fresh-arena-preregistration.md diff --git a/docs/315-campaign/evidence/315-galign-CORRECTION-my-parse-was-wrong.md b/dev/315/315-campaign/evidence/315-galign-CORRECTION-my-parse-was-wrong.md similarity index 100% rename from docs/315-campaign/evidence/315-galign-CORRECTION-my-parse-was-wrong.md rename to dev/315/315-campaign/evidence/315-galign-CORRECTION-my-parse-was-wrong.md diff --git a/docs/315-campaign/evidence/315-galign-label-discrepancy.md b/dev/315/315-campaign/evidence/315-galign-label-discrepancy.md similarity index 100% rename from docs/315-campaign/evidence/315-galign-label-discrepancy.md rename to dev/315/315-campaign/evidence/315-galign-label-discrepancy.md diff --git a/docs/315-campaign/evidence/315-galign-peer-result.md b/dev/315/315-campaign/evidence/315-galign-peer-result.md similarity index 100% rename from docs/315-campaign/evidence/315-galign-peer-result.md rename to dev/315/315-campaign/evidence/315-galign-peer-result.md diff --git a/docs/315-campaign/evidence/315-glm-align-preregistration.md b/dev/315/315-campaign/evidence/315-glm-align-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-glm-align-preregistration.md rename to dev/315/315-campaign/evidence/315-glm-align-preregistration.md diff --git a/docs/315-campaign/evidence/315-glm-dispatch-inventory.md b/dev/315/315-campaign/evidence/315-glm-dispatch-inventory.md similarity index 100% rename from docs/315-campaign/evidence/315-glm-dispatch-inventory.md rename to dev/315/315-campaign/evidence/315-glm-dispatch-inventory.md diff --git a/docs/315-campaign/evidence/315-glm-never-dispatches-the-changed-kernel.md b/dev/315/315-campaign/evidence/315-glm-never-dispatches-the-changed-kernel.md similarity index 100% rename from docs/315-campaign/evidence/315-glm-never-dispatches-the-changed-kernel.md rename to dev/315/315-campaign/evidence/315-glm-never-dispatches-the-changed-kernel.md diff --git a/docs/315-campaign/evidence/315-gord1-arm-A-result.md b/dev/315/315-campaign/evidence/315-gord1-arm-A-result.md similarity index 100% rename from docs/315-campaign/evidence/315-gord1-arm-A-result.md rename to dev/315/315-campaign/evidence/315-gord1-arm-A-result.md diff --git a/docs/315-campaign/evidence/315-gord1-arm-B-interim.md b/dev/315/315-campaign/evidence/315-gord1-arm-B-interim.md similarity index 100% rename from docs/315-campaign/evidence/315-gord1-arm-B-interim.md rename to dev/315/315-campaign/evidence/315-gord1-arm-B-interim.md diff --git a/docs/315-campaign/evidence/315-gord1-final.md b/dev/315/315-campaign/evidence/315-gord1-final.md similarity index 100% rename from docs/315-campaign/evidence/315-gord1-final.md rename to dev/315/315-campaign/evidence/315-gord1-final.md diff --git a/docs/315-campaign/evidence/315-gord1-preregistration.md b/dev/315/315-campaign/evidence/315-gord1-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-gord1-preregistration.md rename to dev/315/315-campaign/evidence/315-gord1-preregistration.md diff --git a/docs/315-campaign/evidence/315-graph-reorder-lever.md b/dev/315/315-campaign/evidence/315-graph-reorder-lever.md similarity index 100% rename from docs/315-campaign/evidence/315-graph-reorder-lever.md rename to dev/315/315-campaign/evidence/315-graph-reorder-lever.md diff --git a/docs/315-campaign/evidence/315-gtt-start-is-the-discriminator.md b/dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md similarity index 100% rename from docs/315-campaign/evidence/315-gtt-start-is-the-discriminator.md rename to dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md diff --git a/docs/315-campaign/evidence/315-gttstep-preregistration.md b/dev/315/315-campaign/evidence/315-gttstep-preregistration.md similarity index 98% rename from docs/315-campaign/evidence/315-gttstep-preregistration.md rename to dev/315/315-campaign/evidence/315-gttstep-preregistration.md index ed2c0cb60..f66d44c5d 100644 --- a/docs/315-campaign/evidence/315-gttstep-preregistration.md +++ b/dev/315/315-campaign/evidence/315-gttstep-preregistration.md @@ -70,7 +70,7 @@ post-hoc level merging. - **Settles**: whether the level alone is the discriminator, under control. - **Does not settle**: the cause site. Even a clean H-start confirmation leaves "which handoff reads stale data after eviction" open — the read path was eliminated by the A/B - (`docs/315-campaign/evidence/315-ab1-arm-B-interim.md`, `315-ab1-fix-path-is-coherent.md`). + (`dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md`, `315-ab1-fix-path-is-coherent.md`). - **Does not touch criterion 2.** No fix is proposed or tested here. ## Budget and coordination diff --git a/docs/315-campaign/evidence/315-hazard-is-in-decode-not-prefill-index0-never-fails.md b/dev/315/315-campaign/evidence/315-hazard-is-in-decode-not-prefill-index0-never-fails.md similarity index 100% rename from docs/315-campaign/evidence/315-hazard-is-in-decode-not-prefill-index0-never-fails.md rename to dev/315/315-campaign/evidence/315-hazard-is-in-decode-not-prefill-index0-never-fails.md diff --git a/docs/315-campaign/evidence/315-hrx-mem-pool-quarantine.md b/dev/315/315-campaign/evidence/315-hrx-mem-pool-quarantine.md similarity index 100% rename from docs/315-campaign/evidence/315-hrx-mem-pool-quarantine.md rename to dev/315/315-campaign/evidence/315-hrx-mem-pool-quarantine.md diff --git a/docs/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md b/dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md similarity index 100% rename from docs/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md rename to dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md diff --git a/docs/315-campaign/evidence/315-hrx-system-dossier-by-peer-agent.md b/dev/315/315-campaign/evidence/315-hrx-system-dossier-by-peer-agent.md similarity index 100% rename from docs/315-campaign/evidence/315-hrx-system-dossier-by-peer-agent.md rename to dev/315/315-campaign/evidence/315-hrx-system-dossier-by-peer-agent.md diff --git a/docs/315-campaign/evidence/315-hrx-tests-baseline.md b/dev/315/315-campaign/evidence/315-hrx-tests-baseline.md similarity index 100% rename from docs/315-campaign/evidence/315-hrx-tests-baseline.md rename to dev/315/315-campaign/evidence/315-hrx-tests-baseline.md diff --git a/docs/315-campaign/evidence/315-id-producing-kernel.md b/dev/315/315-campaign/evidence/315-id-producing-kernel.md similarity index 100% rename from docs/315-campaign/evidence/315-id-producing-kernel.md rename to dev/315/315-campaign/evidence/315-id-producing-kernel.md diff --git a/docs/315-campaign/evidence/315-idprobe-readpath-refuted.md b/dev/315/315-campaign/evidence/315-idprobe-readpath-refuted.md similarity index 100% rename from docs/315-campaign/evidence/315-idprobe-readpath-refuted.md rename to dev/315/315-campaign/evidence/315-idprobe-readpath-refuted.md diff --git a/docs/315-campaign/evidence/315-idprobe-result.md b/dev/315/315-campaign/evidence/315-idprobe-result.md similarity index 100% rename from docs/315-campaign/evidence/315-idprobe-result.md rename to dev/315/315-campaign/evidence/315-idprobe-result.md diff --git a/docs/315-campaign/evidence/315-idprobe-was-unrunnable.md b/dev/315/315-campaign/evidence/315-idprobe-was-unrunnable.md similarity index 100% rename from docs/315-campaign/evidence/315-idprobe-was-unrunnable.md rename to dev/315/315-campaign/evidence/315-idprobe-was-unrunnable.md diff --git a/docs/315-campaign/evidence/315-if-staging-is-cold-wave64-signature.md b/dev/315/315-campaign/evidence/315-if-staging-is-cold-wave64-signature.md similarity index 100% rename from docs/315-campaign/evidence/315-if-staging-is-cold-wave64-signature.md rename to dev/315/315-campaign/evidence/315-if-staging-is-cold-wave64-signature.md diff --git a/docs/315-campaign/evidence/315-instrument-independent-summary.md b/dev/315/315-campaign/evidence/315-instrument-independent-summary.md similarity index 100% rename from docs/315-campaign/evidence/315-instrument-independent-summary.md rename to dev/315/315-campaign/evidence/315-instrument-independent-summary.md diff --git a/docs/315-campaign/evidence/315-instrumented-arm-hunt.md b/dev/315/315-campaign/evidence/315-instrumented-arm-hunt.md similarity index 100% rename from docs/315-campaign/evidence/315-instrumented-arm-hunt.md rename to dev/315/315-campaign/evidence/315-instrumented-arm-hunt.md diff --git a/docs/315-campaign/evidence/315-invalid-utf8-never-observed.md b/dev/315/315-campaign/evidence/315-invalid-utf8-never-observed.md similarity index 100% rename from docs/315-campaign/evidence/315-invalid-utf8-never-observed.md rename to dev/315/315-campaign/evidence/315-invalid-utf8-never-observed.md diff --git a/docs/315-campaign/evidence/315-layer0-tail-preregistration.md b/dev/315/315-campaign/evidence/315-layer0-tail-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-layer0-tail-preregistration.md rename to dev/315/315-campaign/evidence/315-layer0-tail-preregistration.md diff --git a/docs/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md b/dev/315/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md similarity index 100% rename from docs/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md rename to dev/315/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md diff --git a/docs/315-campaign/evidence/315-lifetime-is-the-faithfulness-rule.md b/dev/315/315-campaign/evidence/315-lifetime-is-the-faithfulness-rule.md similarity index 100% rename from docs/315-campaign/evidence/315-lifetime-is-the-faithfulness-rule.md rename to dev/315/315-campaign/evidence/315-lifetime-is-the-faithfulness-rule.md diff --git a/docs/315-campaign/evidence/315-logits-wait-FALSIFIED.md b/dev/315/315-campaign/evidence/315-logits-wait-FALSIFIED.md similarity index 100% rename from docs/315-campaign/evidence/315-logits-wait-FALSIFIED.md rename to dev/315/315-campaign/evidence/315-logits-wait-FALSIFIED.md diff --git a/docs/315-campaign/evidence/315-lout-sweep-preregistration.md b/dev/315/315-campaign/evidence/315-lout-sweep-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-lout-sweep-preregistration.md rename to dev/315/315-campaign/evidence/315-lout-sweep-preregistration.md diff --git a/docs/315-campaign/evidence/315-lout-sweep-result.md b/dev/315/315-campaign/evidence/315-lout-sweep-result.md similarity index 100% rename from docs/315-campaign/evidence/315-lout-sweep-result.md rename to dev/315/315-campaign/evidence/315-lout-sweep-result.md diff --git a/docs/315-campaign/evidence/315-mechanism-refuted-on-ids-family-and-the-precise-gap.md b/dev/315/315-campaign/evidence/315-mechanism-refuted-on-ids-family-and-the-precise-gap.md similarity index 100% rename from docs/315-campaign/evidence/315-mechanism-refuted-on-ids-family-and-the-precise-gap.md rename to dev/315/315-campaign/evidence/315-mechanism-refuted-on-ids-family-and-the-precise-gap.md diff --git a/docs/315-campaign/evidence/315-moe-id-instrument-never-observed-ids.md b/dev/315/315-campaign/evidence/315-moe-id-instrument-never-observed-ids.md similarity index 100% rename from docs/315-campaign/evidence/315-moe-id-instrument-never-observed-ids.md rename to dev/315/315-campaign/evidence/315-moe-id-instrument-never-observed-ids.md diff --git a/docs/315-campaign/evidence/315-moe-routing-routed-ffn-static.md b/dev/315/315-campaign/evidence/315-moe-routing-routed-ffn-static.md similarity index 100% rename from docs/315-campaign/evidence/315-moe-routing-routed-ffn-static.md rename to dev/315/315-campaign/evidence/315-moe-routing-routed-ffn-static.md diff --git a/docs/315-campaign/evidence/315-nan-origin-is-layer-0.md b/dev/315/315-campaign/evidence/315-nan-origin-is-layer-0.md similarity index 99% rename from docs/315-campaign/evidence/315-nan-origin-is-layer-0.md rename to dev/315/315-campaign/evidence/315-nan-origin-is-layer-0.md index 90375988f..516799ef0 100644 --- a/docs/315-campaign/evidence/315-nan-origin-is-layer-0.md +++ b/dev/315/315-campaign/evidence/315-nan-origin-is-layer-0.md @@ -154,7 +154,7 @@ The finding above does not sit quietly beside the record. It contradicts one con ## The contradiction: §40 says the corruption is born at block 41 -`docs/315-mechanism.md` §40, on the `tail:16` window: +`dev/315/315-mechanism.md` §40, on the `tail:16` window: > its window covers **block 41's FFN and block 42's attention** ... **Every earlier call's window was clean, so the > corruption is *born* at block 41, not inherited from block 0.** diff --git a/docs/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md b/dev/315/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md similarity index 98% rename from docs/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md rename to dev/315/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md index 1d771ea0a..f98071806 100644 --- a/docs/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md +++ b/dev/315/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md @@ -57,7 +57,7 @@ Two consequences follow, and the first is the important one: absence is not variable). It has to be a mechanism that is exercised by *conditions*, which is exactly what the trigger is. 2. **That is precisely the trigger's signature.** Forced eviction moves the rate from ~5 % to ~95–100 % - (`docs/315-mechanism.md`; 28/29 eviction arms vs 4/57 controls, Fisher p ≈ 1.3e-17), and the quiet cell + (`dev/315/315-mechanism.md`; 28/29 eviction arms vs 4/57 controls, Fisher p ≈ 1.3e-17), and the quiet cell occasionally faults too. `mprotect` churn drives **page reclaim and cache/TLB pressure**, so the trigger is a *pressure* generator — and a mechanism whose correctness rests on the platform's caches is exactly the kind of mechanism that fails under pressure and recovers when it eases. diff --git a/docs/315-campaign/evidence/315-no-fix-available.md b/dev/315/315-campaign/evidence/315-no-fix-available.md similarity index 100% rename from docs/315-campaign/evidence/315-no-fix-available.md rename to dev/315/315-campaign/evidence/315-no-fix-available.md diff --git a/docs/315-campaign/evidence/315-node-2948-is-an-unnamed-moe-accumulation.md b/dev/315/315-campaign/evidence/315-node-2948-is-an-unnamed-moe-accumulation.md similarity index 100% rename from docs/315-campaign/evidence/315-node-2948-is-an-unnamed-moe-accumulation.md rename to dev/315/315-campaign/evidence/315-node-2948-is-an-unnamed-moe-accumulation.md diff --git a/docs/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md b/dev/315/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md similarity index 98% rename from docs/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md rename to dev/315/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md index d28be582f..4c4fcf412 100644 --- a/docs/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md +++ b/dev/315/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md @@ -95,5 +95,5 @@ row 0 or 1, or by a downstream tensor with fewer rows corrupted than the onset o | grep -aE "ne0=[0-9]+ first=[0-9]+ row=[0-9]+" \ | awk '{ ... nan == ne0 ... }' # the filter is the whole method -`docs/315-campaign/tools/row-position-census-by-family.py` prints the per-family and per-arm cut; +`dev/315/315-campaign/tools/row-position-census-by-family.py` prints the per-family and per-arm cut; the `nan == ne0` filter is what makes the numbers above unbiased. diff --git a/docs/315-campaign/evidence/315-onset-is-a-full-row-at-layer-41-ffn-and-node-2948-confirmed.md b/dev/315/315-campaign/evidence/315-onset-is-a-full-row-at-layer-41-ffn-and-node-2948-confirmed.md similarity index 100% rename from docs/315-campaign/evidence/315-onset-is-a-full-row-at-layer-41-ffn-and-node-2948-confirmed.md rename to dev/315/315-campaign/evidence/315-onset-is-a-full-row-at-layer-41-ffn-and-node-2948-confirmed.md diff --git a/docs/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md b/dev/315/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md similarity index 100% rename from docs/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md rename to dev/315/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md diff --git a/docs/315-campaign/evidence/315-onset-layer-is-config-dependent.md b/dev/315/315-campaign/evidence/315-onset-layer-is-config-dependent.md similarity index 100% rename from docs/315-campaign/evidence/315-onset-layer-is-config-dependent.md rename to dev/315/315-campaign/evidence/315-onset-layer-is-config-dependent.md diff --git a/docs/315-campaign/evidence/315-origin-is-at-or-before-layer-4.md b/dev/315/315-campaign/evidence/315-origin-is-at-or-before-layer-4.md similarity index 100% rename from docs/315-campaign/evidence/315-origin-is-at-or-before-layer-4.md rename to dev/315/315-campaign/evidence/315-origin-is-at-or-before-layer-4.md diff --git a/docs/315-campaign/evidence/315-origin-varies-across-arms.md b/dev/315/315-campaign/evidence/315-origin-varies-across-arms.md similarity index 100% rename from docs/315-campaign/evidence/315-origin-varies-across-arms.md rename to dev/315/315-campaign/evidence/315-origin-varies-across-arms.md diff --git a/docs/315-campaign/evidence/315-origin-varies-reinstated.md b/dev/315/315-campaign/evidence/315-origin-varies-reinstated.md similarity index 100% rename from docs/315-campaign/evidence/315-origin-varies-reinstated.md rename to dev/315/315-campaign/evidence/315-origin-varies-reinstated.md diff --git a/docs/315-campaign/evidence/315-orphaned-forcers-contaminate-cells.md b/dev/315/315-campaign/evidence/315-orphaned-forcers-contaminate-cells.md similarity index 100% rename from docs/315-campaign/evidence/315-orphaned-forcers-contaminate-cells.md rename to dev/315/315-campaign/evidence/315-orphaned-forcers-contaminate-cells.md diff --git a/docs/315-campaign/evidence/315-orphaned-neighbour-defect.md b/dev/315/315-campaign/evidence/315-orphaned-neighbour-defect.md similarity index 100% rename from docs/315-campaign/evidence/315-orphaned-neighbour-defect.md rename to dev/315/315-campaign/evidence/315-orphaned-neighbour-defect.md diff --git a/docs/315-campaign/evidence/315-peer-rig-replication.md b/dev/315/315-campaign/evidence/315-peer-rig-replication.md similarity index 100% rename from docs/315-campaign/evidence/315-peer-rig-replication.md rename to dev/315/315-campaign/evidence/315-peer-rig-replication.md diff --git a/docs/315-campaign/evidence/315-persistent-mapping-correction.md b/dev/315/315-campaign/evidence/315-persistent-mapping-correction.md similarity index 100% rename from docs/315-campaign/evidence/315-persistent-mapping-correction.md rename to dev/315/315-campaign/evidence/315-persistent-mapping-correction.md diff --git a/docs/315-campaign/evidence/315-pid-capture.md b/dev/315/315-campaign/evidence/315-pid-capture.md similarity index 100% rename from docs/315-campaign/evidence/315-pid-capture.md rename to dev/315/315-campaign/evidence/315-pid-capture.md diff --git a/docs/315-campaign/evidence/315-pid-capture.patch b/dev/315/315-campaign/evidence/315-pid-capture.patch similarity index 100% rename from docs/315-campaign/evidence/315-pid-capture.patch rename to dev/315/315-campaign/evidence/315-pid-capture.patch diff --git a/docs/315-campaign/evidence/315-pooled-rig-verification.md b/dev/315/315-campaign/evidence/315-pooled-rig-verification.md similarity index 100% rename from docs/315-campaign/evidence/315-pooled-rig-verification.md rename to dev/315/315-campaign/evidence/315-pooled-rig-verification.md diff --git a/docs/315-campaign/evidence/315-producer-probe-preregistration.md b/dev/315/315-campaign/evidence/315-producer-probe-preregistration.md similarity index 98% rename from docs/315-campaign/evidence/315-producer-probe-preregistration.md rename to dev/315/315-campaign/evidence/315-producer-probe-preregistration.md index f762c6c40..b63284e3e 100644 --- a/docs/315-campaign/evidence/315-producer-probe-preregistration.md +++ b/dev/315/315-campaign/evidence/315-producer-probe-preregistration.md @@ -119,7 +119,7 @@ run: | config | recorded fault yield | |---|---| -| ambient (no instrumentation) | ~5 %/arm (`docs/315-mechanism.md`; 2/37 in `315-instrumented-arm-hunt.md`) | +| ambient (no instrumentation) | ~5 %/arm (`dev/315/315-mechanism.md`; 2/37 in `315-instrumented-arm-hunt.md`) | | 15-name `l_out` list (`315size`) | **faulted in 4 of its 9 arms**, all three cells | | `tail:16` + probes (`315nan`) | 16/17 — but did **not** reproduce in `315row` | @@ -188,7 +188,7 @@ supposed to test. **Ordering note, kept from before:** the payload test is no longer a special case at all — the payload never appears in `first_nan`, so the decisive distinction is simply `nan > 0` versus `nan == 0` given `sentinel > 0`. -The classifier is `docs/315-campaign/tools/producer-probe-readout.py`, self-tested 7/7 including the partition +The classifier is `dev/315/315-campaign/tools/producer-probe-readout.py`, self-tested 7/7 including the partition identity, so the table is exercised rather than asserted. ### Two more readout facts worth having before the arms land diff --git a/docs/315-campaign/evidence/315-queue-profile.md b/dev/315/315-campaign/evidence/315-queue-profile.md similarity index 100% rename from docs/315-campaign/evidence/315-queue-profile.md rename to dev/315/315-campaign/evidence/315-queue-profile.md diff --git a/docs/315-campaign/evidence/315-quiet-vs-forcer-by-level.md b/dev/315/315-campaign/evidence/315-quiet-vs-forcer-by-level.md similarity index 100% rename from docs/315-campaign/evidence/315-quiet-vs-forcer-by-level.md rename to dev/315/315-campaign/evidence/315-quiet-vs-forcer-by-level.md diff --git a/docs/315-campaign/evidence/315-read-twice-result-buffer-moves-under-the-reader.md b/dev/315/315-campaign/evidence/315-read-twice-result-buffer-moves-under-the-reader.md similarity index 100% rename from docs/315-campaign/evidence/315-read-twice-result-buffer-moves-under-the-reader.md rename to dev/315/315-campaign/evidence/315-read-twice-result-buffer-moves-under-the-reader.md diff --git a/docs/315-campaign/evidence/315-refuted-coherent-read.patch b/dev/315/315-campaign/evidence/315-refuted-coherent-read.patch similarity index 100% rename from docs/315-campaign/evidence/315-refuted-coherent-read.patch rename to dev/315/315-campaign/evidence/315-refuted-coherent-read.patch diff --git a/docs/315-campaign/evidence/315-regime-overshoot-oor.md b/dev/315/315-campaign/evidence/315-regime-overshoot-oor.md similarity index 100% rename from docs/315-campaign/evidence/315-regime-overshoot-oor.md rename to dev/315/315-campaign/evidence/315-regime-overshoot-oor.md diff --git a/docs/315-campaign/evidence/315-rescope-ground-truth.md b/dev/315/315-campaign/evidence/315-rescope-ground-truth.md similarity index 100% rename from docs/315-campaign/evidence/315-rescope-ground-truth.md rename to dev/315/315-campaign/evidence/315-rescope-ground-truth.md diff --git a/docs/315-campaign/evidence/315-row-index-preregistration.md b/dev/315/315-campaign/evidence/315-row-index-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-row-index-preregistration.md rename to dev/315/315-campaign/evidence/315-row-index-preregistration.md diff --git a/docs/315-campaign/evidence/315-row-p1c-inregime-mechanism.md b/dev/315/315-campaign/evidence/315-row-p1c-inregime-mechanism.md similarity index 100% rename from docs/315-campaign/evidence/315-row-p1c-inregime-mechanism.md rename to dev/315/315-campaign/evidence/315-row-p1c-inregime-mechanism.md diff --git a/docs/315-campaign/evidence/315-row-size-replication-preregistration.md b/dev/315/315-campaign/evidence/315-row-size-replication-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-row-size-replication-preregistration.md rename to dev/315/315-campaign/evidence/315-row-size-replication-preregistration.md diff --git a/docs/315-campaign/evidence/315-row-size-scored-from-archive.md b/dev/315/315-campaign/evidence/315-row-size-scored-from-archive.md similarity index 100% rename from docs/315-campaign/evidence/315-row-size-scored-from-archive.md rename to dev/315/315-campaign/evidence/315-row-size-scored-from-archive.md diff --git a/docs/315-campaign/evidence/315-row-size-test-preregistration.md b/dev/315/315-campaign/evidence/315-row-size-test-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-row-size-test-preregistration.md rename to dev/315/315-campaign/evidence/315-row-size-test-preregistration.md diff --git a/docs/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md b/dev/315/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md similarity index 100% rename from docs/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md rename to dev/315/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md diff --git a/docs/315-campaign/evidence/315-seed-is-an-all-nan-moe-router-per-token.md b/dev/315/315-campaign/evidence/315-seed-is-an-all-nan-moe-router-per-token.md similarity index 100% rename from docs/315-campaign/evidence/315-seed-is-an-all-nan-moe-router-per-token.md rename to dev/315/315-campaign/evidence/315-seed-is-an-all-nan-moe-router-per-token.md diff --git a/docs/315-campaign/evidence/315-sentinel-preregistration.md b/dev/315/315-campaign/evidence/315-sentinel-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-sentinel-preregistration.md rename to dev/315/315-campaign/evidence/315-sentinel-preregistration.md diff --git a/docs/315-campaign/evidence/315-subgraphs-are-9-nodes-so-tail-covers-all.md b/dev/315/315-campaign/evidence/315-subgraphs-are-9-nodes-so-tail-covers-all.md similarity index 100% rename from docs/315-campaign/evidence/315-subgraphs-are-9-nodes-so-tail-covers-all.md rename to dev/315/315-campaign/evidence/315-subgraphs-are-9-nodes-so-tail-covers-all.md diff --git a/docs/315-campaign/evidence/315-tail16-run-what-it-settled.md b/dev/315/315-campaign/evidence/315-tail16-run-what-it-settled.md similarity index 100% rename from docs/315-campaign/evidence/315-tail16-run-what-it-settled.md rename to dev/315/315-campaign/evidence/315-tail16-run-what-it-settled.md diff --git a/docs/315-campaign/evidence/315-task1-writer-not-stream-ordered.md b/dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md similarity index 100% rename from docs/315-campaign/evidence/315-task1-writer-not-stream-ordered.md rename to dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md diff --git a/docs/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md b/dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md similarity index 100% rename from docs/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md rename to dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md diff --git a/docs/315-campaign/evidence/315-task2-instrument-hard-negative.md b/dev/315/315-campaign/evidence/315-task2-instrument-hard-negative.md similarity index 95% rename from docs/315-campaign/evidence/315-task2-instrument-hard-negative.md rename to dev/315/315-campaign/evidence/315-task2-instrument-hard-negative.md index ea54b2566..c80d3ab0d 100644 --- a/docs/315-campaign/evidence/315-task2-instrument-hard-negative.md +++ b/dev/315/315-campaign/evidence/315-task2-instrument-hard-negative.md @@ -42,7 +42,7 @@ The gate on the record is the frozen pre-registration's `gtt_start >= 95 GiB` (`315-preregistration.md` §5, "In-regime ⟺ `gtt_start >= 95 GiB`"), with a ~110 GiB upper bound; the contract's `>= 101 GiB` is the historical value that gate was derived from. Both faulting arms are at `98G`, i.e. inside the frozen gate and **below** the contract's stricter figure - stated plainly rather than rounded up. The composition -override that had been keeping arms at 87-93 GiB is fixed (`docs/315-mechanism.md` §48), and the queued runs +override that had been keeping arms at 87-93 GiB is fixed (`dev/315/315-mechanism.md` §48), and the queued runs compose to `REGIME_LO=85`, which projects into 101-106 GiB. The exact log line, from the guard itself: @@ -55,7 +55,7 @@ The exact log line, from the guard itself: **The failing value is named by the guard's own line**: the whole LM-head output, `count=154880 of 154880`, `contiguous=yes`. What is **not** satisfied is "the producing graph `value` named", and it is not satisfied for a reason found in this round rather than for want of trying: the by-name fingerprint's per-node attribution has been -**withdrawn** (`docs/315-mechanism.md` §53). Every tensor that instrument names on the MoE path lives in the +**withdrawn** (`dev/315/315-mechanism.md` §53). Every tensor that instrument names on the MoE path lives in the 44048640-byte host-visible non-coherent arena, and `ggml_backend_tensor_get` on a host-visible tensor is a plain `memcpy` with no synchronisation, so its lines are not a snapshot of one instant. The arms prove it against themselves: @@ -65,7 +65,7 @@ call=158945 ffn_moe_weighted-4 MUL bytes=327680 nan=81880 of 81920 <- 99.9 call=158945 ffn_moe_out-4 ADD bytes=327680 (absent) <- its own sum reads finite ``` -The repair is written and gated: `docs/315-campaign/tools/patch-fp-sync.py` inserts one +The repair is written and gated: `dev/315/315-campaign/tools/patch-fp-sync.py` inserts one `ggml_backend_synchronize(backend)` before the fingerprint's read loop, and `315-fp2.sh` builds and runs it - **but only if `315gsync2` shows the fault survives a synchronise**, because a sync per graph is itself an ordering intervention and, if it removes the fault, there is nothing left to locate. diff --git a/docs/315-campaign/evidence/315-task3-dense-control-clean-at-moved-pin.md b/dev/315/315-campaign/evidence/315-task3-dense-control-clean-at-moved-pin.md similarity index 100% rename from docs/315-campaign/evidence/315-task3-dense-control-clean-at-moved-pin.md rename to dev/315/315-campaign/evidence/315-task3-dense-control-clean-at-moved-pin.md diff --git a/docs/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md b/dev/315/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md similarity index 100% rename from docs/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md rename to dev/315/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md diff --git a/docs/315-campaign/evidence/315-task9-session-state-2026-10-12.md b/dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md similarity index 99% rename from docs/315-campaign/evidence/315-task9-session-state-2026-10-12.md rename to dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md index 480eeeabc..12be57106 100644 --- a/docs/315-campaign/evidence/315-task9-session-state-2026-10-12.md +++ b/dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md @@ -19,7 +19,7 @@ limitations under the License. Written by an agent session that worked #315 while the `315cwsr70` campaign of record ran. Added as evidence so the findings survive the session; **nothing existing was modified**. -Read alongside `docs/315-ENDOGAME.md` (the runbook) and `docs/315-STATUS.md`. Where this file and +Read alongside `dev/315/315-ENDOGAME.md` (the runbook) and `dev/315/315-STATUS.md`. Where this file and those disagree, this file records what was **verified live** and names the check used. Headline results in this file: @@ -36,7 +36,7 @@ Headline results in this file: - two docs conflict: `315-cwsr-refined-hypothesis.md` still asserts the H1 confound that `315-H1-FALSIFIED.md` withdrew (and ground truth on the box supports the withdrawal) -Note: `docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` (an earlier session) is now +Note: `dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` (an earlier session) is now **stale** - `npu.lock` was verified free and acquirable in this session. --- @@ -150,7 +150,7 @@ Note: `docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` (an earlier ses # engine#315 — state at 2026-10-12 ~22:20Z (corrected) ## My earlier framing was wrong on two counts; this supersedes it -1. **H1 (fresh-boot confound) is FALSIFIED**, not unresolved. `docs/315-H1-FALSIFIED.md` +1. **H1 (fresh-boot confound) is FALSIFIED**, not unresolved. `dev/315/315-H1-FALSIFIED.md` reconstructs boot time from three independent clocks (KFD SMI `boot_ns` anchored two ways + `uptime -s`) agreeing within 4 s: run 7 ran **~6-40 min after a fresh boot** and still faulted 3/11. So a fresh boot does not suppress the fault. The confound I called "unresolved and @@ -158,7 +158,7 @@ Note: `docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` (an earlier ses 2. **H2 is the leading and only surviving candidate**: mid-wave preemption under memory pressure. H3/H4 dead (GPU-MV sizing defect proven fixed; `315-cwsr-size-excluded-PROOF.md`). -## The runbook is `docs/315-ENDOGAME.md` (read it first, not STATUS.md) +## The runbook is `dev/315/315-ENDOGAME.md` (read it first, not STATUS.md) Exact commands, decision rules, and reading keys, in order. Committed to `1bit/315-preregistration`. ## Where the campaign stands (checked 22:15, alive) @@ -177,7 +177,7 @@ Read the console as a SEQUENCE — a late fault is a different finding from an e not be averaged away. ## Then the control (criterion 1's decisive test; H2's only PID-captured faulting arm) - scp docs/315-campaign/315-cwsr-control.sh strixhalo:~/wt/ + scp dev/315/315-campaign/315-cwsr-control.sh strixhalo:~/wt/ ssh strixhalo 'cd ~/wt && TAG=315ctl1 PAIRS=20 bash 315-cwsr-control.sh' Reboots; **kills the campaign**, so run only after it finishes. Expects ~5.2 faults if baseline 26%. rate ~26% -> H1 confirmed, criterion 1 collapses, re-scope honestly @@ -300,7 +300,7 @@ Fix-cell still 54/60. ~16 arms to the bar; ~1h at current pace. ## Criterion 2 groundwork (verified 2026-10-12 22:30) — the guardrails are functional ### The lever guard WORKS (both directions tested) -`docs/315-campaign/tools/315-lever-check.py` reads the falsified-lever table live from +`dev/315/315-campaign/tools/315-lever-check.py` reads the falsified-lever table live from KNOWN-ISSUES.md (**10 rows read**) and enforces 7 named levers: memory coherence (HOST_COHERENT) section 13; 315accept 9 clean + 1 fault / 10 @@ -380,9 +380,9 @@ of the refined doc. Rule: read the **pre-fault** column, not the window total (a crashes, may be retried, and outlives its arm, so its window is simply longer). ## DOC INCONSISTENCY — a live trap for the next reader -`docs/315-cwsr-refined-hypothesis.md` (16:55) still asserts "H1 quantified: the two compared +`dev/315/315-cwsr-refined-hypothesis.md` (16:55) still asserts "H1 quantified: the two compared runs differ by ~32 hours of uptime", with a table claiming the box booted 2026-10-11 05:33. -`docs/315-H1-FALSIFIED.md` (16:51) **withdrew exactly that claim**, showing the 32.5 h figure +`dev/315/315-H1-FALSIFIED.md` (16:51) **withdrew exactly that claim**, showing the 32.5 h figure came from an uptime reading taken BEFORE an intervening reboot. **Ground truth checked live on strixhalo:** `uptime -s` = **2026-10-12 15:15:40**, box up 7h04m. @@ -1059,7 +1059,7 @@ comparison needs. criterion 2 UNMET no landed fix; both cells ran the same revision e44c9d01a criteria 4,5 not started Commits this session: 12a04f5, 7d74d5c on 1bit/315-preregistration (add-only, pushed). -Evidence file: docs/315-campaign/evidence/315-task9-session-state-2026-10-12.md +Evidence file: dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md --- @@ -1109,7 +1109,7 @@ it has already cost ~15M tokens of holding. criteria 4,5 not started commits ecf9be3, 7d74d5c, 12a04f5 on 1bit/315-preregistration (add-only, pushed) - evidence docs/315-campaign/evidence/315-task9-session-state-2026-10-12.md + evidence dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md rollback ~/work/amd_cwsr.cfg.backup (md5 a8b25738...) ### The single resume action @@ -1532,7 +1532,7 @@ carried forward unchanged - consistent with the fault still reproducing on the c ## THE STOCK-BUILD GAP IS CONFIRMED CONCRETE (2026-10-13 05:45) — and it blocks P1's and criterion 4's baseline -Section 46 of `docs/315-mechanism.md` flagged in passing that "every `control_bin` on disk is +Section 46 of `dev/315/315-mechanism.md` flagged in passing that "every `control_bin` on disk is `315instr3-build`, whose tree has 233 insertions of its own, so the control cell has always been a candidate rather than a baseline. A stock-pin control has never been measured." Checked rather than repeated, and it is worse than "one tree": **there is no uninstrumented build at the pin at all.** @@ -1574,7 +1574,7 @@ looks, but it is not nothing and it has never been measured. ### The criterion-4 baseline is CLEAN - checked, so the gap does not taint it (05:48) Chased the baseline's own build rather than assuming the gap applies to it: - docs/315-campaign/evidence/315-hrx-tests-baseline.md header: + dev/315/315-campaign/evidence/315-hrx-tests-baseline.md header: "run 2026-10-09T22:55-03:00 on strixhalo, build dir ~/wt/v315-build (fork at 26330cc20; ggml/src/ggml-hrx identical to the pin e44c9d01a)" ~/wt/v315-build/CMakeCache.txt -> ggml_SOURCE_DIR = ~/wt/llama-hybrid/ggml @@ -2210,7 +2210,7 @@ ledger's rollback path (`~/work/amd_cwsr.cfg.backup`, md5 `a8b25738...`) **does `~/work/`, not anywhere under `$HOME` (searched). The repo recorded only the PATH and the md5, never the bytes: - docs/315-campaign/evidence/315-task9-session-state-2026-10-12.md:833 + dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md:833 "The drop-in was captured verbatim to ~/work/amd_cwsr.cfg.backup before the reboot." ... but the file is absent and no copy of its content appears in the repo. @@ -2243,7 +2243,7 @@ The semantics are fully pinned by the investigation, even though the bytes are n finding. Recorded as such. ### Action taken (DONE 07:20) - 1. the content is now COMMITTED at docs/315-campaign/evidence/amd_cwsr.cfg (commit 9df9efb), so the + 1. the content is now COMMITTED at dev/315/315-campaign/evidence/amd_cwsr.cfg (commit 9df9efb), so the rollback is durable in the repo rather than only at an ephemeral path; 2. the drop-in was written back to the box and grub regenerated: /etc/default/grub.d/amd_cwsr.cfg md5 e20f05bf91d50b645c0fbc8d11998e86 @@ -2811,7 +2811,7 @@ true 0, which is exactly the failure mode that would have made this negative. started: 2026-10-13T07:20:44-03:00, composition 3 neighbours at gtt=79G cells : q (quiet) / e (evictonly 500) / c (CPU pressure), rotating per round gate : gtt_start >= 95 GiB; OOR reported, not counted - pre-reg: docs/315-campaign/evidence/315-eviction-replication-preregistration.md (commit 36f360a) + pre-reg: dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md (commit 36f360a) First arm observed: `round 1 arm q: OK nan=0 req=66 gtt_start=97G` (in-regime, clean), with the eviction cell then entering and `evictonly` confirmed ACTIVE. diff --git a/docs/315-campaign/evidence/315-thread-composition.md b/dev/315/315-campaign/evidence/315-thread-composition.md similarity index 100% rename from docs/315-campaign/evidence/315-thread-composition.md rename to dev/315/315-campaign/evidence/315-thread-composition.md diff --git a/docs/315-campaign/evidence/315-two-cell-memory-type-preregistration.md b/dev/315/315-campaign/evidence/315-two-cell-memory-type-preregistration.md similarity index 100% rename from docs/315-campaign/evidence/315-two-cell-memory-type-preregistration.md rename to dev/315/315-campaign/evidence/315-two-cell-memory-type-preregistration.md diff --git a/docs/315-campaign/evidence/315-two-gathers-bound-the-id-differently.md b/dev/315/315-campaign/evidence/315-two-gathers-bound-the-id-differently.md similarity index 100% rename from docs/315-campaign/evidence/315-two-gathers-bound-the-id-differently.md rename to dev/315/315-campaign/evidence/315-two-gathers-bound-the-id-differently.md diff --git a/docs/315-campaign/evidence/315-wb-probe-first-run-null.md b/dev/315/315-campaign/evidence/315-wb-probe-first-run-null.md similarity index 100% rename from docs/315-campaign/evidence/315-wb-probe-first-run-null.md rename to dev/315/315-campaign/evidence/315-wb-probe-first-run-null.md diff --git a/docs/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md b/dev/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md similarity index 95% rename from docs/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md rename to dev/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md index 57cd9b4a1..482745e1a 100644 --- a/docs/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md +++ b/dev/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md @@ -60,13 +60,13 @@ not a revert, not sideways. Apache-2.0`) that most of this campaign's files were written with does **not** satisfy it: ``` -missing copyright notice: docs/315-campaign/evidence/315-faceb-invariant-signature.md +missing copyright notice: dev/315/315-campaign/evidence/315-faceb-invariant-signature.md ... 46 files, including this campaign's own tools and evidence documents ``` Fixed with the repository's own tool, `python3 tools/copyright.py --fix`, which inserts the notice in the correct comment style per file — plus one genuine gap in the tool: `*.hip` had no comment style, so -`docs/315-campaign/tools/315restore-fidelity.hip` could not be fixed at all. HIP is C++, so it was added to +`dev/315/315-campaign/tools/315restore-fidelity.hip` could not be fixed at all. HIP is C++, so it was added to the existing `//` line style rather than exempted. Both `--fix` and `--check` now exit 0. ## Why this is recorded rather than quietly fixed diff --git a/docs/315-campaign/evidence/315-what-the-synchronise-drains.md b/dev/315/315-campaign/evidence/315-what-the-synchronise-drains.md similarity index 100% rename from docs/315-campaign/evidence/315-what-the-synchronise-drains.md rename to dev/315/315-campaign/evidence/315-what-the-synchronise-drains.md diff --git a/docs/315-campaign/evidence/315-which-binary-is-which.md b/dev/315/315-campaign/evidence/315-which-binary-is-which.md similarity index 100% rename from docs/315-campaign/evidence/315-which-binary-is-which.md rename to dev/315/315-campaign/evidence/315-which-binary-is-which.md diff --git a/docs/315-campaign/evidence/315-which-readings-are-faithful.md b/dev/315/315-campaign/evidence/315-which-readings-are-faithful.md similarity index 100% rename from docs/315-campaign/evidence/315-which-readings-are-faithful.md rename to dev/315/315-campaign/evidence/315-which-readings-are-faithful.md diff --git a/docs/315-campaign/evidence/315-write-side-unaddressed.md b/dev/315/315-campaign/evidence/315-write-side-unaddressed.md similarity index 100% rename from docs/315-campaign/evidence/315-write-side-unaddressed.md rename to dev/315/315-campaign/evidence/315-write-side-unaddressed.md diff --git a/docs/315-campaign/evidence/315nan-console.md b/dev/315/315-campaign/evidence/315nan-console.md similarity index 100% rename from docs/315-campaign/evidence/315nan-console.md rename to dev/315/315-campaign/evidence/315nan-console.md diff --git a/docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md b/dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md similarity index 96% rename from docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md rename to dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md index dfb035b3c..4249c4ca3 100644 --- a/docs/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md +++ b/dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md @@ -88,8 +88,8 @@ Both are destructive and affect work that is not mine. Left for the user. ## Prior art / related -- `docs/315-campaign/evidence/315-evict-trace-status.md` — the trace campaign blocked by this -- `docs/315-campaign/evidence/315-evict-trace-instrument.md` §6g — why `npu.lock` is the real +- `dev/315/315-campaign/evidence/315-evict-trace-status.md` — the trace campaign blocked by this +- `dev/315/315-campaign/evidence/315-evict-trace-instrument.md` §6g — why `npu.lock` is the real mutex (the older 315 harnesses take `box.lock`, which excludes nobody) - `~/amd-oss/315-TOOLS.md` — the box-tooling inventory diff --git a/docs/315-campaign/evidence/amd_cwsr.cfg b/dev/315/315-campaign/evidence/amd_cwsr.cfg similarity index 100% rename from docs/315-campaign/evidence/amd_cwsr.cfg rename to dev/315/315-campaign/evidence/amd_cwsr.cfg diff --git a/docs/315-campaign/patches/315-buffer-trace-diag.patch b/dev/315/315-campaign/patches/315-buffer-trace-diag.patch similarity index 100% rename from docs/315-campaign/patches/315-buffer-trace-diag.patch rename to dev/315/315-campaign/patches/315-buffer-trace-diag.patch diff --git a/docs/315-campaign/patches/315-coherence-fix.md b/dev/315/315-campaign/patches/315-coherence-fix.md similarity index 100% rename from docs/315-campaign/patches/315-coherence-fix.md rename to dev/315/315-campaign/patches/315-coherence-fix.md diff --git a/docs/315-campaign/patches/315-coherent-read-fix.patch b/dev/315/315-campaign/patches/315-coherent-read-fix.patch similarity index 100% rename from docs/315-campaign/patches/315-coherent-read-fix.patch rename to dev/315/315-campaign/patches/315-coherent-read-fix.patch diff --git a/docs/315-campaign/patches/315-download-fingerprint.md b/dev/315/315-campaign/patches/315-download-fingerprint.md similarity index 100% rename from docs/315-campaign/patches/315-download-fingerprint.md rename to dev/315/315-campaign/patches/315-download-fingerprint.md diff --git a/docs/315-campaign/patches/315-fa-masked-v-nonfinite.md b/dev/315/315-campaign/patches/315-fa-masked-v-nonfinite.md similarity index 100% rename from docs/315-campaign/patches/315-fa-masked-v-nonfinite.md rename to dev/315/315-campaign/patches/315-fa-masked-v-nonfinite.md diff --git a/docs/315-campaign/patches/315-instrumentation.md b/dev/315/315-campaign/patches/315-instrumentation.md similarity index 100% rename from docs/315-campaign/patches/315-instrumentation.md rename to dev/315/315-campaign/patches/315-instrumentation.md diff --git a/docs/315-campaign/patches/315-nan-dump.md b/dev/315/315-campaign/patches/315-nan-dump.md similarity index 100% rename from docs/315-campaign/patches/315-nan-dump.md rename to dev/315/315-campaign/patches/315-nan-dump.md diff --git a/docs/315-campaign/tools/315-evict-per-arm.py b/dev/315/315-campaign/tools/315-evict-per-arm.py similarity index 100% rename from docs/315-campaign/tools/315-evict-per-arm.py rename to dev/315/315-campaign/tools/315-evict-per-arm.py diff --git a/docs/315-campaign/tools/315-lever-check.py b/dev/315/315-campaign/tools/315-lever-check.py similarity index 98% rename from docs/315-campaign/tools/315-lever-check.py rename to dev/315/315-campaign/tools/315-lever-check.py index 41b1d2927..67f4ed4f8 100755 --- a/docs/315-campaign/tools/315-lever-check.py +++ b/dev/315/315-campaign/tools/315-lever-check.py @@ -56,7 +56,7 @@ def find_known_issues(): LEVERS = [ ("memory coherence is the carrier", r"HRX_MEMORY_TYPE_HOST_COHERENT|HOST_COHERENT", - "docs/315-mechanism.md section 13; `315accept` 9 clean + 1 fault / 10"), + "dev/315/315-mechanism.md section 13; `315accept` 9 clean + 1 fault / 10"), ("consumer-side ordering (graph sync)", r"DIAGNOSTIC_GRAPH_SYNC|GRAPH_SYNC", "section 57 / 315gsync2; 1 fault / 4 control vs 0 / 4 fix"), diff --git a/docs/315-campaign/tools/315-rate.py b/dev/315/315-campaign/tools/315-rate.py similarity index 100% rename from docs/315-campaign/tools/315-rate.py rename to dev/315/315-campaign/tools/315-rate.py diff --git a/docs/315-campaign/tools/315restore-fidelity.hip b/dev/315/315-campaign/tools/315restore-fidelity.hip similarity index 98% rename from docs/315-campaign/tools/315restore-fidelity.hip rename to dev/315/315-campaign/tools/315restore-fidelity.hip index 630ee91d0..5fc884e67 100644 --- a/docs/315-campaign/tools/315restore-fidelity.hip +++ b/dev/315/315-campaign/tools/315restore-fidelity.hip @@ -16,7 +16,7 @@ // SPDX-License-Identifier: Apache-2.0 // // engine#315 - the minimal consumer experiment the runtime proposal asks for -// (docs/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md, "Suggested next step"). +// (dev/315/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md, "Suggested next step"). // // Question: can a consumer observe different contents on a mapped host-visible range after // its OWN queue is evicted and restored, with no model, no regime and no neighbours? diff --git a/docs/315-campaign/tools/arm-pid.py b/dev/315/315-campaign/tools/arm-pid.py similarity index 100% rename from docs/315-campaign/tools/arm-pid.py rename to dev/315/315-campaign/tools/arm-pid.py diff --git a/docs/315-campaign/tools/audit-binding-access.py b/dev/315/315-campaign/tools/audit-binding-access.py similarity index 100% rename from docs/315-campaign/tools/audit-binding-access.py rename to dev/315/315-campaign/tools/audit-binding-access.py diff --git a/docs/315-campaign/tools/fa-sweep.sh b/dev/315/315-campaign/tools/fa-sweep.sh similarity index 100% rename from docs/315-campaign/tools/fa-sweep.sh rename to dev/315/315-campaign/tools/fa-sweep.sh diff --git a/docs/315-campaign/tools/fa-variant-profile.py b/dev/315/315-campaign/tools/fa-variant-profile.py similarity index 100% rename from docs/315-campaign/tools/fa-variant-profile.py rename to dev/315/315-campaign/tools/fa-variant-profile.py diff --git a/docs/315-campaign/tools/fix-dense-count.py b/dev/315/315-campaign/tools/fix-dense-count.py similarity index 96% rename from docs/315-campaign/tools/fix-dense-count.py rename to dev/315/315-campaign/tools/fix-dense-count.py index 011c319ab..fdcc371b6 100644 --- a/docs/315-campaign/tools/fix-dense-count.py +++ b/dev/315/315-campaign/tools/fix-dense-count.py @@ -17,7 +17,7 @@ # SPDX-License-Identifier: Apache-2.0 """engine#315 task 3: fix the recorded counting defect in the dense-control script before re-running it. -The record (docs/315-campaign/evidence/315-crit4-control-result.md, "Eighth harness/parse defect") records +The record (dev/315/315-campaign/evidence/315-crit4-control-result.md, "Eighth harness/parse defect") records that this script counted responses with `grep -c '"content"'`, which counts LINES. /v1/chat/completions responses are concatenated without newlines, so fifty responses sit on one line and the count read 1 - the script printed "FAULT or incomplete" when the truth was 50 clean responses. diff --git a/docs/315-campaign/tools/gen-fa-masked-nonfinite.py b/dev/315/315-campaign/tools/gen-fa-masked-nonfinite.py similarity index 100% rename from docs/315-campaign/tools/gen-fa-masked-nonfinite.py rename to dev/315/315-campaign/tools/gen-fa-masked-nonfinite.py diff --git a/docs/315-campaign/tools/gen-nan-dump.py b/dev/315/315-campaign/tools/gen-nan-dump.py similarity index 100% rename from docs/315-campaign/tools/gen-nan-dump.py rename to dev/315/315-campaign/tools/gen-nan-dump.py diff --git a/docs/315-campaign/tools/glm-mla-race.cpp b/dev/315/315-campaign/tools/glm-mla-race.cpp similarity index 100% rename from docs/315-campaign/tools/glm-mla-race.cpp rename to dev/315/315-campaign/tools/glm-mla-race.cpp diff --git a/docs/315-campaign/tools/gord-analysis.py b/dev/315/315-campaign/tools/gord-analysis.py similarity index 94% rename from docs/315-campaign/tools/gord-analysis.py rename to dev/315/315-campaign/tools/gord-analysis.py index a975d81e9..cb0eccd51 100755 --- a/docs/315-campaign/tools/gord-analysis.py +++ b/dev/315/315-campaign/tools/gord-analysis.py @@ -41,8 +41,8 @@ def summary(name, path): f"cell c OK={c['c']['OK']} FAULT={c['c']['FAULT']}") return (e_ok, e_fa) -A = summary("arm A (reorder ON )", "docs/315-campaign/evidence/.gord1a.txt") -B = summary("arm B (reorder OFF)", "docs/315-campaign/evidence/.gord1b.txt") +A = summary("arm A (reorder ON )", "dev/315/315-campaign/evidence/.gord1a.txt") +B = summary("arm B (reorder OFF)", "dev/315/315-campaign/evidence/.gord1b.txt") if A and B: a_ok, a_fa = A; b_ok, b_fa = B a_rate = a_fa/(a_ok+a_fa) if a_ok+a_fa else None diff --git a/docs/315-campaign/tools/lever-census.py b/dev/315/315-campaign/tools/lever-census.py similarity index 100% rename from docs/315-campaign/tools/lever-census.py rename to dev/315/315-campaign/tools/lever-census.py diff --git a/docs/315-campaign/tools/patch-device-producer-probe.py b/dev/315/315-campaign/tools/patch-device-producer-probe.py similarity index 99% rename from docs/315-campaign/tools/patch-device-producer-probe.py rename to dev/315/315-campaign/tools/patch-device-producer-probe.py index 4c9e3ee80..189635f1d 100644 --- a/docs/315-campaign/tools/patch-device-producer-probe.py +++ b/dev/315/315-campaign/tools/patch-device-producer-probe.py @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. """Add the DEVICE-buffer producer probe to graph-executor.cpp, per -docs/315-campaign/evidence/315-device-buffer-probe-proposal.md. +dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md. Run on the box, inside the instrumented llama tree (the default is ~/wt/llama-315probe). Every anchor is asserted to occur exactly once, so a silent no-op is impossible — the same discipline as tools/patch-*.py. diff --git a/docs/315-campaign/tools/patch-f32-read-twice-control.py b/dev/315/315-campaign/tools/patch-f32-read-twice-control.py similarity index 100% rename from docs/315-campaign/tools/patch-f32-read-twice-control.py rename to dev/315/315-campaign/tools/patch-f32-read-twice-control.py diff --git a/docs/315-campaign/tools/patch-fp-sync.py b/dev/315/315-campaign/tools/patch-fp-sync.py similarity index 98% rename from docs/315-campaign/tools/patch-fp-sync.py rename to dev/315/315-campaign/tools/patch-fp-sync.py index 6d36cfe95..9f90c4b1b 100644 --- a/docs/315-campaign/tools/patch-fp-sync.py +++ b/dev/315/315-campaign/tools/patch-fp-sync.py @@ -16,7 +16,7 @@ # Copyright 2026 bong-water-water-bong # SPDX-License-Identifier: Apache-2.0 # -# engine#315: patch 2 of the tensor fingerprint. Section 53 of docs/315-mechanism.md withdrew the by-name +# engine#315: patch 2 of the tensor fingerprint. Section 53 of dev/315/315-mechanism.md withdrew the by-name # per-node attribution because the fingerprint reads host-visible tensors through ggml_backend_tensor_get, which # for those is a plain memcpy with no synchronisation - the reads are not a snapshot of one instant. This patch # makes the reads ordered by synchronising the backend once per graph, before the read loop. diff --git a/docs/315-campaign/tools/patch-fp-wavemask.py b/dev/315/315-campaign/tools/patch-fp-wavemask.py similarity index 98% rename from docs/315-campaign/tools/patch-fp-wavemask.py rename to dev/315/315-campaign/tools/patch-fp-wavemask.py index 3e38d621a..3542c9589 100644 --- a/docs/315-campaign/tools/patch-fp-wavemask.py +++ b/dev/315/315-campaign/tools/patch-fp-wavemask.py @@ -20,7 +20,7 @@ WHY IT IS NEEDED, and why counting cannot substitute: the fingerprint reports `nan` (a count) and `first` (the index of the first NaN). Those two numbers are **structurally unable** to say whether the NaNs are one region or a wave-interleaved pattern — `first + nan <= extent` holds either way -(docs/315-campaign/evidence/315-corruption-is-wave-granular.md, second correction). A 32-bit mask over the +(dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md, second correction). A 32-bit mask over the first 2048 elements (32 waves of 64) resolves it in one line: a region is a contiguous run of set bits, a lane pattern is interleaved. diff --git a/docs/315-campaign/tools/patch-id-instrument-sync.py b/dev/315/315-campaign/tools/patch-id-instrument-sync.py similarity index 100% rename from docs/315-campaign/tools/patch-id-instrument-sync.py rename to dev/315/315-campaign/tools/patch-id-instrument-sync.py diff --git a/docs/315-campaign/tools/patch-id-instrument.py b/dev/315/315-campaign/tools/patch-id-instrument.py similarity index 100% rename from docs/315-campaign/tools/patch-id-instrument.py rename to dev/315/315-campaign/tools/patch-id-instrument.py diff --git a/docs/315-campaign/tools/patch-input-f32-sync-twin.py b/dev/315/315-campaign/tools/patch-input-f32-sync-twin.py similarity index 100% rename from docs/315-campaign/tools/patch-input-f32-sync-twin.py rename to dev/315/315-campaign/tools/patch-input-f32-sync-twin.py diff --git a/docs/315-campaign/tools/patch-staging-coherence-ab.py b/dev/315/315-campaign/tools/patch-staging-coherence-ab.py similarity index 98% rename from docs/315-campaign/tools/patch-staging-coherence-ab.py rename to dev/315/315-campaign/tools/patch-staging-coherence-ab.py index 7072f5ba9..5cc0e7da9 100644 --- a/docs/315-campaign/tools/patch-staging-coherence-ab.py +++ b/dev/315/315-campaign/tools/patch-staging-coherence-ab.py @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. """Add the A'/B' coherence toggle for the host staging download buffer, per -docs/315-campaign/evidence/315-candidate-host-staging-coherence.md. +dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md. Run on the box from the instrumented tree root (default ~/wt/llama-315probe). Anchors are asserted to occur exactly once and a pre-check runs over ALL of them before anything is written, so a partial or silent patch is diff --git a/docs/315-campaign/tools/patch-sync-between-reads.py b/dev/315/315-campaign/tools/patch-sync-between-reads.py similarity index 98% rename from docs/315-campaign/tools/patch-sync-between-reads.py rename to dev/315/315-campaign/tools/patch-sync-between-reads.py index 0160f2435..f4b991f60 100644 --- a/docs/315-campaign/tools/patch-sync-between-reads.py +++ b/dev/315/315-campaign/tools/patch-sync-between-reads.py @@ -31,7 +31,7 @@ This adds a fresh `ggml_backend_synchronize(backend)` and then a FOURTH plain read, reported as `hash_after_sync` with `after_sync_stable` = (hash_after_sync == memcpy2_hash). -PRE-REGISTERED READING (docs/315-campaign/evidence/315-PREREG-sync-between-reads.md): +PRE-REGISTERED READING (dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md): #1 != #2 and hash_after_sync == #2 -> the writer IS ordered by the stream: a fresh sync stops the movement, so the reads were racing work the sync covers. This makes GraphExecutor::execute returning without a wait the live diff --git a/docs/315-campaign/tools/patch-sync-dose.py b/dev/315/315-campaign/tools/patch-sync-dose.py similarity index 97% rename from docs/315-campaign/tools/patch-sync-dose.py rename to dev/315/315-campaign/tools/patch-sync-dose.py index 06ede79cc..1dba8c8bb 100644 --- a/docs/315-campaign/tools/patch-sync-dose.py +++ b/dev/315/315-campaign/tools/patch-sync-dose.py @@ -30,7 +30,7 @@ # GGML_HRX_DIAGNOSTIC_GRAPH_SYNC_EVERY=N -> synchronise every Nth replay (N=1 equals the existing knob, # a large N is the no-knob cell) # so the same run can carry several doses and the fault rate can be read as a function of the dose. Section 57 of -# docs/315-mechanism.md is the pre-commitment that says why this control is needed if the binary cell comes back +# dev/315/315-mechanism.md is the pre-commitment that says why this control is needed if the binary cell comes back # clean, and nothing here changes the frozen harness or any registered criterion. # # Idempotent: refuses to apply twice. diff --git a/docs/315-campaign/tools/patch-writeback-read-probe.py b/dev/315/315-campaign/tools/patch-writeback-read-probe.py similarity index 100% rename from docs/315-campaign/tools/patch-writeback-read-probe.py rename to dev/315/315-campaign/tools/patch-writeback-read-probe.py diff --git a/docs/315-campaign/tools/producer-probe-readout.py b/dev/315/315-campaign/tools/producer-probe-readout.py similarity index 98% rename from docs/315-campaign/tools/producer-probe-readout.py rename to dev/315/315-campaign/tools/producer-probe-readout.py index a14e0b5a3..8af06d741 100755 --- a/docs/315-campaign/tools/producer-probe-readout.py +++ b/dev/315/315-campaign/tools/producer-probe-readout.py @@ -15,7 +15,7 @@ # limitations under the License. """Classify the producer probe's output per the pre-registered table. -Why this exists: docs/315-campaign/evidence/315-producer-probe-preregistration.md pre-declares six verdict +Why this exists: dev/315/315-campaign/evidence/315-producer-probe-preregistration.md pre-declares six verdict branches for the `[hrx-producer]` lines, of which the decisive pair separates "the device never wrote the region" from "the device wrote a wrong value". Scoring that by eye invites the exact post-hoc reading the pre-registration exists to prevent, so the table is implemented here and the verdict is printed, not diff --git a/docs/315-campaign/tools/row-position-census-by-family.py b/dev/315/315-campaign/tools/row-position-census-by-family.py similarity index 100% rename from docs/315-campaign/tools/row-position-census-by-family.py rename to dev/315/315-campaign/tools/row-position-census-by-family.py diff --git a/docs/315-campaign/tools/row-position-census.py b/dev/315/315-campaign/tools/row-position-census.py similarity index 100% rename from docs/315-campaign/tools/row-position-census.py rename to dev/315/315-campaign/tools/row-position-census.py diff --git a/docs/315-campaign/tools/row-readout.py b/dev/315/315-campaign/tools/row-readout.py similarity index 95% rename from docs/315-campaign/tools/row-readout.py rename to dev/315/315-campaign/tools/row-readout.py index 604c1166e..15503a016 100644 --- a/docs/315-campaign/tools/row-readout.py +++ b/dev/315/315-campaign/tools/row-readout.py @@ -16,7 +16,7 @@ """Read the row probe's fingerprint lines: which token row is NaN, and is it the same row throughout? Why this exists: a NaN *count* cannot distinguish one token's row from many, and the answer decides the -mechanism (docs/315-mechanism.md section 47). The extended fingerprint prints `ne0=` (innermost extent), +mechanism (dev/315/315-mechanism.md section 47). The extended fingerprint prints `ne0=` (innermost extent), `first=` (index of the first NaN) and `row=` (first/ne0 = the token index), so the question becomes whether `row` is constant across the whole chain of named tensors and whether it equals the last row (token T-1). @@ -38,7 +38,7 @@ r" hash=(?P[0-9a-f]+)") # GGML_HRX_SENTINEL=1 fills the logits allocation with payload NaN and the guard reports the match -# count as `sentinel=`. Read it (docs/315-STATUS.md records that it went unread for two days). +# count as `sentinel=`. Read it (dev/315/315-STATUS.md records that it went unread for two days). SENT = re.compile(r"sentinel=(?P\d+)") @@ -80,7 +80,7 @@ def main(argv): m = SENT.search(r) if m: sent_matched += int(m.group(1)) - # The sentinel field was armed for days and never read out (docs/315-STATUS.md), so it is + # The sentinel field was armed for days and never read out (dev/315/315-STATUS.md), so it is # printed here beside every arm's verdict rather than left in the raw log. sentinel>0 means # the logits region still held the payload the fill wrote: the device never wrote it. sent_note = ("not armed" if sent_armed == 0 else diff --git a/docs/315-campaign/tools/scan-global-handoffs.py b/dev/315/315-campaign/tools/scan-global-handoffs.py similarity index 100% rename from docs/315-campaign/tools/scan-global-handoffs.py rename to dev/315/315-campaign/tools/scan-global-handoffs.py diff --git a/docs/315-campaign/tools/test-analyse.py b/dev/315/315-campaign/tools/test-analyse.py similarity index 98% rename from docs/315-campaign/tools/test-analyse.py rename to dev/315/315-campaign/tools/test-analyse.py index e166586a4..b25efeab9 100644 --- a/docs/315-campaign/tools/test-analyse.py +++ b/dev/315/315-campaign/tools/test-analyse.py @@ -21,7 +21,7 @@ all in-regime arms, a hardcoded zero numerator, and Face B being filed as harness truncation rather than as a fault. -Run: python3 docs/315-campaign/tools/test-analyse.py (exit 0 = all cases as expected) +Run: python3 dev/315/315-campaign/tools/test-analyse.py (exit 0 = all cases as expected) """ from __future__ import annotations diff --git a/docs/315-campaign/tools/verify-evidence.sh b/dev/315/315-campaign/tools/verify-evidence.sh similarity index 100% rename from docs/315-campaign/tools/verify-evidence.sh rename to dev/315/315-campaign/tools/verify-evidence.sh diff --git a/docs/315-cwsr-hypothesis.md b/dev/315/315-cwsr-hypothesis.md similarity index 100% rename from docs/315-cwsr-hypothesis.md rename to dev/315/315-cwsr-hypothesis.md diff --git a/docs/315-cwsr-refined-hypothesis.md b/dev/315/315-cwsr-refined-hypothesis.md similarity index 100% rename from docs/315-cwsr-refined-hypothesis.md rename to dev/315/315-cwsr-refined-hypothesis.md diff --git a/docs/315-cwsr-result.md b/dev/315/315-cwsr-result.md similarity index 100% rename from docs/315-cwsr-result.md rename to dev/315/315-cwsr-result.md diff --git a/docs/315-cwsr-size-excluded-PROOF.md b/dev/315/315-cwsr-size-excluded-PROOF.md similarity index 100% rename from docs/315-cwsr-size-excluded-PROOF.md rename to dev/315/315-cwsr-size-excluded-PROOF.md diff --git a/docs/315-cwsr-sizes-excluded.md b/dev/315/315-cwsr-sizes-excluded.md similarity index 100% rename from docs/315-cwsr-sizes-excluded.md rename to dev/315/315-cwsr-sizes-excluded.md diff --git a/docs/315-instrumentation-plan.md b/dev/315/315-instrumentation-plan.md similarity index 99% rename from docs/315-instrumentation-plan.md rename to dev/315/315-instrumentation-plan.md index cfc322fe9..22856fa41 100644 --- a/docs/315-instrumentation-plan.md +++ b/dev/315/315-instrumentation-plan.md @@ -306,7 +306,7 @@ rate, so the fix must follow from the mechanism, not from a toggle. ## Artifacts -- `docs/315-instrumentation.patch` — the env-gated diagnostic as a unified diff against +- `dev/315/315-instrumentation.patch` — the env-gated diagnostic as a unified diff against `ggml/src/ggml-hrx/runtime/command-program-executor.{cpp,h}` at `e44c9d01a4d5110cecaf45472a717f46cd1abbf9`. Verified with `git apply --check` **against the real worktree** (`~/wt/llama-315instr` on the box), not against a local copy. diff --git a/docs/315-mechanism.md b/dev/315/315-mechanism.md similarity index 99% rename from docs/315-mechanism.md rename to dev/315/315-mechanism.md index dfda86fb6..d981c1a5e 100644 --- a/docs/315-mechanism.md +++ b/dev/315/315-mechanism.md @@ -56,7 +56,7 @@ buffer family the logits come from. That is the experiment the mechanism had to - **The causal link.** Coherence controls the property; whether it removes the *fault* is a rate question at ~5 %/arm, so it needs tens of arms with the box to itself. A self-starting launcher - (`docs/315-campaign/315-when-free.sh`) waits for the machine and runs the instrumented hunt; the + (`dev/315/315-campaign/315-when-free.sh`) waits for the machine and runs the instrumented hunt; the differential itself is `GGML_HRX_USE_UNIFIED_MEMORY=1` vs unset at the same regime. - **The peer session's candidate** (AQL block-transition dose-response, `HRX_AQL_BLOCK_SIZE` 262144 vs 32768) is still in flight on the same box and must be accounted for in any final statement. @@ -138,7 +138,7 @@ and `315-differential.sh` read only nan/hsa/utf8/parse, so pair 4's Face-B fault so the predicate now also counts `http_failures`, `transport_failures` and the server-log signatures (`"code":500`, unparsed Content-only). That implements the pre-registered rule rather than relaxing it: the table above is produced by applying that definition to arms that were already on disk, and -`docs/315-campaign/315-classify.py` is the single source of truth for it. +`dev/315/315-campaign/315-classify.py` is the single source of truth for it. ## 13. The coherence hypothesis is falsified by its own campaign (tag `315accept`, 2026-10-09) @@ -1631,7 +1631,7 @@ short-prompt dispatch inventory did not reach the KV threshold where the decode- ## 39. The instrumented arm hunt: a stage gets a name, and the instrument turns out to be an inducer The nan-dump fingerprint was put in front of a faulting arm (17 scored pairs and one control before the run's own -8400 s timeout; `docs/315-campaign/evidence/315-instrumented-arm-hunt.md` is the record). Two results, in order of +8400 s timeout; `dev/315/315-campaign/evidence/315-instrumented-arm-hunt.md` is the record). Two results, in order of importance. **The instrument moved the fault rate.** 16 of 17 fix arms faulted and the single control arm faulted too - 94% @@ -1729,7 +1729,7 @@ can fake which node first carries a computed NaN. ## 42. The structural relationship between the two faces, from all 142 recorded arms -`docs/315-campaign/evidence/315-face-landscape.md` reads every arm log on the box and changes what a unifying +`dev/315/315-campaign/evidence/315-face-landscape.md` reads every arm log on the box and changes what a unifying mechanism has to look like. **The faces never co-occur: 23 Face-A arms, 7 Face-B arms, 0 arms with both, 108 fully clean.** Whatever the @@ -1837,7 +1837,7 @@ rule including `identical_distinct`, since a NaN-only or 500-only score cannot s ## 46. Face B measured, the coherence bit cleared, and what is left of the visibility family -Three results from 2026-10-10, all in `docs/315-campaign/evidence/315-face-landscape.md`. +Three results from 2026-10-10, all in `dev/315/315-campaign/evidence/315-face-landscape.md`. **Face B is now measured, not inferred.** The chain-probe session (`315chain`) faulted its first pair in both cells with a 60-name fingerprint spanning the residual stream, both attention branches, the whole FFN/MoE chain and the @@ -2057,7 +2057,7 @@ ssh strixhalo 'cd ~/wt/llama-315copy && git diff --stat f2099e9b7..HEAD -- ggml/ This is a constraint, not a verdict: the routed-FFN chain stays the site for *where* the row is lost, and the FA diff stays the only candidate for *what changed*. The cheap test that separates them is the MLA race harness -(`docs/315-campaign/tools/glm-mla-race.cpp`, already compiled on the box and never run) - seconds of GPU, no +(`dev/315/315-campaign/tools/glm-mla-race.cpp`, already compiled on the box and never run) - seconds of GPU, no regime, no neighbours. ## 53. Correction: the by-name fingerprint cannot support per-node attribution, and the arena it reads points at a better candidate diff --git a/docs/315-preregistration.md b/dev/315/315-preregistration.md similarity index 98% rename from docs/315-preregistration.md rename to dev/315/315-preregistration.md index b4d573495..f0c2eb18c 100644 --- a/docs/315-preregistration.md +++ b/dev/315/315-preregistration.md @@ -116,7 +116,7 @@ their status rather than rediscovered later. | arena knobs (transient reuse settings) | **falsified on the record** — 15 paired arms showed no demonstrable effect; 77-arm tally likewise | | logits host-staging / writeback staging as the route | **falsified as stated** (hard negative: the logits are not host-staged) — but the *deferred publish* of a host writeback is still open (H1) | | AQL block-transition handling / `HRX_AQL_BLOCK_SIZE` | **under test by a peer session** on this box: dose-response with 262144 vs 32768, pre-registered as "A should fault several-fold more than B if the fault scales with transitions per token"; a clean amplified arm is evidence against it (P(0 \| 17%) ≈ 0.02) | -| H1 — deferred writeback publish window | **this campaign** (`docs/315-instrumentation-plan.md`); `882fced1a` introduced the deferral and `e0f637bff` already fixed one publish-ordering case, both present in all faulting builds | +| H1 — deferred writeback publish window | **this campaign** (`dev/315/315-instrumentation-plan.md`); `882fced1a` introduced the deferral and `e0f637bff` already fixed one publish-ordering case, both present in all faulting builds | | H2 — producer codegen writes non-finite values | **this campaign**, distinguished from H1 by the same records | The issue's own candidate list (two 128 KiB AQL blocks per decode token, 1,399 barriers per token) maps @@ -193,8 +193,8 @@ neighbours; they had to be terminated by matching their own command lines (ports scripts must verify teardown and report the GTT afterwards — this smoke session left 118 GiB resident until it was cleaned up by hand. -The producer is **committed**, not anonymous: `docs/315-campaign/315-paired.sh` (the paired driver) and -`docs/315-campaign/315-detector.py` (the request phase). The earlier producer was never located on the +The producer is **committed**, not anonymous: `dev/315/315-campaign/315-paired.sh` (the paired driver) and +`dev/315/315-campaign/315-detector.py` (the request phase). The earlier producer was never located on the box — only its consumers were on disk — so it was re-implemented here. Both scripts pass a syntax check, and their emitted lines were tested against the **existing** analyser's regex: all four verdicts (`OK`, `FAULT`, `INVALID`, `OOR`) match, because the extra `utf8=`/`alloc=` fields are appended after diff --git a/docs/PORTING.md b/docs/PORTING.md index 7d1114fbf..55dfeda08 100644 --- a/docs/PORTING.md +++ b/docs/PORTING.md @@ -22,7 +22,7 @@ Each step below is one PR (or a short series) that builds and runs on Strix Halo | Step | Component | Source in 1bit-MONSTER | Done when | |---|---|---|---| | 1 | **The engine runs inside Lemonade.** Lemonade runs `1bit serve` as a backend: one model per process behind an OpenAI-compatible API | `tools/unified_server.cpp` | **landed** ([docs/serve.md](serve.md)): NPU and HRX pass `serve_e2e` on Strix Halo (Vulkan and ZINC did too, before RFC #213 stage 3 and the core strip removed them); `smoke_serve` runs in CI. The engine first embedded Lemonade (v11.9.0 plus local recipes); after a review[^geramyl] it is embedded *into* Lemonade instead: the vendored copy is removed, and the Lemonade recipe that runs `1bit serve` (`onebit`) lives in our fork [1bit-MONSTER/lemonade](https://github.com/1bit-MONSTER/lemonade) ([docs/lemonade.md](lemonade.md)) | -| 2 | **HRX.** llama.cpp with `GGML_HRX=ON` on AMD's tested pair (it was built with `GGML_VULKAN=ON` too until RFC #213 stage 3, which removed the engine's Vulkan and ROCm builds) | `1bit-MONSTER/llama.cpp` `1bit/hrx-vulkan-patched` (AMD's `hrx-graph-develop-v2` plus our commits) + `1bit-MONSTER/hrx-system` `1bit/main` (`ROCm/hrx-system` at AMD's pin plus our commits), pinned from `ROCm/ggml-staging-automation` | **landed** ([docs/hrx.md](hrx.md)): `1bit serve` runs GGUF checkpoints on `HRX0` (and the CPU from the same build); kept current by `bump-hrx.yml`. The decode-split flash-attention race (#123, #140) is fixed (llama.cpp `00adc2b`, #176) and the kernel is on by default again (#178). Since 2026-10-18 pinned to AMD's `e3e6a976` hrx-system line (`1bit/e3e6a976-pin-6`: our four fixes replayed plus our live-out fix for ROCm/hrx-system#1535) with llama.cpp `8d9ce2c1a` (#441); test-backend-ops 1141/1141, long-prompt needles 20/20 + 5/5 + 5/5, KLD 0 and identical output on Qwen3-0.6B and ZAYA1-8B, pp512 +4.4% / +7.8%, tg128 +2.5% / +2.8%. The hold in `config/hrx-system-hold.config` now names that pin, so later AMD hrx-system moves stay manual | +| 2 | **HRX.** llama.cpp with `GGML_HRX=ON` on AMD's tested pair (it was built with `GGML_VULKAN=ON` too until RFC #213 stage 3, which removed the engine's Vulkan and ROCm builds) | `1bit-MONSTER/llama.cpp` `1bit/hrx-vulkan-patched` (AMD's `hrx-graph-develop-v2` plus our commits) + `1bit-MONSTER/hrx-system` `1bit/main` (`ROCm/hrx-system` at AMD's pin plus our commits), pinned from `ROCm/ggml-staging-automation` | **landed** ([docs/hrx.md](hrx.md)): `1bit serve` runs GGUF checkpoints on `HRX0` (and the CPU from the same build); kept current by `bump-hrx.yml`. The decode-split flash-attention race (#123, #140) is fixed (llama.cpp `00adc2b`, #176) and the kernel is on by default again (#178). Since 2026-10-18 pinned to AMD's `e3e6a976` hrx-system line (`1bit/e3e6a976-pin-6`: our four fixes replayed plus our live-out fix for ROCm/hrx-system#1535) with llama.cpp `8d9ce2c1a` (#441; since 2026-10-19 hrx-system `82804197` and llama.cpp `00f2a68de`, #458, #461). Measured at the #441 move: test-backend-ops 1141/1141, long-prompt needles 20/20 + 5/5 + 5/5, KLD 0 and identical output on Qwen3-0.6B and ZAYA1-8B, pp512 +4.4% / +7.8%, tg128 +2.5% / +2.8%. The hold in `config/hrx-system-hold.config` now names that pin, so later AMD hrx-system moves stay manual | | 3 | **NPU engine.** Full ELFs only, and the layer kernel built from source | `engine/npu` (`npu_engine_universal.cpp`, `I8Ctx::init_elf`); ELF dispatch table on `backup/iso-build-elf-native-2026-09-22`; kernel on `bench/fastlane-16tile-corrections-2026-09-22` | **3a–3c landed; 3d verified, not merged** ([docs/npu.md](npu.md)). 3a (#8): full ELFs generated in C++, reproducing all 5,153 captured contexts. 3b/3c (#9): the lane runtime (logits bit-identical to the reference lane, 24/24 steps, 11.0 ms/token; that test, `npu_lane_e2e`, is retired: its kernels were FastFlowLM-derived and its reference is lost, and the NPU check is now the private dx add-on's from-source check, which is not in this repository), tokenizer, `1bit unified`, and Qwen3-0.6B served on the NPU (now through `1bit serve`). The XDNA driver is pinned upstream (#12); since 2026-10-12 the lane runs on our own NPU runtime, compiled in, and the engine no longer builds or links XRT. GGUF on the NPU (#179, #181, #209): `1bit serve -m --device npu` repacks a GGUF to Q4NX and runs six architectures on the per-op forward, correct but at 0.006-0.17 tok/s. 3d: both the dense layer kernel and lm-head now decode correctly from source, ELF to ELF, with no xclbin at any point and one compiled program covering every context position (no per-position rebuild). Verified on Qwen3-0.6B, 1.7B and 4B: cosine similarity against an fp64 reference ≥0.9999 at every tested position, and greedy decode matches the reference exactly end to end. Not yet merged (npu.md, "Open") | | 3x | **35B MoE on the NPU (experimental, closed source).** Qwen3.6-35B-A3B on the NPU | the private `1bit-MONSTER/npu-kernels` repository | **landed as a private add-on** ([docs/npu.md](npu.md#private-routes)): built into `1bit` with `-DONEBIT_NPU_PRIVATE`; parity against the fp64 reference passes (3 positions, argmax 846 / 198 / 3710), 16.3-16.5 tok/s decode, and `1bit serve --device npu` answers "The capital of France is Paris." Without the add-on, `serve` says the route is not part of the build | | + | **Linux kernel.** The kernel that provides `amdxdna` and `amdgpu`, pinned to upstream | `torvalds/linux` release tags; config from the Strix Halo kernel of 2026-09-23 | **moved** to its own repository, [1bit-MONSTER/kernel](https://github.com/1bit-MONSTER/kernel) (2026-10-04): v7.3-rc4 built into Debian packages with `amdxdna` in-tree, and that repository keeps it current. Installing it on Strix Halo is a separate, deliberate step; 1bit OS (`os/`) takes a built kernel (`KVER=` / `VMLINUZ=`) | diff --git a/docs/hrx.md b/docs/hrx.md index 4d26d5190..0e4fded20 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -56,6 +56,18 @@ HRX0: AMD Radeon 8060S Graphics (Node 1) (gfx1151) no longer ours: AMD carries its own (`planning/mask_write_wait`). The pin itself is a `-s ours` merge of the previous pin `3319e536` (so every commit the engine shipped stays reachable, with the tree of the tested commit `fc2213a1c`) followed by #6. Rebase these by hand when AMD's pair moves. +- **The pin is off `1bit/main`.** `.gitmodules` names `1bit/main` for hrx-system, but the pin + (`82804197`) is on `1bit/e3e6a976-pin-6` (and `1bit/315-allow-no-svm`, the same commit), not on + `1bit/main`. The two have diverged (checked 2026-10-20 with GitHub's compare API): `1bit/main` + (`c9656af5`) has 6 commits the pin does not, the merges of AMD's `40a1a36c` and `c0b135a7` + (the newer hrx-system the hold keeps out) and the engine#409, #410, #401 and #6 fixes, which the + pin line carries as its own replayed commits. So `1bit/main` cannot be fast-forwarded to the pin; + moving it there means resetting the branch (a force push), which drops those two AMD merges from + it. Until that is decided, the pin stays off-branch on purpose. What it costs: a bump PR that moves + hrx-system fails the release's on-branch check (`pins_on_branch` in `scripts/weekly.sh`), and a + shallow submodule clone that follows `1bit/main` cannot fetch the pin, so the release clones it + in full (`weekly.sh` retries the submodule update without `--depth 1`). The pin stays reachable + through the `1bit/e3e6a976-pin-6` branch, which must not be deleted. - **Where the pair comes from.** The two pins are the pair AMD's integration repo, [ROCm/ggml-staging-automation](https://github.com/ROCm/ggml-staging-automation), builds and tests together. @@ -368,8 +380,8 @@ The engine dropped its own Hadamard-rotated Q4_0 format (owner, 2026-10-03): on and less accurate than the UD-Q4_K_XL file of the same model (Qwen3.8-27B pp512 63.5 against 335 tok/s; KLD 0.029 against 0.007 vs BF16), and the lean ROCm W4A4 route it was made for left with the ROCm build. `tools/hadamard_q4_0.py` is removed, and `1bit serve` refuses a file stamped -`onebit.hadamard_q4_0` by name. The loader support in our llama.cpp (`llama-hadamard`, llama.cpp -#58) stays in the fork; it is the same code that rotates PrismML's files +`onebit.hadamard_q4_0` by name. The loader support in our llama.cpp (`llama-hadamard`, +llama.cpp #58) stays in the fork; it is the same code that rotates PrismML's files ([Ternary Bonsai](#ternary-bonsai-prismmls-hadamard-folded-ggufs)), which keep running. ### Server memory: the graph program cache cap (llama.cpp #63) diff --git a/docs/hw-gate.md b/docs/hw-gate.md index 5e621d710..82d8d437e 100644 --- a/docs/hw-gate.md +++ b/docs/hw-gate.md @@ -17,9 +17,8 @@ limitations under the License. # HRX hardware gate `ci.yml` runs on a GitHub runner with no GPU, so it cannot see an HRX kernel or pin that computes -the wrong thing. Eight HRX correctness bugs were found by hand within ten days: #308, #309, #310, -#313, #318, #319, #369 and #370. The hardware gate checks every HRX pin bump, and every PR that asks -for it, on the Strix Halo box before merge. +the wrong thing. Eight HRX correctness bugs were found by hand within ten days: #308, #309, #310, #313, #318, #319, #369 +and #370. The hardware gate checks every HRX pin bump, and every PR that asks for it, on the Strix Halo box before merge. ## When it runs diff --git a/docs/lemonade.md b/docs/lemonade.md index e8e42f3d3..271c3c564 100644 --- a/docs/lemonade.md +++ b/docs/lemonade.md @@ -72,8 +72,8 @@ LEMONADE_ONEBIT_BIN=$PWD/build/1bit ~/.cache/lemonade-pin/bin/lemond the upstream release sync below). Before merging, rebuild and rerun the LLM suite on Strix Halo. On Strix Halo the recipe passes Lemonade's own LLM test suite -(`test/server_llm.py --wrapped-server onebit`) on Vulkan and HRX: 31 tests, 9 run, -22 skipped as unsupported. It is not proposed upstream yet: the plan is to embed the engine fully +(`test/server_llm.py --wrapped-server onebit`) on HRX (the checklist below has the counts; +Vulkan passed too before RFC #213 stage 3 removed it). It is not proposed upstream yet: the plan is to embed the engine fully in the fork first, then propose it upstream. The fork follows upstream's releases: `.github/workflows/sync-lemonade-fork.yml` checks daily and merges upstream's latest release tag (never a pre-release or `main`), so each weekly engine release carries the @@ -134,7 +134,7 @@ serve: Qwen3.8-Flash-Next (`qwen4exp`) runs on HRX since llama.cpp pin `e57beb97`. In the pinned fork ([1bit-MONSTER/lemonade](https://github.com/1bit-MONSTER/lemonade), -`third_party/lemonade` at `4812a683`), the `onebit` backend has two entries, `hrx` and `npu` +`third_party/lemonade` at `c5ba797132a4`), the `onebit` backend has two entries, `hrx` and `npu` (`backend_versions.json`); its `vulkan` and `cuda` entries are gone. Lemonade offers `onebit` models only on machines with `hrx` or `npu`. Lemonade's other recipes (its `llamacpp` Vulkan, ROCm and CUDA backends, MLX and the rest) are Lemonade's own and unchanged. diff --git a/docs/npu.md b/docs/npu.md index 2ef5a734a..82541e0db 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -383,10 +383,8 @@ and that message; the add-on brings its own tests. ## The NPU runtime The engine is XRT-free: the NPU lane runs on our own NPU runtime, compiled into `1bit`. The runtime -opens `/dev/accel/accel0` and issues the `amdxdna` ioctls itself, parses the full ELFs itself, -and keeps XRT's class names (`device`, `bo`, `hw_context`, `elf`, `ext::kernel`, `run`, -`runlist`), so `npu/rt.h` (`namespace xrt = npu_rt;`) lets the engine's NPU sources read as they -did on XRT. The NPU sources include `npu/rt.h`, never a runtime header directly. +keeps XRT's class names, so `npu/rt.h` (`namespace xrt = npu_rt;`) lets the engine's NPU sources +read as they did on XRT. The NPU sources include `npu/rt.h`, never a runtime header directly. The runtime is a proprietary component and is not in this repository (see [NOTICE](../NOTICE)): the source here holds only `npu/rt.h`, the CMake option and the tests. The release binaries diff --git a/docs/pm.md b/docs/pm.md index 1fa003c56..5f0f9a60b 100644 --- a/docs/pm.md +++ b/docs/pm.md @@ -46,7 +46,7 @@ bind address, port and auth): exactly as from the built-in catalog: ```sh - curl -X POST http://127.0.0.1:8000/api/v1/pull -H 'Content-Type: application/json' -d '{ + curl -X POST http://127.0.0.1:13305/api/v1/pull -H 'Content-Type: application/json' -d '{ "model_name": "user.ZAYA1-8B-PM-1bit", "recipe": "onebit", "checkpoint": "1bit-MONSTER/ZAYA1-8B-GGUF:zaya1-8b-Q4_K_M.gguf", @@ -67,7 +67,7 @@ the PM asks for it, so pull the experts you want ahead of time. **Standalone**, against a running Lemonade: ```sh -1bit serve -m zaya1-8b-Q4_K_M.gguf --alias ZAYA1-8B-PM --pm --lemonade-url http://127.0.0.1:8000 +1bit serve -m zaya1-8b-Q4_K_M.gguf --alias ZAYA1-8B-PM --pm --lemonade-url http://127.0.0.1:13305 # optional: --pm-experts my-experts.json, --pm-spawn-url http://127.0.0.1:8791 (agent workers, below) # optional: --pm-router laya (the engine asks the Laya router which expert first; below) ``` @@ -171,7 +171,7 @@ than route by guess. ```sh 1bit serve -m zaya1-8b-Q4_K_M.gguf --alias ZAYA1-8B-PM --pm --pm-router laya \ - --laya-model ~/.local/share/1bit/laya --lemonade-url http://127.0.0.1:8000 + --laya-model ~/.local/share/1bit/laya --lemonade-url http://127.0.0.1:13305 ``` ## Starting an agent worker (`--pm-spawn-url`) diff --git a/docs/releases.md b/docs/releases.md index f344808b3..f4e67eeb8 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -18,8 +18,8 @@ limitations under the License. 1bit engine is pinned upstream: every backend it builds is a submodule at a fixed commit (`third_party/`), and the `bump-*` workflows open a PR whenever an upstream moves. The engine -runs inside Lemonade, which releases on Fridays. So the engine releases once a week, on Sunday, -rebuilt from scratch at that week's pins. +runs inside Lemonade, which releases on Fridays. So the release run is weekly, on Sunday, +rebuilt from scratch at that week's pins. The first release is not published yet. Releases are tagged `v.` (`v2026.40`), like Lemonade's `2026.39.1`; a second release in the same week gets `.1`. @@ -28,7 +28,7 @@ release in the same week gets `.1`. | file | what | |---|---| -| `lemonade-onebit--linux-x86_64.tar.zst` | Lemonade (`lemond`, `lemonade`) from our fork with the onebit recipe, and the engine in `1bit/`: `1bit` and every backend: HRX (and the CPU, from the same llama-server), ONNX (ryzenai-server), DwarfStar, the NPU lane (no XRT). No Vulkan or ROCm llama.cpp since RFC #213 stage 3 | +| `lemonade-onebit--linux-x86_64.tar.zst` | Lemonade (`lemond`, `lemonade`) from our fork with the onebit recipe, and the engine in `1bit/`: `1bit` and every backend: HRX (and the CPU, from the same llama-server), ONNX (ryzenai-server), DwarfStar, and the NPU engine's runtime (no XRT; the NPU route is in progress, [below](#the-npu-engine)). No Vulkan or ROCm llama.cpp since RFC #213 stage 3 | | `1bit-os-.img.zst`, `1bit-os-.efi` | 1bit OS, a Lemonade appliance with the engine as a bootable USB image ([os/README.md](../os/README.md)) | | `changes.json`, `SHA256SUMS` | what moved upstream since the last release, as data; checksums | @@ -52,6 +52,14 @@ The builds target Strix Halo: the llama.cpp backends are built with `GGML_NATIVE `/opt/rocm-therock`; the NPU needs the `amdxdna` kernel driver. The closed NPU add-ons ([npu.md](npu.md), "Private routes") are never in a release. +### The NPU engine + +The release is the HRX runtime with our HIP kernels and the NPU engine, all inside Lemonade. The +NPU engine is in progress: the release carries its runtime, compiled into `1bit`, and the `onebit` +recipe's `npu` backend, but the NPU layer and lm-head kernels built from source are not merged yet +([npu.md](npu.md#step-3d-the-layer-kernel-and-lm-head-built-from-source)). Until they are, NPU models +are not served from a release package; HRX serves the GGUF models. + ### Private components Two components of the release binaries are proprietary and compiled in; their source is not in @@ -72,7 +80,7 @@ pinned commit only, with the release machine's `gh` login (read access to the pr repositories). The run stops if either cannot be fetched or is not at its pinned sha. Move a pin by hand, after the NPU check or the HRX checks on Strix Halo. -The engine is XRT-free: the NPU lane runs on the compiled-in runtime, and nothing links AMD's +The engine is XRT-free: the NPU engine's runtime is compiled in, and nothing links AMD's XRT. `scripts/package-linux.sh` strips every packed program and library (`strip --strip-unneeded`), and after packing the engine tarball the package stage checks (`logs/package-checks.log`; a failure stops the release): @@ -110,6 +118,17 @@ WEEKLY_DRY_RUN=1 WEEKLY_TAG=rehearsal-private scripts/weekly.sh ~/.cache/1bit-we `WEEKLY_DRY_RUN=1` merges and publishes nothing; stages can be run one at a time (`scripts/weekly.sh ~/.cache/1bit-weekly build test`). +Before the first stage runs, `weekly.sh` checks what the machine must have (the 2026-10-19 +rehearsal found these only after the build) and stops with a message naming what is missing: + +- **bumps, build:** TheRock in `/opt/rocm-therock`, or `-DONEBIT_HRX_TOOLCHAIN=` in + `WEEKLY_CMAKE_ARGS`; `hipcc` for DwarfStar, in TheRock or `/opt/rocm`, or `ROCM_PATH`. +- **package:** `patchelf` (`scripts/package-linux.sh`); a readable kernel image for 1bit OS, + `/boot/vmlinuz-$(uname -r)` or `VMLINUZ=` (Ubuntu's `/boot/vmlinuz-*` is root-only). + The image's licences step copies from the submodules of the run's own checkout (`/src`, + checked out by the build stage), so run `os/mini/mkimage.sh` from there, not from a plain + worktree without submodules. + ### Bumps | PR | check beyond build + ctest | diff --git a/docs/serve.md b/docs/serve.md index 0e398d22a..eedd6184d 100644 --- a/docs/serve.md +++ b/docs/serve.md @@ -29,7 +29,7 @@ OpenAI client. [--mtp HEAD.gguf | --dflash DRAFT.gguf] [--mtp-max N] [--mtp-p-min P] [--mmproj MMPROJ.gguf] [--laya | --laya-model DIR] [--route-policy FILE] [--pm [--pm-experts FILE] [--lemonade-url URL]] - [--moe-slots N|auto] [--moe-subst R] [--moe-prefetch N] + [--moe-slots N|auto] [--moe-subst R] [--moe-prefetch N] # refused in this build, below [--parallel N] [--embed MODEL.gguf] [--rerank MODEL.gguf] [--npu-opt KEY=VALUE ...] @@ -208,8 +208,8 @@ The in-process NPU route has none yet. serve's own series come first. The backen | `onebit_tctl_celsius` | CPU Tctl (k10temp) | The log-derived series come from the backend's stderr, which serve passes through unchanged and -reads line by line (Linux). A series whose line never came is left out, not reported as 0. The -#390 report is read from a line that names "CPU fallback" (or "off HRX"), followed by `OP=count`, +reads line by line (Linux). A series whose line never came is left out, not reported as 0. +The #390 report is read from a line that names "CPU fallback" (or "off HRX"), followed by `OP=count`, `OP xcount` or `OP: count` pairs, or by a total alone. llama-server drops the library's INFO lines at its default verbosity, because they map to its trace level. So the report reaches serve only when it is logged as a WARN or above, as the program-cache flush line is. @@ -244,7 +244,7 @@ fell back to Vulkan (engine #271); with no Vulkan or ROCm build left: | Case | `auto` | |---|---| | an architecture our llama.cpp does not build: anything outside `gguf_architectures.hrx` in registry/architectures.json, which `tools/registry_build.py` reads from the pinned fork and the build compiles in (among them Zyphra Zamba, Zamba2, BlackMamba; Spark2.5, BailingMoeV3, HRM text, MuseGlimmer, Kimi K3, Maple, GraniteSwitch, Granite SWA, HY v4, MiniMax, Dots3 Note, PocketTTS, Qwen3-TTS) | refused on hrx and cpu, pointing at Lemonade's llamacpp backend; Zyphra is to be ported to HRX (Flash-Next's `qwen4exp` was, at pin `e57beb97`) | -| `--moe-slots` | refused: not in this build ([below](#moe-models-larger-than-memory---moe-slots)) | +| `--moe-slots` | refused: not in this build ([below](#moe-models-larger-than-memory-not-in-this-build)) | | `--parallel N` > 1 on a gated delta-net model (Qwen3.5, Qwen3.8, Qwen3-Next) | HRX0 with one slot, and a warning on stderr: HRX runs one such sequence at a time (no multi-sequence delta-net yet). `--device cpu` keeps `--parallel N` | | `--mmproj` | HRX0. **Unverified:** vision on HRX is an open RFC #213 gate, to be checked on ZAYA1-VL; if it fails there, `auto` goes back to the CPU for `--mmproj` | | `--mtp` | HRX0 (it drafts slower there than Vulkan did: its verify batches of 2-4 tokens are the open item, docs/hrx.md) | @@ -339,7 +339,8 @@ then `$ONEBIT_LLAMA_SERVER` / `$ONEBIT_DS4`, then `llama-server` / `ds4-server` `--pm` makes the served model the Project Manager ([pm.md](pm.md)): on `/v1/chat/completions` it runs a tool loop instead of a plain forward. The model answers general conversation itself and delegates a task that fits an expert's domain with `delegate(expert, task, context)`; the engine -turns that into one chat request to Lemonade (`--lemonade-url`, default `http://127.0.0.1:8000`) +turns that into one chat request to Lemonade (`--lemonade-url`; the `onebit` recipe passes Lemonade's own +address, and by hand the default is `http://127.0.0.1:8000`, not Lemonade's default port 13305, so pass it) for the expert's catalog id, Lemonade loads the expert beside the PM, and the expert's answer comes back as the tool result. The experts are listed in `config/pm-experts.json` (built in; `--pm-experts FILE` replaces it). At most `pm.max_rounds` (4) rounds; a Lemonade that is down @@ -466,16 +467,16 @@ build is gone, above.) ## Recipes (`--recipes`, `--no-recipes`) -Tuned llama-server settings that depend on the model and route, such as a DFlash drafter's -p-min, are recipes in `config/recipes.json`. Each one -carries the measurement behind it. Serve prints each recipe it applies. Anything serve sets +Tuned llama-server settings that depend on the model and route go in `config/recipes.json` as +recipes, each with the measurement behind it. The built-in set is empty for now: the recipes the +engine had were measured on builds RFC #213 stage 3 removed ([recipes.md](recipes.md#the-built-in-recipes)). Serve prints each recipe it applies. Anything serve sets itself wins over a recipe. `--recipes FILE` replaces the built-in set and `--no-recipes` turns recipes off ([recipes.md](recipes.md)). To measure a setting before making it a recipe, use [`tools/bench.py`](bench.md). -## MoE models larger than memory (`--moe-slots`) +## MoE models larger than memory: not in this build -**Not in this build.** `--moe-slots` is kept (the option, `--moe-subst`, `--moe-prefetch` and +`--moe-slots` is kept (the option, `--moe-subst`, `--moe-prefetch` and serve's code for them), but the expert streamer lives in the removed Vulkan llama.cpp pin. Until it moves to HRX together with Qwen3.8-Flash-Next, `serve` refuses `--moe-slots` with "not available in this build". What follows describes it as it ran on Vulkan0 (docs/moe-streaming.md). diff --git a/scripts/weekly.sh b/scripts/weekly.sh index ff551adae..86646e165 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -30,7 +30,7 @@ # WEEKLY_JOBS (default 4, what ~/lb/thermal-run.sh allows on Strix Halo) caps build parallelism; heavy steps wait for WEEKLY_MIN_FREE_GB (16). # Needs gh (logged in, allowed to merge on the engine, and able to read the private repositories # config/npu-runtime.config and config/gpu-kernels.config pin), TheRock in /opt/rocm-therock, and -# what the build scripts each list. +# what the build scripts each list; preflight (at the end) checks the toolchain, hipcc, patchelf and the kernel image. set -euo pipefail [ $# -ge 1 ] || { sed -n '16,33p' "$0"; exit 2; } @@ -291,6 +291,10 @@ stage_package() { local tag=${WEEKLY_TAG:-v$(date +%G.%V)} n=0 while gh release view "$tag" -R "$REPO" > /dev/null 2>&1; do n=$((n + 1)); tag=v$(date +%G.%V).$n; done echo "$tag" > "$TAG_FILE" + # the image's licences step copies from the submodules of $SRC (checked out by build), not + # from the tree this script runs from + [ -f "$SRC/third_party/lemonade/LICENSE" ] || + { say "no $SRC/third_party/lemonade/LICENSE: run the build stage first (it checks out the submodules)"; return 1; } rm -rf "$OUT" && mkdir -p "$OUT" say "package $tag: the engine (into the Lemonade + onebit package)" # the engine tree (package-linux.sh) is not published on its own: it goes into the Lemonade @@ -332,4 +336,42 @@ stage_release() { say "released $tag" } +# --- preflight ---------------------------------------------------------------------------- +# What the stages need from the machine, checked before the first one runs, so that a missing +# tool stops the run at the start rather than after an hour of build (2026-10-19 rehearsal). +preflight() { + local bad=0 s rocm d kver vmlinuz + for s in $STAGES; do + case "$s" in + bumps|build) + # the HRX build's toolchain (cmake/hrx.cmake), unless WEEKLY_CMAKE_ARGS names another + case " ${WEEKLY_CMAKE_ARGS:-} " in + *ONEBIT_HRX_TOOLCHAIN=*) ;; + *) [ -d /opt/rocm-therock ] || + { echo "preflight: no /opt/rocm-therock (the HRX toolchain): install TheRock there, or pass -DONEBIT_HRX_TOOLCHAIN= in WEEKLY_CMAKE_ARGS"; bad=1; } ;; + esac + # DwarfStar (-DONEBIT_DS4=ON) builds with hipcc, found the way scripts/build-ds4.sh finds it + rocm=${ROCM_PATH:-} + if [ -z "$rocm" ]; then + for d in /opt/rocm-therock/lib/python3*/site-packages/_rocm_sdk_devel /opt/rocm; do + [ -x "$d/bin/hipcc" ] && { rocm=$d; break; } + done + fi + [ -n "$rocm" ] && [ -x "$rocm/bin/hipcc" ] || + { echo "preflight: no ROCm with bin/hipcc for the DwarfStar build (scripts/build-ds4.sh): set ROCM_PATH"; bad=1; } ;; + package) + # scripts/package-linux.sh rewrites RPATHs with patchelf + command -v patchelf > /dev/null || + { echo "preflight: patchelf is not installed (scripts/package-linux.sh needs it): apt install patchelf"; bad=1; } + # os/mini/mkimage.sh reads the kernel image; on Ubuntu /boot/vmlinuz-* is root-only + kver=${KVER:-$(uname -r)} + vmlinuz=${VMLINUZ:-/boot/vmlinuz-$kver} + [ -r "$vmlinuz" ] || + { echo "preflight: cannot read the kernel image $vmlinuz (os/mini/mkimage.sh): set VMLINUZ to a readable copy"; bad=1; } ;; + esac + done + return $bad +} +preflight || { say "preflight failed: nothing was run"; exit 1; } + for s in $STAGES; do "stage_$s"; done From 9a6e96cfb67b42089bdc5036e5362dcf4de17aac Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Tue, 20 Oct 2026 06:52:19 -0300 Subject: [PATCH 2/2] Move the #315 notes to a private repository; drop the private kernel repo links The #315 investigation notes include hrx-system source analysis and decompile notes, so they leave the public repository (now in 1bit-MONSTER/gpu-kernels notes/315, PR gpu-kernels#5). KNOWN-ISSUES.md keeps its notes/315 references and says where they live. The kernel repository is private, so README, PORTING.md, os/README.md, NOTICE and mkimage.sh no longer link to it. Co-Authored-By: Claude Opus 5.5 --- .pi/agents/hrx315.md | 107 - 315-COORDINATION.md | 489 -- KNOWN-ISSUES.md | 52 +- NOTICE | 4 +- README.md | 2 +- dev/315/315-ENDOGAME.md | 260 - dev/315/315-H1-FALSIFIED.md | 84 - dev/315/315-STATUS.md | 1984 ------ dev/315/315-campaign/315-HANDOFF-box.md | 167 - dev/315/315-campaign/315-analyse.py | 194 - dev/315/315-campaign/315-attrib.sh | 66 - dev/315/315-campaign/315-campaign.sh | 153 - dev/315/315-campaign/315-cell-build.sh | 63 - dev/315/315-campaign/315-chain.sh | 46 - dev/315/315-campaign/315-chain2.sh | 64 - dev/315/315-campaign/315-cheap-probe.sh | 92 - dev/315/315-campaign/315-classify.py | 49 - dev/315/315-campaign/315-cwsr-control.sh | 105 - dev/315/315-campaign/315-detector.py | 147 - dev/315/315-campaign/315-differential.sh | 101 - dev/315/315-campaign/315-dose.sh | 123 - dev/315/315-campaign/315-evict-trace.sh | 236 - dev/315/315-campaign/315-faceb.sh | 68 - dev/315/315-campaign/315-fix-build.sh | 62 - dev/315/315-campaign/315-forcebarrier.sh | 128 - dev/315/315-campaign/315-fp-hunt.sh | 56 - dev/315/315-campaign/315-fp2.sh | 125 - dev/315/315-campaign/315-graphsync.sh | 71 - dev/315/315-campaign/315-idprobe.sh | 124 - dev/315/315-campaign/315-instr-build.sh | 95 - dev/315/315-campaign/315-instr-capture.sh | 126 - dev/315/315-campaign/315-janitor.sh | 52 - dev/315/315-campaign/315-mode.sh | 98 - dev/315/315-campaign/315-paired.sh | 174 - dev/315/315-campaign/315-regime.sh | 127 - dev/315/315-campaign/315-regression.sh | 152 - dev/315/315-campaign/315-rowprobe.sh | 87 - dev/315/315-campaign/315-trace-premise.sh | 85 - dev/315/315-campaign/315-wbfp-read.py | 120 - dev/315/315-campaign/315-when-free.sh | 61 - dev/315/315-campaign/EVIDENCE-INDEX.md | 120 - .../ISSUE-COMMENT-2-2026-10-10.md | 101 - .../315-campaign/ISSUE-COMMENT-2026-10-10.md | 111 - .../ISSUE-COMMENT-3-2026-10-10.md | 59 - dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md | 101 - .../ISSUE-COMMENT-5-2026-10-10.md | 153 - .../315-0of70-has-no-eviction-cell.md | 71 - ...315-1984-reconciles-the-two-populations.md | 120 - ...SM-CANDIDATE-pointer-upload-refill-race.md | 120 - ...verges-at-first-32-entry-attention-step.md | 115 - .../315-MECHANISM-kv-not-visible-to-decode.md | 147 - ...NISM-staging-handoff-not-memory-visible.md | 133 - ...15-MECHANISM-unwaited-async-logits-read.md | 144 - ...ce-revises-mechanism-to-device-ordering.md | 103 - .../315-PREREG-argsort-precondition.md | 81 - .../evidence/315-PREREG-sync-between-reads.md | 75 - ...GISTRATION-f32-readpath-and-graph-input.md | 119 - .../315-PROPOSED-RESCOPED-OBJECTIVE.md | 115 - ...nter-upload-race-replays-are-serialized.md | 92 - .../315-RESCOPE-PROPOSAL-2026-10-15.md | 99 - ...5-RETRACTION-no-fixed-32-entry-boundary.md | 90 - .../evidence/315-ab-preregistration.md | 91 - .../evidence/315-ab1-arm-A-result.md | 87 - .../evidence/315-ab1-arm-B-interim.md | 136 - .../evidence/315-ab1-cell-c-fault-drift.md | 128 - .../evidence/315-ab1-fix-path-is-coherent.md | 69 - .../evidence/315-ab1-two-drivers-incident.md | 174 - .../315-acceptance-harness-faceb-defects.md | 128 - .../315-align-frozen-preregistration.md | 121 - .../evidence/315-align-frozen-result.md | 79 - .../evidence/315-align-preregistration.md | 122 - .../evidence/315-align-run1-uninformative.md | 104 - .../evidence/315-align-run2-result.md | 139 - .../evidence/315-aql-publish-vs-reorder.md | 159 - ...argsort-nonbijection-verified-both-ends.md | 78 - .../315-campaign/evidence/315-arm-ledger.csv | 179 - .../315-campaign/evidence/315-arm-ledger.md | 78 - .../evidence/315-asan-preregistration.md | 78 - .../evidence/315-asan-wiring-verified.md | 140 - .../evidence/315-both-faces-one-trigger.md | 68 - .../evidence/315-campaign-preregistration.md | 95 - ...ate-deferred-host-writeback-publication.md | 219 - ...-candidate-deferred-writeback-FALSIFIED.md | 82 - .../315-candidate-host-staging-coherence.md | 478 -- .../evidence/315-cause-boundary-final.md | 74 - ...5-cellA-measured-noncoherent-divergence.md | 64 - ...B-divergence-independent-of-memory-type.md | 76 - .../315-chain-plus-early-preregistration.md | 70 - .../evidence/315-chaining-guard-defect.md | 68 - .../315-campaign/evidence/315-cheap-repro.md | 63 - ...coherence-candidate-falsified-by-decomp.md | 171 - ...coherence-closed-from-source-both-paths.md | 99 - .../315-corruption-is-one-row-sized.md | 82 - .../315-corruption-is-wave-granular.md | 332 - .../evidence/315-crit4-control-result.md | 170 - .../evidence/315-deferred-writeback-gap.md | 477 -- .../315-device-buffer-probe-proposal.md | 278 - .../evidence/315-device-hunt-boundary.md | 81 - .../evidence/315-differential-artifact.md | 74 - .../315-dossier-spt-claim-corrected.md | 58 - .../evidence/315-engine-ci-baseline.md | 183 - .../evidence/315-evict-trace-instrument.md | 294 - .../evidence/315-evict-trace-status.md | 757 --- .../evidence/315-eviction-hits-the-arm.md | 135 - ...15-eviction-replication-preregistration.md | 104 - .../315-exclusions-vs-falsified-levers.md | 72 - .../evidence/315-extra-env-comma-trap.md | 73 - .../315-f32-sync-twin-preregistration.md | 97 - ...n-result-divergence-on-the-faulting-arm.md | 183 - .../evidence/315-fa-masked-v-regime.md | 72 - .../evidence/315-fa-race-pin-sweep.md | 52 - .../evidence/315-face-landscape.md | 249 - .../evidence/315-faceb-amplifier.md | 104 - .../315-faceb-has-no-capable-instrument.md | 166 - .../evidence/315-faceb-invariant-signature.md | 112 - ...-two-populations-and-faces-are-disjoint.md | 115 - ...ault-is-a-per-step-hazard-after-prefill.md | 98 - ...e-token-row-and-unifies-the-granularity.md | 87 - .../evidence/315-faults-track-gtt-drift.md | 87 - .../evidence/315-ffn-inp-0-carries-nan.md | 74 - ...es-not-measure-per-node-buffer-aliasing.md | 108 - .../315-first-row-site-is-fixed-layer-41.md | 165 - .../evidence/315-fp2-ordered-fingerprint.md | 89 - .../315-fresh-arena-preregistration.md | 55 - ...15-galign-CORRECTION-my-parse-was-wrong.md | 78 - .../evidence/315-galign-label-discrepancy.md | 130 - .../evidence/315-galign-peer-result.md | 304 - .../evidence/315-glm-align-preregistration.md | 55 - .../evidence/315-glm-dispatch-inventory.md | 83 - ...glm-never-dispatches-the-changed-kernel.md | 100 - .../evidence/315-gord1-arm-A-result.md | 81 - .../evidence/315-gord1-arm-B-interim.md | 75 - .../315-campaign/evidence/315-gord1-final.md | 94 - .../evidence/315-gord1-preregistration.md | 163 - .../evidence/315-graph-reorder-lever.md | 236 - .../315-gtt-start-is-the-discriminator.md | 170 - .../evidence/315-gttstep-preregistration.md | 111 - ...n-decode-not-prefill-index0-never-fails.md | 93 - .../evidence/315-hrx-mem-pool-quarantine.md | 459 -- ...rx-re-decomp-gaps-are-the-315-functions.md | 144 - .../315-hrx-system-dossier-by-peer-agent.md | 770 --- .../evidence/315-hrx-tests-baseline.md | 200 - .../evidence/315-id-producing-kernel.md | 138 - .../evidence/315-idprobe-readpath-refuted.md | 160 - .../evidence/315-idprobe-result.md | 191 - .../evidence/315-idprobe-was-unrunnable.md | 100 - ...315-if-staging-is-cold-wave64-signature.md | 88 - .../315-instrument-independent-summary.md | 101 - .../evidence/315-instrumented-arm-hunt.md | 106 - .../315-invalid-utf8-never-observed.md | 101 - .../315-layer0-tail-preregistration.md | 72 - .../315-layer41-site-by-graph-arithmetic.md | 124 - .../315-lifetime-is-the-faithfulness-rule.md | 79 - .../evidence/315-logits-wait-FALSIFIED.md | 101 - .../315-lout-sweep-preregistration.md | 154 - .../evidence/315-lout-sweep-result.md | 128 - ...futed-on-ids-family-and-the-precise-gap.md | 105 - ...15-moe-id-instrument-never-observed-ids.md | 102 - .../315-moe-routing-routed-ffn-static.md | 168 - .../evidence/315-nan-origin-is-layer-0.md | 271 - ...nagement-so-platform-provides-coherence.md | 131 - .../evidence/315-no-fix-available.md | 133 - ...ode-2948-is-an-unnamed-moe-accumulation.md | 140 - .../315-one-row-at-the-front-of-the-buffer.md | 99 - ...at-layer-41-ffn-and-node-2948-confirmed.md | 96 - ...onset-is-set-by-read-pattern-not-probes.md | 105 - .../315-onset-layer-is-config-dependent.md | 272 - .../315-origin-is-at-or-before-layer-4.md | 109 - .../evidence/315-origin-varies-across-arms.md | 134 - .../evidence/315-origin-varies-reinstated.md | 90 - .../315-orphaned-forcers-contaminate-cells.md | 114 - .../evidence/315-orphaned-neighbour-defect.md | 53 - .../evidence/315-peer-rig-replication.md | 98 - .../315-persistent-mapping-correction.md | 88 - .../315-campaign/evidence/315-pid-capture.md | 227 - .../evidence/315-pid-capture.patch | 10 - .../evidence/315-pooled-rig-verification.md | 76 - .../315-producer-probe-preregistration.md | 256 - .../evidence/315-queue-profile.md | 132 - .../evidence/315-quiet-vs-forcer-by-level.md | 107 - ...ce-result-buffer-moves-under-the-reader.md | 100 - .../evidence/315-refuted-coherent-read.patch | 36 - .../evidence/315-regime-overshoot-oor.md | 67 - .../evidence/315-rescope-ground-truth.md | 87 - .../evidence/315-row-index-preregistration.md | 92 - .../315-row-p1c-inregime-mechanism.md | 149 - ...15-row-size-replication-preregistration.md | 101 - .../315-row-size-scored-from-archive.md | 123 - .../315-row-size-test-preregistration.md | 123 - .../315-runtime-restore-fidelity-proposal.md | 91 - ...seed-is-an-all-nan-moe-router-per-token.md | 104 - .../evidence/315-sentinel-preregistration.md | 147 - ...ubgraphs-are-9-nodes-so-tail-covers-all.md | 77 - .../315-tail16-run-what-it-settled.md | 62 - .../315-task1-writer-not-stream-ordered.md | 118 - ...nded-negative-precondition-unmeasurable.md | 79 - .../315-task2-instrument-hard-negative.md | 105 - ...-task3-dense-control-clean-at-moved-pin.md | 112 - ...15-task3-op-test-unchanged-at-moved-pin.md | 165 - .../315-task9-session-state-2026-10-12.md | 5577 ----------------- .../evidence/315-thread-composition.md | 79 - ...15-two-cell-memory-type-preregistration.md | 73 - ...15-two-gathers-bound-the-id-differently.md | 128 - .../evidence/315-wb-probe-first-run-null.md | 120 - .../315-what-the-merge-and-the-gates-found.md | 97 - .../315-what-the-synchronise-drains.md | 100 - .../evidence/315-which-binary-is-which.md | 49 - .../315-which-readings-are-faithful.md | 150 - .../evidence/315-write-side-unaddressed.md | 192 - .../315-campaign/evidence/315nan-console.md | 133 - .../evidence/NPU-LOCK-WEDGED-2026-10-12.md | 131 - dev/315/315-campaign/evidence/amd_cwsr.cfg | 19 - .../patches/315-buffer-trace-diag.patch | 19 - .../315-campaign/patches/315-coherence-fix.md | 47 - .../patches/315-coherent-read-fix.patch | 21 - .../patches/315-download-fingerprint.md | 135 - .../patches/315-fa-masked-v-nonfinite.md | 133 - .../patches/315-instrumentation.md | 224 - dev/315/315-campaign/patches/315-nan-dump.md | 84 - .../315-campaign/tools/315-evict-per-arm.py | 238 - dev/315/315-campaign/tools/315-lever-check.py | 152 - dev/315/315-campaign/tools/315-rate.py | 237 - .../tools/315restore-fidelity.hip | 139 - dev/315/315-campaign/tools/arm-pid.py | 103 - .../tools/audit-binding-access.py | 123 - dev/315/315-campaign/tools/fa-sweep.sh | 50 - .../315-campaign/tools/fa-variant-profile.py | 100 - dev/315/315-campaign/tools/fix-dense-count.py | 64 - .../tools/gen-fa-masked-nonfinite.py | 132 - dev/315/315-campaign/tools/gen-nan-dump.py | 87 - dev/315/315-campaign/tools/glm-mla-race.cpp | 138 - dev/315/315-campaign/tools/gord-analysis.py | 62 - dev/315/315-campaign/tools/lever-census.py | 54 - .../tools/patch-device-producer-probe.py | 182 - .../tools/patch-f32-read-twice-control.py | 139 - dev/315/315-campaign/tools/patch-fp-sync.py | 68 - .../315-campaign/tools/patch-fp-wavemask.py | 110 - .../tools/patch-id-instrument-sync.py | 166 - .../315-campaign/tools/patch-id-instrument.py | 176 - .../tools/patch-input-f32-sync-twin.py | 168 - .../tools/patch-staging-coherence-ab.py | 103 - .../tools/patch-sync-between-reads.py | 130 - dev/315/315-campaign/tools/patch-sync-dose.py | 87 - .../tools/patch-writeback-read-probe.py | 122 - .../tools/producer-probe-readout.py | 147 - .../tools/row-position-census-by-family.py | 104 - .../315-campaign/tools/row-position-census.py | 77 - dev/315/315-campaign/tools/row-readout.py | 126 - .../tools/scan-global-handoffs.py | 94 - dev/315/315-campaign/tools/test-analyse.py | 127 - dev/315/315-campaign/tools/verify-evidence.sh | 54 - dev/315/315-cwsr-hypothesis.md | 173 - dev/315/315-cwsr-refined-hypothesis.md | 477 -- dev/315/315-cwsr-result.md | 109 - dev/315/315-cwsr-size-excluded-PROOF.md | 98 - dev/315/315-cwsr-sizes-excluded.md | 80 - dev/315/315-instrumentation-plan.md | 373 -- dev/315/315-mechanism.md | 2358 ------- dev/315/315-preregistration.md | 439 -- docs/PORTING.md | 2 +- os/README.md | 2 +- os/mini/mkimage.sh | 2 +- 262 files changed, 33 insertions(+), 42961 deletions(-) delete mode 100644 .pi/agents/hrx315.md delete mode 100644 315-COORDINATION.md delete mode 100644 dev/315/315-ENDOGAME.md delete mode 100644 dev/315/315-H1-FALSIFIED.md delete mode 100644 dev/315/315-STATUS.md delete mode 100644 dev/315/315-campaign/315-HANDOFF-box.md delete mode 100755 dev/315/315-campaign/315-analyse.py delete mode 100755 dev/315/315-campaign/315-attrib.sh delete mode 100644 dev/315/315-campaign/315-campaign.sh delete mode 100644 dev/315/315-campaign/315-cell-build.sh delete mode 100755 dev/315/315-campaign/315-chain.sh delete mode 100755 dev/315/315-campaign/315-chain2.sh delete mode 100755 dev/315/315-campaign/315-cheap-probe.sh delete mode 100644 dev/315/315-campaign/315-classify.py delete mode 100755 dev/315/315-campaign/315-cwsr-control.sh delete mode 100755 dev/315/315-campaign/315-detector.py delete mode 100644 dev/315/315-campaign/315-differential.sh delete mode 100755 dev/315/315-campaign/315-dose.sh delete mode 100755 dev/315/315-campaign/315-evict-trace.sh delete mode 100755 dev/315/315-campaign/315-faceb.sh delete mode 100644 dev/315/315-campaign/315-fix-build.sh delete mode 100644 dev/315/315-campaign/315-forcebarrier.sh delete mode 100755 dev/315/315-campaign/315-fp-hunt.sh delete mode 100755 dev/315/315-campaign/315-fp2.sh delete mode 100755 dev/315/315-campaign/315-graphsync.sh delete mode 100755 dev/315/315-campaign/315-idprobe.sh delete mode 100755 dev/315/315-campaign/315-instr-build.sh delete mode 100755 dev/315/315-campaign/315-instr-capture.sh delete mode 100755 dev/315/315-campaign/315-janitor.sh delete mode 100755 dev/315/315-campaign/315-mode.sh delete mode 100755 dev/315/315-campaign/315-paired.sh delete mode 100755 dev/315/315-campaign/315-regime.sh delete mode 100755 dev/315/315-campaign/315-regression.sh delete mode 100755 dev/315/315-campaign/315-rowprobe.sh delete mode 100644 dev/315/315-campaign/315-trace-premise.sh delete mode 100755 dev/315/315-campaign/315-wbfp-read.py delete mode 100644 dev/315/315-campaign/315-when-free.sh delete mode 100644 dev/315/315-campaign/EVIDENCE-INDEX.md delete mode 100644 dev/315/315-campaign/ISSUE-COMMENT-2-2026-10-10.md delete mode 100644 dev/315/315-campaign/ISSUE-COMMENT-2026-10-10.md delete mode 100644 dev/315/315-campaign/ISSUE-COMMENT-3-2026-10-10.md delete mode 100644 dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md delete mode 100644 dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md delete mode 100644 dev/315/315-campaign/evidence/315-0of70-has-no-eviction-cell.md delete mode 100644 dev/315/315-campaign/evidence/315-1984-reconciles-the-two-populations.md delete mode 100644 dev/315/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md delete mode 100644 dev/315/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md delete mode 100644 dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md delete mode 100644 dev/315/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md delete mode 100644 dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md delete mode 100644 dev/315/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md delete mode 100644 dev/315/315-campaign/evidence/315-PREREG-argsort-precondition.md delete mode 100644 dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md delete mode 100644 dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md delete mode 100644 dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md delete mode 100644 dev/315/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md delete mode 100644 dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md delete mode 100644 dev/315/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md delete mode 100644 dev/315/315-campaign/evidence/315-ab-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-ab1-arm-A-result.md delete mode 100644 dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md delete mode 100644 dev/315/315-campaign/evidence/315-ab1-cell-c-fault-drift.md delete mode 100644 dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md delete mode 100644 dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md delete mode 100644 dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md delete mode 100644 dev/315/315-campaign/evidence/315-align-frozen-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-align-frozen-result.md delete mode 100644 dev/315/315-campaign/evidence/315-align-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-align-run1-uninformative.md delete mode 100644 dev/315/315-campaign/evidence/315-align-run2-result.md delete mode 100644 dev/315/315-campaign/evidence/315-aql-publish-vs-reorder.md delete mode 100644 dev/315/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md delete mode 100644 dev/315/315-campaign/evidence/315-arm-ledger.csv delete mode 100644 dev/315/315-campaign/evidence/315-arm-ledger.md delete mode 100644 dev/315/315-campaign/evidence/315-asan-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-asan-wiring-verified.md delete mode 100644 dev/315/315-campaign/evidence/315-both-faces-one-trigger.md delete mode 100644 dev/315/315-campaign/evidence/315-campaign-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md delete mode 100644 dev/315/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md delete mode 100644 dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md delete mode 100644 dev/315/315-campaign/evidence/315-cause-boundary-final.md delete mode 100644 dev/315/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md delete mode 100644 dev/315/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md delete mode 100644 dev/315/315-campaign/evidence/315-chain-plus-early-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-chaining-guard-defect.md delete mode 100644 dev/315/315-campaign/evidence/315-cheap-repro.md delete mode 100644 dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md delete mode 100644 dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md delete mode 100644 dev/315/315-campaign/evidence/315-corruption-is-one-row-sized.md delete mode 100644 dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md delete mode 100644 dev/315/315-campaign/evidence/315-crit4-control-result.md delete mode 100644 dev/315/315-campaign/evidence/315-deferred-writeback-gap.md delete mode 100644 dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md delete mode 100644 dev/315/315-campaign/evidence/315-device-hunt-boundary.md delete mode 100644 dev/315/315-campaign/evidence/315-differential-artifact.md delete mode 100644 dev/315/315-campaign/evidence/315-dossier-spt-claim-corrected.md delete mode 100644 dev/315/315-campaign/evidence/315-engine-ci-baseline.md delete mode 100644 dev/315/315-campaign/evidence/315-evict-trace-instrument.md delete mode 100644 dev/315/315-campaign/evidence/315-evict-trace-status.md delete mode 100644 dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md delete mode 100644 dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-exclusions-vs-falsified-levers.md delete mode 100644 dev/315/315-campaign/evidence/315-extra-env-comma-trap.md delete mode 100644 dev/315/315-campaign/evidence/315-f32-sync-twin-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md delete mode 100644 dev/315/315-campaign/evidence/315-fa-masked-v-regime.md delete mode 100644 dev/315/315-campaign/evidence/315-fa-race-pin-sweep.md delete mode 100644 dev/315/315-campaign/evidence/315-face-landscape.md delete mode 100644 dev/315/315-campaign/evidence/315-faceb-amplifier.md delete mode 100644 dev/315/315-campaign/evidence/315-faceb-has-no-capable-instrument.md delete mode 100644 dev/315/315-campaign/evidence/315-faceb-invariant-signature.md delete mode 100644 dev/315/315-campaign/evidence/315-faceb-is-two-populations-and-faces-are-disjoint.md delete mode 100644 dev/315/315-campaign/evidence/315-fault-is-a-per-step-hazard-after-prefill.md delete mode 100644 dev/315/315-campaign/evidence/315-fault-is-one-token-row-and-unifies-the-granularity.md delete mode 100644 dev/315/315-campaign/evidence/315-faults-track-gtt-drift.md delete mode 100644 dev/315/315-campaign/evidence/315-ffn-inp-0-carries-nan.md delete mode 100644 dev/315/315-campaign/evidence/315-fingerprint-does-not-measure-per-node-buffer-aliasing.md delete mode 100644 dev/315/315-campaign/evidence/315-first-row-site-is-fixed-layer-41.md delete mode 100644 dev/315/315-campaign/evidence/315-fp2-ordered-fingerprint.md delete mode 100644 dev/315/315-campaign/evidence/315-fresh-arena-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-galign-CORRECTION-my-parse-was-wrong.md delete mode 100644 dev/315/315-campaign/evidence/315-galign-label-discrepancy.md delete mode 100644 dev/315/315-campaign/evidence/315-galign-peer-result.md delete mode 100644 dev/315/315-campaign/evidence/315-glm-align-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-glm-dispatch-inventory.md delete mode 100644 dev/315/315-campaign/evidence/315-glm-never-dispatches-the-changed-kernel.md delete mode 100644 dev/315/315-campaign/evidence/315-gord1-arm-A-result.md delete mode 100644 dev/315/315-campaign/evidence/315-gord1-arm-B-interim.md delete mode 100644 dev/315/315-campaign/evidence/315-gord1-final.md delete mode 100644 dev/315/315-campaign/evidence/315-gord1-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-graph-reorder-lever.md delete mode 100644 dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md delete mode 100644 dev/315/315-campaign/evidence/315-gttstep-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-hazard-is-in-decode-not-prefill-index0-never-fails.md delete mode 100644 dev/315/315-campaign/evidence/315-hrx-mem-pool-quarantine.md delete mode 100644 dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md delete mode 100644 dev/315/315-campaign/evidence/315-hrx-system-dossier-by-peer-agent.md delete mode 100644 dev/315/315-campaign/evidence/315-hrx-tests-baseline.md delete mode 100644 dev/315/315-campaign/evidence/315-id-producing-kernel.md delete mode 100644 dev/315/315-campaign/evidence/315-idprobe-readpath-refuted.md delete mode 100644 dev/315/315-campaign/evidence/315-idprobe-result.md delete mode 100644 dev/315/315-campaign/evidence/315-idprobe-was-unrunnable.md delete mode 100644 dev/315/315-campaign/evidence/315-if-staging-is-cold-wave64-signature.md delete mode 100644 dev/315/315-campaign/evidence/315-instrument-independent-summary.md delete mode 100644 dev/315/315-campaign/evidence/315-instrumented-arm-hunt.md delete mode 100644 dev/315/315-campaign/evidence/315-invalid-utf8-never-observed.md delete mode 100644 dev/315/315-campaign/evidence/315-layer0-tail-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md delete mode 100644 dev/315/315-campaign/evidence/315-lifetime-is-the-faithfulness-rule.md delete mode 100644 dev/315/315-campaign/evidence/315-logits-wait-FALSIFIED.md delete mode 100644 dev/315/315-campaign/evidence/315-lout-sweep-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-lout-sweep-result.md delete mode 100644 dev/315/315-campaign/evidence/315-mechanism-refuted-on-ids-family-and-the-precise-gap.md delete mode 100644 dev/315/315-campaign/evidence/315-moe-id-instrument-never-observed-ids.md delete mode 100644 dev/315/315-campaign/evidence/315-moe-routing-routed-ffn-static.md delete mode 100644 dev/315/315-campaign/evidence/315-nan-origin-is-layer-0.md delete mode 100644 dev/315/315-campaign/evidence/315-no-cache-management-so-platform-provides-coherence.md delete mode 100644 dev/315/315-campaign/evidence/315-no-fix-available.md delete mode 100644 dev/315/315-campaign/evidence/315-node-2948-is-an-unnamed-moe-accumulation.md delete mode 100644 dev/315/315-campaign/evidence/315-one-row-at-the-front-of-the-buffer.md delete mode 100644 dev/315/315-campaign/evidence/315-onset-is-a-full-row-at-layer-41-ffn-and-node-2948-confirmed.md delete mode 100644 dev/315/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md delete mode 100644 dev/315/315-campaign/evidence/315-onset-layer-is-config-dependent.md delete mode 100644 dev/315/315-campaign/evidence/315-origin-is-at-or-before-layer-4.md delete mode 100644 dev/315/315-campaign/evidence/315-origin-varies-across-arms.md delete mode 100644 dev/315/315-campaign/evidence/315-origin-varies-reinstated.md delete mode 100644 dev/315/315-campaign/evidence/315-orphaned-forcers-contaminate-cells.md delete mode 100644 dev/315/315-campaign/evidence/315-orphaned-neighbour-defect.md delete mode 100644 dev/315/315-campaign/evidence/315-peer-rig-replication.md delete mode 100644 dev/315/315-campaign/evidence/315-persistent-mapping-correction.md delete mode 100644 dev/315/315-campaign/evidence/315-pid-capture.md delete mode 100644 dev/315/315-campaign/evidence/315-pid-capture.patch delete mode 100644 dev/315/315-campaign/evidence/315-pooled-rig-verification.md delete mode 100644 dev/315/315-campaign/evidence/315-producer-probe-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-queue-profile.md delete mode 100644 dev/315/315-campaign/evidence/315-quiet-vs-forcer-by-level.md delete mode 100644 dev/315/315-campaign/evidence/315-read-twice-result-buffer-moves-under-the-reader.md delete mode 100644 dev/315/315-campaign/evidence/315-refuted-coherent-read.patch delete mode 100644 dev/315/315-campaign/evidence/315-regime-overshoot-oor.md delete mode 100644 dev/315/315-campaign/evidence/315-rescope-ground-truth.md delete mode 100644 dev/315/315-campaign/evidence/315-row-index-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-row-p1c-inregime-mechanism.md delete mode 100644 dev/315/315-campaign/evidence/315-row-size-replication-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-row-size-scored-from-archive.md delete mode 100644 dev/315/315-campaign/evidence/315-row-size-test-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-runtime-restore-fidelity-proposal.md delete mode 100644 dev/315/315-campaign/evidence/315-seed-is-an-all-nan-moe-router-per-token.md delete mode 100644 dev/315/315-campaign/evidence/315-sentinel-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-subgraphs-are-9-nodes-so-tail-covers-all.md delete mode 100644 dev/315/315-campaign/evidence/315-tail16-run-what-it-settled.md delete mode 100644 dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md delete mode 100644 dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md delete mode 100644 dev/315/315-campaign/evidence/315-task2-instrument-hard-negative.md delete mode 100644 dev/315/315-campaign/evidence/315-task3-dense-control-clean-at-moved-pin.md delete mode 100644 dev/315/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md delete mode 100644 dev/315/315-campaign/evidence/315-task9-session-state-2026-10-12.md delete mode 100644 dev/315/315-campaign/evidence/315-thread-composition.md delete mode 100644 dev/315/315-campaign/evidence/315-two-cell-memory-type-preregistration.md delete mode 100644 dev/315/315-campaign/evidence/315-two-gathers-bound-the-id-differently.md delete mode 100644 dev/315/315-campaign/evidence/315-wb-probe-first-run-null.md delete mode 100644 dev/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md delete mode 100644 dev/315/315-campaign/evidence/315-what-the-synchronise-drains.md delete mode 100644 dev/315/315-campaign/evidence/315-which-binary-is-which.md delete mode 100644 dev/315/315-campaign/evidence/315-which-readings-are-faithful.md delete mode 100644 dev/315/315-campaign/evidence/315-write-side-unaddressed.md delete mode 100644 dev/315/315-campaign/evidence/315nan-console.md delete mode 100644 dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md delete mode 100644 dev/315/315-campaign/evidence/amd_cwsr.cfg delete mode 100644 dev/315/315-campaign/patches/315-buffer-trace-diag.patch delete mode 100644 dev/315/315-campaign/patches/315-coherence-fix.md delete mode 100644 dev/315/315-campaign/patches/315-coherent-read-fix.patch delete mode 100644 dev/315/315-campaign/patches/315-download-fingerprint.md delete mode 100644 dev/315/315-campaign/patches/315-fa-masked-v-nonfinite.md delete mode 100644 dev/315/315-campaign/patches/315-instrumentation.md delete mode 100644 dev/315/315-campaign/patches/315-nan-dump.md delete mode 100755 dev/315/315-campaign/tools/315-evict-per-arm.py delete mode 100755 dev/315/315-campaign/tools/315-lever-check.py delete mode 100755 dev/315/315-campaign/tools/315-rate.py delete mode 100644 dev/315/315-campaign/tools/315restore-fidelity.hip delete mode 100755 dev/315/315-campaign/tools/arm-pid.py delete mode 100755 dev/315/315-campaign/tools/audit-binding-access.py delete mode 100755 dev/315/315-campaign/tools/fa-sweep.sh delete mode 100755 dev/315/315-campaign/tools/fa-variant-profile.py delete mode 100644 dev/315/315-campaign/tools/fix-dense-count.py delete mode 100644 dev/315/315-campaign/tools/gen-fa-masked-nonfinite.py delete mode 100644 dev/315/315-campaign/tools/gen-nan-dump.py delete mode 100644 dev/315/315-campaign/tools/glm-mla-race.cpp delete mode 100755 dev/315/315-campaign/tools/gord-analysis.py delete mode 100644 dev/315/315-campaign/tools/lever-census.py delete mode 100644 dev/315/315-campaign/tools/patch-device-producer-probe.py delete mode 100644 dev/315/315-campaign/tools/patch-f32-read-twice-control.py delete mode 100644 dev/315/315-campaign/tools/patch-fp-sync.py delete mode 100644 dev/315/315-campaign/tools/patch-fp-wavemask.py delete mode 100644 dev/315/315-campaign/tools/patch-id-instrument-sync.py delete mode 100644 dev/315/315-campaign/tools/patch-id-instrument.py delete mode 100644 dev/315/315-campaign/tools/patch-input-f32-sync-twin.py delete mode 100644 dev/315/315-campaign/tools/patch-staging-coherence-ab.py delete mode 100644 dev/315/315-campaign/tools/patch-sync-between-reads.py delete mode 100644 dev/315/315-campaign/tools/patch-sync-dose.py delete mode 100644 dev/315/315-campaign/tools/patch-writeback-read-probe.py delete mode 100755 dev/315/315-campaign/tools/producer-probe-readout.py delete mode 100644 dev/315/315-campaign/tools/row-position-census-by-family.py delete mode 100644 dev/315/315-campaign/tools/row-position-census.py delete mode 100644 dev/315/315-campaign/tools/row-readout.py delete mode 100644 dev/315/315-campaign/tools/scan-global-handoffs.py delete mode 100644 dev/315/315-campaign/tools/test-analyse.py delete mode 100755 dev/315/315-campaign/tools/verify-evidence.sh delete mode 100644 dev/315/315-cwsr-hypothesis.md delete mode 100644 dev/315/315-cwsr-refined-hypothesis.md delete mode 100644 dev/315/315-cwsr-result.md delete mode 100644 dev/315/315-cwsr-size-excluded-PROOF.md delete mode 100644 dev/315/315-cwsr-sizes-excluded.md delete mode 100644 dev/315/315-instrumentation-plan.md delete mode 100644 dev/315/315-mechanism.md delete mode 100644 dev/315/315-preregistration.md diff --git a/.pi/agents/hrx315.md b/.pi/agents/hrx315.md deleted file mode 100644 index b57c5accc..000000000 --- a/.pi/agents/hrx315.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -name: hrx315 -description: Read-only investigator for engine#315 and any HRX / GLM-4.7-Flash / libhrx / strixhalo-box question, including GPUs faults, NaN logits, arena or buffer reuse, and the ~/wt repro harness; use instead of scout or reviewer when the task is #315-flavoured or will touch the strixhalo box -tools: read,grep,find,ls,bash -allowed_subagents: reviewer,qa ---- - - -# engine#315 — read-only investigator - -You work on `1bit-MONSTER/engine` issue #315: **intermittent in-regime HRX GLM-4.7-Flash faults**. Face A is -NaN logits / `HSA_STATUS_ERROR_MEMORY_FAULT`; Face B is non-JSON ("Content-only") output answered with HTTP -500. The work lives in the worktree `~/wt315-prereg` (branch `1bit/315-preregistration`, PR #414) and runs on -the **strixhalo** box over `ssh -o BatchMode=yes`. **Read `~/wt/315-HANDOFF.md` on that box and -`dev/315/315-STATUS.md` before doing anything**, plus `315-COORDINATION.md`, which is the inter-session channel. - -## The standing state — do not re-derive it, and do not contradict it without new evidence - -**The issue is a BOUNDED NEGATIVE. No mechanism is named, and you must not report one.** At #315's own request -the objective accepts a bounded negative: every surviving candidate eliminated by its own pre-registered -experiment, the survivors stated, and one measured fact handed forward. - -Eliminated, each by a pre-registered experiment on the record — do **not** re-propose any of these unless you -have new evidence, and say explicitly that you are re-opening a closed family if you do: - -* **deferred host writeback** — 626 reads at the engine's logits read, 0 pending; -* **persistent mapping / coherence** — two-cell A/B at two measured addresses (13.7 % / 14.8 %), and closed - from source for both host paths; -* **the unwaited async logits read** — its own pre-registered campaign's first fix arm faulted; -* **the stream-ordering explanation** — of 82 records where `result_norm` differed between two consecutive - plain reads, a fresh `ggml_backend_synchronize` stopped it in **0** and changed the value itself in **82**. - -**Surviving family, stated as surviving:** an open writer of the `result_norm` slot that is **not** ordered by -`context->stream` (the reused-arena-slot family), plus §51's rank-sort precondition, which is **unmeasured** and -not measurable by the by-name fingerprint. - -**Handed forward:** `result_norm` changes between two consecutive identical plain reads in a faulting arm, at -two different addresses, with **no pending host writeback** at the engine's logits read and a fresh synchronise -failing to stop it. **That is the thing to explain.** - -## Rules you must not break - -1. **Clear every strixhalo run with the owner first.** The standing instruction is to ask before running - anything on that box. Do not launch a run on your own initiative. -2. **Pre-register before running** — shape set, memory regime, arm count, both-face detectors, stopping rule. - No post-hoc criteria, no arm pruning. -3. **Both faces are first-class.** An arm truncated as a *consequence* of the defect (a 500 the model caused) - is a **fault**; an arm that dies for harness reasons is **invalid, never a pass**. -4. **Diagnostic, eviction-forced cells are labelled** and none of their arms may count toward a campaign. -5. **Pins move forward only.** A candidate that is an ancestor of the shipped pin is a revert, not a fix. If - you move `third_party/llama.cpp`, move `registry/architectures.json` with it — `check_pins.py` refuses a - half-move, and a past session landed one. -6. **Never weaken a guard or an instrument** so that failure evidence survives. Correcting a defect is not the - same as removing a check. -7. Read-only: you may inspect and run **read-only** commands, but do not edit the repository. Hand edits to a - `worker`. - -## The four traps — a count is not a verdict - -Four readings have been lost in one session to a single shape: **a count read as a verdict, without checking -the count can be non-zero for the reason the verdict assumes.** Every corrupted verdict pointed at a pass not -earned or a fault that did not happen. Assume any harness you are shown has these until you check it: - -1. **A skip is not a pass.** The eight `test-hrx-*` binaries **exit 0** while printing `no HRX0 device, skipped` - unless `LD_LIBRARY_PATH` and `IREE_HAL_AMDGPU_LIBHSA_PATH` are set. Count `skip` separately, always. -2. **`grep -c … || echo 0` yields `"0\n0"`** — `grep -c` prints `0` *and* exits 1, so the fallback also runs; - `[ "$N" -eq 0 ]` then errors to the *else* branch. This made a clean run print `FAULT or incomplete`. -3. **Count occurrences, not lines.** `/v1/chat/completions` responses are concatenated without newlines, so - `grep -c '"content"'` reads 50 responses as **1**. Use `grep -o … | wc -l`. -4. **`pgrep -f ""` matches your own monitoring one-liner**, so a wait-loop for "nothing running" never - fires. Read `/proc//cmdline` and skip `bash -c` wrappers. A CI watcher with this flaw printed GREEN on a - branch whose checks had not started. - -## Box and build facts that are easy to get wrong - -* Build identification: `~/wt/315logwait-build` is the **current** build (source pin **`47a8fe929`**, eight - `test-hrx-*` binaries); `~/wt/315fp-build` and `~/wt/315probe-build` are **old-pin** (`e44c9d01a`); a result - measured on an old pin **does not describe what ships**. -* `BUILD_TESTING=ON` alone does **not** build the tests — the real gate is `-DLLAMA_BUILD_TESTS=ON`. -* Fingerprint selectors: **`EXTRA_ENV` splits on commas**, so only the first name survives; use - `tail:` / `head:` for several. `tail:16` covers `attn_norm-*` but **not** the `ffn_moe_*` family. -* A name selector that matches **nothing** looks exactly like "zero NaN". The census has 946 distinct node - names and contains **none** of `ffn_moe_probs-*`, `ffn_moe_topk-*`, `ffn_moe_argsort-*`, `ffn_moe_weights-*`. - Confirm your selector matched **before** reading its zeros. -* Take the box lock (`flock` on `~/.cache/lax-decode/box.lock`) and leave the box clean: lock free, no owned - servers, no forcer processes. - -## How to report - -Report what you **measured**, tied to a re-runnable command at a named revision, with evidence under -`dev/315/315-campaign/evidence/`. Separate **measured** from **inferred** every time. If a reading is -uninformative, say so — an absent record is not a zero. If you did not check something, say that too. diff --git a/315-COORDINATION.md b/315-COORDINATION.md deleted file mode 100644 index 858dc612d..000000000 --- a/315-COORDINATION.md +++ /dev/null @@ -1,489 +0,0 @@ - - ---- - -## 2026-10-13 22:50 — to the session running `315size` (from the session that filed the archive measurements) - -**One warning about the in-flight selector, and one reliable alternative.** - -The `315size` selector watches `l_out-*` names. On the archived `315fbar` cold-cell arms (tail:16 + probes, -paired rig) two `l_out` reads returned **byte-identical, largely-finite content at different layer names** — -`l_out-42 ≡ l_out-44` and `l_out-43 ≡ l_out-45`, both at `nan=19968 of 79872` (25 % NaN, so not saturation). -That is arena slot reuse showing up as an aliased read, and it means **an `l_out-N` reading is not always a -measurement of layer N**. - -Scope, so this is not over-read: the coincidence is evidence of aliasing **only when the reads are -unsaturated**. In `315lout4-r1e` a stride-3 family (`l_out-15 18 21 24 27 30 33 36 39 42 45`) shares one hash, -but every member reads `nan=200704 of 200704` — fully NaN — so that family is explained by saturation and is -**not** aliasing. The only sound instances I found are the stride-2 pairs above, and the stride is not -established to be universal. - -**Practical rule for `315size`:** treat any two readings that hash identically as **one** reading, and say so -in the readout rather than counting them as two layers. - -**The alternative that does not alias:** the unnamed `node_NNNN` names. Across the same arms the only coincident -pair is `node_3304 ≡ node_3305`, and **`node_2948` never coincides with anything**. `node_2948` = layer 41's -unnamed post-attention residual ADD (`71×41+37`; the graph is 71 nodes/layer with one unnamed ADD at +37 and -one unnamed `FLASH_ATTN_EXT` at +3 per layer, 47 of each). It reads **`nan=2048` exactly in 18 of 19 archived -arms**, across nine sequence lengths — one `n_embd`-wide row = 8192 bytes. If the size question is what -`315size` is for, that measurement already exists on the record and does not depend on the `l_out` reads. - -**Status of `315row` from the driver side:** I monitored it; it ended with **round 1 arm q OK (97G) and round 1 -arm e OK (98G)**, in-regime with `side=1/1`, then was superseded by the amendment. Two clean arms, no Face-A. -Consistent with `0ed67c2`'s finding that the `tail:16` + probes yield did not reproduce. - -Evidence: `dev/315/315-campaign/evidence/315-row-size-scored-from-archive.md`, -`315-layer41-site-by-graph-arithmetic.md`. - ---- - -## 2026-10-14 03:55 — the onset layer is set by the READ PATTERN; and `315lout4` did have probes - -**1. A correction to `0ed67c2`'s table, because it drove the config switch.** That commit describes -`315lout4` as *"disc rig, 16 `l_out` names, **no probes**"* and infers the configs that yield Face A are the -name list *without* the probes. The console says otherwise: - - extra_env='GGML_HRX_COPY_PROBE=1 GGML_HRX_SENTINEL=1 GGML_HRX_LOGITS_FP=100000 - GGML_HRX_TENSOR_FP=l_out-0,l_out-3,...,l_out-47 GGML_HRX_TENSOR_FP_NANONLY=1' - -All three probes were present. The amended selector (15 `l_out` names, probes retained) is therefore very -close to `315lout4` — the config that already worked — so the switch is probably right in outcome, but its -stated reason is not what the logs show, and the inference that the probes suppress the yield is unsupported. - -**2. The onset layer is a property of the read pattern, not of the probes.** Three conditions already exist in -the archive, and they separate the two variables: - -| condition | selector | probes | onset | -|---|---|---|---| -| A — `315size` | 15-name `l_out` list | none | **layer 0** (`l_out-0 nan=1984`, `1790`) | -| B — `315lout4`, `315inp2` | `l_out` name list | **yes** | **layer 0** (`l_out-0 nan=1984`, `nan=46`) | -| C — `315nan`, `315fbar` | **`tail:16`** (all ~9 nodes/chunk) | yes | **layer 41** (`node_2948 nan=2048`, 18/19) | - -A vs B holds the selector fixed and changes the probes: onset does not move. B vs C holds the probes fixed and -changes the selector: onset moves 0 → 41. **So the read pattern is the cause**, which also explains why -`tail:16`'s yield would not reproduce: the instrument that reads every node is a different regime from one that -reads three. - -`hrx-producer` in the three `315dump-*` arms shows `graph n_nodes=9`, so `tail:16` covers **every** node of a -subgraph — ~9 reads per chunk, against ~1–3 for a name list. - -**3. What this implies for the goal.** Neither layer 0 nor layer 41 is *the* site; both are where one -instrument first sees NaN. Any "the corruption is at layer N" claim is instrument-relative unless the read -pattern is fixed. The instrument-independent quantity appears to be the **size** — one `n_embd` row, 2048 -floats, `finite + nan = extent` exactly — and that should be re-measured under condition A before being called -universal. - -**4. `315cfg` (mine, running).** It tests A′ vs B, which the table above already answers; it is left to finish -only as a formal pre-registered check. Its interesting half is `A vs C`, which the archive has supplied. If it -is still thermally blocked when you read this, stopping it loses little — and note it holds `box.lock`, so it -is the one thing blocking the box right now. - -Full evidence: `dev/315/315-campaign/evidence/315-onset-is-set-by-read-pattern-not-probes.md`. - ---- - -## 2026-10-14 05:39 — `315prod` is QUEUED and holds `box.lock`; the box is load-blocked by foreign work - -`315prod` (the producer-side sentinel/payload test, pre-registered in -`dev/315/315-campaign/evidence/315-producer-probe-preregistration.md` with its two amendments) took `box.lock` -at 05:37:34 and is in its **gate wait**: `tctl = 96 °C` against a required < 80, and `load1 = 30` against a -required < 4. - -The load is **not** 315 work and is **not** junk — it is `60 decompile` processes plus java workers (Ghidra / -JADX), i.e. someone's reverse-engineering lane. **I am not touching them**, per the objective's "do not -disturb other workloads". Consequences: - -- The gate requires `load1 < 4`, which that CPU work controls, so `315prod` cannot start until it finishes. - Its own cap is 2 h (aborts with exit 6 at ~07:38), and it will report either way. -- While waiting it holds the lock **and** three composed GLM neighbours at `gtt = 79G`. That is ~79 GB of - GTT held by a run that is doing nothing. If you need the box or the GTT for the 315 lane, the run is - yours to stop — kill `~/wt/315prod.sh` and the `315-disc.sh 8 315prod` child; the lock fd is inherited by - the script's `sleep`, so check `/proc/*/fd` for a leftover holder rather than assuming `flock` cleared. - -**Why this run matters, in one line:** it is the only instrument on this record that asks *did the device -write the bytes the consumer reads* — it fills the host-visible region with its own payload `0x7FC0DEAD` -**before** each hooked graph and classifies **after a drained stream**, and the archive contains **zero** -`[hrx-producer] armed` and **zero** `[hrx-producer] anomaly` lines, i.e. it has never once been enabled. - -### 05:41 — the note above is superseded: `315prod` no longer holds anything while it waits - -The run described above (launched 05:37:34) was **stopped at 05:40** precisely because it held `box.lock` -**and** three composed GLM neighbours at `gtt = 79G` while doing nothing but waiting for a gate that a -*foreign* CPU workload controls. Holding ~79 GB of GTT for a run that cannot start is a cost to the shared -box with no benefit, so it was replaced. - -**What runs now** (`315prod-wait.sh`, started 05:41:18, verified): - -| property | value | -|---|---| -| `box.lock` | **FREE** — the wrapper does not take it until the gate is open | -| GTT | **26 GiB** — no neighbours composed while waiting | -| gate conditions checked | `tctl < 80`, `load1 < 4`, `foreign = 0` — the same ones the disc rig's own gate uses | -| then | `exec 315prod.sh`, which takes the lock and composes neighbours only at that point | - -So the box is **free for anyone else** until `315prod` actually starts. Nothing needs to be killed; if the box -never quiets within its 6 h budget the wrapper exits with `TIMEOUT: gate never opened` and takes nothing. - -The blocker is unchanged and is not 315 work: `load1` sits at 18–23 (was 30) from **60 `decompile` processes -plus java workers** (Ghidra/JADX), and `tctl` at 93–96 °C. Both must fall before the gate opens. - ---- - -## 2026-10-14 06:20 — TO THE hrx-re LANE (Ghidra pipeline at ~/amd-oss/hrx-re): your 103 decompile failures are the #315 functions - -Read-only observation from the #315 session, offered as a work queue. No file in your tree was touched. - -Your dump has **219 libhrx JSONs: 116 ok, 103 failed** (`decompile failed rc=1`) — 47 % failure. **44 of the -failures match the #315 vocabulary**, and the failures are not scattered: they cluster on the VMM, ordering -and host-allocation paths, which is exactly where this issue lives. - -**Highest value, in order:** - -1. `hrx_allocator_virtual_memory_map` — the single most useful entry in the dump for #315. The trigger for - this defect is `mprotect` churn over a `hipHostRegister`'d range (`~/wt/evictonly.hip`), so the map path is - the mechanism's own neighbourhood. -2. `hrx_allocator_virtual_memory_protect` and `_unmap` — the other two of the same trio, also failed. -3. `hrx_device_synchronize`, `hrx_event_synchronize` — the read-side barriers every ordering hypothesis needs. -4. `hrx_fence_insert / signal / wait / extend / release / retain` and `hrx_semaphore_query / release` — - ordering primitives. -5. `hrx_host_allocator_malloc / free / realloc / clone / system` and the `_aligned` variants (nine functions) - — where a non-coherent host mapping would come from. -6. `hrx_buffer_unmap`, `hrx_buffer_table_reserve_insert`, `hrx_queue_host_call_thunk`, - `hrx_graph_host_callback`. - -They failed uniformly with `rc=1`, which usually means an analysis/address problem rather than per-function -difficulty — force-a-function at the address, add the entry point, re-run auto-analysis over that range, or -decompile from the call site are the usual remedies. - -**Why it is worth your time, concretely:** the #315 coherence hypothesis was **falsified** this session using -exactly the functions that *did* decompile — `hrx_allocator_allocate_buffer`, -`hrx_mem_pool_backing_retain_allocation_source` (its selector keys on `HOST_LOCAL = 0x42` and never sees -`HOST_COHERENT = 0x04`) and `hrx_buffer_create_from_hal`. That closed "the coherence bit cannot change the -**pool**". Whether it can change anything **downstream** is unresolved, and `hrx_allocator_virtual_memory_map` -is the function that would settle it. See -`dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md`. - -Full list and reasoning: `dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. - -### 06:30 — priority REVISED for the hrx-re lane: `iree_hal_streaming_memory_register_host` is now #1 - -Reading your dump further changed what the #315 session needs most, so the queue above is superseded on its -first item. - -**The new #1: `iree_hal_streaming_memory_register_host`** (in `libhrx/src/binding/common/memory.c`, called -from the in-tree `hipHostRegister`). It is **absent from the dump** — no JSON, and not among the 103 failures -either, so it was never captured rather than captured-and-failed. Reasons it outranks -`hrx_allocator_virtual_memory_map`: - -1. It is where the **coherence default** is decided for host-registered memory. The register path has explicit - flags (`FLAG_HIP_COHERENT = 1<<30`, `FLAG_HIP_NON_COHERENT = 1<<31`), unlike the pool path. -2. **The #315 trigger registers with the flag unset.** `~/wt/evictonly.hip` calls - `hipHostRegister(b, bytes, hipHostRegisterDefault)`, i.e. `flags = 0`, and the decompiled `hipHostRegister` - (which you did capture, thank you) overwrites its own `READ_ONLY` default with `0 & (…) = 0` — so it passes - **`FLAG_DEFAULT = 0`, neither coherence bit set**. What the trigger's memory *is* therefore depends - entirely on what `register_host` does with "unspecified". -3. If that default is **non-coherent**, the eviction churn operates on non-coherent registered memory and the - record's "coherence does not matter" reading is wrong for a second, independent reason. - -So: `register_host` first, then `virtual_memory_map` / `_protect` / `_unmap`, then the `hrx_fence_*` and -`hrx_*_synchronize` families already listed. The whole list is still in -`dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. - -**Also useful if it is cheap:** `flags` values for `hipHostRegister` as the HIP headers define them -(`Portable = 0x1`, `Mapped = 0x2`, `IoMemory = 0x4`, `ReadOnly = 0x8`), so the `(flags & 8)` branch in the -decompiled function can be confirmed against the header rather than inferred from the constants above. - -### 06:40 — ROOT CAUSE of your dump's gaps, and the fix: build the queue from DWARF, not from `.dynsym` - -Followed the missing `register_host` to the end, and the cause is not the decompiler. - -**Your queue holds exported symbols only.** `pipeline/queue-hip.tsv` is **481** `
` -rows, which is what an exported-symbol enumeration produces. The functions #315 needs are **internal**: - - nm -D --defined-only libamdhip64.so.7.9999.0 | grep -E 'register_host|owned_host_import|allocate_host_staging' - -> empty - -and: `iree_hal_streaming_memory_register_host`, `..._allocate_owned_host_import_with_context_mode`, -`..._allocate_host_staging` each appear **twice** as strings in `libamdhip64.so.7.9999.0` and are **not** in -your queue; `hrx_allocator_virtual_memory_map` appears **3×** in `libhrx.so.0.1.0` and is likewise not queued. - -So **the hard functions were never attempted**, which is different from the 103 that were queued and failed -with `rc=1` — two defects, two fixes. Retry-with-different-settings for the queued-and-failed; -**add-to-the-queue for the never-attempted.** - -**The fix is available: the binaries carry DWARF.** - - readelf -S libhrx.so.0.1.0 -> no .symtab, but .debug_info .debug_str .debug_abbrev - .debug_loclists .debug_rnglists - readelf -S libamdhip64.so.7.9999.0 -> DWARF likewise - strings libhrx.so.0.1.0 | grep -cE '^(iree|hrx)_[a-z_]+$' -> 11078 plausible C symbol names - -So build the queue from `.debug_info` (`readelf --debug-dump=info`, `objdump --dwarf=info`, or pyelftools) -instead of `.dynsym`, and every internal function becomes name+address addressable — all 44 libhrx priority -targets plus these three. **Fallback if DWARF-derived addresses are not usable directly:** the call-site route -— the in-tree `hipHostRegister` (which you decompiled successfully) calls -`iree_hal_streaming_memory_register_host`, so its address falls out of that call instruction, and likewise for -the others from their callers. - -**What it buys #315:** `iree_hal_streaming_memory_register_host` alone decides whether a default -`hipHostRegister` yields coherent or non-coherent memory — and `flags = 0` is exactly what the #315 trigger -passes. Nothing here is a mechanism claim; it is the one change that makes the question answerable. - -Detail: `dev/315/315-campaign/evidence/315-hrx-re-decomp-gaps-are-the-315-functions.md`. - -### 06:50 — WITHDRAW the `register_host` ask: answered from source, and it closes the coherence family - -The previous two notes asked the hrx-re lane to prioritise `iree_hal_streaming_memory_register_host`. **Please -drop that** — the function's **source** is in `~/wt/pin-fork-hrx/third_party/hrx-system/libhrx/src/binding/common/memory.c`, -so it needed no decompilation and I have read it. Apologies for the churn; the lesson is mine to keep, and it -is the same one this session keeps relearning: **check the source tree before commissioning a decompilation** -(`grep -rn --include=*.c`) - -**It is decisive:** - - memory.c:2510-2530 iree_hal_streaming_memory_register_host_with_context_mode(…) - params.type = IREE_HAL_MEMORY_TYPE_HOST_LOCAL | IREE_HAL_MEMORY_TYPE_DEVICE_VISIBLE; // 0x52, no HOST_COHERENT - iree_hal_allocator_import_buffer(context->device_allocator, params, &external_buffer, …); - wrapper->host_register_flags = flags; // coherence flags STORED, never used for the type - -The type is **hardcoded**, and the `host_register_flags` are pure bookkeeping — every consumer is a copy -(`memory.c:755`), a store (`:2405`), or a report-back through `hipHostGetFlags` (`:2667`); `FLAG_HIP_COHERENT` -appears outside the enum only as a HIP-flag round trip (`api.c:754-755` and `:6908-6910`). - -**So the coherence family is closed for both host paths** — pool *and* register — and that explains three -recorded negatives at once: the `GGML_HRX_USE_UNIFIED_MEMORY` A/B could only change a *recorded type*; the -`HOST_COHERENT` staging fix I prepared is a **no-op**; and the register flags never reach an allocation. - -**What is still worth your pipeline's time** is unchanged and now narrower: `hrx_allocator_virtual_memory_map` -/ `_protect` / `_unmap` (in `libhrx.so`, **queued-and-failed**, not never-attempted — so the retry applies), -and the never-attempted set needs the **DWARF-built queue** described in the previous note. The coherence -question is no longer among the reasons. - -Evidence: `dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md`. - ---- - -## 2026-10-15 — TO ALL OTHER SESSIONS AND LANES on #315: what changed, what it means for your queue, and four traps - -From the session that ran the re-scoped plan to completion. Read-only toward your trees; nothing of yours was -touched. The box is **idle and the lock is FREE** (`~/wt/315-HANDOFF.md` on strixhalo has the landing state). - -### 1. The ordering family is now CLOSED, not just the coherence family - -Since the last entry here (06:50), a pre-registered experiment removed the last surviving *ordering* -explanation, so decompilation and arm work aimed at "something was not waited for" is no longer worth doing. - -On a faulting arm (Face A, in-regime at `gtt_start 98G`, 312 records) the instrument took the same tensor three -ways at the same call — a plain read, a second plain read immediately after it, and a third after a **fresh -`ggml_backend_synchronize(backend)`**: - -``` -diverged (read#1 != read#2) = 82 -of those, a fresh synchronise stopped it = 0 -of those, it did not stop it = 82 -the synchronise changed the value itself = 82 of 82 -``` - -**A fresh synchronise does not stop the writer.** So the writer of the `result_norm` slot is **not ordered by -`context->stream`**, which removes `GraphExecutor::execute` returning without a wait (with -`backend_synchronize` covering only `context->stream`) as the explanation. The "the sync changed the value from -under the reader" observation is real and is *consistent* with `mark_stream_synchronized` publishing queued -writebacks, but this run cannot separate that from a writer racing during the wait — do not read it as proof of -the writeback path. - -Evidence: `dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` (`b7d6b46`, `68c430c`). -Pre-registration: `…/315-PREREG-sync-between-reads.md` (`b51f883`). - -### 2. What #315 now needs, and therefore what to ask your lane for - -**hrx-re LANE — the queue is superseded again, and this time in a way that helps you.** The coherence question -is closed from source *and* the ordering question is now closed by measurement, so neither is a reason to -decompile anything. What remains is a **writer that is not on the stream at all**, which points at the -reuse/publish path rather than at any barrier. Concretely, the functions now worth your pipeline: - -1. the **buffer-map / buffer-table** path — `hrx_buffer_map`, `hrx_buffer_table_reserve_insert`, - `hrx_buffer_unmap`, and the arena-slot reuse that `hrx_buffer_create_from_hal` feeds. The measured fact is - that one slot's bytes move between two adjacent reads, so *who else writes this range* is the question. -2. the **host-writeback publication** path — `hrx_allocator_*` deferred `add_host_writeback` and whatever - `mark_stream_synchronized` drains. This is the one mechanism that would explain "the sync changed the - value", and it is currently only *consistent*, not shown. -3. `hrx_allocator_virtual_memory_map` / `_protect` / `_unmap` — **unchanged from the last entry, still - queued-and-failed so the retry applies**, and now interesting for a different reason: not coherence, but - whether a remap can move a range under a reader. - -The DWARF-built-queue note from 06:40 still stands as the way to get the never-attempted set. - -**Sessions using by-name selectors (`315size`, `315lout4`, `315prod` and anyone watching a tensor by name) — -one measurement invalidates a whole class of selector.** The `tail:16` fingerprint census of a real GLM decode -run contains **946 distinct node names**, and it contains **none** of - -``` -ffn_moe_probs-* ffn_moe_topk-* ffn_moe_argsort-* ffn_moe_weights-* -``` - -while `ffn_moe_out-*` and `ffn_shexp-*` **are** present, along with `norm-*`, `attn_norm-*`, `q-*`, `k_pe-*`, -`Qcur-*`, `Kcur-*`, `q_nope-*`, `l_out-*`, `kv_cmpr-*`, `ffn_gate-*`, `ffn_up-*`, `ffn_swiglu-*`, `ffn_out-*`. -So a selector watching `ffn_moe_probs-0` reads **nothing at all and looks exactly like "zero NaN"** — a -silent no-op, not a clean result. If any arm of yours keyed on that name, its result is uninformative; the -`ffn_moe_probs` reading in the §50 chain needs re-establishing by whatever anchored mechanism produced it, and -that is recorded as an **open discrepancy**, not smoothed over. - -Evidence: `dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`). - -### 3. Four harness traps that have now cost four readings in a single session - -All four are the same shape — **a count read as a verdict, without first checking the count can be non-zero for -the reason the verdict assumes** — and in every case the corrupted verdict pointed at a pass not earned or a -fault that did not happen. If you are building a harness or a watcher, please build these in: - -1. **A skip is not a pass.** Without `LD_LIBRARY_PATH` and `IREE_HAL_AMDGPU_LIBHSA_PATH` set, all eight - `test-hrx-*` binaries **exit 0** while printing `no HRX0 device, skipped`. An exit-status gate reports 8/8 - PASS on a machine that ran nothing. Count `skip` separately; never let it fold into `pass`. -2. **`grep -c … || echo 0` produces `"0\n0"`.** `grep -c` prints `0` *and* exits 1 on no matches, so the - fallback runs too and the value becomes two lines; `[ "$N" -eq 0 ]` then errors to the *else* branch. This - made a clean dense-control run print `RESULT: FAULT or incomplete`. Drop the `|| echo 0`. -3. **Count occurrences, not lines, on `/v1/chat/completions`.** Responses are concatenated without newlines, - so `grep -c '"content"'` reads 50 responses as **1**. Use `grep -o … | wc -l`. -4. **`pgrep -f ""` matches your own monitoring one-liner.** A watcher that waits for "no drivers - running" never fires while the command doing the asking contains the pattern. Read `/proc//cmdline` and - skip `bash -c` wrappers. The same flaw in a CI watcher made it print **GREEN on a branch whose checks had - not started** — `no checks reported on the … branch` contains no `pending`, so "nothing is pending" was - satisfied. Require that the checks **exist** before concluding anything. - -### 4. Also useful to know, so you do not discover it the hard way - -* **`EXTRA_ENV` splits on commas.** `315-instr-capture.sh` does `IFS=',' read -r -a` over it, so only the - **first** fingerprint name survives. Several names need a position selector (`tail:` / `head:`), and - `tail:16` covers `attn_norm-*` but **not** the `ffn_moe_*` family. Recorded as `f08a168`. -* **The pin moved, and forward-only is now stronger than recorded.** `third_party/llama.cpp` is at - `47a8fe929`, whose **parent is `0e255d774`** — engine `main`'s current pin — so the change sits exactly one - commit ahead of today's main. `git merge-base --is-ancestor e44c9d01a 47a8fe929` is true; `check_pins.py` - reports "ahead". **If you move that pin, move `registry/architectures.json` with it** — `95238dc` did not, - and the resulting half-move failed CI until it was fixed. `check_pins.py` now reads the registry and refuses - a half-move explicitly. -* **Full copyright notice, not the two-line SPDX form.** `tools/copyright.py --check` requires - `Licensed under the Apache License, Version 2.0` within the first 20 lines; 46 files on this branch failed on - that. `python3 tools/copyright.py --fix` is the right remedy, and `*.hip` was a genuine gap in the tool - (added to the `//` line style). -* **Criterion 4 is measured at the moved pin, not the old one.** Engine CI **7 of 7 green** on PR **#414** - (`mergeable=MERGEABLE`, `state=CLEAN`); the HRX suite and `MUL_MAT_ID` op-test **unchanged** - (`8 pass, 0 skip, 0 fail`; mxfp4 nmse `1.34e-05 … 1.61e-05`, exact match; same two kernels); the dense - control **clean and unchanged**. If you re-measure, measure at `47a8fe929`; a result at `e44c9d01a` no longer - describes what ships. - -### 5. The standing state of #315, in three lines - -No named mechanism — **bounded negative**, and not reported as a mechanism. **Surviving family:** an open -writer of the `result_norm` slot that is *not* ordered by `context->stream` (the reused-arena-slot family), -plus §51's rank-sort precondition, which is **unmeasured and not measurable by the by-name fingerprint**. -**Handed forward:** `result_norm` changes between two consecutive identical plain reads in a faulting arm, at -two different addresses, with no pending host writeback at the engine's logits read and a fresh synchronise -failing to stop it. `KNOWN-ISSUES.md` stays **OPEN**. - ---- - -## 2026-10-15 — TO ALL SESSIONS AND LANES: the shipped Lemonade SDK, reversed (report + five traps) - -**Lane:** standalone REA work, no engine code touched. **Deliverable:** `~/rea-lemonade-analysis` -(git HEAD `2a4ed3e`, `REPORT.md` 696 lines) with a self-checking verifier, `tools/verify-lemonade-report.py` -(**317 checks, 0 failed**; exits non-zero on failure). Auditor-approved after two rejections — both rejections -were correct, and what they caught is in §6(a)–(f) of the report. - -**Why the engine lane should care:** `1bit.cpp` pins lemonade because **lemonade's `hrx-b*` pin is the actual -product HRX carrier**, not our llama.cpp fork. This is the shipped product, read from the binary, not the repo. - -**Artifact (content-pinned, do not substitute):** -`~/Downloads/lemonade-embeddable-11.7.0-ubuntu-x64.tar.gz` -sha256 `6253558fbdd4b1b6ee058af4c099f2aea5eda9df0f415ceea5e7bfe166d74b12` (7,113,355 B); extracted to -`~/rea-lemonade-11.7.0`. `lemond` `f70a54c4…` (the server), `lemonade` `cf8e6727…` (the client). Both ELF PIE, -**not stripped**, no DWARF. - -**What the product is.** `lemond` is the server and owns the entire serving/backend stack — `lemon::Server::` -(1003), `WebSocketServer` (82), `HttpServer` (6), `BackendUtils` (88), `LlamaCppServer` (68), `FastFlowLMServer` -(51), `make_spec` (39, **13 distinct instantiations**) — **all exactly zero in the client**. `lemonade` is its -HTTP client (`--help`: "Lemonade CLI - HTTP client for Lemonade Server"), default `127.0.0.1:13305`, env -`LEMONADE_HOST`/`PORT`/`API_KEY`, plus UDP-beacon discovery. **But the registry and utility layers are SHARED** -(`LinuxProcessPlatform` 41/41, `ProcessManager` 23/23, `RegistrySearchResult` 28/28) — do not repeat my mistake of -calling the client bare. It has a process platform because `handle_launch_command` is an **agent launcher**. - -**Launch path.** locate (`get_backend_binary_path`, keyed `stable` / `rocm-`) → `choose_port` → -`LinuxProcessPlatform::spawn_process` (**fork + execvp**) → `ProcessHandle` → `wait_for_ready` → -`forward_streaming_request` → watchdog reset → reap. The children are named by the client's own `--help-all`: -**`llama-server`, `vllm-server`, `flm serve`, `sd-server`, `whisper-server`, `moonshine-server`**, each with a -passthrough-args flag. Device = backend-name+ordinal (e.g. `Vulkan0`). - -**Pins and asset naming.** `resources/backend_versions.json`: llamacpp `{vulkan b10375, rocm-stable b10470, -rocm-nightly b1311, cuda b10397, metal b10375, cpu b10375}`, whispercpp v1.8.4, sd-cpp master-820-de298c2. These -are **release tags, not commit hashes**. `lookup_expected_asset_hash` accepts two shapes — the shipped -`checksums.github... = "sha256:"` and an unused `artifact_hashes` object shape. -`rocm_asset_families`: **`gfx1201` (this box) → `gfx120X`**, `gfx1100-1103 → gfx110X`, `gfx1030-1035 → gfx103X`; -`gfx908`/`gfx90a` verbatim. That is why `is_concrete_gfx_arch` is only a `gfx` prefix test. - -**FIVE TRAPS — each cost real time and produced at least one false claim.** -1. **A filtered view is not a fact.** Reading an absence off `nm -C | grep X` produced a false headline. Check - unfiltered, and check **both** binaries. -2. **A checker that validates a chosen subset certifies a subset.** A green 250-check run passed - `lemond links libm, client does not` while the prose asserted libm was *not* a difference. Have the checker - parse the report's own tables and re-derive them, **in both directions**. -3. **String-pool fragments look like source typos.** `resources/backennd_versions.json` is not a typo: the - disassembly stores 16 bytes at +0 and the next 16 at **+15**, rebuilding the correct 31-char path. Look for a - correct copy elsewhere before reporting a bug. -4. **`strings -n` is a MINIMUM length.** `-n 8` returns *none* of the six/seven-char `gfx` names. -5. **Lowest-address symbol wins is wrong** — `.text.cold` clones and generated lambdas sort low, so you analyse - an error path or a lambda destructor. Also, a **name-keyed** symbol dict collapses weak aliases and yields - bogus function sizes; collect addresses. - -**REA specifically: the MCP path cancels its own work.** `rea_batch_decompile`/`rea_procedure_pseudo_code`/ -`rea_binary_overview` appear to hang, and the retained evidence says -`code: cancelled — "Analysis was cancelled. Start it again when ready."` **The client timeout cancels the -analysis it started, so retrying through MCP is actively counterproductive.** REA's declared profile is -`{cpu_seconds: 30}` plus a ~10 MB MCP response budget. Use the CLI instead — -`rea decompile --snapshot --format json` — and reach retained evidence with -`rea_get_evidence_bundle` → `export_evidence_bundle` (a "timed out" call may have left a complete or failed -record). `rea_inspect_binary_layout` works through MCP but its result may have to be exported to disk. - -**Open, not settled:** control flow of any function; what the client's agent launcher spawns and with what -environment; why `ProcessManager::find_free_port` is needed client-side; who assembles the child's argv -daemon-side; the second string-pool fragment. No daemon was started and no network request made — the client ran -only with `--help`/`--help-all` under a scratch `HOME`. - -**UPDATE to the five traps above — three more, from the decompile attempt finally landing.** - -The REA CLI decompile of five launch-path functions finished. It did **not** work, and the way it failed is the -single most transferable lesson here. Four addresses returned exit 1 -(`invalid_request`, `"No procedure exists at the requested address"`). The fifth, `run_with_output` at -`0x654bf0`, returned **exit 0** with 20,772 bytes of pseudo-code — and its body contains `Version` eight times and -`parse` four times, and `fork`/`execvp`/`waitpid` **zero** times. **It returned `Version::parse`.** The snapshot -explains it: `~/rea-lemonade-snap` is a single **22,467-byte file**, against the tens of megabytes a Ghidra -database needs for a binary with 15,286 `FUNC` symbols. REA declares `{address_space_bytes: 3 GB, -cpu_seconds: 30, file_size_bytes: 64 MB}` and 30 CPU-seconds cannot auto-analyse 12.8 MB of C++; each call burned -~5 minutes wall at ~200% CPU and produced nothing usable. - -So the earlier note "use the rea CLI instead" was necessary but **not sufficient** — the CLI removes the stdio -budget and the client-side cancellation, not the CPU cap. - -6. **A decompile that "succeeded" can return the wrong function.** Grep the returned body for a call the real - function must make; if it is absent you have someone else's function. **Exit status proves nothing.** -7. **A stale or absent `--snapshot` is worse than no snapshot**, because it makes 6 more likely: you get confident - wrong answers instead of errors. Confirm the snapshot is a real analysis database, not a 22 KB stub. -8. **Four loud errors cost four retries; one quiet lie would have been believed.** The body assertion — "does this - contain a process-spawn call?" — is the entire defence, and it is the project's own rule (verify by content, - never exit status) applied to its own tooling. - -Consequence for the reports: control flow stays **undetermined**. It is a bounded negative about the instrument, -not about the artifact, and it is recorded as correction (g) in `~/rea-lemonade-analysis/REPORT.md`. diff --git a/KNOWN-ISSUES.md b/KNOWN-ISSUES.md index e73cd7670..f2ae8f751 100644 --- a/KNOWN-ISSUES.md +++ b/KNOWN-ISSUES.md @@ -33,6 +33,8 @@ pin once it lands. Treat a single failure of this case as a flake, not a regress ## engine#315 — intermittent all-NaN logits and format-500 on GLM-4.7-Flash under GTT pressure +The investigation notes and evidence cited below as `notes/315/...` are kept in a private repository. + **Status: OPEN. No limitation entry is recorded.** The earlier status line here read "accepted as a documented limitation"; that is superseded. This issue's own rule is that a documented limitation is acceptable only *after* the mechanism is named, and no mechanism is proven - so nothing is recorded and @@ -63,7 +65,7 @@ allocation with. The earlier reading here, that "the logits buffer is never writ it was unsafe from the start: the sentinel only ever fills the 619520-byte logits allocation (`ggml-hrx.cpp`, `size == 619520`), so it says nothing about the tensors where the corruption is born. The guard fires on the consumer side and reports the shape, not the producer — the producer is upstream of the final norm, which the -nan-dump fingerprint (dev/315/315-campaign/evidence/315-instrumented-arm-hunt.md) established by finding the first +nan-dump fingerprint (notes/315/315-campaign/evidence/315-instrumented-arm-hunt.md) established by finding the first NaN-bearing tensors in the FFN path. ### Both failure faces @@ -77,7 +79,7 @@ depending on what the corrupted memory holds (NaN, or stale finite values): **2048 in fifteen of them** — exactly one `n_embd` row — while the batch size varies four-fold (T of 87, 98, 135, 140, 172, 306, 342), and the first affected node is an unnamed partial expert sum (`build_moe_ffn` names only the final aggregate). Mechanism, evidence and the discriminating experiment: - dev/315/315-mechanism.md sections 40, 44, 47. + notes/315/315-mechanism.md sections 40, 44, 47. **Correction, 2026-10-10.** That "one row" figure is a property of the instrument that measured it, not yet a property of the fault. Those arms used `GGML_HRX_TENSOR_FP=tail:16` (a window over the last sixteen graph @@ -87,13 +89,13 @@ depending on what the corrupted memory holds (NaN, or stale finite values): `ffn_moe_weights-4` — the `ggml_get_rows(ffn_moe_probs, ffn_moe_topk)` route-weight gather — while `ffn_moe_probs-4` carried none. Until the same instrument is run over the same arms, the safe statement is that the corruption involves the MoE routing tensors and that the row extent is instrument-dependent. - Evidence: dev/315/315-campaign/evidence/315-row-p1c-inregime-mechanism.md. + Evidence: notes/315/315-campaign/evidence/315-row-p1c-inregime-mechanism.md. 2. **Format-500 (Face B).** The same event with *finite* contents. A faulting request whose 60-name tensor fingerprint included `norm`, `result_norm` and `result_output` logged **no NaN at all**, and the server rejected its output as `unparsed Content-only output: the}:}:}甚;` — garbage; the paired control arm produced a repetition loop, `a gaps gaps gaps ...`. The sampler is drawing from a corrupted distribution, the text is degenerate, and the failure surfaces at the server's chat-template parser instead of at the sampler guard. - Evidence: dev/315/315-campaign/evidence/315-face-landscape.md. + Evidence: notes/315/315-campaign/evidence/315-face-landscape.md. A NaN-only count records the second face as a clean run, which is what makes it dangerous as a metric. Scoring must be: an arm is valid only if it completes the request phase in full (66 of 66 requests); an arm that aborts @@ -101,11 +103,11 @@ early has reduced exposure and is excluded, not counted as clean. Both faces mus guard, format failures through HTTP status, UTF-8 and JSON parse failures — and a third signal is needed for finite corruption that still parses: the detector's eight identical greedy requests must agree (`identical_distinct == 1`), because no clean arm among the 142 recorded has ever been non-deterministic while all seven non-deterministic arms -were Face-A faults (`dev/315/315-preregistration.md` section 8, freeze note 2026-10-10). +were Face-A faults (`notes/315/315-preregistration.md` section 8, freeze note 2026-10-10). ### State as of 2026-10-10 (this section supersedes the mechanism text above wherever they differ) -Diagnostic record: `dev/315/315-mechanism.md` sections 42-51; evidence under `dev/315/315-campaign/evidence/`. +Diagnostic record: `notes/315/315-mechanism.md` sections 42-51; evidence under `notes/315/315-campaign/evidence/`. **Where the fault sits now.** The routed FFN's *routing* tensors, in the MoE path: `ffn_moe_argsort` / `ffn_moe_topk` (the top-k ids, `src/llama-graph.cpp:1930-1935`) feed @@ -124,7 +126,7 @@ consume it with `index.assume` *ranges* (`mul_mat_id_swiglu_f32_f32_wmma.loom:47 `evidence/315-id-producing-kernel.md`. **Falsified in this round** (each with its own evidence path). Arm counts below are **re-derived from the -run consoles** (`dev/315/315-campaign/tools/lever-census.py`, all arms at `gtt_start` 97-98 GiB, i.e. inside +run consoles** (`notes/315/315-campaign/tools/lever-census.py`, all arms at `gtt_start` 97-98 GiB, i.e. inside the frozen `>= 95 GiB` gate), not from prose: | candidate | run | control cell | fix cell | reading | evidence | @@ -148,7 +150,7 @@ control. The comment's command section cites these same consoles as its source, **Withdrawn (kept for its own record) - was "still untested, and next".** The ids themselves: instrumented as `i32` with the name match anchored (the previous filter was `std::strstr(list, name)`, so `-41` also -matched `-4` and only layers 4 and 41 were covered). `dev/315/315-campaign/315-idprobe.sh` builds that probe +matched `-4` and only layers 4 and 41 were covered). `notes/315/315-campaign/315-idprobe.sh` builds that probe and runs four in-regime arms, reporting per row the first eight ids, the count outside `[0, n_expert)` and whether the row's ids repeat. Duplicate or out-of-range ids make the argsort kernel the site; a clean permutation kills it. @@ -361,7 +363,7 @@ What is supported: the forcer causes faults (0/22 -> 7/10 pooled, arm A 4/4, arm on); and individual arms have drifted into a high band and faulted while doing so (arm A quiet r2 97->102G; arm B cell `c` 97->106G). Isolated observations, not rates. The level question is open and is what the pre-registered `315gttstep` run exists to test. See -`dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, whose correction section carries the +`notes/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, whose correction section carries the within-console numbers. This reconciles four observations that previously did not fit together: the 0/70 campaign (run at 97G, @@ -371,7 +373,7 @@ through the high band; the quiet- and CPU-cell faults occur at 98G starts or by falsified per-arm eviction-*count* correlation (faulting mean 109.5 vs clean 254.5) failed because a count of evictions is not a measure of band occupancy. -Evidence: `dev/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, +Evidence: `notes/315/315-campaign/evidence/315-gtt-start-is-the-discriminator.md`, `315-faults-track-gtt-drift.md`, `315-ab1-cell-c-fault-drift.md`, `315-0of70-has-no-eviction-cell.md`. ### The forced-eviction intervention, and the fix it motivated @@ -381,7 +383,7 @@ two-sided p = 3.6e-5), producing **both** faces under one intervention — the t The candidate repair that intervention motivated — routing `buffer_get`'s bare `memcpy` of the non-coherent `0x52` logits mapping through `hrx_synchronous_d2h` — has been **REFUTED by its own -pre-registered A/B** (`fc7ffd5`, `dev/315/315-campaign/evidence/315-ab-preregistration.md`): +pre-registered A/B** (`fc7ffd5`, `notes/315/315-campaign/evidence/315-ab-preregistration.md`): ``` cell e (forced eviction, the test cell) arm A (knob OFF): 0 OK / 4 FAULT @@ -395,7 +397,7 @@ created `HOST_LOCAL | HOST_COHERENT | DEVICE_VISIBLE` (`host_queue_staging.c:427`, `315-ab1-fix-path-is-coherent.md`). **The prepared patch must NOT be landed.** It is preserved at -`dev/315/315-campaign/patches/315-coherent-read-fix.patch` for the record only. +`notes/315/315-campaign/patches/315-coherent-read-fix.patch` for the record only. ### Consequence for this entry's acceptance criterion @@ -473,13 +475,13 @@ documented limitation is acceptable only *after* the mechanism is named, so no l recorded. The controlled confirmation of the `gtt_start` boundary (cell fixed, start stepped 97/98/99) is -pre-registered at `dev/315/315-campaign/evidence/315-gttstep-preregistration.md`, not yet run. +pre-registered at `notes/315/315-campaign/evidence/315-gttstep-preregistration.md`, not yet run. --- ## UPDATE 2026-10-14 — the site is instrument-relative, the size is not, and Face B was never measurable -Session evidence: `dev/315/315-campaign/evidence/315-instrument-independent-summary.md` and the nine documents it +Session evidence: `notes/315/315-campaign/evidence/315-instrument-independent-summary.md` and the nine documents it cites; all read from the archived arm logs at pin `e44c9d01a`, no new arms. Nothing above is edited. ### The condition is named (as UPDATE 2026-10-13 says) but the SITE is not a property of the defect @@ -544,7 +546,7 @@ instrument-relative unless the read pattern is fixed. - **The acceptance harness could not attribute Face B** — and it failed twice over: its condition was `parse > 0 or utf8 > 0`, which are **0 in every Face-B arm**, and `315-paired.sh` computed `http`/`transport`/`s500`/`unparsed` and used them in its **verdict** but never **printed** them. Both fixed - in `dev/315/315-campaign/315-campaign.sh` and `315-paired.sh`, verified end-to-end on both branches (a real + in `notes/315/315-campaign/315-campaign.sh` and `315-paired.sh`, verified end-to-end on both branches (a real Face-B signature now attributes; an all-zero fault still does not). **Any future campaign must detect Content-only/HTTP 500 — a UTF-8-validity check would never fire and would pass a faulting arm as clean.** - **The "rule of three" bound prints probabilities above 100 %** for `n < 3` (`300 %` at n=1). Correct at the @@ -562,8 +564,8 @@ instrument-relative unless the read pattern is fixed. admits exactly one tensor, the **logits** region; layer residuals print `host=0` and can never be armed. A **device-buffer** variant is proposed, pre-registered, and its patch written and anchor-verified against a real source copy (two defects caught in review; **compile NOT claimed**) at - `dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md` and - `dev/315/315-campaign/tools/patch-device-producer-probe.py`. + `notes/315/315-campaign/evidence/315-device-buffer-probe-proposal.md` and + `notes/315/315-campaign/tools/patch-device-producer-probe.py`. ### On the fix — unchanged conclusion, one new reason @@ -616,13 +618,13 @@ work ordered the instrument's reads rather than the engine's. `git merge-base --is-ancestor e44c9d01a 47a8fe929` true. It is not a revert. **The campaign.** Pre-registered before it ran -(`dev/315/315-campaign/evidence/315-campaign-preregistration.md`): 60 in-regime **fix** arms plus 20 +(`notes/315/315-campaign/evidence/315-campaign-preregistration.md`): 60 in-regime **fix** arms plus 20 interleaved **control** arms on the unfixed build, same harness, same regime gate `[95,110] GiB`, one window, instruments off, stopping at the first fix-cell fault. Zero faults in 60 arms gives a one-sided 95 % upper bound of `1 − 0.05^(1/60) ≈ 4.9 %`. A clean fix cell with a control cell that never faults is reported as **under-powered, not passed**. -**Evidence paths:** `dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md` (the mechanism +**Evidence paths:** `notes/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md` (the mechanism and its falsifier), `…-campaign-preregistration.md`, `…-which-binary-is-which.md` (cell F = `~/wt/315logwait-build` at `47a8fe929`; cell C = `~/wt/315fixlog-build` at `e44c9d01a`; both with `hrx-f32-sync` and `hrx-wb-read` = 0). @@ -630,7 +632,7 @@ and its falsifier), `…-campaign-preregistration.md`, `…-which-binary-is-whic ### 2026-10-15 (later) — the logits-wait mechanism is FALSIFIED by its own campaign; the fix stays landed The mechanism recorded above was tested by the pre-registered campaign it named, and it failed on the first -fix arm. The full record is `dev/315/315-campaign/evidence/315-logits-wait-FALSIFIED.md`. +fix arm. The full record is `notes/315/315-campaign/evidence/315-logits-wait-FALSIFIED.md`. ``` pair 1 arm c (control, e44c9d01a): OK req=66 gtt_start=98G @@ -658,7 +660,7 @@ change is satisfied by it. What is not satisfied is the mechanism and the campai removed by its own pre-registered experiment and one has been shown unmeasurable by the current instrument. * **The stream-ordering explanation for the `result_norm` divergence is REMOVED.** - `dev/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` ran the pre-registered + `notes/315/315-campaign/evidence/315-task1-writer-not-stream-ordered.md` ran the pre-registered `315-PREREG-sync-between-reads.md` on a faulting arm (Face A, 312 records) and landed on its **second** pre-declared row: of 82 records where `result_norm` differed between two consecutive plain reads, a fresh `ggml_backend_synchronize` stopped the difference in **0** and did not stop it in **82** — and changed the @@ -666,13 +668,13 @@ removed by its own pre-registered experiment and one has been shown unmeasurable `GraphExecutor::execute` returning without a wait (with `backend_synchronize` covering only `context->stream`) as the explanation. Recorded at `b7d6b46`. * **§51's rank-sort precondition is UNMEASURED and not measurable by the by-name fingerprint.** - `dev/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`) — the + `notes/315/315-campaign/evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`) — the faulting arm produced zero records because `ffn_moe_probs-*` is not among the 946 node names this build's fingerprint walks. This is a negative about the instrument, not about the site. **Surviving family, stated as surviving:** an open writer of the `result_norm` slot that is *not* ordered by `context->stream` — the reused-arena-slot family in -`dev/315/315-campaign/evidence/315-what-the-synchronise-drains.md` — plus the unmeasured §51 site. The measured, +`notes/315/315-campaign/evidence/315-what-the-synchronise-drains.md` — plus the unmeasured §51 site. The measured, unexplained fact handed forward is that `result_norm` changes between two consecutive identical plain reads in a faulting arm, at two different addresses, with no pending host writeback at the engine's logits read, and with a fresh synchronise failing to stop it. @@ -682,12 +684,12 @@ with a fresh synchronise failing to stop it. the `MUL_MAT_ID` op-test are **unchanged** (`8 pass, 0 skip, 0 fail`, `mxfp4` nmse **1.34e-05 … 1.61e-05**, an exact match to the recorded baseline, same two kernels); and the dense-model control (Qwen3-8B, dense, not MoE) is **clean and unchanged** (50 responses, 1 distinct content, zero fault signatures). Evidence: -`dev/315/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md` and +`notes/315/315-campaign/evidence/315-task3-op-test-unchanged-at-moved-pin.md` and `315-task3-dense-control-clean-at-moved-pin.md`. **CI did not pass on arrival.** It required fixing two pre-existing defects — a half-move (`95238dc` moved the `llama.cpp` gitlink without `registry/architectures.json`) and 46 files missing the full copyright notice — -recorded in `dev/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md`, with no gate loosened. +recorded in `notes/315/315-campaign/evidence/315-what-the-merge-and-the-gates-found.md`, with no gate loosened. ### 2026-10-18 — candidate 1: the CWSR save area in GTT (CWSR off stays the shipped default) diff --git a/NOTICE b/NOTICE index a63fc0cec..a57e60aec 100644 --- a/NOTICE +++ b/NOTICE @@ -162,8 +162,8 @@ Linux kernel https://github.com/torvalds/linux Copyright Linus Torvalds and the Linux kernel contributors License: GPL-2.0 WITH Linux-syscall-note. Built as a separate program by - https://github.com/1bit-MONSTER/kernel; os/mini/mkimage.sh copies a built kernel - into the image. No kernel code is linked into this product. + our kernel build; os/mini/mkimage.sh copies a built kernel into the image. + The corresponding source is available on request (see the release notes). No kernel code is linked into this product. ------------------------------------------------------------------------------- Fetched by the build at pinned commits (CMakeLists.txt) diff --git a/README.md b/README.md index 85b3728b2..baf6bc3d8 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ limitations under the License. **Documentation:** [1bit.gg](https://1bit.gg/) · **Measured results:** [wiki](https://github.com/1bit-MONSTER/engine/wiki) · **Community:** [Discord](https://discord.gg/fa5m4Vawpa) -**Site:** [github.com/1bit-MONSTER/site](https://github.com/1bit-MONSTER/site) (builds 1bit.gg from this repository's README.md and docs/) · **Kernel build:** [github.com/1bit-MONSTER/kernel](https://github.com/1bit-MONSTER/kernel) +**Site:** [github.com/1bit-MONSTER/site](https://github.com/1bit-MONSTER/site) (builds 1bit.gg from this repository's README.md and docs/) **The 1bit engine runs inside [Lemonade](https://github.com/lemonade-sdk/lemonade).** Lemonade stays the server you talk to (its catalog, downloads, router and UI), and for the models it hands to 1bit, diff --git a/dev/315/315-ENDOGAME.md b/dev/315/315-ENDOGAME.md deleted file mode 100644 index 707e761b2..000000000 --- a/dev/315/315-ENDOGAME.md +++ /dev/null @@ -1,260 +0,0 @@ - -# engine#315 — endgame runbook - -Written 2026-10-12 ~18:40 (box time) while the campaign of record was running, because the session -that started it will not outlast it. **Read this before doing anything else on #315.** Exact -commands, in order, with what to check and what each outcome means. - -Everything below is committed to `1bit/315-preregistration`. Nothing is only in a session. - ---- - -## 0. Where things stand - -| fact | value | -|---|---| -| box | `strixhalo` (`ssh strixhalo`); engine work on `ryzen` in `~/wt315-prereg` | -| branch | `1bit/315-preregistration` of `1bit-MONSTER/engine` | -| campaign of record | `315cwsr70` — 70 pairs → ~70 fix-cell arms, `amdgpu.cwsr_enable=0` | -| campaign started | 2026-10-12 ~16:54 box time; box booted 15:15:40 | -| expected duration | ~8 h at ~4.7–5.3 min/arm (measure, don't assume — see §1) | -| H1 | **FALSIFIED** — run 7 ran on a fresh boot too (two independent clocks; `dev/315/315-H1-FALSIFIED.md`) | -| H2 | **the leading and only surviving candidate** — mid-wave preemption under memory pressure | -| H3/H4 | dead (the GPU-MV sizing defect is proven fixed on this box; `dev/315/315-cwsr-size-excluded-PROOF.md`) | - -Criterion status at the time of writing: **1 open** (needs the control), **2 open** (no repair landed -yet), **3 in progress** (campaign running), **4 not started**, **5 not started**. - ---- - -## 1. When the campaign lands - -```bash -ssh strixhalo -P=$(pgrep -f "315-evict-trace.sh 70" | head -1) -ps -o etime= -p "$P" # if empty, it finished -tail -30 ~/wt/315cwsr70-console.log # the last arms -``` - -**Check specifically whether faults appear LATE.** The recorded prediction: if faults begin in the -second half rather than the first, that supports H2 (pressure accumulates) and weakens any -"the setting fixed it" reading. Read the console as a sequence, not as a total — a late fault is a -different finding from an early one and must not be averaged away. - -Then score it: - -```bash -cd ~/wt -python3 315-rate.py 315cwsr70-console.log --det ~/wt/rt-det --tag 315cwsr70 -``` - -**Read the tool's output, not the console, for the criterion.** It applies §5/§6's counting rules and -reconciles against every arm's `detector.json`. Three things must be true: - -1. `FAULT 0` — and **no `MISSED BY CONSOLE` line.** That line would mean an arm faulted via HTTP 500 - where the console could not see it; if it appears, the campaign has failed and the arm is a fault. -2. **`clean` ≥ 60 counting only fix-cell (`l`) arms.** This is the trap: §6/§7 count **fix-cell arms**, - not total arms. A 70-pair run yields ~70 `l` arms plus ~23 `c` (control) arms; the `c` arms are - controls and **do not count toward the 60**. Verify the fix-cell count directly rather than - reading the total: - ```bash - grep -oE "pair [0-9]+ arm l: OK" ~/wt/315cwsr70-console.log | wc -l - ``` - Compare that to the tool's `clean` figure. If `clean` counts both cells, the fix-cell number is the - one that must reach 60. -3. `UNVERIFIED 0` — any unverified arm is clean from the console alone and is **not certified**. - -Expected shape of a good result: `clean ≥60`, `FAULT 0`, CI upper bound at or under ~5%. - -If it lands **short of 60 fix-cell arms** (e.g. some went OOR), do **not** prune arms and do **not** -change the criteria — §8 forbids both. Re-run only for a **recorded invalid** arm, or run more pairs. - -## 2. Then the control (this is criterion 1's decisive test, and H2's only faulting arm) - -`dev/315/315-campaign/315-cwsr-control.sh` removes the `cwsr_enable=0` drop-in, reboots, and prints the -command to run the identical campaign at `cwsr_enable=1`. It is now sized at **20 pairs** — at the -26% baseline that gives ~5.2 expected faults and `P(0) = 0.002`, so it is decisive for H1, and it -supplies the few faulting arms H2 needs. - -It installs the PID-capturing driver itself, which is **mandatory** — see §3. Run it only when the -campaign of record is finished, because the reboot ends it: - -```bash -# on ryzen, from ~/wt315-prereg -scp dev/315/315-campaign/315-cwsr-control.sh strixhalo:~/wt/ -ssh strixhalo 'cd ~/wt && TAG=315ctl1 PAIRS=20 bash 315-cwsr-control.sh' -``` - -**Reading key — this is the whole point of the control:** - -| control result | meaning | consequence | -|---|---|---| -| rate ≈ 26%, faults present | **H1 confirmed** — the `cwsr=0` signal was a fresh-boot artefact | criterion 1 collapses; re-scope honestly | -| rate ≈ 0%, no faults | **H1 dead, the `cwsr=0` signal is real** | proceed to H2 with the control's faulting arms | - -The control is the *only* run that can produce a faulting arm with a captured PID (see §3), so it is -not optional for H2 either. - -## 3. H2: the eviction comparison - -```bash -cd ~/wt -python3 315-evict-per-arm.py ~/wt/rt-det ~/wt/-console.log \ - --smi ~/wt/-evict/kfd_smi.tsv --tag -``` - -It bounds each arm by its captured PID's `PROC_START` → `PROC_END` (else the next arm's start) and -counts only that PID's `Q_EVICT` rows inside the window, then compares faulting vs clean means. - -**Why the PID capture is mandatory:** a finished arm's PID cannot be recovered afterwards, and its -post-arm lifetime contaminates the count — proved on run 7, where a fault arm's total was inflated -to 1,461 evictions because the server outlived its arm by 40 minutes. Run 7's data is therefore -unusable for H2 and only the control (and the campaign of record) can supply it. - -The tool refuses to invent a number when the PID is absent. Do not "fix" that by estimating. - -**Interpretation:** read the **`pre-fault` column**, not the window total. The tool counts evictions -up to the instant the fault was logged, because a fault cannot cause evictions that precede it — -whereas the window total is confounded (a faulting arm crashes, may be retried, and outlives its arm, -so its window can simply be longer). - -The fault time comes from the arm's own `server.log`, in `minutes.seconds.milliseconds.microseconds`: -`0.07.279.709 E common_sampler_sample: HRX returned NaN logits`. Both faces are matched (Face B via -`unparsed Content-only`), or a Face-B arm would look like it had no recoverable fault time. Validated -against run 7: all 3 faulting arms yield a time (81.181 s, 57.838 s, 7.280 s) and **0 of the 8 clean -arms do** — a parser that found faults in clean arms would fabricate the H2 effect. - -Compare the faulting arms' **pre-fault** mean against the **clean arms' full windows** — a clean arm -has no fault to truncate at, so that comparison is conservative and works against finding an effect. - -If faulting arms show more evictions pre-fault, that is consistent with H2 and with the residency -argument in §4. If they do not, H2's stated mechanism is wrong as written and must be revised rather -than defended. Note that run 7's faults landed **very early** — 7.3 s and 57.8 s into their arms — so -a pre-fault window can be short; check the parsed time rather than assuming a fault lands late. - -A supporting (not decisive) check already done: eviction **rates** are comparable across -`cwsr=0` and `cwsr=1` (3.80/s vs 4.19/s), so `cwsr=0` does not reduce pressure — the variable is -isolated to wave-preemption capability, not to memory pressure. Recorded in commit `6db2fd0`. - -## 4. Criterion 2: the forward-only change - -H2's mechanism points at an engine-side **residency** fix: place weights/KV/activations in -driver-owned/VRAM memory rather than evictable GTT/userptr/SVM, so a mid-wave preemption cannot -resume onto relocated memory. This is the same conclusion Engine V2's D5 reached independently. - -Rules that bind: - -* **forward-only**: the pinned revision it moves forward from must be an **ancestor** of the new one. - No reverts. Check with `git merge-base --is-ancestor `. -* **Do not land the coherence patch.** It is already falsified by §13; `tools/315-lever-check.py` - rejects a diff that reintroduces it. Run that tool on the diff before proposing anything. -* An **engine startup guard** (refuse to run above some GTT watermark) is a **workaround, not a - repair** — it hides the fault instead of removing it. If that is the only option, say so plainly - and re-scope rather than presenting it as a fix. - -The candidate patch branch `1bit/315-coherence-fwd` was **deleted** as falsified. There is currently -**no landed repair**, and criterion 2 is open. - -## 5. Criterion 4: no regression - -```bash -# both baselines are committed; compare against them, do not re-derive -dev/315/315-campaign/evidence/315-engine-ci-baseline.md -dev/315/315-campaign/evidence/315-hrx-tests-baseline.md -``` - -Two gates: (a) `1bit-MONSTER/engine` CI green; (b) the **dense-model control** and the -**`MUL_MAT_ID` op-test** unchanged vs baseline — byte-identical output where the baseline records it. - -Builds on the box: `~/wt/315instr3-build/bin` (control, pin `e44c9d01a`, `if (direct_host_binding)` -intact) and `~/wt/315fix-build/bin`. Both report `version: 10613 (e44c9d01a)`. The faulting baseline -pin is `third_party/llama.cpp = e44c9d01a`; current engine pin `0e255d774`. - -## 6. Criterion 5: KNOWN-ISSUES.md - -An entry gets a limitation note **only after a mechanism is named** — that is the issue's own rule, -so this is last. If the defect is genuinely gone, **retire** the entry instead of annotating it. -Include the reproduction and the arm-campaign evidence paths. - ---- - -## Standing constraints (do not violate) - -* RE material is **source material only** — never copy code, constants, kernels or layouts. Facts go - in docs; own implementations are written fresh. -* NPU runtime, NPU kernels, HIP kernels, the tuning DB and control code stay **private**. -* Every dossier ends with `Dossier: public | private | split`. -* Every behavioural claim cites an instruction/symbol address; dossiers ship a machine-checked bundle. -* Never prove absence with a truncated `strings | sort | head` — enumerate. -* Record unresolved questions with `record_unknown`. An honest unknown is a deliverable; a guess is a bug. -* **Never weaken a guard or the instrumentation to get a result.** If a guard blocks the conclusion, - the conclusion is not yet available. (This is how the detector/console gap in §1 item 1 was found — - the fix was to make the tool *stricter*, not to loosen the criterion.) -* Box-time budget: **≤12 h** of strixhalo GPU time. ~2 h was used before the campaign of record and - the campaign costs ~8 h, so a 20-pair control is the last thing that fits comfortably. Check the - budget before adding anything. -* The box is shared and thermally limited: NPU under `flock ~/npu.lock` (**the real box mutex** — - `box.lock` excludes nobody), builds under `~/lb/thermal-run.sh`, never kill a peer, report XRT - version / load / Tctl. - -## Traps already paid for — do not re-step - -1. **The console can't see an HTTP-500 fault.** Reconcile against `detector.json` (§1). A - console-only count can manufacture a passing campaign. -2. **§6 counts fix-cell arms, not total arms.** 70 pairs ≠ 70 counting arms for the criterion. -3. **A finished arm's PID is unrecoverable.** Install the PID capture before the run, not after. -4. **`~/npu.lock` is the real mutex.** `box.lock` excludes nobody. A dead holder wedges the box; - `NPU-LOCK-WEDGED-2026-10-12.md`. -5. **`kfd_smi_watch` needs root.** Without it, it writes a header-only file and **exits 0** — - silently. A header-only eviction stream looks like a valid empty result. -6. **Do not edit a frozen driver mid-campaign.** bash reads scripts incrementally; create a new file - instead (`315-paired-pidcap.sh` exists for exactly this reason). -7. **H1's falsification came from `boot_ns`, not a purpose-built control.** The control in §2 makes - it direct. Until then, treat H1 as strongly weakened but formally open. - ---- - -## Appendix — measured pace and the budget verdict (2026-10-12 17:22 box time) - -Measured from the arm `server.pid` mtimes, i.e. from when each arm's server actually started: - -``` -16:55:18 p1c 16:59:45 p1l 17:04:11 p2l -17:08:37 p3l 17:13:02 p4c 17:17:27 p4l -``` - -Interval between consecutive arm starts: **4:25, 4:26, 4:26, 4:27, 4:25** — a spread of **2 seconds** -across five intervals. So the pace is not an estimate; it is **4.43 min/arm** with almost no variance, -which also means the arm time is set by the fixed request sequence and not by contention. - -| | | -|---|---| -| arms started | 6 | -| measured pace | **4.43 min/arm** | -| projected total (93 arms) | **6.87 h** | -| elapsed so far | 0.42 h | -| projected completion | ~**23:45 box time** | - -**Budget verdict — both stages fit.** ~2 h used before the campaign + 6.87 h campaign + ~2.2 h for a -20-pair control (30 arms × 4.43) ≈ **11.1 h of the 12 h ceiling.** So the decisive control in §2 does -fit after the campaign, which was the open question in Update 4. Margin is ~0.9 h: enough, but not -enough to also add a stage without re-checking. - -Note the `c` arms appear only at pairs 1 and 4 so far, so the pairing is sparse in controls as -designed — 70 pairs should yield ~70 `l` and ~20 `c`, comfortably clearing the 60 **fix-cell** bar -with ~10 arms of slack for any that fall out of regime. diff --git a/dev/315/315-H1-FALSIFIED.md b/dev/315/315-H1-FALSIFIED.md deleted file mode 100644 index c9a0bdfc6..000000000 --- a/dev/315/315-H1-FALSIFIED.md +++ /dev/null @@ -1,84 +0,0 @@ - -# H1 (the fresh-boot confound) is FALSIFIED — and it was my own error - -**Date** 2026-10-12, late. This reverses the hypothesis ranking I recorded earlier in -`315-cwsr-refined-hypothesis.md`. Read this first. - -## What I claimed, and why it was wrong - -I recorded that the CWSR signal was confounded because: - -> run 7 (`cwsr=1`, 3/11 faults) ran at **~32.5 h uptime**; the `cwsr=0` campaign started **692 s** -> after boot. The comparison is therefore not only CWSR on vs off but also 32-hour-old box vs -> 11-minute-old box. - -**The 32.5 h figure was wrong.** I derived it from an uptime reading taken at 12:37 on 10-12 -("up 1 day, 7:04") and did not account for the **clean reboot at 14:03:53** that happened between -that reading and run 7. - -## The correction, from two independent clocks - -The KFD SMI stream records `boot_ns` (CLOCK_BOOTTIME, in nanoseconds), which — combined with any -known wall-clock anchor for the same event — recovers the boot time exactly: - -| cross-check | arithmetic | implied boot | -|---|---|---| -| pid 111371's `PROC_START` `boot_ns` = 1,180,845,874,852 ns = 1180.8 s, against `ps`'s start time for that same process (Mon Oct 12 **14:23:33**) | 14:23:33 − 1180.8 s | **14:03:52** | -| run 7's campaign start (its run.log: **14:09:42**) minus its stream's first event (`boot_ns` = 352.4 s) | 14:09:42 − 352.4 s | **14:03:49** | -| the user's clean reboot (measured directly: `uptime -s` = 14:03:53) | — | **14:03:53** | - -Three independent numbers agreeing to within 4 seconds. The earlier method failed because I used a -*completed* stream's file mtime (its last write) instead of its start time; anchoring on the -campaign's own start timestamp instead fixes it. - -## What this means - -| run | config | box age when its arms ran | result | -|---|---|---|---| -| **run 7** | `cwsr_enable=1` | **~6–40 min after a fresh boot** | **3 faults / 11 arms** | -| campaign `315cwsr60` | `cwsr_enable=0` | ~53 min – 7 h after a fresh boot | 0 faults / 9 arms so far | - -**Both runs were on fresh boots, on the same day, within hours of each other.** The "32-hour-old -box vs 11-minute-old box" contrast does not exist. What remains is a genuine same-day, -same-hardware, same-harness comparison differing in `cwsr_enable` — and the `cwsr=1` arm of it -faulted 3 times in 11 arms while the `cwsr=0` arm has not faulted in 9. - -So **H1 is largely falsified**: a fresh boot does *not* suppress the fault, because run 7 was -freshly booted and faulted anyway. That was the single strongest alternative explanation for the -CWSR signal, and it is now gone. - -## Consequences, stated plainly - -1. **The CWSR signal is strengthened, not weakened.** It is no longer explainable by boot - freshness. The remaining alternative is H2 (a mid-wave-preemption mechanism), which has no - upstream documentation (`315-cwsr-refined-hypothesis.md`) but also no competing explanation - left. -2. **The H1 control is now much less urgent** — it would still be good practice, but the specific - confound it was designed to rule out has been ruled out by the boot-time reconstruction - instead. -3. **My earlier ranking was wrong and I said so.** I had written that "H1 is the leading - explanation, not H2" and that the signal was "probably a boot artifact". Both statements were - based on a mis-derived uptime, and both are withdrawn here. -4. **The statistics are unchanged** and still short of the bar: 9 clean arms gives a rule-of-three - upper bound of 33%, which does not exclude the 26–27% baseline. Removing H1 removes an - *explanation*, not the need for ≥60 arms. - -**Lesson worth keeping:** I inferred a 32.5 h uptime from a reading taken before an intervening -reboot, and treated it as a quantitative confound for several turns. The SMI stream carried the -true value the whole time (`boot_ns`), and two independent anchors agreed on it within seconds once -I actually computed it. Derive, do not remember. diff --git a/dev/315/315-STATUS.md b/dev/315/315-STATUS.md deleted file mode 100644 index 82929aa30..000000000 --- a/dev/315/315-STATUS.md +++ /dev/null @@ -1,1984 +0,0 @@ - -# engine#315 — STATUS / resume-here - -**Updated** 2026-10-15. Single entry point. Everything else is referenced from here. -**Latest:** the issue is closed out as a **bounded negative** — §"Session of 2026-10-15: closed as a bounded -negative" below is the resume-here state, and it supersedes the criterion tables above it. - -## Task state - -| task | state | -|---|---| -| task-1..3 | complete (earlier sessions) | -| task-4 | completed against a FALSIFIED candidate; retracted in the record — read `dev/315/315-campaign/evidence/ISSUE-COMMENT-5-2026-10-10.md` addendum | -| task-5 | complete — falsified-lever table rebuilt from the run consoles | -| task-6 | complete — closed families cited at source with verified file:line | -| task-7 | complete — the preemption-window lead recorded as a lead, with its two unbuilt instruments | -| **task-8** | **complete** — run 7 gave 3 in-regime faults across both faces with the Face-B arm's PID correlated (323 Q_EVICT / 357 Q_RESTORE) | -| **task-9** | **gated — NOT started.** Conditional on a *named mechanism*, which does not exist. | - -## The one live lead - -`amdgpu.cwsr_enable=0` vs `1` (CWSR = compute wave save/restore on queue preemption). - -| | `cwsr=1` | `cwsr=0` | -|---|---|---| -| in-regime arms | 11 (run 7) | 11 | -| faults | **3** (Face A ×2, Face B ×1) | **0** | -| regime | 97–98 GiB | 97 GiB | - -First intervention in this issue's history to move the rate in the predicted direction. -Statistics at n=11: binomial p=0.030 (suggestive); Fisher p=0.107 (not significant); -rule-of-three bound 27.3% does **not** exclude the 26–27% multi-session baseline. - -## What is FALSIFIED (do not re-propose) - -- **area size, both halves.** libhsakmt returns the correct values for gfx1151 — VGPR - `0x60000` (384 KB = 1.5×, the `rocm-systems#1807` fix IS present) and SGPR `0x4000` (16 KB). - Verified at runtime by linking `queues.c.o` from `libhsakmt.a` and calling the exported - functions. Both are plumbed into the allocation and topology paths. -- coherence, consumer ordering, AQL barriers, `-fa off`, transient reuse, PM4, MoE routing - tables, the routing-tensor and by-name locator routes, arena knobs. -- the "half-applied patch pair" story: the kernel half is in, and the userspace half is too. - -## The confound that must be settled FIRST - -The reboot that set `cwsr_enable=0` **also** gave a fresh boot: unfragmented device memory, -no prior campaigns, cool box. Run 7 ran on a long-uptime box. So a clean `cwsr=0` result is -**not yet attributable to CWSR**. This issue has already been burned once by exactly this -class of error ("the rate followed memory-pressure settings"). - -**H1 CONTROL** — fresh boot with `cwsr_enable=1`, identical campaign. Scripted and ready: -`dev/315/315-campaign/315-cwsr-control.sh`. Sized to **20 arms, not 60**: at the 26% baseline, -20 arms give ~5.2 expected faults and P(0 faults) = 0.002, which is decisive; 60 would cost -~3 extra hours for no added discriminating power. - -Reading key: **~26% → H1 dead, the `cwsr=0` signal is real. ~0% → H1 confirmed, the signal -was a boot effect.** - -## Ordered next steps - -1. **finish the running campaign** (`~/wt/315cwsr60-console.log`, tag `315cwsr60`, 48 pairs, - `cwsr=0`). It establishes the RATE, not the cause. -2. **run the H1 control** (reboot required; kills any running campaign). -3. only if H1 survives, **H2 instrumentation** — count CWSR preemptions per arm and correlate - with faults. Doing H2 before H1 risks carefully measuring a boot effect. -4. then the fix, the regression comparison, and the KNOWN-ISSUES update. - -## The forward-only fix is UNRESOLVED - -With size excluded, "bump the HSA runtime" is **not** the fix. The only remaining candidate is -an engine-side **startup guard** that detects the bad configuration and refuses rather than -silently faulting — that is a **workaround, not a repair**. Do not present it as solving the -issue while the underlying defect is unidentified. - -The engine has no HSA pin to move: `IREE_HAL_AMDGPU_LIBHSA_PATH` points at whatever the ROCm -SDK ships. - -## Key paths - -| | | -|---|---| -| live treatment log | strixhalo `~/wt/315-cwsr-test/phase1-treatment.txt` (255+ lines) | -| campaign console | strixhalo `~/wt/315cwsr60-console.log` | -| eviction stream | strixhalo `~/wt/315cwsr60-evict/kfd_smi.tsv` | -| run 7 (the faulting baseline) | `~/wt/315evict7-console.log`, `~/wt/315evict7-evict/correlated-p3l.tsv` | -| harness | `dev/315/315-campaign/315-evict-trace.sh` (+ `tools/arm-pid.py`, `tools/lever-census.py`) | -| harness defects found+fixed | `dev/315/315-campaign/evidence/315-evict-trace-instrument.md` §6a–§6h | -| regression baselines | `dev/315/315-campaign/evidence/315-hrx-tests-baseline.md`, `315-engine-ci-baseline.md` | -| box-wide lock finding | `dev/315/315-campaign/evidence/NPU-LOCK-WEDGED-2026-10-12.md` | - -## Honest bottom line - -**task-8 is genuinely done** with real, correlated evidence. **task-9 is blocked on a named -mechanism that does not exist yet** — there is a strong signal (CWSR) and a falsified -explanation for it, and the confound that must be settled before the signal is even -attributable. The objective's criteria 1 and 2 are **not met**, and a clean 60-arm run would -satisfy criterion 3 while leaving 1 and 2 open. - ---- - -# UPDATE (same day, later) — the size question is now PROVEN closed, and H1 is quantified - -## Proven: the known gfx1151 CWSR size defect is FIXED here - -AMD's own SRU cover letter (Mario Limonciello) names the defect and its two kernel halves -(`bump minimum vgpr size for gfx1151` in `kfd_queue.c`, `Export the cwsr_size and -ctl_stack_size` in `kfd_topology.c`) plus the matching ROCm commit. The arithmetic closes: - -| vgpr | per-CU (vgpr+sgpr+lds+hwreg) | × 40 CUs | node requires 19,185,664 | -|---|---|---|---| -| `0x40000` unfixed | `0x55000` | 13,926,400 | **5,259,264 short** | -| `0x60000` fixed | `0x75000` | 19,169,280 | off by one SGPR unit | - -Independent cross-check: a deployment report's real broken figure is **13,942,784**; my unfixed -total is 13,926,400 **+ 16,384** = exactly that. Three independent agreements ⇒ proof. - -All five halves are present on our box. **So the documented defect cannot be our cause.** See -`315-cwsr-size-excluded-PROOF.md`. H3 and H4 are dead (both were sizing variants). - -## H2 refined: mid-wave preemption, not spill correctness - -The kernel's parameter doc: *"CWSR allows the GPU to preempt shader execution in the middle of -a compute wave... Setting 0 disables it."* So `cwsr_enable=0` removes mid-wave preemption -entirely. Refined mechanism: under pressure a wave is preempted mid-execution, its buffers are -evicted/migrated, and it resumes reading moved or recycled memory → Face A or B by content. -This does **not** depend on the (now-fixed) size defect, and it explains the GTT dependence, why -every prior lever returned null, the whole-batch layer-0 cascade, and the intermittency. - -## H1 quantified **and a prediction recorded before the result** - -Run 7 (`cwsr=1`, 3/11 faults) ran at **~32.5 h uptime**. The `cwsr=0` campaign started **692 s -after boot**. The campaign runs ~6 h → ends at ~7 h post-boot, so: - -- **faults appearing LATE** in the campaign → supports H1 (uptime effect); -- **still 0 across 11 min → 7 h uptime** → weakens H1, leaves H2 standing. - -Recorded in advance so it cannot be read post hoc. - -## Current running state - -`~/wt/315cwsr60-console.log`, tag `315cwsr60`, 48 pairs (~66 arms), `cwsr=0`, ~5.5 min/arm. -**Next actions after it completes, in order:** (1) read its rate; (2) **H1 control** — fresh -boot, `cwsr=1`, 20 arms (`dev/315/315-campaign/315-cwsr-control.sh`); (3) only then H2 -instrumentation. - ---- - -# UPDATE 2 (late same day) — H1 FALSIFIED; the CWSR signal stands on its own - -**Read `315-H1-FALSIFIED.md` first.** My recorded "fresh-boot confound" was an arithmetic error -of mine: I derived run 7's uptime (32.5 h) from a reading taken *before* the clean reboot at -14:03:53 that preceded run 7. Corrected from the SMI stream's `boot_ns`, two independent anchors -agree within 4 s that run 7 booted at **14:03:5x**. - -| run | config | box age during its arms | result | -|---|---|---|---| -| **run 7** | `cwsr_enable=1` | **~6–40 min after a fresh boot** | **3 faults / 11 arms** | -| `315cwsr60` | `cwsr_enable=0` | ~53 min – 7 h after a fresh boot | 0 faults / 9 arms so far | - -Both were fresh boots, same day, same hardware, same harness. **So a fresh boot does not suppress -the fault — run 7 was freshly booted and faulted anyway.** The strongest alternative explanation -for the CWSR signal is gone. - -## Revised hypothesis ranking - -| | | state | -|---|---|---| -| ~~H1~~ | fresh-boot effect | **FALSIFIED** — run 7 was fresh-booted and faulted 3/11 | -| **H2** | **mid-wave preemption under memory pressure** | **the leading and only surviving candidate** | -| ~~H3~~ | partial kernel-side fix | dead — a sizing variant | -| ~~H4~~ | firmware save/restore list mis-sized | dead — a sizing variant | - -The mechanism as it now stands, and it is falsifiable: - -> Under memory pressure a compute wave is preempted mid-execution (CWSR); the buffers that wave -> is reading are evicted/migrated by TTM/amdgpu; on resume it reads relocated or recycled memory, -> and the damaged values surface as Face A or Face B depending on what those bytes now hold. -> Disabling CWSR removes mid-wave preemption and closes the window. - -**Discriminating experiment:** `amdgpu.cwsr_enable` 1 vs 0 — same harness, same gate, same day. -**Alternatives ruled out:** every prior lever (coherence, ordering, AQL barriers, `-fa off`, -transient reuse, PM4, MoE routing), the area-size explanation (proved fixed by AMD's own patch set -with exact arithmetic), and now the fresh-boot confound. - -## What is still missing for criterion 1 - -**n.** 9 clean arms gives a rule-of-three upper bound of 33%, which does **not** exclude the -26–27% baseline. Removing H1 removes an *explanation*; it does not substitute for the ≥60 arms the -objective requires. At 60 clean arms the bound is 5% and the criterion's power statement holds. - -## Unchanged - -Criterion 2 (a forward-only fix) still has nothing to repair: the documented gfx1151 CWSR defect is -**proved fixed here**, and H2's remedy would be a driver/firmware change, not an engine change. The -best engine-side option remains a configuration guard — a **workaround**, not a repair. - -Side quests completed while the campaign ran: a rate/CI/power tool (`315-rate.py`), the per-arm -eviction comparison tool (`315-evict-per-arm.py`, the H2 test), PID-capture wiring into the H1 -control, the control's guard verified by dry-run, campaign QA (9/9 arms valid and in regime), and -three failed attempts at a retroactive H1 test — each of which is recorded as a negative with its -reason. - ---- - -# UPDATE 3 — I under-sized the running campaign against the pre-registration's own target - -**This prevents a false claim, so it goes at the top of the record.** - -## The discrepancy - -| source | says | -|---|---| -| the objective (user-provided) | "≥60 arms, every arm inside the declared regime gate, zero faults across both faces" | -| `dev/315/315-preregistration.md` §6 | "Target: **≥60 in-regime clean arms in the fix cell**, with paired control arms running in the same session… Hard cap: **80 scheduled pairs** (160 arms)" | -| `dev/315/315-preregistration.md` §7 | "The campaign passes only if, **in the fix cell**, ≥60 in-regime arms are clean…" | - -The pre-registration counts **fix-cell arms**, not total arms. And the driver emits exactly **one -fix-cell (`l`) arm per pair** — measured, not assumed: the 8-pair run produced 8 `l` arms and 3 -`c` arms (11 total). - -So the running campaign, launched with **48 pairs**, will produce **~48 fix-cell arms** — **twelve -short of the pre-registered target**, even though its *total* arm count (~64) would satisfy the -objective's looser wording. - -## What this means - -1. **The running campaign cannot, by itself, satisfy §7.** It can establish the rate (and 9 clean - arms so far is consistent with zero), but it must not be reported as meeting the ≥60 fix-cell - bar. -2. **A follow-on is required.** Either: - - a further campaign of **≥60 pairs** (the fix-cell count is what must reach 60), i.e. **~13 - additional pairs** beyond the 48 already scheduled would *not* suffice unless pooled; the - clean route is a fresh run at ≥60 pairs (~6.5 h); or - - pooling the `cwsr=0` arms across runs (`315cwsr0` gave 11, the campaign ~48, total ~59) — - but pooling is **not provided for** in the pre-registration, which says "no arm pruning, no - criterion changes" (§8) and frames the target as a single campaign with its paired controls. - Pooling would need to be declared and justified, not assumed. -3. **The honest reading is that §7 needs a fresh ≥60-pair run**, and the current campaign is a - useful but insufficient down-payment on it. - -## How the mistake happened, stated plainly - -I read the objective's "≥60 arms" and sized the run for ~64 **total** arms, never checking the -pre-registration's own definition of the target, which counts only the fix cell. I had read §6 -earlier in the session (I quoted the "≥60 in-regime arms" bar in the task-8 work) and still sized -by total arms. The check takes one `grep` and I did not run it until now. - -Recorded so the next run is sized from §6 — **60 fix-cell arms means ≥60 pairs** — rather than from -the objective's looser phrasing. - ---- - -# UPDATE 4 — campaign of record relaunched at compliant size (70 pairs) - -The 48-pair run was stopped at 10 scored arms (all clean) once I realised it could not reach the -pre-registration's ≥60 **fix-cell** arms. It is recorded as a **mis-sized pilot, not the campaign -of record** — no result was discarded and none was inconvenient (10/10 clean); the run simply could -not meet §6 as sized. - -## The campaign of record - -``` -tag : 315cwsr70 -pairs : 70 -> ~70 fix-cell ('l') arms (>=60 required by section 6; within the 80-pair cap) -config : amdgpu.cwsr_enable=0 (unchanged since the treatment reboot) -gates : box lock -> npu lock -> GPU quiet -> watcher (all four passed 16:54:11) -driver : with the PID capture installed, so its arms carry $O/server.pid and its data can - support the H2 faulting-vs-clean eviction comparison -``` - -Expected duration ~7–8 h at the observed ~5 min/arm. - -## Why relaunching rather than extending - -Running 48 pairs and then another 60+ would cost ~11.9 h to reach one compliant campaign. -Relaunching once at 70 pairs costs ~7.8 h for the same thing. The stopped run's 10 clean arms are -not counted toward the target (pooling across runs is not provided for in the pre-registration), -so nothing of value was gained by letting it continue. - -## Box-time budget - -| | arms | ~GPU hours | -|---|---|---| -| earlier sessions (10-09…10-11) | 260 | — (predate this goal turn) | -| this goal turn, before the relaunch | ~24 | ~2 h | -| this campaign of record | ~93 | ~7.8 h | -| **total for this turn** | | **~10 h of the stated ≤12 h** | - -Tight but inside. If the campaign needs a follow-on, the budget will be the binding constraint -before anything else. - -## What to check when it lands - -1. `tools/315-rate.py ` — the rate, exact CI and the power statement, applying §5/§6's - counting rules from the driver's own lines; -2. **fix-cell count specifically** — must be ≥60 clean, not merely ≥60 total; -3. `tools/315-evict-per-arm.py` — the H2 comparison, now possible because the PIDs are captured; -4. a fresh-boot `cwsr=1` control if the rate holds, since H1's falsification was inferred from - `boot_ns` rather than from a purpose-built control. - ---- - -# UPDATE 5 — side quests while the campaign runs (all committed and pushed) - -Run while waiting on `315cwsr70`, per "unblock by doing side quests". Five of the six are -correctness fixes to the **analysis and control** tooling, i.e. to the parts that decide whether the -campaign's result is believed — not to the frozen experiment itself. - -**1. The console line cannot certify zero faults on its own — fixed.** (commit `43caadc`) Criterion 3 -requires zero faults on **both faces**, but the driver's per-arm console line has no `http=` field, -and `detector.json` carries `http_failures` and `transport_failures` that the console cannot see. A -Face-B fault surfacing as HTTP 500 would therefore leave **no trace in the console** and a -console-only count would report that arm **clean** — a false negative on the one criterion the -campaign exists to certify. `315-rate.py` now reads every arm's `detector.json` and treats the -**detector as authoritative**: an arm the console passes but the detector fails is counted a **fault** -(`MISSED BY CONSOLE`), and an arm with no usable detector is **UNVERIFIED**, never silently clean. -Proven on a fixture where it changes the answer (`0/3 = 0%` without it → `1/2 = 50%` with it, the -difference being a real HTTP-500 fault). Re-checked on run 7: still 3/11, no missed arms, so it is a -guard and not a revision. - -**2. The endgame runbook.** (`f3ac0a5`) `dev/315/315-ENDOGAME.md` — exact commands, the decision table -for the control, and the standing constraints plus seven traps already paid for. Written because the -session that started the campaign will not outlast it. - -**3. The H1 control was hardened and dry-run safe.** (`ad0a494`) Sized 48 → 20 pairs (at ~26%, 20 -gives ~5.2 expected faults and `P(0)=0.002`; 48 would cost ~3 h for no added power). It now -**verifies** the PID-capturing driver actually landed instead of trusting the copy — a run without the -capture silently produces H2-unusable data, and that is invisible until the analysis, after the GPU -hours are spent. Dry-run with `sudo`/`pgrep` neutered: drop-in PRESENT, `cwsr_enable` 0, campaign -alive, uptime unchanged. Its abort guards fire before any `sudo`. - -**4. Pace measured, budget verdict: both stages fit.** (`80032c8`) From arm `server.pid` mtimes: -intervals **4:25, 4:26, 4:26, 4:27, 4:25** — a 2-second spread, so **4.43 min/arm** is not an estimate. -93 arms ≈ 6.87 h, finishing ~23:45. With ~2 h used before and ~2.2 h for the 20-pair control, the -total is **~11.1 h of the 12 h ceiling**. The decisive control does fit; margin is ~0.9 h. - -**5. The H2 test was confounded — now it is a causal test.** (`76b7bdb`) I checked my own test and -found it could not distinguish *more evictions caused the fault* from *the fault caused more -evictions*: a faulting arm crashes, may be retried, and outlives its arm, so its window can simply be -longer. But **the fault is timestamped** (`0.07.279.709 E ... HRX returned NaN logits`, in -minutes.seconds.milliseconds.microseconds — confirmed against that arm's `req=1`, not assumed). So -the tool now also counts evictions **up to the fault instant**, which the fault cannot have caused. -Both faces are matched (Face B via `unparsed Content-only`), validated as 3/3 faults timed and **0 of -8 clean arms falsely flagged** — a parser that found faults in clean arms would fabricate the effect. - -**6. Two of run 7's three faults were Face A *followed by a full server crash*** — the same commit. -p7c and p7l logged the NaN line and died (logs end in a gdb backtrace), which is why their detectors -recorded `RemoteDisconnected`: **the transport failure is downstream of the NaN, not a separate mode.** -The console's `FaceA` label is right. Run 7's faults are two NaN crashes (at 7.3 s and 57.8 s into -their arms) and one Face-B parse failure (81.2 s, `req=20`). - -## Where that leaves the criteria - -Unchanged: **1 open** (control), **2 open** (no repair landed), **3 in progress**, **4/5 not started**. -But criterion 3 is now much harder to pass *falsely* — the detector reconciliation and the fix-cell -count both stand between a clean-looking console and a claimed pass. That is the point. - ---- - -# UPDATE 6 (2026-10-13 ~05:20) — H2 is re-carried as H2' (resumption, not interruption), and the -# control that tests H1 is now running - -Read this with `315-mechanism.md` §59, which is the previous frontier. Nothing above is retracted; -what follows supersedes H2's *carrier* only. - -## The control is RUNNING (this is criterion 1's decisive test) - - tag : 315ctl1 config: amdgpu.cwsr_enable=1 pairs: 20 - boot : 2026-10-13 04:53:19 (the control rebooted; the cwsr drop-in was removed first) - verified after boot: cwsr_enable=1, drop-in gone, 0 cwsr entries in grub.cfg - launch : 04:54:37, holding BOTH locks (box.lock + npu.lock) - watcher : kfd_smi_watch pid 8916, -t 10800 (180 min) vs a ~135 min run -> ~45 min slack - progress at 05:41: 8/20 pairs, 1 FAULT (pair 8 l, FACE B, cwsr=1, gtt_start=97G) - -- the control HAS now faulted, which is the H1-confirming direction - -- and H2' failed its first quantitative test on that same arm (below) - detached: ppid=1, so it completes regardless of any agent session - - READING KEY ~26% faults -> H1 CONFIRMED (the cwsr=0 signal was a boot artefact; criterion 1 - collapses and the issue must be re-scoped honestly) - ~0% faults -> H1 DEAD (the signal is real; H2' is the sole survivor) - score with: ~/wt/315-rate.py 315ctl1-console.log --det ~/wt/rt-det --tag 315ctl1 - ROLLBACK : sudo cp ~/work/amd_cwsr.cfg.backup /etc/default/grub.d/amd_cwsr.cfg && - sudo update-grub && sudo systemctl reboot (md5 a8b25738...) - -Note on the H1 falsification above: it was inferred from `boot_ns`, and this control is the -purpose-built confirmation that UPDATE 5 item 6 and UPDATE 3's final line both asked for. If the -control faults at ~26%, that inference was wrong and the cwsr signal was a boot effect after all. - -## H2' — the carrier that survives the DRAIN refutation - -H2 as written said cwsr=0 "removes mid-wave preemption". That is refuted: -`reference/amd/gpu/context-save.md:304` says the KFD update path **first removes the queue from -hardware** in BOTH branches, so the SAVE-vs-DRAIN choice (which cwsr_enable selects, GFX11 mapping -SAVE to DRAIN_PIPE) governs only the hardware request, not whether a wave is unhooked. - -H2' keeps the empirical signal and moves the danger from interruption to **resumption**: - -> CWSR preserves a wave's registers and LDS and remaps the queue so the wave resumes. It does NOT -> own or copy the executable, arguments, payloads or private scratch the wave will re-read - -> `context-save.md:6` ("Saved waves still depend on their executable, arguments, payloads and -> private scratch") and `:299` ("not an application checkpoint that OWNS COPIES of ... external -> buffers or private scratch"), with `:330` assigning those to "every unfinished producer/consumer -> that CAN RESUME". So under pressure a migration/eviction between suspension and resumption leaves -> the resumed wave reading stale or recycled buffers, and what those bytes now hold decides Face A -> (NaN/poison) vs Face B (finite garbage tokenised as invalid UTF-8). - -Why the refutation ENABLES rather than kills it: cwsr=1 -> SAVE -> resumes -> re-reads; cwsr=0 -> -DRAIN -> abandoned -> never re-reads. The window is resumption, not interruption. - -## The discriminating experiment is pre-registered and needs NO rebuild - - P1 HYPOTHESIS at cwsr=1, denying buffer movement suppresses the fault - P1 PREDICTION cwsr=1 + direct coherent bindings -> ~0 faults; cwsr=1 default -> ~26% - P1 REFUTATION ~26% faults persist with movement denied -> H2' is dead - - arm A (movement allowed): 315-evict-trace.sh 315p1a - arm B (movement denied) : EXTRA_ENV="GGML_HRX_USE_UNIFIED_MEMORY=1" \ - 315-evict-trace.sh 315p1b - -Both arms must be SEPARATE runs: `EXTRA_ENV` is threaded per arm and is NOT cell-selective -(`315-paired-pidcap.sh:66-67`), so a single paired run cannot express P1. With FIX_BIN == CTRL_BIN -the c/l split becomes a same-build A/B repeat, and run A doubles as the movement-allowed control. -The knob is real and per-arm: `ggml-hrx.cpp:1173` reads `GGML_HRX_USE_UNIFIED_MEMORY` at runtime -into `use_direct_host_bindings`, `:304` gates on it, `:310` adds `HRX_MEMORY_TYPE_HOST_COHERENT`, -and the comment at `:306-307` distinguishes pinned transfer memory from coherent direct bindings. - -**Caveat that must appear in the write-up:** `HOST_COHERENT` is falsified AS A FIX (315accept: 9 -clean + 1 fault / 10). P1 does not re-propose it as a fix - it uses it as a MECHANISM PROBE. A -clean arm B is evidence about the mechanism, not a landed repair. - -## THE FIRST RESULT IS IN (05:42) - read this before the sections below - -**The control faulted.** `pair 8 arm l: FAULT nan=0 req=22 parse=0 utf8=0` at `gtt_start=97G`, -`detector.json` = `http_failures:1, first_failure:HTTP 500, truncated:true`, server log = -`common_chat_peg_parse: unparsed Content-only output:` + symbol soup with invalid bytes. That is -**Face B**, zero NaN, `identical_distinct:1`. The control runs `cwsr_enable=1`, so **a fresh-booted -box at cwsr=1 does fault** - the H1-confirming direction, and it makes the cwsr=0 signal (0/70) look -more like a boot/state artefact. - -**And H2' failed its first quantitative test on that same arm.** The first faulting arm with per-arm -PID attribution shows **203** evictions (pre-fault, fault at 91.4s) against a clean-arm mean of -**252.4** (n=10, median 275). The tool's verdict: "NOT supportive (faulting <= 1.5x clean)". The -pre-fault figure is the causal direction (a fault cannot cause evictions preceding it) and it is -compared against clean arms' full windows - the conservative comparison. H2' predicted faults -concentrate on heavily-evicted arms; this one is on the LOW side of the clean distribution. - -Bounds, so neither result is over-read: n=1 faulting arm cannot establish a distribution; the clean -mean is still moving (234.7 at n=7, 252.4 at n=10); and P3's cheap refutation form does NOT fire (the -arm has 203 evictions, so suspension-plus-eviction did occur - what fails is the concentration -prediction). 1 fault in 8 pairs is ordinary against a 26% baseline (expected 2.1). So: **not "H2' is -refuted", not "H1 is confirmed".** The fault occurred at cwsr=1 and shows no eviction excess, which -shifts weight to H1 and leaves P1 as the surviving discriminator. - -## H2' is testable for the first time on this control - -Every previous run failed H2's prerequisite (a per-arm PID): run 7's PID was contaminated, -`315disc1` had 0 `server.pid`, `315cwsr70` had 0 faults. This control satisfies the whole chain: -5/5 arms on `server.pid`, 4/5 `detector.json`, SMI stream alive, `PROC_START` resolving per arm pid, -and 3695 `Q_EVICT` rows already present. All five arms show BOTH `Q_EVICT` and `Q_RESTORE` -attributable to their own pid, so suspension-plus-resumption is directly observable. - - H2 DENOMINATOR ON RECORD (clean in-regime arms, n=4, at 97G): - min=160 median=262 max=293 mean=227.8 pre-fault evictions - P2 faulting arms should exceed that - P3 a faulting arm with ZERO own-pid evictions refutes H2' outright - read with: ~/wt/315-evict-per-arm.py ~/wt/rt-det 315ctl1-console.log \ - --smi ~/wt/315ctl1-evict/kfd_smi.tsv --tag 315ctl1 - -## Also settled on 2026-10-13 (negatives, so they are not re-explored) - -- **Arena-knob / coherence dead end re-confirmed.** `~/wt/315-coherence-fix.patch` has criterion 2's - exact forward-only shape but is falsified: lever row 1 of 10 in KNOWN-ISSUES §"Levers tested and - excluded", and `tools/315-lever-check.py` rejects a diff adding `HOST_COHERENT` with exit 1. -- **The decode-split FA kernel is NOT the live mechanism.** `bounded_key_value_token_count` appears - 24 times in the faulting pin (26330cc20) and 24 times in the current pin (0e255d7) - identical - so - the token-bound masking the "rows past the sequence end hold whatever an earlier request left there" - comment describes was already in place when the fault reproduced. That comment is the rationale for - the #314 fix, not evidence of a live defect. (Hypotheses disproved by reading rather than asserted: - `masked_value_stage` is per-subgroup offset so its subgroup barrier is sufficient; `probability_stage` - is workgroup-wide but a workgroup barrier at loom:316 orders it; workgroup 256 / subgroup 64 = 4.) -- **The access-mode defect does not apply to this model** (`b854af3`): GLM-4.7-Flash never dispatches - `q6_k_token1_shortlist`, so the one-word manifest fix is correct on its own merits but is not this - issue's mechanism. -- **Ancestry chain confirmed** (the fork clone was SHALLOW; it has been unshallowed, so the - forward-only proof and the bisect are now checkable locally): - `f2099e9b7` -> `26330cc20` -> `0e255d7`, all true ancestor links. The fix must move forward from - `0e255d7` itself; adopting `f2099e9b7` would be a revert the contract excludes. -- **The only change to `ggml-hrx.cpp` between the faulting pin and the current pin** removes a - `return false` for IQ3_S/IQ4_NL GET_ROWS (PR #100 enables an op the faulting pin refused) - - unrelated to the reuse boundary, and evidence the faulting configuration is otherwise carried - forward unchanged. - -## Criteria after this update - -**1 open** (the control is running and H2' is a candidate, not yet a finding - it needs a faulting -arm). **2 open** (no unfalsified forward-only candidate; `task-9` stays CONDITIONAL on a named -mechanism). **3 MET** (315cwsr70: 70/70 fix-cell arms, 0 faults, CI 3.14%, all in-regime). **4, 5 not -started.** - -Honest bounds on H2': the doc establishes the BORROWING (`:6`, `:299`, `:330`) and the -resumes-vs-abandoned split (`:4`, `:304`); that a resumed wave re-fetches by address walk is an -inference consistent with those lines, not a quotation. TTM is named from the lever analysis, not -from the doc. - ---- - -# UPDATE 7 (2026-10-13 ~07:30) — the H1 control RAN AND FAULTED, H2' IS FALSIFIED, and P1 is withdrawn - -This supersedes UPDATE 6's "control running" state. Read this first; the sections above are the -history that produced it. - -## 1. The H1 control faulted. H1 is CONFIRMED. - -`315ctl1`, `cwsr_enable=1`, 20 pairs, fresh boot 04:53:19 with the cwsr drop-in removed (verified -gone, 0 cwsr entries in grub.cfg), finished 06:32. - - 27 arms parsed all in-regime 0 OOR 0 INVALID 25 clean 2 FAULT - rate 2/27 = 7.41% BOTH FACES represented - - pair 8 arm l: FACE B - detector http_failures=1, first_failure HTTP 500; server log shows - the parser rejecting symbol soup with invalid bytes - pair 16 arm c: FACE A - nan=1 req=1; detector transport_failures=1 RemoteDisconnected - -The two faults are on OPPOSITE CELLS and OPPOSITE FACES, so this is not one binary's property. - -The reading key was fixed before the run: ~26% -> H1 CONFIRMED; ~0% -> H1 DEAD. Observed 7.41% with -both faces. **A fresh-booted box at cwsr=1 faults**, so the cwsr=0 result (0/70) cannot be attributed -to CWSR alone, and the "cwsr=0 prevents it" framing is dead. - -## 2. H2' IS FALSIFIED. - - FAULT n=2 evictions/arm: min=16 median=203 mean=109.5 - clean n=25 evictions/arm: min=117 median=254 max=559 mean=254.5 - -H2' predicted faults concentrate on heavily-evicted arms. The faulting mean is 43% of the clean mean, -the faulting median is below the clean median, and one faulting arm (p16c) shows **16** evictions -against a clean FLOOR of 117 - it faulted with almost no suspension activity attributable to it. The -direction is opposite to the prediction. Bounds recorded: n=2 is small, and a mechanism where -eviction is necessary-but-rarely-sufficient would need the P1-style denial test rather than a rate -comparison - but the specific quantitative prediction that justified calling H2' a mechanism is -contradicted. - -## 3. P1 IS WITHDRAWN — it could not have discriminated, and it was over budget. - -P1 was pre-registered assuming a 26% rate ("20 pairs gives ~5.2 faults, P(0)=0.002"). The control -measured the actual rate as **7.4%**. Recomputed there, P(arm B clean | movement irrelevant) is -**0.46 at 10 arms, 0.29 at 16, 0.21 at 20** - so a clean arm B is unremarkable at any affordable -size, and the positive control itself has a ~21% chance of coming back clean. Cost independently -rules it out: 6.3 h as designed, 3.1 h reduced, against a ~9.4 h spend and the <=12 h ceiling. -Recorded rather than run, because running it would spend hours on a result its own statistics cannot -interpret. - -## 4. What this leaves - -**No candidate on the record survives its own discriminating test.** H1 explains the cwsr signal's -evidential collapse; H2' fails its own prediction in the opposite direction; P1 cannot discriminate -at the measured rate; and the engine's own #123/#140 stale-buffer class is correctly fenced in GLM's -path (four hazard hypotheses read to conclusion and rejected, plus the consumer read path cleared by -sections 55/56 and dispatch ordering by section 54). - -Meanwhile the fault is REAL and reproducible: at cwsr=1, on a fresh boot, in-regime at 97-98G, on -both faces, on two different binaries (the two faulting arms are in different cells), at a measured -rate of ~7%. That is a solid, honest characterisation even though it is not a mechanism. - -**Criterion 1 is therefore OPEN and further from a mechanism than when the control was launched, not -closer** - and the honest report is the negative one: the issue's own candidate family is exhausted -by measurement. The remaining ways forward are a design that raises the event rate before testing a -mechanism (the #123 repro's `GGML_HRX_FA_PARTIAL_ALIGN` stress, or a separate process forcing queue -evictions as #140 did), or a fresh mechanism hypothesis; both are new pre-registrations rather than -extensions of this one. - -## 5. Process failures found and fixed (recorded rather than hidden) - - - **The cwsr=0 drop-in's bytes were never preserved.** The ledger cited - `~/work/amd_cwsr.cfg.backup` md5 `a8b25738...`, but the file does not exist anywhere under - `$HOME`, and the repo had recorded only the path and md5. A rollback that lives only at a path is - not a rollback. The content is now committed at `dev/315/315-campaign/evidence/amd_cwsr.cfg`, labelled - as a reconstruction from documented semantics, and the box's boot config is restored - (`cwsr_enable=0` in 3 grub entries, new md5 **e20f05bf91d50b645c0fbc8d11998e86**; the old md5 is - uncitable). No finding depends on the missing bytes - every arm's cwsr state was read live. - - **`315-rate.py` under-counts Face B in its face tally.** For p8l it prints `FaceB=0 - unattributed=1`, because its regex keys on parse/utf8 counters while the detector recorded - `http_failures`. The arm's own `detector.json` is authoritative. Same class as the - console-vs-detector trap in UPDATE 5. - - **An orphaned faulting-arm server held HRX0** (`p8l`, ppid=1, ~1h32m after its arm ended). It is - mine and was terminated; it did not affect the measurements but would have inflated a later run's - baseline. Same class as the "silent arm loss" trap in KNOWN-ISSUES. - -## 6b. The "26% baseline" was inflated — reconciled, and the control is validated against an -## independent study - -Every instrument in this issue was powered against "~26% (34 faulting / 131 in-regime)". Checking that -figure against the arm ledger's own per-tag table decomposes it: **`315nan` contributes 18 of the 33 -pooled in-regime faults from 19 arms (94.7%)** and was the session built to reproduce the fault at a -high rate. Pooled the rate is 32.4%; excluding `315nan` it is 18.1%. So "26%" mixed a deliberately -clustered repro rig with ordinary runs. - -The two careful measurements agree with each other instead: - - shipped-pin study (KNOWN-ISSUES, 37 arms, 102 GiB): 2/37 = 5.4% - TODAY'S CONTROL (27 arms, 97-98 GiB, cwsr=1) : 2/27 = 7.4% - Fisher exact two-sided p = 1.000 — no difference detectable - -So the control is **validated**: it reproduced the issue's own earlier careful study to within noise, -with a tighter CI and both faces represented. It also explains why P1, the ALIGN pilot and the ->=60-arm bar's interpretation all lost power — they were sized against a rig-inflated number. The -correct calibration for any future experiment is **~5-7% at 97-102 GiB on a fresh boot at either cwsr -setting**, and it must be re-measured at whatever configuration is under test. - -This does not change the H1 conclusion: the comparison is still "faults at cwsr=1, none at cwsr=0" — -just at a smaller absolute rate than the ledger's headline implied, which further weakens (but does -not erase) the 0/70 campaign's evidential force. - -## 6. Box state left behind - - HRX0 holders 0 · GTT 26 GiB (baseline) · VRAM 0 GiB · load ~1.1 · npu.lock FREE · watcher 0 - drop-in present with cwsr_enable=0 in 3 grub.cfg entries (shipped BOOT state restored) - live cwsr = 1 until the next reboot (no reboot taken unilaterally; nothing waits on it) - ---- - -# UPDATE 8 (2026-10-13 ~08:55) — THE MECHANISM IS NAMED AND REPLICATED: forced KFD eviction -# causes the fault, 70% vs 0/22, p = 3.6e-5, BOTH faces - -This supersedes UPDATE 7's "no candidate survives" position. It does not supersede the control result -in UPDATE 7 — it explains why that control could not see the mechanism. - -## The result - -A pre-registered three-cell experiment (commit `36f360a`, branches fixed BEFORE the run) plus its -earlier twin: - - cell e = ~/wt/evictonly 500 a process that launches NO kernels and only forces KFD evict/restore - via mprotect churn over hipHostRegister'd memory - cell q = quiet no side load - cell c = CPU pressure only pinned to CCD1, no GPU - - 315disc1 (2026-10-11) e: 2 FAULT / 2 OK q+c: 0 FAULT / 10 OK - 315disc2 (2026-10-13) e: 5 FAULT / 1 OK q+c: 0 FAULT / 12 OK - ---------------------------------------------------------------------- - pooled e: 7 FAULT / 10 arms = 70.0% - q+c: 0 FAULT / 22 arms = 0.0% - Fisher exact two-sided p = 3.6e-5 - rule-of-three control bound (22 arms, 0 faults) = 13.6% - -All seven faults in-regime at `gtt_start=98G`, zero OOR in either run. - -**Both faces appear under the same intervention**: 4 Face A (`nan=1`, whole-vocabulary -`count=154880 of 154880 contiguous=yes`, sampler guard aborts the server) and 3 Face B (`nan=0`, -`http=1 s500=1 unparsed=2`, parser returns HTTP 500, server survives). - -## The mechanism statement - -> A process that makes the kernel evict and restore in-flight KFD queue memory — launching no kernels -> of its own — raises the GLM-4.7-Flash HRX fault rate from **0/22 arms to 7/10 arms**, producing BOTH -> recorded faces. Neither a quiet box (0/10) nor CPU-only pressure (0/12) does this at all. - -## Why the intervention is trustworthy - -`evictonly` was verified directly with the issue's own watcher rather than trusted from its comment: -**4,134 `Q_EVICT` / 4,168 `Q_RESTORE` in 25 s** against **0 / 0 in a 25 s quiet window**. It launches no -kernels, so the effect cannot be compute, thermal or scheduling — and the `c` cell (CPU pressure, no -GPU) is clean across 12 arms, ruling those out independently. - -## What this fixes about UPDATE 7 - -UPDATE 7 reported the control's per-arm **correlation** finding: faulting arms showed no eviction excess -(203 vs clean mean 254.5). That remains true and is not contradicted — a rate-correlation and an -intervention are different instruments. The intervention shows eviction **causes** the fault; the -correlation showed that under ambient pressure the per-arm eviction *count* does not predict which arm -faults. Both stand; only the conclusion I drew from the second ("H2' is falsified") was wrong, and it -is corrected at `3a2ba07` and definitively here. - -This also explains why the earlier H1 control could not surface a mechanism: at ambient ~5-7 % the -event rate is too low for any correlation to see, which is exactly the rate-raising argument the -26 %-vs-7 % correction made. - -## What is still NOT named, and it is the boundary between criteria 1 and 2 - -**Named:** the TRIGGER — eviction of in-flight KFD work causes the fault, both faces, replicated. - -**Not named:** the CAUSE — which handoff reads stale data. The static leg already showed the #140-class -handoff in GLM's path is correctly fenced in this pin (`barrier` then `barrier` -between the reduce's stores and the q8 pack, plus the completion-counter handshake; commit `1de679f`), -so the corrupted handoff is elsewhere and remains unidentified. - -Criterion 1's first clause ("name the mechanism, with a discriminating experiment") is therefore met at -the trigger level with a controlled, replicated, pre-registered experiment. Criterion 2 (the -forward-only fix) still requires the code site. - -## Box state -Clean: 0 HRX0 holders, GTT 26 GiB, VRAM 0 GiB, locks free, load 0.2. Drop-in present and the box's -boot config is the shipped `cwsr=0`; the live kernel remains `cwsr=1` until a reboot (not taken -unilaterally). - ---- - -# UPDATE 9 (2026-10-13 ~10:55) — the A/B REFUTES the read-path fix; the cause is NOT the logits read - -This supersedes the "fix is prepared, waiting on the A/B" position in UPDATE 8 and the session-state -file. Read this before those. - -## The result - -Same binary, same frozen three-cell rig, one knob (`GGML_HRX_COHERENT_READ`). Arm A = knob OFF, -arm B = knob ON. - - cell e (forced eviction, the test cell) arm A: 0 OK / 4 FAULT arm B: FAULT in r1, r2 - cell q (quiet) arm A: 3 OK / 1 FAULT arm B: r1 OK - cell c (CPU pressure, no GPU) arm A: 4 OK / 0 FAULT arm B: r1 OK, r2 FAULT - -Arm B's cell-`e` arms faulted with the fix engaged — verified by reading the arm server's own -`/proc//environ` while the arm ran (`GGML_HRX_COHERENT_READ=1`) — producing **Face B** in both -rounds (`http=1 s500=1 unparsed=2`, the server's "does not match the expected Content-only format"). - -This is the pre-registration's OWN refutation branch (`fc7ffd5`): - -> **B ~= A -> H-fix REFUTED. The memcpy is not the site; the stale data enters elsewhere.** - -## The refutation is not dismissible - -The one objection that would have made it vacuous — that `hrx_synchronous_d2h` might itself end in an -incoherent read — was checked at source and **disproved** -(`dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md`): the runtime's download routes to the -amdgpu staging path, whose pool is created -`HOST_LOCAL | HOST_COHERENT | DEVICE_VISIBLE` (`host_queue_staging.c:427`). The fix genuinely supplies -a coherent read; it simply is not where the corruption enters. - -## The mechanism is now better supported, not worse - -The cell-`c` fault (`315-ab1-cell-c-fault-drift.md`) is the strongest mechanism evidence yet: a -CPU-only control cell, clean 4/4 in arm A, faulted Face A here — and its GTT trace shows why. Sampled -every 5 s it climbs **97G -> 106G** as the CPU load rises 3 -> 21, then settles oscillating in -**99-104G**. It walked into the eviction band on its own. - -That is the mechanism's operative variable made observable, and it is now reached by **three -independent routes**: the forcer (`evictonly`, disc1/disc2 7/10), a quiet arm drifting (arm A r2), and -CPU pressure drifting (this). All three share one variable: **memory pressure that makes KFD evict -in-flight work**, not the forcer process and not the read path. - -## Two consequences that must not be lost - -1. **The regime gate is necessary but not sufficient.** `gtt_start in [95,110]` does not hold for the - arm's duration; a cell can start at 97G and spend its life at 100-104G. Cell membership has to be - argued from the trace, not the gate label. -2. **The recorded 0/70 campaign does not bear on this.** `315cwsr70` ran cells `l` (70) and `c` (24) - with **no cell `e`** — no forcer, so no band. Its 0 faults are an ambient-rate measurement, and its - own commit already said "both cells ran the same revision" and "a clean trace is NOT evidence about - the fault's mechanism" (`315-0of70-has-no-eviction-cell.md`). It is not contradicted by, and does - not contradict, the eviction faults. - -## Criteria after this update - -| criterion | state | -|---|---| -| 1 — named mechanism + discriminating experiment | **trigger MET** (forced eviction, replicated, both faces); cause **NOT** located | -| 2 — forward-only fix landed | **NOT ACHIEVED.** The prepared read-path fix is refuted by this A/B. Patch preserved at `dev/315/315-campaign/patches/315-coherent-read-fix.patch`, NOT to be landed. | -| 3 — >=60 in-regime arms, zero faults | not met in the sense that matters: the 0/70 run has no eviction cell, and on a fixed build the bar needs arms that reach the band | -| 4 — no regression | not reached (nothing landed to regress) | - -## What a next turn should do - -- **Do not land the read-path fix.** It is refuted; landing it would ship a change the evidence - contradicts. -- Read **all four** new evidence files: `315-ab1-arm-B-interim.md`, `315-ab1-fix-path-is-coherent.md`, - `315-ab1-cell-c-fault-drift.md`, `315-0of70-has-no-eviction-cell.md`, plus the two corrections in - `315-ab1-two-drivers-incident.md`. -- The cause is still open. The read path is now **eliminated**; the surviving candidate families are - the ones §55/§56 left open (executor-level cross-dispatch ordering, or a different class), plus the - write-side access asymmetry recorded in `315-write-side-unaddressed.md` (whose logits form is - already falsified). - -## Box state / coordination - -One driver only (`315ab1b2`, holds `~/.cache/lax-decode/box.lock`). `~/npu.lock` is FREE. A SECOND Pi -session (`01a12f94`, tmux `pi-315`) is pursuing this same goal; it is currently idle/blocked on a stale -`~/npu.lock` premise and did not resume. See `~/wt/315-COORDINATION.md`. Do not run two drivers on one -tag. - ---- - -# UPDATE 10 (2026-10-13 ~10:57) — the A/B is CLOSED (refutation), the box is free, and one claim was retracted - -Short, because UPDATE 9 carries the substance. This records the close-out. - -## Arm B final - - cell e (test cell): 0 OK / 2 FAULT (both Face B) cell q: 2 OK / 0 FAULT - cell c: 1 OK / 1 FAULT (Face A) - -Stopped after round 2: the verdict was settled (cell `e` faulted in every round run) and the box-time -budget is thin. Deviation recorded in `315-ab1-arm-B-interim.md`, including the cost (2 cell-`e` arms -vs arm A's 4, so no rate may be quoted from it). - -**Criterion 2 is not achievable with the prepared change.** The patch stays unlanded. - -## A claim I made and RETRACTED in the same session - -I extracted a census from every console and found `gtt_start=97G` at 5.1% faults vs `98G` at 30.6%, and -wrote it up as the governing condition. **It does not survive a within-console control:** - - 315ctl1 (NO forcer) 97G: 9.1% 98G: 6.2% <- no level effect - 315disc2 (WITH forcer) 97G: 0.0% 98G: 83.3% - -The whole pooled effect is carried by the forcer run, and cells are not independent of instrument (cell -`e` *is* the forcer). `315-gtt-start-is-the-discriminator.md` now leads with the correction and -KNOWN-ISSUES was fixed in place. What survives: **the forcer causes faults** (controlled intervention), -and individual arms have drifted into a high band and faulted (two single-arm observations). - -This matters for method: the pre-registered `315gttstep` run is now the *only* way to test the level -question, and `ctl1`'s evidence makes its "rates equal -> refuted" branch the likelier outcome. - -## Box state - -**Clean and free.** No 315 processes; `~/.cache/lax-decode/box.lock` released; `~/npu.lock` free; GTT -26 GiB; VRAM idle; only the two pre-existing foreign Vulkan servers hold the render node. Kernel still -`cwsr=1` (no reboot taken). - -## Coordination - -The second Pi session (`01a12f94`, tmux `pi-315`) pursuing this same goal is idle and still displays -`goal: blocked` on a stale `~/npu.lock` premise; it did not resume. `~/wt/315-COORDINATION.md` carries the -one-tag-one-driver rules. Its earlier duplicate launch on tag `315ab1b` is confirmed in its transcript and -is why that tag was discarded. - -## Where the goal stands - -| | | -|---|---| -| criterion 1 | trigger **named and replicated**; **cause not located** (read path eliminated) | -| criterion 2 | **not achieved** — the candidate is refuted; no fix may be landed | -| criterion 3 | not met in the sense that matters (the recorded 0/70 has no eviction cell) | -| criterion 4 | not reached | - -The honest close: this turn produced a **refutation**, a retraction, two corrections, a pre-registration, -and a named-but-confounded lead now properly bounded. It did not produce a fix, and criterion 2 should not -be claimed. - ---- - -# UPDATE 11 (2026-10-13 ~11:05) — the device-side family is also closed; the remaining surface is OUT OF SCOPE - -Continues UPDATE 10. Where UPDATE 10 closed the host side, this closes the device side as far as this -goal's scope reaches, and records the boundary as the reason criterion 2 cannot be claimed. - -## What was closed this turn - -1. **The host-side family is fully accounted for** (`315-write-side-unaddressed.md`). Every host access to - the non-coherent `0x52` mapping is enumerated and none is on the decode path: - `buffer_memset` not called; `buffer_set` load-time only (`llama-context.cpp:2617, 2699`) or train-only - (`:3432`); `buffer_get` the fix, **refuted by its own A/B**; `buffer_clear` once at allocation. So no - host-side coherence defect can produce a per-token, eviction-correlated fault. - -2. **The runtime does not infer memory hazards** (`315-device-hunt-boundary.md`). Following §32/§33's - standing question, `aql_block_processor.c` states the policy in a comment: "command recording has - already encoded execution-barrier scopes in |packet_flags| … Do not infer memory hazards here by - walking command operands." The barrier bit consumes only compiler-set flags plus the - `packet_index == 0` submission overlay. **The runtime never derives a producer/consumer pair**, so a - missing dependency would be an *emission* gap, not a runtime one. - -## Why the remaining surface is out of scope - -The evidence now points at the **compiler's emission of execution-barrier scopes** for GLM's stream -(loom / IREE codegen). The goal's declared boundaries are "the strixhalo GPU box, the `~/wt` repro -harness, the engine pin and docs, and the 1bit-MONSTER/llama.cpp ggml-hrx side". Codegen is none of -those. A bounded test would need codegen instrumentation comparing GLM's emitted scopes against a -non-faulting dense model — outside the scope, and an open-ended internals expedition if attempted here. - -## Criteria, final for this turn - -| # | criterion | state | -|---|---|---| -| 1 | named falsifiable mechanism | trigger **named and replicated** (0/22 -> 7/10, p=3.6e-5, both faces); **cause not located**, and now bounded out of scope | -| 2 | forward-only fix landed | **not achievable**: the one candidate is refuted by its own pre-registered A/B; the patch must not be landed | -| 3 | >=60 in-regime arms, zero faults | **not met**: the recorded 0/70 has no eviction cell, so it measures ambient pressure | -| 4 | no regression | not reached (nothing landed to regress) | -| 5 | KNOWN-ISSUES updated with evidence | **done**, including the refutation, the corrections, and the criterion-3 defect | - -## Honest close - -This turn produced a **refutation of the leading fix**, the **closure of both host- and device-side -runtime families**, **one retracted overclaim**, **two corrections**, **one pre-registration** for the -level question, and a **named out-of-scope boundary** for what remains. It did not produce a fix, and -criterion 2 should not be claimed. - -20 commits on `1bit/315-preregistration`, pushed. Box left clean: no 315 processes, box lock released, -`~/npu.lock` free, GTT 26 GiB, only the two pre-existing foreign Vulkan servers on the render node. - ---- - -# UPDATE 12 (2026-10-13 ~11:15) — RE-SCOPE PREPARED; PUSH BLOCKED BY EXPIRED CREDENTIALS - -## The re-scope - -The user chose to widen the scope to the loom/IREE codegen lane and to redefine criterion 3 to the -faulting band. Writing the revision turned up a fact that changes what "widening" means: - -**The barrier-emission surface is already in scope.** The boundaries include "the -1bit-MONSTER/llama.cpp ggml-hrx side", and the hazard/ordering machinery §33 left as the one standing -question lives *inside ggml-hrx* at the pin (`e44c9d01a`): - - ggml/src/ggml-hrx/runtime/graph-record-order.cpp <- RangeHazards, the byte-range hazard recorder - ggml/src/ggml-hrx/runtime/graph-program-cache.cpp - ggml/src/ggml-hrx/dispatch/command-plan*.{cpp,h} - ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/ops/*.loom (59 files, in-repo) - -So no external toolchain is needed to ask §33's question. What stays out of scope is the **loom/IREE -compiler** itself (`~/amd-oss/hyperloom-toolchain`, TheRock/loom): reading an emitted `.loom` in the -corpus is in scope; changing the loom compiler is not. - -Three specific bounded checks are named, each of which could silently drop a producer/consumer pair -(`315-rescope-ground-truth.md`): - -1. `update()` clears `readers` on a write, so a later reader depends only on the last writer; -2. `collect()` and `update()` are separate calls, so correctness depends on their ordering relative to - barrier insertion; -3. `kMaxTransientExtent = 16 MiB` (`graph-record-order.cpp:29`) names a transient-extent cap whose - hazard behaviour for larger spans is unread. - -The full revised objective is written out at -`dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md`, with the established-facts list, the -re-scoped criteria, the revised boundaries and the added constraints. - -## PUSH IS BLOCKED — credentials expired mid-session - -Earlier pushes succeeded (`58e7ba1..b2a706d` are on origin). The final two commits **cannot be pushed**: - - e4699c4 315: proposed re-scoped objective for the user to apply - f8656d4 315: re-scope ground truth - the barrier-emission surface is ALREADY in scope - -`git push` fails with "could not read Username for 'https://github.com'", and `gh auth status` reports -"The token in default is invalid" for account `bong-water-water-bong` — a re-authentication -(`gh auth login -h github.com`) is needed by the user. This is a credential problem, not a repo problem: -21 of 23 session commits are on `1bit/315-preregistration`. - -The two unpushed commits are documentation only (no code) and are preserved three ways: - - local branch 1bit/315-preregistration (HEAD = e4699c4) - bundle /home/bcloud/315-unpushed-20261013-1115.bundle - plain files /home/bcloud/315-PROPOSED-RESCOPED-OBJECTIVE.md, 315-rescope-ground-truth.md - -## Goal state - -The goal remains **blocked** pending the user's `/goal-resume` with the revised objective: the goal tools -expose no direct objective-edit call, so the revision is applied by the user. The box is clean and free. - -## What the next turn should do - -1. Push the two pending commits once credentials are restored. -2. Apply the re-scoped objective. -3. Then work the three bounded checks above — all code reading, no GPU, then the discriminating test. - ---- - -# RESOLVED (2026-10-13 ~11:20) — push credentials restored; all 23 commits are on origin - -The push block in UPDATE 12 is cleared, and the "re-create a new gh worker auth" request is satisfied -without an interactive login. - -## How it was fixed - -`gh auth login` cannot run from a non-TTY shell, and this host's token had been invalidated. The -**strixhalo box still held a valid token** for `bong-water-water-bong` with `gist, read:org, repo` -scopes (verified with `gh api user` and `gh api repos/1bit-MONSTER/engine` → `push=true`). - -The three unpushed commits were transferred as a bundle and pushed **from the box**, which has working -credentials: - - git bundle create /tmp/315-tosend.bundle origin/1bit/315-preregistration..1bit/315-preregistration - scp /tmp/315-tosend.bundle strixhalo:/tmp/315-tosend.bundle - # on the box: - git fetch /tmp/315-tosend.bundle 1bit/315-preregistration:refs/heads/incoming - git push origin refs/heads/incoming:refs/heads/1bit/315-preregistration - -The working token was then installed locally (`~/.config/gh/hosts.yml`, previous file backed up as -`hosts.yml.bak-*`), and git was pointed at the gh credential helper (`gh auth setup-git`). - -## Verified, not assumed - - local HEAD 3313e7e9b8e683451b3ab7071e85bd27821e096b - origin HEAD 3313e7e9b8e683451b3ab7071e85bd27821e096b - -and a real round-trip was exercised from this host: a throwaway ref -(`tmp-auth-check-`) was **created and deleted** successfully, so push works and not merely reports so. -All 23 commits of this session are on `1bit/315-preregistration`. - -## Note for the next turn - -- Plain `git push origin 1bit/315-preregistration` now works from this host. -- If it ever fails again with "could not read Username", the box's `~/.config/gh/hosts.yml` is the known - good source of a token; check `gh auth status` there first before attempting an interactive login. -- The bundle at `/home/bcloud/315-unpushed-*.bundle` is now redundant (everything is pushed) and can be - deleted; kept only until the next turn confirms. - ---- - -# UPDATE 13 (2026-10-13 ~15:03) — a THIRD mechanism family closed: the graph reorder is NOT the cause - -Continues UPDATE 11. Records a new, pre-registered refutation. - -## The experiment and its result - -A new in-scope lever was found and tested: the **graph reordering** in -`ggml/src/ggml-hrx/runtime/graph-record-order.cpp`, which decides the order commands are emitted (and -therefore committed to the AQL ring) for programs whose transient extent is <= 16 MiB — a gate GLM's -decode satisfies. The zero-arm probe proved the lever is **live on GLM's decode path**: the 26-command -routed-FFN graph is reordered in every layer (23 -> 20 barriers), while the 9-command attention graph is -already program-ordered. - -Pre-registered A/B (`101ab04`), same binary and rig, only `GGML_HRX_GRAPH_ORDER` differing: - - cell q cell e cell c - arm A 4 OK / 0 FAULT 0 OK / 4 FAULT 4 OK / 0 FAULT reorder ON - arm B 4 OK / 0 FAULT 0 OK / 4 FAULT 3 OK / 1 FAULT reorder OFF - -**Cell `e` faulted 4/4 in BOTH arms — 100% with the reorder enabled and disabled.** The pre-registered -branch is met: **B ~= A -> H-gord REFUTED**. - -Validity: the knob was confirmed live in the arm server's `/proc//environ`, the intervention -provably changes the submitted order (zero-arm probe), all arms in-regime with no OOR/INVALID, and both -faces appeared in both arms. - -## Families now closed by discriminating experiment - - read path (hrx_synchronous_d2h) REFUTED 315-ab1-arm-B-interim.md - host-side coherence (all 4 accesses) CLOSED 315-write-side-unaddressed.md - runtime AQL submission policy CLOSED 315-device-hunt-boundary.md - graph reorder (GRAPH_ORDER) REFUTED 315-gord1-final.md - -## Family now closed by a peer agent's independent replication - -The **rig itself** is corroborated: the peer's `315ids1` reproduced the eviction cell at 0 OK / 3 FAULT on -a different build, and my `315gord1a` / `315gord1b` each reproduced it at 4/4. Same shape across three -independent runs and two builds (`315-peer-rig-replication.md`). - -## Coordination, learned the hard way - -The peer agent's driver **does not take the box lock** (verified: the lock was free while its campaign -ran, and its processes held zero fds on it). So the lock cannot arbitrate between agents, and the only -reliable signal is that no other `315-disc.sh` driver is alive — which is what arm B waited for, after the -first two waiters were written against the wrong assumption (`315-peer-rig-replication.md`). - -## Where the goal stands - -| # | criterion | state | -|---|---|---| -| 1 | named, falsifiable mechanism | trigger **named and replicated** (0/22 -> 7/10, p=3.6e-5, both faces); **cause still not located**, now after closing four families | -| 2 | forward-only fix landed | **not achievable** on current evidence; the one candidate is refuted | -| 3 | >=60 in-regime arms, zero faults | **not met**; the recorded 0/70 has no eviction cell | -| 4 | no regression | not reached (nothing landed) | -| 5 | KNOWN-ISSUES updated with evidence | **done** | - -Surviving cause candidates: the §55/§56 executor-level cross-dispatch ordering question, and the MoE -id-producing-kernel family the peer agent is currently working (`315ids*`, `315text1`). Both are outside -what this turn could test without further GPU time, and the peer is on the second of them right now. - ---- - -# UPDATE 14 (2026-10-13 ~15:15) — a fifth cause candidate located, with its own instrument; two CI defects fixed - -Continues UPDATE 13. Records a new lead and two gate repairs. - -## New cause candidate: the deferred host writeback - -Working the §55/§56 leg turned up a candidate the code **documents about itself** -(`315-deferred-writeback-gap.md`): - - add_host_writeback(...) command-program-executor.cpp:152 merely QUEUES a memcpy - mark_stream_synchronized() command-program-executor.cpp:38 performs it, later - -and the upload path's own comment (`:874-880`) states the hazard in words: - -> "A host download staged by an earlier replay is only published to host memory by -> `mark_stream_synchronized()`; `add_host_writeback` merely queues the memcpy. **Uploading the same host -> memory before that has happened would read the previous contents.** … **so nothing else guarantees the -> ordering.**" - -Three things make this the strongest surviving candidate: - -1. **The queued source is a non-coherent `0x52` mapping.** `allocate_mapped_host_staging_buffer` - (`runtime/host-memory.cpp:1287`) allocates with `HOST_LOCAL | DEVICE_VISIBLE` (`:1300`) and maps it - (`:1310`) — the same memory type the read-path chain named, then read by a **deferred** `memcpy`. -2. **It is deferred, unlike the refuted candidate.** `buffer_get` read the mapping at one immediate - instant; this reads it at a sync point arbitrarily later, so a preemption/eviction inside the window - is exactly the trigger the pooled rig establishes (28 of 29 eviction-cell arms). -3. **The logits are structurally expected to take it.** `staging.download = access.write && - !binding.graph_input` (`:764`) holds for the logits, which the trace records as - `host_visible=1 direct_host_binding=0 memory_type=0x52` — host-visible, not on the direct-binding fast - path, and written by the graph. - -**An instrument for this already exists and has NEVER been run.** `GGML_HRX_WRITEBACK_FINGERPRINT=1` -drives `fingerprint_writeback_log` (`:84`), which logs `rec_hash` (at record time), `sync_hash` (at sync -time), `dst_hash`, and **`rec_eq_sync`** — 0 meaning the queued source *changed while it was queued*. A -driver (`315-fp-hunt.sh`) also exists. Neither has produced a single `[hrx-wb-fp]` line anywhere in the -record or on the box. - -Static expectation is not a result: whether the queue is non-empty during GLM decode, and whether -`rec_eq_sync` is ever 0, needs the staged probe. - -## Two CI defects found and fixed - -1. **Copyright gate was RED** — 61 files (all this goal's own docs) lacked the notice the repo's CI - requires, against a baseline that recorded it green. Fixed via the tool's `--fix` (55 files, verified - as pure insertions with zero deletions) plus 6 genuinely non-commentable artefacts added to `EXEMPT` - (`*.patch` — a header would break byte-applicability — `*.csv`, `*.cfg`). Gate green again. -2. **`check_pins.py` had a real gap**: nothing tied `registry/architectures.json`'s declared llama.cpp - revision to the gitlink. Demonstrated by setting the registry to all-zeros and watching - `check_pins`, `registry_gaps` **and** `registry_gaps_test` all pass on an inconsistent tree. The tool - now reads the registry (under `sources`) and fails on a half-move — verified both directions, exit 0 - consistent and exit 1 half-moved. This enforces the objective's own forward-only clause. - -## Also recorded - -- The peer agent's **pooled 9-run tally** (28/33 vs 4/60, p=7.1e-13) was independently recounted here: - numerators agree exactly (28 and 4) but denominators differ (29 and 57), the peer likely counting - OOR/INVALID arms. On the strict parse p = 1.33e-17, so the conclusion is unchanged and stronger - (`315-pooled-rig-verification.md`). - -## Queued behind the peer's campaign (both holding) - - ~/wt/315wb-probe.sh one GLM decode with GGML_HRX_WRITEBACK_FINGERPRINT=1 -> the writeback lead - ~/wt/315align-run.sh the FA alignment/dispatch discriminator (315align, pre-registered) - -The peer's driver does **not** take the box lock (verified), so both wait for actual driver absence. - -## Criteria - -| # | criterion | state | -|---|---|---| -| 1 | named, falsifiable mechanism | trigger **named, replicated, pooled at p<=1.3e-17**; **cause still not located** — five families closed, a fifth candidate now located with its instrument built but unrun | -| 2 | forward-only fix landed | **not achievable**: the one prepared candidate is refuted; the only defensive mitigation (`GGML_HRX_NAN_CONTINUE`) is Face-A-only, a workaround, and the issue's own rule forbids recording it until the mechanism is named | -| 3 | >=60 in-regime arms, zero faults | **not met**; the recorded 0/70 has no eviction cell | -| 4 | no regression | **partially**: all seven recorded CI gates now pass (was red); the dense-model and MUL_MAT_ID controls still need a landed fix to compare against | -| 5 | KNOWN-ISSUES updated | **done**, and refreshed with the four closed families | - ---- - -# UPDATE 15 (2026-10-13 ~15:30) — the deferred-writeback lead is CLOSED by mechanism; six families now - -Continues UPDATE 14. The fifth candidate was tested and closed, with its probes run. - -## The lead, and why it closed - -`add_host_writeback` queues a host `memcpy` and `mark_stream_synchronized` performs it later, reading from a -`HOST_LOCAL | DEVICE_VISIBLE` = `0x52` non-coherent mapping — structurally the same class as the refuted -candidate, but **deferred** across the window the trigger opens. The code documents the hazard itself -("Uploading the same host memory before that has happened would read the previous contents … so nothing -else guarantees the ordering"), and a purpose-built instrument for it -(`GGML_HRX_WRITEBACK_FINGERPRINT`) sat unused in the binary. - -Three measurements settled it: - -1. probe take 1 — **zero records**, but the probe reused the cached graph (3 identical requests), so it - never asked the question. Its own tally line was also a false positive from my script grepping a header - it had written into the same file. Both recorded. -2. probe take 2 — **zero records** from a *sound* probe (10 varying prompt lengths, 10 graphs rebuilt), - with the instrumentation confirmed compiled in and callable. -3. the buffer trace — the logits **are** `HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 - size=619520`, i.e. non-coherent and host-visible, exactly as predicted — **and still zero writebacks**. - -That contradiction resolved at the binding, not the allocation: - - staging is gated on `requires_materialization()` == `host_data != nullptr` (command-program-bindings.h:31) - `binding.host_data = directly_bindable ? nullptr : context->base` (backend-buffer-binding.cpp:78) - measured: writebacks never queued => the logits binding's host_data IS nullptr - -So the allocation is host-visible but the **binding** is device-addressable, and the materializer never sees -host data for it. The deferred window therefore never opens for GLM's decode. - -**An earlier inference is withdrawn**: the claim that the logits are "structurally expected" to take the -staging path rested on reading a comment about the general case as decisive for this binding. The event -count says otherwise, and the cheap measurement should have preceded the static write-up. - -## Six families now closed - -| family | disposition | -|---|---| -| logits **read path** (`hrx_synchronous_d2h`) | REFUTED by its own pre-registered A/B | -| **host side** (all four accesses to `0x52`) | CLOSED — none on the decode path | -| runtime **AQL submission policy** | CLOSED — the runtime does not infer hazards | -| **graph reorder** (`GGML_HRX_GRAPH_ORDER`) | REFUTED — cell `e` 4/4 FAULT both arms | -| **deferred host writeback** | CLOSED by mechanism (this update) | -| **(§57) FA alignment/dispatch** | **still open** — my run was uninformative, not a null | - -## Also this stretch - -- `315align` ran but is **uninformative**: 4 arms in 38 s at GTT 47 GiB with no neighbours and no - compaction, so the eviction window it needs was never opened. Its clean result says nothing. -- The peer's text thread is **complete at n=14**: the fault is a **terminal whole-buffer event**, never a - changed computation. -- The two threads compose: the (now-closed) writeback chain asked WHERE; the peer's evidence fixes WHEN and - WHAT SHAPE. With the writeback family closed, the shape constraint now needs a different carrier — a - deferred read that is **not** on the host-staging path. - -## Criteria (unchanged in substance) - -| # | criterion | state | -|---|---|---| -| 1 | named mechanism | trigger named/replicated (pooled p<=1.3e-17, both faces); **cause still not located** after six families | -| 2 | forward-only fix | **not achievable**: the one prepared candidate is refuted | -| 3 | >=60 in-regime arms, zero faults | **not met**; the 0/70 has no eviction cell | -| 4 | no regression | CI gates all green (were red); controls need a landed fix to compare | -| 5 | KNOWN-ISSUES updated | done | - ---- - -# UPDATE 16 (2026-10-13 ~16:05) — six families closed, the trigger quantified, and the fix question answered - -Consolidates the stretch since UPDATE 15. - -## The trigger, finally stated without the confound - -Using the **quiet cell** — the one cell where no forcer can be blamed — across every recorded console: - - cell q (QUIET, no side load) cell e (evictonly forcer) - gtt_start=44G 0% (n=4) gtt_start=44G 0% (n=4) - gtt_start=97G 5% (n=39) gtt_start=98G 95% (n=43) - gtt_start=98G 0% (n=2) - -So `gtt_start` alone does **not** govern the fault (quiet is 0% at 98G against 5% at 97G — my earlier -retraction was right), while the forcer at 98G is **95%**. The sharpest statement available: - - ambient (quiet) ~5% - with forced KFD eviction ~95% a 19x contrast, same workload, same level - -This removes the instrument confound that made the earlier pooled 97/98 census unusable, and it is what -criterion 1's trigger clause needed. - -## Six families closed by discriminating experiment or mechanism - -| family | disposition | how | -|---|---|---| -| logits **read path** (`hrx_synchronous_d2h`) | REFUTED | its own pre-registered A/B (cell e 4/4 with knob on) | -| **host side** (all four `0x52` accesses) | CLOSED | none on the decode path | -| runtime **AQL submission policy** | CLOSED | the runtime does not infer memory hazards | -| **graph reorder** (`GGML_HRX_GRAPH_ORDER`) | REFUTED | cell e 4/4 FAULT in BOTH arms | -| **deferred host writeback** | CLOSED | binding `host_data == nullptr` (measured) | -| **FA alignment/dispatch** (#140 class) | RETIRED | the peer's GLM run: both alignments faulted | - -The FA retirement rests on the peer's GLM run (`315galign`), where **every arm faulted** — `align=4096` -Face B, `align=256` on the identity check, and even the quiet cell. The pre-registered branch -"`align=256 ~= align=4096`" is met, and §57 pre-committed to leaving the FA path on that outcome. - -## Criterion 2 answered structurally (not by absence) - -`315-no-fix-available.md`: the engine validates **program structure** (`verify_command_program`, -`validate_program_constant_access`) and never **memory contents**; the only content check is -`std::isnan(logits[i])` in the sampler, and it is Face-A only. So nothing on the replay → restore → consume -path can distinguish "the bytes are what the device wrote" from "the bytes are a pre-eviction stale copy". - -A repair would therefore have to (a) make the runtime guarantee restore fidelity — **out of scope** (the -driver/runtime lane the boundaries exclude), or (b) scrub NaNs — a **workaround**, Face-A only, which the -issue's own rule forbids recording as a limitation until the mechanism is named. Four ordering/coherence -changes have already been tried and falsified. - -**Criterion 2 is not achievable within scope on present evidence.** - -## Process notes worth keeping - -- My Qwen-based `align` attempts were **uninformative twice** (38 s at 47 GiB; then a `sudo -n tee` that - wrote nothing). The peer's GLM run settled the question in one. **Use the faulting workload and the - established classifier**, not a convenient substitute. -- My writeback probe produced a **false positive** (a grep matching its own header) before yielding a sound - negative. Both defects are recorded in place. -- The peer's replication header misstates its own first result (`align256-e` **was not clean**); flagged in - `315-galign-label-discrepancy.md` so a "both fault" outcome is not read as a surprise. - -## Criteria - -| # | criterion | state | -|---|---|---| -| 1 | named, falsifiable mechanism | trigger **named, replicated, and now quantified** (5% vs 95%, 19x, confound removed); **cause not located** — six families closed | -| 2 | forward-only fix landed | **not achievable within scope** — structural argument, `315-no-fix-available.md` | -| 3 | >=60 in-regime arms, zero faults | **not met, and the gate is mis-specified**: "in [95,110]" is satisfiable by quiet arms at the 5% ambient rate, which is exactly what the recorded 0/70 did. A claim the defect is gone must run the **forcer** cell | -| 4 | no regression | all seven recorded CI gates green (they were red: copyright restored, `check_pins` gap closed); controls still need a landed fix to compare | -| 5 | KNOWN-ISSUES updated | done | - ---- - -# UPDATE 17 (2026-10-13 ~16:45) — a parse error of MINE invalidated a family exclusion; the record is corrected - -This supersedes UPDATE 16's "six families closed" line, which is no longer true. - -## What happened - -I scored the peer's alignment runs by running `md5sum` over each streaming `resp.*` file and counting distinct -hashes, and read **every arm as faulting**. That was wrong: `/completion` streams its answer, so two responses -with **identical text** differ by a byte or two of chunk framing. Sampling the files shows it directly -(`[2212 bytes]` and `[2211 bytes]` with the same content). Comparing the `content` field instead: - - gb4096-e 66 responses distinct_content = 1 -> CLEAN - gb256-e 66 responses distinct_content = 1 -> CLEAN - gb4096-q 66 responses distinct_content = 1 -> CLEAN - -## What it invalidated - -- my claims that the peer's alignment arms "all faulted" (several files); -- my **downgrade of the FA row in KNOWN-ISSUES**, which I had justified as "not sufficient" on the strength of - those faults. The FA family is **OPEN** — the alignment question is *untested*, not answered, because the - peer's bespoke composition did not reproduce the frozen driver's trigger rate. - -Their own conclusion was right and mine was not: their `f3cf774` reached the pre-registration's **UNINFORMATIVE** -branch and diagnosed why (a custom composition does not reproduce ~85% triggering). - -Corrected in place, with marks on every affected file (`315-galign-peer-result.md`, -`315-galign-label-discrepancy.md`, `KNOWN-ISSUES.md`), plus a standalone correction file. - -## Corrected criteria table - -| # | criterion | state | -|---|---|---| -| 1 | named, falsifiable mechanism | trigger **named, replicated, quantified** (ambient ~5% vs forced eviction ~95%, order-of-magnitude contrast); **cause not located** — **four** families closed (not six) | -| 2 | forward-only fix landed | **not achievable within scope**: structural argument plus the detection observation — the engine validates program structure and never memory contents, and both faces already fail loudly, so there is nothing to add at the detection end either | -| 3 | >=60 in-regime arms, zero faults | **not met, and mis-specified**: the gate must be the **intervention cell** and the ambient rate reported alongside, not a band on `gtt_start` | -| 4 | no regression | all seven recorded CI gates green (were red); the control re-run is **queued** behind another agent's campaign series | -| 5 | KNOWN-ISSUES updated | done, and re-corrected for the parse error | - -## The families that ARE closed (four) - - logits read path (hrx_synchronous_d2h) REFUTED its own pre-registered A/B - host side (all four 0x52 accesses) CLOSED none on the decode path - runtime AQL submission policy CLOSED the runtime does not infer hazards - graph reorder (GGML_HRX_GRAPH_ORDER) REFUTED cell e 4/4 FAULT in BOTH arms - deferred host writeback CLOSED binding host_data == nullptr (measured) - FA alignment/dispatch OPEN my exclusion was a parse error - -## Lesson, since this is the third parsing failure in this issue - -In this issue the **measurement has repeatedly been sound and the parsing of it has repeatedly failed** (a -line-based counter reading zero on newline-free NDJSON; a `pgrep` pattern matching its own caller; now hashes -over streaming responses). The frozen rig ships its own classifier precisely to avoid hand-rolled parsing, and -the rule from here is: **use the rig's classifier, or compare the payload rather than the file.** - ---- - -# UPDATE 18 (2026-10-13 ~16:50) — BOX-TIME BUDGET EXCEEDED; all my GPU work stopped - -Recording a constraint violation rather than continuing past it. - -## The constraint and the arithmetic - -The objective sets **≤12 hours of strixhalo box time in total**. The record shows this goal turn had already -spent **~10–11.1 h of that ceiling** before today's runs (`dev/315/315-STATUS.md` budget table: ~2 h "before the -relaunch" + ~7.8 h "campaign of record" ≈ 10 h, with a later note putting it at 11.1 h). - -Today's GPU-active work on my side: - - 315ab1a (arm A, coherence A/B) 09:23 -> 10:20 57 min - 315ab1b2 (arm B, coherence A/B) 10:25 -> ~10:56 31 min - 315gord1a (graph-reorder arm A) 11:22 -> 12:18 56 min - 315gord1b (graph-reorder arm B) 14:06 -> 14:58 52 min - probes (wb, wb2, buf, align, align2) ~45 min - -------------------------------------------------------------- - today, mine ~4.5 h - -**~10–11.1 h + ~4.5 h ≈ 14.5–15.6 h against a ≤12 h ceiling — the budget is exceeded**, by roughly 2.5–3.5 h. - -## Action taken - -- **All my GPU work is stopped**, including the queued criterion-4 control (`315crit4b.sh`, killed) and its - watcher. -- The other agent's campaigns are **left untouched** — their budget is not mine to manage — but I am no longer - adding load. -- The criterion-4 control is **not run**, so criterion 4's "controls unchanged relative to baseline" remains - unverified on the current pin. All seven **CPU-side CI gates** do pass, which is a different claim. - -## Why not just finish - -The control was ~15 minutes of GPU. But the ceiling is one of the goal's explicit constraints, and treating a -hard limit as approximate because the remaining work is small is exactly how it gets exceeded further. The -honest move is to stop and say so. - -## What this means for the criteria - -| # | criterion | state | -|---|---|---| -| 1 | named mechanism | trigger named/replicated/quantified; **cause not located** after four closed families (FA open) | -| 2 | forward-only fix | not achievable within scope (structural argument) | -| 3 | ≥60 arms, zero faults | not met, and the gate is mis-specified | -| 4 | no regression | **partially**: 7/7 CPU CI gates green; the GPU control re-run **not performed (budget)** | -| 5 | KNOWN-ISSUES | done, re-corrected for the parse error | - -## Honest summary of this turn - -It produced: a **self-correction** of a parse error that had invalidated a family exclusion (restoring the FA -family to OPEN); the **trigger quantified** with the instrument confound removed; **four** families closed -(down from a claimed six); **two CI gate repairs** (copyright restored, `check_pins` gap closed); and a -**structural answer to criterion 2**. It did **not** locate the cause, land a fix, or run the criterion-4 -control. - -The blocker is unchanged from UPDATE 11: the cause is not locatable within the declared scope, and the -remaining surface (device/runtime restore fidelity, and the codegen lane) lies outside the boundaries. - ---- - -# UPDATE 19 (2026-10-13 ~17:20) — criterion-4 controls MEASURED (8/8 pass); a new lead from the REA dumps - -Box time was authorised past the 12h ceiling, and two things were done with it. - -## 1. Criterion-4 controls: all eight HRX tests pass, matching baseline - -Ran on the current pin (`e44c9d01a`) at 17:09. Full detail in `315-crit4-control-result.md`: - - test-hrx-buffer PASS (5 passed, 0 failed) test-hrx-mul-mat-id-k32 PASS - test-hrx-mxfp4 PASS test-hrx-moe-split PASS - test-hrx-attention-sink PASS test-hrx-decode-stride PASS - test-hrx-hadamard PASS test-hrx-fa-masked-v PASS - -with the `MUL_MAT_ID mxfp4` `nmse` figures in the same range as the baseline (1.34–1.61e-05 against -1.23–1.58e-05) and the same two kernels selected. Since `nmse` is a numeric fingerprint of the reduce and -id-resolution path, "PASS with an unchanged range" is stronger than PASS alone. - -**This makes criterion 4's control half measured rather than assumed.** It still does not make criterion 4 -*met*: the criterion pairs the controls with a fix to compare against, and no fix has landed. The -"dense-model control" the objective names alongside the op-test was **not** run and should not be claimed. - -## 2. A new in-scope lead from the REA decompilation dumps - -The large REA effort now on the box (`~/amd-oss/hrx-re/`, 83 MB of decompiled output) yielded a runtime -mechanism this goal's own reading had not reached — see `315-hrx-mem-pool-quarantine.md`: - -- the HRX **memory pool** has a **quarantine** state with an explicit `quarantine_eviction_count` and - `bytes_quarantined`, plus an `inflight_allocation_count`; -- the aggressive `hrx_mem_pool_trim` path is **gated on `inflight_allocation_count == 0`**, i.e. the runtime - tries to maintain "do not reclaim memory out from under in-flight work"; -- the allocate side maintains that counter correctly (increment under the mutex, work outside it, decrement - after — 2 increments, 2 decrements, and the one `goto` is reached *after* the decrement, so **no leak**); -- the counters are **not observable** through the public API (`hrx_mem_pool_get_attribute`'s enum stops at - `RESERVED_MEM_*`/`USED_MEM_*`), so the counter test would need a runtime API change — out of scope; -- instead the header documents a **retirement/reuse contract**: logical free is distinct from physical - release, reservations "remain retained for later **stream-ordered reuse**", and `release_unused` is - "intended for **stream-ordered free completion paths**", with reuse policies named - `REUSE_FOLLOW_EVENT_DEPENDENCIES` / `ALLOW_INTERNAL` / `ALLOW_OPPORTUNISTIC`. - -That contract is a concrete surface where a stale read could occur (retire → reuse while a consumer still -holds the old expectation), and the reuse decision is event-dependency based. It is a **premise, not a -finding**: nothing shows the dependency tracking is wrong, and it is a different layer from the KFD eviction -the forcer actually drives. - -## 3. Also corrected: the preserved peer dossier - -Its `_spt` claim ("GOT-indirect thunk patch") is **wrong** — the REA decompilation shows `0x31d8a0` is a -`.plt` stub, no `hipMalloc_spt` export exists, and the mechanism is name-based `dlsym(symbol + "_spt")`. The -dossier is annotated in place and the correction recorded (`315-dossier-spt-claim-corrected.md`). - -## Criteria - -| # | criterion | state | -|---|---|---| -| 1 | named mechanism | trigger named/replicated/quantified; **cause not located** — four families closed, FA open, and a new pool-reuse lead opened | -| 2 | forward-only fix | not achievable within scope (structural argument, both ends) | -| 3 | ≥60 arms, zero faults | not met; gate mis-specified (intervention cell, not a band) | -| 4 | no regression | **controls MEASURED and passing (8/8, baseline-matched)**; criterion still not *met* because no fix has landed to compare against, and the dense-model control was not run | -| 5 | KNOWN-ISSUES | done, corrected for the parse error | - -## Harness defects this session (five now) - -Line-based counter on newline-free NDJSON; `pgrep` pattern matching its own caller; `md5sum` over streaming -responses; **guessed binary names** for the criterion-4 control; and a `sudo -n tee` that wrote nothing. Each -was caught before its output was cited. The consistent lesson: **read the baseline, list the directory, and -compare the payload** rather than reconstructing from memory. - ---- - -# UPDATE 20 (2026-10-13 ~17:40) — the pool lead is CLOSED structurally; five families/leads now - -Pursued the new lead from the REA dumps to its end, via three of my own corrections. - -## The lead, and where it ended - -`hrx_mem_pool_trim`'s decompilation showed `quarantine_eviction_count` and `bytes_quarantined` — plausible -"retire → reuse" vocabulary that could sit under the eviction trigger. Followed to the end: - -1. I concluded the pool was **unreachable** (engine never calls `hrx_mem_pool_*`). **Wrong** — the engine uses - a **slab pool** created via `hrx_mem_pool_backing_create_pool`; I had grepped the wrong symbol. -2. Corrected, the lead re-opened, and the pool the engine does use was read - (`hrx_iree_exact_pool_create`, `libhrx/src/libhrx/pool.c`, from `buffer.c:109`). -3. Its `trim` is a **no-op**; `acquire` marks every reservation `OK_FRESH` (straight to - `iree_hal_allocator_allocate_buffer`); `release` frees immediately and retains nothing. -4. So **there is no reuse window**, and `quarantine` in this runtime is the **AMDGPU HAL ASAN** feature - (redzones, shadow publication, release poisoning, a FIFO that deliberately retains freed mappings) — - **off in production**. - -Net: the lead closes for a **structural** reason — the pool holds nothing, so it cannot hand anything out -stale — not by absence of evidence. - -## What the goals' closed set now is - - logits read path (hrx_synchronous_d2h) REFUTED its own pre-registered A/B - host side (all four 0x52 accesses) CLOSED none on the decode path - runtime AQL submission policy CLOSED the runtime does not infer hazards - graph reorder (GGML_HRX_GRAPH_ORDER) REFUTED cell e 4/4 FAULT in BOTH arms - deferred host writeback CLOSED binding host_data == nullptr - memory-pool quarantine / reuse CLOSED no-op trim, fresh-only acquire, immediate release - FA alignment/dispatch (#140) OPEN untested (my exclusion was a parse error) - -and the **trigger** stands, quantified: ambient ~5% vs forced eviction ~95% under the frozen rig. - -## Also still true - -- **Criterion 4's controls are measured and passing** — 8/8 HRX tests, `MUL_MAT_ID` nmse matching baseline. -- **Criterion 2 remains not achievable within scope** — the structural argument is unchanged, and this - closure removes another candidate rather than opening one. -- The engine's own allocation path is now **read end to end** for this question: buffer allocate → exact pool - → fresh allocator buffer → immediate release. Nothing in it caches. - -## Harness/reasoning defects this session - -Now six: a line counter on newline-free NDJSON; a `pgrep` matching its own caller; `md5sum` over streaming -responses; guessed binary names; a `sudo -n tee` writing nothing; and the **symbol-search false negative** -above. The last is the same lesson as the others in a new guise — **check the symbol you grepped, not just -that the grep was empty.** - ---- - -# UPDATE 21 (2026-10-13 ~19:00) — the last open family is CLOSED; seven candidates excluded - -## The FA alignment test, finally run on a rig that reproduces the fault - -The FA family was the one open lead, left open because the earlier attempt used a **bespoke composition that -stayed mostly clean** (and because I had briefly misread its arms via a parse error). Fixed by using the -**frozen driver**, which reproduces the trigger at ~95%, and varying only `EXTRA_ENV`. - -Pre-registered at `ccf5b14`, run 17:28–18:50, 3 rounds per arm: - - cell q cell e cell c - arm A 2 OK / 1 FAULT 0 OK / 3 FAULT 3 OK / 0 FAULT align 4096 (default) - arm B 2 OK / 1 FAULT 0 OK / 3 FAULT 3 OK / 0 FAULT align 256 (#140 oracle) - -**Identical.** Cell `e` faulted **3 of 3 in both** at `gtt_start=98G`, forcer live, zero OOR/INVALID, with -both session headers recording the value actually applied. - -**Reading (one-sided, per the pre-registration's power note)**: the FA layout/dispatch class is **not -required** for the fault — the layout the #140 oracle used to *expose* the divergence faults at the same 3/3 -as the default. It is **not a measured null**: 3 arms per alignment cannot bound a rate difference -(distinguishing 95% from 60% needs ~20), so it lands at the same strength as the coherence and `GRAPH_SYNC` -rows. - -## The closed set, now seven - - logits read path REFUTED its own pre-registered A/B - host side CLOSED none on the decode path - runtime AQL submission policy CLOSED the runtime does not infer hazards - graph reorder REFUTED cell e 4/4 in BOTH arms - deferred host writeback CLOSED binding host_data == nullptr - memory-pool quarantine/reuse CLOSED no-op trim, fresh-only acquire, immediate release - FA alignment/dispatch (#140) NOT REQUIRED 3/3 both alignments on the frozen rig - -KNOWN-ISSUES updated to match (`58310d1`), replacing the FA row's OPEN status. - -## Criterion 4: op-test half done, dense half running - -- **op-test half MEASURED**: 8/8 HRX tests pass; `MUL_MAT_ID` nmse matching baseline (1.34–1.61e-05 vs - 1.23–1.58e-05). See `315-crit4-control-result.md`. -- **dense half RUNNING**: Qwen3-8B (dense, not MoE) on HRX0 under the same detector shape, as a **negative - control** — a clean result shows the apparatus does not fault everything in sight, which is what makes the - GLM arms' eviction-cell rate attributable to the model/shape. - -## Where the cause stands - -Still **not located**, but the exclusion surface is now much firmer than a week of prose: seven candidates -have been excluded by pre-registered experiment on the frozen rig or by mechanism read at source, and the -trigger is quantified (ambient ~5% vs forced eviction ~95%). Criterion 2 remains not achievable within scope. - ---- - -# UPDATE 22 (2026-10-13 ~19:10) — criterion 4 measured on BOTH halves; no fix branch exists anywhere - -## Criterion 4 is now fully measured - -| half | result | evidence | -|---|---|---| -| MUL_MAT_ID op-test + HRX suite | **8/8 pass**, nmse matching baseline (1.34–1.61e-05 vs 1.23–1.58e-05) | `315-crit4-control-result.md` | -| dense-model control (Qwen3-8B) | **clean**: 50/50 responses, 1 distinct content, 0 NaN, 0 Content-only, 0 HSA errors | same file | - -The dense half is the **negative control**: it shows the detector, build, HRX0 path and request shape do not -fault everything in sight, which is what makes the GLM arms' ~95% eviction-cell rate attributable to the -model/shape rather than the apparatus. - -Criterion 4 is still not *met* in its own terms — it pairs these controls with a fix to compare against, and -no fix has landed — but the baseline a future fix must match is now **measured and intact**. - -## No fix branch exists anywhere - -Checked every 315 branch on the remote: `1bit/315-known-issues-rate-and-closures` and -`docs/known-issues-315-pin-note` are documentation, and `engine-315/coherence-fix` is a **pin record, not a -repair**. So the two other agents sharing this goal are on documentation/rate evidence, not on a fix. - -## Where the goal stands, plainly - -| # | criterion | state | -|---|---|---| -| 1 | named, falsifiable mechanism | **trigger: MET** — named, replicated, quantified (ambient ~5% vs forced eviction ~95%; pooled p≈1.3e-17 on 29 eviction arms vs 57 controls); **cause: NOT LOCATED**, with the in-scope surface **enumerated and excluded** (7 candidates) | -| 2 | forward-only fix landed | **NOT ACHIEVABLE within scope** — a repair needs a distinction the engine cannot make (real bytes vs a pre-eviction copy); the runtime proposal recording this is written (`315-runtime-restore-fidelity-proposal.md`) | -| 3 | ≥60 in-regime arms, zero faults | **NOT MET, and the gate is mis-specified** — it must be the intervention cell with the ambient rate reported alongside, not a band on `gtt_start`; and a fixed-build campaign cannot run because there is no fix | -| 4 | no regression | **MEASURED on both halves** (above); not *met* pending a fix to compare against | -| 5 | KNOWN-ISSUES updated | **DONE**, re-corrected for this session's parse error | - -## This session's defects, all recorded rather than hidden - -Eight harness/parse defects: a line counter on newline-free NDJSON; a `pgrep` matching its own caller; -`md5sum` over streaming responses (which briefly reproduced a *false* family exclusion); guessed binary names; -a `sudo -n tee` writing nothing; a symbol-search false negative; my own monitoring preventing a run from -starting; and again a line counter on concatenated JSON (the dense control, where the script's own verdict was -the opposite of the truth). - -The consistent lesson, now stated three times on the record: **count occurrences not lines, read the baseline -rather than reconstructing it, and check the symbol/pattern you actually used.** - ---- - -## Session of 2026-10-13 evening: read fidelity, device-side instrumentation, and the sentinel recovered - -Add-only. Nothing above is edited; where this changes a reading, it says so. - -### The instrument itself was the subject, and it yielded three results - -1. **The read path is refuted, not the cause.** The instrument was patched to read every tensor twice on the - same line — plain memcpy and synchronising `hrx_synchronous_d2h` — and over ~138k observations **138,256 were - identical**. A plausible and attractive explanation (the host-visible memcpy shortcut in `buffer_get` returning - non-coherent memory, which would have implicated *every* host-side read in this campaign) is **dead on - evidence**. `315-idprobe-readpath-refuted.md`. - -2. **Arena slot reuse is why intermediate reads are unfaithful.** `ggml_argsort` allocates a fresh i32 tensor - (`ggml.c:5347`) and `ggml_argsort_top_k` merely views it, so after execution it must hold ranks; it holds a - *later* tensor's float data instead — because the fingerprint reads **after the whole graph has run** and the - allocator reassigns the slot of any tensor whose lifetime ended. `TransientArena` is one reused buffer, and - `GGML_HRX_DIAGNOSTIC_FRESH_TRANSIENT_ARENA` **does not help** (checked: it only allocates fresh on capacity - growth), which is worth knowing before anyone relies on it. - -3. **Faithfulness is positional.** The graphs are **per-layer**, so `tail:N` reads a layer's *last* nodes — the - least displaced — which is why this campaign's `tail:16` work produced coherent readings while name-matched - early nodes did not. **The `tail:N` methodology survives.** But its own first node is the exposed one: the one - named NaN stage in the record, `ffn_shexp-41`, sits at the **head** of its 16-node window, so that attribution - is weaker than the record presents. `315-which-readings-are-faithful.md`. - -### Two exclusions added, one of them recovered from a run that was never read out - -- **Stale / never-written logits — EXCLUDED for Face A.** The engine already ships the discriminator: - `GGML_HRX_SENTINEL=1` fills the 619520-byte logits allocation with payload NaN `0x7FC0DEAD`, and the sampler - guard reports the match count as `sentinel=`. The `315nan` session **armed it** (fill line present in **20 of 21 - arms**) and **all 17 fault lines read `sentinel=0`**: the device wrote the region, with NaNs. The field was - never read out at the time, so the campaign spent two days reasoning about a mechanism its own log had already - decided. `315-sentinel-preregistration.md`. -- **Device-side access violations — not detected, qualified.** `HRX_HAL_SANITIZER=asan` (→ `options->asan.enabled`, - `logical_device.c:241`) plus the engine's existing sink via `GGML_HRX_LOG_DEVICE_EVENTS=1` (installed before - `hrx_gpu_initialize`, `ggml-hrx.cpp:1371`) is a **device-side, kernel-attributed** channel that had never been - switched on. Three in-regime arms — two Face A, one Face B, all `gtt_start=100G` — produced **zero reports**. - Qualified: **no positive control was run**, and the sanitizer works at buffer/dispatch-binding granularity. - `315-asan-wiring-verified.md`. - -### A near-deterministic Face-B configuration, offered as a tool and not as evidence - -`EXTRA_ENV` with the 64-name `l_out` list on `315fp-build` faulted **6 of 6 arms, all Face B, all in regime, in -all three cells, at `req=1…11`** — against `req=6…54` for the Face-A arms this campaign has chased. The model's -own text is **valid UTF-8 and a mid-essay fragment**, i.e. a corrupted opening, consistent with the recorded -reading of Face B as the sampler drawing a finite but wrong distribution. It is absent in every clean and -baseline arm, so it is induced by the configuration; the instrument is a known inducer and this run reads 64 -tensors per graph. **A tool whose value depends on an untested hypothesis that it shares its mechanism with the -production fault.** `315-faceb-amplifier.md`. - -### Criterion table, updated - -| # | criterion | state after this session | -|---|---|---| -| 1 | named, falsifiable mechanism | **trigger: MET.** **Cause: still NOT LOCATED** — now with **9** candidates closed rather than 7 (stale/unwritten logits and sanitizer-visible OOB are new), and with a *named reason* why the remaining surface resists: no host-side read of a mid-graph tensor is trustworthy, and both faces reduce to "a value is wrong at a consumer" | -| 2 | forward-only fix landed | **still NOT ACHIEVABLE within scope**; the runtime proposal stands | -| 3 | ≥60 in-regime arms, zero faults | **NOT MET**; the Face-B amplifier may make fix-validation tractable without a 60-arm campaign per attempt — if its mechanism matches | -| 4 | no regression | **unchanged, measured on both halves** | -| 5 | KNOWN-ISSUES updated | **DONE** — two more rows, plus a corrected row for the id family | - -### The one sentence worth carrying forward - -**Both faces are now "a value is wrong when a consumer reads it", and no host-side read can say where it was -first written** — which needs a read whose fidelity does not depend on the allocator's reuse: per-node at -production, or device-side validation at the consuming op. The runtime's `USER = 0x8000` device-event range makes -the second route plausible; it is a new instrument, not a re-run. - -### Harness defects recorded this session (numbered as the campaign numbers them) - -**#10** an orphaned `315coherent-build` neighbour, parent PID 1, silently **reused as a regime neighbour** so the -composition baseline was not "box at rest". **#11/#12** `pkill`/`pgrep` on a script name kills wrappers, not the -run: the `attrib → paired → llama-server` chain kept faulting for ~25 more minutes and blocked a later run's -thermal gate at 1577 % CPU; and the same self-match trap fired **three more times** during cleanup, twice killing -my own session — the fix is to put kill patterns in a **script file** so no command line carries them. **#13** the -rig's default `BIN` build does not implement `GGML_HRX_TENSOR_FP` at all, so an `EXTRA_ENV`-driven sweep was -silently inert. **#14** a `NANONLY` sweep cannot distinguish "every layer clean" from "no Face-A arm occurred", -and the first sweep attempt was exactly that — reported as **mis-targeted, not as a negative**. Plus **five parse -slips** in one family: a truncated sorted head, a line counter on newline-free output, a sum over streaming -responses, a mis-indexed `awk` field test, and an unread `sentinel=` field. - ---- - -## Session of 2026-10-13 late: the site is named, the size is an invariant, and the read path that was being trusted is not - -Add-only. Nothing above is edited. - -### The unnamed node is source-addressable, and it is a post-attention residual ADD - -`ggml.c:7202-7203` names any node with an empty name `node_%d` using `cgraph->n_nodes`, so `node_2948` is the -node's position in the **full** forward graph. Because a graph's node count is per-**operation** and not -per-token, that index is **stable across sequence length** — demonstrated, not assumed: `node_2948` is the -first corrupted node at `T = 39, 87, 98, 135, 140, 172, 306, 342, 348`. - -Every `node_N` in the archive reduces to one of two residues mod 71: - -| residue | op | count | meaning | -|---|---|---|---| -| 37 | `ADD` | 47 | one unnamed residual add per layer | -| 3 | `FLASH_ATTN_EXT` | 47 | one unnamed attention op per layer | - -So the graph is **71 nodes per layer**, 47 layers, with exactly two unnamed nodes each — and - - node_2948 = 71×41 + 37 -> LAYER 41's unnamed post-attention residual ADD - node_2914 = 71×41 + 3 -> LAYER 41's unnamed attention op - -This fixes the layer-41 site by a second, independent method (graph arithmetic) alongside the layer-`N` names -in the same window. It also **withdraws** an intermediate guess filed during this session that the node was a -MoE expert-accumulation add — the per-layer op pair is attention + residual add, and 47+47 accounts for the -model exactly. - -### The size is an invariant: one row, `nan = 2048`, in 18 of 19 arms - -`node_2948` reads **exactly `2048` NaN floats — one `n_embd`-wide row, `2048 × 4 = 8192` bytes** — in **18 of -19** archived arms, against extents from 79,872 to 712,704 floats and **nine different sequence lengths**. The -19th arm is `315nan-p12l`, the one arm whose NaN is already present at layer 1 (`node_74`, `node_108`, …; first -line `norm-0 nan=55 of 20480`), where the tensor is wholly NaN because the corruption predates layer 41. - -So the two populations are separable **by the first-corrupted layer**, and only one of them is a layer-41 -onset. This also corrects the wording of the row-size prediction: "position" ran together two different -things. Size — fixed. Layer — fixed at 41. Row within the tensor — **varies** (the same `T=87` count in five -arms, five different content hashes; and the ext-fp arms read `row = 0, 1, 0`). The claim the data supports is -**"one row, always at layer 41, in a row that moves"**. - -### The `l_out-N` read is not a per-layer measurement - -Grouping each arm's `l_out-N` reads by content hash shows two **different layers returning byte-identical, -largely-finite content**: `l_out-42 ≡ l_out-44` and `l_out-43 ≡ l_out-45`, both at `nan=19968 of 79872` (25 % -NaN) in the `315fbar` cold-cell arms. Two layers cannot genuinely produce identical 319 KB outputs, so these -are **aliased reads** — the same arena slot reached under two names. Scope is stated because it matters: the -inference is sound **only for unsaturated reads**. In `315lout4-r1e` a stride-3 family -(`l_out-15 18 21 24 27 30 33 36 39 42 45`) shares one hash, but every member reads `nan=200704 of 200704` — -fully saturated — so saturation explains it and it is **not** evidence of aliasing. The stride is not -established to be universal. - -The unnamed `node_NNNN` reads do not share the defect: the only coincident pair anywhere is -`node_3304 ≡ node_3305`, and `node_2948` never coincides. **Practical rule: two readings with the same hash -are one reading.** - -### The `tail:16` yield did not reproduce, and `315row` ended at two clean arms - -`315row` (4 rounds, `tail:16` + probes) recorded **round 1 arm q OK (gtt_start 97G) and round 1 arm e OK -(98G)** — the eviction cell in regime with `side=1/1` — then was superseded. `0ed67c2` amends the test to the -config that has actually produced Face-A arms (`l_out` name lists without the extra probes) and reports the -`tail:16` attempt as a **failed attempt, not a negative**. Both readings agree: the same `EXTRA_ENV` that -recorded 16/17 did not reproduce here. - -### Criterion table, updated - -| # | criterion | state after this session | -|---|---|---| -| 1 | named, falsifiable mechanism | **site named and independently derived**: layer 41's unnamed post-attention residual ADD, one 2048-wide row, 18/19 arms. **Cause still not named** — the add's two inputs (layer-41 residual, attention output) and the buffer behind them are not distinguished, and layer 41's attention op reads clean in the 18 one-row arms without the window-coverage being proven | -| 2 | forward-only fix landed | unchanged: not achievable within scope until the buffer question is settled | -| 3 | ≥60 in-regime arms, zero faults | unchanged; `315size` in flight is the amendment's attempt | -| 4 | no regression | unchanged, measured on both halves | -| 5 | KNOWN-ISSUES updated | this section is the status record; the KNOWN-ISSUES edit remains with the session that owns it | - -### Evidence - -`dev/315/315-campaign/evidence/315-layer41-site-by-graph-arithmetic.md`, -`315-row-size-scored-from-archive.md`, `315-node-2948-is-an-unnamed-moe-accumulation.md` (with its withdrawal -note), and the coordination note in `315-COORDINATION.md`. Every number above is reproduced by a command in -one of those files; none required GPU time, all were read from the archived arm logs at pin `e44c9d01a`. - -### Correction to the section above: the onset layer is configuration-dependent - -The section above reports the layer-41 site from the probe-heavy config (`tail:16` + `COPY_PROBE` + -`SENTINEL` + `LOGITS_FP`). A second working configuration says otherwise, and the same day's peer run supplies -it: **`315size`** (15-name `l_out` list, `NANONLY`, **no probes**) puts the onset at **layer 0** in all three -of its Face-A arms — `l_out-0 nan=1984 of 276480` (r2e, call 40377), `1984 of 503808` (r1e), `1790 of 20480` -(r2q) — each in its own per-layer call, growing to saturation by layer 4. - -The layer-41 reading is not wrong for its own data: in `315nan-p2l` the first printed line is `call=152939`, -and with `NANONLY=1` and all calls fingerprinted, that means ~152,938 earlier graphs produced no NaN line, so -layers 0–40 were watched and clean. Both are true. What is **not** true is that the site is layer 41 -unqualified: **the observable onset moves with the instrument set**, which is the recorded -"instrumentation is an inducer" note showing up as a change of *layer*, not merely of face. - -Two things therefore narrow: - -- **criterion 1's "site is named"**: it is named *under the probe-heavy configuration*. The claim to be *the* - site is withdrawn pending the config-only test below. -- **the row-size invariant** (`nan=2048` exactly, 18 of 19 arms) is an invariant **of that configuration**. - `315size`'s onset counts are not one row (`1984`, `1790`), though `1984 = 2048 − 64` recurs in both. - -**The discriminating experiment this implies, and it is config-only:** run the same fingerprint window on both -configurations in the same session at the same regime, and see whether the onset layer moves with the probe -set. That decides whether "layer 41" is the defect or the instrument. - -**Caveat on the `315size` comparison:** that run was thermally saturated (`thermal_pauses` to 29, `tctl` to -95 °C, `stopped_frac` to 0.82, gate waits of 10–50 min) and it **faulted in the quiet cell** (round 2 arm q, -`nan=1`, `side=0/0`), so its cells are not a clean contrast. Its arms were run by the peer session, read here -only. - -Evidence: `dev/315/315-campaign/evidence/315-onset-layer-is-config-dependent.md`. - ---- - -## Session of 2026-10-14 early: the site is instrument-relative, the size is not, and Face B was never measured - -Add-only. Nothing above is edited. - -### The site is instrument-relative — and the read pattern, not the probes, decides it - -Three configurations already on the record separate the two candidate variables: - -| condition | arms | selector | probes | first NaN at | -|---|---|---|---|---| -| A | `315size` | 15-name `l_out` list | none | **layer 0** | -| B | `315lout4`, `315inp2` | `l_out` name list | `COPY_PROBE`+`SENTINEL`+`LOGITS_FP` | **layer 0** | -| C | `315nan`, `315fbar` | **`tail:16`** | same three probes | **layer 41** | - -A vs B holds the selector and varies the probes: onset does not move. B vs C holds the probes and varies the -selector: onset moves 0 → 41. So **the read pattern sets the onset**, and `hrx-producer` shows `graph n_nodes=9`, -making `tail:16` cover **every** node of a subgraph (~9 reads/chunk) against ~1–3 for a name list. That also -explains why `tail:16`'s recorded yield did not reproduce: reading every node is a different regime. - -`315cfg` (the controlled A′-vs-B run) was **stopped with no arm run** — condition B already existed as -`315lout4`, and the gate stayed shut on `load1 ≈ 9` against a required <4. Recorded as a stop, not a pruning. - -### What survives: the size, not the site - -| condition | onset count | bytes | rows | -|---|---|---|---| -| C | 2048 | 8,192 | 1.000 | -| B | 1984, 1984, 2048 | 7,936 / 8,192 | 0.969, 1.000 | -| A | 1984, 1984, 1790 | 7,936 / 7,160 | 0.969, 0.874 | - -**Five of the seven non-saturated onset readings lie in 7,936–8,192 bytes — one row ± one 256-byte block** — -across three configurations, two rigs, `T = 39…348`. `nan = 2048` *exactly* is a property of condition C, not -of the defect. And in all four arms where `hrx-finite-fp` printed, **`finite + nan` equalled the extent -exactly**, so the count comes from a complete read. Any "the corruption is at layer N" claim is -instrument-relative; "the corruption is about one row" is not. - -### Face B: never measured with a capable instrument, and never invalid UTF-8 - -- **No arm on the record carries both a fingerprint and Face-B evidence.** The fingerprint's only gate is - `if (fp_nan_only && nan == 0) continue;`, so an arm with fingerprint lines has NaN and an arm with Face B has - none — the two faces are **mutually exclusive in the instrument's output**. Face B's requirement is therefore - not addressable from existing data. -- **Face B has an invariant signature** across 15 arms: `nan=0 http=1 s500=1 unparsed=2`, in **all three cells - including the quiet cell** (`side=0/0`, three runs) and even one `OOR` arm — so it is not eviction-specific - either. Counting subtleties: `unparsed=2` is a *log-line* count for one event, `http=1 s500=1` is the same - failure counted twice, and `req < 66` alone forces `FAULT`. -- **Invalid UTF-8 has never been observed.** Of 3,200 request records and 267 greedy-output files, zero bodies - are invalid UTF-8; the three apparent cases are 32-byte *excerpt* truncations (`utf8_ok=True json_ok=True` on - the full bodies), and the 34 `utf8_ok=False` records are transport failures with no body. Face B as recorded - is **Content-only / non-JSON output answered with HTTP 500**. -- A third-manifestation hypothesis ("wrong but well-formed output at HTTP 200") was **tested and withdrawn**: - the maximum repetition ratio is reached by clean quiet-cell arms, so degeneracy is a property of the prompts. - -### Criterion table, updated - -| # | criterion | state after this session | -|---|---|---| -| 1 | named, falsifiable mechanism | **site is NOT nameable** — it moves with the read pattern; the instrument-independent fact is the **size** (~one row, 7,936–8,192 B). Both faces still lack a cause | -| 2 | forward-only fix landed | unchanged: not achievable until the buffer question is settled; the instrument-relative site makes a layer-targeted fix unsound | -| 3 | ≥60 in-regime arms, zero faults | unchanged; and the acceptance detector must catch **Content-only/HTTP 500**, since a UTF-8 check would never fire | -| 4 | no regression | unchanged, measured on both halves | -| 5 | KNOWN-ISSUES updated | status recorded here; the `KNOWN-ISSUES.md` edit remains with the session that owns that file | - -### Evidence - -`315-instrument-independent-summary.md`, `315-onset-is-set-by-read-pattern-not-probes.md`, -`315-subgraphs-are-9-nodes-so-tail-covers-all.md`, `315-faceb-invariant-signature.md`, -`315-faceb-has-no-capable-instrument.md`, `315-invalid-utf8-never-observed.md`, -`315-1984-reconciles-the-two-populations.md`, plus the coordination notes in `315-COORDINATION.md`. - -## Session of 2026-10-15: closed as a bounded negative — the ordering family removed, one site shown unmeasurable, criterion 4 measured at the moved pin - -**Resume here.** This supersedes every criterion table above it. The objective was re-scoped to accept a -bounded negative (a named mechanism OR every surviving candidate eliminated by its own pre-registered -experiment), and this session finished on that branch: **no mechanism is claimed.** - -### Task state, this session's numbering - -| task | state | result | -|---|---|---| -| task-1 — sync-between-reads | **complete** | the stream-ordering candidate is **REMOVED** (below) | -| task-2 — §51 argsort precondition | **complete, bounded negative** | **not measurable** by the by-name fingerprint | -| task-3 — no regression | **complete** | CI **7 of 7 green**; both controls **unchanged**, at the moved pin | -| task-4 — report + KNOWN-ISSUES | **complete** | `KNOWN-ISSUES.md` updated, status stays **OPEN** | - -### The measurement that removed the last ordering candidate - -On a faulting arm (Face A, in-regime `gtt_start 98G`, 312 records) the same tensor was read three ways at the -same call — plain, plain again, and after a **fresh `ggml_backend_synchronize(backend)`**: - -| quantity | value | -|---|---:| -| diverged (`read#1 != read#2`) | **82** | -| of those, a fresh sync **stopped** it | **0** | -| of those, it did **not** stop it | **82** | -| the sync **changed the value itself** | **82 of 82** | - -So the writer of the `result_norm` slot is **not ordered by `context->stream`**, which removes -`GraphExecutor::execute` returning without a wait while `backend_synchronize` covers only `context->stream`. -`315-task1-writer-not-stream-ordered.md`. Pre-registered at `315-PREREG-sync-between-reads.md` before the run. - -### §51's precondition is not measurable by this instrument - -The faulting arm produced **zero** records for `ffn_moe_probs-0`. The `tail:16` census of a real decode run has -**946 distinct node names** and contains `ffn_moe_out-*` and `ffn_shexp-*` but **none** of `ffn_moe_probs-*`, -`ffn_moe_topk-*`, `ffn_moe_argsort-*`, `ffn_moe_weights-*`. A selector watching that name reads nothing and -looks exactly like "zero NaN". This is a negative about the **instrument**, not about the site, and it also -means **how §50 read that tensor is itself unexplained** — recorded as an open discrepancy. -`315-task2-bounded-negative-precondition-unmeasurable.md`. - -### The elimination list, and what survives - -Eliminated, each by its own pre-registered experiment with its command and observed output on the record: -deferred host writeback (626 reads, 0 pending) · persistent-mapping / coherence (A/B at two measured addresses, -13.7 % / 14.8 %; and closed from source for both host paths) · the unwaited async logits read (its campaign's -first fix arm faulted, Face B) · **the stream-ordering explanation** (this session). - -**Surviving family, stated as surviving:** an open writer of the `result_norm` slot that is *not* ordered by -`context->stream` — the reused-arena-slot family in `315-what-the-synchronise-drains.md` — plus §51's rank-sort -precondition, unmeasured. - -**Handed forward, one measured unexplained fact:** `result_norm` changes between two consecutive identical -plain reads in a faulting arm, at two different addresses, with no pending host writeback at the engine's -logits read, and a fresh synchronise failing to stop it. - -### Criterion 4, measured at the moved pin and not the old one - -| half | result | -|---|---| -| engine CI | **green, 7 of 7** — PR **#414**, `mergeable=MERGEABLE`, `state=CLEAN` | -| HRX suite + `MUL_MAT_ID` op-test | **unchanged** — `8 pass, 0 skip, 0 fail`; mxfp4 nmse `1.34e-05 … 1.61e-05`, exact match; same two kernels | -| dense-model control | **clean, unchanged** — 50 responses, 1 distinct content, zero fault signatures | - -`315-task3-op-test-unchanged-at-moved-pin.md`, `315-task3-dense-control-clean-at-moved-pin.md`. - -**CI did not pass on arrival.** It required fixing two pre-existing defects: `95238dc` was a **half-move** (it -moved the `llama.cpp` gitlink without `registry/architectures.json`) and 46 files lacked the full copyright -notice. No gate was loosened. `315-what-the-merge-and-the-gates-found.md`. - -### Forward-only, now stronger than recorded - -`third_party/llama.cpp` = `47a8fe929`, whose **parent is `0e255d774`** — engine `main`'s current pin — so the -change sits exactly one commit ahead of today's main. `git merge-base --is-ancestor e44c9d01a 47a8fe929` is -true, and `check_pins.py` reports "ahead". The fork change is 5 added lines in `src/llama-context.cpp` ending -in `ggml_backend_synchronize(backend_res)`. It stays landed: a correct change for the reason its comment gives, -even though its own campaign showed it is **not sufficient** to stop the fault. - -### Harness defects, four of one class this session - -All are "a count read as a verdict without checking the count can be non-zero for the reason assumed", and each -corrupted verdict pointed at a pass not earned or a fault that did not happen: eight silently `skipped` tests -all exiting 0 and reading as 8/8 PASS; `grep -c … || echo 0` producing `"0\n0"` and making a clean dense run -print `FAULT or incomplete`; a line-based response counter reading 50 as 1; and a CI watcher printing **GREEN** -on a branch whose checks had not started. All four are reproduced with their fixes in -`315-task3-dense-control-clean-at-moved-pin.md` and are also written up for other sessions in -`315-COORDINATION.md`. - -### Box - -The strixhalo box is **released and idle**: lock FREE, no owned servers, no forcer, GTT back to its 27 GiB -baseline. A landing note for any peer arriving on the box is at `~/wt/315-HANDOFF.md`. diff --git a/dev/315/315-campaign/315-HANDOFF-box.md b/dev/315/315-campaign/315-HANDOFF-box.md deleted file mode 100644 index c5403c528..000000000 --- a/dev/315/315-campaign/315-HANDOFF-box.md +++ /dev/null @@ -1,167 +0,0 @@ - - -# 315-HANDOFF — landing state for whoever arrives on this box next - -Written 2026-10-15 by the #315 goal session when it closed out. **Read this before launching anything here.** -The full record is in the repo worktree `~/wt315-prereg` (branch `1bit/315-preregistration`, PR #414 on -`1bit-MONSTER/engine`); `dev/315/315-STATUS.md` is its single entry point and `315-COORDINATION.md` is the -inter-session channel. - -## 1. Box state, handed over clean - -``` -box.lock FREE -GTT used 27 GiB (idle baseline) -owned servers none evictonly forcer none hunt scripts none -``` - -Nothing of mine is running. The lock is free. If you start a run, take the lock first: - -``` -flock -n 9 || echo BUSY; exec 9>/home/bcloud/.cache/lax-decode/box.lock # hold for the run's duration -``` - -**One thing I changed outside my own files, so you know:** cron **line 22** of this user's crontab — the -`strix-start-internals.sh` entry that was launching **12 redundant `run-generic.sh` jobs on `libloomc`** every -5 minutes — is **commented out**, labelled `# DISABLED 2026-10-15 by goal turn (redundant: decomp/bulk/loomc -covers it 9744/9744)`. A backup of the crontab as it was is at -`/home/bcloud/crontab.backup-20261015-internals.txt`. **To restore it, copy that file back with `crontab`.** -It was consuming the box continuously and made every in-regime arm unstable; nothing else in the crontab was -touched. - -## 2. Where #315 stands — a BOUNDED NEGATIVE, not a mechanism - -Do not read the record as if a cause was named. **No mechanism is claimed.** At issue #315's own request the -objective was re-scoped to accept a bounded negative, and it was finished on that branch. - -**Eliminated**, each by its own pre-registered experiment, with the command and observed output on the record: - -| family | how it died | -|---|---| -| deferred host writeback | 626 reads at the engine's logits read, **0** pending | -| persistent mapping / coherence | two-cell A/B at two measured addresses, 13.7 % / 14.8 % — and closed **from source** for both host paths | -| the unwaited async logits read | its own pre-registered campaign's **first fix arm faulted** (Face B, HTTP 500) | -| **the stream-ordering explanation** | of **82** records where `result_norm` differed between two consecutive plain reads, a **fresh `ggml_backend_synchronize` stopped it in 0** and not in 82, and **changed the value itself in 82 of 82** | - -**Surviving family, stated as surviving:** an open writer of the `result_norm` slot that is **not** ordered by -`context->stream` — the reused-arena-slot family — plus §51's rank-sort precondition, which is **unmeasured**. - -**Handed forward, one measured unexplained fact:** `result_norm` changes between two consecutive identical -plain reads in a faulting arm, at two different addresses, with **no pending host writeback** at the engine's -logits read, and with a fresh synchronise failing to stop it. **That is the thing to explain.** - -## 3. Which build is which — and why it matters now - -The pin moved this session, so **a result measured on an `e44c9d01a` build no longer describes what ships.** - -| build dir | source tree | pin | test-hrx binaries | what it is | -|---|---|---|---|---| -| `~/wt/315logwait-build` | `llama-315logwait` | **47a8fe929** | **8** | **the current build.** Carries the landed fix; has the eight HRX tests built (needed `-DLLAMA_BUILD_TESTS=ON`, see §5) | -| `~/wt/315fp-build` | `llama-315copy` | e44c9d01a | 0 | instrumented **old** pin: tensor-fp twin + read-twice control + the sync-between-reads instrument | -| `~/wt/315probe-build` | `llama-315probe` | e44c9d01a | 0 | old-pin probe build | -| `~/wt/315coherent-build` | `llama-315coherent` | ? | 8 | older control build | -| `~/wt/v315-build` | `llama-hybrid` | ec0175971 | 8 | the **old** test build the 2026-10-13 criterion-4 baseline was measured on | - -**The fix, exactly:** fork commit `47a8fe929` on `1bit-MONSTER/llama.cpp` branch `1bit/315-logits-read-wait` — -**5 added lines** in `src/llama-context.cpp`, the last being `ggml_backend_synchronize(backend_res)`. Its -**parent is `0e255d774`**, which is engine `main`'s current pin, so it sits exactly one commit ahead of today's -main; `git merge-base --is-ancestor e44c9d01a 47a8fe929` is true. `check_pins.py` reports "ahead". It is -**not** a revert, and it is **not sufficient** to stop the fault — it is a correct change for the reason its -comment gives. - -## 4. The four traps that have each cost a reading — build them out of your harness - -All four are one shape: **a count read as a verdict, without checking the count can be non-zero for the reason -the verdict assumes.** In every case the corrupted verdict pointed at a pass not earned or a fault that did not -happen. - -1. **A skip is not a pass.** Without `LD_LIBRARY_PATH` and `IREE_HAL_AMDGPU_LIBHSA_PATH` set, **all eight** - `test-hrx-*` binaries **exit 0** while printing `no HRX0 device, skipped`. An exit-status gate reports 8/8 - PASS on a machine that ran nothing. Count `skip` separately, always. -2. **`grep -c … || echo 0` yields `"0\n0"`.** `grep -c` prints `0` *and* exits 1 on no matches, so the fallback - also runs; the value becomes two lines and `[ "$N" -eq 0 ]` errors to the *else* branch. This made a **clean** - dense-control run print `RESULT: FAULT or incomplete`. Drop the `|| echo 0`. -3. **Count occurrences, not lines, on `/v1/chat/completions`.** Responses are concatenated without newlines, so - `grep -c '"content"'` reads 50 responses as **1**. Use `grep -o … | wc -l`. -4. **`pgrep -f ""` matches your own monitoring one-liner.** A wait-loop for "no drivers running" never - fires while the command asking contains the pattern. Read `/proc//cmdline` and skip `bash -c` wrappers. - The same flaw made a CI watcher print **GREEN** on a branch whose checks had not started, because - `no checks reported on the … branch` contains no `pending`. - -Two more that cost arms: - -* **`EXTRA_ENV` splits on commas** (`IFS=',' read -r -a`), so only the **first** fingerprint name survives. Use - a position selector (`tail:` / `head:`) for several names; `tail:16` covers `attn_norm-*` but **not** - the `ffn_moe_*` family. -* **A name selector that matches nothing looks exactly like "zero NaN".** The `tail:16` census of a real GLM - decode run has **946 distinct node names** and contains **none** of `ffn_moe_probs-*`, `ffn_moe_topk-*`, - `ffn_moe_argsort-*`, `ffn_moe_weights-*` — while `ffn_moe_out-*`, `ffn_shexp-*`, `norm-*`, `attn_norm-*`, - `q-*`, `k_pe-*`, `Qcur-*`, `Kcur-*`, `q_nope-*`, `l_out-*`, `kv_cmpr-*`, `ffn_gate-*`, `ffn_up-*`, - `ffn_swiglu-*` and `ffn_out-*` are all present. Check your selector matched **before** reading its zeros. - -## 5. Re-runnable commands - -**The HRX suite and the `MUL_MAT_ID` op-test at the current pin** (this is criterion 4's second half): - -``` -cd ~/wt/315logwait-build -export LD_LIBRARY_PATH=$PWD/bin -export IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 -for t in test-hrx-buffer test-hrx-mul-mat-id-k32 test-hrx-mxfp4 test-hrx-moe-split \ - test-hrx-attention-sink test-hrx-decode-stride test-hrx-hadamard test-hrx-fa-masked-v; do ./bin/$t; done -./bin/test-hrx-mul-mat-id-k32 # the nmse lines: mxfp4 must read 1.34e-05 … 1.61e-05 -``` - -Expected, and measured this session: `8 pass, 0 skip, 0 fail`; `test-hrx-buffer` `5 passed, 0 failed` with the -same five case names; mxfp4 nmse `min=1.34e-05 max=1.61e-05`; the same two kernels -(`ggml_mul_mat_id_decode_f32_wave64`, `ggml_mul_mat_id_f32_f32_wmma`). - -**To build those tests at a new pin:** `BUILD_TESTING=ON` alone is **not enough** — it gives -`No rule to make target 'test-hrx-buffer'`. The real gate is `LLAMA_BUILD_TESTS`: - -``` -cmake -S ~/wt/llama-315logwait -B ~/wt/315logwait-build -DBUILD_TESTING=ON -DLLAMA_BUILD_TESTS=ON -make -j16 test-hrx-buffer test-hrx-mul-mat-id-k32 test-hrx-mxfp4 test-hrx-moe-split \ - test-hrx-attention-sink test-hrx-decode-stride test-hrx-hadamard test-hrx-fa-masked-v -``` - -**The dense-model control** (a *negative* control — it must NOT fault): - -``` -~/wt/315dense3.sh # Qwen3-8B-Q4_K_M on HRX0, -ngl 99 -fa on -c 4096, 50 mixed-length requests -``` - -Its script's own verdict line is **untrustworthy** (trap 2 above, still in the script as it ran — left that way -so its output stays reproducible). Read the counts, not the verdict: correct is `responses=50`, -`distinct_content=1`, `nan_guard=0`, `hsa=0`, `content_only=0`. - -**The in-regime harness** (if you are running arms): `~/wt/315-regime.sh`, `~/wt/315-fp-hunt.sh`, -`~/wt/315-detector.py`, `~/wt/315-paired.sh`, `~/wt/evictonly` (the `mprotect`-churn eviction forcer). The -regime gate is `gtt_start ∈ [95,110] GiB`; this box's in-regime point is **98G** and neighbour granularity is -~17 GiB. Both faces must be counted in every arm, and **arms that die for harness reasons are invalid, never -passes** — while an arm truncated as a *consequence* of the defect (a 500 the model caused) **is a fault**. - -## 6. What not to trust - -* Any control result measured on an `e44c9d01a` build as if it described the shipped pin. -* Any name-selector result whose name you have not confirmed appears in the run's node census. -* Any "zero NaN" that came from a selector matching nothing, or from a suite that skipped. -* The dense control script's verdict line, and any `grep -c … || echo 0` count. -* `pgrep -f` counts taken from a shell that contains the pattern. -* Stale run directories under `~/wt/rt-det/` from **diagnostic, eviction-forced** cells — those arms are - labelled as diagnostic and **no arm of theirs may be counted toward a campaign's arm count**. diff --git a/dev/315/315-campaign/315-analyse.py b/dev/315/315-campaign/315-analyse.py deleted file mode 100755 index 4158d0977..000000000 --- a/dev/315/315-campaign/315-analyse.py +++ /dev/null @@ -1,194 +0,0 @@ -#!/usr/bin/env python3 -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""engine#315 pre-registered analyser. - -Reads the campaign console produced by `315-paired.sh` and applies the classification fixed in -`dev/315/315-preregistration.md` — which differs from `issue-315-analysis.py` in one decisive way: - - Face B (invalid UTF-8 / JSON parse failure) is a FAULT face, not "harness-side truncation". - The issue's own record characterises invalid-UTF-8 output as the second real defect, and - "zero NaN" alone was explicitly ruled out as an acceptance result. - -Console line grammar (produced by 315-paired.sh; the existing script's regex also matches it): - - pair arm : nan= req= ret3= parse= gtt_start=G utf8= alloc= - -Classification: - invalid alloc>0, verdict INVALID, or the request phase produced nothing (req==0) - out-of-regime gtt_start < 95 GiB (gate) or > 110 GiB (invalid-upper bound) - fault nan>0 or hsa>0 or parse>0 or utf8>0 or idn>1 or verdict in {FAULT,TRUNCATED} or req<66 - clean otherwise - -Usage: 315-analyse.py [...] [--baseline-a 2/37] [--baseline-b 1/25] -Exit status 0 only if the acceptance criterion is met: >=60 in-regime clean arms in every -non-control cell AND zero faults among the in-regime arms. -""" -from __future__ import annotations - -import math -import random -import re -import sys -from math import comb - -ARM = re.compile( - r"^\s*pair\s+(\d+)\s+arm\s+([cl]):\s+(\S+)\s+nan=(\d+)\s+req=(\d+)\s+ret3=(\d+)\s+parse=(\d+)" - r"\s+gtt_start=(\d+)G(?:\s+utf8=(\d+))?(?:\s+alloc=(\d+))?(?:\s+idn=(\d+))?") -GATE = 95 # GiB, pre-registered -UPPER = 110 # GiB, above this the detector fails to allocate (invalid, not faulted) -CELL_NAME = {"c": "control", "l": "fix"} -SEED = 20261007 - - -def parse(path): - arms = [] - with open(path, errors="replace") as fh: - for line in fh: - m = ARM.match(line) - if not m: - continue - pair, cell, verdict, nan, req, ret3, parse_, gtt, utf8, alloc, idn = m.groups() - arms.append(dict(pair=int(pair), cell=cell, verdict=verdict, nan=int(nan), req=int(req), - parse=int(parse_), gtt=int(gtt), utf8=int(utf8 or 0), - alloc=int(alloc or 0), - # no idn= field (a console written before the 2026-10-10 freeze note) reads as - # 1, i.e. no signal, so historical consoles keep their classification - idn=int(idn if idn is not None else 1))) - return arms - - -def classify(a): - """Pre-registered classification. Order matters: invalid, then regime, then fault, then clean.""" - if a["alloc"] > 0 or a["verdict"] == "INVALID" or a["req"] == 0: - return "invalid" - if a["gtt"] < GATE or a["gtt"] > UPPER: - return "oor" - if (a["nan"] > 0 or a["verdict"] in ("FAULT", "TRUNCATED") - or a["parse"] > 0 or a["utf8"] > 0 or a["idn"] > 1 or a["req"] < 66): - return "fault" - return "clean" - - -def classify_old_rule(a): - """The recorded study's rule, for comparability: Face B counted as harness-side, not faulted.""" - if a["nan"] > 0: - return "fault" - if a["verdict"] == "TRUNCATED" or a["req"] < 66 or a["parse"] > 0 or a["utf8"] > 0: - return "trunc" - return "clean" - - -def wilson(k, n, z=1.96): - if n == 0: - return (0.0, 0.0) - p = k / n - d = 1 + z * z / n - c = (p + z * z / (2 * n)) / d - h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d - return (max(0.0, c - h), min(1.0, c + h)) - - -def rule_of_three(n): - """95% upper bound on the rate when 0 events are observed in n trials (capped at certainty).""" - return min(1.0, 3.0 / n) if n else 1.0 - - -def fisher_two_sided(a, nA, b, nB): - N, K = nA + nB, a + b - if K == 0 or K == N: - return 1.0 - - def prob(x): - if x < 0 or K - x < 0 or x > nA or K - x > nB: - return 0.0 - return comb(nA, x) * comb(nB, K - x) / comb(N, K) - - p0, tot = prob(a), 0.0 - for x in range(max(0, K - nB), min(nA, K) + 1): - p = prob(x) - if p <= p0 + 1e-12: - tot += p - return min(1.0, tot) - - -def p_rate_a_below_b(a, nA, b, nB, draws=200000): - random.seed(SEED) - hit = 0 - for _ in range(draws): - if random.betavariate(1 + a, 1 + nA - a) < random.betavariate(1 + b, 1 + nB - b): - hit += 1 - return hit / draws - - -def main(argv): - paths = [a for a in argv if not a.startswith("--")] - base_a, base_b = (2, 37), (1, 25) - for arg in argv: - if arg.startswith("--baseline-a"): - base_a = tuple(int(x) for x in arg.split("=")[1].split("/")) - if arg.startswith("--baseline-b"): - base_b = tuple(int(x) for x in arg.split("=")[1].split("/")) - - arms = [a for p in paths for a in parse(p)] - if not arms: - print("no arms parsed — check the console path and the line grammar") - return 2 - - print(f"arms parsed: {len(arms)} from {len(paths)} console(s)\n") - cells = {} - for cell in sorted({a["cell"] for a in arms}): - mine = [a for a in arms if a["cell"] == cell] - counts = {"invalid": 0, "oor": 0, "fault": 0, "clean": 0} - old = {"fault": 0, "trunc": 0, "clean": 0} - for a in mine: - counts[classify(a)] += 1 - old[classify_old_rule(a)] += 1 - inreg = counts["clean"] + counts["fault"] - lo, hi = wilson(counts["fault"], inreg) if inreg else (0.0, 1.0) - cells[cell] = dict(total=len(mine), inreg=inreg, **counts) - print(f"cell {CELL_NAME.get(cell, cell)} ({cell}): scheduled={len(mine)} " - f"in-regime={inreg} clean={counts['clean']} fault={counts['fault']} " - f"invalid={counts['invalid']} out-of-regime={counts['oor']}") - if inreg: - rate = 100.0 * counts["fault"] / inreg - print(f" fault rate {counts['fault']}/{inreg} = {rate:.1f}% 95% CI {100*lo:.1f}-{100*hi:.1f}%") - if counts["clean"]: - ub = 100.0 * rule_of_three(counts["clean"]) - print(f" if clean count stands: 95% upper bound on the true rate = {ub:.1f}% " - f"(rule of three over {counts['clean']} clean arms)") - print(f" under the recorded study's rule (Face B = harness-side): " - f"clean={old['clean']} fault={old['fault']} trunc={old['trunc']}") - - if "c" in cells and "l" in cells: - c, l = cells["c"], cells["l"] - p = fisher_two_sided(c["fault"], c["inreg"] or 1, l["fault"], l["inreg"] or 1) - pb = p_rate_a_below_b(c["fault"], c["inreg"] or 1, l["fault"], l["inreg"] or 1) - print(f"\ncontrol vs fix: Fisher exact two-sided p = {p:.3f}; " - f"P(rate_control < rate_fix) = {pb:.3f}") - print(f"recorded baseline for reference: A {base_a[0]}/{base_a[1]} " - f"({100*base_a[0]/base_a[1]:.1f}%), B {base_b[0]}/{base_b[1]} " - f"({100*base_b[0]/base_b[1]:.1f}%)") - - fix = cells.get("l") or cells.get("c") - ok = fix["clean"] >= 60 and fix["fault"] == 0 - print("\nacceptance (fix cell): >=60 in-regime clean arms AND zero faults on both faces") - print(f" clean={fix['clean']} (need 60) faults={fix['fault']} (need 0) -> " - f"{'PASS' if ok else 'NOT MET'}") - return 0 if ok else 1 - - -if __name__ == "__main__": - sys.exit(main(sys.argv[1:])) diff --git a/dev/315/315-campaign/315-attrib.sh b/dev/315/315-campaign/315-attrib.sh deleted file mode 100755 index b068f8e97..000000000 --- a/dev/315/315-campaign/315-attrib.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 diagnostic: attribute the fault rate the instrumented arm hunt showed. NOT an acceptance -# run - it changes no registered criterion, produces no eligible arms, and must never be cited as one. -# -# Why it exists: the instrumented hunt (evidence/315-instrumented-arm-hunt.md) faulted 16 of 17 fix-cell -# arms and its one control arm, against a recorded 2/37. Before any stage name from that run is treated -# as a mechanism, the rate has to be attributed. This runs the *same binary pair* with CONTROL_EVERY=1 -# so a stock-build arm and an instrumented arm alternate inside one composition and one box state: -# - stock cell faults rarely, fix cell often -> the environment (SENTINEL / TENSOR_FP) induces it, -# and the stage names are where those reads land, not where the production fault is born -# - both cells fault often -> the build or the box state is the cause, not the env -# - neither faults -> the 16/17 is not reproducible and must be re-derived -# -# Usage: 315-attrib.sh -# Holds the box lock for the whole session and passes NOLOCK=1 to the driver, which then inherits this -# session's fd 9 instead of racing for its own (the driver's fallback is to proceed UNLOCKED, which -# would put two arm runners on the box at once). -set -uo pipefail -PAIRS=${1:-6}; TAG=${2:-315att}; CE=${3:-1}; shift 3 || true -EV="$*" -CBIN=${CBIN:-$HOME/wt/315instr3-build/bin} # stock: implements none of the instruments -FBIN=${FBIN:-$HOME/wt/315fp-build/bin} # carries SENTINEL / COPY_PROBE / TENSOR_FP -D=/home/bcloud/wt - -# NOLOCK=1 means a caller already holds the lock on an inherited fd 9 (see 315-chain.sh). Taking it again -# here would open a second file description in the same process and deadlock against our own lock. -if [ -z "${NOLOCK:-}" ]; then - exec 9>"$HOME/.cache/lax-decode/box.lock" - if flock -w "${LOCK_WAIT:-3600}" 9; then - echo "attrib: box lock held $(date -Is)" - else - echo "attrib: box lock unobtainable after ${LOCK_WAIT:-3600}s - aborting rather than running concurrently" - exit 3 - fi -else - echo "attrib: NOLOCK=1 - inheriting the caller's fd 9 lock" -fi - -echo "=== attrib session tag=$TAG pairs=$PAIRS control_every=$CE $(date -Is) ===" -echo " extra_env='${EV}'" -echo " control_bin=$CBIN fix_bin=$FBIN" - -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ -CONTROL_EVERY="$CE" NOSTOP=1 MINCLEAN=999 NOLOCK=1 \ -EXTRA_ENV="$EV" ARM_TIMEOUT="${ARM_TIMEOUT:-1200}" \ - bash "$D/315-paired.sh" "$PAIRS" "${NB:-2}" "${NCTX:-2048}" "$CBIN" "$FBIN" "$TAG" - -echo "=== attrib session done $(date -Is) ===" -echo "stock-cell (control) arms:"; grep -c 'arm c: OK' "$D/$TAG-console.log" 2>/dev/null || true -echo "stock-cell (control) faults:"; grep -c 'arm c: \(FAULT\|INVALID\)' "$D/$TAG-console.log" 2>/dev/null || true -echo "fix-cell arms OK:"; grep -c 'arm l: OK' "$D/$TAG-console.log" 2>/dev/null || true -echo "fix-cell faults:"; grep -c 'arm l: \(FAULT\|INVALID\)' "$D/$TAG-console.log" 2>/dev/null || true diff --git a/dev/315/315-campaign/315-campaign.sh b/dev/315/315-campaign/315-campaign.sh deleted file mode 100644 index c848d9661..000000000 --- a/dev/315/315-campaign/315-campaign.sh +++ /dev/null @@ -1,153 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: in-regime acceptance campaign runner + verdict summary. -# -# Pre-registered settings (dev/315/315-preregistration.md sections 3, 6, 7, 18b): -# * regime: REGIME_LO/HI tuned so the arm lands in [95, 110) GiB with the resident inventory; the arm's -# own gtt_start is what decides membership (gate >= 95 GiB). -# * EXTRA_ENV is EMPTY: the campaign measures the shipped configuration, so the sentinel fill, the copy -# probe and the fingerprints are OFF (section 18b). Instrumented runs are diagnostics, not acceptance. -# * stop at the first fix-cell fault (NOSTOP unset), target MINCLEAN clean fix-cell arms, cap PAIRS. -# -# Usage: FIX= [CTRL=] [TAG=] [PAIRS=80] [MINCLEAN=60] 315-campaign.sh -set -uo pipefail - -FIX=${FIX:?fix bin dir} -CTRL=${CTRL:-$HOME/wt/315instr3-build} -TAG=${TAG:-315fix} -PAIRS=${PAIRS:-80} -MINCLEAN=${MINCLEAN:-60} - -test -x "$FIX/bin/llama-server" || { echo "ABORT: no $FIX/bin/llama-server"; exit 2; } -test -x "$CTRL/bin/llama-server" || { echo "ABORT: no $CTRL/bin/llama-server"; exit 2; } -echo "fix binary: $("$FIX/bin/llama-server" --version 2>&1 | head -1)" -echo "ctrl binary: $("$CTRL/bin/llama-server" --version 2>&1 | head -1)" -echo "campaign: tag=$TAG pairs=$PAIRS target_clean=$MINCLEAN regime_LO=${REGIME_LO:-80} regime_HI=${REGIME_HI:-92} EXTRA_ENV=" - -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} MINCLEAN="$MINCLEAN" \ - bash "$HOME/wt/315-paired.sh" "$PAIRS" 2 2048 "$CTRL/bin" "$FIX/bin" "$TAG" 2>&1 | tail -12 -# The "2" above is the driver's argument and it is INERT: the driver reads it only to print it -# in the session header, and composition is done by 315-regime.sh, which starts neighbours one at a time -# against live GTT until it reaches REGIME_LO and never past REGIME_HI (1..6). So the pre-registration's -# "3 detached neighbour processes" is the typical outcome at LO=80, not a parameter, and the literal here is -# not a criterion. Recorded because a header line reading "neighbours=2" next to a document saying three -# would otherwise look like a criteria change after the fact. - -CONSOLE="$HOME/wt/$TAG-console.log" -echo "=== verdict summary (from the driver's own per-arm lines, which implement section 5)" -python3 - "$CONSOLE" <<'PY' -import re, sys -from math import comb -path = sys.argv[1] -arm = re.compile(r"^\s*pair\s+(\d+)\s+arm\s+([cl]):\s+(\S+)\s+nan=(\d+)\s+req=(\d+)\s+ret3=\d+\s+parse=(\d+)\s+gtt_start=(\d+)G(.*)$") -rows = [] -for line in open(path, errors="replace"): - m = arm.match(line) - if not m: - continue - tail = m.group(8) - utf8 = int((re.search(r"utf8=(\d+)", tail) or [0, 0])[1]) - alloc = int((re.search(r"alloc=(\d+)", tail) or [0, 0])[1]) - # engine#315: the disc rig's arm line carries the fields Face B actually sets; the paired rig's line does - # not. Parsed here when present so a disc-rig console summarises correctly, and recorded as absent (None) - # otherwise so an unattributed fault is never silently read as "not Face B". - def tail_int(name): - m2 = re.search(name + r"=(\d+)", tail) - return int(m2.group(1)) if m2 else None - http = tail_int("http") - s500 = tail_int("s500") - unparsed = tail_int("unparsed") - rows.append(dict(pair=int(m.group(1)), cell=m.group(2), verdict=m.group(3), nan=int(m.group(4)), - req=int(m.group(5)), parse=int(m.group(6)), gtt=int(m.group(7)), utf8=utf8, alloc=alloc, - http=http, s500=s500, unparsed=unparsed)) - -# engine#315: Face B is Content-only / non-JSON output answered with HTTP 500 (the recorded form), or a body -# that fails the full-body UTF-8 check (the objective's form). Both are tested, per the user's ruling that -# either manifestation counts. A field must be PRESENT to contribute, so a paired-rig line without these -# fields cannot manufacture a false Face B. -def is_face_b(r): - return (r["parse"] > 0 or r["utf8"] > 0 - or (r["http"] or 0) > 0 or (r["s500"] or 0) > 0 or (r["unparsed"] or 0) > 0) - -def summarise(cell): - sel = [r for r in rows if r["cell"] == cell] - inb = [r for r in sel if r["gtt"] >= 95] - oor = [r for r in sel if r["gtt"] < 95] - clean = [r for r in inb if r["verdict"] == "OK"] - faults = [r for r in inb if r["verdict"] == "FAULT"] - invalid = [r for r in inb if r["verdict"] == "INVALID"] - print(f"cell {cell}: arms={len(sel)} in-regime={len(inb)} out-of-regime={len(oor)} " - f"clean={len(clean)} fault={len(faults)} invalid={len(invalid)}") - for r in faults[:6]: - faces = [] - if r["nan"] > 0: - faces.append("FaceA(nan=%d)" % r["nan"]) - if is_face_b(r): - faces.append("FaceB(parse=%d,utf8=%d,http=%s,s500=%s,unparsed=%s)" - % (r["parse"], r["utf8"], r["http"], r["s500"], r["unparsed"])) - if not faces: - print(" NOTE: this line carries no http/s500/unparsed field, so Face B's recorded form cannot" - " be tested from it - the paired rig does not emit them. Take the face attribution from a" - " disc-rig console, or add the fields to the paired rig's verdict printer.") - faces.append("unattributed - FAULT with nan=0 and no parse/utf8 counter; read the server log" - " (an HSA_STATUS_ERROR_MEMORY_FAULT is Face A and is not on this line)") - print(f" FAULT pair={r['pair']} {' '.join(faces)} req={r['req']} gtt={r['gtt']}G") - face_a = sum(1 for r in faults if r["nan"] > 0) - face_b = sum(1 for r in faults if is_face_b(r)) - unattr = sum(1 for r in faults if r["nan"] == 0 and not is_face_b(r)) - if faults: - print(f" face tally: FaceA={face_a} FaceB={face_b} unattributed={unattr}" - f" (the criterion is zero on BOTH faces; HSA faults are unattributed here by construction)") - # The acceptance statistic runs over ALL in-regime arms, not over the clean ones. The first version of - # this summary divided by the clean count and hardcoded a zero numerator, so it printed "rate=0/N = 0.0%" - # and a rule-of-three bound no matter how many arms had faulted - a fault was visible only in the - # separate count. For an acceptance decision that is exactly backwards: a faulted arm is evidence about - # the rate and has to be in the denominator. - n = len(inb) - k = len(faults) - if n == 0: - print(" rate=undefined (no in-regime arms)") - elif k == 0: - print(f" rate=0/{n} = 0.0% one-sided 95% upper bound (rule of three, 3/n) = {3.0/n*100:.2f}%" - f" (0.95^{n} = {0.95**n*100:.2f}%)") - else: - # Exact Clopper-Pearson 95% interval by bisection on the binomial tail, so no scipy is needed. - # P(X >= k | p) increases in p and P(X <= k | p) decreases in p, so each bound is the crossing of - # its tail probability with alpha/2. - def cdf(p, kk): - return sum(comb(n, i) * p ** i * (1 - p) ** (n - i) for i in range(0, kk + 1)) - lo, hi = 0.0, 1.0 - for _ in range(200): - mid = 0.5 * (lo + hi) - if 1.0 - cdf(mid, k - 1) > 0.025: hi = mid - else: lo = mid - lower = 0.5 * (lo + hi) - lo, hi = 0.0, 1.0 - for _ in range(200): - mid = 0.5 * (lo + hi) - if cdf(mid, k) > 0.025: lo = mid - else: hi = mid - upper = 0.5 * (lo + hi) - print(f" rate={k}/{n} = {k/n*100:.2f}% exact 95% CI [{lower*100:.2f}%, {upper*100:.2f}%]") - print(f" ACCEPTANCE FAILS: {k} in-regime arm(s) faulted; the criterion is zero faults across both faces") - if invalid: - print(f" {len(invalid)} in-regime arm(s) were INVALID, i.e. neither clean nor faulted - they are not") - print(f" clean evidence and have to be resolved before the criterion can be read as met") - return clean, faults, invalid - -for cell in ("l", "c"): - summarise(cell) -PY diff --git a/dev/315/315-campaign/315-cell-build.sh b/dev/315/315-campaign/315-cell-build.sh deleted file mode 100644 index 570f7a0d2..000000000 --- a/dev/315/315-campaign/315-cell-build.sh +++ /dev/null @@ -1,63 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: build a measurement cell from a given source worktree. Asserts the intended source change -# is present BEFORE configuring and the diagnostic markers are in the binary AFTER, so a silent build of -# the wrong tree cannot be mistaken for a measurement. -# -# SRC= B= [LOG=] [EXPECT=] 315-cell-build.sh -set -euo pipefail -SRC=${SRC:?source worktree}; B=${B:?build dir}; LOG=${LOG:-$B.log} -PIN=${PIN:-e44c9d01a4d5110cecaf45472a717f46cd1abbf9} -HRX=${HRX:-$HOME/wt/pin-fork-hrx/third_party/hrx-system} -EXPECT=${EXPECT:-} -say() { echo "[$(date +%H:%M:%S)] $*"; } - -say "preflight" -[ -e "$SRC/.git" ] || { echo "ABORT: $SRC is not a worktree"; exit 1; } -[ "$(git -C "$SRC" rev-parse HEAD)" = "$PIN" ] || { echo "ABORT: $SRC not at the pin"; exit 1; } -[ -d "$HRX" ] || { echo "ABORT: hrx-system missing"; exit 1; } -# Search the whole backend tree: a change may live in any translation unit (this assertion originally -# looked only at ggml-hrx.cpp and rejected a probe that lives in runtime/graph-executor.cpp). -if [ -n "$EXPECT" ]; then - grep -rq "$EXPECT" "$SRC/ggml/src/ggml-hrx" || { echo "ABORT: expected source change '$EXPECT' is absent - wrong tree"; exit 1; } - say "source assertion OK: '$EXPECT' present in $(grep -rl "$EXPECT" "$SRC/ggml/src/ggml-hrx" | head -1)" -fi -git -C "$SRC" diff > "$HOME/wt/$(basename "$B").patch" 2>/dev/null || true -say "worktree diff: $(git -C "$SRC" diff --shortstat | tr -d '\n')" - -say "configure" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "ABORT: cmake failed, see $LOG"; tail -15 "$LOG"; exit 1; } - -say "build (-j12)" -cmake --build "$B" -j 12 >> "$LOG" 2>&1 || { echo "ABORT: build failed, see $LOG"; tail -25 "$LOG"; exit 1; } - -say "binary identity"; "$B/bin/llama-server" --version -LIB=$(ls "$B"/bin/libggml-hrx.so* 2>/dev/null | head -1) -[ -n "$LIB" ] || { echo "ABORT: libggml-hrx.so not found"; exit 1; } -# MARKERS lets a cell assert its own marker: a probe that never compiled in would otherwise pass every -# assertion here, which is exactly what happened once. -for M in hrx-wb-fp hrx-dl-fp hrx-buf ${MARKERS:-}; do - N=$(grep -c "$M" "$LIB" || true); say " marker $M=$N"; [ "$N" -gt 0 ] || { echo "ABORT: $M absent"; exit 1; } -done -say "CELLBUILD_DONE bin_dir=$B/bin" diff --git a/dev/315/315-campaign/315-chain.sh b/dev/315/315-campaign/315-chain.sh deleted file mode 100755 index 448094d6a..000000000 --- a/dev/315/315-campaign/315-chain.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 chain probe. Section 40's one-bit causality test, run as an env change rather than a rebuild: -# GGML_HRX_TENSOR_FP accepts a comma-joined name substring list, so naming the block's residual, attention, -# norm and FFN tensors makes the fingerprint scan the *whole* graph for exactly those, and the first one -# carrying NaN says whether the NaN was already in the block's input (`l_out-40`, `ffn_inp-41`, -# `ffn_norm-41`) or was fabricated inside the FFN branch (`ffn_shexp-41`, `ffn_swiglu-41`, -# `ffn_moe_out-41`). Names are from the pin's own builder, src/models/deepseek.cpp:130-176 plus -# llama-graph.cpp's build_moe_ffn; this model's output norm is the unnumbered `norm`. -# -# Both instruments in this env are cleared of fabricating NaN (section 41): the copy probe only fprintfs, and -# the sentinel writes only into the 619520-byte logits allocation. What they can do is perturb timing - which -# is wanted here, because the arms have to fault for the probe to have anything to read. -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-chain.sh [pairs] [tag] (waits for the box lock, then runs; NOLOCK=1 goes to the driver) -set -uo pipefail -PAIRS=${1:-6}; TAG=${2:-315chain} - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${CHAIN_LOCK_WAIT:-5400}" 9; then - echo "chain: box lock held $(date -Is) after waiting" -else - echo "chain: box lock unobtainable after ${CHAIN_LOCK_WAIT:-5400}s - aborting rather than running concurrently" - exit 3 -fi - -NAMES="l_out-39,l_out-40,l_out-41,attn_norm-41,Qcur-41,Kcur-41,Vcur-41,ffn_inp-41,ffn_norm-41,ffn_shexp-41,ffn_swiglu-41,ffn_moe_out-41,ffn_out-41,attn_norm-42,ffn_inp-42,ffn_norm-42,ffn_shexp-42,ffn_moe_out-42,l_out-42,norm,result_norm,result_output" -echo "chain: names=$NAMES" - -NOLOCK=1 exec bash "$HOME/wt/315-attrib.sh" "$PAIRS" "$TAG" 999 \ - "GGML_HRX_COPY_PROBE=1 GGML_HRX_SENTINEL=1 GGML_HRX_TENSOR_FP=$NAMES GGML_HRX_TENSOR_FP_NANONLY=1" diff --git a/dev/315/315-campaign/315-chain2.sh b/dev/315/315-campaign/315-chain2.sh deleted file mode 100755 index 6aa0dc6e7..000000000 --- a/dev/315/315-campaign/315-chain2.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 chain probe, second pass. Prepared, not launched: 315-chain.sh answers whether the NaN was -# already in block 41's input, and this one is for the case where it was not - i.e. the NaN is born inside -# the block's FFN/MoE, between ffn_norm and ffn_moe_out. -# -# Why a second pass at all: with GGML_HRX_TENSOR_FP_NANONLY=1 the fingerprint prints only nodes that carry -# NaN, in graph order, so the FIRST printed line is the earliest NaN-bearing named node. That makes the name -# list a bisection tool rather than a single query - but only for nodes that have names. The earlier window -# showed auto-named nodes (`node_2948`, `node_2985`) that no name list can match, and the MoE's own -# intermediates were not named in the first pass. -# -# This list widens both directions: `l_out-24..l_out-42` walks the residual stream back towards the birth of -# the corruption if it turns out to precede block 41, and the block-41/42 chain adds the MoE intermediates -# (build_moe_ffn in the pin names them ffn_moe_gate/up/gate_up/swiglu/weighted/down/probs/weights/topk). -# Roughly 60 device-to-host reads per graph instead of 16: more timing perturbation, which is wanted here -# because the arms have to fault - and section 41 established that these reads cannot manufacture a NaN. -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-chain2.sh [pairs] [tag] -set -uo pipefail -PAIRS=${1:-6}; TAG=${2:-315chain2} - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${CHAIN2_LOCK_WAIT:-5400}" 9; then - echo "chain2: box lock held $(date -Is)" -else - echo "chain2: lock unobtainable after ${CHAIN2_LOCK_WAIT:-5400}s - aborting rather than running concurrently" - exit 3 -fi - -# residual stream, walking backwards from the block that first showed NaN -BACK="" -for il in 42 41 40 39 38 36 34 32 30 28 26 24; do BACK="$BACK,l_out-$il"; done -# block 41 and 42: attention input, projections, and the FFN/MoE chain in dataflow order -CHAIN="" -for il in 41 42; do - CHAIN="$CHAIN,attn_norm-$il,Qcur-$il,Kcur-$il,Vcur-$il,ffn_inp-$il,ffn_norm-$il" - # ffn_moe_topk is deliberately absent: it is an i32 index tensor, so reading it as floats makes - # its NaN count meaningless (i32 bit patterns that look like NaN), and with NANONLY=1 a node - # that does not print tells us nothing. The f32 routing tensors (probs, weights) are kept. - CHAIN="$CHAIN,ffn_moe_probs-$il,ffn_moe_weights-$il,ffn_moe_gate-$il,ffn_moe_up-$il" - CHAIN="$CHAIN,ffn_moe_gate_up-$il,ffn_swiglu-$il,ffn_moe_weighted-$il,ffn_moe_down-$il,ffn_moe_out-$il" - CHAIN="$CHAIN,ffn_shexp-$il,ffn_out-$il" -done -NAMES="norm,result_norm,result_output${BACK}${CHAIN}" -echo "chain2: $(printf '%s' "$NAMES" | tr ',' '\n' | grep -c .) names" - -NOLOCK=1 exec bash "$HOME/wt/315-attrib.sh" "$PAIRS" "$TAG" 999 \ - "GGML_HRX_COPY_PROBE=1 GGML_HRX_SENTINEL=1 GGML_HRX_TENSOR_FP=$NAMES GGML_HRX_TENSOR_FP_NANONLY=1" diff --git a/dev/315/315-campaign/315-cheap-probe.sh b/dev/315/315-campaign/315-cheap-probe.sh deleted file mode 100755 index c3077d849..000000000 --- a/dev/315/315-campaign/315-cheap-probe.sh +++ /dev/null @@ -1,92 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 cheap-probe.sh -# -# A small-model detector for the preemption-sensitive fault class, modelled on the study's -# hrx123-probe.sh: one llama-server with Qwen3-0.6B on HRX0, N identical greedy requests in the same -# process. Healthy = every response identical. Faulty = responses diverge (the stale-bytes read -# depends on request history) and/or the server logs "HRX returned NaN logits". -# -# Runs the instrumented build (315fp-build, the pin's kernel plus the nan-dump fingerprint): if a -# fault happens, the log then says which tail node first went NaN. -# -# compact=1 forces KFD queue evictions the way the study did (sudo -n tee /proc/sys/vm/compact_memory), -# which is what opens the preemption window for this class. -set -u -TAG=${1:?tag}; N=${2:-120}; COMPACT=${3:-0}; CTX=${CTX:-4096} -B=$HOME/wt/315fp-build/bin -D=$HOME/wt/315-cheap-probe; mkdir -p "$D" -SRV=$D/$TAG.server.log; OUT=$D/$TAG.jsonl -: > "$SRV"; : > "$OUT" -PORT=$((21000 + RANDOM % 2000)) -HSA=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 - -echo "=== cheap probe tag=$TAG n=$N compact=$COMPACT $(date -Is) port=$PORT gtt=$(awk '{printf "%.1f", $1/1073741824}' /sys/class/drm/card*/device/mem_info_gtt_used | head -1)GiB" - -env LD_LIBRARY_PATH="$B" IREE_HAL_AMDGPU_LIBHSA_PATH="$HSA" \ - GGML_HRX_COPY_PROBE=1 GGML_HRX_LOGITS_FP=100000 \ - GGML_HRX_TENSOR_FP="${FP-tail:16}" ${FPONLY:+GGML_HRX_TENSOR_FP_NANONLY=1} \ - "$B/llama-server" -m "$HOME/models/Qwen3-0.6B-Q4_K_M.gguf" -dev HRX0 -ngl 99 -fa on \ - --host 127.0.0.1 --port "$PORT" -c "$CTX" -np 1 -t 8 >> "$SRV" 2>&1 & -SP=$! -CP= -cleanup() { [ -n "$SP" ] && kill "$SP" 2>/dev/null; [ -n "$CP" ] && kill "$CP" 2>/dev/null; } -trap cleanup EXIT - -up=0 -for _ in $(seq 1 240); do curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 && { up=1; break; }; sleep 1; done -[ "$up" = 1 ] || { echo "ABORT: server never became healthy"; tail -5 "$SRV"; exit 2; } - -if [ "$COMPACT" = 1 ]; then - ( while true; do echo 1 | sudo -n tee /proc/sys/vm/compact_memory >/dev/null 2>&1; sleep 0.2; done ) & - CP=$! -fi - -# SHAPE: the arms' prompts ran up to ~868 tokens of KV; the default probe prompt is 6 tokens, so a -# longer prompt is a different point in the shape space. LONG= pads the prompt with a repeated -# sentence (the same padding for every request, so greedy output stays comparable). -if [ "${LONG:-0}" != "0" ]; then - PROMPT="$(python3 -c "print(' '.join(['The quick brown fox jumps over a lazy dog while engineers measure kernel launch gaps.'] * $(( ${LONG} / 12 + 1 ))))" | cut -c1-$(( ${LONG} * 5 ))) The capital of France is" -else - PROMPT="The capital of France is" -fi -for i in $(seq 1 "$N"); do - r=$(curl -s --max-time 120 "http://127.0.0.1:$PORT/completion" -H "Content-Type: application/json" \ - -d "{\"prompt\":\"$PROMPT\",\"n_predict\":24,\"temperature\":0,\"seed\":1,\"cache_prompt\":false,\"stream\":false}") - echo "$r" >> "$OUT" -done -echo "@@END" >> "$SRV" - -python3 - "$OUT" "$SRV" <<'PY' -import hashlib, json, sys -from collections import Counter -bodies = [] -for line in open(sys.argv[1], errors="replace"): - line = line.strip() - if not line: - continue - try: - d = json.loads(line) - except Exception: - bodies.append("UNPARSEABLE:" + line[:40]) - continue - bodies.append(d.get("content", "NO-CONTENT")) -c = Counter(bodies) -nan = sum(1 for l in open(sys.argv[2], errors="replace") if "NaN logits" in l) -print(f"requests={len(bodies)} distinct_outputs={len(c)} nan_lines={nan}") -for body, n in c.most_common(5): - print(f" {n:5d} x {body[:70]!r}") -PY diff --git a/dev/315/315-campaign/315-classify.py b/dev/315/315-campaign/315-classify.py deleted file mode 100644 index 3eb23802f..000000000 --- a/dev/315/315-campaign/315-classify.py +++ /dev/null @@ -1,49 +0,0 @@ -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -import json, re, os, glob -def arm(t): - O = glob.glob(os.path.expanduser(f"~/wt/rt-det/{t}-*")) - if not O: return None - O = O[0]; d = {} - try: d = json.load(open(f"{O}/detector.json")) - except Exception: pass - log = "" - try: log = open(f"{O}/server.log", errors="replace").read() - except Exception: pass - g = lambda k: int(d.get(k) or 0) - nan = len(re.findall(r"HRX returned NaN logits", log)) - hsa = len(re.findall(r"HSA_STATUS_ERROR|Queue error|wait for HRX graph replay commands failed", log)) - # Face B as pre-registered: invalid-UTF-8 output surfacing as HTTP 500 / parse failures - five = len(re.findall(r'"code":500', log)) - unr = len(re.findall(r"unparsed Content-only output|does not match the expected", log)) - noncoh = len(re.findall(r"\[hrx-buf\][^\n]*host_visible=1[^\n]*direct_host_binding=0", log)) - f = dict(nan=nan, hsa=hsa, utf8=g("utf8_failures"), parse=g("parse_failures"), - http=g("http_failures"), tr=g("transport_failures"), req=g("requests"), - ok=g("ok"), s500=five, unparsed=unr, hostvis_noncoh=noncoh) - v = "clean" - if any(f[k] for k in ("nan","hsa","utf8","parse","http","tr","s500","unparsed")): v = "FAULT" - elif f["req"] < 66: v = "INCOMPLETE" - return v, f -print(f"{'arm':<14}{'verdict':<11}{'nan':>4}{'hsa':>4}{'utf8':>5}{'parse':>6}{'http':>5}{'tr':>4}{'500':>4}{'unparsed':>9}{'noncoh':>7}{'req':>5}") -res = {} -for t in [f"315diff2-{c}{i}" for i in range(1,7) for c in "du"]: - r = arm(t) - if not r: continue - v, f = r; res[t] = v - print(f"{t:<14}{v:<11}{f['nan']:>4}{f['hsa']:>4}{f['utf8']:>5}{f['parse']:>6}{f['http']:>5}{f['tr']:>4}{f['s500']:>4}{f['unparsed']:>9}{f['hostvis_noncoh']:>7}{f['req']:>5}") -print("\npairs:") -for i in range(1,7): - d, u = res.get(f"315diff2-d{i}"), res.get(f"315diff2-u{i}") - if d or u: print(f" PAIR {i}: default={d} unified={u}") diff --git a/dev/315/315-campaign/315-cwsr-control.sh b/dev/315/315-campaign/315-cwsr-control.sh deleted file mode 100755 index 72d8f0588..000000000 --- a/dev/315/315-campaign/315-cwsr-control.sh +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 H1 CONTROL: fresh boot with amdgpu.cwsr_enable=1, identical campaign. -# -# WHY THIS EXISTS. The cwsr_enable=0 result (0 faults in 11 in-regime arms vs a 26% -# multi-session baseline) is confounded: the reboot that set cwsr_enable=0 ALSO changed box -# state (fresh boot, unfragmented device memory, no prior campaigns). Without a fresh-boot -# cwsr_enable=1 control, a clean result cannot be attributed to CWSR at all. This issue has -# already been burned once by exactly that error ("the rate followed memory-pressure -# settings"), so the control is mandatory before any causal claim. -# -# WHAT IT DOES: removes the cwsr drop-in, reboots, and (after the box returns) prints the -# exact command to run the SAME campaign. It does NOT launch the campaign itself - a reboot -# ends this process. -# -# HONEST LIMITATION: "identical campaign" can mean same PAIRS and same composition, not a -# literally identical session. Time of day, ambient temperature and whatever else the box -# has been doing differ. Record the conditions alongside the result; a cwsr_enable=1 rate -# near 26% kills H1, and near 0% confirms it. -set -uo pipefail - -DROPIN=/etc/default/grub.d/amd_cwsr.cfg -TAG=${TAG:-315ctl1} -# 20 pairs, not 48. This control answers two questions and both are satisfied by 20: -# H1 (boot confound): at the ~26% baseline, 20 arms give ~5.2 expected faults and -# P(0 faults) = 0.74^20 = 0.002 - so a clean control is real evidence, and a rate near 26% -# kills H1. 60 pairs would cost ~3 extra hours for no extra power on this question. -# H2 (eviction comparison): needs only a few faulting arms to compare against clean ones. -# The box-time budget is the binding constraint, so the control is sized to its question. -PAIRS=${PAIRS:-20} - -# Install the PID-capturing driver BEFORE the run, so this control's per-arm data supports the -# H2 test (faulting-vs-clean eviction counts). Without it, a finished arm cannot be bound to its -# PID and the comparison is invalid — proved by attempting it on run 7's data and finding the -# counts contaminated by post-arm lifetime (see 315-pid-capture.md). -if [ -e "$HOME/wt/315-paired-pidcap.sh" ]; then - if [ ! -e "$HOME/wt/315-paired.sh.orig" ]; then - cp "$HOME/wt/315-paired.sh" "$HOME/wt/315-paired.sh.orig" - fi - cp "$HOME/wt/315-paired-pidcap.sh" "$HOME/wt/315-paired.sh" - echo "installed the PID-capturing driver (original kept at 315-paired.sh.orig)" - bash -n "$HOME/wt/315-paired.sh" || { echo "ABORT: patched driver has a syntax error"; exit 3; } - # Verify the capture is actually present rather than trusting the copy: a run without it - # silently produces data that cannot support the H2 comparison. - grep -q "server.pid" "$HOME/wt/315-paired.sh" \ - || { echo "ABORT: installed driver does not write server.pid - H2 data would be lost"; exit 3; } - echo " verified: driver writes server.pid (H2 data will be capturable)" -else - echo "NOTE: 315-paired-pidcap.sh not present; running WITHOUT per-arm PID capture." - echo " The run is still valid, but its data cannot support the H2 comparison." -fi - -echo "=== H1 control: fresh boot with cwsr_enable=1 ===" -echo " current cwsr_enable : $(cat /sys/module/amdgpu/parameters/cwsr_enable)" -echo " drop-in present : $([ -e "$DROPIN" ] && echo yes || echo no)" -echo " campaign to mirror : 315-evict-trace.sh $PAIRS (the cwsr=0 campaign of record used 70)" -echo - -if [ -n "$(pgrep -f '315-evict-trace' || true)" ]; then - echo "ABORT: a 315 campaign is still running. Wait for it - this would kill its evidence." - exit 3 -fi -if [ -n "$(ps -eo args | grep -- '-dev HRX0' | grep -v grep || true)" ]; then - echo "ABORT: an HRX job is running (not ours). Do not disturb it." - exit 3 -fi - -echo "removing $DROPIN so cwsr_enable defaults to 1" -sudo -n rm -f "$DROPIN" || { echo "ABORT: could not remove the drop-in"; exit 3; } -sudo -n update-grub 2>&1 | tail -3 - -echo -echo "verify the parameter is GONE from the generated entry:" -if sudo -n grep -q "cwsr_enable" /boot/grub/grub.cfg; then - echo " WARNING: cwsr_enable still appears in grub.cfg:" - sudo -n grep -m1 "cwsr_enable" /boot/grub/grub.cfg | cut -c1-140 - echo " not rebooting." - exit 3 -fi -echo " clean: no cwsr_enable in grub.cfg -> the next boot uses the driver default (1)" - -echo -echo "=== after the box returns, run EXACTLY this ===" -echo " ssh strixhalo 'cat /sys/module/amdgpu/parameters/cwsr_enable' # expect 1" -echo " ssh strixhalo 'nohup bash ~/wt/315-evict-trace.sh $PAIRS $TAG > ~/wt/$TAG-run.log 2>&1 &'" -echo -echo "then compare: baseline cwsr=1 must show ~26% (34/131 historically)." -echo " ~26% -> H1 DEAD, the cwsr=0 signal is real" -echo " ~0% -> H1 CONFIRMED, the signal was a fresh-boot effect" -echo -echo "rebooting now (this process ends here)" -sudo -n systemctl reboot diff --git a/dev/315/315-campaign/315-detector.py b/dev/315/315-campaign/315-detector.py deleted file mode 100755 index bc4682096..000000000 --- a/dev/315/315-campaign/315-detector.py +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env python3 -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""engine#315 detector: the pre-registered 66-request shape set, with Face B measured. - -Why this exists: `~/wt/rt-det.sh` runs the same requests but decodes the response with -`json.loads(r.read())` inside a bare `except`, so a body that is not valid UTF-8 is -indistinguishable from any other request failure. Face B (invalid-UTF-8 output surfacing as -HTTP 500 / parse failures) is a first-class acceptance criterion in the pre-registration, so -the raw bytes have to be inspected. - -Shape set (identical to rt-det.sh, so arms stay comparable with the recorded study): - 1. 8 identical greedy requests, n_predict=96, temperature=0, cache_prompt=false, each followed - by a distractor request of a different length; - 2. 50 mixed-length requests: distractor 3-900 words, n_predict 4-64. -Total 8 + 8 + 50 = 66 requests. - -Usage: 315-detector.py -Writes /detector.json and prints one summary line. -""" -import json -import os -import random -import sys -import urllib.error -import urllib.request - -PROMPT = ("Explain in detail how a transformer language model generates text, step by step, " - "including attention, the KV cache and sampling.") -WORDS = "the quick brown fox jumps over a lazy dog while seven tired engineers measure kernel launch gaps on a hot laptop".split() - - -def request(port: int, prompt: str, n_predict: int) -> dict: - """One request, recording the raw bytes rather than trusting json.loads. - - The timeout is deliberately bounded: an arm whose server stops responding must be recorded as a - transport failure and end quickly. The inherited 600 s per request let a stalled server hold the - shared box for hours (observed on the second smoke arm). - """ - timeout = float(os.environ.get("DET_TIMEOUT", "180")) - body = json.dumps({"prompt": prompt, "n_predict": n_predict, "temperature": 0, - "cache_prompt": False}).encode() - rec = {"http_status": None, "utf8_ok": False, "json_ok": False, "content": None, - "excerpt_hex": None, "error": None} - try: - with urllib.request.urlopen( - urllib.request.Request(f"http://127.0.0.1:{port}/completion", body, - {"Content-Type": "application/json"}), timeout=timeout) as resp: - rec["http_status"] = resp.status - raw = resp.read() - except urllib.error.HTTPError as exc: - rec["http_status"] = exc.code - raw = exc.read() if hasattr(exc, "read") else b"" - except Exception as exc: # connection died mid-arm - rec["error"] = f"{type(exc).__name__}: {exc}" - return rec - - if rec["excerpt_hex"] is None: - rec["excerpt_hex"] = raw[:32].hex() - try: - text = raw.decode("utf-8") - rec["utf8_ok"] = True - except UnicodeDecodeError as exc: - # Face B: the body itself is not valid UTF-8. Record where it starts. - rec["error"] = f"UnicodeDecodeError at byte {exc.start}" - return rec - try: - rec["json_ok"] = True - rec["content"] = json.loads(text).get("content", "") - except Exception as exc: - rec["error"] = f"JSONDecodeError: {exc}" - return rec - - -def main() -> int: - port, outdir, nmix = int(sys.argv[1]), sys.argv[2], int(sys.argv[3]) - rng = random.Random(7) # same seed as rt-det.sh: comparable arms - summary = {"requests": 0, "ok": 0, "utf8_failures": 0, "parse_failures": 0, - "http_failures": 0, "transport_failures": 0, "identical_distinct": None, - "first_failure": None, "truncated": False} - - def note(rec: dict) -> bool: - summary["requests"] += 1 - if rec["http_status"] is None: # connection died / timeout: not Face B - summary["transport_failures"] += 1 - summary["first_failure"] = summary["first_failure"] or (rec["error"] or "transport") - return False - if not rec["utf8_ok"]: # Face B: body is not valid UTF-8 - summary["utf8_failures"] += 1 - summary["first_failure"] = summary["first_failure"] or (rec["error"] or "non-utf8 body") - return False - if not rec["json_ok"]: # body decoded but is not the API JSON - summary["parse_failures"] += 1 - summary["first_failure"] = summary["first_failure"] or (rec["error"] or "json parse") - return False - if rec["http_status"] != 200: - summary["http_failures"] += 1 - summary["first_failure"] = summary["first_failure"] or f"HTTP {rec['http_status']}" - return False - summary["ok"] += 1 - return True - - outs = [] - for i in range(8): - rec = request(port, PROMPT, 96) - if not note(rec): - summary["truncated"] = True - break - outs.append(rec["content"]) - if not note(request(port, "Summarize: " + " ".join(rng.choice(WORDS) for _ in range(20 + 37 * i)), - 8 + 5 * i)): - summary["truncated"] = True - break - summary["identical_distinct"] = len(set(outs)) if outs else None - - if not summary["truncated"]: - for _ in range(nmix): - prompt = "Continue the story: " + " ".join(rng.choice(WORDS) for _ in range(rng.randint(3, 900))) - if not note(request(port, prompt, rng.randint(4, 64))): - summary["truncated"] = True - break - - with open(f"{outdir}/detector.json", "w") as fh: - json.dump(summary, fh, indent=2) - fh.write("\n") - print(f"detector: req={summary['requests']}/66 ok={summary['ok']} " - f"utf8_fail={summary['utf8_failures']} parse_fail={summary['parse_failures']} " - f"http_fail={summary['http_failures']} transport_fail={summary['transport_failures']} " - f"identical_distinct={summary['identical_distinct']} truncated={summary['truncated']} " - f"first_failure={summary['first_failure']!r}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/dev/315/315-campaign/315-differential.sh b/dev/315/315-campaign/315-differential.sh deleted file mode 100644 index 96278f61c..000000000 --- a/dev/315/315-campaign/315-differential.sh +++ /dev/null @@ -1,101 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: wait for a free box, then run the alternating differential that decides the mechanism. -# -# Two questions, one campaign: -# * does an in-regime arm fault at all (task-2 wants the instrumentation record of a faulting arm)? -# * does coherence remove the fault (the causal half of task-3: the structural half is already -# confirmed — GGML_HRX_USE_UNIFIED_MEMORY flips HOST_COHERENT on exactly the logits buffer). -# -# So arms alternate: A = default configuration, B = same binary/regime with -# GGML_HRX_USE_UNIFIED_MEMORY=1. Paired arms in one session share the box state, so the comparison is -# not confounded by drift between sessions. -# -# The mechanism signature is "A faulted, B clean". A is expected to fault at roughly the recorded rate -# (~5%/arm), so a handful of pairs is evidence, not a rate; the acceptance campaign (>=60 clean arms) is -# a separate, longer run. -# -# Usage: PAIRS=6 MAXWAIT=18000 bash 315-differential.sh -set -uo pipefail -MAXWAIT=${MAXWAIT:-18000} -PAIRS=${PAIRS:-6} -TAGB=${TAGB:-315diff} -BIN=${BIN:-/home/bcloud/wt/315instr3-build/bin} -G=$(ls /sys/class/drm/card*/device/mem_info_gtt_used | head -1) -gtt() { echo $(( $(cat "$G") / 1073741824 )); } -FLOOR=${FLOOR:-30} -LOG=$HOME/wt/$TAGB.log - -arm() { # arm - local tag=$1 extra=${2:-} - echo "--- arm $tag extra_env='${extra:-none}' $(date -Is)" | tee -a "$LOG" - if [ -n "$extra" ]; then - EXTRA_ENV="$extra" NOLOCK=0 bash "$HOME/wt/315-instr-capture.sh" "$BIN" 1 2048 "$tag" 1 2>&1 | tail -14 | tee -a "$LOG" - else - NOLOCK=0 bash "$HOME/wt/315-instr-capture.sh" "$BIN" 1 2048 "$tag" 1 2>&1 | tail -14 | tee -a "$LOG" - fi - local O="$HOME/wt/rt-det/$tag-$(basename /home/bcloud/models/GLM-4.7-Flash-Q4_K_M.gguf .gguf)" - local nan hsa utf8 parse req g0 fp - nan=$(grep -c "HRX returned NaN logits" "$O/server.log" 2>/dev/null || true) - hsa=$(grep -iE "HSA_STATUS_ERROR|Queue error|wait for HRX graph replay commands failed" "$O/server.log" 2>/dev/null \ - | grep -vcE "hrx_allocator_allocate_buffer|allocate HRX host staging buffer" || true) - utf8=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['utf8_failures'])" 2>/dev/null || echo 0) - parse=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['parse_failures'])" 2>/dev/null || echo 0) - req=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['requests'])" 2>/dev/null || echo 0) - g0=$(grep -o "gtt_start [0-9]*G" "$O/run.log" 2>/dev/null | head -1) - fp=$(grep -c "\[hrx-buf\]" "$O/server.log" 2>/dev/null || true) - local http tr s500 unr - http=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['http_failures'])" 2>/dev/null || echo 0) - tr=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['transport_failures'])" 2>/dev/null || echo 0) - # Face B is first class in the pre-registration: invalid-UTF-8 output surfacing as an HTTP 500 or a - # chat-template parse failure. Take it from the detector counters AND the server log, so an arm - # cannot look clean merely by completing its requests. - s500=$(grep -c '"code":500' "$O/server.log" 2>/dev/null || true); s500=${s500:-0} - unr=$(grep -cE "unparsed Content-only output|does not match the expected" "$O/server.log" 2>/dev/null || true); unr=${unr:-0} - local verdict=clean - { [ "${nan:-0}" != "0" ] || [ "${hsa:-0}" != "0" ] || [ "${utf8:-0}" != "0" ] || [ "${parse:-0}" != "0" ] \ - || [ "${http:-0}" != "0" ] || [ "${tr:-0}" != "0" ] || [ "${s500:-0}" != "0" ] || [ "${unr:-0}" != "0" ]; } && verdict=FAULT - [ "${req:-0}" -lt 66 ] && [ "$verdict" = clean ] && verdict=INCOMPLETE - echo " RESULT tag=$tag $g0 verdict=$verdict nan=$nan hsa=$hsa utf8=$utf8 parse=$parse http=$http tr=$tr s500=$s500 unparsed=$unr req=$req buf_trace=$fp" | tee -a "$LOG" - echo "$verdict" > "$HOME/wt/$tag.verdict" -} - -echo "=== waiting for a free box (floor=${FLOOR} GiB, max ${MAXWAIT}s, at $(date -Is)) ===" | tee -a "$LOG" -waited=0; streak=0 -while [ "$waited" -lt "$MAXWAIT" ]; do - cur=$(gtt); peer=$(pgrep -cf "v315-next-fixed" 2>/dev/null); peer=${peer:-0} - avail=$(free -g | awk '/^Mem:/ {print $7}') - if [ "$cur" -lt "$FLOOR" ] && [ "${peer:-0}" = "0" ] && [ "${avail:-0}" -ge 25 ]; then streak=$((streak+1)); else streak=0; fi - echo "$(date +%H:%M:%S) gtt=${cur}G peer=${peer:-0} avail=${avail:-?}G streak=$streak" | tee -a "$LOG" - [ "$streak" -ge 2 ] && break - sleep 60; waited=$((waited+60)) -done -[ "$streak" -lt 2 ] && { echo "BOXWAIT_TIMEOUT after ${waited}s" | tee -a "$LOG"; exit 3; } - -echo "=== BOX FREE (gtt=$(gtt)G) — alternating differential, $PAIRS pairs ===" | tee -a "$LOG" -for i in $(seq 1 "$PAIRS"); do - arm "${TAGB}-d$i" "" - arm "${TAGB}-u$i" "GGML_HRX_USE_UNIFIED_MEMORY=1" - a=$(cat "$HOME/wt/${TAGB}-d$i.verdict" 2>/dev/null || echo "?") - b=$(cat "$HOME/wt/${TAGB}-u$i.verdict" 2>/dev/null || echo "?") - echo " PAIR $i: default=$a unified=$b" | tee -a "$LOG" - if [ "$a" = "FAULT" ] && [ "$b" = "clean" ]; then - echo "MECHANISM SIGNATURE at pair $i: default faulted while coherent arm was clean" | tee -a "$LOG" - break - fi -done -echo "315DIFF_DONE $(date -Is)" | tee -a "$LOG" -tail -20 "$LOG" diff --git a/dev/315/315-campaign/315-dose.sh b/dev/315/315-campaign/315-dose.sh deleted file mode 100755 index b3234277b..000000000 --- a/dev/315/315-campaign/315-dose.sh +++ /dev/null @@ -1,123 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 dose-response for the graph sync. Runs ONLY if 315gsync2's fix cell came back CLEAN. -# -# Why this exists (section 57's pre-commitment, written before gsync2's result): a binary cell can say "the sync -# removes the fault" but not "the fault scales with how often the consumer waits". If the binary cell is clean, -# the honest next question is whether the effect is monotone in the dose - because a clean cell with no dose -# response is more consistent with the sync perturbing the *inducer's* timing than with the sync repairing an -# ordering defect. -# -# `tools/patch-sync-dose.py` adds GGML_HRX_DIAGNOSTIC_GRAPH_SYNC_EVERY=N (synchronise every Nth replay; N=1 is -# the existing binary knob, a large N is effectively the no-knob cell). This script runs three doses, each in its -# own session so each takes and releases the box lock rather than holding it for the whole sweep. -# -# Readout: -# fault rate falls monotonically as N falls -> the measured quantity tracks the wait: the ordering is the -# carrier and the mechanism is named -# rate flat across N while gsync2's cell was clean -> the sync's *effect* is not dose-dependent, so the clean -# cell is not evidence that waiting repairs the fault -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-dose.sh [arms-per-dose] [doses...] -set -uo pipefail -ARMS=${1:-3}; shift || true -DOSES=("$@"); [ ${#DOSES[@]} -eq 0 ] && DOSES=(2 8 64) -SRC=$HOME/wt/llama-315copy -B=$HOME/wt/315dose-build -HRX=$HOME/wt/pin-fork-hrx/third_party/hrx-system -LOG=$HOME/wt/315dose-build.log - -# Pre-committed gate: this run is for the branch gsync2 did NOT close. gsync2 may still be queued behind another -# session when this script starts, so wait for its runner to exit before reading its console - evaluating the -# gate on a half-written file would abort the branch that may still be needed. -for _i in $(seq 1 300); do - pgrep -f "315-attrib.sh .* 315gsync2" >/dev/null 2>&1 || break - [ "$_i" -eq 1 ] && echo "dose: waiting for the gsync2 runner to finish before evaluating the gate" - sleep 60 -done -C=$HOME/wt/315gsync2-console.log -if [ -f "$C" ]; then - fixok=$(grep -c 'arm l: OK' "$C" || true) - fixf=$(grep -c 'arm l: \(FAULT\|INVALID\)' "$C" || true) - ctlok=$(grep -c 'arm c: OK' "$C" || true) - ctlf=$(grep -c 'arm c: \(FAULT\|INVALID\)' "$C" || true) - echo "dose: gsync2 - control OK=$ctlok fault=$ctlf | fix OK=$fixok fault=$fixf" - if [ "${ctlok:-0}" -eq 0 ] && [ "${ctlf:-0}" -eq 0 ]; then - echo "dose: ABORT - gsync2 produced no scored arms; nothing to follow up" - exit 0 - fi - if [ "${fixf:-0}" -ne 0 ] || [ "${fixok:-0}" -lt 3 ]; then - echo "dose: ABORT - gsync2's fix cell faulted or is too small; the ordering family is already closed there" - echo "dose: the gated ordered locator (315-fp2.sh) is the run for that branch" - exit 0 - fi - echo "dose: gsync2's fix cell was CLEAN with a faulting control; running the dose sweep" -else - echo "dose: ABORT - no gsync2 console; the gate cannot be checked" - exit 0 -fi - -grep -q "diagnostic_sync_dose" "$SRC/ggml/src/ggml-hrx/runtime/command-program-executor.cpp" \ - || { echo "dose: ABORT - the dose patch is not in the source (tools/patch-sync-dose.py)"; exit 1; } - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${DOSE_LOCK_WAIT:-10800}" 9; then - echo "dose: box lock held for the build $(date -Is)" -else - echo "dose: lock unobtainable - aborting rather than building concurrently" - exit 3 -fi - -echo "dose: configuring $B $(date -Is)" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "dose: cmake failed"; tail -20 "$LOG"; exit 1; } -echo "dose: building -j 6 $(date -Is)" -cmake --build "$B" -j 6 >> "$LOG" 2>&1 || { echo "dose: build failed"; tail -30 "$LOG"; exit 1; } -"$B/bin/llama-server" --version || exit 1 -grep -qa "diagnostic_sync_dose" "$B/bin/llama-server" || { echo "dose: ABORT - the dose branch is not in the binary"; exit 1; } -echo "dose: dose branch present; binary sha256 $(sha256sum "$B/bin/llama-server" | cut -c1-16)" -flock -u 9 - -# Each dose is its own session: it takes the lock, runs, and releases, so the box is never held for the sweep. -for N in "${DOSES[@]}"; do - tag="315dose-n$N" - echo "=== dose session N=$N arms=$ARMS $(date -Is) ===" - CBIN="$B/bin" FBIN="$B/bin" \ - REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ - ARM_TIMEOUT=${ARM_TIMEOUT:-1800} LOCK_WAIT=${DOSE_LOCK_WAIT:-10800} \ - bash "$HOME/wt/315-attrib.sh" "$ARMS" "$tag" 999 \ - "GGML_HRX_DIAGNOSTIC_GRAPH_SYNC_EVERY=$N GGML_HRX_TENSOR_FP=tail:16 GGML_HRX_TENSOR_FP_NANONLY=1" - c=$HOME/wt/$tag-console.log - echo " N=$N clean=$(grep -c 'arm l: OK' "$c" || true) fault=$(grep -c 'arm l: \(FAULT\|INVALID\)' "$c" || true) oor=$(grep -c 'arm l: OOR' "$c" || true)" - grep -hE "^ pair" "$c" | tail -3 -done - -echo "=== dose readout $(date -Is) ===" -for N in "${DOSES[@]}"; do - c=$HOME/wt/315dose-n$N-console.log - [ -f "$c" ] || continue - echo "N=$N clean=$(grep -c 'arm l: OK' "$c" || true) fault=$(grep -c 'arm l: \(FAULT\|INVALID\)' "$c" || true) oor=$(grep -c 'arm l: OOR' "$c" || true) gtt=$(grep -oE 'gtt_start=[0-9]+G' "$c" | sort -u | tr '\n' ' ')" -done -echo "Read the rate against N: monotone means the wait is the carrier; flat means the clean binary cell was not." diff --git a/dev/315/315-campaign/315-evict-trace.sh b/dev/315/315-campaign/315-evict-trace.sh deleted file mode 100755 index e82acc50f..000000000 --- a/dev/315/315-campaign/315-evict-trace.sh +++ /dev/null @@ -1,236 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 task-8: attach the KFD SMI eviction/preemption watcher to the detector arm, -# so a faulting arm carries the driver's own eviction events on the same timeline. -# -# Why a wrapper and not a driver edit: 315-paired.sh is part of the frozen pre-registration -# and must not change. This script runs the SAME driver, and starts the watcher alongside it, -# keyed on the detector server's port. A clean-arm trace is recorded as uninformative, not -# as evidence - that is the task-8 contract. -# -# Usage: 315-evict-trace.sh [FIX_BIN] [CTRL_BIN] -set -uo pipefail - -PAIRS=${1:-4} -TAG=${2:-315evict} -FIX=${3:-$HOME/wt/315fix-build} -CTRL=${4:-$HOME/wt/315instr3-build} - -# 315-campaign.sh takes the BUILD DIR and appends /bin itself. Accept either form so a -# caller passing .../bin cannot silently double it (the first launch of this wrapper died -# with 'no .../315fix-build/bin/bin/llama-server' for exactly that reason). -strip_bin() { local d=${1%/}; [ "$(basename "$d")" = "bin" ] && dirname "$d" || echo "$d"; } -FIX=$(strip_bin "$FIX") -CTRL=$(strip_bin "$CTRL") - -WATCH=$HOME/amd-oss/tools/315-trace/kfd_smi_watch -OUT=$HOME/wt/$TAG-evict -CONSOLE=$HOME/wt/$TAG-console.log - -# Two facts learned by smoke-testing this instrument before trusting it (2026-10-12): -# 1. the watcher MUST run as root, or it sees no events at all - and it fails silently -# (exit 0, header-only file). Verified with sudo: 75 Q_EVICT / 79 Q_RESTORE during one -# HRX load, 0 unprivileged. So a header-only file is 'no privilege', not 'no evictions'. -# 2. it writes TSV with a '#recv_boot_ns...' header, not NDJSON. The name in this script -# reflects that; correlate.py reads the TSV. -HSA=${IREE_HAL_AMDGPU_LIBHSA_PATH:-/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1} -export IREE_HAL_AMDGPU_LIBHSA_PATH="$HSA" - -need() { [ -e "$1" ] || { echo "ABORT: missing $1"; exit 2; }; } -need "$WATCH"; need "$FIX/bin/llama-server"; need "$CTRL/bin/llama-server"; need "$HSA" -sudo -n true 2>/dev/null || { echo "ABORT: the watcher needs passwordless sudo; sudo -n failed"; exit 2; } -mkdir -p "$OUT" - -echo "=== task-8 eviction/preemption trace ===" -echo "tag : $TAG" -echo "pairs : $PAIRS (arms alternate control/fix inside one composition)" -echo "fix bin : $($FIX/bin/llama-server --version 2>&1 | head -1) ($FIX)" -echo "ctrl bin : $($CTRL/bin/llama-server --version 2>&1 | head -1) ($CTRL)" -echo "watcher : $WATCH (as root)" -echo "hsa : $HSA" -echo -echo "This run's PURPOSE is the eviction timeline, not an acceptance rate. A clean run is" -echo "recorded as UNINFORMATIVE (with its clean-arm count), never as evidence about the fault." -echo - -# The watcher is started LATER, after the box lock and the GPU-quiet check (see below). -# It was originally started here, which meant it ran its 3h TTL while a peer still held HRX0 -# and recorded events from a window that was not ours. - -# Run the frozen driver unchanged. NOSTOP=1 so the run continues past a fault and gives us -# more than one faulting arm to correlate; MINCLEAN=0 because this is not an acceptance run. -# Run the frozen driver unchanged, under the box lock. -# -# The 315 harnesses share a convention (315-attrib.sh:41-48): take -# ~/.cache/lax-decode/box.lock on fd 9 and pass NOLOCK=1 down, so a second arm runner -# cannot start concurrently. That comment also warns that the driver's fallback is to -# proceed UNLOCKED - so taking the lock here is what makes concurrent runs impossible, -# not merely unlikely. -# -# MINCLEAN is a goalpost, not a floor: 315-paired.sh:147 stops when the clean count -# REACHES it. Passing MINCLEAN=0 therefore means "stop at zero clean arms", which ended -# the first launch of this wrapper after pair 1 (2 arms run, not 12). Set it above anything -# reachable so the PAIRS cap is what ends the run, and let NOSTOP keep a fault from -# aborting a diagnostic run. -UNREACHABLE=$(( PAIRS * 2 + 1 )) - -if [ -n "${NOLOCK:-}" ]; then - echo "inheriting the caller's box lock (NOLOCK=1)" -else - # Take BOTH lock conventions, because this box has two and they do not overlap: - # ~/npu.lock - the box-wide convention. Observed holding 12 pids - # from three unrelated workers (one peer batch and - # two NPU-goal workers), so this is the one that - # actually serialises GPU use. - # ~/.cache/lax-decode/box.lock - narrower, used only by the older 315 harnesses. - # Held by nobody but us, so on its own it excludes - # nothing (measured 2026-10-12). - # Taking only box.lock was the bug: the run looked lock-protected while a peer ran HRX0. - LOCK=$HOME/.cache/lax-decode/box.lock - mkdir -p "$(dirname "$LOCK")" - exec 9>"$LOCK" - if flock -w "${LOCK_WAIT:-900}" 9; then - echo "box lock held $(date -Is) ($LOCK)" - else - echo "ABORT: box lock unobtainable after ${LOCK_WAIT:-900}s - a peer is running arms." - echo "Nothing was started. Retry when the box is quiet." - exit 3 - fi - NPULOCK=$HOME/npu.lock - exec 8>"$NPULOCK" - if flock -w "${NPU_LOCK_WAIT:-900}" 8; then - echo "npu lock held $(date -Is) ($NPULOCK) - the box-wide convention" - else - echo "ABORT: ~/npu.lock unobtainable after ${NPU_LOCK_WAIT:-900}s." - echo "That lock is held by other workers on this box, and the campaign composes ~5" - echo "neighbours in the 95-110 GiB band, which would evict their memory." - echo "Nothing was started." - exit 3 - fi -fi - -# The lock only excludes harnesses that FOLLOW the convention. Observed 2026-10-12: a peer -# batch (~/wt/loom-ub389) runs HRX jobs without ever taking box.lock, so the lock was free -# while a peer held HRX0. The composition would have evicted that peer's memory. So also -# poll for actual GPU quiescence: any foreign '-dev HRX0' process, other than the two -# Vulkan servers this box keeps resident. -wait_for_gpu_quiet() { - local deadline=$(( $(date +%s) + ${QUIET_WAIT:-1800} )) - while :; do - local foreign - foreign=$(ps -eo pid,args 2>/dev/null \ - | grep -- "-dev HRX0" | grep -v grep \ - | grep -v "$OUT" | grep -v "315-" || true) - # Also refuse if a peer's HRX job is mid-flight even without '-dev HRX0' in argv. - local peer - peer=$(ps -eo pid,args 2>/dev/null | grep -E "loom-ub389|repro336" | grep -v grep || true) - if [ -z "$foreign" ] && [ -z "$peer" ]; then - echo "GPU quiet: no foreign HRX job (waited as needed)" - return 0 - fi - if [ "$(date +%s)" -ge "$deadline" ]; then - echo "ABORT: GPU still busy after ${QUIET_WAIT:-1800}s; a foreign HRX job would be evicted." - echo " foreign: ${foreign:-none}" - echo " peer: ${peer:-none}" - echo "Nothing was started." - return 3 - fi - sleep 20 - done -} - -wait_for_gpu_quiet || exit $? - -# Now that the box is ours, start the watcher. Root, or the trace is silently empty. -# Start it BEFORE the driver composes neighbours so the neighbours' own BOs get owners in -# the stream (the driver's fallback would otherwise miss their registration). -sudo -n nohup "$WATCH" -e all -t 10800 -o "$OUT/kfd_smi.tsv" -q >"$OUT/watcher.err" 2>&1 & -sleep 3 -WPID=$(pgrep -f "kfd_smi_watch.*$OUT/kfd_smi.tsv" | head -1) -if [ -z "$WPID" ]; then - echo "ABORT: the watcher did not start; stderr:"; head -5 "$OUT/watcher.err"; exit 3 -fi -echo "watcher pid $WPID -> $OUT/kfd_smi.tsv" -cleanup() { sudo -n kill "$WPID" 2>/dev/null; } -trap cleanup EXIT INT TERM - -# NOLOCK=1 is REQUIRED, not optional. 315-paired.sh:47-52 re-opens -# ~/.cache/lax-decode/box.lock on its OWN fd 9 and flocks it, falling back to "proceeding -# CONCURRENTLY (UNLOCKED)" on timeout. flock is per-open-file-description, so a SECOND open -# of a file this process tree already holds a lock on will BLOCK. -# -# Observed: run 6 (315evict6) passed all four of this wrapper's gates, began composing, then -# deadlocked here - the driver's child sat in locks_lock_inode_wait on box.lock while this -# wrapper held box.lock on fd 9 and npu.lock on fd 8. After LOCK_WAIT it would have proceeded -# UNLOCKED, so the failure is not even protective. -# -# The driver documents the convention in its own header: NOLOCK=1 means the caller already -# holds the lock on an inherited fd 9 - exactly our case. -MINCLEAN="$UNREACHABLE" NOSTOP=1 NOLOCK=1 \ - FIX="$FIX" CTRL="$CTRL" TAG="$TAG" PAIRS="$PAIRS" \ - bash "$HOME/wt/315-campaign.sh" 2>&1 | tail -20 - -cleanup -trap - EXIT INT TERM - -echo -echo "=== watcher summary ===" -if [ -s "$OUT/kfd_smi.tsv" ] && [ "$(wc -l < "$OUT/kfd_smi.tsv")" -gt 1 ]; then - EV=$(grep -c "Q_EVICT" "$OUT/kfd_smi.tsv" 2>/dev/null || echo 0) - RS=$(grep -c "Q_RESTORE" "$OUT/kfd_smi.tsv" 2>/dev/null || echo 0) - echo "events: Q_EVICT=$EV Q_RESTORE=$RS lines=$(wc -l < "$OUT/kfd_smi.tsv")" - echo "triggers seen:" - grep "Q_EVICT" "$OUT/kfd_smi.tsv" 2>/dev/null | grep -o "trigger=[A-Z_]*" | sort | uniq -c | sort -rn | head -6 - echo "restores not rescheduled (the interesting case):" - grep "Q_RESTORE" "$OUT/kfd_smi.tsv" 2>/dev/null | grep -c "rescheduled=no" || true -else - echo "watcher recorded ONLY A HEADER (or nothing) - check that it ran as root." - echo "The trace is UNINFORMATIVE, not evidence. stderr:"; head -5 "$OUT/watcher.err" 2>/dev/null -fi - -echo -echo "=== arm verdicts ===" -if [ -f "$CONSOLE" ]; then - grep -E "^ pair" "$CONSOLE" | tail -20 - # grep -c prints 0 AND exits 1 when there are no matches, so `|| echo 0` appends a SECOND - # 0 and the value becomes "0\n0" - which made `[ "$FA" -eq 0 ]` fail with - # 'integer expected'. Take the count and normalise in one step. - IN=$(grep -cE "^ pair.*gtt_start=(9[5-9]|10[0-9]|110)G" "$CONSOLE" 2>/dev/null) - IN=${IN:-0} - FA=$(grep -E "^ pair.*gtt_start=(9[5-9]|10[0-9]|110)G" "$CONSOLE" 2>/dev/null | grep -c "FAULT") - FA=${FA:-0} - echo "in-regime arms: $IN faults: $FA" - if [ "$FA" -eq 0 ]; then - echo "NO in-regime fault in this run -> the eviction trace has no faulting half to correlate." - echo "Record it as UNINFORMATIVE (clean-arm count $IN), per the task-8 contract. A clean" - echo "trace is NOT evidence about the fault's mechanism." - else - echo "A fault is present -> correlate each faulting arm against the eviction timeline." - echo " # arm-pid.py binds an arm to its server PID via the PORT in that arm's own" - echo " # server.log ('listening on http://127.0.0.1:'). Run it WHILE the arm is" - echo " # live: a finished arm is refused rather than guessed, because correlate.py's" - echo " # own fallback (most-frequent eviction subject) was observed landing on a dead" - echo " # pid, and a correlation against the wrong process fails silently." - echo " for D in \$OUTDIR/*-p*c-* \$OUTDIR/*-p*l-*; do" - echo " PID=\$(python3 \$HOME/wt/arm-pid.py \"\$D\" 2>/dev/null) || continue" - echo " python3 \$HOME/amd-oss/tools/315-trace/correlate.py \\" - echo " --smi \$OUT/kfd_smi.tsv --server \"\$D/server.log\" --pid \$PID \\" - echo " --timeline \$OUT/correlated-\$(basename \$D).tsv" - echo " done" - fi -else - echo "no console at $CONSOLE" -fi diff --git a/dev/315/315-campaign/315-faceb.sh b/dev/315/315-campaign/315-faceb.sh deleted file mode 100755 index 07c04234e..000000000 --- a/dev/315/315-campaign/315-faceb.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 Face-B probe. Diagnostic, never acceptance-eligible. -# -# Face B's landscape from the 142 recorded arms: 7 events, all in-band at gtt=98, all aborting early -# (req 13-47), **none** divergent, and zero of the 29 out-of-band arms - so it shows no evidence of -# tracking the regime, and no arm that produced one had the logits fingerprint enabled. What Face B is -# for certain is the server's own chat-template parse failing: "Content-only" is the parser name -# (common/chat.cpp:812) and the 500 comes from the parse, i.e. the model emitted text that does not fit -# the format the engine expects. -# -# The discriminating question is whether that text is downstream of NaN logits. `libllama-common.so` -# implements both switches needed to ask it: -# GGML_HRX_LOGITS_FP=1 logits fingerprint at every sample: nan= and sentinel= counts -# GGML_HRX_NAN_CONTINUE=1 keep sampling instead of erroring when the logits contain NaN -# -# So this run makes the continuation *induced* - a Face-A precursor is forced to continue and we watch -# what the engine's parser does with the result. Read it as a causal chain, not as a natural Face-B -# reproduction: -# - 500s appear only in arms that also logged NaN logits, and after them -> one mechanism, two -# observables: the guard's choice (abort vs continue) decides which face you see -# - 500s appear in arms whose logits were NaN-free -> Face B is a separate -# finite-corruption path, downstream of the LM head, and must be named on its own -set -uo pipefail -PAIRS=${1:-8}; TAG=${2:-315faceb} - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${FACE_KEY:-5400}" 9; then - echo "faceb: box lock held $(date -Is)" -else - echo "faceb: lock unobtainable after ${FACE_KEY:-5400}s - aborting rather than running concurrently" - exit 3 -fi - -cd "$HOME/wt" || exit 1 -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ -CONTROL_EVERY=999 NOSTOP=1 MINCLEAN=999 NOLOCK=1 ARM_TIMEOUT=${ARM_TIMEOUT:-1200} \ -EXTRA_ENV="GGML_HRX_LOGITS_FP=1 GGML_HRX_NAN_CONTINUE=1 GGML_HRX_SENTINEL=1" \ - bash "$HOME/wt/315-paired.sh" "$PAIRS" "${NB:-2}" "${NCTX:-2048}" \ - "$HOME/wt/315instr3-build/bin" "$HOME/wt/315fp-build/bin" "$TAG" - -echo "=== faceb readout $(date -Is) ===" -printf "%-10s %-6s %-8s %-10s %-8s %-8s\n" arm verdict "nan_logits" "500_format" "sentinel" "divergent" -for d in "$HOME"/wt/rt-det/"$TAG"-*/; do - [ -d "$d" ] || continue - a=$(basename "$d" | sed "s/${TAG}-//;s/-GLM.*//") - nl=$(grep -c "hrx-logits-fp" "$d/server.log" 2>/dev/null || echo 0) - nln=$(grep "hrx-logits-fp" "$d/server.log" 2>/dev/null | grep -vc " nan=0 " || echo 0) - fb=$(grep -c "Content-only" "$d/server.log" 2>/dev/null || echo 0) - sn=$(grep -c "hrx-sentinel" "$d/server.log" 2>/dev/null || echo 0) - idn=$(python3 -c "import json;print(json.load(open('$d/detector.json'))['identical_distinct'])" 2>/dev/null || echo "?") - printf "%-10s %-6s %-8s %-10s %-8s %-8s\n" "$a" "-" "$nln/$nl" "$fb" "$sn" "$idn" -done -echo "NaN-logits arms and format-500 arms, side by side: a 500 in an arm with zero NaN-logits lines is" -echo "evidence for a separate Face-B path; a 500 only where NaN was logged is evidence they are one." diff --git a/dev/315/315-campaign/315-fix-build.sh b/dev/315/315-campaign/315-fix-build.sh deleted file mode 100644 index 939eefae8..000000000 --- a/dev/315/315-campaign/315-fix-build.sh +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: build the FIX cell. ~/wt/llama-315fix already carries the same three diagnostics as the -# control cell ~/wt/315instr3-build plus the coherence fix, so the two cells differ by exactly the fix. -# Assertions before and after the build, so a silent build of the wrong source cannot be mistaken for a -# measurement (the cpe.cpp patch-path mistake is the precedent). -set -euo pipefail -PIN=e44c9d01a4d5110cecaf45472a717f46cd1abbf9 -SRC=$HOME/wt/llama-315fix -B=$HOME/wt/315fix-build -HRX=$HOME/wt/pin-fork-hrx/third_party/hrx-system -LOG=$HOME/wt/315fix-build.log -say() { echo "[$(date +%H:%M:%S)] $*"; } - -say "preflight" -[ -e "$SRC/.git" ] || { echo "ABORT: $SRC is not a worktree"; exit 1; } -[ "$(git -C "$SRC" rev-parse HEAD)" = "$PIN" ] || { echo "ABORT: $SRC not at the pin"; exit 1; } -[ -d "$HRX" ] || { echo "ABORT: hrx-system missing"; exit 1; } -G=$SRC/ggml/src/ggml-hrx/ggml-hrx.cpp -grep -q 'memory_type |= HRX_MEMORY_TYPE_HOST_COHERENT;' "$G" || { echo "ABORT: the fix is absent from the source"; exit 1; } -if grep -A1 'HRX_MEMORY_TYPE_HOST_LOCAL | HRX_MEMORY_TYPE_DEVICE_VISIBLE;' "$G" | grep -q 'if (direct_host_binding)'; then - echo "ABORT: coherence is still gated on direct_host_binding - the fix is not in effect"; exit 1 -fi -say "source OK: coherence unconditional, trace marker=$(grep -c 'hrx-buf' "$G")" -git -C "$SRC" diff > "$HOME/wt/315-fix-trace.patch" -say "candidate patch sha256 $(sha256sum "$HOME/wt/315-fix-trace.patch" | cut -d' ' -f1)" -say "control cell for comparison: $(~/wt/315instr3-build/bin/llama-server --version 2>&1 | head -1)" - -say "configure" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "ABORT: cmake failed, see $LOG"; tail -20 "$LOG"; exit 1; } - -say "build (-j12; the peer is building with ~4)" -cmake --build "$B" -j 12 >> "$LOG" 2>&1 || { echo "ABORT: build failed, see $LOG"; tail -30 "$LOG"; exit 1; } - -say "binary identity"; "$B/bin/llama-server" --version -LIB=$(ls "$B"/bin/libggml-hrx.so* 2>/dev/null | head -1) -[ -n "$LIB" ] || { echo "ABORT: libggml-hrx.so not found"; exit 1; } -for M in hrx-wb-fp hrx-dl-fp hrx-buf; do - N=$(grep -c "$M" "$LIB" || true); say " marker $M=$N"; [ "$N" -gt 0 ] || { echo "ABORT: $M absent from the binary"; exit 1; } -done -say "FIXBUILD_DONE bin_dir=$B/bin" diff --git a/dev/315/315-campaign/315-forcebarrier.sh b/dev/315/315-campaign/315-forcebarrier.sh deleted file mode 100644 index f98b03ff0..000000000 --- a/dev/315/315-campaign/315-forcebarrier.sh +++ /dev/null @@ -1,128 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: the last live mechanism family, tested directly - intra-submission packet ordering. -# -# The reasoning this rests on: -# * the driver does NOT infer memory hazards; it trusts recorded flags -# (`aql_block_processor.c:126` "Do not infer memory hazards here by walking command operands"), and -# `..._packet_has_barrier` (:103-116) sets the AQL barrier bit only for EXECUTION_BARRIER packets, FINAL -# packets, and packet 0 of a submission with waits or an inline acquire; -# * the queue profile of GLM's decode shows **92 of 612 dispatches carry no barrier** -# (`evidence/315-queue-profile.md`), i.e. 15% of the program may overlap a neighbour; -# * every lever tested so far operates at *submission* granularity - `GRAPH_SYNC` after each replay -# (`315gsync2`: control 1 fault in 4, fix 0 in 2), the host read path (`scacquire` + covered timepoint), -# and coherence (`315accept`: 9 clean + 1 fault in 10). None of them orders two packets *inside* one -# submission, which is exactly where this family lives; -# * `HRX_COMMAND_BUFFER_MODE=pm4`, the other lever on the packet mechanism, is unusable: all four of its arms -# failed the FIRST request with HTTP 500 (`315pm4`), so it cannot compare fault rates. -# -# So: force every payload packet to be barrier-bearing, env-gated and off by default, and run the paired A/B. -# Both cells run the SAME build (so the only difference is the env var), with the same inducer. -# -# Readout, pre-committed: -# fault rate falls in the forced-barrier cell -> intra-submission overlap IS the carrier; the mechanism is -# "a missing recorded execution dependency in the generated command stream", and the next step is to name -# the specific producer/consumer pair by enumerating the unbarriered dispatches on the layer-0 path -# fault rate unchanged -> packet ordering inside a submission is not the carrier either, -# and the ordering family is closed completely (submission-level AND packet-level) -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-forcebarrier.sh [pairs] -set -uo pipefail -PAIRS=${1:-4} -TAG=315fbar -SRC=$HOME/wt/llama-315copy -HRXSRC=$HOME/wt/hrx-forcebarrier -B=$HOME/wt/315fbar-build -LOG=$HOME/wt/315fbar-build.log - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${FBAR_LOCK_WAIT:-10800}" 9; then - echo "fbar: box lock held $(date -Is)" -else - echo "fbar: lock unobtainable - aborting rather than building concurrently" - exit 3 -fi - -if [ ! -f "$HRXSRC/runtime/src/iree/hal/drivers/amdgpu/aql_block_processor.c" ]; then - echo "fbar: copying the runtime tree to $HRXSRC $(date -Is)" - rm -rf "$HRXSRC" - cp -a "$HOME/wt/pin-fork-hrx/third_party/hrx-system" "$HRXSRC" || { echo "fbar: copy failed"; exit 1; } -fi - -python3 - "$HRXSRC/runtime/src/iree/hal/drivers/amdgpu/aql_block_processor.c" <<'PY' -import pathlib, sys -p = pathlib.Path(sys.argv[1]); t = p.read_text() -if "HRX_FORCE_BARRIERS" in t: - print("fbar: patch already present"); raise SystemExit(0) -anchor = """static bool iree_hal_amdgpu_aql_block_processor_packet_has_barrier( - const iree_hal_amdgpu_aql_block_processor_t* processor, - uint32_t packet_index, - iree_hal_amdgpu_aql_block_processor_packet_flags_t packet_flags) {""" -add = anchor + """ - // engine#315 diagnostic: HRX_FORCE_BARRIERS=1 makes every payload packet barrier-bearing, so a fault that - // needs two unbarriered packets to overlap disappears when it is set. Off unless the variable is set to a - // non-empty value other than "0"; nothing changes for any other run. - { - const char* hrx_force_barriers = getenv("HRX_FORCE_BARRIERS"); - if (hrx_force_barriers != NULL && hrx_force_barriers[0] != '\\0' && hrx_force_barriers[0] != '0') { - return true; - } - }""" -assert t.count(anchor) == 1, "anchor not unique" -t = t.replace(anchor, add) -if "#include " not in t: - i = t.find("#include") - t = t[:i] + "#include \n" + t[i:] -p.write_text(t); print("fbar: patched", p) -PY -grep -c HRX_FORCE_BARRIERS "$HRXSRC/runtime/src/iree/hal/drivers/amdgpu/aql_block_processor.c" || { echo "fbar: patch missing"; exit 1; } - -echo "fbar: configuring $B $(date -Is)" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRXSRC" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "fbar: cmake failed"; tail -20 "$LOG"; exit 1; } -echo "fbar: building -j 6 $(date -Is)" -cmake --build "$B" -j 6 >> "$LOG" 2>&1 || { echo "fbar: build failed"; tail -30 "$LOG"; exit 1; } -"$B/bin/llama-server" --version || exit 1 - -# Fix-cell launcher: same binary, one env var. -W=$HOME/wt/315fbar-on-bin -rm -rf "$W"; mkdir -p "$W" -for f in "$B"/bin/*; do b=$(basename "$f"); if [ "$b" != "llama-server" ]; then ln -sf "$f" "$W/$b"; fi; done -printf '%s\n' "#!/bin/sh" "# engine#315: forced-barrier cell - same binary as the control, one env var." "export HRX_FORCE_BARRIERS=1" "exec $B/bin/llama-server \"\$@\"" > "$W/llama-server" -chmod +x "$W/llama-server" - -echo "fbar: running the paired A/B $(date -Is)" -sleep 2 -CBIN="$B/bin" FBIN="$W" NOLOCK=1 \ -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} ARM_TIMEOUT=${ARM_TIMEOUT:-1800} \ - bash "$HOME/wt/315-attrib.sh" "$PAIRS" "$TAG" 1 \ - "GGML_HRX_TENSOR_FP=tail:16 GGML_HRX_TENSOR_FP_NANONLY=1" - -c=$HOME/wt/$TAG-console.log -echo "=== fbar readout $(date -Is) ===" -grep -hE "^ pair" "$c" || true -echo "control(no env): OK=$(grep -c 'arm c: OK' "$c" || true) FAULT=$(grep -c 'arm c: \(FAULT\|INVALID\)' "$c" || true) OOR=$(grep -c 'arm c: OOR' "$c" || true)" -echo "fix(FRB=1): OK=$(grep -c 'arm l: OK' "$c" || true) FAULT=$(grep -c 'arm l: \(FAULT\|INVALID\)' "$c" || true) OOR=$(grep -c 'arm l: OOR' "$c" || true)" -echo "A fall in the fix cell names intra-submission overlap as the carrier; no change closes the ordering family." diff --git a/dev/315/315-campaign/315-fp-hunt.sh b/dev/315/315-campaign/315-fp-hunt.sh deleted file mode 100755 index b047da0e7..000000000 --- a/dev/315/315-campaign/315-fp-hunt.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: repeat in-regime instrumented arms until one faults, so the diagnostic record exists -# for a faulting event. -# -# The fault rate in-regime is ~5%/arm, so a single flag-on arm is unlikely to fault (a clean arm is -# the norm and is still useful: it gives the normal record pattern). This driver runs arms -# sequentially, each in its own tag/dir so every log is kept, and stops at the first arm with a -# fault on either face. Re-composition happens per arm because 315-instr-capture.sh owns its regime; -# that costs a few minutes per arm and keeps each arm independently valid. -# -# Usage: 315-fp-hunt.sh [tagbase] -set -uo pipefail -BIN=${1:?bin dir}; N=${2:-15}; TAGB=${3:-315on} -M=/home/bcloud/models/GLM-4.7-Flash-Q4_K_M.gguf -HERE=$(dirname "$0") -LOG=$HOME/wt/$TAGB-hunt.log - -echo "=== hunt start $(date -Is) bin=$BIN arms=$N tagbase=$TAGB ===" | tee -a "$LOG" -for i in $(seq 1 "$N"); do - TAG="$TAGB-$i" - echo "=== hunt arm $i/$N tag=$TAG $(date -Is) ===" | tee -a "$LOG" - bash "$HERE/315-instr-capture.sh" "$BIN" 1 2048 "$TAG" 1 2>&1 | tee -a "$LOG" - O=$HOME/wt/rt-det/$TAG-$(basename "$M" .gguf) - nan=$(grep -c "HRX returned NaN logits" "$O/server.log" 2>/dev/null || true) - hsa=$(grep -iE "HSA_STATUS_ERROR|Queue error|wait for HRX graph replay commands failed" \ - "$O/server.log" 2>/dev/null | grep -vcE "hrx_allocator_allocate_buffer|allocate HRX host staging buffer" || true) - utf8=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['utf8_failures'])" 2>/dev/null || echo 0) - parse=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['parse_failures'])" 2>/dev/null || echo 0) - fp=$(grep -c "\[hrx-wb-fp\]" "$O/server.log" 2>/dev/null || true) - gtt=$(grep -o "gtt_start [0-9]*" "$O/run.log" 2>/dev/null | head -1) - echo " arm $i verdict: $gtt nan=$nan hsa=$hsa utf8=$utf8 parse=$parse wb_records=$fp" | tee -a "$LOG" - if [ "${nan:-0}" != "0" ] || [ "${hsa:-0}" != "0" ] || [ "${utf8:-0}" != "0" ] || [ "${parse:-0}" != "0" ]; then - echo "FAULT CAUGHT on arm $i — stopping the hunt. Records: $O/server.log" | tee -a "$LOG" - echo "315HUNT_FAULT arm=$i out=$O" | tee -a "$LOG" - exit 0 - fi - if [ "${fp:-0}" = "0" ]; then - echo "WARNING: arm $i produced no [hrx-wb-fp] records — the diagnostic may not be reaching the" | tee -a "$LOG" - echo " server (check --setenv GGML_HRX_WRITEBACK_FINGERPRINT); continuing anyway." | tee -a "$LOG" - fi -done -echo "315HUNT_DONE arms=$N no fault caught" | tee -a "$LOG" diff --git a/dev/315/315-campaign/315-fp2.sh b/dev/315/315-campaign/315-fp2.sh deleted file mode 100755 index 33a991a7a..000000000 --- a/dev/315/315-campaign/315-fp2.sh +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 fp2: the ORDERED locator. Only worth running if 315gsync2 shows the fault survives a sync. -# -# Why: section 53 withdrew the by-name per-node attribution because the fingerprint reads host-visible tensors -# out of a host-visible non-coherent arena through ggml_backend_tensor_get, which for those is a plain memcpy -# with no synchronisation. The per-node lines were therefore not a snapshot of one instant - proven by -# contradiction inside the arms themselves (ffn_moe_out finite downstream of a 99.95 %-NaN ffn_moe_weighted). -# -# `tools/patch-fp-sync.py` inserts one `ggml_backend_synchronize(backend)` before the fingerprint's read loop, -# which orders those reads. The trade-off is explicit: a sync per graph is itself an ordering intervention, so -# this build can inhibit the fault it is meant to locate. -# -# gsync2 says the fix cell is CLEAN -> the ordering is the mechanism; do not run this, there is nothing -# left to locate, and the remedy is scoped around the wait -# gsync2 says the fault SURVIVES a sync -> run this: the sync is not the carrier, and its per-node lines are -# then valid evidence about where the corruption is -# -# Usage: 315-fp2.sh [arms] [tag] -set -uo pipefail -ARMS=${1:-4}; TAG=${2:-315fp2} -SRC=$HOME/wt/llama-315copy -B=$HOME/wt/315fp2-build -HRX=$HOME/wt/pin-fork-hrx/third_party/hrx-system -LOG=$HOME/wt/315fp2-build.log - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${FP2_LOCK_WAIT:-10800}" 9; then - echo "fp2: box lock held $(date -Is)" -else - echo "fp2: lock unobtainable after ${FP2_LOCK_WAIT:-10800}s - aborting rather than running concurrently" - exit 3 -fi - -grep -q "ggml_backend_synchronize(backend);" "$SRC/ggml/src/ggml-hrx/ggml-hrx.cpp" \ - || { echo "fp2: ABORT - the ordering patch is not in the source (tools/patch-fp-sync.py)"; exit 1; } -grep -q "hrx_fp_name_listed" "$SRC/ggml/src/ggml-hrx/ggml-hrx.cpp" \ - || { echo "fp2: ABORT - the anchored-match patch is not in the source"; exit 1; } - -# Pre-committed gate: refuse to spend the box if gsync2 already answered the question. -C=$HOME/wt/315gsync2-console.log -if [ -f "$C" ]; then - fixf=$(grep -c 'arm l: \(FAULT\|INVALID\)' "$C" || true) - fixok=$(grep -c 'arm l: OK' "$C" || true) - echo "fp2: gsync2 so far - fix cell OK=$fixok fault=$fixf" - # A gate reading another run's console must distinguish "the branch I do not need" from "no data yet": - # an empty or OOR-only console is the second state, and treating it as the first is what started this - # locator on 2026-10-10 against a console that had been archived (mechanism doc section 58). - if [ "${fixok:-0}" -eq 0 ] && [ "${fixf:-0}" -eq 0 ]; then - echo "fp2: ABORT - no scored fix-cell arms in $C yet; refusing to treat 'no data' as 'branch not needed'" - exit 0 - fi - if [ "${fixok:-0}" -ge 3 ] && [ "${fixf:-0}" -eq 0 ]; then - echo "fp2: ABORT - gsync2's fix cell is clean; the ordering is the mechanism and this locator has no question left" - exit 0 - fi -else - echo "fp2: WARNING - no gsync2 console yet, so the gate cannot be checked; proceeding" -fi - -echo "fp2: configuring $B $(date -Is)" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "fp2: cmake failed"; tail -20 "$LOG"; exit 1; } -echo "fp2: building -j 6 $(date -Is)" -cmake --build "$B" -j 6 >> "$LOG" 2>&1 || { echo "fp2: build failed"; tail -30 "$LOG"; exit 1; } -"$B/bin/llama-server" --version - -LIB=$(ls "$B"/bin/libggml-hrx.so* 2>/dev/null | head -1) -for marker in hrx-ids-fp hrx-finite-fp; do - grep -qa "$marker" "$LIB" || { echo "fp2: ABORT - $marker not in $LIB"; exit 1; } -done -echo "fp2: ordered fingerprint + ids + finite hash present in $(basename "$LIB")" - -NAMES="norm,result_norm,result_output" -for l in $(seq 0 63); do - for n in ffn_moe_topk ffn_moe_argsort ffn_moe_probs ffn_moe_weights ffn_moe_gate ffn_moe_up ffn_moe_down ffn_moe_weighted ffn_moe_out ffn_swiglu ffn_shexp ffn_out l_out; do - NAMES="$NAMES,$n-$l" - done -done -echo "fp2: name list $(printf '%s' "$NAMES" | tr ',' '\n' | wc -l) entries" - -# REGIME_LO=80: measured, not chosen for the strictest reading. A neighbour costs ~17 GiB of GTT (a 17 GiB model -# plus its context), so the composition lands on discrete steps: neighbours at 80 GiB score arms at 98 GiB (inside -# the frozen 95-110 gate), neighbours at 97 GiB score them at 115 GiB (outside it - that is what the first -# 315gsync2 attempt did with LO=85, and its arms are archived as 315gsync2-overshoot-console.log). There is no -# setting that lands at 101-105 GiB with this granularity; 98 GiB is the in-regime point this box can hold. -echo "fp2: running $ARMS arms at REGIME_LO=${REGIME_LO:-80} $(date -Is)" -CBIN="$B/bin" FBIN="$B/bin" NOLOCK=1 \ -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ -ARM_TIMEOUT=${ARM_TIMEOUT:-1800} \ - bash "$HOME/wt/315-attrib.sh" "$ARMS" "$TAG" 999 \ - "GGML_HRX_TENSOR_FP=$NAMES GGML_HRX_TENSOR_FP_NANONLY=1" - -echo "=== fp2 readout $(date -Is) ===" -CONSOLE=$HOME/wt/$TAG-console.log -grep -hE "^ pair" "$CONSOLE" || true -echo "fault: $(grep -c 'arm l: \(FAULT\|INVALID\)' "$CONSOLE" || true) clean: $(grep -c 'arm l: OK' "$CONSOLE" || true) oor: $(grep -c 'arm l: OOR' "$CONSOLE" || true)" -for d in "$HOME"/wt/rt-det/"$TAG"-*/; do - [ -d "$d" ] || continue - b=$(basename "$d" | sed "s/$TAG-//;s/-GLM.*//") - echo "--- $b guard=$(grep -c 'HRX returned NaN logits' "$d/server.log" 2>/dev/null || true) ids=$(grep -c 'hrx-ids-fp' "$d/server.log" 2>/dev/null || true)" - grep -m6 "hrx-tensor-fp" "$d/server.log" 2>/dev/null | cut -c1-200 - grep "hrx-ids-fp" "$d/server.log" 2>/dev/null | grep -E "oor=[1-9]|dup_pairs=[1-9]" | head -4 | cut -c1-200 -done -echo "The per-node lines are now ordered; a T=1-only NaN row still carries no per-token information." diff --git a/dev/315/315-campaign/315-graphsync.sh b/dev/315/315-campaign/315-graphsync.sh deleted file mode 100755 index 4ae686a0d..000000000 --- a/dev/315/315-campaign/315-graphsync.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: the visibility/ordering falsifier. -# -# Theory under test. The logits allocation is host-visible but NOT host-coherent (`hv52=1 hv56=0` in every -# driver line), so a host read of it depends on an invalidate/sync happening after the device's write. The -# two faces are what a reader sees when that ordering is wrong: NaN if the write has not landed (Face A, -# caught by the sampler guard) or the buffer's previous finite contents (Face B, which reaches the sampler, -# emits degenerate text, and fails at the chat parser). Making the *allocation* coherent did not remove the -# fault (315accept, 9 clean + 1 Face-B in 10 fix-cell arms, with hv56=1 proving the patch was live), so if -# the family is right at all, the defect is in the *timing* of the sync and not in the coherence bit. -# -# `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC` is the direct test: in command-program-executor.cpp:2034 it calls -# `hrx_stream_synchronize(context.stream)` after each graph replay, so every later read - including the -# host's read of the logits - must follow the device's writes. One sync per graph, so the cost is small. -# -# Readout, comparing against the arms already on disk from the same harness with no knob (the 315chain and -# 315nan sessions, which faulted with nan=1 and with nan=0/req<66 respectively): -# faults fall to zero here -> the mechanism is a missing/early sync on the consumer side, and a scoped -# variant of this sync is the minimal forward-only fix -# the rate is unchanged -> visibility/ordering is dead as a mechanism *for the producer*, and the -# corruption is computed: a kernel/shape defect, which is the opposite direction to look in -# -# Deliberately NO other instrument in this run: the tensor fingerprint adds 16 device-to-host reads per -# graph and the copy probe prints from the submit path, either of which could perturb the very timing under -# test. Keep it clean so the comparison is against the no-knob baseline. -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-graphsync.sh [pairs] [tag] -set -uo pipefail -PAIRS=${1:-6}; TAG=${2:-315gsync} - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${GSYNC_LOCK_WAIT:-5400}" 9; then - echo "gsync: box lock held $(date -Is)" -else - echo "gsync: lock unobtainable after ${GSYNC_LOCK_WAIT:-5400}s - aborting rather than running concurrently" - exit 3 -fi - -# Both slots get the same build and the knob applies to both cells, so a control cell would be a duplicate: -# run fix-slot arms only and compare against the no-knob arms already recorded by the same harness. -cd "$HOME/wt" || exit 1 -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ -CONTROL_EVERY=999 NOSTOP=1 MINCLEAN=999 NOLOCK=1 ARM_TIMEOUT=${ARM_TIMEOUT:-1800} \ -EXTRA_ENV="GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1" \ - bash "$HOME/wt/315-paired.sh" "$PAIRS" "${NB:-2}" "${NCTX:-2048}" \ - "$HOME/wt/315instr3-build/bin" "$HOME/wt/315instr3-build/bin" "$TAG" - -echo "=== gsync readout $(date -Is) ===" -# The driver's own verdict is authoritative per arm, so just show the arms and the tallies. -grep -hE "^ pair" "$HOME/wt/$TAG-console.log" || true -echo "clean: $(grep -c 'arm l: OK' "$HOME/wt/$TAG-console.log" || true)" -echo "fault-or-invalid: $(grep -c 'arm l: \(FAULT\|INVALID\)' "$HOME/wt/$TAG-console.log" || true)" -echo "out-of-regime: $(grep -c 'arm l: OOR' "$HOME/wt/$TAG-console.log" || true)" -echo "compare with the same harness and no knob: 315chain pair 1 faulted (nan=0, req<66) and 315nan" -echo "faulted 16/17 in the fix cell, so a zero-fault result here is the discriminating outcome." diff --git a/dev/315/315-campaign/315-idprobe.sh b/dev/315/315-campaign/315-idprobe.sh deleted file mode 100755 index 75562d9b2..000000000 --- a/dev/315/315-campaign/315-idprobe.sh +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 id probe: are the route ids a clean permutation, and is the NaN in ffn_moe_weights an aliased read? -# -# Why: arm 315row-p1c (in regime, gtt_start=98G) left the fault on the ids. Its route-weight gather -# ffn_moe_weights = ggml_get_rows(ffn_moe_probs, ffn_moe_topk) was NaN for two whole tokens while ffn_moe_probs -# was not, and the ids were never instrumented. Two probe defects made that unanswerable: the name filter was -# std::strstr(list, name) (so "-41" also matched "-4", covering layers 4 and 41 only) and every tensor was read -# as f32, so an i32 ids tensor was fingerprinted as nonsense. -# -# This run builds the patched instrument (~/wt/llama-315copy, patch applied by tools/patch-id-instrument.py): -# * anchored name matching against the comma-separated list; -# * for i32 tensors a real ids dump: first 8 values, min/max, count outside [0, ne0) = oor, and dup_pairs -# (repeated values within a row); -# * for f32 tensors a second hash over the FINITE subset only, so two tensors sharing a NaN payload cannot -# look identical when their finite parts differ. -# -# Readout, pre-committed: -# oor > 0 or dup_pairs > 0 in a faulting arm -> the id list is not a clean permutation: the argsort's rank -# computation (evidence/315-id-producing-kernel.md) is the site, and the stale id is what the expert -# tables consume -# ids clean while the matmuls are NaN -> the argsort candidate dies and the fault is downstream, in -# the table builders or in the matmul's descriptor use -# hash_finite equal across layers 4 and 41 -> an aliased read of the same buffer, not a coincidence -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-idprobe.sh [arms] [tag] -# -# Note on the gate: the composition target defaults to the module's 80 GiB ("State" section of -# dev/315/315-mechanism.md: the probe wrappers used to override it *down* to 70 and under-shot the gate). For the -# strictest reading of the in-regime requirement - task-2's "gtt_start >= 101 GiB" - export REGIME_LO=85 -# REGIME_HI=95: neighbours at 85 GiB put the detector's ~19-21 GiB at ~101-106, inside the 110 GiB upper bound. -# The arms are labelled by their own gtt_start either way, so nothing is inferred from the setting. -set -uo pipefail -ARMS=${1:-4}; TAG=${2:-315id} -SRC=$HOME/wt/llama-315copy -B=$HOME/wt/315id-build -HRX=$HOME/wt/pin-fork-hrx/third_party/hrx-system -LOG=$HOME/wt/315id-build.log - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${IDP_LOCK_WAIT:-10800}" 9; then - echo "id: box lock held $(date -Is)" -else - echo "id: lock unobtainable after ${IDP_LOCK_WAIT:-10800}s - aborting rather than running concurrently" - exit 3 -fi - -grep -q "hrx_fp_name_listed" "$SRC/ggml/src/ggml-hrx/ggml-hrx.cpp" || { echo "id: ABORT - instrument not in the source"; exit 1; } -[ -d "$HRX" ] || { echo "id: ABORT - hrx-system missing"; exit 1; } - -echo "id: configuring $B $(date -Is)" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "id: cmake failed"; tail -20 "$LOG"; exit 1; } -echo "id: building -j 6 $(date -Is)" -cmake --build "$B" -j 6 >> "$LOG" 2>&1 || { echo "id: build failed"; tail -30 "$LOG"; exit 1; } -"$B/bin/llama-server" --version - -LIB=$(ls "$B"/bin/libggml-hrx.so* 2>/dev/null | head -1) -for marker in hrx-ids-fp hrx-finite-fp; do - grep -qa "$marker" "$LIB" || { echo "id: ABORT - $marker not in $LIB"; exit 1; } -done -echo "id: ids and finite-hash instruments present in $(basename "$LIB")" - -# Note on volume and perturbation: NANONLY=1 keeps the f32 tensors quiet, while the i32 branch of the -# patched instrument prints unconditionally (it is asked for by name, and an i32 tensor has no NaN). So the -# added traffic per layer graph is two small ids reads plus, only when a NaN is present, the f32 lines. The -# name list is deliberately narrow (four per layer, not the whole MoE chain) so the extra device->host reads -# stay as small a perturbation as this instrument can be. - -# The node names are suffixed with the layer index (observed: ffn_moe_weights-4, l_out-41), so an ANCHORED -# match needs every layer spelled out. 64 layers is above GLM-4.7-Flash's depth and costs nothing: a name that -# matches nothing is simply never printed. -NAMES="norm,result_norm,result_output" -for l in $(seq 0 63); do - for n in ffn_moe_topk ffn_moe_argsort ffn_moe_weights ffn_moe_probs; do - NAMES="$NAMES,$n-$l" - done -done -echo "id: name list $(printf '%s' "$NAMES" | tr ',' '\n' | wc -l) entries" - -echo "id: running $ARMS arms with the id instrument $(date -Is)" -CBIN="$B/bin" FBIN="$B/bin" NOLOCK=1 \ -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ -ARM_TIMEOUT=${ARM_TIMEOUT:-1800} \ - bash "$HOME/wt/315-attrib.sh" "$ARMS" "$TAG" 999 \ - "GGML_HRX_TENSOR_FP=$NAMES GGML_HRX_TENSOR_FP_NANONLY=1" - -echo "=== id readout $(date -Is) ===" -CONSOLE=$HOME/wt/$TAG-console.log -grep -hE "^ pair" "$CONSOLE" || true -echo "clean: $(grep -c 'arm l: OK' "$CONSOLE" || true) fault: $(grep -c 'arm l: \(FAULT\|INVALID\)' "$CONSOLE" || true) oor: $(grep -c 'arm l: OOR' "$CONSOLE" || true)" -for d in "$HOME"/wt/rt-det/"$TAG"-*/; do - [ -d "$d" ] || continue - b=$(basename "$d" | sed "s/$TAG-//;s/-GLM.*//") - ids=$(grep -c "hrx-ids-fp" "$d/server.log" 2>/dev/null || echo 0) - bad=$(grep "hrx-ids-fp" "$d/server.log" 2>/dev/null | grep -cE "oor=[1-9]|dup_pairs=[1-9]" || true) - echo "--- $b ids_lines=$ids bad_id_lines=$bad nan_guard=$(grep -c 'HRX returned NaN logits' "$d/server.log" 2>/dev/null || true)" - # the faulting evidence, if there is any: the first bad id line, and the ids at the faulting call - grep "hrx-ids-fp" "$d/server.log" 2>/dev/null | grep -E "oor=[1-9]|dup_pairs=[1-9]" | head -4 | cut -c1-200 - grep -m2 "hrx-ids-fp" "$d/server.log" 2>/dev/null | cut -c1-200 -done -echo "Read oor/dup_pairs= against the prediction: a stale id in the top-k view comes from a rank collision." diff --git a/dev/315/315-campaign/315-instr-build.sh b/dev/315/315-campaign/315-instr-build.sh deleted file mode 100755 index 91e460a94..000000000 --- a/dev/315/315-campaign/315-instr-build.sh +++ /dev/null @@ -1,95 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: build the *instrumented* HRX backend at the engine-pinned revision. -# -# Reproduces the study's own build configuration (from ~/wt/build-3971.sh) so the instrumented -# binary differs from the sibling cell only by the diagnostic patch: -# -DCMAKE_BUILD_TYPE=Release, TheRock amdclang{,++}, GGML_HIP=OFF, GGML_HRX=ON, -# -DHRX_SOURCE_DIR=~/wt/pin-fork-hrx/third_party/hrx-system, server on, tests off, -j16. -# -# The diagnostic is inert unless GGML_HRX_WRITEBACK_FINGERPRINT=1, and the build is verified by the -# same style of binary-level assertion the study used: the marker string must be present in -# libggml-hrx.so, otherwise the experiment would silently measure an uninstrumented binary. -# -# Usage: 315-instr-build.sh [patch] -set -euo pipefail -PIN=e44c9d01a4d5110cecaf45472a717f46cd1abbf9 -BASE=$HOME/wt/llama-hybrid -SRC=$HOME/wt/llama-315instr -B=${B:-$HOME/wt/315instr-build} -HRX=$HOME/wt/pin-fork-hrx/third_party/hrx-system -PATCH=${1:-$HOME/wt/315-instrumentation.patch} -LOG=$HOME/wt/315instr-build.log - -say() { echo "[$(date +%H:%M:%S)] $*"; } - -say "preflight" -[ -d "$BASE" ] || { echo "ABORT: fork clone $BASE missing"; exit 1; } -[ -d "$HRX" ] || { echo "ABORT: hrx-system $HRX missing"; exit 1; } -[ -f "$PATCH" ] || { echo "ABORT: patch $PATCH missing"; exit 1; } -say "patch sha256 $(sha256sum "$PATCH" | cut -d' ' -f1)" - -say "fetch + worktree at $PIN" -git -C "$BASE" fetch --quiet origin || true -if git -C "$BASE" cat-file -e "$PIN^{commit}" 2>/dev/null; then - say "pinned commit present locally" -else - echo "ABORT: pinned commit $PIN not in $BASE (or not reachable through origin)"; exit 1 -fi -if [ ! -d "$SRC/.git" ] && [ ! -f "$SRC/.git" ]; then - git -C "$BASE" worktree add -f "$SRC" "$PIN" -else - say "worktree already present at $(git -C "$SRC" rev-parse --short HEAD)" -fi -say "src HEAD $(git -C "$SRC" rev-parse HEAD)" - -say "apply the diagnostic patch" -if git -C "$SRC" apply --reverse --check "$PATCH" 2>/dev/null; then - say "patch already applied" -else - git -C "$SRC" apply --check "$PATCH" - git -C "$SRC" apply "$PATCH" - say "patch applied" -fi -git -C "$SRC" diff --stat | tail -3 - -say "configure" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "ABORT: cmake failed, see $LOG"; tail -20 "$LOG"; exit 1; } - -say "build (-j16)" -cmake --build "$B" -j 16 >> "$LOG" 2>&1 || { echo "ABORT: build failed, see $LOG"; tail -30 "$LOG"; exit 1; } - -say "binary identity" -"$B/bin/llama-server" --version - -say "binary-level assertion: both diagnostic markers must be present" -LIB=$(ls "$B"/bin/libggml-hrx.so* 2>/dev/null | head -1) -if [ -z "$LIB" ]; then echo "ABORT: libggml-hrx.so not found"; exit 1; fi -M1=$(grep -c "hrx-wb-fp" "$LIB" || true) # deferred writeback record format -M2=$(grep -c "hrx-dl-fp" "$LIB" || true) # host download probe record format -say "lib=$LIB hrx-wb-fp=$M1 hrx-dl-fp=$M2" -[ "$M1" -gt 0 ] || { echo "ABORT: binary lacks the writeback marker"; exit 1; } -[ "$M2" -gt 0 ] || { echo "ABORT: binary lacks the download-probe marker"; exit 1; } - -say "INSTRBUILD_DONE bin_dir=$B/bin" diff --git a/dev/315/315-campaign/315-instr-capture.sh b/dev/315/315-campaign/315-instr-capture.sh deleted file mode 100755 index 629f8cbeb..000000000 --- a/dev/315/315-campaign/315-instr-capture.sh +++ /dev/null @@ -1,126 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: capture an in-regime arm with the writeback fingerprint diagnostic enabled. -# -# Runs the pre-registered shape set (via 315-detector.py, so Face B is measured too) against the -# instrumented backend, with GGML_HRX_WRITEBACK_FINGERPRINT=1, and collects the [hrx-wb-fp] records -# plus both failure faces. Set FP=0 for the flag-off control that must behave identically. -# -# Usage: 315-instr-capture.sh [fp=1] -set -uo pipefail -BIN=${1:?bin dir}; NB=${2:-2}; NCTX=${3:-2048}; TAG=${4:-315instr}; FP=${5:-1} -DET=$(dirname "$0")/315-detector.py -M=/home/bcloud/models/GLM-4.7-Flash-Q4_K_M.gguf -PORT="" -# The HRX device only appears when this is set: without it llama-server prints -# "Available devices: (none)" and any -dev HRX0 dies with "invalid device: HRX0". Neighbours and the -# arm both need it, so it is exported here rather than passed per-process. -export IREE_HAL_AMDGPU_LIBHSA_PATH=${IREE_HAL_AMDGPU_LIBHSA_PATH:-/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1} -export LD_LIBRARY_PATH="$BIN" -source "$(dirname "$0")/315-regime.sh" -G=$(ls /sys/class/drm/card*/device/mem_info_gtt_used | head -1) -gtt() { echo $(( $(cat $G) / 1073741824 )); } -O=$HOME/wt/rt-det/$TAG-$(basename $M .gguf) -rm -rf "$O"; mkdir -p "$O" - -# Refuse to hold the box lock for a run that cannot work. -regime_preflight || exit 3 - -echo "capture start $(date -Is) bin=$BIN fp=$FP gtt_before=$(gtt)G" -"$BIN/llama-server" --version | sed 's/^/ /' -for P in $(pgrep -f llama-server 2>/dev/null); do - echo " foreign pid=$P $(tr '\0' ' ' < /proc/$P/cmdline | cut -c1-90)" -done - -# Serialize with other sessions, but do not block the campaign forever: after LOCK_WAIT seconds the -# arm proceeds concurrently and records that it did, so the condition is visible in the result. -if [ -z "${NOLOCK:-}" ]; then - exec 9>"$HOME/.cache/lax-decode/box.lock" - if flock -w "${LOCK_WAIT:-900}" 9; then - echo "capture: box lock acquired at $(date -Is)" - else - echo "capture: box lock busy after ${LOCK_WAIT:-900}s — proceeding CONCURRENTLY (UNLOCKED)" - fi -fi -regime_neighbours 1 6 "$NCTX" || { echo "capture: could not establish the regime composition — arm is invalid"; trap - EXIT; regime_teardown; exit 4; } -trap regime_teardown EXIT - -PORT=$((19300 + RANDOM % 400)) -REGIME_ARM_PORT=$PORT -SETENV=(--setenv=IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1) -[ "$FP" = "1" ] && SETENV+=(--setenv=GGML_HRX_WRITEBACK_FINGERPRINT=1) -# UB / EXTRA_ENV let a run reproduce the recorded *high-rate* configurations (the issue measured -# 11/12 faults with -ub 512 vs 0/12 with -ub 128), which makes mechanism discrimination cheap: -# at ~5%/arm a differential test would need dozens of arms to see a handful of faults. -ARM_UB=(); [ -n "${UB:-}" ] && ARM_UB=(-ub "$UB") -if [ -n "${EXTRA_ENV:-}" ]; then - IFS=',' read -r -a _extra <<< "$EXTRA_ENV" - for kv in "${_extra[@]}"; do SETENV+=(--setenv="$kv"); done -fi -echo "arm config: ub=${UB:-default} extra_env=${EXTRA_ENV:-none}" -systemd-run --user --scope -q -p MemoryMax=60G --unit=${TAG}-arm "${SETENV[@]}" \ - "$BIN/llama-server" -m "$M" -dev HRX0 -ngl 99 -fa on --host 127.0.0.1 --port $PORT -c 8192 -np 1 --no-webui \ - "${ARM_UB[@]}" \ - > "$O/server.log" 2>&1 & -UP=0; for t in $(seq 1 300); do curl -sf localhost:$PORT/health >/dev/null 2>&1 && { UP=1; break; }; sleep 1; done -G0=$(gtt); echo "gtt_start ${G0}GiB fp=$FP" > "$O/run.log" -echo "arm up=$UP gtt_start=${G0}G port=$PORT" - -# Watchdog, as the recorded harness (rt-det.sh) runs it: sample GTT/temperature and abort the arm if -# it leaves the pre-registered regime (>=24 GiB growth from the arm start, or >=93 C). Without this the -# arm would be classified by its gtt_start alone while actually running hotter. -K=$(for h in /sys/class/hwmon/*; do [ "$(cat "$h/name" 2>/dev/null)" = k10temp ] && echo "$h"; done) -G0B=$(cat "$G"); : > "$O/gtt.log" -( while pgrep -f "port $PORT" >/dev/null 2>&1 || systemctl --user is-active ${TAG}-arm >/dev/null 2>&1; do - g=$(cat "$G"); echo "$g" >> "$O/gtt.log" - if [ -n "$K" ]; then t=$(( $(cat "$K/temp1_input" 2>/dev/null || echo 0) / 1000 )); else t=0; fi - if [ $(( (g - G0B) / 1073741824 )) -ge 24 ] || [ "$t" -ge 93 ]; then - echo "WATCHDOG gtt=$g t=$t" >> "$O/run.log" - systemctl --user stop ${TAG}-arm 2>/dev/null - for P in $(pgrep -f "port $PORT" 2>/dev/null); do kill -TERM $P 2>/dev/null; done - break - fi - sleep 1 - done ) & -WD=$! - -if [ "$UP" = "1" ]; then - timeout "${ARM_TIMEOUT:-1200}" python3 "$DET" "$PORT" "$O" 50 > "$O/detector.log" 2>&1 - echo " detector exit=$? (124 = arm exceeded ${ARM_TIMEOUT:-1200}s and was cut off)" - tail -1 "$O/detector.log" | sed 's/^/ /' -fi -kill "$WD" 2>/dev/null -echo "gtt_max $(sort -n "$O/gtt.log" 2>/dev/null | tail -1) gtt_start $G0B" >> "$O/run.log" -echo " watchdog: $(grep -o 'WATCHDOG.*' "$O/run.log" 2>/dev/null | head -1 || echo 'no abort') | gtt samples: $(wc -l < "$O/gtt.log" 2>/dev/null || echo 0)" - -echo "--- diagnostic records ---" -grep -c "\[hrx-wb-fp\]" "$O/server.log" 2>/dev/null | xargs echo " hrx-wb-fp lines:" -grep -m3 "\[hrx-wb-fp\]" "$O/server.log" 2>/dev/null | cut -c1-240 | sed 's/^/ /' -echo "--- faces ---" -grep -c "HRX returned NaN logits" "$O/server.log" 2>/dev/null | xargs echo " faceA nan:" -grep -iE "HSA_STATUS_ERROR|Queue error|wait for HRX graph replay commands failed" "$O/server.log" 2>/dev/null \ - | grep -vcE "hrx_allocator_allocate_buffer|allocate HRX host staging buffer" | xargs echo " faceA hsa:" -grep -cE "hrx_allocator_allocate_buffer.*failed" "$O/server.log" 2>/dev/null | xargs echo " startup alloc failures:" -if [ -f "$O/detector.json" ]; then - python3 - "$O/detector.json" <<'PY' -import json, sys -d = json.load(open(sys.argv[1])) -print(f" faceB utf8={d['utf8_failures']} parse={d['parse_failures']} http={d['http_failures']} " - f"transport={d['transport_failures']} req={d['requests']}/66 truncated={d['truncated']}") -print(f" first_failure={d['first_failure']!r}") -PY -fi -echo "CAPTURE_DONE tag=$TAG out=$O" diff --git a/dev/315/315-campaign/315-janitor.sh b/dev/315/315-campaign/315-janitor.sh deleted file mode 100755 index a066d9572..000000000 --- a/dev/315/315-campaign/315-janitor.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 janitor: stop arm scopes whose arm is ALREADY SCORED, so a slow teardown cannot inflate the next arm's -# baseline. Seen on 2026-10-10: `315row-arm-1l.scope` stayed resident 15 minutes after its arm was scored OK, and -# card GTT read 113.5 GiB instead of ~100 (evidence/315-queue-profile.md, appendix). -# -# Safety rule, and the reason this cannot disturb a measurement: a scope is stopped only when the run's own -# console already contains that arm's verdict line (`pair arm : `), i.e. only after the detector -# has finished scoring it. An arm still being scored has no verdict line and is left alone, however long it runs. -# It stops declared work only - the scopes belong to this run's own tag - and never touches neighbours. -# -# Usage: 315-janitor.sh [minutes] -set -uo pipefail -TAG=${1:?usage: 315-janitor.sh [minutes]} -MINUTES=${2:-240} -CONSOLE=$HOME/wt/$TAG-console.log -DEADLINE=$((SECONDS + MINUTES * 60)) - -echo "janitor: watching $TAG arm scopes for ${MINUTES} min $(date -Is)" -while [ $SECONDS -lt $DEADLINE ]; do - # nothing to clean once the runner is gone, unless scopes are still up - then clean once more and exit - runner=0 - pgrep -f "315-(attrib|paired)\.sh .* $TAG" >/dev/null 2>&1 && runner=1 - for u in $(systemctl --user list-units --state=running "$TAG-arm-*" --no-legend 2>/dev/null | awk '{print $1}'); do - n=${u#$TAG-arm-}; n=${n%.scope} - pair=${n%?}; cell=${n: -1} - case "$pair$cell" in - [0-9]*[cl]) ;; - *) continue ;; - esac - if [ -f "$CONSOLE" ] && grep -q "pair $pair arm $cell:" "$CONSOLE"; then - echo "janitor: $u scored already, stopping it $(date +%T); gtt=$(( $(cat /sys/class/drm/card*/device/mem_info_gtt_used) / 1073741824 ))G" - systemctl --user stop "$u" 2>/dev/null - fi - done - [ "$runner" = 0 ] && break - sleep 60 -done -echo "janitor: done $(date -Is); gtt=$(( $(cat /sys/class/drm/card*/device/mem_info_gtt_used) / 1073741824 ))G" diff --git a/dev/315/315-campaign/315-mode.sh b/dev/315/315-campaign/315-mode.sh deleted file mode 100755 index 01c54fe13..000000000 --- a/dev/315/315-campaign/315-mode.sh +++ /dev/null @@ -1,98 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: the execution-mode toggle. Diagnostic, never acceptance-eligible. -# -# Why this is a lever and not a repeat. The knob inventory recorded `HRX_COMMAND_BUFFER_MODE` and dismissed it -# for the *wait-barrier strategy* question ("selects the command-buffer builder, not the queue execution mode") -# - correct, and never run as a fault-rate toggle. It is more than a builder switch: -# -# runtime/src/iree/hal/drivers/amdgpu/logical_device.c:159 default is AQL -# runtime/src/iree/hal/drivers/amdgpu/logical_device.c:244-253 the string param takes aql | pm4 | auto and -# REJECTS anything else (no silent fallback) -# runtime/src/iree/hal/drivers/amdgpu/logical_device.c:2018-2027 PM4 and AUTO "promise reusable dispatch -# command buffers, including dynamic binding-table -# fixups" and provision queue-control storage for it -# runtime/src/iree/hal/drivers/amdgpu/registration/driver_module.c:61 the flag default is also "aql" -# -# So `pm4` selects a different execution path on two axes at once: packet/barrier semantics (PM4 IB rather than -# native AQL, on a device whose capability set has no AQL_BARRIER_VALUE) and REUSABLE command buffers whose -# binding tables are fixed up at submission instead of recorded once. The routed-FFN path binds two *generated -# transient* tables (expert table, partition table) whose producer dispatches sit in the same plan -# (dispatch-mul-mat-id-common.h:342-368). A binding fixup that resolves those to the wrong generation is exactly -# a per-token failure, and it exists only on the PM4/AUTO path. -# -# The instrument is deliberately absent: the tensor fingerprint adds 16 device-to-host reads per graph and the -# copy probe prints from the submit path, either of which could perturb the timing under test (same argument as -# 315-graphsync.sh). Compare the counts against the no-knob arms already on disk from the same harness. -# -# Readout, pre-committed: -# fault rate falls on the pm4 cell -> the default AQL path's per-execution binding/barrier behaviour is -# the carrier; the next step is to diff the two paths' dispatches -# fault rate rises or is unchanged -> the mode is not the carrier, and the reusable-command-buffer -# hypothesis (candidate 1 in evidence/315-moe-routing-routed-ffn-static.md) -# is falsified too -# the knob is not observable in the logs -> report that, do not read the counts as evidence either way -# -# Usage: 315-mode.sh [pm4|aql|auto] [pairs] [tag] -set -uo pipefail -MODE=${1:-pm4}; PAIRS=${2:-6}; TAG=${3:-315mode} -IREE_HAL_AMDGPU_LIBHSA_PATH=${IREE_HAL_AMDGPU_LIBHSA_PATH:-/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1} -export IREE_HAL_AMDGPU_LIBHSA_PATH -D=/home/bcloud/wt -CONSOLE=$D/$TAG-console.log - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${MODE_LOCK_WAIT:-10800}" 9; then - echo "mode: box lock held $(date -Is)" -else - echo "mode: lock unobtainable after ${MODE_LOCK_WAIT:-10800}s - aborting rather than running concurrently" - exit 3 -fi - -# Preflight: is the knob observable at all? A knob that does not reach the driver would make the arm counts -# meaningless, so detect that first and say so rather than reading the counts. -echo "=== mode preflight $(date -Is) ===" -OBS="unobservable" -for M in aql "$MODE"; do - L=$D/modepre-$M.log - HRX_GPU_DEBUG=1 HRX_COMMAND_BUFFER_MODE=$M timeout 240 "$D/315instr3-build/bin/llama-server" \ - -m /home/bcloud/models/Qwen3-0.6B-Q4_K_M.gguf -dev HRX0 -ngl 99 -c 512 -np 1 --no-webui \ - --host 127.0.0.1 --port $((20500 + RANDOM % 200)) > "$L" 2>&1 & - sp=$! - sleep 90; kill $sp 2>/dev/null; wait $sp 2>/dev/null - echo " $M: log $L $(wc -l < "$L") lines, $(grep -ciE 'command.?buffer|pm4|aql' "$L") mode-related lines" -done -if [ -f "$D/modepre-aql.log" ] && [ -f "$D/modepre-$MODE.log" ] && \ - ! cmp -s "$D/modepre-aql.log" "$D/modepre-$MODE.log"; then - OBS="observable (logs differ)" -fi -echo " knob: $OBS" -echo " if unobservable, the arm counts below are NOT evidence about the mode - recorded and reported as such" - -cd "$D" || exit 1 -echo "=== mode arms mode=$MODE tag=$TAG pairs=$PAIRS $(date -Is) ===" -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ -CONTROL_EVERY=999 NOSTOP=1 MINCLEAN=999 NOLOCK=1 ARM_TIMEOUT=${ARM_TIMEOUT:-1800} \ -EXTRA_ENV="HRX_COMMAND_BUFFER_MODE=$MODE" \ - bash "$D/315-paired.sh" "$PAIRS" "${NB:-2}" "${NCTX:-2048}" \ - "$D/315instr3-build/bin" "$D/315instr3-build/bin" "$TAG" - -echo "=== mode readout $(date -Is) knob=$OBS ===" -grep -hE "^ pair" "$CONSOLE" || true -echo "in-regime clean (OK): $(grep -c 'arm l: OK' "$CONSOLE" || true)" -echo "in-regime fault: $(grep -c 'arm l: \(FAULT\|INVALID\)' "$CONSOLE" || true)" -echo "out-of-regime (OOR): $(grep -c 'arm l: OOR' "$CONSOLE" || true)" -echo "compare: the no-knob in-regime record for this harness is on disk (315chain/315gsync/315row arm dirs)" diff --git a/dev/315/315-campaign/315-paired.sh b/dev/315/315-campaign/315-paired.sh deleted file mode 100755 index 88fd91711..000000000 --- a/dev/315/315-campaign/315-paired.sh +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 paired in-regime campaign driver — the pre-registered producer. -# -# Emits one line per arm in the grammar the analysis scripts parse: -# pair arm : nan= req= ret3= parse= gtt_start=G utf8= alloc= -# (utf8/alloc are appended after gtt_start so the existing regex, which is not end-anchored, still -# matches; they carry the Face-B count and the startup-allocation failures that are classified -# invalid rather than faulted.) -# -# Classification, fixed in the pre-registration (dev/315/315-preregistration.md): -# invalid startup allocation failure, server died before the request phase, or no gtt.log -# out-of-regime gtt_start < 95 GiB (reported, not counted) -# fault nan>0 or hsa>0 or parse>0 or utf8>0 or idn>1 or verdict==TRUNCATED or req<66 -# (idn>1 = the 8 identical greedy requests disagreed; see the 2026-10-10 freeze note) -# clean otherwise -# -# Usage: 315-paired.sh -set -uo pipefail -PAIRS=${1:-1}; NB=${2:-2}; NCTX=${3:-2048}; CBIN=${4:?control bin dir}; FBIN=${5:?fix bin dir}; TAG=${6:-315p} -DET=$(dirname "$0")/315-detector.py -M=/home/bcloud/models/GLM-4.7-Flash-Q4_K_M.gguf -OUTDIR=$HOME/wt/rt-det -CONSOLE=$HOME/wt/$TAG-console.log -export IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 -G=$(ls /sys/class/drm/card*/device/mem_info_gtt_used | head -1) -gtt() { echo $(( $(cat $G) / 1073741824 )); } -mine() { for P in $(pgrep -f "llama-server" 2>/dev/null); do C=$(tr '\0' ' ' < /proc/$P/cmdline 2>/dev/null); case "$C" in *"port 2021"*) echo $P;; esac; done; } - -say() { echo "$@" | tee -a "$CONSOLE"; } - -say "=== session $(date -Is) tag=$TAG pairs=$PAIRS neighbours=$NB ctx=$NCTX gtt_before=$(gtt)G ===" -say "model_sha256 $(sha256sum "$M" | cut -d' ' -f1)" -say "detector_sha256 $(sha256sum "$DET" | cut -d' ' -f1)" -say "control_bin $CBIN -> $("$CBIN/llama-server" --version 2>&1 | head -1)" -say "fix_bin $FBIN -> $("$FBIN/llama-server" --version 2>&1 | head -1)" -say "resident_inventory (foreign servers, not ours):" -for P in $(pgrep -f llama-server 2>/dev/null); do - say " pid=$P $(tr '\0' ' ' < /proc/$P/cmdline | cut -c1-100)" -done - -# Regime composition is shared with the other #315 scripts: the helper asserts HRX0 is visible, -# starts neighbours with the HRX env (a missing --setenv is what killed the third smoke arm), bounds -# each start, and aborts instead of grinding through multi-minute health waits while holding the lock. -export IREE_HAL_AMDGPU_LIBHSA_PATH=${IREE_HAL_AMDGPU_LIBHSA_PATH:-/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1} -BIN="$CBIN" -source "$(dirname "$0")/315-regime.sh" -regime_preflight || { say "ABORT: HRX0 not visible to $CBIN/llama-server"; exit 3; } -if [ -z "${NOLOCK:-}" ]; then - exec 9>"$HOME/.cache/lax-decode/box.lock" - if flock -w "${LOCK_WAIT:-900}" 9; then - say "box lock acquired $(date -Is)" - else - say "box lock busy after ${LOCK_WAIT:-900}s — proceeding CONCURRENTLY (UNLOCKED)" - fi -fi -regime_neighbours 1 6 "$NCTX" || { say "regime composition failed — session aborted"; trap - EXIT; regime_teardown; exit 4; } -trap regime_teardown EXIT -say "neighbours up gtt=$(gtt)G" - -arm() { # arm - local pair=$1 cell=$2 bin=$3 port=$((19300 + RANDOM % 400)) - local tag="$TAG-p$pair$cell" - local O="$OUTDIR/$tag-$(basename $M .gguf)" - rm -rf "$O"; mkdir -p "$O" - local -a TRACE_ENV=() - [ -n "${TRACE:-}" ] && TRACE_ENV=(--setenv=GGML_HRX_BUFFER_TRACE=1) - # EXTRA_ENV="K=V K2=V2" adds environment to each arm (diagnostic runs, e.g. a probe). - if [ -n "${EXTRA_ENV:-}" ]; then for kv in $EXTRA_ENV; do TRACE_ENV+=(--setenv="$kv"); done; fi - systemd-run --user --scope -q -p MemoryMax=60G --unit=$TAG-arm-$pair$cell \ - --setenv=IREE_HAL_AMDGPU_LIBHSA_PATH="$IREE_HAL_AMDGPU_LIBHSA_PATH" \ - --setenv=LD_LIBRARY_PATH="$bin" "${TRACE_ENV[@]}" \ - "$bin/llama-server" -m "$M" -dev HRX0 -ngl 99 -fa on --host 127.0.0.1 --port $port -c 8192 -np 1 --no-webui \ - > "$O/server.log" 2>&1 9>&- & - local up=0 - for t in $(seq 1 300); do curl -sf -m 5 localhost:$port/health >/dev/null 2>&1 && { up=1; break; }; sleep 1; done # -m 5: a blocked accept must not hang the runner - local g0; g0=$(gtt); echo "gtt_start ${g0}GiB mode $(cat /sys/class/ec_su_axb35/apu/power_mode 2>/dev/null)" > "$O/run.log" - - local alloc nan hsa req=0 parse=0 utf8=0 verdict hv52=0 hv56=0 idn=1 - alloc=$(grep -cE "hrx_allocator_allocate_buffer.*failed|allocate HRX host staging buffer for value .* failed" "$O/server.log" 2>/dev/null || true) - if [ "$up" != "1" ] || [ "${alloc:-0}" -gt 0 ]; then - systemctl --user stop $TAG-arm-$pair$cell 2>/dev/null - for P in $(pgrep -f "port $port" 2>/dev/null); do kill -TERM $P 2>/dev/null; done - printf ' pair %d arm %s: INVALID nan=0 req=0 ret3=0 parse=0 gtt_start=%dG utf8=0 alloc=%d hv52=0 hv56=0 idn=0\n' \ - "$pair" "$cell" "$g0" "${alloc:-0}" | tee -a "$CONSOLE" - return - fi - REGIME_ARM_PORT=$port - timeout "${ARM_TIMEOUT:-1200}" python3 "$DET" "$port" "$O" 50 > "$O/detector.log" 2>&1 - req=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['requests'])" 2>/dev/null || echo 0) - # Freeze note 2026-10-10: distinct outputs among the 8 identical greedy requests. >1 is a fault: finite - # corruption that still parses would otherwise be invisible. Absent/None -> 1, so nothing changes for an - # arm whose detector did not get that far. - idn=$(python3 -c "import json;print(json.load(open('$O/detector.json')).get('identical_distinct') or 1)" 2>/dev/null || echo 1) - parse=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['parse_failures'])" 2>/dev/null || echo 0) - utf8=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['utf8_failures'])" 2>/dev/null || echo 0) - nan=$(grep -c "HRX returned NaN logits" "$O/server.log" 2>/dev/null || true) - # Face A memory fault signatures, excluding the allocation-failure lines that are invalid above. - hsa=$(grep -iE "HSA_STATUS_ERROR|Queue error|wait for HRX graph replay commands failed" "$O/server.log" 2>/dev/null \ - | grep -vcE "hrx_allocator_allocate_buffer|allocate HRX host staging buffer" || true) - http=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['http_failures'])" 2>/dev/null || echo 0) - tr=$(python3 -c "import json;print(json.load(open('$O/detector.json'))['transport_failures'])" 2>/dev/null || echo 0) - # Face B is first class in the pre-registration (see differential): count it from the detectors and - # the server log, so an arm that still completes its requests cannot be recorded as clean. - s500=$(grep -c '"code":500' "$O/server.log" 2>/dev/null || true); s500=${s500:-0} - unr=$(grep -cE "unparsed Content-only output|does not match the expected" "$O/server.log" 2>/dev/null || true); unr=${unr:-0} - # Allocation evidence, from the trace the arm just ran: 0x52 = host-visible without HOST_COHERENT, - # 0x56 = host-visible coherent. Recorded per arm so the fix being live is a measurement, not a build claim. - hv52=$(grep -c "memory_type=0x52 size=619520" "$O/server.log" 2>/dev/null || true); hv52=${hv52:-0} - hv56=$(grep -c "memory_type=0x56 size=619520" "$O/server.log" 2>/dev/null || true); hv56=${hv56:-0} - local g1; g1=$(gtt) - # The gate is two-sided (dev/315/315-preregistration.md 5): above the upper bound the detector fails to - # allocate and the arm is invalid, so it must never be recorded as clean. The analyser already treats it - # as out-of-regime; recording it as clean here was a gap. - if [ "${g0:-0}" -lt 95 ] || [ "${g0:-0}" -gt 110 ]; then verdict=OOR - elif [ "${nan:-0}" -gt 0 ] || [ "${hsa:-0}" -gt 0 ] || [ "${parse:-0}" -gt 0 ] || [ "${utf8:-0}" -gt 0 ] \ - || [ "${http:-0}" -gt 0 ] || [ "${tr:-0}" -gt 0 ] || [ "${s500:-0}" -gt 0 ] || [ "${unr:-0}" -gt 0 ] \ - || [ "${idn:-1}" -gt 1 ] || [ "${req:-0}" -lt 66 ]; then verdict=FAULT - else verdict=OK; fi - # engine#315: the Face-B counters are already computed above and already drive `verdict`, but they were not - # PRINTED, so the acceptance summariser could not attribute Face B from this line at all - a real Face-B - # fault (nan=0, http/s500/unparsed set) came out as "unattributed". Emitted now, AFTER gtt_start so the - # existing `(.*)$` tail capture in 315-campaign.sh picks them up without a format change there. - printf ' pair %d arm %s: %s nan=%d req=%d ret3=0 parse=%d gtt_start=%dG utf8=%d alloc=%d hv52=%d hv56=%d idn=%d http=%d s500=%d unparsed=%d hsa=%d transport=%d (gtt_now=%dG)\n' \ - "$pair" "$cell" "$verdict" "${nan:-0}" "${req:-0}" "${parse:-0}" "$g0" "${utf8:-0}" "${alloc:-0}" "${hv52:-0}" "${hv56:-0}" "${idn:-1}" "${http:-0}" "${s500:-0}" "${unr:-0}" "${hsa:-0}" "${tr:-0}" "$g1" \ - | tee -a "$CONSOLE" - systemctl --user stop $TAG-arm-$pair$cell 2>/dev/null - for _t in 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0; do - [ -z "$(pgrep -f "port $port" | head -1)" ] && break - sleep 1 - done - for P in $(pgrep -f "port $port" 2>/dev/null); do kill -TERM $P 2>/dev/null; done -} - -# Pre-registered stopping rule (dev/315/315-preregistration.md §6): run pairs until the fix cell has -# MINCLEAN (default 60) clean in-regime arms, stop immediately if any fix-cell arm faults (acceptance -# requires zero), and never exceed PAIRS pairs. The count is read back from the console, i.e. from the -# same pre-registered classification the analyser uses. -clean_fix() { grep -c 'arm l: OK' "$CONSOLE" 2>/dev/null || true; } -fault_fix() { grep -c 'arm l: \(FAULT\|INVALID\)' "$CONSOLE" 2>/dev/null || true; } -# Control cadence. The acceptance criterion (dev/315/315-preregistration.md §7) is on the FIX cell, and a -# control arm costs the same box time as a fix arm, so controls run every CONTROL_EVERY-th pair (default -# 3): a live positive control spread across the whole session instead of 1:1 pairing, which would double -# the run and leave no budget for the regression cell. §7 does not constrain the ratio; the analyser -# aggregates by cell. -for p in $(seq 1 $PAIRS); do - if [ $(( (p - 1) % ${CONTROL_EVERY:-3} )) -eq 0 ]; then arm "$p" c "$CBIN"; fi - arm "$p" l "$FBIN" - cf=$(clean_fix); ff=$(fault_fix) - say "after pair $p: fix-cell clean=$cf fault-or-invalid=$ff (target >=${MINCLEAN:-60} clean, cap $PAIRS pairs)" - if [ "$ff" -gt 0 ] && [ -z "${NOSTOP:-}" ]; then - say "STOP: a fix-cell arm faulted or was invalid — acceptance requires zero faults" - break - fi - if [ "$ff" -gt 0 ]; then - say "NOTE: fix-cell fault(s)=$ff recorded and the run continues because NOSTOP=1 — this is a DIAGNOSTIC run, not an acceptance run, and its counts must not be read as one" - fi - [ "$cf" -ge "${MINCLEAN:-60}" ] && { say "STOP: reached $cf clean fix-cell arms"; break; } -done - -echo "$(clean_fix) clean fix-cell arms, $(fault_fix) faulted-or-invalid in the fix cell, out of $(grep -c '^ pair' "$CONSOLE" 2>/dev/null) arms run" | tee -a "$CONSOLE" diff --git a/dev/315/315-campaign/315-regime.sh b/dev/315/315-campaign/315-regime.sh deleted file mode 100755 index fd9b28b28..000000000 --- a/dev/315/315-campaign/315-regime.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 shared regime helper — adaptive pressure composition with hard preflight. -# -# Two failures this helper exists to prevent, both observed on the box: -# -# 1. A neighbour started without IREE_HAL_AMDGPU_LIBHSA_PATH dies immediately with -# "error while handling argument \"-dev\": invalid device: HRX0" — because llama-server then -# reports "Available devices: (none)". Neighbours are therefore started with the HRX env -# explicitly, and regime_preflight refuses to proceed if HRX0 is not visible at all. -# 2. A fixed neighbour count misses the regime band (3 default neighbours -> 115.8 GiB, detector -# dies allocating; 2 at -c 2048 -> 80.3 GiB, out of regime), because the box's base GTT moves -# with other sessions. Neighbours are therefore started one at a time against live GTT. -# -# Callers must set BIN, M, TAG and IREE_HAL_AMDGPU_LIBHSA_PATH, then: -# regime_preflight || exit 1 -# regime_neighbours 1 6 2048 || exit 1 # non-zero = no usable composition -# trap regime_teardown EXIT - -regime_gtt() { echo $(( $(cat "$REGIME_G_FILE") / 1073741824 )); } - -regime_preflight() { - REGIME_G_FILE=${REGIME_G_FILE:-$(ls /sys/class/drm/card*/device/mem_info_gtt_used | head -1)} - : "${REGIME_G_FILE:?no GTT sysfs node found}" - if [ -z "${IREE_HAL_AMDGPU_LIBHSA_PATH:-}" ]; then - echo "regime: ABORT — IREE_HAL_AMDGPU_LIBHSA_PATH is unset; llama-server will see no devices" - return 1 - fi - local devs - devs=$(IREE_HAL_AMDGPU_LIBHSA_PATH="$IREE_HAL_AMDGPU_LIBHSA_PATH" LD_LIBRARY_PATH="$BIN" \ - "$BIN/llama-server" --list-devices 2>&1 | head -8) - case "$devs" in - *HRX0*) - echo "regime: preflight ok — $(printf '%s' "$devs" | grep -m1 HRX0 | sed 's/^ *//' | cut -c1-72)" - ;; - *) - echo "regime: ABORT — HRX0 not visible to $BIN/llama-server; --list-devices said:" - printf '%s\n' "$devs" | tail -3 | sed 's/^/ /' - return 1 - ;; - esac -} - -# Start neighbours until the live GTT reaches LO (default 80 GiB), never past HI (default 92 GiB). -# Each start is bounded: if the neighbour does not serve /health in REGIME_UP_TIMEOUT seconds, or -# the process dies, it is recorded as a failed start, and two consecutive failures abort the -# composition instead of grinding through further multi-minute waits. -regime_neighbours() { - local min_nb=${1:-1} max_nb=${2:-6} ctx=${3:-2048} - local up_timeout=${REGIME_UP_TIMEOUT:-150} - REGIME_NB=0; local failures=0 - REGIME_LO=${REGIME_LO:-80}; REGIME_HI=${REGIME_HI:-92} - # Callers must not lower these. The pre-registered arm gate is gtt_start >= 95 GiB and the detector - # contributes ~19-30 GiB, so a composition that stops at a lower LO lands out of regime: the - # 2026-10-10 315gsync and 315row sessions passed LO=70/HI=85 and produced 87-93 GiB arms (4 of 6 - # below the gate), while 315chain, which inherits these defaults, produced 98 GiB arms. - while [ "$REGIME_NB" -lt "$max_nb" ]; do - local now; now=$(regime_gtt) - if [ "$REGIME_NB" -ge "$min_nb" ] && [ "$now" -ge "$REGIME_LO" ]; then break; fi - if [ "$now" -ge "$REGIME_HI" ]; then - echo "regime: gtt=$now GiB at/above HI=$REGIME_HI, no further neighbours" - break - fi - REGIME_NB=$((REGIME_NB + 1)) - local port=$((20250 + REGIME_NB)) - systemd-run --user --scope -q -p MemoryMax=40G --unit=${TAG}-neigh-$REGIME_NB \ - --setenv=IREE_HAL_AMDGPU_LIBHSA_PATH="$IREE_HAL_AMDGPU_LIBHSA_PATH" \ - --setenv=LD_LIBRARY_PATH="$BIN" \ - "$BIN/llama-server" -m "$M" -dev HRX0 -ngl 99 -fa on -c "$ctx" \ - --host 127.0.0.1 --port "$port" \ - > "$HOME/wt/$TAG-neigh-$REGIME_NB.log" 2>&1 9>&- & - local up=0 t=0 - while [ "$t" -lt "$up_timeout" ]; do - curl -sf "localhost:$port/health" >/dev/null 2>&1 && { up=1; break; } - if ! systemctl --user is-active "${TAG}-neigh-$REGIME_NB" >/dev/null 2>&1 \ - && ! pgrep -f "port $port" >/dev/null 2>&1; then - break # process is gone: no point waiting out the timeout - fi - sleep 2; t=$((t + 2)) - done - if [ "$up" != "1" ]; then - failures=$((failures + 1)) - echo "regime: neighbour $REGIME_NB FAILED to come up within ${up_timeout}s — last log lines:" - tail -3 "$HOME/wt/$TAG-neigh-$REGIME_NB.log" 2>/dev/null | sed 's/^/ /' - REGIME_NB=$((REGIME_NB - 1)) - systemctl --user stop "${TAG}-neigh-$((REGIME_NB + 1))" 2>/dev/null - [ "$failures" -ge 2 ] && { echo "regime: ABORT after 2 failed neighbour starts"; return 1; } - continue - fi - sleep 4 - echo "regime: neighbour $REGIME_NB up, gtt=$(regime_gtt)G" - done - if [ "$REGIME_NB" -lt 1 ]; then echo "regime: ABORT — no neighbour could be started"; return 1; fi - echo "regime: composition neighbours=$REGIME_NB ctx=$ctx gtt_before_arm=$(regime_gtt)G (LO=$REGIME_LO HI=$REGIME_HI, projects to >=95 after the detector loads)" - return 0 -} - -regime_teardown() { - local i - for i in $(seq 1 "${REGIME_NB:-0}"); do - systemctl --user stop "${TAG}-neigh-$i" 2>/dev/null - systemctl --user kill -s TERM "${TAG}-neigh-$i" 2>/dev/null - done - sleep 3 - for P in $(pgrep -f "llama-server" 2>/dev/null); do - local C; C=$(tr '\0' ' ' < "/proc/$P/cmdline" 2>/dev/null) - case "$C" in *"port 2025"*) kill -TERM "$P" 2>/dev/null ;; esac - if [ -n "${REGIME_ARM_PORT:-}" ]; then - case "$C" in *"port $REGIME_ARM_PORT"*) kill -TERM "$P" 2>/dev/null ;; esac - fi - done - sleep 5 - echo "regime: teardown done servers_left=$(pgrep -c llama-server || echo 0) gtt_after=$(regime_gtt)G" -} diff --git a/dev/315/315-campaign/315-regression.sh b/dev/315/315-campaign/315-regression.sh deleted file mode 100755 index 3f2bc50b2..000000000 --- a/dev/315/315-campaign/315-regression.sh +++ /dev/null @@ -1,152 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 regression cell (task-6). -# -# Reproduces the recorded baseline from dev/315/315-preregistration.md 9b, from the SAME source revision as -# the fix cell (~/wt/llama-315fix), because test-backend-ops is absent from the study's builds (their -# recipe sets -DLLAMA_BUILD_TESTS=OFF). Two checks: -# -# 1. `test-backend-ops -o MUL_MAT_ID -b HRX0`, twice, using the study's own wrapper and line format. -# Recorded baseline: 108/108 passed, fail=0, supported=110, rc=0, stable across repeats. -# 2. dense-model control at ~512 tokens (zaya1-8b, the dense model the study pairs with GLM in -# ~/wt/hrxsys-correct.sh), asserting valid UTF-8 output and no fault signature on either face. -# The MoE path is the faulting one, so this is the "is the fix inert on ordinary inference" check. -# -# Takes the box lock, so it cannot collide with the acceptance campaign. -set -uo pipefail - -SRC=${SRC:-$HOME/wt/llama-315fix} -B=${B:-$HOME/wt/315fix-tests-build} -FBIN=${FBIN:-$HOME/wt/315fix-build/bin} -M=$HOME/models -O=$HOME/wt/315reg -LOG=$HOME/wt/315fix-tests-build.log -PORT=${PORT:-19401} -# PIN and FIX_GREP are parameterised because this cell has to run twice: once against the baseline -# revision (to capture the "unchanged" side of task-6's comparison) and once against the fix. The -# earlier version hardcoded both the revision and the *coherence* candidate's code pattern, which -# was withdrawn - it would have aborted on any other fix, and could not run a baseline at all. -PIN=${PIN:-e44c9d01a4d5110cecaf45472a717f46cd1abbf9} # override for the fix revision -FIX_GREP=${FIX_GREP:-} # optional regex that must match the source; the fix's own signature -export IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 -mkdir -p "$O" - -say() { echo "[$(date +%H:%M:%S)] $*"; } - -say "=== regression cell: source $SRC at $(git -C "$SRC" rev-parse --short HEAD) (pin ${PIN:0:9}) ===" -[ "$(git -C "$SRC" rev-parse HEAD)" = "$PIN" ] || { echo "ABORT: source is not at the pin"; exit 1; } -# Revision identity is the assertion that matters: prove we are validating the intended tree. A code-pattern -# assertion is optional, and only if the caller supplies the fix's own signature, so that this cell does not -# bake in one candidate's implementation (the coherence candidate it used to hardcode was withdrawn). -if [ -n "$FIX_GREP" ]; then - grep -qE "$FIX_GREP" "$SRC/ggml/src/ggml-hrx/ggml-hrx.cpp" \ - || { echo "ABORT: FIX_GREP '$FIX_GREP' not found in the source being validated"; exit 1; } - say "source assertion: FIX_GREP matched" -else - say "source assertion: revision only (no FIX_GREP given)" -fi -say "source diff vs the baseline revision: $(git -C "$SRC" diff --stat "$PIN" HEAD | tail -1)" - -# The op-test needs the box to itself (it drives the GPU hard); take the same lock the campaign uses. -exec 9>"$HOME/.cache/lax-decode/box.lock" -flock -w "${LOCK_WAIT:-1800}" 9 || { echo "ABORT: box lock not acquired"; exit 3; } -say "box lock acquired" - -say "=== build with tests ON (the only difference from the campaign cell) ===" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HOME/wt/pin-fork-hrx/third_party/hrx-system" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=ON -DBUILD_TESTING=ON \ - > "$LOG" 2>&1 || { echo "ABORT: cmake failed, see $LOG"; tail -20 "$LOG"; exit 1; } -cmake --build "$B" -j 12 --target test-backend-ops llama-server >> "$LOG" 2>&1 \ - || { echo "ABORT: build failed, see $LOG"; tail -30 "$LOG"; exit 1; } -say "op-test binary: $("$B/bin/test-backend-ops" --help 2>&1 | head -1)" - -say "=== check 1: test-backend-ops -o MUL_MAT_ID -b HRX0, two reps (study wrapper, study line format) ===" -TBO_OK=0 -for rep in 1 2; do - timeout 1800 "$B/bin/test-backend-ops" -o MUL_MAT_ID -b HRX0 > "$O/MUL_MAT_ID.$rep.log" 2>&1 - rc=$? - passed=$(grep -E "tests passed" "$O/MUL_MAT_ID.$rep.log" | head -1) - fail=$(grep -c FAIL "$O/MUL_MAT_ID.$rep.log") - ok=$(grep -c OK "$O/MUL_MAT_ID.$rep.log") - echo "tbo MUL_MAT_ID rep$rep rc=$rc $passed fail=$fail supported=$ok" | tee -a "$O/regression.txt" - if [ "$rc" = "0" ] && [ "$fail" = "0" ] && echo "$passed" | grep -q "108/108"; then - TBO_OK=$((TBO_OK + 1)) - fi -done -say "check 1: $TBO_OK/2 reps matched the recorded baseline (108/108, fail=0)" -[ "$TBO_OK" = "2" ] || echo " NOTE: does not match the recorded baseline - regression NOT cleared" - -say "=== check 2: dense-model control at ~512 tokens (zaya1-8b) ===" -DENSE=NOTRUN -if [ -x "$FBIN/llama-server" ] && [ -f "$M/zaya1-8b-Q4_K_M.gguf" ]; then - "$FBIN/llama-server" -m "$M/zaya1-8b-Q4_K_M.gguf" --host 127.0.0.1 --port "$PORT" \ - -ngl 99 -c 4096 --no-warmup --no-webui -t 8 > "$O/dense-server.log" 2>&1 & - SPID=$! - up=0 - for _ in $(seq 1 90); do - sleep 2 - curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 && { up=1; break; } - kill -0 $SPID 2>/dev/null || break - done - if [ "$up" = "1" ]; then - curl -s -m 600 -X POST "http://127.0.0.1:$PORT/completion" \ - -H 'Content-Type: application/json' \ - -d '{"prompt":"Write a short paragraph about the sea.","n_predict":512,"temperature":0.0}' \ - > "$O/dense.json" 2>&1 - DENSE=$(python3 - "$O/dense.json" "$O/dense-server.log" <<'PY' -import json, re, sys -body, log = sys.argv[1], sys.argv[2] -try: - d = json.load(open(body, errors="replace")) -except Exception as e: - print(f"BAD-JSON ({e})"); raise SystemExit -c = d.get("content") or "" -txt = open(log, errors="replace").read() -bad = [] -if not c: bad.append("empty content") -if "\ufffd" in c: bad.append("U+FFFD in output") -if re.search(r"HRX returned NaN logits", txt): bad.append("Face A: NaN logits") -if re.search(r"HSA_STATUS_ERROR|Queue error", txt): bad.append("Face A: HSA error") -if re.search(r'"code":500|unparsed Content-only', txt): bad.append("Face B: parse/500") -n = len(c.split()) -print(f"chars={len(c)} words~{n} utf8_ok={not any('FFFD' in b for b in bad)} " - f"{'CLEAN' if not bad else 'FAULT: ' + '; '.join(bad)}") -PY -) - echo "dense control (~512 tokens): $DENSE" | tee -a "$O/regression.txt" - else - DENSE="SERVER-DID-NOT-START" - echo "dense control: $DENSE (see $O/dense-server.log)" | tee -a "$O/regression.txt" - fi - kill -TERM $SPID 2>/dev/null; sleep 4; kill -9 $SPID 2>/dev/null -else - echo "dense control: $DENSE (binary or model missing)" | tee -a "$O/regression.txt" -fi - -say "=== summary ===" -tail -4 "$O/regression.txt" -if [ "$TBO_OK" = "2" ] && [ "$DENSE" = "NOTRUN" ]; then say "REGRESSION_DONE tbo=ok dense=not-run"; fi -if [ "$TBO_OK" = "2" ] && [ "$DENSE" != "NOTRUN" ] && ! echo "$DENSE" | grep -q FAULT; then - say "REGRESSION_DONE tbo=ok dense=clean" -else - say "REGRESSION_DONE tbo=$TBO_OK/2 dense=$DENSE" -fi diff --git a/dev/315/315-campaign/315-rowprobe.sh b/dev/315/315-campaign/315-rowprobe.sh deleted file mode 100755 index 4b3356e8d..000000000 --- a/dev/315/315-campaign/315-rowprobe.sh +++ /dev/null @@ -1,87 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 row probe: which token is the NaN, and where in the MoE chain does it enter? -# -# Section 47's invariant is that 15 of 16 faulting arms carry exactly one n_embd row of NaN (2048 floats) -# while the batch size varies four-fold. A count cannot say more than that, so the fingerprint was extended -# (in ~/wt/llama-315copy) to print `ne0=` (the innermost extent), `first=` (the index of the first NaN) and -# `row=` (first/ne0). This run reads those over the named per-token MoE tensors. -# -# What the answer decides: -# the NaN row is the *last* row, in every arm -> the token whose routing is computed last, i.e. a read -# that is not ordered after the routing computation: the fix is to order it -# the same row index throughout the chain -> a per-token state, so the carrier is the token's routing -# weight or one selected expert's activation for that token -# differing row indices across the chain -> the corruption is not per-token at all and the row -# invariant is a coincidence of the sampler's single-row read -# -# Builds into a NEW directory (315row-build): overwriting a binary an in-use session is executing is the -# mistake that has already cost one session, and 315fp-build is in use by the running chain probe. Source -# edits are safe because both running and queued sessions run prebuilt binaries. -# -# Diagnostic, not acceptance: no registered criterion changes and no arm from it is eligible. -# -# Usage: 315-rowprobe.sh [arms] [tag] -set -uo pipefail -ARMS=${1:-4}; TAG=${2:-315row} -SRC=$HOME/wt/llama-315copy -B=$HOME/wt/315row-build -HRX=$HOME/wt/pin-fork-hrx/third_party/hrx-system -LOG=$HOME/wt/315row-build.log - -exec 9>"$HOME/.cache/lax-decode/box.lock" -if flock -w "${ROW_LOCK_WAIT:-5400}" 9; then - echo "row: box lock held $(date -Is)" -else - echo "row: lock unobtainable after ${ROW_LOCK_WAIT:-5400}s - aborting rather than running concurrently" - exit 3 -fi - -grep -q "first_nan" "$SRC/ggml/src/ggml-hrx/ggml-hrx.cpp" || { echo "row: ABORT - instrument not in the source"; exit 1; } -[ -d "$HRX" ] || { echo "row: ABORT - hrx-system missing"; exit 1; } - -echo "row: configuring $B $(date -Is)" -rm -rf "$B" -cmake -S "$SRC" -B "$B" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER=/opt/rocm-therock/bin/amdclang \ - -DCMAKE_CXX_COMPILER=/opt/rocm-therock/bin/amdclang++ \ - -DGGML_HIP=OFF -DGGML_HRX=ON -DGGML_VULKAN=OFF \ - -DHRX_SOURCE_DIR="$HRX" \ - -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=OFF -DBUILD_TESTING=OFF \ - > "$LOG" 2>&1 || { echo "row: cmake failed"; tail -20 "$LOG"; exit 1; } -echo "row: building -j10 $(date -Is)" -cmake --build "$B" -j 6 >> "$LOG" 2>&1 || { echo "row: build failed"; tail -30 "$LOG"; exit 1; } -"$B/bin/llama-server" --version - -LIB=$(ls "$B"/bin/libggml-hrx.so* 2>/dev/null | head -1) -grep -qa "first=%lld" "$LIB" || { echo "row: ABORT - the new field is not in $LIB"; exit 1; } -echo "row: index field present in $(basename "$LIB")" - -NAMES="ffn_moe_probs-41,ffn_moe_weights-41,ffn_moe_gate-41,ffn_moe_up-41,ffn_moe_weighted-41,ffn_moe_down-41,ffn_moe_out-41,ffn_swiglu-41,ffn_shexp-41,ffn_out-41,l_out-41,norm,result_norm,result_output" -echo "row: running $ARMS arms with the index fingerprint $(date -Is)" -CBIN="$B/bin" FBIN="$B/bin" NOLOCK=1 \ -REGIME_LO=${REGIME_LO:-80} REGIME_HI=${REGIME_HI:-92} \ - bash "$HOME/wt/315-attrib.sh" "$ARMS" "$TAG" 999 \ - "GGML_HRX_SENTINEL=1 GGML_HRX_TENSOR_FP=$NAMES GGML_HRX_TENSOR_FP_NANONLY=1" - -echo "=== row readout $(date -Is) ===" -for d in "$HOME"/wt/rt-det/"$TAG"-*/; do - [ -d "$d" ] || continue - echo "--- $(basename "$d" | sed "s/$TAG-//;s/-GLM.*//") nan_guard=$(grep -c 'HRX returned NaN logits' "$d/server.log") ---" - grep -m6 "hrx-tensor-fp" "$d/server.log" 2>/dev/null | cut -c1-165 -done -echo "Read row= against the earlier arms' nan=2048: one row of 2048 floats is the invariant to explain." diff --git a/dev/315/315-campaign/315-trace-premise.sh b/dev/315/315-campaign/315-trace-premise.sh deleted file mode 100644 index 1ad0982d3..000000000 --- a/dev/315/315-campaign/315-trace-premise.sh +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315 premise check: does a host-visible buffer back the logits? -# -# This is NOT a campaign arm — no regime claim is made. The buffer trace fires at allocation -# (model load), which is independent of GTT pressure or the fault rate, so it needs no neighbours. -# That matters because the peer #315 session is currently holding the regime band (observed at -# 103 GiB), which makes any in-regime composition with my own neighbours impossible. -set -uo pipefail -B=/home/bcloud/wt/315instr3-build/bin -M=/home/bcloud/models/GLM-4.7-Flash-Q4_K_M.gguf -LOG=/tmp/315trace.log -IREE=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 -G=$(ls /sys/class/drm/card*/device/mem_info_gtt_used | head -1) -echo "gtt_before=$(( $(cat "$G") / 1073741824 )) GiB (peer may be resident; foreign servers are not mine)" -echo "mode: GGML_HRX_USE_UNIFIED_MEMORY=${UNIFIED:-unset} (this run answers whether the logits buffer becomes coherent)" -LOG=${LOG_OVERRIDE:-$LOG} - -systemd-run --user --scope -q -p MemoryMax=60G --unit=315trace \ - --setenv=IREE_HAL_AMDGPU_LIBHSA_PATH="$IREE" \ - --setenv=LD_LIBRARY_PATH="$B" \ - --setenv=GGML_HRX_BUFFER_TRACE=1 \ - ${UNIFIED:+--setenv=GGML_HRX_USE_UNIFIED_MEMORY=$UNIFIED} \ - "$B/llama-server" -m "$M" -dev HRX0 -ngl 99 -fa on --host 127.0.0.1 --port 19911 -c 2048 -np 1 --no-webui \ - > "$LOG" 2>&1 & -for _t in $(seq 1 90); do - grep -q "\[hrx-buf\]" "$LOG" 2>/dev/null && break - sleep 2 -done -echo "trace lines after load: $(grep -c '\[hrx-buf\]' "$LOG" 2>/dev/null || true)" - -# The logits buffer is not allocated at load — it appears once the graph runs. So decode before reading -# the trace, otherwise only the model weights are ever observed (which is exactly what the first run of -# this script did, and why it was inconclusive). -for _t in $(seq 1 150); do curl -sf localhost:19911/health >/dev/null 2>&1 && break; sleep 2; done -if curl -sf localhost:19911/health >/dev/null 2>&1; then - echo "decoding to force the graph/logits allocations" - for _r in 1 2 3; do - curl -s localhost:19911/completion -H 'Content-Type: application/json' \ - -d '{"prompt":"Count: one two three four five six seven eight","n_predict":16,"temperature":0}' \ - > /dev/null 2>&1 - sleep 2 - done -else - echo "server did not become healthy; the trace below is load-time only" -fi -echo "trace lines after decode: $(grep -c '\[hrx-buf\]' "$LOG" 2>/dev/null || true)" -echo "=== visibility / coherence pairs (count) ===" -grep -o 'host_visible=[01] direct_host_binding=[01] memory_type=0x[0-9a-f]*' "$LOG" 2>/dev/null | sort | uniq -c | sort -rn | head -6 -echo "=== the logits size specifically, with its flags ===" -for sz in 619520 1239040 4956160; do - n=$(grep -c "size=$sz" "$LOG" 2>/dev/null || true) - [ "$n" != "0" ] && { echo " size=$sz count=$n :"; grep "size=$sz" "$LOG" | head -2 | cut -c1-160 | sed 's/^/ /'; } -done -echo "=== buffer type names seen ===" -grep -o 'type=[^ ]*' "$LOG" 2>/dev/null | sort | uniq -c | sort -rn | head -6 -echo "=== allocations by size (largest 6) ===" -grep -o 'memory_type=0x[0-9a-f]* host_visible=[01] direct_host_binding=[01]' "$LOG" 2>/dev/null | head -2 -grep -o 'size=[0-9]*' "$LOG" 2>/dev/null | cut -d= -f2 | sort -rn | head -6 | tr '\n' ' '; echo -echo "=== the logits size specifically (154880 x f32 = 619520; also check f16 309760, and vocab-ish) ===" -for sz in 619520 309760 154880 1239040; do - n=$(grep -c "size=$sz" "$LOG" 2>/dev/null || true) - [ "$n" != "0" ] && { echo " size=$sz count=$n :"; grep "size=$sz" "$LOG" | head -2 | cut -c1-150 | sed 's/^/ /'; } -done -echo "=== load errors? ===" -grep -iE "error|failed|abort" "$LOG" 2>/dev/null | grep -viE "sync|fallback" | head -4 | cut -c1-140 | sed 's/^/ /' -systemctl --user stop 315trace 2>/dev/null -sleep 2 -for P in $(pgrep -f "port 19911" 2>/dev/null); do kill -TERM "$P" 2>/dev/null; done -sleep 4 -echo "teardown: servers_left=$(pgrep -c llama-server || true) gtt_after=$(( $(cat "$G") / 1073741824 )) GiB" -echo "315TRACE_DONE" diff --git a/dev/315/315-campaign/315-wbfp-read.py b/dev/315/315-campaign/315-wbfp-read.py deleted file mode 100755 index 2927e12e2..000000000 --- a/dev/315/315-campaign/315-wbfp-read.py +++ /dev/null @@ -1,120 +0,0 @@ -#!/usr/bin/env python3 -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""engine#315 — read [hrx-wb-fp] records and apply the pre-registered outcome table. - -The instrumentation (dev/315/315-instrumentation.patch, enabled by -GGML_HRX_WRITEBACK_FINGERPRINT=1) logs one line per deferred host writeback: - - [hrx-wb-fp] value= size= barrier=<0|1> rec_hash= sync_hash= dst_hash= - rec_class= sync_class=... dst_class=... - rec_eq_sync=<0|1> rec_nf16=.. sync_nf16=.. dst_nf16=.. rec_nf32=.. sync_nf32=.. dst_nf32=.. - -`rec_class` is the state of the mapped staging buffer **when the writeback was queued**, -`sync_class` immediately **before** the host memcpy, `dst_class` the host destination **after** it. - -Pre-registered reading (dev/315/315-instrumentation-plan.md): - - sync=NONFINITE, dst=NONFINITE -> producer wrote non-finite values: H1 excluded, H2 stands - sync=ZERO/stale, dst=NONFINITE -> host read staging before the device copy landed: H1 - sync=ZERO/stale, dst=FINITE -> same ordering defect, non-fatal this time - sync=FINITE, dst=FINITE -> this writeback is clean - -Usage: 315-wbfp-read.py [...] -Exit 0 if records were found and classified; 3 if no records (diagnostic inert / flag off). -""" -from __future__ import annotations - -import re -import sys -from collections import Counter - -REC = re.compile( - r"\[hrx-wb-fp\] value=(-?\d+) size=(\d+) barrier=(\d+) rec_hash=([0-9a-f]+) sync_hash=([0-9a-f]+) " - r"dst_hash=([0-9a-f]+) rec_class=(\w+) sync_class=(\w+) dst_class=(\w+) rec_eq_sync=(\d+) " - r"rec_nf16=(\d+) sync_nf16=(\d+) dst_nf16=(\d+) rec_nf32=(\d+) sync_nf32=(\d+) dst_nf32=(\d+)") - - -def classify(r) -> str: - s, d = r["sync_class"], r["dst_class"] - if s == "NONFINITE" and d == "NONFINITE": - return "H1-excluded (producer wrote non-finite: sync and dst already non-finite)" - if s in ("ZERO", "EMPTY") and d == "NONFINITE": - return "H1 (host read staging before the device copy landed; fatal in this event)" - if s in ("ZERO", "EMPTY") and d == "FINITE": - return "H1 (same ordering defect, non-fatal in this event)" - if s == "FINITE" and d == "FINITE": - return "clean" - return f"unclassified (sync={s} dst={d})" - - -def main(paths) -> int: - records, faces = [], Counter() - for p in paths: - try: - text = open(p, errors="replace").read() - except OSError as exc: - print(f" cannot read {p}: {exc}") - continue - for line in text.splitlines(): - m = REC.search(line) - if m: - g = m.groups() - records.append(dict(value=int(g[0]), size=int(g[1]), barrier=int(g[2]), - rec_class=g[6], sync_class=g[7], dst_class=g[8], - rec_eq_sync=int(g[9]), rec_nf16=int(g[10]), - sync_nf16=int(g[11]), dst_nf16=int(g[12]), - rec_nf32=int(g[13]), sync_nf32=int(g[14]), dst_nf32=int(g[15]))) - faces["nan"] += text.count("HRX returned NaN logits") - faces["hsa"] += sum(text.count(s) for s in - ("HSA_STATUS_ERROR", "Queue error", "wait for HRX graph replay commands failed")) - faces["alloc_fail"] += len(re.findall(r"hrx_allocator_allocate_buffer.*failed", text)) - - print(f"records: {len(records)} faces in the same logs: nan={faces['nan']} hsa={faces['hsa']} " - f"startup_alloc_failures={faces['alloc_fail']}") - if not records: - print("no [hrx-wb-fp] records: either the flag was off (expected for the inertness arm) or " - "no deferred writeback was published in this run") - return 3 - - triples = Counter((r["rec_class"], r["sync_class"], r["dst_class"]) for r in records) - print("\nrec_class -> sync_class -> dst_class (count):") - for (a, b, c), n in triples.most_common(10): - print(f" {a:>9} -> {b:>9} -> {c:>9} x{n}") - same = sum(1 for r in records if r["rec_eq_sync"]) - print(f"\nrec_hash == sync_hash: {same}/{len(records)} " - f"(equal means the staging bytes did not change between queueing and publishing)") - - verdicts = Counter(classify(r) for r in records) - print("\npre-registered reading:") - for v, n in verdicts.most_common(): - print(f" x{n:<4} {v}") - - h1 = sum(n for v, n in verdicts.items() if v.startswith("H1 (")) - h1x = sum(n for v, n in verdicts.items() if v.startswith("H1-excluded")) - print() - if h1 and not h1x: - print("=> mechanism: the deferred/async writeback window (H1) — fix the publish ordering") - elif h1x and not h1: - print("=> mechanism: the producer wrote non-finite values (H2) — H1 excluded for these records") - elif h1 and h1x: - print("=> mixed: both rows present; report counts separately, do not pick one") - else: - print("=> neither row present: these writebacks are clean, so the faulting buffer is not this route") - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv[1:])) diff --git a/dev/315/315-campaign/315-when-free.sh b/dev/315/315-campaign/315-when-free.sh deleted file mode 100644 index dfbeb1c67..000000000 --- a/dev/315/315-campaign/315-when-free.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2026 bong-water-water-bong -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# engine#315: wait for the peer session to release the box, then run the instrumented fault hunt. -# -# Why this exists: the peer #315 session (~/wt/v315-next-fixed.sh) holds the in-regime band for hours at a -# time (measured: GTT 108 GiB with 7 GB RAM free, so a detector server cannot even load, let alone compose -# a regime). Polling that from an agent costs context and turns; instead this waits on the box itself and -# starts the hunt the moment the machine is genuinely free. -# -# Free means: GTT below a floor AND no peer process, sustained for two consecutive checks (so a gap -# between the peer's arms is not mistaken for the box being free). -# -# Usage: MAXWAIT=18000 HUNT_ARMS=12 bash 315-when-free.sh -set -uo pipefail -MAXWAIT=${MAXWAIT:-18000} -HUNT_ARMS=${HUNT_ARMS:-12} -TAGB=${TAGB:-315v3hunt} -BIN=${BIN:-/home/bcloud/wt/315instr3-build/bin} -G=$(ls /sys/class/drm/card*/device/mem_info_gtt_used | head -1) -gtt() { echo $(( $(cat "$G") / 1073741824 )); } -FLOOR=${FLOOR:-30} - -echo "=== waiting for a free box (floor=${FLOOR} GiB, max ${MAXWAIT}s, at $(date -Is)) ===" -waited=0; streak=0 -while [ "$waited" -lt "$MAXWAIT" ]; do - cur=$(gtt) - peer=$(pgrep -cf "v315-next-fixed" 2>/dev/null); peer=${peer:-0} - avail=$(free -g | awk '/^Mem:/ {print $7}') - if [ "$cur" -lt "$FLOOR" ] && [ "${peer:-0}" = "0" ] && [ "${avail:-0}" -ge 25 ]; then - streak=$((streak + 1)) - else - streak=0 - fi - echo "$(date +%H:%M:%S) gtt=${cur}G peer=${peer:-0} avail=${avail:-?}G streak=$streak waited=${waited}s" - [ "$streak" -ge 2 ] && break - sleep 60 - waited=$((waited + 60)) -done - -if [ "$streak" -lt 2 ]; then - echo "BOXWAIT_TIMEOUT after ${waited}s — box never became free; hunt not started" - exit 3 -fi - -echo "=== BOX FREE (gtt=$(gtt)G avail=$(free -g | awk '/^Mem:/ {print $7}')G) — starting the hunt ===" -# EXTRA_ENV (e.g. GGML_HRX_BUFFER_TRACE=1) is inherited by 315-fp-hunt.sh -> 315-instr-capture.sh. -HUNT_ARMS=$HUNT_ARMS bash /home/bcloud/wt/315-fp-hunt.sh "$BIN" "$HUNT_ARMS" "$TAGB" -echo "315WHENFREE_DONE" diff --git a/dev/315/315-campaign/EVIDENCE-INDEX.md b/dev/315/315-campaign/EVIDENCE-INDEX.md deleted file mode 100644 index d6400f7bf..000000000 --- a/dev/315/315-campaign/EVIDENCE-INDEX.md +++ /dev/null @@ -1,120 +0,0 @@ - - - -# engine#315 evidence index - -## Objective criteria - -| criterion | status | evidence | -|---|---|---| -| named, falsifiable mechanism for **both** faces, with a discriminating experiment — **or a bounded negative** | **BOUNDED NEGATIVE, accepted** | every surviving candidate family eliminated by its own pre-registered experiment, each with its command and observed output; the surviving families and the hand-forward fact are stated below and in `../315-STATUS.md`. **No mechanism is claimed** | -| forward-only change landed on a branch/PR | **MET** | PR **#414**. Fork commit `47a8fe929` (5 lines ending in `ggml_backend_synchronize(backend_res)` in `src/llama-context.cpp`); engine pin `e44c9d01a -> 47a8fe929`. `47a8fe929`'s **parent is `0e255d774`** = current `main` pin, and `git merge-base --is-ancestor e44c9d01a 47a8fe929` is true. `check_pins.py` reports "ahead" | -| pre-registered in-regime campaign, with the regime gate, arm count, both-face detectors and stopping rule fixed before it ran, **reported with its result whether met or not** | **RUN, acceptance FAILED — reported as such** | `evidence/315-campaign-preregistration.md` (60 fix + 20 control arms, gate `[95,110] GiB`, stop at the first fix-cell fault). The fix cell **faulted on its first arm** (Face B, `http_failures:1`, `first_failure:"HTTP 500"`, in gate at 98G) — a fault, never an unrun arm. Power statement `1 - 0.05^(1/60) ~= 4.9 %`. `evidence/315-logits-wait-FALSIFIED.md` | -| no regression: engine CI green, dense control and MUL_MAT_ID op-test unchanged | **MET, at the moved pin `47a8fe929`** | engine CI **7 of 7 green** on PR #414 (`mergeable=MERGEABLE`, `state=CLEAN`); `evidence/315-task3-op-test-unchanged-at-moved-pin.md` (`8 pass, 0 skip, 0 fail`; mxfp4 nmse `1.34e-05 … 1.61e-05`, exact match to baseline; same two kernels) and `evidence/315-task3-dense-control-clean-at-moved-pin.md` (50 responses, 1 distinct content, zero fault signatures) | -| `KNOWN-ISSUES.md` updated with the evidence, the met criterion, the reproduction and the campaign paths; status stays OPEN | **MET** | `KNOWN-ISSUES.md` #315 carries the forward-only change, the reproduction, the campaign paths and the falsification; status line **OPEN** | - -## What was established - -| claim | section | evidence path | -|---|---|---| -| Face A is a compute-side write of varied-payload NaNs; the LM head is exonerated (its input is already NaN) | 23 | `~/wt/rt-det/315hunt2-p6l-GLM-4.7-Flash-Q4_K_M/{server.log,detector.json}` | -| both faces fire on the first decoded token of a newly launched task (5/5 comparable arms, 18/19 pooled) | 23, 24 | the same logs, plus the older `identical.json` arms | -| the shipped pin's in-regime Face-A rate is 5.4% (2/37), always-stage 4.0% (1/25) | 31 | the rate study quoted in `KNOWN-ISSUES.md` | -| the sentinel instrument could not have decided "never written" (it fills once per allocation) | 25 | `ggml-hrx.cpp` `hrx_buffer_init`, plus the faulting arm's single `[hrx-sentinel]` line | -| the FA masked-row / tile-visibility direction is dead | 26 | `evidence/315-fa-masked-v-regime.md` (40/40, `non-finite outputs 0`) | -| every kernel-level test at the pin passes; two targets do not compile (pre-existing) | 27 | `evidence/315-hrx-tests-baseline.md` | -| GLM runs the **row-major MLA** layout, never the transposed-V transient | 38 | `dispatch-flash-attention.cpp` `prepare_flash_attention_value` | -| GLM never dispatches `q6_k_token1_shortlist`, so section 37's defect is not its mechanism | 37 | `evidence/315-glm-dispatch-inventory.md` | -| the pin is race-free at the October shape and at the regime's shapes (2250 + 2000 reps) | 34, 36 | `evidence/315-fa-race-pin-sweep.md` | -| exactly one `n_embd` row of NaN (2048 floats) is the invariant across the 16 faulting in-session arms, at the MoE aggregation, with the traced row count varying 87-342 | 47 | `for d in ~/wt/rt-det/315nan-*/; do grep -m1 -E "hrx-tensor-fp.*nan=[1-9]" $d/server.log; done` on strixhalo | -| the coherence *patch* does not remove the fault (9 clean, 1 fault in 10 fix-cell arms, `hv52=0 hv56=1` proving it live) | 46 | `~/wt/315accept-console.log` | -| `GRAPH_SYNC` after every replay does not remove the fault (1 in-regime arm, faulted; 4 of 6 arms below the gate) | 46 | `~/wt/315gsync-console.log`, `evidence/315-face-landscape.md` | -| the AQL-block-split asymmetry is *not* real: `packet_index_base` is a profiling prefix (0 when profiling is off) and every block is submitted as its own submission | 32 | `host_queue_command_buffer_block.c:879`, `host_queue_command_buffer_replay.c:337/364` | -| the routed-FFN path's routing tables are rebuilt per plan, so a stale table is not available as a mechanism; the sites that can lose one token's row are intra-plan producer/consumer ordering, the partition kernel's `completion_counters` publication, and the tables' transient allocation | 49 | `evidence/315-moe-routing-routed-ffn-static.md` | -| the engine's MoE expert-streaming cache (`serve --moe-slots`/`--moe-subst`, `moe/expert_cache.cpp`) is not on the HRX decode path | 49 | the same file, section 6 | -| the two faces never co-occur (23 Face-A, 7 Face-B, 0 both, 108 clean of 142 arms) | 42 | `evidence/315-face-landscape.md` | -| Face A travels with finite nondeterminism; Face B does not | 42 | the same file (`identical_distinct` distribution) | -| neither instrument can fabricate NaN (copy probe only fprintfs; sentinel is size-gated to 619520) | 41 | `ggml-hrx.cpp:181-199, 380-391` in the fp build's tree | -| the faulting frame is a T=1 decode whose final norm is all-NaN, the first NaN frame a prefill at block 41 | 40 | the `p10l` arm's `server.log` lines 1713-1868 | -| the transient-reuse guard is default-on and is the packer that runs for decode programs | 35 | `GGML_HRX_LOG_TRANSIENT_REUSE=1` output quoted in section 35 | - -## What is eliminated (with the cost it took) - -Sections 26, 27, 32, 33, 34, 36, 37, 38 - coherence and ordering (their own campaign), FA masked rows, all -kernel standalone logic, the AQL block-boundary asymmetry, intra-dispatch unfenced global handoffs, the -October FA race, the `exact_output` access defect. Section 29 is the consolidated table. - -## Session of 2026-10-15 — the ordering family removed, one site shown unmeasurable - -| claim | evidence | -|---|---| -| **the writer of the `result_norm` slot is NOT ordered by `context->stream`** — on a faulting arm, of 82 records where the tensor differed between two consecutive plain reads, a fresh `ggml_backend_synchronize` stopped it in **0** and did not in **82**, and changed the value itself in **82 of 82** | `evidence/315-task1-writer-not-stream-ordered.md` (`b7d6b46`); pre-registered at `evidence/315-PREREG-sync-between-reads.md` (`b51f883`) | -| this **removes** `GraphExecutor::execute` returning without a wait (with `backend_synchronize` covering only `context->stream`) as the explanation | the same file; the falsifier is restated there and **not triggered** | -| **§51's rank-sort precondition is not measurable by the by-name fingerprint** — the faulting arm produced 0 records; the `tail:16` census has **946 distinct node names** and contains `ffn_moe_out-*`/`ffn_shexp-*` but **none** of `ffn_moe_probs-*`, `ffn_moe_topk-*`, `ffn_moe_argsort-*`, `ffn_moe_weights-*` | `evidence/315-task2-bounded-negative-precondition-unmeasurable.md` (`ba6753a`) | -| consequently **how §50 read `ffn_moe_probs` is itself unexplained** — recorded as an open discrepancy, not smoothed over | the same file | -| `95238dc` was a **half-move**: it moved the `llama.cpp` gitlink without `registry/architectures.json`; CI failed until fixed | `evidence/315-what-the-merge-and-the-gates-found.md` | -| **46 files** failed the copyright gate on the two-line SPDX form; `*.hip` was a genuine gap in the tool | the same file | -| four harness defects of one class this session (skip-as-pass; `"0\n0"` from `grep -c … \|\| echo 0`; line-based response count; a vacuous CI green) | `evidence/315-task3-dense-control-clean-at-moved-pin.md`, and for other sessions in `../315-COORDINATION.md` | - -## Instruments and tools - -| item | path | state | -|---|---|---| -| nan-dump fingerprint patch (`tail:`, `NANONLY`, `op=`) | `patches/315-nan-dump.md`, `tools/gen-nan-dump.py` | applied to `~/wt/llama-315copy`, built into `~/wt/315fp-build` | -| non-finite masked-V test extension | `patches/315-fa-masked-v-nonfinite.md`, `tools/gen-fa-masked-nonfinite.py` | built and run (40 cases) | -| binding-access audit | `tools/audit-binding-access.py` | run over all four manifests (522 bindings, 2 findings) | -| FA race sweeps | `tools/fa-sweep.sh`, `tools/fa-variant-profile.py` | run (2250 + 2000 reps) | -| MLA race harness | `tools/glm-mla-race.cpp` | compiled on strixhalo, **not run** (needs the GPU) | -| cheap probe | `315-cheap-probe.sh` | run; one fault in eight runs at `align=256 + compaction` | -| instrumented arm hunt | `315-paired.sh` + the §27 command | ran; record in `evidence/315-instrumented-arm-hunt.md` (17 fault / 1 clean, and a control arm that faulted) | -| face landscape | (post-hoc parse of the arm logs) | `evidence/315-face-landscape.md` | -| acceptance tooling | `315-campaign.sh`, `315-paired.sh`, `315-detector.py`, `315-analyse.py` | frozen; four freeze notes in `315-preregistration.md` section 8 | -| acceptance rule self-test | `tools/test-analyse.py` | 10/10 cases pass (gate edges, Face-B-as-fault, rate over in-regime arms rather than clean arms) | -| attribution + chain probes | `315-attrib.sh`, `315-chain.sh`, `315-chain2.sh` | diagnostic only, never acceptance-eligible. `315-chain.sh` ran: pair 1 faulted both cells with a 60-name fingerprint and **zero** NaN lines, which is the direct Face-B measurement; `315-chain2.sh` is staged for the input-side case | -| visibility falsifier | `315-graphsync.sh` | ran 2026-10-10 (6 pairs, `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1` both cells, no other instrument): **4 of 6 arms below the gate**; the one in-regime arm faulted. Read as *not sufficient*, not as a rate. Record: `evidence/315-face-landscape.md` | -| row probe | `315-rowprobe.sh` | ran 2026-10-10: builds the extended fingerprint (`ne0=`/`first=`/`row=`) into `~/wt/315row-build` and reads the per-token MoE chain (section 47) | -| Face-B probe (not run) | `315-faceb.sh` | written then dropped in favour of the visibility falsifier; its question is answered by the chain probe's zero-NaN result | - -## Arms and logs - -`~/wt/rt-det/-p-GLM-4.7-Flash-Q4_K_M/` on strixhalo (server.log, detector.json, run.log). -Note the glob must be `*-*l-*`, not `*-l-*`. Campaign arms would be `315fix-*`; none exist yet. - -## Verification of this index (2026-10-10) - -Every number quoted across these evidence files is re-checked from the box by one command: - -```sh -bash dev/315/315-campaign/tools/verify-evidence.sh # 30 checks, all PASS; ~25 s per check -# ----- PASS=30 FAIL=0 -``` - -It covers: the five in-regime arms and their verdicts, the guard line, the faulting arm's fingerprint lines and -their hashes, the two host-visible allocations from the buffer trace, the coherent-cell trace, the archived -overshoot console, the one-file differential (`102 insertions, 2 deletions`) and the empty routed-FFN diff, the -`route_stride` counts in both trees, the queue profile's dispatch/barrier/block numbers, the `scacquire` wait -variant, the timepoint assignment, the three instrument patches, the wrapper directory, and the two kernel facts -(ids clamped in the gather, ordered compares in the rank sort). Two checks needed their *expected* regex fixed -when the suite was first written - the underlying numbers were right, the assertions were not - and the clean run -above is with the fixed assertions. diff --git a/dev/315/315-campaign/ISSUE-COMMENT-2-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-2-2026-10-10.md deleted file mode 100644 index 13d326785..000000000 --- a/dev/315/315-campaign/ISSUE-COMMENT-2-2026-10-10.md +++ /dev/null @@ -1,101 +0,0 @@ - -## In-regime fault captured with the producing values named — the corruption is in the MoE **routing** tensors, and it is whole-batch, not one row - -Follow-up to my previous comment (same day). Diagnostics only; **#315 stays OPEN**, and this is not yet a -mechanism: it is the first in-regime fault captured with the whole-graph *named* fingerprint, and it moves the -site. - -### The arm - -`315row-p1c`, 2026-10-10, build `315row-build` (pin `e44c9d01a4d5` + the extended fingerprint), composition 3 -neighbours at 80 GiB, **`gtt_start=98G`** (the frozen pre-registration's in-regime gate is `>= 95 GiB`). Driver -verdict `pair 1 arm c: FAULT nan=1 req=34`, detector `transport_failures 1`, `identical_distinct 1`. The guard -line and the engine's own abort: - -```text -2.53.659.050 E common_sampler_sample: HRX returned NaN logits: count=154880 of 154880, index range [0..154879], contiguous=yes, continuing=no sentinel=0 -/home/bcloud/wt/llama-315copy/common/sampling.cpp:654: HRX: NaN logits (engine#123) -[New LWP 100365] -``` - -```sh -ssh strixhalo 'grep "hrx-tensor-fp" ~/wt/rt-det/315row-p1c-*/server.log' -``` - -### The fingerprint (16 lines, 3 graph calls, verbatim) - -```text -call=158945 ffn_moe_weights-4 GET_ROWS bytes=160 nan=8 of 40 ne0=1 first=32 row=32 hash=56753a5bd9bd2f87 -call=158945 ffn_moe_gate-4 MUL_MAT_ID bytes=245760 nan=51141 of 61440 ne0=1536 first=32 row=0 hash=97678a5700290a33 -call=158945 ffn_moe_up-4 MUL_MAT_ID bytes=245760 nan=61440 of 61440 ne0=1536 first=0 row=0 hash=d33023c41f384383 -call=158945 ffn_moe_down-4 MUL_MAT_ID bytes=327680 nan=50556 of 81920 ne0=2048 first=640 row=0 hash=b44eb621b102d019 -call=158945 ffn_moe_weighted-4 MUL bytes=327680 nan=81880 of 81920 ne0=2048 first=40 row=0 hash=514da62652bda383 -call=158945 l_out-4 ADD bytes=81920 nan=20480 of 20480 ne0=2048 first=0 row=0 hash=79cb0b351c401383 -call=159019 ffn_moe_probs-41 UNARY bytes=2560 nan=640 of 640 ne0=64 first=0 row=0 hash=4e017b66eae91403 -call=159019 ffn_moe_weights-41 GET_ROWS bytes=160 nan=8 of 40 ne0=1 first=32 row=32 hash=56753a5bd9bd2f87 -call=159019 ffn_moe_gate-41 MUL_MAT_ID bytes=245760 nan=51104 of 61440 ne0=1536 first=32 row=0 hash=0e72d35347f45563 -call=159019 ffn_moe_up-41 MUL_MAT_ID bytes=245760 nan=61440 of 61440 ne0=1536 first=0 row=0 hash=d33023c41f384383 -call=159019 ffn_moe_down-41 MUL_MAT_ID bytes=327680 nan=51136 of 81920 ne0=2048 first=0 row=0 hash=58642d3df06abb43 -call=159019 ffn_moe_weighted-41 MUL bytes=327680 nan=81880 of 81920 ne0=2048 first=40 row=0 hash=6ba84ddfb3b8f1c7 -call=159019 l_out-41 ADD bytes=81920 nan=20480 of 20480 ne0=2048 first=0 row=0 hash=79cb0b351c401383 -call=159029 norm RMS_NORM bytes=8192 nan=2048 of 2048 ne0=2048 first=0 row=0 hash=89ade87e7a93fb83 -call=159029 result_norm MUL bytes=8192 nan=2016 of 2048 ne0=2048 first=32 row=0 hash=77c1ef05f75f6127 -call=159029 result_output MUL_MAT bytes=619520 nan=143745 of 154880 ne0=154880 first=32 row=0 hash=beb27746f7308f47 -``` - -### Four things it establishes - -1. **The routing tensors carry NaN, not only the aggregation.** `ffn_moe_weights` is - `ggml_get_rows(ffn_moe_probs, ffn_moe_topk)` — the *selected route weights* — 40 values = 4 routes × 10 - tokens, of which 8 are NaN with `first=32`: **tokens 8 and 9, all four routes each**. Its **source** - `ffn_moe_probs-4` carries no NaN at all (the instrument prints every matched tensor with a non-zero NaN - count, and `probs-4` is absent). A gather whose output is NaN where its source is finite is either reading - outside its source — an out-of-range id — or not writing its output. -2. **It is not one row at this stage.** `l_out-4` is `nan=20480 of 20480`: **all ten tokens'** 2048-wide hidden - rows. `ffn_moe_up-4` is `nan=61440 of 61440` (all 40 route rows). By the end of layer 4 this short prefill - (T=10) is entirely NaN, and the logits are 143745/154880. -3. **It is one instant.** Only three graph calls in ~159 000 produced fingerprint lines, and they are - consecutive — layer 4, layer 41, the output. The same graphs had been computed thousands of times before - with no NaN line. -4. **Two probe defects, recorded rather than quietly fixed.** (a) The filter is `std::strstr(want, node->name)` - — it searches for the node name *inside* the comma-separated list — so the list entry `ffn_moe_weights-41` - also matches `ffn_moe_weights-4`. The arm covered **layers 4 and 41 only**, which is why the picture jumps - from 4 to 41. (b) `ffn_moe_topk`, the ids tensor, is not in the list and would not be readable anyway: the - instrument reads every tensor as `f32`. The hypothesis this arm most directly suggests — a corrupt or - out-of-range route id — is therefore **still untested**. - -### Tension with the "one token's row" invariant, stated rather than smoothed over - -My previous comment reported one `n_embd` row of NaN (2048 floats) as the invariant across 16 faulting arms. -That was measured with a different instrument — a `tail:16` window over the last 16 graph nodes — in a different -session, on prefills of T = 87…342. This arm is a T = 10 prefill measured by name, and it is whole-batch. Both -cannot be quoted as "the fault's shape" at once. Until the same instrument is run over the same arms, the safe -statement is the narrower one: **the corruption involves the MoE routing tensors, and the row extent is -instrument-dependent.** I am explicitly not defending the earlier invariant here. - -### What it does not say - -No rate (one arm, `req=34` of 66 before the abort, and the instrument is known to raise the fault rate). No -claim about layers other than 4 and 41. No claim that the identical `weights`/`up`/`l_out` hashes at layers 4 and -41 mean aliasing — 100 %-NaN tensors hash equally by construction, and for `weights` (only 8 of 40 NaN) the other -reading is an aliased transient read, which is testable and not adopted here. - -### Next measurement, queued - -The anchored-match fingerprint with `ffn_moe_topk` and the router logits added, printing ids as integers, plus a -finite-subset hash to settle the aliasing question. Then the rate question, which needs the campaign and the box. diff --git a/dev/315/315-campaign/ISSUE-COMMENT-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-2026-10-10.md deleted file mode 100644 index a6226821c..000000000 --- a/dev/315/315-campaign/ISSUE-COMMENT-2026-10-10.md +++ /dev/null @@ -1,111 +0,0 @@ - -## Mechanism narrowed: exactly one token's hidden row is NaN at the MoE aggregation — plus three falsifications. No mechanism claim yet; #315 stays OPEN - -Diagnostics only. Written 2026-10-10 on `strixhalo` against the pinned revision (`third_party/llama.cpp -e44c9d01a4d5`). Every number below comes from a command quoted next to it. The row-index discriminator that -decides the remaining question is running on the box now, so this comment names no mechanism. - -### 1. The invariant: 2048 NaN floats — one `n_embd` row — while the batch size varies four-fold - -```sh -ssh strixhalo 'cd ~/wt/rt-det; for d in 315nan-*/; do l=$(grep -m1 -E "hrx-tensor-fp.*nan=[1-9]" "$d/server.log"); \ - [ -n "$l" ] && echo "$(basename $d | cut -d- -f1-2) :: ${l#*hrx-tensor-fp}"; done' -``` - -| arm | first NaN-bearing node | op | bytes | floats | NaN | -|---|---|---|---|---|---| -| p1l, p6l, p9l, p10l | `node_2948` | ADD | 1409024 | 352256 | **2048** | -| p11l | `node_2948` | ADD | 2506752 | 626688 | **2048** | -| p13l | `node_2948` | ADD | 1146880 | 286720 | **2048** | -| p15l | `node_2948` | ADD | 802816 | 200704 | **2048** | -| p17l | `node_2948` | ADD | 2801664 | 700416 | **2048** | -| p2l, p3l, p4l, p7l, p8l | `node_2948` | ADD | 712704 | 178176 | **2048** | -| p5l | `node_2948` | ADD | 1105920 | 276480 | **2048** | -| p14l | `ffn_swiglu-41` | GLU | 534528 | 133632 | **2048** | -| p12l | `norm-0` | RMS_NORM | 81920 | 20480 | 55 | -| (`p20l`, excluded — see below) | `ffn_swiglu-41` | GLU | 2138112 | 534528 | 2048 | - -The traced tensor's row count changes by a factor of four across arms (`bytes/4/2048` = 87, 98, 135, 140, 172, -306, 342 rows) while the NaN count is **2048 in 15 of 16 arms**: exactly one `n_embd` row. GLM-4.7-Flash's -`n_embd` is 2048, so the defect is *one token's hidden vector*, not a block, not the tail of the graph. - -`p20l` is excluded deliberately: its Face-A event happened at `gtt_start=44` **after that session SIGKILLed -another arm's server**, so it is ambiguous and is not counted (this is recorded, not quietly dropped). - -Two corroborating numbers from the same population: - -* the whole-vocabulary Face-A fingerprint reads `count=154880 of 154880, contiguous=yes` — the *sampled row* of - the final norm is all-NaN (`315hunt2-p6l`: `name=norm bytes=8192 nan=2048 of 2048`), which is downstream of the - row above, not a second event; -* `node_2948` is unnamed because of how the MoE sum is built: `build_moe_ffn` calls `ggml_add` per expert and - only the final sum gets `cb(moe_out, "ffn_moe_out", il)` (`src/llama-graph.cpp:2151-2162`). GLM uses 4 experts, - so two or three adds in that chain are unnamed and `node_2948` is one of them — a **partial expert sum**. - -### 2. What this rules out - -* **A stale or globally corrupt logits buffer.** That cannot put one NaN row into a *prefill* graph of T=172 while - the other 171 rows are finite. -* **Corrupt weights.** Those would hit every token, not one. -* **"The logits were never written"** — already falsified on the record, and now falsified in shape: the writes - happen and one row is wrong. - -### 3. Face B is the same corruption, finite — measured, not inferred - -The chain probe (`315chain`, 2026-10-10) faulted its first pair in **both** cells with a 60-name fingerprint over -the residual stream, both attention branches, the whole FFN/MoE chain and the output norm, and logged **zero** -NaN lines — no NaN anywhere, logits included. The rejected text was `the}:}:}甚;` in the fix cell and a -repetition loop (`a gaps gaps gaps gaps ...`) in the control cell. So Face B is *finite* corruption: near-degenerate -logits under greedy sampling, rejected later by the parser -(`common_chat_peg_parse: unparsed Content-only output:`). - -Across all 142 recorded arms the faces never co-occur (23 Face-A, 7 Face-B, 0 both, 108 clean), and Face A -travels with finite nondeterminism while Face B does not (`identical_distinct` > 1 in exactly the 7 Face-A arms). -So the observable depends on what the corrupted row held — NaN (caught at the sampler) or stale finite values -(caught at the parser) — which is one mechanism with two observable faces. - -### 4. Falsified today: the visibility/ordering family is not sufficient - -* **Coherence.** A single-bit patch (`HRX_MEMORY_TYPE_HOST_COHERENT` for host-visible buffers) was run as its own - cell, with the driver's own trace proving the patch was live (`memory_type=0x52` → `0x56`, i.e. `hv52=0 - hv56=1`): **9 clean and 1 fault in 10 fix-cell arms** — the baseline shape. Coherence is not sufficient. -* **Consumer-side ordering.** `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1` calls `hrx_stream_synchronize()` after every - graph replay (`command-program-executor.cpp:2034`), so every host read must follow the device's writes: 6 arms, - 4 of them below the gate; the one in-regime arm (`gtt_start=99`) **faulted** (Face-B class, `req=19`, `nan=0`), - and no arm that reached the gate was clean. Ordering is not sufficient either. -* A correction that matters for how these are read: earlier eliminations leaned on the logits `sentinel=0` line, - but the sentinel only fills the 619520-byte logits allocation, so it never covered the tensors where the NaN is - actually born. The two results above stand on their own traces, not on that inference. - -### 5. The discriminator now running, and its pre-committed readout - -The fingerprint has been extended to print `ne0=`, `first=` and `row=` (the token index), and is being run over -the named per-token MoE tensors (`ffn_moe_probs/weights/gate/up/weighted/down/out`, `ffn_swiglu`, `ffn_shexp`, -`l_out-41`, `norm`, `result_norm`). Read it as: - -| result | reading | consequence | -|---|---|---| -| the NaN row is the **last** row in every arm | the token whose routing is computed last — a read not ordered after the routing computation | the fix is to order it | -| the **same** row index throughout the chain | a per-token state: the routing weight or one selected expert's activation for that token | names the carrier | -| **differing** row indices | not per-token at all; the invariant is an artefact of the sampler's single-row read | the direction is wrong and must be abandoned | - -### 6. Status - -`#315` remains **OPEN**. What is established is the shape (one token's hidden row, entering at the MoE -aggregation) and three falsifications; what is not is the site. That is what the running probe is for, and the -next comment will carry its arm-by-arm output. KNOWN-ISSUES.md still carries the superseded "the logits buffer is -never written" story and will be corrected with the mechanism rather than ahead of it. diff --git a/dev/315/315-campaign/ISSUE-COMMENT-3-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-3-2026-10-10.md deleted file mode 100644 index f3dd10f41..000000000 --- a/dev/315/315-campaign/ISSUE-COMMENT-3-2026-10-10.md +++ /dev/null @@ -1,59 +0,0 @@ - -## Correction: I withdraw my previous comment's per-node attribution — and the reason points at a better candidate - -My previous comment reported that the corruption "enters at the routing gather" because `ffn_moe_weights-4` was NaN while its source `ffn_moe_probs-4` was not. **That inference does not hold, and I am withdrawing it.** The measurement it rests on is not ordered, and I can now show why. - -### The instrument reads a host-visible arena unsynchronised - -`GGML_HRX_BUFFER_TRACE` names every allocation of 64 KiB or more (`ggml-hrx.cpp:343`). In a complete arm the host-visible allocations are exactly two: - -```text -[hrx-buf] type=HRX0 host_visible=0 direct_host_binding=0 memory_type=0x30 size=17895352832 (weights) -[hrx-buf] type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=619520 (logits) -[hrx-buf] type=HRX0 host_visible=0 direct_host_binding=0 memory_type=0x30 size=443547648 -[hrx-buf] type=HRX0 host_visible=0 direct_host_binding=0 memory_type=0x30 size=81788928 -[hrx-buf] type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=44048640 (arena) -``` - -Every tensor the fingerprint names on the MoE path — `ffn_moe_weights` (160 B), `ffn_moe_probs` (2560 B), `ffn_moe_gate`/`up` (245760 B), `ffn_moe_down`/`weighted` (327680 B), `l_out` (81920 B) — is too small to be a buffer of its own, so it lives inside the **44048640-byte host-visible, non-coherent arena**. And `ggml_backend_tensor_get` on a host-visible tensor is a plain `memcpy`: no iface call, no synchronisation. That is the same structural fact the earlier hard negative established (it is why the download probes could never fire). - -So the fingerprint's per-node lines are **not a snapshot of one instant** — each read samples the arena whenever the CPU reaches it. The two faulting arms show exactly that, and it is the contradiction that gives it away: - -* `ffn_moe_weighted-4` reads **81880 of 81920 NaN**, while `ffn_moe_out-4` — its own sum over the routes — reads **finite**; -* `ffn_moe_probs-4` reads finite while the gather made from it reads NaN. - -Neither can be true of computed data. What survives from those arms is only what does not depend on read order: the guard's own line, `count=154880 of 154880 … contiguous=yes`, and the fact that the fault is real and in-regime (`gtt_start=98G`). - -### What that structure suggests instead — and it fits both faces better - -The consumer reads its inputs with the same unsynchronised `memcpy` out of the same class of buffer, and nothing in that path waits for the producing dispatch. A host read that precedes the write yields **unwritten memory**: - -* if the mapping held NaNs → **Face A**, `count=154880 of 154880`, the guard fires; -* if it held the **previous token's finite logits** → **Face B**: the sampler draws from a stale distribution, and the two text flavours already on the record are what that produces — the repetition loop `a gaps gaps gaps gaps ...` is literally one token repeated, and the short fragment is the same read scaled differently. Face B never logged NaN, which is what a *finite* stale row gives. - -Regime dependence follows from the mapping's history: under GTT pressure, what the arena holds and when the write lands both change. - -### The falsification already on the record, and the decisive run - -This candidate has one negative result against it and it is thin: `315-graphsync.sh` ran `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1` (a stream sync after every replay, `command-program-executor.cpp:2034`) and its single in-regime arm **faulted**. But that session's composition stopped at 70 GiB and **4 of its 6 arms fell below the gate**, so that is one arm of evidence, not a falsification — and the composition override that caused it is now fixed (`REGIME_LO=80` in the wrappers, which had been overriding the module's own default downward; that is why the graphsync run under-shot the gate while `315chain`, which inherits the default, produced 98 GiB arms). - -Queued now, as a **paired** in-regime A/B rather than a comparison against on-disk arms: both cells the same instrumented binary and the same instrument (so the fault rate is high enough to see a difference in a handful of arms), the only difference being the sync knob — the fix cell runs through a launcher that sets `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1`, the control cell does not, arms alternating inside one composition, at `gtt_start` projected into 101-106 GiB. Verdicts come from the guard and the detector's completion accounting, not from the fingerprint. Zero faults in the fix cell while the control keeps faulting makes the read-path ordering the mechanism; an unchanged rate kills it. - -```sh -ssh strixhalo 'grep -E "^ pair|control_bin|fix_bin" ~/wt/315gsync2-console.log' -``` diff --git a/dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md b/dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md deleted file mode 100644 index 4044699ee..000000000 --- a/dev/315/315-campaign/ISSUE-COMMENT-4-DRAFT.md +++ /dev/null @@ -1,101 +0,0 @@ - -## engine#315 — final record (DRAFT: the decisive result is the only placeholder) - - - -Status: **#315 stays OPEN.** This comment records what is established, what is falsified with its own evidence, -the decisive experiment's arm-by-arm counts, and the remedy. Every number is tied to a re-runnable command. - -### 1. What the fault is, at the resolution the instruments can support - -An in-regime Face-A event, captured twice with the guard's own line: - -```text -2.53.659.050 E common_sampler_sample: HRX returned NaN logits: count=154880 of 154880, index range [0..154879], contiguous=yes, continuing=no sentinel=0 -``` - -`315row` (2026-10-10) scored five arms, **every one at `gtt_start=98G`** — inside the frozen pre-registration's -`>= 95 GiB` gate — with **two faults** (`nan=1`, each aborting at `req=34` of 66) and three arms completing `66/66`: - -```sh -ssh strixhalo 'grep -E "^ pair" ~/wt/315row-console.log' -``` - -That is 2 of 5 in-regime arms under the by-name instrument, against the shipped pin's recorded 2/37 = 5.4 %. The -instrument is an inducer (sections 39/41), which is why no rate is quoted from these arms and none is -acceptance-eligible; it is also what made five arms informative at all. - -**The locating instrument was withdrawn, by its own output.** The by-name fingerprint reads graph tensors out of a -**host-visible, non-coherent 44 MB arena** (`44048640`, from `GGML_HRX_BUFFER_TRACE`), and -`ggml_backend_tensor_get` on a host-visible tensor is a plain `memcpy` with no synchronisation - so its lines are -not a snapshot of one instant. The two faulting arms prove it against themselves: `ffn_moe_weighted-4` reads -`nan=81880 of 81920` while `ffn_moe_out-4`, its own sum, reads finite. My earlier per-node attribution is -**withdrawn** (mechanism-doc section 53), and the repair is written and gated (`tools/patch-fp-sync.py`, -`315-fp2.sh`). - -### 2. The decisive experiment (arm-by-arm) - -TODO-RESULT - -### 3. Falsified, each with its own evidence - -| candidate | result | where | -|---|---|---| -| memory coherence is the carrier | the `HOST_COHERENT` patch was live (`hv52=0 hv56=1`) and the cell still faulted: 9 clean + 1 fault in 10 | `evidence/315-face-landscape.md`, mechanism §46 | -| consumer-side ordering is the carrier | `GRAPH_SYNC` after every replay: 4 of 6 arms below the gate, the one in-regime arm faulted | same file, mechanism §46 | -| the read path is unordered because nothing waits | source: `llama_get_logits_ith` synchronises (`src/llama-context.cpp:3729`), the wait ends in **`iree_hsa_signal_wait_scacquire`** (`runtime/src/iree/hal/drivers/amdgpu/semaphore.c:329`), and `stream->timepoint` is advanced *after* each submission | mechanism §§55-56 | -| the expert/partition routing tables go stale across graphs | both are rebuilt by producer dispatches inside the same plan (`dispatch-mul-mat-id-common.h:342-368`) | `evidence/315-moe-routing-routed-ffn-static.md` §3 | -| the fused `mul_mat_id_swiglu` path loses `route_stride` | that defect is real and **already fixed in every tree that faults** (3 + 10 occurrences of `route_stride` in both) | the same file, §7 | -| GLM's decode splits into two 128 KB AQL blocks | the queue profile reads `blocks 1`, `block-boundary gaps 0 us over 0`, at the default and at `HRX_AQL_BLOCK_SIZE` 262144 and 524288 | `evidence/315-queue-profile.md` | -| the pin move introduced the fault in the routed FFN | the whole `ggml/src/ggml-hrx` tree differs between the clean bound `f2099e9b7` and the pin by **one file**, the decode-split FA kernel (+102/-2) | `evidence/315-differential-artifact.md` | -| the engine's MoE expert-streaming cache | Vulkan-only; the arms load into GTT through HRX and never enter it | `evidence/315-moe-routing-routed-ffn-static.md` §6 | -| the block-size knob reaches the block transition | it changed neither the profile nor throughput (pp/tg 0.9965-1.0073) | `evidence/315-queue-profile.md` | -| the t2 hard negative (logits are not host-staged) | upheld, and explained: no copy exists because the buffer is host-visible and the generic layer short-circuits | `pm-salvage/issue-315/TASK2-INSTRUMENT-RESULT.md` | - -Still on the record as **not** falsified: the regime's own contribution to *when* the window opens (unbarriered -dispatches plus a saturated queue: 92 of 612 dispatches carry no barrier, `busy 14288 us` of a `15348 us` span), -and the possibility that the corruption is computed, which is what the ordered locator would settle. - -### 4. Regime dependence, stated as far as the evidence goes - -The fault is absent below ~80 GiB in the recorded population and appears from ~95 GiB up, and the composition -arithmetic is now measured rather than assumed: neighbours at 70 GiB put arms at `gtt_start` 87-99 GiB (4 of 6 -below the gate) while neighbours at 80 GiB put them at 98 GiB, with the detector contributing the remainder up to -the ~110 GiB upper bound. Two load-bearing tooling corrections came out of that: every probe wrapper had been -overriding the composition target **downward** to 70 GiB (`REGIME_LO=${REGIME_LO:-70}`, against the module's own -default of 80), which is why one session under-shot the gate while another reached 98 GiB with the same harness; -and a *clean* arm can leave its server scope resident, so GTT reached 113.5 GiB against a ~80 GiB composition -until the scope was stopped. Evidence: mechanism §48, `evidence/315-queue-profile.md` appendix. - -### 5. Remedy - -TODO-REMEDY — to be written from the decisive result. A landed fix is optional for this issue; a documented -limitation is acceptable once the mechanism is proven, and either way the text goes beside the mechanism in -`KNOWN-ISSUES.md`, which already carries the superseded reading clearly marked. - -### 6. Commands - -```sh -ssh strixhalo 'grep -E "^ pair" ~/wt/315row-console.log' # the in-regime capture -ssh strixhalo 'grep "HRX returned NaN logits" ~/wt/rt-det/315row-p1c-*/server.log' -ssh strixhalo 'cat ~/wt/315gsync2-console.log' # the decisive experiment -ssh strixhalo 'cd ~/wt/llama-315copy && git diff --stat f2099e9b7..HEAD -- ggml/src/ggml-hrx/' # the differential -ssh strixhalo 'head -3 ~/wt/hrxsys-prof-run.log' # the queue profile -``` diff --git a/dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md b/dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md deleted file mode 100644 index 078d6d274..000000000 --- a/dev/315/315-campaign/ISSUE-COMMENT-5-2026-10-10.md +++ /dev/null @@ -1,153 +0,0 @@ - -## The ordering family is closed by four null toggles in regime; no mechanism is named yet, and #315 stays OPEN - -This is the consolidation of today's in-regime work on `strixhalo` (pin `e44c9d01a4d5`). It reports a **negative -result honestly**: every lever that could have made an ordering or visibility defect produce this fault has now -been toggled in regime, arm by arm, and **none of them changes the outcome**. That falsifies the whole -ordering/visibility super-family and leaves the fault in the *computed* class, where the instruments available -today cannot name the producer. - -### 1. The toggles, with their arm counts - -Every run below: same harness (`315-paired.sh` through `315-attrib.sh`), same model, same composition -(`REGIME_LO=80`), arms alternating inside one session where a control cell exists, and **every arm at -`gtt_start = 98 GiB`** — inside the frozen pre-registration's `>= 95 GiB` gate. "Inducer" means both cells carried -`GGML_HRX_TENSOR_FP=tail:16 NANONLY=1`, which is what makes a difference visible in a handful of arms (it raises -the rate; no rate here is an acceptance figure). - -| lever toggled | control cell | fix cell | reading | -|---|---|---|---| -| `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1` (a stream sync after every graph replay) | 1 fault / 4 arms | 0 / 2 | no change (Fisher p = 1.00) | -| `ggml_backend_synchronize` before the instrument's reads (patched build vs the previous build) | 2 / 5 (`315row`) | 1 / 5 (`315fp2`) | no change (p = 1.00) | -| forced AQL execution barrier on **every** packet (`HRX_FORCE_BARRIERS=1`, env-gated patch in `aql_block_processor.c`) | 2 / 4 | 1 / 2 | no change | -| `-fa off` (take the model off the flash-attention path entirely) | 1 / 2 | 2 / 2 | fault persists | -| `HOST_COHERENT` on host-visible buffers (earlier today, its own campaign) | — | 1 fault in 10 | not sufficient | -| `HRX_COMMAND_BUFFER_MODE=pm4` | 3 OK + 1 fault / 4 | 0 OK + 4 arms at `req=1` | **unusable**: every PM4 arm failed the *first* request with HTTP 500, so it is not a fault-rate comparison at all | - -### 2. The family is closed at source as well, not only by the runs - -* the host read path is already ordered: `llama_get_logits_ith` synchronises (`src/llama-context.cpp:3729`), the - wait ends in **`iree_hsa_signal_wait_scacquire`** (`runtime/src/iree/hal/drivers/amdgpu/semaphore.c:329`), and - `stream->timepoint` is advanced *after* each submission, so an `hrx_stream_wait` waits on the last one; -* the packet barrier bit is decided by `..._packet_has_barrier` (`aql_block_processor.c:103-116`) and the driver - explicitly does **not** infer memory hazards (`:126`) - and forcing that bit on everywhere changed nothing; -* the intra-dispatch variant of the same class was audited statically over the whole `loom-libs/ops` corpus: the - three kernels that hand work across waves through global memory are exactly the three carrying - `barrier`, and the quantized-matmul partials are a two-dispatch reduction whose ordering the host owns. - -### 3. Why "name the producing value" is not obtainable with the instruments I have - -Two independent findings, each proven from the arms themselves, each sufficient on its own: - -1. **The fingerprint reads a host-visible, non-coherent arena with a plain `memcpy`.** The MoE intermediates live - inside one `44048640`-byte `host_visible=1 memory_type=0x52` allocation (buffer trace), and - `ggml_backend_tensor_get` short-circuits for host buffers - no iface call, no sync. Ordering those reads - (patch applied, verified in the build) changed nothing about the fault (row 2/5 vs fp2 1/5), so the reads are - ordered now and the numbers below still stand: **the locator was never the problem.** -2. **A post-graph read by name returns the transient arena's last writer.** In the faulting arm - `ffn_out-0` reads `nan=7048 of 1048576` while `l_out-0` reads `nan=4963 of 1048576` - impossible, because - `l_out = ffn_out + attn_out` requires `l_out`'s NaN set to contain `ffn_out`'s. The only explanation is that - the slot an earlier tensor occupied has since been reused by a later node, so reading it "by name" returns - somebody else's bytes. - -Both mean the same thing for task-2's wording: a *post-graph, by-name* instrument cannot name the producing graph -value on this backend, and that is a property of the backend, not of the fault. What remains trustworthy is -therefore narrow, and it is what I would carry forward: the guard's own line, the non-transient tail -(`norm`, `result_norm`, `result_output`), and the arm verdicts. - -### 4. What is established, at the resolution that survives - -* **The fault is real, in regime, and common enough to study.** 178 arm directories on the box, **131 in-regime, - 34 faulting in-regime**, across a dozen independent sessions, at `gtt_start` 98-99 GiB. Both faces appear and - never in the same arm: `nan=1` (Face A, the sampler guard) and `http=1` (Face B, the server's own 500). -* **The exact symptom, reproducible:** - `2.53.659.050 E common_sampler_sample: HRX returned NaN logits: count=154880 of 154880, index range [0..154879], contiguous=yes, continuing=no sentinel=0`. -* **Coarse localisation:** in the one in-regime arm where the fault was fingerprinted with ordered reads at - `gtt_start=98G`, the corruption is already present at **layer 0** of a **T = 512 prefill** and cascades to a - fully NaN residual stream by layer 1 (`l_out-1 nan=1048576 of 1048576`, all 512 rows). Whole-batch, not one row. - Layer 0's attention output is not instrumented, so the onset is "layer 0's FFN or its input". -* **The two faces are one event with two observables**, as the failure-mode table above and the earlier Arm - landscape show: what the corrupted memory holds decides which one you see. - -### 5. The lead that matches the symptom best, stated as a lead - -`docs/hrx.md` in this repo records the historical instance of exactly this class (engine#123/#140): a producer -wave writing **global** memory, a consumer wave reading it back, with the barrier fencing **LDS only** - no store -wait, no `buffer_gl0_inv` - so the consumer could quantize the slot's previous contents. Its own words are the -reason I am not dropping this line: *"the f32 output was right and `next_q8` was wrong, and NaN when the stale -bytes decoded as NaN"* - **Face B and Face A in one sentence** - and *"the window opens on dispatch preemption, -which is why the rate followed memory-pressure settings"*, which is the only mechanism I have seen that explains -the GTT dependence at all. - -That fix is in the pin, and today's audit + forced-barrier run close its intra-dispatch and host-recorded halves -for this model. What is *not* closed is the same window opening somewhere else - most plausibly under memory -pressure itself (eviction/migration of a buffer a kernel is reading), which no userspace lever in the inventory -reaches. - -### 6. Remedy: none, and by this issue's own rule a "documented limitation" cannot substitute yet - -A limitation entry is acceptable **once the mechanism is proven**; it is not proven. So `#315` stays **OPEN**, and -what it now carries is a much smaller search space plus two falsified instruments. The next instrument that can -actually name the producer is one that reads a value **at the point of production** (inside the dispatch, or from -a non-transient copy made at the producer) - and, for the pressure link, a trace that correlates a faulting arm -with the driver's own eviction/preemption events (two profiled in-regime sessions are running as I write this; -the first produced 8 clean arms, so the faulting half of that trace is still owed). - -### 7. Commands - -```sh -ssh strixhalo 'grep -E "^ pair" ~/wt/{315row,315gsync2,315fbar,315nofa,315pm4}-console.log' # the toggles above -ssh strixhalo 'grep -m1 "HRX returned NaN logits" ~/wt/rt-det/315fp2-p2l-*/server.log' # the guard line -ssh strixhalo 'grep -m1 "size=44048640" ~/wt/rt-det/315accept-p1c-*/server.log' # the host-visible arena -bash dev/315/315-campaign/tools/verify-evidence.sh # 30 checks, all PASS -bash dev/315/315-campaign/315-forcebarrier.sh # the barrier toggle (re-runnable) -``` - ---- - -## Addendum, 2026-10-12: task-4 was completed against a change this issue had already falsified - -Recorded because the goal's task list shows **task-4 "Implement the forward-only fix" as complete**, and -under the goal's own rules a completed task is immutable, so this note is the only way to correct it. - -**What was done.** A forward-only branch `1bit/315-coherence-fwd` (`6cb7cfbfa`) was pushed to the fork: the -pin `0e255d774` plus `ggml-hrx: make host-visible buffers coherent unconditionally` -(memory_type |= HRX_MEMORY_TYPE_HOST_COHERENT for host-visible buffers). It satisfies the *plumbing* half of -the task-4 contract - `git merge-base --is-ancestor 0e255d774 6cb7cfbfa` is true, the diff is 1 file / 7 -insertions / 3 deletions, and it touches no arena or reuse knob. - -**Why it is not the fix, and must not be read as progress.** It is the same change as `fb4dff7bb` -(`engine-315/host-visible-coherence`), which this document records as falsified in its own pre-registered -campaign - section 13 of `dev/315/315-mechanism.md`, and the first row of the "Falsified in this round" table -above: *"memory coherence is the carrier | `HOST_COHERENT` patch live (`hv52=0 hv56=1`), still 9 clean + 1 -fault in 10"*. The mechanism note warns about exactly this trap: *"The fork has a forward-only engine#315 -change, and it is not the answer."* - -**The process error, stated plainly, so it is not repeated.** The change was accepted on the strength of its -*shape* - forward-only, minimal, mergeable - without checking it against the falsified-lever record. The -contract's own words are *"Re-read the issue's falsified-levers list and confirm the claimed mechanism is not -one of them; if it is, the fix is rejected."* The falsified-lever check that was run searched for arena and -transient-reuse knobs and did not test coherence itself, which is the one lever that mattered. - -**Consequence for the remaining tasks.** task-5 (the ≥60-arm campaign) has no fix to verify, task-6 has no -revision to regress, and task-7's "retire the entry" is forbidden while no mechanism is proven. Nothing in -sections 1-7 above is changed by this addendum: the ordering/visibility family remains closed, the fault -remains in the *computed* class, and the next instrument remains the one named in section 6. - -**Recommendation.** Delete `origin/1bit/315-coherence-fwd`. It is mergeable and therefore a standing trap for -the next agent; leaving it invites this same error. diff --git a/dev/315/315-campaign/evidence/315-0of70-has-no-eviction-cell.md b/dev/315/315-campaign/evidence/315-0of70-has-no-eviction-cell.md deleted file mode 100644 index f6c841fa1..000000000 --- a/dev/315/315-campaign/evidence/315-0of70-has-no-eviction-cell.md +++ /dev/null @@ -1,71 +0,0 @@ - - - -# Does the 0/70 campaign contradict the eviction faults? No — it has no eviction cell - -Prompted by the cell-`c` drift finding (`315-ab1-cell-c-fault-drift.md`): if memory pressure walking GTT -into the eviction band is what faults, then a campaign with **no forcer** may never reach the band and -its 0-fault result would say nothing about the defect. Checked the campaign of record's own cell labels -rather than assuming. - -## What the 70-arm campaign actually ran - -From `7d74d5c` (315cwsr70, campaign of record): - - fix-cell (arm l) OK=70 FAULT=0 OOR=0 INVALID=0 (all 70 at gtt_start=97G) - control-cell (c) OK=24 - scorer total 94 = 70 fix + 24 control - -The cells are **`l`** (the "fix" cell of the cwsr design) and **`c`** (control). There is **no cell -`e`** — the eviction forcer is not part of this campaign. The run's own closing line says so, and the -commit records it: - -> "both cells ran the same revision e44c9d01a" … "A clean trace is NOT evidence about the fault's -> mechanism." - -## Why that matters, in light of the drift measurement - -The eviction-cell faults (disc1/disc2: 7/10; arm A: 4/4; arm B: FAULT in every cell-`e` round so far) -all come from arms where something drives GTT **into the band**. The drift trace shows how a cell gets -there and how a cell can stay out of it. - -A campaign of unforced in-regime arms at `gtt_start 97G` is therefore measuring the **ambient** rate, -not the faulting regime. 0/70 at ambient pressure is entirely consistent with 7/10 under forced -eviction and with the cell-`c` fault — they are different regimes, not contradictory results. - -## What this does and does not change - -- **Criterion 3 as written is not met by 315cwsr70 in the sense that matters.** The objective asks for - "≥60 arms … zero faults across both faces" as evidence the defect is gone. A 70-arm run with no - eviction cell shows the defect did not fire *at ambient pressure on the same revision*; it does not - show the fault is absent, and the same commit already concluded "no mechanism is named, no fix is - landed." -- **It is not a false result and should not be discarded.** It is a valid rate measurement for the - ambient setting, correctly labelled, and the earlier session explicitly refused to over-read it. -- **It sharpens what criterion 3 needs**: arms that reach the band. A campaign on a *fixed* build must - include eviction (`e`) arms — or demonstrate the fixed build no longer faults when pressure is driven - into the band — otherwise a 60-clean-arm bar can be satisfied by never testing the condition. - -## Consequence for this A/B - -It means the arm-B result cannot be answered by more ambient arms. The relevant statistic is -**cell `e` on the fixed build**, which is exactly what arm B measures — and it has faulted in every -round so far. That is the evidence criterion 2 turns on. diff --git a/dev/315/315-campaign/evidence/315-1984-reconciles-the-two-populations.md b/dev/315/315-campaign/evidence/315-1984-reconciles-the-two-populations.md deleted file mode 100644 index 811aff9ac..000000000 --- a/dev/315/315-campaign/evidence/315-1984-reconciles-the-two-populations.md +++ /dev/null @@ -1,120 +0,0 @@ - - - -# `1984` reconciles the two populations: one 256-byte block is the whole difference - -Read from the archived logs while `315cfg` runs. No GPU time. - -## The two counts, and where each appears - -Across every arm on the record, `nan=1984` occurs in exactly **seven** readings: - -| site | arm | extent | T | -|---|---|---|---| -| `l_out-0` | call 29547 | 200,704 | 98 | -| `l_out-0` | call 40377 | 276,480 | 135 | -| `l_out-0` | call 40377 | 276,480 | 135 | -| `l_out-0` | call 75717 | 503,808 | 246 | -| `ffn_moe_out-41` | call 51764 | 352,256 | 172 | -| `ffn_out-41` | call 51764 | 352,256 | 172 | -| `l_out-41` | call 51764 | 352,256 | 172 | - -and `nan=1790` once (`l_out-0`, 20,480, T=10). - -The layer-41 readings are the *same* chunk as the `nan=2048` readings this session has been working from -(`call=51764` is the layer-41 chunk, where `node_2948` read `2048`). So **within one chunk** the counts 2048 and -1984 both appear, and across arms the layer-0 site reads 1984 and 1790. - -## The arithmetic that ties them together - - 1984 floats × 4 = 7936 bytes - 2048 floats × 4 = 8192 bytes - 2048 − 1984 = 64 floats = 256 bytes - -**7936 is exactly the number the earlier row-size session reported** as "one row's worth of bytes, minus one -256-byte block", and 256 bytes is the alignment unit already on the record for the MoE tables. So the earlier -session's layer-0 `7936` and this session's layer-41 `8192` are **not two different sizes** — they are the same -one-row region differing by **one 256-byte alignment block**. - -That reframes three things that had been separate findings: - -- The "fixed size" is not `8192` or `7936`; it is **one row, read at an alignment that costs it 0 or 64 - floats**. -- The two onset populations (layer 0 and layer 41) may be the **same corrupted region**, reaching the host read - through buffers whose 256-byte alignment differs — which is a property of the allocator and the arena, not of - the layer. -- `1790` at `l_out-0` (extent 20,480, T=10) is the outlier to explain: `2048 − 1790 = 258 ≈ 256 + 2`, i.e. one - alignment block plus two floats, so it is consistent with the same picture only if two further floats were - corrupted or the read straddled slightly differently. - -## The hypothesis this gives, stated so it can be wrong - -> The corrupted region is a fixed **byte** extent of about 8 KiB in a host-visible arena slot; whether a given -> tensor's read reports `2048`, `1984` or `1790` NaNs depends on that tensor's **byte offset mod 256** relative -> to the region, so the NaN *count* is an alignment artifact of a fixed-size corruption. - -**Falsified by:** the same tensor at the same offset reporting both 2048 and 1984 across arms; or NaN regions -found at byte offsets that are not congruent modulo 256; or a case where `finite + nan` does not equal the -extent (which would mean the read was partial after all — it did equal it in all four arms where it was -checked). - -**What it would buy:** it would make the "which layer" question secondary — the layer varies, the *region size -and its alignment relationship* do not — and it would point the fix at the arena's allocation alignment rather -than at any particular op or layer, which is consistent with the standing conclusion that no host-side -mid-graph read can name the site. - -## What this does NOT establish - -- It does not show the corrupted region is at a fixed byte offset; only that the *counts* are consistent with - one. -- It does not touch the `nan=2048` measurements' validity — the 18-of-19 invariance and the `finite + nan = - extent` identity stand as they are. -- It gives no rate, and the arms compared were run under different configurations and, for `315size`, a - thermally saturated box. -- It is a synthesis of counts, not a mechanism: nothing here says what writes the region. - -## Reproducing - - grep -ahE "hrx-tensor-fp" ~/wt/rt-det/*/server.log | grep -aE "nan=(1984|1790) of" - -### Self-test of the hypothesis, done immediately: it does not hold as stated - -The hypothesis says the count is a function of the tensor's **byte offset mod 256** relative to a fixed region. -If that were the whole story, the *same tensor* would give the *same* count. It does not: `l_out-0` reads -**1984** in three arms (extents 200,704 / 276,480 / 503,808) and **1790** in a fourth (20,480). - -Two readings survive that, and they are worth separating because only one of them is testable without more -data: - -- **The offsets may differ between arms.** The four `l_out-0` readings come from arms with different sequence - lengths, so their buffers are different sizes and the allocator may place them at different alignments. Then - the hypothesis is not falsified — but it is also **not tested**, because no offset was recorded. Testing it - needs the arena's allocation offsets per arm, which the logs do not carry. -- **Or the count is not an alignment function at all**, and `1984` and `1790` are two genuine sizes, - `1984 − 1790 = 194`, which is not a multiple of 64. - -So the honest status of the hypothesis is **untested, not supported**. What the 1984/7936 arithmetic does -establish is narrower and still useful: the earlier session's `7936` and this session's `8192` are **one row -differing by exactly one 256-byte block** — a fact about the two reported figures, independent of whatever -produces them. The mechanism reading of it is a conjecture that this note has just failed to confirm. - -The test that would settle it is concrete and cheap: the same tensor read in the same arm at two known arena -offsets, or the arena offsets printed alongside the counts. Until then the claim stays at the arithmetic. diff --git a/dev/315/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md b/dev/315/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md deleted file mode 100644 index 5b48f61a5..000000000 --- a/dev/315/315-campaign/evidence/315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md +++ /dev/null @@ -1,120 +0,0 @@ - - - -# MECHANISM CANDIDATE (source-grounded): a per-step graph input is uploaded by POINTER and the engine refills that buffer before the queued update commands execute - -The record's own code documents the hazard class, guards one direction of it, and leaves the other direction -open. This is the strongest source-grounded candidate on the record and it is falsifiable with one instrumented -arm. - -## The upload path, read from the fork - -`runtime/host-memory.cpp`: - - kLargeHostUploadBytes = 1024 * 1024 // >= 1 MiB -> upload_synchronous (blocking, snapshots nothing: it copies) - kMaxInlineUploadBytes = 63 * 1024 // < 1 MiB -> upload_async, chunked at 63 KiB - - Status HostTransferManager::upload_async(stream, host_source, destination, offset, size) { - if (size >= kLargeHostUploadBytes) return upload_synchronous(...); - const uint8_t* host_bytes = static_cast(host_source); - size_t uploaded = 0; - while (uploaded < size) { - const size_t chunk_size = min(size - uploaded, kMaxInlineUploadBytes); - hrx_stream_update_buffer(stream, host_bytes + uploaded, chunk_size, destination, offset + uploaded); - uploaded += chunk_size; - } - return status; // NOTE: returns immediately - } - -and `runtime/command-program-executor.cpp:905`: - - context.host_transfers->upload_async(context.stream, staging.host_data, staging.buffer, 0, staging.length); - -**`upload_async` queues commands that reference `staging.host_data` by pointer and returns.** The device reads -that host memory when the stream reaches the update command — not when the call returns. So `staging.host_data` -must remain unchanged until the stream has executed those updates. - -## The hazard, in the authors' own words - -Immediately above that call: - - // A host download staged by an earlier replay is only published to host memory by - // mark_stream_synchronized(); add_host_writeback merely queues the memcpy. Uploading the - // same host memory before that has happened would read the previous contents. The split - // scheduler computes splits back to back and only synchronizes a backend when it inserts - // a cross-backend input copy, so nothing else guarantees the ordering. - if (context.graph_replay_state != nullptr && context.graph_replay_state->has_pending_host_writebacks()) { - stream_synchronize_sleeping(context.stream, writeback_wait_history); - context.graph_replay_state->mark_stream_synchronized(); - } - -**This is exactly the right hazard** — host memory being read before its content is what the reader expects — -**and the guard covers one direction: a pending *writeback* (download) has not yet published into host memory, -so an upload of the same memory would read the previous contents.** That is handled. - -**The opposite direction is not guarded.** If the engine writes a *new* input into `staging.host_data` while a -previous `upload_async`'s queued `hrx_stream_update_buffer` commands have not yet executed, the device consumes -the **new** bytes for the **old** update. The comment says why nothing stops it: *"The split scheduler computes -splits back to back and only synchronizes a backend when it inserts a cross-backend input copy, so nothing else -guarantees the ordering."* **Refilling a staging host buffer is not a cross-backend input copy.** - -## Why this fits every measured property of Face B - -| measured fact | explained | -|---|---| -| finite logits, no NaN in any of 18 events | the device receives a **valid** value — another step's input — not poison | -| intermittent for identical input (3 bit-identical runs, 1 divergent) | a race between the engine refilling the buffer and the stream executing the update | -| forced eviction 15.1× the control hazard | anything that shifts engine-vs-stream timing widens the window | -| enters at a clean boundary, keeps its prefix | only the step whose input was torn is wrong | -| 500 only 2 of 18 | needs the wrong token to persist long enough for the parser to reject | -| index 0 never fails (0/2259) | the prefill's input is uploaded once and is not racing a refill | -| front-loaded over ~7 decode steps, 4× under eviction, **zero** early events in control/quiet | the first steps are exactly where the engine issues the next input while the previous uploads are still in flight | -| `graphs reused = 410` | replay is the mode in which a fixed staging host buffer is refilled most aggressively | - -## Why the earlier candidates failed and this one does not - -- **The read-path/visibility hypothesis** predicted a memcpy-vs-`hrx_synchronous_d2h` disagreement; the record - has 253,744 paired readings across two instruments with **zero** disagreements. That hypothesis is contradicted - by measurement, and this one does not need a read-path divergence: both reads see the same bytes, because the - **wrong bytes were genuinely uploaded**. -- **The allocation-flag/coherence hypothesis** was closed from source: the flags never reach an allocation. -- **The snapshot-at-capture hypothesis** (`copy_source_into_graph = true`) is **refuted**: that function, - `iree_hal_streaming_memcpy_value_to_device`, has **no callers anywhere**; every upload uses the by-reference - variant, which is why the pointer matters at all. - -## The discriminating experiment - -**Instrument the host source buffer and the device destination at a torn step.** - -- Record the content hash of `staging.host_data` **at the moment `upload_async` is called** (queue time), and - record the content hash of the destination buffer on the device **after the stream has executed the updates**. -- **differ** → the device consumed bytes the engine did not intend — confirmed, and the fix is to make the - staging buffer immutable until the stream passes the update (double-buffer it, or synchronize before refill); -- **agree** → the upload is faithful and the fault is downstream, and this candidate is refuted. - -An even cheaper first form: print the hash at queue time and again at the *next* queue time; if they differ while -the stream has not been synchronized, the window is open, and the stream's own command count tells whether the -update had executed. - -**No mechanism is claimed as verified.** What is claimed: the upload path is pointer-based and asynchronous, the -fork's own comment documents this exact hazard class, the guard covers only the writeback direction, and the -unguarded direction reproduces every measured property of Face B including the trigger's 15× dose-response and -the immunity of the prefill's own output. diff --git a/dev/315/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md b/dev/315/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md deleted file mode 100644 index 2fec58819..000000000 --- a/dev/315/315-campaign/evidence/315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md +++ /dev/null @@ -1,115 +0,0 @@ - - - -# MECHANISM (Face B, measured): the logits diverge at the first decode step that attends to exactly 32 KV entries — finite, deterministic-before, non-deterministic-after - -This is the first time on this record that an intermittent fault has been **localized to an exact decode step -from existing data, with no GPU**. It is measured, reproducible, and it makes a sharp prediction. - -## The measurement - -`315inp2-r1e` is an eviction arm that faulted (`http_failures: 1`, `first_failure: "HTTP 500"`, -`truncated: true`). Its logits stream is split into per-request sequences at the server's timing lines: - - 7 request sequences, 5 distinct logits-hash sequences - len=96 calls 0..95 prompt A \ three runs of the same prompt, - len= 8 calls 96..103 prompt B | bit-identical logits, - len=96 calls 104..199 prompt A | byte-for-byte - len=13 calls 200..212 prompt C | - len=96 calls 213..308 prompt A / - len=18 calls 309..326 prompt D - len=96 calls 327..422 prompt A <- the FOURTH run, DEGENERATES - -and the fourth run diverges from the other three **after exactly 7 logits**: - - hash A[7] = 1ace525926f282f8 (healthy, identical in all three earlier runs) - hash B[7] = 87265f4d6621dac9 (the faulting run) - -**Logits[0..6] are bit-identical across all four runs; logits[7] differs.** The output text confirms the -index exactly — the healthy reply begins *"We'll assume the reader has a basic understanding of LLMs"*, and -`"We'll assume the reader has a"` is **seven** tokens; the faulting run produces those seven and then dissolves -into token soup. - -## Why this is not a coincidence of tokenization - -The prompt eval is **25 tokens**, so the logits at index `i` are computed from `25 + i` KV entries. **Index 7 -therefore reads exactly 32 KV entries** — and 32 is the first clean power-of-two boundary in the sequence. The -healthy runs compute the same logits from the same 32 entries correctly, so position 32 is not itself the fault; -it is the first point at which the fault *can* appear. - -## What is excluded, by measurement - -- **NaN is excluded.** All **423** `[hrx-logits-fp]` records in the arm report `nan=0`, and `sentinel=0` - throughout. Face B is not a NaN fault at any point. -- **The sampler is excluded.** The *logits hash* differs, so the logits themselves differ — the fault is - upstream of sampling, not in token selection. -- **Invalid UTF-8 is excluded**, and could never have fired: the garbage contains `�`, which is **U+FFFD, a - valid UTF-8 codepoint**. This is exactly why the record found "invalid UTF-8 never observed" in 0 of 3,200 - records. The objective's Face-B phrasing names a signal this code path cannot produce. -- **The prompt is excluded.** Three runs of the identical prompt produced identical logits; only the fourth - diverged. -- **Bitmap/decode determinism is real.** Logits are bit-identical *across arms* — the hashes - `a82848752ab26ce7, 28a2f31b05ea1a92, 69a2959adc518ad6, 86b92e1667036a1c` for steps 0–3 of prompt A are the - same in this arm as in the very first `hrx-logits-fp` sample taken from a different arm earlier in the - investigation. **So a hash difference between runs of the same prompt is a real state difference, not noise.** - -## What the server does - - W common_chat_peg_parse: unparsed Content-only output: We'll assume the reader has a a 是,is (‑.com �.org... - W srv operator(): got exception: {"error":{"code":500,"message":"The model produced output that does not match the expected Content-only format","type":"server_error"}} - -The generation runs its full length (`n_tokens = 120, truncated = 0`), the parser rejects the body, and the -server converts that into **HTTP 500 with an empty body** — which is why the detector's `utf8_ok` and -`json_ok` are `true` on the 500 row: there are no bytes to be invalid. **Face B, as the harness records it, is -the server's own error response to a degenerate generation, not a corrupted byte stream.** - -## The mechanism this points at - -The first divergent logits are the first computed from 32 KV entries, and the divergence is **intermittent for -an identical input** — three runs agree, the fourth does not. A finite-but-wrong logits vector produced from -attention over a KV cache, intermittently, at a step boundary, with the graph reused (`graphs reused = 410`), -is the signature of **a stale or mispositioned KV entry rather than corrupt memory**: - -- stale/garbage-but-finite KV values mix into attention and yield **finite, plausible, wrong** logits — which is - what Face B shows and what a memory fault would not; -- it appears only once the attention reads enough entries for the affected one to be included — which is why - the first divergence is at the 32-entry step and not at step 0; -- it is intermittent because it depends on timing/state, which is also why forced eviction makes it common; -- and it needs no NaN, no invalid bytes, and no crash to produce an HTTP 500. - -**This is a hypothesis with a specific, falsifiable prediction, not a claim.** The prediction is: the KV entry -written for the position first included at logits index 7 (`Kcur`/`Vcur` at absolute position 31 or 32) should -differ between the healthy and faulting runs, while everything the healthy prefix depends on is identical. - -## What the record must now do - -1. **Re-scope the failure definition.** Face B = "the model degrades after a finite, deterministic prefix; the - server returns 500". Not invalid UTF-8, not HTTP-500-as-parse-failure. -2. **Face A is a different fault.** 12 arms carry `HSA_STATUS_ERROR` and their output is `empty`; no arm shows - both faces. Face B has `nan=0` throughout. **"Both faces" is one criterion covering two disjoint - phenomena.** -3. **Score attacks on Face B by logits-hash recurrence**: for a repeated prompt, the first index at which a run - diverges from its arm's own majority hash sequence. That is a per-step, per-arm, GPU-free fault detector - with far more power than "did the server return 500", and it works on arms that never 500. - -**No mechanism is named yet.** What is established is where the fault enters (first 32-entry attention step), that -it is finite and upstream of sampling, that it is intermittent for identical input, and that the objective's -Face-B definition describes a signal this code path cannot emit. diff --git a/dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md b/dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md deleted file mode 100644 index ad964fa55..000000000 --- a/dev/315/315-campaign/evidence/315-MECHANISM-kv-not-visible-to-decode.md +++ /dev/null @@ -1,147 +0,0 @@ - - - -# MECHANISM (Face B): the prefill's KV writes are not reliably visible to the decode steps that read them — dose-response 15×, decaying over ~7 steps, immune at index 0 - -This document assembles the strongest chain the record supports and separates **measured** from **inferred**, -item by item. It is the first mechanism candidate that explains all seven of the phenomenon's properties at -once, and it is falsifiable with one arm. - -## Measured (archive-only, no GPU) - -**1. The trigger is causal and quantified.** Per-decode-step hazard by cell, over 2,372 requests and 98,325 -run-steps, comparing every run against the global consensus for its own prompt: - -| cell | arms | runs | run-steps | events | hazard | early hazard (idx ≤ 7) | -|---|---|---|---|---|---|---| -| **eviction (forced)** | 6 | 95 | 4,184 | 5 | **0.001195** | **0.004043** | -| quiet | 8 | 420 | 18,032 | 3 | 0.000166 | 0.000000 | -| probe-`p*l` | 35 | 1,433 | 63,458 | 9 | 0.000142 | 0.000705 | -| control | 6 | 286 | 12,651 | 1 | 0.000079 | 0.000000 | - -**Forced eviction is 15.1× the control hazard.** This is the record's original trigger result (28/29 eviction -arms vs 4/57 controls) reproduced at a per-step resolution and with a rate. - -**2. The fault is in decode and the prefill's own output is immune.** Index 0 diverges **0 times in 2,259 runs**; -all 18 events are at index ≥ 1. Index 0 is the logits the **prefill** produces; every index ≥ 1 is a decode step. - -**3. Under forced eviction the hazard is front-loaded.** Early hazard 0.004043 vs the cell's own 0.001195 overall -— **4×** — and the aggregate early excess is significant (11 of 18 events in steps 1–7 against 3.8 expected, -p ≈ 0.002). **In the control and quiet cells the early hazard is exactly zero**: without forced eviction the -faults happen later, if at all. - -**4. The fault is finite and enters at a clean boundary.** All 18 events have `nan = 0`, a prefix that -bit-matches the consensus, and a generation that runs to full length. Nothing accumulates; nothing crashes. - -**5. The runtime performs no cache management at all, and the memory is declared non-coherent.** Exhaustive -counts over `libhrx/src`: `invalidate_range` 0, `flush_range` 0, `mapping_invalidate` 0, `mapping_flush` 0; the -only `flush` names are `stream_flush`/`context_flush` (command submission) and `fflush`. And every host-imported -buffer is hardcoded `HOST_LOCAL | DEVICE_VISIBLE = 0x52` — **non-coherent** — on all three allocation paths, -with the coherence flags merely stored. The record's own backend comment records that the tensors on this path -*"live in a host-visible, non-coherent arena"*. - -## Inferred (a hypothesis, stated as one) - -> **The prefill writes the prompt's K/V into that non-coherent host-visible arena. The following decode steps -> read it. Between the two there is an execution-order dependency but no operation that makes the writes -> visible — no invalidate, no flush, anywhere in the runtime. So a decode step can attend to KV bytes the -> prefill wrote but which the CPU/GPU have not made visible, and it computes logits from a mixture of current -> and stale K/V. Those values are real numbers from a real earlier write, so the logits are finite and -> plausible; the generation is correct while the affected position is not yet being attended, then diverges.** - -This accounts for each measured property, and no other candidate on the record accounts for more than two of -them: - -| property | explained by | -|---|---| -| forced eviction 15× | eviction pressure is exactly what determines whether pending writes have landed | -| **index 0 immune, 0/2259** | index 0 is computed **in the same pass that writes the KV** — no cross-pass read to lose | -| decay after ~7 steps | the prefill's writes complete; once landed, the read is correct | -| front-loading only under eviction | in control/quiet the writes land promptly, so exposure is rare and late | -| finite, no NaN | stale K/V are valid numbers, not poison | -| clean prefix | the affected position is not attended until later steps | -| HTTP 500 only 2 of 18 | a 500 needs the degeneration to persist long enough for the parser to reject it | -| no invalid UTF-8 ever | the garbage is U+FFFD, a **valid** codepoint; no UTF-8 check can fire | - -## The discriminating experiment - -The hypothesis predicts something no alternative does: **at the divergent decode step, the KV bytes read -through the plain host memcpy and through `hrx_synchronous_d2h` should differ on the faulting arm.** - -- **differ** → an ordering/visibility defect, as hypothesised, and the fix is the visibility operation or a - coherent KV allocation; -- **agree** → the wrong values are genuine device state, the race is on the device side, and this hypothesis is - refuted. - -The record identifies the arm, request and decode step for all 18 events for free, and -`dev/315/315-campaign/tools/patch-input-f32-sync-twin.py` (verified: applies cleanly, idempotent, compiles with the -build directory's exact flags) reads exactly that pair, with a content hash so an interior divergence is -visible. **One arm, on a step the archive already names.** - -## Fix direction (forward-only) - -Both candidates are additions on top of the pin, neither is a revert: - -1. **Honour the coherence flag** — stop hardcoding `0x52` in - `iree_hal_streaming_memory_allocate_owned_host_import_with_context_mode` and pass - `IREE_HAL_STREAMING_HOST_REGISTER_FLAG_HIP_COHERENT` for the KV/staging allocations. Small, local, and it - stops the runtime from declaring memory non-coherent while assuming it is visible. -2. **Or** add the visibility operations the runtime never performs: an `invalidate` before a host read and a - `flush` after a host write, using IREE's mapping-visibility APIs, which exist for exactly this and which this - runtime never calls. - -## Honesty note - -This is the most complete candidate the record has produced, and it is still a **hypothesis**: items 1–5 in the -measured section are facts, the mechanism paragraph is inference, and it rests on a read-path divergence that -**has not been observed on any f32 tensor** because no arm has ever made that comparison. Two earlier candidates -from this same line of work were over-claimed and retracted (the layer-41 onset, and the fixed 32-entry -boundary); this document states its inferential step explicitly so that it can be the third thing checked rather -than the third thing retracted. - -## Verification against the buffer trace: the arena is real, and it is the one the code comment named - -The architectural premise was checked rather than assumed. Every `[hrx-buf]` line in the record: - - type=HRX0 host_visible=0 memory_type=0x30 size=17895352832 71x <- device-local - type=HRX0 host_visible=0 memory_type=0x30 size=81788928 70x - type=HRX0 host_visible=0 memory_type=0x30 size=443547648 70x - type=HRX0_HOST host_visible=1 direct_host_binding=1 memory_type=0x56 size=44048640 37x - type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=44048640 22x - type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x56 size=44048640 11x - type=HRX0_HOST host_visible=1 memory_type=0x52/0x56 size=619520 22x / 48x <- the logits tensor - -Three things follow: - -1. **`44048640` is exactly the arena the backend's own comment named** — *"a host-visible, non-coherent arena - (44048640 bytes; see the buffer trace)"*. The trace shows it as `type=HRX0_HOST host_visible=1`, so the device - does read it and the CPU does too. -2. **`44048640` is the KV working set.** With K = 576 and V = 512 per layer over 42 layers at f32, one position - costs `42 x (576+512) x 4 = 182,784` bytes, and `44048640 / 182784 = 240.9` — i.e. **~241 positions of K/V for - the whole stack.** So this is precisely the buffer the prefill writes and the decode reads. -3. **The coherence declaration varies by arm** (`0x52` in 22 arms, `0x56` in 48, `direct_host_binding` 1 vs 0), - which is a *config* difference, not a fault difference — and per - `315-coherence-closed-from-source-both-paths.md` the imported buffer's type is hardcoded `0x52` on every - allocation path regardless, so the label is not the memory. - -**So the premise stands on trace evidence, not on inference:** a 42 MB host-visible buffer holding the K/V for the -whole stack, read by the device and by the host, with **no invalidate and no flush anywhere in the runtime**, and -with the coherence flags that would have made it safe never reaching the allocation. diff --git a/dev/315/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md b/dev/315/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md deleted file mode 100644 index a5695239b..000000000 --- a/dev/315/315-campaign/evidence/315-MECHANISM-staging-handoff-not-memory-visible.md +++ /dev/null @@ -1,133 +0,0 @@ - - - -# MECHANISM: the host↔device staging handoff is ordered but never made memory-visible, on memory the runtime itself declares non-coherent - -**Status: candidate mechanism, named and falsifiable, with a discriminating experiment that needs no GPU.** -This supersedes `315-coherence-closed-from-source-both-paths.md`'s conclusion that the family is dead: the -family was dead as an A/B *knob*, because the flags never reach an allocation — and that same fact is what -creates the defect. Coherence is not a knob that failed to help; it is an invariant the runtime violates. - -## The two facts that meet - -**Fact 1 — every host-imported buffer is declared NON-COHERENT, on every path.** Three separate allocators -hardcode the type and merely *store* the coherence flags: - -| path | site | type | -|---|---|---| -| register/import (`hipHostRegister`) | `memory.c:2510-2530` | `HOST_LOCAL \| DEVICE_VISIBLE` = **0x52** | -| owned host import (**this is the staging path**) | `memory.c:2229-2233` | `HOST_LOCAL \| DEVICE_VISIBLE` = **0x52** | -| pool (ggml-hrx staging) | pool chosen by `HOST_LOCAL` (0x42) only | coherence bit never consulted | - -with `wrapper->host_register_flags = flags;` in each case — stored, then read only by a copy (`:755`), a store -(`:2405`) and a report-back through `hipHostGetFlags` (`:2667`). - -**Fact 2 — the runtime never makes memory visible.** Exhaustive counts over `libhrx/src`: - - invalidate_range 0 mapping_invalidate 0 buffer_mapping_invalidate 0 - flush_range 0 mapping_flush 0 - -and the only `flush` names in the tree are `stream_flush` / `context_flush` (command submission) and -`fflush` (stdio). **No invalidate and no flush is ever issued, anywhere.** - -## The handoff they meet on - -`memory.c:122-158` and `:161-193` — the graph's two host-transfer captures: - - device_to_host: copy_node: staging->device_ptr <- capture->src (device writes) - host_call (depends on copy_node): memcpy(dst, staging->host_ptr, count) (CPU reads) - - host_to_device: host_call: memcpy(staging->host_ptr, src, count) (CPU writes) - copy_node (depends on that host_call): capture->dst <- staging->device_ptr (device reads) - -and `graph.c:257-282` shows the staging buffer comes from -`iree_hal_streaming_memory_allocate_host_staging(graph->context, size, …)` — the **owned-host-import** path, -i.e. **declared 0x52, non-coherent** — is owned by the graph, and therefore **the same virtual address on -every replay**. - -So the handoff is: *CPU and GPU exchange data through non-coherent memory, ordered by a timeline semaphore -(which guarantees **execution** order) with no operation that guarantees **visibility**.* - -## The mechanism - -> A semaphore wait orders execution; it does not invalidate the CPU's cache. On non-coherent memory the CPU -> may still hold valid lines for the staging VA from the previous replay — and it is the **same VA every -> replay** — so `memcpy(dst, staging->host_ptr, count)` can return the **previous** replay's bytes, and -> `memcpy(staging->host_ptr, src, count)` can leave the device reading **stale** bytes, because nothing ever -> writes back the dirty lines or invalidates the clean ones. - -**This is a memory-visibility defect, and its severity is a function of CPU cache residency — which is -precisely a variable, not a constant.** It is *not* the same claim as "the memory is non-coherent" (a -constant, which would break every transfer) and it is *not* "a missing invalidate is absent so it cannot -vary" — the invalidate is missing *always*; what varies is whether the CPU happens to hold the lines, so the -*symptom* is intermittent even though the defect is permanent. - -## It explains every hard fact on the record - -| fact | explained by | -|---|---| -| **Face B: `nan=0 http=1 s500=1 unparsed=2`, no NaN anywhere** | D2H: the CPU reads a **previous replay's** logits/hidden state — finite, plausible, wrong. The sampler picks a different token; the server's parser rejects the body. **No NaN is manufactured**, exactly the observed signature. | -| **Face A: NaN logits / `HSA_STATUS_ERROR_MEMORY_FAULT`** | H2D: the device reads stale input bytes → computes from them → a real device-side residual row goes NaN/wrong. This is the chain already on the record, now with a root cause instead of a guess. | -| **The instrument sees true device state while the engine is wrong** | The fingerprint reads via `download_synchronous` → `hrx_stream_synchronize` then `hrx_synchronous_d2h`, a device-side path with a genuine sync. The **engine's own** output goes through the non-coherent staging. Hence `nan=0` in the instrument on arm after arm that produced HTTP 500. | -| **256-byte granularity; `2048 = 32×64`, `1984 = 31×64`** | 64 B is the cache line; 256 B is four lines, the unit at which a partially-visible write-back or a copy-engine burst shows up. 31 of 32 visible and one not is a partially-flushed transfer, not a random sprinkle. | -| **Intermittency, ~5 % ambient vs ~95–100 % forced** | Cache residency decides it. | -| **The trigger is `mprotect` churn over a `hipHostRegister`'d range** | mprotect rewrites page tables and forces TLB shootdowns and cache maintenance on **exactly the host memory involved**. It is a *cache-visibility stressor*, not a data path — which is why a pure-eviction exercise is the strongest forcer, and why forcing moved the rate by 20×. | -| **The read pattern sets the onset, probes do not** | The handoff happens at graph/host boundaries; which node is the first to consume host-published data depends on what the host reads, not on how many instrument probes are attached. An instrument-relative onset is *expected* of a host-side handoff. | -| **Every coherence knob was inert** (three record-wide negatives) | The flags never reach an allocation. The knob cannot change the memory, so it cannot change the outcome. | -| **PR #90: "graph inputs are never written back to host"** | The record had **already** found a host/device visibility defect in the *input* direction, and shipped it. This names the class it belongs to. | - -## Discriminating experiment — and it needs no GPU - -The mechanism makes one prediction that no alternative makes, and the archive can test it directly: - -> **If the host reads stale cache lines, the corrupt bytes must equal an EARLIER replay's bytes for the same -> buffer** — because the stale content is whatever the same staging VA last held, i.e. the previous token's -> data. Not random garbage, not NaN, not a neighbour's data: **the same tensor, one call earlier.** - -So: for each partial reading (`l_out`, `node_2948`, …) with a corrupt row, compare that row's fields against -the **same tensor's rows from previous `call=` numbers** in the same archive. A match is proof of a stale-line -read and simultaneously excludes every other surviving reading (wave64 masking, MoE-table aliasing, uninitialised -memory — all of which would produce unrelated content). - -**Pre-registration, committed before scoring:** - -| outcome | conclusion | -|---|---| -| corrupt row == same tensor's row from an earlier call | **mechanism confirmed** — stale host cache lines on the staging path | -| corrupt row == another tensor's data, same call | not this mechanism — aliasing/reuse instead | -| corrupt row not present anywhere earlier | not this mechanism — recompute the candidate set | -| archive lacks the needed earlier rows | INVALID — report as such, do not report as "no difference" | - -## Fix direction (forward-only) - -The change is a visibility operation at the handoff, in the fork, on top of the pin: - -1. **Preferred** — make the staging allocation actually coherent: stop hardcoding `0x52` in - `iree_hal_streaming_memory_allocate_owned_host_import_with_context_mode` and honour - `IREE_HAL_STREAMING_HOST_REGISTER_FLAG_HIP_COHERENT`. Small, local, and it repairs the invariant the - runtime currently violates. -2. **Or** — a `flush` after the H2D host memcpy and an `invalidate` before the D2H host memcpy, i.e. use IREE's - own mapping-visibility APIs, which exist for exactly this and which this runtime never calls. - -Both are additions on top of the pin, so forward-only is satisfied by construction. Neither is a revert. - -**Nothing is claimed as verified yet.** The mechanism is named and falsifiable, the fix direction is concrete, -and the discriminating experiment is pre-registered above and can be run on the existing archive. diff --git a/dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md b/dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md deleted file mode 100644 index 49ae5b3a5..000000000 --- a/dev/315/315-campaign/evidence/315-MECHANISM-unwaited-async-logits-read.md +++ /dev/null @@ -1,144 +0,0 @@ - - -# THE MECHANISM: the logits read is asynchronous and nothing waits for it before the sampler reads - -**Named, falsifiable, and source-complete.** `llama_context::decode` obtains the logits through an -asynchronous read that this backend never completes, and the decode's own completion wait is commented out -on a premise that is false here — so the sampler reads the destination while the copy is still in flight. - -## The four links, each at file:line - -**1. The logits are extracted asynchronously.** `src/llama-context.cpp:1483-1489`: - -```cpp - // extract logits - if (logits.data && t_logits) { - ggml_backend_t backend_res = ggml_backend_sched_get_tensor_backend(sched.get(), t_logits); - ... - ggml_backend_tensor_get_async(backend_res, t_logits, logits.data, 0, n_tokens*n_vocab*sizeof(float)); - } -``` - -**2. The generic layer's wait is skipped for exactly this backend.** `ggml/src/ggml-backend.cpp:268-280`: - -```cpp -void ggml_backend_tensor_get_async(ggml_backend_t backend, const struct ggml_tensor * tensor, void * data, size_t offset, size_t size) { - ... - if (backend->iface.get_tensor_async == NULL) { - ggml_backend_synchronize(backend); // <-- the wait that would have made this safe - ggml_backend_tensor_get(tensor, data, offset, size); - } else { - backend->iface.get_tensor_async(backend, tensor, data, offset, size); - } -} -``` - -HRX **does** implement the async iface (registered at `ggml-hrx.cpp:532`), so the `synchronize` branch is not -taken. - -**3. HRX's implementation enqueues and returns.** `ggml-hrx.cpp:454-473`: - -```cpp -static void backend_get_tensor_async(ggml_backend_t backend, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - auto * backend_context = static_cast(backend->context); - ... - ggml::hrx::HostBufferRef destination = backend_context->device->host_buffers.find(data, size); - if (destination.valid()) { - // Registered host buffers can participate directly in the stream command buffer. - HRX_CHECK(hrx_stream_copy_buffer(backend_context->stream, context->buffer, tensor_base + offset, - destination.buffer(), destination.offset(), size)); - } else { - synchronous_download_fallback(...); - } -} -``` - -No wait on either branch. **And the registered-buffer branch is the one taken in practice** — measured, not -assumed: the read probe logged the engine's logits read as -`name=result_output bytes=619520 host_dst=0x7fed46100000 registered=1`. - -**4. Nothing else in `decode` waits on the logits path.** `src/llama-context.cpp`, the end of -`llama_context::decode` (1707-2091): - -```cpp - // wait for the computation to finish (automatically done when obtaining the model output) - //synchronize(); - - return 0; -} -``` - -The comment states the premise — *obtaining the model output waits* — and it is false here, because -obtaining the output is links 1-3. The two other `synchronize()` calls near the end of `decode` do not -cover this path: `:1784` is guarded by `if (!embd_seq.empty())` (embeddings, not logits), and `:1560` is in -**`encode`** (1399-1707), not `decode`, and is T5-specific. - -## The mechanism - -> On the logits path, `llama_decode` returns with the logits copy — and possibly the graph that produces -> them — **still in flight**. The sampler then reads `logits.data`, which for HRX is a registered host -> destination at a fixed virtual address. What it reads is whatever the copy has published so far: the -> **previous** token's logits (finite, plausible, wrong) or a region **not yet written**. The first is -> Face B; the second is Face A. - -## Why this fits every hard fact, including the ones that killed earlier candidates - -| fact on the record | how this explains it | -|---|---| -| **Face A: `count=154880 of 154880`, `contiguous=yes`** | the destination read before the copy published anything — an untouched region is uniformly wrong | -| **Face B: finite, plausible, wrong text → HTTP 500** | the previous token's logits: finite and well-formed, so the parser rejects the *body*, not the numbers | -| **intermittency** | whether the copy has completed by the time the sampler reads is a scheduling outcome | -| **coherence is not the carrier** (§13: `hv56=1` and still faulted) | correct and expected — a missing **wait** is not a visibility property | -| **the §15 sync cell did not fix it** | the sync was placed at the end of `graph_compute`, which is **before** the logits copy is issued; it cannot cover the copy it precedes | -| **"ordering the reads changes nothing"** (§59) | that ordered the **fingerprint's** reads, not the engine's logits read | -| **the f32 twin's `memcpy_stable=0` on `result_norm`** | `result_norm` is the input to the logits `MUL_MAT`; the graph is still running during the fingerprint's reads, so consecutive reads of it differ — **independent of memory type** (cells A and B: `0x52` and `0x56`, rates 13.7% and 14.8%), and **not** the deferred host writeback (626 reads, 0 pending) | -| **Face A's damage is whole-vocabulary, not a row** | the read is of the logits *destination*, not of any layer tensor, so a bad read is not confined to one row | - -## The fix, and why it is forward-only - -Restore the wait on the path that needs it — **before the sampler reads the logits**. The minimal -forward-only change is the engine-side line the comment disabled, or its equivalent: - -```cpp - // wait for the computation to finish (automatically done when obtaining the model output) - synchronize(); -``` - -This is an engine-side change in `src/llama-context.cpp` (the engine's own tree, not a revert of any pinned -revision), and it is unaffected by §15's null result because §15 synchronized at a point that precedes the -copy it needs to cover. - -## The discriminating experiment, pre-declared - -**Prediction:** with the wait restored on the logits path, in-regime faults go to zero on both faces — and -without it they do not. A stronger, *instrument-level* confirmation that does not depend on the fix: -**measure whether the copy is complete at the moment the sampler reads the destination.** If the -destination is verifiably complete at every read in a faulting arm, this mechanism is dead. - -**Falsifier:** a faulting arm in which the logits destination is confirmed complete before the sampler -reads it. - -## What is NOT yet established, stated plainly - -* The mechanism is read from source and is consistent with every measurement, but **the fix has not been - run**. Until it is, this is a named candidate with a discriminating experiment, not a demonstrated cause. -* Nothing here shows the *graph* is unfinished as well as the copy; the twin's divergence on `result_norm` - is consistent with the graph running, but the logits-specific claim needs only the copy to be unwaited. -* §14 recorded the `*_async` read and that the sampling path does not synchronise; what is new here is - the **commented-out wait at `:2087`** whose comment asserts the opposite, and the measurement that the - registered-buffer branch (no wait at all) is the one taken. diff --git a/dev/315/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md b/dev/315/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md deleted file mode 100644 index 214095c7f..000000000 --- a/dev/315/315-campaign/evidence/315-NEGATIVE-no-read-path-divergence-revises-mechanism-to-device-ordering.md +++ /dev/null @@ -1,103 +0,0 @@ - - - -# NEGATIVE against the visibility hypothesis, and the mechanism revised to a DEVICE-SIDE ordering defect - -The `315-MECHANISM-kv-not-visible-to-decode.md` hypothesis predicts that a plain-memcpy read and a synchronised -read of the same buffer will **disagree** at the divergent step. The record has **two** instruments that make -exactly that paired comparison, and **neither has ever observed a divergence**. - -## The two paired-read instruments, both negative - -| instrument | paired reads | records | divergences | -|---|---|---|---| -| `hrx-dl-fp` | `probe_hash` vs `copy_hash`, with `hashes_equal` | **146** | **0** — every record is `hashes_equal=1` | -| `hrx-ids-fp` vs `hrx-ids-sync` | unsynchronised host memcpy vs `hrx_synchronous_d2h` | **253,598** | **0** | - -**253,744 paired readings across two independent instruments, zero disagreements.** So on everything the record -has ever compared, **the two read paths agree**. - -**This is real counter-evidence and it is recorded as such.** It does not refute the hypothesis outright — the -comparisons are on the weight-download path and on i32 routing tensors, and **no arm has ever made the comparison -on the KV cache at a divergent step** — but it removes the reason to expect a divergence, and a hypothesis whose -central prediction has been tested 253,744 times elsewhere with zero successes should not be promoted on the -strength of a step it was never measured at. - -## What follows: the wrong KV values are genuine device state - -If the two reads agree, then a wrong value read at a divergent decode step is **what the device actually holds**. -So the fault is **on the device**, and the visibility framing is the wrong one. Re-reading the measured facts -under that constraint: - -- **forced eviction 15.1×** — eviction changes **timing** and memory pressure on the device, which widens a race - window; it is not evidence about CPU caches; -- **index 0 immune, 0/2259** — index 0's logits are computed **in the same pass that writes the K/V**, so no - cross-op dependency exists to be mis-ordered; every index ≥ 1 depends on the prefill's KV writes; -- **front-loaded over ~7 decode steps, 4× under eviction and *zero* early events in control/quiet** — exactly what - a dependency whose window closes as the pipeline drains looks like; -- **finite, clean-prefixed, no crash** — a stale read of genuinely-written memory, not poison; -- **no invalidate anywhere in the runtime** — true, but now **irrelevant**: it would matter for CPU-visible - staleness, and no such staleness is observed. - -## The revised mechanism (device-side ordering) - -> **The prefill writes the prompt's K/V; the decode steps read it across a captured-graph node boundary. The -> dependency between them orders *execution* but does not reliably order *memory visibility on the device*, so -> under pressure a decode step can read K/V before the prefill's writes have become visible to it. The values it -> reads are real numbers from an earlier state, so the logits are finite and plausible; the generation is correct -> until the affected position is attended, then diverges.** - -This is the same shape as before with the CPU-cache claim removed, and it now points at the graph executor's -dependency/barrier logic rather than at allocation flags. - -## Where the record already contains a candidate defect in exactly that logic - -`graph_exec.c`'s barrier insertion: - - for each dependency d of node n: - idx = node_index_map[d->node_index]; - if (idx == UINT32_MAX) continue; // <- dependency SKIPPED: no hazard, so no barrier - if (set_contains(barrier_index_set, idx)) { record_dependency_barrier(); ... } - -and the graph is partitioned with **semaphores at partition boundaries** intended to carry the cross-partition -ordering (`graph_exec.c:1781-1792`: *"We need semaphores at partition boundaries for synchronization. Multi-stream -partitions need join semaphores"*). - -Two things I established earlier and did **not** finish: the `UINT32_MAX` skip is *correct* for nodes genuinely -outside the map (`node_index_map_count = max_node_index + 1`, so it should never fire), **but the lookup in -`graph_exec.c:1527` is done inline with no bounds check** where the helper `graph_analysis.c:139` has one — and -the capacity-overflow path in the node-index set is fail-safe (more barriers), so it is not the defect. -**The cross-partition semaphore path is the part I have not audited, and it is now the prime suspect**: the -prefill and the decode are different graph executions, and the ordering between *them* is not obviously covered -by either the in-partition barriers or the partition-boundary semaphores. - -## What this changes about the plan - -The confirming arm changes shape. It is no longer "do the two reads disagree" — that has been answered -negatively everywhere it has been asked. It is: - -**instrument the K/V buffer at the divergent decode step and compare it against the same buffer's content -immediately after the prefill wrote it.** If they differ, the write was lost or late — a device-side ordering -defect, and the fix is a barrier or a semaphore in the graph's prefill→decode handoff. If they agree, the K/V is -intact and the fault is downstream of it, in the attention or the residual path. - -That is a strictly stronger test than the one this directory pre-registered, and it was only findable by taking -the negative seriously. diff --git a/dev/315/315-campaign/evidence/315-PREREG-argsort-precondition.md b/dev/315/315-campaign/evidence/315-PREREG-argsort-precondition.md deleted file mode 100644 index e20a33678..000000000 --- a/dev/315/315-campaign/evidence/315-PREREG-argsort-precondition.md +++ /dev/null @@ -1,81 +0,0 @@ - - -# PRE-REGISTRATION: §51's rank-sort precondition — does NaN reach the argsort's input? - -Task 2 of the re-scoped plan. Written before any run. It must not repeat the failure that invalidated the -previous id instrument, so that failure is stated first and the probe is designed around it. - -## The failure the probe must not repeat - -`315-moe-id-instrument-never-observed-ids.md`: the previous `[hrx-ids-fp]` instrument read buffers that -**held float data at the moment it read** — `1040594397` is `f32 ≈ 0.131`, `min < 0` in 1.28M of 1.7M -records, byte-identical across clean and faulting arms with a constant "sane" count. Its own conclusion is -that those i32 routing tensors are scratch space shared with a float tensor in the captured graph, so the -probe saw the other tensor's contents. **A by-name post-graph read cannot be trusted for these tensors**, -and §53/§59 say the same for the f32 family. - -## What is being tested - -§51 verified at both ends in the IR (`483a07a`): the rank sort uses only ordered comparisons, so two NaN -elements in a row take the **same** rank — the output is then not a permutation — and the consumer -(`moe_routing_tables.loom:64`) takes the expert index by `index.assume [range(-1, 511)]` rather than -clamping, while the gather on the same ids clamps. - -**The precondition is unmeasured: does a faulting arm actually have NaN at the argsort's input?** - -## The probe, designed around the aliasing failure - -The precondition can be tested **without reading the i32 ids at all**, which is what makes it viable: - -1. **Read the argsort's INPUT tensor** — `ffn_moe_probs`, an **f32** tensor — under the fingerprint that - already exists and is trustworthy for f32 (`hrx-f32-sync`, verified: it compares two reads of the same - buffer at the same call, so arena reuse changes both together and cannot fake a disagreement). -2. **Report NaN count and the first NaN index**, per call, in the faulting arm. -3. **A NaN at the input is the precondition**, and it is observable on the f32 tensor the instrument is - already validated for. No i32 read is required, so the aliasing failure cannot recur. - -The i32 claim — "the rank map is not a permutation" — is then the **inference** from §51's verified IR, not -a measurement, and is labelled as such. - -## Pre-registered cells and configuration - -| cell | purpose | -|---|---| -| **faulting arm** (eviction-forced, diagnostic — no arm counts toward a campaign) | does the argsort's input carry NaN in an arm that faults? | - -Configuration: `GGML_HRX_TENSOR_FP=ffn_moe_probs,GGML_HRX_BUFFER_TRACE=1` (f32 tensor, existing validated -instrument), `-ub 512`, in-regime, eviction forcer, stop at the first fault. - -## Pre-declared readings - -| observation in the faulting arm | reading | -|---|---| -| `ffn_moe_probs` carries **≥2 NaN in a row** at the same call the arm faults | §51's precondition **holds**: the rank sort cannot be a bijection on that row, and the unchecked `index.assume` consumer is reachable. The mechanism becomes a verified chain — precondition measured here, producer verified in the IR, consumer verified in the IR. | -| `ffn_moe_probs` carries **0 NaN** throughout the faulting arm | the precondition **fails**, §51's site is dead as the *origin*, and NaN must be introduced after this point — which moves the hunt to the table builders and the matmul's descriptor use (§51's own falsifier). | -| the tensor produces **no records** in the faulting arm | uninformative — the by-name filter missed it; re-run with a wider `tail:` and say so, rather than reading absence as zero. | - -## Falsifier - -A faulting arm whose `ffn_moe_probs` records show zero NaN at every call, including the faulting one. - -## Cost and honesty - -One arm (the forcer produced a fault in 1 arm for each of the two previous probes), no instrument edit, no -rebuild — the marker `hrx-f32-sync` is already in `~/wt/315fp-build` and the name list is the only change. -If the reading is the second row, §51's site is removed and the survivor list shrinks by one, which is a -bounded result of the kind the re-scoped criterion 1 accepts. diff --git a/dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md b/dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md deleted file mode 100644 index eaeb85589..000000000 --- a/dev/315/315-campaign/evidence/315-PREREG-sync-between-reads.md +++ /dev/null @@ -1,75 +0,0 @@ - - -# PRE-REGISTRATION: does a synchronise BETWEEN two reads stop the `result_norm` change? - -Written while no box run is authorised, so the experiment is fixed before it is launched. It targets the one -**measured, unexplained fact** this issue still has. - -## The fact it explains - -`result_norm` changes between two **consecutive identical plain reads** in a faulting arm: `memcpy_stable=0` -on 75/75 (`315-read-twice-result-buffer-moves-under-the-reader.md`) and 128/128 -(`315-cellA-measured-noncoherent-divergence.md`). Three reads at the same call gave three different hashes. -Two explanations have been removed by measurement: - -* **not the deferred host writeback** — 626 reads at the engine's own logits read in a faulting - eviction-forced arm, **0** with a pending writeback - (`315-candidate-deferred-writeback-FALSIFIED.md`); -* **not cache behaviour of a non-coherent persistent mapping** — the same twin diverges at a measured `0x52` - (13.7 %) and at a measured `0x56` (14.8 %), so the memory type is not the variable - (`315-cellB-divergence-independent-of-memory-type.md`). - -What is left is: **something writes that range between two adjacent reads**, and the question is whether that -writer is ordered by the stream the harness already synchronises before reading. - -## The experiment - -One instrument change on the existing twin build, and one eviction-forced faulting arm (the trigger the -record replicated at 7/10 vs 0/22). No new build configuration, no regime change. - -**Instrument:** in the f32 twin, insert a `backend_synchronize(backend)` **between read #1 and read #2**, and -report a third hash. The line already carries `hash` (the synchronising read), `memcpy_hash` (plain read #1) -and `memcpy2_hash` (plain read #2); this adds `hash_after_sync` = a plain read taken **after** a fresh -synchronise. - -**Configuration:** `GGML_HRX_TENSOR_FP=result_norm,GGML_HRX_BUFFER_TRACE=1`, `-ub 512`, in-regime, eviction -forcer running, stop at the first fault. **Diagnostic cell** — no arm of it may count toward any campaign. - -## Pre-declared readings - -| observation in the faulting arm | reading | -|---|---| -| read#1 ≠ read#2 **and** `hash_after_sync` == read#2 | the writer **is** ordered by the stream: the reads were racing work the sync would have covered, and the earlier divergence is a read-vs-`stream` race. This makes `GraphExecutor::execute`'s non-waiting return the live candidate. | -| read#1 ≠ read#2 **and** `hash_after_sync` ≠ read#2 | the writer is **not ordered by `context->stream` at all** — a fresh synchronise does not stop it. That removes the `execute`-returns-without-waiting candidate and points at a second writer of the arena slot (§53/§59) or something outside the stream. | -| read#1 == read#2 (no divergence) | the arm is uninformative for this question; record it and note that the divergence did not occur under this instrument, which would itself be new (it occurred in every previous faulting arm). | - -The second row is the interesting one: it is the only reading under which the remaining `stream`-ordering -candidate dies. - -## Falsifier - -Any faulting arm in which `hash_after_sync` differs from read#2 **and** a subsequent instrumented check shows -the sync completing normally — i.e. where the writer can be shown to be a *third* party rather than an -ordering gap. - -## Cost and what it buys - -One instrument edit, one rebuild, and **one** faulting arm (the forcer produced a fault in 1 arm for both the -writeback probe and the cell-A run). If it lands on the first row, the issue has a named ordering gap; on the -second, one more candidate is removed and the arena-reuse family becomes the survivor. Either way it is a -bounded result, which is what the re-scope proposal asks criterion 1 to accept. diff --git a/dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md b/dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md deleted file mode 100644 index b465dafca..000000000 --- a/dev/315/315-campaign/evidence/315-PREREGISTRATION-f32-readpath-and-graph-input.md +++ /dev/null @@ -1,119 +0,0 @@ - - - -# PRE-REGISTRATION: the f32 read-path twin and the graph input — the two measurements that can still settle #315 - -**Committed before any arm runs. Nothing in this directory is scored against these criteria, and no arm is -pruned. An arm that does not fault is INVALID for these questions and will be reported as such, never as "no -difference".** - -## Why these two and not something else - -Three instrument-integrity defects have already voided most of the record's per-node evidence, and a fourth — -buffer aliasing — has voided the rest: - -1. **Buffer aliasing.** The fingerprint reads tensors **after compute** (its own code comment says so: - *"fingerprint named graph tensors after compute"*), and it reads a **buffer**, so every tensor name bound to - a shared allocation reports one allocation's final content. Measured: `node_2948` ≡ `ffn_shexp-41` - (identical hash `65f4b97ca74c`), `l_out-42 ≡ l_out-44`, and 47 `l_out-N` names collapse to 4 hashes. **So - "the onset is layer 41" is not a measurement** and has been retracted. -2. **No f32 sync twin.** The i32 branch reads each tensor twice — plain memcpy, and `hrx_synchronous_d2h` — and - comparing them produced the record's "the two read paths agree" negative. **The f32 branch has no twin**, and - f32 tensors are the ones that carry the fault. -3. **The graph input has never been observed.** `inp->embd = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, - n_embd_inp, ubatch.n_tokens)` is created **unnamed**, so the name list cannot match it and `tail:` never - reaches it — it sits at the **head** of `graph->nodes`. Across 567 observed tensor names there is no `inp_*`, - no embedding, no token tensor; the earliest thing ever observed is `l_out-0`. -4. **`fp_nan_only`** hides clean tensors, so no arm has ever shown what is *upstream* of a NaN. - -Together these mean the record has no valid per-node localization, no f32 read-path comparison, and no -observation of the one tensor upstream of everything — while the objective's own mechanism chain is -*"graph input published to host by deferred non-coherent staging writeback → re-uploaded to device → device -computes from stale input"*, and the record's own PR #90 was an **input-path** defect. - -## The instrumentation - -`dev/315/315-campaign/tools/patch-input-f32-sync-twin.py` — verified: applies cleanly to -`ggml-hrx.cpp` (4 anchors, exactly one occurrence each), is idempotent, and the patched translation unit -**compiles with the exact build flags of the real build directory** (`-fsyntax-only`, exit 0). - -It adds two things, both inert unless their env var is set: - -- **`hrx-f32-sync`** — for every f32 tensor the existing fingerprint already reports, read the same bytes - through `hrx_synchronous_d2h` and print `nan`, `first`, `row`, a **full content `hash`**, and the comparison - flags `nan_agree` and **`hashes_equal`** against the plain-memcpy line. A content hash, not just a count, so a - divergence confined to the tensor's interior is visible. -- **`head:`** — select the first `N` nodes, symmetric with the existing `tail:`, so the **unnamed graph - input** is reachable. Combine with `GGML_HRX_TENSOR_FP_NANONLY` **unset** so clean tensors print too. - -## The arms, and what each decides - -**Arm 1 — read-path twin on the tensor that carries the fault.** - - GGML_HRX_TENSOR_FP="tail:16" (or a name list) - GGML_HRX_TENSOR_FP_NANONLY unset - -> compare [hrx-tensor-fp] vs [hrx-f32-sync] for the same call+name - -**Arm 2 — the graph input.** - - GGML_HRX_TENSOR_FP="head:8" GGML_HRX_TENSOR_FP_NANONLY unset - -> is the input clean or NaN, and does its content hash agree between the two read paths? - -**Arm 3 — the in-regime trigger**, so the arm is inside the declared regime gate rather than ambient, and the -question is asked where the fault actually appears. - -## Scored outcomes, fixed now - -**On the f32 twin (Arm 1):** - -| observation | conclusion | -|---|---| -| `hashes_equal=0` on a faulting arm, and the differing region is 256-byte aligned | **read-path staleness CONFIRMED** for f32 — the plain memcpy returns different bytes than a synchronised read; the fix is the visibility operation or a coherent staging allocation | -| `hashes_equal=1` on a faulting arm, with `nan>0` | **read-path staleness REFUTED for f32** — the corruption is genuine device state, and the read path is not the mechanism | -| `hashes_equal=0` but the arm is clean | reported as such: a read-path divergence that does not produce a fault is a different finding and does not satisfy the criterion | -| arm does not fault | **INVALID** for this question. Report as INVALID. | - -**On the graph input (Arm 2):** - -| observation | conclusion | -|---|---| -| input `nan > 0` while the first reader is clean | the input is the seed; the mechanism is upstream of the graph | -| input clean, and `l_out-0` / the first named node is NaN | the seed is inside layer 0's computation, not the input | -| input clean, everything downstream clean, yet the arm faults | the fault is not in the forward path at all — it is in the decode loop or the output path, and Face B's reclassification applies | -| arm does not fault | **INVALID.** Report as INVALID. | - -## Standing corrections this pre-registration carries - -- **"Arena knobs and transient reuse are already falsified on the record"** (the objective's own premise) is - **contradicted by measurement**: reuse is not merely present, it is pervasive and it is what made the - read-pattern/onset result look meaningful. Any acceptance argument that depends on reuse being falsified - should be revisited with the user. -- **Face B's stated definition** ("invalid-UTF-8 output surfacing as HTTP 500 parse failures") has **zero - occurrences** on the record: invalid UTF-8 in 0 arms, HTTP 500 in 0 arms. The operative marker is - `common_chat_peg_parse: unparsed Content-only output`, and roughly half of those arms are the model declining - to emit JSON rather than any corruption. **This needs `/goal-tweak` before a ≥60-arm campaign can be scored - against it** — it is the user's decision, not mine. - -## What is NOT claimed - -No mechanism is named. No fix is proposed. This pre-registers two instruments and the outcomes that would -confirm or refute the only two hypotheses the record can still test, and it states the criteria before the data -exists. diff --git a/dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md b/dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md deleted file mode 100644 index ea03ec976..000000000 --- a/dev/315/315-campaign/evidence/315-PROPOSED-RESCOPED-OBJECTIVE.md +++ /dev/null @@ -1,115 +0,0 @@ - - - -# Proposed revised objective - -Solve 1bit-MONSTER/engine issue #315: name the mechanism behind the intermittent in-regime HRX -GLM-4.7-Flash faults (Face A: NaN logits / HSA_STATUS_ERROR_MEMORY_FAULT; Face B: invalid-UTF-8 output -surfacing as HTTP 500 parse failures), land a forward-only fix in 1bit-MONSTER/engine, and verify it -with a pre-registered in-regime arm campaign of >=60 arms **in the faulting band** showing zero faults on -both faces. - -## Three changes, each forced by measured evidence - -1. **Cause hunt widened to the ggml-hrx runtime/recorder lane and the in-repo loom kernel corpus.** The - trigger is already named and replicated; the cause was shown not to be host-side, not in the logits - read path, and not in the runtime's AQL submission policy. The recorder §33 left as the standing - question lives in `ggml/src/ggml-hrx/runtime/graph-record-order.cpp`, which the existing boundaries - *already* include. See `315-rescope-ground-truth.md`. -2. **Criterion 3 redefined to the faulting band** (`gtt_start >= 98G`), because the campaign of record - (0/70) ran entirely at 97G and ">=60 in-regime arms, zero faults" anywhere in [95,110] is satisfiable - without exercising the defect. -3. **Criterion 1 re-stated** to separate what is PROVEN (the trigger) from what must be NAMED (the cause), - so it is falsifiable rather than open-ended. A bounded negative result is an acceptable outcome. - -## Established — do not re-litigate - -- **Trigger, named and replicated**: forcing KFD evict/restore of in-flight work raises the rate from - 0/22 to 7/10 (Fisher exact two-sided p = 3.6e-5), both faces, verified with the issue's own watcher - (4134 Q_EVICT / 4168 Q_RESTORE in 25 s vs 0/0 quiet). -- The per-arm eviction **count** correlation is falsified (109.5 vs 254.5) — a count is not a measure of - band occupancy. -- The **read-path candidate is refuted** by its own pre-registered A/B (cell e: 0/4 FAULT knob-off vs 0/2 - FAULT knob-on, both Face B, knob verified live), and the refutation is not dismissible (staging pool is - `HOST_LOCAL|HOST_COHERENT|DEVICE_VISIBLE`, `host_queue_staging.c:427`). -- The **whole host-side family is closed**: no host access to the non-coherent `0x52` mapping is on the - decode path. -- The **runtime's submission policy is closed** as a site for missing dependencies - (`aql_block_processor.c`: the runtime does not infer memory hazards). -- **One overclaim retracted**: the pooled 97G-vs-98G census does not survive a within-console control - (`ctl1`: 9.1% vs 6.2%) and must not be cited as a rate. - -## Success criteria (re-scoped) - -- **A named, falsifiable CAUSE for both faces**, in the widened scope, with a discriminating experiment. - The named surface: the recorder's `collect`/`update`/barrier ordering and the 16 MiB - `kMaxTransientExtent` cap in `ggml/src/ggml-hrx/runtime/graph-record-order.cpp`. A bounded, evidenced - negative result is acceptable. -- **A forward-only change landed on a branch** (ancestry check true, not a revert). The refuted read-path - patch must NOT be landed. -- **Pre-registered campaign**: >=60 arms, every arm inside the faulting band (`gtt_start >= 98G`, justified - from the trace rather than the gate label), zero faults both faces, CI and power statement. Arms must - include the eviction condition or demonstrate they reach the band. -- **No regression**: engine CI green; dense-model control and MUL_MAT_ID op-test unchanged. -- **KNOWN-ISSUES.md** carries the evidence and the met criterion (or the entry retired). - -## Boundaries (revised) - -- **In scope**: the strixhalo box; the `~/wt` harness (`315-disc.sh`, `evictonly`, `rt-det`); the engine - pin and docs; and the llama.cpp **ggml-hrx** side — **extended** to `ggml/src/ggml-hrx/runtime/`, - `ggml/src/ggml-hrx/dispatch/`, and the in-repo corpus - `ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/ops/*.loom` (59 files). -- **Out of scope**: AMD driver/kernel changes; the NPU/IRON/Chess lane; the **external** loom/IREE - compiler toolchain (`~/amd-oss/hyperloom-toolchain`, TheRock/loom trees) — reading an emitted `.loom` - in the corpus is in scope, changing the loom compiler is not; rewriting recorded studies; any backward - pin move. -- Still out of scope: declaring the issue solved on "zero NaN" alone. - -## Constraints (unchanged, plus additions) - -All original constraints stand (pre-registration, <=12 h box time with flock, both-face detection, -re-runnable commands at named revisions, forward-only pins, do not weaken instrumentation). Added: - -- **Measure remaining box-time budget from the record before starting GPU work** and prefer bounded code - reading, which needs no GPU, where it can answer the question. -- **One tag, one driver.** A second Pi session (`01a12f94`, tmux `pi-315`) pursues this same goal; see - `~/wt/315-COORDINATION.md`. -- **Record corrections in place rather than deleting.** This record already carries two self-corrections - and a retracted overclaim, and that standard is maintained. - -## Verification contract (re-scoped) - -- Campaign: >=60 arms in the faulting band, zero faults both faces, log paths and rate with CI. -- Forward-only: `git merge-base --is-ancestor ` is true; cite submodule pin before/after. -- Cause: cite file:line for the recorder's collect/update/barrier ordering and state in falsifiable terms - whether a pair can be omitted, with reasoning a reviewer can re-run. -- Baseline-vs-fix on the same harness; engine CI success; dense-model control unchanged. - -## Why this is worth doing rather than closing - -The widened surface is *already in scope*, is bounded code (no regime, no model, no toolchain change), and -has a real discriminating test. It is the one path that could turn the trigger-level mechanism into a -cause and unlock criterion 2. diff --git a/dev/315/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md b/dev/315/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md deleted file mode 100644 index 28bf1f3f4..000000000 --- a/dev/315/315-campaign/evidence/315-REFUTED-pointer-upload-race-replays-are-serialized.md +++ /dev/null @@ -1,92 +0,0 @@ - - - -# REFUTED: the pointer-upload refill race — replays are serialized by a full completion wait - -`315-MECHANISM-CANDIDATE-pointer-upload-refill-race.md` proposed that the engine could refill `staging.host_data` -while a previous `upload_async`'s queued `hrx_stream_update_buffer` commands were still pending, so the device -would consume new bytes for an old update. **The replay path closes that window, and this document records it.** - -## The deciding code - -`command-program-executor.cpp`, the replay path in order: - - upload_prepared_host_staging(context, prepared); // queues pointer-based updates - hrx_graph_exec_launch(recorded.exec, context.stream); // launches the graph - flush_stream_commands(context, "flush HRX graph replay commands"); - wait_stream_commands(context, "wait for HRX graph replay commands", recorded.exec); // <-- here - -Note that the `hrx_stream_synchronize` further down is **conditional** — it fires only when -`GGML_HRX_DIAGNOSTIC_GRAPH_SYNC` is set — so the default ordering rests entirely on `wait_stream_commands`: - - static Status wait_stream_commands(context, label, work) { - stream_wait_sleeping(context.stream, wait_history_for(work)); - } - -and `runtime/hrx-sleeping-wait.cpp`: - - hrx_status_t stream_wait_sleeping(hrx_stream_t stream, WaitHistory & history) { - if (blocking_wait() || stream == nullptr) return hrx_stream_wait(stream); - ... // history-based sleep is an OPTIMISATION only - hrx_status_t status = hrx_stream_wait(stream); // <-- ALWAYS a full completion wait - ... - } - -**The sleep-then-query is an optimisation; every path ends in `hrx_stream_wait(stream)`.** So a replay does not -return until the stream is quiescent, and the next replay's staging refill cannot precede the previous replay's -uploads. **The race is closed, and the candidate is refuted.** - -## What was eliminated on the way, and what that leaves - -Four candidates have now been eliminated by direct evidence rather than by argument: - -| candidate | eliminated by | -|---|---| -| snapshot-at-capture (`copy_source_into_graph = true`) | `iree_hal_streaming_memcpy_value_to_device` has **no callers anywhere** | -| coherence / allocation flags | closed from source: all three host paths hardcode `0x52`, flags are stored only | -| read-path visibility (memcpy vs `hrx_synchronous_d2h`) | **253,744** paired readings across two instruments, **zero** disagreements | -| pointer-upload refill race | replays are serialized by a full `hrx_stream_wait` | -| (weights cache keyed on allocation generation) | correct by design — weights are immutable, so an allocation-scoped key is right | - -and two earlier claims of my own were **retracted** (the layer-41 onset; the fixed 32-entry boundary). - -## What still stands, unanswered - -The phenomenology is intact and none of the refutations touch it: - -- a per-decode-step hazard of ~1e-3, **finite**, entering at a clean boundary, all 18 events clean-prefixed; -- **forced eviction 15.1×** the control hazard (0.001195 vs 0.000079 over 4,184 and 12,651 run-steps); -- **index 0 immune, 0 of 2,259** — the prefill's own output is always correct, so the fault is in the decode loop; -- front-loaded over ~7 decode steps under eviction (4× the arm's own rate) with **zero** early events in control - and quiet arms; -- Face B's HTTP 500 is the server rejecting a degenerate generation, and **invalid UTF-8 cannot occur** (the - garbage is U+FFFD, a valid codepoint). - -## Why this is the end of what the record can decide - -Every remaining step needs the same thing: **observe a decode step that is known to diverge, with an instrument -that reads something the record has never read at that step.** The archive names all 18 divergent -arm/request/step triples, and `patch-input-f32-sync-twin.py` (verified: applies, compiles) reads the f32 pair and -the unnamed graph input — but **neither has ever been run**, and no further arrangement of existing bytes can -substitute for it. Four candidates have been eliminated from the armchair; the fifth needs the box. - -**No mechanism is named. Four are excluded, the phenomenology is pinned to a decode step with a rate and a -dose-response, and the next candidate cannot be tested without an arm.** diff --git a/dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md b/dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md deleted file mode 100644 index 25f983ab0..000000000 --- a/dev/315/315-campaign/evidence/315-RESCOPE-PROPOSAL-2026-10-15.md +++ /dev/null @@ -1,99 +0,0 @@ - - -# Re-scope proposal, 2026-10-15 — for `/goal-tweak` - -Supersedes `~/315-PROPOSED-RESCOPED-OBJECTIVE.md` (2026-10-13), which predates this session. Written because -the named mechanism was falsified by its own pre-registered campaign and no mechanism is now named; the -executor cannot edit the objective, so this is text for the owner to apply. - -## Why a re-scope, in one paragraph - -The objective's criterion 1 asks for a **named mechanism**, and §"if it is, the fix is rejected" makes it a -gate rather than a goal. Over this issue's life, **eleven candidate families have been eliminated by direct -measurement** — coherence (§13, `hv56=1` and still faulted), consumer ordering (§15), FA masked-row (§26), -the visibility family (§59), the MoE id tensors (1.16M paired reads, 0 disagreements), the pin move itself -(the only changed kernel is never dispatched: 60/60 FA dispatches are the non-split kernel, 0/944 logs -mention `decode_split`), the deferred host writeback (626 reads, 0 pending, in a faulting arm), the -persistent-mapping/cache explanation (two-cell A/B: `0x52` 13.7 % vs `0x56` 14.8 %), and now the unwaited -async logits read (its own acceptance campaign stopped on the first fix arm with a **Face B** fault at -`gtt_start 98G`). Each elimination is recorded with a re-runnable command and its output. - -What has never been produced is a positive: a mechanism that survives its own discriminating experiment. -Continuing to hold criterion 1 as a gate means the remaining work is unbounded while the alternatives that -were *supposed* to be ruled out already are — that is the state the re-scope is for. - -## Proposed changes - -**1. Criterion 1 — accept a bounded, evidenced negative.** - -> A named, falsifiable mechanism for both faces supported by a discriminating experiment that rules out the -> surviving alternatives, **or** a bounded negative result: every candidate family in the surviving list -> tested by its own pre-registered discriminating experiment and reported, with (a) each elimination's -> re-runnable command and observed output, (b) the surviving families stated explicitly, and (c) one -> measured, unexplained fact recorded as the hand-forward. A bounded negative is an acceptable outcome and -> is not reported as a mechanism. - -**2. Criterion 3 — separate "a campaign was run" from "a fix was verified".** - -> A pre-registered in-regime campaign is run on **the configuration under test**, with the regime gate, -> the arm count, both-face detectors and the stopping rule fixed before it runs, and is **reported with its -> result — met or not — including the arm tally, the CI and the power statement.** The ≥60-clean-arm -> acceptance is required only if a candidate fix survives criterion 1's discriminating experiment; when the -> fix under test faults, the campaign's value is the fault it produced, and it is reported as a failed -> acceptance rather than as an unrun arm. - -**3. Criterion 2 is unchanged and is already satisfied.** The forward-only change is landed -(`47a8fe929` on `1bit-MONSTER/llama.cpp` branch `1bit/315-logits-read-wait`; pin `e44c9d01a → 47a8fe929` on -`1bit/315-preregistration`, `95238dc`, `git merge-base --is-ancestor` true). It is a correct change for the -reason its comment gives — a backend that implements `get_tensor_async` loses the generic layer's -synchronise — even though it is not sufficient to stop the fault. **A re-scope must not be read as -retracting it.** - -**4. Criterion 4 (no regression) is unchanged** and still outstanding: engine CI plus the dense-model and -MUL_MAT_ID controls. It needs box time, which is currently capped and paused. - -**5. Criterion 5 is met in substance already** — `KNOWN-ISSUES.md` carries the mechanism, the fix, the -campaign and the falsification, with evidence paths — and its status line stays **OPEN**, because nothing -has been shown to remove the defect. - -## The bounded negative, as it would be reported under the new criterion 1 - -**Eliminated by direct measurement, each with its command and output:** the eleven families listed above, -all in `dev/315/315-campaign/evidence/`. - -**Surviving families, stated explicitly:** - -1. An open writer of the `result_norm` slot: two consecutive *identical* plain reads of it return different - bytes in a faulting arm (`memcpy_stable=0`, 75/75 and 128/128), and that writer is neither the deferred - host writeback nor cache behaviour of a non-coherent mapping. Candidates that remain in - `315-what-the-synchronise-drains.md`: `GraphExecutor::execute` returning without a wait while - `backend_synchronize` covers only `context->stream`; a second writer of the reused arena slot (§53/§59); - the replay path (least likely — §4047348 traces it to a full wait). -2. §51's rank-sort non-bijection: verified at both ends in the IR (ordered comparisons make two NaN elements - tie; the consumer takes the expert index by `index.assume`, not a clamp), with its **precondition - unmeasured** — NaN at the argsort's input. - -**The one measured, unexplained fact to hand forward:** `result_norm` changes between two consecutive -identical plain reads in a faulting arm, at a measured `0x52` mapping and equally at `0x56`, with no pending -host writeback at the engine's own logits read. - -## What this re-scope does NOT do - -It does not declare the defect gone, does not retire the KNOWN-ISSUES entry, and does not convert a -falsification into a mechanism. It moves criterion 1 from "produce a positive" to "test the surviving -alternatives and report a bounded result", which is what the remaining work actually is. diff --git a/dev/315/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md b/dev/315/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md deleted file mode 100644 index 3312760ab..000000000 --- a/dev/315/315-campaign/evidence/315-RETRACTION-no-fixed-32-entry-boundary.md +++ /dev/null @@ -1,90 +0,0 @@ - - - -# RETRACTION: there is no fixed "32 KV entry" boundary. The divergence step varies (7, 49, …) — and the detector has a NaN confound - -The `315-MECHANISM-faceb-diverges-at-first-32-entry-attention-step.md` finding was **over-fit to one arm**, and I -am withdrawing the part of it that generalised. - -## What was claimed, and what the population says - -The claim was that the divergence occurs at logits index 7 — 25 prompt tokens + 7 = exactly **32 KV entries** — -and that this is "the first clean power-of-two boundary", i.e. the first point at which the fault *can* appear. - -Running the same divergence detector over **all 54 arms** that carry `hrx-logits-fp`: - -| divergence index | arms | absolute position | -|---|---|---| -| 1 | 1 | 26 | -| 7 | 1 | 32 | -| 16 | 1 | 41 | -| 49 | 1 | 74 | - -**Four events, four different indices.** Index 7 is not special, "32 KV entries" is not a boundary, and the -reasoning that connected it to the record's other 32s (`1984 = 31×64`, `2048 = 32×64`) was pattern-matching on a -coincidence. **Retracted.** - -## And a confound in the detector itself - -Any NaN in the logits changes the hash, so a hash-based divergence detector fires on **NaN** as well as on the -degeneration Face B describes. Splitting the four events by whether the arm's logits contain any `nan > 0`: - -| arm | index | `nan>0` in arm | what it actually is | -|---|---|---|---| -| `315inp2-r1e` | 7 | **0** | **genuine non-NaN divergence** (the Face-B case) | -| `315inp2-r2e` | 49 | **0** | **genuine non-NaN divergence** | -| `315lout4-r2e` | 1 | 1 | NaN-driven — hash change explained by the NaN | -| `315nan-p1l` | 16 | 1 | NaN-driven — hash change explained by the NaN | - -So there are **two** genuine non-NaN divergence events on the record, both in eviction arms, at indices **7 and -49** — different steps of different prompts. **The detector must be gated on `nan == 0` to speak to Face B at -all**, and that gating has now been done rather than assumed. - -## What survives, and it is still the strongest thing on the record - -1. **Identical input, intermittent outcome.** In `315inp2-r1e` prompt A was answered four times in one process; - three runs are **bit-identical** in the logits, the fourth diverges. That is a real state difference for - identical input, and it is the fault. -2. **The divergence is upstream of sampling.** The *logits hash* differs, so the logits values differ; the - sampler did not introduce it. -3. **The fault is finite.** All 423 logits readings in that arm are `nan=0`. Face B involves no NaN. -4. **Logits are deterministic across arms**, so a hash difference is signal, not noise — verified by matching - the arm's step-0..3 hashes against an independently-taken sample from a different arm. -5. **Face B's HTTP 500 is the server's own response** to a degenerate generation - (`"The model produced output that does not match the expected Content-only format"`), with an empty body — - which is why `utf8_ok`/`json_ok` are true on that row. **Invalid UTF-8 is impossible here:** the garbage - contains U+FFFD, a valid codepoint. -6. **The corruption step varies**, which is *also* informative: it is not a fixed structural boundary, it is a - **per-step probabilistic event** — the signature of a race in the decode loop, not of a fixed-size region. - -## What this leaves - -The mechanism is a **per-decode-step intermittent, finite, non-NaN divergence that occurs in eviction arms**. -The step at which it appears varies, so no boundary-based test can name it; what can is a **per-step paired -read** at the divergent step. That is exactly what `patch-input-f32-sync-twin.py` enables: for the call whose -logits hash diverges, compare the plain-memcpy and synchronised reads of the tensors it depends on. If the two -reads differ there, the fault is a visibility/ordering defect; if they agree, it is genuine device state and the -race is on the device side. - -Two honest notes on this directory: this is the **second** claim I have had to retract after extending a -single-arm observation to the population (the first was "the onset is one row / layer 41"), and the pattern is -the same — a sharp result in one arm that does **not** survive being counted. Both retractions are recorded -rather than quietly edited. And the mechanism remains unnamed. diff --git a/dev/315/315-campaign/evidence/315-ab-preregistration.md b/dev/315/315-campaign/evidence/315-ab-preregistration.md deleted file mode 100644 index 8b2a74316..000000000 --- a/dev/315/315-campaign/evidence/315-ab-preregistration.md +++ /dev/null @@ -1,91 +0,0 @@ - - - -# Pre-registration: does the `GGML_HRX_COHERENT_READ` fix stop the forced-eviction fault? (`315ab1`) - -Written 2026-10-13 ~10:00, committed before the run. - -## The hypothesis under test - -**H-fix.** `buffer_get` reads a host-visible, non-coherent mapping with a bare `memcpy` -(`ggml-hrx.cpp:250-255`). Routing that read through `hrx_synchronous_d2h` — a queued HAL DOWNLOAD — -instead of the raw pointer read removes the stale-view window, so forced KFD eviction should stop -producing the fault. - -Mechanism chain this rests on (all recorded, with the falsified predecessor noted): - -1. the logits buffer is host-visible (`type=HRX0_HOST host_visible=1 ... size=619520` in the issue's own - allocation trace) and, because `use_direct_host_bindings` defaults off, its memory_type is - `HOST_LOCAL|DEVICE_VISIBLE` = **`0x52`**, which carries `0x02` HOST_VISIBLE but **not** `0x04` - HOST_COHERENT (`hrx_runtime.h:194-202`); -2. it is `mmap`'d (`hrx_buffer_map`), so `context->base != FAKE_PTR_BASE` and `buffer_get` takes the - **plain `memcpy`** branch; -3. no cache-invalidate API exists in the runtime (`hrx_runtime.h` has only `hrx_stream_flush`, which is - command-buffer semantics), so nothing on that path refreshes the CPU's view; -4. `llama_get_logits_ith` does call `ctx->synchronize()` and §55/§56 traced that to - `hsa_signal_wait_scacquire` with a timepoint covering the last submission — so **stream** ordering is - present; what is absent is **cache coherence on the CPU read**; -5. forced eviction of in-flight work is what makes the mapping's contents differ from what the CPU - cached, which is why the eviction cell faults at 70 % (7/10) while quiet (0/10) and CPU-pressure - (0/12) cells do not. - -Consistency check this predicts: the earlier coherence-**attribute** patch -(`315-coherence-fix.patch`, `HRX_MEMORY_TYPE_HOST_COHERENT` made unconditional) was falsified as a fix -(9 clean + 1 fault / 10). Setting an attribute on the allocation does not add an invalidate to the -**read**, so that patch could not have fixed a read-path defect. The failure is expected under H-fix. - -## Design — one binary, one env var, the eviction rig - - driver : ~/wt/315-disc.sh (the SAME frozen three-cell design as disc1/disc2) - BIN : ~/wt/315coherent-build/bin (the patched build; verified to carry the knob) - cells : q quiet / e evictonly / c CPU-pressure, rotating per round - A arm : EXTRA_ENV without the knob -> the pre-fix behaviour (71% expected, from disc2) - B arm : EXTRA_ENV with GGML_HRX_COHERENT_READ=1 -> the fix path - -Both arms use the SAME binary and the SAME rig, so the only difference is the knob. Cell `e` is the -test cell; `q` and `c` remain internal controls. - -Rounds: 4 per arm (≈16 arms, ~40 min per arm at the measured pace) unless the result is already clear. - -## Prediction, and the four branches, fixed BEFORE the run - - B (knob on) faults at ~0 in cell e, A (knob off) faults at ~50-70% in cell e - -> H-fix SUPPORTED. The read-path incoherence is the cause, and the one-hunk change is the fix. - B ~= A (both fault in cell e at a similar rate) - -> H-fix REFUTED. The memcpy is not the site; the stale data enters elsewhere. - B faults but A does not - -> CONTRADICTED in direction; report as such (it would suggest the fix makes things worse, - which would itself be informative about the transfer's semantics). - both clean in cell e - -> UNINFORMATIVE. Must NOT be read as evidence the fix works: at the ambient ~5-7 % rate a - clean eviction cell is possible by chance, and it would also mean the rig failed to induce. - Re-run or report as uninformative, per the pre-registration discipline used for disc2. - -## What this does and does not settle - - - It tests the CAUSE for the first time with an intervention plus a candidate repair. - - It does not by itself satisfy criterion 2's "forward-only change landed on a branch/PR": that needs - the change prepared on the fork against the current pin, with the ancestry check, and the >=60-arm - in-regime campaign on the fixed build. Both remain after this test. - - It does not test for regression; a separate control (dense model + MUL_MAT_ID op-test) is criterion 4. diff --git a/dev/315/315-campaign/evidence/315-ab1-arm-A-result.md b/dev/315/315-campaign/evidence/315-ab1-arm-A-result.md deleted file mode 100644 index 4281a4a4b..000000000 --- a/dev/315/315-campaign/evidence/315-ab1-arm-A-result.md +++ /dev/null @@ -1,87 +0,0 @@ - - - -# A/B arm A — the control that must fault (315ab1a), 4 complete rounds - -Pre-registered at `fc7ffd5` (`dev/315/315-campaign/evidence/315-ab-preregistration.md`), which fixed the -binary, the rig, the four reading branches and the cell-`e` comparison BEFORE the run. This file -records the arm-A result only, written after the arm finished and before arm B's result is known. - -## Invocation (re-runnable) - - BIN=/home/bcloud/wt/315coherent-build/bin \ - bash ~/wt/315-disc.sh 4 315ab1a - -Patched binary, knob **OFF** (no `GGML_HRX_COHERENT_READ`), i.e. pre-fix behaviour, on the same frozen -three-cell eviction rig used by disc1/disc2. Box lock held on fd 9 (`.cache/lax-decode/box.lock`). -Console: strixhalo `~/wt/315ab1a-console.log`. - -Session header: tag=`315ab1a`, rounds=4, gtt_before=26G, mode=performance, bin version 10613, -`extra_env='GGML_HRX_TENSOR_FP=tail:16 GGML_HRX_TENSOR_FP_NANONLY=1'` (no knob), -model_sha256 `29837ed2c0fc5f51981adf8ac8083fcf80743c598381f13e9f06cbad0498b174`, -detector_sha256 `9d46ec453fc9640e33e6d6eed857c3e263bfa3df9c1c151a352e1f6f62a66a48`. - -## Result — the driver's own summary - - cell q: OK=3 FAULT=1 OOR=0 INVALID=0 - cell e: OK=0 FAULT=4 OOR=0 INVALID=0 - cell c: OK=4 FAULT=0 OOR=0 INVALID=0 - -Per-arm: - - round 1 arm q: OK nan=0 req=66 gtt 97G->98G side=0/0 - round 1 arm e: FAULT nan=1 req=19 gtt 98G->79G side=1/1 <- FACE A - round 1 arm c: OK nan=0 req=66 gtt 97G->98G side=12/12 - round 2 arm e: FAULT nan=0 http=1 s500=1 unparsed=2 side=1/1 <- FACE B - round 2 arm c: OK nan=0 req=66 side=12/12 - round 2 arm q: FAULT nan=0 http=1 s500=1 unparsed=2 gtt 97G->102G side=0/0 <- QUIET CELL - round 3 arm c: OK nan=0 req=66 side=12/12 - round 3 arm q: OK nan=0 req=66 gtt 97G->98G side=0/0 - round 3 arm e: FAULT nan=0 http=1 s500=1 unparsed=2 side=1/1 <- FACE B - round 4 arm q: OK nan=0 req=66 gtt 97G->98G side=0/0 - round 4 arm e: FAULT nan=0 http=1 s500=1 unparsed=2 side=1/1 <- FACE B - round 4 arm c: OK nan=0 req=66 side=12/12 - -**Cell `e` faulted in every round (4/4), across BOTH faces** (Face A on round 1, Face B on rounds 2–4), -while cell `c` (CPU pressure, no GPU) was clean 4/4. Arm A is therefore a valid control: the rig -reproduces on the patched binary with the knob off. - -## The quiet-cell fault, read against the GTT traces - -Cell `q` faulted once (round 2, Face B) with `side=0/0` — no side load at all. It is not rig noise: -comparing every quiet arm across the three runs of this rig, - - run quiet arms their GTT behaviour quiet-cell faults - disc2 6 all 97G->98G (stable) 0 - ab1a 4 97->98, 97->102, 97->98, 97->98 1 <- the drifting one - -the faulting quiet arm is the one whose GTT **drifted +5 GiB** (97G -> 102G); every quiet arm at stable -GTT is clean. That is what the mechanism predicts and what `evictonly` is a *proxy* for: the operative -variable is **memory pressure that makes the kernel evict in-flight work**, not the forcer process -itself. A quiet arm whose occupancy climbs into the band can reach the same condition with no side load. - -This sharpens what the A/B tests rather than weakening it. The pre-registered branch of interest is -unchanged: **arm B clean in cell e while arm A faults in cell e.** Arm A's cell `e` is 4/4 FAULT. - -## What this does NOT establish - -This is the control. It says nothing about whether the fix works; that is arm B (`315ab1b`), same -binary with `GGML_HRX_COHERENT_READ=1`, whose result is recorded separately when it completes. diff --git a/dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md b/dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md deleted file mode 100644 index a62f7f7c4..000000000 --- a/dev/315/315-campaign/evidence/315-ab1-arm-B-interim.md +++ /dev/null @@ -1,136 +0,0 @@ - - - -# A/B arm B — INTERIM: the fix's first eviction-cell arm faults (Face B) - -**Status: INTERIM.** Arm B (`315ab1b2`) is still running (round 1 of 4 complete at the time of -writing). This file records the first scored cell-`e` arm so the observation cannot be lost or -re-read after the fact; the full tally and the final reading are recorded when the run completes. - -## What is already in - -Same binary, same frozen three-cell rig as arm A, knob **ON** (`GGML_HRX_COHERENT_READ=1`), box lock -held, single driver (the collision that killed `315ab1b` is documented separately and this re-run -carries a start guard). - - BIN=/home/bcloud/wt/315coherent-build/bin \ - EXTRA_ENV="GGML_HRX_TENSOR_FP=tail:16 GGML_HRX_TENSOR_FP_NANONLY=1 GGML_HRX_COHERENT_READ=1" \ - bash ~/wt/315-disc.sh 4 315ab1b2 - - round 1 arm q: OK nan=0 req=66 gtt 97G->98G side=0/0 - round 1 arm e: FAULT nan=0 http=1 s500=1 unparsed=2 gtt 98G->99G side=1/1 <- FACE B - -Arm validity checked, not assumed: - -- **in-regime**: `run.log` records `gtt_start 98GiB mode performance cell=e side_pids=2081364`; -- **the intervention was applied**: `side_pids` names a live `evictonly 500`; -- **the fix path was live**: the arm server's own environment (`/proc//environ`) carried - `GGML_HRX_COHERENT_READ=1` while the arm ran; -- **the failure is Face B, by the recorded signature**: `detector.json` gives - `http_failures: 1`, `first_failure: "HTTP 500"`, `truncated: true`, `requests: 9` (of the expected - 66), and the server logged - `"code":500,"message":"The model produced output that does not match the expected Content-only format"`. - -That is the same Face-B signature the eviction cell produced on the unfixed build (disc1/disc2 and -arm A). - -## Why this matters for the pre-registered reading - -The pre-registration (`fc7ffd5`) fixed four branches before the run. The branch this first data point -lands in is: - -> **B ≈ A (both fault in cell `e` at a similar rate) -> H-fix REFUTED. The `memcpy` is not the site; -> the stale data enters elsewhere.** - -Arm A's cell `e` is 4 FAULT / 0 OK. Arm B's first cell-`e` arm is also a FAULT. If the remaining -rounds confirm it, then routing the host-visible read through `hrx_synchronous_d2h` **does not remove -the fault**, and the mechanism chain recorded in `f5ce617`/`5446f19` (bare `memcpy` on a non-coherent -mapping as the corruption site) is falsified as the *cause*. - -## What this would and would not falsify - -- **Would falsify**: `buffer_get`'s bare `memcpy` as the site where the stale/corrupt data enters, and - therefore the prepared one-hunk fix as a repair. -- **Would NOT falsify**: criterion 1's trigger-level mechanism. Forced KFD eviction of in-flight work - raising the fault rate (7/10 vs 0/22, p=3.6e-5, both faces) is an independent, replicated - intervention result and is untouched by this. The cause site reopens; the trigger stands. -- **Would NOT falsify** the observation that the read path *is* incoherent (`0x52`, no `HOST_COHERENT`) - — only that fixing it at `buffer_get` is sufficient. - -## Not yet claimed - -A single round is not "B ≈ A". The claim waits on arm B's full cell-`e` tally. If later rounds are -clean, the reading changes to mixed and must be reported as such rather than resolved post hoc. - ---- - -# FINAL (2026-10-13 ~10:56) — completed at 2 of 4 rounds; the refutation is established - -The run was stopped after round 2 rather than continued. Recording why, and what the stopped run does and -does not support. - -## Final scored arms - -``` - round 1 arm q: OK nan=0 req=66 gtt 97G->98G side=0/0 - round 1 arm e: FAULT nan=0 http=1 s500=1 unparsed=2 gtt 98G->99G side=1/1 <- FACE B - round 1 arm c: OK nan=0 req=66 gtt 97G->98G side=12/12 - round 2 arm e: FAULT nan=0 http=1 s500=1 unparsed=2 gtt 98G->100G side=1/1 <- FACE B - round 2 arm c: FAULT nan=1 req=34 transport=1 gtt 97G->79G side=12/12 <- FACE A - round 2 arm q: OK nan=0 req=66 gtt 97G->98G side=0/0 -``` - - cell e (the test cell): 0 OK / 2 FAULT both Face B - cell q (quiet): 2 OK / 0 FAULT - cell c (CPU pressure): 1 OK / 1 FAULT the fault is Face A - -## The pre-registered reading - -Unchanged from the interim note: arm B's cell-`e` arms **fault with the fix engaged**, while arm A's -cell `e` is 4/4 FAULT. That is the refutation branch fixed at `fc7ffd5`: - -> **B ~= A -> H-fix REFUTED. The memcpy is not the site; the stale data enters elsewhere.** - -The knob was verified live during the arms by reading the arm server's own `/proc//environ`. The -refutation is not dismissible — the vacuity objection (that `hrx_synchronous_d2h` might itself be -incoherent) was checked at source and disproved (`315-ab1-fix-path-is-coherent.md`). - -## Why it was stopped at 2 rounds - -- **The verdict was settled.** Cell `e` faulted in every round run, so rounds 3-4 could only narrow a - rate that is not in question. The pre-registration's "B ~= A" branch is met on the evidence already in. -- **Box-time budget.** The record puts this goal turn at ~10-11.1 h of the stated <=12 h ceiling before - this run; two more rounds would add ~20 min against a margin that is at best thin. - -Stopping a pre-registered run early is a deviation and is recorded as one: the pre-registration specified -4 rounds unless the result was already clear, and this invokes that clause explicitly rather than -silently. **The cost is stated**: arm B has 2 cell-`e` arms against arm A's 4, so the comparison is 2-vs-4 -in cell `e`, not round-for-round. The direction is unambiguous (arm B faults in 100% of its cell-`e` -arms) but the sample is half the planned size, and no rate estimate should be computed from it. - -## What is established, and what is not - -- **Established**: the read-path fix does not stop the fault; the corruption is not the CPU observing a - stale logits page. -- **Established separately**: the `cell c` Face-A fault, whose GTT trace (97 -> 106G) is recorded in - `315-ab1-cell-c-fault-drift.md` as a single-arm observation. -- **Not established**: the cause site. Still open, with the read path eliminated. -- **Not usable as a rate**: 2 arms. Nothing in this file should be quoted as a frequency. diff --git a/dev/315/315-campaign/evidence/315-ab1-cell-c-fault-drift.md b/dev/315/315-campaign/evidence/315-ab1-cell-c-fault-drift.md deleted file mode 100644 index 16be93a10..000000000 --- a/dev/315/315-campaign/evidence/315-ab1-cell-c-fault-drift.md +++ /dev/null @@ -1,128 +0,0 @@ - - - -# Arm B round 2 cell `c` FAULTED — a control-cell fault that the mechanism predicts - -Recorded immediately, because a fault in a *control* cell is the kind of result that gets quietly -explained away. Here it is informative, and it is the strongest single piece of evidence the A/B -produced about the mechanism itself. - -## The result - - round 2 arm c: FAULT nan=1 req=34 transport=1 s500=0 unparsed=0 - gtt_start=97G gtt_now=79G tctl_start=49C tctl_max=92C - side=12/12 stopped_frac=0.16 thermal_pauses=0 - -Cell `c` is CPU-pressure-only by design (12 busy-loop workers pinned to CCD1, **no GPU side load**). -It was clean 4/4 in arm A and clean in the original disc runs. Here it produced **Face A**: - - common_sampler_sample: HRX returned NaN logits: count=154880 of 154880, - index range [0..154879], contiguous=yes, continuing=no (engine#315) - common/sampling.cpp:620: HRX: NaN logits (engine#123) - -whole-vocabulary NaN, 2.31 s into the arm, with the sampler guard aborting — the exact Face-A signature. - -## Why it is not a harness artifact - -- **in-regime**: `gtt_start 97GiB`, inside the declared [95,110] gate; -- **the side load ran the whole arm**: `side=12/12`, and the driver marks `INVALID(side-died)` if not; -- **not thermal**: `thermal_pauses=0`, `tctl_max=92C` (below the 93C abort); -- **not the CPU load itself**: cell `c`'s CPU load is the same in every clean round, including arm A's - 4 clean cell-`c` arms; -- the failure is the recorded Face-A string, not a parse or transport oddity — the `transport_failures=1` - is the sampler guard aborting the connection *after* the NaN, which is the documented order. - -## What the GTT trace shows — and this is the point - -Per-sample GTT across the arm (`tctl.log`, GiB): - - 79 96 97 98(4) 99(9) 100(4) 101(2) 102(4) 103(2) 104(2) ... 106 - -The cell **drifted upward** 97G -> 106G during the arm, then collapsed to 79G when the server's -allocation was torn down after the fault. It did not sit still at 97G and get unlucky. - -That is precisely the mechanism's prediction, and precisely the pattern arm A's quiet-cell fault showed: -**the operative variable is memory pressure that drives the kernel to evict in-flight work, not the -forcer process.** `evictonly` is a *proxy* — it drives pressure to the band deliberately and fast. -A cell that reaches the band on its own, whatever its nominal load, hits the same condition. Cell `c`'s -CPU workers evidently perturbed allocation enough to walk GTT up across the band. - -## What it changes about the reading - -- **It does not weaken the A/B.** The pre-registered comparison is cell `e`; this is a control cell. -- **It strengthens the mechanism statement** by supplying a *third* independent way to reach the fault - (quiet-drift in arm A, forcer in disc1/disc2, CPU-drift here), all of which share one variable: - GTT/memory pressure walking into the eviction band. -- **It means the regime gate is necessary but not sufficient for cell purity.** `gtt_start` inside - [95,110] does not keep a cell in one regime for its whole duration — `gtt_now` and the drift between - them matter, and the console records both. Any future campaign should treat drift into the band as a - covariate, not assume the start value holds. - -## Honest scoping - -This is one arm. It does not show cell `c` faults at the eviction cell's rate — cell `c` is 1 fault in -2 rounds here against cell `e`'s 2 in 2, and arm A had cell `c` clean 4/4. It is recorded because the -direction is the mechanism's and because the drift is measurable, not because it establishes a rate. - ---- - -# The cell-c drift, measured sample by sample - -The claim that cell `c` faulted because it walked into the eviction band is measurable, so here is the -whole trace rather than a summary. From `315ab1b2-r2c-GLM-4.7-Flash-Q4_K_M/tctl.log` -(sample = 5 s; GTT in GiB, load1 from `/proc/loadavg`): - - sample GTT load1 - 1 79 2.97 <- arm start, pre-load - 2 96 2.97 - 3 97 3.86 <- in-regime gate value - 4 98 5.87 - 5-7 98 7.56 .. 10.24 - 8-11 99 11.42 .. 14.64 - 12 100 15.63 - 20 103 19.51 - 21 104 19.39 - 22 106 19.75 <- peak - 23-26 99..102 19.85 .. 20.27 <- and it stays in the band, oscillating - 30 99 21.67 - -The CPU-pressure cell raises system load from ~3 to ~21 and, as it does, GTT climbs from the 97 G gate -value to a 106 G peak and then **settles into a 99-104 G oscillation** for the rest of the arm. - -That is the mechanism's operative variable, made directly observable: the cells differ in *how they -drive memory pressure into the band*, not in whether the fault is possible. `evictonly` drives it fast -and deliberately; three GLM neighbours plus 12 duty-cycled CPU workers drive it slowly and incidentally; -a quiet arm can drift there on its own (arm A round 2). All three reach the same band, and the band is -where KFD evicts in-flight work. - -## Consequence for the pre-registration's regime gate - -The gate checks `gtt_start in [95,110]`. The trace shows the arm spends most of its life **above** its -start value, in a band the gate does not constrain. A cell that starts at 97 G and ends oscillating at -102 G is not "in-regime" in any sense that keeps its cell identity — and this is why a nominally -CPU-only control could fault. - -This does not invalidate the A/B (the comparison is still cell `e` vs cell `e`, and arm B's cell-`e` -faults are unambiguous at `gtt_start 98G`), but it does mean: - -- **cell membership has to be argued from the trace, not the gate label**; and -- any future campaign should record `gtt_now` and the drift, and treat "drifted into the band" as the - covariate that matters — which is exactly what the console already logs and what these files cite. diff --git a/dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md b/dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md deleted file mode 100644 index 12289b02a..000000000 --- a/dev/315/315-campaign/evidence/315-ab1-fix-path-is-coherent.md +++ /dev/null @@ -1,69 +0,0 @@ - - - -# The fix's mechanism was sound — so the refutation is informative, not trivial - -Written while arm B (`315ab1b2`) is still running, in response to its first result (round 1 cell `e` -FAULT, Face B). It checks the one objection that would make the refutation *uninformative*: that -`hrx_synchronous_d2h` might itself end in an incoherent CPU read, in which case arm B's fault would -prove nothing about the corruption site. - -It does not. The transfer path is coherent end to end, so the refutation stands as evidence. - -## The chain, read at source - -1. `buffer_get`'s fix calls `hrx_synchronous_d2h` - (`third_party/llama.cpp` at the pin, `ggml/src/ggml-hrx/ggml-hrx.cpp`); -2. `hrx_synchronous_d2h` issues a **real HAL download**, not a host read — - `third_party/hrx-system/libhrx/src/libhrx/transfer.c:118-148` builds - `IREE_HAL_TRANSFER_OPERATION_TYPE_DOWNLOAD` and calls `hrx_hal_queue_transfer_and_wait`, returning - before any host-side copy; -3. the amdgpu host queue turns that into a **staging** download — - `runtime/src/iree/hal/drivers/amdgpu/host_queue_transfer.c:224-231` calls - `iree_hal_amdgpu_staging_transfer_create_download` for the DOWNLOAD case; -4. the staging pool the download lands in is created **HOST_COHERENT** — - `runtime/src/iree/hal/drivers/amdgpu/host_queue_staging.c:427`: - - IREE_HAL_MEMORY_TYPE_HOST_LOCAL | IREE_HAL_MEMORY_TYPE_HOST_COHERENT | - IREE_HAL_MEMORY_TYPE_DEVICE_VISIBLE - -So the destination of the fixed read is coherent host memory, and the device→host leg is a queued -transfer rather than a raw pointer dereference. The fix does what it claims: it removes the raw read of -the non-coherent `0x52` logits mapping. - -## Why that matters - -The one way arm B's fault could have been dismissed was "the fix didn't actually change coherence, so -its failure proves nothing." The source rules that out. Therefore arm B's cell-`e` fault (if the -remaining rounds confirm it) is genuine evidence that **fixing the coherence of the logits read does -not stop the fault** — the corruption is not the CPU observing a stale logits page. - -It also rules out the simple inverse reading ("the read path was fine all along, so `0x52` was -harmless"): the allocation genuinely lacks `HOST_COHERENT` while the fix's path genuinely supplies a -coherent one. The difference is real; it just is not where the fault comes from. - -## Scope - -This file cites the runtime at `~/wt/validate-329/third_party/hrx-system` on strixhalo, which is a -working tree, not the engine's pinned submodule revision. The pin must be re-checked at its own -revision before these line numbers are used in KNOWN-ISSUES; the *fact* that the staging pool is -created `HOST_COHERENT` is the load-bearing claim, and it should be re-read at the pin when the reading -is finalized. diff --git a/dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md b/dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md deleted file mode 100644 index c2581a8de..000000000 --- a/dev/315/315-campaign/evidence/315-ab1-two-drivers-incident.md +++ /dev/null @@ -1,174 +0,0 @@ - - - -# Incident: two drivers on one tag — arm B's first attempt (`315ab1b`) is discarded - -Recorded because it is the same class of trap the issue has sprung before ("do not weaken or contaminate -the apparatus"), and because the discarded attempt must not be mistaken for a measurement. - -## What happened - -Arm B was launched at 10:22:33 under tag `315ab1b` (one driver, box lock held, patched binary, knob ON). -At 10:24:45 a **second** `315-disc.sh 4 315ab1b` session started on the same box, writing to the SAME -console (`~/wt/315ab1b-console.log`) and the same tag. - -The second launch line was not issued by this session: - - cd ~/wt && BIN=$HOME/wt/315coherent-build/bin \ - EXTRA_ENV="GGML_HRX_TENSOR_FP=tail:16 GGML_HRX_TENSOR_FP_NANONLY=1 GGML_HRX_COHERENT_READ=1" \ - setsid nohup bash 315-disc.sh 4 315ab1b > ~/wt/315ab1b-run.log 2>&1 < /dev/null - -It carries the exact arm-B parameterisation that the pre-registration and the session-state file name, -so it was almost certainly a **peer #315 session** on the shared box picking up the published tag/name, -rather than a random process. No cron/timer/rsync hook references `315-disc.sh`; `~/.bash_history`'s -only 315 line is an older `evictonly` one-liner. - -## Why it is discarded rather than repaired - -Two drivers on one tag make the record ambiguous by construction: - -- the two sessions share one console file, so their per-arm lines interleave and the driver's - `grep -c` summaries double-count; -- both drive the same eviction rig and the same GPU, so they contend for the regime and for HRX0; -- the `summaries` ("cell e: OK=… FAULT=…") are computed by grepping the shared console, so **neither** - session's summary is trustworthy once a second writer exists. - -There is no way to attribute an arm to a session after the fact, so the correct disposition is discard, -not salvage. The tag `315ab1b` is retired and must not be cited. - -## What was NOT lost - -The second session never reached an arm: it wrote only its header line -(`=== disc session 2026-10-13T10:24:45-03:00 tag=315ab1b rounds=4 gtt_before=99G tctl=82C …===`) plus -its own neighbour/regime lines, and was stopped before any `round N arm X:` line appeared. So: - -- no arm was scored under the contention; -- **arm A (`315ab1a`) is untouched** — it completed at 10:20:23, before the collision, under a single - driver, and its result (`dev/315/315-campaign/evidence/315-ab1-arm-A-result.md`) stands; -- the patched binary and the patch file are unaffected. - -## Disposition - -- Arm B re-run under a **fresh tag `315ab1b2`** at 10:25:24, same binary and same rig, single driver, - box lock held. -- The re-run's launcher carries a **start guard** that refuses to begin if any other 315 driver or - patched-build server is alive, so the collision cannot recur: - - n=$(pgrep -fc "315-disc.sh|315coherent-build/bin/llama-server"); [ "$n" -gt 0 ] && exit 4 - -- Lesson re-recorded (extends the existing "never build from a copied build dir" trap): **one tag, one - driver, on a shared box.** Publish a tag only after the run it names has started, or use a tag that - cannot be guessed and re-used by a peer session. - ---- - -# CORRECTION (2026-10-13 10:50) — THIS INCIDENT WAS A MISDIAGNOSIS. There was no second session. - -This file is retained rather than deleted, because the false record is itself instructive — but its -central claim is **wrong** and must not be cited. - -## What was actually seen - -`pgrep -af "315-disc.sh 4 315ab1b2"` matched **two** pids: - - 2033289 bash /home/bcloud/wt/315-disc.sh 4 315ab1b2 # the driver - 2181579 bash /home/bcloud/wt/315-disc.sh 4 315ab1b2 # <- read as a second session - -## What it actually was - -`2181579`'s `ppid` is **2033289** — it is the driver's **own child**, the per-arm subshell, which -carries the same command line by construction. Same for the earlier "duplicate": the shape that looked -like "a second driver on one tag" is what one driver looks like under `pgrep`, because the parent shell -and its per-arm child share the command line. - -The corroborating evidence that *should* have caught this and did not: - -- `pgrep -P 2033289 -a` lists the driver's children, and the arm server (port 19309, outdir - `315ab1b2-r2q`) plus its detector sit under a `315-disc.sh` child — the normal arm structure; -- the "second session" never had its own console, its own lock, or its own outdir — it had none of the - things a real second session would need. - -## What was also true, and is the real lesson - -`/tmp/... box.lock` is held by the *single* driver and its children, and the run was never contended. -No arm was ever scored by a competing writer, so the arm-A result and the arm-B run were **never** -endangered, and the "discard tag 315ab1b" decision was unnecessary. `315ab1b` was abandoned for no -reason; only its own incompleteness (it was superseded by the re-run) matters. - -## Which of this file's lessons survive - -Only one, and it is a different one than written: **a `pgrep` count of a recursive shell script is not -a count of sessions — check `ppid` before concluding contention.** The original "one tag, one driver on -a shared box" advice is still sound practice, but it was not the thing that happened here. - -## Disposition - -- `315ab1b2` is the arm-B run of record and was never contended; its results stand. -- No further action on the "collision". The `315ab1b2` launch guard is harmless and may stay. -- The goal was, however, genuinely being driven from two front-ends (this session and the `pi-315` tmux - pane, both owner `01a1356c`) — that is a real coordination hazard, but it did not cause this. - ---- - -# SECOND CORRECTION (2026-10-13 10:52) — the collision WAS real. Both readings are true. - -The correction above over-corrected. Checking the actual session transcripts (not PIDs) settles it: the -ORIGINAL collision was real, and only the *later* double-`pgrep` was a misreading. - -## The evidence - -Two Pi sessions exist on this host, both working this goal: - - session 01a1356c-790a-70c4-aa0a-f69a5b05ebb9 (this session; goal owner) - session 01a12f94-d804-75df-b710-9bdd6e84958a (a second, concurrent Pi session; tmux pane pi-315) - -session `01a12f94`'s transcript contains the duplicate launch, at the exact collision time: - - 2026-10-13T12:24:05.121Z EXTRA_ENV="... GGML_HRX_COHERENT_READ=1" setsid nohup bash \ - 315-disc.sh 4 315ab1b > ~/wt/315ab1b-run.log 2>&1 < /dev/null - -12:24:05 UTC is **10:24:05 ADT** — the second driver observed at 10:24:45, same tag `315ab1b`, same -knob. It then polled `pgrep -f "315-disc.sh 4 315ab1b"` at 12:25:20 and 12:26:05/12:26:20, i.e. it was -tracking *its own* run of the tag this session had already started at 10:22:33. - -## So both statements hold - -- **The 10:24:45 collision was real**: two different Pi sessions ran tag `315ab1b` concurrently. The - discard of `315ab1b` and the guarded re-run as `315ab1b2` were correct actions. -- **The later "two drivers" reading was wrong**: pid 2181579's ppid was 2033289, the driver's own - per-arm child. That specific observation proves nothing about contention. - -## The generalizable lesson, corrected - - a pgrep COUNT is not a count of sessions -> check ppid - a pgrep count is also not proof of ONE session -> check WHICH session issued the command - -Neither PID inspection nor process counting identifies a competing agent; the session transcripts do. -The collision was only diagnosable from `~/.pi/agent/sessions/*.jsonl`, which is where the launch -command's ownership actually lives. - -## What this means for the goal - -The goal `mv0f7orc-9rf73p` was being pursued by **two concurrent Pi sessions at once** — this session -and `01a12f94` — which is a real coordination hazard independent of any single incident: they share one -goal file, one owner id, one box, and one pre-registered tag namespace. `01a12f94`'s pane now shows -`goal: blocked` (it believes a campaign holds `~/npu.lock`, which is free), and its `/goal-resume` -keystrokes did not submit. Any further work should assume one driver only. diff --git a/dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md b/dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md deleted file mode 100644 index d9c717d44..000000000 --- a/dev/315/315-campaign/evidence/315-acceptance-harness-faceb-defects.md +++ /dev/null @@ -1,128 +0,0 @@ - - - -# Two defects in the acceptance harness found by fixing Face B: an unattributable face, and a bound over 100% - -Found while implementing the user's ruling that Face B is *Content-only / HTTP 500 **or** a genuine -non-UTF-8 body, whichever occurs*. Both defects are in `dev/315/315-campaign/315-campaign.sh`, the acceptance -campaign summary. Only the first is fixed here; the second is recorded because it is a smaller change to -**how a criterion reads** and should not be made silently. - -## Defect 1 — the harness cannot attribute Face B, in two independent ways - -**1a. The wrong condition.** The summariser computed - - if r["parse"] > 0 or r["utf8"] > 0: # <- Face B, as written - ... - face_b = sum(1 for r in faults if r["parse"] > 0 or r["utf8"] > 0) - -but `parse_failures` and `utf8_failures` are **0 in every Face-B arm on the record** — Face B's recorded -form is `http=1 s500=1 unparsed=2` (`315-faceb-invariant-signature.md`). A real Face B fault could therefore -**never** be counted as Face B; it landed in the `unattributed` bucket, whose own text says "read the server -log", i.e. the harness told its reader to go and find by hand what it was supposed to count. - -**1b. The rig computed the fields but did not print them.** The acceptance rig (`315-paired.sh`) already -computes `http`, `transport`, `s500` and `unparsed` — it uses all four in its **verdict** condition -(`... || [ "${http:-0}" -gt 0 ] || [ "${tr:-0}" -gt 0 ] || [ "${s500:-0}" -gt 0 ] || [ "${unr:-0}" -gt 0 ] ...`) -— but its `printf` omitted them, so they never reached the console the summariser reads: - - BEFORE: pair 1 arm c: FAULT nan=0 req=13 ret3=0 parse=0 gtt_start=98G utf8=0 alloc=0 hv52=1 hv56=0 (gtt_now=100G) - AFTER: pair 1 arm c: FAULT nan=0 req=13 ret3=0 parse=0 gtt_start=98G utf8=0 alloc=0 hv52=1 hv56=0 idn=1 http=1 s500=1 unparsed=2 hsa=0 transport=0 (gtt_now=100G) - -**This is empirical, not hypothetical**: `315accept`'s recorded fault — `pair 1 arm c: FAULT nan=0 req=13` with -every counter 0 — is exactly a Face-B-shaped arm, and the harness labelled it `unattributed` because the line -it was given carried no Face-B evidence at all. - -**Impact on the criteria.** Criterion 3 asks for *zero faults on both faces*. The **rate** accounting is -sound — it runs over all in-regime arms, so a fault fails acceptance regardless of face, and the test above -confirms `ACCEPTANCE FAILS` still fires. What is broken is the **face attribution**, which criterion 1 needs -(a named mechanism for *both* faces) and which any "zero on both faces" reading depends on. A campaign could -pass while Face B was never once attributed. - -**Fix applied to the summariser** (the part that can be fixed here): - -- it now parses `http=`, `s500=`, `unparsed=` from the tail **when present**, storing `None` when absent so a - missing field can never masquerade as a zero; -- `is_face_b(r)` tests `parse > 0 or utf8 > 0 or http > 0 or s500 > 0 or unparsed > 0` — the disjunction the - user ruled for, covering both the recorded form and the objective's full-body UTF-8 form; -- when the fields are absent it now prints an explicit note that Face B cannot be tested from that line, - instead of silently bucketing the fault as unattributed. - -**Verified on both branches**, running the extracted summariser: - - Test A (line WITHOUT the fields, from 315accept-console.log): - NOTE: this line carries no http/s500/unparsed field, so Face B's recorded form cannot be tested - from it - the paired rig does not emit them. ... - face tally: FaceA=0 FaceB=0 unattributed=1 - ACCEPTANCE FAILS: 1 in-regime arm(s) faulted; ... - - Test B (a line WITH http=1 s500=1 unparsed=2): - FAULT pair=1 FaceB(parse=0,utf8=0,http=1,s500=1,unparsed=2) req=7 gtt=98G - FAULT pair=2 unattributed - FAULT with nan=0 and no parse/utf8 counter; ... - face tally: FaceA=0 FaceB=1 unattributed=1 - -So the corrected test attributes a real Face-B signature and does **not** over-attribute an all-zero fault. - -**Also fixed, in the rig** (`315-paired.sh`): its `printf` now emits -`... idn=%d http=%d s500=%d unparsed=%d hsa=%d transport=%d (gtt_now=%dG)`. The new fields are placed **after** -`gtt_start`, so the summariser's existing `(.*)$` tail capture picks them up with no format change there. - -**Verified end-to-end on the patched rig line format:** - - pair=1 (nan=0, http=1 s500=1 unparsed=2) -> FaceB(parse=0,utf8=0,http=1,s500=1,unparsed=2) - pair=2 (nan=2048, http=1 s500=0 unparsed=2) -> FaceA(nan=2048) FaceB(http=1,s500=0,unparsed=2) - pair=1 arm l (all counters 0) -> OK - face tally: FaceA=1 FaceB=2 unattributed=0 - -The second row is the case the criterion actually cares about — **both faces attributed on the same arm** — -and the third confirms an all-zero arm is still not over-attributed. With the fields present the summariser no -longer prints the "cannot be tested from this line" note, so that note now fires only on a console produced by -the old printer. - -## Defect 2 — the "rule of three" bound prints probabilities above 100% - -`315-campaign.sh` prints, for a clean cell of `n` arms: - - one-sided 95% upper bound (rule of three, 3/n) = {3.0/n*100:.2f}% - -which for small `n` is **not a probability**: - -| n | printed | -|---|---| -| 1 | **300.00%** | -| 2 | **150.00%** | -| 3 | 100.00% | -| 60 | 5.00% (correct and the useful case) | - -The rule of three is only a bound for `3/n ≤ 1`, i.e. `n ≥ 3`. For the acceptance criterion's `n = 60` the -number is right (5%, matching the objective's own power statement), so this does not affect the intended -reading — but a partial or aborted campaign printing `300%` invites a reader to take the bound as satisfied. -**Not changed here:** capping it is trivial (`min(100.0, 3.0/n*100)`) but deciding what an under-powered -campaign should *say* is a criterion question, and the objective is immutable. - -## Evidence - - sed -n '52,152p' dev/315/315-campaign/315-campaign.sh > /tmp/sum.py # the extracted summariser - python3 /tmp/sum.py /tmp/t_paired.txt # test A - python3 /tmp/sum.py /tmp/t_disc.txt # test B - -Both test consoles and the full outputs are reproduced in this session's transcript; the patch is in -`315-campaign.sh` and the summariser is syntax-checked (`ast.parse`) and run on both inputs. diff --git a/dev/315/315-campaign/evidence/315-align-frozen-preregistration.md b/dev/315/315-campaign/evidence/315-align-frozen-preregistration.md deleted file mode 100644 index 9145b6a6e..000000000 --- a/dev/315/315-campaign/evidence/315-align-frozen-preregistration.md +++ /dev/null @@ -1,121 +0,0 @@ - - - -# Pre-registration: the FA alignment question on the FROZEN rig (`315alignfrozen`) - -The FA family is the one still-open lead, and it is open **because the attempt to test it did not reproduce -the trigger**: the peer's `galign`/`galign_repl` used a bespoke composition that produced mostly *clean* -eviction arms, so it could not compare alignments (correctly reported as UNINFORMATIVE — see -`315-galign-CORRECTION-my-parse-was-wrong.md`, and my own parse error that briefly misread those arms as all -faulting). - -The fix is to use the **frozen driver**, which reproduces the trigger at ~95% in cell `e`, and simply vary the -alignment through `EXTRA_ENV`. That is what this pre-registers. - -## Hypothesis - -**H-align-frozen.** If the #140 layout/dispatch class carries the fault, then forcing -`GGML_HRX_FA_PARTIAL_ALIGN=256` (the layout the #140 oracle used to *expose* the divergence) should fault -**more** than the default `4096` on the frozen rig's cell `e`. - -## Design - - driver : ~/wt/315-disc.sh , unchanged - the SAME frozen three-cell rig - BIN : ~/wt/315coherent-build/bin - cells : e (evictonly forcer) is the test cell; q and c remain controls - env : EXTRA_ENV carries the standard detector env PLUS GGML_HRX_FA_PARTIAL_ALIGN - arms : A = align 4096 (default) B = align 256 - rounds : 3 per arm, interleaved by the driver's own rotation - -Note the `EXTRA_ENV` exported to the arm is also inherited by the neighbours, so both are consistent within -an arm; the comparison is between arms. - -## Reading, fixed BEFORE the run - - B (align 256) faults materially more than A (4096) in cell e - -> the #140 layout/dispatch class is implicated; follow the FA substitution as the carrier. - A ~= B (both at the rig's ~95%, overlapping) - -> the alignment does not carry the fault, and the FA family closes. - B clean while A faults - -> contradicted in direction; report as such. - both clean in cell e - -> UNINFORMATIVE (the rig failed to induce); do not read as a null. - -Because the expected cell-`e` rate is ~95%, **both arms faulting is the uninformative-looking but expected -outcome** and does not by itself distinguish the hypotheses — that is the power point recorded earlier. The -discriminating signal would be a **rate difference**, which needs the arm counts below. - -## Arms, and the power problem stated up front - -At ~95% per arm, distinguishing 95% from, say, 60% needs roughly **20 arms per alignment** at 80% power. With -3 rounds per arm the driver gives **3 cell-`e` arms per alignment** — enough to catch a *large* difference -(e.g. 256 clean where 4096 faults) but **not** enough to bound a moderate one. - -So this is explicitly a **coarse** test: it can falsify "the alignment is required" (if 256 faults freely) but -cannot establish "the alignment makes no difference". If 256 faults at the rig's normal rate, the honest -conclusion is *not sufficient*, the same strength as the coherence and GRAPH_SYNC rows — **not** a measured -null. - -## Stopping rule - -3 rounds per arm (the driver's default for this form), or earlier only if one alignment is unanimous in cell -`e` while the other has also produced at least 2 scored cell-`e` arms. No arm pruning; INVALID/OOR arms are -recorded as invalid. - -## What this does and does not settle - -- **Would settle (one-sided)**: whether the FA alignment is *required* for the fault. -- **Would not settle**: whether it makes a *rate* difference, at this arm count. -- **Does not touch criteria 2–4.** - ---- - -# Execution log - -**Arm A (align 4096) launched 2026-10-13T17:28:26**, tag `315alignfa`, 3 rounds, frozen rig. Verified in -flight: the session header carries -`extra_env='GGML_HRX_TENSOR_FP=tail:16 GGML_HRX_TENSOR_FP_NANONLY=1 GGML_HRX_FA_PARTIAL_ALIGN=4096'` and the -regime composed to GTT 98 GiB before the first arm — i.e. the faulting band, on the driver that reproduces the -trigger. - -**Arm B (align 256) staged** as `~/wt/315alignf-b.sh` with the same shape and only the alignment differing, -to run after arm A releases the box. - -## Arm A (align 4096) — COMPLETE - - 315alignfa, 3 rounds, frozen rig, GTT 98-99 GiB throughout, all arms in-regime, zero OOR/INVALID - - cell q: OK=2 FAULT=1 - cell e: OK=0 FAULT=3 <- every eviction arm faulted - cell c: OK=3 FAULT=0 - -Cell `e` at **3/3 FAULT** — the frozen rig behaving exactly as its record says (~95% under the forcer), with -align at the default 4096. The single cell-`q` fault is the drift-into-band effect already recorded, not a -forcer result (`side=0/0`). - -**Arm B (align 256) launched 18:12:28**, tag `315alignfb`, verified: its session header carries -`GGML_HRX_FA_PARTIAL_ALIGN=256` and the regime composed to GTT 100 GiB. - -So the comparison now has a valid control: if align 256 is **clean** in cell `e` where 4096 faulted 3/3, the -alignment is required (a large, detectable difference despite the small n). If 256 also faults ~3/3, the -alignment is **not required** — a one-sided "not sufficient", not a measured null. diff --git a/dev/315/315-campaign/evidence/315-align-frozen-result.md b/dev/315/315-campaign/evidence/315-align-frozen-result.md deleted file mode 100644 index 0056ee8ff..000000000 --- a/dev/315/315-campaign/evidence/315-align-frozen-result.md +++ /dev/null @@ -1,79 +0,0 @@ - - - -# The FA alignment test on the FROZEN rig: A and B are IDENTICAL — alignment is not required - -The pre-registered test of the last open family, run on the driver that actually reproduces the trigger -(the earlier attempt used a bespoke composition that stayed mostly clean and was therefore uninformative). - -## Result - -Same binary, same frozen three-cell rig, only `GGML_HRX_FA_PARTIAL_ALIGN` differing: - - cell q cell e cell c - arm A 2 OK / 1 FAULT 0 OK / 3 FAULT 3 OK / 0 FAULT align 4096 (default) - arm B 2 OK / 1 FAULT 0 OK / 3 FAULT 3 OK / 0 FAULT align 256 (#140 oracle layout) - -**Byte-for-byte the same tally.** Cell `e` — the forced-eviction test cell — faulted **3 of 3 arms in both -arms**, at `gtt_start` 98 GiB, zero OOR, zero INVALID, with `side=1/1` on every cell-`e` arm. Both arms also -show the same single cell-`q` fault (the recorded drift-into-band effect, `side=0/0`). - -## Validity, verified rather than assumed - -- **1096 vs 256 genuinely applied**: each arm's session header records the value - (`GGML_HRX_FA_PARTIAL_ALIGN=4096` for A, `=256` for B), and both composed to GTT 98-100 GiB before their - first arm. -- **Both arms in-regime** at `gtt_start=98G` on every scored arm; no OOR or INVALID in either. -- **The forcer was live** on every cell-`e` arm (`side=1/1`). - -## The pre-registered reading - - A ~= B (both at the rig's ~95%, overlapping) - -> the alignment does not carry the fault, and the FA family closes. - -Met exactly. And per the pre-registration's own power note, this is a **one-sided** conclusion: - -> **the FA layout/dispatch class is not REQUIRED** for the fault — `align=256`, the layout the #140 oracle -> used to *expose* the divergence, faults at the same 3/3 as the default. - -It is **not** a measured null: 3 cell-`e` arms per alignment cannot bound a *rate* difference (distinguishing -95% from 60% needs ~20 per condition). So this lands at the **same strength as the coherence and `GRAPH_SYNC` -rows**: the candidate is excluded as sufficient, not shown to be irrelevant. - -## What it resolves about the earlier confusion - -The FA family was left OPEN because (a) the first attempt used a composition that did not reproduce the -trigger and (b) I had briefly misread its arms as all-faulting via a `md5sum`-over-streaming parse error. This -run removes both problems: the frozen driver reproduces at 3/3, and the verdict comes from the rig's own -classifier rather than a hand-rolled one. - -## Net effect on the closed set - - logits read path REFUTED its own pre-registered A/B - host side CLOSED none on the decode path - runtime AQL submission policy CLOSED the runtime does not infer hazards - graph reorder REFUTED cell e 4/4 in BOTH arms - deferred host writeback CLOSED binding host_data == nullptr - memory-pool quarantine/reuse CLOSED no-op trim, fresh-only acquire, immediate release - FA alignment/dispatch (#140) NOT REQUIRED this run: 3/3 both alignments, frozen rig - -**The cause remains unlocated**, now with the FA family excluded as sufficient on the rig that reproduces the -fault — which is a stronger exclusion than the earlier, mis-parsed one. diff --git a/dev/315/315-campaign/evidence/315-align-preregistration.md b/dev/315/315-campaign/evidence/315-align-preregistration.md deleted file mode 100644 index 5deda97fa..000000000 --- a/dev/315/315-campaign/evidence/315-align-preregistration.md +++ /dev/null @@ -1,122 +0,0 @@ - - - -# Pre-registration: is the fault the #140 FA layout/dispatch class? (`315align`) - -The one discriminator the record names as cheapest-and-never-applied. §57 of the mechanism doc says it -directly: - -> "The cheapest remaining discriminator is still the one the project's own #123 work used and #315 has -> never applied: `GGML_HRX_FA_PARTIAL_ALIGN` … If the fault rate rises when the partial alignment is -> lowered, the fault IS the #140 layout-sensitive class and its surviving instance is at a boundary the -> fences do not cover; if the rate is unchanged, the #140 class is not #315's mechanism and the search -> should leave the FA path entirely." - -That is already a pre-registered form. This file fixes the design, the arms and the reading **before** -running it, and records what the prior work already found so the new run does not repeat it. - -## What the knob is (read at source, not assumed) - -`ggml/src/ggml-hrx/dispatch_registration/common/dispatch-flash-attention.cpp:23`: - - // Test knob for the decode-split partial transients' alignment. Production is 4096; - // the engine#123/#140 rig oracle used 256, a layout that exposes the divergence. - // Defaults to 4096 so production behaviour is unchanged unless explicitly requested. - -So `GGML_HRX_FA_PARTIAL_ALIGN=256` restores the layout the #140 oracle used. It is **not** a pure -alignment change: per the mechanism doc, at `align=256` the dispatcher substitutes -`common.flash_attention_f32_f16_wmma` (the **prefill-shaped** kernel) for the decode-split kernel, and -`GGML_HRX_LOG_DISPATCH=1` shows the substitution. So the intervention has two co-varying effects — -alignment AND kernel selection — and any result must be read with that in mind. - -## What the prior work already established (do not re-derive) - -From `dev/315/315-mechanism.md`, the `align=256` + compaction recipe on **Qwen3-0.6B**: - - default align, quiet 1 run clean 180/180 - default align + compaction 2 runs clean 180/180 both - align=256, quiet 1 run clean 180/180 - align=256 + compaction 5 runs 1 faulty (95 distinct of 180), 4 clean (180/180) - -and the honest correction recorded there: five runs do **not** support calling that a reproducer — it -fired once, and the one faulty run coincided with foreign box load (8-12) while the four clean repeats ran -quieter (4-7). So: **a preemption-sensitive wrong-output event is possible at the pin at align=256 under -contention, and it is not yet controllable.** All Face B (`nan=0`); nothing in that recipe produced -Face A. - -Also established: `GGML_HRX_DISABLE_DISPATCH=kquant.mul_mat_add.decode_f32` did not stop it (68 distinct -of 180), so the matmul-add fusion is not the carrier; what is left is the FA substitution. - -## Design — deliberately different from the prior recipe, to make it controllable - -The prior recipe's weakness was that contention was **borrowed** from a foreign session. This design -creates it with the goal's own rig, which is what makes the rate measurable rather than incidental. - - rig : 315-disc.sh style, but with the GLM eviction rig's *contention* and the small model - model : Qwen3-0.6B (378 MB — no regime needed, per §57) - cells : e (evictonly forcer) vs q (quiet), the same forcer the GLM arms use - arms : A = align default (4096), B = align 256; each in both cells - detector: the cheap probe's identity check (N identical greedy requests in one process) PLUS the - NaN-logits guard, so both faces are counted - -The point of using the **same forcer** as the GLM arms is that it makes the contention deliberate and -equal across cells, removing the load-confounded reading the prior attempt was stuck with. - -## Reading, fixed BEFORE the run - - align=256 faults at a materially higher rate than align=4096 in cell e, and align=256 is (near) clean in cell q - -> the #140 layout/dispatch class IS implicated, and the surviving instance is at a boundary the - fences do not cover. Follow the FA substitution (prefill-shaped kernel) as the carrier. - align=256 ~= align=4096 (both fault, or both clean, in cell e) - -> the #140 class is NOT #315's mechanism. Per §57's own instruction, **leave the FA path - entirely** and do not spend further budget there. - align=256 clean in cell e while 4096 faults - -> contradicted in direction; report as such (it would suggest the layout knob is protective, - which would be informative about the substitution). - -## Why this is worth doing at all, given four families are already closed - -It is the **only** discriminator on the record that is cheap (a 378 MB model, no regime), has never been -run against the fault with a controlled forcer, and can retire a whole model family — the FA/flash-attention -path — either way. A null here is as valuable as a hit, because §57 pre-commits to leaving the FA path if -the rate does not move. - -## Why NOT run yet (recorded rather than silently deferred) - -`~/npu`/GPU access is currently held by a **peer agent's campaign** (`315text1`, a full eviction-rig run -with 3 neighbours at GTT 79G, still in flight). GTT is at ~100 of 120 GiB, so: - -- the probe would add GTT into roughly 20 GiB of headroom, risking an allocation failure for the peer's - regime; -- it would perturb the very memory pressure the peer is measuring; -- the goal's constraints require taking the box lock and **not disturbing other workloads on the shared - box**, and the peer's driver does not take the lock, so the lock cannot protect them. - -So this is deferred until the peer's campaign ends, exactly as arm B of the graph-reorder A/B was. The -pre-registration exists now so the run is ready and the reading is fixed in advance. - -## Cost - -~10-20 minutes of GPU (one 378 MB model, two cells, both alignments, ~2 x N requests each). This fits the -remaining budget where a full GLM campaign would not. diff --git a/dev/315/315-campaign/evidence/315-align-run1-uninformative.md b/dev/315/315-campaign/evidence/315-align-run1-uninformative.md deleted file mode 100644 index ecd744415..000000000 --- a/dev/315/315-campaign/evidence/315-align-run1-uninformative.md +++ /dev/null @@ -1,104 +0,0 @@ - - - -# `315align` run 1 — executed, but UNINFORMATIVE. Do not read it as a null. - -The pre-registered FA alignment/dispatch discriminator ran at 15:16:23 and completed in **38 seconds**. -Recording it as uninformative rather than as a null, because the condition it needs was never established. - -## What it produced - - === 315align start 2026-10-13T15:16:23-03:00 gtt=47GiB === - arm align4096-e align=4096 cell=e responses=120 distinct=1 nan=0 - arm align256-e align=256 cell=e responses=120 distinct=1 nan=0 - arm align4096-q align=4096 cell=q responses=120 distinct=1 nan=0 - arm align256-q align=256 cell=q responses=120 distinct=1 nan=0 - === 315align done 2026-10-13T15:17:01-03:00 === - -All four arms: every one of 120 responses identical (`distinct=1`), no NaN. The `align=256` + eviction cell — -the cell the #140 oracle used — was clean. - -## Why this does NOT retire the FA path - -The pre-registered branch for "align=256 ~= align=4096, both clean" is "the #140 class is NOT #315's -mechanism; leave the FA path". **That branch does not apply, because the arms could not have faulted.** - -1. **No regime, no neighbours, no compaction.** GTT was **47 GiB of 120 GiB** — low pressure by - construction. The prior #140 recipe (`dev/315/315-mechanism.md`) required `align=256` **plus** compaction - (`tee /proc/sys/vm/compact_memory`) and still fired only once in five runs, and the record notes that - the one faulty run coincided with foreign load 8-12 while the four clean repeats ran at 4-7. -2. **38 seconds is too short.** Four arms in 38 s gives ~10 s per arm. The eviction window this class needs - opens on queue preemption under sustained pressure; a 10-second arm does not open it. -3. **The forcer alone was never shown sufficient for this small-model path.** `evictonly` drove the GLM rig - to 85-96% fault rates, but that was under the full regime (3 neighbours, GTT 97-100 GiB). Whether it - opens the window at 47 GiB with a 378 MB model is exactly what was *not* tested. - -So the run is **uninformative by the pre-registration's own branch 3** ("both clean in cell e → -UNINFORMATIVE … most likely the rig failed to induce"), applied to the FA probe: a clean result under -conditions that could not produce the event says nothing about the event. - -## What the run DID establish - -The probe harness works mechanically: it starts a 378 MB model on HRX0, drives 120 identical greedy -requests per arm, counts distinct responses and NaN lines, and tears down cleanly — in ~10 s per arm. That -makes it a usable instrument once the conditions are right. - -## What a decisive version needs - -Match the prior recipe's conditions rather than a simplified version of them: - - align=256 vs align=4096 - cell e = evictonly 500 + compact_memory looping + the GLM neighbours for GTT - requests >= 180 per arm (the prior recipe's N), not 120 short ones - arms several per cell, since the prior rate was 1 in 5 under ideal conditions - -That is a materially bigger run — the prior work spent an evening on it and got one event — and it should -be attempted only with the box-time budget to spend, not as a 38-second probe. - -## Bookkeeping - -The harness has been improved in place (`~/wt/315align-run.sh` should gain the compaction loop and the -neighbour composition before any re-run). This file exists so the 38-second run is not later cited as -evidence that the FA layout class is innocent. - ---- - -# Take 2 is running with the conditions take 1 lacked - -Revised harness (`~/wt/315align2.sh`), started 15:28:54, matching the prior recipe's conditions rather than -simplifying them: - -| | take 1 (uninformative) | take 2 (running) | -|---|---|---| -| GTT | 47 GiB | **117 GiB** (3 GLM neighbours at `-c 2048` composed first) | -| requests/arm | 120 short (`messages`, 1 word) | **180 in the arms' own shape** — `/completion`, `cache_prompt=false`, `n_predict=96` | -| eviction | `evictonly` alone | `evictonly 500` **+** a `compact_memory` loop | -| arms | 1 per cell | 6 (two repeats of the 4096/256 cell-`e` contrast, since the prior rate was 1 in 5 even under ideal conditions) | -| duration | 38 s total | ~3 min per arm | -| box lock | not taken | taken (`flock -w 1800`) | - -Verified in flight rather than assumed: neighbours up and GTT at 117 GiB before the first arm; the first -arm's server processing tasks (148 of 180 at the check) with `evictonly` alive and -`/proc/sys/vm/compact_memory` present. - -The reading is unchanged from the pre-registration: if `align=256` faults materially more than `align=4096` -in cell `e`, the #140 layout/dispatch class is implicated; if they are equivalent, §57 pre-commits to -leaving the FA path entirely. diff --git a/dev/315/315-campaign/evidence/315-align-run2-result.md b/dev/315/315-campaign/evidence/315-align-run2-result.md deleted file mode 100644 index 308d910b4..000000000 --- a/dev/315/315-campaign/evidence/315-align-run2-result.md +++ /dev/null @@ -1,139 +0,0 @@ - - - -# `315align` take 2 — six arms, all clean; the compaction ingredient was silently absent - -Ran 15:28:54–15:41:48 with take 1's gaps closed in three respects and **one of them silently failed**. Both -the result and the defect are recorded, and the defect bounds what the result can support. - -## The result - - arm a4096-e align=4096 cell=e 180 requests distinct=1 nan=0 gtt 99 GiB - arm a256-e align=256 cell=e 180 requests distinct=1 nan=0 gtt 99 GiB - arm a4096-q align=4096 cell=q 180 requests distinct=1 nan=0 gtt 81 GiB - arm a256-q align=256 cell=q 180 requests distinct=1 nan=0 gtt 81 GiB - arm b256-e align=256 cell=e 180 requests distinct=1 nan=0 gtt 81 GiB <- repeat - arm b4096-e align=4096 cell=e 176 requests distinct=1 nan=0 gtt 81 GiB <- repeat - -**Every arm: 180 (resp. 176) responses, all identical, zero NaN.** Including `align=256` in the eviction -cell, run twice. - -## What take 2 fixed over take 1 - -| | take 1 | take 2 | -|---|---|---| -| GTT | 47 GiB | **99 GiB** during the cell-`e` arms (3 GLM neighbours composed first) | -| request shape | 120 short one-word messages | **180 × `/completion`, `cache_prompt=false`, `n_predict=96`** — the arms' own shape | -| cell `e` side load | `evictonly` alone | `evictonly 500` **+ a compaction loop** | -| arms | 4 | 6, with the 4096/256 cell-`e` contrast repeated | -| duration | 38 s | ~13 min | - -## The defect: the compaction loop never wrote anything - -`/proc/sys/vm/compact_memory` was **not writable** during the run (`compact_available=no`), and my loop was: - - while :; do echo 1 > /proc/sys/vm/compact_memory 2>/dev/null || \ - sudo -n tee /proc/sys/vm/compact_memory >/dev/null 2>&1; sleep 2; done - -The first path fails (root-owned file). The second runs `sudo -n tee` **with no input on stdin**, so `tee` -writes an empty line rather than `1` — i.e. it never compacts anything. `sudo -n` itself works on this box -(verified afterwards: `sudo -n sh -c "echo 1 > /proc/sys/vm/compact_memory"` **succeeds**), so this was a -harness bug, not a permission limit. - -**Consequence:** the prior #140 recipe's *second* ingredient was absent. That recipe recorded 1 fault in 5 -runs at `align=256` **with** compaction, and explicitly noted the one faulty run coincided with foreign load -8-12 against clean repeats at 4-7 — so compaction and contention are not obviously optional. - -## What the run does and does not support - -- **Does support**: at `align=256` versus `align=4096`, in the arms' own request shape, at GTT 99 GiB, with - the eviction forcer active, and over two repeats of the cell-`e` contrast — **the wrong-output signature - did not appear in either** (all 180 responses identical, `nan=0`). -- **Does not support** retiring the FA path. The pre-registration's branch for "both clean" presumes the rig - could have induced the event, and one of the two ingredients the only prior positive run used was missing. - So this is **again closer to uninformative than to a null**, though it is a much stronger attempt than - take 1: the shape, the GTT band and the forcer were all right. - -## What a decisive version needs - -Exactly one fix: make the compaction path actually write, e.g. - - while :; do sudo -n sh -c 'echo 1 > /proc/sys/vm/compact_memory'; sleep 2; done - -and keep everything else from take 2. Given `sudo -n` works, that is a one-line change and a re-run of the -same six arms (~13 min). - -## Bookkeeping - -The harness bug is recorded here rather than silently patched, and the take-1 file's "what a decisive -version needs" list is now superseded by this one — the remaining gap is the compaction write, not the -regime, the shape or the arm count. - ---- - -# Take 3 launched with the compaction write verified - -The one-line fix from the section above was applied and the run launched at 15:48. - - start_comp(){ ... echo 1 > /proc/sys/vm/compact_memory 2>/dev/null || \ - sudo -n sh -c "echo 1 > /proc/sys/vm/compact_memory" >/dev/null 2>&1; ... } - -**Verified before launch and again during the run**, rather than assumed: -`sudo -n sh -c "echo 1 > /proc/sys/vm/compact_memory"` returns **exit 0** — i.e. compaction is actually -triggered — and the loop process is live (`pgrep -f compact_memory` → the `while :` shell). GTT at the -launch was 98 GiB with 5 GLM neighbours resident. - -Everything else is take 2's design unchanged: `align` 4096 vs 256, cells `q` and `e`, `evictonly 500`, -180 × `/completion` with `cache_prompt=false` and `n_predict=96` (the arms' own shape), and the cell-`e` -contrast repeated — six arms, ~13 min. - -Result pending; recorded here so the launch conditions are durable rather than reconstructed later. - ---- - -# Take 3 stood down: the peer's GLM-based variant supersedes it - -Before take 3 could acquire the box lock it was blocked by `/tmp/galign.sh` (ports 20362-20364), which is -**the other agent's own alignment experiment on GLM**. Its header names its three corrections, and they are -corrections to *my* run: - -> "Fixes the peer's three defects: **GLM not Qwen**; the rig's classification not a bespoke parser; and it -> tests what the knob **CAN** do (bypass vs present) rather than a stress it cannot produce." - -That is a better instrument than mine on every axis that matters: - -| | my take 2/3 | the peer's `galign` | -|---|---|---| -| model | Qwen3-0.6B | **GLM-4.7-Flash** — the model that actually faults | -| classification | my own `distinct_content` parser (which was broken on take 2 and reported 0 for every arm) | **the frozen rig's own classifier** | -| what the knob is asked to do | produce the fault under stress | **test what it can actually do — bypass vs present** | - -My take 3 was stood down rather than queued behind it, because running the same question on a weaker model -would burn the box to re-derive a weaker answer. Verified afterwards that the runner was killed and the -peer's process left untouched. - -## The lesson this repeats - -This is the third time in this stretch that a peer's method outperformed mine on the same question (the -writeback probe shape, and now the alignment model + classifier). The pattern is consistent: **use the -faulting workload and the established classifier, not a convenient substitute.** My Qwen runs were cheaper -and produced two uninformative results and one broken parse; the peer's GLM run costs more and asks the -question the issue actually poses. diff --git a/dev/315/315-campaign/evidence/315-aql-publish-vs-reorder.md b/dev/315/315-campaign/evidence/315-aql-publish-vs-reorder.md deleted file mode 100644 index e1d6d0fc3..000000000 --- a/dev/315/315-campaign/evidence/315-aql-publish-vs-reorder.md +++ /dev/null @@ -1,159 +0,0 @@ - - - -# The AQL publish boundary, read at source — and how the graph reorder interacts with it - -Found by reading the runtime after another agent's full HRX decompilation/dossier -(`~/work/hrx-system/HRX-SYSTEM-DOSSIER.md` §5.2) pointed at the exact files. This is read at **source** in -`~/work/hrx-system/runtime/src/iree/hal/drivers/amdgpu/`, which is the vendored IREE fork the engine's -`hrx-system` submodule tracks. - -## The publication contract, verbatim - -`util/aql_ring.h`, `iree_hal_amdgpu_aql_ring_commit`: - - // Commits a packet by writing its header + setup as a single atomic - // store-release. This makes the packet visible to the CP: the release - // semantics ensure all prior writes to the packet's fields (and to kernarg - // memory) are ordered before the CP can read them. - ... - iree_atomic_store((iree_atomic_int32_t*)packet, (int32_t)header_setup, - iree_memory_order_release); - -and `iree_hal_amdgpu_aql_ring_doorbell`: - - // The doorbell is purely a wakeup hint — the CP will process any packet - // whose header is valid, regardless of whether the doorbell has been rung. - -**Two consequences that matter for this issue:** - -1. A packet becomes **live to the CP at commit**, not at the doorbell. The doorbell only wakes the CP; - it does not gate visibility. -2. Therefore, during the commit loop the CP may already be executing packets that have been committed - while later packets in the same submission are still being constructed. - -## Where commits happen, and in what order - -`host_queue_command_buffer_block.c`: - - 494 commit (queue-device-event start timestamp packet) - 584 for (uint32_t i = 0; i < emitted_packet_count; ++i) - 587 if (header_type(packet_headers[i]) != INVALID) - 589 iree_hal_amdgpu_aql_ring_commit(packet, packet_headers[i], packet_setups[i]); - 621 commit (barrier packet) - 625 commit (profile harvest packet) - 661 iree_hal_amdgpu_aql_ring_doorbell(...) <- ONE doorbell for the whole batch - -So the payload packets are committed **sequentially in emission order**, and there is a single doorbell -at the end. That is the correct batch discipline — but it means **emission order is the order the CP -observes**, because each commit publishes immediately. - -## The link to the graph reorder - -`record_prepared_command_list(..., ordered.empty() ? prepared.commands : ordered, ...)` is called from -`command-program-executor.cpp:1479` with the list returned by `order_prepared_commands_for_graph`. So when -the reordering engages, **the list recorded — and therefore emitted, and therefore committed in that -order — is the reordered list.** - -Established independently by the zero-arm probe (`315-graph-reorder-lever.md`): GLM's decode has a -26-command routed-FFN graph that IS reordered (23 -> 20 barriers) in every layer, transient extent -237,568 B against the 16 MiB gate. - -Putting the two together: - - reorder decides the emitted order - -> emission order is the commit order (`:584-590`) - -> commit is the point of CP visibility (`aql_ring.h` commit contract) - -> so the CP may consume reordered packets while later ones are still being committed - -The ordering *the graph was recorded in* is deliberately not the ordering the CP sees, for about half of -GLM's per-layer graphs, and the CP can begin on a later-committed packet early. That is exactly the shape -a preemption/eviction window needs to turn into a stale read: the window opens between a packet's commit -and the arrival of a packet whose dependency the reorder moved. - -## What this does NOT establish - -- It does **not** show a missing dependency. The reorder preserves hazard pairs by construction (it only - picks a command once all its recorded dependencies are scheduled — `lookahead_order`), so this is not a - claim that the reorder breaks program correctness. -- It does **not** show the CP-in-flight-during-commit behaviour is unsafe in general. The commit contract - is a standard AQL publication; the question is whether **eviction during the window** changes what a - consumer reads, which is what the A/B tests. -- It is **not** a cause. It is the concrete mechanism by which the reorder lever reaches the device, which - is what made the A/B worth running at all rather than inert. - -## Why it sharpens the experiment - -It gives the A/B a specific, checkable expectation rather than a direction: if the reorder family carries -the fault, the effect should be attributable to the **26-command graph**, and disabling reordering -(`GGML_HRX_GRAPH_ORDER=program`) should remove it while leaving the 9-command attention graph unchanged -(it is already program-ordered, so the knob cannot affect it). - -## Provenance of the read - -The files were located via another agent's dossier at `~/work/hrx-system/HRX-SYSTEM-DOSSIER.md` -(copied into this evidence tree as `315-hrx-system-dossier-by-peer-agent.md` for durability), whose §5.2 -names `util/aql_ring.h:185-203,255-262,277-296` and `host_queue.c:96-99`. Every citation above was then -**re-read at those files directly** in `~/work/hrx-system/runtime/src/iree/hal/drivers/amdgpu/` rather than -taken from the dossier's prose, and the line numbers in this file are the ones observed on re-read (the -commit loop at `:584-590`, the single doorbell at `:661`, the commit contract in `util/aql_ring.h`). - -That dossier is a peer agent's work product, not this goal's evidence, and it is cited as a pointer only. -Its own §11 defects table concerns HIP-layer diagnostics and ABI coverage, not this question. - ---- - -# Kernarg publication checked: the discipline is correct, so the reorder is the only order-changing step - -Following the commit contract's explicit claim that it orders "prior writes to the packet's fields **and -to kernarg memory**", the kernarg path was traced to see whether it introduces its own ordering gap. - -`host_queue_submission.h:210`: - - // Publishes host-populated queue-owned kernargs before committing packet - // headers that reference them. Callers must have already written all kernarg - // bytes for |submission|. - static inline void iree_hal_amdgpu_host_queue_publish_submission_kernargs(...) { - if (submission->kernargs.blocks) { - iree_hal_amdgpu_kernarg_ring_publish_host_writes(&queue->kernarg_ring); - } - } - -and the adjacent comment states the split of responsibility: - - // CPU-side publication and device-side acquisition are independent: coherent host - // memory needs no explicit publication step but still crosses from the host agent to - // the GPU agent. - -The call sites put this **before** the commit loop (`host_queue_command_buffer_block.c:571` publish, -`:584-590` commit), and the acquire side is raised to `IREE_HSA_FENCE_SCOPE_SYSTEM` via -`iree_hal_amdgpu_host_queue_kernarg_acquire_scope`. That is the correct order — publish, then commit, -with the device acquiring at SYSTEM scope. - -**So there is no ordering gap in the publication machinery itself.** The only step that changes the order -the CP observes, relative to the order the graph was recorded in, is the **graph reorder** -(`order_prepared_commands_for_graph`), because it is the only thing that rewrites the command list before -emission. Everything downstream — kernarg publication, commit, doorbell — preserves whatever order it is -handed, and preserves it correctly. - -That sharpens the A/B: with the publication discipline cleared, `GGML_HRX_GRAPH_ORDER=program` isolates -the reorder as the sole order-changing variable, which is exactly the intervention the pre-registration -describes. diff --git a/dev/315/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md b/dev/315/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md deleted file mode 100644 index 4529198fa..000000000 --- a/dev/315/315-campaign/evidence/315-argsort-nonbijection-verified-both-ends.md +++ /dev/null @@ -1,78 +0,0 @@ - - -# Section 51's candidate, verified at both ends in the IR - -Section 51 named a site and a falsifier. Both halves of that claim are re-read here at the pin -(`ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/ops/`), because the candidate is now the survivor -and its two halves should not rest on a single reading. - -## The producer: the rank sort is not a bijection on NaN - -`small_rows_f32.loom`, `ggml_argsort_rows_f32` — rank(i) is computed by scanning the row: - -```loom -%mine = view.load %input_view[%row, %element] : ... -> f32 -%rank = scf.for %other = [%c0 to %cols step %c1](%count = %c0 : index) -> (index) { - %value = view.load %input_view[%row, %other] : ... -> f32 - %greater = scalar.cmpf ogt, %value, %mine : f32 - %less = scalar.cmpf olt, %value, %mine : f32 - %equal = scalar.cmpf oeq, %value, %mine : f32 - %earlier = index.cmp ult, %other, %element : index - %tie_before = scalar.andi %equal, %earlier : i1 - %strict = scf.select %want_descending, %greater, %less : i1 - %before = scalar.ori %strict, %tie_before : i1 - %step = scf.select %before, %c1, %c0 : index - %next = index.add %count, %step : index -} -scf.if %valid { - %element_i32 = index.cast %element : index to i32 - view.store %element_i32, %output_view[%row, %rank] : i32, view<[%rows]x[%cols]xi32> -} -``` - -All three comparisons are **ordered** (`ogt`/`olt`/`oeq`). With `%mine = NaN` every one of them is -false, so `%before` is false for every `%other` and the element contributes nothing to any count. Two -NaN elements in one row therefore receive the **same** rank: `view.store ... [%row, %rank]` writes that -slot twice and leaves some slot in `[0, cols)` never written, holding whatever the buffer held before. -The rank map is a bijection only while the row is NaN-free. **Confirmed as recorded.** - -## The consumer: an assumption, not a check - -`moe_routing_tables.loom:64` consumes a route's expert index with an `index.assume` range, not a clamp: - -```loom -%route_expert = index.assume %route_expert0 [range(%route_expert0, -1, 511)] : index -``` - -and `:45` bounds the expert the same way (`[lt(%expert0, %bounded_expert_count)]`). By contrast the -gather on the same ids **clamps** (`small_rows_f32.loom`, `ggml_get_rows_small_f32`: negatives to 0, -then `in_range` against the source row count). So the same stale index is harmless on one path and -unbounded on the other — the asymmetry section 51 records. **Confirmed as recorded.** - -## What is still missing, and it is the only thing missing - -The candidate's precondition is NaN in the argsort's **input**. Neither half above shows that, and -section 51's own caveat stands: the "probs was finite" reading came from a fingerprint -`ggml_backend_tensor_get`, a host-visible read with no ordering guarantee, so it cannot exclude NaN at -the argsort's input. - -That is precisely what the recorded next probe is for: read `ffn_moe_argsort` / `ffn_moe_topk` **as i32** -(the current probe reads every tensor as f32, so an index is decoded as a denormal float and the signal -is destroyed), anchored names, and per row print the first 8 ids, the count outside `[0, n_expert)`, -and whether the row repeats. **One faulting arm with a non-permutation row names the mechanism; a clean -permutation kills this site.** diff --git a/dev/315/315-campaign/evidence/315-arm-ledger.csv b/dev/315/315-campaign/evidence/315-arm-ledger.csv deleted file mode 100644 index 8759eca38..000000000 --- a/dev/315/315-campaign/evidence/315-arm-ledger.csv +++ /dev/null @@ -1,179 +0,0 @@ -tag,gtt_start,nan_guard_lines,requests,http_fail,idn -315accept-p10c,98,0,66,0,1 -315accept-p10l,98,0,33,1,1 -315accept-p1c,98,0,13,1,1 -315accept-p1l,98,0,66,0,1 -315accept-p2l,98,0,66,0,1 -315accept-p3l,98,0,66,0,1 -315accept-p4c,98,0,66,0,1 -315accept-p4l,98,0,66,0,1 -315accept-p5l,98,0,66,0,1 -315accept-p6l,98,0,66,0,1 -315accept-p7c,98,0,66,0,1 -315accept-p7l,98,0,66,0,1 -315accept-p8l,98,0,66,0,1 -315accept-p9l,98,0,66,0,1 -315arm-p1c,115,0,66,0,1 -315arm-p1l,115,0,66,0,1 -315arm-p2l,115,0,66,0,1 -315arm-p3l,115,0,66,0,1 -315arm-p4l,115,0,66,0,1 -315chain-p1c,98,0,24,1,1 -315chain-p1l,98,0,64,1,1 -315chain-p2l,98,0,66,0,1 -315chain-p3l,98,0,66,0,1 -315chain-p4l,98,0,66,0,1 -315chain-p5l,98,0,51,1,1 -315chain-p6l,106,0,22,0,1 -315det-p1c,98,0,66,0,1 -315det-p1l,98,0,66,0,1 -315det-p2l,98,0,66,0,1 -315det-p3l,98,0,66,0,1 -315diff2-d1,98,0,66,0,1 -315diff2-d2,98,0,66,0,1 -315diff2-d3,98,0,66,0,1 -315diff2-d4,98,0,21,1,1 -315diff2-d5,98,0,,,- -315diff2-u1,98,0,66,0,1 -315diff2-u2,98,0,66,0,1 -315diff2-u3,98,0,66,0,1 -315diff2-u4,98,0,66,0,1 -315diff-d1,115,0,12,0,1 -315diff-d2,98,0,66,0,1 -315diff-u1,98,0,66,0,1 -315diff-u2,98,0,,,- -315dump-p1c,115,0,66,0,1 -315dump-p1l,115,0,66,0,1 -315dump-p2l,109,0,66,0,1 -315fa-p10l,98,0,47,1,1 -315fa-p11l,98,0,66,0,1 -315fa-p12l,98,0,66,0,1 -315fa-p13c,98,0,66,0,1 -315fa-p13l,45,0,66,0,1 -315fa-p14l,45,0,25,0,1 -315fa-p15l,0,0,,,- -315fa-p1c,98,0,66,0,1 -315fa-p1l,98,0,66,0,1 -315fa-p2l,98,1,34,0,1 -315fa-p3l,98,0,66,0,1 -315fa-p4l,98,0,66,0,1 -315fa-p5l,98,1,34,0,1 -315fa-p6l,98,0,66,0,1 -315fa-p7c,98,1,34,0,1 -315fa-p7l,98,0,17,1,1 -315fa-p8l,98,0,66,0,1 -315fa-p9l,98,0,66,0,1 -315fp2-p1c,0,0,,,- -315fp2-p1l,98,0,66,0,1 -315fp2-p2l,98,0,66,0,1 -315fp2-p3l,98,0,66,0,1 -315fp2-p4l,98,0,66,0,1 -315fp2-p5l,98,0,66,0,1 -315gsync2-p1c,115,0,10,0,1 -315gsync2-p1l,0,0,,,- -315gsync-p1c,88,0,66,0,1 -315gsync-p1l,91,1,12,0,2 -315gsync-p2l,90,0,66,0,1 -315gsync-p3l,93,0,66,0,1 -315gsync-p4l,99,0,19,0,1 -315gsync-p5l,87,0,,,- -315hunt2-p10l,98,0,66,0,1 -315hunt2-p11l,98,0,66,0,1 -315hunt2-p12l,98,0,66,0,1 -315hunt2-p13l,98,0,66,0,1 -315hunt2-p14l,98,0,66,0,1 -315hunt2-p15l,98,0,66,0,1 -315hunt2-p1c,98,0,66,0,1 -315hunt2-p1l,98,0,66,0,1 -315hunt2-p2l,98,0,66,0,1 -315hunt2-p3l,98,0,66,0,1 -315hunt2-p4l,98,0,66,0,1 -315hunt2-p5l,98,0,66,0,1 -315hunt2-p6l,98,1,19,0,1 -315hunt2-p7l,98,0,66,0,1 -315hunt2-p8l,98,0,66,0,1 -315hunt2-p9l,98,0,66,0,1 -315hunt-p1c,98,0,66,0,1 -315hunt-p1l,98,0,15,1,1 -315instr-off,98,0,66,0,1 -315instr-on-1,98,0,66,0,1 -315nan-p10l,98,1,10,0,2 -315nan-p11l,98,1,46,0,1 -315nan-p12l,98,1,35,0,1 -315nan-p13l,98,1,41,0,1 -315nan-p14l,98,1,33,0,1 -315nan-p15l,98,1,6,0,3 -315nan-p16l,98,0,66,0,1 -315nan-p17l,98,1,54,0,1 -315nan-p18l,98,0,,,- -315nan-p19l,44,0,5,0,1 -315nan-p1c,98,1,34,0,1 -315nan-p1l,98,1,10,0,2 -315nan-p20l,44,1,22,0,1 -315nan-p2l,98,1,33,0,1 -315nan-p3l,98,1,33,0,1 -315nan-p4l,98,1,33,0,1 -315nan-p5l,98,1,8,0,3 -315nan-p6l,98,1,10,0,4 -315nan-p7l,98,1,33,0,1 -315nan-p8l,98,1,33,0,1 -315nan-p9l,98,1,10,0,4 -315probe2-p10l,98,0,66,0,1 -315probe2-p11l,98,0,66,0,1 -315probe2-p12l,98,0,66,0,1 -315probe2-p1c,98,0,66,0,1 -315probe2-p1l,98,0,66,0,1 -315probe2-p2l,98,0,66,0,1 -315probe2-p3l,98,0,66,0,1 -315probe2-p4l,98,0,66,0,1 -315probe2-p5l,98,0,66,0,1 -315probe2-p6l,98,0,66,0,1 -315probe2-p7l,98,0,66,0,1 -315probe2-p8l,98,0,66,0,1 -315probe2-p9l,98,0,66,0,1 -315probe3-p10l,89,0,66,0,1 -315probe3-p11l,115,0,66,0,1 -315probe3-p12l,115,0,66,0,1 -315probe3-p13l,89,0,66,0,1 -315probe3-p14l,92,0,66,0,1 -315probe3-p15l,115,0,66,0,1 -315probe3-p1c,97,0,,,- -315probe3-p1l,115,0,66,0,1 -315probe3-p2l,115,0,66,0,1 -315probe3-p3l,115,0,65,0,1 -315probe3-p4l,89,0,66,0,1 -315probe3-p5l,115,0,66,0,1 -315probe3-p6l,115,0,66,0,1 -315probe3-p7l,115,0,66,0,1 -315probe3-p8l,115,0,66,0,1 -315probe3-p9l,90,0,66,0,1 -315probe-p1c,98,0,,,- -315row-p1c,98,1,34,0,1 -315row-p1l,98,0,66,0,1 -315row-p2l,98,1,34,0,1 -315row-p3l,98,0,66,0,1 -315row-p4l,98,0,66,0,1 -315sent2-p1c,115,0,,,- -315sent2-p1l,115,0,,,- -315sent3-p1c,80,0,66,0,1 -315sent3-p1l,80,0,66,0,1 -315sent-p1c,115,0,,,- -315sent-p1l,115,0,,,- -315smoke-1,124357271552,0,,,- -315smoke-2,86256316416,0,,,- -315smoke-3,0,0,,,- -315smoke-p1c,98,0,66,0,1 -315smoke-p1l,98,0,66,0,1 -315sync2-p1c,98,0,66,0,1 -315sync2-p1l,98,0,66,0,1 -315sync2-p2l,98,0,66,0,1 -315sync2-p3l,98,0,66,0,1 -315sync2-p4c,98,0,66,0,1 -315sync2-p4l,98,0,66,0,1 -315sync2-p5l,98,1,10,0,2 -315sync-p1c,98,0,42,1,1 -315sync-p1l,98,0,66,0,1 -315sync-p2l,98,0,,,- -315v2-ub512,98,0,66,0,1 -315v3-buftrace2,0,0,,,- -315v3-buftrace,0,0,,,- diff --git a/dev/315/315-campaign/evidence/315-arm-ledger.md b/dev/315/315-campaign/evidence/315-arm-ledger.md deleted file mode 100644 index 3a0f2f12b..000000000 --- a/dev/315/315-campaign/evidence/315-arm-ledger.md +++ /dev/null @@ -1,78 +0,0 @@ - - - -# The arm ledger: 178 arms, 131 in-regime, 34 faulting in-regime - -```sh -ssh strixhalo 'cd ~/wt/rt-det; for d in 315*-GLM-4.7-Flash-Q4_K_M/; do t=$(basename "$d" | sed "s/-GLM.*//"); \ - g=$(grep -m1 -oE "gtt_start [0-9]+" "$d/run.log" 2>/dev/null | awk "{print \$2}"); \ - n=$(grep -c "HRX returned NaN logits" "$d/server.log" 2>/dev/null); \ - r=$(python3 -c "import json;j=json.load(open(\"$d/detector.json\"));print(j[\"requests\"],j.get(\"http_failures\",0),j.get(\"identical_distinct\",1) or 1)" 2>/dev/null | tr " " ","); \ - echo "$t,${g:-0},${n:-0},${r:-,,-}"; done' -``` - -Columns, in order: `tag,gtt_start,nan_guard_lines,requests,http_fail,idn`. (Five values, not seven - the tag already -carries the arm suffix, e.g. `315row-p1c`. A six-name header shifts every column and turns the whole table into -noise; that mistake was made and corrected once while writing this.) - -## The population - -| bucket | arms | -|---|---| -| all arm directories under `~/wt/rt-det` | 178 | -| in-regime (`gtt_start` 95-110 GiB) | **131** | -| faulting in-regime | **34** | - -Fault classes, from the ledger (an arm counts as faulting if any of `nan_guard_lines > 0`, `http_fail > 0`, or -`idn > 1`): - -| tag | arms | in-regime | faults | -|---|---|---|---| -| `315nan` | 21 | 19 | 18 | -| `315fa` | 18 | 15 | 5 | -| `315hunt2` | 16 | 16 | 1 | -| `315accept` | 14 | 14 | 2 | -| `315chain` | 7 | 7 | 3 | -| `315sync2` | 7 | 7 | 1 | -| `315gsync` | 6 | 1 | 1 | -| `315row` | 5 | 5 | 2 | -| `315det` | 4 | 4 | 0 | -| `315probe2` | 13 | 13 | 0 | -| `315probe3` | 16 | 1 | 0 | - -Both faces appear in the in-regime population and never in the same arm: the `nan=1` arms (Face A, the sampler -guard) and the `http=1` arms (Face B, the server's own 500). Twelve of the 34 in-regime faults are Face B. - -## Two traps this table exposed - both worth knowing before it is used as evidence - -1. **A tag can be reused across days, so a tag is not a session.** `~/wt/rt-det/315fp2-*` held five arms scored - `req=66, nan=0, gtt=98G` whose mtimes were `2026-10-09 20:00-20:19` - a different session's run under the same - tag - while the current run's first arm sat beside them with no `detector.json` yet. Read the mtime, or the - run's own console, before attributing an arm to the session you are running. -2. **"No fault in the ledger" is not "clean".** An arm still in flight has no `detector.json`, and an arm the - harness scored `INVALID` (server never served, or the detector could not allocate) has no fault signal of its - own; both look like `nan=0, http=0` in this table. The verdict is in each run's console - (`pair arm : OK|FAULT|INVALID|OOR`), and the ledger is a population statement, not a verdict list. - -With those two caveats applied, the table is the direct answer to "how often does this reproduce": **34 faulting -arms in regime, across a dozen independent sessions, at `gtt_start` 98-99 GiB**, split between the two faces. diff --git a/dev/315/315-campaign/evidence/315-asan-preregistration.md b/dev/315/315-campaign/evidence/315-asan-preregistration.md deleted file mode 100644 index 15953b3bc..000000000 --- a/dev/315/315-campaign/evidence/315-asan-preregistration.md +++ /dev/null @@ -1,78 +0,0 @@ - - - -# Pre-registration: device-ASAN with the HRX device-event sink (never run in this campaign) - -Written **before** the run. The frozen rig (`~/wt/315-disc.sh`, three cells, `GLO=95`/`GHI=110`) is unchanged -except for `EXTRA_ENV`. - -## Why this is the experiment the campaign is missing - -Every instrument used so far reads **host** memory and reports after the fact, which this stretch showed is not -tensor-faithful for mid-graph tensors (`315-which-readings-are-faithful.md`). The runtime ships a **device-side, -source-attributed** channel that has never been switched on here: - -- **`GGML_HRX_LOG_DEVICE_EVENTS=1`** — `ggml-hrx.cpp:1371-1378` installs `log_hrx_device_event` as the HRX - device-event sink **before** `hrx_gpu_initialize(0)`. The sink is already written to print, for every ASAN - report: `executable=… export=… site=… access=… address=0x… length=… workgroup=(…) workitem=(…) shadow=0x… - value=0x… dispatch=0x…` -- **`HRX_HAL_SANITIZER=asan`** — `libhrx/src/libhrx/runtime.c:576` sets - `IREE_HAL_DEVICE_RUNTIME_FEATURE_FLAG_ASAN | …_FEEDBACK`, i.e. device address sanitizer plus feedback. - -Both exist in **every** `libhrx.so` on the box (checked: `HRX_HAL_SANITIZER` ×2, the ASAN report path ×3 in each -build). **Neither the env var nor the sink appears anywhere in the campaign record or in any driver/probe -script** — the channel is implemented, wired, and has never been used. - -It matters because it is the only channel that can attribute a fault to a **kernel**: `export_ordinal` names the -export, `executable_id` the executable, and `fault_address`/`access_kind`/`access_length` the access. - -## Hypothesis - -In-regime faults include, or are preceded by, an **out-of-bounds buffer access performed by a specific kernel**, -which the AMDGPU HAL's device address sanitizer can report. - -## Readout, fixed in advance - -1. **An `HRX ASAN:` line in a faulting arm** → the mechanism is an out-of-bounds access. The line names the - kernel (`executable`, `export`), the access (`address`, `length`, `access`), the site (`site`) and the - sanitizer's view (`shadow`, `value`). **That is the named mechanism**, and it is falsifiable by inspection of - the named kernel. -2. **No `HRX ASAN:` line, fault still occurs** → the fault is **not** a dispatch-visible out-of-bounds access and - the sanitizer does not see it. A genuine exclusion of the OOB family. -3. **Neither report nor fault** → **INVALID**, not a pass. The intervention may have masked the fault; per the - campaign's standing rule an arm that does not exercise the fault never counts as evidence of absence. - -## Confounds, declared before the run - -- **ASAN is an intervention.** It adds shadow memory and per-dispatch validation, and may move the fault rate - (the record already holds that the fingerprint instrument *raises* it, §39). No arm here is an acceptance arm. -- **Granularity.** The sanitizer validates buffer/dispatch bindings; it is not a per-memory-operation checker and - can miss an out-of-bounds access that stays inside a mapped buffer. A null result therefore bounds the OOB - family *as the sanitizer defines it*, not "no OOB anywhere". -- **Liveness.** A null result is only meaningful if the sanitizer was provably active. If the run shows neither - an ASAN report nor any other evidence the feature was accepted, the outcome is **INVALID**, not a pass. -- Device creation may simply **fail** with the sanitizer on; that is itself a recorded finding, not an error to - be discarded. - -## What would be recorded either way - -Whichever branch fires, the run is recorded with its arm tally, its `gtt_start`s, and the exact `EXTRA_ENV` used. -No criterion changes. No arm eligible. diff --git a/dev/315/315-campaign/evidence/315-asan-wiring-verified.md b/dev/315/315-campaign/evidence/315-asan-wiring-verified.md deleted file mode 100644 index 804c42f2d..000000000 --- a/dev/315/315-campaign/evidence/315-asan-wiring-verified.md +++ /dev/null @@ -1,140 +0,0 @@ - - - -# The ASAN channel is genuinely live: wiring verified at every step - -`315-asan-preregistration.md` requires that a null result be discarded as INVALID unless the sanitizer was -**provably active**. This records the proof, taken before any faulting arm is read. Every step is quoted from -source or from a running process, not inferred. - -**1. The knob reaches the process.** Read from the running arm server's own environment -(`/proc//environ`): - - EXTRA_ENV=GGML_HRX_LOG_DEVICE_EVENTS=1 HRX_HAL_SANITIZER=asan - GGML_HRX_LOG_DEVICE_EVENTS=1 - HRX_HAL_SANITIZER=asan - -**2. The knob sets the runtime feature flags.** `libhrx/src/libhrx/runtime.c:576`: - - const char* sanitizer = getenv("HRX_HAL_SANITIZER"); - if (!sanitizer || !sanitizer[0]) return iree_ok_status(); - runtime_features = IREE_HAL_DEVICE_RUNTIME_FEATURE_FLAG_FEEDBACK; - if (strcmp(sanitizer, "asan") == 0) { - runtime_features |= IREE_HAL_DEVICE_RUNTIME_FEATURE_FLAG_ASAN; - } ... - -**3. The runtime feature flags turn the sanitizer ON in the AMDGPU HAL.** -`runtime/src/iree/hal/drivers/amdgpu/logical_device.c:241-260`: - - static void ...apply_runtime_features(options, runtime_features) { - options->feedback.enabled |= iree_any_bit_set(runtime_features, ..._FEEDBACK); - options->asan.enabled |= iree_any_bit_set(runtime_features, ..._ASAN); - options->tsan.enabled |= iree_any_bit_set(runtime_features, ..._TSAN); - } - ... iree_hal_amdgpu_logical_device_options_apply_runtime_features( - options, create_params->runtime_features); - -So `HRX_HAL_SANITIZER=asan` does set `asan.enabled` — it is not a knob that silently does nothing. - -**4. The device accepts it.** The arms create their device and serve requests normally, so -`iree_hal_amdgpu_logical_device_options_verify_supported_features` passed with ASAN requested; device creation -did not fail. - -**5. The sink is installed, before accelerator init.** `ggml-hrx.cpp:1371-1378`, inside -`create_registry_context()`, gated on `GGML_HRX_LOG_DEVICE_EVENTS` and called **before** `hrx_gpu_initialize(0)` -— which is what the API demands (`hrx_runtime_set_device_event_sink` returns `FAILED_PRECONDITION` if called -after accelerator initialisation). The registered callback, `log_hrx_device_event`, prints, for each ASAN -report: - - HRX ASAN: executable=%llu export=%u site=%llu access=%u address=0x%016llx length=%llu - workgroup=(%u,%u,%u) workitem=(%u,%u,%u) shadow=0x%016llx value=0x%016llx dispatch=0x%016llx - -**6. The report path exists in the deployed binary.** Every `libhrx.so` on the box carries the knob string (×2) -and the ASAN report path (×3). - -**Conclusion:** the channel is live end to end, and it is device-side — the one class of attribution this -campaign has never had. A run that produces no `HRX ASAN` line is therefore a **meaningful** exclusion, not an -INVALID one: ASAN was on, the sink was installed, and nothing was reported. - -## One arm lost to my own cleanup (recorded, not hidden) - -The first arm (`315asan-r1q`) recorded **`gtt_start 47GiB`** — **out of regime**, against the gate's `[95,110]`. -Cause: I killed leftover servers to free the box *while the rig was composing*, so GTT fell away from the -composition the rig had projected (`gtt_before_arm=82G`). Per the standing rule an out-of-regime arm is never -evidence, in either direction, and this one is discarded. It is recorded because a silenced OOR arm is -indistinguishable from a clean one in a summary line, which is exactly the failure mode this campaign has -already been bitten by. - -## Defect #12, carried - -**Killing the top-level probe script does not stop the run.** `pkill -f 315-idprobe` matched and killed the -`315-idprobe.sh` wrapper only; the real chain — `315-attrib.sh` → `315-paired.sh` → the arm's `llama-server` — -kept running for another ~25 minutes, because the diagnostic mode sets `NOSTOP=1`. It was found only because it -pinned the box at 1577 % CPU, holding tctl at 91 °C and blocking the ASAN run's gate. Two lessons, both the shape -of this campaign's earlier harness defects: **kill the process group, not the script name**, and **a leftover can -block a later run's gate rather than merely slowing it**. - ---- - -# The outcome: no device-level access violation in either face - -Both ASAN runs are complete. Tallies, verbatim from the session logs: - -**`315asan2`** — `EXTRA_ENV="GGML_HRX_LOG_DEVICE_EVENTS=1 HRX_HAL_SANITIZER=asan"`, rig otherwise untouched: - - round 1 arm q: FAULT nan=1 req=6 transport=1 gtt_start=100G <- Face A - round 1 arm e: FAULT nan=1 req=10 transport=1 gtt_start=100G <- Face A - round 1 arm c: FAULT nan=0 req=17 http=1 s500=1 unparsed=2 gtt_start=100G <- Face B - - $ grep -h "HRX ASAN" ~/wt/rt-det/315asan2-*/server.log | wc -l -> 0 - $ grep -h "HRX ASAN" ~/wt/rt-det/315asan-*/server.log | wc -l -> 0 (the discarded OOR run) - -**Three arms, both faces represented, every arm in regime, and not one sanitizer report.** The channel was live -(proved section by section above: the knob reached the process, set `asan.enabled`, and the device was created -with feedback, which requires a valid sink). So the readout is the pre-registered **branch 2**: the fault is not -an access violation of the class the AMDGPU HAL sanitizer detects — a buffer/dispatch-binding access leaving its -allocation, caught by pool redzones and shadow. - -## What this does and does not exclude, stated at the strength the evidence supports - -**Does:** no sanitizer-visible out-of-bounds buffer access accompanied either face in these arms. - -**Does not**, and this is the whole of the caveat: - -1. **No positive control.** The sanitizer's own detection was never demonstrated on this device — the argument - that it *would* fire rests on its configuration and delivery path, both verified, not on an induced violation - being caught. A null result with no positive control bounds the family only as far as the sanitizer's own - functioning goes. -2. **Granularity.** It validates at buffer/dispatch-binding granularity. An access that leaves its *row* but stays - inside the allocation is not an ASAN event — and a gather reading a wrong-but-in-bounds row would produce - exactly the finite-but-wrong distribution this campaign calls Face B. -3. **n = 3.** Two Face-A arms and one Face-B arm. - -## Where this leaves the two faces - -| face | excluded | still open | -|---|---|---| -| **A** (NaN logits) | stale/never-written logits region (the engine's own sentinel: `sentinel=0`, fill present in 20 of 21 arms); sanitizer-visible OOB | a NaN **computed** upstream of the final norm — the residual stream is all-NaN at the head, and *which* node produced it is unattributable with the host-side instrument (arena slot reuse) | -| **B** (format failure) | same two | a finite but **wrong** distribution at the sampler; the arm's own text is valid UTF-8 and begins mid-essay, i.e. a corrupted opening | - -Both faces therefore point at the same unresolved question: **a value is wrong at a consumer, and no -host-side read can be trusted to say where it was first written.** Settling it needs a read whose fidelity does -not depend on the allocator's reuse — per-node at production, or device-side validation of the consuming op. diff --git a/dev/315/315-campaign/evidence/315-both-faces-one-trigger.md b/dev/315/315-campaign/evidence/315-both-faces-one-trigger.md deleted file mode 100644 index 3e43b5978..000000000 --- a/dev/315/315-campaign/evidence/315-both-faces-one-trigger.md +++ /dev/null @@ -1,68 +0,0 @@ - - - -# Both faces, one trigger: 11 eviction-cell faults split 7 Face A / 4 Face B - -Criterion 1 asks for a mechanism covering **both faces**. This records the numeric basis for that claim -under the single intervention that moves the rate, re-derived from the consoles rather than from prose. - -## The split, per run, in the forced-eviction cell - - run Face A (nan=1) Face B (nan=0, content/500) - 315gord1a 2 2 - 315gord1b 3 1 - 315ids1 2 1 - total 7 4 - -So **11 forced-eviction faults split 7 Face A / 4 Face B**, and both faces occur under the *same* cell of the -*same* frozen rig. Neither face requires a different intervention: - -- **Face A** is the whole-vocabulary NaN signature - (`HRX returned NaN logits: count=154880 of 154880 … continuing=no`), which the sampler guard catches. -- **Face B** is the format failure (`common_chat_peg_parse: unparsed Content-only output` → HTTP 500), which - the guard does not catch because the logits are finite. - -## Why this matters for the criterion, and what it does NOT claim - -- **It satisfies the "both faces" clause**: one intervention, both observable outcomes, replicated across - three runs and two builds. That is stronger than the record had before, where the faces were sometimes - attributed to different conditions. -- **It does not claim the faces share a cause.** What is established is that the trigger produces both; the - *cause* is unlocated after six closed families. In particular, the peer's n=14 text constraint shows that - faulting arms still emit the correct answer for the identical prompt, so Face B is a **terminal - whole-buffer** event rather than a changed computation — which is a constraint on any future cause, not a - cause itself. -- **The face that appears is a property of the session, not the event.** An earlier session faulted 16 of 17 - arms with `nan=1` each while another faulted half its arms with zero NaN anywhere - (`dev/315/315-campaign/evidence/315-face-landscape.md`). So the two faces are not two mechanisms so much as - two observables of one event, selected by whether the corrupted bytes happen to decode as NaN or as - parseable-but-wrong text. - -## The trigger, quantified (for the same claim) - -From the quiet cell — the only cell with no instrument confound — across every recorded console: - - cell q (QUIET) ~5% (2 faults in 39 arms at 97G; 0% at 44G and 98G) - cell e (FORCER) ~95% (41 faults in 43 arms at 98G) - -a **19x contrast on the same workload at the same GTT level**. That is the intervention the both-faces claim -rests on, and it is why the mechanism statement can be made at trigger level with confidence while the cause -stays open. diff --git a/dev/315/315-campaign/evidence/315-campaign-preregistration.md b/dev/315/315-campaign/evidence/315-campaign-preregistration.md deleted file mode 100644 index b283cf43f..000000000 --- a/dev/315/315-campaign/evidence/315-campaign-preregistration.md +++ /dev/null @@ -1,95 +0,0 @@ - - -# PRE-REGISTRATION: the in-regime acceptance campaign for the logits-read wait - -Committed **before** the campaign runs. The mechanism and the landed fix are in -`315-MECHANISM-unwaited-async-logits-read.md`; the fix is fork commit `47a8fe929`, and the engine pin on -`1bit/315-preregistration` moves `e44c9d01a → 47a8fe929` (`95238dc`), forward-only. - -## What is being tested - -**One prediction, stated before the result:** with the completion wait restored on the logits path, the -faults stop. If they do not stop, the mechanism is wrong regardless of how well it explains the record. - -## Cells - -| cell | build | purpose | arms | -|---|---|---|---| -| **F — fix** | clean, `47a8fe929`, **no instruments** | the acceptance cell | **≥60, in-regime** | -| **C — control** | clean, `e44c9d01a` (the pin's previous revision), no instruments | positive control: shows this harness in this regime can still fault | as many as the box budget allows, same regime, same window | - -Both cells run on **the same harness** (`315-detector.py` shape set), **the same regime gate**, and -interleaved in one window, so a clean fix cell cannot be explained by a regime or harness difference. - -`GGML_HRX_BUFFER_TRACE` and any fingerprint instrument are **off** in both cells: this measures the fault, -not the mechanism. - -## Regime gate - -Frozen at `gtt_start ∈ [95, 110] GiB`, the gate the record already uses, read from the harness's own -`gtt_start` line. **Every counted arm must be inside it**; an arm outside is recorded as out-of-regime and -does not count toward either cell's n (neither as clean nor as fault). - -Composition caveat, already measured (§58): a neighbour moves GTT in ~17 GiB steps, so 80 GiB of -neighbours lands the arm at ~98 GiB (in gate) and one more neighbour lands it at 115 GiB (out). **98 GiB is -this box's in-regime point** and is the expected landing. - -## Both-face detectors, and what counts as a fault - -* **Face A:** the sampler guard line `HRX returned NaN logits` in the arm's `server.log`; and - `HSA_STATUS_ERROR` / queue-error text (the `hsa` counter the harness already computes). -* **Face B:** `detector.json`'s `utf8_failures` and `parse_failures` — `detector.json` is authoritative, - not the console and not the face tally (the record's own rule, because a Face-B fault's text is invalid - UTF-8 and makes `grep` treat the log as binary). -* An arm that dies for a **harness** reason (`transport_failures`, `TimeoutError`, an arm the detector - reports as truncated) is recorded as **invalid**, never as a pass — the record's own rule, and it is why - the previous runs recorded truncated arms separately. - -**Fault = any of the four counters non-zero on a complete arm.** - -## Stopping rule (pre-registered) - -Stop cell F at the **first fault**, exactly as the record's acceptance rule does. A fault in cell F ends the -campaign as **not met** and the arm is reported with its `gtt_start` and face. Cell C is not stopped by a -fault — a fault there is the positive control succeeding. - -## Acceptance, and the power statement - -**Acceptance: ≥60 counted in-regime arms in cell F, zero faults on both faces.** - -Power, stated the way the objective asks: with zero faults in 60 arms, the one-sided 95 % upper confidence -bound on the fault rate is `1 − 0.05^(1/60) ≈ 4.9 %`. So **60 clean arms ⇒ ~95 % confidence the true -in-regime rate is below 5 %.** The bound is reported as this number, not as "the fault is gone". - -**The vacuous-pass guard, pre-registered:** a clean cell F is only meaningful if the harness can still -fault in this regime. If **cell C also produces zero faults in ≥20 complete in-regime arms**, the campaign -is reported as **underpowered** — *not* as a pass — because a harness that cannot fault cannot show a fix. -This is the trap the issue records as already sprung once. - -## Budget - -The objective caps strixhalo at **≤12 h total**. Cell F at ≥60 arms is ~5-6 h at the measured -~5 min/arm; cell C is capped at whatever remains, with a floor of 20 arms for the vacuous-pass guard. If -the budget cannot fund both, cell F's ≥60 is funded first and the shortfall in cell C is reported as a -stated limitation rather than silently dropped. - -## What will be reported - -Every arm with its `gtt_start`, tag and verdict; the counted n per cell; faults by face with the faulting -arm's own log path; the CI above; the control cell's result; and the exact commands and revisions to -re-run it. Any arm excluded, and why. diff --git a/dev/315/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md b/dev/315/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md deleted file mode 100644 index b65aaa294..000000000 --- a/dev/315/315-campaign/evidence/315-candidate-deferred-host-writeback-publication.md +++ /dev/null @@ -1,219 +0,0 @@ - - -# MECHANISM CANDIDATE (source-grounded): a device→host download is a DEFERRED CPU memcpy, published only by `mark_stream_synchronized()` - -Found while tracing what could write `result_norm` between two adjacent host reads, after -`315-read-twice-result-buffer-moves-under-the-reader.md` established that it does (75/75 records -`memcpy_stable=0` in a faulting arm). The mechanism is stated **by the code's own comment**, which is what -makes it worth naming rather than inferring. - -## The download is not a device operation - -`runtime/command-program-executor.h:39-62` — the state that stages downloads: - -```cpp -struct GraphReplayStreamState { - void mark_stream_synchronized(); - void add_host_writeback(void * host_destination, const void * mapped_source, size_t size, hrx_buffer_t buffer); - // True while a host download staged by an earlier replay has not been - // published to host memory yet (see mark_stream_synchronized). - bool has_pending_host_writebacks() const { return !pending_host_writebacks_.empty(); } - private: - std::vector pending_host_writebacks_; -}; -``` - -`runtime/command-program-executor.cpp` — the two operations: - -```cpp -void GraphReplayStreamState::add_host_writeback(void * host_destination, const void * mapped_source, - size_t size, hrx_buffer_t buffer) { - if (buffer != nullptr) { hrx_buffer_retain(buffer); } - pending_host_writebacks_.push_back({ host_destination, mapped_source, size, buffer }); // QUEUES ONLY -} - -void GraphReplayStreamState::mark_stream_synchronized() { - for (PendingHostWriteback & writeback : pending_host_writebacks_) { - if (writeback.host_destination != nullptr && writeback.mapped_source != nullptr && writeback.size > 0) { - std::memcpy(writeback.host_destination, writeback.mapped_source, writeback.size); // PUBLISHES - } - ... - } - pending_host_writebacks_.clear(); -} -``` - -**A device→host download is a plain CPU `memcpy` from the mapped device-visible source into the host -destination, and it does not run when it is staged — it runs when the stream is next marked synchronised.** - -## The hazard, in the code's own words - -`command-program-executor.cpp:873-882`, guarding the host-staging **upload** path: - -> A host download staged by an earlier replay is only published to host memory by -> `mark_stream_synchronized()`; `add_host_writeback` merely queues the memcpy. Uploading the same host memory -> before that has happened would **read the previous contents**. The split scheduler computes splits back to -> back and only synchronizes a backend when it inserts a cross-backend input copy, so **nothing else -> guarantees the ordering**. - -So the codebase (a) knows the destination holds the **previous** contents until publication, and (b) has -written a guard for **exactly one direction** — an *upload* must not read the destination before a pending -download is published. There is no equivalent guard recorded on the **read** side. - -## The read side, which has no such guard - -`ggml-hrx.cpp:238` — the function the engine's host read actually goes through: - -```cpp -static void buffer_get(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, - size_t offset, size_t size) { - auto * context = buffer_context(buffer); - const size_t source_offset = tensor_offset(context, tensor) + offset; - if (context->base != reinterpret_cast(GGML_HRX_FAKE_PTR_BASE)) { - std::memcpy(data, context->base + source_offset, size); // <-- no publication check - return; - } - if (!HRX_CHECK(hrx_synchronous_d2h(context->device->device, context->buffer, source_offset, data, size))) { ... } -} -``` - -When `context->base` is a real host mapping (`!= GGML_HRX_FAKE_PTR_BASE`) the read is a plain CPU `memcpy` -out of the host-visible region, and it consults **neither** `has_pending_host_writebacks()` **nor** any -publication point. So a read that arrives before the queued memcpy has run returns the destination's -**previous** contents — the exact consequence the upload-side comment names, on the direction that has no -check. - -And the two endpoints are the same memory. The only `add_host_writeback` call site is -`command-program-executor.cpp:988`: - -```cpp -if (ErrorResult error = take_status(hrx_stream_copy_buffer(context.stream, staging.buffer, 0, - download_buffer, 0, staging.length))) { ... } -context.graph_replay_state->add_host_writeback(staging.host_data, download_data, staging.length, download_buffer); -``` - -so the device writes `download_buffer`, and `staging.host_data` — passed as `host_destination` — is the -host-side destination that a later read of that buffer returns. *That the host destination `buffer_get` -computes as `context->base + source_offset` is the same region as `staging.host_data` is the one link not -yet read from source; the discriminating experiment below confirms or kills it directly, which is why it is -the experiment.* - -## The mechanism, complete - -> A device→host download is two steps. The device writes `download_buffer`; the step that puts the bytes in -> the host-visible destination is a **queued CPU `memcpy`** (`add_host_writeback`), executed only when the -> stream is next marked synchronised (`mark_stream_synchronized`). The reader (`buffer_get`, `ggml-hrx.cpp:238`) -> reads that destination with a plain `memcpy` and checks nothing. **A read that beats publication returns the -> previous contents.** The upload direction carries exactly this guard (`command-program-executor.cpp:881`); -> the read direction does not. - -## The read path cannot see the publication machinery — and where the probe belongs - -A first attempt to instrument this put the probe in `buffer_get`, and the compiler rejected it: *"no member -named `graph_replay_state` in `ggml_backend_hrx_device_context`"*. That is a structural fact, not a typo. -The two contexts are (`backend-context.h:39-67`): - -```cpp -struct ggml_backend_hrx_device_context { // what a BUFFER context points at - hrx_device_t device; ... std::mutex buffer_stream_mutex; hrx_stream_t buffer_stream; -}; - -struct ggml_backend_hrx_context { // the BACKEND context - ggml_backend_hrx_device_context * device; - hrx_stream_t stream; // <- here - ... - ggml::hrx::GraphReplayStreamState graph_replay_state; // <- and here -}; -``` - -`buffer_get` holds only `ggml_backend_hrx_buffer_context`, whose `device` is the **device** context — so the -synchronous read path has **no route to `stream` and no route to `graph_replay_state`**. It cannot publish a -pending writeback even if it wanted to, which is a sharper statement of the defect than "it does not check". - -`backend_get_tensor_async` (`ggml-hrx.cpp:454`) is where the engine's sampling read actually goes, and it is -the right probe site for two reasons: it holds the **backend** context, so both fields are in scope; and **it -does not synchronise** — it enqueues `hrx_stream_copy_buffer` on `backend_context->stream` and returns: - -```cpp -static void backend_get_tensor_async(ggml_backend_t backend, const ggml_tensor * tensor, void * data, - size_t offset, size_t size) { - auto * backend_context = static_cast(backend->context); - ... - ggml::hrx::HostBufferRef destination = backend_context->device->host_buffers.find(data, size); - if (destination.valid()) { - HRX_CHECK(hrx_stream_copy_buffer(backend_context->stream, context->buffer, tensor_base + offset, - destination.buffer(), destination.offset(), size)); - } else { - synchronous_download_fallback(backend_context, context->buffer, tensor_base + offset, data, size); - } -} -``` - -So the engine's read of the logits is **issued, not completed**, and the copy it queues is on the same -`stream` that `backend_synchronize` drains — which is consistent with §14's finding that the sampling path -does not perform the wait. - -## Instrument status - -`patch-writeback-read-probe.py` reports `graph_replay_state.has_pending_host_writebacks()` and the -destination at that call, gated on `GGML_HRX_WRITEBACK_READ_PROBE`. **Validated as far as it can be without -running it:** the anchors match the live source, and the patched translation unit passes `-fsyntax-only` -under the build's own `compile_commands.json` flags (`rc=0`) — after the first version failed that same -check on the device-context mistake above. It is not yet applied, because an arm is running against that -build. - -## Why this fits every hard fact - -| fact on the record | how this explains it | -|---|---| -| **two adjacent plain reads of the same host buffer differ** (`memcpy_stable=0`, 75/75) | the publication `std::memcpy` runs *between* them; there is no cache involved | -| **Face A: whole-vocabulary NaN** (`count=154880 of 154880`) | the reader gets the destination **before** publication — the previous contents, or an unwritten region | -| **Face B: finite but wrong text** | the previous contents are the *previous token's* logits: finite, plausible, wrong → the parser rejects the body | -| **coherence is not the carrier** (lever 1, §13: `hv56=1` and still faulted) | the transfer is a CPU memcpy; making the memory coherent changes nothing about whether it has run | -| **ordering the reads changes nothing** (§59) | a stream/device ordering primitive does not publish a queued CPU memcpy either | -| **the sync cell did not fix it** (§15) | `backend_synchronize` *does* call `mark_stream_synchronized` (`ggml-hrx.cpp:510`), so a sync at the right point publishes — the question is whether the **reader's** path passes through one at the right time, which §15's placement does not settle | -| **intermittency, and per-token dependence** | whether the reader beats the publication is a scheduling outcome | -| **§14's own correction**: "a copy issued through the buffer stream is invisible to them" | the download is not a device copy at all; instruments watching device transfers would not see it | - -## The discriminating experiment (proposed, not yet run) - -The mechanism makes one prediction no surviving alternative makes: - -> **At the moment the engine reads the logits, a `PendingHostWriteback` for that destination is still -> queued.** In a faulting arm `has_pending_host_writebacks()` is true at the read; in a clean arm it is -> false or was already published. - -Concretely: log `graph_replay_state->has_pending_host_writebacks()` (and the destination range) at the -`ggml_backend_tensor_get` for `result_output`, and correlate with the fault. This is a **one-line observation -plus a counter**, not a fix, and it does not require a new transfer path. - -**Falsifier:** if `has_pending_host_writebacks()` is false on every read in a faulting arm, this candidate is -dead and the writer is something else (see `315-what-the-synchronise-drains.md` for the remaining list). - -## Honest bounds - -* This is a **candidate read at source**, not yet measured. Nothing here shows a pending writeback actually - exists at a read in a real arm. -* It does not yet name the *reader* that loses the race. The engine's sampling-path read is the natural - candidate (§14 records that the sampling path does not synchronise; `llama-context.cpp:1488` issues the - logits copy with `*_async` and returns), but that has not been traced to this queue. -* It does not claim the hazard is unintended: the upload-side guard shows the authors consider the ordering - real and worth guarding; whether the read side is intended to be covered elsewhere is not established here. -* `mark_stream_synchronized` also being called at `ggml-hrx.cpp:510` and `graph-program-cache-limit.cpp:56` - means publication can happen at several points; which of them precedes a given read is the measurement, - not an assumption. diff --git a/dev/315/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md b/dev/315/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md deleted file mode 100644 index a45d124a5..000000000 --- a/dev/315/315-campaign/evidence/315-candidate-deferred-writeback-FALSIFIED.md +++ /dev/null @@ -1,82 +0,0 @@ - - -# FALSIFIED: the deferred-host-writeback candidate — zero pending writebacks across a faulting arm - -`315-candidate-deferred-host-writeback-publication.md` named a mechanism and pre-declared its falsifier: - -> **Falsifier:** if `has_pending_host_writebacks()` is false on every read in a faulting arm, this candidate -> is dead and the writer is something else. - -The falsifier fired on the first arm. - -## The measurement - -`patch-writeback-read-probe.py` reports `graph_replay_state.has_pending_host_writebacks()` at -`backend_get_tensor_async` — the engine's own async logits read, the one that does **not** synchronise — -gated on `GGML_HRX_WRITEBACK_READ_PROBE`. Build `~/wt/315fp-build`, marker `hrx-wb-read` verified in -`libggml-hrx.so.0`. - -Run `315wbe`, `-ub 512`, in-regime, **with the record's replicated eviction forcer running concurrently** -(`~/wt/evictonly 500`, which launches no kernels and only mprotect-churns `hipHostRegister`'d memory — the -intervention the record measured at 7/10 faults against 0/22 controls). Two earlier runs at the ambient -rate produced 9 clean arms and no faulting arm at all, which is the expected outcome at ~5%/arm; the -forcer was used to reach a faulting arm in one, which is a **diagnostic** use and not an acceptance cell. - -``` -315HUNT_FAULT arm=1 out=~/wt/rt-det/315wbe-1-GLM-4.7-Flash-Q4_K_M - arm 1 verdict: gtt_start 98 nan=1 hsa=0 utf8=0 parse=0 wb_records=0 - probe census: reads = 626 pending_writebacks=1 → 0 -``` - -**A faulting arm — Face A (`nan=1`), in-regime at `gtt_start 98G` — with 626 reads and not one of them -carrying a pending host writeback.** - -## What that means - -The candidate's entire content was: *the read of the host destination happens while the CPU `memcpy` that -publishes it is still queued, so the reader gets the previous contents.* If the queue is empty at every -read of a faulting arm, that cannot be what the reader is doing. **The candidate is dead, by its own -pre-registered test, and it is not weakened but removed** — this is the direct measurement the record -lacked, not an argument. - -Notes that keep the falsification honest: - -* The probe is a **read-side observation only**; it changes no ordering, so it cannot have suppressed the - fault it measured. The arm faulted with the probe on. -* `pending_writebacks=0` is not "the probe never ran": 626 records were produced, and the mechanism's own - text (`command-program-executor.cpp:873-882`) plus the earlier 9-arm run confirm the counter is wired to - `pending_host_writebacks_`. -* One arm is one arm. It is a falsification because the prediction was **universal** ("still queued at the - read"), and a single counter-example refutes a universal claim — not because n is large. A run that - showed `pending=1` would have been a confirmation of the same strength. - -## Where the hunt goes - -Not back to visibility or coherence — those remain closed by §13 (coherent `hv56=1` still faulted) and -§59/§55-56 (read-path ordering). The remaining candidates are the ones -`315-what-the-synchronise-drains.md` lists after eliminating the `buffer_stream` path: - -1. `GraphExecutor::execute` returns without waiting, and the fingerprint/read sequence is - `execute → backend_synchronize → read` — so anything read that is not on `stream` is unordered. -2. A second writer of the same arena slot (§53/§59 established the slot is reused). -3. The replay path — least likely, since §4047348 traces it to a full wait. - -And it re-points at the two things the record has measured but not explained: the f32 twin's -`memcpy_stable=0` divergence on `result_norm` (`315-read-twice-result-buffer-moves-under-the-reader.md`), -whose writer this note has now ruled *out* being the deferred writeback, and §51's rank-sort defect, whose -precondition (NaN at the argsort's input) is still unmeasured. diff --git a/dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md b/dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md deleted file mode 100644 index 93ae327e3..000000000 --- a/dev/315/315-campaign/evidence/315-candidate-host-staging-coherence.md +++ /dev/null @@ -1,478 +0,0 @@ - - - -# Candidate mechanism: the deferred host-staging download is a CPU memcpy out of a device-written host buffer that requests no HOST_COHERENT - -Read at the pin (`e44c9d01a` / hrx-system `98d05d94a9`) while the box was blocked. **This is a named, -falsifiable mechanism with a cheap discriminating experiment — and it comes with a falsifier I have not -eliminated, which is stated first because it decides whether this is the mechanism at all.** - -## The falsifier that must be eliminated first - -The runtime's HIP VMM backend **unconditionally grants coherence to host-located memory**: - - libhrx/src/binding/hip/vmm.c:468-473 - if (properties->location.type == hipMemLocationTypeHost) { - // The location selects the physical backing owner, not merely which agent - // may access the eventual mapping. ... - return HRX_MEMORY_TYPE_HOST_LOCAL | HRX_MEMORY_TYPE_HOST_COHERENT | - HRX_MEMORY_TYPE_DEVICE_VISIBLE | cache_type; - } - -So **requesting no `HOST_COHERENT` does not mean the achieved mapping lacks it** — the backend may add it -regardless. The record's `memory_type=0x52` (host-visible, **no** `0x04`) was measured on a *different* -allocation path (`buffer_alloc` → the `HRX0_HOST` logits buffer), not on this staging path, so it is not -evidence either way here. - -**Therefore the mechanism below is a candidate whose central premise is untested.** The discriminating -measurement is one line of logging (below), and until it is run this must not be reported as the cause. - -## The candidate - -The graph-input download path stages device data through a mapped host buffer and then publishes it with a -**plain CPU `memcpy`**: - - runtime/command-program-executor.cpp:1093-1107 (per staging download) - allocate_mapped_host_staging_buffer(context.device, staging.length, download_buffer, download_data); - hrx_stream_copy_buffer(context.stream, staging.buffer, 0, download_buffer, 0, staging.length); - add_host_writeback(staging.host_data, download_data, staging.length, download_buffer, ...); - hrx_buffer_release(download_buffer); - - runtime/command-program-executor.cpp:123-130 (at sync) - std::memcpy(writeback.host_destination, writeback.mapped_source, writeback.size); - -and the staging buffer is allocated **without** the coherence bit: - - runtime/host-memory.cpp:1299-1304 - hrx_buffer_params_t params = { - HRX_MEMORY_TYPE_HOST_LOCAL | HRX_MEMORY_TYPE_DEVICE_VISIBLE, // <- no HOST_COHERENT - HRX_MEMORY_ACCESS_ALL, - HRX_BUFFER_USAGE_DEFAULT | HRX_BUFFER_USAGE_MAPPING_SCOPED | HRX_BUFFER_USAGE_MAPPING_PERSISTENT, - 0, - }; - -The copy at `1099` is **enqueued** (`hrx_stream_copy_buffer`) — the device writes the mapping — and the -staging→host copy is **deferred** to `mark_stream_synchronized`, which then reads that mapping with **no -invalidate**. - -## The same file tree states the rule, for an analogous buffer - - libhrx/src/binding/common/event_timestamp_pool.c:136-146 - // The device writes a tick into this buffer at a queue point and the host - // maps it back, so the allocation has to serve both: DEVICE_VISIBLE is what - // states the device writes here, and every other host allocation this binding - // hands the device asks for it beside HOST_LOCAL. HOST_COHERENT is a - // requirement here and not a preference: the read path never invalidates, so - // non-coherent memory would return stale ticks. - params.type = IREE_HAL_MEMORY_TYPE_HOST_LOCAL | - IREE_HAL_MEMORY_TYPE_DEVICE_VISIBLE | - IREE_HAL_MEMORY_TYPE_HOST_COHERENT; - -This is a device-writes / host-reads buffer, exactly like the staging buffer, and the authors call coherence -*"a requirement here and not a preference"* with the reason given explicitly: **"the read path never -invalidates, so non-coherent memory would return stale"** data. The staging path has the same shape and does -not ask for it. - -## Why this is NOT the falsified lever - -The campaign's coherence A/B toggled `GGML_HRX_USE_UNIFIED_MEMORY`, and both arms faulted, which refuted the -knob. That knob sets coherence **only** inside `buffer_alloc`: - - ggml-hrx.cpp:307-314 - const bool direct_host_binding = host_visible && type_context->device->use_direct_host_bindings; - if (direct_host_binding) { memory_type |= HRX_MEMORY_TYPE_HOST_COHERENT; } - ggml-hrx.cpp:1190 - device_ctx->use_direct_host_bindings = environment_flag_enabled("GGML_HRX_USE_UNIFIED_MEMORY"); - -It is a **ggml buffer-type** decision. `allocate_mapped_host_staging_buffer` does not consult it, does not take -a ggml buffer type, and hardcodes its own params. **So the A/B never touched this buffer**, which is a -coherent explanation for why a coherence experiment refuted coherence: it was applied to the wrong allocation. - -## What it would explain, if the premise holds - -| observation | how this mechanism produces it | -|---|---| -| intermittency | whether the CPU holds a stale line for that address depends on cache state, so the fault is timing- and pressure-dependent by nature | -| the eviction correlation | eviction/paging disturbs cache residency and the mapping — the record's forced-eviction trigger is a cache-pressure amplifier | -| **row 0 or 1, front of buffer** | the first cache lines of a freshly reused arena slot are the ones most likely to still hold the previous tenant's data | -| **arena slot reuse** (the record's own finding) | it is what makes the stale bytes *plausible* rather than obvious garbage — the previous tenant was a same-shaped tensor | -| Face B (finite but wrong → HTTP 500) | stale finite values are a plausible-looking wrong answer | -| Face A (NaN logits) | a stale line that held a NaN pattern, or an unwritten region, yields NaN | -| faults in the **quiet** cell | cache/mapping state is box-wide, not forcer-local — which is why both faces appeared with `side=0/0` | -| `finite + nan == extent` (complete read) | the read *is* complete; it is the *contents* that are stale, so nothing about completeness is violated | - -## The discriminating experiment — cheap, and it settles it before any fix - -Log the **achieved** memory type for the staging allocation (or read it back) in one arm: - -| outcome | conclusion | -|---|---| -| achieved type **lacks** `HOST_COHERENT` | the staging mapping is non-coherent while being device-written and host-read with no invalidate ⇒ **the mechanism is live**, and the fix is to add `HRX_MEMORY_TYPE_HOST_COHERENT` to the staging params (host-memory.cpp:1299) | -| achieved type **includes** `HOST_COHERENT` | the backend grants it anyway ⇒ **this candidate is dead**, and the earlier "read path never invalidates" reasoning does not apply to this buffer | - -This is a single `GGML_LOG` beside the allocation, run in the configuration that already faults. It does not -need the device probe, a build of the probe, or a new campaign — and it is worth doing **before** spending box -time on anything else, because a positive result names the mechanism and specifies a forward-only fix inside -the declared scope (the fork-side `ggml-hrx` runtime). - -## What is proven and what is not - -**Proven, read at source:** the staging params omit `HOST_COHERENT`; the device writes that mapping and the -host later reads it with a plain `memcpy` and no invalidate; the deferred writeback exists and is ordered as -described; the coherence knob does not reach this allocation; and an analogous buffer in the same tree asks for -coherence explicitly, calling it a requirement because the read path never invalidates. - -**NOT proven:** that the *achieved* mapping is non-coherent — the HIP VMM path may add it -(`vmm.c:468-473`). **That single fact decides the mechanism, and it is the first thing to measure.** - -### The discriminating experiment, made runnable: toggle the bit on *this* allocation - -The readback version above is not runnable as written — there is **no getter for a buffer's achieved memory -type** (`hrx_buffer_get_size`, `hrx_buffer_get_device_ptr`, `hrx_buffer_lookup`, `hrx_buffer_map` exist; no -`get_memory_type`), and the `[hrx-buf]` trace prints `memory_type` only inside `buffer_alloc` -(`ggml-hrx.cpp:322`), not for buffers allocated directly through `hrx_allocator_allocate_buffer`. So the -achieved type cannot be read out without new API. - -The runnable form is therefore a **direct A/B on this one allocation**, which is also the stronger test because -it measures the fault rather than the flag: - -| arm | change | prediction if the mechanism is live | -|---|---|---| -| A′ | `allocate_mapped_host_staging_buffer` unchanged | faults at the recorded rate | -| B′ | add `HRX_MEMORY_TYPE_HOST_COHERENT` to the params at `host-memory.cpp:1300` | **fault rate falls** | - -**This is the experiment the earlier coherence A/B should have been.** That one toggled -`GGML_HRX_USE_UNIFIED_MEMORY`, which reaches only `buffer_alloc`'s ggml buffer types and provably cannot touch -this allocation; both its arms faulted, which refuted the knob and not the property. A′ vs B′ moves the bit on -the buffer that is actually device-written and host-read with no invalidate. - -**Falsifier, unchanged and still first in this document:** if the backend already grants coherence to -host-located memory (`vmm.c:468-473` is the reverse mapping and suggests it might), then B′ changes nothing -and the fault rate is unmoved — in which case this candidate is dead and A′ vs B′ is the cheapest possible way -to discover that. - -### Honest status after reading the provider paths - -- **Not our provider:** `vmm_slab_provider.c:401-409` **rejects** any request carrying `HOST_LOCAL | - HOST_VISIBLE | HOST_COHERENT | HOST_CACHED` ("VMM slab provider requires device-local memory without host - access"), so the staging buffer cannot be served there. -- **Unresolved:** the hip path handles `hipMemLocationTypeHost` (`vmm.c:797` is access-location validation, not - the type conversion), and I could not locate the forward conversion from a requested hrx memory type to the - `hipMemAllocationProp` actually used. **So the achieved coherence remains unresolved from source**, and this - document does not claim otherwise. - -**Bottom line: a well-founded candidate with one unresolved premise and a cheap A′/B′ that settles it either -way.** That is a strictly better position than the record had — it is a named mechanism with a discriminating -experiment and an in-scope, forward-only, one-bit fix — but it is **not** yet the named mechanism, and it must -not be reported as the cause. - ---- - -# PRE-REGISTRATION: the A′/B′ staging-coherence campaign (`315abA` / `315abB`) - -Written **before** the run. The design above is a sketch; this fixes the shape, the readout, the stopping -rule and the confounds, so the result is a test rather than a search. - -## Shape - -Two conditions, **one binary**, differing in exactly one environment variable: - -| condition | `EXTRA_ENV` | tag | rounds | -|---|---|---|---| -| **A′** (as shipped) | `GGML_HRX_PRODUCER_PROBE=1 GGML_HRX_PRODUCER_DEVICE=1 GGML_HRX_TENSOR_FP=<15-name l_out list> GGML_HRX_TENSOR_FP_NANONLY=1` | `315abA` | 4 | -| **B′** (bit added) | the same **plus** `GGML_HRX_STAGING_COHERENT=1` | `315abB` | 4 | - -Run **back to back under one lock**, A′ first, so session drift and the box's thermal state are shared rather -than split across conditions. 4 rounds × 3 cells = **12 arms per condition**, of which **4 are eviction-cell -arms** — the cell the recorded rate lives in. - -The two instruments ride along in both arms: the **producer probe** (device mode, finite sentinel) reports -whether the device wrote the residual-family tensors, and the **fingerprint** provides the fault-yield -enabler and the onset readout. Neither is a difference between the conditions. - -## Readout, fixed in advance - -Primary: **fault count per condition over the eviction-cell arms** (`e`), both faces counted. - -| observation | verdict | -|---|---| -| B′'s eviction-cell fault rate is materially **lower** than A′'s | the mechanism is **live** — coherence on this allocation matters, and the fix direction is confirmed | -| B′ and A′ fault at **similar** rates, with ≥3 eviction-cell faults total | the candidate is **dead**; the achieved mapping was already coherent, and this line is closed | -| ≤1 eviction-cell fault in **both** conditions | **INVALID / under-powered** — reported as such, never as "no difference" | -| a condition produces **no Face-A arm at all** | that condition contributes nothing to the rate; reported, not read as clean | - -Secondary (free, from the same arms): the producer probe's `armed` / `anomaly` lines — `sentinel == n` means the -device never wrote that tensor, `0 < sentinel < n` partial, `sentinel == 0` fully written. - -## Stopping rule - -Fixed: run both conditions to completion at 4 rounds each. **No early stopping on a trend**, and no extra -rounds added after seeing the counts — if the result lands in the under-powered row, the correct response is a -**new** pre-registration with more rounds, not an extension of this one. - -## Confounds, declared before the run - -- **Not a rate estimate for acceptance.** 4 eviction arms per condition cannot bound a rate; this is a - directional diagnostic, and no criterion-3 claim may be read from it. -- **The probe perturbs the regime.** Filling live tensors changes downstream values and adds a host⇄device - copy per hooked graph; the record already holds that this instrumentation is itself an inducer. So this - characterises *the probe's regime*, and a difference between A′ and B′ is a difference **within** that regime. -- **The box was thermally saturated earlier today** (`thermal_pauses` to 29, `tctl` to 95 °C, `stopped_frac` - to 0.82) and ran under foreign decompilation load. Both conditions share that, which is why they are run back - to back — but a thermally noisy box widens the interval on any rate, and that is stated rather than adjusted - for. -- **A face may be under-represented.** Face A and Face B have not been observed at equal rates, and Face B has - never been measured with a capable instrument; a fault-count comparison that is all-Face-A is reported as - such. - -## BUILT and RUNNING (2026-10-14 06:00) — both instruments compile and coexist in one binary - - [ 26%] Building CXX object .../runtime/host-memory.cpp.o - [ 26%] Linking CXX shared library ../../../bin/libggml-hrx.so - [100%] Built target ggml-hrx - === EXIT=0 === - strings bin/libggml-hrx.so.0.18.0 | grep -cE 'GGML_HRX_STAGING_COHERENT|GGML_HRX_PRODUCER_DEVICE' -> 2 - -So the A′/B′ toggle **compiles** and the device probe is still in the same artifact, which is what makes a -single run answer both questions. `host-memory.cpp` backed up at `/tmp/host-memory.cpp.orig` -(sha256 `cc7efbc4b873fe8d…`) before patching. - -**Campaign launched**: `315ab` — A′ (`315abA`) then B′ (`315abB`), 4 rounds each, back to back under one lock, -behind the gate-aware wrapper that holds no lock and no GTT while waiting. **Still NOT claimed: that either -condition will fault, or that B′ differs from A′.** The arm verdicts and the producer-probe lines are that -evidence, and they do not exist yet. - -**Running with the probe in both arms is a declared confound, not an oversight**: the probe fills live tensors -and adds a host⇄device copy per hooked graph, and the record holds that this instrumentation is itself an -inducer. Both conditions carry it identically, so the A′/B′ contrast is within the probe's regime — but it is -not the ambient regime, and the pre-registration says so. - ---- - -# The chain, completed: device reads are clean, so the corruption is real and the staleness must enter host-side - -Reading `HostTransferManager::download_synchronous` settles which reads are trustworthy and where staleness can -enter. Both matter, and one of them refutes a competing hypothesis I was about to record. - -## Two download paths, only one of them clean - -| tensor class | path | trustworthy? | -|---|---|---| -| **device** tensors (layer residuals — everything the fingerprint read for the size/position findings) | `backend_get_tensor_async` → unregistered pointer → `synchronous_download_fallback` → `download_synchronous` → **`hrx_stream_synchronize` then `hrx_synchronous_d2h`** | **yes: a real runtime D2H** | -| **host-visible** tensors (the logits region) | generic ggml layer short-circuits for host buffers → **plain memcpy, no backend call, no invalidate** | **no** | -| graph inputs published by the **staging download** | `add_host_writeback` → **deferred plain memcpy** from a mapped staging buffer | **no** | - - runtime/host-memory.cpp:1421-1425 (download_synchronous) - hrx_stream_synchronize(stream); - hrx_synchronous_d2h(device, source, offset, host_destination, size); - -So the "the fingerprint's own reads are stale" hypothesis is **refuted**: my instrument reads device tensors -through a synchronised D2H, not through the deferred mapped copy. **That is a validation of this session's -measurements** — the one-row size invariant, the front-of-buffer position, and `finite + nan == extent` are -readings of **real device state**, not of instrument memory. It also agrees with the record's independent -result that the read path was refuted (138,256 of 138,256 identical when every tensor was read twice). - -## What that forces - -The corruption was measured on the **device**, through a clean read. So the wrong bytes were on the device, and -the question becomes how they got there — and the answer is available in the same file: the **host→device** -direction. - - command-program-executor.cpp:994-996 (comment, on the upload path) - "A host download staged by an earlier replay is only published to host memory by - mark_stream_synchronized(); add_host_writeback merely queues the memcpy. Uploading the same - host memory before that has happened would read the previous contents." - -The code guards *that* ordering explicitly. **The chain that survives is therefore:** - -1. a graph input is published to host memory by the **deferred, non-coherent** staging writeback (or read - before its invalidate); -2. that host buffer is then **uploaded to the device** for a later graph (`staging.upload`); -3. the device computes from a **stale input**, so a real device tensor — the onset residual row — becomes NaN - or wrong; -4. the fingerprint reads that residual through a **synchronised D2H** and reports it **accurately**. - -This is more parsimonious than a device-side defect: every measurement stays valid, the corruption is real and -device-resident, and the mechanism is stale host data being re-uploaded. It also keeps every observation that -had to be explained — intermittency (cache state), the eviction correlation (cache pressure), the front of the -buffer (first cache lines of a reused staging buffer), arena/staging reuse (which makes the stale bytes -plausible, i.e. Face B), the quiet-cell faults (host cache state is box-wide), and the refutation of the -coherence knob (wrong buffer). - -**The A′/B′ campaign now tests exactly this bit**: adding `HOST_COHERENT` to the staging allocation makes the -published bytes visible before they are re-uploaded. If B′ reduces the fault rate, the chain holds. - -### Amendment, still BEFORE any arm: `GGML_HRX_WRITEBACK_FINGERPRINT=1` added to BOTH conditions - -A design risk found while re-reading the publish path, and it would have made the run uninterpretable. - -**The risk.** The chain above needs the staging path to be **warm**. The record already measured the deferred -writeback route as **cold** — *"0 `[hrx-wb-fp]` records with the flag on over a complete 66/66 arm"*. If it is -cold in the disc-rig configuration too, then `add_host_writeback` is never called, no graph input is ever -published by this path, and **A′ and B′ cannot differ** — a null result that would say nothing about the -mechanism while looking like evidence against it. - -**The amendment.** Both conditions now also carry `GGML_HRX_WRITEBACK_FINGERPRINT=1` (the instrument the record -used; inert unless set). It is added to **both** arms, so it remains a non-difference between the conditions, -and it converts the risk from an assumption into a measurement. This is a diagnostic added to the *existing* -run; the primary readout, the arm counts, and the stopping rule are unchanged. - -**The extra readout, fixed in advance:** - -| observation | verdict | -|---|---| -| `[hrx-wb-fp]` lines present in the arms | the staging writeback path **was exercised** ⇒ the A′/B′ contrast is meaningful | -| **no** `[hrx-wb-fp]` line in either condition | the path is **cold in this configuration** ⇒ the chain has no first step here, the A′/B′ null is **structurally expected and uninformative**, and this candidate is dead **for the disc rig** (it would then need a configuration that actually exercises host-staging downloads — the multi-ubatch/split path that #90's message implicates — as a new pre-registration) | - -That second row is the important one: it makes the run **self-falsifying**, so a null cannot be mistaken for a -refutation and cannot be mistaken for support either. - -### Amendment 2, still BEFORE any arm: the two instrument trees are DISJOINT, and the run was aimed at the wrong one - -Caught while preparing the run, and it would have made the campaign **uninterpretable** rather than wrong — the -worst kind of failure, because it looks like a null. - -**The defect.** The instrumentation is split across two source trees, and they do not overlap: - -| tree | what it carries | built by | -|---|---|---| -| `~/wt/llama-315copy` | the **fingerprint** (`hrx-tensor-fp`, `hrx-finite-fp`) | `315fp-build`, `315fp2-build`, `315fbar-build`, … | -| `~/wt/llama-315probe` | the **producer probe** (`hrx-producer`, `GGML_HRX_PRODUCER_PROBE`, `GGML_HRX_PRODUCER_DEVICE`) | `315probe-build` | - -Checked in the artefacts, not inferred: `315probe-build/bin/libggml-hrx.so.0.18.0` contains **7** -`hrx-producer` strings and **0** `hrx-tensor-fp` strings; `llama-315copy`'s `graph-executor.cpp` contains -**0** occurrences of `kProbeSentinel|producer_probe_target|hrx-producer`. - -**What that would have done.** The campaign was written to run on `315probe-build` while setting -`GGML_HRX_TENSOR_FP=<15-name list>`. In that binary the variable is **inert** — so the **15-name list, which is -the fault-yield enabler**, would not have been active. The conditions would have degenerated to *"probe, with -and without a coherence bit"*, at an unknown (probably ambient ~5 %) fault rate, with 4 eviction arms per -condition. That lands in the pre-registration's own **INVALID / under-powered** row — and a reader would have -seen two arms with no faults and no fingerprint lines. - -**Why the amendment is a target change, not a redesign.** A′/B′ needs exactly three things from its binary: the -**fingerprint** (yield enabler + onset readout), the **coherence toggle** (the intervention), and the -**writeback instrument** (the warmth check). Two of the three live in `llama-315copy`, so the fix is to patch -*that* tree — the coherence toggle applies there with its anchor **unique** (the only other -`hrx_buffer_params_t params = {` block is `DEVICE_LOCAL`, so the two are not confusable) — and to build -`315fp-build`. Verified present in `llama-315copy` after patching: the fingerprint, `GGML_HRX_STAGING_COHERENT`, -and `GGML_HRX_WRITEBACK_FINGERPRINT` (2 occurrences, in `command-program-executor.cpp`). - -**The producer probe is dropped from this campaign, deliberately.** It is not needed for A′/B′, it is not in -this tree, and leaving `GGML_HRX_PRODUCER_PROBE=1` in the environment of a binary without it would have been a -second inert variable — so those two variables are **removed from the run** rather than left in and ignored. The -device-probe question keeps its own pre-registration and its own tree. - -**The generalisable trap, recorded because it will recur:** *an environment variable whose instrument is not -compiled into the binary is silently inert, and nothing in the run's output says so.* The only defence is to -check the artefact (`strings | grep `) rather than the source tree you believe you built. This -is the same class as the record's defect **#13** ("the rig's default BIN build does not implement -`GGML_HRX_TENSOR_FP` at all, so an `EXTRA_ENV`-driven sweep was silently inert") — the same trap, a second -time, which is why it is written down rather than just fixed. - -### Correction applied and rebuilt: `315fp-build` now carries all FOUR instruments - - [ 26%] Building CXX object .../ggml/src/ggml-hrx/CMakeFiles/ggml-hrx.dir/ggml-hrx.cpp.o - [ 26%] Linking CXX shared library ../../../bin/libggml-hrx.so - [100%] Built target ggml-hrx === EXIT=0 === - - artefact check: hrx-tensor-fp 1 | GGML_HRX_STAGING_COHERENT 1 | - GGML_HRX_WRITEBACK_FINGERPRINT 1 | GGML_HRX_TENSOR_FP_WAVEMASK 1 - -So the campaign now runs on a binary holding the **fingerprint** (yield enabler + onset readout), the -**coherence toggle** (the intervention), the **writeback instrument** (the warmth check) and the **wave mask** -(region vs lane) — all four verified in the artefact rather than assumed from the build. Rebuilt twice: once -for the toggle, once to add the wave mask. - -**Corrected campaign relaunched** (`315ab`, task `b9b1ee268`): A′ then B′ back to back under one lock, 4 rounds -each, behind the gate-aware wrapper holding no lock and no GTT. Both arms carry the mask and the writeback -instrument; **exactly one variable differs** — `GGML_HRX_STAGING_COHERENT`. - -It now answers three things in one run: - -| question | readout | -|---|---| -| does coherence on the staging allocation change the fault rate? | eviction-cell fault counts, A′ vs B′ | -| was the staging path even exercised? | `[hrx-wb-fp]` line counts per condition | -| is the corruption a region or a lane pattern? | `[hrx-wavemask]` 32-bit masks | - -**Still NOT claimed:** that it compiles *and runs* as intended — the artefact check establishes compilation and -the presence of the env strings, not that any of the four instruments arms at run time. The verdicts and the -`hrx-*` lines are that evidence, and they do not exist yet. - -### Amendment 3, still BEFORE any arm: the launcher would have run B′ with no control - -A defect in my own launcher, found by reading its exit paths rather than by trusting it. - -`315-disc.sh` has three abort exits — **3** (`HRX0 not visible`), **4** (`regime composition failed`) and -**6** (`box not quiet for 2 h before round … arm`, the thermal gate) — and `315ab.sh` ran the two conditions as -two bare commands: - - bash 315-disc.sh 4 315abA # A' - bash 315-disc.sh 4 315abB # B' <- ran even if A' had aborted - -So an A′ that died at composition or on the thermal gate would have left the script **proceeding into B′ with no -control**, producing a set of B′ arms that could not be compared with anything. That is worse than the INVALID -outcome the pre-registration already names, because INVALID is a verdict a reader can see, while this would -have looked like a completed run with a suspiciously one-sided result. - -**Fixed**: each condition is now guarded — - - … bash 315-disc.sh 4 315abA || { echo "ABORT: A-prime exited $? - refusing to run B-prime without its control"; exit 8; } - … bash 315-disc.sh 4 315abB || { echo "ABORT: B-prime exited $?"; exit 9; } - -so an abort in either condition **stops the run and says which one failed**, and a partial comparison cannot be -produced. `bash -n` clean. - -This is the third defect caught in the campaign's own scaffolding before it ran (the disjoint instrument trees, -the wrong-target binary, and now the unguarded launcher), which is the argument for reading a rig before -trusting it — the record has the same lesson three times over in its defect list. - -### Amendment 4: the fault-yield assumption in the pre-registration was wrong, and in the campaign's favour - -The pre-registration justifies 4 eviction arms per condition from "the 15-name list faulted in **4 of its 9 -arms**" (`315size`). That was the **pooled** arm yield. The **eviction-cell** yield is what matters, and it is -much higher: - -| config | eviction-cell arms | result | -|---|---|---| -| `315lout4` (name list + probes) | round 1 `e`, round 2 `e` | **FAULT `nan=1`**, **FAULT `nan=1`** | -| `315size` (name list, no probes) | round 1 `e`, round 2 `e` | **FAULT `nan=1`**, **FAULT `nan=1`** | -| `315inp2` (name list + probes) | round 1 `e`, round 2 `e` | **FAULT `nan=0`**, **FAULT `nan=0`** | - -**6 of 6 eviction-cell arms faulted — essentially 100 %**, against the 44 % I had assumed. Quiet-cell arms in -the same runs were `OK` (5 of 5), so the cell contrast holds. - -Two consequences, and one of them is a caution: - -- **The campaign is better powered than stated.** At ~100 % yield, 4 eviction arms per condition is ample to - show a large difference; the "≤1 fault in both → INVALID" clause is very unlikely to trigger, and the - under-powered risk I worried about does not apply. -- **But a *fix* now has to move ~100 % to ~0 %** to be visible, and any acceptance criterion demanding zero - faults has a high bar given this base rate. That is a fact about the defect, not about the campaign, and it - is worth stating before the arms land rather than after. -- **Face B appears in eviction-cell arms too** (`315inp2`'s two are `nan=0`), which is consistent with the - earlier finding that Face B is not eviction-specific. So an all-Face-A outcome in this campaign would be a - result about the configuration, not proof that Face B is absent. - -**Not changed:** the arm counts (4 rounds per condition each), the stopping rule, and the readout, because the -correction makes the existing shape adequate rather than inadequate. Recorded because a pre-registration whose -stated justification is wrong should say so, even when the shape it justified still holds. diff --git a/dev/315/315-campaign/evidence/315-cause-boundary-final.md b/dev/315/315-campaign/evidence/315-cause-boundary-final.md deleted file mode 100644 index 28d980d79..000000000 --- a/dev/315/315-campaign/evidence/315-cause-boundary-final.md +++ /dev/null @@ -1,74 +0,0 @@ - - - -# The cause boundary, stated finally: seven exclusions and one unbridgeable gap - -After seven candidates were excluded, this records what the engine **can** know, so the boundary between -"cause found" and "out of scope" is explicit rather than implied. - -## The physical trigger, from the record - -`~/wt/evictonly.hip` drives `mprotect` churn over a `hipHostRegister`'d range, which makes the kernel -**evict and restore KFD queues**. Measured per-arm: every arm's **own** buffers show `Q_EVICT`/`Q_RESTORE` -while the arm is live. So the event is *the arm's memory being evicted and restored underneath it*, and the -fault is what a consumer reads afterwards. - -## What the engine tracks about a buffer - -Read at the pin: - - backend-context.h:35 uint64_t generation; per-buffer field - ggml-hrx.cpp:43 std::atomic g_allocation_generation{ 1 }; - ggml-hrx.cpp:338 const uint64_t generation = g_allocation_generation.fetch_add(1); - command-program-bindings.cpp:69 mix_hash(hash, binding.generation); - -So `generation` is a **monotonic counter bumped once per allocation** and mixed into the binding hash used -for **command-program caching**. `identity` is likewise a per-buffer key. - -**Neither tracks contents across a sync.** They answer "which allocation is this?" — not "are the bytes I am -about to read the ones the device wrote?". There is no checksum, epoch, or validity flag consulted after a -synchronisation anywhere on the replay → restore → consume path, which is the same conclusion the writeback -and Face-B-detection reads reached from their ends. - -## The gap, stated as one sentence - -> The engine can name **which** allocation it holds, and can detect **that** a read produced garbage (Face A's -> NaN guard) or malformed text (Face B's parse failure) — but it cannot distinguish "the device wrote these -> bytes" from "a restore left a pre-eviction copy", and that distinction is precisely what a repair requires. - -## Why the remaining candidates are out of scope - - KFD eviction/restore fidelity driver/kernel lane - the boundaries say record, do not attempt - executor cross-dispatch ordering §55/§56 cleared the consumer's own read path - (hsa_signal_wait_scacquire + timepoint covering the last submission) - -and the seven in-scope candidates are excluded above. So the cause is not merely unfound: **the surfaces -within the declared scope have been enumerated and excluded**, which is a stronger statement than repeated -failure to find it. - -## What this means for the criteria, plainly - -- **Criterion 1**: trigger named/replicated/quantified — the strongest result in the issue. Cause **not - locatable within scope**, and now with the in-scope surface enumerated rather than merely searched. -- **Criterion 2**: not achievable within scope, because any repair needs the distinction above. -- **Criterion 3**: gate mis-specified (intervention cell, not a band). -- **Criterion 4**: op-test half measured and passing; dense half running. -- **Criterion 5**: done. diff --git a/dev/315/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md b/dev/315/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md deleted file mode 100644 index fd5961b5e..000000000 --- a/dev/315/315-campaign/evidence/315-cellA-measured-noncoherent-divergence.md +++ /dev/null @@ -1,64 +0,0 @@ - - -# Cell A of the two-cell comparison: the divergence tracks a MEASURED non-coherent mapping - -`315-two-cell-memory-type-preregistration.md` pre-registered the comparison and required -`GGML_HRX_BUFFER_TRACE=1` so the divergence is attached to a **measured** type. Cell A is now measured. - -## The measurement - -Run `315twin`, `-ub 512`, in-regime, **with the eviction forcer running** (`evictonly 500`) so a faulting -arm arrives in one, and with `GGML_HRX_TENSOR_FP=result_norm` + `GGML_HRX_BUFFER_TRACE=1`. - -``` -315twin-1 nan_guard = 1 (Face A: HRX returned NaN logits → FAULT) - twin records = 932 - divergences = 128 - memcpy_stable=0 on 128 of 128 - measured host-visible memory_type = 0x52 (HOST_LOCAL|DEVICE_VISIBLE, NON-coherent) - (device-local allocations measured 0x30 as expected) -``` - -**Every divergence is on a mapping measured `0x52`, and the arm faulted.** - -## What this does and does not settle - -It settles cell A's half of the pre-registration: the divergence is real, it is `memcpy_stable=0` -throughout — so no read-to-read agreement at any point — and it occurs on a memory type that is -**recorded in the same log**, not inferred from a comment. The rate is also higher than the earlier -ambient-rate control run (128 in 932 records against 75 in 669), which is what one expects if the -divergence is driven by cache pressure: this arm ran under the eviction forcer, which mprotect-churns host -memory. - -It does **not** by itself establish that the memory type *causes* the divergence, because cell B — the -`0x56` configuration — has not run. `315-persistent-mapping-correction.md` is the reason the distinction -matters: `buffer_get` reads a **persistent mapping** created at allocation and never invalidated, so a -divergence between two plain reads is consistent with either the device having written the range or the -CPU's cached view of a non-coherent mapping having changed. Cell A cannot separate those; the two cells -together can, because a `0x56` mapping removes the second explanation. - -**Also recorded, because it bounds the claim:** the eviction forcer is a **diagnostic** trigger (the record -measured it at 7/10 faults against 0/22 controls). Nothing here is an acceptance cell, and no arm in this -run may be counted toward the objective's ≥60 in-regime clean-arm campaign. - -## Next - -Cell B: apply the twin to the coherent tree (`llama-315coherent` / `315coherent-build`, which currently has -`hrx-f32-sync = 0`) and run the same configuration. The pre-registered readings in -`315-two-cell-memory-type-preregistration.md` then apply as written, including the case where cell B's -achieved type turns out not to be `0x56`, which voids the comparison and must be recorded as such. diff --git a/dev/315/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md b/dev/315/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md deleted file mode 100644 index 1c6629dcb..000000000 --- a/dev/315/315-campaign/evidence/315-cellB-divergence-independent-of-memory-type.md +++ /dev/null @@ -1,76 +0,0 @@ - - -# Cell B vs cell A: the read-to-read divergence does NOT track the memory type - -The two-cell comparison `315-two-cell-memory-type-preregistration.md` pre-registered, run. The -coherence explanation my own correction introduced (`315-persistent-mapping-correction.md`) is closed. - -## The two cells - -Both cells: **the same build** (`~/wt/315fp-build`), **the same instrument** (the f32 fp/sync twin plus the -read-twice control), `-ub 512`, in-regime, `GGML_HRX_BUFFER_TRACE=1` so the type is measured, and the -eviction forcer running so a faulting arm arrives in one. They differ in **one environment variable**: - -| cell | configuration | **measured** host-visible type | records | diverged | rate | fault | -|---|---|---|---:|---:|---:|---| -| **A** | default | `direct_host_binding=0 memory_type=0x52` (non-coherent) | 932 | 128 | **13.7%** | `nan_guard=1` | -| **B** | `GGML_HRX_USE_UNIFIED_MEMORY=1` | `direct_host_binding=1 memory_type=0x56` (coherent) | 426 | 63 | **14.8%** | `nan_guard=1` | - -All divergences in both cells are `memcpy_stable=0` (128/128 and 63/63). - -`GGML_HRX_USE_UNIFIED_MEMORY` gates `device_ctx->use_direct_host_bindings` (`ggml-hrx.cpp:1175`, logging -*"direct coherent host bindings enabled"*), so the coherent cell is reached on the same build with the -instrument untouched — the A/B difference is the memory type and nothing else. - -## Pre-declared reading applied - -The pre-registration's table: - -> divergences in **both** cells → the divergence is not the memory type. It is then either independent of -> coherence, or both cells achieved the same type — which the trace in the same log settles, so the two -> cases are separable within this run rather than by argument. - -**Divergences in both cells, and the same-log traces show the types are different (`0x52` vs `0x56`).** So -the second branch is excluded by measurement: the divergence is **independent of coherence**. - -The rates are indistinguishable at this n (13.7% vs 14.8%; Fisher exact on 128/932 vs 63/426 is far from -significant), so this is not "coherence reduces it" either. Both cells also faulted, which is consistent -with §13's finding that a coherent allocation does not remove the fault. - -## What this settles, and what it hands back - -* **Closed:** the third explanation for `memcpy_stable=0` that `315-persistent-mapping-correction.md` - introduced — that the CPU's cached view of a non-coherent *persistent mapping* changes between two reads. - Coherent memory has no such mechanism, and the divergence is unchanged. The persistent-mapping observation - itself remains true (`buffer_get` does read a mapping created at allocation and never invalidated); it is - simply not what moves between the reads. -* **Restored, but narrowed:** `memcpy_stable=0` again means **the buffer changed between two identical - reads**, i.e. something wrote it — with the writer now having two fewer candidates: not the deferred - host writeback (falsified by a counter, `315-candidate-deferred-writeback-FALSIFIED.md`) and not cache - behaviour of a non-coherent mapping (falsified here by the A/B). -* **Handed back:** the remaining writers are the ones `315-what-the-synchronise-drains.md` lists — - `GraphExecutor::execute` returning without a wait while `backend_synchronize` covers only - `context->stream`; a second writer of the reused arena slot (§53/§59); the replay path (least likely, - §4047348 traces it to a full wait). - -## Bounds - -Both cells are **diagnostic** (eviction-forced, which the record measured at 7/10 faults against 0/22 -controls). **No arm of either cell may be counted toward the objective's ≥60 in-regime clean-arm campaign.** -n is two arms per cell; what makes this decisive is that the prediction was a *ratio comparison between two -configurations*, and the measured types differ by exactly the bit under test while the ratio does not move. diff --git a/dev/315/315-campaign/evidence/315-chain-plus-early-preregistration.md b/dev/315/315-campaign/evidence/315-chain-plus-early-preregistration.md deleted file mode 100644 index 8640c314b..000000000 --- a/dev/315/315-campaign/evidence/315-chain-plus-early-preregistration.md +++ /dev/null @@ -1,70 +0,0 @@ - - - -# Pre-registration: the §40 chain probe **plus** the early layers, in one arm - -Written before the run. This replaces the queued `tail:16` run, because a single arm can now answer two questions -that `tail:16` answers only one of. - -## The two questions, and why one arm suffices - -1. **§40's claim**: *"Every earlier call's window was clean, so the corruption is **born at block 41**, not - inherited from block 0."* — the campaign already wrote, and explicitly did not run, the probe that settles it - (`315-mechanism.md` §40). -2. **This session's gradient**: `l_out-0` carries NaN (1,984), growing monotonically to saturation by layer 15 — - which says the origin is at or before layer 0's output and therefore **contradicts** (1). - -Both are answered by one name list, at the same instrumentation dose (~16 names) that produced the Face-A -signature with a clean quiet cell. - -## The run - - BIN=~/wt/315fp-build/bin - EXTRA_ENV="GGML_HRX_COPY_PROBE=1 GGML_HRX_SENTINEL=1 GGML_HRX_LOGITS_FP=100000 \ - GGML_HRX_TENSOR_FP=l_out-0,l_out-1,l_out-2,l_out-3,l_out-4,l_out-40,ffn_inp-41,ffn_norm-41,ffn_shexp-41,\ - ffn_swiglu-41,ffn_moe_out-41,ffn_out-41,l_out-41,attn_norm-42,ffn_inp-42,ffn_norm-42 \ - GGML_HRX_TENSOR_FP_NANONLY=1" - -The block-41 chain is **verbatim from §40**; the early layers are added ahead of it. 16 names total. - -## Readout, fixed in advance - -| observation in a faulting arm | meaning | -|---|---| -| `l_out-0` NaN **and** `ffn_inp-41`/`ffn_norm-41` **clean** | §40's generalisation is **refuted**: block 41's input was clean while layer 0's output was not, so the corruption is earlier. The early origin stands. | -| early layers **clean** and `ffn_inp-41`/`ffn_norm-41` NaN | **this arm is a different episode from this session's gradient**; §40's account holds for it and the two must be compared arm-by-arm, not generalised from either | -| `l_out-0` NaN **and** `ffn_inp-41` NaN | both are downstream of the same early NaN — consistent with the gradient, and block 41's chain adds nothing about the origin | -| clean arm (66/66) | nothing, as always | - -Second, within-block-41 causality, exactly as §40 specifies: if `ffn_inp-41`/`ffn_norm-41` are clean while -`ffn_shexp-41`/`ffn_swiglu-41`/`ffn_moe_out-41` are not, **the FFN branch fabricated it**; if `ffn_inp-41` is NaN, -**the block's input was already poisoned**. - -## Confounds, declared - -- **Dose.** ~16 names, matching the configuration whose quiet cell was clean (66/66) and whose eviction cell gave - Face A. Whether the *origin layer* is itself dose-dependent is untested; this arm shares the dose of the gradient - it is checking, which is what makes them comparable. -- **Name-matched reads are displacement-exposed**; `l_out-*` is least so (last in its chunk), and the block-41 - chain nodes are mid-chunk. A "clean early layer" reading is therefore weaker evidence than a NaN reading. -- **The instrument induces**; no rate is quoted and no arm is an acceptance arm. -- One arm cannot distinguish "born here" from "inherited" for a *single* node; it distinguishes the two competing - **accounts**, which is the point. diff --git a/dev/315/315-campaign/evidence/315-chaining-guard-defect.md b/dev/315/315-campaign/evidence/315-chaining-guard-defect.md deleted file mode 100644 index 94296ec4e..000000000 --- a/dev/315/315-campaign/evidence/315-chaining-guard-defect.md +++ /dev/null @@ -1,68 +0,0 @@ - - - -# Defect #16: "wait while the predecessor runs" cannot tell *finished* from *never ran* - -## What happened - -To run three probes back to back (the §40 chain, the layer-0 `ffn_inp` split, the shape measurement) I wrote three -small scripts, each of the form: - - while ps -eo cmd | grep -F "3 315chain" | grep -qv grep; do sleep 30; done # wait for the predecessor - ...clean orphans, cool down... - launch the next probe - -Three disc sessions then appeared on the box at once, including two copies of `315lout4`: - - 1495773 19:30 bash /home/bcloud/wt/315-disc.sh 3 315lout4 - 1583913 04:57 bash /home/bcloud/wt/315-disc.sh 3 315lout4 - 1605854 01:16 bash /home/bcloud/wt/315-disc.sh 3 315inp0 <- stage 2, launched while stage 1 never ran - -`315chain-run.log` was **empty** — stage 1 never started — and stage 2 launched anyway, because its guard is -**"wait *while* the predecessor is running"**, which is **satisfied immediately when the predecessor does not exist -at all**. The guard answers "is it running now?", not "did it ever run and finish?". - -Two sessions then composed the regime simultaneously, so neither arm's cell means what its label says, and every arm -in that window is unusable. - -## Why this is the same defect, one layer up - -It is defect #14 — *a `NANONLY` sweep cannot distinguish "every layer clean" from "no Face-A arm occurred"* — with -the sign flipped: there, absence of output was read as a clean result; here, absence of a process is read as -completion. Both are a **predicate on a proxy** standing in for the event of interest, and both silently proceed. - -The campaign now has four instances of this shape: #13 (an `EXTRA_ENV` naming a feature the build does not -implement — absence of effect read as absence of an effect), #14, #16, and the `pkill`-on-name traps (#11/#12) where -killing the wrapper was read as stopping the run. - -## The fix, and what I did instead - -The correct guard records **evidence of the predecessor's completion** — a sentinel line the predecessor writes -after its last arm — and additionally requires that it *started*. Chaining is not worth that machinery here. - -**I abandoned the chaining and ran the stages directly, one at a time, verifying each launch by its own log header -before starting the next.** That is slower but it cannot silently skip a stage, which is the failure mode that cost -this session two composed regimes and ~20 minutes of box time it is already over budget on. - -## Cost - -Two simultaneous regime compositions, whose arms are discarded; the §40 chain probe (stage 1) never ran and must be -re-issued; one duplicate `315lout4` session. No conclusions were drawn from any arm in that window. diff --git a/dev/315/315-campaign/evidence/315-cheap-repro.md b/dev/315/315-campaign/evidence/315-cheap-repro.md deleted file mode 100644 index 7e77f2931..000000000 --- a/dev/315/315-campaign/evidence/315-cheap-repro.md +++ /dev/null @@ -1,63 +0,0 @@ - - - -## The cheap probe: align=256 + compaction - -``` -# engine#315: cheap reproducer for the wrong-output class at the pin (no GLM, no regime, no neighbours) -# strixhalo, 2026-10-09 23:06-23:15. Model Qwen3-0.6B-Q4_K_M on HRX0, -fa on, -c 4096, greedy, seed 1, -# cache_prompt=false, 180 identical requests per configuration, in ONE server process. -# Build: 315fp-build (the pin e44c9d01a plus the nan-dump fingerprint). - -## per-token logits-hash divergence (call index within a request) - -| config | requests | requests with NaN | distinct hash at token 0 | token 1 | token 2 | token 7 | token 23 | -|---|---|---|---|---|---|---|---| -| default align, quiet | 180 | 0 | 1 | 1 | 1 | 1 | 1 | -| default align + compaction | 180 | 0 | 1 | 1 | 1 | 1 | 1 | -| align=256 + compaction | 180 | 0 | 2 | 3 | 7 | 35 | 121 | - -## the same runs as produced outputs (distinct completions out of 180) -- no compaction: 120 requests, 1 distinct -- default align, quiet: 180 requests, 1 distinct -- default align + compaction: 180 requests, 1 distinct -- align=256 + compaction: 180 requests, 95 distinct - -## dispatch selection differs with the knob (GGML_HRX_LOG_DISPATCH=1, 12 requests, ctx 1024) -- align=256: 21 distinct dispatches -## dispatch selection differs with the knob (GGML_HRX_LOG_DISPATCH=1, 12 requests, ctx 1024) -- align=256: 21 distinct dispatches - flash-attention kernels: ['common.flash_attention_f32_f16_wmma (single)'] - decode matmul kernels: ['common.mul_mat.f32_f32_decode (fused)', 'kquant.mul_mat.decode_f32 (fused)', 'kquant.mul_mat_add.decode_f32 (fused)'] -- align=4096: 21 distinct dispatches - flash-attention kernels: ['common.flash_attention_decode_split_next_q8 (single)', 'common.flash_attention_f32_f16_wmma (single)'] - decode matmul kernels: ['common.mul_mat.f32_f32_decode (fused)', 'kquant.mul_mat.decode_f32 (fused)'] - -## reading -The knob does not merely misalign a buffer: it changes which kernels the dispatcher selects. -At align=256 the decode-split flash attention (common.flash_attention_decode_split_next_q8) -never runs, and kquant.mul_mat_add.decode_f32 (fused) does; at the shipped align=4096 both are -the other way round. Divergence begins at the first decoded token (token 0 has 2 distinct -hashes, i.e. it is already wrong for some requests) and compounds, which is the locality -engine#315 records for its own faults. All sampled logits stay finite (nan=0), so this shape -is Face B, not Face A. -``` diff --git a/dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md b/dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md deleted file mode 100644 index 4c9894a4b..000000000 --- a/dev/315/315-campaign/evidence/315-coherence-candidate-falsified-by-decomp.md +++ /dev/null @@ -1,171 +0,0 @@ - - - -# The coherence candidate is FALSIFIED by the decompiled runtime — and it explains the record's refuted A/B - -Found in the **Ghidra RE dump at `~/amd-oss/hrx-re`** (the lane that was loading the box), which decompiles -`libhrx`, `LoomC` and the `amdhip64`/HIP binding. Reading `hrx_allocator_allocate_buffer` and its callees -settles the premise I could not resolve from source, and it settles it **against** the candidate. - -## The type values, read from the headers (not recalled) - - runtime/src/iree/hal/buffer.h:45-112 - OPTIMAL = 1u << 0 = 0x01 - HOST_VISIBLE = 1u << 1 = 0x02 - HOST_COHERENT = 1u << 2 = 0x04 - HOST_CACHED = 1u << 3 = 0x08 - DEVICE_VISIBLE = 1u << 4 = 0x10 - HOST_LOCAL = HOST_VISIBLE | (1u << 6) = 0x42 - DEVICE_LOCAL = DEVICE_VISIBLE | (1u << 5) = 0x30 - DEVICE_UNCACHED= 1u << 7 = 0x80 - -and the HRX types are bit-identical to these (a wall of `static_assert`s in `hrx_internal.h`). - -## What the decompiled allocator does - - hrx_allocator_allocate_buffer(allocator, params, size, buffer) [decomp/libhrx/] - hrx_mem_pool_backing_retain_allocation_source(device, params.type, params.usage, &pool) - iree_hal_pool_allocate_buffer(pool, request, …) - hrx_buffer_create_from_hal(hal_buffer, device, params.type, size, NULL, buffer) - -and the pool selector: - - hrx_mem_pool_backing_retain_allocation_source(device, memory_type, usage, out_pool) - kind = HRX_MEM_POOL_SOURCE_HOST; - if ((~memory_type & 0x42) != 0) { kind = memory_type >> 7 & …; } // 0x42 == HOST_LOCAL - … - hrx_mem_pool_backing_create_source(device, kind, usage, out_pool); // no memory_type argument - -**`0x42` is exactly `HOST_LOCAL`, and `HOST_COHERENT` is `0x04` — which is not in that mask, and is not passed -to pool creation either.** So: - -| question | answer from the decompilation | -|---|---| -| does the coherence bit change **which pool** backs the buffer? | **No.** The kind is decided by `HOST_LOCAL` (0x42) alone; `0x42 & 0x04 == 0` | -| does the pool get the memory type at all? | **No** — `create_source(device, kind, usage, …)` takes kind and usage only | -| does the buffer record it? | Yes: `hrx_buffer_create_from_hal(…, mem_type, …)` stores the requested type | -| is it consulted in `query_mapping_requirements`? | **No** — that also routes through the same pool selector | - -And the staging request is `HOST_LOCAL | DEVICE_VISIBLE = 0x52` — **precisely the type the record measured for -the logits buffer**, and the type the campaign's arm B′ was going to OR `0x04` into. - -## Consequence 1: my coherence fix is a no-op, and I stopped the campaign - -`315ab` (A′ vs B′, differing only in `GGML_HRX_STAGING_COHERENT`) was **killed** rather than allowed to consume -box time: the bit cannot change the pool, so both arms would have allocated from the **same** pool with the -same usage. A null was structurally guaranteed, and a null would have read as evidence about coherence. - -**Honest limit, stated because it is the difference between "no-op" and "proven inert":** I could not read the -VMM map path (`hrx_allocator_virtual_memory_map` decompiled to nothing in this dump). A downstream mapping step -could in principle branch on the type recorded on the buffer. What is established is that the **backing pool — -which determines the memory — is chosen without reference to the coherence bit**, so the mechanism I proposed -(a different memory provider for a coherent request) is dead. - -## Consequence 2: it explains the record's refuted coherence A/B - -The campaign's earlier coherence experiment toggled `GGML_HRX_USE_UNIFIED_MEMORY` and **both arms faulted**, -which was read as coherence being irrelevant. The decompilation gives the structural reason: - - ggml-hrx.cpp:313-314 if (direct_host_binding) { memory_type |= HRX_MEMORY_TYPE_HOST_COHERENT; } - ggml-hrx.cpp:1190 device->use_direct_host_bindings = environment_flag_enabled("GGML_HRX_USE_UNIFIED_MEMORY"); - -That knob changes only the **type recorded on the buffer** (`0x52` → `0x56`). By the analysis above it cannot -change the pool. **So the knob could not change the memory, and both arms faulting is exactly what that -implies.** A negative result that has sat in the record as "coherence does not matter" is, on this evidence, -better read as *"the knob could not test coherence"* — the same failure mode as the `tail:16`-yield trap and -the disjoint-instrument-trees defect: **the intervention never reached the thing it was named for.** - -## What this leaves standing - -- **The 256-byte granularity** — 96.6% of partial readings begin on a 64-element boundary, ≥89.5% within - **every** tensor family, against 1.6% chance. Unaffected by any of this. -- **The two remaining interpretations** of it: one **wave64** (64 lanes × 4 B = 256 B exactly) and the - record's **MoE-table block** alignment (independent support: the FRONT group is MoE-dominated). The - copy/region reading was already weakened (no 256-**byte** copy granularity in libhrx). -- **The wave mask**, built and pre-committed, which decides region-vs-lane in one run. -- **The coherence/staging family is now closed** as the explanation, on decompiled evidence, unless a - map-path effect is found — and that is a specific, cheap next read, not an open-ended search. - ---- - -# SCOPE CORRECTION: the falsification covers the POOL path only — the REGISTER path has explicit coherence, and its decider is missing from the dump - -The section above concludes "the coherence/staging family is closed". Reading further into the same dump -narrows that: it is closed for **one** of the two host-memory paths and **open for the other**, and the -function that would close the second is one of the dump's gaps. - -## There are two host-memory paths, and they treat coherence differently - -| path | how memory is obtained | is coherence consulted? | -|---|---|---| -| **pool** (what the ggml-hrx staging buffer uses) | `hrx_allocator_allocate_buffer` → `hrx_mem_pool_backing_retain_allocation_source` → pool chosen by `HOST_LOCAL` (**0x42**); `create_source(device, kind, usage)` never sees the type | **no** — this is the falsification, and it stands | -| **register / import** (what `hipHostRegister` and managed buffers use) | `iree_hal_streaming_memory_register_host(context, ptr, size, flags, &buffer)` | **yes** — there are explicit flags | - -## The register path's coherence flags, read from the headers - - libhrx/src/binding/common/internal.h:1089-1107 - IREE_HAL_STREAMING_HOST_REGISTER_FLAG_DEFAULT = 0 - IREE_HAL_STREAMING_HOST_REGISTER_FLAG_READ_ONLY = 1ull << 3 (0x8) - IREE_HAL_STREAMING_HOST_REGISTER_FLAG_HIP_UNCACHED = 1ull << 28 - IREE_HAL_STREAMING_HOST_REGISTER_FLAG_HIP_COHERENT = 1ull << 30 - IREE_HAL_STREAMING_HOST_REGISTER_FLAG_HIP_NON_COHERENT = 1ull << 31 - -and the decompiled `hipHostRegister` (which decompiled **OK**): - - hipHostRegister(ptr, size, flags) - flags_00 = flags | FLAG_READ_ONLY; - if ((flags & 8) == 0) { // 8 == hipHostRegisterReadOnly - flags_00 = flags & (HIP_NON_COHERENT | HIP_COHERENT | HIP_NUMA_USER | HIP_UNCACHED | - WRITE_COMBINED | MAPPED | PORTABLE); // <- overwrites the READ_ONLY default - } - status = iree_hal_streaming_memory_register_host(context, ptr, size, flags_00, &buffer); - -## What that means for the #315 trigger specifically - -The forcer `~/wt/evictonly.hip` registers with **`hipHostRegisterDefault`, i.e. `flags = 0`**: - - flags_00 = 0 | READ_ONLY = 0x8 -> then (flags & 8) == 0 is TRUE, so overwritten by 0 & (...) = 0 - -so it passes **`FLAG_DEFAULT` = 0 — neither `HIP_COHERENT` (1<<30) nor `HIP_NON_COHERENT` (1<<31) set.** -**The registration is coherence-unspecified**, and `iree_hal_streaming_memory_host_flags` (decompiled OK) -likewise *initialises* `*out_flags = FLAG_DEFAULT` before overwriting it from the buffer. - -**So the behaviour of registered host memory under the #315 trigger is decided by what -`iree_hal_streaming_memory_register_host` does with an unspecified coherence flag — and that function is -ABSENT from the dump** (no JSON, not among the failures either, so it was never captured). - -That is now the single most valuable missing function in the dump, ahead of -`hrx_allocator_virtual_memory_map`: the trigger registers with flags = 0, so *its own memory's coherence is -whatever that function's default is*. If it defaults to **non-coherent**, the eviction churn in the trigger is -operating on non-coherent registered memory — which is exactly the scenario the record's coherence hypotheses -were reaching for — and the earlier "coherence does not matter" reading would be wrong for a second, -independent reason. - -## Corrected standing of the coherence family - -- **Closed:** the **pool** path cannot be influenced by a coherence bit (the falsification; the fix I had - prepared is a no-op). -- **Open:** the **register/import** path distinguishes coherence explicitly, the trigger registers with the - flag *unset*, and the deciding function is not in the dump. So the family is **not** closed — it is closed - *for the buffer I had proposed to patch* and open *for the path the trigger actually uses*. - -**Nothing here is a mechanism claim.** It is a precise statement of which path is decided, which is not, and -which one function would decide the rest. diff --git a/dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md b/dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md deleted file mode 100644 index 2f38573b9..000000000 --- a/dev/315/315-campaign/evidence/315-coherence-closed-from-source-both-paths.md +++ /dev/null @@ -1,99 +0,0 @@ - - - -# Coherence is CLOSED, definitively and from source: this runtime cannot make host memory coherent - -Following the register path to the end (the source is in the tree, so no decompilation was needed) settles the -question the record has carried for weeks, and settles it in the strongest form yet. - -## Both host-memory paths ignore coherence, and one of them hardcodes it away - -**Path 1 — pool** (the ggml-hrx staging buffer): - - hrx_mem_pool_backing_retain_allocation_source(device, memory_type, usage, &pool) - kind = HRX_MEM_POOL_SOURCE_HOST; if ((~memory_type & 0x42) != 0) { … } // 0x42 == HOST_LOCAL - hrx_mem_pool_backing_create_source(device, kind, usage, out_pool); // no memory_type argument - -`HOST_LOCAL = 0x42`, `HOST_COHERENT = 0x04`; `0x42 & 0x04 == 0`. The bit is not consulted, and the pool never -receives the type at all. - -**Path 2 — register/import** (`hipHostRegister`, managed buffers) — and this is the one the #315 trigger uses: - - memory.c:2510-2530 iree_hal_streaming_memory_register_host_with_context_mode(…) - iree_hal_buffer_params_t params = { - .type = IREE_HAL_MEMORY_TYPE_HOST_LOCAL | IREE_HAL_MEMORY_TYPE_DEVICE_VISIBLE, // = 0x52, no 0x04 - }; - iree_hal_allocator_import_buffer(context->device_allocator, params, &external_buffer, …); - … - wrapper->host_register_flags = flags; // the computed coherence flags: STORED, not used - -**The type is hardcoded.** Whatever `flags` arrive — `HIP_COHERENT`, `HIP_NON_COHERENT`, or nothing — the -imported buffer is always `0x52`. - -## The flags are pure bookkeeping - -Every consumer of `host_register_flags` in libhrx, with the assignment lines excluded: - -| site | what it does | -|---|---| -| `memory.c:755` | `out_ref->host_register_flags = owner_wrapper->host_register_flags;` — copies it into a ref | -| `memory.c:2667` | `*out_flags = retained_ref.host_register_flags;` — **reports it back via `hipHostGetFlags`** | -| `memory.c:2405-2406` | stores `allocation_flags` | - -and `FLAG_HIP_COHERENT` appears exactly twice outside the enum, both in the HIP binding: - - api.c:754-755 if (hip_flags & hipHostMallocCoherent) { flags |= …FLAG_HIP_COHERENT; } - api.c:6908-6910 if (internal_flags & …FLAG_HIP_COHERENT) { hip_flags |= hipHostMallocCoherent; } - -**So the coherent bit round-trips — set from the HIP flag, stored on the wrapper, reported back by -`hipHostGetFlags` — and never reaches a memory type.** It is set, stored, reported, and never honoured. - -## What this closes, and the three-recorded-negatives it explains - -**The coherence hypothesis family is closed.** Not "we could not make coherence matter" but "this runtime -cannot make host memory coherent": both paths allocate `0x52`, and no flag changes that. Three separate -recorded results are now explained by one mechanism: - -1. `GGML_HRX_USE_UNIFIED_MEMORY=1` (the A/B in `315-mechanism.md`): only changes the **type recorded** on the - ggml buffer (`ggml-hrx.cpp:313-314`) — the pool is chosen by `HOST_LOCAL` alone, so both arms got the same - memory. **"Coherence does not matter" was really "the knob could not change the memory".** -2. The fix I prepared this session (OR `HOST_COHERENT` into the staging params) is a **no-op** — same pool. -3. The register path's coherence flags — computed, stored, reported — **never affect the allocation**. - -## The sharper question this leaves - -If host-visible memory is **always** non-coherent, then something else must make host reads of device-written -data correct *most of the time* — the campaign's reads are overwhelmingly fine. Whatever that something is, a -fault that appears **intermittently** cannot be "the memory is non-coherent" (that would be a constant, and it -would break every read, not one arm in twenty). So the mechanism must be something that **usually works and -sometimes does not**, and the coherence family is not it. - -That reframes the remaining candidate cleanly, and it is consistent with everything else measured: - -- the corruption is **one 256-byte-aligned region** (96.6 %, ≥89.5 % in every tensor family, vs 1.6 % chance) - — 256 B = one wave64 of f32, or the record's MoE-table block; -- it appears in **all three cells including the quiet one**, so the forcer is not necessary; -- and it is **intermittent**, which is what a *race or a lane-level drain* looks like, and is not what a - missing-invalidate looks like. - -**Nothing here names the mechanism.** It removes a family that six weeks of the record could not remove, and it -removes it in a form that also explains why the earlier attempts failed rather than merely disagreeing with -them. diff --git a/dev/315/315-campaign/evidence/315-corruption-is-one-row-sized.md b/dev/315/315-campaign/evidence/315-corruption-is-one-row-sized.md deleted file mode 100644 index 4c2ecf71c..000000000 --- a/dev/315/315-campaign/evidence/315-corruption-is-one-row-sized.md +++ /dev/null @@ -1,82 +0,0 @@ - - - -# The corruption is one hidden-state row's worth of bytes, at a position that moves - -## The observation, in bytes rather than counts - -The counts in `315-origin-varies-reinstated.md` look like different numbers until they are converted to bytes. -Every tensor here is 2048 wide (`ne0 = 2048`), so one row is `2048 × 4 = 8192` bytes: - -| arm | cell | in regime | first corruption | **first bytes** | layer | -|---|---|---|---|---|---| -| `315lout4` r1e | eviction | 98G | `l_out-0` = 1,984 | **7,936** | 0 | -| `315lout4` r2e | eviction | 98G | `l_out-0` = 1,984 | **7,936** | 0 | -| `315inp2` r3e | eviction | 98G | `l_out-1` = 2,048 | **8,192** | 1 | -| `315lout4` r1c | CPU | 98G | `l_out-0` = 46 | 184 | 0 | - -**Under the eviction trigger the first corrupted region is one row, to within 3 %** — 7,936, 7,936 and 8,192 -bytes, in arms whose first-corrupted *layer* differs (0, 0, 1) and whose token counts differ (98, 135, 98). - -And `7,936 = 8,192 − 256`, with **256 the alignment the record gives for the MoE tables** -(`dispatch-mul-mat-id-common.h:311-317`). So the two `l_out-0` arms are one row minus exactly one 256-byte block, -and the `l_out-1` arm is one full row. - -The CPU-pressure arm does not share the size (184 bytes) — so **the size is a property of the eviction path**, the -same way the layer-0 signature was earlier. - -## What this makes coherent - -Three things that had been separate now line up: - -1. **The size is fixed** (~one row) while **the position varies** (layer 0 in three arms, layer 1 in a fourth). - A fixed-size, variably-placed corruption is what a **region-oriented** event produces — and the intervention is - KFD eviction, which acts on memory regions, not on ops or layers. -2. **The row is the natural unit of the affected region.** One hidden-state row is exactly what a single - row-oriented write moves: a `get_rows` result, or **one KV-cache slot's row** for one token. §40's own timing - note already points at a slot — *"the first decoded token after `slot launch_slot_`"*. -3. **The spread then follows causally-masked attention.** A single corrupted row at position *k* is attended to by - positions **≥ k** and by no earlier one, so the NaN fraction after the next attention layer should be about - `(n − k)/n`. The observed growth is of that shape in every arm, and the implied *k* is consistent between - consecutive layers **within** an arm (`r1e`: 34.7 % then 36.7 % ⇒ k ≈ 64 then ≈ 62; `r2e`: 5.2 % then 5.9 % ⇒ - k ≈ 128 then ≈ 127) while differing **between** arms. **The position moves; the size does not.** - -## The hypothesis, stated so it can be wrong - -> Under forced KFD eviction, **one hidden-state row's worth of bytes (~8 KiB) is corrupted at a position that varies -> between episodes**, and the all-NaN logits are that single row propagating through causally-masked attention to -> the end of the sequence and then through the head — Face A when the corruption is NaN, Face B when it is finite -> but wrong. - -**Falsified by**: measuring the first corruption in ≥5 further in-regime Face-A arms and finding sizes that are not -~8 KiB, or finding the size varies while the position is fixed. - -## What is *not* claimed, and the caveat that has to travel with it - -- **Four arms, one session, and the same run supplied two of the identical numbers** — the withdrawal of the - `1984` "constant" two commits ago applies here too. What is new is that the *bytes* agree while the *layer* does - not, which is a different claim from "1984 is a constant" and is not yet replicated outside this session. -- **The temporal faithfulness caveat stands**: `ffn_inp-3` reading *fewer* NaN than `l_out-2` is impossible by - propagation, so at least one reading in this session is not the graph's value. The per-arm spread curve and the - byte sizes are therefore reliable as a *scale*, not as exact figures. -- **The KV-slot reading is a candidate, not a finding.** Nothing here distinguishes "one row of the residual - stream" from "one KV slot's row" from "one row of some other row-oriented buffer"; the observations constrain the - **size and motion**, not the identity of the buffer. diff --git a/dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md b/dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md deleted file mode 100644 index 0f73e4f34..000000000 --- a/dev/315/315-campaign/evidence/315-corruption-is-wave-granular.md +++ /dev/null @@ -1,332 +0,0 @@ - - - -# The corruption is WAVE-GRANULAR: it starts on a wave64 boundary in 96.6% of cases (chance 1.6%) - -Measured from the archive, controlled for the artifact that would have faked it. This is the sharpest -quantitative result in the #315 work, and it **discriminates between two candidates** that were both open. - -## The measurement - -Every hidden-state (`ne0 = 2048`) fingerprint reading that carries the extended `first=`/`row=` fields, split by -whether it is saturated (because a saturated reading is uninformative here — see the control below): - -| subset | readings | -|---|---| -| partial (informative) | **266** | -| saturated (`nan == extent`, excluded) | 450 | - -On the **266 partial readings**: - -| quantity | observed | chance | enrichment | -|---|---|---|---| -| `first` (start of corruption) divisible by **64 elements** | **257 of 266 = 96.6%** | 1.6% | **~60×** | -| `nan` (extent) divisible by 64 elements | 157 of 266 = 59.0% | 1.6% | ~37× | - -`first mod 64` histogram: **257 at 0**, then 4 at 40, 3 at 32, 1 at 37. There is essentially one residue. - -## The control that made this trustworthy - -The first cut over all 716 readings gave 98.7% and 84.8% — **and both numbers were inflated by saturation**, which -is the trap: for a fully-NaN tensor the extent is `2048 × T`, always divisible by 64, and `first = 0`, always -divisible by 64. So 450 of the 716 readings were structurally guaranteed to satisfy both tests. Excluding them -costs almost nothing — 96.6% and 59.0% — which is why the result stands. The uncontrolled version is reported -here rather than dropped, because the difference between them is the whole reason to trust the second. - -## Why the unit matters, and what it discriminates - -**64 *elements* of f32 = 256 bytes.** That is **one wave64** — 64 lanes of 32 bits each. It is **not** the 64-byte -cache line, which would be only **16 elements**. - -That distinction decides between the two candidates that were open: - -| candidate | its natural granularity | prediction for `first mod 64 == 0` | -|---|---|---| -| **host-staging coherence** (mine, `315-candidate-host-staging-coherence.md`) | **cache line = 64 bytes = 16 elements** | if corruption were only cache-line aligned, `first` would be a multiple of 16 and residue-0 mod 64 with probability ~**25%** | -| **wave64 lane masking** | **one wave = 64 elements = 256 bytes** | **~100%**, the observed 96.6% | - -The observation is 96.6% at residue 0 — an order of magnitude above what cache-line alignment predicts. -**So the data favours the wave64 candidate and disfavours the coherence candidate** as the *granularity* of the -corruption. (It does not by itself kill the coherence candidate, which could still be a *cause* whose effect -happens to be wave-quantised — but the wave shape is no longer explained by it, and the burden shifts.) - -## The mechanism this points at - -A write that operates on **whole waves** — an entire wave64 written, or left unwritten, atomically — at the -**front** of a residual row (`first` also lands at row 0 or 1), with the rest of the row correct. That is the -signature of **lane masking**: a wave whose lanes are not all written, or whose lanes were not drained, rather -than of a short byte-range write (which would corrupt a prefix and leave a clean tail — the opposite of what is -seen, since it is the *rest* of the row that is clean and a *hole* that appears). - -The record already carries this hardware class: the pin's `hrx-system` includes *"the GFX11 wave64 lane-mask -drain backport (**#313 NaN**)"*. **That backport is already in the pin**, so this is not #313 itself — it is the -same class of defect surviving it. - -## What it does NOT establish, and how to test further - -- **Not a cause.** It says the corruption is applied per wave, not what applies it. No instrument on the record - observes the write (the producer probe is the closest and sees only host-visible tensors). -- **Predictions it makes, all checkable:** - 1. corrupted elements should fall in **whole waves within a row**, with a clean wave beside them — testable by - dumping the row element-by-element rather than counting; - 2. counts should be `2048 − 64k` for small `k` when exactly one row is hit — the observed `k = 1` case; - 3. a **lane-mask drain** should be visible when the wave64 drain path is instrumented. -- **Sampling caveat:** 266 partial readings from a handful of arms and one configuration (`tail:16`, whose onset - site this session showed to be instrument-relative). The *granularity* result does not depend on where the - onset lands, but it should be re-measured under a name-list configuration before being treated as universal. -- **No rate:** arms were selected for having faulted. - ---- - -# CORRECTION to the interpretation above: the measured shape is a contiguous 256-byte-aligned PREFIX - -The granularity result above (96.6% of corruptions begin on a 64-element boundary) is sound and stands. But I -took it further than it goes by reading it as "whole waves with a clean neighbour", and the fuller shape -measurement corrects that. Both are now recorded, the measurement first. - -## What the extents actually show (266 partial readings) - -| property | measurement | -|---|---| -| NaN is a **single contiguous run** from `first` | **266 of 266** — `first + nan ≤ extent` in every reading | -| the run **starts at element 0** (`first = 0`) | **227 of 266 = 85.4%** | -| the run **covers the end of the buffer** | 30 of 266 | -| the run is **interior** (neither) | **9 of 266** | -| `first` is 256-byte (64-element) aligned | 257 of 266 = 96.6% | -| the run length is 256-byte aligned | 157 of 266 = 59.0% | - -Representative shapes: - - first= 0 nan= 2048 extent= 79872 rows [0.00: 1.00] of 39.00 <- exactly one row, the first - first= 0 nan= 55296 extent= 79872 rows [0.00: 27.00] of 39.00 <- first 27 rows - first= 0 nan= 1856 extent= 2048 waves[0:29] of 32 <- 29 of one row's 32 waves - first= 0 nan=1572928 extent=2441216 rows [0.00:768.03] of 1192.00 <- large prefix - first= 59904 nan= 19968 extent= 79872 rows [29.25: 39.00] of 39.00 <- the 30 tail cases - first= 264192 nan= 88064 extent=352256 rows[129.00:172.00] of 172.00 <- tail from row 129 - -So the picture is **a contiguous run of wrong data starting at the front of the buffer in 85% of cases**, not a -scattered set of waves. The `1984` case is "the first 31 waves of a 32-wave row are wrong and the **last** wave -is clean" — i.e. a prefix of the row, not a hole in the middle of it. - -## What this does and does not settle - -**Settled by measurement:** the corruption is contiguous, and its boundaries fall on **256-byte** (64-element) -lines far more often than chance — 96.6% versus 1.6% for the start. That is a real granularity result and it is -much coarser than a 64-byte cache line (16 elements), which is what the staging/coherence candidate's mechanism -would act on. - -**NOT settled — and I over-read it before:** *what* applies 256-byte-aligned contiguous prefixes. Two readings -remain live, and the data does not separate them: - -| reading | what it would mean | what would separate it | -|---|---|---| -| **a copy/region boundary** | a `copy_buffer`/region operation whose extent is short, or is applied to a sub-range, and whose source held NaN — the runtime's own 256-byte alignment unit (the record's "256 the alignment the record gives for the MoE tables", and `7936 = 8192 − 256`) would then be the natural boundary | whether the *unwritten tail* keeps its previous content or the *prefix* is actively wrong: a short copy leaves the tail stale; a wrong prefix means the copy's source is bad | -| **wave64 lane masking** | a kernel writing whole waves, with the first waves' lanes wrong and the last wave's drain correct | a lane-mask drain visible with the wave64 drain path instrumented, and a per-element dump showing whole waves wrong with a clean neighbour inside a row | - -**The discriminating observation is therefore the same for both, and it is cheap:** dump one corrupted row -element by element and see whether the wrong elements are (a) a prefix of whole waves with the tail clean -(region/copy), or (b) whole waves interleaved with clean waves (lane masking). Counting NaN cannot distinguish -them — which is exactly why the count-based reading went as far as it did and no further. - -**Also unchanged:** this is not #313 (whose wave64 lane-mask drain backport is already in the pin), it does not -establish a cause, and no instrument on the record observes the write. Sampling remains 266 partial readings -from a handful of arms in one (`tail:16`) configuration, whose onset site this session showed to be -instrument-relative — though the *shape* result does not depend on where the onset lands. - ---- - -# SECOND CORRECTION: contiguity was never measured — `nan` is a count and `first` is an index - -The section above states that "the NaN is a single contiguous run from `first`" on the evidence that -`first + nan ≤ extent` in 266 of 266 readings. **That evidence does not support that claim and the claim is -withdrawn.** - -`first + nan ≤ extent` is an arithmetic identity that any reading satisfies whenever the NaNs fit in the space -after the first one — contiguous or scattered. It is satisfied by a single run, by two runs, by every other -element, and by a run at the end plus one at the start. **The fingerprint reports exactly two numbers, `first` -(the index of the first NaN) and `nan` (how many there are); it never measures whether they are adjacent.** So -"one contiguous run" was an assumption dressed as a measurement, in a document whose whole point was to avoid -that — and it is the same failure mode as the raw row-index census earlier in this session (a count used to -support a positional claim). - -## What is actually measured - -| property | status | -|---|---| -| the **first** NaN is at element 0 (`first = 0`) | **measured**: 227 of 266 = 85.4% ⇒ the corruption *includes* the front of the buffer | -| the NaN **count** is 256-byte aligned | **measured**: 157 of 266 = 59.0% | -| the first NaN is at a 256-byte boundary | **measured**: 257 of 266 = 96.6% | -| the NaNs form one contiguous run | **NOT measured — withdrawn** | -| a "clean tail" or a "clean wave beside a wrong one" | **NOT measured** — both presuppose contiguity | - -So the two shapes the last section offered as alternatives — *prefix of whole waves with a clean tail* versus -*waves interleaved with clean waves* — **cannot be separated from this data at all**, and neither can be -asserted. What survives is narrower and still useful: **the corruption always reaches down to the front of the -buffer (85%), and its boundaries/counts fall on 256-byte lines far more often than chance.** - -## The consequence for the next instrument - -The per-element dump was named as the discriminator between the two readings. It is now not merely *a* way to -choose between them — it is **the only way**, because the count-based instrument is structurally blind to the -distinction. Any run that wants to settle whether the corruption is a region or a lane pattern must dump a row -**element by element** (or report per-wave NaN presence), not a total. - -That also means: **not one of the 266 readings on the record can distinguish a region from a lane pattern**, and -no amount of further counting will. The instrument to add is a per-wave (32-bit mask across the row's 32 waves) -or per-element dump — small, diagnostic-only, and in scope. - ---- - -# PRE-REGISTRATION: what the wave mask must read for each shape, fixed before the run - -The mask instrument is built and rides in both arms of the corrected `315ab` campaign. Its output is a 32-bit -hex word; these are the values the two candidate shapes must produce, fixed **before** any mask line exists, so -the result is scored rather than read. - -Bit *w* is set iff wave *w* — elements `[64w, 64w+64)` — contains at least one NaN, for the first 32 waves -(elements 0…2047 = one `n_embd` row). `0x7FFFFFFF` = `0b0111…1111`; `0xFFFFFFFF` = all 32 set. - -| shape | predicted mask | why | -|---|---|---| -| **one corrupted region at the front of the row** | a **contiguous run of set bits from bit 0**, e.g. `0x7FFFFFFF` for a 31-wave region | a region is contiguous by definition | -| **one clean wave inside an otherwise-wrong row** (the `1984` = 31-of-32 case) | `0x7FFFFFFF` if the *last* wave is clean; `0xFFFFFFFE` if the *first* is | it pins which wave the hole is in, which the count alone cannot | -| **whole-row corruption** | `0xFFFFFFFF` | every wave poisoned | -| **a lane/wave-interleaved pattern** | bits set with **gaps**, e.g. `0x55555555`, `0xAAAAAAAA`, or any non-contiguous run | this is the one shape a *count* can never show | -| **no mask line printed, though a fault occurred** | the instrument did not arm in that arm | INVALID for the mask question, and it must not be read as "no waves" | - -**The decisive comparison, stated in advance:** if every faulting arm's mask is a contiguous run, the corruption -is a **region**, and the wave64-lane reading loses; if any arm shows a **non-contiguous** mask, it is a -**lane/wave pattern**, and the region reading loses. A count of 1984 is consistent with both, which is exactly -why this instrument exists. - -**Also pre-committed, because it is easy to get wrong after the fact:** the mask covers only the **first 32 -waves (2048 elements)**. For a tensor whose `ne0 != 2048` the bit indices no longer correspond to one row's -waves, and such a reading must be **excluded** rather than reinterpreted. Every hidden-state tensor on this -record has `ne0 = 2048`, so this excludes nothing observed so far — but it is stated now, not after a -surprising mask appears. - -**What a mask result does not do:** it does not name the writer, and it does not by itself establish a -mechanism. It decides one binary question — region or lane pattern — that every previous instrument was blind -to. - ---- - -# LIMITATION: the front/tail split is dominated by TENSOR FAMILY, so "85% front" must not be pooled - -Checked because a position result that aggregates across tensor families can hide a family effect, and it does. - -| region shape | tensors it is made of | -|---|---| -| **FRONT** (`first = 0`), 227 readings | `ffn_moe_down` **89**, `ffn_moe_weighted` **63**, `ffn_shexp` 21, `norm` 19, `ffn_out` 7, `node_3019` 4 | -| **TAIL** (covers the end), 30 readings | **`l_out` 13**, `ffn_out` 10, `ffn_moe_weighted` 4, `result_norm` 2, `node_3305` 1 | -| **INTERIOR**, 9 readings | scattered, no family above 2 | - -So the two shapes are **not** the same phenomenon seen at different places — they are **largely different tensors**. -FRONT is overwhelmingly the **MoE path** (`ffn_moe_down`, `ffn_moe_weighted`); TAIL is the **residual output** -(`l_out`, `ffn_out`). Per arm the mix varies (e.g. `315fp2-p2l` is 156 front / 0 tail, while `315row-p1c` is -1 front / 3 tail), so the pooled 85.4% is a weighted average over families that behave differently. - -**Consequences, all of which narrow the earlier result:** - -- **"The corruption starts at the front" is a statement about the MoE-path tensors** that dominate the pool, not - a law about the corruption. The `l_out` family — the one the size invariant was measured on — is - **over-represented in the TAIL** group. -- **The family split is a live alternative to any positional reading**: the two shapes could be one mechanism - seen through two different tensors, or two different mechanisms, and **this data cannot tell which**. -- **A pooled percentage is not the right statistic here.** Any future statement of the position result must be - **per tensor family**, with the family's own denominator — which this table now supplies. - -**A read-faithfulness caveat that applies to the FRONT group specifically.** `ffn_moe_down` and -`ffn_moe_weighted` are *intermediate* MoE tensors, and the record already establishes that intermediate reads -can be unfaithful (arena slot reuse; the aliased `l_out-42 ≡ l_out-44` case). So the FRONT group leans on -exactly the reads the record distrusts most, while the TAIL group leans on `l_out`, the family with the -best-validated readings. That is an argument for trusting the TAIL observations more, not less — the reverse of -what the pooled count suggests. - -**Unchanged by this:** the 96.6% 256-byte alignment of `first`, which is computed over the same partial readings -and is not obviously family-specific — but it should be re-cut per family before being quoted as a headline. - ---- - -# The alignment result SURVIVES the per-family re-cut — it is not a family artifact - -The limitation above says a pooled percentage is the wrong statistic and the alignment result must be re-cut -per family. Done, and it holds in every family: - -| family | n | first on a 256-byte boundary | % | -|---|---|---|---| -| `ffn_moe_down` | 91 | 91 | **100.0%** | -| `ffn_moe_weighted` | 67 | 63 | 94.0% | -| `ffn_shexp` | 22 | 22 | **100.0%** | -| `norm` | 21 | 21 | **100.0%** | -| `ffn_out` | 19 | 17 | 89.5% | -| `l_out` | 16 | 15 | 93.8% | -| `node_2948`, `node_3019`, `node_3090`, `node_3161`, `node_3232` | 4 each | 4 each | **100.0%** | -| `ffn_moe_out` | 3 | 3 | **100.0%** | - -**Pooled: 252 of 259 = 97.3%** across families with n ≥ 3 (and 257 of 266 = 96.6% over all partial readings), -against a **1.6%** chance rate. **Eight of the twelve families are at exactly 100%**, and the lowest is 89.5%. - -So the alignment is **family-independent**. That matters in both directions: - -- It **narrows** the position claim (the front/tail split really is a family effect, as the limitation says), and -- it **strengthens** the alignment claim, because it is no longer an average that a single dominant family could - carry. `ffn_moe_down` alone is 91 of 259 readings; even excluding it entirely the remaining families are - ~96% aligned. -- It also **weakens the read-faithfulness objection to the alignment** specifically: the caveat says the FRONT - group leans on intermediate MoE tensors whose reads the record distrusts, but the alignment holds just as - firmly on `l_out` (15/16) and `node_2948` (4/4) — the residual-stream tensors with the **best-validated** - readings. An artifact of unreliable intermediate reads would not reproduce on the families the record trusts. - -**Standing result, stated at the strength the data supports:** - -> The corruption **begins** on a 256-byte (64-element) boundary in **96.6%** of partial readings overall and in -> **at least 89.5%** within every individual tensor family tested — including `l_out` and `node_2948`, whose -> readings the record treats as faithful — against a 1.6% chance rate. It is therefore a **256-byte-granular** -> corruption, not a byte- or element-granular one. - -**What remains open, unchanged:** *what* applies 256-byte-aligned damage (a copy/region boundary versus wave64 -lane masking), whether the NaNs are contiguous at all, and the writer. The mask instrument decides the second -of those and rides in the running campaign. - ---- - -# Narrowing the interpretations of the 256-byte granularity: the copy/region reading is weakened - -Three readings were left open for *what* applies 256-byte-aligned damage. One is now weakened by checking the -runtime rather than by argument: - -| reading | status after checking | -|---|---| -| **copy / region granularity** — the runtime copies in 256-byte blocks, so a short or sub-range copy lands on those boundaries | **weakened.** No 256-**byte** alignment or granularity constant exists in the copy path: `libhrx/src/binding/common/memory.c` has `staging_threshold = 256 * 1024` (256 **KiB**, a size threshold, nothing to do with 256 bytes), and the HIP copy path adds copy nodes without size rounding. The allocator's own alignment machinery is page-granular (`virtual_memory_protect` etc.), i.e. 4096, not 256 | -| **one wave64** — 64 lanes × 4 bytes = **256 bytes exactly** | **unaffected**, and relatively favoured by the above: 256 bytes is intrinsic to the hardware's wave width rather than a policy constant someone chose | -| **a 256-byte data-layout block** — the record's own figure, *"256 the alignment the record gives for the MoE tables"* | **unaffected**, and it has independent support: the FRONT group is dominated by MoE tensors (`ffn_moe_down` 91, `ffn_moe_weighted` 67), i.e. exactly the path that indexes those tables | - -**What this does not do:** establish the wave64 reading. Absence of a 256-byte copy constant is weak evidence — -a granularity can exist without a named constant — and the MoE-table reading is not excluded at all. It narrows -three candidates to two, which is what a source check can honestly do here. - -**It also means the alignments may have more than one cause.** The residual-stream families (`l_out`, -`node_2948`) and the MoE families (`ffn_moe_down`, `ffn_moe_weighted`) both align at ≥89.5%, and they could -align for different reasons — a wave-wide write in one case and a table-block boundary in the other. Nothing in -this data separates those two, and the mask instrument (which reports per-wave NaN presence for one row) is -**not** a test of the MoE-table reading, because it looks at a tensor, not at a table index. Stated so the mask -result is not over-read as settling everything. diff --git a/dev/315/315-campaign/evidence/315-crit4-control-result.md b/dev/315/315-campaign/evidence/315-crit4-control-result.md deleted file mode 100644 index 78f34167e..000000000 --- a/dev/315/315-campaign/evidence/315-crit4-control-result.md +++ /dev/null @@ -1,170 +0,0 @@ - - - -# Criterion-4 control re-run: ALL EIGHT HRX TESTS PASS, matching the baseline - -Ran 2026-10-13 17:09 on the current pin (`e44c9d01a`), from `~/wt/v315-build`, under -`systemd-run --user --scope -p MemoryMax=6G`. This is the objective's **"no regression: engine CI green, and -the dense-model control plus the MUL_MAT_ID op-test unchanged relative to baseline"**. - -## Result - - --- test-hrx-buffer PASS (5 passed, 0 failed) - --- test-hrx-mul-mat-id-k32 PASS - --- test-hrx-mxfp4 PASS - --- test-hrx-moe-split PASS - --- test-hrx-attention-sink PASS - --- test-hrx-decode-stride PASS - --- test-hrx-hadamard PASS ("test-hrx-hadamard: PASS" printed by the binary) - --- test-hrx-fa-masked-v PASS - -**8 of 8 pass.** The `test-hrx-ops` and `test-hrx-loom-jit` binaries the baseline notes as not compiling at -this revision are absent from the build, consistently with that note. - -## Direct comparison against the recorded baseline - -| measure | baseline (2026-10-09) | today (2026-10-13) | verdict | -|---|---|---|---| -| HRX buffer cases | `5 passed, 0 failed, 5 total` | `5 passed, 0 failed, 5 total` | **unchanged** | -| individual cases | `backend_buffer`, `host_buffer`, `host_transfer`, `host_staging`, `host_weight_cache` all `[PASS]` | same five, all `[PASS]` | **unchanged** | -| MUL_MAT_ID `mxfp4` nmse | 1.23e-05 … 1.58e-05 | 1.34e-05 … 1.61e-05 | **same range, no drift** | -| kernels selected | `ggml_mul_mat_id_decode_f32_wave64`, `ggml_mul_mat_id_f32_f32_wmma` | same two | **unchanged** | - -The `nmse` figures are the important half: they are a numeric fingerprint of the reduce and id-resolution -path, so "PASS" plus an unchanged `nmse` range means the op-test has genuinely not drifted, not merely that -it avoided failing. - -## What this does and does not establish - -- **Establishes**: the engine's own regression controls are **unchanged on the shipped pin today**. That is - criterion 4's claim about the controls, and it is now measured rather than assumed. -- **Does not establish criterion 4 as MET**: the criterion pairs this with a fix to compare against - ("baseline-vs-fix on the same harness"), and **no fix has landed** — the one prepared candidate is refuted. - With nothing landed, these tests confirm the baseline the future fix must match, and nothing more. -- Also **not run**: the "dense-model control". The objective names it alongside the op-test; this run covers - the HRX test suite and the MUL_MAT_ID op-test. The dense-model control should be named explicitly before it - is claimed either way. - -## Harness note - -Two attempts were needed and both failures are recorded rather than hidden: the first waited on the wrong -clear-window rule, and the second **matched no tests at all** because I used guessed binary names -(`test-hrx-backend-buffer` etc.) for cases that actually live inside `test-hrx-buffer` -(`315-engine-ci-baseline.md`). The corrected run is `315crit4c.sh`. - ---- - -# The dense-model half of criterion 4: staged - -The objective's criterion 4 names **"the dense-model control plus the MUL_MAT_ID op-test unchanged relative -to baseline"**. The op-test half is done (8/8 HRX tests pass, `MUL_MAT_ID` nmse baseline-matched, above). The -dense half is now staged. - -## What is staged - -`~/wt/315dense.sh` — runs a **dense** model on HRX0 under the same detector shape the arms use and records -whether it faults: - - model : Qwen3-8B-Q4_K_M (4795 MB) - a DENSE model, not MoE, so it lacks the routed-FFN/id-producing - structure the fault's leading family involves - device : HRX0, -ngl 99, -fa on, c 4096, --no-webui - load : 50 mixed-length requests (the detector's shape), max_tokens 32, temperature 0 - detects: nan_guard lines, HSA_STATUS_ERROR/Queue error, Content-only parse failures, and a response count - gate : waits for a sustained 120 s clear window (another agent launches campaigns back-to-back) - -## Why it is a meaningful control - -The recorded finding is that dense models **do not** exhibit this fault while GLM (MoE, routed FFN) does. So a -**clean** dense run is the expected baseline, and its value is as a *negative control* for the harness: it -shows the detector, the regime composition and the build do not fault everything in sight, which is what makes -the GLM arms' high eviction-cell rate attributable to the model/shape rather than to the apparatus. - -## Status - -Staged, not run — arm B of the alignment test holds the box. It will be run after arm B releases it, and its -result recorded either way. - ---- - -# Harness defect (seventh of the session): my monitoring prevented the dense control from starting - -The first dense-control script waited on - - pgrep -f "galign|315-disc.sh|315alignf" - -to find a clear window. It never cleared — because **every monitoring command I ran contained that same -pattern text** (`pgrep -af "...315-disc.sh..."`), so `pgrep -f` matched my own shell's command line and reset -the clear counter each time. The control sat in its wait loop for >4 minutes, `do_wait` on a `sleep 30`, with -the box genuinely idle. - -That is an **observer effect of my own making**: the act of checking whether the run could start prevented it -from starting. Diagnosed by running the pattern from a script file (where the text is not in any argv) and -seeing two `bash -c` matches that were *my own SSH one-liners*. - -Fixed as `315dense2.sh`, which counts only real driver processes — reading each candidate's -`/proc//cmdline` and skipping `bash -c` wrappers — so no monitoring one-liner can mimic a driver. - -This is the seventh harness/reasoning defect recorded this session, and it belongs to the same family as the -`pgrep` self-match scare earlier: **`pgrep -f` sees whatever text is in a command line, including the one -asking the question.** - ---- - -# The dense-model half: VALID and CLEAN — criterion 4 is now measured on both halves - -Ran 19:02–19:05, Qwen3-8B-Q4_K_M (a **dense** model, not MoE) on HRX0 at GTT 44→55 GiB, the same detector -shape the arms use (50 mixed-length requests, `max_tokens=32`, `temperature=0`). - - responses 50 of 50 - distinct content 1 (every response identical — deterministic and coherent) - NaN guard 0 - Content-only 0 - HSA_STATUS_ERROR 0 - server tasks 1617 (0 errors; "graphs reused = 1539") - -**Clean.** That is the expected negative control: a dense model does not exhibit the fault, so it -demonstrates the apparatus — detector, build, HRX0 path, request shape — does not fault everything in sight. -That is what makes the GLM arms' high eviction-cell rate attributable to the model/shape rather than to the -harness. - -## Criterion 4, both halves now MEASURED - -| half | result | -|---|---| -| MUL_MAT_ID op-test + HRX suite | **8/8 pass**; `MUL_MAT_ID` nmse matching baseline (1.34–1.61e-05 vs 1.23–1.58e-05) | -| dense-model control | **clean**: 50/50 responses, 1 distinct content, no fault signatures | - -Criterion 4 is still not *met* in the criterion's own terms, because it pairs these controls with a **fix to -compare against** ("baseline-vs-fix on the same harness") and no fix has landed. What is established is that -the baseline the future fix must match is measured and currently intact. - -## Eighth harness/parse defect of the session - -My own script reported `responses=1` and printed `RESULT: FAULT or incomplete`. **That was wrong.** The -script counted with `grep -c '"content"'`, which counts *lines* — and `/v1/chat/completions` responses are -concatenated without newlines, so fifty responses sit on one "line" and the count is 1. - -Counted correctly (`grep -o … | wc -l`) it is **50**. Anyone reading only the script's verdict would have -concluded the dense control faulted, which is the opposite of the truth. - -This is the **same class** as an earlier defect this session (a line-based counter reading zero on -newline-free NDJSON). Twice on the same mistake is worth stating plainly: **for any concatenated-JSON -endpoint, count occurrences, never lines.** diff --git a/dev/315/315-campaign/evidence/315-deferred-writeback-gap.md b/dev/315/315-campaign/evidence/315-deferred-writeback-gap.md deleted file mode 100644 index 69ea6d129..000000000 --- a/dev/315/315-campaign/evidence/315-deferred-writeback-gap.md +++ /dev/null @@ -1,477 +0,0 @@ - - - -# A self-documented ordering gap: the deferred host writeback (`mark_stream_synchronized`) - -Found while working the §55/§56 cross-dispatch leg after four families were closed. This is the first -candidate located **by the code saying so about itself**, rather than by inference from behaviour. - -## The mechanism, read at the pin (`e44c9d01a`, ggml-hrx) - -`ggml/src/ggml-hrx/runtime/command-program-executor.cpp` keeps a **deferred host writeback queue**: - - GraphReplayStreamState::add_host_writeback(host_destination, mapped_source, size, buffer) // :51 - -> push_back({host_destination, mapped_source, size, buffer}) // merely QUEUES the memcpy - - GraphReplayStreamState::mark_stream_synchronized() // :38 - for each pending writeback: - std::memcpy(writeback.host_destination, writeback.mapped_source, writeback.size) - pending_host_writebacks_.clear() - -So a **host download staged by an earlier replay is not published to host memory when it happens** — it is -queued, and only performed later, at whatever point `mark_stream_synchronized()` runs. - -`mark_stream_synchronized()` is reached from exactly three places: - - :888 the host-staging UPLOAD guard (see below) - :1926 mark_graph_replay_synchronized(...) after a replay - :1985 mark_graph_replay_synchronized(...) after a "wait for HRX command program reading host memory" - -## The gap, in the code's own words - -The upload path carries this comment (`:874-880`), and it is the load-bearing find: - -> A host download staged by an earlier replay is only published to host memory by -> `mark_stream_synchronized()`; `add_host_writeback` merely queues the memcpy. **Uploading the same host -> memory before that has happened would read the previous contents.** The split scheduler computes splits -> back to back and only synchronizes a backend when it inserts a cross-backend input copy, **so nothing -> else guarantees the ordering.** - -Two things follow: - -1. The code **knows** there is a window in which host memory holds the *previous* contents while a - download that should have updated it is still queued. The comment names the exact consequence: a - subsequent read "would read the previous contents". -2. The only guard is at the **host-staging upload** site, and it is conditional: - `if (context.graph_replay_state != nullptr && has_pending_host_writebacks())`. Nothing guarantees the - ordering generally — the comment says so outright. - -## Why this is a candidate for THIS fault - -- **It is a stale-read of host memory**, which is the fault's shape: a consumer observes "the previous - contents" of a buffer rather than what a producer wrote. That is Face B almost literally (a stale token - buffer producing malformed output), and plausibly Face A if the stale bytes are non-finite. -- **It is deferred across a window.** Unlike `buffer_get` (the refuted candidate), which read at one - instant, this queue holds a *pending* copy that is flushed at some later, conditionally-chosen point. - A preemption/eviction landing anywhere inside that window can change what the queued `mapped_source` - contains, which is exactly the trigger the pooled rig establishes (28 of 29 eviction-cell arms fault; - `315-pooled-rig-verification.md`). -- **It explains why the read-path fix failed.** The A/B refuted `hrx_synchronous_d2h` at the `buffer_get` - site; this is a *different* host read of a mapped buffer, on a different path, with a deferral the - refuted site did not have. -- **It is on GLM's decode path.** The writeback queue is driven by host-staging downloads, which is how - the executor publishes results back to host memory during decode. - -## What is NOT established - -- **Not shown to be reached with GLM's shapes.** Whether GLM's decode actually produces - `pending_host_writebacks_` entries, and whether an upload of the same host memory ever follows while - they are pending, is checkable — the queue is countable and `GGML_HRX_LOG_*` exists — but has not been - checked yet. -- **Not shown to be a defect.** The guard at `:888` may cover the only ordering that matters in practice, - and the `:1926`/`:1985` sync points may be sufficient for every real sequence. The comment's "nothing - else guarantees the ordering" is about the *general* case, not proof of a reachable failure. -- **Not a cause.** It is a located, self-documented hazard of the right shape on the right path. - -## The bounded next steps, cheapest first - -1. **Count it.** Run one GLM decode with the existing instrumentation and establish whether - `pending_host_writebacks_` is ever non-empty, and whether `mark_stream_synchronized` runs between a - download and a later upload of the same host memory. Zero arms, no regime, ~2 minutes — the same - class of probe that established the graph reorder was live (`315-graph-reorder-lever.md`). -2. **If it is reached**, the discriminating test has the same cheap A/B form: force the sync - (`mark_stream_synchronized`) before every host-staging upload unconditionally, and compare cell `e` - against the shipped conditional guard. If the fault rate drops, this is the carrier. -3. **If it is not reached**, this family closes like the others, with a measured reason. - -Recorded because it is the first candidate found by the code documenting its own ordering hazard, and -because it is the only surviving cause candidate that is both in scope and cheaply testable. - ---- - -# The instrument for this already exists — and has never been run - -Checked before building anything, and the writeback queue is **already instrumented** in the tree that -`315coherent-build` was built from (`~/wt/llama-315coherent`, and also `~/wt/llama-315copy`): - - GGML_HRX_WRITEBACK_FINGERPRINT=1 - -> fingerprint_writeback_log(...) command-program-executor.cpp:84 - "[hrx-wb-fp] value=%d size=%zu barrier=%d rec_hash=%016llx sync_hash=%016llx dst_hash=%016llx - rec_class=%s sync_class=%s dst_class=%s rec_eq_sync=%d - rec_nf16=%llu sync_nf16=%llu dst_nf16=%llu rec_nf32=%llu sync_nf32=%llu dst_nf32=%llu" - -That is purpose-built for exactly the question this lead poses. Per writeback it records: - -- **`rec_hash`** — the source fingerprint **when the writeback was recorded** (the download); -- **`sync_hash`** — the source fingerprint **at the sync point**, when the deferred `memcpy` would run; -- **`dst_hash`** — the destination after the copy; -- **`rec_eq_sync`** — **whether the source changed between recording and sync.** `rec_eq_sync=0` is the - deferred-window hazard made directly observable: the queued `mapped_source` held different contents at - record time than at sync time; -- **`rec_nf16/32`, `sync_nf16/32`** — non-finite counts on each side, which is the link to Face A. - -And there is a driver already written for it: `dev/315/315-campaign/315-fp-hunt.sh` runs instrumented -in-regime arms and reports `wb_records=$(grep -c "\[hrx-wb-fp\]" ...)` per arm. - -**Neither has ever produced output.** A search of the whole record (`docs/`, `KNOWN-ISSUES.md`) and of the -box (`~/wt/*console*.log`, `~/wt/rt-det/*/server.log`) finds **zero** `[hrx-wb-fp]` lines anywhere. The -instrument was built on 2026-10-13 and, as far as the record shows, never switched on. - -## Consequence for the plan above - -Step 1 as written ("count it with existing instrumentation") needs no new code and no fault — one GLM -decode with `GGML_HRX_WRITEBACK_FINGERPRINT=1` answers whether the queue is used at all, and if it is, -whether `rec_eq_sync` is ever 0 under the eviction rig. That probe is staged at -`~/wt/315wb-probe.sh` and queued behind the peer agent's campaign. - -This also means the four closed families were worked while a purpose-built instrument for a *fifth* -candidate sat unused in the same binary — worth recording as a process observation, not a criticism: the -instrument was built for a different framing (download fingerprints) and its writeback half had no -consumer until this lead gave it one. - ---- - -# The queued source is a NON-COHERENT `0x52` mapping — the same type the refuted candidate named - -Traced the writeback's source buffer to its allocation, which sharpens this lead from "a deferred read" -to "a deferred read of the same memory type the refuted fix was about". - -Chain (`~/wt/llama-315coherent`, the tree `315coherent-build` came from): - - download_prepared_host_staging(...) command-program-executor.cpp:1065 - for each prepared.host_staging with .download: - allocate_mapped_host_staging_buffer(...) -> download_data (a mapped pointer) - hrx_stream_copy_buffer(staging.buffer -> download_buffer) device -> staging copy - add_host_writeback(staging.host_data, download_data, ...) QUeues: dest=host_data, src=download_data - - allocate_mapped_host_staging_buffer(...) runtime/host-memory.cpp:1287 - hrx_buffer_params_t params = { - HRX_MEMORY_TYPE_HOST_LOCAL | HRX_MEMORY_TYPE_DEVICE_VISIBLE, // :1300 == 0x52 - HRX_MEMORY_ACCESS_ALL, - HRX_BUFFER_USAGE_DEFAULT | HRX_BUFFER_USAGE_MAPPING_SCOPED | HRX_BUFFER_USAGE_MAPPING_PERSISTENT, - }; - hrx_allocator_allocate_buffer(...); - hrx_buffer_map(buffer, HRX_MAP_READ | HRX_MAP_WRITE, 0, size, &host_data); // :1310 - -So `download_data` — the pointer the deferred `memcpy` reads from, at whatever later moment -`mark_stream_synchronized()` runs — is a **`HOST_LOCAL | DEVICE_VISIBLE` = `0x52` mapping with no -`HOST_COHERENT`**: exactly the memory type the read-path mechanism chain named, and exactly the type the -refuted candidate read, just at a different site. - -## Why that makes this materially stronger than the refuted candidate - -- **The refuted candidate** read the non-coherent logits mapping at a **single, immediate** point, and its - fix (route the read through a coherent queued transfer) was shown not to help — cell `e` faulted 4/4 - with the fix on (`315-ab1-arm-B-interim.md`). -- **This one is deferred.** The non-coherent source is read at the *sync point*, arbitrarily later than the - device copy that populated it, and the code states the ordering is not otherwise guaranteed. A - preemption/eviction landing anywhere in that window is precisely the trigger the pooled rig establishes - (28 of 29 eviction-cell arms; `315-pooled-rig-verification.md`). -- The instrument already in the binary measures exactly this: `rec_hash` at record time vs `sync_hash` at - sync time, with `rec_eq_sync=0` marking a source that changed while queued. - -## Still not established - -- Whether GLM's decode produces `host_staging` buffers with `.download` at all, and therefore whether - `pending_host_writebacks_` is ever non-empty on this path. That is what the staged probe - (`~/wt/315wb-probe.sh`) answers. -- Whether the `:994` guard covers every reachable sequence even when the queue is non-empty. -- It is not a cause; it is a located, self-documented hazard of the right memory type, deferred across the - window the trigger opens. - ---- - -# The logits are structurally expected to take this path: `direct_host_binding=0` - -Completed the chain from the other end — from GLM's logits binding to the deferred writeback — using the -recorded allocation trace rather than inference. - -## The gating condition - -The writeback is queued in `materialize_bindings` (`command-program-executor.cpp:764`) for any binding that -is host-visible and not a graph input: - - staging.download = access.write && !binding.graph_input; - -guarded by `if (binding.host_data == nullptr) { ... continue; }` (`:750`) — i.e. only **host-visible** -bindings reach it. A host-visible binding that the graph **writes** therefore queues a deferred writeback. - -## The logits are exactly that, and not by the direct path - -The recorded trace (`dev/315/315-mechanism.md:34`, and `315-task9-session-state-2026-10-12.md:3304`) shows the -logits allocation as: - - type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=619520 - -and `ggml-hrx.cpp:319` defines the flag: - - const bool direct_host_binding = host_visible && type_context->device->use_direct_host_bindings; - -with `use_direct_host_bindings = environment_flag_enabled("GGML_HRX_USE_UNIFIED_MEMORY")` (`:1202`), which -**defaults off**. So on the shipped configuration the logits are `host_visible=1` but -`direct_host_binding=0` — host-visible *without* the direct binding — which is precisely the condition that -routes them through host staging rather than binding host memory in place. - -## What that means for this lead - -- The logits are **host-visible** (so `host_data != nullptr` holds) and are **written by the graph** (so - `access.write` holds), and they are **not** a graph input. All three terms of - `access.write && !binding.graph_input` are satisfied. -- They are **not** on the direct-binding fast path (`direct_host_binding=0`), which is the path that would - bypass staging entirely. -- The staging buffer that the deferred read later dereferences is the non-coherent `0x52` mapping traced - above. - -So the deferred-writeback path is **structurally expected** to carry the logits, not merely reachable in -principle. - -## Still owed, and it is only a measurement - -All of the above is static. Whether `pending_host_writebacks_` is non-empty at the right moments during a -GLM decode, and whether `rec_eq_sync` is ever 0 under the eviction rig, is what the staged probe -(`~/wt/315wb-probe.sh`, one decode with `GGML_HRX_WRITEBACK_FINGERPRINT=1`) reports. Static expectation is -not a result, and this section is explicitly not claiming one. - ---- - -# Completing the mechanism: the writeback is queued at the END of a replay and NOT flushed there - -Traced the replay path end to end. The deferral is not incidental — the queue is populated as the last -act of a replay, and the replay returns without flushing it. - -## The three download sites, and what follows each - -`download_prepared_host_staging(context, prepared, insert_barrier)` is called from three places: - - :1978 download_prepared_host_staging(context, prepared) insert_barrier = true (default) - -> followed by `result.success = true; return result;` (the diagnostic-kernel path) - :2047 download_prepared_host_staging(context, prepared, false) insert_barrier = FALSE - -> followed by `result.success = true; return result;` (the main path) - :2082 download_prepared_host_staging(context, commands) (the execute_program path) - -Inside, the function queues each download and returns: - - for each host_staging with .download: - allocate_mapped_host_staging_buffer(...) -> non-coherent 0x52 mapped buffer - hrx_stream_copy_buffer(staging.buffer -> download_buffer) device -> mapped copy - add_host_writeback(staging.host_data, download_data, ...) QUEUES the host memcpy - return status; - -**None of the three call sites calls `mark_stream_synchronized()` afterwards.** The two that return -directly (`:1978`, `:2047`) return with the writeback still pending; the third hands control back to -`execute_prepared_command_program`, whose flush point is conditional. - -## So the sequence on the main path is - - replay launches the graph - -> download_prepared_host_staging(..., insert_barrier=false): - device -> mapped (0x52) copy, then QUEUE a host memcpy from that mapped buffer - -> replay returns success, writeback still PENDING - -and the queued memcpy is executed only when some later caller reaches `mark_stream_synchronized()` — -which, per the code's own comment, is the host-staging **upload** guard plus the two -`mark_graph_replay_synchronized` sites, and "nothing else guarantees the ordering." - -## Why the asymmetry matters - -The `insert_barrier` argument controls whether a **stream execution barrier** is inserted before the -download — a device-side ordering fence. It does **not** control the host-side flush of the queued -writeback. So: - -- the device-side copy into the staging buffer is ordered by the barrier when `insert_barrier=true`, and - by ring program order otherwise; -- the **host-side** read of that buffer (`download_data`) happens at an unrelated, later moment, and - nothing ties it to the device copy that populated it beyond the eventual sync. - -That is a genuine window, and it is wider on the main path (`insert_barrier=false`, `:2047`) than on the -diagnostic one (`:1978`). - -## The instrument answers this exactly - -`GGML_HRX_WRITEBACK_FINGERPRINT=1`'s `rec_hash` is taken when the writeback is queued and `sync_hash` when -it is executed. **`rec_eq_sync=0` means the mapped source held different bytes at flush time than when it -was recorded** — which is what an eviction or the CP writing through during the window would produce. The -probe staged at `~/wt/315wb-probe.sh` reports exactly that tally. - -Explicitly still static: this section shows the window exists and is unflushed at the end of a replay. It -does not show the source ever actually differs at flush time, which is the measurement. - ---- - -# MEASURED: the logits ARE host-visible `0x52`, and the writeback path is still NOT entered - -The buffer trace was run in the build the earlier probes used, with both knobs on, to settle whether this -lead is reachable at all. It produced the sharpest result of the whole lead — and it **contradicts the -static reading** above. - -## The trace - - [hrx-buf] type=HRX0 host_visible=0 direct_host_binding=0 memory_type=0x30 size=17895352832 - [hrx-buf] type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=619520 <- the logits - [hrx-buf] type=HRX0 host_visible=0 direct_host_binding=0 memory_type=0x30 size=443547648 - [hrx-buf] type=HRX0 host_visible=0 direct_host_binding=0 memory_type=0x30 size=81788928 - [hrx-buf] type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=44048640 - -and, from the same run: - - writeback records: 0 - -So the static prediction **held**: the logits are `HRX0_HOST`, `host_visible=1`, `direct_host_binding=0`, -`memory_type=0x52` (not `HOST_COHERENT`) — exactly the allocation the chain above says should flow through -host staging. **And no writeback was queued anyway.** - -## Where the static reading and the measurement disagree - -`backend-buffer-binding.cpp:76-80` appears to support the static reading outright, in its own comment: - - const bool directly_bindable = !ggml_backend_buffer_is_host(buffer) || context->direct_host_binding; - // Coherent HRX host allocations are directly device-addressable. ... Noncoherent host allocations - // remain host data. - binding.buffer = directly_bindable ? context->buffer : nullptr; - binding.host_data = directly_bindable ? nullptr : context->base; - -with `direct_host_binding = 0` in the trace, `directly_bindable` should be `!is_host(buffer)`. For the logits -that is the open question — and the honest statement is that **this file's comment predicts staging for a -non-coherent host allocation, while the measurement shows staging is not entered.** One of the two is not -being read correctly, and the measurement is the one that cannot be wrong about the event count. - -The likely resolutions, none verified: - -- the logits' binding reaches the **first** early-return branch (`backend-buffer-binding.cpp:45-68`), which - sets `binding.host_data = base` and returns immediately — host data, but **never passed through - `materialize_bindings`**, so `staging.download` is never evaluated; -- or `ggml_backend_buffer_is_host(buffer)` is true for the logits, making `directly_bindable` false and - sending them down a different route than assumed; -- or the host-staging entries are materialized elsewhere for this binding shape. - -Distinguishing these is a bounded read of one more function, not a GPU question. - -## What is now established, and what is not - -- **Established**: the logits are a non-coherent `0x52` host-visible allocation in this build (measured, - not inferred). -- **Established**: the `GGML_HRX_WRITEBACK_FINGERPRINT` instrumentation is compiled in and callable, and - it recorded nothing across two sound probes (10 varying graphs) plus this trace run. -- **NOT established**: that the deferred writeback is unreachable in general. It is unreached on the paths - exercised here. -- **Corrected**: the earlier section's claim that the logits are "structurally expected" to take the - staging path is **not supported by measurement**. It rests on reading one comment as decisive when the - event count says otherwise. That inference is withdrawn pending the bounded read named above. - ---- - -# CLOSED by mechanism: the logits binding carries `host_data == nullptr`, so it is never staged - -Completed the bounded read named above. The contradiction resolves, and it resolves **against** this lead — -not by absence of output but by the binding's own resolution. - -## The predicate chain, now complete - -`materialize_host_bindings` (`command-program-executor.cpp:686`) is where staging is decided, and its gate is -`:700`: - - if (binding.length == 0 || (!binding.requires_materialization() && !materialize_weight)) { - ... continue; // NOT staged - } - -and the predicate is one line (`dispatch/command-program-bindings.h:31`, identically in -`graph/value-map.h:67`): - - bool requires_materialization() const { return host_data != nullptr; } - -So **staging happens if and only if the binding's `host_data` is non-null** — nothing else. And -`backend-buffer-binding.cpp:78` sets it: - - binding.host_data = directly_bindable ? nullptr : context->base; - -with `directly_bindable = !ggml_backend_buffer_is_host(buffer) || context->direct_host_binding`. - -The measurement fixes the answer: the writeback path is **never entered** across two sound probes and the -trace run, so for GLM's decode the logits binding resolves with **`host_data == nullptr`** — i.e. as -directly bindable (an HRX buffer handle), *not* as host data. The allocation is host-visible `0x52`; the -**binding** is nonetheless device-addressable. - -## What I got wrong - -The earlier section read `backend-buffer-binding.cpp:78`'s comment — "Noncoherent host allocations remain -host data" — as decisive for the logits, and concluded the staging path was "structurally expected". That -was an inference from a comment about the *general* case, contradicted by an event count for *this* binding. -The prediction is withdrawn, and the reason is recorded: the comment describes what happens to non-coherent -host allocations **that take this branch**, whereas the logits resolve through the directly-bindable branch -and never present host data to the materializer at all. - -## Consequence - -The deferred host writeback is **not** on GLM's decode path in this configuration. This family closes the -same way the other four did — with a mechanism, not a shrug: - - gate binding.host_data != nullptr - actual the logits binding's host_data IS nullptr (writebacks never queued, measured) - therefore the queued-writeback window never opens for this workload - -The relevant non-coherent host read on the logits still exists — that much the trace confirms — but it is -the **immediate** read the A/B already refuted (`buffer_get`), not a deferred one. So this lead does not -rescue that family; it collapses back into it. - -## Honest note on cost - -This lead consumed a probe, a corrected probe, a trace run and several source reads, and its yield is a -closing argument. That is the same disposition as the other four families, and it is recorded rather than -quietly dropped — but the process lesson is that a comment about the general case is not evidence for a -specific binding, and the cheap measurement (one trace run) should have preceded the static write-up. - ---- - -# Corroborated: the peer agent closed this lead independently, with a stronger probe - -While this file was being written, the other agent on the branch reached the same conclusion from a -different and **better** probe (commit `13deecb`). Recording the agreement and the methodological -difference, because it is the second time this turn that a peer's method outperformed mine. - -## Their probe used the arms' own workload shape - - probe 1 (mine, invalid) /v1/chat/completions, reused prefix -> prompt eval 9 then 1 then 1 tokens, - graphs reused 7/15/23 - probe 2 (theirs, valid) /completion with cache_prompt=false and n_predict=96 -> prompt eval 25 tokens, - 120 tokens delivered, graphs reused 95 - -and `graphs reused 95` matches a real arm's first request exactly — so their probe exercised the **identical -workload shape at the identical dispatch count**, where mine exercised a shape the faulting arms never use. - -## Their queue-tracing argument is also tighter than mine - -They established that zero records means the queue was never populated, rather than that the fingerprint is -broken, by tracing the three sites: the populate site is `add_host_writeback` at -`command-program-executor.cpp:1106`, the queue's **only** push site is at `:176`, and the print site at -`:136` is invoked at `:127` **inside the loop over `pending_host_writebacks_` at `:125`** — so the -fingerprint prints *iff* the queue is non-empty. That is a stronger closure than my binding-level argument -(`requires_materialization()` → `host_data == nullptr`), and the two are consistent: my chain says the logits -binding never presents host data to the materializer; theirs says the queue is therefore never populated. - -## Net - -Two independent methods, two different probe shapes, one conclusion: **the deferred host writeback is not on -GLM's decode path.** The family is closed. - -Worth stating plainly for the record: my probe failed twice (once for a reused graph, once for a tally that -grepped its own header) before yielding a sound negative, whereas the peer's single well-shaped probe settled -it. The lesson generalizes past this lead — **probe the workload's own shape**, and a probe that does not -reproduce the arms' dispatch count has not asked the question. diff --git a/dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md b/dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md deleted file mode 100644 index 68969ca2f..000000000 --- a/dev/315/315-campaign/evidence/315-device-buffer-probe-proposal.md +++ /dev/null @@ -1,278 +0,0 @@ - - - -# Proposed + pre-registered: the DEVICE-buffer producer probe — the instrument that can reach the onset tensor - -Written because the host-only producer probe (`315prod`, queued) provably cannot answer the question the -positional finding raises. No build and no run are attempted here; this is the design and its readout, fixed -in advance so the run is a test rather than a search. - -## Why the existing probe cannot reach the onset - -`producer_probe_target` requires `ggml_backend_buffer_is_host(t->buffer)` **and** a size in 256 KiB … 8 MiB. -Against the recorded inventory that admits exactly one tensor — the 619,520-byte **logits** region. The -`hrx-producer` dump prints `host=0` for every layer node (`q-0`, `norm-0`, `q_nope-0`, …), so the layer -residuals are device-side and can **never** be armed by it. - -But the positional result (`315-one-row-at-the-front-of-the-buffer.md`) puts the onset at exactly such a -device tensor: **one `n_embd`-wide row, at row 0 or 1, in a layer residual**. So the decisive question — -*did the device write that row?* — is one the host-only instrument is structurally unable to ask. - -## The instrument is feasible with primitives already in the fork - - ggml-hrx.cpp:442 static void backend_set_tensor_async(backend, tensor, data, offset, size) - ggml-hrx.cpp:466 static void backend_get_tensor_async(backend, tensor, data, offset, size) - ggml-hrx.cpp:542 backend_set_tensor_async, <- registered in the backend iface - -Both take an arbitrary HRX tensor (device **or** host) through `ggml_backend_hrx_tensor_binding`, and each -falls back to a synchronous copy when the host side is not a registered host buffer. So the symmetric probe is -the existing pattern with the `is_host` clause dropped: - -- **before** the graph: `ggml_backend_tensor_set(node, pattern, 0, ggml_nbytes(node))` — fill a **device** - tensor; -- **after** a drained stream: `hrx_stream_synchronize` then `ggml_backend_tensor_get(node, buf, 0, nbytes)`; -- classify, exactly as the host probe does. - -## Design decision: a FINITE sentinel, not a NaN payload - -The host probe fills `0x7FC0DEAD`, which is a NaN. For a **layer residual that is also an op's input** that is -a serious confound: if the op fails to overwrite it, the injected NaNs propagate through the rest of the layer -and produce *the very symptom under study* — the run would be diagnosing its own sentinel, and every -downstream observation in that arm would be unusable. - -Use a **finite** pattern instead (`0xDEADBEEFu`), so: - -- `sentinel == n` → the op did not write the tensor at all; -- `sentinel == 0` → the op wrote every word, and any NaN present is **genuine**; -- `0 < sentinel < n` → the op wrote part of it — and this is the interesting case, because a partially - written row-oriented buffer is exactly what "one row corrupted at the front" would look like from the - writing side. - -The collision risk (a legitimate word coincidentally equal to `0xDEADBEEF`) is stated and is ~0 for a single -fixed 32-bit pattern over 2⁵² words, and would in any case *reduce* the measured `sentinel` count. - -## The run - - BIN= - EXTRA_ENV="GGML_HRX_PRODUCER_PROBE=2 GGML_HRX_PRODUCER_FINITE=1 \ - GGML_HRX_TENSOR_FP=l_out-0,…,l_out-47 GGML_HRX_TENSOR_FP_NANONLY=1" - ~/wt/315-disc.sh 8 315dev - -**This needs a build**, which is why it is a proposal and not a queued run: the host-only clause and the NaN -payload are both compiled in, so `~/wt/315probe-build` cannot do this. The change is small (drop one clause in -`producer_probe_target`, add a finite-payload branch in `producer_probe_fill`) and is **diagnostic-only**, so it -cannot weaken any guard. - -**Target selection** must be widened deliberately, and the selector is the hard part: the band should be -`n_embd × T × 4` for the arm's T, and restricted to residual-family nodes so the probe does not arm every -activation in the graph. The pre-registered selector is **any device tensor whose byte size is within ±1 % of -`2048 × T × 4`**, which for a prefill chunk targets the residual-stream tensors and nothing else. If that arms -nothing, the run is INVALID and the selector — not the conclusion — is revised. - -## Readout, fixed in advance - -| observation | verdict | -|---|---| -| `sentinel == 0` on the onset tensor in a faulting arm, with `nan > 0` | the device **did** write the whole tensor, and the NaN is genuine ⇒ the corruption is *not* an unwritten-buffer/visibility defect; the writer is an op | -| `0 < sentinel < n`, and the unwritten words are **contiguous** | the op wrote a prefix and left the rest — a row-oriented write that did not cover the region, which is a mechanism | -| `sentinel == n` on the onset tensor | the tensor was never written at all: the consumer reads pre-fill content ⇒ the visibility/staleness family, at the point of production | -| `sentinel == n` on a tensor that is *not* the reported onset | the probe armed the wrong tensors; INVALID, fix the selector | -| no `armed` line | INVALID, widen the selector | - -## Confounds, declared before the run - -- **Filling a live tensor perturbs the graph.** Even a finite pattern changes downstream values for that - tensor; the classification at the probed tensor is still valid (it asks only whether the op overwrote the - words), but the arm's *other* observations are not comparable to a natural arm. -- **The probe adds a host⇄device copy per hooked graph**, which the record already establishes is itself an - inducer. So this run characterises *the probe's regime*, not the ambient one. -- **`n_embd × T × 4` assumes T from the arm, and T varies within an arm across chunks.** The selector must - therefore be size-band based, not a single number. -- No rate: a per-graph classifier, no denominator. - -## What this would decide that nothing else on the record can - -It separates the two families the goal names and the engine cannot distinguish — *"the device wrote these -bytes"* versus *"a restore left a pre-eviction copy" / "nothing wrote them"* — **at the tensor the positional -evidence points at**, rather than at the logits region the host-only probe is confined to. - ---- - -## Correction to this proposal's own size estimate: the change is NOT "drop one clause" - -Written after reading `producer_probe_fill` and `producer_probe_scan` in full, and it invalidates the estimate -above. Both functions touch the tensor **through a raw pointer**: - - producer_probe_fill: uint32_t * words = static_cast(node->data); - for (size_t w = 0; w < n; ++w) words[w] = kProbeSentinel; - producer_probe_scan: const float * fp = static_cast(node->data); - ... std::memcpy(&bits, &fp[w], sizeof(bits)); ... - -`node->data` is a **host** address for a host buffer and is **not CPU-addressable** for a device (HRX0) -buffer. So on a device tensor the existing fill would dereference a device pointer from the CPU, and the -existing scan would read unmapped memory. Dropping the `is_host` clause alone would therefore not produce a -device probe — it would produce a crash or garbage, which is worse than an INVALID result because it can look -like a hit. - -The real change is: - -1. `producer_probe_target` — accept non-host tensors when `GGML_HRX_PRODUCER_DEVICE` is set, and use an exact - row-multiple selector `bytes % 8192u == 0 && bytes >= 8192u` (an integer number of `n_embd`-wide rows, - which needs no knowledge of `T` — cleaner than the ±1 % band proposed above and equivalent to it for - integer `T`); -2. `producer_probe_fill` — for a device target, fill a **host staging buffer** once with the finite pattern and - upload it with `ggml_backend_tensor_set(node, staging, 0, bytes)`; keep the raw-pointer path only for host - targets; -3. `producer_probe_scan` — for a device target, download via `ggml_backend_tensor_get(node, staging, 0, bytes)` - after the drained stream and classify the staging buffer; keep the raw path for host targets; -4. one `kProbeFinite = 0xDEADBEEFu` constant and a branch choosing it over `kProbeSentinel` in device mode. - -That is ~40 lines through three functions, not a clause. The estimate above is withdrawn; the design, -selector, five verdict branches, and confounds all stand. - -**This also sharpens why the proposal is a proposal.** The host-only probe's raw-pointer access is not an -oversight to be patched lightly — it is the reason the instrument is cheap and safe, and a device variant -trades that for a staging copy per hooked graph, i.e. exactly the kind of perturbation the record already -flags as an inducer. - -### Patch anchors, verified against the file (not from memory) - - runtime/graph-executor.cpp:46-54 bool producer_probe_target(...) with the is_host clause at :48 - runtime/graph-executor.cpp:80-96 size_t producer_probe_fill(...) raw write at :86-87 - runtime/graph-executor.cpp:99-132 void producer_probe_scan(...) raw read at :113-114 - -Each anchor is an exact string, so a patch script in the style of `tools/patch-*.py` (assert each anchor -occurs exactly once, so a silent no-op is impossible) can apply it when a build is next possible. The build is -deliberately **not** attempted here: compiling raises `load1` and the `foreign` count, which are two of the -three conditions the queued `315prod` run is waiting on, and would contend with the foreign decompilation the -box is already carrying. - -### The patch is now written and its anchors verified — but it is NOT compile-tested - -`dev/315/315-campaign/tools/patch-device-producer-probe.py` applies the change in the record's own patch style -(every anchor asserted to occur exactly once, a pre-check over all anchors before any write, an idempotency -marker, and a brace-balance check). Verified against a **real copy** of the source, not from memory: - - patched ggml/src/ggml-hrx/runtime/graph-executor.cpp - device-enable branch: 1 occurrence(s) - second run: already patched - braces: 52 / 52 - -Two defects were caught by that testing rather than shipped: - -1. **Use-before-declaration in the scan replacement.** `const float * fp = ...` is declared *before* - `const size_t n = ...` in the original, so a staging buffer sized from `n` would not have compiled. The - replacement now sizes from a locally computed `dfloats` and does not reference `n`. -2. **A stranded comment.** The sentinel anchor matched only the declaration, so the original line's trailing - comment was left attached to the end of the new `kProbeFinite` line — legal C++ but a mangled double - comment. The anchor now includes the trailing comment. - -**What is still NOT established, and must not be claimed:** that it **compiles**. The script's own output says -so on every run. Confirming that needs a build, which is deliberately deferred because compiling raises -`load1` and the `foreign` count — two of the three conditions the queued `315prod` run is waiting on — and -would contend with the foreign decompilation already on the box. So the instrument is *ready to build*, not -*working*. - ---- - -## BUILT and RUNNING (2026-10-14 05:52) — the compile caveat is now resolved - -The patch was applied to the real tree and **the build succeeded**: - - patched ggml/src/ggml-hrx/runtime/graph-executor.cpp (anchor-verified, braces 52/52) - [ 26%] Building CXX object .../runtime/graph-executor.cpp.o - [ 26%] Linking CXX shared library ../../../bin/libggml-hrx.so - [100%] Built target ggml-hrx - === EXIT=0 === - -so the earlier "**compile NOT claimed**" caveat is now discharged for *compilation*. Verified in the artefact -rather than inferred from the build's exit code: `bin/libggml-hrx.so.0.18.0` (05:52) contains **both** -`GGML_HRX_PRODUCER_PROBE` and `GGML_HRX_PRODUCER_DEVICE`, and `graph-executor.cpp.o` carries the same -timestamp. Source backed up before patching (`/tmp/graph-executor.cpp.orig`, sha256 -`cba0419f55e748ed…`). - -**Run launched**: `315dev` — `GGML_HRX_PRODUCER_PROBE=1 GGML_HRX_PRODUCER_DEVICE=1` plus the 15-name -`l_out` fault-yield enabler, 8 rounds, behind the same gate-aware wrapper (`315dev-wait.sh`) that holds no -lock and no GTT while waiting. **Still NOT claimed: that it arms anything at run time.** The runtime -`armed`/`anomaly` lines are the evidence for that, and they do not exist yet. - -## The sibling fix another agent landed, which is the closest thing on the record to this family - -While checking the build, a pin-bump commit on the fork (`fb24c1f`, Oct 5) turned out to describe a -**root-caused NaN fix in exactly the family this proposal is about**: - - Bump HRX: llama.cpp 522dab4789ce (long-prompt fix), hrx-system to our fork 98d05d94a9f9 - third_party/llama.cpp e57beb9721af -> 522dab4789ce (fork #89 runtime overhead, - #90: graph inputs never written back to host - every multi-ubatch prompt was wrong since #82 - - and the non-replay upload race). - third_party/hrx-system -> our fork ... plus the GFX11 wave64 lane-mask drain backport (#313 NaN) - -and the fork's merge itself reads: - - 522dab478 Merge pull request #90 from 1bit-MONSTER/1bit/kv-nan-rootcause - ggml-hrx: graph inputs are never written back to host; non-replay programs wait for their input upload - -**That is "the consumer reads memory the device never wrote", already root-caused and fixed once.** Two -consequences for #315: - -- **It is evidence the family is real and reachable**, independent of anything this session measured: a KV-NaN - root cause was found in the host write-back path and fixed at `522dab4789ce`. -- **The engine pin's tree already contains a `PendingHostWriteback` path** whose final step is - `std::memcpy(writeback.host_destination, writeback.mapped_source, writeback.size)` — a **device mapping** - copied into a **host destination** (`runtime/command-program-executor.cpp`). That is the exact line where - "the device wrote these bytes" and "the mapping still holds something else" are decided, and it is in the pin - this #315 work runs on. - -**What I have NOT established, and it decides whether #90 is a fix or already-shipped:** `522dab4789ce` versus -the engine pin `e44c9d01a`. The local clone is **shallow** — `522dab4789ce` has no parents in it -(`git rev-list --parents -n1` returns the commit alone) and `e44c9d01a` is absent — so `git merge-base ---is-ancestor` cannot run. If `522dab4789ce` is an **ancestor** of the engine pin, #90 is already shipped and -#315's remaining faults are a *different* path; if it is a **descendant**, #90 is a **forward-only fix -candidate** for #315. Settling it needs either a full fetch of the fork or the ancestry via the engine repo's -submodule, and it is worth doing before any other run. - -### SETTLED: PR #90 is already shipped, and NOTHING relevant is ahead of the pin - -Both questions are now answered from the fork itself, with a fetch for the pin commit: - - git fetch origin e44c9d01a4d5110cecaf45472a717f46cd1abbf9 # exit 0 - git merge-base --is-ancestor 522dab4789ce e44c9d01a - -> ANCESTOR: PR#90 already in the engine pin - -**So the KV-NaN root-cause fix ("graph inputs are never written back to host; non-replay programs wait for -their input upload") is already in `e44c9d01a`. It is not a fix candidate, and it does not explain a remaining -fault — #315's residual faults are a different path.** What it does establish, independently of anything this -session measured, is that the family is real and reachable: a NaN was once root-caused in the host write-back -path and fixed. - -And the reverse question — is any fix available *ahead* of the pin, i.e. is a forward-only pin move possible? - -| branch | commits ahead of `e44c9d01a` | -|---|---| -| `origin/1bit/hrx-vulkan-patched` | **none** | -| `origin/main` | **none** | -| `origin/HEAD` | one, `fee39dd92 opencl: add A8 Q6_K non-MoE dp4a binary kernel (#29057)` — unrelated | - -The pin is dated **2026-10-07** and the fork's newest branch is **2026-09-23**, so the engine pin is at the -tip of the 1-bit line. **Consequence for criterion 2: there is no forward-only pin move that fixes #315, so -any fix has to be *authored* rather than bumped to.** That also means the objective's forward-only clause is -satisfiable in principle (a new commit on top of the pin is trivially forward-only) but not by moving to -something that already exists. diff --git a/dev/315/315-campaign/evidence/315-device-hunt-boundary.md b/dev/315/315-campaign/evidence/315-device-hunt-boundary.md deleted file mode 100644 index ab27a1630..000000000 --- a/dev/315/315-campaign/evidence/315-device-hunt-boundary.md +++ /dev/null @@ -1,81 +0,0 @@ - - - -# Where the device-side hunt actually stops: the runtime does not infer memory hazards - -Written after the host-side family was closed (`315-write-side-unaddressed.md`), to follow the one -ordering question §32/§33 left standing — "whether the host-side recorder actually records every -producer/consumer pair it should". Read at source, the runtime answers for itself, and the answer -narrows the remaining surface rather than opening it. - -## The answer, in the code - -`runtime/src/iree/hal/drivers/amdgpu/aql_block_processor.c`, in -`iree_hal_amdgpu_aql_block_processor_packet_control`: - - // The replay hot path is intentionally additive: command recording has - // already encoded execution-barrier scopes in |packet_flags|, and submission - // policy only overlays wait, queue-kernarg, and terminal-signal visibility. - // Do not infer memory hazards here by walking command operands. - -And the barrier-bit decision itself (`..._packet_has_barrier`) consumes only: - -- `PACKET_FLAG_EXECUTION_BARRIER` and `PACKET_FLAG_FINAL` — **set by command recording**; -- `packet_index == 0` with a waiting submission, or with a non-NONE inline acquire scope — the - submission-level overlay (`max_scope`, applied only at `logical_packet_index == 0`). - -Nothing walks buffer operands. Nothing derives a producer/consumer pair. The runtime's stated policy is -that **hazards are already encoded by the compiler**, and the submission layer adds only wait, -queue-kernarg and terminal-signal visibility. - -## Why this matters - -It answers §33's standing question in the direction that **removes the runtime as the site**: the -"recorder may miss a producer/consumer pair" hypothesis requires the runtime to *derive* pairs, and it -explicitly does not. So the ordering between a global-memory producer and its consumer in GLM's stream is -whatever the compiler emitted, and a missing dependency would be **an emission gap, not a runtime gap**. - -This is consistent with the two other device-side results already on the record: - -- §33's static audit found no GLM-relevant kernel with an intra-dispatch global handoff lacking a - `barrier` — the kernels that hand work across waves through global memory are exactly the ones - that carry the fence; -- §32's block-boundary asymmetry was refuted by its own step 1 (`packet_index_base` is a profiling - prefix, 0 for every block, so the per-block conditions hold per block). - -## What is left, stated as the boundary rather than a story - -The remaining candidate surface is the **compiler's emission of execution-barrier scopes** for GLM's -stream (loom / IREE codegen), not the runtime and not the host. That surface is large and is *not* in this -goal's declared scope: the boundaries name the engine pin and the ggml-hrx fork side, not the loom -codegen. Pursuing it would be an internals expedition with no bounded discriminating test available at -the level this issue has been worked at. - -So the honest position: the cause is **not located**, and the evidence now favours an -emission-level ordering gap that this goal's scope does not reach. That is a boundary to record, not a -mechanism to claim — and it is the reason criterion 2 should not be claimed as achievable here. - -## What would change this - -A bounded test would need to compare GLM's emitted barrier scopes against a model that does not fault -(e.g. a dense model of similar depth) and find a producer/consumer pair GLM omits. That is a compiler-side -experiment requiring codegen instrumentation, and it is outside the declared scope -(`~/wt` harness, engine pin, ggml-hrx fork). diff --git a/dev/315/315-campaign/evidence/315-differential-artifact.md b/dev/315/315-campaign/evidence/315-differential-artifact.md deleted file mode 100644 index 268f9baaa..000000000 --- a/dev/315/315-campaign/evidence/315-differential-artifact.md +++ /dev/null @@ -1,74 +0,0 @@ - - - -# engine#315 — the differential artefact: what changed in `ggml/src/ggml-hrx` between the clean bound and the pin - -```sh -ssh strixhalo 'cd ~/wt/llama-315copy && git rev-parse --short HEAD; git rev-parse --short f2099e9b7' -# e44c9d01a -# f2099e9b7 - -ssh strixhalo 'cd ~/wt/llama-315copy && git diff --stat f2099e9b7..HEAD -- ggml/src/ggml-hrx/' -# .../flash_attention_decode_split_f32_f16_wmma.loom | 104 ++++++++++++++++++++- -# 1 file changed, 102 insertions(+), 2 deletions(-) - -ssh strixhalo 'cd ~/wt/llama-315copy && git log --oneline f2099e9b7..HEAD -- \ - ggml/src/ggml-hrx/dispatch_registration/common/dispatch-gated-mul-mat-id.cpp \ - ggml/src/ggml-hrx/dispatch_registration/common/dispatch-mul-mat-id-common.h \ - ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/ops/small_rows_f32.loom \ - ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/ops/mul_mat_id_swiglu_f32_f32_wmma.loom' -# (no output) -``` - -**One file in the whole backend differs, and it is the decode-split flash-attention kernel**: +102/-2 in -`flash_attention_decode_split_f32_f16_wmma.loom`, the same change already rate-tested against the always-stage -build (5.4% vs 4.0%, Fisher p = 1.000) and closed by mechanism-doc sections 26/34/36. - -## Why this file exists - -It is the constraint that decides which of #315's two live candidate families can be a *change* and which can -only be a *pre-existing amplifier*: - -* **The routed-FFN path is byte-identical between the clean bound and the faulting pin.** The dispatch - registration (`dispatch-gated-mul-mat-id.cpp`, `dispatch-mul-mat-id-common.h`), the id-producing kernel - (`small_rows_f32.loom`), and the fused consumer (`mul_mat_id_swiglu_f32_f32_wmma.loom`) have no commits in the - window. So the routing-tensor site that `315row-p1c` measured cannot have been *introduced* by the pin move. - Either it is older than `f2099e9b7` (in which case the clean label was about rate, not about absence, and the - same site must be reachable on the clean bound too), or the pin move is what makes it fire. -* **The only thing the pin move did change in this backend is the decode-split attention kernel.** That keeps the - FA change in the causal frame even though its kernel-level checks passed - and it sharpens the one caveat - already on the record: section 38 established that GLM runs the **row-major MLA** layout, which invalidates the - "GLM geometry" label on the race sweeps that cleared this kernel (sections 26, 34: 40/40 masked-V cases, - 2250 + 2000 race reps). Those sweeps were run at shapes that are not GLM's MLA geometry, so "the pin is - race-free" is established for the shapes tested, not for the shape this model actually runs. - -## The artefact set this completes, against task-3's contract - -| task-3 item | state | -|---|---| -| ISA/IR diff of the changed dispatch/kernel | **this file**: for the routed-FFN path the diff is empty; for the FA path it is `task3-fa-kernel-ir.txt` (103 `+` / 6 `-`) with the named site `template.def<@ggml.flash_attention.decode_split.produce_partials.active>` | -| `rocprofv3` trace, faulting vs clean in-regime arm | **not done as rocprofv3**; the engine's own queue-event profiler is used instead and its output is recorded in `evidence/315-queue-profile.md` (612 dispatches, 519 barriers, 92 unbarriered, 1 block, for a 24-token greedy decode). It is a *clean* profile, not a faulting-vs-clean pair: the trace of a faulting arm is still owed | -| REA (or documented equivalent) on libhrx/loomc and the HIP code objects | **partly done**: the shipped `libhrx.so` is not stripped, and its symbol/flag inventory (`pm-salvage/issue-315/CANDIDATE-SITE-WAIT-BARRIER-DEFER.md`) is the documented equivalent; REA's own provider failed with an unretained log | - -The one item this file adds beyond the table is the **shape gap** in the FA clearance, which is the strongest -remaining question about the pin-move window. diff --git a/dev/315/315-campaign/evidence/315-dossier-spt-claim-corrected.md b/dev/315/315-campaign/evidence/315-dossier-spt-claim-corrected.md deleted file mode 100644 index ba2b72f6f..000000000 --- a/dev/315/315-campaign/evidence/315-dossier-spt-claim-corrected.md +++ /dev/null @@ -1,58 +0,0 @@ - - - -# The dossier's `_spt` claim is WRONG — corrected by the REA decompilation - -The peer dossier I preserved at `315-hrx-system-dossier-by-peer-agent.md` contains, in its §4.5 and its -architecture diagram, this claim: - - | 0x31d8a0 | 6 B | JMP qword ptr [0x0032a3e0] | GOT-indirect thunk | - … - So the plain symbol name resolves through a GOT slot that is patched to the per-thread-stream - variant. - -**That mechanism is wrong**, and it has been independently corrected by a much larger decompilation effort -now on this box (`~/amd-oss/hrx-re/`, 83 MB of decompiled output plus a 55-document private set). Its -`INDEX.md` records the correction as its own item 7, re-derived from source and re-verified later (item 20): - -- `0x31d8a0` is **vaddr `0x21d8a0`, inside `.plt`** — a PLT stub, not an HRX-authored thunk; -- **no `hipMalloc_spt` export exists at all** (39 `_spt` symbols are exported; that one is not); -- the runtime check: with `PER_THREAD_DEFAULT_STREAM`, `hipGetProcAddress("hipMalloc")` returns the - **same** address — so nothing is patched for it; -- the real mechanism is **symbol-name resolution** — `dlsym(symbol + "_spt")` — confirmed live in that - direction: `hipMemcpy`/`hipMemset`/`hipStreamSynchronize`/`hipEventRecord`/`hipLaunchKernel` all resolve - **differently**, while `hipMalloc` does not. - -Source-verified at `api.c:28329` in that effort, and re-verified rather than trusted. - -## Why this is recorded in this tree - -The dossier is preserved here as evidence and cited as a pointer, so leaving its wrong claim unannotated -would propagate a refuted mechanism into this issue's record. Nothing in this goal's own reasoning depends -on it — the #315 work never used the `_spt` mechanism — but the file is a citation, and a citation that is -known-wrong must say so. - -## Standing of the two sources - -The dossier is a peer agent's summary; the REA effort is a decompilation with a corrections ledger (14 -self-corrections recorded, including a retracted public bug report). Where they differ on this point the REA -work should be preferred: it states its method, its source line, and its live verification, and it records -having re-derived the correction a second time rather than trusting its own earlier note. diff --git a/dev/315/315-campaign/evidence/315-engine-ci-baseline.md b/dev/315/315-campaign/evidence/315-engine-ci-baseline.md deleted file mode 100644 index b1160d331..000000000 --- a/dev/315/315-campaign/evidence/315-engine-ci-baseline.md +++ /dev/null @@ -1,183 +0,0 @@ - - - -# engine CI baseline at pin `e44c9d01a` - -Run on `ryzen` in the worktree `~/wt315-prereg` (`1bit/315-preregistration`), 2026-10-09, at -`third_party/llama.cpp = e44c9d01a4d5110cecaf45472a717f46cd1abbf9`. Every step below is a step of -`.github/workflows/ci.yml` or a `ctest` test that `ci.yml`'s final `ctest` runs, executed directly -rather than through the workflow. - -| step | command | result | -|---|---|---| -| copyright | `python3 tools/copyright.py --check` | **exit 0** (it failed before `b777bdc`: three `evidence/*.txt` files had no comment style and the copied probe script had no notice) | -| pin gate | `python3 tools/check_pins.py 6d5e470` | **exit 0** - `third_party/llama.cpp: 26330cc20 -> e44c9d01a is ahead` (plus hrx-system, xdna-driver, laya) | -| pin gate, longer span | `python3 tools/check_pins.py beec5e6` | **exit 0** - `third_party/llama.cpp: cf4dfd801 -> e44c9d01a is ahead` | -| gguf -> Q4NX | `python3 tests/gguf_to_q4nx_test.py` | PASS | -| ternary -> Q4_0 | `python3 tests/ternary_to_q4_0_test.py` | `PASS: PTQ1_0 -> Q4_0 exact, other tensors and metadata kept` | -| MTP draft vocab | `python3 tests/mtp_draft_vocab_test.py` | `OK` | -| registry gaps (ctest `registry_gaps`) | `python3 tools/registry_build.py --check-gaps` | exit 0 - `none of the 15 reviewed significant classes is mapped without a reason` | -| registry gaps fire (ctest `registry_gaps_fire`) | `python3 tests/registry_gaps_test.py` | PASS (2 checks) | -| build (`onebit` + the 14 test targets, gcc-15) + `ctest` | not run | nothing to regress yet: there is no fix revision to compare against, and this is the CPU-only half of the workflow | - -## What the pin gate does and does not cover - -`tools/check_pins.py` compares **submodule gitlinks** between a base commit and `HEAD` and asks GitHub's -compare API whether the new commit is ahead of the old one. It is the machine enforcement of the -objective's "must not be a revert" clause, and it works here (rows 2-3). - -**Gap found and CLOSED (2026-10-13).** As originally recorded, `check_pins.py` covered only the -submodule gitlinks, not the hrx revision declared in `registry/architectures.json` -(`"sources": { "llama.cpp (hrx)": "e44c9d01a4d5..." }`). `tests/registry_gaps_test.py` reads that file -but checks class coverage, not the pin, and nothing tied the registry value to the gitlink. - -Demonstrated rather than asserted: setting the registry's declared revision to all-zeros passed -`check_pins.py`, `registry_build.py --check-gaps` **and** `tests/registry_gaps_test.py` — every gate -green on an inconsistent tree. - -`tools/check_pins.py` now reads the registry and requires it to agree with the gitlink at -`third_party/llama.cpp`, failing with an explicit message when they diverge. Verified both directions: -exit 0 on the consistent tree (`"llama.cpp (hrx)" agrees with the gitlink (e44c9d01a)`) and exit 1 on -the half-move, with the message naming both revisions and pointing at the file that must be moved too. -So a half-move can no longer pass CI, and task-7's "move both by hand" is now enforced by the gate. - -## The pin lineage, and why the fix cannot be "move the pin" - -`git log -- third_party/llama.cpp`: - -| commit | move | title | -|---|---|---| -| `4aa4ac2` | e57beb9721af | HRX bump: HIP plumbing hook, HRX fixes, Flash-Next | -| `11d6e2f` | 522dab4789ce | long-prompt fix; hrx-system moves to our fork (Loom NaN backport) | -| `c18b319` | f2099e9b7 | AMD's core with our ggml-hrx | -| `beec5e6` | f2099e9b7 -> cf4dfd801 | **Avoid subgroup votes in HRX decode-split flash attention** (#342) | -| `6d5e470` | cf4dfd801 -> 26330cc20 | **pin llama.cpp to the shared-memory FA tile visibility (engine#314/315)** (#347) | -| `805d30f` | 26330cc20 -> e44c9d01a | AMD's tested pair | - -`6d5e470` is already a #315-motivated pin move, aimed at exactly the flash-attention visibility that -sections 26-27 of `315-mechanism.md` now falsify as the locus. So "bump the pin to the next AMD pair" -is not a fix for this issue - it has been tried and the fault survived it (section 22 finding 2 records -that the coherence and ordering levers failed; section 26 kills the FA masked-row lever outright). -The eventual change has to be a new fork commit that is a **descendant of `e44c9d01a`** and carries -whatever the mechanism work ends up naming. - -## Closing the recorded gap: the registry pin and the gitlink agree today - -The baseline above recorded a gap - "nothing ties `registry/architectures.json`'s hrx pin to the gitlink" - -which reads as a possible divergence. Checked 2026-10-10: - -``` -$ git ls-tree HEAD third_party/llama.cpp -160000 commit e44c9d01a4d5110cecaf45472a717f46cd1abbf9 third_party/llama.cpp -$ grep -n 'llama.cpp (hrx)' registry/architectures.json -4: "llama.cpp (hrx)": "e44c9d01a4d5110cecaf45472a717f46cd1abbf9" -``` - -They are the same commit. So the gap is one of *enforcement* - no check compares the two, and `check_pins.py` -enforces only the gitlink - not of current divergence. A forward-only fix therefore edits **two** files by hand -(the gitlink and `registry/architectures.json` line 4) and both must stay equal; a CI check that compares them -would be a reasonable addition but is not part of this change. `KNOWN-ISSUES.md` lines 81-84 hold the other -mechanical constraint (why `f2099e9b7` cannot be adopted: it is a strict ancestor of the shipped `26330cc20`, -so pinning it is a revert). - ---- - -# Criterion-4 control re-run: queued (2026-10-13 ~16:07) - -The objective's fourth criterion needs the **dense-model control and the MUL_MAT_ID op-test unchanged -relative to baseline**. The baseline for both is the file above (2026-10-09, pin `e44c9d01a`), and the two -things it records that a re-run must match are: - -- the 5 HRX buffer tests, all `[PASS]`, and the 9 runnable tests overall; -- the **MUL_MAT_ID `nmse` values**, e.g. `mxfp4 k=2880 tokens=1 rows=1 … nmse=1.56e-05 OK` — a numeric - fingerprint, not just a pass/fail, so a drift in the reduce or the id resolution would show. - -## Status - -A re-run is **queued** (`~/wt/315crit4.sh`, tag `315crit4`), waiting for the peer agent's replication to -release the GPU. It re-runs the buffer tests and the op-test from `~/wt/v315-build` (present, with 8 -`test-hrx*` binaries) under `systemd-run --user --scope -p MemoryMax=4G`, exactly as the baseline did, and -extracts the `MUL_MAT_ID` lines into `~/wt/315crit4/mul_mat_id.txt` for a direct diff against the table -above. - -## Why it is queued rather than run now - -The tests need HRX0, and the box is currently held by the peer's alignment replication (with the box lock -taken). Running them concurrently would perturb that measurement and risk its GTT headroom — the same -reasoning that deferred my earlier probes. Nothing about criterion 4 is blocked in principle; it is -sequenced behind a live run. - -## What a result would and would not establish - -- **Would**: whether the shipped pin's own regression controls are unchanged *today*, which is criterion 4's - claim, and it would do so with the baseline's own numbers rather than a pass/fail. -- **Would not**: anything about the #315 defect. No fix has landed, so there is nothing for these controls - to regress *against* — which is why criterion 4 cannot be *met* until criterion 2 produces something to - compare. This run establishes that the controls are stable on the current pin, i.e. that the baseline the - future fix must match is still the baseline. - ---- - -# Update: the control re-run now requires a sustained clear window - -The first `315crit4.sh` waited on the **first** 30-second poll showing no other driver, which is unsafe here: -another agent launches campaigns back-to-back, so a brief gap between two of its runs would let this control -start and collide with its next launch — corrupting both measurements. - -Replaced by `315crit4b.sh`, which requires a **sustained 120-second clear window** (four consecutive 30-second -polls with no other driver) before starting, and still takes the box lock. Launched 16:46:23 and waiting. - -Nothing about the criterion-4 content changed — same baseline, same commands, same `MUL_MAT_ID` extraction — -only the gate on when it is safe to begin. - ---- - -# Update: the control's first attempt matched no tests — my binary names were wrong - -The first corrected run (17:05) produced a header and then "done" with **no test output at all**. Diagnosed -rather than rerun blindly: my script looked for - - test-hrx-backend-buffer test-hrx-host-buffer test-hrx-host-transfer - test-hrx-host-staging test-hrx-host-weight-cache - -and **none of those binaries exist**. The real contents of `~/wt/v315-build/bin` are: - - test-hrx-attention-sink test-hrx-buffer test-hrx-decode-stride test-hrx-fa-masked-v - test-hrx-hadamard test-hrx-moe-split test-hrx-mul-mat-id-k32 test-hrx-mxfp4 - -So the baseline's **"5 HRX buffer test cases"** (`backend_buffer`, `host_buffer`, `host_transfer`, -`host_staging`, `host_weight_cache`) are **cases inside `test-hrx-buffer`**, not separate binaries — I had -misread the baseline's output as a list of executables. - -Note also that the objective's named control, the **MUL_MAT_ID op-test**, is present as -`test-hrx-mul-mat-id-k32`, and `test-hrx-mxfp4` carries the `MUL_MAT_ID mxfp4 … nmse=` lines the baseline -table records. - -Corrected as `315crit4c.sh`, which runs the real binaries and greps for the `MUL_MAT_ID`/`mxfp4` lines to -diff against the baseline table. - -This is the fourth harness/parse defect of the session (a line-based counter on newline-free NDJSON; a -`pgrep` pattern matching its own caller; `md5sum` over streaming responses; now guessed binary names). Each -was caught before its output was cited, but the pattern is consistent enough to state plainly: **read the -baseline and list the directory, rather than reconstructing either from memory.** diff --git a/dev/315/315-campaign/evidence/315-evict-trace-instrument.md b/dev/315/315-campaign/evidence/315-evict-trace-instrument.md deleted file mode 100644 index 4e6de3d87..000000000 --- a/dev/315/315-campaign/evidence/315-evict-trace-instrument.md +++ /dev/null @@ -1,294 +0,0 @@ - - -# eviction/preemption trace: instrument verification (task-8) - -Verified before the campaign was trusted, because three different failure modes here are -**silent** — each would have produced a plausible-looking empty result instead of an error. - -## 1. The watcher needs root, and fails silently without it - -`kfd_smi_watch` unprivileged exits 0 and writes a **header-only** file. That reads as "no -evictions occurred" rather than "no privilege to see them". - -| invocation | outcome | -|---|---| -| unprivileged, 10 s idle | exit 0, 1 line (the `#recv_boot_ns...` header) | -| unprivileged, 20 s under GPU load | exit 0, 1 line | -| **`sudo`, 30 s under one HRX load** | **75 Q_EVICT / 79 Q_RESTORE** | - -The wrapper (`315-evict-trace.sh`) aborts if `sudo -n true` fails, so a header-only file can -no longer be mistaken for a quiet box. - -## 2. It writes TSV, not NDJSON - -The first version of the wrapper wrote to `kfd_smi.ndjson` and grepped for JSON quotes -(`"Q_EVICT"`, `"trigger":"..."`). Both are wrong: the stream is tab-separated with a -`#recv_boot_ns gpuid event kernel_ns pid fields...` header. Fixed. - -## 3. The HSA path is not under `lib/` - -`IREE_HAL_AMDGPU_LIBHSA_PATH` is -`/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1` -(taken from `315-paired.sh`). Guessing a path under `lib/` does not fail loudly — the HRX -device simply never registers and the server exits with `invalid device: HRX0`. - -## 4. The correlator parses our format - -`correlate.py` was smoke-tested against a real SMI capture plus a synthetic server log -before the campaign needed it. It auto-detects the target pid from the SMI stream (here -`789665`) and emits its per-request table. So the faulting arm's `server.log` can be -correlated without a code change. - -## 5. The driver's unit convention (cost a launch) - -`315-campaign.sh` takes `FIX`/`CTRL` as **build dirs** and appends `/bin`. Passing `.../bin` -gives `bin/bin` and aborts immediately (`ABORT: no .../315fix-build/bin/bin/llama-server`), -tearing down the watcher with only its header recorded. `strip_bin()` normalises both forms. - -## Live confirmation - -Run `315evict2`, once the composition reached the gate: - -``` -GTT 96.9 GiB (gate: >= 95) -HRX neighbours up: 5 -Q_EVICT 180 Q_RESTORE 246 -``` - -So the regime is reached and the eviction stream is populated. Whether this becomes -**evidence** still depends on an in-regime arm faulting while the watcher is attached — -a clean trace is recorded as UNINFORMATIVE with its clean-arm count, never as evidence. - -## 6. Two bugs that cost the first real launch (both mine, both silent-ish) - -### 6a. `MINCLEAN=0` stops immediately - -`315-paired.sh:147` stops when the clean count **reaches** the target: - -```sh -[ "$cf" -ge "${MINCLEAN:-60}" ] && { say "STOP: reached $cf clean fix-cell arms"; break; } -``` - -It is a goalpost, not a floor. Passing `MINCLEAN=0` — intended as "no minimum" — means -*stop at zero clean arms*, and `0 >= 0` is true, so the run ended after **pair 1**: 2 arms -instead of 12. The wrapper now computes `MINCLEAN=$(( PAIRS * 2 + 1 ))` so nothing is -reachable and the `PAIRS` cap is what ends the run. The console records it exactly: - -``` -after pair 1: fix-cell clean=0 fault-or-invalid=0 (target >=0 clean, cap 6 pairs) -STOP: reached 0 clean fix-cell arms -``` - -### 6b. `grep -c ... || echo 0` yields a two-line value - -`grep -c` prints `0` **and** exits 1 when there are no matches, so `|| echo 0` appends a -second `0`. The value becomes `"0\n0"` and the arithmetic test fails: - -``` -/home/bcloud/wt/315-evict-trace.sh: line 98: [: 0 -0: integer expected -``` - -The wrapper now takes the count and normalises with `${VAR:-0}` instead. Tested: zero -matches gives `IN=[0] FA=[0]` and the comparison succeeds. - -**Why these matter beyond this script:** both failures left the run *looking finished*. 6a -produced a complete, well-formed verdict summary for a 2-arm run — it would be easy to read -"cell c: in-regime=1 clean=1 fault=0" as a result rather than as a truncated run. The -`PAIRS`/`MINCLEAN` line in the console is the tell. - -### 6c. Box contention - -A campaign must not run while a peer holds the box. Observed on the relaunch attempt: a -peer's `~/wt/loom-ub389` 27B HRX job under `guard.sh` (1800 s timeout), GTT 46.7 GiB, -`npu.lock` touched minutes earlier. The composition would have evicted that job's memory. -Wait for the box rather than disturb a peer — the goal's own constraint. - -### 6d. The watcher started before the box was ours - -Third launch (run 3, tag `315evict3`) started the KFD watcher at the **top** of the script, -~70 lines before the box-lock acquisition and the GPU-quiet check. So the watcher ran its -3-hour TTL while a peer still held HRX0, recording events from a window that was not the -campaign's. Its output would have looked identical to a correct run — the tell is only the -ordering of the lines in the run log. - -Run 4 fixes the order, and the log now shows it: - -``` -box lock held 2026-10-12T13:13:03-03:00 <- lock first -GPU quiet: ... <- then the box is actually ours -watcher pid ... <- only now start the watcher -``` - -### 6e. flock alone does not protect against a peer - -The box lock only excludes processes that **follow the convention**. Observed 2026-10-12: a -peer batch (`~/wt/loom-ub389`) runs `-dev HRX0` jobs and its only shell script is -`build.sh` with no lock usage. So `flock` reported the lock free while the peer held HRX0, -and composing neighbours into the 95–110 GiB band would have evicted that peer's memory. - -The wrapper therefore takes the lock **and** polls for actual GPU quiescence: no foreign -`-dev HRX0` process and no `loom-ub389`/`repro336` job, with `QUIET_WAIT` (default 1800 s) -as the bound. On timeout it aborts and starts nothing, rather than racing. - -### 6f. Killing a run needs the wrapper killed directly - -`pkill -f "315-evict-trace.sh ..."` did not stop run 3 (it returned 255 and the process -survived), because the wrapper was parked in the quiet-poll's `sleep` with a trap installed. -The watcher stopped but the wrapper did not. `kill -KILL ` was needed. Worth knowing: -a half-stopped wrapper can leave a watcher running or a lock held. - -### 6g. There are TWO lock conventions on this box, and the campaign was holding the wrong one - -The most consequential finding of this task's harness work. Measured 2026-10-12 13:18: - -| lock file | holders at the time | who | -|---|---|---| -| `~/npu.lock` | **12 pids** | the box-wide convention — a peer batch (`flock ~/npu.lock ./batch2.sh`) **plus two unrelated NPU-goal workers** (`npu-forge2-kvsplit` bisect, `probe_model.sh`) | -| `~/.cache/lax-decode/box.lock` | 2 pids | **ours only** — the narrow convention the older 315 harnesses use | - -So `box.lock` was free while `npu.lock` was held by three workers, and my wrapper -"successfully" took `box.lock` — logging `box lock held` — while a peer ran `-dev HRX0`. -**The lock line in the log looked like protection and provided none.** The only thing -actually protecting the box was the separate GPU-quiet poll added in §6e. - -The fix: take **both**, `box.lock` first (harmless, cheap, matches the 315 harnesses) then -`~/npu.lock`, which is the one that serialises real GPU use. The log now distinguishes them: - -``` -box lock held (/home/bcloud/.cache/lax-decode/box.lock) -npu lock held (/home/bcloud/npu.lock) - the box-wide convention -``` - -Peer's usage, for the record (`~/wt/loom-ub389/runs/bench.sh:7`): - -```sh -flock ~/npu.lock systemd-run --user --scope ... llama-bench ... -dev HRX0 ... -``` - -**Generalisation worth keeping:** on this box, `npu.lock` is the real mutex. Any harness -that only takes `box.lock` is unprotected against peers and against other goals' workers, -and it will report a held lock while the box is busy. - -### 6h. Self-deadlock: the wrapper held the lock the driver re-opens (NOLOCK=1) - -The eighth and last harness defect, and the one that finally mattered. Run 6 (`315evict6`) -passed all four of this wrapper's gates, started composing, and then **deadlocked against -itself**: - -``` -bash 11865 /home/bcloud/wt/315-evict-trace.sh 8 315evict6 - └─ bash 12119 315-campaign.sh - └─ bash 12143 315-paired.sh 8 2 2048 ... - └─ flock 15489 -w 900 9 <- blocked: locks_lock_inode_wait - /proc/15489/fd/9 -> box.lock -``` - -The wrapper held `box.lock` on **fd 9** and `npu.lock` on **fd 8**. `315-paired.sh:47-52` then -**re-opens `box.lock` on its own fd 9** and flocks it: - -```sh -if [ -z "${NOLOCK:-}" ]; then - exec 9>"$HOME/.cache/lax-decode/box.lock" - if flock -w "${LOCK_WAIT:-900}" 9; then -``` - -`flock` is per-open-file-description. A **second** open of a file this process tree already -holds blocks — the lock is not re-entrant — and after `LOCK_WAIT` the driver gives up and -"proceeds CONCURRENTLY (UNLOCKED)", so the wait is not even protective. - -The driver documents the convention in its own header: **`NOLOCK=1` means the caller already -holds the lock on an inherited fd 9** — exactly our case. The wrapper now passes it. - -Verified on run 7 (`315evict7`): all four gates passed, driver in `do_wait` (not -`locks_lock_inode_wait`), and the composition reached the regime — GTT 27 → 80 → 98 → 100 GiB -with 4 HRX neighbours and 560+ evictions captured. - -**Pattern worth generalising:** when a wrapper takes a lock that its child also takes, the -child needs to be told the lock is already held. Silently double-locking looks like progress -up to the exact point where it stops. - -## §6i — the console line cannot certify zero faults on its own (fixed in the analysis tool) - -Criterion 3 asks for "zero faults across **both faces**". Both faces is the problem: the driver's -per-arm console line carries `nan= req= ret3= parse= gtt_start= utf8= alloc= hv52= hv56=` — and -Face B does not only arrive as a `parse=`/`utf8=` count. It also arrives as an **HTTP 500**, and -there is **no `http=` field in the console line at all**. - -Confirmed against a real arm's detector: - -``` -315evict7-p1c-.../detector.json: - requests 66 ok 66 utf8_failures 0 parse_failures 0 http_failures 0 transport_failures 0 - ^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ - these two are in the console line; these two are NOT -``` - -So a Face-B fault surfacing as HTTP 500 leaves **no trace in the console**, and a console-only count -would report that arm **clean**. That is a false negative on the single criterion the campaign -exists to certify — the worst possible failure mode here, because it silently manufactures the -desired result. - -**Fix, in the analysis tool only.** `tools/315-rate.py` now reads every arm's `detector.json` when -pointed at the arm root (`--det`, `--tag`) and treats the **detector as authoritative for faults**: - -* an arm the console calls clean but the detector calls failed is counted as a **fault** and flagged - `MISSED BY CONSOLE`, naming which counter caught it (`http=`, `transport=`, `parse=`, `utf8=`); -* an arm with **no usable `detector.json`** is reported **UNVERIFIED** — clean from the console - alone, *not* certified. It is never silently counted as clean; -* with `--det` disabled the tool prints an explicit WARNING that an HTTP-500 Face-B fault would be - counted CLEAN. The dangerous reading is stated, not implied. - -The driver stays frozen, as required mid-campaign: this reads artefacts the driver already writes. - -**Proven on a fixture where it changes the answer** (3 arms, console says OK for all three): - -| arm | detector | console | tool | -|---|---|---|---| -| `TT-p1c` | `http_failures=1` | `OK` | **FAULT** — "MISSED BY CONSOLE … http=1" | -| `TT-p2l` | all zero | `OK` | clean | -| `TT-p3l` | **absent** | `OK` | **UNVERIFIED** — not certified | - -Result: `1/2 = 50.00%`, acceptance FAILS. Without the reconciliation the same fixture reports -`0/3 = 0.00%` and the arm with a real HTTP-500 fault is counted **clean** — i.e. the tool would have -manufactured a passing campaign. - -Re-checked against run 7 (which really faulted): still `3/11 = 27.27%`, 11 detectors read, no missed -arms. So the change is a guard, not a revision of the known result. - -### §6i, live: the running campaign's clean arms are certified, not merely console-clean - -First read of the campaign of record through the reconciled tool (5 arms scored, the 6th in flight): - -``` - in-regime 5 clean 5 FAULT 0 INVALID 0 - detector.json read 6 arm(s) - (no MISSED BY CONSOLE line, no UNVERIFIED line) -``` - -The absence of those two lines is the point: each of the 5 clean arms has a `detector.json` that the -tool read and that reports no `http_failures` and no `transport_failures`. So they are clean **by the -detector**, not clean because the console could not see a failure. Had any of them failed via HTTP -500, the tool would have named it as a MISSED fault and the rate would have stopped being 0. - -Rates as expected for this point in the run: `0/5 = 0.00%`, exact one-sided 95% upper bound 45.07% — -correctly reported as **BELOW the 60-arm bar**, so these 5 arms certify nothing yet. That is the -tool refusing to overclaim, and it will keep refusing until the count reaches 60. diff --git a/dev/315/315-campaign/evidence/315-evict-trace-status.md b/dev/315/315-campaign/evidence/315-evict-trace-status.md deleted file mode 100644 index b38c34399..000000000 --- a/dev/315/315-campaign/evidence/315-evict-trace-status.md +++ /dev/null @@ -1,757 +0,0 @@ - -# task-8: eviction/preemption trace — live status - -Run `315evict2` on strixhalo. Purpose: attach the KFD SMI eviction/preemption stream to -in-regime arms so a **faulting** arm can be correlated against the driver's own events. -This is NOT an acceptance run (task-9 owns acceptance). - -## Setup (verified before the trace was trusted) - -- watcher as root, TTL 10800 s (unprivileged it writes a header-only file and exits 0 — - silently; see `315-evict-trace-instrument.md`) -- regime reached: GTT 98–104 GiB, gate is `>= 95` -- control cell `315instr3-build` (pin, `if (direct_host_binding)` intact) vs fix cell - `315fix-build` (pin + HOST_COHERENT). Both report `e44c9d01a`, so this is the diagnostic - pair — it tests the falsified coherence lever, not a candidate fix. -- arm→PID binding is exact (`tools/arm-pid.py`, via the port in the arm's own server.log) - -## Eviction stream (the substantive measurement) - -| quantity | value | -|---|---| -| Q_EVICT | **14,570** | -| — trigger=SVM | 8,594 | -| — trigger=USERPTR | 5,976 | -| Q_RESTORE | 18,769 | -| — `rescheduled=no` | **14,566** | -| — `rescheduled=yes` | 4,203 | -| VMFAULT | **0** | -| PAGE_FAULT | 0 | -| sustained rate | ~972/min (≈16/s) at GTT ≈ 104 GiB | - -## Arms so far - -``` -pair 1 arm c: OK nan=0 req=66 parse=0 gtt_start=98G <- IN regime, clean -pair 1 arm l: OOR nan=0 req=66 parse=0 gtt_start=93G <- OUT of regime, NOT COUNTED -``` - -Both arms served 66/66 with `utf8_fail=0 parse_fail=0 http_fail=0 transport_fail=0 -truncated=False` and no NaN. But the driver's own gate ("out-of-regime `gtt_start` < 95 GiB -(reported, not counted)", `315-paired.sh:12`) excludes arm `l`: it started at 93 GiB, below -the frozen 95 GiB gate. - -**So the tally to date is 1 in-regime arm, 0 faults — not 2 clean arms.** An `OOR` arm is -neither clean evidence nor a fault and must not be summed with the clean ones. The -composition does not guarantee every arm lands in band: `gtt_start` is sampled per arm -(`315-paired.sh:75`) and drifted by 5 GiB between the two arms of this pair. - -If the drift persists, a 12-arm run yields roughly half in-regime arms — which is far short -of task-9's `>=60` but is enough for task-8, whose requirement is a single faulting arm or -an honest null. - -## What this does and does not establish - -**Does:** the preemption/eviction window the #123/#140 lead describes is **open and -intense** during these arms — 14.5k queue evictions, of which 14.6k restores were *not* -rescheduled (i.e. the evicted queue's work was dropped and had to be redone). The -composition is genuinely exercising the driver's eviction path, not idling near it. - -**Does NOT:** produce a fault. Both scored arms are clean under that pressure. - -So the honest reading at this point is **negative for the lead as stated**: heavy -eviction/preemption in regime is present *and* the arm is clean. Eviction pressure is -therefore **not sufficient** on its own at this resolution. That is narrower than the -#123/#140 record predicts ("the window opens when a dispatch is preempted, and any -process's KFD queue eviction ... does that"), and it must not be written up as support for -the mechanism. - -Two caveats against over-reading the negative, both required before it is treated as a -result: -1. the run is incomplete (1 of 12 arms scored), so the fault may simply not have occurred - yet in this composition; -2. the fix cell here carries HOST_COHERENT — a lever already falsified — so a clean fix - arm is expected and says nothing about the fault. - -## Contract reminder (task-8) - -Satisfied by EITHER `>=1` in-regime faulting arm with the correlation attached, OR the -trace recorded **UNINFORMATIVE with its clean-arm count**. A clean trace is not evidence -about the mechanism. If the run ends clean, that is what will be recorded. - ---- - -## Run 1 (`315evict2`) — completed, but TRUNCATED, and why - -The first real trace run ended after **2 arms, not 12**. Cause: my wrapper passed -`MINCLEAN=0` intending "no minimum", but `315-paired.sh:147` treats MINCLEAN as a goalpost -and stops when the clean count REACHES it — so `0 >= 0` stopped the run at pair 1. See -`315-evict-trace-instrument.md` §6a. Fixed (`MINCLEAN=$((PAIRS*2+1))`), plus §6b (a -`grep -c || echo 0` producing a two-line integer). - -What the truncated run did establish, at full strength, is the **eviction environment**: - -| quantity | value | -|---|---| -| Q_EVICT | **15,357** (8,936 SVM + 6,421 USERPTR) | -| Q_RESTORE | 19,649 | -| — `rescheduled=no` | **15,353** | -| VMFAULT | 0 | -| arms scored | 2 — `c` in-regime OK, `l` **OOR** (93 GiB < 95 gate) | -| in-regime arms | **1** | -| in-regime faults | **0** | - -So: 15.4k evictions, 15.4k of them unrestored, sustained across the window, with one -in-regime clean arm. **This is a null for task-8, not a result** — and it is explicitly -*not* recorded as evidence about the mechanism. - -## Why run 1 cannot be read as anything stronger - -1. one in-regime arm is nothing like the `>=60` task-9 needs, and task-8 itself wants a - *faulting* arm to correlate; -2. an `OOR` arm is neither clean nor faulted, so the fix cell contributed **no** in-regime - data at all; -3. the campaign was truncated by a harness bug, so its length was not the designed length. - -The honest statement is: **the eviction/preemption window was open and intense, and the -one in-regime arm was clean.** That is a datapoint about eviction pressure being present -without the fault, and nothing more. - -## Run 2 — not launched: box contention - -A relaunch on 2026-10-12 13:09+ found a peer agent's HRX workload on the box -(`~/wt/loom-ub389`, `guard.sh`-wrapped, 1800 s timeout each, first a 27B Qwen3.8 run then a -Zamba2-VL-2.7B run, `npu.lock` freshly touched). The campaign composes ~5 neighbours into -the 95–110 GiB band, which would have evicted that peer's memory. Per the goal's own -constraint (take the box lock, do not disturb other workloads) the relaunch **waits** for -the box rather than competing for it. - -This is the second time the box has been unavailable for a long in-regime window (the -first was `repro336` holding 103–111 GiB). It is a real constraint on task-8, not an -excuse: the trace needs an uninterrupted composition, and the box is shared. - -## Run 2: deferred, with the reason and the resume command - -Not launched. The box is held by a peer agent's **batch** (`~/wt/loom-ub389/runs/batch2.log`, -active 13:10, jobs queued, several `-dev HRX0`) plus a CPU embedding pass. The campaign's -composition drives ~5 neighbours into the 95–110 GiB band, which would evict that peer's -memory. The goal's constraint is explicit — take the box lock, do not disturb other -workloads — so the trace waits. - -Resume when the box is quiet, with the two launch bugs fixed: - -```sh -ssh strixhalo -# confirm free: no "-dev HRX0" jobs of anyone else's, GTT near the ~26 GiB baseline -ps -eo pid,args | grep -- "-dev HRX0" | grep -v grep -awk '{printf "%.1f GiB\n", $1/1073741824}' /sys/class/drm/card*/device/mem_info_gtt_used - -# then, with the corrected wrapper (MINCLEAN=$((PAIRS*2+1)), PAIRS cap ends the run): -nohup bash ~/wt/315-evict-trace.sh 8 315evict3 > ~/wt/315evict3-run.log 2>&1 & -``` - -8 pairs = 16 arms, which at the observed ~1-in-2 in-regime rate gives ~8 in-regime arms. -That is still short of task-9's `>=60`; task-8 needs only one **faulting** in-regime arm, or -an honest null with its clean-arm count. - -**If the box stays contended, task-8's honest close is the null already in hand**: 1 -in-regime clean arm under 15.4k evictions, recorded as uninformative, with the reason being -box contention rather than a design choice. - -## Box contention, measured 2026-10-12 13:10–13:17 - -The failure mode here is not a bug in the harness — it is that the box is continuously busy. -Measured, not assumed: - -| observation | value | -|---|---| -| peer batch | `~/wt/loom-ub389/runs/` — grew `batch1` → `batch2` → `batch3` while waiting | -| job types in the batch | `emb_hrx`, `emb_cpu`, `rr_hrx`, `rr_cpu`, `vis_hrx`, `zamba_base/fix/fin_ub`, `txt_fin_ub`, `ms_fin_b`, `fin_` | -| HRX0 occupancy, sampled every 15 s for 2 min | **1–2 holder processes at every sample, never 0** | -| GTT while a peer job ran | steady ~27 GiB (a small job, not the 95–110 band) | -| one peer job's duration | 6 min 21 s elapsed against a 1200 s cap, with more queued behind it | - -So the peer's batch takes HRX0 continuously in small jobs. My composition needs ~5 -neighbours resident in the 95–110 GiB band for ~5 minutes per arm; running it would evict -the peer's live job. The goal's constraint is to not disturb other workloads, so run 4 -polls (30 min budget) and aborts having started nothing rather than racing. - -**This is the second long unavailability this task has hit** (the first was `repro336` -holding 103–111 GiB on 2026-10-09). It is a property of a shared box, and it is the honest -reason task-8 may close on the null already in hand rather than on a new run. - -Note also: GTT stays ~27 GiB during peer jobs, i.e. the peer never approaches the regime. -A cheaper future option — not taken here because it would change the pre-registered -composition — would be to interleave arms into gaps rather than requiring an empty box. - -## Runs 4 and 5: the lock finding, and where the campaign now stands - -**Run 4** (`315evict4`) ran the wrapper *before* the two-lock fix. It "acquired the box -lock" at 13:13:03 and sat in the GPU-quiet poll for 5m32s while `~/npu.lock` was held by 12 -other processes. Its log line `box lock held` read as protection and provided none — the -poll was doing all the work. Stopped and discarded rather than left running in a -semantically unprotected state. - -**Run 5** (`315evict5`) runs the corrected wrapper. Its log now distinguishes the two locks -and, crucially, **stops at the one that matters**: - -``` -box lock held 2026-10-12T13:18:52-03:00 (/home/bcloud/.cache/lax-decode/box.lock) - ... then waits on ~/npu.lock, held by 14 processes ... -``` - -That is the expected behaviour and it is a behaviour change worth stating: before the fix -the run would have proceeded into the composition while another worker held the box-wide -lock. Now it waits, and aborts having started nothing if the lock is unobtainable within -its budget. - -### Status: task-8 is blocked on the shared box, not on the harness - -The harness has now been hardened against seven distinct silent-failure modes -(`315-evict-trace-instrument.md` §6a–§6g). What remains is not engineering: - -| | | -|---|---| -| `~/npu.lock` holders while run 5 waited | 12 → 14 | -| workers holding it | a peer batch (`loom-ub389`) **and** two other goals' workers (`npu-forge2-kvsplit`, `probe_model.sh`) | -| HRX0 occupancy | 1–2 processes at every sample over 2 min | -| regime requirement | ~5 neighbours resident in the 95–110 GiB band, ~5 min per arm | - -So the honest position on task-8 is: - -- **evidence in hand**: run 1 — 1 in-regime arm, 0 faults, under 15,457 evictions - (15,353 of them unrestored). A **null**. -- **blocked**: a fresh run needs the box-wide lock free for a sustained window, and three - unrelated workers are currently holding it. - -Per the task-8 contract, the null is recordable as **uninformative with its clean-arm -count**, and the reason is box contention. That is a legitimate close, not a design choice — -but it is a weaker close than a faulting arm would be, and it should be reported as such. - -## The lock holder is another #315 campaign, thermally stuck while holding the lock - -Diagnosed 2026-10-12 13:19–13:22. `~/npu.lock` was not held by a peer agent's batch alone — -it was held by **another #315 campaign**, `~/wt/315fix-tools/camp5.sh` (started 13:03:31), -running arms tagged `fx-c5-*` on the same control binary via an `LD_PRELOAD` shim -(`cwsr_shim2.so`) with modes `lockall`, `pinall`, `pincwsrstaging`. - -**It has produced nothing in 16 minutes.** Its arm directory -`~/wt/rt-det/fx-c5-lockall-r1` is entirely empty — no `run.log`, no `server.log`, no -`detector.json` — and its only child is `flock 7` (holding `npu.lock`, 13+ minutes). - -**Why, exactly.** The arm script gates on temperature *before* it starts work -(`/tmp/315arm-6xh7.sh:33`): - -```sh -while [ "$(tctl)" -ge ${TSTART:-72} ]; do sleep 10; done -exec 7>"$HOME/npu.lock"; flock 7 -``` - -and it takes the lock on the next line. Measured `Tctl` over 60 s: - -| t | Tctl | edge | -|---|---|---| -| 10 s | 70.8 °C | 37.0 °C | -| 20 s | 72.5 °C | 37.0 °C | -| 30 s | 72.1 °C | 37.0 °C | -| 40 s | 71.8 °C | 37.0 °C | -| 50 s | 72.1 °C | 36.0 °C | -| 60 s | 72.1 °C | 36.0 °C | - -`Tctl` oscillates **70.8–72.5 °C** around the 72 °C gate, so the wait never reliably -completes. Note the two sensors disagree wildly: `edge` is 36–37 °C while `Tctl` sits at -~72 °C. The gate is on the hot one, and the arm holds `npu.lock` for the entire wait. - -**Consequence for task-8:** run 5 is starved — it correctly took `box.lock`, then blocked on -`~/npu.lock`, which the thermally-stuck arm holds. Waiting cannot fix this; the other -campaign must clear or its lock must be released. - -**This is a design flaw in the other harness, not a bug in mine**, and it is worth flagging -plainly: an arm that holds the box-wide lock **while waiting for a thermal precondition** -can block the box indefinitely when the precondition sits on the wrong side of its -oscillation. The safe order is to wait for thermal conditions *before* taking the lock (or to -bound the wait and release). - -## Root blocker identified: a peer batch holds `npu.lock` while its jobs fail - -Traced the queue depth on 2026-10-12 13:24. `~/npu.lock` had 14 holders, but only one -matters, because `flock` is first-come: **pid 99828**, `flock ~/npu.lock ./batch2.sh`, -started 11:39:11 — **held 1 h 44 m**. Every other holder is queued behind it, including: - -| holder | waiting | what | -|---|---|---| -| 948331 + 948371 | — | **run 5** (this task), correctly blocked | -| 847551 + 867745 | 16 m 32 s | camp5's `fx-c5-lockall-r1` arm, whose `flock 7` has not yet acquired | -| 816800 / 826913 / 826914 | — | **two other goals' workers** (`npu-forge2-kvsplit` bisect, `probe_model.sh`) | -| 816808 / 826939 / 826940 | — | their `flock -w 7200 9` children | - -**The lock owner is not healthy.** `batch2.log` stopped growing (7185 bytes, unchanged over -40 s) and its current job is failing outright: - -``` -0.00.126.893 E llama_model_load: error loading model: unknown model architecture: 'zamba2' -0.00.126.905 E llama_model_load_from_file_impl: failed to load model -0.00.127.071 E common_fit_params: ... failed to load -client_rc=1 -urllib.error.URLError: -``` - -18 × "failed to load model" and 5 × fit-params errors in that batch. And its current job is -**CPU-only** (`-dev none -ngl 0 -t 4 --no-mmproj-offload`), i.e. it is holding the GPU-wide -lock while running something that does not use the GPU, against a model its binary does not -support. - -**Not touched.** That is a peer's process. The goal's constraints say do not disturb other -workloads, and killing another agent's batch to win the box is not mine to decide. Recorded -here precisely so the decision is visible to whoever owns it. - -**Consequence for task-8:** the trace cannot start while this holds the lock. Run 5 will time -out on `NPU_LOCK_WAIT` (900 s) and abort having started nothing — which is the correct -behaviour, and better than the pre-fix version that would have run unprotected. - -## Where this leaves task-8 - -- harness: hardened against seven silent-failure modes, verified -- evidence in hand: **run 1** — 1 in-regime arm, 0 faults, under 15,457 evictions (15,353 unrestored). A **null** -- fresh run: blocked on `~/npu.lock`, held by a peer batch whose jobs are failing, with two - other goals' workers also queued - -The task-8 contract permits closing on the null, recorded as uninformative with its -clean-arm count. The reason is box contention — externally caused and now documented — not a -design choice and not a harness defect. - -## Final measured state at the block (2026-10-12 13:26) - -`~/npu.lock` queue, oldest first — the head does not drain: - -| waited | pid | what | -|---|---|---| -| **6,376 s (1 h 46 m)** | 99828 | `flock ~/npu.lock ./batch2.sh` — **the head, and wedged** | -| 1,583 s | 816808 / 816800 | another goal's `npu-forge2-kvsplit` bisect | -| 1,479 s | 826939 / 826940 / 826913 / 826914 | another goal's `probe_model.sh` ×2 | -| 1,345 s | 845081 | `./batch2.sh` itself | -| 6:31 | 948371 | **our run 5** (`flock -w 900 8`), will abort at 900 s | - -`batch2.log`: **7,185 bytes, unchanged over 30 s — wedged.** It has 18 × "failed to load -model" (`unknown model architecture: 'zamba2'`) and its current job is CPU-only -(`-dev none -ngl 0 -t 4 --no-mmproj-offload`). - -**The queue is growing, not draining:** three unrelated workers are now parked behind a -batch that is failing and not progressing. That is why retrying is not a strategy — a fresh -attempt does not move closer to the head, it joins the queue behind a stalled owner. - -## Correction to the first blocker note - -The blocked summary says "three consecutive goal turns hit the same cause". That is right in -substance but understated the first two: they were plain contention (`repro336` at -103–111 GiB). **Today's is a wedged lock owner**, which is a different and worse problem — -contention eventually clears, a stalled batch does not. Recorded so the decision is not made -against the weaker description. - -## CORRECTION + the real blocker: `~/npu.lock` is wedged by a dead holder - -### What I got wrong - -I previously recorded the blocker as "a wedged peer batch (pid 99828/99830) holds -`npu.lock` while its jobs fail". **That was wrong.** `ps` shows `flock` for both holders and -waiters, and I read elapsed time as evidence of holding. The kernel's own lock table says -otherwise. - -### What the kernel actually says - -`/proc/locks` for the inode behind `~/npu.lock` (`103:02:80249278`) — unindented line is the -**granted** lock, `->` lines are **waiters**: - -``` -10: FLOCK ADVISORY WRITE 826940 103:02:80249278 0 EOF <- HOLDER -10: -> FLOCK ADVISORY WRITE 867745 103:02:80249278 0 EOF <- waiting -10: -> FLOCK ADVISORY WRITE 99830 103:02:80249278 0 EOF <- waiting -10: -> FLOCK ADVISORY WRITE 1155054 103:02:80249278 0 EOF <- waiting -``` - -**The holder `826940` no longer exists.** `/proc/826940` is gone, its whole group -(826913/826939/826940, `probe_model.sh`) has exited, yet the kernel still grants it the -lock. The three waiters are alive and parked in `locks_lock_inode_wait`: - -| pid | state | wchan | -|---|---|---| -| 867745 | S | `locks_lock_inode_wait` | -| 99830 | Ss | `locks_lock_inode_wait` | -| 1155054 | S | `locks_lock_inode_wait` | - -### It is reproducible, not a transient read - -``` -t=10s HELD -t=20s HELD -t=30s HELD -box.lock: FREE <- same filesystem, same mechanism: flock itself is fine -``` - -`flock -n` against `~/npu.lock` fails on every attempt while `box.lock` acquires immediately. -So this is specific to that lock file, and it is **stuck**, not busy. - -### Why this matters beyond my task - -`~/npu.lock` is the **box-wide mutex for GPU work** (§6g). Every GPU harness on this box -queues on it — mine, the peer batch, and two other goals' workers. With a dead holder, **all -of them are blocked permanently**, and no amount of retrying or waiting helps. This is not -contention that clears; it is a wedged mutex. - -**Possible cause, stated as a hypothesis not a finding:** the holder was a `flock` in a -`systemd-run --scope` unit. If the scope was torn down while its flock was granted, the -granted entry can outlive the process in `/proc/locks`. That would make -`flock ~/npu.lock ...` inside a transient scope an unsafe pattern on this box — an advisory -lock whose holder can be reaped out from under it. - -### What this means for task-8 - -The blocker is **not** a peer batch I should leave alone; it is a **stuck kernel lock -holding back every GPU consumer on the box**, including two other goals. Retrying (option c) -cannot work. Waiting (which I have done for three turns) cannot work. - -I have not cleared it: `flock` offers no break mechanism from userspace, and the only ways -that occur to me are destructive (reboot, or `fuser -k` on the lock, which would kill three -innocent waiters' work). That is the user's call, not mine. - -## The wedge is cleared by a clean reboot, and the campaign is finally running - -**2026-10-12 14:03:53 — box rebooted** (user decision, option (b)). State after boot: - -| check | value | -|---|---| -| `~/npu.lock` | **FREE** (was wedged 36+ min) | -| `~/.cache/lax-decode/box.lock` | FREE | -| stale `/proc/locks` entries for it | **0** | -| HRX0 | 0 processes | -| GTT | 26.6 GiB (baseline) | -| `Tctl` | 40.2 °C | -| `MemAvailable` | 90 GiB | - -The dead holder `826940` is gone and the three waiters it had blocked are gone with it. Cost: -the three queued workers' in-flight work (a #315 arm, `batch3.sh`, a `bisect4.sh` worker), -accepted by the user. - -**Run 6 (`315evict6`) launched 14:05:45 with the full corrected guard sequence — and it is the -first run to get through all four gates:** - -``` -box lock held 2026-10-12T14:05:45-03:00 (/home/bcloud/.cache/lax-decode/box.lock) -npu lock held 2026-10-12T14:05:45-03:00 (/home/bcloud/npu.lock) - the box-wide convention -GPU quiet: no foreign HRX job (waited as needed) -watcher pid 11922 -> /home/bcloud/wt/315evict6-evict/kfd_smi.tsv -``` - -`315-campaign.sh` and `315-paired.sh` are composing (8 pairs = 16 arms). This is the first -attempt in this task that reached composition rather than aborting at a gate, which is what -the harness work was for. - -**Not yet evidence.** task-8 still needs an in-regime arm to *fault* while the watcher is -attached, or the null recorded with its clean-arm count. Nothing here changes that. - -## Run 7 (`315evict7`) — first campaign to run properly in regime - -After the reboot and the NOLOCK fix (§6h), run 7 passed all four gates and reached the regime: - -| | | -|---|---| -| gates | box lock → npu lock → GPU quiet → watcher (all four passed) | -| composition | 4 HRX neighbours, GTT 27 → 80 → 98 → **104 GiB** | -| arm 1 (`p1c`) | `run.log: gtt_start 97GiB` — **in regime** | -| requests | 140 → 208 in 60 s (68/min) | -| evictions | 662 → 668 | -| deadlock | none (`do_wait`) | - -This is the first run in the whole task that got past composition and into a scored -in-regime arm while the eviction watcher was attached. Everything before it aborted at a -gate or deadlocked: run 1 was truncated by `MINCLEAN=0`, runs 3–5 never cleared the locks, -run 6 self-deadlocked on the lock the driver re-opens. - -**What it is for:** task-8 needs an in-regime arm to **fault** while the watcher runs, so the -fault can be correlated against the driver's own eviction events. 16 arms (8 pairs) is the -target; the observed in-regime rate is roughly 1 in 2, so expect ~8 in-regime arms. - -**What it cannot do:** satisfy task-9, which is gated on a named mechanism no instrument has -produced. A clean run here is task-8 evidence, not the objective's fix+≥60-arm deliverable. - -**If all 16 arms are clean**, the contract requires recording this trace as -**UNINFORMATIVE with its clean-arm count** — never as evidence about the mechanism. - -## RUN 7 RESULT: an in-regime FAULT on Face B, correlated against its own evictions - -**This satisfies the first branch of the task-8 contract.** - -``` -pair 3 arm l: FAULT nan=0 req=20 parse=0 gtt_start=97G utf8=0 alloc=0 hv52=0 hv56=0 (gtt_now=102G) -``` - -| quantity | value | -|---|---| -| arm | `pair 3 arm l` (the **fix** cell — diagnostics + HOST_COHERENT) | -| regime | **`gtt_start=97 GiB`** — inside the frozen `>= 95 GiB` gate | -| face | **B** — `http_failures: 1`, `first_failure: "HTTP 500"`, `truncated: true`, `nan=0` | -| requests | 20 served, 19 ok, then the 500 stopped the arm | -| detector.json | `{"requests":20,"ok":19,"utf8_failures":0,"parse_failures":0,"http_failures":1,"transport_failures":0,"truncated":true,"first_failure":"HTTP 500"}` | - -### The failure, verbatim from the arm's server.log - -``` -1.21.180.831 W common_chat_peg_parse: unparsed Content-only output: laptop too -1.21.185.082 W srv operator(): got exception: {"error":{"code":500,"message":"The model produced - output that does not match the expected Conten... (truncated) -``` - -That is Face B exactly as the issue describes it: output that is **not valid UTF-8 / not -parseable**, surfacing as an HTTP 500 — not a NaN. `nan=0` confirms it is Face B and not -Face A. - -### The correlation (task-8's actual deliverable) - -Bound to the arm's own PID **exactly** (`tools/arm-pid.py` → `111371`, from the port in the -arm's own `server.log`; not guessed). Against the eviction stream, for that PID alone: - -| event | count | -|---|---| -| `Q_EVICT` | **323** | -| `Q_RESTORE` | **357** | -| timeline rows | 681 → `~/wt/315evict7-evict/correlated-p3l.tsv` | -| trigger | `USERPTR` throughout | - -So the faulting arm can be placed against **its own driver evictions**, which is the thing the -#123/#140 lead predicted would be necessary and which no earlier run could obtain (runs 3–6 -never reached an arm; run 1's one in-regime arm was clean). - -### What this does and does not establish — stated before the run finishes - -**Does:** an in-regime arm faulted on Face B while its own PID was being evicted hundreds of -times, with both recorded on one clock. The correlation exists and is reproducible from the -paths above. That is what task-8 asked for. - -**Does NOT:** show that the evictions *caused* the fault. A clean arm in this same run -(`p1c`, `p1l`, `p2l`, all `gtt_start=97G`, all `OK`) was also being evicted — the eviction -counts are not yet compared between faulting and clean arms, and the fault is intermittent -by construction. **Eviction is necessary-looking, not sufficient**, exactly as run 1 already -suggested with 15,357 evictions and zero faults. - -**Must not be written up as:** "the evictions caused it", or as a mechanism. The mechanism is -still unnamed; this is the correlation evidence, which is the input to naming one. - -## The comparison I cannot make retrospectively — and why - -The obvious next step is to compare eviction counts between the **faulting** arm and the -**clean** arms, all at `gtt_start=97G`. I can produce the faulting arm's count and **not** -the clean arms', and the reason is a known limit of this instrument: - -| arm | port known | PID known | own evictions | -|---|---|---|---| -| `p3l` (FAULT) | 19508 | **111371** — captured **while live** | 323 Q_EVICT / 357 Q_RESTORE | -| `p1c`, `p1l`, `p2l` (OK) | 19498 / 19636 / 19461 | **not recoverable** | — | - -The SMI stream's `PROC_START` rows carry no port (`raw= llama-server`), so once an arm's -server exits there is no exact way to bind it to a PID after the fact. `tools/arm-pid.py` -deliberately **refuses** rather than guessing in that case, because `correlate.py`'s own -fallback ("most frequent eviction subject") was observed landing on a dead PID. - -**Consequence:** this run gives the faulting arm's correlation but **not** the faulting-vs-clean -eviction comparison. That comparison is what would turn correlation into a start on causation. -It needs the PIDs captured **live** for every arm — a one-line change to record the PID into -each arm directory at start, which the frozen driver does not do and I have not edited -(editing it mid-run would invalidate the run). - -**Recorded as the concrete next step, not as done.** Until it exists, the honest statement -remains: the faulting arm was evicted heavily, the clean arms were in the same regime and were -presumably also evicted, and nothing here separates them. - ---- - -# RUN 7 FINAL: 11 in-regime arms, 3 faults, BOTH FACES captured - -Run 7 (`315evict7`, wrapper with all eight fixes) completed. **11 in-regime arms, 3 faults** — -and it is the first run in this task to capture **both** failure faces in regime with the -eviction watcher attached. - -``` -pair 1 arm c: OK nan=0 req=66 gtt_start=97G -pair 1 arm l: OK nan=0 req=66 gtt_start=97G -pair 2 arm l: OK nan=0 req=66 gtt_start=97G -pair 3 arm l: FAULT nan=0 req=20 gtt_start=97G <- FACE B -pair 4 arm c: OK nan=0 req=66 gtt_start=98G -pair 4 arm l: OK nan=0 req=66 gtt_start=98G -pair 5 arm l: OK nan=0 req=66 gtt_start=98G -pair 6 arm l: OK nan=0 req=66 gtt_start=98G -pair 7 arm c: FAULT nan=1 req=16 gtt_start=98G <- FACE A -pair 7 arm l: FAULT nan=1 req=1 gtt_start=98G <- FACE A -pair 8 arm l: OK nan=0 req=66 gtt_start=98G -``` - -Every arm at `gtt_start` 97–98 GiB, inside the frozen `>= 95 GiB` gate. - -## Face A — the canonical guard line - -Both `p7c` and `p7l` produced the exact symptom the issue records: - -``` -0.57.838.128 E common_sampler_sample: HRX returned NaN logits: count=154880 of 154880, - index range [0..154879], contiguous=yes, continuing=no (engine#315) -``` - -`count == vocab == 154880` with `contiguous=yes` — the whole LM-head output NaN, exactly as -`KNOWN-ISSUES.md` describes. Detector: `transport_failures: 1`, -`first_failure: "RemoteDisconnected: Remote end closed connection without..."` (the server -aborting on NaN). - -## Face B — `p3l` - -Detector: `http_failures: 1`, `first_failure: "HTTP 500"`, `truncated: true`, `nan=0`. Server: - -``` -1.21.180.831 W common_chat_peg_parse: unparsed Content-only output: laptop too -1.21.185.082 W srv operator(): got exception: code 500 - The model produced output that does - not match the expected Content... -``` - -## Both faces were captured with the eviction stream attached - -`8419 Q_EVICT` / `8417 restores not rescheduled` across the run, with `trigger=USERPTR 4866` + -`trigger=SVM 3553`. The Face-B arm's own PID was bound exactly while live: -**323 Q_EVICT / 357 Q_RESTORE** for pid 111371 → -`~/wt/315evict7-evict/correlated-p3l.tsv` (681 rows). - -## What this establishes, and what it does not - -**Establishes:** the fault is reproducible **on demand in regime** — 3 faults in 11 arms -(~27%), both faces, from a run where every arm was inside the gate and the driver's eviction -events were recorded on one clock alongside. That is the input task-8 asked for, and it -supersedes the earlier null (run 1: 1 in-regime arm, 0 faults). - -**Does not establish:** that evictions *cause* it. The clean arms ran in the same regime, -presumably also evicted, and I could not compare them because finished arms are not -PID-bindable (§ above). Faulting-vs-clean eviction counts remain the missing measurement. - -**Does not name a mechanism.** Three faults in one run is a reproducibility result, not a -mechanism. The four falsified lever families and the closed ordering/visibility super-family -are untouched by it. - ---- - -# The recorded blocker is STALE — both its premises are resolved - -The goal's blocker text (written before the reboot) says: - -> `~/npu.lock` is WEDGED: the granted holder is pid 826940, which no longer exists ... -> Needs a user decision: (a) accept task-8 closing on the null ... (b) clear the wedged lock -> ... (c) authorise running despite the queue. - -**Every clause of that is now out of date**, verified 2026-10-12 15:07: - -| claim in the blocker | measured now | -|---|---| -| `npu.lock` is wedged | **FREE** (`flock -n` succeeds) | -| holder 826940 still holds it | `/proc/826940` gone, **0** stale `/proc/locks` entries | -| need to choose (a) accept the null | **superseded** — run 7 produced 3 in-regime faults, both faces | -| need to choose (b) clear the lock | **done** — the user's clean reboot cleared it | -| need to choose (c) authorise running | **done** — run 7 ran, in regime, and completed | - -The blocker was correct when written and is now resolved by events. It is left in place only -because goal blockers are not editable from here; this note is the correction of record. - -**Where the goal actually stands:** - -- task-5, task-6, task-7: complete -- task-8: **evidence obtained** — run 7, 11 in-regime arms, 3 faults (Face A ×2, Face B ×1), - with the eviction stream attached and the Face-B arm's PID correlated exactly. The one - remaining measurement (faulting-vs-clean eviction counts) has a verified one-line fix - proposed in `315-pid-capture.md`. -- task-9: still gated on a **named mechanism**, which no instrument has produced. Four lever - families are falsified and the ordering/visibility super-family is closed both by run - (§59, Fisher p=1.00) and at source. This is unaffected by the box being free. - -So the goal's blocker should be cleared, and the open question is not the box — it is whether -the objective should be re-scoped from "land a fix + ≥60-arm campaign" (unreachable, no named -mechanism) to "name the mechanism, or record the closed families and the next instrument" -(what the evidence supports, and largely written). - -## Data-quality check of the running campaign (2026-10-12, ~9 arms in) - -A valid arm is one whose detector completed all 66 requests with no truncation and no first -failure. Every scored arm so far qualifies, and every arm is inside the gate: - -``` - p1c: req=66 ok=66 trunc=False first=None -> VALID - p1l: req=66 ok=66 trunc=False first=None -> VALID - p2l: req=66 ok=66 trunc=False first=None -> VALID - p3l: req=66 ok=66 trunc=False first=None -> VALID - p4c: req=66 ok=66 trunc=False first=None -> VALID - p4l: req=66 ok=66 trunc=False first=None -> VALID - p5l: req=66 ok=66 trunc=False first=None -> VALID - p6l: req=66 ok=66 trunc=False first=None -> VALID - p7c: req=66 ok=66 trunc=False first=None -> VALID - p7l: (in flight) - - gtt_start distribution: 9 x 97G (gate is >= 95) -``` - -So there are **no malformed arms and no out-of-regime arms** — the two ways this campaign could -have produced a technically-clean but invalid result. Nine clean in-regime arms against run 7's -3 faults in its first 11 at the same 97G gate. - -**Still not a result.** Nine arms gives a rule-of-three upper bound of 33%, which does not -exclude the 26–27% baseline, and the H1 confound (fresh boot) is unaddressed. Recorded as a -quality assurance datum, not evidence about the mechanism. - -## The H1 control script: hardened and dry-run verified safe (2026-10-12) - -Sizing changed from 48 to **20 pairs**: at the ~26% baseline that gives ~5.2 expected faults and -`P(0) = 0.74^20 = 0.002`, so a clean control is real evidence and a ~26% control kills H1. 48 pairs -would cost ~3 extra hours for no added power on that question, and the box-time budget is the binding -constraint. - -**One real hazard removed.** The script copies the PID-capturing driver into place and previously -trusted the copy. A run without the capture silently produces data that *cannot* support the H2 -comparison — and the failure is invisible until the analysis, after the GPU hours are spent. It now -verifies the installed driver actually writes `server.pid` and aborts if not: - -``` - installed the PID-capturing driver (original kept at 315-paired.sh.orig) - verified: driver writes server.pid (H2 data will be capturable) -``` - -**Dry-run, with `sudo` and `pgrep` neutered so no reboot and no drop-in removal could occur:** - -``` - current cwsr_enable : 0 - drop-in present : yes - campaign to mirror : 315-evict-trace.sh 20 (the cwsr=0 campaign of record used 70) - ABORT: an HRX job is running (not ours). Do not disturb it. -``` - -Checked immediately after: - -``` - drop-in: PRESENT cwsr_enable: 0 campaign alive: 2 uptime: up 2 hours, 5 minutes -``` - -So the destructive steps were not reached and the running campaign was not disturbed. The abort came -from the script's **second** guard, the HRX check — it fires because the campaign's own arm server -holds the GPU, and it cannot distinguish our server from a peer's. That is the correct conservative -behaviour on a shared box: it refuses to reboot while anything holds the GPU, at the cost of a -slightly misleading "not ours" message. Worth knowing when the control is run for real: it will abort -if any foreign HRX job is up, so run it when the box is genuinely quiet. - -`bash -n` clean, and the two abort guards (`pgrep 315-evict-trace`, foreign HRX job) both fire before -any `sudo`, which is the ordering that makes the script safe to invoke by mistake. diff --git a/dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md b/dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md deleted file mode 100644 index 1c2d65603..000000000 --- a/dev/315/315-campaign/evidence/315-eviction-hits-the-arm.md +++ /dev/null @@ -1,135 +0,0 @@ - - - -# The eviction hits the ARM's own buffers, not only the forcer's — read from the trace - -Written while arm B (`315ab1b2`) runs, to fix a misreading before it enters the record. Reading -`~/wt/evictonly.hip` suggests the intervention is self-contained ("only makes KFD evict and restore -**its own** queues"), and the mechanism statement's wording ("eviction of in-flight KFD work") could be -read either way. The trace settles it, and the settled reading is stronger. - -## What `evictonly` does - - hipHostRegister(16 MiB anon range) # a host-visible, KFD-managed range - hipMalloc(4B); hipMemset; synchronize # ensures a queue exists - loop: mprotect(PROT_READ) <-> mprotect(PROT_READ|WRITE); touch a page; usleep(500) - -The `mprotect` churn on a registered range is what makes KFD evict and restore queues. The comment in -the source frames this as the process's *own* queues — but the file is describing what it *causes*, not -who *receives* it. - -## What the trace shows - -The control run's SMI stream, per arm, attributed by the arm's own server pid -(`315-task9-session-state-2026-10-12.md`, P3 section): - - arm pid Q_EVICT Q_RESTORE - 21139 293 411 - 44443 160 173 - 68030 262 288 - 91692 196 257 - 114850 275 307 - -**Every arm's own buffers are evicted and then restored while the arm is live.** So the eviction is a -device-wide event under memory pressure, not something confined to the forcer process. `evictonly` is a -*pressure generator*; the arm is a *victim*. - -## Why this sharpens the mechanism - -The mechanism statement should be read as: - -> generating KFD evict/restore pressure while the GLM server has in-flight work causes **the server's -> own queue memory** to be evicted and restored mid-flight, and something read across that boundary is -> consumed without re-validation, producing both faces. - -That is a materially different and more specific claim than "the forcer's memory gets evicted." Two -things follow, both checkable: - -1. **The stale data is in the server's own evicted-then-restored memory**, not in the forcer's — which - is exactly why the per-arm correlator (`315-evict-per-arm.py`) compares the *arm's* pre-fault - eviction count against a clean-arm baseline (~228/arm). The instrument is already aimed at the right - pid. -2. **The read fix addressed a real incoherence but the wrong one.** The logits read is host-side; the - eviction is of device-side queue memory. Routing the host read through a coherent transfer cannot - repair a device-side restore that left different bytes behind — consistent with arm B's cell-`e` - fault with the knob on. - -## The pre-registered refutation test this enables - -P3 as written: *"if an arm faults at cwsr=1 with ZERO Q_EVICT for its own pid, H2' is dead."* That test -is aimed at the arm's own counters and remains the right instrument. It is live on the current run: the -re-run's per-arm SMI stream should be read for arm `315ab1b2-r1e`, where a fault already landed at -10:33:19, to test P3 against a real faulting arm rather than assuming it. - -## What this does not do - -It does not locate the cause. It narrows the cause to "something the server reads from its own -evicted-then-restored memory without re-validating," and rules the forcer's memory out as the site. -The surviving candidate families are the same ones §55/§56 left open (executor-level cross-dispatch -ordering, or a different class entirely), not the read path this A/B tested. - ---- - -# HARNESS GAP (same turn) — P3 cannot be tested on the A/B runs, because no SMI watcher was launched - -The section above promises to "read the re-run's per-arm SMI stream for arm `315ab1b2-r1e`". That is -**not possible on these runs**, and the gap is recorded rather than worked around. - -## Why not - -The eviction counters come from a separate privileged watcher, launched by -`dev/315/315-campaign/315-evict-trace.sh`, not by the campaign driver: - - WATCH=$HOME/amd-oss/tools/315-trace/kfd_smi_watch # 315-evict-trace.sh:25 - sudo -n true || { echo "ABORT: the watcher needs passwordless sudo"; exit 2; } # :40 - sudo -n nohup "$WATCH" -e all -t 10800 -o "$OUT/kfd_smi.tsv" -q & # :146 - -`~/wt/315-disc.sh` — the driver the A/B used — contains no SMI watcher and no `sudo` call. It records -only `side=/` per arm, which proves the forcer was *running*, not what KFD did. - -Neither `315ab1b2` nor its discarded predecessor launched the watcher, so for both A/B arms: - - ~/wt/315ab1b2-evict/ does not exist - pgrep kfd_smi_watch nothing - -The raw interface is also not open to me: `/sys/kernel/debug/kfd/kfd_smi_events` does not exist on this -box's mount, and the script's own guard requires passwordless sudo, which is the documented reason the -watcher is a separate opt-in step. - -## What this costs, and what it does not - -- **Costs**: P3 (the pre-registered "a faulting arm with zero Q_EVICT for its own pid kills H2'") cannot - be evaluated on the A/B. The eviction refinement above rests on the *earlier* control run's counters, - not on this one. A future run that wants P3 must launch the watcher with the driver. -- **Does not cost**: the A/B's own reading. The A/B tests the fix against the forcer, and `side=1/1` - establishes the forcer ran for the whole arm, which is what the pre-registered cell design needs. The - missing counters bear on *which pid* is evicted, not on whether arm B faulted. -- **Is not a reason to re-run the A/B.** Arm B's cell-`e` fault with the knob on already lands in the - pre-registered "H-fix REFUTED" branch. Adding eviction counters would refine the mechanism story; it - would not change the A/B verdict. - -## Lesson - -The campaign driver and the eviction tracer are separate entry points, and the pre-registration did not -say which one to use. For any future run whose reading depends on P3, the tracer must be named in the -pre-registration and launched with the driver — otherwise the run cannot answer the question it was -designed around, which is the same class of harness/measurement mismatch recorded in -`315-evict-trace-instrument.md` §6a–§6h. diff --git a/dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md b/dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md deleted file mode 100644 index bc0adb829..000000000 --- a/dev/315/315-campaign/evidence/315-eviction-replication-preregistration.md +++ /dev/null @@ -1,104 +0,0 @@ - - - -# Pre-registration: replication of the eviction-intervention contrast (`315disc2`) - -Written 2026-10-13 ~09:15, committed before the run. - -## Why this run exists - -`315disc1` (2026-10-11) is the only experiment in this issue's history that **forces** the candidate -mechanism rather than observing it. Its `315-disc.sh` design alternates three cells inside one box -composition: - -| cell | side load | -|---|---| -| `q` | quiet — no side load (control) | -| `e` | `~/wt/evictonly 500` — a process that launches **no kernels** and only forces KFD evict/restore via `hipHostRegister` + `mprotect` churn | -| `c` | CPU pressure only, pinned to CCD1 (no GPU) | - -Its recorded result: - - cell q (quiet) 5 arms 0 FAULT - cell e (EVICTION) 4 arms 2 FAULT (nan=1 transport=1, req=6 and req=4) - cell c (CPU pressure) 5 arms 0 FAULT - Fisher exact, e (2/4) vs q+c pooled (0/10): p = 0.066 - -Both faults are **Face A** (`nan=1`), both abort early (`req=4`, `req=6` of 66), both at -`gtt_start=98G`. The two control cells are clean across 10 arms. - -That is the strongest causal contrast in the issue, and it is the rate-raising design the -26 %-vs-7 % rate correction identified as missing. It has never been replicated. - -## Hypothesis (falsifiable) - -**H-e:** forcing KFD eviction/restore from a separate process that runs no kernels increases the -fault rate, relative to both a quiet box and CPU-only pressure, at the same composition. - -## Design - - driver : ~/wt/315-disc.sh 315disc2 (the SAME frozen design as disc1) - cells : q (quiet) / e (evictonly 500) / c (CPU pressure) — rotating order per round - build : the arm binary the arms have always used (315instr3-build, 10613 / e44c9d01a) - gate : gtt_start >= 95 GiB (the frozen regime gate); OOR arms reported, NOT counted - rounds : 6 -> ~12 e + ~6 q + ~6 c arms, ~43 min at the measured 2.4 min/arm - -The rotating cell order is the design's own protection against session drift aligning with one cell, -and it is why this contrast is interpretable where a start-to-finish A/B would not be. - -## Prediction - -If H-e holds: cell `e` accumulates faults at a rate clearly above cells `q` and `c`, whose arms stay -clean. Combining with disc1, ~6/10 eviction arms faulting against ~0/16 control arms would put the -contrast beyond the p = 0.066 seen at disc1's size. - -## Refutation and the branches, fixed in advance - - e faults clear of q and c -> H-e REPLICATED. Eviction forced from outside causes the - fault; the mechanism is eviction-of-in-flight-work. - e ~= q ~= c (all clean, or all alike) -> H-e NOT REPLICATED. disc1's contrast was a small-sample - artefact or a session-specific condition, and the eviction - intervention is not a mechanism. - eviction cell clean but q or c faults -> H-e CONTRADICTED in direction; report as such. - all cells clean -> UNINFORMATIVE for the contrast. Must NOT be read as - evidence that eviction is irrelevant: at the measured ~5-7 % - ambient rate, ~16 arms would be expected to be clean most of - the time, which is exactly the under-powering trap the - 26 %-vs-7 % correction documented. - -**The uninformative branch is pre-declared as uninformative**, because the ambient rate is low enough -that a clean replication proves nothing on its own; only a POSITIVE (eviction faults, controls do not) -carries information at this size. - -## What this run does NOT do - - - It does not land a fix (criterion 2) - it tests a mechanism (criterion 1). - - It is not the >=60-arm campaign (criterion 3, already MET by `315cwsr70`). - - It cannot by itself close criterion 1: even a clean replication names the TRIGGER (forced - eviction) rather than the CAUSE (which handoff reads stale data). The cause would still need the - static leg, which is already recorded as inconclusive for the #140 class in GLM's path. - -## Cost and budget - -~43 min at 2.4 min/arm, against ~114 min remaining of the objective's 12 h box-time ceiling. It fits. diff --git a/dev/315/315-campaign/evidence/315-exclusions-vs-falsified-levers.md b/dev/315/315-campaign/evidence/315-exclusions-vs-falsified-levers.md deleted file mode 100644 index 4b9563597..000000000 --- a/dev/315/315-campaign/evidence/315-exclusions-vs-falsified-levers.md +++ /dev/null @@ -1,72 +0,0 @@ - - - -# Verification-contract check: the seven exclusions against the recorded falsified levers - -The objective's verification contract requires: *"Re-read the issue's falsified-levers list and confirm the -claimed mechanism is not one of them; if it is, the fix is rejected."* Done explicitly here rather than -assumed, because two of this session's exclusions are **adjacent** to recorded levers and the difference -matters. - -## The recorded falsified levers - -From `KNOWN-ISSUES.md`'s lever table: - -1. memory coherence is the carrier — `HOST_COHERENT` made unconditional (`315accept`, 9 clean + 1 fault / 10) -2. consumer-side ordering is the carrier — `GGML_HRX_DIAGNOSTIC_GRAPH_SYNC=1` (0/4 vs 1/4, p = 1.00) -3. ordering the instrument's own reads — `ggml_backend_synchronize` before each read (p = 1.00) -4. forced AQL execution barrier on every packet — `HRX_FORCE_BARRIERS=1` (3/4 vs 1/4) -5. flash attention is the carrier — `-fa off` (1/4 vs 2/4, fault persists) -6. `HRX_COMMAND_BUFFER_MODE=pm4` — unusable (all fix arms failed at `req=1`) -7. the expert/partition tables go stale — static, both rebuilt by producer dispatches -8. the fused path loses `route_stride` — static, already fixed in every tree that faults -9. the engine's MoE expert-streaming cache — static, Vulkan-only - -## This session's seven exclusions, mapped - -| this session's exclusion | relation to the recorded list | -|---|---| -| logits **read path** (`hrx_synchronous_d2h` at `buffer_get`) | **new** — a specific code site, not on the recorded list | -| **host side** (all four host accesses to the `0x52` mapping) | **adjacent to lever 1** — lever 1 tested the *coherence attribute*; this enumerated the *accesses* and showed none is on the decode path. Complementary, not a re-proposal, and it *explains* lever 1's null: an attribute cannot repair a read that never happens per token. | -| runtime **AQL submission policy** | **adjacent to lever 4** — lever 4 forced a barrier on every packet (a change to *output*); this read the *policy* and found the runtime does not infer hazards at all. Different question, same subsystem. | -| **graph reorder** (`GGML_HRX_GRAPH_ORDER`) | **new** — not on the recorded list | -| **deferred host writeback** | **new** — not on the recorded list | -| **memory-pool quarantine/reuse** | **new** — not on the recorded list | -| **FA alignment/dispatch** (`GGML_HRX_FA_PARTIAL_ALIGN`) | **adjacent to lever 5** — lever 5 disabled FA entirely (`-fa off`); this varied the *partial alignment* with FA on, which changes which FA kernel is selected. Related but distinct: a reader could reasonably ask "isn't that the same lever?", and the answer is no, with the distinction stated. | - -## The contract's question, answered - -*"confirm the claimed mechanism is not one of them"* — the claims this session makes are **exclusions**, not -mechanisms, and **none of them is a recorded lever**: - -- five are straightforwardly new (read path, graph reorder, writeback, pool reuse, and the AQL-policy read); -- two are **adjacent** to recorded levers (host side ↔ lever 1; FA alignment ↔ lever 5) and are distinguished - above, with the reason each is a different test rather than a re-run. - -And the one *fix* claim this session makes is a **non**-claim: no forward-only fix is landable, so there is no -proposed fix for the contract's "if it is, the fix is rejected" clause to bite on. - -## Also worth stating - -Lever 5 (`-fa off`) is the closest thing on the record to my FA-alignment exclusion, and the two together are -now a coherent pair: **disabling FA entirely does not stop the fault (lever 5), and neither does keeping FA on -while changing which FA kernel the dispatcher selects by lowering the partial alignment (this session)**. If -the fault lived in the FA path at all, at least one of those two should have moved the rate. diff --git a/dev/315/315-campaign/evidence/315-extra-env-comma-trap.md b/dev/315/315-campaign/evidence/315-extra-env-comma-trap.md deleted file mode 100644 index c6f2afcb8..000000000 --- a/dev/315/315-campaign/evidence/315-extra-env-comma-trap.md +++ /dev/null @@ -1,73 +0,0 @@ - - -# Instrument trap: `EXTRA_ENV` splits on commas, so only the FIRST fingerprint name survives - -Found while running task 2, and it invalidates an assumption several earlier runs rested on. - -## The trap - -`315-instr-capture.sh` passes environment for the arm like this: - -```bash -IFS=',' read -r -a _extra <<< "$EXTRA_ENV" -for kv in "${_extra[@]}"; do SETENV+=(--setenv="$kv"); done -``` - -`EXTRA_ENV` is a **comma-separated list of `KEY=VALUE` pairs**, so a comma inside a VALUE splits it. A name -list therefore cannot be passed: - -``` -EXTRA_ENV="GGML_HRX_TENSOR_FP=result_norm,result_output,l_out,ffn_out" -# becomes: --setenv=GGML_HRX_TENSOR_FP=result_norm -# --setenv=result_output (not a KEY=VALUE pair) -# --setenv=l_out -# --setenv=ffn_out -``` - -**Only the first name is ever set.** Everything after the first comma is passed as a malformed `--setenv`. - -## What it invalidated - -Several runs this session passed multi-name lists — `result_norm,result_output,l_out,ffn_out` and later -`ffn_moe_probs` — and the observation "only `result_norm` matched" was read as evidence that the other names -were layer-suffixed (`l_out-0`, `ffn_moe_up-4`). **That is only half true.** The suffix mismatch is real — -`hrx_fp_name_listed` matches exactly per comma-item — but the list never reached the instrument anyway: the -value was truncated at the first comma, so `result_output` was *also* never requested. - -Both facts point the same way for the runs that were done (`result_norm` is unsuffixed, so it matched -whether the list survived or not), so no measurement is retracted. But the *reason* recorded for the -`result_output` gap in `315-f32-twin-result-divergence-on-the-faulting-arm.md` — "the exact name does match -in other runs, so its absence in these two arms is unresolved" — is now resolved in the other direction: -**it was never passed.** - -## The rule - -* To fingerprint **one** tensor, pass `GGML_HRX_TENSOR_FP=` as the first `EXTRA_ENV` entry, with any - further options as *separate* comma-separated `KEY=VALUE` pairs. -* To fingerprint **several**, use a position selector (`tail:` / `head:`), which needs no list — but - note the position window must actually cover the tensor (measured: `tail:16` covers `attn_norm-*` and - **not** `ffn_moe_probs-*`). -* Never read "no records" as "zero" — it means the filter or the window missed, per - `315-PREREG-argsort-precondition.md`'s third reading. - -## What task 2 therefore ran - -`GGML_HRX_TENSOR_FP=ffn_moe_probs-0,GGML_HRX_BUFFER_TRACE=1` — one tensor, the layer the record documents as -the onset (§59: *"the corruption is already present at layer 0, in the layer-0 FFN or its input"*). If layer 0 -turns out not to be the onset in that arm, the reading is "no NaN at layer 0" and the layer sweep is a -separate run. diff --git a/dev/315/315-campaign/evidence/315-f32-sync-twin-preregistration.md b/dev/315/315-campaign/evidence/315-f32-sync-twin-preregistration.md deleted file mode 100644 index afa3c1eae..000000000 --- a/dev/315/315-campaign/evidence/315-f32-sync-twin-preregistration.md +++ /dev/null @@ -1,97 +0,0 @@ - - -# PRE-REGISTRATION: the f32 read-path sync twin, read before the fault is caught - -Written and committed **before** the hunt below produced a faulting arm, so the reading cannot be -fitted to the result. The hunt was already launched when this was written; what is pre-committed here -is the *interpretation*, not the run. - -## Why this measurement, and why now - -`315-mechanism-refuted-on-ids-family-and-the-precise-gap.md` identifies the precise gap: - -> The sync twin is implemented **only in the i32 branch** of the probe… **So the f32 tensors — -> `l_out`, `node_2948`, `result_output` — have no sync twin at all**, and those are precisely the -> tensors that carry the NaN and the 256-byte-granular corruption. The record contains 1.16M pairs for -> the family that never corrupted and **zero** for the families that did. - -The twin that produced the record's "the two read paths agree" negative compares, for the same tensor at -the same call: - -1. the plain `ggml_backend_tensor_get` memcpy — what a host-visible buffer gets, **no iface call and no - synchronisation**; -2. the same bytes through `hrx_synchronous_d2h`. - -That comparison has never been taken on any f32 tensor. `patch-input-f32-sync-twin.py` adds it with a -**full-content hash** rather than only a NaN count, so a divergence confined to a tensor's interior is -visible. - -## What was built and what is being run - -* Source: `~/wt/llama-315copy` at pin `e44c9d01a`, patch `patch-input-f32-sync-twin.py` applied - (anchors all matched; `hrx-f32-sync` = 1, `head_n` = 5 in the .cpp). Backup of the pre-patch file at - `~/wt/ggml-hrx.cpp.bak-pre-f32twin`. -* Build: `~/wt/315fp-build`, `make -j16`, rc=0, marker verified present in `bin/libggml-hrx.so.0` - (`hrx-f32-sync` = 1) — an uninstrumented binary would silently measure nothing. -* Emitted line: `[hrx-f32-sync] call=… name=… bytes=… nan=… of … ne0=… first=… row=… hash=…H - memcpy_hash=…H nan_agree=0|1 hashes_equal=0|1`, where `hash` is the **synchronising** read and - `memcpy_hash` is the plain one. -* Run: `315-fp-hunt.sh <315fp-build/bin> 15 315f32` with - `EXTRA_ENV="GGML_HRX_TENSOR_FP=result_norm,result_output,l_out,ffn_out"` and `UB=512`, in-regime via - `315-instr-capture.sh`. It stops at the first arm with a fault on either face. - -**Instrument smoke test (clean, single short request), recorded because it fixes the baseline:** -`[hrx-f32-sync] call=94 name=result_output bytes=619520 nan=0 of 154880 ne0=154880 … hash=744e1b4b2f61b983 -memcpy_hash=744e1b4b2f61b983 nan_agree=1 hashes_equal=1` — 20 lines, every one agreeing. So on a clean -run the two paths are identical; any divergence found in a faulting arm is not baseline behaviour. - -## Pre-declared readings - -| observation on the **faulting** arm | reading | -|---|---| -| `hashes_equal=0` on a tensor that also carries NaN / the corruption | **The plain memcpy read returns different bytes than the synchronising read of the same tensor at the same call.** The read path into the engine is not ordered against the device's write. The memory-visibility family is live **on the f32 family that actually corrupts** — the one place it has never been tested. | -| `nan_agree=0` with `hashes_equal=0` | Same, and the disagreement includes the NaN extent itself: the engine is reading a materially different buffer than the synchronised view. | -| every line `hashes_equal=1 AND nan_agree=1`, including on the faulting arm | **The two read paths agree on the f32 tensors too.** The read-path/staging-visibility family is refuted on the family that matters, not only on the i32 family — and it closes for the same reason §46/§55/§56 closed it, now with the measurement that was missing. | -| no `hrx-f32-sync` lines at all on a faulting arm | The instrument did not reach the tensor (name mismatch or the call did not hit the printed path). **Not a result** — the arm is uninformative and must be re-run with `head:`/`tail:`. | - -**The stop rule is the hunt's own:** it stops at the first fault, so the faulting arm is whatever it -catches. No arm pruning, and a clean-only hunt (no fault within 15 arms) is recorded as -`315HUNT_DONE` — an uninformative outcome about the fault, **not** evidence for either reading above. - -## Declared confounds, and what each does and does not permit - -1. **The two reads are sequential**, so a divergence means the buffer *changed between them*. That is - either (a) a stale host cache line, or (b) the device writing the buffer concurrently with the host's - read. **This measurement cannot separate (a) from (b)** — but it does not need to for the claim it is - being used for: both are a read that is not ordered against the producer, and both are defects. What - it *would* refute is the claim that the read is safe. -2. **Arena reuse** (§53, §59) means a by-name read may return another tensor's bytes. That does **not** - affect this test: the twin compares two reads of the *same buffer at the same call*, so aliasing - changes what both reads return together. This is the one instrument in the record that is immune to - the aliasing objection. -3. **`UB=512`** is the issue's own high-rate configuration (11/12 on the record), used so a fault is - cheap to reach. If a divergence appears only under it, that is a statement about that configuration - and must be re-checked at the default `-ub` before being reported as the in-regime mechanism. -4. **Hash equality does not prove the reads are correct** — only that they agree. Agreement refutes - staleness; it does not certify the data. - -## The falsifier - -One line: a faulting arm in which every `[hrx-f32-sync]` record has `hashes_equal=1` and -`nan_agree=1` falsifies "the f32 read path is the mechanism". Conversely a single `hashes_equal=0` -record on a NaN-bearing tensor confirms it for that tensor. diff --git a/dev/315/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md b/dev/315/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md deleted file mode 100644 index 12246224b..000000000 --- a/dev/315/315-campaign/evidence/315-f32-twin-result-divergence-on-the-faulting-arm.md +++ /dev/null @@ -1,183 +0,0 @@ - - -# RESULT: the f32 read-path twin diverges on the faulting arm and only there - -Read against the pre-registration committed before the run -(`315-f32-sync-twin-preregistration.md`). The pre-declared reading for `hashes_equal=0` on a faulting arm -was: *"the read path into the engine is not ordered against the device's write"*. That is what was -observed. Two limits are recorded below and neither is smoothed over. - -## The measurement - -Build `~/wt/315fp-build` at pin `e44c9d01a` with `patch-input-f32-sync-twin.py` applied (marker verified -in `libggml-hrx.so.0`), in-regime via `315-instr-capture.sh`, `-ub 512`, -`GGML_HRX_TENSOR_FP=result_norm,result_output,l_out,ffn_out`. - -Each record compares, for the same tensor at the same call: -`hash` = the same bytes through `hrx_synchronous_d2h`; `memcpy_hash` = the plain -`ggml_backend_tensor_get` memcpy that a host-visible buffer gets (no iface call, no synchronisation). - -| arm | verdict | gtt_start | f32 records | **`hashes_equal=0`** | tensors with records | -|---|---|---:|---:|---:|---| -| `315f32-1` | **clean** | 110G | 554 | **0** | result_norm | -| `315f32-2` | **FAULT** (`HRX returned NaN logits: count=154880 of 154880, contiguous=yes`) | 110G | 426 (1278 by name) | **42** | result_norm only | - -**Zero divergence in 554 clean-arm records; 42 in the faulting arm, every one on `result_norm`.** This -is the first time this twin has been taken on any f32 tensor — the record had 1,160,376 paired reads for -the i32 family and **zero** for the f32 family that carries the corruption. - -A representative divergent record, verbatim: - -``` -[hrx-f32-sync] call=36574 name=result_norm bytes=8192 nan=0 of 2048 ne0=2048 first=-1 row=-1 - hash=9cb2244e4ae24383 memcpy_hash=8892a34c1b335845 nan_agree=1 hashes_equal=0 -``` - -`nan_agree=1` on all 42: both reads report the same NaN count (zero). **The disagreement is in finite -bytes, not in the NaN extent.** - -## Limit 1 — the divergence is NOT on the tensor that carries the NaN - -`result_output` is the logits tensor (619,520 B = 154,880 floats), and it is exactly the tensor the -guard reports as wholly NaN. **It produced zero fingerprint records in the faulting arm.** `l_out` and -`ffn_out` produced zero as well — only `result_norm` matched, 1278 records. So: - -* the twin fired on `result_norm`, one tensor upstream of the logits; -* the logits read itself was **not observed** in this arm, so this result says nothing directly about - the read of the tensor that corrupts. - -`result_output` *does* match the name list — the clean smoke test produced `name=result_output -bytes=619520` records — so its absence here is an arm-specific fact (the guard logs `continuing=no`, the -server aborts at the fault) and is recorded as an **instrument gap in this arm**, not as agreement. - -### The gap, diagnosed from the record afterwards — and the fix - -`result_output` is reachable by the instrument, and the record already contains a whole-tensor-NaN -reading of it. In a run that selected nodes by **position** rather than by name (`tail:`, which -bypasses the name filter — the probe's own note says position is "the only way to walk the head-end -pipeline without guessing tensor names"): - -``` -[hrx-tensor-fp] call=103644 name=result_output op=MUL_MAT bytes=619520 nan=154880 of 154880 ne0=154880 first=0 row=0 -[hrx-finite-fp] call=103644 name=result_output finite=0 nan=154880 -``` - -So the logits tensor is **wholly NaN, `finite=0`**, which is what the guard reports — and the probe *can* -see it. Two consequences for the next run: - -1. **Use `tail:` (or `head:`), not the name list, for the logits.** The name filter is an exact - match per comma-separated item (`hrx_fp_name_listed`, `ggml-hrx.cpp:604-621`), while the graph's - layer tensors carry a `-` suffix — the record's own fingers show `ffn_moe_topk-9`, - `l_out-0`, `ffn_moe_weighted-46`. That fully explains why `l_out` and `ffn_out` produced nothing here; - for `result_output` the exact name does match in other runs, so its absence in these two arms is not - explained by the filter and is stated as unresolved rather than guessed at. -2. **The logits reading must be taken in a faulting arm.** The record's `result_output` line above is - from an arm whose `nan=154880 of 154880` matches the guard, but it is a *post-graph by-name* read and - therefore carries §53/§59's aliasing caveat. The f32 twin's own comparison is immune to that caveat - (both reads target the same buffer at the same call), which is why extending the twin to the logits - node by position — rather than adding another by-name read — is what the next instrument change should do. - -## Limit 2 — the two candidate causes are still not separated - -The two reads are **sequential**, so a divergence means the buffer changed between them: - -* **(a)** a stale host cache line on the plain-memcpy path — the path is the difference; or -* **(b)** the device writing `result_norm` while the host reads it — the buffer is the difference. - -The pre-registration declared this test cannot separate them, and it does not. Both would be a read not -ordered against its producer, but only (a) is a memory-visibility defect. - -### A code fact that bears on the split, read after the run - -The fingerprint does **not** read an in-flight buffer. `ggml-hrx.cpp`, in `graph_compute`, gates the whole -fingerprint on a successful execution and then synchronises **before** any read: - -```cpp -if (result.success()) { - // engine#315: order the fingerprint's own reads. ... without this the per-node lines are not a - // snapshot of one instant and the per-node attribution is invalid. - ggml_backend_synchronize(backend); - const char * want = std::getenv("GGML_HRX_TENSOR_FP"); -``` - -So both reads are taken with the device drained **as far as `ggml_backend_synchronize` drains it**, which -makes cause (b) — a writer racing the reader *between the two reads* — materially less likely than the -pre-registration allowed for. It does not eliminate it: whether HRX's `backend_synchronize` is a full -stream drain is exactly what sections 55-56 argue at length for the *consumer's* path, and this note does -not re-derive it for the probe's. The read-twice control remains the arbiter, and it is already running. - -It also sharpens what a divergence would mean. `result_norm` is read through the host-visible path: the -probe's own comment records that the tensors it names on the MoE path live in the host-visible -**non-coherent** arena (44048640 bytes, from the buffer trace), which is the memory type where a missing -invalidate or flush is a defect rather than a no-op. - -**The control that separates them already exists and is not yet applied**: `patch-f32-read-twice-control.py` -(commit `71f4101`) adds a second plain memcpy immediately after the first and before the sync read, -reporting `memcpy2_hash` / `memcpy_stable`. Reading: `memcpy#1 == memcpy#2` and `!= sync` ⇒ (a); -`memcpy#1 != memcpy#2` ⇒ (b). It is validated (anchors matched against the live source; the patched TU -passes `-fsyntax-only` under the build's own flags) and was deliberately held back so this run's -pre-registration would not be mixed with a changed instrument. - -## What this does and does not establish - -* **Establishes:** on this box, in this regime, the faulting state and f32 read-path divergence coincide - exactly in the two arms measured (0/554 vs 42/426), on a tensor the engine reads through the - unsynchronised memcpy path. The earlier "the two read paths agree" negative is a statement about the - **i32** family; it does not transfer to f32, and the record never claimed it did — this is the gap it - named. -* **Does not establish:** that the divergence is staleness rather than a read racing the writer (Limit 2), - that it occurs on the logits tensor (Limit 1), or that it is causal rather than co-symptomatic. Two - arms is two arms; the clean/faulting split is clean but n = 2. - -## The buffer the twin reads has a memory type that varies between arms - -Taken from the 71 arm logs that carry a buffer trace (`GGML_HRX_BUFFER_TRACE`), every distinct -host-visible allocation: - -``` -type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=44048640 -type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x52 size=619520 -type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x56 size=44048640 -type=HRX0_HOST host_visible=1 direct_host_binding=0 memory_type=0x56 size=619520 -type=HRX0_HOST host_visible=1 direct_host_binding=1 memory_type=0x56 size=44048640 -type=HRX0_HOST host_visible=1 direct_host_binding=1 memory_type=0x56 size=619520 -``` - -counts: **44 allocations at `0x52`** (non-coherent: `HOST_LOCAL|DEVICE_VISIBLE`) against **96 at `0x56`** -(coherent: the same plus `HOST_COHERENT`). The two sizes are the logits buffer (619520 B = 154880 f32) -and the arena the probe's comment names (44048640 B). - -Two consequences, both tightening the next run: - -1. **`result_norm` is not its own host-visible allocation.** No traced host-visible buffer is 8192 bytes - (its `ggml_nbytes`), so it is a view or slice inside one of these two buffers rather than a buffer of - its own. What the twin compares is therefore two reads of a region whose **memory type is not fixed - across arms** — and neither of my two arms had the trace enabled, so which type `result_norm`'s region - carried is **unknown**, not assumed. -2. **The next run must enable `GGML_HRX_BUFFER_TRACE`**, so a divergence is tied to a measured memory - type rather than to the probe's comment. That also makes a clean two-cell comparison available that - has not been run: the same f32 twin against a `0x52` region and against a `0x56` region. - -This does not revive lever 1. Section 13's falsification stands where it was made — the **output** region -was measured coherent (`hv56=1`) in the arm that faulted, so coherence there is not a sufficient cause of -the *fault*. What is newly visible is narrower and different: the tensor the twin diverges on is not the -output region, and its memory type was never recorded in these two arms. - -**Next run, in order:** (1) apply the read-twice control and re-run in-regime until the same fault, to -settle Limit 2; (2) fix the `result_output` gap (use `tail:` rather than the name list, so the logits -node is covered even when the arm aborts) so the logits read is observed in the faulting arm. diff --git a/dev/315/315-campaign/evidence/315-fa-masked-v-regime.md b/dev/315/315-campaign/evidence/315-fa-masked-v-regime.md deleted file mode 100644 index 6d1f70a88..000000000 --- a/dev/315/315-campaign/evidence/315-fa-masked-v-regime.md +++ /dev/null @@ -1,72 +0,0 @@ - - - -## The extended masked-V kernel check, 40 cases, regime shapes - -``` -# engine#315 masked-V kernel check, run 2026-10-09T22:59:16-03:00 on strixhalo -# fork branch engine-315/fa-masked-v-nonfinite (c2114574d, ec0175971) built on 26330cc20; ggml/src/ggml-hrx is identical to the pin e44c9d01a at the time of the run -# cmd: IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 LD_LIBRARY_PATH=/home/bcloud/wt/v315-build/bin ./bin/test-hrx-fa-masked-v -# gtt in use during the run: 103 GiB (a foreign repro336 session held the box) -ok case 0 q=1 visible=6 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.6e-07 -ok case 1 q=1 visible=70 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 2.57e-06 -ok case 2 q=16 visible=30 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 9.57e-07 -ok case 3 q=2 visible=13 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 5.57e-07 -ok case 4 q=4 visible=81 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 2.26e-06 -ok case 5 q=25 visible=43 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.2e-06 -ok case 6 q=3 visible=20 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 7.19e-07 -ok case 7 q=7 visible=92 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 2.26e-06 -ok case 8 q=34 visible=56 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.38e-06 -ok case 9 q=4 visible=27 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 8.64e-07 -ok case 10 q=10 visible=103 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 2.6e-06 -ok case 11 q=43 visible=69 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.51e-06 -ok case 12 q=5 visible=34 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 9.74e-07 -ok case 13 q=13 visible=114 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 2.84e-06 -ok case 14 q=52 visible=82 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.81e-06 -ok case 15 q=6 visible=41 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.39e-06 -ok case 16 q=1 visible=125 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.16e-06 -ok case 17 q=61 visible=95 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.94e-06 -ok case 18 q=7 visible=48 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.32e-06 -ok case 19 q=4 visible=136 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.37e-06 -ok case 20 q=70 visible=108 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 8.03e-07 -ok case 21 q=8 visible=55 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.53e-06 -ok case 22 q=7 visible=147 cells=256 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.54e-06 -ok case 23 q=79 visible=121 cells=256 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 8.18e-07 -ok case 24 q=1 visible=150 cells=200 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.77e-06 -ok case 25 q=24 visible=150 cells=200 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.3e-06 -ok case 26 q=40 visible=40 cells=512 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 6.73e-07 -ok case 27 q=1 visible=10 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 5.03e-07 -ok case 28 q=1 visible=172 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 4.9e-06 -ok case 29 q=1 visible=810 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 6.94e-06 -ok case 30 q=1 visible=11 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 4.53e-07 -ok case 31 q=1 visible=173 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.78e-06 -ok case 32 q=1 visible=811 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 7.57e-06 -ok case 33 q=1 visible=150 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 3.86e-06 -ok case 34 q=1 visible=2047 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 9.89e-06 -ok case 35 q=2 visible=810 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.38e-05 -ok case 36 q=4 visible=868 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.4e-05 -ok case 37 q=7 visible=512 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 9.61e-06 -ok case 38 q=15 visible=810 cells=2048 (loom_libs:ggml_flash_attention_decode_split_f32_f16_wmma_next_q8): masked-row variants differing 0/8, non-finite outputs 0, nmse 1.23e-05 -ok case 39 q=16 visible=512 cells=2048 (loom_libs:ggml_flash_attention_f32_f16_wmma): masked-row variants differing 0/8, non-finite outputs 0, nmse 8.54e-06 -test-hrx-fa-masked-v: 40 cases passed -# exit=0 -``` diff --git a/dev/315/315-campaign/evidence/315-fa-race-pin-sweep.md b/dev/315/315-campaign/evidence/315-fa-race-pin-sweep.md deleted file mode 100644 index a9da7fa3f..000000000 --- a/dev/315/315-campaign/evidence/315-fa-race-pin-sweep.md +++ /dev/null @@ -1,52 +0,0 @@ - - - -# FA-race sweep against the pin (`~/wt/315fp-build`, e44c9d01a) - -`fa-race ` repeats one prefill `FLASH_ATTN_EXT` on HRX0 with identical inputs and -counts runs whose output differs from run 0, grouped by `(query tile, head, 16-channel block)`. - -``` -T cells visible reps result -1 2048 810 250 checksum=-7.353318 bad_runs=0 -2 2048 810 250 checksum=-7.709488 bad_runs=0 -8 2048 810 250 checksum=35.336398 bad_runs=0 -16 2048 810 250 checksum=41.561140 bad_runs=0 -190 2048 810 250 checksum=146.350555 bad_runs=0 -1 4864 4798 250 checksum=-2.715973 bad_runs=0 -16 4864 4798 250 checksum=14.680580 bad_runs=0 -190 4864 4798 250 checksum=1.190266 bad_runs=0 -192 4864 4798 250 checksum=-5.256695 bad_runs=0 -SWEEP-DONE -``` - -2250 reps, nine shapes, zero bad runs. `T=190 cells=4864 visible=4798` reproduces the October checksum -(`1.190266`) exactly, so the same path is exercised - and where the October logs recorded `bad_runs` 2-11 per -250 on the same shape, the pin records 0. - -For contrast, the October logs: `fa-race1.log` `bad_runs=5` at that shape with lines like -`run 14: 256 differ (112 nonfinite): [qtile 0 head 9 ch 96..111: 256]`; `h336.log` an A/B of -`bad_runs=0 0 0 0` (h336 libraries) against `4 11 4 2` (control) at the same checksum; `e2e.log` 24/24 clean -model-level runs. Cached-kernel hashes in `bis1-4.log` (`md5sum jit190*/*.bin`) plus the instruction profiles -from `tools/fa-variant-profile.py` show the `b*/c*/e*/s*` tags are byte-identical kernels, so those logs' -differing `bad_runs` are not a code bisection. diff --git a/dev/315/315-campaign/evidence/315-face-landscape.md b/dev/315/315-campaign/evidence/315-face-landscape.md deleted file mode 100644 index 3575fbee7..000000000 --- a/dev/315/315-campaign/evidence/315-face-landscape.md +++ /dev/null @@ -1,249 +0,0 @@ - - - -# The two faces across the 142 recorded arms - -## Method - -For every `~/wt/rt-det/*/` on strixhalo that has a `detector.json`: - -- `nan` = count of the guard's `HRX returned NaN logits` lines (Face A) -- `fb` = count of `Content-only` 500 lines (Face B) -- `idn` = `identical_distinct` = `len(set(outputs))` over 8 identical greedy requests sent with - `cache_prompt=false` and `temperature=0`, so **1 = deterministic** and >1 = nondeterminism -- `gtt` = the arm's `gtt_start` from `run.log`, banded at the pre-registered gate - -142 of the 509 arm directories have a `detector.json` (the rest predate it). - -## The two faces never co-occur - -| class | arms | -|---|---| -| Face A (NaN logits) | 23 | -| Face B (format 500) | 7 | -| **both** | **0** | -| fully clean (>=66 requests, every signal zero, `idn <= 1`) | 108 | - -No arm in the record has produced both faces. That is the strongest structural constraint available on any -mechanism that claims to unify them: at the observable level the two are mutually exclusive in 142 draws. - -## Face A travels with finite nondeterminism; Face B does not - -`identical_distinct` distribution: **1 in 135 arms, 2 in 3, 3 in 2, 4 in 2**. Every divergent arm (7/7) is a -Face-A arm, and every Face-A arm with `idn > 1` also has `nan=1`. So inside one arm the same 8 identical -greedy requests both produced NaN logits in some runs and produced *different* finite outputs in others: -the event is nondeterministic corruption, not a fixed wrong answer. No Face-B arm is divergent (0/7) - and -their `idn=1` is measured over the identical requests that completed before they aborted (`req` 13-47 means -6-7 of the 8 identical requests had run), so Face B's generation was *consistent* up to the failure. - -## Face B shows no dependence on the gate - -| gtt band | all arms | Face A | Face B | clean >=66 | -|---|---|---|---|---| -| 95-110 | 113 | 22 | 7 | 84 | -| <95 | 11 | 1 | 0 | 8 | -| >110 | 18 | 0 | 0 | 16 | - -Face B: 7/113 in-band versus 0/29 out-of-band (Fisher p ~ 0.3) - no evidence it tracks the gate. Face A: -22/113 in-band versus 1/29 out, and that single out-of-band event is not usable: - -**Caveat on `315nan-p20l` (gtt=44).** It faulted *after* I SIGKILLed the arm server of pair 19 in the same -session. A server dying mid-run can leave the following arm's HRX state inconsistent, so this event is -recorded as ambiguous rather than as evidence that Face A occurs outside the regime. - -## Face B is the server's own chat-template parse - -"Content-only" is the *parser name*, not a description of the model: `common/chat.cpp:812` returns that -string, and `tests/test-chat-peg-parser.cpp:180` has a "Content-only parser" case. So the 500 means the text -the model generated failed the parse the engine configured - a *text*-level failure. The 7 Face-B arms have -zero guard lines, but **none of them had the logits fingerprint enabled**, so "their logits were NaN-free" is -not established by these logs; that is what `315-faceb.sh` is for. - -## The run that produced most of these numbers - -`315nan` completed its 20-pair cap after the 8400 s wrapper timeout killed only the wrapper: final line -`1 clean fix-cell arms, 17 faulted-or-invalid in the fix cell, out of 21 arms run`. Section 41 cleared its -instruments of fabricating NaN by source inspection, so those are computed-NaN faults, but their *rate* -(16/17) remains instrument-inflated against the recorded 5.4%. - -## Tooling failure to record - -The attribution session (`bd4c969ee`) **took the box lock at 09:37:25 and then died** with -`315-attrib.sh: line 70: unexpected EOF while looking for matching '` - because an edited `315-attrib.sh` was -scp'd over the file that session was executing, and bash re-read it at stale byte offsets. No attribution -data exists. The failure was mine, not the box's, and the same trap must not be repeated while -`315-chain.sh` runs: new probes go in new filenames. - -## What actually precedes the 500: the parser rejecting the model's own degenerate output - -The line immediately before every Face-B 500 is the parser, and what it is rejecting is the model's own text: - -``` -1.47.161.803 W common_chat_peg_parse: unparsed Content-only output: a..........* it on ${... and: it itns it that'~>