Repository navigation
spec(hslm): nn/hslm_tokenizer and ml/hslm_corpus -- the 729-token BPE and a pinned TinyStories token file (Refs #10372) - #10380
Merged
Merged
Conversation
Contributor
…d ternary_mac discard fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
PR DashboardGenerated at: 2026-10-18 10:37:58 UTC
Summary
Seal Status
|
This was referenced Oct 18, 2026
Merged
Merged
This was referenced Oct 19, 2026
Merged
Merged
Merged
gHashTag
marked this pull request as ready for review
October 19, 2026 03:34
This was referenced Oct 19, 2026
gHashTag
added a commit
that referenced
this pull request
Oct 19, 2026
…, cap stays 129 (Closes #10837) (#10862) * t27b: a bare sqrt(x) with no fn sqrt declared lowers as @sqrt, as t27c's Zig backend prints it (Refs #10837) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * ledger(t27b): rows for hslm_tokenizer (pass) and hslm_corpus (blocked on host I/O), 11 sqrt specs flip to pass, cap 139 -> 129 (Closes #10837) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * t27b: @truncate by truncate_plan, which frees a ledger slot for #10380's two unlisted hslm specs (Closes #10837) #10380 merged specs/nn/hslm_tokenizer.t27 and specs/ml/hslm_corpus.t27 with no ledger rows, so every PR built on master got UNLISTED 2 from t27b-native-ratchet. hslm_corpus is blocked (cross-module use-resolution), and the ledger already sat at max_not_pass 139: its row would have been the 140th not-pass row, which reads as OVER CAP. The cap stays where it is. This change frees a slot instead. specs/tri/utils/utf8.t27 was blocked only by ExprCall(@truncate). @truncate now lowers through int_cast_plan.t27: - truncate_plan is plan in wrap mode, so it keeps the low bits and never traps; - a comptime operand folds with wrap; - a result type wider than the operand is refused (REFUSE_WIDER), as Zig refuses it; - from_destination now names @truncate, so `x = @truncate(e)` takes x's type. lower.rs is glue only, net 0 lines: int_cast takes a trunc flag that picks ic::truncate_plan and the @truncate site texts. gen/rust/tri/t27b/int_cast_plan.rs is regenerated by t27c gen-rust, and the seal is re-saved. Ledger: utf8 -> pass, hslm_tokenizer pass, hslm_corpus blocked. That leaves 138 blocked + 1 codegen = 139, at the cap, which is unchanged. The t27b-loop skill records four traps: the cap trap, the x86 lab without a qemu runner, the full /data volume, and old negative tests that flip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ledger: rows for six specs master merged unlisted, all pass under t27b (Refs #10837) mos_psp103, mos_psp103_data, sar_switch (specs/analog), idea_factory, particle_ds, petal_menu (specs/t27f) landed on master with no rows in docs/reports/t27b_expectations.json. t27b passes all six on the lab (qemu-aarch64 corpus on 535334d), so they cost no not-pass slot: the ledger stays at 139 not-pass rows, the cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * t27b-loop skill: three traps from consolidating #10853 into #10862 (Refs #10837) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * t27b ledger: pass rows for analog/sar_logic and policy/secret_gate (Refs #10837) Master added both specs after this branch's last ledger update; the native ratchet on 7ab095a reported them UNLISTED, with every other count at 0. Both pass under t27b, so the not-pass count stays 130. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * t27b ledger: cap 129 as master has it; eval_hslm and ngram_hslm go back to #10858 (Refs #10837) #10853 merged on its own with max_not_pass 129, the exact count, so master has no room for eval_hslm and ngram_hslm. With @truncate freeing utf8, keeping both rows would need 130 and raise master's cap. This branch drops them and takes master's 129: 128 not-pass rows, one slot free for #10858. The cap line now matches master's, so the branch merges without conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * t27b-loop skill: a master merge touching lefthook.yml is refused; match master and check merge-tree (Refs #10837) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #10372 (stays open for the training run). Part of #10024, cell 25.
What this adds
Two specs, nothing else (no foreign-language code, no generated files):
specs/nn/hslm_tokenizer.t27-- the HSLM byte-level BPE from gHashTag/trinity-training @ 3e9648c1817a97c5f9b8c727b0f8e2291928eead (src/hslm/tokenizer.zig,bpe_merges.zig). Vocab 729 = 4 specials + 256 bytes (offset 4) + 469 merges (ids 260..728). The merge table is const data converted frombpe_merges.zigwith sed/awk, nothing retyped.encodeanddecodekeep the Zig priority order (one full-sequence scan per rule, in index order) and its edge semantics (short output truncation, UNK -> '?', ids >= 729 ignored).specs/ml/hslm_corpus.t27--main()reads/tmp/hslm/valid.txtand/tmp/hslm/train.txtand writes/tmp/hslm/valid.tokand/tmp/hslm/train.tok. No heap, fixed buffers, streamed 64 KiB at a time. Read-back verification insidemain().Token file contract
File size is exactly 12 + 2N.
Story boundaries: a story is the text between two lines that are exactly
<|endoftext|>; each story is its tokens plus one EOS (id 2), no BOS; the newline before the marker is dropped; newlines inside a story are kept. This is deliberately NOT the Zig loader's cut (src/hslm/data.zigDataset.addText/loadTextFile: one line = one story, 2000-byte pieces each wrapped in BOS/EOS, newlines lost, marker tokenized as text). Documented in the spec header.Corpus (lab, Railway)
Token files (also in
/data/hslm/on the lab):Wall time of one
main()run (both files, ReleaseFast, 48-core lab, single thread): about 7.3 s.Tests
t27c test-report specs/nn/hslm_tokenizer.t27: 29 tests, 29 pass, 0 fail, 0 vacuous.t27c test-report specs/ml/hslm_corpus.t27: 25 tests, 25 pass, 0 fail, 0 vacuous.Tokenizer.encode/decodeitself on the lab (17 strings including "Once upon a time", a newline string, a non-ASCII UTF-8 string, the empty string,<|endoftext|>, control and high bytes; truncation and decode-into-short-buffer edges), plus structural checks over the whole table (layout, running sum, operands earlier, byte spelling, every token decodes, all 256 byte values round-trip) and a decode(encode(s)) == s round trip.valid.tok: 27630 stories, 0 mismatches, all 7689354 tokens accounted for.Exact lab commands
Not done / caveats
main()cannot take argv in the ml backend (host_openneeds a literal path), so the paths are fixed..tok.bodytemporary files are left behind empty (no unlink host call).tokenToTrits,tritsToToken) of the Zig tokenizer are not ported; encode and decode never use the former.🤖 Generated with Claude Code