Skip to content

Port gHashTag/trinity:src/vibeec/trinity_tokenizer.zig (Zig, 4 functions) to specs/port/trinity/src/vibeec/trinity_token - #8495

Merged
gHashTag merged 2 commits into
masterfrom
queen-8162
Oct 10, 2026
Merged

gHashTag merged 2 commits into
masterfrom
queen-8162

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Closes #8162

Written by a bee on queen-8162 and published by tools/queen/publish.py. The branch itself is the bee's; the second commit is the coordination entry every pull request must add, which a bee has no way to know about.

1 file changed, 160 insertions(+)

🤖 Generated with Claude Code

Trinity Bee and others added 2 commits October 10, 2026 02:59
…trinity/src/vibeec/trinity_tokenizer.t27

- Add Tokenizer struct with allocator field
- Port Tokenizer_init function to create tokenizer with allocator
- Port Tokenizer_tokenize function to convert text to normalized float tokens
- Port Tokenizer_detokenize function to convert tokens back to text
- Port main function with undefined body (plumbing)
- Add 5 test functions covering:
  - Basic tokenizer initialization
  - Tokenization with BOS/EOS tokens and normalization
  - Round-trip tokenization/detokenization
  - Special token handling (BOS, EOS, PAD, UNK)
  - Invalid character handling in detokenization

Closes #8162
A pull request must add exactly one docs/now entry and a bee has no way
to know that: its brief names a boundary file and acceptance criteria,
and docs/now/ is neither. The publisher adds it rather than failing the
gate.

Closes #8162

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-10 10:11:52 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 43
PRs with All Checks Green 7
READY 0
FAILING 43
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 43 + 0 + 0 = 43, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=bd3352ccb90e != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port gHashTag/trinity:src/vibeec/trinity_tokenizer.zig (Zig, 4 functions) to specs/port/trinity/src/vibeec/trinity_tokenizer.t27

2 participants