Repository navigation

Rust tokenizer and normalize unicode spaces is implemented - #164

Open

tkaunlaky-e6 wants to merge 7 commits into

e6data:mainfrom

tkaunlaky-e6:rust-tokenizer-unicode-optimization

Open

Rust tokenizer and normalize unicode spaces is implemented#164
tkaunlaky-e6 wants to merge 7 commits into
e6data:mainfrom
tkaunlaky-e6:rust-tokenizer-unicode-optimization

Commits on Sep 23, 2025
- Rust tokenizer and normalize unicode spaces is implemented
  tkaunlaky-e6
  committed
  30dd2a0
Commits on Sep 24, 2025
- Removed print statements
  tkaunlaky-e6
  committed
  8e86a66
- Added fallback to python as well
  tkaunlaky-e6
  committed
  40fef01
Commits on Sep 25, 2025
- Change fallback from parse_one to tokenizer itself
  tkaunlaky-e6
  committed
  4dff0c3
Commits on Sep 26, 2025
- Updated it to just be a feature flag
  tkaunlaky-e6
  committed
  f701ad2
- Added logs and gets from environment
  tkaunlaky-e6
  committed
  4f871c2
- Added error message instead of warning
  tkaunlaky-e6
  committed
  a9fe27c