Repository navigation
Place: text files in UTF-16 without a byte-order mark place as UTF-16 - #7554
Merged
bflatastic merged 3 commits intoNov 14, 2026
Merged
Conversation
The Graph Data window's Import Data read UTF-16 only after a byte-order mark, so a UTF-16 file without one filled the cells with NUL characters. It now reads such a file in the byte order Import Data Sets' rule (storytold#7348) gives it: at least half of the two-byte units hold a NUL byte, and the byte order is the one that reads the line breaks. The text is then read as after a byte-order mark: an odd last byte is dropped and an unpaired surrogate reads as U+FFFD. A file the rule refuses is read as UTF-8, as before. The rule now also gives a byte order to a one-line file, which reads a line break in neither order: big-endian when more of its units start with a NUL byte than end with one, else little-endian as before. Import Data Sets still refuses a one-line file, which has no data rows; read in the new order, such a file can be refused with a different error first.
File › Place of a text file with the Unicode character set read UTF-16 only after a byte-order mark. Without one, UTF-16 text was read as UTF-8 (or the 8-bit set): characters outside U+0001 to U+00FF came out as other characters, and each CR LF placed as two paragraph breaks. Place now reads a file without a byte-order mark as UTF-16 by the rule Import Data Sets uses, through the shared byte-order helper: an even length, at least half of the two-byte units holding a NUL byte, the byte order that reads a line break (for one line, the order its NUL bytes give), and valid UTF-16 with no NUL character in that order. Any other file is read as before, as UTF-8 or else the platform's 8-bit set. Docs: docs/mcp.md, file.place params text, docs/target-app-parity.md.
Contributor
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 15 seconds. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Member
|
Merged, thanks @dotCooCoo! It passed its CI checks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #7522, which adds the shared byte-order helper; until it merges, this diff also shows its commit.
What and why
File › Place of a text file with the Unicode character set read UTF-16 only after a byte-order mark. A UTF-16 file without one was read as UTF-8 or the 8-bit set, so characters outside U+0001 to U+00FF came out as other characters and each CR LF placed as two paragraph breaks.
Place now reads such a file as UTF-16 by the rule Import Data Sets uses, through #7522's
utf16_order_without_mark: an even length, at least half of the two-byte units holding a NUL byte, the byte order that reads a line break (for one line, the order its NUL bytes give), and valid UTF-16 with no NUL character in that order. Any other file is read as before, as UTF-8 or else the platform's 8-bit set. This is VectorCraft's own detection rule. Files with a byte-order mark and the ANSI character set are read as before.Docs: the
file.placeparams text, the Place text paragraph indocs/mcp.md, and a line indocs/target-app-parity.md.How it was tested
tests::textimport::utf16_without_a_byte_order_mark_places_by_its_nul_bytes_and_line_breaks: both byte orders with CR LF, one-line files, a one-line file with NUL bytes in both positions, and the files that fall back (odd length, an unpaired surrogate, a NUL character, line breaks in both byte orders, too few NUL bytes in UTF-8 and in Windows-1252 text, and ANSI). It fails without the change, and it also fails with the NUL-character check removed.cargo test -p vectorcraft-engine --lib -- textimport datasets_csv: 39 passed.cargo test -p vectorcraft-engine: 21 test binaries, 7185 passed, 0 failed, 3 ignored.cargo clippy -p vectorcraft-engine --all-targets -- -D warnings: clean.cargo fmt --all -- --check: the only diff is incrates/svg/src/lib.rs, which this branch does not change. Graphs: Import Data reads UTF-16 text files without a byte-order mark #7522's base predates Fix CI formatting after craft-libs cutover #7454, which formats that file onmain.git merge-treewithmainatedb228ac33: clean.Checklist
cargo xtask cipasses (fmt, clippy, tests, assets, brands, layers, wasm).unwrap(),expect(),panic!,unreachable!,todo!orunimplemented!outside tests, and no unchecked indexing on data from files, parameters ormessages. Errors are returned as
Resultand shown to the user(why and how).
ASSETS.md.