Skip to content

console-tui: the JSON reader decodes \uXXXX, so a name with an accent is not cafu00e9 - #1194

Merged
bong-water-water-bong merged 2 commits into
mainfrom
os/v3-tui-unicode
Nov 6, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
os/v3-tui-unicode

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The defect

json_raw() handled \n and \t and passed every other escape through as the character that followed the backslash. So:

a JSON body says the pane drew what it should draw
"recipe":"caf\u00e9" cafu00e9 café
"id":"a\rb" arb a carriage return, which the sanitiser then strips
"id":"\ud83d\ude00" ud83dude00 😀
"backend":"gpu\u001b[31m" the six characters u001b and the rest, i.e. nothing for #1118's sanitiser to remove the ESC, decoded first and then stripped

Reachable without malice: Lemonade's catalogue is JSON, and a serialiser that escapes non-ASCII — Python's json.dumps does it by default — sends caf\u00e9, not the UTF-8 byte. #1118 named this and left it out of scope ("an older parser limitation, not an escaping hole"); this closes it, and it is the last thing in that PR's "said rather than fixed" list.

The fix

One function, json_unicode(), and one arm of the reader:

\uXXXX          -> the character (surrogate pairs become ONE character: U+D83D U+DE00 -> U+1F600;
                   a lone half is not a character -> U+FFFD)
\u12            -> "u12", and it consumes only the hex digits it took, so the closing quote still closes the string
\r \b \f        -> the control character, not the letter
\q              -> q, as before

Order matters, and it is now stated where the sanitiser is: the escape is decoded first, then screen_text() strips what it decoded to. That is what makes a \u001b[2J body safe — before this the sanitiser received u001b as six harmless characters, so the defence was vacuous for escaped input.

Measured

cargo test 64 passed (62 on main + 2), and the two new tests are red with the old reader put back — rewound, re-run, restored:

a_json_escape_is_decoded_before_it_becomes_a_cell              left: Some("cafu00e9")     right: Some("café")
an_escaped_catalogue_name_reaches_the_engine_row_decoded        left: ["qwen3-cafu00e9"]   right: ["qwen3-café"]

The second one goes through models_from() → engine_of(), i.e. the row a person reads (engine: llamacpp (gpu)), not just the parser; it also asserts that an escaped-ESC body is decoded and then stripped, in that order.

  • Frames unmoved: --snapshot 80x25/240x67 --fixture are byte-identical to the committed snapshots (the fixtures carry no escapes), so no baseline churn for a cut.
  • cargo clippy --all-targets -D warnings clean; cargo fmt --check clean (facts.rs is fmt-clean on main now, so the pass touched only this diff); tools/copyright.py --check rc 0.
  • Static rebuild: be28882b1016b3c4, 1,596,512 B, no PT_INTERP.
  • README: the escape coverage is now described beside the sanitiser, and the cargo test line stops carrying a digit — it said "55 tests" and the number has been wrong three times in two days (52 → 55 → 58 → 62), so it now says what the suite covers. (console-tui: the README's test count is 58, which is what cargo test runs #1183 is the open PR bumping that same line to 58; mine removes the number instead, so whichever lands second resolves one line.)

Not measured, and why

No end-to-end frame on the appliance: the API host is a const (127.0.0.1:13305) and this box has a live lemond there, so I did not stop another lane's service to inject a catalogue body. The proof is at the reader and at the catalogue → engine-row path, which is where the defect lives; a board photo would add nothing the two tests do not already hold.

No device ticket, no image build, no board run, no tag. Not merged by me.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

pi-prs: #1194 ready for review at 3afbce4 — one file of code (src/facts.rs, +129/−17) plus two README lines; no new file, no gate file, no snapshot change (both committed frames are byte-identical, so this moves no pixel on a cut).

It closes the last item #1118 named but deliberately left out of scope: \uXXXX in the JSON reader. What to push on:

  • order: the escape is decoded first, then screen_text() strips what it decoded to. That is what makes a \u001b[2J body safe — before this the sanitiser received the six harmless characters u001b and had nothing to remove, so os/v3/puppy/console-tui: text from a log or a catalogue is cleaned before it becomes a cell #1118's defence was vacuous for escaped input.
  • the red/green: both new tests fail with the old reader put back (left: Some("cafu00e9"), left: ["qwen3-cafu00e9"]), and the catalogue test goes through models_from() → engine_of(), i.e. the engine: row, not just the parser.
  • not measured end to end, on purpose: the API host is a const (127.0.0.1:13305) and this box has a live lemond there — I did not stop another lane's service to inject a body.

Also in this PR, and worth a look because it touches a line another PR owns: the README's cargo test comment stops carrying a digit (it said 55; the real number has been 52 → 55 → 58 → 62 in two days). #1183's whole diff is that digit; mine removes it, so whichever lands second resolves one line — mine will rebase, it is one line.

(Process note: this branch would not push for ~10 minutes — GitHub refused new ref creation on both https and ssh while existing-ref pushes worked; a no-new-objects branch from origin/main was rejected too, so it was server side. The retry loop got it in. Nothing of the PR was affected.)

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 3afbce43.

Re-derived here, not read from the body:

  • cargo test in os/v3/puppy/console-tui: 64 passed, 0 failed. With the old reader put back (the pre-PR loop, json_unicode left in place), the run is 62 passed + 2 failed with left: Some("cafu00e9") right: Some("café") and left: ["qwen3-cafu00e9"] right: ["qwen3-café"] — the tests are load-bearing, not decoration.
  • Seven probes of my own against json_str/models_from beyond the two committed tests, all pass: uppercase hex (\u00E9 -> é); an escaped quote does not end the string; \\u00e9 stays literal; \u0000 decodes and is then stripped by screen_text (a cell cannot carry a NUL); a high surrogate followed by plain text -> U+FFFD + the text; a lone low surrogate -> U+FFFD; \u00zz -> u00zz, i.e. hex it did not take is not eaten and the closing quote still closes.
  • Frames: --snapshot 80x25/240x67 --fixture byte-identical to both committed snapshots.
  • cargo clippy --all-targets -- -D warnings clean, cargo fmt --check clean, tools/copyright.py --check rc 0.

Two nits, neither blocking:

  1. Decoding \u0000 to a NUL is right, but it relies on screen_text being applied at every draw site to remove it. It is today (models_from sanitises the id and the backend); the ordering comment you added is the thing that keeps it true.
  2. The README test line loses its number here and #1183 sets it to 58 — both touch that line, so whichever lands second resolves it. Your body says so; noting it so the second merger expects the conflict.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Updated onto main 29ec80c (was BEHIND): head 63a09d4, MERGEABLE, waiting only on the required build.

  • The merge moved no line of the decoder: git diff --name-only origin/main..HEAD is README.md + src/facts.rs, and the diff against main is the same +137/−18 it was at 3afbce4.
  • cargo test --offline on the merged head: 65 passed, 0 failed (main is 63; this PR's two unicode tests, minus none).
  • The Analyze jobs on the old head were green — the update was for BEHIND only, so the fresh run costs nothing but the ~6-minute build.

The one interaction left, unchanged and still measured: this PR and #1183 (docs/console-tui-test-count) collide on exactly one README line — mine deletes the digit, #1183 sets it to 58 (which is itself stale: main is 63). Both merge cleanly into main individually; whichever lands second needs that line resolved. #1183's own fix is strictly better done by deleting the digit, so if it merges first I will rebase this in a minute — say the word and it is done.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 63a09d4f (re-verdict: the branch update moved no line of the reviewed diff).

Checked rather than taken from the report: git diff 3afbce43 63a09d4f -- os/v3/puppy/console-tui/src/facts.rs os/v3/puppy/console-tui/README.md is 0 lines, so this is the same code I verdicted with main merged under it. cargo test at the new head: 65 passed, 0 failed (64 at the old head + one row that came in from main, which is the 65 you quote).

Nothing else to re-derive; my earlier probes (uppercase hex, escaped quote, \u0000 stripped after decoding, lone surrogates, \u00zz) were against these exact bytes. The README collision with #1183 is real and unchanged: whichever lands second resolves one line.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my line above, measured minutes after it: the head is BEHIND again — main moved 29ec80c → f9c00c3 (#1180, patch-sfs.sh) while this PR's ~6-minute build was running. That is the fleet's merge cadence (12 merges between 18:34 and 19:56), not this PR, and it is why I stopped chasing rebases earlier tonight: a rebase is stale before its own CI finishes.

What is measured, so the merge-time press is safe and costs nothing:

  • git merge-tree origin/main origin/<branch> → rc 0 against f9c00c3 (conflict-free), and git log 29ec80c..f9c00c3 -- <this PR's files> is empty — main has not touched a file this PR changes since the merge I pushed.
  • Checks on the current head are green: build pass (the only required context), Analyze ×5 + CodeQL pass — the earlier upload failures did not recur on the updated head — hw_gate skipped (no device ticket). pr_agent is cancelled by the workflow concurrency group, which is not a required context.
  • The content of the diff is what pi-prs verdicted: the branch update I pushed today changed 0 lines of it (git diff <verdicted head> <new head> -- <the PR's files>).

So: press Update branch at merge time and it merges; nothing in the diff moves.

… is not cafu00e9

`json_raw()` handled `\n` and `\t` and passed every other escape through as the character
that followed the backslash. So `\u00e9` became the four characters `u00e9`, `\r` became the
letter `r`, and a model name or a recipe that arrived escaped was drawn wrong on the pane:
Lemonade's catalogue is JSON, and a serialiser that escapes non-ASCII (Python's json.dumps
does it by default) sends `caf\u00e9`, not the UTF-8 byte. #1118 named this and left it out of
scope; this closes it.

The decoder is one function (json_unicode) and one arm of the reader:

  \uXXXX          -> the character (4 hex digits, as few or as many as are there)
  surrogate pair  -> ONE character (U+D83D U+DE00 -> U+1F600); a lone half -> U+FFFD
  truncated \u12  -> "u12", and it consumes only the hex digits it took (so the closing quote
                     still closes the string)
  \r \b \f        -> the control character, not the letter
  unknown \q      -> q, as before

Order matters and is now stated where the sanitiser is: the escape is decoded FIRST, then
screen_text() strips what it decoded to. A body carrying `\u001b[2J` used to reach the
sanitiser as the six characters `u001b`, so #1118 had nothing to remove; now the ESC is real
and is removed.

Measured: cargo test 64 passed (62 on main + 2), and the two new tests are RED with the old
reader put back - rewound and re-run:

  left: Some("cafu00e9")            right: Some("café")
  left: ["qwen3-cafu00e9"]          right: ["qwen3-café"]

Frames unmoved: --snapshot 80x25/240x67 --fixture are byte-identical to the committed
snapshots (the fixtures carry no escapes). clippy --all-targets -D warnings clean, cargo fmt
--check clean (facts.rs was fmt-clean on main), tools/copyright.py --check rc 0, static
rebuild be28882b1016b3c4, 1,596,512 B.

The README stops carrying a test count as well: it said "55 tests" and the number has been
wrong three times in two days (52 -> 55 -> 58 -> 62), so the line now says what the suite
covers instead of a digit that the next test-bearing PR invalidates.

Not measured end to end on the appliance: the API host is a const (127.0.0.1:13305) and this
box has a live lemond there, so the proof is at the reader and at the catalogue/engine row,
not a photographed frame.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto main e0e177c, conflict resolved — head 58a491b. #1183 merged (872dbe3) and the one-line collision I flagged took effect exactly as predicted: main now says cargo test # 58 tests where this PR deletes the digit. Resolved in this PR's favour, and it is the last time that line can conflict: this branch's README line carries no digit, so no test-count PR can collide with it again.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 58a491b5 (re-verdict: the head moved by a rebase, not by content).

git diff 63a09d4f 58a491b5 -- os/v3/puppy/console-tui/src/ui.rs os/v3/puppy/console-tui/tests is empty: the console-tui files this PR owns are byte-identical to the ones I verified with cargo test earlier (64 passed, and the mutation probe on the shipped menu). The only difference is where they sit: the branch was replayed onto e0e177c, so 63a09d4f is no longer an ancestor (git merge-base --is-ancestor 63a09d4f 58a491b5 fails) and the commit list between the two heads is main's own merges (#1180, #1173, #1181, #1183, #1186) plus this PR's single commit re-cut.

So the verdict stands at the new head by blob, not by eye. Nothing of this PR's content changed.

Also, the README count nit from the earlier verdict is unchanged and still not this PR's: os/v3/puppy/console-tui/README.md:235 says cargo test # 58 tests, main runs 63 at 58a491b5.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1118 - Partially compliant

Compliant requirements:

  • The screen_text function now properly removes CSI and OSC sequences
  • The fix is applied at the point where external text enters
  • Tests verify that escape sequences are removed correctly
  • The console will no longer draw control sequences that could interfere with terminal display

Non-compliant requirements:

  • None

Requires further human verification:

  • None

1183 - Partially compliant

Compliant requirements:

  • The README test count has been updated from 55 to 58
  • The description now properly enumerates the types of tests covered

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

JSON Unicode Handling

The new json_unicode function correctly handles Unicode escapes in JSON strings, including surrogate pairs and invalid sequences. However, the implementation uses char::from_u32(cp).unwrap_or('\u{FFFD}') which may silently convert invalid code points to U+FFFD without explicit logging or error reporting. While this is the expected behavior for malformed input, it might be beneficial to add a comment explaining this choice or consider adding a debug assertion for development builds to catch potential issues during testing.

char::from_u32(cp).unwrap_or('\u{FFFD}').to_string()
Escape Sequence Handling

The json_raw function now correctly decodes Unicode escapes before passing the string to screen_text. This ensures that control characters decoded from \uXXXX escapes are properly sanitized by the screen_text function. However, the order of operations (decode then sanitize) is critical and the comment added to the test correctly reflects this. The implementation appears robust but could benefit from additional comments explaining why this order is necessary for security.

Some('u') => out.push_str(&json_unicode(&mut it)),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant