[dzid26]

What

This is the lexer-provenance follow-up to #35. The preprocessor already parses calls into nodes and the compiler oracle already proves the generated .bep parses. This PR makes the TypeScript lexer itself checkable against the vendored Berry lexer and removes the last hand-synced lexical tables.

Changes

• Delete the one-time old-parser differential. The fix(webui): make the DBC preprocessor lexer-faithful #35 PR body records the 0-diff migration result; the C compiler and C lexer are the durable references.
• Share the compiler-oracle source list. firmware/tools/parse_check_sources.txt now feeds both parse_check_examples.sh and the Node oracle test.
• Add firmware/tools/lex_dump.c. A test-only harness dumps numeric token type, decoded value, and line from be_lexer.c / be_parser.c. It uses numeric types because be_tokentype2str is off by one for keywords.
• Generate the lexer tables. webui/scripts/gen-lexer-tables.ts probes the C harness for keywords, punctuation, and simple escapes, writes lexer-tables.ts, and writes lexer-source.lock. The lock hashes normalized text so Windows and Linux agree.
• Add lexer-conformance.test.ts. It compares the TypeScript token stream with the C stream over all real .be files and a valid edge-case corpus covering both comment forms, both quote styles, every escape family, adjacent literals, line endings, call vs grouping (, and all keywords/operators. It also asserts the TS lexer reports no errors where the C lexer accepts the input.
• Refactor lexer.ts. scanString is flattened into decodeEscape + unfinishedString; the hand tables and hexValue are gone. 1. is a real, 1..2 stays 1 .. 2, trailing malformed suffixes recover like C, and a leading ( is call-form (C resets had_whitespace before the first token).
• CI/docs. The webui workflow watches the harness, manifest, be_lexer.h, and berry_conf.h. The README documents both native gates and the regeneration command.

Verification

• npm test: 201 pass, including the compiler oracle, lexer conformance, and fuzz invariants.
• npm run check: 0 errors (2 pre-existing Svelte warnings).
• node webui/scripts/gen-lexer-tables.ts: idempotent; no diff after running.
• firmware/tools/parse_check_examples.sh test_scripts/bench_test.be: OK.

Limits

• F-strings stay opaque in the TypeScript lexer and are excluded from token conformance; calls inside interpolation remain a documented limitation.
• \xNN values above ASCII are byte strings in C but JS code points in TS; the conformance corpus keeps escaped values ASCII. DBC names are ASCII.
• The conformance test skips locally without gcc and fails in CI.

PR 2 will follow for the policy layer: move the argument-shape helpers to calls.ts, add the top-level/wholeCall argument view, and extract the variable tracker to berry/tracker.ts.