Repository navigation
lab-ops-hub: Worker-fallback extractor — complete HTML character-reference decoding parity #4007
Description
Problem
The lab-ops-hub Worker-fallback extractor (pkgs/lab-ops-hub/src/lib.ts, decodeHtmlEntities) only decodes a small named-entity table and requires a trailing semicolon, so its output diverges from the container Extractor (Python html.parser + html.unescape) on standard references — stored titles/text then depend on which crawling path ran.
Found by Codex review round 4 on PR #3918 (comment 4437512013), dispositioned there as a follow-up because the fix (complete HTML5 named-reference data + semicolon-omission rules) is larger than a review-fix round. Sibling of #569.
Evidence (verified against the container Extractor, Python 3.13.13)
Input: <title>Α &</title><p>→ A & B</p>
- Container: title
Α &, text→ A & B - Worker fallback: every reference preserved literally
Scope
- Decode the complete HTML5 named character references (the data behind Python's html.entities.html5), including the legacy semicolon-less forms and the spec's semicolon-omission rules for text content (e.g.
A->A,&->&in the cases html.unescape expands). - Keep the existing numeric-reference normalization (C1 remap, U+FFFD rules) intact.
- Add shared parity fixtures (test/extractor-fixtures.json) pinning the cases above plus a spread of named references; expected values verified against the container suite before pinning. Both suites (node + python) must stay green.
Acceptance
- The example above produces identical {title, links, text} from the Worker fallback and the container Extractor.
- New fixtures cover: a non-Latin-1 named reference (→), an uppercase legacy reference (&), a semicolon-less numeric reference, and a semicolon-less named reference.
Acceptance criteria
- The Worker-fallback extractor (
pkgs/lab-ops-hub/src/lib.ts,decodeHtmlEntities) decodes the complete HTML5 named character references, including legacy semicolon-less forms, with the spec's semicolon-omission rules for text content matching Pythonhtml.unescape. - Filed evidence case: input
<title>Α &</title><p>→ A & B</p>yields titleΑ &and text→ A & Bfrom the Worker fallback, identical to the container Extractor. - A differential test corpus (named references with/without semicolons, numeric references with/without semicolons, ambiguous ampersands in text and in attribute-like contexts) matches
html.unescapeoutput case for case. - Existing extractor test suites pass; no change to the container Extractor.
Metadata
Metadata
Assignees
Labels
Package/app: lab-ops-hub (pkgs/ or apps/ in this repo)Medium: expected under 1 day of workSomething isn't working (prefixed form of the classic bug label)Intake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Filed or worked by the queue/worker pilotQueue packet kind: bounded fix with reproduction and targeted tests
Activity
dezren39 commentedon Nov 1, 2026
Intake review
html.unescapesemantics for text content are the parity target.html.entities.html5), implement longest-prefix matching with the text-content omission rules, and pin parity with a generated differential corpus againsthtml.unescape.developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
52 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · picked up
Merge pull request #5147 from developing-today/queue/8754-4007-html5-…
dezren39 commentedon Nov 4, 2026
🤖 Agent note · done — fixed and merged
19aa97c2f52228b1f8f9b47ba2ee72d8f5e79f24; issue lab-ops-hub: Worker-fallback extractor — complete HTML character-reference decoding parity #4007 closed as completed. Removed queue/claimed, status/in-progress, and work/assigned labels.git diff --checkpass. Added generated coverage for 2,231 HTML5 entity keys, legacy suffixes, and numeric boundaries; retained mainline RCDATA/raw-text changes.