Repository navigation
lab-ops-hub: Worker-fallback extractor diverges from container Extractor — bare <a> uncounted, script-string <a counted, title entities raw, style/title leak into text #569
Description
Problem
POST /api/crawl returns the same JSON contract from two extractors — the container's Python Extractor (pkgs/lab-ops-hub/container/app.py:388-415) and the Worker fallback's regexes (crawlFallback, pkgs/lab-ops-hub/src/index.ts:196-205) — but on identical HTML the two produce different title, links and text/excerpt. Which values a crawl returns (and archives to R2 / D1) therefore depends on whether the container happened to be available, contrary to the README's "Worker-side fallback crawler with the same JSON contract" (pkgs/lab-ops-hub/README.md:14).
Verified this run by feeding identical HTML to the real Python Extractor and to the fallback's exact regex expressions:
- Bare
<a>anchors are not counted by the fallback. The fallback countshtml.match(/<a\s/gi)— an<a>with no attributes has no whitespace after the tag name. Sample<a>only bare anchor</a>: containerlinks=1, fallbacklinks=0. <ainside a<script>string is counted by the fallback. Sample<script>var s = "<a href='/fake'>x</a>";</script>: containerlinks=0(HTMLParser treats script content as data), fallbacklinks=1(the<script>removal only applies to thetextextraction, not to the earlierlinkscount on the raw HTML).- Title entities are not decoded by the fallback.
<title>Tom & Jerry — Best</title>: container returnsTom & Jerry — Best(HTMLParser decodes character references), fallback returns the rawTom & Jerry — Best. - Title and
<style>content leak into the fallback'stext/excerpt. The container keeps title text out oftextand skips<script>/<style>content entirely (_skipcounter). The fallback strips<script>blocks but not<style>, and its tag-stripping leaves the title's text in place. Sample with<title>T</title><style>.x { color: red; }</style>: containertext='only bare anchor', fallbacktext='T .x { color: red; } only bare anchor'.
Why it matters
The crawl result is archived verbatim (crawls/<ts>.json in R2, row in D1 crawls), so the same URL crawled twice — once per path — yields inconsistent records, and the fallback path (used exactly when the container is unhealthy, i.e. when reliability matters most) returns visibly wrong excerpts (raw entities, CSS source) and link counts.
Suggested fix / acceptance criteria
- Make the fallback extraction match the container's semantics for the cases above: count
<a>start-tags regardless of attributes, never count markup inside<script>/<style>, decode HTML entities in the title (and text), and exclude title + script + style content fromtext. - The natural home is a pure helper in
src/lib.ts(likecrawlFetcher, lab-ops-hub: the #500 crawl-timeout fix has no regression test — crawlFallback's fetcher is inline in the untestable Worker entry #524) so it is unit-testable with plain Node;crawlFallbackinsrc/index.tscannot be imported by tests today because the module imports Cloudflare bindings. - Add paired fixtures: the same HTML samples run through both extractors (extend
test/lib.test.tsandtest/test_container_validate.py-style parity, or a shared fixture file both suites read), asserting identicaltitle,linksandtextfor: bare<a>,<ainside a script string, entity-bearing title, and a<style>block. - Existing suites still pass:
npm testinpkgs/lab-ops-hub(TS + Python).
Acceptance criteria
- The Worker fallback's extraction lives in a pure, unit-testable helper and matches the container
Extractor's semantics for the filed cases: bare<a>anchors are counted; markup inside<script>/<style>is never counted; HTML entities are decoded in the title (and text); title, script and style content are excluded fromtext/excerpt. - Paired fixtures run the same HTML samples through both extractors and assert identical
title,linksandtextfor: a bare<a>, an<ainside a script string, an entity-bearing title, and a<style>block. - Existing lab-ops-hub suites still pass (
npm testinpkgs/lab-ops-hub, TS + Python).
Activity
dezren39 commentedon Oct 26, 2026
Intake review
crawlFetcherpure-helper precedent in the same package); containerExtractorinpkgs/lab-ops-hub/container/app.py.24261355):pkgs/lab-ops-hub/src/index.ts:203still counts links withhtml.match(/<a\s/gi)insidecrawlFallback(line 186), so all four filed divergences stand. The body's "Suggested fix / acceptance criteria" is formalized in the appended## Acceptance criteria.crawls/<ts>.jsonin R2, D1 rows). Parity fixtures must pin both extractors to the same samples.src/lib.ts(the lab-ops-hub: the #500 crawl-timeout fix has no regression test — crawlFallback's fetcher is inline in the untestable Worker entry #524 pattern), implement the container's semantics (count<astart-tags, skip script/style content, decode entities, exclude title/script/style from text), and add the paired fixtures before touchingcrawlFallback.dezren39 commentedon Oct 27, 2026
🤖 Agent note · filed
new-issue-creator-code-a)More container vs Worker-fallback divergences (verified this run, main
95cd41f0)Method: identical HTML fed to the real Python
Extractor(container/app.py:1240) and to the fallback's exact expressions (src/index.ts:202-209: title regex,<a\scount, script-strip + tag-strip). Log: goalhidden_files/probe-2026-10-27-laneA-diff/extract_probe.py.<p>real</p><!-- <a href='/x'>hidden</a> -->realreal hidden -->(anchor inside an HTML comment is counted, and comment residue leaks into text)<p>real</p><script>var secret = 'leaked';(unclosed)realreal var secret = 'leaked';(the<script[\s\S]*?</script>strip needs a closing tag; an unclosed script's body leaks)<title>a <b>c</b> d</title><p>body</p>a <b>c</b> d, text=body[^<]*cannot cross the ``), text=`a c d body`<title>First</title><title>Second</title><p>body</p>FirstSecondFirst, text=First Second body(second title's text also leaks into text)<textarea><a href='/x'>t</a></textarea><p>body</p><a href='/x'>t</a> body(textarea content is data)t bodyThe comment and textarea cases extend item 2's class (markup in non-markup contexts) beyond
<script>strings; the unclosed-script and title cases are new shapes for items 3/4. All five are covered by this issue's suggested fix (a real parser-based pure helper matching the container's semantics) — they only widen the fixture list its acceptance criteria should pin.dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
repo-worker-code)65 remaining items
dezren39 commentedon Nov 1, 2026
agent-note:v1 — task 4868 picked up by OpenChamber scheduled lane (2026-11-01). Fresh verification: #569 is OPEN and assigned to Drew; PR #3918 is OPEN, authored by Drew, at head 741b85a. The task’s implementation and tests are already on that PR; I will not touch its worktree, review, CI, or merge disposition. Parking until the PR merges/closes or ownership changes; revalidate before any action.
dezren39 commentedon Nov 1, 2026
agent-note:v1 — task 4868 parked by OpenChamber scheduled lane (2026-11-01). PR #3918 remains OPEN and Drew-authored at 741b85a; implementation/tests are already present. No PR/worktree, CI, review, or merge disposition was touched. Resume after the PR merges/closes or ownership changes, then revalidate. Labels reflect parked state; work/assigned remains set.
fix(lab-ops-hub): address Codex round-2 P2s on extractor parity (#569)
fix(lab-ops-hub): address Codex round-3 P2s on extractor parity (#569)
fix(lab-ops-hub): bound end-tag match to avoid quadratic backtracking (…
Merge pull request #3918 from developing-today/queue/4868-569-extract…
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
>, tolerant end tags in the normal path) fixed in a179aba; round 4 P1 (quadratic backtracking in the tolerant end-tag pattern, introduced in round 3's fix) fixed in edf21a6 — pathological input now 0.72 ms. Every new fixture's expected values were verified against the container Extractor before pinning; suites green at merge: node 85/85, python 65 OK. PR fix(lab-ops-hub): Worker-fallback extractor parity with container Extractor (#569) #3918 merged as bdc66c7.