fix(lab-ops-hub): Worker-fallback extractor parity with container Extractor (#569) by dezren39 · Pull Request #3918 · developing-today/code

dezren39 · 2026-11-01T18:29:08Z

Fixes #569 (queue task 4868).

The Worker fallback (crawlFallback) and the container crawler disagreed on extraction: the fallback's regexes missed bare <a>, counted anchors inside script strings and comments, returned title entities raw, and leaked style/title into text.

src/lib.ts: new extractHtml mirroring the container Extractor (Python HTMLParser) semantics — every <a> start tag counts (bare and self-closing included); script/style are CDATA-skipped (an unclosed one swallows the rest); <title> is raw character data with entities decoded and never leaks into text; comments/declarations are not markup; text chunks are space-joined and whitespace-collapsed exactly as app.py does.
src/index.ts: crawlFallback uses extractHtml. No status-code or output-limit change on either door.
Paired fixtures: one shared matrix, test/extractor-fixtures.json (17 cases: bare anchors, script/comment markup, entities, titles, styles, unclosed script/style, whitespace, truncation-adjacent chunking, quoted > in attributes), driven by both test/extractor.test.ts and test/test_extractor_parity.py so the two doors cannot drift again.

Tests (laptop): npm run test:ts 73 pass; npm run test:py 65 pass.

^{Need help on this PR? Tag @codesmith-bot with what you need.}

…ractor (#569) Add extractHtml to src/lib.ts mirroring the container Extractor (HTMLParser) semantics: every <a> start tag counts (bare/self-closing included), anchors in script/style/comments do not, title is raw character data with entities decoded and never leaks into text, script/style content is skipped (unclosed swallows the rest), text chunks are space-joined and whitespace-collapsed. crawlFallback now uses it; status codes and output limits unchanged. One shared fixture matrix (test/extractor-fixtures.json) drives both the TS and Python parity tests.

coderabbitai · 2026-11-01T18:29:17Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 7 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: fffaa95b-2503-4554-961a-b2795f8e4c1a

Commits

Reviewing files that changed from the base of the PR and between 33aefa7 and edf21a6.

Files selected for processing: 5

pkgs/lab-ops-hub/src/index.ts
pkgs/lab-ops-hub/src/lib.ts
pkgs/lab-ops-hub/test/extractor-fixtures.json
pkgs/lab-ops-hub/test/extractor.test.ts
pkgs/lab-ops-hub/test/test_extractor_parity.py

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

chatgpt-codex-connector · 2026-11-01T18:29:20Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T22:03:37.125543Z	`edf21a6`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 50ed061c3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- Self-closing detection now parses attributes: a slash ending an unquoted attribute value (src=/x/) is part of the value, so script/ style raw-text mode still engages, matching HTMLParser. - Numeric character references get HTML normalization: C1 controls 0x80-0x9F remap (e.g.  -> euro), NUL/surrogates/out-of-range become U+FFFD, matching html.unescape. - Comment scanner accepts --!> and the abrupt  forms instead of swallowing the rest of the document. - Six paired fixtures added to extractor-fixtures.json; TS suite 79/79 and Python parity suite 65/65 pass on the laptop.

dezren39 · 2026-11-01T20:30:37Z

@codex review

Fix round 741b85a pushed for the three P2 findings on 50ed061 (all threads replied + resolved): attribute-aware self-closing detection, HTML numeric character-reference normalization (C1 remap + U+FFFD), and --!> / comment terminators — six new paired fixtures; TS suite 79/79 and Python parity suite 65/65 green on the laptop. Please re-review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 741b85ac21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-01T20:35:17Z

+        if (quote !== "") {
+          if (ch === quote) quote = "";
+        } else if (ch === '"' || ch === "'") quote = ch;
+        else if (ch === ">") break;


Track attribute boundaries before opening quote mode

When an unquoted attribute value contains an apostrophe, the scanner treats it as the start of a quoted value and can consume the remaining document instead of recognizing subsequent tags. For example, <a href=/it's>link</a><p>body</p> returns the entire HTML as text with zero links, while the container Extractor under Python 3.13.13 returns link body with one link. The previous fallback also stripped this markup correctly. Only enter quote mode when beginning a quoted attribute value, rather than whenever a quote character appears.

Useful? React with 👍 / 👎.

@/tmp/reply1.md

chatgpt-codex-connector · 2026-11-01T20:35:17Z

+      const code = Number.parseInt(body.slice(hex ? 2 : 1), hex ? 16 : 10);
+      return String.fromCodePoint(normalizeNumericRef(code));
+    }
+    return NAMED_ENTITIES[body] ?? whole;


Restrict named-entity lookup to own properties

For unknown entity references whose names match inherited object properties, this lookup substitutes JavaScript functions into the extracted content. For example, <p>&constructor; &toString;</p> becomes function Object() { [native code] } function toString() { [native code] }, whereas both the previous fallback and the container Extractor under Python 3.13.13 preserve the literal references. Use an own-property check or a null-prototype entity map so unknown names remain unchanged.

Useful? React with 👍 / 👎.

@/tmp/reply2.md

chatgpt-codex-connector · 2026-11-01T20:35:17Z

+      continue;
+    }
+    if (next === "/") {
+      const m = /^<\/([a-zA-Z][^\s/>]*)\s*\/?>/.exec(html.slice(i));


Accept recoverable end tags before leaving raw-text mode

When a script or style closing tag contains trailing attributes, the raw-text scanner recognizes the closing name, but this stricter regex rejects the tag, so handleEndTag never clears skipDepth and the remaining article text is discarded. For example, <script>x</script foo><p>body</p> returns empty text here, whereas the container Extractor under Python 3.13.13 returns body; the previous fallback also preserved the body. Parse and consume these recoverable end tags consistently with the container so they actually terminate the skipped region.

Useful? React with 👍 / 👎.

@/tmp/reply3.md

Three parity divergences from the container Extractor, each with a shared fixture verified against the Python suite first: (1) tag scanner enters quote mode only where a quoted attribute value begins (right after =), so an apostrophe inside an unquoted value (<a href=/it-s>) no longer swallows the document; (2) named-entity decoding uses an own-property lookup, so &constructor; and &toString; stay literal instead of resolving to Object prototype functions; (3) raw-text mode consumes recoverable end tags with trailing attributes (</script foo>), so skipDepth clears and following text is kept. Both parity suites green: node 82/82, python 65 OK.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17347f07b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-01T21:42:50Z

+  while (k < len) {
+    while (k < len && /\s/.test(inner[k])) k += 1;
+    if (k >= len) return false;
+    if (inner[k] === "/") return /^\s*$/.test(inner.slice(k + 1));


Require the self-closing slash to immediately precede >

For a page containing <script / >var s='<a href=x>fake</a>';</script><p>real</p>, accepting whitespace after the slash incorrectly makes the script self-closing. The Worker consequently exposes script source in the extracted text and counts the fake anchor. Checked against the container's Extractor under Python 3.13.13, the result is instead text real with zero links; the previous Worker fallback also excluded this script source. Require the slash to be the final character before > rather than accepting trailing whitespace.

Useful? React with 👍 / 👎.

@/tmp/replyA.md

chatgpt-codex-connector · 2026-11-01T21:42:50Z

+      continue;
+    }
+    if (next === "/") {
+      const m = /^<\/([a-zA-Z][^\s/>]*)\s*\/?>/.exec(html.slice(i));


Consume recoverable end tags outside raw-text mode

For HTML containing trailing attributes on an ordinary closing tag, such as <p>a</p foo><p>b</p>, this strict pattern rejects the closing tag and sends its markup into the extracted article text. The Worker now returns a < /p foo> b, whereas the container's Extractor under Python 3.13.13 and the previous Worker fallback both return a b. Apply tolerant closing-tag consumption in the normal parsing path as well, so recoverable markup does not contaminate stored text and excerpts.

Useful? React with 👍 / 👎.

@/tmp/replyB.md

Self-closing slash must immediately precede the closing bracket: isSelfClosingTag now requires the slash to be the final character of the tag, so a script tag with a spaced slash stays an open raw-text element. Recoverable end tags are consumed in the normal parsing path too, not only in raw-text mode: trailing attributes on an end tag no longer leak into extracted text. Two shared fixtures added, expected values verified against the container Extractor first; node 84/84, python 65 OK.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a179aba170

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-01T21:53:42Z

+      // End tags may carry (invalid but recoverable) attributes, which
+      // HTMLParser consumes rather than emitting as text: </p foo>
+      // still closes the paragraph.
+      const m = /^<\/([a-zA-Z][^\s/>]*)[^>]*>/.exec(html.slice(i));


Avoid quadratic backtracking when parsing incomplete end tags

For an untrusted page ending with </p followed by a long string of letters and no >, the overlapping [^\s/>]* and [^>]* repetitions cause quadratic backtracking. An input containing just 20,000 trailing letters took approximately 434 ms in the new extractor, versus 0.05 ms in the previous fallback; larger inputs remain well below the 2 MB response cap but can exhaust Worker CPU. Check for a terminating > before matching, or parse the tag name and remainder without overlapping repetitions.

Useful? React with 👍 / 👎.

@/tmp/replyP1.md

chatgpt-codex-connector · 2026-11-01T21:53:42Z

+export function decodeHtmlEntities(data: string): string {
+  return data.replace(/&(#(?:[xX][0-9a-fA-F]+|\d+)|[a-zA-Z][a-zA-Z0-9]*);/g, (whole, body: string) => {


Decode the complete set of HTML character references

For pages using standard entities outside the small lookup table or references without semicolons, the new extractor does not match the container. Verified against the container Extractor under Python 3.13.13: <title>Α &AMP;</title><p>→ &#65 &amp B</p> produces title Α & and text → A & B, while the Worker preserves every reference literally. This makes stored titles and article text depend on which crawling path runs. Use complete HTML named-reference data and support the semicolon-omission rules rather than requiring every reference to match this restricted pattern.

Useful? React with 👍 / 👎.

@/tmp/replyE.md

chatgpt-codex-connector · 2026-11-01T21:53:42Z

+  const handleStartTag = (tag: string, selfClosing: boolean) => {
+    if (tag === "title") {
+      if (!selfClosing) cdataTag = "title";
+    } else if (tag === "a") links += 1;
+    else if (tag === "script" || tag === "style") {


Honor raw-text modes for textarea and iframe elements

For pages containing literal markup inside textarea, iframe, xmp, noembed, or noframes, the scanner incorrectly interprets that content as ordinary HTML. For example, <textarea><script></textarea><p>Article body</p> returns empty text because the literal <script> activates script skipping and swallows the remaining document. Verified against the container Extractor under Python 3.13.13, the expected text is <script> Article body; the previous Worker fallback also retained the article body. Implement the container parser's additional RCDATA/raw-text modes so literal content cannot activate nested tags or discard subsequent article text.

Useful? React with 👍 / 👎.

@/tmp/replyR.md

…#569) Codex round-4 P1: the tolerant end-tag pattern (name class plus remainder class, both unbounded) backtracked quadratically on an incomplete end tag at EOF -- 20k trailing letters took ~434ms vs 0.05ms for the previous fallback, on untrusted page input. The end-tag match is now bounded at the next closing bracket before matching the name (0.72ms on the same input, output unchanged), and a letter-led incomplete end tag at EOF is dropped, matching HTMLParser. Fixture added for the EOF case; node 85/85, python 65 OK.

dezren39 requested a review from a team as a code owner November 1, 2026 18:29

chatgpt-codex-connector Bot reviewed Nov 1, 2026

View reviewed changes

Comment thread pkgs/lab-ops-hub/src/lib.ts Resolved

Comment thread pkgs/lab-ops-hub/src/lib.ts Resolved

Comment thread pkgs/lab-ops-hub/src/lib.ts Resolved

dezren39 mentioned this pull request Nov 1, 2026

lab-ops-hub: Worker-fallback extractor diverges from container Extractor — bare <a> uncounted, script-string <a counted, title entities raw, style/title leak into text #569

Closed

chatgpt-codex-connector Bot reviewed Nov 1, 2026

View reviewed changes

github-actions Bot assigned dezren39 Nov 1, 2026

chatgpt-codex-connector Bot reviewed Nov 1, 2026

View reviewed changes

This was referenced Nov 1, 2026

lab-ops-hub: Worker-fallback extractor — complete HTML character-reference decoding parity #4007

Closed

lab-ops-hub: Worker-fallback extractor — RCDATA/raw-text modes for textarea, iframe, xmp, noembed, noframes #4008

Closed

dezren39 merged commit bdc66c7 into main Nov 1, 2026
1 check passed

dezren39 deleted the queue/4868-569-extractor-parity branch November 1, 2026 22:02

developing-today Bot mentioned this pull request Nov 3, 2026

Fix #4007: lab-ops-hub: Worker-fallback extractor — complete HTML character-referen #5115

Closed

		export function decodeHtmlEntities(data: string): string {
		return data.replace(/&(#(?:[xX][0-9a-fA-F]+\|\d+)\|[a-zA-Z][a-zA-Z0-9]*);/g, (whole, body: string) => {

Conversation

dezren39 commented Nov 1, 2026 • edited by blacksmith-sh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

chatgpt-codex-connector Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dezren39 commented Nov 1, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

dezren39 commented Nov 1, 2026 •

edited by blacksmith-sh Bot

Loading

coderabbitai Bot commented Nov 1, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 1, 2026 •

edited

Loading