Skip to content

lab-ops-hub crawl validators disagree on special IPv6 ranges — Python allows multicast/reserved, TS allows 2001::/23 specials #474

Closed

Description

@dezren39

Problem

The two lab-ops-hub crawl validators claim to apply the same policy independently (src/lib.ts: "The Python container crawler applies the same policy independently"; container/app.py: "Only public http(s) targets"), but they disagree on special-use IPv6 literals in both directions. Verified this run on main 4b8f252d by calling both validators directly on the same URLs:

Python container validator ALLOWS, TS Worker validator blocks

Python relies on ipaddress.IPv6Address.is_global, which is True for any address outside its narrow private list — including multicast and reserved space:

URL Python (container/app.py validate_crawl_url) TS (src/lib.ts validateCrawlUrl)
http://[ff00::1]/ (multicast ff00::/8) ALLOW BLOCK
http://[4000::1]/ (reserved, outside global unicast 2000::/3) ALLOW BLOCK
http://[fec0::1]/ (site-local fec0::/10, deprecated) ALLOW BLOCK
http://[5f00::1]/ (SRv6 SIDs 5f00::/16, IANA Global=False) ALLOW BLOCK
http://[100:1::1]/ (outside the 100::/64 discard prefix, not global unicast) ALLOW BLOCK

The Python service is the component that actually fetches, so these non-public targets are reachable through the container path even though the Worker would refuse them as direct targets and as redirect targets.

TS Worker validator ALLOWS, Python blocks

ipv6IsGlobal in src/lib.ts treats everything in 2000::/3 as global except a short blocklist that omits the IANA special-purpose block 2001::/23:

URL TS Python
http://[2001::1]/ (Teredo 2001::/32) ALLOW BLOCK
http://[2001:2::1]/ (benchmarking 2001:2::/48) ALLOW BLOCK
http://[2001:10::1]/ (ORCHID 2001:10::/28) ALLOW BLOCK
http://[2001:40::1]/, http://[2001:50::1]/ (2001::/23 specials) ALLOW BLOCK

Python's verdict follows the IANA special registry encoded in ipaddress: 2001::/23 is non-global except 2001:1::1/128, 2001:1::2/128, 2001:3::/32, 2001:4:112::/48, 2001:20::/28 and 2001:30::/28 — those exceptions must keep passing in both validators (e.g. http://[2001:20::1]/ and http://[2001:3::1]/ are ALLOW in both today).

Why it matters

The two validators are defence-in-depth for the same SSRF policy (#441, #460–#472). A policy that depends on which component happens to fetch is not defence in depth: the container path currently accepts multicast/reserved/site-local literals outright, and the Worker path accepts the 2001::/23 special ranges its own redirect validation is supposed to refuse.

Acceptance criteria

  • Both validators refuse: ff00::1, 4000::1, fec0::1, 5f00::1, 100:1::1, 2001::1, 2001:2::1, 2001:10::1, 2001:40::1, 2001:50::1.
  • Both validators still accept the global exceptions inside 2001::/23 (2001:1::1, 2001:1::2, 2001:3::1, 2001:4:112::1, 2001:20::1, 2001:30::1) and ordinary global unicast (e.g. 2606:4700:4700::1111, 2001:4860:4860::8888).
  • The same test vectors appear in both suites (test/lib.test.ts and test/test_container_validate.py) with identical verdicts, in the style of the lab-ops-hub crawl validators don't judge NAT64/6to4 literals by their embedded IPv4 — Python allows embedded loopback/metadata, TS and Python disagree #472 shared-vectors test.
  • Suggested fix: in Python, don't trust is_global wholesale for IPv6 — require global unicast 2000::/3 (as the TS side does) on top of the existing checks; in TS, add the 2001::/23 block with the six global exceptions above.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions