Repository navigation
Description
Problem
The two lab-ops-hub crawl validators claim to apply the same policy independently (src/lib.ts: "The Python container crawler applies the same policy independently"; container/app.py: "Only public http(s) targets"), but they disagree on special-use IPv6 literals in both directions. Verified this run on main 4b8f252d by calling both validators directly on the same URLs:
Python container validator ALLOWS, TS Worker validator blocks
Python relies on ipaddress.IPv6Address.is_global, which is True for any address outside its narrow private list — including multicast and reserved space:
| URL | Python (container/app.py validate_crawl_url) |
TS (src/lib.ts validateCrawlUrl) |
|---|---|---|
http://[ff00::1]/ (multicast ff00::/8) |
ALLOW | BLOCK |
http://[4000::1]/ (reserved, outside global unicast 2000::/3) |
ALLOW | BLOCK |
http://[fec0::1]/ (site-local fec0::/10, deprecated) |
ALLOW | BLOCK |
http://[5f00::1]/ (SRv6 SIDs 5f00::/16, IANA Global=False) |
ALLOW | BLOCK |
http://[100:1::1]/ (outside the 100::/64 discard prefix, not global unicast) |
ALLOW | BLOCK |
The Python service is the component that actually fetches, so these non-public targets are reachable through the container path even though the Worker would refuse them as direct targets and as redirect targets.
TS Worker validator ALLOWS, Python blocks
ipv6IsGlobal in src/lib.ts treats everything in 2000::/3 as global except a short blocklist that omits the IANA special-purpose block 2001::/23:
| URL | TS | Python |
|---|---|---|
http://[2001::1]/ (Teredo 2001::/32) |
ALLOW | BLOCK |
http://[2001:2::1]/ (benchmarking 2001:2::/48) |
ALLOW | BLOCK |
http://[2001:10::1]/ (ORCHID 2001:10::/28) |
ALLOW | BLOCK |
http://[2001:40::1]/, http://[2001:50::1]/ (2001::/23 specials) |
ALLOW | BLOCK |
Python's verdict follows the IANA special registry encoded in ipaddress: 2001::/23 is non-global except 2001:1::1/128, 2001:1::2/128, 2001:3::/32, 2001:4:112::/48, 2001:20::/28 and 2001:30::/28 — those exceptions must keep passing in both validators (e.g. http://[2001:20::1]/ and http://[2001:3::1]/ are ALLOW in both today).
Why it matters
The two validators are defence-in-depth for the same SSRF policy (#441, #460–#472). A policy that depends on which component happens to fetch is not defence in depth: the container path currently accepts multicast/reserved/site-local literals outright, and the Worker path accepts the 2001::/23 special ranges its own redirect validation is supposed to refuse.
Acceptance criteria
- Both validators refuse:
ff00::1,4000::1,fec0::1,5f00::1,100:1::1,2001::1,2001:2::1,2001:10::1,2001:40::1,2001:50::1. - Both validators still accept the global exceptions inside 2001::/23 (2001:1::1, 2001:1::2, 2001:3::1, 2001:4:112::1, 2001:20::1, 2001:30::1) and ordinary global unicast (e.g.
2606:4700:4700::1111,2001:4860:4860::8888). - The same test vectors appear in both suites (
test/lib.test.tsandtest/test_container_validate.py) with identical verdicts, in the style of the lab-ops-hub crawl validators don't judge NAT64/6to4 literals by their embedded IPv4 — Python allows embedded loopback/metadata, TS and Python disagree #472 shared-vectors test. - Suggested fix: in Python, don't trust
is_globalwholesale for IPv6 — require global unicast 2000::/3 (as the TS side does) on top of the existing checks; in TS, add the 2001::/23 block with the six global exceptions above.
Activity
style(lab-ops-hub): biome format for #474 condition