Surfaced while fixing #507 (PR #511), which retired one crawler whose data source is gone. A sweep of all 136 files under pcapkit/vendor/** shows it is not an isolated case.
Census: 95 classes with a string LINK, 92 distinct URLs. Fetching each with the default requests User-Agent: 200 → 87, 403 → 4, 404 → 1.
The four 403s are all Wikipedia, and all pure User-Agent refusals
| crawler |
default UA |
descriptive UA |
pcapkit/vendor/vlan/priority_level.py |
403, 126 B |
200, 119717 B |
pcapkit/vendor/ftp/return_code.py |
403, 126 B |
200, 78192 B |
pcapkit/vendor/ipx/packet.py |
403, 126 B |
200, 114117 B |
pcapkit/vendor/ipx/socket.py |
retired in #511 |
— |
403 body: Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also https://phabricator.wikimedia.org/T400119.
Only the header changes between the two columns — no redirect — reproduced across three different pages.
The fix belongs in one place: pcapkit/vendor/default.py:404 is page = requests.get(self.LINK) with no headers, in both the direct and proxy branches. A descriptive agent is what Wikipedia's policy asks for; a browser spoof is not.
vendor/ipx/packet.py is additionally broken the same way #507 was
Same article, same removal revision (1371327031, 2026-08-25). :82 does soup.find_all('table', class_='wikitable')[1] against a page that now carries one table, so even with the User-Agent fixed it raises IndexError. It needs the same treatment #511 gave the socket crawler — and unlike that one, its constant file was out of scope there.
The 404 is a genuinely dead URL, unaffected by User-Agent
pcapkit/vendor/pcapng/{block_type,option_type,record_type}.py all point at https://www.ietf.org/staging/draft-tuexen-opsawg-pcapng-02.html — 404 with any agent. Its body is a 77968-byte HTML error page, so default.py:405's if not page.ok or not page.text catches it on page.ok.
Working replacements, both measured:
https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-02.html — 200, 215857 Bhttps://datatracker.ietf.org/doc/html/draft-tuexen-opsawg-pcapng-02 — 200, 233176 B
The same dead URL also appears in the documentation, where sphinx-build -b linkcheck reports it as 8 broken links (docs/source/pcapkit/protocols/misc/pcapng.rst:773, docs/source/pcapkit/const/pcapng.rst:123-129). One fix covers both.
Worth pairing with a reproduction gate
cron-vendor.yml runs these weekly and on every push to main, and auto-commits the result (:88, :98) — so a crawler defect lands unreviewed. Measured today: 22/22 offline generators regenerate their constant files byte-for-byte, and 87/93 network ones do (the 6 are 5 network-blocked hosts plus 1 genuine diff). A CI gate asserting byte-identical regeneration is ~40 lines and would have caught pcapkit/const/mh/handover_initiate_status.py, which is the only one of 134 constant files missing the :meta private: marker its template gained in 07bf98d46 — i.e. the weekly crawl has not successfully rewritten it since.
Surfaced while fixing #507 (PR #511), which retired one crawler whose data source is gone. A sweep of all 136 files under
pcapkit/vendor/**shows it is not an isolated case.Census: 95 classes with a string
LINK, 92 distinct URLs. Fetching each with the defaultrequestsUser-Agent: 200 → 87, 403 → 4, 404 → 1.The four 403s are all Wikipedia, and all pure User-Agent refusals
pcapkit/vendor/vlan/priority_level.pypcapkit/vendor/ftp/return_code.pypcapkit/vendor/ipx/packet.pypcapkit/vendor/ipx/socket.py403 body:
Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also https://phabricator.wikimedia.org/T400119.Only the header changes between the two columns — no redirect — reproduced across three different pages.
The fix belongs in one place:
pcapkit/vendor/default.py:404ispage = requests.get(self.LINK)with no headers, in both the direct and proxy branches. A descriptive agent is what Wikipedia's policy asks for; a browser spoof is not.vendor/ipx/packet.pyis additionally broken the same way #507 wasSame article, same removal revision (
1371327031, 2026-08-25).:82doessoup.find_all('table', class_='wikitable')[1]against a page that now carries one table, so even with the User-Agent fixed it raisesIndexError. It needs the same treatment #511 gave the socket crawler — and unlike that one, its constant file was out of scope there.The 404 is a genuinely dead URL, unaffected by User-Agent
pcapkit/vendor/pcapng/{block_type,option_type,record_type}.pyall point athttps://www.ietf.org/staging/draft-tuexen-opsawg-pcapng-02.html— 404 with any agent. Its body is a 77968-byte HTML error page, sodefault.py:405'sif not page.ok or not page.textcatches it onpage.ok.Working replacements, both measured:
https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-02.html— 200, 215857 Bhttps://datatracker.ietf.org/doc/html/draft-tuexen-opsawg-pcapng-02— 200, 233176 BThe same dead URL also appears in the documentation, where
sphinx-build -b linkcheckreports it as 8 broken links (docs/source/pcapkit/protocols/misc/pcapng.rst:773,docs/source/pcapkit/const/pcapng.rst:123-129). One fix covers both.Worth pairing with a reproduction gate
cron-vendor.ymlruns these weekly and on every push tomain, and auto-commits the result (:88,:98) — so a crawler defect lands unreviewed. Measured today: 22/22 offline generators regenerate their constant files byte-for-byte, and 87/93 network ones do (the 6 are 5 network-blocked hosts plus 1 genuine diff). A CI gate asserting byte-identical regeneration is ~40 lines and would have caughtpcapkit/const/mh/handover_initiate_status.py, which is the only one of 134 constant files missing the:meta private:marker its template gained in07bf98d46— i.e. the weekly crawl has not successfully rewritten it since.