Add the pypcap and pypcapfile extraction engines

[JarryShaw]

Closes the New Engines item on the Help Wanted page — both candidates it names are now selectable, each with a matching pcapkit.toolkit module, a pyproject.toml extra, docs and tests.

Registration: built-in, not the public hook

Both are registered in Extractor.__engine__ as ModuleDescriptors, like dpkt/scapy/pyshark. That is not merely for consistency — the public auto-registration path cannot work for an engine shipped with the library. pcapkit/__init__.py never imports pcapkit.foundation.engines, so Engine.__init_subclass__ would never fire for a bare import pcapkit; Extractor(engine='pypcap') would then fall through to "unsupported engine" and silently use the built-in parser — precisely the trap these engines exist to avoid. ModuleDescriptor also keeps the third-party import lazy, so import pcapkit stays free of pcap/pcapfile. The public path remains the documented route for user engines and keeps its own coverage.

Capability gaps are surfaced, not papered over

pypcap is a libpcap binding with zero dissection:

• No reassembly or flow tracing — run() clears the flags, replaces the managers, and warns; the four toolkit adapters raise UnsupportedCall.
• PCAP only. Measured: libpcap 1.11 opens a PCAP-NG file happily, reports datalink()==1, and yields 0 frames — a silently empty capture. So the engine gates on the magic number and raises FormatError instead.
• File-on-disk only (pcap_open_offline opens by name), so non-file input raises UnsupportedCall.
• follow_tcp_stream(engine='pypcap') now falls back with an EngineWarning, as it cannot trace.

pypcapfile decodes Ethernet/IPv4/TCP/UDP only:

• No IPv6 — run() clears the flag and warns, and ipv6_reassembly raises rather than returning None, which would be indistinguishable from "no fragment here".
• Design note: it loads with layers=0 and decodes to depth 2 itself, because pypcapfile's decoders replace payload bytes; stopping at the network layer keeps the TCP segment verbatim so the header/payload split is exact rather than reconstructed. The IPv4 header is reconstructed, and that reconstruction is asserted byte-exact against real packets including options and fragment flags.

Neither installs cleanly from PyPI, and pep.rst says so

• pypcapfile 0.12.0 imports imp, removed in Python 3.12, so it cannot be imported at all on 3.12+. Upstream master (0.12.1, unreleased) fixes it, and that is what the engine was verified against.
• pypcap 1.3.0 ships a pre-generated pcap.c from an old Cython that no longer compiles against the 3.12+ C API (ob_digit, _PyLong_AsByteArray), and its setup.py searches a fixed list of prefixes for pcap.h. It builds once the headers are visible and pcap.c is regenerated with Cython 3.

Both were genuinely executed here, on a locally rebuilt wheel — neither is claimed as unverified. The caveats are about installability, not coverage, and the note in pep.rst states them rather than claiming both are simply done.

Parity, and proving the engine actually ran

tests/foundation/engines/test_new_engine_parity.py runs over in.pcap, arp.pcap, tcp.pcap, ipv4.pcap (1137 frames) and asserts, per frame: capture length, original length, timestamp, and the full Ethernet header — all exact. Every parity test goes through a helper that patches warn, asserts no EngineWarning was raised, and asserts extractor._exnam is the engine requested — because Extractor does not raise when an engine's package is missing, it warns and silently falls back, so a naive test would happily report parity for work the built-in parser did. pypcapfile IPv4 reassembly is additionally asserted byte-identical to the default engine.

One pre-existing crash fixed, because nothing worked without it

Extractor(trace=True) died with AttributeError: 'dict' object has no attribute 'packet'. The guard at extraction.py listed only pyshark and only matched trace_format == 'pcap', while TraceFlow.__init__ substitutes 'pcap' for None — so it never fired. It now matches ('pcap', 'cap', None), which incidentally fixes pyshark too. dpkt and scapy remain broken there by design and are called out in a code comment; that is its own change.

Also worth knowing for future tests: BaseWarning.__init__ calls warnings.simplefilter('ignore', type(self)) outside devmode, so assertWarns can never fire on a pcapkit warning. These tests patch warn and inspect the call instead. (That behaviour is #364.)

Verification

56 new test functions across 5 files. Full suite on this branch: 550 passed, 18 skipped, 335 subtests — the 18 skips are exactly the availability-gated engine tests, since the project venv deliberately does not carry either library. In a venv with both installed the same suite runs 8 more tests.

[Copilot]

🟢 Approval recommended

The changes appear functionally complete with extensive unit/parity coverage, and the only noted issues are minor typos in user-facing strings/docstrings.

Pull request overview

Adds two new selectable third-party extraction engines (pypcap, pypcapfile) to PyPCAPKit/pcapkit, including lazy engine registration, matching toolkit adapters, documentation, and extensive unit + parity test coverage to ensure the requested engine actually ran and matches the default engine on key wire-level fields.

Changes:

• Register pypcap and pypcapfile as built-in engines in Extractor.__engine__, and adjust extractor trace-format handling for engines that emit mapping frames.
• Introduce new engine implementations and toolkit adapters for pypcap and pypcapfile, including explicit capability-gap behavior (warnings/exceptions).
• Add docs updates and a broad test suite (unit tests + end-to-end parity tests), plus packaging extras for the new dependencies.

File summaries

File	Description
tests/toolkit/test_pypcapfile_unit.py	Unit tests for PyPCAPFile toolkit helpers and adapters (stand-ins + real-decoder gated tests).
tests/toolkit/test_pypcap_unit.py	Unit tests for PyPCAP toolkit helpers and “refuse loudly” adapters.
tests/foundation/engines/test_pypcapfile_engine.py	Engine-level unit tests for PyPCAPFile behavior using stand-ins/mocks.
tests/foundation/engines/test_pypcap_engine.py	Engine-level unit tests for PyPCAP behavior using a fake pcap handle.
tests/foundation/engines/test_new_engine_parity.py	End-to-end parity tests proving the requested engine ran and matches default on key fields.
pyproject.toml	Adds PyPCAP/PyPCAPFile extras and includes them in all.
Pipfile	Adds pypcap and pypcapfile to dev dependencies.
pcapkit/toolkit/pypcapfile.py	New toolkit adapter module for pypcapfile packets, including IPv4/TCP support and explicit IPv6 refusal.
pcapkit/toolkit/pypcap.py	New toolkit adapter module for pypcap raw frames and explicit refusal for higher-level adapters.
pcapkit/toolkit/init.py	Documents (commented) import surface for the new toolkit modules.
pcapkit/interface/misc.py	Updates engine type alias and adds pypcap to follow_tcp_stream fallback list.
pcapkit/foundation/extraction.py	Adds new engines to typing/registration; adjusts trace-format fallback for engines emitting dict frames.
pcapkit/foundation/engines/pypcapfile.py	New PyPCAPFile engine implementation (PCAP-only, IPv6 gap surfaced, decode-depth control).
pcapkit/foundation/engines/pypcap.py	New PyPCAP engine implementation (PCAP-only, file-on-disk only, no dissection, disables reassembly/trace).
pcapkit/foundation/engines/init.py	Exposes the new engines from the engines package.
docs/source/pep.rst	Updates “New Engines” section to reflect completion and document installability caveats.
docs/source/pcapkit/toolkit/3rdparty.rst	Adds Sphinx docs entries for the new toolkit modules.
docs/source/pcapkit/foundation/engines/index.rst	Updates engine index and adds a capability-gap table including the new engines.
docs/source/pcapkit/foundation/engines/3rdparty.rst	Adds Sphinx docs entries for the new engine classes and internals.
docs/source/ext.rst	Updates extraction engine support table and adds notes about capability gaps.

Review details

Suppressed comments (1)

pcapkit/foundation/engines/pypcap.py:167

• Typo in warning message: "dose" should be "does".

            warn("'Extractor(engine=pypcap)' object dose not support flow tracing; "
                 f"so 'tcp={ext._tcp}' will be ignored", AttributeWarning, stacklevel=stacklevel())

• Files reviewed: 20/20 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JarryShaw]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

[Copilot]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply ...

Fixed all four typos ("dose" → "does" in engines/pypcap.py lines 159/166 and engines/pypcapfile.py line 193; "seperated" → "separated" in toolkit/pypcap.py line 61 and toolkit/pypcapfile.py line 250).

[JarryShaw]

Copilot's four typo comments were already fixed by its own commit 1cbb9e726; I fast-forwarded onto it rather than replaying over it. One heads-up though, because it changes what an extra means.

pypcap had to come out of the all extra. deploy-pages failed on this branch, and it is not a flake:

Getting requirements to build wheel ... error
  Found pcap headers in /Applications/.../MacOSX26.sdk/usr/include/pcap.h
  None of the following found: ['libpcap.a', 'libpcap.so', 'libpcap.dylib', 'wpcap.lib']
ERROR: Failed to build 'pypcap' when getting requirements to build wheel

pypcap 1.3.0 publishes no wheels at all — PyPI has only an sdist — so pip install pypcapkit[all] now compiles a C extension against libpcap and needs the headers and the shared library. The macOS runner has the header and no library, which is exactly the failing case.

deploy-pages was only the first to hit it. .[all] is installed by five workflows:

workflow	occurrences
deploy-pages	1
cron-conda	3
cron-vendor	1
create-release	2

So the release and conda paths would have broken on merge too, and every user running pip install pypcapkit[all] would need a compiler and libpcap dev files.

pypcap stays as its own opt-in extra — pip install pypcapkit[PyPCAP] — documented as needing libpcap present. pypcapfile is sdist-only as well but pure Python, so it installs anywhere and stays in all. Also dropped from the Pipfile dev-packages, where it would break pipenv install --dev and even pipenv lock on any machine without libpcap — this one included.

Worth noting pep.rst already said pip install pypcapkit[PyPCAP] "can fail to build" while also claiming both extras were in all; that contradiction is now resolved in favour of what the packaging does.

Verified: pip install --dry-run '.[all]' on this libpcap-less host resolves and would install only pypcapfile. Unit tier 438 passed / 14 skipped; engines and toolkit 74 passed / 14 skipped.

Unrelated and left alone: seperated also appears pre-existing in toolkit/scapy.py, toolkit/dpkt.py, protocols/misc/pcapng.py, protocols/link/ethernet.py and protocols/link/arp.py — the new modules had copied it from their siblings. Out of scope here.