engines: add the pcap_ct engine, and make unsupported_reason a real preflight

Pipfile

@@ -21,6 +21,25 @@ cryptography = "*"
 # ``pipenv lock``, which has to build its metadata -- on any machine without the
 # libpcap development files. Install it by hand to work on that engine:
 # ``pipenv run pip install pypcap``.
+#
+# ``pcap-ct`` and ``libpcap``, which drive the PCAP_CT engine, are listed --
+# the reason ``pypcap`` cannot be does not apply to them. Both are
+# ``py3-none-any`` wheels, so there is nothing to compile and ``pipenv lock``
+# works on a machine with no libpcap *development* files at all. (A system
+# ``libpcap.so.1`` is still needed to actually run the engine; that is a runtime
+# prerequisite, not something the lock file can express.)
+#
+# The versions are spelled out rather than left as ``"*"`` because neither
+# project has ever published a non-pre-release, and naming a pre-release in the
+# specifier is the PEP 440 way to opt into one. That keeps the opt-in scoped to
+# these two: ``allow_prereleases`` below stays commented out, since switching it
+# on would let every other package in this file resolve to a beta as well.
+#
+# The marker matches the PCAP_CT extra in pyproject.toml. ``libpcap`` 1.11.0b29
+# declares ``Requires-Python: <4.0.0,>=3.10.0``, which is the binding floor
+# (``pcap-ct`` itself allows 3.9).
+pcap-ct = {version = ">=1.3.0b3",markers = "python_version >= '3.10'"}
+libpcap = {version = ">=1.11.0b29",markers = "python_version >= '3.10'"}
 pypcapfile = "*"
 beautifulsoup4 = {extras = ["html5lib"],version = "*"}
 requests = {extras = ["socks"],version = "*"}

README.rst

@@ -11,7 +11,7 @@ The PyPCAPKit project is an open source Python program focus on network packet
 parsing and analysis, which works as a comprehensive `PCAP`_ file extraction,
 construction and analysis library.
 
-   The whole project supports **Python 3.6** or later.
+   The whole project supports **Python 3.6** or later; CI covers 3.10 to 3.14, and 3.15 as an allowed-to-fail leg.
 
 -----
 About
@@ -79,23 +79,67 @@ fact that ``pcapkit`` is a **comprehensive** packet processing module.
 
 Additionally, ``pcapkit`` introduced alternative extraction engines to accelerate
 this procedure. By now ``pcapkit`` supports `Scapy`_, `DPKT`_, `PyShark`_,
-`PyPCAP`_ and `PyPCAPFile`_, selected through ``engine='scapy'``,
-``'dpkt'``, ``'pyshark'``, ``'pypcap'`` and ``'pypcapfile'`` respectively;
-``engine='default'`` (also spelled ``'pcapkit'``) is ``pcapkit``'s own parser
-and the only one with no third-party requirement.
+`PyPCAP`_, `pcap-ct`_ and `PyPCAPFile`_, selected through ``engine='scapy'``,
+``'dpkt'``, ``'pyshark'``, ``'pypcap'``, ``'pcap_ct'`` and ``'pypcapfile'``
+respectively; ``engine='default'`` (also spelled ``'pcapkit'``) is ``pcapkit``'s
+own parser and the only one with no third-party requirement.
+
+`PyPCAP`_ and `pcap-ct`_ are two independent distributions of the same
+``libpcap(3)`` interface, and both install a top-level ``pcap`` module, so
+they are two engines rather than one. Upstream `PyPCAP`_ stops at Python 3.11;
+`pcap-ct`_ covers 3.10 and newer. **Install exactly one of them** -- with both
+present, ``pcap-ct`` wins the import and the other becomes unselectable, which
+each engine detects and reports.
 
 Speed is not free. Every third-party engine supports **less** than the
-``default`` one, and the two newest support markedly less:
+``default`` one, and the newest ones support markedly less:
 
 - `PyPCAP`_ performs no protocol dissection at all, so it offers neither
   reassembly nor flow tracing, and reads PCAP savefiles from disk only.
+- `pcap-ct`_ reads the same interface, so it has exactly the same gaps.
 - `PyPCAPFile`_ has no IPv6 decoder, so IPv6 reassembly is unavailable; IPv4
   and TCP reassembly still work, and it too is PCAP-only.
 - `PyShark`_ performs no reassembly.
 
 Each gap is announced with a warning or an exception rather than silently
 returning nothing. The `engine support documentation`_ tabulates them.
 
+Every engine also answers a preflight check before it is used --
+``unsupported_reason()`` -- so asking for one that cannot run in the current
+environment produces a single warning naming the actual cause (a Python version, a
+missing ``tshark``, a missing ``libpcap``, the wrong ``pcap`` distribution) and a
+clean fall back to ``pcapkit``'s own parser, rather than an error from inside the
+third-party package.
+
+Engine support by Python version
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Which engines can run at all, by interpreter. Verified by installing each engine
+and extracting a capture on 3.10, 3.11, 3.12 and 3.14; 3.13 and 3.15 were not
+available and are marked accordingly.
+
+============== ======== ======== ======== ======== ======== ========
+Engine          3.10     3.11     3.12     3.13     3.14     3.15
+============== ======== ======== ======== ======== ======== ========
+``pcapkit``     yes      yes      yes      yes*     yes      yes*
+``dpkt``        yes      yes      yes      yes*     yes      yes*
+``scapy``       yes      yes      yes      yes*     yes      yes*
+``pcap_ct``     yes      yes      yes      yes*     yes      yes*
+``pypcap``      yes      yes      no       no       no       no
+``pypcapfile``  yes      yes      no       no       no       no
+``pyshark``     yes†     yes†     yes†     yes*†    no       no
+============== ======== ======== ======== ======== ======== ========
+
+``*`` inferred, not measured -- no 3.13 or 3.15 interpreter was available.
+``†`` also needs Wireshark's ``tshark``, which was absent, so only the
+interpreter half was verified for ``pyshark``.
+
+``pypcap`` and ``pypcapfile`` stop at 3.11, and ``pyshark`` at 3.13, for the
+reasons under `Engine prerequisites`_. **Python 3.11 is the last version on which
+every engine can run** -- and even there ``pypcap`` and ``pcap_ct`` are mutually
+exclusive, since both provide the ``pcap`` module, so no single environment ever
+has all seven at once.
+
 Test Environment
 ~~~~~~~~~~~~~~~~
 
@@ -121,19 +165,51 @@ Engine         Performance (ms per packet)
 ``dpkt``        0.010390_056723
 ``scapy``       0.091690_233567
 ``pcapkit``     0.200390_390390
-``pyshark``    24.682185_018351
+``pyshark``    24.682185_018351 [3]_
 ``pypcap``      *not measured* [1]_
+``pcap_ct``     *not measured* [4]_
 ``pypcapfile``  *not measured* [2]_
 ============== ===========================
 
-Both figures will be filled in once the two engines can be timed on the same
-host, capture and iteration count as the existing rows.
+**These figures are historical, and three of the rows can no longer be
+reproduced on a current Python.** The table was taken on the environment above,
+whose interpreter still ran every engine. Since then ``pyshark``, ``pypcap`` and
+``pypcapfile`` have each acquired a hard Python ceiling -- 3.13, 3.11 and 3.11
+respectively, for the reasons under `Engine prerequisites`_ -- so on the latest
+Python only ``pcapkit``, ``dpkt``, ``scapy`` and ``pcap_ct`` can be timed at all.
+A re-run on a modern interpreter would therefore not extend this table; it would
+replace it with a shorter one, measured on different hardware and not comparable
+row-for-row with what is here.
+
+The empty cells stay empty for the same reason: a figure taken on a different
+host, capture or iteration count is not comparable with these, and inventing one
+would be worse than admitting the gap.
 
 ------------
 Installation
 ------------
 
-   **Note** -- ``pcapkit`` supports Python versions **since 3.6**.
+   **Note** -- ``pcapkit`` declares support for **Python 3.6 and later**, and CI
+   covers **3.10 through 3.14**, plus 3.15 as an allowed-to-fail leg.
+
+   The sources themselves use 3.8 syntax; the ``bpc-walrus``/``bpc-poseur``
+   backport tools in ``setup.py`` convert it at install time, which is what makes
+   the lower bound possible. Measured: 3.9 and 3.8 import and extract straight
+   from source with no conversion needed, and 3.7 needs the conversion.
+
+   **That conversion is currently blocked by an upstream bug**, so below 3.8 the
+   declaration is intent rather than something that works today: ``bpc-poseur``
+   0.4.3.post1 crashes on positional-only parameters declared on a *method*
+   rather than a plain function, and exits 0 so the build does not notice. The
+   12 such parameters in ``pcapkit/corekit/io.py`` then survive into the
+   installed package and ``import pcapkit`` fails. It is a one-line fix
+   upstream -- ``poseur.py:744`` passes ``cls_ctx=name.name`` where ``name`` is
+   already a parso ``Name`` and wants ``.value`` -- and with it applied,
+   ``walrus`` then ``poseur`` produce a file Python 3.7 parses cleanly. Tracking
+   that fix is what will make 3.6/3.7 real again.
+
+   3.8 and 3.9 are end-of-life and best-effort. Individual *engines* also stop
+   earlier than the library does; see `Engine prerequisites`_.
 
 Simply run the following to install the current version from PyPI:
 
@@ -188,28 +264,60 @@ plug-in functions, you may want to install the optional ones:
    pip install pypcapkit[PyPCAPFile]
    # for PyPCAP only -- see the note below, this one builds from source
    pip install pypcapkit[PyPCAP]
+   # for pcap-ct only -- the pure-Python alternative to PyPCAP, and the one that
+   # works on Python 3.12+; do not install it alongside PyPCAP
+   pip install pypcapkit[PCAP_CT]
    # for ESP payload decryption
    pip install pypcapkit[crypto]
-   # and to install the optional packages -- note this excludes PyPCAP
+   # and to install the optional packages -- note this excludes PyPCAP and pcap-ct
    pip install pypcapkit[all]
    # or to do this explicitly
    pip install pypcapkit dpkt scapy pyshark pypcapfile
 
-   **Important** -- The ``all`` extra deliberately does **not** include
-   ``pypcap``. Everything
+   **Important** -- The ``all`` extra deliberately excludes both ``pypcap`` and
+   ``pcap-ct``, for different reasons. Everything
    else in ``all`` is a pure-Python wheel, whereas ``pypcap`` compiles a C
    extension; pulling it into ``all`` would demand a working compiler and the
    `libpcap`_ development files from everyone installing ``pypcapkit[all]``.
-   Install it explicitly with ``pip install pypcapkit[PyPCAP]``.
+   ``pcap-ct`` needs no compiler, but it and its ``libpcap`` dependency are
+   published only as **pre-releases** (1.3.0b3 and 1.11.0b29), and ``all`` should
+   not be how somebody ends up with a beta they did not ask for. Install either
+   explicitly: ``pip install pypcapkit[PyPCAP]`` or
+   ``pip install pypcapkit[PCAP_CT]``.
+
+   **Install only one of them.** Both distributions own the top-level ``pcap``
+   module, and ``pip`` will install both without complaint. With both present the
+   ``pcap-ct`` package wins the import and ``pypcap``'s extension module is
+   shadowed and unreachable, so ``engine='pypcap'`` stops working. ``pcapkit``
+   detects that state and warns, naming both distributions and which one won, but
+   it cannot undo it.
 
 Engine prerequisites
 --------------------
 
-Three of the engines need something beyond a ``pip install``:
+Four of the engines need something beyond a ``pip install``. Each constraint is
+also enforced in code -- the engine's ``unsupported_reason()`` is consulted before
+anything is imported -- so hitting one produces a warning naming the cause and a
+fall back to ``pcapkit``'s own parser, not an error from inside the third-party
+package.
 
 ``pyshark``
-   Drives Wireshark's ``tshark`` binary, which must be on ``PATH``. Install
-   Wireshark (or just ``tshark``) from your platform's package manager.
+   Two requirements, and neither is visible to an import: the package imports
+   cleanly and then fails when used.
+
+   - Drives Wireshark's ``tshark`` binary. It need not be on ``PATH``:
+     ``pyshark`` looks at ``tshark_path`` in its ``config.ini`` first, then
+     ``PATH`` on POSIX, both Program Files directories on Windows, and
+     ``/Applications/Wireshark.app`` on macOS. Install Wireshark (or just
+     ``tshark``) from your platform's package manager.
+   - Requires Python **3.13 or older**. ``pyshark`` 0.6 builds its event loop
+     with ``asyncio.get_event_loop_policy().get_event_loop()``, and from Python
+     **3.14** ``asyncio.get_event_loop()`` raises ``RuntimeError`` when there
+     is no current event loop instead of quietly creating one. Measured: a loop
+     is returned silently on 3.10 and 3.11, returned with a
+     ``DeprecationWarning`` on 3.12, and refused on 3.14. (3.13 was not available
+     to test and is expected to work, being on the deprecated-but-functional side
+     of that change.)
 
 ``pypcap``
    Ships **no wheels** -- only an sdist -- so ``pip`` compiles it, and the build
@@ -239,6 +347,27 @@ Three of the engines need something beyond a ``pip install``:
      found even though it is installed. Installing into ``sys.prefix``, or into
      ``/opt/libpcap``, is what that search will pick up.
 
+``pcap_ct``
+   The way to drive the same `libpcap`_ interface on Python **3.12 and newer**,
+   where ``pypcap`` cannot be built. Nothing to compile and no ``pcap.h`` needed:
+   `pcap-ct`_ is a ``ctypes`` reimplementation, and both it and its ``libpcap``
+   dependency ship ``py3-none-any`` wheels. Verified reading a capture on Python
+   3.10 and 3.14.
+
+   Two caveats:
+
+   - **A system ``libpcap`` is still required at run time.** The ``libpcap``
+     distribution ships a vendored ``libpcap.so`` and, as published, does not use
+     it: its ``libpcap.cfg`` says ``LIBPCAP = None``, which sends its loader to
+     ``ctypes.util.find_library('pcap')``. So the library actually loaded is the
+     host's ``libpcap.so.1``, and with none present ``import pcap`` raises
+     ``OSError`` rather than ``ImportError``. Set ``LIBPCAP = tcpdump`` in
+     ``libpcap.cfg`` to use the vendored copy instead.
+   - Both distributions are **pre-releases**, and ``pcap-ct`` documents itself as
+     tracking the ``pypcap`` *1.2.3* interface. Every attribute the engine uses
+     was measured behaving identically to ``pypcap`` 1.3.0, but that is a
+     statement about the versions tested.
+
 ``pypcapfile``
    Version 0.12.0 imports the ``imp`` module, which was **removed in Python
    3.12**, so ``pcapfile.savefile`` -- the module needed to read a capture --
@@ -248,10 +377,11 @@ Three of the engines need something beyond a ``pip install``:
 
    **Note** -- ``pcapkit`` itself, and its ``default``, ``dpkt`` and ``scapy``
    engines, work
-   fine on current Python versions. Only the three engines above carry these
-   extra constraints, and asking for an engine whose package is unavailable
-   emits a warning and falls back to ``pcapkit``'s own parser rather than
-   failing outright.
+   fine on current Python versions -- ``dpkt`` 1.9.8 and ``scapy`` 2.7.0 were both
+   measured reading a capture on Python 3.14. Only the four engines above carry
+   extra constraints, and asking for an engine that cannot run in the current
+   environment emits a warning naming the reason and falls back to ``pcapkit``'s
+   own parser rather than failing outright.
 
 For CLI usage, you will need to install the optional packages:
 
@@ -294,11 +424,20 @@ engine, and is not needed by the test suite.
 .. _DPKT: https://dpkt.readthedocs.io
 .. _PyShark: https://kiminewt.github.io/pyshark
 .. _PyPCAP: https://github.com/pynetwork/pypcap
+.. _pcap-ct: https://pypi.org/project/pcap-ct/
 .. _PyPCAPFile: https://github.com/kisom/pypcapfile
 .. _libpcap: https://www.tcpdump.org
 .. _DictDumper: https://github.com/JarryShaw/DictDumper
 .. _engine support documentation: https://jarryshaw.github.io/PyPCAPKit/pcapkit/foundation/engines/index.html
 
+.. [3] This figure is **historical**. `PyShark`_ 0.6 builds its event loop with
+   ``asyncio.get_event_loop_policy().get_event_loop()``, and Python 3.14 made
+   ``asyncio.get_event_loop()`` raise ``RuntimeError`` when no current event
+   loop exists rather than quietly creating one -- measured working on 3.10 and
+   3.11, working with a ``DeprecationWarning`` on 3.12, and raising on 3.14. The
+   number therefore cannot be reproduced on a current interpreter; it stands as
+   what was measured when it could be.
+
 .. [1] `PyPCAP`_ could not be installed on the machine available for
    benchmarking, so no figure was taken. Its 1.3.0 sdist compiles a C extension
    and needs `libpcap`_'s headers *and* shared library present, and the
@@ -307,6 +446,12 @@ engine, and is not needed by the test suite.
    hardware and a different Python from the rows above -- which would not be
    comparable with them -- the cell is left empty.
 
+.. [4] `pcap-ct`_ was verified working on Python 3.10 and 3.14, so unlike the two
+   rows above it *could* be timed -- but only on the machine this engine was added
+   on, which is neither the hardware nor the operating system the rows above were
+   measured with. A number from it would not be comparable, so the cell is left
+   empty rather than filled with something misleading.
+
 .. [2] `PyPCAPFile`_ 0.12.0 cannot be imported on Python 3.12 or newer, so it
    could only be timed on an older interpreter than the rows above were measured
    with. That number would not be comparable, so the cell is left empty.

docs/source/ext.rst

@@ -325,23 +325,35 @@ file formats:
 |                     +-----------------------------------------------------------+----------------------------------------+
 |                     | :class:`pcapkit.foundation.engines.pypcap.PyPCAP`         | PCAP only, and from a file on disk     |
 |                     +-----------------------------------------------------------+----------------------------------------+
+|                     | :class:`pcapkit.foundation.engines.pcap_ct.PCAP_CT`       | PCAP only, and from a file on disk     |
+|                     +-----------------------------------------------------------+----------------------------------------+
 |                     | :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` | PCAP only                              |
 +---------------------+-----------------------------------------------------------+----------------------------------------+
 
 .. note::
 
    An engine is free to support less than :class:`~pcapkit.foundation.extraction.Extractor`
-   offers, and several do. `PyPCAP`_ performs no protocol dissection whatsoever, so
-   it supports neither reassembly nor flow tracing; `PyPCAPFile`_ has no IPv6
-   decoder, so it supports IPv4 and TCP reassembly but not IPv6. What matters is
-   that the gap is *announced* -- each engine warns, or raises, for the capability
-   it cannot provide, rather than silently producing an empty result. See
+   offers, and several do. `PyPCAP`_ and `pcap-ct`_ perform no protocol dissection
+   whatsoever, so they support neither reassembly nor flow tracing; `PyPCAPFile`_
+   has no IPv6 decoder, so it supports IPv4 and TCP reassembly but not IPv6. What
+   matters is that the gap is *announced* -- each engine warns, or raises, for the
+   capability it cannot provide, rather than silently producing an empty result. See
    :doc:`pcapkit/foundation/engines/index` for the full table.
 
+.. note::
+
+   `PyPCAP`_ and `pcap-ct`_ are two distributions of one :manpage:`libpcap(3)`
+   interface, and both install a top-level :mod:`pcap` module, so they are two
+   engines rather than one: ``engine='pypcap'`` and ``engine='pcap_ct'``. Upstream
+   `PyPCAP`_ cannot be installed on Python 3.12 or newer, and `pcap-ct`_ can --
+   which is why both exist. See
+   :doc:`pcapkit/foundation/engines/index` for which to pick.
+
 .. _Scapy: https://scapy.net
 .. _DPKT: https://dpkt.readthedocs.io
 .. _PyShark: https://kiminewt.github.io/pyshark
 .. _PyPCAP: https://github.com/pynetwork/pypcap
+.. _pcap-ct: https://pypi.org/project/pcap-ct/
 .. _PyPCAPFile: https://github.com/kisom/pypcapfile
 
 Samples