Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions docs/source/demo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ its main interface. Several scenarios are shown as below.
.. code-block:: python

from pcapkit import HTTP, extract
# set strict to make sure full reassembly
extraction = extract(fin='in.pcap', store=False, nofile=True, reassembly=True, tcp=True, strict=True)
# set reasm_strict to make sure full reassembly
extraction = extract(fin='in.pcap', store=False, nofile=True, reassembly=True, tcp=True, reasm_strict=True)
# print extracted packet if HTTP in reassembled payloads
for datagram in extraction.reassembly.tcp:
if datagram.packet is not None and HTTP in datagram.packet:
Expand All @@ -58,7 +58,9 @@ The CLI (command line interface) of :mod:`pcapkit` has two different access.

* through Python module

``python -m pypcapkit [...]`` works exactly the same as above.
``python -m pcapkit [...]`` works exactly the same as above. Note that the
module name is ``pcapkit``, even though the distribution on PyPI is named
``pypcapkit``.

Here are some usage samples:

Expand All @@ -67,7 +69,7 @@ Here are some usage samples:

.. code-block:: shell

$ pcapkit-cli in --format plist --verbose
$ pcapkit-cli in --auto-extension --format plist --verbose
🚨Loading file 'in.pcap'
Frame 1: Ethernet:IPv6:IPv6_ICMP
Frame 2: Ethernet:IPv6:IPv6_ICMP
Expand All @@ -77,11 +79,18 @@ Here are some usage samples:
Frame 6: Ethernet:IPv4:UDP:Raw
🍺Report file stored in 'out.plist'

2. export to a JSON file (with no format specified)
2. export to a JSON file

.. note::

The output format is **not** presumed from the output file name, so
``-f``/``--format`` (or the ``-j``/``--json`` switch) has to be given;
without it the default tree view is written into whatever file name was
supplied.

.. code-block:: shell

$ pcapkit-cli in --output out.json --verbose
$ pcapkit-cli in --auto-extension --output out.json --format json --verbose
🚨Loading file 'in.pcap'
Frame 1: Ethernet:IPv6:IPv6_ICMP
Frame 2: Ethernet:IPv6:IPv6_ICMP
Expand All @@ -95,7 +104,7 @@ Here are some usage samples:

.. code-block:: shell

$ pcapkit-cli in --output out.txt --format tree --verbose
$ pcapkit-cli in.pcap --output out.txt --format tree --verbose
🚨Loading file 'in.pcap'
Frame 1: Ethernet:IPv6:IPv6_ICMP
Frame 2: Ethernet:IPv6:IPv6_ICMP
Expand Down
2 changes: 2 additions & 0 deletions docs/source/ext.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ The following table shows all available protocol classes in :mod:`pcapkit`:
| | | | :class:`pcapkit.protocols.internet.ipsec.IPsec` |
+ + + IPsec Family +-------------------------------------------------------------+
| | | | :class:`pcapkit.protocols.internet.ah.AH` |
+ + + +-------------------------------------------------------------+
| | | | :class:`pcapkit.protocols.internet.esp.ESP` |
+ +----------------+-----------------------+-------------------------------------------------------------+
| | :class:`pcapkit.protocols.internet.ipx.IPX` |
+ +----------------+-----------------------+-------------------------------------------------------------+
Expand Down
6 changes: 3 additions & 3 deletions docs/source/pcapkit/const/ipv4.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ enumerations include:
- ToS (DS Field) Delay
* - :class:`IPv4_ToSECN <pcapkit.const.ipv4.tos_ecn.ToSECN>`
- ToS ECN Field
* - :class:`IPv4_ToSPrecedence <pcapkit.const.ipv4.tos_pre.TOSPrecedence>`
* - :class:`IPv4_ToSPrecedence <pcapkit.const.ipv4.tos_pre.ToSPrecedence>`
- ToS (DS Field) Precedence
* - :class:`IPv4_ToSReliability <pcapkit.const.ipv4.tos_rel.TOSReliability>`
* - :class:`IPv4_ToSReliability <pcapkit.const.ipv4.tos_rel.ToSReliability>`
- ToS (DS Field) Reliability
* - :class:`IPv4_ToSThroughput <pcapkit.const.ipv4.tos_thr.TOSThroughput>`
* - :class:`IPv4_ToSThroughput <pcapkit.const.ipv4.tos_thr.ToSThroughput>`
- ToS (DS Field) Throughput
* - :class:`IPv4_TSFlag <pcapkit.const.ipv4.ts_flag.TSFlag>`
- TS Flag
Expand Down
6 changes: 3 additions & 3 deletions docs/source/pcapkit/dumpkit/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ of :mod:`pcapkit.dumpkit`:
A --> Tree & XML & JSON

subgraph pcapkit [PyPCAPKit Dumpers]
DumperBase --> Dumper --> PCAPIO & NotImplementedIO
Dumper --> E([user customisation ...])
DumperBase --> PCAPIO & NotImplementedIO
DumperBase --> Dumper --> E([user customisation ...])
end
A --> DumperBase

Expand All @@ -49,4 +49,4 @@ of :mod:`pcapkit.dumpkit`:
click DumperBase "/pcapkit/dumpkit/common.html#pcapkit.dumpkit.common.DumperBase"
click Dumper "/pcapkit/dumpkit/common.html#pcapkit.dumpkit.common.Dumper"
click PCAPIO "/pcapkit/dumpkit/pcap.html#pcapkit.dumpkit.pcap.PCAPIO"
click NotImplementedIO "/pcapkit/dumpkit/pcap.html#pcapkit.dumpkit.pcap.NotImplementedIO"
click NotImplementedIO "/pcapkit/dumpkit/null.html#pcapkit.dumpkit.null.NotImplementedIO"
2 changes: 1 addition & 1 deletion docs/source/pcapkit/dumpkit/null.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Null Dumper
===========

.. module:: pcapkit.dumper.null
.. module:: pcapkit.dumpkit.null

:mod:`pcapkit.dumpkit.null` is the dumper for :mod:`pcapkit` implementation,
specifically for **NotImplemented** format, which is alike those described in
Expand Down
2 changes: 1 addition & 1 deletion docs/source/pcapkit/dumpkit/pcap.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
PCAP Dumper
===========

.. module:: pcapkit.dumper.pcap
.. module:: pcapkit.dumpkit.pcap

:mod:`pcapkit.dumpkit.pcap` is the dumper for :mod:`pcapkit` implementation,
specifically for PCAP format, which is alike those described in
Expand Down
101 changes: 101 additions & 0 deletions docs/source/pcapkit/foundation/engines/3rdparty.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,43 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. _Scapy: https://scapy.net

.. note::

Constructing this engine imports :mod:`scapy.all`, which is what populates
`Scapy`_'s layer registries -- ``conf.l2types`` and the ``bind_layers``
payload table, both of which exist only as import side effects of the layer
modules. Importing a narrower submodule leaves them empty, and
:class:`~scapy.utils.PcapReader` then returns every frame as one opaque
:class:`~scapy.packet.Raw` layer without raising, so the engine dissected
nothing at all and said so only on :data:`sys.stderr`. See
:meth:`Scapy.__init__` for why naming the layer modules individually is not a
cheaper route to the same place.

One side effect is worth knowing about in advance: :mod:`scapy.all` loads
:mod:`scapy.layers.dcerpc`, which reaches `Scapy`_'s TLS layer and there
triggers a ``CryptographyDeprecationWarning`` from :mod:`cryptography` about
finite-field Diffie-Hellman. It concerns a key-exchange code path
:mod:`pcapkit` never executes, but it subclasses :exc:`UserWarning` rather
than :exc:`DeprecationWarning`, so Python's default filters show it.

:mod:`pcapkit` deliberately does not filter it away -- it is `Scapy`_'s to
emit and the consumer's to silence, on the same footing as every other
category (see :mod:`pcapkit.utilities.warnings`)::

import warnings

from cryptography.utils import CryptographyDeprecationWarning

warnings.filterwarnings('ignore', category=CryptographyDeprecationWarning)

.. autoclass:: pcapkit.foundation.engines.scapy.Scapy
:no-members:
:show-inheritance:

.. autoattribute:: __engine_name__
.. autoattribute:: __engine_module__

.. automethod:: __init__
.. automethod:: run
.. automethod:: read_frame

Expand Down Expand Up @@ -173,8 +203,12 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. autoattribute:: __engine_name__
.. autoattribute:: __engine_module__
.. autoattribute:: __engine_distribution__

.. automethod:: unsupported_reason

.. autoproperty:: dlink
.. autoproperty:: backend

.. automethod:: run
.. automethod:: read_frame
Expand Down Expand Up @@ -320,8 +354,12 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. autoattribute:: __engine_name__
.. autoattribute:: __engine_module__
.. autoattribute:: __engine_distribution__

.. automethod:: unsupported_reason

.. autoproperty:: dlink
.. autoproperty:: backend

.. automethod:: __init__
.. automethod:: run
Expand Down Expand Up @@ -360,6 +398,9 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
.. autoattribute:: __engine_name__
.. autoattribute:: __engine_module__
.. autoattribute:: LAYERS
.. autoattribute:: PYTHON_CEILING

.. automethod:: unsupported_reason

.. autoproperty:: dlink

Expand All @@ -383,3 +424,63 @@ Internal Definitions

.. automethod:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile._get_decoder
.. automethod:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile._decode

Backend Detection
=================

.. module:: pcapkit.foundation.engines._pcap_backend

Two unrelated PyPI distributions install a top-level module named :mod:`pcap` --
`PyPCAP`_, a Cython binding shipped as a single extension module, and `pcap-ct`_,
a :mod:`ctypes` reimplementation shipped as a package. They therefore collide,
and ``import pcap`` resolves to whichever the import system finds first.
:class:`~pcapkit.foundation.engines.pypcap.PyPCAP` and
:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` each have to know which one
they actually got rather than assume, and this module is the one place that
answers it -- deliberately shared, since the two engines must agree and two
copies of the detection would be two chances to disagree. It is *only* detection,
and it imports nothing from :mod:`pcapkit`, so it cannot introduce an import
cycle.

.. autodata:: pcapkit.foundation.engines._pcap_backend.PYPCAP

.. autodata:: pcapkit.foundation.engines._pcap_backend.PCAP_CT

.. autodata:: pcapkit.foundation.engines._pcap_backend.DISTRIBUTIONS

.. autodata:: pcapkit.foundation.engines._pcap_backend.ENGINE_NAMES

.. autoclass:: pcapkit.foundation.engines._pcap_backend.Probe
:no-members:
:show-inheritance:

.. note::

This is an :class:`~pcapkit.corekit.infoclass.Info` subclass, so it is a
:class:`~collections.abc.Mapping` rather than a :class:`tuple`: its fields
are reached by name, not by position, and it cannot be unpacked as a
sequence.

.. autoattribute:: name
.. autoattribute:: version
.. autoattribute:: origin
.. autoattribute:: failure
.. autoattribute:: missing
.. autoattribute:: installed

.. automethod:: describe

.. autofunction:: pcapkit.foundation.engines._pcap_backend.probe

.. autofunction:: pcapkit.foundation.engines._pcap_backend.identify

.. autofunction:: pcapkit.foundation.engines._pcap_backend.installed_distributions

.. autofunction:: pcapkit.foundation.engines._pcap_backend.wrong_backend_reason

.. autofunction:: pcapkit.foundation.engines._pcap_backend.collision_reason

Internal Definitions
--------------------

.. autofunction:: pcapkit.foundation.engines._pcap_backend._purge
2 changes: 2 additions & 0 deletions docs/source/pcapkit/foundation/engines/engine.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ all engine support functionality.

.. autoproperty:: extractor

.. automethod:: unsupported_reason

.. automethod:: run
.. automethod:: read_frame
.. automethod:: close
Expand Down
10 changes: 5 additions & 5 deletions docs/source/pcapkit/foundation/engines/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -122,9 +122,10 @@ so it is worth knowing in advance.
| | loop with :func:`asyncio.get_event_loop`, which raises from |
| | 3.14 |
+-----------------------------------------------------------------+---------------------------------------------------------------+
| :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` | `libpcap`_ headers and library, a C compiler, and Python |
| | **3.11 or older** -- ``pypcap`` 1.3.0 publishes no wheel and |
| | its pre-generated :file:`pcap.c` does not compile on 3.12+ |
| :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` | :manpage:`libpcap(3)` headers and library, a C compiler, and |
| | Python **3.11 or older** -- ``pypcap`` 1.3.0 publishes no |
| | wheel and its pre-generated :file:`pcap.c` does not compile |
| | on 3.12+ |
+-----------------------------------------------------------------+---------------------------------------------------------------+
| :class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` | a system ``libpcap.so.1`` at *run* time -- nothing to build, |
| | since ``pcap-ct`` and ``libpcap`` ship pure-Python wheels, |
Expand All @@ -137,7 +138,7 @@ so it is worth knowing in advance.

Every one of these constraints is also enforced in code rather than only
documented: each engine overrides
:meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, which
:meth:`~pcapkit.foundation.engines.engine.Engine.unsupported_reason`, which
:meth:`Extractor.run <pcapkit.foundation.extraction.Extractor.run>` consults
*before* the import test, so asking for an engine that cannot run here produces
one warning naming the actual cause and a clean fall back to the built-in parser.
Expand Down Expand Up @@ -214,4 +215,3 @@ actually got, via
.. _PyPCAP: https://github.com/pynetwork/pypcap
.. _pcap-ct: https://pypi.org/project/pcap-ct/
.. _PyPCAPFile: https://github.com/kisom/pypcapfile
.. _libpcap: https://www.tcpdump.org
15 changes: 9 additions & 6 deletions docs/source/pcapkit/foundation/extraction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,13 @@ extracts parametres from a PCAP file.

.. seealso::

Engine support for |pypcap|_ and |pypcapfile|_ has since landed, as
:class:`pcapkit.foundation.engines.pypcap.PyPCAP` (``engine='pypcap'``) and
:class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile`
(``engine='pypcapfile'``). Both support less than the ``default`` engine
does; :doc:`engines/index` tabulates the gaps, and :doc:`../../index`
documents the installation prerequisites each of the two carries.
Engine support for |pypcap|_, |pcap-ct|_ and |pypcapfile|_ has since landed,
as :class:`pcapkit.foundation.engines.pypcap.PyPCAP` (``engine='pypcap'``),
:class:`pcapkit.foundation.engines.pcap_ct.PCAP_CT` (``engine='pcap_ct'``)
and :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile`
(``engine='pypcapfile'``). All three support less than the ``default``
engine does; :doc:`engines/index` tabulates the gaps, and :doc:`../../index`
documents the installation prerequisites each of the three carries.

.. autoclass:: pcapkit.foundation.extraction.Extractor
:no-members:
Expand Down Expand Up @@ -91,5 +92,7 @@ Type Variables

.. |pypcap| replace:: ``pypcap``
.. _pypcap: https://github.com/pynetwork/pypcap
.. |pcap-ct| replace:: ``pcap-ct``
.. _pcap-ct: https://pypi.org/project/pcap-ct/
.. |pypcapfile| replace:: ``pypcapfile``
.. _pypcapfile: https://github.com/kisom/pypcapfile
4 changes: 2 additions & 2 deletions docs/source/pcapkit/foundation/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ Library Foundation

:mod:`pcapkit.foundation` is a collection of foundations for
:mod:`pcapkit`, including PCAP file extraction tool
:class:`~pcapkit.foundation.extraction.Extrator`, TCP flow tracer
:class:`~pcapkit.foundation.tractflow.TraceFlow`, registry management
:class:`~pcapkit.foundation.extraction.Extractor`, TCP flow tracer
:class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow`, registry management
APIs for :mod:`pcapkit`, and TCP/IP reassembly implementations.

.. toctree::
Expand Down
6 changes: 3 additions & 3 deletions docs/source/pcapkit/foundation/reassembly/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,9 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.reassembly`:
click C "/pcapkit/foundation/reassembly/reassembly.html#pcapkit.foundation.reassembly.reassembly.Reassembly"
click D "/ext.html#reassembly-and-flow-tracing"

click IP "/pcapkit/foundation/reassembly/ip/index.html#pcapkit.foundation.reassembly.ip.IP"
click IPv4 "/pcapkit/foundation/reassembly/ip/ipv4.html#pcapkit.foundation.reassembly.ip.ipv4.IPv4"
click IPv6 "/pcapkit/foundation/reassembly/ip/ipv6.html#pcapkit.foundation.reassembly.ip.ipv6.IPv6"
click IP "/pcapkit/foundation/reassembly/ip/ip.html#pcapkit.foundation.reassembly.ip.IP"
click IPv4 "/pcapkit/foundation/reassembly/ip/ipv4.html#pcapkit.foundation.reassembly.ipv4.IPv4"
click IPv6 "/pcapkit/foundation/reassembly/ip/ipv6.html#pcapkit.foundation.reassembly.ipv6.IPv6"
click TCP "/pcapkit/foundation/reassembly/tcp.html#pcapkit.foundation.reassembly.tcp.TCP"

Auxiliary Data
Expand Down
2 changes: 1 addition & 1 deletion docs/source/pcapkit/foundation/reassembly/ip/ip.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ Type Variables
--------------

.. data:: pcapkit.foundation.reassembly.data.ip._AT
:type: ipaddress.IPv4Address | ipaddress.IPv4Address
:type: ipaddress.IPv4Address | ipaddress.IPv6Address
Loading