Repository navigation
fix(toolkit): key pyshark link-type resolution on frame.encap_type - #850
Conversation
* `packet.layers[0].layer_name` is a Wireshark display-filter name, and one name serves several DLTs, so resolving from it answered a valid but wrong DLT with no error: 101 for `rawip4`/`rawip6`, 0 for a `DLT_LOOP` capture, 9 for `ppp-with-direction`, 203 for `linux-lapd`. * Resolve from `frame.encap_type` instead, through a new `ENCAP_TYPE_TO_LINKTYPE` of 152 entries; the filter name is now only a fallback for a frame layer that carries no such field. A `WTAP_ENCAP_*` number is not a DLT, hence the table rather than a cast. * Expand `FILTER_NAME_TO_LINKTYPE` from 2 to 58 entries, per the ruling on #842: every filter name measured as unambiguous, and no upper-case fallback onto a like-named `LinkType` member -- that fallback is what returned the wrong DLTs above. * Both tables were generated by measurement, not transcribed: each entry is one `editcap -F pcap -T <encap>` rewrite, its DLT read from the resulting pcap file header and its key read back with `tshark -T pdml` 4.6.9. The `editcap` name sits in a comment on every line so an entry can be re-checked when upstream moves. * An encapsulation or name outside the tables still raises `MissingKeyError` rather than standing in a plausible DLT. tests/toolkit: 69 passed, 11 skipped, 257 subtests; pyshark.py at 100% statement and branch coverage.
|
GOOD TO GO on
One thing it explicitly did not verify, matching my own caveat: no Sphinx build was run, so the Ready for you to merge. |
|
Wondering if the ENCAP_TYPE and DISPLAY_FILTER maps can be generated by a vendor script or maybe just a util script - so that we don't have to maintain them manually. They may exist in toolkit still - if generated by scripts. |
|
Recorded, verbatim: "Wondering if the ENCAP_TYPE and DISPLAY_FILTER maps can be generated by a vendor Yes — and the generator effectively already exists, since that is how these tables were produced. But a
Two things such a script has to handle, both learned the hard way while producing this table:
Filed as its own issue rather than folded into #850, since #850 is a defect fix and this is tooling — |
…ap_type pcapkit/toolkit/pyshark.py resolved a frame's link type from packet.layers[0].layer_name, a Wireshark display-filter name, with a blanket Enum_LinkType.get(name.upper()) fallback. One filter name serves several encapsulations, so that fallback returned wrong DLTs silently -- null for a DLT_LOOP capture, 101 (RAW) for rawip6. Re-keyed onto frame.encap_type, which is distinct per encapsulation, and the fallback is gone: an unrecognised encapsulation or filter name now raises MissingKeyError. Both lookup tables were measured through editcap/tshark round trips rather than transcribed from Wireshark source, which is not on the build host: ENCAP_TYPE_TO_LINKTYPE grows to 152 entries and FILTER_NAME_TO_LINKTYPE grows from #838's 2 entries to 58.
Adds util/pyshark_encap_map.py, a generator that regenerates ENCAP_TYPE_TO_LINKTYPE (152 entries) and FILTER_NAME_TO_LINKTYPE (58) in place inside pcapkit/toolkit/pyshark.py, so the two tables #850 hand-built stop being hand-maintained. Guards against its own worst failure mode: a naive LinkType(dlt) lookup cannot detect an unmapped DLT, because _missing_ mints a placeholder rather than raising, so the generator snapshots every known value before any lookup. The real-tshark sweep is gated on a new HAS_WIRESHARK flag and is version-pinned -- editcap -T accepts 226 encapsulations on Wireshark 4.6.9, 224 on 4.2.2 (CI's Ubuntu noble) -- so the count assertions run only under the measured version. One line of pyshark.py itself changes as a result: the IPMB_LINUX table entry becomes I2C_LINUX, the canonical name #848 gave value 209. util/changelog_md.py regenerated CHANGELOG.md for all seven entries added across this and the six preceding commits (#838, #846, #847, #848, #849, #850, #853); `--check` exit 0.
#846 and #850 - #838's "the 21 whose vendor crawler leaves Vendor.process() unmodified" measures 22, not 21 -- pcapkit.const.hip.transport.Transport also has an unmodified process(), but its FLAG-bound range has no unassigned gap, so its _missing_ carries no bounded-range extend_enum branch to fix. The real discriminator is that branch, not the process() override; reworded, and the "other 84" split into the 83 that do override process() and the one that doesn't but has nothing to fix either. Re-derived independently against 5e25db2^..5e25db2 (not main, which would fold in #847's own edit to ipx/socket.py): 22 registries have no process() override, 21 of them have the bounded-range branch, matching tests/const/test_const_enum_no_mint.py's own REGISTRIES_WITH_UNASSIGNED_RANGES (21 entries) vs ALL_REGISTRIES (22, +hip.transport, with a NOTE explaining the exclusion). - #846's "Only tests/integration/test_engine_runtime.py and test_engine_parity.py change" is false -- 30bca99's own numstat also touches .github/workflows/unit-tests.yml (17+/2-). Qualified to "only these two test files change". - #850 gets the breaking marker: its own prose already says the fallback is gone and an unrecognised encapsulation or filter name now raises MissingKeyError instead of substituting a plausible DLT -- the same shape as the two existing markers at AppType.get (1.5.0.rst:2456) and the four .get()-backed enum fields (1.5.0.rst:2720). Restructured to lead with the marker and the subject, matching their wording and placement; the later restatement of the same fact is dropped. None of these are code or test changes -- prose only, matching the cross-review's own framing. #848 stays unmarked (its own "Not breaking" paragraph holds); #838, #849 and #853 stay unmarked per explicit instruction not to add markers beyond what was asked.
…ap_type pcapkit/toolkit/pyshark.py resolved a frame's link type from packet.layers[0].layer_name, a Wireshark display-filter name, with a blanket Enum_LinkType.get(name.upper()) fallback. One filter name serves several encapsulations, so that fallback returned wrong DLTs silently -- null for a DLT_LOOP capture, 101 (RAW) for rawip6. Re-keyed onto frame.encap_type, which is distinct per encapsulation, and the fallback is gone: an unrecognised encapsulation or filter name now raises MissingKeyError. Both lookup tables were measured through editcap/tshark round trips rather than transcribed from Wireshark source, which is not on the build host: ENCAP_TYPE_TO_LINKTYPE grows to 152 entries and FILTER_NAME_TO_LINKTYPE grows from #838's 2 entries to 58.
Adds util/pyshark_encap_map.py, a generator that regenerates ENCAP_TYPE_TO_LINKTYPE (152 entries) and FILTER_NAME_TO_LINKTYPE (58) in place inside pcapkit/toolkit/pyshark.py, so the two tables #850 hand-built stop being hand-maintained. Guards against its own worst failure mode: a naive LinkType(dlt) lookup cannot detect an unmapped DLT, because _missing_ mints a placeholder rather than raising, so the generator snapshots every known value before any lookup. The real-tshark sweep is gated on a new HAS_WIRESHARK flag and is version-pinned -- editcap -T accepts 226 encapsulations on Wireshark 4.6.9, 224 on 4.2.2 (CI's Ubuntu noble) -- so the count assertions run only under the measured version. One line of pyshark.py itself changes as a result: the IPMB_LINUX table entry becomes I2C_LINUX, the canonical name #848 gave value 209. util/changelog_md.py regenerated CHANGELOG.md for all seven entries added across this and the six preceding commits (#838, #846, #847, #848, #849, #850, #853); `--check` exit 0.
#846 and #850 - #838's "the 21 whose vendor crawler leaves Vendor.process() unmodified" measures 22, not 21 -- pcapkit.const.hip.transport.Transport also has an unmodified process(), but its FLAG-bound range has no unassigned gap, so its _missing_ carries no bounded-range extend_enum branch to fix. The real discriminator is that branch, not the process() override; reworded, and the "other 84" split into the 83 that do override process() and the one that doesn't but has nothing to fix either. Re-derived independently against 5e25db2^..5e25db2 (not main, which would fold in #847's own edit to ipx/socket.py): 22 registries have no process() override, 21 of them have the bounded-range branch, matching tests/const/test_const_enum_no_mint.py's own REGISTRIES_WITH_UNASSIGNED_RANGES (21 entries) vs ALL_REGISTRIES (22, +hip.transport, with a NOTE explaining the exclusion). - #846's "Only tests/integration/test_engine_runtime.py and test_engine_parity.py change" is false -- 30bca99's own numstat also touches .github/workflows/unit-tests.yml (17+/2-). Qualified to "only these two test files change". - #850 gets the breaking marker: its own prose already says the fallback is gone and an unrecognised encapsulation or filter name now raises MissingKeyError instead of substituting a plausible DLT -- the same shape as the two existing markers at AppType.get (1.5.0.rst:2456) and the four .get()-backed enum fields (1.5.0.rst:2720). Restructured to lead with the marker and the subject, matching their wording and placement; the later restatement of the same fact is dropped. None of these are code or test changes -- prose only, matching the cross-review's own framing. #848 stays unmarked (its own "Not breaking" paragraph holds); #838, #849 and #853 stay unmarked per explicit instruction not to add markers beyond what was asked.
Please follow the guide below
make pylint,make mypy,make isort)make testpasses, and a test case covers the changeWhat is the purpose of your pull request?
fix— corrects a defectDescription of your pull request and other information
Closes #843.
Link-type resolution was keyed on
packet.layers[0].layer_name, a Wireshark display-filter name,with a blanket
Enum_LinkType.get(name.upper())fallback. One filter name serves severalencapsulations, so that fallback returned wrong DLTs silently —
nullfor aDLT_LOOPcapture,101forrawip6. It is now keyed onframe.encap_type, which is distinct per encapsulation, andthe fallback is gone: an unknown encapsulation or name raises
MissingKeyErrorrather thansubstituting a plausible DLT, per the standing ruling on this file.
Per your ruling — "register only when PyShark is used, so still local to toolkit" — nothing is
aliased onto
LinkType; both tables live inpcapkit/toolkit/pyshark.py.The tables were measured, not transcribed — and that is deliberate
Wireshark's source is not on this host, so transcribing
wiretap/pcap-common.cwould have meantcopying a table nobody here could read. Instead each entry is one round trip through upstream's own
table:
editcap -F pcap -T <encap>writes a real capture, the DLT is read from that file's pcapheader (
network, offset 20), and the key is read back withtshark -r … -T pdml. Every entrycarries a trailing comment naming its
editcap -Tsource, per your "comments point to the source oftruth".
ENCAP_TYPE_TO_LINKTYPE— 152 entries, no DLT serving two keys, no key serving two DLTsFILTER_NAME_TO_LINKTYPE— 2 → 58 entries, every root filter name measured single-DLT (includingones that already spell their member, since there is no upper-case fallback left to catch them)
Verification, re-run by me on this rebased branch
editcap -Taccepts 226, 157 writable fromexamples/captures/in.pcap, 69 refused. This corrects a figure I had been quoting as 158/68 —the worker's count is the right one.
15→NULL,174→LOOP,25→LINUX_SLL,210→LINUX_SLL2,7→RAW,129→IPV4,130→IPV6,1→ETHERNET,2→IEEE802_5.loopfile-DLT 108,linux-sll2276,rawip6229 — all match the table.main→ 26 failed. On this branch,tests/toolkit→69 passed, 11 skipped, 257 subtests. Plain
unittestre-derivation →Ran 14 tests … OK,so no
pytest-subtestsundercount.pcapkit/toolkit/pyshark.py→ 100%, 42 stmts / 0 miss, 14 branch / 0 partial.Caveats, stated rather than buried
pyshark.FileCaptureis broken on Python 3.14(
asyncio has no attribute 'set_child_watcher'). The tests use a fake whose semantics were derivedfrom pyshark's source (
LayerFieldsContainer(str, Pickleable);BaseLayer.__getattr__raisingAttributeError;XmlLayer._sanitize_field_namestripping theframe.prefix soframe_info.encap_typeresolvesframe.encap_type). That the field is present on every packet isverified from tshark PDML across all 157 captures — not from a live pyshark run.
ether-nettl/tr-nettlare unswept, soethandtrbeing unique is unproven beyond the sweep. The 69 refused encapsulations have no evidenceeither way and raise.
llc,sctp,pppoes,fpp,irlap. Single-DLT as measured, but weaker evidence; named in the table's comment.editcap -T hhdlc→ DLT 121 has noLinkTypemember (verified). Left out rather than invented;a candidate follow-up for
const/reg/linktype.py, which this PR does not touch... autodata::rendering is unverified. No hyperlinkreference was added, so the
Unknown target namefailure mode is avoided by construction.