fix(ipv6): parse unrecognised extension headers generically so the chain survives

[JarryShaw]

Describe the bug

When an IPv6 extension header fails to parse, @beholder substitutes a Raw — but IPv6's own chain-walk then reads .next off it, which Raw does not carry:

ipv6.py:329   info = next_.info      # next_ is whatever _import_next_layer returned
ipv6.py:338   proto = info.next      # AttributeError when next_ is the Raw fallback

The consequence is not confined to the extension header. The AttributeError escapes IPv6.read before IPv6's own info is returned, so the @beholder one level up turns the whole IPv6 packet into Raw — source, destination, hop limit and flow label lost along with the extension header.

Reproduction

Measured on main at 21b121c0e, and on PR #889's head, with PYTHONSAFEPATH=1 and pcapkit.__file__ asserted to the tree under test. A well-formed Mobility Header (an IPv6 extension header — Mobility_Header = 135 is in pcapkit/const/ipv6/extension_header.py:42) whose status byte is unassigned:

Ethernet/IPv6  assigned status   -> payload=IPv6 protochain=Ethernet:IPv6:MH
Ethernet/IPv6  unassigned status -> payload=Raw  protochain=Ethernet:Internet_Protocol_version_6
IPv6 alone     unassigned status -> AttributeError: 'Raw' object has no attribute 'next'

Note both IPv6 and MH are absent from the degraded protochain.

It is not specific to that enum. On unmodified main, two unrelated MH failure modes raise the identical AttributeError: a truncated MH (6 bytes) and an absurd Header Len of 0xff. So any extension-header parse failure reaches it.

Expected behavior

A failed extension header degrades to Raw in place, leaving the IPv6 header's own parsed fields intact and the rest of the chain walked as far as it can be — which is what @beholder exists to provide. At minimum the walk should stop cleanly on a Raw rather than raising AttributeError, since Raw is a value _import_next_layer is documented to return.

System information

• Python 3.14.7, CPython, checkout at 21b121c0e.

Additional context

Surfaced while reviewing #889, which stops four mh.py helper enums from minting on an unassigned byte. #889 does not introduce this — it makes it far more reachable, turning a malformed-packet-only path into one a well-formed packet with an unrecognised status byte takes. It is out of scope there and #889 documents the real cost instead.

This also corrects a claim I made on #877 when recommending those enums raise: I said the cost was one MH message's parse. On the IPv6 path it is the whole IPv6 packet. See issues/877 for the correction.

[JarryShaw]

One design decision needed before this is implemented, because "degrade in place" can mean three different things and they are not equivalent.

The mechanism is settled and measured: ipv6.py:329 takes info = next_.info, :338 does proto = info.next, and a Raw fallback's info carries no .next, so the AttributeError escapes IPv6.read before IPv6's own info is returned and the outer @beholder replaces the whole packet. Confirmed pre-existing — a truncated extension header and an absurd Header Len both reach it on unmodified main.

What is not settled is what should happen instead. Three candidates:

(1) Stop the chain walk cleanly on a Raw. Treat a Raw from _import_next_layer as end-of-chain: keep IPv6's own parsed fields, attach the Raw as the final layer, stop walking. A capture with a bad Mobility Header would then read Ethernet:IPv6:Raw instead of Ethernet:Internet_Protocol_version_6, with source, destination, hop limit and flow label intact.

(2) Let the walk continue past it. Would need Raw to carry a usable next, which it cannot — the whole reason it is a Raw is that the header did not parse, so the next-header byte is not trustworthy. I do not think this is viable and mention it only to rule it out explicitly.

(3) Leave the wholesale degrade, and only stop it being an AttributeError. Catch it, log it, and let the packet become Raw as it does today — cheaper, but it keeps losing the IPv6 header fields, which is the actual damage.

My recommendation is (1). It is what @beholder exists to provide — a bad layer costs that layer — and it is the only option where an operator inspecting a capture still sees the addresses. It does change observable output for malformed captures (protochain gains a Raw tail where the packet used to vanish into one), so it wants a breaking label and a test pinning the new shape.

One consequence you should weigh, because it cuts the other way: #889 added a test asserting today's wholesale-degrade behaviour precisely so that fixing this would break it and force a look. Taking (1) means that test gets re-pointed rather than deleted — which is correct, but it means the shape lands in two places and the changelog entry for #889 will describe behaviour the library no longer has by then.

Labelled needs: decision. Nothing dispatched until you rule.

[JarryShaw]

But i think IPv6 extension headers share the same first few fields (including length) formats? Then we can actually have a RawExtension fallback that carries the unparsed payload as if a Raw protocol then next to the actual next one.

[JarryShaw]

Good idea, and mostly right — but I measured the layouts and there are three exceptions that decide the design. Field order taken from the schemas on main:

header	proto	first fields
HOPOPT	0	next, len, options
IPv6_Opts	60	next, len, options
IPv6_Route	43	next, length, type, seg_left
MH	135	next, length, type, reserved, chksum
HIP	139	next, len, pkt, ver, …
AH	51	next, len, reserved, spi, seq
IPv6_Frag	44	next, reserved, flags, id — no length at all
ESP	50	no next, no length — has neither
Shim6	140	no schema in the tree

So next at offset 0 holds for seven of the nine, which is what makes your RawExtension idea work — the walk can keep going, which is strictly better than my option (1) of stopping. But the length needed to skip to that next header is not uniform:

• IPv6_Frag is fixed at 8 octets — byte 1 is reserved, not a length. A generic reader must special-case 44.
• AH's len is in 4-octet units minus 2, not the 8-octet-units-minus-1 that HOPOPT/Opts/Route/MH/HIP use. Same field position, different arithmetic.
• ESP has neither field — its next-header byte is inside the encrypted payload, so the chain genuinely ends there and no fallback can continue it.

So the shape I would build, if you agree: RawExtension parses next at offset 0 and computes its own length from a per-protocol rule (8-octet units for the five, fixed 8 for Frag, 4-octet units for AH), carries the remainder as opaque bytes like Raw, and hands next back so ipv6.py:338's walk continues to the real next header. ESP stays a terminal — it already is, and cannot be otherwise. That gets an operator the IPv6 header, a marker for the unparsable extension, and every layer after it, which none of my three options delivered.

Two things to decide, since they are yours not mine:

1. Should RawExtension trust a length it cannot validate? If the bad header is bad because its length field is garbage, skipping by that length lands mid-packet and the next "header" is noise. Safer is to stop the walk when the computed length would overrun the remaining payload — so it continues where it can and stops where it cannot, rather than fabricating layers.
2. protochain output changes either way — a bad Mobility Header currently makes the whole packet Raw, and would become something like Ethernet:IPv6:RawExtension:UDP. That is the point, but it is breaking for anyone matching on today's shape, and it means re-pointing the test fix(mh): stop the four RFC-inline helper enums from minting on unassigned bytes #889 added to pin the current behaviour.

Confirm the shape and I will dispatch it. Keeping needs: decision until you do, since point 1 changes the implementation materially.

[JarryShaw]

Let's re-read IPv6 extension header's RFC documentation? Maybe some of these values are not considered as extension headers? Maybe there have been a standardised format of how legit extension headers should contain/look like so that we can have our RawExtension (name subject to your recommendation based on the library's naming convention and future facing considerations).

But if we really cannot find the trustworthy source or shared format for them, then we fall back to original recommendation on (1) - stop with a Raw containing whatever is remained.

[JarryShaw]

Ruling recorded, verbatim:

Let's re-read IPv6 extension header's RFC documentation? Maybe some of these values are not considered as extension headers? Maybe there have been a standardised format of how legit extension headers should contain/look like so that we can have our RawExtension (name subject to your recommendation based on the library's naming convention and future facing considerations).

But if we really cannot find the trustworthy source or shared format for them, then we fall back to original recommendation on (1) - stop with a Raw containing whatever is remained.

Right question to ask — my table was a census of this library's schemas, which is evidence about our code rather than about what a legitimate extension header is. Investigating against the specifications, with the fallback you named as the default if the answer comes back "no trustworthy shared format".

Four things being established, each to be answered with quoted text rather than summary:

• The authoritative list. IANA's Protocol Numbers registry carries an explicit IPv6 Extension Header column, so the question "are some of these not extension headers?" has a published answer rather than a judgement. That also catches anything the library is missing, which the census could not.
• Whether a standardised format exists. RFC 8200 §4.8, Defining New Extension Headers and Options, is the section most likely to settle it — whether the layout requirement is normative, and whether it binds the headers defined before RFC 8200 or only new ones.
• The three non-conformers individually, from their own specs: IPv6_Frag's fixed 8 octets (RFC 8200 §4.5), AH's Payload Length unit and bias (RFC 4302, to be read rather than recalled), and ESP's next-header byte living inside the encrypted payload (RFC 4303). Plus Mobility_Header, HIP, Shim6, BIT_EMU and whether 253/254 are extension headers at all or merely reserved protocol numbers.
• The decisive test, stated so it can come back negative: can a generic reader handed an unparsable extension header reliably determine both the next-header value and how many octets to skip? If that is "yes for the conforming set, with a small closed set of special cases", RawExtension is buildable. If not, your fallback applies.

Name deferred to the same investigation, as you asked — against the library's own conventions and where it should live (protocols/internet/ beside the other extension headers, or protocols/misc/ beside Raw).

Keeping needs: decision until the findings are in, since they decide which design you are choosing between.

[JarryShaw]

Findings in, and your instinct was right — there is a trustworthy shared format, so the fallback is stricter than the evidence requires. It is RFC 6564, not RFC 8200 §4.8.

1. IANA says all twelve are extension headers, so that question is settled and not in our favour. The Protocol Numbers registry has an explicit column, and the Y rows are an exact element-for-element match with pcapkit/const/ipv6/extension_header.py — nothing extra, nothing missing. Unsurprising once you look: the module is generated from that column. (The CSV is protocol-numbers-1.csv; the unsuffixed URL 404s.)

0 HOPOPT   43 IPv6-Route   44 IPv6-Frag   50 ESP   51 AH   60 IPv6-Opts
135 Mobility Header   139 HIP   140 Shim6   147 BIT-EMU   253   254

2. But the RFCs do narrow it, and the format requirement is a real MUST. RFC 6564, A Uniform Format for IPv6 Extension Headers, Standards Track, Updates 2460, RFC 2119 keywords declared. §4, verbatim:

Any IPv6 extension headers defined in the future … MUST use the consistent format defined in Figure 1.

Hdr Ext Len 8-bit unsigned integer. Length of the extension header in 8-octet units, not including the first 8 octets.

And §5 answers the retroactivity question outright, verbatim:

The scheme proposed in this document is not intended to be backward compatible with all the currently defined IPv6 extension headers. It applies only to newly defined extension headers. Specifically, the fragment header predates this document and does not follow the format proposed in this document.

Cite RFC 6564 §4, not RFC 8200 §4.8 — I had pointed at the wrong section. RFC 8200 §4.8 restates the layout without an RFC 2119 keyword ("they need to use the following format") and its field description carries a copy-paste defect, describing the generic length field as "Length of the Destination Options header".

3. So the answer to your decisive question is "yes, with a closed six-entry exception table":

	next header	skip length
HOPOPT 0, IPv6_Route 43, IPv6_Opts 60, MH 135, HIP 139, Shim6 140	octet 0	(octet1 + 1) * 8
IPv6_Frag 44	octet 0	constant 8 — octet 1 is Reserved (RFC 8200 §4.5)
AH 51	octet 0	(octet1 + 2) * 4 — RFC 4302 §2.2 flags its own deviation
ESP 50, BIT_EMU 147, 253, 254	no	no — terminal

The four terminals are terminal for stated reasons, not by inference: RFC 8200 §4.5 says "the Encapsulating Security Payload (ESP) is not considered an extension header" and RFC 4303 puts its Next Header in a trailer after a variable-length encrypted payload with no length field anywhere; RFC 9801 defines 147 as a payload type ("the payload is an emulated bit-stream") with no header layout at all, making IANA's Y a filing artefact; and RFC 3692 assigns 253/254 for local experiment with no wire format by design.

Recommendation: build it. Six conformers today, plus every extension header standardised from April 2012 onward by the MUST — which is precisely the population a generic fallback exists to serve. Two need a documented per-value length rule, four terminate into a plain Raw. That is enumerable and spec-cited, not heuristic. So: stop the walk, but only at those four values.

Name: RawExtension, in pcapkit/protocols/internet/raw_extension.py, subclassing Internet — your name, kept. The argument for the location is that Raw is defined by refusing to continue the chain: protocol raises UnsupportedCall, length is hardcoded 0, _next is always NoPayload(). RawExtension's whole purpose is the opposite — to yield a real next-header value and a real octet length. It belongs beside the extension headers it stands in for. It should still follow Raw's precedent of raising UnsupportedCall from __index__, having no single protocol number.

Two things still yours to say before I dispatch: whether it should refuse to skip when the computed length would overrun the remaining payload (I would — a garbage length field is the likeliest reason the header failed to parse, and skipping by it fabricates layers), and confirmation that protochain changing from Ethernet:Internet_Protocol_version_6 to Ethernet:IPv6:RawExtension:UDP is breaking and re-points the test #889 added to pin today's behaviour.

[JarryShaw]

whether it should refuse to skip when the computed length would overrun the remaining payload

yes, it should, or actually i think the current existing workflow or house convention already handles/covers that.

confirmation that protochain changing from Ethernet:Internet_Protocol_version_6 to Ethernet:IPv6:RawExtension:UDP is breaking

yes confirmed - the RawExtension name is still up to your recommendations, as well as where it should be housed.

[JarryShaw]

Ruling recorded, verbatim:

whether it should refuse to skip when the computed length would overrun the remaining payload

yes, it should, or actually i think the current existing workflow or house convention already handles/covers that.

confirmation that protochain changing from Ethernet:Internet_Protocol_version_6 to Ethernet:IPv6:RawExtension:UDP is breaking

yes confirmed - the RawExtension name is still up to your recommendations, as well as where it should be housed.

You are right that a convention already exists, and it is worth naming precisely because it does not quite cover this case. The house pattern is warn-and-clip: emit a SchemaWarning saying what was declared against what remains, then read what remains.

pcapkit/protocols/schema/misc/pcapng.py:373   "option declares {nominal} octet(s) of payload with
                                               {remaining} octet(s) left in its area; reading {remaining}"
pcapkit/protocols/schema/misc/pcapng.py:443   the same shape for a block's option area
pcapkit/corekit/fields/field.py:507           "Field {name} declares a length of {length} octet(s), ..."
pcapkit/protocols/misc/pcapng.py:1119         "{length} exceeds the {read} octet(s) left in ..."

The gap: clipping a payload and clipping a skip-distance are different operations. Clipping is right when the declared length only governs how much of this header to read — you take what is there and carry on. But RawExtension's length also decides where the next header starts, and a clipped skip-distance points at whatever happens to sit at the end of the buffer, which is not a header. Continuing from there fabricates a layer out of trailing bytes, which is worse than stopping: a wrong protochain entry reads as a parsed fact.

So the implementation will follow the convention's warning and diverge on the action: emit a SchemaWarning in the established wording, then stop the walk rather than clip-and-continue — RawExtension absorbs the remaining octets and reports no next header. The chain ends honestly at the bad header instead of inventing what follows it. I will say so in the docstring, citing the convention so the divergence reads as deliberate rather than as an oversight.

Name and location, per your deferral: RawExtension, in pcapkit/protocols/internet/raw_extension.py, subclassing Internet. Raw is defined by refusing to continue a chain — protocol raises UnsupportedCall, length is hardcoded 0, _next is always NoPayload(). RawExtension exists to do the opposite for a conforming header, so it belongs beside the extension headers it substitutes for rather than beside Raw. It will follow Raw's precedent of raising UnsupportedCall from __index__, having no single protocol number of its own.

needs: decision removed, breaking and wip applied, worker dispatched. Scope: the six RFC 6564 conformers walk generically, IPv6_Frag uses a constant 8 and AH uses (len + 2) * 4, and ESP/BIT_EMU/253/254 terminate into a plain Raw as they do today.

[JarryShaw]

Confused tho. Now that we confirmed that IPv6 extension headers should have the uniformed format, why dont we make RawExtension dynamic, meaning, it reads and parses according to the shared format then its property is still able to return the necessary information as index, next, length, protocol (derived from the actual Enum value), etc.?

RawExtension name may sound ambiguous - it's not directly pointing to IPv6 extension headers.