ForwardMatchField's non-consuming bytes count toward Schema.__len__, so correct input fails a declared-length check #446

ForwardMatchField exists to look ahead without consuming — it reads bytes so a later field's length can depend on them, and leaves the buffer position alone. But the bytes it matched still occupy a slot in the schema's __buffer__, and Schema.__len__ is defined as the length of the concatenated buffer:

# pcapkit/protocols/schema/schema.py:443
def __len__(self) -> 'int':
    return len(self.__bytes__())

So any schema containing a ForwardMatchField reports itself longer than the octets it actually read, by exactly the width of the forward match. Where that length is then checked against a declared area, the check fails on correct input.

Observed

The CGA Parameters option is the case that exposed it. CGAParameter (pcapkit/protocols/schema/internet/mh.py:499-521) carries

public_key_test: 'ANSIKeyLengthTest' = ForwardMatchField(BitField(length=2, namespace={
    'len': (8, 8),
}))

and is nested by CGAParametersOption through ListField(length=lambda pkt: pkt['length'], item_type=SchemaField(schema=CGAParameter)). With the enclosing-length lookup corrected (that is issue #445, a separate fault that hits first), a well-formed 40-octet option then fails:

FieldValueError: Field parameters has invalid length.

because the ListField compares its declared area against item lengths that are each 2 octets too large.

The same root is already filed once

#441 is the same mechanism seen from the other end: ipv6_opts.py's SMF-DPD schema carries a test ForwardMatchField that hopopt.py's equivalent does not, so identical octets parse under HOPOPT and fail under IPv6-Opts, the nested schema reporting 4 octets where it read 3. That issue can be closed by deleting the stray field, which is worth doing on its own — but it treats the symptom. As long as a non-consuming field counts toward len(schema), the next schema that legitimately needs a forward match hits the same wall, and CGAParameter is a case where the field is not stray and cannot simply be deleted: the public key's length genuinely has to be read before the key can be sized.

Fix

The question is what len(schema) is for. It is used as "how many octets did this schema account for", which is the consumed count — so a non-consuming field should not contribute. Either:

• exclude forward-match slots from __bytes__/__len__, keeping them in __buffer__ for value access; or
• keep __len__ as the buffer length and give schemas a separate consumed-octet measure, then use that everywhere a declared area is checked.

The first is less disruptive but changes what bytes(schema) round-trips; the second is more honest but touches every length check. Worth a deliberate choice, since both OptionField and ListField depend on the answer.

Reproduction note

A minimal standalone repro is currently blocked by a third, unrelated fault: Schema.unpack(data, length) cannot be called at all (#444), so the isolated case has to be driven through a protocol or with three positional arguments. Once #444 is fixed a two-field schema plus one ForwardMatchField demonstrates this in three lines.

Raised out of docs/source/pep.rst, which had been carrying this defect as prose. That page tracks feature requests, not defects.