Skip to content

NumberField with a callable length cannot pack at 8 octets: _need_process latches True from the placeholder and is never cleared #591

Description

@JarryShaw

A NumberField whose length is a callable is unpackable once that callable resolves to 8 octets: _need_process is latched True while the field is still a placeholder and is never cleared when the real length arrives, so pre_process returns bytes to a struct template that now expects an integer.

Mechanism

numbers.py:109 sets the flag false in __init__, and the template builder sets it true for any length that is not 8, 4, 2 or 1:

else:                 # do not unpack
    struct_fmt = f'{length}s'
    self._need_process = True

A callable length is a placeholder of -1 at construction, which takes that branch. When __call__ later resolves the real width the template is rebuilt correctly — but nothing resets _need_process. So pre_process at numbers.py:194 takes the process path, self._length is now 8 rather than negative so the _length < 0 repair at :196-203 is skipped, and :205 returns

return value.to_bytes(self._length, self._byteorder, signed=self._signed)

bytes, into a template that is by then >Q.

Measured

Executed on origin/main (9c240a60e), CPython 3.14.7. The comparison is the proof — identical resolved length and identical template, differing only in how the length was supplied:

--- callable length (resolves to 8) ---
length= 8  _need_process= True   template= >Q
  error: required argument is not an integer

--- static length 8, for comparison ---
length= 8  _need_process= False  template= >Q
pack -> 0000010000000000

So NumberField(length=lambda pkt: 8) cannot pack, while NumberField(length=8) packs the same value through the same template.

Impact on the wire

This makes every extended (8-octet) MPTCP DSS form unpackable, since those widths are selected at runtime from the DSS flags and therefore must come from a callable. Reported as:

_make_mptcp_dss(DSS, ack=1 << 40)  ->  struct.error: required argument is not an integer

Any other schema that resolves a numeric width to 8 octets through a callable has the same failure.

Notes

Activity

  1. JarryShaw commented on Sep 21, 2026

    @JarryShaw
    OwnerAuthor

    Scope correction to this issue, established by the fix in #598 and verified here.

    This issue understated the defect in two ways.

    1. All four native widths were affected, not only 8. The body said whether 1/2/4 are equally affected "is NOT something I checked". They are: a callable length resolving to 1, 2, 4 or 8 each latched _need_process = True against templates >B, >H, >I and >Q respectively, and each raised required argument is not an integer. Width 3 correctly packed throughout, because a 3-octet field genuinely does need byte processing. The latch has nothing to do with the width it latches into — which is obvious in hindsight and is why the 8-octet DSS case was simply the one that got noticed.

    EnumField was affected identically, since it also leaves __template__ unset. The eight subclasses that fix __template__ never latched anything, because __init__ skips build_template for them.

    2. Parsing was broken in the mirror direction, not just packing. The title says "cannot pack". The same flag made post_process (numbers.py:220-224) call int.from_bytes on the integer struct.unpack had already produced. So the field could not read what it could not write.

    On the fix's shape, since this issue speculated about needing to distinguish "placeholder" from "genuinely needs processing": #598 does not add such a bit. It makes build_template assign _need_process rather than only ever raise it, so the flag becomes a function of the width currently in force — True for the -1 placeholder, False for 8, True again for 3. The placeholder's True was never wrong; it was only sticky.

    One separate defect was found on the same path and is filed independently, because #598 neither fixes nor masks it: pre_process's repair at numbers.py:198 uses math.ceil(value.bit_length() // 8), which is floor division, so every octet boundary is sized one octet short and raises OverflowError.

  2. added
    bugIssues reporting a defect (set by the bug report template; a default, not an assessment)
    on Sep 22, 2026
  3. added this to the 1.5 milestone on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssues reporting a defect (set by the bug report template; a default, not an assessment)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions