Skip to content

perf(const): load the TCP and UDP app-type registries on first use (#1538) - #1560

Merged
JarryShaw merged 1 commit into
mainfrom
perf/1538-p7-lazy-apptype
Oct 10, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
perf/1538-p7-lazy-apptype

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Please follow the guide below

What is the purpose of your pull request?

Tick the commit type your subject line carries.

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • release — bumps the version or rolls up a distribution
  • chore — anything else

Description of your pull request and other information

Refs #1538 (P7). pcapkit/const/reg/apptype/__init__.py now imports tcp/udp on first use: a PEP 562 __getattr__ for TCP/UDP/tcp/udp, and AppType.__registries__ holds a ModuleDescriptor per lazy registry until a read resolves it. Nothing on any import path needed them eagerly (import pcapkit, pcapkit.all, from pcapkit import *, extraction, engines, registry, transport/schema modules all checked). The other proposed importers (registry/protocols.py, fields/numbers.py, corekit/enum.py, const/reg/__init__.py) only import the base AppType, so they are unchanged.

Measured (3 runs each): -X importtime -c 'import pcapkit' 695/700/722 → 516/521/503 ms (tcp+udp were 206–214 ms, 29.5%, not 33%). tests/utilities/test_logging.py + test_const_enum_no_mint.py + test_const_apptype_split_unit.py: 43.0/42.8/43.1 → 37.7/37.4/40.8 s. One run of tests/const: 84.8 → 75.3 s, executing tcp.py 77 → 27 times (udp.py 77 → 26). Transport + application + capture round trip: tcp.py 76 → 16, udp.py 76 → 6 executions; 174.9/165.8/166.1 → 183.6/156.9/158.1 s (the first pair ran at load ~6, the rest at ~3). Estimate: ≤ ~0.19 s per cold re-import, minus ~0.1 s back for each registry a class then touches; per leg unmeasured.

Unchanged, pinned by tests: dir(), __all__, star-import, __registries__ keys/order/len/in and every read view, pickle/copy of members, a port registered before vs after first use, vendor output (regenerated all four from one IANA CSV under both inits: identical, and identical to the committed files). Suites green one directory at a time: const, corekit, protocols (all), foundation, interface, dumpkit, utilities, vendor, cli, toolkit, project, integration, top-level.

Contract changes for the breaking call:

  • A bare import pcapkit no longer leaves pcapkit.const.reg.apptype.tcp/udp in sys.modules, nor TCP/UDP in the package __dict__.
  • AppType.__registries__ is a dict subclass; __registries__.get costs 129 ns vs 56 ns, AppType.get 761 vs 691 ns.
  • Test harness: tests/conftest.py now imports both registries before pinning its module snapshot. Without that, the restore dropped them after the first test that loaded them, and a later direct import built a second TCP/UDP; under xdist this failed test_the_audit_population_figures_are_the_measured_ones with 127 != 129 on 3.10–3.13. SharedImportProbe/PinnedSnapshotTests fail without the warm-up.

@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 10, 2026
@JarryShaw JarryShaw added perf Pull requests that improve performance (perf: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 90.03% (unit tier, Python 3.14, ed83fd807, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18858 842 2366 756 92.11%
pcapkit/corekit 2240 67 722 34 96.32%
pcapkit/dumpkit 258 2 90 2 98.85%
pcapkit/foundation 2766 129 980 47 94.23%
pcapkit/interface 108 7 36 5 91.67%
pcapkit/protocols 16996 230 4548 191 98.02%
pcapkit/toolkit 635 98 198 5 83.31%
pcapkit/utilities 431 4 124 4 98.56%
pcapkit/vendor 4409 2342 1006 157 43.25%

Per-file detail: the coverage-html artifact of this run.

…1538)

pcapkit.const.reg.apptype imported its tcp and udp submodules (about 49,700
generated lines) with the package, though nothing on any import path reads
them; they were about 30% of `import pcapkit`.

- The package binds TCP and UDP, and the tcp and udp submodules, from a
  PEP 562 __getattr__ on first access. dir(), __all__ and star-imports
  answer as before.
- AppType.__registries__ becomes a dict subclass that holds a
  ModuleDescriptor for each lazy registry until a read resolves it, so
  AppType.get() and PortEnumField load only the transport they dispatch to.
  Keys, their order and every read view are unchanged.
- SCTP and DCCP cost about 3 ms together and stay eager.
- tests/conftest.py imports both registries before pinning its module
  snapshot, so its restore no longer leaves a second TCP and UDP behind.

New tests show a cold import leaves both unimported, pin every read path
and the conftest pin; the const, corekit, protocols, foundation and project
suites pass.
@JarryShaw
JarryShaw force-pushed the perf/1538-p7-lazy-apptype branch from 6bd1988 to ed83fd8 Compare October 10, 2026 04:05
@JarryShaw JarryShaw added review: pending No verdict for the current head - never reviewed, or the head moved since the last one review: running A cross-review is in flight against the current head - no verdict yet and removed review: running A cross-review is in flight against the current head - no verdict yet review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Oct 10, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at ed83fd807. Cross-reviewed on Sonnet; the author ran Opus.

  • Laziness: apptype.tcp and apptype.udp stay unimported after import pcapkit, pcapkit.all, a star-import and 12 other entry points. All 21 first-read routes resolve on first use: get, __registries__[...], iteration, ==, from ... import TCP, and so on. I checked the cold-import case myself.
  • Behaviour: dir(), __all__, key order, pickle, copy and get identity all match main. The one visible change is that type(AppType.__registries__) is dict is now False. In 16-thread barrier runs, tcp.py and udp.py were each executed at most once. I don't think this needs breaking.
  • CI fix: the conftest warm-up fixes the +2 counts failure. The old head fails 3 tests on the ordering repro; the new head passes 1039 of 1039 across tests/const, transport and project, in both orders. The laziness tests use child processes, so the warm-up can't hide a regression.
  • Fail-before: 11 of 14 new tests fail on main's library. The 3 that pass are conftest guards, as intended.
  • Numbers: import time goes from 695–722 ms to 499–521 ms. The two tests/const figures in the body didn't reproduce. A single run of 84.8 → 75.3 s came out at about 0–5 s net (89–95 s → 89–90 s), because the new test file costs about as much as it saves. AppType.get measured +15% rather than +10% (620 → 715 ns).

Non-blocking nit: the comment above __registries__ still says it is populated "once all four have been imported". That text is at pcapkit/const/reg/apptype/apptype.py:2400 and in the vendor template pcapkit/vendor/reg/apptype/apptype.py:355, and it is stale now that two entries start as descriptors.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 10, 2026
@JarryShaw
JarryShaw merged commit 2770404 into main Oct 10, 2026
41 checks passed
@JarryShaw
JarryShaw deleted the perf/1538-p7-lazy-apptype branch October 10, 2026 04:38
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 10, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

perf Pull requests that improve performance (perf: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant