Skip to content

Add: adopt #2185's kernel entry hardening on the integration line - #2247

Merged
YunjiQin merged 1 commit into
hw-native-sys:feat/kernel-mode-integration-testfrom
YunjiQin:feat/adopt-2185-entry-hardening
Sep 15, 2026
Merged

YunjiQin merged 1 commit into
hw-native-sys:feat/kernel-mode-integration-testfrom
YunjiQin:feat/adopt-2185-entry-hardening

Conversation

@YunjiQin

Copy link
Copy Markdown
Collaborator

Summary

D15 took only the test hardening from #2185's refreshed head and deferred the
rest as accompanying a signature change; D16 has since adopted that signature
independently. What is left is no longer a signature question, and one part of
it closes a real leak.

The leak. ChipWorker::kernel_init dropped the device context on any
refused init, on the stated ground that a refused kernel init "has adopted
nothing — it took no ACL state and bound no thread". That is false here:
init_kernel_context acquires the context claim, then
kernel_exec_state_.initialize creates two streams and five events, and the
four steps after it — ensure_binaries_loaded, ensure_aicpu_init_launched,
prepare_launch_shape, prepare_aicpu_affinity — each return a failure without
unwinding them; only the claim has a rollback guard. A refusal from any of them
therefore reached ChipWorker with live resources, where
destroy_device_context refuses the context and returns void, and the
DlHandleGuard then dlcloses the library whose release routines are the only
way back to them.

What this adopts

  • ChipWorkerError carries a PTO_RUNTIME_ERR_* code, and
    UnsupportedRuntimeOperation derives from it. The nanobind layer registers
    both with translators that copy the code onto the Python exception, and
    exports the codes as module attributes. The registered
    UnsupportedRuntimeOperation also subclasses NotImplementedError, so
    device_memory_info's ad-hoc conversion is deleted rather than duplicated —
    existing except NotImplementedError callers are unaffected.
  • A refused kernel init calls finalize_device first. When that fails the
    handle, the bindings and the library all stay, device_teardown_owed_ is
    recorded, and no later init runs until a finalize() succeeds.
    INVALID_ARGUMENT and UNSUPPORTED skip the teardown: the entry returns them
    before taking anything.
  • kernel_mode_supported() answers instead of throwing — false whenever no
    runtime is bound, which initialized() already distinguishes from a bound
    runtime without kernel support.
  • A failed finalize() on a kernel context raises and is retriable. The
    Python wrapper's registries already documented that behaviour for a raising
    finalize that could not yet happen.
  • Kernel callables get their own registry. _callable_registry keys are
    program slots that init() replays into register_callable and that
    _allocate_slot_locked reads as occupancy; a runtime-minted kernel id has
    neither meaning.
  • Sixteen Python cases in tests/ut/py/test_chip_worker.py, plus the C++
    and Python entry-test updates.

Kept against #2185

Four places where this line is right and #2185's head is an artifact of its own
stub platform:

  • the D14 item 6 capability gate stays, so the adopted fake runtimes declare
    kernel support to reach the entry behind it;
  • a program-mode teardown status is reported on stderr rather than dropped —
    the runner gives up its device either way, so there is nothing to retry, but
    the status is the only evidence;
  • kernel_init succeeding is the a2a3 case, so the hardware test keeps the
    live-context scenario;
  • the Python twin keeps loading host_build_graph for a genuine UNSUPPORTED.

Decisions are D20 in the integration log.

Testing

Ran twice — once on 29a1cd40, then again after rebasing onto 4a5f28c9
(which carries the merged #2245 and #2242).

  • Native builds: a2a3, a5, a2a3sim, a5sim, nanobind extension — all succeeded
  • C++ unit tests, no hardware — 166/166 passed
  • C++ unit tests with SIMPLER_ENABLE_HARDWARE_TESTS=ON — the two added no-hardware targets passed
  • a2a3 hardware, test_kernel_mode_entry with a CTest resource spec — passed
  • Python unit tests, tests/ut -m "not requires_hardware" — 2448 passed (baseline + the 16 new cases)
  • Python hardware unit tests, tests/ut -m requires_hardware --platform a2a3 — 21/21 scheduled cases passed
  • a2a3sim scenes — 82 passed, 8 skipped (baseline exact)
  • pre-commit hooks on changed files — passed

Hardware work passed the architecture precheck and acquired devices through
task-submit --device auto. Not re-run: the a2a3 onboard scene phases, the SDMA
phase, a5sim scenes, and tests/ut/py/test_kernel_mode_c_api.py — the change is
confined to the ChipWorker entry layer, its bindings and its tests, and does
not reach the platform entries those suites exercise.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: e3b53e7c-f278-48e2-bb83-9f0b36e3df8a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…tion line

The remainder of hw-native-sys#2185's refreshed head, now that D16 has taken its entry
signature independently: a status-carrying exception, a capability query
that answers instead of throwing, and bookkeeping for a refused init or a
failed teardown.

One of those closes a leak. kernel_init dropped the device context on any
refused init, on the stated ground that a refused kernel init has adopted
nothing. init_kernel_context creates the context's two streams and five
events before four steps that can still fail and unwinds only its context
claim, so a refusal from any of them reached ChipWorker with live
resources — where destroy_device_context refuses the context and returns
void, and the DlHandleGuard then unloads the library whose release
routines are the only way back to them. A refused init now calls
finalize_device first and, when that fails, keeps the handle, the
bindings and the library, records the owed teardown, and refuses a later
init until one succeeds. INVALID_ARGUMENT and UNSUPPORTED skip it: the
entry returns them before taking anything.

ChipWorkerError carries a PTO_RUNTIME_ERR_* code,
UnsupportedRuntimeOperation derives from it, and nanobind translators
copy the code onto the Python exception with the codes exported as module
attributes. The registered UnsupportedRuntimeOperation also subclasses
NotImplementedError, so device_memory_info's ad-hoc conversion goes.
kernel_mode_supported() reports false when no runtime is bound rather
than throwing, and a failed finalize() on a kernel context raises and can
be retried. Kernel callables move to their own registry: the program one
is keyed by slots that init() replays into register_callable.

Four things stay as this line has them, each an artifact of hw-native-sys#2185's stub
platform: the D14 item 6 capability gate, a program-mode teardown status
reported on stderr rather than dropped, the successful-kernel_init
scenario in the a2a3 hardware test, and the Python twin loading
host_build_graph for a genuine UNSUPPORTED. The adopted fake runtimes
declare kernel support so they reach the entry behind that gate.

Decisions are D20 in the integration log.
@YunjiQin
YunjiQin merged commit bd7a7c4 into hw-native-sys:feat/kernel-mode-integration-test Sep 15, 2026
3 checks passed
YunjiQin added a commit to sunkaixuan2018/simpler-PTO that referenced this pull request Sep 16, 2026
Rebasing onto the current integration head brings in hw-native-sys#2247, which changed
the contract this branch's close() compensated for. ChipWorker.finalize
now raises ChipWorkerError when a kernel context's device teardown fails,
and clears initialized before it does, so the Worker-side check for a
still-initialized native worker after a returning finalize() can never
fire: raising is already what keeps the CleanupJournal entry, the
ChipWorker, its prepared images and the GC pin for a later close().

The fake ChipWorker in test_worker_kernel_mode.py modelled the old
behaviour — a failed finalize returned and left the native worker
initialized — so leaving it would have kept asserting a contract the real
one no longer has. It now clears initialized and raises, and
test_failed_teardown_keeps_the_context_for_retry expects ChipWorkerError
with the status the entry reports.

Three of the four rebase conflicts resolve by keeping both sides:
probe_kernel_mode_supported joins hw-native-sys#2247's finalize() doc in chip_worker.h,
TestChipWorkerKernelProbe joins TestChipWorkerKernelEntryLayer, and the
fake runtime keeps hw-native-sys#2247's live-handle tracking alongside this branch's
created-context counter. The kernel_prepare_callable docstring takes this
branch's account of the registration synchronize plus hw-native-sys#2247's sentence on
how long the image stays referenced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
YunjiQin added a commit that referenced this pull request Sep 16, 2026
…robe (#2248)

* Add: kernel-mode entry on Worker(level=2) and a pre-init capability probe

PyPTO drives simpler through simpler.worker.Worker, but the kernel-mode
entries existed only on ChipWorker. A level-2 Worker now runs in one of two
execution modes fixed at construction, so a framework process that already
owns a device and a stream can bind, prepare, launch and close a kernel
context through the public Worker.

Worker(level=2, execution_mode="kernel", device_id=..., platform=...,
runtime=...) accepts init(config=CallConfig(...)), which routes to
ChipWorker.kernel_init on the device the calling thread already has current.
Nothing is prewarmed, no registration is replayed and no SDMA workspace is
provisioned. config is keyword-only and required in kernel mode; prewarm_config
is refused there, and config= is refused in program mode. Kernel mode has no
L3+ form, because an L3 chip lives in a forked child that cannot see the
caller's current device or stream, and it is refused at construction for any
other level.

kernel_prepare_callable(chip_callable) returns the id the runtime minted. It
is unrelated to register(): there is no digest dedup, no handle and no
pre-init recording, so the same callable prepared twice takes two ids. The
Worker keeps every prepared image referenced until a native teardown
succeeds, because the device holds addresses into it.

kernel_launch(callable_id, args, *, caller_stream) enqueues one invocation on
the caller's stream and returns. It takes no operation lease, creates no
RunHandle and waits for nothing. args is ChipStorageTaskArgs; the stream is
keyword-only and nonzero; an id this Worker did not mint, or a non-integer
id, is refused before the native call.

kernel_mode_supported answers before init. The new static
ChipWorker::probe_kernel_mode_supported loads the host runtime, asks
simpler_kernel_mode_supported on a fresh device context that no init touches,
and destroys that context before the library handle is released; the C ABI
requires that entry to answer from the runtime build alone, so the probe takes
no device. The Worker caches the answer, answers False for a level other than
2, and answers True for a READY kernel-mode Worker without probing. A missing
runtime build raises instead of reading as unsupported.

The two modes exclude each other. A kernel-mode Worker refuses register,
unregister, submit, run, malloc, free, copy_to, copy_from, create_buffer,
make_tensor_arg, release_buffer and device_memory_info; the native memory
query already refuses a kernel context, and the rest would reach program
state a kernel context does not have. committed_device_memory stays
available. A program-mode Worker refuses the kernel entries.

A lock linearizes prepare and launch against the native finalize. A launch
that overlaps a prepare, another launch or close() fails immediately, since
the caller serializes them. close() publishes CLOSED before it takes the lock,
so a launch either observes CLOSED or finishes before finalize; if a prepare
or launch still holds the lock after the rollback grace period, close() raises
and can be called again. On this base ChipWorker.finalize returns without
raising when the device teardown fails and leaves the native worker
initialized, so kernel close() checks that state and raises, which keeps the
CleanupJournal entry, the ChipWorker, its images and the GC pin for a retry.
The process that bound the context is recorded, and a forked child that
inherited the Worker cannot prepare, launch or tear it down.

~ChipWorker finalizes on whichever thread destroys it, without the quiescence
kernel teardown requires. A kernel-mode Worker garbage-collected without
close(), or still alive at interpreter exit, therefore emits a ResourceWarning
and deliberately pins its ChipWorker for the rest of the process instead of
letting that destructor run. The finalizer receives only the pinned object,
never the Worker.

The Worker and init/close docstrings, docs/user/reference/python-api.md,
docs/chip-level-arch.md and the kernel-mode integration test page describe the
mode, its preconditions and its refusals. The ChipWorker.kernel_prepare_callable
docstring and the residency guide state that registration synchronizes the
context's own stream, so a registration failure raises from prepare.

test_worker_kernel_mode.py covers the Worker against a fake ChipWorker:
argument and mode validation, routing, the probe, id minting, launch
forwarding and refusals, the gate against prepare, launch and close,
retryable teardown, failed-init rollback, the GC pin and the process fence,
plus the real a2a3sim build reporting no kernel mode. test_chip_worker.py
drives the probe against the generated fake runtime.
test_worker_kernel_mode_hw.py runs on a2a3 without torch: the probe before
init, eager launches with exact numerics on the caller's stream, repeated
prepares, Worker- and runtime-side refusals followed by a valid launch, a
second Worker refused on the same device, and host_build_graph refused, each
case checking that the caller's stream and device tear down cleanly
afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fix: drop the finalize compensation #2247 made unreachable

Rebasing onto the current integration head brings in #2247, which changed
the contract this branch's close() compensated for. ChipWorker.finalize
now raises ChipWorkerError when a kernel context's device teardown fails,
and clears initialized before it does, so the Worker-side check for a
still-initialized native worker after a returning finalize() can never
fire: raising is already what keeps the CleanupJournal entry, the
ChipWorker, its prepared images and the GC pin for a later close().

The fake ChipWorker in test_worker_kernel_mode.py modelled the old
behaviour — a failed finalize returned and left the native worker
initialized — so leaving it would have kept asserting a contract the real
one no longer has. It now clears initialized and raises, and
test_failed_teardown_keeps_the_context_for_retry expects ChipWorkerError
with the status the entry reports.

Three of the four rebase conflicts resolve by keeping both sides:
probe_kernel_mode_supported joins #2247's finalize() doc in chip_worker.h,
TestChipWorkerKernelProbe joins TestChipWorkerKernelEntryLayer, and the
fake runtime keeps #2247's live-handle tracking alongside this branch's
created-context counter. The kernel_prepare_callable docstring takes this
branch's account of the registration synchronize plus #2247's sentence on
how long the image stays referenced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: YunjiQin <a1339924773@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant