Add: adopt #2185's kernel entry hardening on the integration line - #2247
Merged
YunjiQin merged 1 commit intoSep 15, 2026
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…tion line The remainder of hw-native-sys#2185's refreshed head, now that D16 has taken its entry signature independently: a status-carrying exception, a capability query that answers instead of throwing, and bookkeeping for a refused init or a failed teardown. One of those closes a leak. kernel_init dropped the device context on any refused init, on the stated ground that a refused kernel init has adopted nothing. init_kernel_context creates the context's two streams and five events before four steps that can still fail and unwinds only its context claim, so a refusal from any of them reached ChipWorker with live resources — where destroy_device_context refuses the context and returns void, and the DlHandleGuard then unloads the library whose release routines are the only way back to them. A refused init now calls finalize_device first and, when that fails, keeps the handle, the bindings and the library, records the owed teardown, and refuses a later init until one succeeds. INVALID_ARGUMENT and UNSUPPORTED skip it: the entry returns them before taking anything. ChipWorkerError carries a PTO_RUNTIME_ERR_* code, UnsupportedRuntimeOperation derives from it, and nanobind translators copy the code onto the Python exception with the codes exported as module attributes. The registered UnsupportedRuntimeOperation also subclasses NotImplementedError, so device_memory_info's ad-hoc conversion goes. kernel_mode_supported() reports false when no runtime is bound rather than throwing, and a failed finalize() on a kernel context raises and can be retried. Kernel callables move to their own registry: the program one is keyed by slots that init() replays into register_callable. Four things stay as this line has them, each an artifact of hw-native-sys#2185's stub platform: the D14 item 6 capability gate, a program-mode teardown status reported on stderr rather than dropped, the successful-kernel_init scenario in the a2a3 hardware test, and the Python twin loading host_build_graph for a genuine UNSUPPORTED. The adopted fake runtimes declare kernel support so they reach the entry behind that gate. Decisions are D20 in the integration log.
YunjiQin
merged commit Sep 15, 2026
bd7a7c4
into
hw-native-sys:feat/kernel-mode-integration-test
3 checks passed
YunjiQin
added a commit
to sunkaixuan2018/simpler-PTO
that referenced
this pull request
Sep 16, 2026
Rebasing onto the current integration head brings in hw-native-sys#2247, which changed the contract this branch's close() compensated for. ChipWorker.finalize now raises ChipWorkerError when a kernel context's device teardown fails, and clears initialized before it does, so the Worker-side check for a still-initialized native worker after a returning finalize() can never fire: raising is already what keeps the CleanupJournal entry, the ChipWorker, its prepared images and the GC pin for a later close(). The fake ChipWorker in test_worker_kernel_mode.py modelled the old behaviour — a failed finalize returned and left the native worker initialized — so leaving it would have kept asserting a contract the real one no longer has. It now clears initialized and raises, and test_failed_teardown_keeps_the_context_for_retry expects ChipWorkerError with the status the entry reports. Three of the four rebase conflicts resolve by keeping both sides: probe_kernel_mode_supported joins hw-native-sys#2247's finalize() doc in chip_worker.h, TestChipWorkerKernelProbe joins TestChipWorkerKernelEntryLayer, and the fake runtime keeps hw-native-sys#2247's live-handle tracking alongside this branch's created-context counter. The kernel_prepare_callable docstring takes this branch's account of the registration synchronize plus hw-native-sys#2247's sentence on how long the image stays referenced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
YunjiQin
added a commit
that referenced
this pull request
Sep 16, 2026
…robe (#2248) * Add: kernel-mode entry on Worker(level=2) and a pre-init capability probe PyPTO drives simpler through simpler.worker.Worker, but the kernel-mode entries existed only on ChipWorker. A level-2 Worker now runs in one of two execution modes fixed at construction, so a framework process that already owns a device and a stream can bind, prepare, launch and close a kernel context through the public Worker. Worker(level=2, execution_mode="kernel", device_id=..., platform=..., runtime=...) accepts init(config=CallConfig(...)), which routes to ChipWorker.kernel_init on the device the calling thread already has current. Nothing is prewarmed, no registration is replayed and no SDMA workspace is provisioned. config is keyword-only and required in kernel mode; prewarm_config is refused there, and config= is refused in program mode. Kernel mode has no L3+ form, because an L3 chip lives in a forked child that cannot see the caller's current device or stream, and it is refused at construction for any other level. kernel_prepare_callable(chip_callable) returns the id the runtime minted. It is unrelated to register(): there is no digest dedup, no handle and no pre-init recording, so the same callable prepared twice takes two ids. The Worker keeps every prepared image referenced until a native teardown succeeds, because the device holds addresses into it. kernel_launch(callable_id, args, *, caller_stream) enqueues one invocation on the caller's stream and returns. It takes no operation lease, creates no RunHandle and waits for nothing. args is ChipStorageTaskArgs; the stream is keyword-only and nonzero; an id this Worker did not mint, or a non-integer id, is refused before the native call. kernel_mode_supported answers before init. The new static ChipWorker::probe_kernel_mode_supported loads the host runtime, asks simpler_kernel_mode_supported on a fresh device context that no init touches, and destroys that context before the library handle is released; the C ABI requires that entry to answer from the runtime build alone, so the probe takes no device. The Worker caches the answer, answers False for a level other than 2, and answers True for a READY kernel-mode Worker without probing. A missing runtime build raises instead of reading as unsupported. The two modes exclude each other. A kernel-mode Worker refuses register, unregister, submit, run, malloc, free, copy_to, copy_from, create_buffer, make_tensor_arg, release_buffer and device_memory_info; the native memory query already refuses a kernel context, and the rest would reach program state a kernel context does not have. committed_device_memory stays available. A program-mode Worker refuses the kernel entries. A lock linearizes prepare and launch against the native finalize. A launch that overlaps a prepare, another launch or close() fails immediately, since the caller serializes them. close() publishes CLOSED before it takes the lock, so a launch either observes CLOSED or finishes before finalize; if a prepare or launch still holds the lock after the rollback grace period, close() raises and can be called again. On this base ChipWorker.finalize returns without raising when the device teardown fails and leaves the native worker initialized, so kernel close() checks that state and raises, which keeps the CleanupJournal entry, the ChipWorker, its images and the GC pin for a retry. The process that bound the context is recorded, and a forked child that inherited the Worker cannot prepare, launch or tear it down. ~ChipWorker finalizes on whichever thread destroys it, without the quiescence kernel teardown requires. A kernel-mode Worker garbage-collected without close(), or still alive at interpreter exit, therefore emits a ResourceWarning and deliberately pins its ChipWorker for the rest of the process instead of letting that destructor run. The finalizer receives only the pinned object, never the Worker. The Worker and init/close docstrings, docs/user/reference/python-api.md, docs/chip-level-arch.md and the kernel-mode integration test page describe the mode, its preconditions and its refusals. The ChipWorker.kernel_prepare_callable docstring and the residency guide state that registration synchronizes the context's own stream, so a registration failure raises from prepare. test_worker_kernel_mode.py covers the Worker against a fake ChipWorker: argument and mode validation, routing, the probe, id minting, launch forwarding and refusals, the gate against prepare, launch and close, retryable teardown, failed-init rollback, the GC pin and the process fence, plus the real a2a3sim build reporting no kernel mode. test_chip_worker.py drives the probe against the generated fake runtime. test_worker_kernel_mode_hw.py runs on a2a3 without torch: the probe before init, eager launches with exact numerics on the caller's stream, repeated prepares, Worker- and runtime-side refusals followed by a valid launch, a second Worker refused on the same device, and host_build_graph refused, each case checking that the caller's stream and device tear down cleanly afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix: drop the finalize compensation #2247 made unreachable Rebasing onto the current integration head brings in #2247, which changed the contract this branch's close() compensated for. ChipWorker.finalize now raises ChipWorkerError when a kernel context's device teardown fails, and clears initialized before it does, so the Worker-side check for a still-initialized native worker after a returning finalize() can never fire: raising is already what keeps the CleanupJournal entry, the ChipWorker, its prepared images and the GC pin for a later close(). The fake ChipWorker in test_worker_kernel_mode.py modelled the old behaviour — a failed finalize returned and left the native worker initialized — so leaving it would have kept asserting a contract the real one no longer has. It now clears initialized and raises, and test_failed_teardown_keeps_the_context_for_retry expects ChipWorkerError with the status the entry reports. Three of the four rebase conflicts resolve by keeping both sides: probe_kernel_mode_supported joins #2247's finalize() doc in chip_worker.h, TestChipWorkerKernelProbe joins TestChipWorkerKernelEntryLayer, and the fake runtime keeps #2247's live-handle tracking alongside this branch's created-context counter. The kernel_prepare_callable docstring takes this branch's account of the registration synchronize plus #2247's sentence on how long the image stays referenced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: YunjiQin <a1339924773@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
D15 took only the test hardening from #2185's refreshed head and deferred the
rest as accompanying a signature change; D16 has since adopted that signature
independently. What is left is no longer a signature question, and one part of
it closes a real leak.
The leak.
ChipWorker::kernel_initdropped the device context on anyrefused init, on the stated ground that a refused kernel init "has adopted
nothing — it took no ACL state and bound no thread". That is false here:
init_kernel_contextacquires the context claim, thenkernel_exec_state_.initializecreates two streams and five events, and thefour steps after it —
ensure_binaries_loaded,ensure_aicpu_init_launched,prepare_launch_shape,prepare_aicpu_affinity— each return a failure withoutunwinding them; only the claim has a rollback guard. A refusal from any of them
therefore reached
ChipWorkerwith live resources, wheredestroy_device_contextrefuses the context and returns void, and theDlHandleGuardthendlcloses the library whose release routines are the onlyway back to them.
What this adopts
ChipWorkerErrorcarries aPTO_RUNTIME_ERR_*code, andUnsupportedRuntimeOperationderives from it. The nanobind layer registersboth with translators that copy the code onto the Python exception, and
exports the codes as module attributes. The registered
UnsupportedRuntimeOperationalso subclassesNotImplementedError, sodevice_memory_info's ad-hoc conversion is deleted rather than duplicated —existing
except NotImplementedErrorcallers are unaffected.finalize_devicefirst. When that fails thehandle, the bindings and the library all stay,
device_teardown_owed_isrecorded, and no later init runs until a
finalize()succeeds.INVALID_ARGUMENTandUNSUPPORTEDskip the teardown: the entry returns thembefore taking anything.
kernel_mode_supported()answers instead of throwing — false whenever noruntime is bound, which
initialized()already distinguishes from a boundruntime without kernel support.
finalize()on a kernel context raises and is retriable. ThePython wrapper's registries already documented that behaviour for a raising
finalize that could not yet happen.
_callable_registrykeys areprogram slots that
init()replays intoregister_callableand that_allocate_slot_lockedreads as occupancy; a runtime-minted kernel id hasneither meaning.
tests/ut/py/test_chip_worker.py, plus the C++and Python entry-test updates.
Kept against #2185
Four places where this line is right and #2185's head is an artifact of its own
stub platform:
kernel support to reach the entry behind it;
the runner gives up its device either way, so there is nothing to retry, but
the status is the only evidence;
kernel_initsucceeding is the a2a3 case, so the hardware test keeps thelive-context scenario;
host_build_graphfor a genuineUNSUPPORTED.Decisions are D20 in the integration log.
Testing
Ran twice — once on
29a1cd40, then again after rebasing onto4a5f28c9(which carries the merged #2245 and #2242).
SIMPLER_ENABLE_HARDWARE_TESTS=ON— the two added no-hardware targets passedtest_kernel_mode_entrywith a CTest resource spec — passedtests/ut -m "not requires_hardware"— 2448 passed (baseline + the 16 new cases)tests/ut -m requires_hardware --platform a2a3— 21/21 scheduled cases passedHardware work passed the architecture precheck and acquired devices through
task-submit --device auto. Not re-run: the a2a3 onboard scene phases, the SDMAphase, a5sim scenes, and
tests/ut/py/test_kernel_mode_c_api.py— the change isconfined to the
ChipWorkerentry layer, its bindings and its tests, and doesnot reach the platform entries those suites exercise.