Skip to content

Add: admit prepared capture nodes and qualify TMR replay - #2625

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
HaoZhang1024:tmr-kernel-capture-replay-scope
Nov 5, 2026
Merged

ChaoWao merged 1 commit into
hw-native-sys:mainfrom
HaoZhang1024:tmr-kernel-capture-replay-scope

Conversation

@HaoZhang1024

@HaoZhang1024 HaoZhang1024 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Allow multiple fully prepared native cold-capture nodes to coexist, and qualify TMR capture/replay through main's common execution. The admission change is shared by TMR and HBG; the TMR device reconstruction and signed terminal handoff are already in merged #2862. Unfinished preparation continues to block any bind that could replace its shared arena/configuration. This follows the fixed-source alignment review.

Base and changes

Rebased on main 91f0004517a9f1415228d33bfd8ca95eddbf936d, including #2862's actual merged commit a20f8b2e6f2670532b9e1ecc84718022e4b1559a. This PR contains one final commit and eight files: three Host product files and five test files. The prerequisite's implementation/history is part of main and is no longer carried as another PR commit.

  • Publish Prepared only after backing ownership and the resource plan are complete. Permit completed cold capture preparations in distinct slots; reject unfinished preparations, mixed eager/capture, duplicate slots and excess capacity before shared state can change.

  • Preserve ordinary eager admission, shared-arena checks, exclusive execution claims, registration guards, actual-stream error observation and delivery-aware finalize. Preparation and graph resource retention use main's existing common APIs and owners.

  • Cover immutable arguments, no user computation during recording, numerical replay, serial independent models, two cold nodes, destroy-without-execution and signed failures. Reconstruction preserves the user heap. Balance test-owned primary contexts and explicitly retry the injected Program cleanup failure.

Kernel eager and recording use the same executor. Program remains eager-only. No AICore entry/argument extension, second graph owner, replay Host callback, private completion protocol or DFX/D2H success gate is introduced.

Validation

  • Actual final tree cc75bac48 was revalidated after the alignment review: eight C++ targets covering common admission/HBG eager controls, both-architecture TMR reconstruction, MPI and retirement; 37 C API cases per simulation architecture; and both fake-RTS guarded-reset lifecycle arms passed. Current-main cleanup-retry and caller-no-reset assertions are preserved.

  • The final extension and A5 TMR SDK were rebuilt with PTO pin 5487a6b0. On DT/CANN 9.3.0, allocated development device 0, the same final tree passed release/Program cleanup retry two cases and native raw capture five cases. Source/extension identity, allocated architecture and before/after SDK hashes matched. This is a final-tree hardware receipt, not an automatic carry-over of an older head.

  • Additional integration evidence: six HBG Graph/explicit-predecessor/admission C++ targets, 52 CI policy checks, 55 communication workspace/source tests and applicable hooks passed. Unchanged earlier 458-target and 29-startup/ABI/package coverage remains recorded separately.

  • Automatic CI for cc75bac48 completed: 13 successful, two skipped, zero failed or pending (run 41997872897). Main CI: run A5 PR hardware checks on demand #2898 runs the full A5 hardware suites on an admin /run-a5 request; A5 simulation, CPU and profiling checks remain automatic. The focused DT tests above do not represent the full optional A5 CI suite, multi-device communication capture, DFX or performance qualification.

Technical review, CI and formal maintainer approval remain separate. No merge is performed by this update.

@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

This PR adds runtime-assigned callable registration and validated kernel invocation packets. It also adds TMR kernel capture and replay APIs for onboard builds, capture admission and completion paths, simulation refusal behavior, and related tests and documentation.

Changes: TMR invocation and capture

Layer / File(s) Summary
Callable registration and execution mode
docs/design/run-completion-fence.md, src/common/platform/include/host/execution_mode_latch.h, src/common/platform/onboard/host/c_api_shared.cpp, src/common/platform/sim/host/c_api_shared.cpp, src/common/worker/*, python/bindings/task_interface.cpp, tests/ut/cpp/common/hierarchical/*, tests/ut/cpp/common/platform/test_execution_mode_latch.cpp, tests/ut/py/test_chip_worker.py, tests/ut/py/test_chip_caller_stream.py, tests/ut/py/test_kernel_mode_c_api.py
The C API and ChipWorker add runtime-assigned callable registration and kernel-callable preparation. The execution-mode latch now represents an unclaimed mode with an empty optional value. Tests cover automatic ID assignment, mode constraints, and context validation.
Eager invocation packet path
src/common/tensormap_and_ringbuffer/kernel_invocation.h, src/common/tensormap_and_ringbuffer/kernel_launch_admission.h, src/common/platform/onboard/host/device_runner_*, src/common/platform/include/common/launch_entry_args.h, src/common/platform/include/aicpu/platform_entry_args.h, src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp, src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp, tests/ut/cpp/common/platform/test_launch_entry_args_tmr.cpp, tests/ut/cpp/common/tensormap_and_ringbuffer/test_kernel_invocation.cpp, tests/st/common/kernel_eager_state/test_stateful_calls.py
Kernel invocation packets carry callable identity, binding, tensor descriptors, and scalar values. Host publication selects the kernel-invocation route; AICPU validates the packet before using its arguments.
Device capture admission and completion
src/common/task_interface/kernel_capture.h, src/common/tensormap_and_ringbuffer/kernel_capture_*, src/a2a3/platform/onboard/*, src/a5/platform/onboard/*, src/a2a3/runtime/tensormap_and_ringbuffer/*, src/a5/runtime/tensormap_and_ringbuffer/*, tests/ut/cpp/common/tensormap_and_ringbuffer/*
Capture control structures and device registries validate capture identity, invocation packets, replay epochs, and completion state. AICore entry points check admission before kernel setup, while AICPU capture hooks report execution completion.
Host capture lease lifecycle
src/common/platform/include/host/kernel_capture_*, src/common/platform/onboard/host/kernel_capture_api.*, src/common/platform/onboard/host/device_runner_base.*, src/common/worker/native_run_context.h, src/common/worker/runtime_c_api.h, src/common/platform/onboard/host/c_api_shared.cpp, docs/design/tmr-kernel-capture.md, mkdocs.yml, tests/ut/cpp/common/platform/test_kernel_capture_binder.cpp, tests/ut/py/test_kernel_capture.py
Host APIs add capture preparation, recording, waiting, and release. The lease retains resources through captured replay, synchronizes the caller and hidden streams, and revokes registration before cleanup. The documentation and tests describe the lifecycle and its supported configurations.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant CaptureAPI as Capture API
  participant AICPU
  participant Registry as Capture registry
  participant AICore
  Caller->>CaptureAPI: Prepare lease
  CaptureAPI->>Registry: Register capture
  Caller->>CaptureAPI: Record capture
  AICPU->>Registry: Admit replay packet
  Registry-->>AICPU: Return replay and binding epochs
  AICPU->>AICore: Launch admitted core work
  AICPU->>Registry: Finish replay with status
  Caller->>CaptureAPI: Wait for completion
  Caller->>CaptureAPI: Release lease
Loading
Merge Risk: Minimal · up to 5bd65

This change adds native TMR kernel capture and replay, with validated invocation packets and lease lifecycle handling. No concrete defects remain open after review. The author should still restack onto the final registration prerequisite and revalidate as planned.

Pre-merge checks: Passed 4

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title identifies the main changes: admitting prepared capture nodes and qualifying TMR replay. It is concise and related to the changeset.
Description check Passed The description is related to the changeset. It describes admission changes that allow multiple prepared native cold-capture nodes to coexist, the TMR capture/replay qualification, the Host product fi…
Full details: Description check

Explanation

The description is related to the changeset. It describes admission changes that allow multiple prepared native cold-capture nodes to coexist, the TMR capture/replay qualification, the Host product files and test files touched, and the validation performed. The file-level summaries cover capture registry, admission, and prepare/record/wait/release APIs, so the description addresses the same work. It is specific enough to decide the check passes, though it refers to a prerequisite (#2862) and commit hashes that are not in the provided context.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit packed the kernels tight,
Then queued their arguments just right.
Through caller streams they hopped along,
Each replay found its epoch strong.
The lease stayed safe until release,
Then carrots danced in captured peace.

ℹ️ Recent review info

⚙️ Run configuration

  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8640d7f4-80ed-4872-950e-24957be2ec73
📥 Commits

Reviewing files that changed from the base of the PR and between f50f900 and 5bd65c2.

📒 Files selected for processing: 53

  • docs/design/run-completion-fence.md
  • docs/design/tmr-kernel-capture.md
  • mkdocs.yml
  • python/bindings/task_interface.cpp
  • src/a2a3/platform/include/common/kernel_args.h
  • src/a2a3/platform/onboard/aicore/kernel.cpp
  • src/a2a3/platform/onboard/aicpu/kernel.cpp
  • src/a2a3/platform/onboard/host/CMakeLists.txt
  • src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp
  • src/a2a3/runtime/tensormap_and_ringbuffer/host/runtime_maker.cpp
  • src/a5/platform/include/common/kernel_args.h
  • src/a5/platform/onboard/aicore/kernel.cpp
  • src/a5/platform/onboard/aicpu/kernel.cpp
  • src/a5/platform/onboard/host/CMakeLists.txt
  • src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp
  • src/a5/runtime/tensormap_and_ringbuffer/host/runtime_maker.cpp
  • src/common/platform/include/aicpu/platform_entry_args.h
  • src/common/platform/include/common/launch_entry_args.h
  • src/common/platform/include/host/execution_mode_latch.h
  • src/common/platform/include/host/kernel_capture_binder.h
  • src/common/platform/include/host/kernel_capture_owner.h
  • src/common/platform/onboard/host/c_api_shared.cpp
  • src/common/platform/onboard/host/device_runner_base.cpp
  • src/common/platform/onboard/host/device_runner_base.h
  • src/common/platform/onboard/host/device_runner_helpers.cpp
  • src/common/platform/onboard/host/device_runner_helpers.h
  • src/common/platform/onboard/host/kernel_capture_api.cpp
  • src/common/platform/onboard/host/kernel_capture_api.h
  • src/common/platform/sim/host/c_api_shared.cpp
  • src/common/task_interface/kernel_capture.h
  • src/common/tensormap_and_ringbuffer/kernel_capture_device.h
  • src/common/tensormap_and_ringbuffer/kernel_capture_gate.h
  • src/common/tensormap_and_ringbuffer/kernel_invocation.h
  • src/common/tensormap_and_ringbuffer/kernel_launch_admission.h
  • src/common/worker/chip_worker.cpp
  • src/common/worker/chip_worker.h
  • src/common/worker/native_run_context.h
  • src/common/worker/runtime_c_api.h
  • tests/st/common/kernel_eager_state/test_stateful_calls.py
  • tests/ut/cpp/common/hierarchical/test_chip_run_lane_joined_launch.cpp
  • tests/ut/cpp/common/platform/CMakeLists.txt
  • tests/ut/cpp/common/platform/test_execution_mode_latch.cpp
  • tests/ut/cpp/common/platform/test_kernel_capture_binder.cpp
  • tests/ut/cpp/common/platform/test_launch_entry_args_tmr.cpp
  • tests/ut/cpp/common/tensormap_and_ringbuffer/CMakeLists.txt
  • tests/ut/cpp/common/tensormap_and_ringbuffer/test_kernel_capture_device.cpp
  • tests/ut/cpp/common/tensormap_and_ringbuffer/test_kernel_capture_gate.cpp
  • tests/ut/cpp/common/tensormap_and_ringbuffer/test_kernel_invocation.cpp
  • tests/ut/py/kernel_capture_state.cpp
  • tests/ut/py/test_chip_caller_stream.py
  • tests/ut/py/test_chip_worker.py
  • tests/ut/py/test_kernel_capture.py
  • tests/ut/py/test_kernel_mode_c_api.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 496f8fe to fd73a37 Compare October 19, 2026 08:35
@HaoZhang1024 HaoZhang1024 changed the title Add: TMR kernel capture and replay (scope draft) Add: support native TMR kernel capture and replay Oct 19, 2026
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from fd73a37 to 5bd65c2 Compare October 19, 2026 09:00
@HaoZhang1024
HaoZhang1024 marked this pull request as ready for review October 19, 2026 09:36
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch 3 times, most recently from e887f56 to 7848e06 Compare October 21, 2026 02:54
@HaoZhang1024 HaoZhang1024 reopened this Oct 21, 2026
@ChaoWao

ChaoWao commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Review — 7848e06e: request changes

Reviewed the full capture contribution against prerequisite #2624 (e9c16b04): 38 files, +2432/-50. The single-lease/single-model/same-stream TMR scope is clear. Admission before the processor fork, fixed binding versus device-minted replay epoch, passive rejection of queued successors, and signed terminal-error propagation fit the shared native executor. HBG, capture DFX and multiple graphs are not missing requirements of this scoped batch.

Spec / correctness

P2-1 — Exception exits lose partial-recording ownership (CPU-reproduced)

At kernel_capture_api.cpp:290–295, owner flags and poisoning are updated only after record_kernel_capture() returns. The real callbacks can throw: the symbol passed into LaunchBuiltInOp(const std::string&) can allocate, and AicpuKernelLaunch:427 explicitly calls make_unique. No intervening layer catches exceptions.

I drove the real exported record entry and production recorder with the existing CPU transport. After the Core submission callback succeeded, I armed the existing throwing-allocation hook so the next CPU-launch symbol construction threw. Observed:

escaped=1 core=1 records=3 waits=2 owner_attempted=0 owner_recorded=0 poisoned=0

The ownership/no-exception assertions fail. A temporary catch-to-status control, preserving the existing sink writeback and poisoning, makes the identical probe pass:

escaped=0 core=1 records=3 waits=2 owner_attempted=1 owner_recorded=0 poisoned=1

The false enqueue_attempted also makes quiesce_capture:85 skip hidden-stream completion proofs, contradicting the documented partial-recording contract. This experiment establishes the host bookkeeping/C-boundary defect; it does not establish an actual silicon hang or use-after-free. Models still must be destroyed before release.

Please preserve progress on every exit, translate exceptions into the integer error contract, retain Partial ownership and poison admission. Add exception coverage alongside the 12-operation return-code matrix. Also qualify “without allocation” if it means no device/runtime-resource allocation: literal host-allocation freedom does not hold through the current launcher. Wording alone does not fix the exception path.

P2-2 — Close A3 cold-control visibility before registration reads it (conditional, not board-reproduced)

KernelCaptureRegistry::update:63–66 directly reads the newly H2D-initialized control. Previous replay callbacks clean it using cache_flush_range, implemented as dc cvac, which does not invalidate cached lines. After release, a later lease can reuse the allocation address with a new generation and reset epochs.

The repository's host-DMA cache contract states that A3 host DMA does not snoop AICPU cache. The visibility investigation explicitly warns that allocation is not proof of cold cache, and leaves implicit launch-boundary maintenance unestablished. With retained cache lines, a valid new registration can therefore compare against an old lease's generation/epochs.

Please supply/document the RTS guarantee that discharges this edge, or add the platform-appropriate cold-registration freshness operation. Add capture → replay → release → capture coverage exercising/reporting control-address reuse. Existing hardware cases use one capture lease followed by eager work.

This is a source/contract-derived conditional risk, not a measured board failure. It is A3-specific under the documented premise; A5 is coherent, A2 is unestablished. This is not a request for unconditional shared invalidation.

Standards

P3, nonblocking: kernel_capture.h:86–87 guards KernelCaptureReference only by size and KernelCaptureRegistrationArgs only by trivially-copyable. Please add both standard-layout and trivially-copyable assertions, as required by codestyle §8 and already done for KernelCaptureControl. Current fields satisfy both; this is a regression guard.

Validation and restack

  • Independently rebuilt and ran 8 related C++ targets: 137 individual tests passed. The extra exception probe fails on this head and passes with the temporary control above. Product sources were not modified.
  • Read final-head A3/A5 hardware-UT logs: 76/78 passed, including all six native capture scenarios per architecture and six RTS contract probes each. Final CI has 18 successful checks plus one conditional deploy skip.
  • Read the earlier A5 cancellation: approximately 19 minutes elapsed between task submission and pytest startup before the job was canceled. The later same-head run is green; I found no failed capture assertion in that canceled log.
  • Add: validate TMR eager invocations on the shared native run path #2624 has now squash-merged as fdcbdcc95. A trial merge reports conflicts in both TMR aicpu_executor.cpp files, common/launch_entry_args.h, and kernel_launch_admission.h (add/add). Restack the three capture commits onto current main and revalidate after resolution; existing green CI covers the old-base tree.

Summary: Standards: one nonblocking P3. Spec/correctness: two P2 items—one reproduced host exception defect, one explicitly conditional A3 visibility obligation. Request changes.

@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 7848e06 to 71f5b4f Compare October 22, 2026 09:47
@HaoZhang1024
HaoZhang1024 marked this pull request as draft October 22, 2026 09:48
@HaoZhang1024 HaoZhang1024 changed the title Add: support native TMR kernel capture and replay Add: support native TMR kernel capture and replay [blocked: A5 timeout] Oct 22, 2026
@HaoZhang1024
HaoZhang1024 marked this pull request as ready for review October 22, 2026 11:46
@HaoZhang1024 HaoZhang1024 changed the title Add: support native TMR kernel capture and replay [blocked: A5 timeout] Add: support native TMR kernel capture and replay Oct 23, 2026
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 71f5b4f to 684be8d Compare October 23, 2026 09:16
@HaoZhang1024 HaoZhang1024 reopened this Oct 23, 2026
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 684be8d to 26dd50d Compare October 23, 2026 10:22
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 26dd50d to 1fb9393 Compare October 23, 2026 17:21
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 1fb9393 to c9f43c1 Compare October 24, 2026 07:19
@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch 5 times, most recently from a63cace to 1685c4f Compare October 30, 2026 12:14
zxx0288 added a commit to zxx0288/simpler that referenced this pull request Oct 30, 2026
Use common native registration, preparation, submission, completion and
cleanup for Program and Kernel execution. Share Host graph construction
and remove dedicated execution interfaces and obsolete capture controls.

Support A2/A3 HBG caller-stream capture with cold reservation and immutable
parameter generations. Retain graph-referenced code, images, workspace,
streams and events until model destruction and drain prove safe release.
Reject mutations of referenced callables while allowing unrelated ones.

Prepare shared device state before the fork, preserving first errors and
Unknown outside restored images. Freeze Execute/Skip and aggregate every
RTS native callback after its work ends. Nonfinal callbacks return zero;
the final callback preserves the selected signed result and its source.
Keep the AICore ABI unchanged and use no dynamic Core epoch handshake.

Preserve Program terminal cleanup and Kernel retryable ownership. Cover
replay identity, result isolation, callback aggregation, parameter versions
and capture retirement with CPU, simulation and A3 hardware regressions.

A5 hardware, captured-failure propagation, public replay result retrieval,
per-replay DFX integration and detailed cost qualification remain open.
Related to hw-native-sys#2773, hw-native-sys#2666 and hw-native-sys#2625; this does not close those tracks.

Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
Co-authored-by: zhangxingxing33@hisilicon.com <zhangxingxing33@hisilicon.com>
zxx0288 added a commit to zxx0288/simpler that referenced this pull request Oct 30, 2026
Use common native registration, preparation, submission, completion and
cleanup for Program and Kernel execution. Share Host graph construction
and remove dedicated execution interfaces and obsolete capture controls.

Support A2/A3 HBG caller-stream capture with cold reservation and immutable
parameter generations. Retain graph-referenced code, images, workspace,
streams and events until model destruction and drain prove safe release.
Reject mutations of referenced callables while allowing unrelated ones.

Prepare shared device state before the fork, preserving first errors and
Unknown outside restored images. Freeze Execute/Skip and aggregate every
RTS native callback after its work ends. Nonfinal callbacks return zero;
the final callback preserves the selected signed result and its source.
Keep the AICore ABI unchanged and use no dynamic Core epoch handshake.

Preserve Program terminal cleanup and Kernel retryable ownership. Cover
replay identity, result isolation, callback aggregation, parameter versions
and capture retirement with CPU, simulation and A3 hardware regressions.

A5 hardware, captured-failure propagation, public replay result retrieval,
per-replay DFX integration and detailed cost qualification remain open.
Related to hw-native-sys#2773, hw-native-sys#2666 and hw-native-sys#2625; this does not close those tracks.

Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
Co-authored-by: zhangxingxing33@hisilicon.com <zhangxingxing33@hisilicon.com>
zxx0288 added a commit to zxx0288/simpler that referenced this pull request Oct 31, 2026
Use common native registration, preparation, submission, completion and
cleanup for Program and Kernel execution. Share Host graph construction
and remove dedicated execution interfaces and obsolete capture controls.

Support A2/A3 HBG caller-stream capture with cold reservation and immutable
parameter generations. Retain graph-referenced code, images, workspace,
streams and events until model destruction and drain prove safe release.
Reject mutations of referenced callables while allowing unrelated ones.

Prepare shared device state before the fork, preserving first errors and
Unknown outside restored images. Freeze Execute/Skip and aggregate every
RTS native callback after its work ends. Nonfinal callbacks return zero;
the final callback preserves the selected signed result and its source.
Keep the AICore ABI unchanged and use no dynamic Core epoch handshake.

Preserve Program terminal cleanup and Kernel retryable ownership. Cover
replay identity, result isolation, callback aggregation, parameter versions
and capture retirement with CPU, simulation and A3 hardware regressions.

A5 hardware, captured-failure propagation, public replay result retrieval,
per-replay DFX integration and detailed cost qualification remain open.
Related to hw-native-sys#2773, hw-native-sys#2666 and hw-native-sys#2625; this does not close those tracks.

Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com>
Co-authored-by: zhangxingxing33@hisilicon.com <zhangxingxing33@hisilicon.com>
@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

@ChaoWao 2625 已更新到 head 1685c4f01(基于 74ddf236d,单提交),20 pass / 2 skip、0 未解决 thread、mergeable。本轮统一 capture 与 eager 的公共执行机制,分离 Host run 结果身份与设备 replay 序号,补齐失败路径异常诊断;AICore 入口、参数布局、Runtime/Handshake 接口未改。请复审;这是你 10-27 CHANGES_REQUESTED 后的新 head。

@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch from 1685c4f to 5b866c0 Compare October 31, 2026 01:37
@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

@ChaoWao 2625 已重排到 c3423e1(含 #2747),新 head 5b866c0f6,20 pass / 2 skip、0 未解决 thread、mergeable;功能补丁与原 1685c4f01 一致,AICore 入口/参数布局/Runtime/Handshake 接口未改。请复审。

@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

@HaoZhang1024 2625 现在与 main 冲突(head 5b866c0f6),请 rebase 到当前 main 并保持单提交;rebase 后请 ChaoWao 复审。

@HaoZhang1024
HaoZhang1024 force-pushed the tmr-kernel-capture-replay-scope branch 6 times, most recently from e59230b to 7918e50 Compare November 2, 2026 05:29
@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

@ChaoWao Head 7918e5098 is now MERGEABLE with 20 checks passing and 2 skipped. The prior CHANGES_REQUESTED review is still recorded; please re-review this head and resolve or restate any remaining blockers.

@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

@HaoZhang1024 pre-commit is failing at head 42fa207. Please fix it before the re-review; the prior CHANGES_REQUESTED review is still standing.

@ChaoWao

ChaoWao commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Updated review and concrete convergence split

Reviewed fixed head 0bd1214c5c1241991cbe4e57003cee17d38862ed. This replaces the superseded parts of my older reviews; it does not reopen the resolved objection that eager and recording use separate launch implementations. The present common enqueue direction is correct.

I read zxx0288's complete #2773 proposal and posted the corrected contract. In particular, my older instruction to preserve persistent first-fault Execute/Skip as a requirement was too broad and is withdrawn. Callback coverage and signed RTS error reporting remain necessary; a new counter/last-return protocol must be justified by actual callback behavior, not by my old wording.

Delivery split to converge with #2862

Please use one common prepared-resource/use owner, one retain/release implementation and the existing common prepare/launch/completion flow. Use #2862's PreparedRunResources / PreparedRunUse design as the starting point for that common ownership, subject to its review corrections and port to merged main. This is not approval of that code as-is.

  • Add: support native HBG capture and serial replay #2862: deliver the common retained-resource/partial-cleanup foundation together with working HBG restoration, as a complete feature. Own the shared platform-entry/callback completion contract once.
  • Add: admit prepared capture nodes and qualify TMR replay #2625: deliver TMR per-execution reconstruction and TMR's own signed functional error propagation on that same common foundation. Do not preserve a second context-wide graph owner or another common completion protocol.
  • Our local Core2: source/test donor for missing multi-node preparation, immutable publication-source lifetime/generation and end-consumer regressions. We will not stack its competing owner implementation on top of these PRs.

The likely integration order is common+HBG first, then the TMR increment; dependent local development can proceed against the shared branch without waiting for merge. This is a proposed coordination boundary for the authors to confirm, not a GitHub assignment change. The user will coordinate with the original authors.

Required corrections in this fixed head

  1. Remove the persistent private error as a future business-dispatch gate. capture_admit clears arrival counts but requires first_error == 0; capture_finish latches the first error without clearing it for the next execution. The later admission can therefore choose Rejected solely from the prior invocation. Keep fault evidence and real last-use/reset ordering, but separate them from deciding whether an independent invocation executes. This does not require replaying on a damaged RTS stream or skipping required recovery.
  2. Do not restore a normal private-D2H success gate over merged Update: separate native completion from result observation #2829. observe_kernel_run / read_kernel_run_status require control readback and private epoch completion after stream completion; poll also reads it. Keep actual RTS error observation, functional error reporting and physical-stop/queued-wait obligations. Optional private detail is not an additional normal-success prerequisite. Preserve Update: separate native completion from result observation #2829's physical retirement versus explicit result observation and delivery-aware finalize when rebasing from 70499ab1 to main 313499d61 or later; do not copy an older whole finalizer back over it.
  3. Replace whole-context graph exclusivity with resource-set retention. simpler_retain_run_resources refuses when kernel_capture_owner() is non-null, then installs the first retained run as that owner. Prepare and launch of other states also check it. This prevents separately prepared A/B nodes or a second independently retained model even when their real device use is serial. Retain the actual backing/code/source/event set; multiple retained calls do not imply overlapping device execution or a second Worker. Keep finite capacity checks based on resources, not a blanket context-owner gate.
  4. Drop the unexplained internal revision protocol. KERNEL_CAPTURE_REGISTRATION_VERSION = 2 is a same-build internal host/device contract; no independently evolving consumer is identified. Keep meaningful geometry, identity, generation and address validation. This request is only about the unsupported version negotiation, not deletion of those protections.

Acceptance

Program remains eager only. Kernel eager and capture call the same launch. With DFX off and no user-kernel warmup: capture leaves sentinel outputs untouched, stable-input updates followed by raw RTS replay produce full-array correct results repeatedly, the original run can retire while the model keeps its resources, and queued-copy/partial failure/release behavior remains safe. Add serial multi-node/model ownership coverage against the shared owner. Evidence must name the exact integrated source; older PR/local-branch results are not new-main qualification.

Source/contract review only: no new hardware run or current-CI approval is claimed here. Please address these corrections and the ownership split; no second capture executor, PreparedCall cache or Program capture is requested.

@ChaoWao

ChaoWao commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Reviewed 7883b36343dff4c0c7571f8becec873f99e337ca against merged main 355ce3015e3e344d1fc3a61c206ee01823ce292c after #2862.

The previous architecture objections are resolved in this revision. The common prepared-resource/use owners, retain/release API, entry/completion and #2829 observation/finalization contracts match merged main. Please do not reintroduce a separate owner, capture executor, internal version protocol, or normal-success private-D2H gate to address older reviews.

The actual new increment over your stacked #2862 prerequisite (ea2f2fb6d) is 8 files: 3 Host implementation files and 5 tests. Its product change is shared admission of multiple fully prepared cold-capture nodes; this also benefits HBG. TMR device reconstruction and signed terminal handoff are already present in the merged prerequisite. The new native TMR tests are the acceptance contribution. The full 113-file three-dot diff should not be treated as another standalone core implementation.

Next action: restack only that increment onto actual main, rather than merge/reapply the old prerequisite stack. A read-only merge preview found one textual conflict in test_kernel_mode_c_api.py around the fake-RTS guarded-reset self-check; preserve current-main cleanup-retry and caller-no-reset assertions. The preview otherwise preserves #2869's explicit Graph predecessors and #2898's A5 scheduling policy. Update the PR body to describe #2862 as merged and distinguish shared multi-node admission from TMR qualification.

Then validate the resulting tree: common admission and HBG eager controls, both-architecture TMR reconstruction, native/fake-RTS lifecycle tests, and the actual TMR capture scenarios. Current-head/older hardware receipts do not automatically qualify the rebased tree. This comment is a fixed-source alignment review, not a claim that the remaining CI or untested A5/DFX/framework combinations have passed.

Our parallel HBG retained-model/INOUT candidate stays independent: one prepared node per model, finalize A's Host run before preparing B. Two simultaneously prepared nodes remain dependent on this shared admission increment, so we will not duplicate it in another PR.

Allow completed cold capture preparations in distinct granted slots while
preserving eager admission, shared-arena checks and exclusive execution.
Publish preparation only after backing ownership and the resource plan are
complete, so another cold bind cannot detach unfinished preparation state.

Use the common native descriptor and prepare/launch/completion flow with
shared prepared-resource retention. Keep TMR reconstruction and signed
functional error reporting on that foundation without another graph owner,
Host replay callback, private success gate or AICore interface extension.

Cover raw replay with immutable arguments, serial models, two cold nodes,
unexecuted model destruction and signed RTS errors. Verify per-invocation
reconstruction without clearing user storage. Balance test-owned primary
contexts after borrowed Kernel close and its no-reset assertions, retry
the injected one-shot Program cleanup failure, and preserve fatal quarantine.
@HaoZhang1024

Copy link
Copy Markdown
Collaborator Author

已按最新对齐意见完成更新:消费 #2862 的实际合入提交 a20f8b2e,基于 main 91f00045 重排,当前 cc75bac48,1 个 commit、8 个文件(3 个 Host 产品文件、5 个测试文件),无合并冲突。

  • 共享准入: TMR/HBG 的冷准备在 backing owner 和资源计划完整后才发布 Prepared,允许不同 slot 的已完成准备并存;未完成准备、混合 eager/capture 和执行 claim 仍拒绝越过。普通 eager、注册、共享 arena、完成观察及清理重试守卫保留。
  • TMR 验收: 设备重建和 signed terminal handoff 已在合入的共同前置中,本 PR 直接复用;没有新增 AICore 入口/参数、第二套 owner/执行器或私有 D2H 成功门禁。保留 current-main cleanup-retry 与 caller-no-reset 断言。
  • 最终树复验: 8 个 C++ 目标(含共享准入/HBG eager 控制和双架构 TMR 重构)、两架构 sim 各 37 项、两臂 fake-RTS 生命周期全部通过。cc75bac48 的 extension/A5 TMR SDK 重建后,在 DT 开发卡 0 上通过释放/Program 重试 2 项、raw capture 5 项,源码、分配内架构和前后产物 hash 一致。
  • CI 已完成: 当前 head 13 项成功、2 项跳过,0 失败/排队:CI run 41997872897。A3 UT 为 103 项通过,五种 raw TMR capture 逐名成功;framework、ST/DFX smokes、双架构仿真及 Ubuntu UT/打包均通过。

按主线 #2898,A5 三套全量 CI 现由管理员 /run-a5 按需触发;上述 DT 结果是 TMR 专项,不冒称 A5 全量/多卡通信 capture、DFX 或性能验收。供 committer 按当前单提交复审,未合并。

@hw-native-sys-bot

Copy link
Copy Markdown
Collaborator

@ChaoWao 2625 has been restacked to head cc75bac4: 1 commit, 8 files (3 Host implementation + 5 tests), based on 91f00045, MERGEABLE, 13 pass / 2 skip, 0 unresolved threads. The author addressed the restack-only request and the test_kernel_mode_c_api.py conflict; please re-review the current head and confirm the increment still matches your 16:48 alignment review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants