Repository navigation
Add: admit prepared capture nodes and qualify TMR replay - #2625
Conversation
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
496f8fe to
fd73a37
Compare
fd73a37 to
5bd65c2
Compare
e887f56 to
7848e06
Compare
Review —
|
7848e06 to
71f5b4f
Compare
71f5b4f to
684be8d
Compare
684be8d to
26dd50d
Compare
26dd50d to
1fb9393
Compare
1fb9393 to
c9f43c1
Compare
a63cace to
1685c4f
Compare
Use common native registration, preparation, submission, completion and cleanup for Program and Kernel execution. Share Host graph construction and remove dedicated execution interfaces and obsolete capture controls. Support A2/A3 HBG caller-stream capture with cold reservation and immutable parameter generations. Retain graph-referenced code, images, workspace, streams and events until model destruction and drain prove safe release. Reject mutations of referenced callables while allowing unrelated ones. Prepare shared device state before the fork, preserving first errors and Unknown outside restored images. Freeze Execute/Skip and aggregate every RTS native callback after its work ends. Nonfinal callbacks return zero; the final callback preserves the selected signed result and its source. Keep the AICore ABI unchanged and use no dynamic Core epoch handshake. Preserve Program terminal cleanup and Kernel retryable ownership. Cover replay identity, result isolation, callback aggregation, parameter versions and capture retirement with CPU, simulation and A3 hardware regressions. A5 hardware, captured-failure propagation, public replay result retrieval, per-replay DFX integration and detailed cost qualification remain open. Related to hw-native-sys#2773, hw-native-sys#2666 and hw-native-sys#2625; this does not close those tracks. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com> Co-authored-by: zhangxingxing33@hisilicon.com <zhangxingxing33@hisilicon.com>
Use common native registration, preparation, submission, completion and cleanup for Program and Kernel execution. Share Host graph construction and remove dedicated execution interfaces and obsolete capture controls. Support A2/A3 HBG caller-stream capture with cold reservation and immutable parameter generations. Retain graph-referenced code, images, workspace, streams and events until model destruction and drain prove safe release. Reject mutations of referenced callables while allowing unrelated ones. Prepare shared device state before the fork, preserving first errors and Unknown outside restored images. Freeze Execute/Skip and aggregate every RTS native callback after its work ends. Nonfinal callbacks return zero; the final callback preserves the selected signed result and its source. Keep the AICore ABI unchanged and use no dynamic Core epoch handshake. Preserve Program terminal cleanup and Kernel retryable ownership. Cover replay identity, result isolation, callback aggregation, parameter versions and capture retirement with CPU, simulation and A3 hardware regressions. A5 hardware, captured-failure propagation, public replay result retrieval, per-replay DFX integration and detailed cost qualification remain open. Related to hw-native-sys#2773, hw-native-sys#2666 and hw-native-sys#2625; this does not close those tracks. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com> Co-authored-by: zhangxingxing33@hisilicon.com <zhangxingxing33@hisilicon.com>
Use common native registration, preparation, submission, completion and cleanup for Program and Kernel execution. Share Host graph construction and remove dedicated execution interfaces and obsolete capture controls. Support A2/A3 HBG caller-stream capture with cold reservation and immutable parameter generations. Retain graph-referenced code, images, workspace, streams and events until model destruction and drain prove safe release. Reject mutations of referenced callables while allowing unrelated ones. Prepare shared device state before the fork, preserving first errors and Unknown outside restored images. Freeze Execute/Skip and aggregate every RTS native callback after its work ends. Nonfinal callbacks return zero; the final callback preserves the selected signed result and its source. Keep the AICore ABI unchanged and use no dynamic Core epoch handshake. Preserve Program terminal cleanup and Kernel retryable ownership. Cover replay identity, result isolation, callback aggregation, parameter versions and capture retirement with CPU, simulation and A3 hardware regressions. A5 hardware, captured-failure propagation, public replay result retrieval, per-replay DFX integration and detailed cost qualification remain open. Related to hw-native-sys#2773, hw-native-sys#2666 and hw-native-sys#2625; this does not close those tracks. Co-authored-by: Chao Wang <26245345+ChaoWao@users.noreply.github.com> Co-authored-by: zhangxingxing33@hisilicon.com <zhangxingxing33@hisilicon.com>
|
@ChaoWao 2625 已更新到 head |
1685c4f to
5b866c0
Compare
|
@HaoZhang1024 2625 现在与 main 冲突(head |
e59230b to
7918e50
Compare
|
@ChaoWao Head |
|
@HaoZhang1024 |
Updated review and concrete convergence splitReviewed fixed head I read zxx0288's complete #2773 proposal and posted the corrected contract. In particular, my older instruction to preserve persistent first-fault Execute/Skip as a requirement was too broad and is withdrawn. Callback coverage and signed RTS error reporting remain necessary; a new counter/last-return protocol must be justified by actual callback behavior, not by my old wording. Delivery split to converge with #2862Please use one common prepared-resource/use owner, one retain/release implementation and the existing common prepare/launch/completion flow. Use #2862's
The likely integration order is common+HBG first, then the TMR increment; dependent local development can proceed against the shared branch without waiting for merge. This is a proposed coordination boundary for the authors to confirm, not a GitHub assignment change. The user will coordinate with the original authors. Required corrections in this fixed head
AcceptanceProgram remains eager only. Kernel eager and capture call the same launch. With DFX off and no user-kernel warmup: capture leaves sentinel outputs untouched, stable-input updates followed by raw RTS replay produce full-array correct results repeatedly, the original run can retire while the model keeps its resources, and queued-copy/partial failure/release behavior remains safe. Add serial multi-node/model ownership coverage against the shared owner. Evidence must name the exact integrated source; older PR/local-branch results are not new-main qualification. Source/contract review only: no new hardware run or current-CI approval is claimed here. Please address these corrections and the ownership split; no second capture executor, PreparedCall cache or Program capture is requested. |
|
Reviewed The previous architecture objections are resolved in this revision. The common prepared-resource/use owners, retain/release API, entry/completion and #2829 observation/finalization contracts match merged main. Please do not reintroduce a separate owner, capture executor, internal version protocol, or normal-success private-D2H gate to address older reviews. The actual new increment over your stacked #2862 prerequisite ( Next action: restack only that increment onto actual main, rather than merge/reapply the old prerequisite stack. A read-only merge preview found one textual conflict in Then validate the resulting tree: common admission and HBG eager controls, both-architecture TMR reconstruction, native/fake-RTS lifecycle tests, and the actual TMR capture scenarios. Current-head/older hardware receipts do not automatically qualify the rebased tree. This comment is a fixed-source alignment review, not a claim that the remaining CI or untested A5/DFX/framework combinations have passed. Our parallel HBG retained-model/INOUT candidate stays independent: one prepared node per model, finalize A's Host run before preparing B. Two simultaneously prepared nodes remain dependent on this shared admission increment, so we will not duplicate it in another PR. |
Allow completed cold capture preparations in distinct granted slots while preserving eager admission, shared-arena checks and exclusive execution. Publish preparation only after backing ownership and the resource plan are complete, so another cold bind cannot detach unfinished preparation state. Use the common native descriptor and prepare/launch/completion flow with shared prepared-resource retention. Keep TMR reconstruction and signed functional error reporting on that foundation without another graph owner, Host replay callback, private success gate or AICore interface extension. Cover raw replay with immutable arguments, serial models, two cold nodes, unexecuted model destruction and signed RTS errors. Verify per-invocation reconstruction without clearing user storage. Balance test-owned primary contexts after borrowed Kernel close and its no-reset assertions, retry the injected one-shot Program cleanup failure, and preserve fatal quarantine.
|
已按最新对齐意见完成更新:消费 #2862 的实际合入提交
按主线 #2898,A5 三套全量 CI 现由管理员 |
|
@ChaoWao 2625 has been restacked to head |
Allow multiple fully prepared native cold-capture nodes to coexist, and qualify TMR capture/replay through main's common execution. The admission change is shared by TMR and HBG; the TMR device reconstruction and signed terminal handoff are already in merged #2862. Unfinished preparation continues to block any bind that could replace its shared arena/configuration. This follows the fixed-source alignment review.
Base and changes
Rebased on main
91f0004517a9f1415228d33bfd8ca95eddbf936d, including #2862's actual merged commita20f8b2e6f2670532b9e1ecc84718022e4b1559a. This PR contains one final commit and eight files: three Host product files and five test files. The prerequisite's implementation/history is part of main and is no longer carried as another PR commit.Publish
Preparedonly after backing ownership and the resource plan are complete. Permit completed cold capture preparations in distinct slots; reject unfinished preparations, mixed eager/capture, duplicate slots and excess capacity before shared state can change.Preserve ordinary eager admission, shared-arena checks, exclusive execution claims, registration guards, actual-stream error observation and delivery-aware finalize. Preparation and graph resource retention use main's existing common APIs and owners.
Cover immutable arguments, no user computation during recording, numerical replay, serial independent models, two cold nodes, destroy-without-execution and signed failures. Reconstruction preserves the user heap. Balance test-owned primary contexts and explicitly retry the injected Program cleanup failure.
Kernel eager and recording use the same executor. Program remains eager-only. No AICore entry/argument extension, second graph owner, replay Host callback, private completion protocol or DFX/D2H success gate is introduced.
Validation
Actual final tree
cc75bac48was revalidated after the alignment review: eight C++ targets covering common admission/HBG eager controls, both-architecture TMR reconstruction, MPI and retirement; 37 C API cases per simulation architecture; and both fake-RTS guarded-reset lifecycle arms passed. Current-main cleanup-retry and caller-no-reset assertions are preserved.The final extension and A5 TMR SDK were rebuilt with PTO pin
5487a6b0. On DT/CANN 9.3.0, allocated development device 0, the same final tree passed release/Program cleanup retry two cases and native raw capture five cases. Source/extension identity, allocated architecture and before/after SDK hashes matched. This is a final-tree hardware receipt, not an automatic carry-over of an older head.Additional integration evidence: six HBG Graph/explicit-predecessor/admission C++ targets, 52 CI policy checks, 55 communication workspace/source tests and applicable hooks passed. Unchanged earlier 458-target and 29-startup/ABI/package coverage remains recorded separately.
Automatic CI for
cc75bac48completed: 13 successful, two skipped, zero failed or pending (run 41997872897). Main CI: run A5 PR hardware checks on demand #2898 runs the full A5 hardware suites on an admin/run-a5request; A5 simulation, CPU and profiling checks remain automatic. The focused DT tests above do not represent the full optional A5 CI suite, multi-device communication capture, DFX or performance qualification.Technical review, CI and formal maintainer approval remain separate. No merge is performed by this update.