Repository navigation
Add: support native HBG capture and serial replay - #2862
Conversation
|
Important Review skippedToo many files! This PR contains 111 files, which is 11 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
 |
8df8cb6 to
ece1f73
Compare
Review and concrete convergence with #2625Reviewed fixed head Proposed delivery ownershipUse this PR's
Our local Core2 is an overlapping implementation, not a mandatory third PR to stack after the two author PRs. We will not introduce another retain/release or model registry. In particular, your existing image/scheduler detachment already covers some of its duties. Choose the authoritative writer and order for each restored region: do not combine its pristine DMA with your device reset indiscriminately. Common+HBG is the proposed first complete delivery; TMR can develop against that shared foundation before merge. Please confirm this coordination boundary; the user will coordinate with the original authors. No assignment change or request for an additional Worker is implied. Required fixes1. Reject Program capture before publishing recording state. 2. Handle Host orchestration effects before they escape preparation. Provide an equivalent ordered replay effect, or reject unsupported effects before caller-visible writes. A pre-effect capability check or private staging/deferred effects may establish that boundary; finding the effect only at retain/record after it happened does not. Preserve read-only Host-control snapshots separately, and do not introduce implicit Kernel IO H2D/D2H or a requirement to rerun Host orchestration for replay. Add a writable-HOST negative control and a read-only snapshot positive control. Our old Core2 effect flag was recorded after the underlying operation, so copying it alone is not a solution either. 3. Preserve merged #2829's completion/observation/delivery behavior during rebase. This branch is based on Clarifications / small corrections
Acceptance and review statusKeep one common native prepare/launch, no eager user warmup, no separate capture executor or AICore ABI. The initial delivery must prove DFX-off capture does not compute, stable input addresses give repeated full-array numerical results, original-run retirement does not end model-held lifetimes, caller-stream observation is the RTS error route, and partial/failed cleanup preserves live resources. HBG initial task states may be nonzero; keep the existing inline/nested restore semantics and do not clear persistent user IO. Your declared limited A3 evidence is useful; A5, SDK capture/cancel/destroy failures and newer-main integration must retain their actual qualification boundaries. The proposed common callback aggregation is not rejected by name, but its necessity and coverage must follow the real callbacks and error-return behavior. Changes requested on the concrete Program and Host-effect boundaries, plus the shared-main integration above. This is pinned-source review; no new build/hardware run or blanket capture approval is claimed. |
ece1f73 to
d7087cb
Compare
3a1aa6b to
2b2ac4d
Compare
08b31dd to
452d028
Compare
b0fa3af to
fdea665
Compare
fdea665 to
81e3cea
Compare
81e3cea to
b979736
Compare
b979736 to
3c61548
Compare
3c61548 to
30e365d
Compare
Retain preparation backing independently of a host run and record Kernel capture through the common native submission path. Restore HBG execution state at device entry without an extra host reset task. Integrate retention with per-call completion, original-stream observation and controlled output delivery. Keep old observations alive across slot reuse, preserve unknown-use cleanup and block owned reset on explicit external obligations. Reject Program capture before mutating run state. Declare capture preparation before Host orchestration and reject writable HOST/NONE arguments before effects escape. Preserve eager writes and readonly control snapshots; disallow upgrading eager preparation to capture. Distinguish optional private publication from invalid producer state and verify A5 runtime results through the common native completion owner. Aggregate native AICPU callbacks, preserve bounded fault evidence and retain cleanup failures for retry. Add lifetime, observation and callback regressions plus A3 capture tests with changed inputs and repeated replay. Document the current native API, ownership and support boundaries in English. Value-initialize arena preparation objects in place so A3 default Werror builds preserve object lifetime and arena reuse without warning overrides. Exercise real RTS capture cancellation, recoverable End argument rejection, retained-resource release retry and subsequent capture on the same context. Qualify real RTS capture conflicts and unjoined End failures, and instrument test-only partial submission to verify retention, stop and recovery. Preserve unlaunched run cleanup ownership when thread attachment fails. Separate external release requests from device completion so confirmed owned reset can retire faulted resources without resetting borrowed ones. Invalidate TMR handshake reports only after acknowledged core retirement. Use per-run simulated callback rendezvous and stage the entire thread cohort before releasing any device work. Cancel partial thread creation without stranding callbacks or simulated cores. Share native invocation initialization, completion and required-symbol loading across backends. Use one current prepare/run descriptor and entry family for eager and capture preparation, with native callers rebuilt against that layout. Retain complete simulation fault evidence until explicit acknowledgement or backing destruction. Recover owned close from sticky device errors through confirmed reset and centralize aligned storage and result ownership. Keep ordinary Program reset failures reachable for close retry after resource release, without unloading a retired binary again. Report call-owned failure diagnostics after output bindings have retired, without copying user outputs or using diagnostics as physical completion proof. Value-initialize A5 arena objects and include CANN profiling headers in the capture driver fixture. Cover staged unload/reset failure and post-retirement error observation in regression tests. Qualify teardown fault injection by resource identity and outermost reset requests, excluding SDK-internal ACL-to-RT delegation and unrelated frees. Exercise both SDK reset shapes and failed-allocation retry accounting without hardware, while retaining real-device lifecycle regressions. Release obsolete managed backing by retained allocation identity and retry its unmapped free failures without consuming unresolved resource tokens. Preserve unrelated failure, mapping and unknown-use debt; do not retry successfully freed or reused addresses. Cover staging and arena cleanup through the public API on both architectures and runtimes. Keep affinity-gate tests portable by restricting Linux CPU lookup to Linux. State eager, consumed Host-control and captured device-resource lifetimes at the native API boundary. Provide one scheduler-timeout fixture for the A5 retirement wiring tests. Use a bounded override only for the missing-handshake case, preserving normal retirement tests' default watchdog and avoiding duplicate symbols. Recheck MPI group predicates after timed waits reacquire their mutex. An already-completed reply does not withdraw frames moved by its leader, and a retired predecessor does not reject the next batch as still active. Regression cases hold the mutex across each deadline without changing production timeout budgets.
30e365d to
ea2f2fb
Compare
Summary
Enable cold native HBG capture through the existing
simpler_prepare_runandsimpler_launch_run_on_streampath. RTS records and serially replays device work without an eager warmup or another Simpler host prepare/reset between replays. Program remains eager-only.Related to #2666 and #2773. This PR supplies the common preparation-resource/use ownership and HBG reconstruction; #2625 can consume the same execution and retention contracts for TMR rather than introducing another model/resource owner.
API and execution
NativeRunDescriptor.prepare_purposedeclares eager or capture preparation before Host orchestration. There is one current descriptor and one prepare/run entry family; native clients rebuild together. No ABI-version field, PreparedCall cache or separate capture executor is added.Ownership, device state and errors
Current contracts: execution/API, resource lifetime, HBG restoration. These are English product contracts. This PR adds no Chinese counterpart; preexisting Chinese Graph-task documentation is unchanged.
Earlier review-repair validation (
b9797366f)sched_getcpufailure was reproduced by compiling that non-Linux source path; the corrected guard preserves the test target.Integration validation (
30e365db3)Rebased onto main
a19801e86(#2826). Preserve its retryable incomplete-wait result, undrained SIM carrier/claim retention and ChipWorker observation/finalize retry behavior. Resolve both architecture SIM startup conflicts using one shared definition of the existing main startup gate, with native callback initialization/completion layered inside successful startup. The gate is relocated to the common host include directory without implementation changes; the duplicate gate and obsolete staging wrapper are removed.The A5 retirement wiring fixture uses one timeout provider. Only the missing-handshake regression enables the short watchdog; normal retirement cases retain their default timeout. This closes the duplicate-definition build failure observed in the preceding macOS job and independently reproduced locally.
MPI timeout repair and current validation (
ea2f2fb6d)Current base is main
89278fff3. The macOS UT job at 30e365db3 compiled successfully and passed Python tests, then failed onlytest_remote_endpoint(449/450 CTest entries passed).MPI group timed waits now recheck their protected predicates after reacquiring the mutex. A completed leader can have moved the frame vector before a waiter reports timeout; completed replies must bypass arrival withdrawal. A retired predecessor similarly must not be rejected as still active. Production deadlines, batching, error cleanup and public APIs are unchanged.
zxx0288/simpler@ea2f2fb6de1e7bfd39e7d3dc2817cda8616d4d1e, macOS ARM64 and Apple Clang. This is targeted native acceptance, not a rerun of all 450 macOS CTest entries. Main now schedules macOS under Daily; its absence from PR CI was not used as a fix. The diagnostic workflow is not part of this PR and books no hardware.Hardware evidence and limits
Pre-repair head
81e3cea59actually executed and passed all 14 A3 HBG native capture scenarios in CI job 127026853292. They include numerical ordinary/early/nested replay, readonly Host snapshots, model lifetime beyond Host finalize, cancellation/invalidated/partial capture cleanup, two bounded scheduler timeouts and controlled post-handshake boot failure recovery. That evidence remains tied to its tested revision; it is not a new hardware run of these repairs.A5/TMR capture hardware, DFX capture, concurrent replay, model parameter/address updates, zero-user-task/persistent-INOUT hardware cases and a retained-model-A/prepare-B/replay-A hardware case remain unqualified. Permanent missing callbacks, unknown physical exit and arbitrary device faults do not have a claimed recovery guarantee. CPU/Host substitutes do not establish silicon behavior or physical retirement. No performance or full-platform qualification is claimed.
The earlier A3 queue/client-budget interruption is tracked separately in #2887; it is not treated as a hung kernel or as a substitute for passing CI.