Skip to content

fix: join owned Python tasks before execution cancellation returns - #126

Draft
Brian Krabach (bkrabach) wants to merge 1 commit into
mainfrom
fix/owned-execution-cancellation-20261025
Draft

Brian Krabach (bkrabach) wants to merge 1 commit into
mainfrom
fix/owned-execution-cancellation-20261025

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

Summary

Make cancellation of Rust-backed Python RustSession.execute() acknowledge completion of the execution's owned Python work, rather than only dropping the native receiver. The public await joins its debug stage, orchestrator stage, and submitted Python hook runners, including their asynchronous finally blocks, before re-raising asyncio.CancelledError.

This is a draft for behavioral and implementation review, not a merge-ready release PR. The branch remains at version 2.0.2; its locally built test wheel is not a published fixed release. The atomic version bump and release gates below are intentionally incomplete, not waived.

Source: 11ca41713add9f16af050685cd282b851840632a, based on 9614052f7c9e493977dcba3f5c98ebd4d8591ec5.

Why and how

The unmodified native bridge reproduced a public cancellation returning while its separately scheduled Python orchestrator still awaited a cooperatively cancellable tool. A caller could then save, replace, or clean up the session while the old execution remained active.

The change keeps the ownership mechanism at the native/Python boundary:

  • session.rs returns a dormant public coroutine and captures a per-execution loop/context and Rust scope guard. Python stage coroutines are created lazily only when their loop accepts the stage.
  • _async_compat.py retains and observes exactly those stage tasks and submitted hook owners. Rust scope release and Python completion jointly acknowledge settlement. Repeated cancellation cannot abandon a join already draining.
  • bridges.rs enrolls hooks from the emitting task's captured context before submission. Exact hook ownership remains the cancellation authority, so hooks awaited by cancellation finalizers still run. Stale descendant context cannot reopen an already settled execution.
  • CONTRACTS.md specifies the boundary. No cancel-all loop sweep, new lifecycle event, global execution owner, kernel deadline, or dependency/version change is introduced.

The trade-off is deliberate: cancellation completion becomes stronger and can take longer. A noncooperative finalizer can hold the join indefinitely. asyncio.wait_for() around a cancellation-draining operation is not a hard abandonment boundary. Hosts retain responsibility for external supervision; this does not guarantee termination of arbitrary detached descendants, threads, sockets, or WASM work.

Verification evidence

Previously completed isolated checks used the actual maturin-built ABI3 native extension, not a mocked engine, on Linux aarch64 with normal-GIL CPython 3.14.7:

  • All 9 native cancellation regressions passed: ordinary/repeated cancellation, child-finalizer failure, raw/debug hooks, held stage startup, normal/error controls, cancellation before the first public step, finalization hooks/stale context, and held unstarted hook submission. Assertions check settlement at public return, before any cancel-all safety cleanup. The original baseline reproduced the abandonment failures.
  • Python/binding suites: 1,196 passed, with 1 existing KeyboardInterrupt-policy skip. Rust library tests using the dist profile and default WASM features: 534 passed.
  • The same ABI3 wheel on normal-GIL CPython 3.11, 3.12, and 3.13 passed the 9 cancellation cases plus 454 binding/context tests and 1 existing skip per interpreter. CPython 3.14 received the full suite above.
  • A joint four-candidate-wheel offline terminal check observed completed history with four messages and cancelled partial history with two messages, exactly preserved by fresh public resume. Owned native work was settled before return. This is joint-stack evidence, not deployed-host qualification.

To reproduce the focused source regression after preparing a native build, run python -m pytest bindings/python/tests/test_session_execution_cancellation.py -q; use tests/ and bindings/python/tests/ for the broader Python/binding checks.

Cross-host impact and adoption limits

Static traces at Unified 9289170b36dee02686df926486a807946bc4cca8, Agent 08853e3f3814af4501acc0231d26bb1d6e516a84, and Resolve a09242596c9af3f04edc0c68c77ccb044bfa65d0 support the ownership fix, but do not qualify adopting it:

  • Unified remount/shutdown cancels and gathers execution before cleanup. Stronger settlement prevents overlapping old work, but may lengthen remount when cleanup is noncooperative; separate process-stop supervision remains.
  • Agent already uses native Core at its 2.0.1 pin and requires durable conversation settlement before terminal reporting. This fix matches that intent, but does not update the pin or establish its finalization/terminal ordering at runtime.
  • Resolve's host Core 1.6.1 and worker Core 2.0.1 live in distinct immutable dependency/image closures. Only an interpreter actually adopting the changed native bridge gets this behavior. Daemon upgrades do not upgrade existing workers.

No host companion PR, automatic adoption, deployment, or CLI dependency floor is proposed. Synthetic offline/native-local history checks do not cover shared-root UUID sessions, real provider accounts, deployed hosts, or whole Click startup.

Required before merge/release

Under context/release-mandate.md ("Pre-Merge Gate") and docs/CORE_DEVELOPMENT_PRINCIPLES.md section 10:

  • Apply the atomic version bump with scripts/bump_version.py to pyproject.toml, crates/amplifier-core/Cargo.toml, and bindings/python/Cargo.toml; this branch has no bump.
  • Complete the normal freshly built-wheel and scripts/e2e-smoke-test.sh release-readiness evidence, including real tool dispatch, delegation, and recipe execution. The offline checks above do not replace it.
  • Verify downstream Core source-override and other pre-merge requirements for the actual release candidate.
  • Complete the six-platform/18-cell native release qualification and matching receipts required by docs/NATIVE_ARTIFACTS.md. The local interpreter matrix is not that release matrix.

The Core owner, not a delegated author/reviewer, owns the eventual merge/release decision. No merge, tag, package publication, or release artifact publication is requested here.

Compatibility and review focus

No public method signature or dependency floor changes. The observable cancellation timing/ownership contract is strengthened, so review noncooperative-finalizer behavior and loop-shutdown limits explicitly. Recommendation: review the ownership mechanism now; keep this draft until the release-readiness requirements are actually satisfied.

Own each native execution scope and join its Python stages and hook runners
before acknowledging cancellation, including repeated and pre-start cancellation.
Keep teardown hooks and unrelated executions independent. Document the contract
and retain native cancellation regressions.

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants