Skip to content

fix(actor): hold the turn until a cancelled call finishes - #949

Open
eisen0419 wants to merge 1 commit into
dapr:mainfrom
eisen0419:fix/actor-cancelled-call-turn-lock
Open

eisen0419 wants to merge 1 commit into
dapr:mainfrom
eisen0419:fix/actor-cancelled-call-turn-lock

Conversation

@eisen0419

Copy link
Copy Markdown

Description

When a caller cancels an in-flight actor invocation, daprd releases its turn lock and can dispatch the next turn for the same (actorType, actorId) before Node finishes executing the first method and saving state.

Following the merged reference implementation in dapr/dotnet-sdk#1937, this change introduces a reference-counted per-actor ActorTurnLock inside ActorManager so the SDK retains turn ownership until the active invocation (activation, method/reminder/timer callback, post-hook, and state save) finishes.

Summary

 HTTPServer/actor.ts (invoke / remind / timer)
   extract Dapr-Reentrancy-Id header -> ActorRuntime -> ActorManager
   ActorManager.runInTurn(actorId, reentrancyId, fn)
+    lease = await turnLock.acquire(actorId.getId(), effectiveReentrancyId)
+    try:
       getActiveActor(actorId)        # onActivate runs before reentrancy opens
+      lease.allowReentrancy()        # matching non-empty Dapr-Reentrancy-Id may reenter
       callActorMethod(...)           # pre-hook -> method -> post-hook + saveState
+    finally:
+      lease.release()                # depth-- ; wake FIFO waiter only when depth === 0
  • Exclusive per (actorType, actorId): invoke, fireReminder, fireTimer, and deactivate acquire the same per-actor turn lock while allowing distinct actor IDs and actor types to execute concurrently.
  • Reentrancy semantics: Only when actor.reentrancy.enabled === true and Dapr-Reentrancy-Id is non-empty and matches the active turn's ID can a nested call reenter — and only after onActivate finishes (lease.allowReentrancy()). Deactivation ignores Dapr-Reentrancy-Id and waits for activeCalls === 0.
  • Bounded memory: Lock entries are reference-counted across active frames and queued waiters, and evicted immediately once idle.

Evidence

  • Before: If daprd released its sidecar lock early after caller cancellation and dispatched a second request for the same actor ID while the first call awaited async I/O, ActorManager executed both methods concurrently (maxConcurrent === 2), interleaving actor state mutations.
  • After: Subsequent non-reentrant calls, reminders, timers, and deactivations queue in FIFO order until the running method and its state save settle (maxConcurrent === 1); npm run build and npm run test:unit:all pass (87 suites, 2,518 tests including 21 new concurrency/reentrancy/exception/cleanup tests in test/unit/actor/actorTurnLock.test.ts).

Merge Danger

Door: Two-way

Internal runtime synchronization (src/actors/runtime/ActorTurnLock.ts, ActorManager.ts, ActorRuntime.ts, HTTPServer/actor.ts) with backward-compatible optional reentrancyId?: string parameters and zero new dependencies.

Blast Radius: Narrow

Scoped to actor runtime dispatch on the server side; different actor IDs and actor types remain fully concurrent, and outbound client calls are untouched.

Issue reference

Fixes #930

Checklist

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The tests lack coverage proving that turn ownership persists through asynchronous state saves and that queued work resumes and locks are cleaned up after both successful and rejected saves.

1 open issue
What changed in this PR

Adds SDK-side actor turn locking to address #930, preventing caller cancellation from allowing another turn to overlap unfinished actor execution and state persistence.

Changes:

  • Serializes invocations, reminders, timers, and deactivation per actor while preserving concurrency across distinct actors.
  • Passes reentrancy IDs through HTTP dispatch and permits matching chains after activation, with idle lock cleanup.
  • Adds concurrency and lifecycle tests, plus documentation of cancellation behavior.
File Description
test/​unit/​actor/​actorTurnLock.test.ts Adds serialization, reentrancy, lifecycle, cleanup, and HTTP header tests.
src/​implementation/​Server/​HTTPServer/​actor.ts Extracts and forwards reentrancy headers.
src/​actors/​runtime/​ActorTurnLock.ts Implements reference-counted turn locking and waiter handling.
src/​actors/​runtime/​ActorRuntime.ts Forwards optional reentrancy IDs to actor managers.
src/​actors/​runtime/​ActorManager.ts Holds turn leases through actor execution and lifecycle completion.
src/​actors/​runtime/​AbstractActor.ts Documents turn retention after caller cancellation.
CHANGELOG.md Describes the cancellation fix and locking behavior.

Review effort: Balanced

Comment thread src/actors/runtime/ActorManager.ts Outdated
res = await actorObject[actorMethodName](args);
}

await actorObject.onActorMethodPostInternal();

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added regression tests in f6750d7 with a state-mutating actor and deferred ActorClientHTTP.stateTransaction mocks covering post-method, activation, and deactivation saves across both resolved and rejected transactions (asserting queued invocations and deactivation stay blocked until persistence settles and lock entries drop to 0 afterward).

Keep one turn per actor id until the method and its state save finish.
A matching non-empty Dapr-Reentrancy-Id joins that turn after activation,
including when the outer activation id is blank or the inner id differs,
and reuses the actor already activated for the chain.

A call outside that chain with a new non-empty Dapr-Actor-Activation-Id
detaches the stale actor before waiting, so the in-flight call cannot
save. The stale onDeactivate runs in the next root turn, after the
in-flight call releases the lock and before the new onActivate. DELETE
unregisters the actor before waiting and can still save unless a newer
activation id supersedes it. An empty activation id still does not wait
on the sidecar version lookup before the call queues.

Fixes dapr#930

Signed-off-by: eisen0419 <usetoken0419@gmail.com>
@eisen0419
eisen0419 force-pushed the fix/actor-cancelled-call-turn-lock branch from f6750d7 to f6e44b0 Compare November 16, 2026 21:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants