[Tech Debt] Align Java and Python recovery model #1016

Activity

1. da-daken

   added

   tech debt[Issue Type] User-unaware issues, such as code refactor and infrastructure maintenance.[Issue Type] User-unaware issues, such as code refactor and infrastructure maintenance.

   priority/majorDefault priority of the PR or issue.Default priority of the PR or issue.

   on Aug 16, 2026
2. wenjin272

   assigned

   da-daken

   on Aug 27, 2026
3. da-daken commented on Sep 4, 2026

   

   da-daken

   on Sep 4, 2026

   Last edited by da-daken

   ContributorAuthor

   More actions

   @weiqingy @wenjin272 Before starting this work, I think we can first align on the recovery model. I believe that for recovery, the function only needs to maintain a unique ID. Currently, matching is done using both the ID and the argsDigest. The argsDigest is essentially an enhancement added by the system to account for the possibility of user operation errors. However, in practice, even if the parameters and the method remain the same, if the data source of that method has changed, it should logically be re‑run. But currently, both the Python side and the Java side can still incorrectly recover in such cases. Fundamentally, we need to maintain a unique ID for all persistent methods within a single action for matching purposes. The multiple tool IDs returned by the LLM are naturally unique; other built‑in actions, such as the chat method of chatAction, are called only once and are thus also unique. For user‑invoked calls, we will hand over control to the user. WDYT?
4. wenjin272 commented on Sep 10, 2026

   

   wenjin272

   on Sep 10, 2026

   Contributor

   More actions

   I think using a unique ID as the recovery identifier makes sense. That said, I’m not very familiar with the Durable Execution mechanism. @joeyutong and @pltbkd, could you also take a look and share your thoughts?
5. pltbkd commented on Sep 10, 2026

   

   pltbkd

   on Sep 10, 2026

   Contributor

   More actions

   Yes — this problem also surfaced in the sub-agent framework work, and it was discussed in #926. The durable id mechanisms of Java and Python differ, and Python's original mechanism cannot distinguish multiple lambdas captured by the same method. Given the different iteration needs on each side, and to keep coordination cost low, we did not previously force the two onto the same scheme. But now both PRs are merged, and the pre-0.4 API cleanup is coming, so I think this is a good moment to address it.

   In an earlier community meeting we reached a consensus: Java and Python differ in language characteristics — since Java methods are not first-class citizens, some natural Python usages require complex interface implementations in Java. We should respect Python's ergonomics, not force Python into a strict correspondence with Java, and instead keep the two sides semantically consistent.

   In the sub-agent framework PR, durable_id was a hard requirement. We introduced durable_id as an optional parameter of the durable_execute family, and to avoid parameter contagion we introduced the with_durable_id / get_durable_id helpers, which inject the durable_id into the durable call instead of actually constructing a type. This lets users opt into a deterministic durable_id, while the default derivation is still used when none is set explicitly. Users keep their Python usage habits and upgrade compatibility is unaffected, but when there is a need they can specify a deterministic id to avoid the lambda-capture problem.

   In the durable batch execution PR, the DurableCall class was introduced to encapsulate a single call within a batch, but to stay aligned with the existing mechanism it did not add any durable_id-related field. I understand this is also the background of the current problem.

   There are possibly several improvement directions, with corresponding divergences:

   1. Should DurableCall gain a durable_id field, or should the func passed into DurableCall be wrapped with with_durable_id?
   2. If DurableCall gains a durable_id field, should durable_execute accept a DurableCall? Further, should with_durable_id then be removed, unifying this need onto DurableCall with durable_id?
   3. Should the automatic digest derivation be kept? It favors Python's ergonomics and avoids upgrade risk for existing jobs, but may implicitly bring the lambda information-loss problem.

   My personal preference is the with_durable_id approach: construct DurableCall with a func wrapped by with_durable_id. On the digest question, my leaning is: when a deterministic id is used, recovery no longer matches the digest; when no id is provided, deriving the digest is still allowed as the fallback.

   I think this is not only a design question of the durable execute mechanism itself, but also a matter of API tidiness. We still have the chance to change the API form, so we can shape the API into its most suitable form. What do you all think would be best?

   👍React with 👍1Reacted by daken
6. wenjin272 commented on Sep 11, 2026

   

   wenjin272

   on Sep 11, 2026

   Contributor

   More actions

   I agree that this issue is primarily about the recovery identity model, rather than the batch execution API. However, pltbkd’s question about how durable_id should be propagated into DurableCall made me revisit the Python batch API introduced in #926. I did not look closely enough at the Python API shape during that review, so I missed this concern at the time.

   Instead of adding another durable_id field to DurableCall, or requiring users to wrap its func with with_durable_id, could we avoid exposing DurableCall as a required user-facing abstraction?

   durable_execute_async already represents a single durable invocation and accepts durable_id. Ideally, batch execution could compose those existing calls:

   outcomes = await ctx.durable_gather(
       ctx.durable_execute_async(foo, x, durable_id="foo-x"),
       ctx.durable_execute_async(bar, y, durable_id="bar-y"),
   )

   If standard asyncio primitives can be supported, this could even be asyncio.gather(...), similar to LangGraph’s Functional API. Otherwise, ctx.durable_gather(...) could provide the same Python-native shape while using the existing durable batch implementation internally.

   With this design, each call carries its own recovery identity when it is created, while durable_gather is only responsible for coordinating concurrency, preserving result order, and recovering the calls as one batch. Therefore, we would not need a separate durable_id representation or propagation mechanism for DurableCall; DurableCall could remain an internal implementation detail.

   There is still some implementation work involved because the current AsyncExecutionResult is lazy and explicitly does not support asyncio.gather. The runtime would need to collect the call handles and plan or reserve the complete batch before starting execution. However, that complexity can remain internal instead of being reflected in the public Python API.

   This would also allow the current discussion to stay focused on the actual identity semantics: an explicit durable_id should be the authoritative recovery identifier, while the automatically derived identity can remain the backward-compatible fallback when no durable_id is provided. WDYT @pltbkd @da-daken ?

   👍React with 👍1Reacted by daken
7. da-daken commented on Sep 13, 2026

   

   da-daken

   on Sep 13, 2026

   Last edited by da-daken

   ContributorAuthor

   More actions

   In an earlier community meeting we reached a consensus: Java and Python differ in language characteristics — since Java methods are not first-class citizens, some natural Python usages require complex interface implementations in Java. We should respect Python's ergonomics, not force Python into a strict correspondence with Java, and instead keep the two sides semantically consistent.

   @pltbkd Thanks for the additional information. Adding args on the Java side would also be a fairly breaking change, so I propose removing args to keep the two languages consistent. From this perspective, keeping this inconsistency also seems to be expected.

   outcomes = await ctx.durable_gather(
   ctx.durable_execute_async(foo, x, durable_id="foo-x"),
   ctx.durable_execute_async(bar, y, durable_id="bar-y"),
   )

   @wenjin272 Great proposal ! Unifying the structure of the Python-side API perfectly resolves the three issues raised by @pltbkd. I also agree that the unique ID should be used first, and that when no durable_id is provided, an automatically derived identifier should be used. This also applies only to the Python-side API; the Java-side API still uses the unique ID. I’m not sure whether my understanding is correct.
8. wenjin272 commented on Sep 14, 2026

   

   wenjin272

   on Sep 14, 2026

   Contributor

   More actions

   This also applies only to the Python-side API; the Java-side API still uses the unique ID

   Yes, this understanding is correct.

   What do you think about the proposed API @pltbkd ?
9. pltbkd commented on Sep 14, 2026

   

   pltbkd

   on Sep 14, 2026

   Contributor

   More actions

   Java won't enable the digest — it's already id-only via DurableCallable.getId(). For API tidiness I'd lean toward dropping the digest field from the state and unifying on a single durable_id; on the Python side, when no durable_id is given we'd implicitly derive one from the current function-id + args-digest, which keeps Python's ergonomics while aligning both sides. But this still leaves the lambda hazard, so I'm also fine with just requiring an explicit durable_id.

   durable_gather is a good Pythonic proposal, +1 — and if we adopt it I'd prefer ctx.durable_gather(...). Java can't take the same shape: durableExecuteAsync returns the value directly (implicitly submitting and yielding), leaving no lazy handle to compose, so Java still needs List<DurableCallable>. Considering that we agreed on respecting the language habits, I think that difference is acceptable.

   But there is one concern: this shape holds only because durable_execute_async is lazy — submitted on await, so the runtime can reserve the whole batch's pending slots before anything starts. If we ever move to eager submission, that reserve-before-execute mechanism (the consistency design from the earlier batch PR) breaks. Do you see a problem here? @da-daken @joeyutong @wenjin272
10. wenjin272 commented on Sep 15, 2026

    

    wenjin272

    on Sep 15, 2026

    Contributor

    More actions

    @pltbkd Thanks, this is a valid concern. After further discussion, we plan to make Java support the same compositional model, tracked in #1122. durableExecuteAsync would return a runtime-managed DurableFuture, while ctx.await() would yield the Continuation instead of blocking:

    var first = ctx.durableExecuteAsync(call1);
    var second = ctx.durableExecuteAsync(call2);
    var outcomes = ctx.await(ctx.gather(List.of(first, second)));

    Regarding lazy execution, I agree that it should be an explicit contract. Initially, both Java and Python handles can remain deferred: gather first reserves all pending slots and then submits the batch, preserving the reserve-before-execute guarantee from #926. Supporting eager submission later would require a separate reservation design.
11. joeyutong commented on Sep 15, 2026

    

    joeyutong

    on Sep 15, 2026

    Collaborator

    More actions

    I agree with keeping durable_execute_async lazy. Submission on await is already part of its public contract, so I think it is reasonable for ctx.durable_gather(...) to rely on that.

    One implementation detail we should account for is recovery matching. Without a reconciler, durable_execute_async currently matches persisted results before the returned AsyncExecutionResult is awaited. This works with the existing await ctx.durable_execute_async(...) pattern: recovery proceeds sequentially, and the cursor is advanced before the next durable_execute_async is evaluated.

    With the proposed durable_gather shape, however, multiple AsyncExecutionResult objects are created before durable_gather starts planning the batch. Recovery matching during their creation could advance the cursor or clear subsequent results before the whole batch is prepared.

    I would therefore defer recovery matching until await as well: standalone await ctx.durable_execute_async(...) would use the existing serial recovery path, while ctx.durable_gather(...) would coordinate recovery matching and reserve the whole batch’s pending slots before execution. This would preserve the reserve-before-execute mechanism.

    👍React with 👍1Reacted by daken
12. da-daken commented on Sep 16, 2026

    

    da-daken

    on Sep 16, 2026

    ContributorAuthor

    More actions

    Thanks everyone for the additional input. It seems we’ve reached consensus on the recovery model: the Java side will only use the unique ID, and the Python side will prioritize the unique ID and fall back to the digest when there isn’t one. I’ll submit a PR for this. Also, it seems we’ve reached consensus on unifying the API. @wenjin272 has already opened an issue: #1122

    👍React with 👍1Reacted by Wenjin Xie
13. da-daken

    closed this as completedon Oct 14, 2026
14. github-actions

    added

    fixVersion/missingThe Bot applies this label when fixVersion is not set.The Bot applies this label when fixVersion is not set.

    on Oct 15, 2026