Skip to content

Workflow Data References at the SDK-level - #109

Open
WhitWaldo wants to merge 1 commit into
dapr:mainfrom
WhitWaldo:data-reference-envelopes
Open

WhitWaldo wants to merge 1 commit into
dapr:mainfrom
WhitWaldo:data-reference-envelopes

Conversation

@WhitWaldo

Copy link
Copy Markdown
Contributor

This proposal documents the workflow data reference implementation that exists in the 1.19.0 release of Dapr.Workflow for .NET docs and the 3.19.0 release of @dapr/dapr for JavaScript docs. This approach requires no changes on the runtime and is strictly an SDK-level implementation that leaves the door open to future runtime-based functionality, but without any expectation as to the timing of the same.

The two implementations are interoperable with one another when both opt into data references at the cross-app boundary, but the SDK implementation for either also supports interoperability using inline-data by default so as to remain compatible with existing cross-app SDKs that lack this functionality.

JavaScript has no concept of types at runtime - to my knowledge, this isn't a gap that affects any of the other Dapr SDKs, but it's because of this language shortcoming that the implementation has been designed to accommodate this: supporting types in C# and others where it's supported, but not requiring said support in the SDK where those languages like JavaScript lack support.

In the .NET SDK, this is fully compatible with the type-based workflow versioning. JavaScript only supports patch-based versioning. This should not be used alongside patch-based versioning as that's not intended to accommodate changed inputs and outputs on activities. I would urge each of you to emphasize this in your documentation.

I talked about this functionality on today's community call (October 28, 2026).

Local benchmarking of an 11 step .NET workflow with the in-memory binary store provider in which a 2 MB file is input to a workflow and not used across any of the activities until the last one demonstrates that using the automatic mode (preserve existing inline or data references at boundaries, but fully materialize referenced data from the store) requires about 40% more memory in .NET but speeds up workflows by 35%. The increased memory comes from a deference towards backwards, cross-app and opt-out capability than a poor implementation and is a mode intended as a stepping stone until developers can fully embrace explicit payload references at the boundaries of their workflows and activities. Once supported, benchmarking shows that the amount of downloaded data drops by about 82% (as object materialization only happens on explicit request), there's an 81% drop in the amount of memory allocated in the application and the workflow itself completes 2.47x faster.

image

I urge the maintainers of the Python, Java, Go and Rust SDKs to consider implementing both the binary store and this workflow data reference functionality throughout the 1.19 release so we can get some cross-app conformity testing in place, get a quickstart put together and start removing a lot of the hedging in the documentation around using care when implementing against non-conformant SDKs.

…d .NET SDKs

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant