Repository navigation
[EPIC] Apply try_to_proto / try_from_proto pattern to ExecutionPlan #22419
Description
#21929 added a try_to_proto / try_from_proto hook on PhysicalExpr so each expression can serialize itself and keep its internal state private, instead of round-tripping through a central ~300-line downcast_ref chain in datafusion-proto. The "Future work" section of that PR calls out applying the same pattern to ExecutionPlan; this issue tracks that work.
Why
ExecutionPlan serialization has the same shape (and the same problems) as PhysicalExpr had:
- A central downcast chain in
datafusion/proto/src/physical_plan/mod.rs— ~30plan.downcast_ref::<…Exec>()branches on encode, plus a symmetricmatchonPhysicalPlanTypeon decode. - Internal state has to be made
pubso the central encoder can read it. proto: serialize dynamic filters on Sort, Aggregate, HashJoin plan nodes #22011 and proto: serialize and dedupe dynamic filters v2 #21807 are recent examples — both added public methods onExecutionPlanimpls (from_parts,inner(),original_children,remapped_children, etc.) for the sole purpose of lettingdatafusion-protoreach in. - Easy to forget when adding a new
ExecutionPlan, or when adding a field to an existing one — the bug only fires when someone serializes a plan that uses it (see e.g. feat(proto): Add protobuf serialization for HashExpr #19379, logical JoinNode proto missing null_aware field (follow-up to #19635) #22065). - Third-party
ExecutionPlanimpls have to go throughPhysicalExtensionCodec, while built-ins go through the central match — same asymmetry thePhysicalExprwork just fixed.
What
Mirror the pattern from #21929:
- Add
try_to_proto/try_from_protohooks onExecutionPlan, feature-gated, defaultOk(None)(fall through to the existing downcast chain). - Migrate one
ExecutionPlanas the working demo (e.g.FilterExecorProjectionExec). - Open one follow-up PR per remaining built-in
ExecutionPlanto migrate it. No wire-format change at any step. - Once all built-ins are migrated, delete the central downcast chain and revert the
pub-for-proto scaffolding added in proto: serialize dynamic filters on Sort, Aggregate, HashJoin plan nodes #22011 / proto: serialize and dedupe dynamic filters v2 #21807.
Scope
Built-in ExecutionPlan impls under datafusion/physical-plan, including (non-exhaustive): ProjectionExec, FilterExec, GlobalLimitExec, LocalLimitExec, HashJoinExec, SymmetricHashJoinExec, SortMergeJoinExec, CrossJoinExec, NestedLoopJoinExec, AggregateExec, EmptyExec, PlaceholderRowExec, CoalesceBatchesExec, CoalescePartitionsExec, RepartitionExec, SortExec, SortPreservingMergeExec, UnionExec, InterleaveExec, WindowAggExec, BoundedWindowAggExec, DataSinkExec, UnnestExec, CooperativeExec, LazyMemoryExec, AsyncFuncExec, BufferExec, ScalarSubqueryExec, ExplainExec, AnalyzeExec, DataSourceExec (and its FileScanConfig / MemorySourceConfig sub-encoders).
The DataSourceExec branch is the gnarliest — it itself dispatches on FileSource / MemorySourceConfig / file format sinks (CsvSink, JsonSink, ParquetSink). It probably wants the same trait-hook treatment one layer down, tracked as a sub-task.
Related: #22418 (the PhysicalExpr equivalent).
Activity
adriangb commentedon May 22, 2026
I'll flag a couple related issues / examples of bugs this fix will eliminate
batch_size— added in Introduce unifiedDataSourceExecfor provided datasources, removeParquetExec,CsvExec, etc #14224/MoveFileSourceConfigandFileStreamto the newdatafusion-datasource#14838 → fixed in feat: Support serde for FileScanConfigbatch_size#15335 (no issue)projection_exprs— added in LetFileScanConfigown a list ofProjectionExprs #18253 → fixed in Split proto serialization to encapsulate private state (#21835) #21929 (issue Split proto serialization to encapsulate private state #21835)partitioned_by_file_group— added in feat: Preserve File Partitioning From File Scans #19124 (reworked Rewrite FileStream in terms of Morsel API #21342, Dynamic work scheduling in FileStream #21351) → fixed in Fix missing fieldpartitioned_by_file_groupin serialization #22365 (issuepartitioned_by_file_groupnot serialized inFileScanExecConfprotobuf breaks datafusion-distributed #22363)file_compression_type— still not serialized (no fix/issue)expr_adapter_factory— intentionally not serialized (runtime factory)Athul0491 commentedon Jul 8, 2026
Hi. I’d like to take a small, scoped part of this issue.
My plan is as follows:
That should establish the pattern without mixing in the broader migration and cleanup. If that sounds right, I can
open a PR for the hook plumbing plus the first migrated plan.
prototype: ExecutionPlan try_to_proto/try_from_proto hook + Projectio…
prototype: add typed bytes-only function serde to the ExecutionPlan p…
prototype: ExecutionPlan try_to_proto/try_from_proto hook + Projectio…
prototype: migrate 7 self-contained ExecutionPlans onto the proto hook (
prototype: migrate 14 more ExecutionPlans onto the proto hook incl. A…
prototype: migrate WindowAggExec + BoundedWindowAggExec onto the prot…
prototype: DataSource/FileSource proto hook + CsvSource migration (ap…
prototype: migrate Json/Arrow/Avro/Parquet FileSources onto the proto…
prototype: migrate AsyncFuncExec + ScalarSubqueryExec onto the proto …
prototype: migrate MemorySourceConfig; DataSourceExec fully on the ho…
prototype: migrate the sink family (DataSink hook + 3 file sinks) ont…
prototype: migrate SortPreservingMergeExec onto the proto hook (apach…
8 remaining items
prototype: migrate the sink family (DataSink hook + 3 file sinks) ont…
prototype: migrate SortPreservingMergeExec onto the proto hook (apach…
prototype: finalize encode dispatch; drop the migration guide (apache…
prototype: DataSource/FileSource proto hook + CsvSource migration (ap…
prototype: migrate Json/Arrow/Avro/Parquet FileSources onto the proto…
prototype: migrate AsyncFuncExec + ScalarSubqueryExec onto the proto …
prototype: migrate MemorySourceConfig; DataSourceExec fully on the ho…
prototype: migrate the sink family (DataSink hook + 3 file sinks) ont…
prototype: migrate SortPreservingMergeExec onto the proto hook (apach…
prototype: finalize encode dispatch; drop the migration guide (apache…
refactor(proto): delegate deprecated ProjectionExec serde shims to ne…
refactor(proto): delegate deprecated ProjectionExec serde shims to ne…
adriangb commentedon Aug 24, 2026
I think this is completed. Anything remaining can be treated in followup issues.
feat: proto serialization for PiecewiseMergeJoinExec (apache#24378)