Repository navigation
[EPIC] Full lambda support #21172
Description
Some features haven't been implemented and others pulled out from #18921 #21679 to keep it from getting too large (>5k LOC) and should be implemented in follow on PR's:
Major features:
- Add support for lambda column capture #21323 to support construction like
select a, exists(array(1, 2, 3), x-> case when a = 2 then x > 1 else null end) - Add lambda substrait support #21193
- Add resolve_lambda_variables helper to Expr and LogicalPlan #22101
- Add exact HigherOrderSignature #22326
- Add transform_values UDF #22689
- fix(lambda): only push referenced params into the merged batch #22853
- feat(lambda): support map non empty nulls cleanup for
HigherOrderFunctionExpr#22847 - Add protobuf support for lambdas #22362
- refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct #22593 (also add alias support)
- add HigherOrderUDF to the public user guide docs
- feat(functions-nested): add array_filter higher-order function #21895
- array_transform index parameter
- add any_match higher-order function #21903
- feat: minor lambda perf improvements #21896
- Add higher-order functions changes to upgrade guide #22107
- minor: make HigherOrderSignature less error-prone #22106
- minor: add higher-order function methods to SessionContext #21950
- Minor: Disallow async function in lambdas #22097
Minor features:
- Named parameters like ScalarUDF already supports:
func(a => 3) - Expr simplifier optimizations for
LambdaVariable - ListView support in remove_list_null_values and HigherOrderUDF::clean_null_values
- Optional ScalarUDFImpl methods not present on HigherOrderUDF(some are unlikely to be used and could be added on demand only):
- simplify
- with_updated_config
- preimage
- placement
- evaluate_bounds
- propagate_constraints
- output_ordering
- preserves_lex_ordering
Activity
LiaCastaneda commentedon Apr 13, 2026
@gstvg I think we should also include as a folloup adding
HigherOrderUDFto the public user guide docs https://datafusion.apache.org/library-user-guide/functions/adding-udfs.htmlgstvg commentedon Apr 14, 2026
@LiaCastaneda added, thanks
Add lambda support and array_transform udf (apache#21679)
LiaCastaneda commentedon Apr 28, 2026
Opened a PR to implement
any_match- #21903@ologlogn also opened a PR for
array_filter- #21895LiaCastaneda commentedon Apr 28, 2026
@gstvg if you don't have a PR/branch already I can open a PR for #21679 (comment), since I think it would simplify the code a bit as well
gstvg commentedon Apr 29, 2026
@LiaCastaneda Feel free to do it, yes, I haven't done nothing on that
[-][DRAFT, EPIC] Full lambda support[/-][+][EPIC] Full lambda support[/+]Add support for lambda column capture (apache#21323)
Add resolve_lambda_variables helper to Expr and LogicalPlan (apache#2…
LiaCastaneda commentedon May 18, 2026
@gstvg I opened a small PR #22326 to add the
exactsignatureLiaCastaneda commentedon May 19, 2026
@gstvg are you working on adding protobuf support? I can open a PR for it this week otherwise.
gstvg commentedon May 19, 2026
@LiaCastaneda I was working on exactly that today 😅 #22362 . Hope that it make into the current release 🤞
LiaCastaneda commentedon May 19, 2026
nice!
Add lambda support and array_transform udf (apache#21679)
Add lambda support and array_transform udf (apache#21679)
Add lambda support and array_transform udf (apache#21679) (#127)
Add support for lambda column capture (apache#21323)
Add support for lambda column capture (apache#21323)
Add support for lambda column capture (apache#21323) (#130)
Add lambda substrait support (apache#21193)
Add lambda substrait support (apache#21193)
Add lambda substrait support (apache#21193) (#134)
refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct (…
Adam-Alani commentedon Jun 1, 2026
Hey, opened up a pr to add map_transform support, similar to array_transform here: #22689
gstvg commentedon Jun 2, 2026
thanks @Adam-Alani, added to the list
Add protobuf support for lambdas (apache#22362)
Add protobuf support for lambdas (apache#22362)
Add protobuf support for lambdas (apache#22362)
Add protobuf support for lambdas (apache#22362)
LiaCastaneda commentedon Sep 7, 2026
👋 I noticed structs as lambda parameters are not supported #25017