Repository navigation
Add SessionConfig reference to ScalarFunctionArgs #13519
Description
Activity
Well, small issue.
ScalarFunctionArgsis indatafusion-exprwhich can't useSessionConfigdirectly since it's indatafusion-executionwhich depends ondatafusion-exprwhich would cause a circular dependency. We can useConfigOptionsthere which would give us access to all config except the opaque extensions inSessionConfigwhich I think is acceptable.Reacted by niebayesGreat work!
Reacted by niebayes and Andrew LambStarted delving into this trying to find a good way to impl. Trying to add
config_options: ConfigOptionstoScalarFunctionExprHaving lots of fun with eq and hash with SessionConfig and f64 atm.I wonder if we can use this trait: https://docs.rs/datafusion/latest/datafusion/catalog/trait.Session.html
We could but it wouldn't solve the issue. The issue is PhysicalExpr requires implementations to impl Eq and Hash or to have implementations for DynEq and DynHash. That is fine until something like SessionConfig which doesn't is introduced. I am looking at updating the either the 'config_namespace' macro or add explicit implementations for 'ScalarFunctionExpr' to try and impl either of the above and use f64.to_bits() and f64::from_bits(..) to handle the problematic f64's in the config. I think it's possible
It makes sense to me to move
SessionConfigto common or common-runtime crateReacted by Andrew Lambtake
It makes sense to me to move
SessionConfigto common or common-runtime crateI agree
I mistyped above - I meant we couldn't use SessionContext, not SessionConfig. Sorry about the confusion.... or not. I need more caffeine today. PR uses ConfigOptions.I guess we can also add
nullableinfo toScalarFunctionArgs#11923Reacted by Andrew LambI guess we can also add
nullableinfo toScalarFunctionArgs#11923It might already be present in
ScalarFunctionArgs::data_type🤔 :datafusion/datafusion/expr/src/udf.rs
Line 336 in f2de2c4
Not really,
DataTypehas no nullable info, we have to sendnullabletoScalarFunctionArgsdatafusion/datafusion/physical-expr/src/scalar_function.rs
Lines 148 to 153 in 6c9355d
// evaluate the function let nullable = self.nullable; let output = self.fun.invoke_with_args(ScalarFunctionArgs { args: inputs.as_slice(), number_rows: batch.num_rows(), return_type: &self.return_type, nullable, })?;
Not really, DataType has no nullable info, we have to send nullable to ScalarFunctionArgs
Ah I was thinking about the nullable info that is part of embedded
Fields -- but that only affects Lists/Maps, etcAdd SessionConfig reference to ScalarFunctionArgs
This is pretty heavy dependency.
SessionConfig contains a LOT of information no scalar function should depend on and very little that has legitimate usage.
I would prefer not adding SessionConfig to ScalarFunctionArgs. Rather, we could add a stripped down config object dedicated to scalar functions. If time zone is all we need, we should add the time zone directly toScalarFunctionArgs(like proposed in #16573).https://docs.rs/datafusion/latest/datafusion/execution/config/struct.SessionConfig.html is mostly a wrapper around the ConfigOptions
If we just want specific fields, we could perhaps reuse the existing ExecutionProps
I think the challenge is that
- The subset of
ConfigOptionsthat functions might want to access is not clear to me - User defined functions may well want to use settings we haven't anticipated (and SessionConfig has a user defined way to pass it)
That doesn't mean we couldn't copy a subset of the fields into ScalarFunctionArgs, it just seems unecessary to try and pick that subset 🤔
- The subset of
If we just want specific fields, we could perhaps reuse the existing ExecutionProps
Are AliasGenerator and VarProviders relevant to function implementors?
The subset of ConfigOptions that functions might want to access is not clear to me
It's not clear to me either. We know about session time zone. Session start time may be relevant.
Do we need to know all of them from the start, or just anticipate we gonna evolve set of what's accessible as the need arises?By avoiding full SessionConfig / ConfigOptions, I would like to retain the state where it's very easy to invoke a function, either from a test, a benchmark, or any other execution context.
Exposing sub-configs such SqlParserOptions, ExplainOptions, or OptimizerOptions to UDFs looks like breaking abstractions, or invitation to break abstractions.
I think my next attempt at solving this was to take the same approach that async-udf took - which is to move the invocation of the udf into an ExecutionPlan. That approach provides the full config_options.
Exposing sub-configs such SqlParserOptions, ExplainOptions, or OptimizerOptions to UDFs looks like breaking abstractions, or invitation to break abstractions.
In light of a push to reduce breaking changes like #16622 (and #16078, #16541, #13648) we could try to be more judicious about growing the public API. IF we don't need to expose full SessionConfig / ConfigOptions in ScalarFunctionArgs let's maybe not expose them.
In light of a push to reduce breaking changes like #16622 (and #16078, #16541, #13648) we could try to be more judicious about growing the public API. IF we don't need to expose full SessionConfig / ConfigOptions in ScalarFunctionArgs let's maybe not expose them.
My counter argument is that having to add new fields to ExecutionProps basically increases the public API over time.
ConfigOptionsis already public so it is my opinion that exposing just theConfigOptionsis less API surface area. I can see how there are alternate opinions to thisHere is what passing in the entire ConfigOptions looks like (it is pretty straightforward actually): #16661
Is your feature request related to a problem or challenge?
ScalarUDFImpl::invoke_with_argsto support passing the return type created for the udf instance #13290 is merged (thanks @joseph-isaacs) we have a place to add new@Omega359 noted that by adding
SessionConfigto theScalarFunctionArgsunblock several tasks such asdatafusion.execution.time_zoneis not used for basic time zone inference #13212Describe the solution you'd like
Add
&SessionConfigtoScalarFunctionArgs, and ideally add a test that shows the config gets through.Describe alternatives you've considered
No response
Additional context
We should try and get this done before DataFusion 44 is released so it isn't a breaking change
44.0.0#13334