Register SQL planners in SessionState::new() #11216

Activity

1. dharanad commented on Jul 2, 2024

   

   dharanad

   on Jul 2, 2024

   Contributor

   More actions

   hello @jayzhan211 , i would like to work on this. QQ: Is this issue any straight as it sounds or there is more to it?
2. jayzhan211 commented on Jul 2, 2024

   

   jayzhan211

   on Jul 2, 2024

   ContributorAuthor

   More actions

   hello @jayzhan211 , i would like to work on this. QQ: Is this issue any straight as it sounds or there is more to it?

   A little more than it. Ideally we need to update map capacity in provider with the number of references. But, I think you can try it.
   

   datafusion/datafusion/core/src/execution/session_state.rs

   Lines 590 to 608 in 75b9c9b

   	let references = self.resolve_table_references(&statement)?;
   	let mut provider = SessionContextProvider {
   	state: self,
   	tables: HashMap::with_capacity(references.len()),
   	};
   	for reference in references {
   	let resolved = &self.resolve_table_ref(reference);
   	if let Entry::Vacant(v) = provider.tables.entry(resolved.to_string()) {
   	if let Ok(schema) = self.schema_for_ref(resolved.clone()) {
   	if let Some(table) = schema.table(&resolved.table).await? {
   	v.insert(provider_as_source(table));
   	}
   	}
   	}
   	}
   	let query = self.build_sql_query_planner(&provider);
3. dharanad commented on Jul 2, 2024

   

   dharanad

   on Jul 2, 2024

   Contributor

   More actions

   take
4. github-actions

   assigned

   dharanad

   on Jul 2, 2024
5. dharanad commented on Jul 3, 2024

   

   dharanad

   on Jul 3, 2024

   Last edited by dharanad

   Contributor

   More actions

   hello @jayzhan211 , i would like to work on this. QQ: Is this issue any straight as it sounds or there is more to it?

   A little more than it. Ideally we need to update map capacity in provider with the number of references. But, I think you can try it.

   datafusion/datafusion/core/src/execution/session_state.rs

   Lines 590 to 608 in 75b9c9b

   	let references = self.resolve_table_references(&statement)?;
   	let mut provider = SessionContextProvider {
   	state: self,
   	tables: HashMap::with_capacity(references.len()),
   	};
   	for reference in references {
   	let resolved = &self.resolve_table_ref(reference);
   	if let Entry::Vacant(v) = provider.tables.entry(resolved.to_string()) {
   	if let Ok(schema) = self.schema_for_ref(resolved.clone()) {
   	if let Some(table) = schema.table(&resolved.table).await? {
   	v.insert(provider_as_source(table));
   	}
   	}
   	}
   	}
   	let query = self.build_sql_query_planner(&provider);

   Do you mean something like this

           let mut provider = SessionContextProvider {
               state: self,
               tables: HashMap::with_capacity(references.len() + self.user_defined_sql_planners.len()),
           };
   

   If yes, why is required ? I this that we are already updating the capacity with reference count. If i misinterpreted it can you please elaborate more ?
6. alamb

   closed this as completedin #11253on Jul 3, 2024
7. alamb commented on Jul 5, 2024

   

   alamb

   on Jul 5, 2024

   Contributor

   More actions

   Update here is I think the current formulation is about as optimal as we can get it:
   

   datafusion/datafusion/core/src/execution/session_state.rs

   Lines 234 to 242 in dce77db

   	let user_defined_sql_planners: Vec<Arc<dyn UserDefinedSQLPlanner>> = vec![
   	Arc::new(functions::core::planner::CoreFunctionPlanner::default()),
   	// register crate of array expressions (if enabled)
   	#[cfg(feature = "array_expressions")]
   	Arc::new(functions_array::planner::ArrayFunctionPlanner),
   	#[cfg(feature = "array_expressions")]
   	Arc::new(functions_array::planner::FieldAccessPlanner),
   	#[cfg(feature = "datetime_expressions")]
   	Arc::new(functions::datetime::planner::ExtractPlanner),

   (namely to make a Vec with a size known at compile time