Repository navigation
Support registration of custom TableProviders through SQL #3310
Description
Activity
Sorry for commenting on closed thread, question for @avantgardnerio and/or @alamb
question, should
datafusion::datasource::datasource::TableProviderFactory::createhaveOptional<DFSchemaRef>or justDFSchemaRefas a parameter to support queries where schema can be provided as part of crate statementCREATE EXTERNAL TABLE demo_table (c1 integer, c2 integer) STORED AS ...?@milenkovicm I think that's not a bad idea, and some versions of the PR did exactly that... however I do think there's one choice to be made:
- do we add an optional parameter to the existing
asyncmethod? - or do we make a new synchronous method (perhaps
with_schema) to do the same?
I would be fine either way, but I think the purpose of the
asyncpart was to allow providers to access the network to infer the schema, so it might not be needed if the schema is provided.- do we add an optional parameter to the existing
thanks @avantgardnerio ,
I believe first option may make more sense, might be easier api to digest.
Other small note, as I'm working on a branch where I've added this change Optional may be redundant as empty schema will be provided by context.rs
empty schema will be provided by context.rs
FWIW, I would advocate for the presence of an empty schema to be represented differently than the absence of a schema.
@avantgardnerio do you mean changing
CreateExternalTable::schemafromDFSchemaReftoOptional<DFSchemaRef>?I meant as a concept in general - not using
""or0or[]to meanNone. But specifically here I meantpub trait TableProviderFactory: Sync + Send { /// Create a TableProvider given name and url fn create(&self, name: &str, url: &str, schema: Option<DfSchemaRef>) -> Arc<dyn TableProvider>; }vs
pub trait TableProviderFactory: Sync + Send { /// Create a TableProvider given name and url /// /// Arguments: /// schema - the schema if known, otherwise an empty schema object to be populated later fn create(&self, name: &str, url: &str, schema: DfSchemaRef) -> Arc<dyn TableProvider>; }Reacted by Marko Milenković@avantgardnerio please have a look at #4142 should support what we've discussed. thanks
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As a DataFusion / Ballista user, one awesome feature is that I can create custom table providers and register them on the context to support datasources not built into DataFusion itself (e.g. DeltaLake). However, if I make my own server (i.e. Ballista fork or Ballista with plugins) I cannot allow my users to register arbitrary tables at run time through SQL - I need to hard-code those into my app.
Describe the solution you'd like
I would like to extend the
create external tablesyntax to allow registration of arbitrary table types.Describe alternatives you've considered
create external table