I sketched out some of the upgrade in #662 and wanted to share what I encountered.

Some major changes in between datafusion 36 and 37.

Needs Completing

• 1) Porting functions.rs BuiltInFunctons to datafusion-functions
• 2) Replacing SectionContext::tables or updating the tests to not rely on it
• 3) new trait methods for ExecutionPlan and ExecutionPlanProperties for DatasetExec need implementing

1) Porting BuiltinFunctions

See this epic apache/datafusion#9285.

That means much of datafusion-python::functions.rs needs an update. An incomplete first pass is 83249fe

2) SectionContext::tables

It was removed in apache/datafusion#9627

Many of the tests rely on this method. Does each test get updated to a new API or do we implement our own version to start and then update the tests until it's not necessary?

3) New Trait Methods

I just marked these as TODO to get the upgrade to compile. See 8cd4fd4