feat(csharp/src/Drivers/Databricks): Optimize GetColumnsExtendedAsync via DESC TABLE EXTENDED

[jackyhu-db]

Optimize GetColumnsExtendedAsync in Databricks via DESC TABLE EXTENDED

Motivation

Currently, HiveServer2Statement.GetColumnsExtendedAsync will make 3 thrift calls GetColumns, GetPrimaryKeys and GetCrossReference to get all the information and then join them into one result set. Now Databricks introduces a SQL DESC TABLE EXTENDED <table> AS JSON that will return all of these information in one query, this can save 2 extra roundtrips and improve the performance.

Description

Override HiveServer2Statement.GetColumnsExtendedAsync in DatabricksStatement by executing single SQL DESC TABLE EXTENDED <table> AS JSON to get all the required column info for GetColumnsExtendedAsync then combine and join these info into the result. As this SQL DESC TABLE EXTENDED <table> AS JSON is only available in Databricks Runtime 16.2 or above, it will check the ServerProtocolVersion. if it does not meet the requirement, it will fallback the implementation of base class.

Change

• Added DescTableExtendedResult that represents the result of DESC TABLE EXTENDED <table> AS JSON (see format here). It also
  • Parses the string-based table_constraints to the structured primary key and foreign key constraints
  • Adds calculated properties DataType, ColumnSize, FullTypeName which are calculated from the column type and type specific properties
• Changed the access modifiers of some properties and methods in HiveServer2Statement to protected so that they can be used/overridden in subclass DatabricksStatement
  • PrimaryKeyFields
  • ForeignKeyFields
  • PrimaryKeyPrefix
  • ForeignKeyPrefix
  • GetColumnsExtendedAsync
  • CreateEmptyExtendedColumnsResult
• Updated DatabricksStatement with the changes below
  • Added GetColumnsExtendedAsync in DatabricksStatement, it executes query DESC TABLE EXTENDED <table> AS JSON to get all the required info and then json them into the QueryResult
  • Moved the column metadata schema creation logic from GetColumnsAsync to a reusable method CreateColumnMetadataSchema
  • Moved the column metadata data array creation logic from GetColumnsAsync to a reusable method CreateColumnMetadataEmptyArray
• Added a Databricks connection Parameter adbc.databricks.use_desc_table_extended
• Added an internal calculated property CanUseDescTableExtended in DatabricksConnection, DatabricksStatement will call it to decide if it will override GetColumnsExtendedAsync
• Improved the Databricks driver integration test StatementTest:CanGetColumnsExtended
  • Setup the required table resources during the test instead of depending on manual setup before the test
  • Support running multiple test cases from the inputs
  • Add a deep result check to make sure all the properties of all the columns are set correctly

Testing

• Added Test class DescTableExtendedResultTest to cover all the deserialization and parsing cases from raw result of DESC TABLE EXTENDED <table> AS JSON
• Tested all the ColumnsExtended relevant test cases in Databricks/StatementTest

[eric-wang-1990]

Can we mocify/improve the existing GetColumnExtended tests to verify that:
With the flag on/off, they get same number of columns, same schema, and same data?

[jackyhu-db]

I updated it to support test for both flag on/off as below, they are almost same except for the IS_NULLABLE on primary key.

        [InlineData("all_column_types", "Resources/create_table_all_types.sql", "Resources/result_get_column_extended_all_types.json", true, new[] { "PK_IS_NULLABLE:NO" })]
        [InlineData("all_column_types", "Resources/create_table_all_types.sql", "Resources/result_get_column_extended_all_types.json", false, new[] { "PK_IS_NULLABLE:YES" })]
        public async Task CanGetColumnsExtended(string tableName,string createTableSqlLocation, string resultLocation, bool useDescTableExtended, string[] ?extraPlaceholdsInResult = null)

[CurtHagenlocher]

Thanks, looks good! I've just got a few nits about formatting and casing of method names.

[CurtHagenlocher]

Thanks! The changes look good but there's a merge conflict. Could you rebase on top of the latest sourced?

[jackyhu-db]

Thanks! The changes look good but there's a merge conflict. Could you rebase on top of the latest sourced?

Done.

[eric-wang-1990]

LGTM!