Skip to content

fix(csharp): use BigQuery REST API in GetTableSchema to support PCNT - #358

Open
logachev wants to merge 4 commits into
adbc-drivers:mainfrom
logachev:fix/csharp-pcnt-get-table-schema
Open

logachev wants to merge 4 commits into
adbc-drivers:mainfrom
logachev:fix/csharp-pcnt-get-table-schema

Conversation

@logachev

Copy link
Copy Markdown
Contributor

Summary

Replaces the INFORMATION_SCHEMA.COLUMNS SQL query in BigQueryConnection.GetTableSchema with Client.GetTable via the BigQuery REST API. This enables GetTableSchema to support 4-part Project.Catalog.Namespace.Table (PCNT) table references (such as BigLake Iceberg tables in namespaces), where INFORMATION_SCHEMA views are not supported, and avoids running a BigQuery SQL job to inspect table schemas.

Changes

  • Replace SELECT * FROM INFORMATION_SCHEMA.COLUMNS in GetTableSchema with Client.GetTable(projectId, datasetId, tableName).
  • Build Arrow Field definitions directly from TableFieldSchema (TableFieldToArrowField), including recursive RECORD/STRUCT subfields and NUMERIC/BIGNUMERIC precision and scale.
  • Allow 2-part catalog.namespace dataset identifiers in SanitizeIdentifier while enforcing single-segment validation for project/catalog identifiers in SanitizeCatalogIdentifier.
  • Add AddTable(Table) helper to BigQueryMockServer and update GetTableSchemaTests, MetadataInjectionTests, and SearchPatternTests with REST API and PCNT coverage.

Behavioral Changes from INFORMATION_SCHEMA

  • No SQL Job Execution: GetTableSchema now calls the tables.get REST endpoint instead of submitting a jobs.insert / jobs.query SQL query against INFORMATION_SCHEMA.COLUMNS.
  • Missing Table Handling: If the target table does not exist (404 NotFound from tables.get), GetTableSchema returns an empty Arrow Schema (matching the previous behavior when WHERE table_name = @tableName returned 0 rows).
  • Field DATA_TYPE Metadata: The field metadata entry DATA_TYPE now reflects the BigQuery REST TableFieldSchema.Type string (e.g. INTEGER, FLOAT, BOOLEAN, RECORD) rather than the GoogleSQL INFORMATION_SCHEMA.COLUMNS.data_type string (e.g. INT64, FLOAT64, BOOL, STRUCT<...>). The resulting Arrow DataType and XdbcDataType mappings remain unchanged.
  • Nested STRUCT / RECORD Fields: Nested struct fields are constructed recursively from TableFieldSchema.Fields instead of splitting the STRUCT<...> SQL type string on spaces and commas, which also properly supports deeply nested structs.
  • NUMERIC / BIGNUMERIC Precision and Scale: Precision and scale are read from TableFieldSchema.Precision and TableFieldSchema.Scale (defaulting to (38, 9) for NUMERIC and (76, 38) for BIGNUMERIC when unparameterized).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

REPEATED BigQuery fields are incorrectly mapped to nullable scalars or structs instead of non-nullable lists, causing the returned schema to lose array structure.

4 open issues
What changed in this PR

This PR moves the C# BigQuery driver’s GetTableSchema from SQL queries to the REST API, enabling four-part Project.Catalog.Namespace.Table references without running a query job.

Changes:

  • Build Arrow schemas from REST field definitions, including nested records and decimal precision and scale.
  • Accept catalog.namespace dataset identifiers while retaining single-segment project validation.
  • Add mock table registration and update schema, injection, and identifier tests.
File Description
csharp/​test/​AdbcDrivers.BigQuery.Tests/​SearchPatternTests.cs Tests two-part dataset identifier validation.
csharp/​test/​AdbcDrivers.BigQuery.Tests/​MockServer/​MetadataInjectionTests.cs Updates assertions for REST requests rather than SQL queries.
csharp/​test/​AdbcDrivers.BigQuery.Tests/​MockServer/​GetTableSchemaTests.cs Migrates schema fixtures to REST and adds PCNT coverage.
csharp/​test/​AdbcDrivers.BigQuery.MockServer/​BigQueryMockServer.cs Adds table registration for mock REST responses.
csharp/​src/​AdbcDrivers.BigQuery/​BigQueryConnection.cs Retrieves schemas through REST and converts fields to Arrow types.

Review effort: Balanced

@davidhcoe
davidhcoe self-requested a review November 4, 2026 01:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Omitted-catalog schema lookups can target the billing project instead of the configured data project, returning empty or incorrect schemas, while the unauthorized-credentials test now encounters local identifier validation before authentication.

2 open issues
4 resolved since last review

Review effort: Balanced

{
string query = string.Format("SELECT * FROM `{0}`.`{1}`.INFORMATION_SCHEMA.COLUMNS WHERE table_name = @tableName",
SanitizeIdentifier(catalog), SanitizeIdentifier(dbSchema));
string projectId = string.IsNullOrEmpty(catalog) ? (this.ProjectId ?? string.Empty) : SanitizeCatalogIdentifier(catalog);
});
Func<Task<BigQueryTable?>> func = () => Task.Run(() =>
{
return Client?.GetTable(projectId, datasetId, tableName);
- Prefer configured project_id (including bigquery-public-data) over billing_project_id when catalog is omitted in GetTableSchema, with mock tests.
- Pass valid project, dataset, and table identifiers in AuthenticationTests.CanDetectUnauthorizedCredentials.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants