Skip to content

fix(csharp): use BigQuery REST API in GetObjects to support PCNT - #357

Open
logachev wants to merge 4 commits into
adbc-drivers:mainfrom
logachev:fix/csharp-pcnt-get-objects
Open

logachev wants to merge 4 commits into
adbc-drivers:mainfrom
logachev:fix/csharp-pcnt-get-objects

Conversation

@logachev

@logachev logachev commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replaces INFORMATION_SCHEMA SQL queries (TABLES, COLUMNS, TABLE_CONSTRAINTS, KEY_COLUMN_USAGE, and CONSTRAINT_COLUMN_USAGE) in GetObjects (GetTableSchemas, GetColumnSchema, and GetConstraintSchema) with Client.ListTables and Client.GetTable via the BigQuery REST API. This enables GetObjects to support 4-part Project.Catalog.Namespace.Table (PCNT) datasets, aligns the C# driver with the Go driver implementation, and eliminates per-dataset and per-table SQL queries during metadata enumeration.

Changes

  • Replace INFORMATION_SCHEMA.TABLES in GetTableSchemas with Client.ListTables(catalog, dbSchema), matching tableNamePattern (via PatternToRegEx) and tableTypes client-side.
  • When depth != GetObjectsDepth.Tables, fetch full table metadata once per matching table via Client.GetTable(catalog, dbSchema, tableName) and pass the BigQueryTable resource to GetColumnSchema and GetConstraintSchema.
  • Populate GetColumnSchema from table.Schema.Fields (TableFieldSchema) and GetConstraintSchema from table.Resource.TableConstraints (PrimaryKey and ForeignKeys), removing GetConstraintColumnNames and GetConstraintsUsage.
  • Allow 2-part catalog.namespace dataset identifiers in SanitizeIdentifier while enforcing single-segment validation for project/catalog identifiers in SanitizeCatalogIdentifier.
  • Add tables.list support and AddTable(Table) to BigQueryMockServer, and update MetadataInjectionTests and SearchPatternTests with PCNT GetObjects tests.

Behavioral Changes from INFORMATION_SCHEMA

  • No SQL Job Execution: GetObjects now uses tables.list and tables.get REST API calls instead of executing up to $1 + 4N$ INFORMATION_SCHEMA SQL queries per dataset.
  • Primary Key constraint_name:
    • In BigQuery DDL, PRIMARY KEY constraints are unnamed. INFORMATION_SCHEMA.TABLE_CONSTRAINTS synthesized <table_name>.pk$ for constraint_name, whereas the REST API's TableConstraints.PrimaryKey has no Name property. Because StandardSchemas.ConstraintSchema requires a non-null string for constraint_name, GetConstraintSchema now populates "" (string.Empty) for primary key constraint_name.
    • constraint_type ("PRIMARY KEY"), constraint_column_names (PrimaryKey.Columns), and constraint_column_usage (null) remain unchanged.
  • Foreign Key Scoping: Previously, GetConstraintsUsage queried INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE WHERE constraint_name = @constraintName across the entire dataset without filtering by table name, which could return rows from other tables in the same dataset if they shared a foreign key name. Reading table.Resource.TableConstraints.ForeignKeys scopes foreign keys strictly to the current table.
  • Column remarks and xdbc_is_generatedcolumn in GetColumnSchema:
    • remarks now populates field.Description ?? string.Empty instead of always hardcoding "".
    • xdbc_is_generatedcolumn is set to false (BigQuery does not support generated columns; INFORMATION_SCHEMA.COLUMNS.is_generated was always "NEVER").
  • Column xdbc_type_name in GetColumnSchema: xdbc_type_name now reflects the REST TableFieldSchema.Type string (e.g. INTEGER, FLOAT, BOOLEAN, RECORD) rather than the GoogleSQL INFORMATION_SCHEMA.COLUMNS.data_type string (INT64, FLOAT64, BOOL, STRUCT). xdbc_sql_data_type (ToXdbcDataType) handles both naming conventions and produces the same values.
  • Table Type Mapping & Access Handling:
    • tables.list returns "TABLE" for standard tables, which GetTableSchemas maps to "BASE TABLE" to match INFORMATION_SCHEMA.TABLES (tableTypes filter accepts either "BASE TABLE" or "TABLE").
    • If a table is concurrently deleted (404 NotFound) or the caller lacks table-level permission (403 Forbidden) during Client.GetTable, the table is skipped instead of failing the entire GetObjects call.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved findings show that lazy table listing bypasses retry and token-refresh handling, while REST metadata changes regress materialized-view and clone filtering and array type and nullability reporting.

5 open issues
What changed in this PR

This PR moves C# GetObjects metadata enumeration from INFORMATION_SCHEMA queries to BigQuery REST APIs, enabling four-part Project.Catalog.Namespace.Table (PCNT) datasets without SQL jobs.

Changes:

  • Use table listing and metadata APIs for tables, columns, and constraints, with client-side filtering.
  • Accept two-part dataset identifiers while keeping project identifier validation single-segment.
  • Extend the mock server and tests for REST enumeration and PCNT datasets.
File Description
csharp/​test/​AdbcDrivers.BigQuery.Tests/​SearchPatternTests.cs Update identifier validation cases for two-part datasets.
csharp/​test/​AdbcDrivers.BigQuery.Tests/​MockServer/​MetadataInjectionTests.cs Add REST enumeration and PCNT test cases.
csharp/​test/​AdbcDrivers.BigQuery.MockServer/​MockRequest.cs Add a table-list request category.
csharp/​test/​AdbcDrivers.BigQuery.MockServer/​BigQueryMockServer.cs Support table registration and REST table listing.
csharp/​src/​AdbcDrivers.BigQuery/​BigQueryConnection.cs Replace metadata SQL queries with REST calls and update identifier validation.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Regression coverage is insufficient: the PCNT and pattern-filter tests can pass without verifying the intended behavior, and the REST foreign-key conversion lacks tests for ordering, referenced metadata, and table scoping.

3 open issues
5 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants