Skip to content

feat(go): use exact catalog and schema names directly in GetStatistics - #379

Open
logachev wants to merge 1 commit into
adbc-drivers:mainfrom
logachev:feat/go-get-statistics-exact-names
Open

logachev wants to merge 1 commit into
adbc-drivers:mainfrom
logachev:feat/go-get-statistics-exact-names

Conversation

@logachev

Copy link
Copy Markdown
Contributor

What's Changed

GetStatistics resolved the catalog and schema arguments only by matching them against the connection's project and the output of datasets.list. That has two consequences:

  • A catalog other than the connection's project always returned an empty result, so statistics for tables in another project (for example bigquery-public-data) were not available.
  • A schema was only found if datasets.list returned it. Every call listed all datasets in the project first, even when the caller named one schema.

This change resolves exact names directly:

  • Catalog: a catalog given without % or _ is used as the project. A pattern still matches only the connection's project.
  • Schema: a schema given without % is used as it is, and the dataset listing is skipped. A named schema that does not exist returns an empty result, as before. Schemas with % take the listing path unchanged.

Behaviour changes to review

  • An underscore in a schema without % is now taken literally. Under the ADBC pattern rules _ matches any single character, so data_et used to also match dataset; now it matches only data_et. Nearly every BigQuery dataset name contains an underscore, so treating it as a wildcard would send almost every exact name through the listing path.
  • For an exact schema, GetStatistics no longer calls datasets.list. Against a project with about 300 datasets this took a single-table call from 4-5 s to about 1.8 s.

Testing

  • New integration test TestGetStatisticsPublicDataset: statistics for bigquery-public-data.samples.shakespeare from a connection to a different project.
  • TestGetStatistics, TestGetStatisticsHiddenDataset and the new test pass against a live project.

@lidavidm

Copy link
Copy Markdown
Contributor

An underscore in a schema without % is now taken literally. Under the ADBC pattern rules _ matches any single character, so data_et used to also match dataset; now it matches only data_et. Nearly every BigQuery dataset name contains an underscore, so treating it as a wildcard would send almost every exact name through the listing path.

We've dealt with this for Snowflake too and decided to enable this behavior with a flag. We're also working on a proposal that will (among other things) overhaul the metadata APIs to make pattern vs literal match explicit; I should remember to add statistics to the list of followups.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants