Skip to content

feat(go): implement ExecutePartitions and ReadPartition - #371

Open
geoHeil wants to merge 3 commits into
adbc-drivers:mainfrom
geoHeil:fix/read-partition-stream-options
Open

geoHeil wants to merge 3 commits into
adbc-drivers:mainfrom
geoHeil:fix/read-partition-stream-options

Conversation

@geoHeil

@geoHeil geoHeil commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Implement ExecutePartitions and ReadPartition using BigQuery Storage Read API sessions so callers can distribute query results across connections or processes. Partition descriptors contain only stream names; each reader uses its own connection's credentials.

ExecutePartitions returns the session's Arrow schema and the exact row count from destination-table metadata, which requires one extra request. Session and reader schemas normalize JSON, GEOGRAPHY, and nested extension fields consistently. Empty results return no partitions.

Two statement options control the requested stream count: bigquery.query.partitions.max_streams (default 4) and optional bigquery.query.partitions.target_rows_per_stream. BigQuery may return fewer streams than requested. Compression is set on the session because session streams reject per-request serialization options. Dry runs and bound parameters are refused.

Assisted-by: OpenAI Codex (GPT-6)

Closes #367.

ExecutePartitions and ReadPartition returned NotImplemented. Now they
work.

ExecutePartitions runs the query as ExecuteQuery does. Then it opens a
multi-stream Storage Read API session on the destination table of the
query job. It returns one partition for each stream. It also returns the
Arrow schema of the session and the exact row count of the result.

A partition descriptor holds a version byte and the name of a stream. It
holds no credential. ReadPartition decodes the descriptor and reads the
stream with the credentials of its own connection. A malformed
descriptor gives InvalidArgument.

A session stream accepts no per-request Arrow serialization options.
BigQuery refuses them with InvalidArgument. So ReadPartition sends only
the stream name, and the session carries the ZSTD compression. The
session also asks for a minimum stream count that equals the maximum.
Without it, BigQuery can return one stream for a small table. BigQuery
can still return fewer streams than requested.

Two new statement options set the stream count:
bigquery.query.partitions.max_streams (default 4) and
bigquery.query.partitions.target_rows_per_stream (default: unset).

ExecutePartitions refuses a dry run and bound parameters.
geoHeil added a commit to telekom/sutura that referenced this pull request Nov 6, 2026
The partitions patch lacked two fixes that the upstream change
(adbc-drivers/bigquery#371) carries, so sutura did not run what it
proposed:

- ExecutePartitions ran a statement with bound parameters and ignored
  them, so a caller got rows for a query it never parameterised. It now
  refuses them as NotImplemented before running the query.
- readPartitionRecordReader opened the ReadRows gRPC stream on the
  caller's context and never cancelled it when decoding the stream's
  schema failed, leaving the stream open behind an error that returns no
  reader to release. It now cancels its context on any error return.

The patch is regenerated from the pinned source (go/v1.14.1) with the
three earlier patches applied: the old patch came back byte-for-byte
before the new one was cut, and the new one applies to that tree and
reproduces the edited tree.

Cells (Go, run by nix/bigquery-adbc.nix's check phase and listed there;
not run locally, CI only):
- TestExecutePartitionsRefusesBoundParameters: red when the params
  refusal is neutralised (`st.params != nil && false`, the field still
  read), as the call then goes on past the refusal and panics on the
  test statement's missing connection instead of returning NotImplemented.
- TestPartitionReadClosesStreamWhenDecodingFails: a fake Storage Read API
  sends an undecodable schema and holds the stream open; red when the
  deferred cancel is dropped, as the server never sees the stream end.

VENDOR.md names adbc-drivers/bigquery#371 and the removal condition: the
first driver release that contains it.

Limit: the cells are written against the module cache but were not run
here; the CI check phase is the first run.

Part of #828

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Automated review generated by GPT-6.1 Sol.

Adds partitioned query execution through Storage Read API sessions.

  • [P0] Validate the current implementation against BigQuery before submission. The description explicitly says this version has not run against BigQuery.
  • [P0] Shorten the description: remove the test inventory and execution report, retain the relevant design decisions, and include the required Assisted-by: or Generated-by: trailer identifying any AI assistance.
  • [P1] Do not add mocked tests for this feature. Instead, add actual integration tests against BigQuery and run those tests yourself before submission.

Comment thread go/job_query.go
return nil, errToAdbcErr(adbc.StatusIO, err, "open read partition stream")
}

rdr, schema, err := ipcReaderFromArrowIterator(arrowIterator, nil, "", alloc)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply the same schema normalization when reading partitions. ExecutePartitions normalizes BigQuery extension metadata, but this path retains the original IPC schema. For JSON, I reproduced arrow.json in the advertised schema versus google:sqlType:json in the reader schema. GEOGRAPHY and nested extension fields are affected too. Normalize the reader and batch schemas consistently so consumers receive the advertised types.

Keep session, reader, batch, and nested column schemas consistent, and
return no partitions for empty results. Replace mocked Storage Read API
tests with live cross-connection regressions and enable partition
conformance coverage.

Assisted-by: OpenAI Codex (GPT-6)
@geoHeil

geoHeil commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hope that my last commit adresses your findings.

Comment thread go/docs/bigquery.md Outdated
Comment on lines +142 to +146
### Partitioned Results

`ExecutePartitions` runs the query, then opens a multi-stream [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage) session on the destination table of the query job. It returns one partition for each stream and reports the exact row count of the result. `ReadPartition` reads one partition. A partition descriptor holds only the name of a stream and a version byte. It holds no credential, so any connection can read it with its own credentials.

BigQuery can return fewer streams than the driver asks for. `ExecutePartitions` does not support dry runs or bound parameters. See `bigquery.query.partitions.max_streams` and `bigquery.query.partitions.target_rows_per_stream` in [](#query-options) below.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Partitioned Results
`ExecutePartitions` runs the query, then opens a multi-stream [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage) session on the destination table of the query job. It returns one partition for each stream and reports the exact row count of the result. `ReadPartition` reads one partition. A partition descriptor holds only the name of a stream and a version byte. It holds no credential, so any connection can read it with its own credentials.
BigQuery can return fewer streams than the driver asks for. `ExecutePartitions` does not support dry runs or bound parameters. See `bigquery.query.partitions.max_streams` and `bigquery.query.partitions.target_rows_per_stream` in [](#query-options) below.
### Partitioned Results
`ExecutePartitions` runs the query, then opens a multi-stream [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage) session on the destination table. It returns one partition for each stream. A partition does not contain credentials, so any connection can read it with its own credentials.
BigQuery can return fewer streams than the driver asks for. Also, dry runs and bound parameters are not supported. Also see `bigquery.query.partitions.max_streams` and `bigquery.query.partitions.target_rows_per_stream` in [](#query-options) below.

Comment thread go/docs/bigquery.md Outdated
`bigquery.query.partitions.max_streams`
: **Type:** integer. **Default:** 4

The largest number of Storage Read API streams that `ExecutePartitions` asks for. The value must be positive and fit in 32 bits. The driver also asks BigQuery for at least this many streams, but BigQuery can return fewer.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The largest number of Storage Read API streams that `ExecutePartitions` asks for. The value must be positive and fit in 32 bits. The driver also asks BigQuery for at least this many streams, but BigQuery can return fewer.
The number of Storage Read API streams that `ExecutePartitions` asks for.

Comment thread go/connection.go Outdated
Comment on lines +567 to +571
// ReadPartition turns a partition descriptor produced by (*statement).ExecutePartitions
// into a RecordReader, by decoding the Storage Read API stream name it carries and
// opening it with this connection's own credentials -- never the credentials of
// whichever connection created the partition. A malformed descriptor is reported
// as adbc.StatusInvalidArgument rather than panicking.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// ReadPartition turns a partition descriptor produced by (*statement).ExecutePartitions
// into a RecordReader, by decoding the Storage Read API stream name it carries and
// opening it with this connection's own credentials -- never the credentials of
// whichever connection created the partition. A malformed descriptor is reported
// as adbc.StatusInvalidArgument rather than panicking.

Comment thread go/driver_test.go
schema, partitions, rows, err := suite.stmt.ExecutePartitions(suite.ctx)
suite.Require().NoError(err)
suite.Zero(rows)
suite.Require().NotNil(schema)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate the schema itself?

Comment thread go/partitions.go Outdated
const defaultPartitionMaxStreams = 4

// maxPartitionStreams is OptionQueryPartitionsMaxStreams, or its default when unset.
func (st *statement) maxPartitionStreams() int32 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather keep this kind of thing inside statement.go

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@geoHeil

geoHeil commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ExecutePartitions/ReadPartition are unconditionally NotImplemented

2 participants