Repository navigation
Conversation
ExecutePartitions and ReadPartition returned NotImplemented. Now they work. ExecutePartitions runs the query as ExecuteQuery does. Then it opens a multi-stream Storage Read API session on the destination table of the query job. It returns one partition for each stream. It also returns the Arrow schema of the session and the exact row count of the result. A partition descriptor holds a version byte and the name of a stream. It holds no credential. ReadPartition decodes the descriptor and reads the stream with the credentials of its own connection. A malformed descriptor gives InvalidArgument. A session stream accepts no per-request Arrow serialization options. BigQuery refuses them with InvalidArgument. So ReadPartition sends only the stream name, and the session carries the ZSTD compression. The session also asks for a minimum stream count that equals the maximum. Without it, BigQuery can return one stream for a small table. BigQuery can still return fewer streams than requested. Two new statement options set the stream count: bigquery.query.partitions.max_streams (default 4) and bigquery.query.partitions.target_rows_per_stream (default: unset). ExecutePartitions refuses a dry run and bound parameters.
The partitions patch lacked two fixes that the upstream change (adbc-drivers/bigquery#371) carries, so sutura did not run what it proposed: - ExecutePartitions ran a statement with bound parameters and ignored them, so a caller got rows for a query it never parameterised. It now refuses them as NotImplemented before running the query. - readPartitionRecordReader opened the ReadRows gRPC stream on the caller's context and never cancelled it when decoding the stream's schema failed, leaving the stream open behind an error that returns no reader to release. It now cancels its context on any error return. The patch is regenerated from the pinned source (go/v1.14.1) with the three earlier patches applied: the old patch came back byte-for-byte before the new one was cut, and the new one applies to that tree and reproduces the edited tree. Cells (Go, run by nix/bigquery-adbc.nix's check phase and listed there; not run locally, CI only): - TestExecutePartitionsRefusesBoundParameters: red when the params refusal is neutralised (`st.params != nil && false`, the field still read), as the call then goes on past the refusal and panics on the test statement's missing connection instead of returning NotImplemented. - TestPartitionReadClosesStreamWhenDecodingFails: a fake Storage Read API sends an undecodable schema and holds the stream open; red when the deferred cancel is dropped, as the server never sees the stream end. VENDOR.md names adbc-drivers/bigquery#371 and the removal condition: the first driver release that contains it. Limit: the cells are written against the module cache but were not run here; the CI check phase is the first run. Part of #828 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
lidavidm
left a comment
There was a problem hiding this comment.
Warning
Automated review generated by GPT-6.1 Sol.
Adds partitioned query execution through Storage Read API sessions.
- [P0] Validate the current implementation against BigQuery before submission. The description explicitly says this version has not run against BigQuery.
- [P0] Shorten the description: remove the test inventory and execution report, retain the relevant design decisions, and include the required
Assisted-by:orGenerated-by:trailer identifying any AI assistance. - [P1] Do not add mocked tests for this feature. Instead, add actual integration tests against BigQuery and run those tests yourself before submission.
| return nil, errToAdbcErr(adbc.StatusIO, err, "open read partition stream") | ||
| } | ||
|
|
||
| rdr, schema, err := ipcReaderFromArrowIterator(arrowIterator, nil, "", alloc) |
There was a problem hiding this comment.
[P2] Apply the same schema normalization when reading partitions. ExecutePartitions normalizes BigQuery extension metadata, but this path retains the original IPC schema. For JSON, I reproduced arrow.json in the advertised schema versus google:sqlType:json in the reader schema. GEOGRAPHY and nested extension fields are affected too. Normalize the reader and batch schemas consistently so consumers receive the advertised types.
Keep session, reader, batch, and nested column schemas consistent, and return no partitions for empty results. Replace mocked Storage Read API tests with live cross-connection regressions and enable partition conformance coverage. Assisted-by: OpenAI Codex (GPT-6)
|
Hope that my last commit adresses your findings. |
| ### Partitioned Results | ||
|
|
||
| `ExecutePartitions` runs the query, then opens a multi-stream [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage) session on the destination table of the query job. It returns one partition for each stream and reports the exact row count of the result. `ReadPartition` reads one partition. A partition descriptor holds only the name of a stream and a version byte. It holds no credential, so any connection can read it with its own credentials. | ||
|
|
||
| BigQuery can return fewer streams than the driver asks for. `ExecutePartitions` does not support dry runs or bound parameters. See `bigquery.query.partitions.max_streams` and `bigquery.query.partitions.target_rows_per_stream` in [](#query-options) below. |
There was a problem hiding this comment.
| ### Partitioned Results | |
| `ExecutePartitions` runs the query, then opens a multi-stream [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage) session on the destination table of the query job. It returns one partition for each stream and reports the exact row count of the result. `ReadPartition` reads one partition. A partition descriptor holds only the name of a stream and a version byte. It holds no credential, so any connection can read it with its own credentials. | |
| BigQuery can return fewer streams than the driver asks for. `ExecutePartitions` does not support dry runs or bound parameters. See `bigquery.query.partitions.max_streams` and `bigquery.query.partitions.target_rows_per_stream` in [](#query-options) below. | |
| ### Partitioned Results | |
| `ExecutePartitions` runs the query, then opens a multi-stream [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage) session on the destination table. It returns one partition for each stream. A partition does not contain credentials, so any connection can read it with its own credentials. | |
| BigQuery can return fewer streams than the driver asks for. Also, dry runs and bound parameters are not supported. Also see `bigquery.query.partitions.max_streams` and `bigquery.query.partitions.target_rows_per_stream` in [](#query-options) below. |
| `bigquery.query.partitions.max_streams` | ||
| : **Type:** integer. **Default:** 4 | ||
|
|
||
| The largest number of Storage Read API streams that `ExecutePartitions` asks for. The value must be positive and fit in 32 bits. The driver also asks BigQuery for at least this many streams, but BigQuery can return fewer. |
There was a problem hiding this comment.
| The largest number of Storage Read API streams that `ExecutePartitions` asks for. The value must be positive and fit in 32 bits. The driver also asks BigQuery for at least this many streams, but BigQuery can return fewer. | |
| The number of Storage Read API streams that `ExecutePartitions` asks for. |
| // ReadPartition turns a partition descriptor produced by (*statement).ExecutePartitions | ||
| // into a RecordReader, by decoding the Storage Read API stream name it carries and | ||
| // opening it with this connection's own credentials -- never the credentials of | ||
| // whichever connection created the partition. A malformed descriptor is reported | ||
| // as adbc.StatusInvalidArgument rather than panicking. |
There was a problem hiding this comment.
| // ReadPartition turns a partition descriptor produced by (*statement).ExecutePartitions | |
| // into a RecordReader, by decoding the Storage Read API stream name it carries and | |
| // opening it with this connection's own credentials -- never the credentials of | |
| // whichever connection created the partition. A malformed descriptor is reported | |
| // as adbc.StatusInvalidArgument rather than panicking. |
| schema, partitions, rows, err := suite.stmt.ExecutePartitions(suite.ctx) | ||
| suite.Require().NoError(err) | ||
| suite.Zero(rows) | ||
| suite.Require().NotNil(schema) |
There was a problem hiding this comment.
validate the schema itself?
| const defaultPartitionMaxStreams = 4 | ||
|
|
||
| // maxPartitionStreams is OptionQueryPartitionsMaxStreams, or its default when unset. | ||
| func (st *statement) maxPartitionStreams() int32 { |
There was a problem hiding this comment.
I'd rather keep this kind of thing inside statement.go
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Addressed the review feedback. |
Implement
ExecutePartitionsandReadPartitionusing BigQuery Storage Read API sessions so callers can distribute query results across connections or processes. Partition descriptors contain only stream names; each reader uses its own connection's credentials.ExecutePartitionsreturns the session's Arrow schema and the exact row count from destination-table metadata, which requires one extra request. Session and reader schemas normalize JSON, GEOGRAPHY, and nested extension fields consistently. Empty results return no partitions.Two statement options control the requested stream count:
bigquery.query.partitions.max_streams(default 4) and optionalbigquery.query.partitions.target_rows_per_stream. BigQuery may return fewer streams than requested. Compression is set on the session because session streams reject per-request serialization options. Dry runs and bound parameters are refused.Assisted-by: OpenAI Codex (GPT-6)
Closes #367.