Skip to content

feat(bigquery): resume interrupted Arrow ReadRows streams - #9602

Merged
ohmayr merged 4 commits into
mainfrom
feat/bigquery-arrow-stream-resume
Oct 21, 2026
Merged

feat(bigquery): resume interrupted Arrow ReadRows streams#9602
ohmayr merged 4 commits into
mainfrom
feat/bigquery-arrow-stream-resume

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Long ReadRows streams can drop on transient gRPC errors, which failed the whole read even though the Storage Read API can resume from any row. This reopens the stream at the next unread row.

  • Retries UNAVAILABLE, and INTERNAL errors caused by RST_STREAM, up to 3 times in a row. Any received batch resets the count. DEADLINE_EXCEEDED isn't retried, since it means the caller's deadline ran out.
  • An abandoned stream loses only our own listeners and keeps a no-op error listener, since google-gax emits CANCELLED after cancel().
  • Errors that aren't retryable, or persist after 3 retries, destroy the stream after the batches already yielded.

@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 13, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 13, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces automatic stream resumption for BigQuery's Arrow stream reader (createQueryArrowStream) by retrying transient gRPC errors (such as UNAVAILABLE, DEADLINE_EXCEEDED, and INTERNAL with RST_STREAM) from the last successfully read offset up to a maximum of three times. Comprehensive unit tests are also added to verify this behavior. The review feedback highlights a critical issue regarding the use of removeAllListeners() on the gRPC stream during cancellation and retries, which can inadvertently strip internal library listeners and cause resource leaks. It is recommended to avoid removeAllListeners() and instead explicitly detach only the registered listeners (data, error, end) using named functions.

Comment thread handwritten/bigquery/src/arrow.ts Outdated
Comment thread handwritten/bigquery/src/arrow.ts Outdated
@ohmayr
ohmayr added this pull request to stack #9597 October 13, 2026 07:08
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from e8fd2fb to 72ed966 Compare October 13, 2026 09:33
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from d4bf688 to da4e16d Compare October 13, 2026 10:42
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from a1856e7 to 663534b Compare October 13, 2026 12:00
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from da77b2d to c8290d3 Compare October 15, 2026 09:55
quirogas
quirogas reviewed Oct 15, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Outdated
Comment thread handwritten/bigquery/src/arrow.ts Outdated
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from 7614972 to 9197508 Compare October 15, 2026 18:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 3 times, most recently from 8fe5111 to 35977c3 Compare October 16, 2026 02:19
@ohmayr
ohmayr marked this pull request as ready for review October 16, 2026 02:34
@ohmayr
ohmayr requested review from a team as code owners October 16, 2026 02:34
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 35977c3 to 9df7133 Compare October 16, 2026 21:36
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 9df7133 to 35977c3 Compare October 16, 2026 21:50
@ohmayr

ohmayr commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on the #9598–#9601 changes. Same behavior and retry rules. The retry is now a short async loop in streamReadRows around a single ReadRows attempt (readRowsOnce), so each failed attempt goes through the same listener cleanup as a normal finish, and the next attempt resumes at the next unread row. The tests check that every abandoned stream keeps none of our listeners.

@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 35977c3 to ec5ff11 Compare October 20, 2026 21:48
Base automatically changed from feat/bigquery-arrow-destination-table to main October 21, 2026 19:05
ohmayr added 3 commits October 21, 2026 12:05
@ohmayr
feat(bigquery): resume interrupted Arrow ReadRows streams
3c7e39d 
Resume Storage Read API ReadRows streams at offset + rowsReceived when
interrupted by transient gRPC errors (DEADLINE_EXCEEDED, UNAVAILABLE,
or INTERNAL with RST_STREAM), up to MAX_STREAM_RETRIES consecutive
attempts, waiting 100 ms before the first reopen and doubling the wait
for each consecutive retry. Surface non-retryable errors or exhausted
retries on the stream even after earlier batches have been emitted.

Abandoned gRPC streams (on reconnect or early cancel) keep a no-op
'error' listener, because google-gax emits CANCELLED asynchronously
after cancel() and an unhandled 'error' event would crash the process.
@ohmayr
fix(bigquery): stop retrying DEADLINE_EXCEEDED; stub Arrow retry waits
b703117 
DEADLINE_EXCEEDED on ReadRows means the caller's deadline ran out, so
pass it to the caller instead of reopening the stream, matching the
design doc and the Python reader. Route the retry backoff through
retryBackoff.wait so unit tests replace it instead of sleeping on real
timers, and check the 100/200/400 ms sequence. Mark readRowsOnce async.
@ohmayr
fix(bigquery): reconnect ReadRows immediately without client-side delay
ef7ca58 
Reopen ReadRows at the next unread row offset without an added
client-side timer on mid-stream UNAVAILABLE and RST_STREAM drops,
matching ReadStream in @google-cloud/bigquery-storage and Python's
ReadRowsStream. Initial connection failures when opening or reopening
a ReadRows stream are already retried with exponential backoff by
google-gax (retry-request).
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 6742228 to ef7ca58 Compare October 21, 2026 19:06
quirogas
quirogas approved these changes Oct 21, 2026
View reviewed changes

@quirogas quirogas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread handwritten/bigquery/src/arrow.ts
Comment on lines +448 to +458
function isRetryableStreamError(err: Error & {code?: number}): boolean {
// DEADLINE_EXCEEDED isn't retried: it means the caller's deadline ran out.
if (err.code === GRPC_STATUS_UNAVAILABLE) {
return true;
}
// Long-lived HTTP/2 streams closed by the server with RST_STREAM surface as
// INTERNAL (code 13) in gRPC and are safe to reconnect at the next offset.
return (
err.code === GRPC_STATUS_INTERNAL && err.message.includes('RST_STREAM')
);
}

@quirogas quirogas Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit : Consider guarding err.message with optional chaining or a case-insensitive regex (typeof err.message === 'string' && /rst_stream/i.test(err.message) or Boolean(err.message?.includes('RST_STREAM'))) in case an error object with code: 13 has a missing or lowercase message.

@ohmayr ohmayr Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 90f8ea1.

shivanee-p
shivanee-p approved these changes Oct 21, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Outdated

@shivanee-p shivanee-p Oct 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency, I'd suggest changing this if statement to if (!emittedBatch && !stream.destroyed) to match streamDefaultJobStream

@ohmayr ohmayr Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 90f8ea1.

@ohmayr
fix(bigquery): guard err.message and destroyed destination streams
90f8ea1 
Use optional chaining on err.message in isRetryableStreamError and skip
decoding the 0-row schema batch in streamDestinationTable when the
stream is already destroyed.
@ohmayr
ohmayr merged commit 8372c9e into main Oct 21, 2026
45 of 46 checks passed
@ohmayr
ohmayr deleted the feat/bigquery-arrow-stream-resume branch October 21, 2026 23:04
@release-please release-please Bot mentioned this pull request Oct 21, 2026
Merged
shivanee-p pushed a commit that referenced this pull request Oct 23, 2026
@release-please
chore: release main (#9684)
b3a9687 
🤖 I have created a release *beep* *boop*
---


<details><summary>common: 8.2.2</summary>

##
[8.2.2](common-v8.2.1...common-v8.2.2)
(2026-10-23)


### Bug Fixes

* **common:** Preserve {{projectId}} token during URI path encoding
([#9668](#9668))
([14b0cc6](14b0cc6))
</details>

<details><summary>gapic-generator: 5.7.2</summary>

##
[5.7.2](gapic-generator-v5.7.1...gapic-generator-v5.7.2)
(2026-10-23)


### Bug Fixes

* **generator:** Compile protos before running tests in generator-tests
([#9701](#9701))
([cde8ae3](cde8ae3))
* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>gaxios: 8.3.2</summary>

##
[8.3.2](gaxios-v8.3.1...gaxios-v8.3.2)
(2026-10-23)


### Bug Fixes

* **gaxios:** Narrow fetch types to call signature for @types/bun
([#9699](#9699))
([2a34528](2a34528))
</details>

<details><summary>bigquery: 9.5.0</summary>

##
[9.5.0](bigquery-v9.4.0...bigquery-v9.5.0)
(2026-10-23)


### Features

* **bigquery:** Allow WRITE_APPEND in createQueryArrowStream
([#9648](#9648))
([2062403](2062403))
* **bigquery:** Resume interrupted Arrow ReadRows streams
([#9602](#9602))
([8372c9e](8372c9e))
* **bigquery:** Stream multi-page Arrow results from default job stream
([#9598](#9598))
([1aa6eb4](1aa6eb4))
* **bigquery:** Support arrowSerializationOptions in
createQueryArrowStream
([#9603](#9603))
([2449fe4](2449fe4))
* **bigquery:** Support explicit destination tables in
createQueryArrowStream
([#9601](#9601))
([f22e2dd](f22e2dd))
* **bigquery:** Support jobs.insert in createQueryArrowStream
([#9600](#9600))
([4c414c6](4c414c6))
* **bigquery:** Support maxResults in createQueryArrowStream
([#9605](#9605))
([a7fdd05](a7fdd05))
* **bigquery:** Wait for unfinished jobs before streaming Arrow results
([#9599](#9599))
([5e465b7](5e465b7))


### Bug Fixes

* **bigquery:** Preserve timestamp nanoseconds when microseconds are
zero
([#9650](#9650))
([f6ca61d](f6ca61d))
* **bigquery:** Raise when an Arrow batch arrives before its schema
([#9681](#9681))
([576a42e](576a42e))
</details>

<details><summary>datastore: 11.3.1</summary>

##
[11.3.1](datastore-v11.3.0...datastore-v11.3.1)
(2026-10-23)


### Bug Fixes

* **datastore:** Align Transaction JSDoc
[@extends](https://github.com/extends) tag with DatastoreRequest
([#9671](#9671))
([733e9f1](733e9f1))
</details>

<details><summary>datamanager: 1.4.0</summary>

##
[1.4.0](datamanager-v1.3.0...datamanager-v1.4.0)
(2026-10-23)


### Features

* Update API sources and regenerate
([#9472](#9472))
([514e2f6](514e2f6))


### Bug Fixes

* Update package.json and .gitignore across packages
([#9378](#9378))
([9c54cec](9c54cec))
</details>

<details><summary>agentidentity: 0.3.1</summary>

##
[0.3.1](agentidentity-v0.3.0...agentidentity-v0.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>agentregistry: 0.6.1</summary>

##
[0.6.1](agentregistry-v0.6.0...agentregistry-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>apptopology: 0.4.1</summary>

##
[0.4.1](apptopology-v0.4.0...apptopology-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>bigquery-storage-api: 0.6.1</summary>

##
[0.6.1](bigquery-storage-api-v0.6.0...bigquery-storage-api-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>bigtable-api: 0.11.1</summary>

##
[0.11.1](bigtable-api-v0.11.0...bigtable-api-v0.11.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>commerceproducer: 0.7.1</summary>

##
[0.7.1](commerceproducer-v0.7.0...commerceproducer-v0.7.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>datastore-api: 0.5.1</summary>

##
[0.5.1](datastore-api-v0.5.0...datastore-api-v0.5.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>firestore-api: 0.7.1</summary>

##
[0.7.1](firestore-api-v0.7.0...firestore-api-v0.7.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>ftp: 0.4.1</summary>

##
[0.4.1](ftp-v0.4.0...ftp-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>logging-api: 0.5.1</summary>

##
[0.5.1](logging-api-v0.5.0...logging-api-v0.5.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>policy-troubleshooter-iam: 0.11.1</summary>

##
[0.11.1](policy-troubleshooter-iam-v0.11.0...policy-troubleshooter-iam-v0.11.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>productregistry: 2.2.1</summary>

##
[2.2.1](productregistry-v2.2.0...productregistry-v2.2.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>spanner-api: 0.6.1</summary>

##
[0.6.1](spanner-api-v0.6.0...spanner-api-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>speech: 8.2.1</summary>

##
[8.2.1](speech-v8.2.0...speech-v8.2.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>tasks: 7.3.1</summary>

##
[7.3.1](tasks-v7.3.0...tasks-v7.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>workloadidentity: 0.4.1</summary>

##
[0.4.1](workloadidentity-v0.4.0...workloadidentity-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>isochrones: 0.3.1</summary>

##
[0.3.1](isochrones-v0.3.0...isochrones-v0.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>inventories: 0.15.1</summary>

##
[0.15.1](inventories-v0.15.0...inventories-v0.15.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>loyaltycustomers: 0.4.1</summary>

##
[0.4.1](loyaltycustomers-v0.4.0...loyaltycustomers-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>reports: 0.18.1</summary>

##
[0.18.1](reports-v0.18.0...reports-v0.18.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>reviews: 0.9.0</summary>

##
[0.9.0](reviews-v0.8.0...reviews-v0.9.0)
(2026-10-23)


### Features

* **google/shopping/merchant/reviews/v1:** Onboard a new library
([#9691](#9691))
([f369d0b](f369d0b))
</details>

<details><summary>storage-control: 0.13.1</summary>

##
[0.13.1](storage-control-v0.13.0...storage-control-v0.13.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@quirogas quirogas quirogas approved these changes
@shivanee-p shivanee-p shivanee-p approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ohmayr @quirogas @shivanee-p