Skip to content

fix(backup): reject repeated Workspace backup page tokens - #1082

Closed
SebTardif wants to merge 1 commit into
openclaw:mainfrom
SebTardif:fix/backup-workspace-page-token
Closed

SebTardif wants to merge 1 commit into
openclaw:mainfrom
SebTardif:fix/backup-workspace-page-token

Conversation

@SebTardif

Copy link
Copy Markdown
Contributor

What Problem This Solves

gog backup push with workspace lists Google Forms responses (Forms.Responses.List) and native Drive files by MIME (Files.List) by walking Google page tokens. Both helpers in internal/cmd/backup_workspace.go copied nextPageToken into the next request with no seen-set.

When Forms or Drive repeats a continuation token, those loops never terminate. Backup collection keeps requesting the same page and never finishes a workspace snapshot.

The same hang class is already closed for Chat and Classroom backup (#1063), Drive sync push listing (#1065), Drive audit permission listing (#1066), calendar and Gmail listing (#1004), and People/Gmail-from-contact/contacts-export listing (#1044, #1045, #1046). Sibling PRs cover Drive backup (#1078), Groups/Admin/Keep (#1079), Drive collaboration (#1080), and Calendar/Contacts/Tasks (#1081). Workspace backup listing was still on the unguarded loop.

Evidence

terminal output from the compiled internal/cmd listing binary after the patch. A stuck continuation token is rejected after two list calls, with no third request:

$ gogcli-backup-workspace.exe -test.run TestFetchBackupFormResponsesRejectsRepeatedPageToken -test.v
    backup_workspace_test.go:55: err = pagination loop: repeated page token "stuck" after 2 list calls

$ gogcli-backup-workspace.exe -test.run TestFetchDriveFilesByMimeRejectsRepeatedPageToken -test.v
    backup_workspace_test.go:103: err = pagination loop: repeated page token "stuck" after 2 list calls

Before the patch, the same stuck token made a third list request and returned HTTP 400 from the safety cap (unexpected extra ... page request after 3 list calls) instead of stopping on the repeated token.

Real behavior proof

  • Behavior or issue addressed: Workspace backup form-response listing and Drive-by-mime listing hung when Google repeated a page token. gog backup push with workspace never finished those listings.
  • Real environment tested: Windows, Go 1.27.1, branch fix/backup-workspace-page-token at current HEAD, compiled internal/cmd listing binary.
  • Exact steps or command run after this patch: Compiled the listing binary with go, then ran gogcli-backup-workspace.exe with -test.v on the form-response and Drive-by-mime hang-guard cases.
  • Evidence after fix: terminal output from that binary shows pagination loop: repeated page token "stuck" after 2 list calls for both fetchBackupFormResponses and fetchDriveFilesByMime.
  • Observed result after fix: A repeated Workspace backup page token now fails closed with the shared paginator error after two requests. Distinct pages still concatenate. Drive-by-mime files stay sorted by id. File listing still uses driveFilesListCallWithDriveSupport (corpora=allDrives).
  • What was not tested: A live Google Workspace account with a real repeated token. Snapshot write / --best-effort handling after the listing error.

Summary

Route fetchBackupFormResponses and fetchDriveFilesByMime through existing collectAllPages. Keep the original field lists, page sizes, MIME query, order, shared-drive flags, and file sort.

Related: #1063, #1065, #1066, #1004, #1044, #1045, #1046, #1078, #1079, #1080, #1081.

Introduced in 068ff0e5 (2026-04-27, feat(backup): expand workspace backup coverage).

Route form-response and Drive-by-mime backup listing through
collectAllPages so a repeated Google page token fails closed
instead of hanging gog backup push with workspace.

Signed-off-by: Sebastien Tardif <SebTardif@ncf.ca>
@clawsweeper

clawsweeper Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 5, 2026
@clawsweeper

clawsweeper Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge. Reviewed September 5, 2026, 3:40 AM ET / 07:40 UTC.

ClawSweeper review

What this changes

Reuse the shared pagination guard for Forms responses and Drive file listings used by Workspace and Apps Script backups, with four regression tests.

Merge readiness

⛔ Blocked before merge - 4 items remain

The two backup collectors remain unguarded on main and in v0.39.0; the linked pagination fixes cover different paths, so this PR remains useful.

Priority: P2
Reviewed head: 1a742b05e9b4710915d6b281de096a67eee9d8a0
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused implementation with credible HTTP fault proof; the remaining concerns are the inherited page-limit contract and release documentation.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (terminal): Captured Windows terminal output exercises both changed production collectors through generated Google HTTP clients against injected repeated-token responses, showing rejection after exactly two requests. This proves the listing guard; snapshot updates and best-effort handling were explicitly outside the supplied run.
Patch quality 🐚 platinum hermit (4/6) 1 actionable review finding remain.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): Captured Windows terminal output exercises both changed production collectors through generated Google HTTP clients against injected repeated-token responses, showing rejection after exactly two requests. This proves the listing guard; snapshot updates and best-effort handling were explicitly outside the supplied run.
Evidence reviewed 11 items Defect remains on main: Both collectors copy each nextPageToken into another request without tracking previously visited tokens; neither has a listing-wide page bound.
Latest release remains affected: The v0.39.0 source contains the same unguarded Forms-response and Drive-by-MIME loops.
Verified introduced change: The pinned base-to-head diff changes only the two collectors and adds four tests. Request contexts, queries, fields, page sizes, shared-drive flags, sorting, and nil-on-provider-error behavior are preserved.
Findings 1 actionable finding [P3] Add the required Unreleased backup entry
Security None None.

How this fits together

The backup adapters collect Google API pages and turn their results into encrypted snapshots. Listing errors flow through existing per-form or optional-service error handling before the backup repository is updated.

flowchart TD
  A[Backup request] --> B[Workspace and Apps Script adapters]
  C[Google API pages] --> D[Shared pagination guard]
  B --> D
  D --> E{Listing complete?}
  E -->|Yes| F[Collected backup rows]
  E -->|Error| G[Existing backup error handling]
  F --> H[Encrypted snapshot]
  G --> H
Loading

Decision needed

Question Recommendation
Should Workspace and Apps Script adopt the shared 10,000-page ceiling, or retain unrestricted pagination when every continuation token is distinct? Adopt the documented shared ceiling: Follow the merged Chat/Classroom precedent, documenting and validating the ceiling's effects on fresh and existing backups before landing.

Why: The repeated-token repair is clear, but the additional ceiling changes successful large-backup behavior and can affect the latest stored snapshot; the submitted proof does not establish acceptance of that tradeoff.

Before merge

  • Add the required Unreleased backup entry (P3) - This changes user-visible Workspace and Apps Script backup behavior, but the diff contains no changelog entry. Root AGENTS.md requires user-visible fixes in CHANGELOG.md under Unreleased, with a PR reference and contributor thanks. Add an entry under 0.39.1 - Unreleased, including the accepted pagination bound.
  • Resolve merge risk (P1) - The inherited 10,000-page ceiling can turn a previously completing listing into an error. Existing snapshot handling can then replace latest Workspace or Apps Script data with encrypted error rows; Forms errors remain embedded even with --no-best-effort. The supplied runs do not cover updates to an existing backup repository.
  • Complete next step (P2) - Resolve the page-limit compatibility choice, document and validate its effects on fresh and existing backups, and add the required Unreleased entry.
  • Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

  • [P3] Add the required Unreleased backup entry — internal/cmd/backup_workspace.go:262
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test LOC Production +10/-20 (net -10); tests +195/-0 The branch reduces production code by reusing the existing guard and adds four focused HTTP regression cases.

Merge-risk options

Maintainer options:

  1. Document and verify the accepted ceiling (recommended)
    After owner acceptance, cover the page-limit error on fresh and populated backups, including service-level best-effort handling and embedded Forms errors.
  2. Keep successful distinct-token listings unrestricted
    Separate cycle protection from the page ceiling for these collectors without changing other paginator callers.

Technical review

Best possible solution:

Retain shared loop protection under an explicitly accepted page-limit policy, with documented and verified error handling for existing Workspace and Apps Script backups.

Do we have a high-confidence way to reproduce the issue?

Yes, source establishes the failure path: a provider repeatedly returning the same nonempty token keeps both current-main loops requesting pages. The contributor supplies bounded before/after HTTP-fault evidence; this read-only review did not execute tests.

Is this the best way to solve the issue?

Unclear overall: reusing the shared collector is the narrow repair, but its additional page ceiling needs an explicit compatibility contract for existing backups.

Full review comments:

  • [P3] Add the required Unreleased backup entry — internal/cmd/backup_workspace.go:262
    This changes user-visible Workspace and Apps Script backup behavior, but the diff contains no changelog entry. Root AGENTS.md requires user-visible fixes in CHANGELOG.md under Unreleased, with a PR reference and contributor thanks. Add an entry under 0.39.1 - Unreleased, including the accepted pagination bound.
    Confidence: 0.99

Overall correctness: patch is correct
Overall confidence: 0.92

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 25703c789da5.

Labels

Label changes:

  • add P2: This repairs a bounded backup reliability defect triggered by repeated provider pagination tokens.
  • add merge-risk: 🚨 compatibility: Using the shared collector additionally caps formerly unrestricted Workspace and Apps Script listings at 10,000 pages, potentially changing existing backup results.
  • add proof: sufficient: Contributor real behavior proof is sufficient. Captured Windows terminal output exercises both changed production collectors through generated Google HTTP clients against injected repeated-token responses, showing rejection after exactly two requests. This proves the listing guard; snapshot updates and best-effort handling were explicitly outside the supplied run.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Captured Windows terminal output exercises both changed production collectors through generated Google HTTP clients against injected repeated-token responses, showing rejection after exactly two requests. This proves the listing guard; snapshot updates and best-effort handling were explicitly outside the supplied run.

Label justifications:

  • P2: This repairs a bounded backup reliability defect triggered by repeated provider pagination tokens.
  • merge-risk: 🚨 compatibility: Using the shared collector additionally caps formerly unrestricted Workspace and Apps Script listings at 10,000 pages, potentially changing existing backup results.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Captured Windows terminal output exercises both changed production collectors through generated Google HTTP clients against injected repeated-token responses, showing rejection after exactly two requests. This proves the listing guard; snapshot updates and best-effort handling were explicitly outside the supplied run.
  • proof: sufficient: Contributor real behavior proof is sufficient. Captured Windows terminal output exercises both changed production collectors through generated Google HTTP clients against injected repeated-token responses, showing rejection after exactly two requests. This proves the listing guard; snapshot updates and best-effort handling were explicitly outside the supplied run.

Evidence

What I checked:

  • Defect remains on main: Both collectors copy each nextPageToken into another request without tracking previously visited tokens; neither has a listing-wide page bound. (internal/cmd/backup_workspace.go:261, 25703c789da5)
  • Latest release remains affected: The v0.39.0 source contains the same unguarded Forms-response and Drive-by-MIME loops. (internal/cmd/backup_workspace.go:261, b058dbbaeb44)
  • Verified introduced change: The pinned base-to-head diff changes only the two collectors and adds four tests. Request contexts, queries, fields, page sizes, shared-drive flags, sorting, and nil-on-provider-error behavior are preserved. (internal/cmd/backup_workspace.go:262, 1a742b05e9b4)
  • Production HTTP fault proof: The complete captured PR body, in context sourceRevision c87793a646cebdf2ce9a9b36168e6348baa5b6bf806657cb43f18d067a955bb1, reports Windows runs returning the repeated-token error after two requests for each collector. The inspected fixtures call the production collectors through generated Forms and Drive clients using real HTTP connections to synthetic endpoints. Assertions verify token handoff, two requests, and no partial results. Snapshot updates were explicitly not exercised. (internal/cmd/backup_workspace_test.go:15, 1a742b05e9b4)
  • Additional page ceiling: The reused collector rejects listings needing more than 10,000 pages, including listings whose tokens never repeat. Its existing tests cover this bound and discarded partial results. (internal/cmd/paging.go:24, 1a742b05e9b4)
  • Actual error-handling boundary: Drive-by-MIME listing also feeds Apps Script. Native Workspace and Apps Script listing errors propagate to optional-service handling, whereas Forms inventory and response errors become embedded error rows even with --no-best-effort. The PR does not change these existing distinctions. (internal/cmd/backup_workspace.go:72, 1a742b05e9b4)

Likely related people:

  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • salmonumbrella: Raw commit 3371e3f adds internal/cmd/paging.go:24 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 3371e3f3ad6d; files: internal/cmd/paging.go)
  • SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Confirm and document the page-limit policy for Workspace and Apps Script, including the distinct Forms error behavior.
  • Validate the accepted error policy on fresh and populated backup repositories under both best-effort modes.
  • Add the required Unreleased changelog entry with contributor credit.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete

steipete commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Superseded by #1087, which incorporates Workspace and Forms pagination protection without introducing new page limits. Stock-CLI runs verified fatal inventory failures and embedded Forms error records. Thanks @SebTardif. Superseded by #1087, which incorporates contact-dedupe cycle protection while preserving early completion at --max. Repeated tokens stop before a partial plan or contact changes. Thanks @SebTardif. Superseded by #1087, which incorporates recurring-instance cycle protection while preserving immediate success when the requested occurrence is found. Failed lookup performs no event deletion. Thanks @SebTardif.

@steipete steipete closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants