Skip to content

fix(raw): preserve canonical Google JSON across readers - #1158

Merged
steipete merged 2 commits into
mainfrom
codex/fix-lossless-raw-readers-20260924
Sep 24, 2026
Merged

steipete merged 2 commits into
mainfrom
codex/fix-lossless-raw-readers-20260924

Conversation

@steipete

Copy link
Copy Markdown
Collaborator

Raw readers promised Google's canonical JSON but decoded responses into SDK structs before writing them. That discarded unknown fields and explicit zero/false/empty values. For example, Slides bold: false became an unset inherited style, Drive's zero-byte size: "0" disappeared, and Gmail's empty part IDs and zero body sizes were lost.

Read the final resource response through authenticated HTTP and preserve fields as json.RawMessage. All raw readers share the same GET/error/JSON-validation helper, including Sheets. Preserve existing scopes, selectors, formatting, untrusted wrapping, and redirect refusal. Calendar, Tasks, and email-to-contact resolution retain their typed lookup clients; People/Contacts raw keeps the existing contact-only scope.

Drive still redacts the same capability fields by default and honors explicit field masks. Docs keeps its default first-tab response and all-tabs tree; selected-tab projection preserves raw values and unknown fields while retaining root document identity and clearing unrelated first-tab content.

Validation:

  • Regression coverage across all ten raw commands for explicit false/zero/null/empty values, unknown nested fields, integers beyond float64 precision, compact/pretty/wrapped output, and ignored ordinary selection transforms.
  • Existing selector, Gmail format, Docs tab, and Drive redaction tests retained; added raw projection, nested redaction, cancellation, HTTP error, and contact-scope coverage.
  • Live synthetic before/after/direct-Google comparisons passed for Docs, Slides, Drive, Calendar, Gmail, Forms, People/Contacts, and Tasks. The original data loss was reproduced for Docs, Slides, Drive, Calendar, Gmail full-format messages, and Tasks. Other live responses matched without requiring Google to emit optional false fields; deterministic fixtures cover those cases.
  • Live Docs checks include nested tab title/ID selection and all-tabs output; Gmail checks include all four formats; Contacts includes email resolution. Drive comparisons exclude only its asynchronously changing version counter. All disposable objects were cleaned up and verified, including the unsent draft; no mail was sent.
  • A separate live Sheets compact/pretty comparison verified the shared reader; that fixture was also trashed and verified.
  • Full make ci passed, including formatting, lint, deadcode, Go tests, Docker version check, generated docs, and skill checks. Independent P2 Auto Review completed with no actionable findings.

@clawsweeper

clawsweeper Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 24, 2026
@clawsweeper

clawsweeper Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 24, 2026, 8:04 AM ET / 12:04 UTC.

ClawSweeper review

What this changes

The branch moves eight Google service raw readers to authenticated HTTP responses, reuses the helper for Sheets, and updates regression tests, documentation, and the changelog.

Merge readiness

✅ Ready for maintainer review

Current main fixes lossless raw output for Sheets, but the documented JSON-preservation behavior remains unmet in other readers. This PR is a coherent candidate to complete that work.

Priority: P2
Reviewed head: dababa7c91b3a3d7b13b96548ff1ea2d96c4509d

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The patch has focused command-path coverage and detailed reported live validation, with no blocking source-review finding.
Proof confidence 🐚 platinum hermit (4/6) Not applicable: The collaborator-authored PR is exempt from the contributor proof gate. Its body reports live before/after comparisons for the changed raw-reader paths, but includes no inspectable transcript; no stored-data contract changes.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The collaborator-authored PR is exempt from the contributor proof gate. Its body reports live before/after comparisons for the changed raw-reader paths, but includes no inspectable transcript; no stored-data contract changes.
Evidence reviewed 8 items Current contract: The raw API documentation promises Google's response shape for scripts and debugging.
Remaining main-branch gap: Current main reads Docs through a typed SDK response before writing JSON; the same pattern remains in other affected readers.
Existing Sheets implementation: Current main already reads Sheets JSON without decoding it into an SDK resource.
Findings None None.
Security None None.

How this fits together

The gog raw commands fetch Google Workspace resources for scripts and agents, then emit JSON through the CLI output layer. Some commands resolve a name or email before fetching the final resource.

flowchart LR
A[Account and resource ID] --> B[Optional selector lookup]
B --> C[Authenticated Google GET]
C --> D[Validate JSON]
D --> E[Tab projection or redaction]
E --> F[Raw JSON output]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Raw commands covered 10 command paths The shared response handling affects every documented raw reader.
Code and test delta production +291/−95, tests +318/−11 lines The production growth supports shared HTTP wiring and tab projection, with more test lines than production lines added.

Technical review

Best possible solution:

Keep one authenticated JSON-preserving read path while retaining each command's selectors, Drive redaction, error handling, and output contract.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main sends typed SDK resources to the raw JSON writer, so SDK-omitted explicit values cannot survive. This read-only review did not execute a live reproduction.

Is this the best way to solve the issue?

Yes. Retaining the authenticated response as JSON values addresses the shared loss mechanism without special-casing individual Google fields.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9413a6c24382.

Labels

Label changes:

  • add P2: The PR repairs a documented raw JSON contract with limited impact outside raw-reader workflows.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The collaborator-authored PR is exempt from the contributor proof gate. Its body reports live before/after comparisons for the changed raw-reader paths, but includes no inspectable transcript; no stored-data contract changes.

Label justifications:

  • P2: The PR repairs a documented raw JSON contract with limited impact outside raw-reader workflows.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The collaborator-authored PR is exempt from the contributor proof gate. Its body reports live before/after comparisons for the changed raw-reader paths, but includes no inspectable transcript; no stored-data contract changes.

Evidence

What I checked:

Likely related people:

  • Peter Steinberger: Raw commit 7381ec9 adds internal/cmd/sheets_raw.go:20 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 7381ec9d1ccb; files: internal/cmd/sheets_raw.go)
  • SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete
steipete merged commit c24b7dc into main Sep 24, 2026
15 checks passed
@steipete

Copy link
Copy Markdown
Collaborator Author

Landed as c24b7dc.

All raw readers now preserve Google's explicit values and unknown fields through the shared HTTP/JSON reader. Drive's existing redaction and field-mask behavior remain intact; Docs tab projection preserves raw content and root identity. The Contacts scope remains contact-only, and the other service scopes are unchanged.

Verified full local make ci, independent P2 review, and final-head hosted CI run 35996248808 (Linux, Windows, macOS build, minimum Go, worker), Docker run 35996248831, and CodeQL. ClawSweeper found no actionable findings. The final merged tree exactly matches the computed combination of reviewed head dababa7 and current main, including both newly landed input-validation fixes.

Live proof covered all ten raw command groups. Baseline loss was reproduced for Docs/Slides false style values, Drive zero size/false flags, Calendar zero sequence/false reminders, Gmail FULL empty part fields, and Tasks empty links. Forms, People/Contacts, other Gmail formats, and the shared Sheets reader passed direct-response preservation checks. Docs nested-tab title/ID projection and Contacts email resolution also passed. Drive comparisons exclude only its asynchronous version counter; deterministic fixtures cover precision and unknown/explicit fields.

All synthetic objects from both fixture attempts were cleaned up with readback: Drive-backed objects trashed, Calendar event cancelled, and the exact draft/contact/task list absent. No mail was sent and no scopes were added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant