Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

## 0.46.1 - Unreleased

- Gmail: retry 403 rate limits with the existing bounded backoff and stop remaining detail fetches when a search fails; preserve error bodies and open the watch circuit on 403 rate limits. (#1241, #1242) — thanks @xyrolle.
- Gmail: leave quoted search operators to Gmail instead of silently narrowing results with label filters. (#1247) — thanks @SebTardif.
- Gmail: include nonempty Cc and Bcc headers in thread text output. (#1248) — thanks @alexpriest.
- Gmail: preserve the direction of reply and forward headers beside right-to-left text, and add Gmail's quote classes without altering plain text or quoted content. (#1250) — thanks @malob.
- Drive: preserve requested false booleans in list/search JSON, including grouped fields, slash paths, and wildcards; keep conditionally absent fields omitted. (#1246) — thanks @SebTardif.

- Auth: omit OAuth provider descriptions, URLs, and response bodies from command errors, per-item results, backup diagnostics, health checks, and watch/verbose logs; retain fixed recovery guidance, authentication exit codes, and both Calendar/Zoom rollback causes. (#1243) — thanks @pkyIntelligence.

- Security: upgrade Alpine's default CA packages in Docker image build and runtime stages to reject the compromised eMudhra certificate chains. (#1244) — thanks @joshavant.
Expand Down
6 changes: 4 additions & 2 deletions docs/automation.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,10 @@ least-privilege read against the intended service.

## Exit codes

Google API rate-limit retries honor `Retry-After` delays up to 60 seconds per
retry, including numeric seconds and HTTP dates. Cancelling the command also
Google API rate-limit retries cover HTTP 429 and HTTP 403 responses whose
reason is `rateLimitExceeded` or `userRateLimitExceeded`. They honor
`Retry-After` delays up to 60 seconds per retry, including numeric seconds and
HTTP dates. Cancelling the command also
cancels a pending retry wait. Upload bodies are closed even when a request is
rejected by the circuit breaker or cannot be buffered for retry, so those
failures release the associated file or stream resources.
Expand Down
19 changes: 13 additions & 6 deletions docs/gmail-workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ gog gmail messages search --max 2 --json -- '-in:inbox newer_than:7d'
```

Thread searches fail if any thread detail cannot be fetched, without emitting a
partial result list. Retry the search after resolving the reported API error.
partial result list, and stop the detail requests still pending. Retry the
search after resolving the reported API error.

Quoted operators such as `"is:unread"` are literal phrases and are passed to Gmail without adding a label filter. `gmail thread get` includes nonempty Cc and Bcc headers in its text output.

### Full payloads and raw MIME

Expand Down Expand Up @@ -435,11 +438,15 @@ gog gmail drafts forward <messageId> --note "FYI"
```

The forwarded-message block lists whichever of the original's From, Date,
Subject, To and Cc it has, in line with Gmail's forwards: the sender and each
recipient as `Name <email>`, or `<email>` without a name, with the addresses
linked in the HTML part. A group lists only its members, so a To of only
`undisclosed-recipients:;` is left empty, and a value that isn't a valid
address list is shown as it appears in the original.
Subject, To and Cc it has that aren't blank, in line with Gmail's forwards: the
sender and each recipient as `Name <email>`, or `<email>` without a name, with
the addresses linked in the HTML part. A group lists only its members, so a To
of only `undisclosed-recipients:;` is left empty, and a value that isn't a
valid address list is shown as it appears in the original. In the HTML part
this block and a reply's "On … wrote:" line are marked left-to-right, since gog
writes them in English, and each value taken from the original (a name, an
address, the subject, the date) keeps its own direction, so a Hebrew or Arabic
name doesn't reorder the line around it.

Remote HTTP images remain remote references. Only MIME parts referenced with
`cid:` are copied into the outgoing message.
Expand Down
43 changes: 43 additions & 0 deletions internal/cmd/drive_fields.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,49 @@ var driveFileConditionalBools = map[string]bool{
"WritersCanShare": true,
}

// driveListRequestFields returns the Files.list mask for this request.
// An empty override keeps the default list mask.
func driveListRequestFields(override string) string {
if strings.TrimSpace(override) != "" {
return override
}
return driveFileListFields
}

// driveFileListItemMask extracts file selectors from a Files.list mask,
// including grouped fields, slash paths, and whole-object wildcards.
func driveFileListItemMask(mask string) string {
var fields []string
add := func(part string) {
part = strings.TrimSpace(part)
name, group, grouped := strings.Cut(part, "(")
switch {
case part == "*" || part == "files":
fields = append(fields, "*")
case strings.HasPrefix(part, "files/"):
fields = append(fields, strings.TrimPrefix(part, "files/"))
case grouped && strings.TrimSpace(name) == "files" && strings.HasSuffix(group, ")"):
fields = append(fields, strings.TrimSuffix(group, ")"))
}
}
depth, start := 0, 0
for i, r := range mask {
switch r {
case '(':
depth++
case ')':
depth--
case ',':
if depth == 0 {
add(mask[start:i])
start = i + 1
}
}
}
add(mask[start:])
return strings.Join(fields, ",")
}

// forceRequestedDriveFileBools keeps top-level boolean fields named in mask in
// JSON output. drive.File tags them omitempty, so a requested `trashed: false`
// would otherwise disappear.
Expand Down
94 changes: 94 additions & 0 deletions internal/cmd/drive_fields_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -224,3 +224,97 @@ func TestDriveGet_JSONKeepsRequestedFalseBools(t *testing.T) {
})
}
}

// TestDriveLs_JSONKeepsRequestedFalseBools proves list JSON keeps requested
// false bools. The list mask wraps file fields in files(...), so the get-path
// top-level scan does not see them.
func TestDriveLs_JSONKeepsRequestedFalseBools(t *testing.T) {
t.Parallel()

type testCase struct {
name string
args []string
body map[string]any
want map[string]bool
notWant []string
}
cases := make([]testCase, 0, 7)
cases = append(cases, testCase{
name: "default fields",
args: nil,
body: map[string]any{
"id": "f1",
"name": "photo.png",
"hasThumbnail": false,
"trashed": false,
},
want: map[string]bool{"hasThumbnail": false},
notWant: []string{"trashed"},
}, testCase{
name: "files mask",
args: []string{"--fields", "files(id,trashed)"},
body: map[string]any{
"id": "f1",
"trashed": false,
"hasThumbnail": false,
},
want: map[string]bool{"trashed": false},
notWant: []string{"hasThumbnail"},
})
for _, mask := range []string{"files/trashed", "files(id),files(trashed)", "files", "files/*", "*"} {
cases = append(cases, testCase{
name: mask,
args: []string{"--fields", mask},
body: map[string]any{"id": "f1", "trashed": false},
want: map[string]bool{"trashed": false},
notWant: []string{"ownedByMe", "shared", "writersCanShare", "hasAugmentedPermissions"},
})
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
srv := newDriveFieldsTestServer(t, func(r *http.Request) map[string]any {
path := strings.TrimPrefix(r.URL.Path, "/drive/v3")
if path != "/files" {
return nil
}
return map[string]any{"files": []map[string]any{tc.body}}
}, nil)
defer srv.Close()
svc := newGoogleTestServiceWithEndpoint(t, srv.Client(), srv.URL+"/", drive.NewService)

var stdout bytes.Buffer
ctx := newCmdRuntimeJSONOutputContext(t, &stdout, io.Discard)
ctx = withDriveTestService(ctx, svc)
flags := &RootFlags{Account: "a@b.com"}

if err := runKong(t, &DriveLsCmd{}, tc.args, ctx, flags); err != nil {
t.Fatalf("run: %v", err)
}
var envelope struct {
Files []map[string]any `json:"files"`
}
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
t.Fatalf("invalid JSON: %v\nraw: %s", err, stdout.String())
}
if len(envelope.Files) != 1 {
t.Fatalf("files = %s", stdout.String())
}
file := envelope.Files[0]
for key, want := range tc.want {
got, ok := file[key]
if !ok {
t.Fatalf("missing %q in output: %s", key, stdout.String())
}
if got != want {
t.Fatalf("%q = %v, want %v", key, got, want)
}
}
for _, key := range tc.notWant {
if _, ok := file[key]; ok {
t.Fatalf("unexpected %q in output: %s", key, stdout.String())
}
}
})
}
}
15 changes: 8 additions & 7 deletions internal/cmd/drive_listing.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ func (c *DriveLsCmd) Run(ctx context.Context, flags *RootFlags) error {
return err
}

return writeDriveFileList(ctx, resp, "No files")
return writeDriveFileList(ctx, resp, "No files", c.Fields)
}

func (c *DriveSearchCmd) Run(ctx context.Context, flags *RootFlags) error {
Expand Down Expand Up @@ -103,7 +103,7 @@ func (c *DriveSearchCmd) Run(ctx context.Context, flags *RootFlags) error {
return err
}

return writeDriveFileList(ctx, resp, "No results")
return writeDriveFileList(ctx, resp, "No results", "")
}

func listDriveFiles(ctx context.Context, svc *drive.Service, opts driveFileListOptions) (*drive.FileList, error) {
Expand All @@ -117,16 +117,17 @@ func listDriveFiles(ctx context.Context, svc *drive.Service, opts driveFileListO
PageToken(opts.page).
OrderBy(orderBy)
call = driveFilesListCallWithDriveSupport(call, opts.allDrives, opts.driveID)
mask := driveFileListFields
if strings.TrimSpace(opts.fields) != "" {
mask = opts.fields
}
mask := driveListRequestFields(opts.fields)
return call.Fields(gapi.Field(mask)).Context(ctx).Do()
}

func writeDriveFileList(ctx context.Context, resp *drive.FileList, emptyMessage string) error {
func writeDriveFileList(ctx context.Context, resp *drive.FileList, emptyMessage, fields string) error {
u := ui.FromContext(ctx)
if outfmt.IsJSON(ctx) {
itemMask := driveFileListItemMask(driveListRequestFields(fields))
for _, f := range resp.Files {
forceRequestedDriveFileBools(f, itemMask)
}
return outfmt.WriteJSON(ctx, stdoutWriter(ctx), map[string]any{
"files": resp.Files,
"count": len(resp.Files),
Expand Down
99 changes: 97 additions & 2 deletions internal/cmd/gmail_concurrent_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"io"
"net/http"
"net/http/httptest"
"path"
"strings"
"sync/atomic"
"testing"
Expand Down Expand Up @@ -303,8 +304,8 @@ func TestFetchThreadDetails_ConcurrentError(t *testing.T) {
if items != nil {
t.Fatalf("expected nil items on error, got %d", len(items))
}
if got := calls.Load(); got != 3 {
t.Errorf("expected 3 concurrent calls and no sequential retry, got %d", got)
if got := calls.Load(); got > 3 {
t.Errorf("expected at most 3 concurrent calls and no sequential retry, got %d", got)
}
}

Expand Down Expand Up @@ -341,3 +342,97 @@ func TestFetchThreadDetails_ContextCanceled(t *testing.T) {
t.Fatalf("error = %v, want context.Canceled", err)
}
}

// newFirstRequestFailsGmailService serves Gmail detail requests. The first one
// fails with 403 rateLimitExceeded; every later one stays in flight until the
// client cancels it, so no slot is freed by anything but the fan-out itself.
// The handlers are released at cleanup, which lets a fan-out that never
// cancels finish after its test has failed.
func newFirstRequestFailsGmailService(t *testing.T) (*gmail.Service, *atomic.Int32) {
t.Helper()

var calls atomic.Int32
release := make(chan struct{})
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if calls.Add(1) == 1 {
w.WriteHeader(http.StatusForbidden)
_, _ = io.WriteString(w, `{"error":{"code":403,"message":"User-rate limit exceeded.","errors":[{"domain":"usageLimits","reason":"rateLimitExceeded"}]}}`)
return
}
select {
case <-r.Context().Done():
return
case <-release:
}
id := path.Base(r.URL.Path)
_, _ = fmt.Fprintf(w, `{"id":%q,"threadId":%q,"messages":[]}`, id, id)
}))
t.Cleanup(func() {
close(release)
server.Close()
})

return newGoogleTestServiceWithEndpoint(t, server.Client(), server.URL+"/", gmail.NewService), &calls
}

// awaitFanOut fails the test when fetch is still running after the first
// failure has had ample time to cancel it; it never decides a passing run.
func awaitFanOut(t *testing.T, fetch func() error) error {
t.Helper()

done := make(chan error, 1)
go func() { done <- fetch() }()

select {
case err := <-done:
return err
case <-time.After(10 * time.Second):
t.Fatal("the fan-out kept waiting on detail requests after its first failure")
return nil
}
}

func TestFetchThreadDetails_StopsAfterFirstError(t *testing.T) {
svc, calls := newFirstRequestFailsGmailService(t)
threads := make([]*gmail.Thread, 30)
for i := range threads {
threads[i] = &gmail.Thread{Id: fmt.Sprintf("thread-%02d", i)}
}

err := awaitFanOut(t, func() error {
_, err := fetchThreadDetails(context.Background(), svc, threads, nil, false, time.UTC)
return err
})
var apiErr *googleapi.Error
if !errors.As(err, &apiErr) || apiErr.Code != http.StatusForbidden {
t.Fatalf("error = %v, want the thread detail HTTP 403", err)
}
// Only the failed request frees a slot before cancellation: at most the
// 10 requests in flight plus one that took that slot ever reach the server.
if got := calls.Load(); got > 11 {
t.Fatalf("fetched %d of 30 threads after the search had failed, want at most 11", got)
}
}

func TestFetchMessageDetails_StopsAfterFirstError(t *testing.T) {
svc, calls := newFirstRequestFailsGmailService(t)
messages := make([]*gmail.Message, 30)
for i := range messages {
messages[i] = &gmail.Message{Id: fmt.Sprintf("message-%02d", i)}
}

err := awaitFanOut(t, func() error {
_, err := fetchMessageDetails(context.Background(), svc, messages, map[string]string{}, time.UTC, false, gmailMessageBodyFormatText, false, false, false, false)
return err
})
var apiErr *googleapi.Error
if !errors.As(err, &apiErr) || apiErr.Code != http.StatusForbidden {
t.Fatalf("error = %v, want the message detail HTTP 403", err)
}
// Only the failed request frees a slot before cancellation: at most the
// 10 requests in flight plus one that took that slot ever reach the server.
if got := calls.Load(); got > 11 {
t.Fatalf("fetched %d of 30 messages after the search had failed, want at most 11", got)
}
}
Loading
Loading