Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ADR-60531: Share Rate-Limit State Across Multi-Target Logs Downloads

**Date**: 2026-09-13
**Status**: Draft
**Deciders**: gh-aw maintainers

---

### Context

The PR changes `gh aw logs` when it downloads logs for multiple workflow targets under a configured `--max-github-api-rate-limit`. The existing behavior let each target enforce the API ceiling independently, which meant one target could detect the limit while other queued or in-flight targets kept working until they hit the same condition themselves or waited for reset. The diff introduces shared orchestration state across multi-target downloads, preserves single-target wait-for-reset behavior, and adds continuation data for targets interrupted by a shared rate-limit stop. The architectural question is how rate-limit enforcement should behave when several targets share one overall download operation.

### Decision

We will enforce `--max-github-api-rate-limit` through shared rate-limit state across multi-target `gh aw logs` downloads, cancel remaining target work as soon as the shared ceiling is reached, and reuse that reached state without additional GitHub API checks. We decided that multi-target downloads should stop promptly rather than wait for reset target-by-target, while single-target downloads should keep the existing wait-for-reset behavior. We will also emit resumable continuation data for interrupted targets so users can continue after the API window resets.

### Alternatives Considered

#### Alternative 1: Keep per-target independent rate-limit checks

This preserves the prior behavior where each target checks the configured limit separately and may wait for reset on its own. It was considered because it is simpler and reuses the existing single-target logic directly. It was not chosen because the PR evidence shows that independent checks leave queued work blocked behind a limit already reached elsewhere in the same command and allow unnecessary extra API checks after the shared ceiling is known.

#### Alternative 2: Wait for reset once, then resume remaining multi-target work automatically

Another option would be to coordinate targets under shared state but pause the whole multi-target operation until the reset time, then continue automatically. It was considered because it preserves completeness for one command invocation. It was not chosen because the diff explicitly adds immediate cancellation for queued and in-flight work in multi-target mode, and the updated CLI guidance distinguishes single-target waiting from multi-target stopping.

#### Alternative 3: Fail the command without continuation data when the shared ceiling is reached

The command could stop all remaining work immediately and return only an error. It was considered because it minimizes new continuation logic. It was not chosen because the diff adds rate-limit-specific continuation messages and preserves cursor options for interrupted targets, indicating resumability is a required part of the new behavior.

### Consequences

#### Positive
- Multi-target downloads stop promptly once the shared API ceiling is reached, avoiding wasted work and redundant rate-limit checks.
- Queued targets retain continuation parameters, making partial multi-target results resumable after the rate limit resets.
- Single-target downloads keep their existing wait-for-reset semantics, limiting behavioral change to the concurrent multi-target case.

#### Negative
- The logs download orchestration now depends on additional shared mutable state and cancellation coordination, increasing implementation complexity.
- Multi-target commands may now stop earlier than before, so users who expected automatic waiting across all targets must re-run with continuation data.
- Rate-limit handling now has distinct single-target and multi-target semantics, which raises the documentation and testing burden.

#### Neutral
- The change threads shared rate-limit state through orchestration, batch processing, and artifact download helpers without changing the public flag name.
- Tests now cover queue clearing, state reuse, and continuation behavior for rate-limit stops.
- CLI and MCP help text are updated to describe the different behaviors for one target versus multiple targets.

---

*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*
4 changes: 2 additions & 2 deletions pkg/cli/logs_command.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ const logsCommandExampleTemplate = ` # Basic usage
%[1]s logs weekly-research --repo owner/repo # Download logs from specific repository

# Resource budgets
%[1]s logs --timeout 30 --max-github-api-rate-limit 12000 # Wait after 12000 core API requests are used
%[1]s logs --timeout 30 --max-github-api-rate-limit 12000 # Pause one target or stop multiple targets after 12000 requests
%[1]s logs --timeout 30 --max-github-api-rate-limit -2000 # Keep 2000 core API requests available
%[1]s logs --timeout 30 --max-storage 10240 # Prune cache data and stop downloads at 10 GB
%[1]s logs --max-storage 10240 --prune-older-runs # Remove oldest runs if cache pruning is insufficient
Expand Down Expand Up @@ -629,7 +629,7 @@ func addLogsCommandFlags(logsCmd *cobra.Command, validArtifactSets string) {
logsCmd.Flags().Int("timeout", 0, "Total download timeout in minutes across all targets (0 = no timeout)")
logsCmd.Flags().Int("timeout-seconds", 0, "Download timeout in seconds (0 = use --timeout)")
_ = logsCmd.Flags().MarkHidden("timeout-seconds")
logsCmd.Flags().Int("max-github-api-rate-limit", 0, "Maximum used GitHub core API requests before waiting for reset (positive = absolute, negative = reserve from API limit; e.g. 12000 or -2000)")
logsCmd.Flags().Int("max-github-api-rate-limit", 0, "Maximum used GitHub core API requests before pausing one target or stopping multiple targets (positive = absolute, negative = reserve from API limit; e.g. 12000 or -2000)")
logsCmd.Flags().Int("max-storage", 0, "Maximum logs storage in MB after pruning non-essential cache data (0 = unlimited)")
logsCmd.Flags().Bool("prune-older-runs", false, "Remove oldest completed runs when non-essential cache pruning cannot satisfy --max-storage")
logsCmd.Flags().String("summary-file", "summary.json", "Path to write the summary JSON file relative to output directory (use empty string to disable)")
Expand Down
29 changes: 21 additions & 8 deletions pkg/cli/logs_multi.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,16 +109,21 @@ func queuedLogsTargetResult(opts LogsDownloadOptions, ctx context.Context) workf
if !timeoutReached {
return workflowLogsResult{}
}
continuation := buildContinuationIfNeeded(nil, timeoutReached, false, false, logsTargetContinuationOptions(opts))
continuation := buildContinuationIfNeeded(nil, timeoutReached, false, false, false, logsTargetContinuationOptions(opts))
return workflowLogsResult{timeoutReached: true, continuation: continuation}
}

func countLimitedLogsTargetResult(opts LogsDownloadOptions) workflowLogsResult {
fetchAllInRange := opts.StartDate != "" || opts.EndDate != ""
continuation := buildContinuationIfNeeded(nil, false, fetchAllInRange, false, logsTargetContinuationOptions(opts))
continuation := buildContinuationIfNeeded(nil, false, fetchAllInRange, false, false, logsTargetContinuationOptions(opts))
return workflowLogsResult{countLimitReached: fetchAllInRange, continuation: continuation}
}

func rateLimitedLogsTargetResult(opts LogsDownloadOptions) workflowLogsResult {
continuation := buildContinuationIfNeeded(nil, false, false, false, true, logsTargetContinuationOptions(opts))
return workflowLogsResult{continuation: continuation}
}

func logsTargetContinuationOptions(opts LogsDownloadOptions) continuationOptions {
return continuationOptions{
workflowName: opts.WorkflowName,
Expand Down Expand Up @@ -246,24 +251,24 @@ func collectLogsTargets(ctx context.Context, opts LogsDownloadOptions, targets [
var wg sync.WaitGroup
workerCount := min(len(targets), getMaxConcurrentWorkflowDownloads())
countLimit := newLogsCountLimit(opts.Count)
targetsCtx, cancelTargets := context.WithCancel(ctx)
defer cancelTargets()
rateLimitState := newLogsRateLimitState(cancelTargets)
// targetsCtx is canceled the instant the shared run-count budget is
// exhausted (see logsCountLimit.cancelRemainingTargets), so every target
// still running or queued observes it at its next cooperative
// cancellation checkpoint instead of only checking countLimit.isReached()
// between its own batches/iterations.
targetsCtx := ctx
if countLimit != nil {
var cancel context.CancelFunc
targetsCtx, cancel = context.WithCancel(ctx)
defer cancel()
countLimit.cancel = cancel
countLimit.cancel = cancelTargets
}
shared := logsTargetSharedState{
sem: make(chan struct{}, workerCount),
perTargetDownloads: max(1, getMaxConcurrentDownloads()/workerCount),
cleanupErrors: make(map[string]error, len(targets)),
storageLimit: newLogsStorageLimit(opts.OutputDir, opts.MaxStorageMB, opts.PruneOlderRuns),
countLimit: countLimit,
rateLimitState: rateLimitState,
}
for _, target := range targets {
cleanupOpts := opts
Expand Down Expand Up @@ -295,6 +300,7 @@ type logsTargetSharedState struct {
cleanupErrors map[string]error
storageLimit *logsStorageLimit
countLimit *logsCountLimit
rateLimitState *logsRateLimitState
}

// collectSingleLogsTarget runs one workflow target's log collection, recovering
Expand Down Expand Up @@ -322,6 +328,7 @@ func collectSingleLogsTarget(ctx context.Context, opts LogsDownloadOptions, targ
targetOpts.maxConcurrentDownloads = shared.perTargetDownloads
targetOpts.storageLimit = shared.storageLimit
targetOpts.countLimit = shared.countLimit
targetOpts.rateLimitState = shared.rateLimitState
// The shared deadline is already installed on ctx by the caller, so the
// target must not build a second timeout context of its own. TimeoutMinutes
// and TimeoutSeconds are deliberately preserved (rather than zeroed) so the
Expand All @@ -344,6 +351,9 @@ func collectSingleLogsTarget(ctx context.Context, opts LogsDownloadOptions, targ
logsOrchestratorLog.Printf("Skipping workflow target %s: shared maximum run count reached while queued", target.displayName())
return logsTargetResult{target: target, result: countLimitedLogsTargetResult(targetOpts)}
}
if shared.rateLimitState.isReached() {
return logsTargetResult{target: target, result: rateLimitedLogsTargetResult(targetOpts), err: errLogsAPIRateLimitReached}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collectSingleLogsTarget sets err: errLogsAPIRateLimitReached for every queued target once the ceiling is reached, unlike the analogous count-limit path a few lines above which returns err: nil (countLimitedLogsTargetResult(targetOpts) with no err). In mergeLogsTargetResults, every non-nil err is appended to allErrors, and DownloadWorkflowLogsForTargets has:

if len(processedRuns) == 0 {
    if len(allErrors) > 0 {
        return errors.Join(allErrors...)
    }
    _, err := handleEmptyProcessedRuns(...) // graceful path with continuations
    return err
}

So when the ceiling is hit before any target has produced a processed run (e.g. it's reached very early, or --count is small), every target — the one that detected the ceiling and all the queued ones — contributes an error, allErrors is non-empty, and the whole multi-target download now returns a hard error via errors.Join, instead of taking the graceful handleEmptyProcessedRuns branch that the count-limit case takes (which reports "no runs" plus the built continuations, exit 0). This turns an expected/handled condition (ceiling reached) into a CLI failure and discards the otherwise-correctly-built continuation data for the caller.

Suggested fix: don't set err on the logsTargetResult for the rate-limit-reached branches (mirror the count-limit branches), or explicitly filter errLogsAPIRateLimitReached out of allErrors before the len(allErrors) > 0 hard-fail check in DownloadWorkflowLogsForTargets.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in b5f569b and follow-up commits: rate-limit sentinel errors are now excluded from multi-target hard failures, preserving merged continuations.

}
// The target never started (e.g. it was still queued behind the
// semaphore when the shared deadline or cancellation fired). Build a
// continuation from its own options so a resumable date-range target
Expand All @@ -359,6 +369,9 @@ func collectSingleLogsTarget(ctx context.Context, opts LogsDownloadOptions, targ
logsOrchestratorLog.Printf("Skipping workflow target %s: shared maximum run count reached while queued", target.displayName())
return logsTargetResult{target: target, result: countLimitedLogsTargetResult(targetOpts)}
}
if shared.rateLimitState.isReached() {
return logsTargetResult{target: target, result: rateLimitedLogsTargetResult(targetOpts), err: errLogsAPIRateLimitReached}
}
return logsTargetResult{
target: target,
result: queuedLogsTargetResult(targetOpts, ctx),
Expand All @@ -385,7 +398,7 @@ func mergeLogsTargetResults(
storageLimitReached := false
var continuations []WorkflowContinuation
for _, targetResult := range results {
if targetResult.err != nil {
if targetResult.err != nil && !errors.Is(targetResult.err, errLogsAPIRateLimitReached) {
allErrors = append(allErrors, fmt.Errorf("%s: %w", targetResult.target.displayName(), targetResult.err))
}
processedRuns = append(processedRuns, targetResult.result.processedRuns...)
Expand Down
49 changes: 49 additions & 0 deletions pkg/cli/logs_multi_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,55 @@ func TestCollectLogsTargetsDoesNotStartQueuedTargetsAfterGlobalCountReached(t *t
assert.Equal(t, int64(1), calls.Load(), "queued targets must not start after the shared count is reached")
}

func TestCollectLogsTargetsClearsQueueWhenRateLimitReached(t *testing.T) {
t.Setenv("GH_AW_MAX_CONCURRENT_DOWNLOADS", "1")
originalCollector := collectWorkflowLogsForTarget
originalFetchRateLimit := fetchRateLimitFunc
t.Cleanup(func() {
collectWorkflowLogsForTarget = originalCollector
fetchRateLimitFunc = originalFetchRateLimit
})

fetchRateLimitFunc = func(context.Context) (rateLimitResource, error) {
return rateLimitResource{
Limit: 15000,
Remaining: 3000,
Reset: time.Now().Add(10 * time.Minute).Unix(),
Used: 12000,
}, nil
}
var calls atomic.Int64
collectWorkflowLogsForTarget = func(ctx context.Context, opts LogsDownloadOptions) (workflowLogsResult, error) {
calls.Add(1)
return workflowLogsResult{}, opts.rateLimitState.check(ctx, false, 12000, 1)
}

results := collectLogsTargets(context.Background(), LogsDownloadOptions{
Count: 10,
OutputDir: t.TempDir(),
MaxGitHubAPIRateLimit: 12000,
}, []logsWorkflowTarget{
{workflowName: "first"},
{workflowName: "second"},
{workflowName: "third"},
})

assert.Equal(t, int64(1), calls.Load(), "queued targets must not start after the shared API ceiling is reached")
require.Len(t, results, 3)
continuationCount := 0
for _, result := range results {
require.ErrorIs(t, result.err, errLogsAPIRateLimitReached)
if result.result.continuation != nil {
continuationCount++
assert.Contains(t, result.result.continuation.Message, "GitHub API rate limit ceiling reached")
}
}
_, continuations, _, _, _, errs := mergeLogsTargetResults(results, nil)
assert.Empty(t, errs, "rate-limit termination must not suppress continuations with a hard error")
assert.Equal(t, 2, continuationCount, "each queued target must retain a continuation")
assert.Len(t, continuations, 2, "each queued target must retain a continuation after merging")
}

func TestCountLimitedLogsTargetResultPreservesDateRangeContinuation(t *testing.T) {
result := countLimitedLogsTargetResult(LogsDownloadOptions{
WorkflowName: "queued",
Expand Down
15 changes: 12 additions & 3 deletions pkg/cli/logs_orchestrator.go
Original file line number Diff line number Diff line change
Expand Up @@ -239,10 +239,10 @@ func selectPaginationCursorDate(filteredRuns []WorkflowRun, oldestFetchedCreated
// returned in this batch.
func buildContinuationIfNeeded(
processedRuns []ProcessedRun,
timeoutReached, countLimitReached, storageLimitReached bool,
timeoutReached, countLimitReached, storageLimitReached, rateLimitReached bool,
opts continuationOptions,
) *ContinuationData {
if !timeoutReached && !countLimitReached && !storageLimitReached {
if !timeoutReached && !countLimitReached && !storageLimitReached && !rateLimitReached {
return nil
}
// A target can make zero progress (e.g. a timeout, shared count limit, or
Expand Down Expand Up @@ -277,6 +277,8 @@ func buildContinuationIfNeeded(
message = "Count limit reached. Use these parameters to continue fetching more logs from the same date range."
} else if storageLimitReached {
message = "Storage limit reached. Use these parameters to continue fetching more logs after freeing space or changing max_storage."
} else if rateLimitReached {
message = "GitHub API rate limit ceiling reached. Use these parameters to continue fetching more logs after the rate limit resets."
}
return &ContinuationData{
Message: message,
Expand Down Expand Up @@ -350,17 +352,24 @@ func collectWorkflowLogs(ctx context.Context, opts LogsDownloadOptions) (workflo

processedRuns, timeoutReached, countLimitReached, storageLimitReached, lastFetchedBeforeDate, err := collectProcessedWorkflowRuns(runtime, opts)
if err != nil {
var continuation *ContinuationData
if errors.Is(err, errLogsAPIRateLimitReached) {
continuationOpts := logsTargetContinuationOptions(opts)
continuationOpts.lastFetchedBeforeDate = lastFetchedBeforeDate
continuation = buildContinuationIfNeeded(processedRuns, false, false, false, true, continuationOpts)
}
return workflowLogsResult{
processedRuns: processedRuns,
artifactFilter: runtime.artifactFilter,
continuation: continuation,
countLimitReached: countLimitReached,
timeoutReached: timeoutReached,
storageLimitReached: storageLimitReached,
}, err
}
processedRuns = limitProcessedRuns(processedRuns, opts.Count, opts.Verbose)
logsOrchestratorLog.Printf("Collected %d processed runs (timeoutReached=%v, countLimitReached=%v)", len(processedRuns), timeoutReached, countLimitReached)
continuation := buildContinuationIfNeeded(processedRuns, timeoutReached, countLimitReached, storageLimitReached, continuationOptions{
continuation := buildContinuationIfNeeded(processedRuns, timeoutReached, countLimitReached, storageLimitReached, false, continuationOptions{
workflowName: opts.WorkflowName,
startDate: opts.StartDate,
endDate: opts.EndDate,
Expand Down
Loading
Loading