Skip to content

[Fleet Monitor] combsmith/worldchest β€” .github/workflows/dependabot-rebase.ymlΒ #29

Closed

Description

@don-petry

🟠 DEGRADED: combsmith/worldchest / .github/workflows/dependabot-rebase.yml

Metric Value
Failure Rate 33.3% (1 / 3 runs)
Status DEGRADED
p50 Duration 13s
p95 Duration 55s
Successful Runs 2
Cancelled Runs 0

Workflow: .github/workflows/dependabot-rebase.yml

Last updated by Fleet Monitor run on 2026-11-10.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. added
    dev-leadRequires dev-lead attention
    fleet-trackerTracked by the Actions Fleet Monitor
    health-checkAutomated health check result
    on Nov 10, 2026
  2. don-petry commented on Nov 10, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: issue #29 implementation failed β€” will retry

    All engines were rate-limited. This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

    Last 40 lines of engine output (redacted)
    ::notice::[gemini] model gemini-3.8-flash: API key 2 of 4 throttled β€” trying the next key
    ::notice::[gemini] model gemini-3.8-flash: key index 3 cooling down for 59614s (daily quota wording β†’ next reset)
    ::notice::[gemini] model gemini-3.8-flash: API key 3 of 4 throttled β€” trying the next key
    ::notice::[gemini] model gemini-3.8-flash: key index 4 cooling down for 59612s (daily quota wording β†’ next reset)
    ::warning::[gemini] model gemini-3.8-flash throttled (rc=173) β€” trying next in chain
    Warning: 256-color support not detected. Using a terminal with at least 256-color support is recommended for a better visual experience.
    Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
    Ripgrep is not available. Falling back to GrepTool.
    Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
    Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
    Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-11-10T15-26-28-404Z.json TerminalQuotaError: You have exhausted your daily quota on this model.
        at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:308718:16)
        at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:309441:29)
        at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
        at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:334666:28)
        at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:334361:29)
        at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:335403:24)
        at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:349394:22)
        at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:349491:14)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-VF5FS3OM.js:13658:26
        at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-VF5FS3OM.js:19036:5) {
      cause: {
        code: 429,
        message: 'You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://aistudio.google.com/docs/rate-limits. To monitor your current usage, head to: https://ai.dev/rate-limit. \n' +
          '* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 20, model: gemini-3.8-flash, description: Generate content requests per day limit exceeded on Free Tier. See https://aistudio.google.com/docs/rate-limits.\n' +
          'Please retry in 16h33m31.608322775s.',
        details: [ [Object], [Object], [Object] ]
      },
      retryDelayMs: undefined,
      reason: undefined,
      status: 429
    }
    {
      "session_id": "1373e0a5-367e-4975-af43-cc85b2c32a04",
      "error": {
        "type": "Error",
        "message": "You have exhausted your daily quota on this model.",
        "code": 429
      }
    }
    
    
  3. don-petry commented on Nov 10, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: implementation deferred β€” automation PR queue is full

    No pull request was opened for issue #29. The org-wide
    open-automation-PR queue is currently at its signed-off cap (see
    standards/pr-limits.json in combsmith/.github).

    This issue stays labeled dev-lead and will be re-attempted automatically once
    the queue drains β€” no action is required.

  4. don-petry commented on Nov 11, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #29 β€” [Fleet Monitor] combsmith/worldchest β€” .github/workflows/dependabot-rebase.yml

    Scope

    Investigate the single failed run behind the 33.3% (1/3) failure rate. Result: no code change needed. This was a transient failure that has already cleared.

    Diagnosis

    • Failed run: 42671916326 (schedule, 2026-11-09T18:12Z, 55s). GitHub reports "This run likely failed because of a workflow file issue", but the run has zero jobs and no annotations, so the workflow never got as far as running any repo logic.
    • The caller stub didn't change: .github/workflows/dependabot-rebase.yml matches combsmith/.github/standards/workflows/dependabot-rebase.yml except for a missing trailing newline. Its last commit is 4d1c420, which is older than the failure.
    • The reusable didn't change: dependabot-rebase/v2-stable β†’ 7d91f9c (2026-11-04). There have been no commits to dependabot-rebase-reusable.yml since then, so the same stub and the same reusable ran green both before and after the failure.
    • It has cleared: the 4 runs since then all succeeded (42743280246, 42812337580, 42906818566, 42959410082), and so did 5 of the 6 runs before it. Overall that's 9 of 10 recent runs green. The 1/3 window was too small to separate a one-off from a real problem.

    Implementation Checklist

    • No source changes. The stub is a thin caller pinned to a central channel tag, and AGENTS.md / the stub header forbid editing its uses: ref, permissions, concurrency or secrets. If this comes back, the fix belongs in the reusable in combsmith/.github, not here.

    Tests to Write

    • None. This is a transient CI infrastructure failure with no behaviour to change (Phase 2 exception: no observable behaviour change).

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh (repo baseline; no test suite yet per AGENTS.md)

    Recommendation

    Close as transient once the Fleet Monitor window rolls past the 2026-11-09 run. Reopen only if a failure with actual jobs/logs shows up.

  5. don-petry commented on Nov 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #29 β€” [Fleet Monitor] combsmith/worldchest β€” .github/workflows/dependabot-rebase.yml

    Scope

    Investigate the single failed run. No source change is warranted: the failure was transient and outside this repo.

    Findings

    • The only failure in the window is run 42671916326 (2026-11-09 18:12 UTC, schedule, 55s). GitHub reports it as "likely failed because of a workflow file issue" and has no failed-step logs.
    • That run used head SHA 4d1c420. The successful runs before it (42561475812) and all 7 scheduled runs after it (2026-11-10 β†’ 2026-11-12, latest 43162058766) used the same SHA. The stub didn't change between failure and recovery.
    • The stub calls combsmith/.github/...dependabot-rebase-reusable.yml@dependabot-rebase/v2-stable, a moving channel tag (now 7d91f9c). A "workflow file issue" with an unchanged caller fits a transient problem resolving or validating that reusable at the time, or a GitHub-side blip. It doesn't point to a defect in this repo.
    • Per AGENTS.md, .github/workflows/*.yml thin-caller stubs are org-managed and must not be hand-edited, and the stub header forbids changing the uses: ref.

    Implementation Checklist

    • No file changes in this repo (stub is org-managed; failure not reproducible; workflow healthy since)

    Tests to Write

    • None (no behavior change; config-only / no-op investigation)

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh (repo has no test suite yet)

    Recommendation

    Close as transient once Fleet Monitor's rolling window drops the 2026-11-09 run. If it happens again, look into the dependabot-rebase/v2-stable channel history in combsmith/.github around the failure time.

  6. don-petry commented on Nov 13, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #29 β€” [Fleet Monitor] combsmith/worldchest β€” .github/workflows/dependabot-rebase.yml

    Scope

    Investigation only. No code change is warranted: the one failure was a short incident across the org on 2026-11-09, and it has already cleared. There is nothing wrong in this repo.

    Findings

    • One failure in the last 87 runs: run 42671916326, scheduled at 2026-11-09T18:12Z. GitHub reported "This run likely failed because of a workflow file issue." No jobs were created, so the run failed while GitHub was setting it up, before any of our steps ran.
    • The same failure hit other repos within a 14-minute window: broodly (18:26Z), markets (18:20Z), worldchest (18:12Z). None of these repos has any other failure in its recent history. That points to something upstream (GitHub, or the dependabot-rebase/v2-stable reusable), not to this repo.
    • Every run since then has succeeded (9 in a row, through 2026-11-13). The 33% figure comes from a 3-run monitoring window that happened to include the incident.
    • The caller stub matches combsmith/.github/standards/workflows/dependabot-rebase.yml. The only difference is a missing trailing newline. The stub header forbids local edits to the uses: channel pin, permissions:, secrets: and triggers.

    Implementation Checklist

    • Root-cause the failed run (failed during setup, same failure in other repos at the same time)
    • Confirm the caller stub still matches the org standard
    • No source change. This issue can close once Fleet Monitor's window rolls past 2026-11-09 and the workflow reports healthy.

    Tests to Write

    • None. This repo has no test suite (STACK=none), and the workflow behaviour is not changing.

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh

  7. don-petry commented on Nov 13, 2026

    @don-petry
    ContributorAuthor

    βœ… Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-11-10 (3 days) β€” the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadRequires dev-lead attentionfleet-trackerTracked by the Actions Fleet Monitorhealth-checkAutomated health check result

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions