Skip to content

[Fleet Monitor] combsmith/broodly β€” .github/workflows/dependabot-rebase.ymlΒ #647

Closed

Description

@don-petry

🟠 DEGRADED: combsmith/broodly / .github/workflows/dependabot-rebase.yml

Metric Value
Failure Rate 33.3% (1 / 3 runs)
Status DEGRADED
p50 Duration 17s
p95 Duration 50s
Successful Runs 2
Cancelled Runs 0

Workflow: .github/workflows/dependabot-rebase.yml

Last updated by Fleet Monitor run on 2026-11-10.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. don-petry commented on Nov 10, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: issue #647 implementation failed β€” will retry

    All engines were rate-limited. This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

    Last 40 lines of engine output (redacted)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:267703:7
        at async Models.generateContentStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:268799:16)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:313130:19
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:289147:23
        at async retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:309418:23)
        at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:334666:28)
        at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:334361:29)
        at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:335403:24) {
      status: 503
    }
    Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-11-10T15-24-47-206Z.json TerminalQuotaError: You have exhausted your daily quota on this model.
        at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:308718:16)
        at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:309441:29)
        at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
        at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:334666:28)
        at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:334361:29)
        at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:335403:24)
        at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:349394:22)
        at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NGQU3QTQ.js:349491:14)
        at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-VF5FS3OM.js:13658:26
        at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-VF5FS3OM.js:19036:5) {
      cause: {
        code: 429,
        message: 'You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://aistudio.google.com/docs/rate-limits. To monitor your current usage, head to: https://ai.dev/rate-limit. \n' +
          '* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 20, model: gemini-3.8-flash, description: Generate content requests per day limit exceeded on Free Tier. See https://aistudio.google.com/docs/rate-limits.\n' +
          'Please retry in 16h35m12.825836415s.',
        details: [ [Object], [Object], [Object] ]
      },
      retryDelayMs: undefined,
      reason: undefined,
      status: 429
    }
    {
      "session_id": "38db3e02-7f99-4bcf-92ed-d5176e805671",
      "error": {
        "type": "Error",
        "message": "You have exhausted your daily quota on this model.",
        "code": 429
      }
    }
    
    
  2. don-petry commented on Nov 10, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: implementation deferred β€” automation PR queue is full

    No pull request was opened for issue #647. The org-wide
    open-automation-PR queue is currently at its signed-off cap (see
    standards/pr-limits.json in combsmith/.github).

    This issue stays labeled dev-lead and will be re-attempted automatically once
    the queue drains β€” no action is required.

  3. don-petry commented on Nov 11, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #647 β€” [Fleet Monitor] combsmith/broodly β€” .github/workflows/dependabot-rebase.yml

    Scope

    Find out why the one run failed. Result: it was a one-off startup failure. No code change is warranted in this repo.

    Findings

    • The failed run, 42674043780 (schedule, 2026-11-09 18:26 UTC), ended at startup with 0 jobs. GitHub's summary says "This run likely failed because of a workflow file issue."
    • The same caller-stub commit (f7150c3) ran successfully before and after the failure: 42565058923 before it, and 42743763145, 42814095643 and 42907376695 after it.
    • The dependabot-rebase/v2-stable channel tag points at combsmith/.github@7d91f9c (committed 2026-11-04). That is earlier than both the failure and the later successful runs, so the reusable did not change between them.
    • .github/workflows/dependabot-rebase.yml has not changed since f7150c3 (2026-11-05).
    • The most likely cause is a one-off GitHub Actions startup problem, such as failing to resolve the cross-repo reusable. It does not come from anything in this repo.

    Implementation Checklist

    • Investigate the failed run (no logs, because no job started)
    • No file changes. The stub is a managed thin caller (its uses: channel pin, permissions, triggers and concurrency must not be edited), and nothing here caused the failure.

    Tests to Write

    • None (no behaviour change).

    Test Command

    N/A

    Recommendation

    Close as transient. If startup failures come back, raise the problem against the reusable in combsmith/.github, not here. The 33.3% rate is based on only 3 runs in the monitored window.

  4. don-petry commented on Nov 11, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #647 β€” [Fleet Monitor] combsmith/broodly β€” .github/workflows/dependabot-rebase.yml

    Scope

    Look into the one failed run out of 3. Conclusion: no code change needed. It was a one-off GitHub-side startup failure, and runs since then have succeeded.

    Findings

    • The failed run was 42674043780 (schedule, 2026-11-09 18:26 UTC). It shows 0 jobs and the banner "This run likely failed because of a workflow file issue". That means the workflow never started, so no step inside the reusable failed.
    • Neither the caller stub nor the reusable changed around that time:
      • Stub last changed in f7150c3 (2026-11-05).
      • The dependabot-rebase/v2-stable tag still points at 7d91f9c (2026-11-04), which is the latest commit to dependabot-rebase-reusable.yml.
    • The same unchanged workflow passed in the run before the failure and in all 4 scheduled runs after it (2026-11-10 02:35, 12:14, 21:40 and 2026-11-11 05:05). Across the last 10 runs it failed once.
    • The stub is a thin caller pinned to a moving tag. Org rules say not to edit its uses: ref, triggers, concurrency, secrets or permissions, and nothing in the evidence points to a defect there.

    Implementation Checklist

    • Look into the failed run (startup failure, 0 jobs)
    • Check the stub and the reusable/tag history for changes (none)
    • Confirm later runs passed (4/4)
    • No file changes

    Tests to Write

    None. Nothing changes in behaviour or config (TDD exception: investigation only).

    Recommendation

    Close as transient. If the failure rate stays above the threshold over a longer window, report it against the reusable in combsmith/.github, not this stub.

  5. don-petry commented on Nov 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #647 β€” [Fleet Monitor] combsmith/broodly β€” .github/workflows/dependabot-rebase.yml

    Scope

    Investigation found a single transient startup failure. No code change is warranted.

    Findings

    • The only failure in the window is run 42674043780 (2026-11-09T18:26Z, schedule). GitHub reports "This run likely failed because of a workflow file issue". No jobs were created, so there are no logs and no annotations. That makes it a startup failure that happened before the run was queued.
    • The failed run and every run since then used the same caller SHA (f7150c3) and the same resolved reusable SHA (dependabot-rebase/v2-stable β†’ 7d91f9cb).
    • Since then there have been 7 consecutive schedule runs. All succeeded, the latest being 43166103340 at 2026-11-12T06:09Z. The 33% rate comes from a 3-run sample.
    • Identical inputs failed once and then passed repeatedly, so the cause was transient and on the GitHub side (reusable-workflow resolution or fetch). It was not a defect in this repo's stub.

    Implementation Checklist

    • None. .github/workflows/dependabot-rebase.yml is a thin caller stub pinned to a central channel. Its header forbids editing the uses: ref, the triggers, the concurrency group, the secrets or the permissions, and none of them are implicated.

    Tests to Write

    • None. There is no behavioural change. This is a config-only investigation, so the TDD exception applies.

    Recommendation

    Close once Fleet Monitor's next window shows the failure rate back under 10%. If startup failures come back, the investigation belongs in the central reusable (combsmith/.github), not in this stub.

  6. don-petry commented on Nov 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #647 β€” [Fleet Monitor] combsmith/broodly β€” .github/workflows/dependabot-rebase.yml

    Investigation

    • The only failing run, 42674043780 (2026-11-09, schedule), ended with "This run likely failed because of a workflow file issue". No jobs started.
    • It ran on head f7150c3. Runs on that same commit passed before it (2026-11-09 05:03) and after it (every run since). Across the last 40 runs: 39 success, 1 failure.
    • The uses: ref dependabot-rebase-reusable.yml@dependabot-rebase/v2-stable resolves today (tag β†’ 7d91f9c). The failure was transient: it came from resolving the central reusable or channel tag, not from anything in this repo.
    • The local stub matches combsmith/.github/standards/workflows/dependabot-rebase.yml except for comments: the header says dependabot-rebase/stable instead of v2-stable, and the permission comments are outdated.

    Scope

    No change to how the workflow behaves. Sync the stub's comments with the central source of truth so it matches the canonical file exactly. The uses: ref, triggers, concurrency, permissions and secrets stay the same, as the stub's guardrails require.

    Implementation Checklist

    • .github/workflows/dependabot-rebase.yml: copy the comment lines from the canonical standard (header channel name and the two permission comments)

    Tests to Write

    • None. This only changes comments, so the TDD exception applies. Verified with a diff against the canonical standard and a YAML parse.

    Test Command

    diff <(gh api repos/combsmith/.github/contents/standards/workflows/dependabot-rebase.yml -H 'Accept: application/vnd.github.raw') .github/workflows/dependabot-rebase.yml

  7. don-petry commented on Nov 12, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete β€” PR #651

    The implementation for issue #647 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks β€” not asserted here. If a later stage fails, this claim is retracted in place.

  8. don-petry commented on Nov 13, 2026

    @don-petry
    ContributorAuthor

    βœ… Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-11-10 (3 days) β€” the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions