You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Investigate why dev-lead.yml is at 28.9% failure rate, document the failure
modes in standards/ci-standards.md so future Fleet Monitor flags route to
the right repo/secret, and confirm the stub itself is compliant. No change to
the stub.
Root-cause analysis (from recent failed runs)
The stub in this repo is a Tier-1 caller that delegates entirely to petry-projects/.github-private/.github/workflows/dev-lead-reusable.yml@main.
It already matches standards/workflows/dev-lead.yml byte-for-byte and carries
the full permission set (incl. statuses: read per #435), so the stub is not
the source of the failures. Sampled failed runs show three distinct signatures:
Signature
Where it surfaces
Where itβs fixed
Process completed with exit code 124 (12+ min dispatch step)
_ApiError ... code:429 ... Resource has been exhausted then RetryableQuotaError
Gemini engine quota depleted; cascade fallback runs out of options
Google AI Studio billing β refill GOOGLE_API_KEY's project
Error: Classic Personal Access Tokens (ghp_) are not supported by Copilot
Copilot engine fallback rejects Classic PAT
Org secret GH_PAT must be a fine-grained PAT, not ghp_β¦
The 17 cancellations / 38 runs are expected: the reusableβs centralised cancel-in-progress: false lanes still emit cancelled runs whenever a higher
lane preempts a queued event for the same PR/issue.
Implementation Checklist
.github/workflows/dev-lead.yml β verify it remains byte-for-byte
identical to standards/workflows/dev-lead.yml (no edits needed)
standards/ci-standards.md β append a "Dev-Lead failure-mode runbook"
subsection under Β§5 Dev-Lead Agent mapping the three signatures above to
their root-cause repo / secret, so the next Fleet Monitor flag is triaged in
seconds instead of hours
Tests to Write
None. This is a pure documentation change with no observable behavioural
change to any workflow, script, or runtime artifact β the
Phase-2 exception in the dev-lead prompt applies. Verification is the standard
repo lint suite (.dev-lead/scripts/dev-lead-lint.sh).
Verified .github/workflows/dev-lead.yml already matches standards/workflows/dev-lead.yml (carries the full permission set incl. statuses: read, no concurrency: block, pinned to petry-projects/.github-private/.github/workflows/dev-lead-reusable.yml@main). No stub edit needed β check_dev_lead_stub in scripts/compliance-audit.sh is green.
Classified the 11 failing runs in the monitored window: all three error signatures originate outside this repo (reusable timeout, Google AI Studio quota, Classic PAT rejected by Copilot).
Added a "Failure-mode runbook" subsection to standards/ci-standards.md Β§ Dev-Lead Agent mapping each signature to the upstream owner (private reusable / AI Studio billing / org secret), plus a note that the high Cancelled Runs count is expected per the centralised cancel-in-progress: false lanes.
Updated the GH_PAT row in the secrets table to call out the fine-grained-PAT requirement (Classic ghp_β¦ PATs are rejected at runtime by the Copilot engine path).
Test Results
Phase 2 (TDD) was skipped β this change is pure documentation in standards/ci-standards.md with no observable behaviour change to any
workflow, script, or runtime artifact, which is the Phase-2 exception clause.
Verification via the required dev-lead-lint.sh and the repo's markdown
linter:
standards/ci-standards.md: Added a "Failure-mode runbook" subsection under Β§Dev-Lead Agent mapping the three observed error signatures (exit code 124, Gemini _ApiError 429, Classic PAT β¦ not supported by Copilot) plus startup_failure to their root-cause repo or secret. Strengthened the GH_PAT row in the secrets table to call out the fine-grained-PAT requirement.
Notes
This issue surfaced infrastructure / engine-quota / secret-format failures
that the Tier-1 stub in this repo cannot fix by design. Concrete follow-ups
that can fix the failure rate live elsewhere β the new runbook routes them
explicitly:
Refill GOOGLE_API_KEY Google AI Studio prepayment credits β until done, every Gemini-step retry in the engine-fallback cascade will 429.
Rotate GH_PAT to a fine-grained PAT β required for the Copilot engine path; the current Classic ghp_β¦ is rejected before any work runs.
Investigate dispatch-step timeouts in petry-projects/.github-private/dev-lead-reusable.yml β exit-code-124 runs were 10β12 min long; if the engine is expected to take that long for review fix-ups, raise the step-level timeout-minutes; otherwise add a per-engine call timeout.
Two of the three are org-secret rotations and cannot be done from inside a
PR; they are the right humans-only escalation path that the runbook now makes
discoverable from this issue.
Activity
don-petry commentedon Jun 9, 2026
Dev-Lead Implementation Plan
Issue: #330 β [Fleet Monitor] petry-projects/.github β dev-lead.yml
Scope
Investigate why
dev-lead.ymlis at 28.9% failure rate, document the failuremodes in
standards/ci-standards.mdso future Fleet Monitor flags route tothe right repo/secret, and confirm the stub itself is compliant. No change to
the stub.
Root-cause analysis (from recent failed runs)
The stub in this repo is a Tier-1 caller that delegates entirely to
petry-projects/.github-private/.github/workflows/dev-lead-reusable.yml@main.It already matches
standards/workflows/dev-lead.ymlbyte-for-byte and carriesthe full permission set (incl.
statuses: readper #435), so the stub is notthe source of the failures. Sampled failed runs show three distinct signatures:
Process completed with exit code 124(12+ min dispatch step)petry-projects/.github-privatereusable / engine timeout config_ApiError ... code:429 ... Resource has been exhaustedthenRetryableQuotaErrorGOOGLE_API_KEY's projectError: Classic Personal Access Tokens (ghp_) are not supported by CopilotGH_PATmust be a fine-grained PAT, notghp_β¦The 17 cancellations / 38 runs are expected: the reusableβs centralised
cancel-in-progress: falselanes still emit cancelled runs whenever a higherlane preempts a queued event for the same PR/issue.
Implementation Checklist
.github/workflows/dev-lead.ymlβ verify it remains byte-for-byteidentical to
standards/workflows/dev-lead.yml(no edits needed)standards/ci-standards.mdβ append a "Dev-Lead failure-mode runbook"subsection under Β§5 Dev-Lead Agent mapping the three signatures above to
their root-cause repo / secret, so the next Fleet Monitor flag is triaged in
seconds instead of hours
Tests to Write
None. This is a pure documentation change with no observable behavioural
change to any workflow, script, or runtime artifact β the
Phase-2 exception in the dev-lead prompt applies. Verification is the standard
repo lint suite (
.dev-lead/scripts/dev-lead-lint.sh).Test Command
bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jun 9, 2026
Dev-Lead: Implementation Complete
Plan Execution
.github/workflows/dev-lead.ymlalready matchesstandards/workflows/dev-lead.yml(carries the full permission set incl.statuses: read, noconcurrency:block, pinned topetry-projects/.github-private/.github/workflows/dev-lead-reusable.yml@main). No stub edit needed βcheck_dev_lead_stubinscripts/compliance-audit.shis green.standards/ci-standards.mdΒ§ Dev-Lead Agent mapping each signature to the upstream owner (private reusable / AI Studio billing / org secret), plus a note that the highCancelled Runscount is expected per the centralisedcancel-in-progress: falselanes.GH_PATrow in the secrets table to call out the fine-grained-PAT requirement (Classicghp_β¦PATs are rejected at runtime by the Copilot engine path).Test Results
Phase 2 (TDD) was skipped β this change is pure documentation in
standards/ci-standards.mdwith no observable behaviour change to anyworkflow, script, or runtime artifact, which is the Phase-2 exception clause.
Verification via the required
dev-lead-lint.shand the repo's markdownlinter:
Files Changed
standards/ci-standards.md: Added a "Failure-mode runbook" subsection under Β§Dev-Lead Agent mapping the three observed error signatures (exit code 124, Gemini_ApiError 429,Classic PAT β¦ not supported by Copilot) plusstartup_failureto their root-cause repo or secret. Strengthened theGH_PATrow in the secrets table to call out the fine-grained-PAT requirement.Notes
This issue surfaced infrastructure / engine-quota / secret-format failures
that the Tier-1 stub in this repo cannot fix by design. Concrete follow-ups
that can fix the failure rate live elsewhere β the new runbook routes them
explicitly:
GOOGLE_API_KEYGoogle AI Studio prepayment credits β until done, every Gemini-step retry in the engine-fallback cascade will 429.GH_PATto a fine-grained PAT β required for the Copilot engine path; the current Classicghp_β¦is rejected before any work runs.petry-projects/.github-private/dev-lead-reusable.ymlβ exit-code-124 runs were 10β12 min long; if the engine is expected to take that long for review fix-ups, raise the step-leveltimeout-minutes; otherwise add a per-engine call timeout.Two of the three are org-secret rotations and cannot be done from inside a
PR; they are the right humans-only escalation path that the runbook now makes
discoverable from this issue.
don-petry commentedon Jun 9, 2026
feat: implement issue #330 β [Fleet Monitor] petry-projects/.github ββ¦
feat: implement issue #330 β [Fleet Monitor] petry-projects/.github ββ¦