Skip to content

[Fleet Monitor] petry-projects/.github β€” dev-lead.ymlΒ #330

Closed

Description

@don-petry

🟑 WARNING: petry-projects/.github / dev-lead.yml

Metric Value
Failure Rate 15.5% (15 / 97 runs)
Status WARNING
p50 Duration 18s
p95 Duration 632s
Successful Runs 26
Cancelled Runs 48

Workflow: dev-lead.yml

Last updated by Fleet Monitor run on 2026-06-10.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Jun 8, 2026
  2. don-petry commented on Jun 9, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #330 β€” [Fleet Monitor] petry-projects/.github β€” dev-lead.yml

    Scope

    Investigate why dev-lead.yml is at 28.9% failure rate, document the failure
    modes in standards/ci-standards.md so future Fleet Monitor flags route to
    the right repo/secret, and confirm the stub itself is compliant. No change to
    the stub.

    Root-cause analysis (from recent failed runs)

    The stub in this repo is a Tier-1 caller that delegates entirely to
    petry-projects/.github-private/.github/workflows/dev-lead-reusable.yml@main.
    It already matches standards/workflows/dev-lead.yml byte-for-byte and carries
    the full permission set (incl. statuses: read per #435), so the stub is not
    the source of the failures. Sampled failed runs show three distinct signatures:

    Signature Where it surfaces Where it’s fixed
    Process completed with exit code 124 (12+ min dispatch step) Engine timeout inside the reusable petry-projects/.github-private reusable / engine timeout config
    _ApiError ... code:429 ... Resource has been exhausted then RetryableQuotaError Gemini engine quota depleted; cascade fallback runs out of options Google AI Studio billing β€” refill GOOGLE_API_KEY's project
    Error: Classic Personal Access Tokens (ghp_) are not supported by Copilot Copilot engine fallback rejects Classic PAT Org secret GH_PAT must be a fine-grained PAT, not ghp_…

    The 17 cancellations / 38 runs are expected: the reusable’s centralised
    cancel-in-progress: false lanes still emit cancelled runs whenever a higher
    lane preempts a queued event for the same PR/issue.

    Implementation Checklist

    • .github/workflows/dev-lead.yml β€” verify it remains byte-for-byte
      identical to standards/workflows/dev-lead.yml (no edits needed)
    • standards/ci-standards.md β€” append a "Dev-Lead failure-mode runbook"
      subsection under Β§5 Dev-Lead Agent mapping the three signatures above to
      their root-cause repo / secret, so the next Fleet Monitor flag is triaged in
      seconds instead of hours

    Tests to Write

    None. This is a pure documentation change with no observable behavioural
    change to any workflow, script, or runtime artifact β€” the
    Phase-2 exception in the dev-lead prompt applies. Verification is the standard
    repo lint suite (.dev-lead/scripts/dev-lead-lint.sh).

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jun 9, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • Verified .github/workflows/dev-lead.yml already matches standards/workflows/dev-lead.yml (carries the full permission set incl. statuses: read, no concurrency: block, pinned to petry-projects/.github-private/.github/workflows/dev-lead-reusable.yml@main). No stub edit needed β€” check_dev_lead_stub in scripts/compliance-audit.sh is green.
    • Classified the 11 failing runs in the monitored window: all three error signatures originate outside this repo (reusable timeout, Google AI Studio quota, Classic PAT rejected by Copilot).
    • Added a "Failure-mode runbook" subsection to standards/ci-standards.md Β§ Dev-Lead Agent mapping each signature to the upstream owner (private reusable / AI Studio billing / org secret), plus a note that the high Cancelled Runs count is expected per the centralised cancel-in-progress: false lanes.
    • Updated the GH_PAT row in the secrets table to call out the fine-grained-PAT requirement (Classic ghp_… PATs are rejected at runtime by the Copilot engine path).

    Test Results

    Phase 2 (TDD) was skipped β€” this change is pure documentation in
    standards/ci-standards.md with no observable behaviour change to any
    workflow, script, or runtime artifact, which is the Phase-2 exception clause.

    Verification via the required dev-lead-lint.sh and the repo's markdown
    linter:

    $ bash .dev-lead/scripts/dev-lead-lint.sh
      [lint] shellcheck on 12 file(s)...
      [lint] all checks passed
    
    $ npx -y markdownlint-cli2 "**/*.md" "#node_modules" "#.claude"
    markdownlint-cli2 v0.22.1 (markdownlint v0.40.0)
    Finding: **/*.md !node_modules !.claude !.claude/** !node_modules/**
    Linting: 40 file(s)
    Summary: 0 error(s)
    

    Files Changed

    • standards/ci-standards.md: Added a "Failure-mode runbook" subsection under Β§Dev-Lead Agent mapping the three observed error signatures (exit code 124, Gemini _ApiError 429, Classic PAT … not supported by Copilot) plus startup_failure to their root-cause repo or secret. Strengthened the GH_PAT row in the secrets table to call out the fine-grained-PAT requirement.

    Notes

    This issue surfaced infrastructure / engine-quota / secret-format failures
    that the Tier-1 stub in this repo cannot fix by design. Concrete follow-ups
    that can fix the failure rate live elsewhere β€” the new runbook routes them
    explicitly:

    1. Refill GOOGLE_API_KEY Google AI Studio prepayment credits β€” until done, every Gemini-step retry in the engine-fallback cascade will 429.
    2. Rotate GH_PAT to a fine-grained PAT β€” required for the Copilot engine path; the current Classic ghp_… is rejected before any work runs.
    3. Investigate dispatch-step timeouts in petry-projects/.github-private/dev-lead-reusable.yml β€” exit-code-124 runs were 10–12 min long; if the engine is expected to take that long for review fix-ups, raise the step-level timeout-minutes; otherwise add a per-engine call timeout.

    Two of the three are org-secret rotations and cannot be done from inside a
    PR; they are the right humans-only escalation path that the runbook now makes
    discoverable from this issue.

  4. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Jun 9, 2026
  5. don-petry commented on Jun 9, 2026

    @don-petry
    ContributorAuthor
    Already working on this: an open PR exists for issue #330.
  6. added a commit that references this issue on Jun 10, 2026
  7. added a commit that references this issue on Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions