Skip to content

[Fleet Monitor] petry-projects/TalkTerm β€” .github/workflows/ci.ymlΒ #438

Closed

Description

@don-petry

🟠 DEGRADED: petry-projects/TalkTerm / .github/workflows/ci.yml

Metric Value
Failure Rate 45.2% (14 / 31 runs)
Status DEGRADED
p50 Duration 142s
p95 Duration 1391s
Successful Runs 17
Cancelled Runs 0

Workflow: .github/workflows/ci.yml

Last updated by Fleet Monitor run on 2026-08-19.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. added
    dev-leadFor dev-lead agent pickup
    fleet-trackerTracked by the Actions Fleet Monitor
    health-checkAutomated health check result
    on Aug 19, 2026
  2. don-petry commented on Aug 19, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #438 β€” [Fleet Monitor] petry-projects/TalkTerm β€” .github/workflows/ci.yml

    Root-cause analysis

    I reviewed all runs in the monitored window. Every one of the 15 failures came from a single long-lived WIP branch (worktree-implement-sprint-1), all pull_request events, all failing the genuine 90% branch-coverage gate (ERROR: Coverage for branches (89.47%) does not meet global threshold (90%)). Zero failures originated from main or any other branch. The elevated p95 (1391s) likewise comes from that branch's full 3-OS quality matrix, not from ci.yml on main (which runs secret-scan + workflow-tests in ~142s).

    So the "DEGRADED" signal is inflated by one WIP branch's legitimate coverage failures β€” not a defect or flake in ci.yml itself. Weakening the coverage gate or editing that unrelated feature branch is out of scope and would be wrong.

    Scope

    Harden the one genuine latent reliability weakness in ci.yml: its four curl release downloads (gitleaks CLI + checksums, yq + checksums) run with no --retry, so a transient network blip (DNS, connection reset, 5xx) would deterministically fail CI and inflate the failure rate. Add retries and lock the invariant with a regression guard β€” mirroring how #380 and #430 were resolved.

    Implementation Checklist

    • scripts/test-ci-workflow.sh β€” add Check 6: every curl download in the workflow must include --retry (folds backslash line-continuations so multi-line curls are evaluated as one command)
    • .github/workflows/ci.yml β€” add --retry 3 --retry-delay 2 --retry-all-errors to all four curl invocations (secret-scan gitleaks download + checksums; workflow-tests yq download + checksums)

    Tests to Write

    • scripts/test-ci-workflow.test.sh β€” fixture case: a workflow whose curl lacks --retry is REJECTED
    • scripts/test-ci-workflow.test.sh β€” fixture case: a workflow whose curl (incl. multi-line continuation) has --retry is ACCEPTED
    • Red proof: run the guard against the current ci.yml β†’ fails before the fix; passes after

    Test Command

    bash scripts/test-ci-workflow.sh && bash scripts/test-ci-workflow.test.sh (plus full bash ".dev-lead/scripts/dev-lead-lint.sh" and the repo workflow-guard suite)

  3. don-petry commented on Aug 19, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Root cause (important context)

    All 15 failures in the monitored window came from one long-lived WIP branch, worktree-implement-sprint-1 (all pull_request events), each failing the genuine 90% branch-coverage gate:
    ERROR: Coverage for branches (89.47%) does not meet global threshold (90%).
    Zero failures came from main or any other branch, and the high p95 (1391s) is that branch's full 3-OS quality matrix. ci.yml on main (secret-scan + workflow-tests, ~142s) is healthy.

    So the "DEGRADED" signal is inflated by a WIP branch's legitimate coverage failures β€” not a flake or defect in ci.yml. Weakening the coverage gate or editing that unrelated branch would be wrong and out of scope, so I did neither.

    What I did fix is the one real, durable reliability weakness in ci.yml: its four release downloads ran curl with no --retry, so a transient network blip (DNS, reset, 429/5xx) would fail CI deterministically and add to the failure rate. I hardened them and locked the invariant with a regression guard β€” the same pattern used for #380 and #430.

    Plan Execution

    • Added Check 6 to scripts/test-ci-workflow.sh: every curl download must pass --retry (folds backslash line-continuations so multi-line curls are evaluated whole; no-op when no curl present)
    • Added --retry 3 --retry-delay 2 --retry-all-errors to all four curl invocations in .github/workflows/ci.yml
    • TDD: new guard check fails against the pre-fix ci.yml (red), passes after the fix (green)
    • Added fixture self-tests (Cases 7 & 8) to scripts/test-ci-workflow.test.sh and updated the "accepted" fixtures to stay fully valid

    Test Results

    === test-ci-workflow (guard vs real ci.yml) ===
    PASS: 'concurrency.group' / keyed on github.ref / cancel-in-progress true
    PASS: 'Install yq' extracts the SHA-256 (column 19)
    PASS: every 'curl' download retries on transient failures
    All checks passed.
    
    === self-test (scripts/test-ci-workflow.test.sh) ===
    ok - buggy yq extraction is rejected
    ok - correct yq extraction is accepted
    ok - workflow without an Install yq step passes
    ok - workflow missing a concurrency block is rejected
    ok - decoy $19 in unnamed step does not rescue buggy Install yq
    ok - correct Install yq with following unnamed step is accepted
    ok - curl download without --retry is rejected            (NEW)
    ok - curl downloads with --retry (multi-line) are accepted (NEW)
    All tests passed.
    
    === full workflow-guard suite (CI workflow-tests job) ===
    dependabot-rebase: PASS   pr-review: PASS   copilot-setup: PASS
    
    === dev-lead lint (shellcheck + CODEOWNERS) ===
    [lint] all checks passed
    

    Files Changed

    • .github/workflows/ci.yml: added --retry 3 --retry-delay 2 --retry-all-errors to the gitleaks CLI + checksum downloads (secret-scan) and the yq binary + checksum downloads (workflow-tests)
    • scripts/test-ci-workflow.sh: new Check 6 enforcing --retry on every curl download
    • scripts/test-ci-workflow.test.sh: new fixture cases 7 & 8; existing "accepted" fixtures updated to include --retry so they remain fully valid

    Notes

    • No test was skipped, weakened, or deleted; the 90% coverage gate is untouched.
    • The guard matches curl only as a command word and folds line-continuations, so multi-line curls and command substitutions are handled without false positives (verified by the self-test).
    • --retry-all-errors (curl 7.71+) is available on the ubuntu-latest runners these two jobs use.
    • Suggested follow-up (separate issue): the elevated failure rate will persist until worktree-implement-sprint-1 lands β‰₯90% branch coverage or is closed; that is a branch-content matter, not a ci.yml defect.
  4. added a commit that references this issue on Aug 19, 2026
  5. added 3 commits that reference this issue on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupfleet-trackerTracked by the Actions Fleet Monitorhealth-checkAutomated health check result

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions