Skip to content

[CI Failure Doctor] CI Failure Investigation - Run #33691 #13685

Closed

Description

@github-actions

🏥 CI Failure Investigation - Run #33691

Summary

CI run 33691 failed because the coverage gate counted 137 tests that exist in the codebase but were not executed by the canary jobs.

Failure Details

Root Cause Analysis

The compare-test-coverage.sh step in the failing jobs enumerated 3,977 tests from all-tests.txt but only observed 3,840 entries in executed-tests.txt. The gate therefore failed with ❌ FAILURE: Found 137 tests that are NOT being executed in CI, even though the referenced integration suites completed successfully. The gap exists because the CI job did not run those 137 tests (such as TestAllCommandsExist, TestCompilationStats, TestCompileWorkflowWithValidation_InvalidFile, etc.) that are still tracked in the codebase.

Failed Jobs and Errors

  • Integration: CLI Completion & Other — aborted once compare-test-coverage.sh reported missing tests.
  • canary_go — failed at the same coverage gate step, logging the 137 tests that are NOT being executed in CI message.

Investigation Findings

  • ./scripts/compare-test-coverage.sh all-tests.txt executed-tests.txt pulled the discrepancy between defined and executed tests.
  • The missing list grows when new tests are added without ensuring the CI job runs them or when executed-tests.txt isn’t generated from the suites that include the new names.
  • This job-level coverage gate is stricter than the individual test suites and therefore fails fast whenever the two files drift apart.

Recommended Actions

  • Update the integration jobs so that their go test invocations cover the missing test names before running compare-test-coverage.sh.
  • Run ./scripts/compare-test-coverage.sh all-tests.txt executed-tests.txt locally (after regenerating all-tests.txt) to catch coverage gate failures before pushing.
  • If certain tests should remain excluded intentionally, document them and have the coverage script skip those names explicitly.

Prevention Strategies

  • Automate regeneration of all-tests.txt and validation of executed-tests.txt as part of pre-commit hooks or CI prechecks whenever new tests are added.
  • Introduce a faster preflight step that runs the coverage comparison in a lightweight job so developers discover mismatches before reaching the final canary job.

AI Team Self-Improvement

Before adding or renaming tests, rerun ./scripts/compare-test-coverage.sh and ensure the changes are reflected in the CI jobs that build executed-tests.txt; document any intentional omissions in instructions.md so future agents know how to keep the coverage files aligned.

Historical Context

No prior investigation matching this coverage-gate failure pattern was found in the cached /tmp/gh-aw/cache-memory/investigations directory.

AI generated by CI Failure Doctor

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. See usage guide.

  • expires on Feb 6, 2026, 8:08 AM UTC

Activity

  1. added
    bugSomething isn't working
    cookieIssue Monster Loves Cookies!
    on Feb 4, 2026
  2. pelikhan commented on Feb 4, 2026

    @pelikhan
    Collaborator

    /q cli failure doctor should be 1d

  3. github-actions commented on Feb 4, 2026

    @github-actions
    ContributorAuthor

    🎩 Mission equipment ready! Q has optimized your workflow. Use wisely, 007! 🔫

  4. github-actions commented on Feb 4, 2026

    @github-actions
    ContributorAuthor

    🎩 Q reporting mission complete!

    I've analyzed the CI Failure Doctor workflow based on your request and created an optimization.

    What I Found

    The CI Failure Doctor workflow currently creates issues that expire in 2 days. Based on your comment "/q cli failure doctor should be 1d", I've optimized this configuration.

    Changes Made

    ✅ Reduced issue expiration from 2d to 1d in .github/workflows/ci-doctor.md

    This is a surgical change that only modifies the safe-outputs.create-issue.expires field from 2d to 1d (line 31).

    Expected Benefits

    • CI failure investigation issues will auto-close after 1 day instead of 2 days
    • Reduces issue clutter while still providing adequate time for investigation
    • Aligns with your preference for a faster issue lifecycle

    Next Steps

    A pull request has been created with this optimization. Once merged, the Copilot agent will automatically compile the updated workflow file.


    Investigation Details: Analyzed recent CI Doctor runs (runs #1562, #1561, #1558, #1554, #1553) and issues (#13716, #13685) to confirm the current 2-day expiration pattern.

    AI generated by Q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcookieIssue Monster Loves Cookies!

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions