Repository navigation
feat: implement issue #2235 — CI Failure Analyst never runs for failed GitHub Actions jobs: the check_run trigger only fires for third-party checks - #2308
Conversation
…d GitHub Actions jobs: the check_run trigger only fires for third-party checks
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Code Review
This pull request introduces support for the workflow_run event in the CI Failure Analyst, allowing it to diagnose failed GitHub Actions runs in addition to third-party check_run events. It adds a new payload resolution script scripts/ci-failure-analyst-resolve.sh, updates documentation and test scenarios, and implements a comprehensive BATS test suite. The review feedback suggests improving the test setup and teardown by adding error handling to the mktemp -d command and guarding the rm -rf cleanup command with a non-empty check to prevent accidental deletions.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2eb942a6c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
QA Lead — test-risk advisoryRisk tier: MEDIUM. This is shared workflow automation used by every caller stub. If the new checkout or output wiring is wrong, the analyst breaks quietly in every repo once What I'd shore up (highest leverage first):
Escalate? no Advisory only — I comment, I do not change code. Ground: BMAD Test Architecture (bmad-tea v1.19.0). Opt out on this item with the qa-lead:hands-off label. |
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T21:17:24Z. |
Dev-Lead — review-changes (applied)Changes committed and pushed. Requested items addressed:
|
|
No description provided. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 43241964c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
No description provided. |
- Fix Issue 1: Filter payload PRs by open state, matching the fallback behavior (lines 73-74). Accepts only open PRs from the webhook payload, consistent with the commits-to-pulls API fallback which also filters by state == "open". - Fix Issue 2: Remove error suppression in idempotency function (line 118). Changed '|| true' to '|| return' so API failures propagate instead of silently returning empty, which could cause duplicate comment posts. - Update test fixtures to include PR state field matching real GitHub webhook payloads that include state information. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
Fixed in two places:
Both test fixtures and the test cases have been updated to include the field that real GitHub webhook payloads provide. |
This comment has been minimized.
This comment has been minimized.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
No description provided. |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Dev-Lead — fix-reviews (partial)A commit was pushed, but not every requested change was applied. Per requested item:
The unaddressed items above still need work. |
Dev-Lead — fix-reviews (applied)Changes committed and pushed. Requested items addressed:
|
|
No description provided. |
|
Original reply, kept for the record (unverified — it does not count as a fix):
|
|
Original reply, kept for the record (unverified — it does not count as a fix):
|
Test suite newly red — human attention neededThe |
|
dev-lead is withholding action on this item. It is labeled To re-enable automated pickup: remove the |
|



User description
Problem
CI Failure Analyst never runs for failed GitHub Actions jobs: the check_run trigger only fires for third-party checks
From the issue: CI Failure Analyst is documented as diagnosing failed CI, Lint and Test runs. In practice it never runs for a failed GitHub Actions job, in any repo, on any release. It only runs when a third-party app's check fails.
Risk
Medium — changes GitHub Actions workflow behavior, which is exercised only post-merge; verify via the affected workflow runs.
Test plan
Tests added/updated:
tests/aw/ci-failure-analyst/scenarios.md,tests/ci_failure_analyst_resolve.bats. Verification:bash scripts/dev-lead-lint.sh(shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.Rollback
Revert this PR. No non-revertible side effects (no tags, migrations, or external state).
Monitoring
Watch the affected workflow run(s) in the Actions tab and this PR's Lint check for regressions.
Refs #2235. This PR delivers criteria 1 and 3 of that issue: the reusable accepts a
check_runor aworkflow_runpayload. The stub trigger (criterion 2) and the live proof (criterion 4) follow in a second PR onceci-failure-analyst/v1-stablecarries this reusable, so the issue stays open after this merges. The change is inert until a channel is cut.CodeAnt-AI Description
Add GitHub Actions run support to CI Failure Analyst
What Changed
Impact
✅ Failed Actions runs can be analyzed after trigger rollout✅ One diagnosis comment per commit✅ No failure comments on closed PRs or default-branch pushes💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.