Skip to content

feat: implement issue #1605 — engine-liveness monitor aborts before writing its report — the fleet-outage alert can never fire, and the run's exit code contradicts its own Should-fail decision - #1606

Merged
don-petry merged 2 commits into
mainfrom
dev-lead/issue-1605-20260831-1142
Aug 31, 2026

Conversation

@don-petry

@don-petry don-petry commented Aug 31, 2026 •

Copy link
Copy Markdown
Collaborator

User description

Closes #1605

Implemented by dev-lead agent. Please review.


CodeAnt-AI Description

Prevent engine-liveness monitoring from aborting before reporting results

What Changed

  • Missing token scope or expiry headers no longer stop the monitor; they are recorded as unknown or unavailable and the report and workflow flags are still written
  • Fleet-wide outages now produce the alert body before the workflow marks the run as failed
  • The monitor stays reachable after completing its report, while the workflow’s final status reflects whether the liveness check should fail
  • Added end-to-end checks for missing headers, valid classic tokens, all-clear runs, and fleet outages

Impact

✅ Fleet outage alerts are generated
✅ No false failures for fine-grained or app tokens
✅ Workflow status matches the liveness decision

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

Summary by CodeRabbit

  • Bug Fixes

    • Improved engine liveness checks when authentication responses omit scope or expiration headers.
    • Prevented valid authentication scenarios from being incorrectly classified as failures.
    • Ensured liveness reports, environment flags, and fleet alerts are generated reliably.
  • Tests

    • Added coverage for missing headers, valid classic tokens, all-clear results, and fleet-wide outage escalation.

…riting its report — the fleet-outage alert can never fire, and the run's exit code contradicts its own Should-fail decision
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing

@codeant-ai

codeant-ai Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Reviewed your PR 328894c Aug 31, 2026 · 11:51 11:53

@codeant-ai

codeant-ai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@codeant-ai codeant-ai Bot added the size:L This PR changes 100-499 lines, ignoring generated files label Aug 31, 2026
@coderabbitai

coderabbitai Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 86f8ff2f-ab8a-422e-993a-b1ceb1610f24

📥 Commits

Reviewing files that changed from the base of the PR and between 82c1f76 and 328894c.

📒 Files selected for processing (3)
  • .github/workflows/lint.yml
  • scripts/engine_liveness.sh
  • tests/test_engine_liveness.bats

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


Important

Approval pending

CodeRabbit has no unresolved comments, but it could not review the latest commit because the review limit was reached. Follow the review guidance in this comment to continue.

📝 Walkthrough

Walkthrough

The engine liveness wrapper now handles missing authentication headers without aborting. It classifies these headers, writes reports and workflow flags, and leaves the final status decision to the workflow. New BATS tests cover token classification and fleet escalation.

Changes

Engine liveness monitor

Layer / File(s) Summary
Handle optional headers and complete the monitor
scripts/engine_liveness.sh
The wrapper tolerates missing scope and expiry headers, classifies missing values, writes its outputs, and exits successfully after processing.
Validate monitor outcomes
tests/test_engine_liveness.bats, .github/workflows/lint.yml
End-to-end tests cover missing headers, classic PATs, all-clear runs, fleet outages, workflow flags, alert output, and test-job integration.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: 🔵 Low · up to 32889

The monitor now produces outage alerts and aligns workflow status with liveness results, but recovered runs do not close or mark the standing outage issue as resolved, so operators may see stale alerts. The PR is mergeable with explicit owner awareness and follow-up for alert recovery handling.

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant EngineLiveness as engine_liveness.sh
  participant GitHubCLI as gh
  participant Outputs as GITHUB_ENV and report files
  Workflow->>EngineLiveness: Run monitor
  EngineLiveness->>GitHubCLI: Query user headers and workflow data
  GitHubCLI-->>EngineLiveness: Return optional headers and preflight results
  EngineLiveness->>Outputs: Write classifications, flags, report, and alert body
  EngineLiveness-->>Workflow: Exit 0
  Workflow->>Workflow: Apply ENGINE_LIVENESS_SHOULD_FAIL
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the implementation and impact, but it omits the required Interaction contract section and Checklist. It also does not confirm the shellcheck or documentation requirements. Add the required template sections. Mark the Interaction contract as N/A, or complete the applicable agentic-role requirements. Complete the Checklist, including shellcheck results for changed shell scripts and documentation status.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies issue #1605 and the two main fixes: preventing monitor aborts and correcting fleet-alert and exit-status behavior. It is longer than preferred but remains specific and rel…
Linked Issues check ✅ Passed The changes address the coding objectives in #1605. They guard missing scope and expiry headers, preserve report and environment-file generation, separate monitor completion from workflow failure, and…
Out of Scope Changes check ✅ Passed The changed workflow entry, monitor logic, and new BATS tests directly support issue #1605. No unrelated code or configuration changes are identified.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. (1 skipped: 1 …
Full details: Title check

Explanation

The title clearly identifies issue #1605 and the two main fixes: preventing monitor aborts and correcting fleet-alert and exit-status behavior. It is longer than preferred but remains specific and relevant.

Full details: Linked Issues check

Explanation

The changes address the coding objectives in #1605. They guard missing scope and expiry headers, preserve report and environment-file generation, separate monitor completion from workflow failure, and add regression tests for missing headers, valid tokens, all-clear runs, and fleet escalation. The requested workflow execution is a manual verification task and is excluded from this coding assessment.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev-lead/issue-1605-20260831-1142

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes a critical issue in the engine liveness script where missing headers (such as x-oauth-scopes) caused the script to abort prematurely under set -euo pipefail. The extraction pipelines are now guarded with || true, and missing scopes are gracefully classified as UNKNOWN. Additionally, a comprehensive BATS integration test suite has been added. The review feedback suggests avoiding GNU-specific date flags to ensure portability across BSD/macOS platforms, and recommends using hardcoded ISO dates or stubbing the date command to make the tests fully deterministic.

Comment thread tests/test_engine_liveness.bats Resolved
Comment thread tests/test_engine_liveness.bats Resolved
Comment thread tests/test_engine_liveness.bats Resolved
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #1606
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-08-31T12:24:23Z

@don-petry

Copy link
Copy Markdown
Collaborator Author

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-08-31T12:24:23Z

@don-petry
don-petry enabled auto-merge (squash) August 31, 2026 11:54
coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 31, 2026
@don-petry

Copy link
Copy Markdown
Collaborator Author

No description provided.

@don-petry
don-petry disabled auto-merge August 31, 2026 11:56
@donpetry-bot

Copy link
Copy Markdown
Contributor

CI checks on this PR are still running. Once they complete, re-mention @donpetry-bot to trigger a fresh review.

Posted by the donpetry-bot PR-review cascade.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

@donpetry-bot

Copy link
Copy Markdown
Contributor

CI checks on this PR are still running. Once they complete, re-mention @donpetry-bot to trigger a fresh review.

Posted by the donpetry-bot PR-review cascade.

@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry disabled auto-merge August 31, 2026 12:04
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning
Issues addressed: 0
Skipped (informational): 1
- Quality Gate passed: no action required (passing report with 0 new issues, 0 security hotspots)
```
The PR is in good state — all completed checks pass, no Tier 1 blockers, and the SonarCloud gate passed. The two in_progress checks (bats, unit) are still running.

@don-petry
don-petry enabled auto-merge (squash) August 31, 2026 12:04

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: MEDIUM
Reviewed commit: c4d32fce8363f662dbb78c3744e413afcc283b66
Review mode: triage-approved (single reviewer)

Summary

Fixes #1605: guards the PAT-audit grep pipelines with '|| true' so an absent x-oauth-scopes/expiry header (normal for fine-grained PATs/App tokens) is classified as UNKNOWN instead of killing the script under errexit before the report, GITHUB_ENV flags, and fleet-alert body are written. Adds an explicit 'exit 0' so the workflow's always() fail-gate owns red/green and the non-always() alert step stays reachable. Adds 4 end-to-end bats tests and registers them in lint.yml.

Linked issue analysis

Issue #1605 documented two defects: (1) the monitor aborts at the scope-extraction grep when the header is absent, so the report/flags/alert body were never written and the fleet-outage alert could never fire; (2) the run's exit code contradicted its own Should-fail decision. Both are substantively addressed: the grep pipelines are guarded, absent scopes classify as UNKNOWN (not a false MISSING alarm), and the wrapper always exits 0 once the report is written, deferring red/green to the workflow's always() gate — verified against engine-token-liveness.yml at the PR head, where the alert step lacks always() and the fail step is 'always() && env.ENGINE_LIVENESS_SHOULD_FAIL == true'. Tests cover exactly the four scenarios the issue demanded: missing headers, classic token, all-clear, and fleet outage (with the alert body asserted non-empty).

Findings

No blocking findings.

  • The unconditional 'exit 0' does not mask genuine failures: set -euo pipefail still aborts non-zero on real errors before reaching it; only a completed classification run exits 0, matching the documented contract.
  • The test helper workflow_run_outcome parses the real fail-gate out of engine-token-liveness.yml (returns 2 on parse failure), so a deleted/reordered gate fails the test rather than passing silently — this resolved the codeant review thread.
  • All 3 bot review threads (gemini x2, codeant x1) are resolved; no human-reviewer questions outstanding.
  • Secret scan MCP tool unavailable in this environment; gitleaks CI check is green and the diff introduces no credentials (test stub uses a dummy token).

CI status

All code checks green: bats, unit-tests, ShellCheck, actionlint, Lint, CodeQL (actions+python), SonarCloud quality gate, gitleaks, agent-shield, and all repo policy gates. One 'review / review' check-run shows cancelled — a 1-second concurrency-superseded run of the review cascade itself at this SHA, followed by a successful run of the same name; not a code failure.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-08-31T13:09:33Z.

@don-petry
don-petry merged commit 758a99f into main Aug 31, 2026
67 of 68 checks passed
@don-petry
don-petry deleted the dev-lead/issue-1605-20260831-1142 branch August 31, 2026 12:10

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: MEDIUM
Reviewed commit: c4d32fce8363f662dbb78c3744e413afcc283b66
Review mode: triage-approved (single reviewer)

Summary

Fixes #1605: the engine-liveness monitor aborted under errexit when 'gh api -i user' returned no x-oauth-scopes header (normal for fine-grained PATs / App tokens), so the report, GITHUB_ENV flags, and fleet-outage alert were never produced. The fix guards both header extractions with '|| true', classifies an absent scopes header as UNKNOWN (not MISSING), and adds an explicit 'exit 0' so the workflow's always()-gated SHOULD_FAIL step owns the run's red/green — verified consistent with the fail-gate in engine-token-liveness.yml at the head SHA. A new 263-line bats suite drives the wrapper end-to-end via a gh PATH stub, covering every acceptance criterion including the previously-unreachable FLEET alert path, and is wired into lint.yml.

Linked issue analysis

All acceptance criteria of #1605 are addressed: (1) missing scope/expiry headers are classification inputs ('|| true' guards, UNKNOWN / NO_EXPIRY states, '<none/fine-grained>' placeholder asserted in tests); (2) the same errexit+grep shape was audited — the two header extractions were the only unguarded instances in the PAT audit; (3) exit status now tracks ENGINE_LIVENESS_SHOULD_FAIL via the workflow fail-gate, with tests parsing the gate's actual if: expression out of the workflow rather than duplicating it; (4) regression test with a gh stub emitting no x-oauth-scopes header asserts completion + report + GITHUB_ENV flags; (5) end-to-end test asserts ESCALATION=FLEET, the alert body, and SHOULD_FAIL=true for a 2-repo sustained outage. The 'verify by running' criterion is met to the extent possible pre-merge: the bats suite executes the wrapper end-to-end in CI (bats check green).

Findings

No blocking findings. All 3 inline review threads (gemini-code-assist x2, codeant-ai x1) are resolved — the codeant suggestion led to the workflow_run_outcome helper parsing the real fail-gate instead of a hand-copied condition. The trailing 'exit 0' cannot mask genuine failures: set -euo pipefail still aborts the script non-zero on real errors before that line is reached. Secret-scanning MCP tool was not available in this environment; the gitleaks CI check passed. No secrets, auth-flow, or credential-handling changes — the script only reads token metadata headers for classification.

CI status

All substantive checks green at c4d32fc: shellcheck, ShellCheck, bats, unit, unit-tests, Lint, actionlint, CodeQL, Analyze (actions/python), Secret scan (gitleaks), SonarCloud (quality gate passed), CodeRabbit, agent-shield, and all repo-policy gates. One 'review / review' run shows CANCELLED — a superseded run of the review cascade itself, followed by two SUCCESS runs of the same job. Skipped checks are ecosystem-conditional (dependency-audit variants) or not-applicable stubs.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

2 participants