Repository navigation
feat: implement issue #645 — no fix when auto rebase workflow fails #646
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # ───────────────────────────────────────────────────────────────────────────── | ||
| # Auto-rebase retry — self-heals a FAILED Auto-rebase workflow run. | ||
| # | ||
| # REPO-SPECIFIC WORKFLOW (no central template). See AGENTS.md | ||
| # §"Workflow Files" exception note. Documented gap: the Auto-rebase reusable | ||
| # handles merge *conflicts* gracefully (sentinel comment → dev-lead `rebase` | ||
| # intent), but a failed *run* (transient API error, rate-limit, infra hiccup) | ||
| # posts no sentinel and nothing retries it. This handler re-runs the failed | ||
| # jobs, bounded by the run's attempt counter (MAX_ATTEMPTS), and surfaces a | ||
| # warning once retries are exhausted. | ||
| # | ||
| # workflow_run handlers run from the default branch only, so changes here take | ||
| # effect after merge to main. | ||
| # ───────────────────────────────────────────────────────────────────────────── | ||
| name: Auto-rebase retry | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: ["Auto-rebase non-Dependabot PRs"] | ||
| types: [completed] | ||
|
|
||
| permissions: {} | ||
|
|
||
| concurrency: | ||
| group: auto-rebase-retry-${{ github.event.workflow_run.id }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| retry: | ||
| # Only act on failed runs; the script re-checks but gating here avoids | ||
| # spinning up a runner for every successful/cancelled Auto-rebase run. | ||
| if: github.event.workflow_run.conclusion == 'failure' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| permissions: | ||
| actions: write # gh run rerun --failed | ||
| steps: | ||
| - name: Checkout agent repo | ||
| uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 | ||
| with: | ||
| persist-credentials: false | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| - name: Retry failed Auto-rebase run | ||
| env: | ||
| # Prefer a PAT: a re-run requested via GITHUB_TOKEN does not re-fire | ||
| # workflow_run events (GitHub's recursion guard), so only GH_PAT_WORKFLOWS | ||
| # lets attempts past the first chain back into this handler toward | ||
| # MAX_ATTEMPTS. With github.token alone, a single retry still fires. | ||
| GH_TOKEN: ${{ secrets.GH_PAT_WORKFLOWS || github.token }} | ||
| REPO: ${{ github.repository }} | ||
| CONCLUSION: ${{ github.event.workflow_run.conclusion }} | ||
| RUN_ID: ${{ github.event.workflow_run.id }} | ||
| RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }} | ||
| WORKFLOW_NAME: ${{ github.event.workflow_run.name }} | ||
| HTML_URL: ${{ github.event.workflow_run.html_url }} | ||
| MAX_ATTEMPTS: "3" | ||
| run: bash scripts/auto-rebase-retry.sh | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| # auto-rebase-retry.sh — self-heal a failed Auto-rebase workflow run. | ||
| # | ||
| # The Auto-rebase workflow handles *merge conflicts* gracefully (it posts a | ||
| # sentinel comment that dev-lead's `rebase` intent then resolves). But when the | ||
| # run itself fails — a transient GitHub API error, a rate-limit, an infra | ||
| # hiccup — no sentinel is posted and nothing retries it. This handler closes | ||
| # that gap: on a failed run it re-runs the failed jobs, bounded by the run's | ||
| # attempt counter so a deterministically-broken run cannot loop forever. When | ||
| # the attempt cap is reached it stops retrying and surfaces a warning so a | ||
| # human can intervene. | ||
| # | ||
| # Bounding works without external state: `gh run rerun --failed` increments the | ||
| # run's attempt counter, and each completed re-run fires another workflow_run | ||
| # event, so RUN_ATTEMPT naturally climbs toward MAX_ATTEMPTS. (Chaining past the | ||
| # first retry requires GH_PAT_WORKFLOWS — re-runs requested via the default | ||
| # GITHUB_TOKEN do not re-fire workflow_run events.) | ||
| # | ||
| # Env inputs: | ||
| # CONCLUSION — github.event.workflow_run.conclusion | ||
| # RUN_ID — github.event.workflow_run.id | ||
| # RUN_ATTEMPT — github.event.workflow_run.run_attempt (defaults to 1) | ||
| # WORKFLOW_NAME — github.event.workflow_run.name (logging only) | ||
| # HTML_URL — github.event.workflow_run.html_url (logging only) | ||
| # REPO — owner/repo (defaults to GITHUB_REPOSITORY) | ||
| # MAX_ATTEMPTS — give up after this many attempts (default: 3) | ||
| # DRY_RUN — if "true", log intent but do not call gh (default: false) | ||
| # GITHUB_STEP_SUMMARY — path for the run summary (optional) | ||
| # | ||
| # Always exits 0: a handler failure would itself be noise. Outcomes are surfaced | ||
| # via ::notice:: / ::warning:: annotations and the step summary instead. | ||
|
|
||
| CONCLUSION="${CONCLUSION:-}" | ||
| RUN_ID="${RUN_ID:-}" | ||
| RUN_ATTEMPT="${RUN_ATTEMPT:-1}" | ||
| WORKFLOW_NAME="${WORKFLOW_NAME:-Auto-rebase}" | ||
| HTML_URL="${HTML_URL:-}" | ||
| REPO="${REPO:-${GITHUB_REPOSITORY:-}}" | ||
| MAX_ATTEMPTS="${MAX_ATTEMPTS:-3}" | ||
| DRY_RUN="${DRY_RUN:-false}" | ||
| GITHUB_STEP_SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}" | ||
|
|
||
| summary() { echo "$1" >> "$GITHUB_STEP_SUMMARY" 2>/dev/null || true; } | ||
|
|
||
| # Default any non-integer attempt to 1 so a malformed payload still retries once | ||
| # rather than silently skipping or crashing under `set -e`. | ||
| if ! [[ "$RUN_ATTEMPT" =~ ^[0-9]+$ ]]; then | ||
| echo "::notice::RUN_ATTEMPT '${RUN_ATTEMPT}' is not an integer — defaulting to 1" | ||
| RUN_ATTEMPT=1 | ||
| fi | ||
| if ! [[ "$MAX_ATTEMPTS" =~ ^[0-9]+$ ]] || [ "$MAX_ATTEMPTS" -lt 1 ]; then | ||
| MAX_ATTEMPTS=3 | ||
| fi | ||
|
|
||
| # ── guard: only act on failed runs ──────────────────────────────────────────── | ||
| if [[ "$CONCLUSION" != "failure" ]]; then | ||
| echo "::notice::${WORKFLOW_NAME} run concluded '${CONCLUSION:-unknown}' (not failure) — nothing to retry" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # ── guard: need a run id to retry ───────────────────────────────────────────── | ||
| if [ -z "$RUN_ID" ]; then | ||
| echo "::warning::No RUN_ID provided — cannot retry the failed ${WORKFLOW_NAME} run" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # ── give up once the attempt cap is reached ─────────────────────────────────── | ||
| if [ "$RUN_ATTEMPT" -ge "$MAX_ATTEMPTS" ]; then | ||
| echo "::warning::${WORKFLOW_NAME} run ${RUN_ID} failed after ${RUN_ATTEMPT} attempt(s) (cap ${MAX_ATTEMPTS}) — giving up, manual intervention required" | ||
| summary "### Auto-rebase retry exhausted" | ||
| summary "" | ||
| summary "Run [\`${RUN_ID}\`](${HTML_URL}) failed after **${RUN_ATTEMPT}** attempt(s) (cap ${MAX_ATTEMPTS}). Automated retries are exhausted — a maintainer should investigate manually." | ||
| exit 0 | ||
| fi | ||
|
|
||
| next_attempt=$(( RUN_ATTEMPT + 1 )) | ||
| echo "::notice::${WORKFLOW_NAME} run ${RUN_ID} failed on attempt ${RUN_ATTEMPT} — re-running failed jobs (attempt ${next_attempt}/${MAX_ATTEMPTS})" | ||
|
|
||
| if [ "$DRY_RUN" = "true" ]; then | ||
| echo " [dry-run] would run: gh run rerun ${RUN_ID} --failed (repo ${REPO})" | ||
| summary "### Auto-rebase retry (dry-run)" | ||
| summary "Would re-run failed jobs of run \`${RUN_ID}\` (attempt ${next_attempt}/${MAX_ATTEMPTS})." | ||
| exit 0 | ||
| fi | ||
|
|
||
| if gh run rerun "$RUN_ID" --failed ${REPO:+--repo "$REPO"}; then | ||
| echo "::notice::Re-run of run ${RUN_ID} requested (attempt ${next_attempt}/${MAX_ATTEMPTS})" | ||
| summary "### Auto-rebase retry dispatched" | ||
| summary "Re-ran failed jobs of run [\`${RUN_ID}\`](${HTML_URL}) — attempt ${next_attempt}/${MAX_ATTEMPTS}." | ||
| else | ||
| echo "::warning::Failed to request re-run of run ${RUN_ID} — manual intervention may be required" | ||
| summary "### Auto-rebase retry failed to dispatch" | ||
| summary "Could not re-run run [\`${RUN_ID}\`](${HTML_URL}). A maintainer should re-run it manually." | ||
| fi | ||
|
|
||
| exit 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| #!/usr/bin/env bats | ||
| # Unit tests for scripts/auto-rebase-retry.sh | ||
| # Covers the workflow_run failure handler that self-heals a failed Auto-rebase | ||
| # run by re-running its failed jobs, bounded by run_attempt, and surfacing a | ||
| # warning to humans once retries are exhausted. | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "$BATS_TEST_FILENAME")/../../.." && pwd)" | ||
| RETRY_SCRIPT="$SCRIPT_DIR/scripts/auto-rebase-retry.sh" | ||
|
|
||
| setup() { | ||
| STUB_BIN_DIR="$(mktemp -d)" | ||
| RERUN_MARKER="$(mktemp -u)" | ||
| export RERUN_MARKER | ||
| # Mock gh: record any `run rerun` invocation to RERUN_MARKER. | ||
| cat > "$STUB_BIN_DIR/gh" <<'GHEOF' | ||
| #!/usr/bin/env bash | ||
| case "$*" in | ||
| *"run rerun"*) | ||
| echo "$*" > "$RERUN_MARKER" | ||
| [ "${GH_STUB_RERUN_FAILS:-0}" = "1" ] && exit 1 | ||
| echo "Requested rerun of run" | ||
| ;; | ||
| *) | ||
| echo "{}" | ||
| ;; | ||
| esac | ||
| GHEOF | ||
| chmod +x "$STUB_BIN_DIR/gh" | ||
| export PATH="$STUB_BIN_DIR:$PATH" | ||
|
|
||
| export GITHUB_STEP_SUMMARY | ||
| GITHUB_STEP_SUMMARY="$(mktemp)" | ||
|
|
||
| # Sensible defaults for a failing run on its first attempt. | ||
| export REPO="petry-projects/.github-private" | ||
| export RUN_ID="123456" | ||
| export RUN_ATTEMPT="1" | ||
| export CONCLUSION="failure" | ||
| export WORKFLOW_NAME="Auto-rebase non-Dependabot PRs" | ||
| export HTML_URL="https://github.com/petry-projects/.github-private/actions/runs/123456" | ||
| export MAX_ATTEMPTS="3" | ||
| unset GH_STUB_RERUN_FAILS DRY_RUN | ||
| } | ||
|
|
||
| teardown() { | ||
| rm -rf "$STUB_BIN_DIR" | ||
| rm -f "$RERUN_MARKER" "$GITHUB_STEP_SUMMARY" | ||
| } | ||
|
|
||
| _reran() { [ -f "$RERUN_MARKER" ]; } | ||
|
|
||
| @test "auto-rebase-retry: failure on first attempt → reruns failed jobs" { | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| _reran | ||
| grep -q "rerun" "$RERUN_MARKER" | ||
| grep -q "$RUN_ID" "$RERUN_MARKER" | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: success conclusion → noop, no rerun" { | ||
| export CONCLUSION="success" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| ! _reran | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: cancelled conclusion → noop, no rerun" { | ||
| export CONCLUSION="cancelled" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| ! _reran | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: attempt at cap → no rerun, gives up with warning" { | ||
| export RUN_ATTEMPT="3" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| ! _reran | ||
| [[ "$output" == *"::warning::"* ]] | ||
| grep -qi "exhaust\|gave up\|giving up\|manual" "$GITHUB_STEP_SUMMARY" | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: attempt past cap → no rerun" { | ||
| export RUN_ATTEMPT="9" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| ! _reran | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: last allowed attempt (cap-1) → reruns" { | ||
| export RUN_ATTEMPT="2" | ||
| export MAX_ATTEMPTS="3" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| _reran | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| @test "auto-rebase-retry: missing RUN_ID → noop, no rerun" { | ||
| export RUN_ID="" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| ! _reran | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: non-numeric RUN_ATTEMPT defaults to 1 → reruns" { | ||
| export RUN_ATTEMPT="notanumber" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| _reran | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: DRY_RUN → logs intent, does not rerun" { | ||
| export DRY_RUN="true" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| ! _reran | ||
| [[ "$output" == *"dry-run"* ]] | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: rerun API failure → warns but exits 0" { | ||
| export GH_STUB_RERUN_FAILS="1" | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| _reran | ||
| [[ "$output" == *"::warning::"* ]] | ||
| } | ||
|
|
||
| @test "auto-rebase-retry: empty REPO → reruns without --repo flag" { | ||
| export REPO="" | ||
| unset GITHUB_REPOSITORY | ||
| run bash "$RETRY_SCRIPT" | ||
| [ "$status" -eq 0 ] | ||
| _reran | ||
| ! grep -q -- "--repo" "$RERUN_MARKER" | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.