Description
Summary
The apply-repo-settings reusable already does everything the org needs — it runs apply-rulesets.sh and apply-repo-settings.sh under the classic admin PAT. No repo in the fleet can reach it. Four independent gaps sit between the working reusable and the repos that need it, and they form a cycle.
This is the root cause of the recurring ruleset-drift-pr-quality-* findings across broodly, TalkTerm, google-app-scripts and bmad-bgreat-suite.
The chain
Gap 1 — the stub is not deployable. standards/workflows/apply-repo-settings.yml exists, but apply-repo-settings.yml is absent from DEPLOYABLE_WORKFLOWS in scripts/deploy-standard-workflows.sh:
pr-review-mention.yml dev-lead.yml agent-shield.yml auto-rebase.yml
dependabot-automerge.yml dependabot-rebase.yml dependency-audit.yml
add-to-project.yml initiative-driver.yml pr-auto-review.yml feature-ideation.yml
The template is never deployed to anyone. The weekly standards-deploy.yml sweep cannot pick it up.
Gap 2 — the channel tags the stub needs do not exist. ring_tier_for_repo puts TalkTerm and bmad-bgreat-suite in ring1, the rest of the fleet at stable. Present tags:
next ring0 v0.0.1–v0.0.5 v0.1.0 v1-next v1-ring0 v1.0.0
apply-repo-settings/v1-ring1 and apply-repo-settings/v1-stable are both missing. The stub pins @apply-repo-settings/v1-stable. Deploying it today would pin a nonexistent ref — a fleet-wide startup_failure.
Gap 3 — the canary will not promote those tags. Blocker #1042 holds apply-repo-settings at ring0->ring1, citing TalkTerm run 33446111321.
Gap 4 — that run fails because the repo vendors a stale script. TalkTerm, markets and bmad-bgreat-suite each carry their own .github/workflows/apply-repo-settings.yml doing run: bash scripts/apply-repo-settings.sh with GH_TOKEN: ${{ secrets.GH_PAT_WORKFLOWS }}:
TalkTerm scripts/apply-repo-settings.sh 320 lines 403-handling: 0 apply-rulesets refs: 0
.github scripts/apply-repo-settings.sh 497 lines 403-handling: 4
The vendored copy has no 403 tolerance and never invokes apply-rulesets.sh. It fails on the check-suites PATCH every run — 30/30 across the three repos — and performs no ruleset convergence at all.
The cycle: canary held → TalkTerm fails → vendored stale script → cannot adopt the stub → stub pins a tag that does not exist → tag not cut → canary held.
Ordered unblock — the order is load-bearing
Doing these out of order breaks the fleet. In particular Step 4 before Step 3 pins every repo to a nonexistent tag.
Step 1 — break the cycle at the vendored repos (maintainer)
Make TalkTerm, markets and bmad-bgreat-suite stop failing. Either sync canonical scripts/apply-repo-settings.sh into each, or remove the vendored workflow + script pending stub adoption. Removing is the smaller change and loses nothing: the vendored path has never successfully applied anything.
Verify: a run of apply-repo-settings.yml in TalkTerm reaches success, or the workflow is gone.
Step 2 — let the canary clear
#1042 auto-closes when the gate passes; it regenerates each run and must not be edited by hand.
Step 3 — cut the channel tags (maintainer)
apply-repo-settings/v1-ring1 and apply-repo-settings/v1-stable, via scripts/cut-release.sh <agent> <ver> --channel <tier> --promote --push. Dry-run first.
Verify: both refs resolve.
Step 4 — add the stub to DEPLOYABLE_WORKFLOWS (agent-safe, once Step 3 is done)
One line in scripts/deploy-standard-workflows.sh. Gate a test on it so the list and standards/workflows/ cannot drift apart again — every template that is meant to deploy should be listed, and a template that is deliberately excluded (ci.yml, sonarcloud.yml) should say so in a comment.
Step 5 — deploy and confirm the drift actually heals
Run standards-deploy.yml, let the stub PRs merge, then confirm convergence against live state, not CI colour:
for r in broodly TalkTerm google-app-scripts bmad-bgreat-suite; do
echo -n "$r "
gh api repos/petry-projects/$r/rulesets --jq '.[]|select(.name=="pr-quality")|.id' \
| xargs -I{} gh api repos/petry-projects/$r/rulesets/{} \
--jq '.rules[]|select(.type=="pull_request")|.parameters
|"codeowner=\(.require_code_owner_review) lastpush=\(.require_last_push_approval) dismissStale=\(.dismiss_stale_reviews_on_push)"'
doneAll three must read true everywhere. Current state — broodly false/false/false, TalkTerm true/false/false, google-app-scripts true/false/false, bmad-bgreat-suite true/true/false.
Not released to dev-lead
Only Step 4 is safe for an agent, and only after Step 3. Steps 1 and 3 need maintainer judgement and admin credentials; an agent doing Step 4 early would pin the fleet to a missing ref. Label deliberately withheld.
Related: #1038 (the reconciler abort itself — PR #1044 in review), #1036 (findings closed while live), #1037 (routing decision), .github-private#1620 (zero-diff PRs), canary blocker #1042.
Filed from a manual compliance sweep, 2026-09-01.
ORDERED UNBLOCK — REVISED 2026-09-07. Supersedes the five steps above.
Two of the four gaps in the original report have closed, one was misdiagnosed, and the sequence has changed. Use this list, not the one above.
Gap status
| Gap | Original claim | Status |
|---|---|---|
| 1 | stub absent from DEPLOYABLE_WORKFLOWS |
Still open, but it was never the cause of the pin failure — see below |
| 2 | ring tags missing | Closed — v1-ring1 created 2026-09-02 (by hand; the automation defect is #1065) |
| 3 | canary deadlocked | Closed — #1042 cleared; next→ring0 and ring0→ring1 both promoted |
| 4 | vendored stale scripts | Neutralised — all three set disabled_manually 2026-09-01 |
The misdiagnosis, corrected
I attributed the TalkTerm#489 / bmad#463 mis-pin to --workflow bypassing DEPLOYABLE_WORKFLOWS. That was wrong. The cause was RING_REUSABLES in scripts/lib/ring-pins.sh omitting apply-repo-settings, so ring_is_ring_reusable returned false, emit_ref_for returned empty, and the template deployed verbatim with its hardcoded @apply-repo-settings/v1-stable. That would have happened through the normal fleet sweep too.
Fixed in #1088 (merged): RING_REUSABLES now equals the registry (16/16), and the deploy refuses any computed ref that does not resolve on the host.
Revised sequence
Step A — deploy the stub to the ring1 repos. Now safe: emit_ref_for computes @apply-repo-settings/v1-ring1 (exists, edbac2551e4c), and #1088's assert-exists check refuses anything that does not resolve.
gh workflow run standards-deploy.yml -R petry-projects/.github \
-f target_repo=TalkTerm -f target_workflow=apply-repo-settings.yml -f dry_run=trueVerify the PR pins v1-ring1 before merging — read the file on the branch, do not infer it. Repeat for bmad-bgreat-suite. Per #1038 AC5″, the same change should remove each repo's vendored caller and local scripts/apply-repo-settings.sh.
markets is included at this step by maintainer decision (2026-09-07): standards/canary-rings.json is authoritative, so markets is ring1. ring_tier_for_repo in scripts/lib/ring-pins.sh still maps it to stable and must be corrected first, or markets will compute the non-existent v1-stable and be refused.
Step B — feed the gate. Manual workflow_dispatch of apply-repo-settings.yml in the ring1 repos, per the registry note: "ring members feed the gate via a manual workflow_dispatch after the stub lands. Do NOT paper over a starved gate with --override."
Success criterion is the live read, not the run's colour:
gh api repos/petry-projects/TalkTerm/rulesets --jq '.[]|select(.name=="pr-quality")|.id' \
| xargs -I{} gh api repos/petry-projects/TalkTerm/rulesets/{} \
--jq '.rules[]|select(.type=="pull_request")|.parameters
|{require_code_owner_review, require_last_push_approval, dismiss_stale_reviews_on_push}'Step C — promote ring1→stable (dwell 12h, sample_min: 1, now satisfiable). Creates stable / v1-stable.
Step D — add apply-repo-settings.yml to DEPLOYABLE_WORKFLOWS (Gap 1, still open) so the weekly sweep carries the stub to the rest of the fleet. Only after Step C — before it, stable-tier repos compute the non-existent v1-stable and #1088 will refuse them.
Step E — confirm convergence across broodly, google-app-scripts, bmad-bgreat-suite and markets with the live read above. All three parameters true everywhere.
Live state 2026-09-10 — Step C will not fire on its own
Verified against the scheduled promoter, not inferred. apply-repo-settings/v1-stable still does
not exist, yet the 05:07Z Canary Rollout run (34439789612, conclusion: success) logs:
──────── agent: apply-repo-settings ────────
nothing to promote — apply-repo-settings is fully rolled out.
The promoter reads the bare channel set (next/ring0/ring1/stable, all present) and
concludes rollout is complete, while consumers pin the major-scoped v1-stable, which was
never created. So Step C is not merely pending — the automation believes it is already done and
will keep reporting success every 4h without ever creating the tag. This is #1065 exactly, one
tier further along than the v1-ring1 case that was patched by hand on 2026-09-02.
persona-mention is the same defect, worse: it has ring0, ring1, v1-next, v1-ring0,
v1-ring1 and releases through v1.5.0, but no next, no stable, no v1-stable at all.
The fix is already in flight: PR #1097 implements #1065. Landing it is what unblocks Step C
here, and #1045 in turn unblocks .github-private#1740. Until #1097 merges, the only way to
complete Step C is another hand-created tag — which is what the standing caution below warns
against treating as evidence the pipeline works.
For the record, one thing this is not: a "no release was ever cut" problem. Every registered
ring agent has channel tags; they live in the agent's host repo (cross-repo reusables in
petry-projects/.github, dev-lead/pr-review/ci-failure-analyst in .github-private), so an
inventory of only one repo will appear to show 14 agents with no tags. The genuine gaps are the two
partially-seeded agents named above.
Standing caution
TalkTerm currently reads true/true/true, but that came from two hand-applied edits by don-petry (ruleset history: 2026-09-04T16:56Z, 2026-09-06T17:03Z), not from the reconciler. A compliant end state is not evidence the pipeline works. Step B must demonstrate convergence by the reusable, or the drift returns and the audit re-files it.
Step C done, 2026-09-13
apply-repo-settings/v1-stable now exists at a781ad9d5e50 (release v1.4.0), equal to v1-ring1. It was created by a gated canary-rollout promote apply-repo-settings (run 34738419511, 04:58Z, no override) after PR #1097 (#1065) fixed channel-major resolution. The dry run (34737895599) showed exactly that one move.
Step D is filed as #1128 (add apply-repo-settings.yml to DEPLOYABLE_WORKFLOWS). It is held until the org Actions queue drains (#1126). Step E (live-read convergence) stays here.
Activity
don-petry commentedon Sep 1, 2026
Unblock started 2026-09-01 — Step 1 in progress
Correction to the plan above: this is a THREE-hop promotion, not one
canary-rollout evaluate apply-repo-settingsreports:The pipeline is at next→ring0, not ring0→ring1. It advances one ring per promote, so the sequence is
next→ring0→ring0→ring1(createsv1-ring1) →ring1→stable(createsv1-stable).cum_fail=0— the window is currently clean. The candidate was cut at23:36:34Z, after TalkTerm’s last failing run at22:25:05Z, so no failure has entered the per-candidate window yet.Two gaps from the original report have already cleared
_gh_move_tagnow does create-if-missing (PATCH, then POST on a genuine 404 / 422 "Reference does not exist"). A missingv1-ring1/v1-stableis no longer a hard blocker.canary-rollout.ymldispatchagentenum now includesapply-repo-settings, and matchesstandards/canary-rings.json.Gap 1 (
DEPLOYABLE_WORKFLOWS) and Gap 4 (vendored scripts) remain.Action taken: vendored workflows disabled to protect the soak
apply-repo-settings.ymlset todisabled_manuallyin TalkTerm, markets and bmad-bgreat-suite.bmad was the live risk — its vendored caller has no path filter:
so every push to main fired a run, and every one failed:
bmad has an active dev-lead pipeline pushing to main, so a single push during the soak would have taken
cum_failto 1 and blocked the gate. TalkTerm and markets are path-filtered with weekly Monday crons and were lower risk, but they are disabled too — this is Step 3 of the plan brought forward, and these vendored paths have never successfully applied anything, so nothing is lost.Reversible with
gh workflow enable apply-repo-settings.yml -R petry-projects/<repo>, though the intent is to replace them with the channel-pinned stub at Step 2 rather than re-enable.Next
Soak completes at ~
2026-09-01T03:36Z. Thenpromotefornext→ring0,ring0→ring1, and a targeteddeploy-standard-workflows.sh --workflow apply-repo-settings.yml --repo <ring1 repo>before the ring1 members can feed thering1→stablesample.Mismatch to fix before deploying to markets:
ring_tier_for_repoinscripts/lib/ring-pins.shmapsmarkets → stable, butstandards/canary-rings.jsonlists markets in ring1. Deploying the stub to markets today would pin@apply-repo-settings/v1-stable, which does not exist →startup_failure. TalkTerm and bmad-bgreat-suite map toring1correctly.don-petry commentedon Sep 2, 2026
Step 1 COMPLETE — two hops promoted, plus a manual tag to unblock Step 2
Promotions landed (overnight, 2026-09-01 → 09-02)
Both promotions ran clean —
cum_fail=0held the whole way, because the three vendored callers were disabled first. The promoted candidateedbac2551e4ccontains the complete AC6 fix from #1038 (pp_apply_security_and_analysishas 2return 1paths; the earlier candidatedc0b6909had 1). The candidate was re-cut at09:33:56Zby an autocut, which restarted the dwell but replaced a pre-AC6 candidate with the correct one.Manual intervention:
apply-repo-settings/v1-ring1The
ring0→ring1promotion created only the barering1tag, not the major-scopedv1-ring1— even though the precedingnext→ring0hop had correctly createdv1-ring0.That blocked Step 2.
deploy-standard-workflows.shkeys the stub pin on the channel major, and its own comment states: "Once a channel tag exists, a bare stub is drift and must migrate to the tier'sv<M>-<tier>form." Sincev1-nextandv1-ring0exist, the deploy emits@apply-repo-settings/v1-ring1for any ring1 repo — a ref that did not exist.Created
refs/tags/apply-repo-settings/v1-ring1→edbac2551e4c091d5ee0c86195f9a34682c2bfe1, matchingring1, as a lightweight ref consistent with howring1andv1-ring0are stored.This is a workaround for an automation defect, not a fix. Filed as #1065 —
persona-mentionhas the same gap atring0andring1and is still broken. The bug hides because_resolved_channelfalls back to the bare tag, so the canary keeps working; only a stub deploy exposes it.Step 2 is now unblocked — for TWO of the three ring1 repos
ring_tier_for_repoTalkTermring1@apply-repo-settings/v1-ring1bmad-bgreat-suitering1@apply-repo-settings/v1-ring1marketsstable@apply-repo-settings/v1-stablemarketsmust not be deployed to yet.ring_tier_for_repoinscripts/lib/ring-pins.shmaps it tostable, whilestandards/canary-rings.jsonlists it inring1. Deploying today pins a nonexistentv1-stable→startup_failure. Either resolve which file is authoritative, or hold markets until after thering1→stablepromotion.Also observed
Every scheduled
canary-rolloutrun since 2026-09-01 has come backcancelled(16:51Z,22:46Z,04:57Z). The scheduledpromote-allarm is not completing, which is why these promotions only happen on manual dispatch. Worth its own look — it is the reason the first promotable window on 09-01 at ~03:36Z was missed entirely.Findings re-closed while still drifted
TalkTerm#454(closed09-01T12:30) andmarkets#418(closed09-01T14:23) both closed with no explanatory comment while their live state is unchanged —TalkTermrequire_last_push_approval=false,marketscode-qualitybypass_actors: 0.markets#418had been deliberately reopened during this sweep and was closed again. TalkTerm's workflow has beendisabled_manuallysince 09-01, so nothing could have remediated it.This is #1036 continuing to fire. Not reopening them again — that is churn against an automated closer until #1036 lands.
Next
Targeted dry-run deploy to the two eligible repos:
Then the ring1 members feed the
ring1→stablesample via manualworkflow_dispatch(registry: "Do NOT paper over a starved gate with--override").don-petry commentedon Sep 4, 2026
Resolved! This check is now passing as of 2026-09-04. Closing automatically.
don-petry commentedon Sep 7, 2026
Step 2 attempted 2026-09-07 — rolled back, root cause found
Targeted stub deploys were run for the two eligible ring1 repos. Both opened PRs pinning
@apply-repo-settings/v1-stable, a tag that does not exist. Both closed unmerged:TalkTerm#489,bmad-bgreat-suite#463.Root cause — a THIRD list, not the two named above
Gap 1 in this issue names
DEPLOYABLE_WORKFLOWS. That is real but is not what caused this. The actual cause isRING_REUSABLESinscripts/lib/ring-pins.sh, which omitsapply-repo-settings. So:No per-repo ring-tier pin is ever computed. Seven of the sixteen registered agents are affected, and
persona-mentionhas the same missing-tag problem. Filed as #1088 with ACs to derive the list from the registry and to make the deploy refuse a ref that does not resolve.Correction to my earlier note
I previously wrote that bypassing
DEPLOYABLE_WORKFLOWSvia--workflowalso bypasses re-pinning. That was wrong —--workflowonly overrides which templates are considered. The verbatim deploy happens because ofRING_REUSABLES, and it would happen the same way through the normal fleet sweep.Revised ordering for this issue
Step 2 cannot proceed until #1088 lands. After that a ring1 deploy resolves to
@apply-repo-settings/v1-ring1(exists,edbac2551e4c) and the ring1 members can feed thering1→stablesample.Promote to ring1— done;v1-ring1created by hand (see canary-rollout ring promotion can skip the v-scoped channel tag, breaking every stub deployed to that tier #1065).v1-ring1.workflow_dispatchin those repos to feedsample_min: 1.ring1→stable, then addapply-repo-settings.ymltoDEPLOYABLE_WORKFLOWSfor the fleet.markets
Maintainer decision recorded:
standards/canary-rings.jsonis authoritative, so markets isring1.ring_tier_for_repoinscripts/lib/ring-pins.shstill maps it tostableand needs correcting — a natural companion to #1088 since it is the same file, but it is NOT in #1088 scope as filed. Track separately or fold in deliberately.