Repository navigation
Three github-config Repository resources never sync: every PATCH is rejected 422 for disabling org-enforced commit signoff #3094
Description
🤖 Generated by the Agentic Engineer
Symptom
CrossplaneManagedResourceNotSynced fired 2026-08-11 ~16:11Z — Synced=False continuously for ≥15 min with reason: ReconcileError. Per the alert's own wording, Crossplane has stopped applying spec to the external system, so these resources can report Ready=True while drifting from Git.
Scope — 3 of 20 Repository managed resources
All in namespace github-config:
agent-pluginsagent-skillsplatform-tenant-template
The other 17 are Synced=True.
Root cause — every update PATCH is rejected by GitHub with 422
Identical error on all three:
update failed: async update failed: failed to update the resource:
PATCH https://api.github.com/repos/devantler-tech/<repo>:
422 Commit signoff is enforced by the organization and cannot be disabled
GitHub returns that when a repository PATCH carries web_commit_signoff_required: false while the organization enforces commit signoff. The update fails as a whole, so nothing else in the payload is applied either, and the provider retries indefinitely.
The trigger is not the signoff field being wrong in Git. Both sides already agree, on all three resources and on all 17 healthy ones:
spec.forProvider.webCommitSignoffRequired = truestatus.atProvider.webCommitSignoffRequired = true
So the provider is emitting false for a field whose spec and observed state are both true — the value in the PATCH comes from the provider's internal Terraform state, not from the manifest.
The decisive evidence: one of the three has no drift at all
| Repo | spec.topics |
atProvider.topics |
Any other spec-vs-live drift |
|---|---|---|---|
agent-plugins |
10 | null |
none |
agent-skills |
11 | null |
none |
platform-tenant-template |
absent | null |
none whatsoever |
platform-tenant-template has zero differing fields between spec.forProvider and status.atProvider, yet the provider still issues a PATCH — and that PATCH is rejected. So this cannot be explained as "the manifest asks for something GitHub refuses". Something is making the provider believe an update is due, and every such update is fatal because of the signoff field it carries.
For agent-plugins/agent-skills the visible drift is topics (10 and 11 topics in Git, null live) — those topics are the user-visible thing that is silently not being applied. Note that topics do apply fine elsewhere (ksail 20, platform-template 8, unifi 8, all synced), so topics are not themselves the problem.
Why it matters
Beyond the three repos' own config drifting from Git, this is a silent class of failure: Ready stays True, the repos look managed, and only the Synced condition and this alert reveal that Crossplane stopped reconciling them. The alert did its job here — the gap is that the failure is unfixable from the manifest side.
What I could not determine
The provider logs at default verbosity carry no request payload (2 log lines since its 15:55Z restart, no signoff mentions), so I could not capture the exact PATCH body. Confirming that body is the next step, and it decides between the candidate fixes below rather than guessing between them.
Candidate fixes — deliberately not applied
- Stop managing
webCommitSignoffRequiredat the repo level. The organization enforces it, so the repo-level field is redundant and is the thing making every PATCH fatal. This is the most likely correct fix, but it currently appears on all 20 repos and 17 reconcile fine, so removing it wholesale would trigger an update on every one of them — that needs to be a considered change, not a hotfix. - Force re-observation of the affected resources so the provider's Terraform state picks up the true
truevalue, if the stale internal state is the cause. - Upstream provider issue if the provider is unconditionally serialising the field on update.
Acceptance criteria
- Capture the actual PATCH body the provider sends for
platform-tenant-template(raise provider log verbosity or trace the request) and confirm whetherweb_commit_signoff_required: falseis present. - All three resources reach
Synced=True. -
agent-pluginsandagent-skillstopics are applied and visible on the repositories. - Whatever is fixed applies to the class, so a future repo added with the same shape does not silently stop reconciling.
Activity
coderabbitai commentedon Aug 11, 2026
🔗 Related PRs
#3080 - feat(observability): alert when a Crossplane managed resource stops reconciling [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Aug 11, 2026
Hypotheses ruled out — recording the negative results so the next attempt does not repeat them
Still
Synced=Falseon all three (agent-plugins,agent-skills,platform-tenant-template) as of 17:2xZ. I tried to crack it from observable state rather than provider logs, and eliminated the obvious candidates:1. The org setting is exactly what the error says — confirmed, not assumed.
orgs/devantler-techreportsweb_commit_signoff_required: true. So GitHub genuinely will reject any repo PATCH carryingfalse.2. The failing repos are NOT misconfigured on GitHub's side.
All three report
web_commit_signoff_required: true,archived: false,is_template: false— identical to the healthy ones (ksail,platform,unifisampled). There is no GitHub-side difference to explain why these three attempt an update and the other 17 do not.3. "Missing topics is the trigger" — ruled out.
It is true that all three have
topics: 0on GitHub while healthy repos have them set (ksail20,unifi8,platform1). Butplatform-tenant-templatedeclares no topics in its spec either, so for that resource there is no topic drift to drive an update — and it still PATCHes. Topics cannot be the common cause.4. "The spec shape is unusual" — ruled out.
Comparing
spec.forProviderkey sets across all 20:platform-tenant-templatehas a reduced set (13 keys vs the common 20) — butworld-at-ruinhas an even smaller set (9 keys) and isSynced=True.agent-plugins/agent-skillscarryhasProjects+hasWikibeyond the common shape — butawscarries the same two and isSynced=True.Neither the size nor the composition of the spec discriminates failing from healthy.
What that leaves
The
falsein the PATCH does not come from the manifest, from GitHub's current state, or from the shape of the spec. The remaining explanation is the one this issue opened with: the provider's internal Terraform state for these three resources, which is not visible inspec.forProviderorstatus.atProvider.Capturing the actual PATCH body remains the deciding evidence, and it needs provider log verbosity raised — a
DeploymentRuntimeConfigchange (the pattern already exists in-repo for the UniFi provider atk8s/providers/hetzner/infrastructure/crossplane/deployment-runtime-config-upjet-unifi.yaml; the GitHub provider has no equivalent). That is a manifest change and therefore shippable, but it only takes effect after a prod deploy, so it is not an in-incident action.Impact meanwhile: unchanged and bounded —
agent-pluginsandagent-skillstopics stay unapplied, and all three resources silently stop reconciling from Git while reportingReady=True. No production workload is affected.devantler commentedon Sep 24, 2026
Status re-read live on prod, 2026-09-24:
Synced=True.platform-tenant-templatenow runs Observe + Create only, with signoff declaredtrue, so no update is sent to GitHub. ItsLastAsyncOperation=False(the 422) dates from 2026-08-16, before that change. It is leftover state, not a current failure.agent-skillscarries its topic.agent-pluginshas none: .github#158 took topics out of management because provider v0.19.1 bundles terraform-provider-github 6.6.0, which sends the signoff field on every repository update.Synced=Falsehere, which the Crossplane sync alerter (Exclude ReconcilePaused from Crossplane sync alerts #2987) now pages on after 15 minutes.What's left: provider v0.20.0 is published and bundles terraform-provider-github 6.13.0, which I confirmed from its
go.modon the Go module proxy. platform#4140 repins to it. Once that deploys, restoring the topics indevantler-tech/.githubcloses the remaining criterion.devantler commentedon Sep 26, 2026
Closing: the remaining criterion landed after #4140
I re-read it live on prod (read-only) on 2026-09-26 at about 22:50Z:
Synced=TrueandReady=True.platform-tenant-template'sLastAsyncOperationis nowTrueas of 2026-09-24 20:38Z, so the 422 left over from 2026-08-16 is gone. That was the first reconcile after fix(crossplane): repin provider-upjet-github to v0.20.0 #4140 repinned the provider to v0.20.0 (terraform-provider-github 6.13.0).deploy/repositories/agent-plugins.yamlindevantler-tech/.githubdeclares its ten topics underUpdatemanagement, andGET /repos/devantler-tech/agent-pluginsreturns exactly those ten.Synced=False, which the Crossplane sync alerter (Exclude ReconcilePaused from Crossplane sync alerts #2987) pages on after 15 minutes.