Repository navigation
ksail cluster create cannot bootstrap: Talos dies composing its /etc overlay, so apid never gets certificates #3269
Description
🤖 Generated by the Agentic Engineer
Evidence
ksail cluster create fails at bootstrap on the docker provider. Both Talos containers start and stay
Up, every registry mirror reports healthy, and the Talos API never becomes reachable — the CLI
retries for the full window and exits 1:
✗ failed to execute cluster lifecycle: failed to create cluster: failed to save kubeconfig:
bootstrap failed: 15 error(s) occurred:
rpc error: code = Unavailable desc = connection error: desc = "transport:
authentication handshake failed: ... read: connection reset by peer"
timeout
The CLI error is a symptom. The cause is in the control-plane container's own log, seconds after start:
[talos] service[apid](Waiting): Waiting for service "containerd" to be "up", api certificates, config to be ready
[talos] controller runtime goroutine error: fatal controller runtime error: failed to set up /etc
overlay: failed to compose writable /etc overlay: failed to compose writable overlay:
openfs failed: failed to create root filesystem: FSCONFIG_SET_FD failed:
bad file descriptor: key="lowerdir+" fd=4
The controller runtime dies composing its /etc overlay, so apid never receives its certificates and
stays in Waiting forever. Every handshake after that is refused. machined and containerd both
report healthy, which is why the containers look fine from the outside.
Observed with:
| Talos | v1.14.0-alpha.2 (as pinned by the current ksail) |
| ksail | 7.178.34 |
| Docker | 29.6.2 (client and server) |
| Host | Darwin 25.6.0 arm64 |
Affected audience and impact
Anyone bringing up the local cluster — which is the repo's documented workflow for trying a component
before promoting it to prod, and the only way to satisfy any acceptance criterion that says "verified on
a running local cluster".
This is currently blocking real work rather than being a theoretical gap: #3263 asks for exactly such a
verification, and #3267 is parked as a draft because that check cannot be run. Any issue whose
definition-of-done needs live admission behaviour, scheduling behaviour, or reconcile behaviour is
blocked the same way. The failure is also quiet in the wrong direction — the containers stay Up, so
this reads as a slow or flaky bootstrap rather than a hard incompatibility.
Expected behaviour
ksail cluster create brings up the local Talos + Docker cluster, or fails with an error that names the
overlay incompatibility instead of a wall of handshake timeouts.
Notes toward a fix
FSCONFIG_SET_FD ... key="lowerdir+" is the new mount API composing an overlayfs lowerdir. This is a
Talos-in-Docker/kernel interaction, not a manifest problem — nothing in k8s/ is reached, since the
cluster never finishes bootstrapping. Worth checking, roughly in cost order:
- whether the pinned Talos is the issue.
v1.14.0-alpha.2is an alpha; a releasedv1.13.x/v1.14.x
may not hit this path. Pinning the local provider to a stable Talos is the cheapest test. - whether the host Docker version changed underneath. Docker 29.x is recent, and the new mount API
behaviour differs across its VM kernel. - whether upstream already tracks it, before opening anything there.
Acceptance criteria
-
ksail cluster createcompletes on the docker provider andkubectl --context admin@local get nodesshows both nodesReady. - The root cause is recorded — Talos version, Docker version, or host kernel — so the next occurrence is recognisable from the handshake symptom.
- If the cause is the pinned alpha Talos, the local provider pins a version known to bootstrap.
Rough size: medium — diagnosis is cheap to repeat, but the fix may sit upstream or in a version pin.
Current dependency (verified 2026-10-24): devantler-tech/ksail#7583 remains open and supplies the create/delete path needed for this local cluster; it follows the provider slice devantler-tech/ksail#7582. The research in devantler-tech/ksail#7577 is complete and that Feature stays open as the parent of four implementation slices. The native dependency follows #7583 so unrelated parent slices do not delay local-cluster validation. Successful KSail cluster creation and the remaining acceptance checks are still outstanding.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
coderabbitai commentedon Aug 20, 2026
🔗 Related PRs
#2725 - fix(umami): bootstrap tenant provisioning immediately [merged]
#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Oct 2, 2026
Re-checked 2026-10-02 as the oldest unblocked bug. It can't be reproduced from the agent host, which has no Docker installed. Neither the platform's CI nor KSail's main-branch CI creates the Talos-on-Docker local cluster on macOS, and KSail's Docker system test is skipped on main pushes, so CI offers no recent evidence either way. KSail still pins Talos
v1.14.0-alpha.2(its default node image and Go modules), so the report's conditions still apply: the cheapest test is still a stable Talos image, run on a Mac with Docker. The ask went to the maintainer today and is recorded in the blocker line.devantler commentedon Oct 6, 2026
Maintainer answer, given in the Slack thread under the ask on 2026-10-02 and missed until now (no run was reading thread replies; the maintainer pointed this out on 2026-10-06):
Meaning: the answer to "Docker or Colima on the agent host?" is neither — the local cluster should run on the Apple container runtime. Next: re-scope this issue to creating the local cluster on that runtime (the
containerCLI is installed on the agent host), and reproduce or close the original Docker-specific failure on that basis.devantler commentedon Oct 6, 2026
Tried it on Apple's
containerruntime today: Talos boots there, and the failure in this report does not happen.v1.14.0-alpha.2) composes its/etcoverlay, bringsapidup healthy and reaches maintenance mode in about 3 seconds. That is the step that dies under Docker here.v1.14.2I took one node all the way: config applied, bootstrapped, nodeReady, CoreDNS running, and a test pod resolved a cluster service name.containerCLI by hand.ksail cluster createcannot do this yet, because KSail has no provider for this runtime.So the remaining work is in KSail, and it is tracked in devantler-tech/ksail#7577 (evidence, the run flags that worked, three pitfalls found, and what is still unknown: several nodes, Cilium, registry mirrors). This issue is now marked as blocked by it. The trial container and its volumes are removed.
The original Docker failure was not re-tested and stays unexplained; with the direction to use Apple's runtime it no longer needs to be.
Update 2026-10-06 (later): the unknowns are settled and the build is decided (devantler-tech/ksail#7577). This issue is unblocked by devantler-tech/ksail#7583, the slice that makes
ksail cluster creatework on this runtime; it follows devantler-tech/ksail#7582.chore: agent-claim #3269 nonce=fb41870efa679d52bc0d09d95ca14d9b
chore: agent-claim #3269 nonce=87f17b7a9f8aa2271c3af5a35f33cabd
chore: agent-claim #3269 nonce=e9663336e948789d7ef24c147455e939
chore: agent-claim #3269 nonce=a3a3ed2effc457b781c652464340f3a1
devantler commentedon Nov 2, 2026
Closing as not planned: the original Docker-specific bootstrap repair is superseded by the recorded move to Apple's container runtime. The October investigation already established that the reported overlay failure does not occur on that runtime; it did not prove KSail integration.
The unfinished integration has a complete implementation owner in devantler-tech/ksail#7583, which remains open and explicitly requires the Platform local configuration to create successfully. It follows the provider work in #7582. Keeping another blocked bootstrap issue adds no separate delivery criterion.
The remaining Platform admission and scheduling observations stay open in #3263. Its native blocking dependency now points directly to KSail #7583, and its body records the same prerequisite. This consolidation does not claim the Docker defect was diagnosed or repaired, that KSail supports the new runtime yet, or that the Platform local-cluster acceptance checks have passed.