docker-container: wait for BuildKit client readiness - #4027
Merged
crazy-max merged 2 commits intoSep 1, 2026
Merged
Conversation
crazy-max
force-pushed
the
fix-docker-container-first-build-race
branch
2 times, most recently
from
August 20, 2026 14:06
3f8e6c6 to
3c9abc6
Compare
crazy-max
marked this pull request as ready for review
August 20, 2026 14:06
This was referenced Aug 20, 2026
crazy-max
force-pushed
the
fix-docker-container-first-build-race
branch
2 times, most recently
from
August 28, 2026 11:52
82259f9 to
833613b
Compare
Only memoize a BuildKit client after driver client creation succeeds. This lets bootstrap retry after transient driver readiness errors instead of keeping the first failed Client call for the lifetime of the handle. Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Use the typed client wait during the container startup window. Avoid delaying stopped or established broken builders. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
crazy-max
force-pushed
the
fix-docker-container-first-build-race
branch
from
September 1, 2026 07:17
833613b to
2d4e67c
Compare
crazy-max
changed the title
vvoland
approved these changes
Sep 1, 2026
VirSanghavi
added a commit
to antifailure/antifailure
that referenced
this pull request
Sep 13, 2026
…uildKit the rest of 20 seconds (#425) ## The failure The first attempt of v1.4.0's release run, [34779202432](https://github.com/antifailure/antifailure/actions/runs/34779202432), failed the sidecar job's *Push both architectures* step about 100 ms after it started: ``` ERROR: failed to build: waiting for BuildKit: DeadlineExceeded: context deadline exceeded while waiting for connections to become ready ``` The same builder had finished the `--load` build seconds earlier. The steps between the two builds only inspect an image, run it and log in, and none of them touches the builder. Re-running the failed jobs passed, `publish` included, which is how v1.4.0 shipped. ## The cause, and the evidence The message is buildx's. [docker/buildx#4027](docker/buildx#4027), first released in v0.37.0 and carried by ubuntu-latest, makes the docker-container driver wait for BuildKit readiness while the builder container is less than 20 seconds old. The wait's deadline is the fixed moment `StartedAt + 20s`, not a timeout of its own. A client call that arrives near the end of the window gets whatever is left, and the connection, a `docker exec` of `buildctl dial-stdio`, needs more than a tenth of a second. The job's own timestamps: - **Attempt 1**, job 103783025245: builder container created 19:57:02.38 to 02.60, push started 19:57:22.19, failed 22.29. That is inside the window with roughly 100 to 300 ms left. The `--load` build started 2 s after boot and had 18 s to spare. - **Attempt 2**, job 103783498485: builder container created 20:00:52.89 to 53.04, push started 20:01:14.38. That is 21.4 s after, outside the window, and it passed, as did `publish`. [docker/buildx#4050](docker/buildx#4050) quotes the identical error from buildx's own CI and gives the wait a timeout of its own. It is released in v0.37.1, whose `driver.go` carries `clientWaitReady` and `buildkitdReadyTimeout`. So this is a timing defect rather than a flake: any release whose first buildx call after boot lands just short of 20 seconds fails the same way. ## The fix One step right after `setup-buildx-action`, which gains `id: buildx`: - reads `buildx_buildkit_<builder>0`'s `StartedAt` and sleeps until the container is at least 22 seconds old. `StartedAt` never moves after boot, so every buildx call after the step, the `--load` build included, is outside the window. Cost: at most about 20 seconds a release. - prints `docker buildx version` and the container's age before and after, so a release log shows what the step did. - fails, rather than skipping, when there is no builder name, no container of that name, or a start time that does not parse. A wait that silently does nothing is a green step protecting nothing. - says in its comment to remove it once the runner's `docker buildx version` is v0.37.1 or later. Not taken: pinning `version: v0.37.1`. At the commit pinned here, `setup-buildx-action` downloads a pinned buildx without `verifySignature`, which trades a timing race for an unverified binary, and the pin would outlive the runner image's own upgrade. ## Proof The step was extracted from this branch's `release.yml` by a YAML parser, and run under a throwaway Docker configuration against a real local docker-container builder, with GNU `date`: - **young**: builder 3 s old. `waiting 19s for buildx's 20 second startup window to close`, then `is now 22s old`, exit 0 - **old**: the same builder at 22 s. No wait, exit 0 - **missing container**: a builder name with no container. `::error::could not read the start time of buildx_buildkit_afdemorepo-no-such-builder0`, exit 1 - **empty builder name**: `::error::setup-buildx-action reported no builder name`, exit 1 - **unparseable start time**, from a docker shim answering `not-a-time`: `::error::... reported a start time that does not parse`, exit 1 Every run printed `docker buildx version` first. `actionlint` 1.7.12 on `release.yml` reports nothing new. Its two `SC2035` notes are on `publish`'s *Checksums* step, and main's copy has the same two. ## What this does NOT prove **The race was not reproduced locally, and this step has never been run against the failure it exists to prevent.** Read the proof above as "the step waits, reports, and refuses correctly", not as "the step stops the push failing". The attempts, all with buildx v0.37.0 and v0.37.1 linux-arm64 binaries verified by checksum and by cosign against their `.sigstore.json` bundles, run standalone in a throwaway container against a local docker-container builder: - **9 arms timed on the Mac's clock**, builds issued at +18.96s to +19.93s after the builder container's `StartedAt`: every one passed, on both versions. - **The Docker VM's clock was 0.378s ahead of the Mac's.** `StartedAt` is stamped by the VM, so those 9 arms were aimed at the wrong moment by about a third of a second, and `docker exec` added its own variable delay on top. - **10 arms timed inside the VM**, sleeping and firing on the clock that stamps `StartedAt` with no `docker exec` in between, v0.37.0 builds fired at +19.746s to +20.080s: every one passed. A likely reason, not measured: locally the `dial-stdio` connection is ready in milliseconds, so the slot in which the leftover window is shorter than the connection is tiny, and sleep jitter of around 100 ms steps over it. Repro attempts were then stopped, because more sweeps would cost Docker time other work needs and would not change the evidence below. **The diagnosis rests on three things instead:** 1. buildx v0.37.0's `driver/docker-container/driver.go` `Client()`, which sets the readiness wait's deadline to the fixed moment `StartedAt + 20s`. 2. docker/buildx#4050, which quotes the identical `waiting for BuildKit: DeadlineExceeded: context deadline exceeded while waiting for connections to become ready` from buildx's own CI and fixes it in v0.37.1. 3. Run 34779202432's own timestamps: attempt 1's push inside the window and failing 100 ms in, attempt 2's push at +21.4s outside it and passing. No changelog entry: this is release workflow plumbing, and the published artifacts and what a customer runs do not change. The commit carries a `Changelog-None` trailer saying so. Signed-off-by: Vir Sanghavi <67278851+VirSanghavi@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes #4021
closes #4023
A freshly created
docker-containerbuilder can report as running beforebuildkitdis ready to serve its socket. Concurrent first builds can then fail with dial-stdio or server preface errors, and a cached failed client creation can make that transient startup failure sticky.This change only caches successfully created driver clients and makes the
docker-containerdriver wait for BuildKit readiness during the container startup window before returning a client. Established broken builders still fail promptly, while newly started builders get a bounded readiness wait through BuildKit's typed client API.