build: add OCI artifact default opt-out by crazy-max · Pull Request #3991 · docker/buildx

crazy-max · 2026-08-04T08:58:00Z

relates to

revert "exporter: fix attestation manifest push order" or make it opt-in moby/buildkit#7014
Attestations as OCI artifacts fail to push to gitlab.com when the subject manifest is new moby/buildkit#7007
exporter: revert attestation manifest push order moby/buildkit#7015

This adds BUILDX_NO_DEFAULT_OCI_ARTIFACT=true as a compatibility escape hatch for Buildx users who need to keep provenance attestations enabled while disabling the newer OCI artifact attestation storage format.

When this environment variable is set, Buildx now sets oci-artifact=false when attestation support is available and the user did not already set oci-artifact explicitly. This keeps explicit exporter options authoritative, mirrors the existing BUILDX_NO_DEFAULT_ATTESTATIONS compatibility pattern, and applies to both buildx build and buildx bake.

BuildKit started defaulting attestation manifests to OCI artifacts in moby/buildkit#6914 after moby/buildkit#6171 tracked registry support for that format. The issue reported in moby/buildkit#7007 showed that GitLab can still reject a push when the attestation manifest has a subject that points at a manifest that is new in the same push. The initial BuildKit-side fix in moby/buildkit#7012 changed the manifest push order, but the follow-up discussion in moby/buildkit#7014 pointed out that this is a registry compliance issue because OCI distribution requires registries to initially accept valid manifests whose subject does not exist yet.

This PR does not change the default behavior and does not make Buildx prefer the non-OCI artifact format for compliant registries. It only provides a narrow opt-out for environments that need time to deal with registry compatibility problems, while preserving provenance attestations instead of requiring users to disable them entirely with BUILDX_NO_DEFAULT_ATTESTATIONS.

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>

thaJeztah · 2026-08-04T09:44:28Z


+const (
+	noDefaultAttestationsEnv = "BUILDX_NO_DEFAULT_ATTESTATIONS"
+	noDefaultOCIArtifactEnv  = "BUILDX_NO_DEFAULT_OCI_ARTIFACT"


Thinking out loud; what are the formats we (currently) support for including attestations? Would it make sense to have a "attestations-format" selection instead of a boolean? (e.g. BUILDX_ATTESTATIONS_FORMAT=(oci|foo|bar|compat|legacy)) or was the old format not complying to any standards? (ISTR there were options for legacy registries, so those would still be valid and potentially work around the issue with non-OCI-compliant registries?)

I'd rather keep this as a narrow opt-out for now (specially since this change aims for a patch release). BuildKit currently exposes this as a boolean exporter option, oci-artifact=true|false, so mirroring that as BUILDX_NO_DEFAULT_OCI_ARTIFACT keeps Buildx from inventing a new format-selection layer on top.

The explicit format selection already exists through --output type=image,oci-artifact=false when users want to choose it directly. This env var only changes the default when the option was not set, similar to BUILDX_NO_DEFAULT_ATTESTATIONS.

A generic BUILDX_ATTESTATIONS_FORMAT=legacy|compat|oci feels a bit too broad to me because Buildx would then need to define what those names mean and keep mapping them to BuildKit exporter behavior. If BuildKit grows more attestation storage formats later, I think that should first be exposed as a BuildKit exporter option, and Buildx can follow that model.

Yeah, that's ok with me; mostly wanted to avoid having to introduce yet-another env-var if there's more options than enable/disable (not even sure what format the "disable" means and if that's a legacy fallback defined by OCI?)

thaJeztah · 2026-08-04T11:05:09Z

+	if isMobyWorker(sb) {
+		t.Skip("attestations are not supported by the docker worker")
+	}


Interesting; are these two the same? (I went looking if we had a more specific "supports attestations" instead of driver-name 😂)

buildx/tests/integration.go

Lines 126 to 129 in 1988826

func isMobyWorker(sb integration.Sandbox) bool {

name, _, hasFeature := driverName(sb.Name())

return name == "docker" && !hasFeature

}

buildx/tests/integration.go

Lines 136 to 139 in 1988826

func isDockerWorker(sb integration.Sandbox) bool {

name, _, _ := driverName(sb.Name())

return name == "docker"

}

Hum indeed, should rework this 😅

thaJeztah

LGTM

GitLab CI builds and pushes the SM120 image to registry.ocnr.org/infra/vllm (tag derived from the VERSION file), VERSION override so the wheel reports 0.30.0+sm120.cu130, and Dockerfile tuning for the ocnr build host: - torch_cuda_arch_list narrowed to 12.0 (build + kv_connectors stages) - DeepEP built for 12.0a only; surface the NVIDIA wheel's nccl lib dir via LIBRARY_PATH (DeepEP links -l:libnccl.so.2 without a -L for it) - MAX_JOBS=32 / NVCC_THREADS=1 / CARGO_BUILD_JOBS=32 for the CI host - wheel version override replaces the .git bind mount - VLLM_MAX_SIZE_MB=1000 (SM120-only wheel still exceeds the PyPI limit) - runtime: g++ for FlashInfer's runtime-JIT SM120 kernels (cuda-nvrtc-dev is already upstream), py-spy + dump-jam-state.sh pre-installed for incident triage Also folded in (previously separate CI commits, net content unchanged): renovate bot wiring (schedule job, repo config, hostRules), positive pipeline filters, build gated to the default branch, docker/renovate base-image digest bumps, build-guard nested-path fix, unpinned CI image tags, the 0.30.0+sm120.cu130 VERSION bump, the tag-derivation comment, and the registry-attestation opt-out below. GitLab registry attestations: first builds of new content intermittently fail at push with "blob unknown to registry - sha256:<image manifest digest>": buildkit v0.32+ emits the provenance attestation as an OCI artifact manifest whose `subject` references the image manifest, and the GitLab container registry rejects a manifest whose subject is not already present (strict referential integrity). Push ordering between the two sibling manifests is not guaranteed, so the push dies when the attestation manifest lands first; a retry re-orders and succeeds. BUILDX_NO_DEFAULT_OCI_ARTIFACT=true (buildx >= v0.36.1) keeps provenance attestations but emits the older docker attestation format without a `subject`, which the GitLab registry accepts. Refs: - moby/buildkit#7007 - docker/buildx#3991 - https://gitlab.com/gitlab-org/container-registry/-/work_items/2367 Deterministic rebuilds: buildkit stamps every re-executed step's wall-clock time into the image config (created/history), and setuptools stamps wheel archive entries with checkout mtimes, so a rebuild of an identical tree pushed a different digest even with a warm cache (observed: 65/111 steps CACHED, digest still changed d6b7d2c6 -> 849b0338). Thread SOURCE_DATE_EPOCH (the commit timestamp) into buildkit (image config metadata) and into the csrc-build stage so setuptools/wheel clamp archive timestamps. Signed-off-by: Nick O'Connor <nick@ocnr.org>

GitHub is canonical: no build, test, publication, or documentation path depends on GitLab's CI or container registry anymore. - Chart defaults, install commands, and docs use ghcr.io/haproxy-haptic; releases up to 0.5.0 are mirrored there with their original digests. - .goreleaser.yml publishes to GitHub directly. CI images are pinned to the GHCR mirror, and ci-images.yml publishes images for changed build inputs from main so later pull requests reuse them. - Upgrade tests discover and install chart baselines from GHCR; the Ingress conformance echo image comes from registry.k8s.io at its pinned digest. - Publication containers set BUILDX_NO_DEFAULT_OCI_ARTIFACT, as the GitLab release jobs did (docker/buildx#3991). - .gitlab-ci.yml and its GitLab-only tooling are removed. Tests that read it now check the GitHub workflows: job inventory, prose profile, nightly selections, feature profiles, benchmark bounds, release ordering, per-repository attestation, maint/ tagging, and upgrade coverage. - Agent docs, CLAUDE.md, contributor docs, SECURITY.md, and Renovate describe GitHub only.

Moves HAPTIC's development, CI, and publication from GitLab to GitHub, and adds the opt-in Pod readiness gate that lets single-replica rollouts drop no requests. After this merges, no build, test, publication, or documentation path depends on GitLab's CI or container registry. ## Size +32k / −7.5k lines. About half of the additions are generated data: the offline Pod schema for the readiness gate's tests (7.6k), the generated 98-job CI matrix (`ci-speed-proof.yml`), and per-test duration weights the scheduler balances shards with (4.5k). The deletions are mostly `.gitlab-ci.yml` and the migration's one-time tooling. ## CI on GitHub Actions - `CI` runs the complete matrix on every pull request, merge-queue group, and push: unit shards, chart and docs builds, the e2e matrix (HAProxy 3.0–3.4), minimum Kubernetes, upgrade baselines, Ingress and Gateway API conformance, and the former nightly suites (Gateway API release matrix, churn, scale, upstream canary). Only the canary may fail. No workflow runs on a schedule. - Measured earlier: the full matrix finished in 20m53s on GitHub ([run 38432655131](https://github.com/haproxy-haptic/HAPTIC/actions/runs/38432655131)) against 28m04s on GitLab (pipeline 2940262274). - The former nightly suites cost about a minute of lead time (21m42s and 22m12s against 20m53s without them, [run 38555150518](https://github.com/haproxy-haptic/HAPTIC/actions/runs/38555150518)), traded for one gate instead of a separate schedule. - Publication (`release.yml`, `snapshot.yml`, `pages.yml`) verifies the producing CI run, its job inventory, and its artifact digests before it publishes. The website publishes after every successful `main` CI run. The publication containers set `BUILDX_NO_DEFAULT_OCI_ARTIFACT=true`, as the GitLab release jobs did (docker/buildx#3991). - `ci-images.yml` publishes CI images for changed build inputs from `main` (`ghcr.io/haproxy-haptic/haptic-ci:<input-hash>-hp<series>`); jobs pull that tag before building in-job, so a toolchain bump slows down only the PR that makes it. - The startup load gate's budget follows the controller's startup probe window (80%, 240 s by default) instead of a fixed 120 s. Two controllers starting together on a 4-vCPU runner spent 104–116 s on ~1,130 validation tests, and `e2e-rate-limit` ran out of time at 98% with no failure. The reinit grace stays 45 s above the budget. ## Registry, docs, and GitLab retirement - Chart defaults, install commands, the landing page, and examples use `ghcr.io/haproxy-haptic/haptic`. Releases up to 0.5.0 are already there with their original digests and signatures ([run 38498045175](https://github.com/haproxy-haptic/HAPTIC/actions/runs/38498045175)). - Dependency updates come from Renovate, self-hosted in [haproxy-haptic/renovate-runner](https://github.com/haproxy-haptic/renovate-runner) with its own GitHub App. Workflow action pins carry their release versions so Renovate can track them. - `.goreleaser.yml` publishes to GitHub directly. Upgrade tests discover chart baselines from GHCR; the Ingress conformance echo image comes from `registry.k8s.io` at its pinned digest. - The documentation site packages the eight preserved playground engines as Brotli, so the complete site fits GitHub Pages (1.32 GB → 478 MB). - Deleted `.gitlab-ci.yml` and its train, budget, nightly-dispatch, and failure-reporting tooling, the one-time registry mirrors, the GitLab comparison baseline, and 23 migration-only proof and diagnosis workflows. - Tests that read `.gitlab-ci.yml` are replaced by checks on the GitHub workflows: job inventory and prose profile (`.github/ci-verification.json`), nightly selections and failure policy, feature e2e profiles, the bounded benchmark smoke, chart-publication ordering, per-repository image attestation, `maint/` tagging, the `latest` ordering check, and upgrade baseline coverage. - `CLAUDE.md`, agent docs, contributor docs, the Gateway API upgrade playbook, the issue tracker guide, the chart's `home`/`sources`, the PR skill, `SECURITY.md` (private vulnerability reporting), and Renovate (GitLab preset inlined) describe GitHub only. The Go module path stays `gitlab.com/haproxy-haptic/haptic` for now. ## Pod readiness gate `controller.templateLibraries.kubernetesBackends.podReadinessGate.enabled` adds the condition `haproxy-haptic.org/load-balancer-ready`. A pod listing it under `readinessGates` becomes Ready only after every HAProxy pod runs a configuration that routes to it, so a Deployment keeps the old pod until then. It's chart-only: `kubernetes-backends` watches pods, routes a gated pod as soon as its containers are ready, and sets the condition in the `deployed` status phase. See ADR-0034 and `docs/site/docs/operations/pod-readiness-gate.md`. The GitHub runs exposed the problem it fixes: an ungated single-replica rollout stopped the old backend 9 ms after HAPTIC saw the new endpoint, and the deploy landed 482 ms later, so two probes failed ([run 38483303507](https://github.com/haproxy-haptic/HAPTIC/actions/runs/38483303507)). - `TestIngressRollingRestartZeroDowntime` gates its backend and allows no failed request. - `TestIngressRollingRestartUngatedRecovers` keeps the ungated case: only fast 503s, within one window no longer than the scale suite's 3 s convergence budget. This replaces the old "at most one 503" bound for ungated pods, which asserted runner speed rather than a guarantee HAPTIC can give. - Enabling the gate grants the controller cluster-wide `pods` read and `pods/status` patch; the RBAC golden test pins that this happens only with the gate on. ## Local verification of the final commit - `make lint audit`, `make test` (11,804 tests), `make lint-chart`, and `./scripts/test-templates.sh` (1,144/1,144). - Strict MkDocs builds of the docs site and landing page. - e2e on a fresh kind cluster with the controller limited to 4 CPUs: `TestObservedGenerationIsTruthful|TestHaptic|TestGateway|TestIngress|TestHTTPRoute`, 126 passed, 0 failed, including both rolling-restart tests with zero failed requests.

crazy-max added this to the v0.36.1 milestone Aug 4, 2026

crazy-max added the status/needs-cherry-pick/v0.36 label Aug 4, 2026

github-actions Bot assigned crazy-max Aug 4, 2026

github-actions Bot added area/build area/tests labels Aug 4, 2026

build: add OCI artifact default opt-out

6ee129a

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>

crazy-max force-pushed the no-default-oci-artifact-env branch from a94cd36 to 6ee129a Compare August 4, 2026 09:02

crazy-max added the status/needs-docs-follow-up label Aug 4, 2026

crazy-max requested review from thaJeztah, tianon-sso and vvoland August 4, 2026 09:18

thaJeztah reviewed Aug 4, 2026

View reviewed changes

thaJeztah mentioned this pull request Aug 4, 2026

revert "exporter: fix attestation manifest push order" or make it opt-in moby/buildkit#7014

Closed

crazy-max marked this pull request as ready for review August 4, 2026 09:53

crazy-max mentioned this pull request Aug 4, 2026

[v0.36] cherry-picks for v0.36.1 #3993

Merged

crazy-max removed the status/needs-cherry-pick/v0.36 label Aug 4, 2026

thaJeztah mentioned this pull request Aug 4, 2026

Attestations as OCI artifacts fail to push to gitlab.com when the subject manifest is new moby/buildkit#7007

Closed

4 tasks done

thaJeztah reviewed Aug 4, 2026

View reviewed changes

thaJeztah approved these changes Aug 4, 2026

View reviewed changes

crazy-max merged commit 37e5bb8 into docker:master Aug 4, 2026
162 checks passed

crazy-max deleted the no-default-oci-artifact-env branch August 4, 2026 11:53

This was referenced Aug 4, 2026

ci(e2e): fix GitLab e2e push attestations docker/build-push-action#1593

Merged

build: Buildx OCI artifact opt-out docker/docs#25713

Merged

rancherbot mentioned this pull request Aug 5, 2026

rddepman: bump dockerBuildx from 0.36.0 to 0.36.1 rancher-sandbox/rancher-desktop#10776

Merged

thaJeztah mentioned this pull request Sep 1, 2026

29.7.0 (containerd image store) breaks docker push with "blob unknown to registry"; references blob skipped as "already exists" moby/moby#53293

Open

Al-thi mentioned this pull request Sep 10, 2026

Docker repository push fails after update from 3.95.1 to 3.96.0 sonatype/nexus-public#1059

Open

r-desktop-workflow-from-workflow Bot mentioned this pull request Sep 10, 2026

rddepman: bump dockerBuildx from 0.35.0 to 0.37.0 rancher-sandbox/rancher-desktop-2#746

Merged

phihos mentioned this pull request Oct 14, 2026

Move HAPTIC to GitHub and add the Pod readiness gate haproxy-haptic/HAPTIC#44

Merged

Navigation Menu

Repository navigation

build: add OCI artifact default opt-out - #3991

build: add OCI artifact default opt-out#3991
crazy-max merged 1 commit into
docker:masterfrom
crazy-max:no-default-oci-artifact-env

crazy-max commented Aug 4, 2026 •

edited

Loading

Uh oh!

thaJeztah Aug 4, 2026 •

edited

Loading

Uh oh!

crazy-max Aug 4, 2026

Uh oh!

thaJeztah Aug 4, 2026

Uh oh!

thaJeztah Aug 4, 2026

Uh oh!

crazy-max Aug 4, 2026

Uh oh!

thaJeztah left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	func isMobyWorker(sb integration.Sandbox) bool {
	name, _, hasFeature := driverName(sb.Name())
	return name == "docker" && !hasFeature
	}

	func isDockerWorker(sb integration.Sandbox) bool {
	name, _, _ := driverName(sb.Name())
	return name == "docker"
	}

Conversation

crazy-max commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crazy-max Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

thaJeztah Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

thaJeztah Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

crazy-max Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

crazy-max commented Aug 4, 2026 •

edited

Loading

thaJeztah Aug 4, 2026 •

edited

Loading