driver/kubernetes: skip terminating pods when choosing a pod

[chagui]

We noticed in our production environment that some builds are routed to terminating pods. A build routed to such a pod is killed mid-flight when the grace period expires, and the client doesn't fail fast in that case, so it hangs until the caller's own timeout (Refs #556). In CI that means the job burns its whole time budget instead of failing and retrying elsewhere.
Even with recent fixes for #556 we would benefit from not sending builds to terminating pods in the first place.

ListRunningPods currently filters on pod.Status.Phase alone. A pod marked for deletion keeps Phase=Running for its entire termination grace period, so it stays selectable right up until its containers are killed. Our set-up configure a preStop hook with a 900secs grade period for the buildkitd container.

Skipping pods with a DeletionTimestamp matches how upstream Kubernetes decides set membership: IsPodActive in pkg/controller/controller_utils.go pairs the phase checks with DeletionTimestamp == nil. This doesn't close the race entirely, since a pod can be deleted between the List and the exec, but it shrinks the window from minutes to sub-second.

Added unit tests for the pod filtering and both pod choosers; the package had none.

Repro / testing

Setup:

# create kind cluster
kind create cluster --name buildkit-terminating

# create a builder
docker buildx create --driver kubernetes --name kindbuilder \
  --driver-opt namespace=default,replicas=2,loadbalance=random --bootstrap

# buildx sets no terminationGracePeriodSeconds, so pods default to 30s.
# Widen it so a terminating pod stays observable.
kubectl patch deploy kindbuilder0 \
  -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":900}}}}'
kubectl rollout status deploy/kindbuilder0

Create a Dockerfile which takes some time, eg.:

FROM alpine:latest
RUN echo "step1 starting" && sleep 900 && echo "step1 done"

Start a build and leave it running. This keeps one pod busy, which is what makes it linger later: GracefulStop waits for the in-flight Solve instead of exiting on SIGTERM.
docker buildx build --builder kindbuilder --progress=plain -f Dockerfile .

Once step1 starting appears, find the busy pod and delete only that one. Delete an idle pod instead and buildkitd exits immediately, so nothing lingers.

for p in $(kubectl get pods -l app=kindbuilder0 -o name); do
  echo "$p busy=$(kubectl exec $p -- ps aux 2>/dev/null | grep -c '[s]leep 900')"
done

kubectl delete pod <the busy one> --wait=false

# wait for the replacement, so ReadyReplicas >= replicas and Bootstrap passes
kubectl wait --for=condition=available --timeout=120s deploy/kindbuilder0

kubectl get pods -l app=kindbuilder0 \
  -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,DEL:.metadata.deletionTimestamp'
# NAME                            PHASE     DEL
# kindbuilder0-6bb75db655-5nq7s   Running   <none>
# kindbuilder0-6bb75db655-rprcs   Running   <none>
# kindbuilder0-74d679dd75-nw96b   Running   2026-08-05T10:10:19Z   <- terminating, still Running

Now run a second build and look at the candidate list.
Note: I used a custom binary with log statement patched to Info level because the debug level was not reliable.

Before, the terminating pod is a valid candidate:

pod running: "kindbuilder0-6bb75db655-5nq7s"
pod running: "kindbuilder0-6bb75db655-rprcs"
pod running: "kindbuilder0-74d679dd75-nw96b"
RandomPodChooser.ChoosePod(): len(pods)=3

With this PR, against the same three pods:

pod running: "kindbuilder0-6bb75db655-5nq7s"
pod running: "kindbuilder0-6bb75db655-rprcs"
pod terminating, skipping: "kindbuilder0-74d679dd75-nw96b"
RandomPodChooser.ChoosePod(): len(pods)=2

[crazy-max]

LGTM thanks!

PTAL @AkihiroSuda

[slonopotamus]

This doesn't close the race entirely, since a pod can be deleted between the List and the exec, but it shrinks the window from minutes to sub-second.

So, the bug is still there but it will be order of magnitudes harder to reproduce and debug it? =/

[chagui]

This doesn't close the race entirely, since a pod can be deleted between the List and the exec, but it shrinks the window from minutes to sub-second.

So, the bug is still there but it will be order of magnitudes harder to reproduce and debug it? =/

In our case, this takes the window from 900 seconds down to sub-second, which significantly reduces the chance of targeting a terminating pod. From what we observed, the builds getting killed are not the ones being scheduled just after a pod is marked as terminating but the ones getting scheduled on terminating pods for which the remaining grace period is short (e.g. under 30 seconds). Builds scheduled in the sub-second window would be fine.