Description
The core E2E suite spec Group Remove E2E Tests > thv rm --group command > should remove workloads from group (test/e2e/rm_group_test.go:74) intermittently fails in CI, timing out while waiting for a freshly-started workload to reach the running state before it can be removed. This is a test-infrastructure/timing flake (container startup / image pull under CI load), not a product regression.
Observed failure
Seen on the E2E Tests / E2E Tests Core (core) job:
[FAIL] Group Remove E2E Tests thv rm --group command [It] should remove workloads from group [core, groups, e2e]
test/e2e/rm_group_test.go:87
timeout waiting for MCP server rm-group-workload-1-1785152373 to be running
Ran 60 of 506 Specs in 588.365 seconds
FAIL! -- 59 Passed | 1 Failed | 0 Pending | 446 Skipped
--- FAIL: TestE2e (588.39s)
- Failing job: https://github.com/stacklok/toolhive/actions/runs/30262060868/job/89965993534
- The spec waits for
rm-group-workload-1-<ts> to become running (via the shared WaitForMCPServer helper), then exercises thv rm --group. The workload never reached running within the timeout.
- All other jobs in the same run were green, and the failure reproduced on a PR that does not touch group/workload lifecycle code — so it is not caused by the change under test.
Suspected cause
Workload startup (image pull + container start) occasionally exceeds the per-workload readiness timeout used by this spec, especially on the workload-heavy core matrix entry under CI load. Same family as previously observed E2E workload-startup flakes.
Suggested fix directions
- Increase / make configurable the readiness timeout in
rm_group_test.go (and/or the shared WaitForMCPServer helper) for the workload-heavy specs.
- Consider pre-pulling the backend image in
BeforeAll/suite setup so per-spec timing excludes the pull.
- Add a short retry around the "wait for running" step before failing.
Prior art (all closed, same family)
Notes
Surfaced during review of #6006; unrelated to that PR's changes (every check exercising that PR passed — E2E Core (vmcp), MCP Conformance (vmcp), all Operator CI tiers). Filed so a re-run isn't the only record of the failure.
Description
The core E2E suite spec
Group Remove E2E Tests > thv rm --group command > should remove workloads from group(test/e2e/rm_group_test.go:74) intermittently fails in CI, timing out while waiting for a freshly-started workload to reach the running state before it can be removed. This is a test-infrastructure/timing flake (container startup / image pull under CI load), not a product regression.Observed failure
Seen on the
E2E Tests / E2E Tests Core (core)job:rm-group-workload-1-<ts>to become running (via the sharedWaitForMCPServerhelper), then exercisesthv rm --group. The workload never reached running within the timeout.Suspected cause
Workload startup (image pull + container start) occasionally exceeds the per-workload readiness timeout used by this spec, especially on the workload-heavy
corematrix entry under CI load. Same family as previously observed E2E workload-startup flakes.Suggested fix directions
rm_group_test.go(and/or the sharedWaitForMCPServerhelper) for the workload-heavy specs.BeforeAll/suite setup so per-spec timing excludes the pull.Prior art (all closed, same family)
Notes
Surfaced during review of #6006; unrelated to that PR's changes (every check exercising that PR passed —
E2E Core (vmcp),MCP Conformance (vmcp), all Operator CI tiers). Filed so a re-run isn't the only record of the failure.