You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Investigate intermittent Docker dynamic host-port collisions in integration CI #136
failed to bind host port for 0.0.0.0::172.17.0.19:9092/tcp: address already in use
tests/Relayer.Kafka.Tests/KafkaBrokerTests.cs:22-23 on current origin/main still uses KafkaBuilder's default dynamic host-port allocation. An isolated Docker 28.0.4 daemon reproduces the exact failure with ten loopback listeners occupying 32768-32777: the allocator exhausts ten bind attempts. This mechanism is tracked by moby/moby#54079. The destroyed hosted runners' socket owners are unknown; the reproduction establishes the mechanism, not those runners' exact socket state.
Remaining work
Open PR #137 implements the Linux Kafka mitigation: Docker selects and reserves both listener ports within 20000-29999, outside the default outbound ephemeral range. Windows retains its existing mapping. The PR reports eight passing Kafka tests on both Windows and Linux, including the Linux reproduction with occupied ports. The fix has not merged, so this issue remains open and in-progress remains valid.
Main run 41330218030, attempt 1 failed before Service Bus assertions while its SQL Server dependency published 1433/tcp. tests/Relayer.Tests/ServiceBusBindingEmulatorTests.cs:16-20 uses ServiceBusBuilder without a fixed host-port binding. Attempt 2 completed successfully. This supports an intermittent container-startup failure but does not prove the Kafka reproduction is the SQL Server failure's root cause. #137 changes only Kafka setup; it does not claim to fix Service Bus setup. No run wait or hold remains for that completed rerun.
The same Docker startup failure occurred in PR #124's CI run 41329127537, attempt 1. The Kafka test failed at KafkaBrokerTests.cs:23, during broker.StartAsync(), before broker assertions. Docker reported failed to bind host port for 0.0.0.0::172.17.0.19:9092/tcp: address already in use. The other 716 tests passed, and the performance regression gate passed.
The test uses the default KafkaBuilder dynamic host-port allocation, with no fixed host port. This is additional evidence of the runner/container startup problem tracked here. PR #124 also has an independent detector review fix, so it returns to draft and will receive fresh CI after review; the failed run is not rerun to seek green.
changed the title [-]Investigate intermittent Docker host-port collision in Kafka integration CI[/-][+]Investigate intermittent Docker dynamic host-port collisions in integration CI[/+]on Nov 1, 2026
The latest push CI on main, 41330218030, attempt 1, failed during the Service Bus emulator's SQL Server container startup. This broadens the affected fixtures beyond Kafka. The Build & Test / Test step reported:
failed to bind host port for 0.0.0.0::172.18.0.2:1433/tcp: address already in use
tests/Relayer.Tests/ServiceBusBindingEmulatorTests.cs:16-20 uses the default ServiceBusBuilder configuration and no fixed host-port binding. The exception occurs in DockerContainer.StartAsync, before client creation or settlement assertions. The runner reports Docker 28.0.4 and Ubuntu 24.04.5 LTS. The other 733 tests passed, including Kafka and PostgreSQL integration tests. These observations support a container-startup infrastructure failure; they do not establish the allocator's root cause or prove this is identical to the Kafka failure.
I requested the one whole-run infrastructure rerun permitted by the red-main procedure. Attempt 1 had not previously been rerun. Do not repeatedly rerun this head if attempt 2 reproduces the failure; investigate it as a real failure under this issue. No tests were skipped or changed. Failed-job logs are retained at C:/git/Relayer-evidence/main-ci/run-41330218030-failed.log under the normal seven-day evidence retention policy.
The earlier red-main blocker has cleared: #132 merged as 300e288, and run 41327937423 completed successfully. This new failure is on 668d1229edc6842fcce539a1b89cf23a895c060b. This issue remains open because the investigation and targeted-fix acceptance criteria are not yet satisfied; no hold label or dependency on #132 is needed.
PR #134's only allowed rerun also fails with the identical Kafka startup error (run 41329961992, attempt 2); no further rerun is requested. Working #136 as the blocker under its own lock.
A fresh isolated Docker 28.0.4 daemon reproduces the exact error deterministically when ten ordinary loopback listeners occupy 32768-32777. Its log records ten EADDRINUSE failures. This matches the upstream allocator defect already tracked by moby/moby#54079. It establishes a mechanism for the CI failure, not the identity of the sockets on the now-destroyed hosted runner.
The same daemon successfully publishes both Kafka ports from 20000-29999 while the ten listeners remain open. The fix keeps Linux Kafka test publications outside the default outbound ephemeral range, leaves Docker selecting and reserving ports, and adds no retries or skipped broker assertions. Windows keeps its default mapping because Docker Desktop must also respect Windows excluded ports.
Activity
thomhurst commentedon Nov 1, 2026
The same Docker startup failure occurred in PR #124's CI run 41329127537, attempt 1. The Kafka test failed at
KafkaBrokerTests.cs:23, duringbroker.StartAsync(), before broker assertions. Docker reportedfailed to bind host port for 0.0.0.0::172.17.0.19:9092/tcp: address already in use. The other 716 tests passed, and the performance regression gate passed.The test uses the default KafkaBuilder dynamic host-port allocation, with no fixed host port. This is additional evidence of the runner/container startup problem tracked here. PR #124 also has an independent detector review fix, so it returns to draft and will receive fresh CI after review; the failed run is not rerun to seek green.
[-]Investigate intermittent Docker host-port collision in Kafka integration CI[/-][+]Investigate intermittent Docker dynamic host-port collisions in integration CI[/+]thomhurst commentedon Nov 1, 2026
The latest push CI on main, 41330218030, attempt 1, failed during the Service Bus emulator's SQL Server container startup. This broadens the affected fixtures beyond Kafka. The Build & Test / Test step reported:
tests/Relayer.Tests/ServiceBusBindingEmulatorTests.cs:16-20uses the default ServiceBusBuilder configuration and no fixed host-port binding. The exception occurs inDockerContainer.StartAsync, before client creation or settlement assertions. The runner reports Docker 28.0.4 and Ubuntu 24.04.5 LTS. The other 733 tests passed, including Kafka and PostgreSQL integration tests. These observations support a container-startup infrastructure failure; they do not establish the allocator's root cause or prove this is identical to the Kafka failure.I requested the one whole-run infrastructure rerun permitted by the red-main procedure. Attempt 1 had not previously been rerun. Do not repeatedly rerun this head if attempt 2 reproduces the failure; investigate it as a real failure under this issue. No tests were skipped or changed. Failed-job logs are retained at
C:/git/Relayer-evidence/main-ci/run-41330218030-failed.logunder the normal seven-day evidence retention policy.The earlier red-main blocker has cleared: #132 merged as
300e288, and run 41327937423 completed successfully. This new failure is on668d1229edc6842fcce539a1b89cf23a895c060b. This issue remains open because the investigation and targeted-fix acceptance criteria are not yet satisfied; no hold label or dependency on #132 is needed.thomhurst commentedon Nov 1, 2026
PR #134's only allowed rerun also fails with the identical Kafka startup error (run 41329961992, attempt 2); no further rerun is requested. Working #136 as the blocker under its own lock.
A fresh isolated Docker 28.0.4 daemon reproduces the exact error deterministically when ten ordinary loopback listeners occupy 32768-32777. Its log records ten EADDRINUSE failures. This matches the upstream allocator defect already tracked by moby/moby#54079. It establishes a mechanism for the CI failure, not the identity of the sockets on the now-destroyed hosted runner.
The same daemon successfully publishes both Kafka ports from 20000-29999 while the ten listeners remain open. The fix keeps Linux Kafka test publications outside the default outbound ephemeral range, leaves Docker selecting and reserving ports, and adds no retries or skipped broker assertions. Windows keeps its default mapping because Docker Desktop must also respect Windows excluded ports.
fix: separate Linux Kafka test ports from ephemeral sockets (#136) (#137