Skip to content

Investigate intermittent Docker dynamic host-port collisions in integration CI #136

Closed

Description

@thomhurst

Investigated failure

Kafka broker startup failed before assertions in PR #134's run 41329961992, attempts 1 and 2 and PR #124's run 41329127537. Docker 28.0.4 on Ubuntu 24.04.5 reported:

failed to bind host port for 0.0.0.0::172.17.0.19:9092/tcp: address already in use

tests/Relayer.Kafka.Tests/KafkaBrokerTests.cs:22-23 on current origin/main still uses KafkaBuilder's default dynamic host-port allocation. An isolated Docker 28.0.4 daemon reproduces the exact failure with ten loopback listeners occupying 32768-32777: the allocator exhausts ten bind attempts. This mechanism is tracked by moby/moby#54079. The destroyed hosted runners' socket owners are unknown; the reproduction establishes the mechanism, not those runners' exact socket state.

Remaining work

Open PR #137 implements the Linux Kafka mitigation: Docker selects and reserves both listener ports within 20000-29999, outside the default outbound ephemeral range. Windows retains its existing mapping. The PR reports eight passing Kafka tests on both Windows and Linux, including the Linux reproduction with occupied ports. The fix has not merged, so this issue remains open and in-progress remains valid.

Acceptance criteria:

Related Service Bus observation

Main run 41330218030, attempt 1 failed before Service Bus assertions while its SQL Server dependency published 1433/tcp. tests/Relayer.Tests/ServiceBusBindingEmulatorTests.cs:16-20 uses ServiceBusBuilder without a fixed host-port binding. Attempt 2 completed successfully. This supports an intermittent container-startup failure but does not prove the Kafka reproduction is the SQL Server failure's root cause. #137 changes only Kafka setup; it does not claim to fix Service Bus setup. No run wait or hold remains for that completed rerun.

Activity

  1. added
    bugSomething isn't working
    area: testingTest harness, in-memory transport, conformance
    P2Medium priority
    on Nov 1, 2026
  2. thomhurst commented on Nov 1, 2026

    @thomhurst
    OwnerAuthor

    The same Docker startup failure occurred in PR #124's CI run 41329127537, attempt 1. The Kafka test failed at KafkaBrokerTests.cs:23, during broker.StartAsync(), before broker assertions. Docker reported failed to bind host port for 0.0.0.0::172.17.0.19:9092/tcp: address already in use. The other 716 tests passed, and the performance regression gate passed.

    The test uses the default KafkaBuilder dynamic host-port allocation, with no fixed host port. This is additional evidence of the runner/container startup problem tracked here. PR #124 also has an independent detector review fix, so it returns to draft and will receive fresh CI after review; the failed run is not rerun to seek green.

  3. changed the title [-]Investigate intermittent Docker host-port collision in Kafka integration CI[/-] [+]Investigate intermittent Docker dynamic host-port collisions in integration CI[/+] on Nov 1, 2026
  4. thomhurst commented on Nov 1, 2026

    @thomhurst
    OwnerAuthor

    The latest push CI on main, 41330218030, attempt 1, failed during the Service Bus emulator's SQL Server container startup. This broadens the affected fixtures beyond Kafka. The Build & Test / Test step reported:

    failed to bind host port for 0.0.0.0::172.18.0.2:1433/tcp: address already in use
    

    tests/Relayer.Tests/ServiceBusBindingEmulatorTests.cs:16-20 uses the default ServiceBusBuilder configuration and no fixed host-port binding. The exception occurs in DockerContainer.StartAsync, before client creation or settlement assertions. The runner reports Docker 28.0.4 and Ubuntu 24.04.5 LTS. The other 733 tests passed, including Kafka and PostgreSQL integration tests. These observations support a container-startup infrastructure failure; they do not establish the allocator's root cause or prove this is identical to the Kafka failure.

    I requested the one whole-run infrastructure rerun permitted by the red-main procedure. Attempt 1 had not previously been rerun. Do not repeatedly rerun this head if attempt 2 reproduces the failure; investigate it as a real failure under this issue. No tests were skipped or changed. Failed-job logs are retained at C:/git/Relayer-evidence/main-ci/run-41330218030-failed.log under the normal seven-day evidence retention policy.

    The earlier red-main blocker has cleared: #132 merged as 300e288, and run 41327937423 completed successfully. This new failure is on 668d1229edc6842fcce539a1b89cf23a895c060b. This issue remains open because the investigation and targeted-fix acceptance criteria are not yet satisfied; no hold label or dependency on #132 is needed.

  5. thomhurst commented on Nov 1, 2026

    @thomhurst
    OwnerAuthor

    PR #134's only allowed rerun also fails with the identical Kafka startup error (run 41329961992, attempt 2); no further rerun is requested. Working #136 as the blocker under its own lock.

    A fresh isolated Docker 28.0.4 daemon reproduces the exact error deterministically when ten ordinary loopback listeners occupy 32768-32777. Its log records ten EADDRINUSE failures. This matches the upstream allocator defect already tracked by moby/moby#54079. It establishes a mechanism for the CI failure, not the identity of the sockets on the now-destroyed hosted runner.

    The same daemon successfully publishes both Kafka ports from 20000-29999 while the ten listeners remain open. The fix keeps Linux Kafka test publications outside the default outbound ephemeral range, leaves Docker selecting and reserving ports, and adds no retries or skipped broker assertions. Windows keeps its default mapping because Docker Desktop must also respect Windows excluded ports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priorityarea: testingTest harness, in-memory transport, conformancebugSomething isn't workingin-progressClaimed by an agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions