Flaky: remote-L3 sim tests connect after a fixed 0.3s sleep, not on the daemon being ready #1508

Five tests in tests/ut/py/test_callable_identity.py start a simpler.remote_l3_worker subprocess on a free TCP port, time.sleep(0.3), then connect. Under load the daemon is not listening yet and the connect is refused.

Observed

ut (macos-latest, 3.10) on PR #1466 head c5c50ca6:

FAILED tests/ut/py/test_callable_identity.py::test_remote_sim_buffer_copy_roundtrip
  - ConnectionRefusedError: [Errno 61] Connection refused

and locally on aarch64 in a full pytest tests/ut run, a sibling in the same family:

FAILED tests/ut/py/test_callable_identity.py::test_remote_sim_health_lane_stays_live_during_long_task
  - ConnectionRefusedError: [Errno 111] Connection refused
  python/simpler/worker.py:2486 sock.connect(sockaddr)

Both cleared on re-run. Each test passes 6/6 when run alone on both main and the PR branch, so the trigger is contention during the full suite, not the code under test.

Cause

daemon = subprocess.Popen([sys.executable, "-m", "simpler.remote_l3_worker", ...])
worker = Worker(level=4, ...)
try:
    time.sleep(0.3)          # <- hope the daemon bound the port
    worker_id = worker.add_remote_worker(...)

0.3s is a guess at interpreter start + import + bind() + listen(). It holds on an idle machine and fails when the runner is busy. The five call sites are at test_callable_identity.py:1026, 1059, 1094, 1128, 1161.

Note the failure surfaces inside Worker._connect_within_deadline, whose own bounded-deadline logic is working correctly — it reports the refusal immediately because the port is closed, not because the deadline expired. Raising the worker-side timeout would not help; nothing is listening yet.

Suggested fix

Wait for readiness rather than for a duration: poll connect() on the port until it succeeds or a generous deadline expires, and fail with the daemon subprocess's captured output if it exits first. A small shared helper would cover all five call sites.

This is the same class as #1504 (that test guessed at how to identify children; these guess at when a daemon is up) — replacing the guess removes the failure mode rather than widening the margin.