Remote gRPC MIP solve: trailing log lines dropped by StreamLogs, causing test_cli_mip_remote to fail with objective=nan #1510

Description

TestCuoptCliCPUOnly::test_cli_mip_remote in python/cuopt/cuopt/tests/linear_programming/test_cpu_only_execution.py failed on CI:

CI run: https://github.com/NVIDIA/cuopt/actions/runs/28603627708/job/84825516658?pr=1503
Job: conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps

AssertionError: Objective nan differs from expected 2.0 (rel error nan)
assert nan < 0.01
tests/linear_programming/test_cpu_only_execution.py:642: AssertionError

The failure occurred on PR #1503, which only touches documentation/Doxygen/Sphinx branding (cpp/doxygen/Doxyfile, cpp/include/cuopt/mathematical_optimization/cuopt_c.h comment, docs/cuopt/source/conf.py, and a logo image) — nothing related to the CLI, gRPC server, or MIP solver. So this is a pre-existing bug/flaky test unrelated to that PR's changes, surfaced incidentally.

Root cause

cuopt_cli run in remote-execution mode (CUOPT_REMOTE_HOST/CUOPT_REMOTE_PORT set) forwards the MIP solve to cuopt_grpc_server over gRPC. The client has no independent "objective summary" print for MIP — it depends entirely on log-line relay from the server via StreamLogs:

• Server side: solve_mip_helper() (cpp/src/mip_heuristics/solve.cu:753) calls sol.log_detailed_summary(), which emits the "Best objective {}, best bound {}, gap {}%." line (cpp/src/mip_heuristics/solver_solution.cu:239-250) — this is the line the CLI/test depends on to report the objective. Immediately afterward (solve.cu:755-756), if settings.sol_file != "" (set by cuopt_cli), a second line "Writing solution to file %s" is logged, right before the job is marked COMPLETED.
• StreamLogs (cpp/src/grpc/server/grpc_service_impl.cpp:822-847) tails the per-job log file. Once it detects the job status has flipped to COMPLETED/FAILED/CANCELLED, it performs exactly one extra getline() call ("read once more before closing the stream") to catch a line written after its last EOF poll, then sends the completion sentinel and returns.
• If two or more lines are flushed to the log file in the window between the tailer's last EOF poll and the completion-status check (which is the normal case for MIP: "Best objective..." followed by "Writing solution to file..."), only the first of those lines is guaranteed to be picked up — any additional line(s) can be silently dropped. This matches the observed symptom exactly: status in _parse_cli_output was correctly parsed as "Optimal", but the "Best objective ..." line (or whichever line lost the race) never reached the client, leaving objective_value at its nan default.

This is a genuine race condition in the log-relay path (grpc_service_impl.cpp's StreamLogs), not a test-authoring bug. It is inherently flaky — timing-dependent on how many lines the worker flushes vs. how quickly the tailer polls job status after reaching EOF.

Suggested fix

In StreamLogs, after detecting a terminal job status, loop reading all remaining lines until EOF (rather than a single getline()) before sending the completion sentinel, e.g.:

while (std::getline(in, line)) {
  ...
  if (!write_log_message(line, next_byte_offset, false)) { break; }
}

Reproduction

Not easily reproducible locally without a GPU + cuopt_grpc_server binary in a CPU-only remote-execution test environment; the race window is narrow, which is consistent with this test passing on most CI runs and failing intermittently. It failed on this specific run of PR #1503's CI, on the conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4, latest-driver, latest-deps job.

Environment

• CI job: https://github.com/NVIDIA/cuopt/actions/runs/28603627708/job/84825516658?pr=1503
• Test: python/cuopt/cuopt/tests/linear_programming/test_cpu_only_execution.py::TestCuoptCliCPUOnly::test_cli_mip_remote