Repository navigation
[Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs #889
Description
Search before asking
- I searched in the issues and found nothing similar.
Description
The it-java and it-python CI jobs fail intermittently for reasons unrelated to the code under test. A recent run shows both failure modes at once, with every unit-test job green.
1. The Ollama install step fails on a transient network blip. tools/start_ollama_server.sh pipes https://ollama.com/install.sh into sh. That upstream script probes for ollama-linux-${ARCH}.tar.zst with a curl --fail --silent --head, and on any failure of that single probe it falls through to a .tgz asset. Current releases publish only .tar.zst — .tgz returns 404. So one flaky HEAD request turns into a hard failure:
>>> Downloading ollama-linux-amd64.tgz
curl: (22) The requested URL returned error: 404
gzip: stdin: unexpected end of file
tar: Child returned status 1
##[error]Process completed with exit code 2.
Two it-java jobs failed this way, before compiling or running a single test. Retrying the install would recover, since the underlying asset does exist.
2. The wrapper masks a failed download. In tools/start_ollama_server.sh, ret=$? is read after curl -fsSL ... | sh, so it captures the exit status of sh, not curl. Without pipefail, a failed curl leaves sh reading empty stdin and exiting 0 — CI then proceeds with no Ollama installed and fails later somewhere less obvious.
3. Integration jobs have no runtime bound. No job in ci.yml sets timeout-minutes. In the run above, it-python [python-3.12] [flink-2.2] hung and was killed at GitHub's 6-hour ceiling (6h0m16s), consuming shared runner capacity for a result that was knowable in minutes. A timeout-minutes on the integration jobs would surface the same failure far sooner.
Open question — the hang itself. In that job Ollama installed correctly, then llama-server crashed while serving qwen3:1.7b:
ollama._types.ResponseError: llama-server process has terminated: signal: segmentation fault (core dumped)
Every failing test there is Ollama-backed; the py4j errors are downstream of the dead model server. I don't yet have evidence for the root cause (upstream regression, runner memory, or model artifact). One option is pinning OLLAMA_VERSION — install.sh supports it, and the workflow already pins uv to 0.11.0 in the adjacent step — which would at least make the toolchain deterministic. I'd welcome opinions on whether that's worth doing, and on how the project wants to handle bumping such a pin.
If this direction sounds right, I'd propose splitting the work into two independent PRs: the retry plus pipefail fix in tools/start_ollama_server.sh, and the timeout-minutes addition in ci.yml.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
weiqingy commentedon Jul 10, 2026
Will work on this.
weiqingy commentedon Jul 10, 2026
Status map for anyone picking this up, since the work has spread across a few PRs and a sibling issue.
This issue produced two independent PRs:
.tgz404).timeout-minutesso a hung job fails fast instead of running to the 6-hour ceiling.Both currently show red CI, but the red is not these changes. The Ollama install succeeds in every failing run; only the Ollama-backed integration tests crash, and the failing jobs roam from run to run while the changes stay the same. That intermittent flake is tracked separately in #893.
Root cause (detail in #893): a subset of GitHub
ubuntu-latestrunner VMs trip an upstreamllama.cppsegmentation fault while servingqwen3:1.7bon CPU. A given VM is effectively deterministic, so re-running the whole matrix keeps hitting some cursed VM. The dated onset lines up with a GitHub runner-image rollout — and notably the installed Ollama version has been constant across the green→red boundary, so pinning the Ollama version does not apply here.The fix is #895: a
workflow_runcompanion that re-runs only the failed integration-test legs on a fresh runner (which re-rolls the VM), capped at two attempts, with unit/lint/build failures never re-run. It was validated end-to-end on a fork. One caveat noted on that PR:workflow_runcompanions only run from the default branch, so #895 cannot heal its own CI and takes effect only once merged.Suggested order, entirely at the maintainers' discretion: #895 first — once merged it heals the integration-test flake for every subsequent run, including #891 and #892 — then #891/#892, which are otherwise green apart from the flake. They can also be reviewed independently. I'll leave the sequencing and the decisions to the maintainers.
weiqingy commentedon Jul 10, 2026
@wenjin272 When you get a chance, could you take a look at this?
CI has been failing more often recently, so it would be good to mitigate this to make PR creation, validation, and review smoother. My suggestion is to prioritize #895 first, since the fresh-runner auto-rerun would unblock the flaky integration-test jobs and give us the most immediate impact.
The map above has the full context.
wenjin272 commentedon Jul 13, 2026
@weiqingy Regarding issue #895, I suggest we first evaluate whether we can resolve the llama server segmentation fault before deciding if launching a new runner for retries is necessary when CI fails.
weiqingy commentedon Jul 14, 2026
Agreed, a real fix beats a retry, so let's hold #895 until we know whether the segfault can be resolved at the source.
I've put the details on #893. The hardware lead looks right, but
OLLAMA_LLM_LIBRARYwon't get us there on 0.31.x, since it only filters GPU library directories now. The CPU kernel is chosen byllama-server, which loads the sapphirerapids ggml variant, the only one built with AMX, on those hosts. So the fix would be to stop that variant loading, intools/start_ollama_server.sh.Next step I'd suggest is a small diagnostic in the integration jobs to log the CPU flags and the loaded variant, so we can confirm it rather than guess. If it holds, I'll put up the fix and close #895.