Repository navigation
[serge] integration failure triage - 2026-10-26 #50005
Description
Automated integration-failure triage for the daily CI window 2026-10-06 → 2026-10-26.
This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.
Serge dispatched one task per failure group below — each opens or updates its own PR on a serge/fix/itf-<fingerprint> branch. This table is refreshed in place as Serge runs: a group links its #<pr> when opened, shows 🚫 no fix when Serge found no safe change, ⚠️ task failed on error, or (pending) while still running (a late PR links on the next nightly run).
The Task column links Serge's own page for the run — the steps it went through, what each tool cost, and the full task text. 🔒 means it is not public: the host is VPN-internal and the page needs a Serge login. A task links once Serge has accepted it, so the column fills in on the first refresh after dispatch.
Dispatched failure groups
| Model | Error | Occurrences | PR | Task |
|---|---|---|---|---|
peft_integration |
other — other (8) | 8 | 🔒 task | |
kosmos2 |
output_mismatch — tensor values differ (2), other (2), list output differs (2) | 6 | 🚫 no fix | 🔒 task |
moshi |
load_error — other (6) | 6 | 🚫 no fix | 🔒 task |
blt |
import_or_config — other (4) | 4 | 🚫 no fix | 🔒 task |
deepseek_v4 |
other — other (4) | 4 | 🚫 no fix | 🔒 task |
deepseek_vl_hybrid |
OOM — other (2) | 2 | #49581 | 🔒 task |
hrm_text |
output_mismatch — tensor values differ (2) | 2 | #50006 | 🔒 task |
kimi_k25 |
cuda_runtime — other (2) | 2 | 🚫 no fix | 🔒 task |
deepseek_vl_hybrid |
cuda_runtime — other (1) | 1 | #49335 | 🔒 task |
qwen3_vl_moe |
OOM — other (1) | 1 | 🔒 task |
Not dispatched — environment / dependency
These groups were triaged but NOT handed to Serge: their failure mode is a property of the runner or the environment, so no minimal source patch can fix them. They need a human (runner capacity, a dependency pin).
2 models ran out of device memory (5 failures) — needs runner capacity, not a patch, so none of these were dispatched: llama4 (3), exaone4_5 (2).
6 group(s) skipped — stopped failing before this run (10 integration tests regressed by commit 92cd495f2720 (PR #49364), 2 integration tests regressed by commit c48824e84024 (PR #49404), minicpmv4_7, minicpm3 +2 more). Not dispatched and not awaiting a human.
Outcome recap
Why each group that opened no PR ended without one, and what it cost — surfaced here from the Serge dashboard. Failing tests links straight to each test's dashboard view, since these are the groups a human has to pick up.
| Group | Reason | LLM | Tokens (in / out) | Failing tests |
|---|---|---|---|---|
peft_integration |
LLM returned unparseable output (finish_reason=stop, 11 LLM turns · 11 tool calls · 1006.7s · 316342 in / 21051 out tokens, 1 salvage re-ask(s) did not recover it) | Qwen/Qwen3.8-27B:deepinfra |
316,342 / 21,051 | test_hotswap_with_compile_and_higher_rank_works · test_hotswap_with_compile_and_lower_rank_works · test_hotswap_without_compile_and_with_higher_rank_works · +1 more |
kosmos2 |
The 6 kosmos2 integration tests fail with large, non-drift differences: the vision-model slice differs by up to 1.33 absolute (44% of elements), and the generated captions are entirely different sentences ("A snowman is sitting in front of a fire..." vs the expected "The image features a snowman sitting by a campfire..."), with the logit for token id 1 collapsing to exactly 0.0 in every actual score row. That is a behavioral change in the image-feature path, not hardware drift (~0.5% precedent), so recording the new values would enshrine a regression. I could not pin the root cause from the available evidence, so I am producing no patch rather than a false green. | Qwen/Qwen3.8-27B:deepinfra |
249,114 / 28,029 | test_inference_interpolate_pos_encoding · test_snowman_image_captioning · test_snowman_image_captioning_batch |
moshi |
[reproduced] GPU reproduce + classify: this is an ENVIRONMENT issue (The traceback shows the Hub repository kmhf/hf-moshiko cannot be resolved/downloaded, which is a missing/renamed/gated checkpoint environment issue rather than a library or test-expectation bug.) — no source patch can fix it, so no investigation and no PR. |
Qwen/Qwen3.8-27B:deepinfra |
— / — | test_moshiko_greedy_unconditional_fp16 · test_moshiko_greedy_unconditional_fp16_eager · test_moshiko_greedy_unconditional_fp32 |
blt |
[not_fixed] GPU verification did not confirm the fix (not_fixed): the patch did not turn the targeted tests green. |
Qwen/Qwen3.8-27B:deepinfra |
1,485,800 / 44,103 | test_model_logits · test_model_logits_bf16 |
deepseek_v4 |
[timeout] A candidate patch was committed to serge/fix/itf-db679591e1c4-fea01d20 but GPU verification could not run (timeout) (https://github.com/huggingface/transformers/actions/runs/40420362642), so it is UNVERIFIED and no PR was opened. The branch is kept: re-run verification against it, or delete it if the group has moved on. — unverified patch kept on serge/fix/itf-db679591e1c4-fea01d20 |
Qwen/Qwen3.8-27B:deepinfra |
334,501 / 6,458 | test_v4_flash_dequantized_chat_seven_prompts · test_v4_flash_dequantized_generation |
kimi_k25 |
[not_fixed] GPU verification did not confirm the fix (not_fixed): the patch did not turn the targeted tests green. |
Qwen/Qwen3.8-27B:deepinfra |
679,553 / 84,887 | test_model_logits · test_model_logits_batched |
qwen3_vl_moe |
LLM returned unparseable output (finish_reason=stop, 12 LLM turns · 10 tool calls · 905.7s · 236734 in / 24305 out tokens, 1 salvage re-ask(s) did not recover it) | Qwen/Qwen3.8-27B:deepinfra |
236,734 / 24,305 | test_small_model_integration_test_batch |
Unverified branches — how to finish them
Serge committed a patch for each group below but the GPU gate never judged it, so no PR was opened and the branch was kept. Check the already-dispatched run before re-running: it often completes after Serge stops waiting, and reading its artifact is free where a re-run is not.
deepseek_v4 — unverified patch on serge/fix/itf-db679591e1c4-fea01d20 (a3b9f7448f4c)
# 1. The verify run Serge dispatched may have finished after it gave up —
# check that run (URL in the Reason cell above) before spending GPU time:
# gh run download <run-id> -R huggingface/transformers -n serge-verify-result-aws-g5-12xlarge-cache
# 2. Only if no verdict exists, re-run it:
gh workflow run serge-verify-caller.yml -R huggingface/transformers \
-f mode=verify \
-f base_sha=$(gh api repos/huggingface/transformers/commits/a3b9f7448f4ce91af2b120ba877bbdf1c170e314 --jq '.parents[0].sha') \
-f commit_sha=a3b9f7448f4ce91af2b120ba877bbdf1c170e314 \
-f test_nodeids='tests/models/deepseek_v4/test_modeling_deepseek_v4.py::DeepseekV4IntegrationTest::test_v4_flash_dequantized_chat_seven_prompts tests/models/deepseek_v4/test_modeling_deepseek_v4.py::DeepseekV4IntegrationTest::test_v4_flash_dequantized_generation' \
-f model=deepseek_v4 \
-f machine_type=aws-g5-12xlarge-cacheGenerated 2026-10-27T02:38:25+00:00.
Activity
github-actions commentedon Oct 27, 2026
Superseded by #50048 — closing this stale nightly integration-failure triage issue. Open PRs stay linked from the current issue.