[trtllm-agent]

Description

Temporarily waive 15 failed case(s) on DGX_H100. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_allows_guided_context_logits_on_graph_hit broken on main for DGX_H100: failure signature tensorrt_llm.bindings.internal.batch_manager.kv_ca....hostoomerror: mmap failed: cannot allocate memory, onset 2026-11-05T07:53:46Z, latest failure 2026-11-05T19:23:27Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7053210 tracks the fix. Scope DGX_H100 is the stage prefix that ran the failing test.

• Bug: https://nvbugs/7053210 (details, logs and 7-day test history are in the bug)
• Restore coverage when: Remove this line when https://nvbugs/7053210 is fixed or closed and the test passes on DGX_H100 post-merge; un-waive follows the bug, not the PR.
• Scope reviewed by: auto-waive at 2026-11-05T21:05:50.165789+00:00

Tests in this break event:

• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_allows_guided_context_logits_on_graph_hit on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_callback_observes_eager_and_replay_execution on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_commits_candidate_only_on_graph_hit on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_graph_miss_uses_semantic_eager_batch on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_all_text_only_drops_deltas on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_generation_with_partial_segments on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_mixed_text_only_keeps_deltas_dense on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_promoted_context_uses_prompt_token_during_overlap on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_promoted_mrope_context_uses_decode_state_contract on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_steady_overlap_generation_advances_positions on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_kv_cache_manager_with_execution_stream on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_position_id_preparation on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup_skips_padding_dummy_during_estimation on DGX_H100
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup_skips_padding_dummy_when_padding_impossible on DGX_H100

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

• [TRTLLMINF-451][infra] Capture setup timeouts and defer unfinished cleanup until JUnit save #19333
• [https://nvbugs/6858496][fix] Fail disagg gen requests whose grammar matcher cannot be created #20903
• [TRTLLM-15944][feat] Enable SWA scratch reuse for Gemma3/4 with TRTLLM attention #20971
• [https://nvbugs/6891165][fix] Pass chat_template_kwargs in the request body of the JSON-schema serve example #21026
• [None][feat] Rubin merge back: MLA, DSpark, DWDP, Helix #21067
• [TRTLLM-17516][fix] Bound context logits memory with CPU fallback #21315
• [TRTLLM-16303][refact] Keep hybrid speculative replay state in KV cache regions #21324
• [None][feat] GEN slot preemption: pause decode requests in place to admit at-risk waiters #21334
• [TRTLLM-17272][feat] Add public LLM health queries and a private-access lint for the serving frontends #21393
• [None][fix] Fix draft KV over-allocation during chunked prefill #21546
• [TRTLLM-16624][fix] Fix Helix MLA decode kernel selection #21582
• [None][feat] Responses: carry the request id in the reply id #21602
• [None][perf] Build MSA fmha_sm100 JIT kernels for the visible GPUs' target only #21681

full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_allows_guided_context_logits_on_graph_hit SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_callback_observes_eager_and_replay_execution SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_commits_candidate_only_on_graph_hit SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_graph_miss_uses_semantic_eager_batch SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_all_text_only_drops_deltas SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_generation_with_partial_segments SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_mixed_text_only_keeps_deltas_dense SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_promoted_context_uses_prompt_token_during_overlap SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_promoted_mrope_context_uses_decode_state_contract SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_steady_overlap_generation_advances_positions SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_kv_cache_manager_with_execution_stream SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_position_id_preparation SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup_skips_padding_dummy_during_estimation SKIP (https://nvbugs/7053210)
full:DGX_H100/unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup_skips_padding_dummy_when_padding_impossible SKIP (https://nvbugs/7053210)

What to check before merging:

• The bug's test-history links show the cases failing on main, not only on the trigger PR.
• The scope DGX_H100 matches the failing stage and no narrower waiver already exists.
• Nothing else is needed from you after merging: the bug owner removes these lines with the fix.

Provenance

• Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
• Run 2e53fc1c-881a-4bae-860a-8a58bfecaccc; snapshot SHA-256 b3efa90d2937fc83ec49ddd8ca2285fda6706230bb41e303acbc0e6b26a20290; config SHA-256 622a43f13fb241c37a015e4ec2c812d7c8cecc1a04413b2618ca4ac119a77c81
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_allows_guided_context_logits_on_graph_hit: incident 362eb61cdef2fb8054b648a36d4dbf6d497f95083d3fb41e0670d447c6d8065e, observation 848a756b9caa50d87a56642c2a49218f09f57cce17c416544497bb844641a283
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_callback_observes_eager_and_replay_execution: incident 2b3540aa47995a5b24010e00c3f34a7f23b7c704402bc5803f75b8217ed1d9d8, observation 7511bb3b1b57c6e652db8955aa346a1b7222a1d09634a12c371a4e700004ceab
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_commits_candidate_only_on_graph_hit: incident 9880b4063a835a9eebbce85a193ad9c8cf4998b20d0f7c24c1004e4a8c50c8ae, observation 853c16f9b874bde2c0ae7b9fbf07f195835b71fa9020f5ae3fca0727b84f4ee6
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_forward_graph_miss_uses_semantic_eager_batch: incident 7b7e5b4e787a5fbf13c0f2d88cb043ca3adf418088e47914bbb0a24e6e76ba4f, observation 61639e94331217afda4cc374ca50862885be3abbfc47df97e6dc7bb9777f6eac
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_all_text_only_drops_deltas: incident a3da7ff9f3957477774c18f83bfa4898e58062ecc5758b9bdd202ab742df0541, observation ae294eb7fe0cea54f72982e6dce30d3667ace757a4d7a1ed33f2d6dde4b35f70
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_generation_with_partial_segments: incident 5c25f9a3182c46b6c8325ffcf9f4152ba5dc3ce3bc700212ba0cfeed49edb6dc, observation 7db997a3fa9470179f6db8b387dc07e86629a36d54177a7e3d6b674d717fe13a
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_mrope_mixed_text_only_keeps_deltas_dense: incident cea41105cbaad7694c6d44c86bc453903abcf8ccb292591c3766be32b98d372d, observation c0ad43c0b2a5096278a6a17e45035fb2e45e6d99c2cbdb8d0b5295f6f1520d35
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_promoted_context_uses_prompt_token_during_overlap: incident 7155a366d06761c10d454d06f546c7ddbb47881e8595b5d465e6b6b2b2aacd59, observation 14084093d2846d5060378d099fff4b6c703d88842dea6d452cea29d7cbfb4f56
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_promoted_mrope_context_uses_decode_state_contract: incident 2ed8b0eab0d45dcb9eb83d82a191e52ce78c1f1ee51803f4e813a175103cd4a1, observation 853e1db098ea39a15d7c5ebf13d27f154b654f605a35d97edca25c224d54ad6e
• unittest/_torch/pyexecutor/test_model_engine.py::DecoderForwardTestCase::test_steady_overlap_generation_advances_positions: incident 4c6bdaa9cc4afe9dcf5fbe5156c47bfebb61125c1c8b9b333fd4aa050030faa1, observation c1a1564b65af55b415b784f198ffb43e4d082e2b038ebacf490a1c88f9c269dc
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_kv_cache_manager_with_execution_stream: incident 5807cec925e8166825acb1f1536397cfca35bd43bbb3f81c0b6cbec633e6b526, observation 06bcd035e610499d77a7192ce698de0ddaff5cdbe14c7f7e1645c678ab6863d9
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_position_id_preparation: incident 6c78a62a7db1fd7cd23ab946182d7b8848aef871e7ee5bc5dcfda2bd2195ca6a, observation b3c2ccfede95a4ac855be0f58cc6498105bcae4d8b74a9b611b8f5fab350493e
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup: incident efdb6eb9c1e8d0d86a6b43d4593cfb07fcfd874d6549e11c4e76beb04dea61f7, observation 3648942a6f2d3e24e4be889b4b1458b5c5f0ea9dce32f02160565aba45b792cf
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup_skips_padding_dummy_during_estimation: incident f9cb87ca7a2dcd735f5ab975eb1517281d915f22dc609691a98a02b8e81b8d21, observation 3e07d3f7b31a5eb319ca2d32997654bbf157cf5eab8281166145fb818ac5d2a9
• unittest/_torch/pyexecutor/test_model_engine.py::ModelEngineTestCase::test_warmup_skips_padding_dummy_when_padding_impossible: incident 9761e971eeda8a8fed030cb84b89b4f18e4bae4c9844dbbf3fd91790b1f1b0b3, observation c223db7811ffa44cff049fea119a9a5e02950a47ea2fba2e74931dc21e67aa5c

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

[karljang]

Closing: these failures were a DGX_H100 host-memory condition (host OOM / CUDA invalid device context across 16 nodes, 37 of 50 builds passing), not a code break, and the stage recovered on its own. No waiver needed. The linked NVBug is being closed as not a bug.