[trtllm-agent]

Description

Temporarily waive 7 failed case(s) on DGX_H100. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_all_reusable_policy_recomputes_the_tail_across_chunks broken on main for DGX_H100: failure signature tensorrt_llm.bindings.internal.batch_manager.kv_ca....hostoomerror: mmap failed: cannot allocate memory, onset 2026-11-05T08:25:22Z, latest failure 2026-11-05T19:23:27Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7053310 tracks the fix. Scope DGX_H100 is the stage prefix that ran the failing test.

• Bug: https://nvbugs/7053310 (details, logs and 7-day test history are in the bug)
• Restore coverage when: Remove this line when https://nvbugs/7053310 is fixed or closed and the test passes on DGX_H100 post-merge; un-waive follows the bug, not the PR.
• Scope reviewed by: auto-waive at 2026-11-05T21:37:55.880696+00:00

Tests in this break event:

• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_all_reusable_policy_recomputes_the_tail_across_chunks on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_deferred_commit_policies_disable_the_spec_recompute_tail[per_conversation] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_deferred_commit_policies_disable_the_spec_recompute_tail[per_request] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_live_storage_stats_use_the_manager_api on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_recurrent_state_managers_allow_only_a_full_reprefill_tail[False-6] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_recurrent_state_managers_allow_only_a_full_reprefill_tail[True--1] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_windowed_layers_keep_the_recompute_tail_claim_capped on DGX_H100

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

• [TRTLLMINF-451][infra] Capture setup timeouts and defer unfinished cleanup until JUnit save #19333
• [https://nvbugs/6858496][fix] Fail disagg gen requests whose grammar matcher cannot be created #20903
• [TRTLLM-15944][feat] Enable SWA scratch reuse for Gemma3/4 with TRTLLM attention #20971
• [https://nvbugs/6891165][fix] Pass chat_template_kwargs in the request body of the JSON-schema serve example #21026
• [None][feat] Rubin merge back: MLA, DSpark, DWDP, Helix #21067
• [TRTLLM-16303][refact] Keep hybrid speculative replay state in KV cache regions #21324
• [None][feat] GEN slot preemption: pause decode requests in place to admit at-risk waiters #21334
• [TRTLLM-17272][feat] Add public LLM health queries and a private-access lint for the serving frontends #21393
• [None][fix] Fix draft KV over-allocation during chunked prefill #21546
• [TRTLLM-16624][fix] Fix Helix MLA decode kernel selection #21582
• [TRTLLM-16731][fix] Autotuner: skip redundant per-profile work for exclude_from_cache ops in tuning mode #21593
• [None][feat] Responses: carry the request id in the reply id #21602
• [None][perf] Build MSA fmha_sm100 JIT kernels for the visible GPUs' target only #21681

full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_all_reusable_policy_recomputes_the_tail_across_chunks SKIP (https://nvbugs/7053310)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_deferred_commit_policies_disable_the_spec_recompute_tail[per_conversation] SKIP (https://nvbugs/7053310)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_deferred_commit_policies_disable_the_spec_recompute_tail[per_request] SKIP (https://nvbugs/7053310)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_live_storage_stats_use_the_manager_api SKIP (https://nvbugs/7053310)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_recurrent_state_managers_allow_only_a_full_reprefill_tail[False-6] SKIP (https://nvbugs/7053310)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_recurrent_state_managers_allow_only_a_full_reprefill_tail[True--1] SKIP (https://nvbugs/7053310)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_windowed_layers_keep_the_recompute_tail_claim_capped SKIP (https://nvbugs/7053310)

What to check before merging:

• The bug's test-history links show the cases failing on main, not only on the trigger PR.
• The scope DGX_H100 matches the failing stage and no narrower waiver already exists.
• Nothing else is needed from you after merging: the bug owner removes these lines with the fix.

Provenance

• Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
• Run 91685f6e-5aff-47e8-b513-9a331b095319; snapshot SHA-256 6b9de299e0a60b16f56c9f5e3f10cc7bc240b9439bec64e644b42f4db79d9f9f; config SHA-256 622a43f13fb241c37a015e4ec2c812d7c8cecc1a04413b2618ca4ac119a77c81
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_all_reusable_policy_recomputes_the_tail_across_chunks: incident a0bc4fc4cfe6b0704612e71dcfe0fc1d2a3c1262e9455a1ec346a021dfc10fdc, observation 0946c94b89a5f96b2c0fa0659587a207ed1c76cc5d84bd4ea13b253a71143802
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_deferred_commit_policies_disable_the_spec_recompute_tail[per_conversation]: incident 2a4fb01f65edfda0c98f2fe4b0ac567a41b4d830488af2a57e54c891fc011f8f, observation ad0389909fed0de0c37eec0e0975775169042a776e131ba817aed7297b9434b9
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_deferred_commit_policies_disable_the_spec_recompute_tail[per_request]: incident 282c1b56bfe1205ef240676f644a42edce6a0411922d639f01b06a5f9f222ad4, observation f9d2355a71efc48d7495f01f9db003fcf83427b5c8673ca1fe53488032551ba6
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_live_storage_stats_use_the_manager_api: incident d5fb5072be9e45645eba946afdeb301d9ea78c8f4cb8bc0e83ff3d9c5a352ff8, observation 8967943626162f78bb865645690934bb89aaa09d64fa29af838c7957a7225ab2
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_recurrent_state_managers_allow_only_a_full_reprefill_tail[False-6]: incident dd980d408b01832deb5f232826dc5beb38687f0f89bc740635b938ed4c51e697, observation 2846513ad22eb4133a7e2d925e840486c1c0108023c91b08dc9c9ebd36f37422
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_recurrent_state_managers_allow_only_a_full_reprefill_tail[True--1]: incident 6626867057a8d4bf9df651b9911ed200a68bf4a93bd1a56cd7eda29486388437, observation 8d7e1c5952c3624bb4c212ed75256bbfe4fdf7f9d62764d822252a18c5246405
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_windowed_layers_keep_the_recompute_tail_claim_capped: incident 906c370286b6dd328da763930475e3987dad51ffee356c17f7b30bb0ed976c34, observation 750f83bf2688c18df754715e1dad1996c7f59c52d4075f8876e7ab4925629137

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

[karljang]

Closing: these failures were a DGX_H100 host-memory condition (host OOM / CUDA invalid device context across 16 nodes, 37 of 50 builds passing), not a code break, and the stage recovered on its own. No waiver needed. The linked NVBug is being closed as not a bug.