[Bug]: trtllm-serve stays alive returning 503 forever after a rank-crash hard kill (no exit for supervisors) #18663

System Info

• GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
• Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 (CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4)
• TensorRT-LLM: 1.3.0rc25 (tag v1.3.0rc25, 785c948); the code paths cited below were also checked on main @ a6616d6 (2026-09-03)
• Backend: PyTorch (trtllm-serve --backend pytorch)

Who can help?

No response (trtllm-serve owners: @NVIDIA/trt-llm-runtime-devs per CODEOWNERS)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run trtllm-serve (PyTorch backend, tp_size > 1, MPI orchestrator) in a container with --restart supervision, then make the executor die on one rank (any of the KV-manager failures in #18660 / #18661 reproduces it; so does killing one MPI worker process). Observed sequence:

[TRT-LLM] [E] [_torch][RANK 7] Executor loop crashed on this rank; hard-killing all 8 ranks in 9.99947s (peers cannot make progress without this rank). Set TLLM_RANK_CRASH_HARD_KILL_GRACE=-1 to disable.
[TRT-LLM] [E] [_torch][RANK 4] HangDetector: propagating hard-kill to all ranks via MPI_Abort.
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD with errorcode 137.
[TRT-LLM] [E] [executor] Fatal engine error recorded: RuntimeError('MPI worker rank 4 (pid 357) exited unexpectedly')

From then on the frontend process (PID 1, uvicorn) keeps running for hours: GET /health returns 503, /v1/chat/completions returns RuntimeError: LLM is shutting down / EngineDeadError: Engine has died for every request, GPU memory is released, and the process never exits.

Expected behavior

After a fatal engine error the server process exits with a non-zero status (after optionally draining in-flight responses), so container restart policies and process supervisors restart it. Alternatively /health should report a terminal "dead" state distinct from "starting" so a supervisor can act on it.

actual behavior

The process stays alive indefinitely in a state where it can never serve again; a restart policy (docker --restart unless-stopped, Kubernetes liveness on /health only if configured for 503) is not triggered because the process is up. We had to detect it externally (503 on /health plus "HangDetector" in the log) and kill the container by hand.

additional notes

• [TRTLLM-13409][feat] trtllm-serve: listen during startup so /health can say STARTING #16939 (listen during startup so /health can say STARTING) is about the startup phase; this report is about the post-crash phase and would compose with it (STARTING / READY / DEAD).
• Observed with rc25 on 8x H200 NVL, pp_size 2 --tp_size 4 and tp_size 8.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.