Skip to content

[Bug][AutoDeploy]: All the perf dashboard's distributed models hang when using torch-opt #9847

New issue
New issue
Closed
#10087
Closed
[Bug][AutoDeploy]: All the perf dashboard's distributed models hang when using torch-opt#9847
#10087
Assignees
MrGeva
Labels
AutoDeploy/Dashboard<NV> AutoDeploy Backend: Dashboard relatedDisaggregated serving<NV>Deploying with separated, distributed components (params, kv-cache, compute). Arch & perf.bugSomething isn't working

Description

@MrGeva
MrGeva
opened on Dec 9, 2025

System Info

All the distributed models in the dashboard hang since the default allreduce strategy was changed from NCCL to AUTO. https://autodeploy-dashboard-a68618.gitlab-master-pages.nvidia.com/trtllm-bench

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

see the dashboard log

Expected behavior

no hang

actual behavior

hang :)

additional notes

na

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Activity

  1. MrGeva
    added this to AutoDeploy Boardon Dec 9, 2025
  2. MrGeva
    added
    bugSomething isn't working
    on Dec 9, 2025
  3. github-project-automation
    moved this to Backlog in AutoDeploy Boardon Dec 9, 2025
  4. MrGeva
    self-assigned this
    on Dec 9, 2025
  5. github-actions
    added
    Disaggregated serving<NV>Deploying with separated, distributed components (params, kv-cache, compute). Arch & perf.
    AutoDeploy/Dashboard<NV> AutoDeploy Backend: Dashboard related
    on Dec 9, 2025
  6. MrGeva
    moved this from Backlog to In progress in AutoDeploy Boardon Dec 15, 2025
  7. MrGeva
    changed the title [-][Bug][AutoDeploy]: All the perf dashboard's distributed models hang when setting AUTO allreduce strategy[/-] [+][Bug][AutoDeploy]: All the perf dashboard's distributed models hang when using torch-opt[/+] on Dec 15, 2025

  8. MrGeva commented on Dec 15, 2025

    @MrGeva
    MrGeva
    on Dec 15, 2025
    CollaboratorAuthor

    this is due to torch-opt. looking into it.

    trtllm-bench --model meta-llama/Llama-3.1-8B throughput --dataset dataset.txt --backend _autodeploy --tp 2 --extra_llm_api_options extra_llm_api_options.yaml --max_batch_size 256 --max_num_tokens 8192 

    compile_backend: torch-opt
free_mem_ratio: 0.8
runtime: trtllm
skip_loading_weights: True

    dataset.txt: /lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_modelopt/users/egeva/TensorRT-LLM/dataset.txt

  9. MrGeva commented on Dec 15, 2025

    @MrGeva
    MrGeva
    on Dec 15, 2025
    CollaboratorAuthor

    bisecting, narrowed down to:
    build 2406 - pass - 078d3a5
    build 2409 - failed - 47f650c

    now moved to bisecting with building from source because there are no whl's available between the impacted range

  10. MrGeva commented on Dec 15, 2025

    @MrGeva
    MrGeva
    on Dec 15, 2025
    CollaboratorAuthor

    bisected the culprit commit is: [c9771eb] [#9198][feat] Refactor dist ops in AutoDeploy (#9301)

  11. MrGeva commented on Dec 16, 2025

    @MrGeva
    MrGeva
    on Dec 16, 2025
    CollaboratorAuthor

    disabling rms norm allreduce fusion works:

      fuse_allreduce_residual_rmsnorm:
    enabled: false

  12. MrGeva commented on Dec 16, 2025

    @MrGeva
    MrGeva
    on Dec 16, 2025
    CollaboratorAuthor

    the story is more complicated:

    • The failure occurs also on latest main with nccl
• In my dist commit the rms+allreduce fusion started working. On the commit before it did not match anything. Hence my commit was suspect as the culprit, but it's not.
• In my enlarge AUTO strategy commit I had a bug in the rms norm allreduce fusion so it did not fuse in non-AUTO mode. Hence it passed.
• The change before it fails on shapes due to missing greg's .add calls - which I fixed in the commit above. When I cherry picked greg's fix (b894d40a1a06878c2bc874ebb7c26fcd5420db41) it hung with NCCL (AUTO hangs anyway at that point, missing my fix to AUTO hang)
• So NCCL torch-opt with rms+allreduce fusion was broken even before my fixes.
• This is the last dashboard run that succeeded 20 Nov. 63 rms norm patterns fused
	○ ++ TRTLLM_PRECOMPILED_LOCATION=https://urm.nvidia.com/artifactory/sw-tensorrt-generic/llm-artifacts/LLM/main/L0_PostMerge/2392/x86_64-linux-gnu/dlfw/tensorrt_llm-1.2.0rc3-cp312-cp312-linux_x86_64.whl

    ++ export TRTLLM_PRECOMPILED_COMMIT=96cfdd8a72cea169f5ff2427124b6cde3984a75f
    • Next dashboard run hangs and it detected 64 patterns.
    ○ ++ TRTLLM_PRECOMPILED_LOCATION=https://urm.nvidia.com/artifactory/sw-tensorrt-generic/llm-artifacts/LLM/main/L0_PostMerge/2403/x86_64-linux-gnu/dlfw/tensorrt_llm-1.2.0rc5-cp312-cp312-linux_x86_64.whl
    ++ export TRTLLM_PRECOMPILED_COMMIT=ae0124ef84e84dac59627094f8a12ad4f06ed89f
    ./start_bisect.sh 96cfdd8 ae0124e 2>&1 | tee bisect.log

  13. MrGeva commented on Dec 17, 2025

    @MrGeva
    MrGeva
    on Dec 17, 2025
    CollaboratorAuthor

    bad: [ae0124e] [None][infra] Check in most recent lock file from nightly pipeline

    git bisect bad ae0124e

    status: waiting for good commit(s), bad commit known

    good: [96cfdd8] [None][chore] Change trt-server to trtlllm-server in opentelemetry readme (#9173)

    git bisect good 96cfdd8

    bad: [a761585] [None][infra] Check in most recent lock file from nightly pipeline

    git bisect bad a761585

    bad: [5d118e0] [None][chore] Revise the description of enable_autotuner. (#9320)

    git bisect bad 5d118e0

    bad: [99ba723] [None][fix] logits device and shape issues in dynamic draft path (#9079)

    git bisect bad 99ba723

    good: [f0b68e4] [None][feat] AutoDeploy: Perf improvement for small batch size (#9163)

    git bisect good f0b68e4

    good: [7c4777a] [TRTLLM-9286][feat] Integration of CuteDSL NVFP4 grouped GEMM (#8880)

    git bisect good 7c4777a

    bad: [92f86a5] [#9137][feat] Factory sharding as default (#9144)

    git bisect bad 92f86a5

    good: [9b0f452] [None][feat] Have ability to cancel disagg request if KV cache resource are exhausted (#9155)

    git bisect good 9b0f452

    first bad commit: [92f86a5] [#9137][feat] Factory sharding as default (#9144)

  14. MrGeva commented on Dec 17, 2025

    @MrGeva
    MrGeva
    on Dec 17, 2025
    CollaboratorAuthor

    bisection shows that this commit caused the hang: 92f86a5

    comparing the log to the commit before I see the amount of rmsnorm+allreduce fusions increased by 1. that extra fusion seems to be problematic for the torch-opt mode regardless the allreduce strategy

    the commit before the culprit:
    20251216_144055_9b0f45298.log
    the culprit:
    20251216_142350_92f86a50d.log

  15. MrGeva commented on Dec 17, 2025

    @MrGeva
    MrGeva
    on Dec 17, 2025
    CollaboratorAuthor

    easiest reproduction instructions:

    git checkout egeva/repro_hang # this is the culprit commit, along with a fix to use .add instead of .append to the sharding info objects, it also comment out some unused ops which caused an error due to using latest whl. It adds logging IR option too.

TRTLLM_PRECOMPILED_LOCATION=https://urm.nvidia.com/artifactory/sw-tensorrt-generic/llm-artifacts/LLM/main/L0_PostMerge/2428/x86_64-linux-gnu/dlfw/tensorrt_llm-1.2.0rc6-cp312-cp312-linux_x86_64.whl pip install -e ".[devel]"

trtllm-bench --model meta-llama/Llama-3.1-8B  throughput --dataset dataset.txt --backend _autodeploy --tp 2 --extra_llm_api_options extra_llm_api_options.yaml --max_batch_size 256 --max_num_tokens 8192 2>&1 | tee log.txt

compile_backend: torch-opt
free_mem_ratio: 0.8
runtime: trtllm
skip_loading_weights: True
transforms:
  detect_sharding:
    allreduce_strategy: 'NCCL'
    sharding_source: ['heuristic'] # replace with ['factory','heuristic'] will cause it to hang
  fuse_allreduce_residual_rmsnorm:
    enabled: true

  16. MrGeva commented on Dec 17, 2025

    @MrGeva
    MrGeva
    on Dec 17, 2025
    CollaboratorAuthor

    see the IR dumps of ['heuristic'] vs. ['factory','heuristic']

    after_fusion_rank0_good.txt

    after_fusion_rank0_bad.txt

  17. MrGeva commented on Dec 17, 2025

    @MrGeva
    MrGeva
    on Dec 17, 2025
    CollaboratorAuthor

    WAR is in PR: #10084

  18. suyoggupta commented on Dec 17, 2025

    @suyoggupta
    suyoggupta
    on Dec 17, 2025
    Collaborator

    summary :

    1. meta-llama/Llama-3.1-8B , tp>1 works with torch-cudagraph and fuse_allreduce_residual_rmsnorm enabled
    2. meta-llama/Llama-3.1-8B , tp>1 works with torch-opt and fuse_allreduce_residual_rmsnorm disabled.

  19. MrGeva commented on Dec 17, 2025

    @MrGeva
    MrGeva
    on Dec 17, 2025
    CollaboratorAuthor

    Switched the perf dashboard to use torch-cudagraph to unblock it.
    We shall continue to investigate this hang on a lower priority

  20. greg-kwasniewski1
    closed this as completedin #10087on Dec 22, 2025
  21. github-project-automation
    moved this from In progress to Done in AutoDeploy Boardon Dec 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

AutoDeploy/Dashboard<NV> AutoDeploy Backend: Dashboard relatedDisaggregated serving<NV>Deploying with separated, distributed components (params, kv-cache, compute). Arch & perf.bugSomething isn't working

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions