Repository navigation
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughCUDA IPC initialization now coordinates failures across tensor-parallel ranks, cleans up partial resources, and reports unavailable IPC. Allreduce initialization selects NCCL or raises an error based on IPC availability. Tests cover success, failure, fallback, and cleanup paths. ChangesCUDA IPC fallback
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant AllReduce
participant IpcMemory
participant TPCollective
participant NCCL
AllReduce->>IpcMemory: initialize IPC workspace
IpcMemory->>TPCollective: exchange IPC failure status
TPCollective-->>IpcMemory: return TP-wide status
IpcMemory-->>AllReduce: return workspace and failure flag
AllReduce->>NCCL: select NCCL when IPC failed
🚥 Pre-merge checks | ✅ 4 | ❌ 1
Finishing Touches
Tip 🎯 Customize your risk assessments
✅ Automatic approvals, with your rules
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/unittest/_torch/distributed/test_ipc_memory_fallback.py (1)
74-157: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftExpand coverage for the remaining fallback contracts.
Coverage summary — insufficient. The three added tests cover successful IPC setup, local import failure, and peer-reported failure. CI/QA test-list membership cannot be verified because no
tests/integration/test_lists/test-db/ortests/integration/test_lists/qa/file was provided.
- Make
FakeCudart.cudaIpcGetMemHandle()configurable and test its new dummy-handle/cleanup path.- In
test_local_ipc_open_failure_falls_back_instead_of_raising, useFakeDist(TP_SIZE)so a local failure alone drives agreement;peer_ipc_ok=Falsecurrently masks failures to propagate the local error.- Add mocked coverage that unavailable IPC downgrades
AllReduceto NCCL withworkspace is None, and thatlamport_initialize()is skipped.As per path instructions, test changes require coverage and test-list status review.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/distributed/test_ipc_memory_fallback.py` around lines 74 - 157, Expand the IPC fallback tests around FakeCudart.cudaIpcGetMemHandle, making its result configurable and covering the dummy-handle and cleanup behavior. Update test_local_ipc_open_failure_falls_back_instead_of_raising to use FakeDist(TP_SIZE), then add mocked coverage verifying unavailable IPC downgrades AllReduce to NCCL with workspace=None and skips lamport_initialize(); also review coverage and test-list status for these test changes.Source: Path instructions
tensorrt_llm/_ipc_utils.py (1)
18-18: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse built-in generics in
open_ipc_memoryReplace
Optional[Tuple[List[int], int]]withtuple[list[int], int] | None.Listis still needed by the other annotations in this file, so thetypingimport can’t be dropped yet.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_ipc_utils.py` at line 18, Update the return annotation of open_ipc_memory to use the built-in generic form tuple[list[int], int] | None instead of Optional[Tuple[List[int], int]]. Retain the existing typing imports because List remains required by other annotations in the file.Sources: Coding guidelines, Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_ipc_utils.py`:
- Around line 161-163: Update the CUDA IPC warning in the logger.warning call to
use the f-string conversion flag for ipc_error instead of explicitly calling
repr(), resolving Ruff RUF010 while preserving the existing message.
---
Nitpick comments:
In `@tensorrt_llm/_ipc_utils.py`:
- Line 18: Update the return annotation of open_ipc_memory to use the built-in
generic form tuple[list[int], int] | None instead of Optional[Tuple[List[int],
int]]. Retain the existing typing imports because List remains required by other
annotations in the file.
In `@tests/unittest/_torch/distributed/test_ipc_memory_fallback.py`:
- Around line 74-157: Expand the IPC fallback tests around
FakeCudart.cudaIpcGetMemHandle, making its result configurable and covering the
dummy-handle and cleanup behavior. Update
test_local_ipc_open_failure_falls_back_instead_of_raising to use
FakeDist(TP_SIZE), then add mocked coverage verifying unavailable IPC downgrades
AllReduce to NCCL with workspace=None and skips lamport_initialize(); also
review coverage and test-list status for these test changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ab9b4213-0365-4477-a4b7-557a34646fe2
📒 Files selected for processing (4)
tensorrt_llm/_ipc_utils.pytensorrt_llm/_torch/distributed/allreduce_helper.pytensorrt_llm/_torch/distributed/ops.pytests/unittest/_torch/distributed/test_ipc_memory_fallback.py
There was a problem hiding this comment.
Pull request overview
This PR addresses a Tensor Parallel initialization failure where CUDA IPC handle import/export can fail (e.g., cudaIpcOpenMemHandle returning cudaErrorInvalidDevice) even when cudaDeviceCanAccessPeer reports P2P capability, by degrading to the already-supported NCCL path instead of aborting engine initialization.
Changes:
- Make
IpcMemory.open_ipc_memory()returnNoneon CUDA IPC handle exchange failures (with TP-wide agreement + cleanup), and haveIpcMemory.__init__disable IPC (null pointers) instead of raising. - Gate Lamport buffer initialization on
lamport_buffers.open_ipcrather than the optimistic P2P check. - In
AllReduce.__init__, detect workspaces that have no usable IPC buffers and downgrade the strategy to NCCL; add a focused unit test module with stubbed CUDA runtime / distributed collectives.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
tensorrt_llm/_ipc_utils.py |
Make CUDA IPC failures non-fatal by returning None and cleaning up; disables IPC state in IpcMemory on failure. |
tensorrt_llm/_torch/distributed/allreduce_helper.py |
Prevent lamport_initialize() from running when IPC buffers are not actually available. |
tensorrt_llm/_torch/distributed/ops.py |
Add IPC-workspace detection and force NCCL fallback when IPC buffers are missing; reorder LOWPRECISION allocation to avoid null-workspace usage. |
tests/unittest/_torch/distributed/test_ipc_memory_fallback.py |
Add stubbed unit tests covering IPC-success, local IPC-open failure, and peer IPC failure agreement behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@BowenFu @schetlur-nv @JunyiXu-nv @allisonlim-nv could one of you take a look when you get a chance? The reporter of #16899 applied the patch on 8x RTX 5090 (PCIe, no NVLink), a different platform from the RTX 6000D in the original report: Before: crash at engine init, cudaErrorInvalidDevice: 101 from cudaIpcOpenMemHandle, all 8 ranks. So the failure is not specific to the RTX 6000D, it affects PCIe-only parts generally. Also pushed one more commit for Ruff nit. |
|
/bot run |
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
The _ipc_utils.py fix is the right one and the collective-agreement + cleanup details are handled carefully. Two things before merge.
The ops.py downgrade is broader than the bug. allreduce_workspace_has_ipc() is False whenever can_access_peer() is False — which includes the entirely normal inter-node TP case (can_access_peer returns False on a node-rank mismatch, and IpcMemory.__init__ already forces open_ipc=False when tp_size > gpus_per_node). See the inline comment at [ops.py:785](https://github.com/NVIDIA/TensorRT-LLM/pull/17034/files#diff-8343f10ef2d25d7245cbff50bea93ad47fd865db34a1e3c1d116ae57f7ab7ac1R785); as written this disables MNNVL on multi-node NVLink systems. The _ipc_utils + allreduce_helper changes alone already fix #16899; if you keep the ops.py guard, it should trigger only on the new failure mode (P2P reported, IPC unusable) and run after the MNNVL block.
Test coverage stops at _ipc_utils. The three new tests are good, but nothing covers the two behavioral changes that can break working hardware: the AllReduce.__init__ strategy downgrade and the lamport_initialize gating. A test asserting that an MNNVL-eligible mapping still gets mnnvl_allreduce set would have caught the issue above. tests/unittest/_torch/distributed is enrolled as a directory in l0_dgx_h100.yml, so the new file will run — worth confirming the cudart patching is safe alongside the real-GPU tests in that stage.
Description nits: the last paragraph is cut off mid-sentence ("trading a "), and the claim that FP8 "worked because it fits in fewer bytes, not because it dodges IPC" contradicts the following clause about not hitting the init path. Also, MoEAllReduce and MiniMaxAllReduceRMS still take the workspace unconditionally, so the "defense in depth" argument doesn't cover them — worth saying so explicitly rather than implying full coverage.
| # Every custom all-reduce kernel reads the peers' IPC buffers. When | ||
| # CUDA IPC is unavailable those pointers are null, so NCCL is the | ||
| # only strategy that can run. See allreduce_workspace_has_ipc. | ||
| if not allreduce_workspace_has_ipc(self.mapping): |
There was a problem hiding this comment.
This fires far more often than the bug it targets. allreduce_workspace_has_ipc() is False for any workspace built with is_p2p_supported=False, and can_access_peer() returns False on a plain node-rank mismatch ([_ipc_utils.py:46](https://github.com/NVIDIA/TensorRT-LLM/pull/17034/files#diff-58a9acafed7455cb039511131ef7c8fb87d8d39200b75e733c93914b25e26eecR46)) — plus IpcMemory.__init__ forces open_ipc=False when tp_size > gpus_per_node. So on any multi-node TP config this rewrites self.strategy to NCCL, and the MNNVL block right below ([ops.py:795](https://github.com/NVIDIA/TensorRT-LLM/pull/17034/files#diff-8343f10ef2d25d7245cbff50bea93ad47fd865db34a1e3c1d116ae57f7ab7ac1R795)) is keyed on self.strategy in (AUTO, MNNVL) — so mnnvl_allreduce is never constructed. That's a silent perf regression on GB200-class multi-node NVLink, which is exactly the topology MNNVL exists for.
Two fixes, either works: (a) only downgrade when the new failure mode occurred — i.e. can_access_peer(mapping) was True but open_ipc came back False — so systems that never had P2P keep today's behavior; or (b) move this block below the MNNVL init and skip it when self.mnnvl_allreduce is not None. (a) is narrower and matches the PR's stated scope.
There was a problem hiding this comment.
Went with (a). IpcMemory tracks ipc_failed separately from open_ipc now, set only when IPC was requested and the group fits in one node but the exchange still failed. can_access_peer False, node-rank mismatch, tp_size > gpus_per_node all leave it False, so multi-node TP does not touch self.strategy and the MNNVL block runs like before.
Added test_mnnvl_kept_without_p2p for it.
| # CUDA IPC is unavailable those pointers are null, so NCCL is the | ||
| # only strategy that can run. See allreduce_workspace_has_ipc. | ||
| if not allreduce_workspace_has_ipc(self.mapping): | ||
| logger.warning( |
There was a problem hiding this comment.
AllReduce is instantiated per decoder layer (often several per layer), so on an affected system this prints a warning ~100+ times per rank. Use logger.warning_once(..., key=...), or better, emit the message once from open_ipc_memory where the condition is actually detected.
There was a problem hiding this comment.
Moved it. warning_once with a key, out of open_ipc_memory where the condition is detected, so one line per process instead of one per AllReduce. Left a logger.debug at the strategy site that names what got downgraded.
| ipc_error = None if error == cudart.cudaError_t.cudaSuccess else error | ||
| if ipc_error is not None: | ||
| # Exchange a dummy handle to keep the collective below symmetric. | ||
| local_handle = cudart.cudaIpcMemHandle_t() |
There was a problem hiding this comment.
Exchanging an all-zero cudaIpcMemHandle_t means every other rank calls cudaIpcOpenMemHandle on garbage. It probably just returns an error you catch below, but you're feeding an uninitialized handle to the driver to preserve collective symmetry when a simpler option exists: allgather the cudaIpcGetMemHandle success flag first and skip the handle exchange + open loop entirely if any rank failed. Same number of collectives (you need a second agreement allgather for open failures either way), no garbage handles.
There was a problem hiding this comment.
Dropped the dummy handle. A rank that cannot export sends None in the handle allgather, and everyone bails before the open loop if any entry is None. That collective carries the payload and the export agreement together, so still two collectives, second one covers open failures.
| peer_ptrs.append(ptr) | ||
| opened_ptrs.append(ptr) | ||
|
|
||
| if not all(dist.tp_allgather(ipc_error is None)): |
There was a problem hiding this comment.
Only ranks that failed locally log; ranks that succeeded and are falling back because a peer failed degrade silently. Since the group decision is what matters operationally, log unconditionally in this branch (with the peer-driven case worded differently), so a user reading rank 0's log can tell why custom all-reduce is off.
There was a problem hiding this comment.
Both branches log now, through a shared disable_ipc(). Message is either failed here with or failed on a peer rank of the TP group.
| still cannot be imported. In that case the workspace pointers are null and only | ||
| NCCL can be used. get_allreduce_workspace must have been called first. | ||
| """ | ||
| allreduce_workspaces = getattr(_thread_local, |
There was a problem hiding this comment.
getattr without a default raises AttributeError if get_allreduce_workspace hasn't run for this pp_rank — the docstring notes the precondition, but an unguarded getattr/[mapping] turns a misuse into an opaque traceback. Cheap fix: take the workspace tuple as an argument, or have get_allreduce_workspace return both the tensor and the flag so the invariant is structural rather than documented.
There was a problem hiding this comment.
get_allreduce_workspace returns (workspace, ipc_failed) and computes the flag when it fills the thread-local, so no second reader and no bare getattr. MoEAllReduce and MiniMaxAllReduceRMS go through _require_ipc_workspace() which raises, they have no NCCL path.
Addresses review feedback on NVIDIA#17034. allreduce_workspace_has_ipc() was False for any workspace built without P2P, which includes ordinary inter-node TP, so it rewrote the strategy to NCCL before the MNNVL block and mnnvl_allreduce was never constructed on multi-node NVLink systems. IpcMemory now records ipc_failed separately from open_ipc, and only that failure mode, P2P reported but handles unusable, downgrades the strategy. Configurations that never had P2P keep the behaviour they had. get_allreduce_workspace returns the flag alongside the workspace, so a caller cannot read the thread-local state before it has been populated. The fallback is reported once, from open_ipc_memory where it is detected, and on every rank of the group rather than only on the ones that failed locally. AllReduce is constructed per decoder layer, so it no longer warns per instance. A rank that cannot export its handle contributes None to the allgather instead of an uninitialized cudaIpcMemHandle_t, so no rank imports a garbage handle. The number of collectives is unchanged. MoEAllReduce and MiniMaxAllReduceRMS have no NCCL path, so they now raise an actionable error instead of letting the kernel dereference null peer pointers. Tests cover the strategy downgrade, MNNVL still being selected when the workspace has no IPC buffers by design, and the lamport_initialize gating. Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
pjdurden
changed the title
|
All five inline points addressed. Pushed as a separate commit on top of the original fix so the delta reads on its own, then re-signed the branch to clear DCO, it was failing on the first commit. ops.py downgrade: went with (a). IpcMemory tracks ipc_failed separately from open_ipc, set only when IPC was requested and possible but the exchange failed. can_access_peer returning False leaves it False, so inter-node TP keeps current behaviour and still reaches the MNNVL block. get_allreduce_workspace returns (workspace, ipc_failed), which also kills the bare getattr at ops.py:62. Tests are 11 now. New: test_mnnvl_kept_without_p2p (the guard for the above), test_downgrade_to_nccl, test_lamport_skipped which drives the real allocate_allreduce_fusion_workspace and asserts lamport_initialize is not called, test_moe_allreduce_raises, test_peer_export_fails, plus two for ipc_failed staying False. On the cudart patching in that stage: every patch is unittest.mock.patch scoped to a with block on module attributes, nothing survives the test, and the fallback frees its fake pointers through the fake runtime before the patch unwinds. test_safe_mpi_comm.py in the same dir already patches this way. Only test_lamport_skipped touches real CUDA, it is gated on torch.cuda.is_available() because the helper allocates device tensors. MoEAllReduce and MiniMaxAllReduceRMS: you are right that defense in depth overclaimed. Neither has a non-IPC path, so they raise explicitly when ipc_failed is set. Same init-time failure they had before this PR, just with a message instead of a null deref in the kernel. Called out in the description now. Description rewritten. Truncated paragraph gone, FP8 claim dropped, I have no evidence for the mechanism there, only that those runs did not hit this path. Title follows [None][fix] now. Same caveat as before: no GPU or torch here so pytest did not run. The _ipc_utils logic is covered by a standalone harness that loads the real edited module with the CUDA runtime and collectives stubbed, six scenarios, all pass. Section 6 of the description lists them. Rest needs CI. DCO is green. Can you kick off another /bot run? |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tests/unittest/_torch/distributed/test_ipc_memory_fallback.py (1)
271-302: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest coverage summary: insufficient.
Added tests cover IPC success, local and peer IPC failures, intentional IPC absence, MNNVL preservation, NCCL fallback, MoE failure, and Lamport initialization.
Add a direct
MiniMaxAllReduceRMSconstruction test withipc_failed=True. The production change now routes this class through_require_ipc_workspace(), but no test verifies that integration.Test-list membership is not applicable to this review because no
tests/integration/test_lists/file was supplied. Runpytest tests/unittest/after the update.As per path instructions, test-code changes must include a coverage verdict.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/distributed/test_ipc_memory_fallback.py` around lines 271 - 302, The tests lack direct coverage for MiniMaxAllReduceRMS when IPC is unavailable. Add a unit test that constructs MiniMaxAllReduceRMS with ipc_failed=True and verifies the production path raises the expected requires-CUDA-IPC error through _require_ipc_workspace(); then run pytest tests/unittest/ and include a coverage verdict for the added test.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/distributed/ops.py`:
- Around line 69-81: Update _require_ipc_workspace to distinguish IPC usability
from ipc_failed, and reject workspaces when IPC is intentionally unavailable as
well as when initialization fails; do not return a workspace containing null
peer pointers to MoEAllReduce or MiniMaxAllReduceRMS. Preserve the upstream
contract that intentional IPC absence leaves ipc_failed false, and add
inter-node coverage for both IPC-only fused operations.
In `@tests/unittest/_torch/distributed/test_ipc_memory_fallback.py`:
- Around line 140-143: Update _run so the IpcMemory created with open_ipc=True
is explicitly cleaned up while cudart_patch and dist_patch remain active, then
set its open_ipc attribute to False before returning it to prevent
IpcMemory.__del__ from performing a second cleanup.
---
Nitpick comments:
In `@tests/unittest/_torch/distributed/test_ipc_memory_fallback.py`:
- Around line 271-302: The tests lack direct coverage for MiniMaxAllReduceRMS
when IPC is unavailable. Add a unit test that constructs MiniMaxAllReduceRMS
with ipc_failed=True and verifies the production path raises the expected
requires-CUDA-IPC error through _require_ipc_workspace(); then run pytest
tests/unittest/ and include a coverage verdict for the added test.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f1588ceb-fd00-410f-8baf-31aafa0793da
📒 Files selected for processing (4)
tensorrt_llm/_ipc_utils.pytensorrt_llm/_torch/distributed/allreduce_helper.pytensorrt_llm/_torch/distributed/ops.pytests/unittest/_torch/distributed/test_ipc_memory_fallback.py
🚧 Files skipped from review as they are similar to previous changes (2)
- tensorrt_llm/_torch/distributed/allreduce_helper.py
- tensorrt_llm/_ipc_utils.py
| def _require_ipc_workspace(mapping: Mapping, op_name: str) -> torch.LongTensor: | ||
| """Workspace for fused ops that reinterpret it as `void**` and have no NCCL path. | ||
|
|
||
| AllReduce degrades to NCCL when CUDA IPC turns out to be unusable, but these | ||
| kernels cannot: they would dereference the null peer pointers. Fail here with an | ||
| actionable message rather than in the kernel with an illegal memory access. | ||
| """ | ||
| workspace, ipc_failed = get_allreduce_workspace(mapping) | ||
| if ipc_failed: | ||
| raise RuntimeError( | ||
| f"{op_name} requires CUDA IPC, which is unavailable on this system, " | ||
| "and it has no NCCL fallback.") | ||
| return workspace |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Reject workspaces with no usable IPC buffers.
ipc_failed is false when IPC was intentionally unavailable, such as inter-node TP. In that state, IpcMemory still serializes null peer pointers. Lines 76-81 return this workspace to MoEAllReduce and MiniMaxAllReduceRMS, and their fused kernels can dereference the null pointers.
Track IPC usability separately from IPC failure. Make _require_ipc_workspace() reject both states. Add inter-node tests for both IPC-only fused operations.
Based on the supplied upstream contract, intentionally absent IPC keeps ipc_failed=False and leaves peer pointers null.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/_torch/distributed/ops.py` around lines 69 - 81, Update
_require_ipc_workspace to distinguish IPC usability from ipc_failed, and reject
workspaces when IPC is intentionally unavailable as well as when initialization
fails; do not return a workspace containing null peer pointers to MoEAllReduce
or MiniMaxAllReduceRMS. Preserve the upstream contract that intentional IPC
absence leaves ipc_failed false, and add inter-node coverage for both IPC-only
fused operations.
There was a problem hiding this comment.
the inter-node case is already the same on main: get_allreduce_workspace builds the IpcMemory buffers with open_ipc=False when tp_size > gpus_per_node, and MoEAllReduce and MiniMaxAllReduceRMS take that null workspace unconditionally. this PR only adds the failure case where P2P was reported but the handle exchange did not work. Rejecting the by-design inter-node workspace in those two ops would change which configurations can build them at all, which I think belongs in its own change rather than in this one. leaving it as is.
Addresses review feedback on NVIDIA#17034. allreduce_workspace_has_ipc() was False for any workspace built without P2P, which includes ordinary inter-node TP, so it rewrote the strategy to NCCL before the MNNVL block and mnnvl_allreduce was never constructed on multi-node NVLink systems. IpcMemory now records ipc_failed separately from open_ipc, and only that failure mode, P2P reported but handles unusable, downgrades the strategy. Configurations that never had P2P keep the behaviour they had. get_allreduce_workspace returns the flag alongside the workspace, so a caller cannot read the thread-local state before it has been populated. The fallback is reported once, from open_ipc_memory where it is detected, and on every rank of the group rather than only on the ones that failed locally. AllReduce is constructed per decoder layer, so it no longer warns per instance. A rank that cannot export its handle contributes None to the allgather instead of an uninitialized cudaIpcMemHandle_t, so no rank imports a garbage handle. The number of collectives is unchanged. MoEAllReduce and MiniMaxAllReduceRMS have no NCCL path, so they now raise an actionable error instead of letting the kernel dereference null peer pointers. Tests cover the strategy downgrade, MNNVL still being selected when the workspace has no IPC buffers by design, and the lamport_initialize gating. Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Addresses review feedback on NVIDIA#17034. allreduce_workspace_has_ipc() was False for any workspace built without P2P, which includes ordinary inter-node TP, so it rewrote the strategy to NCCL before the MNNVL block and mnnvl_allreduce was never constructed on multi-node NVLink systems. IpcMemory now records ipc_failed separately from open_ipc, and only that failure mode, P2P reported but handles unusable, downgrades the strategy. Configurations that never had P2P keep the behaviour they had. get_allreduce_workspace returns the flag alongside the workspace, so a caller cannot read the thread-local state before it has been populated. The fallback is reported once, from open_ipc_memory where it is detected, and on every rank of the group rather than only on the ones that failed locally. AllReduce is constructed per decoder layer, so it no longer warns per instance. A rank that cannot export its handle contributes None to the allgather instead of an uninitialized cudaIpcMemHandle_t, so no rank imports a garbage handle. The number of collectives is unchanged. MoEAllReduce and MiniMaxAllReduceRMS have no NCCL path, so they now raise an actionable error instead of letting the kernel dereference null peer pointers. Tests cover the strategy downgrade, MNNVL still being selected when the workspace has no IPC buffers by design, and the lamport_initialize gating. Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
|
Rebased onto main. It was 162 behind, now 0, and it was a clean rebase, none of One thing that changed under us and is worth flagging, since your review note assumed otherwise: #16498 moved
On the Separately, checking the MNNVL interaction more closely: |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unittest/_torch/distributed/test_ipc_memory_fallback.py`:
- Around line 92-98: Update FakeCudart.cudaMalloc and FakeCudart.cudaMemset to
support configurable CUDA failures, then add tests covering each RuntimeError
path and verifying errors propagate without falling back from IPC to NCCL. Keep
the existing success behavior unchanged and exercise the failure configuration
through the relevant distributed IPC test flows.
- Around line 46-51: Update _has_allreduce_op to catch only AttributeError when
accessing torch.ops.trtllm.allreduce, allowing unrelated setup failures to
propagate while preserving the false result for a missing operator.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 39b3239f-8583-45ba-9fca-ce8317e5b2d7
📒 Files selected for processing (4)
tensorrt_llm/_ipc_utils.pytensorrt_llm/_torch/distributed/allreduce_helper.pytensorrt_llm/_torch/distributed/ops.pytests/unittest/_torch/distributed/test_ipc_memory_fallback.py
🚧 Files skipped from review as they are similar to previous changes (3)
- tensorrt_llm/_torch/distributed/allreduce_helper.py
- tensorrt_llm/_ipc_utils.py
- tensorrt_llm/_torch/distributed/ops.py
cudaDeviceCanAccessPeer reporting P2P support does not guarantee that CUDA IPC handles can be exported and imported. On 8x RTX 6000D, and on PCIe-only parts generally, cudaIpcOpenMemHandle fails with cudaErrorInvalidDevice and engine initialization aborted instead of using the NCCL path that already exists. IpcMemory.open_ipc_memory now returns None on such a failure, after agreeing on the outcome across the whole TP group and releasing whatever it managed to allocate, so that either every rank gets IPC buffers or none of them does. IpcMemory keeps null pointers in that case, and lamport_initialize is gated on the buffers actually being open. Fixes NVIDIA#16899 Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
Addresses review feedback on NVIDIA#17034. allreduce_workspace_has_ipc() was False for any workspace built without P2P, which includes ordinary inter-node TP, so it rewrote the strategy to NCCL before the MNNVL block and mnnvl_allreduce was never constructed on multi-node NVLink systems. IpcMemory now records ipc_failed separately from open_ipc, and only that failure mode, P2P reported but handles unusable, downgrades the strategy. Configurations that never had P2P keep the behaviour they had. get_allreduce_workspace returns the flag alongside the workspace, so a caller cannot read the thread-local state before it has been populated. The fallback is reported once, from open_ipc_memory where it is detected, and on every rank of the group rather than only on the ones that failed locally. AllReduce is constructed per decoder layer, so it no longer warns per instance. A rank that cannot export its handle contributes None to the allgather instead of an uninitialized cudaIpcMemHandle_t, so no rank imports a garbage handle. The number of collectives is unchanged. MoEAllReduce and MiniMaxAllReduceRMS have no NCCL path, so they now raise an actionable error instead of letting the kernel dereference null peer pointers. Tests cover the strategy downgrade, MNNVL still being selected when the workspace has no IPC buffers by design, and the lamport_initialize gating. Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
NVIDIA#16498 moved tests/unittest/_torch/distributed from l0_dgx_h100 to l0_cpu, so this file now runs with zero GPUs. The three tests that build an AllReduce skip when the trtllm custom ops are not registered, since AllReduce.__init__ resolves torch.ops.trtllm.allreduce before any of the logic under test. test_lamport_skipped still needs a GPU because the workspace helper it drives allocates device tensors. Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
|
Rebased onto current main and pushed as One real conflict this time, and it is worth flagging because it sits exactly on the code this PR rewrites. #19184 (Rubin kernels) refactored the handle exchange in I resolved it by routing this PR's fallback through those new helpers rather than keeping the raw
That is the only semantic change in the whole rebase. I diffed this PR's own net diff before and after: the four touched files are otherwise byte-identical. The fallback logic, the group agreement and the cleanup path are unchanged. I also checked the helpers against the stubs in On the actual blocker: the last pipeline run here is PR_Github #68209 from Aug 21, against @bo-nv @JunyiXu-nv @schetlur-nv could one of you kick a fresh run on |
Signed-off-by: pjdurden <prajjwalchittori1@gmail.com> # Conflicts: # tensorrt_llm/_torch/distributed/ops.py
|
merged current main in as |
JunyiXu-nv
left a comment
JunyiXu-nv
left a comment
There was a problem hiding this comment.
Requesting changes: after a rebase the tuple return breaks main's new get_allreduce_workspace caller, the MNNVL guard test is broken, and the fallback catches every CUDA IPC error.
|
|
||
|
|
||
| def get_allreduce_workspace(mapping: Mapping) -> torch.LongTensor: | ||
| def get_allreduce_workspace(mapping: Mapping) -> Tuple[torch.LongTensor, bool]: |
There was a problem hiding this comment.
main gained another caller after your Oct 16 merge: allreduce_group_args() in tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allreduce.py:53 (#20267) passes get_allreduce_workspace(mapping) straight through as workspace. With the tuple return it hands (tensor, False) to trtllm::allreduce, which rejects it ("Expected a value of type 'Optional[Tensor]' for argument 'workspace' but instead found type 'tuple'"), and the rebase will not show a conflict in that file. Please update that caller here too (unpack, and use NCCL with no workspace when ipc_failed is set), or keep this function returning the tensor and expose ipc_failed through a separate helper.
There was a problem hiding this comment.
kept get_allreduce_workspace returning the tensor, so main's callers are untouched. the flag is read through a separate allreduce_workspace_ipc_failed(mapping) now. also updated allreduce_group_args to switch to NCCL with no workspace when it is set, same as the AllReduce module, plus a test for it. merged main in, 946979a
| # P2P was reported but the IPC handles could not be exchanged those | ||
| # pointers are null, so NCCL is the only strategy left. The user | ||
| # already got a warning from open_ipc_memory at that point. | ||
| if ipc_failed: |
There was a problem hiding this comment.
ipc_failed is set for any non-success code from cudaIpcGetMemHandle / cudaIpcOpenMemHandle (_ipc_utils.py:186 and :209), not only the capability gap from #16899. So a bug like the broken handle round-trip #19184 fixed (cudaErrorInvalidValue / cudaErrorInvalidResourceHandle) would switch every custom all-reduce on NVLink nodes to NCCL with a single warning line, and numerics tests still pass, so CI would not catch it. Could the fallback be limited to the capability errors (cudaErrorInvalidDevice, cudaErrorNotSupported, ...) and raise otherwise? Allgathering the error code instead of None/bool would let every rank take the same raise-or-fallback decision.
There was a problem hiding this comment.
narrowed it. every rank allgathers its CUDA error code instead of None/bool, and the group only falls back when all failures are cudaErrorInvalidDevice, cudaErrorNotSupported or cudaErrorPeerAccessUnsupported. anything else (InvalidValue, InvalidResourceHandle, ...) releases the buffers and raises on every rank, naming the rank and error. tests cover local, peer and mixed cases. 946979a
|
|
||
| class FakeMNNVLAllReduce: | ||
| @staticmethod | ||
| def is_mnnvl(mapping, dtype): |
There was a problem hiding this comment.
MNNVLAllReduce.is_mnnvl takes an explicitly_requested keyword since #18231, and AllReduce.__init__ passes it (ops.py:1163), so this fake now raises TypeError: FakeMNNVLAllReduce.is_mnnvl() got an unexpected keyword argument 'explicitly_requested'. test_mnnvl_kept_without_p2p therefore fails wherever the trtllm op is registered and is skipped elsewhere, so the MNNVL regression guard does not guard anything. Adding explicitly_requested=False to the fake fixes it.
There was a problem hiding this comment.
fixed, the fake takes explicitly_requested=False now. the other ops tests patch get_allreduce_workspace / allreduce_workspace_ipc_failed separately to match the new API. 946979a
Signed-off-by: pjdurden <prajjwalchittori1@gmail.com> # Conflicts: # tensorrt_llm/_torch/distributed/ops.py
…ty errors get_allreduce_workspace returns the tensor again, so callers that pass it straight to the op (allreduce_group_args in modeling_v2) keep working. The IPC failure flag is read through allreduce_workspace_ipc_failed(), and allreduce_group_args now switches to NCCL with no workspace when it is set, like the AllReduce module. open_ipc_memory allgathers the CUDA error code of each rank instead of a None/bool, and only falls back to NCCL when every failure is a capability error (cudaErrorInvalidDevice, cudaErrorNotSupported, cudaErrorPeerAccessUnsupported). Any other error, a broken handle for instance, releases the buffers and raises on every rank. The MNNVL guard test fake takes the explicitly_requested keyword that AllReduce passes to is_mnnvl, so the test runs again instead of raising a TypeError. Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
JunyiXu-nv
left a comment
JunyiXu-nv
left a comment
There was a problem hiding this comment.
Thanks, all three of my comments are addressed. LGTM.
Signed-off-by: pjdurden <prajjwalchittori1@gmail.com> # Conflicts: # tensorrt_llm/_torch/distributed/ops.py
|
merged current main in (e7813ca). only conflict was |
| if allreduce_workspace_ipc_failed(mapping): | ||
| raise RuntimeError( | ||
| f"{op_name} requires CUDA IPC, which is unavailable on this system, " | ||
| "and it has no NCCL fallback.") |
There was a problem hiding this comment.
MoEAllReduce is built whenever TP>1 (Qwen3-MoE, DeepSeek-V3/V4, EXAONE-MoE; GLM and Llama4 always), but only called on the TRTLLM-nvfp4 fused path. So a BF16 Qwen3-MoE TP run on the #16899 GPUs still dies here at init, though the kernel never runs. Could we raise in forward, or fold allreduce_workspace_ipc_failed into the models' is_p2p_supported? MiniMax-M2's MiniMaxAllReduceRMS too.
There was a problem hiding this comment.
you're right, fixed in e66a68c. MoEAllReduce and MiniMaxAllReduceRMS now build fine and raise from forward / forward_qk instead, and I did both of your options: Qwen3-MoE, DeepSeek-V3/V4, EXAONE-MoE, GLM, the sm_107 GLM _model.py and MiniMaxRMSNorm fold allreduce_workspace_ipc_failed into is_p2p_supported, so they take their existing non-fused path, including the NCCL path in MiniMax-M2. Llama4 only calls moe_allreduce under cutlass_min_latency_mode, and MiniMax-M3 has no is_p2p_supported gate, so M3 still raises at its first forward on these GPUs. test_moe_allreduce_raises_in_forward, test_minimax_allreduce_rms_raises_in_forward and test_minimax_m2_norm_uses_nccl_without_ipc cover this, but I could not run them here since this machine has no torch or GPU.
| This directory runs in the CPU-only l0_cpu stage since #16498, so everything here has | ||
| to pass with zero GPUs. `test_lamport_skipped` is the exception and skips there: the |
There was a problem hiding this comment.
This file has no pytest.mark.cpu_only, and the CPU stages run unit tests with -m cpu_only, so pytest_ignore_collect in tests/unittest/conftest.py drops it from the unittest/_torch/distributed entry. No GPU list has this dir either, so none of these tests run in CI. Could we add pytestmark = pytest.mark.cpu_only (and give test_lamport_skipped a GPU home, or drop it)?
There was a problem hiding this comment.
fixed, the file has pytestmark = pytest.mark.cpu_only now, so pytest_ignore_collect keeps it in the unittest/_torch/distributed entry. test_lamport_skipped lost its skipif and runs on CPU too: it patches torch.tensor to drop the device argument, since the only thing it needed a GPU for was the flag and pointer tensors in allocate_allreduce_fusion_workspace. not run here, this machine has no torch. e66a68c
| def is_mnnvl(mapping, dtype, explicitly_requested=False): | ||
| return True | ||
|
|
||
| def __init__(self, mapping, dtype): |
There was a problem hiding this comment.
AllReduce.__init__ now calls MNNVLAllReduce(mapping, dtype, max_num_tokens=..., hidden_size=...) (#20815), so this fake raises TypeError, the except Exception there swallows it and mnnvl_allreduce stays None. test_mnnvl_kept_without_p2p fails once it actually runs. Could the fake take **kwargs?
…back tests in CI Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
Signed-off-by: pjdurden <prajjwalchittori1@gmail.com>
Fall back to NCCL when CUDA IPC handles cannot be exchanged
Fixes #16899
1. Root cause
The reporter's follow-up (v1.3.0rc22) pins the failure precisely:
and notes that
can_access_peer()returnedTrueon the same machine.can_access_peer()(tensorrt_llm/_ipc_utils.py:39) only callscudaDeviceCanAccessPeer. That answers "can GPU A address GPU B's memory", whichis necessary but not sufficient for CUDA IPC: exporting and importing an IPC
memory handle is a separate capability that also fails on GPUs without CUDA IPC
support and when the ranks do not share an IPC namespace. On this system
cudaDeviceCanAccessPeersucceeds whilecudaIpcOpenMemHandledoes not.Because
can_access_peer()saidTrue,IpcMemorywas constructed withopen_ipc=True, andopen_ipc_memory()passed the raw CUDA error straight to_raise_if_error(). That unhandledRuntimeErroraborts engine initialization.This is a hard crash for a situation the runtime already fully supports. The
non-IPC path exists and works:
IpcMemory(open_ipc=False)leaves all pointersnull, and on the C++ side
AllReduceOp::ifFallbackToNCCL(
cpp/tensorrt_llm/thop/allreduceOp.cpp:1436) already routes every all-reduce toNCCL when P2P/NVLink is absent, which is exactly this hardware (RTX 6000D is a
PCIe part with no NVLink, so
mIsNVLINKSupportedis false and the custom kernelswould never have been selected anyway). Only the eager Python-side workspace
allocation stood in the way.
One secondary note that matches the report:
TRTLLM_DISABLE_CUSTOM_ALLREDUCE=1had no effect because no such env var exists anywhere in the tree (
grepreturnsnothing). The supported knob is
allreduce_strategy: NCCLin the LLM args, whichskips the workspace allocation entirely. That is a valid workaround, but users
should not need it.
2. The fix
Degrade to the already-supported NCCL path instead of crashing.
tensorrt_llm/_ipc_utils.py.IpcMemory.open_ipc_memoryno longer raiseswhen IPC handles cannot be exchanged. It returns
None, andIpcMemory.__init__reacts by setting
open_ipc = Falseandipc_failed = True, leaving thepointers null. Three details matter:
either every rank gets IPC buffers or none does. Without this, a rank that
succeeded would keep using buffers its peers never mapped, and the subsequent
tp_allgathercalls could desync.Noneto thehandle allgather rather than an uninitialized
cudaIpcMemHandle_t, so no rankever passes an uninitialized handle to
cudaIpcOpenMemHandle. That onecollective carries both the payload and the agreement on the export step, so
the total is still two collectives, as before.
is freed, so the fallback does not leak device memory.
cudaMalloc/cudaMemsetfailures still raise. Those are genuine errors, not acapability gap.
tensorrt_llm/_torch/distributed/allreduce_helper.py. Gatelamport_initialize()onlamport_buffers.open_ipcrather than onis_p2p_supported. Those two can now disagree, and callinglamport_initializewith a null
local_ptrwould be an illegal memory access.tensorrt_llm/_torch/distributed/ops.py.get_allreduce_workspacereturnsthe workspace together with an
ipc_failedflag, andAllReduce.__init__downgrades the strategy to
NCCLonly when that flag is set. The flagdistinguishes the new failure mode (P2P reported, IPC handles unusable) from a
workspace that holds no IPC buffers by design, which is the ordinary inter-node
TP case. That distinction is load-bearing: reacting to the latter would rewrite
the strategy before the MNNVL block and silently disable
mnnvl_allreduceonmulti-node NVLink systems. The
LOWPRECISIONworkspace allocation moved belowthe check so
initialize_static_lowprecision_buffersis not called on a nullworkspace.
SYMM_MEMis unaffected:AllReduce.forwardtriesself.symm_mem_allreducebefore consulting
self.strategy, so the downgrade only changes the fallbackpath.
Fused ops that have no NCCL path.
MoEAllReduceandMiniMaxAllReduceRMShand the workspace to custom kernels that reinterpret it as
void**, and neitherhas a non-IPC alternative, so the strategy downgrade above does not cover them.
They now raise an explicit error when
ipc_failedis set, rather thandereferencing the null peer pointers inside the kernel. That is the same
init-time failure those paths had before this PR, with a message that says why.
3. Files changed
tensorrt_llm/_ipc_utils.pyopen_ipc_memoryreturnsOptional[...]; collective IPC-failure agreement without exchanging uninitialized handles; cleanup;IpcMemory.ipc_failed; single warning on the grouptensorrt_llm/_torch/distributed/allreduce_helper.pylamport_initializeon the actual IPC statetensorrt_llm/_torch/distributed/ops.pyget_allreduce_workspacereturns(workspace, ipc_failed); strategy downgrade to NCCL on real IPC failure only; reorder LOWPRECISION allocation; explicit error forMoEAllReduce/MiniMaxAllReduceRMStests/unittest/_torch/distributed/test_ipc_memory_fallback.py4. Reported validation on a second platform
The reporter of #16899 applied the first revision of this patch on 8x RTX 5090
(PCIe, no NVLink), which is a different platform from the RTX 6000D in the
original report:
cudaErrorInvalidDevice: 101fromcudaIpcOpenMemHandle, on all 8 ranks.benchmark run to completion.
So the failure is not specific to the RTX 6000D, it affects PCIe-only parts
generally. That run predates the changes in section 2 that narrow the downgrade
and rework the handle exchange.
5. Risk and uncertainty
environment has neither GPUs nor
torch. The root cause is derived from thereporter's traceback plus reading the code; the specific reason
cudaIpcOpenMemHandlereturns 101 on that GPU is not established. The fix isdeliberately agnostic to that reason, it handles "IPC handles cannot be
exchanged" however it arises.
slower but correct NCCL path. That is the right trade-off here, since the
runtime already treats non-P2P topologies this way, but it does mean a
genuinely broken IPC setup degrades instead of failing loudly. The warning is
emitted once per process, from
open_ipc_memorywhere the condition isdetected, and on every rank of the group rather than only on the ones that
failed locally.
hit this initialization path; I have no evidence about the mechanism and make
no claim about it.
allgather that already existed, plus the one that covers the open step. Two
collectives per
IpcMemoryconstruction, unchanged.can_access_peer()still reportsTrueonsuch systems, so
self.is_p2p_supportedinmodeling_deepseekv3.py,modeling_glm.py,modeling_qwen3_moe.pyand friends remains optimistic. Acomplete fix would also plumb the IPC verdict into
AllReduceOp::setGroupTopology(cpp/tensorrt_llm/thop/allreduceOp.cpp) sothat C++ and Python agree from the start; the strategy downgrade above covers
the same ground from the Python side without touching C++.
concurrency 1) is a different failure from the rc22 init crash addressed here.
Their rc22 report supersedes it, and that is what this fix targets.
6. How I verified it
Constraints: no GPU and no
torchin this environment, sopytestcannot importtensorrt_llm. The pytest file was not executed. Instead:committed) loads the actual
tensorrt_llm/_ipc_utils.pyviaimportlibwiththe CUDA runtime,
Distributed,loggerandMappingstubbed, and runs sixscenarios. All assertions pass:
open_ipc=True,ipc_failed=False, non-null pointers, onehandle opened, two collectives, no warning.
cudaIpcOpenMemHandlereturnscudaErrorInvalidDevice(101), the reportedfailure: no exception,
ipc_failed=True, all pointers null, local bufferfreed, warning names the failing call.
opened handles closed, buffer freed, and it still warns rather than
degrading silently.
cudaIpcGetMemHandlefails locally: no handle is ever opened, buffer freed.cudaIpcOpenMemHandleis never called, so nouninitialized handle reaches the driver.
open_ipc=Falsewithipc_failed=False, no allocation, no collective. This is the case that mustnot disturb strategy selection.
ruffpinnedto the
v0.9.4this repo's pre-commit uses:ruff checkandruff format --diffclean on_ipc_utils.py,allreduce_helper.pyand the new test (non-legacy files).yapf --diffandisort --diffclean onops.py, which is inlegacy-files.txt, andscripts/legacy_utils.py lint-precommitreportsnothing on it.
Not verified: end-to-end
trtllm-bench --tp 8on affected hardware, and thatNCCL throughput is acceptable for this workload. Both need the reporter's
machine. The new tests are in
tests/unittest/_torch/distributed, which #16498moved from
l0_dgx_h100.ymlto the CPU-onlyl0_cpu.ymlon 2026-08-04, so CIruns them with zero GPUs. Everything there passes without a GPU; the three tests
that build an
AllReduceskip if the trtllm custom ops are not registered, andtest_lamport_skippedskips because the workspace helper it drives allocatesdevice tensors. The CUDA stubbing is
unittest.mock.patchscoped to a contextmanager and touches no real CUDA or MPI state, matching how
test_safe_mpi_comm.pyin the same directory already patches.Rebased onto
mainon 2026-08-05 (was 162 behind, now 0). Clean rebase: none ofthe three source files had upstream commits since the branch point.
Dev Engineer Review
AllReducefalls back to NCCL only after a genuine IPC exchange failure.get_allreduce_workspacereturn type is consistent with its callers.QA Engineer Review
tests/unittest/_torch/distributed/test_ipc_memory_fallback.py.tests/integration/test_lists/,test-db/, orqa/.