Repository navigation
Conversation
Existing signal tests do not exercise repeated consumption and acknowledged reuse of per-peer destination slots across byte alignment combinations. Add a focused device test that checks every payload byte and the source/destination red zones on all directed PE edges, records the first mismatch, and returns a failure status to the launcher. Each signal and ACK carries a 64-bit generation. The sender waits for the previous consumption ACK before recycling a destination slot and completes its device-side source accesses before refilling the source buffer. A single consumer CTA waits, synchronizes its readers, validates the payload, and acknowledges only after the reads finish. Defaults: 257 bytes, four slots, 32 epochs. Options select payload size, slot count, epochs, independent source/destination residues modulo 16, and an alternating submission/consumption delay; --all-offsets runs all 256 residue pairs in one process. Motivated by the signal visibility and reuse questions in NVIDIA#75 and NVIDIA#84. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
0z5a
force-pushed
the
codex/s2-signal-slot-reuse
branch
from
October 26, 2026 13:56
8a5d6fb to
f51f7bd
Compare
0z5a
marked this pull request as ready for review
October 26, 2026 13:57
Contributor
Author
|
More data for the test, on GPU-to-GPU paths without a NIC:
Together with the InfiniBand runs in the description, the test now passes on IB, PCIe P2P and NVLink. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Existing signal tests do not exercise repeated consumption and acknowledged reuse of per-peer destination slots across byte alignment combinations. Add a focused device test that checks every payload byte and source/destination red zone on all directed PE edges, records the first mismatch, and returns a failure status to the launcher.
Each signal and ACK carries a 64-bit generation. The sender waits for the previous consumption ACK before recycling a destination slot and completes its device-side source accesses before refilling the source buffer. A single consumer CTA waits, synchronizes its readers, validates the payload, and acknowledges only after the reads finish. The test uses the existing initialization and finalization helpers and a collective kernel launch.
The default is 257 bytes, four slots and 32 epochs. Optional arguments select payload size, slot count, epochs, independent source/destination residues modulo 16, and alternating submission/consumption delay.
--all-offsetsruns all 256 residue pairs in one process, with completion and a PE barrier between configurations. CMake registration and usage documentation are included.H100 validation at the original commit
8a5d6fbfef997e28439b1647d253d898704b30d0. The current headf51f7bd15e96bb4b121142db000197e0fef58a12is the same change rebased ontodevel2dca770with a DCO sign-off and a commit message; no code differs from8a5d6fb.cffd2a512787cdd08dce21453c3296b2908ad432(development artifact3.9.0.dev0).This is a regression test contribution motivated by signal/payload visibility and reuse questions such as #75 and #84. It does not establish a new transport defect, measure per-QP WQE counter wraparound, or report performance.
Cross-node validation (added after rebase onto devel 2dca770)
2× GH200 (aarch64, CUDA 13.2, driver 595.58, ConnectX-7 NDR, 1 PE per node), so every edge is a remote edge. 30/30 runs exit 0, every record
"status":"PASS"withwrong_payload_or_guard0 (32 directed messages per rank per configuration; 16384 per rank for--all-offsets):--all-offsets(256 residue pairs)NVSHMEM_HEAP_KIND=SYSMEM(heap in Grace memory)The IBGDA rows exercise the GPU NIC handler, which the original H100 validation (CPU handler) did not cover. This test is the kind of check discussed in #75 (
nvshmem_signal_wait_untilvisibility); it does not change the library.