Skip to content

test: check signal payloads across acknowledged slot reuse - #118

Open
0z5a wants to merge 1 commit into
NVIDIA:develfrom
0z5a:codex/s2-signal-slot-reuse
Open

0z5a wants to merge 1 commit into
NVIDIA:develfrom
0z5a:codex/s2-signal-slot-reuse

Conversation

@0z5a

@0z5a 0z5a commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Existing signal tests do not exercise repeated consumption and acknowledged reuse of per-peer destination slots across byte alignment combinations. Add a focused device test that checks every payload byte and source/destination red zone on all directed PE edges, records the first mismatch, and returns a failure status to the launcher.

Each signal and ACK carries a 64-bit generation. The sender waits for the previous consumption ACK before recycling a destination slot and completes its device-side source accesses before refilling the source buffer. A single consumer CTA waits, synchronizes its readers, validates the payload, and acknowledges only after the reads finish. The test uses the existing initialization and finalization helpers and a collective kernel launch.

The default is 257 bytes, four slots and 32 epochs. Optional arguments select payload size, slot count, epochs, independent source/destination residues modulo 16, and alternating submission/consumption delay. --all-offsets runs all 256 residue pairs in one process, with completion and a PE barrier between configurations. CMake registration and usage documentation are included.

H100 validation at the original commit 8a5d6fbfef997e28439b1647d253d898704b30d0. The current head f51f7bd15e96bb4b121142db000197e0fef58a12 is the same change rebased onto devel 2dca770 with a DCO sign-off and a commit message; no code differs from 8a5d6fb.

  • Built the matching devel host/device libraries, MPI bootstrap, IBRC/IBGDA transports and test with CUDA 13.2 / SM90 / GNU 11.5 / Open MPI 4.1.5. Base: cffd2a512787cdd08dce21453c3296b2908ad432 (development artifact 3.9.0.dev0).
  • Checked 25 parser cases and 112 guard-layout cases on the host. Five invalid invocations of the compiled native executable returned 2 before CUDA initialization. The actual build, loader and CPU step waits returned 0.
  • Completed 41 independent process starts on two hosts with four distinct H100 PEs per host. All 328 native children returned 0, and their PID/start-time/GPU UUID joins were independently checked. All 28,888 rank/configuration records passed full byte payload and source/destination guard checks.
  • The matrix covered 8, 64, 256, 257, 4096, 65536 and 1048577 bytes with IBGDA disabled/enabled; all 256 source/destination residues for each size and transport arm; slots 2/4/8; delay 0/8192/65536 cycles; RC counts 1/2/4; and RC mapping none/cta/sm/warp. The configured RC count and mapping were checked against every rank's actual initialization logs.
  • Enabled-arm logs selected IBGDA devices and GPU-resident NIC buffers, with the NIC handler using the CPU mapping backend. This validation does not cover the GPU NIC-handler backend. Complete original outputs were archived and verified by two full readers; the driver, node guards and finalizer actually joined, and original GPU/I/O locks were released and independently reacquired.

This is a regression test contribution motivated by signal/payload visibility and reuse questions such as #75 and #84. It does not establish a new transport defect, measure per-QP WQE counter wraparound, or report performance.

Cross-node validation (added after rebase onto devel 2dca770)

2× GH200 (aarch64, CUDA 13.2, driver 595.58, ConnectX-7 NDR, 1 PE per node), so every edge is a remote edge. 30/30 runs exit 0, every record "status":"PASS" with wrong_payload_or_guard 0 (32 directed messages per rank per configuration; 16384 per rank for --all-offsets):

Transport 257 B / 4 slots / 32 epochs / offsets 3,11 / delay 8192 --all-offsets (256 residue pairs) 8 / 4096 / 65536 / 1048577 B
IBRC (proxy) PASS (2/2 records) PASS (512/512 records) PASS ×4
IBDevX (proxy) PASS PASS (512/512) PASS ×4
IBGDA, GPU NIC handler, RC_MAP_BY=none PASS PASS (512/512) PASS ×4
IBGDA, GPU NIC handler, RC_MAP_BY=cta PASS PASS (512/512) PASS ×4
IBRC with NVSHMEM_HEAP_KIND=SYSMEM (heap in Grace memory) PASS PASS (512/512) PASS ×4

The IBGDA rows exercise the GPU NIC handler, which the original H100 validation (CPU handler) did not cover. This test is the kind of check discussed in #75 (nvshmem_signal_wait_until visibility); it does not change the library.

Existing signal tests do not exercise repeated consumption and acknowledged
reuse of per-peer destination slots across byte alignment combinations. Add a
focused device test that checks every payload byte and the source/destination
red zones on all directed PE edges, records the first mismatch, and returns a
failure status to the launcher.

Each signal and ACK carries a 64-bit generation. The sender waits for the
previous consumption ACK before recycling a destination slot and completes
its device-side source accesses before refilling the source buffer. A single
consumer CTA waits, synchronizes its readers, validates the payload, and
acknowledges only after the reads finish.

Defaults: 257 bytes, four slots, 32 epochs. Options select payload size, slot
count, epochs, independent source/destination residues modulo 16, and an
alternating submission/consumption delay; --all-offsets runs all 256 residue
pairs in one process.

Motivated by the signal visibility and reuse questions in NVIDIA#75 and NVIDIA#84.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@0z5a
0z5a force-pushed the codex/s2-signal-slot-reuse branch from 8a5d6fb to f51f7bd Compare October 26, 2026 13:56
@0z5a
0z5a marked this pull request as ready for review October 26, 2026 13:57
@0z5a

0z5a commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

More data for the test, on GPU-to-GPU paths without a NIC:

  • B300, 2 PEs on one node over NVSwitch (P2P with native atomics), devel 2dca770 plus this PR, library built for sm_103: default parameters 2/2 PASS; --all-offsets 512/512 PASS with wrong_payload_or_guard 0 on every line; payload sizes 8, 4096, 65536 and 1048577 bytes 2/2 PASS each.
  • Two RTX PRO 6000 (sm_120) over PCIe, P2P without native atomics: default parameters 2/2 PASS and --all-offsets 512/512 PASS, with and without NVSHMEM_CFT_HANDLES_SUPPORT.

Together with the InfiniBand runs in the description, the test now passes on IB, PCIe P2P and NVLink.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant