Batched RMA on IBRC/IBDevX: device BATCH_RMA regions run at half the rate of plain puts, implicit MORE_FOLLOWS batching does not help (GH200 + InfiniBand data) #134

Related: roadmap #112 ("Batched RMA: explicit batching of iovec RMA operations and improved implicit batching for the IBRC and IBDevX remote proxy transports"), #105.

I measured the batching paths that are already on devel (2dca770) over InfiniBand, before proposing any code for the roadmap item. Three results seem worth sharing, and one question.

Setup: two nodes, one GH200 and one ConnectX-7 NDR port each, aarch64, CUDA 13.2, 2 PEs (1 per node), MPI bootstrap, both GPUs otherwise idle. Device perftests use 1 CTA × 32 threads, so every put goes through the CPU proxy. Numbers are logical operations per second (MMOPS), mean of 3–5 repetitions; stddev ≤ 0.01 unless noted.

1. Device-side BATCH_RMA regions run at about half the rate of plain puts on IBRC and IBDevX

shmem_region_nbi_rate --dir write, 8 B–4 KB (the rate does not depend on size, so one value per cell):

Two controls narrow it down:

• NVSHMEM_REGION_HOST_BATCH_MAX_OPS = 0 / 1 / 4 / 16 (the transport state applies it to the proxy role too) leaves the BATCH_RMA column unchanged (0.35–0.38 for both scopes). Submitting after every operation is as slow as holding all 64, so this is not batch deferral.
• With 64 operations per region, a region without hints costs only 4–6% over no region (more with 8 operations, where region start and stop amortize over fewer puts), so region bookkeeping and the per-request region metadata are not the main cost either.

That leaves something specific to the BATCH_RMA path between the device request and the transport submit, about 1.5 µs per operation on this machine. I could not profile the proxy thread here (no perf access). Is this expected for the current implementation, or is it worth digging into? I can run instrumented builds on this setup.

On the host side (region_nbi_rate, 64 operations per region) the hint does what it is meant to on IBRC: 2.2–2.6 vs 1.1–1.5 MMOPS for no region (noisy, stddev up to 0.9). On IBDevX it makes no difference.

2. IBRC and IBDevX ignore NVSHMEM_TRANSPORT_OP_FLAG_MORE_FOLLOWS, and honoring it does not help here

The proxy marks runs of adjacent puts to the same PE and QP with MORE_FOLLOWS (proxy_dma_more_follows, bounded by NVSHMEM_TRANSPORT_BATCH_MAX_OPS). Only the libfabric transport uses it (FI_MORE). IBRC and IBDevX batch only inside an explicit BATCH_RMA region, and otherwise post each operation on its own.

I prototyped the missing part: each transport holds a MORE_FOLLOWS run per caller role and submits it through the existing region submit path (a WR chain for IBRC, one doorbell per run for IBDevX). Held operations are flushed before any rma, atomic, fence or quiet of the same role. The runs do form. With NVSHMEM_DEBUG=TRACE, shmem_nbi_issue_rate on IBRC produced 3,086 batched submits for 14,644 puts (mean 4.75 per submit, many at 15), and the correctness tests pass (putmem_signal_slot_reuse all 256 offset pairs, device put). However, over three alternating rounds the rate did not change: shmem_nbi_issue_rate stays at 0.86–0.95 MMOPS for both builds, and shmem_put_bw is about 5% lower (64 KB: 15.7–16.0 vs 16.6–16.9 GB/s on IBRC). At roughly 1.1 µs per request the proxy thread is the bottleneck, not the doorbell. So I am not opening a PR for it. The branch is available if you want to look at it or measure it on x86, where the per-request proxy cost may be lower.

3. Explicit iovec batching: #105 numbers over InfiniBand

#105 compares one host-side nvshmemx_putmem_on_stream per fragment (v0) with application-side packing per destination (v2) on the same routing, with oracles on every round. On this pair (bottleneck-PE P50 of the full round, 20 measured rounds per rank, all oracles pass):

On IBRC the v0 dispatch phase costs about 11.4 µs per put (6.09 ms for 534 puts). That gap is what an iovec put, gathering several local ranges into one remote range or scattering one local range, would close without application packing. If such an API is planned, I would be glad to add a v3 variant to #105 that uses it and to test it on aarch64 + InfiniBand. I do not want to propose an API shape without knowing your direction.