[Bug] MPI group mailbox completion racing wait timeout can dereference empty frames and segfault #2899

Platform

All / Unknown — host-side MPI group mailbox; no NPU is needed to reproduce.

Runtime Variant

All / Unknown — independent of the device runtime.

Description

MpiGroupMailboxChannel::exchange_group_task() can dereference an empty group_frames_ after a condition-variable wait times out concurrently with batch completion.

The race is confirmed on Linux (aarch64) using the existing MpiGroupMailboxChannel.GroupArrivalTimeoutFailsTaskWithoutKillingGroup test and a debugger-controlled scheduling delay, without modifying repository source:

1. The last arriving rank moves group_frames_ into the leader's local storage and clears the shared vector.
2. The leader completes run_exchange(), publishes replies and group_done_ = true, then sets group_leader_running_ = false under group_mu_.
3. The waiting rank reacquires the mutex after its deadline, so wait_until() returns timeout even though the batch is now complete.
4. The timeout branch does not recheck group_done_. It interprets !group_leader_running_ as an incomplete rendezvous and indexes the empty group_frames_.
5. rank_frame.clear() segfaults.

This is a production synchronization defect, not just a test assertion failure. A stopped leader and a leader that never started are different states.

Steps to Reproduce

At the commit below, configure and build the existing target with debug symbols:

cmake -S tests/ut/cpp -B build/ut-rendezvous-debug -DCMAKE_BUILD_TYPE=Debug
cmake --build build/ut-rendezvous-debug --target test_remote_endpoint -j8

Use the project's normal Python/build dependencies when configuring. Save this debugger script as /tmp/rendezvous-repro.gdb:

set pagination off
set confirm off
break remote_endpoint.cpp:1020
commands
silent
printf "Holding the completed batch mutex past the follower deadline\n"
shell sleep 0.15
continue
end
run --gtest_filter=MpiGroupMailboxChannel.GroupArrivalTimeoutFailsTaskWithoutKillingGroup
thread apply all bt 8
frame 1
print group_done_
print group_leader_running_
print group_frames_

Run:

gdb -batch -x /tmp/rendezvous-repro.gdb build/ut-rendezvous-debug/test_remote_endpoint

Line 1020 is group_leader_running_ = false immediately after successful publication of group_done_, while the leader still holds group_mu_. If source lines move, place the breakpoint at that statement. The 150 ms debugger pause exceeds this test's 50 ms deadline; it changes scheduling only, not channel data or source code.

Observed locally: ordinary execution passed 500 consecutive repetitions; the controlled schedule triggered SIGSEGV on all 3 attempts. The pause is a reproduction aid, not a proposed production delay.

Expected Behavior

After reacquiring group_mu_, a waiter must check the authoritative completion predicate before treating the wait as an incomplete rendezvous. If completion has been published, it should consume the result or published error and perform normal departure accounting. A genuinely incomplete rendezvous should retain its existing timeout and recovery semantics.

Actual Behavior

Representative local stack, with machine-specific paths omitted:

Thread 2 received signal SIGSEGV, Segmentation fault.
#0 std::vector<unsigned char>::clear(this=0x0)
#1 MpiGroupMailboxChannel::exchange_group_task(... target_rank=0,
   task_slot=10, group_size=2)
   at src/common/hierarchical/remote_endpoint.cpp:1037
#2 test lambda
   at tests/ut/cpp/common/hierarchical/test_remote_endpoint.cpp:1114

State at the fault:

group_active_ = true
group_done_ = true
group_leader_running_ = false
group_arrived_ = 2
group_departed_ = 0
group_frames_ = empty
group_replies_ = two replies
group_error_ = null

A macOS CI run also segfaulted in this same test:
https://github.com/hw-native-sys/simpler/actions/runs/41981081113/job/127093413853

The job passed 449 of 450 C++ targets; test_remote_endpoint stopped after the RUN line for this case and CTest reported SEGFAULT. CI did not provide a crash stack, so the exact CI crash path is not independently proven. The local reproduction above does prove the production race.

Git Commit ID

30e365d

CANN Version

Not applicable; reproduction uses a host-only C++ unit test.

Driver Version

Not applicable; no device access.

Host Platform

Linux (aarch64), with a matching test-level SIGSEGV observed on macOS CI.

Additional Context

Origin: introduced by #1624, merge commit 43094d4415e4e5f789df49dbf6981ec033a0f2ec (2026-08-14). The entire current exchange_group_task() function is byte-for-byte identical to that merged version, and the function did not exist in its parent. #2862 does not modify the affected implementation or test file. The original timeout-recovery review is #1624 (comment).

Suggested repair: in the final completion wait, recheck group_done_ under the reacquired mutex before entering timeout cleanup, or use the predicate overload with equivalent semantics. Published completion must go through the existing result/error consumption and departure path. Do not merely increase the test timeout or silently skip empty-vector cleanup: that would hide or incompletely handle the state transition.

Acceptance criteria:

• Add a controlled regression for completion publication racing the deadline; verify the current implementation fails and the fix returns both replies without invalid access.
• Cover the same race with a published execution error, preserving that error and departure accounting.
• Retain incomplete-arrival timeout recovery: a later complete batch succeeds, without marking the group terminal solely for missing arrivals.
• Run the full test_remote_endpoint suite and macOS/Linux no-hardware CI.

Related but distinct: #2053 concerns a socket-connect/peer-RST test race; it does not fix this mailbox completion race. No other PR with this exact SIGSEGV was confirmed in the recent CI records inspected.