Skip to content

Set OMPI env var to prevent mpirun failures due to race conditions - #60

Open
bartgol wants to merge 1 commit into
mainfrom
bartgol/ompi-env-fix
Open

bartgol wants to merge 1 commit into
mainfrom
bartgol/ompi-env-fix

Conversation

@bartgol

@bartgol bartgol commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

The env var prevents mpirun to return a non-zero exit code just b/c a race condition completely internal to MPI implementation has caused one of the rank to exit before the MPI runtime expected it to. This env var forces mpirun to allow rank to exit without waiting for ORTE to send an acknowledgement that it received their exit request (or something along that line).


For context, the error I was seeing was of this form:

152/688 Test  #76: grid_np1 ..............................................................***Failed   28.50 sec
HAS RESOURCE SPECS
RUN: OMP_PROC_BIND=spread OMP_PLACES=threads taskset -c 22 ./grid --use-colour no
FROM: /home/e3sm/e3sm/components/eamxx/ctest-build/ghci-snl-gnu/release/src/share/grid/tests
Calling initialize_kokkos
 Default Execution Space name: Serial
 #host threads: 1
-------- EKAT CONFIGS --------
Active AVX settings: 
Compiler Id: GCC
 Default Execution Space name: Serial
 #host threads: 1
-------- EAMXX CONFIGS --------
 sizeof(Real) = 8
 default pack size = 16
 default FPE mask: 0 (NONE) 
-------------------------------
Starting catch session on rank 0 out of 1
 For test get_owners, random number generator seed: 303771390
    Note: catch rng seed was 0 (default). We interpret that as a request to pick a random seed.
    To reproduce a previous run, use --rng-seed N to provide the rng seed.
 For test gid2lid_map, random number generator seed: 791464951
    Note: catch rng seed was 0 (default). We interpret that as a request to pick a random seed.
    To reproduce a previous run, use --rng-seed N to provide the rng seed.
 For test get_remote_pids_and_lids, random number generator seed: 1178953384
    Note: catch rng seed was 0 (default). We interpret that as a request to pick a random seed.
    To reproduce a previous run, use --rng-seed N to provide the rng seed.
===============================================================================
All tests passed (43 assertions in 3 test cases)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 0 on
node mappy exiting improperly. There are three reasons this could occur:
1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.
2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"
3. this process called "MPI_Abort" or "orte_abort" and the mca parameter
orte_create_session_dirs is set to false. In this case, the run-time cannot
detect that the abort call was an abnormal termination. Hence, the only
error message you will receive is this one.
This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
You can avoid this message by specifying -quiet on the mpirun command line.
--------------------------------------------------------------------------

As you can see, the test itself passed. I even added a print statement in eamxx's test-launcher to print the executable rc, and it was always 0, even when MPI crapped out. So this env var is truly the right thing to do for us.

Copilot AI lite review requested due to automatic review settings September 22, 2026 20:59
@bartgol
bartgol requested review from mahf708 and removed request for Copilot September 22, 2026 21:00
@bartgol bartgol self-assigned this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants