Repository navigation
Threading tests that fail on Derecho #2296
Description
Brief summary of bug
These are failing with seg faults on the release-clm5.0 branch. I didn't see this same behavior on the release-cesm2.2 or master branches, but the tests were also slightly different.
ERS_D_Ld5_P512x3.f19_g16.I2000Clm50SpGs.derecho_intel.clm-waccmx_offline
ERP_P180x2_D.f19_g17.I2000Clm50SpRtmFl.derecho_intel.clm-default
ERP_P180x2_D_Ld5.f19_g17.I2000Clm50BgcDvCrop.derecho_intel.clm-crop
LII2FINIDATAREAS_D_P360x2_Ld1.f09_g16_gl4.I1850Clm50BgcCrop.derecho_intel.clm-compatible_finidat_f09
General bug information
CTSM version you are using: release-clm5.0.36-10-gfd829cf5d
Does this bug cause significantly incorrect results in the model's science? No
Configurations affected:
Some threaded layouts on Derecho. I'm not sure what makes some of these run slow and these to fail.
Important output or errors that show the problem
For ERP_D_P36x2_Ld3.f10_f10_musgs.I2000Clm50BgcCruGs.derecho_intel.clm-flexCN_FUN
cesm.log:
dec2447.hsn.de.hpc.ucar.edu 0: MCT::m_Router::initp_: GSMap indices not increasing...Will correct
dec2447.hsn.de.hpc.ucar.edu 31: WARNING: qice out of bounds: g, n, qice = 248 1
dec2447.hsn.de.hpc.ucar.edu 31: -13.5012076451853
dec2447.hsn.de.hpc.ucar.edu 1: Setting mpi info: striping_factor=4
dec2447.hsn.de.hpc.ucar.edu 1: Setting mpi info: striping_unit=1048576
dec2447.hsn.de.hpc.ucar.edu 1: Setting mpi info: striping_factor=4
dec2447.hsn.de.hpc.ucar.edu 1: Setting mpi info: striping_unit=1048576
dec2447.hsn.de.hpc.ucar.edu 2: double free or corruption (fasttop)
dec2447.hsn.de.hpc.ucar.edu 2: forrtl: error (76): Abort trap signal
dec2447.hsn.de.hpc.ucar.edu 2: Image PC Routine Line Source
dec2447.hsn.de.hpc.ucar.edu 2: libpthread-2.31.s 00001478FFB018C0 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBBD3CBB gsignal Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBBD5355 abort Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBC19AE7 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBC21B6A Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBC23744 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBC25E00 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libc-2.31.so 00001478FBC2704F realloc Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 0000000003C3056E Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 0000000003C31255 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 0000000003C24EE0 perf_mod_mp_t_sta 760 perf_mod.F90
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 00000000008A9F0A clm_driver_mp_clm 1012 clm_driver.F90
dec2447.hsn.de.hpc.ucar.edu 2: libiomp5.so 0000147900E98493 __kmp_invoke_micr Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libiomp5.so 0000147900E06533 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libiomp5.so 0000147900E077B8 __kmp_fork_call Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: libiomp5.so 0000147900DBED23 __kmpc_fork_call Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 0000000000884C2F clm_driver_mp_clm 417 clm_driver.F90
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 00000000008521F5 lnd_comp_mct_mp_l 456 lnd_comp_mct.F90
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 0000000000472004 component_mod_mp_ 728 component_mod.F90
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 0000000000441684 cime_comp_mod_mp_ 2720 cime_comp_mod.F90
dec2447.hsn.de.hpc.ucar.edu 2: cesm.exe 000000000045A12B MAIN__ 125 cime_driver.F90
.
.
.
<While other tasks fail in a barrier as follows...
.
dec2447.hsn.de.hpc.ucar.edu: rank 2 died from signal 6 and dumped core
dec2447.hsn.de.hpc.ucar.edu 3: forrtl: error (78): process killed (SIGTERM)
dec2447.hsn.de.hpc.ucar.edu 3: Image PC Routine Line Source
dec2447.hsn.de.hpc.ucar.edu 3: libpthread-2.31.s 000014CE84DF28C0 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: libmpi_intel.so.1 000014CE83A6FE48 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: libmpi_intel.so.1 000014CE83B46895 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: libmpi_intel.so.1 000014CE81E56BE0 Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: libmpi_intel.so.1 000014CE81E5711F PMPI_Barrier Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: libmpifort_intel. 000014CE84976DCB pmpi_barrier Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 000000000088574E clm_driver_mp_clm 1095 clm_driver.F90
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 00000000008521F5 lnd_comp_mct_mp_l 456 lnd_comp_mct.F90
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 0000000000472004 component_mod_mp_ 728 component_mod.F90
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 0000000000441684 cime_comp_mod_mp_ 2720 cime_comp_mod.F90
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 000000000045A12B MAIN__ 125 cime_driver.F90
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 000000000041917D Unknown Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: libc-2.31.so 000014CE80EAF29D __libc_start_main Unknown Unknown
dec2447.hsn.de.hpc.ucar.edu 3: cesm.exe 00000000004190AA Unknown Unknown Unknown
Activity
ekluzek commentedon Feb 5, 2025
I think this was due to an issue in ccs_config and that this is resolved now. We have passing threading tests now.