High RAM memory usage when using CGNS mesh format

[jf-thomas]

Issue description

Hi all, I am trying to run a rather large mesh computation with SU2. The mesh counts 170M cells. I managed to run it with the native SU2 mesh format with 480 cores but the preprocessing step took 90 minutes with a huge load on the file system impacting all the other users of the cluster.

To reduce the load, I was thinking about using the CGNS mesh file format. However the preprocessing step fills all the RAM memory available and the job fails. All the setup being identical.

By reducing the number of cells, I am able to run with the CGNS file format. Is it a common behavior with the CGNS file format?

Hardware and software :

• OS: Linux Red Hat ELS 6.5
• Nodes: Intel Xeon Ivy bridge
• C++ compiler and version: intel/2018.0.1
• MPI implementation and version: intelmpi/2018.0.1
• SU2 Version: v7.2

[pcarruscag]

Hi, I've seen this with some (one) cgns file but not others (larger ones), for example I never had any issue with the finest HL-CRM reference grids on nodes with 128 cores.
I suspect this has something to do with how the files are generated, but I did not investigate further. I think the HL-CRM are made by Pointwise but I do not know the export settings used, do let us know if you find a solution.
Regarding the ASCII format, we have recently improved the mesh reader (in the develop branch), if you are generating the file yourself, and if you list the "points" before the "elements" in the file, it may be up to 4 times faster to read it.
Finally, the file systems used in the clusters of some very fine institutions are absolute trash (not saying this is your case, just making you aware of the fact), and they respond very badly to having many cores reading the same file. Especially for ASCII because here all cores read the entire file, and not just some sections.
If you run the hybrid parallel version of SU2 this will be less of a problem because fewer cores will be participating in file-IO (but all of them will still be used for computation).
https://su2code.github.io/docs_v7/Build-SU2-Linux-MacOS/#basic-configuration
https://su2foundation.org/wp-content/uploads/2020/06/Gomes.pdf
Cheers,
Pedro

[jf-thomas]

Hi, I can confirm that the ASCII mesh reader has been improved a lot in version 7.2.1. This reduces the need for CGNS format. Hybrid parallel helps also, thanks for the hints. I did not find a way for the CGNS to work yet.

[MicK7]

Hi @jf-thomas which version of CGNS are you using ? Maybe applying a cgnsupdate on the mesh file prior to running SU2 might help since SU2 rely on recent cgns version to reduce memory footprint.

[jf-thomas]

Hi @MicK7, I am using CGNS version 2.55 which comes from the meshing tool. I did not know about the cgnsupdate tool, I will try to upgrade the CGNS with this. Thanks

[jf-thomas]

That's great, I did convert to CGNS 4.2.0 and it's working nicely now ! There was a trick, I had to first convert to hdf5 (adf2hdf), cgnsupdate and then hdf2adf.
Thank you all for the help.

[raphobt]

Hi all,
I am facing the same issue as @jf-thomas with SU2 v7.5. The only persistent problem is that updating CGNS mesh version from 3.2 to 4.3 now raises a conflict with the CGNS library version and the run fails.

I acknowledge the warning message following this thread when using old CGNS versions:

WARNING: The CGNS file version (3.21)  is old and may cause high memory usage issues, consider updating the file with the cgnsupdate tool.

However, with the updated CGNS mesh 4.3:

*** Warning:The file being read is more recent that the CGNS library used ***

Checking in SU2/externals/cgns/cgnslib.c where the above message is encoded, there is a comment stating that:

/* This code allows reading version newer than the lib, as long as the 1st digit of the versions are equal */

The same issue is manifest with SU2 v8.
Should one only use the exact CGNS 4.0? I wonder if @jf-thomas faced this issue with v4.2 before doing the trick with hdf5?
Thanks

[MicK7]

Maybe you should open a separate issue to have more visibility.
The solution comes down to upgrading the cgns version of su2 (see #2179)