OMP: Error #15: Initializing libiomp5.dylib in Image Classification with AutoML

[justinormont]

System information

• OS version/distro:

OS Name: Mac OS X
OS Version: 10.14
OS Platform: Darwin
RID: osx.10.14-x64
Base Path: /usr/local/share/dotnet/sdk/5.0.101/

• .NET Version (eg., dotnet --info):

Version: 5.0.101
Commit: d05174dc5a

Issue

• What did you do?
  I was creating a new ML․NET sample showcasing multi-modal (photos, text, categorical, numeric) techniques and model stacking.

• What happened?
  When I stack an AutoML model using image features, I get the below error.

Error:

OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented work a round you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

This seems to occur only after adding the second AutoML model which works on the image features:
https://github.com/justinormont/ImgurClassifier/blob/13b034fa311e0e314a74faf955066ca276296be7/ImgurClassifier.ConsoleApp/ModelBuilder.cs#L154-L169

The same image pipeline is present (and causes no errors) within the main pipeline:
https://github.com/justinormont/ImgurClassifier/blob/13b034fa311e0e314a74faf955066ca276296be7/ImgurClassifier.ConsoleApp/ModelBuilder.cs#L222-L232

Speculation on cause:

• Error didn't occur until an AutoML stacked model was created for the image features. Perhaps it occurs when having two sets of the image featurizers?
• The error is possibly/likely system dependent (and on macOS)
• NuGet runtime.osx.10.10-x64.CoreCompat.System.Drawing is included in the project to get around a libgdiplus error; this may link to a separate version of libomp

Source code / logs

Repro

Checkout sample at current commit:

git clone https://github.com/justinormont/ImgurClassifier.git
cd ImgurClassifier
git checkout 13b034fa311e0e314a74faf955066ca276296be7

Runtime is expected to be ~2hr, and fails at ~30-45min. You may want reduce the AutoML time to 0 seconds (code, code), so only one model will be created. Though, the error may be occurring within the fourth model created (SymbolicSgdLogisticRegressionOva); I'll catch the stack trace on my next run.

Anti-Repro

The error doesn't occur if the second AutoML sub-pipeline (using images) is removed, even though the same image featurizers are still utilized in the main pipeline.