Skip to content

[mono][Perf] URI-related regressions on Mono: 12 Regressions on 9/10/2024 8:39:38 AM #107910

Closed

Description

@performanceautofiler

Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline 176754d0489c3a37a4e91304d52027d801d3fdfd
Compare 43f22c651184fe63935f5e509ab060d099d55f9d
Diff Diff
Configs CompilationMode:tiered, LLVM:true, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
262.62 ns 319.03 ns 1.21 0.11 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.CredentialCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.CredentialCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200
Details

System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString: "http://notfound", uriCount: 0)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline 176754d0489c3a37a4e91304d52027d801d3fdfd
Compare 43f22c651184fe63935f5e509ab060d099d55f9d
Diff Diff
Configs CompilationMode:tiered, LLVM:true, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
232.61 ns 277.06 ns 1.19 0.05 False
466.68 ns 541.03 ns 1.16 0.04 False
736.04 ns 865.83 ns 1.18 0.30 False
1.17 μs 1.28 μs 1.10 0.07 False
1.54 μs 1.70 μs 1.11 0.03 False
459.40 ns 554.61 ns 1.21 0.01 True
594.53 ns 694.06 ns 1.17 0.24 False
255.62 ns 311.39 ns 1.22 0.05 True
249.98 ns 309.01 ns 1.24 0.04 True
1.23 μs 1.40 μs 1.14 0.11 False
1.50 μs 1.69 μs 1.13 0.04 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Uri*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Uri* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Uri*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Uri* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200
Details

System.Tests.Perf_Uri.Ctor(input: "http://dot.net")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "https://a.much.longer.domain.name")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.DnsSafeHost

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://contoso.com/path/with?key=value#fragment")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://xn--hst-sna.with.xn--nicode-2ya/path/with?key=value#fragment")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "http://xn--hst-sna.with.xn--nicode-2ya")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.CombineAbsoluteRelative(input: "/new/path")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "https://CONTOSO.com")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "https://contoso.com")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path?key1=value1&key2=value2&key3=value3&key4=value4")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://a.much.longer.domain.name/path/with?key=value#fragment")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. changed the title [-][Perf] Linux/arm64: 12 Regressions on 9/10/2024 8:39:38 AM[/-] [+][mono][Perf] URI-related regressions on Mono: 12 Regressions on 9/10/2024 8:39:38 AM[/+] on Sep 17, 2024
  2. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Sep 17, 2024
  3. added and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Sep 17, 2024
  4. dotnet-policy-service commented on Sep 17, 2024

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @dotnet/ncl
    See info in area-owners.md if you want to be subscribed.

  5. matouskozak commented on Sep 17, 2024

    @matouskozak
    Member

    @stephentoub could the regressions be caused by #107357 ?

  6. stephentoub commented on Sep 17, 2024

    @stephentoub
    Member

    Is this one of the platforms that fails to accelerate SearchValues?

    cc: @MihaZupan

  7. matouskozak commented on Sep 17, 2024

    @matouskozak
    Member

    Is this one of the platforms that fails to accelerate SearchValues?

    cc: @MihaZupan

    Based on the ADX plot, it's hitting basically all Mono configs (JIT/Interpreter/AOT).

  8. stephentoub commented on Sep 17, 2024

    @stephentoub
    Member

    Thanks. Yeah, so if I'm reading things correctly (which a revised version of your plot that includes all target) it looks like coreclr improves across the board, and mono regresses across the board. Presumably that's something in SearchValues that mono doesn't accelerate?

  9. matouskozak commented on Sep 25, 2024

    @matouskozak
    Member

    @stephentoub could the regressions be caused by #107357 ?

    Sorry for the late reply. I did a bit of investigating on the codegen difference before and after your change and these are my findings:

    1. The SearchValue API calls into IndexOfAnyExcept which bubbles down to:
      private int IndexOfAny<TNegator>(ref char searchSpace, int searchSpaceLength)
      where TNegator : struct, IndexOfAnyAsciiSearcher.INegator
      {
      ref char searchSpaceEnd = ref Unsafe.Add(ref searchSpace, searchSpaceLength);
      ref char cur = ref searchSpace;
      uint[] bitmap = _bitmap;
      while (!Unsafe.AreSame(ref cur, ref searchSpaceEnd))
      {
      char c = cur;
      if (TNegator.NegateIfNeeded(Contains(bitmap, c)))
      {
      return (int)((nuint)Unsafe.ByteOffset(ref searchSpace, ref cur) / sizeof(char));
      }
      cur = ref Unsafe.Add(ref cur, 1);
      }
      return -1;
      }

    These Unsafe APIs should be intrinsified on Mono by

    emit_unsafe_intrinsics (MonoCompile *cfg, MonoMethod *cmethod, MonoMethodSignature *fsig, MonoInst **args)

    1. While looking at the interpreter and LLVM outputs I noticed that the new call System.Buffers.SearchValues'1<char>:IndexOfAnyExcept (System.ReadOnlySpan1)` doesn't get inlined. Do you know if CoreCLR is inlining this call and if that could be the reason why is it slower for Mono?
  10. added this to the 10.0.0 milestone on Sep 25, 2024
  11. removed
    untriagedNew issue has not been triaged by the area owner
    on Sep 27, 2024
  12. dotnet-policy-service commented on Jun 11, 2025

    @dotnet-policy-service
    Contributor

    Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
    See info in area-owners.md if you want to be subscribed.

  13. karelz commented on Jun 11, 2025

    @karelz
    Member

    Adding android label to fall into scope of @vitek-karas's team

  14. MihaZupan commented on Jun 18, 2025

    @MihaZupan
    Member

    While looking at the interpreter and LLVM outputs I noticed that the new call System.Buffers.SearchValues'1:IndexOfAnyExcept (System.ReadOnlySpan1)` doesn't get inlined. Do you know if CoreCLR is inlining this call and if that could be the reason why is it slower for Mono?

    Yes, it should. Since it's in a static readonly the JIT should see the exact type and inline away all the indirection such that the call site in Uri ends up with just a direct call to the core implementation. On most hardware that'd be this: IndexOfAnyCore

    private static TResult IndexOfAnyCore<TResult, TNegator, TOptimizations, TUniqueLowNibble, TResultMapper>(ref short searchSpace, int searchSpaceLength, ref AsciiState state)

    Or the implementation in BitmapCharSearchValues that you pointed to otherwise.

  15. BrzVlad commented on Jul 9, 2025

    @BrzVlad
    Member

    I don't think this regression is significant enough to further investigate.

  16. locked and limited conversation to collaborators on Aug 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions