Skip to content

[mono][Perf] System.Tests.Perf_Type.GetType_Resolvers regression on 4/24/2024 9:24:37 PM #106402

Closed as not planned
Closed as not planned
[mono][Perf] System.Tests.Perf_Type.GetType_Resolvers regression on 4/24/2024 9:24:37 PM#106402
Assignees
adamsitnik
Labels
PGOarch-arm64area-System.Reflection.Metadatabinaryformatter-migrationIssues related to the removal of BinaryFormatter and migrations away from itinvestigateos-linuxLinux OS (any supported distro)regression-from-last-releaseruntime-monospecific to the Mono runtime
Milestone
9.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on May 3, 2024

Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline edb6e7c154a4a75c8d1430cd6a996e6f1f9b401e
Compare 74d927967dcf9dc1801c310603d6dc843dc28ad6
Diff Diff
Configs CompilationMode:tiered, LLVM:true, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_Type

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.19 μs 1.37 μs 1.15 0.31 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Type*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Type* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Type*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Type* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200
Details

System.Tests.Perf_Type.GetType_Resolvers(input: typeof(System.Tests.Nested))

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    added
    arch-arm64
    os-linuxLinux OS (any supported distro)
    PGO
    runtime-monospecific to the Mono runtime
    untriagedNew issue has not been triaged by the area owner
    on May 3, 2024

  2. matouskozak commented on May 7, 2024

    @matouskozak
    matouskozak
    on May 7, 2024
    Member

    The range of regression is d92ac1f...5b4e770. Could it be the library changes from #100094 @adamsitnik ?

  3. matouskozak
    removed
    untriagedNew issue has not been triaged by the area owner
    on May 7, 2024

  4. matouskozak commented on Jun 27, 2024

    @matouskozak
    matouskozak
    on Jun 27, 2024
    Member

    @adamsitnik do you think it could be related to #100094?

  5. kotlarmilos commented on Aug 1, 2024

    @kotlarmilos
    kotlarmilos
    on Aug 1, 2024
    Member

    This looks like noise.

  6. matouskozak
    transferred this issue fromdotnet/perf-autofiling-issueson Aug 14, 2024
  7. ghost added
    area-Infrastructure-mono
    on Aug 14, 2024
  8. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on Aug 14, 2024
  9. matouskozak
    added
    area-System.Reflection.Metadata
    and removed
    area-Infrastructure-mono
    on Aug 14, 2024

  10. dotnet-policy-service commented on Aug 14, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Aug 14, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @directhex, @matouskozak
    See info in area-owners.md if you want to be subscribed.

  11. dotnet-policy-service commented on Aug 14, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Aug 14, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @dotnet/area-system-reflection-metadata
    See info in area-owners.md if you want to be subscribed.

  12. matouskozak commented on Aug 14, 2024

    @matouskozak
    matouskozak
    on Aug 14, 2024
    Member

    This looks like noise.

    I think this could be a real regression based on the trend:
    image

    @adamsitnik do you think that your change #100094 could have caused this?

  13. steveharter
    added
    investigate
    and removed
    untriagedNew issue has not been triaged by the area owner
    on Aug 14, 2024
  14. steveharter
    added this to the 9.0.0 milestone on Aug 14, 2024
  15. matouskozak
    changed the title [-][Perf] Linux/arm64: 1 Regression on 4/24/2024 9:24:37 PM[/-] [+][mono][Perf] System.Tests.Perf_Type.GetType_Resolvers regression on 4/24/2024 9:24:37 PM[/+] on Aug 15, 2024
  16. matouskozak
    added
    regression-from-last-release
    on Aug 15, 2024
  17. ericstj
    added
    binaryformatter-migrationIssues related to the removal of BinaryFormatter and migrations away from it
    on Aug 19, 2024

  18. dotnet-policy-service commented on Aug 19, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Aug 19, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to 'binaryformatter-migration': @adamsitnik, @bartonjs, @jeffhandley, @terrajobst

  19. ericstj commented on Aug 28, 2024

    @ericstj
    ericstj
    on Aug 28, 2024
    Member

    @adamsitnik it seems this is a regression in GetType from the type name parser changes. Could you please have a look?

  20. ericstj
    assigned
    adamsitnik
    on Aug 28, 2024

  21. adamsitnik commented on Aug 29, 2024

    @adamsitnik
    adamsitnik
    on Aug 29, 2024
    Member

    Yes, this is related to TypeName parser changes. The parser got unified for CLR, Mono and NativeAOT. In most cases, it's faster:

    https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu%2022.04_LLVM%3dfalse_MonoAOT%3dfalse_MonoInterpreter%3dfalse_RunKind%3dmicro_mono/System.Tests.Perf_Type.GetType_Resolvers(input%3a%20typeof(int)).html

    image

    https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu%2022.04_LLVM%3dfalse_MonoAOT%3dfalse_MonoInterpreter%3dfalse_RunKind%3dmicro_mono/System.Tests.Perf_Type.GetType_Resolvers(input%3a%20typeof(System.Tests.Nested)).html

    image

    https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu%2022.04_LLVM%3dfalse_MonoAOT%3dfalse_MonoInterpreter%3dfalse_RunKind%3dmicro_mono/System.Tests.Perf_Type.GetType_Resolvers(input%3a%20typeof(System.Int32%5b%5d)).html

    image

    The last image is Mono x64 Ubuntu same benchmark as the one reported here and as you can see it has improved. So there must be something in Mono arm64 AOT codegen that makes it slower. Since parsing nested types is rather a niche scenario I believe it's fair to say it's a good tradeoff (unified parser & a lot of perf improvements) and close the issue.

  22. adamsitnik
    closed this as not plannedon Aug 29, 2024
  23. github-actions
    locked and limited conversation to collaborators on Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

PGOarch-arm64area-System.Reflection.Metadatabinaryformatter-migrationIssues related to the removal of BinaryFormatter and migrations away from itinvestigateos-linuxLinux OS (any supported distro)regression-from-last-releaseruntime-monospecific to the Mono runtime

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions