Skip to content

[Perf] Linux/arm64: 2 Regressions on 4/30/2024 1:50:24 PM #102281

Closed
#103714
Closed
[Perf] Linux/arm64: 2 Regressions on 4/30/2024 1:50:24 PM#102281
#103714
Assignees
rzikm
Labels
arch-arm64area-System.Netos-linuxLinux OS (any supported distro)runtime-monospecific to the Mono runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark
Milestone
9.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on May 7, 2024

Run Information

Name Value
Architecture arm64
OS ubuntu 22.04
Queue AmpereUbuntu
Baseline f9207e6c85bb6aebd9e7af7504fd3db28b32d058
Compare 017593d90781d797df8b2241f6d1f83c236c442b
Diff Diff
Configs CompilationMode:tiered, LLVM:true, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.72 μs 1.80 μs 1.05 0.09 False
1.41 μs 1.51 μs 1.07 0.05 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.CredentialCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Net.Primitives.Tests.CredentialCacheTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200
Details

System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString: "http://name5", uriCount: 10)

ETL Files

Histogram

JIT Disasms

System.Net.Primitives.Tests.CredentialCacheTests.GetCredential_Uri(uriString: "http://notfound", uriCount: 10)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    added
    arch-arm64
    os-linuxLinux OS (any supported distro)
    runtime-monospecific to the Mono runtime
    untriagedNew issue has not been triaged by the area owner
    on May 7, 2024

  2. matouskozak commented on May 7, 2024

    @matouskozak
    matouskozak
    on May 7, 2024
    Member

    The range of regression is f9207e6...8283049. Could it be #101053 @rzikm ?

  3. matouskozak
    removed
    untriagedNew issue has not been triaged by the area owner
    on May 7, 2024

  4. rzikm commented on May 9, 2024

    @rzikm
    rzikm
    on May 9, 2024
    Member

    Sounds plausible, I can have a look when I have time.

  5. matouskozak
    assigned
    rzikm
    on May 9, 2024
  6. matouskozak
    transferred this issue fromdotnet/perf-autofiling-issueson May 15, 2024
  7. ghost added
    area-Infrastructure-mono
    on May 15, 2024
  8. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on May 15, 2024

  9. dotnet-policy-service commented on May 15, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on May 15, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @directhex
    See info in area-owners.md if you want to be subscribed.

  10. matouskozak
    added
    area-System.Net
    tenet-performancePerformance related issue
    tenet-performance-benchmarksIssue from performance benchmark
    and removed
    area-Infrastructure-mono
    on May 15, 2024

  11. dotnet-policy-service commented on May 15, 2024

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on May 15, 2024 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @dotnet/ncl
    See info in area-owners.md if you want to be subscribed.

  12. wfurt commented on May 15, 2024

    @wfurt
    wfurt
    on May 15, 2024
    Member

    It seems like all the interesting part is under if (preAuthenticate) that I doubt is used in perf tests....

  13. rzikm
    closed this as completedin #103714on Jun 25, 2024
  14. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged by the area owner
    on Jun 25, 2024
  15. github-actions
    locked and limited conversation to collaborators on Jul 26, 2024
  16. karelz
    added this to the 9.0.0 milestone on Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-arm64area-System.Netos-linuxLinux OS (any supported distro)runtime-monospecific to the Mono runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions