Skip to content

[Perf] Regressions on Multiple platforms, 8/8/2024 11:52:00 PM #106345

Closed

Description

@performanceautofiler

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.32 ns 40.57 ns 1.12 0.16 False
26.24 ns 28.18 ns 1.07 0.14 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'
Details

System.Tests.Perf_Version.TryParse3

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Version.TryParse2

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.10 ns 10.19 ns 1.26 0.06 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'
Details

System.Text.Perf_Ascii.Equals_Chars(Size: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.SearchValuesByteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.04 ns 10.56 ns 1.31 0.16 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.SearchValuesByteTests*'
Details

System.Buffers.Tests.SearchValuesByteTests.IndexOfAnyExcept(Values: "abcdefABCDEF0123456789Ü")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Xml.Tests.Perf_XmlConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
166.98 ns 185.41 ns 1.11 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Tests.Perf_XmlConvert*'
Details

System.Xml.Tests.Perf_XmlConvert.DateTime_ToString_Local

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
196.99 μs 218.85 μs 1.11 0.02 False
109.61 μs 131.62 μs 1.20 0.04 False
262.93 μs 302.78 μs 1.15 0.04 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'
Details

PerfLabTests.LowLevelPerf.StaticDelegate

ETL Files

Histogram

JIT Disasms

PerfLabTests.LowLevelPerf.ClassVirtualMethod

ETL Files

Histogram

JIT Disasms

PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodLongHierarchy

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_SubstringFrozenDictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
513.41 ns 566.84 ns 1.10 0.12 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_SubstringFrozenDictionary*'
Details

System.Collections.Perf_SubstringFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 100)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.75 μs 9.50 μs 1.09 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;String&gt;*'
Details

System.Collections.IndexerSet<String>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
48.20 ns 55.08 ns 1.14 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'
Details

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: JavaScript,no escaping required,512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.98 ns 11.46 ns 1.15 0.28 False
9.99 ns 11.91 ns 1.19 0.30 False
10.13 ns 11.26 ns 1.11 0.33 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
Details

System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.69 ns 5.91 ns 1.26 0.07 False
7.49 ns 9.33 ns 1.25 0.08 False
15.67 ns 22.77 ns 1.45 0.25 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'
Details

System.Memory.Span<Char>.BinarySearch(Size: 33)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.BinarySearch(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.Fill(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.82 ns 31.84 ns 1.11 0.13 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'
Details

System.Reflection.Invoke.StaticMethod4_int_string_struct_class

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.SearchValuesCharTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
6.19 ns 7.37 ns 1.19 0.33 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.SearchValuesCharTests*'
Details

System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAnyExcept(Values: "abcdefABCDEF0123456789")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
833.19 ns 1.04 μs 1.25 0.21 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'
Details

System.Collections.TryGetValueFalse<Int32, Int32>.ConcurrentDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.49 μs 2.94 μs 1.18 0.10 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;Int32&gt;*'
Details

System.Collections.IndexerSet<Int32>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
87.28 μs 105.50 μs 1.21 0.23 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
Details

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: None)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.40 μs 1.65 μs 1.18 0.25 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;Int16&gt;*'
Details

System.Collections.Perf_Frozen<Int16>.Contains_True(Count: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler commented on Aug 13, 2024

    @performanceautofiler
    Author

    Run Information

    Name Value
    Architecture x64
    OS ubuntu 22.04
    Queue TigerUbuntu
    Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
    Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
    Diff Diff
    Configs CompilationMode:tiered, RunKind:micro

    Regressions in System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Int32>

    Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
    135.79 ns 173.59 ns 1.28 0.05 False

    graph
    Test Report

    Repro

    General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

    git clone https://github.com/dotnet/performance.git
    python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives&lt;Int32&gt;*'
    Details

    System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Int32>.ShiftLeft(BufferLength: 3079)

    ETL Files

    Histogram

    JIT Disasms

    Docs

    Profiling workflow for dotnet/runtime repository
    Benchmarking workflow for dotnet/runtime repository

  2. LoopedBard3 commented on Aug 13, 2024

    @LoopedBard3
    Member

    Likely caused by: #106014, potentially due to unintended side effects. Compare range: 105bf4d...48aa315

    This is a mix of regressions that appear to potentially just be noise and ones that are clearly not just noise, so keeping them all.

  3. changed the title [-][Perf] Linux/x64: 24 Regressions on 8/8/2024 11:52:00 PM[/-] [+][Perf] Regressions on Multiple platforms, 8/8/2024 11:52:00 PM[/+] on Aug 13, 2024
  4. removed
    untriagedNew issue has not been triaged by the area owner
    on Aug 13, 2024
  5. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Aug 13, 2024
  6. removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Aug 16, 2024
  7. dotnet-policy-service commented on Aug 16, 2024

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @dotnet/area-meta
    See info in area-owners.md if you want to be subscribed.

  8. tarekgh commented on Aug 16, 2024

    @tarekgh
    Member

    Nothing related to metrics in this issue.

  9. added this to the 9.0.0 milestone on Aug 23, 2024
  10. removed
    untriagedNew issue has not been triaged by the area owner
    on Aug 23, 2024
  11. jeffhandley commented on Oct 20, 2024

    @jeffhandley
    Member

    Closing this out as many of the results show a bi-modal or multi-modal history, and the results are otherwise acceptable.

  12. locked and limited conversation to collaborators on Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions