[Perf] Regressions from "Remove BBJ_NONE" #95646

Activity

1. performanceautofiler

   added

   arch-x64

   os-linuxLinux OS (any supported distro)Linux OS (any supported distro)

   PGO

   runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Dec 5, 2023
2. performanceautofiler commented on Dec 5, 2023

   

   performanceautofilerbot

   on Dec 5, 2023 – with PerformanceAutoFiler

   Last edited by performanceautofiler

   Author

   More actions

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.ContainsFalse<String>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   FrozenSet - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	4.89 μs	5.23 μs	1.07	0.03	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<String>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.ContainsFalse<String>.FrozenSet(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in PerfLabTests.CastingPerf

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   ObjFooIsObj2 - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	206.14 μs	218.82 μs	1.06	0.02	False
   IFooFooIsIFoo - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	206.25 μs	218.05 μs	1.06	0.03	False

   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'

   Details

   Payloads

   Baseline
   Compare

   PerfLabTests.CastingPerf.ObjFooIsObj2

   ETL Files

   Histogram

   JIT Disasms

   PerfLabTests.CastingPerf.IFooFooIsIFoo

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   DivisionOperatorBenchmark - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	27.51 ns	31.92 ns	1.16	0.03	True

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Int16>*'

   Details

   Payloads

   Baseline
   Compare

   System.Numerics.Tests.Perf_VectorOf<Int16>.DivisionOperatorBenchmark

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in SciMark2.kernel

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   benchFFT - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	543.92 ms	607.67 ms	1.12	0.02	True
   benchSparseMult - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	576.98 ms	723.01 ms	1.25	0.04	False

   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'

   Details

   Payloads

   Baseline
   Compare

   SciMark2.kernel.benchFFT

   ETL Files

   Histogram

   JIT Disasms

   SciMark2.kernel.benchSparseMult

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.IterateFor<Int32>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   ImmutableSortedSet - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	9.25 μs	10.78 μs	1.17	0.33	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor<Int32>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.IterateFor<Int32>.ImmutableSortedSet(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in Benchstone.BenchI.Array1

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   Test - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	4.57 ms	5.05 ms	1.10	0.01	True

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.Array1*'

   Details

   Payloads

   Baseline
   Compare

   Benchstone.BenchI.Array1.Test

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.IndexerSet<Int32>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   SortedList - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	17.50 μs	19.34 μs	1.10	0.00	True

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet<Int32>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.IndexerSet<Int32>.SortedList(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Memory.ReadOnlySpan

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   IndexOfString - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	15.78 ns	19.83 ns	1.26	0.16	True
   IndexOfString - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	15.51 ns	19.79 ns	1.28	0.12	True

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

   Details

   Payloads

   Baseline
   Compare

   System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

   ETL Files

   Histogram

   JIT Disasms

   System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Text.Perf_Utf8Encoding

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   GetByteCount - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	22.91 μs	27.29 μs	1.19	0.08	False
   GetByteCount - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	11.33 μs	13.50 μs	1.19	0.06	False
   GetByteCount - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	10.28 μs	12.34 μs	1.20	0.06	False
   GetByteCount - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	8.94 μs	10.71 μs	1.20	0.07	False

   
   
   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

   Details

   Payloads

   Baseline
   Compare

   System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)

   ETL Files

   Histogram

   JIT Disasms

   System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)

   ETL Files

   Histogram

   JIT Disasms

   System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)

   ETL Files

   Histogram

   JIT Disasms

   System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.MathBenchmarks.Single

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   ScaleB - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	7.64 μs	8.61 μs	1.13	0.01	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'

   Details

   Payloads

   Baseline
   Compare

   System.MathBenchmarks.Single.ScaleB

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in Burgers

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   Test2 - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	133.69 ms	175.75 ms	1.31	0.00	True

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Burgers*'

   Details

   Payloads

   Baseline
   Compare

   Burgers.Test2

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	2158031420214184d22a29029ff24420472fcaa0
   Compare	e07f80ac344b291b54a3fd5a333906782ab4809e
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   EncodeUtf8 - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	35.59 ns	37.65 ns	1.06	0.09	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

   Details

   Payloads

   Baseline
   Compare

   System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,�2020,16)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository
3. EgorBo

   changed the title [-][Perf] Linux/x64: 43 Regressions on 11/28/2023 1:58:26 PM[/-] [+][Perf] Regressions from "Remove BBJ_NONE"[/+] on Dec 5, 2023
4. EgorBo

   transferred this issue fromdotnet/perf-autofiling-issueson Dec 5, 2023
5. ghost added

   needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area ownersAn area label is needed to ensure this gets routed to the appropriate area owners

   on Dec 5, 2023
6. EgorBo

   added

   tenet-performancePerformance related issuePerformance related issue

   tenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark

   area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

   and removed

   needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area ownersAn area label is needed to ensure this gets routed to the appropriate area owners

   on Dec 5, 2023
8. EgorBo commented on Dec 5, 2023

   

   EgorBo

   on Dec 5, 2023

   Member

   More actions

   We suspect #94239 cc @amanasifkhalid
9. EgorBo commented on Dec 5, 2023

   

   EgorBo

   on Dec 5, 2023

   Last edited by cincuranet

   Member

   More actions

   Also:

   x64:

   • [Perf] Windows/x64: 9 Regressions on 11/28/2023 7:36:40 PM perf-autofiling-issues#25432
   • [Perf] Linux/x64: 2 Regressions on 12/2/2023 7:39:13 PM perf-autofiling-issues#25401
   • [Perf] Windows/x64: 4 Regressions on 11/28/2023 7:36:40 PM perf-autofiling-issues#25792

   arm64:

   • [Perf] Linux/arm64: 9 Regressions on 11/28/2023 5:48:12 PM perf-autofiling-issues#25600
   • [Perf] Windows/arm64: 3 Regressions on 11/29/2023 6:46:09 PM perf-autofiling-issues#25612
10. amanasifkhalid commented on Dec 5, 2023

    

    amanasifkhalid

    on Dec 5, 2023

    Contributor

    More actions

    Looking at the asmdiffs from that PR, code size is a net improvement, but some collections were a net regression in FullOpts (in particular, benchmarks.run_pgo).

    This change is part of a much larger work item that will (hopefully) be a net perf improvement. Is there anything we can/should do to address these transient regressions in the meantime?
11. AndyAyersMS commented on Dec 5, 2023

    

    AndyAyersMS

    on Dec 5, 2023

    Member

    More actions

    I would recommend something like:

    • wait for this Thursday's perf triage results (arm64)
    • wait a bit longer (say a week)
    • build a table collating across all the regression reports: benchmark x arch/os x initial/current results. I have a tool I can share with you to help do this.
    • Investigate the top N entries in this table (note performance fluctuates over time and we want to be able to separate out noise from signal, so the waiting above ensures we have a good sample of data to look at)—especially the largest ones, the most stable ones, and the ones that appear in multiple arch/os results. Write up what you've learned.
    • see if there is any immediate mitigation available. If so work on that or open an issue to remember to work on that. If not, see if there is a reasonable expectation that the rest of the work we plan will eventually fix the problem, and if so, add notes to the overall work item saying we expect it to resolve these specific regressions.

    You can look at #84264 or #87194 as examples.

    Also look at improvements, since just looking at regressions is going to give you an unbalanced picture of the impact of the change. If the improvements largely outweigh the regressions and the regressions are not things we can easily fix we may just decide to live with things as is.

    👍React with 👍1Reacted by Aman Khalid
12. AndyAyersMS commented on Dec 5, 2023

    

    AndyAyersMS

    on Dec 5, 2023

    Member

    More actions

    Also I'll be going through the same process with #95645...
13. JulieLeeMSFT

    assigned

    amanasifkhalid

    on Dec 8, 2023
14. JulieLeeMSFT

    added this to the 9.0.0 milestone on Dec 8, 2023
15. ghost removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Dec 8, 2023
16. amanasifkhalid

    added

    Priority:2Work that is important, but not critical for the releaseWork that is important, but not critical for the release

    on May 3, 2024
17. amanasifkhalid commented on Jul 19, 2024

    

    amanasifkhalid

    on Jul 19, 2024

    Contributor

    More actions

    Updated chart:

    Notes	Recent Score	Orig Score	Linux arm64	Windows arm64	Linux x64	Windows x64	Windows amd64	Benchmark
    	1.51	1.55				1.51 1.55		System.Collections.Sort(BigStruct).List(Size: 512)
    	1.50	1.31			1.50 1.31			Burgers.Test2
    	1.31	1.10			1.31 1.10			Benchstone.MDBenchI.MDAddArray2.Test
    	1.27	1.13			1.27 1.13			System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
    	1.27	1.16			1.27 1.16			System.Numerics.Tests.Perf_VectorOf(Int16).DivisionOperatorBenchmark
    	1.20	1.10			1.20 1.10			System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")
    	1.19	1.11			1.19 1.11			System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
    	1.18	1.12			1.18 1.12			SciMark2.kernel.benchFFT
    	1.18	1.13			1.18 1.13			System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)
    	1.17	1.13			1.17 1.13			System.Memory.Span(Char).Reverse(Size: 512)
    	1.16	1.09			1.16 1.09			System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")
    	1.16	1.08			1.16 1.08			System.Collections.IterateFor(String).ImmutableSortedSet(Size: 512)
    	1.15	1.32			1.15 1.32			System.Tests.Perf_String.Concat_CharEnumerable
    	1.12	1.14		1.12 1.14				System.Collections.IndexerSetReverse(Int32).Span(Size: 512)
    	1.12	1.10			1.12 1.10			Benchstone.BenchI.Array1.Test
    	1.10	1.06			1.10 1.06			System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,�2020,16)
    	1.10	1.10					1.10 1.10	System.Collections.TryGetValueTrue(String, String).ImmutableDictionary(Size: 512)
    	1.10	1.17			1.10 1.17			System.Collections.IterateFor(Int32).ImmutableSortedSet(Size: 512)
    	1.09	1.07			1.09 1.07			System.Collections.ContainsFalse(String).FrozenSet(Size: 512)
    	1.09	1.09	1.09 1.09					System.Tests.Perf_Random.ctor_seeded
    	1.08	1.07	1.08 1.07					Benchstone.BenchI.BubbleSort2.Test
    	1.07	1.06			1.07 1.06			PerfLabTests.CastingPerf.ObjFooIsObj2
    	1.07	1.07					1.07 1.07	System.Collections.ContainsKeyFalse(String, String).ImmutableSortedDictionary(Size: 512)
    	1.07	1.07	1.07 1.07					System.Tests.Perf_Double.IsNaN(value: NaN)
    	1.07	1.07	1.07 1.07					System.Tests.Perf_Single.IsNaN(value: NaN)
    	1.07	1.07	1.07 1.07					System.Tests.Perf_Double.IsNaN(value: 0)
    	1.07	1.07	1.07 1.07					System.Tests.Perf_Single.IsNaN(value: 0)
    	1.07	1.12			1.07 1.12			System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)
    	1.06	1.25			1.06 1.25			SciMark2.kernel.benchSparseMult
    	1.06	1.06	1.06 1.06					System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishAllAscii)
    	1.06	1.06					1.06 1.06	Benchstone.BenchI.AddArray.Test
    	1.06	1.06			1.06 1.06			PerfLabTests.CastingPerf.IFooFooIsIFoo
    	1.04	1.41			1.04 1.41			System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    	1.02	1.23			1.02 1.23			System.Tests.Perf_Int128.TryParse(value: "170141183460469231731687303715884105727")
    	1.02	1.07			1.01 1.07	1.03 1.08		System.Collections.ContainsFalse(Int32).ImmutableHashSet(Size: 512)
    	1.02	1.10			1.02 1.10			System.Collections.IndexerSet(Int32).SortedList(Size: 512)
    	1.02	1.26			0.78 1.19		1.33 1.33	System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)
    	1.01	1.14				1.01 1.14		System.MathBenchmarks.Double.ScaleB
    	1.01	1.13			1.01 1.13			System.MathBenchmarks.Single.ScaleB
    	1.01	1.11	1.09 1.09	0.93 1.13				System.Collections.IndexerSet(Int32).Array(Size: 512)
    	1.01	1.37			1.01 1.37			System.Tests.Perf_String.IndexerCheckPathLength
    	1.00	1.11		1.00 1.11				System.Collections.IndexerSetReverse(Int32).Array(Size: 512)
    	1.00	1.13	1.06 1.06		0.94 1.21			Benchstone.BenchI.CSieve.Test
    	1.00	1.13			1.00 1.13			System.Collections.ContainsFalse(Int32).ImmutableSortedSet(Size: 512)
    	0.99	1.08			0.99 1.08			System.Collections.ContainsKeyTrue(Int32, Int32).SortedList(Size: 512)
    	0.99	1.25			0.99 1.25			System.Tests.Perf_Int128.ParseSpan(value: "170141183460469231731687303715884105727")
    	0.98	1.40				0.98 1.40		System.Collections.Sort(BigStruct).Array(Size: 512)
    	0.98	1.05			0.98 1.05			System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)
    	0.97	1.08			0.97 1.08			System.Collections.ContainsFalse(Int32).SortedSet(Size: 512)
    	0.96	1.22			0.96 1.22			System.Tests.Perf_Int128.TryParseSpan(value: "170141183460469231731687303715884105727")
    	0.96	1.25			0.96 1.25			System.Tests.Perf_Int128.Parse(value: "170141183460469231731687303715884105727")
    	0.95	1.28			0.95 1.28			System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    	0.94	1.10				0.94 1.10		System.Collections.ContainsKeyTrue(String, String).ImmutableDictionary(Size: 512)
    	0.92	1.19			0.75 1.19	1.13 1.19		System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)
    	0.88	1.49			0.88 1.49			System.Memory.Span(Char).IndexOfAnyThreeValues(Size: 512)
    	0.88	1.26			0.88 1.26			System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ
    	0.88	1.20			0.73 1.20	1.05 1.21		System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)
    	0.86	1.33				0.86 1.33		System.Collections.Sort(BigStruct).LinqQuery(Size: 512)
    	0.85	1.29				0.85 1.29		System.Collections.Sort(BigStruct).LinqOrderByExtension(Size: 512)
    	0.84	1.29			0.84 1.29			System.Collections.ContainsFalse(Int32).Queue(Size: 512)
    	0.79	1.20			0.79 1.20			System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
    	0.78	1.38			0.78 1.38			System.Memory.Span(Char).LastIndexOfAnyValues(Size: 512)
    	0.76	1.27			0.76 1.27			System.Collections.ContainsKeyTrue(Int32, Int32).IDictionary(Size: 512)
    	0.70	1.10			0.70 1.10			System.Memory.Span(Char).Clear(Size: 512)
18. amanasifkhalid commented on Jul 26, 2024

    

    amanasifkhalid

    on Jul 26, 2024

    Contributor

    More actions

    I've looked at all of the benchmarks still regressed by >=10% in the above chart, and it's clear that these regressions are from recent churn in block layout (#102763) and/or block compaction (#103972). For example, take a look at the graphs of the top regressions:

    
    
    
    
    
    (This one regressed before we turned on the new block layout. I dug into this and found something actionable for .NET 10 here.)

    Windows data is split between Windows 10 (purple) and 11 (blue) configs:
    System.Collections.Sort<BigStruct>.List(Size: 512)
    

    System.Collections.IndexerSetReverse<Int32>.Span(Size: 512) (turquoise is 10, blue is 11)
    

    System.Collections.TryGetValueTrue<String, String>.ImmutableDictionary(Size: 512)
    

    BBJ_NONE removal doesn't seem relevant to these remaining regressions, so I'm going to retire this issue in favor of #102763 and #103972.
19. amanasifkhalid

    closed this as completedon Jul 26, 2024
20. github-actions

    locked and limited conversation to collaborators on Aug 27, 2024