[Perf] Linux/x64: 41 Regressions on 10/18/2023 3:03:02 PM #93926

Activity

1. performanceautofiler

   added

   arch-x64

   os-linuxLinux OS (any supported distro)Linux OS (any supported distro)

   PGO

   runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Oct 24, 2023
2. performanceautofiler commented on Oct 24, 2023

   

   performanceautofilerBot

   on Oct 24, 2023

   Last edited by performanceautofiler

   Author

   More actions

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.Tests.Perf_BitArray

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   BitArraySetLengthGrow - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	192.19 ns	242.05 ns	1.26	0.00	True
   BitArrayCopyToBoolArray - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	122.76 ns	156.11 ns	1.27	0.07	False
   BitArraySetLengthShrink - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	132.70 ns	178.55 ns	1.35	0.01	True
   BitArrayByteArrayCtor - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	129.55 ns	180.65 ns	1.39	0.01	True
   BitArrayBoolArrayCtor - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	28.97 ns	36.10 ns	1.25	0.12	True

   
   
   
   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   System.Collections.Tests.Perf_BitArray.BitArrayCopyToBoolArray(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   System.Collections.Tests.Perf_BitArray.BitArrayByteArrayCtor(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Tests.Perf_String

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   Concat_CharEnumerable - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	2.71 μs	3.57 μs	1.32	0.29	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

   Details

   Payloads

   Baseline
   Compare

   System.Tests.Perf_String.Concat_CharEnumerable

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.CtorFromCollection<String>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   HashSet - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	10.28 μs	11.20 μs	1.09	0.19	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection<String>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.CtorFromCollection<String>.HashSet(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.AddGivenSize<String>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   Dictionary - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	9.45 μs	10.52 μs	1.11	0.04	False
   IDictionary - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	10.39 μs	11.61 μs	1.12	0.07	False

   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize<String>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.AddGivenSize<String>.Dictionary(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   System.Collections.AddGivenSize<String>.IDictionary(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.IndexerSet<Int32>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   SortedDictionary - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	19.18 μs	21.91 μs	1.14	0.01	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet<Int32>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.IndexerSet<Int32>.SortedDictionary(Size: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Collections.TryAddGiventSize<String>

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   Dictionary - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	9.56 μs	10.79 μs	1.13	0.07	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddGiventSize<String>*'

   Details

   Payloads

   Baseline
   Compare

   System.Collections.TryAddGiventSize<String>.Dictionary(Count: 512)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   IsMatch - Duration of single invocation 📈 - ADX Test Multi Config Graph	168.10 ns	184.02 ns	1.09	0.16	False
   IsMatch - Duration of single invocation 📈 - ADX Test Multi Config Graph	80.22 ns	85.34 ns	1.06	0.10	False

   
   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

   Details

   Payloads

   Baseline
   Compare

   System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 3, Options: NonBacktracking)

   ETL Files

   Histogram

   JIT Disasms

   System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository

   ---

   Run Information

   Name	Value
   Architecture	x64
   OS	ubuntu 22.04
   Queue	TigerUbuntu
   Baseline	d18295e107a297dfe4e12ef5b34c28a63548ab31
   Compare	d3bf287cce90d30c634b57c4cf8b568812dfc91b
   Diff	Diff
   Configs	CompilationMode:tiered, RunKind:micro

   Regressions in System.IO.Tests.Perf_FileInfo

   Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
   ctor_str - Duration of single invocation 📝 - Benchmark Source 📈 - ADX Test Multi Config Graph	76.12 ns	94.84 ns	1.25	0.11	False

   
   Test Report

   Repro

   General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

   git clone https://github.com/dotnet/performance.git
   python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_FileInfo*'

   Details

   Payloads

   Baseline
   Compare

   System.IO.Tests.Perf_FileInfo.ctor_str

   ETL Files

   Histogram

   JIT Disasms

   Docs

   Profiling workflow for dotnet/runtime repository
   Benchmarking workflow for dotnet/runtime repository
3. cincuranet

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Oct 24, 2023
4. cincuranet

   transferred this issue fromdotnet/perf-autofiling-issueson Oct 24, 2023
5. ghost added

   area-System.Linq

   on Oct 24, 2023
6. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Oct 24, 2023
8. cincuranet commented on Oct 24, 2023

   

   cincuranet

   on Oct 24, 2023

   Contributor

   More actions

   The diff is always 2a8e2ff...a6e3732. Only suspect is #93371. cc @AndyAyersMS
9. cincuranet

   added

   tenet-performancePerformance related issuePerformance related issue

   tenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark

   and removed

   area-System.Linq

   on Oct 24, 2023
10. cincuranet commented on Oct 24, 2023

    

    cincuranet

    on Oct 24, 2023

    Contributor

    More actions

    More from same diff:

    • [Perf] Windows/x64: 2 Regressions on 10/18/2023 3:03:02 PM perf-autofiling-issues#23640
11. AndyAyersMS

    self-assigned this

    on Oct 24, 2023
12. AndyAyersMS

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Oct 24, 2023
13. AndyAyersMS commented on Oct 24, 2023

    

    AndyAyersMS

    on Oct 24, 2023

    Member

    More actions

    I mentioned this to @cincuranet privately, but there's a divergence betweeen the lab charted data and the azure reported data.

    For instance the lab sees a clear regression here

    

    But the azure data shows something quite different

    

    You can sort of see the lab data tracking the lower bound of the azure data here... not sure what all this means.
14. cincuranet commented on Oct 25, 2023

    

    cincuranet

    on Oct 25, 2023

    Contributor

    More actions

    The ADX query is wrong. I'm going to fix it. In the meantime, this is the correct one.

    On another note, looks like with latest runs, the regression is gone.

    
15. jeffschwMSFT

    added

    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

    on Oct 25, 2023
17. JulieLeeMSFT

    added this to the 9.0.0 milestone on Jan 13, 2024
18. AndyAyersMS

    added

    Priority:2Work that is important, but not critical for the releaseWork that is important, but not critical for the release

    on May 8, 2024
19. AndyAyersMS commented on Jul 9, 2024

    

    AndyAyersMS

    on Jul 9, 2024

    Last edited by AndyAyersMS

    Member

    More actions

    Most of these seem to be fixed now (or were fixed but regressed again), but the various BitArray methods are still regressed

    Notes	Recent Score	Orig Score	main_x64_ubuntu%2022.04/	Benchmark
    Fixed by #94387, but regressed again (more seriously) with #103143 (see dotnet/perf-autofiling-issues#36447)	1.47	1.27	1.47 1.27	System.Collections.Tests.Perf_BitArray.BitArrayCopyToBoolArray(Size: 512)
    ditto	1.38	1.39	1.38 1.39	System.Collections.Tests.Perf_BitArray.BitArrayByteArrayCtor(Size: 512)
    fixed but re-regressed	1.37	1.21	1.37 1.21	System.Linq.Tests.Perf_Enumerable.Aggregate_Seed(input: IEnumerable)
    ditto (1st issue)	1.36	1.35	1.36 1.35	System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 512)
    noisy	1.28	1.26	1.28 1.26	System.Collections.Tests.Perf_BitArray.BitArraySetLengthGrow(Size: 512)
    fixed but now noisy	1.26	1.26	1.26 1.26	System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: Array)
    	1.14	1.25	1.14 1.25	System.IO.Tests.Perf_FileInfo.ctor_str
    fixed but now noisy	1.14	1.32	1.14 1.32	System.Tests.Perf_String.Concat_CharEnumerable
    fixed but since regressed	1.13	1.27	1.13 1.27	System.Linq.Tests.Perf_Enumerable.CastToSameType(input: IEnumerable)
    	1.08	1.22	1.08 1.22	System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False))
    	1.05	1.11	1.05 1.11	System.Collections.Perf_DefaultFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 1000)
    	1.03	1.48	1.03 1.48	SeekUnroll.Test(boxedIndex: 27)
    	1.02	1.84	1.02 1.84	SeekUnroll.Test(boxedIndex: 3)
    	1.02	1.85	1.02 1.85	SeekUnroll.Test(boxedIndex: 1)
    	1.01	1.14	1.01 1.14	System.Linq.Tests.Perf_Enumerable.Where(input: IEnumerable)
    	1.01	1.25	1.01 1.25	System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)
    	1.00	1.32	1.00 1.32	System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: Array)
    	1.00	1.56	1.00 1.56	SeekUnroll.Test(boxedIndex: 11)
    	1.00	1.22	1.00 1.22	System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)
    	1.00	1.24	1.00 1.24	System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)
    	1.00	1.17	1.00 1.17	System.Linq.Tests.Perf_Enumerable.Intersect(input: IEnumerable)
    	0.99	1.06	0.99 1.06	System.Linq.Tests.Perf_Enumerable.Aggregate(input: IEnumerable)
    	0.99	1.10	0.99 1.10	System.Linq.Tests.Perf_Enumerable.WhereSelect(input: IEnumerable)
    	0.99	1.09	0.99 1.09	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 3, Options: NonBacktracking)
    	0.99	1.22	0.99 1.22	System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)
    	0.98	1.06	0.98 1.06	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)
    	0.97	1.25	0.97 1.25	System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)
    	0.95	1.52	0.95 1.52	SeekUnroll.Test(boxedIndex: 19)
    	0.95	1.11	0.95 1.11	System.Collections.AddGivenSize(String).Dictionary(Size: 512)
    	0.94	1.12	0.94 1.12	System.Collections.AddGivenSize(String).IDictionary(Size: 512)
    	0.94	1.13	0.94 1.13	System.Collections.TryAddGiventSize(String).Dictionary(Count: 512)
    	0.93	1.18	0.93 1.18	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: None)
    	0.92	1.14	0.92 1.14	System.Linq.Tests.Perf_Enumerable.Select(input: IEnumerable)
    	0.91	1.14	0.91 1.14	System.Linq.Tests.Perf_Enumerable.Select(input: IList)
    	0.91	1.48	0.91 1.48	System.Collections.Perf_Frozen(Int16).Contains_True(Count: 4)
    	0.88	1.09	0.88 1.09	System.Collections.CtorFromCollection(String).HashSet(Size: 512)
    	0.85	1.14	0.85 1.14	System.Collections.IndexerSet(Int32).SortedDictionary(Size: 512)
    	0.83	1.25	0.83 1.25	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: Compiled)
    	0.75	1.05	0.75 1.05	System.Linq.Tests.Perf_Enumerable.LastWithPredicate_FirstElementMatches(input: IOrderedEnumerable)
    	0.57	1.11	0.57 1.11	System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: Array)
    	0.10	1.40	0.10 1.40	System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: Array)
20. AndyAyersMS commented on Jul 15, 2024

    

    AndyAyersMS

    on Jul 15, 2024

    Member

    More actions

    Seems like all are fixed or were fixed and subsequently re-regressed.
21. AndyAyersMS

    closed this as completedon Jul 15, 2024
22. github-actions

    locked and limited conversation to collaborators on Aug 15, 2024