Skip to content

Dynamic PGO Microbenchmark Regressions #87194

Closed
Closed
Dynamic PGO Microbenchmark Regressions#87194
Assignees
AndyAyersMS
Labels
Priority:2Work that is important, but not critical for the releasearea-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
Milestone
8.0.0

Description

@AndyAyersMS
AndyAyersMS
opened on Jun 6, 2023

This issue tracks investigation into microbenchmarks that have reported regressions with Dynamic PGO enabled. It is a continuation of #84264 which tracked regressions from PGO before it was enabled.

The report below is collated from the following autofiling reports.

The table is auto generated by a tool written by @EgorBo but may be edited by hand as regression analysis produces results. The "Score" is the geomean regression across all architectures; benchmarks that did not regress (or get reported) on some architectures are assumed to have produced the same results with and without PGO. "Recent Score" is the current performance (as of 2023-0606) versus the non-PGO result; "Orig Score" is based on the results of auto filing. They will differ if benchmark performance has improved or regressed since the auto filing ran (see for example the results for System.Text.Json.Tests.Perf_Get.GetByte, which has improved already).

Only the 36 entries with recent scores >= 1.3 are included; this leaves off approximately 220 more rows with scores between 1.3 or lower. Our plan is to prioritize investigation of these benchmarks initially, as they have the largest aggregate regressions. If time permits, we will regenerate this chart to pick up the impact of any fixes and see how much of the remainder we can tackle.

Each arch/os result is a hyperlink to the performance data graph for that benchmark. ~Note we currently have no autofiling data for win-x64-intel. If/when that shows up we will regenerate the table.~~

[edit: had to regenerate the table once already, as the scoring logic was off]
[edit: have x64 win intel data now, new table. Not current results have shifted so table is somewhat different...]

cc @dotnet/jit-contrib

Notes Recent Score Orig Score arm64-lin-ampere arm64-win-surface arm64-win-ampere x64-lin-intel x64-win-intel x64-win-amd Benchmark
noise 3.38 1.37 3.37
1.36		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None)
noise 3.36 1.37 3.36
1.37		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: NonBacktracking)
notes 2.71 3.39 2.71
3.39		 System.Memory.Span(Int32).EndsWith(Size: 4)
likely same as above 2.62 3.03 2.55
2.27		 2.59
3.04		 System.Memory.Span(Int32).SequenceEqual(Size: 4)
likely same as above 1.87 1.76 1.87
1.76		 System.Memory.Span(Int32).SequenceCompareToDifferent(Size: 512)
(lack of) if conversion 1.82 1.80 1.67
1.63		 1.93
1.92		 1.86
1.85		 System.Tests.Perf_Random.NextSingle
budget 1.75 1.88 1.33
1.47		 1.35
1.49		 1.90
1.99		 2.29
2.43		 2.10
2.19		 System.Text.Json.Tests.Perf_Get.GetInt16
BDN 1.73 2.81 3.55
3.54		 1.89
2.00		 1.28
4.73		 1.32
2.01		 1.39
2.68		 System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace(NumberOfBytes: 200000000)
notes 1.64 1.63 1.84
1.82		 1.65
1.64		 System.Tests.Perf_UInt32.TryParseHex(value: "0")
budget 1.61 1.70 1.27
1.44		 1.28
1.46		 1.24
1.18		 2.09
2.17		 2.25
2.33		 1.86
1.94		 System.Text.Json.Tests.Perf_Get.GetSByte
bimodal 1.61 1.59 1.60
1.58		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: Compiled)
cast expansion 1.60 1.64 1.82
1.87		 1.41
1.43		 System.Buffers.Tests.ReadOnlySequenceTests(Char).FirstSingleSegment
cast expansion 1.58 1.62 1.58
1.62		 System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSpanTenSegments
cast expansion 1.52 1.65 1.48
1.81		 1.56
1.50		 System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSingleSegment
cast expansion 1.50 1.73 1.88
2.13		 1.20
1.41		 System.Buffers.Tests.ReadOnlySequenceTests(Char).FirstTenSegments
likely same as span cases above 1.48 1.28 1.48
1.28		 System.Memory.Span(Int32).Reverse(Size: 4)
cast expansion 1.47 1.44 1.47
1.44		 System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSpanSingleSegment
notes 1.47 1.42 1.46
1.42		 Benchstone.BenchF.InvMt.Test
unclear 1.46 1.15 1.46
1.15		 MicroBenchmarks.Serializers.Json_FromStream(MyEventsListerViewModel).DataContractJsonSerializer_
fixed itself 1.45 1.09 1.45
1.09		 System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{
unclear 1.44 1.44 1.44
1.44		 Burgers.Test1
unclear 1.43 1.27 1.43
1.27		 System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers)
unclear, linux arm64 only 1.41 1.58 1.41
1.58		 System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity(length: 100000)
unclear, linux arm64 only 1.39 1.62 1.39
1.62		 BenchmarksGame.RegexRedux_5.RunBench(options: Compiled)
bimodal 1.39 1.39 1.39
1.39		 System.MathBenchmarks.Single.Min
bimodal 1.39 1.39 1.39
1.39		 System.MathBenchmarks.Single.Max
unclear, linux arm64 only 1.39 1.32 1.39
1.32		 System.IO.Pipes.Tests.Perf_NamedPipeStream.ReadWriteAsync(size: 1000000, Options: Asynchronous)
noise 1.38 1.29 1.38
1.29		 System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 10000000)
bimodal 1.37 1.37 1.37
1.37		 System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: Compiled)
notes 1.37 1.36 1.26
1.29		 1.42
1.43		 1.24
1.28		 1.60
1.48		 System.Collections.Sort(IntStruct).Array(Size: 512)
budget 1.36 1.93 1.15
1.56		 1.15
1.58		 1.27
1.66		 1.42
2.14		 1.80
2.67		 1.49
2.24		 System.Text.Json.Tests.Perf_Get.GetByte
noise 1.35 1.31 1.36
1.33		 System.Memory.Span(Char).IndexOfAnyTwoValues(Size: 512)
arm64 only; ldar vs dmb 1.35 1.36 1.35
1.34		 1.38
1.40		 System.Collections.CtorFromCollection(Int32).ConcurrentBag(Size: 512)
fixed by physical promotion 1.35 1.36 1.35
1.36		 Devirtualization.EqualityComparer.ValueTupleCompareWrapped
budget 1.34 1.42 1.42
1.26		 1.28
1.38		 1.35
1.44		 1.35
1.42		 1.35
1.55		 1.31
1.46		 System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToStream(Mode: SourceGen)
notes 1.34 1.45 1.18
1.29		 1.40
1.44		 1.13
1.41		 1.71
1.71		 System.Collections.Sort(IntStruct).List(Size: 512)
notes 1.33 1.33 1.33
1.33		 System.Tests.Perf_HashCode.Combine_1
inlining different; exposed local 1.33 1.32 1.34
1.33		 1.32
1.32		 System.Memory.ReadOnlySequence.Slice_Repeat(Segment: Multiple)
notes 1.33 1.18 1.33
1.18		 System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfStrings)
budget 1.32 1.37 1.24
1.39		 1.20
1.28		 1.37
1.15		 1.39
1.46		 1.45
1.57		 1.27
1.39		 System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToWriter(Mode: SourceGen)
budget 1.32 1.39 1.37
1.28		 1.22
1.42		 1.34
1.31		 1.32
1.38		 1.30
1.50		 1.34
1.38		 System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToUtf8Bytes(Mode: SourceGen)
budget 1.31 1.88 1.15
1.59		 1.18
1.62		 1.03
1.37		 1.49
2.22		 1.66
2.49		 1.49
2.24		 System.Text.Json.Tests.Perf_Get.GetUInt16
budget 1.31 1.33 1.38
1.25		 1.20
1.23		 1.23
1.26		 1.35
1.46		 1.40
1.40		 1.41
1.43		 System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToString(Mode: SourceGen)
jcc errata 1.31 1.39 1.31
1.39		 Span.Sorting.QuickSortSpan(Size: 512)
lack of cold inline exposes local 1.31 1.29 1.31
1.31		 1.31
1.27		 System.Memory.ReadOnlySequence.Slice_Start_And_Length(Segment: Multiple)
budget 1.31 1.39 1.32
1.19		 1.20
1.50		 1.31
1.37		 1.40
1.50		 1.31
1.34		 System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeObjectProperty(Mode: SourceGen)
lack of ldapr 1.30 1.30 1.29
1.30		 1.30
1.30		 System.Collections.CtorFromCollection(String).ConcurrentBag(Size: 512)

Activity

  1. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Jun 6, 2023
  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Jun 6, 2023
  3. AndyAyersMS
    self-assigned this
    on Jun 6, 2023
  4. AndyAyersMS
    added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    and removed
    untriagedNew issue has not been triaged by the area owner
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Jun 6, 2023
  5. AndyAyersMS
    added this to the 8.0.0 milestone on Jun 6, 2023

  7. AndyAyersMS commented on Jun 6, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jun 6, 2023
    MemberAuthor

    System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace(NumberOfBytes: 200000000)

    image

    System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace(NumberOfBytes: 200000000)

    image ----

    These benchmarks were analyzed before PGO was enabled: #84264 (comment)

    BDN's strategy doesn't run the benchmark enough, because each iteration is long running, and so (since the key benchmark methods are R2R'd) the test ends up measuring tier1-instr code.

  8. AndyAyersMS commented on Jun 6, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jun 6, 2023
    MemberAuthor

    System.Text.Json.Tests.Perf_Get.GetInt16

    image

    System.Text.Json.Tests.Perf_Get.GetSByte

    image

    System.Text.Json.Tests.Perf_Get.GetByte

    image

    Likely the analysis from #84264 (comment) is still relevant and explains the related tests regressions as well: we run out of inlining budget, in part because the benchmark method is small and there are quite a few large aggressive inline methods, and so we're unable to do some key inlines.

    Tracking issue for this is #85531.

    Some of these improved with #86551.

  9. AndyAyersMS commented on Jun 6, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jun 6, 2023
    MemberAuthor

    System.Tests.Perf_UInt32.TryParseHex(value: "0")

    image

    This regresses across the board, so not sure why we don't have more autofiling for it.

    I can't repro on win-x64

    Method Job Toolchain value Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
    TryParseHex Job-EEXSZV \base\corerun.exe 0 7.860 ns 0.1366 ns 0.1573 ns 7.814 ns 7.640 ns 8.281 ns 1.00 0.00 - NA
    TryParseHex Job-EZIDSX \diff\corerun.exe 0 7.834 ns 0.1461 ns 0.1683 ns 7.783 ns 7.634 ns 8.344 ns 1.00 0.02 - NA

    But I can (perhaps) on win-arm64 (volterra)

    Method Job Toolchain value Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
    TryParseHex Job-DFURZJ \base\corerun.exe 0 3.265 ns 0.0267 ns 0.0237 ns 3.257 ns 3.235 ns 3.316 ns 1.00 0.00 - NA
    TryParseHex Job-URZCKJ \diff\corerun.exe 0 3.876 ns 0.0899 ns 0.0751 ns 3.849 ns 3.794 ns 4.034 ns 1.19 0.02 - NA 
    base

00.81%   4.1E+05     ?        Unknown
48.82%   2.458E+07   Tier-1   [491fed18-8a14-46f6-b30c-6320dc770919]Runnable_0.WorkloadActionUnroll(int64)
23.52%   1.184E+07   Tier-1   [System.Private.CoreLib]Number.TryParseBinaryIntegerHexOrBinaryNumberStyle(value class System.ReadOnlySpan`1<!!0>,value class System.Globalization.NumberStyles,!!1&)
12.69%   6.39E+06    Tier-1   [MicroBenchmarks]Perf_UInt32.TryParseHex(class System.String)
08.92%   4.49E+06    Tier-1   [System.Private.CoreLib]NumberFormatInfo.<GetInstance>g__GetProviderNonNull|58_0(class System.IFormatProvider)
04.37%   2.2E+06     Tier-1   [System.Private.CoreLib]CastHelpers.IsInstanceOfClass(void*,class System.Object)

diff

50.62%   2.666E+07   Tier-1   [d735f3a6-719a-4885-9631-16a7aeff132c]Runnable_0.WorkloadActionUnroll(int64)
23.94%   1.261E+07   Tier-1   [System.Private.CoreLib]Number.TryParseBinaryIntegerHexOrBinaryNumberStyle(value class System.ReadOnlySpan`1<!!0>,value class System.Globalization.NumberStyles,!!1&)
10.73%   5.65E+06    Tier-1   [MicroBenchmarks]Perf_UInt32.TryParseHex(class System.String)
08.62%   4.54E+06    Tier-1   [System.Private.CoreLib]NumberFormatInfo.<GetInstance>g__GetProviderNonNull|58_0(class System.IFormatProvider)
04.08%   2.15E+06    Tier-1   [System.Private.CoreLib]CastHelpers.IsInstanceOfClass(void*,class System.Object)

    Note the very high overhead. WorkoadActionUnroll should have the AggressiveOptimization attribute so its codegen does not vary.

    PGO codegen for TryParseBinaryIntegerHexOrBinaryNumberStyle looks good, all the hot code is adjacent and the method is straight line code. So not clear why it is ~7% or so slower.

    Path lengths are similar but PGO does one extra STP/LDP and a few more register arg moves. So perhaps that's the explanation?

  10. AndyAyersMS commented on Jun 6, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jun 6, 2023
    MemberAuthor

    System.Memory.Span<Int32>.EndsWith(Size: 4)

    image

    System.Memory.Span.SequenceEqual(Size: 4)

    image

    These two appear to be arm64 specific.

    Method Job Toolchain Size Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
    EndsWith Job-INARIQ \base-rel\corerun.exe 4 2.712 ns 0.0209 ns 0.0163 ns 2.712 ns 2.680 ns 2.734 ns 1.00 0.00 - NA
    EndsWith Job-YXKXJS \diff-rel\corerun.exe 4 3.357 ns 0.2569 ns 0.2959 ns 3.175 ns 3.079 ns 3.928 ns 1.23 0.10 - NA 
    base
01.52%   5.7E+05     ?        Unknown
35.26%   1.319E+07   Tier-1   [18850c97-dca6-4aad-8393-0715e0b95a4a]Runnable_0.WorkloadActionUnroll(int64)
34.51%   1.291E+07   Tier-1   [MicroBenchmarks]System.Memory.Span`1[System.Int32].EndsWith()
27.72%   1.037E+07   Tier-1   [System.Private.CoreLib]SpanHelpers.SequenceEqual(unsigned int8&,unsigned int8&,unsigned int)

diff
01.29%   5.1E+05     ?        Unknown
35.20%   1.392E+07   Tier-1   [MicroBenchmarks]System.Memory.Span`1[System.Int32].EndsWith()
31.40%   1.242E+07   Tier-1   [72707307-0596-4e84-8515-1a5c07b80d3a]Runnable_0.WorkloadActionUnroll(int64)
31.10%   1.23E+07    Tier-1   [System.Private.CoreLib]SpanHelpers.SequenceEqual(unsigned int8&,unsigned int8&,unsigned int)

    So issue appears to be in SequenceEqual ?

    Seems like PGO and non-PGO codegen is the same. Method is R2R'd and with PGO we do a tier1 instr, but do not add any probes. Explanation: this method is an intrinsic and not on the whitelist.

    Fixing that (hack) gives:

    Method Job Toolchain Size Mean Error StdDev Median Min Max Ratio Allocated Alloc Ratio
    EndsWith Job-FBJVVQ \base-rel\corerun.exe 4 2.732 ns 0.0330 ns 0.0275 ns 2.729 ns 2.692 ns 2.793 ns 1.00 - NA
    EndsWith Job-UKVWAD \diff-rel\corerun.exe 4 3.146 ns 0.0472 ns 0.0419 ns 3.153 ns 3.084 ns 3.208 ns 1.15 - NA
    EndsWith Job-BJUBBV \hack-rel\corerun.exe 4 2.660 ns 0.0364 ns 0.0322 ns 2.659 ns 2.610 ns 2.731 ns 0.97 - NA

    and more broady

    Method Job Toolchain Size Mean Error StdDev Median Min Max Ratio RatioSD Allocated Alloc Ratio
    EndsWith Job-SDVJLL \base-rel\corerun.exe 4 2.685 ns 0.0479 ns 0.0448 ns 2.679 ns 2.627 ns 2.766 ns 1.00 0.00 - NA
    EndsWith Job-FTSDAY \diff-rel\corerun.exe 4 3.151 ns 0.0178 ns 0.0158 ns 3.151 ns 3.126 ns 3.181 ns 1.17 0.02 - NA
    EndsWith Job-EGIDWC \hack-rel\corerun.exe 4 3.125 ns 0.0394 ns 0.0368 ns 3.125 ns 3.080 ns 3.180 ns 1.16 0.02 - NA
    EndsWith Job-SDVJLL \base-rel\corerun.exe 33 4.739 ns 0.0558 ns 0.0495 ns 4.728 ns 4.686 ns 4.840 ns 1.00 0.00 - NA
    EndsWith Job-FTSDAY \diff-rel\corerun.exe 33 5.322 ns 0.0576 ns 0.0539 ns 5.299 ns 5.257 ns 5.446 ns 1.12 0.01 - NA
    EndsWith Job-EGIDWC \hack-rel\corerun.exe 33 5.369 ns 0.0265 ns 0.0235 ns 5.369 ns 5.333 ns 5.419 ns 1.13 0.01 - NA
    EndsWith Job-SDVJLL \base-rel\corerun.exe 512 32.020 ns 0.0209 ns 0.0175 ns 32.023 ns 31.999 ns 32.059 ns 1.00 0.00 - NA
    EndsWith Job-FTSDAY \diff-rel\corerun.exe 512 32.422 ns 0.0351 ns 0.0328 ns 32.417 ns 32.383 ns 32.495 ns 1.01 0.00 - NA
    EndsWith Job-EGIDWC \hack-rel\corerun.exe 512 31.819 ns 0.0370 ns 0.0346 ns 31.815 ns 31.776 ns 31.893 ns 0.99 0.00 - NA

    But does not explain why there is a PGO regression. And as you can see the Size=4 results are not very stable.

    Similar diffs for EndsWith (better layout, slightly higher prolog/epilog costs).

  11. AndyAyersMS commented on Jun 6, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jun 6, 2023
    MemberAuthor

    System.Memory.Span<Int32>.SequenceCompareToDifferent(Size: 512)

    image

    Also seems to be arm64 specific.

  12. AndyAyersMS
    added
    Priority:2Work that is important, but not critical for the release
    on Jun 9, 2023

  13. AndyAyersMS commented on Jul 5, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jul 5, 2023
    MemberAuthor

    System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None)

    These spiked up but then recovered and match their longer-term behavior

    newplot - 2023-07-05T110411 317
    newplot - 2023-07-05T110407 909

  14. 25 remaining items

  15. AndyAyersMS commented on Jul 28, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jul 28, 2023
    MemberAuthor

    System.Collections.CtorFromCollection<String>.ConcurrentBag(Size: 512)

    arm64 only

    image

    Repros on my volterra:

    BenchmarkDotNet v0.13.7-nightly.20230724.45, Windows 11 (10.0.22621.2070/22H2/2022Update/SunValley2)
    Snapdragon 8cx Gen 3 3.0 GHz, 1 CPU, 8 logical and 8 physical cores
    .NET SDK 8.0.100-preview.6.23330.14
    [Host] : .NET 8.0.0 (8.0.23.32907), Arm64 RyuJIT AdvSIMD
    Job-EGKPHU : .NET 8.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
    Job-FFUPKC : .NET 8.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD

    PowerPlanMode=00000000-0000-0000-0000-000000000000 Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true IterationTime=250.0000 ms
    MaxIterationCount=20 MinIterationCount=15 WarmupCount=1

    Method Job Toolchain Size Mean Error StdDev Median Min Max Ratio Gen0 Gen1 Gen2 Allocated Alloc Ratio
    ConcurrentBag Job-EGKPHU \base-rel\corerun.exe 512 15.78 us 0.107 us 0.100 us 15.73 us 15.68 us 16.00 us 1.00 2.6309 2.5667 0.0642 16.16 KB 1.00
    ConcurrentBag Job-FFUPKC \diff-rel\corerun.exe 512 22.78 us 0.119 us 0.112 us 22.76 us 22.63 us 22.99 us 1.44 2.5491 2.4547 - 16.16 KB 1.00 
    base
12.70%   4.89E+06    ?        Unknown
43.43%   1.672E+07   Tier-1   [System.Collections.Concurrent]System.Collections.Concurrent.ConcurrentBag`1+WorkStealingQueue[System.__Canon].LocalPush(!0,int64&)
32.36%   1.246E+07   native   coreclr.dll
03.12%   1.2E+06     native   ntoskrnl.exe
02.68%   1.03E+06    Tier-1   [System.Private.CoreLib]System.SZGenericArrayEnumerator`1[System.__Canon].get_Current()
02.05%   7.9E+05     Tier-1   [System.Collections.Concurrent]System.Collections.Concurrent.ConcurrentBag`1[System.__Canon]..ctor(class System.Collections.Generic.IEnumerable`1<!0>)
01.51%   5.8E+05     Tier-1   [System.Private.CoreLib]SZGenericArrayEnumeratorBase.MoveNext()

diff

06.89%   2.71E+06    ?        Unknown
60.37%   2.376E+07   Tier-1   [System.Collections.Concurrent]System.Collections.Concurrent.ConcurrentBag`1+WorkStealingQueue[System.__Canon].LocalPush(!0,int64&)
23.04%   9.07E+06    native   coreclr.dll
02.69%   1.06E+06    Tier-1   [System.Collections.Concurrent]System.Collections.Concurrent.ConcurrentBag`1[System.__Canon]..ctor(class System.Collections.Generic.IEnumerable`1<!0>)
02.59%   1.02E+06    Tier-1   [System.Private.CoreLib]System.SZGenericArrayEnumerator`1[System.__Canon].get_Current()
02.57%   1.01E+06    native   ntoskrnl.exe

    So issue is evidently in LocalPush.

    In the base (no-pgo) jit we form a CSE and this lets us use ldar, in the diff (pgo) jit we don't do the cse and end up emitting a separate barrier. That may be the culprit.

    BASE

Generating: N507 (  1,  1) [000436] -----------                  t436 =    LCL_VAR   byref  V16 cse0          x28 REG x28 $1c8
                                                                        /--*  t436   byref  
Generating: N509 (  3,  2) [000174] V---GO-----                  t174 = *  IND       ref    REG x0 $156
IN0062:             ldapr   x0, [x28]

DIFF

Generating: N347 (  1,  1) [000172] -----------                  t172 =    LCL_VAR   ref    V00 this         u:1 x22 REG x22 $80
                                                                        /--*  t172   ref    
Generating: N349 (  3,  4) [000356] -c---------                  t356 = *  LEA(b+8)  byref  REG NA
                                                                        /--*  t356   byref  
Generating: N351 (  6,  6) [000174] V---GO-----                  t174 = *  IND       ref    REG x0 $156
IN00ac:             ldr     x0, [x22, #0x08]
IN00ad:             dmb     ishld

    In base there are just two sampling hot spots, both tied to ldapr:

    0x0044 : 1037
0x00DC : 162

IN0005: 000040      swpal   w1, w1, [x21]
IN0006: 000044      add     x22, x0, #28
IN0007: 000048      ldapr   w23, [x22]

IN002b: 0000D8      ldapr   w25, [x24]
IN002c: 0000DC      ldrb    w1, [x0, #0x34]

    In diff there are more hot spots, and all hottest samples are near the dmbs.

    0x003C : 655
0x0340 : 367
0x00C4 : 239
0x00AC : 201
0x0064 : 208

IN0006: 00003C      ldr     w1, [x0, #0x1C]
IN0007: 000040      dmb     ishld

IN00c6: 00033C      dmb     ish
IN00c7: 000340      str     wzr, [x0, #0x2C]

IN0027: 0000C0      dmb     ish
IN0028: 0000C4      str     w1, [x0, #0x1C]

IN0021: 0000A8      dmb     ishld
IN0022: 0000AC      ldr     x2, [fp, #0x28]	// [V01 arg1]

IN000f: 000060      dmb     ishld
IN0010: 000064      ldrb    w1, [x0, #0x34]

    @EgorBo example we were chatting about.

  16. adamsitnik commented on Jul 31, 2023

    @adamsitnik
    adamsitnik
    on Jul 31, 2023
    Member

    @adamsitnik seems like we should up the iterations per invocation for these tests to something like 10_000 (at 1000, each benchmark interval is only 10ms).

    I took a look at the benchmark source code and it's safe to do it (the benchmark ID won't change, as the InvocationsPerIteration const is not an argument or a parameter for this benchmark)

  17. AndyAyersMS commented on Jul 31, 2023

    @AndyAyersMS
    AndyAyersMS
    on Jul 31, 2023
    MemberAuthor

    While there are still a few benchmarks where the analysis is unclear, they are isolated to specific OS/ABI combinations. So I'm going to close this out.

  18. AndyAyersMS
    closed this as completedon Jul 31, 2023

  19. DrewScoggins commented on Aug 24, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 24, 2023
    Member

    Did BubbleSort2 ever get looked at?

    image

  20. DrewScoggins commented on Aug 24, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 24, 2023
    Member
    image

  21. adamsitnik commented on Aug 25, 2023

    @adamsitnik
    adamsitnik
    on Aug 25, 2023
    Member

    Please keep in mind that both bubble sort and IndexOf are heavily dependent on memory alignment, so it can be unrelated to PGO. The easiest way to verify is to allocate an aligned memory using NativeMemory.AlignedAlloc, create a span out of it and try to repro.

  22. DrewScoggins commented on Aug 25, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 25, 2023
    Member

    Please keep in mind that both bubble sort and IndexOf are heavily dependent on memory alignment, so it can be unrelated to PGO. The easiest way to verify is to allocate an aligned memory using NativeMemory.AlignedAlloc, create a span out of it and try to repro.

    Makes sense. I am just adding tests here that were not previously included, but regressed over the same commit range as this check-in.

  23. DrewScoggins commented on Aug 25, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 25, 2023
    Member
    image

  24. DrewScoggins commented on Aug 25, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 25, 2023
    Member

    Major instability starting with this commit.

    image

  25. AndyAyersMS commented on Aug 25, 2023

    @AndyAyersMS
    AndyAyersMS
    on Aug 25, 2023
    MemberAuthor

    Please keep in mind that both bubble sort and IndexOf are heavily dependent on memory alignment, so it can be unrelated to PGO. The easiest way to verify is to allocate an aligned memory using NativeMemory.AlignedAlloc, create a span out of it and try to repro.

    Makes sense. I am just adding tests here that were not previously included, but regressed over the same commit range as this check-in.

    Also note that bubble sort runs for a very long time, and so likely BDN + lab customization is not reliably measuring the tier1 codegen , but instead some mixture of Tier0, Tier0 + instrumentation, OSR, and or R2R code.

  26. AndyAyersMS commented on Aug 25, 2023

    @AndyAyersMS
    AndyAyersMS
    on Aug 25, 2023
    MemberAuthor

    Major instability starting with this commit.

    image

    Feel free to add this kind of thing to #87324

  27. DrewScoggins commented on Aug 25, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 25, 2023
    Member

    I will do that going forward, didn't know about that issue :)

  28. DrewScoggins commented on Aug 28, 2023

    @DrewScoggins
    DrewScoggins
    on Aug 28, 2023
    Member

    This is specifically on Windows x86.

    image
  29. ghost locked as resolved and limited conversation to collaborators on Sep 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Priority:2Work that is important, but not critical for the releasearea-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions