Investigate microbenchmarks that regress with PGO enabled

Look into cases where microbenchmarks are slower with PGO enabled.

#74873 (comment) has an initial list to check.

Issues uncovered

Fixes

PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

Default [link]

PGO [link]

Improvements in both on 12/4 were from #79060.
Regression in PGO on 1/13 was from #80481.

Looks like quite a few tests were impacted by #80481. So I'll just focus on that for now.

@AndyAyersMS

@AndyAyersMS I think that CastHelpers should not be instrumented since they were already carefully optimized for a general case by @VSadov e.g.

runtime/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/CastHelpers.cs

Lines 277 to 322 in e6226e6

    
           private static object? IsInstanceOfClass(void* toTypeHnd, object? obj) 
        
           { 
        
               if (obj == null || RuntimeHelpers.GetMethodTable(obj) == toTypeHnd) 
        
                   return obj; 
        
               MethodTable* mt = RuntimeHelpers.GetMethodTable(obj)->ParentMethodTable; 
        
               for (; ; ) 
        
               { 
        
                   if (mt == toTypeHnd) 
        
                       goto done; 
        
                   if (mt == null) 
        
                       break; 
        
                   mt = mt->ParentMethodTable; 
        
                   if (mt == toTypeHnd) 
        
                       goto done; 
        
                   if (mt == null) 
        
                       break; 
        
                   mt = mt->ParentMethodTable; 
        
                   if (mt == toTypeHnd) 
        
                       goto done; 
        
                   if (mt == null) 
        
                       break; 
        
                   mt = mt->ParentMethodTable; 
        
                   if (mt == toTypeHnd) 
        
                       goto done; 
        
                   if (mt == null) 
        
                       break; 
        
                   mt = mt->ParentMethodTable; 
        
               } 
        
               // this helper is not supposed to be used with type-equivalent "to" type. 
        
               Debug.Assert(!((MethodTable*)toTypeHnd)->HasTypeEquivalence); 
        
               obj = null; 
        
           done: 
        
               return obj; 
        
           }

PS: we can probably propose a new API [MethodImpl(MethodImplOptions.NoInstrumenting)]

Yes we are certain that these methods will be hot and want the best possible codegen right away.

The cast helpers are also initialized in a special way -

runtime/src/coreclr/vm/ecall.cpp

Line 99 in 4cbe6f9

void ECall::PopulateManagedCastHelpers()

I do not remember all the details, but there were reasons for that.
Maybe it is possible to disable instrumentation for things initialized that way.
(I do not have preferences, just want mention that there could be another way to detect and special case these)

Yes we are certain that these methods will be hot and want the best possible codegen right away.

The cast helpers are also initialized in a special way -

runtime/src/coreclr/vm/ecall.cpp

Line 99 in 4cbe6f9

void ECall::PopulateManagedCastHelpers()

I do not remember all the details, but there were reasons for that. Maybe it is possible to disable instrumentation for things initialized that way. (I do not have preferences, just want mention that there could be another way to detect and special case these)

The ones you listed have AggressiveOptimization attribute, am I right? In that case we won't instrument them, but there are other (less important) members in CastHelpers which don't have AggressiveOptimization so are eligble for instrumentation

All the cast helpers are initialized in that method.
If you have other uses and reasons to introduce MethodImplOptions.NoInstrumenting, then this part is unimportant.

It would only be interesting if you somehow do not want MethodImplOptions.NoInstrumenting just for a few special methods and need another way to identify them.

The issue caused by #80481 is with benchmark methods like PerfLabTests.CastingPerf:CheckArrayIsInterfaceNo that trigger OSR by looping a lot internally.

For such methods we never see any Tier0 + instr method return, so with sparse profiles enabled Tier1 profile reconstruction thinks the entry block is run rarely, and this messes up optimizations. eg in this case we fail to hoist/cse the cast out of the loop.

Trees after Profile incorporation

-----------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight       IBC  lp [IL range]     [jump]      [EH region]         [flags]
-----------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             0          0    [000..006)-> BB03 (always)                     rare IBC 
BB02 [0001]  1       BB03                 5630k 5630382    [006..018)                                     bwd bwd-target IBC 
BB03 [0002]  2       BB01,BB02            5630k 5630382    [018..020)-> BB02 ( cond )                     bwd bwd-src IBC 
BB04 [0003]  1       BB03                  0          0    [020..022)        (return)                     rare IBC 
-----------------------------------------------------------------------------------------------------------------------------------------

The fix I'm proposing is to run synthesis repair in such cases. It addresses the above case, looking now to see what else it does.

JIT: use synthesis to repair some reconstruction issues

JIT: use blend rather then repair for profile inconsistencies (#85171)

System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 4)

Default [link]

PGO [link]

Feb 2002 regression seems to be #82412

Method	Job	Toolchain	Size	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
SequenceCompareToDifferent	Job-HHXDGF	\base-rel\corerun.exe	4	4.590 ns	0.1019 ns	0.1001 ns	4.543 ns	4.478 ns	4.791 ns	1.00	0.00	-	NA
SequenceCompareToDifferent	Job-UIGKEB	\diff-rel\corerun.exe	4	4.817 ns	0.1166 ns	0.1033 ns	4.785 ns	4.700 ns	5.083 ns	1.05	0.03	-	NA

;; default

03.21%   1.2E+06     ?        Unknown
36.40%   1.36E+07    Tier-1   [MicroBenchmarks]System.Memory.Span`1[System.Byte].SequenceCompareToDifferent()
35.95%   1.343E+07   Tier-1   [System.Private.CoreLib]SpanHelpers.SequenceCompareTo(unsigned int8&,int32,unsigned int8&,int32)
17.91%   6.69E+06    Tier-1   [System.Private.CoreLib]MemoryExtensions.SequenceCompareTo(value class System.Span`1<!!0>,value class System.ReadOnlySpan`1<!!0>)

;; pgo

38.51%   1.483E+07   Tier-1   [MicroBenchmarks]System.Memory.Span`1[System.Byte].SequenceCompareToDifferent()
35.63%   1.372E+07   Tier-1   [System.Private.CoreLib]SpanHelpers.SequenceCompareTo(unsigned int8&,int32,unsigned int8&,int32)
16.26%   6.26E+06    Tier-1   [System.Private.CoreLib]MemoryExtensions.SequenceCompareTo(value class System.Span`1<!!0>,value class System.ReadOnlySpan`1<!!0>)

Looks like slightly different PGO data for an System.ReadOnlySpan1[ubyte]:.ctor(ubyte[]):this` (static vs dynamic) leads to an LSRA block order difference which leads LSRA to use two callee saves in the PGO case.

;; default (static pgo)

-----------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC  lp [IL range]     [jump]      [EH region]         [flags]
-----------------------------------------------------------------------------------------------------------------------------------------
BB01 [0008]  1                           100    100    [000..003)-> BB03 ( cond )                     IBC 
BB02 [0009]  1       BB01                  0.58   1    [003..00B)        (return)                     IBC 
BB03 [0010]  1       BB01                 99.42  99    [00B..021)        (return)                     IBC 
-----------------------------------------------------------------------------------------------------------------------------------------

Start LSRA Block Sequence: 
Current block: BB01
	Succ block: BB03, Criteria: weight, Worklist: [BB03 ]
	Succ block: BB02, Criteria: weight, Worklist: [BB03 BB02 ]
Current block: BB03
	Succ block: BB04, Criteria: weight, Worklist: [BB04 BB02 ]
Current block: BB04
	Succ block: BB06, Criteria: bbNum, Worklist: [BB06 BB02 ]
	Succ block: BB05, Criteria: bbNum, Worklist: [BB05 BB06 BB02 ]
Current block: BB05
	Succ block: BB07, Criteria: bbNum, Worklist: [BB06 BB07 BB02 ]
Current block: BB06
Current block: BB07
Current block: BB02
Final LSRA Block Sequence:
BB01 (  1   )
BB03 (  0.50)
BB04 (  1   )
BB05 (  0.00)
BB06 (  0.50)
BB07 (  1   )
BB02 (  0   )

;; pgo

-----------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight   IBC  lp [IL range]     [jump]      [EH region]         [flags]
-----------------------------------------------------------------------------------------------------------------------------------------
BB01 [0008]  1                           100    100    [000..003)-> BB03 ( cond )                     IBC 
BB02 [0009]  1       BB01                  0      0    [003..00B)        (return)                     rare IBC 
BB03 [0010]  1       BB01                100    100    [00B..021)        (return)                     IBC 
-----------------------------------------------------------------------------------------------------------------------------------------

Start LSRA Block Sequence: 
Current block: BB01
	Succ block: BB03, Criteria: weight, Worklist: [BB03 ]
	Succ block: BB02, Criteria: weight, Worklist: [BB03 BB02 ]
Current block: BB03
	Succ block: BB04, Criteria: weight, Worklist: [BB04 BB02 ]
Current block: BB04
	Succ block: BB06, Criteria: bbNum, Worklist: [BB06 BB02 ]
	Succ block: BB05, Criteria: weight, Worklist: [BB06 BB02 BB05 ]
Current block: BB06
	Succ block: BB07, Criteria: bbNum, Worklist: [BB07 BB02 BB05 ]
Current block: BB07
Current block: BB02
Current block: BB05
Final LSRA Block Sequence:
BB01 (  1   )
BB03 (  0.50)
BB04 (  1   )
BB06 (  0.50)
BB07 (  1   )
BB02 (  0   )
BB05 (  0   )

and the prologs

;; default
       push     rdi
       push     rsi
       sub      rsp, 72

;; pgo
       push     rdi
       push     rsi
       push     rbp
       push     rbx
       sub      rsp, 72

System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400KB)

Default [link]

PGO [link]

May 2022 (small) regression may have been #68804
February 2023 regression may be #77357 or #81936
April 2023 regression possibly #84427

Locally I can repro, but annoyingly when I profile PGO is faster.

Method	Job	Toolchain	IsDataIndented	TestRandomAccess	TestCase	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen0	Allocated	Alloc Ratio
Parse	Job-FHNYPZ	Default	False	True	Json400KB	4.592 ms	0.0531 ms	0.0444 ms	4.599 ms	4.515 ms	4.679 ms	1.00	0.00	92.5926	597.51 KB	1.00
Parse	Job-LHXDIX	PGO	False	True	Json400KB	5.507 ms	0.9467 ms	1.0522 ms	4.760 ms	4.490 ms	6.996 ms	1.29	0.22	-	597.59 KB	1.00

;; default

65.72%   2.552E+07   Tier-1   [System.Text.Json]JsonDocument.GetArrayIndexElement(int32,int32)
07.37%   2.86E+06    FullOpt  [System.Text.Json]JsonDocument.TryGetNamedPropertyValue(int32,int32,value class System.ReadOnlySpan`1<unsigned int8>,value class System.Text.Json.JsonElement&)
03.37%   1.31E+06    native   coreclr.dll
01.85%   7.2E+05     Tier-1   [System.Text.Json]JsonDocument.Parse(value class System.ReadOnlySpan`1<unsigned int8>,value class System.Text.Json.JsonReaderOptions,value class MetadataDb&,value class StackRowStack&)
01.75%   6.8E+05     Tier-1   [System.Text.Json]Utf8JsonReader.ReadSingleSegment()
01.75%   6.8E+05     Tier-1   [System.Private.CoreLib]IndexOfAnyAsciiSearcher.IndexOfAnyVectorized(unsigned int8&,int32,value class System.Runtime.Intrinsics.Vector128`1<unsigned int8>)
01.39%   5.4E+05     Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeString()

;; pgo

66.14%   2.531E+07   Tier-1   [System.Text.Json]JsonDocument.GetArrayIndexElement(int32,int32)
07.66%   2.93E+06    FullOpt  [System.Text.Json]JsonDocument.TryGetNamedPropertyValue(int32,int32,value class System.ReadOnlySpan`1<unsigned int8>,value class System.Text.Json.JsonElement&)
03.29%   1.26E+06    native   coreclr.dll
02.59%   9.9E+05     Tier-1   [System.Text.Json]JsonDocument.Parse(value class System.ReadOnlySpan`1<unsigned int8>,value class System.Text.Json.JsonReaderOptions,value class MetadataDb&,value class StackRowStack&)
01.86%   7.1E+05     Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeNextToken(unsigned int8)
01.44%   5.5E+05     Tier-1   [System.Private.CoreLib]Ascii.WidenAsciiToUtf16(unsigned int8*,wchar*,unsigned int)
01.31%   5E+05       Tier-1   [System.Text.Json]Utf8JsonReader.ReadSingleSegment()
01.23%   4.7E+05     Tier-1   [System.Private.CoreLib]IndexOfAnyAsciiSearcher.IndexOfAnyVectorized(unsigned int8&,int32,value class System.Runtime.Intrinsics.Vector128`1<unsigned int8>)
01.12%   4.3E+05     FullOpt  [System.Text.Json]JsonDocument.TryGetNamedPropertyValue(int32,value class System.ReadOnlySpan`1<wchar>,value class System.Text.Json.JsonElement&)
01.07%   4.1E+05     Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeValue(unsigned int8)
01.07%   4.1E+05     native   clrjit.dll

Note the TryGetNamedPropertyValue methods bypass tiering. Looks like they have both stackalloc and a loop, so aren't eligible for OSR.

Codegen for GetArrayIndexElement is identical w/wo PGO, and next hottest method bypasses tiering, and I consistently get faster results with PGO running BDN -p ETW as admin, so am going to move on.

My only guess is that perhaps GetArrayIndexElement is running into JCC errata. I don't have PGO perf history for other machines so can't easily cross-check (say against arm64).

we now have some PGO data both on linux x64 and ampere arm64, for both of those PGO is comparable or a bit faster.

System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch(total: 400000, unique: 1, cacheSize: 15)

Default [link]

PGO [link]

PGO seems consistently faster, so likely this was just a noisy comparison.

System.Collections.IndexerSet<Int32>.Dictionary(Size: 512)

Default [link]

PGO [link]

Jan 2023 regression looks like #80481

Locally I have PGO consistently faster:

Method	Job	Toolchain	Size	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
Dictionary	Job-OHYCZK	Default	512	3.506 us	0.1136 us	0.1216 us	3.491 us	3.328 us	3.792 us	1.00	0.00	-	NA
Dictionary	Job-NZIXNJ	PGO	512	3.219 us	0.0467 us	0.0414 us	3.214 us	3.122 us	3.287 us	0.92	0.03	-	NA

Both default and PGO have since improved and are showing comparable results in the lab:

Default

Improvement was one or more of:

PGO

Improvement was likely from #85805

System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!")

Note in a windows batch script you may need to use

--filter "System.Tests.Perf_Char.Char_ToLowerInvariant(input: ""Hello World!"")"

to get this properly passed to BenchmarkDotNet.

Default [link]

November 2022 improvement was #78593
April 2023 regression is possibly #83624

PGO [link]

November 2022 regression was #78593
April 2023 regression is possibly #83624

So oddly enough these are reacting oppositely to some changes.

I see PGO as faster or at least comparable locally.

Method	Job	Toolchain	input	Mean	Error	StdDev	Median	Min	Max	Ratio	Allocated	Alloc Ratio
Char_ToLowerInvariant	Job-ZSHNYR	Default	Hello World!	13.34 ns	0.143 ns	0.127 ns	13.36 ns	13.15 ns	13.52 ns	1.00	-	NA
Char_ToLowerInvariant	Job-QOYHZR	PGO	Hello World!	11.80 ns	0.137 ns	0.128 ns	11.76 ns	11.61 ns	11.99 ns	0.88	-	NA

The above covers the initial screen of the 50 worst ratios (as of 3 weeks ago). I am going to add in a few other cases based on a refreshed and slightly broadened query.

Broadened: the results above excluded any test that had less than a 2ns reported time. A lot of HW intrinsic/vector tests come in under this and report slower with PGO. So let's look at a few.

System.Numerics.Tests.Perf_Vector3.DotBenchmark

Default [link]

PGO [link]

Seems comparable, PGO result is just a hair greater than zero, but default is zero so ratio is infinite.

System.Numerics.Tests.Perf_Vector3.ReflectBenchmark

Default [link]

PGO [link]

Regression appears to be from #83036 which seems odd.

This one is still a mystery. Default and PGO seem to have identical code for the benchmark methods, but with release bits PGO reports consistently worse performance. If I drop in a checked jit into the release PGO build, it speeds up and matches the default perf.

Suspecting something like the following -- the workload action unroll for release PGO ends up always calling the precode somehow, and this never gets updated/backpatched?

Looks like the issue was that the jit was not fully initializing some of the SIMD12 vector constants used by this code, causing the high float to pick up random values, that might be denorms, which cause huge stalls.

Fix is #85362

System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace(NumberOfBytes: 200000000)

Default [link]

January 2023 improvement is #79346
February 2023 improvement is possibly #79381

PGO [link]

October 2022 regression is #70941
January 2023 improvement looks like #80481
February 2023 improvement possibly #79381
April 2023 regression is from #84427

This repros locally

Method	Job	Toolchain	NumberOfBytes	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
Base64DecodeInPlace	Job-BGTHBT	Default	200000000	123.6 ms	1.15 ms	0.89 ms	123.5 ms	122.3 ms	125.8 ms	1.00	0.00	400 B	1.00
Base64DecodeInPlace	Job-DGNZZW	PGO	200000000	261.5 ms	47.18 ms	50.48 ms	278.5 ms	123.0 ms	287.5 ms	2.04	0.49	400 B	1.00

But when I enable profiling -p ETW it vanishes

Method	Job	Toolchain	NumberOfBytes	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
Base64DecodeInPlace	Job-ODJYCK	Default	200000000	125.4 ms	6.63 ms	6.20 ms	123.7 ms	121.3 ms	141.2 ms	1.00	0.00	400 B	1.00
Base64DecodeInPlace	Job-NQEVUA	PGO	200000000	124.7 ms	4.63 ms	4.33 ms	123.4 ms	122.0 ms	139.8 ms	1.00	0.06	400 B	1.00

Seems like the "fast" run of this test executes R2R code and never tiers up:

(also looks like something is broken with R2R rundown as neither PerfView nor my tool can resolve which R2R method this is).

Think the issue is in BDN itself: dotnet/BenchmarkDotNet#2296

So, why are we looking at R2R code here? (Potentially if I can catch a slow run in the profiler, it may show hits at Tier0, OSR, or Tier1...will keep trying).

Caught a slow run, and despite what I wrote above the "fast" runs look like they are indeed at Tier1.

;; default

86.93%   1.63E+07    Tier-1   [System.Private.CoreLib]Base64.DecodeFromUtf8InPlace(value class System.Span`1<unsigned int8>,int32&)
13.07%   2.45E+06    native   System.Private.CoreLib.dll

;; pgo

48.42%   2.669E+07   native   coreclr.dll
47.01%   2.591E+07   Tier-1 instr  [System.Private.CoreLib]Base64.DecodeFromUtf8InPlace(value class System.Span`1<unsigned int8>,int32&)
04.44%   2.45E+06    native   System.Private.CoreLib.dll
00.09%   5E+04       native   ntoskrnl.exe

The hot method is evidently R2R and so we go to Tier-1 instr with PGO but it doesn't get called enough to ever finish tiering up. If I rev up the iterations ....

WorkloadResult  29: 1 op, 283866900.00 ns, 283.8669 ms/op
WorkloadResult  30: 1 op, 291856400.00 ns, 291.8564 ms/op
WorkloadResult  31: 1 op, 286046600.00 ns, 286.0466 ms/op
WorkloadResult  32: 1 op, 287347400.00 ns, 287.3474 ms/op
WorkloadResult  33: 1 op, 121137000.00 ns, 121.1370 ms/op
WorkloadResult  34: 1 op, 121175600.00 ns, 121.1756 ms/op
WorkloadResult  35: 1 op, 122286200.00 ns, 122.2862 ms/op
WorkloadResult  36: 1 op, 123082000.00 ns, 123.0820 ms/op
WorkloadResult  37: 1 op, 122119700.00 ns, 122.1197 ms/op

as you can see we need to get to iteration 33 to switch to Tier1 with PGO, but by default we only run 15 iterations. And once we switch we get basically the same perf as without PGO.

BDN could probably do a better job here of warning you that if there are less than say 30 invocations (perhaps 60 to be safe) of your benchmark method you may not see the results you're expecting, and either need to drop the per-invocation time or up the invocation or iteration counts.

Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty

Default [link]

April 2023 improvement possibly #84582

PGO [link]

October 2022 improvement was #70941
February 2023 improvement was #81267
April 2023 regression was possibly #84582

Comparable for me locally

Method	Job	Toolchain	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
AddChainedConfigurationEmpty	Job-SNZWKD	\base-rel\corerun.exe	13.41 ms	0.263 ms	0.270 ms	13.30 ms	13.18 ms	14.01 ms	1.00	0.00	143.24 KB	1.00
AddChainedConfigurationEmpty	Job-AMBTRB	\diff-rel\corerun.exe	13.87 ms	1.199 ms	1.380 ms	13.10 ms	12.58 ms	16.12 ms	1.05	0.10	143.32 KB	1.00

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 0)

Default [link]

Recent improvement was from #85277

PGO [link]

Regression was from something in this range: 887c043...f92b9ef (74 commits)

From Azure Data Explorer, regression was after 2023-04-25T00:29:35Z and before 2023-04-25T03:46:39Z

Tue Apr 25 03:46:39 2023 f74138f Jeremy Koritzinsky Enable ComWrappers for NativeAOT on non-Windows (#85000)
Tue Apr 25 03:33:48 2023 ad59672 Jeremy Koritzinsky Build Mono LLVM legs on the CBL-Mariner images, but run the AOTing steps on CentOS Stream 8 with binutils (#85273)
Tue Apr 25 02:51:18 2023 516aa8c Stephen Toub Add Parallel.ForAsync (#84804)
Tue Apr 25 02:50:59 2023 759fabe Stephen Toub Fix perf regressions in Utf8Formatter for integers (#85277)
Tue Apr 25 01:14:10 2023 01e2455 Michał Petryka Add CollectionsMarshal.SetCount(list, count) (#82146)
Tue Apr 25 00:49:02 2023 41f6e79 Aleksey Kliger (λgeek) [native-library] If a dllimport is specified with an absolute path, look for it first (#85255)
Tue Apr 25 00:29:35 2023 620b0db Stephen Toub Add parsing error information to composite format string parsing exceptions (#85106)

So looks like the change that lead to this was #85277.

Method	Job	Toolchain	value	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
FormatterUInt64	Job-WWTDFM	PGO	0	6.214 ns	0.0098 ns	0.0082 ns	6.213 ns	6.203 ns	6.231 ns	2.91	0.03	-	NA
FormatterUInt64	Job-QWMANF	Default	0	2.137 ns	0.0243 ns	0.0227 ns	2.148 ns	2.082 ns	2.162 ns	1.00	0.00	-	NA


;; default

05.74%   2.18E+06    ?        Unknown
81.72%   3.103E+07   Tier-1   [MicroBenchmarks]Utf8FormatterTests.FormatterUInt64(unsigned int64)
11.72%   4.45E+06    Tier-1   [9c1446ac-187e-4ddf-b6bd-964603a8e2dd]Runnable_0.WorkloadActionUnroll(int64)

;; pgo

02.50%   1.97E+06    ?        Unknown
93.66%   7.369E+07   Tier-1   [MicroBenchmarks]Utf8FormatterTests.FormatterUInt64(unsigned int64)
03.23%   2.54E+06    Tier-1   [ea5814fd-a81b-4c82-bc90-c88fd94286d8]Runnable_0.WorkloadActionUnroll(int64)

Looks like all the time is spent in the benchmark method. Hopefully this will be easy to track down.

Seems like what is happening is that with PGO we see a lot of cold blocks, and so we don't inline some methods that we inline without PGO. One of those takes a struct by address that is also used in hot code, and so we don't fully promote, we dependently promote. And then we make some poor choices in codegen that leads to what is likely a partial stall. Sampling shows a lot of time piling up in this little stretch of code:

G_M000_IG05:                ;; offset=0035H
       mov      word  ptr [rsp+48H], 0
       mov      eax, dword ptr [rsp+48H]
       or       al, byte  ptr [rsp+49H]
       jne      G_M000_IG14

For some reason we do a wide load after a narrow store; my recollection is that the HW does not like this at all. In the non PGO this code doesn't exist because we know all the values are zero.

Looks like we widen this load in fgMorphCastedBitwiseOp. If I fix that to not widen normalize on load locals, then we recover most (but not all) of the perf. Suspect the remainder is in the higher prolog/epilog costs we see with PGO.

Method	Job	Toolchain	value	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
FormatterUInt64	Job-FINTBR	PGO/Fixed	0	2.516 ns	0.0596 ns	0.0558 ns	2.506 ns	2.455 ns	2.631 ns	1.16	0.04	-	NA
FormatterUInt64	Job-EZPSRN	Default	0	2.159 ns	0.0497 ns	0.0441 ns	2.149 ns	2.097 ns	2.254 ns	1.00	0.00	-	NA

PGO has now improved, thanks to #86491 fixing the store-foreward stall.

Default "regressed" because we enabled PGO by default. then improved with #86491

Closing this; remaining regressions will be analyzed in #87194.

	private static object? IsInstanceOfClass(void* toTypeHnd, object? obj)
	{
	if (obj == null \|\| RuntimeHelpers.GetMethodTable(obj) == toTypeHnd)
	return obj;

	MethodTable* mt = RuntimeHelpers.GetMethodTable(obj)->ParentMethodTable;
	for (; ; )
	{
	if (mt == toTypeHnd)
	goto done;

	if (mt == null)
	break;

	mt = mt->ParentMethodTable;
	if (mt == toTypeHnd)
	goto done;

	if (mt == null)
	break;

	mt = mt->ParentMethodTable;
	if (mt == toTypeHnd)
	goto done;

	if (mt == null)
	break;

	mt = mt->ParentMethodTable;
	if (mt == toTypeHnd)
	goto done;

	if (mt == null)
	break;

	mt = mt->ParentMethodTable;
	}

	// this helper is not supposed to be used with type-equivalent "to" type.
	Debug.Assert(!((MethodTable*)toTypeHnd)->HasTypeEquivalence);

	obj = null;

	done:
	return obj;
	}

Investigate microbenchmarks that regress with PGO enabled #84264

Description

Activity

AndyAyersMS commented on Apr 3, 2023

PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

Default [link]

PGO [link]

EgorBo commented on Apr 3, 2023

EgorBo commented on Apr 3, 2023

VSadov commented on Apr 3, 2023

EgorBo commented on Apr 3, 2023

VSadov commented on Apr 3, 2023

AndyAyersMS commented on Apr 4, 2023

33 remaining items

AndyAyersMS commented on Apr 24, 2023

System.Memory.Span<Byte>.SequenceCompareToDifferent(Size: 4)

Default [link]

PGO [link]

AndyAyersMS commented on Apr 24, 2023

System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400KB)

Default [link]

PGO [link]

AndyAyersMS commented on Apr 24, 2023

System.Text.RegularExpressions.Tests.Perf_Regex_Cache.IsMatch(total: 400000, unique: 1, cacheSize: 15)

Default [link]

PGO [link]

AndyAyersMS commented on Apr 24, 2023

System.Collections.IndexerSet<Int32>.Dictionary(Size: 512)

Default [link]

PGO [link]

AndyAyersMS commented on Apr 24, 2023

System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!")

Default [link]

PGO [link]

AndyAyersMS commented on Apr 24, 2023

System.Numerics.Tests.Perf_Vector3.DotBenchmark

Default [link]

PGO [link]

AndyAyersMS commented on Apr 24, 2023

System.Numerics.Tests.Perf_Vector3.ReflectBenchmark

Default [link]

PGO [link]

AndyAyersMS commented on Apr 25, 2023

System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace(NumberOfBytes: 200000000)

Default [link]

PGO [link]

AndyAyersMS commented on Apr 25, 2023

Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty

Default [link]

PGO [link]

AndyAyersMS commented on May 5, 2023

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 0)

Default [link]

PGO [link]

AndyAyersMS commented on Jun 6, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions