PlaintextMVC benchmark is slow on arm64

@Maoni0

Plaintext-PlaintextMVC benchmark should benefit a lot from PGO (namely, from guarded devirtualizations and inlining) and it does benefit from it on all x64 platforms (Linux, Windows, Intel, Amd, etc) - up to +40% more RPS. Unfortunately, it's not the case for arm64 where there is no difference between DynamicPGO and Default. Moreover, the benchmark is 7-8x times slower on arm64 in comparison with x64-dynamicpgo (while I'd expect it to be 1.5-2x slower only).

It looks to me that on arm64 it's bound to JIT_New:

while on x64 it looks like this:

Namely, this call-site ("drilled") (https://github.com/dotnet/aspnetcore/blob/a0b950fc51c43289ab3c6dbea15926d32f3556cc/src/Mvc/Mvc.Core/src/Routing/ControllerRequestDelegateFactory.cs#L68-L101):
Arm64:

same call-site on x64:

Flamegraph for arm64 (two JIT_New are highlighted) for the first th:

x64:

Does it ring a bell to anyone (e.g. JIT_NewS_MP_FastPortable is not used, some gc feature is not implemented for arm64, some allocation tracker/profiler is enabled, etc.)?
/cc @dotnet/jit-contrib @Maoni0 @jkotas @davidwrighton
I can re-run the benchmark with any modifications in jit/vm/gc you suggest.

Steps to get the native traces:

Arm64-Linux:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --scenario mvc --profile aspnet-citrine-arm --application.collect true --application.collectStartup false --application.options.collectCounters false   --application.dotnetTrace false --application.framework net6.0 --application.environmentVariables DOTNET_TieredPGO=1  --application.environmentVariables DOTNET_TC_QuickJitForLoops=1  --application.environmentVariables DOTNET_ReadyToRun=0

x64-Linux:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --scenario mvc --profile aspnet-citrine-lin --application.collect true --application.collectStartup false --application.options.collectCounters false   --application.dotnetTrace false --application.framework net6.0 --application.environmentVariables DOTNET_TieredPGO=1  --application.environmentVariables DOTNET_TC_QuickJitForLoops=1  --application.environmentVariables DOTNET_ReadyToRun=0

Powerbi link: https://aka.ms/aspnet/benchmarks (open "PGO" page there)

A similar benchmark "Plaintex-Plaintext" looks much better:

Does it ring a bell to anyone (e.g. slow jit helper is used, some gc feature is not implemented for arm64, etc)?

Do we know that is actually slower as opposed to maybe something else is causing that lambda to be invoked many more times?

Does it ring a bell to anyone (e.g. slow jit helper is used, some gc feature is not implemented for arm64, etc)?

Do we know that is actually slower as opposed to maybe something else is causing that lambda to be invoked many more times?

Not sure, but I'd expect codepaths to be exactly the same, btw, here is a comparison for all benchmarks we run PGO for:

Exactly the same anomalies in JsonMVC, Foruntes and FortunesEF where arm64 is 5-7x times slower.

^ Most of the benchmarks get a nice boost from DynamicPGO (to be more precise - from FullPGO where we disable R2R and enable QuickJitForLoops) on x64, but I don't see the same boost on arm64 for Fortunes, FortunesEF, JsonMVC and PlaintextMVC - and I suspect that it's because of "bound to JIT_New".

@sebastienros

I don't know the exact arm64 hardware we have, but NodeJS benchmarks show around 50-100% difference between arm64 and x64 so I guess we can expect the same ratio on .NET.

JsonMvc and Plaintext benchmarks on NodeJS:

It'd be nice if we could run Actix (Rust) benchmarks on arm64 too - it should show a more precise ratio due to the fact it uses LLVM which is heavily optimized for both x64 and arm64.

Currently, we only run Actix for x64.

cc @sebastienros

AFAIR the Docker image defined in TE's repo for Actix is not working for ARM64. Will try again.

AFAIR the Docker image defined in TE's repo for Actix is not working for ARM64. Will try again.

It's not necessary to run Actix but anything fast and other than .NET on both x64 and arm64 just to feel the difference

Could you please take a look at the GC info tab in perf-view that shows the GC stats, and share it here for x64 vs. arm64?

@jkotas

@jkotas while I'm re-opening the traces (it takes quite some time on my PC) here are the perf counters (--application.options.collectCounters true):

NB: these are markdown formatted and also correctly spaced so you can either paste them as-is or wrap them in triple back ticks.

Stating the obvious:
Some x64 numbers are multiples of arm64 because RPS is just higher, e.g. Allocations, TP, so nothing weird here.
But the GC is abnormally high on arm64: 66% cpu and 59 gen 0 collections per second and the Gen 0 size significantly higher too.

Also can you use --chart, this will show if the max GC (66%) is just a small period of time, of if it's for the whole run.

I think the GC is clearly part of the problem. Gen0 GC are firing 4x more frequently per second and throughput is 8x lower. It means that the GC rate is effectively 30x higher. @dotnet/gc should take a look.

@Maoni0

If I am reading the traces, it seems that GC pause time on arm64 is 5X than x64.

Both scenarios were ran using flags
--application.dotnetTrace true --application.dotnetTraceProviders gc-collect --application.memoryLimitInBytes 17179869184 --application.cpuSet 0-7.

x64:

arm64: (in addition to above flags, also added -application.environmentVariables DOTNET_GCgen0size=1E00000)

@Maoni0 - can you suggest any further steps to track this down?

does it also collect more? what's the survived bytes for these GCs? you could also share the traces.

does it also collect more? what's the survived bytes for these GCs? you could also share the traces.

I will email them to you.

@Maoni0

Looking at the traces with @Maoni0, there are some unusual callstack present in StackWalker code that does more work in arm64 compared to x64:

arm64:

x64:

@mangod9 - Do you know what would make arm64 do more work than x64 in stack walking code?

@AntonLapounov

@AntonLapounov @janvorli in case they have seen this before. Could you please share traces with us?

Could you please share traces with us?

Done over the email.

@Maoni0

Update:

On @Maoni0 suggestion, we came up with a concentrated repro without the needing of threadpool/thread.sleep. With that we can see the slowness in allocation rate and regression in max pause time.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp15
{
    public class GC_Test
    {
        static int depth = 25;
        public class Node
        {
            public Node left;
            public Node right;
        }
        static void Populate(int iDepth, Node thisNode)
        {
            if (iDepth <= 0)
                return;
            else
            {
                iDepth--;
                thisNode.left = new Node();
                thisNode.right = new Node();
                Populate(iDepth, thisNode.left);
                Populate(iDepth, thisNode.right);
            }
        }

        public static void Main(string[] args)
        {
            Console.WriteLine("Started");
            var stopwatch = Stopwatch.StartNew();
            Node root = new Node();
            Populate(depth, root);
            for (int i = 0; i < 100; i++)
            {
                GC.Collect();
            }
            Console.WriteLine($"Finished in {stopwatch.ElapsedMilliseconds}ms");
        }
    }
}

application	x64_gc_collectCounters	arm64_gc_collectCounters	Diff	Comments
Execution Time	10334msec	28873msec	+178%	<--
CPU Usage (%)	4	4	0.00%
Cores usage (%)	112	143	+27.68%
Working Set (MB)	449	558	+24.28%
Private Memory (MB)	1,318	1,413	+7.21%
Build Time (ms)	1,913	5,885	+207.63%
Start Time (ms)	44	98	+122.73%
Published Size (KB)	70,441	77,501	+10.02%
.NET Core SDK Version	6.0.101	6.0.101
Max CPU Usage (%)	14	17	+21.43%
Max Working Set (MB)	520	585	+12.50%
Max GC Heap Size (MB)	453	501	+10.60%
Size of committed memory by the GC (MB)	454	503	+10.79%
Max Number of Gen 0 GCs / sec	26.00	5.00	-80.77%
Max Number of Gen 1 GCs / sec	14.00	3.00	-78.57%
Max Number of Gen 2 GCs / sec	2.00	1.00	-50.00%
Max Time in GC (%)	86.00	99.00	+15.12%
Max Gen 0 Size (B)	24	24	0.00%
Max Gen 1 Size (B)	20,185,184	62,914,656	+211.69%
Max Gen 2 Size (B)	433,981,480	471,853,312	+8.73%
Max LOH Size (B)	24	24	0.00%
Max Allocation Rate (B/sec)	245,607,168	115,520,480	-52.97%	<--
Max GC Heap Fragmentation	0	0	+25.45%
# of Assemblies Loaded	6	6	0.00%
Max Exceptions (#/s)	0	0
Max Lock Contention (#/s)	0	0
Max ThreadPool Threads Count	0	0
Max ThreadPool Queue Length	0	0
Max ThreadPool Items (#/s)	0	0
Max Active Timers	0	0
IL Jitted (B)	11,984	21,655	+80.70%	<-- why different?
Methods Jitted	69	155	+124.64%	<-- why different?

Below, the screenshots are for GCStats and call stack for both x64 vs. arm64. Some observations:

The execution time and GC allocation rate is 2X slower on Arm64
There are 34 more GCs that happened in x64 compared to arm64 in addition to the 100 GCs that were induced using GC.Collect(). I am assuming this is because of overall speed difference. On x64, code is executing faster leading to more allocation of objects and hence more frequently GC is triggered. On arm64, the slowness lead to lower allocation and hence lower no. of GCs.
Alike previous data, even in this short repro, we can see the "Max Pause Time" is 4X more on Arm64 than on x64.
Looking into the callstack screenshots, we can see that the plan_phase and mark_phase are the biggest contributor of the diff. In plan_phase, even the simple introsort_loop takes longer to execute. My initial assumption was the difference in the code produced by LLVM (which we use to build linux x64/arm64). I tried to extract the introsort_loop (and surrounding code) and checked the diff in godbolt, but I don't see anything that would stand out.
In mark_phase, I can see a clear difference in mark_object_simple1, but I don't see why it would be the case. May be there is huge difference in the count of data both platforms operate on?
Lastly, as seen in above table, I see the no. of methods/IL Jitted is different. I will add more traces to see why that is the case.

x64:

arm64:

The execution time and GC allocation rate is 2X slower on Arm64

One of the factors that influences the GC allocation rate is memory bandwidth. It may be a good idea to measure the memory bandwidth on the two machines: Write a simple micro-benchmark that clears 100MB buffer (no GC involved).

There are 34 more GCs that happened in x64 compared to arm64 in addition to the 100 GCs that were induced using GC.Collect(). I am assuming this is because of overall speed difference. On x64, code is executing faster leading to more allocation of objects and hence more frequently GC is triggered.

I do not think that this explanation makes sense. The benchmark you have shared allocates same amount on both arm64 and x64. I think that the different number of GCs between arm64 and x64 is caused by GC setting the allocation budgets differently. The allocation budgets are computed from the cache size, so the difference is most likely caused by the different (or incorrectly computed) cache sizes between arm64 and x64. There was a discussion about the buggy cache size computation earlier in this thread. Are you using any env. variables to compensate for it?

There was a discussion about the buggy cache size computation earlier in this thread. Are you using any env. variables to compensate for it?

Yes. I am using the environment variable on arm64 to set the cache size. I am also setting the memory and CPU same for x64/arm64.

--application.environmentVariables DOTNET_GCgen0size=1E00000 --application.memoryLimitInBytes 17179869184 --application.cpuSet 0-7.

@kunalspathak

@kunalspathak How is it possible that Max Gen 0 Size is only 24 bytes for your small repro and only 192 bytes for the benchmark on ARM64?

@kunalspathak

@kunalspathak How is it possible that Max Gen 0 Size is only 24 bytes for your small repro and only 192 bytes for the benchmark on ARM64?

Honestly I am not sure how those numbers are collected and reported. Currently I am working with @Maoni0 to gather traces for https://github.com/dotnet/performance/tree/main/src/benchmarks/gc/src/exec/GCPerfSim.

@AntonLapounov

@AntonLapounov that's just the generation size after a GC, if there's no pinning, it could easily be 24 bytes (which is just a min object size)

Closing as completed

Navigation Menu

Repository navigation

PlaintextMVC benchmark is slow on arm64 #60166

Steps to get the native traces:

37 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PlaintextMVC benchmark is slow on arm64 #60166

Description

Steps to get the native traces:

Activity

EgorBo commented on Oct 8, 2021

stephentoub commented on Oct 8, 2021

EgorBo commented on Oct 8, 2021

EgorBo commented on Oct 8, 2021

EgorBo commented on Oct 8, 2021

sebastienros commented on Oct 8, 2021

EgorBo commented on Oct 8, 2021

jkotas commented on Oct 8, 2021

EgorBo commented on Oct 8, 2021

sebastienros commented on Oct 8, 2021

jkotas commented on Oct 8, 2021

37 remaining items

kunalspathak commented on Jan 3, 2022

Maoni0 commented on Jan 3, 2022

kunalspathak commented on Jan 3, 2022

kunalspathak commented on Jan 3, 2022

mangod9 commented on Jan 3, 2022

kunalspathak commented on Jan 5, 2022

kunalspathak commented on Jan 14, 2022

jkotas commented on Jan 15, 2022

kunalspathak commented on Jan 17, 2022

AntonLapounov commented on Feb 2, 2022

kunalspathak commented on Feb 2, 2022

Maoni0 commented on Feb 2, 2022

EgorBo commented on Jul 6, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions