Repository navigation
Performance regression: 6x slower array allocation on Alpine #41398
Description
From the data that I got from @danmosemsft which was collected by running dotnet/performance microbenchmarks on alpine 3.11 via WSL2, it looks like allocating arrays of both value and reference types became 6 times slower compared to 3.1.
Initially, I thought that it was just an outlier, but I can see the same pattern for other collections that internally use arrays (queue, list, stack etc). The regression is specific to alpine. Ubuntu 18.04 (with and without WSL2) is fine.
@jkotas @janvorli who would be the best person to investigate that?
Repro
git clone https://github.com/dotnet/performance.git
python3 ./performance/scripts/benchmarks_ci.py -f netcoreapp3.1 netcoreapp5.0 --filter 'System.Collections.CtorGivenSize<Int32>.Array'System.Collections.CtorGivenSize.Array(Size: 512)
| Conclusion | Base | Diff | Base/Diff | Modality | Operating System | Bit | Processor Name | Base Runtime | Diff Runtime |
|---|---|---|---|---|---|---|---|---|---|
| Same | 181.79 | 183.96 | 0.99 | Windows 10.0.18363.1016 | Arm | Microsoft SQ1 3.0 GHz | .NET Core 3.1.6 | 5.0.100-rc.1.20413.9 | |
| Same | 92.89 | 94.47 | 0.98 | Windows 10.0.18363.959 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 3.1.6 | 5.0.100-rc.1.20404.3 | |
| Same | 96.05 | 94.36 | 1.02 | Windows 10.0.18363.959 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 3.1.6 | 5.0.100-rc.1.20418.3 | |
| Same | 114.74 | 111.94 | 1.03 | Windows 10.0.19041.450 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 3.1.6 | 5.0.100-rc.1.20413.9 | |
| Same | 80.49 | 79.98 | 1.01 | Windows 10.0.19041.450 | X64 | Intel Core i7-6700 CPU 3.40GHz (Skylake) | .NET Core 3.1.6 | 5.0.100-rc.1.20419.9 | |
| Same | 67.30 | 67.66 | 0.99 | bimodal | Windows 10.0.19042 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 3.1.6 | 5.0.100-rc.1.20418.3 |
| Same | 86.10 | 79.17 | 1.09 | bimodal | Windows 10.0.19041.450 | X64 | Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) | .NET Core 3.1.6 | 5.0.100-rc.1.20419.14 |
| Same | 97.50 | 98.77 | 0.99 | Windows 10.0.18363.959 | X86 | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 3.1.6 | 5.0.100-rc.1.20420.14 | |
| Slower | 127.02 | 150.46 | 0.84 | bimodal | Windows 10.0.19041.450 | X86 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 3.1.6 | 5.0.100-rc.1.20419.5 |
| Slower | 193.61 | 287.83 | 0.67 | bimodal | ubuntu 18.04 | Arm64 | Unknown processor | .NET Core 3.1.6 | 6.0.100-alpha.1.20421.6 |
| Same | 99.85 | 103.42 | 0.97 | ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 3.1.6 | 5.0.100-rc.1.20403.23 | |
| Slower | 138.73 | 151.37 | 0.92 | macOS Mojave 10.14.5 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 3.1.6 | 5.0.100-rc.1.20404.2 | |
| Slower | 72.85 | 515.56 | 0.14 | alpine 3.11 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 3.1.6 | 6.0.100-alpha.1.20421.6 | |
| Slower | 78.85 | 90.76 | 0.87 | ubuntu 18.04 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 3.1.6 | 5.0.100-rc.1.20418.3 |
Activity
[-]Performance regression: 5x slower array allocation on Alpine[/-][+]Performance regression: 6x slower array allocation on Alpine[/+]danmoseley commentedon Aug 26, 2020
@billwert
jkotas commentedon Aug 26, 2020
These tests are measuring raw GC allocation speed. This is GC performance regression.
@mangod9 Could you please find somebody to investigate this?
mangod9 commentedon Aug 26, 2020
Ok will have someone take a look. @adamsitnik , Is the regression specifically with Alpine on WSL2, or Alpine in general?
adamsitnik commentedon Aug 26, 2020
mangod9 The benchmarks were executed with Alpine on WSL2, I don't have "Alpine only" results to be able to answer. FWIW the Ubuntu on WSL2 results are fine so it might be a general Alpine problem
jkotas commentedon Aug 26, 2020
It repros without WSL2 too.
The problem is that the GC is running like 100x more often than it should. It is likely problem in the budget computation. One of the places to check is
PAL_GetLogicalProcessorCacheSizeFromOS.mangod9 commentedon Aug 26, 2020
ok thanks for the context. We will look if there is any Alpine specific behavior which is leading to this.
19 remaining items
danmoseley commentedon Sep 1, 2020
Thanks @adamsitnik and @mangod9 for resolving, I'm very happy that my data turned out to be rather useful!
adamsitnik commentedon Sep 1, 2020
@danmosemsft @mangod9 I wonder how we could prevent from a similar problem in the future. Perhaps we should add some asserts to the method to ensure that it never returns 0 again?
mangod9 commentedon Sep 1, 2020
@danmosemsft Yeah it was very helpful in pinpointing where the issue might be. @adamsitnik yeah I will create a separate issue to track how we can add a test/asserts for this.
danmoseley commentedon Sep 1, 2020
@Lxiamail @billwert @DrewScoggins this is evidence I think that regular Alpine runs in the lab are important. This would have justified servicing, I think. I'm not sure where we left that conversation @billwert ?
Lxiamail commentedon Sep 1, 2020
@danmosemsft Yes, we have an email discussion about adding additional OS coverage in perf lab. We will looks into @adamsitnik's finalized report of exercises' data, and I'm trying to get .NET Core OS usage telemetry data. Hope we can identify the commonly used OSes and add them to perf lab.
danmoseley commentedon Sep 1, 2020
@Lxiamail sounds good, thanks! Telemetry aside, I think if we can only do 2 Linux flavors, Alpine probably needs to be one of those because of its unique characteristics.
Lxiamail commentedon Sep 1, 2020
Lxiamail commentedon Sep 1, 2020
Just as side note, https://stackshare.io/stackups/alpine-linux-vs-ubuntu shows Alpine has a lot less users than Ubuntu. If only using OS telemetry to pick the Linux OSes for perf lab, Alpine may not pop to the top rank.
mangod9 commentedon Sep 1, 2020
Alpine is different since it uses the
musl libcinstead of GNU, thus its different from other distros. For this particular fix constants such as_SC_LEVEL1_DCACHE_SIZEare not defined for musl. Hence the fallback to using a difference method to retrieve the cache size was mostly alpine specific (there are several such subtle differences on Alpine).danmoseley commentedon Sep 1, 2020
Right, anything related to libc can have different characteristics. Beyond musl, Alpine is the most "size optimized" distro we currently support, so it may have unique characteristics elsewhere because of those optimizations (eg possibly missing bits of /proc - I don't know, just an example). Plus, folks who deploy to Alpine are presumably particularly sensitive to perf (or at least size) so we should particularly care about the perf on Alpine.
cc @richlander in case he wants to add anything.
Lxiamail commentedon Sep 1, 2020
@mangod9 @danmosemsft thanks for the info! Looks like we definitly should add Alpine to perf lab. Is there any other OSes have unique charactenstics, which should be considered to monitor in perf lab?
danmoseley commentedon Sep 1, 2020
Others are better positioned to answer that one, off the top of my head I cannot remember Linux regressions that wouldn't show up in either Ubuntu or Alpine.
mangod9 commentedon Sep 1, 2020
Adding Alpine is a good start, we can always re-evaluate if we find any other distro specific issues.