Repository navigation
[Perf -118%] System.Collections.Tests.Perf_BitArray for small inputs (3) #37813
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Changes | diff |
Regressions in System.Collections.Tests.Perf_BitArray
| Benchmark | Baseline | Test | Test/Base | Modality | Baseline Outlier |
|---|---|---|---|---|---|
| BitArrayNot | 1.25 ns | 2.34 ns | 1.87 | False | |
| BitArrayIntArrayCtor | 143.62 ns | 151.32 ns | 1.05 | False | |
| BitArraySetAll | 1.56 ns | 5.66 ns | 3.62 | False |
Historical Data in Reporting System
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.Tests.Perf_BitArray*';Details
Histogram
System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 4)
[0.372 ; 1.006) | @@@@@@@@@@@@@@
[1.006 ; 1.753) | @@@@@@@
[1.753 ; 2.387) | @@@@@@@@@@@
[2.387 ; 3.012) | @@@
System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)
[136.771 ; 143.213) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[143.213 ; 149.753) | @@
[149.753 ; 153.484) | @@@@@@@@
[153.484 ; 157.663) | @@@@@@@@@@@
System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 4)
[0.647 ; 2.210) | @@@@@@@@@@@@@@@@
[2.210 ; 3.772) |
[3.772 ; 4.562) |
[4.562 ; 6.125) | @@@@@@@@@@@@@@@@@@@
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Activity
DrewScoggins commentedon Jun 12, 2020
@adamsitnik
danmoseley commentedon Jul 2, 2020
@DrewScoggins do I understand correctly that the regression here is between the last datapoint before the repo consolidation and the first datapoint after ? It seems the regression was at that time, and there was some overlap (consistent with it being the old and new infrastructure)?
I was trying to get something useful out of the diff link, but it just encompasses 6 months worth of work so it isn't useful. But even scoping to the point of the regression, if the regression is across the consolidation point, the product code may not even have changed (?).
The nearest potentially relevant BitArray change entered corefx master in Nov.
DrewScoggins commentedon Jul 9, 2020
As you can see from the graph it does look like this regression came in in November, so good tracking down to that commit.
Historical Data in Reporting System
danmoseley commentedon Jul 9, 2020
Ah good that helps. In the original PR, @Gnbrkm41 noted that there was some regression for the smallest size (which is non vectorized). These scenarios were added at the same time it looks like (dotnet/performance#1026)
I think we figure that's OK - these are tiny bit arrays and tiny absolute times. @DrewScoggins is there still a regression in the int ctor 512 size? You reported it above, but it's not in the latest report
DrewScoggins commentedon Jul 9, 2020
That is because it is floating right around the regression threshold. So in the latest run it just slightly dropped, but that is enough to move it below 5%. Given that we have many other issues with much bigger regressions, we can probably leave this for now and if we see it come back we can take another look then.
danmoseley commentedon Jul 9, 2020
OK I'm going to close this then as acceptable/known based on current data and discussion in original PR.
cc @eiriktsarpalis who owns collections
[-][Perf -118%] System.Collections.Tests.Perf_BitArray (3)[/-][+][Perf -118%] System.Collections.Tests.Perf_BitArray for small inputs (3)[/+]