Repository navigation
[Perf] Changes at 6/2/2022 10:27:31 PM #70499
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | f873951b78d0a8123f4cd2e28cba08f08d67fbc4 |
| Compare | 45589f2a1ea20b4bce66058994de52f317716558 |
| Diff | Diff |
Regressions in System.Memory.Span<Byte>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IndexOfAnyFourValues - Duration of single invocation | 903.31 ns | 1.09 μs | 1.20 | 0.09 | False | |||||
| LastIndexOfValue - Duration of single invocation | 19.17 ns | 21.99 ns | 1.15 | 0.51 | False |
_1.png)
_2.png)
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span<Byte>*'Details
Payloads
Histogram
System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0867249393118492 > 948.7690292147156.
IsChangePoint: Marked as a change because one of 3/31/2022 1:45:25 PM, 5/11/2022 7:18:06 PM, 6/2/2022 5:58:32 PM, 6/9/2022 12:28:12 AM falls between 5/31/2022 10:47:41 AM and 6/9/2022 12:28:12 AM.
IsRegressionStdDev: Marked as regression because -16.877893884964973 (T) = (0 -1049.0027725183602) / Math.Sqrt((501.55979866854403 / (20)) + (1240.232149616815 / (25))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (25) - 2, .025) and -0.16150501615257018 = (903.1409747959003 - 1049.0027725183602) / 903.1409747959003 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.Span<Byte>.LastIndexOfValue(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.98536000052354 > 20.17013587940221.
IsChangePoint: Marked as a change because one of 2/23/2022 2:01:04 AM, 5/31/2022 5:02:26 PM, 6/9/2022 12:28:12 AM falls between 5/31/2022 10:47:41 AM and 6/9/2022 12:28:12 AM.
IsRegressionStdDev: Marked as regression because -13.936089658846777 (T) = (0 -25.901843992515968) / Math.Sqrt((0.2063552136918562 / (11)) + (6.7214928616690255 / (34))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (34) - 2, .025) and -0.33389633896855037 = (19.418183584299246 - 25.901843992515968) / 19.418183584299246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Activity
DrewScoggins commentedon Jun 9, 2022
Regression is only in the second test, and seems related to #68874
1 remaining item
tannergooding commentedon Jun 16, 2022
First one is noise, the other is a known issue related to #70182. The overall improvements were greater than the few small regressions.