Repository navigation
Regressions in System.Memory.ReadOnlySpan.IndexOfString #80131
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 1d15f2140f7eb30a976c66290491ec89cd628da0 |
| Compare | 5b8ebeabb32f7f4118d0cc8b8db28705b62469ee |
| Diff | Diff |
Regressions in System.Memory.ReadOnlySpan
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IndexOfString - Duration of single invocation | 36.31 ns | 40.85 ns | 1.13 | 0.01 | False | |||||
| IndexOfString - Duration of single invocation | 31.12 ns | 34.71 ns | 1.12 | 0.03 | False | |||||
| IndexOfString - Duration of single invocation | 34.41 ns | 37.86 ns | 1.10 | 0.02 | False | |||||
| IndexOfString - Duration of single invocation | 43.46 ns | 46.60 ns | 1.07 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'Details
Payloads
Histogram
System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.85457512057035 > 38.125698392876544.
IsChangePoint: Marked as a change because one of 12/29/2022 3:42:58 PM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -79.23928720326055 (T) = (0 -40.82951839583939) / Math.Sqrt((0.09892261063622323 / (32)) + (0.0035919758079250097 / (11))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (11) - 2, .025) and -0.12798122119304559 = (36.19698415958977 - 40.82951839583939) / 36.19698415958977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.705853042045256 > 32.7695864837413.
IsChangePoint: Marked as a change because one of 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -32.065618011914694 (T) = (0 -34.47368339491605) / Math.Sqrt((0.295698930939249 / (33)) + (0.022469384377310872 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.10922615215631908 = (31.079039497851475 - 34.47368339491605) / 31.079039497851475 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.86361599332396 > 36.17684605418915.
IsChangePoint: Marked as a change because one of 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -121.32784063659697 (T) = (0 -37.83310557360609) / Math.Sqrt((0.0150146128606248 / (33)) + (0.0032601126619518227 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.09844463061357793 = (34.44243298132649 - 37.83310557360609) / 34.44243298132649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 46.59984337401849 > 45.51546186967618.
IsChangePoint: Marked as a change because one of 10/25/2022 2:34:53 PM, 11/18/2022 8:48:12 PM, 12/13/2022 4:26:35 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -29.70329656771073 (T) = (0 -46.50571894072624) / Math.Sqrt((0.39859475663884625 / (33)) + (0.006630446944236772 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.07769776189188264 = (43.15283986401357 - 46.50571894072624) / 43.15283986401357 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Activity
[-][Perf] Windows/x64: 4 Regressions on 12/30/2022 6:06:02 AM[/-][+]Regressions in System.Memory.ReadOnlySpan[/+][-]Regressions in System.Memory.ReadOnlySpan[/-][+]Regressions in System.Memory.ReadOnlySpan.IndexOfString[/+]EgorBo commentedon Jan 3, 2023
#79787 cc @MihaZupan
MihaZupan commentedon Jan 3, 2023
The regressed cases all share the fact that the values are short, and therefore hit the non-vectorized path. The numbers here match what was expected on that PR #79787 (comment).
I'm working on an improvement to
IndexOfAnyValuesthat would also vectorize inputs of 4-7 chars, but we'd still have the regression for inputs of 1-3 chars.EgorBo commentedon Jan 3, 2023
@MihaZupan feel free to close this one in that case