Skip to content

Regression from hoisting out of nested loop #71059

Closed
Closed
Regression from hoisting out of nested loop#71059
Assignees
kunalspathak
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtime
Milestone
7.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on Jun 21, 2022

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e5acd4dfd7b31ed790687e9f3da642fe1c964dc3
Compare 261574bf4121b40c7023c73571fb7a690397bd0f
Diff Diff

Regressions in Benchstone.MDBenchI.MDGeneralArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 13.31 ms 16.47 ms 1.24 0.00 False
Test2 - Duration of single invocation 13.21 ms 16.47 ms 1.25 0.00 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'
Details

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDGeneralArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.47376638701923 > 13.9781989521875.
IsChangePoint: Marked as a change because one of 6/14/2022 10:05:19 AM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -367.3133210148429 (T) = (0 -16562240.309642797) / Math.Sqrt((970113886.8242489 / (30)) + (1800460994.5065267 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.24455339319226346 = (13307778.03526843 - 16562240.309642797) / 13307778.03526843 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Benchstone.MDBenchI.MDGeneralArray.Test2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.472966578125 > 13.863741196189906.
IsChangePoint: Marked as a change because one of 6/14/2022 10:05:19 AM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -314.5082831379247 (T) = (0 -16561237.119994191) / Math.Sqrt((2077293600.8144262 / (30)) + (1676453099.0390358 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.2518517315082016 = (13229391.870586464 - 16561237.119994191) / 13229391.870586464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline e5acd4dfd7b31ed790687e9f3da642fe1c964dc3
Compare 261574bf4121b40c7023c73571fb7a690397bd0f
Diff Diff

Regressions in Burgers

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test3 - Duration of single invocation 87.95 ms 115.96 ms 1.32 0.08 False
Test1 - Duration of single invocation 170.74 ms 261.06 ms 1.53 0.00 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Burgers*'
Details

Payloads

Baseline
Compare

Histogram

Burgers.Test3

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.955026975 > 92.70167878125.
IsChangePoint: Marked as a change because one of 6/14/2022 10:05:19 AM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -76.86974915808065 (T) = (0 -117696004.90405983) / Math.Sqrt((712968655004.949 / (30)) + (4743413455200.672 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.3316230571523148 = (88385376.23083334 - 117696004.90405983) / 88385376.23083334 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Burgers.Test1

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 261.05918285714284 > 179.29468991538465.
IsChangePoint: Marked as a change because one of 6/14/2022 10:05:19 AM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -1718.0951023682271 (T) = (0 -261284212.0427163) / Math.Sqrt((23363070275.099754 / (30)) + (77899970943.01614 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.530140609621464 = (170758301.8186508 - 261284212.0427163) / 170758301.8186508 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e5acd4dfd7b31ed790687e9f3da642fe1c964dc3
Compare 261574bf4121b40c7023c73571fb7a690397bd0f
Diff Diff

Regressions in Benchstone.BenchI.Array2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 679.53 ms 906.73 ms 1.33 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.Array2*'
Details

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.Array2.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 906.7278614615384 > 712.94808081875.
IsChangePoint: Marked as a change because one of 6/14/2022 10:05:19 AM, 6/21/2022 6:07:15 AM falls between 6/12/2022 5:56:07 PM and 6/21/2022 6:07:15 AM.
IsRegressionStdDev: Marked as regression because -605.4595292637864 (T) = (0 -907427594.9044238) / Math.Sqrt((3461697173857.5376 / (30)) + (1015343504891.7698 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.33497085726601056 = (679735883.3456594 - 907427594.9044238) / 679735883.3456594 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    assigned
    EgorBo
    on Jun 21, 2022
  2. performanceautofiler
    added
    CoreClr
    refs/heads/main
    Regression
    RunKind=micro
    ubuntu 18.04
    untriagedNew issue has not been triaged by the area owner
    x64
    on Jun 21, 2022
  3. kunalspathak
    changed the title [-][Perf] Changes at 6/14/2022 2:27:40 PM[/-] [+]Regression from hoisting out of nested loop[/+] on Jun 21, 2022
  4. kunalspathak
    transferred this issue fromdotnet/perf-autofiling-issueson Jun 21, 2022
  6. kunalspathak
    assigned
    kunalspathak
    and unassigned
    EgorBo
    on Jun 21, 2022

  7. kunalspathak commented on Jun 21, 2022

    @kunalspathak
    kunalspathak
    on Jun 21, 2022
    Contributor

    #68061

  8. kunalspathak commented on Jun 21, 2022

    @kunalspathak
    kunalspathak
    on Jun 21, 2022
    Contributor

    Some other: dotnet/perf-autofiling-issues#6211 and dotnet/perf-autofiling-issues#6276

  9. 7 remaining items

  10. kunalspathak commented on Aug 1, 2022

    @kunalspathak
    kunalspathak
    on Aug 1, 2022
    Contributor

    Likewise for Benchstone.BenchI.Array2, there is no change to the generated code that would have impacted the performance.

    https://www.diffchecker.com/NyBN3k3F

  11. kunalspathak commented on Aug 2, 2022

    @kunalspathak
    kunalspathak
    on Aug 2, 2022
    Contributor

    I added a flag to disable "hoisting expression out of nested loop" in kunalspathak@a29daa4 to see what the impact is on performance post #70271. Hoisting doesn't change the generated code as seen in https://www.diffchecker.com/wQsuJBye. Here, left is code generated without "hoisting out of nested loop" and right is code generated with "hoisting out of nested loop" (same as main).

    In fact, we hoist fetching upper/lower bounds out of nested loops as seen below.

    image

    So, to conclude, there could be regression from hoisting changes because of increasing register pressure, but MD optimizations bring that down. Also, by verifying the code without loop hoisting, we don't see much difference and we can conclude that it has not blocked any further improvements that we see in #71059 (comment). There is no further action needed for this issue, so I am closing it.

  12. kunalspathak
    closed this as completedon Aug 2, 2022

  13. mrsharm commented on Aug 8, 2022

    @mrsharm
    mrsharm
    on Aug 8, 2022
    Member

    @kunalspathak, this regression showed up in multiple configurations while working on the monthly perf report. Should this regression be triaged as "by design"?

    Thanks!

    Details

    Benchstone.BenchI.Array2.Test

    Result Ratio Alloc Delta Operating System Bit Processor Name
    Slower 0.86 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
    Same 1.02 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
    Same 0.94 +0 macOS Monterey 12.3 Arm64 Apple M1 Max
    Slower 0.89 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
    Same 0.90 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
    Slower 0.64 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
    Slower 0.68 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
    Slower 0.87 +0 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
    Same 0.95 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
    Same 1.01 +0 Windows 11 X64 AMD Ryzen 9 3950X
    Slower 0.89 +0 Windows 11 X64 AMD Ryzen 9 5900X
    Same 0.97 +0 Windows 11 X64 AMD Ryzen 9 5950X
    Slower 0.66 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
    Same 0.94 +0 Windows 11 X64 Intel Core i9-10900K CPU 3.70GHz
    Slower 0.66 -288 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
    Same 1.03 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
    Same 1.04 +0 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
    Slower 0.77 +0 ubuntu 18.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
    Same 0.93 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
    Same 1.02 +0 ubuntu 20.04 X64 Intel Core i9-10900K CPU 3.70GHz
    Same 1.00 +0 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
    Same 1.00 +0 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
    Same 0.99 +0 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
    Slower 0.85 +0 macOS Big Sur 11.6.8 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
    Same 0.92 -672 macOS Monterey 12.3.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
    Same 0.91 +0 macOS Monterey 12.4 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

  14. kunalspathak commented on Aug 9, 2022

    @kunalspathak
    kunalspathak
    on Aug 9, 2022
    Contributor

    I re-checked the Array2 benchmarks on unix/x64 as well as on Arm64 with/without loop hoisting and nothing pops out. The code is definitely better where we hoist certain invariants outside the nested loop. There could be JCC erratum triggering (at least for unix x64 case) that is causing it. E.g. https://www.diffchecker.com/XdBiwwH4

    Other diffs are included in zip file for all the methods involved in the benchmark.

    71059.zip

    I will call it as "by design".

  15. ghost locked as resolved and limited conversation to collaborators on Sep 8, 2022
  16. jeffhandley
    added
    runtime-coreclrspecific to the CoreCLR runtime
    os-linuxLinux OS (any supported distro)
    arch-x64
    and removed
    CoreClr
    refs/heads/main
    RunKind=micro
    Regression
    ubuntu 18.04
    x64
    on Dec 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtime

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions