Skip to content

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex #96496

New issue
New issue
Closed as not planned
Closed as not planned
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex#96496
Assignees
MihaZupan
Labels
arch-arm64area-System.Text.RegularExpressionsos-windowsruntime-coreclrspecific to the CoreCLR runtime
Milestone
9.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on Jan 4, 2024

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 9e57b6a280308b6e5dc85d1f0c9200cb698fb98b
Compare 653739cc0e4f6c7a44e4cda6c9ac8bb2a265505a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
161.49 μs 174.35 μs 1.08 0.33 False
7.38 ms 10.62 ms 1.44 0.27 False
145.01 μs 173.20 μs 1.19 0.39 False
110.25 μs 116.58 μs 1.06 0.00 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
Details

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: Compiled)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\b\w+n\b", Options: NonBacktracking)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: Compiled)

ETL Files

Histogram

JIT Disasms

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: Compiled)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 623cf77a58f7a233b94dcc1c3ef8eb8d67e8d948
Compare 653739cc0e4f6c7a44e4cda6c9ac8bb2a265505a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.63 secs 1.75 secs 1.07 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*'
Details

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, Compiled)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    added
    arch-arm64
    os-windows
    runtime-coreclrspecific to the CoreCLR runtime
    untriagedNew issue has not been triaged by the area owner
    on Jan 4, 2024
  2. EgorBo
    changed the title [-][Perf] Windows/arm64: 5 Regressions on 1/2/2024 10:16:29 PM[/-] [+]Regressions in System.Text.RegularExpressions.Tests.Perf_Regex[/+] on Jan 4, 2024
  3. EgorBo
    transferred this issue fromdotnet/perf-autofiling-issueson Jan 4, 2024
  4. ghost added
    area-System.Text.RegularExpressions
    on Jan 4, 2024

  6. EgorBo commented on Jan 4, 2024

    @EgorBo
    EgorBo
    on Jan 4, 2024
    Member

    Some graphs are a bit out-of-date e.g. the last one is:
    image

    Looks like it's regressed by #96402 cc @stephentoub

  7. EgorBo commented on Jan 4, 2024

    @EgorBo
    EgorBo
    on Jan 4, 2024
    Member

    Also:

  8. stephentoub commented on Jan 4, 2024

    @stephentoub
    stephentoub
    on Jan 4, 2024
    Member

    cc: @MihaZupan

  9. steveharter
    removed
    untriagedNew issue has not been triaged by the area owner
    on Jan 9, 2024
  10. steveharter
    added this to the 9.0.0 milestone on Jan 9, 2024

  11. stephentoub commented on Jun 27, 2024

    @stephentoub
    stephentoub
    on Jun 27, 2024
    Member

    @MihaZupan, could you take a look at this one? The regressions that remain all appear to be due to using SearchValues.Create(["something"], ...) rather than IndexOf("something", ...) in the generated regex code.

  12. MihaZupan commented on Jun 27, 2024

    @MihaZupan
    MihaZupan
    on Jun 27, 2024
    Member

    I'll try to take a look next week.

  13. MihaZupan commented on Jul 3, 2024

    @MihaZupan
    MihaZupan
    on Jul 3, 2024
    Member

    Rough numbers from X64:

    SliceSlice

    Method Toolchain Options Mean Error Ratio
    Count \literal\corerun.exe Compiled 450.7 ms 3.43 ms 1.00
    Count \searchvalues\corerun.exe Compiled 404.0 ms 2.28 ms 0.90
    Count \literal\corerun.exe IgnoreCase, Compiled 716.6 ms 4.60 ms 1.00
    Count \searchvalues\corerun.exe IgnoreCase, Compiled 630.7 ms 2.91 ms 0.88

    Sherlock

    Method Toolchain Pattern Mean Error Ratio
    Count \literal\corerun.exe (?i)Sherlock 58.29 us 0.657 us 1.00
    Count \searchvalues\corerun.exe (?i)Sherlock 51.43 us 0.807 us 0.88
    Count \literal\corerun.exe (?i)Sherlock Holmes 56.84 us 0.548 us 1.00
    Count \searchvalues\corerun.exe (?i)Sherlock Holmes 51.02 us 0.389 us 0.90
    Count \literal\corerun.exe aqj 40.96 us 0.643 us 1.00
    Count \searchvalues\corerun.exe aqj 43.32 us 0.847 us 1.06

    I do see that practically all regressions linked against #96402 (including this one) were arm64-based.
    I'll check that as well.

  14. MihaZupan commented on Jul 4, 2024

    @MihaZupan
    MihaZupan
    on Jul 4, 2024
    Member

    Seeing the same kind of thing on ARM.

    Method Toolchain Options Mean Error Ratio
    Count /literal/corerun Compiled 1.224 s 0.0017 s 1.00
    Count /searchvalues/corerun Compiled 1.137 s 0.0002 s 0.93
    Count /literal/corerun IgnoreCase, Compiled 1.994 s 0.0024 s 1.00
    Count /searchvalues/corerun IgnoreCase, Compiled 1.606 s 0.0012 s 0.81
    Method Toolchain Pattern Mean Error Ratio
    Count /literal/corerun (?i)Sherlock 216.2 us 0.39 us 1.00
    Count /searchvalues/corerun (?i)Sherlock 183.8 us 0.33 us 0.85
    Count /literal/corerun (?i)Sherlock Holmes 214.7 us 0.24 us 1.00
    Count /searchvalues/corerun (?i)Sherlock Holmes 180.6 us 0.19 us 0.84
    Count /literal/corerun aqj 116.0 us 2.29 us 1.00
    Count /searchvalues/corerun aqj 121.9 us 1.35 us 1.05

    @stephentoub I think this one is a non-issue. The regressions described in this issue were taken a day before #96429 was merged, so it's possible that improved things.

    Though looking at the perf graphs, I still see the regression 😕
    image

  15. stephentoub commented on Jul 4, 2024

    @stephentoub
    stephentoub
    on Jul 4, 2024
    Member

    It's even more pronounced in this one:
    https://pvscmdupload.z22.web.core.windows.net/reports/allTestHistory/refs/heads/main_arm64_Windows%2010.0.19041/System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%22%2c%20Options%3a%20Compiled).html

  16. MihaZupan commented on Jul 4, 2024

    @MihaZupan
    MihaZupan
    on Jul 4, 2024
    Member

    I'm a bit confused, the above ARM numbers showing improvements are from an Azure Dplsv5 VM (Ampere Altra).
    But the perf lab results (presumably from the same type of Ampere CPUs?) show a regression.

    FWIW this is what I was testing against for literal MihaZupan@8ef2f17#diff-f57fafe69c92d1c6acae34ecb5078ce2993286d9d9d2f7a639a4ab90178d4b67

  17. ericstj
    assigned
    MihaZupan
    on Aug 7, 2024

  18. ericstj commented on Aug 7, 2024

    @ericstj
    ericstj
    on Aug 7, 2024
    Member

    @MihaZupan I assigned this to you for now. Do you think there is something we can do in SearchValues to gain back this perf? Or is this something we need to accept as a tradeoff in using this new type?

  19. MihaZupan commented on Aug 23, 2024

    @MihaZupan
    MihaZupan
    on Aug 23, 2024
    Member

    I'm sure there are tweaks we could make to SearchValues to improve things further here.

    The problem is that I've so far been unable to reproduce this regression -- I see it as an improvement on both X64 and Arm64.

  20. ericstj commented on Sep 16, 2024

    @ericstj
    ericstj
    on Sep 16, 2024
    Member

    @MihaZupan / @stephentoub - do you think this warrants further investigation or a fix for 9.0?

  21. stephentoub commented on Sep 16, 2024

    @stephentoub
    stephentoub
    on Sep 16, 2024
    Member

    do you think this warrants further investigation or a fix for 9.0?

    At this point, given an inability to reproduce, I don't see a change being made here for 9.

  22. ericstj commented on Sep 17, 2024

    @ericstj
    ericstj
    on Sep 17, 2024
    Member

    Ok, I'm going to go ahead and resolve this then.

  23. ericstj
    closed this as not plannedon Sep 17, 2024
  24. github-actions
    locked and limited conversation to collaborators on Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-arm64area-System.Text.RegularExpressionsos-windowsruntime-coreclrspecific to the CoreCLR runtime

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions