Skip to content

[ARM64] Performance regression: PerfLabTests.CastingPerf2.CastingPerf.IntObj #41706

Closed
Closed
[ARM64] Performance regression: PerfLabTests.CastingPerf2.CastingPerf.IntObj#41706
Assignees
CarolEidt
Labels
arch-arm64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issue
Milestone
6.0.0

Description

@adamsitnik
adamsitnik
opened on Sep 1, 2020

After running benchmarks for 3.1 vs 5.0 using "Ubuntu arm64 Qualcomm Machines" owned by the JIT Team, I've found out that PerfLabTests.CastingPerf2.CastingPerf.IntObj has regressed x2.

It looks like these are ARM64 specific regressions, I was not able to reproduce it for ARM (the 32-bit variant).

Repro

git clone https://github.com/dotnet/performance.git
py ./performance/scripts/benchmarks_ci.py -f netcoreapp3.1 netcoreapp5.0 --architecture arm64 --filter 'PerfLabTests.CastingPerf2.CastingPerf.IntObj'
BenchmarkDotNet=v0.12.1.1405-nightly, OS=ubuntu 16.04
Unknown processor
  [Host]     : .NET Core 3.1.8 (CoreCLR 4.700.20.41105, CoreFX 4.700.20.41903), Arm64 RyuJIT
  Job-PVNQZA : .NET Core 3.1.8 (CoreCLR 4.700.20.41105, CoreFX 4.700.20.41903), Arm64 RyuJIT
  Job-PXIHWO : .NET Core 5.0.0 (CoreCLR 5.0.20.41714, CoreFX 5.0.20.41714), Arm64 RyuJIT
Method Toolchain Mean Ratio
IntObj netcoreapp3.1 466.4 us 1.00
IntObj netcoreapp5.0 1,001.1 us 2.15

cc @kunalspathak

category:cq
theme:needs-triage
skill-level:expert
cost:large

Activity

  1. adamsitnik
    added
    arch-arm64
    tenet-performancePerformance related issue
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    on Sep 1, 2020
  2. Dotnet-GitSync-Bot
    added
    untriagedNew issue has not been triaged by the area owner
    on Sep 1, 2020
  3. JulieLeeMSFT
    assigned
    CarolEidt
    on Sep 1, 2020

  4. JulieLeeMSFT commented on Sep 1, 2020

    @JulieLeeMSFT
    JulieLeeMSFT
    on Sep 1, 2020
    Member

    @CarolEidt please help look into this.

    CC @dotnet/jit-contrib

  5. JulieLeeMSFT
    removed
    untriagedNew issue has not been triaged by the area owner
    on Sep 1, 2020
  6. JulieLeeMSFT
    added this to the 5.0.0 milestone on Sep 1, 2020

  7. CarolEidt commented on Sep 2, 2020

    @CarolEidt
    CarolEidt
    on Sep 2, 2020
    Contributor

    @adamsitnik - could you clarify how to run these on one of the Ubuntu arm64 systems? The command you give above appears to be using windows pathnames, and if I try this:

    python scripts/benchmarks_ci.py -f netcoreapp3.1 netcoreapp5.0 --architecture arm64 --filter 'PerfLabTests.CastingPerf2.CastingPerf.IntObj'

    I get a syntax error on line 39

  8. AndyAyersMS commented on Sep 2, 2020

    @AndyAyersMS
    AndyAyersMS
    on Sep 2, 2020
    Member

    Assuming you have the right builds of 3.1 and 5.0 installed and dotnet is on your path, and you have cloned the perf repo (to say ~/repos/performance, then

    cd ~/repos/performance/src/benchmarks/micro
dotnet run -c Release -f net5.0 -- -f netcoreapp3.1 netcoreapp5.0 --architecture arm64 --filter 'PerfLabTests.CastingPerf2.CastingPerf.IntObj'

    should work. Add -d to get the BDN produced disassembly.

  9. CarolEidt commented on Sep 2, 2020

    @CarolEidt
    CarolEidt
    on Sep 2, 2020
    Contributor

    Thanks @AndyAyersMS !

  10. AndyAyersMS commented on Sep 2, 2020

    @AndyAyersMS
    AndyAyersMS
    on Sep 2, 2020
    Member

    Also this may be touching on the relatively new cast caching, so cc @VSadov.

  11. VSadov commented on Sep 2, 2020

    @VSadov
    VSadov
    on Sep 2, 2020
    Member

    If this is a simple unbox, it should be jit-inlined.
    Even if not inlined it should not hit cast cache, since unbox requires exact type match*, but it could be calling a managed helper now and there is a small penalty due to tiering/R2R indirection. It is not a lot, but may be noticeable on super fast casts.

    I will take a look.

    *enums match with underlying types as well and thus have some special handling, but that still does not need to use cache.

  12. adamsitnik commented on Sep 2, 2020

    @adamsitnik
    adamsitnik
    on Sep 2, 2020
    MemberAuthor

    could you clarify how to run these on one of the Ubuntu arm64 systems? The command you give above appears to be using windows pathname

    please excuse me for that, I must have copy-pasted it from previous Windows specifc issue. I've fixed the description

    I get a syntax error on line 39

    If you append 3 to python it should use python 3.x and work:

    python3 scripts/benchmarks_ci.py -f netcoreapp3.1 netcoreapp5.0 --architecture arm64 --filter 'PerfLabTests.CastingPerf2.CastingPerf.IntObj'

    Add -d to get the BDN produced disassembly.

    Unfortunately, the BDN disassembler does not support ARM. Some time ago we have switched to use Iced library and it does not support ARM yet (https://github.com/0xd4d/iced/issues/79, https://github.com/0xd4d/iced/issues/80)

  13. CarolEidt commented on Sep 2, 2020

    @CarolEidt
    CarolEidt
    on Sep 2, 2020
    Contributor

    Looking at the code generation, the only difference is that in 5.0 several address constants have been CSE'd. The loop goes from 108 bytes down to 64 bytes, but in both cases is small enough that alignment could make a big difference. Unless there's objection I would suggest we close this, and reference #8108 once again. (Perhaps we should add support for a specified loop alignment, even if only for use in benchmarking to validate or invalidate hypotheses such as this.)

    BEFORE:

G_M590_IG03:        ; offs=000028H, size=003CH, gcrefRegs=0000 {}, byrefRegs=0000 {}, byref, isz

IN0008: 000028      movz    x1, #0xd1ffab1e
IN0009: 00002C      movk    x1, #0xd1ffab1e LSL #16
IN000a: 000030      movk    x1, #0xd1ffab1e LSL #32
IN000b: 000034      ldr     x20, [x1]
IN000c: 000038      ldr     x1, [x20]
IN000d: 00003C      movz    x0, #0xd1ffab1e
IN000e: 000040      movk    x0, #0xd1ffab1e LSL #16
IN000f: 000044      movk    x0, #0xd1ffab1e LSL #32
IN0010: 000048      cmp     x1, x0
IN0011: 00004C      beq     G_M590_IG04
IN0012: 000050      mov     x1, x20
IN0013: 000054      movz    x0, #0xd1ffab1e
IN0014: 000058      movk    x0, #0xd1ffab1e LSL #16
IN0015: 00005C      movk    x0, #0xd1ffab1e LSL #32
IN0016: 000060      bl      CORINFO_HELP_UNBOX

G_M590_IG04:        ; offs=000064H, size=0030H, gcrefRegs=100000 {x20}, byrefRegs=0000 {}, byref, isz

IN0017: 000064      ldr     w0, [x20,#8]
IN0018: 000068      movz    x1, #0xd1ffab1e
IN0019: 00006C      movk    x1, #0xd1ffab1e LSL #16
IN001a: 000070      movk    x1, #0xd1ffab1e LSL #32
IN001b: 000074      str     w0, [x1]
IN001c: 000078      add     w19, w19, #1
IN001d: 00007C      movz    x0, #0xd1ffab1e
IN001e: 000080      movk    x0, #0xd1ffab1e LSL #16
IN001f: 000084      movk    x0, #0xd1ffab1e LSL #32
IN0020: 000088      ldr     w0, [x0]
IN0021: 00008C      cmp     w19, w0
IN0022: 000090      blt     G_M590_IG03

AFTER:

G_M50398_IG03:        ; offs=000038H, size=001CH, bbWeight=4    PerfScore 36.00, gcrefRegs=0000 {}, byrefRegs=0000 {}, byref, isz

IN000b: 000038      movz    x1, #0xd1ffab1e
IN000c: 00003C      movk    x1, #0xd1ffab1e LSL #16
IN000d: 000040      movk    x1, #0xd1ffab1e LSL #32
IN000e: 000044      ldr     x22, [x1]
IN000f: 000048      ldr     x1, [x22]
IN0010: 00004C      cmp     x1, x21
IN0011: 000050      beq     G_M50398_IG05

G_M50398_IG04:        ; offs=000054H, size=000CH, bbWeight=1    PerfScore 2.00, gcrefRegs=400000 {x22}, byrefRegs=0000 {}, byref

IN0012: 000054      mov     x1, x22
IN0013: 000058      mov     x0, x21
IN0014: 00005C      bl      CORINFO_HELP_UNBOX

G_M50398_IG05:        ; offs=000060H, size=0018H, bbWeight=4    PerfScore 36.00, gcrefRegs=400000 {x22}, byrefRegs=0000 {}, byref, isz

IN0015: 000060      ldr     w0, [x22,#8]
IN0016: 000064      str     w0, [x20,#8]
IN0017: 000068      add     w19, w19, #1
IN0018: 00006C      ldr     w0, [x20]
IN0019: 000070      cmp     w19, w0
IN001a: 000074      blt     G_M50398_IG03

  14. 15 remaining items

  15. JulieLeeMSFT
    modified the milestones: 5.0.0, 6.0.0 on Sep 4, 2020

  16. CarolEidt commented on Sep 4, 2020

    @CarolEidt
    CarolEidt
    on Sep 4, 2020
    Contributor

    @AndyAyersMS - it would be interesting to try implementing #2249 for arm64.
    We've marked this 6.0.0, though, as it seems that modifying alignment would require significant perf analysis to ensure that we understand the impact.

  17. CarolEidt commented on Sep 4, 2020

    @CarolEidt
    CarolEidt
    on Sep 4, 2020
    Contributor

    Here's the output from perf stat -e "branch-misses,cache-misses,cpu-cycles,instructions,stalled-cycles-frontend,stalled-cycles-backend" (the first one is 5.0 and the second is 3.1):

    Performance counter stats for '/home/robox/cteidt/performance/tools/dotnet/arm64/dotnet run --project /home/robox/cteidt/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework netcoreapp5.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf.IntObj --packages /home/robox/cteidt/performance/artifacts/packages --runtimes netcoreapp5.0 --cli /home/robox/cteidt/performance/tools/dotnet/arm64/dotnet':

       927,236,914      branch-misses
       435,682,900      cache-misses
    90,726,403,850      cpu-cycles
    59,721,463,264      instructions              #    0.66  insn per cycle
                                                  #    0.67  stalled cycles per insn
    40,144,534,076      stalled-cycles-frontend   #   44.25% frontend cycles idle
    26,028,533,828      stalled-cycles-backend    #   28.69% backend cycles idle

      36.046207216 seconds time elapsed

Performance counter stats for '/home/robox/cteidt/performance/tools/dotnet/arm64/dotnet run --project /home/robox/cteidt/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework netcoreapp5.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf.IntObj --packages /home/robox/cteidt/performance/artifacts/packages --runtimes netcoreapp3.1 --cli /home/robox/cteidt/performance/tools/dotnet/arm64/dotnet':

       746,009,625      branch-misses
       347,766,489      cache-misses
    76,174,119,876      cpu-cycles
    84,227,660,637      instructions              #    1.11  insn per cycle
                                                  #    0.38  stalled cycles per insn
    32,418,821,468      stalled-cycles-frontend   #   42.56% frontend cycles idle
    14,048,337,909      stalled-cycles-backend    #   18.44% backend cycles idle

  18. TamarChristinaArm commented on Sep 7, 2020

    @TamarChristinaArm
    TamarChristinaArm
    on Sep 7, 2020
    Contributor

    @TamarChristinaArm is there a recommended set of NOP sequences of varying length, or is there some other practice to ensure alignment?

    No, we just emit multiple NOPs. For the larger alignment constraints like 32 we only align it if it means adding less than 16 bytes of padding. For the smaller ones we generally always align it.

  19. BruceForstall
    added
    JitUntriagedCLR JIT issues needing additional triage
    on Oct 28, 2020
  20. BruceForstall
    removed
    JitUntriagedCLR JIT issues needing additional triage
    on Nov 10, 2020

  21. BruceForstall commented on Nov 10, 2020

    @BruceForstall
    BruceForstall
    on Nov 10, 2020
    Contributor

    It appears the consensus here is the issue is loop alignment, for which we already have linked issues tracking the work, so I'm going to close this. If that's incorrect, then feel free to re-open with a clear note about what unique work/issue this will address.

  22. BruceForstall
    closed this as completedon Nov 10, 2020

  23. kunalspathak commented on Nov 10, 2020

    @kunalspathak
    kunalspathak
    on Nov 10, 2020
    Contributor

    On my Windows x64, when I tested this benchmark with my changes in #44370, I did see this benchmark improved.

    Faster base/diff Base Median (ns) Diff Median (ns) Modality
    PerfLabTests.CastingPerf.IntObj 1.04 368774.36 355509.29

  24. adamsitnik commented on Nov 12, 2020

    @adamsitnik
    adamsitnik
    on Nov 12, 2020
    MemberAuthor

    On my Windows x64

    @kunalspathak This particular regression was specific to ARM64 (not x64). Is there any chance you could check the ARM results?

  25. kunalspathak commented on Nov 12, 2020

    @kunalspathak
    kunalspathak
    on Nov 12, 2020
    Contributor

    On my Windows x64

    @kunalspathak This particular regression was specific to ARM64 (not x64). Is there any chance you could check the ARM results?

    Ah, that's true. Currently my loop alignment is for xarch, but once I get it for arm, I will verify this.

  26. ghost locked as resolved and limited conversation to collaborators on Dec 12, 2020
  27. JulieLeeMSFT
    added this to .NET Core CodeGenon Jun 5, 2024
  28. JulieLeeMSFT
    moved this to Done in .NET Core CodeGenon Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-arm64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issue

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions