JIT: APX NDD register arithmetic immediately kills the result's byref GC liveness #135428

Description

When byref arithmetic is emitted in APX new-data-destination (NDD) register-register form, emitter::emitOutputRRR marks the destination as a live byref and immediately marks it dead at the same instruction offset. A fully interruptible method can consequently have a live interior pointer missing from its GC register state until the next instruction-group reconciliation.

The bookkeeping defect was verified at 313f777aa09607a97fe43e4a2cc17f10696569f2 and the same code is still present on upstream main at 7fd4be4b29175b1007cb5a1bd701f057b2fe08be.

Reproduction Steps

At the emitter level, use an APX NDD descriptor with a byref result and three distinct registers:

instruction:        INS_sub
format:             IF_RWR_RRD_RRD
GC type:            GCT_BYREF (EA_BYREF)
destination:        r10
source 1:           rcx, holding a byref
source 2:           rdx, holding a native-integer offset
EVEX.ND:            set
initial byref set:  {rcx}

Execute the post-instruction GC bookkeeping from emitOutputRRR, then inspect emitRegGCtype(REG_R10). Repeat with INS_add for the same emitter-level failure.

A managed route to the subtraction is Unsafe.SubtractByteOffset: its importer produces a byref GT_SUB, and binary codegen can select NDD when the destination differs from both source registers and the feature is enabled. The exact managed register allocation was not reproduced.

Validation used a source-extracted C++ probe containing the actual GC live/dead helper bodies, emitHandleGCrefRegs, and the output methods' GC-update tails. This exercised emitter register-state bookkeeping only, not instruction encoding or APX execution. No full CoreCLR build, managed APX run, or GC-stress crash reproduction was performed.

Expected behavior

The destination remains classified as GCT_BYREF after the NDD arithmetic instruction, so the live interior pointer can be reported and relocated by GC.

Actual behavior

For both register-register INS_add and INS_sub, the probe observed emitRegGCtype(REG_R10) == GCT_NONE after the GC-update tail. Equivalent typed NDD register/memory descriptors through emitOutputAM and emitOutputSV retained GCT_BYREF.

Regression?

The introducing commit and affected released versions have not been established. This is pre-existing in the revision reviewed for dotnet/runtime#135415, not introduced by that PR.

Known Workarounds

Avoid APX NDD codegen: DOTNET_EnableApxNDD=0, which is already the default in the referenced source.

Configuration

• Architecture: x64, with APX NDD selected (EnableApxNDD=1 and the required APX encoding support).
• The bookkeeping probe was compiled and run on Windows/x64 with MSVC; APX hardware instructions were not executed.
• This does not require an extended GPR: the failing descriptor used r10.
• No comparison against released .NET versions was performed.

Other information

The two GC-update branches in emitxarch.cpp:16873-16894 are independent:

if (id->idGCref())
{
    emitHandleGCrefRegs(dst, id);
}

if (!emitInsCanOnlyWriteSSE2OrAVXReg(id))

For IF_RWR_RRD_RRD byref add/sub, the first branch marks idReg1 live as a byref. The second branch also runs, since the instruction writes a GPR, and calls emitGCregDeadUpd(id->idReg1(), dst).

The destination-killing branch should be mutually exclusive with the typed GC-result handling, as in the other output paths. Changing the second if to else if preserves the typed result while retaining GC deaths for non-GC GPR writes.

This is separate from the already-tracked AVX10.2 classifier omission in dotnet/runtime#135387.