Arm64: Add SVE/SVE2 support in .NET 9

Until Armv8, NEON architecture enabled users to write vectorized code using SIMD instructions. The vector length for NEON instructions remains fixed at 128 bits. To take into account High Performance Computing applications, newer versions of Arm like v9 has developed Scalable Vector Extension (SVE). SVE is a new programming model that contains SIMD instructions operating on flexible vector length, ranging from 128 bits to 2048 bits. SVE2 extends SVE to enable domains like Computer vision, multimedia, etc. More details about SVE can be found on Arm's website. In .NET 9, we want to start the foundational work in .NET libraries and RyuJIT to add support for SVE2 feature.

Design: The instructions present in SVE2 needs to be exposed through .NET libraries to our customers. However, we want to make sure that the customers don't have to rewrite their code to consume SVE2 functionality and the experience of using SVE2 features should be seamless. Currently, .NET has Vector<T> that represents flexible vector length depending on the underlying hardware and the idea is to expose SVE2 "flexible vector length" functionality using Vector<T>. We need to think about the common instructions and validate if they can be represented by APIs that just takes Vector<T> as parameter. Additionally, SVE2 introduces "predicate register" to mark vector lanes active/inactive. We do not want to expose this concept as well to our consumer through .NET libraries because of reason mentioned above. Hence, for every API, need to come up with a pseudo code of how the API should be implemented internally in JIT such that the "predicate register" concept is created and consumed inside JIT implicitly. There is a good discussion that happened in the past about the API proposal in [API Proposal]: Example usages of a VectorSVE API #88140.
Implement APIs in System.Runtime.Intrinsic.Arm.SVE2: Once design is finalized, need to add all the APIs in a new SVE2 class under System.Runtime.Intrinsic.Arm. They need to be plumbed through the JIT (by transforming tree nodes to represent "predicate register" concept, if needed). all the way to generating code. Arm64: Implement SVE APIs #99957
If possible, automatically generate the test template for various APIs
Update Antigen to pick up the new SVE APIs
SVE hardware availability in CI
Backend support: Regardless of API design, we need to add instruction encoding of all the SVE2 instructions that we are planning to support. Here is a rough list of things that needs to happen to add the new instructions. Covered in Arm64: Implement SVE encodings #94549.
- Add new entries in hwinstrinsiclistarm64.h for AdvSimd.SVE2 APIs
- Depending on the API call the right emitIns_*() code
- emitIns_*() methods add the new instructions support
- emitfmtsarm64.h - needs to add new instruction formatting Arm64: SVE/SVE2 encodings #94285
- instrsarm64.h - needs to add new instruction to instruction formatting mapping Arm64: SVE/SVE2 encodings #94285
- Add the encoding for new instructions in emitOutputInstr()
- Add new Zx registers and predicate registers LSRA: Add support to track more than 64 registers #99658
- Add JitStressRegs mode to always allocate high Z/P register unless the candidate specifically says to use low registers.
- Make sure TP regression is minimum
- Testing if the encoding matches the display ins using windbg/msvc/cordistools
- Add entries for the new instructions in genArm64EmitterUnitTests()
- Fix the formatEncode* data
- Refactor and consolidate the common code/asserts, convert some methods lookup like insSveIsLslN / insSveGetLslOrModN into table drive loop up.
- Possibly frame layout code update such that the SVE_simd and SVE_mask are towards the end of the frame in order for offset in stack to take into consider the variable VL.
Automation for encoding: If we see above list, it is very time consuming to add each instruction's encoding in the code base. There are 800+ instructions and considering 30 mins for each instruction, it will take 400+ human hours to add and validate the encoding. Hence, there is an experiment going on to generate the encoding data and the C++ code around it automatically. The understanding is that the C++ code generated will not be accurate and manual inspection will still be required.
Edit: A working prototype of the tool that generates 2 C++ files for encoding is in Arm64: SVE/SVE2 encodings #94285.
- Produce a json/xml file that contains all the encoding data of all the instructions. A good resource from which this can be extracted is here.
- Recreate instrsarm64.h that contains the existing as well as the new formats along with encoding binary representation and hexadecimal representation.
- If there are new instruction formats, add their entries in emitfmtsarm64.h file.
- Based on the mmmm, dddd, etc. in the binary representation of encoding, have to tool write a logic to produce the instruction bytes. In other words, this will generate code that can be pasted in emitOutputInstr() function.
- Depending on the number of encodings in each group of instructions, they can be tied to appropriate INST* like INST9 or INST8, etc. and regenerated in sorted order.
  Note that if we need to regenerate existing files like instrsarm64.h and emitfmtsarm64.h , existing instruction's encoding also needs to be generated by the tool.
MOVPRFX instruction support: GCStress, sanity check. Also, when emitting every single instruction, need to make sure that if previous instruction was movprfx, we verify it follows all the rules with regards to destination registers, size, etc. Arm64/Sve: Validation for movprfx in emitter #105514
Before consuming the new APIs in libraries, make sure that Mono has support for it or else SVE.IsSupported should return false for it.
On Windows, make sure that new SVE registers as well as predicate registers are available during debugging as well as Vector<T> shows the right number of elements.
ABI related changes for SVE/predicate registers in VM
Support breakpoint patching for SVE instructions
- ICorDebug needs to be updated to know which SVE instructions have relative read/write/jumps - walker.cpp is specifically the place that needs to be updated
- Similar work has been done for AVX512 - Support breakpoints on AVX-512 instructions #89705

@a74nh

cc: @a74nh @TamarChristinaArm @SwapnilGaikwad

Currently, .NET has Vector that represents flexible vector length depending on the underlying hardware and the idea is to expose SVE2 "flexible vector length" functionality using Vector.

We should think about how this is going to work with the new SVE streaming mode. Do we expect to support the SVE streaming mode in .NET eventually? If yes, how it is going to affect the design?

Do we expect to support the SVE streaming mode in .NET eventually

I'd presume it would be desirable to support. Giving users the power of the per hardware lightup is a significant advantage to many backing frameworks that power .NET applications.

If yes, how it is going to affect the design?

The streaming instructions are more abstract and I don't think the design for C++ has been finalized yet either. I at least don't see any of the new instructions under https://developer.arm.com/architectures/instruction-sets/intrinsics

I expect they will be a completely separate consideration for how they're supported as compared to the more standard SIMD processing instructions and we'll need to cross that bridge when we come to it.

This is particularly given how it allows the effective SVE length to be changed and requires explicit start/stop instructions. And so I imagine the runtime itself will require some complex changes to work with the concept and ensure it doesn't impact the ABI, field usage, inlining boundaries, etc.

The streaming instructions are more abstract and I don't think the design for C++ has been finalized yet either. I at least don't see any of the new instructions under

My understanding that the streaming mode uses the same instructions as regular SVE mode (subset of them), except that the instructions operate on larger vector sizes. If we were to use the current Vector<T> in both streaming and non-streaming mode, the size of Vector<T> would need to change between these two modes somehow. I do not see how we would pull this off.

This observation made me think that reusing Vector<T> for SVE registers may be a poor choice. Would it make more sense to have a new special type that can be only use on stack for locals and arguments and that does not have a constant size at runtime?

Would it make more sense to have a new special type that can be only use on stack for locals and arguments and that does not have a constant size at runtime?

That is effectively how C describes its SVE support. They are similar to "incomplete" types in many aspects, but with a few less restrictions. They can't be used with atomics, arrays, sizeof, pointer arithmetic, fields, captures, etc. They can still be used with ref T, T*, templates, parameters/return types, etc.

At least for non-streaming, a lot of that is only restrictive because C/C++ has existing requirements for things like sizeof(T) to be constant and is namely AOT. There's not actually anything "preventing" most of it from working, it would just require the relevant cnt* instruction to be executed and included as part of the size computation dynamically (and can be constant in a JIT environment).

If we did define a new type, I think we'd functionally be defining a ref struct ScalableVector + writing an analyzer to help enforce it doesn't get used for some other edge cases. I don't think that a platform specific vector type would warrant a proper language feature or language level enforcement for C#/F#/etc.

My understanding that the streaming mode uses the same instructions as regular SVE mode (subset of them)

This is my understanding as well.

For reference, the Procedure Call Standard` is here: https://github.com/ARM-software/abi-aa/releases/download/2023Q1/aapcs64.pdf

While the Vector Function ABI is here: https://github.com/ARM-software/abi-aa/releases/download/2023Q1/vfabia64.pdf

Non-streaming SVE has clear and concrete benefits and can easily integrate with our existing types and concepts. It doesn't really require any significant changes to make light up (no more than AVX-512 did, anyways). The biggest open question for it is how exactly to support it for AOT. But we also have a couple solutions to support both the VLA (vector length agnostic) and VLS (vector length specific) concepts. We can also section that work if required and only support VLS for AOT in the first iteration (much as is already done for Vector<T> today). -- Any work to improve Vector<T> for AOT or R2R scenarios then benefits all platforms and can likely be shared.

Streaming SVE still has some concrete benefits, however, it works significantly differently from anything we've exposed so far. There are simply too many open ended questions around it, how the user will start/stop the functionality, how it will impact the managed/native ABI boundary, inlining, callee/caller save state, etc. Trying to solve all of these would likely push non-streaming SVE support out of .NET 9 and potentially put us in a place where non-streaming SVE is functionally worse off from a general usability perspective. So I think it's ultimately better to, for now, treat it like it doesn't exist and say its UB if a user enables it (much as we define for other similar user togglable CPU state such as: floating-point exceptions, changing the default rounding mode, flushing denormals, enabling strict alignment validation, etc).

Then, when we're ready to take a deeper look at the feature, we can determine whether using Vector<T> + an analyzer is feasible for it or whether we truly need a new type. We can also, at the same time, answer the other questions about how it works at the ABI level for managed code, including GC pause points. -- I imagine, based on the general blog posts and other areas around the topic, we likely don't want to look at it until we have the ability to generate multiple versions of a function (one streaming aware, one streaming unaware), to efficiently handle the various dynamic callee/caller save state support around the ZA storage area, to do relevant state tracking for streaming enablement, etc

Non-streaming SVE has clear and concrete benefits and can easily integrate with our existing types and concepts. It doesn't really require any significant changes to make light up (no more than AVX-512 did, anyways). The biggest open question for it is how exactly to support it for AOT.

Good point about AOT support for non-streaming SVE. How are we going to do that with Vector<T>? I do not see any good options.

How are we going to do that with Vector? I do not see any good options.

It ultimately depends on the code the user writes and how much we want to invest.

For recommended coding patterns, where the user treats Vector<T> and Vector64/128/256/512<T> like a ref struct, then it works much as it does in C/C++ and we can trivially generate VLA (vector length agnostic) code. The ABI already exists to support passing any number of Vector<T> arguments, declaring Vector<T> locals, and other scenarios; so we just need to support it as it is supported in native.

If the user deviates from this and starts using Vector<T> as the field of a struct, declares arrays of it, or other scenarios that depend on the size, then it gets a little more involved. However, we can still generate VLA code by making some computations dynamic. For example, sizeof(Vector<int>) functionally becomes cntd * 4. Since the packing of Vector<T> is well-defined, regardless of size, we can then use this to make offset calculations or other sizeof calculations work just as well and also for cheap.

We also have the option of generation VLS (vector length specific) code. This functionally means we can say a given method only support Vector<T> if it is 128-bits. This is basically how Vector<T> works on x64 when you say it should be 32-bytes, after all. If we ever get to the point of being able to generate multiple versions of a function, we could support generating both 128-bit and 256-bit versions in the future, or however we need based on the sizes we or the user wants to support. -- In the worst case, this simply fails to launch the app if SVE support exists and the size is larger than appropriate, but we also have the option of explicitly using predication if the actual size is larger. In practice, most SVE hardware today remains 128-bit and only a couple exceptions are larger (a 512-bit super computer and a 256-bit implementation for Graviton3). We can likewise guard paths based on some if (Sve.IsSupported and sizeof(Vector<T>) == ...) check if that's desirable. It really just comes down to how much effort we want to spend supporting the usage of Vector<T> in already non-standard scenarios (noting that support can be incremental over time as well).

It ultimately depends on the code the user writes and how much we want to invest.

The design that we choose has a large influence over the cost required to support it.

If we choose a design that mirrors the C/C++ design, the cost of supporting SVE for AOT and for streaming mode is going to be very low.

If we choose the current proposed design that uses existing Vector<T> type, the cost of properly supporting SVE for AOT and for streaming mode seems to be very large, close to prohibitive. We should have a discussion about this and explicitly decide whether we like this long-term outcome.

For recommended coding patterns, where the user treats Vector and Vector64/128/256/512 like a ref struct, then it works much as it does in C/C++ and we can trivially generate VLA (vector length agnostic) code.

It is not unusual to see the Vector types used as regular static and instance fields. The design needs to support all situations where Vector* types can be used today. It is ok if it is slower, but it needs to work. Otherwise, it would be a major breaking change.

If the user deviates from this and starts using Vector as the field of a struct, declares arrays of it, or other scenarios that depend on the size, then it gets a little more involved. However, we can still generate VLA code by making some computations dynamic.

In the limit, this means creating a type loader that computes field layout at runtime and teaching codegen to use dynamic sizes and offsets produced by the type loader. We have done that in .NET Native for UWP, it was very complicated and produced poor results. I do not think we would ever want to do that again for form-factors without a JIT.

In practice, most SVE hardware today remains 128-bit and only a couple exceptions are larger (a 512-bit super computer and a 256-bit implementation for Graviton3).

Yes, SVE is a nascent technology. It is reasonable to expect that there will implementations that take advantage of the full range of allowed lengths, with both streaming and non-streaming modes. We should not take shortcuts based on what is available today.

Nit: byref-like types cannot show up on the GC heap. byref-like type would address this particular concern by construction.

Right, sorry. That's what I meant.

That is, I meant ref struct SveVector<T> fixes the concern around static fields and such fields existing on the GC heap. It doesn't fix the remaining cases such as a user passing ref SveVector<T> or SveVector<T>* or ref struct S { SveVector<T> x; SveVector<T> y; } or parameters or locals leaking across an enablement boundary or ....

We do not know what this set. I expect that BCL is going be streaming incompatible by default. It would be prohibitively expensive to audit the whole BCL and annotate it as streaming compatible/incompatible. We may choose to audit and annotate small part of BCL as streaming compatible and this set can grow over time.

👍. My only real concern is about the eventual scope creep of "sve compatible" and that it may explode to a much larger set of annotations. In practice the only code that should be "incompatible" is if they're using some of the SVE instructions that are incompatible without FEAT_SME_FA64. Especially in the BCL, users shouldn't be having Vector as a static field or in an array, span, etc; because that's significantly less efficient and less portable than operating over a Span<T> directly. Most of the things that would cause you to be SME incompatible are anti-patterns for code that is meant to be used in high perf scenarios, and so its things we already strongly recommend against doing.

I do not see the benefit of unmanaged calling convention compared to an attribute. An analyzer to enforce the streaming compatible rules is the main gatekeeper. An attribute is a more natural way to mark methods that the analyzer should enforce the rules for.

Using just an attribute works, but it doesn't provide a clear boundary for users and will come with additional implications that the VM and JIT need to handle. For example, they may have to special case what can happen with inlining or optimizations across the boundary. They may also require additional tracking to know that we're in a method context that has SVE enabled. If a method is SVE compatible and can be used from both SVE and non-SVE code, then we may need to "specialize" for each to ensure good/efficient codegen. I'm concerned that this additional handling may not be "pay for play" in the JIT.

However, for UnmanagedCallersOnly we already have a lot of this special tracking integrated and it provides a very clear boundary under which power users are already familiar that special limitations may exists. Since enabling streaming mode is in effect dynamically changing the ABI/calling convention, this also makes sense from a logical perspective since that's how users do the same for calling into native.

It's really much the same thing, just trying to play off the existing support and functionality the JIT has.

However, for UnmanagedCallersOnly we already have a lot of this special tracking integrated and it provides a very clear boundary under which power users are already familiar that special limitations may exists

I am not sure what kind of special tracking for UnmanagedCallersOnly you have in mind. The main parts of UnmanagedCallersOnly are:

UnmanagedCallersOnly require all types in the method signatures to be blittable. It would be unnecessary limitation for streaming methods. It is perfectly fine for streaming compatible methods to take object references as argument.
UnmanagedCallersOnly switches GC mode. It would be unnecessary overhead for streaming compatible methods.

The JIT is not able to inline through unmanaged calli + UnmanagedCallersOnly combo today. I do not think we have it written down anywhere that such optimization is prohibited.

For example, they may have to special case what can happen with inlining or optimizations across the boundary. They may also require additional tracking to know that we're in a method context that has SVE enabled. If a method is SVE compatible and can be used from both SVE and non-SVE code, then we may need to "specialize" for each to ensure good/efficient codegen. I'm concerned that this additional handling may not be "pay for play" in the JIT.

There is a large spectrum of how the streaming support can look: from simple explicit approach that enables it and requires users to do more work, all the way to streaming-aware auto-vectorization on the other end.

I think it would be reasonable to start with simple explicit approach. Something like:

Code is streaming-incompatible by default. I believe that this matches the approach taken for C/C++ for streaming. Is that correct?
Methods can be annotated as streaming compatible using a new attribute. Streaming compatible methods have to avoid patterns incompatible with streaming (no access of Vector fields, etc.) and have to explicitly switch to non-streaming mode around any calls to streaming incompatible code. The rules are enforced by Roslyn analyzer. There is no enforcement of the rules by the runtime or JIT.
Switches between streaming and non-streaming modes are inlining boundaries. The JIT does not need to worry about encountering both streaming compatible and streaming incompatible code in the same method.

I think it would be reasonable to start with simple explicit approach

That's what I had in mind in option 3 above. The only reason I introduced Local_Streaming is that will give a choice to user to not having to turn on/off/save/restore streaming state, but JIT will do it for them.

The only reason I introduced Local_Streaming is that will give a choice to user to not having to turn on/off/save/restore streaming state

If we were to do automatically inserted mode switches, I think we would want to do them in both directions: turn on streaming around streaming required method calls and turn off streaming around streaming incompatible method calls. It gets more complicated with optimizations - for example, if there are two calls in a row or in a loop, should the JIT be smart enough to turn the streaming on/off just once for the two calls or around the whole loop? I would wait to see the evidence that the automatic mode switches are really needed. Starting with the fully explicit scheme should not prevent us from introducing automatically inserted mode switches later.

For intrinsics themselves, most SVE instructions are streaming compatible and when FEAT_SME_FA64 is available, it changes to all instructions. My expectation is that we basically want to have Sve and something like Sve.NestedClass. The nested class would contain the instructions which may be streaming incompatible and which require a separate check to use. This then allows an analyzer to help prove correctness.

I don't think this ideal:

It's confusing for a user. They now have to be aware of the implementation of SME to understand why a particular SVE instruction is in a different subclass
Most, but not all NEON instructions are disabled when FEAT_SME_FA64 is enabled. We need to consider what to do here. The remaining ones are mostly scalar instructions, and I don't think we gain much by keeping them. We certainly don't want to adding subclasses.
The instruction disabling is gated on the feature FEAT_SME_FA64. There is nothing in architecture that prevents another extension from enabling or disabling a different set of instructions. Whereas subclassing would be hard to change later.

My suggestion would be to use a method attribute which contains the name of the feature that disables the method. I hope that's something that's not too tricky to retrospectively add to an existing API, as for most use cases it's not changing the API. That would enable us to continue the SVE design, and then retrospectively add the attributes to SVE and AdvSimd.

Most, but not all NEON instructions are disabled when FEAT_SME_FA64 is enabled

Do you have this inversed? Based on the architecture manual it looks like NEON is disabled in streaming mode (much like many SVE instructions) unless FEAT_SME_FA64 is enabled.

The manual states that FEAT_SME_FA64 means the full A64 ISA is available in streaming mode and that would logically include NEON.

They now have to be aware of the implementation of SME to understand why a particular SVE instruction is in a different subclass

Users must be aware when an instruction might be supported or not. Any temporary state changes, like SME, would have to be accounted for as part of context switching to ensure other threads don't suddenly fail.

Using attributes + an analyzer to determine if a given intrinsic API is supported is just a different approach to the same problem, but is inconsistent with how we've handled it so far. It also forces reliance on an analyzer for correctness, rather than a more implicit understanding that if Isa.IsSupported returns true, you can use the APIs and they will otherwise throw.

Based on what the manual covers, we have the following separate checks:

AdvSIMD
- Documented via CheckFPAdvSIMDEnabled64()
  - This checks CheckFPEnabled() and requires !HAVESME() || (PSTATE.SM != 1) || IsFullA64Enabled())
SVE
- Documented via CheckSVEEnabled()
SVE - Non Streaming
- Documented via CheckNonStreamingSVEEnabled()
  - This checks CheckSVEEnabled() and requires !HaveSME() || (PSTATE.SM != 1) || IsFullA64Enabled())
SME
- Documented via CheckSMEEnabled()

Thus, it logically tracks that having 4 classes here would also work, each one exactly corresponding to a documented operation check around the support:

AdvSimd
Sve
SveNonStreaming : Sve
- This could be nested Sve.NonStreaming : AdvSimd
Sme

This ensures we have a very clean and clear separation of support that can be constant in most contexts. The only thing that really changes is in an SME enabled context and based on whether FEAT_SME_FA64 is enabled.

AdvSimd.IsSupported
- returns true by default
- returns false in an SME enabled context, unless FEAT_SME_FA64 is enabled.
Sve
- returns true if FEAT_SVE exists
SveNonStreaming (or Sve.NonStreaming)
- returns the same value as Sve by default
- returns false in an SME enabled context, unless FEAT_SME_FA64 is enabled
Sme
- returns true in an SME enabled context

Going into an SME enabled context requires FEAT_SME and would be dependent on the new attribute approach that Jan laid out above. This attribute is largely only relevant to the JIT and only needs minimal support to ensure that users don't call methods that aren't marked as streaming compatible. The intrinsic APIs would be considered compatible by default and rely on the IsSupported check to determine if it was safe to call or not.

Most, but not all NEON instructions are disabled when FEAT_SME_FA64 is enabled

Do you have this inversed? Based on the architecture manual it looks like NEON is disabled in streaming mode (much like many SVE instructions) unless FEAT_SME_FA64 is enabled.

The manual states that FEAT_SME_FA64 means the full A64 ISA is available in streaming mode and that would logically include NEON.

Yes, you're right I had this inverted.

AdvSimd.IsSupported

returns true by default

returns false in an SME enabled context, unless FEAT_SME_FA64 is enabled.

Sve

In a world without FEAT_SME_FA64, some AdvSimd instructions are still available:

Quoting DDI0616B_a_SME_Supplement.pdf:

For the avoidance of doubt, A64 scalar floating-point instructions which match following encoding patterns remain legal when the PE is in Streaming SVE mode:
A64 Encoding Pattern Instructions or Instruction Class
x001 111x xxxx xxxx xxxx xxxx xxxx xxxx Scalar floating-point operations
xx10 110x xxxx xxxx xxxx xxxx xxxx xxxx Load/store pair of FP registers
xx01 1100 xxxx xxxx xxxx xxxx xxxx xxxx Load FP register (PC-relative literal)
xx11 1100 xx0x xxxx xxxx xxxx xxxx xxxx Load/store FP register (unscaled imm)
xx11 1100 xx1x xxxx xxxx xxxx xxxx xx10 Load/store FP register (register offset)
xx11 1101 xxxx xxxx xxxx xxxx xxxx xxxx Load/store FP register (scaled imm)
With the exception of the following floating-point operation which is illegal when the PE is in Streaming SVE
mode:
A64 Encoding Pattern Instructions or Instruction Class
0001 1110 0111 1110 0000 00xx xxxx xxxx FJCVTZS

You plan on not making these available in C# when in streaming mode?

@tannergooding

I will setup a meeting with @tannergooding @a74nh @TamarChristinaArm to discuss this.

Thanks for working on this. Very excited about future SVE2 support!

[ ] SVE hardware availability in CI

First step might be having GitHub hosted arm64 runners and images at all:

While useful for many developers, it might help if a request came from an internal Microsoft team.

As for SVE and SVE2, there a CPU with Neoverse V1 (SVE) or Neoverse N2, V2, N3 or V3 cores is needed. Since Ampere doesn't have processors released with these cores, the Microsoft Azure Cobalt 100 CPU looks like the best options, which uses N2 cores.

It seems all pieces of the puzzle are already in-house at Microsoft!

Closing as completed for .NET 9 effort. We will create a new user story on Arm64 performance for .NET 10 and move some of the remaining items there.

Navigation Menu

Repository navigation

Arm64: Add SVE/SVE2 support in .NET 9 #93095

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Arm64: Add SVE/SVE2 support in .NET 9 #93095

Description

Activity

kunalspathak commented on Oct 6, 2023

jkotas commented on Oct 13, 2023

tannergooding commented on Oct 13, 2023

jkotas commented on Oct 14, 2023

tannergooding commented on Oct 14, 2023

jkotas commented on Oct 14, 2023

tannergooding commented on Oct 14, 2023

jkotas commented on Oct 16, 2023

25 remaining items

tannergooding commented on Feb 16, 2024

jkotas commented on Feb 20, 2024

kunalspathak commented on Feb 20, 2024

jkotas commented on Feb 20, 2024

a74nh commented on Mar 1, 2024

tannergooding commented on Mar 1, 2024

a74nh commented on Mar 6, 2024

kunalspathak commented on Mar 7, 2024

EwoutH commented on Apr 4, 2024

JulieLeeMSFT commented on Aug 14, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions