Implement all ARM NEON intrinsics

Steps for implementing an intrinsic:

Select an intrinsic below
Review coresimd/arm/neon.rs and coresimd/aarch64/neon.rs
Consult ARM official documentation about your intrinsic
Consult godbolt for how the intrinsic should be codegen'd, using clang as an example. Use the links below and replace the name of the intrinsic in the code with your intrinsic. Note that if ARM is an error then your intrinsic may be AArch64-only
- ARM
- AArch64
If the codegen is the same on ARM/AArch64, place the intrinsic in coresimd/arm/neon.rs. If it's different place it in both with appropriate #[cfg] in coresimd/arm/neon.rs. If it's only AArch64 place it in coresimd/aarch64/neon.rs
Write a test for your intrinsic at the bottom of the file as well
Test! Probably use rustup run nightly sh ci/run-docker.sh aarch64-unknown-linux-gnu.
When ready, send a PR!

Is there a blocker for these, or is it just finding time to do it? I'd like to help, but I'd need a more experienced compiler/SIMD person to point me in the right direction.

I can mentor. Start by taking a look at some of the intrinsics in the coresimd/aarch64/neon.rs module :)

Is there some upstream source that these all get copied from, or are they actually written by hand?

@Amanieu

I am not sure I understand the question ? The neon modules in this repository are written by hand, although @Amanieu has expressed interest into generating some parts of them automatically.

Sorry yeah that was unclear. The other thing I wanted to ask was, what's the upstream Source Of Truth for defining these functions?

…

On Fri, Nov 16, 2018, 11:57 AM gnzlbg ***@***.*** wrote: I am not sure I understand the question ? The neon modules in this repository are written by hand, although @Amanieu <https://github.com/Amanieu> has expressed interest into generating some parts of them automatically. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#148 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0jBJQbXX6E4XFv28wWNuDfvUYpY6mDks5uvu5ugaJpZM4QEQB6> .

Ah, I see, that would be the ARM NEON spec: https://developer.arm.com/technologies/neon/intrinsics

@gnzlbg

Now might be a great time to help make some more progress on this! We've got tons of intrinsics already implemented (thanks @gnzlbg!), and I've just implemented automatic verification of all added intrinsics, so we know if they're added they've got the correct signature at least!

I've updated the OP of this issue with more detailed instructions about how to bind NEON intrinsics. Hopefully it's not too bad any more!

We'll probably want to reorganize modules so they're a bit smaller and more manageable over time, but for now if anyone's interested to add more intrinsics and needs some help let me know!

more manageable

I have a proposal for this: using a macro to make definitions one-line e.g.:

neon_op!(binary vadd_s8 : int8x8_t == simd_add, assert vadd / add, doc "Vector add");
neon_op!(binary vaddl_s8 : int8x8_t -> int16x8_t == simd_add, assert vaddl / saddl, doc "Vector long add");
neon_op!(unary vmovn_s16 : int16x8_t -> int8x8_t == simd_cast, assert vmovn / xtn, doc "Vector narrow integer");

This will make adding new ones easier (scrolling through a bolierplate-filled file just feels awful), and I'll add a lot more simd_sub simd_mul simd_lt etc. ones. Would this be accepted?

macro definition I currently have

macro_rules! neon_op {
    (binary $name:ident : $type:ident == $op:ident, assert $instr32:ident / $instr64:ident, doc $doc:literal) => {
        #[inline]
        #[doc = $doc]
        #[target_feature(enable = "neon")]
        #[cfg_attr(target_arch = "arm", target_feature(enable = "v7"))]
        #[cfg_attr(all(test, target_arch = "arm"), assert_instr($instr32))]
        #[cfg_attr(all(test, target_arch = "aarch64"), assert_instr($instr64))]
        pub unsafe fn $name(a: $type, b: $type) -> $type {
            $op(a, b)
        }
    };
    (binary $name:ident : $type:ident -> $result_type:ident == $op:ident, assert $instr32:ident / $instr64:ident, doc $doc:literal) => {
        #[inline]
        #[doc = $doc]
        #[target_feature(enable = "neon")]
        #[cfg_attr(target_arch = "arm", target_feature(enable = "v7"))]
        #[cfg_attr(all(test, target_arch = "arm"), assert_instr($instr32))]
        #[cfg_attr(all(test, target_arch = "aarch64"), assert_instr($instr64))]
        pub unsafe fn $name(a: $type, b: $type) -> $result_type {
            let a: $result_type = simd_cast(a);
            let b: $result_type = simd_cast(b);
            $op(a, b)
        }
    };
    (unary $name:ident : $type:ident -> $result_type:ident == $op:ident, assert $instr32:ident / $instr64:ident, doc $doc:literal) => {
        #[inline]
        #[doc = $doc]
        #[target_feature(enable = "neon")]
        #[cfg_attr(target_arch = "arm", target_feature(enable = "v7"))]
        #[cfg_attr(all(test, target_arch = "arm"), assert_instr($instr32))]
        #[cfg_attr(all(test, target_arch = "aarch64"), assert_instr($instr64))]
        pub unsafe fn $name(a: $type) -> $result_type {
            $op(a)
        }
    };
}

For the definitions, I think that using macros is ok.

I am not sure I follow how does macros generate run-time tests for the intrinsics, that's usually the bulk of the work.

What is the reasoning behind some intrinsics linking in the LLVM intrinsic directly while others are using the generic simd_XXX functions?

For example:

stdarch/crates/core_arch/src/arm/neon/generated.rs

Lines 1416 to 1430 in a371069

    
           /// Halving add 
        
           #[inline] 
        
           #[target_feature(enable = "neon")] 
        
           #[cfg_attr(target_arch = "arm", target_feature(enable = "v7"))] 
        
           #[cfg_attr(all(test, target_arch = "arm"), assert_instr("vhadd.u16"))] 
        
           #[cfg_attr(all(test, target_arch = "aarch64"), assert_instr(uhadd))] 
        
           pub unsafe fn vhadd_u16(a: uint16x4_t, b: uint16x4_t) -> uint16x4_t { 
        
               #[allow(improper_ctypes)] 
        
               extern "C" { 
        
                   #[cfg_attr(target_arch = "arm", link_name = "llvm.arm.neon.vhaddu.v4i16")] 
        
                   #[cfg_attr(target_arch = "aarch64", link_name = "llvm.aarch64.neon.uhadd.v4i16")] 
        
                   fn vhadd_u16_(a: uint16x4_t, b: uint16x4_t) -> uint16x4_t; 
        
               } 
        
           vhadd_u16_(a, b) 
        
           }

Versus:

stdarch/crates/core_arch/src/aarch64/neon/generated.rs

Lines 12 to 18 in a371069

    
           /// Compare bitwise Equal (vector) 
        
           #[inline] 
        
           #[target_feature(enable = "neon")] 
        
           #[cfg_attr(test, assert_instr(cmeq))] 
        
           pub unsafe fn vceq_u64(a: uint64x1_t, b: uint64x1_t) -> uint64x1_t { 
        
               simd_eq(a, b) 
        
           }

Given the sheer volume of neon intrinsics, it seems rather daunting to implement them all by hand using the guide in the first post. I'm wondering if there's a deterministic data driven way to generate all of them using #[link_name = "llvm.*"] as done in the first example. Maybe the llvm c headers could be useful?

What is the reasoning behind some intrinsics linking in the LLVM intrinsic directly while others are using the generic simd_XXX functions?

Not all intrinsics have a corresponding simd_* platform-intrinsic.

I'm wondering if there's a deterministic data driven way to generate all of them using #[link_name = "llvm.*"] as done in the first example. Maybe the llvm c headers could be useful?

Please don't. The simd_* platform intrinsics are much easier to implement in alternative codegen backends than the llvm intrinsics, as they are generic over vector types and they are backend independent.

@aloucks

@aloucks most of the intrinsics (AFAIK) have been added piecemeal over time, so it's sort of expected that they're not 100% consistent. Otherwise though I'd imagine that whatever works best would be fine to add to this repository. Auto-generation sounds pretty reasonable to me, and for an implementation we strive to match what Clang does in its implementation of these intrinsics.

Also, to be clear, this library is not designed for ease of implementation in alternate codegen backends. The purpose of this crate is to get the LLVM backend up and running with SIMD. Discussions and design constraints for alternate backends should be discussed in a separate issue.

@Lokathor

@Lokathor Several instructions have been added recently: vaddhn, vbic, vorn, vceqz, vtst, vabd, vaba. Though some of them are not fully supported( like vceqzd). If you don’t have time to maintain this google sheet, I think I can help

Awesome, looking forward to this!

Any updates after a long time...? Thanks

If you look at the pull request list you can see that there has been activity on this quite recently. For example #1224 was opened yesterday.

@bjorn3

@bjorn3 Thanks! Indeed I mostly want to know when can we see it in stable version.
By the way do you suggest use nightly in production environment? If so I can use it now.

@SparrowLii

@SparrowLii You marked the following instructions as completed (same for min):

It doesn't seem like those instructions are actually part of your recent PR (nor were they on the master branch before that) so I unmarked them again.

@CryZe

@CryZe They can be found in the master branch now:
https://github.com/rust-lang/stdarch/blob/master/crates/core_arch/src/aarch64/neon/generated.rs#L8519-L8539
https://github.com/rust-lang/stdarch/blob/master/crates/core_arch/src/aarch64/neon/generated.rs#L8545-L8565
Sorry I marked them before #1230 merged, this is to prevent others from submitting duplicate PRs

Welp, I'll mark them again then. Somehow the GitHub Pull Request UI doesn't show them as diffs at all: https://i.imgur.com/BsHR5in.gif

Github’s comparison tool will always have problems when changing a large amount of code XD

As in #1230, except for the following instructions and those use 16-bit floating-point, other instructions have been implemented:

The following instructions are only available in aarch64 now, because the corresponding target_feature cannot be found in the available features of arm:
vcadd_rot、vcmla、vdot
The feature i8mm is not valid:
vmmla、vusmmla: https://rust.godbolt.org/z/8GbKW5ef4
LLVM ERROR(Can be reproduced in godbolt):
vsm4e: https://rust.godbolt.org/z/xhT1xvGTP
LLVM ERROR(Normal in gotbolt, but LLVM ERROR: Cannot select: intrinsic raises at runtime)
vsudot、vusdot: https://rust.godbolt.org/z/aMnEvab3n
vqshlu: https://rust.godbolt.org/z/hvGhrhdMT
Not implmented in LLVM and cannot be implemented manually:
vmull_p64（for arm）、vsm3、vrax1q_u64、vxarq_u64、vrnd32、vrnd64、vsha512

As in #1230, except for the following instructions and those use 16-bit floating-point, other instructions have been implemented:
1. The following instructions are only available in aarch64 now, because the corresponding `target_feature` cannot be found in the available features of arm:
   `vcadd_rot`、`vcmla`、`vdot`

On LLVM's ARM backend, vcadd_rot and vcmla are under the v8.3a feature. vdot is under the dotprod feature. I got this information from llvm-project/llvm/lib/Target/ARM/ARMInstrNEON.td.

2. The feature `i8mm` is not valid:
   `vmmla`、`vusmmla`: [rust.godbolt.org/z/8GbKW5ef4](https://rust.godbolt.org/z/8GbKW5ef4)

Already discussed in rust-lang/rust#90079.

3. LLVM ERROR(Can be reproduced in godbolt):
   `vsm4e`: [rust.godbolt.org/z/xhT1xvGTP](https://rust.godbolt.org/z/xhT1xvGTP)

Use llvm.aarch64.crypto.sm4ekey instead of llvm.aarch64.sve.sm4ekey.

4. LLVM ERROR(Normal in gotbolt, but `LLVM ERROR: Cannot select: intrinsic` raises at runtime)
   `vsudot`、`vusdot`: [rust.godbolt.org/z/aMnEvab3n](https://rust.godbolt.org/z/aMnEvab3n)
   `vqshlu`: [rust.godbolt.org/z/hvGhrhdMT](https://rust.godbolt.org/z/hvGhrhdMT)

You need to make you test function pub in godbolt, otherwise it will be optimized away as unreachable by rustc before LLVM.

vsudot/vusdot require the i8mm target feature. vqshlu seems to work fine in godbolt after changing the pub.

5. Not implmented in LLVM and cannot be implemented manually:
   `vmull_p64`（for arm）、`vsm3`、`vrax1q_u64`、`vxarq_u64`、`vrnd32`、`vrnd64`、`vsha512`

These all seem to exist in LLVM at least for AArch64. For ARM we can just leave these out for now.

Hope someone can help implement the remaining instructions.

@Amanieu

@Amanieu v8.5a feature is non-runtime detected so we can't use #[simd_test(enable = "neon,v8.5a")]. So how do we add tests for instructions that use v8.5a, like vrnd32x and vrnd64x?

@SparrowLii

@SparrowLii Shouldn't that work with the frintts feature?

@SparrowLii

@SparrowLii Shouldn't that work with the frintts feature?

Looks useful: https://rust.godbolt.org/z/894W8cndG

LLVM only supports frintts on AArch64, so it's fine to not support this intrinsic on ARM.

Navigation Menu

Repository navigation

Implement all ARM NEON intrinsics #148

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	/// Halving add
	#[inline]
	#[target_feature(enable = "neon")]
	#[cfg_attr(target_arch = "arm", target_feature(enable = "v7"))]
	#[cfg_attr(all(test, target_arch = "arm"), assert_instr("vhadd.u16"))]
	#[cfg_attr(all(test, target_arch = "aarch64"), assert_instr(uhadd))]
	pub unsafe fn vhadd_u16(a: uint16x4_t, b: uint16x4_t) -> uint16x4_t {
	#[allow(improper_ctypes)]
	extern "C" {
	#[cfg_attr(target_arch = "arm", link_name = "llvm.arm.neon.vhaddu.v4i16")]
	#[cfg_attr(target_arch = "aarch64", link_name = "llvm.aarch64.neon.uhadd.v4i16")]
	fn vhadd_u16_(a: uint16x4_t, b: uint16x4_t) -> uint16x4_t;
	}
	vhadd_u16_(a, b)
	}

	/// Compare bitwise Equal (vector)
	#[inline]
	#[target_feature(enable = "neon")]
	#[cfg_attr(test, assert_instr(cmeq))]
	pub unsafe fn vceq_u64(a: uint64x1_t, b: uint64x1_t) -> uint64x1_t {
	simd_eq(a, b)
	}

Implement all ARM NEON intrinsics #148

Description

Activity

gnzlbg commented on Oct 24, 2017

oconnor663 commented on Nov 15, 2018

gnzlbg commented on Nov 15, 2018

oconnor663 commented on Nov 16, 2018

gnzlbg commented on Nov 16, 2018

oconnor663 commented on Nov 16, 2018

gnzlbg commented on Nov 16, 2018

alexcrichton commented on Dec 20, 2018

valpackett commented on Jul 28, 2019

gnzlbg commented on Jul 28, 2019

aloucks commented on Jul 8, 2020

bjorn3 commented on Jul 8, 2020

alexcrichton commented on Jul 8, 2020

alexcrichton commented on Jul 8, 2020

3 remaining items

nano-bot commented on Mar 9, 2021

fzyzcjy commented on Sep 27, 2021

bjorn3 commented on Sep 27, 2021

fzyzcjy commented on Sep 27, 2021

CryZe commented on Oct 21, 2021

SparrowLii commented on Oct 21, 2021

CryZe commented on Oct 21, 2021

SparrowLii commented on Oct 21, 2021

SparrowLii commented on Oct 21, 2021

Amanieu commented on Oct 21, 2021

SparrowLii commented on Oct 25, 2021

SparrowLii commented on Nov 9, 2021

hkratz commented on Nov 9, 2021

SparrowLii commented on Nov 9, 2021

Amanieu commented on Nov 9, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions