[rmlarsen]

Results, compared with main

This PR force-inlines the fpmp2 arithmetic and comparison operators, the polynomial evaluators and the public math functions. Results do not change: it only stops host compilers from calling those functions out of line, inside the fp32mp2 math functions or in their callers. It is based on main and lands first: #12519, #12530 and #12548 are stacked on it.

Performance. Speedup (old time / new time), as the geometric mean over 30 fp32mp2 math functions, with each function called through one non-inlined call ("out of line") or inlined into the benchmark loop ("inlined"):

Platform	Over	Out of line	Inlined
aarch64: DGX Spark Cortex-X925, GCC 13.3 -O3 -mcpu=neoverse-v2	main	2.251	1.492
x86-64: AMD Ryzen Threadripper PRO 7965WX (Zen 4), GCC 15.2 -O3 -march=native	main	1.207	1.030
aarch64: Raspberry Pi 5 (Cortex-A76), GCC 14 -O3 -mcpu=cortex-a76	#12530	1.092	1.078
x86-64: Intel Core i5-11500T, GCC 15.2 -O3 -march=x86-64-v3	#12530	1.044	1.027
aarch64: Apple M4 (Mac mini), Apple clang 21 -O3 -mcpu=apple-m4	#12530	1.071	1.060
GPU: sm_75 to sm_121 (RTX 2070, H100, B200, B300, Rubin, DGX Spark GB10)	main	1.00 (identical SASS)	1.00

• The rows. The DGX Spark and Zen 4 rows measure this PR on main, with [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530's second harness. The Pi, i5-11500T and M4 rows were measured earlier, when this PR sat on top of [fpmp] Faster, more accurate fpmp2 arithmetic with single-limb scalars and a shorter pair product #12519 and [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530 and did not yet force-inline the public math functions; "inlined" there is [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530's own harness.

• Largest gains. Out of line on the Cortex-X925, where GCC 13 called the operators out of line in most math functions: atanh 3.8×, sin, cos and cospi 3.7×, sinh 3.4×, expm1 3.3× and pow 3.1×. On Zen 4: tanh, sinh and expm1 1.55×, pow 1.53×. On the Pi (over [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530), pow 1.76×, atanh 1.65×, acosh 1.47× and tanh 1.41×.

• Out of line, no function is slower than 0.98× on any of these platforms.

• Why it lands first. Without it, [fpmp] Faster, more accurate fpmp2 arithmetic with single-limb scalars and a shorter pair product #12519 and [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530 make some functions slower on aarch64 with GCC, which calls their new operators out of line: tanh ran at 0.73× of main on the Pi, and at 0.64× on the Cortex-X925 (out of line), with sinh and expm1 at 0.79–0.85×. With this PR underneath, no function in [fpmp] Faster, more accurate fpmp2 arithmetic with single-limb scalars and a shorter pair product #12519 or [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530 is slower than 0.97× out of line, on the Cortex-X925 or on Zen 4.

• Loops that call the math functions. Force-inlining the operators alone made the math functions look larger to GCC, which then stopped inlining exp, log, log2, atan and cosh into the loops that call them, where main inlines and vectorizes them: y[i] = cuda::log(x[i]) over an array took 22 ns per element instead of 2.2 on Zen 4. The public math functions and their cuda::std forwarders are therefore force-inlined too, and such loops are now as fast as on main or faster, within 3% (atan on Zen 4):

  ns per element, y[i] = f(x[i]), 4096 values	main	This PR	Whole stack ([fpmp] Accurate, C99-conforming and faster fp32mp2 pow #12548)
  Zen 4, GCC 15.2: exp	1.81	1.84	1.73
  Zen 4: log	2.19	2.23	2.08
  Zen 4: log2	25.4	2.33	2.12
  Zen 4: atan	3.75	3.62	3.85
  Zen 4: cosh	2.52	2.50	2.44
  Cortex-X925, GCC 13.3: exp	5.87	5.87	5.04
  Cortex-X925: log	17.8	6.01	5.31
  Cortex-X925: log2	19.7	7.37	6.48
  Cortex-X925: cosh	7.87	7.76	7.28

  sin, tanh, expm1, atanh and pow are faster than on main in the same loops, on both CPUs.

• Still slower: a math function inside the caller's own non-inline helper, as in the "inlined" mode of [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530's second harness. GCC sizes the helper by the fully expanded function and may not inline it into the loop, which then is not vectorized across its chains: atan and cosh run at 0.23–0.24× of main there on Zen 4 (and log at 0.26× with [fpmp] Faster, more accurate fpmp2 arithmetic with single-limb scalars and a shorter pair product #12519 on top), and log at 0.38× on the Cortex-X925. Force-inlining the helper, or calling the function in the loop directly, avoids it. With [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530's harness, the Pi showed sin and cosh at 0.85× and 0.84× of [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530 for the same reason, and the i5-11500T sin, cospi and cosh at 0.88–0.93×.

Accuracy. Unchanged.

• The device SASS is identical to main's (sm_89 and sm_90; measured on top of [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530 also on sm_100, sm_103, sm_107, sm_120 and sm_121), so device results are bit-identical.
• On the hosts the values are the same, and host/device agreement is unchanged for every function.

Special values. Unchanged. Only the sign of a few NaN results changes on the host, which IEEE 754 leaves unspecified, measured on top of #12530:

• boys_f0(NaN) on x86-64 (GCC 15.2, with and without -ffp-contract=off);
• cos(±inf), cos(NaN), erfc(NaN) and boys_f0(inf) on the Pi (GCC 14).

There, inlining lets GCC contract differently.

Implementation

The fpmp2 arithmetic and comparison operators, add/sub/mul/div, the __fpmp2_*_scalar helpers and the three polynomial evaluators do nothing but dispatch to kernels that are already force-inlined (_CCCL_FPMP_CORE_API). They were plain inline functions, though, and GCC 14 for aarch64 calls several of them out of line in the fp32mp2 math functions:

• the low-accuracy pair product and quotient;
• the pair-by-float product;
• three __fpmp_poly_horner_mixed instances.

This PR declares those functions in fpmp.h (39 on main, 44 once #12519 adds its scalar operators), and the evaluators in fpmp_math_impl.h, _CCCL_HOST_DEVICE_API _CCCL_FORCEINLINE, as libcudacxx does in cuda/__stream/invalid_stream.h. Unlike _CCCL_TRIVIAL_HOST_DEVICE_API, this does not mark them nodebug or artificial, so a debugger can still step into them; the generated code is the same. The constructors, the conversion and assignment operators, the volatile members and the named functions of fpmp.h (sqrt, fma, ...) are unchanged. CUB already force-inlines friend operators the same way (cub/detail/null_type.cuh).

It also declares the 90 public math functions of fpmp_math.h and their 13 cuda::std forwarders _CCCL_HOST_DEVICE_API _CCCL_FORCEINLINE. Their implementations were already force-inlined (_CCCL_FPMP_CORE_API), so these wrappers were the only level left to the compiler. The named functions of fpmp.h (sqrt, fma, ...) stay as they are: GCC inlines them into such loops anyway.

Measurements

The Pi, i5-11500T and M4 figures come from #12530's harness, measured on top of #12530: each function runs on one core, and the figure is the fastest of at least 30 rounds, with a 3 s warm-up on the M4. The DGX Spark and Zen 4 figures come from #12530's second harness on main: the five variants of the stack interleaved, the median of 15 rounds, best of three passes, on one core.

Per-function speedups over #12530: out-of-line call / plain

Function	Pi 5	i5-11500T	Apple M4
exp	1.00 / 1.01	1.00 / 0.99	1.00 / 1.01
exp2	1.00 / 1.00	1.00 / 1.00	1.00 / 1.01
exp10	1.00 / 1.00	1.00 / 1.00	1.00 / 1.00
expm1	1.31 / 1.02	1.11 / 1.00	1.09 / 1.10
log	1.00 / 1.00	1.00 / 1.01	1.00 / 1.00
log2	1.00 / 1.00	1.00 / 1.01	1.00 / 1.00
log10	1.00 / 1.00	1.00 / 0.97	1.00 / 1.00
log1p	1.27 / 1.02	1.09 / 0.99	1.18 / 1.07
sin	1.01 / 0.85	1.01 / 0.93	1.07 / 1.01
cos	1.00 / 1.00	1.00 / 1.00	1.05 / 1.00
tan	1.00 / 1.00	1.00 / 1.00	1.00 / 1.01
sinpi	1.00 / 1.00	1.00 / 1.00	0.98 / 1.00
cospi	1.00 / 0.99	1.00 / 0.93	1.00 / 1.06
asin	0.99 / 1.00	1.00 / 1.00	1.11 / 1.14
acos	1.00 / 1.00	1.00 / 1.00	1.14 / 1.15
atan	1.00 / 1.00	1.00 / 1.00	1.19 / 1.17
erf	1.01 / 1.00	0.99 / 1.01	0.99 / 0.99
erfc	1.00 / 1.00	1.00 / 1.00	1.24 / 1.25
normcdfinv	1.00 / 1.00	1.00 / 1.00	1.36 / 1.30
boys_f0	1.02 / 1.02	1.00 / 1.00	1.04 / 1.07
tanh	1.41 / 1.41	1.16 / 1.00	1.08 / 1.05
sinh	1.00 / 1.00	1.01 / 1.01	1.08 / 1.04
cosh	0.99 / 0.84	1.01 / 0.88	1.00 / 1.02
atanh	1.65 / 1.83	1.33 / 1.34	1.17 / 1.17
asinh	1.37 / 1.53	1.15 / 1.20	1.10 / 1.16
acosh	1.47 / 1.59	1.22 / 1.28	1.12 / 1.11
cbrt	1.00 / 1.00	1.00 / 0.98	1.01 / 1.00
rcbrt	1.00 / 1.05	1.02 / 1.02	1.01 / 0.93
pow	1.76 / 1.90	1.34 / 1.36	1.24 / 1.07
sincos	1.00 / 1.00	1.00 / 1.01	0.99 / 1.00
geomean	1.092 / 1.078	1.044 / 1.027	1.071 / 1.060

The M4's plain rcbrt (0.93×) comes from one fast round of #12530: 15.1 ns, against 16.2 ns in its other three rounds and in all four of this PR's.

Per-function speedups of this PR over main on the DGX Spark's Cortex-X925 and on Zen 4: out of line / inlined

Function	DGX Spark Cortex-X925	Zen 4
exp	2.15 / 0.94	1.00 / 0.94
exp2	1.45 / 1.00	1.00 / 1.00
exp10	1.86 / 1.00	1.09 / 1.00
expm1	3.34 / 1.00	1.55 / 1.00
log	1.69 / 0.38	1.00 / 1.00
log2	1.84 / 0.93	1.04 / 1.00
log10	1.84 / 0.93	1.04 / 1.00
log1p	2.44 / 0.99	1.46 / 1.00
sin	3.74 / 3.70	1.21 / 1.21
cos	3.74 / 3.70	1.21 / 1.21
tan	2.63 / 2.61	1.16 / 1.16
sincos	2.60 / 2.61	1.16 / 1.15
sinpi	3.13 / 3.10	1.34 / 1.34
cospi	3.72 / 3.68	1.40 / 1.41
asin	1.06 / 1.06	1.05 / 1.03
acos	1.07 / 1.07	1.05 / 1.03
atan	1.00 / 1.00	1.00 / 0.23
erf	1.89 / 1.85	1.03 / 1.01
erfc	2.78 / 2.78	1.26 / 1.26
normcdfinv	1.87 / 1.87	1.22 / 1.22
boys_f0	1.90 / 1.94	1.22 / 1.22
tanh	3.01 / 0.99	1.55 / 1.54
sinh	3.41 / 0.98	1.55 / 1.55
cosh	3.01 / 1.00	1.31 / 0.24
atanh	3.78 / 3.75	1.49 / 1.47
asinh	2.37 / 2.40	1.39 / 1.39
acosh	2.88 / 2.88	1.39 / 1.40
cbrt	1.46 / 1.00	1.00 / 1.00
rcbrt	1.62 / 0.96	1.00 / 0.95
pow	3.08 / 1.00	1.53 / 0.99
geomean	2.251 / 1.492	1.207 / 1.030

Testing

• Device code. The SASS of the harness's kernels is identical to main's on sm_89 and sm_90, because nvcc already inlined all of these functions. On top of [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530, it was also identical on sm_100, sm_103, sm_107, sm_120 and sm_121 (CUDA 13.4).
• Bit-exact comparisons. On top of [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530, the 31 fp32mp2 math functions × 2¹⁸ inputs were bit-identical on sm_89 and value-identical on the x86-64 and Pi hosts, as described under Special values. Force-inlining the public functions changes the bits of four special-input results on the hosts, not their values: one erfc result on Zen 4 (GCC 15.2), three cos and one normcdfinv result on the Cortex-X925 (GCC 13.3).
• lit on the GPU. The full cuda/fp/units suite passes on sm_89 on main with this PR (53 passed; fpemu/float64 and fpmp/atomic_dd unsupported), with CUDA 13.4 and GCC 15.2. On top of [fpmp] Faster fp32mp2 math functions; accurate trig near multiples of π/2 and erfc at large x #12530 it passed on sm_89 (56 passed) and on sm_120, where only fpemu/charbool.pass.cpp fails, as on main.

Merge order

Based on main; it should merge first. #12519, #12530 and #12548 are stacked on it and merge its branch.

The CI override of ab86603 (the cuda/fp/units lit suite on a T4 and an H100, MSVC and arm64 precompiles) is reset by 627abd3, so this PR now runs the full CI.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[miscco]

Looks fine by me, note that this also prevents the debugger to step into those functions, which might not be what you want.

In that case you really want to just add _CCCL_FORCEINLINE

[coderabbitai]

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

• Refactor
  • Updated the implementation of multi-precision floating-point arithmetic and math operations, including comparisons, increment and decrement, and math functions used in host and device execution.
  • Supported operations, calculations, accuracy selection, and results remain unchanged. This update does not add or remove user-facing capabilities.

The fpmp2 operators, arithmetic overloads, math wrappers, cuda::std forwarding overloads, and polynomial helpers now use _CCCL_FORCEINLINE. Their signatures, calculations, dispatch, and forwarding behavior remain unchanged.

Changes: fpmp2 annotations

Layer / File(s)	Summary
Operators and arithmetic overloads libcudacxx/include/cuda/__fp/fpmp.h	fpmp2 operators and accuracy-explicit add, sub, mul, and div overloads use _CCCL_FORCEINLINE. Their logic and delegation remain unchanged.
Math APIs and polynomial helpers libcudacxx/include/cuda/__fp/fpmp_math.h, libcudacxx/include/cuda/__fp/fpmp_math_impl.h	fpmp2 math APIs, cuda::std forwarding wrappers, and polynomial helpers use _CCCL_FORCEINLINE. Their calls and helper bodies remain unchanged.

Possibly related PRs

• NVIDIA/cccl#12548: It changes the same operator and mixed-scalar overload declarations in fpmp.h and is stated to depend on this force-inline change landing first.
• NVIDIA/cccl#12530: It builds on the force-inlined declarations and math paths changed here, and the supplied summary states a direct stack relationship.

Suggested reviewers: akolesov-nvidia

Change: Refactor

Merge Risk: Minimal · up to 91455

This change force-inlines fpmp2 operators and math functions to speed up host code. No actionable merge-blocking risk was found.

Architecture risk: Low · up to 91455

The change affects 1 system.

Changed systems: libcudacxx

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

• observed — libcudacxx (service) was modified; 3 changed files map to changed impact.

Before / after behavior

• observed — Modified behavior in libcudacxx/include/cuda/__fp/fpmp_math_impl.h: Updates __fpmp_poly_horner_mixed to use _CCCL_HOST_DEVICE_API _CCCL_FORCEINLINE instead of _CCCL_HOST_DEVICE_API inline; the function interface and body stay the same.
• observed — Modified behavior in libcudacxx/include/cuda/__fp/fpmp_math_impl.h: Updates __fpmp_poly_horner_comp to use _CCCL_HOST_DEVICE_API _CCCL_FORCEINLINE instead of _CCCL_HOST_DEVICE_API inline; the function interface and body stay the same.
• observed — Modified behavior in libcudacxx/include/cuda/__fp/fpmp_math_impl.h: Updates __fpmp_poly_eval to use _CCCL_HOST_DEVICE_API _CCCL_FORCEINLINE instead of _CCCL_HOST_DEVICE_API inline; the function interface and body stay the same.
• observed — Modified behavior in libcudacxx/include/cuda/__fp/fpmp_math.h: The fpmp2 wrappers for exponential, logarithmic, power, trigonometric, hyperbolic, error, Boys, and inverse-normal functions replace inline with _CCCL_FORCEINLINE; signatures and calls to the corresponding __fpmp2_* functions are unchanged.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cccl
• Branch: main
• File: .coderabbit.yaml
• Commit: d0d9d68
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 9b0e1acc-c1a5-4a7e-9f93-a71374bc0e68

Commits

Reviewing files that changed from the base of the PR and between 3057237 and 914553f.

Files selected for processing: 1

• libcudacxx/include/cuda/__fp/fpmp.h

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

[coderabbitai]

Additional comments: 1

libcudacxx/include/cuda/__fp/fpmp.h: 1

1008-1030: Trivial · Maintainability & Code Quality · Low value

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

suggestion: The retrieved learning says to flag inlining hints that lack profiling data. The PR reports host benchmarks, so the learning is satisfied for the arithmetic operators. Add a short comment near the first _CCCL_TRIVIAL_HOST_DEVICE_API use on the operators. The comment should say why force-inlining is needed (host compilers emit out-of-line calls). This prevents a later cleanup from reverting the annotation. Increment, decrement and compound-assignment operators now also force-inline. They add no new logic, only a call to an already force-inlined operator. If code size or compile time matters, drop _CCCL_TRIVIAL_HOST_DEVICE_API from these wrappers and keep it on the core operators.

Also applies to: 1032-1047

Source: Learnings

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cccl
• Branch: main
• File: .coderabbit.yaml
• Commit: 26e9c14
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 75f6fa8a-3c41-49db-8643-438e0807864e

Commits

Reviewing files that changed from the base of the PR and between 26e9c14 and 627abd3.

Files selected for processing: 3

• libcudacxx/include/cuda/__fp/fpmp.h
• libcudacxx/include/cuda/__fp/fpmp_math.h
• libcudacxx/include/cuda/__fp/fpmp_math_impl.h

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

[rmlarsen]

@miscco Thanks, that was not intended. 2fad1ca declares everything this PR force-inlines _CCCL_HOST_DEVICE_API _CCCL_FORCEINLINE instead, as libcudacxx does in cuda/__stream/invalid_stream.h. The operators, the polynomial evaluators and the math functions stay force-inlined, but a debugger can step into them. The generated host code (GCC 15.2) and the sm_89 SASS are unchanged. #12519, which adds more scalar operators on top of this PR, does the same for them in b8b4f8b.

[rmlarsen]

@coderabbitai I added the comment in 3057237: in fpmp.h above the operators, and in fpmp_math.h above the math functions. The increment, decrement and compound-assignment operators stay force-inlined. They are one-line wrappers, so it costs nothing, and leaving them to GCC would bring back the out-of-line calls this PR removes.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

Analysis chain

Script executed:

#!/bin/bash
set -u
for name in fpmp.h fpmp_math.h; do
  echo "=== Locate $name ==="
  fd -t f -F "$name" .
done
for file in libcudacxx/include/cuda/__fp/fpmp.h libcudacxx/include/cuda/__fp/fpmp_math.h; do
  echo "=== Outline $file ==="
  ast-grep outline "$file" --items all --match 'operator|fpmp' 2>/dev/null | head -100 || true
  echo "=== Force-inline and rationale context: $file ==="
  rg -n -C 4 'force.inline|force.inline|out.of.line|_CCCL_FORCEINLINE|operator\+\+|operator--|operator\+=' "$file" | head -250 || true
done
echo "=== Commit changes ==="
git show --format=fuller --no-ext-diff 3057237ff9 -- libcudacxx/include/cuda/__fp/fpmp.h libcudacxx/include/cuda/__fp/fpmp_math.h | head -300 || true
echo "=== Relevant review conversations ==="
gh api graphql -f query='
query {
  repository(owner: "NVIDIA", name: "cccl") {
    pullRequest(number: 12543) {
      reviewThreads(first: 100) {
        nodes {
          path
          isResolved
          comments(first: 30) { nodes { author { login } body url } }
        }
        pageInfo { hasNextPage }
      }
    }
  }
}' --jq '.data.repository.pullRequest.reviewThreads | {hasNextPage: .pageInfo.hasNextPage, matchingThreads: [.nodes[] | select(any(.comments.nodes[]; (.body | ascii_downcase | test("3057237ff9|increment|compound.assignment|force.inline"))))]}' || true

Length of output: 28762

---

@rmlarsen suggestion: The comments are present in both headers. I accept the rationale for keeping the increment, decrement, and compound-assignment operators force-inlined. No further change is needed.

_{You are interacting with an AI system.}

[miscco]

/ok to test 3057237

[rmlarsen]

/ok to test 914553f

[miscco]

/ok to test 914553f

[github-actions]

🥳 CI Workflow Results

🟩 Finished in 1h 28m: Pass: 100%/230 | Total: 2d 01h | Max: 1h 25m | Hits: 93%/696198

See results here.