[fbusato]

Description

#10768 remembered me that there is still low-hanging fruit to optimize WarpReduce.

The PR optimizes bitwise operators for unsigned types > 4B by applying warp_redux to 32-bit chunks. This leads to 4x better perf for 64-bit, and 25x for 128-bit integers.

I also realized that warpspeed is using the internal warp_redux function for a smaller set of types. The original concern was related to the large number of SASS instructions. Instead, switching to the top-level API reduced the number of SASS instructions. The simplification decreases the code fragmentation and potentially the suboptimal paths (@bernhardmgruber for visibility)

[coderabbitai]

📝 Summary

Summary by CodeRabbit

• New Features

  • Added warp-level bitwise reduction support for unsigned integral types wider than 32 bits, including optional 128-bit support.
  • Extended bitwise AND, OR, and XOR reductions across full and partial warps.

• Performance

  • Improved reduction handling across supported GPU architectures by using optimized warp-level reduction paths where available.

• Documentation

  • Added details on hardware-accelerated reductions, supported operators, data types, and architecture requirements.

• Tests

  • Added coverage validating wide-type bitwise reduction results.

Walkthrough

The change adds wide unsigned bitwise reduction support, removes the specialized reduction path from incremental lookahead, documents redux.sync coverage, and adds full and partial reduction tests.

Changes

Warp reduction support

Layer / File(s)	Summary
Wide unsigned bitwise reductions cub/cub/warp/specializations/warp_redux.cuh	Wide unsigned bitwise values are reduced by 32-bit chunks and recombined. Dispatch now selects this path for supported SM80 targets.
Incremental lookahead reduction cub/cub/detail/warpspeed/look_ahead.cuh	warpIncrementalLookahead removes lane-mask state and uses WarpReduce::Reduce for all architectures.
Reduction documentation and validation cub/cub/warp/warp_reduce.cuh, cub/test/warp/catch2_test_warp_reduce.cu	Documentation describes redux.sync support. Tests cover AND, OR, and XOR reductions for unsigned 32-bit, 64-bit, and optional 128-bit types with full and partial inputs.

Suggested reviewers: bernhardmgruber, miscco

Priority: ➖ Normal

Merge Risk: 🟡 Moderate · up to 71378

Partial lookahead batches can produce incorrect aggregates because inactive lanes may contribute uninitialized values before the reduction count is applied. This should be corrected before merge.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 85043dca-9919-4409-9ad9-6a68784ab2b5

📥 Commits

Reviewing files that changed from the base of the PR and between 81aff35 and 71378a9.

📒 Files selected for processing (4)

• cub/cub/detail/warpspeed/look_ahead.cuh
• cub/cub/warp/specializations/warp_redux.cuh
• cub/cub/warp/warp_reduce.cuh
• cub/test/warp/catch2_test_warp_reduce.cu

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

[bernhardmgruber]

We definitely need to see a benchmark of scan on B200 :) I am looking forward to the results!

[fbusato]

Here the summary on a B200, CTK 13.3. There are a couple of cases marked as "slow" but the corresponding Rel. Diff is < 2% and/or the types are not handled by the new code. The only case that has ~2% regression is I8. I will better investigate but could be just noise.

scan.exclusive.sum.lookahead.base

T{ct}	OffsetT{ct}	Elements{io}[pow2]	NoiseRef	MeanRef	NoiseCmp	MeanCmp	Abs. Diff	Rel. Diff	Status
I8	I64	32	0.035285	0.00317111	0.0388331	0.00322586	5.47518e-05	1.72658	🔴 SLOW
I16	I64	32	1.72506	0.00333105	2.07739	0.00331912	-1.19314e-05	-0.358188	🔵 SAME
I32	I64	32	1.54546	0.00510561	1.42902	0.00509621	-9.39611e-06	-0.184035	🔵 SAME
I64	I64	32	1.12445	0.0104627	0.957159	0.0104235	-3.92115e-05	-0.374773	🔵 SAME
I128	I64	32	0.0291948	0.0327232	0.0291965	0.0327211	-2.10106e-06	-0.00642072	🔵 SAME
F32	I64	32	0.260766	0.00485172	0.225715	0.00484841	-3.30852e-06	-0.0681928	🔵 SAME
F64	I64	32	0.762037	0.00991512	0.675908	0.00988734	-2.7772e-05	-0.280098	🔵 SAME
C32	I64	32	0.19341	0.00971669	0.0649842	0.0097101	-6.59376e-06	-0.0678602	🟢 FAST
F16	I64	32	0.03662	0.00334708	0.039661	0.00334687	-2.11643e-07	-0.00632321	🔵 SAME
BF16	I64	32	0.039301	0.00334709	0.0350525	0.00334668	-4.15603e-07	-0.0124168	🔵 SAME

scan.exclusive.sum.base

T{ct}	OffsetT{ct}	Elements{io}[pow2]	NoiseRef	MeanRef	NoiseCmp	MeanCmp	Abs. Diff	Rel. Diff	Status
I8	I64	32	0.0374169	0.0031712	5.93536	0.00324038	6.91803e-05	2.18152	🔴 SLOW
I16	I64	32	1.85317	0.00331815	6.03947	0.00330203	-1.61203e-05	-0.48582	🔵 SAME
I32	I64	32	1.59254	0.00507702	5.79273	0.0050801	3.0864e-06	0.0607917	🔵 SAME
I64	I64	32	1.3126	0.0104258	1.10251	0.0104	-2.58163e-05	-0.247619	🔵 SAME
I128	I64	32	0.029543	0.0327222	0.029831	0.0327212	-9.46224e-07	-0.00289169	🔵 SAME
F32	I64	32	0.245974	0.00484963	0.254201	0.00484997	3.44124e-07	0.00709588	🔵 SAME
F64	I64	32	0.656429	0.00990968	7.27849	0.00995382	4.41335e-05	0.445357	🔵 SAME
C32	I64	32	0.179522	0.00971663	4.01514	0.00972785	1.1215e-05	0.11542	🔵 SAME
F16	I64	32	0.0350774	0.00334702	3.75317	0.00335244	5.42239e-06	0.162007	🔴 SLOW
BF16	I64	32	0.0338799	0.00334675	0.0363017	0.00334658	-1.76951e-07	-0.00528725	🔵 SAME

scan.exclusive.deterministic.base

T{ct}	OffsetT{ct}	Elements{io}[pow2]	NoiseRef	MeanRef	NoiseCmp	MeanCmp	Abs. Diff	Rel. Diff	Status
F32	I64	28	0.487419	0.000321455	0.482355	0.000321486	3.15486e-08	0.00981431	🔵 SAME
F64	I64	28	0.181291	0.00062187	0.179954	0.000621916	4.57512e-08	0.00735704	🔵 SAME

scan.exclusive.custom.base

T{ct}	OffsetT{ct}	Elements{io}[pow2]	NoiseRef	MeanRef	NoiseCmp	MeanCmp	Abs. Diff	Rel. Diff	Status
I8	I64	32	0.033273	0.00344246	0.0324831	0.00344312	6.61472e-07	0.0192151	🔵 SAME
I16	I64	32	0.224582	0.00470878	0.210997	0.00470886	7.82311e-08	0.00166139	🔵 SAME
I32	I64	32	0.416463	0.00491309	0.382851	0.00491041	-2.68593e-06	-0.0546689	🔵 SAME
I64	I64	32	0.81222	0.0105877	0.762628	0.0105814	-6.33858e-06	-0.0598672	🔵 SAME
I128	I64	32	0.017892	0.0346841	0.0227721	0.0346818	-2.32458e-06	-0.00670216	🔵 SAME
F32	I64	32	1.16157	0.00494419	1.10567	0.00494269	-1.4999e-06	-0.0303365	🔵 SAME
F64	I64	32	0.613186	0.0108556	5.13503	0.0108878	3.21893e-05	0.296523	🔵 SAME
C32	I64	32	0.426078	0.410992	2.33511	0.420953	0.00996119	2.42369	🔴 SLOW
F16	I64	32	0.0309065	0.00419285	4.91248	0.0042085	1.56523e-05	0.373309	🔴 SLOW
BF16	I64	32	0.035324	0.00420325	5.17336	0.00421814	1.489e-05	0.354249	🔴 SLOW

[davebayer]

LGTM

[bernhardmgruber]

@fbusato please always post the full benchmark diff. We need to see the results for all problem sizes in the benchmark. You can use --threshold-diff 0.02 to make it a bit shorter.

[fbusato]

@bernhardmgruber I don't want to spend further time on this. I will revert Scan code to the old bloated one.

[github-actions]

🥳 CI Workflow Results

🟩 Finished in 3h 00m: Pass: 100%/176 | Total: 8d 09h | Max: 2h 59m | Hits: 30%/601924

See results here.