sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280 by philip-jingxin · Pull Request #28016 · ggml-org/llama.cpp

philip-jingxin · 2026-08-30T10:33:58Z

Overview

Extends FWHT with Kronecker product to support non-power-of-two common embedding dimensions 384, 640, 768, 1280

Additional information

Changes

Implemented kronecker_kernel to handle base blocks of $m=12$ and $m=20$ using the Paley I matrix construction.
Extended operator tests in tests/test-backend-ops.cpp to cover non-power-of-two sizes for Hadamard transformation.

Testing & Verification

Environment

Device: Intel(R) UHD Graphics 770
Driver Version: 26.05.037020
Command: ./build_hadamard/bin/test-backend-ops test -o MUL_MAT_HADAMARD -b SYCL0

Correctness Test

Passed 16/16 verification tests, including non-power-of-two sizes (m= 384, 640, 768, 1280) across various configurations:

Backend 1/2: SYCL0
  Device description: Intel(R) UHD Graphics 770
  MUL_MAT_HADAMARD(m=384,  n=1,  k=384):  OK
  MUL_MAT_HADAMARD(m=384,  n=32, k=384):  OK
  MUL_MAT_HADAMARD(m=384,  n=4,  k=384):  OK
  MUL_MAT_HADAMARD(m=640,  n=1,  k=640):  OK
  MUL_MAT_HADAMARD(m=640,  n=32, k=640):  OK
  MUL_MAT_HADAMARD(m=768,  n=1,  k=768):  OK
  MUL_MAT_HADAMARD(m=1280, n=1,  k=1280): OK
  16/16 tests passed

Performance Benchmarks

Measured on Intel UHD Graphics 770:

Size	Batch	Latency (us/run)	Throughput
384	1	4.20 us	70.19 GFLOPS
384	32	5.33 us	1.77 TFLOPS
640	1	16.74 us	48.95 GFLOPS
640	32	19.57 us	1.34 TFLOPS
768	1	17.23 us	68.47 GFLOPS
1280	1	72.43 us	45.24 GFLOPS

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES - Gemini was used to debug and ensure the correctness of the implementation.

philip-jingxin

This pr extends the FWHT to support a wider range of common embeddings. If you guys find this useful, I'll extends this to more generalized dimensions. Thanks @ggerganov

arthw

It's good job!
The UT cases are passed on B60 and iGPU in Ultra 7 265K.

Thank you!

Titaniumtown · 2026-09-01T15:47:46Z

@philip-jingxin Thank you! Very reasonable changes.

Titaniumtown · 2026-09-01T15:55:54Z

I too hastilly merged this PR, I believe as it breaks github actions. @philip-jingxin can you create a follow-up PR fixing the build issues?

philip-jingxin · 2026-09-02T16:16:15Z

I too hastilly merged this PR, I believe as it breaks github actions. @philip-jingxin can you create a follow-up PR fixing the build issues?

@Titaniumtown My bad! I've created a follow-up PR fixing the -Werror,-Wunused-variable warning and reintroducing the feature. Here you go: #28254

Thanks!

…T support (#28016) (#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184) This reverts commit c845263. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only

…80 (ggml-org#28016)

…T support (ggml-org#28016) (ggml-org#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (ggml-org#28184) This reverts commit c845263. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only

…80 (ggml-org#28016)

…T support (ggml-org#28016) (ggml-org#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (ggml-org#28184) This reverts commit 79dc68c. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only

…gml-org#28254, ggml-org#29243) (#302) * sycl : fix test-backend-ops CI break && restore Kronecker product FWHT support (ggml-org#28016) (ggml-org#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (ggml-org#28184) This reverts commit c845263. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only (cherry picked from commit 4d91760) * sycl: FWHT kernels for block widths above 512 (ggml-org#29243) The SYCL FWHT covers 64 to 512 via the standard butterfly network, plus 384/640/768/1280 via the Kronecker/Paley construction added separately in Hadamard hint can produce (1024, 2048, 4096, 8192); those still fall through to the default case and run as a dense GEMM against the materialized rotation tensor, correct but O(n^2) instead of O(n log n). fwht_kernel_wide runs one row per work-group instead of per sub-group, so each work-item keeps N/NT values rather than N/WARP_SIZE. Butterflies below the sub-group width still shuffle; those up to the work-group width go through work-group local memory; the rest stay in registers. Same butterfly and sign convention as the existing narrow kernel. ggml's SYCL backend registration (dpct::dev_mgr) unconditionally requires a GPU-labeled platform to exist and throws before any op-level test can run, so test-backend-ops could not be exercised on this box (a GPU-less pod) even via the CPU device. Verified instead with a standalone harness: the same kernel body run through a real SYCL CPU device (Intel oneAPI DPC++ 2026.1, OpenCL CPU backend), checked against an independent recursive-doubling Hadamard reference, cross-validated by first running the existing unmodified narrow kernel through the identical harness and confirming it passes (rules out a reference-convention bug before trusting a pass on the new code). Random-input results for all four widths, single- and multi-row: N=1024 NT=256 rows=1 max_abs_err=1.7e-07 max_rel_err=4.9e-04 PASS N=2048 NT=256 rows=1 max_abs_err=1.9e-07 max_rel_err=2.0e-04 PASS N=4096 NT=256 rows=1 max_abs_err=2.0e-07 max_rel_err=1.4e-04 PASS N=8192 NT=256 rows=1 max_abs_err=2.5e-07 max_rel_err=3.8e-03 PASS N=1024 NT=256 rows=7 max_abs_err=2.4e-07 max_rel_err=1.0e-03 PASS N=2048 NT=256 rows=5 max_abs_err=3.0e-07 max_rel_err=9.4e-04 PASS N=4096 NT=256 rows=3 max_abs_err=2.7e-07 max_rel_err=1.7e-03 PASS N=8192 NT=256 rows=2 max_abs_err=2.5e-07 max_rel_err=1.9e-03 PASS This covers the kernel algorithm itself; it does not exercise the ggml dispatch/supports_op integration end to end, which needs a real GPU (or a SYCL GPU plugin) to get past backend registration. test-backend-ops build is verified: fwht.cpp recompiles with zero warnings as part of ggml-sycl. (cherry picked from commit c829670) --------- Co-authored-by: Jingxin (Philip) Li <philipaslee@gmail.com>

…gml-org#28254, ggml-org#29243) (ggml-org#302) * sycl : fix test-backend-ops CI break && restore Kronecker product FWHT support (ggml-org#28016) (ggml-org#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (ggml-org#28184) This reverts commit c845263. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only (cherry picked from commit 4d91760) * sycl: FWHT kernels for block widths above 512 (ggml-org#29243) The SYCL FWHT covers 64 to 512 via the standard butterfly network, plus 384/640/768/1280 via the Kronecker/Paley construction added separately in Hadamard hint can produce (1024, 2048, 4096, 8192); those still fall through to the default case and run as a dense GEMM against the materialized rotation tensor, correct but O(n^2) instead of O(n log n). fwht_kernel_wide runs one row per work-group instead of per sub-group, so each work-item keeps N/NT values rather than N/WARP_SIZE. Butterflies below the sub-group width still shuffle; those up to the work-group width go through work-group local memory; the rest stay in registers. Same butterfly and sign convention as the existing narrow kernel. ggml's SYCL backend registration (dpct::dev_mgr) unconditionally requires a GPU-labeled platform to exist and throws before any op-level test can run, so test-backend-ops could not be exercised on this box (a GPU-less pod) even via the CPU device. Verified instead with a standalone harness: the same kernel body run through a real SYCL CPU device (Intel oneAPI DPC++ 2026.1, OpenCL CPU backend), checked against an independent recursive-doubling Hadamard reference, cross-validated by first running the existing unmodified narrow kernel through the identical harness and confirming it passes (rules out a reference-convention bug before trusting a pass on the new code). Random-input results for all four widths, single- and multi-row: N=1024 NT=256 rows=1 max_abs_err=1.7e-07 max_rel_err=4.9e-04 PASS N=2048 NT=256 rows=1 max_abs_err=1.9e-07 max_rel_err=2.0e-04 PASS N=4096 NT=256 rows=1 max_abs_err=2.0e-07 max_rel_err=1.4e-04 PASS N=8192 NT=256 rows=1 max_abs_err=2.5e-07 max_rel_err=3.8e-03 PASS N=1024 NT=256 rows=7 max_abs_err=2.4e-07 max_rel_err=1.0e-03 PASS N=2048 NT=256 rows=5 max_abs_err=3.0e-07 max_rel_err=9.4e-04 PASS N=4096 NT=256 rows=3 max_abs_err=2.7e-07 max_rel_err=1.7e-03 PASS N=8192 NT=256 rows=2 max_abs_err=2.5e-07 max_rel_err=1.9e-03 PASS This covers the kernel algorithm itself; it does not exercise the ggml dispatch/supports_op integration end to end, which needs a real GPU (or a SYCL GPU plugin) to get past backend registration. test-backend-ops build is verified: fwht.cpp recompiles with zero warnings as part of ggml-sycl. (cherry picked from commit c829670) --------- Co-authored-by: Jingxin (Philip) Li <philipaslee@gmail.com> (cherry picked from commit f450c76) Assisted-by: GitHub Copilot

sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280

d44bc3b

philip-jingxin requested review from a team and ggerganov as code owners August 30, 2026 10:33

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Aug 30, 2026

philip-jingxin commented Sep 1, 2026

View reviewed changes

arthw approved these changes Sep 1, 2026

View reviewed changes

Titaniumtown approved these changes Sep 1, 2026

View reviewed changes

Titaniumtown merged commit 1f3d318 into ggml-org:master Sep 1, 2026
1 check passed

Titaniumtown mentioned this pull request Sep 1, 2026

Revert "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280" #28184

Merged

philip-jingxin mentioned this pull request Sep 2, 2026

sycl : fix test-backend-ops CI break && restore Kronecker product FWHT support (#28016) #28254

Merged

thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026

sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…

049ddac

…80 (ggml-org#28016)

x1250 pushed a commit to x1250/llama.cpp that referenced this pull request Sep 9, 2026

sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…

c433b78

…80 (ggml-org#28016)

zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026

sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…

442a837

…80 (ggml-org#28016)

Lonny154 mentioned this pull request Sep 17, 2026

cuda: add Kronecker FWHT support #29051

Open

bri-prism mentioned this pull request Sep 21, 2026

sycl: FWHT kernels for block widths above 512 #29243

Merged

Navigation Menu

Repository navigation

sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280 - #28016

sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280#28016
Titaniumtown merged 1 commit into
ggml-org:masterfrom
philip-jingxin:sycl-fwht-kronecker

philip-jingxin commented Aug 30, 2026 •

edited

Loading

Uh oh!

philip-jingxin left a comment •

edited

Loading

Uh oh!

arthw left a comment

Uh oh!

Uh oh!

Titaniumtown commented Sep 1, 2026

Uh oh!

Titaniumtown commented Sep 1, 2026 •

edited

Loading

Uh oh!

philip-jingxin commented Sep 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

philip-jingxin commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Changes

Testing & Verification

Environment

Correctness Test

Performance Benchmarks

Requirements

Uh oh!

philip-jingxin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arthw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Titaniumtown commented Sep 1, 2026

Uh oh!

Titaniumtown commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philip-jingxin commented Sep 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

philip-jingxin commented Aug 30, 2026 •

edited

Loading

philip-jingxin left a comment •

edited

Loading

Titaniumtown commented Sep 1, 2026 •

edited

Loading