ggml-opencl: add opt-in Adreno xmem F16xF32 GEMM for prefill

[happyyzy]

Summary

This PR adds an opt-in Adreno xmem GEMM path for OpenCL prefill matmul.

Scope:

• build-time gated by GGML_OPENCL_USE_ADRENO_KERNELS
• runtime opt-in via GGML_OPENCL_ADRENO_XMEM_GEMM=1
• limited to Adreno GPUs
• limited to F16 x F32 -> F32 GGML_OP_MUL_MAT
• limited to contiguous, single-batch GEMM shapes
• requires N > 1, so token-generation / GEMV decode is not routed through this path

The implementation keeps the existing ggml tensor layout externally and uses a small bridge around the xmem GEMM:

• pack F32 activations into a half image
• pack F16 weights into the xmem kernel layout
• run the Adreno xmem OS8 GEMM
• store the half image result back to F32 output

The generic OpenCL matmul path remains unchanged unless the new runtime opt-in is set.

Results

Tested on Adreno 830 with OpenCL:

OpenCL driver: OpenCL 3.0 QUALCOMM build: 0800.71 Compiler E031.47.18.49
build: 09294365a (468)

Qwen2.5 1.5B F16

Before, baseline OpenCL:

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| qwen2 1.5B F16                 |   2.88 GiB |     1.54 B | OpenCL     |  99 |           pp512 |        204.98 ± 0.61 |
| qwen2 1.5B F16                 |   2.88 GiB |     1.54 B | OpenCL     |  99 |           tg128 |         18.84 ± 0.07 |

After, with GGML_OPENCL_ADRENO_XMEM_GEMM=1:

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| qwen2 1.5B F16                 |   2.88 GiB |     1.54 B | OpenCL     |  99 |           pp512 |        356.19 ± 0.31 |
| qwen2 1.5B F16                 |   2.88 GiB |     1.54 B | OpenCL     |  99 |           tg128 |         18.98 ± 0.06 |

Prefill improved from 204.98 tok/s to 356.19 tok/s, about 1.74x.

Qwen2.5 3B F16

Before, baseline OpenCL:

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| qwen2 3B F16                   |   5.75 GiB |     3.09 B | OpenCL     |  99 |           pp512 |        101.26 ± 0.04 |
| qwen2 3B F16                   |   5.75 GiB |     3.09 B | OpenCL     |  99 |           tg128 |          9.53 ± 0.04 |

After, with GGML_OPENCL_ADRENO_XMEM_GEMM=1:

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| qwen2 3B F16                   |   5.75 GiB |     3.09 B | OpenCL     |  99 |           pp512 |        163.90 ± 2.84 |
| qwen2 3B F16                   |   5.75 GiB |     3.09 B | OpenCL     |  99 |           tg128 |          9.51 ± 0.11 |

Prefill improved from 101.26 tok/s to 163.90 tok/s, about 1.62x.

Decode is intentionally unchanged. Decode-only profiling confirmed that token generation stays on the existing OpenCL path (adreno_xmem count = 0).

Correctness

Checked end-to-end generation with the xmem path enabled on Qwen2.5 1.5B F16 and Qwen2.5 3B F16. Both models produced normal decode output.

Notes

This path depends on Qualcomm Adreno OpenCL subgroup constant-load extensions and is therefore guarded behind the existing Adreno kernel build option plus an explicit runtime environment variable.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - AI was used as an assistant for code review, testing coordination, and drafting text from user-provided results. The contributor is responsible for the submitted changes.

[ggml-gh-bot]

Hi @happyyzy, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[happyyzy]

Thanks for the note. I closed the other open PR (#22117) and will focus on this smaller xmem GEMM PR first.

[lhez]

Thank you - this is much easier. Will take a closer look in the next few days.

[lhez]

I was able to reproduce your results on A840 with Qwen3-1.7B-f16.

A840

ggml_opencl: device: 'QUALCOMM Adreno(TM) 840 (OpenCL 3.0 Adreno(TM) 840)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: 0842.27.1 Compiler E031.50.19.18

With GGML_OPENCL_ADRENO_XMEM_GEMM=1,

ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: Adreno xmem F16xF32 GEMM enabled (temporary weight prepack)
ggml_opencl: loading OpenCL kernels......................................................................................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 840 (OpenCL 3.0 Adreno(TM) 840)'

model	size	params	backend	ngl	test	t/s
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	pp512	394.43 ± 24.45
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	tg128	13.60 ± 0.08

build: f79069d9d (9050)

Without GGML_OPENCL_ADRENO_XMEM_GEMM,

ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels......................................................................................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 840 (OpenCL 3.0 Adreno(TM) 840)'

model	size	params	backend	ngl	test	t/s
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	pp512	169.50 ± 0.25
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	tg128	13.55 ± 0.04

build: f79069d9d (9050)

X2-90

ggml_opencl: device: 'Qualcomm(R) Adreno(TM) X2-90 GPU (OpenCL 3.0 Qualcomm(R) Adreno(TM) X2-90 GPU)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: 863.0 Compiler DX.50.39.00

With xmem,

ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: Adreno xmem F16xF32 GEMM enabled (temporary weight prepack)
ggml_opencl: loading OpenCL kernels........................................................................................................................
ggml_opencl: default device: 'Qualcomm(R) Adreno(TM) X2-90 GPU (OpenCL 3.0 Qualcomm(R) Adreno(TM) X2-90 GPU)'

model	size	params	backend	ngl	test	t/s
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	pp512	681.08 ± 32.66
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	tg128	28.60 ± 0.23

build: 2a52ab71a (9072)

model	size	params	backend	ngl	test	t/s
qwen3 4B F16	7.49 GiB	4.02 B	OpenCL	99	pp512	294.58 ± 6.58
qwen3 4B F16	7.49 GiB	4.02 B	OpenCL	99	tg128	12.88 ± 0.05

build: 2a52ab71a (9072)

Without xmem,

ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels........................................................................................................................
ggml_opencl: default device: 'Qualcomm(R) Adreno(TM) X2-90 GPU (OpenCL 3.0 Qualcomm(R) Adreno(TM) X2-90 GPU)'

model	size	params	backend	ngl	test	t/s
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	pp512	414.83 ± 13.83
qwen3 1.7B F16	3.78 GiB	2.03 B	OpenCL	99	tg128	28.03 ± 0.26

build: 2a52ab71a (9072)

model	size	params	backend	ngl	test	t/s
qwen3 4B F16	7.49 GiB	4.02 B	OpenCL	99	pp512	169.62 ± 2.46
qwen3 4B F16	7.49 GiB	4.02 B	OpenCL	99	tg128	12.91 ± 0.06

build: 2a52ab71a (9072)

[happyyzy]

Thanks, addressed both comments: the xmem program is now local to load_cl_kernels() and released after kernel creation, and the runtime gate now checks Adreno family instead of A8x only.

[happyyzy]

Just checking in on this PR.

The requested review changes have been addressed, and the maintainer-side A840 / X2-90 benchmark results look positive. Please let me know if there is anything else I should adjust on my side.

[happyyzy]

Thanks, addressed the naming comments: the kernel file is now gemm_xmem_f16_f32_os8.cl, the main OpenCL kernel is kernel_gemm_xmem_f16_f32_os8, and the C++ kernel object name matches. I also updated the PR description to say Adreno GPUs instead of A8X.