Proof of concept TCP server mode by tarruda · Pull Request #278 · ggml-org/llama.cpp

tarruda · 2023-03-19T02:46:27Z

This builds on my other PR to implement a very simple TCP mode.

The new mode first loads the model then listens for TCP connections on a port. When a connection is received, arguments will be parsed using a simple protocol:

First the number of arguments will be read followed by a newline character.
Then each argument will be read, separated by the 0 byte.
With this we build an argument vector, similar to what is passed to the program entry point.
The resulting "argv" is passed gpt_params_parse.

Finally llama_main will be executed with the input/output streams connected to the socket.

I've included two sample bash scripts which can be used to test the new mode. This is how it works:

Run ./chat_tcp_server.sh in a terminal.
In a second terminal, run ./chat_tcp_client.sh. This will connect to the server and start a sample chat session.

One thing to note is that this mode is only implemented for Unixes. There's two reasons for that:

I never wrote win32 TCP code, so I'm not familiar with the API
There's really no advantage in implementing this for win32 because it doesn't support fork(). The main advantage of using this mode is that it serves each connection in a separate process which inherits memory from the parent (so the model only has to be loaded once).

While the protocol is a bit "low level", it should be easy to write a higher level API on top of this, such as a node.js web server or next.js app.

ggerganov · 2023-03-19T18:43:00Z

Shall we close #267 now that we have this?

tarruda · 2023-03-19T18:44:34Z

Shall we close #267 now that we have this?

We can, this already includes all changes in #267. Only reason I did in a separate PR was to simplify reviewing.

tarruda · 2023-03-19T19:08:32Z

Rebased

* fix coloring of last `n_batch` of prompt, and refactor line input * forgot the newline that needs to be sent to the model * (per #283) try to force flush of color reset in SIGINT handler

avilum · 2023-03-19T20:37:44Z

If anyone is seeking for working client/server implementation,
I wrote a minimal Go/Python server and client with live streaming, that is based on this awesome repo.
See https://github.com/avilum/llama-saas

tarruda · 2023-03-19T21:26:07Z

If anyone is seeking for working client/server implementation, I wrote a minimal Go/Python server and client with live streaming, that is based on this awesome repo. See https://github.com/avilum/llama-saas

@avilum One problem with your implementation that the service spawns a new llama.cpp instance for every http request, so it can take a long time to respond (model has to be loaded every time).

I suggest you try using this branch with a different approach: Start llama.cpp once, and on every http request you create a TCP connection to the singleton instance. This will get you a clean environment for every request without the overhead of reloading the model.

avideci · 2023-03-19T22:23:43Z

If anyone is seeking for working client/server implementation, I wrote a minimal Go/Python server and client with live streaming, that is based on this awesome repo. See https://github.com/avilum/llama-saas

@avilum One problem with your implementation that the service spawns a new llama.cpp instance for every http request, so it can take a long time to respond (model has to be loaded every time).

I suggest you try using this branch with a different approach: Start llama.cpp once, and on every http request you create a TCP connection to the singleton instance. This will get you a clean environment for every request without the overhead of reloading the model.

For the sake of POC I ran the process every prompt.
I most definitrly Agree - I am working at the moment on loaded the DLL only once (main exe) and once this is merged, I might evensend the data over this TCP socket.
I'll load the model and then feed it with inputs, from the Go code, over any kind of socket / IPC.

jart · 2023-03-20T03:36:00Z

I disagree with this change. This is a large rearchitecting of the project that fundamentally changes its vision. No one will run your daemon. What value does having a TCP server mode offer, aside from fixing loading time? The issue of loading time is solved by #91 which we've implemented in the mmap branch: https://github.com/ggerganov/llama.cpp/tree/mmap It will be much easier to support win32 using mmap than it would be to support winsock.

anzz1 · 2023-03-20T05:36:02Z

Why even the need of spawning multiple processes instead of multiple threads? Threads are already built-in, unlike sockets or mmap. The only truly portable thing is standard I/O, which can be redirected and also easily communicated with using simple file streams which are supported by everything. Instead of changing the main implementation much at all, you could just build any modules outside the main implementation as modules and communicate using these simple file streams. The main llama input would not need any sort of 'protocol' but just listen to '\n' or EOF like it currently does, and whatever modules could just follow that paradigm while communicating through the file streams. Am I missing something here?

Again if the case is that more processes is what is wanted and an ability to share the state between them, a more general approach would be making a C-style API with something simple like struct state{...} , save_state(*state), load_state(*state). Then any implementation could just live as a separate module and use those general funcs to manipulate the state however they wish, and this would keep the main program clean of any non-portable code.

spirobel · 2023-03-20T09:43:51Z

@jart

I disagree with this change. This is a large rearchitecting of the project that fundamentally changes its vision.

That is true. It is better to keep the scope focused and make sure llama.cpp is as stable as possible.

@tarruda

While the protocol is a bit "low level", it should be easy to write a higher level API on top of this, such as a node.js web server or next.js app.

We can bundle llama with node.js in its current form. There is a library called caxa: https://github.com/leafac/caxa

It bundles any nodejs app into a single executable. As a side effect of the way it is architected it unzips any executeables from the dist folder at runtime.

So we can just place a compiled llama version in there and bundle it together. Then we can build a rest-api in node that can be easily called from nextjs.

I also found a way that turns an openapi.yml file into the single source of truth for the nodejs routes.
The result of this is,
that we get an interactive docs page for free that spits out curl commands to interact with the api.

Here is the repo that combines caxa and this technique to make a single binary rpc daemon: https://github.com/spirobel/monerochan-merchant-rpc

If there is an interest I can make something like this for llama.cpp.

If we make a one click template for a provider like digital ocean people could spin up their own on demand instances that work just like the openAI api.

tarruda · 2023-03-20T09:47:43Z

This is a large rearchitecting of the project that fundamentally changes its vision

@jart @spirobel there's no rearchitecting here, in fact my goal was to introduce a server/client model with minimal changes. If you don´t pass the -l option, the app works exactly as it is now. This PR might seem like it is changing a lot, but if you review the commits individually, you will see there's not much change to existing logic.

The way this was implemented was basically by moving all the code (except for model loading code) from the main function to another reusable function called llama_main, then replacing the direct std streams references by parameters passed to it. So the same code runs unmodified on stdin/stdout/stderr or on a TCP connection.

What value does having a TCP server mode offer, aside from fixing loading time?

There are more unexplored applications of this TCP server mode. Here's a few ideas:

Wrap into an http/json/rest/websocket server for using in a web application. Eventually I would like to write an API that is compatible with ChatGPT's API.
Do any kind of processing (such as preload the prompt) prior to accepting connections/requests. This would allow you to essentially respond to queries more quickly by preloading a chatbot prompt, then for each request just process user input.
Server/client usage, you can run llama.cpp in a more powerful computer and share with other devices in the same LAN.

@ggerganov also thinks this is a good idea: #267 (comment)

I don't think this PR is mutually exclusive with the work you are doing, it is still useful to load the model faster on new process startups.

spirobel · 2023-03-20T09:58:42Z

@tarruda

Do any kind of processing (such as preload the prompt) prior to accepting connections/requests. This would allow you to essentially respond to queries more quickly by preloading a chatbot prompt, then for each request just process user input.

Responsiveness and proper concurrency is always good! 😀👍 Nothing more frustrating than lagging or hung up programs.

That being said,

Wrap into an http/json/rest/websocket server for using in a web application. Eventually I would like to write an API that is compatible with ChatGPT's API.

please take a look at the single binary rpc that I built! I am happy to answer any questions! I genuinely believe it is better to implement this in nodejs instead of doing it in cpp. Monero also has a cpp rest rpc daemon and it is not fun to work with. It always hangs up or becomes unresponsive for long times. Also the documentation is always out of date.
Using openapi (not openai 😅) as a single source of truth for routing and documentation solves this issue permanently.

We can bundle llama with node.js in its current form. There is a library called caxa: https://github.com/leafac/caxa

It bundles any nodejs app into a single executable. As a side effect of the way it is architected it unzips any executeables from the dist folder at runtime.

So we can just place a compiled llama version in there and bundle it together. Then we can build a rest-api in node that can be easily called from nextjs.

I also found a way that turns an openapi.yml file into the single source of truth for the nodejs routes. The result of this is, that we get an interactive docs page for free that spits out curl commands to interact with the api.

Here is the repo that combines caxa and this technique to make a single binary rpc daemon: https://github.com/spirobel/monerochan-merchant-rpc

If there is an interest I can make something like this for llama.cpp.

If we make a one click template for a provider like digital ocean people could spin up their own on demand instances that work just like the openAI api.

tarruda · 2023-03-20T09:59:11Z

@anzz1 I don't see how it could work using threads. There's only one instance of the model in memory, AFAIK ggml API is not thread-safe or supports concurrent usage (@ggerganov please correct me if I'm wrong).

tarruda · 2023-03-20T11:52:04Z

implement this in nodejs instead of doing it in cpp

@spirobel If you want to implement a server mode in another program/language such as node.js and without changes to llama.cpp, there's two ways I see you can go about it:

Spawn a single instance of llama.cpp at server startup, and synchronize access to stdin/stdout between all requests. This might work well for a single user, but if llama.cpp the process crashes, you would have to restart it. Not to mention you would not be able to customize parameters per connection/request.
Spawn a new llama.cpp instance for each request (this is what @avilum did), which would be very slow with the way the model is loaded right now (also requires duplicating the model in memory). I'm not familiar with the mmap loading solution, but assuming it makes loading the model instant, would it be able to preload the prompt and/or customize the prompt/seed per request? Preloading the prompt is not implemented right now but it is something that can be done in the server mode I implemented here.

I agree that higher level abstractions are better done in platforms like node.js or python, but in this case I don't think it would be possible to implement a server in purely node.js and have the same efficiency of a fork/connection server approach.

Now, here is the last paragraph of the PR description

While the protocol is a bit "low level", it should be easy to write a higher level API on top of this, such as a node.js web server or next.js app."

As you can see, my goal with this PR was to provide a base server/protocol that can be wrapped in a higher level API. In fact I implemented the parameter passing protocol in a way to allow reusing the existing gpt_params_parse function, so any new parameters added are automatically supported. The server mode is basically a "zygote" for quickly spawning new instances of llama.cpp with custom parameters (except for n_ctx, which has to be passed to ggml_init).

Technically I could have implemented a higher level abstraction such as a simple HTTP endpoint that parses json messages, but it would require me to add at least a JSON parsing library, which goes against the goals of the project ("no dependencies"). I also think we would lose some flexibility by making assumptions about the format of the API, better to have a lower level implementation that can be tailored for various purposes.

Since this server mode is meant to be wrapped in a higher level abstraction, it might be better to implement it using Unix sockets instead of TCP sockets, which I might do later after getting some feedback. This is still an experiment/POC.

* Adding ability to use THP on Linux * Use the actual page size4 used for mmap also in munmap * Add -thp to llama-bench --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

docs : add harness init strided-view bug to coverage limits

Claude-Session: https://claude.ai/code/session_01DSrpF2XYi9dav1e3hSAkaS Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01DSrpF2XYi9dav1e3hSAkaS Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 87268f7)

…#294) * Adds an XMX (DPAS) path for PQ2_0 and PTQ1_0 matrix multiplies on Intel Xe2 GPUs (Battlemage, Lunar Lake). It covers every batch size: prompt processing, single-token decode, parallel decode and speculative verification. On Xe2 it replaces MMVQ and the dequantize-to-FP16 prompt path from ggml-org#278 for these types. The main idea: PQ2_0 packs value+1 in 2-bit fields, lowest first, which as a little-endian dword is already the DPAS 2-bit operand layout. The kernel feeds the codes to DPAS as signed 2-bit (s2) against int8 activations, converting them in registers with a borrow-free per-field subtract. Weights are never widened to 8 or 16 bits in memory. How it works: - **Weight layout.** On first use, each PQ2_0 weight is rewritten in place into an SoA layout (same size): all 32-byte qs blocks first, then all fp16 scales. The qs row pitch is then nb*32 bytes, so one transposed 2D block load fetches 16 rows x 128 weights directly in the DPAS B-operand layout. The 34-byte AoS blocks are only 2-byte aligned and can't be loaded that way. - **PTQ1_0.** Base-3 has no DPAS form, so PTQ1_0 weights are expanded once into the same 2-bit layout. That is 34 bytes a block instead of 28. On Xe2 devices the buffer type reserves the room for it (get_alloc_size), so VRAM holds only the expanded copy: about +25% for these weights, and the file on disk is unchanged. - **Activations.** They are quantized to int8 with one scale per 128 values, one PQ2_0 block. The four DPAS of a block then accumulate in int32 before a single float rescale, instead of one rescale per 32 values. - **One kernel for all batch sizes.** Tiles are 8/16/32 tokens x 32 rows. For small batches, up to 16 threads of a work-group split K for the same tile and reduce through SLM, so a mat-vec keeps enough loads in flight. Scope and fallbacks: - On by default on BMG-G21/G31 and LNL-M. `GGML_SYCL_DISABLE_XMX=1` or `GGML_SYCL_ENABLE_OPT=0` turns it off. - Only plain MUL_MAT on 2D weights in regular (non-split, non-COMPUTE) buffers with K >= 256. Everything else (op offload, split buffers, MUL_MAT_ID, other devices) takes the existing paths unchanged. - Like the existing reorders, a tensor keeps its new layout once rewritten. The gate refuses MUL_MAT_ID expert slices and views. `get_rows` asserts if it ever sees a rewritten PQ2_0/PTQ1_0 tensor. Files: `pq2_xe2.cpp/.hpp` (new), `ggml-sycl.cpp` (gate, alloc size, extras), `common.hpp` (layout flag), `getrows.cpp` (assert). Intel Arc Pro B50 (16 GB), oneAPI 2026.1, Level Zero, Linux `xe` driver, `-ngl 99`, Ternary Bonsai models. "ggml-org#278" is the same build with `GGML_SYCL_DISABLE_XMX=1`, i.e. the current prism code path. Both columns were measured in the same session. `llama-bench`, t/s: | model | test | ggml-org#278 | this PR | change | |---|---|---|---|---| | 1.7B PQ2_0 | pp128 | 2574 | 6133 | 2.38x | | | pp512 | 5218 | 7645 | 1.47x | | | tg64 | 152.4 | 178.1 | +17% | | 4B PQ2_0 | pp128 | 1068 | 3068 | 2.87x | | | pp512 | 2298 | 3272 | 1.42x | | | tg64 | 77.4 | 99.8 | +29% | | 27B PQ2_0 | pp128 | 148 | 346 | 2.33x | | | pp512 | 290 | 370 | 1.28x | | | tg64 | 12.39 | 16.53 | +33% | | 27B PTQ1_0 | pp128 | 127 | 348 | 2.74x | | | pp512 | 267 | 370 | 1.39x | | | tg64 | 12.88 | 16.54 | +28% | Parallel decode (`llama-batched-bench -npp 128 -ntg 64`, aggregate t/s): | sequences | 1.7B PQ2_0 | 4B PQ2_0 | 27B PQ2_0 | 27B PTQ1_0 | |---|---|---|---|---| | 1 | 151 -> 171 | 76 -> 97 | 12.2 -> 16.1 | 13.1 -> 16.1 | | 2 | 250 -> 334 | 125 -> 186 | 18.9 -> 29.7 | 20.2 -> 29.7 | | 4 | 380 -> 613 | 185 -> 355 | 25.2 -> 48.0 | 26.3 -> 47.9 | | 8 | 478 -> 1010 | 224 -> 618 | 29.0 -> 68.6 | 31.0 -> 68.0 | | 16 | 322 -> 1446 | 138 -> 900 | 20.2 -> 85.7 | 17.2 -> 85.8 | | 32 | 588 -> 1769 | 262 -> 1184 | 32.5 -> 98.4 | 29.0 -> 98.4 | On ggml-org#278, batches of 9+ tokens leave MMVQ for the dequantize-to-FP16 path, which is why throughput drops at 16 sequences there. Speculative decoding (27B PQ2_0 target, Qwen3.5-0.8B Q8_0 draft, `--spec-draft-n-max 1`, greedy, 256 tokens), t/s: | | ggml-org#278 | this PR | |---|---|---| | prose, target only | 13.1 | 17.6 | | prose, with draft | 12.1 | 20.5 | | code, target only | 12.7 | 17.5 | | code, with draft | 12.8 | 21.9 | With ggml-org#278, drafting does not pay. With this PR, verification is cheap enough that n_max=1 gains 17-25% over the target alone. Longer drafts lose on acceptance (15-33% at n_max 2-3 with this draft model). Where the decode gain comes from: at the 27B FFN shape (17408x5120, one token), the PQ2_0 mat-vec goes from 199 us (MMVQ, about 53% of the B50's 224 GB/s) to 88 us. End to end, 27B decode now reads about 123 GB/s. Most of the remainder is the ~2300 non-matmul kernels per token. - `test-backend-ops -b SYCL0`: MUL_MAT 1329/1329, MUL_MAT_ID 1035/1035, GET_ROWS 119/119 (all types). Full suite, run per op: 14703/14759. The 56 failures (CONV_2D, ROLL, FLASH_ATTN_EXT, LIGHTNING_INDEXER) are identical with this path disabled, so they predate this PR. SET_ROWS aborts on current prism regardless of this PR (`set_rows.cpp:549: Unsupported tensor type!`), so it was run separately. - Perplexity (`-c 512`, 5 chunks), ggml-org#278 -> this PR: | model | setting | ggml-org#278 | this PR | |---|---|---|---| | 1.7B PQ2_0 | b1 | 8.4089 | 8.4335 | | 1.7B PQ2_0 | b512 | 8.4233 | 8.4149 | | 4B PQ2_0 | b1 | 7.0584 | 7.0565 | | 4B PQ2_0 | b512 | 7.0545 | 7.0506 | | 1.7B PQ2_0 | -ngl 0 (op offload) | 8.4225 | 8.4225 | | 4B PQ2_0 | -ngl 20 (partial) | 7.0541 | 7.0535 | | 27B PTQ1_0 | b512 | 4.2421 | 4.2399 | - KL divergence against a CPU-only reference (1.7B, `-dev none --no-op-offload`), ggml-org#278 vs this PR: - b1: 0.00122 vs 0.00125. - b512: 0.00106 vs 0.00113. - Same top token: 97.4-97.8% vs 97.7-97.9%. - The differences are within the error bars. The per-128 activation scale costs nothing measurable. - This introduces ESIMD/XMX code to the SYCL backend, a new pattern here. I'm happy to adjust structure or naming to fit how the maintainers want it organized. - Tuning (tile sizes, K-split thresholds) was done on the B50 only. BMG-G31 and LNL-M are gated in by architecture but untested. - I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) - AI usage disclosure: YES - the code and this description were written by Claude (Anthropic) under my direction; I tested it on my own hardware (Intel Arc Pro B50) and reviewed the changes. * sycl: gate the PQ2_0/PTQ1_0 XMX path on 16-wide DPAS instead of a device list The path was limited to BMG-G21/G31 and LNL-M by name. It now runs on any XMX device whose int8 DPAS is 16 wide, as the runtime reports it through matrix_combinations: Xe-HPC, Xe2, Xe3 and later. 8-wide XMX (Xe-HPG, Arrow Lake-H) and devices without XMX keep the existing paths. AOT builds compile the kernels for every GGML_SYCL_DEVICE_ARCH device, and they do not build for targets without 16-wide DPAS and 2D block loads (ocloc crashes on acm-g10, and fails on tgllp, dg1, mtl and pvc-vg). pq2_xmx.cpp is now built only when every AOT device is Xe-HPC, Xe2 or later; otherwise it compiles to stubs. Renames pq2_xe2 to pq2_xmx to match. (cherry picked from commit a14c7de) Assisted-by: GitHub Copilot

…B on Intel Arc (port of Torchit1/arc-b580) (ggml-org#356) * Bonsai 2 27B (PTQ1_0) on Intel Arc B580: SYCL + Vulkan performance work - SYCL: PTQ1_0 weights repacked to 2-bit at load and multiplied on XMX via the TernSYCL int8 x int2 DPAS kernels (vendored, BSD 3-Clause), activations quantized to int8 with round-to-nearest (GGML_SYCL_PTQ1_T2=ffn|all). - SYCL: decode attention for q4_0 KV serving 1-4 query tokens per launch; blocked gated delta-net; PTQ1_0 mat-vec kernels (single column, decode-once multi column, same-input merge, fused gate/up); narrow concat. - Vulkan: matching PTQ1_0 multi-column, decode attention, gated delta-net and coopmat work. - Speculative decoding: draft KV cache type (-ctkd/-ctvd) honoured by the MTP draft context, LLAMA_ARG_SPEC_DRAFT_UBATCH to shrink its compute buffer. - docs/bonsai-arc-b580.md: results at 128K on a B580, build and run instructions, switches, credits. Based on the bonsai-combo branch of professorpalmer/llama.cpp-ada-ternary (PrismML PR ggml-org#221). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 753323e on Torchit1/llama.cpp arc-b580) Excluded from this port: .github docs ggml/src/ggml-vulkan Conflicts resolved against prism: fwht.cpp keeps prism's FWHT (ggml-org#302) and adds this commit's signed variant beside it as ggml_sycl_op_fwht_signed; ggml_sycl_op_mul_mat_sycl takes fp16 for PQ2_0/PTQ1_0 as before (ggml-org#278) plus this commit's large-batch rule; the T2 path runs before the PQ2_0 XMX path (ggml-org#294), which still serves PQ2_0/PTQ1_0 when GGML_SYCL_PTQ1_T2 is unset. * sycl: PQ2_0 weights on the TernSYCL XMX path too (GGML_SYCL_PTQ1_T2) PQ2_0 (2-bit codes 00=-1 01=0 10=+1, fp16 scale per 128) is the same size as the 2-bit XMX layout, so it is remapped in place. Code 3 (+2) has no 2-bit two's-complement form: the repack reports it and the matmul aborts with a message. test-backend-ops MUL_MAT type_a=pq2_0 with GGML_SYCL_PTQ1_T2=all: 84/84; ptq1 94/94. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5056bde on Torchit1/llama.cpp arc-b580) * sycl: PTQ1 T2 path only on Xe2+ GPUs, fall back with a warning elsewhere The T2 kernels use 16-lane int8 x int2 DPAS builtins that only Xe2 and newer drivers provide; on an Arc 140T (Xe-LPG+) GGML_SYCL_PTQ1_T2=all failed with 'No kernel named ... was found'. Now checked once against the device architecture. GGML_SYCL_PTQ1_T2_ANYGPU=1 skips the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 731c1a8 on Torchit1/llama.cpp arc-b580) * sycl/ternsycl: 64-bit surface base address on Windows The 2D block I/O builtins took the surface base as 'long', which is 32 bits in SYCL device code on Windows (the device follows the host's LLP64 data model), so the weight loads would have used truncated addresses. 'long long' is 64 bits on both; identical code on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2f0cf6e on Torchit1/llama.cpp arc-b580) * sycl: GGML_SYCL_PTQ1_T2_TILE lab knob for the TernSYCL large-M tile (default stays best) B580 pp1024 (ub 1024, all weights on XMX): 8x128/8x2 939 t/s (default), 8x128/16x1 949, others 480-857. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 244c4d4 on Torchit1/llama.cpp arc-b580) * sycl: two-pass top-k for long rows (GGML_SYCL_TOPK_OLD=1 restores the one-group kernel) The one-work-group top-k took ~550 us per MTP draft token on the B580 (248K-entry vocabulary). Now up to 256 groups per row with a pairwise local-memory merge, then one group merges the winners; ties keep the lower index. test-backend-ops TOP_K 325/325. 200K entries: k=1 241 -> 9.6 us, k=10 647 -> 71 us. Live settings: fresh +1-2%, rename/edit within noise (the speculative step is not purely GPU-bound). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit afe4c82 on Torchit1/llama.cpp arc-b580) * sycl: q4_0 decode attention on XMX via s8 x s4 DPAS (opt-in GGML_SYCL_FA_DEC_DPAS) Q.K uses intel_sub_group_i8_i4_matrix_mad_k32 (IGC provides it on Xe2 though the public extension only lists i4_i4_k64): the raw 16-byte q4_0 block of a key XOR 0x88888888 is the B operand, one key per lane, against int8 Q with one RNE scale per row and block (q4_0 byte order absorbed by pairing dims l and l+16 in A). P.V computes O^T = V^T P^T on f16 DPAS so each lane dequantizes its own key's V block from 5 dword loads. Token chunks of 8 let one kernel serve 1..32 (or =N) query tokens, so n-gram verify batches stop falling back to oneDNN + a full-KV f16 conversion. Per layer (Bonsai 27B shape, real cache layout) old -> new, us: 16K 1 tok 266 -> 123, 4 tok 457 -> 244, 16 tok 2217 -> 776; 48K 1 tok 768 -> 433, 4 tok 1295 -> 671, 16 tok 6629 -> 1915. FLASH_ATTN_EXT decode cases incl. 12/16/17/32 tokens and the permuted layout: 57/57. GGML_SYCL_FA_DEC_DPAS_PROBE skips phases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e5a4d8a on Torchit1/llama.cpp arc-b580) * sycl: MKL flash-attention softmax, one work-group per query row mkl_fa_online_softmax_chunk gave each query row to one work-item that walked the whole KV chunk twice, so neighbouring work-items read addresses chunk_size floats apart. Above the oneDNN KV cap this kernel was 72% of generation time (Bonsai 27B, 115K-token context: verify batches > 32 tokens and every new prompt). Now 256 work-items per row with coalesced reads and group reductions; same math. Bonsai 27B, live flags, 114.6K-token code context: new code 17.6 -> 44.0 t/s, edit 10.8 -> 27.5 t/s, identical output; short prompts unchanged. FLASH_ATTN_EXT with the MKL path forced (ONEDNN_MAX_KV=512): 82/84, the same two pre-existing prompt-batch failures as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a122b2b on Torchit1/llama.cpp arc-b580) * sycl: GGML_SYCL_MMVQ_CHUNK_MAX, largest batch run as MMVQ column chunks Quantized mul_mat with 33+ columns fell through to dequantize-all-of-src0 + GEMM per call. For speculative-verify sized batches that is ruinous: Gemma 4 12B q4_0 with n-gram drafts of 33-64 tokens spent ~94% of GPU time in dequantize_row_q4_0_sycl_reorder. Default stays 4 x MMVQ_MAX_BATCH_SIZE (32); with 128, Gemma 4 12B + QAT assistant (ngram n-max 64): rename 79 -> 186 t/s, edit 57 -> 136. Bonsai 27B unchanged within noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e8bd534 on Torchit1/llama.cpp arc-b580) * sycl: oneDNN / MKL flash attention only for packed interleaved K/V Reported from an Arc 140T and reproduced on a B580: FLASH_ATTN_EXT with quantized K/V in the permuted layout returned ERR ~1.2 for every KV length from 1024 up (ggml-org#27769). The failing test builds the cache view with a gap between tokens (2x the heads allocated, half viewed); the oneDNN SDPA and MKL dequant paths take the token stride from the head count. llama.cpp's own cache view is always packed, so real models were unaffected (Bonsai 27B perplexity at 4K: oneDNN 8.7273, MKL 8.7311, TILE 8.7330, and still 8.7273 with this guard). Interleaved K/V with a gap now falls back to TILE. FLASH_ATTN_EXT 4137/4137 (was 2 failing), plus a new q4_0 real-layout sweep (kv 1024..8192, nb 33/64/512) 29/29. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 226166a on Torchit1/llama.cpp arc-b580) * sycl: XMX decode attention only on Xe2+, fall back with a warning elsewhere GGML_SYCL_FA_DEC_DPAS uses the 16-lane int8 x int4 DPAS builtin that only Xe2 and newer provide; on Alchemist / Xe-LPG it would fail at the first launch (like the T2 path did before 9c25b48). Now checked once against the device architecture, sharing ggml_sycl_device_is_xe2() with the T2 gate; GGML_SYCL_FA_DEC_DPAS_ANYGPU=1 skips the check. B580 unchanged: 4-token verify at 48K 671 us (1295 without). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d441076 on Torchit1/llama.cpp arc-b580) * sycl: XMX q4_0 decode attention for Gemma 4 (head 512 x GQA 16 global, head 256 x GQA 2 sliding) The s8 x s4 DPAS decode kernel takes the head size as a template parameter (K read block by block, P.V subgroups own D/16 dims) and serves two more shapes when GGML_SYCL_FA_DEC_DPAS is on: Gemma 4's global layers (512, 1 KV x 16 Q heads, token chunks of 2) and sliding layers (256, 2 per KV head, chunks of 8). Splits target ~256 work-groups in all (a single KV head used to leave the GPU idle); combine step templated on D. Global-layer attention at 48K, us: 1 token 1114 (f16 TILE) -> 236, 4 tokens 2011 -> 635. Gemma 4 12B QAT + assistant, q4_0 KV, 128K context: short 92 -> 100 t/s new code; at 48K new code 47 -> 65, edit 44 -> 57 (f16 KV was 55 / 47). Bonsai shape unchanged (4-token verify at 48K 638 us). FLASH_ATTN_EXT: Gemma global 20/20, sliding 10/10, Bonsai 36/36. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c759ddb on Torchit1/llama.cpp arc-b580) * sycl: reordered q4_0 small-batch GEMM on XMX via s8 x s4 DPAS (opt-in GGML_SYCL_Q4_0_DPAS) Verify batches of 5..256 columns on q4_0 weights used MMVQ column chunks (weights re-read per 8 columns) or dequantize-everything + GEMM. The new kernel streams each weight row once for up to 64 columns (chunks of 64 beyond): one lane per weight row, B = the row's 16 q4_0 bytes XOR 0x88888888 through intel_sub_group_i8_i4_matrix_mad_k32, A = activations quantized once per call to int8 (per row and 32-block, elements l / l+16 paired). Xe2 only (ggml_sycl_device_is_xe2). GGML_SYCL_Q4_0_DPAS_MIN / _MAX (5 / 256). 15360x3840 q4_0: 8 columns 150 -> 81 us, 16 columns 292 -> 89 us. Gemma 4 12B QAT + assistant: rename 188 -> 275 t/s, edit 138 -> 168 (ngram n-max 96), new code unchanged, rename/edit outputs byte-identical. MUL_MAT q4_0 Gemma shapes 1..256 columns: 20/20. Also Gemma 4 q4_0 mat-mul perf cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 03594d5 on Torchit1/llama.cpp arc-b580) * sycl: large T2 prompt batches through oneDNN's int8 GEMM (GGML_SYCL_T2_W8A8_MIN) Before the TernSYCL path went live, PTQ1_0 prompts ran on the W8A8 oneDNN GEMM (pp512 1013); GemmMT is better for verify-sized batches but slower on full prompt batches. New ggml_sycl_w8a8_mul_mat_t2 converts the repacked 2-bit layout to int8 (per-row scale, group ratios) and uses the same gemm_s8. The conversion is a fixed cost per call, so it only pays from ~1024 columns: Bonsai 27B pp1024 (ub 1024) 941 -> 1062, pp512 901 -> 818 (so MIN=1024 on the live config); 48.6K-token prompt read 67.7 -> 61.6 s (719 -> 789 t/s). PPL c4096 8.7273 -> 8.7359 (+0.1%, int8 rounding as the old W8A8 path). Off by default (0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a68382d on Torchit1/llama.cpp arc-b580) * SYCL decode attention: SLM-staged K/V tiles for 1 query token, one-wave split count; PTQ1_0 T2 NSG knob fattn-dec (DPAS q4_0 path, D 256): single-token decode copies each 128-key K and V tile into SLM with overlapped 16-byte loads and reads lane = key from there; the split count now fills one wave of work-groups (Xe cores x work-groups per core / KV heads) instead of a fixed 64 per head. B580: 48K 1 token 433 -> 244 us, 4 tokens 640 -> 456 us; whole model plain decode +12% at 48K, +21% at 115K, same KLD and output. GGML_SYCL_FA_DEC_STAGE=0 and GGML_SYCL_FA_DEC_SPLITS=64 restore the old behaviour. ptq1-t2: GGML_SYCL_PTQ1_T2_NSG=4 (4 sub-groups for 2-8 row GEMV, +~1%, default off). Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 9cfffcb on Torchit1/llama.cpp arc-b580) * SYCL: s8 x s4 DPAS as inline vISA instead of the intel_sub_group_i8_i4_matrix_mad_k32 builtin The builtin is not in the public subgroup matrix extension list. On a Lunar Lake (Arc 140V) driver that does not provide it, loading the device image failed with "No device image found for external symbol _Z36intel_sub_group_i8_i4_matrix_mad_k32" and every model crashed at the first decode. xe2-dpas.hpp emits the dpas.s4.s8 instruction directly (as libxsmm/TernSYCL's xe2.hpp helpers), so the symbol is gone from libggml-sycl. Used by the q4_0 decode attention and the q4_0 small-batch GEMM. B580: FA 4167/4167, MUL_MAT q4_0 (GGML_SYCL_Q4_0_DPAS=1) 59/59; whole model output byte-identical to the builtin at short / 48K / 115K context, plain and MTP. Single-token attention kernel ~6% slower (48K 244 -> 259 us), whole model within 1-2%. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit f31a4f6 on Torchit1/llama.cpp arc-b580) * SYCL: no IGC builtins left (TernSYCL main, f16 DPAS via vISA), device code split per kernel - ternsycl/: libxsmm/TernSYCL main (11484da) + the long long address fix of TernSYCL#3. Inline vISA only, so the undocumented builtins the old copy declared (intel_sub_group_i8_i2_matrix_mad_k32, __builtin_IB_* 2D block I/O) are gone; it also drops our three JIT workarounds. The per-32 activation scale variant (off by default) is removed. - ptq1-t2.cpp is compiled with -fp-model=precise: under icpx's default fast model the TernSYCL GemmMT returns NaN (upstream's own driver fails validation built that way). - fattn-dec: the f16 P.V DPAS also goes through inline vISA (xe2-dpas.hpp dpas_hf_r8). - CMake: -fsycl-device-code-split=per_kernel (GGML_SYCL_DEVICE_CODE_SPLIT_PER_KERNEL, default ON): the Xe2 XMX kernels are only JIT-compiled when launched, so GPUs without DPAS (Xe-LP, Gen9) don't compile them at all. - test-backend-ops: FA cases for 64-256 query tokens and ~115K history. B580: libggml-sycl has 0 matrix_mad / __builtin_IB references (was f16, i8_i2, 9 IB). FA 4167/4167, MUL_MAT q4_0 59/59, ptq1_0 T2 94/94. KLD (ub 1 / 4 / 512) identical to the previous build. Whole model output identical; MTP 48K 73.6 t/s, 115K 59.6 (was 69.6 / 57.2), plain unchanged. Startup unchanged (first request 4 s). Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 2e8a0e6 on Torchit1/llama.cpp arc-b580) * SYCL decode attention: default DPAS cap 32 -> 256 query tokens Long n-gram verify batches now stay on the q4_0 DPAS kernel instead of oneDNN's path, which converts the whole KV cache to f16 each call. B580 per layer: 48K 64 tokens 28.8 -> 4.9 ms, 115K 256 tokens 253 -> 44 ms. KLD (ub 64-256) 0.012699 vs 0.012800, same top p 92.60 vs 92.89 %, equal within error. Whole model MTP: 48K 73.2 -> 76.1 t/s, 115K 60.3 -> 65.1. GGML_SYCL_FA_DEC_DPAS=32 restores the old cap. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 064461a on Torchit1/llama.cpp arc-b580) * SYCL decode attention: 5-token query chunks for MTP verifies MTP with 4 drafts verifies 5 tokens per step, which ran the 8-token DPAS kernel with 3 empty tokens (48 rows of Q.K / softmax for 30). A 5-token chunk (128-key tiles) cuts the kernel from 637 to 502 us per layer at 48K (215 -> 175 us at 16K); whole model MTP 48K 77.4 -> 78.1 t/s, 115K 65.7 -> 67.1, output identical. GGML_SYCL_FA_DEC_NQ5=0 turns it off (=2 SLM-staged and =3 256-key tiles were slower). test-backend-ops: 5/6-token real-layout cases. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 081d868 on Torchit1/llama.cpp arc-b580) * lab: runtime-switchable SYCL tuning knobs (GGML_SYCL_KNOBS_FILE); LLAMA_ARG_SPEC_NGRAM_FRESH - ggml_sycl_knob(name, def): the attention (DPAS cap, staging, 5-token chunks, split count, probe) and PTQ1_0 T2 (tile, sub-groups, GEMM off) switches are read through it. With GGML_SYCL_KNOBS_FILE set, the NAME=VALUE file is re-read at the start of a graph compute when it changes, so one server can A/B variants. Without it, each knob is read from the environment once, as before. - LLAMA_ARG_SPEC_NGRAM_FRESH=1: the n-gram drafter forgets earlier requests at the start of each one, so repeated benchmark prompts don't replay earlier answers as drafts. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 9db9408 on Torchit1/llama.cpp arc-b580) Excluded from this port: common * sycl: T2 quantized-activation cache + fused Hadamard rotate/quantize Back-to-back T2 mat-vecs on one tensor reuse its int8 activations (GGML_SYCL_PTQ1_T2_QCACHE), and the signed 1024-point FWHT that feeds them runs 128 lanes per block and writes Aq/SA itself (GGML_SYCL_PTQ1_T2_FWHTQ). B580 Bonsai 27B plain decode +3.6-4.9%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 254dfb2 on Torchit1/llama.cpp arc-b580) * sycl: fuse residual add + RMS norm into the T2 rotate/quantize kernel ADD + RMS_NORM + MUL(weight) [+ MUL(signs) + RESHAPE + Hadamard MUL_MAT] runs as one work-group per token (GGML_SYCL_PTQ1_T2_NORMQ). Outputs only used inside the chain are not written; across tokens, outputs must not partly overlap inputs. B580 Bonsai 27B plain decode +1.9-2.6%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4d50f84 on Torchit1/llama.cpp arc-b580) * sycl: vendored TernSYCL -> 8b0ae42, drop the -fp-model=precise override Upstream fixed the GemmMT row-scale broadcast that returned NaN under icpx's default -fp-model=fast (libxsmm/TernSYCL#1). B580: MUL_MAT ptq1_0/pq2_0 178/178 with fast math, KLD unchanged, decode speed unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 546f94c on Torchit1/llama.cpp arc-b580) * sycl: lab knob GGML_SYCL_T2_ZERO_WEIGHTS (zeroes T2 weights to profile non-weight cost) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2dbd839 on Torchit1/llama.cpp arc-b580) * sycl: lab instrumentation - host token timing (GGML_SYCL_HOST_TIMING), W8A8 threshold as a runtime knob; test shapes GGML_SYCL_HOST_TIMING=1 splits decode graphs into between / enqueue / wait host time plus per-op enqueue cost, without a tracer. test-backend-ops: Bonsai attn k/v and output-head mat-vec shapes, FA nh=1,2 at 48K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ec71714 on Torchit1/llama.cpp arc-b580) * sycl: fuse the DeltaNet conv chain for one-token decode (GGML_SYCL_CONV_CHAIN) CONCAT + conv-cache CPY + SSM_CONV + SILU + L2_NORM in one kernel per layer (one work-group per 128-channel head). Outputs are written at the CONCAT's position, so nodes in between must not touch them (checked); an output may sit exactly on the qkv input (same element per channel), and an L2 output on top of the old conv state falls back to its own L2_NORM kernel. Also GGML_SYCL_FAST_SET_TENSOR (opt-in pinned input uploads). B580 Bonsai 27B plain decode +1.3-1.8%, KLD ub1 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b56951e on Torchit1/llama.cpp arc-b580) * sycl: fuse the DeltaNet gate block (GGML_SYCL_DN_GATES) bf16 alpha + beta mat-vecs (2 x 48 rows, same input, up to 8 tokens) with the dt-bias add, softplus, ssm_a multiply and sigmoid in one kernel: 9 graph nodes / 7 launches per layer -> 1 (2 when an output aliases the input and goes through a scratch buffer). GGML_SYCL_DN_GATES_DEBUG=1 prints why a block is not fused. B580 Bonsai 27B plain decode +0.5-1.4%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 38c6be5 on Torchit1/llama.cpp arc-b580) * sycl: same-input T2 mat-vecs in one launch (GGML_SYCL_T2_MULTI) q+k+v, GDN qkv+z and FFN gate+up (<= 8 tokens) share one quantization and one GEMV launch: TernSYCL's Gemv body now takes its work-group indices (run()), and a wrapper maps each group to its matrix through a single call site (three inlined copies spilled: 34.6 vs 46.3 t/s). Grouping and output staging as the PTQ1_0 merge. B580 Bonsai 27B plain decode +1.8-2.0%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 31ae86e on Torchit1/llama.cpp arc-b580) * sycl: fold FFN SwiGLU and the DeltaNet pre-norm into the T2 rotation SWIGLU -> MUL(signs) -> RESHAPE -> Hadamard MUL_MAT: the rotate/quantize kernel computes silu(gate) * up on its load (GGML_SYCL_GLU_ROT). ADD + RMS_NORM + MUL whose rotation comes ~9 nodes later (DeltaNet layers): the rotation is computed with the norm when no node in between touches its output (GGML_SYCL_NORM_ROT_FAR). Same arithmetic, byte-identical outputs. B580 Bonsai 27B plain +0.3-0.8%, MTP rename +2%, edit +1.1%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit cf4b155 on Torchit1/llama.cpp arc-b580) * sycl: per-batch-size GemmMT tile for T2 (small tile up to 32 rows); GEMV_MAX knob arctune sweep on the B580: tile {8,128,2,4} is 7-11% faster per pass for 12-32 rows (n-gram verify batches), the default {8,128,8,2} from 48 rows up. GGML_SYCL_PTQ1_T2_TILE now forces one tile for all sizes; GGML_SYCL_PTQ1_T2_TILE_SMALL_MAX (default 32) picks the crossover. GGML_SYCL_PTQ1_T2_GEMV_MAX (default 8) makes the GEMV/GemmMT switch a knob; the sweep confirmed 8. Outputs and KLD identical; MTP rename/edit +0.5%, 48K copy +1.3%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a70fe64 on Torchit1/llama.cpp arc-b580) * sycl: decode attention prefetches the next phase's key rows (GGML_SYCL_FA_DEC_PF) Phase probes (GGML_SYCL_FA_DEC_DPAS_PROBE) showed Q.K and P.V serialised behind the K / V loads. Each tile now prefetches its V rows before Q.K and the next tile's K rows before P.V (sycl_ext_oneapi_prefetch, L1). Per layer at 1 token: 48K 263 -> 226 us, 115K 554 -> 496 us. Whole model: plain 48K +2.7%, 115K +4.4%; MTP 48K +4.2%, 115K +7.5%; outputs identical. FA Bonsai cases 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit dc575f4 on Torchit1/llama.cpp arc-b580) * tests: arctune fusion tests for the SYCL fusions; fusion guard fixes test-backend-ops ARCLAB_GLU_ROT / ADD_NORM_ROT / CONV_CHAIN / DN_GATES / T2_MULTI build the exact node pattern each fusion matches, at Bonsai 27B shapes and 1-16 tokens, 3 seeds each, and compare every fused output with the CPU backend: 75/75 pass, each fusion confirmed firing (GGML_SYCL_FUSION_LOG, GGML_SYCL_T2_MULTI_LOG). Found while writing them: the far norm->rotation fusion now requires the signs to be a weight (they are read early); the conv chain no longer refuses a later reader of the cache slot; the same-input merge repacks weights on first use, so it also merges on the first graph. Whole model unchanged (plain short / 48K / 115K md5s identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bfb2d76 on Torchit1/llama.cpp arc-b580) Conflict resolved against prism: test-backend-ops keeps prism's fused SwiGLU/sign-flip FWHT cases and adds these. * ptq1-t2: GGML_SYCL_PTQ1_T2_TUNED per-shape GEMV tile table (plumbing, empty table) ARC-LAB: GGML_SYCL_PTQ1_T2_TUNED=1 (default 0) looks up a (N, K, rows, merged) row in t2_tuned_table() for every 1..8-row T2 GEMV (t2_dispatch and ggml_sycl_t2_mul_mat_multi) and launches TernSYCL's Gemv<SGM, NSG, LS, U> (TernSYCL, BSD 3-Clause) under a GRF mode and work-group order through GemvTuned (= gsw::Sweep of /lab/exp/gemv-sweep). Tiles are compiled from the T2_TUNED_KERNELS X-macro; t2_precompile warms them when the knob is on. Table and kernel list are empty until the B580 sweep results are in, so =1 changes nothing. =2 is a plumbing self-test: the default tiles through the tuned launcher (outputs must stay identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a0d82ad on Torchit1/llama.cpp arc-b580) * ptq1-t2: GGML_SYCL_PTQ1_T2_TUNED table from the B580 GEMV sweep (bit-exact tiles) ARC-LAB: five rows from /lab/exp/gemv-sweep (JIT build, medians of 5 interleaved rounds), each with the same LS as the default tile, so the outputs are bit-identical: ffn_down n=5 0.775, gate+up n=5 0.896, output head n=5 0.882, q+k+v n=5 0.923, q+k+v n=1 0.974 (time vs the default tile). Tiles are TernSYCL's Gemv (BSD 3-Clause) under a different launch geometry and work-group order. Batch sizes 2-4 and other shapes keep the default tiles. Knob still defaults to 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4bd894b on Torchit1/llama.cpp arc-b580) * ptq1-t2: tuned GEMV tiles on by default on a B580 (GGML_SYCL_PTQ1_T2_TUNED=0 turns them off) B580 A/B (sd2 head): MTP-only +2.9/+3.1%; live MTP + n-gram short +4.4%, 48K +3.1%, 115K flat; outputs identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0729c9d on Torchit1/llama.cpp arc-b580) * sycl: default to the legacy Level Zero adapter unless SYCL_UR_USE_LEVEL_ZERO_V2 is set Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5336e12 on Torchit1/llama.cpp arc-b580) * sycl: tuned Q6_K mat-vec and fused Q6_K gate+up+SWIGLU for the MTP draft head (knobs, off) The draft head's Q6_K mat-vecs are ~60% of a draft step and read at ~70% of the B580's bandwidth (op timing, 2026-10-10). - GGML_SYCL_Q6K_GEMV=1: a single-column reordered Q6_K mat-vec with the original per-row arithmetic (same lane -> quant-int mapping, blocks in order, same vec_dot expression accumulated in the same order, same sub-group reduction) but U blocks' operands loaded before use, aligned 32-bit ql/qh loads, and GGML_SYCL_Q6K_SG sub-groups per work-group. Sweep knobs GGML_SYCL_Q6K_U (1/2/4, default 2) and GGML_SYCL_Q6K_SG (4/8/16, default 8). GGML_SYCL_Q6K_CHECK=1 reruns the original kernel per launch and counts rows that differ in any bit (stderr), the in-server identity check. - GGML_SYCL_Q6K_GLU=1: Q6_K joins the reorder gate+up+GLU fusion (1..8 columns), whose dot products are the reorder mat-vec's and whose epilogue is the SWIGLU kernel's silu(gate) * up. - test-backend-ops: Q6_K mat-vec and gate+up fusion cases at the head's shapes (eval: 1, 2, 6 tokens; perf: 1..5 tokens). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6dd4481 on Torchit1/llama.cpp arc-b580) * sycl fattn-dec: transposed 2D block loads for the DPAS kernel's lane = key reads (GGML_SYCL_FA_DEC_B2D, default off) xe2dp::ld2d_t32_8x16 (lsc_load_block2d d32.8x16tn): one message gives each lane 8 dwords of its own key row, replacing the per-lane dword gathers in Q.K (5 messages per 16 keys instead of 40) and P.V (1 instead of 5). Same bytes and math, 89/89 FLASH_ATTN_EXT cases. Engages only on 64-byte aligned K/V surfaces (logged once). Slower on its own at 128K (register spills): 5 tokens 1069 -> 1220 us, 16 tokens 2945 -> 3610 us. Kernel launch now goes through a named lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit fb75ee9 on Torchit1/llama.cpp arc-b580) * sycl fattn-dec: 256-register file for the DPAS decode kernel at 3+ query tokens (GGML_SYCL_FA_DEC_GRF, default 256) The 3+ token launch is one wave of 2 work-groups x 16 subgroups per Xe core = 32 threads, which is what the 256-register mode allows, so it costs no occupancy and removes the spills. 1-2 tokens keep the default register file (their wave puts 4 work-groups on a core). GGML_SYCL_FA_DEC_GRF=0 = off. B580, us per layer at 128K (real cache layout) 5 / 8 / 12 / 16 tokens: 1073 / 1434 / 2920 / 2939 -> 1004 / 1240 / 2465 / 2445; 1 token unchanged. Whole model (arctune x3, MTP + n-gram, sd3 head): 48K 106.1 -> 108.2 t/s (+2.0%), 115K 86.8 -> 90.6 (+4.4%), output md5 identical (d09ea3 / 291b0b). FLASH_ATTN_EXT q4_0: 480/480. Perf cases: kv 16K/64K/128K x 1/5/8/12/16 tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1eb2307 on Torchit1/llama.cpp arc-b580) Also carries the FLASH_ATTN_EXT perf cases this commit widens, first added by 1fa26f1 (Jesse Symons), which is not picked separately. * sycl: one in-place layout owner for PQ2_0/PTQ1_0 weights The port's MMVQ reorder (opt_for_reorder, also called by the fused gate/up path) and the PQ2_0 XMX rewrite (ggml-org#294) both change these weights in place, and neither checked the other's flag. With GGML_SYCL_PTQ1_T2 unset a weight could get both layouts, so 27B PTQ1_0 perplexity went to 139615 on the B390 while every op test passed. On 16-wide DPAS devices the XMX layout now owns PQ2_0/PTQ1_0 and the reorder skips them; the fused paths that need the reorder decline and fall back to the unfused XMX path. The XMX path also refuses a weight that is already reordered or T2-repacked. * sycl: drop PQ2_0/PTQ1_0 duplicates left by the arc-b580 merge Keep prism's dequantize, vec_dot and MMVQ dispatch; keep the port's reorder-aware convert/MMVQ cases. Prism (ggml-org#235/ggml-org#294) and the port both added PQ2_0/PTQ1_0 support, so the merged tree had duplicate dequant functions, switch cases and two MMVQ implementations with the same names and did not compile. Deletions only, found and built on the B390 (oneAPI 2025.3.2). * sycl: Xe3 defaults to the MMVQ reorder for PQ2_0/PTQ1_0 and to XMX decode attention Measured on an Arc B390 (Panther Lake) with GGML_SYCL_PTQ1_T2 unset: - PQ2_0/PTQ1_0 weights go to the MMVQ reorder layout instead of the XMX layout: decode +16%, prompt +6%, about 0.9 GB less memory; KLD vs prism 6.7e-4 (XMX 2.4e-4). GGML_SYCL_DISABLE_XMX=0 restores the XMX layout, =1 still forces the reorder on any device. - GGML_SYCL_FA_DEC_DPAS defaults to 1: decode at 16K context +4-6% with a q4_0 KV cache; FLASH_ATTN_EXT head size 256 264/264, KLD with and without it within run-to-run error. =0 turns it off. Other devices keep their previous defaults. --------- Co-authored-by: Jesse Symons <155509533+Torchit1@users.noreply.github.com>

…#294) * Adds an XMX (DPAS) path for PQ2_0 and PTQ1_0 matrix multiplies on Intel Xe2 GPUs (Battlemage, Lunar Lake). It covers every batch size: prompt processing, single-token decode, parallel decode and speculative verification. On Xe2 it replaces MMVQ and the dequantize-to-FP16 prompt path from ggml-org#278 for these types. The main idea: PQ2_0 packs value+1 in 2-bit fields, lowest first, which as a little-endian dword is already the DPAS 2-bit operand layout. The kernel feeds the codes to DPAS as signed 2-bit (s2) against int8 activations, converting them in registers with a borrow-free per-field subtract. Weights are never widened to 8 or 16 bits in memory. How it works: - **Weight layout.** On first use, each PQ2_0 weight is rewritten in place into an SoA layout (same size): all 32-byte qs blocks first, then all fp16 scales. The qs row pitch is then nb*32 bytes, so one transposed 2D block load fetches 16 rows x 128 weights directly in the DPAS B-operand layout. The 34-byte AoS blocks are only 2-byte aligned and can't be loaded that way. - **PTQ1_0.** Base-3 has no DPAS form, so PTQ1_0 weights are expanded once into the same 2-bit layout. That is 34 bytes a block instead of 28. On Xe2 devices the buffer type reserves the room for it (get_alloc_size), so VRAM holds only the expanded copy: about +25% for these weights, and the file on disk is unchanged. - **Activations.** They are quantized to int8 with one scale per 128 values, one PQ2_0 block. The four DPAS of a block then accumulate in int32 before a single float rescale, instead of one rescale per 32 values. - **One kernel for all batch sizes.** Tiles are 8/16/32 tokens x 32 rows. For small batches, up to 16 threads of a work-group split K for the same tile and reduce through SLM, so a mat-vec keeps enough loads in flight. Scope and fallbacks: - On by default on BMG-G21/G31 and LNL-M. `GGML_SYCL_DISABLE_XMX=1` or `GGML_SYCL_ENABLE_OPT=0` turns it off. - Only plain MUL_MAT on 2D weights in regular (non-split, non-COMPUTE) buffers with K >= 256. Everything else (op offload, split buffers, MUL_MAT_ID, other devices) takes the existing paths unchanged. - Like the existing reorders, a tensor keeps its new layout once rewritten. The gate refuses MUL_MAT_ID expert slices and views. `get_rows` asserts if it ever sees a rewritten PQ2_0/PTQ1_0 tensor. Files: `pq2_xe2.cpp/.hpp` (new), `ggml-sycl.cpp` (gate, alloc size, extras), `common.hpp` (layout flag), `getrows.cpp` (assert). Intel Arc Pro B50 (16 GB), oneAPI 2026.1, Level Zero, Linux `xe` driver, `-ngl 99`, Ternary Bonsai models. "ggml-org#278" is the same build with `GGML_SYCL_DISABLE_XMX=1`, i.e. the current prism code path. Both columns were measured in the same session. `llama-bench`, t/s: | model | test | ggml-org#278 | this PR | change | |---|---|---|---|---| | 1.7B PQ2_0 | pp128 | 2574 | 6133 | 2.38x | | | pp512 | 5218 | 7645 | 1.47x | | | tg64 | 152.4 | 178.1 | +17% | | 4B PQ2_0 | pp128 | 1068 | 3068 | 2.87x | | | pp512 | 2298 | 3272 | 1.42x | | | tg64 | 77.4 | 99.8 | +29% | | 27B PQ2_0 | pp128 | 148 | 346 | 2.33x | | | pp512 | 290 | 370 | 1.28x | | | tg64 | 12.39 | 16.53 | +33% | | 27B PTQ1_0 | pp128 | 127 | 348 | 2.74x | | | pp512 | 267 | 370 | 1.39x | | | tg64 | 12.88 | 16.54 | +28% | Parallel decode (`llama-batched-bench -npp 128 -ntg 64`, aggregate t/s): | sequences | 1.7B PQ2_0 | 4B PQ2_0 | 27B PQ2_0 | 27B PTQ1_0 | |---|---|---|---|---| | 1 | 151 -> 171 | 76 -> 97 | 12.2 -> 16.1 | 13.1 -> 16.1 | | 2 | 250 -> 334 | 125 -> 186 | 18.9 -> 29.7 | 20.2 -> 29.7 | | 4 | 380 -> 613 | 185 -> 355 | 25.2 -> 48.0 | 26.3 -> 47.9 | | 8 | 478 -> 1010 | 224 -> 618 | 29.0 -> 68.6 | 31.0 -> 68.0 | | 16 | 322 -> 1446 | 138 -> 900 | 20.2 -> 85.7 | 17.2 -> 85.8 | | 32 | 588 -> 1769 | 262 -> 1184 | 32.5 -> 98.4 | 29.0 -> 98.4 | On ggml-org#278, batches of 9+ tokens leave MMVQ for the dequantize-to-FP16 path, which is why throughput drops at 16 sequences there. Speculative decoding (27B PQ2_0 target, Qwen3.5-0.8B Q8_0 draft, `--spec-draft-n-max 1`, greedy, 256 tokens), t/s: | | ggml-org#278 | this PR | |---|---|---| | prose, target only | 13.1 | 17.6 | | prose, with draft | 12.1 | 20.5 | | code, target only | 12.7 | 17.5 | | code, with draft | 12.8 | 21.9 | With ggml-org#278, drafting does not pay. With this PR, verification is cheap enough that n_max=1 gains 17-25% over the target alone. Longer drafts lose on acceptance (15-33% at n_max 2-3 with this draft model). Where the decode gain comes from: at the 27B FFN shape (17408x5120, one token), the PQ2_0 mat-vec goes from 199 us (MMVQ, about 53% of the B50's 224 GB/s) to 88 us. End to end, 27B decode now reads about 123 GB/s. Most of the remainder is the ~2300 non-matmul kernels per token. - `test-backend-ops -b SYCL0`: MUL_MAT 1329/1329, MUL_MAT_ID 1035/1035, GET_ROWS 119/119 (all types). Full suite, run per op: 14703/14759. The 56 failures (CONV_2D, ROLL, FLASH_ATTN_EXT, LIGHTNING_INDEXER) are identical with this path disabled, so they predate this PR. SET_ROWS aborts on current prism regardless of this PR (`set_rows.cpp:549: Unsupported tensor type!`), so it was run separately. - Perplexity (`-c 512`, 5 chunks), ggml-org#278 -> this PR: | model | setting | ggml-org#278 | this PR | |---|---|---|---| | 1.7B PQ2_0 | b1 | 8.4089 | 8.4335 | | 1.7B PQ2_0 | b512 | 8.4233 | 8.4149 | | 4B PQ2_0 | b1 | 7.0584 | 7.0565 | | 4B PQ2_0 | b512 | 7.0545 | 7.0506 | | 1.7B PQ2_0 | -ngl 0 (op offload) | 8.4225 | 8.4225 | | 4B PQ2_0 | -ngl 20 (partial) | 7.0541 | 7.0535 | | 27B PTQ1_0 | b512 | 4.2421 | 4.2399 | - KL divergence against a CPU-only reference (1.7B, `-dev none --no-op-offload`), ggml-org#278 vs this PR: - b1: 0.00122 vs 0.00125. - b512: 0.00106 vs 0.00113. - Same top token: 97.4-97.8% vs 97.7-97.9%. - The differences are within the error bars. The per-128 activation scale costs nothing measurable. - This introduces ESIMD/XMX code to the SYCL backend, a new pattern here. I'm happy to adjust structure or naming to fit how the maintainers want it organized. - Tuning (tile sizes, K-split thresholds) was done on the B50 only. BMG-G31 and LNL-M are gated in by architecture but untested. - I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) - AI usage disclosure: YES - the code and this description were written by Claude (Anthropic) under my direction; I tested it on my own hardware (Intel Arc Pro B50) and reviewed the changes. * sycl: gate the PQ2_0/PTQ1_0 XMX path on 16-wide DPAS instead of a device list The path was limited to BMG-G21/G31 and LNL-M by name. It now runs on any XMX device whose int8 DPAS is 16 wide, as the runtime reports it through matrix_combinations: Xe-HPC, Xe2, Xe3 and later. 8-wide XMX (Xe-HPG, Arrow Lake-H) and devices without XMX keep the existing paths. AOT builds compile the kernels for every GGML_SYCL_DEVICE_ARCH device, and they do not build for targets without 16-wide DPAS and 2D block loads (ocloc crashes on acm-g10, and fails on tgllp, dg1, mtl and pvc-vg). pq2_xmx.cpp is now built only when every AOT device is Xe-HPC, Xe2 or later; otherwise it compiles to stubs. Renames pq2_xe2 to pq2_xmx to match.

tarruda requested a review from ggerganov March 19, 2023 02:50

tarruda mentioned this pull request Mar 19, 2023

Refactor most code in main.cpp into a separate module (preparing to implement TCP mode) #267

Closed

tarruda force-pushed the tcp_server branch 2 times, most recently from 8f1eeca to 4a79a33 Compare March 19, 2023 19:08

tarruda force-pushed the tcp_server branch from 4a79a33 to 3848f83 Compare March 19, 2023 20:10

tarruda force-pushed the tcp_server branch from 3848f83 to c663305 Compare March 19, 2023 20:58

maxsu reviewed Mar 19, 2023

View reviewed changes

Comment thread CMakeLists.txt Resolved

tarruda force-pushed the tcp_server branch 2 times, most recently from 1075f48 to 5e65c52 Compare March 19, 2023 21:19

tarruda force-pushed the tcp_server branch from 5e65c52 to 3ebf150 Compare March 19, 2023 23:51

tarruda mentioned this pull request Mar 20, 2023

Should use mmap for model loading #91

Closed

This was referenced Mar 20, 2023

High performance API #321

Closed

Ability for ./main to keep the model in memory and pass it more text #23

Closed

gjmulder added the enhancement New feature or request label Mar 20, 2023

tarruda force-pushed the tcp_server branch from 3ebf150 to b6fdbee Compare March 20, 2023 21:52

anzz1 mentioned this pull request Mar 20, 2023

MMAP for Windows (not working atm) #341

Closed

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

ckti pushed a commit to ckti/llama.cpp-consolidated that referenced this pull request Aug 22, 2026

Merge pull request ggml-org#278 from giveen/document-3

e30664a

docs : add harness init strided-view bug to coverage limits

Conversation

tarruda commented Mar 19, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

tarruda commented Mar 19, 2023

Uh oh!

tarruda commented Mar 19, 2023

Uh oh!

avilum commented Mar 19, 2023

Uh oh!

Uh oh!

tarruda commented Mar 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avideci commented Mar 19, 2023

Uh oh!

jart commented Mar 20, 2023

Uh oh!

anzz1 commented Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spirobel commented Mar 20, 2023

Uh oh!

tarruda commented Mar 20, 2023

Uh oh!

spirobel commented Mar 20, 2023

Uh oh!

tarruda commented Mar 20, 2023

Uh oh!

tarruda commented Mar 20, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

tarruda commented Mar 19, 2023 •

edited

Loading

anzz1 commented Mar 20, 2023 •

edited

Loading