Repository navigation
Report: Ternary kernels work on Volta (sm_70) — token-identical with sm_89, ~50 t/s on a V100 #240
Description
Not a bug — a datapoint for an architecture outside the tested set (75-virtual;86-real;89-real;120-real). Since the docs describe missing architectures as "untested, not merely slower," here is a tested one.
Result
Ternary-Bonsai-2-27B (PQ2_0) runs correctly on a Tesla V100 (compute 7.0) built from source against CUDA 12.9. Greedy (temp 0, 600 tokens) output is token-identical to the same build running on an RTX 4070 (compute 8.9, inside the tested set) — diff of the two outputs differs only in the loading spinner and the perf banner. Reasoning trace and final answer both coherent.
| GPU | compute cap | prompt eval | generation |
|---|---|---|---|
| Tesla V100-PCIE-32GB | 7.0 | 132.5 t/s | 50.5 t/s |
| RTX 4070 (control) | 8.9 | 358.2 t/s | 52.9 t/s |
Generation speed is effectively at parity (memory-bound; HBM2 holds up). Prefill is ~2.7x faster on Ada.
Environment
- Fork:
prismbranch, commit3ae4f51(version 0.2.0-dev, build 1) - Model:
prism-ml/Ternary-Bonsai-2-27B-gguf—Ternary-Bonsai-2-27B-PQ2_0.gguf(7.2 GB) - GPU: Tesla V100-PCIE-32GB (Volta, sm_70), driver 580.173.02
- CUDA toolkit 12.9.86 (nvcc off PATH; passed explicitly — see below). Note this route dies with CUDA 13, which dropped Volta support.
- gcc 13.3.0, cmake 3.28.3, Ubuntu 24.04, host: i7-13700KF / 15 GB RAM
Build
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
-DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.9/bin/nvcc \
-DCMAKE_CUDA_ARCHITECTURES="70;89"
cmake --build build -j 4
Compiled clean — no sm_70-related warnings or errors. Load reports loaded 402 Hadamard-folded weight(s) (1 inverse-lookup) using 1 rotation(s) and 3 sign vector(s); all 65/65 layers offloaded; model buffer 6539.67 MiB.
Test
CUDA_VISIBLE_DEVICES=<n> ./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
-ngl 99 --temp 0 -n 600 --single-turn \
-p "List the first five prime numbers, then explain in one sentence why 9 is not prime."
Ran once per card; diffed the outputs. Correct answer on both (2, 3, 5, 7, 11 + a correct sentence about 9 = 3 × 3).
Happy to run additional validation (perplexity run, longer contexts, or PTQ1_0 — untested here) if useful. Filed in case it saves the next person with a Volta card the guesswork; also fine to close if sm_70 remains officially unsupported.
Disclosure: testing and this writeup were done with the help of an AI assistant (Claude); all figures are taken directly from the run logs on the machine described above.
Activity
christopherrobertbrooks-tech commentedon Sep 22, 2026
Follow-up on the same machine, taking up the offer above: PTQ1_0 is now tested, and so is the vision tower. Same box, same binary, same build (3ae4f51, CUDA 12.9,
-DCMAKE_CUDA_ARCHITECTURES="70;89"), one card visible at a time.Three things: a Volta row for the throughput table, a counterexample to the "prompt processing favors PQ2_0 everywhere" claim, and confirmation that the mmproj works on sm_70.
First, a correction to my own numbers above. The 132.5 / 358.2 t/s prefill figures in the original post are not comparable to the README's table — they came from
llama-clion a ~20-token prompt, which is far too short to saturate either card. Measured properly withllama-bench, V100 prefill is 766 t/s, not 132. Please disregard the prompt-eval column in the opening post.Throughput
llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 5, mean ± stddev:PP512 on Volta favours PTQ1_0, not PQ2_0
The model card states that prompt processing, "being compute-bound, favors PQ2_0 everywhere." On the V100 that inverts: PTQ1_0 is ~6.6% faster at pp512 (817.3 vs 766.6), with non-overlapping error bars. Reproduced with the model order swapped and
-r 8: PTQ1_0 814.9 ± 4.8 vs PQ2_0 724.2 ± 30.2.The 4070 in the same chassis, same binary, same invocation behaves exactly as documented — PQ2_0 wins pp512 by 2.05x, and PTQ1_0 edges decode. So this is not a build or harness artifact; the two architectures genuinely disagree.
Volta is the worst case so far for PTQ1_0 decode
PTQ1_0 reaches 67.5% of PQ2_0 decode on the V100, below the A100's 74.0% (the lowest ratio currently in the table). That is consistent with the README's explanation — trit-unpacking arithmetic loses wherever batch-1 decode is not bandwidth-starved — and extends it to a generation the table does not yet cover. On this hardware PQ2_0 is the clear pick for interactive use despite the larger footprint.
Vision works on sm_70
llama-mtmd-cliwithTernary-Bonsai-2-27B-mmproj-Q8_0.ggufalongside PQ2_0, both fully offloaded to the V100:--image-min-tokens 1024: every cell correct, including leading zeros (07,04,00), plus both values buried in the small print.Image encode took 158 ms and 394 ms respectively. No sm_70-specific warnings from the vision path.
Minor
chat_template_kwargs: {"reasoning_effort": "none"}raises a Jinjaraise_exceptionand surfaces as an HTTP 500 fromllama-server. The template acceptsxhigh/medium/low, so this is expected rather than a bug — noting it only because "none" is a common spelling elsewhere and the failure is a hard 500 rather than a fallback to the default.Environment otherwise as described in the opening post. Happy to run perplexity or long-context numbers on either card if that would be useful.
Disclosure: testing and this writeup were done with the help of an AI assistant (Claude); all figures are taken directly from the run logs on the machine described above.