Repository navigation
[Issue]: fail to load Qwen3.8-mtp:27b, flm version 1.0.7 #767
Description
Problem Description
$ flm run qwen3.8-mtp:27b -c 16384 --pmode turbo
[Warning] Version check timed out; continuing without update info.
[FLM] Qwen3.8 pre-resize image height to 720 pixels if larger than that
[FLM] Loading model: /home/tang/.config/flm/models/Qwen3.8-27B-NPU2
[ERROR] Failed to load model: attn_layout: max_len 16512 puts the deepest pass's replay at 1032 chunks, past the 1024 its 10-bit BD dimension can walk. max_len is at most 16384 (MAX_L_LIMIT).
flm 1.0.6 works, but not 1.0.7.
Operating System
NAME="Debian GNU/Linux" VERSION="13 (trixie)"
CPU
model name : AMD Ryzen AI 9 365 w/ Radeon 880M
GPU
AMD Ryzen AI 9 365 w/ Radeon 880M
ROCm Version
not installed.
Installation Method
Built from source
Installed ROCm Packages / Versions
Installed ROCm packages / versions
Paste output here
ROCm Component
No response
Steps to Reproduce
No response
(Optional for Linux users) Output of rocminfo --support
rocminfo --support output
Paste output here
Additional Information
No response
Activity
sunjizheng commentedon Oct 6, 2026
The maximum context length is 8192.
tangmeng09 commentedon Oct 6, 2026
Thanks for the reminder. I was planning to use it to review my code all night. The context length might not be sufficient.
Atomic-Germ commentedon Oct 7, 2026
Wait, 8192!? Is that arbitrary or does the model itself have no utility?
aahlborg commentedon Oct 7, 2026
I guess there is a technical reason for why the model is limited to 8k. I'd like to know why, though.
BoneHorror commentedon Oct 7, 2026
It's just so slow on the NPU it would take hours to get this context filled up anyway
aahlborg commentedon Oct 7, 2026
That's not an issue for overnight code review or background work. Everyone's use case is different. For coding, most of the context is input tokens anyway.
Atomic-Germ commentedon Oct 7, 2026
@aahlborg Fair enough
b3y0urs3lf commentedon Oct 10, 2026
Reproduced on 1.0.7, independent hardware — and the effective ceiling has collapsed to 8192
Confirming this regression on a different machine.
Environment
Repro (same error as OP)
$ flm serve qwen3.8-mtp:27b --ctx-len 16384
[ERROR] Failed to load model: attn_layout: max_len 16512 puts the deepest pass's replay
at 1032 chunks, past the 1024 its 10-bit BD dimension can walk.
max_len is at most 16384 (MAX_L_LIMIT).
Context-length sweep — only 8192 loads
Every value in (8192, 16384] fails identically with max_len 16512, i.e. the engine rounds the request up to the next power-of-two bucket (16384) and then adds ~128 of KV overhead → 16512 →
1032 chunks > the 1024 the 10-bit BD can address. There is no loadable bucket between 8192 and 16384, so in practice the maximum usable context on 1.0.7 is 8192, not the advertised 16384.
Note this means the declared MAX_L_LIMIT = 16384 is itself unreachable: 16384 + 128 overhead already exceeds 1024 chunks. The previous bucket, 8192, is the real ceiling.
Where the limit lives (for triage)
Tracing the open-source path, nothing in the runtime clamps this:
context.context_length.
The rounding + the MAX_L_LIMIT / attn_layout / "10-bit BD" / "deepest pass" logic all live inside the prebuilt libqwen3_8mtp_npu.so + the Qwen3.8-27B-NPU2/*.xclbin kernels (confirmed by
strings), so it can't be worked around from config or flags. config.json has no rope_scaling / sliding_window / context field (max_position_embeddings is 262144 but decoupled from the NPU
bucketing).
Since 1.0.6 reportedly loaded -c 16384, this looks like a regression in the attention-layout sizing for the MTP engine introduced in 1.0.7 — would be great to know whether 16384 can be
restored (even just the 16384 bucket), or whether a finer, non-power-of-two bucket is feasible.
tangmeng09 commentedon Oct 11, 2026
Maybe I ignored the context sized on 1.0.6. This might not be a regression problem.
$flm run qwen3.8-mtp:27b
[FLM] Qwen3.8 pre-resize image height to 720 pixels if larger than that
[FLM] Loading model: /home/tang/.config/flm/models/Qwen3.8-27B-NPU2
[FLM] Loading model: qwen3.8-mtp:27b
[FLM] Type /? for help
/show
Model:
model_name: Qwen3.8-27B-NPU2
compatible_flm_version: >= 1.0.7
head_dim: 256
hidden_size: 5120
hidden_act: silu
intermediate_size: 17408
num_attention_heads: 24
num_hidden_layers: 64
num_key_value_heads: 4
pretraining_tp: 0
rms_norm_eps: 1e-06
max context length : 4096
Compare to Qwen3.6-35B-A3B
/load qwen3.6-moe:35b-a3b
[FLM] Qwen3.6 pre-resize image height to 720 pixels if larger than that
[FLM] Loading model: /home/tang/.config/flm/models/Qwen3.6-35B-A3B-NPU2
/show
Model:
model_name: Qwen3.6-35B-A3B-NPU2
compatible_flm_version: >= 1.0.3
head_dim: 256
hidden_size: 2048
hidden_act: silu
intermediate_size: 512
num_attention_heads: 16
num_hidden_layers: 40
num_key_value_heads: 2
pretraining_tp: 0
rms_norm_eps: 1e-06
Vision:
vision_model_weight: /home/tang/.config/flm/models/Qwen3.6-35B-A3B-NPU2/vision_weight.q4nx
max context length : 32768
The actual context size may be double of the show context length. Qwen3.6-35B-A3B runs happily with -c 65536 in my case.
r34s commentedon Oct 16, 2026
Same issue here. Everything beyond 8192 context does not load. I've got 64GB or RAM so that is not the issue. For some experiments I would like to try to go beyond the 64k of context, as supported by the model.
Environment
What happens
flm serve qwen3.8-mtp:27b --ctx-len N loads for N ≤ 8192. Any N above that fails to load:
[ERROR] Failed to load model: attn_layout: max_len 16512 puts the deepest pass's replay at 1032 chunks, past the 1024 its 10-bit BD dimension can walk. max_len is at most 16384 (MAX_L_LIMIT).