[Model request] KAT-Coder-V2.5-Dev NVFP4 community exports (compressed-tensors MoE): dense-branch misroute + GDN granularity #410

Activity

1. minatoAI

   added

   bugSomething isn't workingSomething isn't working

   on Sep 7, 2026
2. github-actions

   added

   windows

   on Sep 7, 2026
3. Romeriz commented on Sep 8, 2026

   

   Romeriz

   on Sep 8, 2026

   More actions

   Hi, I have a working, hardware-tested patch for this exact family and I don't want to
   duplicate you or the open PRs. I got AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4,
   doth4580/Kwaipilot-KAT-Coder-V2.5-Dev-NVFP4-MIXED and
   Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 to convert and serve cleanly.

   The layouts it handles that are distinct from PR #390 (mixed FP8-attn + NVFP4 experts):

   • per-expert llm-compressor experts (...experts.E.{gate,up,down}_proj.weight_packed,
     one global per expert) -- AEON and Ttimms;
   • stacked per-layer experts (...experts.gate_up_proj / .down_proj as one
     [E*rows, cols] tensor + a per-layer scalar global) with a dedicated bank loader --
     Kwaipilot;
   • format-only configs (no config_groups) detected via
     format: nvfp4-pack-quantized;
   • GDN granularity: ignore-listed GDN stays bf16 (AEON / Ttimms / your sahilchachra
     case) and a gdn:fp8 recipe GDN is dequantized to bf16 (Kwaipilot), while
     full-attention projections stay NVFP4 -- resolved with a probe, matching your point 2;
   • NVFP4 embed_tokens dequantized (no NVFP4 embedding kernel) and lm_head kept
     native NVFP4 when the export quantized it (Kwaipilot);
   • single-file no-index checkpoints (AEON is one 23 GB model.safetensors) -- the
     reciprocal global is read from the header, which PR fix(models): better support for mixed-precision compressed-tensors NVFP4  #390's index-only probe misses.

   It's a one-branch change (fix(models): support llm-compressor NVFP4 MoE export variants) with weight-free unit tests.

   Since you said you'd shape your local change into a PR and PR #390 is open in the same
   files, I'd like to coordinate rather than race: are you still planning to open it? My
   per-expert plumbing and #390's FP8-attn routing touch the same
   iter_weights/_CT_NVFP4_FUSE/_expert_quant spots, so the cleanest is to reconcile
   them (ideally a per-variant dense dispatch: .weight_packed NVFP4 dense MoE CT -> the
   CT reader, FP8-config_groups MoE CT -> the fp8 reader). Happy to rebase mine on
   whichever lands first and drop the overlap.

   Verified locally (CPU, weight-free):

   • AEON config -> expert_quant/attn_quant/dense_quant=nvfp4, gdn_quant=none, lm_head_quant=none; its dense read matches the model state dict exactly (663 tensors,
     0 missing/extra) and is unchanged from before this patch.
   • Kwaipilot config -> expert_quant/attn_quant/dense_quant=nvfp4, gdn_quant=none, lm_head_quant=nvfp4; dense read matches the model state dict exactly (665 tensors,
     0 missing/extra).
   • On the GPU box all three ft checkpoint + ft serve successfully (AEON FTW per my
     log: 663 weight + 240 experts_bank, 20.96 GiB). Happy to paste exact hardware and
     commands in the PR.
4. minatoAI commented on Sep 8, 2026

   

   minatoAI

   on Sep 8, 2026

   Author

   More actions

   Thanks @Romeriz — I won't open a competing PR.

   #413 matches both root causes in this issue: MoE compressed-tensors routing to
   expert_quant=nvfp4, and separate GDN granularity with probe-level
   ignore/recipe handling. #390 looks complementary rather than a replacement for
   these two checkpoints.

   My exact checkpoints remain:

   • sakamakismile/KAT-Coder-V2.5-Dev-NVFP4
   • sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16

   My locally validated results were saka 723 weight + 240 experts_bank, 20.62 GiB
   and sahil 663 weight + 240 experts_bank, 20.96 GiB, both convert + serve clean.
   If useful, I can rerun the #413 branch against those two exact checkpoints and
   post logs/counts for Windows/Desktop verification.
5. Romeriz commented on Sep 8, 2026

   

   Romeriz

   on Sep 8, 2026

   More actions

   @minatoAI - thanks, I'd like to take you up on the verification offer. Please run the current #413 head against your two checkpoints:

   • branch Romeriz:fix/models-ct-nvfp4-moe-variants, commit f8d6a56 (base af71ba4)
   • ft checkpoint --model <dir> --out <ftw> then ft serve --model-path <ftw>

   If you can post the convert/serve logs plus your tensor / FTW counts for sakamakismile/KAT-Coder-V2.5-Dev-NVFP4 and sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16, that independent Windows/Desktop run goes straight into #413 as the "tested on real hardware" evidence the maintainers ask for.

   Frame of reference: your earlier local numbers were saka 723 weight + 240 experts_bank / 20.62 GiB and sahil 663 + 240 / 20.96 GiB. If #413 lands on the same counts, that is the confirmation; if they differ, that is exactly the finding worth surfacing - please post either way.
6. minatoAI commented on Sep 9, 2026

   

   minatoAI

   on Sep 9, 2026

   Last edited by minatoAI

   Author

   More actions

   @Romeriz

   Verified on Romeriz:fix/models-ct-nvfp4-moe-variants @ f8d6a56 (isolated worktree, detached, clean; base af71ba4). Both KAT-Coder-V2.5-Dev NVFP4 variants, convert + serve on real hardware.

   Hardware/software: RTX 5070 Ti 16 GB (Blackwell), 32 GB RAM, Windows 11 native (WDDM), driver 610.62 / CUDA UMD 13.3, torch 2.11.0+cu130, Python 3.12, FreeToken 0.1.2.

   Convert (ft checkpoint --model <src> --out <ftw>):

   checkpoint	parse	FTW
   sakamakismile/KAT-Coder-V2.5-Dev-NVFP4 (uniform NVFP4, single model.safetensors)	expert/attn/gdn/dense=nvfp4, lm_head=none	723 weight + 240 experts_bank, 20.62 GiB, fp af06eb89b89e6efa
   sahilchachra/KAT-Coder-V2.5-Dev-NVFP4A16 (A16, 13 shards)	gdn=none, rest same	663 weight + 240 experts_bank, 20.96 GiB, fp ab30602b5261b8ce

   Serve (ft serve --model-path <ftw>): both boot the same way - split residency 14.39 GiB pinned (cudaHostRegister, 34 GPU layers) + 2.54 GiB pageable (6 CPU layers: [0, 1, 2, 37, 38, 39]), CPU MoE executor fmt=nvfp4 ... act=silu, CUDA graphs captured, /health = serving, and POST generations are coherent and stop-terminated (sahil: Say OK -> OK, finish_reason=stop). The AssertionError (saka) and KeyError (sahil) from #410 are both gone. Same counts -> confirmation.

   Serve env (test-harness only, not part of the PR): FREETOKEN_PIN_BUDGET_GB=14.4, FREETOKEN_ALLOW_CUDA_MISMATCH=1 (nvcc 12.8 vs torch cu130; prebuilt kernel-cache hit after that), one FREETOKEN_ZMQ_BASE per run, plus Windows-only sitecustomize shims (PR tree pristine): posix_fadvise no-op, madvise guard, mmap prot->access translation, kernel-cache .dll probe, ipc://->TCP ZMQ map, SelectorEventLoop.

   Windows findings from this run (happy to file separately, say the word):

   1. Convert drop_page_cache uses os.posix_fadvise -> crash on Windows.
   2. HostBank.release calls madvise -> crash on Windows.
   3. FTW reader uses mmap.PROT_READ -> crash on Windows.
   4. Those no-ops also mean page-cache bloat, which aggravates pin pressure on 32 GB boxes.
   5. _pin_budget_bytes misses the native-Windows WDDM cap (~half of RAM; 15.6 GB shared limit here): pinning ~15.2 GiB OOMs in cudaHostRegister while 14.39 GiB works. Worth detecting non-WSL Windows too - this one bites every big-MoE Windows user.
   6. Kernel-cache loader only probes {name}.so, ignores the shipped {name}.dlls on Windows.
   7. Bank mlock path imports resource (POSIX-only) -> CPU-locked layers silently fall back to pageable (warning only, non-fatal).
   8. Minor: the pin-failure message reports one bank's size (failed for 0.0 GiB), hiding quota exhaustion; cumulative-pinned + cap would debug faster.

   Full convert/serve logs available on request.