Repository navigation
fix: skip layers the AWQ GEMM kernel cannot serve in AWQ export - #2456
Merged
Merged
Conversation
Signed-off-by: losewayy <losemywayy@gmail.com>
losewayy
force-pushed
the
fix-awq-unservable-layers
branch
from
October 10, 2026 01:37
b5e1ecc to
8fc0b4a
Compare
Contributor
Author
|
/azp run Unit-Test-CUDA-AutoRound |
|
Commenter does not have sufficient privileges for PR 2456 in repo intel/auto-round |
wenhuach21
reviewed
Oct 10, 2026
…hapes Layers matched by a user-supplied layer_config entry (exact name or expanded regex) keep their quantization setting instead of being forced to fp16. They are flagged on the module so the AWQ packers attempt packing, and a warning notes the shape may fail on the AWQ GEMM path in serving stacks. Layers that only fall back to the default scheme are still exported in fp16. Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
wenhuach21
reviewed
Oct 10, 2026
Collaborator
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…attribute apply_plan_to_model already writes fixed_by_user onto quantized modules, so a shared awq_layer_user_forced helper reads it directly, falling back to the layer_config entry for modules lacking the flag (e.g. lm_head added via quant_lm_head whose entry carries no fixed_by_user key). The marking pass, the export prescan, and pack_layer on both AWQ paths share the same rule, and nothing is written onto modules during export. Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
wenhuach21
approved these changes
Oct 10, 2026
wenhuach21
left a comment
wenhuach21
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me. Thanks for the PR!
Collaborator
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
AWQ export could pack layers whose shapes the AWQ GEMM kernel cannot serve. When
in_featuresorout_featuresis not divisible bygroup_size, orout_featuresis not a multiple of the kernel's 64-wide CTA tile, the exported model either crashes vLLM at load time (OC is not multiple of Group size) or silently leaves the layer unquantized so the saved model grows instead of shrinking. This hits the small Gated-DeltaNet projections in Qwen3.5 hybrids (e.g.in_proj_ba, OC=64 with group_size=128) and the lm_head of some models.This PR adds one shared predicate,
awq_gemm_kernel_supported, and applies it at three points so unservable layers stay in fp16 and are listed inmodules_to_not_convert:bits=16, data_type=fp) before quantization, so they are excluded from tuning and packing. The marking is re-applied after layer_config expansion inapply_layer_config_special_cases, since regex user entries share one dict across matches and could overwrite the mark.save_quantized_as_autoawqpre-scans and skips unservable layers and merges them intomodules_to_not_convert, which covers thequantize()thensave_quantized(format="auto_awq")split flow where format resolution happens after quantization.pack_layeralso bails out on such shapes, and the same guard is applied to theauto_round:auto_awqpacking path.check_awq_gemm_compatibilitynow delegates to the same predicate, which also closes the missingout_features % 64rule and fixes the stale "only supports 4 bits" message.Type of Change
Bug fix
Related Issues
Fixes #1496, relates to #1650 and #2427
Checklist Before Submitting
/azp run Unit-Test-CUDA-AutoRound.