Repository navigation
Question about leading dimension handling for quantization scales in cuBLASLt matmul #305
Description
I have a question regarding how to handle leading dimensions for quantization scales in cuBLASLtMatmul when the input matrices are not tightly packed.
Suppose the leading dimensions of matrices A and B are larger than their logical sizes (M, N, K), i.e., they are strided / padded layouts rather than contiguous M×K or K×N.
For quantized matmul using:
A_scale = CUBLASLT_MATMUL_MATRIX_SCALE_VEC128_32F
B_scale = CUBLASLT_MATMUL_MATRIX_SCALE_BLK128x128_32F
How should the leading dimensions of the corresponding scale tensors be handled?
Specifically:
1.Are scale tensors always expected to be contiguous in memory?
2.Or do scale tensors also support their own leading dimensions / strides similar to matrices A and B?
3.If A or B has padding due to larger leading dimensions, should the scales reflect the padded layout or only the logical tile layout?
I couldn’t find clear documentation on how scale tensor strides relate to matrix strides, so any clarification would be appreciated.
Activity
rsdubtso commentedon Feb 18, 2026
Hi @Haoyan-Ma. Thanks for the question and apologies for a delayed response.
Currently — yes, the scales are expected to be contiguous in memory.
No, there is no support for leading dimension.
Not completely sure if I understood the question. Scaling factor dimensions are computed based on M, N and K matmul arguments and do not take leading dimensions of A and B into account.
Let me k know if you have any additional questions.
Haoyan-Ma commentedon Feb 24, 2026
Thank you for the clarification and no worries about the delay.