Skip to content

ggml : fix odd blocks for ARM_NEON #8556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ggerganov merged 10 commits into from
Jul 19, 2024
Merged

ggml : fix odd blocks for ARM_NEON #8556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
90e8f81
ggml : fix iq4_nl dot product with odd number of blocks
slaren Jul 17, 2024
e5e7a24
ggml : fix q4_1
ggerganov Jul 18, 2024
67b079f
ggml : fix q5_0
ggerganov Jul 18, 2024
f6f2ff9
ggml : fix q5_1
ggerganov Jul 18, 2024
3f68842
ggml : fix iq4_nl metal
ggerganov Jul 18, 2024
79b95e3
ggml : fix q4_0
ggerganov Jul 18, 2024
62a3185
ggml : fix q8_0
ggerganov Jul 18, 2024
974410a
ggml : remove special Q4_0 code for first 2 blocks
ggerganov Jul 18, 2024
32e9c41
ggml : fix sumf redefinition
ggerganov Jul 19, 2024
5ff83bd
Merge branch 'sl/fix-iqnl-odd-blocks' into gg/fix-odd-blocks-arm
ggerganov Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions ggml/src/ggml-metal.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1786,10 +1786,6 @@ static enum ggml_status ggml_metal_graph_compute(
}
};

if (ggml_is_quantized(src0t)) {
GGML_ASSERT(ne00 >= nth0*nth1);
}

[encoder setComputePipelineState:pipeline];
[encoder setBuffer:id_src0 offset:offs_src0 atIndex:0];
[encoder setBuffer:id_src1 offset:offs_src1 atIndex:1];
Expand Down
4 changes: 2 additions & 2 deletions ggml/src/ggml-metal.metal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4757,7 +4757,7 @@ void kernel_mul_mv_iq4_nl_f32_impl(
device const float4 * y4 = (device const float4 *)yb;
yl[0] = y4[0]; yl[1] = y4[4]; yl[2] = y4[1]; yl[3] = y4[5];

for (int row = 0; row < 2; ++row) {
for (int row = 0; row < 2 && first_row + row < ne01; ++row) {

device const block_iq4_nl & xb = x[row*nb + ib];
device const uint16_t * q4 = (device const uint16_t *)(xb.qs + 8*it);
Expand Down Expand Up @@ -4789,7 +4789,7 @@ void kernel_mul_mv_iq4_nl_f32_impl(
yb += 16 * QK4_NL;
}

for (int row = 0; row < 2; ++row) {
for (int row = 0; row < 2 && first_row + row < ne01; ++row) {
all_sum = simd_sum(sumf[row]);
if (tiisg == 0) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = all_sum;
Expand Down
Loading