Add oneliner for batch quantization

[jooray]

No description provided.

[prusnak]

Sed is not necessary, bash, zsh and other modern shells can perform pattern replacement of a variable:

Suggested change

	for i in models/65B/ggml-model-f16.bin*;do quantized=`echo "$i" | sed -e 's/f16/q4_0/'`; ./quantize "$i" "$quantized" 2 ;done
	for i in models/65B/ggml-model-f16.bin* ; do ./quantize "$i" "${i/f16/q4_0}" 2 ;done

[s-and-witch]

This will generate 'models/65B/ggml-model-q4_0/.bin.2' such paths and will fail with errors, the right command (in bash) should be for i in models/65B/ggml-model-f16.bin* ; do ./quantize "$i" "${i/f16/q4_0}" 2 ;done

[prusnak]

@Player-205 right, updated the suggestion above, thanks

[ggerganov]

Lets put this in a quantize.sh script that accepts argument like 7B, 13B, etc. and update instructions to just run the script:

source quantize.sh 7B

Should be much easier to follow

[leszekhanusz]

Note that if the disk space is limited, it is still useful to quantize each file separately so that we could delete each intermediate file in between.
In my case I added a rm command because I did not have enough disk space otherwise:

for i in models/65B/ggml-model-f16.bin* ; do ./quantize "$i" "${i/f16/q4_0}" 2 ; rm "$i"; done

[ggerganov]

Good point, should have a second parameter for "keep f16" which is on by default

[prusnak]

Superseded by #92