You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Groups of those 1-byte numbers are transformed into 32 bit integers called tokens. This process is called tokenization and not shown here. llama_eval accepts list of tokens. The code I shown demonstrates how each token is substituted with vector known as embedding. Data type of vector elements varies. It can be 32 bit float, it can be 16 bit float, it can be 4 bit quantized number. Theoretically vector components could be bits. It doesn't matter because there's thousands of them in one embedding, plenty to uniquely represent a token that in turn represents bunch of UTF-8 characters.
Ideally none on the side of behavior. Altered behavior is something we try to minimize by picking efficient quantization schemes. Effects that we want from quantization are smaller memory usage and faster speed. Difference between FP32 and 4 bit models is that first requires supercomputer and second runs on Raspberry Pi.
Yes... sadly this runs on CPU. Is there any way to do the same (ie. running int4 quantized 7B model) on GPU / NPU ? Not sure the standard models implementations allow that.
Modern phones typically have pretty good NPU / GPU compared to CPU. Could be interesting for say a on-device autocompletion keyboard.
This codebase is explicitly created to be run on CPU opposed to prevalent GPU implementations. Because everyone have CPU and RAM is upgradeable but not everyone own Nvidia hardware with sufficient VRAM.
Activity
jarcen commentedon Mar 17, 2023
No. Model doesn't operate on characters. Characters are lumped into tokens and tokens are substituted with vectors as shown here:
https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L579-L582
Napolitain commentedon Mar 17, 2023
my bad I think it uses int4 which is even surprising
Napolitain commentedon Mar 17, 2023
characters in C++ is a number within 1 byte
jarcen commentedon Mar 17, 2023
Groups of those 1-byte numbers are transformed into 32 bit integers called tokens. This process is called tokenization and not shown here.
llama_evalaccepts list of tokens. The code I shown demonstrates how each token is substituted with vector known as embedding. Data type of vector elements varies. It can be 32 bit float, it can be 16 bit float, it can be 4 bit quantized number. Theoretically vector components could be bits. It doesn't matter because there's thousands of them in one embedding, plenty to uniquely represent a token that in turn represents bunch of UTF-8 characters.Napolitain commentedon Mar 17, 2023
mhhh whats the effect of quantization ? less nuance in generation / understanding ?
jarcen commentedon Mar 17, 2023
Ideally none on the side of behavior. Altered behavior is something we try to minimize by picking efficient quantization schemes. Effects that we want from quantization are smaller memory usage and faster speed. Difference between FP32 and 4 bit models is that first requires supercomputer and second runs on Raspberry Pi.
Napolitain commentedon Mar 17, 2023
Yes... sadly this runs on CPU. Is there any way to do the same (ie. running int4 quantized 7B model) on GPU / NPU ? Not sure the standard models implementations allow that.
Modern phones typically have pretty good NPU / GPU compared to CPU. Could be interesting for say a on-device autocompletion keyboard.
jarcen commentedon Mar 17, 2023
This codebase is explicitly created to be run on CPU opposed to prevalent GPU implementations. Because everyone have CPU and RAM is upgradeable but not everyone own Nvidia hardware with sufficient VRAM.
GPU implementations:
Original Facebook repo:
https://github.com/facebookresearch/llama
Huggingface Transformers apparently support Llama judging by people converting models to hf format:
https://huggingface.co/decapoda-research
Getting down to 8-bit:
https://github.com/tloen/llama-int8
And 4 bit:
https://github.com/oobabooga/text-generation-webui/
There's lots of them, really.
fix format string vulnerability (ggml-org#223)
ml8 trainer: dispatch profiler + host-bound root-cause findings
ml8 trainer ggml-org#223: pack indices on-GPU + cache packed layer
ml8 trainer profile: ggml-org#223+ggml-org#224 results (1.87x) + bf16…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
7 remaining items
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
vulkan : MoE gate + up + SwiGLU in one weight-streaming MUL_MAT_ID la…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…
rdna-boosts: block 05: flash-attention band, native quantized K/V and…