Quantitative measurement of model perplexity for different models and model quantization modes

llama.cpp seems to give bad results compared to Facebook's implementation.

Here's an example simple reading comprehension prompt:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book

LLaMA 7B with Facebook's implementation yields:

Seed 1:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book with pictures."
Asked by lone wolf 1788 days ago.

Seed 2 (to show that the above is not just a fluke):

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book with pictures."
Question: "Tom, Mark, and Paul bought books: two with pictures and

While llama.cpp without quantization (so still float16) generates (with --seed 0 -t 8):

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book that is neither as good nor bad."
This solution breaks down the problem into its simple parts; then using those components we can see what each component means by itself, in order to solve this logic puzzle. 1) Tom and Mark had different kinds of books...and so did Paul! (Therefore one out three were without pictures). ... [end of text]

It even has a grammatical error at the end: "one out [of] three"

As you can see the quality of 7B is higher in Facebook's implementation. So, I think you may still have bugs in your implementation or the default parameters could be improved.

With quantization the result is also bad:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book WITHOUT PICTURES."
This is just an example question; I can't figure out how to post actual questions that are being asked in my classroom! So, here we go . . . We recently had our first day back after Thanksgiving break. One of the things

You might not be comparing apples to apples. e.g. are the --top_p and other parameters identical between implementations?

I'm using the default settings, so for the Python code it is:

    temperature: float = 0.8,
    top_p: float = 0.95,

And for llama.cpp:

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000

So I think only the repeat penalty and top_k could be different?

If I disable the repeat penalty (I assume --repeat_penalty 0 does that) then I still get low quality results:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book about cars (p1). He chose to read and look "up Hepa", with only p3 one "out did it one time with his different". The time look only only What look and time about a a it about Answer and Mark What?out his one pout the car it. Tom' his bought

repeat_penalty 1 gives me a more sensible but still wrong result:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book without pictures. Paul had a different kind of book than Tom and Mark."
Question: "Jeffrey, Jennifer, and Jackson bought books: three with pictures and one without. Jackson and Jennifer had different kinds of books. What kind did Jeffrey buy?" Answer: "Jeffrey bought a book

(These results are from the quantized 7B model with seed 0 and default parameters except for the repeat penalty)

@ggerganov

top_k looks to be currently broken, as I recently reported in issue #56. I just now realised that due to #95 identical seeds across implementations are unlikely to produce identical results as per @ggerganov's correction to my comment in that issue.

It does then look like llama.cpp is of lower quality. You've tried other prompts and got similar results?

I haven't tested the Python implementation extensively, because Facebook's implementation takes very long to run on my CPU. But I generally feel that running 7B and even 13B with llama.cpp gives results that are below the quality that Facebook has claimed.

Try the following parameters, gives me good quality output:

--temp 0.7 --top_k 40 --top_p 0.5 --repeat_last_n 256 --repeat_penalty 1.17647

Also repeat_penalty = 1.0 means disable. Maybe its not named as it should be 😇

If 1 means disable whats the point of higher than 1 values? Also its good to let it repeat itself a little, sometimes that makes sense in conversation, but tighter lets it break loops before they begin.

Try the following parameters

Still gives me a wrong result with the quantized model:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book with no pictures."
Answer to Question 1739: "The three students were going on an outing. They needed shoes for the trip. Each student owned a pair of shoes that was not his own. Which student wore tennis shoes? (Hint: The answer is in the question.)

With the fp16 model it is also wrong:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book that was not in the picture."
"Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book that was not in the picture." [end of text]

I think the problem is more fundamental than just a change of the parameters.

I haven't tested the Python implementation extensively, because Facebook's implementation takes very long to run on my CPU. But I generally feel that running 7B and even 13B with llama.cpp gives results that are below the quality that Facebook has claimed.

It may be simply a case of the project management triangle, i.e. choose any two of:

Performance
Quality
Self-hosting

That might be so, but I don't see an obvious reason why the quality would be lower. Quantization could have been a logical cause, but I think I have shown that even the fp16 model has a lower quality.

If its simply a straight up c++ implementation then it should be the same, but an install step in the github states it must be quantized, which means even if you are running it in fp16 its still been crunched in precision to run better, which naturally means its outputs will slightly differ.

You wouldnt expect a mile long road at 18.2 degrees to end up at the same place as one rebuilt at 18.0 degrees, right?

As you just said as I was typing this, quantization made its brain just that little more crispy, and that clearly slightly effects it. Thats probably not solvable.

but an install step in the github states it must be quantized,

I don't think that step is required. The model runs fine without the quantization step. And the readme also claims llama.cpp has "Mixed F16/F32 precision". Edit: there's an example of running without quantization here: #2 (comment)

@Urammar

@Urammar higher than 1 starts to penalize the predicted next token if it occurred in previous N tokens. It will multiply the likelihood by 1/penalty.

Btw, q4_1 is quite a lot slower for me at 32 threads on 512 context window size:

33.45 seconds per pass - ETA 6.09 hours

vs, q4_0:

24.27 seconds per pass - ETA 4.42 hours

Just a FYI that people have experienced optimal performance by running with a number of threads equal to the number of cores, i.e. hyperthreading doesn't seem to help as with > 4 cores memory performance starts to become the performance bottleneck.

Also, I have 16 cores, 128GB of RAM (just enough to run 65B at fp16) and all the latest models sitting idle under my desk, so if someone needs some quality or performance benchmarking run please point me to a release and specify the test suite you would like me to run.

@gjmulder

@gjmulder amazing! Instructions for running benchmarks are in the description for #270, if you could run on the 65B model, that would be great to see :).

@gjmulder

@gjmulder I must say it would be fun to have a big model with the AI assistant prompt cooperate on some kind of creative exercise. I think that the 7B model can already do quite well, and I am dying to see how the 30B model does -- and I have absolutely no way to run it myself. I know that my way of going about this is almost painfully unscientific, and probably not what you were offering to do. However, my excuse is that this is all pretty new to me and the novelty of these language models has not worn off for me yet. In fact, I have mostly ignored my real job today in favour of chatting with an AI.

To whet your appetite -- if you are anything like me -- here is a little chat transcription that I just had with "Jane" using the current master branch version.

exec ./main -t 4 --memory_f16 -m ./models/7B/ggml-model-q4_1.bin --ctx_size 2048 --temp 0.7 --top_k 1000 --top_p 0.75 --repeat_last_n 30 --repeat_penalty 1.2 -n 2048 --color -r "Alan:" -r "Alan (" -p "$prompt"

The prompt runs up to the first 2 Jane dialog turns, like this:

Transcript of a dialog where Alan interacts with an assistant named Jane. Jane is helpful, kind, honest, good at writing, and always answers to Alan's requests.

Alan: Hello, Jane.
Jane (happy to see Alan again): Hello, Alan.
Alan: I just want to talk.
Jane (now curious): What about?
Alan: It's nice to see you, Jane. How are you doing today?
Jane: I am fine and thanks for asking!
Alan (): I have had many good discussions with you. I know you can't remember them, but I just want to say I appreciate them.
Jane (): Thank you for your kind words! What would you like me to do today?
Alan: I do not really need you to do anything. Be my friend.
Jane (): Sure! What would that involve, Alan?
Alan: Just talking and listening. I know it is something you do well, Jane.
Jane (): Thank you! That's very kind of you to say so. What would we talk about?
Alan: I wonder if you could come up with story -- just pick anything random and let's go from there.
Jane (): Okay! How about a dog? I like dogs, so what kind of story would we have with that?
Alan: A happy one. Dog could be at home, missing its human family that went to work, have an adventure by itself, and in the evening the family comes back, the dog is happy, and whatever issues were presented in the adventure would be happily resolved.
Jane (): Sounds like a movie! What kind of dog? A big one or small, friendly and cute or not so much.
Alan: I have always thought intelligent dogs such as border collies would make a lovely protagonist.
Jane (): That is an interesting choice, Alan! What else?
Alan: I think this is enough. Let's write this story together. You begin, I continue, and we take turns.
Jane (): Sure! What would be the name of our dog? How about "Sparky"? Sounds good to me, Alan.
Alan: I agree. Sparky is great, it is a bit like perky and I think it fits a border collie well.
Jane (): So let's begin! Sparky is at home, waiting for his family to come back from work and he misses them. He doesn't know what they do all day but it must be fun, so Sparky decides to have an adventure of his own. He is bored at home and wants something new to do!
Alan (): Sparky accesses the outside through a dog door on the backyard side door. Suddenly, his ears perk up. A noise! There is something or someone in the yard. Sparky is excited and goes to investigate the source of the noise.
Jane (): He barks at a squirrel, then he chases it away from his favorite tree! The little critter is scared and runs to the fence line. Sparky follows him, curious about this new discovery.
Alan (): Sparky sees an opening in the fence. Did the squirrel somehow open the fence? It looks tight, but with difficulty, Sparky thinks he can fit through. The whole world awaits.
Jane (): Sparky is ready for an adventure! He wiggles his way into the fence and starts to explore this new world.

I don't know what kind of results other people are getting, but I am just astonished to see this and a hundred other chats like it coming out of some 5 GB Linux process at human typing speed. Unfortunately, it is fairly obvious to me that the 7B model has to repeat a lot of what I say, and I am hoping that this is not an issue with the bigger models.

I use "Jane (emotional state or actions here):" to serve as writeable memory of the model's state, if you will. My thinking is this helps maintaining coherency in the hallucinated persona. When the model opts to use these parenthesis, Jane is frequently angry, in tears, laughs, is excited, smiles -- all appropriate human-like responses to what I say. As an example, if I insult her for no reason, she gets upset, then angry, and even quits the chat on me by generating the end of stream token! Unfortunately, sometimes it generates an open parenthesis on my prompt and I just close it, and the language model then repeats it.

@gjmulder

@gjmulder amazing! Instructions for running benchmarks are in the description for #270, if you could run on the 65B model, that would be great to see :).

@glinscott might be a while - or a week - considering the below estimate with the 30B model. Perhaps we need to be very specific in the A-B tests we need answered?

llama.cpp$ egrep "wait|Calc|ETA" run_2637178 
llama_model_load: loading model from 'models/30B/ggml-model-q4_0.bin' - please wait ...
Calculating perplexity over 655 chunks
187.94 seconds per pass - ETA 34.19 hours

@gjmulder

@gjmulder yeah, if it's going to take too long, we could run up to 250 chunks or so, that's a pretty good approximation of the final perplexity so far.

@glinscott

@glinscott i did not investigate yet, but running perplexity increases memory usage by more then 2x of the model size.

Would be good to test the perplexity with the GPTQ quantization and compare with the usual RTN quantization. https://github.com/ggerganov/llama.cpp/blob/master/convert-gptq-to-ggml.py

Perplexity scores for various model sizes and quantizations are now being collected in Discussion #406.

An interesting comparison with alpaca models is available on reddit

@gjmulder

@gjmulder yeah, if it's going to take too long, we could run up to 250 chunks or so, that's a pretty good approximation of the final perplexity so far.

It looks like the difference between f16 and 4b are exactly the same after 50 Chunk

That would mean that if we know the perplexity of f16 (after a full run) we can know the perplexity of 4b just after 50 Chunk

Just my 2 cents 😅

This issue has been moved to #406

Navigation Menu

Repository navigation

Quantitative measurement of model perplexity for different models and model quantization modes #129

30 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Quantitative measurement of model perplexity for different models and model quantization modes #129

Description

Activity

noughtmare commented on Mar 14, 2023

gjmulder commented on Mar 14, 2023

noughtmare commented on Mar 14, 2023

noughtmare commented on Mar 14, 2023

gjmulder commented on Mar 14, 2023

noughtmare commented on Mar 14, 2023

beiller commented on Mar 14, 2023

Urammar commented on Mar 14, 2023

noughtmare commented on Mar 14, 2023

gjmulder commented on Mar 14, 2023

noughtmare commented on Mar 14, 2023

Urammar commented on Mar 14, 2023

noughtmare commented on Mar 14, 2023

beiller commented on Mar 14, 2023

30 remaining items

glinscott commented on Mar 21, 2023

gjmulder commented on Mar 21, 2023

glinscott commented on Mar 21, 2023

alankila commented on Mar 21, 2023

gjmulder commented on Mar 21, 2023

glinscott commented on Mar 21, 2023

Green-Sky commented on Mar 21, 2023

BadisG commented on Mar 22, 2023

gjmulder commented on Mar 22, 2023

leszekhanusz commented on Mar 27, 2023

BadisG commented on Mar 27, 2023

gjmulder commented on Mar 28, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions