Longer and infinite output

If we use -n 1000000 to have a very long output (for a story for example),
it stops generating quite fast, after around 30 lines, probably because of this line of code.

It would be nice if we could have longer outputs and also the possibility to have infinite output, stopping only on Ctrl-C.
We could maybe specify that -n 0 will trigger that infinite output mode.
That issue is a bit related to issue #23

Depending on how much memory you have you can increase the context size to get longer outputs. On a 64gb machine I was able to have a 12k context with the 7B model and 2k context with the 65B model. You can change it here

Huh... I thought the context size was determined when the model was trained due to the positional encoding used. (I am only a layman.) But #78 is still useful for when you eventually hit the limit of your context, right?

When trying large contexts, I often encounter

aggml_new_tensor_impl: not enough space in the context's memory pool (needed 702840624, available 701883745)

I played around a bit with increasing ctx_size but that did not work, I suspect an underlying memory UB as the cause, as lldb seems to trap on some suspicious memory accesses

Depending on how much memory you have you can increase the context size to get longer outputs. On a 64gb machine I was able to have a 12k context with the 7B model and 2k context with the 65B model. You can change it here

Your link goes to this code snippet if (!llama_model_load(params.model, model, vocab, 512)) { // TODO: set context from user input ?? is this correct to change the context size?

@drewcrawford

ggml_new_tensor_impl: not enough space in the context's memory pool (needed 704905456, available 704155676) Assertion failed: false, file ggml.c, line 2516

@drewcrawford for me, that error doesn't appear to be context size related. I've run the same prompt at different context sizes and they all fail.

Typically if you get the not enough space in the context error you tried setting the context too large, though on the larger models I have had to tweak this line too. The math around memory allocation in this thing isn't perfectly scaling on the larger models and unfortunately my fork has substantially diverged from master and too lazy to work on merging.

llama_model_load: loading model from 'models/13B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 8192
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: ggml ctx size = 20562.56 MB
llama_model_load: memory_size = 12800.00 MB, n_mem = 327680

8192 context size on quantized 13B model

Yes, the math for computing necessary memory for the ggml buffers and graphs has to be updated.

What's the best way to enable infinite output? Can we just shift-out the old contexts in K and V tensors (along the n_ctx dim) when they are full, or is there a better approach?

Hmm, I think yes - we need to shift the KV cache. I haven't implemented this yet in any of the ggml examples.
And when the context is full - stop increasing n_past.

The 65B model sometimes crashes with 2k, but usually works ...

llama_model_load: loading model from './models/65B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 8192
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 64
llama_model_load: n_layer = 80
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 22016
llama_model_load: n_parts = 8
llama_model_load: ggml ctx size = 49157.73 MB
llama_model_load: memory_size = 10240.00 MB, n_mem = 163840

The increased context doesn't increase memory use (still under 41GB) - until the context gets filled.

Yet, on the 7B model the calculation overflow at the start for 16k or greater. Usually, 15360 will work.
llama_model_load: ggml ctx size = 17592186044337.35 MB

eous@e0213e0 here is some easy stuff I pulled out of my local hackery if anyone wants to play with it

Btw if anyone wants to slice/dice/refactor/cleanup/dissect/mixup/etc that changeset feel free, I don't need to be credited.

eous@e0213e0 here is some easy stuff I pulled out of my local hackery if anyone wants to play with it

It's only been 4 days and this is apparently already a dead link. is that because this has already been added or?

e0213e0

@jboero

@jboero can you provide an example llama model u are using that can have 400k context size? i thought context generated is limited by the llama model u use. max i know is 192k which i never get to touch coz it's way above my 8gb vram and 64gb ram.

how do you get 400k tokens generated?

Sorry I should have specified "prompt context" not model context. Ala --ctx-size arg. No I haven't actually used a model with 400k context.

-c N, --ctx-size N: Set the size of the prompt context. The default is 512, but LLaMA models were built with a context of 2048, which will provide better results for longer input/inference. The size may differ in other models, for example, baichuan models were build with a context of 4096.

I realize I'm quite late to this party... but I'm able to get an infinite response running llama.cpp on my raspberry pi 4. When I load it up, I use these commands.

./main -m models/[your model]  --color \  --ctx-size 2048 \ -n -1 \ -ins -b 256 \ --top_k 10000 \ --temp 1.5 \ 
--repeat_penalty 1.1 \ --ignore-eos

I'm going to let you all know that I've been playing around with AI for literally the past 2 weeks, so I barely know what I'm doing, but I'm still learning. (In case you looked at those commands and gagged, laughed or asked yourself, "What in God's holy name is this moron doing?"). I'm kind of like the guy that in an emergency situation that can lift a car off of someone, because in that moment I'm not thinking about all the reasons I can't lift a car. What I mean is, I think I got llama.cpp to work in the first place by brute force and ignorance, so I can't explain why it works, it just does for me.

So I hope that helps anyone who knows less than I do, or it opens doors for someone who knows more than I do.