How to use ggml for Flan-T5

@ggerganov

@ggerganov Thanks for sharing llama.cpp. As usual, great work.

Question rather than issue. How difficult would it be to make ggml.c work for a Flan checkpoint, like T5-xl/UL2, then quantized?

Would love to be able to have those models run on a browser, much like what you did with whisper.cpp wasm.

Thanks again. (I can move this post somewhere else if you prefer since it's not technically about Llama. Just let me know where.)

@i-am-neo

@i-am-neo just curious, have you found T5-xl/UL2 to be in any way superior to FB's llama models (even the smallest 7b model) for certain tasks? IIRC T5-xxl used to be the best open source model before llama, but now...?

Since LLaMA wasn't published under a FOSS license and explicitly does not allow commercial use, it shouldn't be considered an open source model.

@i-am-neo

@i-am-neo just curious, have you found T5-xl/UL2 to be in any way superior to FB's llama models (even the smallest 7b model) for certain tasks? IIRC T5-xxl used to be the best open source model before llama, but now...?

@alexconstant9108 I have found Flan-T5 performant when one needs accurate answers to questions (no inventions allowed). This is from real-life data, details disguised for privacy. Flan-T5 was used in its recipe.

Flan-UL2 looks to be more "fluent"/expressive than Flan-T5, but I've just started to look.

I won't comment on text generation nor "general knowledge queries" since I don't use these models to write novels, nor do I care if they know about Batman and Robin.

What I found beautiful about Stanford Alpaca was that a 7B autoregressive model can be coaxed to follow instructions. The Llama paper mentions "Llama-I" which compared favorably to other instructed models at around 65B parameters). As to how well Alpaca does at 7B, we'll have to see (I've requested the Stanford team to run stats and hopefully they are willing to respond).

What I find beautiful about the idea of llama.cpp or the possibility of a Flan-T5.cpp is running a model locally, hopefully at some point on the browser. If one could shrink Flan-UL2 down to xB parameters and it still performs well... (Leche Flan anyone?).

I'm of the thinking a sequence-to-sequence model and an autogenerative are better harnessed to the best capabilities of each, depending on what one needs to accomplish, rather than pitted against each other on a one-on-one comparison.

In the end, as our field evolves, we all need to ask the question touched on in the Llama paper, italics mine - "The focus of this work is to train a series of
language models that achieve the best possible performance at various inference budgets, by ..."

@i-am-neo

@i-am-neo
Very interesting. So in that setup did you fine-tune Flan-T5 on your own dataset so that it can answer those questions accurately? Also can you give any example question / answer pairs (with any sensitive data masked for privacy reasons, ofcourse )

There is an alpaca quantized version available for download somewhere on github if you look close enough :)

BTW regarding running flan-t5 (and llama) locally (not in the browser tho) you will be very excited to try https://bellard.org/ts_server/ . supports both CPU and GPU inference

Since people in this thread are interested in Instruct models, I recommend checking out chatGLM-6B.

I believe it is more capable than Flan-UL2 in just 6B parameters. I have a one-click web demo of the 4bit GPU version here:

It's not getting much attention due to being the product of a Chinese University. But in my testing it outperforms every other open source instruct model while coming in at just 4GB, even smaller than Alpaca-7B.

Full repo is here: https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md

@MarkSchmidty

@MarkSchmidty useful reference. Thanks
From what I have observed GLM is mostly ignored due to it being weaker with English prompts. But it may turn out to be better when conversing in Chinese for those who can speak it...
If you could provide a small comparison table of English prompts when using Alpaca, Flan-U2L and GLM-6B, that would open the eyes of many people for that particular Instruct model

@MarkSchmidty

@MarkSchmidty useful reference. Thanks From what I have observed GLM is mostly ignored due to it being weaker with English prompts. But it may turn out to be better when conversing in Chinese for those who can speak it...

I have not had this experience. It seems excellent for English.
It is mostly ignored because the team behind it does very little to market it outside of China or to non-Chinese speakers.

>If you could provide a small comparison table of English prompts when using Alpaca, Flan-U2L and GLM-6B, that would open the eyes of many people for that particular Instruct model
It takes 30 seconds to launch and test my free web demo linked above.

But in my testing it outperforms every other open source instruct model while coming in at just 4GB, even smaller than Alpaca-7B.

BTW, the readme says:

Quantization Level	GPU Memory
FP16（no quantization）	13 GB
INT8	10 GB
INT4	6 GB

How did you manage to shrink it to 4GB? Also is it using GPTQ or RTN quantization for 4-bit?
Edit, I saw that in another place, the authors mention 5.2GB CPU memory
and GPT-Q quantization. It would help if there docs had more English examples. Right now, it seems that they are targeting mostly the Chinese audience which is in line with what you said regarding how they "market" it...

That is the GPU memory required to run inference not the model size.

The official int4 model is 4.06GB on HuggingFace before any pruning.

>It would help if there docs had more English examples. Right now, it seems that they are targeting mostly the Chinese audience which is in line with what you said regarding how they "market" it...

Yes, that's why I'm trying to "market" it to English speakers who want instruct models right now. If more people dig into it then we can collectively work out out answers to questions like these. I actually have an all-English fork in the works which I will publish soon.

I actually have an all-English fork in the works which I will publish soon.

Great! You seem to be moving really fast! They only published the 4-bit model 2 days ago :)
Hopefully, we'll see also see support for GLM-6B within llama.cpp or end up with a glm.cpp fork :)
One more thing - please also include more info about what datasets the model has been trained (and RLHF fine tuned) on (was it similar to LLama, e.g. on the Pile, etc.).
Usually with software / models coming out of China people are wary because of any potential (ab)use of party propaganda materials / censorship and so on.

@i-am-neo

@i-am-neo Very interesting. So in that setup did you fine-tune Flan-T5 on your own dataset so that it can answer those questions accurately? Also can you give any example question / answer pairs (with any sensitive data masked for privacy reasons, ofcourse )

There is an alpaca quantized version available for download somewhere on github if you look close enough :)

BTW regarding running flan-t5 (and llama) locally (not in the browser tho) you will be very excited to try https://bellard.org/ts_server/ . supports both CPU and GPU inference

@alexconstant9108 Thanks for sharing about bellard.org. Very good to know!

Re Flan-T5, no special fine-tuning done in this case. Unfortunately I can't expose question-answer pairs as that would violate user privacy (plus it takes a lot of work to disguise details). I think you may feel similarly if it were your data. You could, if so inclined, try it for yourself, though hopefully because you have a real use case instead of just wanting to kick the tires (limited resources). (ggml-org/whisper.cpp#625)

And yes, I saw those Alpaca weights floating around, thanks. :)

@MarkSchmidty

@MarkSchmidty useful reference. Thanks From what I have observed GLM is mostly ignored due to it being weaker with English prompts. But it may turn out to be better when conversing in Chinese for those who can speak it...

I have not had this experience. It seems excellent for English. It is mostly ignored because the team behind it does very little to market it outside of China or to non-Chinese speakers.

If you could provide a small comparison table of English prompts when using Alpaca, Flan-U2L and GLM-6B, that would open the eyes of many people for that particular Instruct model It takes 30 seconds to launch and test my free web demo linked above.

@MarkSchmidty Thanks for sharing. I was excited enough to try it right away (both using your notebook and another gradio demo off HF). The model doesn't seem as adept at answering questions drawn from context I give it. Maybe it's the way I gave prompts? In your tests, how did you ask it to answer from specific text?

Definitely, though it is not high prio atm, so will hope for a community contribution in the meantime

Going to comment out that not T5 specifically, but more support for available Encoder-Decoder models in llama.cpp would be amazing. From what I've seen, Enc-Dec support on most optimized model inference projects seems almost non-existent. It is really a shame since Enc-Dec models currently tend to offer the best performance in tasks such as Machine Translation and Automatic Speech Recognition.

Speaking of T5 support, Aya model(s) is based on mT5 and it's trending now. It's 13B, so it does require quantization.

Would be great if we can support it now that the T5/mT5 architectures are back to interesting for text-generation (they were before, but now trending next to Llama-2 and Mistral) https://huggingface.co/CohereForAI/aya-101/discussions/9

In terms of API, how do these models work?

I imagine something like:

prompt_enc = llama_encode(prompt_text);

while (true) {
    token = llama_decode(prompt_enc);
    
    ? update prompt_enc with sampled token ?
}

Does that make sense? Would be helpful if we get a short summary about how these models are different compared to decoder-only models

@ggerganov

Sorry if I misunderstood your request, but encoder-decoder models perform "translations", like Whisper, or Google Translate. They can do summarisation, Q&A, and even completion too, if so trained.

Encoder-decoder models might be exposed to an end user with a similar GUI. Rather than it being a completion task, it's all about one specific input, and one specific output. That is a separate thing, not a continuation. It's "translate" vs "autocomplete" (even though they can do completion, it's not done as a continuation).

The code you sketched seems to be a completion, which is quite different. The API would probably need to be a bit different to support these sorts of models.

Some encoder-decoder models are inherently less at risk to instruction prompt injection and can be made to run really fast on mobile devices more easily because of their constrained task focus. Both are really important features for solving problems quickly and reliably.

I imagine it would be more like this:

prompt_enc = llama_encode(prompt_text);

// Empty string to collect output tokens
result_output = "";

for (int i = 0; i < max_output_length; ++i) {
    string token = llama_decode_step(prompt_enc, result_output);
    result_output += token;
    if (token == END_TOKEN) {  
        break;
    }
}

Encoder-Decoder models in llama.cpp could be amazing.
I hope this comment is useful, much respect, I'm using Llama.cpp all time time, you're a hero!
@ggerganov

This issue was closed because it has been inactive for 14 days since being marked as stale.

This works on CUDA, but (ggml-org#247)

rdna-boosts: block 05: flash-attention band, native quantized K/V and…

Navigation Menu

Repository navigation

How to use ggml for Flan-T5 #247

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How to use ggml for Flan-T5 #247

Description

Activity

alexconstant9108 commented on Mar 18, 2023

michaelbogdan commented on Mar 19, 2023

i-am-neo commented on Mar 19, 2023

alexconstant9108 commented on Mar 20, 2023

MarkSchmidty commented on Mar 21, 2023

alexconstant9108 commented on Mar 21, 2023

MarkSchmidty commented on Mar 21, 2023

alexconstant9108 commented on Mar 21, 2023

MarkSchmidty commented on Mar 21, 2023

alexconstant9108 commented on Mar 21, 2023

i-am-neo commented on Mar 21, 2023

i-am-neo commented on Mar 21, 2023

18 remaining items

ggerganov commented on Dec 31, 2023

jorirsan commented on Feb 18, 2024

maziyarpanahi commented on Feb 19, 2024

ggerganov commented on Feb 19, 2024

lukestanley commented on Feb 28, 2024

github-actions commented on Apr 14, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions