Repository navigation
High performance API #321
Description
Hey!
I'd love to see this project being able to be used through some TCP socket with a very optimized protocol. One it may make use of something like protobuf, or even grpc.
I think everyone agrees HTTP would be a complete overkill specially for a project focused on high performance. 😆
Thanks
Niansa
Reacted by FireFragment
Activity
LostRuins commentedon Mar 20, 2023
I don't agree that HTTP is overkill. I think it is perfect. You have RESTful APIs for all the large LLM providers like OpenAI. You send your inputs and generation params, the service sends you back the generated response string.
Here's an example of serving llama.cpp over HTTP, as an emulated KoboldAI server.
https://github.com/LostRuins/llamacpp-for-kobold
tarruda commentedon Mar 20, 2023
Try out the
tcp_serverbranch, more details of how it works in #278niansa commentedon Mar 20, 2023
Sure but none of these are performance oriented like this project and none of these run on the local PC.
If done correctly building a REST API on top of THAT would be trivial, even from Python if performance isn't much of a point anyways.
Also supporting HTTP would require lots of code or even external dependencies.
anzz1 commentedon Mar 20, 2023
Yes, keep the core lean, portable, fast and free of dependencies while having the option of building things on top of it, as modules. This could be achieved with a C api which simply uses file streams in place of stdin/stdout. A communication layer based on those file streams could then be built upon to achieve whatever functionality necessary. The tcp_server branch is actually almost like that, but not quite.
LostRuins commentedon Mar 22, 2023
My distributable is a 700kb zip containing literally 3 files, one of which is optional. It requires nothing else except a stock python install to run.
https://github.com/LostRuins/llamacpp-for-kobold/releases/tag/v1.0.3
and if you don't want that, we have a literal 1 file pyinstaller solution with a built-in server.
https://github.com/henk717/llamacpp-for-kobold/releases/tag/v1.0.2-2048
0x090909 commentedon Mar 27, 2023
@tarruda This is great, is it going to be merged?
tarruda commentedon Mar 27, 2023
I don' t think so, check the discussion in #278 for more context.
Calandiel commentedon Apr 25, 2023
No idea what your specialization is, but in my part of the programming world TCP is already considered too heavy ^^
A TCP based API would be very much welcome over restful APIs from my point of view.
LlaMA-4 support (text only) (ggml-org#321)
server: --reasoning-effort-allow/-fallback and --reasoning-max-tokens…