This repository was archived by the owner on Sep 12, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Sep 12, 2024. It is now read-only.
cannot use RWKV models #121
Description
I just tried to use the current version of "llama-node" with the "rwkv.cpp" backend and failed.
The link found in the docs where I should be able to download RWKV models points to nowhere.
Since I could not find pre-quantized models anywhere, I followed the instructions found in the rwkv.cpp repo to download, convert and quantize the 1.5B and 0.1B models - I even uploaded them to HuggingFace.
Then, I copied the example found in your docs added a path to my quantized model, changed the template and tried to run the result.
Unfortunately, I got nothing but an error message:
llama.cpp: loading model from /Users/andreas/rozek/AI/RWKV/RWKV-5-World-0.1B-v1-20230803-ctx4096-Q4_1.bin
error loading model: unknown (magic, version) combination: 67676d66, 00000065; is this really a GGML file?
llama_init_from_file: failed to load model
node:internal/process/promises:288
triggerUncaughtException(err, true /* fromPromise */);
^
[Error: Failed to initialize LLama context from file: /Users/andreas/rozek/AI/RWKV/RWKV-5-World-0.1B-v1-20230803-ctx4096-Q4_1.bin] {
code: 'GenericFailure'
}
Node.js v18.17.0
Do you have any idea what could be wrong?
Activity
rozek commentedon Nov 9, 2023
I just learned that RWKV-5 models are not yet supported by rwkv.cpp.
So I tried RWKV-4 instead - took the .pth model and converted it to .bin following the docs. Unfortunately, however, the result is the same:
Using the same model with
python python/generate_completions.py /rwkv/RWKV-4-World-0.1B-v1-20230520-ctx4096.binfrom rwkv.cpp works, howeverrozek commentedon Nov 9, 2023
how do I check my GGML version? I'm using the current version of rwkv.cpp
rozek commentedon Nov 9, 2023
I just found a section in th rwkv.cpp README.md which says:
may this be the reason for misbehaviour?
rozek commentedon Nov 10, 2023
FYI: I just used the version of rwkv.cpp from from Sept, 20th (before they restructured the Python API) and tried again - with the same results.
Which means: no, the API restructuring is not the reason for not loading the RWKV model
rozek commentedon Nov 11, 2023
FYI: going back to the latest commit (of rwkv.cpp) before "update ggml" fails because the resulting code can not be compiled.
Thus, in order to test if "llama-node" does work with RWKV actually means to go back to commit "update ggml" (8db73b1) and manually revert any changes related to GGML
Damn...
Not being a C++ developer, I have to give up here - I'll mention this problem in rwkv.cpp as well (see issue 144), let's see who will be able to fix it
saharNooby commentedon Nov 11, 2023
Hi!
The module
rwkv-cppinllama-nodeexplicitly points to a specific version ofrwkv.cpp: rwkv.cpp @ 363dfb1. In turn, this version ofrwkv.cppexplicitly points to a specific version ofggml: ggml @ 00b49ec. For all of this to work, I highly recommend not using newest/arbitrary versions of the packages, and stick to the ones that are explicitly referenced -- that way, everything should be compatible with each other.saharNooby commentedon Nov 11, 2023
If it helps debugging, for some reason
llama.cpploads the RWKV file, notrwkv.cpp:rozek commentedon Nov 11, 2023
That was quick - thank you very much.
Unfortunately, I cannot get rwkv.cpp @ 363dfb1 to compile.
Unless I manage to find out why, I may have to wait for RWKV-5 support.
Nevertheless, thank you very much for your effort!
rozek commentedon Nov 11, 2023
FYI: I managed to compile rwkv.cpp again - my mistake was to only
git reset --hardrwkv.cpp itself, but not the included ggml repo...Now I'm trying to use it - a first attempt with the current version of llama-node failed with the same error message as before.
Let's see what the detail
llama.cpp: loading modelmeansrozek commentedon Nov 11, 2023
Ok, I think I have to give up - now RWKV crashes with
I installed llama-node using
which seems to be wrong anyway as the RWKV inference example refers to a file (
20B_tokenizer.json) which is only found within thenode_modules/llama-nodefolder and should not have to be referenced thererozek commentedon Nov 11, 2023
Well, I meanwhile used rwkv.cpp @ 363dfb1 with ggml @ 00b49ec, as mentioned above.
But, as described before
node_modules(which means that simply installing llama-node with its rwkv backend is not sufficient)