Instructions for "Prepare Data & Run" don't seem to work on Ubuntu 22.04

[bobhairgrove]

I was able to build the llama.cpp code with CMake, and I downloaded the 7B and 13B models. However, it seems that the instructions for setting up the data do not work when building it this way:

(1) Instructions say:

# obtain the original LLaMA model weights and place them in ./models
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model

I assume that .models should be created as a subfolder of the build/bin folder, otherwise the instructions given later for running quantize and main will not work? Ok, so I did that... Now my $pwd is ~/code/llama.cpp/build/bin.

(2) Instructions say:

# install Python dependencies:
python3 -m pip install -r requirements.txt

Where is "requirements.txt"? Ok, I found this file back under ../../llama.cpp, so I cd ../.. and run that. Now my $pwd is ~/code/llama.cpp...

(3) Instructions say:

# convert the 7B model to ggml FP16 format
python3 convert.py models/7B/

Assuming that there should be a '.' before models/7B/, I add the dot and run this command. However, it cannot find .models/7B/ (obviously, because it is under llama.cpp/build/bin/, right?)

So I run this command instead, after cding back to llama.cpp:

python3 convert.py build/bin/.models/7B/

This appears to work, wih lots of output lines similar to the following:

Loading model file build/bin/.models/7B/consolidated.00.pth
Loading vocab file build/bin/.models/tokenizer.model
Writing vocab...
[  1/291] Writing tensor tok_embeddings.weight                  | size  32000 x   4096  | type UnquantizedDataType(name='F16')
[  2/291] Writing tensor norm.weight                            | size   4096           | type UnquantizedDataType(name='F32')
(etc.)

So I cd back to llama.cpp/build/bin and look at the contents of the .models folder: Sure enough, there is a file:

.models/7B/ggml-model-f16.bin

So now I try to run, according to instructions:

./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin q4_0

Error received:

main: build = 546 (08737ef)
main: quantizing './models/7B/ggml-model-f16.bin' to './models/7B/ggml-model-q4_0.bin' as q4_0
terminate called after throwing an instance of 'std::runtime_error'
  what():  failed to open ./models/7B/ggml-model-f16.bin: No such file or directory

What next?

[bobhairgrove]

Ok, just found the error with the last instruction:

It should be

./quantize .models/7B/ggml-model-f16.bin .models/7B/ggml-model-q4_0.bin q4_0

And the next command has the same error:

./main -m ./models/7B/ggml-model-q4_0.bin -n 128

Remove the slash before "models" and it appears to work.

[bobhairgrove]

Hmmm ... now I see that the FIRST line said to install the models into ./models and not .models. So it was partly my mistake.

Will this make any difference?

(It is not necessary to prefix relative directories with a './.' , only the command you are running if it is not in the $PATH. I suggest that these be removed, because it is very confusing. However, there are still issues with the other things not being in the expected directories.)

[Green-Sky]

It is not necessary to prefix file paths with ./ , except when you want to execute/call a file in the current directory.
the instructions seem to be written for make, so the models folder is in the root of the repo.

[SlyEcho]

There is actually no requirement to put the model files in a fixed location. All the tools have options to read and write to any path on the system.