Error: llama_model_load: error loading model: failed to open ggml-bagel-2.8b-v0.2-q8_0.gguf

[vineel96]

What happened?

Hello,
I have following error while running llama.cpp with mamba architecture:

Steps to reproduce:

1. git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
2. gh pr checkout 5328 (since this pr is related to mamba incorporation)
3. make -j10
4. ./main -e -p "[INST] Write an essay about AI. [/INST]\n" -m ggml-bagel-2.8b-v0.2-q8_0.gguf

Please help in providing steps to run llama.cpp with mamba architecture.

Name and Version

version: 2403 (39579d3)
built with cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0 for aarch64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[manny-pi]

Hi @vineel96,

Is it possible that your model isn't in the root directory of llama.cpp?

I would suggest you try any of the following you double-check the location of your model, and try to run the program again. If that doesn't work, create a new branch, pull the changes from #5328 into your branch, and try to run the model again.

Please let me know if this helps, and what went wrong if it didn't!

[vineel96]

Hi @manny-pi,

1. I could not find the model in repository
2. I tried creating new branch and pulling changes, then also its throwing the same error.

Is it possible to share steps to run mamba with llama.cpp or where to find the location of that model?

[compilade]

@vineel96 You do not need to pull #5328, since it has been merged a while ago in the master branch. This means you can use the latest version of llama.cpp, or any version released since after March 2024.

Basically, what I use to run Mamba is the following:

First make sure you have a GGUF Mamba model and put it somewhere. (I prefer to use convert_hf_to_gguf.py on the original model files on a big external hard drive, but you can of course use something like ~/Downloads/ggml-bagel-2.8b-v0.2-q8_0.gguf (the tilde (~) expands to the $HOME directory) assuming you've downloaded a pre-converted GGUF from https://huggingface.co/dranger003/mamba-bagel-2.8b-v0.2-GGUF (but note that these files have no pre-tokenizer set, so that will be slightly wrong at run-time; recent conversions do not have this problem, but no recent conversions were published for that model (if you do convert it, I recommend its DPO version instead: https://huggingface.co/jondurbin/bagel-dpo-2.8b-v0.2, but it's a huge download at 11GB of F32 weights.) To avoid downloading too much stuff, I recommend that you start with what you already have.))

$ cd llama.cpp # assume the current directory from now on is a llama.cpp checkout
$ git branch --show-current # Note: I'm on the master branch
master
$ git pull --ff-only origin
$ mkdir build
$ cd build
$ cmake .. -DGGML_NATIVE=TRUE
$ make -j4 # of course you can use more
$ ./bin/llama-cli -m ~/Downloads/ggml-bagel-2.8b-v0.2-q8_0.gguf -p "Hello" # use the correct path for where you stored your model

Note that you don't necessarily have to use cmake, directly using make from the root of the repo is also very fine. Your problem mostly seems to be related to the path of the model (hence the No such file or directory error). Make sure you use a correct path which exists. Also note that if you use a recent llama.cpp version, the binary for main has been renamed to llama-cli somewhere in the last months.

where to find the location of that model?

When you download files, where do they end up? In the shell, when using a command which accepts file names, you can use your <TAB> key to auto-complete file names, which should help you navigate to valid paths.

[vineel96]

@compilade,
Thanks for the reply. It is working now.