Issue with embedding generation

[marcobambini]

I wrote a very simple example using a model that I am sure is ready to be used to generate embeddings.
However, I continue to receive the following error: decode: cannot decode batches with this context (calling encode() instead).

From the source code, I see that it is generated when memory is false:

int llama_context::decode(const llama_batch & batch_inp) {
    GGML_ASSERT((!batch_inp.token && batch_inp.embd) || (batch_inp.token && !batch_inp.embd)); // NOLINT

    if (!memory) {
        LLAMA_LOG_DEBUG("%s: cannot decode batches with this context (calling encode() instead)\n", __func__);
        return encode(batch_inp);
    }

How can I ensure that memory is initialized correctly?
I haven't found any explicit function to initialize it.

[ggerganov]

Most embedding models don't have a memory (a.k.a. a KV cache). This is not an error - just a warning telling you that you can simply use llama_encode() instead of llama_decode().

[marcobambini]

Thanks @ggerganov. I spent two days trying to fix this—don’t you think it should be made clearer that it’s just a warning? Or better yet, do you really think it's necessary to generate a warning in this case at all?

[marcobambini]

@ggerganov can I use llama_get_memory to decide at runtime if llama_decode or llama_encode must be used?

[ggerganov]

If your application is going to support both models with and without a memory, then you should simply call llama_decode() always. The warning message is a debug message (LLAMA_LOG_DEBUG) - it only appears in debug builds and is useful for the llama.cpp developers. It's not something that you should worry about.

For example, llama-server always calls llama_decode() regardless what model is used.

@ggerganov can I use llama_get_memory to decide at runtime if llama_decode or llama_encode must be used?

You can, but it's much simpler to just call llama_decode().