Would you be interested in putting together a PR to fix this?

I cannot implement my own StatelessExecutor implementation because it relies on methods that are internal to NativeApi.

LLamaSharp should expose all native functionality of the llama.cpp API, that's the core goal of the library! Which bits are missing/inaccessible?

Sure - I’ll get one started. In terms of general direction, is simply adding the parameter as a property to any Executor that uses the decoder in order to pass it through an appropriate design? Or would it be better to add it to one of the structures passed in to construct the Executor?

The problem line came from

I was trying to duplicate behavior (naively) in my own class and project, and this internal api was the problem:

Not sure that should be exposed, though.

In terms of general direction

It's up to you, but from a quick skim of the relevant code it looks to me like adding it as a property to IInferenceParams might be the best option.

llama_kv_self_seq_add

That's exposed through SafeLlamaContextHandle.KvCacheSequenceAdd:

It used to be that the entire llama.cpp API was exposed through NativeApi completely unchanged (no checks, accepting raw pointers etc). Over time we're slowly replacing those methods with slightly safer wrappers in the SafeHandle classes.

Awesome - I agree with that from a user perspective 😄 I'll get this fired off quickly.

Makes sense that I got confused if the code is in the process of being changed - since I saw the rm function publicly available, I assumed I would also see the add function. That's on me for a naive copy-paste.