server : improvements and maintenance

[ggerganov]

The server example has been growing in functionality and unfortunately I feel it is not very stable at the moment and there are some important features that are still missing. Creating this issue to keep track on some of these points and try to draw more attention from the community. I guess, some of the tasks are relatively big and would require significant efforts to complete

• Support chat templates
  We need to have separation between the user input and the special tokens, so that the tokenization is performed correctly. See the following comments / commits for more context:
  Server: OpenAI-compatible POST /v1/chat/completions API endpoint #4160 (comment)
  c544fae
  Server: OpenAI-compatible POST /v1/chat/completions API endpoint #4160 (comment)

  We already support extracting meta information from the GGUF model files that can provide the chat template for the specific model:
  gguf-py : export chat templates #4125
  Support chat template for /v1/chat/completions: Server: use llama_chat_apply_template #5593
  List of supported templates: view on wiki

  Supporting this in server would require changes both in the backend and the frontend

• Likely redundant logic for OpenAI (OAI) compatibility that should be removed
  server : OAI API compatibility #4198 (comment)

• Use multiple mount points for the OAI API
  https://github.com/ggerganov/llama.cpp/blob/af19d3573481d409b3c4e55494810eb1f65a9aae/examples/server/server.cpp#L2682-L2684
  Add "/chat/completions" as alias for "/v1/chat/completions" #5722

• Return meaningful errors on KV cache overflow
  update_slots : failed to decode the batch #4185 (comment)

• Refactor the code
  With the recent additions for parallel decoding support for multiple clients and LLaVA, I feel the code base became very cumbersome and there is a lot of room for refactoring and improving the code. There should be some effort dedicated to cleaning up things and simplifying the code.
  Server: try to refactor server.cpp #5065
  Server: Improve work queue stability #5710

• Batched decoding endpoint?
  Although we added parallel decoding support via "slots", we are still lacking batched decoding where a single client could pass an array of prompts to be completed. Or alternatively, generate multiple completions for a single prompt. Would be useful to support this use case
  llama : add batched inference endpoint to server #3478 (comment)

• Tool calls (function calling)
  Support for MeetKai/functionary model by implementing OpenAI-compatible tool calls to chat endpoint.
  Server: add support for "tool_calls" (MeetKai/functionary model) #5695

• Multimodal support
  Support has been temporary dropped in server : refactor #5882, before working in server, we should improve llava-cli and the API for using LLaVA

  • server: Bring back multimodal support #8010
  • llava-cli: improve llava-cli and the API for using LLaVA #6027
  • server : refactor #5882 (comment)
  • server : refactor #5882 (comment)
  • server: multimodal - fix misreported prompt and num prompt tokens #5896
  • llama cpp server not doing parallel inference for llava when using flags -np and -cb #5592
  • Unable to assign mmproj value when running docker  #6226

• Prompt processing improvment

  • Server: Add prompt processing progress endpoint? #6586
  • server: process prompt fairly accross slots #6607

• Server production readiness

  • `server` production readiness #6398
  • kubernetes example #6546

This is likely not a complete list of things - if you think some feature is important to be improved or supported, drop a comment.

Have a look to issues labelled with server/webui.

[IridiumMaster]

Would love if the server could get look ahead decoding and contrastive search. A collection of common presets would be very helpful for fast model evaluation. The ability to edit responses and replies in the UI would be very useful for rapidly testing prompt branches if combined with batching capabilities. Would also appreciate a simple implementation of request queuing and a server interface for the model training example. Edit: Discussion link for contrastive search : #3450 , other related topics / potential substitutes are mentioned in the thread.

[ruped]

Thanks for raising this issue and looking into the server example.

I think this #4201 could be relevant - although it sounds like the fix will be in the core code rather than in the server.

Since the addition of support for batching, llama.cpp could be come a viable competitor to vllm for large scale deployments. This is also helpful for individual hobbyists who are using/building AI agents (because these possibly make multiple requests in parallel to the LLMs to construct answers). So I think your suggestions around improving stability/refactor of the server example would be very valuable. Also focusing on the throughput speed particularly of batched requests (and benchmarking this against vllm).

[mudler]

What'd be lovely is to see also the speculative sampling added to it - would be really a great addition there

[tobi]

Very excited about this! I think that the server should increasingly be thought of as the main deliverable of this repo.

There are 100s of libraries and tools that integrate different subset of backends and inference libraries. Especially in the python world. This doesn't make sense. We need a simple convention by which everything can interopt. The solution is to use openai's API as a protocol on localhost. Could there be better standards? Maybe. But this is the one we have, and it works really well.

My suggestion is that clean we clean up the server and treat it and the /chat/completions endpoint as main deliverable of this repository. We can easily switch the web interface to use that as well. ./server -m ~/model should boot with the ideal default parameters read from the gguf like context size and (if we can pull it off) chat template style.

This means that existing code only needs the api_url override to be modified to work locally.

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1")


completion = client.chat.completions.create(
  model="llama!",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message.content)

This works already. At least as long as you are loading a model that conforms to chatml and are ok with the default context size. I find that a much better vision for how LLM interopt will work in the open source space. Different servers, different, backends, all on the same proto.

[FSSRepo]

@ggerganov

Batched decoding endpoint?

This option to generate multiple alternatives for the same prompt requires the ability to change the seed, and the truth is, I've been having a bit of a struggle with it when adding parallel decoding, as it raises questions about how the seed should be managed.

[spirobel]

@tobi

Very excited about this! I think that the server should increasingly be thought of as the main deliverable of this repo.

LocalAI serves this usecase quite well already and has lots of traction. It is better to not compete with your customers and delight them instead. An easy to link library with a C API should be the main deliverable of this project.

[studiotatsu]

The OAI API included with the server, is great I love it.
Please include llama_params "repeat_penalty" and "min_p" .

These params are much needed. Thanks.

[antcodd]

I think it would be good if the OAI endpoint supported the same set of parameters and defaults as the regular endpoint and sensible or argument driven defaults given many clients won't supply all parameters.

One issue is the seed is defaulting 0 instead of -1, so every regeneration is the same if the client doesn't specify a seed.