Orchestrating LLMs

[jeffotoni]

🌐 Proposal for Integration with LLMs and AI Ecosystem

The main objective is to connect with LLMs and orchestrate communication between them, facilitating the development of applications such as RAGs (Retrieval-Augmented Generation), Intelligent Agents, among others.

⚠️ We will not develop Ollama.
Our focus is to communicate with Ollama, consuming the language models it supports.

🔗 Communication with LLMs

The idea is to support communication with multiple LLM providers, including:

• OpenAI (GPT-4, GPT-3.5)
• Anthropic (Claude)
• Google Gemini
• Mistral (open-source)
• Groq (via LLM Engine)
• And others, as available

We intend to develop a LangChain-inspired library, with a focus on modularity and ease of use.

🧠 Vectors / Embeddings / RAG

Our scope also includes integration with vector and RAG technologies, such as:

• Pinecone
• Weaviate
• Qdrant

🖼️ AI Image Processing

We will support communication with image generation and analysis platforms:

• OpenAI DALL·E 3
• Stability AI (Stable Diffusion)
• Replicate
• Google Cloud Vision API
• Clarifai

🎙️ Text-to-Speech (TTS) / Voice

The proposal also includes integration with voice and text-to-speech APIs:

• OpenAI TTS
• ElevenLabs
• Google Cloud Text-to-Speech
• Amazon Polly
• Microsoft Azure TTS

---

💡 Name suggestions for Quick's LLM library

Here are some simple name suggestions I thought of for our Quick project:

• quick/llm
• quick/langchain
• quick/mind
• quick/qai
• quick/agents
• quick/lang
• quick/brain

Feel free to comment, suggest other names or vote for your favorites! 🚀

🤔 What do you think of this idea?

I'm curious to hear your thoughts!

Let's build something powerful, modular, and easy to integrate with the current AI ecosystem.

[emmadal]

Great idea. for the use of LLMs, they will be outside of the Quick core ? for example each library will be a independant repository ?

Also, for these integration, they will come after the CLI or the both will come together ?

[jeffotoni]

Great idea. for the use of LLMs, they will be outside of the Quick core ? for example each library will be a independant repository ?

Also, for these integration, they will come after the CLI or the both will come together ?

At first, it was supposed to be something like
github.com/jeffotoni/quick/llm

Being part of the Quick ecosystem takes advantage of the power we have with lightweight routes and robust HTTP client that we developed with retry etc.

In this scenario we have to maintain 100% without external dependencies, so below I mapped all LLMs and vector bases that we were able to do 100% using our good old HTTP 🩷.

And below is a draft of the idea, having connects for LLMs and that's where the magic begins.

llm/
├── openai/ # ChatGPT, GPT-4, etc
 │ └── client.go
├── gemini/ # Google Gemini
 │ └── client.go
├── anthropic/ # Claude
 │ └── client.go
├── deepseek/
 │ └── client.go
├── qwen/ # Qwen 2.5
 │ └── client.go
├── ollama/ # Local templates via localhost:11434
 │ └── client.go
├── togetherai/
 │ └── client.go
├── cohere/
 │ └── client.go
├── mistral/
 │ └── client.go

# Vector Bases
├── vector/
│ ├── pinecone/
│ │ └── client.go
│ ├── qdrant/
│ │ └── client.go
│ ├── weaviate/
│ │ └── client.go
│ ├── chromadb/
│ │ └── client.go
│ └── redis/
│ └── client.go

# Core and abstractions
├── client.go # HTTP client with retry, headers, etc (you can use the one from Quick)
├── types.go # Common interfaces, structs (ChatRequest, ChatResponse, etc)
└── llm.go # Generic interface to consume any LLM

Reinforcing without external dependencies, 90% of what I saw can be done.
I'm thinking of starting like this, if in the future we see that it has taken on a gigantic dimension and we need the dependencies for example, then we can separate them without any problems. What do you think of this approach? 🐧🐧

communication flow view

User calls:
llm.Chat(prompt)

↓ Internally calls:
llm.go
├── type LLM interface {
│ Chat(prompt string) (response string, err error)
│ }
├── func New(provider string, cfg Config) LLM
│ ├── switch provider:
│ │ ├── "openai" → return openai.New(cfg)
│ │ ├── "gemini" → return gemini.New(cfg)
│ │ └── etc...

↓ Example for openai:
openai/client.go
├── type Client struct {
│ cfg Config
│ http *quickhttp.Client // using Quick's http client
│ }
├── func (c *Client) Chat(prompt string) (string, error)
│ ├── Builds JSON payload
│ ├── Calls c.http.POST(url, headers, body)
│ ├── Automatically retry via Quick's client
│ ├── Reads the response body
│ └── Returns the clean response (generated text)

↓ quickhttp.Client (already implemented in Quick)
quick/httpclient/
├── client.go
│ ├── Retry, Timeout, Headers, etc
│ └── JSON helpers for request and response

Just a draft of the idea, we will have several types llm interfaces {} ... 🩷
Did I manage to explain it? 🐧
Let me know what you think? 🐧

[emmadal]

I understand well. i'm just excited to work on it too