Add support for HuggingFace LLM Inference

[tasmay]

I'm trying to build a fullstack RAG application with LlamaIndex TS. I don't want to use create-llama as it keeps asking for OpenAI API Key which I don't intend to use as the underlying LLM. Is there any support for HuggingFaceInferenceAPI in LlamaIndexTS as there is in the python version?

[himself65]

See: https://ts.llamaindex.ai/modules/embeddings/available_embeddings/huggingface

[tasmay]

See: https://ts.llamaindex.ai/modules/embeddings/available_embeddings/huggingface

Thanks for your reply. But the code still requires me to provide an OpenAI API key. Is that mandatory to provide? I don't intend to use OpenAI as underlying LLM. I want to call the HuggingFaceInferenceAPI like it's done here https://docs.llamaindex.ai/en/stable/examples/llm/huggingface/ but I want to achieve the same in Typescript. Is that even possible?

[marcusschiesser]

@tasmay Currently, we're not supporting HuggingFace for LLM inference in Typescript (you're referencing the Python version that supports it), but we support a lot of other LLMs besides OpenAI in Typescript; see https://ts.llamaindex.ai/modules/llms/

You just have to replace the OpenAI class with one of these classes in the code generated by create-llama (or generate Python code (FastAPI) with create-llama).

I'll change the title of this issue to add support for HF LLM

[tasmay]

Thanks, @marcusschiesser. That answers my question!

[himself65]

maybe related d10533e

[marcusschiesser]

@tasmay in 0.3.4, you can now use HuggingFaceInferenceAPI, see:

LlamaIndexTS/examples/huggingface/chat.ts

Lines 1 to 22 in f1862cc

	import { HuggingFaceInferenceAPI } from "llamaindex";
	(async () => {
	if (!process.env.HUGGING_FACE_TOKEN) {
	throw new Error("Please set the HUGGING_FACE_TOKEN environment variable.");
	}
	const hf = new HuggingFaceInferenceAPI({
	accessToken: process.env.HUGGING_FACE_TOKEN,
	model: "mistralai/Mixtral-8x7B-Instruct-v0.1",
	});
	const result = await hf.chat({
	messages: [
	{ content: "You want to talk in rhymes.", role: "system" },
	{
	content:
	"How much wood would a woodchuck chuck if a woodchuck could chuck wood?",
	role: "user",
	},
	],
	});
	console.log(result);
	})();

[tasmay]

Thanks @marcusschiesser! This is really helpful.