Support GGUF models

[vladfaust]

See vllm-project/vllm#1002, vllm-project/vllm#5191.

Should be able to set gguf as QUANTIZATION envar, but we also need to provide exact quant. I'm thinking of some MODEL_FILENAME envar containing the exact filename in the model's repository. The model download logic shall be changed, see https://github.com/Isotr0py/vllm/blob/main/examples/gguf_inference.py.

[nerdylive123]

yeah i guess now we dont need set quantization env var but we need to support it like from this one

[mohamednaji7]

@nerdylive123 How could I implement GGUF for my fork?