ggml : get rid of BLAS and all it's variants

[ggerganov]

This is a big one

The only reason we use BLAS is that we don't have efficient implementation of matrix x matrix multiplication. Naively doing parallel dot products is not optimal. We need to implement some of the fundamental GEMM optimizations such as block tiling and we need to implement this in a compact way that reuses the existing dot product code and supports all quantization types

More comments on this:

• MFA Integration llama.cpp#1867 (comment)
• VRAM optimization + matrix multiplication discussion llama.cpp#1935 (comment)
• ggml : get rid of BLAS and all it's variants #293 (comment)

[philipturner]

supports all quantization types

The best place for dequantization would be after you've loaded data from device -> threadgroup memory, and are loading into registers. I hypothesize it will faster to have each of the four simds in a threadgroup unpack on their own, even if that duplicates the work two times.

For example, you might fork my MFA repository, then modify the line here to unpack after loading.

https://github.com/philipturner/metal-flash-attention/blob/fbfd0a028402c0ae6fa293c2f91de318b95e359b/Sources/GEMM.metal#L129

[Dampfinchen]

While I do understand the desire for the project to be independend of other libraries, I personally do not think removing the excellent cuBLAS implementation entirely is a good idea. One of the main benefits of this project is that you can run large models with good speeds on lower end hardware with less VRAM. For example, thanks to the cuBLAS' fast prompt processing of 15 ms/t I can enjoy a 13B parameter model with full context at around 1.6 token/sec, which is much faster than running GPTQ with cpu offloading (0,4 token/s) on my RTX 2060 laptop.

As you've said yourself, a native GEMM implementation would likely be slower than what cuBLAS is offering. Even if the performance difference is not drastic on your hardware, it can make all the difference on hardware like mine, even a slight performance difference (for example 15 ms/t to 23 ms/t) would lead to a worse experience for people who run huge LLMs too big for their systems. Not just for low to mid spec hardware, but for high end hardware trying to run 65B models as well.

Please keep this perspective in mind as you continue to develop the project. I don't think it would be a good outcome for me and many others to have to downgrade to older versions with cuBLAS and not enjoy new enhancements just because older versions run faster due to cuBLAS support.

[philipturner]

Why would custom code be slower than cuBLAS? There is Nvidia CUTLASS. The only things it can be are equal or faster.

[Dampfinchen]

Why would custom code be slower than cuBLAS? There is Nvidia CUTLASS. The only things it can be are equal or faster.

They aren't going to use CUTLASS though, because it'd be another third party lib.

CuBLAS is highly optimized for the hardware and Georgi said himself that he is aware a custom code is not going to perform as well in this comment:

ggml-org/llama.cpp#1867 (comment)

And I fully understand that it will be close to impossible to achieve the maximum performance available from dedicated libraries (such as cuBLAS, for example).

[philipturner]

300 lines of code, (soon to) outperform all of Apple's proprietary Metal Performance Shaders.

https://github.com/philipturner/metal-flash-attention/blob/main/Sources/GEMM.metal

If we take some code from CUTLASS, maybe just optimize for the matrix shapes that exist in LLaMA.

[slaren]

A custom GEMM implementation will be faster with quantized models - that's one of the goals. There may be a small performance regression with f16 and f32 models, though.

[philipturner]

Could we utilize the int4 hardware support in Ampere tensor cores?

[slaren]

From what I have seen of the way int4 works with tensor cores, I don't think so. We cannot do a matrix multiplication directly in int4, we need to dequantize to f16 or f32 first. But we can still use the tensor cores after dequantizing to float. I may be wrong, though.

[Dampfinchen]

A custom GEMM implementation will be faster with quantized models - that's one of the goals. There may be a small performance regression with f16 and f32 models, though.

Glad to hear that. I hope that will be the case.

Could we utilize the int4 hardware support in Ampere tensor cores?

Just a heads up: Tensor cores in Turing, Ampere and Ada Lovelace support INT4, INT8 and FP16 instructions. Ampere and Turing support INT1 as well.

While only Ampere and Ada support FP8 and FP32 in addition to that.

So ideally, the code would use FP16 or the integer instructions I mentioned to cover a wide range of hardware with tensor core support.

[philipturner]

Didn't they remove INT4 on Ada and Hopper?