Backend init_tensor Behaviour

[taronaeo]

While attempting to run matrix multiplication operations via IBM zDNN as a backend, I noticed that the inputs tensor does not contain a valid pointer address in tensor->extra, which means the tensor had not gone through the init_tensor buffer function.

I could only assume that this happened because my backend did not list GGML_OP_MUL as a supported operation. If I were to turn on support for that, it would mean requiring to support all operations that came before it as well, including GGML_OP_RMS_NORM, which we do not have support at all.

If init_tensor is the designated function for initialising additional information within tensor->extra but not all tensors go through this function, how else are we supposed to initialise additional information that is required for our operations to run?

IBM zDNN requires all tensors to be accompanied with pre-transform, transform, and zTensors. If any tensor does not contain it, the operation will not run.

[slaren]

All tensors go through the init_tensor function of the buffer in which they are allocated. If you found a case where that doesn't happen, it would be a bug. One possibility is that the supports_buft function of your backend is not properly setup, and you are receiving tensors from a different buffer type.

If I were to turn on support for that, it would mean requiring to support all operations that came before it as well, including GGML_OP_RMS_NORM, which we do not have support at all.

This is not accurate, ggml_backend_sched schedules each operation to a backend that can run it, and it ensures that the tensors are copied to a buffer type that is supported by the backend.

[slaren]

Your backend is advertising that it can use any buffer type that is in host memory, so of course it also receives tensors allocated in a CPU buffer. If you only want tensors in a zDNN buffer type, your supports_buft function needs to reflect that.

[taronaeo]

Oh, man, you're right; that indeed resolved the issue 😆. Before I resolve this, can I check if the backend is documented anywhere apart from within the GGML codebase, or can we set up documentation for it? That way we can inform future contributors of things like these that they won't understand at first glance.

Thanks so much!

[slaren]

There isn't a lot of documentation in ggml, but there are brief descriptions of what each function in the ggml-backend interfaces are intended to do in ggml-backend-impl.h and ggml-backend.h.

[taronaeo]

Yeah, I did refer to those files and was still thinking that we should have more documentation, in the format of markdown, possibly within the docs/backend directory. Unfortunately, those comments do not carry enough information for people like me to catch the subtle differences in implementation.

If it adds to the understanding, I spent nearly 7 hours today trying to find alternative workarounds to "fix" the missing tensor->extra when the problem was really in supports_buft 😅. We can save time by having these documented.

I can get started on the documentation work in markdown via the suggested directory above, but I can't guarantee that my knowledge is correct or accurate

[slaren]

Having more comprehensive documentation would be great, if you write it I will review it.