iOS library is broken because ggml dependency is not pinned

[azarovalex]

I was debugging some accidental crashes in SwiftUI sample, dug deeper and understood a couple of things.

Some background

1. When llama.cpp is built outside of Xcode, e.g. with make, there is no direct dependency on ggml repo. ggml sources are just duplicated in this repo, they get out of sync but there are scripts used to sync two repos + whisper.cpp, which also use ggml.

2. Xcode build is different (not only the sample, but all external applications that use llama.cpp through SPM). In this case ggml will be fetched from ggml repo, ignoring files present in this repo. Additionally, dependency on ggml repo is not pinned, which technically means that ggml can be resolved to any commit.

Why it's done like this and why it doesn't work

This difference in build logic between Xcode and make was introduced only recently in this PR.
I completely see the reasoning behind this change, indeed whisper.cpp already has it's own copy of ggml, so we got a conflict of sybmols in the resulting binary if we used both llama.cpp and whisper.cpp in one iOS app.

However the current approach just doesn't work for llama.cpp at all. For example right now latest master of llama.cpp (cd108e6) is not compatible with the latest master of ggml (c75db1e), it leads to a crash on this line.

Proposed solution

I really don't want to force Grigori to rethink how he structures his projects, in this case a monorepo might solve the problem, but I guess iOS build just has to adapt to the existing structure.
The best solution I see so far:

• We create a new target in the SPM manifest. It will build llama.cpp exactly how make does it, using local ggml sources and without dependency on external repo. This new target becomes the recommended way to integrate llama.cpp to any iOS app, because it's guaranteed to always build the same sources as other build systems. (this step is basically a revert of Updates Package.swift to use ggml as dependency #4691)
• For rare cases when an app already has ggml, like with whisper.cpp, we create a second target that uses ggml from the external repo and the developer of the app will be responsible for pinning a compatible version, ensuring two commits of llama and ggml work together. (This target is exactly the target we have right now)
• Finally we add a CI job that builds the sample, so this won't happen again. 😄

I'd love to hear opinions on this possible solution from folks who previously contributed into SPM manifest: @ggerganov @1-ashraful-islam @kchro3 @jhen0409.

[ggerganov]

Thank you for creating the issue and summarizing the problem - this helps. I think the main trouble came from me overlooking the dependency problem of the Swift package when we merged #4691. Should be possible to resolve by simply pinning to a specific ggml commit like we just did in whisper.cpp: ggml-org/whisper.cpp@cc7f872

However, there is still one issue that would remain - the ggml.h header in the spm-headers folder. Can we eliminate this and make the llama.cpp Swift package simply use the ggml.h header from the ggml package? Ideally, we should be able to simply remove the ggml.h symbolic link from spm-headers. Will be testing out this later today - any feedback in the meantime is appreciated.

The 2-target solution is also an option, but not sure if it is better.

Btw, #4822 is likely unrelated to the current build issues.

However the current approach just doesn't work for llama.cpp at all. For example right now latest master of llama.cpp (cd108e6) is not compatible with the latest master of ggml (c75db1e), it leads to a crash on this line.

Can you double-check this? The crash initially was caused by the introduction of the new quantization type which changed the enum value for GGML_TYPE_I32 in the ggml.h header. However, after syncing the 2-bit changes to ggml (ggml-org/ggml@8a547fe) it should no longer be an issue.

[ggerganov]

Could you guys review #4878 and let me know if it works for you and if you think this is a good way to proceed with llama.cpp Swift package

[to3d]

Pinning the dependency to a specific commit seems ideal, but I'm struggling to figure out if it's currently possible to build a project that references both the llama.cpp and whisper.cpp Swift packages on github.

If there were more history of pinning the Swift packages like this for both whisper.cpp and llama.cpp, it would allow you to pick from a set of the two that share the same ggml commit. Currently, I don't see a way to resolve a shared ggml version, but maybe it can be done through creating additional targets.

[1-ashraful-islam]

@to3d I made the three pull requests to specifically allow importing both llama.cpp and whisper.cpp in to a single iOS app using SPM. I did not run into the the current issue of needing to pin the ggml dependency to a specific commit at that time.

See:

• Adds Swift Package declaration for GGML ggml#674
• Updates Package.swift to use ggml as dependency #4691
• Updates Package.swift to use ggml as package dependency whisper.cpp#1701

[to3d]

@1-ashraful-islam Yeah, I had been able to successfully work with both of them because of that, so thank you!

If I understand correctly, if the whisper.cpp Swift package was also updated to reference 979cc23b - alongside llama.cpp with #4878 - the dependency issues should be resolved (assuming that this version of ggml doesn't break anything in whisper.cpp)

[ggerganov]

For now I merged the pinned commit version. I just figured we can make the packages track a release branch in ggml and when we sync the repos, we'll update the release branch and it should be all good.

[ggerganov]

Both llama.cpp and whisper.cpp Swift packages now reference the release branch of ggml. Let me know if you run into any problems

[azarovalex]

Great, I think this solution should work, looks like the issues that I saw where caused by ggml.h being taken from llama.cpp and ggml.c from ggml, now that's fixed.

Pinning to a branch is still not perfect, it's not forcing Xcode to fetch the latest commit each time, some resolved hash is stored in Package.resolved and it can get out of sync with resolved commit of llama.cpp itself.
But for a fast moving project like I guess this will do. 👍

[ggerganov]

Pinning to a branch is still not perfect, it's not forcing Xcode to fetch the latest commit each time, some resolved hash is stored in Package.resolved and it can get out of sync with resolved commit of llama.cpp itself.

AFAIK in these cases Xcode -> File -> Packages -> Update to Latest Package Versions should help