epic: Jan integrates Cortex.cpp

[freelerobot]

Goal

• Jan integrates Cortex.cpp and has a bug-free transition to Cortex.cpp
• Planning: planning: Jan's path to cortex.cpp? #3690

Tasklist

• Eng Spec (copy over from planning: Jan's path to cortex.cpp? #3690, replace this post)

Previous Issues

• feat: Jan supports new Cortex's Model Folder and model.yaml architecture #3633

[louis-menlo]

Implementation Specs

Migration Path:

1. App 0.5.8 opens
2. Return model list from cache (given users are on 0.5.7) -> function normally.
3. Scan JSON models (legacy logics - fresh install or older versions) -> function normally.
4. In the background, the app attempts to import models from cortex.cpp and merge them with legacy downloaded models (failed to import models).
5. The app combines models returned by cortex.cpp and legacy JSON models. Cortex.cpp models are prioritized in case of the same ID (Models are imported successfully.)

Changes

Naming convention

• inference-nitro-extension is renamed into inference-cortex-extension.
• cortex.cpp binaries have the same name as engine releases.
• Pre-package everything, include cuda dependencies (dll, so) so users don't have to install separately.
• Support noavx-cuda binaries as a fallback

Simplifed

• Deprecated ModelFile. It's no longer relevant. Now, providers define models, so it should manage how to run itself.
• Remove install cuda toolkit UX, should be ready after installed.

Downloader

App proxies to cortex.cpp or app's downloader, depending on the cortex.cpp model support capability.

Model Hub

• App allows extensions to register models available for download in RAM. After downloading them, the models will have their yaml or json persisted along with the model files.
• App priorities model hub decoration (previous json metadata) over cortex.cpp metadata (such as name, size, tags)

Observability

• cortex-extension should watch cortex.cpp server upon launch. It ensures that the cortex process runs with the application.
• All requests will be queued and run when the server to come online, ensuring the UX remains the same. So there would be no asynchronous requests and server run introduced. E.g. Model import or start should not fail due to server not being online in time.
• So there would be no attempt to kill the cortex process on model start every time. It is just a stop and start model, so it will not block other API requests.

Goals

• Updated from the older version to this version, models will be imported and run normally. Models are not imported will still able to run since we will attempt to do preflight before running.
• Users can download models or app proxies to cortex.cpp, or use the app downloader, depending on the cortex.cpp model support capability.

Subtasks

• Pull latest cortex.cpp and engines to package
• Keep binaries with the same name as their release name
• Support noavx-cuda binaries as a fallback
• Rename nitro-extension to cortex-extension
• Deprecate ModelFile
• Pre-package Cuda dependencies
• Register models should be persisted in memory.
• App manages to download models using cortex.cpp downloader and it's legacy downloader (tensorrt-llm and clip models)
• App gets persisted models from cortex.cpp and scans JSON models itself (legacy)
• App prioritize model decoration metadata from Hub over cortex.cpp
• cortex-extension watches cortex.cpp server on launch
• Model-extension queues cortex.cpp API requests with health checks so requests do not fail due to server uptime
• cortex.cpp supports legacy model load parameters (NGL, context-length, cache enable, clip mmproj)
• App shares CUDA dependencies with extensions (asar.unpackaged) to improve load time and reduce app size when multiple extensions share the same artifacts (tensorrt-llm and llama-cpp)

@janhq/jan @janhq/cortex