Installing llm-sentence-transformers degrades startup time

[ricardog]

On my Mac, after installing the llm-sentence-transformers plugins, startup of the llm process slowed down significantly. I measured startup time by running

time ./my-cli.py

Where the script is simply

❯ cat my-cli.py
#!/usr/bin/env python3

import llm.cli

Without the plugin startup is about 0.5s. With the plugin it is about 4s. The plugin brings in a lot of packages that are not required for startup.

I was able to speed up startup with the package by moving the import of SentenceTransformer to the place where it is needed, i.e.

class SentenceTransformerModel(llm.EmbeddingModel):
    def __init__(self, model_id, model_name, trust_remote_code):
        self.model_id = model_id
        self.model_name = model_name
        self.trust_remote_code = trust_remote_code
        self._model = None

    def embed_batch(self, texts):
        from sentence_transformers import SentenceTransformer
        with disable_logging():

Installing llm_mlx also degrades startup time for the same reason; it load an enormous amount of packages. I haven't looked at how the move imports in llm_mlx to speedup startup.

[ricardog]

Here's a diff to speedup startup with the llm_mlx plugin installed.

< import mlx.core as mx
< from mlx_lm import load, stream_generate
< from mlx_lm.sample_utils import make_sampler
202a200
>         from mlx_lm import load, stream_generate
207a206,207
>         from mlx_lm.sample_utils import make_sampler
>
247a248
>             import mlx.core as mx

Also, I just realized the plugins live in separate repos so perhaps this issue should move. On the other hand, it would be nice if there was a way for llm to limit the time taken for plugin import.