Rerank API not accessible: {"error":{"code":500,"message":"grpc service not ready","type":""}}

[Sarmingsteiner]

I am trying to connect Dify with LocalAI for reranking. However, the connection to the API (http://localhost:8080/v1/rerank) is not possible and results in a 500 error.

But even the proposed CURL test is not successful.
After some time, the follow error message appears:
{"error":{"code":500,"message":"grpc service not ready","type":""}}

I tried with both latest-aio-cpu as well as latest-cpu packages (version 2.16).

Any ideas?

[mudler]

for using rerankers you would need images shipping python dependencies. the latest-aio-cpu and latest-cpu tags automatically points to core images that doesn't ship python.

See also: https://localai.io/basics/container/#standard-container-images

[Sarmingsteiner]

Thanks for your quick reply!
Unfortunately, upon checking the link, I cannot find the tags for the images including python?

[mudler]

just use latest, or for instance a tagged release e.g. v2.16.0

[walbercardoso]

Im getting this error with tts models (coqui, bark). Only piper is working

failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43665: connect: connection refused""

[devMls]

I have problems for access to rerank too:

11:15AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:jina-reranker-v2-base-multilingual ContextSize:512 Seed:1345567851 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:2 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/espeak-ng-data RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/build/models/jina-reranker-v2-base-multilingual Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 LoadFormat: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false ModelPath:/build/models LoraAdapters:[] LoraScales:[]}

11:15AM INF [stablediffusion] Loads OK

**11:15AM ERR Server error error="rpc error: code = Unimplemented desc = method Rerank not implemented" ip=10.0.1.19 latency=14.148126589s method=POST status=500 url=/v1/rerank**

11:16AM INF Success ip=127.0.0.1 latency="411.985µs" method=GET status=200 url=/readyz

11:17AM INF Success ip=127.0.0.1 latency="59.579µs" method=GET status=200 url=/readyz

11:18AM DBG guessDefaultsFromFile: modelPath is empty

11:18AM DBG Request for model: jina-reranker-v2-base-multilingual

11:18AM DBG Model already loaded in memory: jina-reranker-v2-base-multilingual

11:18AM DBG Checking model availability (jina-reranker-v2-base-multilingual)

11:18AM DBG Model 'jina-reranker-v2-base-multilingual' already loaded

my docker swarm config:

services:
  reranker:
    hostname: rerank_ai_server
    image: localai/localai:v2.23.0
    command: run https://gist.github.com/devMls/3e5fb2ae3fd573b29cecc0eb2a7ec822 #file://build/rerank_config.yml
    networks:
      - ragflow
      - bridge
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    environment:
      - DEBUG=true
      - HUGGING_FACE_HUB_TOKEN=secret
    configs:
      - source: rerank_config.yml
        target: /build/rerank_config.yml  
    volumes:
      - localai_rerank:/build/models:cached
networks:
  ragflow:
    external: true
  bridge:
    name: bridge
    external: true    

volumes:
  localai_rerank:
    driver: local

configs:
  rerank_config.yml:
    external: true 

my model config:

name: jina-reranker-v2-base-multilingual
backend: rerankers
parameters:
  model: huggingface://jinaai/jina-reranker-v2-base-multilingual

any clue?

[alex89607]

Also have a problem:
HTTPConnectionPool(host='host.docker.internal', port=8083): Max retries exceeded with url: /v1/rerank/rerank (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7067d02671a0>, 'Connection to host.docker.internal timed out. (connect timeout=1

assigned port to 8083. And nothing..