Skip to content

[Bug]: thread 'tokio-runtime-worker' panicked #4365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dertruffel opened this issue Apr 23, 2025 · 2 comments
Open

[Bug]: thread 'tokio-runtime-worker' panicked #4365

dertruffel opened this issue Apr 23, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@dertruffel
Copy link

What happened?

chromadb latest/1.0.6

Chroma panics randomly after a while, didn't happen on earlier versions, but our data amount also increased.

Versions

chroma 1.0.6 latest, python:3.11, host os is Ubuntu 24.04.2 LTS
shouldn't be relevant, running as docker containers

Relevant log output

thread 'tokio-runtime-worker' panicked at /chroma/rust/system/src/wrapped_message.rs:88:30:
message reply channel was unexpectedly dropped by caller: Ok(())
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
   2: core::result::unwrap_failed
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/result.rs:1679:5
   3: core::result::Result<T,E>::expect
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/result.rs:1059:23
   4: <core::option::Option<chroma_system::wrapped_message::HandleableMessageImpl<M,<C as chroma_system::types::Handler<M>>::Result>> as chroma_system::wrapped_message::HandleableMessage<C>>::handle_and_reply::{{closure}}
             at ./chroma/rust/system/src/wrapped_message.rs:86:25
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
   6: chroma_system::wrapped_message::WrappedMessage<C>::handle::{{closure}}
             at ./chroma/rust/system/src/wrapped_message.rs:55:61
   7: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
   8: chroma_system::executor::ComponentExecutor<C>::run::{{closure}}
             at ./chroma/rust/system/src/executor.rs:98:64
   9: chroma_system::system::System::start_component::{{closure}}
             at ./chroma/rust/system/src/system.rs:54:65
  10: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  11: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/core.rs:331:17
  12: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/loom/std/unsafe_cell.rs:16:9
  13: tokio::runtime::task::core::Core<T,S>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/core.rs:320:30
  14: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:499:19
  15: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  16: std::panicking::try::do_call
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  17: std::panicking::try
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  18: std::panic::catch_unwind
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  19: tokio::runtime::task::harness::poll_future
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:487:18
  20: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:209:27
  21: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:154:15
  22: tokio::runtime::task::raw::RawTask::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/raw.rs:201:18
  23: tokio::runtime::task::LocalNotified<S>::run
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/mod.rs:435:9
  24: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:596:18
  25: tokio::runtime::coop::with_budget
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/coop.rs:107:5
  26: tokio::runtime::coop::budget
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/coop.rs:73:5
  27: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:595:9
  28: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:546:24
  29: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:511:21
  30: tokio::runtime::context::scoped::Scoped<T>::set
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context/scoped.rs:40:9
  31: tokio::runtime::context::set_scheduler::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context.rs:180:26
  32: std::thread::local::LocalKey<T>::try_with
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283:12
  33: std::thread::local::LocalKey<T>::with
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260:9
  34: tokio::runtime::context::set_scheduler
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context.rs:180:17
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:506:9
  36: tokio::runtime::context::runtime::enter_runtime
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context/runtime.rs:65:16
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:498:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/worker.rs:464:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/core.rs:331:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/loom/std/unsafe_cell.rs:16:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/core.rs:320:30
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:499:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs:272:9
  45: std::panicking::try::do_call
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:557:40
  46: std::panicking::try
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:521:19
  47: std::panic::catch_unwind
             at ./rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs:350:14
  48: tokio::runtime::task::harness::poll_future
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:487:18
  49: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:209:27
  50: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/harness.rs:154:15
  51: tokio::runtime::task::raw::RawTask::poll
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/raw.rs:201:18
  52: tokio::runtime::task::UnownedTask<S>::run
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/task/mod.rs:472:9
  53: tokio::runtime::blocking::pool::Task::run
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/pool.rs:161:9
  54: tokio::runtime::blocking::pool::Inner::run
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/pool.rs:511:17
  55: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/pool.rs:469:13
@dertruffel dertruffel added the bug Something isn't working label Apr 23, 2025
@tazarov
Copy link
Contributor

tazarov commented Apr 24, 2025

@dertruffel, can you share during what type of operations your Chroma instance has panicked?

@dertruffel
Copy link
Author

dertruffel commented Apr 24, 2025 •
edited
Loading

90% of operations is retrieval from the base, although i don't log that kind of data. Is there a way to auto-restart upon panic?
@tazarov
Additionally i upped the version to 1.0.7.dev17 and it still happens, same trace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tazarov @dertruffel