Skip to content

Chromadb getting locked. #4038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bbl-my opened this issue Mar 20, 2025 · 2 comments
Open

Chromadb getting locked. #4038

bbl-my opened this issue Mar 20, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@bbl-my
Copy link

bbl-my commented Mar 20, 2025

What happened?

I have indexed 2101951 chunks into chromadb. Each chunk is 512 tokens.
Embeddings used: tiktoken

Below is code that is used for indexing. There is only one process that is doing indexing which is ongoing process with below code.

    def get_or_create_collection(self, client, collection_name):
        """
        Attempts to retrieve a collection. If it does not exist, creates a new one.

        :param client: The database client instance.
        :param collection_name: The name of the collection to retrieve or create.
        :return: The retrieved or newly created collection.
        :raises: Re-raises unexpected exceptions.
        """
        try:
            return client.get_collection(collection_name)
        except Exception as e:
            if "Collection" in str(e) and "does not exist" in str(e):
                logging.warning("Collection %s not found. Creating a new one.", collection_name)
                return client.create_collection(collection_name, metadata={"hnsw:space": "cosine"})
            else:
                logging.error("Unexpected error while getting collection %s: %s", collection_name, str(e))
                raise  # Re-raise the unexpected exception
collection = self.get_or_create_collection(client, collection_name)
collection.add(embeddings=embeddings, ids=ids, metadatas=meta_info, documents=chunks)

There is another process which has exposed rest api over few use cases. They simple do query over the collection on which this document is indexed.

Collection name - BusinessFocus_document
Tenant - BusinessFocus
Database - default_database

The issue that I am facing rite now is that chromadb is getting deadlock (which is more of a sqllite issue it seems). I am running with default chroma setup in a docker.
Below are the logs

INFO:     [20-03-2025 11:12:33] 172.18.0.1:42200 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:12:34] 172.18.0.1:42200 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:12:34] 172.18.0.1:42200 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:12:35] 172.18.0.1:42200 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:12:35] 172.18.0.1:42200 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:12:43] 172.18.0.1:37054 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:12:43] 172.18.0.1:37054 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:12:49] 172.18.0.1:37068 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:12:49] 172.18.0.1:37068 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:12:51] 172.18.0.1:37068 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:12:51] 172.18.0.1:37068 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:12:57] 172.18.0.1:51582 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:12:57] 172.18.0.1:51582 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:13:05] 172.18.0.1:42864 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:13:05] 172.18.0.1:42864 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:13:12] 172.18.0.1:37588 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:13:12] 172.18.0.1:37588 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:17:19] 172.18.0.1:59490 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:17:19] 172.18.0.1:59490 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:17:27] 172.18.0.1:52980 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:17:27] 172.18.0.1:52980 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:17:42] 172.18.0.1:47672 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:17:42] 172.18.0.1:47672 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:17:46] 172.18.0.1:47672 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:17:47] 172.18.0.1:47672 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:17:53] 172.18.0.1:57436 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
INFO:     [20-03-2025 11:17:53] 172.18.0.1:57436 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 201
INFO:     [20-03-2025 11:18:00] 172.18.0.1:38730 - "GET /api/v2/tenants/Pharma_Regulatory HTTP/1.1" 200
INFO:     [20-03-2025 11:18:00] 172.18.0.1:38742 - "GET /api/v2/auth/identity HTTP/1.1" 200
INFO:     [20-03-2025 11:18:00] 172.18.0.1:38746 - "GET /api/v2/tenants/Pharma_Regulatory HTTP/1.1" 200
INFO:     [20-03-2025 11:18:00] 172.18.0.1:38746 - "GET /api/v2/tenants/Pharma_Regulatory/databases/default_database HTTP/1.1" 200
INFO:     [20-03-2025 11:18:00] 172.18.0.1:38742 - "GET /api/v2/tenants/Pharma_Regulatory/databases/default_database/collections/Pharma_Regulatory_document HTTP/1.1" 200
INFO:     [20-03-2025 11:22:27] 172.18.0.1:44220 - "GET /api/v2/tenants/BusinessFocus/databases/default_database/collections/BusinessFocus_document HTTP/1.1" 200
ERROR:    [20-03-2025 11:41:10] database is locked
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 111, in receive
    return self.receive_nowait()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 106, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 124, in receive
    return receiver.item
           ^^^^^^^^^^^^^
AttributeError: 'MemoryObjectItemReceiver' object has no attribute 'item'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 157, in call_next
    message = await recv_stream.receive()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 126, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/chroma/chromadb/server/fastapi/__init__.py", line 107, in catch_exceptions_middleware
    return await call_next(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/server/fastapi/__init__.py", line 131, in check_http_version_middleware
    return await call_next(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/telemetry/opentelemetry/__init__.py", line 134, in async_wrapper
    return await f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/server/fastapi/__init__.py", line 807, in add
    await to_thread.run_sync(
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/server/fastapi/__init__.py", line 789, in process_add
    return self._api._add(
           ^^^^^^^^^^^^^^^
  File "/chroma/chromadb/telemetry/opentelemetry/__init__.py", line 150, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/api/segment.py", line 103, in wrapper
    return self._rate_limit_enforcer.rate_limit(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/rate_limit/simple_rate_limit/__init__.py", line 23, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/api/segment.py", line 437, in _add
    self._producer.submit_embeddings(collection_id, records_to_submit)
  File "/chroma/chromadb/telemetry/opentelemetry/__init__.py", line 150, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/db/mixins/embeddings_queue.py", line 238, in submit_embeddings
    with self.tx() as cur:
  File "/chroma/chromadb/db/impl/sqlite.py", line 55, in __exit__
    self._conn.commit()
  File "/chroma/chromadb/db/impl/sqlite_pool.py", line 33, in commit
    self._conn.commit()
sqlite3.OperationalError: database is locked
INFO:     [20-03-2025 11:41:11] 172.18.0.1:44220 - "POST /api/v2/tenants/BusinessFocus/databases/default_database/collections/34bd3d87-4674-4a6b-a304-6fee07e0ba81/add HTTP/1.1" 500
ERROR:    [20-03-2025 11:41:11] cannot start a transaction within a transaction
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 111, in receive
    return self.receive_nowait()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 106, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 124, in receive
    return receiver.item
           ^^^^^^^^^^^^^
AttributeError: 'MemoryObjectItemReceiver' object has no attribute 'item'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 157, in call_next
    message = await recv_stream.receive()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 126, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/chroma/chromadb/server/fastapi/__init__.py", line 107, in catch_exceptions_middleware
    return await call_next(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/server/fastapi/__init__.py", line 131, in check_http_version_middleware
    return await call_next(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/telemetry/opentelemetry/__init__.py", line 134, in async_wrapper
    return await f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/server/fastapi/__init__.py", line 694, in get_collection
    await to_thread.run_sync(
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/telemetry/opentelemetry/__init__.py", line 150, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/api/segment.py", line 103, in wrapper
    return self._rate_limit_enforcer.rate_limit(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/rate_limit/simple_rate_limit/__init__.py", line 23, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/api/segment.py", line 293, in get_collection
    existing = self._sysdb.get_collections(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/telemetry/opentelemetry/__init__.py", line 150, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/chroma/chromadb/db/mixins/sysdb.py", line 444, in get_collections
    with self.tx() as cur:
  File "/chroma/chromadb/db/impl/sqlite.py", line 41, in __enter__
    self._conn.execute("BEGIN;")
  File "/chroma/chromadb/db/impl/sqlite_pool.py", line 29, in execute
    return self._conn.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: cannot start a transaction within a transaction

After this error the only option left is to retart the docker but issue happens like every 2-4 hours.

Versions

Aws linux - t2.2xlarge
Docker chroma version - 0.5.20

Relevant log output

@bbl-my bbl-my added the bug Something isn't working label Mar 20, 2025
@bbl-my bbl-my changed the title Chromadb getting deadlock. Chromadb getting locked. Mar 20, 2025
@bbl-my
Copy link
Author

bbl-my commented Mar 26, 2025

I have enabled WAL mode under sqlite directly, post this I did not encountered any locking issue.

@bbl-my
Copy link
Author

bbl-my commented Mar 26, 2025

is this correct way to do it ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant