You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When adding some features to the llm-grok plugin, I noticed an issue where if the max tokens is sufficiently low that the model never finishes its reasoning phase and therefore the response is empty, the DB logging will crash.
❯ llm -m grok-3-mini-latest "what color is the sun?" -o max_completion_tokens 5
Traceback (most recent call last):
File "/Users/daniel/.local/bin/llm", line 10, in <module>
sys.exit(cli())
~~~^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/llm/cli.py", line 781, in prompt
response.log_to_db(db)
~~~~~~~~~~~~~~~~~~^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/llm/models.py", line 456, in log_to_db
"prompt_json": condense_json(self._prompt_json, replacements),
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/condense_json/__init__.py", line 81, in condense_json
return process(obj)
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/condense_json/__init__.py", line 53, in process
return {key: process(val) for key, val in value.items()}
~~~~~~~^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/condense_json/__init__.py", line 55, in process
return [process(item) for item in value]
~~~~~~~^^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/condense_json/__init__.py", line 53, in process
return {key: process(val) for key, val in value.items()}
~~~~~~~^^^^^
File "/Users/daniel/.local/share/uv/tools/llm/lib/python3.13/site-packages/condense_json/__init__.py", line 67, in process
replacement_id: str = substr_to_id[matched_text]
~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: ''
This raises a few broader questions of how to control reasoning vs visible tokens, which each provider will have its own approach for. Should reasoning tokens be visible to the user? (Should llm have a flag for that? I am not aware of one currently). If reasoning tokens are not shown to the user, but the max tokens are sufficiently low such that the response is empty, how should that be indicated to the user?
But, more broadly, the DB logging issue seems like a rough edge case.
The text was updated successfully, but these errors were encountered:
When adding some features to the
llm-grok
plugin, I noticed an issue where if the max tokens is sufficiently low that the model never finishes its reasoning phase and therefore the response is empty, the DB logging will crash.This raises a few broader questions of how to control reasoning vs visible tokens, which each provider will have its own approach for. Should reasoning tokens be visible to the user? (Should
llm
have a flag for that? I am not aware of one currently). If reasoning tokens are not shown to the user, but the max tokens are sufficiently low such that the response is empty, how should that be indicated to the user?But, more broadly, the DB logging issue seems like a rough edge case.
The text was updated successfully, but these errors were encountered: