Issues when returning JSON with claude

[johann-petrak]

No description provided.

[johann-petrak]

See python/notebooks/debug-issue15.ipynb

This is rather weird: the claude model does not seem to actually support the response_format parameter. Instead LiteLLM uses claude tooling, creating a request like this:

'{'model': 'claude-3-5-sonnet-20240620', 'messages': [{'role': 'user', 'content': [{'type': 'text', 'text': "What is a monoid? Provide a long and a short answer. Return as a JSON dict with the keys 'short' and 'long'"}]}], 'tools': [{'name': 'json_tool_call', 'input_schema': {'type': 'object', 'additionalProperties': True, 'properties': {}}}], 'tool_choice': {'name': 'json_tool_call', 'type': 'tool'}, 'max_tokens': 4096}'

This then creates a raw response like this:

{
"id":"msg_01FQNqbsMJjUj92EMJ5ZDEFE",
"type":"message",
"role":"assistant",
"model":"claude-3-5-sonnet-20240620",
"content": [
  {"type":"tool_use",
   "id":"toolu_01FMbknPF5cAsSvZEFbwm2o2",
   "name":"json_tool_call",
   "input":{"json_input":{"short":"A monoid is an algebraic structure with an associative binary operation and an identity element.","long":"A monoid is a fundamental concept in abstract algebra and category theory. It consists of a set equipped with a binary operation that satisfies two key properties:\n\n1. Associativity: For all elements a, b, and c in the set, (a * b) * c = a * (b * c), where * represents the binary operation.\n2. Identity element: There exists an element e in the set such that for any element a, e * a = a * e = a.\n\nMonoids are ubiquitous in mathematics and computer science. Some common examples include:\n\n- The set of integers under addition, with 0 as the identity element.\n- The set of strings under concatenation, with the empty string as the identity element.\n- The set of functions from a set to itself under function composition, with the identity function as the identity element.\n\nMonoids are important because they provide a simple yet powerful abstraction for many mathematical and computational concepts. They form the basis for more complex algebraic structures like groups and rings, and play a crucial role in functional programming, where they are used to model and manipulate various data structures and operations.\n\nIn computer science, monoids are particularly useful for parallel and distributed computing, as their associativity property allows for efficient splitting and combining of computations."}}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":360,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":349}}

So what we really want is nested under input/json_input/

However, when just rerunning the same request, the data may be nested under input/content or input/data

What we get is a json dict with json_input, content or data as the top key

This is wrong and should be considered a bug

[johann-petrak]

See BerriAI/litellm#8898