Skip to content

Commit 957f428

Browse files
mudlermudler
authored
fix(tools): correctly render tools response in templates (#1932)
* fix(tools): allow to correctly display both Functions and Tools * models(hermes-2-pro): correctly display function results
1 parent 61e5e6b commit 957f428

File tree

4 files changed

+64
-54
lines changed

4 files changed

+64
-54
lines changed

‎aio/gpu-8g/text-to-text.yaml

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,30 +3,27 @@ mmap: true
33
parameters:
44
model: huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q6_K.gguf
55

6-
roles:
7-
assistant_function_call: assistant
8-
function: tool
96
template:
107
chat_message: |
11-
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "function"}}{{.Role}}{{else if eq .RoleName "user"}}user{{end}}
12-
{{ if eq .RoleName "assistant_function_call" }}<tool_call>{{end}}
13-
{{ if eq .RoleName "function" }}<tool_result>{{end}}
8+
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}
9+
{{ if .FunctionCall }}<tool_call>{{end}}
10+
{{ if eq .RoleName "tool" }}<tool_result>{{end}}
1411
{{if .Content}}{{.Content}}{{end}}
1512
{{if .FunctionCall}}{{toJson .FunctionCall}}{{end}}
16-
{{ if eq .RoleName "assistant_function_call" }}</tool_call>{{end}}
17-
{{ if eq .RoleName "function" }}</tool_result>{{end}}
13+
{{ if .FunctionCall }}</tool_call>{{end}}
14+
{{ if eq .RoleName "tool" }}</tool_result>{{end}}
1815
<|im_end|>
1916
# https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF#prompt-format-for-function-calling
2017
function: |
2118
<|im_start|>system
22-
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
19+
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
2320
<tools>
2421
{{range .Functions}}
2522
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
2623
{{end}}
27-
</tools>
28-
Use the following pydantic model json schema for each tool call you will make:
29-
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
24+
</tools>
25+
Use the following pydantic model json schema for each tool call you will make:
26+
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
3027
For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
3128
<tool_call>
3229
{'arguments': <args-dict>, 'name': <function-name>}

‎aio/intel/text-to-text.yaml

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,30 +4,27 @@ f16: false
44
parameters:
55
model: huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q6_K.gguf
66

7-
roles:
8-
assistant_function_call: assistant
9-
function: tool
107
template:
118
chat_message: |
12-
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "function"}}{{.Role}}{{else if eq .RoleName "user"}}user{{end}}
13-
{{ if eq .RoleName "assistant_function_call" }}<tool_call>{{end}}
14-
{{ if eq .RoleName "function" }}<tool_result>{{end}}
9+
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}
10+
{{ if .FunctionCall }}<tool_call>{{end}}
11+
{{ if eq .RoleName "tool" }}<tool_result>{{end}}
1512
{{if .Content}}{{.Content}}{{end}}
1613
{{if .FunctionCall}}{{toJson .FunctionCall}}{{end}}
17-
{{ if eq .RoleName "assistant_function_call" }}</tool_call>{{end}}
18-
{{ if eq .RoleName "function" }}</tool_result>{{end}}
14+
{{ if .FunctionCall }}</tool_call>{{end}}
15+
{{ if eq .RoleName "tool" }}</tool_result>{{end}}
1916
<|im_end|>
2017
# https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF#prompt-format-for-function-calling
2118
function: |
2219
<|im_start|>system
23-
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
20+
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
2421
<tools>
2522
{{range .Functions}}
2623
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
2724
{{end}}
28-
</tools>
29-
Use the following pydantic model json schema for each tool call you will make:
30-
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
25+
</tools>
26+
Use the following pydantic model json schema for each tool call you will make:
27+
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
3128
For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
3229
<tool_call>
3330
{'arguments': <args-dict>, 'name': <function-name>}

‎core/http/endpoints/openai/chat.go

Lines changed: 37 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -236,7 +236,7 @@ func ChatEndpoint(cl *config.BackendConfigLoader, ml *model.ModelLoader, startup
236236

237237
// if function call, we might want to customize the role so we can display better that the "assistant called a json action"
238238
// if an "assistant_function_call" role is defined, we use it, otherwise we use the role that is passed by in the request
239-
if i.FunctionCall != nil && i.Role == "assistant" {
239+
if (i.FunctionCall != nil || i.ToolCalls != nil) && i.Role == "assistant" {
240240
roleFn := "assistant_function_call"
241241
r := config.Roles[roleFn]
242242
if r != "" {
@@ -246,14 +246,19 @@ func ChatEndpoint(cl *config.BackendConfigLoader, ml *model.ModelLoader, startup
246246
r := config.Roles[role]
247247
contentExists := i.Content != nil && i.StringContent != ""
248248

249+
fcall := i.FunctionCall
250+
if len(i.ToolCalls) > 0 {
251+
fcall = i.ToolCalls
252+
}
253+
249254
// First attempt to populate content via a chat message specific template
250255
if config.TemplateConfig.ChatMessage != "" {
251256
chatMessageData := model.ChatMessageTemplateData{
252257
SystemPrompt: config.SystemPrompt,
253258
Role: r,
254259
RoleName: role,
255260
Content: i.StringContent,
256-
FunctionCall: i.FunctionCall,
261+
FunctionCall: fcall,
257262
FunctionName: i.Name,
258263
LastMessage: messageIndex == (len(input.Messages) - 1),
259264
Function: config.Grammar != "" && (messageIndex == (len(input.Messages) - 1)),
@@ -271,35 +276,49 @@ func ChatEndpoint(cl *config.BackendConfigLoader, ml *model.ModelLoader, startup
271276
content = templatedChatMessage
272277
}
273278
}
279+
280+
marshalAnyRole := func(f any) {
281+
j, err := json.Marshal(f)
282+
if err == nil {
283+
if contentExists {
284+
content += "\n" + fmt.Sprint(r, " ", string(j))
285+
} else {
286+
content = fmt.Sprint(r, " ", string(j))
287+
}
288+
}
289+
}
290+
marshalAny := func(f any) {
291+
j, err := json.Marshal(f)
292+
if err == nil {
293+
if contentExists {
294+
content += "\n" + string(j)
295+
} else {
296+
content = string(j)
297+
}
298+
}
299+
}
274300
// If this model doesn't have such a template, or if that template fails to return a value, template at the message level.
275301
if content == "" {
276302
if r != "" {
277303
if contentExists {
278304
content = fmt.Sprint(r, i.StringContent)
279305
}
306+
280307
if i.FunctionCall != nil {
281-
j, err := json.Marshal(i.FunctionCall)
282-
if err == nil {
283-
if contentExists {
284-
content += "\n" + fmt.Sprint(r, " ", string(j))
285-
} else {
286-
content = fmt.Sprint(r, " ", string(j))
287-
}
288-
}
308+
marshalAnyRole(i.FunctionCall)
309+
}
310+
if i.ToolCalls != nil {
311+
marshalAnyRole(i.ToolCalls)
289312
}
290313
} else {
291314
if contentExists {
292315
content = fmt.Sprint(i.StringContent)
293316
}
294317
if i.FunctionCall != nil {
295-
j, err := json.Marshal(i.FunctionCall)
296-
if err == nil {
297-
if contentExists {
298-
content += "\n" + string(j)
299-
} else {
300-
content = string(j)
301-
}
302-
}
318+
marshalAny(i.FunctionCall)
319+
}
320+
if i.ToolCalls != nil {
321+
marshalAny(i.ToolCalls)
303322
}
304323
}
305324
// Special Handling: System. We care if it was printed at all, not the r branch, so check seperately

‎embedded/models/hermes-2-pro-mistral.yaml

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,30 +3,27 @@ mmap: true
33
parameters:
44
model: huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q6_K.gguf
55

6-
roles:
7-
assistant_function_call: assistant
8-
function: tool
96
template:
107
chat_message: |
11-
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "function"}}{{.Role}}{{else if eq .RoleName "user"}}user{{end}}
12-
{{ if eq .RoleName "assistant_function_call" }}<tool_call>{{end}}
13-
{{ if eq .RoleName "function" }}<tool_result>{{end}}
8+
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}
9+
{{ if .FunctionCall }}<tool_call>{{end}}
10+
{{ if eq .RoleName "tool" }}<tool_result>{{end}}
1411
{{if .Content}}{{.Content}}{{end}}
1512
{{if .FunctionCall}}{{toJson .FunctionCall}}{{end}}
16-
{{ if eq .RoleName "assistant_function_call" }}</tool_call>{{end}}
17-
{{ if eq .RoleName "function" }}</tool_result>{{end}}
13+
{{ if .FunctionCall }}</tool_call>{{end}}
14+
{{ if eq .RoleName "tool" }}</tool_result>{{end}}
1815
<|im_end|>
1916
# https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF#prompt-format-for-function-calling
2017
function: |
2118
<|im_start|>system
22-
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
19+
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
2320
<tools>
2421
{{range .Functions}}
2522
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
2623
{{end}}
27-
</tools>
28-
Use the following pydantic model json schema for each tool call you will make:
29-
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
24+
</tools>
25+
Use the following pydantic model json schema for each tool call you will make:
26+
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
3027
For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
3128
<tool_call>
3229
{'arguments': <args-dict>, 'name': <function-name>}

0 commit comments

Comments
 (0)