@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a324d3a1a7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

Change granite chat template to keep json list formatting for tool calls (#10452)

Signed-off-by: Max de Bayser <maxdebayser@gmail.com>

Maximilien de Bayser committed Nov 19, 2024 at 22:16 UTC a324d3a1a74ab0a3fafc0f2d19860bd1d1301a85
1 file changed +1 -5
examples/tool_chat_template_granite.jinja
+1 -5
@@ -21,11 +21,7 @@
21 {{- '<|start_of_role|>user<|end_of_role|>' + message['content'] + '<|end_of_text|>
22 ' }}
23 {%- elif message['role'] == 'assistant_tool_call' or (message['role'] == 'assistant' and message.tool_calls is defined) %}
24 - {{- '<|start_of_role|>assistant<|end_of_role|>' }}
25 - {% for tc in message.tool_calls %}
26 - {{- '<|tool_call|> ' + {'name': tc.function.name, 'arguments': tc.function.arguments}|tojson }}
27 - {% endfor %}
28 - {{- '<|end_of_text|>
24 + {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message.tool_calls|map(attribute='function')|list|tojson(indent=4) + '<|end_of_text|>
25 ' }}
26 {%- elif message['role'] == 'assistant' %}
27 {{- '<|start_of_role|>assistant<|end_of_role|>' + message['content'] + '<|end_of_text|>