@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 118 lines 4.13 KB
Rendered Raw
1 # Interleaved Thinking
2
3 ## Introduction
4
5 Interleaved thinking allows models to reason between tool calls, enabling more sophisticated decision-making after receiving tool results. This feature helps models chain multiple tool calls with reasoning steps in between and make nuanced decisions based on intermediate results.
6
7 Important: Interleaved thinking increases token usage and response latency. Consider your budget and performance requirements when enabling this feature.
8
9 ## How Interleaved Thinking Works
10
11 With interleaved thinking, the model can:
12
13 - Reason about the results of a tool call before deciding what to do next
14 - Chain multiple tool calls with reasoning steps in between
15 - Make more nuanced decisions based on intermediate results
16 - Provide transparent reasoning for its tool selection process
17
18 ## Supported Models
19
20 vLLM currently supports the following interleaved thinking models:
21
22 | Model Series | Reasoning Parser Name |
23 | ------------ | --------------------- |
24 | moonshotai/Kimi-K2-Thinking | kimi_k2 |
25 | MiniMaxAI/MiniMax-M2 | minimax_m2 |
26
27 ## Example Usage
28
29 To use interleaved thinking with tool calls, specify a model that supports this feature and enable tool calls in your chat completion request. Here's an example:
30
31 ??? code
32
33 ```python
34 """
35 vllm serve MiniMaxAI/MiniMax-M2 \
36 --tensor-parallel-size 4 \
37 --tool-call-parser minimax_m2 \
38 --reasoning-parser minimax_m2 \
39 --enable-auto-tool-choice
40 """
41 import json
42
43 from openai import OpenAI
44
45 client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
46
47
48 def get_current_weather(location: str, unit: "str"):
49 """Get the current weather in a given location"""
50 if unit == "celsius":
51 return f"The current temperature in {location} is 22°C."
52 else:
53 return f"The current temperature in {location} is 72°F."
54
55
56 tools = [
57 {
58 "type": "function",
59 "function": {
60 "name": "get_weather",
61 "description": "Get the current weather in a given location",
62 "parameters": {
63 "type": "object",
64 "properties": {
65 "location": {
66 "type": "string",
67 "description": "City and state, e.g., 'San Francisco, CA'",
68 },
69 "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
70 },
71 "required": ["location", "unit"],
72 },
73 },
74 }
75 ]
76 messages = [{"role": "user", "content": "What's the weather in Fahrenheit like in San Francisco?"}]
77 response = client.chat.completions.create(
78 model=client.models.list().data[0].id,
79 messages=messages,
80 tools=tools,
81 tool_choice="auto",
82 )
83
84 tool_call = response.choices[0].message.tool_calls[0].function
85
86 messages.append(
87 {
88 "role": "assistant",
89 "tool_calls": response.choices[0].message.tool_calls,
90 "reasoning": response.choices[0].message.reasoning, # append reasoning
91 }
92 )
93
94 # Simulate tool execution
95 available_tools = {"get_weather": get_current_weather}
96
97 completion_tool_calls = response.choices[0].message.tool_calls
98 for call in completion_tool_calls:
99 tool_to_call = available_tools[call.function.name]
100 args = json.loads(call.function.arguments)
101 result = tool_to_call(**args)
102 messages.append(
103 {
104 "role": "tool",
105 "content": result,
106 "tool_call_id": call.id,
107 "name": call.function.name,
108 }
109 )
110 response_2 = client.chat.completions.create(
111 model=client.models.list().data[0].id,
112 messages=messages,
113 tools=tools,
114 tool_choice="auto",
115 )
116 print(response_2.choices[0].message.content)
117 ```
118 This example demonstrates how to set up interleaved thinking with tool calls using a weather retrieval function. The model reasons about the tool results before generating the final response.