A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 10, 12:45 UTC
[Doc] Add Claude code usage example (#31188)
Signed-off-by: mgoin <mgoin64@gmail.com>
Michael Goin committed
Jan 8, 2026 at 00:50 UTC
6b2a672e47f15854c2df5d61c8d57ca25547e50c
2 files changed
+75
docs/assets/deployment/claude-code-example.png
Binary files /dev/null and b/docs/assets/deployment/claude-code-example.png differ
docs/serving/integrations/claude_code.md
new
+75
@@ -0,0 +1,75 @@
1
+# Claude Code
2
+
3
+[Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) is Anthropic's official agentic coding tool that lives in your terminal. It can understand your codebase, edit files, run commands, and help you write code more efficiently.
4
+
5
+By pointing Claude Code at a vLLM server, you can use your own models as the backend instead of the Anthropic API. This is useful for:
6
+
7
+- Running fully local/private coding assistance
8
+- Using open-weight models with tool calling capabilities
9
+- Testing and developing with custom models
10
+
11
+## How It Works
12
+
13
+vLLM implements the Anthropic Messages API, which is the same API that Claude Code uses to communicate with Anthropic's servers. By setting `ANTHROPIC_BASE_URL` to point at your vLLM server, Claude Code sends its requests to vLLM instead of Anthropic. vLLM then translates these requests to work with your local model and returns responses in the format Claude Code expects.
14
+
15
+This means any model served by vLLM with proper tool calling support can act as a drop-in replacement for Claude models in Claude Code.
16
+
17
+## Requirements
18
+
19
+Claude Code requires a model with strong tool calling capabilities. The model must support the OpenAI-compatible tool calling API. See [Tool Calling](../../features/tool_calling.md) for details on enabling tool calling for your model.
20
+
21
+## Installation
22
+
23
+First, install Claude Code by following the [official installation guide](https://docs.anthropic.com/en/docs/claude-code/getting-started).
24
+
25
+## Starting the vLLM Server
26
+
27
+Start vLLM with a tool-calling capable model - here's an example using `openai/gpt-oss-120b`:
28
+
29
+```bash
30
+vllm serve openai/gpt-oss-120b --served-model-name my-model --enable-auto-tool-choice --tool-call-parser openai
31
+```
32
+
33
+For other models, you'll need to enable tool calling explicitly with `--enable-auto-tool-choice` and the right `--tool-call-parser`. Refer to the [Tool Calling documentation](../../features/tool_calling.md) for the correct flags for your model.
34
+
35
+## Configuring Claude Code
36
+
37
+Launch Claude Code with environment variables pointing to your vLLM server:
38
+
39
+```bash
40
+ANTHROPIC_BASE_URL=http://localhost:8000 \
41
+ANTHROPIC_API_KEY=dummy \
42
+ANTHROPIC_DEFAULT_OPUS_MODEL=my-model \
43
+ANTHROPIC_DEFAULT_SONNET_MODEL=my-model \
44
+ANTHROPIC_DEFAULT_HAIKU_MODEL=my-model \
45
+claude
46
+```
47
+
48
+The environment variables:
49
+
50
+| Variable | Description |
51
+| -------------------------------- | --------------------------------------------------------------------- |
52
+| `ANTHROPIC_BASE_URL` | Points to your vLLM server (default port is 8000) |
53
+| `ANTHROPIC_API_KEY` | Can be any value since vLLM doesn't require authentication by default |
54
+| `ANTHROPIC_DEFAULT_OPUS_MODEL` | Model name for Opus-tier requests |
55
+| `ANTHROPIC_DEFAULT_SONNET_MODEL` | Model name for Sonnet-tier requests |
56
+| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | Model name for Haiku-tier requests |
57
+
58
+!!! tip
59
+ You can add these environment variables to your shell profile (e.g., `.bashrc`, `.zshrc`), Claude Code configuration file (`~/.claude/settings.json`), or create a wrapper script for convenience.
60
+
61
+## Testing the Setup
62
+
63
+Once Claude Code launches, try a simple prompt to verify the connection:
64
+
65
+
66
+
67
+If the model responds correctly, your setup is working. You can now use Claude Code with your vLLM-served model for coding tasks.
68
+
69
+## Troubleshooting
70
+
71
+**Connection refused**: Ensure vLLM is running and accessible at the specified URL. Check that the port matches.
72
+
73
+**Tool calls not working**: Verify that your model supports tool calling and that you've enabled it with the correct `--tool-call-parser` flag. See [Tool Calling](../../features/tool_calling.md).
74
+
75
+**Model not found**: Ensure the `--served-model-name` matches the model names in your environment variables. You cannot use model names with `/` in them, such as `openai/gpt-oss-120b` directly from Huggingface, so beware of that limitation with Claude Code.