@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 49 lines 1.88 KB
Rendered Raw
1 # Custom Arguments
2
3 You can use vLLM *custom arguments* to pass in arguments which are not part of the vLLM `SamplingParams` and REST API specifications. Adding or removing a vLLM custom argument does not require recompiling vLLM, since the custom arguments are passed in as a dictionary.
4
5 Custom arguments can be useful if, for example, you want to use a [custom logits processor](./custom_logitsprocs.md) without modifying the vLLM source code.
6
7 !!! note
8 Make sure your custom logits processor have implemented `validate_params` for custom arguments. Otherwise, invalid custom arguments can cause unexpected behaviour.
9
10 ## Offline Custom Arguments
11
12 Custom arguments passed to `SamplingParams.extra_args` as a `dict` will be visible to any code which has access to `SamplingParams`:
13
14 ``` python
15 SamplingParams(extra_args={"your_custom_arg_name": 67})
16 ```
17
18 This allows arguments which are not already part of `SamplingParams` to be passed into `LLM` as part of a request.
19
20 ## Online Custom Arguments
21
22 The OpenAI-compatible REST API and Anthropic-compatible `/v1/messages` endpoint allow custom arguments to be passed to the vLLM server via `vllm_xargs`. The example below integrates custom arguments into a vLLM REST API request:
23
24 ``` bash
25 curl http://localhost:8000/v1/completions \
26 -H "Content-Type: application/json" \
27 -d '{
28 "model": "Qwen/Qwen2.5-1.5B-Instruct",
29 ...
30 "vllm_xargs": {"your_custom_arg": 67}
31 }'
32 ```
33
34 Furthermore, OpenAI SDK users can access `vllm_xargs` via the `extra_body` argument:
35
36 ``` python
37 batch = await client.completions.create(
38 model="Qwen/Qwen2.5-1.5B-Instruct",
39 ...,
40 extra_body={
41 "vllm_xargs": {
42 "your_custom_arg": 67
43 }
44 }
45 )
46 ```
47
48 !!! note
49 `vllm_xargs` is assigned to `SamplingParams.extra_args` under the hood, so code which uses `SamplingParams.extra_args` is compatible with both offline and online scenarios.