@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 70 lines 2.42 KB
Rendered Raw
1 # Context Extension
2
3 !!! note
4 The `--rope-scaling` parameter used in older versions of vLLM is no longer supported. Please use the `--hf-overrides` method with `rope_parameters` instead.
5 This directory contains examples for extending the context length of models using vLLM.
6
7 ## Offline Inference Example
8
9 The [`context_extension.py`](../../examples/features/context_extension/context_extension_offline.py) script demonstrates how to extend the context length of a Qwen model using the YARN method (rope_parameters) and run a simple chat example.
10
11 ### Usage
12
13 ```bash
14 python examples/features/context_extension/context_extension_offline.py
15 ```
16
17 ## OpenAI Online Method
18
19 You can also use vLLM's OpenAI-compatible API to serve models with extended context length.
20
21 ### Usage
22
23 Run the vLLM server with the following command to extend the context length using YARN:
24
25 ```bash
26 vllm serve Qwen/Qwen3-0.6B \
27 --hf-overrides '{"rope_parameters": {"factor": 4.0, "original_max_position_embeddings": 32768, "rope_theta": 1000000, "rope_type": "yarn"}}' \
28 --max-model-len 131072
29 ```
30
31 ### Client Example
32
33 After starting the server, you can use the OpenAI Python client to interact with it:
34
35 ```python
36 from openai import OpenAI
37
38 client = OpenAI(
39 base_url="http://localhost:8000/v1",
40 api_key="token-abc123" # Dummy API key, required by the client
41 )
42
43 response = client.chat.completions.create(
44 model="Qwen/Qwen3-0.6B",
45 messages=[
46 {"role": "system", "content": "You are a helpful assistant"},
47 {"role": "user", "content": "Hello"}
48 ],
49 max_tokens=128,
50 temperature=0.8,
51 top_p=0.95
52 )
53
54 print(response.choices[0].message.content)
55 ```
56
57 ### Key Parameters
58
59 The available parameters depend on the `rope_type` you choose. For detailed information about all supported RoPE types and their specific parameters, please refer to the [Hugging Face Transformers RoPE documentation](https://huggingface.co/docs/transformers/main/en/internal/rope_utils#transformers.RopeParameters).
60
61 Common parameters include:
62
63 - `rope_type`: The type of RoPE implementation (e.g., "yarn", "linear", "dynamic")
64 - `factor`: The factor by which to extend the context length
65 - `original_max_position_embeddings`: The original maximum position embeddings of the model
66
67 The following parameters are specific to vLLM:
68
69 - `max_model_len`: The new maximum sequence length after extension (original * factor).
70 Used for KV cache pre‑allocation and request limit at serving time.