A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 4, 16:15 UTC
[Docs] Add docs for context extension using the yarn method (#37430)
Signed-off-by: xiaoming <1259730330@qq.com> Signed-off-by: labAxiaoming <34019940+labAxiaoming@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
labAxiaoming committed
Apr 24, 2026 at 23:26 UTC
f768b4473e1bd55023dcaff63984cfdd08902fc8
1 file changed
+70
docs/features/context_extension.md
new
+70
@@ -0,0 +1,70 @@
1
+# Context Extension
2
+
3
+!!! note
4
+ The `--rope-scaling` parameter used in older versions of vLLM is no longer supported. Please use the `--hf-overrides` method with `rope_parameters` instead.
5
+This directory contains examples for extending the context length of models using vLLM.
6
+
7
+## Offline Inference Example
8
+
9
+The [`context_extension.py`](../../examples/offline_inference/context_extension) script demonstrates how to extend the context length of a Qwen model using the YARN method (rope_parameters) and run a simple chat example.
10
+
11
+### Usage
12
+
13
+```bash
14
+python examples/offline_inference/context_extension.py
15
+```
16
+
17
+## OpenAI Online Method
18
+
19
+You can also use vLLM's OpenAI-compatible API to serve models with extended context length.
20
+
21
+### Usage
22
+
23
+Run the vLLM server with the following command to extend the context length using YARN:
24
+
25
+```bash
26
+vllm serve Qwen/Qwen3-0.6B \
27
+ --hf-overrides '{"rope_parameters": {"factor": 4.0, "original_max_position_embeddings": 32768, "rope_theta": 1000000, "rope_type": "yarn"}}' \
28
+ --max-model-len 131072
29
+```
30
+
31
+### Client Example
32
+
33
+After starting the server, you can use the OpenAI Python client to interact with it:
34
+
35
+```python
36
+from openai import OpenAI
37
+
38
+client = OpenAI(
39
+ base_url="http://localhost:8000/v1",
40
+ api_key="token-abc123" # Dummy API key, required by the client
41
+)
42
+
43
+response = client.chat.completions.create(
44
+ model="Qwen/Qwen3-0.6B",
45
+ messages=[
46
+ {"role": "system", "content": "You are a helpful assistant"},
47
+ {"role": "user", "content": "Hello"}
48
+ ],
49
+ max_tokens=128,
50
+ temperature=0.8,
51
+ top_p=0.95
52
+)
53
+
54
+print(response.choices[0].message.content)
55
+```
56
+
57
+### Key Parameters
58
+
59
+The available parameters depend on the `rope_type` you choose. For detailed information about all supported RoPE types and their specific parameters, please refer to the [Hugging Face Transformers RoPE documentation](https://huggingface.co/docs/transformers/main/en/internal/rope_utils#transformers.RopeParameters).
60
+
61
+Common parameters include:
62
+
63
+- `rope_type`: The type of RoPE implementation (e.g., "yarn", "linear", "dynamic")
64
+- `factor`: The factor by which to extend the context length
65
+- `original_max_position_embeddings`: The original maximum position embeddings of the model
66
+
67
+The following parameters are specific to vLLM:
68
+
69
+- `max_model_len`: The new maximum sequence length after extension (original * factor).
70
+ Used for KV cache pre‑allocation and request limit at serving time.