A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 12:15 UTC
[doc] update wrong model id (#17287)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Apr 28, 2025 at 19:20 UTC
3ad986c28bc2c02d1734d45951e99632564c7a9e
1 file changed
+5
-3
docs/source/features/quantization/gptqmodel.md
+5
-3
@@ -58,7 +58,7 @@ model.save(quant_path)
58
To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:
59
60
```console
61
-python examples/offline_inference/llm_engine_example.py --model DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2
61
+python examples/offline_inference/llm_engine_example.py --model ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2
62
```
63
64
## Using GPTQModel with vLLM's Python API
@@ -80,15 +80,17 @@ prompts = [
80
sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
81
82
# Create an LLM.
83
-llm = LLM(model="DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
83
+llm = LLM(model="ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
84
85
# Generate texts from the prompts. The output is a list of RequestOutput objects
86
# that contain the prompt, generated text, and other information.
87
outputs = llm.generate(prompts, sampling_params)
88
89
# Print the outputs.
90
+print("-"*50)
91
for output in outputs:
92
prompt = output.prompt
93
generated_text = output.outputs[0].text
93
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
94
+ print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}")
95
+ print("-"*50)
96
```