A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 21:00 UTC
[Doc] Correct beam_search using in generative_models.md (#14363)
Irina Yuryeva committed
Mar 6, 2025 at 18:37 UTC
4f27044aab45d4e1c1a74fdc82217eadb0384411
1 file changed
+6
-4
docs/source/models/generative_models.md
+6
-4
@@ -54,14 +54,16 @@ The {class}`~vllm.LLM.beam_search` method implements [beam search](https://huggi
54
For example, to search using 5 beams and output at most 50 tokens:
55
56
```python
57
+from vllm import LLM
58
+from vllm.sampling_params import BeamSearchParams
59
+
60
llm = LLM(model="facebook/opt-125m")
61
params = BeamSearchParams(beam_width=5, max_tokens=50)
59
-outputs = llm.generate("Hello, my name is", params)
62
+outputs = llm.beam_search([{"prompt": "Hello, my name is "}], params)
63
64
for output in outputs:
62
- prompt = output.prompt
63
- generated_text = output.outputs[0].text
64
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
65
+ generated_text = output.sequences[0].text
66
+ print(f"Generated text: {generated_text!r}")
67
```
68
69
### `LLM.chat`