A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 19:45 UTC
Add RLHF document (#14482)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Mar 8, 2025 at 10:51 UTC
cfd0ae823472d7dc33b29864dd576d5c7b5f832f
3 files changed
+14
-1
docs/source/generate_examples.py
+2
-1
@@ -14,13 +14,14 @@ EXAMPLE_DOC_DIR = ROOT_DIR / "docs/source/getting_started/examples"
14
def fix_case(text: str) -> str:
15
subs = {
16
"api": "API",
17
- "Cli": "CLI",
17
+ "cli": "CLI",
18
"cpu": "CPU",
19
"llm": "LLM",
20
"tpu": "TPU",
21
"aqlm": "AQLM",
22
"gguf": "GGUF",
23
"lora": "LoRA",
24
+ "rlhf": "RLHF",
25
"vllm": "vLLM",
26
"openai": "OpenAI",
27
"multilora": "MultiLoRA",
docs/source/index.md
+1
@@ -105,6 +105,7 @@ features/compatibility_matrix
105
:maxdepth: 1
106
107
training/trl.md
108
+training/rlhf.md
109
110
:::
111
docs/source/training/rlhf.md
new
+11
@@ -0,0 +1,11 @@
1
+# Reinforcement Learning from Human Feedback
2
+
3
+Reinforcement Learning from Human Feedback (RLHF) is a technique that fine-tunes language models using human-generated preference data to align model outputs with desired behaviours.
4
+
5
+vLLM can be used to generate the completions for RLHF. The best way to do this is with libraries like [TRL](https://github.com/huggingface/trl), [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) and [verl](https://github.com/volcengine/verl).
6
+
7
+See the following basic examples to get started if you don't want to use an existing library:
8
+
9
+- [Training and inference processes are located on separate GPUs (inspired by OpenRLHF)](https://docs.vllm.ai/en/latest/getting_started/examples/rlhf.html)
10
+- [Training and inference processes are colocated on the same GPUs using Ray](https://docs.vllm.ai/en/latest/getting_started/examples/rlhf_colocate.html)
11
+- [Utilities for performing RLHF with vLLM](https://docs.vllm.ai/en/latest/getting_started/examples/rlhf_utils.html)