| 1 | # Reproducibility |
| 2 | |
| 3 | vLLM does not guarantee the reproducibility of the results by default, for the sake of performance. To achieve |
| 4 | reproducible results: |
| 5 | |
| 6 | - In offline mode, you can either set `VLLM_ENABLE_V1_MULTIPROCESSING=0` which makes scheduling deterministic, |
| 7 | or enable [batch invariance](../features/batch_invariance.md) to make the outputs insensitive to scheduling. |
| 8 | - In online mode, you can only enable [batch invariance](../features/batch_invariance.md). |
| 9 | |
| 10 | Example: [examples/features/batch_invariance/reproducibility_offline.py](../../examples/features/batch_invariance/reproducibility_offline.py) |
| 11 | |
| 12 | !!! warning |
| 13 | |
| 14 | Setting `VLLM_ENABLE_V1_MULTIPROCESSING=0` will change the random state of user code |
| 15 | (i.e. the code that constructs [LLM][vllm.LLM] class). |
| 16 | |
| 17 | !!! note |
| 18 | |
| 19 | Even with the above settings, vLLM only provides reproducibility |
| 20 | when it runs on the same hardware and the same vLLM version. |
| 21 | |
| 22 | ## Setting the global seed |
| 23 | |
| 24 | The `seed` parameter in vLLM is used to control the random states for various random number generators. |
| 25 | |
| 26 | If a specific seed value is provided, the random states for `random`, `np.random`, and `torch.manual_seed` will be set accordingly. |
| 27 | |
| 28 | ### Default Behavior |
| 29 | |
| 30 | In V1, the `seed` parameter defaults to `0` which sets the random state for each worker, so the results will remain consistent for each vLLM run even if `temperature > 0`. |
| 31 | |
| 32 | It is impossible to un-specify a seed for V1 because different workers need to sample the same outputs |
| 33 | for workflows such as speculative decoding. For more information, see: <https://github.com/vllm-project/vllm/pull/17929> |
| 34 | |
| 35 | !!! note |
| 36 | |
| 37 | The random state in user code (i.e. the code that constructs [LLM][vllm.LLM] class) is updated by vLLM |
| 38 | only if the workers are run in the same process as user code, i.e.: `VLLM_ENABLE_V1_MULTIPROCESSING=0`. |
| 39 | |
| 40 | By default, `VLLM_ENABLE_V1_MULTIPROCESSING=1` so you can use vLLM without having to worry about |
| 41 | accidentally making deterministic subsequent operations that rely on random state. |