[docs] Add documentation for pynvvideocodec video decoding backend (#49066)
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Brandon Pelfrey committed
Jul 29, 2026 at 10:23 UTC
5c7a7f9462c97ff8b2f63e6458488b820f2bbd86
1 file changed
+56
-10
docs/features/multimodal_inputs.md
+56
-10
@@ -846,15 +846,17 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
846
vLLM decodes video bytes into frames using a selectable decoding backend. Five
847
backends are supported:
848
849
-- `opencv` (default): OpenCV-based decoder.
850
-- `pyav`: PyAV decoder.
851
-- `torchcodec`: TorchCodec (PyTorch-native) decoder.
852
-- `pynvvideocodec`: NVIDIA NVDEC-based decoder.
853
-- `deepstream`: NVIDIA DeepStream NVDEC-based decoder.
854
-
855
-The CPU backends are backed by FFmpeg. `torchcodec` lets you choose which FFmpeg
856
-version is used while `opencv` and `pyav` rely on whichever FFmpeg build they
857
-were linked against.
849
+| Backend | Device | Description |
850
+| --- | --- | --- |
851
+| `opencv` (default) | CPU | OpenCV-based decoder |
852
+| `pyav` | CPU | PyAV decoder |
853
+| `torchcodec` | CPU | TorchCodec (PyTorch-native) decoder |
854
+| `pynvvideocodec` | GPU | NVIDIA PyNvVideoCodec decoder |
855
+| `deepstream` | GPU | NVIDIA DeepStream decoder |
856
+
857
+The three CPU backends are ultimately backed by FFmpeg. `torchcodec` lets you
858
+choose which FFmpeg version is used while `opencv` and `pyav` rely on whichever
859
+FFmpeg build they were linked against.
860
861
Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
862
@@ -921,11 +923,55 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
923
924
Works with common video formats like MP4 when using OpenCV backends.
925
926
+#### GPU Video Decoding with PyNvVideoCodec (NVDEC)
927
+
928
+The `pynvvideocodec` backend uses NVIDIA NVDEC to decode the sampled video
929
+frames on the GPU before copying them into host memory for multimodal
930
+preprocessing. For workloads with large videos and relatively light inference,
931
+such as video tagging, this can alleviate bottlenecks in CPU-based video
932
+decoders.
933
+
934
+!!! warning
935
+ [CUDA Multi-Process Service (MPS)](https://docs.nvidia.com/deploy/mps/quick-start.html)
936
+ is required when using this backend. Video decoding runs in the API server
937
+ process while model serving runs in the engine process, so multiple CUDA
938
+ processes share the same GPU. Configure and start MPS before starting vLLM.
939
+
940
+You must also set a positive `--mm-ipc-gpu-memory-gb` value to reserve VRAM for
941
+video decoding. vLLM carves this budget out of the memory available to the KV
942
+cache and uses it to bound concurrent frontend decode allocations. If the
943
+budget is exhausted, decode work waits instead of consuming the engine's VRAM
944
+headroom and potentially causing an out-of-memory error while serving requests.
945
+
946
+Select the backend with an environment variable and specify a workload-appropriate
947
+VRAM budget. For example, to reserve 1 GiB:
948
+
949
+```bash
950
+export VLLM_VIDEO_LOADER_BACKEND=pynvvideocodec
951
+vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
952
+ --mm-ipc-gpu-memory-gb 1
953
+```
954
+
955
+Alternatively, select it with `--media-io-kwargs`:
956
+
957
+```bash
958
+vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
959
+ --media-io-kwargs '{"video": {"backend": "pynvvideocodec"}}' \
960
+ --mm-ipc-gpu-memory-gb 1
961
+```
962
+
963
+Choose a budget large enough for the largest sampled video that a single API
964
+server process must decode. When using multiple API server processes, vLLM
965
+divides the configured budget evenly among them.
966
+
967
+For streaming video sources, use the DeepStream backend instead.
968
+
969
#### GPU Video Decoding with DeepStream (NVDEC)
970
971
By default vLLM decodes video on the CPU. On NVIDIA GPUs you can instead decode
972
directly on the hardware video engine (NVDEC) with the DeepStream backend, which
928
-keeps decoding off the CPU and can significantly increase video throughput.
973
+keeps decoding off the CPU and can significantly increase video throughput. It
974
+is the recommended GPU backend for streaming video sources.
975
976
Install the backend (Linux x86-64 only):
977