@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / fc542144c4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[Feature] Fix guided decoding blocking bitmask memcpy (#12563)

**[Guided decoding performance optimization]** Sending the guided decoding bitmask in xgrammar to the GPU (`self.token_bitmask.to(scores.device)`) is a blocking operation that prevents the CPU from pre-launching the sampler kernels. The CPU waits until decode is complete, then copies the bitmask over. This PR changes the operation to async via setting `non-blocking=True`. (Current) The CPU is blocked on a `cudaStreamSynchronize` and only pre-empts the sampling kernels after bitmask application. Below is the Nsys profile for one decode phase from Llama 3.1 8B. ![image](https://github.com/user-attachments/assets/8997eae1-b822-4f52-beb8-ef19a7c6b824) With the optimization, this is no longer the case: ![image](https://github.com/user-attachments/assets/6d5ea83f-f169-4f98-a8c1-41c719b3e1e7) --------- Signed-off-by: Ryan N <ryan.nguyen@centml.ai>

Ryan Nguyen committed Jan 31, 2025 at 18:37 UTC fc542144c4477ffec1d3de6fa43e54f8fb5351e8
1 file changed +2 -2
vllm/model_executor/guided_decoding/xgrammar_decoding.py
+2 -2
@@ -307,8 +307,8 @@ class XGrammarLogitsProcessor:
307 # Note: In this method, if the tensors have different dimensions
308 # on CPU device fails, but on GPU it runs without error. Hence the
309 # unsqueeze above for scores, to match the token bitmask shape
310 - xgr.apply_token_bitmask_inplace(scores,
311 - self.token_bitmask.to(scores.device))
310 + xgr.apply_token_bitmask_inplace(
311 + scores, self.token_bitmask.to(scores.device, non_blocking=True))
312 if device_type != "cuda":
313 scores = scores.to(dtype).to(device_type).squeeze()
314