@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / da6f409246
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

Update deploying_with_k8s.rst (#10922)

AlexHe99 committed Dec 16, 2024 at 08:33 UTC da6f40924609e084ced486cae5b4ddf97133acd9
1 file changed +2 -2
docs/source/serving/deploying_with_k8s.rst
+2 -2
@@ -162,7 +162,7 @@ To test the deployment, run the following ``curl`` command:
162 curl http://mistral-7b.default.svc.cluster.local/v1/completions \
163 -H "Content-Type: application/json" \
164 -d '{
165 - "model": "facebook/opt-125m",
165 + "model": "mistralai/Mistral-7B-Instruct-v0.3",
166 "prompt": "San Francisco is a",
167 "max_tokens": 7,
168 "temperature": 0
@@ -172,4 +172,4 @@ If the service is correctly deployed, you should receive a response from the vLL
172
173 Conclusion
174 ----------
175 -Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation.
\ No newline at end of file
175 +Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation.