A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
Update deploying_with_k8s.rst (#10922)
AlexHe99 committed
Dec 16, 2024 at 08:33 UTC
da6f40924609e084ced486cae5b4ddf97133acd9
1 file changed
+2
-2
docs/source/serving/deploying_with_k8s.rst
+2
-2
@@ -162,7 +162,7 @@ To test the deployment, run the following ``curl`` command:
162
curl http://mistral-7b.default.svc.cluster.local/v1/completions \
163
-H "Content-Type: application/json" \
164
-d '{
165
- "model": "facebook/opt-125m",
165
+ "model": "mistralai/Mistral-7B-Instruct-v0.3",
166
"prompt": "San Francisco is a",
167
"max_tokens": 7,
168
"temperature": 0
@@ -172,4 +172,4 @@ If the service is correctly deployed, you should receive a response from the vLL
172
173
Conclusion
174
----------
175
-Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation.
\ No newline at end of file
175
+Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation.