@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 92cd2e2f21
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 03:00 UTC

[Doc] Fix getting stared to use publicly available model (#3963)

Frαnçois committed Apr 10, 2024 at 20:05 UTC 92cd2e2f21e8ec65b2cb635a9f15de38157a1359
1 file changed +4 -6
docs/source/serving/openai_compatible_server.md
+4 -6
@@ -4,7 +4,7 @@ vLLM provides an HTTP server that implements OpenAI's [Completions](https://plat
4
5 You can start the server using Python, or using [Docker](deploying_with_docker.rst):
6 ```bash
7 -python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --dtype float32 --api-key token-abc123
7 +python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.2 --dtype auto --api-key token-abc123
8 ```
9
10 To call the server, you can use the official OpenAI Python client library, or any other HTTP client.
@@ -16,9 +16,8 @@ client = OpenAI(
16 )
17
18 completion = client.chat.completions.create(
19 - model="meta-llama/Llama-2-7b-hf",
19 + model="mistralai/Mistral-7B-Instruct-v0.2",
20 messages=[
21 - {"role": "system", "content": "You are a helpful assistant."},
21 {"role": "user", "content": "Hello!"}
22 ]
23 )
@@ -38,9 +37,8 @@ Or directly merge them into the JSON payload if you are using HTTP call directly
37
38 ```python
39 completion = client.chat.completions.create(
41 - model="meta-llama/Llama-2-7b-hf",
40 + model="mistralai/Mistral-7B-Instruct-v0.2",
41 messages=[
43 - {"role": "system", "content": "You are a helpful assistant."},
42 {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
43 ],
44 extra_body={
@@ -89,7 +87,7 @@ In order for the language model to support chat protocol, vLLM requires the mode
87 a chat template in its tokenizer configuration. The chat template is a Jinja2 template that
88 specifies how are roles, messages, and other chat-specific tokens are encoded in the input.
89
92 -An example chat template for `meta-llama/Llama-2-7b-chat-hf` can be found [here](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/09bd0f49e16738cdfaa6e615203e126038736eb0/tokenizer_config.json#L12)
90 +An example chat template for `mistralai/Mistral-7B-Instruct-v0.2` can be found [here](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2#instruction-format)
91
92 Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those model,
93 you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat