@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4bfa7e7f75
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[Doc] Add API reference for offline inference (#4710)

Cyrus Leung committed May 14, 2024 at 08:47 UTC 4bfa7e7f75eb5b1a397c93aeea1dea1afa867b2a
4 files changed +17 -5
docs/source/index.rst
+7 -1
@@ -67,6 +67,13 @@ Documentation
67 getting_started/quickstart
68 getting_started/examples/examples_index
69
70 +.. toctree::
71 + :maxdepth: 1
72 + :caption: Offline Inference
73 +
74 + offline_inference/llm
75 + offline_inference/sampling_params
76 +
77 .. toctree::
78 :maxdepth: 1
79 :caption: Serving
@@ -101,7 +108,6 @@ Documentation
108 :maxdepth: 2
109 :caption: Developer Documentation
110
104 - dev/sampling_params
111 dev/engine/engine_index
112 dev/kernel/paged_attention
113 dev/dockerfile/dockerfile
docs/source/offline_inference/llm.rst new
+6
@@ -0,0 +1,6 @@
1 +LLM Class
2 +==========
3 +
4 +.. autoclass:: vllm.LLM
5 + :members:
6 + :show-inheritance:
docs/source/offline_inference/sampling_params.rst renamed
+2 -2
@@ -1,5 +1,5 @@
1 -Sampling Params
2 -===============
1 +Sampling Parameters
2 +===================
3
4 .. autoclass:: vllm.SamplingParams
5 :members:
docs/source/serving/openai_compatible_server.md
+2 -2
@@ -48,7 +48,7 @@ completion = client.chat.completions.create(
48 ```
49
50 ### Extra Parameters for Chat API
51 -The following [sampling parameters (click through to see documentation)](../dev/sampling_params.rst) are supported.
51 +The following [sampling parameters (click through to see documentation)](../offline_inference/sampling_params.rst) are supported.
52
53 ```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
54 :language: python
@@ -65,7 +65,7 @@ The following extra parameters are supported:
65 ```
66
67 ### Extra Parameters for Completions API
68 -The following [sampling parameters (click through to see documentation)](../dev/sampling_params.rst) are supported.
68 +The following [sampling parameters (click through to see documentation)](../offline_inference/sampling_params.rst) are supported.
69
70 ```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
71 :language: python