A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
docs: add instruction for langchain (#1162)
Massimiliano Pronesti committed
Nov 30, 2023 at 19:57 UTC
05a38612b0f2895e480f08dfd77d1b5697c3ed42
2 files changed
+32
docs/source/index.rst
+1
@@ -66,6 +66,7 @@ Documentation
66
serving/run_on_sky
67
serving/deploying_with_triton
68
serving/deploying_with_docker
69
+ serving/serving_with_langchain
70
71
.. toctree::
72
:maxdepth: 1
docs/source/serving/serving_with_langchain.rst
new
+31
@@ -0,0 +1,31 @@
1
+.. _run_on_langchain:
2
+
3
+Serving with Langchain
4
+============================
5
+
6
+vLLM is also available via `Langchain <https://github.com/langchain-ai/langchain>`_ .
7
+
8
+To install langchain, run
9
+
10
+.. code-block:: console
11
+
12
+ $ pip install langchain -q
13
+
14
+To run inference on a single or multiple GPUs, use ``VLLM`` class from ``langchain``.
15
+
16
+.. code-block:: python
17
+
18
+ from langchain.llms import VLLM
19
+
20
+ llm = VLLM(model="mosaicml/mpt-7b",
21
+ trust_remote_code=True, # mandatory for hf models
22
+ max_new_tokens=128,
23
+ top_k=10,
24
+ top_p=0.95,
25
+ temperature=0.8,
26
+ # tensor_parallel_size=... # for distributed inference
27
+ )
28
+
29
+ print(llm("What is the capital of France ?"))
30
+
31
+Please refer to this `Tutorial <https://github.com/langchain-ai/langchain/blob/master/docs/extras/integrations/llms/vllm.ipynb>`_ for more details.
\ No newline at end of file