@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 05a38612b0
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

docs: add instruction for langchain (#1162)

Massimiliano Pronesti committed Nov 30, 2023 at 19:57 UTC 05a38612b0f2895e480f08dfd77d1b5697c3ed42
2 files changed +32
docs/source/index.rst
+1
@@ -66,6 +66,7 @@ Documentation
66 serving/run_on_sky
67 serving/deploying_with_triton
68 serving/deploying_with_docker
69 + serving/serving_with_langchain
70
71 .. toctree::
72 :maxdepth: 1
docs/source/serving/serving_with_langchain.rst new
+31
@@ -0,0 +1,31 @@
1 +.. _run_on_langchain:
2 +
3 +Serving with Langchain
4 +============================
5 +
6 +vLLM is also available via `Langchain <https://github.com/langchain-ai/langchain>`_ .
7 +
8 +To install langchain, run
9 +
10 +.. code-block:: console
11 +
12 + $ pip install langchain -q
13 +
14 +To run inference on a single or multiple GPUs, use ``VLLM`` class from ``langchain``.
15 +
16 +.. code-block:: python
17 +
18 + from langchain.llms import VLLM
19 +
20 + llm = VLLM(model="mosaicml/mpt-7b",
21 + trust_remote_code=True, # mandatory for hf models
22 + max_new_tokens=128,
23 + top_k=10,
24 + top_p=0.95,
25 + temperature=0.8,
26 + # tensor_parallel_size=... # for distributed inference
27 + )
28 +
29 + print(llm("What is the capital of France ?"))
30 +
31 +Please refer to this `Tutorial <https://github.com/langchain-ai/langchain/blob/master/docs/extras/integrations/llms/vllm.ipynb>`_ for more details.
\ No newline at end of file