A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 17:15 UTC
[Minor] Remove TGI launching script (#15646)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Woosuk Kwon committed
Mar 28, 2025 at 09:30 UTC
70e132244a61425a6b88c0b8345e496dc5bdfecd
3 files changed
-22
benchmarks/benchmark_serving.py
-3
@@ -7,9 +7,6 @@ On the server side, run one of the following commands:
7
--swap-space 16 \
8
--disable-log-requests
9
10
- (TGI backend)
11
- ./launch_tgi_server.sh <your_model> <max_batch_total_tokens>
12
-
10
On the client side, run:
11
python benchmarks/benchmark_serving.py \
12
--backend <backend> \
benchmarks/benchmark_serving_structured_output.py
-3
@@ -5,9 +5,6 @@ On the server side, run one of the following commands:
5
(vLLM OpenAI API server)
6
vllm serve <your_model> --disable-log-requests
7
8
- (TGI backend)
9
- ./launch_tgi_server.sh <your_model> <max_batch_total_tokens>
10
-
8
On the client side, run:
9
python benchmarks/benchmark_serving_structured_output.py \
10
--backend <backend> \
benchmarks/launch_tgi_server.sh
deleted
-16
@@ -1,16 +0,0 @@
1
-#!/bin/bash
2
-
3
-PORT=8000
4
-MODEL=$1
5
-TOKENS=$2
6
-
7
-docker run -e "HF_TOKEN=$HF_TOKEN" --gpus all --shm-size 1g -p $PORT:80 \
8
- -v "$PWD/data:/data" \
9
- ghcr.io/huggingface/text-generation-inference:2.2.0 \
10
- --model-id "$MODEL" \
11
- --sharded false \
12
- --max-input-length 1024 \
13
- --max-total-tokens 2048 \
14
- --max-best-of 5 \
15
- --max-concurrent-requests 5000 \
16
- --max-batch-total-tokens "$TOKENS"