@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / ecb33a28cb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

[CI/Build][Doc] Update CI and Doc for VLM example changes (#6860)

Roger Wang committed Jul 27, 2024 at 02:54 UTC ecb33a28cb6c10ebf3b1aa139f72e759cacb8c15
2 files changed +2 -3
.buildkite/test-pipeline.yaml
+1 -2
@@ -140,14 +140,13 @@ steps:
140 working_dir: "/vllm-workspace/examples"
141 mirror_hardwares: [amd]
142 commands:
143 - # install aws cli for llava_example.py
143 # install tensorizer for tensorize_vllm_model.py
144 - pip install awscli tensorizer
145 - python3 offline_inference.py
146 - python3 cpu_offload.py
147 - python3 offline_inference_with_prefix.py
148 - python3 llm_engine_example.py
150 - - python3 llava_example.py
149 + - python3 offline_inference_vision_language.py
150 - python3 tensorize_vllm_model.py --model facebook/opt-125m serialize --serialized-directory /tmp/ --suffix v1 && python3 tensorize_vllm_model.py --model facebook/opt-125m deserialize --path-to-tensors /tmp/vllm/facebook/opt-125m/v1/model.tensors
151
152 - label: Inputs Test
docs/source/models/vlm.rst
+1 -1
@@ -73,7 +73,7 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptI
73 generated_text = o.outputs[0].text
74 print(generated_text)
75
76 -A code example can be found in `examples/llava_example.py <https://github.com/vllm-project/vllm/blob/main/examples/llava_example.py>`_.
76 +A code example can be found in `examples/offline_inference_vision_language.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py>`_.
77
78
79 Online OpenAI Vision API Compatible Inference