@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c5d7fb9ddc
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Doc] fix third-party model example (#9771)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

Russell Bryant committed Oct 28, 2024 at 22:39 UTC c5d7fb9ddc16d9eb68f1018cfb384faf3be301be
1 file changed +4 -2
docs/source/models/adding_model.rst
+4 -2
@@ -133,7 +133,9 @@ If you are running api server with :code:`vllm serve <args>`, you can wrap the e
133 from vllm import ModelRegistry
134 from your_code import YourModelForCausalLM
135 ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
136 - import runpy
137 - runpy.run_module('vllm.entrypoints.openai.api_server', run_name='__main__')
136 +
137 + if __name__ == '__main__':
138 + import runpy
139 + runpy.run_module('vllm.entrypoints.openai.api_server', run_name='__main__')
140
141 Save the above code in a file and run it with :code:`python your_file.py <args>`.