@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b5f49ee55b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

Update README.md (#6847)

Gurpreet Singh Dhami committed Jul 26, 2024 at 20:26 UTC b5f49ee55beac7bb314fd8880bdb718ed117dacb
1 file changed +1 -1
examples/fp8/quantizer/README.md
+1 -1
@@ -16,7 +16,7 @@
16 #### Run on H100 system for speed if FP8; number of GPUs depends on the model size
17
18 #### Example: quantize Llama2-7b model from HF to FP8 with FP8 KV Cache:
19 -`python quantize.py --model_dir ./ll2-7b --dtype float16 --qformat fp8 --kv_cache_dtype fp8 --output_dir ./ll2_7b_fp8 --calib_size 512 --tp_size 1`
19 +`python quantize.py --model-dir ./ll2-7b --dtype float16 --qformat fp8 --kv-cache-dtype fp8 --output-dir ./ll2_7b_fp8 --calib-size 512 --tp-size 1`
20
21 Outputs: model structure, quantized model & parameters (with scaling factors) are in JSON and Safetensors (npz is generated only for the reference)
22 ```