31c579503d · tduncan/Nvidia-Optimization-Stack-VLLM

@@ -314,3 +314,82 @@ lm_eval --model vllm \

314 --tasks mmlu \

315 --batch_size auto

316 ```

317 +

318 +## Online Quantization

319 +

320 +All the workflows above are *offline* quantization: you run a script, write a new quantized checkpoint to disk, and later load it for serving. This produces the most accurate and smallest checkpoints because it can run accuracy-recovery algorithms (rotation, SmoothQuant, GPTQ, AWQ) during export, but it requires a separate quantization step and a second copy of the model on disk.

321 +

322 +*Online* quantization instead quantizes the weights at load time, directly from a high-precision checkpoint, and offers several advantages over the offline flow:

323 +

324 +- **No export step** — serve directly from the original `bf16`/`fp16` checkpoint; no separate quantization run before deployment.

325 +- **No extra disk footprint** — nothing new is written to disk, so there is no second copy of the model to store or manage.

326 +- **No calibration data** — activations are scaled dynamically at runtime, so no calibration dataset is needed.

327 +- **Fast iteration** — switch schemes or per-layer selections instantly by changing a config, without re-exporting.

328 +

329 +### vLLM online quantization

330 +

331 +vLLM has [built-in online quantization](online.md) that skips the export step: it loads an ordinary high-precision (`bf16`) checkpoint and quantizes each layer's weights to schemes such as FP8 or MXFP4 at load time, without a pre-quantized checkpoint or calibration data. No new checkpoint is written to disk. See [Online Quantization](online.md) for the supported schemes and configuration.

332 +

333 +### Quark online quantization

334 +

335 +AMD Quark provides its own online path, exposed as the `quark_online` quantization backend, for users who want parity with Quark's offline export and Quark's per-layer mixed-precision configs. Like vLLM's built-in online quantization, it quantizes weights inside the weight-loading hook just before serving and writes no new checkpoint.

336 +

337 +Its quant math is aligned byte-for-byte with Quark's offline export, so what you validate online is what you get offline. Use it to *find* the scheme and per-layer selection you want.

338 +

339 +Compared to vLLM's built-in online quantization, the `quark_online` plugin adds:

340 +

341 +- **Flexible config parsing** — a terse config expands into Quark's verbose per-layer config, delegating all per-layer matching to a real `QuarkConfig`.

342 +- **Per-layer / mixed schemes** — dispatch a different method per layer (e.g. MXFP4 experts with FP8 attention on an MoE model). Each online method subclasses the matching offline scheme, so a load-time quantized layer runs the identical inference kernel as an offline one.

343 +- **Re-quantizing an already-quantized checkpoint** — an FP8 block-scale checkpoint (e.g. DeepSeek-R1) is dequantized and re-quantized to a target scheme layer-locally at load time, with no new checkpoint.

344 +

345 +The plugin ships in AMD Quark — no fork of vLLM, no patched checkpoint format; see the [Quark documentation](https://quark.docs.amd.com/latest/) for details. Three presets ship ready to use:

346 +

347 +| Preset key | Scheme |

348 +| --- | --- |

349 +| `ptpc_fp8` | FP8 E4M3, per-channel weight + dynamic per-token activation |

350 +| `mxfp4` | MXFP4, per-group (group size 32) with E8M0 block scale |

351 +| `linear_ptpc_fp8_moe_mxfp4` | Mixed: attention in FP8, MoE experts in MXFP4 |

352 +

353 +### Python API

354 +

355 +```python

356 +from vllm import LLM, SamplingParams

357 +from quark.online_quantization.vllm import HF_QUANTIZATION_CONFIGS

358 +

359 +llm = LLM(

360 + model="Qwen/Qwen3-30B-A3B-Thinking-2507",

361 + quantization="quark_online",

362 + hf_overrides=HF_QUANTIZATION_CONFIGS["ptpc_fp8"],

363 + enforce_eager=True,

364 + tensor_parallel_size=1,

365 +)

366 +out = llm.generate(["The capital of France is"], SamplingParams(temperature=0.0, max_tokens=100))

367 +print(out)

368 +```

369 +

370 +### Native vLLM CLI

371 +

372 +```bash

373 +export VLLM_PLUGINS="${VLLM_PLUGINS:-quark_online_quant}"

374 +ONLINE_QUANT_CONFIG='{"online_quant_config": {"global_quant_config": "ptpc_fp8", "exclude_layer": ["lm_head"]}}'

375 +

376 +vllm serve Qwen/Qwen3-8B \

377 + --trust-remote-code \

378 + --tensor-parallel-size 1 \

379 + --additional-config "$ONLINE_QUANT_CONFIG"

380 +```

381 +

382 +### Re-quantizing an offline checkpoint

383 +

384 +No extra arguments — the same `hf_overrides` detects the checkpoint's existing config and merges it automatically:

385 +

386 +```python

387 +llm = LLM(

388 + model="deepseek-ai/DeepSeek-R1", # ships quant_method: "fp8"

389 + quantization="quark_online",

390 + hf_overrides=HF_QUANTIZATION_CONFIGS["ptpc_fp8"],

391 + tensor_parallel_size=8,

392 +)

393 +```

394 +

395 +The plugin can be disabled with `QUARK_DISABLE_VLLM_PLUGIN=1`, and it steps aside automatically if another platform plugin (e.g. ATOM) has already taken over the backend.

update quark docs to include online quantization (#52736)