| 1 | # Weight Checker |
| 2 | |
| 3 | The Weight Checker verifies that an RL weight update landed on every rank. It |
| 4 | hashes each model parameter with SHA-256, so a caller can save the original |
| 5 | digests, reset the weights, transfer them back, and confirm they match. |
| 6 | |
| 7 | Enable it with `VLLM_SERVER_DEV_MODE=1`. All operations use |
| 8 | `POST /weight_checker`: |
| 9 | |
| 10 | | Request | Response | |
| 11 | | --- | --- | |
| 12 | | `{"action": "checksum"}` | `{"checksums": {key: sha256_hex}}` | |
| 13 | | `{"action": "reset"}` | `{"status": "reset"}` | |
| 14 | | `{"action": "compare", "baseline": {key: sha256_hex}}` | `{"match": bool, "mismatches": [key]}` | |
| 15 | |
| 16 | Keys have the form `dp{dp}:pp{pp}:pcp{pcp}:tp{tp}:{tensor_name}`, so each |
| 17 | shard is checked separately. All actions cover the target model's parameters, |
| 18 | which is what weight loading writes; `reset` zeroes them. The endpoint keeps no |
| 19 | state, so the caller holds the baseline. Invalid requests return HTTP 400. |
| 20 | |
| 21 | ## RL weight-update check |
| 22 | |
| 23 | ```bash |
| 24 | URL=http://localhost:8000 |
| 25 | |
| 26 | # 1. Stop serving, then save the original digests. |
| 27 | curl -X POST "$URL/pause?mode=abort" |
| 28 | curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \ |
| 29 | -d '{"action":"checksum"}' > baseline.json |
| 30 | |
| 31 | # 2. Zero the weights. |
| 32 | curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \ |
| 33 | -d '{"action":"reset"}' |
| 34 | |
| 35 | # 3. Transfer the original weights with start/update/finish_weight_update. |
| 36 | |
| 37 | # 4. Compare against the baseline; expect {"match": true, "mismatches": []}. |
| 38 | curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \ |
| 39 | -d "{\"action\":\"compare\",\"baseline\":$(jq .checksums baseline.json)}" |
| 40 | |
| 41 | curl -X POST $URL/resume |
| 42 | ``` |
| 43 | |
| 44 | ## Limitations |
| 45 | |
| 46 | - Stay paused from `checksum` to `compare`: the weights are invalid between |
| 47 | `reset` and the transfer, and EPLB moves experts while serving. |
| 48 | - A request covers the engines managed by the API server it reaches. With |
| 49 | several API servers (for example `--data-parallel-external-lb`), send it to |
| 50 | each one. |
| 51 | - Buffers, draft models and LoRA adapters are not checked. |
| 52 | - CPU backends that repack linear weights hide them from the checker. |
| 53 | - compressed-tensors online transforms are not supported: their Hadamard |
| 54 | weights are normalized once per storage, so restored copies differ. |
| 55 | - MoE weights padded beyond the checkpoint shape (DeepEP hidden size, MXFP4) |
| 56 | keep uninitialized padding, so `compare` reports them as mismatches. |
| 57 | - The engine must be awake: sleep levels 1 and 2 release the weight memory. |
| 58 | - `--offload-backend prefetch` is not supported: parameters point at staging |
| 59 | buffers, not the offloaded weights. |
| 60 | - Hashing copies every weight to CPU, so keep it off latency-sensitive paths. |