| 1 | --- |
| 2 | name: check-api-compat |
| 3 | description: Determine the correct behavior of vLLM's OpenAI-compatible or Anthropic-compatible API by checking the official upstream spec, instead of guessing from vLLM's own code. Use for any bug, issue, PR, review, or question touching /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /v1/messages, or their request/response models — including "is this a bug or intended", "what should this field default to", "does upstream allow this", and any change that adds, removes, or renames a field on those models. Do not use for vLLM-internal APIs (engine, scheduler, LLM class) or for endpoints with no upstream counterpart. |
| 4 | --- |
| 5 | |
| 6 | # Checking vLLM API compatibility against the upstream spec |
| 7 | |
| 8 | vLLM's OpenAI and Anthropic server surfaces are *reimplementations* of someone |
| 9 | else's spec. vLLM's own code, tests, and docstrings are not evidence of correct |
| 10 | behavior — they are the thing under test. Establish what upstream says first, |
| 11 | then compare. |
| 12 | |
| 13 | Never answer "what should this field do" from memory. Field names, types, and |
| 14 | especially defaults drift between spec versions. |
| 15 | |
| 16 | ## 1. Pin down the question |
| 17 | |
| 18 | Name three things before looking anything up: |
| 19 | |
| 20 | - **Which surface** — OpenAI-compatible or Anthropic-compatible. They are separate |
| 21 | code paths in vLLM and separate specs. |
| 22 | - **Which endpoint**, by path. |
| 23 | - **Which field or behavior** — a request field, a response field, a streaming |
| 24 | event shape, an error code, or a status code. |
| 25 | |
| 26 | ## 2. Read vLLM's implementation |
| 27 | |
| 28 | In a vLLM checkout, the request/response models live beside each endpoint: |
| 29 | |
| 30 | | surface | path | |
| 31 | | --- | --- | |
| 32 | | OpenAI chat | `vllm/entrypoints/openai/chat_completion/` | |
| 33 | | OpenAI completions | `vllm/entrypoints/openai/completion/` | |
| 34 | | OpenAI responses | `vllm/entrypoints/openai/responses/` | |
| 35 | | OpenAI embeddings | `vllm/entrypoints/pooling/embed/` | |
| 36 | | OpenAI shared base models, errors, usage | `vllm/entrypoints/serve/engine/protocol.py` | |
| 37 | | Anthropic messages | `vllm/entrypoints/anthropic/` | |
| 38 | | audio | `vllm/entrypoints/speech_to_text/*/protocol.py` | |
| 39 | |
| 40 | Each directory holds `protocol.py` (the Pydantic models), `serving.py` (behavior), |
| 41 | and `api_router.py` (routing and status codes). Read the field declaration *and* |
| 42 | the code that consumes it — a field can be declared faithfully and then ignored, |
| 43 | which is still an incompatibility. |
| 44 | |
| 45 | ## 3. Fetch the upstream spec |
| 46 | |
| 47 | Prefer a source that can actually be read as text, in this order. |
| 48 | |
| 49 | ### OpenAI |
| 50 | |
| 51 | 1. The OpenAPI spec — authoritative, machine-readable, and fetches cleanly: |
| 52 | `https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml` |
| 53 | 2. The installed `openai` Python SDK's typed params, which track the spec closely. |
| 54 | Locate them with `python -c "import openai,os;print(os.path.dirname(openai.__file__))"`, |
| 55 | then read e.g. `types/chat/completion_create_params.py`, |
| 56 | `types/chat/chat_completion.py`, `types/chat/chat_completion_chunk.py`, |
| 57 | `types/responses/`. Note the SDK version — an old SDK is an old spec. |
| 58 | 3. The prose reference at `https://platform.openai.com/docs/api-reference/...` |
| 59 | for semantics the types do not carry. It is script-heavy and often fetches |
| 60 | poorly; treat a failed fetch as a fetch failure, not as absence of a field. |
| 61 | |
| 62 | ### Anthropic |
| 63 | |
| 64 | 1. `https://docs.claude.com/en/api/messages` and |
| 65 | `https://docs.claude.com/en/api/messages-streaming` — these render as clean |
| 66 | text and are the primary source for both fields and streaming event order. |
| 67 | 2. The installed `anthropic` SDK's `types/message_create_params.py`, |
| 68 | `types/message.py`, `types/raw_message_stream_event.py`. |
| 69 | |
| 70 | Say which source and which version answered the question, so the finding can be |
| 71 | re-checked later. |
| 72 | |
| 73 | ## 4. Compare on all four axes |
| 74 | |
| 75 | A field is compatible only when **name**, **type**, **default**, and **semantics** |
| 76 | all match. Most real bugs hide in the last two: a field that exists with the right |
| 77 | type but the wrong default, or one that is accepted and silently dropped. |
| 78 | |
| 79 | Also check, when relevant: |
| 80 | |
| 81 | - **Streaming** — chunk shape, the order of events, and the terminal event |
| 82 | (`data: [DONE]` for Chat Completions and Completions; `response.completed`, |
| 83 | `response.failed`, or `response.incomplete` with no `[DONE]` for Responses; |
| 84 | `message_stop` for Anthropic). |
| 85 | - **Errors** — HTTP status *and* the error body's `type`/`code` strings. |
| 86 | - **Omitted vs null** — upstream often distinguishes an absent field from an |
| 87 | explicit `null`. Pydantic defaults erase that distinction unless the model is |
| 88 | written to preserve it. |
| 89 | |
| 90 | ## 5. Classify before calling it a bug |
| 91 | |
| 92 | Three outcomes, and they are not the same: |
| 93 | |
| 94 | - **Matches upstream** — no action. |
| 95 | - **vLLM extension** — a field upstream has no name for at all (`vllm_xargs`, |
| 96 | `chat_template_kwargs`, sampling knobs like `min_p`). These are intentional and |
| 97 | allowed. Do not report them as violations. Check per surface: `top_k` is an |
| 98 | extension on OpenAI but a real upstream field on Anthropic Messages. |
| 99 | - **Incompatibility** — a field that *shares a name with upstream* but differs in |
| 100 | type, default, or behavior, or an upstream field that is missing, ignored, or |
| 101 | malformed. This is the reportable case. |
| 102 | |
| 103 | An unsupported-but-declared field is its own subcase: decide whether vLLM rejects |
| 104 | it explicitly or accepts and ignores it, and report which, since silently ignoring |
| 105 | a documented field is the more damaging failure. |
| 106 | |
| 107 | ## 6. Report |
| 108 | |
| 109 | State: the endpoint and field, what upstream specifies (with the source), what |
| 110 | vLLM does (with `file:line`), which of the three outcomes it is, and the smallest |
| 111 | fix. When behavior turns on a spec detail that is easy to disbelieve, quote the |
| 112 | upstream line rather than paraphrasing it. |