[Agents] Add API compatibility checking skill (#58003)
Signed-off-by: sfeng33 <4florafeng@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Flora Feng committed
Sep 30, 2026 at 00:59 UTC
a3af78384a1f14f4769f8fa2c64c7ced5216b2ae
3 files changed
+117
.agents/skills/check-api-compat/SKILL.md
new
+112
@@ -0,0 +1,112 @@
1
+---
2
+name: check-api-compat
3
+description: Determine the correct behavior of vLLM's OpenAI-compatible or Anthropic-compatible API by checking the official upstream spec, instead of guessing from vLLM's own code. Use for any bug, issue, PR, review, or question touching /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /v1/messages, or their request/response models — including "is this a bug or intended", "what should this field default to", "does upstream allow this", and any change that adds, removes, or renames a field on those models. Do not use for vLLM-internal APIs (engine, scheduler, LLM class) or for endpoints with no upstream counterpart.
4
+---
5
+
6
+# Checking vLLM API compatibility against the upstream spec
7
+
8
+vLLM's OpenAI and Anthropic server surfaces are *reimplementations* of someone
9
+else's spec. vLLM's own code, tests, and docstrings are not evidence of correct
10
+behavior — they are the thing under test. Establish what upstream says first,
11
+then compare.
12
+
13
+Never answer "what should this field do" from memory. Field names, types, and
14
+especially defaults drift between spec versions.
15
+
16
+## 1. Pin down the question
17
+
18
+Name three things before looking anything up:
19
+
20
+- **Which surface** — OpenAI-compatible or Anthropic-compatible. They are separate
21
+ code paths in vLLM and separate specs.
22
+- **Which endpoint**, by path.
23
+- **Which field or behavior** — a request field, a response field, a streaming
24
+ event shape, an error code, or a status code.
25
+
26
+## 2. Read vLLM's implementation
27
+
28
+In a vLLM checkout, the request/response models live beside each endpoint:
29
+
30
+| surface | path |
31
+| --- | --- |
32
+| OpenAI chat | `vllm/entrypoints/openai/chat_completion/` |
33
+| OpenAI completions | `vllm/entrypoints/openai/completion/` |
34
+| OpenAI responses | `vllm/entrypoints/openai/responses/` |
35
+| OpenAI embeddings | `vllm/entrypoints/pooling/embed/` |
36
+| OpenAI shared base models, errors, usage | `vllm/entrypoints/serve/engine/protocol.py` |
37
+| Anthropic messages | `vllm/entrypoints/anthropic/` |
38
+| audio | `vllm/entrypoints/speech_to_text/*/protocol.py` |
39
+
40
+Each directory holds `protocol.py` (the Pydantic models), `serving.py` (behavior),
41
+and `api_router.py` (routing and status codes). Read the field declaration *and*
42
+the code that consumes it — a field can be declared faithfully and then ignored,
43
+which is still an incompatibility.
44
+
45
+## 3. Fetch the upstream spec
46
+
47
+Prefer a source that can actually be read as text, in this order.
48
+
49
+### OpenAI
50
+
51
+1. The OpenAPI spec — authoritative, machine-readable, and fetches cleanly:
52
+ `https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml`
53
+2. The installed `openai` Python SDK's typed params, which track the spec closely.
54
+ Locate them with `python -c "import openai,os;print(os.path.dirname(openai.__file__))"`,
55
+ then read e.g. `types/chat/completion_create_params.py`,
56
+ `types/chat/chat_completion.py`, `types/chat/chat_completion_chunk.py`,
57
+ `types/responses/`. Note the SDK version — an old SDK is an old spec.
58
+3. The prose reference at `https://platform.openai.com/docs/api-reference/...`
59
+ for semantics the types do not carry. It is script-heavy and often fetches
60
+ poorly; treat a failed fetch as a fetch failure, not as absence of a field.
61
+
62
+### Anthropic
63
+
64
+1. `https://docs.claude.com/en/api/messages` and
65
+ `https://docs.claude.com/en/api/messages-streaming` — these render as clean
66
+ text and are the primary source for both fields and streaming event order.
67
+2. The installed `anthropic` SDK's `types/message_create_params.py`,
68
+ `types/message.py`, `types/raw_message_stream_event.py`.
69
+
70
+Say which source and which version answered the question, so the finding can be
71
+re-checked later.
72
+
73
+## 4. Compare on all four axes
74
+
75
+A field is compatible only when **name**, **type**, **default**, and **semantics**
76
+all match. Most real bugs hide in the last two: a field that exists with the right
77
+type but the wrong default, or one that is accepted and silently dropped.
78
+
79
+Also check, when relevant:
80
+
81
+- **Streaming** — chunk shape, the order of events, and the terminal event
82
+ (`data: [DONE]` for Chat Completions and Completions; `response.completed`,
83
+ `response.failed`, or `response.incomplete` with no `[DONE]` for Responses;
84
+ `message_stop` for Anthropic).
85
+- **Errors** — HTTP status *and* the error body's `type`/`code` strings.
86
+- **Omitted vs null** — upstream often distinguishes an absent field from an
87
+ explicit `null`. Pydantic defaults erase that distinction unless the model is
88
+ written to preserve it.
89
+
90
+## 5. Classify before calling it a bug
91
+
92
+Three outcomes, and they are not the same:
93
+
94
+- **Matches upstream** — no action.
95
+- **vLLM extension** — a field upstream has no name for at all (`vllm_xargs`,
96
+ `chat_template_kwargs`, sampling knobs like `min_p`). These are intentional and
97
+ allowed. Do not report them as violations. Check per surface: `top_k` is an
98
+ extension on OpenAI but a real upstream field on Anthropic Messages.
99
+- **Incompatibility** — a field that *shares a name with upstream* but differs in
100
+ type, default, or behavior, or an upstream field that is missing, ignored, or
101
+ malformed. This is the reportable case.
102
+
103
+An unsupported-but-declared field is its own subcase: decide whether vLLM rejects
104
+it explicitly or accepts and ignores it, and report which, since silently ignoring
105
+a documented field is the more damaging failure.
106
+
107
+## 6. Report
108
+
109
+State: the endpoint and field, what upstream specifies (with the source), what
110
+vLLM does (with `file:line`), which of the three outcomes it is, and the smallest
111
+fix. When behavior turns on a spec detail that is easy to disbelieve, quote the
112
+upstream line rather than paraphrasing it.
.agents/skills/check-api-compat/agents/openai.yaml
new
+4
@@ -0,0 +1,4 @@
1
+interface:
2
+ display_name: "Check API Compat"
3
+ short_description: "Check vLLM's OpenAI/Anthropic-compatible API against the upstream spec."
4
+ default_prompt: "Use the vLLM API compatibility skill to check this endpoint or field against the upstream OpenAI or Anthropic spec."
.claude/skills/check-api-compat
new
+1
@@ -0,0 +1 @@
1
+../../.agents/skills/check-api-compat
\ No newline at end of file