@samitouri / QOSami_Agents_Zero / commits / 9bcef390

Add llama.cpp and vLLM local providers

Add llama.cpp / llama-server support: - Register llama.cpp chat and embedding providers through hosted_vllm defaults on host.docker.internal:8080. - Add onboarding metadata, a bundled icon, no-key provider metadata, model discovery coverage, setup docs, and tests. Add vLLM support: - Register vLLM chat and embedding providers through hosted_vllm defaults on host.docker.internal:8000. - Add onboarding metadata, a bundled icon, no-key provider metadata, model discovery coverage, setup docs, and tests. - Strip empty tools arrays from the Responses-to-chat fallback path so strict OpenAI-compatible servers accept local vLLM calls.

Alessandro committed Jun 15, 2026 at 05:37 UTC 9bcef39028a8d5579cfa9752c1584eb8d39cd4ca
18 files changed +450 -6
conf/model_providers.yaml
+30
@@ -86,6 +86,15 @@ chat:
86 kwargs:
87 api_base: "http://host.docker.internal:1234/v1"
88 api_key: "lm-studio"
89 + llama_cpp:
90 + name: llama.cpp
91 + litellm_provider: hosted_vllm
92 + models_list:
93 + endpoint_url: "/v1/models"
94 + default_base: "http://host.docker.internal:8080"
95 + kwargs:
96 + api_base: "http://host.docker.internal:8080/v1"
97 + api_key: "llama-cpp"
98 mistral:
99 name: Mistral AI
100 litellm_provider: mistral
@@ -167,6 +176,15 @@ chat:
176 api_base: https://api.venice.ai/api/v1
177 venice_parameters:
178 include_venice_system_prompt: false
179 + vllm:
180 + name: vLLM
181 + litellm_provider: hosted_vllm
182 + models_list:
183 + endpoint_url: "/v1/models"
184 + default_base: "http://host.docker.internal:8000"
185 + kwargs:
186 + api_base: "http://host.docker.internal:8000/v1"
187 + api_key: "vllm"
188 xai:
189 name: xAI
190 litellm_provider: xai
@@ -203,6 +221,12 @@ embedding:
221 kwargs:
222 api_base: "http://host.docker.internal:1234/v1"
223 api_key: "lm-studio"
224 + llama_cpp:
225 + name: llama.cpp
226 + litellm_provider: hosted_vllm
227 + kwargs:
228 + api_base: "http://host.docker.internal:8080/v1"
229 + api_key: "llama-cpp"
230 mistral:
231 name: Mistral AI
232 litellm_provider: mistral
@@ -250,6 +274,12 @@ embedding:
274 endpoint_url: "https://api.venice.ai/api/v1/models"
275 kwargs:
276 api_base: https://api.venice.ai/api/v1
277 + vllm:
278 + name: vLLM
279 + litellm_provider: hosted_vllm
280 + kwargs:
281 + api_base: "http://host.docker.internal:8000/v1"
282 + api_key: "vllm"
283 other:
284 name: Other OpenAI compatible
285 litellm_provider: openai
docs/setup/installation.md
+67
@@ -439,6 +439,8 @@ Use the naming format required by your selected provider:
439 | OpenRouter | Provider prefix mostly required | `anthropic/claude-sonnet-4-5` |
440 | Ollama | Model name only | `gpt-oss:20b` |
441 | oMLX | API-visible model name from `/v1/models` | `Qwen3-0.6B-4bit` |
442 +| llama.cpp | API-visible model name from `/v1/models` or `--alias` | `local-gguf` |
443 +| vLLM | Hugging Face model ID or served model alias | `Qwen/Qwen2.5-1.5B-Instruct` |
444
445 > [!TIP]
446 > If you see "Invalid model ID," verify the provider and naming format on the provider website, or search the web for "<name-of-ai-model> model naming".
@@ -504,6 +506,71 @@ omlx serve --model-dir ~/.omlx/models --paged-ssd-cache-dir ~/.omlx/cache
506
507 ---
508
509 +## Installing and Using llama.cpp (GGUF Local Models)
510 +
511 +llama.cpp provides `llama-server`, a lightweight OpenAI-compatible HTTP server for GGUF models. Agent Zero talks to it through the same `/v1` API used by OpenAI-compatible clients.
512 +
513 +### macOS llama.cpp Installation
514 +
515 +**Using Homebrew:**
516 +
517 +```bash
518 +brew install llama.cpp
519 +```
520 +
521 +Start a server with a downloaded GGUF model:
522 +
523 +```bash
524 +llama-server -m ~/models/model.gguf --port 8080 --alias local-gguf
525 +```
526 +
527 +By default, Agent Zero expects llama.cpp at `http://host.docker.internal:8080/v1`. The model name can be the model path returned by `/v1/models`, but using `--alias` gives you a short stable name such as `local-gguf`.
528 +
529 +### Configuring llama.cpp in Agent Zero
530 +
531 +1. Start `llama-server` and confirm `http://localhost:8080/v1/models` returns your model.
532 +2. In Agent Zero Settings, choose **llama.cpp** as the Chat model, Utility model, or Embedding model provider.
533 +3. Use the model ID shown by `/v1/models`, or the alias you passed with `--alias`.
534 +4. Override the API base URL only if you started `llama-server` on another host or port.
535 +5. Click `Save` to confirm your settings.
536 +
537 +> [!NOTE]
538 +> If Agent Zero runs in Docker and cannot reach a host-side `llama-server`, start the server on an address Docker can reach, for example `--host 0.0.0.0`, and keep the port firewalled to trusted clients.
539 +
540 +---
541 +
542 +## Installing and Using vLLM (Local OpenAI-Compatible Serving)
543 +
544 +vLLM is a high-throughput local inference server with an OpenAI-compatible API. It is most common on Linux GPU hosts, and can also run on Apple Silicon through the vLLM Apple Silicon path or vLLM-Metal.
545 +
546 +For Apple Silicon Macs, install and activate vLLM-Metal:
547 +
548 +```bash
549 +curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
550 +source ~/.venv-vllm-metal/bin/activate
551 +```
552 +
553 +Start a basic OpenAI-compatible server:
554 +
555 +```bash
556 +vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000
557 +```
558 +
559 +By default, Agent Zero expects vLLM at `http://host.docker.internal:8000/v1`, matching vLLM's default HTTP port. If another local provider already uses port 8000, start vLLM on another port and update Agent Zero's API base, for example `http://host.docker.internal:8001/v1`.
560 +
561 +### Configuring vLLM in Agent Zero
562 +
563 +1. Start vLLM and confirm `http://localhost:8000/v1/models` returns the served model.
564 +2. In Agent Zero Settings, choose **vLLM** as the Chat model, Utility model, or Embedding model provider.
565 +3. Use the model ID returned by vLLM's model list endpoint.
566 +4. If you started vLLM with `--api-key`, enter the same key in the advanced provider settings or environment.
567 +5. Click `Save` to confirm your settings.
568 +
569 +> [!NOTE]
570 +> vLLM serves one model at a time by default. Use a generation model for Chat and Utility slots, and a separate embedding-capable vLLM server if you want vLLM embeddings.
571 +
572 +---
573 +
574 ## Installing and Using Ollama (Local Models)
575
576 Ollama is a powerful tool that allows you to run various large language models locally.
helpers/litellm_transport.py
+3
@@ -459,6 +459,7 @@ class ChatCompletionsTransport:
459 chat_kwargs = dict(kwargs)
460 _drop_internal_transport_kwargs(chat_kwargs)
461 if not _has_tools(chat_kwargs.get("tools")):
462 + chat_kwargs.pop("tools", None)
463 chat_kwargs.pop("tool_choice", None)
464 chat_kwargs.pop("parallel_tool_calls", None)
465 if _is_openai_prompt_cache_provider(model, chat_kwargs):
@@ -1538,6 +1539,8 @@ def _is_responses_not_supported_error(exc: Exception) -> bool:
1539 "unsupportedparamserror",
1540 "does not support parameters",
1541 "no 'tools' defined while 'tool_choice' is specified",
1542 + "tools` must not be an empty array",
1543 + "tools must not be an empty array",
1544 )
1545 )
1546
helpers/litellm_transport.py.dox.md new
+45
@@ -0,0 +1,45 @@
1 +# litellm_transport.py DOX
2 +
3 +## Purpose
4 +
5 +- Own Agent Zero's LiteLLM transport adapter for Chat Completions and Responses API calls.
6 +- Normalize Agent Zero model-call kwargs into provider-safe LiteLLM requests.
7 +- Preserve canonical response metadata for history, provider-state continuation, and fallback decisions.
8 +
9 +## Ownership
10 +
11 +- `litellm_transport.py` owns the runtime implementation.
12 +- `litellm_transport.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
13 +- Classes:
14 +- `TransportMode`
15 +- `TransportRecovery`
16 +- `TransportPolicy`
17 +- `LiteLLMTransport`
18 +- `ChatCompletionsTransport`
19 +- `ResponsesTransport`
20 +- `ResponsesEventParser`
21 +- Top-level functions include transport cache reset, request normalization, parsing, prompt-cache preparation, and response/error classifiers.
22 +
23 +## Runtime Contracts
24 +
25 +- Keep provider selection and provider-specific defaults outside this helper; callers pass a resolved LiteLLM model name and kwargs.
26 +- Strip Agent Zero internal kwargs before sending requests to LiteLLM.
27 +- Do not send orphan tool controls when no tools are present; strict OpenAI-compatible servers can reject empty `tools` arrays.
28 +- Prefer Responses API when configured, but fallback to Chat Completions when the provider does not support Responses.
29 +- Preserve provider-state metadata when Responses API calls succeed, and fall back to local replay when provider state is unsupported.
30 +- Keep prompt-cache markers only for providers that accept them.
31 +
32 +## Work Guidance
33 +
34 +- Add provider-agnostic request cleanup here when multiple OpenAI-compatible providers can benefit.
35 +- Treat fallback behavior as a shared transport contract, not a provider registry.
36 +- Keep tool conversion symmetric between Chat Completions and Responses requests.
37 +
38 +## Verification
39 +
40 +- Run `pytest tests/test_stream_tool_early_stop.py tests/test_responses_architecture.py -q` after changing transport normalization or fallback behavior.
41 +- Run local-provider smoke checks when changing OpenAI-compatible request cleanup.
42 +
43 +## Child DOX Index
44 +
45 +No child DOX files.
helpers/llm_result.py.dox.md new
+39
@@ -0,0 +1,39 @@
1 +# llm_result.py DOX
2 +
3 +## Purpose
4 +
5 +- Own canonical LLM result metadata shared by model transports, history, and tool-result processing.
6 +- Preserve Responses API output items, provider response IDs, reasoning text, usage, and capability metadata in a serializable form.
7 +
8 +## Ownership
9 +
10 +- `llm_result.py` owns the runtime implementation.
11 +- `llm_result.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
12 +- Classes:
13 +- `ResponseItem`
14 +- `ResponseFunctionCall`
15 +- `LLMResult`
16 +- Top-level functions include metadata conversion, function-call output item construction, object normalization, output-text extraction, reasoning extraction, and function-call argument parsing.
17 +
18 +## Runtime Contracts
19 +
20 +- `LLMResult.metadata()` stores data under `RESPONSE_METADATA_KEY` so history can round-trip provider state.
21 +- `from_response(...)` must preserve provider `response_id`, `previous_response_id`, raw output items, usage, and capability metadata.
22 +- `from_chat(...)` must produce an equivalent chat-completions result with `mode="chat_completions"` and `state="off"`.
23 +- Function-call output items must preserve `call_id` and optional acknowledged safety checks.
24 +- Argument parsing must tolerate JSON strings, dictionaries, and malformed values without throwing.
25 +
26 +## Work Guidance
27 +
28 +- Keep metadata backward-compatible with existing serialized chat history.
29 +- Treat unknown response item types as preserved built-in items unless they are local function calls, message text, or reasoning.
30 +- Avoid provider-specific assumptions in result parsing.
31 +
32 +## Verification
33 +
34 +- Run `pytest tests/test_responses_architecture.py -q` after changing result metadata behavior.
35 +- Run focused history/tool-processing tests when changing function-call serialization.
36 +
37 +## Child DOX Index
38 +
39 +No child DOX files.
helpers/tunnel_origins.py.dox.md new
+37
@@ -0,0 +1,37 @@
1 +# tunnel_origins.py DOX
2 +
3 +## Purpose
4 +
5 +- Own origin normalization for Remote Control tunnel URLs and CSRF/WebSocket same-origin checks.
6 +- Provide a small helper boundary between tunnel discovery and security enforcement.
7 +
8 +## Ownership
9 +
10 +- `tunnel_origins.py` owns the runtime implementation.
11 +- `tunnel_origins.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
12 +- Top-level functions:
13 +- `origin_from_url(value)`
14 +- `origin_key(value)`
15 +- `get_active_tunnel_origins()`
16 +
17 +## Runtime Contracts
18 +
19 +- Normalize URL and Origin header values to `scheme://host[:port]`, omitting default ports.
20 +- Return comparable origin keys with default ports restored for same-origin checks.
21 +- Treat invalid, missing, or malformed origins as `None`.
22 +- Discover active tunnel origins from `TunnelManager` and the Docker tunnel API without raising if either source is unavailable.
23 +- Keep tunnel service lookups short-timeout and local-only.
24 +
25 +## Work Guidance
26 +
27 +- Keep parsing based on `urllib.parse` rather than hand-rolled string checks.
28 +- Preserve defensive exception handling because tunnel services are optional and may not be running.
29 +- Coordinate security-sensitive changes with CSRF and WebSocket tests.
30 +
31 +## Verification
32 +
33 +- Run `pytest tests/test_csrf_tunnel_origins.py tests/test_ws_csrf.py -q` after changing tunnel origin behavior.
34 +
35 +## Child DOX Index
36 +
37 +No child DOX files.
plugins/_browser/helpers/connector_runtime.py
+1 -1
@@ -58,7 +58,7 @@ HOST_BROWSER_PROFILE_MODE_KEY = getattr(
58 "host_browser_profile_mode",
59 )
60 get_browser_config = browser_config.get_browser_config
61 -_LOCAL_PROVIDERS = {"ollama", "lm_studio", "omlx"}
61 +_LOCAL_PROVIDERS = {"ollama", "lm_studio", "llama_cpp", "omlx", "vllm"}
62 _LOCAL_HOSTS = {"localhost", "127.0.0.1", "::1", "host.docker.internal"}
63 _SENSITIVE_ACTIONS = {"content", "detail", "evaluate", "screenshot", "screenshot_file"}
64 _KEY_ALIASES = {
plugins/_model_config/api/model_search.py
+8 -2
@@ -26,6 +26,12 @@ _NON_CHAT_EXCLUDE = frozenset({
26 "omni-moderation",
27 "vision-preview",
28 })
29 +_LOCAL_PLACEHOLDER_KEYS = {
30 + "lm_studio": {"lm-studio"},
31 + "llama_cpp": {"llama-cpp"},
32 + "omlx": {"omlx"},
33 + "vllm": {"vllm"},
34 +}
35
36
37 class ModelSearch(ApiHandler):
@@ -179,8 +185,8 @@ class ModelSearch(ApiHandler):
185 elif provider == "azure":
186 if has_key:
187 headers["api-key"] = api_key
182 - elif provider not in ("ollama", "lm_studio", "omlx"):
183 - if has_key:
188 + elif provider != "ollama":
189 + if has_key and api_key not in _LOCAL_PLACEHOLDER_KEYS.get(provider, set()):
190 headers["Authorization"] = f"Bearer {api_key}"
191
192 extra = (cfg or {}).get("kwargs", {}).get("extra_headers", {})
plugins/_model_config/extensions/python/banners/_20_missing_api_key.py
+1 -1
@@ -6,7 +6,7 @@ from plugins._model_config.helpers import model_config
6 class MissingApiKeyCheck(Extension):
7 """Check if API keys are configured for selected model providers."""
8
9 - LOCAL_PROVIDERS = {"ollama", "lm_studio", "omlx"}
9 + LOCAL_PROVIDERS = {"ollama", "lm_studio", "llama_cpp", "omlx", "vllm"}
10 CONFIGURE_MODEL_SETTINGS_LINK = (
11 """<div class="onboarding-banner-btn-container" style="margin-top: 12px;">"""
12 """<button class="btn btn-ok" onclick="window.openModal('/plugins/_onboarding/webui/onboarding.html');return false;">"""
plugins/_model_config/helpers/model_config.py
+1 -1
@@ -32,7 +32,7 @@ IMPLICIT_PRESET_SLOT_DEFAULTS = {
32 "kwargs": {},
33 },
34 }
35 -LOCAL_PROVIDERS = {"ollama", "lm_studio", "omlx"}
35 +LOCAL_PROVIDERS = {"ollama", "lm_studio", "llama_cpp", "omlx", "vllm"}
36 LOCAL_EMBEDDING = {"huggingface"}
37 _PROVIDER_METADATA_CACHE: dict | None = None
38
plugins/_model_config/provider_metadata.yaml
+8
@@ -1,10 +1,14 @@
1 chat:
2 lm_studio:
3 api_key_mode: none
4 + llama_cpp:
5 + api_key_mode: none
6 ollama:
7 api_key_mode: none
8 omlx:
9 api_key_mode: none
10 + vllm:
11 + api_key_mode: none
12 other:
13 api_key_mode: optional
14
@@ -13,9 +17,13 @@ embedding:
17 api_key_mode: none
18 lm_studio:
19 api_key_mode: none
20 + llama_cpp:
21 + api_key_mode: none
22 ollama:
23 api_key_mode: none
24 omlx:
25 api_key_mode: none
26 + vllm:
27 + api_key_mode: none
28 other:
29 api_key_mode: optional
plugins/_onboarding/webui/assets/provider-logos/llama-cpp.svg new
+17
@@ -0,0 +1,17 @@
1 +<svg xmlns="http://www.w3.org/2000/svg" width="160" height="160" viewBox="0 0 160 160" role="img" aria-labelledby="llama-cpp-title">
2 + <title id="llama-cpp-title">llama.cpp</title>
3 + <defs>
4 + <linearGradient id="llama-cpp-bg" x1="0" y1="0" x2="1" y2="1">
5 + <stop offset="0" stop-color="#242424"/>
6 + <stop offset="1" stop-color="#111111"/>
7 + </linearGradient>
8 + </defs>
9 + <rect x="10" y="10" width="140" height="140" rx="28" fill="url(#llama-cpp-bg)"/>
10 + <path d="M47 111V65c0-18 14-32 32-32h9c18 0 32 14 32 32v46" fill="none" stroke="#f1e4c8" stroke-width="11" stroke-linecap="round" stroke-linejoin="round"/>
11 + <path d="M58 42 47 24M103 42l12-18" fill="none" stroke="#f1e4c8" stroke-width="10" stroke-linecap="round"/>
12 + <path d="M58 72h50" stroke="#79c7b4" stroke-width="9" stroke-linecap="round"/>
13 + <circle cx="67" cy="88" r="5" fill="#f1e4c8"/>
14 + <circle cx="96" cy="88" r="5" fill="#f1e4c8"/>
15 + <path d="M72 106h22" stroke="#f1e4c8" stroke-width="8" stroke-linecap="round"/>
16 + <text x="80" y="133" text-anchor="middle" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace" font-size="23" font-weight="800" fill="#79c7b4">.cpp</text>
17 +</svg>
plugins/_onboarding/webui/assets/provider-logos/vllm.svg new
+18
@@ -0,0 +1,18 @@
1 +<svg xmlns="http://www.w3.org/2000/svg" width="160" height="160" viewBox="0 0 160 160" role="img" aria-labelledby="vllm-title">
2 + <title id="vllm-title">vLLM</title>
3 + <defs>
4 + <linearGradient id="vllm-bg" x1="0" y1="0" x2="1" y2="1">
5 + <stop offset="0" stop-color="#12303a"/>
6 + <stop offset="1" stop-color="#101820"/>
7 + </linearGradient>
8 + <linearGradient id="vllm-mark" x1="0" y1="0" x2="1" y2="1">
9 + <stop offset="0" stop-color="#8ce9dc"/>
10 + <stop offset="1" stop-color="#4da3ff"/>
11 + </linearGradient>
12 + </defs>
13 + <rect x="10" y="10" width="140" height="140" rx="28" fill="url(#vllm-bg)"/>
14 + <path d="M37 43 65 116 93 43" fill="none" stroke="url(#vllm-mark)" stroke-width="15" stroke-linecap="round" stroke-linejoin="round"/>
15 + <path d="M99 68v48M120 68v48" fill="none" stroke="#efffff" stroke-width="12" stroke-linecap="round"/>
16 + <path d="M99 92h21" stroke="#efffff" stroke-width="12" stroke-linecap="round"/>
17 + <text x="80" y="135" text-anchor="middle" font-family="Inter, ui-sans-serif, system-ui, sans-serif" font-size="21" font-weight="900" fill="#8ce9dc">vLLM</text>
18 +</svg>
plugins/_onboarding/webui/onboarding-providers.js
+19 -1
@@ -25,7 +25,7 @@ export const MORE_CLOUD_PROVIDER_IDS = [
25 "other",
26 ];
27
28 -export const LOCAL_PROVIDER_IDS = ["ollama", "lm_studio", "omlx", "other"];
28 +export const LOCAL_PROVIDER_IDS = ["ollama", "lm_studio", "omlx", "llama_cpp", "vllm", "other"];
29
30 export const ONBOARDING_PROVIDER_OVERRIDES = {
31 a0_venice: {
@@ -126,6 +126,15 @@ export const ONBOARDING_PROVIDER_OVERRIDES = {
126 model_list_autoload: true,
127 short_description: "Run local models through LM Studio.",
128 },
129 + llama_cpp: {
130 + logo: "/plugins/_onboarding/webui/assets/provider-logos/llama-cpp.svg",
131 + setup_url: "https://github.com/ggml-org/llama.cpp",
132 + docs_url: "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md",
133 + default_api_base: "http://host.docker.internal:8080/v1",
134 + api_key_mode: "none",
135 + model_list_autoload: true,
136 + short_description: "Run GGUF models with llama-server.",
137 + },
138 mistral: {
139 logo: "https://mistral.ai/favicon.ico",
140 setup_url: "https://console.mistral.ai/",
@@ -181,6 +190,15 @@ export const ONBOARDING_PROVIDER_OVERRIDES = {
190 model_list_autoload: true,
191 short_description: "Apple Silicon local inference with MLX.",
192 },
193 + vllm: {
194 + logo: "/plugins/_onboarding/webui/assets/provider-logos/vllm.svg",
195 + setup_url: "https://docs.vllm.ai/",
196 + docs_url: "https://docs.vllm.ai/en/stable/serving/online_serving/",
197 + default_api_base: "http://host.docker.internal:8000/v1",
198 + api_key_mode: "none",
199 + model_list_autoload: true,
200 + short_description: "High-throughput local OpenAI-compatible serving.",
201 + },
202 openai: {
203 logo: "https://openai.com/favicon.ico",
204 setup_url: "https://platform.openai.com/",
tests/test_model_config_api_keys.py
+61
@@ -274,10 +274,14 @@ def test_provider_key_modes_for_local_and_ollama_cloud():
274
275 assert model_config.provider_requires_api_key("ollama") is False
276 assert model_config.provider_requires_api_key("lm_studio") is False
277 + assert model_config.provider_requires_api_key("llama_cpp") is False
278 assert model_config.provider_requires_api_key("omlx") is False
279 + assert model_config.provider_requires_api_key("vllm") is False
280 assert model_config.provider_requires_api_key("other") is False
281 assert model_config.provider_requires_api_key("ollama_cloud") is True
282 + assert "llama_cpp" in missing_key_banner.MissingApiKeyCheck.LOCAL_PROVIDERS
283 assert "omlx" in missing_key_banner.MissingApiKeyCheck.LOCAL_PROVIDERS
284 + assert "vllm" in missing_key_banner.MissingApiKeyCheck.LOCAL_PROVIDERS
285
286
287 def test_local_provider_defaults_are_docker_friendly():
@@ -293,6 +297,15 @@ def test_local_provider_defaults_are_docker_friendly():
297 assert provider_config["chat"]["lm_studio"]["models_list"]["default_base"] == (
298 "http://host.docker.internal:1234"
299 )
300 + assert provider_config["chat"]["llama_cpp"]["litellm_provider"] == "hosted_vllm"
301 + assert provider_config["chat"]["llama_cpp"]["kwargs"]["api_base"] == (
302 + "http://host.docker.internal:8080/v1"
303 + )
304 + assert provider_config["chat"]["llama_cpp"]["kwargs"]["api_key"] == "llama-cpp"
305 + assert provider_config["chat"]["llama_cpp"]["models_list"]["default_base"] == (
306 + "http://host.docker.internal:8080"
307 + )
308 + assert provider_config["chat"]["llama_cpp"]["models_list"]["endpoint_url"] == "/v1/models"
309 assert provider_config["chat"]["ollama"]["kwargs"]["api_base"] == (
310 "http://host.docker.internal:11434"
311 )
@@ -308,10 +321,24 @@ def test_local_provider_defaults_are_docker_friendly():
321 "http://host.docker.internal:8000"
322 )
323 assert provider_config["chat"]["omlx"]["models_list"]["endpoint_url"] == "/v1/models"
324 + assert provider_config["chat"]["vllm"]["litellm_provider"] == "hosted_vllm"
325 + assert provider_config["chat"]["vllm"]["kwargs"]["api_base"] == (
326 + "http://host.docker.internal:8000/v1"
327 + )
328 + assert provider_config["chat"]["vllm"]["kwargs"]["api_key"] == "vllm"
329 + assert provider_config["chat"]["vllm"]["models_list"]["default_base"] == (
330 + "http://host.docker.internal:8000"
331 + )
332 + assert provider_config["chat"]["vllm"]["models_list"]["endpoint_url"] == "/v1/models"
333 assert provider_config["embedding"]["lm_studio"]["kwargs"]["api_base"] == (
334 "http://host.docker.internal:1234/v1"
335 )
336 assert provider_config["embedding"]["lm_studio"]["kwargs"]["api_key"] == "lm-studio"
337 + assert provider_config["embedding"]["llama_cpp"]["litellm_provider"] == "hosted_vllm"
338 + assert provider_config["embedding"]["llama_cpp"]["kwargs"]["api_base"] == (
339 + "http://host.docker.internal:8080/v1"
340 + )
341 + assert provider_config["embedding"]["llama_cpp"]["kwargs"]["api_key"] == "llama-cpp"
342 assert provider_config["embedding"]["ollama"]["kwargs"]["api_base"] == (
343 "http://host.docker.internal:11434"
344 )
@@ -320,6 +347,11 @@ def test_local_provider_defaults_are_docker_friendly():
347 "http://host.docker.internal:8000/v1"
348 )
349 assert provider_config["embedding"]["omlx"]["kwargs"]["api_key"] == "omlx"
350 + assert provider_config["embedding"]["vllm"]["litellm_provider"] == "hosted_vllm"
351 + assert provider_config["embedding"]["vllm"]["kwargs"]["api_base"] == (
352 + "http://host.docker.internal:8000/v1"
353 + )
354 + assert provider_config["embedding"]["vllm"]["kwargs"]["api_key"] == "vllm"
355
356
357 def test_local_provider_runtime_defaults_and_overrides(monkeypatch):
@@ -344,6 +376,16 @@ def test_local_provider_runtime_defaults_and_overrides(monkeypatch):
376 assert custom_lm_embedding.kwargs["api_base"] == "http://127.0.0.1:1234/v1"
377 assert custom_lm_embedding.kwargs["api_key"] == "real-local-key"
378
379 + llama_cpp_chat = models.get_chat_model("llama_cpp", "local-chat-model")
380 + assert llama_cpp_chat.model_name == "hosted_vllm/local-chat-model"
381 + assert llama_cpp_chat.kwargs["api_base"] == "http://host.docker.internal:8080/v1"
382 + assert llama_cpp_chat.kwargs["api_key"] == "llama-cpp"
383 +
384 + llama_cpp_embedding = models.get_embedding_model("llama_cpp", "local-embedding-model")
385 + assert llama_cpp_embedding.model_name == "hosted_vllm/local-embedding-model"
386 + assert llama_cpp_embedding.kwargs["api_base"] == "http://host.docker.internal:8080/v1"
387 + assert llama_cpp_embedding.kwargs["api_key"] == "llama-cpp"
388 +
389 ollama_embedding = models.get_embedding_model("ollama", "nomic-embed-text")
390 assert ollama_embedding.model_name == "ollama/nomic-embed-text"
391 assert ollama_embedding.kwargs["api_base"] == "http://host.docker.internal:11434"
@@ -368,6 +410,25 @@ def test_local_provider_runtime_defaults_and_overrides(monkeypatch):
410 assert custom_omlx_chat.kwargs["api_base"] == "http://127.0.0.1:8000/v1"
411 assert custom_omlx_chat.kwargs["api_key"] == "real-local-key"
412
413 + vllm_chat = models.get_chat_model("vllm", "local-chat-model")
414 + assert vllm_chat.model_name == "hosted_vllm/local-chat-model"
415 + assert vllm_chat.kwargs["api_base"] == "http://host.docker.internal:8000/v1"
416 + assert vllm_chat.kwargs["api_key"] == "vllm"
417 +
418 + vllm_embedding = models.get_embedding_model("vllm", "local-embedding-model")
419 + assert vllm_embedding.model_name == "hosted_vllm/local-embedding-model"
420 + assert vllm_embedding.kwargs["api_base"] == "http://host.docker.internal:8000/v1"
421 + assert vllm_embedding.kwargs["api_key"] == "vllm"
422 +
423 + custom_vllm_chat = models.get_chat_model(
424 + "vllm",
425 + "local-chat-model",
426 + api_base="http://127.0.0.1:8001/v1",
427 + api_key="real-local-key",
428 + )
429 + assert custom_vllm_chat.kwargs["api_base"] == "http://127.0.0.1:8001/v1"
430 + assert custom_vllm_chat.kwargs["api_key"] == "real-local-key"
431 +
432
433 def test_docker_compose_maps_host_docker_internal_for_local_models():
434 import yaml
tests/test_model_search.py
+10
@@ -72,6 +72,16 @@ def test_model_search_omits_auth_header_for_omlx_placeholder_key():
72 assert handler._build_headers("omlx", "omlx", {}) == {}
73
74
75 +def test_model_search_omits_auth_header_for_local_placeholder_keys():
76 + handler = _handler()
77 +
78 + assert handler._build_headers("llama_cpp", "llama-cpp", {}) == {}
79 + assert handler._build_headers("vllm", "vllm", {}) == {}
80 + assert handler._build_headers("llama_cpp", "real-local-key", {}) == {
81 + "Authorization": "Bearer real-local-key"
82 + }
83 +
84 +
85 def test_model_search_filters_non_chat_models():
86 handler = _handler()
87
tests/test_onboarding_static.py
+14
@@ -65,16 +65,24 @@ def test_onboarding_provider_grid_names_are_present_in_metadata():
65 assert 'docs_url: "https://docs.venice.ai/guides/getting-started/generating-api-key"' in provider_ui
66 assert 'docs_url: "https://docs.tokenfactory.nebius.com/api-reference/introduction"' in provider_ui
67 assert 'docs_url: "https://lmstudio.ai/docs/developer/core/authentication"' in provider_ui
68 + assert 'logo: "/plugins/_onboarding/webui/assets/provider-logos/llama-cpp.svg"' in provider_ui
69 + assert 'docs_url: "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md"' in provider_ui
70 + assert 'default_api_base: "http://host.docker.internal:8080/v1"' in provider_ui
71 assert 'logo: "/plugins/_onboarding/webui/assets/provider-logos/omlx.svg"' in provider_ui
72 assert 'docs_url: "https://github.com/jundot/omlx#readme"' in provider_ui
73 assert 'default_api_base: "http://host.docker.internal:8000/v1"' in provider_ui
74 + assert 'logo: "/plugins/_onboarding/webui/assets/provider-logos/vllm.svg"' in provider_ui
75 + assert 'docs_url: "https://docs.vllm.ai/en/stable/serving/online_serving/"' in provider_ui
76 assert 'docs_url: ""' in provider_ui
77 assert "api_key_mode: none" in model_metadata
78 assert "api_key_mode: optional" in model_metadata
79 assert "Ollama Cloud" in provider_yaml
80 assert "https://ollama.com/v1" in provider_yaml
81 + assert "llama.cpp" in provider_yaml
82 + assert "http://host.docker.internal:8080/v1" in provider_yaml
83 assert "oMLX" in provider_yaml
84 assert "http://host.docker.internal:8000/v1" in provider_yaml
85 + assert "vLLM" in provider_yaml
86 assert "Nebius Token Factory" in provider_yaml
87 assert "https://api.tokenfactory.nebius.com/v1" in provider_yaml
88 assert not (PROJECT_ROOT / "plugins/_model_config/conf/model_providers.yaml").exists()
@@ -92,7 +100,9 @@ def test_onboarding_provider_grid_names_are_present_in_metadata():
100 "Z.AI",
101 "Mistral AI",
102 "Azure OpenAI",
103 + "llama.cpp",
104 "oMLX",
105 + "vLLM",
106 ]:
107 assert name in provider_yaml + provider_ui
108
@@ -122,12 +132,16 @@ def test_onboarding_provider_grid_names_are_present_in_metadata():
132 "sambanova.png",
133 "cometapi.ico",
134 "github-copilot.svg",
135 + "llama-cpp.svg",
136 "zai-logo.svg",
137 "omlx.svg",
138 + "vllm.svg",
139 ]:
140 assert logo in provider_ui
141
142 + assert (PROJECT_ROOT / "plugins/_onboarding/webui/assets/provider-logos/llama-cpp.svg").exists()
143 assert (PROJECT_ROOT / "plugins/_onboarding/webui/assets/provider-logos/omlx.svg").exists()
144 + assert (PROJECT_ROOT / "plugins/_onboarding/webui/assets/provider-logos/vllm.svg").exists()
145
146
147 def test_nebius_provider_config_uses_openai_compatible_token_factory_endpoint():
tests/test_stream_tool_early_stop.py
+71
@@ -643,6 +643,77 @@ def test_responses_request_normalizes_reasoning_and_orphan_tool_choice():
643 assert "reasoning" not in request
644
645
646 +def test_chat_completions_kwargs_omit_empty_tools():
647 + kwargs = litellm_transport.ChatCompletionsTransport.prepare_kwargs(
648 + {
649 + "tools": [],
650 + "tool_choice": "auto",
651 + "parallel_tool_calls": True,
652 + "max_tokens": 8,
653 + }
654 + )
655 +
656 + assert kwargs == {"max_tokens": 8}
657 +
658 + kwargs = litellm_transport.ChatCompletionsTransport.prepare_kwargs(
659 + {
660 + "tools": [
661 + {
662 + "type": "function",
663 + "function": {
664 + "name": "lookup",
665 + "parameters": {"type": "object"},
666 + },
667 + }
668 + ],
669 + "tool_choice": "auto",
670 + }
671 + )
672 +
673 + assert kwargs["tools"][0]["function"]["name"] == "lookup"
674 + assert kwargs["tool_choice"] == "auto"
675 +
676 +
677 +def test_complete_falls_back_to_chat_when_responses_shim_sends_empty_tools(
678 + monkeypatch,
679 +):
680 + calls: list[str] = []
681 +
682 + def fake_responses(*args, **kwargs):
683 + calls.append("responses")
684 + raise RuntimeError(
685 + "Value error, `tools` must not be an empty array. "
686 + "Either provide at least one tool or omit the field entirely."
687 + )
688 +
689 + def fake_completion(*args, **kwargs):
690 + calls.append("chat")
691 + assert kwargs["drop_params"] is True
692 + assert "tools" not in kwargs
693 + assert "tool_choice" not in kwargs
694 + assert "parallel_tool_calls" not in kwargs
695 + return {"choices": [{"message": {"content": "ok"}}]}
696 +
697 + monkeypatch.setattr(litellm_transport, "responses", fake_responses)
698 + monkeypatch.setattr(litellm_transport, "completion", fake_completion)
699 +
700 + transport = litellm_transport.LiteLLMTransport(
701 + model="hosted_vllm/qwen",
702 + messages=[{"role": "user", "content": "hi"}],
703 + kwargs={
704 + "tools": [],
705 + "tool_choice": "auto",
706 + "parallel_tool_calls": True,
707 + "max_tokens": 8,
708 + },
709 + )
710 +
711 + parsed = transport.complete()
712 +
713 + assert parsed["response_delta"] == "ok"
714 + assert calls == ["responses", "chat"]
715 +
716 +
717 def test_responses_request_adds_openai_prompt_cache_key_for_static_prefix():
718 request = litellm_transport.ResponsesTransport.from_chat(
719 [