docs: add model efficiency scorecard
Summarize the provider evidence sweeps into a scored Agent Zero tool-efficiency chart.\n\nDocument the scoring rubric, ranked provider/model results, recurring failure clusters, applied prompt/skill improvements, and follow-up candidates while keeping the raw EVIDENCE files untracked.
Alessandro committed
May 10, 2026 at 07:12 UTC
79fe46cd7bfc5543d6313aa38e932a3096f19c8a
1 file changed
+67
docs/agents/model-efficiency-scorecard.md
new
+67
@@ -0,0 +1,67 @@
1
+# Agent Zero Model Efficiency Scorecard
2
+
3
+Date: 2026-05-10
4
+
5
+This scorecard synthesizes the untracked `EVIDENCE-*.md` provider sweeps. It is not a general intelligence benchmark. It scores how efficiently each provider/model pairing drove Agent Zero's live tool suite under short, median-user prompts.
6
+
7
+## Scoring
8
+
9
+Score is a 0-100 review score based on:
10
+
11
+- first-pass correct tool routing
12
+- successful completion without repair turns
13
+- local/server vs host/remote locality discipline
14
+- correct use of skill-gated tools
15
+- provider support for vision and remote/CUA surfaces
16
+- low final-answer drift, repeated tool loops, and hallucinated success
17
+
18
+Repairs, partial routing, provider-specific limitations, unnecessary tool probes, and cleanup hazards lower the score even when the final user-visible answer was correct.
19
+
20
+## Chart
21
+
22
+| Rank | Provider / model | Score | Chart | Main read |
23
+|---:|---|---:|---|---|
24
+| 1 | Nebius AI / `Qwen/Qwen3.5-397B-A17B` | 96 | `###################-` | Best broad reliability; most tools passed, with repair noise mostly confined to patch/scheduler. |
25
+| 2 | A0 Venice / `z-ai-glm-5v-turbo` | 92 | `##################--` | Strong unified-tool behavior; Venice image validation and patch repair remain the main costs. |
26
+| 3 | OpenRouter / `anthropic/claude-sonnet-4.6` | 89 | `##################--` | Cleanest locality and patching; remaining gaps are higher-level routing and semantic scope. |
27
+| 4 | OpenRouter / `xiaomi/mimo-v2.5-pro` | 88 | `##################--` | Strong explicit-tool performer; natural remote patch and unsupported vision endpoint hurt reliability. |
28
+| 5 | Moonshot AI / `kimi-2.6` | 87 | `#################---` | Good primitives and scheduler; skill read_file, document query routing, and vision perception were weaker. |
29
+| 6 | OpenRouter / `google/gemini-3.1-flash-lite` | 85 | `#################---` | Strong tool surface, but behavior/memory persistence leaked temporary rules into later chats. |
30
+| 7 | A0 Venice / `google-gemma-4-26b-a4b-it` | 83 | `#################---` | Good for a smaller open model after prompt cleanup; memory wording, scheduler, and Venice vision were fragile. |
31
+| 8 | OpenRouter / `anthropic/claude-haiku-4.5` | 81 | `################----` | Usable when explicit; natural memory, scheduler shape, CUA status, and notify defaults are less reliable. |
32
+| 9 | SambaNova / `MiniMax-M2.7` | 71 | `##############------` | Concrete tools often worked, but provider noise, natural memory, notify loops, CUA, and vision were risky. |
33
+| 10 | OpenRouter / `openai/gpt-4.1-mini` | 67 | `#############-------` | Basic syntax is competent, but natural routing often looks successful while using the wrong tool path. |
34
+| 11 | Nebius AI / `nvidia/Nemotron-3-Nano-Omni` | 66 | `#############-------` | Most affected by stale final answers, local/remote confusion, memory misses, and patch instability. |
35
+
36
+## Cross-Provider Failure Clusters
37
+
38
+- Natural document questions often use `text_editor` instead of `document_query`.
39
+- Skill requests sometimes jump straight to `load` and skip `search`, even when the user asks to find a skill.
40
+- Scheduler reminders often start with ISO timestamps in `schedule` instead of cron fields or `plan`.
41
+- Normal notifications often omit `priority: 10` or use `success` styling for plain notes.
42
+- Patch requests are still cognitively expensive for smaller models, especially after locality ambiguity.
43
+- Natural memory requests conflict with prompt-include guidance unless the prompt separates durable memory from project instruction files.
44
+- Behavior adjustments can leave vector-memory residue even when `behaviour.md` is restored.
45
+- Vision reliability is provider-specific; some endpoints reject image input or validate embedded media differently.
46
+- CUA should stay skill-gated and beta; status is generally safer than capture/action testing.
47
+
48
+## Improvements Applied From This Pass
49
+
50
+- Renamed high-impact skills to task-oriented names and moved plugin-owned skills into their plugin folders.
51
+- Updated skill frontmatter on renamed skills toward the official `name` + `description` standard.
52
+- Clarified memory-vs-promptinclude guidance so "remember/forget" routes to memory tools.
53
+- Clarified scheduler one-time vs cron task shapes and timezone handling.
54
+- Clarified `document_query` as the preferred document-QA tool for document paths and URLs.
55
+- Clarified `skills_tool` search/load/read_file order and required arguments.
56
+- Clarified normal notification priority/type.
57
+- Clarified local and host text-editor patch guidance when the user says not to rewrite.
58
+
59
+## Next Candidates
60
+
61
+- Make memory delete/forget clean up derived fragments more predictably, or expose fragment cleanup as an explicit backend result.
62
+- Add behavior-adjustment scoping and cleanup protection so temporary rules do not become persistent vector memories.
63
+- Consider a scheduler convenience path for one-time reminders that maps ordinary time phrases to `create_planned_task` without relying on model cron synthesis.
64
+- Add provider capability checks before `vision_load` injects image content into text-only or image-rejecting endpoints.
65
+- Improve patch ergonomics with a simpler replace-by-exact-text path for both local and host file editors.
66
+- Add notify de-duplication/loop protection at the tool layer for repeated identical notifications.
67
+- Improve A2A error text around required response history so models do not treat `(no response)` as success.