a972f870 · samitouri/QOSami_Agents_Zero

@@ -1,67 +0,0 @@

1 -# Agent Zero Model Efficiency Scorecard

2 -

3 -Date: 2026-05-10

4 -

5 -This scorecard synthesizes the untracked `EVIDENCE-*.md` provider sweeps. It is not a general intelligence benchmark. It scores how efficiently each provider/model pairing drove Agent Zero's live tool suite under short, median-user prompts.

6 -

7 -## Scoring

8 -

9 -Score is a 0-100 review score based on:

10 -

11 -- first-pass correct tool routing

12 -- successful completion without repair turns

13 -- local/server vs host/remote locality discipline

14 -- correct use of skill-gated tools

15 -- provider support for vision and remote/CUA surfaces

16 -- low final-answer drift, repeated tool loops, and hallucinated success

17 -

18 -Repairs, partial routing, provider-specific limitations, unnecessary tool probes, and cleanup hazards lower the score even when the final user-visible answer was correct.

19 -

20 -## Chart

21 -

23 -|---:|---|---:|---|---|

24 -| 1 | Nebius AI / `Qwen/Qwen3.5-397B-A17B` | 96 | `###################-` | Best broad reliability; most tools passed, with repair noise mostly confined to patch/scheduler. |

25 -| 2 | A0 Venice / `z-ai-glm-5v-turbo` | 92 | `##################--` | Strong unified-tool behavior; Venice image validation and patch repair remain the main costs. |

26 -| 3 | OpenRouter / `anthropic/claude-sonnet-4.6` | 89 | `##################--` | Cleanest locality and patching; remaining gaps are higher-level routing and semantic scope. |

27 -| 4 | OpenRouter / `xiaomi/mimo-v2.5-pro` | 88 | `##################--` | Strong explicit-tool performer; natural remote patch and unsupported vision endpoint hurt reliability. |

28 -| 5 | Moonshot AI / `kimi-2.6` | 87 | `#################---` | Good primitives and scheduler; skill read_file, document query routing, and vision perception were weaker. |

29 -| 6 | OpenRouter / `google/gemini-3.1-flash-lite` | 85 | `#################---` | Strong tool surface, but behavior/memory persistence leaked temporary rules into later chats. |

30 -| 7 | A0 Venice / `google-gemma-4-26b-a4b-it` | 83 | `#################---` | Good for a smaller open model after prompt cleanup; memory wording, scheduler, and Venice vision were fragile. |

31 -| 8 | OpenRouter / `anthropic/claude-haiku-4.5` | 81 | `################----` | Usable when explicit; natural memory, scheduler shape, CUA status, and notify defaults are less reliable. |

32 -| 9 | SambaNova / `MiniMax-M2.7` | 71 | `##############------` | Concrete tools often worked, but provider noise, natural memory, notify loops, CUA, and vision were risky. |

33 -| 10 | OpenRouter / `openai/gpt-4.1-mini` | 67 | `#############-------` | Basic syntax is competent, but natural routing often looks successful while using the wrong tool path. |

34 -| 11 | Nebius AI / `nvidia/Nemotron-3-Nano-Omni` | 66 | `#############-------` | Most affected by stale final answers, local/remote confusion, memory misses, and patch instability. |

35 -

36 -## Cross-Provider Failure Clusters

37 -

38 -- Natural document questions often use `text_editor` instead of `document_query`.

39 -- Skill requests sometimes jump straight to `load` and skip `search`, even when the user asks to find a skill.

40 -- Scheduler reminders often start with ISO timestamps in `schedule` instead of cron fields or `plan`.

41 -- Normal notifications often omit `priority: 10` or use `success` styling for plain notes.

42 -- Patch requests are still cognitively expensive for smaller models, especially after locality ambiguity.

43 -- Natural memory requests conflict with prompt-include guidance unless the prompt separates durable memory from project instruction files.

44 -- Behavior adjustments can leave vector-memory residue even when `behaviour.md` is restored.

45 -- Vision reliability is provider-specific; some endpoints reject image input or validate embedded media differently.

46 -- CUA should stay skill-gated and beta; status is generally safer than capture/action testing.

47 -

48 -## Improvements Applied From This Pass

49 -

50 -- Renamed high-impact skills to task-oriented names and moved plugin-owned skills into their plugin folders.

51 -- Updated skill frontmatter on renamed skills toward the official `name` + `description` standard.

52 -- Clarified memory-vs-promptinclude guidance so "remember/forget" routes to memory tools.

53 -- Clarified scheduler one-time vs cron task shapes and timezone handling.

54 -- Clarified `document_query` as the preferred document-QA tool for document paths and URLs.

55 -- Clarified `skills_tool` search/load/read_file order and required arguments.

56 -- Clarified normal notification priority/type.

57 -- Clarified local and host text-editor patch guidance when the user says not to rewrite.

58 -

59 -## Next Candidates

60 -

61 -- Make memory delete/forget clean up derived fragments more predictably, or expose fragment cleanup as an explicit backend result.

62 -- Add behavior-adjustment scoping and cleanup protection so temporary rules do not become persistent vector memories.

63 -- Consider a scheduler convenience path for one-time reminders that maps ordinary time phrases to `create_planned_task` without relying on model cron synthesis.

64 -- Add provider capability checks before `vision_load` injects image content into text-only or image-rejecting endpoints.

65 -- Improve patch ergonomics with a simpler replace-by-exact-text path for both local and host file editors.

66 -- Add notify de-duplication/loop protection at the tool layer for repeated identical notifications.

67 -- Improve A2A error text around required response history so models do not treat `(no response)` as success.

Delete model-efficiency-scorecard.md