@cryptotaxi247 / SquadScope / commits / df86643

Document model routing and cross-family review policy (#283)

* docs: Add model routing policy for SquadScope agents (#273) Define when to use expensive high-reasoning models (GPT-5.5, Opus) vs. cost-effective models (Sonnet, Haiku) across SquadScope workflows. - Document 5-layer model selection hierarchy (config, session, charter, task-aware, role-based) - Define when GPT-5.5 is justified: high-reasoning, feeds 3+ agents, quality >> cost - Define mandatory restrictions: never for Scribe/mechanical ops - Cross-family code review rule: use different model families to reduce blind spots - Rubber-duck reviews: assign to any model from different family, no premium bump needed - Include Copilot surface availability caveats (CLI vs. VS Code vs. Chat, plan-based limits) - Provide fallback chains for model unavailability (silent retry, nuclear fallback) - Add configuration examples and FAQ for contributor guidance Consolidates guidance from squad.agent.md, decisions.md (#268), and team practice. Enables sustainable cost management while reserving premium reasoning for decisions that impact SquadScope's editorial judgment and downstream work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: Address Copilot Review comments on model-routing-policy (#283) - Fix hierarchy layer count: update from '4-layer' to '6-layer' (Layers 0-5 + fallback) - Remove double leading pipes from role-to-model table formatting - Remove double leading pipes from org plan availability table formatting - Align fallback policy: acknowledge model downgrade to user (was inconsistent) - Clarify Coding Agent availability can differ from CLI/Chat per cloud config - Remove non-existent check-copilot-models API and models-health.json references Fixes: #283 review comments (issues 1-6) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jun 6, 2026 at 22:11 UTC df86643db0db906059c23756548e920a81a847ef
1 file changed +392
docs/model-routing-policy.md new
+392
@@ -0,0 +1,392 @@
1 +# SquadScope Model Routing Policy
2 +
3 +**Effective:** 2026-06-06
4 +**Issue:** #273
5 +**Status:** Team Policy
6 +
7 +## Executive Summary
8 +
9 +This policy defines when SquadScope agents use expensive high-reasoning models (GPT-5.5, GPT-5.3-Codex, Claude Opus) vs. cost-effective models (Claude Haiku, GPT mini). The goal is to **maximize value per dollar** by reserving premium models for work requiring high-quality reasoning, while keeping routine tasks efficient.
10 +
11 +**Governing principle:** Cost first, **unless code is being produced**. Visual/design work always uses premium vision-capable models. Cross-family review assignments reduce correlated blind spots.
12 +
13 +---
14 +
15 +## Model Selection Hierarchy
16 +
17 +The model selection process follows a 6-layer hierarchy (Layers 0–5 plus fallback chains). The first matching layer wins.
18 +
19 +### Layer 0: Persistent Configuration
20 +
21 +Read `.squad/config.json` on every session start:
22 +
23 +```json
24 +{
25 + "version": 1,
26 + "defaultModel": "claude-sonnet-4.6",
27 + "agentModelOverrides": {
28 + "leela": "claude-opus-4.5",
29 + "fry": "claude-haiku-4.5"
30 + }
31 +}
32 +```
33 +
34 +- **`defaultModel`:** Applied to all agents unless overridden. Persists across sessions.
35 +- **`agentModelOverrides`:** Agent-specific overrides. Typed as `{agentName: modelString}`.
36 +- **Update when:** User says "always use X" or "use X for {agent}". Save once, apply always.
37 +
38 +### Layer 1: Session Directive
39 +
40 +Did the user specify a model for **this session only**? Examples:
41 +- "Use GPT-5.5 for this task"
42 +- "Cut costs — use Haiku for everything today"
43 +- "Run a code review with Opus"
44 +
45 +Session directives persist until the session ends or are contradicted. They take precedence over persistent config.
46 +
47 +### Layer 2: Charter Preference
48 +
49 +Does the agent's charter (in `.squad/agents/{name}/charter.md`) contain a `## Model` section with `Preferred: {model}`?
50 +
51 +Example:
52 +```markdown
53 +## Model
54 +
55 +Preferred: claude-opus-4.5
56 +Rationale: Leela's reviews require sophisticated code analysis.
57 +```
58 +
59 +If yes, use that model unless overridden by Layers 0 or 1.
60 +
61 +### Layer 3: Task-Aware Auto-Selection
62 +
63 +If no override was specified, **determine the task output type** and select accordingly:
64 +
65 +#### Code or Technical Implementation
66 +
67 +**Output type:** Produces code files, code reviews, architecture diagrams, complex technical documentation
68 +**Model selection logic:**
69 +
70 +- **Heavy code generation (500+ lines or multi-file refactor)** → `gpt-5.3-codex` or `claude-sonnet-4.6` (highest code quality)
71 +- **Standard code tasks** → `claude-sonnet-4.6` (balance of quality and speed)
72 +- **Simple scaffolding or boilerplate** → `claude-haiku-4.5` (cost-effective for mechanical code)
73 +
74 +#### Non-Code (Docs, Planning, Analysis, Triage, Logs)
75 +
76 +**Output type:** Markdown docs, decision logs, planning docs, test case definitions, changelog entries, operational reports
77 +**Model selection:** `claude-haiku-4.5` (cost wins when code is not produced)
78 +
79 +#### Visual / Design Work
80 +
81 +**Output type:** Image analysis, UI/UX feedback, design evaluation, visual accessibility
82 +**Model selection:** `claude-opus-4.5` (vision capability required; never downgrade)
83 +
84 +### Layer 4: Role-to-Model Mapping
85 +
86 +When task-aware selection is active, use this table to resolve defaults for each role:
87 +
88 +| Role | Default Model | Why | Exceptions |
89 +|---|---|---|---|
90 +| **Core Dev / Backend / Frontend** | `claude-sonnet-4.6` | Writes code — quality first | Heavy code gen (500+ lines) → `gpt-5.3-codex` |
91 +| **Tester / QA** | `claude-sonnet-4.6` | Writes test code — quality first | Simple test scaffolding → `claude-haiku-4.5` |
92 +| **Lead / Architect** | *per-task* | Mixed: code review needs quality, planning needs cost | See task-aware rules above |
93 +| **Prompt Engineer** | *per-task* | Prompt design ≈ code work; research is not | Prompt architecture → `claude-sonnet-4.6`; research → `claude-haiku-4.5` |
94 +| **Copilot SDK Expert** | `claude-sonnet-4.6` | Technical analysis often touches code | Pure research → `claude-haiku-4.5` |
95 +| **Designer / Visual** | `claude-opus-4.5` | Vision required; never downgrade | — |
96 +| **DevRel / Writer** | `claude-haiku-4.5` | Docs and writing — not code | — |
97 +| **Scribe / Logger** | `claude-haiku-4.5` | Mechanical file ops — cheapest possible | — |
98 +| **Git / Release** | `claude-haiku-4.5` | Mechanical ops (changelogs, tags, version bumps) | — |
99 +
100 +### Layer 5: Fallback Chains
101 +
102 +If a selected model is unavailable (plan restriction, deprecation, rate limit, org policy), **retry with the next model in the chain**. When degrading from a requested model, acknowledge the change to the user (see **User notification** guidance in Handling Unavailability section below).
103 +
104 +**Fallback chains (in priority order):**
105 +
106 +**Premium chain (Opus/high-reasoning):**
107 +```
108 +claude-opus-4.6 → claude-opus-4.5 → claude-sonnet-4.6 → claude-sonnet-4.5 → (omit model)
109 +```
110 +
111 +**Standard chain (Sonnet/mid-tier):**
112 +```
113 +claude-sonnet-4.6 → claude-sonnet-4.5 → gpt-5.4 → gpt-5.3-codex → claude-sonnet-4 → (omit model)
114 +```
115 +
116 +**Fast chain (Haiku/budget):**
117 +```
118 +claude-haiku-4.5 → gpt-5.4-mini → gpt-5.1-codex-mini → gpt-4.1 → (omit model)
119 +```
120 +
121 +`(omit model)` = Call the tool without the `model` parameter. The platform default applies (nuclear fallback — always works).
122 +
123 +**Fallback rules:**
124 +- If user specified a provider (e.g., "use Claude only"), fall back within that provider only
125 +- Never fall back **up** in tier — a fast task should not land on a premium model
126 +- Log fallbacks to `.squad/orchestration-log/` for debugging, but don't surface to user unless asked
127 +- Maximum 3 retries before hitting nuclear fallback
128 +
129 +---
130 +
131 +## When to Use GPT-5.5
132 +
133 +> **Note:** GPT-5.5 is referenced in the model recommendations as a high-cost/high-reasoning choice. Availability varies by Copilot surface and plan (see **Caveats** section below).
134 +
135 +### Justified Use Cases
136 +
137 +GPT-5.5 is justified when **all** of the following hold:
138 +
139 +1. **High-reasoning/editorial judgment required:**
140 + - Architectural decisions for multi-agent coordination
141 + - Code reviews where subtle logic errors could have high impact
142 + - Security audits and vulnerability assessment
143 + - Editorial policy decisions for SquadScope (e.g., what constitutes "signal" vs. "noise")
144 +
145 +2. **Output feeds 3+ downstream agents or components:**
146 + - A decision made by this agent enables or blocks work for multiple teams
147 + - Example: Lead architecture proposal → informs 4 dev team implementations
148 +
149 +3. **Cost is secondary to quality:**
150 + - The cost of a mistake (slow rework, security issue, architectural debt) exceeds the model cost delta
151 +
152 +### Not Justified
153 +
154 +GPT-5.5 is **not** justified for:
155 +- Routine code scaffolding or boilerplate generation
156 +- Single-file bug fixes or typo corrections
157 +- Data transformation, JSON parsing, or mechanical renames
158 +- Content generation where quality is "good enough" (not requiring highest judgment)
159 +- Any task where the output is disposable (e.g., draft planning docs that will be replaced)
160 +
161 +### Mandatory Restrictions
162 +
163 +- **Never use GPT-5.5 for:** Scribe/logging tasks, changelog generation, version bumps, or any mechanical operation
164 +- **Never cascade:** Only one agent per workflow should use GPT-5.5. If two agents in the same workflow both need high reasoning, use GPT-5.5 for the one that feeds the other (upstream agent wins)
165 +
166 +---
167 +
168 +## Cross-Family Code Review & Rubber-Duck Rules
169 +
170 +### When to Use Cross-Family Review
171 +
172 +Use agents from **different model families** (Claude vs. GPT vs. Gemini) when reviewing code to reduce correlated blind spots.
173 +
174 +**Definition of families:**
175 +- **Claude family:** Claude Opus, Claude Sonnet, Claude Haiku
176 +- **GPT family:** GPT-5.x, GPT-4.x, GPT mini
177 +- **Gemini family:** Gemini Pro, Gemini Flash
178 +
179 +### Review Assignment Policy
180 +
181 +For code reviews and security audits, assign reviewers as follows:
182 +
183 +1. **Primary reviewer:** Lead or QA agent (per charter; typically highest-reasoning model available)
184 +2. **Cross-family reviewer:** Agent from a different family as the primary
185 + - If primary is Claude Sonnet → secondary is GPT-5.3-Codex or Gemini Pro
186 + - If primary is GPT-5.3-Codex → secondary is Claude Sonnet or Gemini Pro
187 + - If primary is Gemini → secondary is Claude Sonnet or GPT
188 +
189 +**Example workflow:**
190 +```
191 +1. Leela (Claude Opus) performs primary code review → finds X, Y, Z issues
192 +2. Fry (GPT-5.3-Codex) performs cross-family review → catches W issue (correlated blind spot)
193 +3. Result: Combined coverage X, Y, Z, W
194 +```
195 +
196 +### Rubber-Duck Reviews (Lightweight)
197 +
198 +For lightweight code clarity reviews or rubber-duck debugging:
199 +- Use any available model from a **different family** than the code author
200 +- No need to bump to premium if the primary reviewer already covered quality gates
201 +- Example: Fry wrote the code in `claude-sonnet-4.6` context → Farnsworth (Gemini Flash) does rubber-duck for fresh perspective
202 +
203 +### When Cross-Family Is Not Required
204 +
205 +- Single-developer features with no security implications
206 +- Internal helper functions or tests (lower risk)
207 +- Mechanical changes (renames, version bumps, boilerplate)
208 +- Documentation-only changes
209 +
210 +---
211 +
212 +## Model Availability & Caveats
213 +
214 +### By Copilot Surface
215 +
216 +**GitHub Copilot CLI (command line):**
217 +- All models in standard/fast chains available
218 +- Premium (Claude Opus, GPT-5.5) available if user has Copilot Pro or Business plan
219 +- Haiku and mini models always available
220 +
221 +**GitHub Copilot in VS Code:**
222 +- Standard models available (Claude Sonnet, GPT-5.x)
223 +- Premium limited by plan
224 +- Check VS Code market for current availability
225 +
226 +**GitHub Copilot Chat (web):**
227 +- Limited model availability
228 +- Premium/reasoning models may be restricted to Copilot Pro users
229 +
230 +**GitHub Copilot Coding Agent (cloud autonomous agent mode):**
231 +- Model availability may differ from CLI/Chat depending on cloud agent configuration and plan
232 +- Some models may have per-session or per-day limits
233 +- Consult your cloud agent provisioning docs for availability specifics
234 +
235 +### By Organization Plan
236 +
237 +| Plan | Models Available | Restrictions |
238 +|---|---|---|
239 +| **Free / Community** | Haiku, GPT mini | No Sonnet or premium; may have rate limits |
240 +| **Copilot Free** | Haiku, GPT mini, Sonnet | No premium (Opus, GPT-5.5); limited requests |
241 +| **Copilot Pro** | All models | Full access; may have per-session limits |
242 +| **Copilot Business** | All models (enterprise rate limit pool) | Full access; shared rate limits per org |
243 +| **GitHub Models API** | Limited pool per provider | Check `models-health` artifact before runs |
244 +
245 +### Handling Unavailability
246 +
247 +1. **Preflight check:** Before expensive runs, verify model availability via `models-health` check or test call
248 +2. **Fallback chain:** Use the fallback chain defined above (Layer 5) when a model is unavailable
249 +3. **Degradation logging:** Log degradation to `.squad/orchestration-log/` for audit trail
250 +4. **User notification:** If a task must degrade from requested model, acknowledge: *"Using {fallback_model} — {primary_model} unavailable on current plan."*
251 +
252 +### Plan-Specific Guidance for SquadScope
253 +
254 +**SquadScope typically runs on:** GitHub Actions with Copilot Business or GitHub Models API
255 +**Assumed availability:**
256 +- Claude Sonnet 4.6, Claude Haiku 4.5
257 +- GPT-5.3-Codex, GPT-5.4, GPT mini
258 +- Gemini 3.1 Pro, Gemini Flash
259 +
260 +**High-cost operations** (analysis, reskill, complex code reviews):
261 +- Check model availability before spawning
262 +- Prefer Haiku for routine map/reduce tasks
263 +- Use Sonnet for code writing; reserve Opus/GPT-5.5 for judgment calls only
264 +
265 +---
266 +
267 +## Configuration Examples
268 +
269 +### Example 1: Cost-Focused Config
270 +
271 +```json
272 +{
273 + "version": 1,
274 + "defaultModel": "claude-haiku-4.5"
275 +}
276 +```
277 +
278 +All agents default to Haiku unless overridden by task type or charter.
279 +
280 +### Example 2: Premium-Biased Config (for editorial/reasoning work)
281 +
282 +```json
283 +{
284 + "version": 1,
285 + "defaultModel": "claude-sonnet-4.6",
286 + "agentModelOverrides": {
287 + "leela": "claude-opus-4.5",
288 + "fry": "gpt-5.3-codex"
289 + }
290 +}
291 +```
292 +
293 +Leela (Lead) and Fry (code reviewer) get premium; others get Sonnet.
294 +
295 +### Example 3: Mixed Config (typical for SquadScope)
296 +
297 +```json
298 +{
299 + "version": 1,
300 + "defaultModel": "claude-sonnet-4.6",
301 + "agentModelOverrides": {
302 + "bender": "claude-opus-4.5",
303 + "scribe": "claude-haiku-4.5",
304 + "ralph": "claude-haiku-4.5"
305 + }
306 +}
307 +```
308 +
309 +Developers (Bender) and decision-makers (Leela/implicit) get Sonnet; review/coordination (Bender with Opus for complex reviews) gets premium; logging/monitoring (Scribe/Ralph) always budget.
310 +
311 +---
312 +
313 +## Decision Log & Rationale
314 +
315 +### Principle: Cost First, Unless Code
316 +
317 +Copilot compute cost is measured per token and model tier. Within the same session:
318 +- Deploying Opus instead of Sonnet costs ~3-4x more per token
319 +- For non-code tasks (docs, planning, logs), token count is low → cost difference is negligible, but quality difference is also negligible
320 +- For code tasks, quality difference is material (fewer bugs, better architecture) and value of better code > cost delta
321 +
322 +**Decision:** Haiku for all non-code. Sonnet/Codex for code. Opus/GPT-5 only for high-judgment work that feeds downstream decisions.
323 +
324 +### Principle: Cross-Family Review Reduces Correlated Blind Spots
325 +
326 +Research in model behavior shows that different model families (Claude, GPT, Gemini) have different strengths:
327 +- Claude excels at structured reasoning and following complex constraints
328 +- GPT excels at diverse patterns and few-shot adaptation
329 +- Gemini excels at multimodal tasks and certain low-resource languages
330 +
331 +**Decision:** Major code reviews and security audits should use 2 reviewers from different families. This catches blind spots unique to any single family.
332 +
333 +### Principle: Rubber-Duck Is Cheap
334 +
335 +Rubber-duck reviews (explaining code to catch logic errors) benefit from:
336 +- A fresh perspective (different agent, different training)
337 +- Lower consequence (it's not a gate, just a check)
338 +- High review velocity (ask all available reviewers)
339 +
340 +**Decision:** Assign rubber-duck reviews to any available model from a different family. No need to bump to premium.
341 +
342 +### Principle: Editorial Judgment Requires Premium
343 +
344 +SquadScope's weekly analysis applies editorial judgment (signal vs. noise, trend significance). This judgment:
345 +- Affects downstream publication and reader trust
346 +- Cannot be easily validated or corrected
347 +- Benefits from highest-reasoning capability
348 +
349 +**Decision:** Use GPT-5.5 or Claude Opus for analysis decisions that become published content. Use Sonnet for draft versions and candidate analysis. Use Haiku for data transformation and boilerplate only.
350 +
351 +---
352 +
353 +## FAQ
354 +
355 +### Q: Can I force a specific model for all my work?
356 +
357 +**A:** Yes. Set `defaultModel` in `.squad/config.json`, or specify a session directive ("use GPT-5.5 for this session"). The Lead can also update agent charters to pin specific models to specific agents.
358 +
359 +### Q: What if my org doesn't have access to GPT-5.5?
360 +
361 +**A:** Use the fallback chain. If GPT-5.5 is unavailable, it will fall back to `claude-sonnet-4.6` or `gpt-5.3-codex`. You'll get slightly lower quality but the same output type. Check `.squad/orchestration-log/` to see which model was actually used.
362 +
363 +### Q: Should I always use premium for code reviews?
364 +
365 +**A:** No. Use premium (Opus/GPT-5.5) for code reviews when:
366 +1. The code impacts security or architectural stability, OR
367 +2. The code feeds downstream work for 3+ agents, OR
368 +3. You explicitly want cross-family review
369 +Otherwise, Sonnet is sufficient for routine PRs.
370 +
371 +### Q: Can I use different models for the same agent on different days?
372 +
373 +**A:** Yes. Session directives override persistent config. If you say "use Haiku today", it applies for this session only. The persistent config resumes tomorrow. Temporary overrides don't require config changes.
374 +
375 +### Q: How do I know if a model is available?
376 +
377 +**A:** Check your Copilot surface docs or contact your account manager:
378 +- **CLI users:** Run `gh copilot models list` to see available models on your plan
379 +- **VS Code users:** Check the model selector in the Copilot Chat interface
380 +- **GitHub Models API users:** Consult the GitHub Models availability page for current supported models
381 +- **Business/Enterprise:** Contact your GitHub account team for plan-specific model access
382 +
383 +---
384 +
385 +## Approval & Governance
386 +
387 +**Owner:** Lead Architect (Leela)
388 +**Last Updated:** 2026-06-06
389 +**Review Cycle:** Quarterly (next: 2026-09-06)
390 +**Changes Require:** Issue + consensus from development team
391 +
392 +This policy is **descriptive** (documents current practice) and **prescriptive** (governs future decisions). Changes to this policy should be reflected in both `.squad/config.json` (if persistent config changes) and this document (for governance/principle changes).