@cryptotaxi247 / SquadScope
Code Commits Issues Pulls Sessions CI/CD
main
main
md 453 lines 20 KB
Rendered Raw
1 # PRD: Cost Estimation and Optimization for Token-Based Copilot Billing
2
3 **Author:** Leela (Lead/Architect)
4 **Date:** 2026-05-19
5 **Status:** Draft
6 **Relates to:** docs/PRD.md, .squad/decisions.md (CI Architecture Decision)
7
8 ---
9
10 ## Executive Summary
11
12 SquadScope runs automated AI analysis weekly using GitHub Copilot CLI inside GitHub Actions; analysis is Copilot-only and has no GitHub Models/OpenAI operational fallback. With GitHub Copilot's shift to token-based consumption billing (AI Credits at $0.01/credit), every pipeline run has a measurable cost. This PRD quantifies per-run and annual costs, projects context growth over 6–12 months, and defines optimization strategies and budget controls to keep SquadScope economically sustainable as a zero-revenue open-source project.
13
14 **Key finding:** A weekly analysis run costs approximately **$0.27–$0.35** in AI credits. The annual projection table totals **$16.18/year** at current configuration. Accounting for context growth (5–10% over 12 months), diagnostic no-AI candidate runs after Copilot failures, and variance in weekly token counts, the realistic annual range is **$16–$20/year** — comparable to a cheap newsletter service, and orders of magnitude cheaper than a human analyst.
15
16 ---
17
18 ## Problem Statement
19
20 ### Why Cost Matters for Automated Copilot Usage
21
22 1. **Predictability:** Unlike interactive Copilot chat (included in subscription), automated CI invocations consume tokens that count against plan allowances and incur overage charges.
23 2. **Context growth:** SquadScope's wisdom, skills, and history accumulate over time, making each run progressively more expensive unless managed.
24 3. **Budget transparency:** As a personal open-source project, jmservera needs clear visibility into the marginal cost of each published page.
25 4. **Plan selection:** Understanding token consumption informs whether Copilot Pro ($10/month, 300 credits included) or Copilot Pro+ ($39/month, 1500 credits) is the right tier.
26 5. **Graceful degradation:** If token budgets are exhausted, the pipeline must degrade gracefully (use cheaper models, skip optional enrichment) rather than fail silently.
27
28 ---
29
30 ## Token Pricing Model Summary
31
32 *(Source: [GitHub Copilot Models and Pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing), fetched 2026-06-06. Prices must be reviewed every two months.)*
33
34 The notification-only `.github/workflows/copilot-pricing-review.yml` workflow runs every two months and opens or updates a GitHub issue when pricing data is due for manual review. It must not mutate pricing tables directly; changes go through code/docs/tests and PR review.
35
36 ### Core Concepts
37
38 - **1 AI Credit = $0.01 USD**
39 - Tokens are consumed for: input (prompt + context), cached input (reused context), and output (generated text)
40 - Plans include monthly credit allowances; overage billed at per-token rates
41 - Code completions remain unlimited on paid plans (not relevant to SquadScope CI)
42
43 ### Relevant Model Pricing (per 1M tokens)
44
45 | Model | Category | Input | Cached Input | Output | Cache Write |
46 |-------|----------|------:|-------------:|-------:|------------:|
47 | GPT-5 mini | OpenAI | $0.25 | $0.025 | $2.00 | — |
48 | GPT-5.3-Codex | OpenAI | $1.75 | $0.175 | $14.00 | — |
49 | GPT-5.4 (≤272K input) | OpenAI | $2.50 | $0.25 | $15.00 | — |
50 | GPT-5.4 (>272K input) | OpenAI | $5.00 | $0.50 | $22.50 | — |
51 | GPT-5.4 mini | OpenAI | $0.75 | $0.075 | $4.50 | — |
52 | GPT-5.4 nano | OpenAI | $0.20 | $0.02 | $1.25 | — |
53 | GPT-5.5 (≤272K input) | OpenAI | $5.00 | $0.50 | $30.00 | — |
54 | GPT-5.5 (>272K input) | OpenAI | $10.00 | $1.00 | $45.00 | — |
55 | Claude Haiku 4.5 | Anthropic | $1.00 | $0.10 | $5.00 | $1.25 |
56 | **Claude Sonnet 4 / 4.5 / 4.6** (primary) | Anthropic | $3.00 | $0.30 | $15.00 | $3.75 |
57 | Claude Opus 4.5 / 4.6 / 4.7 / 4.8 | Anthropic | $5.00 | $0.50 | $25.00 | $6.25 |
58 | Gemini 2.5 Pro | Google | $1.25 | $0.125 | $10.00 | — |
59 | Gemini 3 Flash | Google | $0.50 | $0.05 | $3.00 | — |
60 | Gemini 3.1 Pro (≤200K input) | Google | $2.00 | $0.20 | $12.00 | — |
61 | Gemini 3.1 Pro (>200K input) | Google | $4.00 | $0.40 | $18.00 | — |
62 | Gemini 3.5 Flash | Google | $1.50 | $0.15 | $9.00 | — |
63 | Raptor mini | Fine-tuned/GitHub | $0.25 | $0.025 | $2.00 | — |
64 | MAI-Code-1-Flash | Microsoft | $0.75 | $0.075 | $4.50 | — |
65
66 ### Plan Allowances
67
68 | Plan | Monthly Credits | Equivalent $ | Notes |
69 |------|----------------|--------------|-------|
70 | Copilot Free | ~13.33 | $0.13 | Very limited |
71 | Copilot Pro | 300 | $3.00 | Likely sufficient for SquadScope alone |
72 | Copilot Pro+ | 1500 | $15.00 | Generous headroom |
73 | Copilot Business | 200/user pooled | $2.00/user | Org billing |
74
75 ---
76
77 ## Cost Breakdown per Pipeline Stage
78
79 ### Tokenization Assumptions
80
81 - Average ~4 characters per token (English text/markdown)
82 - JSON is less efficient: ~3.5 characters per token due to structural characters
83 - 1 KB of prose ≈ 250 tokens; 1 KB of JSON ≈ 285 tokens
84
85 ### Stage 1: Weekly Analysis (Copilot CLI — Claude Sonnet 4)
86
87 | Component | Size | Estimated Tokens |
88 |-----------|------|-----------------|
89 | Analyze prompt template | 6.5 KB | ~1,625 |
90 | Raw weekly JSON (`data/raw/2026-W21.json`) | 301 KB | ~86,000 |
91 | Wisdom file (`.squad/identity/wisdom.md`) | 3.2 KB | ~800 |
92 | Skills directory (currently empty) | 0 KB | 0 |
93 | Previous week summary | ~5 KB | ~1,250 |
94 | System/tool overhead (Copilot CLI framing) | ~2 KB | ~500 |
95 | **Total input tokens** | **~318 KB** | **~90,175** |
96
97 | Output Component | Size | Estimated Tokens |
98 |------------------|------|-----------------|
99 | Analyzed summary markdown | ~5 KB | ~1,250 |
100 | Tool calls/internal reasoning overhead | ~3 KB | ~750 |
101 | **Total output tokens** | **~8 KB** | **~2,000** |
102
103 **Weekly analysis cost (Claude Sonnet 4):**
104
105 ```
106 Input: 90,175 tokens × $3.00/1M = $0.2705
107 Output: 2,000 tokens × $15.00/1M = $0.0300
108 ─────────────────────────────────────────────
109 Total per weekly run: ≈ $0.30
110 ```
111
112 **In AI Credits: ~30 credits per weekly run.**
113
114 ### Stage 2: Reskill (Every 5th Week — Copilot CLI — copilot-default)
115
116 The reskill run is larger because it reads 5 weeks of history plus snapshot data.
117
118 | Component | Size | Estimated Tokens |
119 |-----------|------|-----------------|
120 | Reskill prompt template | 2.9 KB | ~725 |
121 | Last 5 analyzed summaries (5 × ~5 KB) | ~25 KB | ~6,250 |
122 | Wisdom file | 3.2 KB | ~800 |
123 | Skills directory (growing) | ~2 KB (month 6) | ~500 |
124 | Star snapshot context (5 weeks) | ~15 KB | ~4,285 |
125 | Quality trend report | ~2 KB | ~500 |
126 | **Total input tokens** | **~50 KB** | **~13,060** |
127
128 | Output Component | Size | Estimated Tokens |
129 |------------------|------|-----------------|
130 | Reskill report | ~4 KB | ~1,000 |
131 | Wisdom updates | ~1 KB | ~250 |
132 | **Total output tokens** | **~5 KB** | **~1,250** |
133
134 **Reskill cost (copilot-default at Claude Sonnet 4 rates):**
135
136 ```
137 Input: 13,060 tokens × $3.00/1M = $0.0392
138 Output: 1,250 tokens × $15.00/1M = $0.0188
139 ─────────────────────────────────────────────
140 Total per reskill run: ≈ $0.058
141 ```
142
143 **In AI Credits: ~6 credits per reskill run.**
144
145 ### Stage 3: Copilot Failure Diagnosis (No GitHub Models/OpenAI fallback)
146
147 When Copilot CLI fails, weekly analysis must fail closed or produce a diagnostic no-AI candidate artifact that is publish-ineligible. There is no GitHub Models/OpenAI analysis fallback configured, so cost tooling may estimate alternate Copilot model choices but must not present GitHub Models as an operational recovery path.
148
149 ```
150 Publishable AI output: none
151 AI token cost for diagnostic no-AI artifact: $0.00
152 Operator action: inspect Copilot failure classification, token/auth issue, and workflow logs
153 ```
154
155 ### Stage 4: GitHub Actions Compute
156
157 | Job | Runner | Duration (est.) | Cost |
158 |-----|--------|-----------------|------|
159 | Crawl | ubuntu-latest | ~3 min | Free (public repo) |
160 | Analyze | ubuntu-latest | ~2 min | Free (public repo) |
161 | Generate | ubuntu-latest | ~1 min | Free (public repo) |
162 | Deploy | ubuntu-latest | ~2 min | Free (public repo) |
163 | Reskill | ubuntu-latest | ~2 min | Free (public repo) |
164
165 **GitHub Actions is free for public repositories.** If the repo becomes private, estimate ~10 minutes/run × $0.008/min = $0.08/run.
166
167 ---
168
169 ## Context Growth Projections
170
171 ### Growth Vectors
172
173 | Component | Current Size | Growth Rate | 6-Month Projection | 12-Month Projection |
174 |-----------|-------------|-------------|--------------------|--------------------|
175 | Wisdom.md | 3.2 KB | +0.5 KB per reskill (~every 5 weeks) | 5.8 KB | 8.4 KB |
176 | Skills directory | 0 KB | +1 KB per reskill (new skill files) | 5.2 KB | 10.4 KB |
177 | Raw JSON (per file) | 301 KB | Stable (weekly crawl scope fixed) | 301 KB | 301 KB |
178 | Star snapshots (per file) | ~3 KB | Stable per file, linear file count | 78 KB total | 156 KB total |
179 | Previous summary | 5 KB | Stable (only last week sent) | 5 KB | 5 KB |
180 | Analyzed archive | 5 KB × weeks | Linear growth | 130 KB (26 files) | 260 KB (52 files) |
181
182 ### Token Cost Trajectory
183
184 | Timeframe | Weekly Input Tokens | Weekly Cost | Reskill Input Tokens | Monthly Cost (4.3 weeks + 0.86 reskill amortized) |
185 |-----------|--------------------:|------------:|---------------------:|---:|
186 | Month 1 (now) | 90,175 | $0.30 | 13,060 | $1.32 |
187 | Month 6 | 92,300 (+2.4%) | $0.31 | 16,200 (+24%) | $1.36 |
188 | Month 12 | 94,500 (+4.8%) | $0.32 | 19,500 (+49%) | $1.40 |
189
190 **Key insight:** Context growth is modest because the dominant cost driver (raw JSON at 86K tokens) is stable. Wisdom and skills growth adds only ~2-5% annually to weekly runs. Reskill grows faster (24-49%) because it accumulates more historical context, but it runs infrequently.
191
192 ---
193
194 ## Cost per Page Calculation
195
196 ### Annual Cost Projection (Year 1)
197
198 | Line Item | Frequency | Unit Cost | Annual Cost |
199 |-----------|-----------|-----------|-------------|
200 | Weekly analysis (Claude Sonnet 4) | 52/year | $0.30 | $15.60 |
201 | Reskill (copilot-default / Claude Sonnet rates) | ~10/year | $0.058 | $0.58 |
202 | GitHub Actions compute | 52/year | $0.00 (public) | $0.00 |
203 | **Total annual AI cost** | | | **$16.18** |
204
205 ### Cost per Published Page
206
207 ```
208 Annual AI cost / 52 pages = $16.18 / 52 = $0.311 per page
209 ```
210
211 Including amortized reskill:
212 ```
213 ($15.60 + $0.58) / 52 = $0.311 per page (reskill is small)
214 ```
215
216 ### Comparative Analysis
217
218 | Approach | Annual Cost | Cost per Page | Quality |
219 |----------|-------------|---------------|---------|
220 | **SquadScope (automated)** | **~$16/year** | **$0.311** | Consistent, opinionated, improving |
221 | Human analyst (freelance) | $5,200–$10,400/year | $100–$200 | High but variable |
222 | Newsletter service (Substack Pro) | $600/year | $11.50 | Platform cost only, still need writer |
223 | Manual GPT-4 chat (copy-paste) | ~$50/year | ~$1 | Inconsistent, no learning loop |
224
225 **SquadScope is 300× cheaper than a human analyst and offers compounding quality via reskill.**
226
227 ---
228
229 ## Cost Optimization Strategies
230
231 ### Strategy 1: Model Selection by Task Criticality
232
233 | Task | Current Model | Optimized Model | Savings |
234 |------|---------------|-----------------|---------|
235 | Weekly analysis | Claude Sonnet 4 ($0.30) | GPT-5 mini ($0.023 input + $0.004 output) | **91%** |
236 | Reskill | copilot-default / Claude Sonnet rates ($0.058) | Keep Copilot-only; no GitHub Models fallback | 0% |
237 | Copilot failure diagnosis | No AI ($0.00) | Fail closed or produce publish-ineligible diagnostic artifact | N/A |
238
239 **Recommendation:** Start with Claude Sonnet 4 for quality. If quality_score consistently ≥ 75, experiment with GPT-5.4 mini, GPT-5 mini, or Claude Haiku 4.5 for weekly analysis. Reserve premium models for reskill where judgment quality matters most.
240
241 ### Strategy 2: Context Window Management
242
243 1. **Summarize raw JSON before sending:** Instead of sending 301 KB of raw JSON, pre-process to extract only the fields used by the prompt (~50 KB, saving ~60% of input tokens).
244 2. **Cap wisdom.md:** Establish a 5 KB soft limit. During reskill, retire obsolete heuristics rather than only appending.
245 3. **Compress star snapshots:** For reskill, send only delta summaries rather than full snapshot JSON.
246
247 **Potential savings:** 40-60% reduction in input tokens = ~$0.12–$0.18 savings per weekly run.
248
249 ### Strategy 3: Skip-If-Unchanged (Caching)
250
251 If the crawled data has fewer than N significant changes from the prior week (e.g., <5 new repos, <10% topic shift), skip analysis and republish last week's summary with an "unchanged" note.
252
253 **Potential savings:** 5-15% of annual runs skipped = $0.80–$2.40/year.
254
255 **Risk:** Breaks the "every week has a page" contract. Implement as opt-in only.
256
257 ### Strategy 4: Token Budget per Run
258
259 Set a hard cap on total tokens per invocation:
260
261 ```yaml
262 env:
263 SQUADSCOPE_TOKEN_BUDGET: 150000 # tokens
264 SQUADSCOPE_COST_CAP: 0.50 # USD per run
265 ```
266
267 If pre-calculated token estimate exceeds budget:
268 1. Truncate raw JSON to top 50 repos by stars_gained
269 2. Omit skills context
270 3. Shorten previous summary to frontmatter-only
271
272 ### Strategy 5: Prompt Optimization
273
274 | Optimization | Token Savings | Effort |
275 |--------------|---------------|--------|
276 | Remove output template (model knows format) | ~500 tokens | Low |
277 | Shorten editorial stance to bullet points | ~200 tokens | Low |
278 | Inline wisdom into prompt (skip file read) | ~100 tokens | Medium |
279 | Use structured JSON output instead of markdown | ~300 output tokens | Medium |
280
281 **Combined prompt optimization: ~1,100 tokens saved = ~$0.004/run (marginal).**
282
283 Prompt optimization has low ROI because the raw JSON dominates input cost. Focus on Strategy 2 (context window management) first.
284
285 ### Strategy 6: Cached Input Optimization
286
287 If the Copilot CLI supports prompt caching (reusing context across calls), the 86K raw JSON tokens could be served at cached rates:
288
289 ```
290 Cached: 86,000 × $0.30/1M = $0.026 (vs $0.258 uncached)
291 Savings: $0.232 per run = 77% reduction on the JSON portion
292 ```
293
294 **Status:** Copilot CLI caching behavior is not yet documented for CI invocations. Monitor for updates.
295
296 ---
297
298 ## Monitoring & Alerting Design
299
300 ### Per-Run Token Tracking
301
302 1. **Copilot CLI transcript:** The `--share=PATH` flag exports a session transcript. Parse it post-run to extract actual token counts.
303 2. **API-compatible response metadata (general tooling only):** The cost ledger can parse OpenAI-compatible `usage` JSON for non-analysis experiments, but weekly analysis remains Copilot-only and does not use GitHub Models/OpenAI fallback.
304 3. **Workflow annotations:** Log token estimates and actuals as workflow summary annotations.
305
306 ### Implementation
307
308 ```yaml
309 - name: Log token usage
310 if: always()
311 run: |
312 # Parse Copilot CLI transcript for usage data
313 if [ -f copilot-session.md ]; then
314 python3 scripts/track_token_usage.py \
315 --transcript copilot-session.md \
316 --stage analysis \
317 --week "$WEEK"
318 fi
319 ```
320
321 ### Usage Dashboard
322
323 Store per-run metrics in `data/metrics/token-usage.jsonl`:
324
325 ```json
326 {"week": "2026-W21", "stage": "analysis", "model": "claude-sonnet-4", "input_tokens": 90175, "output_tokens": 2000, "cost_usd": 0.30, "timestamp": "2026-05-19T08:00:00Z"}
327 ```
328
329 Render a simple chart on the SquadScope site (Hugo shortcode or static SVG) showing:
330 - Weekly cost trend
331 - Cumulative annual spend
332 - Context size growth
333
334 ### Budget Alerts
335
336 | Threshold | Action |
337 |-----------|--------|
338 | Single run > $0.50 | Warning annotation in workflow summary |
339 | Single run > $1.00 | Fail the run, open issue |
340 | Monthly cumulative > $5.00 | Email alert via GitHub Actions notification |
341 | Monthly cumulative > $10.00 | Auto-switch to GPT-5 mini for remaining month |
342
343 ---
344
345 ## Budget Controls (Hard Limits, Graceful Degradation)
346
347 ### Tiered Degradation Strategy
348
349 ```
350 Normal Mode (cost < $0.50/run)
351 └─ Full analysis with Claude Sonnet 4
352 └─ Full context (raw JSON + wisdom + skills + prior week)
353
354 Budget Mode (cost would exceed $0.50/run)
355 └─ Switch to GPT-5.4 mini (saves ~74%)
356 └─ Truncate raw JSON to top 100 repos
357 └─ Omit skills context
358
359 Minimal Mode (monthly budget exhausted)
360 └─ Switch to GPT-5 mini (saves ~91%)
361 └─ Truncate raw JSON to top 30 repos
362 └─ Omit all optional context
363 └─ Quality gate threshold lowered to 50
364
365 Emergency Mode (all credits exhausted)
366 └─ Skip AI analysis entirely
367 └─ Produce diagnostic/staged no-AI artifact only; publish-ineligible
368 └─ Open issue for manual intervention
369 ```
370
371 ### Pre-flight Cost Estimation
372
373 Before invoking the model, estimate cost:
374
375 ```python
376 from scripts.model_pricing import estimate_cost_usd
377
378
379 def estimate_cost(input_tokens: int, output_estimate: int, model: str) -> float | None:
380 # Centralized implementation lives in scripts/model_pricing.py.
381 return estimate_cost_usd(
382 model,
383 input_tokens=input_tokens,
384 output_tokens=output_estimate,
385 )
386 ```
387
388 ---
389
390 ## Implementation Plan
391
392 ### Issues to Create
393
394 | # | Title | Priority | Effort | Dependencies |
395 |---|-------|----------|--------|--------------|
396 | 1 | Add pre-flight token estimation to analyze workflow | High | S | None |
397 | 2 | Implement `scripts/track_token_usage.py` for post-run metrics | High | M | None |
398 | 3 | Create `data/metrics/token-usage.jsonl` schema and writer | Medium | S | #2 |
399 | 4 | Add budget alerts to workflow (annotations + issue creation) | Medium | M | #2 |
400 | 5 | Implement tiered degradation (model downgrade on budget) | Medium | M | #1 |
401 | 6 | Pre-process raw JSON to reduce token count (Strategy 2) | Medium | M | None |
402 | 7 | Add wisdom.md size cap and retirement policy to reskill | Low | S | None |
403 | 8 | Create cost dashboard Hugo shortcode | Low | L | #3 |
404 | 9 | Investigate Copilot CLI caching for CI (Strategy 6) | Low | S | None |
405 | 10 | Document model selection decision matrix | Low | S | None |
406
407 ### Phasing
408
409 - **Phase A (immediate):** Issues 1–3 — visibility into actual costs
410 - **Phase B (month 2):** Issues 4–6 — active cost management
411 - **Phase C (month 3+):** Issues 7–10 — optimization and documentation
412
413 ---
414
415 ## Open Questions
416
417 | # | Question | Impact | Proposed Resolution |
418 |---|----------|--------|---------------------|
419 | OQ1 | Does Copilot CLI expose actual token usage in transcript or exit metadata? | High — needed for accurate tracking | Spike: parse `--share` output for usage data |
420 | OQ2 | Does `copilot-requests: write` permission on GITHUB_TOKEN consume from org pool or personal allowance? | High — affects billing entity | Test in workflow with usage monitoring |
421 | OQ3 | Is prompt caching available for Copilot CLI in non-interactive mode? | Medium — could save 77% on JSON input | Monitor GitHub changelog |
422 | OQ4 | What's the actual token count for the raw JSON? (estimated 86K, need actuals) | Medium — calibration | Add tokenizer count in pre-flight step |
423 | OQ5 | How does GitHub bill for the Copilot CLI invocation itself vs. the underlying model tokens? | High — may have additional overhead | Review billing after first month |
424 | OQ6 | Can Copilot CLI expose more detailed actual token usage for cache/cached-input accounting? | Medium — affects estimate precision | Review Copilot CLI release notes and billing exports |
425 | OQ7 | What happens when the Copilot Pro credit allowance is consumed mid-month? | High — operational risk | Set up overage alerts, test degradation path |
426
427 ---
428
429 ## Appendix: Token Estimation Methodology
430
431 ### Tokenization Rules of Thumb
432
433 - English prose: ~4 characters/token (or ~0.75 words/token)
434 - JSON with short keys: ~3.5 characters/token
435 - Markdown with formatting: ~3.8 characters/token
436 - Code: ~3.2 characters/token
437
438 ### Validation Approach
439
440 Once `scripts/track_token_usage.py` is live, compare estimates against actuals for 4 weeks. Adjust multipliers if estimates deviate by >20%.
441
442 ### Raw JSON Breakdown (2026-W21.json = 301 KB)
443
444 Estimated token distribution:
445 - Structural JSON characters (`{}[],:"`): ~20% = ~17K tokens
446 - Repository names, URLs, descriptions: ~50% = ~43K tokens
447 - Numeric fields (stars, dates): ~15% = ~13K tokens
448 - Topic arrays: ~15% = ~13K tokens
449 - **Total: ~86K tokens** (at 3.5 chars/token)
450
451 ---
452
453 *This PRD will be updated with actuals once token tracking is implemented (Phase A, Issues 1–3).*