feat: cost dashboard, caching investigation, model matrix (#87, #88, #89) (#112)
- Add cost-dashboard Hugo shortcode with SVG bars and HTML tables - Add sample cost-summary.json data file - Add dashboard content page - Document Copilot CLI caching limitations and recommendations - Add model selection decision matrix with budget tiers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
May 19, 2026 at 16:31 UTC
8cde9a7ccf2de757abfc838dac89749092b3c36f
5 files changed
+194
content/dashboard.md
new
+5
@@ -0,0 +1,5 @@
1
+---
2
+title: "Cost Dashboard"
3
+layout: "single"
4
+---
5
+{{< cost-dashboard >}}
data/metrics/cost-summary.json
new
+9
@@ -0,0 +1,9 @@
1
+{
2
+ "weeks": [
3
+ {"week": "2026-W19", "cost": 0.32, "tokens_in": 82000, "tokens_out": 3500, "model": "claude-sonnet-4"},
4
+ {"week": "2026-W20", "cost": 0.28, "tokens_in": 75000, "tokens_out": 3200, "model": "claude-sonnet-4"},
5
+ {"week": "2026-W21", "cost": 0.35, "tokens_in": 86000, "tokens_out": 4000, "model": "claude-sonnet-4"}
6
+ ],
7
+ "cumulative_cost": 0.95,
8
+ "budget_limit": 10.00
9
+}
docs/decisions/copilot-cli-caching.md
new
+58
@@ -0,0 +1,58 @@
1
+# Copilot CLI Caching Investigation
2
+
3
+## Status: No Explicit Cache Support Available
4
+
5
+**Date:** 2026-05-21
6
+**Author:** Amy & Fry (SquadScope Squad)
7
+
8
+## Current Status
9
+
10
+### Copilot CLI (Primary Path)
11
+
12
+- Copilot CLI in CI mode does **NOT** support explicit prompt caching
13
+- No flags or configuration options exist to enable cache reuse between runs
14
+- Each invocation sends the full prompt context to the model
15
+- GitHub has not announced any roadmap for prompt caching in the CLI
16
+
17
+### GitHub Models API (Fallback Path)
18
+
19
+- The GitHub Models API also does **NOT** expose cache control headers or parameters
20
+- Token usage is billed per-request with no discount for repeated prefixes
21
+- No mechanism to pin or reuse cached prompt segments
22
+
23
+### Anthropic API (Direct Access)
24
+
25
+- Anthropic's API **does** support prompt caching via `cache_control` blocks
26
+- Cached input tokens are billed at $0.30/1M instead of $3.00/1M (for Claude Sonnet 4)
27
+- Cache has a 5-minute TTL with automatic extension on cache hits
28
+- Requires direct API access (not available through Copilot CLI or GitHub Models)
29
+
30
+## Cost Impact Analysis
31
+
32
+| Scenario | Input Token Cost | Savings |
33
+|----------|-----------------|---------|
34
+| No caching (current) | $3.00/1M tokens | — |
35
+| With Anthropic caching | $0.30/1M tokens (cached) | ~77% reduction |
36
+| Typical weekly run (~80K tokens) | $0.24 → $0.056 | ~$0.18/run saved |
37
+
38
+For our current usage (~3 runs/week), potential monthly savings: ~$2.16
39
+
40
+## Recommendation
41
+
42
+1. **Short-term:** Continue using Copilot CLI as primary path. The convenience and integration benefits outweigh the caching cost savings at our current volume.
43
+
44
+2. **Medium-term:** Monitor GitHub's announcements for:
45
+ - Prompt caching support in Copilot CLI
46
+ - Cache-aware billing in GitHub Models API
47
+ - Any new `--cache` or `--session` flags
48
+
49
+3. **Long-term / High-volume:** If SquadScope scales to daily runs or multi-org deployments, consider switching to direct Anthropic API access to leverage prompt caching. This becomes worthwhile when:
50
+ - Monthly token volume exceeds 1M input tokens
51
+ - The same system prompt is reused across multiple runs within 5 minutes
52
+ - Cost savings justify the added complexity of API key management
53
+
54
+## References
55
+
56
+- [Anthropic Prompt Caching Docs](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
57
+- [GitHub Copilot CLI Documentation](https://docs.github.com/en/copilot/using-github-copilot/using-github-copilot-in-the-command-line)
58
+- [GitHub Models API](https://docs.github.com/en/github-models)
docs/decisions/model-selection-matrix.md
new
+32
@@ -0,0 +1,32 @@
1
+# Model Selection Decision Matrix
2
+
3
+## Current Configuration
4
+
5
+| Task | Model | Cost/Run | Quality Req | Notes |
6
+|------|-------|----------|-------------|-------|
7
+| Weekly Analysis | Claude Sonnet 4 | ~$0.35 | quality_score ≥ 60 | Primary, full context |
8
+| Reskill | Claude Sonnet 4 | ~$0.10 | N/A (advisory) | Lower token count |
9
+| Fallback Analysis | GitHub Models GPT-4.1 | ~$0.25 | quality_score ≥ 50 | When Copilot CLI unavailable |
10
+| Budget Mode | GPT-4.1 | ~$0.20 | quality_score ≥ 50 | Truncated context |
11
+| Minimal Mode | GPT-5 mini | ~$0.05 | quality_score ≥ 40 | Top 30 repos only |
12
+| Scoring | Local (no AI) | $0.00 | N/A | Heuristic-based |
13
+| Pre-flight | Local (no AI) | $0.00 | N/A | Token counting only |
14
+
15
+## Decision Criteria
16
+
17
+1. Monthly budget remaining > 50%: use Claude Sonnet 4
18
+2. Monthly budget 20-50%: switch to GPT-4.1
19
+3. Monthly budget < 20%: switch to GPT-5 mini
20
+4. Monthly budget exhausted: emergency mode (raw stats only)
21
+
22
+## Quality Thresholds
23
+
24
+- Below quality_score 40: reject and retry with better model
25
+- Below quality_score 50: acceptable for budget mode only
26
+- Above quality_score 60: production quality
27
+
28
+## Evolution Plan
29
+
30
+- Review monthly based on accumulated quality data
31
+- Adjust thresholds if model pricing changes
32
+- Consider direct Anthropic API if caching becomes critical
layouts/shortcodes/cost-dashboard.html
new
+90
@@ -0,0 +1,90 @@
1
+{{- $data := site.Data.metrics.cost_summary -}}
2
+{{- $weeks := $data.weeks -}}
3
+{{- $cumulative := $data.cumulative_cost -}}
4
+{{- $budget := $data.budget_limit -}}
5
+{{- $pctUsed := mul (div $cumulative $budget) 100 -}}
6
+
7
+<div class="cost-dashboard">
8
+ <h2>💰 Cost Dashboard</h2>
9
+
10
+ <!-- Cumulative Spend -->
11
+ <section class="cumulative-spend">
12
+ <h3>Cumulative Spend</h3>
13
+ <p><strong>${{ printf "%.2f" $cumulative }}</strong> of ${{ printf "%.2f" $budget }} budget ({{ printf "%.1f" $pctUsed }}% used)</p>
14
+ <svg width="100%" height="30" viewBox="0 0 400 30" xmlns="http://www.w3.org/2000/svg">
15
+ <rect x="0" y="5" width="400" height="20" rx="4" fill="#e0e0e0"/>
16
+ <rect x="0" y="5" width="{{ mul 4 $pctUsed }}" height="20" rx="4" fill="{{ if lt $pctUsed 50.0 }}#4caf50{{ else if lt $pctUsed 80.0 }}#ff9800{{ else }}#f44336{{ end }}"/>
17
+ </svg>
18
+ </section>
19
+
20
+ <!-- Weekly Cost Trend -->
21
+ <section class="weekly-trend">
22
+ <h3>Weekly Cost Trend</h3>
23
+ <table>
24
+ <thead>
25
+ <tr>
26
+ <th>Week</th>
27
+ <th>Cost</th>
28
+ <th>Tokens In</th>
29
+ <th>Tokens Out</th>
30
+ <th>Model</th>
31
+ <th>Bar</th>
32
+ </tr>
33
+ </thead>
34
+ <tbody>
35
+ {{- $maxCost := 0.0 -}}
36
+ {{- range $weeks -}}
37
+ {{- if gt .cost $maxCost }}{{ $maxCost = .cost }}{{ end -}}
38
+ {{- end -}}
39
+ {{- range $weeks }}
40
+ <tr>
41
+ <td>{{ .week }}</td>
42
+ <td>${{ printf "%.2f" .cost }}</td>
43
+ <td>{{ .tokens_in | lang.FormatNumber 0 }}</td>
44
+ <td>{{ .tokens_out | lang.FormatNumber 0 }}</td>
45
+ <td>{{ .model }}</td>
46
+ <td>
47
+ {{- $barWidth := mul (div .cost $maxCost) 100 -}}
48
+ <svg width="120" height="16" xmlns="http://www.w3.org/2000/svg">
49
+ <rect x="0" y="2" width="{{ $barWidth }}%" height="12" rx="2" fill="#2196f3"/>
50
+ </svg>
51
+ </td>
52
+ </tr>
53
+ {{- end }}
54
+ </tbody>
55
+ </table>
56
+ </section>
57
+
58
+ <!-- Model Usage Distribution -->
59
+ <section class="model-distribution">
60
+ <h3>Model Usage Distribution</h3>
61
+ {{- $models := dict -}}
62
+ {{- range $weeks -}}
63
+ {{- $prev := index $models .model | default 0 -}}
64
+ {{- $models = merge $models (dict .model (add $prev 1)) -}}
65
+ {{- end -}}
66
+ <table>
67
+ <thead>
68
+ <tr><th>Model</th><th>Weeks Used</th><th>Share</th></tr>
69
+ </thead>
70
+ <tbody>
71
+ {{- $total := len $weeks -}}
72
+ {{- range $model, $count := $models }}
73
+ <tr>
74
+ <td>{{ $model }}</td>
75
+ <td>{{ $count }}</td>
76
+ <td>{{ printf "%.0f" (mul (div (float $count) (float $total)) 100) }}%</td>
77
+ </tr>
78
+ {{- end }}
79
+ </tbody>
80
+ </table>
81
+ </section>
82
+</div>
83
+
84
+<style>
85
+.cost-dashboard { font-family: system-ui, sans-serif; max-width: 800px; }
86
+.cost-dashboard section { margin: 1.5rem 0; }
87
+.cost-dashboard table { border-collapse: collapse; width: 100%; }
88
+.cost-dashboard th, .cost-dashboard td { border: 1px solid #ddd; padding: 0.5rem; text-align: left; }
89
+.cost-dashboard th { background: #f5f5f5; }
90
+</style>