Update Copilot pricing and scheduled review
Updates Copilot pricing calculations and adds the two-month scheduled pricing review workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
Jun 7, 2026 at 12:43 UTC
e158bb239816ca919552611318a266aab5c890f0
18 files changed
+681
-82
.github/workflows/copilot-pricing-review.yml
new
+51
@@ -0,0 +1,51 @@
1
+name: Copilot Pricing Review
2
+
3
+on:
4
+ schedule:
5
+ # Run every two months on the 6th, matching the pricing table's 2026-06-06 review baseline.
6
+ - cron: "23 9 6 2,4,6,8,10,12 *"
7
+ workflow_dispatch:
8
+
9
+permissions:
10
+ contents: read
11
+ issues: write
12
+
13
+jobs:
14
+ review-pricing:
15
+ runs-on: ubuntu-latest
16
+ steps:
17
+ # actions/checkout@v4 pinned to a full commit SHA for zizmor.
18
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
19
+ with:
20
+ persist-credentials: false
21
+
22
+ - name: Capture Copilot pricing source metadata
23
+ run: |
24
+ curl -fsSLI "https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing" > copilot-pricing-source-headers.txt || true
25
+
26
+ - name: Check Copilot pricing review status
27
+ id: pricing
28
+ run: |
29
+ python3 scripts/check_copilot_pricing_review.py \
30
+ --source-headers copilot-pricing-source-headers.txt \
31
+ --output copilot-pricing-review.md \
32
+ --github-output "$GITHUB_OUTPUT"
33
+
34
+ - name: Create or update pricing review issue
35
+ if: steps.pricing.outputs.needs_review == 'true'
36
+ env:
37
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
38
+ GH_REPO: ${{ github.repository }}
39
+ run: |
40
+ set -euo pipefail
41
+ TITLE="Copilot model pricing review required"
42
+ EXISTING=$(gh issue list --state open --search "in:title \"$TITLE\"" --json number --jq '.[0].number // empty')
43
+ if [ -n "$EXISTING" ]; then
44
+ gh issue comment "$EXISTING" --body-file copilot-pricing-review.md
45
+ echo "Updated existing pricing review issue #$EXISTING"
46
+ else
47
+ gh issue create \
48
+ --title "$TITLE" \
49
+ --label squad \
50
+ --body-file copilot-pricing-review.md
51
+ fi
.squad/agents/bender/history.md
+7
@@ -50,6 +50,13 @@
50
- Recommended PRD path is hybrid staged fan-out/fan-in: establish validated artifact contracts first, then gate RSS matrix, GitHub query matrix, and analysis map/reduce on measured thresholds.
51
- Run 27030646485 also showed analysis, not crawling, is the critical-path risk: three Copilot attempts consumed ~28m41s, failed quality gates, GitHub Models had no `openai/gpt-4o` access, and the workflow shipped via no-AI fallback with ~112.9k estimated input tokens.
52
- Issue #249 implementation: weekly analysis now writes to `data/candidates/<week>/<run_id>/` first and emits a `publish_eligibility_v1` manifest before any `data/analyzed/<week>-summary.md` promotion; promotion must fail closed on no-AI, stale source evidence, missing checksums, or failed validation.
53
+
54
+## Issue #291: Copilot Pricing Refresh (2026-06-06)
55
+
56
+- Implemented centralized model pricing in `scripts/model_pricing.py` as single source of truth for all model costs.
57
+- Added `.github/workflows/copilot-pricing-review.yml` for scheduled pricing review automation.
58
+- Pricing data now decoupled from scattered configuration; future pipeline cost analysis can rely on unified pricing module.
59
+- All tests pass; policy preservation: Copilot-only analysis requirement maintained.
60
- Analysis preflight now emits raw and prompt-visible repository evidence inventories with byte/token/checksum metadata; analysis gate rejects final repo links outside current raw evidence when inventory is available.
61
62
## Issue #287 — Analysis Gate Preflight Hardening (2026-06-06T21:23:50.664Z)
.squad/agents/fry/history.md
+6
@@ -110,6 +110,12 @@
110
- Good canonical weekly summary/content must remain unchanged when candidates are failed, degraded, no-AI, stale, missing-manifest, or malformed-manifest.
111
- Safe rerun promotion is copy-stable and must not append or duplicate article body content; ineligible candidates should remain in staging with promotion diagnostics for debugging.
112
113
+## Issue #291: Copilot Pricing Validation (2026-06-06)
114
+
115
+- Validated centralized pricing implementation in `scripts/model_pricing.py` for correctness and consistency.
116
+- Workflow safety review: `.github/workflows/copilot-pricing-review.yml` safe for production.
117
+- Full test suite passed (554 tests); no breaking changes detected.
118
+- Pricing validation will support future cost-aware pipeline decisions and analysis budget optimization.
119
## Issue #287 — Analysis Gate Preflight Hardening (2026-06-06T21:23:50.664Z)
120
121
- ✅ APPROVED: Bender's evidence inventory and gate failure classification implementation
.squad/log/2026-06-06T22:08:14Z-issue-291-copilot-pricing.md
new
+30
@@ -0,0 +1,30 @@
1
+# Session Log: Issue #291 Copilot Pricing Refresh
2
+
3
+**Timestamp:** 2026-06-06T22:08:14Z
4
+**Issue:** #291
5
+**PR:** #292
6
+**Agents:** Bender (Crawler, gpt-5.5), Fry (Tester, claude-sonnet-4.6), Coordinator
7
+
8
+## Summary
9
+
10
+Implemented complete Copilot pricing refresh with centralized model pricing and automated review workflow.
11
+
12
+## Deliverables
13
+
14
+- `scripts/model_pricing.py` — Centralized pricing source of truth
15
+- `.github/workflows/copilot-pricing-review.yml` — Scheduled review workflow
16
+- `scripts/check_copilot_pricing_review.py` — Pricing validation script
17
+- Updated documentation (cost, models)
18
+- Full test coverage validated
19
+
20
+## Policy Adherence
21
+
22
+✅ Copilot-only analysis policy preserved
23
+✅ README fallback wording corrected
24
+✅ All tests passing
25
+
26
+## Status
27
+
28
+✅ Ready for merge
29
+✅ PR checks green
30
+✅ Approved by team
.squad/orchestration-log/2026-06-06T22:08:14Z-bender.md
new
+33
@@ -0,0 +1,33 @@
1
+# Bender (Crawler) — Orchestration Log
2
+
3
+**Session:** 2026-06-06T22:08:14Z
4
+**Task:** Implement issue #291 Copilot pricing refresh and scheduled review workflow
5
+**Model:** gpt-5.5
6
+**Mode:** Sync
7
+
8
+## Outcome
9
+
10
+✅ **COMPLETE**
11
+
12
+- Centralized model pricing in `scripts/model_pricing.py`
13
+- Added workflow `.github/workflows/copilot-pricing-review.yml`
14
+- Added pricing review script `scripts/check_copilot_pricing_review.py`
15
+- Updated cost documentation and model documentation
16
+- Updated and verified all related tests
17
+- Preserved Copilot-only analysis policy
18
+- Corrected README fallback wording
19
+
20
+## Key Decisions
21
+
22
+1. **Pricing Centralization:** All model pricing data consolidated into single, maintainable source
23
+2. **Workflow Integration:** Scheduled pricing review workflow added to GitHub Actions
24
+3. **Policy Preservation:** Copilot-only analysis requirement maintained across all changes
25
+4. **Documentation Accuracy:** Fallback behavior correctly documented
26
+
27
+## Test Results
28
+
29
+All tests passed. Pricing data validated. Workflow syntax verified.
30
+
31
+## Git Integration
32
+
33
+Changes ready for merge into feature branch and subsequent PR #292.
.squad/orchestration-log/2026-06-06T22:08:14Z-fry.md
new
+33
@@ -0,0 +1,33 @@
1
+# Fry (Tester) — Orchestration Log
2
+
3
+**Session:** 2026-06-06T22:08:14Z
4
+**Task:** Review and validate pricing/workflow update
5
+**Model:** claude-sonnet-4.6
6
+**Mode:** Sync
7
+
8
+## Outcome
9
+
10
+✅ **APPROVE**
11
+
12
+- Price table verified for accuracy and consistency
13
+- Workflow validation: safe for production
14
+- Full test suite executed: all tests passed
15
+- No breaking changes detected
16
+- Documentation reviewed: clear and correct
17
+
18
+## Validation Coverage
19
+
20
+1. **Pricing Data Integrity:** Verified all model prices correct and consistent
21
+2. **Workflow Safety:** Checked trigger conditions, job dependencies, permissions
22
+3. **Test Coverage:** Ran full test suite with no failures
23
+4. **Backward Compatibility:** Verified existing integrations unaffected
24
+
25
+## Sign-Off
26
+
27
+Ready for merge. No blockers identified. Recommends proceeding with PR #292.
28
+
29
+## Next Steps
30
+
31
+- Merge to main branch
32
+- Deploy to production
33
+- Monitor first pricing review cycle for any issues
README.md
+4
-4
@@ -32,7 +32,7 @@ JSON Markdown Hugo Pages Improvements
32
- Applies heuristic filtering (language, topic, description quality)
33
- Outputs: `data/raw/YYYY-WNN.json`, `data/raw/YYYY-WNN-external-news.json`, `data/snapshots/YYYY-WNN-stars.json`
34
35
-**Stage 2: Analyze** (Copilot CLI or fallback)
35
+**Stage 2: Analyze** (Copilot CLI only)
36
- Reads raw JSON; applies AI analysis to classify repos as signal/noise/gaps
37
- Outputs: `data/analyzed/YYYY-WNN-summary.md` with quality score and summary sections
38
- Quality gate: Blocks publish if quality_score < 60 or missing required sections
@@ -58,7 +58,7 @@ JSON Markdown Hugo Pages Improvements
58
- **Notifications:** RSS feeds + GitHub Releases
59
- **Automation:** GitHub Actions
60
- **Deployment:** GitHub Pages
61
-- **Analysis engine:** Copilot CLI (with GitHub Models API fallback)
61
+- **Analysis engine:** Copilot CLI only; no GitHub Models/OpenAI analysis fallback
62
63
## Quick start
64
@@ -117,8 +117,8 @@ JSON Markdown Hugo Pages Improvements
117
118
### Required secrets
119
120
-- `COPILOT_GH_TOKEN` — Fine-grained PAT with **Account → Copilot Requests** permission (primary analysis)
121
-- `GITHUB_TOKEN` — Built-in; used for crawling, fallback analysis, commits, Pages deployment
120
+- `COPILOT_GH_TOKEN` — Fine-grained PAT with **Account → Copilot Requests** permission for Copilot CLI analysis
121
+- `GITHUB_TOKEN` — Built-in; used for crawling, commits, Pages deployment, and issue/notification automation
122
123
## Crawler notes
124
docs/decisions/model-selection-matrix.md
+9
-5
@@ -1,13 +1,17 @@
1
# Model Selection Decision Matrix
2
3
+Pricing source: [GitHub Copilot Models and Pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing), fetched 2026-06-06. Review prices every two months.
4
+
5
+Scheduled reminder: `.github/workflows/copilot-pricing-review.yml` checks this cadence every two months and opens/updates a review issue; pricing changes still require a normal PR.
6
+
7
## Current Configuration
8
9
| Task | Model | Cost/Run | Quality Req | Notes |
10
|------|-------|----------|-------------|-------|
11
| Weekly Analysis | Claude Sonnet 4 | ~$0.35 | quality_score ≥ 60 | Primary, full context |
12
| Reskill | Claude Sonnet 4 | ~$0.10 | N/A (advisory) | Lower token count |
9
-| Fallback Analysis | GitHub Models GPT-4.1 | ~$0.25 | quality_score ≥ 50 | When Copilot CLI unavailable |
10
-| Budget Mode | GPT-4.1 | ~$0.20 | quality_score ≥ 50 | Truncated context |
13
+| Copilot Failure Diagnosis | No AI | $0.00 | N/A | Copilot failures fail closed or produce publish-ineligible diagnostics; no GitHub Models/OpenAI fallback |
14
+| Budget Mode | Copilot GPT-5.4 mini | ~$0.08 | quality_score ≥ 50 | Truncated context, still through Copilot |
15
| Minimal Mode | GPT-5 mini | ~$0.05 | quality_score ≥ 40 | Top 30 repos only |
16
| Scoring | Local (no AI) | $0.00 | N/A | Heuristic-based |
17
| Pre-flight | Local (no AI) | $0.00 | N/A | Token counting only |
@@ -15,9 +19,9 @@
19
## Decision Criteria
20
21
1. Monthly budget remaining > 50%: use Claude Sonnet 4
18
-2. Monthly budget 20-50%: switch to GPT-4.1
22
+2. Monthly budget 20-50%: switch Copilot model to GPT-5.4 mini
23
3. Monthly budget < 20%: switch to GPT-5 mini
20
-4. Monthly budget exhausted: emergency mode (raw stats only)
24
+4. Monthly budget exhausted: diagnostic no-AI mode only (raw stats are publish-ineligible; no AI fallback)
25
26
## Quality Thresholds
27
@@ -27,6 +31,6 @@
31
32
## Evolution Plan
33
30
-- Review monthly based on accumulated quality data
34
+- Review model pricing every two months and quality data monthly
35
- Adjust thresholds if model pricing changes
36
- Consider direct Anthropic API if caching becomes critical
docs/model-routing-policy.md
+9
-5
@@ -10,6 +10,10 @@ This policy defines when SquadScope agents use expensive high-reasoning models (
10
11
**Governing principle:** Cost first, **unless code is being produced**. Visual/design work always uses premium vision-capable models. Cross-family review assignments reduce correlated blind spots.
12
13
+**Pricing source:** [GitHub Copilot Models and Pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing), fetched 2026-06-06. Pricing assumptions must be reviewed every two months.
14
+
15
+The scheduled workflow `.github/workflows/copilot-pricing-review.yml` runs every two months to open or update a review issue when the table is due. It is notification-only and must not change pricing data without a PR.
16
+
17
---
18
19
## Model Selection Hierarchy
@@ -115,7 +119,7 @@ claude-sonnet-4.6 → claude-sonnet-4.5 → gpt-5.4 → gpt-5.3-codex → claude
119
120
**Fast chain (Haiku/budget):**
121
```
118
-claude-haiku-4.5 → gpt-5.4-mini → gpt-5.1-codex-mini → gpt-4.1 → (omit model)
122
+claude-haiku-4.5 → gpt-5.4-mini → gpt-5-mini → gpt-5.4-nano → (omit model)
123
```
124
125
`(omit model)` = Call the tool without the `model` parameter. The platform default applies (nuclear fallback — always works).
@@ -251,11 +255,11 @@ For lightweight code clarity reviews or rubber-duck debugging:
255
256
### Plan-Specific Guidance for SquadScope
257
254
-**SquadScope typically runs on:** GitHub Actions with Copilot Business or GitHub Models API
258
+**SquadScope weekly analysis runs on:** GitHub Actions with Copilot CLI. GitHub Models/OpenAI fallback is not configured for analysis; Copilot failures fail closed or produce publish-ineligible diagnostic artifacts for operator triage.
259
**Assumed availability:**
260
- Claude Sonnet 4.6, Claude Haiku 4.5
257
-- GPT-5.3-Codex, GPT-5.4, GPT mini
258
-- Gemini 3.1 Pro, Gemini Flash
261
+- GPT-5.3-Codex, GPT-5.4, GPT-5.4 mini, GPT-5 mini
262
+- Gemini 3.1 Pro, Gemini 3 Flash, Gemini 3.5 Flash
263
264
**High-cost operations** (analysis, reskill, complex code reviews):
265
- Check model availability before spawning
@@ -386,7 +390,7 @@ Otherwise, Sonnet is sufficient for routine PRs.
390
391
**Owner:** Lead Architect (Leela)
392
**Last Updated:** 2026-06-06
389
-**Review Cycle:** Quarterly (next: 2026-09-06)
393
+**Review Cycle:** Every two months for pricing assumptions (next: 2026-08-06); broader routing policy quarterly.
394
**Changes Require:** Issue + consensus from development team
395
396
This policy is **descriptive** (documents current practice) and **prescriptive** (governs future decisions). Changes to this policy should be reflected in both `.squad/config.json` (if persistent config changes) and this document (for governance/principle changes).
docs/processed/PRD-cost-estimation.md
+59
-45
@@ -9,9 +9,9 @@
9
10
## Executive Summary
11
12
-SquadScope runs automated AI analysis weekly using GitHub Copilot CLI and GitHub Models API inside GitHub Actions. With GitHub Copilot's shift to token-based consumption billing (AI Credits at $0.01/credit), every pipeline run has a measurable cost. This PRD quantifies per-run and annual costs, projects context growth over 6–12 months, and defines optimization strategies and budget controls to keep SquadScope economically sustainable as a zero-revenue open-source project.
12
+SquadScope runs automated AI analysis weekly using GitHub Copilot CLI inside GitHub Actions; analysis is Copilot-only and has no GitHub Models/OpenAI operational fallback. With GitHub Copilot's shift to token-based consumption billing (AI Credits at $0.01/credit), every pipeline run has a measurable cost. This PRD quantifies per-run and annual costs, projects context growth over 6–12 months, and defines optimization strategies and budget controls to keep SquadScope economically sustainable as a zero-revenue open-source project.
13
14
-**Key finding:** A weekly analysis run costs approximately **$0.27–$0.35** in AI credits. The annual projection table totals **$15.96/year** at current configuration. Accounting for context growth (5–10% over 12 months), occasional fallback runs (~$0.20 each), and variance in weekly token counts, the realistic annual range is **$16–$20/year** — comparable to a cheap newsletter service, and orders of magnitude cheaper than a human analyst.
14
+**Key finding:** A weekly analysis run costs approximately **$0.27–$0.35** in AI credits. The annual projection table totals **$16.18/year** at current configuration. Accounting for context growth (5–10% over 12 months), diagnostic no-AI candidate runs after Copilot failures, and variance in weekly token counts, the realistic annual range is **$16–$20/year** — comparable to a cheap newsletter service, and orders of magnitude cheaper than a human analyst.
15
16
---
17
@@ -29,7 +29,9 @@ SquadScope runs automated AI analysis weekly using GitHub Copilot CLI and GitHub
29
30
## Token Pricing Model Summary
31
32
-*(Source: [GitHub Copilot Models and Pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing), fetched 2026-05-19)*
32
+*(Source: [GitHub Copilot Models and Pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing), fetched 2026-06-06. Prices must be reviewed every two months.)*
33
+
34
+The notification-only `.github/workflows/copilot-pricing-review.yml` workflow runs every two months and opens or updates a GitHub issue when pricing data is due for manual review. It must not mutate pricing tables directly; changes go through code/docs/tests and PR review.
35
36
### Core Concepts
37
@@ -41,12 +43,25 @@ SquadScope runs automated AI analysis weekly using GitHub Copilot CLI and GitHub
43
### Relevant Model Pricing (per 1M tokens)
44
45
| Model | Category | Input | Cached Input | Output | Cache Write |
44
-|-------|----------|-------|--------------|--------|-------------|
45
-| **Claude Sonnet 4** (primary) | Versatile | $3.00 | $0.30 | $15.00 | $3.75 |
46
-| **GPT-4.1** (fallback) | Versatile | $2.00 | $0.50 | $8.00 | — |
47
-| GPT-5 mini | Lightweight | $0.25 | $0.025 | $2.00 | — |
48
-| Claude Haiku 4.5 | Versatile | $1.00 | $0.10 | $5.00 | $1.25 |
49
-| Gemini 3 Flash | Lightweight | $0.50 | $0.05 | $3.00 | — |
46
+|-------|----------|------:|-------------:|-------:|------------:|
47
+| GPT-5 mini | OpenAI | $0.25 | $0.025 | $2.00 | — |
48
+| GPT-5.3-Codex | OpenAI | $1.75 | $0.175 | $14.00 | — |
49
+| GPT-5.4 (≤272K input) | OpenAI | $2.50 | $0.25 | $15.00 | — |
50
+| GPT-5.4 (>272K input) | OpenAI | $5.00 | $0.50 | $22.50 | — |
51
+| GPT-5.4 mini | OpenAI | $0.75 | $0.075 | $4.50 | — |
52
+| GPT-5.4 nano | OpenAI | $0.20 | $0.02 | $1.25 | — |
53
+| GPT-5.5 (≤272K input) | OpenAI | $5.00 | $0.50 | $30.00 | — |
54
+| GPT-5.5 (>272K input) | OpenAI | $10.00 | $1.00 | $45.00 | — |
55
+| Claude Haiku 4.5 | Anthropic | $1.00 | $0.10 | $5.00 | $1.25 |
56
+| **Claude Sonnet 4 / 4.5 / 4.6** (primary) | Anthropic | $3.00 | $0.30 | $15.00 | $3.75 |
57
+| Claude Opus 4.5 / 4.6 / 4.7 / 4.8 | Anthropic | $5.00 | $0.50 | $25.00 | $6.25 |
58
+| Gemini 2.5 Pro | Google | $1.25 | $0.125 | $10.00 | — |
59
+| Gemini 3 Flash | Google | $0.50 | $0.05 | $3.00 | — |
60
+| Gemini 3.1 Pro (≤200K input) | Google | $2.00 | $0.20 | $12.00 | — |
61
+| Gemini 3.1 Pro (>200K input) | Google | $4.00 | $0.40 | $18.00 | — |
62
+| Gemini 3.5 Flash | Google | $1.50 | $0.15 | $9.00 | — |
63
+| Raptor mini | Fine-tuned/GitHub | $0.25 | $0.025 | $2.00 | — |
64
+| MAI-Code-1-Flash | Microsoft | $0.75 | $0.075 | $4.50 | — |
65
66
### Plan Allowances
67
@@ -96,7 +111,7 @@ Total per weekly run: ≈ $0.30
111
112
**In AI Credits: ~30 credits per weekly run.**
113
99
-### Stage 2: Reskill (Every 5th Week — GitHub Models API — GPT-4.1)
114
+### Stage 2: Reskill (Every 5th Week — Copilot CLI — copilot-default)
115
116
The reskill run is larger because it reads 5 weeks of history plus snapshot data.
117
@@ -116,30 +131,27 @@ The reskill run is larger because it reads 5 weeks of history plus snapshot data
131
| Wisdom updates | ~1 KB | ~250 |
132
| **Total output tokens** | **~5 KB** | **~1,250** |
133
119
-**Reskill cost (GPT-4.1 via GitHub Models):**
134
+**Reskill cost (copilot-default at Claude Sonnet 4 rates):**
135
136
```
122
-Input: 13,060 tokens × $2.00/1M = $0.0261
123
-Output: 1,250 tokens × $8.00/1M = $0.0100
137
+Input: 13,060 tokens × $3.00/1M = $0.0392
138
+Output: 1,250 tokens × $15.00/1M = $0.0188
139
─────────────────────────────────────────────
125
-Total per reskill run: ≈ $0.036
140
+Total per reskill run: ≈ $0.058
141
```
142
128
-**In AI Credits: ~4 credits per reskill run.**
143
+**In AI Credits: ~6 credits per reskill run.**
144
130
-### Stage 3: Fallback Analysis (GitHub Models API — GPT-4.1)
145
+### Stage 3: Copilot Failure Diagnosis (No GitHub Models/OpenAI fallback)
146
132
-When Copilot CLI fails and the fallback triggers:
147
+When Copilot CLI fails, weekly analysis must fail closed or produce a diagnostic no-AI candidate artifact that is publish-ineligible. There is no GitHub Models/OpenAI analysis fallback configured, so cost tooling may estimate alternate Copilot model choices but must not present GitHub Models as an operational recovery path.
148
149
```
135
-Input: 90,175 tokens × $2.00/1M = $0.1804
136
-Output: 2,000 tokens × $8.00/1M = $0.0160
137
-─────────────────────────────────────────────
138
-Total per fallback run: ≈ $0.20
150
+Publishable AI output: none
151
+AI token cost for diagnostic no-AI artifact: $0.00
152
+Operator action: inspect Copilot failure classification, token/auth issue, and workflow logs
153
```
154
141
-**Fallback is ~33% cheaper than primary** due to GPT-4.1's lower rates vs Claude Sonnet 4.
142
-
155
### Stage 4: GitHub Actions Compute
156
157
| Job | Runner | Duration (est.) | Cost |
@@ -186,26 +198,26 @@ Total per fallback run: ≈ $0.20
198
| Line Item | Frequency | Unit Cost | Annual Cost |
199
|-----------|-----------|-----------|-------------|
200
| Weekly analysis (Claude Sonnet 4) | 52/year | $0.30 | $15.60 |
189
-| Reskill (GPT-4.1) | ~10/year | $0.036 | $0.36 |
201
+| Reskill (copilot-default / Claude Sonnet rates) | ~10/year | $0.058 | $0.58 |
202
| GitHub Actions compute | 52/year | $0.00 (public) | $0.00 |
191
-| **Total annual AI cost** | | | **$15.96** |
203
+| **Total annual AI cost** | | | **$16.18** |
204
205
### Cost per Published Page
206
207
```
196
-Annual AI cost / 52 pages = $15.96 / 52 = $0.307 per page
208
+Annual AI cost / 52 pages = $16.18 / 52 = $0.311 per page
209
```
210
211
Including amortized reskill:
212
```
201
-($15.60 + $0.36) / 52 = $0.307 per page (reskill is negligible)
213
+($15.60 + $0.58) / 52 = $0.311 per page (reskill is small)
214
```
215
216
### Comparative Analysis
217
218
| Approach | Annual Cost | Cost per Page | Quality |
219
|----------|-------------|---------------|---------|
208
-| **SquadScope (automated)** | **~$16/year** | **$0.31** | Consistent, opinionated, improving |
220
+| **SquadScope (automated)** | **~$16/year** | **$0.311** | Consistent, opinionated, improving |
221
| Human analyst (freelance) | $5,200–$10,400/year | $100–$200 | High but variable |
222
| Newsletter service (Substack Pro) | $600/year | $11.50 | Platform cost only, still need writer |
223
| Manual GPT-4 chat (copy-paste) | ~$50/year | ~$1 | Inconsistent, no learning loop |
@@ -220,11 +232,11 @@ Including amortized reskill:
232
233
| Task | Current Model | Optimized Model | Savings |
234
|------|---------------|-----------------|---------|
223
-| Weekly analysis | Claude Sonnet 4 ($0.30) | GPT-5 mini ($0.02 input + $0.004 output) | **92%** |
224
-| Reskill | GPT-4.1 ($0.036) | Keep (already cheap, quality matters) | 0% |
225
-| Fallback analysis | GPT-4.1 ($0.20) | GPT-5 mini ($0.024) | **88%** |
235
+| Weekly analysis | Claude Sonnet 4 ($0.30) | GPT-5 mini ($0.023 input + $0.004 output) | **91%** |
236
+| Reskill | copilot-default / Claude Sonnet rates ($0.058) | Keep Copilot-only; no GitHub Models fallback | 0% |
237
+| Copilot failure diagnosis | No AI ($0.00) | Fail closed or produce publish-ineligible diagnostic artifact | N/A |
238
227
-**Recommendation:** Start with Claude Sonnet 4 for quality. If quality_score consistently ≥ 75, experiment with GPT-4.1 or Claude Haiku 4.5 for weekly analysis. Reserve premium models for reskill where judgment quality matters most.
239
+**Recommendation:** Start with Claude Sonnet 4 for quality. If quality_score consistently ≥ 75, experiment with GPT-5.4 mini, GPT-5 mini, or Claude Haiku 4.5 for weekly analysis. Reserve premium models for reskill where judgment quality matters most.
240
241
### Strategy 2: Context Window Management
242
@@ -288,7 +300,7 @@ Savings: $0.232 per run = 77% reduction on the JSON portion
300
### Per-Run Token Tracking
301
302
1. **Copilot CLI transcript:** The `--share=PATH` flag exports a session transcript. Parse it post-run to extract actual token counts.
291
-2. **GitHub Models API response headers (assumption to validate):** The API is expected to return `x-ratelimit-remaining` headers and usage metadata in response JSON. Current scripts only parse the JSON body and discard headers; implementation will need to explicitly capture response headers and extract usage fields. This assumption requires validation against live API responses.
303
+2. **API-compatible response metadata (general tooling only):** The cost ledger can parse OpenAI-compatible `usage` JSON for non-analysis experiments, but weekly analysis remains Copilot-only and does not use GitHub Models/OpenAI fallback.
304
3. **Workflow annotations:** Log token estimates and actuals as workflow summary annotations.
305
306
### Implementation
@@ -340,19 +352,19 @@ Normal Mode (cost < $0.50/run)
352
└─ Full context (raw JSON + wisdom + skills + prior week)
353
354
Budget Mode (cost would exceed $0.50/run)
343
- └─ Switch to GPT-4.1 (saves ~33%)
355
+ └─ Switch to GPT-5.4 mini (saves ~74%)
356
└─ Truncate raw JSON to top 100 repos
357
└─ Omit skills context
358
359
Minimal Mode (monthly budget exhausted)
348
- └─ Switch to GPT-5 mini (saves ~92%)
360
+ └─ Switch to GPT-5 mini (saves ~91%)
361
└─ Truncate raw JSON to top 30 repos
362
└─ Omit all optional context
363
└─ Quality gate threshold lowered to 50
364
365
Emergency Mode (all credits exhausted)
366
└─ Skip AI analysis entirely
355
- └─ Publish raw data summary (stats only, no editorial)
367
+ └─ Produce diagnostic/staged no-AI artifact only; publish-ineligible
368
└─ Open issue for manual intervention
369
```
370
@@ -361,14 +373,16 @@ Emergency Mode (all credits exhausted)
373
Before invoking the model, estimate cost:
374
375
```python
364
-def estimate_cost(input_tokens: int, output_estimate: int, model: str) -> float:
365
- rates = {
366
- "claude-sonnet-4": {"input": 3.00, "output": 15.00},
367
- "openai/gpt-4.1": {"input": 2.00, "output": 8.00},
368
- "openai/gpt-5-mini": {"input": 0.25, "output": 2.00},
369
- }
370
- r = rates[model]
371
- return (input_tokens * r["input"] + output_estimate * r["output"]) / 1_000_000
376
+from scripts.model_pricing import estimate_cost_usd
377
+
378
+
379
+def estimate_cost(input_tokens: int, output_estimate: int, model: str) -> float | None:
380
+ # Centralized implementation lives in scripts/model_pricing.py.
381
+ return estimate_cost_usd(
382
+ model,
383
+ input_tokens=input_tokens,
384
+ output_tokens=output_estimate,
385
+ )
386
```
387
388
---
@@ -407,7 +421,7 @@ def estimate_cost(input_tokens: int, output_estimate: int, model: str) -> float:
421
| OQ3 | Is prompt caching available for Copilot CLI in non-interactive mode? | Medium — could save 77% on JSON input | Monitor GitHub changelog |
422
| OQ4 | What's the actual token count for the raw JSON? (estimated 86K, need actuals) | Medium — calibration | Add tokenizer count in pre-flight step |
423
| OQ5 | How does GitHub bill for the Copilot CLI invocation itself vs. the underlying model tokens? | High — may have additional overhead | Review billing after first month |
410
-| OQ6 | Are GitHub Models API calls billed differently from Copilot CLI calls against the same model? | Medium — affects fallback cost comparison | Compare billing line items |
424
+| OQ6 | Can Copilot CLI expose more detailed actual token usage for cache/cached-input accounting? | Medium — affects estimate precision | Review Copilot CLI release notes and billing exports |
425
| OQ7 | What happens when the Copilot Pro credit allowance is consumed mid-month? | High — operational risk | Set up overage alerts, test degradation path |
426
427
---
scripts/check_copilot_pricing_review.py
new
+139
@@ -0,0 +1,139 @@
1
+#!/usr/bin/env python3
2
+"""Check whether the Copilot model pricing table is due for manual review."""
3
+from __future__ import annotations
4
+
5
+import argparse
6
+import json
7
+from datetime import UTC, date, datetime
8
+from pathlib import Path
9
+
10
+from scripts.model_pricing import (
11
+ MODEL_PRICING,
12
+ PRICING_FETCHED_DATE,
13
+ PRICING_REVIEW_INTERVAL_MONTHS,
14
+ PRICING_SOURCE_URL,
15
+ TieredModelRate,
16
+)
17
+
18
+
19
+def parse_source_headers(path: Path | None) -> dict[str, str]:
20
+ if path is None or not path.exists():
21
+ return {}
22
+ metadata: dict[str, str] = {}
23
+ for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
24
+ if ":" not in line:
25
+ continue
26
+ name, value = line.split(":", 1)
27
+ normalized = name.strip().lower()
28
+ if normalized in {"etag", "last-modified"}:
29
+ metadata[normalized] = value.strip()
30
+ return metadata
31
+
32
+
33
+def parse_date(value: str) -> date:
34
+ candidate = value.strip()
35
+ if candidate.endswith("Z"):
36
+ candidate = f"{candidate[:-1]}+00:00"
37
+ if "T" in candidate:
38
+ return datetime.fromisoformat(candidate).date()
39
+ return date.fromisoformat(candidate)
40
+
41
+
42
+def add_months(value: date, months: int) -> date:
43
+ month_index = value.month - 1 + months
44
+ year = value.year + month_index // 12
45
+ month = month_index % 12 + 1
46
+ month_lengths = [31, 29 if year % 4 == 0 and (year % 100 != 0 or year % 400 == 0) else 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
47
+ return date(year, month, min(value.day, month_lengths[month - 1]))
48
+
49
+
50
+def pricing_status(
51
+ current_date: date,
52
+ source_url: str = PRICING_SOURCE_URL,
53
+ source_headers: dict[str, str] | None = None,
54
+) -> dict[str, object]:
55
+ fetched_date = parse_date(PRICING_FETCHED_DATE)
56
+ due_date = add_months(fetched_date, PRICING_REVIEW_INTERVAL_MONTHS)
57
+ source_url_matches = source_url == PRICING_SOURCE_URL
58
+ due = current_date >= due_date
59
+ tiered_models = sorted(model for model, pricing in MODEL_PRICING.items() if isinstance(pricing, TieredModelRate))
60
+ return {
61
+ "needs_review": due or not source_url_matches,
62
+ "review_due": due,
63
+ "source_url_matches": source_url_matches,
64
+ "source_url": PRICING_SOURCE_URL,
65
+ "requested_source_url": source_url,
66
+ "fetched_date": PRICING_FETCHED_DATE,
67
+ "review_interval_months": PRICING_REVIEW_INTERVAL_MONTHS,
68
+ "due_date": due_date.isoformat(),
69
+ "current_date": current_date.isoformat(),
70
+ "model_count": len(MODEL_PRICING),
71
+ "tiered_models": tiered_models,
72
+ "source_headers": source_headers or {},
73
+ }
74
+
75
+
76
+def render_report(status: dict[str, object]) -> str:
77
+ result = "required" if status["needs_review"] else "not due"
78
+ return "\n".join(
79
+ [
80
+ "# Copilot model pricing review",
81
+ "",
82
+ f"**Status:** Review {result}.",
83
+ f"**Source:** {status['source_url']}",
84
+ f"**Repository pricing fetched:** {status['fetched_date']}",
85
+ f"**Review interval:** every {status['review_interval_months']} months",
86
+ f"**Next/due review date:** {status['due_date']}",
87
+ f"**Workflow check date:** {status['current_date']}",
88
+ f"**Tracked pricing entries:** {status['model_count']}",
89
+ f"**Long-context pricing entries:** {', '.join(status['tiered_models'])}",
90
+ f"**Observed source metadata:** {json.dumps(status['source_headers'], sort_keys=True) if status['source_headers'] else 'not captured'}",
91
+ "",
92
+ "This workflow does not change pricing automatically. Please compare the repository pricing table against the GitHub docs, update code/docs/tests if needed, and open a PR.",
93
+ "",
94
+ "Checklist:",
95
+ "- Review `scripts/model_pricing.py` against the source URL.",
96
+ "- Update cost documentation and tests if rates, model names, or thresholds changed.",
97
+ "- Keep the source URL and fetched date in sync with the reviewed table.",
98
+ ]
99
+ ) + "\n"
100
+
101
+
102
+def write_github_output(path: Path, status: dict[str, object]) -> None:
103
+ with path.open("a", encoding="utf-8") as handle:
104
+ handle.write(f"needs_review={str(status['needs_review']).lower()}\n")
105
+ handle.write(f"due_date={status['due_date']}\n")
106
+ handle.write(f"fetched_date={status['fetched_date']}\n")
107
+
108
+
109
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
110
+ parser = argparse.ArgumentParser(description="Check whether Copilot model pricing needs manual review.")
111
+ parser.add_argument("--current-date", default=datetime.now(UTC).date().isoformat(), help="Current UTC date.")
112
+ parser.add_argument("--source-url", default=PRICING_SOURCE_URL, help="Expected GitHub Copilot pricing source URL.")
113
+ parser.add_argument("--output", type=Path, help="Write a Markdown review report to this path.")
114
+ parser.add_argument("--json-output", type=Path, help="Write machine-readable status JSON to this path.")
115
+ parser.add_argument("--github-output", type=Path, help="Append step outputs for GitHub Actions.")
116
+ parser.add_argument("--source-headers", type=Path, help="Optional HTTP response headers captured from the source URL.")
117
+ return parser.parse_args(argv)
118
+
119
+
120
+def main(argv: list[str] | None = None) -> int:
121
+ args = parse_args(argv)
122
+ status = pricing_status(parse_date(args.current_date), args.source_url, parse_source_headers(args.source_headers))
123
+ report = render_report(status)
124
+
125
+ if args.output:
126
+ args.output.write_text(report, encoding="utf-8")
127
+ else:
128
+ print(report, end="")
129
+
130
+ if args.json_output:
131
+ args.json_output.write_text(json.dumps(status, indent=2, sort_keys=True) + "\n", encoding="utf-8")
132
+ if args.github_output:
133
+ write_github_output(args.github_output, status)
134
+
135
+ return 0
136
+
137
+
138
+if __name__ == "__main__":
139
+ raise SystemExit(main())
scripts/model_pricing.py
new
+150
@@ -0,0 +1,150 @@
1
+"""GitHub Copilot model pricing helpers.
2
+
3
+Source: https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing
4
+Fetched: 2026-06-06. Prices are USD per 1M tokens and must be reviewed every two months.
5
+"""
6
+from __future__ import annotations
7
+
8
+from dataclasses import dataclass
9
+
10
+PRICING_SOURCE_URL = "https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing"
11
+PRICING_FETCHED_DATE = "2026-06-06"
12
+PRICING_REVIEW_INTERVAL_MONTHS = 2
13
+
14
+
15
+@dataclass(frozen=True)
16
+class ModelRate:
17
+ input: float
18
+ cached_input: float
19
+ output: float
20
+ cache_write: float | None = None
21
+
22
+ def as_dict(self) -> dict[str, float]:
23
+ values = {
24
+ "input": self.input,
25
+ "cached_input": self.cached_input,
26
+ "output": self.output,
27
+ }
28
+ if self.cache_write is not None:
29
+ values["cache_write"] = self.cache_write
30
+ return values
31
+
32
+
33
+@dataclass(frozen=True)
34
+class TieredModelRate:
35
+ default: ModelRate
36
+ long_context: ModelRate
37
+ long_context_threshold: int
38
+
39
+ def rate_for(self, input_tokens: int) -> ModelRate:
40
+ return self.long_context if input_tokens > self.long_context_threshold else self.default
41
+
42
+ def as_dict(self) -> dict[str, float | int]:
43
+ values: dict[str, float | int] = {
44
+ "input": self.default.input,
45
+ "cached_input": self.default.cached_input,
46
+ "output": self.default.output,
47
+ }
48
+ if self.default.cache_write is not None:
49
+ values["cache_write"] = self.default.cache_write
50
+ values["long_context_threshold"] = self.long_context_threshold
51
+ values["long_context_input"] = self.long_context.input
52
+ values["long_context_cached_input"] = self.long_context.cached_input
53
+ values["long_context_output"] = self.long_context.output
54
+ return values
55
+
56
+
57
+SONNET_RATE = ModelRate(input=3.00, cached_input=0.30, cache_write=3.75, output=15.00)
58
+OPUS_RATE = ModelRate(input=5.00, cached_input=0.50, cache_write=6.25, output=25.00)
59
+
60
+MODEL_PRICING: dict[str, ModelRate | TieredModelRate] = {
61
+ "copilot-default": SONNET_RATE,
62
+ "gpt-5-mini": ModelRate(input=0.25, cached_input=0.025, output=2.00),
63
+ "openai/gpt-5-mini": ModelRate(input=0.25, cached_input=0.025, output=2.00),
64
+ "gpt-5.3-codex": ModelRate(input=1.75, cached_input=0.175, output=14.00),
65
+ "openai/gpt-5.3-codex": ModelRate(input=1.75, cached_input=0.175, output=14.00),
66
+ "gpt-5.4": TieredModelRate(
67
+ default=ModelRate(input=2.50, cached_input=0.25, output=15.00),
68
+ long_context=ModelRate(input=5.00, cached_input=0.50, output=22.50),
69
+ long_context_threshold=272_000,
70
+ ),
71
+ "openai/gpt-5.4": TieredModelRate(
72
+ default=ModelRate(input=2.50, cached_input=0.25, output=15.00),
73
+ long_context=ModelRate(input=5.00, cached_input=0.50, output=22.50),
74
+ long_context_threshold=272_000,
75
+ ),
76
+ "gpt-5.4-mini": ModelRate(input=0.75, cached_input=0.075, output=4.50),
77
+ "openai/gpt-5.4-mini": ModelRate(input=0.75, cached_input=0.075, output=4.50),
78
+ "gpt-5.4-nano": ModelRate(input=0.20, cached_input=0.02, output=1.25),
79
+ "openai/gpt-5.4-nano": ModelRate(input=0.20, cached_input=0.02, output=1.25),
80
+ "gpt-5.5": TieredModelRate(
81
+ default=ModelRate(input=5.00, cached_input=0.50, output=30.00),
82
+ long_context=ModelRate(input=10.00, cached_input=1.00, output=45.00),
83
+ long_context_threshold=272_000,
84
+ ),
85
+ "openai/gpt-5.5": TieredModelRate(
86
+ default=ModelRate(input=5.00, cached_input=0.50, output=30.00),
87
+ long_context=ModelRate(input=10.00, cached_input=1.00, output=45.00),
88
+ long_context_threshold=272_000,
89
+ ),
90
+ "claude-haiku-4.5": ModelRate(input=1.00, cached_input=0.10, cache_write=1.25, output=5.00),
91
+ "claude-sonnet-4": SONNET_RATE,
92
+ "claude-sonnet-4.5": SONNET_RATE,
93
+ "claude-sonnet-4.6": SONNET_RATE,
94
+ "claude-opus-4.5": OPUS_RATE,
95
+ "claude-opus-4.6": OPUS_RATE,
96
+ "claude-opus-4.7": OPUS_RATE,
97
+ "claude-opus-4.8": OPUS_RATE,
98
+ "gemini-2.5-pro": ModelRate(input=1.25, cached_input=0.125, output=10.00),
99
+ "google/gemini-2.5-pro": ModelRate(input=1.25, cached_input=0.125, output=10.00),
100
+ "gemini-3-flash": ModelRate(input=0.50, cached_input=0.05, output=3.00),
101
+ "google/gemini-3-flash": ModelRate(input=0.50, cached_input=0.05, output=3.00),
102
+ "gemini-3.1-pro": TieredModelRate(
103
+ default=ModelRate(input=2.00, cached_input=0.20, output=12.00),
104
+ long_context=ModelRate(input=4.00, cached_input=0.40, output=18.00),
105
+ long_context_threshold=200_000,
106
+ ),
107
+ "google/gemini-3.1-pro": TieredModelRate(
108
+ default=ModelRate(input=2.00, cached_input=0.20, output=12.00),
109
+ long_context=ModelRate(input=4.00, cached_input=0.40, output=18.00),
110
+ long_context_threshold=200_000,
111
+ ),
112
+ "gemini-3.5-flash": ModelRate(input=1.50, cached_input=0.15, output=9.00),
113
+ "google/gemini-3.5-flash": ModelRate(input=1.50, cached_input=0.15, output=9.00),
114
+ "raptor-mini": ModelRate(input=0.25, cached_input=0.025, output=2.00),
115
+ "github/raptor-mini": ModelRate(input=0.25, cached_input=0.025, output=2.00),
116
+ "mai-code-1-flash": ModelRate(input=0.75, cached_input=0.075, output=4.50),
117
+ "microsoft/mai-code-1-flash": ModelRate(input=0.75, cached_input=0.075, output=4.50),
118
+}
119
+
120
+MODEL_RATES = {model: pricing.as_dict() for model, pricing in MODEL_PRICING.items()}
121
+
122
+
123
+def get_model_rate(model: str, input_tokens: int) -> dict[str, float] | None:
124
+ pricing = MODEL_PRICING.get(model)
125
+ if pricing is None:
126
+ return None
127
+ if isinstance(pricing, TieredModelRate):
128
+ return pricing.rate_for(input_tokens).as_dict()
129
+ return pricing.as_dict()
130
+
131
+
132
+def estimate_cost_usd(
133
+ model: str,
134
+ input_tokens: int,
135
+ output_tokens: int,
136
+ cached_input_tokens: int = 0,
137
+ cache_write_tokens: int = 0,
138
+) -> float | None:
139
+ rates = get_model_rate(model, input_tokens)
140
+ if rates is None:
141
+ return None
142
+ if cache_write_tokens > 0 and "cache_write" not in rates:
143
+ return None
144
+ total = (
145
+ input_tokens * rates["input"]
146
+ + cached_input_tokens * rates["cached_input"]
147
+ + cache_write_tokens * rates.get("cache_write", 0)
148
+ + output_tokens * rates["output"]
149
+ ) / 1_000_000
150
+ return round(total, 6)
scripts/preflight_cost_check.py
+3
-1
@@ -10,9 +10,11 @@ import argparse
10
import sys
11
from pathlib import Path
12
13
-from scripts.track_token_usage import (
13
+from scripts.model_pricing import (
14
MODEL_RATES,
15
estimate_cost_usd,
16
+)
17
+from scripts.track_token_usage import (
18
estimate_tokens_from_path,
19
)
20
scripts/tier_selector.py
+1
-1
@@ -13,7 +13,7 @@ import sys
13
# Tier definitions
14
TIERS = {
15
"normal": {"model": "claude-sonnet-4", "max_repos": None, "skip_ai": False},
16
- "budget": {"model": "gpt-4.1", "max_repos": 100, "skip_ai": False},
16
+ "budget": {"model": "gpt-5.4-mini", "max_repos": 100, "skip_ai": False},
17
"minimal": {"model": "gpt-5-mini", "max_repos": 30, "skip_ai": False},
18
"emergency": {"model": None, "max_repos": None, "skip_ai": True},
19
}
scripts/track_token_usage.py
+2
-17
@@ -9,18 +9,11 @@ import sys
9
from datetime import UTC, datetime
10
from pathlib import Path
11
12
+from scripts.model_pricing import estimate_cost_usd
13
+
14
ROOT = Path(__file__).resolve().parent.parent
15
DEFAULT_USAGE_FILE = ROOT / "data" / "metrics" / "token-usage.jsonl"
16
CHARS_PER_TOKEN = 4
15
-MODEL_RATES = {
16
- "copilot-default": {"input": 3.00, "output": 15.00},
17
- "claude-sonnet-4": {"input": 3.00, "output": 15.00},
18
- "openai/gpt-4.1": {"input": 2.00, "output": 8.00},
19
- "gpt-4.1": {"input": 2.00, "output": 8.00},
20
- "openai/gpt-5-mini": {"input": 0.25, "output": 2.00},
21
- "gpt-5-mini": {"input": 0.25, "output": 2.00},
22
- "claude-haiku-4.5": {"input": 1.00, "output": 5.00},
23
-}
17
18
19
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
@@ -141,14 +134,6 @@ def parse_api_response(path: Path) -> tuple[int, int] | None:
134
return None
135
136
144
-def estimate_cost_usd(model: str, input_tokens: int, output_tokens: int) -> float | None:
145
- rates = MODEL_RATES.get(model)
146
- if not rates:
147
- return None
148
- total = (input_tokens * rates["input"] + output_tokens * rates["output"]) / 1_000_000
149
- return round(total, 6)
150
-
151
-
137
def build_record(args: argparse.Namespace) -> dict[str, object]:
138
parsed_datetime = parse_datetime(args.current_datetime).astimezone(UTC)
139
week = args.week or week_slug(parsed_datetime)
tests/test_copilot_pricing_review.py
new
+99
@@ -0,0 +1,99 @@
1
+import json
2
+import tempfile
3
+import unittest
4
+from datetime import date
5
+from pathlib import Path
6
+
7
+import yaml
8
+
9
+import scripts.check_copilot_pricing_review as pricing_review
10
+
11
+
12
+class CopilotPricingReviewTests(unittest.TestCase):
13
+ def test_review_not_due_before_two_month_interval(self) -> None:
14
+ status = pricing_review.pricing_status(date(2026, 8, 5))
15
+ self.assertFalse(status["needs_review"])
16
+ self.assertFalse(status["review_due"])
17
+ self.assertEqual(status["due_date"], "2026-08-06")
18
+
19
+ def test_review_due_at_two_month_interval(self) -> None:
20
+ status = pricing_review.pricing_status(date(2026, 8, 6))
21
+ self.assertTrue(status["needs_review"])
22
+ self.assertTrue(status["review_due"])
23
+
24
+ def test_source_url_mismatch_requires_review(self) -> None:
25
+ status = pricing_review.pricing_status(date(2026, 7, 1), source_url="https://example.invalid/pricing")
26
+ self.assertTrue(status["needs_review"])
27
+ self.assertFalse(status["source_url_matches"])
28
+
29
+ def test_source_headers_are_parsed_for_report_metadata(self) -> None:
30
+ tests_root = Path(__file__).resolve().parent
31
+ with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:
32
+ headers = Path(tmpdir) / "headers.txt"
33
+ headers.write_text('HTTP/2 200\netag: "abc123"\nlast-modified: Sat, 06 Jun 2026 00:00:00 GMT\n', encoding="utf-8")
34
+ status = pricing_review.pricing_status(date(2026, 7, 1), source_headers=pricing_review.parse_source_headers(headers))
35
+ report = pricing_review.render_report(status)
36
+ self.assertEqual(status["source_headers"]["etag"], '"abc123"')
37
+ self.assertIn("last-modified", report)
38
+
39
+ def test_main_writes_report_json_and_github_outputs(self) -> None:
40
+ tests_root = Path(__file__).resolve().parent
41
+ with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:
42
+ base = Path(tmpdir)
43
+ report_path = base / "report.md"
44
+ json_path = base / "status.json"
45
+ github_output = base / "github-output.txt"
46
+
47
+ rc = pricing_review.main(
48
+ [
49
+ "--current-date",
50
+ "2026-08-06",
51
+ "--output",
52
+ str(report_path),
53
+ "--json-output",
54
+ str(json_path),
55
+ "--github-output",
56
+ str(github_output),
57
+ ]
58
+ )
59
+
60
+ self.assertEqual(rc, 0)
61
+ self.assertIn("does not change pricing automatically", report_path.read_text(encoding="utf-8"))
62
+ status = json.loads(json_path.read_text(encoding="utf-8"))
63
+ self.assertTrue(status["needs_review"])
64
+ self.assertIn("needs_review=true", github_output.read_text(encoding="utf-8"))
65
+
66
+
67
+class CopilotPricingReviewWorkflowTests(unittest.TestCase):
68
+ def test_workflow_is_scheduled_and_opens_issue_without_changing_pricing(self) -> None:
69
+ workflow_path = Path(".github/workflows/copilot-pricing-review.yml")
70
+ workflow = yaml.safe_load(workflow_path.read_text(encoding="utf-8"))
71
+
72
+ self.assertEqual(workflow["name"], "Copilot Pricing Review")
73
+ trigger = workflow.get("on", workflow.get(True))
74
+ self.assertIsNotNone(trigger)
75
+ self.assertEqual(trigger["schedule"][0]["cron"], "23 9 6 2,4,6,8,10,12 *")
76
+ self.assertIn("workflow_dispatch", trigger)
77
+ self.assertEqual(workflow["permissions"], {"contents": "read", "issues": "write"})
78
+
79
+ job = workflow["jobs"]["review-pricing"]
80
+ pricing_step = next((step for step in job["steps"] if step.get("id") == "pricing"), None)
81
+ self.assertIsNotNone(pricing_step)
82
+ self.assertIn("scripts/check_copilot_pricing_review.py", pricing_step["run"])
83
+ self.assertIn("--source-headers", pricing_step["run"])
84
+ self.assertIn("--github-output", pricing_step["run"])
85
+ metadata_step = next((step for step in job["steps"] if step.get("name") == "Capture Copilot pricing source metadata"), None)
86
+ self.assertIsNotNone(metadata_step)
87
+ self.assertIn("curl -fsSLI", metadata_step["run"])
88
+
89
+ issue_step = next((step for step in job["steps"] if step.get("name") == "Create or update pricing review issue"), None)
90
+ self.assertIsNotNone(issue_step)
91
+ self.assertEqual(issue_step["if"], "steps.pricing.outputs.needs_review == 'true'")
92
+ self.assertIn("gh issue create", issue_step["run"])
93
+ self.assertIn("gh issue comment", issue_step["run"])
94
+ self.assertNotIn("git commit", issue_step["run"])
95
+ self.assertNotIn("git push", issue_step["run"])
96
+
97
+
98
+if __name__ == "__main__":
99
+ unittest.main()
tests/test_tier_selector.py
+1
-1
@@ -41,7 +41,7 @@ class TestBuildConfig:
41
42
def test_budget_config(self):
43
cfg = build_config("budget")
44
- assert cfg == {"tier": "budget", "model": "gpt-4.1", "max_repos": 100, "skip_ai": False}
44
+ assert cfg == {"tier": "budget", "model": "gpt-5.4-mini", "max_repos": 100, "skip_ai": False}
45
46
def test_minimal_config(self):
47
cfg = build_config("minimal")
tests/test_track_token_usage.py
+45
-3
@@ -65,7 +65,7 @@ class TrackTokenUsageTests(unittest.TestCase):
65
"--source",
66
"github-models",
67
"--model",
68
- "openai/gpt-4.1",
68
+ "gpt-5.4-mini",
69
"--current-datetime",
70
"2026-05-19T08:00:00Z",
71
"--input-tokens",
@@ -83,7 +83,7 @@ class TrackTokenUsageTests(unittest.TestCase):
83
self.assertEqual(record["output_tokens"], 250)
84
self.assertEqual(record["total_tokens"], 1250)
85
self.assertEqual(record["week"], "2026-W21")
86
- self.assertEqual(record["cost_usd"], 0.004)
86
+ self.assertEqual(record["cost_usd"], 0.001875)
87
self.assertFalse(record["estimated"])
88
89
def test_input_manifest_validation_fails_when_final_usage_differs_by_more_than_10_percent(self) -> None:
@@ -368,7 +368,7 @@ class TokenSourcePriorityTests(unittest.TestCase):
368
[
369
"--stage", "reskill",
370
"--source", "github-models",
371
- "--model", "gpt-4.1",
371
+ "--model", "gpt-5.4-mini",
372
"--current-datetime", "2026-05-19T08:00:00Z",
373
"--prompt-file", str(prompt_path),
374
"--api-response", str(api_response),
@@ -441,5 +441,47 @@ class TokenSourcePriorityTests(unittest.TestCase):
441
self.assertTrue(record["estimated"])
442
443
444
+class ModelPricingTests(unittest.TestCase):
445
+ def test_prices_representative_current_models(self) -> None:
446
+ self.assertEqual(track_token_usage.estimate_cost_usd("gpt-5-mini", 1_000_000, 1_000_000), 2.25)
447
+ self.assertEqual(track_token_usage.estimate_cost_usd("claude-haiku-4.5", 1_000_000, 1_000_000), 6.0)
448
+ self.assertEqual(track_token_usage.estimate_cost_usd("gemini-3-flash", 1_000_000, 1_000_000), 3.5)
449
+ self.assertEqual(track_token_usage.estimate_cost_usd("mai-code-1-flash", 1_000_000, 1_000_000), 5.25)
450
+
451
+ def test_long_context_threshold_rates_apply(self) -> None:
452
+ self.assertEqual(track_token_usage.estimate_cost_usd("gpt-5.4", 272_000, 1_000), 0.695)
453
+ self.assertEqual(track_token_usage.estimate_cost_usd("gpt-5.4", 272_001, 1_000), 1.382505)
454
+ self.assertEqual(track_token_usage.estimate_cost_usd("gemini-3.1-pro", 200_001, 1_000), 0.818004)
455
+
456
+ def test_cached_and_cache_write_tokens_are_supported(self) -> None:
457
+ cost = track_token_usage.estimate_cost_usd(
458
+ "claude-sonnet-4.6",
459
+ input_tokens=1_000_000,
460
+ output_tokens=1_000_000,
461
+ cached_input_tokens=1_000_000,
462
+ cache_write_tokens=1_000_000,
463
+ )
464
+ self.assertEqual(cost, 22.05)
465
+
466
+ def test_cached_tokens_do_not_trigger_long_context_rates(self) -> None:
467
+ cost = track_token_usage.estimate_cost_usd(
468
+ "gpt-5.4",
469
+ input_tokens=272_000,
470
+ output_tokens=1_000,
471
+ cached_input_tokens=1,
472
+ )
473
+ self.assertEqual(cost, 0.695)
474
+
475
+ def test_unsupported_cache_write_tokens_return_unknown(self) -> None:
476
+ self.assertIsNone(
477
+ track_token_usage.estimate_cost_usd(
478
+ "gpt-5-mini",
479
+ input_tokens=1_000_000,
480
+ output_tokens=1_000_000,
481
+ cache_write_tokens=1,
482
+ )
483
+ )
484
+
485
+
486
if __name__ == "__main__":
487
unittest.main()