fix: stop pinning Copilot workflow models (#145)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
May 20, 2026 at 20:16 UTC
e57f3ea51cedc04c307951f99ed18bc3489f32bd
8 files changed
+34
-16
.github/workflows/crawl-and-publish.yml
+7
-7
@@ -23,6 +23,7 @@ concurrency:
23
24
env:
25
PYTHONPATH: ${{ github.workspace }}
26
+ GITHUB_MODELS_MODEL: ${{ vars.GITHUB_MODELS_MODEL || 'openai/gpt-4o' }}
27
28
jobs:
29
crawl:
@@ -267,9 +268,9 @@ jobs:
268
for f in .squad/skills/*.md; do
269
[ -f "$f" ] && CONTEXT_FILES+=("$f")
270
done
271
+ # Use the generic Copilot default cost profile so the workflow never pins a stale CLI model.
272
python3 scripts/preflight_cost_check.py \
271
- --context-files "${CONTEXT_FILES[@]}" \
272
- --model claude-sonnet-4
273
+ --context-files "${CONTEXT_FILES[@]}"
274
275
- name: Install Copilot CLI
276
id: install-copilot
@@ -299,12 +300,12 @@ jobs:
300
cat "$PRESS_FILE" >> "$PROMPT_FILE"
301
fi
302
303
+ # Intentionally rely on Copilot CLI's default model so CI follows the platform-supported default.
304
if command -v copilot >/dev/null 2>&1 && copilot \
305
--agent squad \
306
-p "Farnsworth, read the file at ${PROMPT_FILE} — it contains the weekly data and analysis instructions. Follow them exactly and write the analysis to ${OUTPUT_FILE}." \
307
-s \
308
--no-ask-user \
307
- --model claude-sonnet-4 \
309
--allow-tool=read \
310
--allow-tool=write \
311
--allow-tool=glob \
@@ -312,7 +313,7 @@ jobs:
313
--share=data/metrics/copilot-transcript.md \
314
> "$OUTPUT_FILE"; then
315
ANALYSIS_SOURCE="copilot-cli"
315
- ANALYSIS_MODEL="claude-sonnet-4"
316
+ ANALYSIS_MODEL="copilot-default"
317
elif python3 scripts/analyze_fallback.py \
318
--raw-json "$WEEK_FILE" \
319
--output "$OUTPUT_FILE" \
@@ -725,7 +726,6 @@ jobs:
726
COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GH_TOKEN }}
727
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
728
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
728
- GITHUB_MODELS_MODEL: openai/gpt-4o
729
run: |
730
set -euo pipefail
731
git config user.name "github-actions[bot]"
@@ -739,15 +739,15 @@ jobs:
739
740
# Primary path: Copilot CLI with agent identity
741
RESKILL_SOURCE="copilot-cli"
742
- RESKILL_MODEL="claude-sonnet-4"
742
+ RESKILL_MODEL="copilot-default"
743
python3 scripts/reskill.py --current-datetime "$CURRENT_DATETIME" --output "$RESKILL_OUTPUT" --prompt-output "$RESKILL_PROMPT" --print-prompt > "$RESKILL_PROMPT" || true
744
745
+ # Intentionally rely on Copilot CLI's default model so CI follows the platform-supported default.
746
if command -v copilot >/dev/null 2>&1 && copilot \
747
--agent squad \
748
-p "Farnsworth, read the file at ${RESKILL_PROMPT} — it contains the reskill retrospective instructions. Follow them exactly and write the report to ${RESKILL_OUTPUT}." \
749
-s \
750
--no-ask-user \
750
- --model claude-sonnet-4 \
751
--allow-tool=read \
752
--allow-tool=write \
753
--allow-tool=glob \
.squad/agents/farnsworth/history.md
+1
@@ -31,3 +31,4 @@
31
- **2026-05-19T22:52:54+02:00:** Fixed two reader-mode polish issues and reskill 403 crash (PR #139). (1) Count header `N repos have press correlation:` stripped in reader_mode via `re.sub` — it was an AI-prompt artefact leaking into the published page. (2) `_extract_readme_description()` now trims every candidate line to the last sentence boundary (`.` `!` `?` followed by space or end); lines with no boundary are skipped entirely, so truncated snippets never produce half-sentences. The 150-char upper bound was removed — sentence trimming makes it redundant. (3) `reskill.py main()` now catches `RuntimeError` from `call_github_models()` and writes a placeholder report instead of crashing — the job exits 0 even when the configured model (`openai/gpt-4.1`) returns 403. Key lesson: template-rendered content always needs an explicit pass to strip AI-only fields when switching to reader mode — simply replacing the list with narrative paragraphs is not enough if the surrounding template text still contains prompt tokens. 519 tests pass.
32
- **2026-05-19T20:57:55Z:** PR #139 merged. Correlation Summary section now renders as narrative prose in reader_mode (Decision recorded in `.squad/decisions.md`). Groups correlations by organization, ranks by aggregate confidence, fetches README snippets for top 2 repos per group (max 6 total), produces 1–3 interpretive paragraphs with inline links. Graceful failure on README timeout (5s). Key rationale: raw repo names with scores communicate data but not meaning; narrative format helps readers understand organizational impact. AI-mode output unchanged. Additionally: reskill job now catches RuntimeError on model 403, emits placeholder report, exits cleanly. Related decisions: `_format_correlations_narrative()` function pattern, README fetch timeout strategy, segment-by-org grouping logic. All new code covered by tests; 513 total pass.
33
- **2026-05-20T19:15:53.942+02:00:** Article structure restructured from repo-listing format to Gartner/McKinsey-style trend brief (requested by jmservera). New section order: `## This Week's Trends` → `## Where Industry Meets Code` → `## Signal & Noise` → `## Blind Spots` → `## The Week Ahead` → `## Key References` (with `### Notable Projects` and `### Press & Industry`). Key design decisions: (1) Lead with named macro trends — 3-5 synthesized themes, not repo lists. (2) "Where Industry Meets Code" replaces "Industry & Press Correlation" — editorial framing, not ML-sounding. (3) Signal & Noise merged into integrated prose section (no `### Signal`/`### Noise` sub-headings) for better editorial writing. (4) Repos move to `### Notable Projects` at end as references, freeing body prose to cite them as evidence rather than leading with them. (5) `generate_rollups.py` updated with backward compat — tries new heading names first, falls back to old for existing `2026-W21-summary.md`. Files changed: `prompts/analyze-weekly.md`, `docs/analysis-spec.md`, `scripts/analysis_gate.py`, `scripts/analyze_fallback.py`, `scripts/generate_rollups.py`, 5 test files. All 519 tests pass. Decision filed at `.squad/decisions/inbox/farnsworth-article-restructure.md`.
34
+- **2026-05-20T20:09:26+02:00:** Removed stale `claude-sonnet-4` pins from `crawl-and-publish.yml` so analysis and reskill now rely on the Copilot CLI default model instead of hardcoded model IDs. Added a `copilot-default` cost profile for preflight/usage tracking and made `GITHUB_MODELS_MODEL` workflow-wide and configurable via repo vars with `openai/gpt-4o` as the fallback default. Lesson: for platform-managed Copilot CLI runs, pinning a versioned model name is operational debt; keep the CLI on its moving default and only pin fallback API models you explicitly control.
.squad/decisions/inbox/farnsworth-model-resilience.md
new
+12
@@ -0,0 +1,12 @@
1
+# 2026-05-20: Model resilience for weekly CI
2
+
3
+- **Owner:** Farnsworth
4
+- **Date:** 2026-05-20T20:09:26+02:00
5
+- **Status:** Proposed
6
+- **Decision:** The `crawl-and-publish.yml` workflow should never pass a version-pinned `--model` flag to GitHub Copilot CLI. Analysis and reskill should rely on the CLI's platform default model, while GitHub Models fallback remains configurable through `GITHUB_MODELS_MODEL` with `openai/gpt-4o` as the default.
7
+- **Why:** Pinned Copilot CLI model IDs can disappear from the platform and silently degrade the pipeline into fallback or no-AI paths. Letting the CLI choose its current default keeps the primary path available without manual model churn, while the fallback path still has an explicit, overridable model.
8
+- **Implementation notes:**
9
+ - Removed `--model claude-sonnet-4` from Copilot CLI invocations in analysis and reskill.
10
+ - Removed the workflow's pinned preflight model argument and switched cost estimation/usage tracking to a generic `copilot-default` rate profile.
11
+ - Promoted `GITHUB_MODELS_MODEL` to workflow-level env configuration using `${{ vars.GITHUB_MODELS_MODEL || 'openai/gpt-4o' }}`.
12
+- **Scope:** `.github/workflows/crawl-and-publish.yml`, `scripts/preflight_cost_check.py`, `scripts/track_token_usage.py`
scripts/preflight_cost_check.py
+3
-2
@@ -17,6 +17,7 @@ from scripts.track_token_usage import (
17
)
18
19
DEFAULT_OUTPUT_TOKENS = 2000
20
+DEFAULT_MODEL = "copilot-default"
21
HARD_CAP_USD = 1.00
22
23
@@ -33,8 +34,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
34
)
35
parser.add_argument(
36
"--model",
36
- default="claude-sonnet-4",
37
- help="Model name for cost rate lookup.",
37
+ default=DEFAULT_MODEL,
38
+ help="Model or rate profile name for cost lookup (default: copilot-default).",
39
)
40
parser.add_argument(
41
"--output-tokens",
scripts/track_token_usage.py
+1
@@ -12,6 +12,7 @@ ROOT = Path(__file__).resolve().parent.parent
12
DEFAULT_USAGE_FILE = ROOT / "data" / "metrics" / "token-usage.jsonl"
13
CHARS_PER_TOKEN = 4
14
MODEL_RATES = {
15
+ "copilot-default": {"input": 3.00, "output": 15.00},
16
"claude-sonnet-4": {"input": 3.00, "output": 15.00},
17
"openai/gpt-4.1": {"input": 2.00, "output": 8.00},
18
"gpt-4.1": {"input": 2.00, "output": 8.00},
tests/test_pipeline.py
+5
@@ -198,11 +198,14 @@ class WorkflowConfigTests(unittest.TestCase):
198
199
reskill_step = next((s for s in reskill["steps"] if s.get("name") == "Run reskill"), None)
200
self.assertIsNotNone(reskill_step)
201
+ self.assertEqual(workflow["env"]["GITHUB_MODELS_MODEL"], "${{ vars.GITHUB_MODELS_MODEL || 'openai/gpt-4o' }}")
202
self.assertEqual(reskill_step["env"]["COPILOT_GITHUB_TOKEN"], "${{ secrets.COPILOT_GH_TOKEN }}")
203
reskill_run = reskill_step["run"]
204
self.assertIn("python3 scripts/reskill.py --current-datetime", reskill_run)
205
self.assertIn("--prompt-output", reskill_run)
206
self.assertIn("python3 scripts/track_token_usage.py", reskill_run)
207
+ self.assertIn('RESKILL_MODEL="copilot-default"', reskill_run)
208
+ self.assertNotIn("--model claude-sonnet-4", reskill_run)
209
self.assertIn("mkdir -p .squad/skills .squad/reskill", reskill_run)
210
self.assertIn("data/metrics", reskill_run)
211
self.assertIn("trigger-log.txt", reskill_run)
@@ -214,6 +217,8 @@ class WorkflowConfigTests(unittest.TestCase):
217
self.assertIsNotNone(run_analysis_step)
218
run_analysis = run_analysis_step["run"]
219
self.assertIn("python3 scripts/track_token_usage.py", run_analysis)
220
+ self.assertIn('ANALYSIS_MODEL="copilot-default"', run_analysis)
221
+ self.assertNotIn("--model claude-sonnet-4", run_analysis)
222
self.assertIn("mkdir -p data/metrics", run_analysis)
223
224
def test_generate_workflow_runs_rollups_and_commits_all_content(self) -> None:
tests/test_preflight_cost_check.py
+3
-5
@@ -16,7 +16,7 @@ class PreflightCostCheckTests(unittest.TestCase):
16
with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:
17
base = Path(tmpdir)
18
f1 = self._make_file(base, "small.json", 400)
19
- rc = preflight.main(["--context-files", str(f1), "--model", "claude-sonnet-4"])
19
+ rc = preflight.main(["--context-files", str(f1)])
20
self.assertEqual(rc, 0)
21
22
def test_fails_over_cap(self) -> None:
@@ -25,7 +25,7 @@ class PreflightCostCheckTests(unittest.TestCase):
25
base = Path(tmpdir)
26
# 2M chars → 500k tokens; at $3/M input that's $1.50 — over cap
27
f1 = self._make_file(base, "huge.json", 2_000_000)
28
- rc = preflight.main(["--context-files", str(f1), "--model", "claude-sonnet-4"])
28
+ rc = preflight.main(["--context-files", str(f1)])
29
self.assertEqual(rc, 1)
30
31
def test_custom_cap(self) -> None:
@@ -34,9 +34,7 @@ class PreflightCostCheckTests(unittest.TestCase):
34
base = Path(tmpdir)
35
f1 = self._make_file(base, "medium.json", 40_000)
36
# 40k chars → 10k tokens; at $3/M input = $0.03; cap $0.01 should fail
37
- rc = preflight.main(
38
- ["--context-files", str(f1), "--model", "claude-sonnet-4", "--hard-cap", "0.01"]
39
- )
37
+ rc = preflight.main(["--context-files", str(f1), "--hard-cap", "0.01"])
38
self.assertEqual(rc, 1)
39
40
def test_unknown_model_fails(self) -> None:
tests/test_track_token_usage.py
+2
-2
@@ -24,7 +24,7 @@ class TrackTokenUsageTests(unittest.TestCase):
24
"--source",
25
"copilot-cli",
26
"--model",
27
- "claude-sonnet-4",
27
+ "copilot-default",
28
"--current-datetime",
29
"2026-05-19T08:00:00Z",
30
"--week",
@@ -44,7 +44,7 @@ class TrackTokenUsageTests(unittest.TestCase):
44
record = records[0]
45
self.assertEqual(record["stage"], "analysis")
46
self.assertEqual(record["source"], "copilot-cli")
47
- self.assertEqual(record["model"], "claude-sonnet-4")
47
+ self.assertEqual(record["model"], "copilot-default")
48
self.assertEqual(record["week"], "2026-W21")
49
self.assertEqual(record["input_tokens"], 10)
50
self.assertEqual(record["output_tokens"], 5)