@cryptotaxi247 / SquadScope / commits / dacc155

PRD: Topic-specific news channels for SquadScope (#39)

* docs: PRD for topic-specific news channels Defines how to generalize SquadScope into a topic-channel system where each deployment is configured for a specific domain (ai-ml, rust, security, etc.) with per-topic learning, predictions, and RSS. Key decisions: - Feature first, not separate platform - Multi-instance single-topic for v1 - Per-topic learning state isolation - Scoring pipeline (not just keyword filters) - Prediction ledger with hindsight validation - Topics as channels with dedicated RSS feeds Addresses rubber-duck review findings and learning audit gaps (G7, G8, G9, G13). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * squad: update Leela history and decision inbox for topic channels Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PRD topic channel review comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add learning audit reference doc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed May 18, 2026 at 15:23 UTC dacc155d7eae0735b8dc8a209b4e80604d60475c
4 files changed +956
.squad/agents/leela/history.md
+16
@@ -72,6 +72,22 @@
72 - **Team notification:** All agents notified that Phase 0 is complete and architecture is published.
73 - **Next phase:** Phase 1 (crawlers and generators) can proceed independently. Phase 2 (analyzer) is unblocked.
74
75 +### 2026-05-18T13:20:07.067+02:00 — Topic Channels PRD
76 +
77 +- **Deliverable:** `docs/PRD-topic-channels.md` — feature PRD for topic-specific news channels
78 +- **PR:** #39 (squad/topic-channels-prd → main)
79 +- **Key decisions:**
80 + - Feature first, not separate platform — extends existing pipeline with topic namespace
81 + - v1 = single configurable topic per instance (fork per topic); v2 = multi-topic deferred
82 + - Per-topic learning isolation (wisdom, skills, predictions, scorecards)
83 + - New scoring pipeline between crawl and analyze (relevance score 0-100)
84 + - Prediction ledger (`predictions.jsonl`) with hindsight validation at week N+4
85 + - `squadscope.topic.yml` as the single config file controlling all topic behavior
86 + - Two example configs shipped: ai-ml and rust
87 +- **Rubber-duck findings addressed:** All 7 findings incorporated (namespacing, multi-instance, learning isolation, scoring pipeline, prediction ledger, channel structure, quality criteria)
88 +- **Learning audit gaps addressed:** G7 (prompt feedback), G8 (hindsight validation), G9 (prediction registry), G13 (enrichment signals as OQ5)
89 +- **Implementation plan:** 15 issues with dependency graph, ~7-9 sessions estimated
90 +
91 ### 2026-05-18T10:59:10.800+02:00 — Phase 1 PR Review Gate
92
93 - **PR #26 outcome:** Acceptable and merged after validation. The hardened crawler delivered the expected Phase 1 improvements: caching, star snapshots, stronger low-signal filtering, bounded retry/rate-limit behavior, partial-failure metadata, and regression tests for the new query and payload behavior.
.squad/decisions/inbox/leela-topic-channels.md new
+77
@@ -0,0 +1,77 @@
1 +# Decision: Topic-Specific News Channels Architecture
2 +
3 +**Date:** 2026-05-18T13:20:07.067+02:00
4 +**Author:** Leela (Lead/Architect)
5 +**Status:** Proposed
6 +**PRD:** docs/PRD-topic-channels.md
7 +**PR:** #39
8 +
9 +## Decision Summary
10 +
11 +Generalize SquadScope into a topic-channel system where each instance serves a single focused domain with isolated learning and calibrated predictions.
12 +
13 +## Key Architectural Decisions
14 +
15 +### 1. Feature First, Not Platform
16 +
17 +SquadScope topic channels extend the existing pipeline (Crawl → Score → Analyze → Generate → Deploy) by adding a topic namespace. No new platform, no new repo structure. Same codebase, configured differently.
18 +
19 +### 2. Multi-Instance Single-Topic (v1)
20 +
21 +One fork/config per topic. Each fork has its own `squadscope.topic.yml`, its own Actions schedule, its own GitHub Pages site. This avoids orchestration complexity and keeps learning isolation trivial.
22 +
23 +Multi-topic single-instance is v2 — only after v1 proves the model works.
24 +
25 +### 3. Topic Config as Single Source of Truth
26 +
27 +`squadscope.topic.yml` at repo root controls:
28 +- Crawler queries
29 +- Scoring weights and thresholds
30 +- Analysis tone and audience
31 +- Learning state paths
32 +- Quality criteria
33 +
34 +### 4. Scoring Pipeline (New Stage)
35 +
36 +GitHub topic search is noisy. A new `scripts/score_repos.py` sits between crawl and analyze, scoring repos 0-100 on topic relevance, star momentum, language match, noise penalties, and recency. Only repos scoring ≥40 reach analysis.
37 +
38 +### 5. Per-Topic Learning Isolation
39 +
40 +Each topic maintains:
41 +- `topics/{id}/wisdom.md` — domain-specific heuristics
42 +- `topics/{id}/skills/` — extracted patterns
43 +- `topics/{id}/predictions.jsonl` — prediction ledger
44 +- `topics/{id}/scorecards/` — hindsight validation results
45 +
46 +No cross-topic contamination. Wisdom from AI/ML never leaks into Rust analysis.
47 +
48 +### 6. Prediction Ledger with Hindsight Validation
49 +
50 +Every analysis appends machine-readable predictions to `predictions.jsonl`. Four weeks later, `scripts/validate_predictions.py` scores them against actual outcomes (star deltas, fork growth). Scorecards feed into reskill.
51 +
52 +### 7. Topic Quality Criteria
53 +
54 +Topics must meet minimum thresholds to justify a channel:
55 +- Minimum N repos/week passing filters
56 +- Maximum false positive rate
57 +- Minimum genuinely significant repos per issue
58 +
59 +## Implications
60 +
61 +- Crawler must read config instead of using hardcoded queries
62 +- Analysis prompt becomes a template with injection points
63 +- Hugo gains topic taxonomy and per-topic RSS
64 +- All data paths gain `{topic_id}/` prefix
65 +- Reskill reads per-topic state
66 +
67 +## Risks
68 +
69 +- Topic configs may need frequent tuning in early weeks (mitigated by quality threshold warnings)
70 +- Fork-per-topic model doesn't scale past ~5 topics (acceptable; v2 addresses this)
71 +- Scoring weights are subjective initially (mitigated by prediction validation loop)
72 +
73 +## Open for Discussion
74 +
75 +- Should enrichment signals (forks, contributors) be in v1 scorer or deferred?
76 +- Prediction confidence: fixed initial values or prompt-generated?
77 +- Topic config in root vs `topics/` directory?
docs/PRD-topic-channels.md new
+625
@@ -0,0 +1,625 @@
1 +# PRD: Topic-Specific News Channels for SquadScope
2 +
3 +**Author:** Leela (Lead/Architect)
4 +**Date:** 2026-05-18
5 +**Status:** Draft
6 +**Type:** Feature PRD
7 +**Depends on:** docs/analysis-spec.md, docs/pipeline-validation.md, docs/learning-audit.md, .squad/decisions.md
8 +
9 +---
10 +
11 +## Executive Summary
12 +
13 +SquadScope currently crawls all of GitHub looking for "what's interesting this week." This produces broad but shallow coverage — jack of all topics, master of none. This PRD defines how to generalize SquadScope into a **topic-channel system** where each deployment is configured for a specific domain (e.g., `ai-ml`, `rust`, `security`), producing focused, expert-level weekly digests with calibrated learning per topic.
14 +
15 +The approach is **feature first, not a separate platform.** v1 delivers a single configurable topic per instance (one fork/config per topic). Multi-topic single-instance is deferred to v2.
16 +
17 +---
18 +
19 +## Problem Statement
20 +
21 +### Why topic-specific is better than general
22 +
23 +1. **Signal quality degrades with breadth.** A general GitHub crawl returns repos spanning AI, systems programming, web frameworks, security tools, and student homework. No single editorial voice can meaningfully assess "is this Rust crate important?" and "is this ML paper implementation notable?" in the same breath.
24 +
25 +2. **Learning cannot calibrate across domains.** The learning audit (docs/learning-audit.md) identified that wisdom must flow back into analysis (G7). But shared wisdom across domains produces uncalibrated judgments — a heuristic like "repos with >500 stars/week are always significant" is true in AI/ML but false in niche systems programming.
26 +
27 +3. **GitHub topic filtering is noisy.** Simply adding `topic:rust` to a search query returns thousands of results including tutorials, homework, and abandoned projects. Topic filtering needs a multi-stage pipeline: query → score → filter → analyze.
28 +
29 +4. **Readers want depth, not breadth.** A security professional subscribing to SquadScope doesn't want to scroll past 15 ML repos to find the 3 security tools that matter this week.
30 +
31 +5. **Predictions need domain context.** "This will be important" means something different in each field. A prediction ledger must be per-topic to be meaningful.
32 +
33 +---
34 +
35 +## Goals & Non-Goals
36 +
37 +### Goals
38 +
39 +- **G1:** Define a topic configuration format (`squadscope.topic.yml`) that controls all topic-specific behavior
40 +- **G2:** Namespace all data, content, prompts, RSS, and learning state by topic
41 +- **G3:** Deliver per-topic RSS feeds at `/topics/{topic}/index.xml`
42 +- **G4:** Implement a scoring pipeline for GitHub results (not just keyword filters)
43 +- **G5:** Isolate learning state per topic (wisdom, skills, prediction scores)
44 +- **G6:** Add a prediction ledger that tracks claims vs outcomes per topic
45 +- **G7:** Define topic quality criteria (minimum viable coverage thresholds)
46 +- **G8:** Ship 2 example topic configs: `ai-ml` and `rust`
47 +
48 +### Non-Goals
49 +
50 +- Multi-topic single-instance deployment (v2)
51 +- Topic marketplace or discovery
52 +- User-facing topic configuration UI
53 +- Real-time or daily publishing cadence
54 +- Cross-topic trend correlation
55 +- Non-GitHub data sources per topic (future enhancement)
56 +
57 +---
58 +
59 +## Topic Configuration
60 +
61 +### File: `squadscope.topic.yml`
62 +
63 +Each SquadScope instance has exactly one topic config at the repository root. Unless explicitly normalized by the scorer, `language_boost` keys should match the crawler's raw GitHub language strings exactly (for example `Python`, `Rust`, `Jupyter Notebook`).
64 +
65 +```yaml
66 +# squadscope.topic.yml — defines a single topic channel
67 +topic:
68 + id: ai-ml # URL-safe identifier
69 + name: "AI & Machine Learning" # Human-readable name
70 + description: "Weekly digest of significant AI/ML repositories, frameworks, and research implementations on GitHub"
71 +
72 +# Crawler queries — multiple queries combined for coverage
73 +queries:
74 + primary:
75 + - "topic:machine-learning stars:>50 pushed:>{last_week}"
76 + - "topic:deep-learning stars:>50 pushed:>{last_week}"
77 + - "topic:artificial-intelligence stars:>30 pushed:>{last_week}"
78 + - "topic:llm stars:>20 pushed:>{last_week}"
79 + secondary:
80 + - "topic:transformers stars:>100 pushed:>{last_week}"
81 + - "topic:diffusion stars:>30 pushed:>{last_week}"
82 + - "topic:rag stars:>20 pushed:>{last_week}"
83 +
84 +# Scoring pipeline — repos must pass these filters
85 +scoring:
86 + min_stars: 20 # Absolute minimum to consider
87 + min_stars_gained: 10 # Minimum weekly star delta
88 + max_age_days: 365 # Exclude repos older than this from "new" category
89 + min_relevance_score: 40 # Repos below this score do not reach analysis
90 + language_boost: # Keys match raw crawler language values
91 + Python: 1.2
92 + Jupyter Notebook: 1.1
93 + Rust: 1.0
94 + topic_relevance: # Required topic overlap (at least one must match)
95 + - machine-learning
96 + - deep-learning
97 + - artificial-intelligence
98 + - neural-network
99 + - llm
100 + - nlp
101 + - computer-vision
102 + - reinforcement-learning
103 + noise_topics: # Penalty topics (reduce score)
104 + - tutorial
105 + - course
106 + - awesome-list
107 + - homework
108 + noise_name_patterns: # Regex patterns that reduce relevance score
109 + - "^awesome-"
110 + - "-tutorial$"
111 + - "-course$"
112 +
113 +# Quality thresholds — topic must meet these to justify a channel
114 +quality:
115 + min_repos_per_week: 8 # Minimum repos passing filters weekly
116 + max_false_positive_rate: 0.25 # Max 25% irrelevant results after scoring
117 + min_signal_repos: 3 # At least 3 genuinely significant repos per issue
118 +
119 +# Content configuration
120 +content:
121 + tone: "technical, analytical"
122 + audience: "ML engineers and researchers"
123 + emphasis:
124 + - "Novel architectures and training techniques"
125 + - "Production-ready frameworks and tools"
126 + - "Significant performance improvements"
127 + de_emphasis:
128 + - "Yet another wrapper around OpenAI API"
129 + - "Awesome lists and link collections"
130 + - "Course materials and tutorials"
131 +
132 +# Learning configuration
133 +learning:
134 + wisdom_file: "topics/ai-ml/wisdom.md"
135 + skills_dir: "topics/ai-ml/skills/"
136 + predictions_file: "topics/ai-ml/predictions.jsonl"
137 + reskill_context:
138 + - "What ML-specific heuristics should we update?"
139 + - "Are we over/under-weighting any sub-domain?"
140 + - "Which prediction categories are we worst at?"
141 +```
142 +
143 +### Second example: `rust`
144 +
145 +```yaml
146 +topic:
147 + id: rust
148 + name: "Rust Ecosystem"
149 + description: "Weekly digest of significant Rust crates, tools, and ecosystem developments"
150 +
151 +queries:
152 + primary:
153 + - "language:rust stars:>30 pushed:>{last_week}"
154 + - "topic:rust stars:>20 pushed:>{last_week}"
155 + - "topic:rust-lang stars:>20 pushed:>{last_week}"
156 + secondary:
157 + - "topic:cargo stars:>50 pushed:>{last_week}"
158 + - "topic:wasm language:rust stars:>30 pushed:>{last_week}"
159 +
160 +scoring:
161 + min_stars: 15
162 + min_stars_gained: 8
163 + max_age_days: 730
164 + min_relevance_score: 40
165 + language_boost:
166 + Rust: 1.5
167 + C: 1.0
168 + topic_relevance:
169 + - rust
170 + - rust-lang
171 + - cargo
172 + - wasm
173 + - systems-programming
174 + - embedded
175 + noise_topics:
176 + - tutorial
177 + - learning-rust
178 + - rust-exercises
179 + noise_name_patterns:
180 + - "^rust-by-example"
181 + - "-exercises$"
182 +
183 +quality:
184 + min_repos_per_week: 5
185 + max_false_positive_rate: 0.30
186 + min_signal_repos: 2
187 +
188 +content:
189 + tone: "systems-oriented, precise"
190 + audience: "Rust developers and systems programmers"
191 + emphasis:
192 + - "Crates reaching stability milestones"
193 + - "Performance and safety innovations"
194 + - "Ecosystem tooling improvements"
195 + de_emphasis:
196 + - "Beginner tutorials"
197 + - "Reimplementations of existing tools without novel approach"
198 +
199 +learning:
200 + wisdom_file: "topics/rust/wisdom.md"
201 + skills_dir: "topics/rust/skills/"
202 + predictions_file: "topics/rust/predictions.jsonl"
203 + reskill_context:
204 + - "Are we calibrated for the Rust ecosystem's smaller scale?"
205 + - "Which crate categories are we missing?"
206 +```
207 +
208 +---
209 +
210 +## Pipeline Changes
211 +
212 +### Overview
213 +
214 +```
215 +┌─────────────────────────────────────────────────────────────────────────┐
216 +│ Topic-Aware Pipeline (v1) │
217 +├─────────────────────────────────────────────────────────────────────────┤
218 +│ │
219 +│ squadscope.topic.yml │
220 +│ │ │
221 +│ ▼ │
222 +│ ┌───────────┐ query+ ┌───────────┐ scored ┌──────────────┐ │
223 +│ │ Crawler │──────────►│ Scorer │──────────►│ Analyzer │ │
224 +│ └───────────┘ raw JSON └───────────┘ repos └──────────────┘ │
225 +│ │ │
226 +│ ┌───────────┼──────────┐ │
227 +│ ▼ ▼ ▼ │
228 +│ ┌─────────┐ ┌──────────┐ ┌────┐│
229 +│ │ Content │ │Prediction│ │RSS ││
230 +│ │ Page │ │ Ledger │ │Feed││
231 +│ └─────────┘ └──────────┘ └────┘│
232 +│ │
233 +│ Learning loop (per topic): │
234 +│ predictions.jsonl → validate_predictions.py → scorecard → reskill │
235 +│ → updated wisdom.md → injected into next analysis prompt │
236 +│ │
237 +└─────────────────────────────────────────────────────────────────────────┘
238 +```
239 +
240 +### 1. Crawler Changes (`scripts/crawl.py`)
241 +
242 +**Current:** Hardcoded queries in `crawl.py` searching for generic trending repos.
243 +
244 +**Proposed:**
245 +- Read `squadscope.topic.yml` at startup
246 +- Build search queries from `queries.primary` and `queries.secondary`
247 +- Apply `scoring.min_stars` as a pre-filter in the GitHub API query
248 +- Template `{last_week}` in query strings to ISO date of 7 days ago
249 +- Output to `data/raw/{topic_id}/YYYY-WNN.json` (namespaced)
250 +
251 +```python
252 +# Pseudocode for topic-aware crawling
253 +config = load_topic_config("squadscope.topic.yml")
254 +queries = config["queries"]["primary"] + config["queries"]["secondary"]
255 +for q in queries:
256 + q = q.replace("{last_week}", last_week_iso())
257 + results = search_github(q)
258 + all_repos.extend(results)
259 +
260 +# Deduplicate by full_name, then classify into the existing raw payload shape
261 +unique_repos = deduplicate(all_repos)
262 +new_repos, trending_repos = partition_repo_sets(unique_repos)
263 +payload = {
264 + "week": current_iso_week(),
265 + "new_repos": new_repos,
266 + "trending_repos": trending_repos,
267 + "signals": build_signals(new_repos, trending_repos),
268 + "metadata": build_metadata(config, queries),
269 +}
270 +write_json(f"data/raw/{config['topic']['id']}/YYYY-WNN.json", payload)
271 +```
272 +
273 +### 2. New: Scoring Pipeline (`scripts/score_repos.py`)
274 +
275 +A new pipeline stage between crawl and analyze. Repos get a **relevance score** (0-100):
276 +
277 +| Factor | Weight | Scoring Logic |
278 +|--------|--------|---------------|
279 +| Topic overlap | 30% | Count of repo topics matching `scoring.topic_relevance` |
280 +| Star momentum | 25% | `stars_gained / min_stars_gained` ratio (capped at 3x) |
281 +| Language match | 15% | Boost from `scoring.language_boost` |
282 +| Noise penalty | -20% | Repos matching `noise_topics` or `noise_name_patterns` |
283 +| Recency | 10% | Days since last push (more recent = higher) |
284 +
285 +**Output:** `data/scored/{topic_id}/YYYY-WNN.json` — same top-level schema as raw (`week`, `new_repos`, `trending_repos`, `signals`, `metadata`), with `relevance_score` added to repo entries and scoring/filter metadata appended under `metadata`. Only repos with `relevance_score >= scoring.min_relevance_score` pass to analysis.
286 +
287 +### 3. Analysis Prompt Changes (`prompts/analyze-weekly.md`)
288 +
289 +**Current:** Static prompt with no topic context or learned state.
290 +
291 +**Proposed:** Topic-aware prompt template with injection points:
292 +
293 +```markdown
294 +# Weekly Analysis: {{TOPIC_NAME}}
295 +
296 +You are analyzing GitHub repositories for the **{{TOPIC_NAME}}** channel.
297 +Audience: {{AUDIENCE}}
298 +Tone: {{TONE}}
299 +
300 +## Emphasis
301 +{{EMPHASIS_LIST}}
302 +
303 +## De-emphasis
304 +{{DE_EMPHASIS_LIST}}
305 +
306 +## Learned Wisdom (from prior reskill cycles)
307 +{{WISDOM_CONTENT}}
308 +
309 +## Active Skills
310 +{{SKILLS_CONTENT}}
311 +
312 +## Prediction Track Record
313 +{{PREDICTION_SCORECARD}}
314 +
315 +## Instructions
316 +Analyze the scored repositories in `data/scored/{{TOPIC_ID}}/YYYY-WNN.json`.
317 +...
318 +```
319 +
320 +### 4. Content Namespacing
321 +
322 +| Asset | Current Path | Topic-Aware Path |
323 +|-------|-------------|-----------------|
324 +| Raw crawl data | `data/raw/YYYY-WNN.json` | `data/raw/{topic_id}/YYYY-WNN.json` |
325 +| Scored data | N/A (new) | `data/scored/{topic_id}/YYYY-WNN.json` |
326 +| Analysis output | `data/analyzed/YYYY-WNN-summary.md` | `data/analyzed/{topic_id}/YYYY-WNN-summary.md` |
327 +| Star snapshots | `data/snapshots/YYYY-WNN.json` | `data/snapshots/{topic_id}/YYYY-WNN.json` |
328 +| Hugo content | `content/weekly/YYYY-WNN.md` | `content/topics/{topic_id}/YYYY-WNN.md` |
329 +| RSS feed | `/index.xml` | `/topics/{topic_id}/index.xml` |
330 +| Wisdom | `.squad/identity/wisdom.md` | `topics/{topic_id}/wisdom.md` |
331 +| Skills | `.squad/skills/` | `topics/{topic_id}/skills/` |
332 +| Predictions | N/A (new) | `topics/{topic_id}/predictions.jsonl` |
333 +
334 +### 5. RSS Per Topic
335 +
336 +Hugo taxonomy configuration:
337 +
338 +```toml
339 +# hugo.toml additive changes
340 +[taxonomies]
341 + tag = "tags"
342 + category = "categories"
343 + topic = "topics"
344 +
345 +[outputFormats.RSS]
346 + mediaType = "application/rss+xml"
347 + baseName = "index"
348 +
349 +[params]
350 + topicId = "ai-ml" # From squadscope.topic.yml
351 +```
352 +
353 +Each topic gets its own RSS feed at `/topics/{topic_id}/index.xml`. The site root `/index.xml` remains as an aggregate feed (or is removed in single-topic mode).
354 +
355 +---
356 +
357 +## Learning System Integration
358 +
359 +### Per-Topic Learning State
360 +
361 +Each topic maintains isolated learning state:
362 +
363 +```
364 +topics/{topic_id}/
365 +├── wisdom.md # Accumulated heuristics for this domain
366 +├── skills/ # Extracted patterns and rules
367 +│ ├── SKILL-001.md
368 +│ └── SKILL-002.md
369 +├── predictions.jsonl # Prediction ledger (append-only)
370 +└── scorecards/ # Hindsight validation results
371 + ├── 2026-W21.json
372 + └── 2026-W25.json
373 +```
374 +
375 +### Why Isolation Matters
376 +
377 +From the learning audit: "shared wisdom across domains produces uncalibrated judgments." Examples:
378 +
379 +- AI/ML wisdom: "Repos with HuggingFace integrations tend to gain adoption quickly" → **meaningless for Rust**
380 +- Rust wisdom: "Crates with `no_std` support indicate systems-level seriousness" → **meaningless for AI/ML**
381 +- Security wisdom: "CVE-related repos spike and fade within 2 weeks" → **misleading if applied to general software**
382 +
383 +### Prediction Ledger (`predictions.jsonl`)
384 +
385 +Each analysis produces machine-readable predictions appended to the ledger:
386 +
387 +```jsonl
388 +{"week":"2026-W21","repo":"owner/name","claim":"signal","confidence":0.8,"category":"framework","predicted_stars_4w":500}
389 +{"week":"2026-W21","repo":"owner/name","claim":"noise","confidence":0.7,"category":"wrapper","reason":"thin wrapper around existing API"}
390 +{"week":"2026-W21","repo":"owner/name","claim":"gap","confidence":0.6,"category":"missing-tooling","description":"No good Rust WASM debugger exists yet"}
391 +```
392 +
393 +**Fields:**
394 +- `week`: ISO week of the prediction
395 +- `repo`: Full repository name (or null for gap predictions)
396 +- `claim`: One of `signal`, `noise`, `gap`
397 +- `confidence`: 0.0-1.0 how sure the system is
398 +- `category`: Domain-specific category
399 +- `predicted_stars_4w`: Expected star count in 4 weeks (for signal/noise)
400 +- `reason`/`description`: Human-readable explanation
401 +
402 +### Hindsight Validation (`scripts/validate_predictions.py`)
403 +
404 +Runs 4 weeks after predictions are made. Compares claims to outcomes:
405 +
406 +```python
407 +# Validation logic
408 +for prediction in load_predictions(topic_id, target_week):
409 + if prediction["claim"] == "signal":
410 + actual_stars = get_current_stars(prediction["repo"])
411 + predicted = prediction["predicted_stars_4w"]
412 + score = min(actual_stars / predicted, 2.0) # Cap at 2x
413 + scorecard.append({"prediction": prediction, "actual": actual_stars, "score": score})
414 + elif prediction["claim"] == "noise":
415 + # Noise repos should have plateaued or declined
416 + delta = get_star_delta(prediction["repo"], weeks=4)
417 + score = 1.0 if delta < prediction.get("predicted_stars_4w", 50) else 0.0
418 + scorecard.append({"prediction": prediction, "actual_delta": delta, "score": score})
419 +```
420 +
421 +**Scorecard output** feeds into reskill: "Last month we were 72% accurate on signal calls but only 45% on noise calls in ai-ml. We tend to overestimate wrapper libraries."
422 +
423 +### Reskill Integration
424 +
425 +The reskill prompt (run every 5th cycle) now receives:
426 +1. Topic-specific wisdom from `topics/{topic_id}/wisdom.md`
427 +2. Latest scorecard from `topics/{topic_id}/scorecards/`
428 +3. Prediction accuracy trend across last 5 scorecards
429 +4. Topic config context (what we're optimizing for)
430 +
431 +Reskill outputs are written back to topic-specific paths, ensuring one topic's learnings never contaminate another.
432 +
433 +---
434 +
435 +## Content Architecture
436 +
437 +### Topic Channels
438 +
439 +URL structure:
440 +```
441 +/ → Home (links to topic channel)
442 +/topics/{topic_id}/ → Topic landing page (latest + archive)
443 +/topics/{topic_id}/2026-W21 → Weekly issue page
444 +/topics/{topic_id}/index.xml → RSS feed for this topic
445 +```
446 +
447 +### Hugo Content Structure
448 +
449 +```
450 +content/
451 +└── topics/
452 + └── ai-ml/
453 + ├── _index.md # Topic landing page
454 + ├── 2026-W21.md # Weekly issue
455 + ├── 2026-W22.md
456 + └── ...
457 +```
458 +
459 +### Navigation
460 +
461 +For v1 (single-topic instance), the site homepage redirects to the topic channel. The topic archive page lists all weekly issues with summaries.
462 +
463 +For v2 (multi-topic), a topic selector would appear in navigation.
464 +
465 +---
466 +
467 +## v1 Scope: Single Configurable Topic
468 +
469 +### What ships in v1
470 +
471 +1. **`squadscope.topic.yml` config format** — fully specified, validated at pipeline start
472 +2. **Topic-aware crawler** — reads queries from config, outputs to namespaced paths
473 +3. **Scoring pipeline** — `scripts/score_repos.py` with configurable weights
474 +4. **Topic-aware analysis prompt** — injects topic context, wisdom, and scorecard
475 +5. **Prediction ledger** — appended to after each analysis
476 +6. **Hindsight validation script** — runs on 4-week-old predictions
477 +7. **Per-topic learning state** — isolated wisdom, skills, scorecards
478 +8. **Topic RSS feed** — at `/topics/{topic_id}/index.xml`
479 +9. **Two example configs** — `examples/topics/ai-ml.yml` and `examples/topics/rust.yml`
480 +10. **Config validation script** — `scripts/validate_topic_config.py`
481 +
482 +### What does NOT ship in v1
483 +
484 +- Multi-topic in a single instance
485 +- Topic discovery or marketplace
486 +- Cross-topic learning transfer
487 +- Dynamic query generation
488 +- Topic health monitoring dashboard
489 +
490 +### Deployment Model (v1)
491 +
492 +One SquadScope fork per topic. Each fork:
493 +- Has its own `squadscope.topic.yml`
494 +- Runs its own GitHub Actions schedule
495 +- Produces its own GitHub Pages site
496 +- Accumulates its own learning state
497 +- Has its own RSS feed
498 +
499 +This is intentionally simple. Forks share the same codebase but diverge on configuration and learned state.
500 +
501 +---
502 +
503 +## v2 Vision: Multi-Topic Single Instance
504 +
505 +**Deferred.** Documented here for future planning only.
506 +
507 +### What v2 would add
508 +
509 +- Single instance running multiple topics on different schedules
510 +- Shared infrastructure, isolated topic state
511 +- Topic health monitoring (auto-disable topics below quality thresholds)
512 +- Cross-topic signals ("this repo is trending in BOTH ai-ml and rust channels")
513 +- Topic marketplace (community-contributed topic configs)
514 +- Unified navigation across topics
515 +
516 +### Why v2 is premature now
517 +
518 +- Adds orchestration complexity (per-topic cron, per-topic secrets)
519 +- Learning isolation is harder in shared instances (accidental cross-contamination)
520 +- No user demand signal yet — need v1 adoption data first
521 +- GitHub Actions concurrency constraints make multi-topic scheduling complex
522 +
523 +---
524 +
525 +## Implementation Plan
526 +
527 +### Issues to Create
528 +
529 +| # | Title | Phase | Depends On | Assignee Profile |
530 +|---|-------|-------|-----------|-----------------|
531 +| 1 | Define `squadscope.topic.yml` schema and validator | Foundation | — | Architect |
532 +| 2 | Namespace data directories by topic ID | Foundation | #1 | Crawler |
533 +| 3 | Implement scoring pipeline (`scripts/score_repos.py`) | Pipeline | #1, #2 | Crawler |
534 +| 4 | Make crawler read queries from topic config | Pipeline | #1, #2 | Crawler |
535 +| 5 | Create topic-aware analysis prompt template | Pipeline | #1 | Analyzer |
536 +| 6 | Add prediction ledger output to analysis | Pipeline | #5 | Analyzer |
537 +| 7 | Implement hindsight validation script | Learning | #6 | Analyzer |
538 +| 8 | Per-topic learning state directories and seeding | Learning | #1 | Architect |
539 +| 9 | Wire prediction scorecard into reskill prompt | Learning | #7 | Analyzer |
540 +| 10 | Hugo topic taxonomy and per-topic RSS | Content | #2 | Site |
541 +| 11 | Topic landing page template | Content | #10 | Site |
542 +| 12 | Example config: ai-ml | Validation | #1-#4 | Validator |
543 +| 13 | Example config: rust | Validation | #1-#4 | Validator |
544 +| 14 | Topic quality threshold enforcement | Quality | #3 | Crawler |
545 +| 15 | End-to-end integration test with example topic | Validation | All | Validator |
546 +
547 +### Dependencies
548 +
549 +```
550 +#1 (schema) ─┬─► #2 (namespacing) ─┬─► #3 (scorer) ──► #4 (crawler)
551 + │ │ │
552 + │ └─► #10 (Hugo) ▼
553 + │ #12, #13 (examples)
554 + └─► #5 (prompt) ──► #6 (predictions) ──► #7 (validation)
555 + │
556 + ▼
557 + #9 (reskill wiring)
558 +```
559 +
560 +### Estimated Effort
561 +
562 +- **Foundation (Issues 1-2):** 1 session
563 +- **Pipeline (Issues 3-6):** 2-3 sessions
564 +- **Learning (Issues 7-9):** 2 sessions
565 +- **Content (Issues 10-11):** 1 session
566 +- **Validation (Issues 12-15):** 1-2 sessions
567 +
568 +**Total:** ~7-9 work sessions
569 +
570 +---
571 +
572 +## Open Questions
573 +
574 +| # | Question | Impact | Proposed Resolution |
575 +|---|----------|--------|-------------------|
576 +| OQ1 | Should topic configs live in repo root or `topics/` dir? | File organization | Repo root for v1 (single topic); move to `topics/` in v2 |
577 +| OQ2 | How to handle repos that span multiple topics? | Dedup in multi-topic v2 | v1: irrelevant (single topic). v2: each topic scores independently |
578 +| OQ3 | What's the minimum weeks of data before learning is meaningful? | Reskill timing | Propose 4 weeks minimum before first hindsight validation runs |
579 +| OQ4 | Should prediction confidence be system-generated or human-calibrated initially? | Learning accuracy | Start with fixed confidence (0.7 for signal, 0.5 for noise), calibrate after 8 weeks of scorecard data |
580 +| OQ5 | Enrichment signals beyond stars — which to add first? | Prediction quality | Forks and contributor count (cheapest API calls, highest signal per learning-audit G13) |
581 +| OQ6 | Should topic quality thresholds auto-disable a topic or just warn? | Reliability | Warn-only for v1 (log to workflow summary), auto-disable in v2 |
582 +
583 +---
584 +
585 +## Success Metrics
586 +
587 +### Quantitative (measurable after 8 weeks of operation)
588 +
589 +| Metric | Target | Measurement |
590 +|--------|--------|-------------|
591 +| False positive rate | < 25% per topic | Manual audit of 20 random "signal" calls per month |
592 +| Prediction accuracy (signal) | > 65% | Hindsight validation scorecard |
593 +| Prediction accuracy (noise) | > 55% | Hindsight validation scorecard |
594 +| Repos per weekly issue | ≥ quality.min_repos_per_week from config | Automated count |
595 +| RSS subscribers per topic | > 0 within 4 weeks | Analytics (if available) |
596 +| Learning improvement trend | Prediction accuracy increases by ≥ 5% over 8 weeks | Scorecard comparison |
597 +
598 +### Qualitative
599 +
600 +- Topic experts find the digest "saves them time" vs. manual GitHub browsing
601 +- Analysis tone matches configured audience expectations
602 +- Signal/Noise/Gaps sections feel calibrated to the specific domain
603 +- Learned wisdom in `wisdom.md` contains domain-specific (not generic) heuristics after 3 reskill cycles
604 +
605 +---
606 +
607 +## Relationship to Existing Work
608 +
609 +### Analysis Spec (`docs/analysis-spec.md`)
610 +This PRD extends the approved analyzer contract. Topic-aware raw and scored artifacts keep the existing top-level payload shape (`week`, `new_repos`, `trending_repos`, `signals`, `metadata`) while adding a topic namespace prefix and repo-level `relevance_score` data.
611 +
612 +### Pipeline Validation (`docs/pipeline-validation.md`)
613 +This PRD preserves the current Crawl → Analyze → Generate workflow expectations while adding one topic-aware scoring step between crawl and analyze. Existing quality gates and artifact validation remain in force.
614 +
615 +### Learning Audit (`docs/learning-audit.md`)
616 +This PRD directly addresses:
617 +- **G7 (prompt feedback loop):** Topic-aware prompt template with `{{WISDOM_CONTENT}}` injection
618 +- **G8 (hindsight validation):** `scripts/validate_predictions.py` with per-topic scorecards
619 +- **G9 (prediction registry):** `predictions.jsonl` format defined
620 +- **G13 (enrichment signals):** Fork/contributor data noted as OQ5, planned for scorer enrichment
621 +
622 +### Decisions (`.squad/decisions.md`)
623 +- Respects Decision 3 (pipeline stage contracts) — adds a scoring stage but preserves existing boundaries
624 +- Respects Decision 4 (reviewer gate) — quality gate applies per-topic
625 +- Extends Decision 6 (reskill) — reskill reads per-topic state instead of global state
docs/learning-audit.md new
+238
@@ -0,0 +1,238 @@
1 +# SquadScope Learning System Audit
2 +
3 +**Author:** Leela (Lead/Architect)
4 +**Date:** 2026-05-18T13:20:07.067+02:00
5 +**Scope:** End-to-end audit of the learning system — can SquadScope actually learn in production?
6 +
7 +---
8 +
9 +## Executive Summary
10 +
11 +**Verdict: The learning system will NOT work in production today.** The design is sound but the implementation is almost entirely absent. Issues #14 and #15 correctly describe what needs to be built, but zero code exists for the reskill cycle. The workflow has no reskill job, no run counter exists, no `.squad/skills/` directory has been created, `wisdom.md` is empty, and there is no hindsight validation mechanism. The learning loop is a design-only artifact.
12 +
13 +---
14 +
15 +## 1. Reskill Cycle Implementation Status
16 +
17 +### Issues #14 and #15 Assessment
18 +
19 +| Issue | Title | Design Quality | Implementation | Status |
20 +|-------|-------|---------------|----------------|--------|
21 +| #14 | Implement reskill retrospective workflow and squad-state outputs | ✅ Good | ❌ Zero code | OPEN |
22 +| #15 | Add run counter persistence and every-fifth-run reskill trigger | ✅ Good | ❌ Zero code | OPEN |
23 +
24 +### Will the every-5th-run trigger work?
25 +
26 +**Design (from `.squad/decisions.md` Decision 6):**
27 +```bash
28 +COUNTER=$(cat .squad/run-counter.txt)
29 +if [ $((COUNTER % 5)) -eq 0 ]; then
30 + # reskill invocation
31 +fi
32 +```
33 +
34 +**Problems with this design:**
35 +
36 +1. **No run-counter.txt exists.** The file has never been created. First run would fail with `cat: .squad/run-counter.txt: No such file or directory`.
37 +2. **No increment logic shown.** The decision shows the *check* but not who increments the counter or when. If the counter increments before the check, run 5 triggers. If after, run 6 triggers. The ordering matters and isn't specified.
38 +3. **No atomicity guarantee.** The workflow uses stash/pop for commits but the counter update could race with concurrent `workflow_dispatch` triggers.
39 +4. **No reskill job in the workflow.** `crawl-and-publish.yml` has jobs: `crawl`, `analyze`, `generate`, `deploy`, `notify`. There is no `reskill` job.
40 +5. **Counter survives only if committed.** The workflow commits `data/raw`, `data/snapshots`, `data/analyzed`, and `content/weekly/` — but not `.squad/`. Counter changes would be lost.
41 +
42 +### What's missing for #14 and #15:
43 +
44 +- A `reskill` job in `crawl-and-publish.yml` (or a separate workflow)
45 +- Counter initialization (`echo 0 > .squad/run-counter.txt`)
46 +- Counter increment step in the crawl or analyze job
47 +- Commit step that includes `.squad/run-counter.txt` changes
48 +- The actual Copilot CLI invocation for reskill analysis
49 +- Output writing to `.squad/reskill/YYYY-WNN.md`
50 +- Commit step for reskill outputs (`.squad/` directory changes)
51 +
52 +---
53 +
54 +## 2. Learning Inputs
55 +
56 +### Current state of each input source:
57 +
58 +| Input | Exists? | Has Content? | Accessible in CI? | Notes |
59 +|-------|---------|--------------|-------------------|-------|
60 +| `.squad/agents/*/history.md` | ✅ Yes (7 agents) | ✅ Yes | ✅ Yes (checked out) | Contains real project learnings |
61 +| `.squad/decisions.md` | ✅ Yes | ✅ Yes | ✅ Yes | Rich decision log |
62 +| `.squad/identity/wisdom.md` | ✅ Yes | ❌ Empty (header only) | ✅ Yes | Zero accumulated wisdom |
63 +| `.squad/skills/` | ❌ No | N/A | N/A | Directory never created |
64 +| Past analysis outputs | ⚠️ Partial | Only if runs have occurred | ✅ Yes (`data/analyzed/`) | No historical comparison logic exists |
65 +| Star snapshots | ⚠️ Empty | `data/snapshots/` exists but is empty | ✅ Yes | Crawler writes snapshots but none committed yet |
66 +
67 +### Hindsight validation capability:
68 +
69 +**Does not exist.** There is no code or process that:
70 +- Compares "what we said was important 5 weeks ago" with "what actually became important"
71 +- Uses star snapshot deltas to validate past predictions
72 +- Measures whether Signal/Noise/Gaps calls were accurate in retrospect
73 +
74 +The analysis spec mentions comparing to prior weeks (§ Context dimension), but this is only for the *current* analysis — it doesn't feed backward into learning.
75 +
76 +---
77 +
78 +## 3. Learning Outputs
79 +
80 +### What should change after a reskill:
81 +
82 +| Output | Mechanism Defined? | Implementation? | Will It Persist? |
83 +|--------|-------------------|-----------------|-----------------|
84 +| Updated `wisdom.md` heuristics | ⚠️ Implied only | ❌ No code | ❌ No commit step |
85 +| New/updated skills in `.squad/skills/` | ✅ Reskill SKILL.md template exists | ❌ No code | ❌ No commit step, no directory |
86 +| Adjusted significance thresholds | ❌ Not designed | ❌ No code | N/A |
87 +| Updated hype detection patterns | ❌ Not designed | ❌ No code | N/A |
88 +| Revised gap analysis focus | ❌ Not designed | ❌ No code | N/A |
89 +| Reskill report in `.squad/reskill/YYYY-WNN.md` | ✅ Yes (Decision 6) | ❌ No code | ❌ No commit step |
90 +
91 +### Critical gap: No feedback into the analysis prompt
92 +
93 +Even if reskill produces updated wisdom or patterns, the `prompts/analyze-weekly.md` template has no variable or include that would inject learned heuristics. The analysis prompt is static. Learning outputs have no path back into the analysis pipeline.
94 +
95 +---
96 +
97 +## 4. Learning Feedback Loop
98 +
99 +### Can the system measure improvement?
100 +
101 +**No.** There is no mechanism to:
102 +
103 +1. **Record predictions with timestamps.** The analysis output exists (`data/analyzed/YYYY-WNN-summary.md`) but Signal/Noise/Gaps claims are not stored in a machine-readable format that enables later comparison.
104 +
105 +2. **Compare predictions to outcomes.** Star snapshots (`data/snapshots/`) could provide ground truth (did "Signal" repos actually grow? did "Noise" repos fade?), but:
106 + - The snapshots directory is currently empty
107 + - No script compares week N predictions against week N+4 star deltas
108 + - No "scorecard" format exists
109 +
110 +3. **Attribute improvement to reskill changes.** Without a baseline quality metric tracked over time, there's no way to know if reskill actually improved anything.
111 +
112 +### Is star snapshot data sufficient?
113 +
114 +**Partially.** Star snapshots can validate:
115 +- ✅ "This repo is gaining momentum" (compare stars at week N vs N+4)
116 +- ✅ "This is hype" (stars plateau or decline)
117 +- ❌ "This gap matters" (absence can't be validated by stars alone)
118 +- ❌ "This trend is durable" (needs signals beyond stars — commits, forks, adoption)
119 +
120 +### Missing data collection for better learning:
121 +
122 +1. **Prediction registry:** Machine-readable claims from each analysis (repo X will grow, theme Y is noise) with confidence scores
123 +2. **Outcome tracker:** Script that revisits predictions after N weeks using snapshot data
124 +3. **Quality trend log:** `quality_score` from each analysis plotted over time
125 +4. **External validation signals:** Fork counts, contributor growth, dependency adoption — richer than stars alone
126 +
127 +---
128 +
129 +## 5. Persistence
130 +
131 +### Will learnings survive across sessions and workflow runs?
132 +
133 +| Question | Answer | Evidence |
134 +|----------|--------|----------|
135 +| Are `.squad/` files committed after reskill? | ❌ **No** | No commit step for `.squad/` exists in the workflow |
136 +| Does the workflow have `contents: write`? | ✅ **Yes** | `analyze` job has `contents: write`; would need same for reskill job |
137 +| Will Copilot CLI have `.squad/` state during reskill? | ✅ **Yes** (if checkout is full) | The workflow checks out with `fetch-depth: 0` — `.squad/` is in the repo |
138 +| Is `run-counter.txt` persisted? | ❌ **No** | File doesn't exist; no commit step would save it |
139 +| Are reskill outputs persisted? | ❌ **No** | `.squad/reskill/` directory doesn't exist; no commit step |
140 +
141 +### The persistence chain is broken at every link:
142 +
143 +```
144 +Run → Counter increment → [NOT COMMITTED] → Lost
145 +Reskill → wisdom.md update → [NOT COMMITTED] → Lost
146 +Reskill → skill extraction → [NO DIRECTORY] → Lost
147 +Reskill → report → [NOT COMMITTED] → Lost
148 +```
149 +
150 +**The workflow only commits:** `data/raw/`, `data/snapshots/`, `data/analyzed/`, `content/weekly/`. Squad state changes are invisible to git.
151 +
152 +---
153 +
154 +## 6. Gap Analysis
155 +
156 +### Critical Gaps (Learning will not happen without these)
157 +
158 +| # | Gap | Impact | Proposed Fix | New Issue? |
159 +|---|-----|--------|--------------|------------|
160 +| G1 | No reskill job in workflow | Reskill never triggers | Add `reskill` job to `crawl-and-publish.yml` with counter check | Part of #15 |
161 +| G2 | No `run-counter.txt` | Counter check fails on first run | Initialize file; add increment in crawl job commit step | Part of #15 |
162 +| G3 | No `.squad/` commit step | All learning outputs lost between runs | Add commit step for `.squad/` after reskill | Part of #14 |
163 +| G4 | No `.squad/skills/` directory | Skill extraction has nowhere to write | Create directory with `.gitkeep` | Part of #14 |
164 +| G5 | No `.squad/reskill/` directory | Reskill reports have nowhere to go | Create directory with `.gitkeep` | Part of #14 |
165 +| G6 | Empty `wisdom.md` | No heuristics available for first reskill to build on | Seed with initial heuristics from analysis-spec patterns | Part of #14 |
166 +| G7 | Analysis prompt ignores learned state | Even if wisdom exists, it's not injected into analysis | Add `{{WISDOM_CONTENT}}` variable to `prompts/analyze-weekly.md` | **Yes — new issue** |
167 +
168 +### Serious Gaps (Learning will be shallow without these)
169 +
170 +| # | Gap | Impact | Proposed Fix | New Issue? |
171 +|---|-----|--------|--------------|------------|
172 +| G8 | No hindsight validation | Can't measure if past calls were right | Build `scripts/validate_predictions.py` that compares analysis claims to snapshot deltas | **Yes — new issue** |
173 +| G9 | No prediction registry format | Claims aren't machine-readable for later comparison | Define frontmatter or sidecar format for testable predictions | **Yes — new issue** |
174 +| G10 | Star snapshots empty | No ground-truth data for validation | Ensure crawler commits snapshots (workflow does commit `data/snapshots/` — crawler must produce them) | Bug in #6 or crawl schedule |
175 +| G11 | No quality trend tracking | Can't measure improvement over time | Add `scripts/track_quality_trend.py` reading `quality_score` from all `data/analyzed/` files | Part of #14 |
176 +| G12 | Reskill prompt too vague | "Assess what's working" is underspecified | Write structured reskill prompt in `prompts/reskill.md` with specific review criteria | Part of #14 |
177 +
178 +### Minor Gaps (Nice-to-have for deeper learning)
179 +
180 +| # | Gap | Impact | Proposed Fix | New Issue? |
181 +|---|-----|--------|--------------|------------|
182 +| G13 | No fork/contributor signals in snapshots | Star-only validation is one-dimensional | Extend crawler to capture fork_count and contributor_count in snapshots | Future enhancement |
183 +| G14 | No reskill PR review gate | Reskill could degrade squad state | Reskill outputs as PR (not direct commit) for human review | Design decision needed |
184 +| G15 | No rollback mechanism | Bad reskill can't be undone | Git history provides implicit rollback; add explicit `squad revert-reskill` | Future enhancement |
185 +
186 +---
187 +
188 +## 7. Recommendations
189 +
190 +### Immediate (must-do before claiming "SquadScope learns")
191 +
192 +1. **Implement Issue #15 — Run counter:**
193 + - Create `.squad/run-counter.txt` initialized to `0`
194 + - Add counter increment to the `crawl` job commit step
195 + - Include `.squad/run-counter.txt` in the git add paths
196 +
197 +2. **Implement Issue #14 — Reskill job:**
198 + - Add a `reskill` job in `crawl-and-publish.yml` (conditional on counter % 5 == 0)
199 + - Write a structured prompt at `prompts/reskill.md` that reads squad state, past analyses, and star snapshots
200 + - Create `.squad/skills/` and `.squad/reskill/` directories
201 + - Add commit step that pushes `.squad/` changes
202 +
203 +3. **Close the prompt feedback loop (new issue needed):**
204 + - Add `{{WISDOM_CONTENT}}` and `{{SKILLS_CONTENT}}` variables to `prompts/analyze-weekly.md`
205 + - The analysis fallback script must read and inject these
206 + - Without this, learning has no effect on future analysis quality
207 +
208 +4. **Seed `wisdom.md`:**
209 + - Extract initial heuristics from the analysis-spec's editorial dimensions
210 + - This gives the first reskill something to refine rather than starting from zero
211 +
212 +### Near-term (should do within 2 reskill cycles)
213 +
214 +5. **Build hindsight validation (new issue):**
215 + - Script that loads analysis from week N, loads star snapshots from week N+4
216 + - Scores whether "Signal" repos grew, "Noise" repos stalled
217 + - Produces a scorecard that feeds into the next reskill
218 +
219 +6. **Define prediction registry format (new issue):**
220 + - Frontmatter additions: `predictions: [{repo, direction, confidence}]`
221 + - Machine-readable claims enable automated scoring
222 +
223 +7. **Verify star snapshots are being produced:**
224 + - The crawler has snapshot logic but `data/snapshots/` is empty
225 + - Likely because no successful cron run has occurred yet
226 + - Validate on first manual `workflow_dispatch` run
227 +
228 +### Architecture constraints to respect:
229 +
230 +- Reskill MUST NOT modify `data/raw/` or `data/analyzed/` (immutability contract)
231 +- Reskill outputs should be PR-based if they modify prompts or specs (governance)
232 +- Counter must be atomic and race-safe under concurrency controls already in the workflow
233 +
234 +---
235 +
236 +## Conclusion
237 +
238 +SquadScope's learning differentiator is currently a **design document, not a system**. The architecture is well-thought-out (counter mechanism, squad state as context, reskill outputs), but the implementation gap is total: zero lines of reskill code exist in the workflow or scripts. Three new issues are needed beyond #14 and #15 to close the feedback loop completely. Until at minimum #14, #15, and the prompt injection gap (G7) are resolved, SquadScope does not learn — it merely remembers what humans and squad sessions manually append to history files.