main
md 240 lines 13.6 KB
Rendered Raw
1 # SquadScope Learning System Audit
2
3 > **⚠️ Archived / Partially Stale (as of 2026-06-13):** Some claims (e.g., no reskill job, no `.squad/skills/` directory, no run-counter support) no longer match the current repository. See the live workflows and `.squad/` directory for the current learning system state.
4
5 **Author:** Leela (Lead/Architect)
6 **Date:** 2026-05-18T13:20:07.067+02:00
7 **Scope:** End-to-end audit of the learning system — can SquadScope actually learn in production?
8
9 ---
10
11 ## Executive Summary
12
13 **Verdict: The learning system will NOT work in production today.** The design is sound but the implementation is almost entirely absent. Issues #14 and #15 correctly describe what needs to be built, but zero code exists for the reskill cycle. The workflow has no reskill job, no run counter exists, no `.squad/skills/` directory has been created, `wisdom.md` is empty, and there is no hindsight validation mechanism. The learning loop is a design-only artifact.
14
15 ---
16
17 ## 1. Reskill Cycle Implementation Status
18
19 ### Issues #14 and #15 Assessment
20
21 | Issue | Title | Design Quality | Implementation | Status |
22 |-------|-------|---------------|----------------|--------|
23 | #14 | Implement reskill retrospective workflow and squad-state outputs | ✅ Good | ❌ Zero code | OPEN |
24 | #15 | Add run counter persistence and every-fifth-run reskill trigger | ✅ Good | ❌ Zero code | OPEN |
25
26 ### Will the every-5th-run trigger work?
27
28 **Design (from `.squad/decisions.md` Decision 6):**
29 ```bash
30 COUNTER=$(cat .squad/run-counter.txt)
31 if [ $((COUNTER % 5)) -eq 0 ]; then
32 # reskill invocation
33 fi
34 ```
35
36 **Problems with this design:**
37
38 1. **No run-counter.txt exists.** The file has never been created. First run would fail with `cat: .squad/run-counter.txt: No such file or directory`.
39 2. **No increment logic shown.** The decision shows the *check* but not who increments the counter or when. If the counter increments before the check, run 5 triggers. If after, run 6 triggers. The ordering matters and isn't specified.
40 3. **No atomicity guarantee.** The workflow uses stash/pop for commits but the counter update could race with concurrent `workflow_dispatch` triggers.
41 4. **No reskill job in the workflow.** `crawl-and-publish.yml` has jobs: `crawl`, `analyze`, `generate`, `deploy`, `notify`. There is no `reskill` job.
42 5. **Counter survives only if committed.** The workflow commits `data/raw`, `data/snapshots`, `data/analyzed`, and `content/weekly/` — but not `.squad/`. Counter changes would be lost.
43
44 ### What's missing for #14 and #15:
45
46 - A `reskill` job in `crawl-and-publish.yml` (or a separate workflow)
47 - Counter initialization (`echo 0 > .squad/run-counter.txt`)
48 - Counter increment step in the crawl or analyze job
49 - Commit step that includes `.squad/run-counter.txt` changes
50 - The actual Copilot CLI invocation for reskill analysis
51 - Output writing to `.squad/reskill/YYYY-WNN.md`
52 - Commit step for reskill outputs (`.squad/` directory changes)
53
54 ---
55
56 ## 2. Learning Inputs
57
58 ### Current state of each input source:
59
60 | Input | Exists? | Has Content? | Accessible in CI? | Notes |
61 |-------|---------|--------------|-------------------|-------|
62 | `.squad/agents/*/history.md` | ✅ Yes (7 agents) | ✅ Yes | ✅ Yes (checked out) | Contains real project learnings |
63 | `.squad/decisions.md` | ✅ Yes | ✅ Yes | ✅ Yes | Rich decision log |
64 | `.squad/identity/wisdom.md` | ✅ Yes | ❌ Empty (header only) | ✅ Yes | Zero accumulated wisdom |
65 | `.squad/skills/` | ❌ No | N/A | N/A | Directory never created |
66 | Past analysis outputs | ⚠️ Partial | Only if runs have occurred | ✅ Yes (`data/analyzed/`) | No historical comparison logic exists |
67 | Star snapshots | ⚠️ Empty | `data/snapshots/` exists but is empty | ✅ Yes | Crawler writes snapshots but none committed yet |
68
69 ### Hindsight validation capability:
70
71 **Does not exist.** There is no code or process that:
72 - Compares "what we said was important 5 weeks ago" with "what actually became important"
73 - Uses star snapshot deltas to validate past predictions
74 - Measures whether Signal/Noise/Gaps calls were accurate in retrospect
75
76 The analysis spec mentions comparing to prior weeks (§ Context dimension), but this is only for the *current* analysis — it doesn't feed backward into learning.
77
78 ---
79
80 ## 3. Learning Outputs
81
82 ### What should change after a reskill:
83
84 | Output | Mechanism Defined? | Implementation? | Will It Persist? |
85 |--------|-------------------|-----------------|-----------------|
86 | Updated `wisdom.md` heuristics | ⚠️ Implied only | ❌ No code | ❌ No commit step |
87 | New/updated skills in `.squad/skills/` | ✅ Reskill SKILL.md template exists | ❌ No code | ❌ No commit step, no directory |
88 | Adjusted significance thresholds | ❌ Not designed | ❌ No code | N/A |
89 | Updated hype detection patterns | ❌ Not designed | ❌ No code | N/A |
90 | Revised gap analysis focus | ❌ Not designed | ❌ No code | N/A |
91 | Reskill report in `.squad/reskill/YYYY-WNN.md` | ✅ Yes (Decision 6) | ❌ No code | ❌ No commit step |
92
93 ### Critical gap: No feedback into the analysis prompt
94
95 Even if reskill produces updated wisdom or patterns, the `prompts/analyze-weekly.md` template has no variable or include that would inject learned heuristics. The analysis prompt is static. Learning outputs have no path back into the analysis pipeline.
96
97 ---
98
99 ## 4. Learning Feedback Loop
100
101 ### Can the system measure improvement?
102
103 **No.** There is no mechanism to:
104
105 1. **Record predictions with timestamps.** The analysis output exists (`data/analyzed/YYYY-WNN-summary.md`) but Signal/Noise/Gaps claims are not stored in a machine-readable format that enables later comparison.
106
107 2. **Compare predictions to outcomes.** Star snapshots (`data/snapshots/`) could provide ground truth (did "Signal" repos actually grow? did "Noise" repos fade?), but:
108 - The snapshots directory is currently empty
109 - No script compares week N predictions against week N+4 star deltas
110 - No "scorecard" format exists
111
112 3. **Attribute improvement to reskill changes.** Without a baseline quality metric tracked over time, there's no way to know if reskill actually improved anything.
113
114 ### Is star snapshot data sufficient?
115
116 **Partially.** Star snapshots can validate:
117 - ✅ "This repo is gaining momentum" (compare stars at week N vs N+4)
118 - ✅ "This is hype" (stars plateau or decline)
119 - ❌ "This gap matters" (absence can't be validated by stars alone)
120 - ❌ "This trend is durable" (needs signals beyond stars — commits, forks, adoption)
121
122 ### Missing data collection for better learning:
123
124 1. **Prediction registry:** Machine-readable claims from each analysis (repo X will grow, theme Y is noise) with confidence scores
125 2. **Outcome tracker:** Script that revisits predictions after N weeks using snapshot data
126 3. **Quality trend log:** `quality_score` from each analysis plotted over time
127 4. **External validation signals:** Fork counts, contributor growth, dependency adoption — richer than stars alone
128
129 ---
130
131 ## 5. Persistence
132
133 ### Will learnings survive across sessions and workflow runs?
134
135 | Question | Answer | Evidence |
136 |----------|--------|----------|
137 | Are `.squad/` files committed after reskill? | ❌ **No** | No commit step for `.squad/` exists in the workflow |
138 | Does the workflow have `contents: write`? | ✅ **Yes** | `analyze` job has `contents: write`; would need same for reskill job |
139 | Will Copilot CLI have `.squad/` state during reskill? | ✅ **Yes** (if checkout is full) | The workflow checks out with `fetch-depth: 0``.squad/` is in the repo |
140 | Is `run-counter.txt` persisted? | ❌ **No** | File doesn't exist; no commit step would save it |
141 | Are reskill outputs persisted? | ❌ **No** | `.squad/reskill/` directory doesn't exist; no commit step |
142
143 ### The persistence chain is broken at every link:
144
145 ```
146 Run → Counter increment → [NOT COMMITTED] → Lost
147 Reskill → wisdom.md update → [NOT COMMITTED] → Lost
148 Reskill → skill extraction → [NO DIRECTORY] → Lost
149 Reskill → report → [NOT COMMITTED] → Lost
150 ```
151
152 **The workflow only commits:** `data/raw/`, `data/snapshots/`, `data/analyzed/`, `content/weekly/`. Squad state changes are invisible to git.
153
154 ---
155
156 ## 6. Gap Analysis
157
158 ### Critical Gaps (Learning will not happen without these)
159
160 | # | Gap | Impact | Proposed Fix | New Issue? |
161 |---|-----|--------|--------------|------------|
162 | G1 | No reskill job in workflow | Reskill never triggers | Add `reskill` job to `crawl-and-publish.yml` with counter check | Part of #15 |
163 | G2 | No `run-counter.txt` | Counter check fails on first run | Initialize file; add increment in crawl job commit step | Part of #15 |
164 | G3 | No `.squad/` commit step | All learning outputs lost between runs | Add commit step for `.squad/` after reskill | Part of #14 |
165 | G4 | No `.squad/skills/` directory | Skill extraction has nowhere to write | Create directory with `.gitkeep` | Part of #14 |
166 | G5 | No `.squad/reskill/` directory | Reskill reports have nowhere to go | Create directory with `.gitkeep` | Part of #14 |
167 | G6 | Empty `wisdom.md` | No heuristics available for first reskill to build on | Seed with initial heuristics from analysis-spec patterns | Part of #14 |
168 | G7 | Analysis prompt ignores learned state | Even if wisdom exists, it's not injected into analysis | Add `{{WISDOM_CONTENT}}` variable to `prompts/analyze-weekly.md` | **Yes — new issue** |
169
170 ### Serious Gaps (Learning will be shallow without these)
171
172 | # | Gap | Impact | Proposed Fix | New Issue? |
173 |---|-----|--------|--------------|------------|
174 | G8 | No hindsight validation | Can't measure if past calls were right | Build `scripts/validate_predictions.py` that compares analysis claims to snapshot deltas | **Yes — new issue** |
175 | G9 | No prediction registry format | Claims aren't machine-readable for later comparison | Define frontmatter or sidecar format for testable predictions | **Yes — new issue** |
176 | G10 | Star snapshots empty | No ground-truth data for validation | Ensure crawler commits snapshots (workflow does commit `data/snapshots/` — crawler must produce them) | Bug in #6 or crawl schedule |
177 | G11 | No quality trend tracking | Can't measure improvement over time | Add `scripts/track_quality_trend.py` reading `quality_score` from all `data/analyzed/` files | Part of #14 |
178 | G12 | Reskill prompt too vague | "Assess what's working" is underspecified | Write structured reskill prompt in `prompts/reskill.md` with specific review criteria | Part of #14 |
179
180 ### Minor Gaps (Nice-to-have for deeper learning)
181
182 | # | Gap | Impact | Proposed Fix | New Issue? |
183 |---|-----|--------|--------------|------------|
184 | G13 | No fork/contributor signals in snapshots | Star-only validation is one-dimensional | Extend crawler to capture fork_count and contributor_count in snapshots | Future enhancement |
185 | G14 | No reskill PR review gate | Reskill could degrade squad state | Reskill outputs as PR (not direct commit) for human review | Design decision needed |
186 | G15 | No rollback mechanism | Bad reskill can't be undone | Git history provides implicit rollback; add explicit `squad revert-reskill` | Future enhancement |
187
188 ---
189
190 ## 7. Recommendations
191
192 ### Immediate (must-do before claiming "SquadScope learns")
193
194 1. **Implement Issue #15 — Run counter:**
195 - Create `.squad/run-counter.txt` initialized to `0`
196 - Add counter increment to the `crawl` job commit step
197 - Include `.squad/run-counter.txt` in the git add paths
198
199 2. **Implement Issue #14 — Reskill job:**
200 - Add a `reskill` job in `crawl-and-publish.yml` (conditional on counter % 5 == 0)
201 - Write a structured prompt at `prompts/reskill.md` that reads squad state, past analyses, and star snapshots
202 - Create `.squad/skills/` and `.squad/reskill/` directories
203 - Add commit step that pushes `.squad/` changes
204
205 3. **Close the prompt feedback loop (new issue needed):**
206 - Add `{{WISDOM_CONTENT}}` and `{{SKILLS_CONTENT}}` variables to `prompts/analyze-weekly.md`
207 - The analysis fallback script must read and inject these
208 - Without this, learning has no effect on future analysis quality
209
210 4. **Seed `wisdom.md`:**
211 - Extract initial heuristics from the analysis-spec's editorial dimensions
212 - This gives the first reskill something to refine rather than starting from zero
213
214 ### Near-term (should do within 2 reskill cycles)
215
216 5. **Build hindsight validation (new issue):**
217 - Script that loads analysis from week N, loads star snapshots from week N+4
218 - Scores whether "Signal" repos grew, "Noise" repos stalled
219 - Produces a scorecard that feeds into the next reskill
220
221 6. **Define prediction registry format (new issue):**
222 - Frontmatter additions: `predictions: [{repo, claim_type, direction, confidence}]`
223 - Machine-readable claims enable automated scoring
224
225 7. **Verify star snapshots are being produced:**
226 - The crawler has snapshot logic but `data/snapshots/` is empty
227 - Likely because no successful cron run has occurred yet
228 - Validate on first manual `workflow_dispatch` run
229
230 ### Architecture constraints to respect:
231
232 - Reskill MUST NOT modify `data/raw/` or `data/analyzed/` (immutability contract)
233 - Reskill outputs should be PR-based if they modify prompts or specs (governance)
234 - Counter must be atomic and race-safe under concurrency controls already in the workflow
235
236 ---
237
238 ## Conclusion
239
240 SquadScope's learning differentiator is currently a **design document, not a system**. The architecture is well-thought-out (counter mechanism, squad state as context, reskill outputs), but the implementation gap is total: zero lines of reskill code exist in the workflow or scripts. Three new issues are needed beyond #14 and #15 to close the feedback loop completely. Until at minimum #14, #15, and the prompt injection gap (G7) are resolved, SquadScope does not learn — it merely remembers what humans and squad sessions manually append to history files.