| 1 | # SquadScope Learning System Audit |
| 2 | |
| 3 | > **⚠️ Archived / Partially Stale (as of 2026-06-13):** Some claims (e.g., no reskill job, no `.squad/skills/` directory, no run-counter support) no longer match the current repository. See the live workflows and `.squad/` directory for the current learning system state. |
| 4 | |
| 5 | **Author:** Leela (Lead/Architect) |
| 6 | **Date:** 2026-05-18T13:20:07.067+02:00 |
| 7 | **Scope:** End-to-end audit of the learning system — can SquadScope actually learn in production? |
| 8 | |
| 9 | --- |
| 10 | |
| 11 | ## Executive Summary |
| 12 | |
| 13 | **Verdict: The learning system will NOT work in production today.** The design is sound but the implementation is almost entirely absent. Issues #14 and #15 correctly describe what needs to be built, but zero code exists for the reskill cycle. The workflow has no reskill job, no run counter exists, no `.squad/skills/` directory has been created, `wisdom.md` is empty, and there is no hindsight validation mechanism. The learning loop is a design-only artifact. |
| 14 | |
| 15 | --- |
| 16 | |
| 17 | ## 1. Reskill Cycle Implementation Status |
| 18 | |
| 19 | ### Issues #14 and #15 Assessment |
| 20 | |
| 21 | | Issue | Title | Design Quality | Implementation | Status | |
| 22 | |-------|-------|---------------|----------------|--------| |
| 23 | | #14 | Implement reskill retrospective workflow and squad-state outputs | ✅ Good | ❌ Zero code | OPEN | |
| 24 | | #15 | Add run counter persistence and every-fifth-run reskill trigger | ✅ Good | ❌ Zero code | OPEN | |
| 25 | |
| 26 | ### Will the every-5th-run trigger work? |
| 27 | |
| 28 | **Design (from `.squad/decisions.md` Decision 6):** |
| 29 | ```bash |
| 30 | COUNTER=$(cat .squad/run-counter.txt) |
| 31 | if [ $((COUNTER % 5)) -eq 0 ]; then |
| 32 | # reskill invocation |
| 33 | fi |
| 34 | ``` |
| 35 | |
| 36 | **Problems with this design:** |
| 37 | |
| 38 | 1. **No run-counter.txt exists.** The file has never been created. First run would fail with `cat: .squad/run-counter.txt: No such file or directory`. |
| 39 | 2. **No increment logic shown.** The decision shows the *check* but not who increments the counter or when. If the counter increments before the check, run 5 triggers. If after, run 6 triggers. The ordering matters and isn't specified. |
| 40 | 3. **No atomicity guarantee.** The workflow uses stash/pop for commits but the counter update could race with concurrent `workflow_dispatch` triggers. |
| 41 | 4. **No reskill job in the workflow.** `crawl-and-publish.yml` has jobs: `crawl`, `analyze`, `generate`, `deploy`, `notify`. There is no `reskill` job. |
| 42 | 5. **Counter survives only if committed.** The workflow commits `data/raw`, `data/snapshots`, `data/analyzed`, and `content/weekly/` — but not `.squad/`. Counter changes would be lost. |
| 43 | |
| 44 | ### What's missing for #14 and #15: |
| 45 | |
| 46 | - A `reskill` job in `crawl-and-publish.yml` (or a separate workflow) |
| 47 | - Counter initialization (`echo 0 > .squad/run-counter.txt`) |
| 48 | - Counter increment step in the crawl or analyze job |
| 49 | - Commit step that includes `.squad/run-counter.txt` changes |
| 50 | - The actual Copilot CLI invocation for reskill analysis |
| 51 | - Output writing to `.squad/reskill/YYYY-WNN.md` |
| 52 | - Commit step for reskill outputs (`.squad/` directory changes) |
| 53 | |
| 54 | --- |
| 55 | |
| 56 | ## 2. Learning Inputs |
| 57 | |
| 58 | ### Current state of each input source: |
| 59 | |
| 60 | | Input | Exists? | Has Content? | Accessible in CI? | Notes | |
| 61 | |-------|---------|--------------|-------------------|-------| |
| 62 | | `.squad/agents/*/history.md` | ✅ Yes (7 agents) | ✅ Yes | ✅ Yes (checked out) | Contains real project learnings | |
| 63 | | `.squad/decisions.md` | ✅ Yes | ✅ Yes | ✅ Yes | Rich decision log | |
| 64 | | `.squad/identity/wisdom.md` | ✅ Yes | ❌ Empty (header only) | ✅ Yes | Zero accumulated wisdom | |
| 65 | | `.squad/skills/` | ❌ No | N/A | N/A | Directory never created | |
| 66 | | Past analysis outputs | ⚠️ Partial | Only if runs have occurred | ✅ Yes (`data/analyzed/`) | No historical comparison logic exists | |
| 67 | | Star snapshots | ⚠️ Empty | `data/snapshots/` exists but is empty | ✅ Yes | Crawler writes snapshots but none committed yet | |
| 68 | |
| 69 | ### Hindsight validation capability: |
| 70 | |
| 71 | **Does not exist.** There is no code or process that: |
| 72 | - Compares "what we said was important 5 weeks ago" with "what actually became important" |
| 73 | - Uses star snapshot deltas to validate past predictions |
| 74 | - Measures whether Signal/Noise/Gaps calls were accurate in retrospect |
| 75 | |
| 76 | The analysis spec mentions comparing to prior weeks (§ Context dimension), but this is only for the *current* analysis — it doesn't feed backward into learning. |
| 77 | |
| 78 | --- |
| 79 | |
| 80 | ## 3. Learning Outputs |
| 81 | |
| 82 | ### What should change after a reskill: |
| 83 | |
| 84 | | Output | Mechanism Defined? | Implementation? | Will It Persist? | |
| 85 | |--------|-------------------|-----------------|-----------------| |
| 86 | | Updated `wisdom.md` heuristics | ⚠️ Implied only | ❌ No code | ❌ No commit step | |
| 87 | | New/updated skills in `.squad/skills/` | ✅ Reskill SKILL.md template exists | ❌ No code | ❌ No commit step, no directory | |
| 88 | | Adjusted significance thresholds | ❌ Not designed | ❌ No code | N/A | |
| 89 | | Updated hype detection patterns | ❌ Not designed | ❌ No code | N/A | |
| 90 | | Revised gap analysis focus | ❌ Not designed | ❌ No code | N/A | |
| 91 | | Reskill report in `.squad/reskill/YYYY-WNN.md` | ✅ Yes (Decision 6) | ❌ No code | ❌ No commit step | |
| 92 | |
| 93 | ### Critical gap: No feedback into the analysis prompt |
| 94 | |
| 95 | Even if reskill produces updated wisdom or patterns, the `prompts/analyze-weekly.md` template has no variable or include that would inject learned heuristics. The analysis prompt is static. Learning outputs have no path back into the analysis pipeline. |
| 96 | |
| 97 | --- |
| 98 | |
| 99 | ## 4. Learning Feedback Loop |
| 100 | |
| 101 | ### Can the system measure improvement? |
| 102 | |
| 103 | **No.** There is no mechanism to: |
| 104 | |
| 105 | 1. **Record predictions with timestamps.** The analysis output exists (`data/analyzed/YYYY-WNN-summary.md`) but Signal/Noise/Gaps claims are not stored in a machine-readable format that enables later comparison. |
| 106 | |
| 107 | 2. **Compare predictions to outcomes.** Star snapshots (`data/snapshots/`) could provide ground truth (did "Signal" repos actually grow? did "Noise" repos fade?), but: |
| 108 | - The snapshots directory is currently empty |
| 109 | - No script compares week N predictions against week N+4 star deltas |
| 110 | - No "scorecard" format exists |
| 111 | |
| 112 | 3. **Attribute improvement to reskill changes.** Without a baseline quality metric tracked over time, there's no way to know if reskill actually improved anything. |
| 113 | |
| 114 | ### Is star snapshot data sufficient? |
| 115 | |
| 116 | **Partially.** Star snapshots can validate: |
| 117 | - ✅ "This repo is gaining momentum" (compare stars at week N vs N+4) |
| 118 | - ✅ "This is hype" (stars plateau or decline) |
| 119 | - ❌ "This gap matters" (absence can't be validated by stars alone) |
| 120 | - ❌ "This trend is durable" (needs signals beyond stars — commits, forks, adoption) |
| 121 | |
| 122 | ### Missing data collection for better learning: |
| 123 | |
| 124 | 1. **Prediction registry:** Machine-readable claims from each analysis (repo X will grow, theme Y is noise) with confidence scores |
| 125 | 2. **Outcome tracker:** Script that revisits predictions after N weeks using snapshot data |
| 126 | 3. **Quality trend log:** `quality_score` from each analysis plotted over time |
| 127 | 4. **External validation signals:** Fork counts, contributor growth, dependency adoption — richer than stars alone |
| 128 | |
| 129 | --- |
| 130 | |
| 131 | ## 5. Persistence |
| 132 | |
| 133 | ### Will learnings survive across sessions and workflow runs? |
| 134 | |
| 135 | | Question | Answer | Evidence | |
| 136 | |----------|--------|----------| |
| 137 | | Are `.squad/` files committed after reskill? | ❌ **No** | No commit step for `.squad/` exists in the workflow | |
| 138 | | Does the workflow have `contents: write`? | ✅ **Yes** | `analyze` job has `contents: write`; would need same for reskill job | |
| 139 | | Will Copilot CLI have `.squad/` state during reskill? | ✅ **Yes** (if checkout is full) | The workflow checks out with `fetch-depth: 0` — `.squad/` is in the repo | |
| 140 | | Is `run-counter.txt` persisted? | ❌ **No** | File doesn't exist; no commit step would save it | |
| 141 | | Are reskill outputs persisted? | ❌ **No** | `.squad/reskill/` directory doesn't exist; no commit step | |
| 142 | |
| 143 | ### The persistence chain is broken at every link: |
| 144 | |
| 145 | ``` |
| 146 | Run → Counter increment → [NOT COMMITTED] → Lost |
| 147 | Reskill → wisdom.md update → [NOT COMMITTED] → Lost |
| 148 | Reskill → skill extraction → [NO DIRECTORY] → Lost |
| 149 | Reskill → report → [NOT COMMITTED] → Lost |
| 150 | ``` |
| 151 | |
| 152 | **The workflow only commits:** `data/raw/`, `data/snapshots/`, `data/analyzed/`, `content/weekly/`. Squad state changes are invisible to git. |
| 153 | |
| 154 | --- |
| 155 | |
| 156 | ## 6. Gap Analysis |
| 157 | |
| 158 | ### Critical Gaps (Learning will not happen without these) |
| 159 | |
| 160 | | # | Gap | Impact | Proposed Fix | New Issue? | |
| 161 | |---|-----|--------|--------------|------------| |
| 162 | | G1 | No reskill job in workflow | Reskill never triggers | Add `reskill` job to `crawl-and-publish.yml` with counter check | Part of #15 | |
| 163 | | G2 | No `run-counter.txt` | Counter check fails on first run | Initialize file; add increment in crawl job commit step | Part of #15 | |
| 164 | | G3 | No `.squad/` commit step | All learning outputs lost between runs | Add commit step for `.squad/` after reskill | Part of #14 | |
| 165 | | G4 | No `.squad/skills/` directory | Skill extraction has nowhere to write | Create directory with `.gitkeep` | Part of #14 | |
| 166 | | G5 | No `.squad/reskill/` directory | Reskill reports have nowhere to go | Create directory with `.gitkeep` | Part of #14 | |
| 167 | | G6 | Empty `wisdom.md` | No heuristics available for first reskill to build on | Seed with initial heuristics from analysis-spec patterns | Part of #14 | |
| 168 | | G7 | Analysis prompt ignores learned state | Even if wisdom exists, it's not injected into analysis | Add `{{WISDOM_CONTENT}}` variable to `prompts/analyze-weekly.md` | **Yes — new issue** | |
| 169 | |
| 170 | ### Serious Gaps (Learning will be shallow without these) |
| 171 | |
| 172 | | # | Gap | Impact | Proposed Fix | New Issue? | |
| 173 | |---|-----|--------|--------------|------------| |
| 174 | | G8 | No hindsight validation | Can't measure if past calls were right | Build `scripts/validate_predictions.py` that compares analysis claims to snapshot deltas | **Yes — new issue** | |
| 175 | | G9 | No prediction registry format | Claims aren't machine-readable for later comparison | Define frontmatter or sidecar format for testable predictions | **Yes — new issue** | |
| 176 | | G10 | Star snapshots empty | No ground-truth data for validation | Ensure crawler commits snapshots (workflow does commit `data/snapshots/` — crawler must produce them) | Bug in #6 or crawl schedule | |
| 177 | | G11 | No quality trend tracking | Can't measure improvement over time | Add `scripts/track_quality_trend.py` reading `quality_score` from all `data/analyzed/` files | Part of #14 | |
| 178 | | G12 | Reskill prompt too vague | "Assess what's working" is underspecified | Write structured reskill prompt in `prompts/reskill.md` with specific review criteria | Part of #14 | |
| 179 | |
| 180 | ### Minor Gaps (Nice-to-have for deeper learning) |
| 181 | |
| 182 | | # | Gap | Impact | Proposed Fix | New Issue? | |
| 183 | |---|-----|--------|--------------|------------| |
| 184 | | G13 | No fork/contributor signals in snapshots | Star-only validation is one-dimensional | Extend crawler to capture fork_count and contributor_count in snapshots | Future enhancement | |
| 185 | | G14 | No reskill PR review gate | Reskill could degrade squad state | Reskill outputs as PR (not direct commit) for human review | Design decision needed | |
| 186 | | G15 | No rollback mechanism | Bad reskill can't be undone | Git history provides implicit rollback; add explicit `squad revert-reskill` | Future enhancement | |
| 187 | |
| 188 | --- |
| 189 | |
| 190 | ## 7. Recommendations |
| 191 | |
| 192 | ### Immediate (must-do before claiming "SquadScope learns") |
| 193 | |
| 194 | 1. **Implement Issue #15 — Run counter:** |
| 195 | - Create `.squad/run-counter.txt` initialized to `0` |
| 196 | - Add counter increment to the `crawl` job commit step |
| 197 | - Include `.squad/run-counter.txt` in the git add paths |
| 198 | |
| 199 | 2. **Implement Issue #14 — Reskill job:** |
| 200 | - Add a `reskill` job in `crawl-and-publish.yml` (conditional on counter % 5 == 0) |
| 201 | - Write a structured prompt at `prompts/reskill.md` that reads squad state, past analyses, and star snapshots |
| 202 | - Create `.squad/skills/` and `.squad/reskill/` directories |
| 203 | - Add commit step that pushes `.squad/` changes |
| 204 | |
| 205 | 3. **Close the prompt feedback loop (new issue needed):** |
| 206 | - Add `{{WISDOM_CONTENT}}` and `{{SKILLS_CONTENT}}` variables to `prompts/analyze-weekly.md` |
| 207 | - The analysis fallback script must read and inject these |
| 208 | - Without this, learning has no effect on future analysis quality |
| 209 | |
| 210 | 4. **Seed `wisdom.md`:** |
| 211 | - Extract initial heuristics from the analysis-spec's editorial dimensions |
| 212 | - This gives the first reskill something to refine rather than starting from zero |
| 213 | |
| 214 | ### Near-term (should do within 2 reskill cycles) |
| 215 | |
| 216 | 5. **Build hindsight validation (new issue):** |
| 217 | - Script that loads analysis from week N, loads star snapshots from week N+4 |
| 218 | - Scores whether "Signal" repos grew, "Noise" repos stalled |
| 219 | - Produces a scorecard that feeds into the next reskill |
| 220 | |
| 221 | 6. **Define prediction registry format (new issue):** |
| 222 | - Frontmatter additions: `predictions: [{repo, claim_type, direction, confidence}]` |
| 223 | - Machine-readable claims enable automated scoring |
| 224 | |
| 225 | 7. **Verify star snapshots are being produced:** |
| 226 | - The crawler has snapshot logic but `data/snapshots/` is empty |
| 227 | - Likely because no successful cron run has occurred yet |
| 228 | - Validate on first manual `workflow_dispatch` run |
| 229 | |
| 230 | ### Architecture constraints to respect: |
| 231 | |
| 232 | - Reskill MUST NOT modify `data/raw/` or `data/analyzed/` (immutability contract) |
| 233 | - Reskill outputs should be PR-based if they modify prompts or specs (governance) |
| 234 | - Counter must be atomic and race-safe under concurrency controls already in the workflow |
| 235 | |
| 236 | --- |
| 237 | |
| 238 | ## Conclusion |
| 239 | |
| 240 | SquadScope's learning differentiator is currently a **design document, not a system**. The architecture is well-thought-out (counter mechanism, squad state as context, reskill outputs), but the implementation gap is total: zero lines of reskill code exist in the workflow or scripts. Three new issues are needed beyond #14 and #15 to close the feedback loop completely. Until at minimum #14, #15, and the prompt injection gap (G7) are resolved, SquadScope does not learn — it merely remembers what humans and squad sessions manually append to history files. |