Final QA readiness review and reskill-cycle simulation (#53)
* docs: add final qa report Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add Pagefind and wire reskill workflow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot QA review: clarify reskill simulation status and add owners to blocking issues - Updated Test results summary to clarify that only prompt rendering was tested - Explicitly note that full reskill-cycle simulation remains incomplete (required for #24 AC) - Reorganized Known issues section with clear blocking vs non-blocking separation - Added owner assignments (Fry) to all blocking launch issues for actionable QA Fixes Copilot review comments on PR #53. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
May 18, 2026 at 17:38 UTC
f84a8ccf24c2e91e7422194f77708f8ed50bc2c4
4 files changed
+178
-16
.github/workflows/crawl-and-publish.yml
+47
-9
@@ -436,6 +436,9 @@ jobs:
436
- name: Build site
437
run: hugo --minify
438
439
+ - name: Build search index
440
+ run: npx pagefind --site public/
441
+
442
- name: Upload Pages artifact
443
uses: actions/upload-pages-artifact@v3
444
with:
@@ -542,6 +545,9 @@ jobs:
545
needs: [reskill-check]
546
if: needs.reskill-check.outputs.should_reskill == 'true'
547
runs-on: ubuntu-latest
548
+ permissions:
549
+ contents: write
550
+ models: read
551
552
steps:
553
- uses: actions/checkout@v4
@@ -549,21 +555,53 @@ jobs:
555
fetch-depth: 0
556
ref: ${{ github.event.repository.default_branch }}
557
552
- - name: Run placeholder reskill
558
+ - name: Set up Node
559
+ uses: actions/setup-node@v4
560
+ with:
561
+ node-version: '22'
562
+
563
+ - name: Set up Python
564
+ uses: actions/setup-python@v5
565
+ with:
566
+ python-version: '3.12'
567
+
568
+ - name: Install Copilot CLI
569
+ run: npm install -g @github/copilot
570
+
571
+ - name: Run reskill
572
env:
573
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
574
+ COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GH_TOKEN }}
575
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
576
run: |
577
set -euo pipefail
578
git config user.name "github-actions[bot]"
579
git config user.email "github-actions[bot]@users.noreply.github.com"
580
COUNTER=$(cat .squad/run-counter.txt 2>/dev/null || echo 0)
581
+ CURRENT_DATETIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
582
mkdir -p .squad/skills .squad/reskill
561
- echo "Reskill triggered at run #$COUNTER ($(date -Iseconds))" >> .squad/reskill/trigger-log.txt
562
- git add .squad/reskill/trigger-log.txt
563
- git diff --cached --quiet || {
564
- git commit -m "audit: reskill triggered at run #$COUNTER"
565
- git push origin "HEAD:$DEFAULT_BRANCH" || {
566
- echo "Warning: Failed to push trigger log to $DEFAULT_BRANCH, but continuing."
567
- }
583
+
584
+ if python3 scripts/reskill.py --current-datetime "$CURRENT_DATETIME"; then
585
+ echo "🔄 Reskill report generated for run #$COUNTER"
586
+ else
587
+ echo "Reskill script failed; writing placeholder trigger log."
588
+ echo "Reskill triggered at run #$COUNTER ($CURRENT_DATETIME)" >> .squad/reskill/trigger-log.txt
589
+ fi
590
+
591
+ if ! git status --short -- .squad | grep -q .; then
592
+ echo "No .squad changes to commit."
593
+ exit 0
594
+ fi
595
+
596
+ git stash push --include-untracked --message reskill-state -- .squad
597
+ git fetch origin "$DEFAULT_BRANCH"
598
+ git checkout -B "$DEFAULT_BRANCH" "origin/$DEFAULT_BRANCH"
599
+ git stash pop || {
600
+ echo "Failed to reapply .squad changes after syncing $DEFAULT_BRANCH."
601
+ exit 1
602
+ }
603
+ git add .squad/
604
+ git diff --cached --quiet || git commit -m "chore: reskill state update"
605
+ git push origin "HEAD:$DEFAULT_BRANCH" || {
606
+ echo "Warning: Failed to push .squad updates to $DEFAULT_BRANCH, but continuing."
607
}
569
- echo "🔄 Reskill placeholder ran at run #$COUNTER"
docs/qa-report.md
new
+105
@@ -0,0 +1,105 @@
1
+# SquadScope QA Report
2
+
3
+_Date:_ 2026-05-18
4
+_Issue:_ #24 — Final QA readiness review and reskill-cycle simulation
5
+_Agent:_ Fry
6
+
7
+## Test results summary
8
+
9
+- `PYTHONPATH=. pytest -q` → **38 passed** in 0.50s.
10
+- `python3 -m py_compile` succeeded for the pipeline scripts, including `scripts/reskill.py` and `scripts/track_quality.py`.
11
+- `python3 scripts/track_quality.py` produced a valid markdown trend report.
12
+- **Reskill cycle simulation**: Prompt rendering tested (`--print-prompt` flag validates placeholder resolution), but full reskill cycle execution (calling GitHub Models API and generating `.squad/reskill/2026-W21.md` report) remains incomplete. This is **required** for #24 acceptance criterion "A dry-run or simulation of the first reskill cycle is completed and reviewed."
13
+
14
+## Hugo build verification
15
+
16
+- Verified Hugo `v0.161.1` using `.tools/hugo-0.161.1/hugo`.
17
+- `hugo --minify` completed successfully.
18
+- Build emitted two theme/template deprecation warnings from PaperMod/Hugo integration:
19
+ - `.Language.LanguageDirection` → use `.Language.Direction`
20
+ - `.Language.LanguageCode` → use `.Language.Locale`
21
+
22
+## Workflow audit findings
23
+
24
+### `.github/workflows/crawl-and-publish.yml`
25
+
26
+- Job dependency chain is coherent: `crawl` → `analyze` → `generate` → `deploy`, with `notify` depending on published artifacts and `reskill-check` gating `reskill`.
27
+- Top-level permissions are broadly sufficient, and job-level overrides for analysis/deploy are present.
28
+- Concurrency control exists (`weekly-crawl` at workflow level, `pages` in deploy).
29
+- Secrets/vars references look valid for the intended design (`COPILOT_GH_TOKEN`, `GITHUB_TOKEN`, optional `WEBHOOK_URL`).
30
+
31
+**Blocking finding:** the `deploy` job only runs `hugo --minify`. On a clean build, that produces `public/search/index.html` and RSS output, but **does not produce `public/pagefind/` assets**. The search page template expects `pagefind/pagefind-ui.js`, so scheduled/manual publish runs can ship a broken search experience unless Pagefind is built explicitly in this workflow.
32
+
33
+**Readiness note:** the `reskill` job is still a placeholder logger. It does not invoke `scripts/reskill.py`, does not write `.squad/reskill/YYYY-WNN.md`, and does not use the quality trend/reskill prompt path that the repository now contains.
34
+
35
+### `.github/workflows/deploy-site.yml`
36
+
37
+- `build` → `deploy` dependency is correct.
38
+- Permissions and Pages deployment setup are sufficient.
39
+- Concurrency control is present (`pages`).
40
+- This workflow **does** build Pagefind after Hugo, which matches the search page contract.
41
+
42
+## Pipeline contract verification
43
+
44
+### Crawl → Analyze
45
+
46
+- `data/raw/2026-W21.json` is valid JSON and matches the expected top-level contract: `week`, `crawled_at`, `new_repos`, `trending_repos`, `signals`, `metadata`.
47
+- Sample counts: `209` new repos, `215` trending repos.
48
+- `docs/analysis-spec.md` and `prompts/analyze-weekly.md` are aligned on:
49
+ - required frontmatter keys,
50
+ - required section order,
51
+ - required `Signal` / `Noise` / `Gaps` subsections,
52
+ - the rule that missing `stars_gained` must be called out explicitly.
53
+- `data/analyzed/2026-W21-summary.md` satisfies the documented structure and includes `quality_score: 76`.
54
+
55
+### Analyze → Generate
56
+
57
+- `scripts/generate_content.py` successfully reads the analyzed summary and can regenerate the weekly page.
58
+- **Content drift finding:** the committed `content/weekly/2026/W21.md` is still the older manual dry-run page, not the current generator output from `data/analyzed/2026-W21-summary.md`.
59
+
60
+### Rollups
61
+
62
+- `scripts/generate_rollups.py` runs successfully and can regenerate monthly/yearly pages.
63
+- **Metric mismatch finding:** regenerated monthly frontmatter computed `total_repos_featured: 15`, while the weekly summary reports `repos_featured: 424`. The current implementation is effectively counting unique linked repos, not the editorial pass size implied by the field name.
64
+- Current committed monthly/yearly content exists and builds, but future rollup generations need review before production use.
65
+
66
+### Reskill readiness
67
+
68
+- `scripts/reskill.py`, `scripts/track_quality.py`, and `prompts/reskill.md` all exist and are syntactically valid.
69
+- Reskill prompt rendering successfully reads squad wisdom/skills/history-derived state and resolves placeholders.
70
+- `.squad/run-counter.txt` is currently `0`, so automatic reskill will not trigger until successful crawl runs increment the counter and reach a positive multiple of 5.
71
+
72
+## Content verification results
73
+
74
+- Weekly page exists: `content/weekly/2026/W21.md`
75
+- Monthly rollup exists: `content/monthly/2026/05.md`
76
+- Yearly rollup exists: `content/yearly/2026.md`
77
+- Search page exists and is configured: `content/search/_index.md` + `layouts/search/list.html`
78
+- RSS feed is generated by Hugo (`public/index.xml` observed after build)
79
+- Current committed rollup content uses Hugo-aware `relref` links.
80
+
81
+**Forward-looking note:** `scripts/generate_rollups.py` currently constructs literal `/weekly/...` and `/monthly/...` paths for new entries, so future generated rollup entries can regress away from Hugo-aware links even though the committed content currently uses `relref`.
82
+
83
+## Known issues / limitations
84
+
85
+### Blocking issues (must resolve before release)
86
+
87
+1. **Search asset pipeline incomplete** _(Owner: Fry)_
88
+ `crawl-and-publish.yml` deploys search pages without building fresh Pagefind assets on a clean runner. The search page template expects `pagefind/pagefind-ui.js`, so scheduled/manual publish runs can ship a broken search experience.
89
+
90
+2. **Reskill workflow not integrated** _(Owner: Fry)_
91
+ The `reskill` job in `crawl-and-publish.yml` is still a placeholder logger. It does not invoke `scripts/reskill.py`, does not write `.squad/reskill/YYYY-WNN.md`, and does not use the quality trend/reskill prompt path. Additionally, the first reskill-cycle simulation (required for #24 acceptance criterion) has not been run and reviewed.
92
+
93
+3. **Generated content drift** _(Owner: Fry)_
94
+ The committed `content/weekly/2026/W21.md` is stale relative to the current generator output from `data/analyzed/2026-W21-summary.md`. Future pipeline runs may produce inconsistencies.
95
+
96
+### Non-blocking follow-ups (post-launch)
97
+
98
+4. Rollup generation currently undercounts `total_repos_featured` and may emit hardcoded internal paths on new entries.
99
+5. Hugo build passes, but theme deprecation warnings should be cleaned up before a future Hugo upgrade removes those APIs.
100
+
101
+## Readiness verdict
102
+
103
+**FAIL**
104
+
105
+The codebase is close: tests are green, the analyzer contract is well defined, and the core scripts/builds run. But the production pipeline still has blocking readiness gaps in search deployment, reskill automation wiring, and generated-content consistency. I would not call v1 release-ready until those are fixed and re-verified.
docs/rollout-checklist.md
+9
-6
@@ -7,7 +7,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
7
- [ ] **Repository created**
8
- [ ] GitHub repository exists and is accessible
9
- [ ] Submodules initialized: `git submodule update --init --recursive`
10
- - [ ] Hugo version verified: `hugo version` (must be v0.146.0+)
10
+ - [ ] Hugo version verified: `hugo version` (must be v0.161.1 to match CI)
11
12
- [ ] **Copilot auth configured**
13
- [ ] Fine-grained PAT created with **Account → Copilot Requests** permission
@@ -26,6 +26,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
26
- [ ] Run `hugo server` locally and verify site loads at http://localhost:1313
27
- [ ] No Hugo build errors in console
28
- [ ] Run `hugo --minify` and verify `public/` directory created with HTML files
29
+ - [ ] Run `npx pagefind --site public/` and verify `public/pagefind/pagefind-ui.js` exists
30
31
- [ ] **First manual crawl successful**
32
- [ ] Trigger: `gh workflow run crawl-and-publish.yml -R OWNER/REPO`
@@ -37,7 +38,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
38
- [ ] Previous crawl completed
39
- [ ] Verify: `data/analyzed/YYYY-WNN-summary.md` exists and contains valid Markdown + YAML
40
- [ ] Check: YAML frontmatter includes `quality_score`, `title`, `date`, `categories`
40
- - [ ] Check: File contains required sections (`## Signal`, `## Noise`, `## Gaps`)
41
+ - [ ] Check: File contains `## Trend Analysis` with `### Signal` and `### Noise`, plus `## What's Missing` with `### Gaps`
42
43
- [ ] **Quality gate passed**
44
- [ ] Analysis quality_score ≥ 60 (check frontmatter)
@@ -48,6 +49,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
49
- [ ] Wait for full workflow to complete (crawl + analyze + generate + deploy)
50
- [ ] Monitor: **Actions → Crawl and Publish → Latest run → Deploy Pages** shows success
51
- [ ] Verify: GitHub Pages shows "Deployment successful"
52
+ - [ ] Verify: deployed search still works and the build includes fresh `pagefind/` assets
53
54
- [ ] **RSS feed accessible**
55
- [ ] Visit: `https://PAGES_URL/index.xml` (replace with your Pages domain)
@@ -73,7 +75,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
75
- [ ] **Run counter initialized**
76
- [ ] File exists: `.squad/run-counter.txt`
77
- [ ] Contains integer counter (e.g., `1`)
76
- - [ ] Will trigger reskill automatically every 5 runs
78
+ - [ ] Reskill only triggers on positive multiples of 5 (`5`, `10`, `15`, ...), not when the counter is `0`
79
80
## First Automated Run
81
@@ -92,7 +94,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
94
- [ ] Site updated with new week's content
95
96
- [ ] **RSS feed updated**
95
- - [ ] `https://PAGES_URL/feed/` now shows latest week's entry
97
+ - [ ] `https://PAGES_URL/index.xml` now shows the latest entry
98
- [ ] Feed readers can subscribe and receive new entries
99
100
## Post-Launch Monitoring
@@ -113,9 +115,10 @@ Use this checklist to verify your SquadScope instance is ready for production. C
115
116
- [ ] **Manual stage execution (debug only)**
117
- [ ] Crawl: `python3 scripts/crawl.py --as-of 2026-05-18`
116
- - [ ] Analyze (fallback): `python3 scripts/analyze_fallback.py --raw-json data/raw/YYYY-WNN.json --output /tmp/test-analysis.md --current-datetime 2026-05-18T16:00:00Z`
118
+ - [ ] Analyze (fallback): `python3 scripts/analyze_fallback.py --raw-json data/raw/YYYY-WNN.json --output ./scratch-analysis.md --current-datetime 2026-05-18T16:00:00Z`
119
- [ ] Gate: `python3 scripts/analysis_gate.py --analysis-file data/analyzed/YYYY-WNN-summary.md --raw-json data/raw/YYYY-WNN.json --current-datetime 2026-05-18T16:00:00Z`
120
- [ ] Generate: `python3 scripts/generate_content.py data/analyzed/YYYY-WNN-summary.md`
121
+ - [ ] Build search assets after Hugo: `npx pagefind --site public/`
122
123
- [ ] **Fallback analysis path works (Copilot unavailable scenario)**
124
- [ ] Comment out or rename `COPILOT_GH_TOKEN` secret temporarily
@@ -126,7 +129,7 @@ Use this checklist to verify your SquadScope instance is ready for production. C
129
- [ ] **Reskill cycle verified (after 5 runs)**
130
- [ ] Count automated runs (every Monday = 1 run)
131
- [ ] After 5 weeks, check for `.squad/reskill/YYYY-WNN.md` file
129
- - [ ] Verify reskill outputs analysis and recommendations
132
+ - [ ] Verify the workflow invokes the real reskill path (`scripts/reskill.py`) and outputs analysis + recommendations
133
134
## Sign-Off
135
tests/test_pipeline.py
+17
-1
@@ -185,16 +185,32 @@ class WorkflowConfigTests(unittest.TestCase):
185
self.assertIn("reskill=true", check_run)
186
self.assertIn("$GITHUB_OUTPUT", check_run)
187
188
- reskill_step = next((s for s in reskill["steps"] if s.get("name") == "Run placeholder reskill"), None)
188
+ install_step = next((s for s in reskill["steps"] if s.get("name") == "Install Copilot CLI"), None)
189
+ self.assertIsNotNone(install_step)
190
+ self.assertEqual(install_step["run"], "npm install -g @github/copilot")
191
+
192
+ reskill_step = next((s for s in reskill["steps"] if s.get("name") == "Run reskill"), None)
193
self.assertIsNotNone(reskill_step)
194
+ self.assertEqual(reskill_step["env"]["COPILOT_GITHUB_TOKEN"], "${{ secrets.COPILOT_GH_TOKEN }}")
195
reskill_run = reskill_step["run"]
196
+ self.assertIn("python3 scripts/reskill.py --current-datetime", reskill_run)
197
self.assertIn("mkdir -p .squad/skills .squad/reskill", reskill_run)
198
self.assertIn("trigger-log.txt", reskill_run)
199
+ self.assertIn("git add .squad/", reskill_run)
200
201
def test_generate_workflow_runs_rollups_and_commits_all_content(self) -> None:
202
workflow_path = Path(".github/workflows/crawl-and-publish.yml")
203
workflow = yaml.safe_load(workflow_path.read_text(encoding="utf-8"))
204
205
+ deploy_job = workflow["jobs"]["deploy"]
206
+ build_site_step = next((s for s in deploy_job["steps"] if s.get("name") == "Build site"), None)
207
+ self.assertIsNotNone(build_site_step)
208
+ self.assertEqual(build_site_step["run"], "hugo --minify")
209
+
210
+ pagefind_step = next((s for s in deploy_job["steps"] if s.get("name") == "Build search index"), None)
211
+ self.assertIsNotNone(pagefind_step)
212
+ self.assertEqual(pagefind_step["run"], "npx pagefind --site public/")
213
+
214
generate_job = workflow["jobs"]["generate"]
215
generate_rollups_step = next((s for s in generate_job["steps"] if s.get("name") == "Generate rollups"), None)
216
self.assertIsNotNone(generate_rollups_step)