| 1 | # SquadScope QA Report |
| 2 | |
| 3 | > **⚠️ Archived / Partially Stale (as of 2026-06-13):** Some claims in this report (e.g., deploy workflow not building Pagefind, reskill being a placeholder) no longer reflect the current implementation. Refer to the live workflows and `docs/pipeline-validation.md` for canonical operational truth. |
| 4 | |
| 5 | _Date:_ 2026-05-18 |
| 6 | _Issue:_ #24 — Final QA readiness review and reskill-cycle simulation |
| 7 | _Agent:_ Fry |
| 8 | |
| 9 | ## Test results summary |
| 10 | |
| 11 | - `PYTHONPATH=. pytest -q` → **38 passed** in 0.50s. |
| 12 | - `python3 -m py_compile` succeeded for the pipeline scripts, including `scripts/reskill.py` and `scripts/track_quality.py`. |
| 13 | - `python3 scripts/track_quality.py` produced a valid markdown trend report. |
| 14 | - **Reskill cycle simulation**: Prompt rendering tested (`--print-prompt` flag validates placeholder resolution), but full reskill cycle execution (calling GitHub Models API and generating `.squad/reskill/2026-W21.md` report) remains incomplete. This is **required** for #24 acceptance criterion "A dry-run or simulation of the first reskill cycle is completed and reviewed." |
| 15 | |
| 16 | ## Hugo build verification |
| 17 | |
| 18 | - Verified Hugo `v0.161.1` using `.tools/hugo-0.161.1/hugo`. |
| 19 | - `hugo --minify` completed successfully. |
| 20 | - Build emitted two theme/template deprecation warnings from PaperMod/Hugo integration: |
| 21 | - `.Language.LanguageDirection` → use `.Language.Direction` |
| 22 | - `.Language.LanguageCode` → use `.Language.Locale` |
| 23 | |
| 24 | ## Workflow audit findings |
| 25 | |
| 26 | ### `.github/workflows/crawl-and-publish.yml` |
| 27 | |
| 28 | - Job dependency chain is coherent: `crawl` → `analyze` → `generate` → `deploy`, with `notify` depending on published artifacts and `reskill-check` gating `reskill`. |
| 29 | - Top-level permissions are broadly sufficient, and job-level overrides for analysis/deploy are present. |
| 30 | - Concurrency control exists (`weekly-crawl` at workflow level, `pages` in deploy). |
| 31 | - Secrets/vars references look valid for the intended design (`COPILOT_GH_TOKEN`, `GITHUB_TOKEN`, optional `WEBHOOK_URL`). |
| 32 | |
| 33 | **Blocking finding:** the `deploy` job only runs `hugo --minify`. On a clean build, that produces `public/search/index.html` and RSS output, but **does not produce `public/pagefind/` assets**. The search page template expects `pagefind/pagefind-ui.js`, so scheduled/manual publish runs can ship a broken search experience unless Pagefind is built explicitly in this workflow. |
| 34 | |
| 35 | **Readiness note:** the `reskill` job is still a placeholder logger. It does not invoke `scripts/reskill.py`, does not write `.squad/reskill/YYYY-WNN.md`, and does not use the quality trend/reskill prompt path that the repository now contains. |
| 36 | |
| 37 | ### `.github/workflows/deploy-site.yml` |
| 38 | |
| 39 | - `build` → `deploy` dependency is correct. |
| 40 | - Permissions and Pages deployment setup are sufficient. |
| 41 | - Concurrency control is present (`pages`). |
| 42 | - This workflow **does** build Pagefind after Hugo, which matches the search page contract. |
| 43 | |
| 44 | ## Pipeline contract verification |
| 45 | |
| 46 | ### Crawl → Analyze |
| 47 | |
| 48 | - `data/raw/2026-W21.json` is valid JSON and matches the expected top-level contract: `week`, `crawled_at`, `new_repos`, `trending_repos`, `signals`, `metadata`. |
| 49 | - Sample counts: `209` new repos, `215` trending repos. |
| 50 | - `docs/analysis-spec.md` and `prompts/analyze-weekly.md` are aligned on: |
| 51 | - required frontmatter keys, |
| 52 | - required section order, |
| 53 | - required `Signal` / `Noise` / `Gaps` subsections, |
| 54 | - the rule that missing `stars_gained` must be called out explicitly. |
| 55 | - `data/analyzed/2026-W21-summary.md` satisfies the documented structure and includes `quality_score: 76`. |
| 56 | |
| 57 | ### Analyze → Generate |
| 58 | |
| 59 | - `scripts/generate_content.py` successfully reads the analyzed summary and can regenerate the weekly page. |
| 60 | - **Content drift finding:** the committed `content/weekly/2026/W21.md` is still the older manual dry-run page, not the current generator output from `data/analyzed/2026-W21-summary.md`. |
| 61 | |
| 62 | ### Rollups |
| 63 | |
| 64 | - `scripts/generate_rollups.py` runs successfully and can regenerate monthly/yearly pages. |
| 65 | - **Metric mismatch finding:** regenerated monthly frontmatter computed `total_repos_featured: 15`, while the weekly summary reports `repos_featured: 424`. The current implementation is effectively counting unique linked repos, not the editorial pass size implied by the field name. |
| 66 | - Current committed monthly/yearly content exists and builds, but future rollup generations need review before production use. |
| 67 | |
| 68 | ### Reskill readiness |
| 69 | |
| 70 | - `scripts/reskill.py`, `scripts/track_quality.py`, and `prompts/reskill.md` all exist and are syntactically valid. |
| 71 | - Reskill prompt rendering successfully reads squad wisdom/skills/history-derived state and resolves placeholders. |
| 72 | - `.squad/run-counter.txt` is currently `0`, so automatic reskill will not trigger until successful crawl runs increment the counter and reach a positive multiple of 5. |
| 73 | |
| 74 | ## Content verification results |
| 75 | |
| 76 | - Weekly page exists: `content/weekly/2026/W21.md` |
| 77 | - Monthly rollup exists: `content/monthly/2026/05.md` |
| 78 | - Yearly rollup exists: `content/yearly/2026.md` |
| 79 | - Search page exists and is configured: `content/search/_index.md` + `layouts/search/list.html` |
| 80 | - RSS feed is generated by Hugo (`public/index.xml` observed after build) |
| 81 | - Current committed rollup content uses Hugo-aware `relref` links. |
| 82 | |
| 83 | **Forward-looking note:** `scripts/generate_rollups.py` currently constructs literal `/weekly/...` and `/monthly/...` paths for new entries, so future generated rollup entries can regress away from Hugo-aware links even though the committed content currently uses `relref`. |
| 84 | |
| 85 | ## Known issues / limitations |
| 86 | |
| 87 | ### Blocking issues (must resolve before release) |
| 88 | |
| 89 | 1. **Search asset pipeline incomplete** _(Owner: Fry)_ |
| 90 | `crawl-and-publish.yml` deploys search pages without building fresh Pagefind assets on a clean runner. The search page template expects `pagefind/pagefind-ui.js`, so scheduled/manual publish runs can ship a broken search experience. |
| 91 | |
| 92 | 2. **Reskill workflow not integrated** _(Owner: Fry)_ |
| 93 | The `reskill` job in `crawl-and-publish.yml` is still a placeholder logger. It does not invoke `scripts/reskill.py`, does not write `.squad/reskill/YYYY-WNN.md`, and does not use the quality trend/reskill prompt path. Additionally, the first reskill-cycle simulation (required for #24 acceptance criterion) has not been run and reviewed. |
| 94 | |
| 95 | 3. **Generated content drift** _(Owner: Fry)_ |
| 96 | The committed `content/weekly/2026/W21.md` is stale relative to the current generator output from `data/analyzed/2026-W21-summary.md`. Future pipeline runs may produce inconsistencies. |
| 97 | |
| 98 | ### Non-blocking follow-ups (post-launch) |
| 99 | |
| 100 | 4. Rollup generation currently undercounts `total_repos_featured` and may emit hardcoded internal paths on new entries. |
| 101 | 5. Hugo build passes, but theme deprecation warnings should be cleaned up before a future Hugo upgrade removes those APIs. |
| 102 | |
| 103 | ## Readiness verdict |
| 104 | |
| 105 | **FAIL** |
| 106 | |
| 107 | The codebase is close: tests are green, the analyzer contract is well defined, and the core scripts/builds run. But the production pipeline still has blocking readiness gaps in search deployment, reskill automation wiring, and generated-content consistency. I would not call v1 release-ready until those are fixed and re-verified. |