1
+# SquadScope Operator Guide
2
+
3
+This guide covers everything needed to set up, operate, monitor, and troubleshoot SquadScope in production.
4
+
5
+## Prerequisites
6
+
7
+Before starting, ensure you have:
8
+
9
+1. **A GitHub repository** — Fork or create a copy of SquadScope
10
+2. **GitHub Pages enabled** — In repo settings, set source to "GitHub Actions"
11
+3. **GitHub Copilot subscription** — Required for AI-powered analysis (primary path)
12
+ - Copilot Pro ($20/month) or Copilot Team subscription
13
+ - Account with Copilot API access
14
+4. **Fine-grained Personal Access Token (PAT)**
15
+ - Scope: `github_pat_...` (not `ghp_...`)
16
+ - Permission: Account → Copilot Requests (at least)
17
+ - No expiration recommended (or set far future)
18
+5. **Basic CLI tools** — `gh` CLI installed (`brew install gh` on macOS, `apt install gh` on Linux)
19
+
20
+## Initial Setup
21
+
22
+### Step 1: Clone the repository
23
+
24
+```bash
25
+git clone https://github.com/YOUR_USERNAME/SquadScope.git
26
+cd SquadScope
27
+```
28
+
29
+### Step 2: Verify Hugo version locally
30
+
31
+Ensure your Hugo binary is v0.146.0 or newer:
32
+
33
+```bash
34
+hugo version
35
+```
36
+
37
+If needed, download the extended version:
38
+- macOS: `brew install hugo`
39
+- Linux: Download from [hugo releases](https://github.com/gohugoio/hugo/releases)
40
+- Windows: Download from releases or use Chocolatey
41
+
42
+### Step 3: Configure GitHub Copilot secret
43
+
44
+Create a fine-grained PAT on GitHub:
45
+
46
+1. Navigate to **Settings → Developer settings → Personal access tokens → Fine-grained tokens**
47
+2. Click **Generate new token**
48
+3. Configure:
49
+ - **Token name:** `SquadScope-Copilot-Token`
50
+ - **Expiration:** No expiration (or set to 1 year)
51
+ - **Repository access:** Select your SquadScope repository
52
+ - **Account permissions:** Scroll to **Copilot Requests** and select **Read and write**
53
+ - Click **Generate token**
54
+4. Copy the token (you won't see it again)
55
+
56
+Add the token as a repository secret:
57
+
58
+```bash
59
+# Using GitHub CLI
60
+gh secret set COPILOT_GH_TOKEN --body YOUR_PAT_HERE -R YOUR_USERNAME/SquadScope
61
+
62
+# Or manually in repo settings:
63
+# Settings → Secrets and variables → Actions → New repository secret
64
+# Name: COPILOT_GH_TOKEN
65
+# Value: github_pat_...
66
+```
67
+
68
+Verify the secret exists:
69
+
70
+```bash
71
+gh secret list -R YOUR_USERNAME/SquadScope
72
+```
73
+
74
+### Step 4: Enable GitHub Pages
75
+
76
+1. Navigate to repo **Settings → Pages**
77
+2. Set **Source** to "GitHub Actions"
78
+3. (Optional) Configure custom domain if desired
79
+4. Save
80
+
81
+### Step 5: Test local build
82
+
83
+Ensure the Hugo build works locally:
84
+
85
+```bash
86
+hugo server
87
+```
88
+
89
+Visit `http://localhost:1313` and verify the site loads. Press `Ctrl+C` to stop.
90
+
91
+For production build:
92
+
93
+```bash
94
+hugo --minify
95
+```
96
+
97
+This generates the `public/` directory with optimized static assets.
98
+
99
+## Running the Pipeline
100
+
101
+### Option A: Automatic scheduling (default)
102
+
103
+The pipeline runs automatically every **Monday at 08:00 UTC** via `.github/workflows/crawl-and-publish.yml`.
104
+
105
+You don't need to do anything. Go to your repo's **Actions** tab to monitor runs.
106
+
107
+### Option B: Manual trigger
108
+
109
+Run the workflow manually:
110
+
111
+```bash
112
+gh workflow run crawl-and-publish.yml -R YOUR_USERNAME/SquadScope
113
+```
114
+
115
+Or through the GitHub UI:
116
+1. Go to **Actions → Crawl and Publish**
117
+2. Click **Run workflow**
118
+3. Confirm
119
+
120
+The workflow takes ~2-3 minutes depending on GitHub API response times.
121
+
122
+### Option C: Run individual stages locally
123
+
124
+For debugging or testing, run stages separately:
125
+
126
+#### Crawl
127
+
128
+```bash
129
+python3 scripts/crawl.py --as-of 2026-05-18
130
+```
131
+
132
+Output: `data/raw/2026-W21.json`, `data/snapshots/2026-W21-stars.json`
133
+
134
+#### Analyze (Fallback — if Copilot unavailable)
135
+
136
+```bash
137
+python3 scripts/analyze_fallback.py \
138
+ --raw-json data/raw/2026-W21.json \
139
+ --output data/analyzed/2026-W21-summary.md \
140
+ --current-datetime 2026-05-18T16:00:00Z
141
+```
142
+
143
+Output: `data/analyzed/2026-W21-summary.md`
144
+
145
+#### Quality gate
146
+
147
+```bash
148
+python3 scripts/analysis_gate.py \
149
+ --analysis-file data/analyzed/2026-W21-summary.md \
150
+ --raw-json data/raw/2026-W21.json \
151
+ --current-datetime 2026-05-18T16:00:00Z
152
+```
153
+
154
+If the gate fails, it will exit with a non-zero code and log the reason (e.g., quality_score < 60).
155
+
156
+#### Generate
157
+
158
+```bash
159
+python3 scripts/generate_content.py data/analyzed/2026-W21-summary.md
160
+```
161
+
162
+Output: `content/weekly/2026/W21.md`
163
+
164
+#### Build and deploy
165
+
166
+```bash
167
+hugo --minify
168
+```
169
+
170
+Output: `public/` directory ready for GitHub Pages.
171
+
172
+## Monitoring the Cron Schedule
173
+
174
+### View recent runs
175
+
176
+```bash
177
+gh run list -R YOUR_USERNAME/SquadScope --workflow crawl-and-publish.yml --limit 10
178
+```
179
+
180
+### Check a specific run
181
+
182
+```bash
183
+gh run view RUN_ID -R YOUR_USERNAME/SquadScope
184
+```
185
+
186
+### Stream live logs
187
+
188
+```bash
189
+gh run view RUN_ID -R YOUR_USERNAME/SquadScope --log
190
+```
191
+
192
+### Monitor in the UI
193
+
194
+Navigate to **Actions → Crawl and Publish** in your repo. Green checkmarks = success, red X = failure.
195
+
196
+## Troubleshooting Common Failures
197
+
198
+### ❌ Copilot auth failure
199
+
200
+**Error message:** `COPILOT_GITHUB_TOKEN not configured` or `401 Unauthorized`
201
+
202
+**Cause:** The `COPILOT_GH_TOKEN` secret is missing, expired, or has insufficient permissions.
203
+
204
+**Fix:**
205
+1. Verify the secret exists: `gh secret list -R YOUR_USERNAME/SquadScope`
206
+2. Regenerate the PAT if expired (Settings → Developer settings → Personal access tokens)
207
+3. Confirm it has **Account → Copilot Requests** permission
208
+4. Update the secret: `gh secret set COPILOT_GH_TOKEN --body YOUR_NEW_PAT -R YOUR_USERNAME/SquadScope`
209
+5. Re-run the workflow
210
+
211
+**Fallback:** The workflow automatically falls back to GitHub Models API if Copilot fails. Check the workflow logs to see which path was used.
212
+
213
+### ❌ GitHub API rate limits exceeded
214
+
215
+**Error message:** `API rate limit exceeded (60/60)` or `secondary rate limit`
216
+
217
+**Cause:** Too many API calls in a short time window. GitHub's search endpoint is particularly strict.
218
+
219
+**Fix:**
220
+1. The crawler has built-in backoff logic. Wait 15 minutes and retry.
221
+2. For immediate relief, configure a `GITHUB_TOKEN` (built-in) in the workflow to increase rate limits from 60/hour to 5,000/hour.
222
+3. If using a fine-grained PAT, ensure it has minimal required permissions (reduces quota consumption).
223
+
224
+**Avoid:** Running crawl jobs back-to-back in rapid succession.
225
+
226
+### ❌ Copilot quota exhausted
227
+
228
+**Error message:** `Quota exhausted` or `429 Too Many Requests`
229
+
230
+**Cause:** You've used up your Copilot API requests for the billing cycle (typically thousands per month for Copilot Pro).
231
+
232
+**Fix:**
233
+1. Check your Copilot usage: https://github.com/settings/copilot
234
+2. If you hit the limit, wait for the next billing cycle or upgrade your plan
235
+3. The workflow will automatically fall back to GitHub Models API (lower quality, but functional)
236
+
237
+**Mitigation:** Copilot Pro includes generous quota. For automated pipelines, consider Copilot Team (more quota, better for organizations).
238
+
239
+### ❌ Hugo build fails
240
+
241
+**Error message:** `error: /content/weekly/... failed to parse date` or `theme not found`
242
+
243
+**Cause:**
244
+- Hugo version mismatch (too old)
245
+- Corrupted frontmatter in generated content
246
+
247
+**Fix:**
248
+1. Check Hugo version: `hugo version` (must be v0.146.0+)
249
+2. Check generated markdown in `content/weekly/` for valid YAML frontmatter
250
+3. Run locally: `hugo server` to see detailed error messages
251
+
252
+### ❌ GitHub Pages doesn't update
253
+
254
+**Error message:** Site shows old content or doesn't deploy
255
+
256
+**Cause:**
257
+- Pages source not set to "GitHub Actions"
258
+- Deployment workflow didn't complete successfully
259
+- Cache not cleared
260
+
261
+**Fix:**
262
+1. Verify Pages source: Repo **Settings → Pages → Source = "GitHub Actions"**
263
+2. Check the **Actions** tab for failed deploy jobs
264
+3. Clear browser cache (`Ctrl+Shift+Delete`) and refresh
265
+4. Force redeploy: Push a dummy commit to `main` or manually trigger `deploy-site.yml` workflow
266
+
267
+### ❌ Quality gate blocks publish
268
+
269
+**Error message:** `quality_score < 60` or `Missing required sections (Signal, Noise, Gaps)`
270
+
271
+**Cause:** The AI analysis didn't meet quality thresholds. This is a **feature, not a bug** — the gate prevents low-quality content from publishing.
272
+
273
+**Fix:**
274
+1. Check the analysis file: `cat data/analyzed/YYYY-WNN-summary.md`
275
+2. Review the quality_score in the YAML frontmatter
276
+3. If analysis is genuinely poor, this may indicate a problem with the raw data (crawler issue) or Copilot API issues
277
+4. **Do NOT bypass the gate.** Instead:
278
+ - Investigate why analysis quality was low
279
+ - Retry the workflow (Copilot may have had transient issues)
280
+ - If consistent, open an issue with the analyzed output for review
281
+
282
+### ❌ Reskill workflow doesn't trigger
283
+
284
+**Error message:** Reskill job skipped in the logs
285
+
286
+**Cause:** Reskill only runs every 5th pipeline execution. If you've only run 1-4 times, it won't trigger yet.
287
+
288
+**Fix:**
289
+1. Run the workflow 4 more times to trigger a reskill (or wait for 4 more weeks)
290
+2. Check `.squad/run-counter.txt` to see how many runs have executed
291
+3. To manually test reskill logic, run: `gh workflow run squad-heartbeat.yml -R YOUR_USERNAME/SquadScope` (separate workflow)
292
+
293
+## The Reskill Cycle
294
+
295
+Every 5th pipeline run, SquadScope enters a "reskill" phase where it reviews its own behavior and proposes improvements.
296
+
297
+### What happens
298
+
299
+1. **Copilot reads:**
300
+ - `.squad/agents/*/history.md` — Learnings from all agents
301
+ - `.squad/decisions.md` — Architectural decisions made
302
+ - `data/analyzed/` — Recent analysis outputs (quality trend)
303
+ - `.squad/run-counter.txt` — Run history
304
+
305
+2. **Analysis:**
306
+ - What patterns are working well?
307
+ - What should change in prompts or logic?
308
+ - Are there quality drift signals?
309
+
310
+3. **Output:**
311
+ - Recommendations written to `.squad/reskill/YYYY-WNN.md`
312
+ - Optional: PR proposed with prompt refinements (not auto-merged)
313
+
314
+### What to expect
315
+
316
+- **First reskill (run 5):** Observations and baseline recommendations
317
+- **Subsequent reskills:** Trend analysis ("quality is improving" vs. "signal/noise classification drifting")
318
+- **Prompts may be refined:** If reskill suggests prompt changes, review the PR and decide whether to merge
319
+
320
+### No manual action required
321
+
322
+Reskill runs automatically. You can review the outputs in `.squad/reskill/` directory, but the system works fine without human intervention.
323
+
324
+## Updating Configuration
325
+
326
+### Change the crawl schedule
327
+
328
+Edit `.github/workflows/crawl-and-publish.yml`:
329
+
330
+```yaml
331
+on:
332
+ schedule:
333
+ - cron: '0 8 * * 1' # Change to your desired time (UTC)
334
+```
335
+
336
+Times are in UTC. Use crontab.guru to generate your schedule.
337
+
338
+### Change the reskill interval
339
+
340
+Edit `.github/workflows/crawl-and-publish.yml` or `.squad/run-counter.txt`:
341
+
342
+```yaml
343
+# In workflow, modify the modulo check:
344
+if [ $((COUNTER % 5)) -eq 0 ]; then # Change 5 to desired interval
345
+```
346
+
347
+### Update the analyzer prompts
348
+
349
+Prompts live in `prompts/` directory. Edit and commit changes. They take effect on the next workflow run.
350
+
351
+### Customize quality gate thresholds
352
+
353
+Edit `scripts/analysis_gate.py`:
354
+
355
+```python
356
+MIN_QUALITY_SCORE = 60 # Change threshold here
357
+MIN_WORD_COUNT = 200
358
+```
359
+
360
+## Monitoring Site Health
361
+
362
+### Check RSS feed generation
363
+
364
+```bash
365
+curl https://YOUR_SITE/feed/
366
+```
367
+
368
+Should return valid XML with recent article entries.
369
+
370
+### Verify Hugo build output
371
+
372
+```bash
373
+# After running locally:
374
+ls public/
375
+hugo --minify && wc -l public/*.html
376
+```
377
+
378
+Should show HTML files for each content page.
379
+
380
+### Monitor GitHub API usage
381
+
382
+Check your GitHub API rate limits:
383
+
384
+```bash
385
+gh api rate_limit
386
+```
387
+
388
+Output shows `limit`, `remaining`, and `reset` time. If you're consistently hitting limits, consider:
389
+- Running less frequently
390
+- Optimizing the crawler queries
391
+- Using a GitHub App instead of PAT (higher limits)
392
+
393
+## Disaster Recovery
394
+
395
+### Revert a bad analysis
396
+
397
+If analysis was published but is clearly incorrect:
398
+
399
+1. Manually edit `data/analyzed/YYYY-WNN-summary.md` to correct it
400
+2. Commit and push
401
+3. Re-run `generate` and `deploy` stages (or full pipeline)
402
+4. The week's content will be regenerated with corrected data
403
+
404
+### Reset the run counter
405
+
406
+If you want reskill to trigger immediately:
407
+
408
+```bash
409
+echo "5" > .squad/run-counter.txt
410
+git add .squad/run-counter.txt
411
+git commit -m "Reset run counter to trigger reskill"
412
+git push
413
+```
414
+
415
+Next run will trigger reskill logic.
416
+
417
+### Restore from a previous week
418
+
419
+Content is immutable by design. If you need to restore:
420
+
421
+1. Check git history: `git log --oneline -- data/analyzed/YYYY-WNN-summary.md`
422
+2. Revert if necessary: `git revert COMMIT_HASH`
423
+3. Regenerate and deploy
424
+
425
+## Getting Help
426
+
427
+- **Logs:** Check **Actions** tab in GitHub UI for detailed workflow logs
428
+- **Common issues:** See troubleshooting section above
429
+- **Architecture questions:** See `.squad/decisions.md` for design rationale
430
+- **PRD:** See `docs/PRD.md` for feature requirements and future phases
431
+- **Issues:** Open an issue in the repository with logs and error messages
432
+
433
+## What's Next?
434
+
435
+Once SquadScope is running smoothly:
436
+
437
+1. **Monitor quality:** Review weekly analyses for patterns and trends
438
+2. **Iterate prompts:** Based on reskill recommendations, refine analysis quality
439
+3. **Extend sources:** Add additional data sources (HackerNews, Reddit, etc.) via MCP tools
440
+4. **Add notifications:** Configure GitHub Releases or webhook integrations
441
+5. **Topic channels:** Explore multi-topic feature (see `docs/PRD-topic-channels.md`)
442
+
443
+SquadScope is designed to improve itself. Trust the system, monitor the trends, and enjoy curated tech news delivered every week.