| 1 | # Matrix Crawl Operator Runbook |
| 2 | |
| 3 | **Status:** Operational reference |
| 4 | **Audience:** Pipeline operators, on-call engineers |
| 5 | **Related issues:** #435, #436, #437, #438, #439 |
| 6 | |
| 7 | --- |
| 8 | |
| 9 | ## Overview |
| 10 | |
| 11 | This runbook covers how to trigger, monitor, and troubleshoot the SquadScope matrix crawl and map/reduce dry-run pipeline. The default crawl topology is monolithic (non-matrix). Matrix fan-out is opt-in and experimental. |
| 12 | |
| 13 | --- |
| 14 | |
| 15 | ## 1. Triggering a Crawl Run |
| 16 | |
| 17 | ### Scheduled (default) |
| 18 | |
| 19 | The `crawl-and-publish.yml` workflow runs automatically on Sundays at 11:53 UTC (`53 11 * * 0`). Treat this as a best-effort GitHub-hosted schedule, not an exact start minute; scheduled runs can start hours late on shared runners. For the supported mitigation ladder (manual trigger, external scheduler -> `workflow_dispatch`, optional self-hosted runners), see [`docs/operator-guide.md#schedule-latency-and-mitigation-ladder`](operator-guide.md#schedule-latency-and-mitigation-ladder). |
| 20 | |
| 21 | ### Manual dispatch |
| 22 | |
| 23 | Use the GitHub Actions UI or CLI: |
| 24 | |
| 25 | ```bash |
| 26 | # Normal monolithic crawl |
| 27 | gh workflow run crawl-and-publish.yml |
| 28 | |
| 29 | # Dry-run mode (no publishing, safe for experiments) |
| 30 | gh workflow run crawl-and-publish.yml \ |
| 31 | -f run_mode=dry-run |
| 32 | |
| 33 | # Map/reduce dry-run analysis path |
| 34 | gh workflow run crawl-and-publish.yml \ |
| 35 | -f run_mode=dry-run \ |
| 36 | -f analysis_path=map-reduce-dry-run |
| 37 | |
| 38 | # Force refresh all sources (ignore same-day cache) |
| 39 | gh workflow run crawl-and-publish.yml \ |
| 40 | -f source_refresh_policy=force-refresh |
| 41 | |
| 42 | # Restore a specific past week |
| 43 | gh workflow run crawl-and-publish.yml \ |
| 44 | -f run_mode=restore \ |
| 45 | -f rebuild_week=2026-W23 \ |
| 46 | -f source_run_id=26753498571 |
| 47 | ``` |
| 48 | |
| 49 | ### Run modes |
| 50 | |
| 51 | | Mode | Publishes? | Creates release? | Use case | |
| 52 | |------|:----------:|:----------------:|----------| |
| 53 | | `normal` | Yes | If gated | Weekly production run | |
| 54 | | `dry-run` | No | No | Testing changes, experiments | |
| 55 | | `candidate-only` | No | No | Map/reduce validation | |
| 56 | | `restore` | Yes | Optional | Rebuilding a past week | |
| 57 | | `force-replace` | Yes | Yes | Explicit re-publication | |
| 58 | |
| 59 | ### Analysis paths |
| 60 | |
| 61 | | Path | Description | Publishes? | |
| 62 | |------|-------------|:----------:| |
| 63 | | `single-pass` | Current monolithic AI analysis (default) | Yes | |
| 64 | | `map-reduce-dry-run` | Experimental map/reduce scaffolding | Never | |
| 65 | |
| 66 | --- |
| 67 | |
| 68 | ## 2. Monitoring a Run |
| 69 | |
| 70 | ### Key artifacts to check |
| 71 | |
| 72 | | Artifact | Location | Purpose | |
| 73 | |----------|----------|---------| |
| 74 | | `crawl-cache` | Actions artifact | GitHub API response cache | |
| 75 | | `raw-data` | Actions artifact (90 days) | Job transport, same-day reuse, and emergency recovery; not durable storage | |
| 76 | | Immutable raw store | `publish:data/raw-store/<week>/<source_run_id>/` | Durable source-bound payloads, hashes, and artifact provenance | |
| 77 | | `crawl-snapshots` | Actions artifact | Star/trending snapshots | |
| 78 | | Rerun mode summary | `data/diagnostics/rerun-mode.json` | Mode validation output | |
| 79 | | External news | `data/raw/{week}-external-news.json` | RSS crawl result | |
| 80 | | Map/reduce candidates | `data/candidates/map-reduce/` | Dry-run output (if enabled) | |
| 81 | |
| 82 | ### Metrics to watch (per #437) |
| 83 | |
| 84 | - **Crawl duration:** GitHub crawl p95 target < 6 minutes |
| 85 | - **API calls:** Typical range 440–460 per run |
| 86 | - **Search API remaining:** Should stay ≥ 24/30 after crawl |
| 87 | - **Secondary rate limits:** Target: 0 events per run |
| 88 | - **RSS fetch time:** Target < 5s total for all sources |
| 89 | - **Source success rate:** Target ≥ 4/5 sources |
| 90 | |
| 91 | ### Checking run status |
| 92 | |
| 93 | ```bash |
| 94 | # List recent workflow runs |
| 95 | gh run list --workflow=crawl-and-publish.yml --limit 5 |
| 96 | |
| 97 | # View a specific run |
| 98 | gh run view <run-id> |
| 99 | |
| 100 | # Download artifacts for inspection |
| 101 | gh run download <run-id> -n raw-data |
| 102 | ``` |
| 103 | |
| 104 | --- |
| 105 | |
| 106 | ## 3. Troubleshooting |
| 107 | |
| 108 | ### GitHub crawl failures |
| 109 | |
| 110 | | Symptom | Likely cause | Action | |
| 111 | |---------|-------------|--------| |
| 112 | | Secondary rate limit (HTTP 403 with `retry-after`) | Too many concurrent API calls | Check if shard experiment is running; reduce parallelism | |
| 113 | | Search API quota exhausted | Excessive search queries | Wait for quota reset (resets per-minute); check for duplicate queries | |
| 114 | | Crawl timeout (>10 min) | Network issues or API degradation | Retry; check [githubstatus.com](https://githubstatus.com) | |
| 115 | | Cache miss storm | Config/code change invalidated cache | Expected on first run after changes; subsequent runs will rebuild cache | |
| 116 | | Empty results | Token permission issue | Verify `GITHUB_TOKEN` has `contents: read` and is not expired | |
| 117 | |
| 118 | ### RSS/News crawl failures |
| 119 | |
| 120 | | Symptom | Likely cause | Action | |
| 121 | |---------|-------------|--------| |
| 122 | | Single source timeout | Upstream feed slow/down | Check source URL manually; feed will retry once by default | |
| 123 | | All sources failed | Network/DNS issue on runner | Check runner connectivity; retry the run | |
| 124 | | Schema validation failure | Feed format changed | Check `scripts/techcrunch_crawler.py` parsing logic against current feed | |
| 125 | | Deduplication anomaly | URL normalization issue | Check `GITHUB_URL_RE` and URL stripping logic | |
| 126 | |
| 127 | ### Fan-in validation failures |
| 128 | |
| 129 | | Symptom | Likely cause | Action | |
| 130 | |---------|-------------|--------| |
| 131 | | Schema version mismatch | Mixed artifact versions | Ensure all legs use same code SHA (check `code_sha` in run context) | |
| 132 | | Checksum mismatch | Non-deterministic serialization | Check for floating-point ordering or timestamp injection in articles | |
| 133 | | Window mismatch | Leg computed own time | Verify all legs receive shared run context, not local `date` calls | |
| 134 | | Missing required artifact | GitHub shard crashed | Check individual shard job logs; fix and re-run | |
| 135 | | Minimum-source policy failed | ≥3 RSS sources down | Verify upstream feeds; consider temporary source list override | |
| 136 | |
| 137 | ### Map/reduce dry-run failures |
| 138 | |
| 139 | | Symptom | Likely cause | Action | |
| 140 | |---------|-------------|--------| |
| 141 | | Missing raw JSON input | Crawl step didn't complete | Ensure crawl job succeeded before analysis | |
| 142 | | Mapper schema violation | Contract change | Compare mapper output against `MAP_SCHEMA` in `scripts/map_reduce_dry_run.py` | |
| 143 | | QA gate failure | Quality regression | Check `data/candidates/map-reduce/qa-report.json` for specific failures | |
| 144 | | Token budget exceeded | Input growth | Check preflight vs actual token counts; may need input slicing | |
| 145 | |
| 146 | --- |
| 147 | |
| 148 | ## 4. Operational Procedures |
| 149 | |
| 150 | ### Enabling RSS matrix mode (when triggers fire) |
| 151 | |
| 152 | Prerequisites (per PRD triggers): |
| 153 | - RSS p95 > 60 seconds, OR |
| 154 | - Source count > 10, OR |
| 155 | - Source needs independent credentials/isolation |
| 156 | |
| 157 | Steps: |
| 158 | 1. Create a feature branch |
| 159 | 2. Modify workflow to add matrix strategy with `fail-fast: false` |
| 160 | 3. Each leg runs one source, uploads per-source artifact |
| 161 | 4. Add fan-in job that downloads all, validates, and merges |
| 162 | 5. Test with `dry-run` mode first |
| 163 | 6. Monitor metrics for 3+ runs before enabling for production |
| 164 | |
| 165 | ### Running a GitHub shard experiment (#435) |
| 166 | |
| 167 | 1. Trigger with `run_mode=dry-run` and shard configuration |
| 168 | 2. Run baseline (monolithic) and shard variant on same week window |
| 169 | 3. Compare: wall-clock time, API calls, rate-limit events, output stability |
| 170 | 4. Document results in experiment report |
| 171 | 5. Acceptance criteria: ≥25% speedup, ≤10% API growth, 0 rate-limit regression |
| 172 | |
| 173 | ### Recovering from a failed run |
| 174 | |
| 175 | ```bash |
| 176 | # 1. Check what failed |
| 177 | gh run view <run-id> --log-failed |
| 178 | |
| 179 | # 2. If crawl cache is stale, force refresh |
| 180 | gh workflow run crawl-and-publish.yml \ |
| 181 | -f source_refresh_policy=force-refresh |
| 182 | |
| 183 | # 3. If a specific week needs rebuilding |
| 184 | gh workflow run crawl-and-publish.yml \ |
| 185 | -f run_mode=restore \ |
| 186 | -f rebuild_week=2026-W23 \ |
| 187 | -f source_run_id=26753498571 |
| 188 | |
| 189 | # 4. If analysis failed but crawl succeeded, re-run from artifacts |
| 190 | gh run rerun <run-id> --failed |
| 191 | ``` |
| 192 | |
| 193 | ### Validating fan-in locally |
| 194 | |
| 195 | ```bash |
| 196 | # Run the deterministic map/reduce dry-run locally |
| 197 | python scripts/map_reduce_dry_run.py \ |
| 198 | --raw-json data/raw/2026-W23.json \ |
| 199 | --output-dir data/candidates/map-reduce/ \ |
| 200 | --current-datetime "2026-06-05T17:42:56Z" \ |
| 201 | --run-id "local-test" |
| 202 | |
| 203 | # Verify canonical output is byte-stable |
| 204 | python scripts/map_reduce_dry_run.py \ |
| 205 | --raw-json data/raw/2026-W23.json \ |
| 206 | --output-dir /tmp/mr-verify/ \ |
| 207 | --current-datetime "2026-06-05T17:42:56Z" \ |
| 208 | --run-id "local-test" |
| 209 | |
| 210 | diff data/candidates/map-reduce/ /tmp/mr-verify/ |
| 211 | ``` |
| 212 | |
| 213 | --- |
| 214 | |
| 215 | ## 5. Escalation Path |
| 216 | |
| 217 | 1. **Self-serve:** Check this runbook and workflow logs |
| 218 | 2. **Team:** Tag the issue with `squad:bender` (crawler) or `squad:fry` (tests/QA) |
| 219 | 3. **Architecture:** Tag `squad:leela` for design decisions or contract changes |
| 220 | 4. **External:** GitHub API issues → check [githubstatus.com](https://githubstatus.com); RSS feed issues → check upstream provider status |
| 221 | |
| 222 | --- |
| 223 | |
| 224 | ## References |
| 225 | |
| 226 | - Workflow: [`.github/workflows/crawl-and-publish.yml`](../.github/workflows/crawl-and-publish.yml) |
| 227 | - Fan-in contracts: [`docs/matrix-crawl-fan-in-contracts.md`](matrix-crawl-fan-in-contracts.md) |
| 228 | - PRD: [`docs/processed/PRD-matrix-crawl-map-reduce-analysis.md`](processed/PRD-matrix-crawl-map-reduce-analysis.md) |
| 229 | - Operator guide: [`docs/operator-guide.md`](operator-guide.md) |
| 230 | - Pipeline validation: [`docs/pipeline-validation.md`](pipeline-validation.md) |