| 1 | # ADR: Matrix Crawl Fan-In Architecture |
| 2 | |
| 3 | **Status:** Accepted |
| 4 | **Date:** 2026-06-13 |
| 5 | **Decision makers:** Leela (Lead/Architect) |
| 6 | **Related issues:** #331, #333, #356, #435, #436, #437, #438, #439 |
| 7 | |
| 8 | --- |
| 9 | |
| 10 | ## Context |
| 11 | |
| 12 | SquadScope collects weekly GitHub repository signals and external news via RSS, then runs AI-powered analysis to produce trend summaries. As the system grows (more sources, more repos, longer analysis), the question arose: should we parallelize crawling with GitHub Actions matrix jobs? |
| 13 | |
| 14 | Investigation (documented in the [PRD](../processed/PRD-matrix-crawl-map-reduce-analysis.md)) revealed: |
| 15 | |
| 16 | 1. RSS collection is already fast (~1s for 5 feeds) — matrix overhead would exceed actual work time. |
| 17 | 2. GitHub API crawling is rate-limit-constrained, not compute-constrained — parallelism risks quota exhaustion without proportional speedup. |
| 18 | 3. Analysis duration (28+ minutes) and context growth (112k+ tokens) are the actual bottlenecks. |
| 19 | 4. Downstream consumers expect canonical artifact formats regardless of collection topology. |
| 20 | |
| 21 | --- |
| 22 | |
| 23 | ## Decision |
| 24 | |
| 25 | We adopt a **staged, evidence-gated architecture** with three key design choices: |
| 26 | |
| 27 | ### 1. Keep monolithic crawl as default; matrix is opt-in |
| 28 | |
| 29 | **Choice:** The default crawl topology remains monolithic (single GitHub crawl process + in-process RSS fetching). Matrix fan-out is enabled only when measurable triggers fire. |
| 30 | |
| 31 | **Alternatives considered:** |
| 32 | - **Always-on matrix:** Rejected. Adds runner setup cost, artifact I/O overhead, cache merge complexity, and rate-limit instability for no measured benefit at current scale. |
| 33 | - **Hybrid from day one:** Deferred. Premature complexity without evidence of need. |
| 34 | |
| 35 | **Triggers for RSS matrix:** |
| 36 | - RSS p95 > 60 seconds |
| 37 | - Source count > 10 |
| 38 | - Source requires independent credentials or isolation |
| 39 | |
| 40 | **Triggers for GitHub matrix:** |
| 41 | - Shard experiment proves ≥25% speedup |
| 42 | - No more than 10% API-call growth |
| 43 | - Zero secondary-rate-limit regression |
| 44 | |
| 45 | ### 2. Deterministic fan-in with strict contracts |
| 46 | |
| 47 | **Choice:** All matrix legs produce per-shard artifacts with schema validation, shared run context, and checksums. A fan-in job deterministically merges them into canonical payloads that are byte-stable for the same inputs. |
| 48 | |
| 49 | **Alternatives considered:** |
| 50 | - **Append-only merge (no ordering):** Rejected. Non-deterministic output breaks caching, diffing, and downstream idempotency. |
| 51 | - **Last-writer-wins:** Rejected. Data loss risk when shards overlap. |
| 52 | - **Event-stream merge:** Over-engineered for batch workflows. Deferred. |
| 53 | |
| 54 | **Key contract properties:** |
| 55 | - Deterministic ordering (repos by `full_name`, articles by `(source_id, url)`) |
| 56 | - Idempotent: same inputs → same outputs |
| 57 | - Fail-closed on required artifacts; graceful degradation on optional sources |
| 58 | - Schema-versioned with forward-compatible rejection |
| 59 | |
| 60 | ### 3. Map/reduce as analysis experiment, not crawl optimization |
| 61 | |
| 62 | **Choice:** Map/reduce is positioned as an analysis-quality tool (reducing LLM context, improving citations) — not as a crawl-speed mechanism. It runs in dry-run mode only until evidence validates quality parity. |
| 63 | |
| 64 | **Alternatives considered:** |
| 65 | - **Map/reduce for crawl parallelism:** Rejected. Crawl is I/O-bound (API rate limits), not compute-bound. Parallelism doesn't help. |
| 66 | - **Immediate production map/reduce:** Rejected. Must prove quality parity with single-pass analysis before promoting output. |
| 67 | - **Vector DB / embedding approach:** Non-goal for MVP. No external paid infrastructure. |
| 68 | |
| 69 | --- |
| 70 | |
| 71 | ## Consequences |
| 72 | |
| 73 | ### Positive |
| 74 | |
| 75 | - **No premature complexity:** Pipeline stays simple until evidence justifies change. |
| 76 | - **Safe experimentation:** Dry-run and candidate-only modes allow testing without risk. |
| 77 | - **Downstream stability:** Canonical artifact contracts isolate consumers from topology changes. |
| 78 | - **Measurable rollout:** Clear acceptance criteria prevent subjective "good enough" decisions. |
| 79 | - **Deterministic validation:** Byte-stable output enables automated regression testing. |
| 80 | |
| 81 | ### Negative |
| 82 | |
| 83 | - **Delayed parallelism:** If source count grows rapidly, we'll need to implement matrix mode reactively rather than having it pre-built. |
| 84 | - **Experiment overhead:** Shard experiments require dedicated runs comparing baseline vs. variant. |
| 85 | - **Contract maintenance:** Schema versioning adds development overhead for artifact format changes. |
| 86 | |
| 87 | ### Risks and mitigations |
| 88 | |
| 89 | | Risk | Mitigation | |
| 90 | |------|-----------| |
| 91 | | Triggers never fire, matrix code rots | Review triggers quarterly; remove dead code if unused for 6 months | |
| 92 | | Fan-in non-determinism edge cases | Automated QA gates (#438) with fixture-based determinism tests | |
| 93 | | Map/reduce quality never matches single-pass | Keep single-pass as default; map/reduce stays experimental until proven | |
| 94 | | Schema version proliferation | Strict one-version-active policy; old schemas rejected immediately | |
| 95 | |
| 96 | --- |
| 97 | |
| 98 | ## Implementation Roadmap |
| 99 | |
| 100 | | Phase | Issue | Status | Description | |
| 101 | |-------|-------|--------|-------------| |
| 102 | | 0 | #333 | In progress | Define fan-in validation path and run-context schema | |
| 103 | | 1 | #435 | Planned | GitHub shard experiment with guardrails | |
| 104 | | 1 | #436 | Planned | RSS per-source artifacts + deterministic merge | |
| 105 | | 1 | #437 | Planned | Observability metrics wiring | |
| 106 | | 2 | #438 | Planned | Automated QA gates | |
| 107 | | 2 | #439 | This PR | Documentation package (contracts, runbook, ADR) | |
| 108 | | 3 | — | Future | Evidence-based rollout decision (enable or archive) | |
| 109 | |
| 110 | --- |
| 111 | |
| 112 | ## References |
| 113 | |
| 114 | - PRD: [Matrix Crawl and Map/Reduce Analysis](../processed/PRD-matrix-crawl-map-reduce-analysis.md) |
| 115 | - Fan-in contracts: [docs/matrix-crawl-fan-in-contracts.md](../matrix-crawl-fan-in-contracts.md) |
| 116 | - Runbook: [docs/matrix-crawl-runbook.md](../matrix-crawl-runbook.md) |
| 117 | - Issue #356: Triage meta-issue for matrix crawl & map/reduce |
| 118 | - Issue #331: Define map/reduce analysis promotion path |
| 119 | - Issue #333: Define crawl matrix readiness and fan-in validation path |