| 1 | # Matrix Crawl Fan-In Contracts |
| 2 | |
| 3 | **Status:** Reference specification |
| 4 | **Related issues:** #333, #435, #436, #437, #438, #439 |
| 5 | **PRD:** [docs/processed/PRD-matrix-crawl-map-reduce-analysis.md](processed/PRD-matrix-crawl-map-reduce-analysis.md) |
| 6 | |
| 7 | --- |
| 8 | |
| 9 | ## Overview |
| 10 | |
| 11 | This document specifies the fan-in contracts for SquadScope's matrix crawl architecture. Fan-in is the merge step that combines per-shard or per-source artifacts into canonical downstream payloads. These contracts ensure deterministic, reproducible output regardless of whether collection ran as a monolithic process or as parallel matrix legs. |
| 12 | |
| 13 | --- |
| 14 | |
| 15 | ## Shared Run Context Schema |
| 16 | |
| 17 | Every crawl leg, fan-in validator, mapper, and reducer receives the same immutable run context. No matrix leg may compute its own time window from the local wall clock. |
| 18 | |
| 19 | ```json |
| 20 | { |
| 21 | "schema_version": "run_context_v1", |
| 22 | "run_id": "<week>-<sha256-prefix>", |
| 23 | "week": "2026-W23", |
| 24 | "since": "2026-06-01T00:00:00Z", |
| 25 | "until": "2026-06-08T00:00:00Z", |
| 26 | "source_config_checksum": "sha256:<hex>", |
| 27 | "topic_config_checksum": "sha256:<hex>", |
| 28 | "code_sha": "<sha256-of-relevant-source-files>", |
| 29 | "created_at": "2026-06-05T17:42:56Z" |
| 30 | } |
| 31 | ``` |
| 32 | |
| 33 | ### Required fields |
| 34 | |
| 35 | | Field | Type | Description | |
| 36 | |-------|------|-------------| |
| 37 | | `schema_version` | string | Must be `run_context_v1`. Fan-in rejects mismatches. | |
| 38 | | `run_id` | string | Stable identifier: `{week}-{sha256-prefix}`. | |
| 39 | | `week` | string | ISO week in `YYYY-WNN` format. | |
| 40 | | `since` | string (ISO-8601) | Inclusive start of the collection window. | |
| 41 | | `until` | string (ISO-8601) | Exclusive end of the collection window. | |
| 42 | | `source_config_checksum` | string | SHA-256 of the source configuration file. | |
| 43 | | `topic_config_checksum` | string | SHA-256 of `squadscope.topic.yml`. | |
| 44 | | `code_sha` | string | SHA-256 of relevant pipeline source files. | |
| 45 | | `created_at` | string (ISO-8601) | When the run context was generated. | |
| 46 | |
| 47 | --- |
| 48 | |
| 49 | ## Per-Source RSS Artifact Schema |
| 50 | |
| 51 | Each RSS matrix leg emits exactly one artifact per source: |
| 52 | |
| 53 | ```json |
| 54 | { |
| 55 | "schema_version": 2, |
| 56 | "source_id": "techcrunch", |
| 57 | "run_context": { "...shared run context..." }, |
| 58 | "status": "success | partial | failed", |
| 59 | "articles": [ |
| 60 | { |
| 61 | "url": "https://...", |
| 62 | "title": "...", |
| 63 | "published": "2026-06-03T10:00:00Z", |
| 64 | "source": "techcrunch", |
| 65 | "relevance_score": 0.85, |
| 66 | "github_urls": ["https://github.com/..."], |
| 67 | "entities": ["Company A", "Project B"] |
| 68 | } |
| 69 | ], |
| 70 | "metrics": { |
| 71 | "articles_fetched": 20, |
| 72 | "articles_relevant": 12, |
| 73 | "fetch_duration_ms": 1200, |
| 74 | "retries": 0 |
| 75 | }, |
| 76 | "error": null, |
| 77 | "checksum": "sha256:<hex-of-articles-array>" |
| 78 | } |
| 79 | ``` |
| 80 | |
| 81 | ### Validation rules |
| 82 | |
| 83 | - `schema_version` must equal `2` (current canonical version). |
| 84 | - `run_context.week` must match the fan-in job's expected week. |
| 85 | - `run_context.source_config_checksum` must match across all legs. |
| 86 | - `checksum` must match the SHA-256 of the serialized `articles` array (sorted by URL, deterministic JSON). |
| 87 | - `status: "failed"` artifacts carry no `articles` and must include an `error` object. |
| 88 | |
| 89 | --- |
| 90 | |
| 91 | ## GitHub Crawl Shard Artifact Schema |
| 92 | |
| 93 | Each GitHub shard leg (when enabled via experiment) emits: |
| 94 | |
| 95 | ```json |
| 96 | { |
| 97 | "schema_version": "github_shard_v1", |
| 98 | "shard_id": "new_repos | trending_repos | topic_primary | topic_secondary", |
| 99 | "run_context": { "...shared run context..." }, |
| 100 | "repositories": [ |
| 101 | { |
| 102 | "full_name": "owner/repo", |
| 103 | "stars": 1234, |
| 104 | "stars_gained": 45, |
| 105 | "description": "...", |
| 106 | "language": "Python", |
| 107 | "topics": ["ai", "ml"], |
| 108 | "created_at": "2026-01-15T...", |
| 109 | "pushed_at": "2026-06-02T..." |
| 110 | } |
| 111 | ], |
| 112 | "api_metrics": { |
| 113 | "calls_made": 112, |
| 114 | "cache_hits": 45, |
| 115 | "cache_misses": 67, |
| 116 | "secondary_rate_limit_events": 0, |
| 117 | "search_api_remaining": 24 |
| 118 | }, |
| 119 | "checksum": "sha256:<hex>" |
| 120 | } |
| 121 | ``` |
| 122 | |
| 123 | --- |
| 124 | |
| 125 | ## Fan-In Merge Rules |
| 126 | |
| 127 | ### Ordering guarantees |
| 128 | |
| 129 | 1. **Deterministic repository ordering:** Repositories are sorted by `full_name` (lexicographic, case-sensitive). Ties are not possible since `full_name` is unique. |
| 130 | 2. **Deterministic article ordering:** Articles are sorted by `(source_id, url)` tuple. |
| 131 | 3. **Stable output:** Same input artifacts → byte-identical canonical output (excluding the `merged_at` timestamp, which is excluded from checksum computation). |
| 132 | |
| 133 | ### Deduplication |
| 134 | |
| 135 | - **Repositories:** Deduplicated by `full_name`. If the same repo appears in multiple shards, the entry with the highest `stars_gained` is kept. |
| 136 | - **Articles:** Deduplicated by normalized URL (scheme + host + path, query params stripped). First occurrence by source priority order wins. |
| 137 | |
| 138 | ### Idempotency |
| 139 | |
| 140 | - Rerunning fan-in with the same input artifacts produces identical output. |
| 141 | - Fan-in does not mutate input artifacts. |
| 142 | - Reruns do not double-count articles, repos, retries, or restored stale artifacts. |
| 143 | - The `run_id` in the run context serves as the idempotency key; the same `run_id` always maps to the same canonical output given the same inputs. |
| 144 | |
| 145 | ### Failure policy |
| 146 | |
| 147 | | Scenario | Behavior | |
| 148 | |----------|----------| |
| 149 | | Required GitHub shard missing or invalid | **Fail closed.** No canonical output produced. Pipeline halts before analysis. | |
| 150 | | Optional RSS source failed | **Degrade.** Produce canonical output if minimum-source policy passes (≥3/5 sources succeed). Record explicit warnings. | |
| 151 | | Schema version mismatch | **Reject artifact.** Treat as missing. Apply required/optional rules above. | |
| 152 | | Window/checksum mismatch | **Reject artifact.** Log validation error with details. | |
| 153 | | All legs failed | **Fail closed.** Emit diagnostics artifact only. | |
| 154 | |
| 155 | ### Minimum-source policy |
| 156 | |
| 157 | For RSS fan-in, the merge proceeds if: |
| 158 | - At least 60% of configured sources report `status: "success"` or `status: "partial"`. |
| 159 | - All `status: "partial"` sources still have ≥1 valid article. |
| 160 | - The canonical output includes a `warnings` array documenting degraded sources. |
| 161 | |
| 162 | --- |
| 163 | |
| 164 | ## Canonical Output Schema |
| 165 | |
| 166 | Fan-in produces exactly two canonical artifacts consumed by downstream analysis: |
| 167 | |
| 168 | ### `data/raw/{week}.json` (GitHub) |
| 169 | |
| 170 | The existing monolithic format, produced identically whether from one crawl process or merged shards. |
| 171 | |
| 172 | ### `data/raw/{week}-external-news.json` (RSS/News) |
| 173 | |
| 174 | ```json |
| 175 | { |
| 176 | "schema_version": 2, |
| 177 | "run_context": { "..." }, |
| 178 | "sources": { |
| 179 | "techcrunch": { "status": "success", "articles_count": 12 }, |
| 180 | "nvidia_blog": { "status": "success", "articles_count": 5 }, |
| 181 | "huggingface": { "status": "failed", "error": "timeout" } |
| 182 | }, |
| 183 | "articles": [ "...merged, deduplicated, sorted..." ], |
| 184 | "metrics": { |
| 185 | "total_articles": 39, |
| 186 | "total_relevant": 23, |
| 187 | "sources_succeeded": 4, |
| 188 | "sources_failed": 1 |
| 189 | }, |
| 190 | "warnings": ["huggingface: fetch timeout after 15s"], |
| 191 | "merged_at": "2026-06-05T18:00:00Z", |
| 192 | "checksum": "sha256:<hex-of-articles>" |
| 193 | } |
| 194 | ``` |
| 195 | |
| 196 | --- |
| 197 | |
| 198 | ## Contract Versioning |
| 199 | |
| 200 | - Schema versions use `<domain>_v<N>` format (e.g., `run_context_v1`, `github_shard_v1`). |
| 201 | - Breaking changes increment the version number. |
| 202 | - Fan-in rejects artifacts with unknown or mismatched schema versions. |
| 203 | - The `schema_version` field is required in every artifact; omission is treated as a validation failure. |
| 204 | |
| 205 | --- |
| 206 | |
| 207 | ## Downstream Consumers |
| 208 | |
| 209 | The following components depend on canonical fan-in output and must NOT need to know whether collection was monolithic or matrix-based: |
| 210 | |
| 211 | - `scripts/correlate.py` — correlation analysis |
| 212 | - `scripts/render_press_context.py` — press context rendering |
| 213 | - `scripts/generate_content.py` — AI analysis |
| 214 | - `scripts/map_reduce_dry_run.py` — map/reduce dry-run |
| 215 | - `scripts/analysis_gate.py` — quality gate validation |
| 216 | - Publishing workflow steps |
| 217 | |
| 218 | --- |
| 219 | |
| 220 | ## References |
| 221 | |
| 222 | - PRD: [Matrix Crawl and Map/Reduce Analysis](processed/PRD-matrix-crawl-map-reduce-analysis.md) |
| 223 | - Issue #333: Define crawl matrix readiness and fan-in validation path |
| 224 | - Issue #435: Run GitHub crawl shard experiment |
| 225 | - Issue #436: Implement RSS matrix fan-in |
| 226 | - Issue #437: Wire observability metrics |
| 227 | - Issue #438: Automated QA gates |