| 1 | # PRD: Matrix Crawl and Map/Reduce Analysis for SquadScope |
| 2 | |
| 3 | **Author:** Leela (Lead/Architect) |
| 4 | **Date:** 2026-06-05 |
| 5 | **Status:** Phase 2 dry-run scaffolding implemented for issue #258 |
| 6 | **Type:** Product / Design Requirements Document |
| 7 | **Depends on:** docs/analysis-spec.md, docs/pipeline-validation.md, .squad/decisions.md, scripts/analysis_gate.py |
| 8 | **Inputs synthesized:** Bender matrix crawl findings, Farnsworth map/reduce analysis findings, Fry QA gates |
| 9 | |
| 10 | --- |
| 11 | |
| 12 | ## Executive Summary |
| 13 | |
| 14 | SquadScope should not add a GitHub Actions crawl matrix by default yet. The recent implementation avoided matrix fan-out because the measured RSS work is already fast (about one second for five feeds) and the dominant crawl cost is GitHub API collection, which is constrained by cache behavior, Search API quota, and secondary-rate-limit risk. A matrix is not automatically faster; at current scale it can add runner setup, artifact fan-in, cache merge complexity, and rate-limit instability without improving the bottleneck. |
| 15 | |
| 16 | This PRD recommends a staged architecture: |
| 17 | |
| 18 | 1. **Keep current crawl topology by default:** monolithic/cached GitHub crawl plus bounded in-process RSS fetching. |
| 19 | 2. **Make crawl artifacts matrix-ready:** introduce shared run context, schema validation, deterministic fan-in contracts, checksums, source status, and metrics. |
| 20 | 3. **Gate crawl matrix rollout by measurements:** RSS matrix only when source count/runtime/isolation triggers fire; GitHub matrix only after a shard experiment proves at least **25% crawl speedup**, no more than **10% API-call growth**, and no secondary-rate-limit regression. |
| 21 | 4. **Adopt map/reduce as an analysis experiment, not a crawl-speed fix:** use mapper claim ledgers and a reducer/final writer to shrink LLM context, preserve citations, dedupe claims, resolve contradictions, and satisfy the existing analysis spec/gate. |
| 22 | |
| 23 | The major product value is reliability and quality under growing evidence volume, not premature parallelism. Crawl fan-out should be evidence-triggered. Analysis map/reduce should be tested in dry-run/A-B mode before it can publish. |
| 24 | |
| 25 | A completed live workflow run, `27030646485`, strengthens this prioritization. The crawl path was healthy (213 new repos, 236 trending repos, 455 GitHub API calls, about 4m30s main crawler), and external news was healthy (5/5 sources, 39 articles, 23 relevant, zero failures/dedupe, schema v2 telemetry/checksum present). The dominant risk moved downstream: analysis took about 28m41s, failed three Copilot gates, could not use the GitHub Models `openai/gpt-4o` fallback because access was unavailable, and ultimately succeeded only through no-AI fallback. Final token ledger input was about 112.9k versus about 74.3k at preflight, confirming that analysis duration, context growth, and fallback behavior now matter more than crawl parallelism. |
| 26 | |
| 27 | --- |
| 28 | |
| 29 | ## Problem Statement |
| 30 | |
| 31 | The user asked why the team did not use a matrix to run the crawl faster, and whether a map/reduce technique could divide analysis into smaller parts to reduce LLM context. |
| 32 | |
| 33 | Two concerns are related but distinct: |
| 34 | |
| 35 | - **Crawl speed and reliability:** Can GitHub Actions matrix jobs fetch RSS or GitHub data faster than the current crawl? |
| 36 | - **Analysis context and quality:** Can the LLM analysis stage be decomposed into smaller, citation-preserving map outputs that a reducer combines into one coherent weekly summary? |
| 37 | |
| 38 | The recent multi-source news implementation increased RSS coverage from one TechCrunch feed to five external sources, but the measured RSS stage remained about one second. Meanwhile the GitHub crawler still took roughly 4.5-6 minutes and used shared cache/rate-limit behavior. The completed run `27030646485` showed this path healthy: about 4m30s main crawler time, 455 API calls, 213 new repos, 236 trending repos, and a fully successful external-news artifact with 5/5 sources, 39 articles, 23 relevant, zero failures/dedupe, and schema v2 telemetry/checksum. Splitting the wrong work would increase complexity without reducing critical path. However, analysis input size is growing: raw GitHub JSON and external-news/correlation evidence can exceed tens of thousands of token-estimate before prompt instructions, learned state, and previous summaries are added. In the same completed run, final input tokens reached about 112.9k versus about 74.3k at preflight. That creates attention dilution, citation drift, structural gate failures, weaker editorial synthesis, and slow/repeated fallback behavior. |
| 39 | |
| 40 | Therefore this PRD separates the decisions: |
| 41 | |
| 42 | - **Crawl matrix:** justified only by measured runtime, source count, failure isolation, or GitHub shard experiment evidence. |
| 43 | - **Analysis map/reduce:** justified as an experiment to reduce per-call context, improve citation discipline, and keep final output compliant with `docs/analysis-spec.md` and `scripts/analysis_gate.py`. |
| 44 | |
| 45 | --- |
| 46 | |
| 47 | ## Goals & Non-Goals |
| 48 | |
| 49 | ### Goals |
| 50 | |
| 51 | - **G1:** Explain why the recent crawl implementation did not use a matrix for speed. |
| 52 | - **G2:** Define measurable triggers and experiments for enabling RSS and GitHub crawl matrices. |
| 53 | - **G3:** Specify viable matrix/fan-out/fan-in designs for RSS and GitHub crawl while preserving canonical downstream artifacts. |
| 54 | - **G4:** Define an LLM analysis map/reduce architecture that divides work into smaller bounded contexts. |
| 55 | - **G5:** Define mapper and reducer contracts that preserve citations, dedupe claims, expose contradictions, and satisfy the existing analysis spec/gate. |
| 56 | - **G6:** Require observability for crawl legs, fan-in, mappers, reducers, fallbacks, token/cost behavior, and quality gates. |
| 57 | - **G7:** Provide rollout, QA gates, risks, mitigations, and acceptance criteria before any default-on change. |
| 58 | |
| 59 | ### Non-Goals |
| 60 | |
| 61 | - Implementing the matrix or map/reduce pipeline in this PRD. |
| 62 | - Replacing the current canonical raw artifact paths by default. |
| 63 | - Replacing `docs/analysis-spec.md` or weakening `scripts/analysis_gate.py`. |
| 64 | - Letting mapper outputs become publishable article prose. |
| 65 | - Adding paid services, vector databases, or embedding infrastructure for the MVP. |
| 66 | - Publishing map/reduce output before evidence-contract validation, analysis gate pass, and human/editorial comparison. |
| 67 | - Matrixing GitHub crawl merely because matrix jobs are available. |
| 68 | - Treating article volume alone as proof of success. |
| 69 | |
| 70 | --- |
| 71 | |
| 72 | ## Current Baseline / Why No Matrix Yet |
| 73 | |
| 74 | ### Evidence from recent runs |
| 75 | |
| 76 | Observed crawl timings and artifacts show: |
| 77 | |
| 78 | | Run | Shape | Crawl job | Main GitHub crawl | RSS/news step | Output | API observations | |
| 79 | | --- | --- | ---: | ---: | ---: | --- | --- | |
| 80 | | Old run `26753498571 / 78847225991` | single TechCrunch feed | ~6m23s | ~5m58s | sub-second | 196 new repos, 238 trending repos, 20 TechCrunch articles / 7 relevant | 447 API calls; Search API min remaining 24/30; core remained high | |
| 81 | | New run `27026348186 / 79767247136` | five external RSS feeds, in-process parallel | ~5m08s | ~4m47s | ~1s | 213 new repos, 236 trending repos, 54 articles / 27 relevant | 455 API calls; Search API min remaining 24/30; core remained high | |
| 82 | | Completed run `27030646485 / 79781846313` | five external RSS feeds | full workflow success | ~4m30s | ~1s | 213 new repos, 236 trending repos; 39 articles / 23 relevant; 5/5 sources; 0 failures/dedupe; schema v2 telemetry/checksum present | 455 API calls; crawl healthy; analysis became dominant risk | |
| 83 | |
| 84 | ### Live-run analysis signal |
| 85 | |
| 86 | Run `27030646485` did not reveal a crawler problem that needs a new crawler issue. Instead, it exposed analysis as the dominant risk: |
| 87 | |
| 88 | - Analysis duration was about **28m41s**, much longer than the crawl critical path. |
| 89 | - Three Copilot analysis gates failed before fallback. |
| 90 | - GitHub Models fallback could not use `openai/gpt-4o` because access was unavailable. |
| 91 | - The no-AI fallback ultimately passed, preserving workflow success but reducing confidence in AI-generated editorial quality. |
| 92 | - The final token ledger was about **112.9k input tokens** versus about **74.3k preflight**, showing substantial context expansion after preflight. |
| 93 | |
| 94 | This strengthens the map/reduce case: compaction, retry policy, gate-aware slicing, and provider fallback behavior should be prioritized before crawl matrix work. A focused analysis issue may be useful to track these fixes, but a new crawler issue is not warranted by this run. |
| 95 | |
| 96 | ### Why matrix was not used in the recent implementation |
| 97 | |
| 98 | The decision was architectural and evidence-based: |
| 99 | |
| 100 | 1. **RSS was not the bottleneck.** Five-source RSS collection completed in about one second. A per-source matrix would repeat checkout, Python setup, dependency installation, artifact upload/download, and merge logic. That overhead is larger than the current RSS work. |
| 101 | 2. **RSS is already parallelized in-process.** The newer RSS path uses bounded worker concurrency and writes one canonical external-news artifact. This is appropriate for a small number of I/O-bound feeds. |
| 102 | 3. **GitHub API crawl is rate/cache constrained.** The GitHub crawler uses shared cache, shared token/rate-limit visibility, one star snapshot, and deterministic output. Splitting queries before measuring shard behavior risks Search API quota pressure, secondary-rate-limit regressions, duplicate candidates, cache conflicts, and incorrect star-gain semantics. |
| 103 | 4. **Downstream contracts expect canonical artifacts.** `correlate.py`, `render_press_context.py`, analysis, rebuild mode, and publishing should not need to know whether collection was single-process or matrix-based. |
| 104 | 5. **Matrix fan-out is not automatically faster.** If the slow part is GitHub API wait/backoff or cache misses, parallel jobs can simply exhaust quota faster and force serialized backoff. |
| 105 | |
| 106 | ### Current decision baseline |
| 107 | |
| 108 | Default path remains: |
| 109 | |
| 110 | ```text |
| 111 | crawl-github (monolithic, cached, required) |
| 112 | + |
| 113 | crawl-external-news (bounded in-process RSS, optional/degraded) |
| 114 | -> canonical raw artifacts |
| 115 | -> correlation / press context |
| 116 | -> analysis |
| 117 | ``` |
| 118 | |
| 119 | Future work should make the artifact boundary matrix-ready without changing default topology prematurely. |
| 120 | |
| 121 | --- |
| 122 | |
| 123 | ## Proposed Architecture |
| 124 | |
| 125 | ### Overview |
| 126 | |
| 127 | ```text |
| 128 | shared run context |
| 129 | week / since / until / run_id |
| 130 | │ |
| 131 | ┌────────────────┴────────────────┐ |
| 132 | │ │ |
| 133 | required GitHub crawl optional RSS/news crawl |
| 134 | monolith by default; shard in-process by default; matrix |
| 135 | experiment only behind gate per source only behind gate |
| 136 | │ │ |
| 137 | └──────────────┬──────────────────┘ |
| 138 | ▼ |
| 139 | deterministic fan-in / validate |
| 140 | canonical data/raw/{week}.json + {week}-external-news.json |
| 141 | │ |
| 142 | ▼ |
| 143 | deterministic preflight / slicing |
| 144 | │ |
| 145 | ┌──────────────┴──────────────┐ |
| 146 | ▼ ▼ |
| 147 | analysis mapper(s) optional press/correlation mapper(s) |
| 148 | claim ledgers only claim ledgers only |
| 149 | └──────────────┬──────────────┘ |
| 150 | ▼ |
| 151 | reducer: dedupe / resolve / select |
| 152 | │ |
| 153 | ▼ |
| 154 | final writer: one coherent article |
| 155 | │ |
| 156 | ▼ |
| 157 | analysis_gate.py + evidence-contract validation |
| 158 | ``` |
| 159 | |
| 160 | ### Matrix Crawl |
| 161 | |
| 162 | #### Design principle |
| 163 | |
| 164 | Matrix collection is allowed only when it improves a measured property: crawl p95, failure isolation, retry granularity, source-specific quota/credential isolation, or rebuild/debuggability. Matrix jobs must never commit or feed analysis directly. They write validated artifacts, and a fan-in job emits the canonical payload consumed downstream. |
| 165 | |
| 166 | #### RSS matrix option |
| 167 | |
| 168 | **When default path stays in-process:** current five-feed RSS collection, normal p95 below threshold, homogeneous RSS/HTTP behavior, no source-specific credentials or quotas. |
| 169 | |
| 170 | **When RSS matrix is justified:** enable a per-source matrix only if one or more triggers fire: |
| 171 | |
| 172 | - RSS/news stage p95 exceeds **60 seconds** across recent runs. |
| 173 | - Configured external source count exceeds **10**. |
| 174 | - A source needs independent credentials, quota policy, network isolation, retry policy, parser/runtime dependency, or failure semantics. |
| 175 | - A single flaky/slow source repeatedly forces rerunning the whole crawl job. |
| 176 | - Product requires per-source downloadable diagnostics even when the aggregate crawl degrades. |
| 177 | |
| 178 | **RSS matrix design:** |
| 179 | |
| 180 | 1. A setup job computes a shared context: `run_id`, `week`, `since`, `until`, source config checksum, topic config checksum, and code SHA. |
| 181 | 2. Matrix legs run one source per leg with `strategy.fail-fast: false`. |
| 182 | 3. Each leg uploads exactly one artifact on `if: always()` containing either articles or a structured status/error payload. |
| 183 | 4. A fan-in job downloads all source artifacts, validates schemas/checksums/window consistency, dedupes article URLs deterministically, records per-source status, and writes canonical `data/raw/{week}-external-news.json`. |
| 184 | 5. Analysis consumes only the canonical merged artifact, never raw per-source artifacts. |
| 185 | |
| 186 | #### GitHub matrix option |
| 187 | |
| 188 | **When default path stays monolithic:** normal GitHub crawl p95 is acceptable, Search API remains tight, cache behavior is shared, star snapshot semantics remain global, or no shard experiment has proven benefit. |
| 189 | |
| 190 | **When GitHub matrix is justified:** enable only after a no-publish shard experiment proves all of: |
| 191 | |
| 192 | - At least **25% crawl wall-clock speedup** versus monolithic baseline, measured on comparable weekly windows. |
| 193 | - No more than **10% total GitHub API-call growth** versus baseline. |
| 194 | - No secondary-rate-limit regression: same or fewer secondary-limit events/backoff seconds. |
| 195 | - Search API quota remains above an agreed safety floor and is visible per shard. |
| 196 | - Same canonical raw output semantics: deterministic repo ordering, dedupe by `full_name`, stable star-gain/snapshot computation, and no downstream contract change. |
| 197 | |
| 198 | **GitHub matrix designs considered:** |
| 199 | |
| 200 | 1. **Per query/category shard** |
| 201 | - Shard by search query group: new repos, trending repos, topic primary queries, topic secondary queries. |
| 202 | - Fan-in dedupes by `full_name`, applies final filtering, merges API/cache metadata, computes star gains once, and emits canonical raw JSON/snapshot. |
| 203 | - Highest potential speedup, highest rate-limit/cache risk. |
| 204 | |
| 205 | 2. **Per candidate-processing shard after serialized search** |
| 206 | - A monolithic setup/search step gathers candidate repo names, then matrix legs fetch/normalize repository details or README metadata. |
| 207 | - Lower Search API risk because search remains centralized; possible speedup if core API details dominate. |
| 208 | - Fan-in still owns deterministic filtering and snapshot semantics. |
| 209 | |
| 210 | 3. **Hybrid staged fan-out/fan-in** |
| 211 | - Keep GitHub crawl monolithic initially. |
| 212 | - Introduce explicit artifact validation/fan-in first. |
| 213 | - Add correlation/analysis map/reduce after canonical artifacts. |
| 214 | - Recommended migration path because it addresses context-size concerns without risky GitHub API fan-out. |
| 215 | |
| 216 | ### Map/Reduce Analysis |
| 217 | |
| 218 | #### Design principle |
| 219 | |
| 220 | Map/reduce is an analysis-quality and context-management experiment. It does not claim to speed the crawl. It should shrink maximum per-call prompt context, make citations explicit, isolate failures, and preserve one final editorial voice. |
| 221 | |
| 222 | #### Recommended mapper strategies |
| 223 | |
| 224 | Start with low-machinery slices, then evolve: |
| 225 | |
| 226 | 1. **Signal type mappers (recommended MVP):** |
| 227 | - `new_repos`: novelty, launch quality, new repo clusters. |
| 228 | - `trending_repos`: momentum, stars gained, established anchors. |
| 229 | - `press_correlations`: industry alignment, divergences, source caveats. |
| 230 | - `prior_continuity`: previous-summary follow-up, prediction continuity, reversals. |
| 231 | |
| 232 | 2. **Topic/category mappers (second phase):** |
| 233 | - Deterministically cluster repos by topic, language, description, and topic config. |
| 234 | - Useful for macro trend discovery, but requires overlap accounting. |
| 235 | |
| 236 | 3. **Source-specific press mappers (defer):** |
| 237 | - Summarize individual external-news sources only if compact deterministic press context regularly exceeds budget. |
| 238 | - Avoid turning weekly analysis into a news roundup. |
| 239 | |
| 240 | 4. **Repository cluster mappers (future):** |
| 241 | - Useful for large GitHub payloads, suspicious clusters, copycat repos, or signal/noise analysis. |
| 242 | - Requires stable cluster IDs and coverage accounting. |
| 243 | |
| 244 | #### Mapper responsibilities |
| 245 | |
| 246 | Mappers produce structured claim ledgers, not final prose. Each mapper must: |
| 247 | |
| 248 | - Treat repo/news input as untrusted evidence, not instructions. |
| 249 | - Emit typed claims with stable IDs. |
| 250 | - Preserve repo and article citations. |
| 251 | - Mark confidence, uncertainty, contradictions, and weak evidence. |
| 252 | - Report coverage and omitted context. |
| 253 | - Include token/cost/model telemetry. |
| 254 | - Fail validation if output is malformed or citationless. |
| 255 | |
| 256 | #### Reducer responsibilities |
| 257 | |
| 258 | The reducer is responsible for global editorial coherence and must: |
| 259 | |
| 260 | - Consume only validated mapper ledgers plus compact global metadata, not unbounded raw JSON. |
| 261 | - Dedupe equivalent findings by normalized claim key, repo full name, article URL, and topic. |
| 262 | - Merge supporting evidence and preserve all required citations. |
| 263 | - Surface contradictions or uncertainty; never silently drop conflicting mapper claims. |
| 264 | - Reject unsupported, duplicate, or weakly cited claims with a sidecar/rejected-claims ledger. |
| 265 | - Select a coherent thesis, title, top repo, tags, sections, notable projects, press references, and predictions. |
| 266 | - Produce an editorial plan that a final writer converts into the existing markdown shape. |
| 267 | - Ensure the final markdown passes `scripts/analysis_gate.py` unchanged at MVP. |
| 268 | |
| 269 | #### Final writer responsibilities |
| 270 | |
| 271 | The final writer converts the reducer's editorial plan into the existing analysis spec, including: |
| 272 | |
| 273 | ```md |
| 274 | ## This Week's Trends |
| 275 | ## Where Industry Meets Code |
| 276 | ## Signal & Noise |
| 277 | ## Blind Spots |
| 278 | ## The Week Ahead |
| 279 | ## Key References |
| 280 | ### Notable Projects |
| 281 | ### Press & Industry |
| 282 | ``` |
| 283 | |
| 284 | The final writer is the only stage that writes reader-facing prose. This preserves one voice and avoids mapper-by-mapper seams. |
| 285 | |
| 286 | --- |
| 287 | |
| 288 | ## Fan-in Contracts |
| 289 | |
| 290 | ### Shared run context |
| 291 | |
| 292 | Every crawl leg, mapper, reducer, and validator must receive the same generated context: |
| 293 | |
| 294 | ```json |
| 295 | { |
| 296 | "schema_version": "run_context_v1", |
| 297 | "run_id": "2026-W23-<sha>", |
| 298 | "week": "2026-W23", |
| 299 | "since": "2026-06-01T00:00:00Z", |
| 300 | "until": "2026-06-08T00:00:00Z", |
| 301 | "source_config_checksum": "sha256:...", |
| 302 | "topic_config_checksum": "sha256:...", |
| 303 | "code_sha": "...", |
| 304 | "created_at": "2026-06-05T17:42:56Z" |
| 305 | } |
| 306 | ``` |
| 307 | |
| 308 | No matrix leg may compute its own week window from local wall clock. |
| 309 | |
| 310 | ### Fan-in rules |
| 311 | |
| 312 | - Required GitHub artifacts missing or invalid: fail closed before analysis. |
| 313 | - Optional RSS source failures: degrade only if minimum-source policy passes, and record explicit warnings/caveats. |
| 314 | - Fan-in runs on `if: always()` to publish diagnostics even when some legs fail. |
| 315 | - Same inputs must produce byte-stable canonical outputs, excluding documented timestamps from checksums. |
| 316 | - Cache metadata must include week window, config checksum, schema version, and stale-hit status. |
| 317 | - Reruns must not double-count articles, repos, retries, or restored stale artifacts. |
| 318 | |
| 319 | --- |
| 320 | |
| 321 | ## Data Contracts |
| 322 | |
| 323 | ### Crawl leg artifact |
| 324 | |
| 325 | ```json |
| 326 | { |
| 327 | "schema_version": "crawl_leg_v1", |
| 328 | "run_id": "2026-W23-<sha>", |
| 329 | "week": "2026-W23", |
| 330 | "since": "2026-06-01T00:00:00Z", |
| 331 | "until": "2026-06-08T00:00:00Z", |
| 332 | "leg_id": "rss:hugging-face-blog", |
| 333 | "source_type": "rss|github_search|github_repo_details", |
| 334 | "started_at": "ISO-8601", |
| 335 | "finished_at": "ISO-8601", |
| 336 | "duration_seconds": 1.23, |
| 337 | "status": "success|failed|partial|skipped", |
| 338 | "payload": {}, |
| 339 | "errors": [], |
| 340 | "metrics": { |
| 341 | "item_count": 9, |
| 342 | "relevant_count": 4, |
| 343 | "dedupe_count": 0, |
| 344 | "api_calls": 0, |
| 345 | "cache_hits": 0, |
| 346 | "stale_cache_hits": 0, |
| 347 | "retry_count": 0, |
| 348 | "rate_limit_remaining": null, |
| 349 | "rate_limit_resource": null |
| 350 | }, |
| 351 | "checksum": "sha256:..." |
| 352 | } |
| 353 | ``` |
| 354 | |
| 355 | ### RSS source payload |
| 356 | |
| 357 | ```json |
| 358 | { |
| 359 | "schema_version": "rss_source_v1", |
| 360 | "week": "2026-W23", |
| 361 | "source": "hugging_face_blog", |
| 362 | "source_config_checksum": "sha256:...", |
| 363 | "articles": [ |
| 364 | { |
| 365 | "title": "Article title", |
| 366 | "url": "https://example.com/article", |
| 367 | "source": "Hugging Face Blog", |
| 368 | "published_at": "2026-06-03T12:00:00Z", |
| 369 | "summary": "Short retained summary", |
| 370 | "categories": ["ai"], |
| 371 | "github_links": ["https://github.com/owner/repo"], |
| 372 | "relevance_score": 0.82 |
| 373 | } |
| 374 | ], |
| 375 | "source_status": { |
| 376 | "status": "success|failed|partial", |
| 377 | "attempts": 1, |
| 378 | "timeout_seconds": 15, |
| 379 | "duration_seconds": 0.7, |
| 380 | "error_class": null, |
| 381 | "error_message": null |
| 382 | }, |
| 383 | "artifact_checksum": "sha256:..." |
| 384 | } |
| 385 | ``` |
| 386 | |
| 387 | Fan-in emits canonical `data/raw/{week}-external-news.json` with stable metadata: `schema_version`, `sources_requested`, `sources_succeeded`, `sources_failed`, `sources_with_articles`, `dedupe_count`, `errors`, `source_status`, and checksum. |
| 388 | |
| 389 | ### GitHub shard artifact |
| 390 | |
| 391 | ```json |
| 392 | { |
| 393 | "schema_version": "github_shard_v1", |
| 394 | "week": "2026-W23", |
| 395 | "crawl_window": {"since": "ISO-8601", "until": "ISO-8601"}, |
| 396 | "shard_id": "github:new-repos:q1", |
| 397 | "query": "created:... stars:...", |
| 398 | "query_type": "new|trending|topic_primary|topic_secondary|repo_details", |
| 399 | "repos": [ |
| 400 | { |
| 401 | "full_name": "owner/repo", |
| 402 | "url": "https://github.com/owner/repo", |
| 403 | "stars": 123, |
| 404 | "stars_gained": null, |
| 405 | "topics": ["ai"], |
| 406 | "language": "Python", |
| 407 | "pushed_at": "ISO-8601", |
| 408 | "created_at": "ISO-8601", |
| 409 | "fork": false, |
| 410 | "template": false |
| 411 | } |
| 412 | ], |
| 413 | "api_calls_used": 12, |
| 414 | "cache_hits": 4, |
| 415 | "stale_cache_hits": 0, |
| 416 | "rate_limit": { |
| 417 | "resource": "search", |
| 418 | "limit": 30, |
| 419 | "remaining": 24, |
| 420 | "reset": "ISO-8601" |
| 421 | }, |
| 422 | "partial_failures": [], |
| 423 | "artifact_checksum": "sha256:..." |
| 424 | } |
| 425 | ``` |
| 426 | |
| 427 | Fan-in remains responsible for dedupe, final ordering, significance filtering, cache metadata reconciliation, star snapshot/delta computation, and canonical `data/raw/{week}.json` shape. |
| 428 | |
| 429 | ### Analysis mapper output |
| 430 | |
| 431 | ```json |
| 432 | { |
| 433 | "schema_version": "analysis_map_v1", |
| 434 | "run_id": "2026-W23-<sha>", |
| 435 | "week": "2026-W23", |
| 436 | "shard_id": "signal-type:new-repos", |
| 437 | "slice": { |
| 438 | "strategy": "signal_type|topic|source|repo_cluster", |
| 439 | "input_refs": ["data/raw/2026-W23.json#new_repos[0:50]"], |
| 440 | "input_token_estimate": 12000, |
| 441 | "repo_count": 50, |
| 442 | "article_count": 0 |
| 443 | }, |
| 444 | "coverage": { |
| 445 | "repo_ids_seen": ["owner/repo"], |
| 446 | "article_urls_seen": [], |
| 447 | "excluded_reason_counts": {"low_relevance": 3} |
| 448 | }, |
| 449 | "findings": [ |
| 450 | { |
| 451 | "claim_id": "stable-claim-id", |
| 452 | "claim": "A concise evidence-bound claim.", |
| 453 | "category": "trend|signal|noise|gap|press_correlation|press_divergence|continuity", |
| 454 | "source_type": "github|news|mixed|prior_summary", |
| 455 | "evidence_refs": [ |
| 456 | { |
| 457 | "type": "repo", |
| 458 | "ref": "owner/repo", |
| 459 | "url": "https://github.com/owner/repo", |
| 460 | "role": "anchor|supporting|counterexample", |
| 461 | "evidence_note": "Why this supports or weakens the claim" |
| 462 | } |
| 463 | ], |
| 464 | "repo_full_name": "owner/repo", |
| 465 | "news_url": null, |
| 466 | "confidence": 0.72, |
| 467 | "contra_refs": [], |
| 468 | "uncertainties": ["stars_gained unavailable"], |
| 469 | "quality_flags": ["needs_reducer_review"] |
| 470 | } |
| 471 | ], |
| 472 | "citations": [ |
| 473 | {"type": "repo", "url": "https://github.com/owner/repo", "title": "owner/repo"} |
| 474 | ], |
| 475 | "reference_candidates": { |
| 476 | "notable_projects": ["owner/repo"], |
| 477 | "press_articles": [] |
| 478 | }, |
| 479 | "token_estimate": 1800, |
| 480 | "model": "copilot|github-models|none", |
| 481 | "status": "success|failed|partial", |
| 482 | "errors": [] |
| 483 | } |
| 484 | ``` |
| 485 | |
| 486 | ### Reducer input |
| 487 | |
| 488 | ```json |
| 489 | { |
| 490 | "schema_version": "analysis_reduce_input_v1", |
| 491 | "run_id": "2026-W23-<sha>", |
| 492 | "week": "2026-W23", |
| 493 | "global_totals": { |
| 494 | "repos_featured": 449, |
| 495 | "stars_tracked": 123456, |
| 496 | "new_repo_count": 213, |
| 497 | "trending_repo_count": 236 |
| 498 | }, |
| 499 | "source_coverage": { |
| 500 | "sources_requested": ["techcrunch", "github_blog"], |
| 501 | "sources_succeeded": ["techcrunch"], |
| 502 | "sources_failed": ["github_blog"] |
| 503 | }, |
| 504 | "mapper_outputs": ["analysis_map_v1 objects"], |
| 505 | "validation_summary": { |
| 506 | "malformed_maps": 0, |
| 507 | "missing_required_citations": 0, |
| 508 | "week_mismatches": 0 |
| 509 | } |
| 510 | } |
| 511 | ``` |
| 512 | |
| 513 | ### Reducer output / editorial plan |
| 514 | |
| 515 | ```json |
| 516 | { |
| 517 | "schema_version": "analysis_editorial_plan_v1", |
| 518 | "title": "Final headline candidate", |
| 519 | "summary": "One-sentence thesis", |
| 520 | "top_repo": "owner/repo", |
| 521 | "tags": ["ai", "developer-tools"], |
| 522 | "selected_claims": [ |
| 523 | { |
| 524 | "claim_id": "reducer-claim-id", |
| 525 | "section": "This Week's Trends|Where Industry Meets Code|Signal & Noise|Blind Spots|The Week Ahead", |
| 526 | "merged_from": ["mapper-claim-id-1", "mapper-claim-id-2"], |
| 527 | "normalized_claim_key": "topic:agent-runtime-observability", |
| 528 | "citation_bindings": { |
| 529 | "repos": ["owner/repo"], |
| 530 | "articles": ["https://example.com/article"] |
| 531 | }, |
| 532 | "confidence": 0.81, |
| 533 | "rationale": "Why this claim survived reduce" |
| 534 | } |
| 535 | ], |
| 536 | "key_references": { |
| 537 | "notable_projects": ["owner/repo"], |
| 538 | "press_articles": ["https://example.com/article"] |
| 539 | }, |
| 540 | "rejected_claims": [ |
| 541 | {"claim_id": "mapper-claim-id-3", "reason": "duplicate|unsupported|contradicted|weak_citation"} |
| 542 | ], |
| 543 | "contradictions": [ |
| 544 | { |
| 545 | "claim_ids": ["a", "b"], |
| 546 | "resolution": "surface_as_uncertainty|prefer_claim|reject_both", |
| 547 | "rationale": "Evidence comparison" |
| 548 | } |
| 549 | ], |
| 550 | "quality_notes": ["One source failed; press claims caveated"] |
| 551 | } |
| 552 | ``` |
| 553 | |
| 554 | --- |
| 555 | |
| 556 | ## Observability and Metrics |
| 557 | |
| 558 | ### Crawl metrics |
| 559 | |
| 560 | Per crawl leg: |
| 561 | |
| 562 | - `leg_id`, source name/type, status, start/end/duration. |
| 563 | - Item count, relevant count, dedupe count, artifact size, checksum. |
| 564 | - Cache hit/stale-hit count. |
| 565 | - API calls, rate-limit limit/remaining/reset/resource. |
| 566 | - Retry count, timeout count, error class/message. |
| 567 | |
| 568 | Aggregate crawl: |
| 569 | |
| 570 | - Required/optional leg counts. |
| 571 | - Failed/skipped/degraded leg counts. |
| 572 | - Merged artifact size/checksum. |
| 573 | - Total API calls and per-resource rate-limit status. |
| 574 | - Cache hit ratio and stale-cache usage. |
| 575 | - Fan-in validation result and deterministic checksum. |
| 576 | |
| 577 | ### Analysis metrics |
| 578 | |
| 579 | Per mapper: |
| 580 | |
| 581 | - Shard ID, input refs/checksums, input token estimate, output token estimate. |
| 582 | - Model/provider, duration, retries, status. |
| 583 | - Finding count, citation count, malformed citation count. |
| 584 | - Schema validation result and coverage counts. |
| 585 | |
| 586 | Reducer/final writer: |
| 587 | |
| 588 | - Input mapper count, failed/skipped mapper count. |
| 589 | - Duplicate claim count and contradiction count. |
| 590 | - Rejected/merged claim counts. |
| 591 | - Final prompt/output token estimates. |
| 592 | - Model/provider, duration, retry count. |
| 593 | - `analysis_gate.py` result and evidence-contract validation result. |
| 594 | |
| 595 | Pipeline path: |
| 596 | |
| 597 | - Selected path: `single-pass`, `map-reduce`, `github-models`, `no-ai`. |
| 598 | - Fallback reason, provider/model attempted, access/permission failures, and retry count. |
| 599 | - Copilot gate attempts/failures, GitHub Models fallback result, and no-AI fallback result. |
| 600 | - Preflight token estimate versus final token ledger, including unexplained growth. |
| 601 | - Analysis duration by attempt and total analysis wall-clock. |
| 602 | - Final publish eligibility. |
| 603 | - Human comparison score during A/B period. |
| 604 | |
| 605 | ### Thresholds to alert or block |
| 606 | |
| 607 | - GitHub matrix experiment exceeds **10% API-call growth**. |
| 608 | - Secondary-rate-limit events/backoff increase versus baseline. |
| 609 | - RSS p95 exceeds **60 seconds**. |
| 610 | - Source count exceeds **10**. |
| 611 | - Single analysis prompt exceeds configured token budget. |
| 612 | - Final token ledger exceeds preflight by more than an agreed tolerance. |
| 613 | - Analysis duration or retry count exceeds configured budget. |
| 614 | - Copilot/GitHub Models gates repeatedly fail and no-AI fallback becomes the only passing path. |
| 615 | - Mapper/reducer total cost exceeds budget. |
| 616 | - Final reducer output loses required citations or fails `analysis_gate.py`. |
| 617 | |
| 618 | --- |
| 619 | |
| 620 | ## Rollout Plan |
| 621 | |
| 622 | ### Phase 0 — Baseline and contracts |
| 623 | |
| 624 | - Record p50/p95 durations for GitHub crawl, RSS/news crawl, fan-in/correlation, press context rendering, and analysis across at least 5-10 runs. |
| 625 | - Define JSON schemas for crawl legs, RSS source artifacts, GitHub shards, mapper outputs, reducer input, and editorial plan. |
| 626 | - Add validators and fixture tests for deterministic merge and analysis contract validation. |
| 627 | - Keep current workflow behavior unchanged. |
| 628 | |
| 629 | ### Phase 1 — Fan-in dry-run |
| 630 | |
| 631 | - Add a non-publishing fan-in validator that accepts current canonical artifacts and fixture per-source artifacts. |
| 632 | - Verify canonical outputs are byte-stable for same inputs. |
| 633 | - Add failure fixtures: missing optional RSS leg, malformed JSON, mismatched week, stale cache, duplicate URLs. |
| 634 | |
| 635 | ### Phase 2 — Analysis map/reduce dry-run |
| 636 | |
| 637 | - Add feature-flagged/dry-run map/reduce path. |
| 638 | - Mappers emit claim ledgers only. |
| 639 | - Reducer emits editorial plan plus rejected-claims/conflicts sidecar. |
| 640 | - Final writer output is compared to current single-pass output but is not published. |
| 641 | - Existing single-pass/Copilot -> GitHub Models -> no-AI fallback remains source of truth. |
| 642 | |
| 643 | ### Phase 3 — A/B comparison |
| 644 | |
| 645 | Run for at least four weekly cycles or equivalent replay fixtures: |
| 646 | |
| 647 | - Compare gate pass rate. |
| 648 | - Compare citation preservation and unsupported-claim count. |
| 649 | - Compare max per-call token estimate, final ledger growth versus preflight, total cost/runtime, and retry/fallback count. |
| 650 | - Compare editorial quality: synthesis, specificity, skepticism, blind spots, and voice. |
| 651 | - Measure rerun stability: top repo and key reference overlap. |
| 652 | |
| 653 | ### Phase 4 — Controlled crawl matrix experiments |
| 654 | |
| 655 | RSS experiment: |
| 656 | |
| 657 | - Run RSS matrix as dry-run only when trigger threshold fires or via workflow dispatch. |
| 658 | - Compare runtime, artifact reliability, partial-source diagnostics, and merge determinism versus in-process RSS. |
| 659 | |
| 660 | GitHub experiment: |
| 661 | |
| 662 | - Run no-publish shard experiment with isolated caches and strict API/rate telemetry. |
| 663 | - Do not publish sharded output until fan-in matches monolithic canonical output semantics. |
| 664 | - Enable only if >=25% speedup, <=10% API-call growth, and no secondary-rate-limit regression. |
| 665 | |
| 666 | ### Phase 5 — Guarded default-on |
| 667 | |
| 668 | - Start with workflow dispatch flag. |
| 669 | - Then scheduled dry-run. |
| 670 | - Then default-on for map/reduce only if acceptance criteria pass. |
| 671 | - Retain single-pass/no-AI fallback for at least one release cycle after default-on. |
| 672 | - Crawl matrix remains independently gated; map/reduce can ship without crawl matrix. |
| 673 | |
| 674 | --- |
| 675 | |
| 676 | ## QA Gates |
| 677 | |
| 678 | ### Crawl fan-out/fan-in gates |
| 679 | |
| 680 | - All legs share one generated `run_id`, `week`, `since`, `until`, source config checksum, topic config checksum, and code SHA. |
| 681 | - Every leg uploads exactly one status/payload artifact on `if: always()`. |
| 682 | - Fan-in validates schema, checksum, source names, week/window consistency, and required fields. |
| 683 | - Same inputs produce byte-stable canonical artifacts. |
| 684 | - Optional RSS failures are visible and gated by minimum-source-success policy. |
| 685 | - Required GitHub failures fail closed. |
| 686 | - Reruns do not double-count repos/articles. |
| 687 | - Cache metadata prevents silent mixed-window or mixed-config artifacts. |
| 688 | - Downstream analysis consumes one canonical GitHub raw payload and one canonical external-news payload regardless of collection topology. |
| 689 | |
| 690 | ### Map/reduce gates |
| 691 | |
| 692 | - Mapper output is structured JSON and schema-valid, not prose-only markdown. |
| 693 | - Each finding includes claim, evidence refs, confidence, category, source type, optional repo/news IDs, and `contra_refs`. |
| 694 | - Reducer rejects missing/malformed citations before final writer. |
| 695 | - Duplicate findings collapse without losing all citations. |
| 696 | - Contradictions are resolved with rationale or surfaced as uncertainty. |
| 697 | - Final markdown passes `scripts/analysis_gate.py` unchanged for MVP. |
| 698 | - Final repo mentions render as `[owner/repo](https://github.com/owner/repo)` and resolve to seen repo coverage. |
| 699 | - Final press claims cite retained article URLs. |
| 700 | - If map/reduce fails, pipeline falls back to existing single-pass/GitHub Models/no-AI path. |
| 701 | |
| 702 | ### Test matrix |
| 703 | |
| 704 | | Area | Scenario | Expected result | |
| 705 | | --- | --- | --- | |
| 706 | | Shared context | Matrix legs receive generated week/window | Matching `week/since/until/run_id`; mismatch fails fan-in | |
| 707 | | Deterministic fan-in | Same fixture artifacts merged twice | Identical canonical bytes/checksum | |
| 708 | | Optional source failure | One RSS source times out | Canonical artifact records failure; publish can continue if minimum-source policy passes | |
| 709 | | Required GitHub failure | Required raw GitHub shard missing | Analysis blocked; failure notification path applies | |
| 710 | | Retry accounting | Optional RSS leg fails then succeeds on retry | Retry status recorded; no duplicate articles | |
| 711 | | Stale cache | Cache restored for wrong week/config | Rejected or marked unusable; no silent mixing | |
| 712 | | Rate limit | GitHub Search quota near floor | Shard experiment throttles/skips fan-out; no uncontrolled bursts | |
| 713 | | Mapper malformed output | Mapper emits prose or invalid JSON | Reducer rejects map artifact | |
| 714 | | Mapper failure | One required mapper fails | Retry/fallback; reducer cannot silently omit it | |
| 715 | | Duplicate claims | Same claim appears in two mappers | Reducer emits one claim with combined citations | |
| 716 | | Contradiction | One mapper says signal, another says noise | Reducer records resolution or uncertainty | |
| 717 | | Citation loss | Final claim lacks source refs | Evidence validation fails before publish | |
| 718 | | Over-budget context | Reducer input exceeds token budget | Hierarchical reduce or fallback before model call | |
| 719 | | Gate regression | Final markdown missing required heading | `analysis_gate.py` fails; fallback exercised | |
| 720 | |
| 721 | --- |
| 722 | |
| 723 | ## Risks and Mitigations |
| 724 | |
| 725 | | Risk | Impact | Mitigation | |
| 726 | | --- | --- | --- | |
| 727 | | Matrixing GitHub search increases rate-limit pressure | Slower or failed crawls | Require shard experiment, API-call cap, secondary-rate regression gate, central fan-in | |
| 728 | | RSS matrix adds overhead without speed benefit | More complexity, no runtime gain | Gate on p95/source count/isolation triggers; keep in-process default | |
| 729 | | Bad fan-in corrupts data | Duplicate repos/articles, wrong trends | Schema validation, deterministic ordering, checksums, fixture tests | |
| 730 | | Cache conflicts or stale cache mixing | Incorrect crawl results | Cache keys include week/window/config/schema; stale use observable and bounded | |
| 731 | | Mapper loses nuance/citations | Unsupported final claims | Claim ledger schema, citation validation, reducer rejected-claims sidecar | |
| 732 | | Reducer hides contradictions | Misleading analysis | Require contradiction ledger and explicit resolution/surfacing | |
| 733 | | Multiple LLM calls increase total cost | Higher spend despite smaller contexts | Token/cost ledger, budget preflight, A/B threshold before default-on | |
| 734 | | Editorial voice fragments | Patchwork article | Mappers never write final prose; final writer creates one coherent article | |
| 735 | | Partial mapper failure biases analysis | Missing source/category | Required/optional shard policy, retries, visible degraded state, fallback | |
| 736 | | Prompt injection through repo/news content | Unsafe instructions influence mapper | Keep untrusted evidence boundaries in every mapper/reducer prompt | |
| 737 | |
| 738 | --- |
| 739 | |
| 740 | ## Acceptance Criteria |
| 741 | |
| 742 | ### Crawl matrix acceptance |
| 743 | |
| 744 | A crawl matrix may be enabled by default only when: |
| 745 | |
| 746 | 1. Baseline telemetry for current monolithic/in-process path is recorded. |
| 747 | 2. Fan-in validates every shard schema/checksum before analysis. |
| 748 | 3. Canonical downstream artifact paths remain unchanged. |
| 749 | 4. Same shard inputs produce byte-stable canonical artifacts. |
| 750 | 5. Optional RSS failures are reflected in metadata and downstream caveats. |
| 751 | 6. Required GitHub failures fail closed unless an explicit partial-data policy is approved. |
| 752 | 7. RSS matrix is triggered by source count/runtime/isolation need, not by default. |
| 753 | 8. GitHub shard experiment proves >=25% wall-clock crawl speedup. |
| 754 | 9. GitHub shard experiment keeps API-call growth <=10%. |
| 755 | 10. GitHub shard experiment shows no secondary-rate-limit regression. |
| 756 | 11. Existing crawl/correlation/press-context/rebuild tests remain green. |
| 757 | 12. New tests cover schema validation, deterministic fan-in, duplicate handling, missing shard handling, stale cache rejection, and partial failure metadata. |
| 758 | |
| 759 | ### Map/reduce acceptance |
| 760 | |
| 761 | The map/reduce analysis path may become publishable only when: |
| 762 | |
| 763 | 1. Given the same weekly raw GitHub JSON and compact press context, final markdown passes `scripts/analysis_gate.py`. |
| 764 | 2. Every final repo mention resolves to preflight/mapper coverage and is rendered as a proper GitHub markdown link. |
| 765 | 3. Every final press claim cites an article URL retained in source coverage or press context. |
| 766 | 4. Reducer emits rejected-claims and contradictions sidecars for audit. |
| 767 | 5. Duplicate claims collapse without losing citation provenance. |
| 768 | 6. Contradictions are resolved with rationale or surfaced as uncertainty. |
| 769 | 7. Final article has one editorial voice and satisfies the existing section/frontmatter shape. |
| 770 | 8. Max per-call token estimate decreases materially versus the current single-pass prompt; target **>=30% max context reduction**. |
| 771 | 9. Quality does not regress in human review against single-pass output. |
| 772 | 10. Reruns on identical input are stable: same top repo or documented reason for change, and at least 70% overlap in selected key references. |
| 773 | 11. Existing Copilot -> GitHub Models -> no-AI fallback path remains available until map/reduce beats current path on gate pass rate, citation coverage, and editorial review. |
| 774 | |
| 775 | ### Issue #258 dry-run implementation notes |
| 776 | |
| 777 | The first implementation is intentionally candidate-only: |
| 778 | |
| 779 | - `scripts/map_reduce_dry_run.py` performs deterministic local mappers for `new_repos`, `trending_repos`, `press_correlations`, and `prior_continuity`. |
| 780 | - Each mapper writes an `analysis_map_v1` claim ledger with stable claim IDs, evidence refs, confidence/uncertainty, coverage counts, and artifact checksums. |
| 781 | - The reducer writes `analysis_editorial_plan_v1`, plus `sidecars/rejected-claims.json` and `sidecars/contradictions.json`. |
| 782 | - The final writer emits one reader-facing markdown candidate under `data/candidates/.../map-reduce/`; it is marked not publish eligible. |
| 783 | - `qa-comparison-report.json` records structural analysis-gate status, evidence/editorial gate status, expected publish-provenance failure, blockers, and baseline comparison metadata. |
| 784 | - Workflow dispatch exposes `analysis_path=map-reduce-dry-run`, but the workflow rejects it unless `run_mode` is `dry-run` or `candidate-only`. It never runs generate/deploy/notify/promotion paths. |
| 785 | |
| 786 | --- |
| 787 | |
| 788 | ## Open Questions |
| 789 | |
| 790 | 1. What exact RSS p95 observation window should trigger matrix work: last 5 runs, last 10 runs, or rolling 30 days? |
| 791 | 2. Should the RSS source-count trigger be exactly 10, or should it vary by source type/latency class? |
| 792 | 3. What minimum RSS source success policy is acceptable for publishing: at least one source, majority of configured sources, or required source classes? |
| 793 | 4. Which GitHub shard design should be tested first: per-query/category, serialized search plus parallel details, or another hybrid? |
| 794 | 5. What Search API safety floor should block GitHub fan-out? |
| 795 | 6. Should map/reduce mappers run as separate Actions jobs, subprocesses inside one job, or model calls orchestrated by one script? |
| 796 | 7. What is the agreed per-run token/cost/runtime budget for map/reduce experiments? |
| 797 | 8. What tolerance should be allowed between preflight token estimate and final token ledger before blocking or compacting? |
| 798 | 9. Should evidence-contract validation live inside `analysis_gate.py` or remain a separate pre-gate validator initially? |
| 799 | 10. How should hierarchical reduce trigger when reducer input is still over budget? |
| 800 | 11. Who performs the human editorial comparison during the first four A/B runs? |
| 801 | 12. Should a focused analysis issue track compaction, Copilot gate retries, GitHub Models access/fallback behavior, and no-AI fallback quality after run `27030646485`? |
| 802 | |
| 803 | --- |
| 804 | |
| 805 | ## Recommendation Summary |
| 806 | |
| 807 | Do not matrix crawl by default now. RSS is too fast at current scale, and GitHub crawl is constrained by API/cache/rate behavior that needs proof before fan-out. Make crawl artifacts matrix-ready, then gate RSS matrix on source count/runtime/isolation triggers and GitHub matrix on a shard experiment proving >=25% speedup with <=10% API-call growth and no secondary-rate-limit regression. |
| 808 | |
| 809 | Do experiment with analysis map/reduce. Run `27030646485` makes this the priority: crawl and external-news collection were healthy, while analysis duration, token growth, Copilot gate failures, unavailable `openai/gpt-4o` fallback access, and eventual no-AI success were the dominant risks. Use mapper claim ledgers and a reducer/final writer to reduce LLM context, preserve citations, dedupe claims, expose contradictions, and keep the final weekly summary compliant with the existing analysis spec and gate. |