docs: add matrix crawl and map-reduce analysis PRD
Adds PRD for crawl matrix gating and map/reduce analysis experiments.
Juan Manuel Servera committed
Jun 5, 2026 at 20:33 UTC
b94706362da4234ba097de3439d064935eef0432
7 files changed
+2185
-577
.squad/agents/bender/history.md
+7
@@ -42,3 +42,10 @@
42
- Multi-source RSS remains in-process, but downstream reliability depends on a versioned canonical artifact: include crawl window, source config checksum, per-source statuses, partial failures, dedupe count, and deterministic checksum in `*-external-news.json`.
43
- Press correlation must retain bounded source-aware citations and label category/fuzzy-only matches as weak so mirrored coverage or broad topics do not inflate strong press claims.
44
- PR #242 review follow-up: failed RSS fetch exceptions must stamp source-level attempt/timeout telemetry before re-raising, and scheduled external-news crawls should pass both `--since` and `--until` so canonical crawl windows are reproducible.
45
+
46
+## 2026-06-05 Matrix crawl PRD input
47
+
48
+- Current observed run 27030646485 confirmed the pattern from issue #237: crawl job ~4m50s, GitHub `Run crawler` ~4m30s, external RSS/news ~1s.
49
+- Matrixing RSS is an isolation feature at current scale, not a speed feature; matrixing GitHub needs a shard experiment because search quota and secondary limits are shared across jobs.
50
+- Recommended PRD path is hybrid staged fan-out/fan-in: establish validated artifact contracts first, then gate RSS matrix, GitHub query matrix, and analysis map/reduce on measured thresholds.
51
+- Run 27030646485 also showed analysis, not crawling, is the critical-path risk: three Copilot attempts consumed ~28m41s, failed quality gates, GitHub Models had no `openai/gpt-4o` access, and the workflow shipped via no-AI fallback with ~112.9k estimated input tokens.
.squad/agents/farnsworth/history.md
+1
@@ -42,3 +42,4 @@
42
- Validator generates scorecards: `.squad/reskill/scorecards/YYYY-WNN.md` (human) and `data/metrics/scorecards/YYYY-WNN-scorecard.json` (machine)
43
- 548 tests passing
44
- Ready for merge
45
+- 2026-06-05T17:42:56Z: Analysis map/reduce should be justified by context pressure and claim quality, not crawl speed; mappers should emit cited claim ledgers while a single reducer preserves global thesis, dedupe, citations, and the analysis gate contract.
.squad/agents/fry/history.md
+5
@@ -93,3 +93,8 @@
93
- Run 27026348186 completed crawl, analyze, generate, and deploy successfully; only `notify` failed.
94
- Root cause: `gh release create week-2026-W23` is not idempotent when the weekly release tag already exists, so rerunning/publishing the same week produced HTTP 422 `Release.tag_name already exists`.
95
- QA fix: make notify update an existing weekly release with `gh release edit` instead of failing, while preserving release creation for new weeks.
96
+
97
+## Matrix + map/reduce PRD QA input (2026-06-05T18:15:23Z)
98
+
99
+- Matrix crawl should be gated by deterministic shared run context, per-leg artifacts, fan-in checksums, optional-vs-required failure policy, and explicit rate-limit budgets; GitHub search fan-out is riskier than RSS fan-out because search quota is already the tight limit.
100
+- Map/reduce analysis is promising for context reduction only if mapper outputs are schema-validated, reducer output preserves citations and contradictions, and the final markdown still passes the existing analysis gate with current fallback paths retained.
.squad/agents/leela/history.md
+1
@@ -44,3 +44,4 @@
44
- Routing: Bender (implementation), Fry (reliability gates), Farnsworth (press-context quality)
45
- Labels: `squad`, `squad:leela`, `squad:bender`, `go:yes`
46
- Decisions recorded in .squad/decisions.md under four entries (Bender, Farnsworth, Fry, Leela)
47
+- 2026-06-05T17:42:56.819+00:00 matrix/map-reduce PRD note: treat crawl matrix and analysis decomposition as separate decisions; keep crawl fan-out measurement-gated, while map/reduce is a citation-preserving LLM context/quality experiment with mapper claim ledgers and reducer-owned editorial coherence.
.squad/decisions-archive.md
+851
@@ -1486,3 +1486,854 @@ Apply a layered OWASP LLM01 defense for analyzer prompt rendering:
1486
The analyzer keeps using the same editorial structure, but prompt provenance is clearer and repository descriptions have bounded influence. Suspicious descriptions are logged and truncated rather than blocked to avoid false positives disrupting publication.
1487
1488
---
1489
+
1490
+Date: 2026-06-01
1491
+
1492
+## Context
1493
+Issue #217 showed the weekly analysis job can fail even when crawl data is healthy because Copilot sometimes returns a generic placeholder title or no output file at all after retries.
1494
+
1495
+## Decision
1496
+Keep Copilot CLI as the primary analysis generator, but if its output still fails the quality gate after retries, immediately fall back to `scripts/analyze_fallback.py` via GitHub Models. Also render the prompt with concrete `week`, `year`, and title guidance so the model is less likely to echo placeholder frontmatter.
1497
+
1498
+## Rationale
1499
+This keeps the higher-quality primary path, but removes CI flakiness from transient Copilot failures and from prompt placeholders leaking into the final markdown.
1500
+
1501
+---
1502
+
1503
+Date: 2026-06-01
1504
+
1505
+## Context
1506
+Issue #217 showed the weekly analysis job can fail even when crawl data is healthy because Copilot sometimes returns a generic placeholder title or no output file at all after retries.
1507
+
1508
+## Decision
1509
+Keep Copilot CLI as the primary analysis generator, but if its output still fails the quality gate after retries, immediately fall back to `scripts/analyze_fallback.py` via GitHub Models. Also render the prompt with concrete `week`, `year`, and title guidance so the model is less likely to echo placeholder frontmatter.
1510
+
1511
+## Rationale
1512
+This keeps the higher-quality primary path, but removes CI flakiness from transient Copilot failures and from prompt placeholders leaking into the final markdown.
1513
+
1514
+---
1515
+
1516
+Date: 2026-06-01
1517
+
1518
+## Context
1519
+Issue #220 showed the crawl-and-publish workflow could finish crawl and analysis successfully, then fail in the generate handoff because the generated weekly page path was absolute while the publish-branch restore logic assumed a repository-relative path. The same workflow also lacked a failure-to-issue bridge, so repeated pipeline failures did not automatically open or update a GitHub issue.
1520
+
1521
+## Decision
1522
+Normalize `page_path` to a repo-relative `content/weekly/...` path inside the generate commit step before copying weekly output onto the publish branch. Add a dedicated `notify-failure` job that always evaluates after the pipeline jobs and creates or updates a GitHub issue whenever any crawl/analyze/generate/deploy/notify job fails.
1523
+
1524
+## Rationale
1525
+The path normalization fixes the actual handoff bug without changing `scripts/generate_content.py`, which already returns an absolute file path used elsewhere in tests. A separate failure notifier makes regressions visible even when later jobs are skipped, which is the exact reliability gap that hid the recent failures.
1526
+
1527
+---
1528
+
1529
+Date: 2026-06-01
1530
+
1531
+## Context
1532
+Issue #226 adds article-level sharing. PaperMod already ships a share-buttons partial, but SquadScope also needs mobile-native sharing through the Web Share API and token-aligned styling.
1533
+
1534
+## Decision
1535
+Enable PaperMod share support through `hugo.toml` (`params.ShowShareButtons` plus an explicit `params.ShareButtons` allowlist), then override `layouts/partials/share_icons.html` in the project to add a mobile-only native share button while keeping desktop fallback links for X, LinkedIn, and Facebook. To keep the site buildable with the current PaperMod submodule layout, vendor the theme partials the site already relies on into `layouts/partials/` instead of editing the theme.
1536
+
1537
+## Rationale
1538
+This keeps the third-party theme submodule untouched, reuses the existing article-footer insertion point, and scopes the share customization to a project-level partial plus tokenized footer styles. Vendoring the required PaperMod partials also makes the build deterministic for SquadScope without depending on theme-internal `_partials` resolution quirks.
1539
+
1540
+---
1541
+
1542
+Date: 2026-06-01
1543
+
1544
+## Decision
1545
+Use an optional `predictions` frontmatter registry on weekly analysis summaries with entries shaped as `{repo, direction, confidence}`.
1546
+
1547
+## Why
1548
+The published markdown is already the durable editorial artifact, so embedding prediction intent there avoids a separate ledger drifting out of sync. Legacy summaries still need heuristic extraction from Signal/Noise/Gaps prose, but future summaries should register explicit repo-level calls for cleaner hindsight scoring.
1549
+
1550
+## Operational note
1551
+The validator writes a human scorecard to `.squad/reskill/scorecards/YYYY-WNN.md` and a machine-readable companion to `data/metrics/scorecards/YYYY-WNN-scorecard.json` so the current reskill tooling can ingest the same run.
1552
+
1553
+---
1554
+
1555
+Date: 2026-06-01
1556
+
1557
+## Context
1558
+Issue #220 showed the crawl-and-publish workflow could finish crawl and analysis successfully, then fail in the generate handoff because the generated weekly page path was absolute while the publish-branch restore logic assumed a repository-relative path. The same workflow also lacked a failure-to-issue bridge, so repeated pipeline failures did not automatically open or update a GitHub issue.
1559
+
1560
+## Decision
1561
+Normalize `page_path` to a repo-relative `content/weekly/...` path inside the generate commit step before copying weekly output onto the publish branch. Add a dedicated `notify-failure` job that always evaluates after the pipeline jobs and creates or updates a GitHub issue whenever any crawl/analyze/generate/deploy/notify job fails.
1562
+
1563
+## Rationale
1564
+The path normalization fixes the actual handoff bug without changing `scripts/generate_content.py`, which already returns an absolute file path used elsewhere in tests. A separate failure notifier makes regressions visible even when later jobs are skipped, which is the exact reliability gap that hid the recent failures.
1565
+
1566
+---
1567
+
1568
+Date: 2026-06-01
1569
+
1570
+## Decision
1571
+Use an optional `predictions` frontmatter registry on weekly analysis summaries with entries shaped as `{repo, direction, confidence}`.
1572
+
1573
+## Why
1574
+The published markdown is already the durable editorial artifact, so embedding prediction intent there avoids a separate ledger drifting out of sync. Legacy summaries still need heuristic extraction from Signal/Noise/Gaps prose, but future summaries should register explicit repo-level calls for cleaner hindsight scoring.
1575
+
1576
+## Operational note
1577
+The validator writes a human scorecard to `.squad/reskill/scorecards/YYYY-WNN.md` and a machine-readable companion to `data/metrics/scorecards/YYYY-WNN-scorecard.json` so the current reskill tooling can ingest the same run.
1578
+
1579
+---
1580
+
1581
+Date: 2026-06-05
1582
+
1583
+## Decision
1584
+
1585
+Keep external RSS/news in the existing crawl job with bounded in-process parallelism, but promote the handoff to a canonical `schema_version: 2` `data/raw/{week}-external-news.json` artifact. The artifact carries crawl window, source config checksum, requested/succeeded/failed sources, per-source status metrics, dedupe count, deterministic checksum, and partial-failure metadata.
1586
+
1587
+## Rationale
1588
+
1589
+The measured bottleneck remains the GitHub repository crawl, not the five-source RSS step. Source-aware telemetry and schema validation improve downstream reliability without adding Actions matrix startup overhead or splitting cache/API behavior.
1590
+
1591
+## Operational notes
1592
+
1593
+`correlate.py` and `render_press_context.py` now preserve article source/title/date/URL citations, label strong versus weak correlations, bound press context size to an ~8k token estimate, and keep legacy `*-techcrunch.json` and no-press fallbacks.
1594
+
1595
+- PR #242 merged at 2026-06-05T17:24:14Z, closing issue #237.
1596
+
1597
+---
1598
+
1599
+Date: 2026-06-05T15:36:19.379+00:00
1600
+
1601
+## Decision
1602
+
1603
+The crawl-and-publish analysis stage should degrade to a data-only no-AI weekly summary when both Copilot output and GitHub Models output are unavailable or rejected by the quality gate.
1604
+
1605
+## Rationale
1606
+
1607
+A missing or unauthorized model is an operational dependency failure, but the pipeline still has verified crawl data. Publishing a clearly labeled data-only summary is more reliable than failing the entire weekly handoff after preserving no reader-facing output.
1608
+
1609
+## Follow-up
1610
+
1611
+If model access is restored, the AI analysis path remains preferred. The no-AI path is only a terminal fallback after Copilot and GitHub Models fail.
1612
+
1613
+---
1614
+
1615
+Date: 2026-06-05T15:36:19.379+00:00
1616
+
1617
+**By:** Leela
1618
+
1619
+## Decision
1620
+
1621
+Issue #188 was closed as obsolete/unverifiable rather than reconstructed or rerouted. W23 draft files under `.squad/posts/`, the requested `.squad/metrics/2026/w23-distribution.md`, and platform posting evidence were absent from the working tree, git history, related issues, and PR context. PR #190 and `docs/growth/distribution-strategy.md` only provide the launch strategy/template, not the W23 execution artifacts.
1622
+
1623
+## Rationale
1624
+
1625
+Recreating social posts and metrics after the distribution window would create misleading evidence. Future growth execution issues should remain open until artifact-backed proof exists, or be closed explicitly when the posting window expires without evidence.
1626
+
1627
+---
1628
+
1629
+Date: 2026-06-05T15:36:19.379+00:00
1630
+
1631
+PR #236 keeps RSS enrichment in the existing crawl job with bounded in-process parallel fetching instead of separate Actions jobs.
1632
+
1633
+QA verified the diff covers config loading, multi-source crawl aggregation, metadata/errors, legacy `*-techcrunch.json` fallback, correlation handoff, press-context resolution, and rebuild hydration.
1634
+
1635
+Validation run in an isolated PR worktree:
1636
+- `PYTHONPATH=. .venv/bin/python -m pytest tests -q` → 554 passed
1637
+- Live RSS smoke with `--max-workers 5` → 54 articles from 5 sources, no feed errors
1638
+
1639
+Verdict: approve; no follow-up implementation owner required.
1640
+
1641
+---
1642
+
1643
+Date: 2026-06-05T15:36:19.379+00:00
1644
+
1645
+## Verdict
1646
+
1647
+Request changes before merge.
1648
+
1649
+## Rationale
1650
+
1651
+PR #236 keeps workflow secrets out of the RSS step and does not add new dependency classes, but the new config-driven fetcher currently trusts `feed_url` values without enforcing scheme/host boundaries and calls `feedparser.parse(url)` without an explicit per-request timeout. Because the workflow runs this in CI and later grants `contents: write` in the same job, external-network behavior should fail closed around the intended RSS allowlist and fail fast on slow/unresponsive feeds.
1652
+
1653
+## Required fixes
1654
+
1655
+- Validate source config with `urllib.parse.urlparse()` before crawling:
1656
+ - require `https`;
1657
+ - require hostnames to match the repository-owned allowlist for the five intended feeds;
1658
+ - reject credentials, local/private/link-local hosts, and unexpected ports.
1659
+- Fetch feeds through a code path with explicit timeout and bounded retry/backoff behavior; do not rely on the default socket timeout.
1660
+- Keep bounded concurrency; optionally validate `--max-workers` to a safe range.
1661
+
1662
+## Suggested owner
1663
+
1664
+Bender should own the fixes so Leela does not review her own implementation changes.
1665
+
1666
+---
1667
+
1668
+Date: 2026-06-05T15:36:19.379+00:00
1669
+Issue: #234
1670
+
1671
+## Decision
1672
+
1673
+Keep external news crawling in the existing crawl job and make the RSS source list config-driven via `config/external_news_sources.json`. Fetch the configured feeds concurrently inside `scripts/techcrunch_crawler.py` using a bounded thread pool, and write one weekly enrichment artifact: `data/raw/YYYY-WNN-external-news.json`.
1674
+
1675
+## Rubberduck tradeoff
1676
+
1677
+Separate GitHub Actions jobs would parallelize at the runner level, but every source would repeat checkout, Python setup, dependency install, artifact upload/download, and failure-handling boilerplate. For five RSS feeds, that overhead is larger than the network wait we are optimizing away, and it would fragment a single enrichment contract across multiple artifacts.
1678
+
1679
+In-process threading matches the current architecture better: RSS fetching is I/O-bound, feedparser work is light, and the existing crawl job already owns raw data artifact handoff. A bounded pool preserves Actions compute, keeps one failure surface, and lets future sources be added by config without editing workflow topology.
1680
+
1681
+## Scope boundary
1682
+
1683
+This is a small architectural refactor around an existing RSS crawler, so Leela implemented directly rather than reassigning to Bender. Deeper crawler work, such as source-specific parsing, feed health dashboards, or correlation logic, should remain Bender-owned.
1684
+
1685
+---
1686
+
1687
+Date: 2026-06-05T16:00:00+00:00
1688
+
1689
+Hermes re-reviewed PR #236 at Bender fix commit `e91e2a5b33b816191148125d40192b3fff8fbc6a`.
1690
+
1691
+Security blockers from the prior review are resolved:
1692
+- external RSS feed URLs are parsed with `urllib.parse.urlparse()` and restricted to HTTPS on the approved host allowlist;
1693
+- credentials, localhost/local domains, private/link-local IP literals, invalid ports, and non-443 ports are rejected;
1694
+- RSS fetches use `urlopen(..., timeout=DEFAULT_FETCH_TIMEOUT_SECONDS)` with bounded retry attempts;
1695
+- parallel RSS crawling caps workers at `DEFAULT_MAX_WORKERS` and rejects `--max-workers < 1`;
1696
+- tests cover unsafe URL rejection and explicit timeout propagation.
1697
+
1698
+Validation: `PYTHONPATH=. python -m pytest tests -q` in an isolated PR worktree passed with 563 tests.
1699
+
1700
+Decision: Hermes security approval/unblock for merge, with CodeQL checks green on the PR.
1701
+
1702
+---
1703
+
1704
+Date: 2026-06-05T16:26:00Z
1705
+Requested by: jmservera
1706
+Inputs:
1707
+- Old crawler job: https://github.com/jmservera/SquadScope/actions/runs/26753498571/job/78847225991
1708
+- New crawler job: https://github.com/jmservera/SquadScope/actions/runs/27026348186/job/79767247136
1709
+
1710
+## Observations from job logs
1711
+
1712
+### Old run — single TechCrunch RSS source
1713
+
1714
+Run `26753498571`, job `78847225991`, head `59b45137fc3ad674276b1ff8c0aa743d8e43d1bb`:
1715
+
1716
+- `crawl` job duration: 2026-06-01 11:58:51Z → 12:05:14Z, about 6m23s.
1717
+- `Run crawler`: 11:59:02Z → 12:05:00Z, about 5m58s.
1718
+- `Crawl TechCrunch RSS`: started and completed at 12:05:06Z in the step timing metadata, effectively sub-second.
1719
+- GitHub crawl summary: `Wrote data/raw/2026-W23.json with 196 new repos and 238 trending repos, saved data/snapshots/2026-W23-stars.json, used 447 API calls, and served 0 cache hits.`
1720
+- RSS summary: `Crawled 20 articles (7 relevant) → data/raw/2026-W23-techcrunch.json`.
1721
+- Rate-limit evidence: 447 rate-limit log lines; 6 search calls and 441 core calls. Minimum observed remaining quota was 24 search requests out of 30, and final core quota was 4556/5000.
1722
+- Retry/flakiness evidence: 0 `Retrying`, 0 stale-cache fallbacks, 0 search failures in the filtered log summary. The `warning`/`error` counts visible in the raw filtered scan are from workflow script text/hints, not crawler failures.
1723
+
1724
+### New run — five external RSS sources, in-process parallelism
1725
+
1726
+Run `27026348186`, job `79767247136`, head `87e55a227da78b86e9677acc96460968196e9e5a`:
1727
+
1728
+- `crawl` job duration: 2026-06-05 16:15:41Z → 16:20:49Z, about 5m08s.
1729
+- `Run crawler`: 16:15:49Z → 16:20:36Z, about 4m47s.
1730
+- `Crawl external news RSS feeds`: 16:20:42Z → 16:20:43Z, about 1s.
1731
+- GitHub crawl summary: `Wrote data/raw/2026-W23.json with 213 new repos and 236 trending repos, saved data/snapshots/2026-W23-stars.json, used 455 API calls, and served 0 cache hits.`
1732
+- External RSS summary: `Crawled 54 articles from 5 sources (27 relevant) → data/raw/2026-W23-external-news.json`.
1733
+- Rate-limit evidence: 455 rate-limit log lines; 6 search calls and 449 core calls. Minimum observed remaining quota was 24 search requests out of 30, and final core quota was 4458/5000.
1734
+- Retry/flakiness evidence: 0 `Retrying`, 0 stale-cache fallbacks, 0 search failures. The new RSS stage did not visibly bottleneck the job.
1735
+
1736
+## Current implementation shape reviewed
1737
+
1738
+The newer workflow revision changes the RSS stage from a single TechCrunch output to:
1739
+
1740
+```yaml
1741
+python scripts/techcrunch_crawler.py \
1742
+ --sources config/external_news_sources.json \
1743
+ --output "data/raw/${WEEK}-external-news.json" \
1744
+ --since "$SINCE"
1745
+```
1746
+
1747
+The external source config contains five approved feeds: TechCrunch, NVIDIA Blog, Hugging Face Blog, MIT Technology Review, and GitHub Blog.
1748
+
1749
+The new crawler implementation:
1750
+
1751
+- validates feed URLs against an HTTPS host allowlist;
1752
+- fetches RSS with an explicit 15s timeout;
1753
+- uses `ThreadPoolExecutor` with `max_workers=min(requested_or_source_count, source_count, 8)`;
1754
+- records per-source article `source` fields;
1755
+- writes one merged `external_news` artifact with metadata including `source_count`, `sources_with_articles`, totals, GitHub links, and `errors`.
1756
+
1757
+## Topology options
1758
+
1759
+### Option A — keep bounded in-process parallelism in one job
1760
+
1761
+Best fit for the current source count.
1762
+
1763
+Pros:
1764
+- Fast enough now: five-source RSS stage adds about 1s in the new run.
1765
+- No extra checkout/setup/artifact overhead per source.
1766
+- Keeps one downstream news artifact contract, which matches `correlate.py` and `render_press_context.py` expectations.
1767
+- A source failure can be represented inside `metadata.errors` without failing the entire crawl.
1768
+
1769
+Cons:
1770
+- If one feed hangs until timeout, the RSS step is bounded by timeout plus retry delay for that source.
1771
+- GitHub Actions cannot independently retry only one failed source.
1772
+- Per-source logs are less visible unless the script emits explicit source start/end/error lines.
1773
+
1774
+### Option B — GitHub Actions matrix per source/type
1775
+
1776
+Not justified yet for the RSS feeds alone.
1777
+
1778
+Pros:
1779
+- Clean isolation and per-source retry visibility.
1780
+- Natural if sources become heterogeneous: RSS, APIs, browser crawls, paid sources, or sources with independent secrets/quotas.
1781
+- Failure policy can vary by source.
1782
+
1783
+Cons:
1784
+- More runner minutes and more setup overhead than the current 1s RSS crawl.
1785
+- Requires explicit merge job and stricter artifact naming/schema validation.
1786
+- Increases race/branch commit complexity if matrix outputs are committed directly.
1787
+- Does not help the actual current bottleneck, which is the GitHub repo crawl step at roughly 4m47s–5m58s.
1788
+
1789
+### Option C — hybrid/staged topology
1790
+
1791
+Recommended next iteration, but staged lightly: keep RSS in-process now, make the artifact contract merge-ready, and add a separate merge/validate step before analysis.
1792
+
1793
+Pros:
1794
+- Preserves current speed and simplicity.
1795
+- Creates a clean future migration path to a matrix without changing analysis consumers.
1796
+- Lets the pipeline distinguish crawler collection from artifact assembly/validation.
1797
+- Gives downstream stages one canonical `external-news` artifact regardless of whether collection was single-process or matrix.
1798
+
1799
+Cons:
1800
+- Adds one small script/step for validation/merge even before a matrix is needed.
1801
+- Requires schema versioning discipline.
1802
+
1803
+## Recommendation
1804
+
1805
+Use a hybrid/staged approach:
1806
+
1807
+1. Keep the current bounded in-process parallel RSS crawl for the next iteration.
1808
+2. Add explicit per-source logs: start time, duration, article count, relevant count, GitHub-link count, and error if any.
1809
+3. Add `schema_version` and stable `sources_requested` / `sources_succeeded` / `sources_failed` metadata to `external-news.json`.
1810
+4. Add a validation/merge script that accepts either:
1811
+ - current single merged external-news payload, or
1812
+ - future per-source payloads named like `external-news-${source}.json`.
1813
+5. Make analysis consume only the canonical merged artifact: `data/raw/${WEEK}-external-news.json`.
1814
+6. Move to an Actions matrix only when evidence shows RSS collection is material: e.g. external source stage exceeds 60s p95, source count exceeds about 12–15, or a source requires independent credentials/rate policy.
1815
+
1816
+## Risks
1817
+
1818
+- Current metadata has `errors`, but success criteria are ambiguous. A total RSS outage could still return exit 0 if errors are recorded but no minimum-source gate exists.
1819
+- The script name `techcrunch_crawler.py` is now misleading for multi-source external news. Rename later only with backward-compatible CLI/wrapper to avoid breaking existing docs/tests.
1820
+- The current logs show `0 cache hits` in both old and new GitHub crawls, so cache restoration is not reducing runtime in these examples. That may be due to TTL/query churn or artifact mismatch and should be investigated separately from RSS topology.
1821
+- Search API quota is the tighter GitHub limit: both runs reached minimum remaining 24/30 search requests while core remained above 4450/5000. More GitHub search parallelism would risk secondary/rate-limit pressure; RSS parallelism does not consume GitHub API quota.
1822
+
1823
+## Acceptance criteria for next implementation
1824
+
1825
+- A weekly crawl with five RSS sources still completes the external RSS stage in under 30s under normal network conditions.
1826
+- The RSS crawler logs one concise summary line per source with duration and counts.
1827
+- `data/raw/${WEEK}-external-news.json` includes `schema_version`, `source_count`, `sources_requested`, `sources_succeeded`, `sources_failed`, `sources_with_articles`, and `errors`.
1828
+- The workflow fails only when the required GitHub raw payload is missing or the external-news artifact is structurally invalid; individual optional RSS source failures are recorded and do not block analysis unless fewer than an agreed minimum number of sources succeed.
1829
+- Rebuild mode hydrates the canonical external-news artifact and remains backward-compatible with legacy `${WEEK}-techcrunch.json`.
1830
+- Correlation and press-context steps read the canonical merged artifact and do not need to know whether collection was in-process or matrix-based.
1831
+- Tests cover single merged payload validation plus simulated future per-source merge inputs.
1832
+
1833
+---
1834
+
1835
+Date: 2026-06-05T16:26:00Z
1836
+Requested by: jmservera
1837
+
1838
+## Evidence reviewed
1839
+
1840
+- Old crawl job `26753498571 / 78847225991`: crawl job 11:58:51–12:05:14 (~6m23s). GitHub crawl wrote `data/raw/2026-W23.json` with 196 new repos, 238 trending repos, 447 API calls, 0 cache hits. Single TechCrunch RSS step produced 20 articles / 7 relevant. Raw artifact: 199,434 bytes; cache artifact: 10,188,525 bytes.
1841
+- New crawl job `27026348186 / 79767247136`: crawl job 16:15:41–16:20:49 (~5m08s). GitHub crawl wrote 213 new repos, 236 trending repos, 455 API calls, 0 cache hits. External RSS step produced 54 articles from 5 sources / 27 relevant. Raw artifact: 207,606 bytes; cache artifact: 11,874,886 bytes.
1842
+- Current `origin/main` workflow runs GitHub crawl first, then a single in-process parallel `scripts/techcrunch_crawler.py --sources config/external_news_sources.json` step, uploads one `raw-data` artifact, and analysis falls back from `{week}-external-news.json` to legacy `{week}-techcrunch.json`.
1843
+- Existing tests cover source config validation, allowlisted HTTPS feed URLs, explicit fetch timeout, in-process parallel aggregation, metadata/errors in combined output, correlation loading, and press-context rendering.
1844
+
1845
+## Reliability observations
1846
+
1847
+1. Multi-source RSS is not currently the runtime bottleneck. The new five-source RSS step took about one second after dependencies; the GitHub API crawler still dominates the crawl job at ~4m47.
1848
+2. The in-process model is operationally simple and fast, but failure isolation is only at script level. A per-source fetch exception can be represented in `metadata.errors`, but a bad config parse, merge bug, dependency issue, or Python process failure takes out every external source in one step.
1849
+3. Retry granularity is poor in the current shape. A flaky NVIDIA/Hugging Face/MIT feed requires rerunning the whole crawl job, including the GitHub API crawl and cache artifact upload, unless manual surgery is done.
1850
+4. Artifact availability is all-or-nothing for external news. The workflow uploads `raw-data` after the combined step, so failed individual sources do not leave independently downloadable payloads unless the combined script writes a degraded aggregate.
1851
+5. Cache behavior argues against matrixing the GitHub repository crawl right now. The GitHub cache is a single `data/cache/` artifact restored from the previous successful run; splitting GitHub query work would introduce cache merge/conflict questions without evidence it is the bottleneck needing parallel source isolation.
1852
+6. Partial data tolerance exists downstream: correlation only runs if an external-news or legacy TechCrunch file exists, and press context can render a no-press fallback. That is good, but the workflow does not yet make optional-source degradation explicit enough in job summaries or gating.
1853
+7. Reproducibility needs tightening before matrix fan-out. Matrix jobs must share the same centrally computed `week`, `since`, and `until`; otherwise each source can observe a slightly different crawl window.
1854
+
1855
+## Recommendation
1856
+
1857
+For the next iteration, keep the GitHub repository crawl as one core job and split external RSS/news sources into a GitHub Actions matrix with `fail-fast: false`, per-source artifacts, and a deterministic merge job before analysis.
1858
+
1859
+This gives the best reliability improvement without multiplying the GitHub API/cache risk. Because external RSS jobs can run in parallel with the slower GitHub crawl, matrix overhead should not increase the critical path much if analysis depends on a small merge job rather than on the old monolithic crawl job. Do not push source merging into analysis; merge before analysis so correlation, press context, artifacts, and rebuild hydration keep a stable stage boundary.
1860
+
1861
+## Acceptance criteria for the issue
1862
+
1863
+- Workflow defines a shared crawl context (`week`, `since`, `until`, source config checksum) once and passes it to all crawl jobs.
1864
+- GitHub repository crawl remains a required/core job and continues to restore/upload the existing `crawl-cache` artifact.
1865
+- External news uses a matrix over configured source names/URLs with `strategy.fail-fast: false`.
1866
+- Each source uploads a per-source artifact on `if: always()` containing either:
1867
+ - a valid source payload with articles and metadata; or
1868
+ - a status/error JSON with source name, error class/message, attempts, duration, and crawl window.
1869
+- A merge job runs on `if: always()` after core crawl and all news matrix jobs, downloads available source artifacts, validates schemas, deduplicates/sorts deterministically, and writes canonical `data/raw/{week}-external-news.json`.
1870
+- Analysis consumes only the merged canonical external-news file plus the GitHub raw file; it does not crawl or merge feeds itself.
1871
+- Optional external-news failures do not block publication when GitHub raw data is valid; they must produce visible warnings and metadata. A config/schema/security validation failure should fail the workflow because it is deterministic and actionable.
1872
+- Rebuild mode hydrates the merged external-news file and still accepts legacy `{week}-techcrunch.json`.
1873
+- The raw-data artifact remains available even when one or more optional source jobs fail.
1874
+- CI summary reports per-source status and aggregate totals; the next run can identify exactly which feed was slow/flaky.
1875
+
1876
+## Tests to add or update
1877
+
1878
+- Unit tests for a new merge helper/script:
1879
+ - merges multiple valid source artifacts into `source=external_news` canonical output;
1880
+ - preserves `source_count`, `sources_with_articles`, `metadata.errors`, and per-source status;
1881
+ - deduplicates repeated article URLs deterministically without dropping distinct source attribution unexpectedly;
1882
+ - sorts output deterministically by `published_at`, then source/name/url;
1883
+ - tolerates missing/failed optional source artifacts;
1884
+ - fails on malformed JSON, invalid source names, or mismatched `week`/window metadata.
1885
+- CLI tests for fixed `--since` and `--until` propagation so matrix jobs reproduce the same window.
1886
+- Workflow/handoff tests or a validation script fixture that asserts `analyze` depends on the merge artifact, not raw matrix artifacts directly.
1887
+- Correlation and press-context tests with merged `*-external-news.json`, legacy `*-techcrunch.json`, and no external-news file.
1888
+- Regression test that a single source failure still produces a merged canonical file with remaining articles and visible `metadata.errors`.
1889
+- Regression test that all external sources failing produces a no-press fallback path while preserving a valid GitHub raw artifact.
1890
+
1891
+## Metrics/logging to capture
1892
+
1893
+- Per source: source name, URL host, start/end/duration seconds, attempts, timeout seconds, total articles, relevant articles, GitHub links found, error class/message, and success/failure.
1894
+- Aggregate: source_count, successful_source_count, failed_source_count, total/relevant articles, dedupe counts, artifact size, merge duration.
1895
+- Core GitHub crawl: API calls, cache hits, stale cache hits, rate-limit remaining/resource/reset, partial failure count, repo counts, snapshot repo count.
1896
+- Workflow: job durations for core crawl, each source crawl, merge, analyze; whether analysis used full press data, partial press data, or no-press fallback.
1897
+- Reproducibility: source config checksum, code commit SHA, crawl window, and canonical merged file checksum.
1898
+
1899
+## Risks / gates
1900
+
1901
+- Matrix jobs add workflow complexity and more artifacts; keep merge logic small and heavily tested.
1902
+- Matrix setup overhead is only acceptable if source jobs run in parallel with the GitHub crawl. If they remain sequential after core crawl, in-process fan-out is faster for five feeds.
1903
+- Do not treat article volume alone as success. Gate on valid schemas, explicit source statuses, deterministic merge, and downstream correlation/press-context success.
1904
+- Keep optional-source degradation visible. Silent partial data is worse than a failed optional feed.
1905
+
1906
+---
1907
+
1908
+Date: 2026-06-05T16:26:00Z
1909
+Requested by: jmservera
1910
+Issue: https://github.com/jmservera/SquadScope/issues/237
1911
+
1912
+## Decision
1913
+
1914
+Created issue #237, "Improve multi-source crawler telemetry and source-aware press correlation."
1915
+
1916
+The lead decision is:
1917
+
1918
+- Keep GitHub repository crawl monolithic and cached.
1919
+- Keep external RSS/news crawl in-process with bounded parallelism for now.
1920
+- Defer Actions matrix fan-out until evidence triggers it: RSS/news p95 > 60s, source count > 10, or a source needs independent retry, credentials, quota, or network isolation.
1921
+- Treat merge-before-analyze as deterministic data fan-in, not staged LLM map-reduce.
1922
+
1923
+## Scope captured
1924
+
1925
+The issue asks the next iteration to improve:
1926
+
1927
+- per-source external-news status and metrics;
1928
+- schema/versioned deterministic canonical `*-external-news.json`;
1929
+- source-aware and bounded `correlate.py` / `render_press_context.py`;
1930
+- cross-source dedupe to avoid correlation inflation;
1931
+- press-context token/article bounds and telemetry;
1932
+- tests for partial failures, fallback paths, reproducibility, dedupe, and citation preservation.
1933
+
1934
+## Non-goals captured
1935
+
1936
+- Multi-pass/staged LLM analysis.
1937
+- GitHub raw compaction.
1938
+- Matrix split unless the trigger threshold is met.
1939
+- Core GitHub crawler topology changes.
1940
+
1941
+## Routing
1942
+
1943
+Labels applied: `squad`, `squad:leela`, `squad:bender`, `go:yes`.
1944
+
1945
+Bender is the likely implementation owner; Fry should validate reliability gates; Farnsworth should review press-context quality.
1946
+
1947
+---
1948
+
1949
+Date: 2026-06-05T16:26:00.133+00:00
1950
+
1951
+## Context
1952
+
1953
+The old crawler run (`26753498571` / job `78847225991`) produced:
1954
+
1955
+- `data/raw/2026-W23.json`: 196 new repos, 238 trending repos, 447 GitHub API calls.
1956
+- `data/raw/2026-W23-techcrunch.json`: 20 TechCrunch articles, 7 relevant.
1957
+
1958
+The new crawler run (`27026348186` / job `79767247136`) produced:
1959
+
1960
+- `data/raw/2026-W23.json`: 213 new repos, 236 trending repos, 455 GitHub API calls.
1961
+- `data/raw/2026-W23-external-news.json`: 54 articles from 5 sources, 27 relevant, no feed errors.
1962
+- Source mix: TechCrunch 20, NVIDIA Blog 13, Hugging Face Blog 9, MIT Technology Review 10, GitHub Blog 2.
1963
+
1964
+The external-news artifact is roughly 45.5 KB / 11.4k token-estimate by itself; the GitHub raw artifact from the same run is roughly 296 KB / 74k token-estimate. Existing rendered press context can also be large: the W23 TechCrunch-only press context on `publish` is about 27.9 KB / 7k token-estimate before adding the extra sources.
1965
+
1966
+## Analyst assessment
1967
+
1968
+Do not send all raw GitHub and all raw external-news inputs directly to the weekly analysis model. That path is editorially fragile: the model will spend attention on repeated article summaries, source boilerplate, low-relevance items, and broad category matches instead of the actual job — deciding what matters. It also increases prompt-injection surface and makes limited-context models more likely to drop required sections, lose citations, or overfit the latest/longest source.
1969
+
1970
+The current analysis contract already expects a concise `Where Industry Meets Code` comparison, not a press digest. External news should therefore enter analysis as a compact, source-aware correlation artifact: a deterministic press-context file that preserves the top evidence and citations while discarding bulk article text.
1971
+
1972
+## Options considered
1973
+
1974
+### 1. Pass every raw input at once
1975
+
1976
+**Pros**
1977
+- Maximum recall.
1978
+- Simplest implementation if context windows are assumed unlimited.
1979
+
1980
+**Cons**
1981
+- Poor fit for limited-context or cheaper fallback models.
1982
+- Increases prompt size from already-large GitHub raw payloads into 90k+ token territory before learned state and instructions.
1983
+- Encourages article summarization instead of repo-to-industry synthesis.
1984
+- Makes the quality gate less reliable because structural failures, missing references, and citation drift become more likely.
1985
+- Treats all sources equally even when some are lower relevance for developer adoption.
1986
+
1987
+**Analyst verdict:** Reject for the default path.
1988
+
1989
+### 2. Pre-merge and summarize all sources into one artifact
1990
+
1991
+**Pros**
1992
+- Keeps the analyzer prompt smaller.
1993
+- Gives the model one stable press evidence surface.
1994
+- Easier to validate than source-specific LLM steps.
1995
+
1996
+**Cons**
1997
+- If summarization is LLM-generated, it can lose citations or compound hallucinations before the main analysis.
1998
+- If it simply concatenates all sources, it still carries noise.
1999
+- Needs source provenance to avoid TechCrunch/GitHub/NVIDIA/MIT/HF being flattened into one undifferentiated "press" voice.
2000
+
2001
+**Analyst verdict:** Good only if deterministic and citation-preserving.
2002
+
2003
+### 3. Run staged source-specific LLM analyses
2004
+
2005
+**Pros**
2006
+- Keeps each model call small.
2007
+- Can produce richer source-by-source editorial nuance.
2008
+- Scales if future source count grows substantially.
2009
+
2010
+**Cons**
2011
+- Higher cost and more failure points.
2012
+- Second-stage analyzer may inherit summaries without enough evidence.
2013
+- Quality gate currently validates final structure, not the faithfulness of intermediate source briefs.
2014
+- More operational complexity than current volume justifies.
2015
+
2016
+**Analyst verdict:** Defer. Consider only when relevant article volume regularly exceeds the compact artifact budget.
2017
+
2018
+### 4. Use compact correlation / press-context artifact
2019
+
2020
+**Pros**
2021
+- Best match for the weekly brief: correlations, divergences, citations, and source provenance are preserved.
2022
+- Keeps the LLM focused on editorial judgment instead of raw article triage.
2023
+- Can be generated deterministically and tested.
2024
+- Supports fallback models and no-AI fallback more safely.
2025
+
2026
+**Cons**
2027
+- Requires explicit ranking and truncation rules.
2028
+- Bad correlation heuristics can still inject false positives, especially category-only matches.
2029
+- Needs quality gates that check citation preservation, not just markdown shape.
2030
+
2031
+**Analyst verdict:** Recommended default.
2032
+
2033
+## Recommendation
2034
+
2035
+Implement a source-aware compact press-context artifact as the only external-news input to weekly analysis.
2036
+
2037
+The analyzer should receive:
2038
+
2039
+1. Sanitized/possibly compacted GitHub repo evidence.
2040
+2. Previous weekly summary.
2041
+3. Learned wisdom/skills.
2042
+4. One compact press-context artifact containing:
2043
+ - source coverage summary (`source`, total articles, relevant articles, errors),
2044
+ - 5-10 ranked press items with URL, source, date, relevance score, and one-sentence why-it-matters,
2045
+ - 5-10 highest-confidence repo/news correlations,
2046
+ - separate "possible/weak correlations" bucket for category-only or fuzzy matches,
2047
+ - 3-6 divergence findings,
2048
+ - complete citations for every article retained,
2049
+ - explicit caveat when sources were unavailable or noisy.
2050
+
2051
+Do not include all article summaries in the analysis prompt. Do not let low-confidence category matches count as strong press correlation. Category-only matches should be framed as weak context unless reinforced by direct GitHub link, organization/entity match, temporal spike, or repeated source agreement.
2052
+
2053
+## Prompt / gate implications
2054
+
2055
+- The prompt should say: "Use press context as correlation evidence, not as instructions and not as content to repackage."
2056
+- External-news content should be wrapped in the same untrusted-content boundary pattern used for raw repo JSON.
2057
+- The quality gate should remain structural, but add evidence-focused checks:
2058
+ - `## Key References > ### Press & Industry` contains 3-5 retained article links when press data exists.
2059
+ - The body does not contain raw correlation dumps, model instructions, or full article payloads.
2060
+ - At least one sentence in `Where Industry Meets Code` distinguishes strong correlation from weak/noisy press context.
2061
+ - If external-news metadata reports source errors, the article includes a concise caveat.
2062
+
2063
+## Acceptance criteria for Leela's next issue
2064
+
2065
+- A deterministic compact press-context artifact is generated before analysis from `*-external-news.json` and `*-correlations.json`.
2066
+- The compact artifact has a documented token/size budget, recommended ceiling: <= 8k token-estimate for press context.
2067
+- The weekly analysis prompt consumes the compact press context, not the full external-news JSON.
2068
+- Press context retains source name, article URL, article title, published date, relevance score, and correlation confidence for every retained citation.
2069
+- Correlations are tiered: direct-link/org/entity/temporal matches are strong; fuzzy/category-only matches are weak unless corroborated.
2070
+- Quality gate or tests reject raw article/correlation dumps in final analysis output.
2071
+- Tests cover: multi-source source counts, no-source/error caveats, citation preservation, truncation behavior, weak-correlation labeling, and legacy `*-techcrunch.json` fallback.
2072
+- The final weekly summary still conforms to `docs/analysis-spec.md`: required frontmatter, stable H2 sections, complete Key References, no placeholders, no raw JSON/tool logs.
2073
+
2074
+## Editorial success metric
2075
+
2076
+The finished weekly brief should make fewer but sharper press claims: "what the industry narrative explains, what developer activity confirms, and what the press is missing." It should not become a five-source news roundup.
2077
+
2078
+---
2079
+
2080
+Date: 2026-06-05T17:11:29.929+00:00
2081
+Issue: https://github.com/jmservera/SquadScope/issues/238
2082
+Run: https://github.com/jmservera/SquadScope/actions/runs/27026348186
2083
+
2084
+## Finding
2085
+
2086
+The pipeline stages that produce and publish data succeeded. The only failed job was `notify`, where `gh release create week-2026-W23` returned HTTP 422 because the `week-2026-W23` release already existed.
2087
+
2088
+## Decision
2089
+
2090
+Treat this as a real QA-owned workflow idempotency bug, not a transient network or rate-limit failure. Weekly notify must be safe to rerun for an already-published week.
2091
+
2092
+## Fix
2093
+
2094
+Update the notify release step to check for the weekly release tag. If it exists, edit the existing release title/notes and mark it latest; otherwise create it as before.
2095
+
2096
+## Validation
2097
+
2098
+- `PYTHONPATH=. .venv/bin/python -m pytest tests/test_pipeline.py -q` — 9 passed.
2099
+- `PYTHONPATH=. .venv/bin/python -m pytest tests -q` — 563 passed after installing project requirements and pytest in a local venv.
2100
+
2101
+---
2102
+
2103
+Date: 2026-06-05T17:42:56Z
2104
+Requested by: jmservera
2105
+Scope: PRD-ready findings for Leela; no code or issue created.
2106
+
2107
+## Recommendation
2108
+
2109
+Keep the last implementation decision for now: GitHub crawl stays monolithic, and external RSS/news stays bounded in-process. The measured bottleneck is still the GitHub repository crawl, while RSS/news is already parallelized inside one job and completes in about one second for five feeds.
2110
+
2111
+Introduce a matrix only behind measured gates, and prefer a staged fan-out/fan-in design over directly feeding matrix outputs to analysis. If the goal is smaller LLM context, solve that in the analysis handoff with deterministic map/reduce summaries rather than splitting API collection first.
2112
+
2113
+## Why matrix was not used last time
2114
+
2115
+The prior decision was evidence-based:
2116
+
2117
+- Old crawler job `26753498571 / 78847225991`: crawl job ~6m23s; `Run crawler` ~5m58s.
2118
+- New crawler job `27026348186 / 79767247136`: crawl job ~5m08s; `Run crawler` ~4m47s; external RSS step ~1s.
2119
+- Current observed run `27030646485 / 79781846313`: crawl job ~4m50s; `Run crawler` ~4m30s; external RSS step ~1s.
2120
+- RSS fan-out would add repeated checkout/setup/artifact overhead that is larger than the current RSS work.
2121
+- GitHub crawl uses one shared cache, one shared token/rate-limit view, one star snapshot, and one deterministic output. Splitting it before measuring shard behavior risks API thrash and merge bugs.
2122
+
2123
+This decision still holds unless the measured gates below fire.
2124
+
2125
+## Parallelizable work
2126
+
2127
+### GitHub repository crawl
2128
+
2129
+Parallelizable in theory:
2130
+
2131
+- Search query pages (`created:` new repos, `pushed:` trending repos, topic config primary/secondary queries).
2132
+- Candidate filtering and repository normalization.
2133
+- README existence checks.
2134
+- Star snapshot construction after shard outputs are merged.
2135
+
2136
+Not safely parallelizable without coordination:
2137
+
2138
+- GitHub search quota management. Search quota is much tighter than core quota; previous issue context observed search remaining near `24/30` while core stayed around `4450+/5000`.
2139
+- Secondary rate limit backoff and cooling. More jobs can make the aggregate request rate worse.
2140
+- Cache writes unless each shard has an isolated cache namespace and a deterministic post-merge cache artifact.
2141
+- Trending star-gain computation until all candidates and the prior snapshot are available.
2142
+
2143
+### External RSS/news
2144
+
2145
+Parallelizable today and already done in-process:
2146
+
2147
+- Per-feed fetch and parse via `ThreadPoolExecutor`, capped at 8 workers.
2148
+- Per-source status telemetry, partial failures, and deterministic source ordering.
2149
+
2150
+Good matrix candidate later:
2151
+
2152
+- Per-source crawl jobs when source count grows, source p95 gets slow, or source-specific retries/credentials/network failures need isolation.
2153
+
2154
+### Correlation / press context / analysis
2155
+
2156
+Parallelizable for context reduction:
2157
+
2158
+- Correlation can map over repo shards against the same bounded article set, then reduce ranked correlations and divergences.
2159
+- LLM analysis can map over normalized slices such as `new_repos`, `trending_repos`, `press_correlations`, and `divergences`, then reduce to the final weekly summary.
2160
+
2161
+Requires strict contracts because final analysis must remain deterministic, citation-preserving, and bounded.
2162
+
2163
+## Matrix design options
2164
+
2165
+### Option A — RSS per-source matrix
2166
+
2167
+Design:
2168
+
2169
+1. A setup job computes `week`, `since`, `until`, and the source list from `config/external_news_sources.json`.
2170
+2. Matrix job runs one source per leg and emits `external-news-source-{source}.json`.
2171
+3. Fan-in job downloads all source artifacts, validates schema/checksums, dedupes articles, computes canonical `data/raw/{week}-external-news.json`, and uploads raw data for analyze.
2172
+
2173
+Pros:
2174
+
2175
+- Best failure isolation for flaky feeds.
2176
+- Easy per-source retries.
2177
+- Simple ownership and telemetry.
2178
+
2179
+Cons:
2180
+
2181
+- Slower than current state for five feeds because each leg pays Actions startup/setup overhead.
2182
+- More artifact merge code and missing-leg handling.
2183
+- Minimal end-to-end speed gain unless RSS p95 is high.
2184
+
2185
+Use when: RSS/news p95 > 60s, source count > 10, or any source needs independent retry/credential/quota isolation.
2186
+
2187
+### Option B — GitHub per-query/category matrix
2188
+
2189
+Design:
2190
+
2191
+1. Setup job restores previous cache and star snapshot, builds query shards.
2192
+2. Matrix legs run `scripts/crawl.py`-like shard mode for one query/category, writing candidate repo records, API metadata, errors, and shard cache.
2193
+3. Fan-in job validates shards, dedupes repos by `full_name`, applies significance filtering if not already done, checks README policy, merges API/cache metadata, computes star gains, builds snapshots, and emits canonical `data/raw/{week}.json`.
2194
+
2195
+Pros:
2196
+
2197
+- Potentially reduces wall-clock time if API wait and README checks dominate and rate limits permit concurrency.
2198
+- Isolates query failures.
2199
+- Enables targeted rerun of failed query shards.
2200
+
2201
+Cons:
2202
+
2203
+- Highest risk: search quota and secondary rate limits are shared across jobs but not centrally visible.
2204
+- Query shards can produce overlapping repos; merge must be deterministic.
2205
+- Cache artifacts can conflict or balloon.
2206
+- Fan-in must own star snapshot and trending delta semantics to avoid inconsistent star gains.
2207
+
2208
+Use only after experiments prove aggregate API calls, secondary-limit events, and wall-clock improve versus monolith.
2209
+
2210
+### Option C — Hybrid staged fan-out/fan-in
2211
+
2212
+Design:
2213
+
2214
+1. `crawl-github` remains monolithic initially.
2215
+2. `crawl-rss` remains in-process initially, or later becomes RSS matrix.
2216
+3. `merge-crawl-artifacts` is introduced as an explicit fan-in/validation job even before matrixing.
2217
+4. `correlate-map` optionally shards repository analysis and writes bounded correlation shards.
2218
+5. `reduce-analysis-context` emits compact deterministic context for the LLM.
2219
+
2220
+Pros:
2221
+
2222
+- Lowest risk migration path.
2223
+- Creates the artifact contract needed for any future matrix.
2224
+- Targets the user’s context-size concern without forcing risky GitHub API fan-out.
2225
+
2226
+Cons:
2227
+
2228
+- Does not materially speed crawl until matrix gates fire.
2229
+- Adds one fan-in job and contract tests.
2230
+
2231
+Recommended path.
2232
+
2233
+## Expected performance impact
2234
+
2235
+Current baseline:
2236
+
2237
+- GitHub crawler dominates: ~4.5–6 minutes.
2238
+- External RSS/news: ~1 second for five sources.
2239
+- Analyze stage dominates full workflow when LLM retries occur; example prior run analyze was ~36 minutes.
2240
+
2241
+Expected impacts:
2242
+
2243
+- RSS matrix: likely neutral or slower at current scale; improves retry isolation only.
2244
+- GitHub matrix: possible wall-clock improvement, but only if search/core rate limits and secondary limits do not force serialized backoff. Risk of slower runs from quota contention is real.
2245
+- Hybrid fan-in plus analysis map/reduce: biggest context-size benefit; may reduce LLM retries and latency by giving the analyzer smaller, purpose-built context.
2246
+
2247
+Failure/retry behavior:
2248
+
2249
+- RSS matrix can mark one source failed and still publish partial results if fan-in records `sources_failed` and caveats downstream output.
2250
+- GitHub matrix should fail closed if required query shards fail, unless a PRD explicitly allows partial GitHub data with visible `partial_failures`.
2251
+- Analyze map/reduce can retry failed map slices independently, but the reduce stage must fail if required slice summaries are missing or invalid.
2252
+
2253
+## Required artifact contracts
2254
+
2255
+### GitHub shard artifact, if implemented
2256
+
2257
+Each shard must include:
2258
+
2259
+- `schema_version`
2260
+- `week`, `crawl_window`, `shard_id`, `query`, `query_type`
2261
+- `repos` or raw candidates with `full_name`, stars, topics, timestamps, license, URL, fork/template flags as needed
2262
+- `api_calls_used`, `cache_hits`, `stale_cache_hits`
2263
+- `rate_limit_limit`, `rate_limit_remaining`, `rate_limit_reset`, `rate_limit_resource`
2264
+- `partial_failures`
2265
+- deterministic `artifact_checksum`
2266
+
2267
+Fan-in must emit the existing canonical `data/raw/{week}.json` shape plus snapshot, preserving deterministic ordering and existing validation.
2268
+
2269
+### RSS source artifact, if implemented
2270
+
2271
+Each source shard must include:
2272
+
2273
+- `schema_version`
2274
+- `week`, `source`, `source_config_checksum`, `crawl_window`
2275
+- `articles` with source provenance, URL, title, published date, categories, summary, GitHub links, entities, relevance score
2276
+- `source_status` with start/end/duration, attempts, timeout, success/error fields
2277
+- `artifact_checksum`
2278
+
2279
+Fan-in must emit canonical `data/raw/{week}-external-news.json` with `sources_requested`, `sources_succeeded`, `sources_failed`, `source_status`, `sources_with_articles`, `dedupe_count`, `errors`, and stable checksum.
2280
+
2281
+### Analysis map/reduce artifacts
2282
+
2283
+Map outputs should be compact and machine-validatable:
2284
+
2285
+- `schema_version`
2286
+- `week`, `slice_id`, `slice_type`, `source_artifacts`
2287
+- top ranked findings with citations and reason codes
2288
+- token/character estimate
2289
+- `required_context_omitted: false` or explicit omissions
2290
+- `artifact_checksum`
2291
+
2292
+Reduce input should never be raw unbounded crawl JSON. It should consume validated map summaries plus bounded press context.
2293
+
2294
+## Open questions / experiments
2295
+
2296
+1. Measure per-step p50/p95 for `Run crawler`, RSS, correlation, press context, and analyze across at least 5–10 runs.
2297
+2. Run a no-merge experiment that replays GitHub query shards with isolated caches and records total API calls, search remaining, secondary-limit events, and wall-clock.
2298
+3. Measure checkout/setup/artifact overhead for a small RSS matrix versus current in-process RSS.
2299
+4. Determine whether GitHub Actions concurrency with one `GITHUB_TOKEN` worsens search quota or secondary rate limits.
2300
+5. Decide partial-data policy: RSS may degrade; GitHub likely should fail closed unless enough shards succeed by explicit threshold.
2301
+6. Define max shard count and naming to avoid artifact sprawl.
2302
+7. Decide whether map/reduce analysis runs in Actions jobs, Copilot sub-prompts, or a deterministic Python preprocessor plus one LLM reduce.
2303
+8. Validate that reduced context preserves all citations needed by quality gates and Copilot review.
2304
+
2305
+## Acceptance criteria
2306
+
2307
+A crawl matrix PRD should require:
2308
+
2309
+- Baseline telemetry recorded for current monolith before implementation.
2310
+- Fan-in job validates every shard schema and checksum before analyze.
2311
+- Canonical output paths remain unchanged for downstream consumers.
2312
+- Deterministic merge: same inputs produce byte-stable canonical artifacts except timestamps explicitly excluded from checksum.
2313
+- Partial RSS failures are reflected in metadata and downstream caveats.
2314
+- GitHub shard failures either fail the workflow or are surfaced by an explicit accepted degradation policy.
2315
+- No increase in total GitHub API calls greater than 10% versus baseline without approval.
2316
+- No secondary-rate-limit regression versus baseline.
2317
+- End-to-end crawl p95 improves by at least 25% for GitHub matrix, or RSS isolation demonstrates successful partial publication with one failed source.
2318
+- Analysis context token estimate decreases by at least 30% for map/reduce without reducing required citations or quality-gate pass rate.
2319
+- Existing tests pass, plus new contract tests for shard validation, fan-in merge, deterministic ordering, duplicate handling, missing shard handling, and partial failure metadata.
2320
+
2321
+## Metrics gates
2322
+
2323
+Implement matrix only if at least one gate is met:
2324
+
2325
+- RSS/news p95 > 60 seconds.
2326
+- RSS/news configured source count > 10.
2327
+- A source needs independent credentials, retry policy, or failure isolation.
2328
+- GitHub crawl p95 > 8 minutes and shard experiment shows at least 25% wall-clock improvement with <=10% API-call increase and no secondary-rate-limit increase.
2329
+- Analysis prompt/context p95 exceeds agreed token budget or LLM retry rate exceeds 20%, and map/reduce experiment reduces context by >=30% while preserving output quality.
2330
+
2331
+## Key risks
2332
+
2333
+- Matrixing GitHub search can trade wall-clock for rate-limit instability.
2334
+- Bad merge semantics can corrupt star-gain trends, duplicate repos, or lose source provenance.
2335
+- Matrix artifacts increase operational complexity and can make rebuild/hydration paths brittle.
2336
+- Map/reduce can lose nuance if slice summaries omit counterexamples or citations.
2337
+- Retry isolation can hide systemic failures unless fan-in produces clear status and gates.
2338
+
2339
+---
.squad/decisions.md
+522
-577
@@ -1,29 +1,5 @@
1
-# Squad Decisions
2
-
3
-# Squad Decisions
4
-
5
-## Impact
6
-
7
-Applies to future weekly summaries and any generator work that consumes `data/analyzed/*-summary.md`.
8
-
9
----
10
-
11
----
12
-
1
# Fry: quality gate fallback hardening
2
15
-Date: 2026-06-01
16
-
17
-## Context
18
-Issue #217 showed the weekly analysis job can fail even when crawl data is healthy because Copilot sometimes returns a generic placeholder title or no output file at all after retries.
19
-
20
-## Decision
21
-Keep Copilot CLI as the primary analysis generator, but if its output still fails the quality gate after retries, immediately fall back to `scripts/analyze_fallback.py` via GitHub Models. Also render the prompt with concrete `week`, `year`, and title guidance so the model is less likely to echo placeholder frontmatter.
22
-
23
-## Rationale
24
-This keeps the higher-quality primary path, but removes CI flakiness from transient Copilot failures and from prompt placeholders leaking into the final markdown.
25
-
26
----
3
4
---
5
@@ -34,26 +10,6 @@ This keeps the higher-quality primary path, but removes CI flakiness from transi
10
- Decision: Do not bypass the review gate on self-authored pull requests. Treat independent approval as still required before merging branches opened by the same account Leela is operating under.
11
- Why: GitHub blocks self-approval, and preserving the review gate matters more than forcing a merge from the lead seat.
12
37
-## Cost Transparency Placement (2026-05-25)
38
-
39
-**Decision:** The AI pipeline cost dashboard is part of `/about/` under a Pipeline transparency section, with the existing `/dashboard/` page retained as a direct audit link that reuses the same shortcode.
40
-
41
-**Rationale:** Cost reporting is operational transparency, not a primary navigation destination or product dashboard. Keeping it on About matches the editorial-restrained redesign while preserving the old URL for references.
42
-
43
-## Nibbler Review Gate for External-Facing Artifacts (2026-05-25)
44
-
45
-**Source:** Nibbler audit recommendation
46
-**Adopted by:** Leela
47
-**Status:** Adopted
48
-
49
-External-facing launch and announcement artifacts require Nibbler review before publication or merge. This includes Hacker News posts, LinkedIn announcements, Bluesky threads, Reddit posts, launch blogs, press copy, launch graphics, and similar materials that will appear outside this repository.
50
-
51
-PRs that ship this copy or graphics must tag `@squad:nibbler` for RAI sign-off and use the [Responsible AI checklist](skills/responsible-ai-review/SKILL.md) (`.squad/skills/responsible-ai-review/SKILL.md`) before merge.
52
-
53
-**Rationale:** Distribution copy can create reputational, safety, accessibility, or policy risk even when the underlying code is unchanged. Nibbler provides the hostile-reader and responsible-AI perspective before users encounter the material.
54
-
55
----
56
-
13
# Amy — Topic buttons follow-up
14
15
- Date: 2026-06-01
@@ -65,729 +21,718 @@ PRs that ship this copy or graphics must tag `@squad:nibbler` for RAI sign-off a
21
22
# Fry: quality gate fallback hardening
23
68
-Date: 2026-06-01
69
-
70
-## Context
71
-Issue #217 showed the weekly analysis job can fail even when crawl data is healthy because Copilot sometimes returns a generic placeholder title or no output file at all after retries.
72
-
73
-## Decision
74
-Keep Copilot CLI as the primary analysis generator, but if its output still fails the quality gate after retries, immediately fall back to `scripts/analyze_fallback.py` via GitHub Models. Also render the prompt with concrete `week`, `year`, and title guidance so the model is less likely to echo placeholder frontmatter.
75
-
76
-## Rationale
77
-This keeps the higher-quality primary path, but removes CI flakiness from transient Copilot failures and from prompt placeholders leaking into the final markdown.
78
-
79
----
24
25
---
26
27
# Fry — generate-step failure handling
28
85
-Date: 2026-06-01
86
-
87
-## Context
88
-Issue #220 showed the crawl-and-publish workflow could finish crawl and analysis successfully, then fail in the generate handoff because the generated weekly page path was absolute while the publish-branch restore logic assumed a repository-relative path. The same workflow also lacked a failure-to-issue bridge, so repeated pipeline failures did not automatically open or update a GitHub issue.
89
-
90
-## Decision
91
-Normalize `page_path` to a repo-relative `content/weekly/...` path inside the generate commit step before copying weekly output onto the publish branch. Add a dedicated `notify-failure` job that always evaluates after the pipeline jobs and creates or updates a GitHub issue whenever any crawl/analyze/generate/deploy/notify job fails.
92
-
93
-## Rationale
94
-The path normalization fixes the actual handoff bug without changing `scripts/generate_content.py`, which already returns an absolute file path used elsewhere in tests. A separate failure notifier makes regressions visible even when later jobs are skipped, which is the exact reliability gap that hid the recent failures.
95
-
96
----
29
30
---
31
32
# Amy — Share button implementation
33
102
-Date: 2026-06-01
103
-
104
-## Context
105
-Issue #226 adds article-level sharing. PaperMod already ships a share-buttons partial, but SquadScope also needs mobile-native sharing through the Web Share API and token-aligned styling.
106
-
107
-## Decision
108
-Enable PaperMod share support through `hugo.toml` (`params.ShowShareButtons` plus an explicit `params.ShareButtons` allowlist), then override `layouts/partials/share_icons.html` in the project to add a mobile-only native share button while keeping desktop fallback links for X, LinkedIn, and Facebook. To keep the site buildable with the current PaperMod submodule layout, vendor the theme partials the site already relies on into `layouts/partials/` instead of editing the theme.
109
-
110
-## Rationale
111
-This keeps the third-party theme submodule untouched, reuses the existing article-footer insertion point, and scopes the share customization to a project-level partial plus tokenized footer styles. Vendoring the required PaperMod partials also makes the build deterministic for SquadScope without depending on theme-internal `_partials` resolution quirks.
112
-
113
----
34
35
---
36
37
# Farnsworth — Hindsight validation decision
38
119
-Date: 2026-06-01
120
-
121
-## Decision
122
-Use an optional `predictions` frontmatter registry on weekly analysis summaries with entries shaped as `{repo, direction, confidence}`.
123
-
124
-## Why
125
-The published markdown is already the durable editorial artifact, so embedding prediction intent there avoids a separate ledger drifting out of sync. Legacy summaries still need heuristic extraction from Signal/Noise/Gaps prose, but future summaries should register explicit repo-level calls for cleaner hindsight scoring.
126
-
127
-## Operational note
128
-The validator writes a human scorecard to `.squad/reskill/scorecards/YYYY-WNN.md` and a machine-readable companion to `data/metrics/scorecards/YYYY-WNN-scorecard.json` so the current reskill tooling can ingest the same run.
129
-
130
----
39
40
---
41
42
# Fry — Generate-step failure handling
43
136
-Date: 2026-06-01
137
-
138
-## Context
139
-Issue #220 showed the crawl-and-publish workflow could finish crawl and analysis successfully, then fail in the generate handoff because the generated weekly page path was absolute while the publish-branch restore logic assumed a repository-relative path. The same workflow also lacked a failure-to-issue bridge, so repeated pipeline failures did not automatically open or update a GitHub issue.
140
-
141
-## Decision
142
-Normalize `page_path` to a repo-relative `content/weekly/...` path inside the generate commit step before copying weekly output onto the publish branch. Add a dedicated `notify-failure` job that always evaluates after the pipeline jobs and creates or updates a GitHub issue whenever any crawl/analyze/generate/deploy/notify job fails.
143
-
144
-## Rationale
145
-The path normalization fixes the actual handoff bug without changing `scripts/generate_content.py`, which already returns an absolute file path used elsewhere in tests. A separate failure notifier makes regressions visible even when later jobs are skipped, which is the exact reliability gap that hid the recent failures.
146
-
147
----
44
45
---
46
47
# Farnsworth hindsight validation decision
48
153
-Date: 2026-06-01
154
-
155
-## Decision
156
-Use an optional `predictions` frontmatter registry on weekly analysis summaries with entries shaped as `{repo, direction, confidence}`.
157
-
158
-## Why
159
-The published markdown is already the durable editorial artifact, so embedding prediction intent there avoids a separate ledger drifting out of sync. Legacy summaries still need heuristic extraction from Signal/Noise/Gaps prose, but future summaries should register explicit repo-level calls for cleaner hindsight scoring.
160
-
161
-## Operational note
162
-The validator writes a human scorecard to `.squad/reskill/scorecards/YYYY-WNN.md` and a machine-readable companion to `data/metrics/scorecards/YYYY-WNN-scorecard.json` so the current reskill tooling can ingest the same run.
163
-
164
----
49
50
---
51
52
# Fry QA triage decision
53
170
-Date: 2026-06-05T15:36:19.379+00:00
171
-
172
-## Decision
173
-
174
-The crawl-and-publish analysis stage should degrade to a data-only no-AI weekly summary when both Copilot output and GitHub Models output are unavailable or rejected by the quality gate.
175
-
176
-## Rationale
177
-
178
-A missing or unauthorized model is an operational dependency failure, but the pipeline still has verified crawl data. Publishing a clearly labeled data-only summary is more reliable than failing the entire weekly handoff after preserving no reader-facing output.
179
-
180
-## Follow-up
181
-
182
-If model access is restored, the AI analysis path remains preferred. The no-AI path is only a terminal fallback after Copilot and GitHub Models fail.
183
-
184
----
54
55
---
56
57
# Leela: Close unverifiable W23 growth execution
58
190
-Date: 2026-06-05T15:36:19.379+00:00
191
-
192
-**By:** Leela
193
-
194
-## Decision
195
-
196
-Issue #188 was closed as obsolete/unverifiable rather than reconstructed or rerouted. W23 draft files under `.squad/posts/`, the requested `.squad/metrics/2026/w23-distribution.md`, and platform posting evidence were absent from the working tree, git history, related issues, and PR context. PR #190 and `docs/growth/distribution-strategy.md` only provide the launch strategy/template, not the W23 execution artifacts.
197
-
198
-## Rationale
199
-
200
-Recreating social posts and metrics after the distribution window would create misleading evidence. Future growth execution issues should remain open until artifact-backed proof exists, or be closed explicitly when the posting window expires without evidence.
201
-
202
----
59
60
---
61
62
# Fry PR #236 QA Review
63
208
-Date: 2026-06-05T15:36:19.379+00:00
209
-
210
-PR #236 keeps RSS enrichment in the existing crawl job with bounded in-process parallel fetching instead of separate Actions jobs.
211
-
212
-QA verified the diff covers config loading, multi-source crawl aggregation, metadata/errors, legacy `*-techcrunch.json` fallback, correlation handoff, press-context resolution, and rebuild hydration.
213
-
214
-Validation run in an isolated PR worktree:
215
-- `PYTHONPATH=. .venv/bin/python -m pytest tests -q` → 554 passed
216
-- Live RSS smoke with `--max-workers 5` → 54 articles from 5 sources, no feed errors
217
-
218
-Verdict: approve; no follow-up implementation owner required.
219
-
220
----
64
65
---
66
67
# Hermes security review — PR #236 external RSS feeds
68
226
-Date: 2026-06-05T15:36:19.379+00:00
227
-
228
-## Verdict
229
-
230
-Request changes before merge.
231
-
232
-## Rationale
233
-
234
-PR #236 keeps workflow secrets out of the RSS step and does not add new dependency classes, but the new config-driven fetcher currently trusts `feed_url` values without enforcing scheme/host boundaries and calls `feedparser.parse(url)` without an explicit per-request timeout. Because the workflow runs this in CI and later grants `contents: write` in the same job, external-network behavior should fail closed around the intended RSS allowlist and fail fast on slow/unresponsive feeds.
235
-
236
-## Required fixes
237
-
238
-- Validate source config with `urllib.parse.urlparse()` before crawling:
239
- - require `https`;
240
- - require hostnames to match the repository-owned allowlist for the five intended feeds;
241
- - reject credentials, local/private/link-local hosts, and unexpected ports.
242
-- Fetch feeds through a code path with explicit timeout and bounded retry/backoff behavior; do not rely on the default socket timeout.
243
-- Keep bounded concurrency; optionally validate `--max-workers` to a safe range.
244
-
245
-## Suggested owner
246
-
247
-Bender should own the fixes so Leela does not review her own implementation changes.
248
-
249
----
69
70
---
71
72
# PR #236 security unblock
73
255
-Date: 2026-06-05T16:00:00+00:00
256
-
257
-Hermes re-reviewed PR #236 at Bender fix commit `e91e2a5b33b816191148125d40192b3fff8fbc6a`.
258
-
259
-Security blockers from the prior review are resolved:
260
-- external RSS feed URLs are parsed with `urllib.parse.urlparse()` and restricted to HTTPS on the approved host allowlist;
261
-- credentials, localhost/local domains, private/link-local IP literals, invalid ports, and non-443 ports are rejected;
262
-- RSS fetches use `urlopen(..., timeout=DEFAULT_FETCH_TIMEOUT_SECONDS)` with bounded retry attempts;
263
-- parallel RSS crawling caps workers at `DEFAULT_MAX_WORKERS` and rejects `--max-workers < 1`;
264
-- tests cover unsafe URL rejection and explicit timeout propagation.
265
-
266
-Validation: `PYTHONPATH=. python -m pytest tests -q` in an isolated PR worktree passed with 563 tests.
267
-
268
-Decision: Hermes security approval/unblock for merge, with CodeQL checks green on the PR.
269
-
270
----
74
75
---
76
77
# Bender — Crawler parallelism analysis
78
276
-Date: 2026-06-05T16:26:00Z
277
-Requested by: jmservera
278
-Inputs:
279
-- Old crawler job: https://github.com/jmservera/SquadScope/actions/runs/26753498571/job/78847225991
280
-- New crawler job: https://github.com/jmservera/SquadScope/actions/runs/27026348186/job/79767247136
281
-
282
-## Observations from job logs
283
-
284
-### Old run — single TechCrunch RSS source
79
286
-Run `26753498571`, job `78847225991`, head `59b45137fc3ad674276b1ff8c0aa743d8e43d1bb`:
287
-
288
-- `crawl` job duration: 2026-06-01 11:58:51Z → 12:05:14Z, about 6m23s.
289
-- `Run crawler`: 11:59:02Z → 12:05:00Z, about 5m58s.
290
-- `Crawl TechCrunch RSS`: started and completed at 12:05:06Z in the step timing metadata, effectively sub-second.
291
-- GitHub crawl summary: `Wrote data/raw/2026-W23.json with 196 new repos and 238 trending repos, saved data/snapshots/2026-W23-stars.json, used 447 API calls, and served 0 cache hits.`
292
-- RSS summary: `Crawled 20 articles (7 relevant) → data/raw/2026-W23-techcrunch.json`.
293
-- Rate-limit evidence: 447 rate-limit log lines; 6 search calls and 441 core calls. Minimum observed remaining quota was 24 search requests out of 30, and final core quota was 4556/5000.
294
-- Retry/flakiness evidence: 0 `Retrying`, 0 stale-cache fallbacks, 0 search failures in the filtered log summary. The `warning`/`error` counts visible in the raw filtered scan are from workflow script text/hints, not crawler failures.
80
+---
81
296
-### New run — five external RSS sources, in-process parallelism
82
+# Farnsworth: LLM input strategy for multi-source news
83
298
-Run `27026348186`, job `79767247136`, head `87e55a227da78b86e9677acc96460968196e9e5a`:
84
300
-- `crawl` job duration: 2026-06-05 16:15:41Z → 16:20:49Z, about 5m08s.
301
-- `Run crawler`: 16:15:49Z → 16:20:36Z, about 4m47s.
302
-- `Crawl external news RSS feeds`: 16:20:42Z → 16:20:43Z, about 1s.
303
-- GitHub crawl summary: `Wrote data/raw/2026-W23.json with 213 new repos and 236 trending repos, saved data/snapshots/2026-W23-stars.json, used 455 API calls, and served 0 cache hits.`
304
-- External RSS summary: `Crawled 54 articles from 5 sources (27 relevant) → data/raw/2026-W23-external-news.json`.
305
-- Rate-limit evidence: 455 rate-limit log lines; 6 search calls and 449 core calls. Minimum observed remaining quota was 24 search requests out of 30, and final core quota was 4458/5000.
306
-- Retry/flakiness evidence: 0 `Retrying`, 0 stale-cache fallbacks, 0 search failures. The new RSS stage did not visibly bottleneck the job.
85
+---
86
308
-## Current implementation shape reviewed
87
+# Fry QA: crawler reliability and performance next iteration
88
310
-The newer workflow revision changes the RSS stage from a single TechCrunch output to:
89
312
-```yaml
313
-python scripts/techcrunch_crawler.py \
314
- --sources config/external_news_sources.json \
315
- --output "data/raw/${WEEK}-external-news.json" \
316
- --since "$SINCE"
317
-```
90
+---
91
319
-The external source config contains five approved feeds: TechCrunch, NVIDIA Blog, Hugging Face Blog, MIT Technology Review, and GitHub Blog.
92
+# Leela — crawler next-iteration issue
93
321
-The new crawler implementation:
94
323
-- validates feed URLs against an HTTPS host allowlist;
324
-- fetches RSS with an explicit 15s timeout;
325
-- uses `ThreadPoolExecutor` with `max_workers=min(requested_or_source_count, source_count, 8)`;
326
-- records per-source article `source` fields;
327
-- writes one merged `external_news` artifact with metadata including `source_count`, `sources_with_articles`, totals, GitHub links, and `errors`.
95
+---
96
329
-## Topology options
97
+# Bender PR #236 Security Fix
98
331
-### Option A — keep bounded in-process parallelism in one job
99
+## Context
100
+Hermes blocked PR #236 because config-driven external RSS sources were fetched directly without egress URL validation or explicit per-request timeouts.
101
333
-Best fit for the current source count.
102
+## Decision
103
+External news RSS source configs now require HTTPS URLs whose host is in the approved feed allowlist, with credentials, local/private/link-local targets, and unexpected ports rejected before crawl. Fetching now goes through `urllib.request.urlopen` with an explicit bounded timeout before handing bytes to `feedparser`, while retaining the existing config-driven source list and bounded in-process worker pool.
104
335
-Pros:
336
-- Fast enough now: five-source RSS stage adds about 1s in the new run.
337
-- No extra checkout/setup/artifact overhead per source.
338
-- Keeps one downstream news artifact contract, which matches `correlate.py` and `render_press_context.py` expectations.
339
-- A source failure can be represented inside `metadata.errors` without failing the entire crawl.
105
+## Validation
106
+Added tests for invalid/unapproved URL rejection and explicit fetch timeout propagation. Ran `PYTHONPATH=. .venv/bin/python -m pytest tests -q` with 563 passing tests.
107
341
-Cons:
342
-- If one feed hangs until timeout, the RSS step is bounded by timeout plus retry delay for that source.
343
-- GitHub Actions cannot independently retry only one failed source.
344
-- Per-source logs are less visible unless the script emits explicit source start/end/error lines.
108
+---
109
346
-### Option B — GitHub Actions matrix per source/type
110
+---
111
348
-Not justified yet for the RSS feeds alone.
112
+# Leela — Issue 234 external news source architecture
113
350
-Pros:
351
-- Clean isolation and per-source retry visibility.
352
-- Natural if sources become heterogeneous: RSS, APIs, browser crawls, paid sources, or sources with independent secrets/quotas.
353
-- Failure policy can vary by source.
114
355
-Cons:
356
-- More runner minutes and more setup overhead than the current 1s RSS crawl.
357
-- Requires explicit merge job and stricter artifact naming/schema validation.
358
-- Increases race/branch commit complexity if matrix outputs are committed directly.
359
-- Does not help the actual current bottleneck, which is the GitHub repo crawl step at roughly 4m47s–5m58s.
115
+# Fry — Issue #238 notify triage
116
361
-### Option C — hybrid/staged topology
117
363
-Recommended next iteration, but staged lightly: keep RSS in-process now, make the artifact contract merge-ready, and add a separate merge/validate step before analysis.
118
+---
119
365
-Pros:
366
-- Preserves current speed and simplicity.
367
-- Creates a clean future migration path to a matrix without changing analysis consumers.
368
-- Lets the pipeline distinguish crawler collection from artifact assembly/validation.
369
-- Gives downstream stages one canonical `external-news` artifact regardless of whether collection was single-process or matrix.
120
+# Leela PR #241 Review — Idempotent Weekly Release Notify
121
371
-Cons:
372
-- Adds one small script/step for validation/merge even before a matrix is needed.
373
-- Requires schema versioning discipline.
122
+- Date: 2026-06-05
123
+- Context: Issue #238 showed a real rerun failure in `notify`: `gh release create week-2026-W23` returned HTTP 422 because the weekly release already existed.
124
+- Decision: Keep weekly release notification idempotent by resolving the weekly tag first, editing an existing `week-*` release with `gh release edit`, and creating only when no release exists.
125
+- Review result: Approved in substance. Formal GitHub approval was blocked because the authenticated account is the PR author, so Leela posted an explicit lead approval comment instead of bypassing the review gate.
126
+- Validation: `tests/test_pipeline.py` passed locally (9 tests), full `tests` passed locally (563 tests), CodeQL checks were green, and Copilot PR review completed with no comments.
127
+- Merge gate: Do not merge from this account until the repository's independent-review requirement for `jmservera`-authored PRs is satisfied.
128
+- PR #241 merged at 2026-06-05T17:21:05Z, closing issue #238.
129
375
-## Recommendation
130
+---
131
377
-Use a hybrid/staged approach:
132
+---
133
379
-1. Keep the current bounded in-process parallel RSS crawl for the next iteration.
380
-2. Add explicit per-source logs: start time, duration, article count, relevant count, GitHub-link count, and error if any.
381
-3. Add `schema_version` and stable `sources_requested` / `sources_succeeded` / `sources_failed` metadata to `external-news.json`.
382
-4. Add a validation/merge script that accepts either:
383
- - current single merged external-news payload, or
384
- - future per-source payloads named like `external-news-${source}.json`.
385
-5. Make analysis consume only the canonical merged artifact: `data/raw/${WEEK}-external-news.json`.
386
-6. Move to an Actions matrix only when evidence shows RSS collection is material: e.g. external source stage exceeds 60s p95, source count exceeds about 12–15, or a source requires independent credentials/rate policy.
134
+# Bender issue #237 implementation
135
388
-## Risks
136
390
-- Current metadata has `errors`, but success criteria are ambiguous. A total RSS outage could still return exit 0 if errors are recorded but no minimum-source gate exists.
391
-- The script name `techcrunch_crawler.py` is now misleading for multi-source external news. Rename later only with backward-compatible CLI/wrapper to avoid breaking existing docs/tests.
392
-- The current logs show `0 cache hits` in both old and new GitHub crawls, so cache restoration is not reducing runtime in these examples. That may be due to TTL/query churn or artifact mismatch and should be investigated separately from RSS topology.
393
-- Search API quota is the tighter GitHub limit: both runs reached minimum remaining 24/30 search requests while core remained above 4450/5000. More GitHub search parallelism would risk secondary/rate-limit pressure; RSS parallelism does not consume GitHub API quota.
137
+---
138
395
-## Acceptance criteria for next implementation
139
+# Bender PR #242 Copilot Review Fixes
140
397
-- A weekly crawl with five RSS sources still completes the external RSS stage in under 30s under normal network conditions.
398
-- The RSS crawler logs one concise summary line per source with duration and counts.
399
-- `data/raw/${WEEK}-external-news.json` includes `schema_version`, `source_count`, `sources_requested`, `sources_succeeded`, `sources_failed`, `sources_with_articles`, and `errors`.
400
-- The workflow fails only when the required GitHub raw payload is missing or the external-news artifact is structurally invalid; individual optional RSS source failures are recorded and do not block analysis unless fewer than an agreed minimum number of sources succeed.
401
-- Rebuild mode hydrates the canonical external-news artifact and remains backward-compatible with legacy `${WEEK}-techcrunch.json`.
402
-- Correlation and press-context steps read the canonical merged artifact and do not need to know whether collection was in-process or matrix-based.
403
-- Tests cover single merged payload validation plus simulated future per-source merge inputs.
141
+- Keep category/project-name-only press matches weak even when temporally spiking or corroborated by multiple articles/sources.
142
+- Pass both `--since` and `--until` from the crawl workflow to preserve deterministic canonical `crawl_window` metadata.
143
+- Record bounded fetch attempts and timeout telemetry on `NewsFeedSource` even when `fetch_feed()` raises before returning a feed.
144
+- Keep press-context article lookup comments aligned with the actual URL-to-title mapping.
145
+- PR #243 merged at 2026-06-05T17:34:18Z.
146
147
---
148
149
---
150
409
-# Farnsworth: LLM input strategy for multi-source news
410
-
411
-Date: 2026-06-05T16:26:00.133+00:00
412
-
413
-## Context
414
-
415
-The old crawler run (`26753498571` / job `78847225991`) produced:
416
-
417
-- `data/raw/2026-W23.json`: 196 new repos, 238 trending repos, 447 GitHub API calls.
418
-- `data/raw/2026-W23-techcrunch.json`: 20 TechCrunch articles, 7 relevant.
419
-
420
-The new crawler run (`27026348186` / job `79767247136`) produced:
151
+# Leela PR #243 Review
152
422
-- `data/raw/2026-W23.json`: 213 new repos, 236 trending repos, 455 GitHub API calls.
423
-- `data/raw/2026-W23-external-news.json`: 54 articles from 5 sources, 27 relevant, no feed errors.
424
-- Source mix: TechCrunch 20, NVIDIA Blog 13, Hugging Face Blog 9, MIT Technology Review 10, GitHub Blog 2.
153
+- Verdict: approved in substance after independent lead review.
154
+- Scope checked: issue #237 acceptance criteria follow-up, PR #242 Copilot comments, PR #243 diff, tests, CodeQL, Copilot review state.
155
+- Local validation: clean PR worktree ran `pytest tests -q` with 574 passed.
156
+- Formal GitHub approval blocked: the active account is the PR author and GitHub rejected own-PR approval.
157
+- Merge gate: wait for an independent non-Bender reviewer/approval unless repository policy explicitly permits merge with the lead approval comment.
158
426
-The external-news artifact is roughly 45.5 KB / 11.4k token-estimate by itself; the GitHub raw artifact from the same run is roughly 296 KB / 74k token-estimate. Existing rendered press context can also be large: the W23 TechCrunch-only press context on `publish` is about 27.9 KB / 7k token-estimate before adding the extra sources.
159
+---
160
428
-## Analyst assessment
161
+---
162
430
-Do not send all raw GitHub and all raw external-news inputs directly to the weekly analysis model. That path is editorially fragile: the model will spend attention on repeated article summaries, source boilerplate, low-relevance items, and broad category matches instead of the actual job — deciding what matters. It also increases prompt-injection surface and makes limited-context models more likely to drop required sections, lose citations, or overfit the latest/longest source.
163
+### 2026-06-05T17:06:31.753+00:00: User directive — Copilot Review Asynchronous Gate
164
432
-The current analysis contract already expects a concise `Where Industry Meets Code` comparison, not a press digest. External news should therefore enter analysis as a compact, source-aware correlation artifact: a deterministic press-context file that preserves the top evidence and citations while discarding bulk article text.
165
+**By:** jmservera (via Copilot)
166
+**What:** Copilot Review is asynchronous. Before merging a PR, check whether Copilot is still reviewing and do not merge until the review has finished and any review comments are handled.
167
+**Why:** User request — captured for team memory
168
434
-## Options considered
169
+---
170
436
-### 1. Pass every raw input at once
171
+# Bender input — crawl matrix and map/reduce PRD
172
438
-**Pros**
439
-- Maximum recall.
440
-- Simplest implementation if context windows are assumed unlimited.
173
442
-**Cons**
443
-- Poor fit for limited-context or cheaper fallback models.
444
-- Increases prompt size from already-large GitHub raw payloads into 90k+ token territory before learned state and instructions.
445
-- Encourages article summarization instead of repo-to-industry synthesis.
446
-- Makes the quality gate less reliable because structural failures, missing references, and citation drift become more likely.
447
-- Treats all sources equally even when some are lower relevance for developer adoption.
174
+# Farnsworth — PRD input: LLM analysis map/reduce
175
449
-**Analyst verdict:** Reject for the default path.
176
+Date: 2026-06-05T17:42:56.819+00:00
177
+Requested by: jmservera
178
451
-### 2. Pre-merge and summarize all sources into one artifact
179
+## Recommendation
180
453
-**Pros**
454
-- Keeps the analyzer prompt smaller.
455
-- Gives the model one stable press evidence surface.
456
-- Easier to validate than source-specific LLM steps.
181
+Adopt a staged map/reduce design for the **LLM analysis stage**, but do not start by splitting the raw crawl job into a GitHub Actions matrix for speed alone. Existing evidence says RSS collection is already fast and in-process parallelized, while the GitHub crawl dominates crawl runtime. The stronger reason for map/reduce is **analysis quality and reliability under context pressure**: smaller mapper calls can extract cited, typed claims from bounded evidence windows, and one reducer can preserve the weekly editorial voice and final `docs/analysis-spec.md` contract.
182
458
-**Cons**
459
-- If summarization is LLM-generated, it can lose citations or compound hallucinations before the main analysis.
460
-- If it simply concatenates all sources, it still carries noise.
461
-- Needs source provenance to avoid TechCrunch/GitHub/NVIDIA/MIT/HF being flattened into one undifferentiated "press" voice.
183
+Initial PRD should target an experimental path behind a feature flag or dry-run workflow, with deterministic compaction and validation before any generated weekly summary becomes publishable.
184
463
-**Analyst verdict:** Good only if deterministic and citation-preserving.
185
+## Current context pressure
186
465
-### 3. Run staged source-specific LLM analyses
187
+Current weekly analysis input is already large before the model writes anything:
188
467
-**Pros**
468
-- Keeps each model call small.
469
-- Can produce richer source-by-source editorial nuance.
470
-- Scales if future source count grows substantially.
189
+- GitHub raw crawl is the dominant payload. The W23 multi-source run recorded in `.squad/decisions.md` produced `213` new repos and `236` trending repos, roughly `296 KB` / `74k` token-estimate in raw GitHub JSON.
190
+- External news expanded from one TechCrunch feed to five sources. W23 external news was `54` articles / `27` relevant articles, roughly `45.5 KB` / `11.4k` token-estimate.
191
+- Rendered press context is intentionally capped by `scripts/render_press_context.py` at an `8k` token-estimate budget, but prompt-mode output can still include ranked articles, correlations, divergences, caveats, and telemetry.
192
+- The weekly prompt itself injects raw JSON, previous summary, `.squad/identity/wisdom.md`, all `.squad/skills/**/*.md`, analysis instructions, security constraints, and optional press context. This overhead competes with repo evidence for attention.
193
+- `analysis_gate.py` is structural: it enforces frontmatter, required headings, word count, placeholder/raw JSON bans, quality score, dates, and repo format. It does not yet validate intermediate faithfulness, mapper contradictions, or claim-level citation integrity.
194
472
-**Cons**
473
-- Higher cost and more failure points.
474
-- Second-stage analyzer may inherit summaries without enough evidence.
475
-- Quality gate currently validates final structure, not the faithfulness of intermediate source briefs.
476
-- More operational complexity than current volume justifies.
195
+The current failure mode is not only token overflow. It is attention dilution: long raw inputs encourage listing, citation drift, missed required headings, weak press/repo correlation claims, and generic summaries. A map/reduce design should reduce evidence windows and force explicit claim contracts before final prose.
196
478
-**Analyst verdict:** Defer. Consider only when relevant article volume regularly exceeds the compact artifact budget.
197
+## Why a matrix was not used for the crawl
198
480
-### 4. Use compact correlation / press-context artifact
199
+The previous crawler analysis supports not using an Actions matrix yet for RSS/source crawling:
200
482
-**Pros**
483
-- Best match for the weekly brief: correlations, divergences, citations, and source provenance are preserved.
484
-- Keeps the LLM focused on editorial judgment instead of raw article triage.
485
-- Can be generated deterministically and tested.
486
-- Supports fallback models and no-AI fallback more safely.
201
+- New five-source RSS collection took about one second in the observed run; GitHub repo crawl remained about 4m47s.
202
+- `scripts/techcrunch_crawler.py` already uses bounded in-process parallel source fetching in the newer pipeline.
203
+- Matrix jobs would add checkout/setup/artifact/merge overhead and commit-race complexity without addressing the actual bottleneck.
204
+- A matrix becomes justified when source count, source heterogeneity, source-specific credentials/quotas, or p95 external collection latency materially increases.
205
488
-**Cons**
489
-- Requires explicit ranking and truncation rules.
490
-- Bad correlation heuristics can still inject false positives, especially category-only matches.
491
-- Needs quality gates that check citation preservation, not just markdown shape.
206
+For the PRD, separate **crawl parallelism** from **analysis decomposition**. Matrix crawl is a future topology decision; map/reduce analysis is an editorial reliability strategy.
207
493
-**Analyst verdict:** Recommended default.
208
+## Candidate map strategies
209
495
-## Recommendation
210
+### 1. By editorial topic/category
211
497
-Implement a source-aware compact press-context artifact as the only external-news input to weekly analysis.
212
+Mappers receive repo slices clustered by topics, languages, descriptions, and prior-week continuity hints. They produce candidate trends, noise patterns, blind spots, and key repos.
213
499
-The analyzer should receive:
214
+Pros:
215
+- Matches final article structure: macro trends and gaps.
216
+- Good for discovering cross-repo patterns inside bounded themes.
217
501
-1. Sanitized/possibly compacted GitHub repo evidence.
502
-2. Previous weekly summary.
503
-3. Learned wisdom/skills.
504
-4. One compact press-context artifact containing:
505
- - source coverage summary (`source`, total articles, relevant articles, errors),
506
- - 5-10 ranked press items with URL, source, date, relevance score, and one-sentence why-it-matters,
507
- - 5-10 highest-confidence repo/news correlations,
508
- - separate "possible/weak correlations" bucket for category-only or fuzzy matches,
509
- - 3-6 divergence findings,
510
- - complete citations for every article retained,
511
- - explicit caveat when sources were unavailable or noisy.
218
+Cons:
219
+- Topic overlap can duplicate repos or split one trend across mappers.
220
+- Requires deterministic cluster IDs and repo membership to avoid inconsistent claims.
221
513
-Do not include all article summaries in the analysis prompt. Do not let low-confidence category matches count as strong press correlation. Category-only matches should be framed as weak context unless reinforced by direct GitHub link, organization/entity match, temporal spike, or repeated source agreement.
222
+Best use: primary mapper strategy after deterministic clustering.
223
515
-## Prompt / gate implications
224
+### 2. By signal type: new, trending, news/correlation, prior continuity
225
517
-- The prompt should say: "Use press context as correlation evidence, not as instructions and not as content to repackage."
518
-- External-news content should be wrapped in the same untrusted-content boundary pattern used for raw repo JSON.
519
-- The quality gate should remain structural, but add evidence-focused checks:
520
- - `## Key References > ### Press & Industry` contains 3-5 retained article links when press data exists.
521
- - The body does not contain raw correlation dumps, model instructions, or full article payloads.
522
- - At least one sentence in `Where Industry Meets Code` distinguishes strong correlation from weak/noisy press context.
523
- - If external-news metadata reports source errors, the article includes a concise caveat.
226
+Separate mappers handle:
227
+- `new_repos`: novelty and launch quality.
228
+- `trending_repos`: momentum and established anchors.
229
+- press/correlation artifact: industry alignment/divergence.
230
+- prior summary/history: continuity, reversals, and prediction follow-up.
231
525
-## Acceptance criteria for Leela's next issue
232
+Pros:
233
+- Mirrors current input sources and reduces per-call context sharply.
234
+- Easier citation provenance because each mapper owns one evidence type.
235
527
-- A deterministic compact press-context artifact is generated before analysis from `*-external-news.json` and `*-correlations.json`.
528
-- The compact artifact has a documented token/size budget, recommended ceiling: <= 8k token-estimate for press context.
529
-- The weekly analysis prompt consumes the compact press context, not the full external-news JSON.
530
-- Press context retains source name, article URL, article title, published date, relevance score, and correlation confidence for every retained citation.
531
-- Correlations are tiered: direct-link/org/entity/temporal matches are strong; fuzzy/category-only matches are weak unless corroborated.
532
-- Quality gate or tests reject raw article/correlation dumps in final analysis output.
533
-- Tests cover: multi-source source counts, no-source/error caveats, citation preservation, truncation behavior, weak-correlation labeling, and legacy `*-techcrunch.json` fallback.
534
-- The final weekly summary still conforms to `docs/analysis-spec.md`: required frontmatter, stable H2 sections, complete Key References, no placeholders, no raw JSON/tool logs.
236
+Cons:
237
+- Final trends often require combining new + trending + press evidence.
238
+- Reducer needs stronger dedupe and conflict logic.
239
536
-## Editorial success metric
240
+Best use: strong baseline because it requires little new clustering machinery.
241
538
-The finished weekly brief should make fewer but sharper press claims: "what the industry narrative explains, what developer activity confirms, and what the press is missing." It should not become a five-source news roundup.
242
+### 3. By source
243
540
----
244
+Mappers summarize each external source or source family, preserving source name, URL, article title, date, relevance score, and correlation confidence.
245
542
----
246
+Pros:
247
+- Keeps source provenance clear.
248
+- Prevents TechCrunch/GitHub/NVIDIA/MIT/HF from becoming one flattened press voice.
249
544
-# Fry QA: crawler reliability and performance next iteration
250
+Cons:
251
+- Risk of over-weighting press summaries in a GitHub-first analysis.
252
+- More LLM calls for relatively small article volume.
253
546
-Date: 2026-06-05T16:26:00Z
547
-Requested by: jmservera
254
+Best use: only if relevant article volume exceeds the compact press-context budget or source mix becomes heterogeneous.
255
549
-## Evidence reviewed
256
+### 4. By repository clusters
257
551
-- Old crawl job `26753498571 / 78847225991`: crawl job 11:58:51–12:05:14 (~6m23s). GitHub crawl wrote `data/raw/2026-W23.json` with 196 new repos, 238 trending repos, 447 API calls, 0 cache hits. Single TechCrunch RSS step produced 20 articles / 7 relevant. Raw artifact: 199,434 bytes; cache artifact: 10,188,525 bytes.
552
-- New crawl job `27026348186 / 79767247136`: crawl job 16:15:41–16:20:49 (~5m08s). GitHub crawl wrote 213 new repos, 236 trending repos, 455 API calls, 0 cache hits. External RSS step produced 54 articles from 5 sources / 27 relevant. Raw artifact: 207,606 bytes; cache artifact: 11,874,886 bytes.
553
-- Current `origin/main` workflow runs GitHub crawl first, then a single in-process parallel `scripts/techcrunch_crawler.py --sources config/external_news_sources.json` step, uploads one `raw-data` artifact, and analysis falls back from `{week}-external-news.json` to legacy `{week}-techcrunch.json`.
554
-- Existing tests cover source config validation, allowlisted HTTPS feed URLs, explicit fetch timeout, in-process parallel aggregation, metadata/errors in combined output, correlation loading, and press-context rendering.
258
+Deterministically shard repos into clusters by embedding/topic/language/owner/fork-star anomaly patterns, then map each cluster.
259
556
-## Reliability observations
260
+Pros:
261
+- Handles large GitHub raw payloads directly.
262
+- Can isolate suspicious clusters such as fork inflation, star farming, exploit churn, or copycat agent repos.
263
558
-1. Multi-source RSS is not currently the runtime bottleneck. The new five-source RSS step took about one second after dependencies; the GitHub API crawler still dominates the crawl job at ~4m47.
559
-2. The in-process model is operationally simple and fast, but failure isolation is only at script level. A per-source fetch exception can be represented in `metadata.errors`, but a bad config parse, merge bug, dependency issue, or Python process failure takes out every external source in one step.
560
-3. Retry granularity is poor in the current shape. A flaky NVIDIA/Hugging Face/MIT feed requires rerunning the whole crawl job, including the GitHub API crawl and cache artifact upload, unless manual surgery is done.
561
-4. Artifact availability is all-or-nothing for external news. The workflow uploads `raw-data` after the combined step, so failed individual sources do not leave independently downloadable payloads unless the combined script writes a degraded aggregate.
562
-5. Cache behavior argues against matrixing the GitHub repository crawl right now. The GitHub cache is a single `data/cache/` artifact restored from the previous successful run; splitting GitHub query work would introduce cache merge/conflict questions without evidence it is the bottleneck needing parallel source isolation.
563
-6. Partial data tolerance exists downstream: correlation only runs if an external-news or legacy TechCrunch file exists, and press context can render a no-press fallback. That is good, but the workflow does not yet make optional-source degradation explicit enough in job summaries or gating.
564
-7. Reproducibility needs tightening before matrix fan-out. Matrix jobs must share the same centrally computed `week`, `since`, and `until`; otherwise each source can observe a slightly different crawl window.
264
+Cons:
265
+- Needs stable clustering and coverage accounting.
266
+- Cluster labels may be misleading if generated by LLM without deterministic support.
267
566
-## Recommendation
268
+Best use: second iteration once signal-type mapping proves useful.
269
568
-For the next iteration, keep the GitHub repository crawl as one core job and split external RSS/news sources into a GitHub Actions matrix with `fail-fast: false`, per-source artifacts, and a deterministic merge job before analysis.
270
+### 5. Source-specific press summaries before main reduce
271
570
-This gives the best reliability improvement without multiplying the GitHub API/cache risk. Because external RSS jobs can run in parallel with the slower GitHub crawl, matrix overhead should not increase the critical path much if analysis depends on a small merge job rather than on the old monolithic crawl job. Do not push source merging into analysis; merge before analysis so correlation, press context, artifacts, and rebuild hydration keep a stable stage boundary.
272
+A deterministic or LLM-assisted press mapper compresses external news into source-aware press claims, then the main reducer joins those claims with repo claims.
273
572
-## Acceptance criteria for the issue
274
+Pros:
275
+- Strong citation preservation if contract is strict.
276
+- Keeps `Where Industry Meets Code` from becoming a news roundup.
277
574
-- Workflow defines a shared crawl context (`week`, `since`, `until`, source config checksum) once and passes it to all crawl jobs.
575
-- GitHub repository crawl remains a required/core job and continues to restore/upload the existing `crawl-cache` artifact.
576
-- External news uses a matrix over configured source names/URLs with `strategy.fail-fast: false`.
577
-- Each source uploads a per-source artifact on `if: always()` containing either:
578
- - a valid source payload with articles and metadata; or
579
- - a status/error JSON with source name, error class/message, attempts, duration, and crawl window.
580
-- A merge job runs on `if: always()` after core crawl and all news matrix jobs, downloads available source artifacts, validates schemas, deduplicates/sorts deterministically, and writes canonical `data/raw/{week}-external-news.json`.
581
-- Analysis consumes only the merged canonical external-news file plus the GitHub raw file; it does not crawl or merge feeds itself.
582
-- Optional external-news failures do not block publication when GitHub raw data is valid; they must produce visible warnings and metadata. A config/schema/security validation failure should fail the workflow because it is deterministic and actionable.
583
-- Rebuild mode hydrates the merged external-news file and still accepts legacy `{week}-techcrunch.json`.
584
-- The raw-data artifact remains available even when one or more optional source jobs fail.
585
-- CI summary reports per-source status and aggregate totals; the next run can identify exactly which feed was slow/flaky.
278
+Cons:
279
+- Adds hallucination/citation drift risk if source summaries are LLM-generated.
280
+- Current compact deterministic press-context path may be enough.
281
587
-## Tests to add or update
282
+Best use: defer unless `*-external-news.json` regularly breaches press context budget.
283
589
-- Unit tests for a new merge helper/script:
590
- - merges multiple valid source artifacts into `source=external_news` canonical output;
591
- - preserves `source_count`, `sources_with_articles`, `metadata.errors`, and per-source status;
592
- - deduplicates repeated article URLs deterministically without dropping distinct source attribution unexpectedly;
593
- - sorts output deterministically by `published_at`, then source/name/url;
594
- - tolerates missing/failed optional source artifacts;
595
- - fails on malformed JSON, invalid source names, or mismatched `week`/window metadata.
596
-- CLI tests for fixed `--since` and `--until` propagation so matrix jobs reproduce the same window.
597
-- Workflow/handoff tests or a validation script fixture that asserts `analyze` depends on the merge artifact, not raw matrix artifacts directly.
598
-- Correlation and press-context tests with merged `*-external-news.json`, legacy `*-techcrunch.json`, and no external-news file.
599
-- Regression test that a single source failure still produces a merged canonical file with remaining articles and visible `metadata.errors`.
600
-- Regression test that all external sources failing produces a no-press fallback path while preserving a valid GitHub raw artifact.
284
+## Recommended architecture
285
602
-## Metrics/logging to capture
286
+### Phase 0 — deterministic preflight
287
604
-- Per source: source name, URL host, start/end/duration seconds, attempts, timeout seconds, total articles, relevant articles, GitHub links found, error class/message, and success/failure.
605
-- Aggregate: source_count, successful_source_count, failed_source_count, total/relevant articles, dedupe counts, artifact size, merge duration.
606
-- Core GitHub crawl: API calls, cache hits, stale cache hits, rate-limit remaining/resource/reset, partial failure count, repo counts, snapshot repo count.
607
-- Workflow: job durations for core crawl, each source crawl, merge, analyze; whether analysis used full press data, partial press data, or no-press fallback.
608
-- Reproducibility: source config checksum, code commit SHA, crawl window, and canonical merged file checksum.
288
+Inputs:
289
+- sanitized weekly raw JSON,
290
+- compact press context from `*-external-news.json` + `*-correlations.json`,
291
+- previous summary,
292
+- wisdom/skills bundle,
293
+- analysis spec and gate constraints.
294
+
295
+Preflight outputs:
296
+- token estimates per input segment,
297
+- repo coverage counts and star totals,
298
+- source coverage counts/errors,
299
+- deterministic clusters or slices,
300
+- stable IDs for repos, articles, and candidate evidence groups.
301
+
302
+### Phase 1 — mappers produce claim ledgers, not prose articles
303
+
304
+Each mapper receives a bounded evidence slice and returns a strict JSON/markdown-ledger contract. Mappers should not write final publication prose or frontmatter. They should extract:
305
+
306
+- candidate trend claims,
307
+- signal/noise/gap judgments,
308
+- evidence repo IDs and article IDs,
309
+- confidence and uncertainty,
310
+- citation URLs,
311
+- contradiction flags,
312
+- suggested `Key References` candidates,
313
+- token usage/coverage telemetry.
314
+
315
+### Phase 2 — reducer creates one coherent editorial plan
316
+
317
+Reducer consumes only mapper ledgers plus compact global metadata. It:
318
+
319
+- deduplicates candidate claims by normalized claim key/topic/repo/article URL,
320
+- merges supporting evidence across mappers,
321
+- rejects weak unsupported claims,
322
+- resolves contradictions by evidence strength and citation quality,
323
+- selects 3-5 macro trends, 2-4 correlations/divergences, 2-4 blind spots, 5-10 repo references, and 3-5 press references,
324
+- chooses `title`, `top_repo`, `tags`, `quality_score`, and optional `predictions`,
325
+- emits an editorial outline with citation bindings.
326
+
327
+### Phase 3 — final writer/gate
328
+
329
+Final writer converts the reducer plan into the exact `docs/analysis-spec.md` output shape:
330
+
331
+```md
332
+## This Week's Trends
333
+## Where Industry Meets Code
334
+## Signal & Noise
335
+## Blind Spots
336
+## The Week Ahead
337
+## Key References
338
+### Notable Projects
339
+### Press & Industry
340
+```
341
610
-## Risks / gates
342
+Then `scripts/analysis_gate.py` runs unchanged at first, with future enhancements for evidence/citation checks.
343
+
344
+## Reducer responsibilities for global coherence
345
+
346
+The reducer is the only stage allowed to create final reader-facing prose. It must:
347
+
348
+- preserve one editorial voice and avoid mapper-by-mapper seams;
349
+- maintain a single global thesis and title;
350
+- avoid duplicate claims by normalizing repo full names, article URLs, topic labels, and claim keys;
351
+- keep every repository mention renderable as `[owner/repo](https://github.com/owner/repo)`;
352
+- keep every press claim backed by retained article citations;
353
+- distinguish strong correlations from weak/category/fuzzy matches;
354
+- retain source caveats from external-news metadata;
355
+- keep `repos_featured` and `stars_tracked` tied to deterministic preflight totals rather than mapper estimates;
356
+- satisfy `analysis_gate.py` frontmatter/headings/body constraints.
357
+
358
+## Concrete mapper output contract
359
+
360
+Suggested `analysis_map_v1` object:
361
+
362
+```json
363
+{
364
+ "schema_version": "analysis_map_v1",
365
+ "week": "YYYY-WNN",
366
+ "slice": {
367
+ "id": "signal-type:new-repos",
368
+ "strategy": "signal_type|topic|source|repo_cluster",
369
+ "input_token_estimate": 12000,
370
+ "repo_count": 42,
371
+ "article_count": 0
372
+ },
373
+ "coverage": {
374
+ "repo_ids_seen": ["owner/repo"],
375
+ "article_urls_seen": ["https://example.com/article"],
376
+ "excluded_reason_counts": {"low_relevance": 3}
377
+ },
378
+ "claims": [
379
+ {
380
+ "claim_id": "stable-hash-or-slug",
381
+ "claim_type": "trend|signal|noise|gap|press_correlation|press_divergence|continuity",
382
+ "headline": "Short claim label",
383
+ "summary": "One or two sentences, evidence-bound.",
384
+ "evidence_repos": [
385
+ {
386
+ "full_name": "owner/repo",
387
+ "url": "https://github.com/owner/repo",
388
+ "role": "anchor|supporting|counterexample",
389
+ "stars": 123,
390
+ "stars_gained": null,
391
+ "evidence_note": "Why this repo supports the claim"
392
+ }
393
+ ],
394
+ "evidence_articles": [
395
+ {
396
+ "title": "Article title",
397
+ "url": "https://example.com/article",
398
+ "source": "TechCrunch",
399
+ "published_at": "2026-06-01",
400
+ "role": "corroborates|diverges|context",
401
+ "correlation_strength": "strong|weak|none"
402
+ }
403
+ ],
404
+ "confidence": 0.72,
405
+ "uncertainties": ["stars_gained missing for most trending repos"],
406
+ "quality_flags": ["possible_duplicate", "weak_citation", "needs_reducer_review"]
407
+ }
408
+ ],
409
+ "reference_candidates": {
410
+ "notable_projects": ["owner/repo"],
411
+ "press_articles": ["https://example.com/article"]
412
+ }
413
+}
414
+```
415
612
-- Matrix jobs add workflow complexity and more artifacts; keep merge logic small and heavily tested.
613
-- Matrix setup overhead is only acceptable if source jobs run in parallel with the GitHub crawl. If they remain sequential after core crawl, in-process fan-out is faster for five feeds.
614
-- Do not treat article volume alone as success. Gate on valid schemas, explicit source statuses, deterministic merge, and downstream correlation/press-context success.
615
-- Keep optional-source degradation visible. Silent partial data is worse than a failed optional feed.
416
+## Concrete reducer input/output contract
417
+
418
+Reducer input:
419
+
420
+```json
421
+{
422
+ "schema_version": "analysis_reduce_input_v1",
423
+ "week": "YYYY-WNN",
424
+ "run_datetime": "ISO-8601",
425
+ "global_totals": {
426
+ "repos_featured": 449,
427
+ "stars_tracked": 123456,
428
+ "new_repo_count": 213,
429
+ "trending_repo_count": 236
430
+ },
431
+ "source_coverage": {
432
+ "sources_requested": ["techcrunch", "github_blog"],
433
+ "sources_succeeded": ["techcrunch"],
434
+ "sources_failed": ["github_blog"]
435
+ },
436
+ "maps": ["analysis_map_v1 objects"]
437
+}
438
+```
439
617
----
440
+Reducer output should be an editorial plan before prose:
441
+
442
+```json
443
+{
444
+ "schema_version": "analysis_editorial_plan_v1",
445
+ "title": "Punchy headline",
446
+ "summary": "One-sentence thesis",
447
+ "top_repo": "owner/repo",
448
+ "tags": ["ai", "developer-tools", "security"],
449
+ "selected_claims": [
450
+ {
451
+ "claim_id": "...",
452
+ "section": "This Week's Trends|Where Industry Meets Code|Signal & Noise|Blind Spots|The Week Ahead",
453
+ "merged_from": ["mapper-claim-id"],
454
+ "citation_bindings": {
455
+ "repos": ["owner/repo"],
456
+ "articles": ["https://example.com/article"]
457
+ }
458
+ }
459
+ ],
460
+ "key_references": {
461
+ "notable_projects": ["owner/repo"],
462
+ "press_articles": ["https://example.com/article"]
463
+ },
464
+ "rejected_claims": [
465
+ {"claim_id": "...", "reason": "duplicate|unsupported|contradicted|weak_citation"}
466
+ ],
467
+ "quality_notes": ["Caveat missing stars_gained in trend section"]
468
+}
469
+```
470
619
----
471
+The final writer then emits only markdown conforming to the existing spec.
472
621
-# Leela — crawler next-iteration issue
473
+## Risks
474
623
-Date: 2026-06-05T16:26:00Z
475
+- Mapper contradiction: two mappers may classify the same repo as signal and noise. Reducer needs explicit conflict resolution and rejected-claim logging.
476
+- Citation drift: if mappers paraphrase article claims without preserving URLs/source/date, the final summary may cite the wrong article or overstate correlation.
477
+- Duplicate claims: topic and signal-type mappers may independently discover the same pattern.
478
+- Quality gate complexity: structural gate is simple today; claim-ledger validation, citation coverage, and contradiction checks add test and maintenance burden.
479
+- Cost/token growth: multiple smaller LLM calls can exceed one large call if slices overlap or include repeated instructions/history.
480
+- Runtime: parallel mapper calls help wall-clock time only if model/API concurrency is available and reliable.
481
+- Editorial voice loss: mapper prose can create a patchwork article unless final prose is written by one reducer/writer pass.
482
+- Over-pruning: small slices may miss weak cross-cluster patterns that only appear globally.
483
+- Failure policy: partial mapper failure could bias coverage unless reducer sees missing-slice telemetry and either degrades explicitly or falls back.
484
+- Prompt injection surface: every mapper still ingests untrusted repo/news text and must keep untrusted-content boundaries.
485
+
486
+## Evaluation metrics
487
+
488
+### Token and runtime metrics
489
+
490
+- Total prompt token-estimate by stage: preflight, each mapper, reducer, final writer.
491
+- Maximum per-call token-estimate and p95 per-call token-estimate.
492
+- Total generated tokens and total model calls.
493
+- End-to-end wall-clock time versus current single-call path.
494
+- Cost per successful weekly analysis and cost per fallback/retry.
495
+
496
+### Quality and faithfulness metrics
497
+
498
+- `analysis_gate.py` pass rate.
499
+- Required section/headings/frontmatter pass rate.
500
+- Citation coverage: percentage of repo/article claims with retained citations.
501
+- Claim support: percentage of final claims traceable to mapper evidence IDs.
502
+- Hallucination/unsupported-claim count from automated or human review.
503
+- Duplicate claim count before/after reduce.
504
+- Contradiction count and reducer resolution rate.
505
+- Press correlation accuracy: strong vs weak labels preserved correctly.
506
+- Editorial quality score from Farnsworth/Leela rubric: synthesis, specificity, skepticism, blind spots, and voice.
507
+
508
+### Stability metrics
509
+
510
+- Rerun stability: overlap in selected top trends/repos/press references across repeated runs with same inputs.
511
+- Title/top_repo stability across repeated runs.
512
+- Sensitivity to mapper ordering.
513
+- Missing-slice degradation behavior.
514
+
515
+## Non-goals for initial PRD
516
+
517
+- Do not replace the weekly `docs/analysis-spec.md` output contract.
518
+- Do not make each mapper produce publishable prose.
519
+- Do not split the crawl into an Actions matrix as part of the analysis map/reduce MVP unless separate performance evidence justifies it.
520
+- Do not include raw article dumps or raw correlation dumps in final analysis prompts.
521
+- Do not let weak/category-only correlations become strong claims without corroboration.
522
+- Do not optimize for maximum recall at the expense of citation integrity and editorial judgment.
523
+- Do not require new paid services, embeddings infrastructure, or vector databases for MVP.
524
+- Do not publish map/reduce output until it passes the existing gate and a new evidence-contract validator.
525
+
526
+## Guardrails for MVP
527
+
528
+- Feature flag the map/reduce path; preserve the current single-call/fallback path.
529
+- Keep deterministic preflight totals authoritative for `repos_featured`, `stars_tracked`, source status, and citation inventories.
530
+- Wrap all repo/news evidence as untrusted data in every mapper prompt.
531
+- Limit mapper output to structured claims with evidence IDs, not final prose.
532
+- Run final `analysis_gate.py` unchanged initially, then add a separate mapper/reducer contract validator.
533
+- Require a human review comparison against the single-call output for the first several weeks.
534
+- Treat no-AI/data-only fallback as the terminal reliability fallback if mapper/reducer calls fail.
535
+
536
+## Acceptance criteria
537
+
538
+1. Given the same weekly raw GitHub JSON and compact press context, the map/reduce experiment produces a final markdown summary that passes `scripts/analysis_gate.py`.
539
+2. Every final repo mention resolves to a repo seen in preflight or mapper coverage and is rendered as a proper GitHub markdown link.
540
+3. Every final press claim cites an article URL retained in source coverage or press context.
541
+4. The reducer emits a rejected-claims/conflicts ledger for audit, even if not published.
542
+5. The final article contains 3-5 coherent macro trends, explicit signal/noise judgment, useful blind spots, and a single editorial voice.
543
+6. The map/reduce path demonstrates lower max per-call token-estimate than the current single-call prompt, with measured total cost/runtime reported.
544
+7. Reruns on identical input are stable enough for publication: same top_repo or documented reason for change, and at least 70% overlap in selected key references.
545
+8. Partial mapper failure either retries that slice or marks the final output as degraded; it must not silently omit a source/category.
546
+9. Existing single-call and no-AI fallback paths remain available until map/reduce beats them on gate pass rate, citation coverage, and human editorial review.
547
+
548
+---
549
+
550
+# Fry QA input — matrix crawl + map/reduce analysis PRD
551
+
552
+Date: 2026-06-05T18:15:23Z
553
Requested by: jmservera
625
-Issue: https://github.com/jmservera/SquadScope/issues/237
626
-
627
-## Decision
628
-
629
-Created issue #237, "Improve multi-source crawler telemetry and source-aware press correlation."
630
-
631
-The lead decision is:
632
-
633
-- Keep GitHub repository crawl monolithic and cached.
634
-- Keep external RSS/news crawl in-process with bounded parallelism for now.
635
-- Defer Actions matrix fan-out until evidence triggers it: RSS/news p95 > 60s, source count > 10, or a source needs independent retry, credentials, quota, or network isolation.
636
-- Treat merge-before-analyze as deterministic data fan-in, not staged LLM map-reduce.
637
-
638
-## Scope captured
554
+Owner: Fry / QA
555
+
556
+## QA position
557
+
558
+A matrix is not automatically faster for the current crawl. Prior evidence shows the external RSS stage is about one second, while GitHub search/repo crawling dominates and already approaches the tighter Search API budget. The PRD should treat matrix crawl as a measured experiment: first make artifacts merge-ready and deterministic, then fan out only workloads with independent latency, retry, and quota profiles.
559
+
560
+Map/reduce analysis is worth ideating because it can shrink per-model context and isolate failures, but it must not weaken the existing analysis contract. The reducer's final markdown must still pass `scripts/analysis_gate.py`, preserve repo/news citations, produce the current frontmatter shape, and keep the existing Copilot -> GitHub Models -> no-AI fallback path viable.
561
+
562
+## PRD-ready QA gates
563
+
564
+### 1. Matrix crawl fan-out/fan-in
565
+
566
+Required gates before default-on:
567
640
-The issue asks the next iteration to improve:
568
+- **Deterministic run context:** every leg receives the same `week`, `since`, `until`, source config revision, topic config revision, and run id. No leg may compute its own week window from local wall clock except via a shared generated context artifact.
569
+- **Per-leg artifact contract:** each leg writes exactly one JSON artifact with `{schema_version, run_id, week, since, until, leg_id, source_type, started_at, finished_at, duration_seconds, status, payload, errors, metrics, checksum}`.
570
+- **Fan-in determinism:** merge output must be byte-stable for the same inputs: canonical ordering, deterministic dedupe keys, stable error ordering, and a checksum recorded in metadata.
571
+- **Partial failure semantics:** required legs fail the workflow; optional legs degrade with explicit `status=failed` artifacts and a minimum-source-success gate.
572
+- **Retry behavior:** retry only failed optional legs when possible; fan-in must distinguish first-attempt failure, retry success, and terminal failure. A rerun must not double-count articles/repos.
573
+- **Cache consistency:** cache keys include query/source config, week window, and schema version. Stale cache use must be marked in metadata and never silently mix different windows.
574
+- **Rate-limit safety:** GitHub-query fan-out must be capped by search quota remaining and secondary-rate-limit backoff. RSS/API legs need per-host concurrency limits and timeout/retry ceilings.
575
+- **Artifact compatibility:** downstream analysis consumes one canonical raw payload and one canonical external-news payload regardless of matrix vs single-process collection.
576
642
-- per-source external-news status and metrics;
643
-- schema/versioned deterministic canonical `*-external-news.json`;
644
-- source-aware and bounded `correlate.py` / `render_press_context.py`;
645
-- cross-source dedupe to avoid correlation inflation;
646
-- press-context token/article bounds and telemetry;
647
-- tests for partial failures, fallback paths, reproducibility, dedupe, and citation preservation.
577
+### 2. Map/reduce analysis
578
649
-## Non-goals captured
579
+Required gates before default-on:
580
651
-- Multi-pass/staged LLM analysis.
652
-- GitHub raw compaction.
653
-- Matrix split unless the trigger threshold is met.
654
-- Core GitHub crawler topology changes.
581
+- **Mapper schema validation:** each mapper emits structured JSON, not prose-only markdown: `{schema_version, run_id, week, shard_id, input_refs, findings[], citations[], token_estimate, model, status, errors}`.
582
+- **Finding shape:** each finding includes `claim`, `evidence_refs`, `confidence`, `category`, `source_type`, `repo_full_name?`, `news_url?`, and `contra_refs[]`.
583
+- **Citation preservation:** reducer must be able to trace every final claim to repo URLs, raw payload paths, and news URLs. Missing or malformed citations fail reducer validation.
584
+- **Reducer behavior:** reducer must dedupe equivalent findings, surface contradictions instead of hiding them, prefer higher-confidence/evidence-backed findings, and record rejected/merged finding IDs in a sidecar.
585
+- **Contradiction tests:** contradictory mapper outputs must either resolve with documented rationale or appear in the final analysis as uncertainty/blind spot; they must not disappear silently.
586
+- **Duplicate tests:** duplicate repo/news claims across shards must collapse to one final claim without losing all citations.
587
+- **Gate compatibility:** final markdown must pass `analysis_gate.py` unchanged unless the PRD explicitly extends the gate. Frontmatter, headings, week/date, predictions, and no-placeholder rules still apply.
588
+- **Fallback compatibility:** if any map/reduce stage cannot produce a valid final summary, the pipeline must still try the current single-pass/GitHub Models/no-AI fallback path.
589
656
-## Routing
590
+## Test matrix
591
658
-Labels applied: `squad`, `squad:leela`, `squad:bender`, `go:yes`.
592
+| Area | Scenario | Expected QA outcome |
593
+| --- | --- | --- |
594
+| Crawl context | All legs receive shared generated week window | Artifacts have identical `week/since/until/run_id`; mismatch fails fan-in |
595
+| Crawl determinism | Same fixture artifacts merged twice | Identical merged JSON bytes/checksum |
596
+| Crawl optional failure | One RSS/source leg times out | Workflow continues if minimum source threshold met; error recorded; analysis sees canonical artifact |
597
+| Crawl required failure | GitHub raw repo leg fails | Analyze does not run; notify-failure path catches pipeline failure |
598
+| Crawl retry | Failed optional leg succeeds on retry | Final metadata records retry count and no duplicate payload entries |
599
+| Crawl cache | Stale cache restored for wrong week/config | Fan-in rejects or marks unusable; no silent mixed-window output |
600
+| Crawl rate limit | Search quota near floor | GitHub fan-out throttles or skips risky fan-out; no uncontrolled parallel search bursts |
601
+| No news data | External-news artifact absent or empty | Press context says no press data; analysis gate can still pass |
602
+| Mapper schema | Mapper emits malformed JSON/prose | Reducer rejects mapper artifact and records mapper failure |
603
+| Mapper failure | One mapper exits non-zero | Required shard fails workflow or optional shard degrades by configured policy; reducer cannot silently omit |
604
+| Duplicate findings | Same repo trend in two shards | Reducer emits one finding with combined citations |
605
+| Contradictions | One mapper says trend is signal, another says noise | Reducer records rationale or uncertainty; contradiction sidecar includes both sources |
606
+| Citation loss | Reducer final claim lacks source refs | Reducer validation fails before `analysis_gate.py` |
607
+| Over-budget context | Single reducer input exceeds token budget | Reducer switches to hierarchical reduce or fails to fallback before spending unbounded tokens |
608
+| Week mismatch | Mapper output `week` differs from raw payload | Reducer rejects artifact |
609
+| Token spike | Mapper/reducer token estimate exceeds budget threshold | Dry-run blocks default path; metrics identify model/stage/shard |
610
+| Gate regression | Final summary missing heading or generic title | Existing `analysis_gate.py` fails and fallback path is exercised |
611
660
-Bender is the likely implementation owner; Fry should validate reliability gates; Farnsworth should review press-context quality.
612
+## Failure modes to require in PRD
613
662
----
663
-
664
----
665
-
666
-# Bender PR #236 Security Fix
667
-
668
-## Context
669
-Hermes blocked PR #236 because config-driven external RSS sources were fetched directly without egress URL validation or explicit per-request timeouts.
670
-
671
-## Decision
672
-External news RSS source configs now require HTTPS URLs whose host is in the approved feed allowlist, with credentials, local/private/link-local targets, and unexpected ports rejected before crawl. Fetching now goes through `urllib.request.urlopen` with an explicit bounded timeout before handing bytes to `feedparser`, while retaining the existing config-driven source list and bounded in-process worker pool.
614
+- One matrix leg fails: fan-in runs with `if: always()` for diagnostics, but publish/analyze only continue if required artifacts exist and optional-source thresholds pass.
615
+- One mapper fails: reducer must not hide it; either fail the map/reduce path or explicitly degrade based on shard criticality, then fallback to single-pass/no-AI if final gate fails.
616
+- No news data: treated as valid degraded input, not a crash; final summary uses existing "No press data" behavior.
617
+- Over-budget context: preflight estimates for each mapper, reducer, and aggregate final prompt; hard fail or hierarchical reduce before model invocation.
618
+- Stale cache: cache metadata includes created_at, week window, source config checksum, and schema version; stale use is observable and bounded.
619
+- Inconsistent week windows: fan-in/reducer reject mixed `week/since/until` artifacts.
620
+- Token/cost spikes: per-shard and total token ledger records estimates/actuals; alert if p95 or per-run cost exceeds threshold.
621
674
-## Validation
675
-Added tests for invalid/unapproved URL rejection and explicit fetch timeout propagation. Ran `PYTHONPATH=. .venv/bin/python -m pytest tests -q` with 563 passing tests.
622
+## Observability requirements
623
677
----
624
+Minimum notices/metrics per run:
625
679
----
680
-
681
-# Leela — Issue 234 external news source architecture
626
+- Per crawl leg: `leg_id`, source name/type, status, start/end/duration, item count, relevant count, dedupe count, artifact size, checksum, cache hit/stale hit, API calls, retry count, error class.
627
+- Aggregate crawl: required/optional leg counts, failed leg counts, merged artifact size/checksum, total API calls, rate-limit remaining/reset/resource, cache hit ratio.
628
+- Per mapper: shard id, input artifact refs/checksums, prompt size, token estimate/actual, model/source, duration, output size, finding count, citation count, quality/schema validation result.
629
+- Reducer: input shard count, failed/skipped shard count, duplicate count, contradiction count, final prompt/output tokens, duration, model/source, final quality gate result.
630
+- Pipeline path: selected path (`single-pass`, `map-reduce`, `github-models`, `no-ai`), fallback reason, and final `analysis_gate` outcome.
631
683
-Date: 2026-06-05T15:36:19.379+00:00
684
-Issue: #234
632
+## Rollout plan and acceptance thresholds
633
686
-## Decision
634
+1. **Design-only contract:** define artifact schemas and validators; do not change default workflow path.
635
+2. **Local fixture dry-run:** run fan-in and map/reduce reducer on deterministic fixtures with no network/model calls.
636
+3. **CI dry-run mode:** add non-publishing matrix/map-reduce jobs that upload artifacts and metrics but keep single-pass analysis as source of truth.
637
+4. **A/B comparison:** for at least 4 weekly runs, compare current single-pass vs map/reduce outputs for gate pass rate, citation preservation, token use, cost, duration, and human review quality.
638
+5. **Default switch only if thresholds pass:**
639
+ - 100% final `analysis_gate.py` pass rate in dry-run comparison.
640
+ - 0 missing required citations in reducer validation.
641
+ - No increase in failed weekly publishes.
642
+ - >=25% reduction in analysis prompt tokens or >=20% reduction in analysis wall time, without quality regression.
643
+ - Crawl matrix only enabled if measured crawl stage p95 improves by >=20% or it materially improves retry isolation for sources with real failure/latency.
644
+ - Token/cost per run stays within agreed budget and has alerts before hard overrun.
645
+6. **Guarded rollout:** workflow_dispatch flag first, then scheduled dry-run, then default-on with single-pass fallback retained for at least one release cycle.
646
688
-Keep external news crawling in the existing crawl job and make the RSS source list config-driven via `config/external_news_sources.json`. Fetch the configured feeds concurrently inside `scripts/techcrunch_crawler.py` using a bounded thread pool, and write one weekly enrichment artifact: `data/raw/YYYY-WNN-external-news.json`.
647
+## Local and CI validation needed
648
690
-## Rubberduck tradeoff
649
+Local validation:
650
692
-Separate GitHub Actions jobs would parallelize at the runner level, but every source would repeat checkout, Python setup, dependency install, artifact upload/download, and failure-handling boilerplate. For five RSS feeds, that overhead is larger than the network wait we are optimizing away, and it would fragment a single enrichment contract across multiple artifacts.
651
+- Unit tests for artifact schemas, fan-in merge determinism, dedupe ordering, cache metadata rejection, and failure classification.
652
+- Unit tests for mapper schema validator, reducer dedupe/contradiction handling, citation preservation, week-window rejection, and token-budget preflight.
653
+- Existing focused tests should remain green: `tests/test_crawl.py`, `tests/test_techcrunch_crawler.py`, `tests/test_pipeline.py`, `tests/test_analysis_gate.py`, `tests/test_analyze_fallback.py`, `tests/test_track_token_usage.py`, `tests/test_preflight_cost_check.py`, `tests/test_render_press_context.py`, `tests/test_correlate.py`.
654
694
-In-process threading matches the current architecture better: RSS fetching is I/O-bound, feedparser work is light, and the existing crawl job already owns raw data artifact handoff. A bounded pool preserves Actions compute, keeps one failure surface, and lets future sources be added by config without editing workflow topology.
655
+CI validation:
656
696
-## Scope boundary
697
-
698
-This is a small architectural refactor around an existing RSS crawler, so Leela implemented directly rather than reassigning to Bender. Deeper crawler work, such as source-specific parsing, feed health dashboards, or correlation logic, should remain Bender-owned.
657
+- Matrix dry-run job with fixture legs and one forced optional failure.
658
+- Fan-in job using `if: always()` that publishes diagnostics artifacts even on failed legs.
659
+- Map/reduce dry-run job that compares reducer output to single-pass output but does not publish.
660
+- Quality gate runs on final reducer markdown and fallback markdown.
661
+- Token/cost ledger checks include mapper/reducer stages and enforce budget alerts.
662
+- Rebuild mode validation hydrates canonical merged artifacts and does not depend on per-leg artifacts being present forever.
663
664
---
665
702
-# Fry — Issue #238 notify triage
666
+# Leela decision input — matrix crawl + map/reduce analysis PRD
667
704
-Date: 2026-06-05T17:11:29.929+00:00
705
-Issue: https://github.com/jmservera/SquadScope/issues/238
706
-Run: https://github.com/jmservera/SquadScope/actions/runs/27026348186
668
+Date: 2026-06-05T17:42:56.819+00:00
669
+Owner: Leela / Lead
670
+Artifact: `docs/processed/PRD-matrix-crawl-map-reduce-analysis.md`
671
708
-## Finding
672
+## Decision recommendation
673
710
-The pipeline stages that produce and publish data succeeded. The only failed job was `notify`, where `gh release create week-2026-W23` returned HTTP 422 because the `week-2026-W23` release already existed.
674
+Do not enable a crawl matrix by default. The recent implementation correctly avoided it because five-source RSS collection is about one second and already uses bounded in-process parallelism, while GitHub crawling is dominated by API/cache/rate-limit behavior that matrix fan-out could make worse.
675
712
-## Decision
676
+Make crawl artifacts matrix-ready through shared run context, schema validation, checksums, deterministic fan-in, and observability. Gate RSS matrix on source count/runtime/isolation triggers. Gate GitHub matrix on a no-publish shard experiment that proves >=25% crawl speedup with <=10% API-call growth and no secondary-rate-limit regression.
677
714
-Treat this as a real QA-owned workflow idempotency bug, not a transient network or rate-limit failure. Weekly notify must be safe to rerun for an already-published week.
678
+Adopt map/reduce only as an analysis experiment for LLM context and quality. Mappers should emit structured claim ledgers with citations, confidence, contradictions, and coverage. The reducer should own dedupe, citation preservation, contradiction handling, editorial coherence, and final `analysis_gate.py` compliance.
679
716
-## Fix
680
+## Follow-up needed
681
718
-Update the notify release step to check for the weekly release tag. If it exists, edit the existing release title/notes and mark it latest; otherwise create it as before.
719
-
720
-## Validation
721
-
722
-- `PYTHONPATH=. .venv/bin/python -m pytest tests/test_pipeline.py -q` — 9 passed.
723
-- `PYTHONPATH=. .venv/bin/python -m pytest tests -q` — 563 passed after installing project requirements and pytest in a local venv.
682
+- Baseline crawl/analyze p50/p95 and token/cost metrics across multiple runs.
683
+- Define artifact and mapper/reducer JSON schemas plus validators.
684
+- Run map/reduce in dry-run A/B mode before publication eligibility.
685
+- Keep single-pass/GitHub Models/no-AI fallback until map/reduce beats current quality and reliability gates.
686
687
---
688
727
----
689
+# Bender run 27030646485 log review
690
729
-# Leela PR #241 Review — Idempotent Weekly Release Notify
691
+Date: 2026-06-05T17:42:56Z
692
+Run: https://github.com/jmservera/SquadScope/actions/runs/27030646485
693
731
-- Date: 2026-06-05
732
-- Context: Issue #238 showed a real rerun failure in `notify`: `gh release create week-2026-W23` returned HTTP 422 because the weekly release already existed.
733
-- Decision: Keep weekly release notification idempotent by resolving the weekly tag first, editing an existing `week-*` release with `gh release edit`, and creating only when no release exists.
734
-- Review result: Approved in substance. Formal GitHub approval was blocked because the authenticated account is the PR author, so Leela posted an explicit lead approval comment instead of bypassing the review gate.
735
-- Validation: `tests/test_pipeline.py` passed locally (9 tests), full `tests` passed locally (563 tests), CodeQL checks were green, and Copilot PR review completed with no comments.
736
-- Merge gate: Do not merge from this account until the repository's independent-review requirement for `jmservera`-authored PRs is satisfied.
737
-- PR #241 merged at 2026-06-05T17:21:05Z, closing issue #238.
694
+## Findings
695
739
----
696
+- Workflow completed successfully, but success came through the no-AI fallback path.
697
+- Crawl job was healthy: `Run crawler` took ~4m30s, used 455 GitHub API calls, found 213 new repos and 236 trending repos, with 0 cache hits.
698
+- External news behaved correctly at current scale: 5/5 sources succeeded in ~1s total, 39 articles, 23 relevant, 0 deduped, checksum `ebe382a11c0b...`.
699
+- Per-source external news telemetry was present in logs and artifact metadata: source names, hosts, attempts, durations, article counts, relevant counts, GitHub-link counts, errors, config checksum, and artifact checksum.
700
+- Correlation/press-context generation succeeded before analysis: 50 correlations from 449 repos; 9 strong and 41 weak; press context 32,765 bytes / ~7,991 token estimate.
701
+- Analysis was the runtime and reliability concern: three Copilot attempts took ~28m41s and failed quality gates; the fallback GitHub Models request failed with `no_access` for `openai/gpt-4o`; data-only no-AI output passed the gate.
702
+- Quality-gate failures were actionable:
703
+ - attempts 1 and 2: `date must match the current run timestamp`;
704
+ - attempt 3: invalid `predictions[*].claim_type` values plus the date mismatch.
705
+- Token telemetry showed the analysis path estimated 112,911 input tokens / 119,620 total tokens, while the pre-flight check estimated 74,318 input tokens before full rendered prompt accounting.
706
+- Non-blocking platform warning: GitHub Actions reported Node.js 20 actions deprecation for checkout/download/upload/setup/deploy actions.
707
741
----
708
+## Directional read
709
743
-# Bender issue #237 implementation
744
-
745
-Date: 2026-06-05
746
-
747
-## Decision
748
-
749
-Keep external RSS/news in the existing crawl job with bounded in-process parallelism, but promote the handoff to a canonical `schema_version: 2` `data/raw/{week}-external-news.json` artifact. The artifact carries crawl window, source config checksum, requested/succeeded/failed sources, per-source status metrics, dedupe count, deterministic checksum, and partial-failure metadata.
710
+This run supports the current PRD direction to keep external RSS in-process until scale/isolation thresholds are met. RSS is still not the speed bottleneck; the critical path is now analysis duration, prompt size, and retry waste. It also supports deterministic merge/press-context fan-in over LLM map-reduce for now: compact press context worked, but the full analysis prompt is still too large and brittle.
711
751
-## Rationale
712
+## Recommendations
713
753
-The measured bottleneck remains the GitHub repository crawl, not the five-source RSS step. Source-aware telemetry and schema validation improve downstream reliability without adding Actions matrix startup overhead or splitting cache/API behavior.
714
+1. Treat analysis compaction/retry control as higher priority than crawler matrixing.
715
+2. Add or refine telemetry so pre-flight token estimates match the final prompt/token ledger, including press context and rendered instructions.
716
+3. Consider failing faster on repeated deterministic gate failures such as timestamp mismatch and invalid enum values, or patch/sanitize those fields before retrying.
717
+4. Gate Copilot retry count or switch earlier to no-AI/data-only when attempts exceed a duration budget.
718
+5. Resolve the `openai/gpt-4o` GitHub Models access/config mismatch, or configure an accessible fallback model.
719
+6. Track the Node.js 20 Actions deprecation, but it is not run-specific or urgent compared with analysis reliability.
720
755
-## Operational notes
756
-
757
-`correlate.py` and `render_press_context.py` now preserve article source/title/date/URL citations, label strong versus weak correlations, bound press context size to an ~8k token estimate, and keep legacy `*-techcrunch.json` and no-press fallbacks.
758
-
759
-- PR #242 merged at 2026-06-05T17:24:14Z, closing issue #237.
760
-
761
----
762
-
763
----
764
-
765
-# Bender PR #242 Copilot Review Fixes
766
-
767
-- Keep category/project-name-only press matches weak even when temporally spiking or corroborated by multiple articles/sources.
768
-- Pass both `--since` and `--until` from the crawl workflow to preserve deterministic canonical `crawl_window` metadata.
769
-- Record bounded fetch attempts and timeout telemetry on `NewsFeedSource` even when `fetch_feed()` raises before returning a feed.
770
-- Keep press-context article lookup comments aligned with the actual URL-to-title mapping.
771
-- PR #243 merged at 2026-06-05T17:34:18Z.
721
+## Issue recommendation
722
773
----
723
+Do not open a separate crawler/RSS matrix issue from this run. The existing PRD/issue direction is enough for external-news telemetry and fan-in. If a new issue is opened, make it about analysis critical-path reduction and fallback model access, not crawler parallelism.
724
725
---
726
777
-# Leela PR #243 Review
727
+## Scribe: 2026-06-05T18:27:00Z — Merged PRD/run-review decision inputs
728
779
-- Verdict: approved in substance after independent lead review.
780
-- Scope checked: issue #237 acceptance criteria follow-up, PR #242 Copilot comments, PR #243 diff, tests, CodeQL, Copilot review state.
781
-- Local validation: clean PR worktree ran `pytest tests -q` with 574 passed.
782
-- Formal GitHub approval blocked: the active account is the PR author and GitHub rejected own-PR approval.
783
-- Merge gate: wait for an independent non-Bender reviewer/approval unless repository policy explicitly permits merge with the lead approval comment.
729
+**Action:** Merged 5 decision inbox files into decisions.md:
730
+- bender-matrix-crawl-prd-input.md (Bender: matrix crawl and fan-in/fan-out design options)
731
+- farnsworth-map-reduce-analysis-prd-input.md (Farnsworth: analysis map/reduce architecture)
732
+- fry-matrix-mapreduce-qa-prd-input.md (Fry QA: PRD-ready gates and test matrix)
733
+- leela-matrix-mapreduce-prd.md (Leela: decision recommendation summary)
734
+- bender-run-27030646485-log-review.md (Bender: run analysis and directional findings)
735
785
----
786
-
787
----
788
-
789
-### 2026-06-05T17:06:31.753+00:00: User directive — Copilot Review Asynchronous Gate
736
+**Outcome:** decisions.md grew from 45948 → 91562 bytes. Inbox purged. No duplicates found in merge. Added 5 decision dividers. Content addresses crawl matrix topology, analysis map/reduce experiment design, QA gates/tests, run diagnostics, and fallback strategy.
737
791
-**By:** jmservera (via Copilot)
792
-**What:** Copilot Review is asynchronous. Before merging a PR, check whether Copilot is still reviewing and do not merge until the review has finished and any review comments are handled.
793
-**Why:** User request — captured for team memory
738
+**No archiving trigger:** decisions.md is still within typical document lifecycle size; existing PRD scope is fresh and actionable.
docs/processed/PRD-matrix-crawl-map-reduce-analysis.md
new
+798
@@ -0,0 +1,798 @@
1
+# PRD: Matrix Crawl and Map/Reduce Analysis for SquadScope
2
+
3
+**Author:** Leela (Lead/Architect)
4
+**Date:** 2026-06-05
5
+**Status:** Draft
6
+**Type:** Product / Design Requirements Document
7
+**Depends on:** docs/analysis-spec.md, docs/pipeline-validation.md, .squad/decisions.md, scripts/analysis_gate.py
8
+**Inputs synthesized:** Bender matrix crawl findings, Farnsworth map/reduce analysis findings, Fry QA gates
9
+
10
+---
11
+
12
+## Executive Summary
13
+
14
+SquadScope should not add a GitHub Actions crawl matrix by default yet. The recent implementation avoided matrix fan-out because the measured RSS work is already fast (about one second for five feeds) and the dominant crawl cost is GitHub API collection, which is constrained by cache behavior, Search API quota, and secondary-rate-limit risk. A matrix is not automatically faster; at current scale it can add runner setup, artifact fan-in, cache merge complexity, and rate-limit instability without improving the bottleneck.
15
+
16
+This PRD recommends a staged architecture:
17
+
18
+1. **Keep current crawl topology by default:** monolithic/cached GitHub crawl plus bounded in-process RSS fetching.
19
+2. **Make crawl artifacts matrix-ready:** introduce shared run context, schema validation, deterministic fan-in contracts, checksums, source status, and metrics.
20
+3. **Gate crawl matrix rollout by measurements:** RSS matrix only when source count/runtime/isolation triggers fire; GitHub matrix only after a shard experiment proves at least **25% crawl speedup**, no more than **10% API-call growth**, and no secondary-rate-limit regression.
21
+4. **Adopt map/reduce as an analysis experiment, not a crawl-speed fix:** use mapper claim ledgers and a reducer/final writer to shrink LLM context, preserve citations, dedupe claims, resolve contradictions, and satisfy the existing analysis spec/gate.
22
+
23
+The major product value is reliability and quality under growing evidence volume, not premature parallelism. Crawl fan-out should be evidence-triggered. Analysis map/reduce should be tested in dry-run/A-B mode before it can publish.
24
+
25
+A completed live workflow run, `27030646485`, strengthens this prioritization. The crawl path was healthy (213 new repos, 236 trending repos, 455 GitHub API calls, about 4m30s main crawler), and external news was healthy (5/5 sources, 39 articles, 23 relevant, zero failures/dedupe, schema v2 telemetry/checksum present). The dominant risk moved downstream: analysis took about 28m41s, failed three Copilot gates, could not use the GitHub Models `openai/gpt-4o` fallback because access was unavailable, and ultimately succeeded only through no-AI fallback. Final token ledger input was about 112.9k versus about 74.3k at preflight, confirming that analysis duration, context growth, and fallback behavior now matter more than crawl parallelism.
26
+
27
+---
28
+
29
+## Problem Statement
30
+
31
+The user asked why the team did not use a matrix to run the crawl faster, and whether a map/reduce technique could divide analysis into smaller parts to reduce LLM context.
32
+
33
+Two concerns are related but distinct:
34
+
35
+- **Crawl speed and reliability:** Can GitHub Actions matrix jobs fetch RSS or GitHub data faster than the current crawl?
36
+- **Analysis context and quality:** Can the LLM analysis stage be decomposed into smaller, citation-preserving map outputs that a reducer combines into one coherent weekly summary?
37
+
38
+The recent multi-source news implementation increased RSS coverage from one TechCrunch feed to five external sources, but the measured RSS stage remained about one second. Meanwhile the GitHub crawler still took roughly 4.5-6 minutes and used shared cache/rate-limit behavior. The completed run `27030646485` showed this path healthy: about 4m30s main crawler time, 455 API calls, 213 new repos, 236 trending repos, and a fully successful external-news artifact with 5/5 sources, 39 articles, 23 relevant, zero failures/dedupe, and schema v2 telemetry/checksum. Splitting the wrong work would increase complexity without reducing critical path. However, analysis input size is growing: raw GitHub JSON and external-news/correlation evidence can exceed tens of thousands of token-estimate before prompt instructions, learned state, and previous summaries are added. In the same completed run, final input tokens reached about 112.9k versus about 74.3k at preflight. That creates attention dilution, citation drift, structural gate failures, weaker editorial synthesis, and slow/repeated fallback behavior.
39
+
40
+Therefore this PRD separates the decisions:
41
+
42
+- **Crawl matrix:** justified only by measured runtime, source count, failure isolation, or GitHub shard experiment evidence.
43
+- **Analysis map/reduce:** justified as an experiment to reduce per-call context, improve citation discipline, and keep final output compliant with `docs/analysis-spec.md` and `scripts/analysis_gate.py`.
44
+
45
+---
46
+
47
+## Goals & Non-Goals
48
+
49
+### Goals
50
+
51
+- **G1:** Explain why the recent crawl implementation did not use a matrix for speed.
52
+- **G2:** Define measurable triggers and experiments for enabling RSS and GitHub crawl matrices.
53
+- **G3:** Specify viable matrix/fan-out/fan-in designs for RSS and GitHub crawl while preserving canonical downstream artifacts.
54
+- **G4:** Define an LLM analysis map/reduce architecture that divides work into smaller bounded contexts.
55
+- **G5:** Define mapper and reducer contracts that preserve citations, dedupe claims, expose contradictions, and satisfy the existing analysis spec/gate.
56
+- **G6:** Require observability for crawl legs, fan-in, mappers, reducers, fallbacks, token/cost behavior, and quality gates.
57
+- **G7:** Provide rollout, QA gates, risks, mitigations, and acceptance criteria before any default-on change.
58
+
59
+### Non-Goals
60
+
61
+- Implementing the matrix or map/reduce pipeline in this PRD.
62
+- Replacing the current canonical raw artifact paths by default.
63
+- Replacing `docs/analysis-spec.md` or weakening `scripts/analysis_gate.py`.
64
+- Letting mapper outputs become publishable article prose.
65
+- Adding paid services, vector databases, or embedding infrastructure for the MVP.
66
+- Publishing map/reduce output before evidence-contract validation, analysis gate pass, and human/editorial comparison.
67
+- Matrixing GitHub crawl merely because matrix jobs are available.
68
+- Treating article volume alone as proof of success.
69
+
70
+---
71
+
72
+## Current Baseline / Why No Matrix Yet
73
+
74
+### Evidence from recent runs
75
+
76
+Observed crawl timings and artifacts show:
77
+
78
+| Run | Shape | Crawl job | Main GitHub crawl | RSS/news step | Output | API observations |
79
+| --- | --- | ---: | ---: | ---: | --- | --- |
80
+| Old run `26753498571 / 78847225991` | single TechCrunch feed | ~6m23s | ~5m58s | sub-second | 196 new repos, 238 trending repos, 20 TechCrunch articles / 7 relevant | 447 API calls; Search API min remaining 24/30; core remained high |
81
+| New run `27026348186 / 79767247136` | five external RSS feeds, in-process parallel | ~5m08s | ~4m47s | ~1s | 213 new repos, 236 trending repos, 54 articles / 27 relevant | 455 API calls; Search API min remaining 24/30; core remained high |
82
+| Completed run `27030646485 / 79781846313` | five external RSS feeds | full workflow success | ~4m30s | ~1s | 213 new repos, 236 trending repos; 39 articles / 23 relevant; 5/5 sources; 0 failures/dedupe; schema v2 telemetry/checksum present | 455 API calls; crawl healthy; analysis became dominant risk |
83
+
84
+### Live-run analysis signal
85
+
86
+Run `27030646485` did not reveal a crawler problem that needs a new crawler issue. Instead, it exposed analysis as the dominant risk:
87
+
88
+- Analysis duration was about **28m41s**, much longer than the crawl critical path.
89
+- Three Copilot analysis gates failed before fallback.
90
+- GitHub Models fallback could not use `openai/gpt-4o` because access was unavailable.
91
+- The no-AI fallback ultimately passed, preserving workflow success but reducing confidence in AI-generated editorial quality.
92
+- The final token ledger was about **112.9k input tokens** versus about **74.3k preflight**, showing substantial context expansion after preflight.
93
+
94
+This strengthens the map/reduce case: compaction, retry policy, gate-aware slicing, and provider fallback behavior should be prioritized before crawl matrix work. A focused analysis issue may be useful to track these fixes, but a new crawler issue is not warranted by this run.
95
+
96
+### Why matrix was not used in the recent implementation
97
+
98
+The decision was architectural and evidence-based:
99
+
100
+1. **RSS was not the bottleneck.** Five-source RSS collection completed in about one second. A per-source matrix would repeat checkout, Python setup, dependency installation, artifact upload/download, and merge logic. That overhead is larger than the current RSS work.
101
+2. **RSS is already parallelized in-process.** The newer RSS path uses bounded worker concurrency and writes one canonical external-news artifact. This is appropriate for a small number of I/O-bound feeds.
102
+3. **GitHub API crawl is rate/cache constrained.** The GitHub crawler uses shared cache, shared token/rate-limit visibility, one star snapshot, and deterministic output. Splitting queries before measuring shard behavior risks Search API quota pressure, secondary-rate-limit regressions, duplicate candidates, cache conflicts, and incorrect star-gain semantics.
103
+4. **Downstream contracts expect canonical artifacts.** `correlate.py`, `render_press_context.py`, analysis, rebuild mode, and publishing should not need to know whether collection was single-process or matrix-based.
104
+5. **Matrix fan-out is not automatically faster.** If the slow part is GitHub API wait/backoff or cache misses, parallel jobs can simply exhaust quota faster and force serialized backoff.
105
+
106
+### Current decision baseline
107
+
108
+Default path remains:
109
+
110
+```text
111
+crawl-github (monolithic, cached, required)
112
+ +
113
+crawl-external-news (bounded in-process RSS, optional/degraded)
114
+ -> canonical raw artifacts
115
+ -> correlation / press context
116
+ -> analysis
117
+```
118
+
119
+Future work should make the artifact boundary matrix-ready without changing default topology prematurely.
120
+
121
+---
122
+
123
+## Proposed Architecture
124
+
125
+### Overview
126
+
127
+```text
128
+ shared run context
129
+ week / since / until / run_id
130
+ │
131
+ ┌────────────────┴────────────────┐
132
+ │ │
133
+ required GitHub crawl optional RSS/news crawl
134
+ monolith by default; shard in-process by default; matrix
135
+ experiment only behind gate per source only behind gate
136
+ │ │
137
+ └──────────────┬──────────────────┘
138
+ ▼
139
+ deterministic fan-in / validate
140
+ canonical data/raw/{week}.json + {week}-external-news.json
141
+ │
142
+ ▼
143
+ deterministic preflight / slicing
144
+ │
145
+ ┌──────────────┴──────────────┐
146
+ ▼ ▼
147
+ analysis mapper(s) optional press/correlation mapper(s)
148
+ claim ledgers only claim ledgers only
149
+ └──────────────┬──────────────┘
150
+ ▼
151
+ reducer: dedupe / resolve / select
152
+ │
153
+ ▼
154
+ final writer: one coherent article
155
+ │
156
+ ▼
157
+ analysis_gate.py + evidence-contract validation
158
+```
159
+
160
+### Matrix Crawl
161
+
162
+#### Design principle
163
+
164
+Matrix collection is allowed only when it improves a measured property: crawl p95, failure isolation, retry granularity, source-specific quota/credential isolation, or rebuild/debuggability. Matrix jobs must never commit or feed analysis directly. They write validated artifacts, and a fan-in job emits the canonical payload consumed downstream.
165
+
166
+#### RSS matrix option
167
+
168
+**When default path stays in-process:** current five-feed RSS collection, normal p95 below threshold, homogeneous RSS/HTTP behavior, no source-specific credentials or quotas.
169
+
170
+**When RSS matrix is justified:** enable a per-source matrix only if one or more triggers fire:
171
+
172
+- RSS/news stage p95 exceeds **60 seconds** across recent runs.
173
+- Configured external source count exceeds **10**.
174
+- A source needs independent credentials, quota policy, network isolation, retry policy, parser/runtime dependency, or failure semantics.
175
+- A single flaky/slow source repeatedly forces rerunning the whole crawl job.
176
+- Product requires per-source downloadable diagnostics even when the aggregate crawl degrades.
177
+
178
+**RSS matrix design:**
179
+
180
+1. A setup job computes a shared context: `run_id`, `week`, `since`, `until`, source config checksum, topic config checksum, and code SHA.
181
+2. Matrix legs run one source per leg with `strategy.fail-fast: false`.
182
+3. Each leg uploads exactly one artifact on `if: always()` containing either articles or a structured status/error payload.
183
+4. A fan-in job downloads all source artifacts, validates schemas/checksums/window consistency, dedupes article URLs deterministically, records per-source status, and writes canonical `data/raw/{week}-external-news.json`.
184
+5. Analysis consumes only the canonical merged artifact, never raw per-source artifacts.
185
+
186
+#### GitHub matrix option
187
+
188
+**When default path stays monolithic:** normal GitHub crawl p95 is acceptable, Search API remains tight, cache behavior is shared, star snapshot semantics remain global, or no shard experiment has proven benefit.
189
+
190
+**When GitHub matrix is justified:** enable only after a no-publish shard experiment proves all of:
191
+
192
+- At least **25% crawl wall-clock speedup** versus monolithic baseline, measured on comparable weekly windows.
193
+- No more than **10% total GitHub API-call growth** versus baseline.
194
+- No secondary-rate-limit regression: same or fewer secondary-limit events/backoff seconds.
195
+- Search API quota remains above an agreed safety floor and is visible per shard.
196
+- Same canonical raw output semantics: deterministic repo ordering, dedupe by `full_name`, stable star-gain/snapshot computation, and no downstream contract change.
197
+
198
+**GitHub matrix designs considered:**
199
+
200
+1. **Per query/category shard**
201
+ - Shard by search query group: new repos, trending repos, topic primary queries, topic secondary queries.
202
+ - Fan-in dedupes by `full_name`, applies final filtering, merges API/cache metadata, computes star gains once, and emits canonical raw JSON/snapshot.
203
+ - Highest potential speedup, highest rate-limit/cache risk.
204
+
205
+2. **Per candidate-processing shard after serialized search**
206
+ - A monolithic setup/search step gathers candidate repo names, then matrix legs fetch/normalize repository details or README metadata.
207
+ - Lower Search API risk because search remains centralized; possible speedup if core API details dominate.
208
+ - Fan-in still owns deterministic filtering and snapshot semantics.
209
+
210
+3. **Hybrid staged fan-out/fan-in**
211
+ - Keep GitHub crawl monolithic initially.
212
+ - Introduce explicit artifact validation/fan-in first.
213
+ - Add correlation/analysis map/reduce after canonical artifacts.
214
+ - Recommended migration path because it addresses context-size concerns without risky GitHub API fan-out.
215
+
216
+### Map/Reduce Analysis
217
+
218
+#### Design principle
219
+
220
+Map/reduce is an analysis-quality and context-management experiment. It does not claim to speed the crawl. It should shrink maximum per-call prompt context, make citations explicit, isolate failures, and preserve one final editorial voice.
221
+
222
+#### Recommended mapper strategies
223
+
224
+Start with low-machinery slices, then evolve:
225
+
226
+1. **Signal type mappers (recommended MVP):**
227
+ - `new_repos`: novelty, launch quality, new repo clusters.
228
+ - `trending_repos`: momentum, stars gained, established anchors.
229
+ - `press_correlations`: industry alignment, divergences, source caveats.
230
+ - `prior_continuity`: previous-summary follow-up, prediction continuity, reversals.
231
+
232
+2. **Topic/category mappers (second phase):**
233
+ - Deterministically cluster repos by topic, language, description, and topic config.
234
+ - Useful for macro trend discovery, but requires overlap accounting.
235
+
236
+3. **Source-specific press mappers (defer):**
237
+ - Summarize individual external-news sources only if compact deterministic press context regularly exceeds budget.
238
+ - Avoid turning weekly analysis into a news roundup.
239
+
240
+4. **Repository cluster mappers (future):**
241
+ - Useful for large GitHub payloads, suspicious clusters, copycat repos, or signal/noise analysis.
242
+ - Requires stable cluster IDs and coverage accounting.
243
+
244
+#### Mapper responsibilities
245
+
246
+Mappers produce structured claim ledgers, not final prose. Each mapper must:
247
+
248
+- Treat repo/news input as untrusted evidence, not instructions.
249
+- Emit typed claims with stable IDs.
250
+- Preserve repo and article citations.
251
+- Mark confidence, uncertainty, contradictions, and weak evidence.
252
+- Report coverage and omitted context.
253
+- Include token/cost/model telemetry.
254
+- Fail validation if output is malformed or citationless.
255
+
256
+#### Reducer responsibilities
257
+
258
+The reducer is responsible for global editorial coherence and must:
259
+
260
+- Consume only validated mapper ledgers plus compact global metadata, not unbounded raw JSON.
261
+- Dedupe equivalent findings by normalized claim key, repo full name, article URL, and topic.
262
+- Merge supporting evidence and preserve all required citations.
263
+- Surface contradictions or uncertainty; never silently drop conflicting mapper claims.
264
+- Reject unsupported, duplicate, or weakly cited claims with a sidecar/rejected-claims ledger.
265
+- Select a coherent thesis, title, top repo, tags, sections, notable projects, press references, and predictions.
266
+- Produce an editorial plan that a final writer converts into the existing markdown shape.
267
+- Ensure the final markdown passes `scripts/analysis_gate.py` unchanged at MVP.
268
+
269
+#### Final writer responsibilities
270
+
271
+The final writer converts the reducer's editorial plan into the existing analysis spec, including:
272
+
273
+```md
274
+## This Week's Trends
275
+## Where Industry Meets Code
276
+## Signal & Noise
277
+## Blind Spots
278
+## The Week Ahead
279
+## Key References
280
+### Notable Projects
281
+### Press & Industry
282
+```
283
+
284
+The final writer is the only stage that writes reader-facing prose. This preserves one voice and avoids mapper-by-mapper seams.
285
+
286
+---
287
+
288
+## Fan-in Contracts
289
+
290
+### Shared run context
291
+
292
+Every crawl leg, mapper, reducer, and validator must receive the same generated context:
293
+
294
+```json
295
+{
296
+ "schema_version": "run_context_v1",
297
+ "run_id": "2026-W23-<sha>",
298
+ "week": "2026-W23",
299
+ "since": "2026-06-01T00:00:00Z",
300
+ "until": "2026-06-08T00:00:00Z",
301
+ "source_config_checksum": "sha256:...",
302
+ "topic_config_checksum": "sha256:...",
303
+ "code_sha": "...",
304
+ "created_at": "2026-06-05T17:42:56Z"
305
+}
306
+```
307
+
308
+No matrix leg may compute its own week window from local wall clock.
309
+
310
+### Fan-in rules
311
+
312
+- Required GitHub artifacts missing or invalid: fail closed before analysis.
313
+- Optional RSS source failures: degrade only if minimum-source policy passes, and record explicit warnings/caveats.
314
+- Fan-in runs on `if: always()` to publish diagnostics even when some legs fail.
315
+- Same inputs must produce byte-stable canonical outputs, excluding documented timestamps from checksums.
316
+- Cache metadata must include week window, config checksum, schema version, and stale-hit status.
317
+- Reruns must not double-count articles, repos, retries, or restored stale artifacts.
318
+
319
+---
320
+
321
+## Data Contracts
322
+
323
+### Crawl leg artifact
324
+
325
+```json
326
+{
327
+ "schema_version": "crawl_leg_v1",
328
+ "run_id": "2026-W23-<sha>",
329
+ "week": "2026-W23",
330
+ "since": "2026-06-01T00:00:00Z",
331
+ "until": "2026-06-08T00:00:00Z",
332
+ "leg_id": "rss:hugging-face-blog",
333
+ "source_type": "rss|github_search|github_repo_details",
334
+ "started_at": "ISO-8601",
335
+ "finished_at": "ISO-8601",
336
+ "duration_seconds": 1.23,
337
+ "status": "success|failed|partial|skipped",
338
+ "payload": {},
339
+ "errors": [],
340
+ "metrics": {
341
+ "item_count": 9,
342
+ "relevant_count": 4,
343
+ "dedupe_count": 0,
344
+ "api_calls": 0,
345
+ "cache_hits": 0,
346
+ "stale_cache_hits": 0,
347
+ "retry_count": 0,
348
+ "rate_limit_remaining": null,
349
+ "rate_limit_resource": null
350
+ },
351
+ "checksum": "sha256:..."
352
+}
353
+```
354
+
355
+### RSS source payload
356
+
357
+```json
358
+{
359
+ "schema_version": "rss_source_v1",
360
+ "week": "2026-W23",
361
+ "source": "hugging_face_blog",
362
+ "source_config_checksum": "sha256:...",
363
+ "articles": [
364
+ {
365
+ "title": "Article title",
366
+ "url": "https://example.com/article",
367
+ "source": "Hugging Face Blog",
368
+ "published_at": "2026-06-03T12:00:00Z",
369
+ "summary": "Short retained summary",
370
+ "categories": ["ai"],
371
+ "github_links": ["https://github.com/owner/repo"],
372
+ "relevance_score": 0.82
373
+ }
374
+ ],
375
+ "source_status": {
376
+ "status": "success|failed|partial",
377
+ "attempts": 1,
378
+ "timeout_seconds": 15,
379
+ "duration_seconds": 0.7,
380
+ "error_class": null,
381
+ "error_message": null
382
+ },
383
+ "artifact_checksum": "sha256:..."
384
+}
385
+```
386
+
387
+Fan-in emits canonical `data/raw/{week}-external-news.json` with stable metadata: `schema_version`, `sources_requested`, `sources_succeeded`, `sources_failed`, `sources_with_articles`, `dedupe_count`, `errors`, `source_status`, and checksum.
388
+
389
+### GitHub shard artifact
390
+
391
+```json
392
+{
393
+ "schema_version": "github_shard_v1",
394
+ "week": "2026-W23",
395
+ "crawl_window": {"since": "ISO-8601", "until": "ISO-8601"},
396
+ "shard_id": "github:new-repos:q1",
397
+ "query": "created:... stars:...",
398
+ "query_type": "new|trending|topic_primary|topic_secondary|repo_details",
399
+ "repos": [
400
+ {
401
+ "full_name": "owner/repo",
402
+ "url": "https://github.com/owner/repo",
403
+ "stars": 123,
404
+ "stars_gained": null,
405
+ "topics": ["ai"],
406
+ "language": "Python",
407
+ "pushed_at": "ISO-8601",
408
+ "created_at": "ISO-8601",
409
+ "fork": false,
410
+ "template": false
411
+ }
412
+ ],
413
+ "api_calls_used": 12,
414
+ "cache_hits": 4,
415
+ "stale_cache_hits": 0,
416
+ "rate_limit": {
417
+ "resource": "search",
418
+ "limit": 30,
419
+ "remaining": 24,
420
+ "reset": "ISO-8601"
421
+ },
422
+ "partial_failures": [],
423
+ "artifact_checksum": "sha256:..."
424
+}
425
+```
426
+
427
+Fan-in remains responsible for dedupe, final ordering, significance filtering, cache metadata reconciliation, star snapshot/delta computation, and canonical `data/raw/{week}.json` shape.
428
+
429
+### Analysis mapper output
430
+
431
+```json
432
+{
433
+ "schema_version": "analysis_map_v1",
434
+ "run_id": "2026-W23-<sha>",
435
+ "week": "2026-W23",
436
+ "shard_id": "signal-type:new-repos",
437
+ "slice": {
438
+ "strategy": "signal_type|topic|source|repo_cluster",
439
+ "input_refs": ["data/raw/2026-W23.json#new_repos[0:50]"],
440
+ "input_token_estimate": 12000,
441
+ "repo_count": 50,
442
+ "article_count": 0
443
+ },
444
+ "coverage": {
445
+ "repo_ids_seen": ["owner/repo"],
446
+ "article_urls_seen": [],
447
+ "excluded_reason_counts": {"low_relevance": 3}
448
+ },
449
+ "findings": [
450
+ {
451
+ "claim_id": "stable-claim-id",
452
+ "claim": "A concise evidence-bound claim.",
453
+ "category": "trend|signal|noise|gap|press_correlation|press_divergence|continuity",
454
+ "source_type": "github|news|mixed|prior_summary",
455
+ "evidence_refs": [
456
+ {
457
+ "type": "repo",
458
+ "ref": "owner/repo",
459
+ "url": "https://github.com/owner/repo",
460
+ "role": "anchor|supporting|counterexample",
461
+ "evidence_note": "Why this supports or weakens the claim"
462
+ }
463
+ ],
464
+ "repo_full_name": "owner/repo",
465
+ "news_url": null,
466
+ "confidence": 0.72,
467
+ "contra_refs": [],
468
+ "uncertainties": ["stars_gained unavailable"],
469
+ "quality_flags": ["needs_reducer_review"]
470
+ }
471
+ ],
472
+ "citations": [
473
+ {"type": "repo", "url": "https://github.com/owner/repo", "title": "owner/repo"}
474
+ ],
475
+ "reference_candidates": {
476
+ "notable_projects": ["owner/repo"],
477
+ "press_articles": []
478
+ },
479
+ "token_estimate": 1800,
480
+ "model": "copilot|github-models|none",
481
+ "status": "success|failed|partial",
482
+ "errors": []
483
+}
484
+```
485
+
486
+### Reducer input
487
+
488
+```json
489
+{
490
+ "schema_version": "analysis_reduce_input_v1",
491
+ "run_id": "2026-W23-<sha>",
492
+ "week": "2026-W23",
493
+ "global_totals": {
494
+ "repos_featured": 449,
495
+ "stars_tracked": 123456,
496
+ "new_repo_count": 213,
497
+ "trending_repo_count": 236
498
+ },
499
+ "source_coverage": {
500
+ "sources_requested": ["techcrunch", "github_blog"],
501
+ "sources_succeeded": ["techcrunch"],
502
+ "sources_failed": ["github_blog"]
503
+ },
504
+ "mapper_outputs": ["analysis_map_v1 objects"],
505
+ "validation_summary": {
506
+ "malformed_maps": 0,
507
+ "missing_required_citations": 0,
508
+ "week_mismatches": 0
509
+ }
510
+}
511
+```
512
+
513
+### Reducer output / editorial plan
514
+
515
+```json
516
+{
517
+ "schema_version": "analysis_editorial_plan_v1",
518
+ "title": "Final headline candidate",
519
+ "summary": "One-sentence thesis",
520
+ "top_repo": "owner/repo",
521
+ "tags": ["ai", "developer-tools"],
522
+ "selected_claims": [
523
+ {
524
+ "claim_id": "reducer-claim-id",
525
+ "section": "This Week's Trends|Where Industry Meets Code|Signal & Noise|Blind Spots|The Week Ahead",
526
+ "merged_from": ["mapper-claim-id-1", "mapper-claim-id-2"],
527
+ "normalized_claim_key": "topic:agent-runtime-observability",
528
+ "citation_bindings": {
529
+ "repos": ["owner/repo"],
530
+ "articles": ["https://example.com/article"]
531
+ },
532
+ "confidence": 0.81,
533
+ "rationale": "Why this claim survived reduce"
534
+ }
535
+ ],
536
+ "key_references": {
537
+ "notable_projects": ["owner/repo"],
538
+ "press_articles": ["https://example.com/article"]
539
+ },
540
+ "rejected_claims": [
541
+ {"claim_id": "mapper-claim-id-3", "reason": "duplicate|unsupported|contradicted|weak_citation"}
542
+ ],
543
+ "contradictions": [
544
+ {
545
+ "claim_ids": ["a", "b"],
546
+ "resolution": "surface_as_uncertainty|prefer_claim|reject_both",
547
+ "rationale": "Evidence comparison"
548
+ }
549
+ ],
550
+ "quality_notes": ["One source failed; press claims caveated"]
551
+}
552
+```
553
+
554
+---
555
+
556
+## Observability and Metrics
557
+
558
+### Crawl metrics
559
+
560
+Per crawl leg:
561
+
562
+- `leg_id`, source name/type, status, start/end/duration.
563
+- Item count, relevant count, dedupe count, artifact size, checksum.
564
+- Cache hit/stale-hit count.
565
+- API calls, rate-limit limit/remaining/reset/resource.
566
+- Retry count, timeout count, error class/message.
567
+
568
+Aggregate crawl:
569
+
570
+- Required/optional leg counts.
571
+- Failed/skipped/degraded leg counts.
572
+- Merged artifact size/checksum.
573
+- Total API calls and per-resource rate-limit status.
574
+- Cache hit ratio and stale-cache usage.
575
+- Fan-in validation result and deterministic checksum.
576
+
577
+### Analysis metrics
578
+
579
+Per mapper:
580
+
581
+- Shard ID, input refs/checksums, input token estimate, output token estimate.
582
+- Model/provider, duration, retries, status.
583
+- Finding count, citation count, malformed citation count.
584
+- Schema validation result and coverage counts.
585
+
586
+Reducer/final writer:
587
+
588
+- Input mapper count, failed/skipped mapper count.
589
+- Duplicate claim count and contradiction count.
590
+- Rejected/merged claim counts.
591
+- Final prompt/output token estimates.
592
+- Model/provider, duration, retry count.
593
+- `analysis_gate.py` result and evidence-contract validation result.
594
+
595
+Pipeline path:
596
+
597
+- Selected path: `single-pass`, `map-reduce`, `github-models`, `no-ai`.
598
+- Fallback reason, provider/model attempted, access/permission failures, and retry count.
599
+- Copilot gate attempts/failures, GitHub Models fallback result, and no-AI fallback result.
600
+- Preflight token estimate versus final token ledger, including unexplained growth.
601
+- Analysis duration by attempt and total analysis wall-clock.
602
+- Final publish eligibility.
603
+- Human comparison score during A/B period.
604
+
605
+### Thresholds to alert or block
606
+
607
+- GitHub matrix experiment exceeds **10% API-call growth**.
608
+- Secondary-rate-limit events/backoff increase versus baseline.
609
+- RSS p95 exceeds **60 seconds**.
610
+- Source count exceeds **10**.
611
+- Single analysis prompt exceeds configured token budget.
612
+- Final token ledger exceeds preflight by more than an agreed tolerance.
613
+- Analysis duration or retry count exceeds configured budget.
614
+- Copilot/GitHub Models gates repeatedly fail and no-AI fallback becomes the only passing path.
615
+- Mapper/reducer total cost exceeds budget.
616
+- Final reducer output loses required citations or fails `analysis_gate.py`.
617
+
618
+---
619
+
620
+## Rollout Plan
621
+
622
+### Phase 0 — Baseline and contracts
623
+
624
+- Record p50/p95 durations for GitHub crawl, RSS/news crawl, fan-in/correlation, press context rendering, and analysis across at least 5-10 runs.
625
+- Define JSON schemas for crawl legs, RSS source artifacts, GitHub shards, mapper outputs, reducer input, and editorial plan.
626
+- Add validators and fixture tests for deterministic merge and analysis contract validation.
627
+- Keep current workflow behavior unchanged.
628
+
629
+### Phase 1 — Fan-in dry-run
630
+
631
+- Add a non-publishing fan-in validator that accepts current canonical artifacts and fixture per-source artifacts.
632
+- Verify canonical outputs are byte-stable for same inputs.
633
+- Add failure fixtures: missing optional RSS leg, malformed JSON, mismatched week, stale cache, duplicate URLs.
634
+
635
+### Phase 2 — Analysis map/reduce dry-run
636
+
637
+- Add feature-flagged/dry-run map/reduce path.
638
+- Mappers emit claim ledgers only.
639
+- Reducer emits editorial plan plus rejected-claims/conflicts sidecar.
640
+- Final writer output is compared to current single-pass output but is not published.
641
+- Existing single-pass/Copilot -> GitHub Models -> no-AI fallback remains source of truth.
642
+
643
+### Phase 3 — A/B comparison
644
+
645
+Run for at least four weekly cycles or equivalent replay fixtures:
646
+
647
+- Compare gate pass rate.
648
+- Compare citation preservation and unsupported-claim count.
649
+- Compare max per-call token estimate, final ledger growth versus preflight, total cost/runtime, and retry/fallback count.
650
+- Compare editorial quality: synthesis, specificity, skepticism, blind spots, and voice.
651
+- Measure rerun stability: top repo and key reference overlap.
652
+
653
+### Phase 4 — Controlled crawl matrix experiments
654
+
655
+RSS experiment:
656
+
657
+- Run RSS matrix as dry-run only when trigger threshold fires or via workflow dispatch.
658
+- Compare runtime, artifact reliability, partial-source diagnostics, and merge determinism versus in-process RSS.
659
+
660
+GitHub experiment:
661
+
662
+- Run no-publish shard experiment with isolated caches and strict API/rate telemetry.
663
+- Do not publish sharded output until fan-in matches monolithic canonical output semantics.
664
+- Enable only if >=25% speedup, <=10% API-call growth, and no secondary-rate-limit regression.
665
+
666
+### Phase 5 — Guarded default-on
667
+
668
+- Start with workflow dispatch flag.
669
+- Then scheduled dry-run.
670
+- Then default-on for map/reduce only if acceptance criteria pass.
671
+- Retain single-pass/no-AI fallback for at least one release cycle after default-on.
672
+- Crawl matrix remains independently gated; map/reduce can ship without crawl matrix.
673
+
674
+---
675
+
676
+## QA Gates
677
+
678
+### Crawl fan-out/fan-in gates
679
+
680
+- All legs share one generated `run_id`, `week`, `since`, `until`, source config checksum, topic config checksum, and code SHA.
681
+- Every leg uploads exactly one status/payload artifact on `if: always()`.
682
+- Fan-in validates schema, checksum, source names, week/window consistency, and required fields.
683
+- Same inputs produce byte-stable canonical artifacts.
684
+- Optional RSS failures are visible and gated by minimum-source-success policy.
685
+- Required GitHub failures fail closed.
686
+- Reruns do not double-count repos/articles.
687
+- Cache metadata prevents silent mixed-window or mixed-config artifacts.
688
+- Downstream analysis consumes one canonical GitHub raw payload and one canonical external-news payload regardless of collection topology.
689
+
690
+### Map/reduce gates
691
+
692
+- Mapper output is structured JSON and schema-valid, not prose-only markdown.
693
+- Each finding includes claim, evidence refs, confidence, category, source type, optional repo/news IDs, and `contra_refs`.
694
+- Reducer rejects missing/malformed citations before final writer.
695
+- Duplicate findings collapse without losing all citations.
696
+- Contradictions are resolved with rationale or surfaced as uncertainty.
697
+- Final markdown passes `scripts/analysis_gate.py` unchanged for MVP.
698
+- Final repo mentions render as `[owner/repo](https://github.com/owner/repo)` and resolve to seen repo coverage.
699
+- Final press claims cite retained article URLs.
700
+- If map/reduce fails, pipeline falls back to existing single-pass/GitHub Models/no-AI path.
701
+
702
+### Test matrix
703
+
704
+| Area | Scenario | Expected result |
705
+| --- | --- | --- |
706
+| Shared context | Matrix legs receive generated week/window | Matching `week/since/until/run_id`; mismatch fails fan-in |
707
+| Deterministic fan-in | Same fixture artifacts merged twice | Identical canonical bytes/checksum |
708
+| Optional source failure | One RSS source times out | Canonical artifact records failure; publish can continue if minimum-source policy passes |
709
+| Required GitHub failure | Required raw GitHub shard missing | Analysis blocked; failure notification path applies |
710
+| Retry accounting | Optional RSS leg fails then succeeds on retry | Retry status recorded; no duplicate articles |
711
+| Stale cache | Cache restored for wrong week/config | Rejected or marked unusable; no silent mixing |
712
+| Rate limit | GitHub Search quota near floor | Shard experiment throttles/skips fan-out; no uncontrolled bursts |
713
+| Mapper malformed output | Mapper emits prose or invalid JSON | Reducer rejects map artifact |
714
+| Mapper failure | One required mapper fails | Retry/fallback; reducer cannot silently omit it |
715
+| Duplicate claims | Same claim appears in two mappers | Reducer emits one claim with combined citations |
716
+| Contradiction | One mapper says signal, another says noise | Reducer records resolution or uncertainty |
717
+| Citation loss | Final claim lacks source refs | Evidence validation fails before publish |
718
+| Over-budget context | Reducer input exceeds token budget | Hierarchical reduce or fallback before model call |
719
+| Gate regression | Final markdown missing required heading | `analysis_gate.py` fails; fallback exercised |
720
+
721
+---
722
+
723
+## Risks and Mitigations
724
+
725
+| Risk | Impact | Mitigation |
726
+| --- | --- | --- |
727
+| Matrixing GitHub search increases rate-limit pressure | Slower or failed crawls | Require shard experiment, API-call cap, secondary-rate regression gate, central fan-in |
728
+| RSS matrix adds overhead without speed benefit | More complexity, no runtime gain | Gate on p95/source count/isolation triggers; keep in-process default |
729
+| Bad fan-in corrupts data | Duplicate repos/articles, wrong trends | Schema validation, deterministic ordering, checksums, fixture tests |
730
+| Cache conflicts or stale cache mixing | Incorrect crawl results | Cache keys include week/window/config/schema; stale use observable and bounded |
731
+| Mapper loses nuance/citations | Unsupported final claims | Claim ledger schema, citation validation, reducer rejected-claims sidecar |
732
+| Reducer hides contradictions | Misleading analysis | Require contradiction ledger and explicit resolution/surfacing |
733
+| Multiple LLM calls increase total cost | Higher spend despite smaller contexts | Token/cost ledger, budget preflight, A/B threshold before default-on |
734
+| Editorial voice fragments | Patchwork article | Mappers never write final prose; final writer creates one coherent article |
735
+| Partial mapper failure biases analysis | Missing source/category | Required/optional shard policy, retries, visible degraded state, fallback |
736
+| Prompt injection through repo/news content | Unsafe instructions influence mapper | Keep untrusted evidence boundaries in every mapper/reducer prompt |
737
+
738
+---
739
+
740
+## Acceptance Criteria
741
+
742
+### Crawl matrix acceptance
743
+
744
+A crawl matrix may be enabled by default only when:
745
+
746
+1. Baseline telemetry for current monolithic/in-process path is recorded.
747
+2. Fan-in validates every shard schema/checksum before analysis.
748
+3. Canonical downstream artifact paths remain unchanged.
749
+4. Same shard inputs produce byte-stable canonical artifacts.
750
+5. Optional RSS failures are reflected in metadata and downstream caveats.
751
+6. Required GitHub failures fail closed unless an explicit partial-data policy is approved.
752
+7. RSS matrix is triggered by source count/runtime/isolation need, not by default.
753
+8. GitHub shard experiment proves >=25% wall-clock crawl speedup.
754
+9. GitHub shard experiment keeps API-call growth <=10%.
755
+10. GitHub shard experiment shows no secondary-rate-limit regression.
756
+11. Existing crawl/correlation/press-context/rebuild tests remain green.
757
+12. New tests cover schema validation, deterministic fan-in, duplicate handling, missing shard handling, stale cache rejection, and partial failure metadata.
758
+
759
+### Map/reduce acceptance
760
+
761
+The map/reduce analysis path may become publishable only when:
762
+
763
+1. Given the same weekly raw GitHub JSON and compact press context, final markdown passes `scripts/analysis_gate.py`.
764
+2. Every final repo mention resolves to preflight/mapper coverage and is rendered as a proper GitHub markdown link.
765
+3. Every final press claim cites an article URL retained in source coverage or press context.
766
+4. Reducer emits rejected-claims and contradictions sidecars for audit.
767
+5. Duplicate claims collapse without losing citation provenance.
768
+6. Contradictions are resolved with rationale or surfaced as uncertainty.
769
+7. Final article has one editorial voice and satisfies the existing section/frontmatter shape.
770
+8. Max per-call token estimate decreases materially versus the current single-pass prompt; target **>=30% max context reduction**.
771
+9. Quality does not regress in human review against single-pass output.
772
+10. Reruns on identical input are stable: same top repo or documented reason for change, and at least 70% overlap in selected key references.
773
+11. Existing Copilot -> GitHub Models -> no-AI fallback path remains available until map/reduce beats current path on gate pass rate, citation coverage, and editorial review.
774
+
775
+---
776
+
777
+## Open Questions
778
+
779
+1. What exact RSS p95 observation window should trigger matrix work: last 5 runs, last 10 runs, or rolling 30 days?
780
+2. Should the RSS source-count trigger be exactly 10, or should it vary by source type/latency class?
781
+3. What minimum RSS source success policy is acceptable for publishing: at least one source, majority of configured sources, or required source classes?
782
+4. Which GitHub shard design should be tested first: per-query/category, serialized search plus parallel details, or another hybrid?
783
+5. What Search API safety floor should block GitHub fan-out?
784
+6. Should map/reduce mappers run as separate Actions jobs, subprocesses inside one job, or model calls orchestrated by one script?
785
+7. What is the agreed per-run token/cost/runtime budget for map/reduce experiments?
786
+8. What tolerance should be allowed between preflight token estimate and final token ledger before blocking or compacting?
787
+9. Should evidence-contract validation live inside `analysis_gate.py` or remain a separate pre-gate validator initially?
788
+10. How should hierarchical reduce trigger when reducer input is still over budget?
789
+11. Who performs the human editorial comparison during the first four A/B runs?
790
+12. Should a focused analysis issue track compaction, Copilot gate retries, GitHub Models access/fallback behavior, and no-AI fallback quality after run `27030646485`?
791
+
792
+---
793
+
794
+## Recommendation Summary
795
+
796
+Do not matrix crawl by default now. RSS is too fast at current scale, and GitHub crawl is constrained by API/cache/rate behavior that needs proof before fan-out. Make crawl artifacts matrix-ready, then gate RSS matrix on source count/runtime/isolation triggers and GitHub matrix on a shard experiment proving >=25% speedup with <=10% API-call growth and no secondary-rate-limit regression.
797
+
798
+Do experiment with analysis map/reduce. Run `27030646485` makes this the priority: crawl and external-news collection were healthy, while analysis duration, token growth, Copilot gate failures, unavailable `openai/gpt-4o` fallback access, and eventual no-AI success were the dominant risks. Use mapper claim ledgers and a reducer/final writer to reduce LLM context, preserve citations, dedupe claims, expose contradictions, and keep the final weekly summary compliant with the existing analysis spec and gate.