@cryptotaxi247 / SquadScope-1 / commits / 5d0416a

docs: archive completed TechCrunch + superseded Podcast PRDs (#365)

Operator-approved; all review threads resolved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jun 10, 2026 at 22:15 UTC 5d0416aedbfb7f611a3590b23998dc9c4e5333a9

4 files changed +350 -642

docs/PRD-techcrunch-integration.md deleted

-443

@@ -1,443 +0,0 @@

1 -# PRD: TechCrunch RSS Integration for Cross-Signal Enrichment

2 -

3 -**Author:** Farnsworth (Analyst), revised by Bender (Crawler)

4 -**Date:** 2026-05-19

5 -**Status:** Draft

6 -**Type:** Feature PRD (Enrichment Signal)

7 -**Depends on:** .squad/decisions.md (Decision #7 — Crawler Plugin Architecture), docs/PRD-topic-channels.md

8 -

9 ----

10 -

11 -## Executive Summary

12 -

13 -SquadScope tracks GitHub repository trends weekly. This PRD proposes adding TechCrunch RSS as a **supplementary enrichment signal** — not a primary data source — to detect the *delta* between press hype and actual GitHub traction. When TechCrunch covers a technology or project that also shows unusual GitHub star activity, that correlation is newsworthy. When press coverage does NOT correlate with GitHub activity, that absence is equally informative.

14 -

15 -**Key constraint acknowledged upfront:** The correlation hit rate between TechCrunch articles and specific GitHub repositories is estimated at only **5–15%**. This feature is designed as a low-cost enrichment layer that adds value when correlations exist, and degrades gracefully (adds zero noise) when they don't.

16 -

17 ----

18 -

19 -## Problem Statement

20 -

21 -### The Gap: Press Hype vs. Real Adoption

22 -

23 -1. **GitHub stars measure developer interest.** A repo gaining 500 stars in a week signals genuine traction from people who build things.

24 -

25 -2. **TechCrunch coverage measures press/VC interest.** An article about a startup or technology signals attention from the funding and media ecosystem.

26 -

27 -3. **The delta between these signals is the insight.** Three scenarios produce editorial value:

28 - - **Hype confirmed:** TechCrunch covers X, and X's GitHub repos are surging → "Real momentum, developers agree"

29 - - **Hype without substance:** TechCrunch covers Y, but Y has zero or declining GitHub activity → "Marketing over engineering"

30 - - **Quiet breakout:** No press coverage, but a repo is exploding on GitHub → "Under the radar"

31 -

32 -4. **Currently, SquadScope only sees scenario 3.** Adding press signal enables detecting scenarios 1 and 2, making the weekly digest more insightful.

33 -

34 -### Why TechCrunch Specifically

35 -

36 -- TechCrunch has a well-maintained RSS feed (`http://techcrunch.com/feed`) with full article metadata

37 -- It covers the startup/tech ecosystem most likely to overlap with open-source GitHub activity

38 -- RSS is free, requires no API key, and is stable

39 -- Other sources (HN, Reddit) can follow the same plugin pattern later

40 -

41 ----

42 -

43 -## Value Proposition: The Delta Model

44 -

45 -The value of this integration is **NOT** in summarizing TechCrunch articles (readers can read TechCrunch themselves). The value is in the **cross-reference delta**:

46 -

47 -```

48 -Value = f(TechCrunch_coverage, GitHub_activity) where:

49 - - Both high → "Confirmed trend" (correlation)

50 - - TC high, GH low → "Hype alert" (anti-correlation)

51 - - TC low, GH high → "Sleeper hit" (absence signal)

52 - - Both low → No signal (filtered out)

53 -```

54 -

55 -This positions SquadScope as providing analysis that neither TechCrunch nor GitHub alone can offer.

56 -

57 ----

58 -

59 -## Honest Assessment: Correlation Rates

60 -

61 -### Expected Hit Rates

62 -

63 -| Correlation Type | Estimated Rate | Reasoning |

64 -|-----------------|---------------|-----------|

65 -| Direct match (TC mentions a specific repo) | 2–5% | Few TC articles name exact repos |

66 -| Indirect match (TC covers technology X, repo uses topic X) | 10–15% | Broader topic matching catches more |

67 -| No correlation found | 80–93% | Most TC articles have no GitHub signal |

68 -

69 -### Why Low Rates Are Acceptable

70 -

71 -1. **Low false-positive cost:** Uncorrelated articles are simply ignored — they add zero noise to the output.

72 -2. **High value per hit:** When a correlation IS found, it's genuinely interesting editorial content.

73 -3. **Asymmetric payoff:** Even 2–3 notable correlations per week would meaningfully enrich a weekly digest.

74 -4. **Trend over time:** Cross-referencing accumulated over weeks reveals patterns invisible in any single week.

75 -

76 -### What This Feature Is NOT

77 -

78 -- NOT a TechCrunch summarizer

79 -- NOT a primary data source for SquadScope

80 -- NOT expected to produce signal every week

81 -- NOT a replacement for GitHub-native trend detection

82 -

83 ----

84 -

85 -## Filtering Strategy

86 -

87 -### The Problem: Volume

88 -

89 -TechCrunch publishes **30–50 articles per day** (210–350 per week). Without aggressive filtering, this overwhelms the pipeline with noise. The crawler must reduce this to a manageable set before any correlation attempt.

90 -

91 -### Three-Stage Filtering Pipeline

92 -

93 -```

94 -Stage 1: Category Filter (RSS metadata)

95 - Input: ~250 articles/week (full RSS feed)

96 - Filter: Keep only categories relevant to developer tools/open-source

97 - Output: ~60-80 articles/week (70% reduction)

98 - Method: Allowlist of RSS <category> tags

99 -

100 -Stage 2: Keyword Filter (title + description)

101 - Input: ~60-80 articles/week

102 - Filter: Must contain technology/developer keywords

103 - Output: ~20-30 articles/week (60% reduction)

104 - Method: Keyword scoring (open-source, GitHub, developer, API, SDK, framework, etc.)

105 -

106 -Stage 3: Entity Extraction (lightweight)

107 - Input: ~20-30 articles/week

108 - Filter: Extract mentioned technologies, companies, project names

109 - Output: ~20-30 enriched article records with entity tags

110 - Method: Regex patterns + known project name dictionary

111 -```

112 -

113 -### Category Allowlist (Initial)

114 -

115 -```yaml

116 -allowed_categories:

117 - - Apps

118 - - Artificial Intelligence

119 - - Cloud

120 - - Developer

121 - - Enterprise

122 - - Hardware

123 - - Open Source

124 - - Robotics

125 - - Security

126 - - Startups

127 -

128 -blocked_categories:

129 - - Media & Entertainment

130 - - Transportation

131 - - Government & Policy

132 - - Crypto # Too noisy, low GitHub correlation

133 -```

134 -

135 -### Keyword Scoring

136 -

137 -Each article gets a relevance score (0–10) based on title + description:

138 -

139 -| Keyword Group | Weight | Examples |

140 -|--------------|--------|----------|

141 -| Direct GitHub mentions | +5 | "GitHub", "open source", "repository" |

142 -| Developer tools | +3 | "API", "SDK", "framework", "library", "CLI" |

143 -| Technology names | +2 | "Python", "Rust", "Kubernetes", "LLM" |

144 -| Funding/startup | +1 | "raises", "Series A", "launch" |

145 -

146 -**Threshold:** Articles scoring ≥ 3 proceed to entity extraction. Expected pass rate: ~40% of category-filtered articles.

147 -

148 ----

149 -

150 -## Temporal Alignment

151 -

152 -### The Problem: RSS is Real-Time, SquadScope is Weekly

153 -

154 -TechCrunch publishes continuously. SquadScope runs weekly (Monday 06:53 UTC). This creates a timing mismatch:

155 -

156 -- An article published Tuesday about Project X won't be seen until the following Monday

157 -- By then, the GitHub star surge may have already peaked and fallen

158 -

159 -### Solution: Weekly Batch with 7-Day Window

160 -

161 -```

162 -┌─────────────────────────────────────────────────────┐

163 -│ Monday 06:53 UTC: Crawl job runs │

164 -│ │

165 -│ 1. Fetch all RSS items from past 7 days │

166 -│ 2. Filter (3-stage pipeline above) │

167 -│ 3. Extract entities from filtered articles │

168 -│ 4. Cross-reference entities against weekly │

169 -│ GitHub trending repos (already collected) │

170 -│ 5. Output correlation data for Farnsworth │

171 -└─────────────────────────────────────────────────────┘

172 -```

173 -

174 -### Why Weekly Batch Is Sufficient

175 -

176 -1. **SquadScope is a weekly digest.** Real-time alerting is out of scope.

177 -2. **7-day accumulation helps.** A trend covered across multiple articles in a week is stronger signal.

178 -3. **GitHub stars data is also weekly.** Both signals align on the same time window.

179 -4. **Simplicity:** No state management, no incremental polling, no deduplication across runs.

180 -

181 -### Freshness Guarantee

182 -

183 -- RSS feed items older than 7 days are discarded

184 -- If the RSS feed doesn't contain 7 days of history (TechCrunch's feed typically holds 20–30 items), supplement with the feed's full available content

185 -- Each item's `<pubDate>` is checked against the collection window

186 -

187 ----

188 -

189 -## Correlation Approach

190 -

191 -### Entity-to-Repository Matching

192 -

193 -```

194 -TechCrunch Entity → GitHub Signal

195 -─────────────────────────────────────────────────

196 -"Anthropic" (company) → repos with topic:anthropic or org:anthropic

197 -"LangChain" (project) → repo langchain-ai/langchain stars_gained

198 -"Rust 2024 edition" (tech) → repos with topic:rust AND stars_gained > threshold

199 -"Series B: Acme Corp" → repos owned by acme-corp org

200 -```

201 -

202 -### Matching Strategies (in priority order)

203 -

204 -1. **Exact name match:** Article mentions "LangChain" → search for repos named `langchain*`

205 -2. **Organization match:** Article mentions company → search GitHub org

206 -3. **Topic match:** Article discusses technology → match against repo topics

207 -4. **Description match:** Fuzzy match article entities against repo descriptions

208 -

209 -### Scoring Correlation Strength

210 -

211 -| Match Type | Confidence | Example |

212 -|-----------|-----------|---------|

213 -| Exact repo name in article | 0.9 | "...announced on their GitHub repo langchain-ai/langchain..." |

214 -| Organization name + topic overlap | 0.7 | Article about Anthropic + repo topics include "claude" |

215 -| Technology keyword + trending | 0.5 | Article about "Rust" + Rust repo trending |

216 -| Company name only (no GitHub signal) | 0.3 | Article about startup with no public repos |

217 -

218 -### Output Format

219 -

220 -```json

221 -{

222 - "week": "2026-W21",

223 - "correlations": [

224 - {

225 - "article_title": "LangChain raises $25M Series A",

226 - "article_url": "https://techcrunch.com/...",

227 - "article_date": "2026-05-15",

228 - "matched_repos": ["langchain-ai/langchain"],

229 - "match_type": "exact_name",

230 - "confidence": 0.9,

231 - "github_signal": {

232 - "stars_gained": 847,

233 - "percentile": 98

234 - },

235 - "delta_type": "confirmed_trend"

236 - }

237 - ],

238 - "unmatched_articles": 24,

239 - "total_filtered_articles": 27

240 -}

241 -```

242 -

243 ----

244 -

245 -## Technical Implementation

246 -

247 -### Architecture: Plugin Pattern (Decision #7)

248 -

249 -This integration implements the `DataSource` protocol defined in Decision #7:

250 -

251 -```python

252 -class TechCrunchSource:

253 - """TechCrunch RSS crawler plugin for SquadScope."""

254 -

255 - def get_name(self) -> str:

256 - return "techcrunch"

257 -

258 - def get_rate_limits(self) -> RateLimits:

259 - return RateLimits(

260 - requests_per_minute=10, # Polite RSS polling

261 - retry_after_seconds=60

262 - )

263 -

264 - def crawl(self, window_start: datetime, window_end: datetime) -> CrawlResult:

265 - """Fetch, filter, and extract entities from TechCrunch RSS."""

266 - raw_items = self._fetch_rss(window_start, window_end)

267 - filtered = self._apply_filters(raw_items)

268 - enriched = self._extract_entities(filtered)

269 - return CrawlResult(source="techcrunch", items=enriched)

270 -```

271 -

272 -### File Layout

273 -

274 -```

275 -scripts/

276 - sources/

277 - __init__.py

278 - base.py # DataSource protocol

279 - techcrunch.py # TechCrunch RSS plugin

280 - config/

281 - techcrunch.yml # Category allowlist, keywords, thresholds

282 -data/

283 - enrichment/

284 - techcrunch/

285 - 2026-W21.json # Weekly filtered articles + entities

286 - correlations/

287 - 2026-W21.json # Cross-reference results (output for Farnsworth)

288 -```

289 -

290 -### Dependencies

291 -

292 -| Dependency | Purpose | Size Impact |

293 -|-----------|---------|-------------|

294 -| `feedparser` | RSS parsing | ~200 KB, pure Python |

295 -| `re` (stdlib) | Entity extraction patterns | None |

296 -| `datetime` (stdlib) | Window filtering | None |

297 -

298 -No additional API keys or authentication required. RSS is public.

299 -

300 -### Integration with Existing Crawl Workflow

301 -

302 -```yaml

303 -# In crawl-and-publish.yml (additions only)

304 -- name: Crawl TechCrunch RSS

305 - run: |

306 - python3 scripts/sources/techcrunch.py \

307 - --window-days 7 \

308 - --output data/enrichment/techcrunch/$WEEK.json

309 - env:

310 - WEEK: ${{ env.WEEK }}

311 -

312 -- name: Cross-reference correlations

313 - run: |

314 - python3 scripts/correlate.py \

315 - --github-data data/raw/$WEEK.json \

316 - --techcrunch-data data/enrichment/techcrunch/$WEEK.json \

317 - --output data/correlations/$WEEK.json

318 -```

319 -

320 -### Error Handling

321 -

322 -| Failure Mode | Response | Impact |

323 -|-------------|----------|--------|

324 -| RSS feed unreachable | Retry 3×, then skip TechCrunch for this week | None — enrichment is optional |

325 -| RSS feed format changed | Log warning, skip parsing, open issue | None — graceful degradation |

326 -| Zero correlations found | Normal — output empty correlations file | Expected most weeks |

327 -| Malformed XML in feed | Skip malformed items, process rest | Partial data is fine |

328 -

329 ----

330 -

331 -## Cost Estimate

332 -

333 -### Compute Cost

334 -

335 -| Resource | Usage | Cost |

336 -|---------|-------|------|

337 -| RSS fetch | 1 HTTP request/week | $0.00 |

338 -| Python processing | ~5 seconds CPU | $0.00 (free Actions minutes) |

339 -| Correlation script | ~2 seconds CPU | $0.00 |

340 -| **Total infrastructure cost** | | **$0.00/week** |

341 -

342 -### Token Cost (if Farnsworth uses correlations in analysis)

343 -

345 -|-----------|------|--------|-------------|

346 -| Correlations JSON (typical week, 2–5 hits) | ~2 KB | ~570 | +$0.002/week |

347 -| Correlations JSON (zero hits) | ~0.2 KB | ~57 | +$0.0002/week |

348 -| Correlations JSON (exceptional week, 10+ hits) | ~5 KB | ~1,400 | +$0.004/week |

349 -

350 -**Annual token cost impact: $0.10–$0.21/year** (negligible relative to $16/year baseline).

351 -

352 -### Development Cost

353 -

354 -| Task | Effort | Priority |

355 -|------|--------|----------|

356 -| `techcrunch.py` plugin | 2–3 hours | Medium |

357 -| `correlate.py` script | 2–3 hours | Medium |

358 -| Configuration + tests | 1–2 hours | Medium |

359 -| Workflow integration | 1 hour | Low |

360 -| **Total** | **6–9 hours** | |

361 -

362 ----

363 -

364 -## Success Criteria

365 -

366 -### Quantitative Metrics (measured after 8 weeks of operation)

367 -

368 -| Metric | Target | Measurement |

369 -|--------|--------|-------------|

370 -| RSS fetch success rate | ≥ 95% | Weeks with successful fetch / total weeks |

371 -| Filter reduction ratio | 85–95% reduction | (Raw articles - filtered) / raw articles |

372 -| Correlation hit rate | ≥ 5% of filtered articles | Articles with ≥1 GitHub match / filtered articles |

373 -| False positive rate | ≤ 2% | Matches marked incorrect in manual review / total matches |

374 -| Zero-noise weeks | 100% | Weeks where zero-correlation produces zero output noise |

375 -| Enrichment value (subjective) | ≥ 3/5 quality rating | Monthly review: "Did correlations improve the digest?" |

376 -

377 -### Qualitative Success Indicators

378 -

379 -- At least 1 "hype vs reality" insight per month that wouldn't exist without this signal

380 -- Zero instances where TechCrunch noise degrades the digest quality

381 -- The feature is invisible when it has nothing useful to contribute

382 -

383 -### Failure Criteria (triggers feature removal)

384 -

385 -- Hit rate below 2% after 8 weeks → feature adds complexity without value

386 -- False positives above 10% → feature introduces noise

387 -- RSS feed breaks and stays broken for 4+ consecutive weeks → dependency unreliable

388 -- Farnsworth (analyst) consistently ignores correlation data in analysis → no downstream value

389 -

390 ----

391 -

392 -## Phased Rollout

393 -

394 -### Phase 1: RSS Collection Only (Week 1–2)

395 -

396 -- Implement `techcrunch.py` with 3-stage filtering

397 -- Output `data/enrichment/techcrunch/{week}.json`

398 -- No integration with analysis — just collect and validate filter quality

399 -- **Exit criteria:** Filter reduces volume by ≥ 80%, entity extraction produces meaningful tags

400 -

401 -### Phase 2: Correlation Script (Week 3–4)

402 -

403 -- Implement `correlate.py` cross-reference logic

404 -- Output `data/correlations/{week}.json`

405 -- Manual review of correlation quality for 2 weeks

406 -- **Exit criteria:** Hit rate ≥ 3%, false positive rate ≤ 5%

407 -

408 -### Phase 3: Analysis Integration (Week 5–6)

409 -

410 -- Farnsworth consumes `data/correlations/{week}.json` in analysis prompt

411 -- Correlation data appears in weekly digest when relevant

412 -- **Exit criteria:** At least 1 correlation adds editorial value in 2 of 4 weeks

413 -

414 -### Phase 4: Steady State (Week 7+)

415 -

416 -- Monitor success metrics

417 -- Tune keyword lists and category filters based on actual hit rates

418 -- Consider adding second source (HN) if TechCrunch proves the plugin model

419 -

420 ----

421 -

422 -## Open Questions

423 -

425 -|---|----------|--------|---------------------|

426 -| OQ1 | Does TechCrunch's RSS feed include full article text or just excerpts? | Medium — affects entity extraction quality | Spike: inspect actual feed content. If excerpts only, extraction limited to title + summary. |

431 -| OQ6 | Can we use GitHub's topic taxonomy to improve matching? | Medium — could boost hit rate | Investigate `GET /repos/{owner}/{repo}/topics` coverage during Phase 2. |

432 -

433 ----

434 -

435 -## Relationship to Other PRDs

436 -

437 -- **PRD-topic-channels.md:** Topic channels define per-domain crawling. TechCrunch integration is orthogonal — it enriches ANY topic channel with press signal. A `rust` channel could correlate TechCrunch Rust articles with Rust repo trends.

438 -- **PRD-cost-estimation.md:** TechCrunch adds negligible cost ($0.10–$0.21/year in tokens). No budget concern.

439 -- **Decision #7 (Plugin Architecture):** TechCrunch is the first non-GitHub `DataSource` plugin, validating the extensible crawler design.

440 -

441 ----

442 -

443 -*This PRD will be updated with actual hit rates and filter performance after Phase 1 completes (target: 2 weeks post-implementation).*

docs/processed/PRD-podcast-generation.md renamed

docs/processed/PRD-techcrunch-integration.md

+350 -199

@@ -1,301 +1,452 @@

1 -# PRD: TechCrunch RSS Integration for Cross-Source Trend Correlation

1 +# PRD: TechCrunch RSS Integration for Cross-Signal Enrichment

3 -**Author:** Farnsworth (Analyst/Content Curator)

3 +**Author:** Farnsworth (Analyst), revised by Bender (Crawler)

4 **Date:** 2026-05-19

5 -**Status:** Draft

6 -**Type:** Feature PRD

7 -**Depends on:** .squad/decisions.md (Decision #7: Crawler Plugin Architecture), docs/PRD-topic-channels.md

5 +**Status:** Completed — implemented and archived 2026-06-10 (see `scripts/techcrunch_crawler.py`, `tests/test_techcrunch_crawler.py`, and `data/raw/*-external-news.json`)

6 +**Type:** Feature PRD (Enrichment Signal)

7 +**Depends on:** `docs/processed/PRD-topic-channels.md`

8 +

9 +> **Archived as-is.** This PRD is preserved as a historical planning record. The current canonical paths, field names, and interfaces live in code — see `scripts/techcrunch_crawler.py`, `config/external_news_sources.json`, and `.github/workflows/crawl-and-publish.yml`. The most misleading specifics below have been corrected to match what shipped (the crawler now ingests multiple external news sources, not TechCrunch alone).

11 ---

13 ## Executive Summary

13 -SquadScope currently derives all insights from a single signal source: GitHub activity. While GitHub reveals *what developers are building*, it cannot tell us *why* activity is spiking — whether it's organic community interest, a VC-backed launch, or a viral TechCrunch article driving attention. This PRD proposes integrating TechCrunch's RSS feed as SquadScope's first non-GitHub data source, enabling **cross-source trend correlation** that distinguishes organic momentum from press-driven hype.

15 +SquadScope tracks GitHub repository trends weekly. This PRD proposes adding TechCrunch RSS as a **supplementary enrichment signal** — not a primary data source — to detect the *delta* between press hype and actual GitHub traction. When TechCrunch covers a technology or project that also shows unusual GitHub star activity, that correlation is newsworthy. When press coverage does NOT correlate with GitHub activity, that absence is equally informative.

15 -**Key insight:** GitHub star surges often lag TechCrunch coverage by 24–72 hours. Detecting this pattern lets SquadScope editorially distinguish "genuinely important" (organic growth) from "temporarily hyped" (press-driven spike that fades within a week).

17 +**Key constraint acknowledged upfront:** The correlation hit rate between TechCrunch articles and specific GitHub repositories is estimated at only **5–15%**. This feature is designed as a low-cost enrichment layer that adds value when correlations exist, and degrades gracefully (adds zero noise) when they don't.

19 ---

21 ## Problem Statement

21 -### What GitHub data alone cannot tell us

23 +### The Gap: Press Hype vs. Real Adoption

24 +

25 +1. **GitHub stars measure developer interest.** A repo gaining 500 stars in a week signals genuine traction from people who build things.

23 -1. **Causality is invisible.** A repo gaining 2,000 stars in a week is interesting, but *why* matters editorially. Is it because the project shipped a breakthrough feature, or because TechCrunch wrote about it and HN amplified?

27 +2. **TechCrunch coverage measures press/VC interest.** An article about a startup or technology signals attention from the funding and media ecosystem.

25 -2. **Funding and launch context is missing.** When a startup raises a $50M Series B and open-sources their core library, GitHub shows a star spike — but without the funding context, the analysis misattributes organic community excitement.

29 +3. **The delta between these signals is the insight.** Three scenarios produce editorial value:

30 + - **Hype confirmed:** TechCrunch covers X, and X's GitHub repos are surging → "Real momentum, developers agree"

31 + - **Hype without substance:** TechCrunch covers Y, but Y has zero or declining GitHub activity → "Marketing over engineering"

32 + - **Quiet breakout:** No press coverage, but a repo is exploding on GitHub → "Under the radar"

27 -3. **Industry narrative gaps.** SquadScope's "Gaps" section (what's missing from the conversation) is currently limited to what's absent from GitHub. But sometimes the gap is between what the industry *claims* to care about (per press coverage) and what's actually *being built* (per GitHub).

34 +4. **Currently, SquadScope only sees scenario 3.** Adding press signal enables detecting scenarios 1 and 2, making the weekly digest more insightful.

29 -4. **Hype detection requires a baseline.** To identify noise, you need to know what the press machine is amplifying. Without press data, everything on GitHub looks equally "organic."

36 +### Why TechCrunch Specifically

31 -5. **Prediction accuracy suffers.** The topic-channels PRD envisions a prediction ledger. Cross-referencing press coverage with subsequent GitHub activity dramatically improves prediction calibration.

38 +- TechCrunch has a well-maintained RSS feed (`https://techcrunch.com/feed/`) with full article metadata

39 +- It covers the startup/tech ecosystem most likely to overlap with open-source GitHub activity

40 +- RSS is free, requires no API key, and is stable

41 +- Other sources (HN, Reddit) can follow the same plugin pattern later

43 ---

35 -## Value Proposition

45 +## Value Proposition: The Delta Model

46 +

47 +The value of this integration is **NOT** in summarizing TechCrunch articles (readers can read TechCrunch themselves). The value is in the **cross-reference delta**:

48 +

49 +```

50 +Value = f(TechCrunch_coverage, GitHub_activity) where:

51 + - Both high → "Confirmed trend" (correlation)

52 + - TC high, GH low → "Hype alert" (anti-correlation)

53 + - TC low, GH high → "Sleeper hit" (absence signal)

54 + - Both low → No signal (filtered out)

55 +```

37 -### For SquadScope readers

57 +This positions SquadScope as providing analysis that neither TechCrunch nor GitHub alone can offer.

58 +

59 +---

39 -| Current State (GitHub-only) | With TechCrunch Correlation |

40 -|---|---|

41 -| "Repo X gained 3,000 stars this week" | "Repo X gained 3,000 stars after TechCrunch covered their $30M raise — watch if stars sustain past week 2" |

42 -| "These 5 AI repos are trending" | "3 of 5 trending AI repos correlate with press coverage; 2 show organic growth (stronger signal)" |

43 -| "Gap: No new observability tools" | "Gap: TechCrunch covered 4 observability startups this month, but none have meaningful GitHub traction yet — vaporware risk" |

61 +## Honest Assessment: Correlation Rates

45 -### For SquadScope's editorial stance

63 +### Expected Hit Rates

47 -- **Critical thinking becomes measurable:** "Press-amplified vs. organically growing" is a concrete, data-backed editorial judgment

48 -- **Signal vs. noise gets sharper:** Hype detection moves from vibes-based to correlation-based

49 -- **The Gaps section gains depth:** Disconnects between press narrative and actual developer activity become visible

65 +| Correlation Type | Estimated Rate | Reasoning |

66 +|-----------------|---------------|-----------|

67 +| Direct match (TC mentions a specific repo) | 2–5% | Few TC articles name exact repos |

68 +| Indirect match (TC covers technology X, repo uses topic X) | 10–15% | Broader topic matching catches more |

69 +| No correlation found | 80–93% | Most TC articles have no GitHub signal |

70 +

71 +### Why Low Rates Are Acceptable

72 +

73 +1. **Low false-positive cost:** Uncorrelated articles are simply ignored — they add zero noise to the output.

74 +2. **High value per hit:** When a correlation IS found, it's genuinely interesting editorial content.

75 +3. **Asymmetric payoff:** Even 2–3 notable correlations per week would meaningfully enrich a weekly digest.

76 +4. **Trend over time:** Cross-referencing accumulated over weeks reveals patterns invisible in any single week.

77 +

78 +### What This Feature Is NOT

79 +

80 +- NOT a TechCrunch summarizer

81 +- NOT a primary data source for SquadScope

82 +- NOT expected to produce signal every week

83 +- NOT a replacement for GitHub-native trend detection

85 ---

53 -## Correlation Model

87 +## Filtering Strategy

88 +

89 +### The Problem: Volume

90 +

91 +TechCrunch publishes **30–50 articles per day** (210–350 per week). Without aggressive filtering, this overwhelms the pipeline with noise. The crawler must reduce this to a manageable set before any correlation attempt.

92 +

93 +### Three-Stage Filtering Pipeline

55 -### How TechCrunch articles map to GitHub signals

95 +> **As-shipped note:** This three-stage design (category allowlist → keyword filter → entity extraction) was the original plan and is **not** how the crawler ultimately shipped. The implemented crawler (`scripts/techcrunch_crawler.py`) uses a single `compute_relevance_score()` heuristic and keeps articles whose `relevance_score` meets a threshold (currently `>= 0.4`); there is no RSS `<category>` allowlist or separate multi-stage pipeline. The pipeline below is preserved as historical design rationale only.

97 ```

58 -┌─────────────────┐ ┌──────────────────────┐

59 -│ TechCrunch RSS │ │ GitHub Weekly Crawl │

60 -│ (article feed) │ │ (repo activity) │

61 -└────────┬────────┘ └──────────┬───────────┘

62 - │ │

63 - ▼ ▼

64 -┌─────────────────┐ ┌──────────────────────┐

65 -│ Extract: │ │ Extract: │

66 -│ - Company/proj │ │ - Repo name/org │

67 -│ - Category │ │ - Star delta │

68 -│ - Funding amt │ │ - Fork delta │

69 -│ - GitHub links │ │ - Contributor growth │

70 -└────────┬────────┘ └──────────┬───────────┘

71 - │ │

72 - └──────────┬───────────────────┘

73 - ▼

74 - ┌─────────────────────┐

75 - │ Correlation Engine │

76 - │ (fuzzy matching) │

77 - └──────────┬──────────┘

78 - ▼

79 - ┌─────────────────────┐

80 - │ Annotated Analysis │

81 - │ - press_correlated │

82 - │ - organic_growth │

83 - │ - hype_risk_score │

84 - └─────────────────────┘

98 +Stage 1: Category Filter (RSS metadata)

99 + Input: ~250 articles/week (full RSS feed)

100 + Filter: Keep only categories relevant to developer tools/open-source

101 + Output: ~60-80 articles/week (70% reduction)

102 + Method: Allowlist of RSS <category> tags

103 +

104 +Stage 2: Keyword Filter (title + description)

105 + Input: ~60-80 articles/week

106 + Filter: Must contain technology/developer keywords

107 + Output: ~20-30 articles/week (60% reduction)

108 + Method: Keyword scoring (open-source, GitHub, developer, API, SDK, framework, etc.)

109 +

110 +Stage 3: Entity Extraction (lightweight)

111 + Input: ~20-30 articles/week

112 + Filter: Extract mentioned technologies, companies, project names

113 + Output: ~20-30 enriched article records with entity tags

114 + Method: Regex patterns + known project name dictionary

115 ```

116

87 -### Correlation heuristics

117 +### Category Allowlist (Initial)

118 +

119 +```yaml

120 +allowed_categories:

121 + - Apps

122 + - Artificial Intelligence

123 + - Cloud

124 + - Developer

125 + - Enterprise

126 + - Hardware

127 + - Open Source

128 + - Robotics

129 + - Security

130 + - Startups

131 +

132 +blocked_categories:

133 + - Media & Entertainment

134 + - Transportation

135 + - Government & Policy

136 + - Crypto # Too noisy, low GitHub correlation

137 +```

138 +

139 +### Keyword Scoring

140

89 -1. **Direct link match:** TechCrunch article contains a GitHub URL → exact match to crawled repo

90 -2. **Organization match:** Article mentions company X → match to `github.com/X/*` repos gaining stars

91 -3. **Project name match:** Article title/body contains project name → fuzzy match against repo names in weekly crawl

92 -4. **Category correlation:** Article tagged "AI" published Monday → AI-category repos spiking by Thursday

93 -5. **Temporal lag analysis:** Stars gained within 72 hours of article publication → likely press-correlated

141 +Each article gets a relevance score (0–10) based on title + description:

142

95 -### Hype risk scoring

143 +| Keyword Group | Weight | Examples |

144 +|--------------|--------|----------|

145 +| Direct GitHub mentions | +5 | "GitHub", "open source", "repository" |

146 +| Developer tools | +3 | "API", "SDK", "framework", "library", "CLI" |

147 +| Technology names | +2 | "Python", "Rust", "Kubernetes", "LLM" |

148 +| Funding/startup | +1 | "raises", "Series A", "launch" |

149

97 -| Pattern | Hype Risk | Editorial Label |

98 -|---------|-----------|-----------------|

99 -| Stars spike post-article, sustain 2+ weeks | Low | "Press-validated, community-sustained" |

100 -| Stars spike post-article, decay within 7 days | High | "Press-driven hype, fading interest" |

101 -| Stars growing before any press coverage | Very Low | "Organic growth — genuinely interesting" |

102 -| Press coverage but no GitHub activity | Medium | "Announced but unbuilt / closed-source" |

150 +**Threshold:** Articles scoring ≥ 3 proceed to entity extraction. Expected pass rate: ~40% of category-filtered articles.

151

152 ---

153

106 -## Technical Approach

154 +## Temporal Alignment

155

108 -### Data Source: TechCrunch RSS

156 +### The Problem: RSS is Real-Time, SquadScope is Weekly

157

110 -- **Feed URL:** `https://techcrunch.com/feed/`

111 -- **Format:** RSS 2.0 / XML

112 -- **Update frequency:** ~20-40 articles/day

113 -- **Relevant categories:** Startups, Apps, AI, Funding, Open Source

114 -- **Rate limits:** None (public RSS)

115 -- **Content available in feed:** Title, excerpt/summary, author, publish date, categories, link

158 +TechCrunch publishes continuously. SquadScope runs weekly (Monday 06:53 UTC). This creates a timing mismatch:

159

117 -### Architecture: Fits Decision #7 (Crawler Plugin)

160 +- An article published Tuesday about Project X won't be seen until the following Monday

161 +- By then, the GitHub star surge may have already peaked and fallen

162

119 -The existing `DataSource` protocol interface applies directly:

163 +### Solution: Weekly Batch with 7-Day Window

164

121 -```python

122 -class TechCrunchSource:

123 - """Crawler plugin for TechCrunch RSS feed."""

165 +```

166 +┌─────────────────────────────────────────────────────┐

167 +│ Monday 06:53 UTC: Crawl job runs │

168 +│ │

169 +│ 1. Fetch all RSS items from past 7 days │

170 +│ 2. Filter (3-stage pipeline above) │

171 +│ 3. Extract entities from filtered articles │

172 +│ 4. Cross-reference entities against weekly │

173 +│ GitHub trending repos (already collected) │

174 +│ 5. Output correlation data for Farnsworth │

175 +└─────────────────────────────────────────────────────┘

176 +```

177

125 - def get_name(self) -> str:

126 - return "techcrunch"

178 +### Why Weekly Batch Is Sufficient

179

128 - def get_rate_limits(self) -> RateLimits:

129 - return RateLimits(requests_per_hour=10, burst=5)

180 +1. **SquadScope is a weekly digest.** Real-time alerting is out of scope.

181 +2. **7-day accumulation helps.** A trend covered across multiple articles in a week is stronger signal.

182 +3. **GitHub stars data is also weekly.** Both signals align on the same time window.

183 +4. **Simplicity:** No state management, no incremental polling, no deduplication across runs.

184

131 - async def crawl(self, config: CrawlConfig) -> CrawlResult:

132 - """Fetch and parse TechCrunch RSS, extract structured articles."""

133 - ...

134 -```

185 +### Freshness Guarantee

186

136 -### Data flow integration

187 +- RSS feed items older than 7 days are discarded

188 +- If the RSS feed doesn't contain 7 days of history (TechCrunch's feed typically holds 20–30 items), supplement with the feed's full available content

189 +- Each item's `<pubDate>` is checked against the collection window

190 +

191 +---

192 +

193 +## Correlation Approach

194 +

195 +### Entity-to-Repository Matching

196

197 ```

139 -Existing: data/raw/YYYY-WNN.json (GitHub crawl)

140 -New: data/raw/YYYY-WNN-techcrunch.json (TechCrunch crawl)

141 -Merged: data/analyzed/YYYY-WNN-summary.md (cross-referenced analysis)

198 +TechCrunch Entity → GitHub Signal

199 +─────────────────────────────────────────────────

200 +"Anthropic" (company) → repos with topic:anthropic or org:anthropic

201 +"LangChain" (project) → repo langchain-ai/langchain stars_gained

202 +"Rust 2024 edition" (tech) → repos with topic:rust AND stars_gained > threshold

203 +"Series B: Acme Corp" → repos owned by acme-corp org

204 ```

205

144 -### RSS parsing requirements

206 +### Matching Strategies (in priority order)

207

146 -| Requirement | Approach |

147 -|-------------|----------|

148 -| XML parsing | `feedparser` (Python) — battle-tested RSS library |

149 -| Category extraction | Map TC categories to SquadScope topic taxonomy |

150 -| GitHub link extraction | Regex scan article content for `github.com` URLs |

151 -| Entity extraction | Match company/project names against crawled repos |

152 -| Deduplication | Hash on article URL; skip already-processed items |

153 -| Storage | JSON array, same weekly naming as GitHub crawl |

208 +1. **Exact name match:** Article mentions "LangChain" → search for repos named `langchain*`

209 +2. **Organization match:** Article mentions company → search GitHub org

210 +3. **Topic match:** Article discusses technology → match against repo topics

211 +4. **Description match:** Fuzzy match article entities against repo descriptions

212

155 -### Output schema (per article)

213 +### Scoring Correlation Strength

214 +

215 +| Match Type | Confidence | Example |

216 +|-----------|-----------|---------|

217 +| Exact repo name in article | 0.9 | "...announced on their GitHub repo langchain-ai/langchain..." |

218 +| Organization name + topic overlap | 0.7 | Article about Anthropic + repo topics include "claude" |

219 +| Technology keyword + trending | 0.5 | Article about "Rust" + Rust repo trending |

220 +| Company name only (no GitHub signal) | 0.3 | Article about startup with no public repos |

221 +

222 +### Output Format

223

224 ```json

225 {

159 - "source": "techcrunch",

160 - "title": "Anthropic open-sources Claude's tool-use framework",

161 - "url": "https://techcrunch.com/2026/05/15/...",

162 - "published_at": "2026-05-15T14:30:00Z",

163 - "categories": ["ai", "open-source", "funding"],

164 - "github_links": ["https://github.com/anthropics/tool-use-sdk"],

165 - "entities": ["Anthropic", "Claude"],

166 - "funding_amount": null,

167 - "relevance_score": 0.85

226 + "week": "2026-W21",

227 + "correlations": [

228 + {

229 + "article_title": "LangChain raises $25M Series A",

230 + "article_url": "https://techcrunch.com/...",

231 + "article_date": "2026-05-15",

232 + "matched_repos": ["langchain-ai/langchain"],

233 + "match_type": "exact_name",

234 + "confidence": 0.9,

235 + "github_signal": {

236 + "stars_gained": 847,

237 + "percentile": 98

238 + },

239 + "delta_type": "confirmed_trend"

240 + }

241 + ],

242 + "unmatched_articles": 24,

243 + "total_filtered_articles": 27

244 }

245 ```

246

171 -### Analyzer changes

247 +---

248

173 -The analyzer prompt gains a new context block:

249 +## Technical Implementation

250

251 +### Architecture: Plugin Pattern (Decision #7)

252 +

253 +This integration implements the `DataSource` protocol defined in Decision #7:

254 +

255 +```python

256 +# As shipped: scripts/techcrunch_crawler.py

257 +class NewsFeedSource:

258 + """RSS data source following the DataSource protocol."""

259 +

260 + def __init__(self, config: NewsSourceConfig) -> None:

261 + self.config = config

262 +

263 + def get_name(self) -> str:

264 + return self.config.name

265 +

266 + def get_rate_limits(self) -> dict:

267 + # Plain dict, not a RateLimits type.

268 + return {"requests_per_minute": self.config.requests_per_minute}

269 +

270 + def crawl(

271 + self,

272 + since: datetime,

273 + until: datetime,

274 + feed_url: str | None = None,

275 + ) -> list[dict]:

276 + """Fetch, filter, and extract entities from an RSS feed."""

277 + ...

278 ```

176 -## Press Context (TechCrunch, week of {date})

177 -{N} articles published relevant to tech/open-source.

178 -Notable coverage:

179 -- {title} ({category}) — mentions {github_links}

180 -- ...

181 -

182 -Cross-reference: For each trending repo, note if press coverage

183 -preceded the star surge. Label as "press-correlated" or "organic."

279 +

280 +`TechCrunchSource` is a thin subclass of `NewsFeedSource`; each source is

281 +described by a `NewsSourceConfig` (`name`, `feed_url`, `requests_per_minute`)

282 +loaded from `config/external_news_sources.json`.

283 +

284 +### File Layout

285 +

286 +```

287 +scripts/

288 + techcrunch_crawler.py # Multi-source RSS crawler (DataSource protocol)

289 + correlate.py # Cross-reference GitHub vs external news

290 +config/

291 + external_news_sources.json # Per-source feed_url + requests_per_minute

292 +data/

293 + raw/

294 + {week}-external-news.json # Merged crawl output (legacy: {week}-techcrunch.json)

295 + analyzed/

296 + {week}-correlations.json # Cross-reference results (output for Farnsworth)

297 ```

298

186 ----

299 +### Dependencies

300 +

301 +| Dependency | Purpose | Size Impact |

302 +|-----------|---------|-------------|

303 +| `feedparser` | RSS parsing | ~200 KB, pure Python |

304 +| `re` (stdlib) | Entity extraction patterns | None |

305 +| `datetime` (stdlib) | Window filtering | None |

306 +

307 +No additional API keys or authentication required. RSS is public.

308 +

309 +### Integration with Existing Crawl Workflow

310 +

311 +```yaml

312 +# In crawl-and-publish.yml (additions only)

313 +- name: Crawl external news RSS

314 + run: |

315 + python3 scripts/techcrunch_crawler.py \

316 + --sources config/external_news_sources.json \

317 + --output "data/raw/${WEEK}-external-news.json" \

318 + --since "$SINCE" \

319 + --until "$UNTIL"

320 +

321 +- name: Cross-reference correlations

322 + run: |

323 + python3 scripts/correlate.py \

324 + --raw "$WEEK_FILE" \

325 + --techcrunch "data/raw/${WEEK}-external-news.json" \

326 + --output "data/analyzed/${WEEK}-correlations.json"

327 +```

328 +

329 +### Error Handling

330

188 -## Phases

331 +| Failure Mode | Response | Impact |

332 +|-------------|----------|--------|

333 +| RSS feed unreachable | Retry 3×, then skip TechCrunch for this week | None — enrichment is optional |

334 +| RSS feed format changed | Log warning, skip parsing, open issue | None — graceful degradation |

335 +| Zero correlations found | Normal — output empty correlations file | Expected most weeks |

336 +| Malformed XML in feed | Skip malformed items, process rest | Partial data is fine |

337

190 -### Phase 1: RSS Crawl Plugin (1–2 weeks)

338 +---

339

192 -- Implement `TechCrunchSource` crawler plugin

193 -- Parse RSS feed, extract structured article data

194 -- Store as `data/raw/YYYY-WNN-techcrunch.json`

195 -- Filter to tech/open-source relevant articles only

196 -- Basic deduplication

197 -- **Output:** Weekly TechCrunch article JSON alongside GitHub JSON

340 +## Cost Estimate

341

199 -### Phase 2: Correlation Engine (2–3 weeks)

342 +### Compute Cost

343

201 -- Implement GitHub URL extraction from articles

202 -- Fuzzy entity matching (company name → GitHub org)

203 -- Temporal correlation (article date vs. star surge timing)

204 -- Add `press_correlated: bool` and `hype_risk: low|medium|high` to repo analysis

205 -- **Output:** Enriched analysis with cross-source annotations

344 +| Resource | Usage | Cost |

345 +|---------|-------|------|

346 +| RSS fetch | 1 HTTP request/week | $0.00 |

347 +| Python processing | ~5 seconds CPU | $0.00 (free Actions minutes) |

348 +| Correlation script | ~2 seconds CPU | $0.00 |

349 +| **Total infrastructure cost** | | **$0.00/week** |

350

207 -### Phase 3: Editorial Integration (1–2 weeks)

351 +### Token Cost (if Farnsworth uses correlations in analysis)

352

209 -- Update analyzer prompt to consume TechCrunch context

210 -- Add "Press vs. Reality" subsection to weekly summary

211 -- Surface disconnects in Gaps section

212 -- Update Hugo templates to render correlation badges

213 -- **Output:** Reader-facing cross-source insights on the published site

354 +|-----------|------|--------|-------------|

355 +| Correlations JSON (typical week, 2–5 hits) | ~2 KB | ~570 | +$0.002/week |

356 +| Correlations JSON (zero hits) | ~0.2 KB | ~57 | +$0.0002/week |

357 +| Correlations JSON (exceptional week, 10+ hits) | ~5 KB | ~1,400 | +$0.004/week |

358

215 -### Phase 4: Prediction Enhancement (future)

359 +**Annual token cost impact: $0.10–$0.21/year** (negligible relative to $16/year baseline).

360

217 -- Track whether press-correlated repos sustain momentum

218 -- Feed correlation accuracy back into prediction ledger

219 -- Calibrate hype risk scoring over time

220 -- **Output:** Improved prediction accuracy in topic channels

361 +### Development Cost

362 +

363 +| Task | Effort | Priority |

364 +|------|--------|----------|

365 +| `techcrunch.py` plugin | 2–3 hours | Medium |

366 +| `correlate.py` script | 2–3 hours | Medium |

367 +| Configuration + tests | 1–2 hours | Medium |

368 +| Workflow integration | 1 hour | Low |

369 +| **Total** | **6–9 hours** | |

370

371 ---

372

224 -## Cost & Resource Impact

373 +## Success Criteria

374

226 -| Resource | Impact |

227 -|----------|--------|

228 -| RSS fetch | Negligible (1 HTTP request/week, public feed, no auth) |

229 -| Storage | ~50-100 KB/week JSON (40 articles × metadata) |

230 -| Analyzer tokens | +500-800 tokens input context per run (~$0.002/week) |

231 -| API rate limits | Zero impact (RSS is not GitHub API) |

232 -| CI minutes | +5-10 seconds per run (RSS fetch + parse) |

233 -| Dependencies | `feedparser` (Python, MIT license, mature) |

375 +### Quantitative Metrics (measured after 8 weeks of operation)

376

235 -**Total incremental cost: <$0.01/week.** Trivial relative to base pipeline costs documented in PRD-cost-estimation.md.

377 +| Metric | Target | Measurement |

378 +|--------|--------|-------------|

379 +| RSS fetch success rate | ≥ 95% | Weeks with successful fetch / total weeks |

380 +| Filter reduction ratio | 85–95% reduction | (Raw articles - filtered) / raw articles |

381 +| Correlation hit rate | ≥ 5% of filtered articles | Articles with ≥1 GitHub match / filtered articles |

382 +| False positive rate | ≤ 2% | Matches marked incorrect in manual review / total matches |

383 +| Zero-noise weeks | 100% | Weeks where zero-correlation produces zero output noise |

384 +| Enrichment value (subjective) | ≥ 3/5 quality rating | Monthly review: "Did correlations improve the digest?" |

385

237 ----

386 +### Qualitative Success Indicators

387 +

388 +- At least 1 "hype vs reality" insight per month that wouldn't exist without this signal

389 +- Zero instances where TechCrunch noise degrades the digest quality

390 +- The feature is invisible when it has nothing useful to contribute

391

239 -## Risks & Mitigations

392 +### Failure Criteria (triggers feature removal)

393

242 -|------|------------|--------|------------|

247 -| Content extraction blocked | Low | Low | Use RSS summary only, don't scrape full articles |

394 +- Hit rate below 2% after 8 weeks → feature adds complexity without value

395 +- False positives above 10% → feature introduces noise

396 +- RSS feed breaks and stays broken for 4+ consecutive weeks → dependency unreliable

397 +- Farnsworth (analyst) consistently ignores correlation data in analysis → no downstream value

398

399 ---

400

251 -## Open Questions

401 +## Phased Rollout

402

253 -1. **OQ1: Should we also extract from TechCrunch's category-specific feeds?**

254 - - `techcrunch.com/category/artificial-intelligence/feed/` for topic-channel alignment

255 - - Pro: Better relevance filtering. Con: More feeds to manage.

403 +### Phase 1: RSS Collection Only (Week 1–2)

404

257 -2. **OQ2: Full article fetch vs. RSS excerpt only?**

258 - - RSS includes ~200 word excerpt. Full article requires HTTP fetch + HTML parsing.

259 - - Recommendation: Start with RSS excerpt only. Avoids scraping concerns and ToS issues.

405 +- Implement `scripts/techcrunch_crawler.py` with 3-stage filtering

406 +- Output `data/raw/{week}-external-news.json` (legacy `data/raw/{week}-techcrunch.json`)

407 +- No integration with analysis — just collect and validate filter quality

408 +- **Exit criteria:** Filter reduces volume by ≥ 80%, entity extraction produces meaningful tags

409

261 -3. **OQ3: Should correlation annotations be visible to readers or analyst-only?**

262 - - Option A: Show "📰 Press-correlated" badge on repo entries

263 - - Option B: Keep as internal signal that shapes editorial tone only

264 - - Recommendation: Option A for transparency (readers deserve to know *why* something is trending)

410 +### Phase 2: Correlation Script (Week 3–4)

411

266 -4. **OQ4: Add HackerNews as a second correlation source simultaneously?**

267 - - HN has an API, overlaps with TechCrunch coverage, and better represents developer sentiment

268 - - Recommendation: TechCrunch first (simpler, RSS), HN second (API, different signal)

412 +- Implement `correlate.py` cross-reference logic

413 +- Output `data/analyzed/{week}-correlations.json`

414 +- Manual review of correlation quality for 2 weeks

415 +- **Exit criteria:** Hit rate ≥ 3%, false positive rate ≤ 5%

416

270 -5. **OQ5: How to handle TechCrunch articles about closed-source products?**

271 - - Many TC articles cover proprietary SaaS with no GitHub presence

272 - - Recommendation: Filter to articles containing GitHub links OR open-source keywords only

417 +### Phase 3: Analysis Integration (Week 5–6)

418

274 ----

419 +- Farnsworth consumes `data/analyzed/{week}-correlations.json` in analysis prompt

420 +- Correlation data appears in weekly digest when relevant

421 +- **Exit criteria:** At least 1 correlation adds editorial value in 2 of 4 weeks

422

276 -## Success Criteria

423 +### Phase 4: Steady State (Week 7+)

424

278 -| Metric | Target | Measurement |

279 -|--------|--------|-------------|

280 -| Articles crawled per week | 15-40 relevant | Count in weekly JSON |

281 -| Correlation hit rate | >30% of trending repos have press match | Cross-reference accuracy |

282 -| Hype detection accuracy | >70% of "high hype risk" repos show star decay at week +2 | Retrospective validation |

283 -| Reader value signal | Qualitative improvement in Gaps section depth | Editorial review |

284 -| Zero pipeline failures from RSS source | 100% graceful degradation | CI logs |

425 +- Monitor success metrics

426 +- Tune keyword lists and category filters based on actual hit rates

427 +- Consider adding second source (HN) if TechCrunch proves the plugin model

428

429 ---

430

288 -## Relationship to Existing PRDs

431 +## Open Questions

432

290 -- **PRD-topic-channels.md:** TechCrunch correlation enriches per-topic analysis. AI-focused TC articles correlate with `ai-ml` topic channel repos.

291 -- **PRD-cost-estimation.md:** Incremental cost is negligible (<$0.01/week). No tier change needed.

292 -- **decisions.md Decision #7:** This is the first concrete implementation of the crawler plugin architecture.

293 -- **decisions.md MCP Tools:** TechCrunch RSS fetch can be an MCP tool, registered in allowlist per Decision 5.

434 +|---|----------|--------|---------------------|

435 +| OQ1 | Does TechCrunch's RSS feed include full article text or just excerpts? | Medium — affects entity extraction quality | Spike: inspect actual feed content. If excerpts only, extraction limited to title + summary. |

440 +| OQ6 | Can we use GitHub's topic taxonomy to improve matching? | Medium — could boost hit rate | Investigate `GET /repos/{owner}/{repo}/topics` coverage during Phase 2. |

441

442 ---

443

297 -## Editorial Philosophy Note

444 +## Relationship to Other PRDs

445 +

446 +- **`docs/processed/PRD-topic-channels.md`:** Topic channels define per-domain crawling. TechCrunch integration is orthogonal — it enriches ANY topic channel with press signal. A `rust` channel could correlate TechCrunch Rust articles with Rust repo trends.

447 +- **PRD-cost-estimation.md:** TechCrunch adds negligible cost ($0.10–$0.21/year in tokens). No budget concern.

448 +- **Crawler plugin architecture:** TechCrunch is the first non-GitHub `DataSource` plugin, validating the extensible crawler design.

449

299 -TechCrunch integration does NOT mean SquadScope becomes a TechCrunch aggregator. The feed is a **correlation signal**, not content to republish. SquadScope's voice remains: "Here's what's actually happening on GitHub this week, and here's what the press says is happening. Notice the gap? That's where the real story is."

450 +---

451

301 -The editorial value is in the *delta* between press narrative and developer activity — not in summarizing TechCrunch articles.

452 +*Archived as-is. This PRD reflects the original plan; the integration shipped on 2026-06-10. For canonical paths, field names, and interfaces, see the code (`scripts/techcrunch_crawler.py`, `config/external_news_sources.json`, `.github/workflows/crawl-and-publish.yml`).*

docs/processed/podcast-generation-plan.md renamed