38e6a74 · cryptotaxi247/SquadScope

@@ -0,0 +1,443 @@

1 +# PRD: TechCrunch RSS Integration for Cross-Signal Enrichment

2 +

3 +**Author:** Farnsworth (Analyst), revised by Bender (Crawler)

4 +**Date:** 2026-05-19

5 +**Status:** Draft

6 +**Type:** Feature PRD (Enrichment Signal)

7 +**Depends on:** .squad/decisions.md (Decision #7 — Crawler Plugin Architecture), docs/PRD-topic-channels.md

8 +

9 +---

10 +

11 +## Executive Summary

12 +

13 +SquadScope tracks GitHub repository trends weekly. This PRD proposes adding TechCrunch RSS as a **supplementary enrichment signal** — not a primary data source — to detect the *delta* between press hype and actual GitHub traction. When TechCrunch covers a technology or project that also shows unusual GitHub star activity, that correlation is newsworthy. When press coverage does NOT correlate with GitHub activity, that absence is equally informative.

14 +

15 +**Key constraint acknowledged upfront:** The correlation hit rate between TechCrunch articles and specific GitHub repositories is estimated at only **5–15%**. This feature is designed as a low-cost enrichment layer that adds value when correlations exist, and degrades gracefully (adds zero noise) when they don't.

16 +

17 +---

18 +

19 +## Problem Statement

20 +

21 +### The Gap: Press Hype vs. Real Adoption

22 +

23 +1. **GitHub stars measure developer interest.** A repo gaining 500 stars in a week signals genuine traction from people who build things.

24 +

25 +2. **TechCrunch coverage measures press/VC interest.** An article about a startup or technology signals attention from the funding and media ecosystem.

26 +

27 +3. **The delta between these signals is the insight.** Three scenarios produce editorial value:

28 + - **Hype confirmed:** TechCrunch covers X, and X's GitHub repos are surging → "Real momentum, developers agree"

29 + - **Hype without substance:** TechCrunch covers Y, but Y has zero or declining GitHub activity → "Marketing over engineering"

30 + - **Quiet breakout:** No press coverage, but a repo is exploding on GitHub → "Under the radar"

31 +

32 +4. **Currently, SquadScope only sees scenario 3.** Adding press signal enables detecting scenarios 1 and 2, making the weekly digest more insightful.

33 +

34 +### Why TechCrunch Specifically

35 +

36 +- TechCrunch has a well-maintained RSS feed (`http://techcrunch.com/feed`) with full article metadata

37 +- It covers the startup/tech ecosystem most likely to overlap with open-source GitHub activity

38 +- RSS is free, requires no API key, and is stable

39 +- Other sources (HN, Reddit) can follow the same plugin pattern later

40 +

41 +---

42 +

43 +## Value Proposition: The Delta Model

44 +

45 +The value of this integration is **NOT** in summarizing TechCrunch articles (readers can read TechCrunch themselves). The value is in the **cross-reference delta**:

46 +

47 +```

48 +Value = f(TechCrunch_coverage, GitHub_activity) where:

49 + - Both high → "Confirmed trend" (correlation)

50 + - TC high, GH low → "Hype alert" (anti-correlation)

51 + - TC low, GH high → "Sleeper hit" (absence signal)

52 + - Both low → No signal (filtered out)

53 +```

54 +

55 +This positions SquadScope as providing analysis that neither TechCrunch nor GitHub alone can offer.

56 +

57 +---

58 +

59 +## Honest Assessment: Correlation Rates

60 +

61 +### Expected Hit Rates

62 +

63 +| Correlation Type | Estimated Rate | Reasoning |

64 +|-----------------|---------------|-----------|

65 +| Direct match (TC mentions a specific repo) | 2–5% | Few TC articles name exact repos |

66 +| Indirect match (TC covers technology X, repo uses topic X) | 10–15% | Broader topic matching catches more |

67 +| No correlation found | 80–93% | Most TC articles have no GitHub signal |

68 +

69 +### Why Low Rates Are Acceptable

70 +

71 +1. **Low false-positive cost:** Uncorrelated articles are simply ignored — they add zero noise to the output.

72 +2. **High value per hit:** When a correlation IS found, it's genuinely interesting editorial content.

73 +3. **Asymmetric payoff:** Even 2–3 notable correlations per week would meaningfully enrich a weekly digest.

74 +4. **Trend over time:** Cross-referencing accumulated over weeks reveals patterns invisible in any single week.

75 +

76 +### What This Feature Is NOT

77 +

78 +- NOT a TechCrunch summarizer

79 +- NOT a primary data source for SquadScope

80 +- NOT expected to produce signal every week

81 +- NOT a replacement for GitHub-native trend detection

82 +

83 +---

84 +

85 +## Filtering Strategy

86 +

87 +### The Problem: Volume

88 +

89 +TechCrunch publishes **30–50 articles per day** (210–350 per week). Without aggressive filtering, this overwhelms the pipeline with noise. The crawler must reduce this to a manageable set before any correlation attempt.

90 +

91 +### Three-Stage Filtering Pipeline

92 +

93 +```

94 +Stage 1: Category Filter (RSS metadata)

95 + Input: ~250 articles/week (full RSS feed)

96 + Filter: Keep only categories relevant to developer tools/open-source

97 + Output: ~60-80 articles/week (70% reduction)

98 + Method: Allowlist of RSS <category> tags

99 +

100 +Stage 2: Keyword Filter (title + description)

101 + Input: ~60-80 articles/week

102 + Filter: Must contain technology/developer keywords

103 + Output: ~20-30 articles/week (60% reduction)

104 + Method: Keyword scoring (open-source, GitHub, developer, API, SDK, framework, etc.)

105 +

106 +Stage 3: Entity Extraction (lightweight)

107 + Input: ~20-30 articles/week

108 + Filter: Extract mentioned technologies, companies, project names

109 + Output: ~20-30 enriched article records with entity tags

110 + Method: Regex patterns + known project name dictionary

111 +```

112 +

113 +### Category Allowlist (Initial)

114 +

115 +```yaml

116 +allowed_categories:

117 + - Apps

118 + - Artificial Intelligence

119 + - Cloud

120 + - Developer

121 + - Enterprise

122 + - Hardware

123 + - Open Source

124 + - Robotics

125 + - Security

126 + - Startups

127 +

128 +blocked_categories:

129 + - Media & Entertainment

130 + - Transportation

131 + - Government & Policy

132 + - Crypto # Too noisy, low GitHub correlation

133 +```

134 +

135 +### Keyword Scoring

136 +

137 +Each article gets a relevance score (0–10) based on title + description:

138 +

139 +| Keyword Group | Weight | Examples |

140 +|--------------|--------|----------|

141 +| Direct GitHub mentions | +5 | "GitHub", "open source", "repository" |

142 +| Developer tools | +3 | "API", "SDK", "framework", "library", "CLI" |

143 +| Technology names | +2 | "Python", "Rust", "Kubernetes", "LLM" |

144 +| Funding/startup | +1 | "raises", "Series A", "launch" |

145 +

146 +**Threshold:** Articles scoring ≥ 3 proceed to entity extraction. Expected pass rate: ~40% of category-filtered articles.

147 +

148 +---

149 +

150 +## Temporal Alignment

151 +

152 +### The Problem: RSS is Real-Time, SquadScope is Weekly

153 +

154 +TechCrunch publishes continuously. SquadScope runs weekly (Monday 08:00 UTC). This creates a timing mismatch:

155 +

156 +- An article published Tuesday about Project X won't be seen until the following Monday

157 +- By then, the GitHub star surge may have already peaked and fallen

158 +

159 +### Solution: Weekly Batch with 7-Day Window

160 +

161 +```

162 +┌─────────────────────────────────────────────────────┐

163 +│ Monday 08:00 UTC: Crawl job runs │

164 +│ │

165 +│ 1. Fetch all RSS items from past 7 days │

166 +│ 2. Filter (3-stage pipeline above) │

167 +│ 3. Extract entities from filtered articles │

168 +│ 4. Cross-reference entities against weekly │

169 +│ GitHub trending repos (already collected) │

170 +│ 5. Output correlation data for Farnsworth │

171 +└─────────────────────────────────────────────────────┘

172 +```

173 +

174 +### Why Weekly Batch Is Sufficient

175 +

176 +1. **SquadScope is a weekly digest.** Real-time alerting is out of scope.

177 +2. **7-day accumulation helps.** A trend covered across multiple articles in a week is stronger signal.

178 +3. **GitHub stars data is also weekly.** Both signals align on the same time window.

179 +4. **Simplicity:** No state management, no incremental polling, no deduplication across runs.

180 +

181 +### Freshness Guarantee

182 +

183 +- RSS feed items older than 7 days are discarded

184 +- If the RSS feed doesn't contain 7 days of history (TechCrunch's feed typically holds 20–30 items), supplement with the feed's full available content

185 +- Each item's `<pubDate>` is checked against the collection window

186 +

187 +---

188 +

189 +## Correlation Approach

190 +

191 +### Entity-to-Repository Matching

192 +

193 +```

194 +TechCrunch Entity → GitHub Signal

195 +─────────────────────────────────────────────────

196 +"Anthropic" (company) → repos with topic:anthropic or org:anthropic

197 +"LangChain" (project) → repo langchain-ai/langchain stars_gained

198 +"Rust 2024 edition" (tech) → repos with topic:rust AND stars_gained > threshold

199 +"Series B: Acme Corp" → repos owned by acme-corp org

200 +```

201 +

202 +### Matching Strategies (in priority order)

203 +

204 +1. **Exact name match:** Article mentions "LangChain" → search for repos named `langchain*`

205 +2. **Organization match:** Article mentions company → search GitHub org

206 +3. **Topic match:** Article discusses technology → match against repo topics

207 +4. **Description match:** Fuzzy match article entities against repo descriptions

208 +

209 +### Scoring Correlation Strength

210 +

211 +| Match Type | Confidence | Example |

212 +|-----------|-----------|---------|

213 +| Exact repo name in article | 0.9 | "...announced on their GitHub repo langchain-ai/langchain..." |

214 +| Organization name + topic overlap | 0.7 | Article about Anthropic + repo topics include "claude" |

215 +| Technology keyword + trending | 0.5 | Article about "Rust" + Rust repo trending |

216 +| Company name only (no GitHub signal) | 0.3 | Article about startup with no public repos |

217 +

218 +### Output Format

219 +

220 +```json

221 +{

222 + "week": "2026-W21",

223 + "correlations": [

224 + {

225 + "article_title": "LangChain raises $25M Series A",

226 + "article_url": "https://techcrunch.com/...",

227 + "article_date": "2026-05-15",

228 + "matched_repos": ["langchain-ai/langchain"],

229 + "match_type": "exact_name",

230 + "confidence": 0.9,

231 + "github_signal": {

232 + "stars_gained": 847,

233 + "percentile": 98

234 + },

235 + "delta_type": "confirmed_trend"

236 + }

237 + ],

238 + "unmatched_articles": 24,

239 + "total_filtered_articles": 27

240 +}

241 +```

242 +

243 +---

244 +

245 +## Technical Implementation

246 +

247 +### Architecture: Plugin Pattern (Decision #7)

248 +

249 +This integration implements the `DataSource` protocol defined in Decision #7:

250 +

251 +```python

252 +class TechCrunchSource:

253 + """TechCrunch RSS crawler plugin for SquadScope."""

254 +

255 + def get_name(self) -> str:

256 + return "techcrunch"

257 +

258 + def get_rate_limits(self) -> RateLimits:

259 + return RateLimits(

260 + requests_per_minute=10, # Polite RSS polling

261 + retry_after_seconds=60

262 + )

263 +

264 + def crawl(self, window_start: datetime, window_end: datetime) -> CrawlResult:

265 + """Fetch, filter, and extract entities from TechCrunch RSS."""

266 + raw_items = self._fetch_rss(window_start, window_end)

267 + filtered = self._apply_filters(raw_items)

268 + enriched = self._extract_entities(filtered)

269 + return CrawlResult(source="techcrunch", items=enriched)

270 +```

271 +

272 +### File Layout

273 +

274 +```

275 +scripts/

276 + sources/

277 + __init__.py

278 + base.py # DataSource protocol

279 + techcrunch.py # TechCrunch RSS plugin

280 + config/

281 + techcrunch.yml # Category allowlist, keywords, thresholds

282 +data/

283 + enrichment/

284 + techcrunch/

285 + 2026-W21.json # Weekly filtered articles + entities

286 + correlations/

287 + 2026-W21.json # Cross-reference results (output for Farnsworth)

288 +```

289 +

290 +### Dependencies

291 +

292 +| Dependency | Purpose | Size Impact |

293 +|-----------|---------|-------------|

294 +| `feedparser` | RSS parsing | ~200 KB, pure Python |

295 +| `re` (stdlib) | Entity extraction patterns | None |

296 +| `datetime` (stdlib) | Window filtering | None |

297 +

298 +No additional API keys or authentication required. RSS is public.

299 +

300 +### Integration with Existing Crawl Workflow

301 +

302 +```yaml

303 +# In crawl-and-publish.yml (additions only)

304 +- name: Crawl TechCrunch RSS

305 + run: |

306 + python3 scripts/sources/techcrunch.py \

307 + --window-days 7 \

308 + --output data/enrichment/techcrunch/$WEEK.json

309 + env:

310 + WEEK: ${{ env.WEEK }}

311 +

312 +- name: Cross-reference correlations

313 + run: |

314 + python3 scripts/correlate.py \

315 + --github-data data/raw/$WEEK.json \

316 + --techcrunch-data data/enrichment/techcrunch/$WEEK.json \

317 + --output data/correlations/$WEEK.json

318 +```

319 +

320 +### Error Handling

321 +

322 +| Failure Mode | Response | Impact |

323 +|-------------|----------|--------|

324 +| RSS feed unreachable | Retry 3×, then skip TechCrunch for this week | None — enrichment is optional |

325 +| RSS feed format changed | Log warning, skip parsing, open issue | None — graceful degradation |

326 +| Zero correlations found | Normal — output empty correlations file | Expected most weeks |

327 +| Malformed XML in feed | Skip malformed items, process rest | Partial data is fine |

328 +

329 +---

330 +

331 +## Cost Estimate

332 +

333 +### Compute Cost

334 +

335 +| Resource | Usage | Cost |

336 +|---------|-------|------|

337 +| RSS fetch | 1 HTTP request/week | $0.00 |

338 +| Python processing | ~5 seconds CPU | $0.00 (free Actions minutes) |

339 +| Correlation script | ~2 seconds CPU | $0.00 |

340 +| **Total infrastructure cost** | | **$0.00/week** |

341 +

342 +### Token Cost (if Farnsworth uses correlations in analysis)

343 +

345 +|-----------|------|--------|-------------|

346 +| Correlations JSON (typical week, 2–5 hits) | ~2 KB | ~570 | +$0.002/week |

347 +| Correlations JSON (zero hits) | ~0.2 KB | ~57 | +$0.0002/week |

348 +| Correlations JSON (exceptional week, 10+ hits) | ~5 KB | ~1,400 | +$0.004/week |

349 +

350 +**Annual token cost impact: $0.10–$0.21/year** (negligible relative to $16/year baseline).

351 +

352 +### Development Cost

353 +

354 +| Task | Effort | Priority |

355 +|------|--------|----------|

356 +| `techcrunch.py` plugin | 2–3 hours | Medium |

357 +| `correlate.py` script | 2–3 hours | Medium |

358 +| Configuration + tests | 1–2 hours | Medium |

359 +| Workflow integration | 1 hour | Low |

360 +| **Total** | **6–9 hours** | |

361 +

362 +---

363 +

364 +## Success Criteria

365 +

366 +### Quantitative Metrics (measured after 8 weeks of operation)

367 +

368 +| Metric | Target | Measurement |

369 +|--------|--------|-------------|

370 +| RSS fetch success rate | ≥ 95% | Weeks with successful fetch / total weeks |

371 +| Filter reduction ratio | 85–95% reduction | (Raw articles - filtered) / raw articles |

372 +| Correlation hit rate | ≥ 5% of filtered articles | Articles with ≥1 GitHub match / filtered articles |

373 +| False positive rate | ≤ 2% | Matches marked incorrect in manual review / total matches |

374 +| Zero-noise weeks | 100% | Weeks where zero-correlation produces zero output noise |

375 +| Enrichment value (subjective) | ≥ 3/5 quality rating | Monthly review: "Did correlations improve the digest?" |

376 +

377 +### Qualitative Success Indicators

378 +

379 +- At least 1 "hype vs reality" insight per month that wouldn't exist without this signal

380 +- Zero instances where TechCrunch noise degrades the digest quality

381 +- The feature is invisible when it has nothing useful to contribute

382 +

383 +### Failure Criteria (triggers feature removal)

384 +

385 +- Hit rate below 2% after 8 weeks → feature adds complexity without value

386 +- False positives above 10% → feature introduces noise

387 +- RSS feed breaks and stays broken for 4+ consecutive weeks → dependency unreliable

388 +- Farnsworth (analyst) consistently ignores correlation data in analysis → no downstream value

389 +

390 +---

391 +

392 +## Phased Rollout

393 +

394 +### Phase 1: RSS Collection Only (Week 1–2)

395 +

396 +- Implement `techcrunch.py` with 3-stage filtering

397 +- Output `data/enrichment/techcrunch/{week}.json`

398 +- No integration with analysis — just collect and validate filter quality

399 +- **Exit criteria:** Filter reduces volume by ≥ 80%, entity extraction produces meaningful tags

400 +

401 +### Phase 2: Correlation Script (Week 3–4)

402 +

403 +- Implement `correlate.py` cross-reference logic

404 +- Output `data/correlations/{week}.json`

405 +- Manual review of correlation quality for 2 weeks

406 +- **Exit criteria:** Hit rate ≥ 3%, false positive rate ≤ 5%

407 +

408 +### Phase 3: Analysis Integration (Week 5–6)

409 +

410 +- Farnsworth consumes `data/correlations/{week}.json` in analysis prompt

411 +- Correlation data appears in weekly digest when relevant

412 +- **Exit criteria:** At least 1 correlation adds editorial value in 2 of 4 weeks

413 +

414 +### Phase 4: Steady State (Week 7+)

415 +

416 +- Monitor success metrics

417 +- Tune keyword lists and category filters based on actual hit rates

418 +- Consider adding second source (HN) if TechCrunch proves the plugin model

419 +

420 +---

421 +

422 +## Open Questions

423 +

425 +|---|----------|--------|---------------------|

426 +| OQ1 | Does TechCrunch's RSS feed include full article text or just excerpts? | Medium — affects entity extraction quality | Spike: inspect actual feed content. If excerpts only, extraction limited to title + summary. |

431 +| OQ6 | Can we use GitHub's topic taxonomy to improve matching? | Medium — could boost hit rate | Investigate `GET /repos/{owner}/{repo}/topics` coverage during Phase 2. |

432 +

433 +---

434 +

435 +## Relationship to Other PRDs

436 +

437 +- **PRD-topic-channels.md:** Topic channels define per-domain crawling. TechCrunch integration is orthogonal — it enriches ANY topic channel with press signal. A `rust` channel could correlate TechCrunch Rust articles with Rust repo trends.

438 +- **PRD-cost-estimation.md:** TechCrunch adds negligible cost ($0.10–$0.21/year in tokens). No budget concern.

439 +- **Decision #7 (Plugin Architecture):** TechCrunch is the first non-GitHub `DataSource` plugin, validating the extensible crawler design.

440 +

441 +---

442 +

443 +*This PRD will be updated with actual hit rates and filter performance after Phase 1 completes (target: 2 weeks post-implementation).*

docs: TechCrunch RSS integration PRD (#55)