@cryptotaxi247 / SquadScope / commits / 87e55a2

feat: add parallel external news feeds

Closes #234

Juan Manuel Servera committed Jun 5, 2026 at 18:13 UTC 87e55a227da78b86e9677acc96460968196e9e5a

15 files changed +446 -36

.github/workflows/crawl-and-publish.yml

+9 -4

@@ -104,13 +104,14 @@ jobs:

104 - name: Install Python dependencies

105 run: pip install -r requirements.txt

106

107 - - name: Crawl TechCrunch RSS

107 + - name: Crawl external news RSS feeds

108 if: ${{ !inputs.rebuild_week }}

109 run: |

110 WEEK=$(date +%Y-W%V)

111 SINCE=$(date -d '7 days ago' +%Y-%m-%d)

112 python scripts/techcrunch_crawler.py \

113 - --output "data/raw/${WEEK}-techcrunch.json" \

113 + --sources config/external_news_sources.json \

114 + --output "data/raw/${WEEK}-external-news.json" \

115 --since "$SINCE"

116

117 - name: Upload raw crawl artifact

@@ -212,6 +213,7 @@ jobs:

213 git fetch origin publish

214 mkdir -p data/raw data/analyzed data/snapshots

215 git checkout origin/publish -- "data/raw/${REBUILD_WEEK}.json"

216 + git checkout origin/publish -- "data/raw/${REBUILD_WEEK}-external-news.json" 2>/dev/null || true

217 git checkout origin/publish -- "data/raw/${REBUILD_WEEK}-techcrunch.json" 2>/dev/null || true

218 git checkout origin/publish -- "data/snapshots/${REBUILD_WEEK}-stars.json" 2>/dev/null || true

219 # Also hydrate ALL prior analyzed files so rollups have correct historical context

@@ -276,9 +278,12 @@ jobs:

278 set -euo pipefail

279 WEEK="${{ steps.analysis-context.outputs.week }}"

280 WEEK_FILE="${{ steps.analysis-context.outputs.week_file }}"

279 - TC_FILE="data/raw/${WEEK}-techcrunch.json"

281 + TC_FILE="data/raw/${WEEK}-external-news.json"

282 + if [ ! -f "$TC_FILE" ]; then

283 + TC_FILE="data/raw/${WEEK}-techcrunch.json"

284 + fi

285

281 - # Run correlate if TechCrunch data exists

286 + # Run correlate if external news data exists

287 if [ -f "$TC_FILE" ]; then

288 mkdir -p data/analyzed

289 python scripts/correlate.py \

.squad/agents/bender/history.md

@@ -15,3 +15,5 @@

15 - GitHub Actions schedule events do not have an `inputs` object; use `!inputs.X` instead of `github.event.inputs.X == ''` to safely check optional manual inputs without breaking cron triggers (critical fix, PR #164).

16 - Deploy pipeline should hydrate previous-week content/data from publish branch before Hugo build to prevent main/publish divergence and preserve existing data integrity (PR #164, W21 rescue architectural fix).

17 - Fork-safe deploy secrets should default to empty in Hugo config and be injected via `HUGO_PARAMS_*` environment overrides so forks render safe defaults without inherited maintainer secrets (GA4 PR #182/#191).

18 +- External RSS crawlers must validate config URLs against an HTTPS host allowlist and fetch through explicit per-request timeouts; config-driven source lists are not a security boundary by themselves (PR #236).

19 +- Local validation docs for scripts importing `scripts.*` modules should use `python3 -m ...` or set `PYTHONPATH=.` so repo-root imports resolve reliably (PR #236).

.squad/agents/leela/history.md

@@ -5,6 +5,7 @@

5 - Keeps interface contracts stable enough for specialists to work independently.

7 ## Learnings

8 +- 2026-06-05T15:36:19.379+00:00 issue #234: for small RSS enrichment sets in the weekly crawl, prefer a config-driven source list plus bounded in-process parallel fetches over separate Actions jobs; runner setup/artifact overhead outweighs per-feed job parallelism.

9 - Branch protection must never be bypassed; automated write paths should follow the shared `branch-protection-pr-workflow` skill instead.

10 - The learning loop only works when agent identity is loaded before execution, outcomes are persisted after execution, and that state is injected into the next run.

11 - Copilot CLI agent selection uses the registered agent name, not the path to the agent file.

.squad/decisions/inbox/bender-pr-236-security-fix.md new

+10

@@ -0,0 +1,10 @@

1 +# Bender PR #236 Security Fix

2 +

3 +## Context

4 +Hermes blocked PR #236 because config-driven external RSS sources were fetched directly without egress URL validation or explicit per-request timeouts.

5 +

6 +## Decision

7 +External news RSS source configs now require HTTPS URLs whose host is in the approved feed allowlist, with credentials, local/private/link-local targets, and unexpected ports rejected before crawl. Fetching now goes through `urllib.request.urlopen` with an explicit bounded timeout before handing bytes to `feedparser`, while retaining the existing config-driven source list and bounded in-process worker pool.

8 +

9 +## Validation

10 +Added tests for invalid/unapproved URL rejection and explicit fetch timeout propagation. Ran `PYTHONPATH=. .venv/bin/python -m pytest tests -q` with 563 passing tests.

.squad/decisions/inbox/leela-issue-234-sources.md new

+18

@@ -0,0 +1,18 @@

1 +# Leela — Issue 234 external news source architecture

2 +

3 +Date: 2026-06-05T15:36:19.379+00:00

4 +Issue: #234

5 +

6 +## Decision

7 +

8 +Keep external news crawling in the existing crawl job and make the RSS source list config-driven via `config/external_news_sources.json`. Fetch the configured feeds concurrently inside `scripts/techcrunch_crawler.py` using a bounded thread pool, and write one weekly enrichment artifact: `data/raw/YYYY-WNN-external-news.json`.

9 +

10 +## Rubberduck tradeoff

11 +

12 +Separate GitHub Actions jobs would parallelize at the runner level, but every source would repeat checkout, Python setup, dependency install, artifact upload/download, and failure-handling boilerplate. For five RSS feeds, that overhead is larger than the network wait we are optimizing away, and it would fragment a single enrichment contract across multiple artifacts.

13 +

14 +In-process threading matches the current architecture better: RSS fetching is I/O-bound, feedparser work is light, and the existing crawl job already owns raw data artifact handoff. A bounded pool preserves Actions compute, keeps one failure surface, and lets future sources be added by config without editing workflow topology.

15 +

16 +## Scope boundary

17 +

18 +This is a small architectural refactor around an existing RSS crawler, so Leela implemented directly rather than reassigning to Bender. Deeper crawler work, such as source-specific parsing, feed health dashboards, or correlation logic, should remain Bender-owned.

.squad/skills/ci-data-source-integration-pattern/SKILL.md

@@ -195,6 +195,10 @@ def main(argv: list[str] | None = None) -> int:

195 return 0

196 ```

197

198 +### Config-Driven Parallel RSS Sources

199 +

200 +For small sets of external RSS feeds in the weekly Actions pipeline, prefer one config file plus bounded in-process parallel fetches over one job per feed. This avoids repeated checkout/setup/artifact overhead, keeps a single enrichment artifact contract, and lets maintainers add or remove sources without editing workflow topology.

201 +

202 ## Notes

203

204 - Standardize output schemas across all data sources for seamless pipeline integration

README.md

+6 -4

@@ -26,10 +26,11 @@ Crawl → Analyze → Generate → Deploy → Reskill

26 JSON Markdown Hugo Pages Improvements

27 ```

29 -**Stage 1: Crawl** (`scripts/crawl.py`)

29 +**Stage 1: Crawl** (`scripts/crawl.py`, `scripts/techcrunch_crawler.py`)

30 - Queries GitHub API for repos created/trending in the current week

31 +- Fetches configured external RSS feeds from `config/external_news_sources.json` in parallel as an enrichment signal

32 - Applies heuristic filtering (language, topic, description quality)

32 -- Outputs: `data/raw/YYYY-WNN.json`, `data/snapshots/YYYY-WNN-stars.json`

33 +- Outputs: `data/raw/YYYY-WNN.json`, `data/raw/YYYY-WNN-external-news.json`, `data/snapshots/YYYY-WNN-stars.json`

35 **Stage 2: Analyze** (Copilot CLI or fallback)

36 - Reads raw JSON; applies AI analysis to classify repos as signal/noise/gaps

@@ -93,7 +94,8 @@ JSON Markdown Hugo Pages Improvements

94 - `content/weekly/YYYY/WNN.md` — immutable weekly summaries (published once, never modified)

95 - `content/monthly/YYYY/MM.md` — monthly rollups (append-only)

96 - `content/yearly/YYYY.md` — yearly summaries (append-only)

96 -- `data/raw/YYYY-WNN.json` — crawler output (JSON object with keys: `week`, `new_repos`, `trending_repos`, `signals`, `metadata`)

97 +- `data/raw/YYYY-WNN.json` — GitHub crawler output (JSON object with keys: `week`, `new_repos`, `trending_repos`, `signals`, `metadata`)

98 +- `data/raw/YYYY-WNN-external-news.json` — external RSS enrichment output from sources configured in `config/external_news_sources.json`

99 - `data/analyzed/YYYY-WNN-summary.md` — AI analysis with quality score

100 - `data/snapshots/YYYY-WNN-stars.json` — star count snapshots for trending analysis

101

@@ -101,7 +103,7 @@ JSON Markdown Hugo Pages Improvements

103

104 `.github/workflows/crawl-and-publish.yml` runs the full weekly automation every Monday at 08:00 UTC:

105

104 -1. **Crawl:** GitHub API → `data/raw/YYYY-WNN.json`

106 +1. **Crawl:** GitHub API → `data/raw/YYYY-WNN.json`; external RSS feeds → `data/raw/YYYY-WNN-external-news.json`

107 2. **Analyze:** Copilot → `data/analyzed/YYYY-WNN-summary.md`

108 3. **Quality gate:** Validates quality_score ≥ 60; blocks publish if failed

109 4. **Generate:** Markdown → `content/weekly/YYYY/WNN.md`

config/external_news_sources.json new

+27

@@ -0,0 +1,27 @@

1 +[

2 + {

3 + "name": "techcrunch",

4 + "feed_url": "https://techcrunch.com/feed/",

5 + "requests_per_minute": 10

6 + },

7 + {

8 + "name": "nvidia_blog",

9 + "feed_url": "https://blogs.nvidia.com/feed/",

10 + "requests_per_minute": 10

11 + },

12 + {

13 + "name": "hugging_face_blog",

14 + "feed_url": "https://huggingface.co/blog/feed.xml",

15 + "requests_per_minute": 10

16 + },

17 + {

18 + "name": "mit_technology_review",

19 + "feed_url": "https://www.technologyreview.com/feed",

20 + "requests_per_minute": 10

21 + },

22 + {

23 + "name": "github_blog",

24 + "feed_url": "https://github.blog/feed/",

25 + "requests_per_minute": 10

26 + }

27 +]

docs/pipeline-validation.md

@@ -28,6 +28,7 @@ Required secrets/tokens:

29 **Outputs**

30 - `data/raw/YYYY-WNN.json`

31 +- `data/raw/YYYY-WNN-external-news.json`

32 - `data/snapshots/YYYY-WNN-stars.json`

33 - `raw-data` artifact

34 - `crawl-snapshots` artifact

@@ -36,6 +37,7 @@ Required secrets/tokens:

38 **Success criteria**

39 - Raw payload passes `scripts.crawl.validate_payload()`

40 +- External RSS payload is written for the same ISO week from `config/external_news_sources.json`

41 - Snapshot file is written for the same ISO week

42 - Cache artifact uploads even on partial failures

43 - Job permissions include `actions: read` and `contents: write` at workflow level for cache restore and commits

@@ -118,6 +120,7 @@ Required secrets/tokens:

120 ### Trigger locally

121

122 - Crawl: `python3 scripts/crawl.py --as-of YYYY-MM-DD`

123 +- External news crawl: `python3 -m scripts.techcrunch_crawler --sources config/external_news_sources.json --output data/raw/YYYY-WNN-external-news.json --since YYYY-MM-DD --until YYYY-MM-DD`

124 - Analyze fallback: `python3 scripts/analyze_fallback.py --raw-json data/raw/YYYY-WNN.json --output data/analyzed/YYYY-WNN-summary.md --current-datetime YYYY-MM-DDTHH:MM:SSZ`

125 - Gate: `python3 scripts/analysis_gate.py --analysis-file data/analyzed/YYYY-WNN-summary.md --raw-json data/raw/YYYY-WNN.json --current-datetime YYYY-MM-DDTHH:MM:SSZ`

126 - Generate: `python3 scripts/generate_content.py data/analyzed/YYYY-WNN-summary.md`

scripts/analyze_fallback.py

+3 -1

@@ -374,7 +374,9 @@ def _render_press_section_no_ai(press_context_path: Path | None) -> str:

374 stem = press_context_path.stem # e.g. "2026-W21-press-context"

375 week = stem.replace("-press-context", "") # e.g. "2026-W21"

376 data_dir = press_context_path.parent.parent # data/analyzed/ -> data/

377 - tc_path = data_dir / "raw" / f"{week}-techcrunch.json"

377 + external_path = data_dir / "raw" / f"{week}-external-news.json"

378 + legacy_path = data_dir / "raw" / f"{week}-techcrunch.json"

379 + tc_path = external_path if external_path.exists() else legacy_path

380 corr_path = data_dir / "analyzed" / f"{week}-correlations.json"

381

382 if tc_path.exists():

scripts/correlate.py

+7 -5

@@ -1,12 +1,12 @@

1 #!/usr/bin/env python3

2 """Cross-source correlation engine for SquadScope.

4 -Matches TechCrunch articles to GitHub repo activity using fuzzy matching

4 +Matches external news articles to GitHub repo activity using fuzzy matching

5 heuristics to identify press-correlated repositories.

7 Usage:

8 python scripts/correlate.py [--raw data/raw/ai-ml/2026-W21.json] \

9 - [--techcrunch data/raw/ai-ml/2026-W21-techcrunch.json] \

9 + [--techcrunch data/raw/ai-ml/2026-W21-external-news.json] \

10 [--output data/analyzed/ai-ml/2026-W21-correlations.json] \

11 [--topic ai-ml]

12 """

@@ -384,7 +384,7 @@ def main(argv: list[str] | None = None) -> int:

384 )

385 parser.add_argument(

386 "--techcrunch", default=None,

387 - help="Path to TechCrunch articles JSON file",

387 + help="Path to external news articles JSON file",

388 )

389 parser.add_argument(

390 "--output", default=None,

@@ -411,11 +411,13 @@ def main(argv: list[str] | None = None) -> int:

411 log(f"Raw file not found: {raw_path}")

412 return 1

413

414 - # Resolve TechCrunch file

414 + # Resolve external news file, with TechCrunch-only legacy fallback.

415 if args.techcrunch:

416 tc_path = Path(args.techcrunch)

417 else:

418 - tc_path = find_latest_file(raw_dir(topic), "*-techcrunch.json")

418 + tc_path = find_latest_file(raw_dir(topic), "*-external-news.json")

419 + if tc_path is None:

420 + tc_path = find_latest_file(raw_dir(topic), "*-techcrunch.json")

421

422 # Load repos

423 raw_data = load_json(raw_path)

scripts/render_press_context.py

+5 -2

@@ -574,8 +574,11 @@ def render_press_context(

574

575

576 def resolve_paths(topic: str | None, week: str) -> tuple[Path, Path]:

577 - """Resolve file paths for TechCrunch and correlation data."""

578 - tc_path = raw_dir(topic) / f"{week}-techcrunch.json"

577 + """Resolve file paths for external news and correlation data."""

578 + raw_path = raw_dir(topic)

579 + external_path = raw_path / f"{week}-external-news.json"

580 + legacy_path = raw_path / f"{week}-techcrunch.json"

581 + tc_path = legacy_path if legacy_path.exists() and not external_path.exists() else external_path

582 corr_path = analyzed_dir(topic) / f"{week}-correlations.json"

583 return tc_path, corr_path

584

scripts/techcrunch_crawler.py

+197 -19

@@ -1,30 +1,46 @@

1 #!/usr/bin/env python3

2 -"""TechCrunch RSS crawler with entity extraction for SquadScope.

2 +"""External news RSS crawler with entity extraction for SquadScope.

4 -Fetches articles from TechCrunch RSS feed, extracts structured metadata,

4 +Fetches configured RSS feeds in parallel, extracts structured metadata,

5 GitHub URLs, and entities (company/project names).

7 Usage:

8 python scripts/techcrunch_crawler.py [--topic ai-ml] \

9 - [--output data/raw/ai-ml/2026-W21-techcrunch.json] [--since 2026-05-11]

9 + [--output data/raw/ai-ml/2026-W21-external-news.json] [--since 2026-05-11]

10 """

12 from __future__ import annotations

14 import argparse

15 +import ipaddress

16 import json

17 import re

18 import sys

19 import time

20 +from collections import Counter

21 +from concurrent.futures import ThreadPoolExecutor, as_completed

22 +from dataclasses import dataclass

23 from datetime import UTC, datetime, timedelta

24 from pathlib import Path

25 from typing import Any

26 +from urllib.parse import urlparse

27 +from urllib.request import Request, urlopen

29 import feedparser

31 from scripts.topic_paths import raw_dir

33 FEED_URL = "https://techcrunch.com/feed/"

34 +DEFAULT_SOURCES_PATH = Path("config/external_news_sources.json")

35 +DEFAULT_FETCH_TIMEOUT_SECONDS = 15

36 +DEFAULT_MAX_WORKERS = 8

37 +APPROVED_FEED_HOSTS = frozenset({

38 + "techcrunch.com",

39 + "blogs.nvidia.com",

40 + "huggingface.co",

41 + "www.technologyreview.com",

42 + "github.blog",

43 +})

45 GITHUB_URL_RE = re.compile(

46 r"https?://github\.com/[a-zA-Z0-9_.\-]+/[a-zA-Z0-9_.\-]+"

@@ -56,6 +72,78 @@ STOP_WORDS = {

72 }

75 +@dataclass(frozen=True, slots=True)

76 +class NewsSourceConfig:

77 + """Configuration for one external RSS source."""

78 +

79 + name: str

80 + feed_url: str

81 + requests_per_minute: int = 10

82 +

83 + def __post_init__(self) -> None:

84 + validate_feed_url(self.feed_url)

85 +

86 +

87 +def validate_feed_url(url: str) -> None:

88 + """Validate an external RSS URL against the approved egress allowlist."""

89 + parsed = urlparse(url)

90 + if parsed.scheme.lower() != "https":

91 + raise ValueError(f"External RSS feed URL must use HTTPS: {url}")

92 + if parsed.username or parsed.password:

93 + raise ValueError(f"External RSS feed URL must not include credentials: {url}")

94 + host = (parsed.hostname or "").rstrip(".").lower()

95 + if not host:

96 + raise ValueError(f"External RSS feed URL must include a hostname: {url}")

97 + try:

98 + port = parsed.port

99 + except ValueError as exc:

100 + raise ValueError(f"External RSS feed URL has an invalid port: {url}") from exc

101 + if port not in (None, 443):

102 + raise ValueError(f"External RSS feed URL must not use unexpected ports: {url}")

103 + if host in {"localhost", "localhost.localdomain"} or host.endswith(".local"):

104 + raise ValueError(f"External RSS feed URL must not target local hosts: {url}")

105 + try:

106 + ip_addr = ipaddress.ip_address(host)

107 + except ValueError:

108 + pass

109 + else:

110 + if ip_addr.is_private or ip_addr.is_loopback or ip_addr.is_link_local:

111 + raise ValueError(

112 + f"External RSS feed URL must not target private/local IPs: {url}"

113 + )

114 + if host not in APPROVED_FEED_HOSTS:

115 + approved = ", ".join(sorted(APPROVED_FEED_HOSTS))

116 + raise ValueError(

117 + f"External RSS feed host is not approved: {host} (approved: {approved})"

118 + )

119 +

120 +

121 +def load_source_configs(path: Path = DEFAULT_SOURCES_PATH) -> list[NewsSourceConfig]:

122 + """Load external RSS source config from JSON."""

123 + payload = json.loads(path.read_text(encoding="utf-8"))

124 + if not isinstance(payload, list):

125 + raise ValueError(f"Expected a list of source configs in {path}")

126 +

127 + sources: list[NewsSourceConfig] = []

128 + seen_names: set[str] = set()

129 + for raw in payload:

130 + if not isinstance(raw, dict):

131 + raise ValueError(f"Invalid source config in {path}: {raw!r}")

132 + name = str(raw.get("name", "")).strip()

133 + feed_url = str(raw.get("feed_url", "")).strip()

134 + if not name or not feed_url:

135 + raise ValueError(f"Source configs require name and feed_url: {raw!r}")

136 + if name in seen_names:

137 + raise ValueError(f"Duplicate external news source name: {name}")

138 + seen_names.add(name)

139 + sources.append(NewsSourceConfig(

140 + name=name,

141 + feed_url=feed_url,

142 + requests_per_minute=int(raw.get("requests_per_minute", 10)),

143 + ))

144 + return sources

145 +

146 +

147 def iso_timestamp(value: datetime) -> str:

148 """Format datetime as ISO 8601 UTC string."""

149 return value.astimezone(UTC).isoformat().replace("+00:00", "Z")

@@ -138,10 +226,25 @@ def parse_published_date(entry: Any) -> datetime | None:

226 return None

227

228

141 -def fetch_feed(url: str = FEED_URL, retries: int = 1) -> Any:

142 - """Fetch and parse RSS feed with retry on failure."""

229 +def fetch_feed(

230 + url: str = FEED_URL,

231 + retries: int = 1,

232 + timeout: int = DEFAULT_FETCH_TIMEOUT_SECONDS,

233 +) -> Any:

234 + """Fetch and parse RSS feed with bounded retries and an explicit timeout."""

235 + validate_feed_url(url)

236 + if timeout <= 0:

237 + raise ValueError("RSS fetch timeout must be greater than zero")

238 for attempt in range(retries + 1):

144 - feed = feedparser.parse(url)

239 + try:

240 + request = Request(url, headers={"User-Agent": "SquadScope RSS crawler"})

241 + with urlopen(request, timeout=timeout) as response:

242 + feed = feedparser.parse(response.read())

243 + except Exception:

244 + if attempt < retries:

245 + time.sleep(2)

246 + continue

247 + raise

248 if feed.bozo and not feed.entries:

249 if attempt < retries:

250 time.sleep(2)

@@ -152,23 +255,28 @@ def fetch_feed(url: str = FEED_URL, retries: int = 1) -> Any:

255 return feed # pragma: no cover

256

257

155 -class TechCrunchSource:

156 - """TechCrunch RSS data source following the DataSource protocol."""

258 +class NewsFeedSource:

259 + """RSS data source following the DataSource protocol."""

260 +

261 + def __init__(self, config: NewsSourceConfig) -> None:

262 + self.config = config

263

264 def get_name(self) -> str:

159 - return "techcrunch"

265 + return self.config.name

266

267 def get_rate_limits(self) -> dict:

162 - return {"requests_per_minute": 10}

268 + return {"requests_per_minute": self.config.requests_per_minute}

269

270 def crawl(

271 self,

272 since: datetime,

273 until: datetime,

168 - feed_url: str = FEED_URL,

274 + feed_url: str | None = None,

275 ) -> list[dict[str, Any]]:

170 - """Crawl TechCrunch RSS feed and return structured articles."""

171 - feed = fetch_feed(feed_url)

276 + """Crawl an RSS feed and return structured articles."""

277 + resolved_feed_url = feed_url or self.config.feed_url

278 + validate_feed_url(resolved_feed_url)

279 + feed = fetch_feed(resolved_feed_url)

280 articles: list[dict[str, Any]] = []

281

282 for entry in feed.entries:

@@ -197,6 +305,7 @@ class TechCrunchSource:

305 summary = summary[:497] + "..."

306

307 article: dict[str, Any] = {

308 + "source": self.get_name(),

309 "title": getattr(entry, "title", ""),

310 "url": getattr(entry, "link", ""),

311 "published_at": iso_timestamp(pub_date),

@@ -211,9 +320,54 @@ class TechCrunchSource:

320 return articles

321

322

323 +class TechCrunchSource(NewsFeedSource):

324 + """Backward-compatible TechCrunch RSS source."""

325 +

326 + def __init__(self) -> None:

327 + super().__init__(NewsSourceConfig("techcrunch", FEED_URL, 10))

328 +

329 +

330 +def crawl_sources_parallel(

331 + sources: list[NewsSourceConfig],

332 + since: datetime,

333 + until: datetime,

334 + max_workers: int | None = None,

335 +) -> tuple[list[dict[str, Any]], list[dict[str, str]]]:

336 + """Crawl configured RSS sources concurrently and return articles plus errors."""

337 + if not sources:

338 + return [], []

339 +

340 + if max_workers is not None and max_workers < 1:

341 + raise ValueError("--max-workers must be at least 1")

342 + workers = min(max_workers or len(sources), len(sources), DEFAULT_MAX_WORKERS)

343 + articles: list[dict[str, Any]] = []

344 + errors: list[dict[str, str]] = []

345 + with ThreadPoolExecutor(max_workers=workers) as executor:

346 + futures = {

347 + executor.submit(NewsFeedSource(source).crawl, since, until): source

348 + for source in sources

349 + }

350 + for future in as_completed(futures):

351 + source = futures[future]

352 + try:

353 + articles.extend(future.result())

354 + except Exception as exc: # pragma: no cover - defensive around network/parser failures

355 + errors.append({"source": source.name, "error": str(exc)})

356 +

357 + articles.sort(

358 + key=lambda article: (article.get("published_at", ""), article.get("source", "")),

359 + reverse=True,

360 + )

361 + return articles, errors

362 +

363 +

364 def build_output(

365 articles: list[dict[str, Any]],

366 crawled_at: datetime,

367 + *,

368 + source: str = "techcrunch",

369 + source_count: int = 1,

370 + errors: list[dict[str, str]] | None = None,

371 ) -> dict[str, Any]:

372 """Build the final output structure with metadata."""

373 relevant = [a for a in articles if a["relevance_score"] >= 0.4]

@@ -221,22 +375,26 @@ def build_output(

375 for a in articles:

376 all_github_links.update(a.get("github_links", []))

377

378 + by_source = Counter(str(article.get("source", source)) for article in articles)

379 return {

380 "week": week_slug(crawled_at),

226 - "source": "techcrunch",

381 + "source": source,

382 "crawled_at": iso_timestamp(crawled_at),

383 "articles": articles,

384 "metadata": {

385 + "source_count": source_count,

386 + "sources_with_articles": dict(sorted(by_source.items())),

387 "total_articles": len(articles),

388 "relevant_articles": len(relevant),

389 "github_links_found": len(all_github_links),

390 + "errors": errors or [],

391 },

392 }

393

394

395 def main(argv: list[str] | None = None) -> int:

396 parser = argparse.ArgumentParser(

239 - description="Crawl TechCrunch RSS feed for SquadScope"

397 + description="Crawl external news RSS feeds for SquadScope"

398 )

399 parser.add_argument(

400 "--topic", default="general",

@@ -254,6 +412,17 @@ def main(argv: list[str] | None = None) -> int:

412 "--until", default=None,

413 help="End date filter (YYYY-MM-DD, default: now)",

414 )

415 + parser.add_argument(

416 + "--sources",

417 + default=str(DEFAULT_SOURCES_PATH),

418 + help="Path to external RSS source config JSON",

419 + )

420 + parser.add_argument(

421 + "--max-workers",

422 + type=int,

423 + default=None,

424 + help="Maximum parallel RSS fetches (default: one per source, capped at 8)",

425 + )

426 args = parser.parse_args(argv)

427

428 now = datetime.now(UTC)

@@ -268,22 +437,31 @@ def main(argv: list[str] | None = None) -> int:

437 else now

438 )

439

271 - source = TechCrunchSource()

272 - articles = source.crawl(since=since, until=until)

273 - output = build_output(articles, crawled_at=now)

440 + source_configs = load_source_configs(Path(args.sources))

441 + articles, errors = crawl_sources_parallel(

442 + source_configs, since=since, until=until, max_workers=args.max_workers

443 + )

444 + output = build_output(

445 + articles,

446 + crawled_at=now,

447 + source="external_news",

448 + source_count=len(source_configs),

449 + errors=errors,

450 + )

451

452 if args.output:

453 out_path = Path(args.output)

454 else:

455 out_dir = raw_dir(args.topic)

456 out_dir.mkdir(parents=True, exist_ok=True)

280 - out_path = out_dir / f"{week_slug(now)}-techcrunch.json"

457 + out_path = out_dir / f"{week_slug(now)}-external-news.json"

458

459 out_path.parent.mkdir(parents=True, exist_ok=True)

460 with open(out_path, "w", encoding="utf-8") as f:

461 json.dump(output, f, indent=2, ensure_ascii=False)

462

463 print(f"Crawled {output['metadata']['total_articles']} articles "

464 + f"from {output['metadata']['source_count']} sources "

465 f"({output['metadata']['relevant_articles']} relevant) → {out_path}")

466 return 0

467

tests/test_render_press_context.py

+6 -1

@@ -192,10 +192,15 @@ class TestRenderPressContext:

192 class TestResolvePaths:

193 def test_with_topic(self):

194 tc, corr = resolve_paths("ai-ml", "2026-W21")

195 - assert "raw/ai-ml/2026-W21-techcrunch.json" in str(tc)

195 + assert "raw/ai-ml/2026-W21-external-news.json" in str(tc)

196 assert "analyzed/ai-ml/2026-W21-correlations.json" in str(corr)

197

198 def test_without_topic(self):

199 + tc, corr = resolve_paths(None, "2026-W99")

200 + assert "2026-W99-external-news.json" in str(tc)

201 + assert "2026-W99-correlations.json" in str(corr)

202 +

203 + def test_legacy_techcrunch_fallback(self):

204 tc, corr = resolve_paths(None, "2026-W21")

205 assert "2026-W21-techcrunch.json" in str(tc)

206 assert "2026-W21-correlations.json" in str(corr)

tests/test_techcrunch_crawler.py

+148

@@ -2,6 +2,7 @@

3 from __future__ import annotations

5 +import json

6 from datetime import UTC, datetime, timedelta

7 from types import SimpleNamespace

8 from unittest.mock import patch

@@ -9,13 +10,20 @@ from unittest.mock import patch

10 import pytest

12 from scripts.techcrunch_crawler import (

13 + DEFAULT_SOURCES_PATH,

14 + DEFAULT_FETCH_TIMEOUT_SECONDS,

15 + NewsSourceConfig,

16 TechCrunchSource,

17 build_output,

18 compute_relevance_score,

19 + crawl_sources_parallel,

20 extract_entities,

21 extract_github_urls,

22 + fetch_feed,

23 iso_timestamp,

24 + load_source_configs,

25 parse_published_date,

26 + validate_feed_url,

27 week_slug,

28 )

@@ -188,6 +196,7 @@ class TestTechCrunchSourceCrawl:

196 )

197

198 assert len(articles) == 1

199 + assert articles[0]["source"] == "techcrunch"

200 assert articles[0]["title"] == "Test Article"

201

202 def test_crawl_extracts_github_links(self):

@@ -267,9 +276,11 @@ class TestBuildOutput:

276 assert output["source"] == "techcrunch"

277 assert output["week"] == week_slug(now)

278 assert output["crawled_at"] == "2026-05-19T10:00:00Z"

279 + assert output["metadata"]["source_count"] == 1

280 assert output["metadata"]["total_articles"] == 2

281 assert output["metadata"]["relevant_articles"] == 1

282 assert output["metadata"]["github_links_found"] == 1

283 + assert output["metadata"]["errors"] == []

284 assert len(output["articles"]) == 2

285

286

@@ -285,3 +296,140 @@ class TestDataSourceProtocol:

296 limits = source.get_rate_limits()

297 assert "requests_per_minute" in limits

298 assert limits["requests_per_minute"] == 10

299 +

300 +

301 +# --- Config and parallel crawl tests ---

302 +

303 +class TestExternalNewsSources:

304 + def test_load_default_source_configs(self):

305 + sources = load_source_configs(DEFAULT_SOURCES_PATH)

306 + names = {source.name for source in sources}

307 +

308 + assert "techcrunch" in names

309 + assert "nvidia_blog" in names

310 + assert "hugging_face_blog" in names

311 + assert "mit_technology_review" in names

312 + assert "github_blog" in names

313 +

314 + @pytest.mark.parametrize(

315 + "feed_url",

316 + [

317 + "http://techcrunch.com/feed/",

318 + "https://user:pass@techcrunch.com/feed/",

319 + "https://localhost/feed/",

320 + "https://127.0.0.1/feed/",

321 + "https://169.254.169.254/feed/",

322 + "https://example.com/feed/",

323 + "https://techcrunch.com:8443/feed/",

324 + ],

325 + )

326 + def test_rejects_invalid_or_unapproved_feed_urls(self, feed_url):

327 + with pytest.raises(ValueError):

328 + validate_feed_url(feed_url)

329 +

330 + def test_load_source_configs_rejects_unapproved_hosts(self):

331 + payload = json.dumps([

332 + {

333 + "name": "evil",

334 + "feed_url": "https://example.com/feed.xml",

335 + "requests_per_minute": 10,

336 + }

337 + ])

338 +

339 + with patch("pathlib.Path.read_text", return_value=payload):

340 + with pytest.raises(ValueError, match="not approved"):

341 + load_source_configs(DEFAULT_SOURCES_PATH)

342 +

343 + def test_fetch_feed_uses_explicit_timeout(self):

344 + class FakeResponse:

345 + def __enter__(self):

346 + return self

347 +

348 + def __exit__(self, exc_type, exc, traceback):

349 + return None

350 +

351 + def read(self):

352 + return b"<rss><channel></channel></rss>"

353 +

354 + feed = _make_feed()

355 + with (

356 + patch(

357 + "scripts.techcrunch_crawler.urlopen",

358 + return_value=FakeResponse(),

359 + ) as mock_urlopen,

360 + patch("scripts.techcrunch_crawler.feedparser.parse", return_value=feed),

361 + ):

362 + result = fetch_feed("https://techcrunch.com/feed/")

363 +

364 + assert result is feed

365 + assert mock_urlopen.call_args.kwargs["timeout"] == DEFAULT_FETCH_TIMEOUT_SECONDS

366 +

367 + def test_crawl_sources_parallel_combines_sources(self):

368 + alpha_entry = _make_entry(

369 + title="Alpha AI framework",

370 + link="https://example.com/alpha",

371 + published_parsed=(2026, 5, 16, 10, 0, 0, 4, 136, 0),

372 + )

373 + beta_entry = _make_entry(

374 + title="Beta developer API",

375 + link="https://example.com/beta",

376 + published_parsed=(2026, 5, 17, 10, 0, 0, 5, 137, 0),

377 + )

378 + feeds = {

379 + "https://techcrunch.com/feed/": _make_feed(entries=[alpha_entry]),

380 + "https://github.blog/feed/": _make_feed(entries=[beta_entry]),

381 + }

382 +

383 + def fake_fetch(url, retries=1, timeout=DEFAULT_FETCH_TIMEOUT_SECONDS):

384 + return feeds[url]

385 +

386 + sources = [

387 + NewsSourceConfig("alpha", "https://techcrunch.com/feed/"),

388 + NewsSourceConfig("beta", "https://github.blog/feed/"),

389 + ]

390 + with patch("scripts.techcrunch_crawler.fetch_feed", side_effect=fake_fetch):

391 + articles, errors = crawl_sources_parallel(

392 + sources,

393 + since=datetime(2026, 5, 10, tzinfo=UTC),

394 + until=datetime(2026, 5, 20, tzinfo=UTC),

395 + max_workers=2,

396 + )

397 +

398 + assert errors == []

399 + assert [article["source"] for article in articles] == ["beta", "alpha"]

400 + assert {article["title"] for article in articles} == {

401 + "Alpha AI framework",

402 + "Beta developer API",

403 + }

404 +

405 + def test_external_news_output_metadata(self):

406 + now = datetime(2026, 5, 19, 10, 0, 0, tzinfo=UTC)

407 + output = build_output(

408 + [

409 + {

410 + "source": "alpha",

411 + "title": "AI framework",

412 + "summary": "open source",

413 + "categories": [],

414 + "github_links": [],

415 + "relevance_score": 0.4,

416 + },

417 + {

418 + "source": "beta",

419 + "title": "Developer API",

420 + "summary": "sdk",

421 + "categories": [],

422 + "github_links": [],

423 + "relevance_score": 0.4,

424 + },

425 + ],

426 + crawled_at=now,

427 + source="external_news",

428 + source_count=2,

429 + errors=[{"source": "gamma", "error": "timeout"}],

430 + )

431 +

432 + assert output["source"] == "external_news"

433 + assert output["metadata"]["source_count"] == 2

434 + assert output["metadata"]["sources_with_articles"] == {"alpha": 1, "beta": 1}

435 + assert output["metadata"]["errors"] == [{"source": "gamma", "error": "timeout"}]