@cryptotaxi247 / SquadScope / commits / 498496e

feat: harden external news telemetry (#242)

Closes #237. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jun 5, 2026 at 19:24 UTC 498496e3fd00689a460b8a73542eef5c3f8a0f3a
10 files changed +795 -41
.github/workflows/crawl-and-publish.yml
+36 -1
@@ -109,10 +109,29 @@ jobs:
109 run: |
110 WEEK=$(date +%Y-W%V)
111 SINCE=$(date -d '7 days ago' +%Y-%m-%d)
112 + UNTIL=$(date +%Y-%m-%d)
113 python scripts/techcrunch_crawler.py \
114 --sources config/external_news_sources.json \
115 --output "data/raw/${WEEK}-external-news.json" \
115 - --since "$SINCE"
116 + --since "$SINCE" \
117 + --until "$UNTIL"
118 + python3 - <<'PY' "data/raw/${WEEK}-external-news.json"
119 + import json
120 + import sys
121 + from pathlib import Path
122 +
123 + path = Path(sys.argv[1])
124 + payload = json.loads(path.read_text(encoding="utf-8"))
125 + metadata = payload.get("metadata", {})
126 + print(
127 + "::notice::External news artifact "
128 + f"size_bytes={path.stat().st_size} "
129 + f"articles={metadata.get('total_articles', 0)} "
130 + f"relevant={metadata.get('relevant_articles', 0)} "
131 + f"dedupe={metadata.get('dedupe_count', 0)} "
132 + f"checksum={metadata.get('artifact_checksum', '')[:12]}"
133 + )
134 + PY
135
136 - name: Upload raw crawl artifact
137 if: always()
@@ -296,6 +315,19 @@ jobs:
315 PRESS_CONTEXT=$(python scripts/render_press_context.py --week "$WEEK")
316 PRESS_FILE="data/analyzed/${WEEK}-press-context.md"
317 printf '%s\n' "$PRESS_CONTEXT" > "$PRESS_FILE"
318 + python3 - <<'PY' "$PRESS_FILE"
319 + import sys
320 + from pathlib import Path
321 + from scripts.render_press_context import estimate_tokens
322 +
323 + path = Path(sys.argv[1])
324 + content = path.read_text(encoding="utf-8")
325 + print(
326 + "::notice::Press context "
327 + f"size_bytes={path.stat().st_size} "
328 + f"token_estimate={estimate_tokens(content)}"
329 + )
330 + PY
331 echo "press_file=$PRESS_FILE" >> "$GITHUB_OUTPUT"
332
333 - name: Pre-flight cost check
@@ -330,6 +362,7 @@ jobs:
362 WEEK="${{ steps.analysis-context.outputs.week }}"
363 CURRENT_DATETIME="${{ steps.analysis-context.outputs.current_datetime }}"
364 PRESS_FILE="${{ steps.press-context.outputs.press_file }}"
365 + ANALYSIS_STARTED=$(date +%s)
366 mkdir -p data/metrics
367 # Hydrate metrics ledger from publish so track_token_usage.py appends to
368 # the canonical token-usage.jsonl rather than starting fresh each run.
@@ -468,6 +501,8 @@ jobs:
501 --prompt-file "$PROMPT_FILE" \
502 --output-file "$OUTPUT_FILE" \
503 $TRANSCRIPT_ARGS
504 + ANALYSIS_DURATION=$(( $(date +%s) - ANALYSIS_STARTED ))
505 + echo "::notice::Analysis path source=$ANALYSIS_SOURCE model=$ANALYSIS_MODEL duration_seconds=$ANALYSIS_DURATION press_context=$PRESS_FILE"
506 rm -f "$PROMPT_FILE"
507 # Clean up per-attempt transcripts
508 rm -f data/metrics/copilot-transcript-attempt-*.md
.squad/agents/bender/history.md
+5
@@ -36,3 +36,8 @@
36 - Decision recorded in .squad/decisions.md; GitHub issue #237 created for implementation.
37 - External RSS crawlers must validate config URLs against an HTTPS host allowlist and fetch through explicit per-request timeouts; config-driven source lists are not a security boundary by themselves (PR #236).
38 - Local validation docs for scripts importing `scripts.*` modules should use `python3 -m ...` or set `PYTHONPATH=.` so repo-root imports resolve reliably (PR #236).
39 +
40 +## Issue #237 canonical external-news telemetry (2026-06-05)
41 +
42 +- Multi-source RSS remains in-process, but downstream reliability depends on a versioned canonical artifact: include crawl window, source config checksum, per-source statuses, partial failures, dedupe count, and deterministic checksum in `*-external-news.json`.
43 +- Press correlation must retain bounded source-aware citations and label category/fuzzy-only matches as weak so mirrored coverage or broad topics do not inflate strong press claims.
.squad/decisions/inbox/bender-issue-237-implementation.md new
+15
@@ -0,0 +1,15 @@
1 +# Bender issue #237 implementation
2 +
3 +Date: 2026-06-05
4 +
5 +## Decision
6 +
7 +Keep external RSS/news in the existing crawl job with bounded in-process parallelism, but promote the handoff to a canonical `schema_version: 2` `data/raw/{week}-external-news.json` artifact. The artifact carries crawl window, source config checksum, requested/succeeded/failed sources, per-source status metrics, dedupe count, deterministic checksum, and partial-failure metadata.
8 +
9 +## Rationale
10 +
11 +The measured bottleneck remains the GitHub repository crawl, not the five-source RSS step. Source-aware telemetry and schema validation improve downstream reliability without adding Actions matrix startup overhead or splitting cache/API behavior.
12 +
13 +## Operational notes
14 +
15 +`correlate.py` and `render_press_context.py` now preserve article source/title/date/URL citations, label strong versus weak correlations, bound press context size to an ~8k token estimate, and keep legacy `*-techcrunch.json` and no-press fallbacks.
docs/pipeline-validation.md
+8
@@ -38,6 +38,8 @@ Required secrets/tokens:
38 **Success criteria**
39 - Raw payload passes `scripts.crawl.validate_payload()`
40 - External RSS payload is written for the same ISO week from `config/external_news_sources.json`
41 +- `data/raw/YYYY-WNN-external-news.json` uses canonical `schema_version: 2`, includes `crawl_window`, `source_config_checksum`, `sources_requested/succeeded/failed`, per-source status rows, `dedupe_count`, and `artifact_checksum`
42 +- Optional per-source RSS failures are warnings with a valid partial artifact; malformed config/schema/checksum errors fail the crawl step
43 - Snapshot file is written for the same ISO week
44 - Cache artifact uploads even on partial failures
45 - Job permissions include `actions: read` and `contents: write` at workflow level for cache restore and commits
@@ -53,12 +55,16 @@ Required secrets/tokens:
55
56 **Outputs**
57 - `data/analyzed/YYYY-WNN-summary.md`
58 +- `data/analyzed/YYYY-WNN-correlations.json`
59 +- `data/analyzed/YYYY-WNN-press-context.md`
60 - `analyzed-data` artifact
61 - Commit to `main` for `data/analyzed/`
62 - Job outputs: `week`, `summary_file`, `current_datetime`
63
64 **Success criteria**
65 - Current raw file week matches the run week
66 +- Correlation and press-context steps consume compact external-news data with legacy `YYYY-WNN-techcrunch.json` fallback
67 +- Press context preserves source names, article URLs/titles/dates, strong-vs-weak labels, and partial-source caveats while staying under the ~8k token budget
68 - Copilot CLI output or fallback output is written to `data/analyzed/`
69 - `scripts/analysis_gate.py` passes before publish continues
70 - Job permissions include `actions: read`, `contents: write`, `copilot-requests: write`, and `models: read`
@@ -121,6 +127,8 @@ Required secrets/tokens:
127
128 - Crawl: `python3 scripts/crawl.py --as-of YYYY-MM-DD`
129 - External news crawl: `python3 -m scripts.techcrunch_crawler --sources config/external_news_sources.json --output data/raw/YYYY-WNN-external-news.json --since YYYY-MM-DD --until YYYY-MM-DD`
130 +- Correlate press: `python3 -m scripts.correlate --raw data/raw/YYYY-WNN.json --techcrunch data/raw/YYYY-WNN-external-news.json --output data/analyzed/YYYY-WNN-correlations.json`
131 +- Render press context: `python3 -m scripts.render_press_context --week YYYY-WNN`
132 - Analyze fallback: `python3 scripts/analyze_fallback.py --raw-json data/raw/YYYY-WNN.json --output data/analyzed/YYYY-WNN-summary.md --current-datetime YYYY-MM-DDTHH:MM:SSZ`
133 - Gate: `python3 scripts/analysis_gate.py --analysis-file data/analyzed/YYYY-WNN-summary.md --raw-json data/raw/YYYY-WNN.json --current-datetime YYYY-MM-DDTHH:MM:SSZ`
134 - Generate: `python3 scripts/generate_content.py data/analyzed/YYYY-WNN-summary.md`
scripts/correlate.py
+199 -9
@@ -23,6 +23,12 @@ from typing import Any
23
24 from scripts.topic_paths import analyzed_dir, raw_dir
25
26 +MAX_ARTICLES_FOR_CORRELATION = 80
27 +MAX_CORRELATIONS = 50
28 +MAX_MATCHED_ARTICLES_PER_REPO = 5
29 +MAX_DIVERGENCE_ARTICLES = 30
30 +WEAK_MATCH_TYPES = {"category", "project_name"}
31 +
32
33 def log(message: str) -> None:
34 print(f"[correlate] {message}", file=sys.stderr)
@@ -125,6 +131,114 @@ def match_category(repo: dict[str, Any], articles: list[dict[str, Any]]) -> list
131 return matches
132
133
134 +def _normalized_article_url(url: str) -> str:
135 + """Normalize an article URL for dedupe and citation joins."""
136 + if not url:
137 + return ""
138 + from urllib.parse import urlparse
139 +
140 + parsed = urlparse(url.strip())
141 + return f"{parsed.scheme.lower()}://{parsed.netloc.lower()}{parsed.path.rstrip('/')}"
142 +
143 +
144 +def dedupe_articles(articles: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], int]:
145 + """Deduplicate cross-source/mirrored stories by normalized URL."""
146 + grouped: dict[str, dict[str, Any]] = {}
147 + duplicates = 0
148 + for article in sorted(
149 + articles,
150 + key=lambda item: (
151 + item.get("published_at", ""),
152 + item.get("source", ""),
153 + item.get("url", ""),
154 + item.get("title", ""),
155 + ),
156 + reverse=True,
157 + ):
158 + key = _normalized_article_url(str(article.get("url", "")))
159 + if not key:
160 + key = str(article.get("title", "")).strip().lower()
161 + if key not in grouped:
162 + current = dict(article)
163 + current["sources"] = sorted({
164 + str(current.get("source", "")) or "unknown",
165 + *[str(source) for source in current.get("sources", [])],
166 + })
167 + grouped[key] = current
168 + continue
169 + duplicates += 1
170 + existing = grouped[key]
171 + sources = set(existing.get("sources", []))
172 + sources.add(str(article.get("source", "")) or "unknown")
173 + sources.update(str(source) for source in article.get("sources", []))
174 + existing["sources"] = sorted(sources)
175 + existing["relevance_score"] = max(
176 + float(existing.get("relevance_score", 0)),
177 + float(article.get("relevance_score", 0)),
178 + )
179 + existing_links = list(existing.get("github_links", []))
180 + for link in article.get("github_links", []):
181 + if link not in existing_links:
182 + existing_links.append(link)
183 + existing["github_links"] = existing_links
184 + deduped = list(grouped.values())
185 + deduped.sort(
186 + key=lambda item: (
187 + item.get("published_at", ""),
188 + item.get("source", ""),
189 + item.get("url", ""),
190 + item.get("title", ""),
191 + ),
192 + reverse=True,
193 + )
194 + return deduped, duplicates
195 +
196 +
197 +def _article_citation(article: dict[str, Any]) -> dict[str, Any]:
198 + """Return the bounded citation fields downstream renderers are allowed to use."""
199 + return {
200 + "title": article.get("title", ""),
201 + "url": article.get("url", ""),
202 + "source": article.get("source", "unknown"),
203 + "sources": article.get("sources", [article.get("source", "unknown")]),
204 + "published_at": article.get("published_at", ""),
205 + "relevance_score": article.get("relevance_score", 0),
206 + }
207 +
208 +
209 +def _unique_articles(articles: list[dict[str, Any]]) -> list[dict[str, Any]]:
210 + """Return URL-deduped articles preserving order."""
211 + seen: set[str] = set()
212 + unique: list[dict[str, Any]] = []
213 + for article in articles:
214 + key = _normalized_article_url(str(article.get("url", ""))) or str(article)
215 + if key in seen:
216 + continue
217 + seen.add(key)
218 + unique.append(article)
219 + return unique
220 +
221 +
222 +def correlation_strength(
223 + match_type: str,
224 + matched_articles: list[dict[str, Any]],
225 + *,
226 + temporal_spike: bool,
227 +) -> str:
228 + """Label strong vs weak correlations without letting fuzzy/category inflate claims."""
229 + source_names = {
230 + source
231 + for article in matched_articles
232 + for source in article.get("sources", [article.get("source", "unknown")])
233 + }
234 + corroborated = len(source_names) >= 2 or len(_unique_articles(matched_articles)) >= 2
235 + if match_type in WEAK_MATCH_TYPES:
236 + return "weak"
237 + if match_type in {"direct_link", "org_name"} or temporal_spike or corroborated:
238 + return "strong"
239 + return "weak"
240 +
241 +
242 def has_temporal_spike(repo: dict[str, Any], stars_threshold: int = 10) -> bool:
243 """Check if repo had a stars_gained spike in the same week."""
244 stars_gained = repo.get("stars_gained")
@@ -161,13 +275,15 @@ def correlate_repo(repo: dict[str, Any], articles: list[dict[str, Any]]) -> dict
275 best_confidence = 0.0
276 best_type = ""
277 matched_articles: list[str] = []
278 + matched_article_objs: list[dict[str, Any]] = []
279
280 # Priority 1: Direct link match (confidence 1.0)
281 direct = match_direct_link(repo, articles)
282 if direct:
283 best_confidence = 1.0
284 best_type = "direct_link"
170 - matched_articles = [a["url"] for a in direct]
285 + matched_article_objs = _unique_articles(direct)[:MAX_MATCHED_ARTICLES_PER_REPO]
286 + matched_articles = [a["url"] for a in matched_article_objs]
287
288 # Priority 2: Org name match (confidence 0.8)
289 if not matched_articles:
@@ -175,7 +291,8 @@ def correlate_repo(repo: dict[str, Any], articles: list[dict[str, Any]]) -> dict
291 if org:
292 best_confidence = 0.8
293 best_type = "org_name"
178 - matched_articles = [a["url"] for a in org]
294 + matched_article_objs = _unique_articles(org)[:MAX_MATCHED_ARTICLES_PER_REPO]
295 + matched_articles = [a["url"] for a in matched_article_objs]
296
297 # Priority 3: Project name fuzzy match (confidence 0.6)
298 if not matched_articles:
@@ -183,7 +300,8 @@ def correlate_repo(repo: dict[str, Any], articles: list[dict[str, Any]]) -> dict
300 if fuzzy:
301 best_confidence = 0.6
302 best_type = "project_name"
186 - matched_articles = [a["url"] for a in fuzzy]
303 + matched_article_objs = _unique_articles(fuzzy)[:MAX_MATCHED_ARTICLES_PER_REPO]
304 + matched_articles = [a["url"] for a in matched_article_objs]
305
306 # Priority 4: Category correlation (confidence 0.4)
307 if not matched_articles:
@@ -191,23 +309,36 @@ def correlate_repo(repo: dict[str, Any], articles: list[dict[str, Any]]) -> dict
309 if cat:
310 best_confidence = 0.4
311 best_type = "category"
194 - matched_articles = [a["url"] for a in cat]
312 + matched_article_objs = _unique_articles(cat)[:MAX_MATCHED_ARTICLES_PER_REPO]
313 + matched_articles = [a["url"] for a in matched_article_objs]
314
315 if not matched_articles:
316 return None
317
318 # Priority 5: Temporal lag bonus
319 press_correlated = True
201 - if has_temporal_spike(repo):
320 + temporal_spike = has_temporal_spike(repo)
321 + if temporal_spike and best_type not in WEAK_MATCH_TYPES:
322 best_confidence = min(best_confidence + 0.2, 1.0)
323 press_correlated = True
324 + strength = correlation_strength(
325 + best_type,
326 + matched_article_objs,
327 + temporal_spike=temporal_spike,
328 + )
329
330 return {
331 "repo": repo.get("full_name") or f"{repo.get('owner')}/{repo.get('name')}",
332 "press_correlated": press_correlated,
333 "correlation_confidence": round(best_confidence, 2),
334 "matched_articles": matched_articles,
335 + "matched_article_details": [
336 + _article_citation(article) for article in matched_article_objs
337 + ],
338 "match_type": best_type,
339 + "correlation_strength": strength,
340 + "confidence_label": strength,
341 + "temporal_spike": temporal_spike,
342 "hype_risk": assess_hype_risk(best_confidence, repo.get("stars_gained")),
343 }
344
@@ -253,7 +384,10 @@ def detect_divergences(
384 matched_article_urls.update(corr.get("matched_articles", []))
385
386 # Unmatched articles → uncovered tech trends
256 - unmatched_articles = [a for a in articles if a.get("url") not in matched_article_urls]
387 + unmatched_articles = [
388 + a for a in articles
389 + if a.get("url") not in matched_article_urls
390 + ][:MAX_DIVERGENCE_ARTICLES]
391
392 # Group unmatched articles by topic
393 topic_articles: dict[str, list[dict[str, Any]]] = {}
@@ -264,6 +398,10 @@ def detect_divergences(
398 uncovered_tech_trends = [
399 {
400 "topic": topic,
401 + "news_articles": [
402 + {"title": a.get("title", ""), "url": a.get("url", "")}
403 + for a in arts
404 + ],
405 "techcrunch_articles": [
406 {"title": a.get("title", ""), "url": a.get("url", "")}
407 for a in arts
@@ -297,7 +435,7 @@ def detect_divergences(
435 }
436 for r in reps
437 ],
300 - "signal": "No TechCrunch coverage",
438 + "signal": "No external press coverage",
439 }
440 for topic, reps in sorted(topic_repos.items(), key=lambda x: -len(x[1]))
441 ]
@@ -310,6 +448,8 @@ def detect_divergences(
448
449 def correlate_all(repos: list[dict[str, Any]], articles: list[dict[str, Any]], week: str) -> dict[str, Any]:
450 """Run correlation engine across all repos and articles."""
451 + articles, dedupe_count = dedupe_articles(articles)
452 + articles = articles[:MAX_ARTICLES_FOR_CORRELATION]
453 correlations: list[dict[str, Any]] = []
454 uncorrelated: list[str] = []
455
@@ -322,7 +462,14 @@ def correlate_all(repos: list[dict[str, Any]], articles: list[dict[str, Any]], w
462 uncorrelated.append(name)
463
464 # Sort by confidence descending
325 - correlations.sort(key=lambda c: c["correlation_confidence"], reverse=True)
465 + correlations.sort(
466 + key=lambda c: (
467 + c.get("correlation_strength") != "strong",
468 + -c["correlation_confidence"],
469 + c.get("repo", ""),
470 + )
471 + )
472 + correlations = correlations[:MAX_CORRELATIONS]
473
474 articles_matched = len({url for c in correlations for url in c["matched_articles"]})
475
@@ -336,8 +483,23 @@ def correlate_all(repos: list[dict[str, Any]], articles: list[dict[str, Any]], w
483 "uncorrelated_repos": uncorrelated,
484 "metadata": {
485 "repos_analyzed": len(repos),
486 + "articles_analyzed": len(articles),
487 "correlations_found": len(correlations),
488 + "strong_correlations": sum(
489 + 1 for corr in correlations
490 + if corr.get("correlation_strength") == "strong"
491 + ),
492 + "weak_correlations": sum(
493 + 1 for corr in correlations
494 + if corr.get("correlation_strength") == "weak"
495 + ),
496 "articles_matched": articles_matched,
497 + "dedupe_count": dedupe_count,
498 + "limits": {
499 + "max_articles": MAX_ARTICLES_FOR_CORRELATION,
500 + "max_correlations": MAX_CORRELATIONS,
501 + "max_matched_articles_per_repo": MAX_MATCHED_ARTICLES_PER_REPO,
502 + },
503 "uncovered_tech_trends": len(divergences["uncovered_tech_trends"]),
504 "unpublicized_dev_activity": len(divergences["unpublicized_dev_activity"]),
505 },
@@ -363,6 +525,30 @@ def load_json(path: Path) -> dict[str, Any]:
525 return json.load(f)
526
527
528 +def extract_news_metadata(news_data: dict[str, Any] | list[dict[str, Any]]) -> dict[str, Any]:
529 + """Extract source/failure metadata from canonical or legacy news payloads."""
530 + if isinstance(news_data, list):
531 + return {
532 + "schema_version": 1,
533 + "sources_requested": ["techcrunch"],
534 + "sources_succeeded": ["techcrunch"],
535 + "sources_failed": [],
536 + "source_status": [],
537 + "errors": [],
538 + }
539 + metadata = news_data.get("metadata", {})
540 + return {
541 + "schema_version": news_data.get("schema_version", 1),
542 + "source_config_checksum": metadata.get("source_config_checksum", ""),
543 + "sources_requested": metadata.get("sources_requested", [news_data.get("source", "techcrunch")]),
544 + "sources_succeeded": metadata.get("sources_succeeded", []),
545 + "sources_failed": metadata.get("sources_failed", []),
546 + "source_status": metadata.get("source_status", []),
547 + "errors": metadata.get("errors", []),
548 + "artifact_checksum": metadata.get("artifact_checksum", ""),
549 + }
550 +
551 +
552 def extract_week_from_filename(path: Path) -> str:
553 """Extract week slug from filename like '2026-W21.json'."""
554 match = re.search(r"(\d{4}-W\d{2})", path.name)
@@ -433,17 +619,21 @@ def main(argv: list[str] | None = None) -> int:
619
620 # Load articles (graceful if missing)
621 articles: list[dict[str, Any]] = []
622 + news_metadata: dict[str, Any] = {}
623 if tc_path and tc_path.exists():
624 tc_data = load_json(tc_path)
625 + news_metadata = extract_news_metadata(tc_data)
626 articles = tc_data if isinstance(tc_data, list) else tc_data.get("articles", [])
627 else:
440 - log("No TechCrunch data found; producing empty correlations")
628 + log("No external news data found; producing empty correlations")
629
630 # Determine week
631 week = extract_week_from_filename(raw_path)
632
633 # Run correlation
634 result = correlate_all(repos, articles, week)
635 + if news_metadata:
636 + result["metadata"]["news_sources"] = news_metadata
637
638 # Write output
639 if args.output:
scripts/render_press_context.py
+105 -14
@@ -22,6 +22,11 @@ sys.path.insert(0, str(_REPO_ROOT / "scripts"))
22
23 from topic_paths import raw_dir, analyzed_dir # noqa: E402
24
25 +PRESS_CONTEXT_TOKEN_BUDGET = 8000
26 +PRESS_CONTEXT_CHAR_BUDGET = PRESS_CONTEXT_TOKEN_BUDGET * 4
27 +MAX_RENDERED_ARTICLES = 40
28 +MAX_RENDERED_CORRELATIONS = 20
29 +
30
31 def current_week() -> str:
32 """Return the current ISO week as YYYY-WNN."""
@@ -43,15 +48,23 @@ def format_articles_list(articles: list[dict]) -> str:
48 if not articles:
49 return "- (none)"
50 lines = []
46 - for article in articles:
51 + for article in articles[:MAX_RENDERED_ARTICLES]:
52 title = article.get("title", "Untitled")
53 url = article.get("url", "")
54 categories = article.get("categories", [])
55 + source = article.get("source", "unknown")
56 + published_at = article.get("published_at", "")
57 cat_str = f" [{', '.join(categories)}]" if categories else ""
58 + source_str = f" — {source}"
59 + if published_at:
60 + source_str += f", {published_at[:10]}"
61 if url:
52 - lines.append(f"- [{title}]({url}){cat_str}")
62 + lines.append(f"- [{title}]({url}){cat_str}{source_str}")
63 else:
54 - lines.append(f"- {title}{cat_str}")
64 + lines.append(f"- {title}{cat_str}{source_str}")
65 + omitted = len(articles) - MAX_RENDERED_ARTICLES
66 + if omitted > 0:
67 + lines.append(f"…and {omitted} more relevant articles within budget")
68 return "\n".join(lines)
69
70
@@ -206,7 +219,7 @@ def _format_correlations_narrative(
219 arts_str = _join_links(article_links)
220 if idx == 0:
221 para = (
209 - f"This week's TechCrunch coverage closely tracks developer activity "
222 + f"This week's external press coverage closely tracks developer activity "
223 f"across {total} repos. {org.capitalize()} featured prominently: "
224 f"coverage of {arts_str} aligns with activity in {repos_str}."
225 )
@@ -218,7 +231,7 @@ def _format_correlations_narrative(
231 else:
232 if idx == 0:
233 para = (
221 - f"This week's TechCrunch coverage closely tracks developer activity "
234 + f"This week's external press coverage closely tracks developer activity "
235 f"across {total} repos. {org.capitalize()} shows the strongest signal, "
236 f"with {repos_str} seeing notable GitHub traction."
237 )
@@ -282,10 +295,25 @@ def format_correlations_list(
295 repo = corr.get("repo", "unknown")
296 match_type = corr.get("match_type", "unknown")
297 confidence = corr.get("correlation_confidence", 0.0)
298 + strength = corr.get("correlation_strength", corr.get("confidence_label", "unknown"))
299 hype_risk = corr.get("hype_risk", "none")
300 + details = corr.get("matched_article_details", [])
301 + sources = sorted({
302 + source
303 + for detail in details
304 + for source in detail.get("sources", [detail.get("source", "unknown")])
305 + })
306 + citation = ""
307 + if details:
308 + first = details[0]
309 + title = first.get("title", "article")
310 + url = first.get("url", "")
311 + citation = f", cited: [{title}]({url})" if url else f", cited: {title}"
312 lines.append(
313 f"- {repo} — match: {match_type}, "
288 - f"confidence: {confidence:.1f}, hype_risk: {hype_risk}"
314 + f"strength: {strength}, confidence: {confidence:.1f}, "
315 + f"sources: {', '.join(sources) if sources else 'unknown'}, "
316 + f"hype_risk: {hype_risk}{citation}"
317 )
318
319 if omitted > 0:
@@ -378,7 +406,7 @@ def _format_uncovered_narrative(items: list[dict]) -> str:
406 # Collect up to two article links across all topics
407 article_links: list[str] = []
408 for item in display:
381 - for a in item.get("techcrunch_articles", [])[:1]:
409 + for a in item.get("news_articles", item.get("techcrunch_articles", []))[:1]:
410 title = a.get("title", "article")
411 url = a.get("url", "")
412 if url:
@@ -404,7 +432,7 @@ def _format_uncovered_narrative(items: list[dict]) -> str:
432 article_str = "Press articles generated buzz"
433
434 return (
407 - f"TechCrunch heavily covered {topics_str} this week, but GitHub shows minimal "
435 + f"External press heavily covered {topics_str} this week, but GitHub shows minimal "
436 f"matching developer activity. {article_str}, yet no significant new repositories "
437 f"emerged in these spaces — suggesting these are still in the narrative or "
438 f"announcement phase rather than implementation."
@@ -446,10 +474,10 @@ def format_divergences(divergences: dict, *, reader_mode: bool = False) -> str:
474 # AI prompt mode: full raw data for model consumption — keep unchanged
475 if uncovered:
476 lines.append("#### 🔍 Tech Trends Without Dev Activity")
449 - lines.append("Topics heavily covered by TechCrunch with no matching GitHub repos:\n")
477 + lines.append("Topics heavily covered by external press with no matching GitHub repos:\n")
478 for item in uncovered:
479 topic = item.get("topic", "unknown")
452 - articles = item.get("techcrunch_articles", [])
480 + articles = item.get("news_articles", item.get("techcrunch_articles", []))
481 article_refs = ", ".join(
482 f"[{a.get('title', 'article')}]({a.get('url', '')})"
483 for a in articles[:3]
@@ -459,7 +487,7 @@ def format_divergences(divergences: dict, *, reader_mode: bool = False) -> str:
487
488 if unpublicized:
489 lines.append("#### 🚀 Dev Activity Without Press Coverage")
462 - lines.append("GitHub repos/trends with no matching TechCrunch coverage:\n")
490 + lines.append("GitHub repos/trends with no matching external press coverage:\n")
491 for item in unpublicized:
492 topic = item.get("topic", "unknown")
493 repos = item.get("github_repos", [])
@@ -479,6 +507,55 @@ def format_divergences(divergences: dict, *, reader_mode: bool = False) -> str:
507 return "\n".join(lines)
508
509
510 +def _source_caveats(techcrunch_data: dict | None, correlation_data: dict | None) -> str:
511 + """Render concise partial-failure caveats from crawl/correlation metadata."""
512 + metadata: dict = {}
513 + if techcrunch_data:
514 + metadata = techcrunch_data.get("metadata", {})
515 + corr_sources = {}
516 + if correlation_data:
517 + corr_sources = correlation_data.get("metadata", {}).get("news_sources", {})
518 +
519 + requested = metadata.get("sources_requested") or corr_sources.get("sources_requested") or []
520 + succeeded = metadata.get("sources_succeeded") or corr_sources.get("sources_succeeded") or []
521 + failed = metadata.get("sources_failed") or corr_sources.get("sources_failed") or []
522 + errors = metadata.get("errors") or corr_sources.get("errors") or []
523 + if not requested and not failed:
524 + return ""
525 + lines = [
526 + "### Source Coverage",
527 + f"- Sources requested: {', '.join(requested) if requested else 'unknown'}",
528 + f"- Sources succeeded: {', '.join(succeeded) if succeeded else 'none'}",
529 + ]
530 + if failed:
531 + lines.append(f"- Partial crawl caveat: failed sources: {', '.join(failed)}")
532 + for error in errors[:3]:
533 + lines.append(
534 + f" - {error.get('source', 'unknown')}: "
535 + f"{error.get('error_class', 'error')} {error.get('error', '')}".strip()
536 + )
537 + return "\n".join(lines)
538 +
539 +
540 +def estimate_tokens(markdown: str) -> int:
541 + """Return a rough token estimate used for telemetry and hard budget checks."""
542 + return max(1, (len(markdown) + 3) // 4)
543 +
544 +
545 +def enforce_press_context_budget(markdown: str) -> str:
546 + """Keep press context below the documented token budget."""
547 + if estimate_tokens(markdown) <= PRESS_CONTEXT_TOKEN_BUDGET:
548 + return markdown
549 + budget_note = (
550 + "\n\n### Budget Notice\n"
551 + f"Press context truncated to ~{PRESS_CONTEXT_TOKEN_BUDGET} tokens; "
552 + "citations and source caveats above are prioritized.\n"
553 + )
554 + keep_chars = max(0, PRESS_CONTEXT_CHAR_BUDGET - len(budget_note))
555 + truncated = markdown[:keep_chars].rsplit("\n", 1)[0]
556 + return truncated + budget_note
557 +
558 +
559 def render_press_context(
560 techcrunch_data: dict | None,
561 correlation_data: dict | None,
@@ -539,10 +616,12 @@ def render_press_context(
616 ),
617 )
618
542 - top_n = 10 if reader_mode else None
619 + top_n = MAX_RENDERED_CORRELATIONS if reader_mode else None
620
621 # Render template
545 - rendered = template.replace("{date}", week)
622 + source_label = "External news"
623 + rendered = template.replace("TechCrunch", source_label)
624 + rendered = rendered.replace("{date}", week)
625 rendered = rendered.replace("{article_count}", str(article_count))
626 rendered = rendered.replace("{articles_list}", format_articles_list(articles))
627 rendered = rendered.replace("{correlation_count}", str(correlation_count))
@@ -570,7 +649,19 @@ def render_press_context(
649 if divergence_section:
650 rendered += "\n" + divergence_section
651
573 - return rendered
652 + caveats = _source_caveats(techcrunch_data, correlation_data)
653 + if caveats:
654 + rendered += "\n\n" + caveats
655 +
656 + rendered += (
657 + "\n\n### Press Context Telemetry\n"
658 + f"- token_estimate: {estimate_tokens(rendered)}\n"
659 + f"- token_budget: {PRESS_CONTEXT_TOKEN_BUDGET}\n"
660 + f"- article_limit: {MAX_RENDERED_ARTICLES}\n"
661 + f"- correlation_limit: {MAX_RENDERED_CORRELATIONS if reader_mode else 'unbounded-input'}\n"
662 + )
663 +
664 + return enforce_press_context_budget(rendered)
665
666
667 def resolve_paths(topic: str | None, week: str) -> tuple[Path, Path]:
scripts/techcrunch_crawler.py
+267 -15
@@ -12,6 +12,7 @@ Usage:
12 from __future__ import annotations
13
14 import argparse
15 +import hashlib
16 import ipaddress
17 import json
18 import re
@@ -33,7 +34,9 @@ from scripts.topic_paths import raw_dir
34 FEED_URL = "https://techcrunch.com/feed/"
35 DEFAULT_SOURCES_PATH = Path("config/external_news_sources.json")
36 DEFAULT_FETCH_TIMEOUT_SECONDS = 15
37 +DEFAULT_FETCH_RETRIES = 1
38 DEFAULT_MAX_WORKERS = 8
39 +CANONICAL_SCHEMA_VERSION = 2
40 APPROVED_FEED_HOSTS = frozenset({
41 "techcrunch.com",
42 "blogs.nvidia.com",
@@ -83,6 +86,11 @@ class NewsSourceConfig:
86 def __post_init__(self) -> None:
87 validate_feed_url(self.feed_url)
88
89 + @property
90 + def host(self) -> str:
91 + """Return the normalized feed host."""
92 + return (urlparse(self.feed_url).hostname or "").rstrip(".").lower()
93 +
94
95 def validate_feed_url(url: str) -> None:
96 """Validate an external RSS URL against the approved egress allowlist."""
@@ -144,6 +152,20 @@ def load_source_configs(path: Path = DEFAULT_SOURCES_PATH) -> list[NewsSourceCon
152 return sources
153
154
155 +def source_config_checksum(sources: list[NewsSourceConfig]) -> str:
156 + """Return a stable checksum for the effective source config."""
157 + canonical = [
158 + {
159 + "feed_url": source.feed_url,
160 + "name": source.name,
161 + "requests_per_minute": source.requests_per_minute,
162 + }
163 + for source in sorted(sources, key=lambda item: item.name)
164 + ]
165 + payload = json.dumps(canonical, sort_keys=True, separators=(",", ":"))
166 + return hashlib.sha256(payload.encode("utf-8")).hexdigest()
167 +
168 +
169 def iso_timestamp(value: datetime) -> str:
170 """Format datetime as ISO 8601 UTC string."""
171 return value.astimezone(UTC).isoformat().replace("+00:00", "Z")
@@ -228,7 +250,7 @@ def parse_published_date(entry: Any) -> datetime | None:
250
251 def fetch_feed(
252 url: str = FEED_URL,
231 - retries: int = 1,
253 + retries: int = DEFAULT_FETCH_RETRIES,
254 timeout: int = DEFAULT_FETCH_TIMEOUT_SECONDS,
255 ) -> Any:
256 """Fetch and parse RSS feed with bounded retries and an explicit timeout."""
@@ -240,6 +262,8 @@ def fetch_feed(
262 request = Request(url, headers={"User-Agent": "SquadScope RSS crawler"})
263 with urlopen(request, timeout=timeout) as response:
264 feed = feedparser.parse(response.read())
265 + setattr(feed, "squad_fetch_attempts", attempt + 1)
266 + setattr(feed, "squad_fetch_timeout_seconds", timeout)
267 except Exception:
268 if attempt < retries:
269 time.sleep(2)
@@ -250,6 +274,8 @@ def fetch_feed(
274 time.sleep(2)
275 continue
276 # Return partial result even on failure
277 + setattr(feed, "squad_fetch_attempts", attempt + 1)
278 + setattr(feed, "squad_fetch_timeout_seconds", timeout)
279 return feed
280 return feed
281 return feed # pragma: no cover
@@ -260,6 +286,8 @@ class NewsFeedSource:
286
287 def __init__(self, config: NewsSourceConfig) -> None:
288 self.config = config
289 + self.last_attempts = 0
290 + self.last_timeout_seconds = DEFAULT_FETCH_TIMEOUT_SECONDS
291
292 def get_name(self) -> str:
293 return self.config.name
@@ -277,6 +305,10 @@ class NewsFeedSource:
305 resolved_feed_url = feed_url or self.config.feed_url
306 validate_feed_url(resolved_feed_url)
307 feed = fetch_feed(resolved_feed_url)
308 + self.last_attempts = int(getattr(feed, "squad_fetch_attempts", 1))
309 + self.last_timeout_seconds = int(
310 + getattr(feed, "squad_fetch_timeout_seconds", DEFAULT_FETCH_TIMEOUT_SECONDS)
311 + )
312 articles: list[dict[str, Any]] = []
313
314 for entry in feed.entries:
@@ -332,33 +364,186 @@ def crawl_sources_parallel(
364 since: datetime,
365 until: datetime,
366 max_workers: int | None = None,
335 -) -> tuple[list[dict[str, Any]], list[dict[str, str]]]:
336 - """Crawl configured RSS sources concurrently and return articles plus errors."""
367 +) -> tuple[list[dict[str, Any]], list[dict[str, str]], list[dict[str, Any]]]:
368 + """Crawl configured RSS sources concurrently and return articles, errors, statuses."""
369 if not sources:
338 - return [], []
370 + return [], [], []
371
372 if max_workers is not None and max_workers < 1:
373 raise ValueError("--max-workers must be at least 1")
374 workers = min(max_workers or len(sources), len(sources), DEFAULT_MAX_WORKERS)
375 articles: list[dict[str, Any]] = []
376 errors: list[dict[str, str]] = []
377 + statuses: list[dict[str, Any]] = []
378 +
379 + def crawl_one(source: NewsSourceConfig) -> tuple[list[dict[str, Any]], dict[str, Any], dict[str, str] | None]:
380 + started = datetime.now(UTC)
381 + source_client = NewsFeedSource(source)
382 + status: dict[str, Any] = {
383 + "source": source.name,
384 + "host": source.host,
385 + "started_at": iso_timestamp(started),
386 + "timeout_seconds": DEFAULT_FETCH_TIMEOUT_SECONDS,
387 + "attempts": 0,
388 + "total_articles": 0,
389 + "relevant_articles": 0,
390 + "github_links_found": 0,
391 + "success": False,
392 + "error_class": "",
393 + "error_message": "",
394 + }
395 + try:
396 + source_articles = source_client.crawl(since, until)
397 + status["attempts"] = source_client.last_attempts or 1
398 + status["timeout_seconds"] = source_client.last_timeout_seconds
399 + status["total_articles"] = len(source_articles)
400 + status["relevant_articles"] = sum(
401 + 1 for article in source_articles
402 + if article.get("relevance_score", 0) >= 0.4
403 + )
404 + github_links: set[str] = set()
405 + for article in source_articles:
406 + github_links.update(article.get("github_links", []))
407 + status["github_links_found"] = len(github_links)
408 + status["success"] = True
409 + return source_articles, status, None
410 + except Exception as exc: # pragma: no cover - defensive around network/parser failures
411 + status["attempts"] = source_client.last_attempts or (DEFAULT_FETCH_RETRIES + 1)
412 + status["error_class"] = exc.__class__.__name__
413 + status["error_message"] = str(exc)
414 + error = {
415 + "source": source.name,
416 + "error_class": exc.__class__.__name__,
417 + "error": str(exc),
418 + }
419 + return [], status, error
420 + finally:
421 + ended = datetime.now(UTC)
422 + status["ended_at"] = iso_timestamp(ended)
423 + status["duration_seconds"] = round((ended - started).total_seconds(), 3)
424 +
425 with ThreadPoolExecutor(max_workers=workers) as executor:
426 futures = {
347 - executor.submit(NewsFeedSource(source).crawl, since, until): source
427 + executor.submit(crawl_one, source): source
428 for source in sources
429 }
430 for future in as_completed(futures):
351 - source = futures[future]
352 - try:
353 - articles.extend(future.result())
354 - except Exception as exc: # pragma: no cover - defensive around network/parser failures
355 - errors.append({"source": source.name, "error": str(exc)})
431 + source_articles, status, error = future.result()
432 + articles.extend(source_articles)
433 + statuses.append(status)
434 + if error:
435 + errors.append(error)
436
437 articles.sort(
358 - key=lambda article: (article.get("published_at", ""), article.get("source", "")),
438 + key=lambda article: (
439 + article.get("published_at", ""),
440 + article.get("source", ""),
441 + article.get("url", ""),
442 + article.get("title", ""),
443 + ),
444 + reverse=True,
445 + )
446 + statuses.sort(key=lambda status: status["source"])
447 + errors.sort(key=lambda error: error["source"])
448 + for status in statuses:
449 + state = "ok" if status["success"] else f"failed:{status['error_class']}"
450 + print(
451 + "[external-news] "
452 + f"{status['source']} host={status['host']} status={state} "
453 + f"duration={status['duration_seconds']:.3f}s attempts={status['attempts']} "
454 + f"articles={status['total_articles']} relevant={status['relevant_articles']} "
455 + f"github_links={status['github_links_found']}",
456 + file=sys.stderr,
457 + )
458 + return articles, errors, statuses
459 +
460 +
461 +def _normalized_article_url(url: str) -> str:
462 + """Normalize a URL for cross-source dedupe."""
463 + if not url:
464 + return ""
465 + parsed = urlparse(url.strip())
466 + host = (parsed.netloc or "").lower()
467 + path = parsed.path.rstrip("/")
468 + return f"{parsed.scheme.lower()}://{host}{path}"
469 +
470 +
471 +def dedupe_articles(articles: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], int]:
472 + """Deduplicate mirrored articles by URL while preserving provenance."""
473 + grouped: dict[str, dict[str, Any]] = {}
474 + duplicates = 0
475 + for article in sorted(
476 + articles,
477 + key=lambda item: (
478 + item.get("published_at", ""),
479 + item.get("source", ""),
480 + item.get("url", ""),
481 + item.get("title", ""),
482 + ),
483 + reverse=True,
484 + ):
485 + key = _normalized_article_url(str(article.get("url", "")))
486 + if not key:
487 + key = "|".join([
488 + str(article.get("source", "")),
489 + str(article.get("published_at", "")),
490 + str(article.get("title", "")).lower(),
491 + ])
492 + if key not in grouped:
493 + current = dict(article)
494 + current["sources"] = sorted({
495 + str(article.get("source", "")) or "unknown",
496 + *[str(s) for s in article.get("sources", [])],
497 + })
498 + grouped[key] = current
499 + continue
500 + duplicates += 1
501 + existing = grouped[key]
502 + existing_sources = set(existing.get("sources", []))
503 + existing_sources.add(str(article.get("source", "")) or "unknown")
504 + existing_sources.update(str(s) for s in article.get("sources", []))
505 + existing["sources"] = sorted(existing_sources)
506 + existing["relevance_score"] = max(
507 + float(existing.get("relevance_score", 0)),
508 + float(article.get("relevance_score", 0)),
509 + )
510 + existing_links = list(existing.get("github_links", []))
511 + for link in article.get("github_links", []):
512 + if link not in existing_links:
513 + existing_links.append(link)
514 + existing["github_links"] = existing_links
515 + deduped = list(grouped.values())
516 + deduped.sort(
517 + key=lambda article: (
518 + article.get("published_at", ""),
519 + article.get("source", ""),
520 + article.get("url", ""),
521 + article.get("title", ""),
522 + ),
523 reverse=True,
524 )
361 - return articles, errors
525 + return deduped, duplicates
526 +
527 +
528 +def _checksum_payload(output: dict[str, Any]) -> dict[str, Any]:
529 + """Return the deterministic subset covered by artifact_checksum."""
530 + metadata = dict(output.get("metadata", {}))
531 + metadata.pop("artifact_checksum", None)
532 + payload = dict(output)
533 + payload["metadata"] = metadata
534 + payload.pop("crawled_at", None)
535 + return payload
536 +
537 +
538 +def artifact_checksum(output: dict[str, Any]) -> str:
539 + """Return a stable checksum for the canonical artifact content."""
540 + payload = json.dumps(
541 + _checksum_payload(output),
542 + sort_keys=True,
543 + separators=(",", ":"),
544 + ensure_ascii=False,
545 + )
546 + return hashlib.sha256(payload.encode("utf-8")).hexdigest()
547
548
549 def build_output(
@@ -367,29 +552,88 @@ def build_output(
552 *,
553 source: str = "techcrunch",
554 source_count: int = 1,
555 + crawl_window: dict[str, str] | None = None,
556 + source_config_checksum_value: str | None = None,
557 + requested_sources: list[str] | None = None,
558 + source_statuses: list[dict[str, Any]] | None = None,
559 errors: list[dict[str, str]] | None = None,
560 ) -> dict[str, Any]:
561 """Build the final output structure with metadata."""
562 + articles, dedupe_count = dedupe_articles(articles)
563 relevant = [a for a in articles if a["relevance_score"] >= 0.4]
564 all_github_links = set()
565 for a in articles:
566 all_github_links.update(a.get("github_links", []))
567
568 + statuses = source_statuses or []
569 + succeeded = [
570 + str(status.get("source"))
571 + for status in statuses
572 + if status.get("success")
573 + ]
574 + failed = [
575 + str(status.get("source"))
576 + for status in statuses
577 + if not status.get("success")
578 + ]
579 + requested = requested_sources or sorted({
580 + str(article.get("source", source)) for article in articles
581 + }) or [source]
582 by_source = Counter(str(article.get("source", source)) for article in articles)
379 - return {
583 + output = {
584 + "schema_version": CANONICAL_SCHEMA_VERSION,
585 "week": week_slug(crawled_at),
586 "source": source,
587 "crawled_at": iso_timestamp(crawled_at),
588 + "crawl_window": crawl_window or {},
589 "articles": articles,
590 "metadata": {
591 "source_count": source_count,
592 + "source_config_checksum": source_config_checksum_value or "",
593 + "sources_requested": sorted(requested),
594 + "sources_succeeded": sorted(succeeded or requested),
595 + "sources_failed": sorted(failed),
596 + "source_status": sorted(statuses, key=lambda status: status["source"]),
597 "sources_with_articles": dict(sorted(by_source.items())),
598 "total_articles": len(articles),
599 "relevant_articles": len(relevant),
600 "github_links_found": len(all_github_links),
601 + "dedupe_count": dedupe_count,
602 "errors": errors or [],
603 },
604 }
605 + output["metadata"]["artifact_checksum"] = artifact_checksum(output)
606 + validate_canonical_output(output)
607 + return output
608 +
609 +
610 +def validate_canonical_output(output: dict[str, Any]) -> None:
611 + """Validate canonical external-news artifact shape."""
612 + if output.get("schema_version") != CANONICAL_SCHEMA_VERSION:
613 + raise ValueError("External news artifact has unsupported schema_version")
614 + if output.get("source") == "external_news" and not output.get("crawl_window"):
615 + raise ValueError("Canonical external news artifact requires crawl_window")
616 + metadata = output.get("metadata")
617 + if not isinstance(metadata, dict):
618 + raise ValueError("External news artifact requires metadata")
619 + required = {
620 + "source_config_checksum",
621 + "sources_requested",
622 + "sources_succeeded",
623 + "sources_failed",
624 + "source_status",
625 + "total_articles",
626 + "relevant_articles",
627 + "dedupe_count",
628 + "errors",
629 + "artifact_checksum",
630 + }
631 + missing = sorted(required - set(metadata))
632 + if missing:
633 + raise ValueError(f"External news artifact missing metadata keys: {missing}")
634 + expected_checksum = artifact_checksum(output)
635 + if metadata.get("artifact_checksum") != expected_checksum:
636 + raise ValueError("External news artifact checksum mismatch")
637
638
639 def main(argv: list[str] | None = None) -> int:
@@ -438,7 +682,7 @@ def main(argv: list[str] | None = None) -> int:
682 )
683
684 source_configs = load_source_configs(Path(args.sources))
441 - articles, errors = crawl_sources_parallel(
685 + articles, errors, statuses = crawl_sources_parallel(
686 source_configs, since=since, until=until, max_workers=args.max_workers
687 )
688 output = build_output(
@@ -446,6 +690,13 @@ def main(argv: list[str] | None = None) -> int:
690 crawled_at=now,
691 source="external_news",
692 source_count=len(source_configs),
693 + crawl_window={
694 + "since": iso_timestamp(since),
695 + "until": iso_timestamp(until),
696 + },
697 + source_config_checksum_value=source_config_checksum(source_configs),
698 + requested_sources=[source.name for source in source_configs],
699 + source_statuses=statuses,
700 errors=errors,
701 )
702
@@ -462,7 +713,8 @@ def main(argv: list[str] | None = None) -> int:
713
714 print(f"Crawled {output['metadata']['total_articles']} articles "
715 f"from {output['metadata']['source_count']} sources "
465 - f"({output['metadata']['relevant_articles']} relevant) → {out_path}")
716 + f"({output['metadata']['relevant_articles']} relevant, "
717 + f"{output['metadata']['dedupe_count']} deduped) → {out_path}")
718 return 0
719
720
tests/test_correlate.py
+55
@@ -11,6 +11,7 @@ from scripts.correlate import (
11 assess_hype_risk,
12 correlate_all,
13 correlate_repo,
14 + dedupe_articles,
15 extract_week_from_filename,
16 fuzzy_name_score,
17 has_temporal_spike,
@@ -54,13 +55,18 @@ def _article(
55 github_links: list[str] | None = None,
56 entities: list[str] | None = None,
57 categories: list[str] | None = None,
58 + source: str = "techcrunch",
59 + published_at: str = "2026-05-15T10:00:00Z",
60 ) -> dict:
61 return {
62 + "source": source,
63 "title": title,
64 "url": url,
65 + "published_at": published_at,
66 "github_links": github_links or [],
67 "entities": entities or [],
68 "categories": categories or [],
69 + "relevance_score": 0.8,
70 }
71
72
@@ -224,6 +230,8 @@ class TestCorrelateRepo:
230 assert result is not None
231 assert result["match_type"] == "direct_link"
232 assert result["correlation_confidence"] == 1.0
233 + assert result["correlation_strength"] == "strong"
234 + assert result["matched_article_details"][0]["source"] == "techcrunch"
235
236 def test_no_match_returns_none(self):
237 repo = _repo(owner="nobody", name="nothing", topics=[])
@@ -237,6 +245,25 @@ class TestCorrelateRepo:
245 assert result is not None
246 assert result["correlation_confidence"] == 1.0 # 0.8 + 0.2
247
248 + def test_category_only_match_is_weak(self):
249 + repo = _repo(owner="acme", name="tool", topics=["ai"], stars_gained=0)
250 + article = _article(categories=["ai"], entities=[])
251 + result = correlate_repo(repo, [article])
252 + assert result is not None
253 + assert result["match_type"] == "category"
254 + assert result["correlation_strength"] == "weak"
255 + assert result["correlation_confidence"] == 0.4
256 +
257 + def test_corroborated_category_match_stays_weak(self):
258 + repo = _repo(owner="acme", name="tool", topics=["ai"], stars_gained=50)
259 + articles = [
260 + _article(url="https://example.com/a", categories=["ai"], entities=[], source="alpha"),
261 + _article(url="https://example.com/b", categories=["ai"], entities=[], source="beta"),
262 + ]
263 + result = correlate_repo(repo, articles)
264 + assert result is not None
265 + assert result["correlation_strength"] == "weak"
266 +
267
268 # ---------------------------------------------------------------------------
269 # Integration: correlate_all
@@ -257,6 +284,8 @@ class TestCorrelateAll:
284 assert "nobody/unrelated" in result["uncorrelated_repos"]
285 assert result["metadata"]["repos_analyzed"] == 2
286 assert result["metadata"]["correlations_found"] == 1
287 + assert result["metadata"]["strong_correlations"] == 1
288 + assert result["metadata"]["weak_correlations"] == 0
289
290 def test_empty_articles(self):
291 repos = [_repo()]
@@ -264,6 +293,24 @@ class TestCorrelateAll:
293 assert result["correlations"] == []
294 assert len(result["uncorrelated_repos"]) == 1
295
296 + def test_cross_source_dedupe(self):
297 + articles = [
298 + _article(
299 + url="https://example.com/story/",
300 + source="alpha",
301 + github_links=["https://github.com/acme/cool-project"],
302 + ),
303 + _article(
304 + url="https://example.com/story",
305 + source="beta",
306 + github_links=["https://github.com/acme/cool-project"],
307 + ),
308 + ]
309 + result = correlate_all([_repo()], articles, "2026-W21")
310 + assert result["metadata"]["dedupe_count"] == 1
311 + details = result["correlations"][0]["matched_article_details"][0]
312 + assert details["sources"] == ["alpha", "beta"]
313 +
314
315 # ---------------------------------------------------------------------------
316 # Utility functions
@@ -285,6 +332,14 @@ class TestUtilities:
332 def test_fuzzy_name_score_empty(self):
333 assert fuzzy_name_score("", "something") == 0.0
334
335 + def test_dedupe_articles_preserves_provenance(self):
336 + articles, count = dedupe_articles([
337 + _article(url="https://example.com/a/", source="alpha"),
338 + _article(url="https://example.com/a", source="beta"),
339 + ])
340 + assert count == 1
341 + assert articles[0]["sources"] == ["alpha", "beta"]
342 +
343
344 # ---------------------------------------------------------------------------
345 # CLI: main() repo loading from crawl output format
tests/test_render_press_context.py
+1 -1
@@ -281,7 +281,7 @@ class TestFormatDivergencesReaderMode:
281 def test_reader_mode_has_narrative(self):
282 result = format_divergences(self._divergences(), reader_mode=True)
283 # Narrative prose paragraphs, no raw bullet lists
284 - assert "TechCrunch heavily covered" in result
284 + assert "External press heavily covered" in result
285 assert "Developer activity this week" in result
286 assert "- **quantum-computing**:" not in result
287 assert "- **wasm-tooling**:" not in result
tests/test_techcrunch_crawler.py
+104 -1
@@ -12,11 +12,13 @@ import pytest
12 from scripts.techcrunch_crawler import (
13 DEFAULT_SOURCES_PATH,
14 DEFAULT_FETCH_TIMEOUT_SECONDS,
15 + DEFAULT_FETCH_RETRIES,
16 NewsSourceConfig,
17 TechCrunchSource,
18 build_output,
19 compute_relevance_score,
20 crawl_sources_parallel,
21 + dedupe_articles,
22 extract_entities,
23 extract_github_urls,
24 fetch_feed,
@@ -388,7 +390,7 @@ class TestExternalNewsSources:
390 NewsSourceConfig("beta", "https://github.blog/feed/"),
391 ]
392 with patch("scripts.techcrunch_crawler.fetch_feed", side_effect=fake_fetch):
391 - articles, errors = crawl_sources_parallel(
393 + articles, errors, statuses = crawl_sources_parallel(
394 sources,
395 since=datetime(2026, 5, 10, tzinfo=UTC),
396 until=datetime(2026, 5, 20, tzinfo=UTC),
@@ -396,6 +398,8 @@ class TestExternalNewsSources:
398 )
399
400 assert errors == []
401 + assert {status["source"] for status in statuses} == {"alpha", "beta"}
402 + assert all(status["success"] for status in statuses)
403 assert [article["source"] for article in articles] == ["beta", "alpha"]
404 assert {article["title"] for article in articles} == {
405 "Alpha AI framework",
@@ -426,10 +430,109 @@ class TestExternalNewsSources:
430 crawled_at=now,
431 source="external_news",
432 source_count=2,
433 + crawl_window={
434 + "since": "2026-05-12T00:00:00Z",
435 + "until": "2026-05-19T00:00:00Z",
436 + },
437 + source_config_checksum_value="abc123",
438 + requested_sources=["alpha", "beta", "gamma"],
439 + source_statuses=[
440 + {
441 + "source": "alpha",
442 + "host": "techcrunch.com",
443 + "success": True,
444 + "attempts": 1,
445 + "timeout_seconds": 15,
446 + "total_articles": 1,
447 + "relevant_articles": 1,
448 + "github_links_found": 0,
449 + "started_at": "2026-05-19T10:00:00Z",
450 + "ended_at": "2026-05-19T10:00:01Z",
451 + "duration_seconds": 1.0,
452 + "error_class": "",
453 + "error_message": "",
454 + },
455 + {
456 + "source": "beta",
457 + "host": "github.blog",
458 + "success": True,
459 + "attempts": 1,
460 + "timeout_seconds": 15,
461 + "total_articles": 1,
462 + "relevant_articles": 1,
463 + "github_links_found": 0,
464 + "started_at": "2026-05-19T10:00:00Z",
465 + "ended_at": "2026-05-19T10:00:01Z",
466 + "duration_seconds": 1.0,
467 + "error_class": "",
468 + "error_message": "",
469 + },
470 + {
471 + "source": "gamma",
472 + "host": "example.com",
473 + "success": False,
474 + "attempts": 2,
475 + "timeout_seconds": 15,
476 + "total_articles": 0,
477 + "relevant_articles": 0,
478 + "github_links_found": 0,
479 + "started_at": "2026-05-19T10:00:00Z",
480 + "ended_at": "2026-05-19T10:00:01Z",
481 + "duration_seconds": 1.0,
482 + "error_class": "TimeoutError",
483 + "error_message": "timeout",
484 + },
485 + ],
486 errors=[{"source": "gamma", "error": "timeout"}],
487 )
488
489 + assert output["schema_version"] == 2
490 assert output["source"] == "external_news"
491 assert output["metadata"]["source_count"] == 2
492 + assert output["metadata"]["source_config_checksum"] == "abc123"
493 + assert output["metadata"]["sources_requested"] == ["alpha", "beta", "gamma"]
494 + assert output["metadata"]["sources_succeeded"] == ["alpha", "beta"]
495 + assert output["metadata"]["sources_failed"] == ["gamma"]
496 + assert output["metadata"]["artifact_checksum"]
497 assert output["metadata"]["sources_with_articles"] == {"alpha": 1, "beta": 1}
498 assert output["metadata"]["errors"] == [{"source": "gamma", "error": "timeout"}]
499 +
500 + def test_dedupe_articles_preserves_sources(self):
501 + articles, deduped = dedupe_articles([
502 + {
503 + "source": "alpha",
504 + "title": "Same story",
505 + "url": "https://example.com/story/",
506 + "published_at": "2026-05-15T10:00:00Z",
507 + "github_links": ["https://github.com/a/b"],
508 + "relevance_score": 0.4,
509 + },
510 + {
511 + "source": "beta",
512 + "title": "Same story mirror",
513 + "url": "https://example.com/story",
514 + "published_at": "2026-05-15T10:00:00Z",
515 + "github_links": ["https://github.com/c/d"],
516 + "relevance_score": 0.8,
517 + },
518 + ])
519 +
520 + assert deduped == 1
521 + assert len(articles) == 1
522 + assert articles[0]["sources"] == ["alpha", "beta"]
523 + assert articles[0]["relevance_score"] == 0.8
524 +
525 + def test_failed_source_reports_bounded_retry_attempts(self):
526 + source = NewsSourceConfig("alpha", "https://techcrunch.com/feed/")
527 + with patch("scripts.techcrunch_crawler.fetch_feed", side_effect=TimeoutError("boom")):
528 + articles, errors, statuses = crawl_sources_parallel(
529 + [source],
530 + since=datetime(2026, 5, 10, tzinfo=UTC),
531 + until=datetime(2026, 5, 20, tzinfo=UTC),
532 + max_workers=1,
533 + )
534 +
535 + assert articles == []
536 + assert errors[0]["error_class"] == "TimeoutError"
537 + assert statuses[0]["attempts"] == DEFAULT_FETCH_RETRIES + 1
538 + assert statuses[0]["success"] is False