feat: pre-flight token cost estimation (#56) (#90)
* docs: add TechCrunch RSS integration PRD Proposes TechCrunch RSS as SquadScope's first non-GitHub data source, enabling cross-source trend correlation to distinguish press-driven hype from organic developer momentum. Key points: - Implements Decision #7 crawler plugin architecture - Near-zero incremental cost (<$0.01/week) - Phased approach: RSS crawl → correlation engine → editorial integration - Editorial stance: press data is a correlation signal, not content to republish Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(squad): log session — PR #54 merged, TechCrunch PRD opened - Merged 9 decision inbox files into decisions.md (now 27475 bytes) - Updated Leela history: PR #54 cost estimation merged - Updated Farnsworth history: PR #55 TechCrunch integration opened - Added 8 new decisions: Analyze job integration, analysis spec, generate/deploy, run counter, reskill retrospective, cost estimation, topic channels, TechCrunch RSS Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: record Leela's review of PR #55 (rejected — empty branch) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(squad): log review session — PR #55 rejected, Bender revised Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: decompose PRDs into milestoned issues, move to processed - Created milestones v0.5 through v0.9 - Decomposed 3 PRDs into 34 GitHub issues with squad labels - Moved processed PRDs to docs/processed/ - Adopted milestone-based workflow per user directive Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(squad): log milestone decomposition session Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
May 19, 2026 at 15:22 UTC
0d44d22fcb2a7e71fd38e017ac0ef1305ee220a7
18 files changed
+612
-259
.squad/agents/amy/history.md
+1
@@ -23,3 +23,4 @@
23
- **2026-05-18T10:59:10Z:** Issues #3 and #4 complete. Commits c46beab, 05372c3. Ready for Issue #6+. User directive: all future work follows branch → PR → Review → Merge workflow (no direct commits to main).
24
- **2026-05-18T13:20:07.067+02:00:** The weekly pipeline now needs three repo-backed publication stages after crawling: analyzed markdown in `data/analyzed/`, generated Hugo content in `content/weekly/YYYY/WNN.md`, and a Pages artifact built with Hugo 0.161.1 plus Pagefind.
25
- **2026-05-18T13:20:07.067+02:00:** `scripts/generate_content.py` can safely normalize analyzer output into archetype-compatible Hugo frontmatter by dropping analyzer-only fields (`year`, `quality_score`) and keeping the analysis body intact for publication.
26
+- **2026-05-19T15:08:00Z:** Leela milestone decomposition complete. Issues assigned to v0.5–v0.9 milestones. Scribe logged orchestration and merged decision. Your assigned v0.5 frontend and content issues are ready. See `.squad/orchestration-log/2026-05-19T15-08-leela.md` for full decomposition outcome.
.squad/agents/bender/history.md
+3
@@ -26,3 +26,6 @@
26
- **2026-05-18T10:50:21Z:** PR #27 (Issue #8 crawl workflow) review complete. All 7 Copilot findings addressed (abb2a80). Workflow structure: restore `data/cache/` artifact, run `scripts/crawl.py`, upload `crawl-output` + new cache. Permissions `actions: read` + `contents: write`. Ready for merge. Downstream phases can depend on cache artifacts.
27
- **2026-05-18T15:22:25.067+02:00:** Issue #15 should increment `.squad/run-counter.txt` inside the `crawl` commit step after syncing the default branch, so the workflow reads the latest persisted counter, writes the incremented value atomically, and commits it alongside crawl artifacts.
28
- **2026-05-18T15:22:25.067+02:00:** The reskill trigger can stay lightweight for now: a `reskill-check` job only needs the persisted counter from `crawl`, and a gated placeholder `reskill` job can scaffold `.squad/skills/` and `.squad/reskill/` until Issue #14 adds real retrospective outputs and `.squad/` persistence.
29
+- **2026-05-19T11:59:28Z:** Took over PR #55 from Farnsworth (locked out after reviewer rejection). Wrote `docs/PRD-techcrunch-integration.md` — TechCrunch RSS integration PRD positioned as enrichment signal (not primary source). Key design decisions: 3-stage filtering pipeline (category → keyword → entity extraction) to reduce 250 articles/week to ~25; weekly batch alignment instead of real-time; honest 5–15% correlation hit rate with clear failure criteria; delta model (hype vs traction) as the value proposition; implements Decision #7 plugin architecture with `TechCrunchSource` class; negligible cost impact ($0.10–$0.21/year tokens). Force-pushed clean branch, updated PR #55 description, ready for next review.
30
+- **2026-05-19T15:08:00Z:** Leela milestone decomposition complete. Issues assigned to v0.5–v0.9 milestones. Scribe logged orchestration and merged decision. Your assigned v0.5 crawler enhancement issues are ready to pull and start. See `.squad/orchestration-log/2026-05-19T15-08-leela.md` for full decomposition outcome.
31
+
.squad/agents/farnsworth/history.md
+4
@@ -20,3 +20,7 @@
20
- **2026-05-18T12:07:20.778+02:00:** Keep the reader-facing weekly summary in five stable H2 sections, but require labeled `Signal`, `Noise`, and `Gaps` subsections so the editorial lens remains explicit and machine-checkable.
21
- **2026-05-18T13:20:07.067+02:00:** Weekly analysis prose should render repo mentions as explicit GitHub markdown links, and the current raw crawl artifact exposes those repo page URLs under `url` rather than `html_url`, so analyzer/generator prompts should require link formatting without assuming a different field name.
22
- **2026-05-18T15:22:25.067+02:00:** The learning loop only becomes real when learned state is both persisted and injected back into the next weekly prompt. Reskill reports need recent summaries, snapshot hindsight, and quality trend context; the weekly analyzer must read `wisdom.md` plus `.squad/skills/` at prompt-render time so lessons change future judgment instead of sitting idle.
23
+- **2026-05-19T11:48:44.543Z:** TechCrunch RSS integration PRD authored (`docs/PRD-techcrunch-integration.md`). Key architectural insight: non-GitHub sources serve as *correlation signals* for editorial judgment, not content to republish. The value is in the delta between press narrative and actual developer activity. This is the first concrete use of Decision #7's crawler plugin architecture. RSS feeds are trivially cheap (~$0.00/week) and require no API auth, making them ideal first candidates for multi-source expansion before more complex APIs (HN, Reddit).
24
+- **2026-05-19T11:48:44.543Z:** PR #55 opened with TechCrunch integration proposal. Analysis specification decisions finalized and merged into `.squad/decisions.md`: frontmatter superset contract, five stable H2 sections, required Signal/Noise/Gaps subsections, honest degradation of trending when momentum data incomplete. Learned state injection framework (wisdom.md + skills/) integrated into decision log; weekly analyzer will read these at prompt-render time starting Phase 2.
25
+- **2026-05-19T11:55:46Z:** Self-review of TechCrunch RSS PRD (PR #55) completed. Key findings: (1) PRD file missing from branch — blocker. (2) Correlation hit rate realistically 5–15%, not the implicit "most articles correlate" assumption. Name-matching across TC articles and GitHub repos requires entity resolution, not string matching. (3) Filtering is underspecified — no keywords, no category selection, no confidence tiers defined. (4) Temporal mismatch between real-time RSS and weekly analysis means correlations are retrospective explanations, not predictive signals — the "prediction enhancement" phase is premature. (5) No success criteria defined to evaluate whether integration justifies its complexity. Lesson: excitement about architectural patterns (plugin system) must not override skepticism about editorial value-add. The right question isn't "can we?" but "should we, and will it measurably improve output?"
26
+- **2026-05-19T15:08:00Z:** Leela milestone decomposition complete. Issues assigned to v0.5–v0.9 milestones. Scribe logged orchestration and merged decision. Your assigned v0.5 analysis and synthesis issues are ready. See `.squad/orchestration-log/2026-05-19T15-08-leela.md` for full decomposition outcome.
.squad/agents/fry/history.md
+1
@@ -16,3 +16,4 @@
16
17
- **2026-05-18T10:59:10.800+02:00:** The PaperMod theme in this repo needs Hugo `v0.146.0+`; dry-run validation failed on the system `v0.123.7` binary and only passed after switching to a newer Hugo build.
18
- **2026-05-18T10:59:10.800+02:00:** Real crawler output is publishable with manual curation, but `trending_repos` needs historical snapshots and the new-repo filter still lets exploit/cheat noise through.
19
+- **2026-05-19T15:08:00Z:** Leela milestone decomposition complete. Issues assigned to v0.5–v0.9 milestones. Scribe logged orchestration and merged decision. Your assigned v0.5 infrastructure and validation issues are ready. See `.squad/orchestration-log/2026-05-19T15-08-leela.md` for full decomposition outcome.
.squad/agents/leela/history.md
+40
@@ -110,3 +110,43 @@
110
4. Token budget with tiered degradation
111
- **Pricing model hypothesis (pending OQ6 validation):** GitHub Models API and Copilot CLI are assumed to use the same per-token rates, with the difference being auth mechanism and agentic capabilities rather than cost per token. This assumption needs empirical validation — see PRD OQ6.
112
- **Open risk:** Whether Copilot CLI transcript exposes actual token usage (needed for monitoring)
113
+
114
+### 2026-05-19T11:48:44.543Z — PR #54 Merged (Cost Estimation)
115
+
116
+- **Status:** All 4 review comments resolved and PR squash-merged to main
117
+- **Outcome:** Cost estimation framework approved for Phase A implementation
118
+- **Integration:** Cost tracking issues will be added to Phase A backlog
119
+- **Team note:** Cost analysis findings established sustainability baseline; no immediate budget action required but monitoring framework is essential for future growth planning
120
+
121
+### 2026-05-19T11:55:46.116Z — PR #55 Review (TechCrunch RSS PRD)
122
+
123
+- **Verdict:** REJECTED (request-changes, recorded as comment due to self-author constraint)
124
+- **Reason:** PR title/description promises a TechCrunch RSS integration PRD but the branch contains zero TechCrunch-related content. Actual diff is stale cost-estimation work already merged via PR #54. Branch has merge conflicts against main.
125
+- **Architectural observation:** The PR description's editorial framing (cross-source correlation to distinguish press hype from organic momentum) is sound and aligned with Decision #7's plugin architecture. When the actual PRD arrives, key review criteria will be: plugin interface compliance, overlap with topic-channels PRD, and incremental cost impact.
126
+- **Recurring pattern:** This is another instance of a PR being opened before the deliverable is committed — need team discipline on "commit first, then open PR."
127
+
128
+### 2026-05-19T11:59:28Z — PR #55 Resolved by Bender (TechCrunch RSS PRD Revision)
129
+
130
+- **Handoff:** Rejected PR #55 passed to Bender for revision (Farnsworth locked out per protocol)
131
+- **Outcome:** Bender rewrote PRD, rebased branch, committed deliverable, updated PR description
132
+- **Key decision captured:** TechCrunch as enrichment signal (5–15% correlation hit rate), not primary source
133
+- **Status:** PR #55 ready for next review cycle
134
+- **Team learning:** Rollback/rejection-to-revision cycle worked as designed — rejector (Leela) transitioned ownership cleanly, locked reviewer enabled handoff without conflicts
135
+
136
+### 2026-05-19T14:51:48.593+02:00 — PR #55 Re-review (TechCrunch RSS PRD)
137
+
138
+- **Verdict:** APPROVED (recorded as comment due to GitHub self-author constraint)
139
+- **Revision quality:** Excellent. Bender delivered a complete 443-line PRD that addresses all original rejection reasons.
140
+- **Key strengths:** Honest 5–15% correlation rate, graceful zero-noise degradation, Decision #7 plugin compliance, explicit failure criteria with removal triggers, phased rollout with exit gates.
141
+- **Minor suggestions (non-blocking):** Spike OQ1 (RSS content depth) before Phase 1; consider `correlate.py` placement at `scripts/` root since it's a cross-source concern; add URL-based dedup for mid-week article republishes.
142
+- **Pattern confirmed:** The reject → reassign → revise cycle works. Bender's revision was materially better than a "fix the branch" patch — it was a ground-up rewrite with proper editorial framing.
143
+- **Operational note:** GitHub still blocks formal approve/request-changes on self-authored PRs. Approval recorded via PR comment.
144
+
145
+### 2026-05-19T14:59:57+02:00 — PRD Decomposition into Milestones
146
+
147
+- **Milestone structure adopted:** v0.5 (Cost Visibility, 3 issues), v0.6 (Topic Channels Foundation, 6 issues), v0.7 (Learning & Predictions, 9 issues), v0.8 (Cross-Source Intelligence, 7 issues), v0.9 (Cost Optimization & Polish, 9 issues)
148
+- **Total issues created:** 34 issues across 5 milestones (issues #56–#89)
149
+- **PRDs processed:** 3 PRDs moved to docs/processed/ (cost-estimation, topic-channels, techcrunch-integration)
150
+- **Workflow change:** Milestone-based versioning adopted per user directive. PRDs → issues → milestones → docs/processed/
151
+- **Dependencies respected:** TechCrunch (v0.8) follows topic-channels foundation (v0.6); cost optimization (v0.9) follows cost visibility (v0.5)
152
+- **Label convention:** All issues carry `squad` + `squad:{agent}` labels for routing
.squad/decisions.md
+247
@@ -386,6 +386,253 @@ Treat README lookups as a degradable signal instead of a hard-stop path. The cra
386
387
---
388
389
+## Analyze Job Integration & Quality Gate (2026-05-18)
390
+
391
+**Issue:** #10 — Integrate Actions analyze job with Copilot path and reviewer gate
392
+**Author:** Bender (Crawler agent)
393
+**Status:** Approved for Phase 2 implementation
394
+**Date:** 2026-05-18T13:05:53.678+02:00
395
+
396
+### Decision
397
+
398
+Extend `.github/workflows/crawl-and-publish.yml` with an `analyze` job that runs after `crawl` and enforces an automated quality gate before downstream publish steps.
399
+
400
+**Stage handoff artifacts:**
401
+- `raw-data` for crawl → analyze
402
+- `analyzed-data` for analyze → generate
403
+
404
+**Analysis paths:**
405
+1. Primary: Standalone Copilot CLI with `permissions.copilot-requests: write` and `COPILOT_GH_TOKEN`
406
+2. Fallback: `scripts/analyze_fallback.py` using GitHub Models API with `permissions.models: read`
407
+
408
+**Quality Gate Contract** — workflow must fail if any of the following are false:
409
+- YAML frontmatter exists with exact required keys
410
+- `quality_score` is an integer ≥ 60
411
+- Required H2/H3 sections appear in documented order
412
+- Body word count ≥ 200
413
+- Output does not leak raw JSON, traceback, or placeholder content
414
+
415
+**Implications:**
416
+- Generate jobs can safely consume `analyzed-data` without inspecting raw crawl artifacts
417
+- Copilot failures do not block immediately; GitHub Models fallback preserves publishability
418
+- Reviewer-gate failures stop low-quality summaries before downstream stages
419
+
420
+---
421
+
422
+## Analysis Output Specification (2026-05-18)
423
+
424
+**Issue:** #9 — Define weekly analysis contract between crawler output and site generator
425
+**Author:** Farnsworth (Analyst)
426
+**Status:** Approved for Phase 2 implementation
427
+**Date:** 2026-05-18T12:07:20.778+02:00
428
+
429
+### Decisions
430
+
431
+1. **Analyzer output frontmatter is a superset contract.**
432
+ - Required fields: `title`, `date`, `week`, `year`, `tags`, `categories`, `repos_featured`, `stars_tracked`, `top_repo`, `quality_score`, `summary`
433
+
434
+2. **Reader-facing structure: five stable H2 sections** (in order):
435
+ - `Notable New Repositories`
436
+ - `Trending This Week (Stars Gained)`
437
+ - `Trend Analysis` (with required `### Signal` and `### Noise` subsections)
438
+ - `What's Missing` (with required `### Gaps` subsection)
439
+ - `Conclusion`
440
+
441
+3. **Trending must degrade honestly when momentum data is incomplete.**
442
+ - If `stars_gained` is absent or null, summary must say the section is directional, not a true momentum leaderboard
443
+
444
+4. **Analyzer input schema: strict on core fields, tolerant on metadata.**
445
+ - Required: week slug, crawl timestamp, new/trending repo arrays, top topics
446
+ - Optional: `partial_failures`, `filter_summary`, `snapshot_path`
447
+
448
+---
449
+
450
+## Generate & Deploy Workflow (2026-05-18)
451
+
452
+**Issue:** #11 — Implement generate-and-deploy workflow for GitHub Pages
453
+**Author:** Amy (Generator agent)
454
+**Status:** Approved for Phase 2 implementation
455
+**Date:** 2026-05-18T13:20:07.067+02:00
456
+
457
+### Decision
458
+
459
+Keep `.github/workflows/deploy-site.yml` for push-to-main deployments. Weekly automation lives in `.github/workflows/crawl-and-publish.yml` end-to-end (crawl → analyze → generate → deploy).
460
+
461
+**Generate stage:**
462
+1. Read `data/analyzed/YYYY-WNN-summary.md`
463
+2. Write Hugo page to `content/weekly/YYYY/WNN.md` with archetype-compatible frontmatter
464
+3. Commit back to default branch before Pages build so future archive builds retain previously published content
465
+
466
+**Deploy:**
467
+- Build with Hugo 0.161.1 + Pagefind
468
+- Deploy with `actions/deploy-pages@v4` under `github-pages` environment
469
+
470
+---
471
+
472
+## Run Counter & Reskill Trigger (2026-05-18)
473
+
474
+**Issue:** #15 — Add run counter persistence and every-fifth-run reskill trigger
475
+**Author:** Bender (Crawler agent)
476
+**Status:** Approved for Phase 1B implementation
477
+**Date:** 2026-05-18T15:22:25.067+02:00
478
+
479
+### Decisions
480
+
481
+1. **Create `.squad/run-counter.txt`** initialized to `0`
482
+2. **Increment counter** in `crawl` job's git commit step after syncing default branch, then commit `.squad/run-counter.txt` with `data/raw/` and `data/snapshots/`
483
+3. **Add `reskill-check` job** that reads persisted counter and exposes `should_reskill` for downstream jobs
484
+4. **Add placeholder `reskill` job** that logs the trigger and scaffolds `.squad/skills/` and `.squad/reskill/` until Issue #14 adds full retrospective implementation
485
+
486
+**Why:** Reading the counter only after syncing `origin/main` keeps the increment tied to latest persisted state. Committing together ensures survival between weekly runs. Splitting `reskill-check` from `reskill` keeps trigger logic auditable.
487
+
488
+---
489
+
490
+## Reskill Retrospective & Learning State (2026-05-18)
491
+
492
+**Issue:** #14 — Reskill retrospective, learned-state injection, and quality trend tracking
493
+**Author:** Farnsworth (Analyst)
494
+**Status:** Approved for Phase 2 implementation
495
+**Date:** 2026-05-18T15:22:25.067+02:00
496
+
497
+### Decisions
498
+
499
+1. **Reskill context** from latest analyzer evidence, not generic squad history:
500
+ - Inputs: last ~5 `data/analyzed/*-summary.md` files, `data/snapshots/` hindsight, `wisdom.md`, learned skills, quality trend report
501
+ - Why: gives retrospective concrete calibration points and closes gap findings
502
+
503
+2. **Learned state flows back** into weekly analyzer prompt:
504
+ - Inject `.squad/identity/wisdom.md` into `{{WISDOM}}` placeholder
505
+ - Inject concatenated markdown from `.squad/skills/` into `{{SKILLS}}` placeholder
506
+ - Why: without prompt injection, learning artifacts exist but never influence future analysis
507
+
508
+3. **Quality trend tracking** is first-class reskill input:
509
+ - `scripts/track_quality.py` reads `quality_score` from analyzed summaries, produces markdown trend report
510
+ - Why: squad needs lightweight longitudinal measure of editorial quality improvement
511
+
512
+4. **Reskill outputs in persistent squad state:**
513
+ - `.squad/reskill/` for weekly retrospective reports
514
+ - `.squad/skills/` for extracted reusable patterns
515
+ - Both committed to git (not ephemeral workflow output)
516
+
517
+---
518
+
519
+## Cost Estimation & Budget Controls (2026-05-19)
520
+
521
+**Issue:** #17 — Cost estimation framework for SquadScope
522
+**Author:** Leela (Lead/Architect)
523
+**Status:** Proposed
524
+**PRD:** docs/PRD-cost-estimation.md
525
+**Date:** 2026-05-19T05:17:53.102+02:00
526
+
527
+### Summary
528
+
529
+Current SquadScope cost under token-based billing: ~$0.30/week (~$16/year), well within Copilot Pro's 300 credits/month allowance. However, proactive monitoring and budget controls needed before context growth or model upgrades change the picture.
530
+
531
+### Decisions
532
+
533
+1. **Accept current cost profile as sustainable** — $16/year is economically trivial; no immediate model downgrade required
534
+2. **Implement token usage tracking (Phase A)** — Add `scripts/track_token_usage.py` and `data/metrics/token-usage.jsonl` to establish baselines before optimizing
535
+3. **Set budget alert thresholds:**
536
+ - Warn at $0.50/run
537
+ - Fail at $1.00/run
538
+ - Email alert at $5/month cumulative
539
+ - Auto-switch to cheaper model at $10/month cumulative
540
+4. **Defer raw JSON pre-processing** — 40-60% savings significant but adds pipeline complexity; implement only if costs grow beyond $30/year
541
+5. **Wisdom.md cap at 5 KB** — Reskill should retire obsolete heuristics, not only append
542
+
543
+**Rationale:** Dominant cost driver (raw JSON at 86K tokens) is stable and bounded by crawl scope. Growth comes from wisdom/skills/history accumulation, which is slow. Premature optimization would add complexity without meaningful savings at current scale.
544
+
545
+**Risks:**
546
+- OQ5/OQ6: Billing mechanics for Copilot CLI vs Models API may differ in ways not yet visible
547
+- Credit exhaustion mid-month would disrupt weekly pipeline if no degradation path exists
548
+
549
+---
550
+
551
+## Topic-Specific News Channels Architecture (2026-05-18)
552
+
553
+**Issue:** #16 — Topic-specific news channels architecture
554
+**Author:** Leela (Lead/Architect)
555
+**Status:** Proposed
556
+**PRD:** docs/PRD-topic-channels.md
557
+**PR:** #39
558
+**Date:** 2026-05-18T13:20:07.067+02:00
559
+
560
+### Key Architectural Decisions
561
+
562
+**Feature First, Not Platform:** Generalize SquadScope into topic channels by adding topic namespace to existing pipeline. No new platform, no new repo structure. Same codebase, configured differently.
563
+
564
+**Multi-Instance Single-Topic (v1):** One fork/config per topic with isolated learning, own `squadscope.topic.yml`, own Actions schedule, own GitHub Pages site. Multi-topic single-instance is v2.
565
+
566
+**Topic Config as Single Source of Truth:** `squadscope.topic.yml` controls:
567
+- Crawler queries
568
+- Scoring weights and thresholds
569
+- Analysis tone and audience
570
+- Learning state paths
571
+- Quality criteria
572
+
573
+**Scoring Pipeline (New Stage):** GitHub topic search is noisy. New `scripts/score_repos.py` between crawl and analyze, scoring repos 0-100 on relevance/momentum/language/noise/recency. Only repos ≥40 reach analysis.
574
+
575
+**Per-Topic Learning Isolation:**
576
+- `topics/{id}/wisdom.md` — domain-specific heuristics
577
+- `topics/{id}/skills/` — extracted patterns
578
+- `topics/{id}/predictions.jsonl` — prediction ledger
579
+- `topics/{id}/scorecards/` — hindsight validation results
580
+- No cross-topic contamination
581
+
582
+**Prediction Ledger with Hindsight Validation:** Every analysis appends machine-readable predictions to `predictions.jsonl`. Four weeks later, `scripts/validate_predictions.py` scores them against actual outcomes (star deltas, fork growth). Scorecards feed into reskill.
583
+
584
+**Topic Quality Criteria:**
585
+- Minimum N repos/week passing filters
586
+- Maximum false positive rate
587
+- Minimum genuinely significant repos per issue
588
+
589
+### Implications
590
+
591
+- Crawler must read config instead of hardcoded queries
592
+- Analysis prompt becomes a template with injection points
593
+- Hugo gains topic taxonomy and per-topic RSS
594
+- All data paths gain `{topic_id}/` prefix
595
+- Reskill reads per-topic state
596
+
597
+### Open for Discussion
598
+
599
+- Should enrichment signals (forks, contributors) be in v1 scorer or deferred?
600
+- Prediction confidence: fixed initial values or prompt-generated?
601
+- Topic config in root vs `topics/` directory?
602
+
603
+---
604
+
605
+## TechCrunch RSS as First Non-GitHub Data Source (2026-05-19)
606
+
607
+**Issue:** TechCrunch integration as first non-GitHub crawler plugin
608
+**Author:** Farnsworth (Analyst)
609
+**Status:** Proposed
610
+**PRD:** docs/PRD-techcrunch-integration.md
611
+**Date:** 2026-05-19T11:48:44.543Z
612
+
613
+### Decision
614
+
615
+Add TechCrunch RSS (`https://techcrunch.com/feed/`) as SquadScope's first non-GitHub data source, implementing Decision #7's crawler plugin architecture.
616
+
617
+**Rationale:**
618
+1. Cross-source correlation enables hype detection (press-driven vs. organic growth)
619
+2. Near-zero cost and complexity (public RSS, no auth, no rate limits)
620
+3. Directly implements the `DataSource` plugin pattern already approved
621
+4. Enriches editorial judgment without changing SquadScope's voice or pipeline structure
622
+
623
+**Impact:**
624
+- **Bender:** Implements `TechCrunchSource` crawler plugin
625
+- **Farnsworth:** Analyzer prompt gains press-context block; labels repos as press-correlated or organic
626
+- **Amy:** Optional correlation badge in Hugo templates
627
+- **Leela:** No architectural changes needed; plugin arch already designed for this
628
+
629
+### Open for Team Input
630
+
631
+- Should we start with full feed or category-specific feeds?
632
+- Correlation annotations: reader-visible or internal-only?
633
+
634
+---
635
+
636
## Governance
637
638
- All meaningful changes require team consensus
.squad/decisions/decisions.md
+15
@@ -71,3 +71,18 @@
71
- **Issue:** `data/analyzed/2026-W21-summary.md` is stored in the analyzer contract path but does not follow the approved Analyze → Generate section contract (`Signal`, `Noise`, `Gaps`).
72
- **Required follow-up:** Either align the analyzed artifact to the approved contract or move the manual validation artifact out of `data/analyzed/` so the repository does not adopt the wrong schema by accident.
73
- **Status:** Identified in PR #25 review (now merged) — Phase 2 action item.
74
+
75
+## 2026-05-19: TechCrunch RSS as Enrichment Signal (PR #55)
76
+
77
+- **Owner:** Bender
78
+- **Date:** 2026-05-19
79
+- **Decision:** TechCrunch RSS integration is an enrichment signal (not primary source) with explicit low-expectation framing (5–15% correlation hit rate). Feature degrades to zero noise when no correlations found.
80
+- **Why:** Correlation between press articles and repos is inherently low. Value lies in the delta (hype vs traction), not article summarization. Enrichment positioning allows silent failure without degrading digest.
81
+- **Implications:** All future `DataSource` plugins must declare "primary" or "enrichment" status. Enrichment sources require explicit failure/removal criteria. Farnsworth's analysis treats correlation data as optional context, never required input.
82
+
83
+## 2026-05-19: Milestone-based workflow adopted
84
+
85
+- **Owner:** jmservera (via Copilot)
86
+- **Date:** 2026-05-19
87
+- **Decision:** All future work organized into versioned milestones (v0.5, v0.6, etc.). PRDs are decomposed into issues, assigned to milestones, then moved to docs/processed/. This enables progress tracking and versioning.
88
+- **Why:** User directive — makes work easier to follow and enables versioning.
.squad/decisions/inbox/amy-generate-deploy.md
deleted
-15
@@ -1,15 +0,0 @@
1
-# Amy decision inbox — generate and deploy workflow
2
-
3
-- **Date:** 2026-05-18T13:20:07.067+02:00
4
-- **Issue:** #11 — Implement generate-and-deploy workflow for GitHub Pages
5
-
6
-## Proposed decision
7
-
8
-Keep `.github/workflows/deploy-site.yml` for push-to-main deployments, and let `.github/workflows/crawl-and-publish.yml` own the weekly automation path end-to-end.
9
-
10
-### Implementation details
11
-
12
-1. The weekly workflow should run `crawl → analyze → generate → deploy` in a single pipeline.
13
-2. The `generate` stage should write the weekly Hugo page into `content/weekly/YYYY/WNN.md` using archetype-compatible frontmatter derived from `data/analyzed/YYYY-WNN-summary.md`.
14
-3. The generated weekly page should be committed back to the default branch before the Pages build so future archive builds retain previously published weekly content.
15
-4. The publish artifact should be built with Hugo 0.161.1 and Pagefind, then deployed with `actions/deploy-pages@v4` under the `github-pages` environment.
.squad/decisions/inbox/bender-actions-crawl.md
deleted
-19
@@ -1,19 +0,0 @@
1
-# Bender decision inbox: Actions crawl workflow
2
-
3
-- **Date:** 2026-05-18T12:07:20.778+02:00
4
-- **Author:** Bender
5
-- **Issue:** #8
6
-
7
-## Proposed team decision
8
-
9
-Restore `data/cache/` from the latest successful `crawl-and-publish.yml` run before executing `scripts/crawl.py`.
10
-
11
-## Why
12
-
13
-- Reuses the crawler's on-disk GitHub API cache across weekly runs
14
-- Lowers repeated README/search calls on warm runs
15
-- Keeps the crawl stage self-contained until downstream jobs arrive in later issues
16
-
17
-## Workflow implication
18
-
19
-The workflow needs `actions: read` in addition to `contents: write` so it can discover the prior successful run and download the `crawl-cache` artifact.
.squad/decisions/inbox/bender-analyze-job.md
deleted
-38
@@ -1,38 +0,0 @@
1
-# Bender Decision Inbox — Analyze Job
2
-
3
-- **Date:** 2026-05-18T13:05:53.678+02:00
4
-- **Author:** Bender
5
-- **Issue:** #10 — Integrate Actions analyze job with Copilot path and reviewer gate
6
-
7
-## Context
8
-
9
-Phase 2 needs the weekly workflow to transform `data/raw/YYYY-WNN.json` into `data/analyzed/YYYY-WNN-summary.md` inside GitHub Actions, while preserving the approved fallback architecture and enforcing the analyzer contract before any downstream publish step runs.
10
-
11
-## Decision
12
-
13
-1. Extend `.github/workflows/crawl-and-publish.yml` with an `analyze` job that runs after `crawl`.
14
-2. Standardize the stage handoff artifacts as:
15
- - `raw-data` for crawl → analyze
16
- - `analyzed-data` for analyze → generate
17
-3. Use standalone Copilot CLI as the primary analysis path with:
18
- - `permissions.copilot-requests: write`
19
- - PAT secret `COPILOT_GH_TOKEN` exported as `COPILOT_GITHUB_TOKEN`
20
-4. Add `scripts/analyze_fallback.py` as the GitHub Models fallback using `permissions.models: read` and `GITHUB_TOKEN`.
21
-5. Enforce an automated `quality-check` gate in the workflow that blocks publish when the analysis contract is not met.
22
-
23
-## Quality Gate Contract
24
-
25
-The workflow gate should fail if any of the following are false:
26
-
27
-- YAML frontmatter exists.
28
-- The exact required frontmatter keys are present.
29
-- `quality_score` is an integer and at least 60.
30
-- Required H2/H3 sections appear in the documented order.
31
-- Body word count is at least 200.
32
-- Output does not leak raw JSON, traceback text, or placeholder/tool-log content.
33
-
34
-## Implications
35
-
36
-- Future generate jobs can safely consume `analyzed-data` without needing to inspect the raw crawl artifact.
37
-- Copilot CLI failures do not block the pipeline immediately; the GitHub Models fallback preserves publishability.
38
-- Reviewer-gate failures stay machine-detectable and stop low-quality summaries before they reach downstream stages.
.squad/decisions/inbox/bender-run-counter.md
deleted
-26
@@ -1,26 +0,0 @@
1
-# Bender Decision Inbox — Run Counter & Reskill Trigger
2
-
3
-- **Date:** 2026-05-18T15:22:25.067+02:00
4
-- **Author:** Bender
5
-- **Issue:** #15 — Add run counter persistence and every-fifth-run reskill trigger
6
-
7
-## Context
8
-
9
-The weekly crawl workflow already serializes runs with `concurrency`, but the learning audit found two missing pieces: there was no persisted `.squad/run-counter.txt`, and no workflow job checked that counter to trigger the every-5th-run reskill cycle.
10
-
11
-## Decision
12
-
13
-1. Create `.squad/run-counter.txt` in the repository, initialized to `0`.
14
-2. Increment the counter inside the `crawl` job's git commit step **after** syncing the default branch, then commit `.squad/run-counter.txt` together with `data/raw/` and `data/snapshots/`.
15
-3. Add a dedicated `reskill-check` job that reads the persisted counter and exposes `should_reskill` for downstream jobs.
16
-4. Add a gated placeholder `reskill` job that logs the trigger and scaffolds `.squad/skills/` and `.squad/reskill/` until Issue #14 adds the full retrospective implementation.
17
-
18
-## Why
19
-
20
-- Reading the counter only after syncing `origin/main` keeps the increment tied to the latest persisted state.
21
-- Committing `.squad/run-counter.txt` in the same crawl commit ensures the trigger survives between weekly runs.
22
-- Splitting `reskill-check` from `reskill` keeps the trigger logic auditable and makes the future reskill implementation easier to extend.
23
-
24
-## Follow-up
25
-
26
-- Issue #14 should add `.squad/` persistence for reskill outputs and the actual retrospective prompt/output flow.
.squad/decisions/inbox/farnsworth-analysis-spec.md
deleted
-29
@@ -1,29 +0,0 @@
1
-# Farnsworth Analysis Spec Decisions
2
-
3
-- **Date:** 2026-05-18T12:07:20.778+02:00
4
-- **Issue:** #9
5
-- **Scope:** Weekly analysis contract between crawler output and site generator
6
-
7
-## Proposed decisions
8
-
9
-1. **Analyzer output frontmatter is a superset contract.**
10
- - Required fields: `title`, `date`, `week`, `year`, `tags`, `categories`, `repos_featured`, `stars_tracked`, `top_repo`, `quality_score`, `summary`.
11
- - Why: this satisfies Amy’s weekly page needs and Leela’s reviewer gate in one artifact.
12
-
13
-2. **Reader-facing structure stays in five stable H2 sections.**
14
- - Required order: `Notable New Repositories`, `Trending This Week (Stars Gained)`, `Trend Analysis`, `What's Missing`, `Conclusion`.
15
- - Why: matches the approved weekly page shape and keeps generator parsing simple.
16
-
17
-3. **Signal / Noise / Gaps remain explicit as required subsections.**
18
- - `Trend Analysis` must include `### Signal` and `### Noise`.
19
- - `What's Missing` must include `### Gaps`.
20
- - Why: preserves the approved editorial lens without fighting the page-level section structure.
21
-
22
-4. **Trending must degrade honestly when momentum data is incomplete.**
23
- - If `stars_gained` is absent or null, the summary must say the section is directional rather than a true weekly momentum leaderboard.
24
- - Why: avoids overstating popularity as trend movement.
25
-
26
-5. **Analyzer input schema should be strict on core repo fields and tolerant on metadata.**
27
- - Required reads: week slug, crawl timestamp, new/trending repo arrays, and top topics.
28
- - Optional diagnostics: `partial_failures`, `filter_summary`, `snapshot_path`.
29
- - Why: supports current crawler output while leaving room for metadata evolution.
.squad/decisions/inbox/farnsworth-reskill.md
deleted
-23
@@ -1,23 +0,0 @@
1
-# Farnsworth Reskill Workflow Decisions
2
-
3
-- **Date:** 2026-05-18T15:22:25.067+02:00
4
-- **Issue:** #14
5
-- **Scope:** Reskill retrospective, learned-state injection, and quality trend tracking
6
-
7
-## Proposed decisions
8
-
9
-1. **Reskill context should be assembled from the latest analyzer evidence, not generic squad history alone.**
10
- - Inputs: last up to five `data/analyzed/*-summary.md` files, matching `data/snapshots/` hindsight when available, current `wisdom.md`, learned skills, and a quality trend report.
11
- - Why: this gives the retrospective something concrete to calibrate against and closes gaps G3-G7, G11, and G12.
12
-
13
-2. **Learned state must flow back into the weekly analyzer prompt.**
14
- - The analyze job should inject `.squad/identity/wisdom.md` into `{{WISDOM}}` and concatenated markdown from `.squad/skills/` into `{{SKILLS}}` before calling Copilot CLI or the GitHub Models fallback.
15
- - Why: without prompt injection, learning artifacts exist but never influence future analysis.
16
-
17
-3. **Quality trend tracking should be a first-class reskill input.**
18
- - `scripts/track_quality.py` should read `quality_score` from analyzed summaries and produce a markdown trend report for retrospective review.
19
- - Why: the squad needs a lightweight longitudinal measure of whether editorial quality is improving.
20
-
21
-4. **Reskill outputs belong in persistent squad state.**
22
- - Keep `.squad/reskill/` for weekly retrospective reports and `.squad/skills/` for extracted reusable patterns, both committed to git.
23
- - Why: durable learning needs durable storage, not ephemeral workflow output.
.squad/decisions/inbox/leela-cost-estimation.md
deleted
-32
@@ -1,32 +0,0 @@
1
-# Decision Proposal: Cost Estimation Framework for SquadScope
2
-
3
-**Date:** 2026-05-19T05:17:53.102+02:00
4
-**Author:** Leela (Lead/Architect)
5
-**Status:** Proposed
6
-**PRD:** docs/PRD-cost-estimation.md
7
-
8
-## Summary
9
-
10
-SquadScope's automated Copilot CLI and GitHub Models API usage has a quantifiable cost under token-based billing. At current configuration (~$0.30/week), annual cost is ~$16 — well within Copilot Pro's 300 credits/month allowance. However, proactive monitoring and budget controls are needed before context growth or model upgrades change the picture.
11
-
12
-## Decisions Proposed
13
-
14
-1. **Accept current cost profile as sustainable** — $16/year is economically trivial; no immediate action required on model downgrade.
15
-2. **Implement token usage tracking (Phase A)** — Add `scripts/track_token_usage.py` and `data/metrics/token-usage.jsonl` to establish baselines before optimizing.
16
-3. **Set budget alert thresholds** — Warn at $0.50/run, fail at $1.00/run, email alert at $5/month cumulative, auto-switch to cheaper model at $10/month cumulative (aligned with PRD budget alerts table).
17
-4. **Defer raw JSON pre-processing** — The 40-60% savings is significant but adds pipeline complexity; implement only if costs grow beyond $30/year.
18
-5. **Wisdom.md cap at 5 KB** — Reskill should retire obsolete heuristics, not only append.
19
-
20
-## Rationale
21
-
22
-The dominant cost driver (raw JSON at 86K tokens) is stable and bounded by crawl scope. Growth comes from wisdom/skills/history accumulation, which is slow. Premature optimization would add complexity without meaningful savings at current scale.
23
-
24
-## Risks
25
-
26
-- OQ5/OQ6: Billing mechanics for Copilot CLI vs Models API may differ in ways not yet visible
27
-- Credit exhaustion mid-month would disrupt the weekly pipeline if no degradation path exists
28
-
29
-## Next Steps
30
-
31
-- Create implementation issues per PRD Phase A (tracking)
32
-- Validate actual token counts against estimates after 4 weeks of data
.squad/decisions/inbox/leela-topic-channels.md
deleted
-77
@@ -1,77 +0,0 @@
1
-# Decision: Topic-Specific News Channels Architecture
2
-
3
-**Date:** 2026-05-18T13:20:07.067+02:00
4
-**Author:** Leela (Lead/Architect)
5
-**Status:** Proposed
6
-**PRD:** docs/PRD-topic-channels.md
7
-**PR:** #39
8
-
9
-## Decision Summary
10
-
11
-Generalize SquadScope into a topic-channel system where each instance serves a single focused domain with isolated learning and calibrated predictions.
12
-
13
-## Key Architectural Decisions
14
-
15
-### 1. Feature First, Not Platform
16
-
17
-SquadScope topic channels extend the existing pipeline (Crawl → Score → Analyze → Generate → Deploy) by adding a topic namespace. No new platform, no new repo structure. Same codebase, configured differently.
18
-
19
-### 2. Multi-Instance Single-Topic (v1)
20
-
21
-One fork/config per topic. Each fork has its own `squadscope.topic.yml`, its own Actions schedule, its own GitHub Pages site. This avoids orchestration complexity and keeps learning isolation trivial.
22
-
23
-Multi-topic single-instance is v2 — only after v1 proves the model works.
24
-
25
-### 3. Topic Config as Single Source of Truth
26
-
27
-`squadscope.topic.yml` at repo root controls:
28
-- Crawler queries
29
-- Scoring weights and thresholds
30
-- Analysis tone and audience
31
-- Learning state paths
32
-- Quality criteria
33
-
34
-### 4. Scoring Pipeline (New Stage)
35
-
36
-GitHub topic search is noisy. A new `scripts/score_repos.py` sits between crawl and analyze, scoring repos 0-100 on topic relevance, star momentum, language match, noise penalties, and recency. Only repos scoring ≥40 reach analysis.
37
-
38
-### 5. Per-Topic Learning Isolation
39
-
40
-Each topic maintains:
41
-- `topics/{id}/wisdom.md` — domain-specific heuristics
42
-- `topics/{id}/skills/` — extracted patterns
43
-- `topics/{id}/predictions.jsonl` — prediction ledger
44
-- `topics/{id}/scorecards/` — hindsight validation results
45
-
46
-No cross-topic contamination. Wisdom from AI/ML never leaks into Rust analysis.
47
-
48
-### 6. Prediction Ledger with Hindsight Validation
49
-
50
-Every analysis appends machine-readable predictions to `predictions.jsonl`. Four weeks later, `scripts/validate_predictions.py` scores them against actual outcomes (star deltas, fork growth). Scorecards feed into reskill.
51
-
52
-### 7. Topic Quality Criteria
53
-
54
-Topics must meet minimum thresholds to justify a channel:
55
-- Minimum N repos/week passing filters
56
-- Maximum false positive rate
57
-- Minimum genuinely significant repos per issue
58
-
59
-## Implications
60
-
61
-- Crawler must read config instead of using hardcoded queries
62
-- Analysis prompt becomes a template with injection points
63
-- Hugo gains topic taxonomy and per-topic RSS
64
-- All data paths gain `{topic_id}/` prefix
65
-- Reskill reads per-topic state
66
-
67
-## Risks
68
-
69
-- Topic configs may need frequent tuning in early weeks (mitigated by quality threshold warnings)
70
-- Fork-per-topic model doesn't scale past ~5 topics (acceptable; v2 addresses this)
71
-- Scoring weights are subjective initially (mitigated by prediction validation loop)
72
-
73
-## Open for Discussion
74
-
75
-- Should enrichment signals (forks, contributors) be in v1 scorer or deferred?
76
-- Prediction confidence: fixed initial values or prompt-generated?
77
-- Topic config in root vs `topics/` directory?
docs/processed/PRD-cost-estimation.md
renamed
docs/processed/PRD-techcrunch-integration.md
new
+301
@@ -0,0 +1,301 @@
1
+# PRD: TechCrunch RSS Integration for Cross-Source Trend Correlation
2
+
3
+**Author:** Farnsworth (Analyst/Content Curator)
4
+**Date:** 2026-05-19
5
+**Status:** Draft
6
+**Type:** Feature PRD
7
+**Depends on:** .squad/decisions.md (Decision #7: Crawler Plugin Architecture), docs/PRD-topic-channels.md
8
+
9
+---
10
+
11
+## Executive Summary
12
+
13
+SquadScope currently derives all insights from a single signal source: GitHub activity. While GitHub reveals *what developers are building*, it cannot tell us *why* activity is spiking — whether it's organic community interest, a VC-backed launch, or a viral TechCrunch article driving attention. This PRD proposes integrating TechCrunch's RSS feed as SquadScope's first non-GitHub data source, enabling **cross-source trend correlation** that distinguishes organic momentum from press-driven hype.
14
+
15
+**Key insight:** GitHub star surges often lag TechCrunch coverage by 24–72 hours. Detecting this pattern lets SquadScope editorially distinguish "genuinely important" (organic growth) from "temporarily hyped" (press-driven spike that fades within a week).
16
+
17
+---
18
+
19
+## Problem Statement
20
+
21
+### What GitHub data alone cannot tell us
22
+
23
+1. **Causality is invisible.** A repo gaining 2,000 stars in a week is interesting, but *why* matters editorially. Is it because the project shipped a breakthrough feature, or because TechCrunch wrote about it and HN amplified?
24
+
25
+2. **Funding and launch context is missing.** When a startup raises a $50M Series B and open-sources their core library, GitHub shows a star spike — but without the funding context, the analysis misattributes organic community excitement.
26
+
27
+3. **Industry narrative gaps.** SquadScope's "Gaps" section (what's missing from the conversation) is currently limited to what's absent from GitHub. But sometimes the gap is between what the industry *claims* to care about (per press coverage) and what's actually *being built* (per GitHub).
28
+
29
+4. **Hype detection requires a baseline.** To identify noise, you need to know what the press machine is amplifying. Without press data, everything on GitHub looks equally "organic."
30
+
31
+5. **Prediction accuracy suffers.** The topic-channels PRD envisions a prediction ledger. Cross-referencing press coverage with subsequent GitHub activity dramatically improves prediction calibration.
32
+
33
+---
34
+
35
+## Value Proposition
36
+
37
+### For SquadScope readers
38
+
39
+| Current State (GitHub-only) | With TechCrunch Correlation |
40
+|---|---|
41
+| "Repo X gained 3,000 stars this week" | "Repo X gained 3,000 stars after TechCrunch covered their $30M raise — watch if stars sustain past week 2" |
42
+| "These 5 AI repos are trending" | "3 of 5 trending AI repos correlate with press coverage; 2 show organic growth (stronger signal)" |
43
+| "Gap: No new observability tools" | "Gap: TechCrunch covered 4 observability startups this month, but none have meaningful GitHub traction yet — vaporware risk" |
44
+
45
+### For SquadScope's editorial stance
46
+
47
+- **Critical thinking becomes measurable:** "Press-amplified vs. organically growing" is a concrete, data-backed editorial judgment
48
+- **Signal vs. noise gets sharper:** Hype detection moves from vibes-based to correlation-based
49
+- **The Gaps section gains depth:** Disconnects between press narrative and actual developer activity become visible
50
+
51
+---
52
+
53
+## Correlation Model
54
+
55
+### How TechCrunch articles map to GitHub signals
56
+
57
+```
58
+┌─────────────────┐ ┌──────────────────────┐
59
+│ TechCrunch RSS │ │ GitHub Weekly Crawl │
60
+│ (article feed) │ │ (repo activity) │
61
+└────────┬────────┘ └──────────┬───────────┘
62
+ │ │
63
+ ▼ ▼
64
+┌─────────────────┐ ┌──────────────────────┐
65
+│ Extract: │ │ Extract: │
66
+│ - Company/proj │ │ - Repo name/org │
67
+│ - Category │ │ - Star delta │
68
+│ - Funding amt │ │ - Fork delta │
69
+│ - GitHub links │ │ - Contributor growth │
70
+└────────┬────────┘ └──────────┬───────────┘
71
+ │ │
72
+ └──────────┬───────────────────┘
73
+ ▼
74
+ ┌─────────────────────┐
75
+ │ Correlation Engine │
76
+ │ (fuzzy matching) │
77
+ └──────────┬──────────┘
78
+ ▼
79
+ ┌─────────────────────┐
80
+ │ Annotated Analysis │
81
+ │ - press_correlated │
82
+ │ - organic_growth │
83
+ │ - hype_risk_score │
84
+ └─────────────────────┘
85
+```
86
+
87
+### Correlation heuristics
88
+
89
+1. **Direct link match:** TechCrunch article contains a GitHub URL → exact match to crawled repo
90
+2. **Organization match:** Article mentions company X → match to `github.com/X/*` repos gaining stars
91
+3. **Project name match:** Article title/body contains project name → fuzzy match against repo names in weekly crawl
92
+4. **Category correlation:** Article tagged "AI" published Monday → AI-category repos spiking by Thursday
93
+5. **Temporal lag analysis:** Stars gained within 72 hours of article publication → likely press-correlated
94
+
95
+### Hype risk scoring
96
+
97
+| Pattern | Hype Risk | Editorial Label |
98
+|---------|-----------|-----------------|
99
+| Stars spike post-article, sustain 2+ weeks | Low | "Press-validated, community-sustained" |
100
+| Stars spike post-article, decay within 7 days | High | "Press-driven hype, fading interest" |
101
+| Stars growing before any press coverage | Very Low | "Organic growth — genuinely interesting" |
102
+| Press coverage but no GitHub activity | Medium | "Announced but unbuilt / closed-source" |
103
+
104
+---
105
+
106
+## Technical Approach
107
+
108
+### Data Source: TechCrunch RSS
109
+
110
+- **Feed URL:** `https://techcrunch.com/feed/`
111
+- **Format:** RSS 2.0 / XML
112
+- **Update frequency:** ~20-40 articles/day
113
+- **Relevant categories:** Startups, Apps, AI, Funding, Open Source
114
+- **Rate limits:** None (public RSS)
115
+- **Content available in feed:** Title, excerpt/summary, author, publish date, categories, link
116
+
117
+### Architecture: Fits Decision #7 (Crawler Plugin)
118
+
119
+The existing `DataSource` protocol interface applies directly:
120
+
121
+```python
122
+class TechCrunchSource:
123
+ """Crawler plugin for TechCrunch RSS feed."""
124
+
125
+ def get_name(self) -> str:
126
+ return "techcrunch"
127
+
128
+ def get_rate_limits(self) -> RateLimits:
129
+ return RateLimits(requests_per_hour=10, burst=5)
130
+
131
+ async def crawl(self, config: CrawlConfig) -> CrawlResult:
132
+ """Fetch and parse TechCrunch RSS, extract structured articles."""
133
+ ...
134
+```
135
+
136
+### Data flow integration
137
+
138
+```
139
+Existing: data/raw/YYYY-WNN.json (GitHub crawl)
140
+New: data/raw/YYYY-WNN-techcrunch.json (TechCrunch crawl)
141
+Merged: data/analyzed/YYYY-WNN-summary.md (cross-referenced analysis)
142
+```
143
+
144
+### RSS parsing requirements
145
+
146
+| Requirement | Approach |
147
+|-------------|----------|
148
+| XML parsing | `feedparser` (Python) — battle-tested RSS library |
149
+| Category extraction | Map TC categories to SquadScope topic taxonomy |
150
+| GitHub link extraction | Regex scan article content for `github.com` URLs |
151
+| Entity extraction | Match company/project names against crawled repos |
152
+| Deduplication | Hash on article URL; skip already-processed items |
153
+| Storage | JSON array, same weekly naming as GitHub crawl |
154
+
155
+### Output schema (per article)
156
+
157
+```json
158
+{
159
+ "source": "techcrunch",
160
+ "title": "Anthropic open-sources Claude's tool-use framework",
161
+ "url": "https://techcrunch.com/2026/05/15/...",
162
+ "published_at": "2026-05-15T14:30:00Z",
163
+ "categories": ["ai", "open-source", "funding"],
164
+ "github_links": ["https://github.com/anthropics/tool-use-sdk"],
165
+ "entities": ["Anthropic", "Claude"],
166
+ "funding_amount": null,
167
+ "relevance_score": 0.85
168
+}
169
+```
170
+
171
+### Analyzer changes
172
+
173
+The analyzer prompt gains a new context block:
174
+
175
+```
176
+## Press Context (TechCrunch, week of {date})
177
+{N} articles published relevant to tech/open-source.
178
+Notable coverage:
179
+- {title} ({category}) — mentions {github_links}
180
+- ...
181
+
182
+Cross-reference: For each trending repo, note if press coverage
183
+preceded the star surge. Label as "press-correlated" or "organic."
184
+```
185
+
186
+---
187
+
188
+## Phases
189
+
190
+### Phase 1: RSS Crawl Plugin (1–2 weeks)
191
+
192
+- Implement `TechCrunchSource` crawler plugin
193
+- Parse RSS feed, extract structured article data
194
+- Store as `data/raw/YYYY-WNN-techcrunch.json`
195
+- Filter to tech/open-source relevant articles only
196
+- Basic deduplication
197
+- **Output:** Weekly TechCrunch article JSON alongside GitHub JSON
198
+
199
+### Phase 2: Correlation Engine (2–3 weeks)
200
+
201
+- Implement GitHub URL extraction from articles
202
+- Fuzzy entity matching (company name → GitHub org)
203
+- Temporal correlation (article date vs. star surge timing)
204
+- Add `press_correlated: bool` and `hype_risk: low|medium|high` to repo analysis
205
+- **Output:** Enriched analysis with cross-source annotations
206
+
207
+### Phase 3: Editorial Integration (1–2 weeks)
208
+
209
+- Update analyzer prompt to consume TechCrunch context
210
+- Add "Press vs. Reality" subsection to weekly summary
211
+- Surface disconnects in Gaps section
212
+- Update Hugo templates to render correlation badges
213
+- **Output:** Reader-facing cross-source insights on the published site
214
+
215
+### Phase 4: Prediction Enhancement (future)
216
+
217
+- Track whether press-correlated repos sustain momentum
218
+- Feed correlation accuracy back into prediction ledger
219
+- Calibrate hype risk scoring over time
220
+- **Output:** Improved prediction accuracy in topic channels
221
+
222
+---
223
+
224
+## Cost & Resource Impact
225
+
226
+| Resource | Impact |
227
+|----------|--------|
228
+| RSS fetch | Negligible (1 HTTP request/week, public feed, no auth) |
229
+| Storage | ~50-100 KB/week JSON (40 articles × metadata) |
230
+| Analyzer tokens | +500-800 tokens input context per run (~$0.002/week) |
231
+| API rate limits | Zero impact (RSS is not GitHub API) |
232
+| CI minutes | +5-10 seconds per run (RSS fetch + parse) |
233
+| Dependencies | `feedparser` (Python, MIT license, mature) |
234
+
235
+**Total incremental cost: <$0.01/week.** Trivial relative to base pipeline costs documented in PRD-cost-estimation.md.
236
+
237
+---
238
+
239
+## Risks & Mitigations
240
+
241
+| Risk | Probability | Impact | Mitigation |
242
+|------|------------|--------|------------|
243
+| TechCrunch changes RSS format | Low | Medium | feedparser handles format variations; alert on parse failures |
244
+| RSS feed discontinued | Very Low | Low | Graceful degradation — analysis runs without press context |
245
+| False correlations (noise) | Medium | Medium | Require temporal proximity (72h) + name match confidence >0.7 |
246
+| Over-weighting press signal | Medium | High | Editorial rule: press correlation is annotation, not ranking factor |
247
+| Content extraction blocked | Low | Low | Use RSS summary only, don't scrape full articles |
248
+
249
+---
250
+
251
+## Open Questions
252
+
253
+1. **OQ1: Should we also extract from TechCrunch's category-specific feeds?**
254
+ - `techcrunch.com/category/artificial-intelligence/feed/` for topic-channel alignment
255
+ - Pro: Better relevance filtering. Con: More feeds to manage.
256
+
257
+2. **OQ2: Full article fetch vs. RSS excerpt only?**
258
+ - RSS includes ~200 word excerpt. Full article requires HTTP fetch + HTML parsing.
259
+ - Recommendation: Start with RSS excerpt only. Avoids scraping concerns and ToS issues.
260
+
261
+3. **OQ3: Should correlation annotations be visible to readers or analyst-only?**
262
+ - Option A: Show "📰 Press-correlated" badge on repo entries
263
+ - Option B: Keep as internal signal that shapes editorial tone only
264
+ - Recommendation: Option A for transparency (readers deserve to know *why* something is trending)
265
+
266
+4. **OQ4: Add HackerNews as a second correlation source simultaneously?**
267
+ - HN has an API, overlaps with TechCrunch coverage, and better represents developer sentiment
268
+ - Recommendation: TechCrunch first (simpler, RSS), HN second (API, different signal)
269
+
270
+5. **OQ5: How to handle TechCrunch articles about closed-source products?**
271
+ - Many TC articles cover proprietary SaaS with no GitHub presence
272
+ - Recommendation: Filter to articles containing GitHub links OR open-source keywords only
273
+
274
+---
275
+
276
+## Success Criteria
277
+
278
+| Metric | Target | Measurement |
279
+|--------|--------|-------------|
280
+| Articles crawled per week | 15-40 relevant | Count in weekly JSON |
281
+| Correlation hit rate | >30% of trending repos have press match | Cross-reference accuracy |
282
+| Hype detection accuracy | >70% of "high hype risk" repos show star decay at week +2 | Retrospective validation |
283
+| Reader value signal | Qualitative improvement in Gaps section depth | Editorial review |
284
+| Zero pipeline failures from RSS source | 100% graceful degradation | CI logs |
285
+
286
+---
287
+
288
+## Relationship to Existing PRDs
289
+
290
+- **PRD-topic-channels.md:** TechCrunch correlation enriches per-topic analysis. AI-focused TC articles correlate with `ai-ml` topic channel repos.
291
+- **PRD-cost-estimation.md:** Incremental cost is negligible (<$0.01/week). No tier change needed.
292
+- **decisions.md Decision #7:** This is the first concrete implementation of the crawler plugin architecture.
293
+- **decisions.md MCP Tools:** TechCrunch RSS fetch can be an MCP tool, registered in allowlist per Decision 5.
294
+
295
+---
296
+
297
+## Editorial Philosophy Note
298
+
299
+TechCrunch integration does NOT mean SquadScope becomes a TechCrunch aggregator. The feed is a **correlation signal**, not content to republish. SquadScope's voice remains: "Here's what's actually happening on GitHub this week, and here's what the press says is happening. Notice the gap? That's where the real story is."
300
+
301
+The editorial value is in the *delta* between press narrative and developer activity — not in summarizing TechCrunch articles.