fix(correlate): load repos from new_repos/trending_repos keys (#130)
fix: correlator schema mismatch - loads repos from new_repos/trending_repos keys Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
May 19, 2026 at 20:16 UTC
5f2980c834df963f8dfc8ca1a6ace15fc9fdae99
7 files changed
+189
-37
.squad/agents/bender/history.md
+2
@@ -33,3 +33,5 @@
33
- **2026-05-19T19:11:06+02:00:** W21 page was stale because the generate step's direct-push to main was rejected by branch protection. PR #123 fixed the workflow to use PR-based commits but wasn't merged until after the W21 run completed. Created PR #125 to regenerate W21 content (plus rollups) from the existing analyzed summary. Future workflow runs will use the corrected PR-based approach and should succeed without manual intervention.
34
35
- **2026-05-19T19:37:45+02:00:** Issue #128 — the PR-based approach from #123 also failed because the repo setting "Allow GitHub Actions to create or approve pull requests" is disabled. Fix: PR #129 replaces all `gh pr create` + `gh pr merge --auto` steps with direct push to an unprotected `publish` branch. The main branch ruleset only protects `refs/heads/main`, so `publish` accepts pushes from `GITHUB_TOKEN`. Key learnings: (1) Can't use branch name `data` if `data/*` branches already exist (git ref namespace collision). (2) Must use `git checkout -f` when switching branches after artifact downloads modify the working tree. (3) Deploy job's `github-pages` environment only allows deploys from main — expected failure when testing from feature branches.
36
+
37
+**TEAM UPDATE 2026-05-19T17:37:45Z:** Decisions and directives captured in `.squad/decisions.md`: (1) "Use `publish` branch for automated data commits" — approved and implemented in PR #129. (2) "Manual W21 content regeneration" — executed via PR #125. (3) Team directive "Never bypass branch protection rulesets" documented. Scribe orchestration logs created for both Bender and Hermes. All inbox decisions merged and deduplicated.
.squad/agents/farnsworth/history.md
+1
@@ -25,3 +25,4 @@
25
- **2026-05-19T11:55:46Z:** Self-review of TechCrunch RSS PRD (PR #55) completed. Key findings: (1) PRD file missing from branch — blocker. (2) Correlation hit rate realistically 5–15%, not the implicit "most articles correlate" assumption. Name-matching across TC articles and GitHub repos requires entity resolution, not string matching. (3) Filtering is underspecified — no keywords, no category selection, no confidence tiers defined. (4) Temporal mismatch between real-time RSS and weekly analysis means correlations are retrospective explanations, not predictive signals — the "prediction enhancement" phase is premature. (5) No success criteria defined to evaluate whether integration justifies its complexity. Lesson: excitement about architectural patterns (plugin system) must not override skepticism about editorial value-add. The right question isn't "can we?" but "should we, and will it measurably improve output?"
26
- **2026-05-19T15:08:00Z:** Leela milestone decomposition complete. Issues assigned to v0.5–v0.9 milestones. Scribe logged orchestration and merged decision. Your assigned v0.5 analysis and synthesis issues are ready. See `.squad/orchestration-log/2026-05-19T15-08-leela.md` for full decomposition outcome.
27
- **2026-05-19T15:22:00+02:00:** Topic-aware prompt template implemented (Issue #63). Key architecture decisions: (1) Used `{{#IF_TOPIC}}`/`{{#IF_NO_TOPIC}}` conditional blocks rather than Jinja2 to keep the template readable as standalone markdown and avoid adding template engine dependencies. (2) Wisdom injection is two-tier — global wisdom from `.squad/identity/wisdom.md` (existing) plus per-topic wisdom from `topics/{id}/wisdom.md` (new). (3) Render script (`scripts/render_topic_prompt.py`) is zero-dependency (stdlib only, with optional PyYAML), so it works in any CI environment without pip install. (4) Backward compatibility guaranteed: when no `squadscope.topic.yml` exists, the template collapses cleanly to general-mode analysis identical to the existing `analyze-weekly.md` behavior.
28
+- **2026-05-19T20:07:19+02:00:** Fixed correlator "0 repos" bug (PR #130). Root cause: `correlate.py` loaded repos via `raw_data.get("repos")` but `crawl.py` writes them under `new_repos` and `trending_repos`. Key paths: `scripts/correlate.py:320`, `scripts/crawl.py:857-858`. Lesson: when integrating scripts in a pipeline, always verify the producer's *actual output schema* against the consumer's expected input schema — don't assume key names match. The CI skill pattern ("test the wire") would have caught this if applied at integration time.
.squad/agents/hermes/history.md
+2
@@ -16,3 +16,5 @@
16
- AI fallback chain: Copilot CLI → GitHub Models API → no-AI data summary
17
- Token/secrets used: GITHUB_TOKEN (scoped per job), GitHub Models API token
18
- Rate limiting handled with exponential backoff in crawl.py and analyze_fallback.py
19
+
20
+**2026-05-19T17:37:45Z:** Security review of PR #129 completed — **APPROVED**. Findings: (1) Permission changes safe — `publish` branch strategy requires no new secrets. (2) Branch protection remains intact on main. (3) No supply chain risks. Also reviewed PR #126 (TechCrunch RSS) — **SECURITY CLEAR**: No SSRF, retry logic bounded, feedparser is CVE-free, no credential leaks. Non-blocking recommendation: add control character sanitization for defense-in-depth against prompt injection. Decision "PR #126 Security Review — Clear" documented in `.squad/decisions.md`. Orchestration log created.
.squad/decisions.md
+107
@@ -633,6 +633,113 @@ Add TechCrunch RSS (`https://techcrunch.com/feed/`) as SquadScope's first non-Gi
633
634
---
635
636
+## Decision: Use `publish` branch for automated data commits (2026-05-19)
637
+
638
+**Author:** Bender (Crawler)
639
+**Status:** Implemented (PR #129)
640
+**Fixes:** Issue #128
641
+
642
+### Context
643
+
644
+The crawl-and-publish workflow failed because:
645
+1. The repo setting "Allow GitHub Actions to create or approve pull requests" is disabled
646
+2. `gh pr create` with `GITHUB_TOKEN` is blocked by this setting
647
+3. Even if enabled, the `copilot_code_review` rule + `required_review_thread_resolution` on main could block auto-merge unpredictably
648
+
649
+### Decision
650
+
651
+Replace PR-based commits with direct push to an unprotected `publish` branch.
652
+
653
+- The main branch ruleset only protects `refs/heads/main`
654
+- The `publish` branch accepts direct pushes from workflow `GITHUB_TOKEN`
655
+- Inter-job data flow uses artifacts (unchanged)
656
+- Deploy job downloads all artifacts directly (no dependency on branch state)
657
+- `reskill-check` reads `run-counter.txt` from `publish` branch with fallback to main
658
+
659
+### Consequences
660
+
661
+- Automated data no longer lands on `main` automatically — it accumulates on `publish`
662
+- A separate manual or scheduled merge from `publish` → `main` can sync when desired
663
+- Main branch protection remains fully intact (no bypasses)
664
+- Pipeline reliability is decoupled from PR permission settings
665
+
666
+### Alternatives Considered
667
+
668
+1. Enable "Allow GitHub Actions to create PRs" — requires repo admin action, doesn't solve auto-merge reliability
669
+2. Use a PAT/GitHub App token — adds secret management complexity
670
+3. `--admin` flag on merge — bypasses protection, violates team decision
671
+
672
+---
673
+
674
+## Decision: Manual W21 content regeneration (2026-05-19)
675
+
676
+**Author:** Bender
677
+**Status:** Executed
678
+
679
+### Context
680
+
681
+The crawl-and-publish workflow (run #26109935234) generated W21 analysis but failed to commit it because the commit step tried to push directly to `main`, which requires PRs (branch protection). PR #123 fixed the workflow to use PR-based commits, but it merged after the failed run.
682
+
683
+### Decision
684
+
685
+Regenerated W21 content manually and created PR #125 to update the page. No workflow code changes needed — the root cause (direct push) was already fixed by PR #123.
686
+
687
+### Impact
688
+
689
+- `content/weekly/2026/W21.md` updated from stale manual dry-run to full analysis
690
+- Monthly/yearly rollups refreshed
691
+- Once PR #125 merges, deploy-site will publish the updated page
692
+- Future scheduled runs will use the PR-based approach and should not hit this again
693
+
694
+---
695
+
696
+## Directive: Always test the whole publishing cycle before considering work done (2026-05-19)
697
+
698
+**By:** jmservera (via Copilot)
699
+**Date:** 2026-05-19T19:37:45+02:00
700
+
701
+User directive — captured for team memory. Always test the whole publishing cycle before considering work done.
702
+
703
+---
704
+
705
+## Decision: PR #126 Security Review — Clear (2026-05-19)
706
+
707
+**Author:** Hermes (Security)
708
+**PR:** #126 — "feat: wire TechCrunch RSS into CI pipeline and add API retry backoff"
709
+
710
+### Decision
711
+
712
+PR #126 is **security-clear**. No blocking vulnerabilities found.
713
+
714
+### Key Findings
715
+
716
+- No SSRF risk (hardcoded feed URL)
717
+- Retry logic properly bounded (3 retries, exponential backoff + jitter)
718
+- No secrets leaked in logs
719
+- feedparser dependency is well-maintained, no CVEs
720
+- Workflow permissions unchanged
721
+
722
+### Non-blocking Recommendation
723
+
724
+RSS content fed into AI prompts has a theoretical indirect prompt injection surface. Mitigated by HTML stripping and content truncation. Recommend adding control character sanitization in a future PR for defense-in-depth.
725
+
726
+### Impact
727
+
728
+Team can merge PR #126 without security holds.
729
+
730
+---
731
+
732
+## Directive: Never bypass branch protection rulesets (2026-05-19)
733
+
734
+**By:** jmservera (via Squad)
735
+**Date:** 2026-05-19T18:05:10Z
736
+
737
+CI workflows must not push directly to protected branches. Use PR-based commits instead. Never add bypass actors to rulesets to work around branch protection.
738
+
739
+**Why:** Branch protection exists to ensure code review on every change. Bypassing it for convenience undermines the safety net.
740
+
741
+---
742
+
743
## Governance
744
745
- All meaningful changes require team consensus
.squad/decisions/inbox/bender-publish-branch-strategy.md
deleted
-36
@@ -1,36 +0,0 @@
1
-# Decision: Use `publish` branch for automated data commits
2
-
3
-**Date:** 2026-05-19T19:37:45+02:00
4
-**Author:** Bender (Crawler)
5
-**Status:** Implemented (PR #129)
6
-**Fixes:** Issue #128
7
-
8
-## Context
9
-
10
-The crawl-and-publish workflow failed because:
11
-1. The repo setting "Allow GitHub Actions to create or approve pull requests" is disabled
12
-2. `gh pr create` with `GITHUB_TOKEN` is blocked by this setting
13
-3. Even if enabled, the `copilot_code_review` rule + `required_review_thread_resolution` on main could block auto-merge unpredictably
14
-
15
-## Decision
16
-
17
-Replace PR-based commits with direct push to an unprotected `publish` branch.
18
-
19
-- The main branch ruleset only protects `refs/heads/main`
20
-- The `publish` branch accepts direct pushes from workflow `GITHUB_TOKEN`
21
-- Inter-job data flow uses artifacts (unchanged)
22
-- Deploy job downloads all artifacts directly (no dependency on branch state)
23
-- `reskill-check` reads `run-counter.txt` from `publish` branch with fallback to main
24
-
25
-## Consequences
26
-
27
-- Automated data no longer lands on `main` automatically — it accumulates on `publish`
28
-- A separate manual or scheduled merge from `publish` → `main` can sync when desired
29
-- Main branch protection remains fully intact (no bypasses)
30
-- Pipeline reliability is decoupled from PR permission settings
31
-
32
-## Alternatives Considered
33
-
34
-1. Enable "Allow GitHub Actions to create PRs" — requires repo admin action, doesn't solve auto-merge reliability
35
-2. Use a PAT/GitHub App token — adds secret management complexity
36
-3. `--admin` flag on merge — bypasses protection, violates team decision
scripts/correlate.py
+9
-1
@@ -317,7 +317,15 @@ def main(argv: list[str] | None = None) -> int:
317
318
# Load repos
319
raw_data = load_json(raw_path)
320
- repos = raw_data if isinstance(raw_data, list) else raw_data.get("repos", raw_data.get("repositories", []))
320
+ if isinstance(raw_data, list):
321
+ repos = raw_data
322
+ else:
323
+ # The crawl output stores repos under "new_repos" and "trending_repos"
324
+ repos = raw_data.get("repos", raw_data.get("repositories", []))
325
+ if not repos:
326
+ new_repos = raw_data.get("new_repos", [])
327
+ trending_repos = raw_data.get("trending_repos", [])
328
+ repos = new_repos + trending_repos
329
330
# Load articles (graceful if missing)
331
articles: list[dict[str, Any]] = []
tests/test_correlate.py
+68
@@ -284,3 +284,71 @@ class TestUtilities:
284
285
def test_fuzzy_name_score_empty(self):
286
assert fuzzy_name_score("", "something") == 0.0
287
+
288
+
289
+# ---------------------------------------------------------------------------
290
+# CLI: main() repo loading from crawl output format
291
+# ---------------------------------------------------------------------------
292
+
293
+
294
+class TestMainRepoLoading:
295
+ """Test that main() correctly loads repos from new_repos/trending_repos keys."""
296
+
297
+ def test_main_loads_new_and_trending_repos(self, tmp_path):
298
+ from scripts.correlate import main
299
+
300
+ raw_file = tmp_path / "2026-W21.json"
301
+ tc_file = tmp_path / "2026-W21-techcrunch.json"
302
+ output_file = tmp_path / "correlations.json"
303
+
304
+ raw_data = {
305
+ "week": "2026-W21",
306
+ "new_repos": [
307
+ _repo(name="new-project", owner="org1"),
308
+ ],
309
+ "trending_repos": [
310
+ _repo(name="trending-project", owner="org2"),
311
+ ],
312
+ }
313
+ tc_data = {
314
+ "articles": [
315
+ _article(entities=["Org1"]),
316
+ ],
317
+ }
318
+ raw_file.write_text(json.dumps(raw_data))
319
+ tc_file.write_text(json.dumps(tc_data))
320
+
321
+ ret = main([
322
+ "--raw", str(raw_file),
323
+ "--techcrunch", str(tc_file),
324
+ "--output", str(output_file),
325
+ ])
326
+ assert ret == 0
327
+
328
+ result = json.loads(output_file.read_text())
329
+ assert result["metadata"]["repos_analyzed"] == 2
330
+ assert result["metadata"]["correlations_found"] >= 1
331
+
332
+ def test_main_loads_repos_key_format(self, tmp_path):
333
+ """Backward compat: if 'repos' key exists, use it directly."""
334
+ from scripts.correlate import main
335
+
336
+ raw_file = tmp_path / "2026-W21.json"
337
+ output_file = tmp_path / "correlations.json"
338
+
339
+ raw_data = {
340
+ "week": "2026-W21",
341
+ "repos": [
342
+ _repo(name="classic-format", owner="org1"),
343
+ ],
344
+ }
345
+ raw_file.write_text(json.dumps(raw_data))
346
+
347
+ ret = main([
348
+ "--raw", str(raw_file),
349
+ "--output", str(output_file),
350
+ ])
351
+ assert ret == 0
352
+
353
+ result = json.loads(output_file.read_text())
354
+ assert result["metadata"]["repos_analyzed"] == 1