@cryptotaxi247 / SquadScope / commits / 6e952f1

fix: replace URL hostname substring checks with parsed-netloc checks (#164)

* ci: fix deploy divergence and add rebuild-without-recrawl support Fix 1 (deploy-site.yml): - Add workflow_run trigger so deploy auto-fires after crawl-and-publish completes, eliminating the need for a manual trigger after each weekly run. - Update build job if-condition to allow workflow_run (success) events in addition to manual and human-push triggers. - Add 'Hydrate generated content from publish' step before hugo build. Code/theme/config come from main; all generated content (weekly pages, rollups, raw data) is overlaid from the canonical publish branch. The site can no longer diverge from publish. PR #162 was the manual rescue. Fix 2 (crawl-and-publish.yml): - Add rebuild_week workflow_dispatch input. When set, the workflow skips the crawl steps entirely (Run crawler, Crawl TechCrunch RSS, Commit crawl data) to avoid polluting a prior week's good data. - Add 'Hydrate from publish (rebuild mode)' step in the analyze job that restores raw + all prior analyzed files from publish when rebuild_week is set, giving downstream steps correct historical context for rollups. - Update 'Prepare analysis context' Python block to use rebuild_week as WEEK when provided, falling back to the current isocalendar week. Closes the architectural concerns identified during the W21/W22 debacle. Implements the directive captured in .squad/decisions/inbox/copilot-directive-rebuild-no-recrawl.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: replace URL substring checks with parsed-netloc checks in tests Fixes 2 CodeQL alerts (py/incomplete-url-substring-sanitization) in test_render_press_context.py: - Line ~100: replaced '"techcrunch.com" in result' with a check for the full article URL to avoid the hostname-substring anti-pattern. - Line ~500: replaced '"github.com" in result' with a urlparse-based assertion that extracts all markdown link URLs and validates each netloc equals 'github.com'. Added imports: urllib.parse.urlparse, re. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ci): make rebuild_week guard safe across all trigger types Guard pattern github.event.inputs.rebuild_week == '' breaks on schedule events because inputs doesn't exist. Switch to !inputs.rebuild_week which works across schedule, push, and workflow_dispatch. Also: - Add format validation for rebuild_week input (YYYY-WNN) to prevent pathspec injection - Add data/snapshots/ to deploy-site hydration step for full publish parity Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed May 25, 2026 at 16:16 UTC 6e952f11f5b501729bd34d8374db6ce1137e2d80

3 files changed +74 -5

.github/workflows/crawl-and-publish.yml

+38 -2

@@ -10,6 +10,11 @@ on:

10 required: false

11 default: false

12 type: boolean

13 + rebuild_week:

14 + description: 'Rebuild a specific past week (e.g. 2026-W21). Skips crawl, hydrates raw/analyzed from publish, regenerates content+rollups only.'

15 + required: false

16 + default: ''

17 + type: string

19 permissions:

20 contents: read

@@ -91,6 +96,7 @@ jobs:

96 python-version: '3.12'

98 - name: Run crawler

99 + if: ${{ !inputs.rebuild_week }}

100 env:

101 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

102 run: python scripts/crawl.py

@@ -99,6 +105,7 @@ jobs:

105 run: pip install -r requirements.txt

106

107 - name: Crawl TechCrunch RSS

108 + if: ${{ !inputs.rebuild_week }}

109 run: |

110 WEEK=$(date +%Y-W%V)

111 SINCE=$(date -d '7 days ago' +%Y-%m-%d)

@@ -131,6 +138,7 @@ jobs:

138 if-no-files-found: warn

139

140 - name: Commit crawl data to data branch

141 + if: ${{ !inputs.rebuild_week }}

142 env:

143 DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}

144 DATA_BRANCH: publish

@@ -185,11 +193,32 @@ jobs:

193 ref: ${{ github.event.repository.default_branch }}

194

195 - name: Download raw crawl artifact

196 + if: ${{ !inputs.rebuild_week }}

197 uses: actions/download-artifact@v4

198 with:

199 name: raw-data

200 path: data/raw/

201

202 + - name: Hydrate from publish (rebuild mode)

203 + if: ${{ inputs.rebuild_week }}

204 + env:

205 + REBUILD_WEEK: ${{ inputs.rebuild_week }}

206 + run: |

207 + set -euo pipefail

208 + if ! [[ "$REBUILD_WEEK" =~ ^[0-9]{4}-W[0-9]{2}$ ]]; then

209 + echo "::error::Invalid rebuild_week format. Expected YYYY-WNN (e.g. 2026-W21), got: $REBUILD_WEEK"

210 + exit 1

211 + fi

212 + git fetch origin publish

213 + mkdir -p data/raw data/analyzed data/snapshots

214 + git checkout origin/publish -- "data/raw/${REBUILD_WEEK}.json"

215 + git checkout origin/publish -- "data/raw/${REBUILD_WEEK}-techcrunch.json" 2>/dev/null || true

216 + git checkout origin/publish -- "data/snapshots/${REBUILD_WEEK}-stars.json" 2>/dev/null || true

217 + # Also hydrate ALL prior analyzed files so rollups have correct historical context

218 + git ls-tree -r --name-only origin/publish -- data/analyzed/ | while read -r f; do

219 + git checkout origin/publish -- "$f" 2>/dev/null || true

220 + done

221 +

222 - name: Set up Node

223 uses: actions/setup-node@v4

224 with:

@@ -202,19 +231,26 @@ jobs:

231

232 - name: Prepare analysis context

233 id: analysis-context

234 + env:

235 + REBUILD_WEEK: ${{ inputs.rebuild_week || '' }}

236 run: |

237 mkdir -p data/analyzed

238 CURRENT_DATETIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)

239 readarray -t CONTEXT_LINES < <(python3 - <<'PY' "$CURRENT_DATETIME"

240 import json

241 + import os

242 import sys

243 from datetime import UTC, datetime

244 from pathlib import Path

245

246 current_datetime = sys.argv[1]

247 run_datetime = datetime.fromisoformat(current_datetime.replace("Z", "+00:00")).astimezone(UTC)

216 - iso_year, iso_week, _ = run_datetime.isocalendar()

217 - week = f"{iso_year}-W{iso_week:02d}"

248 + rebuild_week = os.environ.get("REBUILD_WEEK", "").strip()

249 + if rebuild_week:

250 + week = rebuild_week

251 + else:

252 + iso_year, iso_week, _ = run_datetime.isocalendar()

253 + week = f"{iso_year}-W{iso_week:02d}"

254 week_file = Path("data/raw") / f"{week}.json"

255 if not week_file.exists():

256 raise SystemExit(f"Missing raw payload for current run: {week_file}")

.github/workflows/deploy-site.yml

+30 -1

@@ -4,6 +4,12 @@ on:

4 push:

5 branches:

6 - main

7 + # Auto-deploy after a successful crawl-and-publish run so the site always

8 + # reflects the latest content committed to the publish branch.

9 + workflow_run:

10 + workflows: ["Crawl and publish weekly data"]

11 + types: [completed]

12 + branches: [main]

13 workflow_dispatch:

15 permissions:

@@ -17,7 +23,10 @@ concurrency:

24 jobs:

25 build:

20 - if: github.actor != 'github-actions[bot]'

26 + # Allow manual triggers, human pushes to main, and post-crawl auto-runs.

27 + # Bot pushes are excluded from push events to avoid loops; workflow_run

28 + # covers the bot-written publish branch updates.

29 + if: ${{ github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.actor != 'github-actions[bot]') || (github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') }}

30 runs-on: ubuntu-latest

31 env:

32 HUGO_VERSION: 0.161.1

@@ -53,6 +62,26 @@ jobs:

62 export PATH="${HOME}/.local/hugo:${PATH}"

63 hugo version

65 + - name: Hydrate generated content from publish

66 + # Architecture: code/theme/config come from main; all generated content

67 + # (weekly pages, rollups, raw data) comes from the canonical publish branch.

68 + # This prevents the site from ever diverging from the crawl-and-publish output.

69 + run: |

70 + set -euo pipefail

71 + git fetch origin publish

72 + # Wipe and re-checkout to mirror publish exactly (deletions propagate)

73 + rm -rf content/weekly content/monthly content/yearly data/analyzed data/raw data/metrics data/snapshots

74 + git checkout origin/publish -- \

75 + content/weekly/ \

76 + content/monthly/ \

77 + content/yearly/ \

78 + data/analyzed/ \

79 + data/raw/ \

80 + data/metrics/ \

81 + data/snapshots/ || true

82 + echo "Hydrated from publish:"

83 + ls content/weekly/2026/ 2>/dev/null || true

84 +

85 - name: Build site

86 run: hugo --minify

tests/test_render_press_context.py

+6 -2

@@ -1,7 +1,9 @@

1 """Tests for scripts/render_press_context.py."""

3 +import re

4 import sys

5 from pathlib import Path

6 +from urllib.parse import urlparse

8 _REPO_ROOT = Path(__file__).resolve().parent.parent

9 sys.path.insert(0, str(_REPO_ROOT / "scripts"))

@@ -95,7 +97,7 @@ class TestFormatArticlesList:

97 def test_single_article(self):

98 result = format_articles_list([_article()])

99 assert "[AI Startup Raises $10M]" in result

98 - assert "techcrunch.com" in result

100 + assert _article()["url"] in result

101 assert "[AI, Startups]" in result

102

103 def test_article_without_url(self):

@@ -495,7 +497,9 @@ class TestFormatCorrelationsNarrative:

497 corr = self._corr(articles=["https://techcrunch.com/unknown-url"])

498 result = _format_correlations_narrative([corr], [])

499 # URL is in the corr but not in the articles list, so no link text

498 - assert "[" not in result or "github.com" in result

500 + # Any links present must point to github.com (repo links), not article URLs

501 + link_urls = re.findall(r'\]$(https?://[^)]+)$', result)

502 + assert all(urlparse(url).netloc == "github.com" for url in link_urls)

503

504 def test_reader_mode_true_uses_narrative(self):

505 corr = self._corr(repo="openai/codex")