@cryptotaxi247 / SquadScope / commits / 6a20450

fix(analyze): never drop populated press context; harden gate + crawl (W30 incident) (#582)

* fix(analyze): never drop populated press context when synthesis narrative present Root cause of the 2026-W30 "No industry press data was available..." incident: _build_prompt blanked press_content whenever a Step-1 synthesis narrative was present (introduced in jmservera/SquadScope#515), so the Step-2 prompt had no Press Context block and the model emitted the no-press fallback despite a successful crawl (45 articles, 22 relevant, 32KB press-context.md). - analyze_fallback.py: when a synthesis narrative exists, keep the real press context (condensed to COMPACTED_PRESS_CONTEXT_CHARS) instead of blanking it, so "Where Industry Meets Code" / "Press & Industry" are written from real data. Sanitize press on this path (_strip_ai_instruction_blocks + _escape_untrusted_boundaries). press_decision now reflects reality ("included: condensed alongside synthesis narrative"). - prompts/analyze-weekly.md: emit the no-press line only when both press context and industry narrative are truly absent. - techcrunch_crawler.py: exponential backoff + jitter in fetch_feed (mirrors crawl.py), retry only transient failures (RETRYABLE_STATUSES) and fail fast on permanent HTTP errors; DEFAULT_FETCH_RETRIES 1->3. Partial results are preserved when a source fails, plus an explicit per-source success/failure stdout summary and ::warning:: on partial crawls (status also persisted in artifact metadata). - docs/operator-guide.md: document the analysis-only / press-only rerun path that reuses existing crawl artifacts without re-crawling GitHub or press. - tests: regression test that a synthesis narrative does not drop press context; crawler retry/backoff and fail-fast tests. Refs: jmservera/SquadScope#515, jmservera/SquadScope-Coordinator#33 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(gate): fail publish when body claims no press data but press context is populated Defense-in-depth for the 2026-W30 incident where a populated press-context.md was silently dropped and the analysis body shipped "No industry press data was available for this week's analysis." The existing contradiction rule only fires when the body ALSO describes press coverage; this adds a rule that fires even when the body does not self-contradict, as long as the week's press context is actually populated. - analysis_gate.py: add STALE_PRESS_CLAIM_PATTERNS (matches "no industry press data was available" and "no press data was provided this week"), stale_press_claim_errors(), and press_context_is_populated() which treats render_press_context.py's non-empty "No press data available for this week." sentinel as empty. New --press-context-path / --press-token-estimate args, wired through main() and validate_publish_quality(). Errors classify as editorial_quality (retryable). - crawl-and-publish.yml: pass --press-context-path to both gate invocations. - test: update mock signature to accept the new kwarg. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(press-context): lock 2026-W30 fix in analyze step and publish gate Strengthen and extend regression coverage for the 2026-W30 incident where a populated press-context.md was silently dropped and the analysis shipped "No industry press data was available for this week's analysis.". analyze_fallback: strengthen the synthesis-narrative test to assert the preflight press_correlations component stays included, add a case proving an oversized press context is condensed (not dropped) alongside a synthesis narrative, and add a case proving the no-press path is not over-corrected. analysis_gate: cover stale_press_claim_errors (fires when press context is available, silent otherwise, catches both phrasings), the validate_publish_quality wiring, and press_context_is_populated (missing/empty/sentinel vs real content and token_estimate>0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(analyze): treat no-press sentinel as absent press context (PR #582 review) Resolve Copilot review: the non-empty "No press data available for this week..." sentinel from render_press_context was being misclassified as real press, which could suppress the required no-press statement and false-positive the stale-press gate on genuinely press-less weeks. - render_press_context: add single source of truth NO_PRESS_SENTINEL and NO_PRESS_SENTINEL_MARKER; render returns the constant. - analyze_fallback: blank press_content when it matches the sentinel at all press read sites, so no `## Press Context` block is emitted and the component is included=False. Real-press retention (synthesis narrative) is unchanged. - analysis_gate.press_context_is_populated: a provided path's sentinel content now wins over token_estimate>0 (returns False); imports the shared marker instead of a duplicated local regex. - tests: regression coverage for the press-less-week path in fallback suppression and the stale-press gate; real-press W30 behavior preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: apply ruff format to press-context fix files CI runs `ruff format --check`; formatting-only changes (blank lines, line wrapping). No behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(press-context): wire --press-token-estimate fallback + correct operator doc Thread 1 (workflow): crawl-and-publish.yml now emits press_token_estimate as a step output and passes --press-token-estimate to analysis_gate.py in both the run-analysis and quality-check gate invocations. The gate already enforced precedence (readable press path authoritative; token estimate is fallback only when path is None/missing/empty/unreadable). This restores the fallback signal for the stale-press gate and matches the PR description. Thread 2 (doc): operator-guide.md now states the ## Press Context block appears only for real press; a press-less week renders the non-empty NO_PRESS_SENTINEL marker (single source of truth in render_press_context.py) which is suppressed (press_correlations.included=false). No hardcoded sentinel string. Test: add test_press_context_token_estimate_is_fallback_only proving the fallback branches and the sentinel-authoritative precedence. Refs jmservera/SquadScope#582 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(analyze): make synthesis-narrative label conditional on real press presence The Step-1 synthesis label hard-coded "press & historical context" even on press-less weeks (press_content empty / no-press sentinel suppressed), misleading the downstream model into thinking press exists. Make the label conditional on the already-normalized press_content emptiness (single source of truth); it now reads "historical context" only when press is absent. Adds regression coverage. Resolves the Copilot review thread on #582. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(press-context): emit genuine 0 token-estimate for empty press file estimate_tokens() clamps to min 1, so int(estimate_tokens(content) or 0) could never be 0 for an empty/whitespace press file, making the --press-token-estimate fallback misclassify an empty press file as populated in analysis_gate.press_context_is_populated(). Add press_token_estimate() helper (single source of truth, reuses estimate_tokens) that returns 0 for empty/whitespace content and a positive estimate otherwise, and wire the crawl-and-publish workflow to it. Readable press_context_path (real vs NO_PRESS_SENTINEL) remains authoritative; token estimate stays fallback-only. Add tests proving empty -> 0 -> not populated and real -> positive -> populated. Refs jmservera/SquadScope#582 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(crawler): honor HTTP-date Retry-After in _retry_after_seconds Retry-After (RFC 9110) may be either delta-seconds or an HTTP-date. _retry_after_seconds only parsed the numeric form and returned None for HTTP-date values, so the crawler silently ignored server-provided retry timing in that case. Parse the HTTP-date form via email.utils.parsedate_to_datetime, assume UTC for naive datetimes, compute the delay against now, and floor at 1.0 (past/near dates never yield a negative sleep). Empty/unparseable headers still return None. Downstream _sleep_before_retry clamps to the max delay. Add tests covering numeric, future HTTP-date, past HTTP-date, and garbage/empty header cases. Refs jmservera/SquadScope#582 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(press-context): treat no-press sentinel as 0 tokens in gate fallback press_token_estimate() feeds the gate's --press-token-estimate fallback signal. It previously returned a positive count for the non-empty NO_PRESS_SENTINEL text, so if analysis_gate.py ever fell back to the token estimate (missing/unreadable --press-context-path), a press-less week could be misclassified as populated and wrongly trip the stale-press gate. Return 0 when the stripped content matches NO_PRESS_SENTINEL_MARKER (in addition to empty/whitespace), reusing the existing marker regex so the fallback stays consistent with the authoritative path logic (sentinel = not populated). Extend tests to cover sentinel -> 0 -> not populated. Refs jmservera/SquadScope#582 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(crawler): honor server Retry-After above the backoff cap _sleep_before_retry unconditionally capped the delay to RETRY_MAX_DELAY_SECONDS (30s), so a server-supplied Retry-After (e.g. 120s on a 429) only slept 30s and defeated the purpose of honoring the header, risking repeated rate-limit hits. Honor a positive Retry-After up to a new, larger RETRY_AFTER_MAX_SECONDS (120s) ceiling, while the computed exponential backoff+jitter branch stays bounded by RETRY_MAX_DELAY_SECONDS. The separate ceiling still protects against a hostile/absurd Retry-After hanging the crawler. Add tests covering honored, bounded, small, and no-header cases. Refs jmservera/SquadScope#582 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(crawler): reject non-finite numeric Retry-After values float('nan')/float('inf') parse successfully, so a numeric Retry-After header of nan/inf could leak a non-finite value into retry logic and make behavior unpredictable. Guard the numeric branch with math.isfinite: only finite values are honored; nan/inf/-inf fall through to HTTP-date parsing (which returns None for such strings) and thus normal backoff. Add tests for nan/inf/-inf/Infinity -> None. Refs jmservera/SquadScope#582 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * style: apply ruff format to press-context and crawler tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jul 20, 2026 at 13:37 UTC 6a2045080f3f6b7b5e9c4f4123b165eccf8606ce

11 files changed +1044 -22

.github/workflows/crawl-and-publish.yml

+19 -3

@@ -479,18 +479,22 @@ jobs:

479 PRESS_CONTEXT=$(python scripts/render_press_context.py --week "$WEEK")

480 PRESS_FILE="data/analyzed/${WEEK}-press-context.md"

481 printf '%s\n' "$PRESS_CONTEXT" > "$PRESS_FILE"

482 - python3 - <<'PY' "$PRESS_FILE"

482 + python3 - <<'PY' "$PRESS_FILE" "$GITHUB_OUTPUT"

483 import sys

484 from pathlib import Path

485 - from scripts.render_press_context import estimate_tokens

485 + from scripts.render_press_context import press_token_estimate

486

487 path = Path(sys.argv[1])

488 + github_output = Path(sys.argv[2])

489 content = path.read_text(encoding="utf-8")

490 + token_estimate = press_token_estimate(content)

491 print(

492 "::notice::Press context "

493 f"size_bytes={path.stat().st_size} "

492 - f"token_estimate={estimate_tokens(content)}"

494 + f"token_estimate={token_estimate}"

495 )

496 + with github_output.open("a", encoding="utf-8") as output:

497 + print(f"press_token_estimate={token_estimate}", file=output)

498 PY

499 echo "press_file=$PRESS_FILE" >> "$GITHUB_OUTPUT"

500

@@ -658,6 +662,7 @@ jobs:

662 IN_MANIFEST_FILE: ${{ steps.analysis-context.outputs.publish_manifest_file }}

663 IN_PUBLISHED_SUMMARY: ${{ steps.analysis-context.outputs.published_output_file }}

664 IN_PRESS_FILE: ${{ steps.press-context.outputs.press_file }}

665 + IN_PRESS_TOKEN_ESTIMATE: ${{ steps.press-context.outputs.press_token_estimate }}

666 IN_ANALYSIS_PATH: ${{ inputs.analysis_path || 'single-pass' }}

667 IN_RUN_MODE: ${{ steps.analysis-context.outputs.run_mode }}

668 IN_PROMPT_FILE: ${{ steps.prompt-preflight.outputs.prompt_file }}

@@ -671,6 +676,8 @@ jobs:

676 MANIFEST_FILE="$IN_MANIFEST_FILE"

677 PUBLISHED_SUMMARY="$IN_PUBLISHED_SUMMARY"

678 PRESS_FILE="$IN_PRESS_FILE"

679 + PRESS_TOKEN_ESTIMATE="${IN_PRESS_TOKEN_ESTIMATE:-0}"

680 + case "$PRESS_TOKEN_ESTIMATE" in ''|*[!0-9]*) PRESS_TOKEN_ESTIMATE=0 ;; esac

681 ANALYSIS_PATH="$IN_ANALYSIS_PATH"

682 RUN_MODE="$IN_RUN_MODE"

683 ANALYSIS_STARTED=$(date +%s)

@@ -690,6 +697,8 @@ jobs:

697 --source "$1" \

698 --model "$2" \

699 --repair-safe \

700 + --press-context-path "$PRESS_FILE" \

701 + --press-token-estimate "$PRESS_TOKEN_ESTIMATE" \

702 --report-json "$3"

703 }

704

@@ -910,13 +919,20 @@ jobs:

919 RAW_JSON_FILE: ${{ steps.analysis-context.outputs.week_file }}

920 CURRENT_DATETIME: ${{ steps.analysis-context.outputs.current_datetime }}

921 GATE_REPORT: ${{ steps.analysis-context.outputs.analysis_gate_report_file }}

922 + PRESS_FILE: ${{ steps.press-context.outputs.press_file }}

923 + IN_PRESS_TOKEN_ESTIMATE: ${{ steps.press-context.outputs.press_token_estimate }}

924 run: |

925 + set -euo pipefail

926 + IN_PRESS_TOKEN_ESTIMATE="${IN_PRESS_TOKEN_ESTIMATE:-0}"

927 + case "$IN_PRESS_TOKEN_ESTIMATE" in ''|*[!0-9]*) IN_PRESS_TOKEN_ESTIMATE=0 ;; esac

928 python3 scripts/analysis_gate.py \

929 --analysis-file "$ANALYSIS_FILE" \

930 --raw-json "$RAW_JSON_FILE" \

931 --current-datetime "$CURRENT_DATETIME" \

932 --source "$ANALYSIS_SOURCE" \

933 --model "$ANALYSIS_MODEL" \

934 + --press-context-path "$PRESS_FILE" \

935 + --press-token-estimate "$IN_PRESS_TOKEN_ESTIMATE" \

936 --report-json "$GATE_REPORT"

937

938 - name: Emit publish eligibility manifest

docs/operator-guide.md

+59

@@ -305,6 +305,65 @@ hugo --minify

305

306 Output: `public/` directory ready for GitHub Pages.

307

308 +### Option D: Analysis-only rerun (reuse crawl artifacts, no re-crawl)

309 +

310 +When a report is wrong because of an **analysis/rendering bug** (not a data

311 +problem) — for example the report claims "No industry press data was available"

312 +even though the press crawl succeeded — you can regenerate the week's analysis

313 +**without re-crawling GitHub and without re-crawling press**. This reuses the

314 +existing immutable crawl artifacts:

315 +

316 +- `data/raw/<WEEK>.json` — GitHub raw payload

317 +- `data/raw/<WEEK>-external-news.json` — press/RSS crawl output

318 +- `data/analyzed/<WEEK>-correlations.json` — press↔repo correlations

319 +- `data/analyzed/<WEEK>-press-context.md` — rendered press context

320 +

321 +**Step 1 (optional) — regenerate press context from existing crawl artifacts**

322 +(press-only; no network calls). Reruns correlation + rendering only:

323 +

324 +```bash

325 +python3 scripts/correlate.py \

326 + --raw data/raw/2026-W30.json \

327 + --techcrunch data/raw/2026-W30-external-news.json \

328 + --output data/analyzed/2026-W30-correlations.json

329 +

330 +python3 scripts/render_press_context.py --week 2026-W30 \

331 + > data/analyzed/2026-W30-press-context.md

332 +```

333 +

334 +**Step 2 — regenerate the analysis prompt/report from existing artifacts**

335 +(no GitHub crawl, no press crawl). Pass the existing raw JSON and press context:

336 +

337 +```bash

338 +python3 scripts/analyze_fallback.py \

339 + --raw-json data/raw/2026-W30.json \

340 + --output data/analyzed/2026-W30-summary.md \

341 + --current-datetime 2026-07-27T12:00:00Z \

342 + --press-context data/analyzed/2026-W30-press-context.md \

343 + --print-prompt # inspect the prompt; drop this flag + wire Copilot CLI to emit the report

344 +```

345 +

346 +The rendered prompt includes a `## Press Context` block only when the press

347 +context contains real press data — even when a Step-1 synthesis narrative is

348 +supplied via `--synthesis-input` (the narrative condenses *historical* context

349 +but never replaces real press data). A press-less week still renders the

350 +non-empty `NO_PRESS_SENTINEL` marker (defined in

351 +`scripts/render_press_context.py`), which `analyze_fallback.py` treats as absent;

352 +the block is suppressed and the `press_correlations` component is recorded as

353 +`included: false`. Confirm whether real press reached the prompt with:

354 +

355 +```bash

356 +python3 scripts/analyze_fallback.py ... --print-prompt | grep -c "## Press Context"

357 +```

358 +

359 +A non-zero count means the "Where Industry Meets Code" and "Press & Industry"

360 +sections will be written from real press data. A zero count on a press-less week

361 +is expected: the sentinel was correctly suppressed. To distinguish real press

362 +from the marker, use `--preflight-report-json` to audit the

363 +`press_correlations` component (`included: true` for real press, `false` for

364 +no press), and consult the `NO_PRESS_SENTINEL` symbol in

365 +`scripts/render_press_context.py` as the authoritative marker definition.

366 +

367 ## Understanding Source Artifacts and Reuse

368

369 ### Source artifact tracking

prompts/analyze-weekly.md

+1 -1

@@ -190,7 +190,7 @@ is the story of the period, not a list.

190

191 16. Keep the section scope aligned with the spec:

192 - `## This Week's Trends`: ~200-350 words. Name 3-5 macro trends of the week. Each trend should have a name, a 1-2 sentence explanation of what it is, and why it matters to practitioners now. Do not just list repos — synthesize across them. Reference specific repos as evidence using `[owner/repo](https://github.com/owner/repo)`.

193 - - `## Where Industry Meets Code`: ~150-250 words. Compare press coverage (TechCrunch or other provided press data) against what developers are actually building. Highlight 2-4 correlations (where press and developer activity align) and call out 2-3 divergences (topics in the press with no dev traction, and developer work the press is ignoring). If no press data was provided, state: "No industry press data was available for this week's analysis." and focus on what the developer activity alone reveals.

193 + - `## Where Industry Meets Code`: ~150-250 words. Compare press coverage (TechCrunch or other provided press data) against what developers are actually building. Highlight 2-4 correlations (where press and developer activity align) and call out 2-3 divergences (topics in the press with no dev traction, and developer work the press is ignoring). Whenever a `## Press Context` block or an industry narrative is present below, treat it as real press data and write this section from it — do NOT claim press data is missing. Only when no press context and no industry narrative are provided at all, state: "No industry press data was available for this week's analysis." and focus on what the developer activity alone reveals.

194 - `## Signal & Noise`: ~150-260 words. Integrated analysis — what is real versus hype. Do not use Signal/Noise as separate sub-headings; write it as coherent editorial prose that distinguishes durable patterns from inflated, low-substance, or marketing-driven activity. Name names. Reference repos as evidence.

195 - `## Blind Spots`: ~80-160 words. Identify 2-4 meaningful absences from both press coverage AND developer attention. Be specific and concrete — name the missing category, why it matters, and what its absence signals.

196 - `## The Week Ahead`: ~50-110 words. Forward-looking editorial close. What should readers watch for next week? What trends are in motion that haven't peaked yet? Where is the ecosystem heading based on this week's evidence?

scripts/analysis_gate.py

+92 -3

@@ -11,6 +11,8 @@ from datetime import UTC, datetime

11 from pathlib import Path

12 from typing import Any

14 +from scripts.render_press_context import NO_PRESS_SENTINEL_MARKER

15 +

16 try: # pragma: no cover - optional dependency on runners

17 import yaml

18 except ImportError: # pragma: no cover - exercised via fallback parser

@@ -110,6 +112,22 @@ CONTRADICTION_PATTERNS = [

112 "claims no meaningful developer activity while citing active repositories.",

113 ),

114 ]

115 +# "No press data" style statements that must NOT appear in the published body when

116 +# the week's press context is actually populated. Unlike CONTRADICTION_PATTERNS these

117 +# fire even when the body does not self-contradict (no other press discussion present) —

118 +# they catch the false-negative where a populated press-context.md is silently dropped.

119 +STALE_PRESS_CLAIM_PATTERNS = [

120 + (

121 + re.compile(r"no industry press data was available", re.IGNORECASE),

122 + 'body states "No industry press data was available for this week\'s analysis." '

123 + "while a populated press context exists for this week.",

124 + ),

125 + (

126 + re.compile(r"no press data was provided this week", re.IGNORECASE),

127 + 'Key References state "No press data was provided this week." '

128 + "while a populated press context exists for this week.",

129 + ),

130 +]

131

132

133 def parse_args(argv: list[str] | None = None) -> argparse.Namespace:

@@ -135,6 +153,20 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:

153 help="Apply deterministic frontmatter/schema repairs before final validation.",

154 )

155 parser.add_argument("--report-json", type=Path, help="Write a machine-readable gate report.")

156 + parser.add_argument(

157 + "--press-context-path",

158 + type=Path,

159 + default=None,

160 + help="Path to the week's rendered press-context.md. When populated, the gate fails "

161 + 'analyses that still claim "no press data".',

162 + )

163 + parser.add_argument(

164 + "--press-token-estimate",

165 + type=int,

166 + default=None,

167 + help="Optional token estimate for the week's press context; a value > 0 marks the "

168 + "press context as populated even without the rendered file.",

169 + )

170 return parser.parse_args(argv)

171

172

@@ -616,6 +648,48 @@ def contradiction_errors(body: str) -> list[str]:

648 return errors

649

650

651 +def press_context_is_populated(

652 + press_context_path: Path | None, token_estimate: int | None = None

653 +) -> bool:

654 + """Return True when the week has real press context to write from.

655 +

656 + render_press_context.py emits a non-empty "No press data available for this week."

657 + sentinel when press is absent, so a bare size/non-empty check is insufficient: the

658 + sentinel is treated as an empty press context.

659 +

660 + When a press_context_path is provided its content is authoritative: sentinel

661 + content wins over a positive token_estimate, so a genuinely press-less week is

662 + classified as *not* populated even if token_estimate > 0. The token_estimate

663 + fallback only applies when no usable path is provided (path is None, missing,

664 + empty, or unreadable).

665 + """

666 + if press_context_path is not None:

667 + try:

668 + if press_context_path.exists() and press_context_path.stat().st_size > 0:

669 + content = press_context_path.read_text(encoding="utf-8").strip()

670 + if content:

671 + return not NO_PRESS_SENTINEL_MARKER.search(content)

672 + except OSError:

673 + pass

674 + return token_estimate is not None and token_estimate > 0

675 +

676 +

677 +def stale_press_claim_errors(body: str, *, press_context_available: bool) -> list[str]:

678 + """Fail analyses that claim "no press data" while press context is populated.

679 +

680 + This is defense-in-depth against the 2026-W30 regression where a populated

681 + press-context.md was silently dropped and the body shipped

682 + "No industry press data was available for this week's analysis.".

683 + """

684 + if not press_context_available:

685 + return []

686 + errors: list[str] = []

687 + for pattern, message in STALE_PRESS_CLAIM_PATTERNS:

688 + if pattern.search(body):

689 + errors.append(f"stale press claim: {message}")

690 + return errors

691 +

692 +

693 def ai_provenance_errors(source: str, model: str) -> list[str]:

694 errors: list[str] = []

695 normalized_source = source.strip()

@@ -635,7 +709,7 @@ def categorize_gate_error(error: str) -> str:

709 ):

710 return "evidence_citation"

711 if (

638 - error.startswith(("editorial analysis", "contradictory claim"))

712 + error.startswith(("editorial analysis", "contradictory claim", "stale press claim"))

713 or "section is too thin" in error

714 or "must explain why" in error

715 ):

@@ -665,6 +739,7 @@ def validate_publish_quality(

739 *,

740 source: str,

741 model: str,

742 + press_context_available: bool = False,

743 ) -> tuple[list[str], dict[str, dict[str, Any]]]:

744 try:

745 _, body = extract_frontmatter(text)

@@ -676,6 +751,9 @@ def validate_publish_quality(

751 errors.extend(evidence_citation_errors(body, raw_payload))

752 errors.extend(editorial_quality_errors(body))

753 errors.extend(contradiction_errors(body))

754 + errors.extend(

755 + stale_press_claim_errors(body, press_context_available=press_context_available)

756 + )

757 return errors, build_gate_results(errors)

758

759

@@ -874,9 +952,16 @@ def main(argv: list[str] | None = None) -> int:

952

953 text = args.analysis_file.read_text(encoding="utf-8")

954 raw_payload = load_json(args.raw_json)

955 + press_context_available = press_context_is_populated(

956 + args.press_context_path, args.press_token_estimate

957 + )

958 errors_before, word_count = validate_analysis(text, raw_payload, args.current_datetime)

959 publish_errors_before, _ = validate_publish_quality(

879 - text, raw_payload, source=args.source, model=args.model

960 + text,

961 + raw_payload,

962 + source=args.source,

963 + model=args.model,

964 + press_context_available=press_context_available,

965 )

966 combined_errors_before = errors_before + [

967 error for error in publish_errors_before if error not in errors_before

@@ -900,7 +985,11 @@ def main(argv: list[str] | None = None) -> int:

985 file=sys.stderr,

986 )

987 publish_errors, _ = validate_publish_quality(

903 - text, raw_payload, source=args.source, model=args.model

988 + text,

989 + raw_payload,

990 + source=args.source,

991 + model=args.model,

992 + press_context_available=press_context_available,

993 )

994 errors = errors + [error for error in publish_errors if error not in errors]

995 gate_results = build_gate_results(errors)

scripts/analyze_fallback.py

+50 -6

@@ -960,6 +960,13 @@ def render_synthesis_prompt(

960 else ""

961 )

962

963 + # The no-press sentinel is a NON-EMPTY string; treat it as ABSENT so the

964 + # synthesis narrative is not built from a fake "## Press Context".

965 + from scripts.render_press_context import NO_PRESS_SENTINEL_MARKER

966 +

967 + if press_content and NO_PRESS_SENTINEL_MARKER.search(press_content):

968 + press_content = ""

969 +

970 if press_content:

971 press_content = _strip_ai_instruction_blocks(press_content)

972 if press_content:

@@ -1034,6 +1041,13 @@ def run_synthesis_step(

1041 else ""

1042 )

1043

1044 + # The no-press sentinel is a NON-EMPTY string; treat it as ABSENT so

1045 + # synthesis does not treat the sentinel as real press.

1046 + from scripts.render_press_context import NO_PRESS_SENTINEL_MARKER

1047 +

1048 + if press_content and NO_PRESS_SENTINEL_MARKER.search(press_content):

1049 + press_content = ""

1050 +

1051 # Strip AI-only instruction blocks from press context before synthesis

1052 if press_content:

1053 press_content = _strip_ai_instruction_blocks(press_content)

@@ -1209,12 +1223,37 @@ def _build_prompt(

1223 and press_context_path.stat().st_size > 0

1224 else ""

1225 )

1212 - # When a synthesis narrative is available (Step 1 output), it replaces

1213 - # the raw press context and historical context — those were already

1214 - # distilled into the narrative. This dramatically reduces token count.

1215 - if synthesis_narrative:

1216 - historical_context_content = f"[Industry narrative synthesized from press & historical context]\n\n{synthesis_narrative}"

1226 + # The no-press sentinel is a NON-EMPTY string, so treat it as ABSENT here:

1227 + # blanking it keeps the "press exists" logic (## Press Context block,

1228 + # included=bool(press_content), press_decision) and the required no-press

1229 + # statement correct for genuinely press-less weeks.

1230 + from scripts.render_press_context import NO_PRESS_SENTINEL_MARKER

1231 +

1232 + if press_content and NO_PRESS_SENTINEL_MARKER.search(press_content):

1233 press_content = ""

1234 + if press_content:

1235 + press_content = _strip_ai_instruction_blocks(press_content)

1236 + press_content = _escape_untrusted_boundaries(press_content)

1237 + # When a synthesis narrative is available (Step 1 output), it distils the

1238 + # *historical* context into a compact narrative that replaces the bulky

1239 + # historical context block and saves tokens. It must NOT drop the press

1240 + # context: the Step-2 sections "Where Industry Meets Code" and

1241 + # "Press & Industry" still have to be written from the real press data.

1242 + # jmservera/SquadScope#515 blanked press_content here, which silently

1243 + # dropped a populated press context and forced the model to emit

1244 + # "No industry press data was available...". Keep a condensed press

1245 + # context so those sections stay evidence-backed.

1246 + press_condensed_for_synthesis = False

1247 + if synthesis_narrative:

1248 + synthesis_source = "press & historical context" if press_content else "historical context"

1249 + historical_context_content = (

1250 + f"[Industry narrative synthesized from {synthesis_source}]\n\n{synthesis_narrative}"

1251 + )

1252 + if press_content and len(press_content) > COMPACTED_PRESS_CONTEXT_CHARS:

1253 + press_content, _ = truncate_with_notice(

1254 + press_content, COMPACTED_PRESS_CONTEXT_CHARS, "press context"

1255 + )

1256 + press_condensed_for_synthesis = True

1257 payload_for_prompt = sanitized_payload

1258 raw_decisions = {"new_repos": "included", "trending_repos": "included"}

1259 previous_decision = "included" if previous_summary_path else "not included: no previous summary"

@@ -1237,7 +1276,12 @@ def _build_prompt(

1276 if continuity_file.exists()

1277 else "not included: no analysis-specific continuity capsule"

1278 )

1240 - press_decision = "included" if press_content else "not included: no press context"

1279 + if not press_content:

1280 + press_decision = "not included: no press context"

1281 + elif press_condensed_for_synthesis:

1282 + press_decision = "included: condensed alongside synthesis narrative"

1283 + else:

1284 + press_decision = "included"

1285 degraded = False

1286

1287 def assemble() -> str:

scripts/render_press_context.py

+17 -1

@@ -29,6 +29,14 @@ MAX_RENDERED_ARTICLES = 40

29 MAX_RENDERED_CORRELATIONS = 20

30 GITHUB_REPO_FULL_NAME_RE = re.compile(r"^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$")

32 +# Single source of truth for the "no press this week" sentinel. This is a

33 +# NON-EMPTY string, so downstream code must use NO_PRESS_SENTINEL_MARKER to

34 +# detect it rather than treating any non-empty press_content as real press.

35 +NO_PRESS_SENTINEL = (

36 + "No press data available for this week. Analyze repos based on GitHub signals only."

37 +)

38 +NO_PRESS_SENTINEL_MARKER = re.compile(r"no press data available for this week", re.IGNORECASE)

39 +

41 def validate_https_url(url: str, *, label: str) -> None:

42 parsed = urlparse(url)

@@ -614,6 +622,14 @@ def estimate_tokens(markdown: str) -> int:

622 return max(1, (len(markdown) + 3) // 4)

623

624

625 +def press_token_estimate(content: str) -> int:

626 + """Return 0 for empty content or no-press sentinel so gate fallback matches path logic."""

627 + stripped = content.strip()

628 + if not stripped or NO_PRESS_SENTINEL_MARKER.search(stripped):

629 + return 0

630 + return estimate_tokens(stripped)

631 +

632 +

633 def enforce_press_context_budget(markdown: str) -> str:

634 """Keep press context below the documented token budget."""

635 if estimate_tokens(markdown) <= PRESS_CONTEXT_TOKEN_BUDGET:

@@ -649,7 +665,7 @@ def render_press_context(

665 Rendered markdown prompt section.

666 """

667 if techcrunch_data is None and correlation_data is None:

652 - return "No press data available for this week. Analyze repos based on GitHub signals only."

668 + return NO_PRESS_SENTINEL

669

670 template_path = _REPO_ROOT / "prompts" / "analyze-press-context.md"

671 template = template_path.read_text(encoding="utf-8")

scripts/techcrunch_crawler.py

+111 -6

@@ -15,16 +15,20 @@ import argparse

15 import hashlib

16 import ipaddress

17 import json

18 +import math

19 import os

20 import re

21 +import secrets

22 import sys

23 import time

24 from collections import Counter

25 from concurrent.futures import ThreadPoolExecutor, as_completed

26 from dataclasses import dataclass

25 -from datetime import UTC, date, datetime, timedelta

27 +from datetime import UTC, date, datetime, timedelta, timezone

28 +from email.utils import parsedate_to_datetime

29 from pathlib import Path

30 from typing import Any

31 +from urllib.error import HTTPError, URLError

32 from urllib.parse import urlparse

33 from urllib.request import Request, urlopen

@@ -43,8 +47,18 @@ from scripts.topic_paths import raw_dir

47 FEED_URL = "https://techcrunch.com/feed/"

48 DEFAULT_SOURCES_PATH = Path("config/external_news_sources.json")

49 DEFAULT_FETCH_TIMEOUT_SECONDS = 15

46 -DEFAULT_FETCH_RETRIES = 1

50 +DEFAULT_FETCH_RETRIES = 3

51 DEFAULT_MAX_WORKERS = 8

52 +# HTTP statuses worth retrying on a transient failure; everything else (e.g.

53 +# 400/401/404) is treated as permanent and fails the single source fast.

54 +RETRYABLE_STATUSES = frozenset({403, 408, 429, 500, 502, 503, 504})

55 +# Exponential backoff bounds for per-source retries (mirrors scripts/crawl.py).

56 +RETRY_BASE_DELAY_SECONDS = 2.0

57 +RETRY_MAX_DELAY_SECONDS = 30.0

58 +# Bound hostile/absurd Retry-After values while honoring reasonable server

59 +# requests well above the crawler's computed 30s backoff cap.

60 +RETRY_AFTER_MAX_SECONDS = 120.0

61 +_JITTER_RANDOM = secrets.SystemRandom()

62 CANONICAL_SCHEMA_VERSION = 2

63 APPROVED_FEED_HOSTS = frozenset(

64 {

@@ -739,15 +753,69 @@ def parse_published_date(entry: Any) -> datetime | None:

753 return None

754

755

756 +def _sleep_before_retry(attempt: int, retry_after: float | None = None) -> float:

757 + """Sleep with exponential backoff + jitter before the next fetch attempt.

758 +

759 + Server-supplied Retry-After is honored up to RETRY_AFTER_MAX_SECONDS, while

760 + computed backoff is bounded by RETRY_MAX_DELAY_SECONDS. Returns the delay

761 + slept so callers/tests can reason about it.

762 + """

763 + if retry_after and retry_after > 0:

764 + delay = min(retry_after, RETRY_AFTER_MAX_SECONDS)

765 + else:

766 + base_delay = min(

767 + RETRY_BASE_DELAY_SECONDS * (2**attempt),

768 + RETRY_MAX_DELAY_SECONDS,

769 + )

770 + delay = min(

771 + base_delay + _JITTER_RANDOM.uniform(0.3, 1.7),

772 + RETRY_MAX_DELAY_SECONDS,

773 + )

774 + time.sleep(delay)

775 + return delay

776 +

777 +

778 +def _retry_after_seconds(exc: HTTPError) -> float | None:

779 + """Extract a positive Retry-After delay from delta-seconds or HTTP-date."""

780 + header = None

781 + try:

782 + header = exc.headers.get("Retry-After") if exc.headers else None

783 + except AttributeError:

784 + header = None

785 + if not header:

786 + return None

787 + try:

788 + value = float(header)

789 + if math.isfinite(value):

790 + return max(value, 1.0)

791 + except (TypeError, ValueError):

792 + pass

793 + try:

794 + retry_at = parsedate_to_datetime(header)

795 + except (TypeError, ValueError):

796 + return None

797 + if retry_at.tzinfo is None:

798 + retry_at = retry_at.replace(tzinfo=timezone.utc)

799 + delay = (retry_at - datetime.now(timezone.utc)).total_seconds()

800 + return max(delay, 1.0)

801 +

802 +

803 def fetch_feed(

804 url: str = FEED_URL,

805 retries: int = DEFAULT_FETCH_RETRIES,

806 timeout: int = DEFAULT_FETCH_TIMEOUT_SECONDS,

807 ) -> Any:

747 - """Fetch and parse RSS feed with bounded retries and an explicit timeout."""

808 + """Fetch and parse an RSS feed with bounded retries and exponential backoff.

809 +

810 + Transient failures (network errors, retryable HTTP statuses, empty/bozo

811 + feeds) are retried with exponential backoff + jitter. Permanent HTTP errors

812 + (e.g. 404) fail fast. Feed bytes are always treated as UNTRUSTED content and

813 + are only parsed, never executed.

814 + """

815 validate_feed_url(url)

816 if timeout <= 0:

817 raise ValueError("RSS fetch timeout must be greater than zero")

818 + feed: Any = None

819 for attempt in range(retries + 1):

820 try:

821 request = Request(url, headers={"User-Agent": "SquadScope RSS crawler"})

@@ -755,14 +823,20 @@ def fetch_feed(

823 feed = feedparser.parse(response.read())

824 setattr(feed, "squad_fetch_attempts", attempt + 1)

825 setattr(feed, "squad_fetch_timeout_seconds", timeout)

758 - except Exception:

826 + except HTTPError as exc:

827 + # Only retry transient HTTP statuses; permanent errors fail fast.

828 + if exc.code in RETRYABLE_STATUSES and attempt < retries:

829 + _sleep_before_retry(attempt, _retry_after_seconds(exc))

830 + continue

831 + raise

832 + except (URLError, TimeoutError, OSError):

833 if attempt < retries:

760 - time.sleep(2)

834 + _sleep_before_retry(attempt)

835 continue

836 raise

837 if feed.bozo and not feed.entries:

838 if attempt < retries:

765 - time.sleep(2)

839 + _sleep_before_retry(attempt)

840 continue

841 # Return partial result even on failure

842 setattr(feed, "squad_fetch_attempts", attempt + 1)

@@ -1382,6 +1456,37 @@ def main(argv: list[str] | None = None) -> int:

1456 refreshed_count = sum(

1457 1 for item in output["metadata"]["source_reuse_summary"] if item["action"] != "reused"

1458 )

1459 +

1460 + # Explicit per-source success/failure summary on stdout so a partial crawl

1461 + # (one source failing while others succeed) is diagnosable straight from CI

1462 + # logs. The same detail is persisted in metadata.source_status / sources_failed.

1463 + succeeded_sources = list(output["metadata"]["sources_succeeded"])

1464 + failed_sources = list(output["metadata"]["sources_failed"])

1465 + print(

1466 + f"[external-news] per-source summary: "

1467 + f"{len(succeeded_sources)} succeeded, {len(failed_sources)} failed "

1468 + f"(requested {len(output['metadata']['sources_requested'])})"

1469 + )

1470 + if succeeded_sources:

1471 + print(f"[external-news] succeeded: {', '.join(sorted(succeeded_sources))}")

1472 + if failed_sources:

1473 + status_by_source = {

1474 + str(status.get("source")): status for status in output["metadata"]["source_status"]

1475 + }

1476 + for source_name in sorted(failed_sources):

1477 + status = status_by_source.get(source_name, {})

1478 + reason = (

1479 + str(status.get("error_message") or status.get("error_class") or "unknown error")

1480 + .replace("\n", " ")

1481 + .strip()

1482 + )

1483 + attempts = status.get("attempts", "?")

1484 + print(f"[external-news] FAILED: {source_name} attempts={attempts} reason={reason}")

1485 + print(

1486 + "::warning::external-news crawl completed with partial results; "

1487 + f"{len(failed_sources)} source(s) failed: {', '.join(sorted(failed_sources))}"

1488 + )

1489 +

1490 print(

1491 f"Crawled {output['metadata']['total_articles']} articles "

1492 f"from {output['metadata']['source_count']} sources "

tests/test_analysis_gate.py

+208 -1

@@ -5,6 +5,7 @@ from pathlib import Path

5 from unittest import mock

7 import scripts.analysis_gate as analysis_gate

8 +from scripts.render_press_context import NO_PRESS_SENTINEL, press_token_estimate

10 RAW_PAYLOAD = {"week": "2026-W23"}

11 RAW_PAYLOAD_WITH_REPOS = {

@@ -373,7 +374,12 @@ summary: "A grounded week focused on practical tools."'''.strip()

374 raw_path.write_text(json.dumps(RAW_PAYLOAD_WITH_REPOS), encoding="utf-8")

375

376 def publish_quality_for(

376 - text: str, raw_payload: dict, *, source: str, model: str

377 + text: str,

378 + raw_payload: dict,

379 + *,

380 + source: str,

381 + model: str,

382 + press_context_available: bool = False,

383 ) -> tuple[list[str], dict]:

384 if text == original_text:

385 return ["pre-repair publish-quality failure"], analysis_gate.build_gate_results(

@@ -636,6 +642,207 @@ No press data was provided this week.

642 self.assertTrue(any("contradictory claim" in error for error in errors))

643 self.assertFalse(gates["editorial_quality"]["passed"])

644

645 + def test_stale_press_claim_fails_when_press_context_available(self) -> None:

646 + """Regression (2026-W30): body claims no press data while a populated press

647 + context exists → gate must fail with a 'stale press claim:' error."""

648 + body = "No industry press data was available for this week's analysis."

649 + errors = analysis_gate.stale_press_claim_errors(body, press_context_available=True)

650 + self.assertTrue(any(error.startswith("stale press claim:") for error in errors))

651 +

652 + def test_stale_press_claim_ignored_when_no_press_context(self) -> None:

653 + """Legitimately press-less weeks must NOT false-positive: the same body with

654 + press_context_available=False produces no stale-press error."""

655 + body = "No industry press data was available for this week's analysis."

656 + errors = analysis_gate.stale_press_claim_errors(body, press_context_available=False)

657 + self.assertEqual(errors, [])

658 +

659 + def test_stale_press_claim_catches_key_references_variant(self) -> None:

660 + """The 'No press data was provided this week.' Key References phrasing is also

661 + caught when a populated press context exists."""

662 + body = "### Press & Industry\n\nNo press data was provided this week."

663 + errors = analysis_gate.stale_press_claim_errors(body, press_context_available=True)

664 + self.assertTrue(any(error.startswith("stale press claim:") for error in errors))

665 + errors_no_press = analysis_gate.stale_press_claim_errors(

666 + body, press_context_available=False

667 + )

668 + self.assertEqual(errors_no_press, [])

669 +

670 + def test_publish_quality_gate_fails_on_stale_press_claim_with_press_available(self) -> None:

671 + """End-to-end: validate_publish_quality wires stale_press_claim_errors into the

672 + editorial_quality gate when press context is available."""

673 + body = make_body() + "\n\nNo industry press data was available for this week's analysis."

674 + errors, gates = analysis_gate.validate_publish_quality(

675 + make_analysis(VALID_FRONTMATTER, body),

676 + RAW_PAYLOAD,

677 + source="copilot-cli",

678 + model="copilot-default",

679 + press_context_available=True,

680 + )

681 + self.assertTrue(any(error.startswith("stale press claim:") for error in errors))

682 + self.assertFalse(gates["editorial_quality"]["passed"])

683 +

684 + def test_publish_quality_gate_allows_no_press_body_when_press_absent(self) -> None:

685 + """The same body passes the stale-press rule when there is genuinely no press

686 + context (the default press_context_available=False)."""

687 + body = make_body() + "\n\nNo industry press data was available for this week's analysis."

688 + errors, gates = analysis_gate.validate_publish_quality(

689 + make_analysis(VALID_FRONTMATTER, body),

690 + RAW_PAYLOAD,

691 + source="copilot-cli",

692 + model="copilot-default",

693 + press_context_available=False,

694 + )

695 + self.assertFalse(any(error.startswith("stale press claim:") for error in errors))

696 +

697 + def test_press_context_is_populated_detects_real_and_empty_context(self) -> None:

698 + """The helper treats missing/empty files and the render sentinel as empty, but

699 + real content (or a positive token estimate) as populated."""

700 + self.assertFalse(analysis_gate.press_context_is_populated(None))

701 + tests_root = Path(__file__).resolve().parent

702 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

703 + base = Path(tmpdir)

704 +

705 + missing = base / "missing-press-context.md"

706 + self.assertFalse(analysis_gate.press_context_is_populated(missing))

707 +

708 + empty = base / "empty-press-context.md"

709 + empty.write_text("", encoding="utf-8")

710 + self.assertFalse(analysis_gate.press_context_is_populated(empty))

711 +

712 + sentinel = base / "sentinel-press-context.md"

713 + sentinel.write_text("No press data available for this week.", encoding="utf-8")

714 + self.assertFalse(analysis_gate.press_context_is_populated(sentinel))

715 +

716 + real = base / "real-press-context.md"

717 + real.write_text(

718 + "## Press Context\n\n22 relevant articles about AI agents.",

719 + encoding="utf-8",

720 + )

721 + self.assertTrue(analysis_gate.press_context_is_populated(real))

722 +

723 + # A positive token estimate short-circuits to populated even without a file.

724 + self.assertTrue(analysis_gate.press_context_is_populated(missing, token_estimate=42))

725 +

726 + def test_press_context_is_populated_sentinel_wins_over_token_estimate(self) -> None:

727 + """Regression (2026-W30 press-less path): a provided path is authoritative, so a

728 + sentinel-only press file is *not* populated even when token_estimate > 0, while a

729 + real press file is. The token_estimate fallback only applies when no usable path

730 + is provided."""

731 + tests_root = Path(__file__).resolve().parent

732 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

733 + base = Path(tmpdir)

734 +

735 + sentinel = base / "sentinel-press-context.md"

736 + sentinel.write_text(NO_PRESS_SENTINEL, encoding="utf-8")

737 + self.assertFalse(analysis_gate.press_context_is_populated(sentinel, token_estimate=500))

738 +

739 + real = base / "real-press-context.md"

740 + real.write_text(

741 + "## Press Context\n\n22 relevant articles about AI agents.",

742 + encoding="utf-8",

743 + )

744 + self.assertTrue(analysis_gate.press_context_is_populated(real, token_estimate=500))

745 +

746 + # token_estimate fallback applies only when no usable path is provided.

747 + self.assertTrue(analysis_gate.press_context_is_populated(None, token_estimate=500))

748 + self.assertFalse(analysis_gate.press_context_is_populated(None, 0))

749 + self.assertFalse(analysis_gate.press_context_is_populated(None, None))

750 +

751 + def test_press_context_token_estimate_is_fallback_only(self) -> None:

752 + """Positive token estimates populate only when no readable authoritative content exists."""

753 + self.assertTrue(analysis_gate.press_context_is_populated(None, token_estimate=1))

754 + self.assertFalse(analysis_gate.press_context_is_populated(None, token_estimate=0))

755 + self.assertFalse(analysis_gate.press_context_is_populated(None, token_estimate=None))

756 +

757 + tests_root = Path(__file__).resolve().parent

758 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

759 + base = Path(tmpdir)

760 +

761 + missing = base / "missing-press-context.md"

762 + self.assertTrue(analysis_gate.press_context_is_populated(missing, token_estimate=1))

763 + self.assertFalse(analysis_gate.press_context_is_populated(missing, token_estimate=0))

764 + self.assertFalse(analysis_gate.press_context_is_populated(missing, token_estimate=None))

765 +

766 + empty = base / "empty-press-context.md"

767 + empty.write_text("", encoding="utf-8")

768 + self.assertTrue(analysis_gate.press_context_is_populated(empty, token_estimate=1))

769 + self.assertFalse(analysis_gate.press_context_is_populated(empty, token_estimate=0))

770 +

771 + unreadable = base / "unreadable-press-context.md"

772 + unreadable.write_text("content that cannot be read", encoding="utf-8")

773 + with mock.patch.object(Path, "read_text", side_effect=OSError("unreadable")):

774 + self.assertTrue(

775 + analysis_gate.press_context_is_populated(unreadable, token_estimate=1)

776 + )

777 +

778 + sentinel = base / "sentinel-press-context.md"

779 + sentinel.write_text(NO_PRESS_SENTINEL, encoding="utf-8")

780 + self.assertFalse(

781 + analysis_gate.press_context_is_populated(sentinel, token_estimate=9999)

782 + )

783 +

784 + real = base / "real-press-context.md"

785 + real.write_text(

786 + "## Press Context\n\n22 relevant articles about AI agents.",

787 + encoding="utf-8",

788 + )

789 + self.assertTrue(analysis_gate.press_context_is_populated(real, token_estimate=0))

790 +

791 + def test_press_context_fallback_uses_rendered_content_token_estimate(self) -> None:

792 + """The gate fallback consumes render_press_context.press_token_estimate, where

793 + empty/whitespace/sentinel content maps to 0 and real press content maps positive."""

794 + self.assertFalse(analysis_gate.press_context_is_populated(None, press_token_estimate("")))

795 + self.assertFalse(

796 + analysis_gate.press_context_is_populated(None, press_token_estimate(" \n\t "))

797 + )

798 + self.assertFalse(

799 + analysis_gate.press_context_is_populated(None, press_token_estimate(NO_PRESS_SENTINEL))

800 + )

801 + self.assertTrue(

802 + analysis_gate.press_context_is_populated(

803 + None,

804 + press_token_estimate("## Press Context\n\nA real article about AI agents."),

805 + )

806 + )

807 +

808 + def test_stale_press_gate_end_to_end_for_sentinel_pressless_week(self) -> None:

809 + """End-to-end press-less path: a body that legitimately states press was absent

810 + must NOT trip the stale-press rule when the week's press file is the sentinel

811 + (even with a positive token estimate); the identical body with a real press file

812 + still trips it, confirming the W30 regression stays caught."""

813 + body = "No industry press data was available for this week's analysis."

814 + tests_root = Path(__file__).resolve().parent

815 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

816 + base = Path(tmpdir)

817 + token_estimate = 500

818 +

819 + sentinel = base / "sentinel-press-context.md"

820 + sentinel.write_text(NO_PRESS_SENTINEL, encoding="utf-8")

821 + sentinel_available = analysis_gate.press_context_is_populated(

822 + sentinel, token_estimate=token_estimate

823 + )

824 + self.assertFalse(sentinel_available)

825 + self.assertEqual(

826 + analysis_gate.stale_press_claim_errors(

827 + body, press_context_available=sentinel_available

828 + ),

829 + [],

830 + )

831 +

832 + real = base / "real-press-context.md"

833 + real.write_text(

834 + "## Press Context\n\n22 relevant articles about AI agents.",

835 + encoding="utf-8",

836 + )

837 + real_available = analysis_gate.press_context_is_populated(

838 + real, token_estimate=token_estimate

839 + )

840 + self.assertTrue(real_available)

841 + errors = analysis_gate.stale_press_claim_errors(

842 + body, press_context_available=real_available

843 + )

844 + self.assertTrue(any(error.startswith("stale press claim:") for error in errors))

845 +

846

847 if __name__ == "__main__":

848 unittest.main()

tests/test_analyze_fallback.py

+305

@@ -9,6 +9,7 @@ from urllib import error

10 import scripts.analyze_fallback as analyze_fallback

11 import scripts.publish_manifest as publish_manifest

12 +from scripts.render_press_context import NO_PRESS_SENTINEL

15 class _FakeHTTPResponse(io.BytesIO):

@@ -726,6 +727,310 @@ class AnalyzeFallbackTests(unittest.TestCase):

727 markdown = analyze_fallback.call_github_models("prompt")

728 self.assertEqual(markdown, "# Summary\n")

729

730 + def test_synthesis_narrative_does_not_drop_press_context(self) -> None:

731 + """Regression (jmservera/SquadScope#515): a synthesis narrative must NOT blank

732 + a populated press context — Step-2 still needs real press data for the

733 + 'Where Industry Meets Code' and 'Press & Industry' sections."""

734 + tests_root = Path(__file__).resolve().parent

735 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

736 + base = Path(tmpdir)

737 + raw_path = base / "data" / "raw" / "2026-W30.json"

738 + prompt_template = base / "prompt.md"

739 + output_path = base / "data" / "analyzed" / "2026-W30-summary.md"

740 + press_path = base / "data" / "analyzed" / "2026-W30-press-context.md"

741 + synthesis_path = base / "synthesis.md"

742 + report_path = base / "diagnostics" / "preflight.json"

743 + raw_path.parent.mkdir(parents=True)

744 + output_path.parent.mkdir(parents=True)

745 + raw_path.write_text(

746 + json.dumps({"week": "2026-W30", "new_repos": [], "trending_repos": []}),

747 + encoding="utf-8",

748 + )

749 + prompt_template.write_text(

750 + "{{RAW_JSON_CONTENT}}\n{{HISTORICAL_CONTEXT}}", encoding="utf-8"

751 + )

752 + press_path.write_text(

753 + "## Press Context (External news, week of 2026-W30)\n\n"

754 + "UNIQUE_PRESS_MARKER: 22 relevant articles about AI agents.",

755 + encoding="utf-8",

756 + )

757 + synthesis_path.write_text(

758 + "Industry narrative distilled from press and history.", encoding="utf-8"

759 + )

760 +

761 + with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout:

762 + exit_code = analyze_fallback.main(

763 + [

764 + "--raw-json",

765 + str(raw_path),

766 + "--output",

767 + str(output_path),

768 + "--current-datetime",

769 + "2026-07-27T12:00:00Z",

770 + "--prompt-template",

771 + str(prompt_template),

772 + "--analyzed-dir",

773 + str(output_path.parent),

774 + "--wisdom-file",

775 + str(base / "w.md"),

776 + "--skills-dir",

777 + str(base / "s"),

778 + "--press-context",

779 + str(press_path),

780 + "--synthesis-input",

781 + str(synthesis_path),

782 + "--preflight-report-json",

783 + str(report_path),

784 + "--print-prompt",

785 + ]

786 + )

787 +

788 + self.assertEqual(exit_code, 0)

789 + rendered = stdout.getvalue()

790 + # Both the synthesis narrative and the real press data must be present.

791 + self.assertIn("Industry narrative distilled", rendered)

792 + self.assertIn(

793 + "[Industry narrative synthesized from press & historical context]",

794 + rendered,

795 + )

796 + # The Step-2 prompt must still carry a real "## Press Context" block.

797 + self.assertIn("## Press Context", rendered)

798 + self.assertIn("UNIQUE_PRESS_MARKER", rendered)

799 + # And the model must NOT be told there was no press data.

800 + self.assertNotIn("No industry press data was available", rendered)

801 +

802 + # Diagnostics must record the press context as *included* (not blanked).

803 + report = json.loads(report_path.read_text(encoding="utf-8"))

804 + components = {component["name"]: component for component in report["components"]}

805 + press_component = components["press_correlations"]

806 + self.assertTrue(press_component["included"])

807 + # Short press content is under the compaction threshold, so it is

808 + # included verbatim (not condensed and definitely not dropped).

809 + self.assertEqual(press_component["compaction_decision"], "included")

810 + self.assertGreater(press_component["token_estimate"], 0)

811 + self.assertGreater(press_component["bytes"], 0)

812 +

813 + def test_synthesis_narrative_condenses_but_keeps_large_press_context(self) -> None:

814 + """A synthesis narrative may *condense* an oversized press context, but must

815 + still keep the real press data in the Step-2 prompt (press_decision reflects

816 + 'included: condensed alongside synthesis narrative')."""

817 + tests_root = Path(__file__).resolve().parent

818 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

819 + base = Path(tmpdir)

820 + raw_path = base / "data" / "raw" / "2026-W30.json"

821 + prompt_template = base / "prompt.md"

822 + output_path = base / "data" / "analyzed" / "2026-W30-summary.md"

823 + press_path = base / "data" / "analyzed" / "2026-W30-press-context.md"

824 + synthesis_path = base / "synthesis.md"

825 + report_path = base / "diagnostics" / "preflight.json"

826 + raw_path.parent.mkdir(parents=True)

827 + output_path.parent.mkdir(parents=True)

828 + raw_path.write_text(

829 + json.dumps({"week": "2026-W30", "new_repos": [], "trending_repos": []}),

830 + encoding="utf-8",

831 + )

832 + prompt_template.write_text(

833 + "{{RAW_JSON_CONTENT}}\n{{HISTORICAL_CONTEXT}}", encoding="utf-8"

834 + )

835 + # Press context larger than COMPACTED_PRESS_CONTEXT_CHARS (14_000) so the

836 + # synthesis path condenses it. A leading marker must survive truncation.

837 + filler = "AI agents infrastructure launch coverage. " * 800

838 + press_path.write_text(

839 + "LEADING_PRESS_MARKER: 40 relevant articles.\n\n" + filler,

840 + encoding="utf-8",

841 + )

842 + synthesis_path.write_text(

843 + "Industry narrative distilled from press and history.", encoding="utf-8"

844 + )

845 +

846 + with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout:

847 + exit_code = analyze_fallback.main(

848 + [

849 + "--raw-json",

850 + str(raw_path),

851 + "--output",

852 + str(output_path),

853 + "--current-datetime",

854 + "2026-07-27T12:00:00Z",

855 + "--prompt-template",

856 + str(prompt_template),

857 + "--analyzed-dir",

858 + str(output_path.parent),

859 + "--wisdom-file",

860 + str(base / "w.md"),

861 + "--skills-dir",

862 + str(base / "s"),

863 + "--press-context",

864 + str(press_path),

865 + "--synthesis-input",

866 + str(synthesis_path),

867 + "--preflight-report-json",

868 + str(report_path),

869 + "--print-prompt",

870 + ]

871 + )

872 +

873 + self.assertEqual(exit_code, 0)

874 + rendered = stdout.getvalue()

875 + self.assertIn("## Press Context", rendered)

876 + self.assertIn("LEADING_PRESS_MARKER", rendered)

877 + self.assertNotIn("No industry press data was available", rendered)

878 +

879 + report = json.loads(report_path.read_text(encoding="utf-8"))

880 + components = {component["name"]: component for component in report["components"]}

881 + press_component = components["press_correlations"]

882 + self.assertTrue(press_component["included"])

883 + self.assertEqual(

884 + press_component["compaction_decision"],

885 + "included: condensed alongside synthesis narrative",

886 + )

887 + self.assertGreater(press_component["token_estimate"], 0)

888 + self.assertGreater(press_component["bytes"], 0)

889 +

890 + def test_synthesis_narrative_with_absent_press_context_takes_no_press_path(self) -> None:

891 + """The fix must not over-correct: when a synthesis narrative is present but

892 + there is genuinely NO press context, the no-press path stays intact —

893 + press_decision is 'not included: no press context' and no '## Press Context'

894 + block is emitted."""

895 + tests_root = Path(__file__).resolve().parent

896 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

897 + base = Path(tmpdir)

898 + raw_path = base / "data" / "raw" / "2026-W30.json"

899 + prompt_template = base / "prompt.md"

900 + output_path = base / "data" / "analyzed" / "2026-W30-summary.md"

901 + synthesis_path = base / "synthesis.md"

902 + report_path = base / "diagnostics" / "preflight.json"

903 + raw_path.parent.mkdir(parents=True)

904 + output_path.parent.mkdir(parents=True)

905 + raw_path.write_text(

906 + json.dumps({"week": "2026-W30", "new_repos": [], "trending_repos": []}),

907 + encoding="utf-8",

908 + )

909 + prompt_template.write_text(

910 + "{{RAW_JSON_CONTENT}}\n{{HISTORICAL_CONTEXT}}", encoding="utf-8"

911 + )

912 + synthesis_path.write_text(

913 + "Industry narrative distilled from press and history.", encoding="utf-8"

914 + )

915 +

916 + with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout:

917 + exit_code = analyze_fallback.main(

918 + [

919 + "--raw-json",

920 + str(raw_path),

921 + "--output",

922 + str(output_path),

923 + "--current-datetime",

924 + "2026-07-27T12:00:00Z",

925 + "--prompt-template",

926 + str(prompt_template),

927 + "--analyzed-dir",

928 + str(output_path.parent),

929 + "--wisdom-file",

930 + str(base / "w.md"),

931 + "--skills-dir",

932 + str(base / "s"),

933 + # No --press-context: genuinely press-less week.

934 + "--synthesis-input",

935 + str(synthesis_path),

936 + "--preflight-report-json",

937 + str(report_path),

938 + "--print-prompt",

939 + ]

940 + )

941 +

942 + self.assertEqual(exit_code, 0)

943 + rendered = stdout.getvalue()

944 + self.assertIn("Industry narrative distilled", rendered)

945 + self.assertIn("[Industry narrative synthesized from historical context]", rendered)

946 + self.assertNotIn("press & historical context", rendered)

947 + self.assertNotIn("## Press Context", rendered)

948 +

949 + report = json.loads(report_path.read_text(encoding="utf-8"))

950 + components = {component["name"]: component for component in report["components"]}

951 + press_component = components["press_correlations"]

952 + self.assertFalse(press_component["included"])

953 + self.assertEqual(

954 + press_component["compaction_decision"], "not included: no press context"

955 + )

956 + self.assertEqual(press_component["token_estimate"], 0)

957 + self.assertEqual(press_component["bytes"], 0)

958 +

959 + def test_synthesis_narrative_with_sentinel_press_context_suppresses_block(self) -> None:

960 + """Regression (press-less week): when the press-context FILE contains the render

961 + NO_PRESS_SENTINEL, the non-empty sentinel must be treated as *no* press — the

962 + '## Press Context' block is suppressed, the model is not told 'No industry press

963 + data was available', and the press component is recorded as not included. A real

964 + press file (contrast, covered by

965 + test_synthesis_narrative_does_not_drop_press_context) still yields the block."""

966 + tests_root = Path(__file__).resolve().parent

967 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

968 + base = Path(tmpdir)

969 + raw_path = base / "data" / "raw" / "2026-W30.json"

970 + prompt_template = base / "prompt.md"

971 + output_path = base / "data" / "analyzed" / "2026-W30-summary.md"

972 + press_path = base / "data" / "analyzed" / "2026-W30-press-context.md"

973 + synthesis_path = base / "synthesis.md"

974 + report_path = base / "diagnostics" / "preflight.json"

975 + raw_path.parent.mkdir(parents=True)

976 + output_path.parent.mkdir(parents=True)

977 + raw_path.write_text(

978 + json.dumps({"week": "2026-W30", "new_repos": [], "trending_repos": []}),

979 + encoding="utf-8",

980 + )

981 + prompt_template.write_text(

982 + "{{RAW_JSON_CONTENT}}\n{{HISTORICAL_CONTEXT}}", encoding="utf-8"

983 + )

984 + # The press file is present but its content IS the no-press sentinel.

985 + press_path.write_text(NO_PRESS_SENTINEL, encoding="utf-8")

986 + synthesis_path.write_text(

987 + "Industry narrative distilled from press and history.", encoding="utf-8"

988 + )

989 +

990 + with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout:

991 + exit_code = analyze_fallback.main(

992 + [

993 + "--raw-json",

994 + str(raw_path),

995 + "--output",

996 + str(output_path),

997 + "--current-datetime",

998 + "2026-07-27T12:00:00Z",

999 + "--prompt-template",

1000 + str(prompt_template),

1001 + "--analyzed-dir",

1002 + str(output_path.parent),

1003 + "--wisdom-file",

1004 + str(base / "w.md"),

1005 + "--skills-dir",

1006 + str(base / "s"),

1007 + "--press-context",

1008 + str(press_path),

1009 + "--synthesis-input",

1010 + str(synthesis_path),

1011 + "--preflight-report-json",

1012 + str(report_path),

1013 + "--print-prompt",

1014 + ]

1015 + )

1016 +

1017 + self.assertEqual(exit_code, 0)

1018 + rendered = stdout.getvalue()

1019 + self.assertIn("Industry narrative distilled", rendered)

1020 + self.assertIn("[Industry narrative synthesized from historical context]", rendered)

1021 + self.assertNotIn("press & historical context", rendered)

1022 + # Sentinel content must NOT be emitted as a real press block.

1023 + self.assertNotIn("## Press Context", rendered)

1024 + self.assertNotIn("No industry press data was available", rendered)

1025 +

1026 + report = json.loads(report_path.read_text(encoding="utf-8"))

1027 + components = {component["name"]: component for component in report["components"]}

1028 + press_component = components["press_correlations"]

1029 + self.assertFalse(press_component["included"])

1030 + self.assertEqual(

1031 + press_component["compaction_decision"], "not included: no press context"

1032 + )

1033 +

1034 def test_run_synthesis_exits_zero_and_writes_prompt(self) -> None:

1035 """--run-synthesis should render synthesis prompt to output file."""

1036 tests_root = Path(__file__).resolve().parent

tests/test_render_press_context.py

+13

@@ -10,12 +10,14 @@ sys.path.insert(0, str(_REPO_ROOT / "scripts"))

11 import render_press_context as render_press_context_module # noqa: E402

12 from render_press_context import ( # noqa: E402

13 + NO_PRESS_SENTINEL,

14 _escape_markdown_url,

15 _extract_readme_description,

16 _format_correlations_narrative,

17 format_articles_list,

18 format_correlations_list,

19 format_divergences,

20 + press_token_estimate,

21 render_press_context,

22 resolve_paths,

23 )

@@ -156,6 +158,17 @@ class TestFormatCorrelationsList:

158

159

160 class TestRenderPressContext:

161 + def test_press_token_estimate_treats_empty_content_as_zero(self):

162 + assert press_token_estimate("") == 0

163 + assert press_token_estimate(" \n\t ") == 0

164 + assert press_token_estimate(NO_PRESS_SENTINEL) == 0

165 + assert press_token_estimate(f" \n## Press Context\n\n{NO_PRESS_SENTINEL}\n ") == 0

166 +

167 + estimate = press_token_estimate("## Press Context\n\nA real article about AI agents.")

168 +

169 + assert isinstance(estimate, int)

170 + assert estimate > 0

171 +

172 def test_no_data_returns_fallback(self):

173 result = render_press_context(None, None, "2026-W21")

174 assert "No press data available" in result

tests/test_techcrunch_crawler.py

+169 -1

@@ -4,10 +4,12 @@ from __future__ import annotations

5 import json

6 import tempfile

7 -from datetime import UTC, datetime

7 +from datetime import UTC, datetime, timedelta, timezone

8 +from email.utils import format_datetime

9 from pathlib import Path

10 from types import SimpleNamespace

11 from unittest.mock import patch

12 +from urllib.error import HTTPError, URLError

14 import pytest

@@ -61,6 +63,10 @@ def _make_feed(entries=None, bozo=False):

63 return SimpleNamespace(entries=entries or [], bozo=bozo)

66 +def _http_error_with_headers(headers):

67 + return HTTPError("https://techcrunch.com/feed/", 503, "Unavailable", headers, None)

68 +

69 +

70 # --- Unit tests: utility functions ---

@@ -166,6 +172,67 @@ class TestComputeRelevanceScore:

172 assert score_with > score_no

173

174

175 +class TestRetryAfterSeconds:

176 + def test_retry_after_numeric_header_preserved_and_floored(self):

177 + assert (

178 + techcrunch_crawler._retry_after_seconds(

179 + _http_error_with_headers({"Retry-After": "120"})

180 + )

181 + == 120.0

182 + )

183 + assert (

184 + techcrunch_crawler._retry_after_seconds(_http_error_with_headers({"Retry-After": "0"}))

185 + == 1.0

186 + )

187 + assert (

188 + techcrunch_crawler._retry_after_seconds(

189 + _http_error_with_headers({"Retry-After": "0.5"})

190 + )

191 + == 1.0

192 + )

193 +

194 + def test_retry_after_http_date_future_returns_positive_delay(self):

195 + retry_at = datetime.now(timezone.utc) + timedelta(seconds=120)

196 + result = techcrunch_crawler._retry_after_seconds(

197 + _http_error_with_headers({"Retry-After": format_datetime(retry_at)})

198 + )

199 +

200 + assert result is not None

201 + assert 60 <= result <= 200

202 +

203 + def test_retry_after_http_date_past_is_floored(self):

204 + retry_at = datetime.now(timezone.utc) - timedelta(seconds=120)

205 +

206 + assert (

207 + techcrunch_crawler._retry_after_seconds(

208 + _http_error_with_headers({"Retry-After": format_datetime(retry_at)})

209 + )

210 + == 1.0

211 + )

212 +

213 + def test_retry_after_garbage_empty_or_missing_returns_none(self):

214 + assert (

215 + techcrunch_crawler._retry_after_seconds(

216 + _http_error_with_headers({"Retry-After": "not-a-date"})

217 + )

218 + is None

219 + )

220 + assert (

221 + techcrunch_crawler._retry_after_seconds(_http_error_with_headers({"Retry-After": ""}))

222 + is None

223 + )

224 + assert techcrunch_crawler._retry_after_seconds(_http_error_with_headers({})) is None

225 +

226 + def test_retry_after_non_finite_numeric_returns_none(self):

227 + for header_value in ("nan", "inf", "-inf", "Infinity"):

228 + assert (

229 + techcrunch_crawler._retry_after_seconds(

230 + _http_error_with_headers({"Retry-After": header_value})

231 + )

232 + is None

233 + )

234 +

235 +

236 class TestParsePublishedDate:

237 def test_published_parsed(self):

238 entry = SimpleNamespace(published_parsed=(2026, 5, 15, 10, 0, 0, 3, 135, 0))

@@ -499,6 +566,107 @@ class TestExternalNewsSources:

566 assert result is feed

567 assert mock_urlopen.call_args.kwargs["timeout"] == DEFAULT_FETCH_TIMEOUT_SECONDS

568

569 + def test_fetch_feed_retries_transient_error_with_backoff(self):

570 + """A transient network error is retried with backoff and can then succeed."""

571 + feed = _make_feed()

572 +

573 + class FakeResponse:

574 + def __enter__(self):

575 + return self

576 +

577 + def __exit__(self, exc_type, exc, traceback):

578 + return None

579 +

580 + def read(self):

581 + return b"<rss><channel></channel></rss>"

582 +

583 + attempts = {"n": 0}

584 +

585 + def flaky_urlopen(request, timeout=None):

586 + attempts["n"] += 1

587 + if attempts["n"] == 1:

588 + raise URLError("temporary DNS failure")

589 + return FakeResponse()

590 +

591 + with (

592 + patch("scripts.techcrunch_crawler.urlopen", side_effect=flaky_urlopen),

593 + patch("scripts.techcrunch_crawler.feedparser.parse", return_value=feed),

594 + patch("scripts.techcrunch_crawler._sleep_before_retry", return_value=0.0) as sleeper,

595 + ):

596 + result = fetch_feed("https://techcrunch.com/feed/", retries=2)

597 +

598 + assert result is feed

599 + assert attempts["n"] == 2

600 + assert sleeper.called

601 +

602 + def test_fetch_feed_fails_fast_on_non_retryable_status(self):

603 + """Permanent HTTP errors (e.g. 404) must not be retried."""

604 + attempts = {"n": 0}

605 +

606 + def not_found(request, timeout=None):

607 + attempts["n"] += 1

608 + raise HTTPError("https://techcrunch.com/feed/", 404, "Not Found", {}, None)

609 +

610 + with (

611 + patch("scripts.techcrunch_crawler.urlopen", side_effect=not_found),

612 + patch("scripts.techcrunch_crawler._sleep_before_retry") as sleeper,

613 + ):

614 + with pytest.raises(HTTPError):

615 + fetch_feed("https://techcrunch.com/feed/", retries=3)

616 +

617 + assert attempts["n"] == 1

618 + assert not sleeper.called

619 +

620 + def test_fetch_feed_retries_retryable_status(self):

621 + """Retryable HTTP statuses (e.g. 503) are retried with backoff."""

622 + attempts = {"n": 0}

623 +

624 + def unavailable(request, timeout=None):

625 + attempts["n"] += 1

626 + raise HTTPError("https://techcrunch.com/feed/", 503, "Unavailable", {}, None)

627 +

628 + with (

629 + patch("scripts.techcrunch_crawler.urlopen", side_effect=unavailable),

630 + patch("scripts.techcrunch_crawler._sleep_before_retry", return_value=0.0) as sleeper,

631 + ):

632 + with pytest.raises(HTTPError):

633 + fetch_feed("https://techcrunch.com/feed/", retries=2)

634 +

635 + assert attempts["n"] == 3

636 + assert sleeper.call_count == 2

637 +

638 + def test_sleep_before_retry_honors_retry_after_120(self):

639 + """Server Retry-After of 120s is honored instead of backoff-capped."""

640 + with patch("scripts.techcrunch_crawler.time.sleep") as sleep_mock:

641 + delay = techcrunch_crawler._sleep_before_retry(0, retry_after=120)

642 +

643 + assert delay == techcrunch_crawler.RETRY_AFTER_MAX_SECONDS

644 + sleep_mock.assert_called_once_with(delay)

645 +

646 + def test_sleep_before_retry_bounds_absurd_retry_after(self):

647 + """Absurd Retry-After values are capped to the Retry-After maximum."""

648 + with patch("scripts.techcrunch_crawler.time.sleep") as sleep_mock:

649 + delay = techcrunch_crawler._sleep_before_retry(0, retry_after=99999)

650 +

651 + assert delay == techcrunch_crawler.RETRY_AFTER_MAX_SECONDS

652 + sleep_mock.assert_called_once_with(delay)

653 +

654 + def test_sleep_before_retry_honors_small_retry_after(self):

655 + """Small positive Retry-After values are honored exactly."""

656 + with patch("scripts.techcrunch_crawler.time.sleep") as sleep_mock:

657 + delay = techcrunch_crawler._sleep_before_retry(0, retry_after=5)

658 +

659 + assert delay == 5.0

660 + sleep_mock.assert_called_once_with(delay)

661 +

662 + def test_sleep_before_retry_without_retry_after_uses_capped_backoff(self):

663 + """Computed backoff remains bounded by the backoff maximum."""

664 + with patch("scripts.techcrunch_crawler.time.sleep") as sleep_mock:

665 + delay = techcrunch_crawler._sleep_before_retry(10, retry_after=None)

666 +

667 + assert 0 < delay <= techcrunch_crawler.RETRY_MAX_DELAY_SECONDS

668 + sleep_mock.assert_called_once_with(delay)

669 +

670 def test_crawl_sources_parallel_combines_sources(self):

671 alpha_entry = _make_entry(

672 title="Alpha AI framework",