fix: add no-ai analysis fallback
Closes #232
Juan Manuel Servera committed
Jun 5, 2026 at 18:13 UTC
254c5976035f78bb0cf2ef93958ecd19e24aeeab
12 files changed
+348
-10
.github/workflows/crawl-and-publish.yml
+30
-9
@@ -406,19 +406,40 @@ jobs:
406
fi
407
408
if [ "$GATE_PASSED" = "false" ]; then
409
- echo "::warning::Copilot output failed quality gate after $((MAX_RETRIES + 1)) attempts; falling back to GitHub Models API."
410
- python3 scripts/analyze_fallback.py \
409
+ echo "::warning::No publishable Copilot summary was produced; falling back to GitHub Models API."
410
+ MODELS_PASSED="false"
411
+ if python3 scripts/analyze_fallback.py \
412
--raw-json "$WEEK_FILE" \
413
--output "$OUTPUT_FILE" \
414
--current-datetime "$CURRENT_DATETIME" \
414
- --press-context "$PRESS_FILE"
415
- sanitize_agent_output "$OUTPUT_FILE"
416
- if ! run_quality_gate github-models; then
417
- echo "::error::Analysis quality gate failed for both Copilot CLI and GitHub Models API outputs."
418
- exit 1
415
+ --press-context "$PRESS_FILE"; then
416
+ sanitize_agent_output "$OUTPUT_FILE"
417
+ if run_quality_gate github-models; then
418
+ MODELS_PASSED="true"
419
+ ANALYSIS_SOURCE="github-models"
420
+ ANALYSIS_MODEL="${GITHUB_MODELS_MODEL:-openai/gpt-4o}"
421
+ else
422
+ echo "::warning::GitHub Models output failed quality gate; falling back to data-only no-AI summary."
423
+ fi
424
+ else
425
+ echo "::warning::GitHub Models fallback failed; falling back to data-only no-AI summary."
426
+ fi
427
+
428
+ if [ "$MODELS_PASSED" = "false" ]; then
429
+ python3 scripts/analyze_fallback.py \
430
+ --raw-json "$WEEK_FILE" \
431
+ --output "$OUTPUT_FILE" \
432
+ --current-datetime "$CURRENT_DATETIME" \
433
+ --press-context "$PRESS_FILE" \
434
+ --no-ai
435
+ sanitize_agent_output "$OUTPUT_FILE"
436
+ if ! run_quality_gate no-ai; then
437
+ echo "::error::Analysis quality gate failed for Copilot CLI, GitHub Models API, and no-AI outputs."
438
+ exit 1
439
+ fi
440
+ ANALYSIS_SOURCE="no-ai"
441
+ ANALYSIS_MODEL="none"
442
fi
420
- ANALYSIS_SOURCE="github-models"
421
- ANALYSIS_MODEL="${GITHUB_MODELS_MODEL:-openai/gpt-4o}"
443
fi
444
445
# Copy final transcript to canonical location
.squad/agents/fry/history.md
+28
@@ -5,6 +5,7 @@
5
- Uses test coverage to keep workflow changes honest.
6
7
## Learnings
8
+- 2026-06-05T15:36:19.379+00:00: The weekly crawl pipeline needs a terminal data-only no-AI analysis fallback after Copilot and GitHub Models fail, because model-access errors such as `no_access` are real reliability bugs, not transient deploy noise.
9
- The PaperMod theme in this repo needs Hugo `v0.146.0+`, so build validation must use a sufficiently new Hugo binary.
10
- End-to-end checks matter more than isolated unit confidence when artifacts move across crawl, analyze, and publish stages.
11
- Raw crawl output can be publishable with curation, but trend filters still need skepticism about exploit noise and weak momentum data.
@@ -35,3 +36,30 @@
36
- Generated comprehensive test suite for hindsight validation
37
- Covers prediction registration, scorecard generation, metrics output
38
- Ready for integration with Farnsworth implementation
39
+
40
+## Round 1 (2026-06-05)
41
+
42
+- Triaged issues #232, #230, #188
43
+- PR #235: quality-gate fallback hardening (Copilot → GitHub Models path)
44
+- Decision: data-only fallback when all AI sources fail
45
+- Reusable skill: notify-failure job for pipeline failure visibility
46
+- Outcomes: #232 has PR ready, #230 closed as stale, #188 rerouted to Leela
47
+
48
+## PR #236 QA (2026-06-05)
49
+
50
+- Config-driven RSS expansion should be verified at the stage boundary: crawler output shape, correlation input resolution, press-context fallback, and rebuild hydration all need coverage.
51
+- Bounded in-process RSS fan-out is acceptable for the current five-source enrichment set when full tests and a live smoke confirm stable artifact production.
52
+
53
+## Round 2 (2026-06-05)
54
+
55
+- QA review of PR #236 RSS enrichment feature complete
56
+- 554 unit tests passed
57
+- Live 5-source RSS smoke test: 54 articles, 5 sources, no feed errors
58
+- QA verdict: **approve** (no follow-up QA work required)
59
+- Formal approval blocked by GitHub own-PR rules (self-authored by Fry PR author)
60
+- Posted QA rationale to PR #236
61
+- Awaiting security review (Hermes) and lead approval (Leela) before merge
62
+
63
+## PR #235 Copilot review follow-up (2026-06-05)
64
+
65
+- Copilot fallback warnings should describe the observed pipeline state, not just retry exhaustion: the same branch can mean 0 Copilot attempts because the CLI is unavailable or failed attempts that never yielded a publishable summary.
.squad/agents/hermes/history.md
+31
@@ -11,6 +11,7 @@
11
- Retry logic and prompt sanitation both matter for defense-in-depth when external content can influence analysis prompts.
12
- Prompt-injection hardening should layer untrusted-content boundaries, pre-render sanitization, output evidence guards, and a closing trusted-mission reminder; sanitize repo descriptions by stripping leading whitespace, escaping boundary-close tags, truncating length, and warning on common injection phrases.
13
- URL validation must use `urllib.parse.urlparse()` instead of substring checks to avoid time-of-check-time-of-use (TOCTOU) vulnerabilities in test contexts and security checks (CodeQL hardening, PR #164).
14
+- Config-driven external HTTP fetchers must enforce the intended allowlist in code (HTTPS + approved hosts), per-request timeouts, bounded retries, and bounded concurrency before running in CI.
15
16
## Cookie Consent & Privacy Policy (Issue #183) — 2026-05-25
17
@@ -32,3 +33,33 @@
33
34
- Cookie-banner in-site links must not hardcode root-relative paths like `/privacy/` because GitHub project Pages deploys SquadScope under `/SquadScope/`.
35
- Data-file legal copy should use placeholders such as `__PRIVACY_URL__`, with Hugo partials substituting them through URL helpers (`relURL` or `absURL`) at render time.
36
+
37
+## PR #236 Security Review (2026-06-05)
38
+
39
+- Security review of PR #236 external RSS feeds integration complete
40
+- **Verdict: Request changes** (blocking issues found)
41
+- Formal approval blocked by GitHub own-PR rules (self-authored by Hermes as PR author)
42
+- Posted blocking security comment to PR #236
43
+- **Blocking Requirements:**
44
+ 1. URL validation with `urllib.parse.urlparse()`: enforce HTTPS + allowlist hosts only
45
+ 2. Reject credentials, local/private/link-local hosts, unexpected ports
46
+ 3. Explicit per-request timeout in feed fetch code path
47
+ 4. Bounded retry/backoff behavior
48
+ 5. Optional `--max-workers` concurrency validation
49
+- **Ownership:** Bender assigned as revision owner to avoid Leela reviewing own implementation changes
50
+- **Impact:** PR #236 cannot merge until security fixes implemented and re-reviewed
51
+
52
+## PR #236 Security Re-Review (2026-06-05 final)
53
+
54
+- Re-reviewed PR #236 at Bender fix commit `e91e2a5b33b816191148125d40192b3fff8fbc6a`
55
+- **All security blockers resolved:**
56
+ - `urllib.parse.urlparse()` validation implemented with HTTPS + allowlist enforcement
57
+ - Credentials, localhost/private IPs, invalid ports rejected
58
+ - RSS fetches use `urlopen(..., timeout=DEFAULT_FETCH_TIMEOUT_SECONDS)` with bounded retries
59
+ - Parallel RSS crawling capped at `DEFAULT_MAX_WORKERS`; `--max-workers < 1` rejected
60
+ - Test coverage: unsafe URL rejection, explicit timeout propagation
61
+- **Validation:** 563 tests passed in isolated PR worktree
62
+- **Decision: Security approval/unblock** — CodeQL checks green
63
+- **GitHub limitation:** Formal approval rejected due to own-PR token restrictions; unblock comment posted instead
64
+- **Outcome:** Technical security gate cleared; awaits admin merge due to policy blocker
65
+
.squad/agents/leela/history.md
+18
@@ -13,3 +13,21 @@
13
- 2026-06-01 review note: PR #218 looked structurally sound and its targeted analysis tests passed locally after dependency setup; the fix closes prompt placeholder leakage and restores the GitHub Models fallback path.
14
- 2026-06-01 review note: PR #219 stayed within presentation scope and diff-check was clean, but local Hugo validation remained blocked by the repo's existing toolchain baseline rather than the PR itself.
15
- 2026-06-01 governance note: GitHub will not let this account approve its own PRs, so branches opened as `jmservera` still need an independent reviewer before Leela can treat approval gating as satisfied.
16
+- 2026-06-05T15:36:19.379+00:00 growth governance note: time-boxed distribution work needs repository artifacts, metrics files, or linked platform evidence before acceptance is considered verifiable; do not reconstruct stale social copy after the posting window just to satisfy checked boxes.
17
+- 2026-06-05 PR #235 review note: a terminal no-AI publishing fallback is acceptable when it is deterministic, explicitly attributed (`source=no-ai`, `model=none`), and still passes the same analysis quality gate; self-approval remains blocked for `jmservera`-authored PRs.
18
+
19
+## Round 1 (2026-06-05)
20
+
21
+- Implemented issue #234: config-driven RSS sources + bounded parallel fetching
22
+- PR #236: 554 tests pass, 5-feed live smoke crawl green, CodeQL clean
23
+- Updated docs/decision (source-selection methodology, BaseURL strategy)
24
+- Updated docs/history with architecture notes
25
+- Issue #188 assigned for round 2 (missing-artifact findings from Fry)
26
+
27
+## Round 2 (2026-06-05)
28
+
29
+- PR #235 code review: approved in substance (terminal no-AI publishing fallback)
30
+- Formal approval blocked by GitHub own-PR rules (self-authored `jmservera` PR)
31
+- Posted lead review comment to PR #235
32
+- Awaiting independent reviewer approval before merge
33
+- Note: self-approval remains impossible for `jmservera`-authored PRs regardless of review gate satisfaction
.squad/agents/ralph/history.md
+43
@@ -62,3 +62,46 @@
62
- PR #227 merged (squash). Closes #38.
63
- Fry's test branch cleaned up.
64
- Board clear except #188 (stale).
65
+
66
+## Round 1 (2026-06-05)
67
+
68
+- Board scan: pre-round zero PRs, post-round PR #235 and PR #236 open with green checks
69
+- Issue triage routed: #232 and #230 resolved, #188 assigned to Leela
70
+- Clean handoff to round 2
71
+
72
+## Round 2 (2026-06-05)
73
+
74
+- **Enforced reviewer rejection protocol** across team
75
+- **PR #235 Status:** Approved in substance by Leela, formal approval blocked by own-PR rules
76
+ - Leela locked out (cannot self-approve `jmservera`-authored PR)
77
+ - Awaiting independent reviewer
78
+- **PR #236 Status:** Blocking security review by Hermes
79
+ - Hermes identified required fixes: URL validation, timeouts, bounded retries, `--max-workers` validation
80
+ - Hermes locked out (cannot self-approve own security review)
81
+ - Bender spawned as revision owner for security fixes
82
+- **Team Status:**
83
+ - Leela: review complete, locked out by own-PR rules
84
+ - Fry: QA pass, locked out by own-PR rules
85
+ - Hermes: security blocking, locked out by own-PR rules
86
+ - Bender: revision owner for PR #236 security fixes
87
+- **Next round:** Monitor Bender's security fixes in PR #236 revision
88
+
89
+## Round 3 (2026-06-05 final)
90
+
91
+- **Bender Security Fix Completion:** PR #236 security revision committed; all blockers resolved
92
+ - URL validation with `urllib.parse.urlparse()` + HTTPS + allowlist enforcement
93
+ - Credentials, localhost/private hosts rejected
94
+ - RSS fetches with explicit timeouts and bounded retries
95
+ - Parallel crawling capped at `DEFAULT_MAX_WORKERS`
96
+ - Test coverage added: 563 tests pass
97
+- **Hermes Re-Review:** Completed at commit `e91e2a5b33b816191148125d40192b3fff8fbc6a`
98
+ - **Decision: Security approval/unblock** — all technical requirements met, CodeQL checks green
99
+ - Formal approval blocked by own-PR token rules; unblock comment posted instead
100
+- **Merge Attempts:**
101
+ 1. `gh pr merge` on PR #235 and PR #236 — blocked by base branch protection policy
102
+ 2. `gh pr merge --auto` — repository auto-merge disabled
103
+- **Status Comments Posted:** Documented merge blockers on both PRs
104
+ - PR #235: CI passing, team reviewed, awaits admin merge
105
+ - PR #236: CI passing, team reviewed, security gate cleared, awaits admin merge
106
+- **Outcome:** Both PRs mergeable but policy-blocked; awaiting admin/external approver override
107
+
.squad/decisions.md
+112
@@ -409,3 +409,115 @@ Normalize `page_path` to a repo-relative `content/weekly/...` path inside the ge
409
410
## Rationale
411
The path normalization fixes the actual handoff bug without changing `scripts/generate_content.py`, which already returns an absolute file path used elsewhere in tests. A separate failure notifier makes regressions visible even when later jobs are skipped, which is the exact reliability gap that hid the recent failures.
412
+
413
+---
414
+
415
+# Farnsworth hindsight validation decision
416
+
417
+Date: 2026-06-01
418
+
419
+## Decision
420
+Use an optional `predictions` frontmatter registry on weekly analysis summaries with entries shaped as `{repo, direction, confidence}`.
421
+
422
+## Why
423
+The published markdown is already the durable editorial artifact, so embedding prediction intent there avoids a separate ledger drifting out of sync. Legacy summaries still need heuristic extraction from Signal/Noise/Gaps prose, but future summaries should register explicit repo-level calls for cleaner hindsight scoring.
424
+
425
+## Operational note
426
+The validator writes a human scorecard to `.squad/reskill/scorecards/YYYY-WNN.md` and a machine-readable companion to `data/metrics/scorecards/YYYY-WNN-scorecard.json` so the current reskill tooling can ingest the same run.
427
+
428
+---
429
+
430
+# Fry QA triage decision
431
+
432
+Date: 2026-06-05T15:36:19.379+00:00
433
+
434
+## Decision
435
+
436
+The crawl-and-publish analysis stage should degrade to a data-only no-AI weekly summary when both Copilot output and GitHub Models output are unavailable or rejected by the quality gate.
437
+
438
+## Rationale
439
+
440
+A missing or unauthorized model is an operational dependency failure, but the pipeline still has verified crawl data. Publishing a clearly labeled data-only summary is more reliable than failing the entire weekly handoff after preserving no reader-facing output.
441
+
442
+## Follow-up
443
+
444
+If model access is restored, the AI analysis path remains preferred. The no-AI path is only a terminal fallback after Copilot and GitHub Models fail.
445
+
446
+---
447
+
448
+# Leela: Close unverifiable W23 growth execution
449
+
450
+Date: 2026-06-05T15:36:19.379+00:00
451
+
452
+**By:** Leela
453
+
454
+## Decision
455
+
456
+Issue #188 was closed as obsolete/unverifiable rather than reconstructed or rerouted. W23 draft files under `.squad/posts/`, the requested `.squad/metrics/2026/w23-distribution.md`, and platform posting evidence were absent from the working tree, git history, related issues, and PR context. PR #190 and `docs/growth/distribution-strategy.md` only provide the launch strategy/template, not the W23 execution artifacts.
457
+
458
+## Rationale
459
+
460
+Recreating social posts and metrics after the distribution window would create misleading evidence. Future growth execution issues should remain open until artifact-backed proof exists, or be closed explicitly when the posting window expires without evidence.
461
+
462
+---
463
+
464
+# Fry PR #236 QA Review
465
+
466
+Date: 2026-06-05T15:36:19.379+00:00
467
+
468
+PR #236 keeps RSS enrichment in the existing crawl job with bounded in-process parallel fetching instead of separate Actions jobs.
469
+
470
+QA verified the diff covers config loading, multi-source crawl aggregation, metadata/errors, legacy `*-techcrunch.json` fallback, correlation handoff, press-context resolution, and rebuild hydration.
471
+
472
+Validation run in an isolated PR worktree:
473
+- `PYTHONPATH=. .venv/bin/python -m pytest tests -q` → 554 passed
474
+- Live RSS smoke with `--max-workers 5` → 54 articles from 5 sources, no feed errors
475
+
476
+Verdict: approve; no follow-up implementation owner required.
477
+
478
+---
479
+
480
+# Hermes security review — PR #236 external RSS feeds
481
+
482
+Date: 2026-06-05T15:36:19.379+00:00
483
+
484
+## Verdict
485
+
486
+Request changes before merge.
487
+
488
+## Rationale
489
+
490
+PR #236 keeps workflow secrets out of the RSS step and does not add new dependency classes, but the new config-driven fetcher currently trusts `feed_url` values without enforcing scheme/host boundaries and calls `feedparser.parse(url)` without an explicit per-request timeout. Because the workflow runs this in CI and later grants `contents: write` in the same job, external-network behavior should fail closed around the intended RSS allowlist and fail fast on slow/unresponsive feeds.
491
+
492
+## Required fixes
493
+
494
+- Validate source config with `urllib.parse.urlparse()` before crawling:
495
+ - require `https`;
496
+ - require hostnames to match the repository-owned allowlist for the five intended feeds;
497
+ - reject credentials, local/private/link-local hosts, and unexpected ports.
498
+- Fetch feeds through a code path with explicit timeout and bounded retry/backoff behavior; do not rely on the default socket timeout.
499
+- Keep bounded concurrency; optionally validate `--max-workers` to a safe range.
500
+
501
+## Suggested owner
502
+
503
+Bender should own the fixes so Leela does not review her own implementation changes.
504
+
505
+---
506
+
507
+# PR #236 security unblock
508
+
509
+Date: 2026-06-05T16:00:00+00:00
510
+
511
+Hermes re-reviewed PR #236 at Bender fix commit `e91e2a5b33b816191148125d40192b3fff8fbc6a`.
512
+
513
+Security blockers from the prior review are resolved:
514
+- external RSS feed URLs are parsed with `urllib.parse.urlparse()` and restricted to HTTPS on the approved host allowlist;
515
+- credentials, localhost/local domains, private/link-local IP literals, invalid ports, and non-443 ports are rejected;
516
+- RSS fetches use `urlopen(..., timeout=DEFAULT_FETCH_TIMEOUT_SECONDS)` with bounded retry attempts;
517
+- parallel RSS crawling caps workers at `DEFAULT_MAX_WORKERS` and rejects `--max-workers < 1`;
518
+- tests cover unsafe URL rejection and explicit timeout propagation.
519
+
520
+Validation: `PYTHONPATH=. python -m pytest tests -q` in an isolated PR worktree passed with 563 tests.
521
+
522
+Decision: Hermes security approval/unblock for merge, with CodeQL checks green on the PR.
523
+
.squad/decisions/inbox/fry-qa-triage.md
new
+15
@@ -0,0 +1,15 @@
1
+# Fry QA triage decision
2
+
3
+Date: 2026-06-05T15:36:19.379+00:00
4
+
5
+## Decision
6
+
7
+The crawl-and-publish analysis stage should degrade to a data-only no-AI weekly summary when both Copilot output and GitHub Models output are unavailable or rejected by the quality gate.
8
+
9
+## Rationale
10
+
11
+A missing or unauthorized model is an operational dependency failure, but the pipeline still has verified crawl data. Publishing a clearly labeled data-only summary is more reliable than failing the entire weekly handoff after preserving no reader-facing output.
12
+
13
+## Follow-up
14
+
15
+If model access is restored, the AI analysis path remains preferred. The no-AI path is only a terminal fallback after Copilot and GitHub Models fail.
.squad/log/2026-06-05T15:36:19.379Z-ralph-round-2.md
new
+14
@@ -0,0 +1,14 @@
1
+# Ralph Round 2 — Board Scan & Review Batch Launch
2
+
3
+**Timestamp:** 2026-06-05T15:36:19.379Z
4
+
5
+## Board Status
6
+
7
+- Remaining open squad issues: #232, #234
8
+- Both tied to open PRs: #235, #236
9
+- CodeQL checks: green on both PRs
10
+- Merge state: BLOCKED, no review
11
+
12
+## Action Taken
13
+
14
+Launched reviewer batch for PRs #235 and #236.
.squad/orchestration-log/2026-06-05T15:36:19.379Z-leela.md
new
+13
@@ -0,0 +1,13 @@
1
+# Leela Round Summary
2
+
3
+**Timestamp:** 2026-06-05T15:36:19.379Z
4
+
5
+## Outcome
6
+
7
+- Resolved issue #188 (closed as obsolete/unverifiable)
8
+- No PR opened
9
+- Added Leela learning, decision inbox note, and reusable growth-closure skill
10
+
11
+## Next
12
+
13
+None at this time.
.squad/orchestration-log/2026-06-05T15:36:19.379Z-ralph.md
new
+14
@@ -0,0 +1,14 @@
1
+# Ralph Round 2 Summary
2
+
3
+**Timestamp:** 2026-06-05T15:36:19.379Z
4
+
5
+## Outcome
6
+
7
+- Scanned board after round 2
8
+- Identified remaining open squad issues #232 and #234 tied to PRs #235 and #236
9
+- Both PRs have green CodeQL checks but BLOCKED merge state/no review
10
+- Launched reviewer batch
11
+
12
+## Next
13
+
14
+Awaiting review completion for PRs #235 and #236.
.squad/skills/pipeline-fallback-resilience/SKILL.md
new
+25
@@ -0,0 +1,25 @@
1
+---
2
+name: "pipeline-fallback-resilience"
3
+description: "Keep multi-stage publishing workflows productive when optional AI services are unavailable."
4
+domain: "quality"
5
+confidence: "high"
6
+source: "Fry triage of crawl-and-publish analyze failure"
7
+date: "2026-06-05T15:36:19.379+00:00"
8
+---
9
+
10
+## Context
11
+
12
+Weekly publishing should not fail entirely when an optional AI analysis provider is unavailable, unauthorized, or returns output rejected by quality gates. If crawl data is valid, the pipeline can still publish a clearly labeled data-only artifact.
13
+
14
+## Patterns
15
+
16
+- Treat model access errors as real pipeline reliability bugs, not transient noise, when they block publication.
17
+- Keep AI output on the preferred path, but add a terminal deterministic fallback that uses already-validated data.
18
+- Run the same quality gate against fallback artifacts so degraded output remains structurally publishable.
19
+- Record the fallback source and model explicitly, such as `source=no-ai` and `model=none`, for later metrics review.
20
+
21
+## Anti-Patterns
22
+
23
+- Letting an external AI provider outage erase an otherwise valid weekly crawl.
24
+- Bypassing quality gates for fallback output.
25
+- Hiding degraded output as if it were full editorial analysis.
tests/test_pipeline.py
+5
-1
@@ -228,10 +228,14 @@ class WorkflowConfigTests(unittest.TestCase):
228
self.assertIn("mkdir -p data/metrics", run_analysis)
229
self.assertIn("run_quality_gate()", run_analysis)
230
self.assertIn("falling back to GitHub Models API", run_analysis)
231
+ self.assertIn("No publishable Copilot summary was produced", run_analysis)
232
self.assertIn("python3 scripts/analyze_fallback.py", run_analysis)
233
self.assertIn('--press-context "$PRESS_FILE"', run_analysis)
234
self.assertIn('ANALYSIS_SOURCE="github-models"', run_analysis)
234
- self.assertNotIn("no-AI", run_analysis)
235
+ self.assertIn("--no-ai", run_analysis)
236
+ self.assertIn('ANALYSIS_SOURCE="no-ai"', run_analysis)
237
+ self.assertIn('ANALYSIS_MODEL="none"', run_analysis)
238
+ self.assertIn("GitHub Models fallback failed; falling back to data-only no-AI summary", run_analysis)
239
240
def test_generate_workflow_runs_rollups_and_commits_all_content(self) -> None:
241
workflow_path = Path(".github/workflows/crawl-and-publish.yml")