@cryptotaxi247 / SquadScope-1 / commits / 7ad6e63

Harden analysis gate and preflight evidence

Adds evidence inventories, repo-link validation, structured gate failure summaries, and Copilot-only publishability documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jun 7, 2026 at 10:23 UTC 7ad6e63eb6db441883f72aacef29cff2be9e153c
14 files changed +460 -10
.squad/agents/bender/history.md
+11
@@ -50,3 +50,14 @@
50 - Recommended PRD path is hybrid staged fan-out/fan-in: establish validated artifact contracts first, then gate RSS matrix, GitHub query matrix, and analysis map/reduce on measured thresholds.
51 - Run 27030646485 also showed analysis, not crawling, is the critical-path risk: three Copilot attempts consumed ~28m41s, failed quality gates, GitHub Models had no `openai/gpt-4o` access, and the workflow shipped via no-AI fallback with ~112.9k estimated input tokens.
52 - Issue #249 implementation: weekly analysis now writes to `data/candidates/<week>/<run_id>/` first and emits a `publish_eligibility_v1` manifest before any `data/analyzed/<week>-summary.md` promotion; promotion must fail closed on no-AI, stale source evidence, missing checksums, or failed validation.
53 +- Analysis preflight now emits raw and prompt-visible repository evidence inventories with byte/token/checksum metadata; analysis gate rejects final repo links outside current raw evidence when inventory is available.
54 +
55 +## Issue #287 — Analysis Gate Preflight Hardening (2026-06-06T21:23:50.664Z)
56 +
57 +- ✅ COMPLETE: Implemented evidence inventories (repo, article, source ledgers with integrity checksums)
58 +- ✅ COMPLETE: Repository evidence validation with citation/link integrity
59 +- ✅ COMPLETE: Structured gate failure summaries with classification (contract, context, timeout, fallback, other)
60 +- ✅ COMPLETE: Documentation and test coverage
61 +- pytest: 673 passed, 2 subtests
62 +- PR #288 created; ready for merge after Fry validation
63 +- Orchestration log recorded at `.squad/orchestration-log/20260606T212350Z-bender.md`
.squad/agents/fry/history.md
+8
@@ -109,3 +109,11 @@
109 - Added deterministic no-network regression coverage around publish eligibility manifests and promotion guard behavior.
110 - Good canonical weekly summary/content must remain unchanged when candidates are failed, degraded, no-AI, stale, missing-manifest, or malformed-manifest.
111 - Safe rerun promotion is copy-stable and must not append or duplicate article body content; ineligible candidates should remain in staging with promotion diagnostics for debugging.
112 +
113 +## Issue #287 — Analysis Gate Preflight Hardening (2026-06-06T21:23:50.664Z)
114 +
115 +- ✅ APPROVED: Bender's evidence inventory and gate failure classification implementation
116 +- ✅ Validated: Evidence schema, citation link integrity, gate failure handling across all cases
117 +- ✅ No regressions: pytest full pass (673 passed, 2 subtests)
118 +- PR #288 validated and approved; ready for Coordinator merge workflow
119 +- Orchestration log recorded at `.squad/orchestration-log/20260606T212350Z-fry.md`
.squad/agents/leela/history.md
+9
@@ -46,3 +46,12 @@
46 - Decisions recorded in .squad/decisions.md under four entries (Bender, Farnsworth, Fry, Leela)
47 - 2026-06-05T17:42:56.819+00:00 matrix/map-reduce PRD note: treat crawl matrix and analysis decomposition as separate decisions; keep crawl fan-out measurement-gated, while map/reduce is a citation-preserving LLM context/quality experiment with mapper claim ledgers and reducer-owned editorial coherence.
48 - 2026-06-05T21:03:35.661+00:00 issue hierarchy refresh: updated #248-#259 and created #261 so safe weekly reruns preserve good AI-authored articles, treat no-AI fallback as non-publishable replacement by default, reuse successful same-day per-source evidence artifacts with provenance/freshness guards, and make signal-type claim-ledger map/reduce the selected dry-run analysis architecture.
49 +
50 +## Round 4 (2026-06-06T21:19:24Z)
51 +
52 +- Orchestration session synthesized next priority from team learnings and run 27055543722 diagnosis
53 +- Ranked recommendations: (1) analysis gate & contract validation [P0 blocking], (2) deterministic preflight compaction [P0 supporting], (3) reader-facing methodology & governance path [P1 strategic]
54 +- Merged 10 inbox decisions from Bender, Farnsworth, Fry into .squad/decisions.md; documented orchestration at .squad/orchestration-log/2026-06-06T21-19-24Z-leela.md
55 +- Run 27056632166 post-mortems confirmed safety gates work; no-AI blocking and preservation validated in production
56 +- P0 safety layer (#250/#251/#252/#253) must complete before #254 atomicity can start
57 +- Issue sequencing finalized: contract/preflight (Phase 1) → atomicity/quality gates (Phase 2) → docs/evidence (Phase 3) → map/reduce stays dry-run (Phase 4)
.squad/decisions.md
+208
@@ -1,3 +1,13 @@
1 +## User Directive: Write Issues Before Development [2026-06-06T21:23:50.664+00:00]
2 +
3 +**By:** jmservera (via Copilot)
4 +
5 +**What:** Respect the way we work: write issues for plans before jumping straight on to development.
6 +
7 +**Why:** User request — captured for team memory
8 +
9 +---
10 +
11 # Run 27055543722 — Analysis Failure Root Cause & Diagnosis [2026-06-06]
12
13 ## Executive Summary
@@ -370,3 +380,201 @@ Added small deterministic promotion-guard helper and regression tests for publis
380 **Captured for team memory:** User request that prioritizes Copilot reliability and token lifecycle management over automatic fallback.
381
382
383 +
384 +---
385 +
386 +## Bender: Issue #250 — Preserve Good Analysis on Rerun Failure
387 +
388 +**Date:** 2026-06-06
389 +
390 +**Decision:** Weekly publish reruns fail closed by default. The analyze workflow now emits a rejected no-AI candidate and manifest instead of exiting before manifest creation, hydrates the currently published summary from `publish`, and lets `publish_manifest.py` decide `promote`, `preserve`, or `block`.
391 +
392 +**Rationale:** Downstream jobs and operators need a stable handoff contract. A failed/degraded/no-AI/stale rerun should leave the last good published weekly summary discoverable while preserving the rejected candidate artifacts for diagnosis.
393 +
394 +---
395 +
396 +## Bender: Issue #255 — Regression-Safe Gate Revision
397 +
398 +**Date:** 2026-06-06
399 +
400 +**Decision:**
401 +- Calibrated deterministic publish quality against the current known-good W22 and W23 summaries instead of relying on brittle exact wording checks.
402 +- Kept Copilot CLI on its platform default model operationally, while recording omitted model provenance as `copilot-default` so existing valid Copilot runs are not blocked as `unknown`.
403 +- Final promotion remains fail-closed: `validation.gate_report.passed` and all required structural/provenance/evidence/editorial gate families must be present and passing.
404 +- Rejected no-AI fallback candidates now still emit a manifest and gate report before the workflow exits, preserving diagnostics without promotion.
405 +
406 +---
407 +
408 +## User Directive: Review Copilot PR Comments
409 +
410 +**Date:** 2026-06-06T17:29:38.291+00:00
411 +**By:** jmservera (via Copilot)
412 +
413 +**Directive:** Review all Copilot Review PR comments and resolve them before considering PR work complete.
414 +
415 +**Reason:** User request — captured for team memory.
416 +
417 +---
418 +
419 +## Farnsworth: Issue #251 — No-AI Fallback Policy
420 +
421 +**Date:** 2026-06-06T08:48:43.587+00:00
422 +
423 +**Decision:** No-AI weekly fallback output is diagnostic by default and cannot replace an existing good AI-authored article unless an operator selects `force-replace` and supplies actor/reason audit metadata.
424 +
425 +**Implementation Note:** Explicit first publish of no-AI fallback uses `allow-no-ai-first-publish`, requires no existing good AI article, a passing analysis gate, source provenance, attempted AI paths, and quality_score >= 70.
426 +
427 +---
428 +
429 +## Farnsworth: Issue #255 — Publish Quality Gate
430 +
431 +**Date:** 2026-06-06T08:48:43.587+00:00
432 +
433 +**Decision:** Keep structural analysis validation deterministic, but make publication depend on a structured gate report with separate structural schema, AI provenance, evidence/citation, and editorial-quality gate families. The publish manifest records those gate outcomes and promotion consumes them before replacing a published article.
434 +
435 +**Rationale:** This preserves existing schema repair behavior while preventing structurally valid but low-quality, stale-evidence-backed, contradictory, or no-AI fallback summaries from becoming publishable artifacts.
436 +
437 +---
438 +
439 +## Fry: QA Audit Decision Memo — Run #27056632166
440 +
441 +**Date:** 2026-06-06T08:38:45.537+00:00
442 +**Context:** Run 27056632166 succeeded end-to-end after PRs #267, #268, #271, #272 merged.
443 +
444 +### Immediate Actions — Close These Issues
445 +- **#251** (Block no-AI fallback from replacing AI-authored summaries)
446 + - ✅ Promotion guard rejects no-ai source; manifest marks no-ai ineligible
447 + - ✅ Run 27056632166 produced no-ai but did NOT promote (guard worked)
448 + - **Verdict:** CLOSE — blocking confirmed in production
449 +
450 +- **#250** (Preserve existing good weekly analysis on failed/degraded reruns)
451 + - ✅ Promotion guard rejects degraded/failed candidates
452 + - ✅ Workflow copies only eligible week; does not overwrite prior
453 + - ✅ Test `test_same_successful_rerun_is_stable_and_does_not_duplicate_content` passing
454 + - **Verdict:** CLOSE — preservation confirmed in production
455 +
456 +### Defer These Issues (Design-Complete, Waiting for Upstream Work)
457 +- **#258** (Add map/reduce dry-run) — Deliberately deferring until #255/#257 (editorial/evidence gates) prove reliable in production. Timeline: 2-3 successful weeks with full gates, then begin map/reduce sidecar.
458 +- **#256** (Add preflight compaction and fallback policy) — Design complete (decisions.md documented); needs to run after this week's analysis completes.
459 +
460 +### Immediate PRs Needed (Next Week)
461 +1. **#255:** Implement editorial_quality_gate and evidence_freshness_gate (manifests and guards check for these fields but they're never populated)
462 +2. **#261:** Implement same-day source artifact reuse in crawl (guard exists but feature missing)
463 +3. **#259:** Create operator guide and safe rerun playbook
464 +4. **#273:** Document model routing policy (decision memo only)
465 +
466 +### Test Results & Validation
467 +- **Test Suite:** 562/563 passing (1 fixture data update needed)
468 +- **Safety Features Verified in Production:**
469 + - No-AI blocking: ✅ Works
470 + - Preservation: ✅ Works
471 + - Publish sync: ✅ Works
472 + - Manifest guards: ✅ Work
473 +
474 +### Recommendation
475 +✅ **Close #250 and #251** — safety verdicts confirmed in production.
476 +
477 +⚡ **Prioritize next-week PRs:** #255 (editorial/evidence gates) > #261 (same-day reuse) > #259 (operator docs) > #273 (model routing).
478 +
479 +---
480 +
481 +## Fry: Review Approval — Issue #250
482 +
483 +**Date:** 2026-06-06T08:48:43.587+00:00
484 +**Verdict:** APPROVE
485 +
486 +**Reviewed Commit:** `3f78be6632eea9ffeeb99c5a1eed1efad4af1648` on branch `squad/250-preserve-good-analysis`.
487 +
488 +**Rationale:**
489 +- Existing good weekly summaries are detected through markdown metadata plus prior candidate manifest provenance when available.
490 +- Failed validation, no-AI, stale evidence, and lower-quality candidates become ineligible and choose `promotion.decision: preserve` when a good published summary exists.
491 +- Preserve manifests record both `preserved_summary_path` and `rejected_candidate_path`; rejected candidate artifacts remain under `data/candidates/YYYY-WNN/<run-id>/`.
492 +
493 +**Tests Run:** 17 passed. Custom validation-failure preservation smoke test passed. Full suite: 602 passed, 1 unrelated existing failure.
494 +
495 +**Non-blocking Follow-up:** Refresh stale prediction fixture/test expectation so full suite is green again.
496 +
497 +---
498 +
499 +## Fry: Review Approval — Issue #251
500 +
501 +**Date:** 2026-06-06T08:48:43.587+00:00
502 +**Verdict:** APPROVE
503 +
504 +**Commit Reviewed:** 18e521e
505 +
506 +**Rationale:**
507 +- No-AI fallback candidates are blocked from default promotion, including over existing good AI-authored summaries.
508 +- No-existing-article behavior stays fail-closed by default and only permits first publish with explicit `allow-no-ai-first-publish` plus quality gates.
509 +- Force replacement requires explicit `force-replace` mode, actor, reason, fallback provenance, attempted AI paths, and emits manifest/audit data.
510 +
511 +**Validation:** `test_publish_manifest.py`, `test_promotion_guard.py`, and `test_pipeline.py` passed. Full suite: 606 passed, 1 failed (known unrelated fixture drift).
512 +
513 +**Non-blocking Follow-up:** Consider adding documented workflow-dispatch inputs for force replacement if operators are expected to use the escape hatch through GitHub Actions.
514 +
515 +---
516 +
517 +## Issue #255 — Manifest/Promotion Compatibility Decision Note
518 +
519 +**Date:** 2026-06-06
520 +
521 +**Context:** Leela rejected the prior #255 revision because `publish_manifest.py create/assert-eligible` could approve `data/candidates/` manifests with `generated_at` and nested `candidate.summary_path`, while `promotion_guard.py` still required `data/staging/`, `run_started_at`, and `candidate_content_path`.
522 +
523 +**Decision:**
524 +- Keep the current candidate manifest workflow intact and make the contract explicitly compatible in both directions.
525 +- `publish_manifest.py` emits compatibility aliases alongside the existing nested manifest fields.
526 +- `promotion_guard.py` accepts both `data/staging/` and `data/candidates/` manifest roots.
527 +- `promotion_guard.py` normalizes `generated_at` as a run timestamp fallback and uses the candidate summary as the content fallback for legacy summary-only manifests.
528 +- Gate-family checks remain fail-closed, and no-AI/degraded/rejected candidates remain blocked.
529 +
530 +**Verification:** Added regression tests proving an eligible manifest created and accepted by `publish_manifest assert-eligible` is accepted by `promotion_guard.promote_candidate()`, while a rejected/no-AI manifest is rejected by both without replacing existing published artifacts. Full suite passed.
531 +
532 +---
533 +
534 +## Leela: Issue Triage After Successful Run #27056632166
535 +
536 +**Date:** 2026-06-06T08:38:45.537+00:00
537 +
538 +**Context:** Run #27056632166 succeeded end-to-end; PRs #267, #268, #271, #272 merged (prediction schema repair, Copilot-only analysis fix, data sync).
539 +
540 +### P0 Critical Safety Layer Issues (All Remain Open)
541 +- **#250** (Preserve good weekly on failed rerun) — Blocking #254, needs implementation
542 +- **#251** (Block no-AI fallback) — Blocking #254, needs implementation
543 +- **#252** (Explicit safe rerun modes) — Blocking #254/#259, needs implementation
544 +- **#253** (Immutable backups) — Blocking #254, needs implementation
545 +- **#254** (Atomic weekly promotion) — Blocked by above four, cannot start yet
546 +
547 +### P0 Quality & Preflight Gates
548 +- **#255** (Strengthen publish gate) — Supports #254, not blocking, needs implementation
549 +- **#256** (Preflight compaction) — Feeds #258 map/reduce, P1 priority
550 +
551 +### P2 Analysis Architecture (Dry-Run Hold)
552 +- **#258** (Map/reduce dry-run) — **REMAINS DRY-RUN ONLY** until P0 safety layer complete. Do not promote until #250/#251/#252/#253/#254/#255/#256/#257 gates pass.
553 +
554 +### P1 Documentation & Evidence Tasks
555 +- **#259** (Document safe rerun/restore) — Depends on #252/#261 implementation, deferred
556 +- **#261** (Reuse same-day artifacts) — Supports #259/#258 evidence freshness
557 +- **#273** (Model routing policy) — Research complete, writeup needed, can start immediately
558 +
559 +### Recommended Work Order
560 +
561 +**Phase 1 (Next): P0 Safety Gates (Parallel Track)**
562 +1. #250, #251, #252, #253 — Implement in parallel; target all 4 complete within same sprint
563 +
564 +**Phase 2: Atomicity & Quality Gates**
565 +1. #254 — Start after Phase 1 complete
566 +2. #255, #256 — In parallel with #254
567 +
568 +**Phase 3: Documentation & Evidence Freshness**
569 +1. #259 — After #252 implementation lands
570 +2. #261 — Parallel or after crawl metrics complete
571 +3. #273 — No blocking dependencies; start anytime
572 +
573 +**Phase 4: Analysis Innovation (Deferred)**
574 +- #258 — Remains dry-run/tests-only until Phase 1-2 complete
575 +
576 +### Key Sequencing Constraints
577 +- ⛔ **#254 cannot start** until all of (#250, #251, #252, #253) are implemented and reviewed
578 +- ⛔ **#258 map/reduce must stay dry-run** until P0 safety layer is solid
579 +- ✓ **#273 (model routing)** has zero blocking dependencies and can start today
580 +
.squad/log/20260606T212350Z-issue-287-analysis-hardening.md new
+14
@@ -0,0 +1,14 @@
1 +# Session Log — 2026-06-06T21:23:50.664Z — Issue #287 Analysis Hardening
2 +
3 +## Context
4 +Team completed P0 analysis gate and preflight hardening work (issue #287) after Bender's implementation and Fry's validation.
5 +
6 +## Session Work
7 +- Pre-check: decisions.md at 31,654 bytes; 1 inbox file
8 +- Archive gate: No entries > 30 days old (only 2026-06-06 entry exists)
9 +- Merged inbox directive: "Write issues for plans before development"
10 +- Created orchestration logs for Bender and Fry
11 +- Updated team memory with user directive
12 +
13 +## Status
14 +✅ Documentation workflow synchronized. Ready for next sprint cycle.
.squad/orchestration-log/20260606T212350Z-bender.md new
+21
@@ -0,0 +1,21 @@
1 +# Bender Orchestration Log — 2026-06-06T21:23:50.664Z
2 +
3 +## Task
4 +Implement P0 analysis gate/preflight hardening for issue #287
5 +
6 +## Outcome
7 +✅ COMPLETE
8 +
9 +- Implemented evidence inventories with repo/article/source ledgers
10 +- Added repository evidence validation with citation/link integrity checks
11 +- Structured gate failure summaries with classification (contract, context, timeout, fallback, other)
12 +- Added comprehensive documentation in `docs/analysis-gate-preflight.md`
13 +- Full pytest passed (673 passed, 2 subtests)
14 +
15 +## Artifacts
16 +- Issue #287 created and tracked
17 +- PR #288 opened on branch `squad/287-analysis-gate-preflight-hardening`
18 +- Changes committed with full test validation
19 +
20 +## Notes
21 +Ready for Fry review and merge.
.squad/orchestration-log/20260606T212350Z-fry.md new
+22
@@ -0,0 +1,22 @@
1 +# Fry Orchestration Log — 2026-06-06T21:23:50.664Z
2 +
3 +## Task
4 +Review and validate Bender's changes for issue #287
5 +
6 +## Outcome
7 +✅ APPROVED
8 +
9 +- Reviewed Bender's analysis gate/preflight hardening implementation
10 +- Verified evidence inventory structure and validation logic
11 +- Confirmed gate failure classification and structured reporting
12 +- Full pytest passed (673 passed, 2 subtests)
13 +- No regressions detected
14 +
15 +## Validation
16 +- Evidence inventory schemas validated
17 +- Citation link integrity checks tested
18 +- Gate failure handling verified across all case types
19 +- Documentation reviewed and complete
20 +
21 +## Notes
22 +PR #288 ready for merge after Coordinator integrates issue/branch/PR workflow.
docs/analysis-spec.md
+4
@@ -344,6 +344,10 @@ Publication also requires a structured gate report with four passing gate famili
344
345 The publish manifest records these gate outcomes and promotion must consume them before replacing a previously published AI-authored article.
346
347 +The preflight stage must also emit a deterministic analysis input manifest (`analysis-preflight.json`) before Copilot runs. That manifest records the rendered prompt byte/token estimate, prompt checksum, every prompt component with byte/token/checksum metadata, deterministic slice names, and evidence inventories for raw and prompt-visible repository sets. Each evidence inventory lists resolvable `owner/repo` names, source slice, GitHub URL when present, stars, and `stars_gained` when present. The gate must reject repository markdown links that do not resolve to the current raw evidence inventory when that inventory is available.
348 +
349 +Gate reports are machine-readable failure contracts. They include per-family `gates`, repair actions, `failure_class`, and a structured `failure_summary` with failed categories and error count so workflow retries and publish preservation can classify failures deterministically.
350 +
351 ## Generator Handoff Rules
352
353 The generator may assume:
docs/operator-guide.md
+12 -3
@@ -502,7 +502,7 @@ data/candidates/YYYY-WNN/RUN_ID/
502 ├── YYYY-WNN-content.md # Generated HTML candidate (if produced)
503 ├── publish-manifest.json # Eligibility and provenance
504 └── diagnostics/
505 - ├── analysis-preflight.json # Pre-analysis context budget check
505 + ├── analysis-preflight.json # Deterministic input manifest, context budget, evidence inventory
506 ├── analysis-preflight.md # Preflight diagnostic report
507 ├── copilot-cli-attempt-N.log # Raw Copilot CLI stderr/stdout
508 ├── gate-copilot-cli-attempt-N.json # Quality gate failure details
@@ -520,7 +520,8 @@ cat data/candidates/2026-W21/RUN_ID/diagnostics/gate-copilot-cli-attempt-0.json
520 gates: .gates,
521 errors_before_repair: .errors_before_repair,
522 repair_actions: .repair_actions,
523 - failure_class: .failure_class
523 + failure_class: .failure_class,
524 + failure_summary: .failure_summary
525 }'
526 ```
527
@@ -561,10 +562,18 @@ Gate report output:
562 "Added 3 spurious/false claims to Noise section"
563 ],
564 "errors_after_repair": [],
564 - "failure_class": "passed"
565 + "failure_class": "passed",
566 + "failure_summary": {
567 + "failure_class": "passed",
568 + "failure_categories": [],
569 + "error_count": 0,
570 + "retryable": false
571 + }
572 }
573 ```
574
575 +`analysis-preflight.json` is the source of truth for prompt inputs. It includes byte/token/checksum metadata for each prompt component and evidence inventories (`raw_new_repos`, `raw_trending_repos`, `prompt_new_repos`, `prompt_trending_repos`) so operators can verify whether a final repo link was present in current crawl evidence or only in compacted prompt context.
576 +
577 ### Quality gate specifics
578
579 The publish manifest records:
docs/pipeline-validation.md
+3 -1
@@ -65,8 +65,10 @@ Required secrets/tokens:
65 - Current raw file week matches the run week
66 - Correlation and press-context steps consume compact external-news data with legacy `YYYY-WNN-techcrunch.json` fallback
67 - Press context preserves source names, article URLs/titles/dates, strong-vs-weak labels, and partial-source caveats while staying under the ~8k token budget
68 +- Analysis preflight writes `analysis-preflight.json` with deterministic prompt component byte/token/checksum metadata and raw/prompt evidence inventories before Copilot is invoked.
69 - Copilot CLI output is staged under `data/candidates/YYYY-WNN/<run-id>/`; if Copilot cannot produce publishable analysis, a no-AI candidate is kept as a rejected artifact and the run fails closed before promotion.
69 -- `scripts/analysis_gate.py` passes before publish continues, and `scripts/publish_manifest.py` records promote-vs-preserve decisions against any existing good published summary.
70 +- `scripts/analysis_gate.py` passes before publish continues, rejects repo links outside the current raw evidence inventory where available, and emits structured failure summaries for deterministic retry/preserve decisions.
71 +- `scripts/publish_manifest.py` records promote-vs-preserve decisions against any existing good published summary.
72 - Job permissions include `actions: read`, `contents: write`, and `issues: write`
73
74 ### 3. Generate
scripts/analysis_gate.py
+18 -2
@@ -43,7 +43,7 @@ REQUIRED_HEADINGS = [
43 "### Notable Projects",
44 "### Press & Industry",
45 ]
46 -PUBLISHABLE_AI_SOURCES = {"copilot-cli", "github-models"}
46 +PUBLISHABLE_AI_SOURCES = {"copilot-cli"}
47 UNPUBLISHABLE_MODEL_VALUES = {"", "unknown", "unavailable", "none", "no-ai"}
48 RAW_MARKERS = [
49 "```json",
@@ -513,6 +513,11 @@ def evidence_citation_errors(body: str, raw_payload: dict[str, Any]) -> list[str
513 linked_repos = set(REPO_LINK_PATTERN.findall(body))
514 if repos and not linked_repos.intersection(repos):
515 errors.append("evidence citations must include at least one repository link from the raw payload.")
516 + unresolved_links = sorted(linked_repos - repos) if repos else []
517 + if unresolved_links:
518 + preview = ", ".join(unresolved_links[:10])
519 + suffix = f" (+{len(unresolved_links) - 10} more)" if len(unresolved_links) > 10 else ""
520 + errors.append(f"repository links must resolve to the current raw evidence inventory: {preview}{suffix}.")
521 if repos and "## Key References" in body:
522 notable = section_text(body, "## Key References")
523 notable_links = set(REPO_LINK_PATTERN.findall(notable))
@@ -573,7 +578,7 @@ def ai_provenance_errors(source: str, model: str) -> list[str]:
578 def categorize_gate_error(error: str) -> str:
579 if error.startswith("AI provenance"):
580 return "ai_provenance"
576 - if error.startswith(("evidence citations", "Key References", "raw evidence")):
581 + if error.startswith(("evidence citations", "Key References", "raw evidence", "repository links")):
582 return "evidence_citation"
583 if error.startswith(("editorial analysis", "contradictory claim")) or "section is too thin" in error or "must explain why" in error:
584 return "editorial_quality"
@@ -742,6 +747,7 @@ def write_gate_report(
747 "passed": not errors_after,
748 "word_count": word_count,
749 "gates": gate_results,
750 + "failure_summary": build_failure_summary(errors_after, gate_results),
751 "errors_before_repair": errors_before,
752 "repair_actions": repair_actions,
753 "errors_after_repair": errors_after,
@@ -750,6 +756,16 @@ def write_gate_report(
756 path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
757
758
759 +def build_failure_summary(errors: list[str], gate_results: dict[str, dict[str, Any]]) -> dict[str, Any]:
760 + categories = sorted(category for category, result in gate_results.items() if not result.get("passed"))
761 + return {
762 + "failure_class": classify_gate_errors(errors),
763 + "failure_categories": categories,
764 + "error_count": len(errors),
765 + "retryable": bool(errors) and not any(category == "ai_provenance" for category in categories),
766 + }
767 +
768 +
769 def classify_gate_errors(errors: list[str]) -> str:
770 if not errors:
771 return "passed"
scripts/analyze_fallback.py
+69
@@ -51,6 +51,26 @@ class PromptComponent:
51 checksum_sha256: str
52
53
54 +@dataclass
55 +class EvidenceRepoRef:
56 + full_name: str
57 + url: str | None
58 + source: str
59 + stars: int | None
60 + stars_gained: int | None
61 +
62 +
63 +@dataclass
64 +class EvidenceInventory:
65 + name: str
66 + path: str
67 + item_count: int
68 + bytes: int
69 + token_estimate: int
70 + checksum_sha256: str
71 + repos: list[EvidenceRepoRef]
72 +
73 +
74 @dataclass
75 class PromptPreflight:
76 prompt_token_budget: int
@@ -65,6 +85,7 @@ class PromptPreflight:
85 fallback_policy: str
86 components: list[PromptComponent]
87 deterministic_slices: list[str]
88 + evidence_inventories: list[EvidenceInventory]
89
90
91 def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
@@ -165,6 +186,48 @@ def _component(
186 )
187
188
189 +def _repo_int(value: Any) -> int | None:
190 + return value if isinstance(value, int) and not isinstance(value, bool) else None
191 +
192 +
193 +def _inventory_repo_refs(payload: dict[str, Any], field: str) -> list[EvidenceRepoRef]:
194 + repos = payload.get(field)
195 + if not isinstance(repos, list):
196 + return []
197 + refs: list[EvidenceRepoRef] = []
198 + for repo in repos:
199 + if not isinstance(repo, dict):
200 + continue
201 + full_name = repo.get("full_name")
202 + if not isinstance(full_name, str) or "/" not in full_name:
203 + continue
204 + url = repo.get("url")
205 + refs.append(
206 + EvidenceRepoRef(
207 + full_name=full_name.strip(),
208 + url=url if isinstance(url, str) and url.strip() else None,
209 + source=field,
210 + stars=_repo_int(repo.get("stars")),
211 + stars_gained=_repo_int(repo.get("stars_gained")),
212 + )
213 + )
214 + return refs
215 +
216 +
217 +def _evidence_inventory(name: str, payload: dict[str, Any], field: str, path: Path) -> EvidenceInventory:
218 + content = json.dumps(payload.get(field, []), indent=2, ensure_ascii=False)
219 + repos = _inventory_repo_refs(payload, field)
220 + return EvidenceInventory(
221 + name=name,
222 + path=path.as_posix(),
223 + item_count=len(repos),
224 + bytes=len(content.encode("utf-8")),
225 + token_estimate=estimate_tokens(content),
226 + checksum_sha256=checksum_text(content),
227 + repos=repos,
228 + )
229 +
230 +
231 def truncate_with_notice(content: str, limit: int, label: str) -> tuple[str, str]:
232 if len(content) <= limit:
233 return content, "included"
@@ -473,6 +536,12 @@ def _build_prompt(
536 ),
537 components=components,
538 deterministic_slices=["new_repos", "trending_repos", "press_correlations", "prior_continuity"],
539 + evidence_inventories=[
540 + _evidence_inventory("raw_new_repos", sanitized_payload, "new_repos", raw_json_path),
541 + _evidence_inventory("raw_trending_repos", sanitized_payload, "trending_repos", raw_json_path),
542 + _evidence_inventory("prompt_new_repos", payload_for_prompt, "new_repos", raw_json_path),
543 + _evidence_inventory("prompt_trending_repos", payload_for_prompt, "trending_repos", raw_json_path),
544 + ],
545 )
546 return prompt, preflight
547
tests/test_analysis_gate.py
+48 -4
@@ -12,7 +12,10 @@ RAW_PAYLOAD_WITH_REPOS = {
12 "week": "2026-W23",
13 "crawled_at": "2026-06-01T00:00:00Z",
14 "new_repos": [{"full_name": "owner/repo", "stars": 1000}],
15 - "trending_repos": [{"full_name": "owner/repo-b", "stars": 200}],
15 + "trending_repos": [
16 + {"full_name": "owner/repo-a", "stars": 300},
17 + {"full_name": "owner/repo-b", "stars": 200},
18 + ],
19 }
20 CURRENT_DATETIME = "2026-06-01T00:00:00Z"
21
@@ -227,8 +230,8 @@ summary: "A grounded week focused on practical tools."'''.strip()
230 self.assertEqual(errors, ["predictions[1].claim_type must be one of signal, noise, gap."])
231 self.assertIn("set date from current run timestamp", actions)
232 self.assertNotIn("claim_type", frontmatter_after["predictions"][0])
230 - self.assertEqual(frontmatter_after["repos_featured"], 2)
231 - self.assertEqual(frontmatter_after["stars_tracked"], 1200)
233 + self.assertEqual(frontmatter_after["repos_featured"], 3)
234 + self.assertEqual(frontmatter_after["stars_tracked"], 1500)
235
236 def test_repair_analysis_normalizes_safe_prediction_claim_alias(self) -> None:
237 frontmatter = VALID_FRONTMATTER + "\npredictions:\n - repo: owner/repo\n claim: Signal\n direction: UP\n confidence: 0.7"
@@ -445,6 +448,12 @@ No press data was provided this week.
448 "trending_repos": [],
449 }
450 text = summary_path.read_text(encoding="utf-8")
451 + linked_repos = sorted(analysis_gate.REPO_LINK_PATTERN.findall(text))
452 + raw_payload["new_repos"].extend(
453 + {"full_name": name, "stars": 100}
454 + for name in linked_repos
455 + if name != repo_name
456 + )
457
458 structure_errors, word_count = analysis_gate.validate_analysis(text, raw_payload, crawled_at)
459 publish_errors, gates = analysis_gate.validate_publish_quality(
@@ -482,7 +491,10 @@ No press data was provided this week.
491 self.assertTrue(gates["ai_provenance"]["passed"])
492
493 def test_publish_quality_gate_rejects_missing_evidence_citations(self) -> None:
485 - body = make_body().replace("[owner/repo-b](https://github.com/owner/repo-b)", "owner/repo-b")
494 + body = make_body().replace("[owner/repo-a](https://github.com/owner/repo-a)", "owner/repo-a").replace(
495 + "[owner/repo-b](https://github.com/owner/repo-b)",
496 + "owner/repo-b",
497 + )
498 errors, gates = analysis_gate.validate_publish_quality(
499 make_analysis(VALID_FRONTMATTER, body),
500 RAW_PAYLOAD_WITH_REPOS,
@@ -493,6 +505,18 @@ No press data was provided this week.
505 self.assertIn("evidence citations must include at least one repository link from the raw payload.", errors)
506 self.assertFalse(gates["evidence_citation"]["passed"])
507
508 + def test_publish_quality_gate_rejects_repo_links_outside_current_inventory(self) -> None:
509 + body = make_body().replace("[owner/repo-a](https://github.com/owner/repo-a)", "[other/repo](https://github.com/other/repo)")
510 + errors, gates = analysis_gate.validate_publish_quality(
511 + make_analysis(VALID_FRONTMATTER, body),
512 + RAW_PAYLOAD_WITH_REPOS,
513 + source="copilot-cli",
514 + model="copilot-default",
515 + )
516 +
517 + self.assertIn("repository links must resolve to the current raw evidence inventory: other/repo.", errors)
518 + self.assertFalse(gates["evidence_citation"]["passed"])
519 +
520 def test_publish_quality_gate_rejects_stale_evidence(self) -> None:
521 stale_payload = dict(RAW_PAYLOAD_WITH_REPOS, crawled_at="2026-05-25T00:00:00Z")
522 errors, gates = analysis_gate.validate_publish_quality(
@@ -533,6 +557,26 @@ No press data was provided this week.
557 self.assertIn("AI provenance model is not publishable: none.", errors)
558 self.assertFalse(gates["ai_provenance"]["passed"])
559
560 + def test_publish_quality_gate_rejects_github_models_provenance(self) -> None:
561 + errors, gates = analysis_gate.validate_publish_quality(
562 + make_analysis(VALID_FRONTMATTER, make_body()),
563 + RAW_PAYLOAD,
564 + source="github-models",
565 + model="openai/gpt-4o",
566 + )
567 +
568 + self.assertIn("AI provenance source is not publishable: github-models.", errors)
569 + self.assertFalse(gates["ai_provenance"]["passed"])
570 +
571 + def test_gate_report_includes_structured_failure_summary(self) -> None:
572 + errors = ["repository links must resolve to the current raw evidence inventory: other/repo."]
573 + gates = analysis_gate.build_gate_results(errors)
574 + summary = analysis_gate.build_failure_summary(errors, gates)
575 +
576 + self.assertEqual(summary["failure_class"], "evidence_citation")
577 + self.assertEqual(summary["failure_categories"], ["evidence_citation"])
578 + self.assertEqual(summary["error_count"], 1)
579 +
580 def test_publish_quality_gate_rejects_contradictory_press_claims(self) -> None:
581 body = make_body() + "\n\nNo press data was provided this week, but TechCrunch reported a major launch."
582 errors, gates = analysis_gate.validate_publish_quality(
tests/test_analyze_fallback.py
+13
@@ -237,6 +237,11 @@ class AnalyzeFallbackTests(unittest.TestCase):
237 components = {component["name"]: component for component in report["components"]}
238 self.assertEqual(components["new_repos"]["inclusion_reason"], "Deterministic mapper slice: newly discovered repositories.")
239 self.assertEqual(components["trending_repos"]["compaction_decision"], "included")
240 + inventories = {inventory["name"]: inventory for inventory in report["evidence_inventories"]}
241 + self.assertEqual(inventories["raw_new_repos"]["item_count"], 1)
242 + self.assertEqual(inventories["raw_new_repos"]["repos"][0]["full_name"], "owner/new")
243 + self.assertEqual(inventories["raw_trending_repos"]["repos"][0]["stars_gained"], 5)
244 + self.assertGreater(inventories["prompt_new_repos"]["token_estimate"], 0)
245
246 def test_preflight_compacts_before_prompt_exceeds_budget(self) -> None:
247 tests_root = Path(__file__).resolve().parent
@@ -298,6 +303,14 @@ class AnalyzeFallbackTests(unittest.TestCase):
303 components = {component["name"]: component for component in report["components"]}
304 self.assertIn("compacted to top", components["new_repos"]["compaction_decision"])
305 self.assertIn("compacted to top", components["trending_repos"]["compaction_decision"])
306 + inventories = {inventory["name"]: inventory for inventory in report["evidence_inventories"]}
307 + self.assertEqual(inventories["raw_new_repos"]["item_count"], 60)
308 + self.assertEqual(inventories["prompt_new_repos"]["item_count"], analyze_fallback.COMPACTED_NEW_REPOS_LIMIT)
309 + self.assertEqual(inventories["raw_trending_repos"]["item_count"], 60)
310 + self.assertEqual(
311 + inventories["prompt_trending_repos"]["item_count"],
312 + analyze_fallback.COMPACTED_TRENDING_REPOS_LIMIT,
313 + )
314
315 def test_extract_markdown_supports_message_parts(self) -> None:
316 payload = {