@cryptotaxi247 / SquadScope / commits / 891e1e5

QA guard prediction schema repair loop (#267)

* Scribe: Consolidate run 27055543722 analysis failure diagnosis Merged decision inbox notes from Farnsworth, Bender, Fry, and Leela: - Farnsworth analysis failure root cause and reliable process proposal - Bender analysis pipeline failure root cause and mechanical causes - Fry analysis reliability QA plan and test cases - Leela analysis failure plan and issue hierarchy Key findings: - Prompt/spec/gate schema drift: predictions contract mismatch - Context bloat: 38% token undercount (74k preflight vs 113k ledger) - Blind deterministic retry loops without error classification - Unavailable AI fallback (GitHub Models 403 no_access) - No-AI provenance not enforced (safety manifest blocked this run) Immediate recommendations for next run: 1. Align prediction contract across docs, prompt, gate 2. Exact rendered-prompt preflight with component token counts 3. Deterministic frontmatter injection and focused repair loop 4. Analysis-specific wisdom capsule (1-2k tokens, not full skills) 5. GitHub Models access preflight before Copilot attempts 6. Direct analyzer invocation (not Squad agent wrapper) 7. No-AI flagged as diagnostic-only, not fallback recovery Acceptance criteria and cross-team notes documented. Related issues: #248-#261, #265, #266 Processed inbox files deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: guard analysis prediction schema repair Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: record analysis schema learning Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: guard retry-after header access Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: harden analysis retry diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: keep gate diagnostics best-effort Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Juan Manuel Servera committed Jun 6, 2026 at 09:47 UTC 891e1e5df9f07e190a4539177066fd35c6879bec

10 files changed +551 -16

.github/workflows/crawl-and-publish.yml

+40 -8

@@ -369,7 +369,8 @@ jobs:

369 CURRENT_DATETIME="${{ steps.analysis-context.outputs.current_datetime }}"

370 PRESS_FILE="${{ steps.press-context.outputs.press_file }}"

371 ANALYSIS_STARTED=$(date +%s)

372 - mkdir -p data/metrics "$(dirname "$OUTPUT_FILE")"

372 + DIAGNOSTICS_DIR="$(dirname "$OUTPUT_FILE")/diagnostics"

373 + mkdir -p data/metrics "$(dirname "$OUTPUT_FILE")" "$DIAGNOSTICS_DIR"

374 # Hydrate metrics ledger from publish so track_token_usage.py appends to

375 # the canonical token-usage.jsonl rather than starting fresh each run.

376 git fetch origin publish 2>/dev/null && \

@@ -392,7 +393,9 @@ jobs:

393 --analysis-file "$OUTPUT_FILE" \

394 --raw-json "$WEEK_FILE" \

395 --current-datetime "$CURRENT_DATETIME" \

395 - --source "$1"

396 + --source "$1" \

397 + --repair-safe \

398 + --report-json "$2"

399 }

400

401 # Retry loop: LLM output can be non-deterministically truncated,

@@ -402,11 +405,12 @@ jobs:

405 GATE_PASSED=false

406 ANALYSIS_SOURCE=""

407 ANALYSIS_MODEL=""

408 + LAST_GATE_FINGERPRINT=""

409

410 if command -v copilot >/dev/null 2>&1; then

411 while [ "$GATE_PASSED" = "false" ] && [ "$ATTEMPT" -le "$MAX_RETRIES" ]; do

412 if [ "$ATTEMPT" -gt 0 ]; then

409 - echo "::warning::Quality gate failed on attempt $ATTEMPT, retrying (attempt $((ATTEMPT + 1))/$((MAX_RETRIES + 1)))..."

413 + echo "::warning::Quality gate failed on attempt $ATTEMPT; retrying with focused gate diagnostics (attempt $((ATTEMPT + 1))/$((MAX_RETRIES + 1)))..."

414 rm -f "$OUTPUT_FILE"

415 # Reset any .squad changes from failed attempt

416 git checkout -- .squad 2>/dev/null || true

@@ -414,12 +418,21 @@ jobs:

418

419 echo "::notice::Running Copilot analysis attempt $((ATTEMPT + 1))/$((MAX_RETRIES + 1))"

420 TRANSCRIPT_FILE="data/metrics/copilot-transcript-attempt-${ATTEMPT}.md"

421 + GATE_REPORT="$DIAGNOSTICS_DIR/gate-copilot-cli-attempt-${ATTEMPT}.json"

422 + CANDIDATE_SNAPSHOT="$DIAGNOSTICS_DIR/candidate-copilot-cli-attempt-${ATTEMPT}.md"

423 rm -f "$TRANSCRIPT_FILE"

424 + REPAIR_CONTEXT=""

425 + if [ "$ATTEMPT" -gt 0 ]; then

426 + PREVIOUS_REPORT="$DIAGNOSTICS_DIR/gate-copilot-cli-attempt-$((ATTEMPT - 1)).json"

427 + if [ -f "$PREVIOUS_REPORT" ]; then

428 + REPAIR_CONTEXT=" Previous gate report: ${PREVIOUS_REPORT}. Correct exactly those validation errors; do not regenerate unrelated content."

429 + fi

430 + fi

431

432 # Intentionally rely on Copilot CLI's default model so CI follows the platform-supported default.

433 if ! copilot \

434 --agent squad \

422 - -p "Farnsworth, read the file at ${PROMPT_FILE} — it contains the weekly data and analysis instructions. Follow them exactly and write the analysis to ${OUTPUT_FILE}." \

435 + -p "Farnsworth, read the file at ${PROMPT_FILE} — it contains the weekly data and analysis instructions. Follow them exactly and write the analysis to ${OUTPUT_FILE}.${REPAIR_CONTEXT}" \

436 -s \

437 --no-ask-user \

438 --allow-tool=read \

@@ -435,12 +448,26 @@ jobs:

448

449 sanitize_agent_output "$OUTPUT_FILE"

450

438 - # Inline quality gate check (suppress step summary to avoid noise)

439 - if run_quality_gate copilot-cli; then

451 + # Inline quality gate check (suppress step summary to avoid noise).

452 + # The gate applies deterministic metadata/schema repairs and writes an auditable report.

453 + if run_quality_gate copilot-cli "$GATE_REPORT"; then

454 + cp "$OUTPUT_FILE" "$CANDIDATE_SNAPSHOT" 2>/dev/null || true

455 GATE_PASSED=true

456 FINAL_TRANSCRIPT="$TRANSCRIPT_FILE"

457 ANALYSIS_SOURCE="copilot-cli"

458 ANALYSIS_MODEL="copilot-default"

459 + else

460 + cp "$OUTPUT_FILE" "$CANDIDATE_SNAPSHOT" 2>/dev/null || true

461 + CURRENT_GATE_FINGERPRINT=$(python3 -c 'import sys; from pathlib import Path; import scripts.analysis_gate as gate; print(gate.gate_report_fingerprint(Path(sys.argv[1])))' "$GATE_REPORT" 2>/dev/null || true)

462 + if [ -n "$CURRENT_GATE_FINGERPRINT" ]; then

463 + if [ -n "$LAST_GATE_FINGERPRINT" ] && [ "$CURRENT_GATE_FINGERPRINT" = "$LAST_GATE_FINGERPRINT" ]; then

464 + echo "::error::Repeated deterministic analysis gate failure after repair. See ${GATE_REPORT} and ${CANDIDATE_SNAPSHOT}."

465 + break

466 + fi

467 + LAST_GATE_FINGERPRINT="$CURRENT_GATE_FINGERPRINT"

468 + else

469 + echo "::warning::Gate report missing or invalid; continuing fallback path without deterministic repeat fingerprint. Expected report: ${GATE_REPORT}"

470 + fi

471 fi

472

473 ATTEMPT=$((ATTEMPT + 1))

@@ -452,17 +479,20 @@ jobs:

479 if [ "$GATE_PASSED" = "false" ]; then

480 echo "::warning::No publishable Copilot summary was produced; falling back to GitHub Models API."

481 MODELS_PASSED="false"

482 + MODELS_GATE_REPORT="$DIAGNOSTICS_DIR/gate-github-models-attempt-0.json"

483 if python3 scripts/analyze_fallback.py \

484 --raw-json "$WEEK_FILE" \

485 --output "$OUTPUT_FILE" \

486 --current-datetime "$CURRENT_DATETIME" \

487 --press-context "$PRESS_FILE"; then

488 sanitize_agent_output "$OUTPUT_FILE"

461 - if run_quality_gate github-models; then

489 + if run_quality_gate github-models "$MODELS_GATE_REPORT"; then

490 + cp "$OUTPUT_FILE" "$DIAGNOSTICS_DIR/candidate-github-models-attempt-0.md" 2>/dev/null || true

491 MODELS_PASSED="true"

492 ANALYSIS_SOURCE="github-models"

493 ANALYSIS_MODEL="${GITHUB_MODELS_MODEL:-openai/gpt-4o}"

494 else

495 + cp "$OUTPUT_FILE" "$DIAGNOSTICS_DIR/candidate-github-models-attempt-0.md" 2>/dev/null || true

496 echo "::warning::GitHub Models output failed quality gate; falling back to data-only no-AI summary."

497 fi

498 else

@@ -477,10 +507,12 @@ jobs:

507 --press-context "$PRESS_FILE" \

508 --no-ai

509 sanitize_agent_output "$OUTPUT_FILE"

480 - if ! run_quality_gate no-ai; then

510 + NO_AI_GATE_REPORT="$DIAGNOSTICS_DIR/gate-no-ai-attempt-0.json"

511 + if ! run_quality_gate no-ai "$NO_AI_GATE_REPORT"; then

512 echo "::error::Analysis quality gate failed for Copilot CLI, GitHub Models API, and no-AI outputs."

513 exit 1

514 fi

515 + cp "$OUTPUT_FILE" "$DIAGNOSTICS_DIR/candidate-no-ai-attempt-0.md" 2>/dev/null || true

516 ANALYSIS_SOURCE="no-ai"

517 ANALYSIS_MODEL="none"

518 fi

.squad/decisions.md

+134

@@ -837,3 +837,137 @@ Safety-first protection layer for analysis reruns across staging/publish workflo

837 ### Notes

838

839 GitHub issue hierarchy represented via parent #248 with linked child issues and inline comments. All issues labeled `squad` with per-owner tracking.

840 +# Run 27055543722 — Analysis Failure Root Cause & Diagnosis [2026-06-06]

841 +

842 +## Executive Summary

843 +

844 +Workflow run 27055543722 confirmed persistent analysis failures traced to **AI output contract inconsistency**, **deterministic retry loops**, and **unavailable fallback**. The crawler was healthy; analysis failed deterministically after ~25 minutes across all Copilot attempts, falling through to no-AI fallback which the safety manifest correctly blocked.

845 +

846 +## Consolidated Root Causes (Ranked by Confidence)

847 +

848 +### 1. Prompt/Spec/Gate Schema Drift — Very High Confidence

849 +- `prompts/analyze-weekly.md` and `docs/analysis-spec.md` specify predictions as `{repo, direction, confidence}`

850 +- `scripts/analysis_gate.py` requires `predictions[].claim_type in {signal, noise, gap}`

851 +- Run 27030646485: All 3 Copilot attempts failed on `predictions[*].claim_type must be one of signal, noise, gap`

852 +- Run 27055543722: Same deterministic failure across all 3 attempts

853 +- **Immediate action:** Align contract; update gate or prompt to match

854 +

855 +### 2. Context Bloat & Undercounted Preflight — Very High Confidence

856 +- Preflight estimates: ~74k tokens (raw JSON + skills only)

857 +- Actual final ledger: ~113k tokens (preflight + template + identity/wisdom + prior summary + press context + agent wrapper)

858 +- Gap of ~39k tokens (38% undercount) dilutes model attention

859 +- Renders skills payload (~45.8 KB / 11.4k tokens) contains unrelated operational/design skills, not analysis-specific

860 +- **Immediate action:** Render exact prompt before preflight; use analysis-specific wisdom capsule

861 +

862 +### 3. Blind Deterministic Retry Loop — Very High Confidence

863 +- Three full Copilot attempts (9-11 min each) repeated identical flawed prompt

864 +- No gate-error classification or deterministic repair (e.g., timestamp/schema normalization)

865 +- Retries waste 25-29 minutes, then fall through to no-AI

866 +- **Immediate action:** Classify gate failures; repair frontmatter once; fail-fast on systematic errors

867 +

868 +### 4. Unavailable AI Fallback — High Confidence

869 +- GitHub Models configured to `openai/gpt-4o` → `403 no_access` on both runs

870 +- No fallback to accessible model; pipeline jumps directly to no-AI

871 +- **Immediate action:** Preflight model access before expensive Copilot attempts

872 +

873 +### 5. No-AI Fallback Provenance Not Enforced — High Confidence

874 +- `analysis_gate.py` checks structure only, not provenance

875 +- No-AI output (quality_score 62, self-referential language) passed structural gate

876 +- Run 27030646485: No-AI published despite existing AI article from previous week

877 +- **Immediate action:** Enforce no-AI publish-blocking in manifest (already done in #249); add to diagnostics

878 +

879 +### 6. Weak Evidence/Citation Validation — Medium-High Confidence

880 +- Gate does not validate repo-link coverage (every mentioned repo must be `[owner/repo](url)`)

881 +- Gate does not validate press citation integrity or that links resolve to retained articles

882 +- No-AI fallback uses raw descriptions without editorial curation

883 +

884 +### 7. Learned Context Overhead & Agent Wrapper — Medium-High Confidence

885 +- Workflow calls `copilot --agent squad` for publication analysis

886 +- Squad agent adds routing/delegation instructions on top of the rendered prompt

887 +- Flat skills bundle mixes unrelated squad operational skills with analysis context

888 +

889 +## Recommended Immediate Fixes

890 +

891 +### 1. Align Prediction Contract (Blocker for Next Run)

892 +- **Decision:** Require `claim_type in {signal, noise, gap}` (supports hindsight classification)

893 +- **Actions:**

894 + - Update `docs/analysis-spec.md` and `prompts/analyze-weekly.md` to specify predictions as `{repo, claim_type, direction, confidence}`

895 + - Add contract-drift test: parse docs/prompt/gate and assert matching prediction schema

896 + - Add regression fixtures for bad-date and prediction-shape errors

897 +

898 +### 2. Deterministic Frontmatter & Repair Loop

899 +- **Actions:**

900 + - Precompute and mechanically inject `date`, `week`, `year`, `repos_featured`, `stars_tracked`

901 + - Add focused repair prompt for gate failures (classifies errors; repairs timestamp/schema once)

902 + - Fail-fast if error is systematic; do not retry if only deterministic fields are wrong

903 +

904 +### 3. Exact Rendered Prompt Preflight

905 +- **Actions:**

906 + - Build `analysis-input-manifest.json` with component byte/token counts

907 + - Render exact prompt (template + raw JSON + wisdom + skills + press context + prior summary)

908 + - Compare rendered-prompt estimate to final ledger; fail or compact if gap > 10%

909 +

910 +### 4. Analysis-Specific Wisdom Capsule

911 +- **Actions:**

912 + - Replace full learned-context injection with `{{EDITORIAL_WISDOM_CAPSULE}}` (~1-2k tokens max)

913 + - Select only analysis-relevant learnings; exclude operational/design/PR workflow skills

914 + - Keep prior-week continuity notes (~500 tokens) for editorial context

915 +

916 +### 5. GitHub Models Access Preflight

917 +- **Actions:**

918 + - Add fast `models-health` check before Copilot attempts

919 + - If configured model is inaccessible, switch to known-good fallback or mark unavailable up front

920 + - Record provider/model/access status in `models-health.json` artifact

921 +

922 +### 6. No-AI as Diagnostic Only

923 +- **Actions:**

924 + - Keep no-AI fallback for diagnostics/artifact purposes

925 + - Tag as `diagnostic_no_ai_candidate` (not fallback recovery)

926 + - Ensure manifest blocks promotion unless explicit force flag is set

927 + - Do not let no-AI replace existing good AI article

928 +

929 +### 7. Direct Analyzer Invocation

930 +- **Actions:**

931 + - Stop calling `--agent squad` for publication analysis

932 + - Use plain Copilot CLI or minimal analyzer agent (Farnsworth-only)

933 + - Remove squad routing/delegation context from analysis prompt

934 +

935 +## Map/Reduce Status

936 +

937 +**Hold for now.** Map/reduce remains dry-run candidate only after immediate fixes land. The signal-type claim-ledger architecture is sound but adding a second pipeline before deterministic contract/repair/preflight are solid risks masking the same root causes across more calls.

938 +

939 +## Observability to Add

940 +

941 +- `analysis-input-manifest.json`: component token counts by segment

942 +- `analysis-attempts.jsonl`: per-attempt provider, model, duration, gate errors, failure class

943 +- `models-health.json`: endpoint, model, access status, fallback selected

944 +- `gate-report.json`: structured `analysis_gate.py` output

945 +- Persist failed candidates under `data/candidates/{week}/{run_id}/` for reproduction

946 +

947 +## Acceptance Criteria for Next Run

948 +

949 +1. ✅ Prediction contract aligned across docs, prompt, gate

950 +2. ✅ Exact rendered-prompt preflight within 10% of final ledger

951 +3. ✅ Deterministic frontmatter/schema failures do not trigger full retries

952 +4. ✅ GitHub Models access is preflighted or marked unavailable

953 +5. ✅ No-AI manifest blocks promotion unless existing AI article is gone

954 +6. ✅ Final markdown passes both structural and evidence-citation gates

955 +7. ✅ Attempt artifacts retained for root-cause analysis

956 +

957 +## Cross-Team Notes

958 +

959 +- **Farnsworth:** Comprehensive root-cause diagnosis and process proposal documented

960 +- **Bender:** Pipeline failure mechanics, contract mismatch, context bloat analysis

961 +- **Fry:** QA test fixtures, gate/spec alignment, publish protection test plan, rollout phases

962 +- **Leela:** Issue hierarchy (#248-#261), GitHub creation/updates, PR #245 safety review

963 +

964 +## Related Issues

965 +

966 +- #248 — Parent epic: protect published analysis from unsafe reruns

967 +- #249 — Publish eligibility manifest with no-AI blocking (✅ safety gate worked this run)

968 +- #255 — Strengthen publish gate beyond structural validation

969 +- #256 — Deterministic preflight compaction and fallback policy

970 +- #265 — Triaged run 27055543722 as real P0 analysis bug

971 +- #266 — New immediate P0 child for contract alignment and deterministic repair

972 +

973 +---

.squad/identity/wisdom.md

@@ -35,3 +35,4 @@ Reusable patterns and heuristics learned through work. NOT transcripts — each

35 - **Use topic counts as supporting evidence only.** `signals.top_topics` can confirm a pattern, but topic frequency alone does not prove significance.

36 - **Prefer repeated technical themes over brand repetition.** Trend calls should come from recurring problem/solution patterns, not from the same large projects staying visible.

37 - **Be explicit about uncertainty.** Honest caveats improve trust; if momentum data or historical context is thin, the analysis should say so rather than pretend precision.

38 +- **Analysis schemas must be single-sourced across prompt, spec, gate, and diagnostics.** Optional prediction registries are only safe when every generated example includes the same machine-validated fields and deterministic repairs are auditable before publish eligibility.

docs/analysis-spec.md

+1 -1

@@ -205,7 +205,7 @@ The analyzer output must begin with YAML frontmatter containing these fields.

208 -| `predictions` | array<object> | no | Optional hindsight registry. Each entry is `{repo, direction, confidence}` using `owner/repo`, `up|flat|down`, and confidence from `0` to `1`. |

208 +| `predictions` | array<object> | no | Optional hindsight registry. Each entry is `{repo, claim_type, direction, confidence}` using `owner/repo`, `signal|noise|gap`, `up|flat|down`, and confidence from `0` to `1`. |

209

210 No extra frontmatter keys should be emitted beyond this contract.

211

docs/learning-audit.md

+1 -1

@@ -217,7 +217,7 @@ Reskill → report → [NOT COMMITTED] → Lost

217 - Produces a scorecard that feeds into the next reskill

218

219 6. **Define prediction registry format (new issue):**

220 - - Frontmatter additions: `predictions: [{repo, direction, confidence}]`

220 + - Frontmatter additions: `predictions: [{repo, claim_type, direction, confidence}]`

221 - Machine-readable claims enable automated scoring

222

223 7. **Verify star snapshots are being produced:**

prompts/analyze-weekly.md

+2 -1

@@ -102,7 +102,7 @@ Be critical, selective, and opinionated.

102 11. `stars_tracked` should equal the total stars across those repos.

103 12. `top_repo` should be the repo that best anchors the editorial narrative, not automatically the most-starred repo.

104 13. `quality_score` must be an honest 0-100 self-assessment; publishable work is `>= 60`.

105 -14. If you include `predictions`, each entry must be `{repo, direction, confidence}` with `direction` in `up|flat|down` and `confidence` from `0` to `1`.

105 +14. If you include `predictions`, each entry must be `{repo, claim_type, direction, confidence}` with `claim_type` in `signal|noise|gap`, `direction` in `up|flat|down`, and `confidence` from `0` to `1`.

106 15. Include all required sections in this exact order:

107

108 ```md

@@ -166,6 +166,7 @@ quality_score: 0

166 summary: "One-sentence editorial thesis."

167 predictions:

168 - repo: owner/repo

169 + claim_type: signal

170 direction: up

171 confidence: 0.72

172 ---

scripts/analysis_gate.py

+208 -3

@@ -2,6 +2,7 @@

2 from __future__ import annotations

4 import argparse

5 +import hashlib

6 import json

7 import os

8 import re

@@ -30,6 +31,8 @@ REQUIRED_FIELDS = [

31 ]

32 OPTIONAL_FIELDS = ["predictions"]

33 PREDICTION_DIRECTIONS = {"up", "flat", "down"}

34 +PREDICTION_CLAIM_TYPES = {"signal", "noise", "gap"}

35 +PREDICTION_FIELDS = {"repo", "claim_type", "direction", "confidence"}

36 REQUIRED_HEADINGS = [

37 "## This Week's Trends",

38 "## Where Industry Meets Code",

@@ -70,6 +73,12 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:

73 parser.add_argument("--raw-json", required=True, type=Path, help="Path to the raw weekly payload.")

74 parser.add_argument("--current-datetime", required=True, help="Current run timestamp in ISO 8601 format.")

75 parser.add_argument("--source", default="unknown", help="Analysis source label for summaries.")

76 + parser.add_argument(

77 + "--repair-safe",

78 + action="store_true",

79 + help="Apply deterministic frontmatter/schema repairs before final validation.",

80 + )

81 + parser.add_argument("--report-json", type=Path, help="Write a machine-readable gate report.")

82 return parser.parse_args(argv)

@@ -205,6 +214,42 @@ def parse_frontmatter(text: str) -> dict[str, Any]:

214 return parse_frontmatter_fallback(text)

215

216

217 +def dump_frontmatter(data: dict[str, Any]) -> str:

218 + if yaml is not None:

219 + dumped = yaml.safe_dump(data, sort_keys=False, allow_unicode=True).strip()

220 + return re.sub(r"^(date): '([^']+)'$", r"\1: \2", dumped, flags=re.MULTILINE)

221 +

222 + def format_scalar(value: Any) -> str:

223 + if isinstance(value, str):

224 + return json.dumps(value)

225 + if isinstance(value, (int, float)):

226 + return str(value)

227 + raise TypeError(f"Unsupported frontmatter value for fallback dump: {value!r}")

228 +

229 + lines: list[str] = []

230 + for key, value in data.items():

231 + if isinstance(value, list):

232 + if not value:

233 + lines.append(f"{key}: []")

234 + continue

235 + lines.append(f"{key}:")

236 + for item in value:

237 + if isinstance(item, dict):

238 + if not item:

239 + lines.append(" - {}")

240 + continue

241 + item_fields = list(item.items())

242 + first_key, first_value = item_fields[0]

243 + lines.append(f" - {first_key}: {format_scalar(first_value)}")

244 + for nested_key, nested_value in item_fields[1:]:

245 + lines.append(f" {nested_key}: {format_scalar(nested_value)}")

246 + else:

247 + lines.append(f" - {format_scalar(item)}")

248 + else:

249 + lines.append(f"{key}: {format_scalar(value)}")

250 + return "\n".join(lines)

251 +

252 +

253 def extract_frontmatter(text: str) -> tuple[dict[str, Any], str]:

254 match = FRONTMATTER_PATTERN.match(text)

255 if not match:

@@ -213,6 +258,88 @@ def extract_frontmatter(text: str) -> tuple[dict[str, Any], str]:

258 return parse_frontmatter(frontmatter_text), body

259

260

261 +def render_analysis(frontmatter: dict[str, Any], body: str) -> str:

262 + return f"---\n{dump_frontmatter(frontmatter)}\n---\n{body}"

263 +

264 +

265 +def expected_repo_counts(raw_payload: dict[str, Any]) -> tuple[int, int]:

266 + repos: list[dict[str, Any]] = []

267 + for field in ("new_repos", "trending_repos"):

268 + value = raw_payload.get(field)

269 + if isinstance(value, list):

270 + repos.extend(item for item in value if isinstance(item, dict))

271 + stars = sum(star for repo in repos if isinstance((star := repo.get("stars")), int) and not isinstance(star, bool))

272 + return len(repos), stars

273 +

274 +

275 +def repair_analysis(

276 + text: str,

277 + raw_payload: dict[str, Any],

278 + current_datetime: str,

279 +) -> tuple[str, list[str]]:

280 + frontmatter, body = extract_frontmatter(text)

281 + repaired = dict(frontmatter)

282 + actions: list[str] = []

283 +

284 + expected_week = raw_payload.get("week")

285 + week_match = WEEK_PATTERN.fullmatch(expected_week) if isinstance(expected_week, str) else None

286 + if isinstance(expected_week, str) and repaired.get("week") != expected_week:

287 + repaired["week"] = expected_week

288 + actions.append(f"set week from raw payload ({expected_week})")

289 + if week_match:

290 + expected_year = int(week_match.group("year"))

291 + if repaired.get("year") != expected_year:

292 + repaired["year"] = expected_year

293 + actions.append(f"set year from raw payload week ({expected_year})")

294 + if repaired.get("date") != current_datetime:

295 + repaired["date"] = current_datetime

296 + actions.append("set date from current run timestamp")

297 +

298 + repos_featured, stars_tracked = expected_repo_counts(raw_payload)

299 + if repaired.get("repos_featured") != repos_featured:

300 + repaired["repos_featured"] = repos_featured

301 + actions.append(f"set repos_featured from raw repo counts ({repos_featured})")

302 + if repaired.get("stars_tracked") != stars_tracked:

303 + repaired["stars_tracked"] = stars_tracked

304 + actions.append(f"set stars_tracked from raw repo stars ({stars_tracked})")

305 +

306 + predictions = repaired.get("predictions")

307 + if isinstance(predictions, list):

308 + repaired_predictions = []

309 + changed_predictions = False

310 + for index, prediction in enumerate(predictions, start=1):

311 + if not isinstance(prediction, dict):

312 + repaired_predictions.append(prediction)

313 + continue

314 + repaired_prediction = dict(prediction)

315 + if "claim_type" not in repaired_prediction:

316 + for alias in ("claim", "claimType", "type", "kind"):

317 + alias_value = repaired_prediction.get(alias)

318 + if isinstance(alias_value, str) and alias_value.strip().lower() in PREDICTION_CLAIM_TYPES:

319 + repaired_prediction["claim_type"] = alias_value.strip().lower()

320 + del repaired_prediction[alias]

321 + changed_predictions = True

322 + actions.append(f"set predictions[{index}].claim_type from {alias}")

323 + break

324 + claim_type = repaired_prediction.get("claim_type")

325 + if isinstance(claim_type, str) and claim_type.strip().lower() in PREDICTION_CLAIM_TYPES and claim_type != claim_type.strip().lower():

326 + repaired_prediction["claim_type"] = claim_type.strip().lower()

327 + changed_predictions = True

328 + actions.append(f"normalized predictions[{index}].claim_type")

329 + direction = repaired_prediction.get("direction")

330 + if isinstance(direction, str) and direction.strip().lower() in PREDICTION_DIRECTIONS and direction != direction.strip().lower():

331 + repaired_prediction["direction"] = direction.strip().lower()

332 + changed_predictions = True

333 + actions.append(f"normalized predictions[{index}].direction")

334 + repaired_predictions.append(repaired_prediction)

335 + if changed_predictions:

336 + repaired["predictions"] = repaired_predictions

337 +

338 + if not actions:

339 + return text, actions

340 + return render_analysis(repaired, body), actions

341 +

342 +

343 def validate_string_field(frontmatter: dict[str, Any], field: str, errors: list[str]) -> None:

344 value = frontmatter.get(field)

345 if value is None:

@@ -272,7 +399,7 @@ def validate_predictions(frontmatter: dict[str, Any], errors: list[str]) -> None

399 confidence = prediction.get("confidence")

400 if not isinstance(repo, str) or not TOP_REPO_PATTERN.fullmatch(repo.strip()):

401 errors.append(f"predictions[{index}].repo must use owner/repo format.")

275 - if not isinstance(claim_type, str) or claim_type.strip().lower() not in {"signal", "noise", "gap"}:

402 + if not isinstance(claim_type, str) or claim_type.strip().lower() not in PREDICTION_CLAIM_TYPES:

403 errors.append(f"predictions[{index}].claim_type must be one of signal, noise, gap.")

404 if not isinstance(direction, str) or direction.strip().lower() not in PREDICTION_DIRECTIONS:

405 errors.append(f"predictions[{index}].direction must be one of up, flat, down.")

@@ -280,7 +407,7 @@ def validate_predictions(frontmatter: dict[str, Any], errors: list[str]) -> None

407 errors.append(f"predictions[{index}].confidence must be numeric.")

408 elif not 0 <= float(confidence) <= 1:

409 errors.append(f"predictions[{index}].confidence must be between 0 and 1.")

283 - extra_fields = sorted(set(prediction) - {"repo", "claim_type", "direction", "confidence"})

410 + extra_fields = sorted(set(prediction) - PREDICTION_FIELDS)

411 if extra_fields:

412 errors.append(f"predictions[{index}] has unexpected fields: {', '.join(extra_fields)}")

413

@@ -401,6 +528,59 @@ def report_success(path: Path, source: str, word_count: int, summary_path: str |

528 handle.write(f"- Word count: `{word_count}`\n")

529

530

531 +def write_gate_report(

532 + path: Path | None,

533 + *,

534 + analysis_file: Path,

535 + source: str,

536 + errors_before: list[str],

537 + errors_after: list[str],

538 + repair_actions: list[str],

539 + word_count: int,

540 +) -> None:

541 + if path is None:

542 + return

543 + path.parent.mkdir(parents=True, exist_ok=True)

544 + payload = {

545 + "analysis_file": analysis_file.as_posix(),

546 + "source": source,

547 + "passed": not errors_after,

548 + "word_count": word_count,

549 + "errors_before_repair": errors_before,

550 + "repair_actions": repair_actions,

551 + "errors_after_repair": errors_after,

552 + "failure_class": classify_gate_errors(errors_after),

553 + }

554 + path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")

555 +

556 +

557 +def classify_gate_errors(errors: list[str]) -> str:

558 + if not errors:

559 + return "passed"

560 + if all(

561 + error.startswith(("date must", "week must", "year must", "repos_featured must", "stars_tracked must"))

562 + or ".claim_type must" in error

563 + for error in errors

564 + ):

565 + return "metadata_schema"

566 + if any(error.startswith("Missing required section heading") or "body" in error for error in errors):

567 + return "content_structure"

568 + return "quality_gate"

569 +

570 +

571 +def gate_report_fingerprint(path: Path) -> str:

572 + try:

573 + report = json.loads(path.read_text(encoding="utf-8"))

574 + except (OSError, json.JSONDecodeError):

575 + return ""

576 + if not isinstance(report, dict):

577 + return ""

578 + errors = report.get("errors_after_repair") or report.get("errors_before_repair") or []

579 + if not isinstance(errors, list):

580 + return ""

581 + return hashlib.sha256(json.dumps(errors, sort_keys=True).encode("utf-8")).hexdigest()

582 +

583 +

584 def main(argv: list[str] | None = None) -> int:

585 args = parse_args(argv)

586 summary_path = os.environ.get("GITHUB_STEP_SUMMARY")

@@ -410,7 +590,32 @@ def main(argv: list[str] | None = None) -> int:

590

591 text = args.analysis_file.read_text(encoding="utf-8")

592 raw_payload = load_json(args.raw_json)

413 - errors, word_count = validate_analysis(text, raw_payload, args.current_datetime)

593 + errors_before, word_count = validate_analysis(text, raw_payload, args.current_datetime)

594 + errors = errors_before

595 + repair_actions: list[str] = []

596 + if errors and args.repair_safe:

597 + try:

598 + repaired_text, repair_actions = repair_analysis(text, raw_payload, args.current_datetime)

599 + except Exception as exc: # noqa: BLE001 - repair is best-effort; validation/reporting must continue.

600 + repair_actions = [f"repair skipped: {exc}"]

601 + else:

602 + if repair_actions and repaired_text != text:

603 + args.analysis_file.write_text(repaired_text, encoding="utf-8")

604 + text = repaired_text

605 + errors, word_count = validate_analysis(text, raw_payload, args.current_datetime)

606 + print(

607 + f"::notice::Analysis gate applied safe repairs: {', '.join(repair_actions)}",

608 + file=sys.stderr,

609 + )

610 + write_gate_report(

611 + args.report_json,

612 + analysis_file=args.analysis_file,

613 + source=args.source,

614 + errors_before=errors_before,

615 + errors_after=errors,

616 + repair_actions=repair_actions,

617 + word_count=word_count,

618 + )

619 if errors:

620 fail(errors, summary_path)

621

scripts/analyze_fallback.py

+9 -2

@@ -197,6 +197,7 @@ def extract_markdown(response_payload: dict[str, Any]) -> str:

197

198

199 RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}

200 +NON_RETRYABLE_STATUS_CLASSES = {400, 401, 403, 404}

201 MAX_RETRIES = 3

202 BASE_DELAY = 2 # seconds

203

@@ -235,11 +236,17 @@ def call_github_models(prompt: str) -> str:

236 except error.HTTPError as exc:

237 if exc.code not in RETRYABLE_STATUS_CODES or attempt == MAX_RETRIES:

238 detail = exc.read().decode("utf-8", errors="replace")

239 + retry_class = (

240 + "non-retryable"

241 + if exc.code in NON_RETRYABLE_STATUS_CLASSES or exc.code not in RETRYABLE_STATUS_CODES

242 + else "retry-exhausted"

243 + )

244 + access_hint = " GitHub Models access is unavailable for this model." if exc.code == 403 else ""

245 raise RuntimeError(

239 - f"GitHub Models API request failed ({exc.code}): {detail}"

246 + f"GitHub Models API request failed ({exc.code}, {retry_class}): {detail}{access_hint}"

247 ) from exc

248 # Determine delay: respect Retry-After header on 429

242 - retry_after = exc.headers.get("Retry-After") if exc.code == 429 else None

249 + retry_after = exc.headers.get("Retry-After") if exc.code == 429 and exc.headers is not None else None

250 if retry_after is not None:

251 try:

252 delay = float(retry_after)

tests/test_analysis_gate.py

+116

@@ -1,9 +1,18 @@

1 +import json

2 +import tempfile

3 import unittest

4 +from pathlib import Path

5 +from unittest import mock

7 import scripts.analysis_gate as analysis_gate

10 RAW_PAYLOAD = {"week": "2026-W23"}

11 +RAW_PAYLOAD_WITH_REPOS = {

12 + "week": "2026-W23",

13 + "new_repos": [{"full_name": "owner/repo", "stars": 1000}],

14 + "trending_repos": [{"full_name": "owner/repo-b", "stars": 200}],

15 +}

16 CURRENT_DATETIME = "2026-06-01T00:00:00Z"

@@ -200,6 +209,42 @@ summary: "A grounded week focused on practical tools."'''.strip()

209

210 self.assertEqual(errors, [])

211

212 + def test_repair_analysis_refuses_to_guess_legacy_prediction_claim_type(self) -> None:

213 + frontmatter = VALID_FRONTMATTER.replace(

214 + "date: 2026-06-01T00:00:00Z",

215 + "date: 2026-06-01T12:00:00Z",

216 + ) + "\npredictions:\n - repo: owner/repo\n direction: up\n confidence: 0.7"

217 +

218 + repaired_text, actions = analysis_gate.repair_analysis(

219 + make_analysis(frontmatter, make_body()),

220 + RAW_PAYLOAD_WITH_REPOS,

221 + CURRENT_DATETIME,

222 + )

223 + errors, _ = analysis_gate.validate_analysis(repaired_text, RAW_PAYLOAD_WITH_REPOS, CURRENT_DATETIME)

224 + frontmatter_after, _ = analysis_gate.extract_frontmatter(repaired_text)

225 +

226 + self.assertEqual(errors, ["predictions[1].claim_type must be one of signal, noise, gap."])

227 + self.assertIn("set date from current run timestamp", actions)

228 + self.assertNotIn("claim_type", frontmatter_after["predictions"][0])

229 + self.assertEqual(frontmatter_after["repos_featured"], 2)

230 + self.assertEqual(frontmatter_after["stars_tracked"], 1200)

231 +

232 + def test_repair_analysis_normalizes_safe_prediction_claim_alias(self) -> None:

233 + frontmatter = VALID_FRONTMATTER + "\npredictions:\n - repo: owner/repo\n claim: Signal\n direction: UP\n confidence: 0.7"

234 +

235 + repaired_text, actions = analysis_gate.repair_analysis(

236 + make_analysis(frontmatter, make_body()),

237 + RAW_PAYLOAD,

238 + CURRENT_DATETIME,

239 + )

240 + errors, _ = analysis_gate.validate_analysis(repaired_text, RAW_PAYLOAD, CURRENT_DATETIME)

241 + frontmatter_after, _ = analysis_gate.extract_frontmatter(repaired_text)

242 +

243 + self.assertEqual(errors, [])

244 + self.assertIn("set predictions[1].claim_type from claim", actions)

245 + self.assertEqual(frontmatter_after["predictions"][0]["claim_type"], "signal")

246 + self.assertNotIn("claim", frontmatter_after["predictions"][0])

247 +

248 def test_validate_analysis_rejects_invalid_prediction_registry(self) -> None:

249 frontmatter = VALID_FRONTMATTER + "\npredictions:\n - repo: bad repo\n claim_type: maybe\n direction: sideways\n confidence: 1.3\n note: nope"

250

@@ -215,6 +260,77 @@ summary: "A grounded week focused on practical tools."'''.strip()

260 self.assertIn("predictions[1].confidence must be between 0 and 1.", errors)

261 self.assertIn("predictions[1] has unexpected fields: note", errors)

262

263 + def test_prediction_contract_examples_stay_aligned_with_gate(self) -> None:

264 + repo_root = Path(__file__).resolve().parent.parent

265 + docs = (repo_root / "docs" / "analysis-spec.md").read_text(encoding="utf-8")

266 + prompt = (repo_root / "prompts" / "analyze-weekly.md").read_text(encoding="utf-8")

267 +

268 + for content in (docs, prompt):

269 + self.assertIn("{repo, claim_type, direction, confidence}", content)

270 + self.assertIn("signal|noise|gap", content)

271 + self.assertNotIn("{repo, direction, confidence}", content)

272 +

273 + def test_gate_report_fingerprint_tolerates_missing_or_invalid_reports(self) -> None:

274 + tests_root = Path(__file__).resolve().parent

275 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

276 + workspace = Path(tmpdir)

277 + missing = workspace / "missing.json"

278 + invalid = workspace / "invalid.json"

279 + report = workspace / "report.json"

280 + invalid.write_text("not json", encoding="utf-8")

281 + report.write_text(

282 + json.dumps({"errors_after_repair": ["date must match the current run timestamp."]}),

283 + encoding="utf-8",

284 + )

285 +

286 + self.assertEqual(analysis_gate.gate_report_fingerprint(missing), "")

287 + self.assertEqual(analysis_gate.gate_report_fingerprint(invalid), "")

288 + self.assertRegex(analysis_gate.gate_report_fingerprint(report), r"^[0-9a-f]{64}$")

289 +

290 + def test_fallback_frontmatter_dump_handles_empty_dict_list_items(self) -> None:

291 + original_yaml = analysis_gate.yaml

292 + try:

293 + analysis_gate.yaml = None

294 + dumped = analysis_gate.dump_frontmatter({"predictions": [{}]})

295 + finally:

296 + analysis_gate.yaml = original_yaml

297 +

298 + self.assertIn(" - {}", dumped)

299 +

300 + def test_repair_exception_still_writes_gate_report(self) -> None:

301 + tests_root = Path(__file__).resolve().parent

302 + with tempfile.TemporaryDirectory(dir=tests_root) as tmpdir:

303 + workspace = Path(tmpdir)

304 + analysis_path = workspace / "candidate.md"

305 + raw_path = workspace / "raw.json"

306 + report_path = workspace / "report.json"

307 + analysis_path.write_text(

308 + make_analysis(VALID_FRONTMATTER + "\npredictions:\n - {}", make_body()),

309 + encoding="utf-8",

310 + )

311 + raw_path.write_text('{"week": "2026-W23"}', encoding="utf-8")

312 +

313 + with mock.patch.object(analysis_gate, "repair_analysis", side_effect=RuntimeError("boom")):

314 + with self.assertRaises(SystemExit) as raised:

315 + analysis_gate.main(

316 + [

317 + "--analysis-file",

318 + str(analysis_path),

319 + "--raw-json",

320 + str(raw_path),

321 + "--current-datetime",

322 + CURRENT_DATETIME,

323 + "--repair-safe",

324 + "--report-json",

325 + str(report_path),

326 + ]

327 + )

328 +

329 + self.assertEqual(raised.exception.code, 1)

330 + report = analysis_gate.load_json(report_path)

331 + self.assertEqual(report["repair_actions"], ["repair skipped: boom"])

332 + self.assertIn("predictions[1].repo must use owner/repo format.", report["errors_after_repair"])

333 +

334

335 if __name__ == "__main__":

336 unittest.main()

tests/test_analyze_fallback.py

+39

@@ -5,6 +5,7 @@ import tempfile

5 import unittest

6 from pathlib import Path

7 from unittest import mock

8 +from urllib import error

10 import scripts.analyze_fallback as analyze_fallback

@@ -289,6 +290,44 @@ class AnalyzeFallbackTests(unittest.TestCase):

290 self.assertEqual(output_path.read_text(encoding="utf-8"), "# Summary\n")

291 self.assertEqual(urlopen_mock.call_args.kwargs["timeout"], analyze_fallback.DEFAULT_MODELS_TIMEOUT)

292

293 + def test_github_models_403_is_non_retryable_access_failure(self) -> None:

294 + forbidden = error.HTTPError(

295 + url=analyze_fallback.DEFAULT_MODELS_ENDPOINT,

296 + code=403,

297 + msg="Forbidden",

298 + hdrs={},

299 + fp=io.BytesIO(b'{"error":{"code":"no_access"}}'),

300 + )

301 +

302 + with mock.patch.dict("os.environ", {"GITHUB_TOKEN": "token"}, clear=False), mock.patch.object(

303 + analyze_fallback.request, "urlopen", side_effect=forbidden

304 + ) as urlopen_mock:

305 + with self.assertRaisesRegex(RuntimeError, "403, non-retryable.*no_access.*access is unavailable"):

306 + analyze_fallback.call_github_models("prompt")

307 +

308 + self.assertEqual(urlopen_mock.call_count, 1)

309 +

310 + def test_github_models_429_without_headers_retries_safely(self) -> None:

311 + rate_limited = error.HTTPError(

312 + url=analyze_fallback.DEFAULT_MODELS_ENDPOINT,

313 + code=429,

314 + msg="Too Many Requests",

315 + hdrs=None,

316 + fp=io.BytesIO(b'{"error":{"code":"rate_limited"}}'),

317 + )

318 + response = _FakeHTTPResponse(json.dumps({"choices": [{"message": {"content": "# Summary\n"}}]}).encode("utf-8"))

319 +

320 + with mock.patch.dict("os.environ", {"GITHUB_TOKEN": "token"}, clear=False), mock.patch.object(

321 + analyze_fallback.request, "urlopen", side_effect=[rate_limited, response]

322 + ) as urlopen_mock, mock.patch.object(analyze_fallback.random, "uniform", return_value=0), mock.patch.object(

323 + analyze_fallback.time, "sleep"

324 + ) as sleep_mock:

325 + markdown = analyze_fallback.call_github_models("prompt")

326 +

327 + self.assertEqual(markdown, "# Summary\n")

328 + self.assertEqual(urlopen_mock.call_count, 2)

329 + sleep_mock.assert_called_once_with(analyze_fallback.BASE_DELAY)

330 +

331

332 if __name__ == "__main__":

333 unittest.main()