| 1 | # Bender — History |
| 2 | |
| 3 | ## Core Context |
| 4 | - Owns crawl automation, raw data capture, and CI wiring for upstream collection. |
| 5 | - Produces structured artifacts for analysis rather than editorial output. |
| 6 | |
| 7 | ## Learnings |
| 8 | - 2026-06-07T21:42:28.011+00:00: Issue #302 Podcaster handoff now belongs after successful normal weekly article deploy: `podcaster-handoff` depends on `analyze`, `generate`, and `deploy`, gates on `run_mode == 'normal'`, and stays non-blocking for article publication. |
| 9 | - 2026-06-07T21:42:28.011+00:00: Podcaster payload contract is produced by `scripts/podcaster_handoff.py` and includes week, article URL/path, article hash when available, publish run ID, publish mode, and source artifact references while reading endpoint/key from Actions variable/secret without logging the key. |
| 10 | - 2026-06-07T21:42:28.011+00:00: Ineligible modes for Podcaster are enforced both at workflow and manifest/script boundaries: dry-run, candidate-only, restore, force-replace, no-AI, and failed publish/deploy paths must not call Podcaster. |
| 11 | - Weekly crawl output should preserve both newly discovered repos and momentum candidates so downstream stages can reason about freshness and star gains. |
| 12 | - Star-gain estimates depend on comparing current search results against the most recent prior snapshot, so snapshot compatibility matters as much as the live crawl. |
| 13 | - Rate-limited integrations should follow the shared `exponential-backoff-with-jitter` skill instead of open-coding retry behavior. |
| 14 | - Analysis execution should prefer Copilot CLI first, then fall back to GitHub Models using the same rendered prompt so output contracts stay aligned. |
| 15 | - Reviewer gates should validate the analyzer contract, not just artifact existence. |
| 16 | - New pipeline stages should follow the `ci-data-source-integration-pattern` skill: wire the script into CI immediately, document the handoff, and test the producer/consumer schema at the boundary. |
| 17 | - Hugo's `ignoreFiles` config is overridden by explicit `[module.mounts]` declarations; use `excludeFiles` on the mount definition to exclude files from mounted directories (W21 rescue, PR #167). |
| 18 | - GitHub Actions schedule events do not have an `inputs` object; use `!inputs.X` instead of `github.event.inputs.X == ''` to safely check optional manual inputs without breaking cron triggers (critical fix, PR #164). |
| 19 | - Deploy pipeline should hydrate previous-week content/data from publish branch before Hugo build to prevent main/publish divergence and preserve existing data integrity (PR #164, W21 rescue architectural fix). |
| 20 | - Fork-safe deploy secrets should default to empty in Hugo config and be injected via `HUGO_PARAMS_*` environment overrides so forks render safe defaults without inherited maintainer secrets (GA4 PR #182/#191). |
| 21 | |
| 22 | ## Round 1 (2026-06-05) |
| 23 | |
| 24 | - Resolved Copilot review thread PRRT_kwDOSgq4hM6HaRkn on PR #236 |
| 25 | - Updated docs command to `python3 -m scripts.techcrunch_crawler ...` |
| 26 | - Validated command in clean venv: `python3 -m scripts.techcrunch_crawler --help` successful |
| 27 | - commit ba2787e pushed; thread resolved |
| 28 | - PR #236 awaiting post-commit CodeQL |
| 29 | |
| 30 | - Comparing W23 crawler runs showed five in-process external RSS feeds added about 1s while the GitHub repo crawl remained the dominant 4m47s–5m58s step; keep RSS bounded in-process until source count/runtime justifies a matrix. |
| 31 | |
| 32 | ## 2026-06-05 Crawler parallelism analysis |
| 33 | |
| 34 | - Analyzed old run (26753498571) vs. new run (27026348186) to assess topology options. |
| 35 | - Key finding: GitHub repo crawl is bottleneck (~5m58s old, ~4m47s new); RSS parallelism not a factor (~1s). |
| 36 | - Topology options: A (bounded in-process), B (matrix per-source), C (hybrid staged). |
| 37 | - Recommendation: Use Option C — keep in-process now, add per-source logs and schema versioning, add validation/merge step before analysis, defer matrix to when RSS p95 > 60s or source count > 10. |
| 38 | - Acceptance criteria documented: per-source logs, schema_version, sources_requested/succeeded/failed, deterministic merge. |
| 39 | - Decision recorded in .squad/decisions.md; GitHub issue #237 created for implementation. |
| 40 | - External RSS crawlers must validate config URLs against an HTTPS host allowlist and fetch through explicit per-request timeouts; config-driven source lists are not a security boundary by themselves (PR #236). |
| 41 | - Local validation docs for scripts importing `scripts.*` modules should use `python3 -m ...` or set `PYTHONPATH=.` so repo-root imports resolve reliably (PR #236). |
| 42 | |
| 43 | ## Issue #237 canonical external-news telemetry (2026-06-05) |
| 44 | |
| 45 | - Multi-source RSS remains in-process, but downstream reliability depends on a versioned canonical artifact: include crawl window, source config checksum, per-source statuses, partial failures, dedupe count, and deterministic checksum in `*-external-news.json`. |
| 46 | - Press correlation must retain bounded source-aware citations and label category/fuzzy-only matches as weak so mirrored coverage or broad topics do not inflate strong press claims. |
| 47 | - PR #242 review follow-up: failed RSS fetch exceptions must stamp source-level attempt/timeout telemetry before re-raising, and scheduled external-news crawls should pass both `--since` and `--until` so canonical crawl windows are reproducible. |
| 48 | |
| 49 | ## 2026-06-05 Matrix crawl PRD input |
| 50 | |
| 51 | - Current observed run 27030646485 confirmed the pattern from issue #237: crawl job ~4m50s, GitHub `Run crawler` ~4m30s, external RSS/news ~1s. |
| 52 | - Matrixing RSS is an isolation feature at current scale, not a speed feature; matrixing GitHub needs a shard experiment because search quota and secondary limits are shared across jobs. |
| 53 | - Recommended PRD path is hybrid staged fan-out/fan-in: establish validated artifact contracts first, then gate RSS matrix, GitHub query matrix, and analysis map/reduce on measured thresholds. |
| 54 | - Run 27030646485 also showed analysis, not crawling, is the critical-path risk: three Copilot attempts consumed ~28m41s, failed quality gates, GitHub Models had no `openai/gpt-4o` access, and the workflow shipped via no-AI fallback with ~112.9k estimated input tokens. |
| 55 | - Issue #249 implementation: weekly analysis now writes to `data/candidates/<week>/<run_id>/` first and emits a `publish_eligibility_v1` manifest before any `data/analyzed/<week>-summary.md` promotion; promotion must fail closed on no-AI, stale source evidence, missing checksums, or failed validation. |
| 56 | |
| 57 | ## Issue #291: Copilot Pricing Refresh (2026-06-06) |
| 58 | |
| 59 | - Implemented centralized model pricing in `scripts/model_pricing.py` as single source of truth for all model costs. |
| 60 | - Added `.github/workflows/copilot-pricing-review.yml` for scheduled pricing review automation. |
| 61 | - Pricing data now decoupled from scattered configuration; future pipeline cost analysis can rely on unified pricing module. |
| 62 | - All tests pass; policy preservation: Copilot-only analysis requirement maintained. |
| 63 | - Analysis preflight now emits raw and prompt-visible repository evidence inventories with byte/token/checksum metadata; analysis gate rejects final repo links outside current raw evidence when inventory is available. |
| 64 | |
| 65 | ## Issue #287 — Analysis Gate Preflight Hardening (2026-06-06T21:23:50.664Z) |
| 66 | |
| 67 | - ✅ COMPLETE: Implemented evidence inventories (repo, article, source ledgers with integrity checksums) |
| 68 | - ✅ COMPLETE: Repository evidence validation with citation/link integrity |
| 69 | - ✅ COMPLETE: Structured gate failure summaries with classification (contract, context, timeout, fallback, other) |
| 70 | - ✅ COMPLETE: Documentation and test coverage |
| 71 | - pytest: 673 passed, 2 subtests |
| 72 | - PR #288 created; ready for merge after Fry validation |
| 73 | - Orchestration log recorded at `.squad/orchestration-log/20260606T212350Z-bender.md` |