| 1 | # Pipeline Validation Checklist |
| 2 | |
| 3 | This checklist validates the automated weekly workflow in `.github/workflows/crawl-and-publish.yml`. |
| 4 | |
| 5 | ## Trigger and scheduling |
| 6 | |
| 7 | - [x] `schedule` is enabled in the workflow. |
| 8 | - [x] Cron is `53 11 * * 0`, which targets Sundays at 11:53 UTC. |
| 9 | - [x] The documented expectation is **best effort**: GitHub-hosted scheduled workflows can start late on shared runners, so validation checks the configured cron plus the mitigation path instead of assuming an exact start minute. |
| 10 | - [x] `workflow_dispatch` is enabled for manual runs from the Actions tab or `gh workflow run crawl-and-publish.yml`. |
| 11 | - [x] `concurrency.group` is `weekly-crawl` with `cancel-in-progress: false`, so a second run waits instead of overlapping the active run. |
| 12 | |
| 13 | ## Secrets and tokens |
| 14 | |
| 15 | Required secrets/tokens: |
| 16 | |
| 17 | - `COPILOT_GH_TOKEN` — fine-grained PAT used as `COPILOT_GITHUB_TOKEN` for Copilot CLI analysis. |
| 18 | - `GITHUB_TOKEN` — built-in workflow token used for crawling, artifact downloads, commits, token-renewal issue creation, and Pages deployment. |
| 19 | - Optional Podcaster handoff: Actions variable `PODCASTER_ENDPOINT` and Actions secret `PODCASTER_API_KEY`. |
| 20 | |
| 21 | ## Stage-by-stage validation |
| 22 | |
| 23 | ### 1. Crawl |
| 24 | |
| 25 | **Job:** `crawl` |
| 26 | |
| 27 | **Inputs** |
| 28 | - GitHub API access via `GITHUB_TOKEN` |
| 29 | - Restored `crawl-cache` artifact from the latest successful workflow run when available |
| 30 | |
| 31 | **Outputs** |
| 32 | - `data/raw/YYYY-WNN.json` |
| 33 | - `data/raw/YYYY-WNN-external-news.json` |
| 34 | - `data/snapshots/YYYY-WNN-stars.json` |
| 35 | - `raw-data` artifact |
| 36 | - Immutable raw evidence at `publish:data/raw-store/YYYY-WNN/<source_run_id>/` |
| 37 | - `crawl-snapshots` artifact |
| 38 | - `crawl-cache` artifact |
| 39 | - Commit to `main` for `data/raw/` and `data/snapshots/` |
| 40 | |
| 41 | **Success criteria** |
| 42 | - Raw payload passes `scripts.crawl.validate_payload()` |
| 43 | - External RSS payload is written for the same ISO week from `config/external_news_sources.json` |
| 44 | - `data/raw/YYYY-WNN-external-news.json` uses canonical `schema_version: 2`, includes `crawl_window`, `source_config_checksum`, `sources_requested/succeeded/failed`, per-source status rows, `dedupe_count`, and `artifact_checksum` |
| 45 | - Optional per-source RSS failures are warnings with a valid partial artifact; malformed config/schema/checksum errors fail the crawl step |
| 46 | - Snapshot file is written for the same ISO week |
| 47 | - Cache artifact uploads even on partial failures |
| 48 | - `raw-data` has explicit 90-day retention for transport/emergency recovery only |
| 49 | - The publish-branch raw store refuses an existing week/run destination and records |
| 50 | source run/artifact identity plus per-file SHA-256 hashes |
| 51 | - Job permissions include `actions: read` and `contents: write` at workflow level for cache restore and commits |
| 52 | |
| 53 | ### 2. Analyze |
| 54 | |
| 55 | **Job:** `analyze` |
| 56 | |
| 57 | **Inputs** |
| 58 | - `raw-data` artifact downloaded into `data/raw/` |
| 59 | - `COPILOT_GH_TOKEN` for Copilot CLI primary path |
| 60 | - `GITHUB_TOKEN` for diagnostics, commits, and Copilot-token renewal issue creation |
| 61 | |
| 62 | **Outputs** |
| 63 | - `data/analyzed/YYYY-WNN-summary.md` |
| 64 | - `data/analyzed/YYYY-WNN-correlations.json` |
| 65 | - `data/analyzed/YYYY-WNN-press-context.md` |
| 66 | - `analyzed-data` artifact |
| 67 | - Commit to `main` for `data/analyzed/` |
| 68 | - Job outputs: `week`, `summary_file`, `current_datetime` |
| 69 | |
| 70 | **Success criteria** |
| 71 | - Current raw file week matches the run week |
| 72 | - Restore mode requires `source_run_id`, hydrates the matching immutable raw-store |
| 73 | directory, and verifies all hashes before analysis accepts the inputs |
| 74 | - Correlation and press-context steps consume compact external-news data with legacy `YYYY-WNN-techcrunch.json` fallback |
| 75 | - Press context preserves source names, article URLs/titles/dates, strong-vs-weak labels, and partial-source caveats while staying under the ~8k token budget |
| 76 | - Analysis preflight writes `analysis-preflight.json` with deterministic prompt component byte/token/checksum metadata and raw/prompt evidence inventories before Copilot is invoked. |
| 77 | - Copilot CLI output is staged under `data/candidates/YYYY-WNN/<run-id>/`; if Copilot cannot produce publishable analysis, a no-AI candidate is kept as a rejected artifact and the run fails closed before promotion. |
| 78 | - `scripts/analysis_gate.py` passes before publish continues, rejects repo links outside the current raw evidence inventory where available, and emits structured failure summaries for deterministic retry/preserve decisions. |
| 79 | - `scripts/publish_manifest.py` records promote-vs-preserve decisions against any existing good published summary. |
| 80 | - Job permissions include `actions: read`, `contents: write`, and `issues: write` |
| 81 | |
| 82 | ### 3. Generate |
| 83 | |
| 84 | **Job:** `generate` |
| 85 | |
| 86 | **Inputs** |
| 87 | - `analyzed-data` artifact downloaded into `data/analyzed/` |
| 88 | - `needs.analyze.outputs.summary_file` |
| 89 | |
| 90 | **Outputs** |
| 91 | - `content/weekly/YYYY/WNN.md` |
| 92 | - `generated-content` artifact |
| 93 | - Commit to `main` for `content/weekly/` |
| 94 | - Job output: `page_path` |
| 95 | |
| 96 | **Success criteria** |
| 97 | - `scripts/generate_content.py` converts the weekly summary into Hugo content |
| 98 | - Generated frontmatter keeps publishable fields and drops analysis-only fields like `quality_score` |
| 99 | - Job permissions include `actions: read` and `contents: write` |
| 100 | |
| 101 | ### 4. Deploy |
| 102 | |
| 103 | **Job:** `deploy` |
| 104 | |
| 105 | **Inputs** |
| 106 | - Repository checkout with submodules |
| 107 | - `raw-data`, `analyzed-data`, and `generated-content` artifacts restored into the workspace |
| 108 | - Hugo extended `0.161.1` |
| 109 | |
| 110 | **Outputs** |
| 111 | - `public/` site build |
| 112 | - GitHub Pages artifact uploaded with `actions/upload-pages-artifact` |
| 113 | - Published Pages deployment via `actions/deploy-pages` |
| 114 | |
| 115 | **Success criteria** |
| 116 | - Hugo build succeeds with the pinned version |
| 117 | - Pages artifact is uploaded from `./public` |
| 118 | - Deployment publishes from the same workflow run that generated the content |
| 119 | - Job permissions include `actions: read`, `contents: read`, `pages: write`, and `id-token: write` |
| 120 | |
| 121 | ## Artifact handoff audit |
| 122 | |
| 123 | - `crawl` → `analyze`: `raw-data` |
| 124 | - `crawl` → later runs: `crawl-cache` |
| 125 | - `analyze` → `generate`: `analyzed-data` |
| 126 | - `generate` → `deploy`: `generated-content` |
| 127 | - Podcaster handoff: triggered from `sync-publish-to-main` only **after** the weekly article is merged into `main`. The handoff runs `scripts/podcaster_handoff.py --require-merged`, which fails closed unless the merged article exists and its sha256 matches the manifest `candidate.content_sha256`. This prevents the prior race where `deploy` (built from artifacts, pre-merge) could trigger the podcaster before the article was merged, producing stub episodes (e.g. W27). |
| 128 | - `crawl` and `analyze` also feed `deploy` so the final build uses the same run's data artifacts |
| 129 | |
| 130 | ## Manual validation flow |
| 131 | |
| 132 | ### Trigger from GitHub |
| 133 | |
| 134 | - Actions tab → **Crawl and publish weekly data** → **Run workflow** |
| 135 | - CLI: `gh workflow run crawl-and-publish.yml` |
| 136 | - External scheduler / automation host: call the same `workflow_dispatch` endpoint or CLI command if punctual timing matters more than GitHub-hosted `schedule` latency |
| 137 | |
| 138 | ### Trigger locally |
| 139 | |
| 140 | - Crawl: `python3 scripts/crawl.py --as-of YYYY-MM-DD` |
| 141 | - External news crawl: `python3 -m scripts.techcrunch_crawler --sources config/external_news_sources.json --output data/raw/YYYY-WNN-external-news.json --since YYYY-MM-DD --until YYYY-MM-DD` |
| 142 | - Correlate press: `python3 -m scripts.correlate --raw data/raw/YYYY-WNN.json --techcrunch data/raw/YYYY-WNN-external-news.json --output data/analyzed/YYYY-WNN-correlations.json` |
| 143 | - Render press context: `python3 -m scripts.render_press_context --week YYYY-WNN` |
| 144 | - Render analysis prompt/diagnostic no-AI output: `python3 scripts/analyze_fallback.py --raw-json data/raw/YYYY-WNN.json --output data/analyzed/YYYY-WNN-summary.md --current-datetime YYYY-MM-DDTHH:MM:SSZ --print-prompt` |
| 145 | - Gate: `python3 scripts/analysis_gate.py --analysis-file data/analyzed/YYYY-WNN-summary.md --raw-json data/raw/YYYY-WNN.json --current-datetime YYYY-MM-DDTHH:MM:SSZ` |
| 146 | - Generate: `python3 scripts/generate_content.py data/analyzed/YYYY-WNN-summary.md` |
| 147 | - Deploy build check: `hugo --minify` |
| 148 | |
| 149 | ## Known limitations and workarounds |
| 150 | |
| 151 | - Copilot CLI in CI depends on `COPILOT_GH_TOKEN`; when token/auth fails, the workflow fails immediately and creates or updates an issue assigned to `@jmservera` to renew the token. There is no GitHub Models/OpenAI fallback for weekly analysis. |
| 152 | - Weekly momentum quality is only as good as the historical star snapshots; first runs and sparse history can make `stars_gained` incomplete. |
| 153 | - Hugo must be `0.146.0+`; the workflow pins `0.161.1` because older runner binaries fail with the current theme. |
| 154 | - The scheduled workflow now deploys Pages directly. `deploy-site.yml` skips bot-authored pushes so the scheduled run does not trigger a duplicate Pages deployment. |
| 155 | - GitHub-hosted `schedule` is best-effort. This repo has observed multi-hour delays on scheduled starts, so the supported mitigation path is manual `gh workflow run`, then external scheduler -> `workflow_dispatch`, with self-hosted runners reserved as optional future work. |