| 1 | [](https://github.com/jmservera/SquadScope/actions/workflows/crawl-and-publish.yml) |
| 2 | [](https://github.com/jmservera/SquadScope/actions/workflows/deploy-site.yml) |
| 3 | # Claracle (SquadScope) |
| 4 | |
| 5 | Claracle is an automated AI-powered site that publishes weekly, monthly, and yearly summaries of tech trends sourced from GitHub. The system crawls trending and newly-created repositories, applies AI analysis to identify signal vs. noise, and publishes curated insights with zero manual intervention. |
| 6 | |
| 7 | ## What is Claracle? |
| 8 | |
| 9 | Claracle solves the information overload problem in open-source development. Each week, millions of repositories are created or gain stars. Claracle: |
| 10 | |
| 11 | 1. **Crawls** GitHub API for new and trending repositories in a given week |
| 12 | 2. **Analyzes** the data with Copilot to identify what's genuinely important vs. hype |
| 13 | 3. **Generates** human-readable weekly summaries with signal, noise, and gaps analysis |
| 14 | 4. **Publishes** to GitHub Pages with RSS feeds for consumption |
| 15 | 5. **Reskills** every 5 runs to improve its own analysis quality |
| 16 | |
| 17 | Result: Curated tech trend insights delivered automatically every week. |
| 18 | |
| 19 | ## Architecture |
| 20 | |
| 21 | Claracle uses a **5-stage pipeline** executed entirely in GitHub Actions: |
| 22 | |
| 23 | ``` |
| 24 | Crawl → Analyze → Generate → Deploy → Reskill |
| 25 | ↓ ↓ ↓ ↓ ↓ |
| 26 | JSON Markdown Hugo Pages Improvements |
| 27 | ``` |
| 28 | |
| 29 | **Stage 1: Crawl** (`scripts/crawl.py`, `scripts/techcrunch_crawler.py`) |
| 30 | - Queries GitHub API for repos created/trending in the current week |
| 31 | - Fetches configured external RSS feeds from `config/external_news_sources.json` in parallel as an enrichment signal |
| 32 | - Applies heuristic filtering (language, topic, description quality) |
| 33 | - Outputs: `data/raw/YYYY-WNN.json`, `data/raw/YYYY-WNN-external-news.json`, `data/snapshots/YYYY-WNN-stars.json` |
| 34 | |
| 35 | **Stage 2: Analyze** (Copilot CLI agents, gpt-5.5) |
| 36 | - Two-step AI analysis using dedicated Copilot CLI agents configured in `.github/agents/`: |
| 37 | 1. **Weekly Synthesis** (`weekly-synthesis`) — generates a compact industry narrative (~2 000 tokens) from press context and historical signals |
| 38 | 2. **Weekly Analysis** (`weekly-analysis`) — consumes the synthesis plus raw crawl data to produce the full editorial with signal/noise/gaps classification |
| 39 | - Both agents use gpt-5.5 (configured in agent frontmatter) with only `read`/`write` tools |
| 40 | - Outputs: `data/analyzed/YYYY-WNN-summary.md` with quality score, `data/analyzed/YYYY-WNN-correlations.json`, `data/analyzed/YYYY-WNN-press-context.md` |
| 41 | - Quality gate: Blocks publish if quality_score < 60 or missing required sections |
| 42 | |
| 43 | **Stage 3: Generate** (`scripts/generate_content.py`) |
| 44 | - Converts analyzed Markdown into Hugo content structure |
| 45 | - Outputs: `content/weekly/YYYY/WNN.md` ready for Hugo build |
| 46 | |
| 47 | **Stage 4: Deploy** (Hugo + GitHub Pages) |
| 48 | - Builds static site from Hugo and publishes to GitHub Pages |
| 49 | - Outputs: Live website + RSS feed at `https://www.claracle.com/index.xml` |
| 50 | |
| 51 | **Stage 5: Reskill** (every 5th run) |
| 52 | - Every 5th run, Copilot reviews squad history and recent analysis outputs |
| 53 | - Writes observations and improvement recommendations to `.squad/reskill/YYYY-WNN.md` |
| 54 | - Optional: Can trigger PR with proposed prompt refinements (not auto-merged) |
| 55 | |
| 56 | ## Theme and stack |
| 57 | |
| 58 | - **Static site generator:** Hugo (extended, v0.146.0+) |
| 59 | - **Theme:** [PaperMod](https://github.com/adityatelange/hugo-PaperMod) |
| 60 | - **Search:** Pagefind (static, client-side) |
| 61 | - **Notifications:** RSS feeds + GitHub Releases |
| 62 | - **Automation:** GitHub Actions |
| 63 | - **Deployment:** GitHub Pages |
| 64 | - **Analysis engine:** Copilot CLI agents (`weekly-synthesis`, `weekly-analysis`) using gpt-5.5; no GitHub Models/OpenAI fallback |
| 65 | |
| 66 | ## Quick start |
| 67 | |
| 68 | ### For end users |
| 69 | |
| 70 | 1. **Visit the site:** [Claracle](https://www.claracle.com/): Browse weekly, monthly, yearly summaries |
| 71 | 2. **Subscribe to RSS:** Add `https://www.claracle.com/index.xml` to your reader |
| 72 | 3. **Check GitHub Releases:** New summaries also posted as releases |
| 73 | |
| 74 | ### For operators (see `docs/operator-guide.md` for full setup) |
| 75 | |
| 76 | 1. Fork this repository |
| 77 | 2. Configure `COPILOT_GH_TOKEN` secret in your repo (fine-grained PAT with Copilot Requests permission) |
| 78 | 3. Enable GitHub Pages (Actions source) |
| 79 | 4. Test manual run: `gh workflow run crawl-and-publish.yml` |
| 80 | 5. Monitor the first automated run |
| 81 | |
| 82 | ⚠️ **First-time operators:** Start with `docs/rollout-checklist.md` to ensure all prerequisites are in place. |
| 83 | |
| 84 | ## Local development |
| 85 | |
| 86 | 1. **Install Hugo:** `brew install hugo` (macOS) or download from [hugo releases](https://github.com/gohugoio/hugo/releases) (v0.146.0 or newer) |
| 87 | 2. **Clone with submodules:** |
| 88 | ```bash |
| 89 | git clone --recurse-submodules https://github.com/jmservera/SquadScope.git |
| 90 | git submodule update --init --recursive |
| 91 | ``` |
| 92 | 3. **Start dev server:** `hugo server` (default: http://localhost:1313) |
| 93 | 4. **Create production build:** `hugo --minify` (output: `public/`) |
| 94 | |
| 95 | ## Content structure |
| 96 | |
| 97 | - `content/weekly/YYYY/WNN.md` — immutable weekly summaries (published once, never modified) |
| 98 | - `content/monthly/YYYY/MM.md` — monthly rollups with synthesis narrative, SEO editorial titles (max 70 chars), cross-links to weekly/yearly pages, trend-arc sections, and structured frontmatter (themes, gaps, repos) |
| 99 | - `content/yearly/YYYY.md` — yearly narrative summaries with SEO titles, ≤155-char meta descriptions, and cross-links to monthly pages |
| 100 | - `data/raw/YYYY-WNN.json` — GitHub crawler output (JSON object with keys: `week`, `new_repos`, `trending_repos`, `signals`, `metadata`) |
| 101 | - `data/raw/YYYY-WNN-external-news.json` — external RSS enrichment output from sources configured in `config/external_news_sources.json` |
| 102 | - `data/analyzed/YYYY-WNN-summary.md` — AI analysis with quality score |
| 103 | - `data/snapshots/YYYY-WNN-stars.json` — star count snapshots for trending analysis |
| 104 | |
| 105 | ## Automated weekly pipeline |
| 106 | |
| 107 | `.github/workflows/crawl-and-publish.yml` runs the full weekly automation on a best-effort Sunday schedule at 11:53 UTC: |
| 108 | |
| 109 | 1. **Crawl:** GitHub API → `data/raw/YYYY-WNN.json`; external RSS feeds → `data/raw/YYYY-WNN-external-news.json` |
| 110 | 2. **Analyze:** Copilot → `data/analyzed/YYYY-WNN-summary.md` |
| 111 | 3. **Quality gate:** Validates quality_score ≥ 60; blocks publish if failed |
| 112 | 4. **Generate:** Markdown → `content/weekly/YYYY/WNN.md` |
| 113 | 5. **Deploy:** Hugo build → GitHub Pages |
| 114 | 6. **Reskill:** Every 5th run, review and improve squad state |
| 115 | |
| 116 | ### Schedule and manual runs |
| 117 | |
| 118 | - **Automated schedule:** Sunday 11:53 UTC (`53 11 * * 0`) on GitHub-hosted runners |
| 119 | - **Timing expectation:** GitHub Actions `schedule` is best-effort on shared runners; this repo has observed multi-hour delays on scheduled starts |
| 120 | - **Manual trigger:** `gh workflow run crawl-and-publish.yml` or GitHub Actions UI |
| 121 | - **If punctuality matters:** trigger the existing `workflow_dispatch` from an external scheduler; see [`docs/operator-guide.md#schedule-latency-and-mitigation-ladder`](docs/operator-guide.md#schedule-latency-and-mitigation-ladder) |
| 122 | |
| 123 | ### Required secrets |
| 124 | |
| 125 | - `COPILOT_GH_TOKEN` — Fine-grained PAT with **Account → Copilot Requests** permission for Copilot CLI analysis |
| 126 | - `GITHUB_TOKEN` — Built-in; used for crawling, commits, Pages deployment, and issue/notification automation |
| 127 | - `PODCASTER_API_KEY` — Optional; used with the `PODCASTER_ENDPOINT` Actions variable for the post-publish Podcaster handoff. The value must never be logged or committed. |
| 128 | |
| 129 | ## Crawler notes |
| 130 | |
| 131 | - `signals.top_topics` de-duplicates repositories by `full_name` across new and trending buckets |
| 132 | - `data/snapshots/YYYY-WNN-stars.json` preserves the broader pre-filter candidate set for consistent week-over-week `stars_gained` comparisons |
| 133 | - Live runs use open-ended `created:>` / `pushed:>` GitHub search filters |
| 134 | - `--as-of` mode switches to bounded date ranges for deterministic historical backfills |
| 135 | |
| 136 | ## Deployment |
| 137 | |
| 138 | - **Standard deploys:** Direct pushes to `main` trigger `.github/workflows/deploy-site.yml` |
| 139 | - **Weekly automation:** `crawl-and-publish.yml` handles full pipeline and deploys Pages |
| 140 | - **Deduplication:** Deploy workflow skips bot-authored pushes to avoid duplicate Pages runs |
| 141 | |
| 142 | ## Documentation |
| 143 | |
| 144 | - **`docs/operator-guide.md`** — Complete setup, configuration, and troubleshooting guide for new operators |
| 145 | - **`docs/rollout-checklist.md`** — Step-by-step verification checklist for first-time deployments |
| 146 | - **`docs/pipeline-validation.md`** — Stage checklist, artifact handoffs, success criteria, and known limitations |
| 147 | - **`docs/analysis-spec.md`** — Detailed specification for analysis output format and quality gate criteria |
| 148 | - **`.squad/decisions.md`** — Architectural decisions and decision history |