docs: align pipeline documentation with current two-step analysis, SEO, and synthesis features (#524)
* docs: align pipeline docs with two-step analysis, SEO, and synthesis layout - README: document two-step Copilot CLI analysis (weekly-synthesis → weekly-analysis, both gpt-5.5), monthly synthesis layout with SEO titles/cross-links, and yearly narrative features - architecture.md: add two-step agent detail and additional analyze outputs - scripts/generate_rollups.py: add module docstring covering cross-links, SEO titles, structured frontmatter, and rolling report - scripts/generate_yearly_narrative.py: add module docstring covering SEO titles, meta descriptions, and monthly cross-links - scripts/month_synthesis.py: add module docstring covering frontmatter fields, body sections, summary constraints, and idempotent checksums Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add data/analyzed/ prefix to output paths in docs Addresses Copilot review comments on PR #524 — correlations and press-context filenames now include the full directory path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
Jun 17, 2026 at 09:39 UTC
35e23478c95e494546e4aef77f01ab2780d7ec38
5 files changed
+47
-9
README.md
+9
-6
@@ -32,9 +32,12 @@ JSON Markdown Hugo Pages Improvements
32
- Applies heuristic filtering (language, topic, description quality)
33
- Outputs: `data/raw/YYYY-WNN.json`, `data/raw/YYYY-WNN-external-news.json`, `data/snapshots/YYYY-WNN-stars.json`
34
35
-**Stage 2: Analyze** (Copilot CLI only)
36
-- Reads raw JSON; applies AI analysis to classify repos as signal/noise/gaps
37
-- Outputs: `data/analyzed/YYYY-WNN-summary.md` with quality score and summary sections
35
+**Stage 2: Analyze** (Copilot CLI agents, gpt-5.5)
36
+- Two-step AI analysis using dedicated Copilot CLI agents configured in `.github/agents/`:
37
+ 1. **Weekly Synthesis** (`weekly-synthesis`) — generates a compact industry narrative (~2 000 tokens) from press context and historical signals
38
+ 2. **Weekly Analysis** (`weekly-analysis`) — consumes the synthesis plus raw crawl data to produce the full editorial with signal/noise/gaps classification
39
+- Both agents use gpt-5.5 (configured in agent frontmatter) with only `read`/`write` tools
40
+- Outputs: `data/analyzed/YYYY-WNN-summary.md` with quality score, `data/analyzed/YYYY-WNN-correlations.json`, `data/analyzed/YYYY-WNN-press-context.md`
41
- Quality gate: Blocks publish if quality_score < 60 or missing required sections
42
43
**Stage 3: Generate** (`scripts/generate_content.py`)
@@ -58,7 +61,7 @@ JSON Markdown Hugo Pages Improvements
61
- **Notifications:** RSS feeds + GitHub Releases
62
- **Automation:** GitHub Actions
63
- **Deployment:** GitHub Pages
61
-- **Analysis engine:** Copilot CLI only; no GitHub Models/OpenAI analysis fallback
64
+- **Analysis engine:** Copilot CLI agents (`weekly-synthesis`, `weekly-analysis`) using gpt-5.5; no GitHub Models/OpenAI fallback
65
66
## Quick start
67
@@ -92,8 +95,8 @@ JSON Markdown Hugo Pages Improvements
95
## Content structure
96
97
- `content/weekly/YYYY/WNN.md` — immutable weekly summaries (published once, never modified)
95
-- `content/monthly/YYYY/MM.md` — monthly rollups (append-only)
96
-- `content/yearly/YYYY.md` — yearly summaries (append-only)
98
+- `content/monthly/YYYY/MM.md` — monthly rollups with synthesis narrative, SEO editorial titles (max 70 chars), cross-links to weekly/yearly pages, trend-arc sections, and structured frontmatter (themes, gaps, repos)
99
+- `content/yearly/YYYY.md` — yearly narrative summaries with SEO titles, ≤155-char meta descriptions, and cross-links to monthly pages
100
- `data/raw/YYYY-WNN.json` — GitHub crawler output (JSON object with keys: `week`, `new_repos`, `trending_repos`, `signals`, `metadata`)
101
- `data/raw/YYYY-WNN-external-news.json` — external RSS enrichment output from sources configured in `config/external_news_sources.json`
102
- `data/analyzed/YYYY-WNN-summary.md` — AI analysis with quality score
architecture.md
+5
-2
@@ -11,7 +11,7 @@ SquadScope, publicly branded as **Claracle**, is an AI-powered GitHub trend obse
11
- **Python** scripts for crawl, analysis prep, content generation, publishing, and handoff automation
12
- **GitHub Actions** for orchestration
13
- **GitHub Pages** for hosting
14
-- **GitHub Copilot CLI** for AI analysis and reskill work
14
+- **GitHub Copilot CLI** for AI analysis (two-step: `weekly-synthesis` → `weekly-analysis`, both gpt-5.5) and reskill work
15
- **No OpenAI / GitHub Models fallback** for weekly analysis
16
17
## Directory Structure
@@ -31,7 +31,10 @@ SquadScope, publicly branded as **Claracle**, is an AI-powered GitHub trend obse
31
1. **Crawl**
32
GitHub API data is written to `data/raw/YYYY-WNN.json`, and external RSS/news enrichment is written to `data/raw/YYYY-WNN-external-news.json`.
33
2. **Analyze**
34
- Copilot CLI consumes raw artifacts and produces `data/analyzed/YYYY-WNN-summary.md`.
34
+ Two-step Copilot CLI analysis using dedicated agents (both gpt-5.5):
35
+ - *Weekly Synthesis* agent produces a compact industry narrative from press/historical context
36
+ - *Weekly Analysis* agent consumes the synthesis plus raw data to generate `data/analyzed/YYYY-WNN-summary.md`
37
+ - Additional outputs: `data/analyzed/YYYY-WNN-correlations.json`, `data/analyzed/YYYY-WNN-press-context.md`
38
3. **Generate**
39
The analyzed summary is transformed into Hugo content at `content/weekly/YYYY/WNN.md`.
40
4. **Deploy**
scripts/generate_rollups.py
+12
@@ -1,4 +1,16 @@
1
#!/usr/bin/env python3
2
+"""Generate append-only monthly and yearly rollup pages from weekly analyses.
3
+
4
+Monthly pages include:
5
+- Month Synthesis section with cross-links to weekly and yearly pages
6
+- SEO-friendly editorial titles (max 70 chars) derived from theme trajectories
7
+- Structured frontmatter: summary, themes, persistent/accelerating/weakening themes,
8
+ key_gaps, top_repos, weeks_covered
9
+- Per-week entries for overview, top repos, trends, and key takeaways
10
+
11
+Yearly pages are delegated to generate_yearly_narrative.build_yearly_narrative_pages().
12
+A rolling 4-week context report can also be generated with --rolling.
13
+"""
14
from __future__ import annotations
15
16
import argparse
scripts/generate_yearly_narrative.py
+10
@@ -1,4 +1,14 @@
1
#!/usr/bin/env python3
2
+"""Generate yearly narrative pages from monthly rollup content.
3
+
4
+Reads monthly pages (content/monthly/YYYY/MM.md), detects trend-family arcs
5
+across months, and synthesizes a cohesive year-in-review narrative. Output
6
+includes:
7
+- SEO-friendly editorial titles (max 70 chars) driven by detected arcs
8
+- Meta description summaries (≤155 chars) extracted from narrative opening
9
+- Cross-links to each contributing monthly report
10
+- Structured frontmatter: months_covered, format, summary, categories
11
+"""
12
from __future__ import annotations
13
14
import argparse
scripts/month_synthesis.py
+11
-1
@@ -1,4 +1,14 @@
1
-from __future__ import annotations
1
+"""Month synthesis: deterministic monthly narrative generator.
2
+
3
+Consumes weekly analysis summaries for a given month and produces a synthesis
4
+artifact with:
5
+- Structured frontmatter: title, summary, themes, persistent/accelerating/weakening
6
+ themes, key_gaps, top_repos, source_checksum, weeks_covered
7
+- Body sections: Month Synthesis (narrative), Weekly Reports (cross-linked list),
8
+ Trend Arc (persistent/accelerating/weakened theme bullets), Prediction Review
9
+- Summary field (≤28 words) suitable for SEO meta descriptions
10
+- Source checksum for idempotent regeneration (skips if input unchanged)
11
+"""
12
13
import hashlib
14
import json