| 1 | # Weekly Analysis Specification |
| 2 | |
| 3 | This document defines the analyzer contract between `data/raw/YYYY-WNN.json` and `data/analyzed/YYYY-WNN-summary.md`. |
| 4 | |
| 5 | ## Purpose |
| 6 | |
| 7 | The analyzer turns a weekly GitHub crawl into a structured editorial summary that is: |
| 8 | |
| 9 | - consistent enough for CI automation, |
| 10 | - opinionated enough to be worth reading, |
| 11 | - strict enough for downstream site generation, and |
| 12 | - traceable enough for reviewer-gate validation. |
| 13 | |
| 14 | The analyzer is a read-only consumer of `data/raw/`. It may interpret, rank, and summarize the crawl, but it must not rewrite the input artifact. |
| 15 | |
| 16 | ## Editorial Lens |
| 17 | |
| 18 | SquadScope analysis uses a three-part editorial lens: |
| 19 | |
| 20 | - **Signal** — projects or shifts that matter because they solve real problems, represent credible technical movement, or reveal durable ecosystem direction. |
| 21 | - **Noise** — activity that is loud but weak: marketing-heavy launches, copycat agents, exploit/bypass churn, or trend-chasing with little substance. |
| 22 | - **Gaps** — meaningful absences: categories, problem spaces, or technical needs that should be showing more energy but are not. |
| 23 | |
| 24 | The reader-facing markdown keeps the five approved weekly sections, but the analysis itself must explicitly surface **Signal**, **Noise**, and **Gaps** as labeled subsections. |
| 25 | |
| 26 | ## Input Contract |
| 27 | |
| 28 | ### File naming |
| 29 | |
| 30 | - **Location:** `data/raw/` |
| 31 | - **Filename:** `YYYY-WNN.json` |
| 32 | - **Example:** `data/raw/2026-W21.json` |
| 33 | |
| 34 | ### Analyzer read scope |
| 35 | |
| 36 | The analyzer reads these fields: |
| 37 | |
| 38 | - `week` |
| 39 | - `crawled_at` |
| 40 | - `new_repos[]` |
| 41 | - `trending_repos[]` |
| 42 | - `signals.top_topics[]` |
| 43 | - `metadata.partial_failures` *(optional diagnostic input; emitted by `scripts/crawl.py` today, but analyzers must tolerate absence)* |
| 44 | - `metadata.filter_summary` *(optional diagnostic input; emitted by `scripts/crawl.py` today, but analyzers must tolerate absence)* |
| 45 | - `metadata.snapshot_path` *(optional diagnostic input; emitted by `scripts/crawl.py` today, but analyzers must tolerate absence)* |
| 46 | |
| 47 | Unknown fields must be ignored. The current crawler emits these diagnostic metadata fields in its own artifacts, but analyzers must not fail when they are missing from backfilled or forward-compatible payloads. |
| 48 | |
| 49 | ### JSON schema |
| 50 | |
| 51 | ```json |
| 52 | { |
| 53 | "$schema": "https://json-schema.org/draft/2020-12/schema", |
| 54 | "title": "SquadScope Weekly Crawl Payload", |
| 55 | "type": "object", |
| 56 | "additionalProperties": true, |
| 57 | "required": [ |
| 58 | "week", |
| 59 | "crawled_at", |
| 60 | "new_repos", |
| 61 | "trending_repos", |
| 62 | "signals", |
| 63 | "metadata" |
| 64 | ], |
| 65 | "properties": { |
| 66 | "week": { |
| 67 | "type": "string", |
| 68 | "pattern": "^[0-9]{4}-W[0-9]{2}$" |
| 69 | }, |
| 70 | "crawled_at": { |
| 71 | "type": "string", |
| 72 | "format": "date-time" |
| 73 | }, |
| 74 | "new_repos": { |
| 75 | "type": "array", |
| 76 | "items": { "$ref": "#/$defs/repo" } |
| 77 | }, |
| 78 | "trending_repos": { |
| 79 | "type": "array", |
| 80 | "items": { "$ref": "#/$defs/trendingRepo" } |
| 81 | }, |
| 82 | "signals": { |
| 83 | "type": "object", |
| 84 | "additionalProperties": true, |
| 85 | "required": ["top_topics"], |
| 86 | "properties": { |
| 87 | "top_topics": { |
| 88 | "type": "array", |
| 89 | "items": { |
| 90 | "type": "object", |
| 91 | "additionalProperties": false, |
| 92 | "required": ["topic", "count"], |
| 93 | "properties": { |
| 94 | "topic": { "type": "string" }, |
| 95 | "count": { "type": "integer", "minimum": 0 } |
| 96 | } |
| 97 | } |
| 98 | } |
| 99 | } |
| 100 | }, |
| 101 | "metadata": { |
| 102 | "type": "object", |
| 103 | "additionalProperties": true, |
| 104 | "properties": { |
| 105 | "api_calls_used": { "type": "integer", "minimum": 0 }, |
| 106 | "cache_hits": { "type": "integer", "minimum": 0 }, |
| 107 | "stale_cache_hits": { "type": "integer", "minimum": 0 }, |
| 108 | "rate_limit_limit": { "type": ["integer", "null"], "minimum": 0 }, |
| 109 | "rate_limit_remaining": { "type": ["integer", "null"], "minimum": 0 }, |
| 110 | "rate_limit_reset": { "type": ["integer", "null"], "minimum": 0 }, |
| 111 | "rate_limit_resource": { "type": ["string", "null"] }, |
| 112 | "partial_failures": { |
| 113 | "type": "array", |
| 114 | "items": { "type": "string" } |
| 115 | }, |
| 116 | "snapshot_path": { "type": "string" }, |
| 117 | "filter_summary": { |
| 118 | "type": "object", |
| 119 | "additionalProperties": { |
| 120 | "type": "object", |
| 121 | "additionalProperties": { "type": "integer", "minimum": 0 } |
| 122 | } |
| 123 | } |
| 124 | } |
| 125 | } |
| 126 | }, |
| 127 | "$defs": { |
| 128 | "repo": { |
| 129 | "type": "object", |
| 130 | "additionalProperties": true, |
| 131 | "required": [ |
| 132 | "name", |
| 133 | "owner", |
| 134 | "full_name", |
| 135 | "description", |
| 136 | "language", |
| 137 | "stars", |
| 138 | "forks", |
| 139 | "created_at", |
| 140 | "topics", |
| 141 | "license", |
| 142 | "url" |
| 143 | ], |
| 144 | "properties": { |
| 145 | "name": { "type": "string" }, |
| 146 | "owner": { "type": "string" }, |
| 147 | "full_name": { "type": "string" }, |
| 148 | "description": { "type": ["string", "null"] }, |
| 149 | "language": { "type": ["string", "null"] }, |
| 150 | "stars": { "type": "integer", "minimum": 0 }, |
| 151 | "forks": { "type": "integer", "minimum": 0 }, |
| 152 | "created_at": { "type": "string", "format": "date-time" }, |
| 153 | "topics": { |
| 154 | "type": "array", |
| 155 | "items": { "type": "string" } |
| 156 | }, |
| 157 | "license": { "type": ["string", "null"] }, |
| 158 | "url": { "type": "string", "format": "uri" } |
| 159 | } |
| 160 | }, |
| 161 | "trendingRepo": { |
| 162 | "allOf": [ |
| 163 | { "$ref": "#/$defs/repo" }, |
| 164 | { |
| 165 | "type": "object", |
| 166 | "properties": { |
| 167 | "stars_gained": { "type": ["integer", "null"], "minimum": 0 } |
| 168 | } |
| 169 | } |
| 170 | ] |
| 171 | } |
| 172 | } |
| 173 | } |
| 174 | ``` |
| 175 | |
| 176 | ### Input interpretation rules |
| 177 | |
| 178 | 1. **New repos** are candidates for editorial novelty. |
| 179 | 2. **Trending repos** are candidates for momentum, but if `stars_gained` is absent or null, the analyzer must say that momentum is not yet fully measurable. |
| 180 | 3. **Top topics** are directional evidence, not conclusions by themselves. |
| 181 | 4. **Metadata diagnostics** can justify caveats about crawl quality, filtering, or missing baselines, but must not dominate the summary. |
| 182 | |
| 183 | ## Output Contract |
| 184 | |
| 185 | ### File naming |
| 186 | |
| 187 | - **Location:** `data/analyzed/` |
| 188 | - **Filename:** `YYYY-WNN-summary.md` |
| 189 | - **Example:** `data/analyzed/2026-W21-summary.md` |
| 190 | |
| 191 | ### Required frontmatter |
| 192 | |
| 193 | The analyzer output must begin with YAML frontmatter containing these fields. |
| 194 | |
| 195 | | Field | Type | Required | Meaning | |
| 196 | |---|---|---:|---| |
| 197 | | `title` | string | yes | Reader-facing weekly title. Use a punchy editorial headline, not a generic week label. | |
| 198 | | `date` | string | yes | Analysis run timestamp in ISO 8601. | |
| 199 | | `week` | string | yes | Week slug from the raw payload (`YYYY-WNN`). | |
| 200 | | `year` | integer | yes | Numeric year for downstream validation and archive logic. | |
| 201 | | `tags` | array[string] | yes | 3-8 topical tags summarizing the week. | |
| 202 | | `categories` | array[string] | yes | Must include `weekly`. | |
| 203 | | `repos_featured` | integer | yes | Total repos considered in the editorial pass. Typically `len(new_repos) + len(trending_repos)`. | |
| 204 | | `stars_tracked` | integer | yes | Sum of `stars` across all repos considered. | |
| 205 | | `top_repo` | string | yes | The repo that anchors the week’s narrative, not necessarily the highest-star repo. | |
| 206 | | `quality_score` | integer | yes | Reviewer-gate score from 0-100. Must be `>= 60` to publish. | |
| 207 | | `summary` | string | yes | One-sentence editorial thesis for the week. | |
| 208 | | `predictions` | array<object> | no | Optional hindsight registry. Each entry is `{repo, claim_type, direction, confidence}` using `owner/repo`, `signal|noise|gap`, `up|flat|down`, and confidence from `0` to `1`. | |
| 209 | |
| 210 | No extra frontmatter keys should be emitted beyond this contract. |
| 211 | |
| 212 | ### Required body structure |
| 213 | |
| 214 | The body must follow this exact top-level section order: |
| 215 | |
| 216 | ```md |
| 217 | ## This Week's Trends |
| 218 | |
| 219 | ## Where Industry Meets Code |
| 220 | |
| 221 | ## Signal & Noise |
| 222 | |
| 223 | ## Blind Spots |
| 224 | |
| 225 | ## The Week Ahead |
| 226 | |
| 227 | ## Key References |
| 228 | ### Notable Projects |
| 229 | ### Press & Industry |
| 230 | ``` |
| 231 | |
| 232 | Every repository mentioned in the body must be rendered as a clickable markdown link in this exact format: `[owner/repo](https://github.com/owner/repo)`. |
| 233 | |
| 234 | ### Image accessibility guidance |
| 235 | |
| 236 | If an analysis ever includes an image, chart, or screenshot: |
| 237 | |
| 238 | - provide concise, descriptive alt text that explains the information a reader would otherwise miss, |
| 239 | - do not use placeholder alt text like `image`, `screenshot`, or the file name, |
| 240 | - keep decorative images rare; only use empty alt text when the image adds no editorial meaning, |
| 241 | - explain any important numbers or trends in the surrounding prose so the page still works without the image. |
| 242 | |
| 243 | ### Section guidance |
| 244 | |
| 245 | #### 1. This Week's Trends |
| 246 | - **Purpose:** Name and explain the week's 3-5 macro trends — the big themes that cut across individual repos. |
| 247 | - **Include:** A clear name for each trend, what is driving it, and its significance to practitioners. Reference specific repos as evidence. |
| 248 | - **Repo links:** Every repo mention must use `[owner/repo](https://github.com/owner/repo)`. |
| 249 | - **Tone:** Analytical and opinionated — write like a Gartner analyst, not a GitHub trending page. |
| 250 | - **Length:** ~200-350 words. |
| 251 | - **Avoid:** Listing repos without synthesis. Every repo reference must support a named trend. |
| 252 | |
| 253 | #### 2. Where Industry Meets Code |
| 254 | - **Purpose:** Compare press coverage against what developers are actually building. |
| 255 | - **Include:** 2-4 correlations (where press and dev activity align) and 2-3 divergences (media-covered topics with no dev traction, and developer movements the press is ignoring). If no press data was available, state that explicitly. |
| 256 | - **Repo links:** Every repo mention must use `[owner/repo](https://github.com/owner/repo)`. |
| 257 | - **Tone:** Editorial and skeptical — the interesting story is usually in the gap. |
| 258 | - **Length:** ~150-250 words. |
| 259 | - **Avoid:** Summarizing press articles without connecting them to developer evidence. |
| 260 | |
| 261 | #### 3. Signal & Noise |
| 262 | - **Purpose:** Deliver integrated editorial judgment on what is real versus hype. |
| 263 | - **Required:** Write as coherent prose — do **not** use `### Signal` and `### Noise` sub-headings. The distinction should emerge from the writing itself. |
| 264 | - **Include:** Durable, technically credible patterns (signal) and inflated, copycat, or marketing-driven patterns (noise). Name specific repos and patterns in both categories. |
| 265 | - **Repo links:** Every repo mention must use `[owner/repo](https://github.com/owner/repo)`. |
| 266 | - **Length:** ~150-260 words. |
| 267 | - **Avoid:** Repeating trend descriptions from section 1 without adding critical judgment. |
| 268 | |
| 269 | #### 4. Blind Spots |
| 270 | - **Purpose:** Surface what is absent from both press coverage and developer activity. |
| 271 | - **Include:** 2-4 specific, concrete blind spots — name the missing category, why it matters, and what its absence signals. |
| 272 | - **Repo links:** Every repo mention must use `[owner/repo](https://github.com/owner/repo)`. |
| 273 | - **Length:** ~80-160 words. |
| 274 | - **Avoid:** Generic filler like "more innovation is needed" or restating known gaps without editorial insight. |
| 275 | |
| 276 | #### 5. The Week Ahead |
| 277 | - **Purpose:** End with a forward-looking editorial close. |
| 278 | - **Include:** What trends are in motion that have not peaked yet? What should readers watch for next week? What does this week's activity suggest about where the ecosystem is heading? |
| 279 | - **Repo links:** Every repo mention must use `[owner/repo](https://github.com/owner/repo)`. |
| 280 | - **Length:** ~50-110 words. |
| 281 | - **Avoid:** Introducing brand-new evidence or restating section 1. |
| 282 | |
| 283 | #### 6. Key References |
| 284 | - **Purpose:** Give readers the 5-10 most important repos and 3-5 most relevant press items in one scannable place. |
| 285 | - **Required subsections:** `### Notable Projects` and `### Press & Industry`. |
| 286 | - **Notable Projects:** 5-10 repos with one sentence of context each — why it matters, not just what it is. Every repo must be a link. |
| 287 | - **Press & Industry:** 3-5 articles or sources with markdown links. If no press data was available, write: "No press data was provided this week." |
| 288 | - **Repo links:** Every repo mention must use `[owner/repo](https://github.com/owner/repo)`. |
| 289 | |
| 290 | ## Analysis Dimensions |
| 291 | |
| 292 | Every weekly analysis must apply these dimensions explicitly. |
| 293 | |
| 294 | ### Importance Assessment |
| 295 | Ask whether a repo or theme solves a real problem, reduces friction, opens a new workflow, or signals credible adoption. Prefer practical utility over novelty theater. |
| 296 | |
| 297 | ### Trend Detection |
| 298 | Look for repeated patterns across topics, repo types, and—when available—previous weekly summaries. A single loud repo is not a trend; clustered movement is. |
| 299 | |
| 300 | ### Hype Detection |
| 301 | Separate genuine substance from branding, wrappers, thinly differentiated agent launches, or exploit-driven attention. If the repo sounds bigger than it is, say so. |
| 302 | |
| 303 | ### Gap Analysis |
| 304 | Identify what should be showing up but is not: missing infrastructure, underrepresented defensive/security work, absent tooling for known pain points, or stagnant categories. |
| 305 | |
| 306 | ### Context |
| 307 | Compare the current week to the prior week when a prior summary exists. Note continuity, acceleration, reversal, or broadening of a theme. If no prior summary exists, say so briefly and avoid pretending longitudinal certainty. |
| 308 | |
| 309 | ## Quality Criteria |
| 310 | |
| 311 | ### Good analysis |
| 312 | - Synthesizes, ranks, and judges instead of listing. |
| 313 | - Connects individual repos into ecosystem-level patterns. |
| 314 | - Names uncertainty honestly when data quality is limited. |
| 315 | - Uses evidence from the payload without sounding like the payload. |
| 316 | - Makes the `Blind Spots` section useful and specific. |
| 317 | - Leaves Amy’s generator with all frontmatter needed for site publication. |
| 318 | |
| 319 | ### Bad analysis |
| 320 | - Reads like release notes or a changelog. |
| 321 | - Repeats repo descriptions without editorial value. |
| 322 | - Confuses total stars with weekly momentum. |
| 323 | - Refuses to criticize obvious hype or noise. |
| 324 | - Omits gaps, caveats, or trend continuity. |
| 325 | - Produces frontmatter that cannot drive the weekly page template. |
| 326 | |
| 327 | ## Reviewer-Gate Expectations |
| 328 | |
| 329 | A weekly analysis is structurally valid only if all of the following are true: |
| 330 | |
| 331 | - `quality_score >= 60` |
| 332 | - all required frontmatter fields are present, |
| 333 | - all six required H2 sections are present in order (`This Week's Trends`, `Where Industry Meets Code`, `Signal & Noise`, `Blind Spots`, `The Week Ahead`, `Key References`), |
| 334 | - `### Notable Projects` and `### Press & Industry` subsections are present under `## Key References`, |
| 335 | - body word count is at least 200, |
| 336 | - the prose contains no raw JSON, tool logs, or placeholder text. |
| 337 | |
| 338 | Publication also requires a structured gate report with four passing gate families: |
| 339 | |
| 340 | - `structural_schema` — frontmatter, schema, dates, headings, section order, and deterministic repair results. |
| 341 | - `ai_provenance` — a publishable AI source and available model; `no-ai`, unknown, unavailable, or deterministic repair failure outputs are staged but not promoted. |
| 342 | - `evidence_citation` — fresh raw evidence and markdown citations to raw-payload repositories where repository evidence exists. |
| 343 | - `editorial_quality` — enough section depth, explanatory trend judgment, no contradictory claims, and no generic low-signal prose. |
| 344 | |
| 345 | The publish manifest records these gate outcomes and promotion must consume them before replacing a previously published AI-authored article. |
| 346 | |
| 347 | The preflight stage must also emit a deterministic analysis input manifest (`analysis-preflight.json`) before Copilot runs. That manifest records the rendered prompt byte/token estimate, prompt checksum, every prompt component with byte/token/checksum metadata, deterministic slice names, and evidence inventories for raw and prompt-visible repository sets. Each evidence inventory lists resolvable `owner/repo` names, source slice, GitHub URL when present, stars, and `stars_gained` when present. The gate must reject repository markdown links that do not resolve to the current raw evidence inventory when that inventory is available. |
| 348 | |
| 349 | Gate reports are machine-readable failure contracts. They include per-family `gates`, repair actions, `failure_class`, and a structured `failure_summary` with failed categories and error count so workflow retries and publish preservation can classify failures deterministically. |
| 350 | |
| 351 | ## Generator Handoff Rules |
| 352 | |
| 353 | The generator may assume: |
| 354 | |
| 355 | - the summary frontmatter already contains the weekly page fields Amy’s Hugo templates expect, |
| 356 | - `summary` is safe to surface in list views, |
| 357 | - `top_repo` is a deliberate editorial choice, |
| 358 | - body headings are stable and machine-detectable, |
| 359 | - body headings use the stable structure defined in this spec; the generator can extract any section by heading name. |
| 360 | |
| 361 | The analyzer may assume: |
| 362 | |
| 363 | - `data/raw/` is authoritative input, |
| 364 | - prior-week continuity is optional but preferred, |
| 365 | - missing `stars_gained` must produce a caveat, not a silent omission. |