feat: define topic config schema and validator (#93)
- Pydantic v2 models for squadscope.topic.yml - Validation script with clear error messages - Example configs: ai-ml, rust - Default config at repo root (ai-ml) - 21 tests covering valid/invalid cases Closes #59 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
May 19, 2026 at 15:36 UTC
1cac63d988d10392fac0b5e00869c8cd3dc072db
7 files changed
+504
-2
.squad/agents/bender/history.md
+1
@@ -28,4 +28,5 @@
28
- **2026-05-18T15:22:25.067+02:00:** The reskill trigger can stay lightweight for now: a `reskill-check` job only needs the persisted counter from `crawl`, and a gated placeholder `reskill` job can scaffold `.squad/skills/` and `.squad/reskill/` until Issue #14 adds real retrospective outputs and `.squad/` persistence.
29
- **2026-05-19T11:59:28Z:** Took over PR #55 from Farnsworth (locked out after reviewer rejection). Wrote `docs/PRD-techcrunch-integration.md` — TechCrunch RSS integration PRD positioned as enrichment signal (not primary source). Key design decisions: 3-stage filtering pipeline (category → keyword → entity extraction) to reduce 250 articles/week to ~25; weekly batch alignment instead of real-time; honest 5–15% correlation hit rate with clear failure criteria; delta model (hype vs traction) as the value proposition; implements Decision #7 plugin architecture with `TechCrunchSource` class; negligible cost impact ($0.10–$0.21/year tokens). Force-pushed clean branch, updated PR #55 description, ready for next review.
30
- **2026-05-19T15:08:00Z:** Leela milestone decomposition complete. Issues assigned to v0.5–v0.9 milestones. Scribe logged orchestration and merged decision. Your assigned v0.5 crawler enhancement issues are ready to pull and start. See `.squad/orchestration-log/2026-05-19T15-08-leela.md` for full decomposition outcome.
31
+- **2026-05-19T15:22:00+02:00:** Issue #59 topic config schema implemented. Schema uses Pydantic v2 models in `scripts/validate_topic_config.py`. Key design: `topic` and `queries` sections required, `scoring`/`quality`/`learning` optional with defaults. `topic.id` enforced as lowercase-alphanumeric-hyphens via regex. Language boosts clamped 0.1–10.0. Quality min/max cross-validated. Examples at `examples/topics/{ai-ml,rust}.yml`, default config at repo root `squadscope.topic.yml`. Added pydantic+pyyaml to `requirements.txt`.
32
examples/topics/ai-ml.yml
new
+39
@@ -0,0 +1,39 @@
1
+topic:
2
+ id: ai-ml
3
+ name: "AI & Machine Learning"
4
+ description: "Weekly digest of trending AI/ML repositories on GitHub"
5
+
6
+queries:
7
+ primary:
8
+ - "topic:machine-learning stars:>50 pushed:>{last_week}"
9
+ - "topic:artificial-intelligence stars:>50 pushed:>{last_week}"
10
+ secondary:
11
+ - "topic:transformers stars:>100 pushed:>{last_week}"
12
+ - "topic:llm stars:>100 pushed:>{last_week}"
13
+
14
+scoring:
15
+ min_stars: 20
16
+ min_stars_gained: 10
17
+ max_age_days: 365
18
+ min_relevance_score: 40
19
+ language_boost:
20
+ Python: 1.2
21
+ Jupyter Notebook: 1.1
22
+ topic_relevance:
23
+ - machine-learning
24
+ - deep-learning
25
+ - artificial-intelligence
26
+ - neural-network
27
+ - llm
28
+ - transformers
29
+
30
+quality:
31
+ min_repos_per_week: 5
32
+ max_repos_per_week: 30
33
+ min_quality_score: 60
34
+
35
+learning:
36
+ wisdom_file: "topics/ai-ml/wisdom.md"
37
+ skills_dir: "topics/ai-ml/skills/"
38
+ prediction_file: "topics/ai-ml/predictions.jsonl"
39
+ scorecard_dir: "topics/ai-ml/scorecards/"
examples/topics/rust.yml
new
+37
@@ -0,0 +1,37 @@
1
+topic:
2
+ id: rust
3
+ name: "Rust Ecosystem"
4
+ description: "Weekly digest of trending Rust repositories and crates on GitHub"
5
+
6
+queries:
7
+ primary:
8
+ - "language:rust stars:>50 pushed:>{last_week}"
9
+ - "topic:rust stars:>50 pushed:>{last_week}"
10
+ secondary:
11
+ - "topic:cargo stars:>30 pushed:>{last_week}"
12
+ - "topic:wasm language:rust stars:>30 pushed:>{last_week}"
13
+
14
+scoring:
15
+ min_stars: 30
16
+ min_stars_gained: 15
17
+ max_age_days: 730
18
+ min_relevance_score: 35
19
+ language_boost:
20
+ Rust: 1.3
21
+ topic_relevance:
22
+ - rust
23
+ - cargo
24
+ - wasm
25
+ - systems-programming
26
+ - async-rust
27
+
28
+quality:
29
+ min_repos_per_week: 5
30
+ max_repos_per_week: 25
31
+ min_quality_score: 55
32
+
33
+learning:
34
+ wisdom_file: "topics/rust/wisdom.md"
35
+ skills_dir: "topics/rust/skills/"
36
+ prediction_file: "topics/rust/predictions.jsonl"
37
+ scorecard_dir: "topics/rust/scorecards/"
requirements.txt
+3
-2
@@ -1,2 +1,3 @@
1
-# No external dependencies required.
2
-# The crawler uses the Python 3 standard library only.
1
+# Core dependencies
2
+pydantic>=2.0,<3.0
3
+pyyaml>=6.0,<7.0
scripts/validate_topic_config.py
new
+198
@@ -0,0 +1,198 @@
1
+#!/usr/bin/env python3
2
+"""Validate a squadscope.topic.yml config file against the topic schema.
3
+
4
+Usage:
5
+ python scripts/validate_topic_config.py <path-to-yaml>
6
+
7
+Exit codes:
8
+ 0 — config is valid
9
+ 1 — config is invalid (errors printed to stderr)
10
+
11
+Schema fields:
12
+ topic (required):
13
+ id — URL-safe identifier (lowercase alphanumeric + hyphens)
14
+ name — Human-readable display name
15
+ description — Short description for the topic
16
+
17
+ queries (required):
18
+ primary — List of GitHub search queries (at least one required)
19
+ secondary — Optional list of supplemental queries
20
+
21
+ scoring (optional, has defaults):
22
+ min_stars, min_stars_gained, max_age_days, min_relevance_score
23
+ language_boost — dict of language → multiplier
24
+ topic_relevance — list of relevant GitHub topics
25
+
26
+ quality (optional, has defaults):
27
+ min_repos_per_week, max_repos_per_week, min_quality_score
28
+
29
+ learning (optional, has defaults):
30
+ wisdom_file, skills_dir, prediction_file, scorecard_dir
31
+"""
32
+
33
+from __future__ import annotations
34
+
35
+import re
36
+import sys
37
+from pathlib import Path
38
+from typing import Dict, List, Optional
39
+
40
+import yaml
41
+from pydantic import BaseModel, Field, field_validator, model_validator
42
+
43
+
44
+# --- Pydantic Models ---
45
+
46
+URL_SAFE_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
47
+
48
+
49
+class TopicInfo(BaseModel):
50
+ """Core topic identification. All fields required."""
51
+
52
+ id: str = Field(..., description="URL-safe identifier (lowercase alphanumeric + hyphens)")
53
+ name: str = Field(..., description="Human-readable display name")
54
+ description: str = Field("", description="Short description of the topic")
55
+
56
+ @field_validator("id")
57
+ @classmethod
58
+ def id_must_be_url_safe(cls, v: str) -> str:
59
+ if not URL_SAFE_RE.match(v):
60
+ raise ValueError(
61
+ f"topic.id must be URL-safe (lowercase alphanumeric + hyphens), got: '{v}'"
62
+ )
63
+ return v
64
+
65
+ @field_validator("name")
66
+ @classmethod
67
+ def name_not_empty(cls, v: str) -> str:
68
+ if not v.strip():
69
+ raise ValueError("topic.name must not be empty")
70
+ return v
71
+
72
+
73
+class Queries(BaseModel):
74
+ """Search queries for GitHub API. At least one primary query is required."""
75
+
76
+ primary: List[str] = Field(..., min_length=1, description="Primary search queries (at least one)")
77
+ secondary: List[str] = Field(default_factory=list, description="Optional secondary queries")
78
+
79
+ @field_validator("primary")
80
+ @classmethod
81
+ def primary_not_empty_strings(cls, v: List[str]) -> List[str]:
82
+ for i, q in enumerate(v):
83
+ if not q.strip():
84
+ raise ValueError(f"queries.primary[{i}] must not be an empty string")
85
+ return v
86
+
87
+
88
+class Scoring(BaseModel):
89
+ """Scoring thresholds and boosts. All fields have sensible defaults."""
90
+
91
+ min_stars: int = Field(default=20, ge=0, description="Minimum star count")
92
+ min_stars_gained: int = Field(default=10, ge=0, description="Minimum stars gained in period")
93
+ max_age_days: int = Field(default=365, ge=1, le=3650, description="Max repo age in days")
94
+ min_relevance_score: int = Field(default=40, ge=0, le=100, description="Minimum relevance score (0-100)")
95
+ language_boost: Dict[str, float] = Field(
96
+ default_factory=dict, description="Language → score multiplier"
97
+ )
98
+ topic_relevance: List[str] = Field(
99
+ default_factory=list, description="GitHub topics that boost relevance"
100
+ )
101
+
102
+ @field_validator("language_boost")
103
+ @classmethod
104
+ def boost_values_reasonable(cls, v: Dict[str, float]) -> Dict[str, float]:
105
+ for lang, boost in v.items():
106
+ if boost < 0.1 or boost > 10.0:
107
+ raise ValueError(
108
+ f"scoring.language_boost['{lang}'] must be between 0.1 and 10.0, got {boost}"
109
+ )
110
+ return v
111
+
112
+
113
+class Quality(BaseModel):
114
+ """Quality gates for output. All fields have sensible defaults."""
115
+
116
+ min_repos_per_week: int = Field(default=5, ge=1, description="Minimum repos to include per week")
117
+ max_repos_per_week: int = Field(default=30, ge=1, description="Maximum repos to include per week")
118
+ min_quality_score: int = Field(default=60, ge=0, le=100, description="Minimum quality score (0-100)")
119
+
120
+ @model_validator(mode="after")
121
+ def min_less_than_max(self) -> "Quality":
122
+ if self.min_repos_per_week > self.max_repos_per_week:
123
+ raise ValueError(
124
+ f"quality.min_repos_per_week ({self.min_repos_per_week}) "
125
+ f"must be <= max_repos_per_week ({self.max_repos_per_week})"
126
+ )
127
+ return self
128
+
129
+
130
+class Learning(BaseModel):
131
+ """Paths for learning/feedback loop artifacts. Supports {topic_id} placeholder."""
132
+
133
+ wisdom_file: str = Field(
134
+ default="topics/{topic_id}/wisdom.md", description="Path to wisdom markdown"
135
+ )
136
+ skills_dir: str = Field(
137
+ default="topics/{topic_id}/skills/", description="Directory for learned skills"
138
+ )
139
+ prediction_file: str = Field(
140
+ default="topics/{topic_id}/predictions.jsonl", description="Path to predictions log"
141
+ )
142
+ scorecard_dir: str = Field(
143
+ default="topics/{topic_id}/scorecards/", description="Directory for scorecards"
144
+ )
145
+
146
+
147
+class TopicConfig(BaseModel):
148
+ """Root model for squadscope.topic.yml configuration."""
149
+
150
+ topic: TopicInfo
151
+ queries: Queries
152
+ scoring: Scoring = Field(default_factory=Scoring)
153
+ quality: Quality = Field(default_factory=Quality)
154
+ learning: Learning = Field(default_factory=Learning)
155
+
156
+
157
+# --- CLI Entrypoint ---
158
+
159
+
160
+def validate_file(path: str) -> TopicConfig:
161
+ """Load and validate a YAML topic config file. Returns the validated model."""
162
+ file_path = Path(path)
163
+ if not file_path.exists():
164
+ raise FileNotFoundError(f"Config file not found: {path}")
165
+
166
+ with open(file_path, "r", encoding="utf-8") as f:
167
+ raw = yaml.safe_load(f)
168
+
169
+ if not isinstance(raw, dict):
170
+ raise ValueError("Config file must contain a YAML mapping at the top level")
171
+
172
+ return TopicConfig.model_validate(raw)
173
+
174
+
175
+def main() -> int:
176
+ if len(sys.argv) != 2:
177
+ print("Usage: python scripts/validate_topic_config.py <path-to-yaml>", file=sys.stderr)
178
+ return 1
179
+
180
+ path = sys.argv[1]
181
+ try:
182
+ config = validate_file(path)
183
+ print(f"✓ Valid topic config: {config.topic.name} ({config.topic.id})")
184
+ return 0
185
+ except FileNotFoundError as e:
186
+ print(f"✗ Error: {e}", file=sys.stderr)
187
+ return 1
188
+ except ValueError as e:
189
+ print(f"✗ Validation error: {e}", file=sys.stderr)
190
+ return 1
191
+ except Exception as e:
192
+ # Pydantic validation errors
193
+ print(f"✗ Validation failed:\n{e}", file=sys.stderr)
194
+ return 1
195
+
196
+
197
+if __name__ == "__main__":
198
+ sys.exit(main())
squadscope.topic.yml
new
+39
@@ -0,0 +1,39 @@
1
+topic:
2
+ id: ai-ml
3
+ name: "AI & Machine Learning"
4
+ description: "Weekly digest of trending AI/ML repositories on GitHub"
5
+
6
+queries:
7
+ primary:
8
+ - "topic:machine-learning stars:>50 pushed:>{last_week}"
9
+ - "topic:artificial-intelligence stars:>50 pushed:>{last_week}"
10
+ secondary:
11
+ - "topic:transformers stars:>100 pushed:>{last_week}"
12
+ - "topic:llm stars:>100 pushed:>{last_week}"
13
+
14
+scoring:
15
+ min_stars: 20
16
+ min_stars_gained: 10
17
+ max_age_days: 365
18
+ min_relevance_score: 40
19
+ language_boost:
20
+ Python: 1.2
21
+ Jupyter Notebook: 1.1
22
+ topic_relevance:
23
+ - machine-learning
24
+ - deep-learning
25
+ - artificial-intelligence
26
+ - neural-network
27
+ - llm
28
+ - transformers
29
+
30
+quality:
31
+ min_repos_per_week: 5
32
+ max_repos_per_week: 30
33
+ min_quality_score: 60
34
+
35
+learning:
36
+ wisdom_file: "topics/ai-ml/wisdom.md"
37
+ skills_dir: "topics/ai-ml/skills/"
38
+ prediction_file: "topics/ai-ml/predictions.jsonl"
39
+ scorecard_dir: "topics/ai-ml/scorecards/"
tests/test_validate_topic_config.py
new
+187
@@ -0,0 +1,187 @@
1
+"""Tests for topic config schema validation."""
2
+
3
+import pytest
4
+from pydantic import ValidationError
5
+
6
+from scripts.validate_topic_config import TopicConfig, validate_file
7
+
8
+
9
+# --- Valid configs pass ---
10
+
11
+
12
+def test_valid_ai_ml_config():
13
+ config = validate_file("examples/topics/ai-ml.yml")
14
+ assert config.topic.id == "ai-ml"
15
+ assert config.topic.name == "AI & Machine Learning"
16
+ assert len(config.queries.primary) >= 1
17
+
18
+
19
+def test_valid_rust_config():
20
+ config = validate_file("examples/topics/rust.yml")
21
+ assert config.topic.id == "rust"
22
+ assert config.scoring.min_stars == 30
23
+
24
+
25
+def test_valid_root_config():
26
+ config = validate_file("squadscope.topic.yml")
27
+ assert config.topic.id == "ai-ml"
28
+
29
+
30
+def test_minimal_config():
31
+ """Only required fields — scoring/quality/learning get defaults."""
32
+ data = {
33
+ "topic": {"id": "minimal", "name": "Minimal Topic"},
34
+ "queries": {"primary": ["stars:>100"]},
35
+ }
36
+ config = TopicConfig.model_validate(data)
37
+ assert config.scoring.min_stars == 20
38
+ assert config.quality.min_repos_per_week == 5
39
+ assert "{topic_id}" in config.learning.wisdom_file
40
+
41
+
42
+# --- Invalid configs fail with correct errors ---
43
+
44
+
45
+def test_invalid_topic_id_uppercase():
46
+ data = {
47
+ "topic": {"id": "AI-ML", "name": "Test"},
48
+ "queries": {"primary": ["stars:>10"]},
49
+ }
50
+ with pytest.raises(ValidationError, match="URL-safe"):
51
+ TopicConfig.model_validate(data)
52
+
53
+
54
+def test_invalid_topic_id_spaces():
55
+ data = {
56
+ "topic": {"id": "ai ml", "name": "Test"},
57
+ "queries": {"primary": ["stars:>10"]},
58
+ }
59
+ with pytest.raises(ValidationError, match="URL-safe"):
60
+ TopicConfig.model_validate(data)
61
+
62
+
63
+def test_invalid_topic_id_special_chars():
64
+ data = {
65
+ "topic": {"id": "ai_ml!", "name": "Test"},
66
+ "queries": {"primary": ["stars:>10"]},
67
+ }
68
+ with pytest.raises(ValidationError, match="URL-safe"):
69
+ TopicConfig.model_validate(data)
70
+
71
+
72
+def test_missing_topic_name():
73
+ data = {
74
+ "topic": {"id": "test"},
75
+ "queries": {"primary": ["stars:>10"]},
76
+ }
77
+ with pytest.raises(ValidationError, match="name"):
78
+ TopicConfig.model_validate(data)
79
+
80
+
81
+def test_empty_topic_name():
82
+ data = {
83
+ "topic": {"id": "test", "name": " "},
84
+ "queries": {"primary": ["stars:>10"]},
85
+ }
86
+ with pytest.raises(ValidationError, match="must not be empty"):
87
+ TopicConfig.model_validate(data)
88
+
89
+
90
+def test_missing_primary_queries():
91
+ data = {
92
+ "topic": {"id": "test", "name": "Test"},
93
+ "queries": {"secondary": ["stars:>100"]},
94
+ }
95
+ with pytest.raises(ValidationError, match="primary"):
96
+ TopicConfig.model_validate(data)
97
+
98
+
99
+def test_empty_primary_queries_list():
100
+ data = {
101
+ "topic": {"id": "test", "name": "Test"},
102
+ "queries": {"primary": []},
103
+ }
104
+ with pytest.raises(ValidationError, match="least"):
105
+ TopicConfig.model_validate(data)
106
+
107
+
108
+def test_empty_string_in_primary():
109
+ data = {
110
+ "topic": {"id": "test", "name": "Test"},
111
+ "queries": {"primary": [""]},
112
+ }
113
+ with pytest.raises(ValidationError, match="empty string"):
114
+ TopicConfig.model_validate(data)
115
+
116
+
117
+def test_invalid_language_boost_too_high():
118
+ data = {
119
+ "topic": {"id": "test", "name": "Test"},
120
+ "queries": {"primary": ["stars:>10"]},
121
+ "scoring": {"language_boost": {"Python": 50.0}},
122
+ }
123
+ with pytest.raises(ValidationError, match="between 0.1 and 10.0"):
124
+ TopicConfig.model_validate(data)
125
+
126
+
127
+def test_invalid_language_boost_too_low():
128
+ data = {
129
+ "topic": {"id": "test", "name": "Test"},
130
+ "queries": {"primary": ["stars:>10"]},
131
+ "scoring": {"language_boost": {"Go": 0.0}},
132
+ }
133
+ with pytest.raises(ValidationError, match="between 0.1 and 10.0"):
134
+ TopicConfig.model_validate(data)
135
+
136
+
137
+def test_invalid_quality_min_greater_than_max():
138
+ data = {
139
+ "topic": {"id": "test", "name": "Test"},
140
+ "queries": {"primary": ["stars:>10"]},
141
+ "quality": {"min_repos_per_week": 50, "max_repos_per_week": 10},
142
+ }
143
+ with pytest.raises(ValidationError, match="must be <="):
144
+ TopicConfig.model_validate(data)
145
+
146
+
147
+def test_invalid_relevance_score_out_of_range():
148
+ data = {
149
+ "topic": {"id": "test", "name": "Test"},
150
+ "queries": {"primary": ["stars:>10"]},
151
+ "scoring": {"min_relevance_score": 150},
152
+ }
153
+ with pytest.raises(ValidationError, match="less than or equal to 100"):
154
+ TopicConfig.model_validate(data)
155
+
156
+
157
+def test_missing_topic_section():
158
+ data = {"queries": {"primary": ["stars:>10"]}}
159
+ with pytest.raises(ValidationError, match="topic"):
160
+ TopicConfig.model_validate(data)
161
+
162
+
163
+def test_missing_queries_section():
164
+ data = {"topic": {"id": "test", "name": "Test"}}
165
+ with pytest.raises(ValidationError, match="queries"):
166
+ TopicConfig.model_validate(data)
167
+
168
+
169
+def test_file_not_found():
170
+ with pytest.raises(FileNotFoundError):
171
+ validate_file("nonexistent.yml")
172
+
173
+
174
+def test_cli_exit_code_valid(monkeypatch):
175
+ """CLI returns 0 for valid config."""
176
+ from scripts.validate_topic_config import main
177
+
178
+ monkeypatch.setattr("sys.argv", ["validate", "examples/topics/ai-ml.yml"])
179
+ assert main() == 0
180
+
181
+
182
+def test_cli_exit_code_invalid(monkeypatch):
183
+ """CLI returns 1 for missing file."""
184
+ from scripts.validate_topic_config import main
185
+
186
+ monkeypatch.setattr("sys.argv", ["validate", "nonexistent.yml"])
187
+ assert main() == 1