Add automated QA gates for matrix crawl and map/reduce acceptance criteria (#466)
Merging QA gates for matrix crawl/map-reduce acceptance criteria. All review comments resolved, all 10 CI checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Juan Manuel Servera committed
Jun 13, 2026 at 23:09 UTC
5c11d4c874a2cfef4bdeab0fdb6a8064af85b12b
2 files changed
+707
docs/qa-gates.md
new
+96
@@ -0,0 +1,96 @@
1
+# QA Gates: Map/Reduce Acceptance Criteria
2
+
3
+This document describes the automated QA gates that validate matrix crawl and
4
+map/reduce changes before rollout (see issue #438).
5
+
6
+## Gate Structure
7
+
8
+The gates live in `tests/test_qa_gates_map_reduce.py` and run as part of the
9
+standard `python -m pytest` CI step. No additional workflow configuration is
10
+needed.
11
+
12
+### 1. Reducer Correctness (`TestReducerDeterministicFanIn`, `TestReducerCitationPreservation`, `TestReducerContradictionHandling`)
13
+
14
+| Gate | What it checks | Failure meaning |
15
+|------|---------------|-----------------|
16
+| Deterministic fan-in | Same input → same output, order-independent | Reducer has non-deterministic behavior |
17
+| Duplicate collapse | Same normalized key is deduplicated | Duplicate claims leak to editorial plan |
18
+| Citation preservation | Repo/article bindings survive reduce | Published claims would lack evidence links |
19
+| Weak citation rejection | Findings without evidence_refs rejected | Unsupported claims reach output |
20
+| Contradiction handling | Mutual/one-sided contradictions rejected | Conflicting claims not caught |
21
+
22
+### 2. End-to-End Dry-Run (`TestEndToEndDryRun`)
23
+
24
+| Gate | What it checks | Failure meaning |
25
+|------|---------------|-----------------|
26
+| Full pipeline QA pass | Complete run produces `status: passed` | Pipeline broke a contract |
27
+| Mapper contracts valid | Every mapper ledger passes `validate_map` | Schema/coverage violation in mapper |
28
+| Sidecars present | rejected-claims.json & contradictions.json exist | Audit trail missing |
29
+| Never publish eligible | manifest.publish_eligible is always False | Dry-run could accidentally publish |
30
+| Provenance expected failure | Provenance gate fails as expected | Dry-run wrongly claims AI provenance |
31
+| Candidate has links | Output markdown contains repo hyperlinks | Citation rendering broken |
32
+
33
+### 3. Cost/Token Guardrails (`TestCostTokenGuardrails`)
34
+
35
+| Gate | What it checks | Failure meaning |
36
+|------|---------------|-----------------|
37
+| Over-budget rejection | 2M char file → preflight fails | Cost guard is disabled |
38
+| Under-budget pass | Small file → preflight passes | False positive blocking pipeline |
39
+| Custom cap | Tighter cap triggers failure | Cap override broken |
40
+| Unknown model fails | Unrecognized model → exit 1 | Silent pass on unknown pricing |
41
+| Token determinism | Same file → same estimate | Non-deterministic budget |
42
+| Missing file → 0 tokens | Missing path doesn't crash | Pipeline crash on optional inputs |
43
+| Manifest token estimate | rendered_prompt_estimate > 0 | Token accounting missing from artifacts |
44
+| All models have rates | Every MODEL_RATES entry yields cost | Pricing table incomplete |
45
+
46
+### 4. Failure Handling (`TestMapperFailureHandling`)
47
+
48
+| Gate | What it checks | Failure meaning |
49
+|------|---------------|-----------------|
50
+| Missing required fields | validate_map catches all missing fields | Schema violation undetected |
51
+| Wrong schema version | Version mismatch flagged | Version drift undetected |
52
+| Failed status flagged | status=failed → error | Silent failure |
53
+| Malformed findings | Non-dict findings rejected | Crash on bad data |
54
+| Empty ledger list | Zero ledgers → empty plan | Crash on empty input |
55
+| No-URL press mapper | Partial status, no crash | Press mapper crash |
56
+| No press context | Pipeline completes without press | Hard dependency on optional input |
57
+
58
+### 5. Gate Output Clarity (`TestGateOutputClarity`)
59
+
60
+| Gate | What it checks | Failure meaning |
61
+|------|---------------|-----------------|
62
+| Descriptive errors | Each error > 10 chars, actionable | CI errors too terse to debug |
63
+| Gate identification | QA report has `passed` key per gate | CI can't determine which gate failed |
64
+| Rejection reasons | Every rejected claim has a reason | No diagnosis path for rejections |
65
+
66
+## Running Locally
67
+
68
+```bash
69
+python -m pytest tests/test_qa_gates_map_reduce.py -v
70
+```
71
+
72
+## CI Integration
73
+
74
+These tests run in the existing CI workflow (`.github/workflows/ci.yml`) under
75
+the `python -m pytest` step. No additional configuration required — pytest
76
+autodiscovers the test file.
77
+
78
+## Interpreting Failures
79
+
80
+When a gate fails in CI:
81
+
82
+1. The test name tells you which category failed (e.g., `TestCostTokenGuardrails::test_preflight_rejects_over_budget`)
83
+2. The assertion message identifies the specific contract violation
84
+3. The QA report JSON (`qa-comparison-report.json`) provides structured gate-by-gate results
85
+
86
+If `test_full_pipeline_produces_passing_qa` fails, inspect the QA report's
87
+`checks` object carefully. **Not all gates share the same schema** — each gate
88
+reports results in its own structure:
89
+
90
+- `mapper_contracts` includes `passed` plus `errors_by_mapper`
91
+- `structural_analysis_gate` and `evidence_and_editorial_gates` include `passed`
92
+ plus `errors`
93
+- `publish_provenance_gate` includes `passed`, `expected_failure`, and `errors`
94
+- `sidecars_present` includes `passed` plus emitted object counts
95
+- `reference_count` is informational only and reports `selected`,
96
+ `notable_projects`, and `press_articles`
tests/test_qa_gates_map_reduce.py
new
+611
@@ -0,0 +1,611 @@
1
+"""Automated QA gates for matrix crawl and map/reduce acceptance criteria.
2
+
3
+Covers:
4
+- Reducer correctness: deterministic fan-in, citation preservation, contradiction handling
5
+- End-to-end dry-run: full pipeline on representative fixtures with gate validation
6
+- Cost/token guardrails: budget accounting, bounds enforcement
7
+- Failure handling: mapper/reducer failure paths and fallback behavior
8
+
9
+These gates run as part of the existing pytest CI path (issue #438).
10
+"""
11
+
12
+from __future__ import annotations
13
+
14
+import json
15
+from pathlib import Path
16
+
17
+import pytest
18
+
19
+from scripts import map_reduce_dry_run as dry_run
20
+from scripts.preflight_cost_check import estimate_input_tokens, main as preflight_main
21
+from scripts.model_pricing import MODEL_RATES, estimate_cost_usd
22
+
23
+
24
+# ---------------------------------------------------------------------------
25
+# Fixtures
26
+# ---------------------------------------------------------------------------
27
+
28
+def make_repo(owner: str, name: str, stars: int, gained: int = 0, **extra) -> dict:
29
+ base = {
30
+ "name": name,
31
+ "owner": owner,
32
+ "full_name": f"{owner}/{name}",
33
+ "description": f"{name} delivers evidence-backed developer tooling for the open-source ecosystem.",
34
+ "language": "Python",
35
+ "stars": stars,
36
+ "stars_gained": gained,
37
+ "created_at": "2026-05-18T10:00:00Z",
38
+ "topics": ["ai", "developer-tools"],
39
+ "url": f"https://github.com/{owner}/{name}",
40
+ }
41
+ base.update(extra)
42
+ return base
43
+
44
+
45
+def valid_finding(claim_id: str = "claim-1", repo: str = "org/repo", **overrides) -> dict:
46
+ base = {
47
+ "claim_id": claim_id,
48
+ "claim": f"{repo} is supported by repository evidence.",
49
+ "category": "trend",
50
+ "source_type": "github",
51
+ "evidence_refs": [{"type": "repo", "ref": repo, "url": f"https://github.com/{repo}"}],
52
+ "repo_full_name": repo,
53
+ "news_url": None,
54
+ "confidence": 0.78,
55
+ "contra_refs": [],
56
+ "uncertainties": [],
57
+ }
58
+ base.update(overrides)
59
+ return base
60
+
61
+
62
+def valid_ledger(findings=None, shard_id="signal-type:test") -> dict:
63
+ if findings is None:
64
+ findings = [valid_finding()]
65
+ return {
66
+ "schema_version": "analysis_map_v1",
67
+ "run_id": "local",
68
+ "week": "2026-W21",
69
+ "shard_id": shard_id,
70
+ "slice": {},
71
+ "coverage": {
72
+ "repo_ids_seen": ["org/repo"],
73
+ "article_urls_seen": [],
74
+ "repo_count_input": 1,
75
+ "repo_count_mapped": 1,
76
+ "article_count_input": 0,
77
+ "article_count_mapped": 0,
78
+ "excluded_reason_counts": {},
79
+ },
80
+ "findings": findings,
81
+ "citations": [],
82
+ "reference_candidates": {"notable_projects": ["org/repo"], "press_articles": []},
83
+ "provenance": {},
84
+ }
85
+
86
+
87
+RAW_PAYLOAD = {
88
+ "week": "2026-W21",
89
+ "crawled_at": "2026-05-20T12:00:00Z",
90
+ "new_repos": [make_repo("octo", "alpha", 1200, 50), make_repo("octo", "beta", 900, 30)],
91
+ "trending_repos": [make_repo("tools", "gamma", 5000, 450), make_repo("tools", "delta", 3000, 250)],
92
+ "signals": {"top_topics": ["ai", "developer-tools", "testing"]},
93
+}
94
+
95
+PRESS_CONTEXT = (
96
+ "### Correlation Summary\n"
97
+ "- Industry article: https://example.com/ai-tooling links repo momentum to developer tools.\n"
98
+ "- Industry article: https://example.com/oss-growth shows OSS ecosystem expansion.\n"
99
+)
100
+
101
+
102
+# ===========================================================================
103
+# Section 1: Reducer correctness
104
+# ===========================================================================
105
+
106
+
107
+class TestReducerDeterministicFanIn:
108
+ """Validates deterministic fan-in assumptions needed by dry-run map/reduce."""
109
+
110
+ def test_identical_ledgers_produce_stable_output(self):
111
+ """Running reduce_ledgers twice with same input produces identical plan."""
112
+ ledger = valid_ledger()
113
+ plan1, rej1, con1 = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
114
+ plan2, rej2, con2 = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
115
+
116
+ assert plan1 == plan2
117
+ assert rej1 == rej2
118
+ assert con1 == con2
119
+
120
+ def test_ledger_ordering_is_deterministic(self):
121
+ """Reducer produces same output regardless of ledger insertion order."""
122
+ ledger_a = valid_ledger(
123
+ [valid_finding("a1", "org/alpha")],
124
+ shard_id="signal-type:new_repos",
125
+ )
126
+ ledger_b = valid_ledger(
127
+ [valid_finding("b1", "org/beta")],
128
+ shard_id="signal-type:trending_repos",
129
+ )
130
+
131
+ plan_ab, _, _ = dry_run.reduce_ledgers([ledger_a, ledger_b], raw_payload=RAW_PAYLOAD)
132
+ plan_ba, _, _ = dry_run.reduce_ledgers([ledger_b, ledger_a], raw_payload=RAW_PAYLOAD)
133
+
134
+ # Selected claims sorted by section/confidence, so order should be stable
135
+ assert [c["claim_id"] for c in plan_ab["selected_claims"]] == [
136
+ c["claim_id"] for c in plan_ba["selected_claims"]
137
+ ]
138
+
139
+ def test_duplicate_claims_collapsed(self):
140
+ """Claims with the same normalized key are deduplicated by the reducer."""
141
+ finding = valid_finding("dup-1", "org/repo")
142
+ finding_dup = valid_finding("dup-2", "org/repo")
143
+ ledger = valid_ledger([finding, finding_dup])
144
+
145
+ plan, rejected, _ = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
146
+
147
+ selected_ids = [c["claim_id"] for c in plan["selected_claims"]]
148
+ assert len(selected_ids) == 1
149
+ dup_rejects = [r for r in rejected if r["reason"] == "duplicate"]
150
+ assert len(dup_rejects) == 1
151
+
152
+
153
+class TestReducerCitationPreservation:
154
+ """Validates citation preservation for mapper/reducer outputs."""
155
+
156
+ def test_selected_claims_retain_citation_bindings(self):
157
+ """Selected claims keep repo and article citation bindings through reduce."""
158
+ finding = valid_finding("cit-1", "org/cited")
159
+ finding["evidence_refs"].append(
160
+ {"type": "article", "ref": "https://news.example.com/1", "url": "https://news.example.com/1"}
161
+ )
162
+ ledger = valid_ledger([finding])
163
+
164
+ plan, _, _ = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
165
+
166
+ assert len(plan["selected_claims"]) == 1
167
+ bindings = plan["selected_claims"][0]["citation_bindings"]
168
+ assert "org/cited" in bindings["repos"]
169
+ assert "https://news.example.com/1" in bindings["articles"]
170
+
171
+ def test_merged_claims_combine_citations(self):
172
+ """When claims merge, citation bindings from both are combined."""
173
+ f1 = valid_finding("merge-1", "org/repo")
174
+ f1["evidence_refs"] = [{"type": "repo", "ref": "org/repo", "url": "https://github.com/org/repo"}]
175
+
176
+ # Second ledger with same normalized key but different citation
177
+ f2 = valid_finding("merge-2", "org/repo")
178
+ f2["evidence_refs"] = [
179
+ {"type": "repo", "ref": "org/repo", "url": "https://github.com/org/repo"},
180
+ {"type": "article", "ref": "https://news.example.com/x", "url": "https://news.example.com/x"},
181
+ ]
182
+ ledger1 = valid_ledger([f1], shard_id="signal-type:shard1")
183
+ ledger2 = valid_ledger([f2], shard_id="signal-type:shard2")
184
+
185
+ plan, _, _ = dry_run.reduce_ledgers([ledger1, ledger2], raw_payload=RAW_PAYLOAD)
186
+
187
+ assert len(plan["selected_claims"]) == 1
188
+ bindings = plan["selected_claims"][0]["citation_bindings"]
189
+ assert "org/repo" in bindings["repos"]
190
+ assert "https://news.example.com/x" in bindings["articles"]
191
+
192
+ def test_weak_citation_claims_rejected(self):
193
+ """Findings without evidence refs are rejected as weak_citation."""
194
+ finding = valid_finding("weak-1", "org/empty")
195
+ finding["evidence_refs"] = []
196
+ ledger = valid_ledger([finding])
197
+
198
+ plan, rejected, _ = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
199
+
200
+ assert plan["selected_claims"] == []
201
+ weak = [r for r in rejected if r["reason"] == "weak_citation"]
202
+ assert len(weak) == 1
203
+ assert weak[0]["claim_id"] == "weak-1"
204
+
205
+
206
+class TestReducerContradictionHandling:
207
+ """Validates contradiction detection and rejection behavior."""
208
+
209
+ def test_mutual_contradiction_rejects_both(self):
210
+ """Claims that reference each other's contra_refs are both rejected."""
211
+ f1 = valid_finding("contra-1", "org/a")
212
+ f1["contra_refs"] = ["contra-2"]
213
+ f2 = valid_finding("contra-2", "org/b")
214
+ f2["contra_refs"] = ["contra-1"]
215
+ ledger = valid_ledger([f1, f2])
216
+
217
+ plan, rejected, contradictions = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
218
+
219
+ assert plan["selected_claims"] == []
220
+ contra_ids = {c["claim_id"] for c in contradictions}
221
+ assert contra_ids == {"contra-1", "contra-2"}
222
+ reject_ids = {r["claim_id"] for r in rejected if r["reason"] == "unresolved_contradiction"}
223
+ assert reject_ids == {"contra-1", "contra-2"}
224
+
225
+ def test_one_sided_contradiction_rejects_contradicted_claim(self):
226
+ """A claim that contradicts another is rejected along with its target."""
227
+ f1 = valid_finding("target-1", "org/target")
228
+ f2 = valid_finding("attacker-1", "org/attacker")
229
+ f2["contra_refs"] = ["target-1"]
230
+ ledger = valid_ledger([f1, f2])
231
+
232
+ plan, rejected, contradictions = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
233
+
234
+ # The attacker has contra_refs, so it's contradicted
235
+ # The target is contradicted_by the attacker
236
+ contra_ids = {c["claim_id"] for c in contradictions}
237
+ assert "attacker-1" in contra_ids
238
+ assert "target-1" in contra_ids
239
+
240
+ def test_contradictions_sidecar_populated(self):
241
+ """Contradictions list is preserved in the plan for audit."""
242
+ f1 = valid_finding("sc-1", "org/x")
243
+ f1["contra_refs"] = ["sc-2"]
244
+ f2 = valid_finding("sc-2", "org/y")
245
+ ledger = valid_ledger([f1, f2])
246
+
247
+ plan, _, contradictions = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
248
+
249
+ assert plan["contradictions"] == contradictions
250
+ assert len(contradictions) > 0
251
+ for c in contradictions:
252
+ assert "claim_id" in c
253
+ assert "resolution" in c
254
+ assert c["resolution"] == "rejected_unresolved"
255
+
256
+
257
+# ===========================================================================
258
+# Section 2: End-to-end dry-run tests
259
+# ===========================================================================
260
+
261
+
262
+class TestEndToEndDryRun:
263
+ """Exercises full map/reduce scaffolding on fixtures and validates gate outcomes."""
264
+
265
+ @pytest.fixture
266
+ def workspace(self, tmp_path):
267
+ """Create representative fixture workspace."""
268
+ raw_path = tmp_path / "data" / "raw" / "2026-W21.json"
269
+ press_path = tmp_path / "data" / "analyzed" / "2026-W21-press-context.md"
270
+ output_dir = tmp_path / "data" / "candidates" / "2026-W21" / "local" / "map-reduce"
271
+ raw_path.parent.mkdir(parents=True)
272
+ press_path.parent.mkdir(parents=True)
273
+ raw_path.write_text(json.dumps(RAW_PAYLOAD), encoding="utf-8")
274
+ press_path.write_text(PRESS_CONTEXT, encoding="utf-8")
275
+ return raw_path, press_path, output_dir
276
+
277
+ def test_full_pipeline_produces_passing_qa(self, workspace):
278
+ """End-to-end run produces qa-comparison-report with status=passed."""
279
+ raw_path, press_path, output_dir = workspace
280
+
281
+ rc = dry_run.main([
282
+ "--raw-json", raw_path.as_posix(),
283
+ "--press-context", press_path.as_posix(),
284
+ "--output-dir", output_dir.as_posix(),
285
+ "--current-datetime", "2026-05-20T12:00:00Z",
286
+ "--run-id", "qa-gate-test",
287
+ ])
288
+
289
+ assert rc == 0
290
+ qa = json.loads((output_dir / "qa-comparison-report.json").read_text(encoding="utf-8"))
291
+ assert qa["status"] == "passed"
292
+ assert qa["publish_eligible"] is False
293
+
294
+ def test_all_mapper_contracts_valid(self, workspace):
295
+ """Each mapper ledger passes validate_map with zero errors."""
296
+ raw_path, press_path, output_dir = workspace
297
+ rc = dry_run.main([
298
+ "--raw-json", raw_path.as_posix(),
299
+ "--press-context", press_path.as_posix(),
300
+ "--output-dir", output_dir.as_posix(),
301
+ "--current-datetime", "2026-05-20T12:00:00Z",
302
+ "--run-id", "qa-gate-test",
303
+ ])
304
+ assert rc == 0
305
+
306
+ for mapper in dry_run.MAPPER_IDS:
307
+ ledger = json.loads((output_dir / "maps" / f"{mapper}.json").read_text(encoding="utf-8"))
308
+ errors = dry_run.validate_map(ledger)
309
+ assert errors == [], f"Mapper {mapper} contract errors: {errors}"
310
+
311
+ def test_sidecars_always_present(self, workspace):
312
+ """rejected-claims.json and contradictions.json are always emitted."""
313
+ raw_path, press_path, output_dir = workspace
314
+ rc = dry_run.main([
315
+ "--raw-json", raw_path.as_posix(),
316
+ "--press-context", press_path.as_posix(),
317
+ "--output-dir", output_dir.as_posix(),
318
+ "--current-datetime", "2026-05-20T12:00:00Z",
319
+ "--run-id", "qa-gate-test",
320
+ ])
321
+ assert rc == 0
322
+
323
+ rejected = json.loads((output_dir / "sidecars" / "rejected-claims.json").read_text(encoding="utf-8"))
324
+ contras = json.loads((output_dir / "sidecars" / "contradictions.json").read_text(encoding="utf-8"))
325
+ assert rejected["schema_version"] == "analysis_rejected_claims_v1"
326
+ assert contras["schema_version"] == "analysis_contradictions_v1"
327
+ assert isinstance(rejected["rejected_claims"], list)
328
+ assert isinstance(contras["contradictions"], list)
329
+
330
+ def test_manifest_is_never_publish_eligible(self, workspace):
331
+ """Dry-run manifest always marks candidate_only=True, publish_eligible=False."""
332
+ raw_path, press_path, output_dir = workspace
333
+ rc = dry_run.main([
334
+ "--raw-json", raw_path.as_posix(),
335
+ "--press-context", press_path.as_posix(),
336
+ "--output-dir", output_dir.as_posix(),
337
+ "--current-datetime", "2026-05-20T12:00:00Z",
338
+ "--run-id", "qa-gate-test",
339
+ ])
340
+ assert rc == 0
341
+
342
+ manifest = json.loads((output_dir / "manifest.json").read_text(encoding="utf-8"))
343
+ assert manifest["publish_eligible"] is False
344
+ assert manifest["candidate_only"] is True
345
+
346
+ def test_qa_report_documents_expected_provenance_failure(self, workspace):
347
+ """The provenance gate fails as expected (dry-run is not publishable AI)."""
348
+ raw_path, press_path, output_dir = workspace
349
+ rc = dry_run.main([
350
+ "--raw-json", raw_path.as_posix(),
351
+ "--press-context", press_path.as_posix(),
352
+ "--output-dir", output_dir.as_posix(),
353
+ "--current-datetime", "2026-05-20T12:00:00Z",
354
+ "--run-id", "qa-gate-test",
355
+ ])
356
+ assert rc == 0
357
+
358
+ qa = json.loads((output_dir / "qa-comparison-report.json").read_text(encoding="utf-8"))
359
+ provenance = qa["checks"]["publish_provenance_gate"]
360
+ assert provenance["passed"] is False
361
+ assert provenance["expected_failure"] is True
362
+
363
+ def test_candidate_markdown_contains_repo_links(self, workspace):
364
+ """Candidate markdown includes hyperlinks to featured repositories."""
365
+ raw_path, press_path, output_dir = workspace
366
+ rc = dry_run.main([
367
+ "--raw-json", raw_path.as_posix(),
368
+ "--press-context", press_path.as_posix(),
369
+ "--output-dir", output_dir.as_posix(),
370
+ "--current-datetime", "2026-05-20T12:00:00Z",
371
+ "--run-id", "qa-gate-test",
372
+ ])
373
+ assert rc == 0
374
+
375
+ candidate = (output_dir / "2026-W21-map-reduce-candidate.md").read_text(encoding="utf-8")
376
+ assert "[tools/gamma](https://github.com/tools/gamma)" in candidate
377
+ assert "Map/reduce dry-run candidate only" in candidate
378
+
379
+
380
+# ===========================================================================
381
+# Section 3: Cost/token guardrails
382
+# ===========================================================================
383
+
384
+
385
+class TestCostTokenGuardrails:
386
+ """Enforces PRD guardrails for token usage and budget accounting."""
387
+
388
+ def test_preflight_rejects_over_budget(self, tmp_path):
389
+ """Cost check fails when token budget exceeds hard cap."""
390
+ big_file = tmp_path / "big.json"
391
+ big_file.write_text("x" * 2_000_000, encoding="utf-8")
392
+
393
+ rc = preflight_main(["--context-files", str(big_file)])
394
+ assert rc == 1
395
+
396
+ def test_preflight_passes_under_budget(self, tmp_path):
397
+ """Cost check passes when under hard cap."""
398
+ small_file = tmp_path / "small.json"
399
+ small_file.write_text("x" * 400, encoding="utf-8")
400
+
401
+ rc = preflight_main(["--context-files", str(small_file)])
402
+ assert rc == 0
403
+
404
+ def test_preflight_custom_cap_enforcement(self, tmp_path):
405
+ """Custom hard cap correctly triggers failure."""
406
+ medium_file = tmp_path / "medium.json"
407
+ medium_file.write_text("x" * 40_000, encoding="utf-8")
408
+
409
+ rc = preflight_main(["--context-files", str(medium_file), "--hard-cap", "0.001"])
410
+ assert rc == 1
411
+
412
+ def test_unknown_model_always_fails(self, tmp_path):
413
+ """Unknown model name causes preflight to fail rather than pass silently."""
414
+ f = tmp_path / "a.json"
415
+ f.write_text("content", encoding="utf-8")
416
+
417
+ rc = preflight_main(["--context-files", str(f), "--model", "no-such-model-xyz"])
418
+ assert rc == 1
419
+
420
+ def test_token_estimate_deterministic(self, tmp_path):
421
+ """Token estimation is deterministic for the same input."""
422
+ f = tmp_path / "stable.json"
423
+ f.write_text("hello world " * 100, encoding="utf-8")
424
+
425
+ t1 = estimate_input_tokens([f])
426
+ t2 = estimate_input_tokens([f])
427
+ assert t1 == t2
428
+ assert t1 > 0
429
+
430
+ def test_missing_file_yields_zero_tokens(self):
431
+ """Missing files contribute zero tokens, don't crash."""
432
+ tokens = estimate_input_tokens([Path("/nonexistent/file.json")])
433
+ assert tokens == 0
434
+
435
+ def test_manifest_token_estimate_is_positive(self, tmp_path):
436
+ """Dry-run manifest includes positive rendered_prompt_estimate tokens."""
437
+ raw_path = tmp_path / "raw.json"
438
+ press_path = tmp_path / "press.md"
439
+ output_dir = tmp_path / "out"
440
+ raw_path.write_text(json.dumps(RAW_PAYLOAD), encoding="utf-8")
441
+ press_path.write_text(PRESS_CONTEXT, encoding="utf-8")
442
+
443
+ rc = dry_run.main([
444
+ "--raw-json", raw_path.as_posix(),
445
+ "--press-context", press_path.as_posix(),
446
+ "--output-dir", output_dir.as_posix(),
447
+ "--current-datetime", "2026-05-20T12:00:00Z",
448
+ "--run-id", "cost-test",
449
+ ])
450
+ assert rc == 0, f"dry_run.main failed with return code {rc}"
451
+
452
+ manifest = json.loads((output_dir / "manifest.json").read_text(encoding="utf-8"))
453
+ est = manifest["component_estimates"]["rendered_prompt_estimate"]
454
+ assert est["tokens"] > 0
455
+ assert est["bytes"] > 0
456
+ assert len(est["checksum_sha256"]) == 64
457
+
458
+ def test_known_models_have_rates(self):
459
+ """All known models in MODEL_RATES produce non-None cost estimates."""
460
+ for model_name in MODEL_RATES:
461
+ cost = estimate_cost_usd(model_name, 1000, 500)
462
+ assert cost is not None, f"Model {model_name} returned None cost"
463
+ assert cost >= 0
464
+
465
+
466
+# ===========================================================================
467
+# Section 4: Failure handling and fallback behavior
468
+# ===========================================================================
469
+
470
+
471
+class TestMapperFailureHandling:
472
+ """Tests mapper and reducer failure paths."""
473
+
474
+ def test_validate_map_catches_missing_required_fields(self):
475
+ """validate_map returns errors for missing required mapper fields."""
476
+ payload = {"schema_version": "analysis_map_v1"}
477
+ errors = dry_run.validate_map(payload)
478
+ missing_fields = {"run_id", "week", "shard_id", "slice", "coverage", "findings",
479
+ "citations", "reference_candidates", "provenance"}
480
+ for field in missing_fields:
481
+ assert any(field in e for e in errors), f"Missing error for {field}"
482
+
483
+ def test_validate_map_catches_wrong_schema_version(self):
484
+ """Wrong schema version produces a validation error."""
485
+ ledger = valid_ledger()
486
+ ledger["schema_version"] = "wrong_version"
487
+ errors = dry_run.validate_map(ledger)
488
+ assert "mapper schema_version mismatch" in errors
489
+
490
+ def test_validate_map_catches_failed_status(self):
491
+ """A ledger with status=failed is flagged."""
492
+ ledger = valid_ledger()
493
+ ledger["status"] = "failed"
494
+ errors = dry_run.validate_map(ledger)
495
+ assert "mapper status failed" in errors
496
+
497
+ def test_malformed_finding_in_reducer(self):
498
+ """Non-dict findings are rejected as malformed."""
499
+ ledger = valid_ledger()
500
+ ledger["findings"] = ["not-a-dict", 42]
501
+
502
+ plan, rejected, _ = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
503
+
504
+ malformed = [r for r in rejected if r["reason"] == "malformed_finding"]
505
+ assert len(malformed) == 2
506
+
507
+ def test_empty_ledger_list_produces_empty_plan(self):
508
+ """Reducing zero ledgers produces an empty selected_claims list."""
509
+ plan, rejected, contradictions = dry_run.reduce_ledgers([], raw_payload=RAW_PAYLOAD)
510
+ assert plan["selected_claims"] == []
511
+ assert rejected == []
512
+ assert contradictions == []
513
+
514
+ def test_partial_press_mapper_handles_no_urls(self, tmp_path):
515
+ """Press mapper with no URLs emits partial status, not a crash."""
516
+ press_path = tmp_path / "empty-press.md"
517
+ press_path.write_text("No links here.\n", encoding="utf-8")
518
+ raw_path = tmp_path / "raw.json"
519
+ raw_path.write_text(json.dumps(RAW_PAYLOAD), encoding="utf-8")
520
+ raw_ref = dry_run.file_ref(raw_path)
521
+
522
+ result = dry_run.map_press(
523
+ run_id="test", week="2026-W21",
524
+ press_path=press_path,
525
+ press_ref=dry_run.file_ref(press_path),
526
+ raw_ref=raw_ref,
527
+ )
528
+
529
+ assert result["status"] == "partial"
530
+ assert result["findings"] == []
531
+ assert any("No press URLs" in e for e in result["errors"])
532
+
533
+ def test_end_to_end_without_press_context(self, tmp_path):
534
+ """Pipeline still runs when press context is unavailable."""
535
+ raw_path = tmp_path / "raw.json"
536
+ output_dir = tmp_path / "out"
537
+ raw_path.write_text(json.dumps(RAW_PAYLOAD), encoding="utf-8")
538
+
539
+ rc = dry_run.main([
540
+ "--raw-json", raw_path.as_posix(),
541
+ "--output-dir", output_dir.as_posix(),
542
+ "--current-datetime", "2026-05-20T12:00:00Z",
543
+ "--run-id", "no-press",
544
+ ])
545
+
546
+ assert rc == 0
547
+ qa = json.loads((output_dir / "qa-comparison-report.json").read_text(encoding="utf-8"))
548
+ # Should still pass structural checks even without press
549
+ assert qa["checks"]["structural_analysis_gate"]["passed"] is True
550
+
551
+
552
+# ===========================================================================
553
+# Section 5: Gate output clarity (CI-facing documentation)
554
+# ===========================================================================
555
+
556
+
557
+class TestGateOutputClarity:
558
+ """Validates that gate failures produce clear, actionable output."""
559
+
560
+ def test_validate_map_errors_are_descriptive(self):
561
+ """Each validation error string identifies what failed and why."""
562
+ payload = {
563
+ "schema_version": "wrong",
564
+ "findings": "not-a-list",
565
+ }
566
+ errors = dry_run.validate_map(payload)
567
+
568
+ # Every error should be a non-empty string
569
+ for error in errors:
570
+ assert isinstance(error, str)
571
+ assert len(error) > 10, f"Error too terse to be actionable: {error!r}"
572
+
573
+ def test_qa_report_identifies_failing_gate(self, tmp_path):
574
+ """QA report structure makes it clear which gate failed."""
575
+ raw_path = tmp_path / "raw.json"
576
+ press_path = tmp_path / "press.md"
577
+ output_dir = tmp_path / "out"
578
+ raw_path.write_text(json.dumps(RAW_PAYLOAD), encoding="utf-8")
579
+ press_path.write_text(PRESS_CONTEXT, encoding="utf-8")
580
+
581
+ rc = dry_run.main([
582
+ "--raw-json", raw_path.as_posix(),
583
+ "--press-context", press_path.as_posix(),
584
+ "--output-dir", output_dir.as_posix(),
585
+ "--current-datetime", "2026-05-20T12:00:00Z",
586
+ "--run-id", "clarity-test",
587
+ ])
588
+ assert rc == 0
589
+
590
+ qa = json.loads((output_dir / "qa-comparison-report.json").read_text(encoding="utf-8"))
591
+ # Each check section has "passed" bool and either "errors" or "expected_failure"
592
+ for gate_name, gate_data in qa["checks"].items():
593
+ assert "passed" in gate_data or "selected" in gate_data, (
594
+ f"Gate {gate_name} missing 'passed' key — CI cannot determine outcome"
595
+ )
596
+
597
+ def test_rejected_claims_include_reason(self, tmp_path):
598
+ """Every rejected claim has a reason field for diagnosis."""
599
+ f1 = valid_finding("rej-1", "org/x")
600
+ f1["evidence_refs"] = [] # will be rejected as weak_citation
601
+ f2 = valid_finding("rej-2", "org/y")
602
+ f2["contra_refs"] = ["rej-1"] # will be rejected as contradiction
603
+ ledger = valid_ledger([f1, f2])
604
+
605
+ _, rejected, _ = dry_run.reduce_ledgers([ledger], raw_payload=RAW_PAYLOAD)
606
+
607
+ for item in rejected:
608
+ assert "reason" in item, f"Rejected claim missing reason: {item}"
609
+ assert item["reason"] in {
610
+ "weak_citation", "unresolved_contradiction", "duplicate", "malformed_finding"
611
+ }, f"Unknown rejection reason: {item['reason']}"