| 1 | # QA Gates: Map/Reduce Acceptance Criteria |
| 2 | |
| 3 | This document describes the automated QA gates that validate matrix crawl and |
| 4 | map/reduce changes before rollout (see issue #438). |
| 5 | |
| 6 | ## Gate Structure |
| 7 | |
| 8 | The gates live in `tests/test_qa_gates_map_reduce.py` and run as part of the |
| 9 | standard `python -m pytest` CI step. No additional workflow configuration is |
| 10 | needed. |
| 11 | |
| 12 | ### 1. Reducer Correctness (`TestReducerDeterministicFanIn`, `TestReducerCitationPreservation`, `TestReducerContradictionHandling`) |
| 13 | |
| 14 | | Gate | What it checks | Failure meaning | |
| 15 | |------|---------------|-----------------| |
| 16 | | Deterministic fan-in | Same input → same output, order-independent | Reducer has non-deterministic behavior | |
| 17 | | Duplicate collapse | Same normalized key is deduplicated | Duplicate claims leak to editorial plan | |
| 18 | | Citation preservation | Repo/article bindings survive reduce | Published claims would lack evidence links | |
| 19 | | Weak citation rejection | Findings without evidence_refs rejected | Unsupported claims reach output | |
| 20 | | Contradiction handling | Mutual/one-sided contradictions rejected | Conflicting claims not caught | |
| 21 | |
| 22 | ### 2. End-to-End Dry-Run (`TestEndToEndDryRun`) |
| 23 | |
| 24 | | Gate | What it checks | Failure meaning | |
| 25 | |------|---------------|-----------------| |
| 26 | | Full pipeline QA pass | Complete run produces `status: passed` | Pipeline broke a contract | |
| 27 | | Mapper contracts valid | Every mapper ledger passes `validate_map` | Schema/coverage violation in mapper | |
| 28 | | Sidecars present | rejected-claims.json & contradictions.json exist | Audit trail missing | |
| 29 | | Never publish eligible | manifest.publish_eligible is always False | Dry-run could accidentally publish | |
| 30 | | Provenance expected failure | Provenance gate fails as expected | Dry-run wrongly claims AI provenance | |
| 31 | | Candidate has links | Output markdown contains repo hyperlinks | Citation rendering broken | |
| 32 | |
| 33 | ### 3. Cost/Token Guardrails (`TestCostTokenGuardrails`) |
| 34 | |
| 35 | | Gate | What it checks | Failure meaning | |
| 36 | |------|---------------|-----------------| |
| 37 | | Over-budget rejection | 2M char file → preflight fails | Cost guard is disabled | |
| 38 | | Under-budget pass | Small file → preflight passes | False positive blocking pipeline | |
| 39 | | Custom cap | Tighter cap triggers failure | Cap override broken | |
| 40 | | Unknown model fails | Unrecognized model → exit 1 | Silent pass on unknown pricing | |
| 41 | | Token determinism | Same file → same estimate | Non-deterministic budget | |
| 42 | | Missing file → 0 tokens | Missing path doesn't crash | Pipeline crash on optional inputs | |
| 43 | | Manifest token estimate | rendered_prompt_estimate > 0 | Token accounting missing from artifacts | |
| 44 | | All models have rates | Every MODEL_RATES entry yields cost | Pricing table incomplete | |
| 45 | |
| 46 | ### 4. Failure Handling (`TestMapperFailureHandling`) |
| 47 | |
| 48 | | Gate | What it checks | Failure meaning | |
| 49 | |------|---------------|-----------------| |
| 50 | | Missing required fields | validate_map catches all missing fields | Schema violation undetected | |
| 51 | | Wrong schema version | Version mismatch flagged | Version drift undetected | |
| 52 | | Failed status flagged | status=failed → error | Silent failure | |
| 53 | | Malformed findings | Non-dict findings rejected | Crash on bad data | |
| 54 | | Empty ledger list | Zero ledgers → empty plan | Crash on empty input | |
| 55 | | No-URL press mapper | Partial status, no crash | Press mapper crash | |
| 56 | | No press context | Pipeline completes without press | Hard dependency on optional input | |
| 57 | |
| 58 | ### 5. Gate Output Clarity (`TestGateOutputClarity`) |
| 59 | |
| 60 | | Gate | What it checks | Failure meaning | |
| 61 | |------|---------------|-----------------| |
| 62 | | Descriptive errors | Each error > 10 chars, actionable | CI errors too terse to debug | |
| 63 | | Gate identification | QA report has `passed` key per gate | CI can't determine which gate failed | |
| 64 | | Rejection reasons | Every rejected claim has a reason | No diagnosis path for rejections | |
| 65 | |
| 66 | ## Running Locally |
| 67 | |
| 68 | ```bash |
| 69 | python -m pytest tests/test_qa_gates_map_reduce.py -v |
| 70 | ``` |
| 71 | |
| 72 | ## CI Integration |
| 73 | |
| 74 | These tests run in the existing CI workflow (`.github/workflows/ci.yml`) under |
| 75 | the `python -m pytest` step. No additional configuration required — pytest |
| 76 | autodiscovers the test file. |
| 77 | |
| 78 | ## Interpreting Failures |
| 79 | |
| 80 | When a gate fails in CI: |
| 81 | |
| 82 | 1. The test name tells you which category failed (e.g., `TestCostTokenGuardrails::test_preflight_rejects_over_budget`) |
| 83 | 2. The assertion message identifies the specific contract violation |
| 84 | 3. The QA report JSON (`qa-comparison-report.json`) provides structured gate-by-gate results with error lists |
| 85 | |
| 86 | If `test_full_pipeline_produces_passing_qa` fails, check the QA report's |
| 87 | `checks` object — each gate has a `passed` boolean and an `errors` list. |