| 1 | # Map/Reduce Analysis Promotion Path |
| 2 | |
| 3 | **Issue:** #331 |
| 4 | **Status:** Candidate-only (promotion blocked by default) |
| 5 | **Effective:** 2026-06-14 |
| 6 | |
| 7 | ## Overview |
| 8 | |
| 9 | This document defines how map/reduce analysis can graduate from candidate-only |
| 10 | dry runs to publish eligibility. Promotion requires passing a multi-week |
| 11 | comparison framework, meeting explicit quality thresholds, and receiving |
| 12 | operator approval. Until all criteria are met, map/reduce output remains |
| 13 | non-publishing. |
| 14 | |
| 15 | --- |
| 16 | |
| 17 | ## Comparison Framework |
| 18 | |
| 19 | Before any promotion decision, at least **3 representative weekly runs** must |
| 20 | produce side-by-side QA reports comparing map/reduce candidate output against |
| 21 | the current single-pass analyzer. |
| 22 | |
| 23 | ### Metrics Compared |
| 24 | |
| 25 | | Metric | Source | Threshold | |
| 26 | |--------|--------|-----------| |
| 27 | | Evidence coverage | Coverage ledger `repo_count_mapped / repo_count_input` | Track weekly delta vs. single-pass; every promotion-window run must still stay ≥ 0.85 absolute coverage | |
| 28 | | Citation integrity | `citation_bindings` vs. rendered markdown links | Zero orphaned citations | |
| 29 | | Contradiction handling | `contradictions` sidecar count + resolution audit | Zero unresolved contradictions in the promotion window | |
| 30 | | Claim rejection | `rejected_claims` with reason audit | Every rejected claim must carry an allowed audit reason; no valid claims incorrectly rejected | |
| 31 | | Quality score | `analysis_gate.validate_analysis` word count + structure | Track weekly delta vs. single-pass; every promotion-window run must still stay ≥ 60 with ≥ 65 average | |
| 32 | | Gate pass/fail | `analysis_gate.validate_publish_quality` | Must pass all non-provenance gates | |
| 33 | |
| 34 | ### Comparison Run Protocol |
| 35 | |
| 36 | 1. **Input:** Same `data/raw/YYYY-WNN.json` for both paths. |
| 37 | 2. **Single-pass baseline:** The existing `scripts/analyze.py` output at |
| 38 | `data/analyzed/YYYY-WNN-summary.md`. |
| 39 | 3. **Map/reduce candidate:** Output of `scripts/map_reduce_dry_run.py` at |
| 40 | `data/map-reduce-candidates/YYYY-WNN/`. |
| 41 | 4. **Comparison report:** Generated by `scripts/map_reduce_comparison.py` at |
| 42 | `data/map-reduce-candidates/YYYY-WNN/comparison-report.json`. |
| 43 | 5. **Minimum runs:** 3 consecutive weeks must pass before promotion is eligible. |
| 44 | 6. **Staleness:** If any comparison run is older than 28 days at promotion time, |
| 45 | it must be re-run against fresh single-pass output. |
| 46 | |
| 47 | ### Comparison Report Schema |
| 48 | |
| 49 | ```json |
| 50 | { |
| 51 | "schema_version": "comparison_report_v1", |
| 52 | "week": "2026-W24", |
| 53 | "run_datetime": "2026-06-14T07:30:00Z", |
| 54 | "single_pass": { |
| 55 | "artifact_path": "data/analyzed/2026-W24-summary.md", |
| 56 | "sha256": "...", |
| 57 | "quality_score": 72, |
| 58 | "gate_passed": true, |
| 59 | "evidence_coverage": 0.92, |
| 60 | "citation_count": 14, |
| 61 | "word_count": 1850 |
| 62 | }, |
| 63 | "map_reduce": { |
| 64 | "artifact_path": "data/map-reduce-candidates/2026-W24/2026-W24-map-reduce-candidate.md", |
| 65 | "sha256": "...", |
| 66 | "quality_score": 68, |
| 67 | "gate_passed": true, |
| 68 | "evidence_coverage": 0.88, |
| 69 | "citation_count": 12, |
| 70 | "word_count": 1720, |
| 71 | "mapper_errors": {}, |
| 72 | "claims_rejected": 5, |
| 73 | "unresolved_contradictions": 0, |
| 74 | "orphaned_citations": 0, |
| 75 | "invalid_rejected_claims": 0 |
| 76 | }, |
| 77 | "deltas": { |
| 78 | "quality_score": -4, |
| 79 | "evidence_coverage": -0.04, |
| 80 | "citation_count": -2, |
| 81 | "word_count": -130, |
| 82 | "gate_regression": false, |
| 83 | "orphaned_citations": 0, |
| 84 | "unresolved_contradictions": 0, |
| 85 | "invalid_rejected_claims": 0 |
| 86 | }, |
| 87 | "verdict": "pass", |
| 88 | "blockers": [] |
| 89 | } |
| 90 | ``` |
| 91 | |
| 92 | --- |
| 93 | |
| 94 | ## Promotion Criteria |
| 95 | |
| 96 | All of the following MUST be true before map/reduce output becomes |
| 97 | publish-eligible: |
| 98 | |
| 99 | ### Quality Gates |
| 100 | |
| 101 | 1. **Gate parity:** Map/reduce candidate passes `validate_analysis` and |
| 102 | `validate_publish_quality` (excluding provenance gate) for all 3+ |
| 103 | comparison runs. |
| 104 | 2. **Quality score floor:** Map/reduce `quality_score` deltas are recorded |
| 105 | against the single-pass baseline for every run, and the candidate must still |
| 106 | stay ≥ 60 for every run AND average ≥ 65 across the promotion window. |
| 107 | 3. **Evidence coverage:** `evidence_coverage` deltas are recorded against the |
| 108 | single-pass baseline for every run, and the candidate must still stay ≥ 0.85 |
| 109 | absolute coverage for every run (i.e., at least 85% of input repos appear in |
| 110 | mapper output). |
| 111 | 4. **Citation integrity:** Zero orphaned citations — every `[repo](url)` in |
| 112 | rendered markdown must trace back to a `citation_bindings` entry in the |
| 113 | editorial plan. |
| 114 | 5. **Contradiction handling:** Promotion-window runs must have zero unresolved |
| 115 | contradictions preserved in the reducer sidecars. |
| 116 | 6. **Claim rejection audit:** Every rejected claim must carry an allowed audit |
| 117 | reason, and no valid claim may be incorrectly rejected. |
| 118 | 7. **No gate regression:** If single-pass passes all gates, map/reduce must |
| 119 | also pass all gates. A map/reduce gate failure when single-pass succeeds |
| 120 | is a blocking regression. |
| 121 | |
| 122 | ### Editorial Requirements |
| 123 | |
| 124 | 8. **Section completeness:** All 5 required sections (`## This Week's Trends`, |
| 125 | `## Where Industry Meets Code`, `## Signal & Noise`, `## Blind Spots`, |
| 126 | `## The Week Ahead`) meet minimum word counts per `analysis-spec.md`. |
| 127 | 9. **Contradiction transparency:** Any contradiction preserved for audit must |
| 128 | be surfaced in the QA sidecar, not silently dropped. |
| 129 | 10. **Claim provenance:** Every selected claim traces to at least one mapper |
| 130 | finding with explicit `evidence_refs`. |
| 131 | |
| 132 | ### Approval Requirements |
| 133 | |
| 134 | 11. **Operator opt-in:** The `MAPREDUCE_PUBLISH_ELIGIBLE` environment variable |
| 135 | or workflow input must be explicitly set to `true`. Default is `false`. |
| 136 | 12. **Human sign-off:** At least one human reviewer must approve the promotion |
| 137 | PR that sets `publish_eligible: true` in the workflow configuration. |
| 138 | 13. **Team sign-off:** Leela (scope/risk), Farnsworth (editorial quality), |
| 139 | Fry (gates), and Bender (artifact determinism) must each approve in the |
| 140 | promotion PR. |
| 141 | |
| 142 | --- |
| 143 | |
| 144 | ## Rollback Rules |
| 145 | |
| 146 | ### Automatic Rollback Triggers |
| 147 | |
| 148 | Map/reduce output is automatically replaced by single-pass output when: |
| 149 | |
| 150 | 1. **Gate failure:** Map/reduce candidate fails any non-provenance gate that |
| 151 | the single-pass baseline passes. |
| 152 | 2. **Quality regression:** `quality_score` drops below 55 (hard floor). |
| 153 | 3. **Coverage collapse:** `evidence_coverage` < 0.70. |
| 154 | 4. **Mapper failure:** Any mapper returns `status: "failed"` and the reducer |
| 155 | cannot recover. |
| 156 | 5. **Timeout:** Map/reduce pipeline exceeds the configured time budget |
| 157 | (default: 5 minutes for full pipeline). |
| 158 | |
| 159 | ### Rollback Behavior |
| 160 | |
| 161 | - The workflow publishes the single-pass output as if map/reduce never ran. |
| 162 | - The failed map/reduce candidate is preserved in |
| 163 | `data/map-reduce-candidates/YYYY-WNN/` for post-mortem analysis. |
| 164 | - The QA report records `"rollback": true` with the trigger reason. |
| 165 | - An alert is emitted to the orchestration log so the team can investigate. |
| 166 | - Rollback does NOT require human intervention — it is automatic and safe. |
| 167 | |
| 168 | ### Manual Rollback |
| 169 | |
| 170 | An operator can force rollback at any time by: |
| 171 | |
| 172 | 1. Setting `MAPREDUCE_PUBLISH_ELIGIBLE=false` in the workflow environment. |
| 173 | 2. Re-running the analysis workflow — single-pass takes over immediately. |
| 174 | 3. No data loss: all map/reduce candidates remain archived. |
| 175 | |
| 176 | ### Rollback Testing |
| 177 | |
| 178 | Before promotion, the rollback path MUST be tested: |
| 179 | |
| 180 | - Inject a deliberately failing mapper (e.g., truncated input) and verify |
| 181 | that single-pass output publishes without interruption. |
| 182 | - Inject a quality_score below 55 and verify automatic rollback triggers. |
| 183 | - Verify that `data/map-reduce-candidates/` preserves the failed artifact. |
| 184 | |
| 185 | --- |
| 186 | |
| 187 | ## Operator Controls |
| 188 | |
| 189 | ### Environment Variables |
| 190 | |
| 191 | | Variable | Default | Effect | |
| 192 | |----------|---------|--------| |
| 193 | | `MAPREDUCE_PUBLISH_ELIGIBLE` | `false` | Must be `true` for map/reduce to publish | |
| 194 | | `MAPREDUCE_FORCE_ROLLBACK` | `false` | Force immediate rollback to single-pass | |
| 195 | | `MAPREDUCE_DRY_RUN_ONLY` | `true` | When `true`, produce candidates but never publish | |
| 196 | | `MAPREDUCE_COMPARISON_MODE` | `true` | Generate comparison reports alongside candidates | |
| 197 | | `MAPREDUCE_TIME_BUDGET_SECONDS` | `300` | Max pipeline runtime before timeout rollback | |
| 198 | | `MAPREDUCE_MIN_QUALITY_SCORE` | `60` | Per-run quality floor for promotion | |
| 199 | | `MAPREDUCE_MIN_COVERAGE` | `0.85` | Per-run evidence coverage floor | |
| 200 | |
| 201 | ### Workflow Integration |
| 202 | |
| 203 | The analysis workflow (`analyze.yml`) checks controls in this order: |
| 204 | |
| 205 | 1. If `MAPREDUCE_FORCE_ROLLBACK=true` → skip map/reduce entirely. |
| 206 | 2. If `MAPREDUCE_DRY_RUN_ONLY=true` → run map/reduce but only save candidate. |
| 207 | 3. If `MAPREDUCE_PUBLISH_ELIGIBLE=true` → run comparison, check all promotion |
| 208 | criteria, publish only if all pass. |
| 209 | 4. On any failure → automatic rollback to single-pass. |
| 210 | |
| 211 | ### Promotion Checklist (for operators) |
| 212 | |
| 213 | Before setting `MAPREDUCE_PUBLISH_ELIGIBLE=true`: |
| 214 | |
| 215 | - [ ] At least 3 consecutive comparison runs passed all criteria. |
| 216 | - [ ] No comparison run is older than 28 days. |
| 217 | - [ ] Rollback path tested with failing mapper injection. |
| 218 | - [ ] Rollback path tested with below-threshold quality score. |
| 219 | - [ ] Team sign-offs obtained (Leela, Farnsworth, Fry, Bender). |
| 220 | - [ ] Human reviewer approved the promotion PR. |
| 221 | - [ ] `docs/model-routing-policy.md` alignment verified for mapper/reducer |
| 222 | model assignments. |
| 223 | |
| 224 | --- |
| 225 | |
| 226 | ## Cost & Model Routing Alignment |
| 227 | |
| 228 | Map/reduce stages MUST follow `docs/model-routing-policy.md`. The table below |
| 229 | specifies the model routing for each pipeline stage: |
| 230 | |
| 231 | | Stage | Role | Model (per routing policy) | Rationale | |
| 232 | |-------|------|---------------------------|-----------| |
| 233 | | Mapper (×4) | Signal extraction | `claude-haiku-4.5` | Mechanical extraction from structured data; cost-first | |
| 234 | | Reducer / Editorial Planner | Claim selection + dedup | `claude-sonnet-4.6` | Judgment required for dedup and section assignment | |
| 235 | | Critic | QA gate validation | `claude-sonnet-4.6` | Must catch quality issues; cross-family if reviewing Sonnet output | |
| 236 | | Final Writer | Article generation | `claude-sonnet-4.6` | Produces code-adjacent editorial content; quality matters | |
| 237 | |
| 238 | ### Cost Budget |
| 239 | |
| 240 | - **Per-run budget:** Mappers (4 × Haiku) + Reducer (1 × Sonnet) + Critic |
| 241 | (1 × Sonnet) + Writer (1 × Sonnet) ≈ target $0.15–$0.40 per weekly run. |
| 242 | - **Budget enforcement:** `scripts/model_pricing.py` tracks per-stage costs; |
| 243 | if total exceeds $0.50 the run is flagged for review but not auto-aborted. |
| 244 | - **Cost comparison:** Each comparison report includes cost delta vs. |
| 245 | single-pass (which uses 1 × Sonnet call). |
| 246 | |
| 247 | ### Cross-Family Review for Promotion |
| 248 | |
| 249 | Per the model-routing-policy cross-family review rules: |
| 250 | - Critic stage SHOULD use a different model family than the Final Writer |
| 251 | when reviewing the generated article for promotion decisions. |
| 252 | - During the 3-run comparison period, at least one run SHOULD use GPT-family |
| 253 | for the Critic while keeping Sonnet for the Writer (or vice versa). |
| 254 | |
| 255 | --- |
| 256 | |
| 257 | ## Lifecycle States |
| 258 | |
| 259 | ``` |
| 260 | ┌─────────────┐ 3+ passing ┌──────────────┐ operator ┌────────────┐ |
| 261 | │ DRY-RUN │ ──── comparison ──→ │ ELIGIBLE │ ── opt-in ──→ │ PROMOTED │ |
| 262 | │ (default) │ runs │ (pending) │ │ (active) │ |
| 263 | └─────────────┘ └──────────────┘ └────────────┘ |
| 264 | ↑ ↑ │ |
| 265 | │ rollback / criteria │ gate failure / │ |
| 266 | └──────────── not met ──────────────┘ quality drop │ |
| 267 | ↓ │ |
| 268 | ┌────────────┐ │ |
| 269 | │ ROLLBACK │ ←──────┘ |
| 270 | └────────────┘ |
| 271 | ``` |
| 272 | |
| 273 | - **DRY-RUN:** Current state. Candidates generated, never published. |
| 274 | - **ELIGIBLE:** Comparison criteria met. Awaiting operator/team approval. |
| 275 | - **PROMOTED:** Map/reduce output is publish-eligible. Single-pass serves as |
| 276 | fallback on failure. |
| 277 | - **ROLLBACK:** Automatic reversion to single-pass. Re-enters DRY-RUN state |
| 278 | until criteria are re-established. |