Code Commits Issues Pulls Sessions CI/CD

main

md 12.1 KB

View source Raw

Map/Reduce Analysis Promotion Path

Issue: #331 Status: Candidate-only (promotion blocked by default) Effective: 2026-06-14

Overview

This document defines how map/reduce analysis can graduate from candidate-only dry runs to publish eligibility. Promotion requires passing a multi-week comparison framework, meeting explicit quality thresholds, and receiving operator approval. Until all criteria are met, map/reduce output remains non-publishing.

Comparison Framework

Before any promotion decision, at least 3 representative weekly runs must produce side-by-side QA reports comparing map/reduce candidate output against the current single-pass analyzer.

Metrics Compared

Metric	Source	Threshold
Evidence coverage	Coverage ledger `repo_count_mapped / repo_count_input`	Track weekly delta vs. single-pass; every promotion-window run must still stay ≥ 0.85 absolute coverage
Citation integrity	`citation_bindings` vs. rendered markdown links	Zero orphaned citations
Contradiction handling	`contradictions` sidecar count + resolution audit	Zero unresolved contradictions in the promotion window
Claim rejection	`rejected_claims` with reason audit	Every rejected claim must carry an allowed audit reason; no valid claims incorrectly rejected
Quality score	`analysis_gate.validate_analysis` word count + structure	Track weekly delta vs. single-pass; every promotion-window run must still stay ≥ 60 with ≥ 65 average
Gate pass/fail	`analysis_gate.validate_publish_quality`	Must pass all non-provenance gates

Comparison Run Protocol

Input: Same data/raw/YYYY-WNN.json for both paths.
Single-pass baseline: The existing scripts/analyze.py output at data/analyzed/YYYY-WNN-summary.md.
Map/reduce candidate: Output of scripts/map_reduce_dry_run.py at data/map-reduce-candidates/YYYY-WNN/.
Comparison report: Generated by scripts/map_reduce_comparison.py at data/map-reduce-candidates/YYYY-WNN/comparison-report.json.
Minimum runs: 3 consecutive weeks must pass before promotion is eligible.
Staleness: If any comparison run is older than 28 days at promotion time, it must be re-run against fresh single-pass output.

Comparison Report Schema

{
  "schema_version": "comparison_report_v1",
  "week": "2026-W24",
  "run_datetime": "2026-06-14T07:30:00Z",
  "single_pass": {
    "artifact_path": "data/analyzed/2026-W24-summary.md",
    "sha256": "...",
    "quality_score": 72,
    "gate_passed": true,
    "evidence_coverage": 0.92,
    "citation_count": 14,
    "word_count": 1850
  },
  "map_reduce": {
    "artifact_path": "data/map-reduce-candidates/2026-W24/2026-W24-map-reduce-candidate.md",
    "sha256": "...",
    "quality_score": 68,
    "gate_passed": true,
    "evidence_coverage": 0.88,
    "citation_count": 12,
    "word_count": 1720,
    "mapper_errors": {},
    "claims_rejected": 5,
    "unresolved_contradictions": 0,
    "orphaned_citations": 0,
    "invalid_rejected_claims": 0
  },
  "deltas": {
    "quality_score": -4,
    "evidence_coverage": -0.04,
    "citation_count": -2,
    "word_count": -130,
    "gate_regression": false,
    "orphaned_citations": 0,
    "unresolved_contradictions": 0,
    "invalid_rejected_claims": 0
  },
  "verdict": "pass",
  "blockers": []
}

Promotion Criteria

All of the following MUST be true before map/reduce output becomes publish-eligible:

Quality Gates

Gate parity: Map/reduce candidate passes validate_analysis and validate_publish_quality (excluding provenance gate) for all 3+ comparison runs.
Quality score floor: Map/reduce quality_score deltas are recorded against the single-pass baseline for every run, and the candidate must still stay ≥ 60 for every run AND average ≥ 65 across the promotion window.
Evidence coverage: evidence_coverage deltas are recorded against the single-pass baseline for every run, and the candidate must still stay ≥ 0.85 absolute coverage for every run (i.e., at least 85% of input repos appear in mapper output).
Citation integrity: Zero orphaned citations — every [repo](url) in rendered markdown must trace back to a citation_bindings entry in the editorial plan.
Contradiction handling: Promotion-window runs must have zero unresolved contradictions preserved in the reducer sidecars.
Claim rejection audit: Every rejected claim must carry an allowed audit reason, and no valid claim may be incorrectly rejected.
No gate regression: If single-pass passes all gates, map/reduce must also pass all gates. A map/reduce gate failure when single-pass succeeds is a blocking regression.

Editorial Requirements

Section completeness: All 5 required sections (## This Week's Trends, ## Where Industry Meets Code, ## Signal & Noise, ## Blind Spots, ## The Week Ahead) meet minimum word counts per analysis-spec.md.
Contradiction transparency: Any contradiction preserved for audit must be surfaced in the QA sidecar, not silently dropped.
Claim provenance: Every selected claim traces to at least one mapper finding with explicit evidence_refs.

Approval Requirements

Operator opt-in: The MAPREDUCE_PUBLISH_ELIGIBLE environment variable or workflow input must be explicitly set to true. Default is false.
Human sign-off: At least one human reviewer must approve the promotion PR that sets publish_eligible: true in the workflow configuration.
Team sign-off: Leela (scope/risk), Farnsworth (editorial quality), Fry (gates), and Bender (artifact determinism) must each approve in the promotion PR.

Rollback Rules

Automatic Rollback Triggers

Map/reduce output is automatically replaced by single-pass output when:

Gate failure: Map/reduce candidate fails any non-provenance gate that the single-pass baseline passes.
Quality regression: quality_score drops below 55 (hard floor).
Coverage collapse: evidence_coverage < 0.70.
Mapper failure: Any mapper returns status: "failed" and the reducer cannot recover.
Timeout: Map/reduce pipeline exceeds the configured time budget (default: 5 minutes for full pipeline).

Rollback Behavior

The workflow publishes the single-pass output as if map/reduce never ran.
The failed map/reduce candidate is preserved in data/map-reduce-candidates/YYYY-WNN/ for post-mortem analysis.
The QA report records "rollback": true with the trigger reason.
An alert is emitted to the orchestration log so the team can investigate.
Rollback does NOT require human intervention — it is automatic and safe.

Manual Rollback

An operator can force rollback at any time by:

Setting MAPREDUCE_PUBLISH_ELIGIBLE=false in the workflow environment.
Re-running the analysis workflow — single-pass takes over immediately.
No data loss: all map/reduce candidates remain archived.

Rollback Testing

Before promotion, the rollback path MUST be tested:

Inject a deliberately failing mapper (e.g., truncated input) and verify that single-pass output publishes without interruption.
Inject a quality_score below 55 and verify automatic rollback triggers.
Verify that data/map-reduce-candidates/ preserves the failed artifact.

Operator Controls

Environment Variables

Variable	Default	Effect
`MAPREDUCE_PUBLISH_ELIGIBLE`	`false`	Must be `true` for map/reduce to publish
`MAPREDUCE_FORCE_ROLLBACK`	`false`	Force immediate rollback to single-pass
`MAPREDUCE_DRY_RUN_ONLY`	`true`	When `true`, produce candidates but never publish
`MAPREDUCE_COMPARISON_MODE`	`true`	Generate comparison reports alongside candidates
`MAPREDUCE_TIME_BUDGET_SECONDS`	`300`	Max pipeline runtime before timeout rollback
`MAPREDUCE_MIN_QUALITY_SCORE`	`60`	Per-run quality floor for promotion
`MAPREDUCE_MIN_COVERAGE`	`0.85`	Per-run evidence coverage floor

Workflow Integration

The analysis workflow (analyze.yml) checks controls in this order:

If MAPREDUCE_FORCE_ROLLBACK=true → skip map/reduce entirely.
If MAPREDUCE_DRY_RUN_ONLY=true → run map/reduce but only save candidate.
If MAPREDUCE_PUBLISH_ELIGIBLE=true → run comparison, check all promotion criteria, publish only if all pass.
On any failure → automatic rollback to single-pass.

Promotion Checklist (for operators)

Before setting MAPREDUCE_PUBLISH_ELIGIBLE=true:

At least 3 consecutive comparison runs passed all criteria.
No comparison run is older than 28 days.
Rollback path tested with failing mapper injection.
Rollback path tested with below-threshold quality score.
Team sign-offs obtained (Leela, Farnsworth, Fry, Bender).
Human reviewer approved the promotion PR.
docs/model-routing-policy.md alignment verified for mapper/reducer model assignments.

Cost & Model Routing Alignment

Map/reduce stages MUST follow docs/model-routing-policy.md. The table below specifies the model routing for each pipeline stage:

Stage	Role	Model (per routing policy)	Rationale
Mapper (×4)	Signal extraction	`claude-haiku-4.5`	Mechanical extraction from structured data; cost-first
Reducer / Editorial Planner	Claim selection + dedup	`claude-sonnet-4.6`	Judgment required for dedup and section assignment
Critic	QA gate validation	`claude-sonnet-4.6`	Must catch quality issues; cross-family if reviewing Sonnet output
Final Writer	Article generation	`claude-sonnet-4.6`	Produces code-adjacent editorial content; quality matters

Cost Budget

Per-run budget: Mappers (4 × Haiku) + Reducer (1 × Sonnet) + Critic (1 × Sonnet) + Writer (1 × Sonnet) ≈ target $0.15–$0.40 per weekly run.
Budget enforcement: scripts/model_pricing.py tracks per-stage costs; if total exceeds $0.50 the run is flagged for review but not auto-aborted.
Cost comparison: Each comparison report includes cost delta vs. single-pass (which uses 1 × Sonnet call).

Cross-Family Review for Promotion

Per the model-routing-policy cross-family review rules: - Critic stage SHOULD use a different model family than the Final Writer when reviewing the generated article for promotion decisions. - During the 3-run comparison period, at least one run SHOULD use GPT-family for the Critic while keeping Sonnet for the Writer (or vice versa).

Lifecycle States

┌─────────────┐     3+ passing     ┌──────────────┐    operator    ┌────────────┐
│  DRY-RUN    │ ──── comparison ──→ │  ELIGIBLE    │ ── opt-in ──→ │  PROMOTED  │
│  (default)  │      runs           │  (pending)   │               │  (active)  │
└─────────────┘                     └──────────────┘               └────────────┘
       ↑                                   ↑                             │
       │         rollback / criteria       │      gate failure /         │
       └──────────── not met ──────────────┘      quality drop          │
                                                        ↓               │
                                                  ┌────────────┐        │
                                                  │  ROLLBACK  │ ←──────┘
                                                  └────────────┘

DRY-RUN: Current state. Candidates generated, never published.
ELIGIBLE: Comparison criteria met. Awaiting operator/team approval.
PROMOTED: Map/reduce output is publish-eligible. Single-pass serves as fallback on failure.
ROLLBACK: Automatic reversion to single-pass. Re-enters DRY-RUN state until criteria are re-established.

#Map/Reduce Analysis Promotion Path

#Overview

#Comparison Framework

#Metrics Compared

#Comparison Run Protocol

#Comparison Report Schema

#Promotion Criteria

#Quality Gates

#Editorial Requirements

#Approval Requirements

#Rollback Rules

#Automatic Rollback Triggers

#Rollback Behavior

#Manual Rollback

#Rollback Testing

#Operator Controls

#Environment Variables

#Workflow Integration

#Promotion Checklist (for operators)

#Cost & Model Routing Alignment

#Cost Budget

#Cross-Family Review for Promotion

#Lifecycle States