| 1 | # Crawl Matrix Trigger Thresholds |
| 2 | |
| 3 | **Status:** Active |
| 4 | **Related issues:** #333, #435, #436 |
| 5 | **PRD:** [docs/processed/PRD-matrix-crawl-map-reduce-analysis.md](processed/PRD-matrix-crawl-map-reduce-analysis.md) |
| 6 | |
| 7 | --- |
| 8 | |
| 9 | ## Overview |
| 10 | |
| 11 | This document specifies the evidence-based trigger thresholds that must be met before enabling RSS matrix fan-out or GitHub shard experiments in the default pipeline path. These thresholds ensure that matrix complexity is only introduced when measurable evidence justifies it. |
| 12 | |
| 13 | **Baseline requirement:** At least 5 representative production runs must be observed before any trigger evaluation is valid (or an explicitly documented shorter window with rationale). |
| 14 | |
| 15 | --- |
| 16 | |
| 17 | ## RSS Matrix Triggers |
| 18 | |
| 19 | The RSS/news collection path remains in-process (monolithic) unless ALL of the following conditions are met: |
| 20 | |
| 21 | | Trigger | Threshold | Measurement | |
| 22 | |---------|-----------|-------------| |
| 23 | | **p95 runtime** | > 60 seconds | Measured across ≥5 runs from observability ledger (`source_type: rss`) | |
| 24 | | **Source count** | > 10 configured sources | Count of active entries in source config | |
| 25 | | **Source-specific isolation** | Required | Any source needing independent credentials, rate limits, or network isolation | |
| 26 | | **Flaky-source retry** | Required | Any source requiring retry logic that would delay other sources | |
| 27 | |
| 28 | ### Decision logic |
| 29 | |
| 30 | ``` |
| 31 | IF baseline_runs >= 5 |
| 32 | AND (rss_p95 > 60s OR source_count > 10 OR isolation_required OR flaky_retry_needed) |
| 33 | THEN propose RSS matrix enablement (requires explicit approval) |
| 34 | ELSE RSS matrix remains disabled |
| 35 | ``` |
| 36 | |
| 37 | ### Current baseline status |
| 38 | |
| 39 | Run `python -m scripts.baseline_telemetry check` to see current values. |
| 40 | |
| 41 | --- |
| 42 | |
| 43 | ## GitHub Shard/Matrix Triggers |
| 44 | |
| 45 | GitHub crawl sharding remains **no-publish experimental** unless ALL of the following are met: |
| 46 | |
| 47 | | Trigger | Threshold | Measurement | |
| 48 | |---------|-----------|-------------| |
| 49 | | **Wall-clock speedup** | ≥ 25% reduction vs. monolithic baseline | Compare shard experiment p50 to monolithic baseline p50 | |
| 50 | | **API-call growth** | ≤ 10% increase | Total API calls (shard sum) vs. monolithic baseline | |
| 51 | | **Secondary rate-limit regression** | Zero events | No secondary rate-limit events in shard experiment | |
| 52 | | **Cache coherence** | Maintained | Cache hit ratio within 5% of monolithic baseline | |
| 53 | | **Output parity** | Byte-stable | Canonical merged output matches monolithic output (excluding timestamps) | |
| 54 | |
| 55 | ### Decision logic |
| 56 | |
| 57 | ``` |
| 58 | IF shard_experiment_runs >= 3 |
| 59 | AND speedup_pct >= 25 |
| 60 | AND api_call_growth_pct <= 10 |
| 61 | AND secondary_rate_limit_events == 0 |
| 62 | AND cache_hit_ratio_delta <= 5% |
| 63 | AND byte_stable_output == true |
| 64 | THEN propose GitHub shard default-on (requires explicit approval + ADR update) |
| 65 | ELSE GitHub shard remains no-publish experiment only |
| 66 | ``` |
| 67 | |
| 68 | ### Experiment execution |
| 69 | |
| 70 | See `scripts/crawl_shard_experiment.py` (issue #435) for the no-publish experiment runner. Results are written to `data/experiments/shard-435/`. |
| 71 | |
| 72 | --- |
| 73 | |
| 74 | ## Trigger Evaluation Process |
| 75 | |
| 76 | 1. **Collect baseline:** Pipeline operator runs ≥5 production cycles with observability metrics enabled. |
| 77 | 2. **Generate report:** `python -m scripts.baseline_telemetry report --output data/metrics/baseline-report.json` |
| 78 | 3. **Check triggers:** `python -m scripts.baseline_telemetry check --min-runs 5` |
| 79 | 4. **If triggered:** Open a proposal issue referencing the baseline report. Requires team review and ADR update before enabling. |
| 80 | 5. **If not triggered:** No action. Re-evaluate after next 5-run window. |
| 81 | |
| 82 | --- |
| 83 | |
| 84 | ## Existing Behavior Preservation |
| 85 | |
| 86 | These triggers are gates ONLY. Until a trigger fires and is explicitly approved: |
| 87 | |
| 88 | - ✅ `scripts/crawl.py` — monolithic GitHub crawl unchanged |
| 89 | - ✅ `scripts/techcrunch_crawler.py` — in-process RSS crawl unchanged |
| 90 | - ✅ `scripts/correlate.py` — correlation analysis unchanged |
| 91 | - ✅ `scripts/render_press_context.py` — press context rendering unchanged |
| 92 | - ✅ `scripts/generate_content.py` — analysis unchanged |
| 93 | - ✅ `scripts/publish_manifest.py` — publish manifest unchanged |
| 94 | - ✅ Canonical artifact paths (`data/raw/{week}.json`, `data/raw/{topic}/{week}-external-news.json`) unchanged |
| 95 | |
| 96 | --- |
| 97 | |
| 98 | ## References |
| 99 | |
| 100 | - [Fan-in contracts](matrix-crawl-fan-in-contracts.md) |
| 101 | - [ADR: Matrix Crawl Fan-In](decisions/adr-matrix-crawl-fan-in.md) |
| 102 | - [Operator Runbook](matrix-crawl-runbook.md) |
| 103 | - Baseline telemetry: `scripts/baseline_telemetry.py` |
| 104 | - Run context schema: `scripts/run_context.py` |
| 105 | - Fan-in validator: `scripts/fan_in_validator.py` |