| 1 | # Observability Metrics Schema |
| 2 | |
| 3 | `scripts/observability_metrics.py` defines the durable ledger schema for crawl and analysis observability artifacts written to `data/metrics/observability/`. |
| 4 | |
| 5 | ## Schema version |
| 6 | |
| 7 | - Current version: `observability_v1` |
| 8 | - Writers must set `schema_version` exactly. |
| 9 | - Downstream checks should fail if `validate_ledger()` reports any missing required field paths. |
| 10 | |
| 11 | ## Artifact layout |
| 12 | |
| 13 | Current pipeline writers emit: |
| 14 | |
| 15 | - `data/metrics/observability/{week}-github-crawl.json` |
| 16 | - `data/metrics/observability/{week}-external-news-crawl.json` |
| 17 | - `data/metrics/observability/{week}-map-reduce.json` |
| 18 | |
| 19 | These are runtime artifacts and are gitignored. |
| 20 | |
| 21 | ## Top-level ledger fields |
| 22 | |
| 23 | Required: |
| 24 | |
| 25 | - `schema_version` |
| 26 | - `run_id` |
| 27 | - `week` |
| 28 | - `timestamp` |
| 29 | - `crawl_metrics` |
| 30 | - `environment` |
| 31 | |
| 32 | Optional: |
| 33 | |
| 34 | - `analysis_metrics` |
| 35 | |
| 36 | ## `crawl_metrics[]` |
| 37 | |
| 38 | Required fields: |
| 39 | |
| 40 | - `source_type` (`github` or `external-news`) |
| 41 | - `duration_seconds` |
| 42 | - `api_calls` |
| 43 | - `cache_hits` |
| 44 | - `cache_misses` |
| 45 | - `stale_cache_hits` |
| 46 | - `rate_limit_events` |
| 47 | - `secondary_rate_limit_hit` |
| 48 | |
| 49 | Optional fields: |
| 50 | |
| 51 | - `duration_p95_seconds` |
| 52 | - `duration_sample_count` |
| 53 | |
| 54 | Notes: |
| 55 | |
| 56 | - GitHub crawl currently records overall run duration as the comparable sampled duration. |
| 57 | - External-news crawl records p95 from per-source fetch durations when available. |
| 58 | |
| 59 | ## `analysis_metrics` |
| 60 | |
| 61 | Required fields when present: |
| 62 | |
| 63 | - `duration_seconds` |
| 64 | - `token_ledger` |
| 65 | - `map_stages` |
| 66 | |
| 67 | Optional: |
| 68 | |
| 69 | - `reduce_stage` |
| 70 | |
| 71 | ### `token_ledger` |
| 72 | |
| 73 | Required: |
| 74 | |
| 75 | - `input_tokens` |
| 76 | - `output_tokens` |
| 77 | - `total_tokens` |
| 78 | |
| 79 | Common additional field: |
| 80 | |
| 81 | - `cost_usd` |
| 82 | |
| 83 | ### `map_stages[]` and `reduce_stage` |
| 84 | |
| 85 | Required: |
| 86 | |
| 87 | - `stage` |
| 88 | - `duration_seconds` |
| 89 | - `input_tokens` |
| 90 | - `output_tokens` |
| 91 | - `cost_usd` |
| 92 | - `status` (`pass` or `fail`) |
| 93 | - `gate_failure_reasons` |
| 94 | |
| 95 | ## Validation |
| 96 | |
| 97 | Use `validate_ledger()` from `scripts.observability_metrics`: |
| 98 | |
| 99 | ```python |
| 100 | from scripts.observability_metrics import validate_ledger |
| 101 | |
| 102 | errors = validate_ledger(payload) |
| 103 | if errors: |
| 104 | raise SystemExit(f"Missing required observability fields: {errors}") |
| 105 | ``` |
| 106 | |
| 107 | `emit_ledger()` already validates before writing and raises `ValueError` on schema gaps. |
| 108 | |
| 109 | ## Example |
| 110 | |
| 111 | Representative end-to-end sample: |
| 112 | |
| 113 | - `tests/fixtures/observability/2026-W21-full-run.json` |
| 114 | |
| 115 | Minimal shape: |
| 116 | |
| 117 | ```json |
| 118 | { |
| 119 | "schema_version": "observability_v1", |
| 120 | "run_id": "12345", |
| 121 | "week": "2026-W21", |
| 122 | "timestamp": "2026-05-20T12:00:00Z", |
| 123 | "crawl_metrics": [ |
| 124 | { |
| 125 | "source_type": "github", |
| 126 | "duration_seconds": 12.4, |
| 127 | "api_calls": 27, |
| 128 | "cache_hits": 14, |
| 129 | "cache_misses": 27, |
| 130 | "stale_cache_hits": 1, |
| 131 | "rate_limit_events": 2, |
| 132 | "secondary_rate_limit_hit": false |
| 133 | } |
| 134 | ], |
| 135 | "analysis_metrics": { |
| 136 | "duration_seconds": 3.2, |
| 137 | "token_ledger": { |
| 138 | "input_tokens": 1234, |
| 139 | "output_tokens": 456, |
| 140 | "total_tokens": 1690, |
| 141 | "cost_usd": 0.0 |
| 142 | }, |
| 143 | "map_stages": [], |
| 144 | "reduce_stage": null |
| 145 | }, |
| 146 | "environment": { |
| 147 | "pipeline": "map-reduce-dry-run" |
| 148 | } |
| 149 | } |
| 150 | ``` |
| 151 | |
| 152 | ## Downstream check guidance |
| 153 | |
| 154 | - Treat `schema_version` as a compatibility gate. |
| 155 | - Call `validate_ledger()` and fail on any returned field path. |
| 156 | - Prefer exact field-path assertions over permissive defaults so missing metrics break CI early. |
| 157 | - For experiment reports tied to issue #356, link the representative fixture above plus the emitted runtime artifacts from the relevant workflow run. |