| 1 | # metrix |
| 2 | |
| 3 | `metrix` is the metrics storage and read API used by go.d `ModuleV2` collectors and runtime/internal components. |
| 4 | |
| 5 | **Audience**: `ModuleV2` collector authors and framework contributors. |
| 6 | |
| 7 | **See also**: [charttpl](/src/go/plugin/framework/charttpl/README.md) (template DSL), |
| 8 | [chartengine](/src/go/plugin/framework/chartengine/README.md) (compile + plan). |
| 9 | |
| 10 | ## Purpose |
| 11 | |
| 12 | | Consumer | Store type | Typical usage | |
| 13 | |----------------------------------|------------------|------------------------------------------------------------| |
| 14 | | Collector jobs (`ModuleV2`) | `CollectorStore` | Cycle-scoped writes, snapshot reads, chart planning input | |
| 15 | | Internal/runtime instrumentation | `RuntimeStore` | Stateful immediate-commit writes, runtime metrics planning | |
| 16 | |
| 17 | ## Core Concepts |
| 18 | |
| 19 | - **Immutable reads** — Readers observe immutable snapshots that are swapped atomically on commit. Multiple goroutines can read concurrently without locking. |
| 20 | - **Cycle-scoped collector writes** — `CollectorStore` writes are staged between `BeginCycle` and `CommitCycleSuccess`. Nothing is visible to readers until commit. |
| 21 | - **Stateful runtime writes** — `RuntimeStore` writes are committed immediately (no cycle API). Each write produces a new overlay snapshot. |
| 22 | - **Label canonicalization** — Label maps (`map[string]string`) are sorted and encoded into a canonical key that uniquely identifies a series (metric name + labels). |
| 23 | - **Typed + flattened views** — Reader supports canonical typed families (Histogram, Summary, StateSet, MeasureSet) and a flattened scalar view where complex types are projected into individual scalar series. |
| 24 | |
| 25 | ## Key Definitions |
| 26 | |
| 27 | - **Freshness** controls which series appear in non-raw reads. |
| 28 | `FreshnessCycle` = series must be observed in the latest successful cycle to be visible. |
| 29 | `FreshnessCommitted` = series is visible as long as it's committed, even if not re-observed. |
| 30 | - **Window** controls how stateful histogram/summary instruments accumulate observations. |
| 31 | `WindowCumulative` = observations accumulate across cycles. |
| 32 | `WindowCycle` = observations reset each cycle. |
| 33 | |
| 34 | ## Stores and Interfaces |
| 35 | |
| 36 | | Interface | Key methods | Notes | |
| 37 | |---------------------|---------------------------------------------------------|-----------------------------------------| |
| 38 | | `CollectorStore` | `Read(...)`, `Write()` | Default collector-facing store | |
| 39 | | `RuntimeStore` | `Read(...)`, `Write()` | Stateful-only writes | |
| 40 | | `CycleManagedStore` | `CycleController()` | Runtime/orchestrator-only cycle control | |
| 41 | | `Reader` | `Value/Delta/Histogram/Summary/StateSet/MeasureSet/...` | Immutable snapshot read API | |
| 42 | |
| 43 | ## Write Model |
| 44 | |
| 45 | ### Collector store |
| 46 | |
| 47 | | Phase | Action | |
| 48 | |---------|-----------------------------------------------------------------------------------------------| |
| 49 | | Begin | Open staged frame (`BeginCycle`) | |
| 50 | | Collect | Collector writes metrics through `Write().SnapshotMeter(...)` or `Write().StatefulMeter(...)` | |
| 51 | | Success | `CommitCycleSuccess` publishes new snapshot and advances success sequence | |
| 52 | | Failure | `AbortCycle` drops staged writes | |
| 53 | |
| 54 | `ModuleV2` collectors should write metrics only; cycle control is handled by job runtime. |
| 55 | |
| 56 | ### Runtime store |
| 57 | |
| 58 | - **No cycle API** — writes commit immediately. |
| 59 | - **Stateful only** — snapshot-mode instrument registration returns an error. |
| 60 | |
| 61 | > [!CAUTION] |
| 62 | > Calling snapshot-mode record methods (`ObserveTotal`, `ObservePoint`) on a `RuntimeStore` **panics**. |
| 63 | |
| 64 | - **Fixed freshness** — runtime store enforces `FreshnessCommitted` semantics; other freshness policies are rejected. |
| 65 | |
| 66 | ## Instrument Modes and Defaults |
| 67 | |
| 68 | | Mode | Typical meter | Freshness default | Window default | |
| 69 | |----------|----------------------|----------------------|--------------------| |
| 70 | | Snapshot | `SnapshotMeter(...)` | `FreshnessCycle` | `WindowCumulative` | |
| 71 | | Stateful | `StatefulMeter(...)` | `FreshnessCommitted` | `WindowCumulative` | |
| 72 | |
| 73 | ## Instrument Options |
| 74 | |
| 75 | | Option | Scope | |
| 76 | |-----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------| |
| 77 | | `WithFreshness(...)` | Freshness policy override (subject to mode constraints) | |
| 78 | | `WithWindow(...)` | Stateful histogram/summary window mode | |
| 79 | | `WithHistogramBounds(...)` | Histogram bucket boundaries | |
| 80 | | `WithSummaryQuantiles(...)` | Summary quantile output (required for quantile series in flattened view) | |
| 81 | | `WithSummaryReservoirSize(...)` | Stateful summary estimator size | |
| 82 | | `WithStateSetStates(...)` | StateSet allowed states | |
| 83 | | `WithStateSetMode(...)` | `ModeBitSet` (multiple simultaneous active states) or `ModeEnum` (exactly one active state) | |
| 84 | | `WithMeasureSetFields(...)` | MeasureSet fixed ordered field schema (required for MeasureSet instruments) | |
| 85 | | `WithDescription(...)`, `WithChartFamily(...)`, `WithUnit(...)`, `WithFloat(...)` | Metric metadata hints for downstream consumers (e.g., autogen chart identity + float SET mode) | |
| 86 | |
| 87 | ## MeasureSet |
| 88 | |
| 89 | - **Structured numeric family** — `MeasureSet` stores one logical metric family with a fixed ordered list of named numeric fields. |
| 90 | - **Family-level semantics** — one `MeasureSet` family is either gauge-like or counter-like; semantics are never mixed per field. |
| 91 | - **Family-level metadata** — `Description`, `ChartFamily`, and `Unit` apply to the whole family. |
| 92 | - **Field-level schema** — `MeasureFieldSpec` declares per-field `Name` and `Float`. |
| 93 | - **Chartengine integration** — chart autogen treats `MeasureSet` as a structured family, similar to `StateSet`; flatten remains the generic reader/tooling path. |
| 94 | |
| 95 | ### Writers |
| 96 | |
| 97 | | Mode | Gauge-like family | Counter-like family | |
| 98 | |----------|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------| |
| 99 | | Snapshot | `MeasureSetGauge(...).ObservePoint(...)` or preferred `ObserveFields(...)` | `MeasureSetCounter(...).ObserveTotalPoint(...)` or preferred `ObserveTotalFields(...)` | |
| 100 | | Stateful | `MeasureSetGauge(...).SetPoint(...)`, `AddPoint(...)`, `SetFields(...)`, `AddFields(...)`, `SetField(...)`, `AddField(...)` | `MeasureSetCounter(...).AddPoint(...)`, `AddFields(...)`, `AddField(...)` | |
| 101 | |
| 102 | ### Phase-1 write contract |
| 103 | |
| 104 | - **Preferred collector-facing API** — use named write helpers instead of raw positional `MeasureSetPoint` values whenever practical. |
| 105 | - **Snapshot handles** support: |
| 106 | - full-family positional writes (`ObservePoint(...)`, `ObserveTotalPoint(...)`) |
| 107 | - full-family named writes (`ObserveFields(...)`, `ObserveTotalFields(...)`) |
| 108 | - **Stateful handles** support: |
| 109 | - full-family positional writes |
| 110 | - full-family named writes |
| 111 | - singular field writes (`SetField(...)`, `AddField(...)`) |
| 112 | - **Snapshot singular field writes do not exist in phase 1.** |
| 113 | - `MeasureSet` still models one sampled family point per collect cycle in snapshot mode. |
| 114 | - Partial snapshot field visibility/completeness semantics are intentionally deferred. |
| 115 | - **Named full-family writes require the exact declared field set.** |
| 116 | - Missing fields panic. |
| 117 | - Unknown extra fields panic. |
| 118 | - **Stateful singular field writes update only the addressed field.** |
| 119 | - Gauge-like `SetField(...)` overwrites one field on top of the committed/staged family. |
| 120 | - Gauge-like `AddField(...)` and counter-like `AddField(...)` apply a delta to one field. |
| 121 | |
| 122 | ### Schema example |
| 123 | |
| 124 | ```go |
| 125 | store := metrix.NewCollectorStore() |
| 126 | meter := store.Write().SnapshotMeter("svc") |
| 127 | latency := meter.MeasureSetGauge( |
| 128 | "latency", |
| 129 | metrix.WithMeasureSetFields( |
| 130 | metrix.MeasureFieldSpec{Name: "value"}, |
| 131 | metrix.MeasureFieldSpec{Name: "ratio", Float: true}, |
| 132 | ), |
| 133 | metrix.WithUnit("seconds"), |
| 134 | ) |
| 135 | latency.ObserveFields(map[string]metrix.SampleValue{ |
| 136 | "value": 1.5, |
| 137 | "ratio": 0.5, |
| 138 | }) |
| 139 | ``` |
| 140 | |
| 141 | Re-registering the same metric name with a different `MeasureSet` schema is rejected like other structured families. |
| 142 | |
| 143 | ### Stateful singular-write example |
| 144 | |
| 145 | ```go |
| 146 | store := metrix.NewRuntimeStore() |
| 147 | meter := store.Write().StatefulMeter("svc") |
| 148 | usage := meter.MeasureSetGauge( |
| 149 | "usage", |
| 150 | metrix.WithMeasureSetFields( |
| 151 | metrix.MeasureFieldSpec{Name: "value"}, |
| 152 | metrix.MeasureFieldSpec{Name: "limit"}, |
| 153 | ), |
| 154 | ) |
| 155 | |
| 156 | usage.SetFields(map[string]metrix.SampleValue{ |
| 157 | "value": 10, |
| 158 | "limit": 20, |
| 159 | }) |
| 160 | usage.SetField("value", 15) |
| 161 | usage.AddField("limit", 3) |
| 162 | ``` |
| 163 | |
| 164 | This yields a committed `MeasureSet` point equivalent to: |
| 165 | |
| 166 | ```go |
| 167 | metrix.MeasureSetPoint{Values: []metrix.SampleValue{15, 23}} |
| 168 | ``` |
| 169 | |
| 170 | ## Read Modes |
| 171 | |
| 172 | `Read(...)` accepts option functions that control two independent axes: |
| 173 | |
| 174 | - **Raw** (`ReadRaw()`) — bypasses freshness filtering, returning all committed series regardless of when they were last observed. |
| 175 | - **Flatten** (`ReadFlatten()`) — projects complex types (Histogram, Summary, StateSet, MeasureSet) into individual scalar series. |
| 176 | |
| 177 | | Read options | Visibility | Shape | |
| 178 | |----------------------------------|----------------------|--------------------------| |
| 179 | | `Read()` | Freshness-filtered | Canonical typed families | |
| 180 | | `Read(ReadRaw())` | All committed series | Canonical typed families | |
| 181 | | `Read(ReadFlatten())` | Freshness-filtered | Flattened scalar view | |
| 182 | | `Read(ReadRaw(), ReadFlatten())` | All committed series | Flattened scalar view | |
| 183 | |
| 184 | ## Flattened View Mapping |
| 185 | |
| 186 | `Read(ReadFlatten())` projects non-scalar families into scalar series: |
| 187 | |
| 188 | | Source kind | Flattened outputs | |
| 189 | |-------------|------------------------------------------------------------------------------------------------------------------| |
| 190 | | Histogram | `<name>_bucket{le=...}`, `<name>_count`, `<name>_sum` | |
| 191 | | Summary | `<name>_count`, `<name>_sum` (always); `<name>{quantile=...}` (only when `WithSummaryQuantiles()` is configured) | |
| 192 | | StateSet | `<name>{<name>=state}` with scalar 0/1 values | |
| 193 | | MeasureSet | `<name>_<field>{measure_field=field}`; flattened kind follows family semantics (`Gauge` or `Counter`) | |
| 194 | |
| 195 | Flatten metadata is exposed via `SeriesMeta.Kind`, `SeriesMeta.SourceKind`, and `SeriesMeta.FlattenRole`. |
| 196 | |
| 197 | `MeasureSet` flattening keeps per-field metric names for `MetricMeta(name)` compatibility and also adds a synthetic `measure_field=<field>` label. This gives chartengine explicit field identity without widening the reader metadata API. |
| 198 | |
| 199 | ## Minimal Usage Snippets |
| 200 | |
| 201 | ### Collector write path |
| 202 | |
| 203 | ```go |
| 204 | store := metrix.NewCollectorStore() |
| 205 | meter := store.Write().SnapshotMeter("mysql") |
| 206 | qps := meter.Counter("queries_total") |
| 207 | qps.ObserveTotal(42) |
| 208 | ``` |
| 209 | |
| 210 | ### Read path for planning |
| 211 | |
| 212 | ```go |
| 213 | reader := store.Read(metrix.ReadRaw(), metrix.ReadFlatten()) |
| 214 | value, ok := reader.Value("mysql.queries_total", nil) |
| 215 | _ = value |
| 216 | _ = ok |
| 217 | ``` |
| 218 | |
| 219 | ### Direct MeasureSet read |
| 220 | |
| 221 | ```go |
| 222 | reader := store.Read() |
| 223 | point, ok := reader.MeasureSet("svc.latency", nil) |
| 224 | _ = point |
| 225 | _ = ok |
| 226 | ``` |
| 227 | |
| 228 | For a complete collector integration pattern (cycle management, error handling), |
| 229 | see [how-to-write-a-collector.md](/src/go/plugin/go.d/docs/how-to-write-a-collector.md). |
| 230 | |
| 231 | ## Contracts and Pitfalls |
| 232 | |
| 233 | - **Label sets** — `LabelSet` is store-owned; do not share between different stores. |
| 234 | - **Counter deltas** — `Delta()` requires contiguous sequence (N, N+1). |
| 235 | In `CollectorStore` this is per-cycle: missing one successful cycle breaks the delta. |
| 236 | In `RuntimeStore` this is per-series per-write: the sequence always increments on each write, so skipping a write cycle does not break deltas. |
| 237 | - **Snapshot freshness** — Snapshot-mode instruments cannot use `FreshnessCommitted`. |
| 238 | - **Runtime writes** — `RuntimeStore` rejects snapshot-mode instrument registration with an error. |
| 239 | Calling snapshot-mode record methods (`ObserveTotal`, `ObservePoint`) **panics**. |
| 240 | - **MeasureSet runtime writes** — `RuntimeStore` supports both gauge-like and counter-like `MeasureSet` families, but only through `StatefulMeter(...)`. |
| 241 | - **MeasureSet named writes** — `ObserveFields(...)`, `ObserveTotalFields(...)`, `SetFields(...)`, and `AddFields(...)` require the exact declared field set. Snapshot singular field writes are intentionally absent in phase 1. |
| 242 | - **Window/freshness coupling** — Stateful histogram/summary with `WindowCycle` requires (and silently forces) `FreshnessCycle`. Setting an explicit non-Cycle freshness with `WindowCycle` returns an error. |
| 243 | - **Schema stability** — Re-registering an existing metric name with different kind/mode/schema returns an error (or panics in strict runtime paths). |
| 244 | - **MeasureSet flatten naming** — Flattened `MeasureSet` series use per-field metric names like `<name>_<field>` and also carry a synthetic `measure_field=<field>` label. |
| 245 | - **MeasureSet counter semantics** — Stateful counter-like `MeasureSet` families reject negative `AddPoint(...)` deltas, just like scalar counters. |
| 246 | - **Summary NaN quantiles** — a summary point may carry NaN quantile *values* (e.g. an empty observation window); they are stored (only Inf is rejected) and render as a chart gap downstream (chartengine emits `SETEMPTY`). Count and Sum must still be finite. |
| 247 | - **Collector retention** — `CollectorStore` evicts series not seen for 10 successful cycles by default. |
| 248 | |
| 249 | ## Internal Architecture Notes |
| 250 | |
| 251 | | Area | Implementation pattern | |
| 252 | |------------------|------------------------------------------------------------------| |
| 253 | | Snapshot publish | Read snapshots are immutable and atomically swapped | |
| 254 | | Collector commit | Staged frame merges into new snapshot on successful cycle commit | |
| 255 | | Runtime commit | Overlay/compaction strategy with retention pruning | |
| 256 | | Iteration | Name-indexed deterministic iteration for reader traversal | |
| 257 | | Identity | Canonical metric+labels key with stable `SeriesIdentity` hash | |