| 1 | # Chart Template Format |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | A **chart template** defines _how a collector's metrics are organized into charts_ in the Netdata dashboard. |
| 6 | |
| 7 | > [!NOTE] |
| 8 | > Chart templates are declarative YAML files. You describe **what** to chart, and the engine handles creating, updating, and removing chart instances at runtime. |
| 9 | |
| 10 | It tells the chart engine: |
| 11 | |
| 12 | - which **metrics** to include |
| 13 | - how to **group** them into charts and families |
| 14 | - how to **name** and **scale** dimensions |
| 15 | - how to create **per-instance charts** (e.g., one chart per host, per disk, per user) |
| 16 | |
| 17 | Each collector has a single `charts.yaml` file that describes all its charts. |
| 18 | |
| 19 | > [!TIP] |
| 20 | > Groups can be nested to **any depth**. Family paths, context namespaces, and metric scopes compose automatically as you nest — no need to repeat prefixes. See [groups](#4-groups) for the full composition rules and examples. |
| 21 | |
| 22 | ### How Chart Templates Work |
| 23 | |
| 24 | When a collector runs, the chart engine: |
| 25 | |
| 26 | 1. Reads the collector's `charts.yaml` file. |
| 27 | 2. Compiles it into an immutable program (validates, resolves defaults, infers algorithms). |
| 28 | 3. On each collection cycle, matches incoming metrics against dimension selectors. |
| 29 | 4. Creates chart instances dynamically based on instance identity labels. |
| 30 | 5. Updates dimension values every cycle; removes stale instances based on lifecycle policy. |
| 31 | |
| 32 | **Template Lifecycle** |
| 33 | |
| 34 | ```text |
| 35 | charts.yaml |
| 36 | | |
| 37 | v |
| 38 | ┌─────────────────────┐ |
| 39 | │ Decode & Validate │ strict YAML parse + semantic checks |
| 40 | └──────────┬──────────┘ |
| 41 | v |
| 42 | ┌─────────────────────┐ |
| 43 | │ Compile (engine) │ selector parsing, algorithm inference, |
| 44 | │ │ context/family/ID composition |
| 45 | └──────────┬──────────┘ |
| 46 | v |
| 47 | ┌─────────────────────┐ |
| 48 | │ Runtime (per cycle)│ match series → create/update/remove |
| 49 | │ │ charts and dimensions |
| 50 | └─────────────────────┘ |
| 51 | ``` |
| 52 | |
| 53 | ### Example: Complete Chart Template |
| 54 | |
| 55 | The example below shows a single template that covers all common metric kinds: |
| 56 | gauge, counter, histogram, summary, and stateset. Inline comments explain each field. |
| 57 | |
| 58 | ```yaml |
| 59 | version: v1 # schema version (only "v1" supported) |
| 60 | context_namespace: myapp # prefix for all chart contexts → myapp.<group context>.<chart context> |
| 61 | |
| 62 | groups: |
| 63 | # ── Gauge: point-in-time values (algorithm: absolute) ────────────── |
| 64 | - family: Resources |
| 65 | metrics: # metrics visible to dimension selectors in this group |
| 66 | - memory_used_bytes |
| 67 | - memory_total_bytes |
| 68 | - cpu_usage_percent |
| 69 | charts: |
| 70 | - id: memory_usage |
| 71 | title: Memory Usage |
| 72 | context: memory_usage # final context: myapp.memory_usage |
| 73 | units: bytes |
| 74 | type: stacked # line (default), area, stacked, heatmap |
| 75 | algorithm: absolute # absolute = raw value; incremental = rate (value - prev) / interval |
| 76 | instances: |
| 77 | by_labels: [host] # one chart per unique "host" label value |
| 78 | dimensions: |
| 79 | - selector: memory_used_bytes |
| 80 | name: used # static dimension name |
| 81 | - selector: memory_total_bytes |
| 82 | name: total |
| 83 | options: |
| 84 | hidden: true # collected but not drawn (useful for % calculations) |
| 85 | |
| 86 | - id: cpu_usage |
| 87 | title: CPU Usage |
| 88 | context: cpu_usage |
| 89 | units: percentage |
| 90 | instances: |
| 91 | by_labels: [host] |
| 92 | dimensions: |
| 93 | - selector: cpu_usage_percent |
| 94 | name: used |
| 95 | options: |
| 96 | divisor: 100 # raw value in basis points → divide by 100 for percent |
| 97 | float: true # use floating-point precision |
| 98 | |
| 99 | # ── Counter: monotonically increasing values (algorithm: incremental) ── |
| 100 | - family: Traffic |
| 101 | metrics: |
| 102 | - http_requests_total |
| 103 | - bytes_received |
| 104 | - bytes_sent |
| 105 | charts: |
| 106 | - id: http_requests |
| 107 | title: HTTP Requests |
| 108 | context: http_requests |
| 109 | units: requests/s |
| 110 | algorithm: incremental # engine computes rate: (current - previous) / interval |
| 111 | instances: |
| 112 | by_labels: [host] |
| 113 | dimensions: |
| 114 | - selector: http_requests_total |
| 115 | name_from_label: method # dynamic name: each unique label value becomes a dimension |
| 116 | # e.g., method="GET" → dim "GET", method="POST" → dim "POST" |
| 117 | |
| 118 | - id: bandwidth |
| 119 | title: Network Bandwidth |
| 120 | context: bandwidth |
| 121 | units: kilobits/s |
| 122 | type: area |
| 123 | algorithm: incremental |
| 124 | instances: |
| 125 | by_labels: [host] |
| 126 | dimensions: |
| 127 | - selector: bytes_received |
| 128 | name: in |
| 129 | options: |
| 130 | multiplier: 8 # bytes → bits |
| 131 | divisor: 1000 # bits → kilobits |
| 132 | - selector: bytes_sent |
| 133 | name: out |
| 134 | options: |
| 135 | multiplier: -8 # negative = drawn below zero line (bidirectional chart) |
| 136 | divisor: 1000 |
| 137 | |
| 138 | # ── Histogram: bucketed distribution (flattened into _bucket, _count, _sum) ── |
| 139 | - family: Latency |
| 140 | metrics: |
| 141 | - request_duration_seconds_bucket |
| 142 | - request_duration_seconds_count |
| 143 | - request_duration_seconds_sum |
| 144 | charts: |
| 145 | - id: request_duration_buckets |
| 146 | title: Request Duration Buckets |
| 147 | context: request_duration_buckets |
| 148 | units: observations/s |
| 149 | type: stacked |
| 150 | algorithm: incremental # histogram buckets are counters |
| 151 | instances: |
| 152 | by_labels: [host] |
| 153 | dimensions: |
| 154 | - selector: request_duration_seconds_bucket |
| 155 | # no name, no name_from_label → engine infers dimension names |
| 156 | # from the "le" (less-than-or-equal) label automatically: |
| 157 | # le="0.005" → dim "0.005", le="0.01" → dim "0.01", etc. |
| 158 | |
| 159 | - id: request_rate |
| 160 | title: Request Rate |
| 161 | context: request_rate |
| 162 | units: requests/s |
| 163 | algorithm: incremental |
| 164 | instances: |
| 165 | by_labels: [host] |
| 166 | dimensions: |
| 167 | - selector: request_duration_seconds_count |
| 168 | name: requests |
| 169 | |
| 170 | # ── Summary: quantile distribution (flattened into quantile values, _count, _sum) ── |
| 171 | - family: Response Time |
| 172 | metrics: |
| 173 | - response_time_seconds |
| 174 | - response_time_seconds_count |
| 175 | - response_time_seconds_sum |
| 176 | charts: |
| 177 | - id: response_time_quantiles |
| 178 | title: Response Time Quantiles |
| 179 | context: response_time_quantiles |
| 180 | units: seconds |
| 181 | algorithm: absolute # quantile values are gauges, not counters |
| 182 | instances: |
| 183 | by_labels: [host] |
| 184 | dimensions: |
| 185 | - selector: response_time_seconds |
| 186 | # no name, no name_from_label → engine infers dimension names |
| 187 | # from the "quantile" label automatically: |
| 188 | # quantile="0.5" → dim "0.5", quantile="0.99" → dim "0.99", etc. |
| 189 | options: |
| 190 | float: true |
| 191 | |
| 192 | # ── StateSet: named boolean states (exactly one active at a time) ── |
| 193 | - family: Health |
| 194 | metrics: |
| 195 | - service_status |
| 196 | charts: |
| 197 | - id: service_health |
| 198 | title: Service Health Status |
| 199 | context: service_health |
| 200 | units: state |
| 201 | instances: |
| 202 | by_labels: [host] |
| 203 | dimensions: |
| 204 | - selector: service_status |
| 205 | # no name, no name_from_label → engine infers dimension names |
| 206 | # from the metric-name label (service_status=<value>) automatically: |
| 207 | # service_status="ready" → dim "ready", service_status="degraded" → dim "degraded", etc. |
| 208 | ``` |
| 209 | |
| 210 | **What this template produces** — if the collector reports metrics for 2 hosts (`host="web-1"`, `host="web-2"`), the engine creates **2 instances of every chart** (one per host). The histogram bucket chart gets one dimension per `le` boundary, the summary chart gets one per quantile, and the stateset chart gets one per state — all named automatically by the engine. |
| 211 | |
| 212 | ## Template Structure |
| 213 | |
| 214 | Every `charts.yaml` follows this structure: |
| 215 | |
| 216 | ```yaml |
| 217 | version: <schema version> |
| 218 | context_namespace: <context prefix> |
| 219 | engine: <engine policy> |
| 220 | groups: |
| 221 | - family: <family name> |
| 222 | context_namespace: <context segment> |
| 223 | metrics: <available metrics> |
| 224 | chart_defaults: <inheritable defaults> |
| 225 | charts: <chart definitions> |
| 226 | groups: <nested groups> |
| 227 | ``` |
| 228 | |
| 229 | | Section | Purpose | |
| 230 | |-----------------------------------------------|----------------------------------------------------| |
| 231 | | [**version**](#1-version) | Schema version (must be `v1`). | |
| 232 | | [**context_namespace**](#2-context_namespace) | Top-level prefix for chart context paths. | |
| 233 | | [**engine**](#3-engine) | Engine-level policy (selectors, autogeneration). | |
| 234 | | [**groups**](#4-groups) | Recursive chart groups — the core of the template. | |
| 235 | |
| 236 | --- |
| 237 | |
| 238 | ## Field Reference |
| 239 | |
| 240 | ### 1. version |
| 241 | |
| 242 | Schema version. Currently only `v1` is supported. Defaults to `v1` if omitted. |
| 243 | |
| 244 | ```yaml |
| 245 | version: v1 |
| 246 | ``` |
| 247 | |
| 248 | ### 2. context_namespace |
| 249 | |
| 250 | Top-level prefix for all chart context paths in the template. Combined with group-level `context_namespace` and chart `context` to form the final context. |
| 251 | |
| 252 | ```yaml |
| 253 | context_namespace: mysql |
| 254 | ``` |
| 255 | |
| 256 | **Context composition** — the final chart context is built by joining all context parts with `.`: |
| 257 | |
| 258 | ``` |
| 259 | <top context_namespace>.<group context_namespace>...<chart.context> |
| 260 | ``` |
| 261 | |
| 262 | For example: |
| 263 | |
| 264 | | Level | Value | |
| 265 | |-------------------------------|---------------------| |
| 266 | | Top-level `context_namespace` | `mysql` | |
| 267 | | Group `context_namespace` | _(empty)_ | |
| 268 | | Chart `context` | `queries` | |
| 269 | | **Resulting context** | **`mysql.queries`** | |
| 270 | |
| 271 | **Autogen charts** — the **top-level** `context_namespace` also prefixes the contexts of charts |
| 272 | created by `engine.autogen` (metrics not matched by any template dimension), joined with the same |
| 273 | `.`. Group-level `context_namespace` does not apply to autogen, since unmatched series belong to |
| 274 | no group. For example, with top-level `context_namespace: nagios`, an unmatched metric |
| 275 | `check_load` autogenerates the context `nagios.check_load`. |
| 276 | |
| 277 | The autogen context is `context_namespace` joined with the **full metric name**, and a metric's name |
| 278 | includes any `SnapshotMeter("<prefix>")` prefix (`<prefix>.<instrument>`). So a non-empty meter prefix |
| 279 | **stacks after** `context_namespace` — e.g. `context_namespace: app` with `SnapshotMeter("app")` and |
| 280 | instrument `foo` yields `app.app.foo`. When you set `context_namespace`, write metrics with |
| 281 | `SnapshotMeter("")` so the namespace has a single source; do not also encode it in the meter prefix. |
| 282 | |
| 283 | ### 3. engine |
| 284 | |
| 285 | Template-level policy that controls metric filtering and autogeneration. |
| 286 | |
| 287 | ```yaml |
| 288 | engine: |
| 289 | selector: |
| 290 | allow: ["cpu_*", "memory_*"] |
| 291 | deny: ["cpu_guest_*"] |
| 292 | autogen: |
| 293 | enabled: true |
| 294 | expire_after_success_cycles: 50 |
| 295 | ``` |
| 296 | |
| 297 | | Field | Type | Default | Description | |
| 298 | |---------------------------------------|---------------|-------------|----------------------------------------------------------------------------------------| |
| 299 | | `selector.allow` | array[string] | _(empty)_ | Include only metrics matching these patterns (simple patterns: `*` and `?` wildcards). | |
| 300 | | `selector.deny` | array[string] | _(empty)_ | Exclude metrics matching these patterns (simple patterns: `*` and `?` wildcards). | |
| 301 | | `autogen.enabled` | bool | `false` | Create charts for metrics not matched by any template dimension. | |
| 302 | | `autogen.max_type_id_len` | int | `0` (=1200) | Max full `type.id` length. Must be `0` or `>= 4`. | |
| 303 | | `autogen.expire_after_success_cycles` | uint64 | `0` | Remove autogenerated charts not seen for N successful cycles (`0` = never). | |
| 304 | |
| 305 | **When to use autogen**: For collectors like Nagios plugins where the set of metrics is unpredictable and user-defined. The engine creates a chart for every unmatched metric automatically. |
| 306 | |
| 307 | **Example: Nagios collector with autogeneration** |
| 308 | |
| 309 | ```yaml |
| 310 | version: v1 |
| 311 | context_namespace: nagios |
| 312 | engine: |
| 313 | autogen: |
| 314 | enabled: true |
| 315 | expire_after_success_cycles: 50 |
| 316 | groups: |
| 317 | - family: Job |
| 318 | context_namespace: job |
| 319 | groups: |
| 320 | - family: Execution |
| 321 | metrics: |
| 322 | - nagios.job.execution_state |
| 323 | - nagios.job.execution_duration |
| 324 | charts: |
| 325 | - id: job_execution_state |
| 326 | title: Job Execution State |
| 327 | context: execution_state |
| 328 | units: state |
| 329 | instances: |
| 330 | by_labels: [nagios_job] |
| 331 | dimensions: |
| 332 | - selector: nagios.job.execution_state |
| 333 | - id: job_execution_duration |
| 334 | title: Execution Duration |
| 335 | context: execution_duration |
| 336 | units: seconds |
| 337 | instances: |
| 338 | by_labels: [nagios_job] |
| 339 | dimensions: |
| 340 | - selector: nagios.job.execution_duration |
| 341 | name: duration |
| 342 | options: |
| 343 | float: true |
| 344 | ``` |
| 345 | |
| 346 | Explicitly defined charts (like `execution_state`) use the template. Any _other_ metrics the Nagios plugin emits get auto-charted by the engine. |
| 347 | |
| 348 | ### 4. groups |
| 349 | |
| 350 | Groups organize charts into a hierarchy that can be nested to **any depth**. Each group defines a **family** segment, can declare **metrics** in scope, and contains **charts** and/or nested **groups**. |
| 351 | |
| 352 | Nesting serves three purposes: |
| 353 | |
| 354 | 1. **Family composition** — each level's `family` is joined with `/`, producing Netdata's hierarchical family structure automatically (the UI renders `/`-separated families as navigable levels). |
| 355 | 2. **Context composition** — each level's `context_namespace` is joined with `.`, so you write short context leaves instead of long prefixed strings. |
| 356 | 3. **Metric scoping** — metrics declared in a group are inherited by all descendants, so you declare once at the appropriate level. |
| 357 | |
| 358 | ```yaml |
| 359 | groups: |
| 360 | - family: <family name> |
| 361 | context_namespace: <optional context segment> |
| 362 | metrics: |
| 363 | - <metric_name> |
| 364 | chart_defaults: |
| 365 | label_promotion: [<label>, ...] |
| 366 | instances: |
| 367 | by_labels: [<label>, ...] |
| 368 | charts: |
| 369 | - <chart definition> |
| 370 | groups: |
| 371 | - <nested group> |
| 372 | ``` |
| 373 | |
| 374 | | Field | Type | Required | Description | |
| 375 | |---------------------|---------------|----------|-------------------------------------------------------------------------------------| |
| 376 | | `family` | string | **yes** | Family segment. Groups compose the chart family hierarchy. | |
| 377 | | `context_namespace` | string | no | Context segment appended to inherited context namespace. | |
| 378 | | `metrics` | array[string] | no | Metrics visible to dimension selectors in this group and descendants. | |
| 379 | | `chart_defaults` | object | no | Inheritable defaults for descendant charts (see [chart_defaults](#chart_defaults)). | |
| 380 | | `charts` | array | no | Chart definitions (see [charts](#5-charts)). | |
| 381 | | `groups` | array | no | Nested groups (recursive). | |
| 382 | |
| 383 | **Family composition** — group families compose hierarchically. The final chart family is built by joining all group `family` segments and the chart's own `family` (if set) with `/`: |
| 384 | |
| 385 | | Level | Family value | |
| 386 | |----------------------|-----------------------------------------| |
| 387 | | Root group | `Storage Engine` | |
| 388 | | Nested group | `InnoDB` | |
| 389 | | Nested group | `Buffer Pool` | |
| 390 | | **Resulting family** | **`Storage Engine/InnoDB/Buffer Pool`** | |
| 391 | |
| 392 | Here is a real-world nesting example showing how family and context compose at each level: |
| 393 | |
| 394 | ```yaml |
| 395 | # context_namespace: mysql (set at top level) |
| 396 | groups: # family context |
| 397 | - family: Storage Engine # Storage Engine (inherited) |
| 398 | groups: |
| 399 | - family: InnoDB # Storage Engine/InnoDB (inherited) |
| 400 | groups: |
| 401 | - family: Buffer Pool # Storage Engine/InnoDB/Buffer Pool |
| 402 | charts: |
| 403 | - context: pages # → mysql.pages |
| 404 | - family: I/O # Storage Engine/InnoDB/I/O |
| 405 | charts: |
| 406 | - context: bandwidth # → mysql.bandwidth |
| 407 | - family: MyISAM # Storage Engine/MyISAM |
| 408 | charts: |
| 409 | - context: key_blocks # → mysql.key_blocks |
| 410 | ``` |
| 411 | |
| 412 | Without nesting, you would repeat `Storage Engine/InnoDB/` in every chart's family and `mysql.` in every context. Nesting eliminates that repetition and makes the structure self-documenting. |
| 413 | |
| 414 | > [!WARNING] |
| 415 | > Dimensions can only reference metrics declared in their group or any ancestor group. Referencing a metric not in scope produces a validation error. |
| 416 | |
| 417 | **Metric scoping** — this prevents accidental cross-references and keeps templates self-documenting: |
| 418 | |
| 419 | ```yaml |
| 420 | groups: |
| 421 | - family: Database |
| 422 | metrics: |
| 423 | - queries_total # visible to all charts in this group and nested groups |
| 424 | groups: |
| 425 | - family: Cache |
| 426 | metrics: |
| 427 | - cache_hits # visible only in this group and its descendants |
| 428 | charts: |
| 429 | - title: Cache Performance |
| 430 | context: cache |
| 431 | units: hits/s |
| 432 | dimensions: |
| 433 | - selector: cache_hits # OK — declared in this group |
| 434 | name: hits |
| 435 | - selector: queries_total # OK — inherited from parent group |
| 436 | name: queries |
| 437 | ``` |
| 438 | |
| 439 | #### chart_defaults |
| 440 | |
| 441 | Inheritable chart configuration applied to all descendant charts in the group subtree. Useful when many charts share the same instance identity or label promotion policy. |
| 442 | |
| 443 | | Field | Type | Description | |
| 444 | |-------------------|---------------|------------------------------------------| |
| 445 | | `label_promotion` | array[string] | Default labels to promote on all charts. | |
| 446 | | `instances` | object | Default instance identity policy. | |
| 447 | |
| 448 | > [!NOTE] |
| 449 | > **Inheritance rules**: nearest group default wins (child overrides parent), chart-local field overrides inherited default, and list/object fields replace the inherited field wholesale — there is no deep merge or append. |
| 450 | |
| 451 | **Example: Azure Monitor — all charts share the same instance identity** |
| 452 | |
| 453 | ```yaml |
| 454 | groups: |
| 455 | - family: Azure Key Vault |
| 456 | context_namespace: key_vault |
| 457 | chart_defaults: |
| 458 | label_promotion: [resource_name, resource_group, region] |
| 459 | instances: |
| 460 | by_labels: [resource_uid] |
| 461 | charts: |
| 462 | # Every chart below inherits instances and label_promotion |
| 463 | # without repeating them. |
| 464 | - id: availability |
| 465 | title: Azure Key Vault Availability |
| 466 | context: availability |
| 467 | units: percentage |
| 468 | dimensions: |
| 469 | - selector: key_vault.availability_average |
| 470 | name: average |
| 471 | - id: api_latency |
| 472 | title: Azure Key Vault API Latency |
| 473 | context: api_latency |
| 474 | units: milliseconds |
| 475 | dimensions: |
| 476 | - selector: key_vault.service_api_latency_average |
| 477 | name: average |
| 478 | ``` |
| 479 | |
| 480 | Without `chart_defaults`, you would need to repeat `instances` and `label_promotion` on every chart. |
| 481 | |
| 482 | ### 5. charts |
| 483 | |
| 484 | A chart defines a single visualization in the Netdata dashboard. |
| 485 | |
| 486 | ```yaml |
| 487 | charts: |
| 488 | - id: <chart ID> |
| 489 | title: <chart title> |
| 490 | family: <optional family leaf> |
| 491 | context: <chart context> |
| 492 | units: <units string> |
| 493 | algorithm: <absolute|incremental> |
| 494 | type: <line|area|stacked|heatmap> |
| 495 | priority: <int> |
| 496 | label_promotion: [<label>, ...] |
| 497 | instances: |
| 498 | by_labels: [<label>, ...] |
| 499 | lifecycle: |
| 500 | max_instances: <int> |
| 501 | expire_after_cycles: <int> |
| 502 | dimensions: |
| 503 | max_dims: <int> |
| 504 | expire_after_cycles: <int> |
| 505 | dimensions: |
| 506 | - <dimension definition> |
| 507 | ``` |
| 508 | |
| 509 | | Field | Type | Required | Default | Description | |
| 510 | |-------------------|---------------|----------|------------------------|------------------------------------------------------------------------------| |
| 511 | | `id` | string | no | derived from `context` | Base chart ID. If omitted, derived by replacing `.` with `_` in `context`. | |
| 512 | | `title` | string | **yes** | | Chart title shown in the dashboard. | |
| 513 | | `family` | string | no | | Optional chart-level family leaf, appended to the group family. | |
| 514 | | `context` | string | **yes** | | Chart context leaf. Combined with context namespaces. | |
| 515 | | `units` | string | **yes** | | Chart units (e.g., `queries/s`, `bytes`, `percentage`). | |
| 516 | | `algorithm` | string | no | inferred from metrics | `absolute` or `incremental`. If omitted, inferred from metric suffixes. | |
| 517 | | `type` | string | no | `line` | `line`, `area`, `stacked`, or `heatmap`. | |
| 518 | | `priority` | int | no | `70000` | Chart ordering priority in the dashboard (`0` = use engine default `70000`). | |
| 519 | | `label_promotion` | array[string] | no | from `chart_defaults` | Labels to promote as chart labels (for filtering/grouping in UI). Entries must be non-empty label keys. | |
| 520 | | `instances` | object | no | from `chart_defaults` | Instance identity policy (see [instances](#instances)). | |
| 521 | | `lifecycle` | object | no | | Instance/dimension cap and expiry (see [lifecycle](#lifecycle)). | |
| 522 | | `dimensions` | array | **yes** | | At least one dimension required (see [dimensions](#6-dimensions)). | |
| 523 | |
| 524 | > [!TIP] |
| 525 | > When `algorithm` is omitted, the engine infers it from metric name suffixes. You only need to set it explicitly when the suffix doesn't match the intended behavior (e.g., a gauge metric named `*_total`). |
| 526 | |
| 527 | | Suffix | Inferred algorithm | |
| 528 | |-------------------------------------------|--------------------| |
| 529 | | `*_total`, `*_count`, `*_sum`, `*_bucket` | `incremental` | |
| 530 | | Everything else | `absolute` | |
| 531 | |
| 532 | > [!WARNING] |
| 533 | > If a chart's dimensions mix counter-like metrics (e.g., `requests_total`) with gauge-like metrics (e.g., `temperature`) and `algorithm` is omitted, the engine fails with a compile error: _"algorithm inference is ambiguous for mixed metric kinds; set algorithm explicitly"_. Set `algorithm` on the chart to resolve this. |
| 534 | |
| 535 | **Example: MySQL queries — incremental counters displayed as rates** |
| 536 | |
| 537 | ```yaml |
| 538 | charts: |
| 539 | - id: queries |
| 540 | title: Queries |
| 541 | context: queries |
| 542 | units: queries/s |
| 543 | algorithm: incremental |
| 544 | dimensions: |
| 545 | - selector: queries |
| 546 | name: queries |
| 547 | - selector: questions |
| 548 | name: questions |
| 549 | - selector: slow_queries |
| 550 | name: slow_queries |
| 551 | ``` |
| 552 | |
| 553 | **Example: MySQL bandwidth — bidirectional area chart with unit conversion** |
| 554 | |
| 555 | ```yaml |
| 556 | charts: |
| 557 | - id: net |
| 558 | title: Bandwidth |
| 559 | context: net |
| 560 | units: kilobits/s |
| 561 | type: area |
| 562 | algorithm: incremental |
| 563 | dimensions: |
| 564 | - selector: bytes_received |
| 565 | name: in |
| 566 | options: |
| 567 | multiplier: 8 |
| 568 | divisor: 1000 |
| 569 | - selector: bytes_sent |
| 570 | name: out |
| 571 | options: |
| 572 | multiplier: -8 # negative = below zero line |
| 573 | divisor: 1000 |
| 574 | ``` |
| 575 | |
| 576 | #### instances |
| 577 | |
| 578 | Instance identity determines how series are grouped into chart instances. When multiple series share the same instance identity label values, they appear as dimensions on the same chart instance. |
| 579 | |
| 580 | > [!TIP] |
| 581 | > Without `instances`, there is one chart instance (all matching series land on the same chart). With `instances`, the engine creates one chart instance per unique combination of the specified label values. |
| 582 | |
| 583 | ```yaml |
| 584 | instances: |
| 585 | by_labels: [host] |
| 586 | ``` |
| 587 | |
| 588 | | Token | Meaning | |
| 589 | |--------------|---------------------------------------------------------------| |
| 590 | | `label_key` | Include this label in instance identity. | |
| 591 | | `*` | Include all labels. | |
| 592 | | `!label_key` | Exclude this label (use with `*` to include all _except_...). | |
| 593 | |
| 594 | Excludes are order-independent and always win. For example, both `["host", "!host"]` and `["!host", "host"]` exclude `host`. |
| 595 | When `instances` is set, `by_labels` must include at least one positive selector: `*` or `label_key`. Exclude tokens use strict `!label_key` syntax; `! host` is invalid. |
| 596 | |
| 597 | **Example: One chart per host** |
| 598 | |
| 599 | ```yaml |
| 600 | instances: |
| 601 | by_labels: [host] |
| 602 | ``` |
| 603 | |
| 604 | If the collector reports metrics for hosts `server-1`, `server-2`, `server-3`, the engine creates 3 separate chart instances — each showing only that host's dimensions. |
| 605 | |
| 606 | **Example: One chart per unique (job, instance) combination** |
| 607 | |
| 608 | ```yaml |
| 609 | instances: |
| 610 | by_labels: [nagios_job, perfdata_value] |
| 611 | ``` |
| 612 | |
| 613 | **Example: All labels except one** |
| 614 | |
| 615 | ```yaml |
| 616 | instances: |
| 617 | by_labels: ["*", "!_collect_job"] |
| 618 | ``` |
| 619 | |
| 620 | #### lifecycle |
| 621 | |
| 622 | Controls cardinality limits and expiry for chart instances and dimensions. |
| 623 | |
| 624 | | Field | Type | Default | Description | |
| 625 | |----------------------------------|------|----------------|--------------------------------------------------------------------------------------| |
| 626 | | `max_instances` | int | `0` (disabled) | Best-effort cap on chart instances per template. Active instances are never evicted. | |
| 627 | | `expire_after_cycles` | int | `5` | Remove chart instances not seen for N successful collection cycles. | |
| 628 | | `dimensions.max_dims` | int | `0` (disabled) | Best-effort cap on dimensions per chart instance. | |
| 629 | | `dimensions.expire_after_cycles` | int | `0` (disabled) | Remove dimensions not seen for N successful collection cycles. | |
| 630 | |
| 631 | **How lifecycle caps work**: |
| 632 | |
| 633 | - Caps are **best-effort** — instances/dimensions actively seen in the current cycle are never evicted. |
| 634 | - Oldest inactive entries are evicted first (by last-seen time). |
| 635 | - Expiry counters only advance on **successful** collection cycles. |
| 636 | |
| 637 | ### 6. dimensions |
| 638 | |
| 639 | A dimension binds a metric from the collector's metric store to a line on the chart. |
| 640 | |
| 641 | ```yaml |
| 642 | dimensions: |
| 643 | - selector: <metric selector> |
| 644 | name: <static name> |
| 645 | name_from_label: <label key> |
| 646 | options: |
| 647 | multiplier: <int> |
| 648 | divisor: <int> |
| 649 | hidden: <bool> |
| 650 | float: <bool> |
| 651 | ``` |
| 652 | |
| 653 | | Field | Type | Required | Default | Description | |
| 654 | |----------------------|--------|----------|---------|------------------------------------------------------------------| |
| 655 | | `selector` | string | **yes** | | Metric selector expression (see [selectors](#selectors) below). | |
| 656 | | `name` | string | no | | Static dimension name shown in the chart. | |
| 657 | | `name_from_label` | string | no | | Dynamic name: use the value of this label as the dimension name. | |
| 658 | | `options.multiplier` | int | no | `1` | Multiply the raw value by this factor. | |
| 659 | | `options.divisor` | int | no | `1` | Divide the raw value by this factor. | |
| 660 | | `options.hidden` | bool | no | `false` | Hide this dimension in the chart (still collected). | |
| 661 | | `options.float` | bool | no | `false` | Use floating-point precision for this dimension. | |
| 662 | |
| 663 | > [!IMPORTANT] |
| 664 | > There are three ways to name a dimension — pick **exactly one**: |
| 665 | > - `name` — static name you choose (e.g., `name: read`). |
| 666 | > - `name_from_label` — dynamic name from a label value (e.g., `name_from_label: method` → dimensions "GET", "POST", ...). |
| 667 | > - **Omit both** — the engine infers the name automatically for histogram buckets (`le`), summary quantiles (`quantile`), and statesets. |
| 668 | > |
| 669 | > `name` and `name_from_label` are mutually exclusive. Duplicate static `name` values within the same chart are rejected. |
| 670 | |
| 671 | #### selectors |
| 672 | |
| 673 | A selector specifies which metric(s) a dimension should match. |
| 674 | |
| 675 | **Syntax:** |
| 676 | |
| 677 | ``` |
| 678 | metric_name |
| 679 | metric_name{label_key=label_value, ...} |
| 680 | ``` |
| 681 | |
| 682 | - The metric name prefix is **required** — label-only selectors like `{label=value}` are rejected. |
| 683 | - The metric must be declared in the current group's `metrics` list (or inherited from an ancestor group). |
| 684 | - Label filters narrow which series match. Without labels, all series of that metric match. |
| 685 | |
| 686 | **Examples:** |
| 687 | |
| 688 | ```yaml |
| 689 | # Match all series of the "queries" metric |
| 690 | - selector: queries |
| 691 | |
| 692 | # Match only series where method="GET" |
| 693 | - selector: http_requests_total{method="GET"} |
| 694 | |
| 695 | # Match a specific histogram bucket |
| 696 | - selector: request_duration_seconds_bucket{le="0.5"} |
| 697 | ``` |
| 698 | |
| 699 | #### Common dimension patterns |
| 700 | |
| 701 | **Unit conversion** — convert bytes to kilobits per second: |
| 702 | |
| 703 | ```yaml |
| 704 | dimensions: |
| 705 | - selector: bytes_received |
| 706 | name: in |
| 707 | options: |
| 708 | multiplier: 8 |
| 709 | divisor: 1000 |
| 710 | ``` |
| 711 | |
| 712 | **Bidirectional charts** — use a negative multiplier to display below zero: |
| 713 | |
| 714 | ```yaml |
| 715 | dimensions: |
| 716 | - selector: bytes_received |
| 717 | name: in |
| 718 | options: |
| 719 | multiplier: 8 |
| 720 | divisor: 1000 |
| 721 | - selector: bytes_sent |
| 722 | name: out |
| 723 | options: |
| 724 | multiplier: -8 |
| 725 | divisor: 1000 |
| 726 | ``` |
| 727 | |
| 728 | **Float precision** — for ratios or small decimal values: |
| 729 | |
| 730 | ```yaml |
| 731 | dimensions: |
| 732 | - selector: efficiency_ratio |
| 733 | name: efficiency |
| 734 | options: |
| 735 | float: true |
| 736 | ``` |
| 737 | |
| 738 | **Dynamic naming from labels** — each unique label value becomes a separate dimension: |
| 739 | |
| 740 | ```yaml |
| 741 | dimensions: |
| 742 | - selector: http_requests_total |
| 743 | name_from_label: method |
| 744 | ``` |
| 745 | |
| 746 | If the metric has series with `method="GET"`, `method="POST"`, etc., each becomes its own dimension on the chart. |
| 747 | |
| 748 | ## Examples |
| 749 | |
| 750 | ### Simple: static metrics, no instances |
| 751 | |
| 752 | A collector that monitors a single MySQL server. Each chart has a fixed set of dimensions. |
| 753 | |
| 754 | ```yaml |
| 755 | version: v1 |
| 756 | context_namespace: mysql |
| 757 | groups: |
| 758 | - family: Queries |
| 759 | groups: |
| 760 | - family: Statistics |
| 761 | metrics: |
| 762 | - queries |
| 763 | - questions |
| 764 | - slow_queries |
| 765 | - com_delete |
| 766 | - com_insert |
| 767 | - com_select |
| 768 | - com_update |
| 769 | charts: |
| 770 | - id: queries |
| 771 | title: Queries |
| 772 | context: queries |
| 773 | units: queries/s |
| 774 | algorithm: incremental |
| 775 | dimensions: |
| 776 | - selector: queries |
| 777 | name: queries |
| 778 | - selector: questions |
| 779 | name: questions |
| 780 | - selector: slow_queries |
| 781 | name: slow_queries |
| 782 | - id: queries_type |
| 783 | title: Queries By Type |
| 784 | context: queries_type |
| 785 | units: queries/s |
| 786 | type: stacked |
| 787 | algorithm: incremental |
| 788 | dimensions: |
| 789 | - selector: com_delete |
| 790 | name: delete |
| 791 | - selector: com_insert |
| 792 | name: insert |
| 793 | - selector: com_select |
| 794 | name: select |
| 795 | - selector: com_update |
| 796 | name: update |
| 797 | ``` |
| 798 | |
| 799 | ### Per-instance: one chart per host |
| 800 | |
| 801 | A ping collector that monitors multiple hosts. Each host gets its own set of charts. |
| 802 | |
| 803 | ```yaml |
| 804 | version: v1 |
| 805 | context_namespace: ping |
| 806 | groups: |
| 807 | - family: latency |
| 808 | metrics: |
| 809 | - min_rtt |
| 810 | - max_rtt |
| 811 | - avg_rtt |
| 812 | charts: |
| 813 | - id: host_rtt |
| 814 | title: Ping round-trip time |
| 815 | context: host_rtt |
| 816 | units: milliseconds |
| 817 | type: area |
| 818 | instances: |
| 819 | by_labels: [host] |
| 820 | dimensions: |
| 821 | - selector: min_rtt |
| 822 | name: min |
| 823 | options: |
| 824 | divisor: 1000 |
| 825 | - selector: max_rtt |
| 826 | name: max |
| 827 | options: |
| 828 | divisor: 1000 |
| 829 | - selector: avg_rtt |
| 830 | name: avg |
| 831 | options: |
| 832 | divisor: 1000 |
| 833 | ``` |
| 834 | |
| 835 | ### Per-instance with multiple labels |
| 836 | |
| 837 | MySQL replication monitoring creates one chart per replication connection. |
| 838 | |
| 839 | ```yaml |
| 840 | groups: |
| 841 | - family: Replication |
| 842 | groups: |
| 843 | - family: Slave Status |
| 844 | metrics: |
| 845 | - seconds_behind_master |
| 846 | - slave_io_running |
| 847 | - slave_sql_running |
| 848 | charts: |
| 849 | - id: slave_behind |
| 850 | title: Slave Behind Seconds |
| 851 | context: slave_behind |
| 852 | units: seconds |
| 853 | instances: |
| 854 | by_labels: [connection] |
| 855 | dimensions: |
| 856 | - selector: seconds_behind_master |
| 857 | name: seconds |
| 858 | - id: slave_thread_running |
| 859 | title: I/O / SQL Thread Running State |
| 860 | context: slave_status |
| 861 | units: boolean |
| 862 | instances: |
| 863 | by_labels: [connection] |
| 864 | dimensions: |
| 865 | - selector: slave_io_running |
| 866 | name: io_running |
| 867 | - selector: slave_sql_running |
| 868 | name: sql_running |
| 869 | ``` |
| 870 | |
| 871 | ### Deeply nested groups |
| 872 | |
| 873 | MySQL's InnoDB storage engine metrics organized in a deep hierarchy. |
| 874 | |
| 875 | ```yaml |
| 876 | groups: |
| 877 | - family: Storage Engine |
| 878 | groups: |
| 879 | - family: InnoDB |
| 880 | groups: |
| 881 | - family: Buffer Pool |
| 882 | metrics: |
| 883 | - innodb_buffer_pool_pages_data |
| 884 | - innodb_buffer_pool_pages_dirty |
| 885 | - innodb_buffer_pool_pages_free |
| 886 | - innodb_buffer_pool_pages_misc |
| 887 | - innodb_buffer_pool_pages_total |
| 888 | charts: |
| 889 | - id: innodb_buffer_pool_pages |
| 890 | title: InnoDB Buffer Pool Pages |
| 891 | context: innodb_buffer_pool_pages |
| 892 | units: pages |
| 893 | dimensions: |
| 894 | - selector: innodb_buffer_pool_pages_data |
| 895 | name: data |
| 896 | - selector: innodb_buffer_pool_pages_dirty |
| 897 | name: dirty |
| 898 | options: |
| 899 | multiplier: -1 |
| 900 | - selector: innodb_buffer_pool_pages_free |
| 901 | name: free |
| 902 | - selector: innodb_buffer_pool_pages_misc |
| 903 | name: misc |
| 904 | options: |
| 905 | multiplier: -1 |
| 906 | - selector: innodb_buffer_pool_pages_total |
| 907 | name: total |
| 908 | - family: I/O |
| 909 | metrics: |
| 910 | - innodb_data_read |
| 911 | - innodb_data_written |
| 912 | charts: |
| 913 | - id: innodb_io |
| 914 | title: InnoDB I/O Bandwidth |
| 915 | context: innodb_io |
| 916 | units: KiB/s |
| 917 | type: area |
| 918 | algorithm: incremental |
| 919 | dimensions: |
| 920 | - selector: innodb_data_read |
| 921 | name: read |
| 922 | options: |
| 923 | divisor: 1024 |
| 924 | - selector: innodb_data_written |
| 925 | name: write |
| 926 | options: |
| 927 | divisor: 1024 |
| 928 | ``` |
| 929 | |
| 930 | The resulting chart families are `Storage Engine/InnoDB/Buffer Pool` and `Storage Engine/InnoDB/I/O`. |
| 931 | |
| 932 | ### chart_defaults: reducing repetition |
| 933 | |
| 934 | When monitoring a cloud resource that has many charts, all sharing the same instance identity. |
| 935 | |
| 936 | ```yaml |
| 937 | groups: |
| 938 | - family: Azure PostgreSQL |
| 939 | context_namespace: postgres_flexible |
| 940 | chart_defaults: |
| 941 | label_promotion: [resource_name, resource_group, region] |
| 942 | instances: |
| 943 | by_labels: [resource_uid] |
| 944 | charts: |
| 945 | - title: CPU Percent |
| 946 | context: cpu_percent |
| 947 | units: percentage |
| 948 | dimensions: |
| 949 | - selector: postgres_flexible.cpu_percent_average |
| 950 | name: average |
| 951 | - title: Memory Percent |
| 952 | context: memory_percent |
| 953 | units: percentage |
| 954 | dimensions: |
| 955 | - selector: postgres_flexible.memory_percent_average |
| 956 | name: average |
| 957 | - title: Storage Percent |
| 958 | context: storage_percent |
| 959 | units: percentage |
| 960 | dimensions: |
| 961 | - selector: postgres_flexible.storage_percent_average |
| 962 | name: average |
| 963 | ``` |
| 964 | |
| 965 | All three charts inherit `instances` and `label_promotion` from `chart_defaults` — no repetition needed. |
| 966 | |
| 967 | ### Autogeneration: handling unpredictable metrics |
| 968 | |
| 969 | For collectors where the metric set is user-defined or discovered at runtime, use autogen to catch metrics that don't match any explicit chart template. |
| 970 | |
| 971 | ```yaml |
| 972 | version: v1 |
| 973 | context_namespace: prometheus_scraper |
| 974 | engine: |
| 975 | autogen: |
| 976 | enabled: true |
| 977 | expire_after_success_cycles: 30 |
| 978 | selector: |
| 979 | deny: ["go_*", "promhttp_*"] # exclude internal Go/Prometheus metrics |
| 980 | groups: |
| 981 | - family: Application |
| 982 | metrics: |
| 983 | - app_http_requests_total |
| 984 | - app_http_response_time_seconds |
| 985 | charts: |
| 986 | - id: app_requests |
| 987 | title: Application HTTP Requests |
| 988 | context: http_requests |
| 989 | units: requests/s |
| 990 | algorithm: incremental |
| 991 | instances: |
| 992 | by_labels: [instance] |
| 993 | dimensions: |
| 994 | - selector: app_http_requests_total |
| 995 | name_from_label: status_code |
| 996 | - id: app_response_time |
| 997 | title: Application Response Time |
| 998 | context: response_time |
| 999 | units: seconds |
| 1000 | instances: |
| 1001 | by_labels: [instance] |
| 1002 | dimensions: |
| 1003 | - selector: app_http_response_time_seconds |
| 1004 | name: p99 |
| 1005 | options: |
| 1006 | float: true |
| 1007 | ``` |
| 1008 | |
| 1009 | The explicitly defined charts handle `app_http_requests_total` and `app_http_response_time_seconds`. Any _other_ application metrics the scraper discovers are automatically charted by the engine, and removed after 30 cycles of inactivity. The `selector.deny` filter excludes noisy internal metrics from autogeneration. |
| 1010 | |
| 1011 | ## Validation Rules |
| 1012 | |
| 1013 | > [!CAUTION] |
| 1014 | > Unknown YAML fields cause an immediate decode error (strict unmarshal). Double-check field names for typos — a misspelled field like `demensions` will be caught at parse time with an unmarshal error, not at runtime with a descriptive message pointing to the affected chart. |
| 1015 | |
| 1016 | All rules below produce semantic validation errors unless noted: |
| 1017 | |
| 1018 | | Rule | Error type | |
| 1019 | |-----------------------------------------------------------------------------------------|---------------------------------| |
| 1020 | | `version` must be `v1` | semantic | |
| 1021 | | `groups[]` must be non-empty | semantic | |
| 1022 | | `group.family` must not be empty or whitespace-only | semantic | |
| 1023 | | `group.metrics[]` entries must not be empty; no duplicates within same group | semantic | |
| 1024 | | `chart.title`, `chart.context`, `chart.units` must be non-empty | semantic | |
| 1025 | | `chart.algorithm` must be `absolute` or `incremental` (when specified) | semantic | |
| 1026 | | `chart.type` must be `line`, `area`, `stacked`, or `heatmap` (when specified) | semantic | |
| 1027 | | `dimension.selector` must include explicit metric name (prefix before `{`) | semantic | |
| 1028 | | Selector metric must be visible in current group metric scope | semantic | |
| 1029 | | `name` and `name_from_label` are mutually exclusive | semantic | |
| 1030 | | `name` and `name_from_label` must not be whitespace-only | semantic | |
| 1031 | | Duplicate dimension `name` values within the same chart are rejected | semantic | |
| 1032 | | `instances.by_labels` must contain at least one token when `instances` is set | semantic | |
| 1033 | | `instances.by_labels` exclude token must use `!label_key` syntax | semantic | |
| 1034 | | `instances.by_labels` must include at least one positive selector (`*` or `label_key`) | semantic | |
| 1035 | | `instances.by_labels` tokens must not be duplicated | semantic | |
| 1036 | | `label_promotion[]` entries must not be empty or whitespace-only | semantic | |
| 1037 | | Lifecycle numeric fields must be `>= 0` | semantic | |
| 1038 | | `engine.autogen.max_type_id_len` must be `0` or `>= 4` | semantic | |
| 1039 | | Unknown YAML fields | decode error (strict unmarshal) | |
| 1040 | |
| 1041 | ## Compiler-Derived Behavior |
| 1042 | |
| 1043 | > [!NOTE] |
| 1044 | > These behaviors are applied by `chartengine` during compilation, not by the template parser. You don't need to configure them — they happen automatically, but knowing about them helps you write simpler templates. |
| 1045 | |
| 1046 | | Input | Derived behavior | |
| 1047 | |-----------------------------------------|------------------------------------------------------------------------------------------| |
| 1048 | | Missing `chart.id` | `id` derived from `context` (`.` replaced with `_`). | |
| 1049 | | Missing `chart.algorithm` | Inferred from metric suffixes (`*_total`, `*_count`, `*_sum`, `*_bucket` = incremental). | |
| 1050 | | `chart.priority = 0` | Treated as `70000` (engine default). | |
| 1051 | | Group family hierarchy + `chart.family` | Composed into `/`-separated chart family. | |
| 1052 | | `options.multiplier = 0` | Treated as `1`. | |
| 1053 | | `options.divisor = 0` | Treated as `1`. | |