@cryptotaxi247 / netdata / commits / b41374bfb

docs: task class: improve_docs - add a prometheus/openmetrics example to the (#22479)

* docs: update docs/netdata-alert-cross-chart-prometheus-syntax * docs: Execution completed * docs: Execution completed * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * docs: Execution completed * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * docs: Execution completed * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: nedi-app[bot] <nedi-app[bot]@users.noreply.github.com> Co-authored-by: Fotis Voutsas <fotis@netdata.cloud> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

nedi-app[bot] committed May 21, 2026 at 08:48 UTC b41374bfb6613a0c8c58a1e4ef3b3968bb5c912c
1 file changed +46
src/health/REFERENCE.md
+46
@@ -995,6 +995,52 @@ Although the `alarm_variables` link shows variables for a particular chart, the
995 - `$system.cpu.user` - User CPU from system.cpu chart
996 - `$disk.sda.reads` - Read operations from sda disk chart
997
998 +##### Cross-Chart Variable Examples from Stock Health Entities
999 +
1000 +Several stock health configurations use host variables to reference dimensions from **other charts** in their `calc`, `warn`, and `crit` expressions.
1001 +
1002 +| Health entity | File | Expression | Cross-chart reference |
1003 +|---------------|------|------------|-----------------------|
1004 +| `30min_ram_swapped_out` | `health.d/swap.conf` | `calc: $this / 1024 * 100 / ( $system.ram.used + $system.ram.cached + $system.ram.free )` | `$system.ram.*` from within an alert on the `mem.swapio` chart |
1005 +| `ram_available` | `health.d/ram.conf` | `calc: $avail * 100 / ($system.ram.used + $system.ram.cached + $system.ram.free + $system.ram.buffers)` | `$system.ram.*` from within an alert on the `mem.available` chart |
1006 +| `system_clock_sync_state` | `health.d/timex.conf` | `warn: $system.uptime.uptime > 17 * 60 AND $this == 0` | `$system.uptime.uptime` from within an alert on the `system.clock_sync_state` chart |
1007 +| `audit_backlog_utilization` | `health.d/audit.conf` | `warn: $this > 50 AND $audit.failure.panic == 1` | `$audit.failure.panic` from within an alert on the `audit.backlog_utilization` chart |
1008 +| `10s_ip_tcp_resets_sent` | `health.d/tcp_resets.conf` | `warn: $netdata.uptime.uptime > (1 * 60) AND ...` | `$netdata.uptime.uptime` from within an alert on the `ip.tcphandshake` chart |
1009 +| `streaming_never_connected` | `health.d/streaming.conf` | `warn: $netdata.uptime.uptime > 30 * 60 AND $this > 0` | `$netdata.uptime.uptime` from within an alert on the `netdata.streaming_inbound` chart |
1010 +
1011 +##### Prometheus Collector Variables
1012 +
1013 +For metrics collected by the go.d `prometheus` collector, each unique Prometheus label set usually produces a separate chart. The chart ID is built from the metric name followed by `-label=value` pairs for every label (e.g. `kubelet_volume_stats_used_bytes-persistentvolumeclaim=my-pvc`). In the Netdata chart registry, the prefix comes from the go.d job `FullName`: it is `prometheus.<metric_name>-<label_set>` only when the job name is literally `prometheus`; otherwise it is `prometheus_<job_name>.<metric_name>-<label_set>` (for example, `prometheus_local.<metric_name>-<label_set>` or `prometheus_kubelet.<metric_name>-<label_set>`). For summary and histogram metric families, the collector may also emit related chart IDs such as `<id>`, `<id>_sum`, and `<id>_count`, so verify the exact chart ID you want to reference.
1014 +
1015 +Because Prometheus chart IDs typically contain hyphens and `=` characters, use the `${...}` brace form to reference them in `calc`/`warn`/`crit` expressions — the unbraced `$var` form stops parsing at `-`. Apply the same rule for both the common `prometheus_<job_name>` prefix and the special-case plain `prometheus` prefix, including any `_sum` or `_count` chart variants.
1016 +
1017 +**Example — PVC volume usage alert using kubelet metrics from a named Prometheus job (`name: kubelet`):**
1018 +
1019 +```text
1020 + alarm: kubelet_pvc_volume_usage
1021 + on: prometheus_kubelet.kubelet_volume_stats_used_bytes-persistentvolumeclaim=my-pvc
1022 + lookup: max -1m unaligned match-names of kubelet_volume_stats_used_bytes
1023 + calc: $this * 100 / ${prometheus_kubelet.kubelet_volume_stats_capacity_bytes-persistentvolumeclaim=my-pvc.kubelet_volume_stats_capacity_bytes}
1024 + warn: $this > 80
1025 + crit: $this > 95
1026 + units: %
1027 + every: 1m
1028 + info: PVC volume usage percentage
1029 + to: sysadmin
1030 +```
1031 +
1032 +:::note
1033 +
1034 +The exact chart ID and dimension names depend on your endpoint's label sets. Use the alarm variables API to discover the correct names:
1035 +
1036 +```text
1037 +http://NODE:19999/api/v1/alarm_variables?chart=prometheus_kubelet.kubelet_volume_stats_used_bytes-persistentvolumeclaim%3Dmy-pvc
1038 +```
1039 +
1040 +URL-encode the chart ID only in API query parameters. In alert expressions, use the chart ID as-is inside `${...}` braces.
1041 +
1042 +:::
1043 +
1044 #### Special Variables
1045
1046 | Variable | Contains | Usage |