@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 1,075 lines 36.9 KB
Rendered Raw
1 # StatsD Collector
2
3 ## What is StatsD?
4
5 [StatsD](https://github.com/statsd/statsd) is a system for collecting metrics from applications. Your applications send metrics to StatsD, usually via non-blocking UDP communication, and StatsD servers collect these metrics, perform simple calculations, and push them to time-series databases.
6
7 Learn more about the [StatsD protocol.](https://blog.netdata.cloud/introduction-to-statsd/)
8
9 ## Overview
10
11 | Feature | Description |
12 |----------------------------|----------------------------------------------------------------------------|
13 | **Metric Collection** | Collect real-time metrics from any application supporting StatsD protocol |
14 | **Visualization** | View metrics as private charts (one per metric) or custom synthetic charts |
15 | **Supported Metric Types** | Gauges, Counters, Meters, Timers, Histograms, Sets, Dictionaries |
16 | **Transport** | Both UDP (low-overhead) and TCP (reliable, higher volume) supported |
17 | **Performance** | Can collect millions of metrics per second using just 1 CPU core |
18 | **Integration** | Built directly into Netdata - no extra installation needed |
19 | **Language Support** | Use with Python, Node.js, Java, Go, Ruby, Shell scripts, and more |
20
21 :::tip
22
23 Want a hands-on example? [Jump to the K6 StatsD Walkthrough](#step-by-step-guide-monitoring-k6-with-statsd)
24
25 :::
26
27 ## Supported Metric Types Summary
28
29 | Metric Type | Purpose | Format | LLM Summary |
30 |--------------|-----------------------------------------------|---------------------|-------------------------------------------------------------------------------------------------|
31 | Gauges | Report current values | `name:value\|g` | Report latest value; can increment/decrement; supports sampling & tags. |
32 | Counters | Count events | `name:value\|c/C/m` | Report rate & event count; `:value` optional (default 1); supports sampling & tags. |
33 | Meters | Count events (rate-focused) | `name:value\|m` | Report rate & event count; `:value` optional (default 1); supports sampling & tags. |
34 | Timers | Statistical analysis of values (duration) | `name:value\|ms` | Report min, max, avg, percentiles, median, stddev, count; supports sampling & tags. |
35 | Histograms | Statistical analysis of values (distribution) | `name:value\|h` | Report min, max, avg, percentiles, median, stddev, count; supports sampling & tags. |
36 | Sets | Count unique occurrences | `name:value\|s` | Report unique count & event count; sampling NOT supported; values as text; supports tags. |
37 | Dictionaries | Count occurrences of distinct values | `name:value\|d` | Report counts per value & total updates; sampling NOT supported; values as text; supports tags. |
38
39 ### How StatsD Works with Netdata
40
41 ```mermaid
42 flowchart TD
43 A("Your Application") -->|"Sends metrics"| B("Netdata StatsD")
44 B -->|"Creates"| C("Private Charts")
45 B -->|"Creates"| D("Synthetic Charts")
46 B -->|"Stores in"| E("Database")
47
48 %% Style definitions
49 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
50 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
51 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
52 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
53
54 %% Apply styles
55 class A alert
56 class B,C,D neutral
57 class E complete
58 ```
59
60 ## Netdata as a StatsD Server
61
62 Netdata comes with a **fully-featured StatsD server built in**. You can:
63
64 - Collect StatsD-formatted metrics
65 - Visualize them on the Netdata dashboard
66 - Store them in Netdata's database for long-term retention
67
68 Since StatsD is embedded in Netdata, **you effectively have a StatsD server on every system where Netdata is installed.**
69
70 :::note
71
72 **Netdata's StatsD implementation is incredibly fast.** It can collect **several million metrics per second** on modern hardware using just one CPU core. The implementation uses two threads: one collects metrics, and the other updates the charts.
73
74 :::
75
76 ## Pre-configured StatsD Applications
77
78 Netdata includes **synthetic chart definitions** to automatically present application metrics consistently. These are defined in configuration files that you can use as-is or customize.
79
80 For synthetic charts, you can set up alerts just like with any other metric or chart.
81
82 Currently available applications:
83
84 - [K6 load testing tool](https://k6.io)
85 - **Description:** k6 is a developer-centric, free, and open-source load testing tool for performance testing
86 - [Documentation](https://github.com/netdata/netdata/blob/master/src/collectors/statsd.plugin/k6.md)
87 - [Configuration](https://github.com/netdata/netdata/blob/master/src/collectors/statsd.plugin/k6.conf)
88 - [Asterisk](https://www.asterisk.org/)
89 - **Description:** Asterisk is an Open Source PBX and telephony toolkit
90 - [Documentation](https://github.com/netdata/netdata/blob/master/src/collectors/statsd.plugin/asterisk.md)
91 - [Configuration](https://github.com/netdata/netdata/blob/master/src/collectors/statsd.plugin/asterisk.conf)
92
93 ## Supported Metric Types
94
95 Netdata fully supports the StatsD protocol and extends it for more advanced use cases. All StatsD client libraries are compatible with Netdata.
96
97 ```mermaid
98 flowchart TD
99 A("Application") -->|"Sends"| B("Metrics")
100 B --> C("Gauges")
101 B --> D("Counters")
102 B --> E("Timers")
103 B --> F("Histograms")
104 B --> G("Sets")
105 B --> H("Dictionaries")
106
107 %% Style definitions
108 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
109 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
110 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
111 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
112
113 %% Apply styles
114 class A alert
115 class B neutral
116 class C,D,E,F,G,H complete
117 ```
118
119 <details>
120 <summary><strong>Gauges</strong></summary>
121 <br/>
122
123 <strong>Purpose:</strong> Report current values (e.g., cache memory used by an application server)
124
125 **Format:** `name:value|g`
126
127 - `value` can be any decimal/fractional number
128 - StatsD reports the latest value and the number of updates (events)
129 - You can increment/decrement previous values by prefixing with `+` or `-`
130 - Sampling rate is supported
131 - Tags can change chart units, family, and [dimension](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#d) name
132 - When not collected, the last value will be shown if "show gaps" is disabled (default)
133
134 </details>
135
136 <details>
137 <summary><strong>Counters and Meters</strong></summary>
138 <br/>
139
140 <strong>Purpose:</strong> Count events (e.g., number of file downloads)
141
142 **Format:** `name:value|c`, `name:value|C`, or `name:value|m`
143
144 - `value` must be an integer (positive or negative)
145 - StatsD reports the rate and update count (events)
146 - `:value` can be omitted (defaults to 1)
147 - `|c`, `|C` and `|m` can be omitted (defaults to `|m`)
148 - Counters use `|c` (etsy/StatsD compatible) or `|C` (brubeck compatible)
149 - Meters use `|m`
150 - Sampling rate is supported
151 - Tags can change chart units, family, and dimension name
152 - When not collected, StatsD shows zero until a new value arrives
153
154 </details>
155
156 <details>
157 <summary><strong>Timers and Histograms</strong></summary>
158 <br/>
159
160 <strong>Purpose:</strong> Statistical analysis of values (e.g., request duration, file sizes)
161
162 **Format:** `name:value|ms` or `name:value|h`
163
164 - `value` can be any decimal/fractional number
165 - StatsD reports min, max, average, 95th percentile, median, standard deviation, and update count
166 - Timers use `|ms` and report in milliseconds
167 - Histograms use `|h`
168 - Sampling rate is supported
169 - Tags can change chart units and family
170 - When not collected, StatsD shows zero until a new value arrives
171
172 </details>
173
174 <details>
175 <summary><strong>Sets</strong></summary>
176 <br/>
177
178 <strong>Purpose:</strong> Count unique occurrences (e.g., unique users, unique filenames)
179
180 **Format:** `name:value|s`
181
182 - `value` can be any string or number (leading/trailing spaces are removed)
183 - StatsD reports the count of unique values and update count
184 - Sampling rate is NOT supported
185 - Values are always treated as text (so `01` and `1` are different)
186 - Tags can change chart units and family
187 - When not collected, StatsD shows zero until a new value arrives
188
189 </details>
190
191 <details>
192 <summary><strong>Dictionaries</strong></summary>
193 <br/>
194
195 <strong>Purpose:</strong> Count occurrences of distinct values
196
197 **Format:** `name:value|d`
198
199 - `value` can be any string or number (leading/trailing spaces are removed)
200 - StatsD reports the count of events for each `value` and total updates
201 - Sampling rate is NOT supported
202 - Values are always treated as text (so `01` and `1` are different)
203 - Tags can change chart units and family
204 - When not collected, StatsD shows zero until a new value arrives
205
206 </details>
207
208 ## Advanced Features
209
210 <details>
211 <summary><strong>Sampling Rates</strong></summary>
212 <br/>
213
214 You can append `|@sampling_rate` to metrics, where `sampling_rate` is between 0.0 and 1.0. This tells StatsD to extrapolate the value for the entire period.
215
216 Example: If your application reports data for only 1/10th of events, append `|@0.1` to have StatsD calculate the total.
217 </details>
218
219 <details>
220 <summary><strong>Tags</strong></summary>
221 <br/>
222
223 You can append `|#tag1:value1,tag2:value2,tag3:value3` to metrics. Netdata currently uses these tags:
224
225 - `units=string` - Sets the units of the automatically generated chart
226 - `family=string` - Sets the family (dashboard submenu) of the chart
227 - `name=string` - Sets the [dimension](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#d) name (for counters, meters, gauges only)
228
229 :::tip
230
231 For consistency, either send tags with every event or use the special `zinit` value to initialize charts. For example, send `my.metric:zinit|c|#units=bytes,name=size` at the beginning, then just `my.metric:VALUE|c` afterward.
232
233 :::
234
235 </details>
236
237 <details>
238 <summary><strong>Sending Multiple Metrics</strong></summary>
239 <br/>
240
241 You can send multiple metrics in a single packet by separating them with newlines (`\n`).
242
243 #### TCP Packets
244
245 Netdata listens for both TCP and UDP packets. With TCP, always append `\n` to each metric so Netdata can detect metrics split across multiple TCP packets.
246
247 #### UDP Packets
248
249 When sending multiple metrics in a UDP message, keep the total size under the network MTU (usually 1500 bytes).
250
251 :::important
252
253 Netdata can accept UDP datagrams up to the maximum UDP payload size (65,507 bytes for IPv4), but packets larger than the network MTU will be fragmented at the IP layer (or dropped if fragmentation is not allowed).
254
255 :::
256
257 </details>
258
259 ## Configuration
260
261 You can find the StatsD configuration in `/etc/netdata/netdata.conf`:
262
263 ```
264 [statsd]
265 # enabled = yes
266 # decimal detail = 1000
267 # update every (flushInterval) = 1s
268 # udp messages to process at once = 10
269 # create private charts for metrics matching = *
270 # max private charts hard limit = 1000
271 # cleanup obsolete charts after = 0
272 # private charts memory mode = save
273 # private charts history = 3996
274 # histograms and timers percentile (percentThreshold) = 95.00000
275 # add dimension for number of events received = no
276 # gaps on gauges (deleteGauges) = no
277 # gaps on counters (deleteCounters) = no
278 # gaps on meters (deleteMeters) = no
279 # gaps on sets (deleteSets) = no
280 # gaps on histograms (deleteHistograms) = no
281 # gaps on timers (deleteTimers) = no
282 # listen backlog = 4096
283 # default port = 8125
284 # bind to = udp:localhost:8125 tcp:localhost:8125
285 ```
286
287 ## Configuration Architecture
288
289 ### How the StatsD Configuration Works
290
291 Netdata's StatsD chart system uses three key sections in its configuration:
292
293 ```mermaid
294 flowchart TD
295 A("statsd.d config") --> B("app")
296 A --> C("dictionary")
297 A --> D("chart definitions")
298 B --> E("metric filtering")
299 C --> F("renaming for display")
300 D --> G("chart family/context/units/priorities")
301
302 %% Style definitions
303 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
304 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
305 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
306 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
307
308 %% Apply styles
309 class A alert
310 class B,C,D neutral
311 class E,F,G complete
312 ```
313
314 The diagram shows how the configuration flows:
315
316 1. The central `statsd.d config` connects to **three main components**:
317 - The **application** configuration
318 - The **dictionary** system
319 - **Chart definitions**
320
321 2. Each of these components serves a specific purpose:
322 - The **app** component handles **metric filtering**
323 - The **dictionary** manages **renaming metrics** for display
324 - **Chart definitions determine properties** like family, context, units, and priorities
325
326 This structure allows for flexible and powerful metric configuration within Netdata's StatsD implementation.
327
328 ### Key Configuration Options
329
330 - **`enabled = yes|no`** - Controls whether StatsD is enabled
331 - **`default port = 8125`** - The default port if not specified in binding
332 - **`bind to = udp:localhost tcp:localhost`** - Space-separated list of IPs and ports to listen on
333 - **`update every (flushInterval) = 1s`** - How often StatsD updates Netdata charts
334 - **`decimal detail = 1000`** - Controls decimal precision in gauges and histograms
335
336 ## StatsD Charts
337
338 Netdata can visualize StatsD collected metrics in two ways:
339
340 1. **Private charts** - Each metric gets its own chart (default, no configuration needed)
341 2. **Synthetic charts** - Combine multiple metrics into custom charts (requires configuration)
342
343 ### Private Metric Charts
344
345 Private charts are controlled with `create private charts for metrics matching = *`. This setting accepts a space-separated list of [simple patterns](https://github.com/netdata/netdata/blob/master/src/libnetdata/simple_pattern/README.md). By default, Netdata creates private charts for all metrics.
346
347 #### Private Chart Naming Convention
348
349 When querying StatsD metrics via the Netdata API, you must use the chart identifier that Netdata constructs from the metric name — not the raw metric name you sent to StatsD. The chart identifier is built as follows:
350
351 - **Chart type** = `statsd_<first_word>` — where `<first_word>` is the portion of the metric name before the first `.` or `_`.
352 - **Chart id** = `<remaining>_<metric_type>` — where `<remaining>` is the portion after the first `.` or `_`, and `<metric_type>` is one of: `gauge`, `counter`, `meter`, `timer`, `histogram`, `set`, or `dictionary`. If the metric name has no `.` or `_`, the chart id is just the `<metric_type>`.
353 - **Full chart reference** = `<chart_type>.<chart_id>` — use this as the `chart` parameter in API queries.
354
355 | Metric sent | Metric type | Chart type | Chart ID | Full chart reference |
356 |------------------------------|-------------|-----------------|---------------------|-------------------------------------|
357 | `test.metric:100\|c` | counter | `statsd_test` | `metric_counter` | `statsd_test.metric_counter` |
358 | `myapp.used_memory:12345\|g` | gauge | `statsd_myapp` | `used_memory_gauge` | `statsd_myapp.used_memory_gauge` |
359 | `myapp.requests:50\|ms` | timer | `statsd_myapp` | `requests_timer` | `statsd_myapp.requests_timer` |
360
361 :::tip
362
363 To discover the exact chart names available on your agent, run:
364
365 ```bash
366 curl http://localhost:19999/api/v1/charts
367 ```
368
369 Filter the results for IDs starting with `statsd_`. Use the returned chart `id` field (in `type.id` format) as the `chart` parameter in data queries.
370
371 :::
372
373 :::info
374
375 **Troubleshooting "No metrics where matched to query":** This error typically occurs when you use the raw StatsD metric name (e.g., `statsd.test.metric`) as the `chart` parameter instead of the constructed chart reference (e.g., `statsd_test.metric_counter`). To resolve this, verify the correct chart name using `/api/v1/charts` before querying metric data.
376
377 For example, after sending `test.metric:100|c` via StatsD, the correct query is:
378
379 ```bash
380 curl "http://localhost:19999/api/v1/data?chart=statsd_test.metric_counter&after=-10&before=now"
381 ```
382
383 The chart name `statsd_test.metric_counter` follows the `type.id` convention: type `statsd_test` (prefix + first dot-delimited word) and id `metric_counter` (remaining words + metric type).
384
385 :::
386
387 Example: To create charts for all `myapp.*` metrics except `myapp.*.badmetric`:
388
389 ```
390 create private charts for metrics matching = !myapp.*.badmetric myapp.*
391 ```
392
393 You can configure a different memory mode specifically for StatsD charts:
394
395 - `private charts memory mode`
396 - `private charts history`
397
398 <details>
399 <summary><strong>Private Chart Examples</strong></summary>
400 <br/>
401
402 Example of a gauge metric chart:
403
404 ![Gauge metric example](https://i.imgur.com/Sr5nJEV.png)
405
406 Example of a histogram metric chart:
407
408 ![Timer metric example](https://i.imgur.com/P4p0hvq.png)
409
410 Histogram chart with "sum" unselected:
411
412 ![Histogram dimensions](https://cloud.githubusercontent.com/assets/2662304/26131598/8076443a-3aa3-11e7-9ffa-ea535aee9c9f.png)
413
414 Example of a counter metric chart:
415
416 ![Counter example](https://cloud.githubusercontent.com/assets/2662304/26131553/4a26d19c-3aa3-11e7-94e8-c53b5ed6ebc3.png)
417
418 Example of a meter metric chart:
419
420 ![Meter example](https://cloud.githubusercontent.com/assets/2662304/26131605/8fdf5a06-3aa3-11e7-963f-7ecf207d1dbc.png)
421
422 Example of a set metric chart:
423
424 ![Set example](https://cloud.githubusercontent.com/assets/2662304/26131612/9eaa7b1a-3aa3-11e7-903b-d881e9a35be2.png)
425
426 Example of a timer metric chart:
427
428 ![Timer example](https://cloud.githubusercontent.com/assets/2662304/26131629/bc34f2d2-3aa3-11e7-8a07-f2fc94ba4352.png)
429 </details>
430
431 #### Storage Optimization
432
433 For performance reasons, Netdata limits private charts. The `max private charts hard limit` (default: 1000) controls this. Metrics above this limit can still be used in synthetic charts.
434
435 For ephemeral metrics, use `set charts as obsolete after` and `cleanup obsolete charts after` to automatically clean up charts that haven't received data recently.
436
437 ### Synthetic StatsD Charts
438
439 Use synthetic charts to create dedicated sections on the dashboard to render your StatsD charts.
440
441 ```mermaid
442 flowchart TD
443 A("StatsD Metrics") --> B("App")
444 A --> C("Dictionary")
445 B --> D("Chart")
446 C --> D
447 D --> E("Dashboard")
448
449 %% Style definitions
450 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
451 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
452 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
453 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
454
455 %% Apply styles
456 class A alert
457 class B,C neutral
458 class D,E complete
459 ```
460
461 Synthetic charts are organized in:
462
463 - **Application** - Section in Netdata Dashboard
464 - **Charts for each application** - Family/submenu in the Dashboard
465 - **StatsD metrics for each chart** - Charts and context in the Dashboard
466
467 #### Basic Configuration Structure
468
469 For example, to monitor the application `myapp` using StatsD and Netdata, create the file `/etc/netdata/statsd.d/myapp.conf`:
470
471 ```
472 [app]
473 name = myapp
474 metrics = myapp.*
475 private charts = no
476 gaps when not collected = no
477 history = 60
478
479 [dictionary]
480 m1 = metric1
481 m2 = metric2
482
483 # Chart definition with ID 'mychart'
484 # The chart will be named: myapp.mychart
485 [mychart]
486 name = mychart
487 title = my chart title
488 family = my family
489 context = chart.context
490 units = tests/s
491 priority = 91000
492 type = area
493 dimension = myapp.metric1 m1
494 dimension = myapp.metric2 m2
495 ```
496
497 Using this configuration, `myapp` gets its own dashboard section with one chart containing two [dimensions](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#d).
498
499 When you send metrics like `foo:10|g` and `bar:20|g`, you'll see both private charts and your synthetic chart.
500
501 <details>
502 <summary><strong>Synthetic Chart Example</strong></summary>
503 <br/>
504
505 Example of a synthetic chart combining multiple metrics:
506
507 ![Synthetic chart example](https://user-images.githubusercontent.com/2662304/28942317-958a2c68-78a3-11e7-853f-32850141dd36.png)
508 </details>
509
510 #### Application Section Options
511
512 The `[app]` section defines the application and has these options:
513
514 :::note
515
516 - **name** - Defines the application name
517 - **metrics** - [Simple pattern](https://github.com/netdata/netdata/blob/master/src/libnetdata/simple_pattern/README.md) matching all metrics for this app
518 - **private charts** - Enable/disable private charts for matched metrics (yes|no)
519 - **gaps when not collected** - Show gaps when no metrics are collected (yes|no)
520 - **memory mode** - Sets memory mode for application charts (optional, default is global Netdata setting)
521 - **history** - Size of round-robin database (optional, only relevant with `memory mode = save`)
522
523 :::
524
525 #### Dictionary Section
526
527 `[dictionary]` defines name-value pairs for renaming metrics in synthetic charts. This allows you to:
528
529 - Define dimension names globally for the whole app
530 - Rename dimensions when using patterns
531 - Create more human-readable names for technical metrics
532
533 The dictionary can be empty or omitted if not needed.
534
535 #### Chart Definitions
536
537 Each chart starts with `[id]` and will be named `app_name.id`. Key settings for charts:
538
539 :::note
540
541 - **family** - Controls dashboard submenu placement
542 - **context** - Controls alert templates
543 - **priority** - Controls chart ordering
544 - **type** - Chart visualization type (line, area, stacked)
545 - **units** - Chart measurement units
546
547 :::
548
549 #### [Dimension](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#d) Format
550
551 Add metrics to charts using `dimension` lines with this format:
552
553 ```
554 dimension = [pattern] METRIC NAME TYPE MULTIPLIER DIVIDER OPTIONS
555 ```
556
557 Where:
558
559 1. **METRIC** - The metric name as collected (must match the `metrics` pattern)
560 2. **NAME** - The dimension name to display (can use dictionary for renaming)
561 3. **TYPE** - (Optional) Value selector like `events`, `last`, `min`, `max`, etc.
562 4. **MULTIPLIER** - (Optional) Value to multiply the metric by
563 5. **DIVIDER** - (Optional) Value to divide the metric by
564 6. **OPTIONS** - (Optional) Flags like `hidden` to include but not display a dimension
565
566 <details>
567 <summary><strong>Renaming StatsD Synthetic Charts' Metrics</strong></summary>
568 <br/>
569
570 You can define a dictionary to rename metrics sent by StatsD clients. This allows you to transmit the response code `200` while Netdata displays it as `successful connection`.
571
572 The `[dictionary]` section accepts any number of `name = value` pairs.
573
574 Netdata uses this dictionary as follows:
575
576 1. When a `dimension` has a non-empty `NAME`, that name is looked up in the dictionary
577 2. If the above lookup finds nothing, the original StatsD metric name is looked up
578 3. If any lookup succeeds, Netdata uses the dictionary's `value` for the dimension name
579
580 The dimensions will have the original StatsD metric name as ID and the dictionary value as name.
581
582 You can use the dictionary in two ways:
583
584 1. Set `dimension = myapp.metric1 ''` and have in the dictionary `myapp.metric1 = metric1 name`
585 2. Set `dimension = myapp.metric1 'm1'` and have in the dictionary `m1 = metric1 name`
586
587 In both cases, the dimension will be added with ID `myapp.metric1` and named `metric1 name`. In alerts, you can reference it as either `${myapp.metric1}` or `${metric1 name}`.
588
589 :::note
590
591 If you add the same StatsD metric multiple times to a chart, Netdata will append `TYPE` to the dimension ID, so `myapp.metric1` will become `myapp.metric1_last` or `myapp.metric1_events`. If you add the same metric with the same `TYPE` multiple times, Netdata will also append an incremental counter, e.g., `myapp.metric1_last1`, `myapp.metric1_last2`, etc.
592
593 :::
594
595 </details>
596
597 <details>
598 <summary><strong>Dimension Patterns</strong></summary>
599 <br/>
600
601 Netdata allows adding multiple dimensions to a chart by matching StatsD metrics with a **pattern**.
602
603 For example, if you have an API that provides StatsD metrics for each response code per method:
604
605 ```
606 myapp.api.get.200
607 myapp.api.get.400
608 myapp.api.get.500
609 myapp.api.del.200
610 myapp.api.del.400
611 myapp.api.del.500
612 myapp.api.post.200
613 myapp.api.post.400
614 myapp.api.post.500
615 myapp.api.all.200
616 myapp.api.all.400
617 myapp.api.all.500
618 ```
619
620 To add all response codes of `myapp.api.get` to a chart:
621
622 ```
623 [api_get_responses]
624 ...
625 dimension = pattern 'myapp.api.get.* '' last 1 1
626 ```
627
628 This adds dimensions named `200`, `400`, and `500`. Netdata extracts the wildcard part of the metric name.
629
630 You can rename these dimensions with the dictionary:
631
632 ```
633 [dictionary]
634 get.200 = 200 ok
635 get.400 = 400 bad request
636 get.500 = 500 cannot connect to db
637
638 [api_get_responses]
639 ...
640 dimension = pattern 'myapp.api.get.* 'get.' last 1 1
641 ```
642
643 The `NAME` prefix `get.` is combined with the wildcarded part to look up in the dictionary. So `500` becomes `get.500`, which is looked up to find `500 cannot connect to db`.
644
645 ### More Pattern Examples
646
647 To add all 200s across all API methods to a chart:
648
649 ```
650 [ok_by_method]
651 ...
652 dimension = pattern 'myapp.api.*.200 '' last 1 1
653 ```
654
655 This adds `get`, `post`, `del`, and `all` to the chart.
656
657 To exclude the `all` method:
658
659 ```
660 [ok_by_method]
661 ...
662 dimension = pattern '!myapp.api.all.* myapp.api.*.200 '' last 1 1
663 ```
664
665 To rename methods automatically:
666
667 ```
668 [dictionary]
669 method.get = GET
670 method.post = ADD
671 method.del = DELETE
672
673 [ok_by_method]
674 ...
675 dimension = pattern '!myapp.api.all.* myapp.api.*.200 'method.' last 1 1
676 ```
677
678 This adds dimensions named `GET`, `ADD`, and `DELETE`.
679 </details>
680
681 ## Using StatsD with Different Languages
682
683 <details>
684 <summary><strong>Python</strong></summary>
685 <br/>
686
687 Using [jsocol/pystatsd](https://github.com/jsocol/pystatsd):
688
689 ```python
690 import statsd
691
692 c = statsd.StatsClient('localhost', 8125)
693 c.incr('foo') # Increment the 'foo' counter.
694 for i in range(100000000):
695 c.incr('bar')
696 c.incr('foo')
697 if i % 3:
698 c.decr('bar')
699 c.timing('stats.timed', 320) # Record a 320ms 'stats.timed'.
700 ```
701
702 See the [full documentation](https://statsd.readthedocs.io/en/v3.3/) for more details.
703 </details>
704
705 <details>
706 <summary><strong>JavaScript and Node.js</strong></summary>
707 <br/>
708
709 Using [sivy/node-statsd](https://github.com/sivy/node-statsd):
710
711 ```javascript
712 var StatsD = require('node-statsd'),
713 client = new StatsD();
714
715 // Timing: sends a timing command with the specified milliseconds
716 client.timing('response_time', 42);
717
718 // Increment: Increments a stat by a value (default is 1)
719 client.increment('my_counter');
720
721 // Decrement: Decrements a stat by a value (default is -1)
722 client.decrement('my_counter');
723
724 // Using the callback
725 client.set(['foo', 'bar'], 42, function (error, bytes) {
726 //this only gets called once after all messages have been sent
727 if (error) {
728 console.error('Oh noes! There was an error:', error);
729 } else {
730 console.log('Successfully sent', bytes, 'bytes');
731 }
732 });
733
734 // Sampling, tags and callback are optional and could be used in any combination
735 client.histogram('my_histogram', 42, 0.25); // 25% Sample Rate
736 client.histogram('my_histogram', 42, ['tag']); // User-defined tag
737 client.histogram('my_histogram', 42, next); // Callback
738 client.histogram('my_histogram', 42, 0.25, ['tag']);
739 client.histogram('my_histogram', 42, 0.25, next);
740 client.histogram('my_histogram', 42, ['tag'], next);
741 client.histogram('my_histogram', 42, 0.25, ['tag'], next);
742 ```
743
744 </details>
745
746 <details>
747 <summary><strong>Other Languages</strong></summary>
748 <br/>
749
750 StatsD clients are available for many languages:
751
752 - Golang: [alexcesaro/statsd](https://github.com/alexcesaro/statsd)
753 - Ruby: [reinh/statsd](https://github.com/reinh/statsd)
754 - Java: [DataDog/java-dogstatsd-client](https://github.com/DataDog/java-dogstatsd-client)
755
756 </details>
757
758 <details>
759 <summary><strong>Shell Script</strong></summary>
760 <br/>
761
762 You can use the Unix shell with `nc` to send StatsD metrics from any script.
763
764 :::important
765
766 You'll need the `netcat` package with the `nc` command. Different versions have different parameters, so experiment to find what works on your system. The examples below assume `openbsd-netcat` is installed.
767
768 :::
769
770 #### Using UDP (for sporadic events)
771
772 ```sh
773 echo "APPLICATION.METRIC:VALUE|TYPE" | nc -u -w 0 localhost 8125
774 ```
775
776 - `-u` enables UDP
777 - `-w 0` tells `nc` not to wait for a response
778
779 Examples:
780
781 ```sh
782 # Set a gauge value
783 echo "myapp.used_memory:123456|g|#units:bytes" | nc -u -w 0 localhost 8125
784
785 # Increment a counter
786 echo "myapp.files_sent:10|c|#units:files" | nc -u -w 0 localhost 8125
787
788 # Send multiple metrics
789 printf "myapp.used_memory:123456|g|#units:bytes\nmyapp.files_sent:10|c|#units:files\n" | nc -u -w 0 localhost 8125
790 ```
791
792 #### Using TCP (for many metrics at once)
793
794 ```sh
795 # send multiple metrics via TCP
796 cat /tmp/statsd.metrics.txt | nc -N -w 120 localhost 8125
797 ```
798
799 #### Helper Function for Shell Scripts
800
801 This function handles both UDP and TCP automatically:
802
803 ```sh
804 #!/usr/bin/env bash
805
806 # we assume nc is from the openbsd-netcat package
807
808 STATSD_HOST="localhost"
809 STATSD_PORT="8125"
810 statsd() {
811 local options="-u -w 0" all="${*}"
812
813 # replace all spaces with newlines
814 all="${all// /\\n}"
815
816 # if the string length of all parameters given is above 1000, use TCP
817 [ "${#all}" -gt 1000 ] && options="-N -w 0"
818
819 # send the metrics to statsd
820 printf "${all}\n" | nc ${options} ${STATSD_HOST} ${STATSD_PORT} || return 1
821
822 return 0
823 }
824
825 if [ ! -z "${*}" ]
826 then
827 statsd "${@}"
828 fi
829 ```
830
831 Usage:
832
833 ```sh
834 # source it in your script
835 source statsd.sh
836
837 # then use it anywhere
838 statsd "myapp.used_memory:123456|g|#units:bytes" "myapp.files_sent:10|c|#units:files" ...
839
840 # or at command line
841 ./statsd.sh "myapp.used_memory:123456|g|#units:bytes" "myapp.files_sent:10|c|#units:files" ...
842 ```
843
844 The function automatically switches to TCP if the metrics exceed 1000 bytes.
845 </details>
846
847 ## Step-by-Step Guide: Monitoring K6 with StatsD
848
849 This guide demonstrates how to use Netdata's StatsD to visualize metrics from [k6](https://k6.io), an open-source load testing tool.
850
851 <details>
852 <summary><strong>Prerequisites</strong></summary>
853 <br/>
854
855 - A node with [Netdata](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md) installed
856 - [k6](https://k6.io/docs/getting-started/installation) installed
857
858 </details>
859
860 <details>
861 <summary><strong>The Process in Brief</strong></summary>
862 <br/>
863
864 1. **Run an experiment** sending StatsD metrics to Netdata without configuration
865 - This creates a private chart per metric
866 - Reload the dashboard after starting to send data
867
868 2. **Create a configuration file** for your app:
869 ```
870 sudo ./edit-config statsd.d/myapp.conf
871 ```
872 - This organizes metrics into meaningful sections
873
874 </details>
875
876 <details>
877 <summary><strong>Understanding Your Metrics</strong></summary>
878 <br/>
879
880 First, understand what metrics your application provides. For k6, check their [metrics documentation](https://k6.io/docs/using-k6/metrics/).
881
882 When instrumenting your own code, you'll need to decide:
883
884 - What to measure
885 - Which StatsD metric type is appropriate for each measurement
886
887 </details>
888
889 <details>
890 <summary><strong>Exploring Available Metrics with Private Charts</strong></summary>
891 <br/>
892
893 Every StatsD metric initially gets its own "private chart." While you'll likely disable this in production, it's helpful during setup to see all available metrics.
894
895 Private charts clearly show the metric type (gauge, timer, etc.) and available operations for complex types like histograms.
896 </details>
897
898 <details>
899 <summary><strong>Creating a StatsD Configuration File</strong></summary>
900 <br/>
901
902 Use Netdata's [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) to create a new file:
903
904 ```bash
905 sudo ./edit-config statsd.d/k6.conf
906 ```
907
908 Start with this basic configuration:
909
910 ```
911 [app]
912 name = k6
913 metrics = k6*
914 private charts = yes
915 gaps when not collected = no
916 memory mode = dbengine
917 ```
918
919 </details>
920
921 <details>
922 <summary><strong>Organizing Metrics</strong></summary>
923 <br/>
924
925 Next, decide how to organize metrics in the Netdata dashboard:
926
927 1. **Dictionary** - Create human-readable names for technical metrics
928 ```
929 [dictionary]
930 http_req_blocked = Blocked HTTP Requests
931 http_req_connecting = Connecting HTTP Requests
932 http_req_receiving = Receiving HTTP Requests
933 http_reqs = Total HTTP requests
934 ```
935
936 2. **Families** - Group charts into dashboard submenus. For k6, we'll use `k6 native metrics` and `http metrics` families.
937
938 3. **[Dimensions](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#d)** - Choose which metrics to show and how to group them in charts
939
940 </details>
941
942 <details>
943 <summary><strong>Complete Configuration Example</strong></summary>
944 <br/>
945
946 Here's a complete configuration for k6:
947
948 ```
949 [app]
950 name = k6
951 metrics = k6*
952 private charts = yes
953 gaps when not collected = no
954 memory mode = dbengine
955
956 [dictionary]
957 http_req_blocked = Blocked HTTP Requests
958 http_req_connecting = Connecting HTTP Requests
959 http_req_receiving = Receiving HTTP Requests
960 http_reqs = Total HTTP requests
961
962 [http_req_total]
963 name = http_req_total
964 title = Total HTTP Requests
965 family = http requests
966 context = k6.http_requests
967 dimension = k6.http_reqs http_reqs last 1 1 sum
968 type = line
969 units = requests/s
970
971 [vus]
972 name = vus
973 title = Virtual Active Users
974 family = k6_metrics
975 dimension = k6.vus vus last 1 1
976 dimension = k6.vus_max vus_max last 1 1
977 type = line
978 unit = vus
979
980 [iteration_duration]
981 name = iteration_duration_2
982 title = Iteration duration
983 family = k6_metrics
984 dimension = k6.iteration_duration iteration_duration last 1 1
985 dimension = k6.iteration_duration iteration_duration_max max 1 1
986 dimension = k6.iteration_duration iteration_duration_min min 1 1
987 dimension = k6.iteration_duration iteration_duration_avg avg 1 1
988 type = line
989 unit = s
990
991 [dropped_iterations]
992 name = dropped_iterations
993 title = Dropped Iterations
994 family = k6_metrics
995 dimension = k6.dropped_iterations dropped_iterations last 1 1
996 units = iterations
997 type = line
998
999 [data]
1000 name = data
1001 title = K6 Data
1002 family = k6_metrics
1003 dimension = k6.data_received data_received last 1 1
1004 dimension = k6.data_sent data_sent last -1 1
1005 units = kb/s
1006 type = area
1007
1008 [http_req_status]
1009 name = http_req_status
1010 title = HTTP Requests Status
1011 family = http requests
1012 dimension = k6.http_req_blocked http_req_blocked last 1 1
1013 dimension = k6.http_req_connecting http_req_connecting last 1 1
1014 units = ms
1015 type = line
1016
1017 [http_req_duration]
1018 name = http_req_duration
1019 title = HTTP requests duration
1020 family = http requests
1021 dimension = k6.http_req_sending http_req_sending last 1 1
1022 dimension = k6.http_req_waiting http_req_waiting last 1 1
1023 dimension = k6.http_req_receiving http_req_receiving last 1 1
1024 units = ms
1025 type = stacked
1026 ```
1027
1028 :::note
1029
1030 Netdata will report the rate for metrics and counters even if your application sends absolute numbers. For example, k6 sends absolute HTTP requests with `http_reqs`, but Netdata visualizes that as `requests/second`.
1031
1032 :::
1033
1034 Restart Netdata to enable this configuration.
1035 </details>
1036
1037 <details>
1038 <summary><strong>Adding Custom Icons and Descriptions</strong></summary>
1039 <br/>
1040
1041 You can customize the section icon and add helpful chart descriptions.
1042
1043 Create a custom dashboard info file:
1044
1045 ```javascript
1046 netdataDashboard.menu = {
1047 'k6': {
1048 title: 'K6 Load Testing',
1049 icon: '<i class="fas fa-cogs"></i>',
1050 info: 'k6 is an open-source load testing tool and cloud service providing the best developer experience for API performance testing.'
1051 },
1052 };
1053
1054 netdataDashboard.context = {
1055 'k6.http_req_duration': {
1056 info: "Total time for the request. It's equal to http_req_sending + http_req_waiting + http_req_receiving (i.e. how long did the remote server take to process the request and respond, without the initial DNS lookup/connection times)"
1057 },
1058 };
1059 ```
1060
1061 These descriptions help users monitor your application, especially during incidents. The `info` field supports HTML, allowing you to embed links and instructions.
1062 </details>
1063
1064 <details>
1065 <summary><strong>Contributing Your Collector</strong></summary>
1066 <br/>
1067
1068 Once you've created a working configuration, consider sharing it with the Netdata community:
1069
1070 1. Follow the [contributing guide](https://github.com/netdata/.github/blob/main/CONTRIBUTING.md)
1071 2. Fork the netdata/netdata repository
1072 3. Place your configuration file in `netdata/collectors/statsd.plugin`
1073 4. Add a reference in `netdata/collectors/statsd.plugin/Makefile.am`
1074
1075 </details>