Document accuracy implications of sampling algorithm (#20991)
Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>
Costa Tsaousis committed
Sep 17, 2025 at 10:08 UTC
b43f46e4c6aa782f4b77c45a12a50f2f241b7523
1 file changed
+24
-1
src/collectors/systemd-journal.plugin/README.md
+24
-1
@@ -282,6 +282,29 @@ The sampling algorithm is designed to be resilient to large datasets. Even if yo
282
283
:::
284
285
+#### Accuracy implications
286
+Netdata’s sampling budget evaluates **up to 1,000,000 log entries** before it ever marks rows as `[unsampled]`. The proportion of the dataset we examine is:
287
+
288
+```
289
+evaluated_entries = min(total_entries, 1_000_000)
290
+evaluated_ratio = evaluated_entries / total_entries
291
+```
292
+
293
+Because the sampling set is so large, percentage breakdowns stay tight even on massive datasets. For a 10 M–entry window where 60 % of logs share a value, the 95 % confidence interval around that percentage is:
294
+
295
+```
296
+standard_error ≈ sqrt(p * (1 - p) / evaluated_entries)
297
+CI95 ≈ 1.96 * standard_error = 1.96 * sqrt(0.6 * 0.4 / 1_000_000) ≈ ±0.9 %
298
+```
299
+
300
+By contrast, evaluating only 5,000 entries (a small-sample approach typical of many log explorers when speed is prioritized) would yield:
301
+
302
+```
303
+CI95 ≈ 1.96 * sqrt(0.6 * 0.4 / 5_000) ≈ ±8.7 %
304
+```
305
+
306
+The result is that even at extreme scale, mainly because Netdata samples 200x more data, it can provide significantly more accurate estimations on value distributions, at comparable performance.
307
+
308
---
309
310
## Best practices for better performance
@@ -560,4 +583,4 @@ Check the Netdata Agent logs for plugin startup messages:
583
sudo journalctl -u netdata | grep journal
584
```
585
563
-Look for lines confirming the journal plugin started successfully and detected sources.
\ No newline at end of file
586
+Look for lines confirming the journal plugin started successfully and detected sources.