| 1 | # Go Helper Packages For go.d Collectors |
| 2 | |
| 3 | Use existing helper packages before adding collector-local plumbing. A helper is |
| 4 | not better because it is shared; it is better when it gives users the same |
| 5 | configuration shape, the same safety behavior, or the same testable parsing path |
| 6 | as other collectors. |
| 7 | |
| 8 | This guide covers helper surfaces used by go.d collectors across: |
| 9 | |
| 10 | - `src/go/pkg/*` for shared Go packages used beyond go.d; |
| 11 | - `src/go/plugin/go.d/pkg/*` for go.d-specific helpers; |
| 12 | - `src/go/logger` for the logger embedded through `collectorapi.Base`. |
| 13 | |
| 14 | It is not an exhaustive API reference. Before adding a local helper, search |
| 15 | these roots for an existing package that already owns the behavior. |
| 16 | |
| 17 | ## Helper Roots |
| 18 | |
| 19 | | Need | Start with | |
| 20 | |---|---| |
| 21 | | V2 metrics, metric stores, host scopes | `src/go/pkg/metrix` | |
| 22 | | Duration and tri-state config option types | `src/go/pkg/confopt` | |
| 23 | | HTTP request/client config | `src/go/pkg/web` | |
| 24 | | TLS config outside HTTP | `src/go/pkg/tlscfg` | |
| 25 | | Prometheus exposition parsing | `src/go/pkg/prometheus` | |
| 26 | | User selector/matcher grammar | `src/go/pkg/matcher` | |
| 27 | | Collector logging and log limiting | `src/go/logger` | |
| 28 | | Function request/response helpers | `src/go/pkg/funcapi` | |
| 29 | | Topology payloads | `src/go/pkg/topology/v1` | |
| 30 | | Agent API / chart emission payloads | `src/go/pkg/netdataapi` | |
| 31 | | TCP/UDP/Unix line-protocol clients | `src/go/plugin/go.d/pkg/socket` | |
| 32 | | Command execution | `src/go/plugin/go.d/pkg/ndexec` | |
| 33 | | Log-file readers/parsers | `src/go/plugin/go.d/pkg/logs` | |
| 34 | | IP range parsing | `src/go/plugin/go.d/pkg/iprange` | |
| 35 | | SQL query/scan helpers | `src/go/plugin/go.d/pkg/sqlquery` | |
| 36 | | Cloud auth config/credentials | `src/go/plugin/go.d/pkg/cloudauth` | |
| 37 | | Ping probing | `src/go/plugin/go.d/pkg/pinger` | |
| 38 | | SNMP utilities | `src/go/plugin/go.d/pkg/snmputils` | |
| 39 | | Kubernetes client helpers | `src/go/plugin/go.d/pkg/k8sclient` | |
| 40 | | Docker host helpers | `src/go/plugin/go.d/pkg/dockerhost` | |
| 41 | | Test helpers for collectors | `src/go/plugin/go.d/pkg/collecttest` | |
| 42 | | Legacy V1 metric helpers | `src/go/pkg/stm`, `src/go/plugin/go.d/pkg/oldmetrix` | |
| 43 | |
| 44 | ## Config Option Types |
| 45 | |
| 46 | Use `src/go/pkg/confopt` for common configuration value types. |
| 47 | |
| 48 | When: |
| 49 | |
| 50 | - users configure durations that should accept strings such as `5s`, `30m`, or |
| 51 | numeric seconds; |
| 52 | - users need explicit `auto` / `enabled` / `disabled` behavior instead of a |
| 53 | plain boolean; |
| 54 | - a migration needs to preserve legacy pointer-boolean semantics without |
| 55 | keeping pointer plumbing in new code. |
| 56 | |
| 57 | Why: |
| 58 | |
| 59 | - `confopt.Duration` and `confopt.LongDuration` centralize YAML/JSON duration |
| 60 | parsing and formatting; |
| 61 | - `confopt.AutoBool` makes tri-state behavior explicit and schema-friendly; |
| 62 | - collectors avoid ad hoc parsers and inconsistent boolean defaults. |
| 63 | |
| 64 | ## HTTP Collectors |
| 65 | |
| 66 | Use `src/go/pkg/web` for HTTP-based collectors. |
| 67 | |
| 68 | When: |
| 69 | |
| 70 | - the collector talks to an HTTP or HTTPS endpoint; |
| 71 | - users need the normal Netdata HTTP options: `url`, timeout, redirects, proxy, |
| 72 | basic auth, bearer token file, headers, body, method, and TLS fields; |
| 73 | - the collector builds repeated requests against the same endpoint. |
| 74 | |
| 75 | Why: |
| 76 | |
| 77 | - `web.HTTPConfig` embeds `web.RequestConfig` and `web.ClientConfig` so HTTP |
| 78 | collectors expose the same option surface; |
| 79 | - `web.NewHTTPClient(c.ClientConfig)` applies timeout, TLS, proxy, redirect, and |
| 80 | HTTP/2 behavior consistently; |
| 81 | - `web.NewHTTPRequest(c.RequestConfig)` and |
| 82 | `web.NewHTTPRequestWithPath(c.RequestConfig, path)` apply user agent, |
| 83 | authentication, headers, body, and safe path joining. |
| 84 | |
| 85 | Pattern: |
| 86 | |
| 87 | ```go |
| 88 | type Config struct { |
| 89 | web.HTTPConfig `yaml:",inline" json:""` |
| 90 | } |
| 91 | ``` |
| 92 | |
| 93 | Use `src/go/pkg/tlscfg` directly only when the collector is not HTTP-based but |
| 94 | still needs TLS, such as Redis or x509-style checks. HTTP collectors should get |
| 95 | TLS behavior through `web.HTTPConfig`. |
| 96 | |
| 97 | ## Prometheus Endpoints |
| 98 | |
| 99 | Use `src/go/pkg/prometheus` when the upstream endpoint exposes Prometheus text |
| 100 | format. |
| 101 | |
| 102 | When: |
| 103 | |
| 104 | - the collector scrapes `/metrics` or another Prometheus exposition endpoint; |
| 105 | - the collector needs to parse metric families or sorted series; |
| 106 | - the collector needs a bounded selector for metric names. |
| 107 | |
| 108 | Why: |
| 109 | |
| 110 | - it reuses `web.RequestConfig` and `*http.Client`; |
| 111 | - it handles Prometheus text parsing and gzip responses; |
| 112 | - selectors avoid parsing or processing metric families the collector will not |
| 113 | use. |
| 114 | |
| 115 | Do not hand-roll text exposition parsing in a collector. |
| 116 | |
| 117 | ## Selectors And Matchers |
| 118 | |
| 119 | Use `src/go/pkg/matcher` for user-facing include/exclude or selector fields. |
| 120 | |
| 121 | When: |
| 122 | |
| 123 | - users select entities by name, ID, interface, queue, topic, or similar labels; |
| 124 | - the selector syntax can be glob, regexp, string, or simple patterns; |
| 125 | - negative matches such as `!*test* *` are sufficient. |
| 126 | |
| 127 | Why: |
| 128 | |
| 129 | - users get one matcher grammar across collectors; |
| 130 | - tests can cover selector behavior without custom parser logic; |
| 131 | - existing logical matchers can combine conditions when needed. |
| 132 | |
| 133 | Do not invent a selector language unless the upstream API requires one. Prefer a |
| 134 | single simple-pattern field for simple cases; add separate include/exclude fields |
| 135 | only when the user problem needs that shape. |
| 136 | |
| 137 | Do not use `src/go/pkg/selectorcore` for user-facing collector selectors. It is |
| 138 | the lower-level selector metadata/parser surface used by template and selector |
| 139 | engines, not the normal collector selector helper. |
| 140 | |
| 141 | ## Limited Logging |
| 142 | |
| 143 | Collectors embed `collectorapi.Base`, which embeds `*logger.Logger`. Use the |
| 144 | logger's built-in limiting before adding collector-local rate-limit state. |
| 145 | |
| 146 | When: |
| 147 | |
| 148 | - an error can repeat every collection cycle; |
| 149 | - a partial failure is useful to report but would spam logs; |
| 150 | - a one-time notice or warning is enough. |
| 151 | |
| 152 | Why: |
| 153 | |
| 154 | - in go.d jobs, `c.Once(key).Warningf(...)` is cycle-local because the runtime |
| 155 | resets `Once` state each `runOnce`; it is useful for suppressing duplicate |
| 156 | messages inside one cycle only; |
| 157 | - `c.Limit(key, n, window).Warningf(...)` logs at most `n` messages per key per |
| 158 | window and is the right default for cross-cycle spam control; |
| 159 | - the limiter is shared through the collector logger and already used by modern |
| 160 | collectors such as Cato Networks, PAN-OS, and vSphere. |
| 161 | |
| 162 | Pattern: |
| 163 | |
| 164 | ```go |
| 165 | c.Limit("mycollector:operation:error", 1, time.Hour). |
| 166 | Warningf("operation failed: %v", err) |
| 167 | ``` |
| 168 | |
| 169 | Use stable keys. Include the operation and bounded error class when needed, but |
| 170 | do not put unbounded IDs, URLs, query strings, customer names, or raw provider |
| 171 | messages in the key. |
| 172 | |
| 173 | Custom warning gates are justified only when the built-in count-per-window |
| 174 | semantics are not the right behavior, for example when logging only on state |
| 175 | transitions. Document that reason in the PR description or design note so |
| 176 | reviewers can see why the built-in limiter was not enough. |
| 177 | |
| 178 | ## Socket Clients |
| 179 | |
| 180 | Use `src/go/plugin/go.d/pkg/socket` for simple TCP, UDP, or Unix-socket |
| 181 | line-protocol collectors. |
| 182 | |
| 183 | When: |
| 184 | |
| 185 | - the collector connects to a local or remote socket and sends text commands; |
| 186 | - the response is processed line by line; |
| 187 | - the collector needs shared timeout, TLS, and max-read-line behavior. |
| 188 | |
| 189 | Why: |
| 190 | |
| 191 | - socket address parsing is shared across collectors; |
| 192 | - connect, command, read, disconnect, deadline, and line-limit behavior stay |
| 193 | consistent; |
| 194 | - tests can use the helper's fake TCP/UDP/Unix servers instead of custom socket |
| 195 | harnesses. |
| 196 | |
| 197 | Do not hand-roll socket dial/read loops for common line-oriented protocols. |
| 198 | |
| 199 | ## External Commands |
| 200 | |
| 201 | Use `src/go/plugin/go.d/pkg/ndexec` for collectors that execute binaries. |
| 202 | |
| 203 | When: |
| 204 | |
| 205 | - the collector needs a local command output; |
| 206 | - the command should run through Netdata's helper wrappers; |
| 207 | - the command may need privilege through `ndsudo`; |
| 208 | - tests need to stub helper paths. |
| 209 | |
| 210 | Why: |
| 211 | |
| 212 | - arguments are passed without a shell; |
| 213 | - timeouts and context cancellation are handled; |
| 214 | - stderr snippets are bounded; |
| 215 | - helpers integrate with Netdata's execution model. |
| 216 | |
| 217 | Use: |
| 218 | |
| 219 | - `RunUnprivileged` / `RunUnprivilegedWithOptions...` for unprivileged commands; |
| 220 | - `RunNDSudo` for commands exposed through `ndsudo`; |
| 221 | - `RunDirect` only when direct execution is intentionally required; |
| 222 | - `FindBinary` for PATH/default-path discovery. |
| 223 | |
| 224 | Do not call `exec.Command` directly unless the helper cannot support the case and |
| 225 | the reason is documented. |
| 226 | |
| 227 | ## Log File Collectors |
| 228 | |
| 229 | Use `src/go/plugin/go.d/pkg/logs` for collectors that parse application log |
| 230 | files. |
| 231 | |
| 232 | When: |
| 233 | |
| 234 | - the collector tails files that can rotate; |
| 235 | - the log format is CSV, LTSV, regexp, or JSON; |
| 236 | - parser errors should be distinguishable from I/O errors. |
| 237 | |
| 238 | Why: |
| 239 | |
| 240 | - `logs.Reader` is log-rotation aware; |
| 241 | - `logs.NewParser` centralizes supported parser types; |
| 242 | - `logs.IsParseError` lets collection logic treat malformed rows differently |
| 243 | from source failures. |
| 244 | |
| 245 | Do not open and seek log files manually unless the collector's source is not a |
| 246 | normal file-tail workflow. |
| 247 | |
| 248 | ## IP Ranges |
| 249 | |
| 250 | Use `src/go/plugin/go.d/pkg/iprange` when users configure address ranges. |
| 251 | |
| 252 | When: |
| 253 | |
| 254 | - the collector filters IPs, networks, peers, or hosts by ranges; |
| 255 | - the config accepts CIDR, range, or other supported range syntax. |
| 256 | |
| 257 | Why: |
| 258 | |
| 259 | - range parsing and membership checks are shared; |
| 260 | - invalid syntax handling is consistent; |
| 261 | - collectors avoid slightly different IP matching semantics. |
| 262 | |
| 263 | ## SQL Helpers |
| 264 | |
| 265 | Use `src/go/plugin/go.d/pkg/sqlquery` for repeated SQL row-scanning patterns. |
| 266 | |
| 267 | When: |
| 268 | |
| 269 | - the collector or Function scans rows into strings, integers, floats, or discard |
| 270 | columns; |
| 271 | - the collector needs table-column discovery with `?` or `$1` placeholders; |
| 272 | - the row-to-value assignment is generic across queries. |
| 273 | |
| 274 | Why: |
| 275 | |
| 276 | - scan holders and null handling are centralized; |
| 277 | - query duration measurement and row iteration behavior stay testable; |
| 278 | - Function code can avoid custom one-off scanners. |
| 279 | |
| 280 | ## Cloud Auth Helpers |
| 281 | |
| 282 | Use `src/go/plugin/go.d/pkg/cloudauth` when a cloud collector needs supported |
| 283 | cloud-provider credentials. |
| 284 | |
| 285 | When: |
| 286 | |
| 287 | - the collector supports `cloud_auth` configuration; |
| 288 | - Azure AD credential construction is needed. |
| 289 | |
| 290 | Why: |
| 291 | |
| 292 | - provider names normalize consistently; |
| 293 | - validation is centralized; |
| 294 | - unsupported providers fail with consistent errors. |
| 295 | |
| 296 | ## Ping Helpers |
| 297 | |
| 298 | Use `src/go/plugin/go.d/pkg/pinger` for ping/latency probing. |
| 299 | |
| 300 | When: |
| 301 | |
| 302 | - a collector needs ICMP-style probing; |
| 303 | - it needs shared latency/jitter derivation. |
| 304 | |
| 305 | Why: |
| 306 | |
| 307 | - probe config validation and derived metrics are shared; |
| 308 | - collectors avoid reimplementing packet sampling and jitter math. |
| 309 | |
| 310 | ## Legacy V1 Helpers |
| 311 | |
| 312 | `src/go/pkg/stm` converts structs into `map[string]int64`. |
| 313 | `src/go/plugin/go.d/pkg/oldmetrix` provides V1 metric vector helper types such |
| 314 | as counters, summaries, histograms, and boolean conversions used by existing V1 |
| 315 | collectors. Both helpers are V1-shaped. New V2 collectors MUST NOT use them as |
| 316 | their metric path. |
| 317 | |
| 318 | Acceptable uses: |
| 319 | |
| 320 | - maintaining an existing V1 collector; |
| 321 | - temporary parity tests during V1-to-V2 migration, provided the helper is not |
| 322 | reachable from the final runtime path. |