| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/troubleshooting.md" |
| 3 | sidebar_label: "Troubleshooting" |
| 4 | learn_status: "Published" |
| 5 | learn_rel_path: "Network Flows" |
| 6 | keywords: ['troubleshooting', 'debugging', 'plugin health', 'failures'] |
| 7 | endmeta--> |
| 8 | |
| 9 | <!-- markdownlint-disable-file --> |
| 10 | |
| 11 | # Troubleshooting |
| 12 | |
| 13 | Concrete recipes for the most common failures, organised by symptom. Most issues are diagnosable from the [plugin health charts](/docs/network-flows/visualization/dashboard-cards.md), the Netdata journal logs, and a couple of OS-level commands. |
| 14 | |
| 15 | ## The plugin doesn't start |
| 16 | |
| 17 | The plugin won't come up at all, or starts and immediately exits. |
| 18 | |
| 19 | **Symptoms:** |
| 20 | - Netdata reports `netflow-plugin` as not running, or restart-looping. |
| 21 | - Nothing in the Network Flows view. |
| 22 | - An error in `journalctl --namespace netdata`. |
| 23 | |
| 24 | **Likely causes:** |
| 25 | |
| 26 | | Cause | What to check | |
| 27 | |---|---| |
| 28 | | YAML typo or unknown key | `journalctl --namespace netdata --since "5 minutes ago" \| grep -E 'failed to load configuration\|netflow'`. The plugin uses strict YAML — any unknown key fails parsing. | |
| 29 | | Required GeoIP DB missing (`optional: false`) | Same log search. Look for `failed to load database`. Either fix the path or set `optional: true`. | |
| 30 | | Listen address conflict | Look for `failed to bind`. Another process is on the configured port (default 2055). | |
| 31 | | Validation error | Look for `must be greater than 0` and similar. The plugin validates the full config at startup. | |
| 32 | | `enabled: false` was set | Look for `netflow plugin disabled by config`. The plugin honours this and shuts down cleanly — looks like "not running" if you don't read the log. | |
| 33 | |
| 34 | **Recovery:** |
| 35 | |
| 36 | ```bash |
| 37 | # Read the failure |
| 38 | sudo journalctl --namespace netdata --since "5 minutes ago" | grep -E 'netflow|failed to|error' |
| 39 | |
| 40 | # Validate the YAML (use an online linter or `yamllint`) |
| 41 | yamllint /etc/netdata/netflow.yaml |
| 42 | |
| 43 | # After fixing, restart |
| 44 | sudo systemctl restart netdata |
| 45 | ``` |
| 46 | |
| 47 | ## The plugin starts, but no flows appear |
| 48 | |
| 49 | The plugin is running, but the Network Flows view is empty. |
| 50 | |
| 51 | **First check:** is anything reaching the plugin? |
| 52 | |
| 53 | ```bash |
| 54 | sudo tcpdump -i any -nn -c 50 'udp port 2055' |
| 55 | ``` |
| 56 | |
| 57 | - **No packets in 30 seconds** — exporter not sending, or firewall blocking. Check the exporter's status (`show flow exporter` on Cisco, equivalents elsewhere) and the network path. The plugin can't help here; the data isn't reaching it. |
| 58 | - **Packets arriving** — keep going. |
| 59 | |
| 60 | **Second check:** is the listener bound? |
| 61 | |
| 62 | ```bash |
| 63 | sudo ss -unlp | grep -E ':2055|netflow' |
| 64 | ``` |
| 65 | |
| 66 | If nothing matches, the plugin isn't listening. See "doesn't start" above. |
| 67 | |
| 68 | **Third check:** what do the plugin's own counters say? |
| 69 | |
| 70 | Open `netflow.input_packets` on the standard Netdata charts page. The dimensions tell the story: |
| 71 | |
| 72 | - `udp_received > 0`, `parsed_packets == 0` — datagrams arriving, none decoding successfully. Wrong protocol on the listener, or all datagrams malformed. |
| 73 | - `udp_received > 0`, `parsed_packets > 0`, but no per-protocol counter (`netflow_v9`, `ipfix`, etc.) is moving — the protocol you're sending may be disabled in the plugin config. Check `protocols.v9`, `protocols.ipfix`, etc. in `netflow.yaml`. |
| 74 | - `parse_errors` rising in lockstep with `udp_received` — datagrams aren't valid for the protocols the plugin supports. Capture a small UDP sample with `tcpdump -w` and inspect it with Wireshark. |
| 75 | |
| 76 | ## Partial data — some flows are dropped |
| 77 | |
| 78 | Counters show received traffic but you suspect data loss. |
| 79 | |
| 80 | **Template errors (NetFlow v9, IPFIX):** |
| 81 | |
| 82 | ```bash |
| 83 | # Watch the template_errors dimension |
| 84 | # In the dashboard: netflow.input_packets > template_errors |
| 85 | ``` |
| 86 | |
| 87 | If it's climbing, the exporter is sending data records before their templates. Either: |
| 88 | |
| 89 | - The exporter restarted and the plugin's template cache is stale. Wait for the exporter to send the next template (the cadence depends on the exporter's `template-refresh` configuration — vendor defaults vary widely), or restart the exporter to force an immediate template refresh. |
| 90 | - Templates are sent rarely. Cisco IOS / IOS-XE Flexible NetFlow ships a default `template data timeout` of **600 seconds (10 minutes)**; Juniper and others have their own defaults, often longer. After a plugin restart, you'll see template errors until the next template re-send. **Fix on the router side**: lower the template refresh interval to 60 seconds (the [Quick Start](/docs/network-flows/quick-start.md) configurations show this). |
| 91 | - The exporter is using template IDs that collide with another exporter's templates. Most common cause: two exporters NATted behind the same public IP. Place the plugin inside the NAT boundary or give each exporter a distinct address. |
| 92 | |
| 93 | **UDP kernel drops:** |
| 94 | |
| 95 | The plugin doesn't count these. Check at the OS level: |
| 96 | |
| 97 | ```bash |
| 98 | sudo ss -uamn sport = :2055 # inspect the d<N> field inside skmem:(...) |
| 99 | grep ^Udp: /proc/net/snmp # RcvbufErrors counter (system-wide) |
| 100 | ``` |
| 101 | |
| 102 | `/proc/net/udp` lists open sockets and includes per-socket `drops`; the kernel-wide UDP `RcvbufErrors` total lives under the `Udp:` line of `/proc/net/snmp` (this is what Netdata's own `ipv4.udperrors` chart and the `1m_ipv4_udp_receive_buffer_errors` alert read). |
| 103 | |
| 104 | If drops are occurring, the kernel UDP receive buffer is too small for the burst rate. Tune: |
| 105 | |
| 106 | ```bash |
| 107 | sudo sysctl -w net.core.rmem_max=33554432 |
| 108 | sudo sysctl -w net.core.rmem_default=8388608 |
| 109 | sudo sysctl -w net.core.netdev_max_backlog=250000 |
| 110 | ``` |
| 111 | |
| 112 | Persist in `/etc/sysctl.d/99-netflow.conf`. |
| 113 | |
| 114 | **Per-protocol switch off:** |
| 115 | |
| 116 | ```yaml |
| 117 | protocols: |
| 118 | v5: false # are you accidentally rejecting v5 datagrams? |
| 119 | ``` |
| 120 | |
| 121 | ## Data is wrong — numbers don't match expectations |
| 122 | |
| 123 | **Volume looks doubled:** |
| 124 | |
| 125 | This is the most common report. When a router is configured to export both ingress and egress on each monitored interface — a common configuration; vendor best practice is ingress-only — every packet generates an ingress record AND an egress record, so traffic appears 2× on a single such router. With two routers on the same path doing the same thing, 4×. Filter to one exporter and one interface (`Ingress Interface Name` OR `Egress Interface Name`, pick one) to see real volume. See [Anti-patterns](/docs/network-flows/anti-patterns.md). |
| 126 | |
| 127 | **Bandwidth doesn't match SNMP:** |
| 128 | |
| 129 | Several legitimate causes: |
| 130 | |
| 131 | - **Doubling**, as above. Filter to one exporter and one interface before comparing. |
| 132 | - **Comparing aggregates to a single interface counter.** SNMP `ifInOctets` / `ifOutOctets` is per-interface; an unfiltered flow aggregate sums many interfaces. Compare like-with-like by filtering the dashboard to the same exporter and the same interface (Input or Output, pick one). |
| 133 | - **Sampling rate not honoured by the exporter.** The plugin multiplies each flow's bytes/packets by that flow's own sampling rate. If the exporter doesn't carry the rate (NetFlow v7 has no field for it; v5 sometimes sends 0 instead of the actual rate; v9 / IPFIX without the Sampling Options Template), the plugin treats those records as unsampled and undercounts. |
| 134 | - **SNMP includes layer-2 traffic** (ARP, STP, LLDP, routing protocols) that flow data filters out. Expect SNMP to be 5-15% higher than flow on a healthy collector. More than that, investigate. |
| 135 | |
| 136 | See [Validation and Data Quality](/docs/network-flows/validation.md). |
| 137 | |
| 138 | **AS resolution chain misbehaving:** |
| 139 | |
| 140 | If `SRC_AS` / `DST_AS` are zero everywhere despite the exporter sending them, check the `asn_providers` chain: |
| 141 | |
| 142 | - `[geoip, ...]` — `geoip` is a terminal short-circuit. The chain stops at `geoip` (it returns 0). Reorder: `[flow, routing, geoip]`. |
| 143 | - `[]` (empty) — no validation rejects this. Every AS is forced to 0. |
| 144 | |
| 145 | See [Enrichment](/docs/network-flows/enrichment.md) (the asn_providers chain section). |
| 146 | |
| 147 | **Decapsulation eating non-tunnel traffic:** |
| 148 | |
| 149 | If you've enabled `decapsulation_mode: vxlan` and traffic that isn't VXLAN suddenly disappears from the L2-section path, that's by design — the decap is destructive on non-matching traffic. Standard NetFlow / IPFIX records (no IE 104 / IE 315) are unaffected. |
| 150 | |
| 151 | ## Performance issues |
| 152 | |
| 153 | **High CPU:** |
| 154 | |
| 155 | ```bash |
| 156 | top -p $(pgrep -f netflow-plugin) |
| 157 | ``` |
| 158 | |
| 159 | If `netflow-plugin` is using a lot of CPU: |
| 160 | |
| 161 | - Check `netflow.input_packets` — high `udp_received` rate? You're at the limit of what one core can do for the post-decode hot path. Each instance is single-process; you can't scale horizontally on one host. |
| 162 | - If `udp_received` is moderate but CPU is high, classifier rules with complex regex might be the cause. Check `enrichment.classifier_cache_duration` — if too short, classifiers re-evaluate too often. |
| 163 | - Investigate with `perf top` or similar to find the hot function. |
| 164 | |
| 165 | See [Sizing and Capacity Planning](/docs/network-flows/sizing-capacity.md) for measured throughput limits on this hardware class. |
| 166 | |
| 167 | **Memory growth:** |
| 168 | |
| 169 | ```bash |
| 170 | # Watch the resident memory chart over time |
| 171 | # netflow.memory_resident_bytes - rss dimension |
| 172 | ``` |
| 173 | |
| 174 | - If `rss` climbs and `netflow.memory_accounted_bytes` shows `unaccounted` growing, that's an unattributed allocation — could be allocator fragmentation, possibly a leak. |
| 175 | - If `tier_indexes` or `open_tiers` is the climbing dimension, ingest is outpacing tier flushes. Check `netflow.materialized_tier_ops` for `flushes` rate and `*_errors`. |
| 176 | - If `netflow.decoder_scopes` is growing without bound, your exporter is rotating template IDs. Investigate per-router behaviour. |
| 177 | |
| 178 | **Disk fill:** |
| 179 | |
| 180 | ```bash |
| 181 | sudo du -sh /var/cache/netdata/flows/* |
| 182 | ``` |
| 183 | |
| 184 | Default retention is `10GB / 7d` per tier. The default is applied separately to raw, 1m, 5m, and 1h tiers, so total can reach roughly 40 GB plus some. If your config left this default and your collector is busy, expect to hit the size cap before the time cap. See [Configuration](/docs/network-flows/configuration.md) for per-tier overrides — most production deployments need them. |
| 185 | |
| 186 | ## Things that look like bugs but aren't |
| 187 | |
| 188 | - **Traffic appears 2×.** When the router is configured to export both ingress + egress (common, but not universal — vendor best practice is ingress-only), the same packet is recorded once on entry and once on exit on a single router. Filter to one exporter and one interface (`Ingress Interface Name` or `Egress Interface Name`, pick one). |
| 189 | - **Bidirectional conversations show twice.** A→B and B→A are real, distinct flows representing different packets going each way. Their volumes are usually asymmetric. Filter by `Source AS Name` (your network) for outbound or `Destination AS Name` (your network) for inbound to see one side. |
| 190 | - **City map empty over long windows.** City + lat/lon are raw-tier-only. Default raw-tier retention is short. Use the country or state map for long ranges. |
| 191 | - **`__overflow__` row in results.** Your aggregation produced more groups than `query_max_groups`. Narrow the filter or reduce group-by depth. |
| 192 | - **30-second query timeout.** Hard limit. Narrow time range, add filters, or reduce group-by depth. |
| 193 | - **Sampled byte counts not exact.** sFlow is statistical by design; even NetFlow with sampling is an estimate. Cross-check against SNMP for sanity, accept some divergence. |
| 194 | - **`enabled: false` makes the plugin look crashed.** It's intentional — the plugin tells the parent to stop respawning it. Look for the "disabled by config" line in the journal. |
| 195 | |
| 196 | ## Diagnostic command quick reference |
| 197 | |
| 198 | ```bash |
| 199 | # What's happening |
| 200 | sudo journalctl --namespace netdata --since "10 minutes ago" | grep -iE 'netflow|geoip|bmp|bioris|network-sources' |
| 201 | |
| 202 | # What's arriving on the wire |
| 203 | sudo tcpdump -i any -nn -c 50 'udp port 2055' |
| 204 | |
| 205 | # Is the listener bound |
| 206 | sudo ss -unlp | grep 2055 |
| 207 | |
| 208 | # UDP kernel drops |
| 209 | sudo ss -uamn sport = :2055 |
| 210 | grep ^Udp: /proc/net/snmp |
| 211 | |
| 212 | # Disk usage by tier |
| 213 | sudo du -sh /var/cache/netdata/flows/* |
| 214 | |
| 215 | # Process resources |
| 216 | top -p $(pgrep -f netflow-plugin) |
| 217 | |
| 218 | # Capture a sample for offline analysis |
| 219 | sudo tcpdump -w /tmp/netflow-sample.cap -c 200 'udp port 2055' |
| 220 | ``` |
| 221 | |
| 222 | ## When to file an issue |
| 223 | |
| 224 | Collect this before opening a bug report: |
| 225 | |
| 226 | - Plugin version (`netdata --version` from the running daemon). |
| 227 | - A sample of `netflow.input_packets` chart for the failure window — all dimensions visible. |
| 228 | - A sample of `netflow.memory_resident_bytes` if performance-related. |
| 229 | - A small packet-capture file (`tcpdump -w` from the agent's interface) reproducing the issue. |
| 230 | - Sanitised `netflow.yaml` (redact internal IPs, customer names, secrets). |
| 231 | - Relevant log lines from `journalctl --namespace netdata`. |
| 232 | |
| 233 | Open issues against [github.com/netdata/netdata](https://github.com/netdata/netdata) with `area/collectors/netflow` in the title. |
| 234 | |
| 235 | ## What's next |
| 236 | |
| 237 | - [Plugin Health Charts](/docs/network-flows/visualization/dashboard-cards.md) — The charts referenced above. |
| 238 | - [Validation and Data Quality](/docs/network-flows/validation.md) — How to spot silent data corruption. |
| 239 | - [Anti-patterns](/docs/network-flows/anti-patterns.md) — Why some "weird" results are actually normal. |
| 240 | - [Configuration](/docs/network-flows/configuration.md) — Tuning that affects most of the symptoms above. |