Add guide for troubleshooting with eBPF metrics (#9352)
* Init guide * Continue work * Continuing work * Finish first draft... finally * Fix broken links * Add links, copyedit pass * Fixes for Thiago * Fix typo and replace image * Add to unfinished sentence
Joel Hans committed
Jun 22, 2020 at 16:08 UTC
0155d313179ee3c05adf3cd5b31b41aef0ca727d
3 files changed
+278
-3
collectors/apps.plugin/README.md
+6
-1
@@ -160,7 +160,12 @@ There are a few command line options you can pass to `apps.plugin`. The list of
160
161
### Integration with eBPF
162
163
-If you don't see charts under the **eBPF syscall** or **eBPF net** sections, you should edit your [`ebpf.conf`](/collectors/ebpf.plugin/README.md#ebpf-programs) file to ensure the eBPF program is enabled.
163
+If you don't see charts under the **eBPF syscall** or **eBPF net** sections, you should edit your
164
+[`ebpf.conf`](/collectors/ebpf.plugin/README.md#ebpf-programs) file to ensure the eBPF program is enabled.
165
+
166
+Also see our [guide on troubleshooting apps with eBPF
167
+metrics](/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md) for ideas on how to interpret these charts in a
168
+few scenarios.
169
170
## Permissions
171
collectors/ebpf.plugin/README.md
+3
-2
@@ -16,8 +16,9 @@ variety of kernel functions, such as `do_sys_open`, `__close_fd`, `vfs_read`, `v
16
`return` program also monitors the return of each kernel functions to deliver more granular metrics about how your
17
system and its applications interact with the Linux kernel.
18
19
-We expect eBPF monitoring to be particularly valuable in observing and debugging how the Linux kernel handles custom
20
-applications.
19
+eBPF monitoring can help you troubleshoot and debug how applications interact with the Linux kernel. See our [guide on
20
+troubleshooting apps with eBPF metrics](/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md) for configuration
21
+and troubleshooting tips.
22
23
<figure>
24
<img src="https://user-images.githubusercontent.com/1153921/74746434-ad6a1e00-5222-11ea-858a-a7882617ae02.png" alt="An example of VFS charts, made possible by the eBPF collector plugin" />
docs/guides/troubleshoot/monitor-debug-applications-ebpf.md
new
+269
@@ -0,0 +1,269 @@
1
+<!--
2
+title: "Monitor, troubleshoot, and debug applications with eBPF metrics"
3
+description: "Use Netdata's built-in eBPF metrics collector to monitor, troubleshoot, and debug your custom application using low-level kernel feedback."
4
+image: /img/seo/guides/troubleshoot/monitor-debug-applications-ebpf.png
5
+-->
6
+
7
+# Monitor, troubleshoot, and debug applications with eBPF metrics
8
+
9
+When trying to troubleshoot or debug a finicky application, there's no such thing as too much information. At Netdata,
10
+we developed programs that connect to the [_extended Berkeley Packet Filter_ (eBPF) virtual
11
+machine](/collectors/ebpf.plugin/README.md) to help you see exactly how specific applications are interacting with the
12
+Linux kernel. With these charts, you can root out bugs, discover optimizations, diagnose memory leaks, and much more.
13
+
14
+This means you can see exactly how often, and in what volume, the application creates processes, opens files, writes to
15
+filesystem using virtual filesystem (VFS) functions, and much more. Even better, the eBPF collector gathers metrics at
16
+an _event frequency_, which is even faster than Netdata's beloved 1-second granularity. When you troubleshoot and debug
17
+applications with eBPF, rest assured you miss not even the smallest meaningful event.
18
+
19
+Using this guide, you'll learn the fundamentals of setting up Netdata to give you kernel-level metrics from your
20
+application so that you can monitor, troubleshoot, and debug to your heart's content.
21
+
22
+## Configure `apps.plugin` to recognize your custom application
23
+
24
+To start troubleshooting an application with eBPF metrics, you need to ensure your Netdata dashboard collects and
25
+displays those metrics independent from any other process.
26
+
27
+You can use the `apps_groups.conf` file to configure which applications appear in charts generated by
28
+[`apps.plugin`](/collectors/apps.plugin/README.md). Once you edit this file and create a new group for the application
29
+you want to monitor, you can see how it's interacting with the Linux kernel via real-time eBPF metrics.
30
+
31
+Let's assume you have an application that runs on the process `custom-app`. To monitor eBPF metrics for that application
32
+separate from any others, you need to create a new group in `apps_groups.conf` and associate that process name with it.
33
+
34
+Open the `apps_groups.conf` file in your Netdata configuration directory.
35
+
36
+```bash
37
+cd /etc/netdata # Replace this path with your Netdata config directory
38
+sudo ./edit-config apps_groups.conf
39
+```
40
+
41
+Scroll down past the explanatory comments and stop when you see `# NETDATA processes accounting`. Above that, paste in
42
+the following text, which creates a new `dev` group with the `custom-app` process. Replace `custom-app` with the name of
43
+your application's process name.
44
+
45
+Your file should now look like this:
46
+
47
+```conf
48
+...
49
+# -----------------------------------------------------------------------------
50
+# Custom applications to monitor with apps.plugin and ebpf.plugin
51
+
52
+dev: custom-app
53
+
54
+# -----------------------------------------------------------------------------
55
+# NETDATA processes accounting
56
+...
57
+```
58
+
59
+Restart Netdata with `sudo service netdata restart` or the appropriate method for your system to begin seeing metrics
60
+for this particular group+process. You can also add additional processes to the same group.
61
+
62
+You can set up `apps_groups.conf` to more show more precise eBPF metrics for any application or service running on your
63
+system, even if it's a standard package like Redis, Apache, or any other [application/service Netdata collects
64
+from](/collectors/COLLECTORS.md).
65
+
66
+```conf
67
+# -----------------------------------------------------------------------------
68
+# Custom applications to monitor with apps.plugin and ebpf.plugin
69
+
70
+dev: custom-app
71
+database: *redis*
72
+apache: *apache*
73
+
74
+# -----------------------------------------------------------------------------
75
+# NETDATA processes accounting
76
+...
77
+```
78
+
79
+Now that you have `apps_groups.conf` set up to monitor your application/service, you can also set up the eBPF collector
80
+to show other charts that will help you debug and troubleshoot how it interacts with the Linux kernel.
81
+
82
+## Configure the eBPF collector to monitor errors
83
+
84
+The eBPF collector has [two possible modes](/collectors/ebpf.plugin#ebpf-load-mode): `entry` and `return`. The default
85
+is `entry`, and only monitors calls to kernel functions, but the `return` also monitors and charts _whether these calls
86
+return in error_.
87
+
88
+Let's turn on the `return` mode for more granularity when debugging Firefox's behavior.
89
+
90
+```bash
91
+cd /etc/netdata # Replace this path with your Netdata config directory
92
+sudo ./edit-config ebpf.conf
93
+```
94
+
95
+Replace `entry` with `return`:
96
+
97
+```conf
98
+[global]
99
+ ebpf load mode = return
100
+ disable apps = no
101
+
102
+[ebpf programs]
103
+ process = yes
104
+ network viewer = yes
105
+```
106
+
107
+Restart Netdata with `sudo service netdata restart` or the appropriate method for your system.
108
+
109
+## Get familiar with per-application eBPF metrics and charts
110
+
111
+Visit the Netdata dashboard at `http://NODE:19999`, replacing `NODE` with the hostname or IP of the system you're using
112
+to monitor this application. Scroll down to the **Applications** section. These charts now feature a `firefox` dimension
113
+with metrics specific to that process.
114
+
115
+Pay particular attention to the charts in the **ebpf syscall** and **ebpf net** sections. These charts are populated by
116
+low-level Linux kernel metrics thanks to eBPF, and showcase the volume of calls to open/close files, call functions like
117
+`do_fork`, IO activity on the VFS, and much more.
118
+
119
+See the [eBPF collector documentation](/collectors/ebpf.plugin/README.md#integration-with-appsplugin) for the full list
120
+of per-application charts.
121
+
122
+Let's show some examples of how you can first identify normal eBPF patterns, then use that knowledge to idenfity
123
+anomalies in a few simulated scenarios.
124
+
125
+For example, the following screenshot shows the number of open files, failures to open files, and closed files on a
126
+Debian 10 system. The first spike is from configuring/compiling a small C program, then from running Apache's `ab` tool
127
+to benchmark an Apache web server.
128
+
129
+
131
+
132
+In these charts, you can see first a spike in syscalls to open and close files from the configure/build process,
133
+followed by a similar spike from the Apache benchmark.
134
+
135
+> 👋 Don't forget that you can view chart data directly via Netdata's API!
136
+>
137
+> For example, open your browser and navigate to `http://NODE:19999/api/v1/data?chart=apps.file_open`, replacing `NODE`
138
+> with the IP address or hostname of your Agent. The API returns JSON of that chart's dimensions and metrics, which you
139
+> can use in other operations.
140
+>
141
+> To see other charts, replace `apps.file_open` with the context of the chart you want to see data for.
142
+>
143
+> To see all the API options, visit our [Swagger
144
+> documentation](https://editor.swagger.io/?url=https://raw.githubusercontent.com/netdata/netdata/master/web/api/netdata-swagger.yaml)
145
+> and look under the **/data** section.
146
+
147
+## Troubleshoot and debug applications with eBPF
148
+
149
+The actual method of troubleshooting and debugging any application with Netdata's eBPF metrics depends on the
150
+application, its place within your stack, and the type of issue you're trying to root cause. This guide won't be able to
151
+explain how to troubleshoot _any_ application with eBPF metrics, but it should give you some ideas on how to start with
152
+your own systems.
153
+
154
+The value of using Netdata to collect and visualize eBPF metrics is that you don't have to rely on existing (complex)
155
+command line eBPF programs or, even worse, write your own eBPF program to get the information you need.
156
+
157
+Let's walk through some scenarios where you might find value in eBPF metrics.
158
+
159
+### Benchmark application performance
160
+
161
+You can use eBPF metrics to profile the performance of your applications, whether they're custom or a standard Linux
162
+service, like a web server or database.
163
+
164
+For example, look at the charts below. The first spike represents running a Redis benchmark _without_ pipelining
165
+(`redis-benchmark -n 1000000 -t set,get -q`). The second spike represents the same benchmark _with_ pipelining
166
+(`redis-benchmark -n 1000000 -t set,get -q -P 16`).
167
+
168
+
170
+
171
+The performance optimization is clear from the speed at which the benchmark finished (the horizontal length of the
172
+spike) and the reduced write/read syscalls and bytes written to disk.
173
+
174
+You can run similar performance benchmarks against any application, view the results on a Linux kernel level, and
175
+continuously improve the performance of your infrastructure.
176
+
177
+### Inspect for leaking file descriptors
178
+
179
+If your application runs fine and then only crashes after a few hours, leaking file descriptors may be to blame.
180
+
181
+Check the **Number of open files (apps.file_open)** and **Files closed (apps.file_closed)** for discrepancies. These
182
+metrics should be more or less equal. If they diverge, with more open files than closed, your application may not be
183
+closing file descriptors properly.
184
+
185
+See, for example, the volume of files opened and closed by `apps.plugin` itself. Because the eBPF collector is
186
+monitoring these syscalls at an event level, you can see at any given second that the open and closed numbers as equal.
187
+
188
+This isn't to say Netdata is _perfect_, but at least `apps.plugin` doesn't have a file descriptor problem.
189
+
190
+
192
+
193
+### Pin down syscall failures
194
+
195
+If you enabled the eBPF collector's `return` mode as mentioned [in a previous
196
+step](#configure-the-ebpf-collector-to-monitor-errors), you can view charts related to how often a given application's
197
+syscalls return in failure.
198
+
199
+By understanding when these failures happen, and when, you might be able to diagnose a bug in your application.
200
+
201
+To diagnose potential issues with an application, look at the **Fails to open files (apps.file_open_error)**, **Fails to
202
+close files (apps.file_close_error)**, **Fails to write (apps.vfs_write_error)**, and **Fails to read
203
+(apps.vfs_read_error)** charts for failed syscalls coming from your application. If you see any, look to the surrounding
204
+charts for anomalies at the same time frame, or correlate with other activity in the application or on the system to get
205
+closer to the root cause.
206
+
207
+### Investigate zombie processes
208
+
209
+Look for the trio of **Process started (apps.process_create)**, **Threads started (apps.thread_create)**, and **Tasks
210
+closed (apps.task_close)** charts to investigate situations where an application inadvertently leaves [zombie
211
+processes](https://en.wikipedia.org/wiki/Zombie_process).
212
+
213
+These processes, which are terminated and don't use up system resources, can still cause issues if your system runs out
214
+of available PIDs to allocate.
215
+
216
+For example, the chart below demonstrates a [zombie factory
217
+program](https://www.refining-linux.org/archives/7-Dr.-Frankenlinux-or-how-to-create-zombie-processes.html) in action.
218
+
219
+
221
+
222
+Starting at 14:51:49, Netdata sees the `zombie` group creating one new process every second, but no closed tasks. This
223
+continues for roughly 30 seconds, at which point the factory program was killed with `SIGINT`, which results in the 31
224
+closed tasks in the subsequent second.
225
+
226
+Zombie processes may not be catastrophic, but if you're developing an application on Linux, you should eliminate them.
227
+If a service in your stack creates them, you should consider filing a bug report.
228
+
229
+## View eBPF metrics in Netdata Cloud
230
+
231
+You can also show per-application eBPF metrics in Netdata Cloud. This could be particularly useful if you're running the
232
+same application on multiple systems and want to correlate how it performs on each target, or if you want to share your
233
+findings with someone else on your team.
234
+
235
+If you don't already have a Netdata Cloud account, go [sign in](https://app.netdata.cloud) and get started for free.
236
+Read the [get started with Cloud guide](https://learn.netdata.cloud/docs/cloud/get-started) for a walkthrough of node
237
+claiming and other fundamentals.
238
+
239
+Add more charts to the Nodes view by clicking on the gear icon at the far end of the table. Click on the **Context**
240
+input and scroll until you find the eBPF chart you're interested in, or type in the name of the context directly. Maybe
241
+something like `apps.vfs_write_call`? Next, click on the **Dimensions** input and find the `dev` dimension, or maybe
242
+`apache`/`redis` if you set your `apps_groups.conf` up like the examples above.
243
+
244
+Click **Save** to add the chart for all your claimed nodes. Now you can see how your application interacts with multiple
245
+Linux kernels on multiple Linux systems.
246
+
247
+Now that you can see these metrics in Netdata Cloud, you can [invite your
248
+team](https://learn.netdata.cloud/docs/cloud/collaborate/invite-your-team) and share your findings with others.
249
+
250
+## What's next?
251
+
252
+Debugging and troubleshooting an application takes a special combination of practice, experience, and sheer luck. With
253
+Netdata's eBPF metrics to back you up, you can rest assured that you see every minute detail of how your application
254
+interacts with the Linux kernel.
255
+
256
+If you're still trying to wrap your head aroud what we offer, be sure to read up on our accompanying documentation and
257
+other resources on eBPF monitoring with Netdata:
258
+
259
+- [eBPF collector](/collectors/ebpf.plugin/README.md)
260
+- [eBPF's integration with `apps.plugin`](/collectors/apps.plugin/README.md#integration-with-ebpf)
261
+- [Linux eBPF monitoring with Netdata](https://www.netdata.cloud/blog/linux-ebpf-monitoring-with-netdata/)
262
+
263
+The scenarios described above are just the beginning when it comes to troubleshooting with eBPF metrics. We're excited
264
+to explore others and see what our community dreams up. If you have other use cases, whether simulated or real-world,
265
+we'd love to hear them: [info@netdata.cloud](mailto:info@netdata.cloud).
266
+
267
+Happy troubleshooting!
268
+
269
+[](<>)