docs: add "With NVIDIA GPUs monitoring" to docker install (#17167)
Ilya Mashchenko committed
Mar 14, 2024 at 22:17 UTC
382bc6c23b3c8c301292296525c3d1955cece71b
2 files changed
+42
-81
packaging/docker/README.md
+37
@@ -172,6 +172,43 @@ Add `- /run/dbus:/run/dbus:ro` to the netdata service `volumes`.
172
</TabItem>
173
</Tabs>
174
175
+### With NVIDIA GPUs monitoring
176
+
177
+
178
+Monitoring NVIDIA GPUs requires:
179
+
180
+- Using official [NVIDIA driver](https://www.nvidia.com/Download/index.aspx).
181
+- Installing [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
182
+- Allowing the Netdata container to access GPU resources.
183
+
184
+
185
+<Tabs>
186
+<TabItem value="docker_run" label="docker run">
187
+
188
+<h3> Using the <code>docker run</code> command </h3>
189
+
190
+Add `--gpus 'all,capabilities=utility'` to your `docker run`.
191
+
192
+</TabItem>
193
+<TabItem value="docker compose" label="docker-compose">
194
+
195
+<h3> Using the <code>docker-compose</code> command</h3>
196
+
197
+Add the following to the netdata service.
198
+
199
+```yaml
200
+ deploy:
201
+ resources:
202
+ reservations:
203
+ devices:
204
+ - driver: nvidia
205
+ count: all
206
+ capabilities: [gpu]
207
+```
208
+
209
+</TabItem>
210
+</Tabs>
211
+
212
### With host-editable configuration
213
214
Use a [bind mount](https://docs.docker.com/storage/bind-mounts/) for `/etc/netdata` rather than a volume.
src/collectors/python.d.plugin/nvidia_smi/README.md
+5
-81
@@ -11,22 +11,18 @@ learn_rel_path: "Integrations/Monitor/Devices"
11
12
Monitors performance metrics (memory usage, fan speed, pcie bandwidth utilization, temperature, etc.) using `nvidia-smi` cli tool.
13
14
-## Requirements and Notes
14
+## Requirements
15
16
-- You must have the `nvidia-smi` tool installed and your NVIDIA GPU(s) must support the tool. Mostly the newer high end models used for AI / ML and Crypto or Pro range, read more about [nvidia_smi](https://developer.nvidia.com/nvidia-system-management-interface).
17
-- You must enable this plugin, as its disabled by default due to minor performance issues:
16
+- The `nvidia-smi` tool installed and your NVIDIA GPU(s) must support the tool. Mostly the newer high end models used for AI / ML and Crypto or Pro range, read more about [nvidia_smi](https://developer.nvidia.com/nvidia-system-management-interface).
17
+- Enable this plugin, as it's disabled by default due to minor performance issues:
18
```bash
19
cd /etc/netdata # Replace this path with your Netdata config directory, if different
20
sudo ./edit-config python.d.conf
21
```
22
Remove the '#' before nvidia_smi so it reads: `nvidia_smi: yes`.
23
-
23
- On some systems when the GPU is idle the `nvidia-smi` tool unloads and there is added latency again when it is next queried. If you are running GPUs under constant workload this isn't likely to be an issue.
25
-- Currently the `nvidia-smi` tool is being queried via cli. Updating the plugin to use the nvidia c/c++ API directly should resolve this issue. See discussion here: <https://github.com/netdata/netdata/pull/4357>
26
-- Contributions are welcome.
27
-- Make sure `netdata` user can execute `/usr/bin/nvidia-smi` or wherever your binary is.
28
-- If `nvidia-smi` process [is not killed after netdata restart](https://github.com/netdata/netdata/issues/7143) you need to off `loop_mode`.
29
-- `poll_seconds` is how often in seconds the tool is polled for as an integer.
24
+
25
+If using Docker, see [Netdata Docker container with NVIDIA GPUs monitoring](https://github.com/netdata/netdata/tree/master/packaging/docker#with-nvidia-gpus-monitoring).
26
27
## Charts
28
@@ -83,75 +79,3 @@ Now you can manually run the `nvidia_smi` module in debug mode:
79
```bash
80
./python.d.plugin nvidia_smi debug trace
81
```
86
-
87
-## Docker
88
-
89
-GPU monitoring in a docker container is possible with [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) installed on the host system, and `gcompat` added to the `NETDATA_EXTRA_APK_PACKAGES` environment variable.
90
-
91
-Sample `docker-compose.yml`
92
-```yaml
93
-version: '3'
94
-services:
95
- netdata:
96
- image: netdata/netdata
97
- container_name: netdata
98
- hostname: example.com # set to fqdn of host
99
- ports:
100
- - 19999:19999
101
- restart: unless-stopped
102
- cap_add:
103
- - SYS_PTRACE
104
- security_opt:
105
- - apparmor:unconfined
106
- environment:
107
- - NETDATA_EXTRA_APK_PACKAGES=gcompat
108
- volumes:
109
- - netdataconfig:/etc/netdata
110
- - netdatalib:/var/lib/netdata
111
- - netdatacache:/var/cache/netdata
112
- - /etc/passwd:/host/etc/passwd:ro
113
- - /etc/group:/host/etc/group:ro
114
- - /proc:/host/proc:ro
115
- - /sys:/host/sys:ro
116
- - /etc/os-release:/host/etc/os-release:ro
117
- deploy:
118
- resources:
119
- reservations:
120
- devices:
121
- - driver: nvidia
122
- count: all
123
- capabilities: [gpu]
124
-
125
-volumes:
126
- netdataconfig:
127
- netdatalib:
128
- netdatacache:
129
-```
130
-
131
-Sample `docker run`
132
-```yaml
133
-docker run -d --name=netdata \
134
- -p 19999:19999 \
135
- -e NETDATA_EXTRA_APK_PACKAGES=gcompat \
136
- -v netdataconfig:/etc/netdata \
137
- -v netdatalib:/var/lib/netdata \
138
- -v netdatacache:/var/cache/netdata \
139
- -v /etc/passwd:/host/etc/passwd:ro \
140
- -v /etc/group:/host/etc/group:ro \
141
- -v /proc:/host/proc:ro \
142
- -v /sys:/host/sys:ro \
143
- -v /etc/os-release:/host/etc/os-release:ro \
144
- --restart unless-stopped \
145
- --cap-add SYS_PTRACE \
146
- --security-opt apparmor=unconfined \
147
- --gpus all \
148
- netdata/netdata
149
-```
150
-
151
-### Docker Troubleshooting
152
-To troubleshoot `nvidia-smi` in a docker container, first confirm that `nvidia-smi` is working on the host system. If that is working correctly, run `docker exec -it netdata nvidia-smi` to confirm it's working within the docker container. If `nvidia-smi` is fuctioning both inside and outside of the container, confirm that `nvidia-smi: yes` is uncommented in `python.d.conf`.
153
-```bash
154
-docker exec -it netdata bash
155
-cd /etc/netdata
156
-./edit-config python.d.conf
157
-```