Add Docker instructions to enable Nvidia GPUs (#14924)
* Add Docker instructions `gcompat` adds a glibc library for binaries linked against glibc, like nvidia drivers, which don't natively work with musl distributions like alpine. * Edit duplicate heading
D34DC3N73R committed
Apr 20, 2023 at 09:05 UTC
5b676d5f912fc27a126ff4ff6ba5b35da9cf930c
1 file changed
+71
-3
collectors/python.d.plugin/nvidia_smi/README.md
+71
-3
@@ -11,9 +11,6 @@ learn_rel_path: "Integrations/Monitor/Devices"
11
12
Monitors performance metrics (memory usage, fan speed, pcie bandwidth utilization, temperature, etc.) using `nvidia-smi` cli tool.
13
14
-> **Warning**: this collector does not work when the Netdata Agent is [running in a container](https://github.com/netdata/netdata/blob/master/packaging/docker/README.md).
15
-
16
-
14
## Requirements and Notes
15
16
- You must have the `nvidia-smi` tool installed and your NVIDIA GPU(s) must support the tool. Mostly the newer high end models used for AI / ML and Crypto or Pro range, read more about [nvidia_smi](https://developer.nvidia.com/nvidia-system-management-interface).
@@ -87,3 +84,74 @@ Now you can manually run the `nvidia_smi` module in debug mode:
84
./python.d.plugin nvidia_smi debug trace
85
```
86
87
+## Docker
88
+
89
+GPU monitoring in a docker container is possible with [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) installed on the host system, and `gcompat` added to the `NETDATA_EXTRA_APK_PACKAGES` environment variable.
90
+
91
+Sample `docker-compose.yml`
92
+```yaml
93
+version: '3'
94
+services:
95
+ netdata:
96
+ image: netdata/netdata
97
+ container_name: netdata
98
+ hostname: example.com # set to fqdn of host
99
+ ports:
100
+ - 19999:19999
101
+ restart: unless-stopped
102
+ cap_add:
103
+ - SYS_PTRACE
104
+ security_opt:
105
+ - apparmor:unconfined
106
+ environment:
107
+ - NETDATA_EXTRA_APK_PACKAGES=gcompat
108
+ volumes:
109
+ - netdataconfig:/etc/netdata
110
+ - netdatalib:/var/lib/netdata
111
+ - netdatacache:/var/cache/netdata
112
+ - /etc/passwd:/host/etc/passwd:ro
113
+ - /etc/group:/host/etc/group:ro
114
+ - /proc:/host/proc:ro
115
+ - /sys:/host/sys:ro
116
+ - /etc/os-release:/host/etc/os-release:ro
117
+ deploy:
118
+ resources:
119
+ reservations:
120
+ devices:
121
+ - driver: nvidia
122
+ count: all
123
+ capabilities: [gpu]
124
+
125
+volumes:
126
+ netdataconfig:
127
+ netdatalib:
128
+ netdatacache:
129
+```
130
+
131
+Sample `docker run`
132
+```yaml
133
+docker run -d --name=netdata \
134
+ -p 19999:19999 \
135
+ -e NETDATA_EXTRA_APK_PACKAGES=gcompat \
136
+ -v netdataconfig:/etc/netdata \
137
+ -v netdatalib:/var/lib/netdata \
138
+ -v netdatacache:/var/cache/netdata \
139
+ -v /etc/passwd:/host/etc/passwd:ro \
140
+ -v /etc/group:/host/etc/group:ro \
141
+ -v /proc:/host/proc:ro \
142
+ -v /sys:/host/sys:ro \
143
+ -v /etc/os-release:/host/etc/os-release:ro \
144
+ --restart unless-stopped \
145
+ --cap-add SYS_PTRACE \
146
+ --security-opt apparmor=unconfined \
147
+ --gpus all \
148
+ netdata/netdata
149
+```
150
+
151
+### Docker Troubleshooting
152
+To troubleshoot `nvidia-smi` in a docker container, first confirm that `nvidia-smi` is working on the host system. If that is working correctly, run `docker exec -it netdata nvidia-smi` to confirm it's working within the docker container. If `nvidia-smi` is fuctioning both inside and outside of the container, confirm that `nvidia-smi: yes` is uncommented in `python.d.conf`.
153
+```bash
154
+docker exec -it netdata bash
155
+cd /etc/netdata
156
+./edit-config python.d.conf
157
+```