@cryptotaxi247 / netdata-1 / commits / d98b20129

Improve system-info.sh to better handle certain cases when gathering info on the system's disk capacity. (#7902)

* Remove trailing whitespace in system-info.sh. * Fix handling of APFS on macOS. APFS can have multiple volumes in a single partition, which means that the same functional 'volume' can appear multiple times in the output of `df`. Duplicate lines for such volumes will show the same total size and available space along with a common prefix for the device name. This updates the parsing logic for `df` on macOS to account for this by deduplicating lines in the `df` output that have the same total size, available space, and same normalized device name. This has the potential to incorrectly under-account space in some cases, but the liklihood of that happeing is much less than the certainty of overaccounting space on standard APFS configurations. * Properly handle VirtIO block devices when using /sys. The VirtIO Block device driver uses a dynamically allocated device major number, meaning that we can't trivially match on it. This updates the handling to properly look it up in `/proc/devices` instead of just using the whole dynamic device number range. * Add handling for NVMe block devices in sysfs code. They use dynamic major numbers just like VirtIO Block devices do. * Switch to device major discovery in /proc/devices for all device types. This converts the code to use `/proc/devices` to look up correct device major numbers for block devices taht we treat as disks just like we are already doing for those that have dynamically assigned numbers. This makes the code both more robust and easier to understand and modify. This also excludes some particularly old hardware that we were originally looking for. If needed, we can add in the required device names, but for now it's better to keep the list concise. * Correct handling of device major discovery. We need to strip leading whitespace before calling cut, not after. * Only use /sys/block if we can read /proc/devices. We use `/proc/devices` to do device number lookups that we then use to filter devices under `/sys/block`. As a result, if we can't read `/proc/devices`, then we won't actually parse anything out of `/sys/block` either, so we need to just fall back to parsing `df` output. * Deduplicate `df` output by device name on Linux. This ensures that we properly handle BTRFS subvolumes, counting each actual volume only once. * Use POSIX math expansion instead of awk to sum disk sizes. This avoids the rather annoying habit of AWK of printing integers in scientific notation instead of as exact values. * Correct `sed` options for POSIX complaince. * Fix disk info fetching for macOS> POSIX tools, as found on macOS, lack a number of rather useful filtering and sorting features, so we need to get rather creative with the handling on macOS to make the disk space computation work correctly. This unfortunately makes the calculation a bit less reliable than it would have been had the existing calculations worked correctly, but it's the best I can come up with without making things exponentiall more complicated. * Properly handle sector size when using sysfs.

Austin S. Hemmelgarn committed May 21, 2020 at 09:16 UTC d98b2012920d8412c1225da2df2b0ce83860a691
1 file changed +21 -18
daemon/system-info.sh
+21 -18
@@ -105,6 +105,7 @@ else
105 CONTAINER_OS_DETECTION="/etc/os-release"
106 fi
107
108 + # shellcheck disable=SC2153
109 if [ "${NAME}" = "unknown" ] || [ "${VERSION}" = "unknown" ] || [ "${ID}" = "unknown" ]; then
110 if [ -f "/etc/lsb-release" ]; then
111 if [ "${OS_DETECTION}" = "unknown" ]; then
@@ -121,9 +122,9 @@ else
122 if [ "${ID}" = "unknown" ]; then CONTAINER_ID="${DISTRIB_CODENAME}"; fi
123 fi
124 if [ -n "$(command -v lsb_release 2>/dev/null)" ]; then
124 - if [ "${OS_DETECTION}" = "unknown" ]; then
125 + if [ "${OS_DETECTION}" = "unknown" ]; then
126 CONTAINER_OS_DETECTION="lsb_release"
126 - else
127 + else
128 CONTAINER_OS_DETECTION="Mixed"
129 fi
130 if [ "${NAME}" = "unknown" ]; then CONTAINER_NAME="$(lsb_release -is 2>/dev/null)"; fi
@@ -307,8 +308,7 @@ if [ "${KERNEL_NAME}" = "Darwin" ]; then
308 fi
309
310 DISK_DETECTION="df"
310 - total="$(/bin/df -k -t ${types} | tail -n +1 | awk '{s+=$2} END {print s}')"
311 - DISK_SIZE="$((total * 1024))"
311 + DISK_SIZE=$(($(/bin/df -k -t ${types} | tail -n +2 | sed -E 's/\/dev\/disk([[:digit:]]*)s[[:digit:]]*/\/dev\/disk\1/g' | sort -k 1 | awk -F ' ' '{s=$NF;for(i=NF-1;i>=1;i--)s=s FS $i;print s}' | uniq -f 9 | awk '{print $8}' | tr '\n' '+' | rev | cut -f 2- -d '+' | rev) * 1024))
312 elif [ "${KERNEL_NAME}" = FreeBSD ] ; then
313 types='ufs'
314
@@ -320,16 +320,20 @@ elif [ "${KERNEL_NAME}" = FreeBSD ] ; then
320 total="$(df -t ${types} -c -k | tail -n 1 | awk '{print $2}')"
321 DISK_SIZE="$((total * 1024))"
322 else
323 - if [ -d /sys/block ] ; then
324 - # List of device majors we actually count towards total disk space.
325 - # The meanings of these can be found in `Documentation/admin-guide/devices.txt` in the Linux sources.
326 - # The ':' surrounding each number are important for matching.
327 - dev_major_whitelist=':3:8:9:21:22:28:31:33:34:44:45:47:48:49:50:51:52:53:54:55:56:57:65:66:67:68:69:70:71:72:73:74:75:76:77:78:79:88:89:90:91:93:94:96:98:101:104:105:106:107:108:109:110:111:112:114:116:128:129:130:131:132:134:135:136:137:138:139:140:141:142:143:153:160:161:179:180:202:256:257:'
328 -
329 - if [ "${VIRTUALIZATION}" != "unknown" ] ; then
330 - # We're running virtualized, add the local range of device major numbers so that we catch paravirtualized block devices.
331 - dev_major_whitelist="${dev_major_whitelist}240:241:242:243:244:245:246:247:248:249:250:251:252:253:254:"
332 - fi
323 + if [ -d /sys/block ] && [ -r /proc/devices ] ; then
324 + dev_major_whitelist=''
325 +
326 + # This is a list of device names used for block storage devices.
327 + # These translate to the prefixs of files in `/dev` indicating the device type.
328 + # They are sorted by lowest used device major number, with dynamically assigned ones at the end.
329 + # We use this to look up device major numbers in `/proc/devices`
330 + device_names='hd sd mfm ad ftl pd nftl dasd intfl mmcblk ub xvd rfd vbd nvme'
331 +
332 + for name in ${device_names} ; do
333 + if grep -qE " ${name}\$" /proc/devices ; then
334 + dev_major_whitelist="${dev_major_whitelist}:$(grep -E "${name}\$" /proc/devices | sed -e 's/^[[:space:]]*//' | cut -f 1 -d ' ' | tr '\n' ':'):"
335 + fi
336 + done
337
338 DISK_DETECTION="sysfs"
339 DISK_SIZE="0"
@@ -338,18 +342,17 @@ else
342 (echo "${dev_major_whitelist}" | grep -q ":$(cut -f 1 -d ':' "${disk}/dev"):") && \
343 grep -qv 1 "${disk}/removable"
344 then
341 - size="$(cat "${disk}/size")"
345 + size="$(($(cat "${disk}/size") * 512))"
346 DISK_SIZE="$((DISK_SIZE + size))"
347 fi
348 done
349 elif df --version 2>/dev/null | grep -qF "GNU coreutils" ; then
350 DISK_DETECTION="df"
347 - DISK_SIZE="$(df -x tmpfs -x devtmpfs -x squashfs -l --total -B1 --output=size | tail -n 1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
351 + DISK_SIZE=$(($(df -x tmpfs -x devtmpfs -x squashfs -l -B1 --output=source,size | tail -n +2 | sort -u -k 1 | awk '{print $2}' | tr '\n' '+' | head -c -1)))
352 else
353 DISK_DETECTION="df"
354 include_fs_types="ext*|btrfs|xfs|jfs|reiser*|zfs"
351 - total="$(df -T -P | grep "${include_fs_types}" | awk '{s+=$3} END {print s}')"
352 - DISK_SIZE="$((total * 1024))"
355 + DISK_SIZE=$(($(df -T -P | tail -n +2 | sort -u -k 1 | grep "${include_fs_types}" | awk '{print $3}' | tr '\n' '+' | head -c -1) * 1024))
356 fi
357 fi
358