| 1 | # Secrets Management |
| 2 | |
| 3 | Keep collector credentials out of plain-text configuration files. |
| 4 | |
| 5 | Netdata lets you reference secret values in collector configs instead of storing them directly in YAML. Depending on where the secret lives, you can resolve it from environment variables, local files, local commands, or remote secretstore backends. |
| 6 | |
| 7 | ### Jump To |
| 8 | |
| 9 | [Resolver Quick Reference](#resolver-quick-reference) • [Choosing a Resolver](#choosing-a-resolver) • [Environment Variables](#environment-variables) • [Files](#files) • [Commands](#commands) • [Secretstores](#secretstores) • [Supported Secretstore Backends](#supported-secretstore-backends) • [How It Works](#how-it-works) • [Troubleshooting](#troubleshooting) |
| 10 | |
| 11 | |
| 12 | ## Resolver Quick Reference |
| 13 | |
| 14 | | Resolver | Syntax | Best for | Notes | |
| 15 | |:---------|:-------|:---------|:------| |
| 16 | | Environment variable | `${env:VAR_NAME}` | Secrets already injected into the Netdata service environment | Value is trimmed. The variable must exist. | |
| 17 | | File | `${file:/absolute/path}` | Secrets stored in local files on disk | The path must be absolute. File contents are trimmed. | |
| 18 | | Command | `${cmd:/absolute/path/to/command args}` | Secrets returned by a trusted local command | The command path must be absolute. Netdata uses a 10-second timeout. | |
| 19 | | Secretstore | `${store:<kind>:<name>:<operand>}` | Secrets stored in remote backends such as Vault, AWS, Azure, or GCP | Configure the secretstore first, then reference it from collector configs. | |
| 20 | |
| 21 | ## Choosing a Resolver |
| 22 | |
| 23 | - Use `${env:...}` or `${file:...}` for simple setups where secrets are already available locally on the Netdata host. |
| 24 | - Use `${cmd:...}` when you need dynamic secret retrieval via a trusted local command, such as 1Password CLI or a custom script. |
| 25 | - Use `${store:...}` when your organization manages secrets centrally in a cloud provider or Vault and you want Netdata to pull from that source directly. |
| 26 | - You can use different resolver types across different collectors, different jobs within the same collector, or even within the same configuration value. See [Mixing resolver types](#mixing-resolver-types). |
| 27 | |
| 28 | ## Environment Variables |
| 29 | |
| 30 | Use `${env:VARIABLE_NAME}` to read a secret from the Netdata process environment. |
| 31 | |
| 32 | ```yaml |
| 33 | jobs: |
| 34 | - name: mysql_prod |
| 35 | password: "${env:MYSQL_PASSWORD}" |
| 36 | ``` |
| 37 | |
| 38 | - Netdata trims leading and trailing whitespace from the environment variable value. |
| 39 | - The variable must be set in the environment of the Netdata service or process that runs the collector. |
| 40 | |
| 41 | ## Files |
| 42 | |
| 43 | Use `${file:/absolute/path}` to read a secret from a local file on disk. |
| 44 | |
| 45 | ```yaml |
| 46 | jobs: |
| 47 | - name: mysql_prod |
| 48 | password: "${file:/run/secrets/mysql_password}" |
| 49 | ``` |
| 50 | |
| 51 | - The file path must be absolute. |
| 52 | - Netdata trims leading and trailing whitespace from the file contents. |
| 53 | - The file must exist on the Netdata host and be readable by the `netdata` user. |
| 54 | - **Docker Secrets**: Docker mounts secrets as files under `/run/secrets/` inside the container. Use `${file:/run/secrets/<secret-name>}` to read them. |
| 55 | - **Kubernetes Secrets**: If you mount Kubernetes Secrets as volume files in the Netdata pod, reference them with `${file:/path/to/mounted/secret}`. |
| 56 | |
| 57 | ## Commands |
| 58 | |
| 59 | Use `${cmd:/absolute/path/to/command args}` to execute a trusted local command and use its stdout as the secret value. |
| 60 | |
| 61 | ```yaml |
| 62 | jobs: |
| 63 | - name: mysql_prod |
| 64 | password: "${cmd:/usr/bin/op read op://vault/netdata/mysql/password}" |
| 65 | ``` |
| 66 | |
| 67 | - The command path must be absolute. |
| 68 | - Arguments are split on whitespace. Netdata does not interpret shell quoting, pipes, redirects, or variable expansion unless you explicitly run a shell such as `/bin/sh -c`. |
| 69 | - Netdata uses a 10-second timeout for command resolvers. |
| 70 | - Netdata trims leading and trailing whitespace from stdout and ignores stderr. |
| 71 | |
| 72 | ## Secretstores |
| 73 | |
| 74 | Use secretstores when you want Netdata collectors to fetch secrets from remote backends at runtime instead of storing them locally in collector configs. |
| 75 | |
| 76 | Configure a secretstore first, then reference it from collector configs with: |
| 77 | |
| 78 | ```text |
| 79 | ${store:<kind>:<name>:<operand>} |
| 80 | ``` |
| 81 | |
| 82 | | Part | Description | |
| 83 | |:-----|:------------| |
| 84 | | `kind` | Secretstore backend kind, such as `vault` or `aws-sm`. | |
| 85 | | `name` | The store name you configured in Netdata, such as `vault_prod`. | |
| 86 | | `operand` | Backend-specific identifier for the secret you want to read. | |
| 87 | |
| 88 | Example: |
| 89 | |
| 90 | ```yaml |
| 91 | jobs: |
| 92 | - name: mysql_prod |
| 93 | password: "${store:vault:vault_prod:secret/data/netdata/mysql#password}" |
| 94 | ``` |
| 95 | |
| 96 | ### Configuration Methods |
| 97 | |
| 98 | #### Dynamic Configuration UI |
| 99 | |
| 100 | 1. Open the Netdata Dynamic Configuration UI. |
| 101 | 2. Go to `Collectors -> go.d -> SecretStores`. |
| 102 | 3. Choose the backend kind you want to configure. |
| 103 | 4. Give the secretstore a name. |
| 104 | 5. Fill in the backend-specific settings. |
| 105 | 6. Save the secretstore and use its `${store:<kind>:<name>:<operand>}` reference in collector configs. |
| 106 | |
| 107 | #### Configuration Files |
| 108 | |
| 109 | Each secretstore backend has its own file under `/etc/netdata/go.d/ss/`: |
| 110 | |
| 111 | | File | Backend | |
| 112 | |:-----|:--------| |
| 113 | | `/etc/netdata/go.d/ss/aws-sm.conf` | AWS Secrets Manager | |
| 114 | | `/etc/netdata/go.d/ss/azure-kv.conf` | Azure Key Vault | |
| 115 | | `/etc/netdata/go.d/ss/gcp-sm.conf` | Google Secret Manager | |
| 116 | | `/etc/netdata/go.d/ss/vault.conf` | Vault | |
| 117 | |
| 118 | Each file contains a `jobs` array. The backend kind is determined by the filename. |
| 119 | |
| 120 | :::note |
| 121 | |
| 122 | File-based secretstores are loaded at agent startup. If you edit these files, restart the Netdata Agent to apply the changes. |
| 123 | |
| 124 | ::: |
| 125 | |
| 126 | If the `/etc/netdata/go.d/ss/` directory does not exist, create it: |
| 127 | |
| 128 | ```bash |
| 129 | sudo mkdir -p /etc/netdata/go.d/ss |
| 130 | sudo chown netdata:netdata /etc/netdata/go.d/ss |
| 131 | sudo chmod 0750 /etc/netdata/go.d/ss |
| 132 | ``` |
| 133 | |
| 134 | Secretstore configuration files may contain sensitive values such as tokens or client secrets. Restrict directory and file permissions to the `netdata` user. |
| 135 | |
| 136 | ### Multiple Secretstores |
| 137 | |
| 138 | Each secretstore config file can contain multiple `jobs` entries, each with a unique store name. You can use different secretstore backends simultaneously. For example, you might configure a Vault store for database credentials and an AWS Secrets Manager store for API keys, then reference each one using its `${store:<kind>:<name>:<operand>}` syntax in the relevant collector configs. |
| 139 | |
| 140 | ### Mixing Resolver Types |
| 141 | |
| 142 | You can mix different resolver types in the same configuration value or the same config file. For example, you might read the username from an environment variable and the password from a secretstore: |
| 143 | |
| 144 | ```yaml |
| 145 | jobs: |
| 146 | - name: mysql_prod |
| 147 | dsn: "${env:MYSQL_USER}:${store:vault:vault_prod:secret/data/netdata/mysql#password}@tcp(127.0.0.1:3306)/" |
| 148 | ``` |
| 149 | |
| 150 | Different jobs within the same collector config file can also use different resolver types. |
| 151 | |
| 152 | ## Supported Secretstore Backends |
| 153 | |
| 154 | Use the backend README for provider-specific authentication, operand rules, configuration examples, and troubleshooting. |
| 155 | |
| 156 | | Backend | Kind | Operand Format | Example Operand | |
| 157 | |:--------|:-----|:---------------|:----------------| |
| 158 | | [AWS Secrets Manager](/src/go/plugin/agent/secrets/secretstore/backends/aws/README.md) | `aws-sm` | `secret-name[#key]` | `netdata/mysql#password` | |
| 159 | | [Azure Key Vault](/src/go/plugin/agent/secrets/secretstore/backends/azure/README.md) | `azure-kv` | `vault-name/secret-name` | `my-keyvault/mysql-password` | |
| 160 | | [Google Secret Manager](/src/go/plugin/agent/secrets/secretstore/backends/gcp/README.md) | `gcp-sm` | `project/secret[/version]` | `my-project/mysql-password` | |
| 161 | | [Vault](/src/go/plugin/agent/secrets/secretstore/backends/vault/README.md) | `vault` | `path#key` | `secret/data/netdata/mysql#password` | |
| 162 | |
| 163 | ## How It Works |
| 164 | |
| 165 | - Secrets are resolved each time a collector job starts or restarts. |
| 166 | - If a secret cannot be resolved, the collector job will fail to start and log an error. |
| 167 | - Updating a secretstore automatically restarts running and failed collector jobs that use it so they pick up the new credentials. |
| 168 | - Accepted or disabled jobs keep their state and use the updated secretstore the next time they start. |
| 169 | - If a secretstore change applies successfully but some dependent collector restarts fail, Netdata reports those restart failures. |
| 170 | |
| 171 | ## Security Notes |
| 172 | |
| 173 | - Prefer secret references over plain-text credentials in collector configs. |
| 174 | - Prefer platform-native identity modes for production when a backend supports them, such as instance roles, managed identities, or metadata-based credentials. |
| 175 | - Secretstore configuration values (such as tokens and client secrets) also support `${env:...}`, `${file:...}`, and `${cmd:...}` resolvers. Use them to avoid storing backend credentials in plain text. Note that `${store:...}` references are not supported inside secretstore configurations. |
| 176 | - Keep local secret material readable only by the `netdata` user, including token files, service account files, and any files used with `${file:...}`. |
| 177 | - Use `${cmd:...}` only with trusted local commands and absolute paths. |
| 178 | |
| 179 | ## Troubleshooting |
| 180 | |
| 181 | - Secret resolution failures appear in agent logs and usually surface as collector jobs failing to start. |
| 182 | - Start by checking the resolver syntax you used in the collector config. |
| 183 | - For `${env:...}`, make sure the variable exists in the Netdata process environment. |
| 184 | - For `${file:...}`, make sure the path is absolute and the file is readable by `netdata`. |
| 185 | - For `${cmd:...}`, make sure the command path is absolute and the command completes within 10 seconds. |
| 186 | - For `${store:...}`, check the backend README for provider-specific operand rules, authentication requirements, and troubleshooting. |
| 187 | |
| 188 | Representative error patterns: |
| 189 | |
| 190 | - `${env:VAR_NAME}`: environment variable is not set |
| 191 | - `${file:relative/path}`: file path must be absolute |
| 192 | - `${cmd:echo hello}`: command path must be absolute |
| 193 | - `${cmd:/path/to/slow-command}`: command timed out after 10s |