| 1 | # Spec - Sensitive data discipline for committed artifacts |
| 2 | |
| 3 | ## Status |
| 4 | |
| 5 | Active. Applies to every SOW, every skill (public or private), every |
| 6 | script, every spec, every committed doc, every commit message, and every |
| 7 | PR text in this repository. |
| 8 | |
| 9 | ## Rule |
| 10 | |
| 11 | The following categories of literal values MUST NOT appear in any |
| 12 | committed file: |
| 13 | |
| 14 | 1. **IP addresses.** IPv4 or IPv6 literals to specific hosts. |
| 15 | Loopback (`127.0.0.1`, `::1`) and link-local addresses used as |
| 16 | loopback / link-local references are fine. |
| 17 | 2. **Tenant-identifying hostnames.** Any hostname that uniquely |
| 18 | identifies a customer, community member, internal ingestion |
| 19 | host, or other non-public destination. **Public Netdata-org |
| 20 | sites used in role-descriptive prose are allowed** (e.g. |
| 21 | `learn.netdata.cloud` when describing the learn site, |
| 22 | `netdata.cloud` when describing the marketing site). The Cloud |
| 23 | REST API host is env-keyed because it's an operational target |
| 24 | that scripts call. |
| 25 | 3. **UUID-shaped identifiers.** Machine GUIDs, node UUIDs, claim |
| 26 | IDs, space IDs, room IDs, agent IDs, ephemeral IDs, |
| 27 | bearer-token UUIDs, tenant IDs, account IDs. |
| 28 | 4. **Credentials.** API tokens, bearer tokens, session cookies, |
| 29 | OAuth tokens, passwords, signing keys, SSH private keys, |
| 30 | anything that grants access. |
| 31 | 5. **Absolute filesystem paths to per-user or per-tenant state.** |
| 32 | User home paths, the user's mirrored-repos tree, the larger |
| 33 | monitoring mirror tree, or any workstation-specific opt/var |
| 34 | path that identifies a particular install or user. |
| 35 | **Documented Netdata default install paths |
| 36 | (`/var/lib/netdata`, `/etc/netdata`, `/usr/lib/netdata`, |
| 37 | `/usr/libexec/netdata`) are allowed** because they are |
| 38 | open-source defaults documented across the codebase. |
| 39 | Netdata can also be bundled (typically rooted at |
| 40 | `/opt/netdata`), so scripts MUST autodetect the install |
| 41 | prefix at runtime by probing the candidate locations and |
| 42 | selecting the first one that exists. Common candidates: |
| 43 | empty (system install), `/opt/netdata`, `/usr/local/netdata`. |
| 44 | Scripts must NOT require a `NETDATA_PREFIX` env knob; the |
| 45 | prefix is a discovered fact, not a user configuration. |
| 46 | 6. **Real-name identifiers.** Usernames, email addresses, real |
| 47 | names of community members, customers, employees, |
| 48 | contributors. |
| 49 | 7. **Tenant identifiers.** Netdata Cloud space names, room |
| 50 | names, or any human-readable identifier that maps to a |
| 51 | specific tenant or organization. |
| 52 | 8. **Proprietary incident details.** Customer support |
| 53 | narratives, non-public bug reports, private correspondence. |
| 54 | |
| 55 | ## Allowed alternatives |
| 56 | |
| 57 | For every reference to a value covered above, use ONE of: |
| 58 | |
| 59 | - An env-key placeholder: `${KEY_NAME}` -- the value lives in `.env` |
| 60 | (gitignored). Examples: |
| 61 | `ssh ${AGENT_EVENTS_HOSTNAME}` rather than the literal address; |
| 62 | `${NETDATA_REPOS_DIR}/learn/ingest.js` and |
| 63 | `${NETDATA_REPOS_DIR}/website/content/...` when committed skill |
| 64 | content needs to point at sibling Netdata-org repositories the |
| 65 | user has cloned locally. Sibling-repo file paths via |
| 66 | `${NETDATA_REPOS_DIR}/...` are explicitly allowed in committed |
| 67 | skill / SOW / spec content; literal workstation roots |
| 68 | (`~/`, `/home/...`) are not. |
| 69 | - A repo-relative path: `<repo>/src/daemon/status-file.c`, |
| 70 | `src/web/api/...` -- these describe locations inside this |
| 71 | repository and are not leaks. |
| 72 | - Standard Linux/POSIX paths that carry no tenant or user identity: |
| 73 | `/tmp`, `/run`, `/etc/passwd` (the file, not the contents), `/proc`, |
| 74 | `/sys`. Use sparingly; prefer not to mention them at all unless the |
| 75 | reference is essential. |
| 76 | - Generic role descriptions: "the production Netdata Cloud REST host", |
| 77 | "the agent's varlib directory", "the user's repo mirror" -- without |
| 78 | the actual value. |
| 79 | - Open-source code references: `file:line` citations into this repo |
| 80 | (e.g. `src/daemon/status-file.c:988`) are fine; they describe code, |
| 81 | not values. |
| 82 | |
| 83 | ## Required env keys |
| 84 | |
| 85 | These keys MUST be defined in `<repo>/.env` (gitignored) when a SOW, |
| 86 | skill, or script references them. If a SOW or script references one and |
| 87 | the key is unset, the script must error loudly and exit non-zero. Values |
| 88 | live ONLY in `.env`; this spec lists names and roles only. |
| 89 | |
| 90 | | Key | Role | |
| 91 | |---|---| |
| 92 | | `NETDATA_CLOUD_TOKEN` | long-lived Cloud REST token | |
| 93 | | `NETDATA_CLOUD_HOSTNAME` | Cloud REST API host (the operational target scripts call) | |
| 94 | | `NETDATA_REPOS_DIR` | user's mirror of Netdata-org repos | |
| 95 | | `AGENT_EVENTS_HOSTNAME` | network address of the ingestion node -- dual-duty: ssh host (`ssh ${AGENT_EVENTS_HOSTNAME}`) AND direct-HTTP host (`http://${AGENT_EVENTS_HOSTNAME}:19999/...`). Can be a DNS name or an IP literal. NOTE: this is NOT the journalctl namespace (hardcoded to `agent-events`) and NOT the Cloud room name (also hardcoded to `agent-events`). | |
| 96 | | `AGENT_EVENTS_MACHINE_GUID` | events-ingestion agent machine GUID | |
| 97 | | `AGENT_EVENTS_NODE_ID` | events-ingestion agent node UUID | |
| 98 | | `CODACY_TOKEN` | Codacy Cloud Account API token; header form `api-token: <value>` | |
| 99 | |
| 100 | Per-user setup is documented at `<repo>/.agents/ENV.md`. The |
| 101 | committed `<repo>/.env.template` is the starting point for a |
| 102 | new contributor's `.env`. |
| 103 | |
| 104 | Things that are intentionally NOT env-keyed (and why): |
| 105 | |
| 106 | - The Cloud Swagger URL is derived as |
| 107 | `${NETDATA_CLOUD_HOSTNAME}/api/docs/`. |
| 108 | - The agent-events producer ingest URL is a `const char *` in |
| 109 | `src/daemon/status-file.c`; reference by `file:line`. |
| 110 | - Public site hostnames (learn site, marketing site) are public |
| 111 | and used in role-descriptive prose. |
| 112 | - Default Netdata install paths (`/var/lib/netdata`, |
| 113 | `/etc/netdata`) are public OSS defaults; bundled installs |
| 114 | (`/opt/netdata/...`, etc.) are handled by runtime |
| 115 | autodetection in scripts, not a config knob. |
| 116 | - This repo's checkout root is found via |
| 117 | `git rev-parse --show-toplevel`. |
| 118 | |
| 119 | Adding new keys to `.env` is the user's prerogative. SOWs and |
| 120 | scripts can REQUEST keys; only the user adds them. |
| 121 | |
| 122 | ## Verification |
| 123 | |
| 124 | Before any commit that touches a SOW, skill, spec, or doc, run: |
| 125 | |
| 126 | ```bash |
| 127 | # Helper: list of files staged for commit, excluding this spec. |
| 128 | files=$(git diff --cached --name-only --diff-filter=ACMR \ |
| 129 | | grep -v '^\.agents/sow/specs/sensitive-data-discipline\.md$') |
| 130 | [ -z "$files" ] && exit 0 |
| 131 | |
| 132 | # Run each pattern. The patterns themselves are not embedded in |
| 133 | # this code block as committable literals; they are constructed |
| 134 | # from concatenated character classes so a grep over THIS spec |
| 135 | # does not flag itself for the very examples it must define. |
| 136 | grep_args=( |
| 137 | --line-number --extended-regexp --binary-files=without-match |
| 138 | ) |
| 139 | |
| 140 | # Domain pattern for the org. |
| 141 | domain='[A-Za-z0-9-]+\.netdata\.(cloud|io)' |
| 142 | git grep "${grep_args[@]}" -- $files -- "$domain" |
| 143 | |
| 144 | # UUID-shaped identifiers. |
| 145 | uuid='[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' |
| 146 | git grep "${grep_args[@]}" -- $files -- "$uuid" |
| 147 | |
| 148 | # IPv4 literals (review hits manually for false positives). |
| 149 | ipv4='([0-9]{1,3}\.){3}[0-9]{1,3}' |
| 150 | git grep "${grep_args[@]}" -- $files -- "$ipv4" |
| 151 | |
| 152 | # Per-user / per-install absolute paths. Documented Netdata |
| 153 | # default paths (/var/lib/netdata, /etc/netdata) are intentionally |
| 154 | # excluded -- they are public OSS defaults; bundled installs use |
| 155 | # the NETDATA_PREFIX knob. |
| 156 | abs="(~/|/$(echo op)t/baddisk|/$(echo op)t/neda|/$(echo op)t/ai-agent|/$(echo ho)me/)" |
| 157 | git grep "${grep_args[@]}" -- $files -- "$abs" |
| 158 | |
| 159 | # Long opaque tokens (40+ char base64-ish). |
| 160 | tok='[A-Za-z0-9_+/=-]{40,}' |
| 161 | git grep "${grep_args[@]}" -- $files -- "$tok" |
| 162 | ``` |
| 163 | |
| 164 | Every match must either be removed or converted to an env-key |
| 165 | reference, OR explicitly justified inline (e.g. a citation of an |
| 166 | upstream open-source project's hostname when documenting how that |
| 167 | project reports its own data). |
| 168 | |
| 169 | ## Exceptions |
| 170 | |
| 171 | - **This spec file itself** must list forbidden patterns and |
| 172 | example regexes in order to define the rule. The verification |
| 173 | grep excludes `<repo>/.agents/sow/specs/sensitive-data- |
| 174 | discipline.md` from its scan. No other SOW, skill, or doc |
| 175 | qualifies for this exemption. |
| 176 | - **Quoted user messages** preserved verbatim in a SOW's "User |
| 177 | Request" section may include literals the user typed. Redact |
| 178 | the literal value and replace with the `.env` key in `[env- |
| 179 | keyed: ${KEY}]` form, with a footnote pointing here. The |
| 180 | user's wording stays; only the literal value moves to `.env`. |
| 181 | - **Repo-relative paths** (`src/daemon/...`, `<repo>/src/...`) |
| 182 | are not absolute paths and are fine. |
| 183 | - **Open-source upstream references** (e.g. |
| 184 | `prometheus/prometheus@<sha>:cmd/...`) are fine; they |
| 185 | describe external code, not tenant data. |
| 186 | - **Code citations** of the form `file:line` are fine |
| 187 | (`src/daemon/status-file.c:988`). |
| 188 | |
| 189 | ## Failure mode |
| 190 | |
| 191 | If verification returns a hit on a committed file, the work that introduced it |
| 192 | failed its Sensitive Data Gate and must be treated as a regression. Create a new |
| 193 | branch-local SOW that links the prior PR/commit, redact the durable artifact, |
| 194 | and force-push only with explicit user approval. Otherwise, create a follow-up |
| 195 | commit that scrubs the leak. |