Add AI-assistant skills (query-netdata, integrations, learn, agent-events, mirror) (#22423)
* agents: query-netdata skills + SOW infrastructure Symmetric public AI skills for querying every Netdata Cloud and Netdata Agent surface (metrics, logs, topology, flows, alerts, dyncfg, functions, nodes, plus cloud-only rooms/members/feed and agent-only streaming) so downstream skills do not reimplement them. Token-safe wrappers keep cloud tokens, agent bearers and claim_ids off assistant-visible stdout, with a no-leak self-test. Adds the sensitive-data-discipline spec, the public-skill convention in AGENTS.md (token-safety contract + live how-tos catalog rule), and tracks the remaining skill work via SOW-0003, 0004, 0005, 0006 and 0007. * agents: integrations-lifecycle private skill (SOW-0007) Documents the integrations pipeline 100%: every script (gen_integrations, gen_docs_integrations, gen_doc_collector_page, gen_doc_secrets_page, gen_doc_service_discovery_page), every JSON Schema (12 of them), every generated artifact and DO-NOT-EDIT banner, both CI workflows (generate-integrations.yml, check-markdown.yml), the ibm.d contexts.yaml -> metadata.yaml chain, the cloud-frontend artifact contract, the five-file collector-consistency rule, and the gotchas including the broken check_collector_metadata.py, the gen_doc_service_discovery_page.py CI gap, and the unused distros.json schema. So an assistant or maintainer never has to ask again how metadata.yaml works, whether integrations/*.md should be hand-edited, or what runs in CI. Sensitive-data-discipline spec tightened with an explicit sibling-repo-via-NETDATA_REPOS_DIR example. * agents: clarify points vs duration in query-metrics skills Assistants commonly request "10 minutes in 30 points" expecting per-second data. The query engine actually divides the time range into N equal buckets, so 600 seconds / 30 points = 20 seconds per point (heavily aggregated). To get per-second data, points MUST equal the duration in seconds. Both query-netdata-cloud and query-netdata-agents now document this explicitly with worked examples, the common mistake, the dbengine tier 0 requirement for per-second storage, and the fact that points: 0 is NOT per-second. * agents: learn-site-structure private skill (SOW-0004) Documents the Netdata learn-site ingest pipeline 100%: the docs/.map/map.yaml source-of-truth (the actual lever -- source filesystem path is irrelevant for routing), the live ingest/ingest.py orchestrator (NOT the legacy ingest.js / ingest.md, which are stale), the 16-step ingest flow, the 6 source repositories, frontmatter injection and slug computation, sidebar autogeneration via Docusaurus filesystem mode, every MDX escape transformation, the 4-mechanism redirect stack with auto-redirect on move/rename and manual surgery on delete, the every-3-hours CI cadence, the Netlify deploy contract, the part_of_learn: True opt-in, and the every dead artifact and gotcha worth knowing. Recipes for add/move/rename/delete walk maintainers through each operation. So an assistant or maintainer never has to ask again how a Learn page is produced. * agents: fix learn-site-structure SKILL.md YAML The description value contained `part_of_learn: True` (with a space after the colon) inside backticks. YAML ignores markdown backticks and parsed the colon as a nested mapping, breaking frontmatter load. Replaced with `part_of_learn=True`. * agents: query-agent-events private skill (SOW-0003) Bug-investigation tool for the agent-events ingestion namespace. Verified producer-side AE_* field map (80+ fields traceable to src/daemon/status-file.c), enums (status / aclk / health / profile / exit_reason / exit_cause / signal_code), 23h client-side dedup (status-file-dedup.c:11), 10-min disk-snapshot cadence (status-file.c:835), and the after-the-fact event timing (POST only on agent restart). Lifted the multi-value selections filter capability into query-netdata-{cloud,agents}/query-logs.md so all callers can use it -- between fields = AND, between values = OR, verified at libnetdata/facets/logs_query_status.h:386-466. Ships SKILL.md plus per-domain guides (AE_FIELDS, transports, update-cadence, query-discipline, finding-crashes, finding-fatals), recipes (find-by-function, find-by-version, find-related-to-work), how-tos catalog, and token-safe scripts: get-events.sh (index-friendly defaults: 24h, latest stable + 3 nightlies, structured selections), analyze-events.sh (group-by 17 dimensions including signal / version / fatal_function / architecture / packaging / kubernetes / aclk), redact-events.sh (opt-in identifier masking for sharing). 40k-200k events/day on stable releases makes index-friendly queries mandatory; the skill writes that rule and bakes it into the defaults. The .local agent-events draft was found to have 14 high-severity divergences from producer source; the committed AE_FIELDS.md is the authoritative reference. * Close MikroTik SNMP gap investigation * agents: mirror-netdata-repos private skill (SOW-0005) Vendors a parameterized copy of the user's battle-tested sync-all.sh as scripts/sync-netdata-repos.sh so AI assistants and developers can bring up a local Netdata-org repos mirror at ${NETDATA_REPOS_DIR} for cross-repo grep / code review without GitHub API round-trips and rate limits. Surgical edits vs the source: env-driven mirror dir (NETDATA_REPOS_DIR), --repo NAME repeatable scoping (skips Phase 2), sanitization for missing env / git / jq / gh (gh missing or unauthed gracefully skips Phase 2), early --help that works without env. All battle-tested logic preserved verbatim (activity-cache sort, skip-on-staged-or-modified, switch-to-default, submodule force recursive, colored output, end-of-run summary). Reset-to-default-branch is documented as the intended safety feature that prevents stale-feature-branch black-hole repos from confusing cross-repo reasoning. Skill is independent from any other repo mirrors this workstation may have. * agents: simplify .env -- single setup guide + correct hostname semantics Drop AGENT_EVENTS_NC_SPACE entirely (unused; cloud queries route via AGENT_EVENTS_NODE_ID; the space is implicit in the token's tenancy). Correct the AGENT_EVENTS_HOSTNAME description across the skill family: it is dual-duty (ssh host + direct-HTTP host, can be IP or DNS name), NOT quadruple-duty. The journal namespace is hardcoded to 'agent-events' on the ingestion server's log2journal invocation; the Cloud room name is also hardcoded 'agent-events'. Neither is derived from the hostname. agentevents_namespace() in scripts/_lib.sh now returns the literal 'agent-events'. Every recipe / discipline doc / transport doc / example payload that previously interpolated ${AGENT_EVENTS_HOSTNAME} into __logs_sources or --namespace= now uses the literal constant. Add .agents/ENV.md as the single setup + reference guide -- per-key table with role, where to find the value, sample format, which skills require it, plus common mistakes (tilde-in-quotes, wrong gh org, trailing whitespace, etc). Add .env.template at repo root with sanitized placeholders + inline comments mapping to the guide. Tighten error messages in query-netdata-agents/_lib.sh and query-agent-events/_lib.sh to point at .agents/ENV.md instead of generic 'is empty in .env'. Drop NETDATA_BEARER_TOKENS_DIR from the spec's example list (the key was never declared in .env, never referenced by any script). Replace with a real example using AGENT_EVENTS_HOSTNAME. Update AGENTS.md to point readers at .env.template and .agents/ENV.md for setup. * Fix analyze-events.sh row projection; renumber duplicate SOWs - analyze-events.sh: bind row before reduce so jq does not try to index the accumulator object with the integer column index; the previous form errored "Cannot index object with number" on every group-by run. Verified end-to-end on a fresh 24h Cloud-transport dump (570 rows; signal/exit_cause/version/ fatal_function/health all group correctly). - SOWs renumbered to remove number collisions: SOW-0001 (project-writing-collectors, 20260502) -> SOW-0009 SOW-0002 (netdata-query-skills, 20260503) -> SOW-0010 qbridge keeps SOW-0001 (its earlier 20260501 slot); unified- topology keeps SOW-0002 (also 20260501). Cross-references in SOW-0003 / 0004 / 0005 / 0006 / 0007 and the sensitive-data spec updated to point at the new numbers. * Address shellcheck findings on AI-skill scripts All findings are mechanical (no behavior change). Verified locally with shellcheck 0.11.0 --external-sources: ALL CLEAN. - query-agent-events: SC1091 silenced via SCRIPTDIR source hint plus disable directive (the source path is dynamic at runtime, not followable by static analysis). One SC2012 documented inline (ls -1t on a known *.json dir is fine, find pipeline is overkill). - mirror-netdata-repos: split declare-and-assign (SC2155) on every local-with-command-substitution to stop masking subcommand exit status; quoted git-rev refs (SC2086); replaced bare `> file` truncation with `: > file` (SC2188); guarded the initial cd into NETDATA_REPOS_DIR with `|| exit 2` (SC2164); dropped the unused `untracked` count in get_uncommitted_details (SC2034) -- the caller already counts untracked separately for the informational print. * Exclude AI-assistant tooling trees from Codacy markdownlint Codacy ran markdownlint on .agents/ (SOWs, specs, internal project skills, work tracking) and docs/netdata-ai/skills/ (AI-skill docs) and reported 864 style findings -- all Info/Warning, mostly MD013 line-length on prose tables and reference dumps that are written for AI/agent consumption rather than narrow-column human reading. Adding both trees to exclude_paths is consistent with the existing precedent for content where markdownlint is the wrong tool (parity/README.md, parity/evidence/**, the TODO-netflow-* files). * agents: codacy-audit skill (SOW-0011) Adds .agents/skills/codacy-audit/ with the same shape as the sister coverity-audit / sonarqube-audit / graphql-audit skills: - SKILL.md with frontmatter, MANDATORY sections, scope (in/out), env-keys table, scripts table, workflow examples. - scripts/_lib.sh -- token-safe wrappers (CODACY_TOKEN never reaches captured stdout) plus a sentinel-driven no-leak self-test; portable repo-root resolution that works under both bash and zsh. - scripts/analyze-local.sh -- runs codacy-analysis-cli locally before `git push` to catch findings without a CI round-trip; auto-detects local binary, falls back to docker; uses the upstream docker-in-docker invocation (sock mount + CODACY_CODE + same-path bind-mount) so the CLI's per-tool child containers can start. - scripts/pr-issues.sh -- paginated v3 PR-issue fetch with a clustered TSV summary on stdout (group-by tool / pattern / severity / file / category) and full JSON dump under .local/audits/codacy/. - how-tos/INDEX.md plus reproduce-pr-22423-markdownlint.md seed entry for the live-catalog rule. Scope is read-only by design. Write actions (mark false-positive, mark fixed, modify ignore-patterns via API) and master-backlog triage are deferred to future SOWs and will open when a real need surfaces. Artifact updates: - .agents/ENV.md -- new Codacy section + per-skill checklist - .env.template -- CODACY_TOKEN block - AGENTS.md -- skill-index entries - sensitive-data-discipline.md -- env-keys table row Validation: - shellcheck --external-sources: ALL CLEAN - no-leak self-test: PASS in both bash and zsh - pr-issues.sh on PR #22423 (post-exclusion): 0 issues - pr-issues.sh on PR #22420 (recent merged PR): 0 issues - analyze-local.sh on a markdownlint fixture: 2 findings * Address SonarCloud findings on AI-skill scripts SonarCloud flagged 8 new issues on PR #22423 (Quality Gate still passed). All are mechanical -- no behavior change. Findings addressed: - shelldre:S131 (1) sync-netdata-repos.sh -- early-help case loop now has an explicit `*) ;;` no-op default. - shelldre:S7679 (2) analyze-events.sh field_for_dim() -- bind positional param to `local dim` once, reference `$dim` in branches and the unknown-dim error path. - shelldre:S7679 (5) sync-netdata-repos.sh -- bind positional args to local in print_status, is_git_repo, and main()'s CLI parser loop (where each iteration now does `local arg=$1` before the case, and the --repo branch does `local val=$2`). Same-pattern scan: query-netdata-agents/_lib.sh's agents_query_agent() and agents_call_function() use the same positional-param-in-case idiom (Sonar didn't flag them this round, likely deduped per-rule per-file). Fixed pre-emptively to avoid a follow-up bot round-trip. Validation: - shellcheck --external-sources on all four touched files: ALL CLEAN - bash -n on all four: OK - analyze-events.sh on the existing dump: produces same output - pr-issues.sh 22423: 0 issues (unchanged) - codacyaudit_selftest_no_token_leak: PASS - agentevents_selftest_no_token_leak: PASS
Costa Tsaousis committed
May 5, 2026 at 17:47 UTC
d8531e4ba7134581b9eb5d6f780fd2dae513b7b4
94 files changed
+18051
-5
.agents/ENV.md
new
+191
@@ -0,0 +1,191 @@
1
+# `.env` setup and reference
2
+
3
+`.env` at the repo root holds per-user secrets and pointers
4
+that AI-skill scripts consume. It is **gitignored** -- values
5
+never reach the committed history.
6
+
7
+This file is the single setup guide. Every skill that needs
8
+`.env` keys lists them here with the role, where to find the
9
+value, sample format, and which scripts consume it. If a
10
+script tells you a key is missing, check this file.
11
+
12
+## Quick start
13
+
14
+```bash
15
+cd <repo>
16
+cp .env.template .env
17
+# Open .env in your editor and fill in the keys you need.
18
+chmod 0600 .env # optional but recommended
19
+```
20
+
21
+You only need to fill in keys for the skills you actually
22
+use. Each script checks its own required keys and exits with
23
+a clear error if any are missing -- it will not corrupt
24
+state if you forget a key.
25
+
26
+## Key reference
27
+
28
+### Netdata Cloud + agents
29
+
30
+| Key | Role | Where to find it | Sample format |
31
+|---|---|---|---|
32
+| `NETDATA_CLOUD_TOKEN` | long-lived Cloud REST token | app.netdata.cloud -> user menu -> Settings -> API Tokens -> Create. `scope:all` (full) or `scope:grafana-plugin` (read-only data). | 36-char UUID-shaped token |
33
+| `NETDATA_CLOUD_HOSTNAME` | Cloud REST API host | Almost always `app.netdata.cloud` | `app.netdata.cloud` |
34
+| `NETDATA_REPOS_DIR` | local Netdata-org repos mirror dir | Pick or create. Will be populated by `mirror-netdata-repos` skill's sync script. | `$HOME/src/netdata` |
35
+
36
+### agent-events ingestion node
37
+
38
+The `agent-events` node is the Netdata-operated ingestion
39
+host that receives status submissions from every Netdata
40
+agent in the wild. The query-agent-events skill triages
41
+crashes / panics / fatals from it.
42
+
43
+| Key | Role | Where to find it | Sample format |
44
+|---|---|---|---|
45
+| `AGENT_EVENTS_HOSTNAME` | Network address of the ingestion node. Dual-duty -- ssh host (`ssh ${AGENT_EVENTS_HOSTNAME}`) AND direct-HTTP host (`http://${AGENT_EVENTS_HOSTNAME}:19999/`). Can be a DNS name or an IP. NOTE -- this is NOT the journalctl namespace (which is hardcoded to `agent-events`) and NOT the Cloud room name (also hardcoded to `agent-events`). | Operations / your records | `10.20.1.105` or `agent-events.example` |
46
+| `AGENT_EVENTS_NODE_ID` | Cloud node UUID for that node | Visit the node in app.netdata.cloud and copy the UUID from the URL; or list nodes via the Cloud API and pick the matching one. | UUID |
47
+| `AGENT_EVENTS_MACHINE_GUID` | Netdata machine GUID for that node | On the host: `sudo cat /var/lib/netdata/registry/netdata.public.unique.id` | UUID |
48
+
49
+### Coverity Scan (coverity-audit skill)
50
+
51
+| Key | Role | Where to find it | Sample format |
52
+|---|---|---|---|
53
+| `COVERITY_HOST` | Scan API host | Always `https://scan4.scan.coverity.com` for the new instance | URL |
54
+| `COVERITY_PROJECT_ID` | integer project id | URL query param `?projectId=...` when you click the project in the dashboard | small integer |
55
+| `COVERITY_COOKIE` | full browser Cookie header (with XSRF-TOKEN) | DevTools -> Network -> any request to scan4 -> Request Headers -> Cookie | long Cookie string |
56
+| `COVERITY_VIEW_OUTSTANDING` | integer viewId for the "Outstanding" view | URL query param `?viewId=...` when you open that view | small integer |
57
+
58
+The cookie expires; refresh by re-pasting from the browser
59
+(or run `coverity-audit/scripts/keepalive.sh` to extend
60
+it during a triage session).
61
+
62
+### SonarCloud (sonarqube-audit skill)
63
+
64
+| Key | Role | Where to find it | Sample format |
65
+|---|---|---|---|
66
+| `SONAR_HOST_URL` | SonarCloud host | Always `https://sonarcloud.io` | URL |
67
+| `SONAR_ORG` | your organization key on SonarCloud | sonarcloud.io organization page | short string |
68
+| `SONAR_PROJECT` | projectKey on SonarCloud | For Netdata: `netdata_netdata` | `org_repo` form |
69
+| `SONAR_TOKEN` | personal access token | https://sonarcloud.io/account/security -> Generate | long opaque token |
70
+
71
+### Codacy Cloud (codacy-audit skill)
72
+
73
+| Key | Role | Where to find it | Sample format |
74
+|---|---|---|---|
75
+| `CODACY_TOKEN` | Account API token (header `api-token: <value>`) | https://app.codacy.com -> top-right avatar -> Account -> API tokens -> "Create API Token" | 20-char opaque string |
76
+| `CODACY_HOST` | API host. Defaults to `https://api.codacy.com`; set only if Codacy moves the API host. | n/a | URL |
77
+| `CODACY_PROVIDER` | git provider. Defaults to `gh` (GitHub). | n/a | `gh` |
78
+| `CODACY_ORG` | Codacy organization (matches the GitHub org). Defaults to `netdata`. | n/a | short string |
79
+| `CODACY_REPO` | Codacy repository name. Defaults to `netdata`. | n/a | short string |
80
+
81
+`CODACY_TOKEN` is required by `pr-issues.sh` and any wrapper that
82
+calls the v3 API. `analyze-local.sh` does NOT need it (the local
83
+CLI runs anonymously).
84
+
85
+## Per-skill checklist
86
+
87
+Set the keys for whichever skills you plan to use. Skills
88
+not listed here either need no `.env` keys or rely on `gh
89
+auth` instead.
90
+
91
+### query-netdata-cloud / query-netdata-agents
92
+
93
+- `NETDATA_CLOUD_TOKEN`
94
+- `NETDATA_CLOUD_HOSTNAME`
95
+- (For agent-events examples in those skills' docs:
96
+ `AGENT_EVENTS_HOSTNAME`, `AGENT_EVENTS_NODE_ID`,
97
+ `AGENT_EVENTS_MACHINE_GUID`.)
98
+
99
+### query-agent-events
100
+
101
+- `NETDATA_CLOUD_TOKEN`
102
+- `NETDATA_CLOUD_HOSTNAME`
103
+- `AGENT_EVENTS_HOSTNAME`
104
+- `AGENT_EVENTS_NODE_ID`
105
+- `AGENT_EVENTS_MACHINE_GUID`
106
+
107
+### mirror-netdata-repos
108
+
109
+- `NETDATA_REPOS_DIR`
110
+
111
+### integrations-lifecycle / learn-site-structure
112
+
113
+- `NETDATA_REPOS_DIR` (for cross-repo path references in
114
+ examples / recipes)
115
+
116
+### coverity-audit
117
+
118
+- `COVERITY_HOST`
119
+- `COVERITY_PROJECT_ID`
120
+- `COVERITY_COOKIE`
121
+- `COVERITY_VIEW_OUTSTANDING`
122
+
123
+### sonarqube-audit
124
+
125
+- `SONAR_HOST_URL`
126
+- `SONAR_ORG`
127
+- `SONAR_PROJECT`
128
+- `SONAR_TOKEN`
129
+
130
+### codacy-audit
131
+
132
+- `CODACY_TOKEN` (required by `pr-issues.sh`; not by `analyze-local.sh`)
133
+- `CODACY_HOST` (optional; defaults to `https://api.codacy.com`)
134
+- `CODACY_PROVIDER` / `CODACY_ORG` / `CODACY_REPO` (optional; default to `gh` / `netdata` / `netdata`)
135
+
136
+### pr-reviews / graphql-audit
137
+
138
+- No `.env` keys required. Both rely on `gh auth login`
139
+ having been run.
140
+
141
+## Common mistakes
142
+
143
+- **Trailing whitespace** in a value: bash variable
144
+ expansion preserves the whitespace; the value comes
145
+ through with the trailing space and breaks API calls
146
+ silently. Strip whitespace inside the quotes.
147
+- **Wrong quoting**: quotes around bash-expansion characters
148
+ (`$`, backticks, `\`) are interpreted. For tokens
149
+ containing those characters, use single quotes:
150
+ `SONAR_TOKEN='abc$def'`.
151
+- **Expired Coverity cookie**: re-paste from the browser.
152
+ The script's error message will tell you when this
153
+ happens.
154
+- **Wrong `gh` org**: `pr-reviews` and `graphql-audit` use
155
+ `gh` against the current repo's remote. Make sure your
156
+ remote points to the right repo (`git remote -v`).
157
+- **Cloud token scope too narrow**: some endpoints require
158
+ `scope:all`. If you get a 403 with what looks like a valid
159
+ token, regenerate with broader scope.
160
+- **`NETDATA_REPOS_DIR` and tilde**: bash does NOT expand
161
+ the home-directory shortcut character inside
162
+ double-quoted strings. If you write `"<TILDE>/src/netdata"`,
163
+ the literal tilde is kept in the value, and scripts will
164
+ fail with "directory does not exist" because that path
165
+ is not real. Use `$HOME` instead, or the full absolute
166
+ path:
167
+ ```
168
+ NETDATA_REPOS_DIR="$HOME/src/netdata"
169
+ ```
170
+
171
+## Why these are env-keyed
172
+
173
+Every value above is either:
174
+- a **secret** (token / cookie) that must never leak into
175
+ committed artifacts, or
176
+- a **per-user / per-deployment** path or identifier (mirror
177
+ dir, ingestion node) that varies between contributors.
178
+
179
+The committed skills, scripts, and docs reference these
180
+values exclusively via `${KEY}` placeholders, never literal
181
+values. The discipline is enforced by the spec at
182
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md`,
183
+which includes a pre-commit grep recipe to catch
184
+literal-value leaks.
185
+
186
+## When a skill says "X is empty in .env"
187
+
188
+That skill's `_lib.sh` ran the bash safety net
189
+`: "${X:?...}"` because `X` was unset or empty. Open this
190
+file, find the row for `X`, follow the "where to find it"
191
+pointer, paste the value into `.env`, and re-run.
.agents/skills/codacy-audit/SKILL.md
new
+119
@@ -0,0 +1,119 @@
1
+---
2
+name: codacy-audit
3
+description: Codacy Cloud workflow for this repository -- run Codacy's analyzers locally before `git push` (mirrors what Codacy CI runs), and fetch/cluster Codacy issues for any PR via the v3 API. Use when the user mentions Codacy, "codacy analysis", `codacy-analysis-cli`, "codacy issues on PR", "fix codacy CI", "codacy markdownlint findings", or any Codacy gate failing on a netdata-org PR. Ships scripts analyze-local.sh (docker/binary runner for codacy-analysis-cli) and pr-issues.sh (paginated v3 issue fetch + group-by tool/pattern/severity/file). Token-safe -- CODACY_TOKEN never reaches assistant-visible stdout. Read-only by design in the current SOW; write actions (mark FP, mark fixed) are deferred.
4
+---
5
+
6
+# Codacy audit skill
7
+
8
+Drives Codacy Cloud for `netdata/netdata`:
9
+
10
+1. **Pre-push prevention** -- run the same analyzers Codacy CI runs, locally, before `git push`. Collapses the "push -> wait minutes -> see findings -> fix -> push again" loop into one push.
11
+2. **Read-only PR triage** -- list Codacy issues for any PR, cluster by tool / pattern / severity / file, drop the JSON dump under `<repo>/.local/audits/codacy/`.
12
+
13
+This skill is the fourth in the static-analysis triage family in this repo:
14
+`coverity-audit/`, `sonarqube-audit/`, `graphql-audit/`, `codacy-audit/`. Same shape, same conventions, same artifact directory.
15
+
16
+## MANDATORY -- keep this skill alive
17
+
18
+If you (the assistant) discover a new pattern, gotcha, working flow, correction, or any operational knowledge while running this skill -- update this `SKILL.md` AND commit it BEFORE proceeding. Knowledge that isn't committed is lost.
19
+
20
+Examples worth capturing:
21
+- New v3 API endpoint or response-shape detail learned the hard way
22
+- Codacy-side rate-limit signals
23
+- A pattern Codacy mismodels for this codebase (so the next assistant can add a path exclusion or mark it FP)
24
+- A new tool the local CLI gained / lost
25
+- Auth-failure surface (e.g. token type mismatch, expired token signs)
26
+
27
+## MANDATORY -- live how-tos catalog
28
+
29
+Each concrete question that requires non-trivial analysis (multiple wrapper calls, jq pipelines, cross-referencing other skills) MUST become a how-to under `how-tos/<slug>.md` AND get an entry in `how-tos/INDEX.md` BEFORE the task is reported complete. Skipping this means the next assistant repeats the analysis from scratch.
30
+
31
+## Scope (current SOW)
32
+
33
+In scope:
34
+
35
+- Local pre-push analysis via `codacy-analysis-cli` (auto-detects local binary, falls back to docker).
36
+- Read-only PR-issue queries against the v3 API.
37
+- Token-safe wrappers (sentinel-driven no-leak self-test).
38
+
39
+Out of scope (deferred to a future SOW):
40
+
41
+- Write actions (mark issue as false-positive, mark as fixed, modify ignore-patterns).
42
+- Master-backlog triage on the 31,425+ open issues.
43
+- Cross-repo aggregation across the netdata org.
44
+
45
+## Required env keys
46
+
47
+| Key | Required for |
48
+|---|---|
49
+| `CODACY_TOKEN` | Account API token, header `api-token: <value>`. Required by `pr-issues.sh` and any wrapper that calls `_codacyaudit_run`. NOT required by `analyze-local.sh` (the CLI runs anonymously). |
50
+| `CODACY_HOST` | Defaults to `https://api.codacy.com`. Override only if Codacy moves the API host. |
51
+| `CODACY_PROVIDER` | Defaults to `gh` (GitHub). |
52
+| `CODACY_ORG` | Defaults to `netdata`. |
53
+| `CODACY_REPO` | Defaults to `netdata`. |
54
+
55
+All values live in `<repo>/.env` (gitignored). See `<repo>/.agents/ENV.md` for setup (where each value comes from, sample formats, common mistakes).
56
+
57
+## Scripts (in scripts/)
58
+
59
+| Script | Purpose |
60
+|---|---|
61
+| `_lib.sh` | Helpers (`codacyaudit_*` prefix). Token-safe; ships `codacyaudit_selftest_no_token_leak`. |
62
+| `analyze-local.sh` | Run `codacy-analysis-cli` locally; auto-pick local-binary or docker; write JSON dump under `.local/audits/codacy/`. |
63
+| `pr-issues.sh` | Fetch all Codacy issues for a PR via the v3 API; cluster summary on stdout; full JSON dump on disk. |
64
+
65
+## Workflow -- pre-push prevention
66
+
67
+```
68
+$ .agents/skills/codacy-audit/scripts/analyze-local.sh
69
+[analyze-local] runner=docker format=json dir=<repo>
70
+[analyze-local] wrote 0 finding(s) to <repo>/.local/audits/codacy/local-<ts>.json
71
+```
72
+
73
+Run this before `git push`. If it returns 0 findings, the Codacy gate on the PR will be green (modulo Codacy server-side patterns the local CLI doesn't bundle). If it returns findings, fix them locally first.
74
+
75
+To restrict to a single tool (matches what Codacy reported on a CI run):
76
+
77
+```
78
+$ .agents/skills/codacy-audit/scripts/analyze-local.sh --tool markdownlint
79
+```
80
+
81
+## Workflow -- PR triage
82
+
83
+```
84
+$ .agents/skills/codacy-audit/scripts/pr-issues.sh 22423
85
+[pr-issues] fetching issues for PR #22423 ...
86
+[pr-issues] wrote 0 issue(s) to <repo>/.local/audits/codacy/pr-22423-<ts>.json
87
+
88
+No issues on PR #22423.
89
+```
90
+
91
+For a PR with findings, the script emits a clustered TSV summary. Default grouping is `--by pattern`; switch to `--by tool`, `--by severity`, `--by file`, or `--by category` for other angles. The JSON dump under `.local/audits/codacy/` carries the full issue payload for follow-up jq queries.
92
+
93
+## Path discipline
94
+
95
+This skill follows `<repo>/.agents/sow/specs/sensitive-data-discipline.md`:
96
+
97
+- Repo files: repo-relative (`<repo>/src/...`).
98
+- Codacy account / org / repo identifiers: env-keyed.
99
+- `CODACY_TOKEN`: NEVER literal in any committed file; ALWAYS via `${CODACY_TOKEN}` and the `_lib.sh` wrappers.
100
+- Audit dumps: gitignored under `<repo>/.local/audits/codacy/`.
101
+
102
+## Related skills
103
+
104
+- `.agents/skills/coverity-audit/` -- Coverity Scan (same triage shape).
105
+- `.agents/skills/sonarqube-audit/` -- SonarCloud (same triage shape).
106
+- `.agents/skills/graphql-audit/` -- GitHub Code Scanning / CodeQL (same triage shape).
107
+
108
+## Token-safe self-test
109
+
110
+Before trusting wrappers in a long-running session, run the self-test:
111
+
112
+```
113
+$ source .agents/skills/codacy-audit/scripts/_lib.sh
114
+$ codacyaudit_load_env
115
+$ codacyaudit_selftest_no_token_leak
116
+PASS: codacyaudit_selftest_no_token_leak
117
+```
118
+
119
+The self-test sets `CODACY_TOKEN` to a sentinel UUID, drives every public wrapper, captures stdout, and asserts the sentinel never appears. Run after editing `_lib.sh` or any wrapper.
.agents/skills/codacy-audit/how-tos/INDEX.md
new
+9
@@ -0,0 +1,9 @@
1
+# codacy-audit how-tos catalog
2
+
3
+This catalog is **live**. When an assistant performs concrete analysis on a Codacy question that requires more than one wrapper call OR more than one jq pipeline OR cross-referencing another skill, AND the answer is not already documented here or in the per-domain SKILL.md, the assistant MUST author a new entry under `how-tos/<slug>.md` AND add it here BEFORE marking the task complete.
4
+
5
+Skipping this rule means the next assistant repeats the analysis from scratch -- that is an explicit framework violation.
6
+
7
+## Entries
8
+
9
+- [reproduce-pr-22423-markdownlint](reproduce-pr-22423-markdownlint.md) -- reproduce the 864 markdownlint findings PR #22423 saw on its first CI run, locally via `analyze-local.sh --tool markdownlint`.
.agents/skills/codacy-audit/how-tos/reproduce-pr-22423-markdownlint.md
new
+73
@@ -0,0 +1,73 @@
1
+# How-to: reproduce PR #22423's 864 markdownlint findings locally
2
+
3
+## When to use
4
+
5
+You want to confirm `analyze-local.sh` matches what Codacy CI reported on a known fixture. PR #22423 is the canonical fixture for this skill: its first CI run reported **864 markdownlint findings** (recorded in the SOW that built this skill, SOW-0011); commit `3a54c9afbc` cleared them by adding `.agents/**` and `docs/netdata-ai/skills/**` to `.codacy.yml`.
6
+
7
+This how-to walks through reproducing those 864 findings on the pre-exclusion state, then confirming the post-exclusion state shows zero on the affected files.
8
+
9
+## Prerequisite
10
+
11
+- `docker` available (or `codacy-analysis-cli` installed locally).
12
+- `<repo>/.env` need NOT contain `CODACY_TOKEN` -- `analyze-local.sh` runs the CLI anonymously.
13
+
14
+## Step 1 -- check out the pre-exclusion state
15
+
16
+PR #22423 introduced the exclusion in commit `3a54c9afbc`. The parent commit `d7791e6838` is the "before" state.
17
+
18
+```bash
19
+git checkout d7791e6838 -- .codacy.yml # restore the pre-exclusion .codacy.yml
20
+# (do NOT switch branches; just stage the older .codacy.yml)
21
+```
22
+
23
+## Step 2 -- run analyze-local on markdownlint only
24
+
25
+```bash
26
+.agents/skills/codacy-audit/scripts/analyze-local.sh --tool markdownlint
27
+```
28
+
29
+Expected: a JSON dump under `<repo>/.local/audits/codacy/local-markdownlint-<ts>.json`. The CLI returns non-zero when findings exist (this is normal; the script tolerates it).
30
+
31
+## Step 3 -- count findings
32
+
33
+```bash
34
+DUMP="$(ls -1t .local/audits/codacy/local-markdownlint-*.json | head -1)"
35
+jq '
36
+ if type=="array" then length
37
+ elif type=="object" and has("issues") then (.issues | length)
38
+ elif type=="object" and has("results") then (.results | length)
39
+ else 0 end
40
+' "$DUMP"
41
+```
42
+
43
+Expected: a count close to 864 (within ~10% tolerance for tool-version drift between the CLI bundle and Codacy Cloud).
44
+
45
+## Step 4 -- restore the exclusion
46
+
47
+```bash
48
+git checkout HEAD -- .codacy.yml
49
+```
50
+
51
+## Step 5 -- re-run and confirm zero on excluded paths
52
+
53
+```bash
54
+.agents/skills/codacy-audit/scripts/analyze-local.sh --tool markdownlint
55
+DUMP="$(ls -1t .local/audits/codacy/local-markdownlint-*.json | head -1)"
56
+jq '[ ... | select(.filePath | startswith(".agents/") or startswith("docs/netdata-ai/skills/")) ] | length' "$DUMP"
57
+```
58
+
59
+(The exact jq filter depends on the dump shape -- consult the dump structure first via `jq 'keys' "$DUMP"`.)
60
+
61
+Expected: zero rows in the excluded trees.
62
+
63
+## What this validates
64
+
65
+- `analyze-local.sh` runs end-to-end against the configured runner (docker or local binary).
66
+- The CLI honours `.codacy.yml` exclude_paths (or, if it doesn't, we have empirical evidence to handle that gap in a future SOW).
67
+- The bundled markdownlint version produces a count consistent with what Codacy CI reports.
68
+
69
+## Troubleshooting
70
+
71
+- **Docker pull is slow on first run**: `codacy/codacy-analysis-cli:latest` is a few hundred MB. Subsequent runs use the warm cache.
72
+- **Different count than 864**: tool-version drift between the bundled `codacy-analysis-cli` and Codacy Cloud is normal. ~10% tolerance is fine. Anything wider warrants checking the CLI version vs Cloud's reported version.
73
+- **CLI exits with non-zero**: that's expected when findings are present. The script suppresses this; the JSON dump is still valid.
.agents/skills/codacy-audit/scripts/_lib.sh
new
+224
@@ -0,0 +1,224 @@
1
+#!/usr/bin/env bash
2
+# Common helpers for codacy-audit scripts.
3
+#
4
+# Token-safe by design: CODACY_TOKEN never reaches the
5
+# assistant-visible stdout. Internal helpers that handle
6
+# credential bytes have leading-underscore names; public
7
+# wrappers read .env internally and emit only the response
8
+# body.
9
+#
10
+# Sourced from the per-action scripts; not executed directly.
11
+
12
+set -euo pipefail
13
+
14
+# ANSI colors. Real ESC bytes via $'...' so the variables work
15
+# uniformly with echo -e and printf. Color vars are referenced
16
+# by sourcing scripts; shellcheck cannot see that.
17
+# shellcheck disable=SC2034
18
+CA_RED=$'\033[0;31m'
19
+# shellcheck disable=SC2034
20
+CA_GREEN=$'\033[0;32m'
21
+# shellcheck disable=SC2034
22
+CA_YELLOW=$'\033[1;33m'
23
+# shellcheck disable=SC2034
24
+CA_GRAY=$'\033[0;90m'
25
+# shellcheck disable=SC2034
26
+CA_CYAN=$'\033[0;36m'
27
+# shellcheck disable=SC2034
28
+CA_NC=$'\033[0m'
29
+
30
+# Resolve this lib's path (zsh + bash compatible). The query-agent-events
31
+# skill uses the same idiom; mirror it here so sourcing from either shell
32
+# works without warnings.
33
+if [ -n "${ZSH_VERSION-}" ]; then
34
+ eval '_codacyaudit_lib_self="${(%):-%x}"'
35
+elif [ -n "${BASH_VERSION-}" ]; then
36
+ _codacyaudit_lib_self="${BASH_SOURCE[0]}"
37
+else
38
+ _codacyaudit_lib_self="$0"
39
+fi
40
+_codacyaudit_lib_dir="$(cd "$(dirname "$_codacyaudit_lib_self")" && pwd)"
41
+
42
+# Locate the repo root from this script's location.
43
+codacyaudit_repo_root() {
44
+ git -C "$_codacyaudit_lib_dir" rev-parse --show-toplevel
45
+}
46
+
47
+# Source <repo-root>/.env. CODACY_TOKEN is required for token-gated
48
+# endpoints (issue search across master, repo metadata, future write
49
+# actions). Read-only PR-issue queries also work anonymously, but
50
+# this skill drives them through the token wrapper for consistency
51
+# and to exercise the no-leak self-test on every run.
52
+codacyaudit_load_env() {
53
+ local root env
54
+ root="$(codacyaudit_repo_root)"
55
+ env="${root}/.env"
56
+ if [[ ! -f "${env}" || ! -r "${env}" ]]; then
57
+ echo -e "${CA_RED}[ERROR]${CA_NC} Missing ${env}. See <repo>/.agents/ENV.md for the setup guide." >&2
58
+ return 1
59
+ fi
60
+ set -a
61
+ # shellcheck disable=SC1090
62
+ source "${env}"
63
+ set +a
64
+
65
+ : "${CODACY_TOKEN:?CODACY_TOKEN is empty -- see <repo>/.agents/ENV.md to set it.}"
66
+ : "${CODACY_HOST:=https://api.codacy.com}"
67
+ : "${CODACY_PROVIDER:=gh}"
68
+ : "${CODACY_ORG:=netdata}"
69
+ : "${CODACY_REPO:=netdata}"
70
+
71
+ export CODACY_TOKEN CODACY_HOST CODACY_PROVIDER CODACY_ORG CODACY_REPO
72
+}
73
+
74
+# Audit artifacts go under .local/audits/codacy/ at the repo root.
75
+# .local/ is gitignored -- see AGENTS.md for the convention.
76
+codacyaudit_audit_dir() {
77
+ local root dir
78
+ root="$(codacyaudit_repo_root)"
79
+ dir="${root}/.local/audits/codacy"
80
+ mkdir -p "${dir}"
81
+ echo "${dir}"
82
+}
83
+
84
+# ---------------------------------------------------------------
85
+# Token-safe HTTP wrappers.
86
+#
87
+# The internal helper handles the token bytes. Public wrappers
88
+# call it and emit response body only on stdout. We never echo
89
+# the curl command line (which would expose the token).
90
+
91
+# _codacyaudit_run METHOD PATH [DATA]
92
+# Returns the response body on stdout. HTTP non-2xx -> non-zero.
93
+# stderr: minimal status line on error.
94
+_codacyaudit_run() {
95
+ local method="$1"
96
+ local path="$2"
97
+ local data="${3:-}"
98
+ local url="${CODACY_HOST}${path}"
99
+
100
+ local -a curl_args=(
101
+ --silent --show-error --fail-with-body
102
+ --max-time 60
103
+ --request "$method"
104
+ --header "api-token: ${CODACY_TOKEN}"
105
+ --header 'Accept: application/json'
106
+ )
107
+ if [ -n "$data" ]; then
108
+ curl_args+=(--header 'Content-Type: application/json' --data-raw "$data")
109
+ fi
110
+
111
+ local body
112
+ if ! body="$(curl "${curl_args[@]}" "$url")"; then
113
+ echo -e "${CA_RED}[ERROR]${CA_NC} ${method} ${path} failed (see body below)" >&2
114
+ printf '%s\n' "$body" >&2
115
+ return 1
116
+ fi
117
+ printf '%s' "$body"
118
+}
119
+
120
+# Public GET. Stdout is the response body; the token never leaks.
121
+codacyaudit_get() {
122
+ local path="$1"
123
+ _codacyaudit_run GET "$path"
124
+}
125
+
126
+# Public POST.
127
+codacyaudit_post() {
128
+ local path="$1"
129
+ local data="$2"
130
+ _codacyaudit_run POST "$path" "$data"
131
+}
132
+
133
+# Paginated GET. Walks the v3 cursor protocol and concatenates
134
+# `data[]` into a single JSON array on stdout.
135
+#
136
+# Codacy v3 pagination:
137
+# request : ?cursor=<c>&limit=<n>
138
+# response: { data: [...], pagination: { cursor, limit, total } }
139
+codacyaudit_get_paged() {
140
+ local path_base="$1"
141
+ local limit="${2:-1000}"
142
+ local sep cursor=""
143
+ local first=true
144
+ local out='[]'
145
+
146
+ while :; do
147
+ if [[ "$path_base" == *'?'* ]]; then sep='&'; else sep='?'; fi
148
+ local path="${path_base}${sep}limit=${limit}"
149
+ if [ -n "$cursor" ]; then
150
+ path="${path}&cursor=${cursor}"
151
+ fi
152
+
153
+ local resp
154
+ resp="$(_codacyaudit_run GET "$path")" || return 1
155
+
156
+ # Append data[] to accumulator.
157
+ out="$(printf '%s\n%s' "$out" "$resp" \
158
+ | jq -sc '.[0] + (.[1].data // [])')"
159
+
160
+ cursor="$(printf '%s' "$resp" | jq -r '.pagination.cursor // ""')"
161
+ [ -z "$cursor" ] && break
162
+ $first || [ "$first" = "false" ] # keep loop simple
163
+ first=false
164
+ done
165
+
166
+ printf '%s' "$out"
167
+}
168
+
169
+# ---------------------------------------------------------------
170
+# Convenience wrappers for the two endpoints this SOW ships.
171
+
172
+# PR issues: GET /v3/analysis/organizations/<p>/<o>/repositories/<r>/pull-requests/<n>/issues
173
+codacyaudit_pr_issues() {
174
+ local pr="$1"
175
+ if [[ ! "$pr" =~ ^[1-9][0-9]*$ ]]; then
176
+ echo -e "${CA_RED}[ERROR]${CA_NC} PR number must be a positive integer, got: '${pr}'" >&2
177
+ return 1
178
+ fi
179
+ codacyaudit_get_paged \
180
+ "/api/v3/analysis/organizations/${CODACY_PROVIDER}/${CODACY_ORG}/repositories/${CODACY_REPO}/pull-requests/${pr}/issues"
181
+}
182
+
183
+# Repo overview: GET /v3/organizations/<p>/<o>/repositories/<r>
184
+codacyaudit_repo_info() {
185
+ codacyaudit_get \
186
+ "/api/v3/organizations/${CODACY_PROVIDER}/${CODACY_ORG}/repositories/${CODACY_REPO}"
187
+}
188
+
189
+# ---------------------------------------------------------------
190
+# No-token-leak self-test.
191
+#
192
+# Drives every public wrapper with a sentinel CODACY_TOKEN and
193
+# asserts the sentinel never appears on captured stdout. Run
194
+# this after editing any wrapper.
195
+
196
+codacyaudit_selftest_no_token_leak() {
197
+ local sentinel="deadbeef-1234-5678-9abc-def012345678"
198
+
199
+ local saved="${CODACY_TOKEN:-}"
200
+ CODACY_TOKEN="$sentinel"
201
+ export CODACY_TOKEN
202
+
203
+ # Drive each public wrapper. The expected outcome is HTTP
204
+ # 401 (sentinel is not a real token); we capture stdout and
205
+ # assert the sentinel does not appear there.
206
+ local out
207
+ out="$( {
208
+ codacyaudit_get "/api/v3/user" 2>/dev/null || true
209
+ codacyaudit_post "/api/v3/user" '{"noop":true}' 2>/dev/null || true
210
+ codacyaudit_pr_issues 22423 2>/dev/null || true
211
+ codacyaudit_repo_info 2>/dev/null || true
212
+ } )"
213
+
214
+ CODACY_TOKEN="$saved"
215
+ export CODACY_TOKEN
216
+
217
+ if printf '%s' "$out" | grep -q "$sentinel"; then
218
+ echo -e "${CA_RED}FAIL${CA_NC}: sentinel ${sentinel} appeared on captured stdout" >&2
219
+ return 1
220
+ fi
221
+
222
+ echo -e "${CA_GREEN}PASS${CA_NC}: codacyaudit_selftest_no_token_leak"
223
+ return 0
224
+}
.agents/skills/codacy-audit/scripts/analyze-local.sh
new
+145
@@ -0,0 +1,145 @@
1
+#!/usr/bin/env bash
2
+# analyze-local.sh -- run codacy-analysis-cli locally on the working tree.
3
+#
4
+# Mirrors what Codacy CI would run on the same source. Useful BEFORE
5
+# `git push` to catch findings in seconds, not minutes.
6
+#
7
+# Output: a JSON dump under <repo>/.local/audits/codacy/.
8
+# stdout (last line): the dump path.
9
+
10
+set -euo pipefail
11
+
12
+usage() {
13
+ cat <<'EOF'
14
+analyze-local.sh [options]
15
+
16
+Runs the official codacy-analysis-cli (https://github.com/codacy/codacy-analysis-cli)
17
+on the current working tree and writes a JSON dump under
18
+<repo>/.local/audits/codacy/. The cli respects the repo's .codacy.yml
19
+exclude_paths.
20
+
21
+Options:
22
+ --tool <name> run a single tool (e.g. shellcheck, markdownlint).
23
+ Omit to run all tools applicable to changed files.
24
+ --directory <path> analyze a subpath (default: <repo-root>)
25
+ --format json|sarif output format (default: json)
26
+ --output PATH explicit dump path (default: auto under .local/audits/codacy/)
27
+ --runner docker|local installer to use (default: auto -- prefer local
28
+ binary, fall back to docker, fall back to npm)
29
+ -h, --help
30
+
31
+Required tools: docker (default) OR a local codacy-analysis-cli binary.
32
+EOF
33
+}
34
+
35
+TOOL=
36
+SUBDIR=
37
+FORMAT=json
38
+OUTPUT=
39
+RUNNER=auto
40
+
41
+while [ $# -gt 0 ]; do
42
+ case "$1" in
43
+ --tool) TOOL="$2"; shift 2 ;;
44
+ --directory) SUBDIR="$2"; shift 2 ;;
45
+ --format) FORMAT="$2"; shift 2 ;;
46
+ --output) OUTPUT="$2"; shift 2 ;;
47
+ --runner) RUNNER="$2"; shift 2 ;;
48
+ -h|--help) usage; exit 0 ;;
49
+ *) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;;
50
+ esac
51
+done
52
+
53
+# shellcheck source=SCRIPTDIR/_lib.sh disable=SC1091
54
+source "$(cd "$(dirname "$0")" && pwd)/_lib.sh"
55
+
56
+# We do not require CODACY_TOKEN here -- the CLI runs without it
57
+# for read-only local analysis. Skip env load to avoid forcing
58
+# users without a token to set one just to run pre-push checks.
59
+
60
+repo_root="$(codacyaudit_repo_root)"
61
+audit_dir="$(codacyaudit_audit_dir)"
62
+
63
+# Resolve target directory.
64
+if [ -z "$SUBDIR" ]; then
65
+ SUBDIR="$repo_root"
66
+else
67
+ case "$SUBDIR" in
68
+ /*) : ;; # absolute
69
+ *) SUBDIR="$(cd "$SUBDIR" && pwd)" ;;
70
+ esac
71
+fi
72
+
73
+# Resolve output path.
74
+if [ -z "$OUTPUT" ]; then
75
+ suffix=""
76
+ [ -n "$TOOL" ] && suffix="-${TOOL}"
77
+ OUTPUT="${audit_dir}/local${suffix}-$(date -u +%Y%m%dT%H%M%SZ).${FORMAT}"
78
+fi
79
+
80
+# Pick a runner.
81
+if [ "$RUNNER" = "auto" ]; then
82
+ if command -v codacy-analysis-cli >/dev/null 2>&1; then
83
+ RUNNER=local
84
+ elif command -v docker >/dev/null 2>&1; then
85
+ RUNNER=docker
86
+ else
87
+ echo -e "${CA_RED}[ERROR]${CA_NC} neither 'codacy-analysis-cli' nor 'docker' found in PATH." >&2
88
+ echo "Install options:" >&2
89
+ echo " - docker: https://docs.docker.com/get-docker/" >&2
90
+ echo " - cli: https://github.com/codacy/codacy-analysis-cli#install" >&2
91
+ exit 2
92
+ fi
93
+fi
94
+
95
+echo -e "${CA_GRAY}[analyze-local] runner=${RUNNER} format=${FORMAT} dir=${SUBDIR}${CA_NC}" >&2
96
+
97
+case "$RUNNER" in
98
+ local)
99
+ # Local binary expects host paths.
100
+ local_args=(analyze --directory "$SUBDIR" --format "$FORMAT")
101
+ [ -n "$TOOL" ] && local_args+=(--tool "$TOOL")
102
+ if ! codacy-analysis-cli "${local_args[@]}" > "$OUTPUT" 2>/dev/null; then
103
+ echo -e "${CA_YELLOW}[analyze-local] cli returned non-zero (this is normal when findings are present)${CA_NC}" >&2
104
+ fi
105
+ ;;
106
+ docker)
107
+ # Per https://github.com/codacy/codacy-analysis-cli the CLI
108
+ # spawns one child container per tool and needs:
109
+ # - the host docker socket (docker-in-docker)
110
+ # - CODACY_CODE pointing at the host path of the source
111
+ # - the source bind-mounted at the SAME path inside the
112
+ # CLI container so child containers can resolve it
113
+ cli_args=(analyze --directory "$SUBDIR" --format "$FORMAT")
114
+ [ -n "$TOOL" ] && cli_args+=(--tool "$TOOL")
115
+ if ! docker run --rm \
116
+ --env CODACY_CODE="$SUBDIR" \
117
+ --volume /var/run/docker.sock:/var/run/docker.sock \
118
+ --volume "$SUBDIR":"$SUBDIR" \
119
+ codacy/codacy-analysis-cli:latest \
120
+ "${cli_args[@]}" > "$OUTPUT" 2>/dev/null; then
121
+ echo -e "${CA_YELLOW}[analyze-local] cli returned non-zero (this is normal when findings are present)${CA_NC}" >&2
122
+ fi
123
+ ;;
124
+ *)
125
+ echo -e "${CA_RED}[ERROR]${CA_NC} unknown --runner '${RUNNER}'" >&2
126
+ exit 2
127
+ ;;
128
+esac
129
+
130
+# Sanity check the output.
131
+if [ ! -s "$OUTPUT" ]; then
132
+ echo -e "${CA_RED}[ERROR]${CA_NC} empty output at ${OUTPUT}; check the runner above" >&2
133
+ exit 1
134
+fi
135
+
136
+# Quick summary if format=json.
137
+if [ "$FORMAT" = "json" ] && jq -e . "$OUTPUT" >/dev/null 2>&1; then
138
+ n="$(jq 'if type=="array" then length elif type=="object" and has("issues") then (.issues|length) else 0 end' "$OUTPUT")"
139
+ echo -e "${CA_GREEN}[analyze-local]${CA_NC} wrote ${n} finding(s) to ${OUTPUT}" >&2
140
+else
141
+ echo -e "${CA_GREEN}[analyze-local]${CA_NC} wrote ${OUTPUT} (${FORMAT} format)" >&2
142
+fi
143
+
144
+# Last line on stdout: the path. Pipe-friendly.
145
+echo "$OUTPUT"
.agents/skills/codacy-audit/scripts/pr-issues.sh
new
+131
@@ -0,0 +1,131 @@
1
+#!/usr/bin/env bash
2
+# pr-issues.sh -- fetch Codacy issues for a PR and emit a clustered summary.
3
+#
4
+# Output:
5
+# - Full JSON dump of issues:
6
+# <repo>/.local/audits/codacy/pr-<NNN>-<timestamp>.json
7
+# - TSV summary clustering by tool / pattern / severity / file
8
+# printed to stdout.
9
+
10
+set -euo pipefail
11
+
12
+usage() {
13
+ cat <<'EOF'
14
+pr-issues.sh <pr-number> [options]
15
+
16
+Fetches the Codacy v3 PR issues list for the given PR, paginating
17
+through every result, and writes a JSON dump under
18
+<repo>/.local/audits/codacy/. Emits a TSV summary on stdout
19
+clustering issues by tool / pattern / severity / file.
20
+
21
+Options:
22
+ --output PATH explicit output path for the JSON dump
23
+ --by <dim> summary dimension: tool|pattern|severity|file|category
24
+ (default: pattern)
25
+ --top N top-N rows in the summary (default: 25)
26
+ -h, --help
27
+
28
+Required env: CODACY_TOKEN (see <repo>/.agents/ENV.md)
29
+Defaults: org=netdata, repo=netdata, provider=gh
30
+ (override via CODACY_ORG / CODACY_REPO / CODACY_PROVIDER in .env)
31
+EOF
32
+}
33
+
34
+PR=
35
+OUTPUT=
36
+BY=pattern
37
+TOP=25
38
+
39
+while [ $# -gt 0 ]; do
40
+ case "$1" in
41
+ --output) OUTPUT="$2"; shift 2 ;;
42
+ --by) BY="$2"; shift 2 ;;
43
+ --top) TOP="$2"; shift 2 ;;
44
+ -h|--help) usage; exit 0 ;;
45
+ -*) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;;
46
+ *)
47
+ if [ -z "$PR" ]; then PR="$1"
48
+ else echo "Unexpected positional arg: $1" >&2; usage >&2; exit 2
49
+ fi
50
+ shift
51
+ ;;
52
+ esac
53
+done
54
+
55
+if [ -z "$PR" ]; then
56
+ usage >&2
57
+ exit 2
58
+fi
59
+
60
+# shellcheck source=SCRIPTDIR/_lib.sh disable=SC1091
61
+source "$(cd "$(dirname "$0")" && pwd)/_lib.sh"
62
+codacyaudit_load_env
63
+
64
+# ---------------------------------------------------------------
65
+# Output path.
66
+
67
+if [ -z "$OUTPUT" ]; then
68
+ audit_dir="$(codacyaudit_audit_dir)"
69
+ OUTPUT="${audit_dir}/pr-${PR}-$(date -u +%Y%m%dT%H%M%SZ).json"
70
+fi
71
+
72
+echo -e "${CA_GRAY}[pr-issues] fetching issues for PR #${PR} ...${CA_NC}" >&2
73
+
74
+# Wrap raw issues array into an envelope with metadata.
75
+issues_array="$(codacyaudit_pr_issues "$PR")"
76
+total="$(printf '%s' "$issues_array" | jq 'length')"
77
+
78
+jq -n \
79
+ --arg pr "$PR" \
80
+ --arg fetched_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
81
+ --argjson total "$total" \
82
+ --argjson data "$issues_array" \
83
+ '{
84
+ pr: ($pr | tonumber),
85
+ fetched_at: $fetched_at,
86
+ total: $total,
87
+ data: $data
88
+ }' > "$OUTPUT"
89
+
90
+echo -e "${CA_GREEN}[pr-issues]${CA_NC} wrote ${total} issue(s) to ${OUTPUT}" >&2
91
+echo "$OUTPUT"
92
+
93
+# ---------------------------------------------------------------
94
+# Summary.
95
+
96
+if [ "$total" -eq 0 ]; then
97
+ echo
98
+ echo -e "${CA_GREEN}No issues on PR #${PR}.${CA_NC}"
99
+ exit 0
100
+fi
101
+
102
+field_for_dim() {
103
+ local dim="$1"
104
+ case "$dim" in
105
+ tool) echo '.commitIssue.toolInfo.name' ;;
106
+ pattern) echo '.commitIssue.patternInfo.id' ;;
107
+ severity) echo '.commitIssue.patternInfo.severityLevel' ;;
108
+ file) echo '.commitIssue.filePath' ;;
109
+ category) echo '.commitIssue.patternInfo.category' ;;
110
+ *) echo "Unknown --by '$dim'" >&2; exit 2 ;;
111
+ esac
112
+}
113
+
114
+field_path="$(field_for_dim "$BY")"
115
+
116
+echo
117
+echo -e "${CA_CYAN}Top ${TOP} by ${BY} (PR #${PR}, ${total} total issue(s)):${CA_NC}"
118
+printf '%s\n' "------------------------------------------------------------"
119
+
120
+jq -r --argjson top "$TOP" "
121
+ .data
122
+ | group_by(${field_path} // \"(none)\")
123
+ | map({key: (.[0] | ${field_path} // \"(none)\"), count: length})
124
+ | sort_by(-.count)
125
+ | .[:\$top]
126
+ | .[]
127
+ | [.count, .key] | @tsv
128
+" "$OUTPUT" \
129
+| awk -F'\t' '{ printf "%8d %s\n", $1, $2 }'
130
+
131
+printf '%s\n' "------------------------------------------------------------"
.agents/skills/integrations-lifecycle/SKILL.md
new
+151
@@ -0,0 +1,151 @@
1
+---
2
+name: integrations-lifecycle
3
+description: Authoritative reference for Netdata's integrations pipeline -- how `metadata.yaml` drives per-integration pages, the `COLLECTORS.md`/`SECRETS.md`/`SERVICE-DISCOVERY.md` umbrellas, the `integrations.js` artifact consumed by the cloud-frontend, and per-integration `.md` files committed to the repo. Use when adding/modifying any integration (collector, exporter, agent or cloud notification, authentication, secretstore, service-discovery, log type, deploy method); editing `metadata.yaml`; checking whether `integrations/*.md` should be hand-edited; reading the four generator scripts under `integrations/`, schemas under `integrations/schemas/`, templates under `integrations/templates/`, the workflows `generate-integrations.yml` or `check-markdown.yml`; ibm.d modules where `metadata.yaml` is generated from `contexts.yaml`; the 5-file consistency rule (metadata.yaml + config_schema.json + stock conf + alerts + README move together).
4
+---
5
+
6
+# integrations-lifecycle
7
+
8
+This skill is the **single place** to learn how Netdata's
9
+integrations pipeline works end to end. It documents:
10
+
11
+- the four-stage generator pipeline rooted in
12
+ `integrations/gen_integrations.py`;
13
+- the 12 JSON-Schema contracts every `metadata.yaml` is validated
14
+ against;
15
+- every artifact the pipeline produces (gitignored runtime files
16
+ AND committed `.md` documentation);
17
+- the `<!--startmeta` banner conventions and DO-NOT-EDIT rules;
18
+- the two CI workflows that auto-PR or gate the regenerated docs
19
+ (`generate-integrations.yml`, `check-markdown.yml`);
20
+- the secondary ibm.d generation chain
21
+ (`contexts.yaml` -> `metadata.yaml`);
22
+- the contract by which the cloud-frontend dashboard consumes
23
+ `integrations.js`;
24
+- the collector-consistency rule (5 files must move together)
25
+ and what is and is NOT enforced by tooling;
26
+- every surprising/dead/edge-case behavior an assistant or
27
+ maintainer is likely to hit.
28
+
29
+After reading SKILL.md plus the per-domain guides linked below,
30
+an assistant should never need to ask "how does metadata.yaml
31
+work?", "are these `integrations/*.md` files generated?", "what
32
+fields does the schema support?", "what runs in CI?", "where
33
+does the in-app integrations page get its data?".
34
+
35
+## Key concepts (read first)
36
+
37
+1. **`metadata.yaml` is the single source of truth.** Every
38
+ per-integration page on every surface (Learn site, in-app
39
+ dashboard, the umbrella `COLLECTORS.md` / `SECRETS.md` /
40
+ `SERVICE-DISCOVERY.md` pages) is rendered from
41
+ `metadata.yaml` by the pipeline. Edit `metadata.yaml`, run
42
+ the pipeline, commit the regenerated artifacts.
43
+
44
+2. **`integrations/*.md` files are GENERATED. DO NOT EDIT.**
45
+ Every per-integration `.md` opens with a
46
+ `<!--startmeta ... endmeta-->` block that ends with
47
+ `message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED
48
+ BY THE COLLECTOR'S/EXPORTER'S/...'S metadata.yaml FILE"`.
49
+ See `artifacts-and-banners.md` for the full banner spec.
50
+ Edit the source `metadata.yaml`, regenerate, commit.
51
+
52
+3. **The CI workflow auto-opens a "Regenerate integrations
53
+ docs" PR.** After a `metadata.yaml` change merges to
54
+ `master`, `.github/workflows/generate-integrations.yml`
55
+ regenerates every per-integration `.md` and the umbrella
56
+ pages and opens a PR for a maintainer to merge. You CAN
57
+ regenerate locally and include the changes in the same PR;
58
+ that is preferred to avoid two PRs per change.
59
+
60
+4. **The five-file consistency rule.** Anything that touches a
61
+ collector's runtime behavior MUST land in one PR with
62
+ matching changes to:
63
+ - `metadata.yaml` (the integration page driver),
64
+ - `config_schema.json` (the dashboard's DYNCFG editor),
65
+ - the stock `.conf` (what `/etc/netdata/...` ships),
66
+ - `health.d/*.conf` (the alert definitions),
67
+ - `README.md` (which is a symlink to the generated
68
+ `integrations/<slug>.md` for single-integration plugins).
69
+ See `consistency.md` for what is and is NOT automatically
70
+ enforced.
71
+
72
+5. **ibm.d is different.** ibm.d module `metadata.yaml`,
73
+ `README.md`, and `config_schema.json` are GENERATED from
74
+ `contexts.yaml` + `config.go` + `module.yaml` via
75
+ `go generate`. NEVER hand-edit them. See `ibm-d.md`.
76
+
77
+6. **The dashboard consumes `integrations/integrations.js`.**
78
+ The cloud-frontend at
79
+ `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/` runs
80
+ `gen_integrations.py` in its own CI to copy
81
+ `integrations.js` into its source tree. The contract is
82
+ that `.js` file's exact shape:
83
+ `export const categories = [...]; export const integrations
84
+ = [...]`. See `in-app-contract.md`.
85
+
86
+## Table of contents
87
+
88
+| Guide | Purpose |
89
+|---|---|
90
+| `pipeline.md` | The 4-stage pipeline graph, every script, every artifact, the CI workflows. |
91
+| `schema-reference.md` | Exhaustive per-field reference for all 12 JSON Schemas under `integrations/schemas/`. |
92
+| `per-type-matrix.md` | One-row-per-integration-type quick lookup: source paths, validator, render keys, output location. |
93
+| `artifacts-and-banners.md` | Every committed and gitignored artifact; banner conventions; symlink rules. |
94
+| `ibm-d.md` | The `contexts.yaml` -> `metadata.yaml` chain for ibm.d modules. |
95
+| `consistency.md` | The 5-file consistency rule and what tooling enforces (mostly nothing). |
96
+| `in-app-contract.md` | How the cloud-frontend dashboard consumes `integrations.js`. |
97
+| `gotchas.md` | Every surprise, dead-code reference, hardcoded marketing anchor, custom Jinja delimiter. |
98
+| `recipes/INDEX.md` | Step-by-step recipes for adding/updating each integration type. |
99
+| `how-tos/INDEX.md` | Live catalog: every analysis question gets a how-to entry. |
100
+
101
+## Live how-to rule (mandatory)
102
+
103
+If an assistant is asked a concrete question about the
104
+integrations pipeline that is NOT already documented under
105
+`how-tos/` or one of the per-domain guides above, AND answering
106
+it requires non-trivial analysis (reading multiple scripts,
107
+running the pipeline, cross-referencing schemas), the assistant
108
+MUST author a new how-to under `how-tos/<slug>.md` and add a
109
+one-line entry to `how-tos/INDEX.md` BEFORE completing the task.
110
+This rule is durable. Skipping it means the next assistant
111
+repeats the analysis from scratch.
112
+
113
+## Path discipline
114
+
115
+This skill follows
116
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md`:
117
+
118
+- Files in this repo: repo-relative
119
+ (`integrations/gen_integrations.py`,
120
+ `<repo>/integrations/...`, `src/...`).
121
+- Files in sibling Netdata-org repos:
122
+ `${NETDATA_REPOS_DIR}/<repo-name>/...` (env-key from `.env`).
123
+- No literal home-directory or workstation-root paths anywhere (use the env-keyed placeholder above instead).
124
+
125
+## Sources of truth referenced by this skill
126
+
127
+- `<repo>/integrations/` -- generators, schemas, templates,
128
+ shared metadata files (`categories.yaml`, `deploy.yaml`).
129
+- `<repo>/integrations/schemas/*.json` -- all 12 schemas.
130
+- `<repo>/integrations/templates/` -- all Jinja templates.
131
+- `<repo>/.github/workflows/generate-integrations.yml` and
132
+ `<repo>/.github/workflows/check-markdown.yml` -- the CI.
133
+- `<repo>/.github/data/distros.yml` -- platform table fed into
134
+ deploy rendering.
135
+- `<repo>/src/go/plugin/ibm.d/` -- the secondary generator
136
+ chain (`docgen/main.go`, `metricgen/main.go`).
137
+- `<repo>/AGENTS.md` -- the "Collector Consistency
138
+ Requirements" policy text.
139
+
140
+## Related skills
141
+
142
+- `project-writing-collectors` -- the broader collector
143
+ authoring context (NIDL contexts, dashboard shaping, plugin
144
+ landscape). Read FIRST when authoring a brand-new collector;
145
+ read THIS skill when working with the integration metadata
146
+ side.
147
+- `learn-site-structure` -- how the per-integration `.md` files
148
+ ultimately get published on `learn.netdata.cloud`. The
149
+ Learn-side mapping is driven by `<repo>/docs/.map/map.yaml`;
150
+ for integration pages, the relevant `<!--startmeta` block
151
+ inside each generated `.md` is what Learn's ingest reads.
.agents/skills/integrations-lifecycle/artifacts-and-banners.md
new
+178
@@ -0,0 +1,178 @@
1
+# Artifacts and banners
2
+
3
+Every file the integrations pipeline reads or writes, with
4
+banner conventions and edit rules.
5
+
6
+## Tracked vs runtime artifacts
7
+
8
+| Path | Producer | Tracked in git? | Auto-regenerated by |
9
+|---|---|---|---|
10
+| `integrations/integrations.js` | `gen_integrations.py:1311-1325` | NO -- gitignored (`.gitignore:159`) | Every CI run; `rm`-ed by the workflow before opening the PR |
11
+| `integrations/integrations.json` | `gen_integrations.py:1311-1325` (clean variant) | NO -- gitignored (`.gitignore:160`) | as above |
12
+| `<plugin-dir>/integrations/<slug>.md` (per-integration) | `gen_docs_integrations.py` | **YES (committed)** | `generate-integrations.yml` auto-PR; verified in `check-markdown.yml` |
13
+| `<plugin-dir>/README.md` (symlink) | `gen_docs_integrations.py:make_symlinks` (`:527-544`) | **YES** (the symlink itself) | as above |
14
+| `src/health/notifications/<dir>/README.md` (DIRECT, not a symlink) | `gen_docs_integrations.py:agent_notification branch` (`:488-496`) | **YES** | as above |
15
+| `src/collectors/COLLECTORS.md` | `gen_doc_collector_page.py` | **YES** | as above |
16
+| `src/collectors/SECRETS.md` | `gen_doc_secrets_page.py` | **YES** | as above |
17
+| `src/collectors/SERVICE-DISCOVERY.md` | `gen_doc_service_discovery_page.py` | **YES** | **NOT** in `generate-integrations.yml` -- manual or cmake `render-docs` only -- see `gotchas.md` |
18
+| `src/go/plugin/ibm.d/modules/<m>/metadata.yaml` | ibm.d `docgen` | **YES** | manual `go generate ./...` -- never CI |
19
+| `src/go/plugin/ibm.d/modules/<m>/README.md` | ibm.d `docgen` | **YES** | as above |
20
+| `src/go/plugin/ibm.d/modules/<m>/config_schema.json` | ibm.d `docgen` | **YES** | as above |
21
+| `src/go/plugin/ibm.d/modules/<m>/contexts/zz_generated_contexts.go` | ibm.d `metricgen` | **YES** | as above |
22
+| Hand-written `metadata.yaml` (non-ibm.d), `config_schema.json`, stock `.conf`, `health.d/<...>.conf`, hand-written `README.md` | collector author | **YES** | none -- author edits + commits manually |
23
+
24
+## Banner conventions per file kind
25
+
26
+### `integrations/integrations.js`
27
+
28
+Top of file (`integrations/templates/integrations.js:1-2`):
29
+
30
+```js
31
+// DO NOT EDIT THIS FILE DIRECTLY
32
+// It gets generated by integrations/gen_integrations.py in the Netdata repo
33
+```
34
+
35
+Body shape:
36
+
37
+```js
38
+export const categories = [/* JSON array of category trees */];
39
+export const integrations = [/* JSON array of integration objects */];
40
+```
41
+
42
+The body uses the rendered-with-folding/`{% details %}` markers
43
+(the `clean=False` variant). Consumed by the cloud-frontend
44
+dashboard which has its own renderer that interprets these
45
+markers.
46
+
47
+### `integrations/integrations.json`
48
+
49
+Pure JSON, no banner. Top-level shape:
50
+
51
+```json
52
+{ "categories": [...], "integrations": [...] }
53
+```
54
+
55
+Uses the `clean=True` variant (markers stripped). Consumed by
56
+`scripts/checkIntegrations.js` in the dashboard repo (validation
57
+fetch from `https://raw.githubusercontent.com/netdata/netdata/master/integrations/integrations.json`)
58
+and by anything that needs GitHub-renderable markdown.
59
+
60
+### Per-integration `.md` (collector, exporter, secretstore, etc.)
61
+
62
+Every per-integration `.md` opens with a `<!--startmeta`
63
+HTML-comment block. This block is consumed by the Learn ingest
64
+pipeline (`<repo>/docs/.map/map.yaml` row matching is by
65
+`custom_edit_url`).
66
+
67
+Schema:
68
+
69
+```markdown
70
+<!--startmeta
71
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/<plugin-dir>/integrations/<slug>.md"
72
+meta_yaml: "https://github.com/netdata/netdata/edit/master/<plugin-dir>/metadata.yaml"
73
+sidebar_label: "<display name>"
74
+learn_status: "Published"
75
+learn_rel_path: "<Learn category path>"
76
+keywords: ['k1', 'k2', ...] # only when metadata has keywords
77
+most_popular: true # only when metadata sets it
78
+message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE"
79
+endmeta-->
80
+```
81
+
82
+The exact wording of `message:` varies by type. From
83
+`gen_docs_integrations.py`:
84
+
85
+| Integration type | `message:` text |
86
+|---|---|
87
+| `collector` | `DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE` |
88
+| `exporter` | `... GENERATED BY THE EXPORTER'S metadata.yaml FILE` |
89
+| `agent_notification`, `cloud_notification` | `... GENERATED BY THE NOTIFICATION'S metadata.yaml FILE` |
90
+| `logs` | `... GENERATED BY THE LOGS' metadata.yaml FILE` |
91
+| `authentication` | `... GENERATED BY THE AUTHENTICATION'S metadata.yaml FILE` |
92
+| `secretstore` | `... GENERATED BY THE SECRETSTORE'S metadata.yaml FILE` |
93
+| `service_discovery` | `... GENERATED BY THE SERVICE DISCOVERY DISCOVERER'S metadata.yaml FILE` |
94
+
95
+After the banner, the script injects a community-or-Netdata
96
+badge image before the first `##` (`:422-437`). Badge color
97
+depends on `meta.community` boolean.
98
+
99
+### `learn_rel_path` mapping
100
+
101
+The first category in `meta.monitored_instance.categories` is
102
+remapped to a Learn navigation path (`gen_docs_integrations.py`
103
+per-type branches):
104
+
105
+| Integration type | Mapping |
106
+|---|---|
107
+| `collector` | `data-collection.<x>` -> `Collecting Metrics/Collectors/<x>` |
108
+| `exporter` | hardcoded `Exporting Metrics/Connectors` |
109
+| `agent_notification` | `<x>` with `notifications` -> `Alerts & Notifications/Notifications` |
110
+| `cloud_notification` | same as above |
111
+| `logs` | `<x>` with `logs` -> `Logs` |
112
+| `authentication` | `<x>` with `authentication` -> `Netdata Cloud/Authentication & Authorization/Cloud Authentication & Authorization Integrations` |
113
+| `secretstore` | hardcoded `Collecting Metrics/Secrets Management/Secret Stores` |
114
+| `service_discovery` | hardcoded `Collecting Metrics/Service Discovery` |
115
+
116
+### `<plugin-dir>/README.md` (symlink case)
117
+
118
+When the symlink rule fires (single-integration directory), the
119
+README is a symlink to `integrations/<slug>.md`. The symlink
120
+target's banner is the per-integration banner above, with two
121
+adjustments:
122
+- `custom_edit_url` points at the README path, not the
123
+ integration file path -- so editing through Learn opens the
124
+ README.
125
+- The body has `{element}/{symlinks[element]}` replaced with
126
+ `{element}/README.md` (`:542-544`) so internal anchor refs
127
+ don't break.
128
+
129
+### `src/health/notifications/<dir>/README.md` (direct case)
130
+
131
+Same banner as above. Written DIRECTLY, not as a symlink. Keep
132
+in mind for the five-file consistency rule -- the README.md is
133
+the generated artifact.
134
+
135
+### `src/collectors/COLLECTORS.md`, `SECRETS.md`, `SERVICE-DISCOVERY.md`
136
+
137
+**No banner.** They open with their marketing headers
138
+(`# Monitor anything with Netdata`, `# Secrets Management`,
139
+`# Service Discovery`). This is a known oversight -- there is
140
+no DO-NOT-EDIT marker on these files even though they are
141
+generated. Editing them directly will be silently overwritten
142
+on the next CI run (for COLLECTORS.md and SECRETS.md). For
143
+SERVICE-DISCOVERY.md the absence of CI wiring means manual
144
+edits stick until someone runs the script.
145
+
146
+### ibm.d-generated files
147
+
148
+| File | Banner |
149
+|---|---|
150
+| `metadata.yaml` | First line: `# Generated metadata.yaml for <module> module` (template at `src/go/plugin/ibm.d/docgen/main.go:562`) |
151
+| `zz_generated_contexts.go` | Filename prefix `zz_generated_`. Package doc at `<module>/contexts/doc.go:2-3` declares `DO NOT EDIT zz_generated_contexts.go - it is auto-generated from contexts.yaml`. |
152
+| `README.md` | Banner depends on `docgen`'s readme template. Includes module info, metric tables, config tables. |
153
+| `config_schema.json` | Pure JSON, no banner. |
154
+
155
+See `ibm-d.md` for the full chain.
156
+
157
+## Maintainer rules
158
+
159
+1. **Per-integration `.md` files: NEVER edit by hand.** Edit
160
+ the source `metadata.yaml`, regenerate, commit.
161
+2. **Symlinked `README.md` files: NEVER edit by hand.** Same
162
+ reason -- they point at the generated `.md`.
163
+3. **`src/health/notifications/<dir>/README.md`: NEVER edit by
164
+ hand.** Direct generated file, looks identical to a normal
165
+ README -- the `<!--startmeta` banner is the giveaway.
166
+4. **`src/collectors/COLLECTORS.md` / `SECRETS.md`: NEVER edit
167
+ the dynamic tables.** Edit the source `metadata.yaml`. The
168
+ static prose ("# Monitor anything with Netdata", the
169
+ secret-store onboarding text) is baked into the generator
170
+ scripts -- to change static prose, edit
171
+ `gen_doc_collector_page.py` / `gen_doc_secrets_page.py` and
172
+ commit.
173
+5. **`src/collectors/SERVICE-DISCOVERY.md`: same rule.** Caveat:
174
+ the script is not in CI, so manual regen is required after
175
+ editing source metadata until that gap is fixed.
176
+6. **ibm.d generated files: NEVER edit.** Edit `contexts.yaml`,
177
+ `config.go`, or `module.yaml` instead and re-run
178
+ `go generate`.
.agents/skills/integrations-lifecycle/consistency.md
new
+140
@@ -0,0 +1,140 @@
1
+# Five-file consistency rule
2
+
3
+`<repo>/AGENTS.md` declares ("Collector Consistency
4
+Requirements") that any change touching a collector MUST land
5
+in one PR with matching changes to all five of:
6
+
7
+1. **The code** -- the collector implementation files.
8
+2. **`metadata.yaml`** -- the integration page driver.
9
+3. **`config_schema.json`** -- the dashboard's DYNCFG editor.
10
+4. **The stock `.conf`** -- what `/etc/netdata/<plugin>/...`
11
+ ships.
12
+5. **`health.d/*.conf`** -- alert definitions for the
13
+ collector's metrics.
14
+6. **`README.md`** -- comprehensive end-user documentation
15
+ (often a symlink into the generated
16
+ `integrations/<slug>.md`; see `artifacts-and-banners.md`).
17
+
18
+(`AGENTS.md` lists 6 items; the "5-file" shorthand merges
19
+"the code" into the implicit driver.)
20
+
21
+The rule covers obvious cases (units change in code -> update
22
+metadata.yaml; new config option -> update schema, stock conf,
23
+and docs; new metric -> update metadata.yaml and README.md)
24
+and subtle ones (renaming a metric label affects the alert
25
+definition that refers to it; changing a default affects the
26
+stock conf example and the documented default value).
27
+
28
+## What enforces this rule today
29
+
30
+**Nothing automated, in most cases.** Specifically:
31
+
32
+- **`gen_integrations.py`** validates each `metadata.yaml`
33
+ against its JSON Schema only. It does NOT cross-check
34
+ against `config_schema.json`, the stock `.conf`, or
35
+ `health.d/*.conf`.
36
+- **`integrations/check_collector_metadata.py`** is broken
37
+ (see `gotchas.md` and `validators.md` for details). Its
38
+ imports refer to symbols that no longer exist in
39
+ `gen_integrations.py`. ImportError on first run. NOT
40
+ invoked from any workflow.
41
+- **No CI workflow** runs a "verify metric names in
42
+ metadata.yaml exist in the collector code" check.
43
+- **No CI workflow** runs a "verify `health.d/*.conf` alert
44
+ metric names exist in the collector" check.
45
+- **`check-markdown.yml`** only validates that generated
46
+ markdown links resolve through Learn ingest -- not that
47
+ metadata.yaml is in sync with the other four files.
48
+
49
+The one exception is **ibm.d modules**: their `metadata.yaml`,
50
+`README.md`, and `config_schema.json` are GENERATED by the
51
+ibm.d `docgen` tool from `contexts.yaml` + `config.go` +
52
+`module.yaml`, so those three files are consistent by
53
+construction. ibm.d does NOT generate the stock `.conf` or
54
+`health.d/<...>.conf`; those still need manual sync. See
55
+`ibm-d.md`.
56
+
57
+## What reviewers should check
58
+
59
+When reviewing a PR that touches a collector, verify:
60
+
61
+1. **Code changes have matching `metadata.yaml` changes.** If
62
+ the diff adds a chart, dimension, label, or unit change in
63
+ the code, the corresponding entry must appear in
64
+ `metadata.yaml`. If a metric is renamed, both files must
65
+ change.
66
+
67
+2. **Config changes propagate to all four config-related
68
+ files.**
69
+ - The Go struct field (in `config.go`).
70
+ - `config_schema.json` -- the field appears with the
71
+ correct type, default, validation.
72
+ - The stock `.conf` -- a representative example shows the
73
+ option.
74
+ - `metadata.yaml` -- the option appears under
75
+ `setup.configuration.options.list`.
76
+
77
+3. **Alert changes have matching `metadata.yaml.alerts`
78
+ entries.** If `health.d/<plugin>.conf` adds, removes, or
79
+ renames an alert, `metadata.yaml.modules.<m>.alerts[]` must
80
+ reflect the change.
81
+
82
+4. **README.md handling.** If the plugin directory has a
83
+ single integration, the README is a symlink to the
84
+ generated `integrations/<slug>.md` -- the symlink target
85
+ already updates when `metadata.yaml` updates. If the
86
+ plugin directory has multiple integrations, the README is
87
+ hand-written and must be updated by the author.
88
+ `agent_notification` is a special case: the README itself
89
+ is the generated artifact (no `integrations/` subdir).
90
+
91
+5. **`integrations/<slug>.md` regenerated.** The author
92
+ should have run the pipeline locally and committed the
93
+ updated `.md` file. `check-markdown.yml` will re-run the
94
+ pipeline in CI; if the author's commit and CI's regen
95
+ diverge, the PR fails.
96
+
97
+6. **Umbrella pages.** If the diff added or removed a
98
+ collector, `src/collectors/COLLECTORS.md` should reflect
99
+ it. Same for `SECRETS.md` (secretstore changes) and
100
+ `SERVICE-DISCOVERY.md` (service-discovery changes -- but
101
+ note this one is NOT in CI; manual regen required).
102
+
103
+## Why the policy is unenforced
104
+
105
+Investigation of the repo found that
106
+`integrations/check_collector_metadata.py` was apparently the
107
+first stab at automation; it appears to have bit-rotted
108
+without a CI hook to catch the rot. A proper enforcement
109
+pipeline would:
110
+
111
+- repair / rewrite `check_collector_metadata.py` (or replace
112
+ with a unit test under `tests/`);
113
+- wire it into `generate-integrations.yml` as a pre-flight
114
+ validator;
115
+- add a metric-name cross-reference check between
116
+ `metadata.yaml.alerts[].metric` and the alert
117
+ configurations under `health.d/`;
118
+- add a config-name cross-reference check between
119
+ `metadata.yaml.setup.configuration.options.list[].name`
120
+ and `config_schema.json` properties.
121
+
122
+Tracked as a followup SOW after `integrations-lifecycle` ships.
123
+For now, the consistency rule is a review-time policy.
124
+
125
+## Anti-patterns to flag in review
126
+
127
+- "I only changed the code; the docs can be a follow-up PR."
128
+ -> No. Five files, one PR.
129
+- "The integration page on Learn doesn't show my new option."
130
+ -> Author forgot to update `metadata.yaml` AND regenerate
131
+ `integrations/<slug>.md`.
132
+- "I edited `integrations/<slug>.md` directly to fix a
133
+ description." -> No. That file is generated. Edit
134
+ `metadata.yaml` and regenerate.
135
+- "I edited `metadata.yaml` for an ibm.d module." -> No. Edit
136
+ `contexts.yaml`, `config.go`, or `module.yaml` and run
137
+ `go generate`.
138
+- "I changed a default in the stock `.conf` only." -> Update
139
+ `config_schema.json` `default`, `metadata.yaml.setup.configuration.options.list[].default_value`,
140
+ and the README in lockstep.
.agents/skills/integrations-lifecycle/gotchas.md
new
+367
@@ -0,0 +1,367 @@
1
+# Gotchas
2
+
3
+Every surprise, dead-code reference, hardcoded marketing
4
+anchor, custom Jinja delimiter, undocumented behavior, and
5
+edge case the integrations pipeline carries today. Read this
6
+before assuming the code does the obvious thing.
7
+
8
+## Dead / broken code in the pipeline
9
+
10
+### `integrations/check_collector_metadata.py` is broken
11
+
12
+- File path: `integrations/check_collector_metadata.py`.
13
+- Line 8 imports `SINGLE_PATTERN`, `MULTI_PATTERN`,
14
+ `SINGLE_VALIDATOR`, `MULTI_VALIDATOR` from `gen_integrations`.
15
+- **None of those names exist in `gen_integrations.py`
16
+ today.** The current names are `METADATA_PATTERN` (single
17
+ pattern) and `COLLECTOR_VALIDATOR` (single validator).
18
+- Therefore: any attempt to run
19
+ `python3 integrations/check_collector_metadata.py <path>`
20
+ exits with `ImportError`.
21
+- It is referenced nowhere actionable: no workflow under
22
+ `.github/workflows/`, no script under `packaging/cmake/`.
23
+ Only mentioned in `integrations/README.md` style prose.
24
+- Its body has a second bug: `'{ check_path } is a valid
25
+ collector metadata file.'` (line 84) -- the f-string `f`
26
+ prefix is missing, so the literal `{ check_path }` would be
27
+ printed even if the imports worked.
28
+- **Treat this file as dead code.** Do NOT rely on it.
29
+ Followup: SOW will track repair-or-delete.
30
+
31
+### `gen_doc_service_discovery_page.py` is NOT in CI
32
+
33
+- File exists, runs correctly when invoked manually.
34
+- Workflow `.github/workflows/generate-integrations.yml`
35
+ runs only `gen_integrations.py`, `gen_docs_integrations.py`,
36
+ `gen_doc_collector_page.py`, `gen_doc_secrets_page.py`
37
+ (`generate-integrations.yml:48-63`).
38
+- Workflow `.github/workflows/check-markdown.yml` has the
39
+ same gap.
40
+- Consequence: `src/collectors/SERVICE-DISCOVERY.md` drifts
41
+ from source `metadata.yaml` until a developer manually
42
+ runs `python3 integrations/gen_doc_service_discovery_page.py`.
43
+- Followup: SOW will track adding the script to both
44
+ workflows.
45
+
46
+### `integrations/schemas/distros.json` is unused
47
+
48
+- Schema declared and well-formed.
49
+- `gen_integrations.py:1330` calls `load_yaml(DISTROS_FILE)`
50
+ WITHOUT validation.
51
+- `DEPLOY_VALIDATOR` is for `deploy.yaml` only, NOT for
52
+ `distros.yml`.
53
+- Garbage in `.github/data/distros.yml` produces broken
54
+ `platform_info` tables silently.
55
+- Followup: SOW will track wiring `distros.json` into
56
+ `gen_integrations.py:1330`.
57
+
58
+## Custom Jinja delimiters
59
+
60
+`gen_integrations.py:233-238` configures Jinja with custom
61
+delimiters:
62
+
63
+| Default | This pipeline |
64
+|---|---|
65
+| `{{ ... }}` | `[[ ... ]]` |
66
+| `{% ... %}` | `[% ... %]` |
67
+| `{# ... #}` | `[# ... #]` |
68
+
69
+Why: so that templates can pass `{% details %}`,
70
+`{% relatedResource %}`, `{% if $showClaimingOptions %}`,
71
+`{{ ... }}` markers through to the rendered output verbatim
72
+(those markers are the cloud-frontend's renderer's syntax,
73
+not Jinja's).
74
+
75
+Documented in `integrations/templates/README.md:12-15` (which
76
+itself is partly stale -- see below).
77
+
78
+## Two-pass templating with `meta.variables`
79
+
80
+`gen_integrations.py:930-934`. When a metadata entry declares
81
+`meta.monitored_instance.variables` (collectors) OR
82
+`meta.variables` (other types), the FIRST pass renders the
83
+section; if the rendered output still contains
84
+`[[ variables.foo ]]` markers, a SECOND Jinja pass is run
85
+with `variables=...` in context.
86
+
87
+**Divergence**: collectors look up
88
+`monitored_instance.variables`; exporters and notifications
89
+look up `meta.variables`. Same goal, different lookup path.
90
+A wart.
91
+
92
+## `{% relatedResource %}` two-pass resolution
93
+
94
+Pass 1: templates emit literal
95
+`{% relatedResource id="..." %}name{% /relatedResource %}`
96
+markers. See `integrations/templates/overview/collector.md:42`
97
+and `integrations/templates/related_resources.md:5`.
98
+
99
+Pass 2: `gen_docs_integrations.py:resolve_related_links`
100
+(`:56-78`) runs AFTER all per-integration `.md` files are
101
+written. It replaces the markers with `[name](/path)` markdown
102
+links using a global `id_to_path` map built from the
103
+just-written files.
104
+
105
+**Silent fallback**: if the marker's `id` is not found in the
106
+map, the marker is replaced with bare `name` text (no link).
107
+No warning.
108
+
109
+## `clean=False` vs `clean=True` divergence
110
+
111
+Every render keys section is rendered TWICE
112
+(`gen_integrations.py:805-947` for collectors, similar for
113
+others):
114
+
115
+- `clean=False` -- preserves `{% details %}` markers; goes
116
+ into `integrations.js` for the cloud-frontend dashboard.
117
+- `clean=True` -- strips folding/details markers; goes into
118
+ `integrations.json` AND into the per-integration `.md`
119
+ files via `gen_docs_integrations.py:50-52`.
120
+
121
+Consumers:
122
+
123
+- Cloud-frontend reads `.js` -> rich variant with markers.
124
+- Per-integration `.md` files (committed, viewed on Learn /
125
+ GitHub) -- clean variant.
126
+- `gen_doc_collector_page.py`, `gen_doc_secrets_page.py`,
127
+ `gen_doc_service_discovery_page.py`,
128
+ `gen_docs_integrations.py` all parse `integrations.js`,
129
+ NOT `integrations.json` -- so they see the rich variant
130
+ but emit the clean variant downstream.
131
+
132
+## Slug rules diverge by integration type
133
+
134
+- Most types: slug = `clean_string(meta.name)` ->
135
+ `<plugin-dir>/integrations/<slug>.md`.
136
+- **Secretstore: slug = `clean_string(meta.kind)`**
137
+ (`gen_docs_integrations.py:640`). The kind matches the
138
+ runtime config filename `/etc/netdata/go.d/ss/<kind>.conf`.
139
+- **Service-discovery: slug = `clean_string(meta.kind)`**
140
+ (`gen_docs_integrations.py:655`). Same reason -- the kind
141
+ is the discoverer registry name.
142
+- **Collector custom_edit_url under `/integrations/functions/`**:
143
+ filename uses the function slug from the URL stem (with `-`
144
+ -> ` `) instead of `monitored_instance.name`. Avoids
145
+ collisions when many integrations share a "Top Queries"
146
+ label.
147
+
148
+`clean_string` rules (`gen_docs_integrations.py:118-126`):
149
+1. lowercase;
150
+2. spaces -> `_`;
151
+3. `/` -> `_`;
152
+4. drop `(`, `)`, `,`, `'`, backtick, `:`.
153
+
154
+So `Apache Kafka` -> `apache_kafka`. `Citrix/NetScaler` ->
155
+`citrix_netscaler`. Note that these characters are stripped
156
+silently; if two source names collide post-cleanup, one
157
+overwrites the other (not warned about).
158
+
159
+## `make_id` allows uppercase
160
+
161
+`gen_integrations.py:768`:
162
+`monitored_instance.name.replace(' ', '_')`.
163
+
164
+So `Apache Kafka` becomes id segment `Apache_Kafka` and full
165
+id `go.d.plugin-kafka-Apache_Kafka`. Mixed case preserved;
166
+only spaces translated. Not URL-safe in the strict sense
167
+(uppercase).
168
+
169
+## Schemas are NOT strict
170
+
171
+`additionalProperties: false` is NOT set on most schemas. Two
172
+known undocumented fields that pass through silently:
173
+
174
+- `alternative_monitored_instances` -- seen in
175
+ `src/go/plugin/go.d/collector/postgres/metadata.yaml:21`.
176
+- `most_popular` -- seen in ibm.d-generated `metadata.yaml`
177
+ and elsewhere.
178
+
179
+Neither is in `collector.json`. They appear in
180
+`integrations.js` but no template renders them. They are
181
+harmless but misleading -- maintainers may assume they do
182
+something.
183
+
184
+## `global` scope renamed to `<instance> instance`
185
+
186
+`gen_integrations.py:914-916`. Many `metadata.yaml` files
187
+declare `metrics.scopes:` with `name: global`. The renderer
188
+rewrites this in-place to `<monitored_instance.name> instance`
189
+before templating. The original file stays as `global`.
190
+
191
+## Default-categories fallback
192
+
193
+`gen_integrations.py:906-908`. If a collector's declared
194
+categories are all bogus (none match `categories.yaml`), the
195
+renderer falls back to ALL categories with
196
+`collector_default: true` from `categories.yaml`. Currently
197
+only `data-collection.applications` (`categories.yaml:50-52`)
198
+is so flagged. So a typo in a collector's categories silently
199
+parks the integration under "Applications".
200
+
201
+## `agent_notification` writes README directly
202
+
203
+For every type EXCEPT `agent_notification`, the per-integration
204
+`.md` lives under `<dir>/integrations/<slug>.md` and a
205
+`README.md` symlink is made when there is exactly one
206
+integration.
207
+
208
+For `agent_notification`, the script writes the per-integration
209
+file DIRECTLY to `<dir>/README.md` (`gen_docs_integrations.py:488-496`).
210
+No `integrations/` subdirectory, no symlink. So
211
+`src/health/notifications/email/README.md` is the generated
212
+artifact, NOT a hand-written README. The
213
+`<!--startmeta` banner is the giveaway.
214
+
215
+## Symlink only fires when exactly one integration
216
+
217
+`gen_docs_integrations.py:466`:
218
+`len(list(integrations_dir.iterdir())) == 1`. If a directory
219
+has multiple integrations (rare), no top-level `README.md`
220
+symlink is created -- the parent's existing README is left
221
+alone (or absent).
222
+
223
+## Hardcoded marketing anchors that don't exist
224
+
225
+`gen_doc_collector_page.py:_render_tech_navigation` (`:424-493`)
226
+writes navigation links like:
227
+`#cloud-provider-managed`, `#kubernetes`, `#search-engines`,
228
+`#freebsd`, `#message-brokers`, `#linux-systems`,
229
+`#windows-systems`, `#macos-systems`.
230
+
231
+None of those category IDs exist in `categories.yaml`. The
232
+real categories are `databases`, `web-servers-and-proxies`,
233
+`containers-and-vms`, `operating-systems`, `networking`,
234
+`cloud-and-devops`, `hardware-and-sensors`, `applications`,
235
+`storage`, `synthetic-testing`. Several of the hardcoded
236
+anchors point at non-existent sections in `COLLECTORS.md`.
237
+
238
+The header text "850+ integrations" is also a baked literal.
239
+
240
+## Static prose baked into Python scripts
241
+
242
+`gen_doc_secrets_page.py:20-203`: the `SECRETS_PAGE` dict
243
+contains the bulk of `SECRETS.md`. Only the "Supported
244
+Secretstore Backends" table is dynamic.
245
+
246
+`gen_doc_service_discovery_page.py:21-257`: the `SD_PAGE` dict
247
+contains the bulk of `SERVICE-DISCOVERY.md`. Only the
248
+discoverer table is dynamic.
249
+
250
+`gen_doc_collector_page.py`: the marketing header in
251
+`_render_tech_navigation` is hardcoded.
252
+
253
+To change the static prose on any of these umbrella pages,
254
+edit the Python script and commit.
255
+
256
+## Scoped regen via `-c plugin/module`
257
+
258
+`gen_docs_integrations.py:578-583`. Allows scoped cleanup +
259
+regen:
260
+
261
+```bash
262
+python3 integrations/gen_docs_integrations.py -c go.d/snmp
263
+```
264
+
265
+NOT used by CI (CI always does full regen). Useful for fast
266
+local iteration.
267
+
268
+## `templates/README.md` is partly stale
269
+
270
+`integrations/templates/README.md:30-31` mentions
271
+`setup-generic.md`, `setup-logs.md`, `setup-secretstore.md` as
272
+the per-type setup templates, but `setup-service_discovery.md`
273
+was added later and is not mentioned. Not pipeline-impacting,
274
+just out-of-date docs.
275
+
276
+## Umbrella pages have NO DO-NOT-EDIT banner
277
+
278
+`src/collectors/COLLECTORS.md`, `src/collectors/SECRETS.md`,
279
+`src/collectors/SERVICE-DISCOVERY.md` all open with marketing
280
+headers (e.g. `# Monitor anything with Netdata`). None has a
281
+`<!--startmeta` block, none has any DO-NOT-EDIT comment.
282
+
283
+A maintainer who edits these files directly will have their
284
+edits silently overwritten on the next CI run (for
285
+COLLECTORS.md and SECRETS.md). For SERVICE-DISCOVERY.md the
286
+absence of CI wiring means manual edits stick until someone
287
+runs the script -- giving a false sense that hand-editing is
288
+acceptable.
289
+
290
+## Edge case in `build_path`
291
+
292
+`gen_docs_integrations.py:81-90` assumes `meta_yaml` URL
293
+starts with `https://github.com/netdata/...`. For forks /
294
+non-`netdata/netdata` sources, it would produce wrong paths.
295
+The pipeline assumes `AGENT_REPO = 'netdata/netdata'`
296
+everywhere (`gen_integrations.py:15`).
297
+
298
+## `convert_local_links` rewrites all `](/...)` links
299
+
300
+`gen_integrations.py` runs `convert_local_links` on the
301
+`integrations.js` output, rewriting any `](/...)` link in
302
+rendered text to absolute
303
+`https://github.com/netdata/netdata/blob/master/...`. This
304
+applies to body links inside per-integration content. So
305
+metadata authors writing `](/src/foo/bar.md)` get a GitHub
306
+link in the dashboard, not a local-relative link.
307
+
308
+## `dependencies` in `collector.json:61-63`
309
+
310
+Draft-7 JSON Schema `dependencies` keyword: when
311
+`monitored_instance_name` is set on a
312
+`related_resources.integrations.list[]` entry,
313
+`module_name` becomes required. Correct semantics, but
314
+non-obvious -- nothing else in the schemas uses
315
+`dependencies`, and no commentary explains it.
316
+
317
+## `fail_on_warnings` makes ALL warnings fatal
318
+
319
+`gen_integrations.py:150-160`. Any single validation warning
320
+-- duplicate id, invalid category, missing related
321
+integration -- causes `fail_on_warnings()` to return 1, which
322
+fails CI. Even cosmetic issues block the regeneration PR.
323
+
324
+Warnings are deduplicated by file path; the failure message
325
+lists each warned file.
326
+
327
+## Cloud-notifications and authentication metadata are single-file arrays
328
+
329
+Most types have one `metadata.yaml` per integration directory.
330
+Two exceptions:
331
+
332
+- `integrations/cloud-notifications/metadata.yaml` -- ONE
333
+ file containing an ARRAY of cloud-notification entries.
334
+- `integrations/cloud-authentication/metadata.yaml` -- ONE
335
+ file containing an ARRAY of authentication-method entries.
336
+
337
+The `_load_*_file` functions handle both shapes via
338
+`if 'id' in data` branches (single entry vs array).
339
+
340
+## ibm.d websphere subdirectories
341
+
342
+`gen_integrations.py:35` adds
343
+`src/go/plugin/ibm.d/modules/websphere` to `COLLECTOR_SOURCES`
344
+separately. That's because `websphere/{jmx,mp,pmi}/` are
345
+sub-modules each with their own `metadata.yaml`,
346
+`module.yaml`, `contexts/`, etc. The default
347
+`src/go/plugin/ibm.d/modules` glob would not catch them at
348
+the right depth.
349
+
350
+## `pip.sh` and the cmake module must stay in sync
351
+
352
+`integrations/pip.sh` is a 2-line script:
353
+`pip install jsonschema referencing jinja2 ruamel.yaml`. The
354
+same four packages are listed at
355
+`packaging/cmake/Modules/NetdataRenderDocs.cmake:21`. Both
356
+must be updated together if the dep set changes (commented
357
+inline in `pip.sh`).
358
+
359
+## Logo contrast analysis makes outbound HTTP calls
360
+
361
+`gen_integrations.py:1647-1681` annotates `<img src="https://(www\.)?netdata\.cloud/img/...">`
362
+tags with `data-integration-logo`, `data-logo-contrast-light`,
363
+`data-logo-contrast-dark`, `data-logo-contrast-confidence`
364
+after fetching each logo and analyzing its luminance. The
365
+result is cached per-URL within a single run. CI runs may
366
+trip over rate limits or transient network errors; per-request
367
+timeout is hardcoded.
.agents/skills/integrations-lifecycle/how-tos/INDEX.md
new
+55
@@ -0,0 +1,55 @@
1
+# How-tos -- INDEX
2
+
3
+Live catalog of analysis-derived how-tos for the
4
+integrations-lifecycle skill.
5
+
6
+**Live-catalog rule** (also stated in `../SKILL.md` and
7
+`../recipes/INDEX.md`): if an assistant is asked a concrete
8
+question about the integrations pipeline that requires
9
+non-trivial analysis (multiple file reads, running the
10
+pipeline, cross-referencing schemas) AND the answer is not
11
+already documented under one of the per-domain guides
12
+(`../pipeline.md`, `../schema-reference.md`, `../per-type-matrix.md`,
13
+`../artifacts-and-banners.md`, `../ibm-d.md`,
14
+`../consistency.md`, `../in-app-contract.md`,
15
+`../gotchas.md`) or the recipes (`../recipes/`), the
16
+assistant MUST author a new `how-tos/<slug>.md` and add a
17
+one-line entry to this INDEX BEFORE completing the task.
18
+
19
+This is a durable rule. Skipping it means the next assistant
20
+repeats the same analysis from scratch -- a framework
21
+violation.
22
+
23
+## Catalog
24
+
25
+(empty -- entries grow as assistants encounter
26
+not-yet-documented questions)
27
+
28
+| Topic | Slug | Notes |
29
+|---|---|---|
30
+| -- | -- | -- |
31
+
32
+## How to add a how-to
33
+
34
+1. Create `how-tos/<slug>.md` with:
35
+ - A one-line summary at the top (the question being
36
+ answered).
37
+ - The answer with file:line citations into this repo.
38
+ - A "How I figured this out" footer naming the files
39
+ read and the commands run, so the next assistant can
40
+ verify or extend.
41
+2. Add a row to the table above with topic, slug, and short
42
+ notes.
43
+3. Commit alongside the work that prompted the analysis.
44
+
45
+## When NOT to add a how-to
46
+
47
+- The question is already covered by an existing per-domain
48
+ guide or recipe -- update that guide instead.
49
+- The answer is a one-liner that doesn't require analysis
50
+ (e.g. "where does X live?" -> just look it up; not how-to
51
+ worthy).
52
+- The answer is highly speculative or version-specific (e.g.
53
+ about a feature that may change in the next release). In
54
+ that case, note it in the conversation but don't bake into
55
+ the durable catalog.
.agents/skills/integrations-lifecycle/ibm-d.md
new
+184
@@ -0,0 +1,184 @@
1
+# ibm.d generation chain
2
+
3
+ibm.d is a Go collector framework whose modules generate their
4
+own `metadata.yaml` (and `README.md`, `config_schema.json`,
5
+`zz_generated_contexts.go`) from a small set of authoritative
6
+inputs. This is fundamentally different from go.d / python.d /
7
+charts.d collectors where `metadata.yaml` is hand-edited.
8
+
9
+**Maintainer rule**: for any ibm.d module, NEVER edit
10
+`metadata.yaml`, `README.md`, or `config_schema.json` directly.
11
+Edit `contexts.yaml`, `config.go`, or `module.yaml`, then run
12
+`go generate ./...`.
13
+
14
+## Layout per module
15
+
16
+```
17
+src/go/plugin/ibm.d/modules/<m>/
18
+├── module.yaml # display name, description, icon, categories, link, keywords
19
+├── config.go # Config struct -- parsed via Go AST
20
+├── contexts/
21
+│ ├── contexts.yaml # metric definitions: classes -> contexts -> dimensions
22
+│ ├── doc.go # //go:generate go run ../../../metricgen/main.go ...
23
+│ └── zz_generated_contexts.go # GENERATED -- DO NOT EDIT
24
+├── generate.go # //go:generate go run ../../docgen ...
25
+├── metadata.yaml # GENERATED -- DO NOT EDIT (consumed by gen_integrations.py)
26
+├── README.md # GENERATED -- DO NOT EDIT
27
+├── config_schema.json # GENERATED -- DO NOT EDIT
28
+└── <module-source>.go ... # the collector implementation (hand-written)
29
+```
30
+
31
+`websphere/` is a special parent: its sub-modules
32
+`websphere/{jmx,mp,pmi}/` each have their own
33
+`metadata.yaml`, `module.yaml`, etc. `gen_integrations.py:35`
34
+adds `src/go/plugin/ibm.d/modules/websphere` separately to
35
+`COLLECTOR_SOURCES` so these one-level-deeper paths get picked
36
+up.
37
+
38
+## The two generators
39
+
40
+### `metricgen` -- contexts.yaml -> zz_generated_contexts.go
41
+
42
+Repo path: `src/go/plugin/ibm.d/metricgen/main.go`.
43
+
44
+Reads a module's `contexts/contexts.yaml`. The file declares
45
+classes -> contexts -> dimensions in a structured form:
46
+
47
+```yaml
48
+classes:
49
+ - name: connection
50
+ contexts:
51
+ - name: connection_count
52
+ title: Connection count
53
+ units: connections
54
+ family: connections
55
+ type: line
56
+ dimensions:
57
+ - name: total
58
+ - name: active
59
+```
60
+
61
+Writes `contexts/zz_generated_contexts.go` -- a Go source file
62
+that registers these contexts with the ibm.d framework so the
63
+collector can emit metrics by name. The generated file is
64
+committed.
65
+
66
+Triggered by:
67
+
68
+```go
69
+//go:generate go run ../../../metricgen/main.go ...
70
+```
71
+
72
+at `src/go/plugin/ibm.d/modules/<m>/contexts/doc.go:5`.
73
+
74
+### `docgen` -- contexts.yaml + config.go + module.yaml -> metadata.yaml + README.md + config_schema.json
75
+
76
+Repo path: `src/go/plugin/ibm.d/docgen/main.go`.
77
+
78
+Inputs (per module):
79
+
80
+- `contexts/contexts.yaml` -- the same metric structure
81
+ metricgen reads. Parsed as `Config` with `Class` entries
82
+ (`docgen/main.go:28-55`).
83
+- `config.go` -- the Go `Config` struct. Parsed via Go AST
84
+ (`docgen/config_parser.go`) to extract `ConfigField` records
85
+ (`docgen/main.go:57-78`).
86
+- `module.yaml` -- module-level metadata: name, display name,
87
+ description, icon, categories, link, keywords.
88
+
89
+Outputs (per module):
90
+
91
+- `metadata.yaml` -- written from `metadataTemplate`
92
+ (`docgen/main.go:562`). The generated file opens with the
93
+ banner: `# Generated metadata.yaml for <module> module`. It
94
+ carries hardcoded scaffolding (`most_popular: false`,
95
+ default `update_every: 1` option, `endpoint: dummy://localhost`,
96
+ and a fixed prerequisite "Enable monitoring interface")
97
+ PLUS the dynamic content extracted from `contexts.yaml` and
98
+ `config.go`. Authors who want richer metadata.yaml content
99
+ must extend the template or `module.yaml`, NOT edit the
100
+ generated file.
101
+- `config_schema.json` -- written from a separate template
102
+ (`docgen/main.go:528`). Used by the dashboard's DYNCFG
103
+ editor.
104
+- `README.md` -- written from a readme template
105
+ (`docgen/main.go:552`). Includes module info, metric tables,
106
+ config tables. Banner depends on the template.
107
+
108
+Triggered by:
109
+
110
+```go
111
+//go:generate go run ../../docgen -module=<m> -contexts=contexts/contexts.yaml -config=config.go -module-info=module.yaml
112
+```
113
+
114
+at `src/go/plugin/ibm.d/modules/<m>/generate.go:3`.
115
+
116
+## End-to-end edit recipe (ibm.d module)
117
+
118
+1. Edit one of:
119
+ - `contexts/contexts.yaml` to add/change/remove a metric
120
+ class, context, or dimension;
121
+ - `config.go` to add/change/remove a config field;
122
+ - `module.yaml` to change the display name, description,
123
+ categories, icon, etc.
124
+2. Run from the repo root:
125
+ ```bash
126
+ go generate ./src/go/plugin/ibm.d/modules/<m>/...
127
+ ```
128
+ This invokes BOTH `metricgen` (on `contexts.yaml`) and
129
+ `docgen` (on the module).
130
+3. Commit ALL generated files together with the source change:
131
+ - `metadata.yaml`
132
+ - `README.md`
133
+ - `config_schema.json`
134
+ - `contexts/zz_generated_contexts.go`
135
+4. Run the integrations regen locally to update the
136
+ per-integration `.md` and the umbrella pages:
137
+ ```bash
138
+ ./integrations/pip.sh
139
+ python3 integrations/gen_integrations.py
140
+ python3 integrations/gen_docs_integrations.py -c ibm.d/<m>
141
+ python3 integrations/gen_doc_collector_page.py
142
+ python3 integrations/gen_doc_secrets_page.py
143
+ ```
144
+5. Commit the regenerated `<plugin-dir>/integrations/<slug>.md`
145
+ and umbrella pages too, in the same PR.
146
+
147
+## Why ibm.d is generated this way
148
+
149
+ibm.d collectors are typically heavy: many metrics, many
150
+config fields, dense documentation. Generating ensures
151
+consistency between:
152
+- the runtime metric registration
153
+ (`zz_generated_contexts.go`),
154
+- the integration metadata (`metadata.yaml`),
155
+- the dashboard schema (`config_schema.json`),
156
+- the user-facing documentation (`README.md`).
157
+
158
+It is the closest thing this repo has to enforcement of the
159
+five-file consistency rule for the integration-page side
160
+(metadata + README + config_schema), but it does NOT cover
161
+the stock `.conf` or `health.d/<...>.conf` -- those still
162
+need manual sync.
163
+
164
+## Risks and gotchas
165
+
166
+- **Hand edits to generated files are silently overwritten on
167
+ next `go generate`.** No warning. The DO-NOT-EDIT banner is
168
+ the only signal.
169
+- **`module.yaml` is the right place for static prose** (e.g.
170
+ description text) that the metadata template inlines. Edits
171
+ to that file survive regeneration; edits to the generated
172
+ `metadata.yaml` do not.
173
+- **The metadata template hardcodes some scaffolding** (e.g.
174
+ `endpoint: dummy://localhost`). Modules that need different
175
+ scaffolding must extend the template at
176
+ `docgen/main.go:562+` -- editing the generated `metadata.yaml`
177
+ is not a fix.
178
+- **`go generate` does not auto-run `gen_integrations.py`**.
179
+ After regenerating ibm.d files, you still need to run the
180
+ integrations pipeline to refresh the per-integration `.md`
181
+ and umbrella pages.
182
+- **`websphere/` sub-modules each have their own generation
183
+ cycle**. Running `go generate ./src/go/plugin/ibm.d/modules/websphere/...`
184
+ hits all three (`jmx`, `mp`, `pmi`).
.agents/skills/integrations-lifecycle/in-app-contract.md
new
+137
@@ -0,0 +1,137 @@
1
+# In-app dashboard contract
2
+
3
+The Netdata cloud-frontend dashboard (the React app that powers
4
+`app.netdata.cloud`) renders the Integrations page from the
5
+`integrations.js` artifact this repo produces. This guide
6
+documents the contract between the two repositories so
7
+maintainers know what is and is NOT in scope when working on
8
+integrations-lifecycle changes.
9
+
10
+The cloud-frontend repo lives at
11
+`${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/` (private,
12
+Netdata-org). React component internals are explicitly OUT OF
13
+SCOPE for this skill; only the artifact contract matters.
14
+
15
+## What gets shipped
16
+
17
+**This repo produces:** `integrations/integrations.js` (and
18
+`integrations/integrations.json`) on every CI run of
19
+`generate-integrations.yml` (or local run of
20
+`gen_integrations.py`). Both files are gitignored in this
21
+repo.
22
+
23
+**The cloud-frontend repo consumes:**
24
+`integrations/integrations.js` -- specifically, it copies the
25
+file into `src/domains/integrations/data/integrations.js` in
26
+its own source tree.
27
+
28
+## How the consumption works
29
+
30
+Confirmed at
31
+`${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/.github/workflows/sync-to-s3.yaml:48-66`:
32
+
33
+1. Cloud-frontend's CI checks out `netdata/netdata` (this repo).
34
+2. Runs `python3 integrations/gen_integrations.py` against
35
+ the freshly checked-out master.
36
+3. `cp ./integrations/integrations.js ../src/domains/integrations/data/integrations.js`
37
+ into the dashboard source tree.
38
+4. The dashboard builds with the just-copied artifact baked
39
+ in.
40
+
41
+A second script in the dashboard repo,
42
+`scripts/checkIntegrations.js`, fetches
43
+`https://raw.githubusercontent.com/netdata/netdata/master/integrations/integrations.json`
44
+and validates against the in-tree copy as a drift detector
45
+(`scripts/checkIntegrations.js:13`).
46
+
47
+A third script, `scripts/checkLinks.js`, validates that links
48
+in `src/domains/integrations/data/integrations.js` and
49
+`src/domains/integrations/utils/integrations.js` resolve.
50
+
51
+## The artifact shape
52
+
53
+`integrations/integrations.js`:
54
+
55
+```js
56
+// DO NOT EDIT THIS FILE DIRECTLY
57
+// It gets generated by integrations/gen_integrations.py in the Netdata repo
58
+
59
+export const categories = [
60
+ /* recursive tree of category objects, each:
61
+ { id, name, description, children: [...], collector_default?: boolean }
62
+ */
63
+];
64
+
65
+export const integrations = [
66
+ /* flat array of integration objects, each carries
67
+ integration_type, id, meta, keywords, plus the rendered
68
+ section keys per type (e.g. setup, troubleshooting, alerts,
69
+ metrics, functions, overview, related_resources for
70
+ collectors) and their `clean_*` siblings (e.g. clean_setup,
71
+ clean_alerts) */
72
+];
73
+```
74
+
75
+The dashboard's renderer interprets the `{% details %}` /
76
+`{% /details %}` markers embedded in the rendered text (the
77
+`clean=False` variant is the one written into the `.js` file).
78
+This is why the dashboard receives rich-text content with
79
+collapsible sections, while GitHub-rendered `.md` files use
80
+the `clean=True` variant where these markers are stripped.
81
+
82
+## Special case: `deploy` integrations
83
+
84
+`deploy` entries are NOT written to disk as per-integration
85
+`.md` files (see `per-type-matrix.md`). They live ONLY inside
86
+`integrations.js`, sorted by `quick_start` integer. The
87
+dashboard's "Add Nodes" dialog is the consumer.
88
+
89
+`quick_start: -1` (or any negative) hides the entry from the
90
+dialog. Positive values define the sort order.
91
+
92
+## Drift detection
93
+
94
+The cloud-frontend's `checkIntegrations.js` is the only
95
+end-to-end drift check between the two repos. It runs in the
96
+dashboard's CI and surfaces a failure if the local copy
97
+diverges from `netdata/master`. There is no symmetric check
98
+in this repo (this repo doesn't know what version of
99
+`integrations.js` the dashboard currently has baked in).
100
+
101
+In practice this means:
102
+
103
+- A PR in this repo that changes `metadata.yaml` does NOT
104
+ break the dashboard immediately. The dashboard rebuilds on
105
+ its own schedule (or when its developers re-run their
106
+ sync-to-s3 workflow).
107
+- A breaking change to the `integrations.js` shape (e.g. a
108
+ removed top-level field) WILL break the dashboard on the
109
+ next sync. There is no shape-versioning today; both repos
110
+ assume the JS export shape is stable.
111
+
112
+## What is OUT of scope for integrations-lifecycle
113
+
114
+- **The React renderer** in cloud-frontend that turns
115
+ `integrations.js` into UI. Not documented here.
116
+- **The Integrations page UX, search behavior, filtering,
117
+ navigation.** Cloud-frontend territory.
118
+- **The "Add Nodes" dialog flow** beyond the `quick_start`
119
+ sort contract.
120
+- **Per-platform install commands rendering** (the `deploy`
121
+ entries' `methods[].commands[]`). The dashboard renders
122
+ them; the metadata produces them.
123
+
124
+## Maintainer rules
125
+
126
+1. **Treat `integrations.js` as a published artifact**. Its
127
+ shape (the two named exports, the per-integration object
128
+ keys) is a contract. Avoid breaking changes; coordinate
129
+ with the cloud-frontend team if a key must be renamed or
130
+ removed.
131
+2. **Custom Jinja markers in metadata** (`{% details %}`,
132
+ `{% relatedResource %}`, `{% if %}`) are part of the
133
+ contract. The dashboard's renderer interprets them. Test
134
+ any new marker against both surfaces before relying on it.
135
+3. **Do not commit `integrations.js` to this repo**. It is
136
+ gitignored on purpose; the dashboard pulls fresh on each
137
+ build.
.agents/skills/integrations-lifecycle/per-type-matrix.md
new
+82
@@ -0,0 +1,82 @@
1
+# Per-type matrix
2
+
3
+One row per integration type. Compact lookup: where the source
4
+YAMLs live, which schema validates them, which sections render,
5
+which template chooses, where the output lands, which surfaces
6
+consume it.
7
+
8
+| `integration_type` | Source YAMLs | Schema (`integrations/schemas/`) | RENDER_KEYS (`gen_integrations.py`) | Overview template (`integrations/templates/overview/`) | Setup template (`integrations/templates/`) | Output `.md` location | Surfaces consuming it |
9
+|---|---|---|---|---|---|---|---|
10
+| `collector` | 9 source roots under `src/collectors`, `src/go/plugin/{go.d,scripts.d,ibm.d}/...`, `src/crates/netdata-otel` (see `pipeline.md` for full list) | `collector.json` | `alerts, metrics, functions, overview, related_resources, setup, troubleshooting` (`:71`) | `collector.md` | `setup-generic.md` (with sample-`<lang>`-config.md per plugin) | `<plugin-dir>/integrations/<slug>.md`, slug = `clean_string(meta.monitored_instance.name)` | learn (per-page), in-app (sidebar/search), `src/collectors/COLLECTORS.md`, `integrations.js` |
11
+| `deploy` | `integrations/deploy.yaml` (`:39-41`) | `deploy.json` | n/a -- only `platform_info.md` template | n/a | n/a | NOT written to disk -- embedded only in `integrations.js` | in-app "Add Nodes" dialog (sorted by `quick_start`) |
12
+| `exporter` | `src/exporting/*/metadata.yaml` (`:43`) | `exporter.json` | `overview, setup, troubleshooting` (`:81`) | `exporter.md` | `setup-generic.md` | `src/exporting/<dir>/integrations/<slug>.md` | learn, in-app |
13
+| `agent_notification` | `src/health/notifications/*/metadata.yaml` (`:47`) | `agent_notification.json` | `overview, setup, troubleshooting` (`:87`) | `notification.md` | `setup-generic.md` (handles short `setup.description` form too) | `src/health/notifications/<dir>/README.md` (DIRECT, NOT a symlink, NOT under `integrations/`) | learn, in-app |
14
+| `cloud_notification` | `integrations/cloud-notifications/metadata.yaml` (`:51`) | `cloud_notification.json` | `setup, troubleshooting` (`:93`) | n/a (image-only header) | `setup-generic.md` (mostly short form) | `integrations/cloud-notifications/integrations/<slug>.md`, slug = `clean_string(meta.name)` | learn, in-app |
15
+| `logs` | `integrations/logs/metadata.yaml` (`:55`) | `logs.json` | `overview, setup` (`:98`) | `logs.md` | `setup-logs.md` | `integrations/logs/integrations/<slug>.md`, slug = `clean_string(meta.name)` | learn, in-app |
16
+| `authentication` | `integrations/cloud-authentication/metadata.yaml` (`:59`) | `authentication.json` | `overview, setup, troubleshooting` (`:103`) | `authentication.md` | `setup-generic.md` | `integrations/cloud-authentication/integrations/<slug>.md`, slug = `clean_string(meta.name)` | learn, in-app |
17
+| `secretstore` | `src/go/plugin/agent/secrets/secretstore/backends/*/metadata.yaml` (`:63`) | `secretstore.json` | `overview, setup, collector_configs, troubleshooting` (`:109`) | `secretstore.md` | `setup-secretstore.md` (renders `collector_configs.md` too) | `<backend-dir>/integrations/<slug>.md`, **slug = `clean_string(meta.kind)`** (NOT `meta.name`) | learn (per-backend page), in-app, `src/collectors/SECRETS.md` (umbrella) |
18
+| `service_discovery` | `src/go/plugin/go.d/discovery/sdext/discoverer/*/metadata.yaml` (`:67`) | `service_discovery.json` | `overview, setup, services, verify, troubleshooting` (`:116`) | `service_discovery.md` | `setup-service_discovery.md` (renders `sd-services.md`, `sd-verify.md` too) | `<discoverer-dir>/integrations/<slug>.md`, **slug = `clean_string(meta.kind)`** (NOT `meta.name`) | learn, in-app, `src/collectors/SERVICE-DISCOVERY.md` (umbrella) |
19
+| `categories` | `integrations/categories.yaml` | `categories.json` | n/a -- embedded as `categories` array in `integrations.js` | n/a | n/a | n/a | in-app navigation, learn navigation |
20
+| `distros` | `.github/data/distros.yml` | `distros.json` (declared but **NOT enforced** -- see `gotchas.md`) | n/a -- consumed only by `render_deploy` | n/a | n/a | n/a | feeds `deploy.platform_info` table |
21
+| `shared` | n/a -- referenced by other schemas via `./shared.json#/$defs/...` | n/a | n/a | n/a | n/a | n/a | building block (instance, full_setup, troubleshooting, _folding) |
22
+
23
+## Slug rules summary
24
+
25
+The slug used in the output filename comes from `clean_string`
26
+(`gen_docs_integrations.py:118-126`):
27
+1. lowercase the source string;
28
+2. replace spaces with `_`;
29
+3. replace `/` with `_`;
30
+4. drop `(`, `)`, `,`, `'`, backtick, `:`.
31
+
32
+Source string per type:
33
+- collector: `meta.monitored_instance.name`
34
+- exporter / agent-notification / cloud-notification /
35
+ authentication / logs: `meta.name`
36
+- **secretstore / service_discovery: `meta.kind`** (deliberately
37
+ -- it must match the runtime config filename
38
+ `/etc/netdata/go.d/ss/<kind>.conf` etc.)
39
+
40
+Special case at `gen_docs_integrations.py`: if a collector's
41
+`custom_edit_url` is under `/integrations/functions/`, the
42
+output filename uses the function slug from the URL stem
43
+(with `-` -> ` `) instead of `monitored_instance.name` -- to
44
+avoid collisions when many integrations share a "Top Queries"
45
+label.
46
+
47
+## Single-integration symlink rule
48
+
49
+`gen_docs_integrations.py:make_symlinks` (`:527-544`):
50
+
51
+- After all per-integration `.md` files are written, the script
52
+ walks each `<plugin-dir>/integrations/` directory.
53
+- If the directory contains EXACTLY one file
54
+ (`len(list(integrations_dir.iterdir())) == 1` at `:466`),
55
+ the script creates `<plugin-dir>/README.md` as a symlink to
56
+ `integrations/<sole-file>.md`.
57
+- Multi-integration directories do NOT get a top-level
58
+ `README.md` symlink; the parent's hand-written README (if
59
+ any) is left alone.
60
+- The internal `{element}/{symlinks[element]}` references in
61
+ the `.md` body are rewritten to `{element}/README.md` so
62
+ anchors don't break (`:542-544`).
63
+
64
+This is the reason a plugin's `README.md` is sometimes a
65
+symlink (e.g. `src/collectors/diskspace.plugin/README.md ->
66
+integrations/disk_space.md`) and sometimes a hand-written
67
+file (e.g. multi-module plugins).
68
+
69
+## `agent_notification` is the odd one out
70
+
71
+For every other type, the per-integration `.md` lives under
72
+`<dir>/integrations/<slug>.md` and a `README.md` symlink is
73
+made when there is only one integration in that directory.
74
+
75
+For `agent_notification`, the script writes the per-integration
76
+file DIRECTLY to `<dir>/README.md` (`:488-496`). No
77
+`integrations/` subdirectory, no symlink. So
78
+`src/health/notifications/email/README.md` IS the generated
79
+artifact, not a symlink. Keep this in mind when checking the
80
+five-file consistency rule (the README.md you would normally
81
+not edit is the same physical file as the generated
82
+integration page).
.agents/skills/integrations-lifecycle/pipeline.md
new
+428
@@ -0,0 +1,428 @@
1
+# Pipeline
2
+
3
+This document maps the integrations pipeline end to end -- every
4
+script, every input, every output, every CI workflow. All path
5
+citations are repo-relative; line citations refer to the file at
6
+HEAD of `master` at the time this skill was last updated.
7
+
8
+## The four-stage pipeline
9
+
10
+```
11
+[ YAML sources ]
12
+ |
13
+ v
14
++------------------------+
15
+| gen_integrations.py | (orchestrator, validator, renderer)
16
++------------------------+
17
+ |
18
+ v reads
19
++------------------------+
20
+| integrations.js (gitignored)
21
+| integrations.json (gitignored)
22
++------------------------+
23
+ |
24
+ v
25
++--------------------------------+
26
+| gen_docs_integrations.py | (per-integration .md files)
27
++--------------------------------+
28
+ |
29
+ v
30
++--------------------------------+
31
+| gen_doc_collector_page.py | (src/collectors/COLLECTORS.md)
32
++--------------------------------+
33
+ |
34
+ v
35
++--------------------------------+
36
+| gen_doc_secrets_page.py | (src/collectors/SECRETS.md)
37
++--------------------------------+
38
+ |
39
+ v
40
++--------------------------------+
41
+| gen_doc_service_discovery_page.py | (src/collectors/SERVICE-DISCOVERY.md)
42
++--------------------------------+ NOT in CI today -- see gotchas.md
43
+```
44
+
45
+All four downstream scripts read the **same** `integrations.js`
46
+(or its data inside; details below). They run sequentially but
47
+do not cross-talk.
48
+
49
+## Stage 1 -- `gen_integrations.py` (orchestrator)
50
+
51
+Repo path: `integrations/gen_integrations.py`.
52
+
53
+### Inputs
54
+
55
+- **Categories**: `integrations/categories.yaml` (validated
56
+ against `integrations/schemas/categories.json` at
57
+ `gen_integrations.py:344-355`).
58
+- **Distros**: `.github/data/distros.yml` (loaded via
59
+ `load_yaml` at `gen_integrations.py:1330` -- WITHOUT
60
+ validation; the `distros.json` schema exists but is not
61
+ consulted -- see `gotchas.md`).
62
+- **Per-integration `metadata.yaml`** files matched by
63
+ `METADATA_PATTERN = '*/metadata.yaml'`
64
+ (`gen_integrations.py:25`) under nine collector source roots:
65
+
66
+| Root | Integration types served |
67
+|---|---|
68
+| `src/collectors` | C plugins (apps, cgroups, diskspace, ebpf, freebsd, idlejitter, macos, proc, slabinfo, statsd, systemd-journal, tc, timex, xenstat, log2journal, charts.d, python.d) |
69
+| `src/collectors/charts.d.plugin` | shell-based charts.d collectors |
70
+| `src/collectors/python.d.plugin` | Python collectors (am2320, etc.) |
71
+| `src/collectors/guides` | tutorial-style content |
72
+| `src/go/plugin/go.d/collector` | the Go collector tree (the bulk) |
73
+| `src/go/plugin/scripts.d/collector` | scripts.d (shell) |
74
+| `src/go/plugin/ibm.d/modules` | ibm.d collectors (db2, mq, etc.) |
75
+| `src/go/plugin/ibm.d/modules/websphere` | websphere/{jmx,mp,pmi}/ subcollectors -- listed separately because they are 1 level deeper |
76
+| `src/crates/netdata-otel` | the OTEL Rust crate's collector metadata |
77
+
78
+- **Exporters**: `src/exporting/*/metadata.yaml`
79
+ (`gen_integrations.py:43`).
80
+- **Agent notifications**:
81
+ `src/health/notifications/*/metadata.yaml` (`:47`).
82
+- **Cloud notifications**:
83
+ `integrations/cloud-notifications/metadata.yaml` (`:51`).
84
+- **Logs**: `integrations/logs/metadata.yaml` (`:55`).
85
+- **Authentication**:
86
+ `integrations/cloud-authentication/metadata.yaml` (`:59`).
87
+- **Secretstore**:
88
+ `src/go/plugin/agent/secrets/secretstore/backends/*/metadata.yaml`
89
+ (`:63`).
90
+- **Service discovery**:
91
+ `src/go/plugin/go.d/discovery/sdext/discoverer/*/metadata.yaml`
92
+ (`:67`).
93
+- **Deploy**: `integrations/deploy.yaml` (`:39-41`).
94
+
95
+- **Schemas**: `integrations/schemas/*.json` -- loaded on demand
96
+ via `Registry(retrieve=retrieve_from_filesystem)`
97
+ (`gen_integrations.py:163-169`). Each integration type has its
98
+ own `Draft7Validator` instance (`:171-219`).
99
+
100
+- **Templates**: `integrations/templates/**` -- Jinja env at
101
+ `gen_integrations.py:230-241`. Custom delimiters: `[[ ]]`
102
+ for variables and `[% %]` for control statements (so that
103
+ the template can pass through embedded `{% ... %}` and
104
+ `{{ ... }}` markers untouched). See `gotchas.md`.
105
+
106
+### Validation behavior
107
+
108
+For each integration type, `gen_integrations.py` runs a
109
+`Draft7Validator.validate(...)` call (e.g. `:350`, `:372`,
110
+`:399`, `:437`, `:485`, `:533`, `:581`, `:629`, `:677`, `:725`).
111
+On any `ValidationError`, the script calls `warn(...)`.
112
+**Warnings are fatal**: `fail_on_warnings()` (`:150-160`)
113
+returns 1, causing the CI workflow to fail and abort doc
114
+regeneration.
115
+
116
+The validator IS strict about declared properties; it is NOT
117
+strict about extra properties (no `additionalProperties: false`
118
+on collector.json). Unknown keys (`alternative_monitored_instances`,
119
+`most_popular`) pass through silently. They appear in
120
+`integrations.js` but no template renders them. See `gotchas.md`.
121
+
122
+### Rendering behavior
123
+
124
+For each integration type, the script:
125
+
126
+1. Loads the YAML(s).
127
+2. Validates each entry against the type's JSON Schema.
128
+3. Calls `make_id` (collectors only -- `:766`,
129
+ `f'{plugin}-{module}-{instance}'`).
130
+4. Computes `edit_link` from `_src_path` (`:777`).
131
+5. Sorts by id/path/index.
132
+6. Calls `dedupe_integrations` (`:789`); duplicate ids yield
133
+ warnings.
134
+7. Renders every section listed in `*_RENDER_KEYS` (`:71-122`)
135
+ through Jinja, storing the result back on the item under
136
+ that key. Sections come from the type's schema (e.g.
137
+ `COLLECTOR_RENDER_KEYS = ['alerts', 'metrics', 'functions',
138
+ 'overview', 'related_resources', 'setup',
139
+ 'troubleshooting']` at `:71`).
140
+8. Each section is rendered TWICE -- with `clean=False` (rich
141
+ variant for the JS / cloud-frontend output) and `clean=True`
142
+ (clean variant for the JSON / GitHub-rendered `.md`
143
+ output). Both variants are kept in parallel `clean_*`
144
+ lists.
145
+9. Strips internal-only keys (`_src_path`, `_repo`, `_index`)
146
+ before serialization.
147
+
148
+### Two-pass templating with `meta.variables`
149
+
150
+When a metadata entry declares
151
+`meta.monitored_instance.variables` (collectors) or
152
+`meta.variables` (other types), the FIRST pass produces
153
+markdown that may still contain `[[ variables.foo ]]` markers.
154
+The renderer detects this with a regex and performs a SECOND
155
+Jinja pass over the rendered string with `variables=...` in
156
+context (`:930-934`). This lets metadata authors inject
157
+runtime-style placeholders into rendered text.
158
+
159
+**Divergence**: collectors look up
160
+`monitored_instance.variables`; exporters and notifications
161
+look up `meta.variables` directly. Same goal, different lookup
162
+path -- a known wart.
163
+
164
+### Outputs
165
+
166
+Two files are written (`gen_integrations.py:1311-1325`):
167
+
168
+- `integrations/integrations.js` -- assembles the
169
+ `integrations/templates/integrations.js` Jinja shell with
170
+ `categories=...` and `integrations=...` JSON, then runs
171
+ `convert_local_links` to rewrite any `](/...)` in the body to
172
+ absolute GitHub URLs at `https://github.com/netdata/netdata/blob/master/...`.
173
+ The first 2 lines are a banner:
174
+ ```
175
+ // DO NOT EDIT THIS FILE DIRECTLY
176
+ // It gets generated by integrations/gen_integrations.py in the Netdata repo
177
+ ```
178
+ The body is `export const categories = [...]; export const
179
+ integrations = [...]`.
180
+- `integrations/integrations.json` -- pure JSON with the
181
+ `clean` variant of `{categories, integrations}`. No banner.
182
+
183
+Both are gitignored (`.gitignore:159-160`). They are produced
184
+fresh on every run; in CI, the workflow `rm`s them after the
185
+downstream scripts read them so they are NOT included in the
186
+auto-PR.
187
+
188
+### Commands a maintainer runs locally
189
+
190
+```bash
191
+cd <repo>
192
+./integrations/pip.sh # installs jsonschema referencing jinja2 ruamel.yaml
193
+python3 integrations/gen_integrations.py
194
+```
195
+
196
+Run from the repo root. The script depends on relative paths
197
+hard-coded in `gen_integrations.py:11-37`.
198
+
199
+## Stage 2 -- `gen_docs_integrations.py`
200
+
201
+Repo path: `integrations/gen_docs_integrations.py`.
202
+
203
+### Inputs
204
+
205
+- `integrations/integrations.js` -- the script parses it by
206
+ string-splitting on `export const categories = ` and
207
+ `export const integrations = ` (`:129-140`). It does NOT
208
+ read `integrations.json`.
209
+
210
+### Outputs
211
+
212
+For each integration entry, the script writes either a
213
+`<plugin-dir>/integrations/<slug>.md` file or a
214
+`<plugin-dir>/README.md` file (depending on type). The full
215
+mapping per integration type is in `per-type-matrix.md`. Slug
216
+rules are in `artifacts-and-banners.md`.
217
+
218
+After writing, the script:
219
+
220
+1. Calls `resolve_related_links()` (`:56-78`) to convert
221
+ `{% relatedResource id="..." %}name{% /relatedResource %}`
222
+ markers (left in by `templates/overview/collector.md:42`
223
+ and `templates/related_resources.md:5`) into
224
+ `[name](/path)` markdown links. **Two-pass resolution**:
225
+ the markers are present in pass 1; they get rewritten in
226
+ pass 2 after every file is written so the id-to-path map
227
+ is complete. If the id is not found, the marker is
228
+ replaced with bare `name` text (silent fallback).
229
+
230
+2. Calls `make_symlinks(symlink_dict)` (`:527-544`) to symlink
231
+ `<plugin-dir>/README.md -> integrations/<sole-file>.md`
232
+ when the directory holds exactly one integration. Only
233
+ fires when `len(list(integrations_dir.iterdir())) == 1`
234
+ (`:466`). Multi-integration directories are NOT
235
+ symlinked.
236
+
237
+3. Cleans the corresponding `**/integrations` directories
238
+ BEFORE writing (`:19-41`), so removed integrations vanish
239
+ from the tree.
240
+
241
+### Scoped regen
242
+
243
+The script accepts `-c plugin/module` to scope cleanup and
244
+regen to one collector (`:578-583`). Useful locally:
245
+
246
+```bash
247
+python3 integrations/gen_docs_integrations.py -c go.d/snmp
248
+```
249
+
250
+NOT used by CI; CI always runs without `-c` (full regen).
251
+
252
+## Stage 3 -- `gen_doc_collector_page.py`
253
+
254
+Repo path: `integrations/gen_doc_collector_page.py`.
255
+
256
+Reads `integrations/integrations.js` (`:37-46`). Walks the
257
+category tree; the "section-level" categories are children of
258
+`data-collection` (`:82` -- `parent == 'data-collection'`).
259
+
260
+Writes `src/collectors/COLLECTORS.md` (committed). This is the
261
+"Monitor anything with Netdata" umbrella marketing page that
262
+lists every collector in tabular form, grouped by category.
263
+
264
+### Notable behaviors
265
+
266
+- Sort order: "Linux first, Other last" (`:279-303`).
267
+- Description extraction: `extract_description_from_overview`
268
+ reads `## Overview` body, uses the first sentence (`:143-183`);
269
+ falls back to `meta.monitored_instance.description`; final
270
+ fallback `Monitor <name>`.
271
+- Slug for table links: `to_slug(display_name)` -- lowercase,
272
+ spaces to `_`, `/` to `-`, strips parentheses (`:213-215`).
273
+- Hardcoded marketing anchors: `_render_tech_navigation`
274
+ (`:424-493`) writes `#cloud-provider-managed`, `#kubernetes`,
275
+ `#search-engines`, `#freebsd`, `#message-brokers`, etc.
276
+ Several of these category IDs do NOT exist in
277
+ `categories.yaml` -- some links go to non-existent anchors.
278
+ See `gotchas.md`. Header literal "850+ integrations" is also
279
+ baked in.
280
+
281
+## Stage 4 -- `gen_doc_secrets_page.py`
282
+
283
+Repo path: `integrations/gen_doc_secrets_page.py`.
284
+
285
+Reads `integrations/integrations.js` (`:213-217`), filters
286
+entries where `integration_type == 'secretstore'`, builds the
287
+"Supported Secretstore Backends" table from each backend's
288
+`meta.kind`, `meta.name`, `collector_configs.summary.{operand_format,
289
+example_operand}`, and renders via
290
+`integrations/templates/secrets.md`. Writes
291
+`src/collectors/SECRETS.md` (committed, `:358`).
292
+
293
+The bulk of `SECRETS.md` is **static content baked into the
294
+script** (`SECRETS_PAGE` dict, `:20-203`). Only the backends
295
+table is dynamic. To change the static prose, edit the script.
296
+
297
+## Stage 5 -- `gen_doc_service_discovery_page.py`
298
+
299
+Repo path: `integrations/gen_doc_service_discovery_page.py`.
300
+
301
+Mirror of the secrets stage for service discovery. Reads
302
+`integrations.js`, filters
303
+`integration_type == 'service_discovery'`, renders via
304
+`integrations/templates/service_discovery.md`. Writes
305
+`src/collectors/SERVICE-DISCOVERY.md` (committed, `:382`).
306
+Most content is static (`SD_PAGE` dict, `:21-257`).
307
+
308
+**KNOWN GAP**: this stage is NOT wired into the
309
+`generate-integrations.yml` workflow. CI does not run it. The
310
+file in tree drifts from metadata.yaml until a developer runs
311
+the script manually (or a future PR adds it to CI). See
312
+`gotchas.md` and the SOW followups.
313
+
314
+## CI workflow 1 -- `generate-integrations.yml`
315
+
316
+Repo path: `.github/workflows/generate-integrations.yml`.
317
+
318
+### Triggers
319
+
320
+- `push` to `master` filtered by paths
321
+ (`generate-integrations.yml:6-25`):
322
+ - `**/metadata.yaml` (every collector / exporter / notification
323
+ metadata)
324
+ - `integrations/templates/**`
325
+ - `integrations/schemas/**`
326
+ - `integrations/categories.yaml`, `integrations/deploy.yaml`
327
+ - `integrations/cloud-notifications/metadata.yaml`,
328
+ `integrations/cloud-authentication/metadata.yaml`
329
+ - the four older Python scripts (NOT
330
+ `gen_doc_service_discovery_page.py` -- the gap)
331
+- `workflow_dispatch` -- manual.
332
+
333
+### Concurrency
334
+
335
+- `integrations-${{ github.ref }}`, `cancel-in-progress: true`.
336
+
337
+### Repo gate
338
+
339
+- `if: github.repository == 'netdata/netdata'` -- forks do NOT
340
+ trigger this workflow.
341
+
342
+### Steps
343
+
344
+1. `actions/checkout@v6` (depth 1, recursive submodules).
345
+2. `apt install python3-venv` + `./integrations/pip.sh` to
346
+ install Python deps.
347
+3. `python3 integrations/gen_integrations.py`.
348
+4. `python3 integrations/gen_docs_integrations.py`.
349
+5. `python3 integrations/gen_doc_collector_page.py`.
350
+6. `python3 integrations/gen_doc_secrets_page.py`.
351
+7. **NOT** `gen_doc_service_discovery_page.py` -- gap.
352
+8. `rm -rf go.d.plugin virtualenv integrations/integrations.js
353
+ integrations/integrations.json` -- prevents the auto-PR from
354
+ committing the runtime artifacts.
355
+9. `peter-evans/create-pull-request@v8` -- branch
356
+ `integrations-regen`, label `integrations-update`, title
357
+ `Regenerate integrations docs`, token
358
+ `NETDATABOT_GITHUB_TOKEN`. Reviewed and merged manually.
359
+10. Slack failure notification on master failures.
360
+
361
+## CI workflow 2 -- `check-markdown.yml`
362
+
363
+Repo path: `.github/workflows/check-markdown.yml`.
364
+
365
+### Triggers
366
+
367
+- `pull_request` filtered by paths:
368
+ - `**/*.md`, `**/*.mdx`
369
+ - `docs/**`, `**/metadata.yaml`, `integrations/**`
370
+
371
+### Steps
372
+
373
+1. Checkout PR branch and the `netdata/learn` repo.
374
+2. Install Python deps (`./integrations/pip.sh`).
375
+3. Run `gen_integrations.py`, `gen_docs_integrations.py`,
376
+ `gen_doc_collector_page.py`, `gen_doc_secrets_page.py`
377
+ (same gap on SD page generator).
378
+4. Run `learn/ingest/ingest.py --local-repo netdata:...
379
+ --ignore-on-prem-repo --fail-links-netdata`
380
+ (`check-markdown.yml:64-69`) -- validates that all
381
+ generated markdown links resolve through Learn's ingest
382
+ pipeline.
383
+
384
+This workflow validates but does NOT auto-commit. It acts as
385
+a gate on PRs. A failure here means a PR cannot merge until
386
+the metadata or links are fixed.
387
+
388
+## CMake target -- `render-docs`
389
+
390
+Repo path: `packaging/cmake/Modules/NetdataRenderDocs.cmake`.
391
+
392
+A developer-facing convenience target. When wired up by the
393
+build system, it runs the same generator chain (with
394
+`gen_integrations` + `gen_docs_integrations` only by default).
395
+Useful for local validation. NOT a substitute for running the
396
+scripts directly during active development.
397
+
398
+## End-to-end: a single PR's flow
399
+
400
+1. Developer edits `src/go/plugin/go.d/collector/foo/metadata.yaml`
401
+ (and the four other consistency-rule files: `config_schema.json`,
402
+ stock conf, `health.d/foo.conf`, `README.md`).
403
+2. Developer runs locally:
404
+ ```bash
405
+ ./integrations/pip.sh
406
+ python3 integrations/gen_integrations.py
407
+ python3 integrations/gen_docs_integrations.py -c go.d/foo
408
+ python3 integrations/gen_doc_collector_page.py
409
+ python3 integrations/gen_doc_secrets_page.py
410
+ ```
411
+3. Developer commits the regenerated `integrations/foo.md`,
412
+ the symlinked `README.md` (if applicable), and the updated
413
+ `src/collectors/COLLECTORS.md` if the collector list
414
+ changed.
415
+4. PR is opened. `check-markdown.yml` runs, regenerates the
416
+ same files in CI, and validates Learn ingest. If the dev's
417
+ committed files differ from CI's regen, the PR fails.
418
+5. Reviewer checks the five-file consistency.
419
+6. PR merges. `generate-integrations.yml` triggers on master,
420
+ regenerates everything, and opens an `integrations-regen`
421
+ PR if anything is now stale (typically nothing, because the
422
+ dev already committed the regen). Maintainer merges.
423
+7. Cloud-frontend's own CI (in
424
+ `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/`) runs
425
+ `gen_integrations.py` against the new master and copies
426
+ `integrations.js` into its source. See `in-app-contract.md`.
427
+8. Learn's ingest pulls the new `integrations/foo.md` on its
428
+ 3-hourly schedule. See the `learn-site-structure` skill.
.agents/skills/integrations-lifecycle/recipes/INDEX.md
new
+49
@@ -0,0 +1,49 @@
1
+# Recipes -- INDEX
2
+
3
+Step-by-step recipes for the most common integration-lifecycle
4
+tasks a maintainer (or AI assistant) performs.
5
+
6
+| Recipe | When to use |
7
+|---|---|
8
+| `add-go-collector.md` | Adding a new go.d module from scratch (most common case). |
9
+| `update-collector.md` | Modifying an existing collector's metrics, config, alerts, or docs. |
10
+| `add-ibm-d-module.md` | Adding a new ibm.d module (where `metadata.yaml` is generated). |
11
+| `add-cloud-notification.md` | Adding a new cloud-notification destination. |
12
+| `add-secretstore-backend.md` | Adding a new secretstore backend. |
13
+| `add-service-discovery.md` | Adding a new service-discovery discoverer. |
14
+| `regenerate-locally.md` | Running the full pipeline locally to verify your changes before pushing. |
15
+
16
+## Common preamble for all recipes
17
+
18
+Every recipe assumes you are in the agent repo root. Install
19
+the Python deps once per machine:
20
+
21
+```bash
22
+./integrations/pip.sh
23
+```
24
+
25
+That installs `jsonschema`, `referencing`, `jinja2`,
26
+`ruamel.yaml` -- the four packages `gen_integrations.py`
27
+needs.
28
+
29
+For fast iteration during development, prefer `-c plugin/module`
30
+scoping on `gen_docs_integrations.py` to skip cleaning/
31
+regenerating other directories:
32
+
33
+```bash
34
+python3 integrations/gen_docs_integrations.py -c go.d/<your-module>
35
+```
36
+
37
+## When in doubt
38
+
39
+1. Read `pipeline.md` for the end-to-end flow.
40
+2. Read `schema-reference.md` for the exact field your
41
+ `metadata.yaml` change needs.
42
+3. Read `consistency.md` for the five-file rule.
43
+4. Read `gotchas.md` BEFORE assuming the pipeline does the
44
+ obvious thing.
45
+5. If you encountered a question that this catalog doesn't
46
+ cover and you had to investigate to answer it, AUTHOR a
47
+ how-to under `../how-tos/<slug>.md` and add it to
48
+ `../how-tos/INDEX.md`. This rule is mandatory; see
49
+ `../SKILL.md` "Live how-to rule".
.agents/skills/integrations-lifecycle/recipes/add-go-collector.md
new
+270
@@ -0,0 +1,270 @@
1
+# Recipe: add a new go.d collector integration
2
+
3
+This recipe assumes you are adding a brand-new go.d module
4
+called `<name>`. For modifying an existing collector, see
5
+`update-collector.md`.
6
+
7
+## 0. Read first
8
+
9
+- `<repo>/.agents/skills/project-writing-collectors/SKILL.md`
10
+ -- the broader "how to write a collector" context (NIDL
11
+ contexts, dashboard shaping, plugin landscape).
12
+- `../SKILL.md` -- this skill's overview.
13
+- `../schema-reference.md` -- the `collector.json` schema
14
+ fields you will be filling in.
15
+
16
+## 1. Create the module skeleton
17
+
18
+Standard go.d layout:
19
+
20
+```
21
+src/go/plugin/go.d/collector/<name>/
22
+├── <name>.go # Module entrypoint, Init/Check/Collect
23
+├── config.go # Config struct
24
+├── config_schema.json # DYNCFG schema
25
+├── metadata.yaml # Integration metadata (this skill's territory)
26
+├── README.md # Will become a symlink to integrations/<slug>.md once gen runs
27
+├── testdata/ # Fixtures
28
+└── ...other .go files
29
+```
30
+
31
+Plus stock conf:
32
+
33
+```
34
+src/go/plugin/go.d/config/go.d/<name>.conf
35
+```
36
+
37
+Plus alerts (if any):
38
+
39
+```
40
+src/health/health.d/<name>.conf
41
+```
42
+
43
+## 2. Author `metadata.yaml`
44
+
45
+Use an existing rich collector as a template:
46
+`src/go/plugin/go.d/collector/postgres/metadata.yaml`.
47
+
48
+Required top-level fields per `collector.json`:
49
+
50
+```yaml
51
+plugin_name: go.d.plugin
52
+modules:
53
+ - meta:
54
+ plugin_name: go.d.plugin
55
+ module_name: <name>
56
+ monitored_instance:
57
+ name: "<Display Name>"
58
+ link: "https://upstream-site.example/"
59
+ categories:
60
+ - data-collection.<category> # see categories.yaml for valid ids
61
+ icon_filename: "<name>.svg"
62
+ keywords: [<keywords>]
63
+ related_resources:
64
+ integrations:
65
+ list: []
66
+ info_provided_to_referring_integrations:
67
+ description: ""
68
+ overview:
69
+ data_collection:
70
+ metrics_description: |
71
+ One paragraph: what we collect.
72
+ method_description: |
73
+ One paragraph: how we collect it.
74
+ supported_platforms:
75
+ include: []
76
+ exclude: []
77
+ multi_instance: true
78
+ additional_permissions:
79
+ description: ""
80
+ default_behavior:
81
+ auto_detection:
82
+ description: ""
83
+ limits:
84
+ description: ""
85
+ performance_impact:
86
+ description: ""
87
+ setup:
88
+ prerequisites:
89
+ list: []
90
+ configuration:
91
+ file:
92
+ name: "go.d/<name>.conf"
93
+ options:
94
+ description: ""
95
+ folding:
96
+ title: "Config options"
97
+ enabled: true
98
+ list: []
99
+ examples:
100
+ folding:
101
+ title: "Config"
102
+ enabled: true
103
+ list:
104
+ - name: "Basic"
105
+ description: "Basic configuration."
106
+ config: |
107
+ jobs:
108
+ - name: local
109
+ url: http://localhost:1234
110
+ troubleshooting:
111
+ problems:
112
+ list: []
113
+ alerts: []
114
+ metrics:
115
+ folding:
116
+ title: "Metrics"
117
+ enabled: false
118
+ description: ""
119
+ availability: []
120
+ scopes:
121
+ - name: global # will be auto-rewritten to "<Display Name> instance"
122
+ description: ""
123
+ labels: []
124
+ metrics:
125
+ - name: <name>.<context>
126
+ description: <Chart title>
127
+ unit: <unit>
128
+ chart_type: line # one of: line, area, stacked, heatmap
129
+ dimensions:
130
+ - name: <dim>
131
+```
132
+
133
+Hit every required field. The validator is strict (fatal on
134
+warnings). Refer to `../schema-reference.md` for the
135
+exhaustive field list.
136
+
137
+## 3. Make sure `categories.yaml` has your category
138
+
139
+If your `monitored_instance.categories` references a category
140
+that doesn't exist in `integrations/categories.yaml`, the
141
+validator will warn (fatal). Either pick an existing category
142
+or add a new one under the appropriate parent (typically
143
+`data-collection`).
144
+
145
+## 4. Stock `.conf`, `config_schema.json`, alerts, README
146
+
147
+These three are the rest of the five-file consistency rule:
148
+
149
+- `src/go/plugin/go.d/config/go.d/<name>.conf` -- the stock
150
+ config users will see at
151
+ `/etc/netdata/go.d/<name>.conf`. Keep it minimal but
152
+ representative. Show every common option with a comment.
153
+- `src/go/plugin/go.d/collector/<name>/config_schema.json` --
154
+ the DYNCFG schema. Each option in the stock `.conf` should
155
+ have a corresponding entry here, with the same default.
156
+- `src/health/health.d/<name>.conf` -- alerts on the metrics
157
+ declared in `metadata.yaml`. Each alert in this file should
158
+ have a matching entry under `metadata.yaml.modules[0].alerts[]`.
159
+- `src/go/plugin/go.d/collector/<name>/README.md` -- this is the
160
+ USER-FACING documentation. After step 5, this file will be
161
+ REPLACED with a symlink to
162
+ `integrations/<slug>.md`. So you do NOT hand-write the
163
+ README; the generator does. Stub it as empty initially.
164
+
165
+## 5. Run the pipeline locally
166
+
167
+From the repo root:
168
+
169
+```bash
170
+./integrations/pip.sh # once
171
+python3 integrations/gen_integrations.py
172
+python3 integrations/gen_docs_integrations.py -c go.d/<name>
173
+python3 integrations/gen_doc_collector_page.py
174
+python3 integrations/gen_doc_secrets_page.py
175
+```
176
+
177
+Expected outputs:
178
+
179
+- `integrations/integrations.js` and `integrations/integrations.json`
180
+ regenerated (gitignored, do NOT commit them).
181
+- `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`
182
+ CREATED. Inspect: it should contain the `<!--startmeta`
183
+ banner with your `sidebar_label` and `learn_rel_path`, then
184
+ the rendered overview / setup / metrics / alerts /
185
+ troubleshooting sections.
186
+- `src/go/plugin/go.d/collector/<name>/README.md` becomes a
187
+ symlink to `integrations/<slug>.md` (because there is
188
+ exactly one integration in this directory).
189
+- `src/collectors/COLLECTORS.md` updated to include your new
190
+ collector in its category section.
191
+
192
+If `gen_integrations.py` exits non-zero, read the warning
193
+output -- a schema validation failed. Fix `metadata.yaml` and
194
+re-run.
195
+
196
+## 6. Verify locally
197
+
198
+- Open the generated `integrations/<slug>.md` and make sure
199
+ every section reads correctly.
200
+- Open `src/collectors/COLLECTORS.md` and find your collector
201
+ in the table.
202
+- Run `git diff` and confirm the only changes are in:
203
+ - `src/go/plugin/go.d/collector/<name>/...` (your new module
204
+ files).
205
+ - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`
206
+ (the generated integration page).
207
+ - `src/go/plugin/go.d/collector/<name>/README.md` (now a
208
+ symlink).
209
+ - `src/collectors/COLLECTORS.md` (umbrella page updated).
210
+ - `src/health/health.d/<name>.conf` (alerts file).
211
+ - Possibly `integrations/categories.yaml` if you added a
212
+ category.
213
+ - NOT `integrations/integrations.js` or
214
+ `integrations.json` (gitignored).
215
+
216
+## 7. Commit and push
217
+
218
+Single PR, single commit (or a few logical commits) covering
219
+the five-file rule plus the generated integration page and
220
+umbrella update. Reviewers will check that all five files
221
+were updated together.
222
+
223
+## 8. CI
224
+
225
+- `check-markdown.yml` will run on the PR. It runs the same
226
+ pipeline scripts and validates Learn ingest. If your
227
+ committed integration page diverges from CI's regen, the
228
+ workflow fails -- fix locally and re-push.
229
+- After merge, `generate-integrations.yml` triggers on master.
230
+ Since you already committed the regen, this should not
231
+ produce changes. If it does, the auto-PR
232
+ (`Regenerate integrations docs`) catches the drift -- merge
233
+ it.
234
+
235
+## 9. Surface arrival timing
236
+
237
+- `src/collectors/COLLECTORS.md` is live in the repo
238
+ immediately after merge.
239
+- The cloud-frontend dashboard's Integrations page rebuilds
240
+ on its own schedule (when the cloud-frontend CI re-runs
241
+ `gen_integrations.py` against master). Coordinate with
242
+ the dashboard team if you need to know the exact next
243
+ build.
244
+- The Learn site's per-integration page lands within a few
245
+ hours -- Learn's `ingest.yml` workflow runs every 3 hours
246
+ (see the `learn-site-structure` skill for details).
247
+
248
+## Common mistakes
249
+
250
+- **Forgetting one of the five files.** The most common
251
+ cause of review feedback. Use `git status` after step 5 to
252
+ confirm all five (or six counting the umbrella) are
253
+ staged.
254
+- **Hand-editing `integrations/<slug>.md` after generation.**
255
+ Never. It is regenerated each time. Edit `metadata.yaml`
256
+ and re-run.
257
+- **Skipping `gen_doc_collector_page.py`.** This forgets to
258
+ update `src/collectors/COLLECTORS.md`, leaving your
259
+ collector invisible in the umbrella table even though the
260
+ per-integration page exists.
261
+- **Categories typo.** A category id that doesn't match
262
+ `categories.yaml` causes validation to fail (warnings are
263
+ fatal). The renderer would silently fall back to
264
+ `data-collection.applications` if your only declared
265
+ category is bogus -- that fallback is itself the symptom
266
+ of a typo, not the desired outcome.
267
+- **Slug collision.** If `clean_string(meta.name)` produces
268
+ the same slug as an existing collector in the same
269
+ directory, one overwrites the other silently. Pick a
270
+ unique enough display name.
.agents/skills/integrations-lifecycle/schema-reference.md
new
+440
@@ -0,0 +1,440 @@
1
+# Schema reference
2
+
3
+Exhaustive per-field reference for all 12 JSON Schemas under
4
+`integrations/schemas/`. Each schema is JSON Schema Draft 7;
5
+cross-refs use `./shared.json#/$defs/...` resolved by
6
+`Registry(retrieve=retrieve_from_filesystem)`
7
+(`gen_integrations.py:163-169`).
8
+
9
+Tables below use these column conventions:
10
+- **Field**: dotted path (`a.b.c[].d` for nested arrays).
11
+- **Type**: JSON Schema type or `$ref` indication.
12
+- **Req**: yes / no / conditional (with the condition).
13
+- **Values**: enum values, regex constraints, `minItems`,
14
+ `minLength`.
15
+- **Surface**: which output(s) the field affects (learn /
16
+ www / in-app / alerts / stock / README / none).
17
+- **Notes**: cross-field constraints, special handling in
18
+ `gen_integrations.py`.
19
+
20
+If the schema declares a field but no template renders it,
21
+"Surface: none" is recorded; the field is still validated and
22
+serialized into `integrations.js` but never appears anywhere
23
+visible.
24
+
25
+`additionalProperties: false` is NOT set on most schemas, so
26
+unknown keys pass through silently. See `gotchas.md`.
27
+
28
+## shared.json -- building blocks
29
+
30
+Referenced by every other schema for common structures.
31
+
32
+### `$defs.id`
33
+
34
+Single string field used in many schemas as an identifier.
35
+
36
+| Field | Type | Req | Values | Surface | Notes |
37
+|---|---|---|---|---|---|
38
+| `id` | string | yes | `minLength: 1` | all | URL-safe identifier; deduplication key in `dedupe_integrations` (`gen_integrations.py:789`). |
39
+
40
+### `$defs.instance`
41
+
42
+The "what is this thing" descriptor used by every per-integration entry.
43
+
44
+| Field | Type | Req | Values | Surface | Notes |
45
+|---|---|---|---|---|---|
46
+| `instance.name` | string | yes | -- | learn / www / in-app | Display name. Drives slug for most types. |
47
+| `instance.link` | string | yes | URL | learn / www | Official upstream site. |
48
+| `instance.categories` | array<string> | yes | each must match a `categories.yaml` id | learn / www / in-app | Validated; bogus removed (`gen_integrations.py:899-912`). If none survive, falls back to `categories.yaml` entries flagged `collector_default: true` (`:906-908`). |
49
+| `instance.icon_filename` | string | yes | -- | learn / www / in-app | Path under `${NETDATA_REPOS_DIR}/website/themes/tailwind/static/img/` (icon repo). |
50
+| `instance.variables` | object | no | values: string / int / bool / number | all rendered text | Triggers two-pass Jinja templating; see `pipeline.md`. |
51
+
52
+### `$defs.keywords`
53
+
54
+Search-keyword array.
55
+
56
+| Field | Type | Req | Values | Surface | Notes |
57
+|---|---|---|---|---|---|
58
+| `keywords` | array<string> | yes (in most parent schemas) | -- | learn frontmatter, in-app search | Emitted in the `<!--startmeta` block as `keywords: ['k1','k2']`. |
59
+
60
+### `$defs.short_setup`
61
+
62
+Minimal "Setup" block. Alternative to `full_setup` for
63
+notification-style integrations.
64
+
65
+| Field | Type | Req | Values | Surface | Notes |
66
+|---|---|---|---|---|---|
67
+| `short_setup.description` | string | yes (when `short_setup` used) | markdown | learn / in-app | Free-form setup text. |
68
+
69
+### `$defs.full_setup`
70
+
71
+The standard setup block for collectors / exporters /
72
+authentication / secretstore / service_discovery.
73
+
74
+| Field | Type | Req | Values | Surface | Notes |
75
+|---|---|---|---|---|---|
76
+| `full_setup.prerequisites.list[]` | array<obj> | yes | objects with `title`, `description` | learn / in-app | Rendered as h4 sections in `setup-generic.md`. |
77
+| `full_setup.prerequisites.list[].title` | string | yes | -- | learn / in-app | h4 text. |
78
+| `full_setup.prerequisites.list[].description` | string | yes | markdown | learn / in-app | body. |
79
+| `full_setup.configuration.file.name` | string | yes | -- | learn | Stock conf filename, e.g. `go.d/postgres.conf`. |
80
+| `full_setup.configuration.file.section_name` | string | no | -- | learn | netdata.conf section, e.g. `[plugin:proc]`. |
81
+| `full_setup.configuration.options.description` | string | yes | markdown | learn / in-app | Intro before the options table. |
82
+| `full_setup.configuration.options.folding.title` | string | yes | -- | learn (clean strips) | Folding section title. |
83
+| `full_setup.configuration.options.folding.enabled` | boolean | yes | -- | learn (clean strips) | Whether the section is collapsed by default. |
84
+| `full_setup.configuration.options.list[].name` | string | yes | -- | learn / in-app | Option name (e.g. `dsn`). |
85
+| `full_setup.configuration.options.list[].group` | string | no | -- | learn | Adds a "Group" column when present. |
86
+| `full_setup.configuration.options.list[].description` | string | yes | markdown | learn / in-app | Short description for the table cell. |
87
+| `full_setup.configuration.options.list[].detailed_description` | string | no | markdown | learn (anchor) | When set, table cell becomes a link to a detailed h5 section below. |
88
+| `full_setup.configuration.options.list[].default_value` | string / number / bool | yes | -- | learn / in-app | Default value as displayed in the table. |
89
+| `full_setup.configuration.options.list[].required` | boolean | yes | -- | learn / in-app | Yes/No column. |
90
+| `full_setup.configuration.examples.folding` | $ref `_folding` | no | -- | learn (clean strips) | Folding for the examples block. |
91
+| `full_setup.configuration.examples.list[].name` | string | yes | -- | learn / in-app | Example title. |
92
+| `full_setup.configuration.examples.list[].description` | string | yes | markdown | learn / in-app | Example explanation. |
93
+| `full_setup.configuration.examples.list[].config` | string | yes | YAML string | learn / in-app | Rendered inside a ```` ```yaml ```` fence. |
94
+| `full_setup.configuration.examples.list[].folding` | $ref `_folding_relaxed` | no | -- | learn (clean strips) | Per-example folding override. When absent, defaults to the parent `examples.folding.enabled` (`gen_integrations.py:918-922`). |
95
+
96
+### `$defs.troubleshooting`
97
+
98
+| Field | Type | Req | Values | Surface | Notes |
99
+|---|---|---|---|---|---|
100
+| `troubleshooting.problems.list[].name` | string | yes | -- | learn / in-app | Rendered as h3. |
101
+| `troubleshooting.problems.list[].description` | string | yes | markdown | learn / in-app | Body. |
102
+
103
+The `troubleshooting.md` template adds debug-mode boilerplate
104
+per plugin (e.g. `python.d.plugin`, `go.d.plugin`,
105
+`charts.d.plugin`); see
106
+`integrations/templates/troubleshooting.md:1-86`.
107
+
108
+### `$defs._folding`
109
+
110
+| Field | Type | Req | Values | Surface | Notes |
111
+|---|---|---|---|---|---|
112
+| `_folding.title` | string | yes | -- | learn (clean strips) | Section title. |
113
+| `_folding.enabled` | boolean | yes | -- | learn (clean strips) | Initial collapsed/expanded state. |
114
+
115
+### `$defs._folding_relaxed`
116
+
117
+Same as `_folding` but only `enabled` is required; `title`
118
+optional.
119
+
120
+## collector.json
121
+
122
+Top-level structure: a `plugin_name` plus a `modules:` array
123
+where each module is one collector integration.
124
+
125
+| Field | Type | Req | Values | Surface | Notes |
126
+|---|---|---|---|---|---|
127
+| `plugin_name` | string | yes | -- | (cascaded into modules) | Auto-copied to each `module.meta.plugin_name` at `gen_integrations.py:381`. |
128
+| `modules` | array<obj> | yes | -- | -- | One entry per integration. |
129
+| `modules[].meta.plugin_name` | string | yes | -- | id / edit_link | Redundant with top-level; both must agree (no enforcement). |
130
+| `modules[].meta.module_name` | string | yes | -- | id / stock conf basename | Matches stock conf section / filename. |
131
+| `modules[].meta.monitored_instance` | $ref `shared.instance` | yes | -- | all | Full instance block; `name` drives slug + sidebar label. |
132
+| `modules[].meta.keywords` | $ref `shared.keywords` | yes | -- | learn / in-app | |
133
+| `modules[].meta.community` | boolean | no | -- | badge color | When true, badge becomes "Community" (`gen_docs_integrations.py:424`). |
134
+| `modules[].meta.related_resources.integrations.list[].plugin_name` | string | yes (in entry) | -- | related-integrations panel | |
135
+| `modules[].meta.related_resources.integrations.list[].module_name` | string | conditional | required if `monitored_instance_name` is set (Draft-7 `dependencies` at `collector.json:61-63`) | related-integrations | See `gotchas.md` for non-obvious dependency semantics. |
136
+| `modules[].meta.related_resources.integrations.list[].monitored_instance_name` | string | no | -- | related-integrations | For cgroups multi-instance disambiguation. |
137
+| `modules[].meta.info_provided_to_referring_integrations.description` | string | yes | markdown | rendered when ANOTHER collector references this one | The "what THIS collector says when referenced from another." |
138
+| `modules[].overview.data_collection.metrics_description` | string | yes | markdown | learn / www | The "what we collect" prose. |
139
+| `modules[].overview.data_collection.method_description` | string | yes | markdown | learn / www | The "how we collect" prose. |
140
+| `modules[].overview.supported_platforms.include` | array<string> | yes (may be empty) | platform names | learn (`overview/collector.md:12-26`) | Allow-list. |
141
+| `modules[].overview.supported_platforms.exclude` | array<string> | yes (may be empty) | platform names | learn (`overview/collector.md:12-26`) | Block-list. |
142
+| `modules[].overview.multi_instance` | boolean | yes | -- | learn (`overview/collector.md:28-32`) | Drives the multi-instance sentence. |
143
+| `modules[].overview.additional_permissions.description` | string | yes (may be empty) | markdown | learn (`overview/collector.md:34-36`) | When non-empty, an extra paragraph. |
144
+| `modules[].overview.default_behavior.auto_detection.description` | string | yes | markdown | learn (`overview/collector.md:46-58`) | |
145
+| `modules[].overview.default_behavior.limits.description` | string | yes | markdown | learn (`overview/collector.md:46-58`) | |
146
+| `modules[].overview.default_behavior.performance_impact.description` | string | yes | markdown | learn (`overview/collector.md:46-58`) | |
147
+| `modules[].setup` | $ref `shared.full_setup` | yes | -- | learn / in-app | Rendered through `setup-generic.md` (with sample-`<lang>`-config.md per plugin). |
148
+| `modules[].troubleshooting` | $ref `shared.troubleshooting` | yes | -- | learn / in-app | |
149
+| `modules[].alerts[].name` | string | yes | -- | learn alerts table | |
150
+| `modules[].alerts[].link` | string | yes | URL or repo-relative | learn alerts table | Deep link to the `health.d/<...>.conf` definition. |
151
+| `modules[].alerts[].metric` | string | yes | metric context | learn alerts table | Must match a metric name in `metrics.scopes[].metrics[].name` (NOT enforced). |
152
+| `modules[].alerts[].info` | string | yes | -- | learn alerts table | Short alert description. |
153
+| `modules[].alerts[].os` | string | no | -- | learn alerts table | OS filter. |
154
+| `modules[].metrics.folding` | $ref `_folding` | yes | -- | learn (clean strips) | Folding for the entire metrics section. |
155
+| `modules[].metrics.description` | string | yes | markdown | learn | Intro to the metrics block. |
156
+| `modules[].metrics.availability` | array<string> | yes | -- | metrics table column-set | Defines which "availability" columns the table will have. |
157
+| `modules[].metrics.scopes[].name` | string | yes | -- | learn metrics table | Special: `global` is rewritten to `<instance> instance` at `gen_integrations.py:914-916`. |
158
+| `modules[].metrics.scopes[].description` | string | yes | markdown | learn metrics table | |
159
+| `modules[].metrics.scopes[].labels[].name` | string | yes | -- | learn | Label name. |
160
+| `modules[].metrics.scopes[].labels[].description` | string | yes | -- | learn | |
161
+| `modules[].metrics.scopes[].metrics[].name` | string | yes | metric context | learn metrics table | Chart context (e.g. `postgres.connections`). |
162
+| `modules[].metrics.scopes[].metrics[].availability` | array<string> | no | matches parent `metrics.availability` | metrics table | Drives column ticks (`metrics.md:32-37`). |
163
+| `modules[].metrics.scopes[].metrics[].description` | string | yes | -- | metrics table | Chart title. |
164
+| `modules[].metrics.scopes[].metrics[].unit` | string | yes | -- | metrics table | |
165
+| `modules[].metrics.scopes[].metrics[].chart_type` | string | yes | enum: `line, area, stacked, heatmap` | metrics table | |
166
+| `modules[].metrics.scopes[].metrics[].dimensions[].name` | string | yes | -- | metrics table | |
167
+| `modules[].functions.description` | string | yes (when `functions` present) | markdown | learn Live Data section | Intro. |
168
+| `modules[].functions.list[].id` | string | yes | -- | learn | Function id (matches the agent's Function name). |
169
+| `modules[].functions.list[].name` | string | yes | -- | learn | Display name. |
170
+| `modules[].functions.list[].description` | string | yes | markdown | learn | |
171
+| `modules[].functions.list[].parameters[].id` | string | yes | -- | learn parameters table | |
172
+| `modules[].functions.list[].parameters[].name` | string | yes | -- | learn parameters table | |
173
+| `modules[].functions.list[].parameters[].description` | string | yes | -- | learn parameters table | |
174
+| `modules[].functions.list[].parameters[].type` | string | yes | -- | learn parameters table | |
175
+| `modules[].functions.list[].parameters[].required` | boolean | yes | -- | learn parameters table | |
176
+| `modules[].functions.list[].parameters[].default` | string / number / bool | yes | -- | learn parameters table | |
177
+| `modules[].functions.list[].parameters[].options[].id` | string | yes | -- | learn parameters table | When present, parameter is enum-style. |
178
+| `modules[].functions.list[].parameters[].options[].name` | string | yes | -- | learn | |
179
+| `modules[].functions.list[].parameters[].options[].description` | string | no | -- | learn | |
180
+| `modules[].functions.list[].parameters[].options[].default` | boolean | no | -- | learn | |
181
+| `modules[].functions.list[].returns.description` | string | yes | markdown | learn | |
182
+| `modules[].functions.list[].returns.columns[].name` | string | yes | -- | learn returns table | |
183
+| `modules[].functions.list[].returns.columns[].type` | string | yes | -- | learn returns table | |
184
+| `modules[].functions.list[].returns.columns[].unit` | string | yes | -- | learn returns table | |
185
+| `modules[].functions.list[].returns.columns[].visibility` | string | no | enum: `hidden` | learn returns table | When `hidden`, column is suppressed. |
186
+| `modules[].functions.list[].performance` | string | yes | markdown | learn | Performance characteristics. |
187
+| `modules[].functions.list[].security` | string | yes | markdown | learn | Security considerations. |
188
+| `modules[].functions.list[].availability` | string | yes | markdown | learn | When the function is available. |
189
+| `modules[].functions.list[].prerequisites.list[].title` | string | yes (if prereqs present) | -- | learn | h4 text. |
190
+| `modules[].functions.list[].prerequisites.list[].description` | string | yes (if prereqs present) | markdown | learn | |
191
+| `modules[].functions.list[].require_cloud` | boolean | no | -- | learn functions table | Yes/No column. |
192
+
193
+Required at module root: `meta`, `overview`, `setup`,
194
+`troubleshooting`, `alerts`, `metrics`
195
+(`collector.json:611-618`).
196
+
197
+Required on `meta`: `plugin_name`, `module_name`,
198
+`monitored_instance`, `keywords`, `related_resources`,
199
+`info_provided_to_referring_integrations` (`collector.json:94-101`).
200
+
201
+## agent_notification.json
202
+
203
+Single object OR array of objects (oneOf).
204
+
205
+| Field | Type | Req | Values | Surface | Notes |
206
+|---|---|---|---|---|---|
207
+| `id` | $ref `shared.id` | yes | -- | id / dedupe | |
208
+| `meta` | $ref `shared.instance` | yes | -- | learn / in-app | `meta.name` drives slug. |
209
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
210
+| `overview.notification_description` | string | yes | markdown | learn (`overview/notification.md`) | The "what gets notified" prose. |
211
+| `overview.notification_limitations` | string | yes (may be empty) | markdown | learn | When non-empty, rendered as `## Limitations`. |
212
+| `global_setup.severity_filtering` | boolean | yes | -- | learn | Sentence in setup template. |
213
+| `global_setup.http_proxy` | boolean | yes | -- | learn | Sentence in setup template. |
214
+| `setup` | oneOf [`shared.short_setup`, `shared.full_setup`] | yes | -- | learn / in-app | Rendered by `setup-generic.md` which handles both shapes. |
215
+| `troubleshooting` | $ref `shared.troubleshooting` | no | -- | learn / in-app | |
216
+
217
+## cloud_notification.json
218
+
219
+Same shape as `agent_notification.json` minus `overview`
220
+(none required), with `setup` required.
221
+
222
+| Field | Type | Req | Values | Surface | Notes |
223
+|---|---|---|---|---|---|
224
+| `id` | $ref `shared.id` | yes | -- | id | |
225
+| `meta` | $ref `shared.instance` | yes | -- | learn / in-app | |
226
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
227
+| `setup` | oneOf [`shared.short_setup`, `shared.full_setup`] | yes | -- | learn / in-app | |
228
+| `troubleshooting` | $ref `shared.troubleshooting` | no | -- | learn / in-app | |
229
+
230
+`integrations/cloud-notifications/metadata.yaml` is a single
231
+file containing an ARRAY of these entries (one per
232
+notification destination).
233
+
234
+## authentication.json
235
+
236
+Same shape as `agent_notification.json` with renamed `overview`
237
+fields for the authentication context.
238
+
239
+| Field | Type | Req | Values | Surface | Notes |
240
+|---|---|---|---|---|---|
241
+| `id` | $ref `shared.id` | yes | -- | id | |
242
+| `meta` | $ref `shared.instance` | yes | -- | learn / in-app | |
243
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
244
+| `overview.authentication_description` | string | yes | markdown | learn (`overview/authentication.md`) | |
245
+| `overview.authentication_limitations` | string | yes (may be empty) | markdown | learn | |
246
+| `setup` | oneOf [`shared.short_setup`, `shared.full_setup`] | yes | -- | learn / in-app | |
247
+| `troubleshooting` | $ref `shared.troubleshooting` | no | -- | learn / in-app | |
248
+
249
+`integrations/cloud-authentication/metadata.yaml` is a single
250
+file with an array of authentication-method entries.
251
+
252
+## logs.json
253
+
254
+Single entry OR array. Required: `id`, `meta`, `keywords`,
255
+`overview`.
256
+
257
+| Field | Type | Req | Values | Surface | Notes |
258
+|---|---|---|---|---|---|
259
+| `id` | $ref `shared.id` | yes | -- | id | |
260
+| `meta` | $ref `shared.instance` | yes | -- | learn / in-app | |
261
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
262
+| `overview.description` | string | yes | markdown | learn (`overview/logs.md`) | h1 body. |
263
+| `overview.visualization.description` | string | yes | markdown | learn | `## Visualization` section. |
264
+| `overview.key_features.description` | string | yes | markdown | learn | `## Key features` section. |
265
+| `setup.prerequisites.description` | string | yes (when `setup` present) | markdown | learn (`setup-logs.md`) | |
266
+
267
+`integrations/logs/metadata.yaml` covers exactly three log
268
+types: `systemd-journal`, `windows-events`, `OpenTelemetry`.
269
+
270
+## secretstore.json
271
+
272
+Per-backend entries.
273
+
274
+| Field | Type | Req | Values | Surface | Notes |
275
+|---|---|---|---|---|---|
276
+| `id` | $ref `shared.id` | yes | -- | id | |
277
+| `meta.kind` | string | yes | -- | slug | **Drives slug** (NOT `meta.name`); matches stock conf filename `/etc/netdata/go.d/ss/<kind>.conf`. |
278
+| `meta.name` | string | yes | -- | learn / in-app | Display name. |
279
+| `meta.link` | string | yes | URL | learn / www | |
280
+| `meta.icon_filename` | string | yes | -- | learn / www / in-app | |
281
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
282
+| `overview.description` | string | yes | markdown | learn (`overview/secretstore.md`) | |
283
+| `overview.limitations` | string | no | markdown | learn | |
284
+| `setup` | $ref `shared.full_setup` | yes | -- | learn / in-app | Rendered via `setup-secretstore.md`. |
285
+| `collector_configs.description` | string | yes | markdown | learn (`collector_configs.md`) | |
286
+| `collector_configs.summary.operand_format` | string | yes | -- | `SECRETS.md` umbrella table | Used by `gen_doc_secrets_page.py` to build the supported-backends table. |
287
+| `collector_configs.summary.example_operand` | string | yes | -- | `SECRETS.md` umbrella table | as above |
288
+| `collector_configs.format.description` | string | no | markdown | learn | |
289
+| `collector_configs.format.syntax` | string | yes | -- | learn | E.g. `${store:<kind>:<name>:<operand>}`. |
290
+| `collector_configs.format.parts.list[].name` | string | yes | -- | learn | |
291
+| `collector_configs.format.parts.list[].description` | string | yes | -- | learn | |
292
+| `collector_configs.examples.list[].name` | string | yes (`minItems: 1`) | -- | learn | |
293
+| `collector_configs.examples.list[].description` | string | yes | -- | learn | |
294
+| `collector_configs.examples.list[].content` | string | yes | -- | learn | Code block. |
295
+| `collector_configs.examples.list[].language` | string | no | language id | learn | Code-fence language; defaults to `text` per schema description but template uses `'yaml'`. |
296
+| `troubleshooting` | $ref `shared.troubleshooting` | yes | -- | learn / in-app | |
297
+
298
+## service_discovery.json
299
+
300
+| Field | Type | Req | Values | Surface | Notes |
301
+|---|---|---|---|---|---|
302
+| `id` | $ref `shared.id` | yes | -- | id | |
303
+| `meta.kind` | string | yes | -- | slug | **Drives slug**; matches discoverer registry name and stock conf filename. |
304
+| `meta.name` | string | yes | -- | learn / in-app | |
305
+| `meta.tagline` | string | yes | -- | SD hub table | One-liner shown in the SERVICE-DISCOVERY.md table. |
306
+| `meta.link` | string | yes | URL | learn / www | |
307
+| `meta.icon_filename` | string | yes | -- | learn / www / in-app | |
308
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
309
+| `overview.description` | string | yes | markdown | learn (`overview/service_discovery.md`) | |
310
+| `overview.how_it_works` | string | no | markdown | learn | h3 under Overview. |
311
+| `overview.limitations` | string | no | markdown | learn | |
312
+| `setup` | $ref `shared.full_setup` | yes | -- | learn (`setup-service_discovery.md`) | |
313
+| `services.description` | string | yes | markdown | learn | |
314
+| `services.evaluation.description` | string | no | markdown | learn | |
315
+| `services.evaluation.list[].name` | string | yes | -- | learn | Evaluation criterion. |
316
+| `services.evaluation.list[].description` | string | yes | -- | learn | |
317
+| `services.template_variables.description` | string | no | markdown | learn | |
318
+| `services.template_variables.list[].name` | string | yes (`minItems: 1`) | -- | learn | Discoverer-specific template var name. |
319
+| `services.template_variables.list[].description` | string | yes | -- | learn | |
320
+| `services.template_variables.list[].type` | string | no | -- | learn | |
321
+| `services.examples.description` | string | no | markdown | learn | |
322
+| `services.examples.list[].name` | string | yes (`minItems: 1`) | -- | learn | |
323
+| `services.examples.list[].description` | string | yes | -- | learn | |
324
+| `services.examples.list[].config` | string | yes | -- | learn | YAML code block. |
325
+| `verify.description` | string | no | markdown | learn | |
326
+| `verify.checks.list[].name` | string | yes (`minItems: 1` when `verify` present) | -- | learn | |
327
+| `verify.checks.list[].description` | string | yes | -- | learn | |
328
+| `troubleshooting` | $ref `shared.troubleshooting` | yes | -- | learn / in-app | |
329
+
330
+Required at entry root: `id`, `meta`, `keywords`, `overview`,
331
+`setup`, `services`, `troubleshooting` (`service_discovery.json:237-245`).
332
+
333
+## deploy.json
334
+
335
+Top-level: ARRAY of objects (one per deploy method).
336
+
337
+| Field | Type | Req | Values | Surface | Notes |
338
+|---|---|---|---|---|---|
339
+| `id` | $ref `shared.id` | yes | -- | id | |
340
+| `meta` | $ref `shared.instance` | yes | -- | in-app dialog | `meta.categories` must include a `deploy.*` id. |
341
+| `keywords` | array<string> | yes | -- | in-app search | |
342
+| `install_description` | string | yes | markdown | in-app dialog | |
343
+| `methods[].method` | string | yes | -- | in-app dialog | E.g. `wget`, `curl`, `kubectl`. |
344
+| `methods[].commands[].channel` | string (enum) | yes | enum: `nightly`, `stable` | in-app dialog | |
345
+| `methods[].commands[].command` | string | yes | -- | in-app dialog | May contain custom tags `{% if $showClaimingOptions %}...{% /if %}`; stripped when `clean=True` (`gen_integrations.py:982-985`). |
346
+| `additional_info` | string | yes | markdown | in-app dialog | May contain custom tags. |
347
+| `clean_additional_info` | string | no | markdown | in-app dialog | Clean-variant override; when present, replaces `additional_info` in the `clean_*` branch (`gen_integrations.py:990-992`). |
348
+| `related_resources` | object | yes | -- | (TBD/empty) | Currently unused. |
349
+| `platform_info.group` | string (enum) | yes | enum: `include`, `no_include`, `""` | in-app dialog | `include`/`no_include` cross-ref `distros.yml` to filter the platform table. |
350
+| `platform_info.distro` | string | yes | -- | in-app dialog | Matches `distros.yml` `distro` field. |
351
+| `quick_start` | integer | yes | -- | in-app "Add Nodes" dialog | Sort order. Negative -> hidden. |
352
+
353
+Custom tag patterns recognized in `command` / `additional_info`:
354
+`{% if X %}...{% /if %}`, `{%...%}` (regex
355
+`gen_integrations.py:124`). Stripped when generating
356
+`clean=True` outputs.
357
+
358
+## exporter.json
359
+
360
+Same pattern as `agent_notification.json` with
361
+`overview.exporter_description` and
362
+`overview.exporter_limitations`.
363
+
364
+| Field | Type | Req | Values | Surface | Notes |
365
+|---|---|---|---|---|---|
366
+| `id` | $ref `shared.id` | yes | -- | id | |
367
+| `meta` | $ref `shared.instance` | yes | -- | learn / in-app | |
368
+| `keywords` | array<string> | yes | -- | learn frontmatter | |
369
+| `overview.exporter_description` | string | yes | markdown | learn (`overview/exporter.md`) | |
370
+| `overview.exporter_limitations` | string | yes (may be empty) | markdown | learn | When non-empty, rendered as `## Limitations`. |
371
+| `setup` | $ref `shared.full_setup` | yes | -- | learn / in-app | |
372
+| `troubleshooting` | $ref `shared.troubleshooting` | yes | -- | learn / in-app | |
373
+
374
+## categories.json
375
+
376
+Recursive tree definition.
377
+
378
+| Field | Type | Req | Values | Surface | Notes |
379
+|---|---|---|---|---|---|
380
+| `id` | string | yes | -- | category lookup | Dotted path, e.g. `data-collection.databases`. |
381
+| `name` | string | yes | -- | navigation | Display name. |
382
+| `description` | string | yes | markdown | navigation | Tooltip / overview. |
383
+| `children` | array<obj> | yes (may be empty) | -- | navigation | Recursive structure. |
384
+| `collector_default` | boolean | no | -- | fallback | When true, this category is the default if a collector's declared categories are all bogus (`gen_integrations.py:906-908`). |
385
+
386
+## distros.json
387
+
388
+Validates `.github/data/distros.yml`. **NOT actually
389
+enforced** -- `gen_integrations.py:1330` calls `load_yaml`
390
+without passing this schema. See `gotchas.md`.
391
+
392
+Top-level keys: `platform_map` (CPU arch -> docker platform
393
+string), `arch_order`, `include[]` (array of platform
394
+descriptors).
395
+
396
+Per-platform descriptor fields:
397
+
398
+| Field | Type | Req | Values | Surface | Notes |
399
+|---|---|---|---|---|---|
400
+| `distro` | string | yes | regex `^[a-z][a-z0-9]*$` | deploy platform table | |
401
+| `version` | string | yes | regex `^[a-z0-9][a-z.0-9]*$` | deploy platform table | |
402
+| `support_type` | string | yes | enum: `Core`, `Intermediate`, `Community`, `Third-Party`, `Unsupported` | deploy platform table | |
403
+| `notes` | string | yes | -- | deploy platform table | |
404
+| `eol_check` | bool / string | no | -- | deploy build matrix | |
405
+| `bundle_sentry` | bool / string | yes | -- | deploy build matrix | |
406
+| `base_image` | string | no | -- | deploy build matrix | |
407
+| `env_prep` | string | no | -- | deploy build matrix | |
408
+| `jsonc_removal` | string | no | -- | deploy build matrix | |
409
+| `test.ebpf-core` | bool | no | -- | deploy build matrix | |
410
+| `packages.type` | string | no | -- | deploy build matrix | |
411
+| `packages.arches` | array<string> | no | -- | deploy build matrix | |
412
+| `packages.repo_distro` | string | no | -- | deploy build matrix | |
413
+| `packages.alt_links` | array | no | -- | deploy build matrix | |
414
+
415
+Required: `distro`, `version`, `support_type`, `notes`,
416
+`bundle_sentry`. Garbage in `distros.yml` produces broken
417
+`platform_info` tables silently.
418
+
419
+## Cross-schema notes
420
+
421
+- The `<plugin-dir>/metadata.yaml` for collectors uses one of
422
+ two top-level shapes (`gen_integrations.py:381`):
423
+ - **Single-module** -- `plugin_name` and `modules: [<one>]`.
424
+ - **Multi-module** -- `plugin_name` and `modules: [<many>]`.
425
+ Both are validated against the same `collector.json` schema.
426
+ The split single/multi validation in
427
+ `check_collector_metadata.py` is dead code (see `gotchas.md`).
428
+
429
+- Schema `shared.json` cross-refs are resolved by
430
+ `Registry(retrieve=retrieve_from_filesystem)` so changes to
431
+ `shared.json` propagate to all consumers immediately.
432
+
433
+- `additionalProperties: false` is NOT set on most schemas.
434
+ Unknown keys (`alternative_monitored_instances`,
435
+ `most_popular`) pass through silently into `integrations.js`
436
+ but no template renders them. See `gotchas.md`.
437
+
438
+- Validation warnings are FATAL: `fail_on_warnings()`
439
+ (`gen_integrations.py:150-160`) returns 1 on any warning,
440
+ causing CI to fail and aborting doc regeneration.
.agents/skills/learn-site-structure/SKILL.md
new
+145
@@ -0,0 +1,145 @@
1
+---
2
+name: learn-site-structure
3
+description: Authoritative reference for how docs in this repo (and 5 other Netdata-org repos) become published pages on `learn.netdata.cloud`. Covers the `<repo>/docs/.map/map.yaml` source-of-truth (the actual lever -- filesystem path is irrelevant for routing), the live `ingest/ingest.py` orchestrator in the learn repo (NOT the legacy `ingest.js`), frontmatter injection, slug rules, sidebar autogeneration, MDX escape rules, versioning, the 4-mechanism redirect stack, the 6 source repositories, the every-3-hours CI ingest, Netlify deploy, and the `part_of_learn=True` opt-in for files hand-authored in the learn repo. Use when adding/moving/renaming/deleting a docs page; when a page on Learn looks wrong; when wondering whether to edit a doc here or in the learn repo; when reading `ingest.py`, `sidebars.js`, `docusaurus.config.js`, `static.toml`, `LegacyLearnCorrelateLinksWithGHURLs.json`, `netlify.toml`, the `<!--startmeta` blocks in `.mdx` files, or the workflows `ingest.yml` and `daily-learn-link-check.yml`.
4
+---
5
+
6
+# learn-site-structure
7
+
8
+This skill is the **single place** to learn how content in this
9
+repo (and adjacent Netdata-org repos) becomes published pages on
10
+the Netdata learn site (`learn.netdata.cloud`). It documents:
11
+
12
+- the `<repo>/docs/.map/map.yaml` source-of-truth that every
13
+ published page must appear in;
14
+- the source-path-to-Learn-URL computation (frontmatter is
15
+ injected from `map.yaml`; the source filesystem path is
16
+ irrelevant for routing);
17
+- the 16-step `ingest/ingest.py` orchestrator in the learn repo
18
+ (the live one -- NOT the legacy `ingest.js`);
19
+- the 6 source repositories the pipeline pulls from;
20
+- sidebar autogeneration via Docusaurus `autogenerated` mode;
21
+- the MDX escape rules every transformation enforces;
22
+- the 4-mechanism redirect stack and how moves auto-redirect;
23
+- the every-3-hours CI ingest cadence and Netlify deploy
24
+ contract;
25
+- the `part_of_learn: True` opt-in for files hand-authored in
26
+ the learn repo;
27
+- every surprise, dead artifact, and gotcha worth knowing.
28
+
29
+After reading SKILL.md plus the per-domain guides linked below,
30
+an assistant should never need to ask "how does a page get
31
+added to learn?", "how is the URL computed?", "should I edit
32
+this here or in the learn repo?", or "what runs in CI?".
33
+
34
+## Key concepts (read first)
35
+
36
+1. **`<repo>/docs/.map/map.yaml` is the lever.** Every page
37
+ published on Learn appears in this file. To add a page,
38
+ add a node. To move a page, move its node. To unpublish a
39
+ page, remove its node and (manually) update the redirect
40
+ catalog. The schema lives at
41
+ `<repo>/docs/.map/map.schema.json`; the authoring guide
42
+ lives at `<repo>/docs/.map/README.md`.
43
+
44
+2. **Source filesystem path is IRRELEVANT for routing.** A
45
+ doc's destination URL on Learn is computed from
46
+ `sidebar_label` + `learn_rel_path` injected from `map.yaml`,
47
+ NOT from where the file lives in this repo. Renaming a
48
+ source file does not break the URL; renaming the
49
+ `meta.label` in `map.yaml` DOES (and triggers an
50
+ auto-redirect).
51
+
52
+3. **The live orchestrator is `ingest/ingest.py`. Ignore
53
+ `ingest.js` and `ingest.md`.** Both are legacy. The README
54
+ at the learn repo root and `docs/.map/README.md` document
55
+ the live Python flow. Don't edit `ingest.js`; don't trust
56
+ `ingest.md`.
57
+
58
+4. **Sidebar is autogenerated.** `${NETDATA_REPOS_DIR}/learn/sidebars.js`
59
+ is one line: `{type: "autogenerated", dirName: "."}`.
60
+ Order is driven by `sidebar_position` frontmatter that
61
+ ingest assigns from `map.yaml` traversal order. To reorder
62
+ the sidebar, reorder rows in `map.yaml`.
63
+
64
+5. **`docs/` (in the learn repo) is OWNED by ingest.** Anything
65
+ under `${NETDATA_REPOS_DIR}/learn/docs/` is cleaned and
66
+ regenerated each ingest run, EXCEPT files marked
67
+ `part_of_learn: True` (currently only `docs/ask-nedi.mdx`).
68
+ Do NOT hand-edit `docs/` in the learn repo unless you set
69
+ that flag and accept that this is a hand-authored
70
+ exception.
71
+
72
+6. **6 source repositories feed Learn.** This repo
73
+ (`netdata/netdata`) is the bulk; also pulled:
74
+ `netdata-cloud-onprem`, `.github`, `agent-service-discovery`,
75
+ `netdata-grafana-datasource-plugin`, `helmchart`. Each can
76
+ be overridden locally with `--repos` for testing.
77
+
78
+7. **Auto-redirects on move/rename, manual on delete.**
79
+ Moving or renaming a page in `map.yaml` produces an
80
+ automatic Netlify redirect from the old URL to the new
81
+ target. Deleting a page requires manual surgery on
82
+ `${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json`
83
+ (documented in `<repo>/docs/.map/README.md`).
84
+
85
+## Table of contents
86
+
87
+| Guide | Purpose |
88
+|---|---|
89
+| `mapping.md` | The map.yaml schema; frontmatter injection; source-path-to-URL computation; slug rules; edge cases. |
90
+| `pipeline.md` | The 16-step `ingest.py` flow; source repos; CI workflow; Netlify deploy. |
91
+| `sidebars.md` | Docusaurus autogenerated sidebar; ordering rules; category overview pages; auto-grid pages. |
92
+| `mdx-rules.md` | Every transformation in `_escape_mdx_braces` and `sanitize_page`; what breaks MDX 3 parsing. |
93
+| `redirects.md` | The 4-mechanism redirect stack; auto-redirect on move/rename; manual unpublish surgery. |
94
+| `pitfalls-and-gotchas.md` | Every surprise, dead artifact, edge case, silent failure mode, undocumented behavior. |
95
+| `authoring-boundary.md` | What is owned by ingest (do not edit) vs hand-authored in learn repo; `part_of_learn: True`; cross-repo authoring. |
96
+| `recipes/INDEX.md` | Step-by-step recipes for add / move / rename / delete a doc page. |
97
+| `how-tos/INDEX.md` | Live catalog: every analysis question becomes a how-to entry. |
98
+
99
+## Live how-to rule (mandatory)
100
+
101
+If an assistant is asked a concrete question about how a Learn
102
+page is produced that is NOT already documented under
103
+`how-tos/` or one of the per-domain guides above, AND answering
104
+it requires non-trivial analysis (reading `ingest.py` or
105
+related scripts, running ingest locally, checking
106
+cross-references between this repo's `map.yaml` and the
107
+learn-repo output), the assistant MUST author a new how-to
108
+under `how-tos/<slug>.md` and add a one-line entry to
109
+`how-tos/INDEX.md` BEFORE completing the task. This rule is
110
+durable.
111
+
112
+## Path discipline
113
+
114
+This skill follows
115
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md`:
116
+
117
+- This repo's files: repo-relative
118
+ (`<repo>/docs/.map/map.yaml`, `<repo>/docs/...`).
119
+- Learn repo files: `${NETDATA_REPOS_DIR}/learn/...` (env-key
120
+ from `.env`).
121
+- Other Netdata-org sibling repos: `${NETDATA_REPOS_DIR}/<repo>/...`.
122
+- No literal home-directory or workstation-root paths anywhere.
123
+
124
+## Sources of truth referenced by this skill
125
+
126
+- `<repo>/docs/.map/map.yaml` -- the publication source of truth.
127
+- `<repo>/docs/.map/map.schema.json` -- the schema.
128
+- `<repo>/docs/.map/README.md` -- the authoring guide.
129
+- `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py` -- live orchestrator.
130
+- `${NETDATA_REPOS_DIR}/learn/ingest/autogenerateRedirects.py` -- redirect generator.
131
+- `${NETDATA_REPOS_DIR}/learn/sidebars.js` -- sidebar config.
132
+- `${NETDATA_REPOS_DIR}/learn/docusaurus.config.js` -- site config.
133
+- `${NETDATA_REPOS_DIR}/learn/static.toml` -- hand-curated static redirects.
134
+- `${NETDATA_REPOS_DIR}/learn/test_escape_mdx_braces.py` -- MDX escape test suite.
135
+- `${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml` -- ingest CI workflow.
136
+- `${NETDATA_REPOS_DIR}/learn/.github/workflows/daily-learn-link-check.yml` -- daily link checker.
137
+
138
+## Related skills
139
+
140
+- `integrations-lifecycle` -- the integrations side
141
+ (`metadata.yaml` -> per-integration `.md`). Integration pages
142
+ are produced before ingest runs and inserted into `map.yaml`
143
+ by ingest's `populate_integrations` step (see `mapping.md`).
144
+- `project-writing-collectors` -- the broader collector
145
+ authoring context.
.agents/skills/learn-site-structure/authoring-boundary.md
new
+185
@@ -0,0 +1,185 @@
1
+# Authoring boundary
2
+
3
+Where do you EDIT a doc to make it appear on Learn? It depends.
4
+This guide draws the boundary between content owned by ingest
5
+(do NOT edit in the learn repo) and content hand-authored
6
+directly in the learn repo.
7
+
8
+## Owned by ingest -- DO NOT edit in the learn repo
9
+
10
+The cleanup step at `ingest.py:1038-1064` walks
11
+`${NETDATA_REPOS_DIR}/learn/docs/` and removes every
12
+`.md` / `.mdx` / `.json` that does NOT carry
13
+`part_of_learn: True`. Anything in `docs/` is wiped and
14
+regenerated each run, EXCEPT files explicitly marked as
15
+hand-authored.
16
+
17
+Specifically:
18
+
19
+- `${NETDATA_REPOS_DIR}/learn/docs/**` -- every file except
20
+ those marked `part_of_learn: True`. Currently only
21
+ `${NETDATA_REPOS_DIR}/learn/docs/ask-nedi.mdx` is so
22
+ flagged.
23
+
24
+- `${NETDATA_REPOS_DIR}/learn/netlify.toml` -- fully
25
+ regenerated by `autogenerateRedirects.main`
26
+ (`ingest.py:2932`). Hand edits are overwritten.
27
+
28
+- `${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json`
29
+ -- appended to on each run. Hand edits to existing entries
30
+ ARE preserved (because they're values in a dict that gets
31
+ read and re-emitted), but new entries get added each ingest.
32
+ This is the file you DO edit for manual unpublish surgery
33
+ (see `redirects.md`).
34
+
35
+- `${NETDATA_REPOS_DIR}/learn/ingest/generated_map.yaml` --
36
+ recomputed each run; committed for diff visibility.
37
+
38
+- `${NETDATA_REPOS_DIR}/learn/ingest/one_commit_back_file-dict.yaml`
39
+ -- written each run as the next run's diff baseline.
40
+
41
+## Hand-authored in the learn repo -- safe to edit
42
+
43
+These files live in the learn repo and are the maintainer's
44
+responsibility:
45
+
46
+- `${NETDATA_REPOS_DIR}/learn/docusaurus.config.js` -- site
47
+ config, plugins, theme.
48
+- `${NETDATA_REPOS_DIR}/learn/sidebars.js` -- one-line
49
+ autogen config.
50
+- `${NETDATA_REPOS_DIR}/learn/babel.config.js`,
51
+ `${NETDATA_REPOS_DIR}/learn/tailwind.config.js`,
52
+ `${NETDATA_REPOS_DIR}/learn/postcss.config.js` -- build
53
+ config.
54
+- `${NETDATA_REPOS_DIR}/learn/static.toml` -- the static
55
+ section of redirects (gets copied verbatim into
56
+ `netlify.toml`).
57
+- `${NETDATA_REPOS_DIR}/learn/docs/ask-nedi.mdx` -- the only
58
+ hand-authored page in `docs/` (carries `part_of_learn: True`).
59
+- `${NETDATA_REPOS_DIR}/learn/src/**` -- React pages,
60
+ components, theme overrides, css, custom Docusaurus plugins.
61
+- `${NETDATA_REPOS_DIR}/learn/static/**` -- favicons, fonts,
62
+ Swagger UI, robots.txt, ahrefs verification, etc.
63
+- `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py` and the ingest
64
+ helpers (`ingest/autogenerateRedirects.py`,
65
+ `ingest/autogenerateSupportedIntegrationsPage.py`,
66
+ `ingest/check_learn_links.py`).
67
+- `${NETDATA_REPOS_DIR}/learn/versioning/remove_edit_links.py`
68
+ -- manual versioning helper.
69
+- `${NETDATA_REPOS_DIR}/learn/package.json`,
70
+ `${NETDATA_REPOS_DIR}/learn/yarn.lock`,
71
+ `${NETDATA_REPOS_DIR}/learn/vitest.config.js`.
72
+- `${NETDATA_REPOS_DIR}/learn/.github/**` -- workflows.
73
+- `${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json`
74
+ -- specifically for **manual unpublish surgery**, per
75
+ `<repo>/docs/.map/README.md:96-104`.
76
+
77
+## Source repos -- this is where most authoring happens
78
+
79
+For documentation content, edits go in the SOURCE repo (almost
80
+always `netdata/netdata`, this repo). Specifically:
81
+
82
+- **Doc body**: `<repo>/docs/<...>/<file>.md`. Path is
83
+ largely cosmetic; what matters is the entry in
84
+ `map.yaml`.
85
+- **Map**: `<repo>/docs/.map/map.yaml`. Required for
86
+ publication.
87
+- **README files for collectors**: NOT hand-edited. They are
88
+ generated by the integrations pipeline (see the
89
+ `integrations-lifecycle` skill) and copied into the learn
90
+ tree by ingest's integration-placeholder mechanism.
91
+
92
+## What if I want a page that LIVES in the learn repo?
93
+
94
+Set `part_of_learn: True` in the page's frontmatter. The
95
+cleanup step at `ingest.py:1038-1064` will preserve it on
96
+every ingest run. Currently `docs/ask-nedi.mdx` is the only
97
+page using this flag.
98
+
99
+This is the right answer for:
100
+- Pages that don't have a natural home in any source repo
101
+ (e.g. site-meta pages, "About" pages, the home page).
102
+- Highly-customized pages with React components that don't
103
+ belong in a markdown source.
104
+
105
+## Cross-repo authoring
106
+
107
+Pages published from non-`netdata/netdata` source repos:
108
+
109
+| Source repo | What lives there | Edit-flow |
110
+|---|---|---|
111
+| `netdata/netdata` | Most docs and integration metadata | This repo (you're here). |
112
+| `netdata/netdata-cloud-onprem` | On-prem doc pages | Edit there; ingest pulls into `Netdata Cloud On-Prem` category. |
113
+| `netdata/.github` | `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md` | Edit there. Branch is `main`, not `master`. |
114
+| `netdata/agent-service-discovery` | Service-discovery docs | Edit there. |
115
+| `netdata/netdata-grafana-datasource-plugin` | Grafana datasource docs | Edit there. |
116
+| `netdata/helmchart` | Kubernetes Helm chart docs | Edit there. |
117
+
118
+For each: the source repo's docs need a corresponding entry in
119
+`<repo>/docs/.map/map.yaml` (in this repo) -- the map is
120
+cross-repo even though it lives in `netdata/netdata`. Adding
121
+or moving a page in a different source repo still requires a
122
+map.yaml change in this repo.
123
+
124
+## Decision tree: where to edit
125
+
126
+Question: I want to edit/add a page. Where?
127
+
128
+1. **Is it an integration page?** (`<plugin-dir>/integrations/<slug>.md`
129
+ or a notification README under
130
+ `src/health/notifications/...`)
131
+ -> NO, it's generated. Edit the source `metadata.yaml` and
132
+ re-run the integrations pipeline (see the
133
+ `integrations-lifecycle` skill).
134
+
135
+2. **Is it a doc page in this repo's `<repo>/docs/`?**
136
+ -> Edit the `.md` file directly in this repo. If it's a new
137
+ file, add a row to `<repo>/docs/.map/map.yaml`. If it
138
+ already has a row in map.yaml, no map change needed unless
139
+ you're moving / renaming.
140
+
141
+3. **Is it the Ask Nedi entrypoint?**
142
+ -> Edit `${NETDATA_REPOS_DIR}/learn/docs/ask-nedi.mdx`
143
+ directly. It's `part_of_learn: True` and survives
144
+ ingest.
145
+
146
+4. **Is it from a sibling Netdata-org repo (cloud-onprem,
147
+ .github, agent-service-discovery, grafana-datasource,
148
+ helmchart)?**
149
+ -> Edit the file in that repo. Then ensure
150
+ `<repo>/docs/.map/map.yaml` has a corresponding row (in
151
+ THIS repo, even though the source is elsewhere).
152
+
153
+5. **Is it a styling/UI/build/redirect change?**
154
+ -> Edit in the learn repo (`docusaurus.config.js`,
155
+ `tailwind.config.js`, `static.toml`, `src/`, `static/`).
156
+
157
+6. **Is it a sidebar reorder?**
158
+ -> Edit `<repo>/docs/.map/map.yaml` in THIS repo. Reorder
159
+ the rows. The sidebar autogeneration follows map traversal
160
+ order.
161
+
162
+## Anti-patterns to flag in review
163
+
164
+- **Editing `${NETDATA_REPOS_DIR}/learn/docs/<page>.mdx`
165
+ without `part_of_learn: True`.** Will be wiped on next
166
+ ingest. The right fix is to edit the source `.md` in the
167
+ appropriate repo and ensure `map.yaml` references it.
168
+- **Editing `netlify.toml` directly.** Regenerated each
169
+ ingest. Edit `static.toml` for static redirects, or rely on
170
+ the auto-redirect mechanism (move/rename in `map.yaml`).
171
+- **Editing `sidebars.js`.** It's one line. Useless. Edit
172
+ `map.yaml` to reorder.
173
+- **Editing `_category_.json` files in the learn repo.** They
174
+ are wiped each ingest by `safe_cleanup_learn_folders`.
175
+ Either set the page in `map.yaml` so the position comes
176
+ from there, or pre-author the `_category_.json` in the
177
+ source repo and arrange for ingest to copy it (currently
178
+ no such mechanism; needs feature work).
179
+- **Editing `LegacyLearnCorrelateLinksWithGHURLs.json` for
180
+ anything OTHER than manual unpublish surgery.** Existing
181
+ entries are stable across ingest runs, but the maintainer
182
+ who added the entry is responsible for it.
183
+- **Editing `ingest.js` or `ingest.md`.** Both are LEGACY.
184
+ The live orchestrator is `ingest/ingest.py`. See
185
+ `pitfalls-and-gotchas.md`.
.agents/skills/learn-site-structure/how-tos/INDEX.md
new
+52
@@ -0,0 +1,52 @@
1
+# How-tos -- INDEX
2
+
3
+Live catalog of analysis-derived how-tos for the
4
+learn-site-structure skill.
5
+
6
+**Live-catalog rule** (also stated in `../SKILL.md` and
7
+`../recipes/INDEX.md`): if an assistant is asked a concrete
8
+question about how a Learn page is produced that requires
9
+non-trivial analysis (multiple file reads, running ingest
10
+locally, cross-referencing this repo's `map.yaml` with the
11
+learn-repo output) AND the answer is not already documented
12
+under one of the per-domain guides
13
+(`../mapping.md`, `../pipeline.md`, `../sidebars.md`,
14
+`../mdx-rules.md`, `../redirects.md`,
15
+`../pitfalls-and-gotchas.md`, `../authoring-boundary.md`) or
16
+the recipes (`../recipes/`), the assistant MUST author a new
17
+`how-tos/<slug>.md` and add a one-line entry to this INDEX
18
+BEFORE completing the task.
19
+
20
+This is a durable rule. Skipping it means the next assistant
21
+repeats the same analysis from scratch -- a framework
22
+violation.
23
+
24
+## Catalog
25
+
26
+(empty -- entries grow as assistants encounter
27
+not-yet-documented questions)
28
+
29
+| Topic | Slug | Notes |
30
+|---|---|---|
31
+| -- | -- | -- |
32
+
33
+## How to add a how-to
34
+
35
+1. Create `how-tos/<slug>.md` with:
36
+ - A one-line summary at the top (the question being
37
+ answered).
38
+ - The answer with file:line citations into this repo or
39
+ the learn repo (`${NETDATA_REPOS_DIR}/learn/...`).
40
+ - A "How I figured this out" footer naming the files
41
+ read and the commands run, so the next assistant can
42
+ verify or extend.
43
+2. Add a row to the table above with topic, slug, and short
44
+ notes.
45
+3. Commit alongside the work that prompted the analysis.
46
+
47
+## When NOT to add a how-to
48
+
49
+- The question is already covered by an existing per-domain
50
+ guide or recipe -- update that guide instead.
51
+- The answer is a one-liner that doesn't require analysis.
52
+- The answer is highly speculative or version-specific.
.agents/skills/learn-site-structure/mapping.md
new
+215
@@ -0,0 +1,215 @@
1
+# Mapping: source -> Learn URL
2
+
3
+The single most important fact to internalize about Learn:
4
+**source filesystem path is irrelevant for routing**. A page's
5
+URL on `learn.netdata.cloud` is computed from frontmatter that
6
+`ingest.py` INJECTS from `<repo>/docs/.map/map.yaml`. Without
7
+this mental model, every other rule seems arbitrary.
8
+
9
+## The map.yaml source of truth
10
+
11
+`<repo>/docs/.map/map.yaml` is the canonical declaration of
12
+which files are published on Learn and where they appear. It is
13
+a hierarchical tree of nodes; each leaf node represents one
14
+published page.
15
+
16
+Schema: `<repo>/docs/.map/map.schema.json`. Authoring guide:
17
+`<repo>/docs/.map/README.md`.
18
+
19
+### Per-node fields
20
+
21
+Each `meta` block declares one publication node:
22
+
23
+| Field | Type | Required | Notes |
24
+|---|---|---|---|
25
+| `meta.label` | string | yes | Display name. Becomes `sidebar_label` AND drives the destination filename. |
26
+| `meta.edit_url` | string | yes (for doc-emitting nodes) | Must match `^https://github\.com/netdata/<repo>/edit/<branch>/.+\.(md\|mdx)$`. The unique key ingest uses to look up source files. Optional ONLY for category-only nodes that contain integration placeholders. |
27
+| `meta.path` | string | no | Single-segment override of the URL path component if it should differ from the label. |
28
+| `meta.description` | string | no | Frontmatter description; powers the page's meta description. |
29
+| `meta.keywords` | array<string> | no | Frontmatter keywords; powers in-app search and Learn page metadata. |
30
+
31
+The schema sets `additionalProperties: false`. Unknown keys
32
+fail validation and abort the run with exit code 2
33
+(`ingest/ingest.py:2815-2819`).
34
+
35
+### Branches
36
+
37
+For all repos in the source list (see `pipeline.md`), the
38
+`edit_url` branch is `master` -- with one exception: the
39
+`netdata/.github` repo uses `main` branch instead of `master`
40
+(`ingest.py:1232-1235`).
41
+
42
+### Integration placeholders
43
+
44
+`integration_placeholder: true` rows in `map.yaml` are
45
+substituted at ingest time by integration pages
46
+auto-discovered from the source repos. The `populate_integrations`
47
+step (`ingest.py:752-1003`) finds every `.md` carrying the
48
+`DO NOT EDIT THIS FILE DIRECTLY` marker
49
+(`INTEGRATION_MARKER`, `ingest.py:113`), parses its hidden
50
+`<!--startmeta...endmeta-->` block, buckets by category
51
+(collectors / exporters / secretstore / functions /
52
+authentication / cloud-notifications / agent-notifications /
53
+logs), sorts by `(learn_rel_path, sidebar_label)`, and
54
+**inserts the rows in place of** the matching placeholder.
55
+Result is written to
56
+`${NETDATA_REPOS_DIR}/learn/ingest/generated_map.yaml`.
57
+
58
+So integration pages do NOT need explicit `map.yaml` rows;
59
+they are pulled in via the placeholder mechanism.
60
+
61
+## Frontmatter that ingest INJECTS
62
+
63
+When ingest matches a source file's `custom_edit_url` to a
64
+`map.yaml` row, it writes a hidden HTML-comment metadata block
65
+at the top of the source file. After `sanitize_page` runs, the
66
+comment delimiters become real YAML frontmatter (`<!--` ->
67
+`---`, `-->` -> `---`).
68
+
69
+Per `ingest.py:1221-1299` plus `ingest.py:1832-1897`:
70
+
71
+| Field | Source | Notes |
72
+|---|---|---|
73
+| `custom_edit_url` | `meta.edit_url` from `map.yaml` | Used by the "Edit this page" link. For autogenerated grid pages, this is `null`. |
74
+| `sidebar_label` | `meta.label` | Drives sidebar AND destination filename. |
75
+| `learn_status` | `"Published"` or `"AUTOGENERATED"` | Anything other than `Published` is excluded from `to_publish`. |
76
+| `learn_rel_path` | reconstructed from `map.yaml` hierarchy | Drives the destination directory tree. `root` means write to `docs/` directly. |
77
+| `keywords` | `meta.keywords` | Normalized to inline YAML array. CSV strings, brackets, and Python lists all accepted. |
78
+| `description` | `meta.description` | Skipped when empty/None to minimize diffs. |
79
+| `sidebar_position` | computed from map traversal order | Always written as quoted string `sidebar_position: "10"`. |
80
+| `slug` | computed (`/<learn_rel_path>/<label>` lowercased) | Author-supplied `slug:` in source is preserved as override (`ingest.py:1890-1895`). |
81
+| `learn_link` | `https://learn.netdata.cloud/docs<slug>` | Updated each ingest. Validated daily by `check_learn_links.py`. |
82
+
83
+Special case: paths containing `Collecting Metrics` get
84
+`toc_max_heading_level: 6` and `toc_collapsible: true` injected
85
+(`ingest.py:1251-1253`).
86
+
87
+## Source-path-to-URL computation
88
+
89
+For files marked `learn_status: Published`, the destination is
90
+computed by `create_mdx_path_from_metadata`
91
+(`ingest.py:1140-1204`):
92
+
93
+```
94
+docs/<learn_rel_path>/<sanitized_sidebar_label>.mdx
95
+```
96
+
97
+### Sanitization rules
98
+
99
+`sidebar_label` is sanitized for the filename (`ingest.py:1159-1175`):
100
+1. Strip `'`, `:`, `/`, `(`, `)`, `,`, backtick.
101
+2. Collapse repeated whitespace to single space.
102
+
103
+For the URL **slug** (different from filename), additionally:
104
+- Lowercase everything.
105
+- Spaces -> `-`.
106
+- `//` -> `/`.
107
+
108
+Filename keeps original case; slug is lowercase. So
109
+`docs/Welcome to Netdata/Welcome to Netdata.mdx` exists with
110
+spaces and capitals.
111
+
112
+### Special-cases
113
+
114
+1. **Functions integrations** -- if the source `custom_edit_url`
115
+ is under `/integrations/functions/`, the filename uses the
116
+ function slug from the URL stem (with `-` -> ` `) instead of
117
+ `sidebar_label` (`ingest.py:1153-1157`). Avoids collisions
118
+ when many integrations share a "Top Queries" label.
119
+
120
+2. **Category overview pages** -- if the last two segments of
121
+ the slug are equal (e.g. `/collecting-metrics/collecting-metrics`),
122
+ `create_mdx_path_from_metadata` returns the slug with the
123
+ duplicate trimmed (`ingest.py:1178-1191`). On disk the file
124
+ is still written as `docs/<X>/<X>.mdx`; Docusaurus then
125
+ routes it to the parent URL `/<X>` because it has the same
126
+ slug.
127
+
128
+3. **Frontmatter `slug:` override** -- if the source file
129
+ already has `slug:` declared in frontmatter
130
+ (custom-authored, like `docs/ask-nedi.mdx`), that value wins
131
+ (`ingest.py:1890-1895`). For these files, the
132
+ author-supplied slug must be set BEFORE the
133
+ `<!-- ... -->` metadata block so it survives the rewrite.
134
+
135
+4. **`id:` in mapDict** -- if the metadata in `mapDict` carries
136
+ an `id`, the last segment of the URL is replaced by the id
137
+ (`ingest.py:2120-2126`).
138
+
139
+### Edge cases
140
+
141
+- `README.md`, `index.md`, `_index.md`, `index.mdx`, hidden
142
+ directories: NONE of these are special-cased in `ingest.py`.
143
+ The Python pipeline never publishes a file unless it appears
144
+ in `map.yaml` (or comes from an integration placeholder), so
145
+ filename has no routing effect.
146
+- Files at repo root vs deep paths: same rule -- destination is
147
+ driven by `map.yaml.meta.label` + the node's position in the
148
+ hierarchy, not by source path depth.
149
+- Special filenames: ignored. Only `map.yaml` matters.
150
+
151
+## Examples
152
+
153
+### Add a new top-level page
154
+
155
+1. Create `<repo>/docs/getting-started-netdata/quick-tour.md`.
156
+ The exact path doesn't matter for routing -- use a
157
+ reasonable location.
158
+2. Add to `<repo>/docs/.map/map.yaml`:
159
+ ```yaml
160
+ - meta:
161
+ label: Quick Tour
162
+ edit_url: https://github.com/netdata/netdata/edit/master/docs/getting-started-netdata/quick-tour.md
163
+ description: A 5-minute tour of Netdata.
164
+ keywords: [tour, getting-started, intro]
165
+ ```
166
+ under the appropriate parent.
167
+3. Resulting Learn URL: `https://learn.netdata.cloud/docs/<parent-slug>/quick-tour`.
168
+4. Resulting filename in learn repo: `docs/<parent>/Quick Tour.mdx` (with spaces and capitals).
169
+
170
+### Move a page (different sidebar location, same source file)
171
+
172
+1. In `map.yaml`, move the node to its new parent.
173
+2. Keep `meta.edit_url` unchanged -- still points to the same
174
+ source file.
175
+3. After ingest, the new `learn_rel_path` is computed from the
176
+ new tree position; the diff between previous and current
177
+ target produces an automatic Netlify redirect from the old
178
+ URL to the new (see `redirects.md`).
179
+
180
+### Rename a page
181
+
182
+Same as a move -- the URL slug is derived from `sidebar_label`,
183
+so changing `meta.label` changes the URL. Old URL is
184
+auto-redirected.
185
+
186
+### Delete a page
187
+
188
+The map.yaml side is just removing the node and the source
189
+file. There is one manual step on the learn-repo side
190
+(redirect surgery in
191
+`LegacyLearnCorrelateLinksWithGHURLs.json`); see
192
+`redirects.md` and `recipes/delete-doc-page.md`.
193
+
194
+## Path-collision rule
195
+
196
+If two `to_publish` entries collide case-insensitively
197
+(`<learn_rel_path>, <sidebar_label>` matches), ingest emits a
198
+warning at `ingest.py:2891-2908`. On case-insensitive
199
+filesystems (macOS / Windows), one file silently overwrites
200
+the other. On Linux they coexist as separate files.
201
+
202
+## What lookups happen
203
+
204
+The lookup key is the **canonical edit URL**:
205
+
206
+```
207
+https://github.com/netdata/<repo>/edit/<branch>/<repo-rel-path>
208
+```
209
+
210
+(`ingest.py:1232-1235`). Branch is `master` for everything
211
+except `.github`, which uses `main`. This is the join key
212
+between ingest's filesystem walk and `map.yaml`.
213
+
214
+If a source file's edit URL doesn't match any `map.yaml` row,
215
+the file is skipped. No warning -- silent skip.
.agents/skills/learn-site-structure/mdx-rules.md
new
+192
@@ -0,0 +1,192 @@
1
+# MDX rules
2
+
3
+Docusaurus uses MDX 3 for `.mdx` files. MDX is markdown plus
4
+JSX, which means certain markdown text that is harmless in
5
+plain GitHub-rendered markdown will break MDX parsing. Ingest
6
+runs a battery of escape transformations to make source `.md`
7
+content survive the conversion to `.mdx`.
8
+
9
+Live in `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py:1721-1799`
10
+(the `_escape_mdx_braces` function and adjacent transforms),
11
+exhaustively tested in
12
+`${NETDATA_REPOS_DIR}/learn/test_escape_mdx_braces.py`. Every
13
+transform is applied to every published file via
14
+`sanitize_page` (`ingest.py:1765-1829`), in this order:
15
+
16
+## 1. Frontmatter conversion
17
+
18
+- First `<!--` -> `---`, first `-->` -> `---` (`ingest.py:1779-1780`).
19
+- Turns the injected hidden metadata block into real YAML
20
+ frontmatter.
21
+
22
+## 2. Strip unhideme markers
23
+
24
+`<!--unhideme` and `unhideme-->` markers are stripped
25
+(`ingest.py:1783-1784`).
26
+
27
+## 3. `<details><summary>` newline fix
28
+
29
+`<details><summary>` -> `<details>\n<summary>` (and the
30
+`<details open>` variant) for MDX 3 compatibility
31
+(`ingest.py:1787-1788`). Without this fix, the inline form
32
+breaks the parser.
33
+
34
+**Caveat**: only the two literal forms it knows are fixed. Any
35
+other variant (e.g. `<details class="x"><summary>`) breaks
36
+MDX rendering silently.
37
+
38
+## 4. `_escape_mdx_braces`
39
+
40
+Escape every bare `{` outside fenced/inline code:
41
+
42
+- **Preserve** fenced ```` ``` ... ``` ``` blocks (DOTALL
43
+ match).
44
+- **Preserve** inline `` `...` `` (no newlines, no nested
45
+ backticks).
46
+- **Preserve** `^import ...$` lines (MDX ESM destructuring
47
+ syntax).
48
+- **Preserve** `^export (default|function|const|let|var|{) ...$`
49
+ lines (regex
50
+ `export\s+(?:default|function|const|let|var|\{)`).
51
+- Replace every bare `{` (not already preceded by `\`) with
52
+ `\{`.
53
+- Restore `style=\{\{` back to `style={{` because that's valid
54
+ JSX.
55
+
56
+**Bash `export VAR=...` lines are NOT preserved as ESM**. The
57
+`export` regex requires `default`, `function`, `const`, `let`,
58
+`var`, or `{` after `export`. Bash-style `export NETDATA_FOO=bar`
59
+passes through escape unchanged because it has no `{`. If a
60
+bash export contained `{`, it would get escaped (probably what
61
+you want).
62
+
63
+## 5. Specific operator escapes
64
+
65
+- `<=` -> `\<=`
66
+- `%<` -> `%\<`
67
+- `<->` -> `\<->`
68
+
69
+(`ingest.py:1792-1794`). MDX would otherwise try to parse
70
+these as JSX tags. Note these are **exact-substring** rules;
71
+near-variants like `< =` (with space) or `<---->` (multi-dash)
72
+are NOT covered. See `pitfalls-and-gotchas.md`.
73
+
74
+## 6. Bare URL angle-bracket links
75
+
76
+Converted to markdown links (`ingest.py:1797-1799`):
77
+
78
+- `<https://...>` -> `[https://...](https://...)`
79
+- `<http://...>` -> `[http://...](http://...)`
80
+- `<email@x.y>` -> `[email@x.y](mailto:email@x.y)`
81
+
82
+So `<...>` used as a "I want this rendered as a link" markdown
83
+shortcut works correctly through ingest.
84
+
85
+## 7. `meta_yaml` rewrite
86
+
87
+If `meta_yaml: "<url>"` is present in the file, the file is
88
+treated as an integration. `meta_yaml:` is removed and
89
+`custom_edit_url` is rewritten to that URL
90
+(`ingest.py:1801-1808`). Silent rewrite -- any file with that
91
+key triggers it.
92
+
93
+## 8. Integration logo annotation
94
+
95
+Integration files (`INTEGRATION_MARKER` present) get
96
+`<img src="https://(www.)?netdata.cloud/img/...">` annotated
97
+with `data-integration-logo`, `data-logo-contrast-light`,
98
+`data-logo-contrast-dark`, `data-logo-contrast-confidence`
99
+after a fetch+luminance analysis
100
+(`_annotate_integration_logo_tags` and `_analyze_remote_logo`,
101
+`ingest.py:1647-1718`). Used by `Grid_integrations` and the
102
+dashboard theme to add a subtle glow on low-contrast logos.
103
+
104
+## 9. Drop analytics pixel lines
105
+
106
+Lines starting with `[![analytics]` are dropped
107
+(`ingest.py:1816-1817`). These are leftover GitHub
108
+README-style tracking pixels that don't make sense on Learn.
109
+
110
+## What survives, what doesn't
111
+
112
+The escape rules cover the most common breakage patterns:
113
+
114
+| Pattern | Survives? | Why |
115
+|---|---|---|
116
+| `{word}` outside code | escaped to `\{word}` | rule 4 |
117
+| `${expr}` | escaped to `\${expr}` | rule 4 |
118
+| `style={{ }}` JSX | yes (un-escaped) | rule 4 restoration |
119
+| Already-escaped braces | unchanged | rule 4 idempotent |
120
+| Fenced or inline code | unchanged | rule 4 preservation |
121
+| MDX `import`/`export` ESM | unchanged | rule 4 preservation |
122
+| Tables with `{` cells | escaped per cell | rule 4 |
123
+| `<=`, `%<`, `<->` exact | escaped | rule 5 |
124
+| `< =` (with space) | NOT covered | rule 5 is exact-substring |
125
+| `<---->` (long arrow) | NOT covered | rule 5 is exact-substring |
126
+| `<htmltag>` body content | NOT escaped | breaks MDX unless wrapped in code |
127
+| `<details>` inline summary | fixed | rule 3 |
128
+| `<details class="x">` | NOT fixed | rule 3 only knows two forms |
129
+| `}` closing brace alone | NOT escaped | rule 4 only escapes `{` |
130
+| Bare URL in `<...>` | converted to MD link | rule 6 |
131
+
132
+## Test suite
133
+
134
+`${NETDATA_REPOS_DIR}/learn/test_escape_mdx_braces.py:74-377`
135
+exercises:
136
+
137
+- simple `{word}`, `${expr}` templates;
138
+- `{{double}}` braces;
139
+- `style={{ }}` JSX (preserved);
140
+- already-escaped braces (idempotent);
141
+- fenced and inline code preservation (multiple code blocks
142
+ per file, mixed inline and fenced);
143
+- table rows with mixed bare and code-fenced braces;
144
+- real Zabbix and Nagios integration content;
145
+- MDX `import`/`export` ESM lines;
146
+- bash `export` lines (currently pass through unchanged
147
+ because they have no braces);
148
+- empty input;
149
+- only-`{}` content;
150
+- nested `{outer{inner}}`;
151
+- unclosed `{`;
152
+- lone `}`;
153
+- newline after `{`;
154
+- the original hardcoded `{attribute_name}` patterns;
155
+- a full-document simulation.
156
+
157
+## Mermaid diagrams
158
+
159
+Mermaid diagrams are enabled at the markdown level in
160
+`docusaurus.config.js:26-30`. The escape rules preserve
161
+fenced code blocks, so ` ```mermaid ... ``` ` blocks survive
162
+intact.
163
+
164
+## What you can put in source `.md` files safely
165
+
166
+- Any plain markdown.
167
+- Code blocks (fenced and inline) -- whatever's inside them is
168
+ preserved.
169
+- MDX import/export ESM at the top of file (works in `.mdx`,
170
+ not in `.md`; ingest converts to `.mdx`).
171
+- `style={{ ... }}` JSX (preserved).
172
+- Mermaid in fenced ```` ```mermaid ``` ```.
173
+- `<details>` and `<summary>` with newlines between them.
174
+
175
+## What you should escape yourself in source
176
+
177
+- `<htmltag>` in body content -- always wrap in code (fenced
178
+ or inline).
179
+- Multi-character operators like `<->` with extra dashes
180
+ (`<-->`, `<--->`).
181
+- `< ` or ` <` with spaces around the `<`.
182
+- Closing-brace-only sequences if they're standalone.
183
+
184
+## Onbroken-links policy
185
+
186
+`docusaurus.config.js:22`: `onBrokenLinks: 'warn'`. Broken
187
+links never fail the Docusaurus build at the parse level; they
188
+only get caught by the ingest's pre-build link checker
189
+(`--fail-links`) and the daily 404 sweep
190
+(`daily-learn-link-check.yml`). So a broken link in your `.md`
191
+will silently make it to production unless caught by one of
192
+those two gates.
.agents/skills/learn-site-structure/pipeline.md
new
+273
@@ -0,0 +1,273 @@
1
+# Pipeline: ingest, CI, deploy
2
+
3
+This document maps the Learn ingest pipeline end to end: every
4
+step `ingest.py` performs, the source repositories it pulls
5
+from, the CI workflow that triggers it, and how the result
6
+gets to `learn.netdata.cloud`.
7
+
8
+## The orchestrator: `ingest/ingest.py`
9
+
10
+Live entrypoint: `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py`.
11
+The legacy `ingest.js` and `ingest.md` at the learn-repo root
12
+are NOT the orchestrator -- ignore them.
13
+
14
+### Argument parsing (`ingest.py:2513-2762`)
15
+
16
+| Flag | Purpose |
17
+|---|---|
18
+| `--repos OWNER/REPO:BRANCH ...` | Override the default repo list. |
19
+| `--local-repo NAME:/path ...` | Use a local copy via `shutil.copytree` instead of cloning. |
20
+| `--dry-run` | Skip output writes. |
21
+| `--debug` | Verbose logging. |
22
+| `--docs-prefix DOCS` | Default `docs`; the target directory in learn for published files. |
23
+| `--fail-links` | Exit 1 if broken internal links are detected at the end of the run. |
24
+| `--fail-links-{netdata,helmchart,onprem,asd,grafana,github}` | Per-source-repo fail-on-broken-links. |
25
+| `--gh-token` | GitHub token for clones. |
26
+| `--use_plain_https` | Force HTTPS clone URLs (no SSH). |
27
+| `--ignore-on-prem-repo` | Skip `netdata-cloud-onprem` clone and ignore broken-link references to it. |
28
+
29
+### The 16-step flow (`ingest.py:2513-3087`, `__main__`)
30
+
31
+1. **Argument parsing**.
32
+
33
+2. **Cleanup** (`ingest.py:2766-2768`):
34
+ - `unsafe_cleanup_folders('ingest-temp-folder')` -- wipes
35
+ the work dir.
36
+ - `safe_cleanup_learn_folders('docs')` -- walks
37
+ `${NETDATA_REPOS_DIR}/learn/docs/` and removes every
38
+ `.md` / `.mdx` / `.json` that does NOT carry
39
+ `part_of_learn: True`. The flag is the **opt-in for
40
+ hand-authored files** that should survive between runs.
41
+ Currently only `docs/ask-nedi.mdx`.
42
+
43
+3. **Clone all repos** (`ingest.py:2776-2800`) into
44
+ `ingest-temp-folder/<repo>/` with `--depth 1`. Each repo
45
+ tracks the branch declared in `default_repos`
46
+ (`ingest.py:74-105`). `--ignore-on-prem-repo` skips the
47
+ on-prem clone and ignores broken-link references to it.
48
+ `--local-repo X:/path` and `--repos /path/to/repo` use
49
+ `shutil.copytree` for developer ergonomics.
50
+
51
+4. **Move + validate map** (`ingest.py:2802-2825`):
52
+ - `ingest-temp-folder/netdata/docs/.map/map.yaml` ->
53
+ `./map.yaml`.
54
+ - Validate against
55
+ `ingest-temp-folder/netdata/docs/.map/map.schema.json`.
56
+ - Build `MAP_SIDEBAR_ORDER` and `MAP_DOC_SCOPE` for sidebar
57
+ position assignment.
58
+ - Schema failure -> exit code 2.
59
+
60
+5. **Enumerate markdowns** (`ingest.py:2828-2829`):
61
+ `glob('ingest-temp-folder/**/*.md*')` plus dot-directories
62
+ (`fetch_markdown_from_repo`, `ingest.py:1207-1211`).
63
+
64
+6. **Populate integrations** (`ingest.py:2832`): scan all
65
+ markdowns for `INTEGRATION_MARKER`, parse hidden metadata,
66
+ bucket by category, replace each `integration_placeholder`
67
+ row in the map with sorted (by `learn_rel_path`,
68
+ `sidebar_label`) integration rows. Writes the resulting
69
+ tabular map to `ingest/generated_map.yaml`.
70
+
71
+7. **Sidebar position assignment** (`ingest.py:2838-2840`):
72
+ `automate_sidebar_position` walks the (now expanded) map
73
+ dataframe and assigns sibling-relative positions in steps
74
+ of 10 within each parent scope, ordered by map traversal.
75
+
76
+8. **Inject metadata into source files** (`ingest.py:2842-2889`):
77
+ for each markdown, look up its row by `custom_edit_url` and
78
+ write the hidden metadata block at the top of the file. If
79
+ `learn_status == Published`, compute the destination via
80
+ `create_mdx_path_from_metadata`, store the destination +
81
+ ingestedRepo in `to_publish`, also inject `slug:` and
82
+ `learn_link:` into the file. Otherwise file is dropped.
83
+
84
+9. **Path collision check** (`ingest.py:2891-2908`): warn if
85
+ any two `to_publish` entries collide case-insensitively.
86
+
87
+10. **Publish each file** (`ingest.py:2916-2919`):
88
+ - `local_to_absolute_links(md_file, to_publish)` --
89
+ convert relative/abs-from-repo-root markdown links into
90
+ GitHub view URLs (rewritten in step 12).
91
+ - `copy_doc(md_file, to_publish[md_file]['learnPath'])` --
92
+ copy temp file to `docs/<...>.mdx` (creating dirs).
93
+ - `sanitize_page(learnPath)` -- runs MDX-escape transforms
94
+ (see `mdx-rules.md`).
95
+
96
+11. **Build cross-reference dictionary** (`ingest.py:2925-2927`):
97
+ `add_new_learn_path_key_to_dict` produces, for every
98
+ published file, a map: GitHub-view-link AND
99
+ GitHub-edit-link -> final `/docs/...` URL. Handles the
100
+ duplicate-segment slug trim and explicit `slug:`
101
+ overrides.
102
+
103
+12. **Rewrite GitHub links to Learn URLs** (`ingest.py:2929-2930`):
104
+ `convert_github_links` walks every published file. Any
105
+ `https://github.com/netdata/<repo>/blob/.../*.md` in the
106
+ body that maps to a file in `to_publish` is rewritten to
107
+ its final Learn URL. Header anchors are validated; broken
108
+ anchors are accumulated. Links to integration md files
109
+ that aren't in the map fall back to the parent README's
110
+ URL (`ingest.py:2153-2209`). Links to GitHub files that
111
+ exist in the repos but aren't in the map stay as GitHub
112
+ links (intentional -- `file_exists_in_repos` at
113
+ `ingest.py:711-719`). Truly missing targets become
114
+ `UNCORRELATED_LINK_COUNTER` increments.
115
+
116
+13. **Generate redirects** (`ingest.py:2932`): see
117
+ `redirects.md`.
118
+
119
+14. **Broken-link reporting** (`ingest.py:2940-3022`): print
120
+ broken URLs grouped by repo, broken anchors grouped by
121
+ repo, decide whether to exit 1 based on `--fail-links`
122
+ flags.
123
+
124
+15. **Post-processing** (`ingest.py:3030-3065`):
125
+ - Save current `(custom_edit_url -> new_learn_path)`
126
+ mapping to `ingest/one_commit_back_file-dict.yaml` for
127
+ next run's redirect diff.
128
+ - Cleanup `ingest-temp-folder` and `map.yaml`.
129
+ - `get_dir_make_file_and_recurse('./docs')` -- auto-create
130
+ grid pages for any integration directory that lacks an
131
+ overview.
132
+ - `ensure_category_json_for_dirs('docs')` -- write
133
+ `_category_.json` for any directory still without an
134
+ overview page.
135
+ - `normalize_sidebar_positions_by_parent('docs')` --
136
+ assign sibling positions deterministically per parent
137
+ scope.
138
+
139
+16. **No git push from the script.** The CI workflow handles
140
+ `git add` / commit / PR.
141
+
142
+### How content gets fetched
143
+
144
+`ingest.py` does NOT use the GitHub REST API for content. It
145
+clones full repos with `gitpython` (`ingest.py:1101-1137`), so
146
+there's no API rate-limit consideration in the live pipeline.
147
+(Legacy `ingest.js` did use the API; ignore it.)
148
+
149
+## The 6 source repositories
150
+
151
+Default at `ingest/ingest.py:74-105`:
152
+
153
+| Logical repo | Owner | Branch | What it feeds |
154
+|---|---|---|---|
155
+| `netdata` | `netdata` | `master` | The bulk of Learn -- all docs declared in `<repo>/docs/.map/map.yaml`, plus integration metadata generated by the agent build (collectors, exporters, secretstore, alerts, functions). Hosts the canonical map files. |
156
+| `netdata-cloud-onprem` | `netdata` | `master` | On-prem documentation (Netdata Cloud On-Prem category). Skippable via `--ignore-on-prem-repo`. |
157
+| `.github` | `netdata` | **`main`** | `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md` -- only those whose edit URLs appear in `map.yaml`. |
158
+| `agent-service-discovery` | `netdata` | `master` | Service-discovery docs for the agent. |
159
+| `netdata-grafana-datasource-plugin` | `netdata` | `master` | Grafana datasource integration docs. |
160
+| `helmchart` | `netdata` | `master` | Kubernetes Helm chart docs. |
161
+
162
+Override at runtime: `--repos OWNER/REPO:BRANCH ...` or
163
+`--local-repo NAME:/path ...` (`ingest.py:2607-2613`).
164
+
165
+The legacy `ingest.js` listed only 4 repos (`netdata`, `.github`,
166
+`go.d.plugin`, `agent-service-discovery`). `go.d.plugin` is no
167
+longer pulled (its docs were absorbed into the netdata
168
+monorepo); `netdata-cloud-onprem`,
169
+`netdata-grafana-datasource-plugin`, `helmchart` are new in the
170
+Python pipeline.
171
+
172
+## CI: `${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml`
173
+
174
+### Triggers (`ingest.yml:2-18`)
175
+
176
+- `workflow_dispatch` -- manual.
177
+- `schedule: cron "10 8-23/3 * * *"` -- every 3 hours from
178
+ 08:10 to 23:10 UTC (6 runs/day).
179
+- `push` to `master` when `plugins/**`, `src/**`, `static/**`,
180
+ `sidebar.js`, `package.json`, `yarn.lock`,
181
+ `tailwind.config.js`, or any `**/*.md` / `**/*.mdx` changes
182
+ in the LEARN repo (not in source repos -- those trigger
183
+ ingest indirectly via the cron).
184
+
185
+### Steps (`ingest.yml:24-101`)
186
+
187
+1. Checkout (full history, `fetch-depth: 0`) using
188
+ `secrets.GITHUB_TOKEN`.
189
+2. SSH agent for `secrets.NETDATABOT_SSH_PRIVATE_KEY` (used to
190
+ clone private repos like `.github`).
191
+3. Python 3.10, install
192
+ `${NETDATA_REPOS_DIR}/learn/.learn_environment/ingest-requirements.txt`
193
+ (`pip`, `requests==2.33.0`, `Pillow`, `PyGithub`, `gitpython`,
194
+ `mergedeep`, `pandas`, `numpy`, `retrypy`, `pyyaml`,
195
+ `jsonschema`).
196
+4. **Run `python ingest/ingest.py --fail-links 2>&1`** --
197
+ captures output for issue creation, propagates exit code 1
198
+ (broken links) into a workflow output but does NOT fail
199
+ the workflow on broken links (only on real errors,
200
+ `ingest.yml:65-77`).
201
+5. **Update kickstart checksum** -- fetches
202
+ `https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/kickstart.sh`,
203
+ MD5s it, replaces literal `@KICKSTART_CHECKSUM@` placeholder
204
+ in `docs/Netdata Agent/Installation/Linux/Linux.mdx`.
205
+6. **`peter-evans/create-pull-request@v6.0.1`** -- opens or
206
+ updates a branch named `ingest` with title
207
+ "Ingest New Documentation", labels `ingest, automation`. The
208
+ PR is reviewed and merged manually by the team.
209
+7. If broken links were found, create or update a single open
210
+ GitHub issue with label `broken-links`.
211
+
212
+## CI: `${NETDATA_REPOS_DIR}/learn/.github/workflows/daily-learn-link-check.yml`
213
+
214
+Daily cron `00 04 * * *` -- runs
215
+`python ingest/check_learn_links.py`, which extracts every
216
+`learn_link:` URL from every `.md` / `.mdx` in `docs/`,
217
+HEAD/GET checks them, exits 1 (failing the workflow) when any
218
+404 is observed.
219
+
220
+## Deploy: Netlify
221
+
222
+There is **no GitHub-side build/deploy workflow**. Build &
223
+deploy are handled by Netlify:
224
+
225
+- README confirms Netlify deploys from `master` automatically.
226
+- Site is `netdata-docusaurus` on Netlify.
227
+- Branches `master`, `staging`, `staging1` each get their own
228
+ deploy (preview URLs).
229
+- Build env pinned by `static.toml:3-4` /
230
+ `netlify.toml [build]`: `NPM_VERSION=10.9.2`,
231
+ `NODE_VERSION=22.14.0`, `NETLIFY_USE_YARN=true`,
232
+ `NODE_OPTIONS=--max_old_space_size=4096`.
233
+- Build command: `yarn build` (Docusaurus default).
234
+- Output dir: `build/`.
235
+- Propagation time: whatever Netlify takes after a master
236
+ commit; redirects ship in `netlify.toml` so they apply at
237
+ the edge immediately upon deploy.
238
+
239
+## End-to-end timing
240
+
241
+1. Maintainer pushes to `netdata/netdata` master (via merged
242
+ PR).
243
+2. Up to 3 hours later, `${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml`
244
+ cron fires.
245
+3. Ingest opens a PR in the learn repo titled "Ingest New
246
+ Documentation".
247
+4. A maintainer reviews and merges the PR.
248
+5. Netlify auto-deploys `master`. Within minutes, the new
249
+ content is live on `learn.netdata.cloud`.
250
+
251
+So end-to-end: 0-3 hours of cron lag + manual review/merge +
252
+minutes of Netlify deploy. Plan accordingly when something
253
+must be live by a deadline -- you can also trigger the ingest
254
+manually via `workflow_dispatch` to skip the cron lag.
255
+
256
+## Local testing
257
+
258
+From within the learn repo:
259
+
260
+```bash
261
+# Install deps once.
262
+python3 -m venv venv && . venv/bin/activate
263
+pip install -r .learn_environment/ingest-requirements.txt
264
+
265
+# Test against your local netdata clone.
266
+python3 ingest/ingest.py --local-repo netdata:<repo> --ignore-on-prem-repo --fail-links-netdata
267
+
268
+# Then build the site locally.
269
+yarn install
270
+yarn start # dev server
271
+# OR
272
+yarn build # full build to ./build
273
+```
.agents/skills/learn-site-structure/pitfalls-and-gotchas.md
new
+307
@@ -0,0 +1,307 @@
1
+# Pitfalls and gotchas
2
+
3
+Every silent failure mode, dead artifact, undocumented
4
+behavior, and edge case the Learn ingest pipeline carries
5
+today. Read this BEFORE assuming the pipeline does the obvious
6
+thing.
7
+
8
+## Hard failures (exit non-zero)
9
+
10
+- **`map.yaml` schema validation** -- exit code 2
11
+ (`ingest.py:2807, 2819`).
12
+- **`--fail-links` plus broken internal links** -- exit 1 at
13
+ the very end of the run.
14
+- **Kickstart checksum step in CI** -- exit 1 if `wget` or the
15
+ placeholder substitution fails.
16
+
17
+## Soft failures / silent breakage
18
+
19
+- **Duplicate `(learn_rel_path, sidebar_label)`** -- only a
20
+ printed warning at `ingest.py:2891-2908`. On case-insensitive
21
+ filesystems (macOS/Windows), one file silently overwrites
22
+ the other. On Linux they coexist but collide in URL space.
23
+
24
+- **Missing `sidebar_label` / `learn_rel_path` in `map.yaml`
25
+ row** -- file is skipped with a printed `KeyError`;
26
+ downstream sidebar may have a hole and links that pointed to
27
+ it become broken anchors.
28
+
29
+- **`sidebar_label` containing characters that the slug
30
+ sanitizer collapses** (`'`, `:`, `/`, `(`, `)`, `,`,
31
+ backtick, repeated whitespace) -- destination filename can
32
+ collide with a sibling that differs only by punctuation.
33
+
34
+- **MDX 3 syntax bombs** -- bare `{word}` outside code is
35
+ escaped, but a single backtick line, smart quotes, or
36
+ non-paired `<...>` can still break the MDX parser. Symptoms:
37
+ "Unexpected character `}`", "Could not parse expression", or
38
+ a Docusaurus warning that becomes a blocking error in
39
+ `production` builds. The ingest only escapes `{`, not `}`,
40
+ not `<`. Bare `<word>` in body still breaks unless wrapped
41
+ in code.
42
+
43
+- **Broken anchor links** -- markdown header anchors are
44
+ recomputed by `extract_headers_from_file`
45
+ (`ingest.py:531-560`); custom anchors via `<a id="...">` may
46
+ not match this slugger and trigger anchor-mismatch warnings.
47
+
48
+- **Missing assets / images referenced by absolute path under
49
+ `/img`** -- Docusaurus `onBrokenLinks: 'warn'`
50
+ (`docusaurus.config.js:22`) and
51
+ `markdown.hooks.onBrokenMarkdownLinks: 'warn'` -- broken
52
+ links **warn but do not fail the build**, so they slip into
53
+ production.
54
+
55
+- **`<details><summary>` without newline** -- the source-side
56
+ fix at `ingest.py:1787-1788` only handles the two literal
57
+ forms it knows. Any other variant (e.g. `<details class="x">`)
58
+ breaks MDX rendering silently.
59
+
60
+- **`<= %< <->` not in code blocks** -- escaped by
61
+ `sanitize_page` only as those exact substrings; near-variants
62
+ like `< =`, `<---->`, or `< -` aren't covered.
63
+
64
+- **`meta_yaml: "<url>"` rewrite** -- silently rewrites
65
+ `custom_edit_url` for any file containing `meta_yaml: "..."`,
66
+ even if the author didn't intend it
67
+ (`ingest.py:1801-1808`).
68
+
69
+- **Frontmatter `slug:` overrides** -- frontmatter `slug:` is
70
+ taken at face value (`ingest.py:1892`); a space in a slug
71
+ breaks every link to that page. Not validated.
72
+
73
+- **Auto-grid file overwrite risk**:
74
+ `get_dir_make_file_and_recurse` will NOT overwrite an
75
+ existing `<dir>/<dir>.mdx` (`ingest.py:2493`). However, if a
76
+ directory contains exactly one published integration plus
77
+ zero non-integrations, the script special-cases that as
78
+ content and skips the grid (`ingest.py:2429-2503`).
79
+ Adding/removing files can flip a directory between "grid"
80
+ and "leaf" presentations, surprising the maintainer.
81
+
82
+- **`safe_cleanup_learn_folders` deletes ALL `.json` files
83
+ unconditionally** (`ingest.py:1061-1063`) without checking
84
+ `part_of_learn`. Hand-authored `_category_.json` in
85
+ `${NETDATA_REPOS_DIR}/learn/docs/` is wiped each run unless
86
+ ingest itself re-creates it via
87
+ `ensure_category_json_for_dirs`.
88
+
89
+## Dead code / stale artifacts
90
+
91
+### `ingest.js` is LEGACY
92
+
93
+`${NETDATA_REPOS_DIR}/learn/ingest.js` is the original Node
94
+orchestrator. It is no longer the live pipeline. The README,
95
+the active workflow, and `<repo>/docs/.map/README.md` all run
96
+`ingest/ingest.py`. **Do not edit `ingest.js`**; do not trust
97
+its behavior as a description of the current pipeline.
98
+
99
+### `ingest.md` is stale
100
+
101
+`${NETDATA_REPOS_DIR}/learn/ingest.md` documents `ingest.js`,
102
+not `ingest.py`. The `README.md` describes the Python pipeline
103
+(at lines 52-172). When in conflict, README + `ingest.py` win
104
+over `ingest.md`.
105
+
106
+### `ingest/create_grid_integration_pages.py` is empty
107
+
108
+`${NETDATA_REPOS_DIR}/learn/ingest/create_grid_integration_pages.py`
109
+is 0 bytes. The README at lines 127-128 still tells users to
110
+run it; the actual grid generation moved into
111
+`ingest.py:get_dir_make_file_and_recurse` (`ingest.py:2333-2510`).
112
+Running the empty script does nothing and produces no error.
113
+
114
+### Duplicate link-checker
115
+
116
+`${NETDATA_REPOS_DIR}/learn/scripts/check_learn_links.py`
117
+duplicates
118
+`${NETDATA_REPOS_DIR}/learn/ingest/check_learn_links.py`
119
+verbatim. Pick one as canonical; today both exist.
120
+
121
+### `search-icons.js` is dead code
122
+
123
+`${NETDATA_REPOS_DIR}/learn/search-icons.js` is a
124
+developer-only icon-picker shim that's never imported. Safe to
125
+ignore.
126
+
127
+### `.bak` workflows
128
+
129
+Three legacy workflows kept as `.bak`:
130
+- `${NETDATA_REPOS_DIR}/learn/.github/workflows/old_ingest.yml.bak`
131
+ (Node `ingest.js` cron),
132
+- `old_check-broken-links.yml.bak`,
133
+- `old_check-broken-links-external.yml.bak`,
134
+- `check-internal-links.yml.bak`.
135
+
136
+Not active. Useful as historical reference only.
137
+
138
+### `produce_gh_edit_link_for_repo` typo
139
+
140
+`ingest.py:1027-1035`: the format string is
141
+**single-quoted** (`"https://github.com/netdata/{repo}/edit/master/{file_path}"`)
142
+-- the `f` prefix is missing -- so the function returns the
143
+literal string with `{repo}` and `{file_path}` unsubstituted
144
+for non-`.github` repos. Not currently called in the live
145
+pipeline, but a real bug that would surface if the function
146
+ever gets invoked.
147
+
148
+## Versioning is effectively unused
149
+
150
+- `${NETDATA_REPOS_DIR}/learn/versioning/remove_edit_links.py`
151
+ is the only versioning helper. It is a manual prep step for
152
+ snapshotting a version: rewrite `custom_edit_url:` to
153
+ `null`. Not automated.
154
+- No `versioned_docs/`, `versioned_sidebars/`, or
155
+ `versions.json` files exist in
156
+ `${NETDATA_REPOS_DIR}/learn/`.
157
+- `package.json` has standard `docusaurus`, `start`, `build`,
158
+ `swizzle`, `deploy`, `clear`, `serve` scripts but no
159
+ version-tagging script.
160
+- Conclusion: there is one live version. Freezing one would
161
+ require manual `versioning/remove_edit_links.py` on a
162
+ snapshot, then `docusaurus docs:version <name>` (no
163
+ automation exists for this).
164
+
165
+## Netlify redirect-rule ceiling
166
+
167
+The dynamic redirect section keeps growing -- already ~12,700
168
+lines in `${NETDATA_REPOS_DIR}/learn/netlify.toml`. The
169
+`LegacyLearnCorrelateLinksWithGHURLs.json` already has ~3,490
170
+entries.
171
+
172
+**Netlify's redirect-rule limit is ~10,000 rules per site.**
173
+This repo is approaching/past that ceiling. Not noted anywhere
174
+in the live code or docs.
175
+
176
+When the limit is exceeded, Netlify deploys still succeed but
177
+some redirects stop working. Followup: either prune the
178
+catalog (drop redirects older than N months) or move the
179
+mechanism to a different layer.
180
+
181
+## Inferring repo from a local `--repos /path` argument
182
+
183
+`ingest.py:2701-2738`: `ingest.py` first matches the basename
184
+to a known repo key; if no exact match, it does a SUBSTRING
185
+match in either direction (`netdata` vs `mynetdata-fork`).
186
+This can pick the wrong repo silently.
187
+
188
+When testing locally with a fork that has an unusual name, use
189
+`--local-repo netdata:/path/to/your/fork` to force the right
190
+mapping.
191
+
192
+## Schemas accept extras silently
193
+
194
+`<repo>/docs/.map/map.schema.json` sets
195
+`additionalProperties: false`. Validation IS strict; unknown
196
+keys fail the run.
197
+
198
+But the per-file metadata block format is NOT schema-validated
199
+end-to-end; the script tolerates extra keys in the
200
+`<!--startmeta...endmeta-->` block. Stray keys pass through to
201
+the rendered frontmatter and may either be ignored by
202
+Docusaurus or, in rare cases, trigger build warnings.
203
+
204
+## `learn_link` is informational only
205
+
206
+`learn_link:` in frontmatter is NOT used by Docusaurus for
207
+routing. It is checked daily by `check_learn_links.py` (HEAD
208
+requests against `learn.netdata.cloud`) -- its purpose is to
209
+catch the case where a slug in the source-controlled file no
210
+longer matches what's actually deployed.
211
+
212
+## Logo contrast analysis makes outbound HTTP calls
213
+
214
+`ingest.py:1647-1681` makes outbound HTTP calls to
215
+`netdata.cloud/img/...` for every integration during ingest.
216
+Result is cached per URL within a run. CI runs over hundreds
217
+of logos; total network time is bounded by
218
+`LOGO_ANALYSIS_TIMEOUT = 8 s` per request. Transient network
219
+errors during ingest can cause subtle visual inconsistencies
220
+(missing data attributes on logos).
221
+
222
+## Sidebar position 0 reservation
223
+
224
+`sidebar_position: 0` is reserved for "Ask Nedi"
225
+(`ingest.py:493-512`). Any other top-level entry assigned
226
+position 0 gets re-stamped to >= 10. So you can't pin a
227
+non-`docs/ask-nedi.mdx` page to the very top of the sidebar.
228
+
229
+## Empty/unmapped directories vanish
230
+
231
+`ensure_category_json_for_dirs` (`ingest.py:357-360`) does NOT
232
+create `_category_.json` when a directory has no `.mdx` files.
233
+So if you remove all docs from a category but leave the empty
234
+dir, it disappears from the sidebar (Docusaurus drops empty
235
+dirs). To intentionally keep an empty section, you need at
236
+least one `learn_status: Published` page in it.
237
+
238
+## Slugs in source frontmatter are sticky
239
+
240
+Ingest writes `slug:` in the frontmatter for every published
241
+file. Authors who want a stable URL across renames must set
242
+`slug:` in the source `.md` (which becomes the override path
243
+at `ingest.py:1890-1895`).
244
+
245
+But the source file's frontmatter is rewritten by ingest, so
246
+`slug:` must be set BEFORE the metadata block, where it
247
+survives the `<!-- ... -->` rewrite. In practice authors
248
+rarely do this; they rely on `meta.label` and the redirect
249
+machinery instead.
250
+
251
+## Auto-redirect chain depth is unbounded
252
+
253
+Every ingest appends new entries to
254
+`LegacyLearnCorrelateLinksWithGHURLs.json`. Old redirects are
255
+re-resolved each run via `UpdateGHLinksBasedOnMap`, so a
256
+multiple-times-moved page keeps working. But the catalog grows
257
+indefinitely.
258
+
259
+## Home page depends on `ask-nedi.mdx`
260
+
261
+`${NETDATA_REPOS_DIR}/learn/src/pages/index.js:1-6` redirects
262
+`/` -> `/docs/ask-nedi`. Anyone removing or renaming
263
+`ask-nedi.mdx` (which is `part_of_learn: True` and otherwise
264
+survives ingest) breaks the site root.
265
+
266
+## Search is local, not Algolia
267
+
268
+`docusaurus.config.js:34-39` uses
269
+`@easyops-cn/docusaurus-search-local` -- client-side, hashed
270
+indexes built at `yarn build` time. There is NO Algolia
271
+DocSearch integration. Search results are not as polished, and
272
+indexing is build-time (changes propagate only on next deploy).
273
+
274
+## Posthog, GTM, gtag, Reo.dev, Nedi UI
275
+
276
+`docusaurus.config.js:179-264` loads Posthog, Google Tag
277
+Manager, Google gtag, Reo.dev, and the Nedi UI as
278
+plugins/scripts. The Nedi embed depends on
279
+`nedi.netdata.cloud/ai-agent-public.js`; if that origin is
280
+down, the chat widget fails silently but the rest of the site
281
+loads.
282
+
283
+## Custom anchor IDs
284
+
285
+`ingest.py:531-560` extracts header anchors via a slugger.
286
+Custom anchors via `<a id="..."> </a>` may not match the
287
+slugger and trigger anchor-mismatch warnings during link
288
+resolution (step 12 of the pipeline). Use `## Heading` text
289
+that produces the desired slug instead of injecting custom
290
+anchors.
291
+
292
+## OpenAPI / Swagger
293
+
294
+`${NETDATA_REPOS_DIR}/learn/static/api/` has Swagger UI files.
295
+Not part of the docs/ ingest tree. Updated separately when the
296
+agent's OpenAPI surface changes.
297
+
298
+## Forks / non-`netdata/netdata` sources
299
+
300
+The pipeline assumes `AGENT_REPO = 'netdata/netdata'`
301
+everywhere (`ingest.py:15`). Running `ingest.py` against a
302
+fork (e.g. `--repos ktsaou/netdata:master`) works, but
303
+`build_path` (`gen_docs_integrations.py:81-90`) assumes
304
+`meta_yaml` URLs start with `https://github.com/netdata/...`,
305
+so integration page URL rewriting may break. For local
306
+testing use `--local-repo netdata:/path/to/your/fork` so the
307
+edit URLs still reference `netdata/netdata` upstream.
.agents/skills/learn-site-structure/recipes/INDEX.md
new
+69
@@ -0,0 +1,69 @@
1
+# Recipes -- INDEX
2
+
3
+Step-by-step recipes for the most common tasks an author /
4
+maintainer performs against the Learn site.
5
+
6
+| Recipe | When to use |
7
+|---|---|
8
+| `add-doc-page.md` | Adding a brand-new page to Learn. |
9
+| `move-doc-page.md` | Moving a page to a different sidebar location (different parent). |
10
+| `rename-doc-page.md` | Renaming a page (changes URL slug). |
11
+| `delete-doc-page.md` | Unpublishing a page. The only recipe that requires manual surgery. |
12
+
13
+## Common preamble for all recipes
14
+
15
+Every recipe assumes the source repo (almost always
16
+`netdata/netdata`, this repo) is your edit target. The map
17
+file `<repo>/docs/.map/map.yaml` is the lever.
18
+
19
+### Quick mental model
20
+
21
+1. **What changes appears on Learn:** anything you add to /
22
+ modify in / remove from `<repo>/docs/.map/map.yaml`,
23
+ plus the corresponding `.md` files in source repos.
24
+2. **When changes appear:** ingest CI runs every 3 hours
25
+ (cron) plus on push to learn-repo master plus on manual
26
+ dispatch. So after merging your docs PR to
27
+ `netdata/netdata` master, expect a 0-3-hour delay before
28
+ the ingest PR opens in the learn repo.
29
+3. **Maintainer step in learn repo:** review and merge the
30
+ auto-opened "Ingest New Documentation" PR. Netlify
31
+ deploys within minutes.
32
+
33
+### Test locally before pushing
34
+
35
+The cheapest way to verify your `map.yaml` change works:
36
+
37
+```bash
38
+cd ${NETDATA_REPOS_DIR}/learn
39
+
40
+# Set up venv once.
41
+python3 -m venv venv && . venv/bin/activate
42
+pip install -r .learn_environment/ingest-requirements.txt
43
+
44
+# Test against your local netdata clone.
45
+python3 ingest/ingest.py --local-repo netdata:<repo> \
46
+ --ignore-on-prem-repo --fail-links-netdata
47
+```
48
+
49
+After ingest produces output in
50
+`${NETDATA_REPOS_DIR}/learn/docs/`, browse it with the dev
51
+server:
52
+
53
+```bash
54
+cd ${NETDATA_REPOS_DIR}/learn
55
+yarn start # runs Docusaurus dev server, opens browser
56
+```
57
+
58
+## When in doubt
59
+
60
+1. Read `../mapping.md` for the source-to-URL computation.
61
+2. Read `../pipeline.md` for what runs in CI.
62
+3. Read `../authoring-boundary.md` to confirm where to edit.
63
+4. Read `../pitfalls-and-gotchas.md` BEFORE assuming the
64
+ pipeline does the obvious thing.
65
+5. If you encountered a question that this catalog doesn't
66
+ cover and you had to investigate to answer it, AUTHOR a
67
+ how-to under `../how-tos/<slug>.md` and add it to
68
+ `../how-tos/INDEX.md`. This rule is mandatory; see
69
+ `../SKILL.md` "Live how-to rule".
.agents/skills/learn-site-structure/recipes/add-doc-page.md
new
+124
@@ -0,0 +1,124 @@
1
+# Recipe: add a new doc page to Learn
2
+
3
+Add a new published page on `learn.netdata.cloud` from the
4
+agent repo (this repo) or any of the 6 source repos.
5
+
6
+## 0. Read first
7
+
8
+- `../SKILL.md` -- skill overview.
9
+- `../mapping.md` -- the map.yaml schema and URL computation.
10
+
11
+## 1. Create the source markdown
12
+
13
+Place the file under a reasonable location in the source repo.
14
+Path is largely cosmetic; what matters is the `map.yaml` row.
15
+
16
+For this repo, suggested locations:
17
+- `<repo>/docs/<existing-section>/<your-page>.md` for general
18
+ user-facing docs;
19
+- `<repo>/docs/developer-and-contributor-corner/<page>.md` for
20
+ developer notes;
21
+- `<repo>/docs/netdata-agent/<page>.md` for agent docs;
22
+- `<repo>/docs/dashboards-and-charts/<page>.md` for dashboard
23
+ docs.
24
+
25
+Author the markdown content normally. No special frontmatter
26
+required; ingest will inject what it needs.
27
+
28
+If you absolutely must pin a stable URL across renames, add
29
+`slug: /your/stable/path/here` at the very top of the file
30
+(before any HTML-comment metadata block) -- but this is rare
31
+and usually unnecessary because the auto-redirect mechanism
32
+handles renames.
33
+
34
+## 2. Add a node to map.yaml
35
+
36
+Open `<repo>/docs/.map/map.yaml` and add a row under the
37
+appropriate parent:
38
+
39
+```yaml
40
+- meta:
41
+ label: My New Page
42
+ edit_url: https://github.com/netdata/netdata/edit/master/docs/<section>/<your-page>.md
43
+ description: Optional one-line description for SEO.
44
+ keywords: [keyword1, keyword2]
45
+```
46
+
47
+The position of the row in the YAML determines the sidebar
48
+position within the parent (sibling-relative; ingest assigns
49
+`sidebar_position` in steps of 10 in traversal order).
50
+
51
+If you want a single-segment override of the URL slug
52
+(different from the label), add `path: <slug>` under the
53
+`meta` block.
54
+
55
+## 3. Test locally
56
+
57
+From the learn repo:
58
+
59
+```bash
60
+cd ${NETDATA_REPOS_DIR}/learn
61
+. venv/bin/activate # if not already
62
+python3 ingest/ingest.py --local-repo netdata:<repo> \
63
+ --ignore-on-prem-repo --fail-links-netdata
64
+yarn start # browse at http://localhost:3000
65
+```
66
+
67
+Check:
68
+- Your new page appears in the sidebar under the expected
69
+ parent.
70
+- The URL is what you expected
71
+ (`/docs/<learn_rel_path>/<sidebar_label>` lowercased).
72
+- Links from / to your page resolve.
73
+
74
+## 4. Open the docs PR in this repo
75
+
76
+Single PR with two commits (or one):
77
+1. The new `.md` file under `<repo>/docs/`.
78
+2. The map.yaml change adding the node.
79
+
80
+Reviewers check:
81
+- The map.yaml row has all required fields (`label`,
82
+ `edit_url`).
83
+- The `edit_url` matches the schema regex
84
+ (`^https://github\.com/netdata/<repo>/edit/<branch>/.+\.(md|mdx)$`).
85
+- The page is in a sensible parent in the tree.
86
+
87
+## 5. After merge
88
+
89
+`${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml`
90
+fires within 3 hours (or you can `workflow_dispatch` it
91
+immediately). Ingest opens an "Ingest New Documentation" PR
92
+in the learn repo. A learn-repo maintainer reviews and merges
93
+it. Netlify deploys within minutes.
94
+
95
+End-to-end timing: 0-3 hours of cron + manual review/merge +
96
+minutes of Netlify deploy.
97
+
98
+## 6. Verify on production
99
+
100
+After deploy, check:
101
+- `https://learn.netdata.cloud/docs/<your-rel-path>/<your-label>`
102
+ resolves.
103
+- The page renders correctly (no MDX errors).
104
+- Sidebar position is what you expected.
105
+- Page metadata (title, description, keywords) is populated
106
+ from your `meta` block.
107
+
108
+## Common mistakes
109
+
110
+- **Forgetting the map.yaml row.** The page won't be
111
+ published. There's no warning -- ingest silently skips
112
+ source files not in the map.
113
+- **Wrong `edit_url` format.** Schema validation fails the
114
+ whole ingest run with exit code 2. Check the regex.
115
+- **Source path implies the URL.** It does NOT. The URL is
116
+ computed from `meta.label` + `learn_rel_path`. Don't expect
117
+ the source filesystem path to influence routing.
118
+- **Hand-editing `${NETDATA_REPOS_DIR}/learn/docs/<page>.mdx`
119
+ directly.** Wiped on next ingest. Use `part_of_learn: True`
120
+ if you really need a hand-authored exception.
121
+- **Wrong source repo for the content.** If your content is
122
+ about Netdata Cloud On-Prem, edit
123
+ `${NETDATA_REPOS_DIR}/netdata-cloud-onprem/`, not this repo.
124
+ But the map.yaml row goes in THIS repo.
.agents/skills/learn-site-structure/recipes/delete-doc-page.md
new
+123
@@ -0,0 +1,123 @@
1
+# Recipe: delete (unpublish) a doc page
2
+
3
+The only recipe that requires **manual surgery** in the learn
4
+repo. Deletion does NOT auto-redirect -- without the manual
5
+step, the old URL serves a 404.
6
+
7
+The canonical reference for this recipe is
8
+`<repo>/docs/.map/README.md:96-104`.
9
+
10
+## 1. Decide where deleted-link traffic should go
11
+
12
+Before deleting, decide one of:
13
+
14
+- **Redirect to a closest replacement page.** Best for SEO
15
+ and external link preservation. Pick a page whose content
16
+ largely replaces the deleted one.
17
+- **Drop the link entirely (404).** Acceptable for pages with
18
+ little inbound traffic that have no good replacement.
19
+
20
+You'll apply the decision in step 4.
21
+
22
+## 2. Delete the source file and remove the map.yaml node
23
+
24
+In this repo:
25
+
26
+1. Delete the source `.md` file.
27
+2. Open `<repo>/docs/.map/map.yaml` and remove the matching
28
+ node.
29
+3. Open the docs PR.
30
+
31
+After this PR merges, ingest's next run produces a "Ingest
32
+New Documentation" PR in the learn repo that DOES NOT
33
+auto-redirect the old URL (because the GH source URL no
34
+longer points to a real file, the `UpdateGHLinksBasedOnMap`
35
+step can't resolve it).
36
+
37
+## 3. Open the learn-repo manual-surgery PR
38
+
39
+Once the ingest PR is merged (or BEFORE if you want to bundle
40
+the surgery with the ingest PR -- see step 4 alternative):
41
+
42
+1. Open `${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json`.
43
+2. Search for the GitHub blob/edit URL of the deleted file:
44
+ ```
45
+ "https://github.com/netdata/netdata/blob/master/docs/<...>/<deleted-page>.md"
46
+ ```
47
+3. Find the entry. Apply your decision from step 1:
48
+ - **Redirect to replacement**: change the value to the
49
+ full Learn URL of the replacement page
50
+ (`https://learn.netdata.cloud/docs/<replacement>`).
51
+ - **Drop**: delete the entry from the JSON entirely.
52
+
53
+Save the file.
54
+
55
+Alternative: if you want a one-off manual `[[redirects]]`
56
+rule (for example, to redirect to an entirely external
57
+domain), edit `${NETDATA_REPOS_DIR}/learn/static.toml` and
58
+add a hand-rolled entry under `# section: static`. This
59
+gets copied into `netlify.toml` on the next ingest.
60
+
61
+## 4. Open a learn-repo PR
62
+
63
+Title: `manual redirect surgery for deleted page <name>`. PR
64
+body explains what was deleted, where the redirect goes, and
65
+why.
66
+
67
+## 5. Merge and verify
68
+
69
+After Netlify deploys:
70
+
71
+```bash
72
+# If you redirected to replacement:
73
+curl -sI https://learn.netdata.cloud<old-path>
74
+# Expect: HTTP/2 301
75
+# Location: https://learn.netdata.cloud<replacement-path>
76
+
77
+# If you dropped:
78
+curl -sI https://learn.netdata.cloud<old-path>
79
+# Expect: HTTP/2 404
80
+```
81
+
82
+## Notes
83
+
84
+- **Daily link checker**: the daily
85
+ `${NETDATA_REPOS_DIR}/learn/.github/workflows/daily-learn-link-check.yml`
86
+ cron will start failing if it finds a `learn_link:`
87
+ pointing at the deleted page. To minimize noise, do step 3
88
+ promptly after step 2.
89
+
90
+- **Internal links from other Learn pages**: search
91
+ `${NETDATA_REPOS_DIR}/learn/docs/` and this repo's `<repo>/docs/`
92
+ for explicit links to the deleted page's URL and update them.
93
+ The link rewriter (step 12 of the ingest pipeline) will
94
+ flag broken internal links during ingest if you missed any.
95
+
96
+- **If you re-publish later**: the redirect entry can stay
97
+ in `LegacyLearnCorrelateLinksWithGHURLs.json`. The
98
+ `UpdateGHLinksBasedOnMap` step will re-resolve the GH URL
99
+ through the new map and route the old URL to the new
100
+ destination automatically. So if you later add a similar
101
+ page back, the old URL keeps working with no further
102
+ surgery.
103
+
104
+## Common mistakes
105
+
106
+- **Forgetting the manual JSON surgery.** Without it, the
107
+ deleted page's old URL serves a 404. External links break
108
+ silently for users.
109
+- **Editing `netlify.toml` directly.** Regenerated each
110
+ ingest. Edit `static.toml` (for hand-curated static rules)
111
+ or `LegacyLearnCorrelateLinksWithGHURLs.json` (for the
112
+ dynamic catalog).
113
+- **Deleting a `part_of_learn: True` page.** That's a
114
+ hand-authored learn-repo page (currently only
115
+ `docs/ask-nedi.mdx`). Deleting it requires editing the
116
+ learn repo directly, NOT this repo's map.yaml. And the
117
+ home page redirects to it -- removing it breaks the site
118
+ root.
119
+- **Deleting an integration page.** Integration pages are
120
+ generated by the integrations pipeline. Don't delete the
121
+ generated `.md` directly. Edit the source `metadata.yaml`
122
+ to remove the integration (or change its category) -- see
123
+ the `integrations-lifecycle` skill.
.agents/skills/learn-site-structure/recipes/move-doc-page.md
new
+102
@@ -0,0 +1,102 @@
1
+# Recipe: move a doc page (different sidebar location)
2
+
3
+Move a page from one sidebar location to another. The source
4
+file stays where it is; only its position in `map.yaml`
5
+changes. Ingest auto-generates a redirect from the old URL to
6
+the new one.
7
+
8
+## 0. Pre-check
9
+
10
+Confirm the page is a regular doc (`.md`), not an integration
11
+page. Integration pages flow through the integrations
12
+pipeline; their position is driven by their `meta.categories`,
13
+not by `map.yaml`. To move an integration page, you change
14
+the integration's category in its `metadata.yaml` (see the
15
+`integrations-lifecycle` skill).
16
+
17
+## 1. Edit map.yaml
18
+
19
+Open `<repo>/docs/.map/map.yaml`.
20
+
21
+1. Find the existing node for the page.
22
+2. Cut it from the old location and paste it under the new
23
+ parent.
24
+3. Keep `meta.edit_url` UNCHANGED -- it still points to the
25
+ same source file.
26
+4. Optionally adjust `meta.label` if the new location calls
27
+ for a different display name (this changes the URL slug
28
+ too -- in that case it's effectively also a rename).
29
+
30
+## 2. Test locally
31
+
32
+```bash
33
+cd ${NETDATA_REPOS_DIR}/learn
34
+. venv/bin/activate
35
+python3 ingest/ingest.py --local-repo netdata:<repo> \
36
+ --ignore-on-prem-repo --fail-links-netdata
37
+```
38
+
39
+Check:
40
+- The page appears under the new parent in the sidebar.
41
+- A redirect entry was added to `netlify.toml` and
42
+ `LegacyLearnCorrelateLinksWithGHURLs.json` from the old URL
43
+ to the new target.
44
+
45
+```bash
46
+grep -A1 "<old-url>" netlify.toml LegacyLearnCorrelateLinksWithGHURLs.json
47
+```
48
+
49
+## 3. Open the docs PR
50
+
51
+Single map.yaml change. The source `.md` file is unchanged.
52
+
53
+## 4. After merge
54
+
55
+Same as add-doc-page: 0-3 hour cron + ingest PR + manual
56
+merge + Netlify deploy. The auto-generated redirect ships
57
+with the ingest PR.
58
+
59
+## 5. Verify
60
+
61
+After deploy:
62
+- The page is at the new URL.
63
+- The OLD URL redirects to the new one (HTTP 301 from
64
+ Netlify edge).
65
+
66
+```bash
67
+curl -sI https://learn.netdata.cloud<old-path>
68
+# Expect: HTTP/2 301
69
+# Location: https://learn.netdata.cloud<new-path>
70
+```
71
+
72
+## Notes
73
+
74
+- **Auto-redirect is via the GitHub blob URL**, not via the
75
+ old Learn URL directly. The catalog is anchored to
76
+ `github.com/netdata/netdata/blob/master/<source-path>` so
77
+ multiple consecutive moves keep working as long as the
78
+ source file still exists.
79
+
80
+- **If you also rename the file in source**, treat that as a
81
+ separate move + you must update `meta.edit_url`. The diff
82
+ mechanism still works because the OLD `meta.edit_url` value
83
+ is what gets stored as the redirect's GH URL key.
84
+
85
+- **`slug:` overrides bypass the diff mechanism.** If the
86
+ page has a frontmatter `slug:`, moving the map.yaml row
87
+ does NOT change the URL, so no redirect is generated. To
88
+ move a slug-override page, edit the slug AND ensure a
89
+ manual entry lands in
90
+ `LegacyLearnCorrelateLinksWithGHURLs.json` (similar to the
91
+ delete recipe).
92
+
93
+## Common mistakes
94
+
95
+- **Editing `meta.edit_url` when only moving.** The edit_url
96
+ should NOT change for a move (the source file stays put).
97
+- **Editing the source file path AND the map.yaml row in the
98
+ same PR without updating `meta.edit_url`.** Will produce
99
+ validation failures because the schema regex requires the
100
+ edit_url to match an existing source file.
101
+- **Expecting the OLD URL to disappear.** It auto-redirects
102
+ forever. Don't break links pointing to it.
.agents/skills/learn-site-structure/recipes/rename-doc-page.md
new
+106
@@ -0,0 +1,106 @@
1
+# Recipe: rename a doc page
2
+
3
+Rename a page (change its display name in the sidebar / URL).
4
+Functionally identical to a move because the URL slug is
5
+derived from `meta.label`. Auto-redirected.
6
+
7
+## 1. Edit map.yaml
8
+
9
+Open `<repo>/docs/.map/map.yaml`. Find the node and change
10
+`meta.label`:
11
+
12
+```yaml
13
+# Before
14
+- meta:
15
+ label: Old Name
16
+ edit_url: https://github.com/netdata/netdata/edit/master/docs/x/page.md
17
+
18
+# After
19
+- meta:
20
+ label: New Name
21
+ edit_url: https://github.com/netdata/netdata/edit/master/docs/x/page.md
22
+```
23
+
24
+`edit_url` stays unchanged (source file is still the same
25
+file).
26
+
27
+## 2. Optionally rename the source file too
28
+
29
+If you want the source file to also have a new name (cleaner),
30
+rename it in this repo AND update `meta.edit_url` to match.
31
+Two scenarios:
32
+
33
+- **Just label change**: edit_url unchanged. Auto-redirect
34
+ works; old URL maps to old GH URL maps (via the now-current
35
+ map) to new URL.
36
+- **Label + source filename change**: edit_url changes. The
37
+ diff mechanism keys on edit_url, so the OLD edit_url is
38
+ what stores the redirect key. After the rename, the OLD
39
+ edit_url no longer points to a real source file in the
40
+ current commit -- but
41
+ `LegacyLearnCorrelateLinksWithGHURLs.json` still has it
42
+ pointing at the new URL via the previous-run snapshot.
43
+ This works for ONE rename cycle. If you rename again later,
44
+ the chain may break -- so prefer to keep edit_url stable.
45
+
46
+## 3. Test locally
47
+
48
+```bash
49
+cd ${NETDATA_REPOS_DIR}/learn
50
+. venv/bin/activate
51
+python3 ingest/ingest.py --local-repo netdata:<repo> \
52
+ --ignore-on-prem-repo --fail-links-netdata
53
+```
54
+
55
+Check that the auto-redirect is generated. See
56
+`../redirects.md`.
57
+
58
+## 4. Open the docs PR
59
+
60
+map.yaml change (and source file rename if applicable).
61
+Reviewers check the `edit_url` regex and that the rename
62
+makes sense.
63
+
64
+## 5. After merge
65
+
66
+Same flow as move/add: 0-3 hour cron + ingest PR + manual
67
+merge + Netlify deploy.
68
+
69
+## 6. Verify
70
+
71
+```bash
72
+# Old URL redirects to new
73
+curl -sI https://learn.netdata.cloud<old-path> # expect 301
74
+# New URL works
75
+curl -sI https://learn.netdata.cloud<new-path> # expect 200
76
+```
77
+
78
+## Notes
79
+
80
+- **Slug computation refresher**: the slug = lowercase
81
+ `<learn_rel_path>/<sanitized-label>`, with spaces -> `-`,
82
+ `//` -> `/`. So `Old Name` -> `old-name`, `New Name` ->
83
+ `new-name`.
84
+- **The filename in the learn repo also changes** because the
85
+ destination filename is derived from `sidebar_label` (with
86
+ case preserved + only certain chars stripped). So
87
+ `${NETDATA_REPOS_DIR}/learn/docs/x/Old Name.mdx` becomes
88
+ `New Name.mdx`. This is normal and ingest handles it.
89
+- **External links to the OLD URL keep working** thanks to
90
+ the auto-redirect. But it's still good practice to update
91
+ any first-party references (within
92
+ `<repo>/docs/...`).
93
+
94
+## Common mistakes
95
+
96
+- **Renaming `meta.label` to something with `/`, `(`, `)`,
97
+ `,`, `'`, backtick, or `:`** -- those characters get
98
+ stripped by the slug sanitizer. Pick a label that survives
99
+ cleanup.
100
+- **Forgetting that the URL changes.** A rename IS a URL
101
+ change. The auto-redirect mitigates external link breakage,
102
+ but the URL surface itself moves.
103
+- **Renaming a page with `slug:` override in source
104
+ frontmatter.** The auto-redirect mechanism is bypassed
105
+ (slug override wins). You'll need manual JSON surgery
106
+ similar to the delete recipe.
.agents/skills/learn-site-structure/redirects.md
new
+206
@@ -0,0 +1,206 @@
1
+# Redirects
2
+
3
+Learn supports four redirect mechanisms layered in precedence
4
+order. Most page moves and renames produce **automatic**
5
+redirects. Page deletions require **manual** surgery.
6
+
7
+## The four mechanisms
8
+
9
+| # | Mechanism | Precedence | Authoritative? | Where configured |
10
+|---|---|---|---|---|
11
+| 1 | Netlify `[[redirects]]` (in `netlify.toml`) | edge -- before SPA loads | YES | Generated end-to-end by `ingest/autogenerateRedirects.main` each ingest |
12
+| 2 | Docusaurus `@docusaurus/plugin-client-redirects` | client-side, after SPA loads | client-only | `docusaurus.config.js:167-178` |
13
+| 3 | React root redirect | client-side fallback | hand-coded | `${NETDATA_REPOS_DIR}/learn/src/pages/index.js:1-6` (`/` -> `/docs/ask-nedi`) |
14
+| 4 | Frontmatter `redirect_from` | client-side, Docusaurus-native | NOT used | grep yields zero hits in `docs/` |
15
+
16
+So mechanism 1 is the workhorse. Mechanisms 2 and 3 are
17
+limited and supplemental. Mechanism 4 is theoretically
18
+supported but not used here.
19
+
20
+## Mechanism 1: Netlify edge redirects (the workhorse)
21
+
22
+`${NETDATA_REPOS_DIR}/learn/netlify.toml` is REGENERATED end
23
+to end by `ingest/autogenerateRedirects.main`
24
+(`autogenerateRedirects.py:172-213`) on every ingest run. It
25
+has two sections; both are regenerated each ingest:
26
+
27
+1. **Static section** (`# section: static << START / END`) --
28
+ copied verbatim from
29
+ `${NETDATA_REPOS_DIR}/learn/static.toml` (hand-curated).
30
+ Currently 28 rules covering legacy `/docs/agent/...` and
31
+ `/docs/nightly/...` paths, the `/guides -> /docs`
32
+ migration, and a handful of category remaps (e.g.
33
+ `kubernetes-k8s-netdata -> /docs/collecting-metrics/kubernetes`).
34
+
35
+2. **Dynamic section** (`# section: dynamic << START / END`)
36
+ -- built from
37
+ `${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json`
38
+ joined with the just-finished ingest's
39
+ `(GH URL -> Learn URL)` map. Currently many hundreds of
40
+ rules; the file is ~12,700 lines.
41
+
42
+Both `netlify.toml` (the live published file) and
43
+`LegacyLearnCorrelateLinksWithGHURLs.json` are committed to
44
+git on each ingest PR.
45
+
46
+### Auto-redirects on move/rename
47
+
48
+When a node moves in `<repo>/docs/.map/map.yaml` (or its
49
+`meta.label` changes -- which changes the slug), the diff-based
50
+`addMovedRedirects` (`autogenerateRedirects.py:124-155`):
51
+
52
+1. Builds the current `(custom_edit_url -> new_learn_path)`
53
+ mapping.
54
+2. Compares with the previous-run snapshot at
55
+ `ingest/one_commit_back_file-dict.yaml`.
56
+3. For every `custom_edit_url` whose target moved, adds
57
+ `https://learn.netdata.cloud<old_path> -> <github_blob_url>`
58
+ to the redirect set.
59
+4. `append_entries_to_json` writes the new entries to
60
+ `LegacyLearnCorrelateLinksWithGHURLs.json`.
61
+5. `UpdateGHLinksBasedOnMap` reads ALL entries from
62
+ `LegacyLearnCorrelateLinksWithGHURLs.json` (legacy + just
63
+ appended), looks up their current Learn target via the
64
+ `mapping`, and folds them into the dynamic-redirect
65
+ section.
66
+
67
+### Indirection: GH-URL anchored
68
+
69
+The redirect store is anchored to the GitHub blob URL of the
70
+source file, NOT to the old Learn URL directly. So a moved
71
+page's old URL keeps redirecting forever, even after several
72
+subsequent moves -- as long as the source file still exists
73
+somewhere in `map.yaml`. Each ingest re-resolves
74
+`GH URL -> current Learn URL` via the live mapping.
75
+
76
+If the source file is deleted (the GH URL no longer points to
77
+real content), the redirect resolves to a missing URL and
78
+serves a 404 unless someone manually edits the JSON. See the
79
+delete recipe.
80
+
81
+## Mechanism 2: Docusaurus client redirects
82
+
83
+`${NETDATA_REPOS_DIR}/learn/docusaurus.config.js:167-178`:
84
+
85
+```js
86
+redirects: [{ from: '/docs/ask-netdata', to: '/docs/ask-nedi' }]
87
+```
88
+
89
+Currently one entry. These are client-side redirects rendered
90
+as static HTML stubs by the Docusaurus build. Used for in-app
91
+links the SPA might encounter.
92
+
93
+## Mechanism 3: React root redirect
94
+
95
+`${NETDATA_REPOS_DIR}/learn/src/pages/index.js:1-6` -- the home
96
+page is a React component that redirects `/` -> `/docs/ask-nedi`.
97
+
98
+This is the LAST FALLBACK for the homepage. If you remove or
99
+rename `ask-nedi.mdx` (which is `part_of_learn: True` and
100
+otherwise survives ingest), you break the site root.
101
+
102
+## Mechanism 4: frontmatter `redirect_from` -- not used
103
+
104
+Docusaurus supports `redirect_from` in frontmatter. A grep
105
+across `${NETDATA_REPOS_DIR}/learn/docs/` returns zero hits.
106
+Don't use this mechanism here -- it's not part of the
107
+established conventions.
108
+
109
+## Order of precedence
110
+
111
+Server-then-client:
112
+
113
+1. Netlify edge rules (mechanism 1) catch the request before
114
+ anything else.
115
+2. If the URL passes through, Docusaurus client redirects
116
+ (mechanism 2) handle in-app navigation.
117
+3. The React root redirect (mechanism 3) is the last fallback
118
+ for the homepage.
119
+
120
+A redirect from one mechanism cannot override a redirect in a
121
+higher-priority mechanism. So an `/docs/old-page` URL caught by
122
+the dynamic Netlify section never reaches the Docusaurus
123
+client redirect plugin.
124
+
125
+## What happens when a page moves
126
+
127
+The diff-based mechanism handles it automatically:
128
+
129
+1. Author updates `<repo>/docs/.map/map.yaml`.
130
+2. Docs PR merges to `master`.
131
+3. The next ingest cycle (within 3 hours) detects the diff and
132
+ adds a redirect to `LegacyLearnCorrelateLinksWithGHURLs.json`
133
+ and `netlify.toml`.
134
+4. Old URL keeps working forever.
135
+
136
+## What happens when a page is renamed
137
+
138
+Same as a move (the URL slug is derived from `sidebar_label`,
139
+so changing `meta.label` is functionally a move).
140
+
141
+## What happens when a page is deleted
142
+
143
+The author MUST manually update
144
+`${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json`
145
+per `<repo>/docs/.map/README.md:96-104`. The recipe:
146
+
147
+1. Delete the source `.md` and remove the matching node from
148
+ `map.yaml`. Open the docs PR.
149
+2. Open `LegacyLearnCorrelateLinksWithGHURLs.json` in the
150
+ learn repo.
151
+3. Search for the GitHub blob/edit URL of the deleted file.
152
+4. Either:
153
+ - Update its value to the closest live replacement on Learn
154
+ (so old links redirect somewhere useful).
155
+ - Or delete the entry (resulting in a 404 for the old URL).
156
+5. Optionally add a manual entry in `static.toml` under
157
+ `# section: static` for hard-coded one-off redirects.
158
+
159
+Without this manual step, the redirect resolves to a missing
160
+URL and serves a 404. See `recipes/delete-doc-page.md`.
161
+
162
+## How redirects are committed
163
+
164
+Each ingest run commits two files to the learn-repo PR:
165
+
166
+- `netlify.toml` (always regenerated end-to-end).
167
+- `LegacyLearnCorrelateLinksWithGHURLs.json` (appended to;
168
+ preserves legacy entries, adds new diff entries).
169
+
170
+Both are inspected during the manual review of the
171
+`Ingest New Documentation` PR.
172
+
173
+## Static `static.toml` redirects
174
+
175
+The hand-curated static section of `static.toml` is for
176
+one-off redirects that the diff mechanism wouldn't catch:
177
+
178
+- Cross-domain migrations (e.g. legacy `/docs/agent/...` ->
179
+ current paths).
180
+- Category remaps (e.g.
181
+ `/docs/collecting-metrics/kubernetes-k8s-netdata` ->
182
+ `/docs/collecting-metrics/kubernetes`).
183
+- The `/guides` -> `/docs` migration.
184
+
185
+Edit `static.toml` directly to add a static redirect; it gets
186
+copied into `netlify.toml` on the next ingest run.
187
+
188
+## Risks
189
+
190
+- **Unbounded redirect growth.** Every ingest appends new
191
+ entries to `LegacyLearnCorrelateLinksWithGHURLs.json`. The
192
+ dynamic redirect section keeps growing -- already ~12,700
193
+ lines. Netlify's redirect-rule limit is ~10,000 rules per
194
+ site. The repo is approaching/past that ceiling. Not noted
195
+ anywhere in the live code or docs. Followup item.
196
+- **Forgotten deletions.** Deleting a source file without
197
+ manual JSON surgery results in a 404 on the old URL with no
198
+ warning. The daily link checker
199
+ (`daily-learn-link-check.yml`) will flag it, but only after
200
+ publication.
201
+- **Slug overrides bypass the diff mechanism.** If a file has
202
+ `slug:` in frontmatter, that value wins. Renaming the file
203
+ or moving its `map.yaml` row does NOT change the URL, so no
204
+ redirect is generated. To rename a slug-override page, edit
205
+ the slug AND ensure the redirect catalog gets a manual
206
+ entry.
.agents/skills/learn-site-structure/sidebars.md
new
+136
@@ -0,0 +1,136 @@
1
+# Sidebars
2
+
3
+The Learn site sidebar is **autogenerated** from the
4
+filesystem. Order is driven by `sidebar_position` frontmatter
5
+that ingest assigns from `map.yaml` traversal order.
6
+
7
+## `sidebars.js` is one line
8
+
9
+`${NETDATA_REPOS_DIR}/learn/sidebars.js:12-19`:
10
+
11
+```js
12
+module.exports = { sidebar: [{ "type": "autogenerated", "dirName": "." }] };
13
+```
14
+
15
+Docusaurus walks `docs/` and builds the entire sidebar from
16
+filesystem structure plus per-file frontmatter. There is NO
17
+manual sidebar list.
18
+
19
+## Ordering rules
20
+
21
+Per Docusaurus, the displayed order in each parent scope is
22
+controlled by:
23
+
24
+1. **Frontmatter `sidebar_label`** -- set by ingest from
25
+ `map.yaml`'s `meta.label`.
26
+
27
+2. **Frontmatter `sidebar_position`** -- set by
28
+ `automate_sidebar_position` (`ingest.py:2238-2299`) and then
29
+ re-normalized per parent scope by
30
+ `normalize_sidebar_positions_by_parent` (`ingest.py:425-519`).
31
+ Values are assigned in steps of 10 within each parent scope;
32
+ map-derived order wins, then alphabetical fallback.
33
+
34
+3. **`_category_.json`** (Docusaurus convention) -- written by
35
+ `ensure_category_json_for_dirs` (`ingest.py:336-382`) for
36
+ any subdirectory that contains `.mdx` files but no overview
37
+ file. Format: `{label, position}`. Currently only one in
38
+ the live tree: `docs/Collecting Metrics/OpenTelemetry/_category_.json`.
39
+
40
+4. **Section overview pages** -- when a directory has
41
+ `<dir>/<dir>.mdx`, it acts as the category landing page
42
+ (Docusaurus convention) AND is routed to the parent URL via
43
+ the duplicate-segment slug trim (`mapping.md`).
44
+
45
+5. **Auto-generated grid pages** --
46
+ `get_dir_make_file_and_recurse` (`ingest.py:2333-2510`)
47
+ creates `<dir>/<dir>.mdx` for any directory that contains
48
+ integration pages and has no overview file. The generated
49
+ MDX uses `<Grid columns="4">` with `<Box ...>` children
50
+ (the `Grid_integrations` component), and carries
51
+ `learn_status: AUTOGENERATED` plus computed `slug:` and
52
+ `learn_link:`.
53
+
54
+## Special rule: "Ask Nedi" pinned to position 0
55
+
56
+`automate_sidebar_position` (`ingest.py:493-512`) forces a
57
+top-level page whose `sidebar_label` is "Ask Nedi" to position
58
+0. The "Ask Nedi" entrypoint is
59
+`${NETDATA_REPOS_DIR}/learn/docs/ask-nedi.mdx`. Other top-level
60
+entries cannot occupy position 0; they get >= 10.
61
+
62
+`docs/ask-nedi.mdx` is the only file currently flagged
63
+`part_of_learn: True` -- meaning it's hand-authored in the
64
+learn repo and survives the cleanup step. See
65
+`authoring-boundary.md`.
66
+
67
+## Section landing page convention
68
+
69
+When a directory has a file matching `<dir>/<dir>.mdx`:
70
+
71
+- Docusaurus treats it as the category landing page.
72
+- `add_new_learn_path_key_to_dict` (`ingest.py:1862-1876`)
73
+ removes the duplicate dir segment from the URL.
74
+- So `docs/Collecting Metrics/Collecting Metrics.mdx` is
75
+ routed to `/docs/collecting-metrics` (parent URL), not
76
+ `/docs/collecting-metrics/collecting-metrics`.
77
+
78
+This is how every category gets a landing page that displays
79
+when the user clicks the parent in the sidebar.
80
+
81
+For directories WITHOUT an overview file, a `_category_.json`
82
+is generated to provide a label and position (otherwise
83
+Docusaurus uses the directory name verbatim, which often has
84
+poor casing/spacing).
85
+
86
+## Auto-grid generation
87
+
88
+`get_dir_make_file_and_recurse` (`ingest.py:2333-2510`) walks
89
+`docs/` after publication. For directories that contain
90
+integration pages (matched by their `<!--startmeta` blocks)
91
+but lack a `<dir>/<dir>.mdx` overview file:
92
+
93
+1. Generate an MDX file with a `<Grid_integrations>` component
94
+ listing the integrations in that directory.
95
+2. `learn_status: AUTOGENERATED` is set in the frontmatter.
96
+3. `slug:` and `learn_link:` are computed and injected.
97
+
98
+If a directory contains EXACTLY one published integration plus
99
+zero non-integration content, the script special-cases that as
100
+content-leaf and skips the grid (`ingest.py:2429-2503`).
101
+Adding/removing files can flip a directory between "grid" and
102
+"leaf" presentations. See `pitfalls-and-gotchas.md`.
103
+
104
+## Top-level vs nested
105
+
106
+There is no structural difference between top-level and nested
107
+sections, except for the "Ask Nedi" position-0 rule, which
108
+applies only to the root scope. All other scopes are ordered by
109
+map traversal first, alphabetical second.
110
+
111
+## To reorder the sidebar
112
+
113
+You do NOT edit `sidebars.js`. You reorder the rows in
114
+`<repo>/docs/.map/map.yaml`. Ingest re-runs
115
+`automate_sidebar_position`, which assigns new
116
+`sidebar_position` values. Docusaurus rebuilds with the new
117
+order on the next ingest cycle.
118
+
119
+## Common mistakes
120
+
121
+- **Editing `sidebars.js` directly.** The file is one line and
122
+ not consulted for ordering. Useless.
123
+- **Editing `_category_.json` files in the learn repo.** These
124
+ get overwritten on the next ingest run (`safe_cleanup_learn_folders`
125
+ deletes ALL `.json` files unconditionally -- not just
126
+ non-`part_of_learn` ones; see `pitfalls-and-gotchas.md`).
127
+ The right fix is to put the page in `map.yaml` so the
128
+ sidebar position comes from there.
129
+- **Setting `sidebar_position: 0` in the source frontmatter.**
130
+ It is overwritten by ingest's `automate_sidebar_position`.
131
+ And the value 0 is reserved for "Ask Nedi" at the top level
132
+ -- any conflict gets re-stamped to >= 10.
133
+- **Empty directory expectations.** Empty directories
134
+ disappear from the sidebar (Docusaurus drops them). Removing
135
+ the last doc in a category removes the category from the
136
+ sidebar.
.agents/skills/mirror-netdata-repos/SKILL.md
new
+243
@@ -0,0 +1,243 @@
1
+---
2
+name: mirror-netdata-repos
3
+description: Maintains a local mirror of Netdata-org source repositories at `${NETDATA_REPOS_DIR}` so AI assistants and developers can do cross-repo grep / code review locally without GitHub API round-trips and rate limits. Ships a vendored sync script (`scripts/sync-netdata-repos.sh`) that updates ~150 repos in two phases (resync existing on default branch, discover and clone new). Safety -- skips repos that have staged or modified changes; otherwise switches to the default branch and recursively updates submodules. Reset-to-default is intentional -- it prevents stale-feature-branch "black hole" repos that confuse cross-repo reasoning. Supports `--repo NAME` (repeatable) to scope to specific repos. Independent from any other repo mirrors this workstation may have. Use when the local mirror is out of date, before a cross-repo grep / review session, when adding a new netdata-org repo (auto-discovered), when an assistant needs cross-repo cognition without `gh` API turnaround.
4
+---
5
+
6
+# mirror-netdata-repos
7
+
8
+A local mirror of every active Netdata-org source repository,
9
+synced by a vendored bash script. Built for AI assistants
10
+(and humans) that need cross-repo grep, code review, and
11
+pattern lookup without paying GitHub API costs.
12
+
13
+## Why this skill exists
14
+
15
+Netdata maintains ~150 active source repos across the
16
+`netdata` GitHub org (the agent monorepo, cloud-* services,
17
+ai-agent, charts, helmchart, blogs, dashboards, ...). Routine
18
+work (cross-repo grep, "how does service X handle this?",
19
+pattern lookup, build) needs all of them locally.
20
+
21
+Without a local mirror:
22
+- Each cross-repo question hits the GitHub API.
23
+- Searches are paginated and rate-limited.
24
+- An AI assistant cannot pipeline grep results across repos.
25
+- Iteration speed and reasoning depth fall through the floor.
26
+
27
+With a local mirror at `${NETDATA_REPOS_DIR}`, all of that is
28
+fast local I/O.
29
+
30
+This is a **netdata repos mirror, independent from any other
31
+repo mirrors this workstation may have**. It exists for this
32
+project's cross-repo work; it is not a generic research mirror.
33
+
34
+## How it works
35
+
36
+The vendored script `scripts/sync-netdata-repos.sh` does two
37
+phases:
38
+
39
+### Phase 1 -- update existing repos
40
+
41
+For each `.git`-bearing subdirectory under `${NETDATA_REPOS_DIR}`,
42
+sorted by recent activity (cached in `.repo-activity-cache`):
43
+
44
+- If staged OR modified files exist -> **skip** with details.
45
+- Else: detect default branch (master / main / develop),
46
+ switch to it (committed feature-branch state survives in
47
+ the branch ref), `git pull`, and `git submodule update
48
+ --init --force --recursive`.
49
+
50
+### Phase 2 -- discover and clone new repos
51
+
52
+Runs only when:
53
+- no `--repo` flag was given (full sync), AND
54
+- `gh` is available AND authenticated.
55
+
56
+Lists `gh repo list netdata --source --no-archived` and clones
57
+any that are not yet in the mirror. The `--source --no-archived`
58
+filter excludes forks and dead repos -- they add no value for
59
+cross-repo grep.
60
+
61
+If `gh` is missing or not authenticated, Phase 2 is skipped
62
+with a clear warning. Phase 1 still runs (it uses local `git`
63
+only, no GitHub API).
64
+
65
+## Reset-to-default-branch is the feature, not a hazard
66
+
67
+Sub-repos in a mirror tend to drift onto stale feature
68
+branches that no one remembers. A repo whose `HEAD` is on
69
+`fix/something-from-six-months-ago` is a **black hole** for
70
+cross-repo reasoning -- the assistant grepping it sees
71
+out-of-date code and reasons wrong.
72
+
73
+The only viable fix: always reset to the default branch when
74
+it's safe to do so. The script's safety conditions:
75
+
76
+- **Untracked files**: OK; they survive checkout.
77
+- **Staged or modified files**: NOT safe; script skips the
78
+ repo and prints what was found.
79
+- **Unpushed feature-branch commits**: SAFE; the branch ref
80
+ preserves them, no data is lost. Script switches to default
81
+ with a warning summarizing the unpushed commits.
82
+
83
+So the rule is: if you have working changes you want to keep,
84
+commit them or stash them before running this. Anything else
85
+the script handles correctly.
86
+
87
+## When to run
88
+
89
+- **Before any cross-repo grep / review** session.
90
+- **After a long absence** from the workstation (catches up to
91
+ upstream on every repo).
92
+- **When you've just added a new netdata-org repo**: nothing
93
+ to do manually; the next full run picks it up via Phase 2.
94
+- **Periodically** (daily / weekly) to keep the mirror fresh.
95
+
96
+There's no automation here; the script is interactive (colored
97
+output, end-of-run summary). Run it on demand.
98
+
99
+## Setup
100
+
101
+### One-time
102
+
103
+1. Pick a directory for the mirror (large; expect 30-50 GB).
104
+2. Set `NETDATA_REPOS_DIR` in `<repo>/.env`:
105
+ ```
106
+ NETDATA_REPOS_DIR="/path/to/your/mirror"
107
+ ```
108
+3. `mkdir -p "$NETDATA_REPOS_DIR"`.
109
+4. Required tools: `git` and `jq`. Install via your package
110
+ manager.
111
+5. For Phase 2 (auto-discovery): install `gh` (the GitHub CLI)
112
+ and run `gh auth login`. SSH access to `git@github.com:netdata/...`
113
+ must work for clones.
114
+
115
+### First sync
116
+
117
+```bash
118
+# Source the env, run the script.
119
+source <(grep -E '^NETDATA_REPOS_DIR=' <repo>/.env)
120
+.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh
121
+```
122
+
123
+Or with the variable inline:
124
+
125
+```bash
126
+NETDATA_REPOS_DIR="/path/to/mirror" \
127
+ .agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh
128
+```
129
+
130
+The first run clones every netdata-org source repo. Expect it
131
+to take several minutes; subsequent runs are fast (only
132
+fetch+pull on each repo).
133
+
134
+## Common usage
135
+
136
+### Sync everything (default)
137
+
138
+```bash
139
+.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh
140
+```
141
+
142
+### Sync just one or two repos
143
+
144
+```bash
145
+.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh \
146
+ --repo netdata \
147
+ --repo cloud-frontend
148
+```
149
+
150
+`--repo` is repeatable. When any `--repo` is given, Phase 2
151
+(discovery) is skipped -- you asked for specific repos, the
152
+script does not go looking for new ones.
153
+
154
+### See help
155
+
156
+```bash
157
+.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh --help
158
+```
159
+
160
+Works without `NETDATA_REPOS_DIR` set.
161
+
162
+## Reading the output
163
+
164
+The script prints colored per-repo progress and ends with a
165
+summary covering:
166
+
167
+- **Branches switched to default**: every repo that was on a
168
+ non-default branch and got switched. Inspect this list if
169
+ you had work in progress.
170
+- **Repositories with uncommitted changes (skipped)**: these
171
+ weren't synced. Commit / stash / revert and re-run.
172
+- **Repositories with unpushed commits**: switched to default,
173
+ but you have feature-branch commits that haven't been
174
+ pushed. The branch ref preserves them; push when you're
175
+ ready.
176
+- **Repositories on wrong branch**: tried to switch but
177
+ failed (rare). Manual intervention needed.
178
+- **Repositories that failed to update**: fetch or pull
179
+ failure. Inspect manually.
180
+
181
+## Adding a new netdata-org repo
182
+
183
+Nothing to do in this skill or its script. Phase 2's
184
+`gh repo list netdata --source --no-archived` discovers any
185
+new netdata-org repo on the next full sync run. The repo
186
+must be:
187
+
188
+- Owned by the `netdata` org (not a fork).
189
+- Not archived.
190
+
191
+Otherwise it's skipped intentionally.
192
+
193
+If you want to mirror a fork or an archived repo (rare),
194
+clone it manually into `${NETDATA_REPOS_DIR}/<name>` and the
195
+next run's Phase 1 will start syncing it.
196
+
197
+## Sanitization (what the script checks)
198
+
199
+The script refuses to run unsafely. Hard errors (exit 2):
200
+
201
+- `NETDATA_REPOS_DIR` not set.
202
+- `NETDATA_REPOS_DIR` set but the directory doesn't exist.
203
+- `git` not in `PATH`.
204
+- `jq` not in `PATH`.
205
+
206
+Soft warnings (Phase 2 skipped, Phase 1 still runs):
207
+
208
+- `gh` not installed.
209
+- `gh` installed but not authenticated.
210
+
211
+## Limitations
212
+
213
+- **Org is hardcoded** (`netdata`). This skill is
214
+ netdata-org-specific.
215
+- **Filter is hardcoded** (`--source --no-archived`). To
216
+ mirror forks or archived repos, clone them manually -- the
217
+ script will then sync them in Phase 1.
218
+- **`gh` rate limit**: Phase 2 calls `gh repo list netdata
219
+ --limit 1000` once per run. On a properly-authed `gh` this
220
+ is well within the limit.
221
+- **Submodule `--force --recursive`**: intentional. Cross-repo
222
+ review and most build steps depend on accurate, up-to-date
223
+ submodule state. Local submodule modifications are
224
+ overwritten -- if you have work-in-progress inside a
225
+ submodule, commit it before running.
226
+
227
+## Path discipline
228
+
229
+This skill follows
230
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md`:
231
+
232
+- All references to the mirror directory go through
233
+ `${NETDATA_REPOS_DIR}` (the env key from `.env`).
234
+- The script itself contains no hardcoded user paths.
235
+- The skill content contains no workstation paths.
236
+
237
+## See also
238
+
239
+- `<repo>/.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh`
240
+ -- the vendored script.
241
+- `<repo>/.agents/skills/mirror-netdata-repos/how-tos/INDEX.md`
242
+ -- live catalog of how-tos.
243
+- `<repo>/.env` -- where `NETDATA_REPOS_DIR` lives.
.agents/skills/mirror-netdata-repos/how-tos/INDEX.md
new
+41
@@ -0,0 +1,41 @@
1
+# How-tos -- INDEX
2
+
3
+Live catalog of analysis-derived how-tos for the
4
+mirror-netdata-repos skill.
5
+
6
+**Live-catalog rule** (also stated in `../SKILL.md`): if an
7
+assistant is asked a concrete question about the mirror that
8
+required non-trivial analysis (multiple file reads, running
9
+the script with custom flags, debugging a failed sync) AND
10
+the answer is not already documented in `../SKILL.md`, the
11
+assistant MUST author a new `how-tos/<slug>.md` and add a row
12
+to this INDEX BEFORE completing the task.
13
+
14
+This is durable. Skipping it means the next assistant repeats
15
+the same analysis from scratch.
16
+
17
+## Catalog
18
+
19
+(empty -- entries grow as assistants encounter
20
+not-yet-documented questions)
21
+
22
+| Topic | Slug | Notes |
23
+|---|---|---|
24
+| -- | -- | -- |
25
+
26
+## How to add a how-to
27
+
28
+1. Create `how-tos/<slug>.md` with:
29
+ - One-line summary at the top (the question being answered).
30
+ - The answer with file/line citations into the script if
31
+ relevant.
32
+ - A "How I figured this out" footer naming the files read
33
+ and the commands run.
34
+2. Add a row to the table above with topic, slug, short notes.
35
+3. Commit alongside the work that prompted the analysis.
36
+
37
+## When NOT to add a how-to
38
+
39
+- The question is already covered by SKILL.md.
40
+- The answer is a one-liner.
41
+- The answer is highly speculative or version-specific.
.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh
new
+546
@@ -0,0 +1,546 @@
1
+#!/usr/bin/env bash
2
+
3
+# Don't use set -e to ensure script continues even on errors
4
+set -uo pipefail
5
+
6
+# Color codes for output
7
+RED='\033[0;31m'
8
+GREEN='\033[0;32m'
9
+YELLOW='\033[1;33m'
10
+BLUE='\033[0;34m'
11
+CYAN='\033[0;36m'
12
+NC='\033[0m' # No Color
13
+BOLD='\033[1m'
14
+
15
+# Configuration
16
+ORG="netdata"
17
+ACTIVITY_CACHE_FILE=".repo-activity-cache"
18
+
19
+# Arrays to track issues
20
+declare -a REPOS_WITH_UNCOMMITTED=()
21
+declare -a REPOS_WITH_UNPUSHED=()
22
+declare -a REPOS_UPDATE_FAILED=()
23
+declare -a REPOS_WRONG_BRANCH=()
24
+declare -a REPOS_BRANCH_SWITCHED=()
25
+
26
+# Scoped subset of repos (--repo flag, repeatable). Empty => all repos.
27
+declare -a SCOPE_REPOS=()
28
+
29
+usage() {
30
+ cat <<EOF
31
+sync-netdata-repos.sh [--repo NAME ...] [-h|--help]
32
+
33
+Maintains a local mirror of Netdata-org source repositories at
34
+\${NETDATA_REPOS_DIR} so that cross-repo grep / code review can run
35
+locally without GitHub API round-trips and rate limits.
36
+
37
+Phase 1 (always): for each repo in scope, skip if there are staged or
38
+modified changes; otherwise switch to the default branch (master/main/
39
+develop), pull, and recursively update submodules.
40
+
41
+Phase 2 (only when no --repo flags AND 'gh' is available + authed):
42
+discover new netdata-org source repos via 'gh repo list netdata
43
+--source --no-archived' and clone any that are missing.
44
+
45
+Options:
46
+ --repo NAME sync ONLY the named repo. Repeatable. Skips Phase 2.
47
+ -h, --help show this help.
48
+
49
+Required environment:
50
+ NETDATA_REPOS_DIR directory holding the mirror (must exist).
51
+
52
+Required tools:
53
+ git, jq always.
54
+ gh only for Phase 2; if missing or unauthed, Phase 2
55
+ is skipped with a warning.
56
+EOF
57
+}
58
+
59
+# ---------------------------------------------------------------
60
+# Early help (works without NETDATA_REPOS_DIR or any other env).
61
+for _arg in "$@"; do
62
+ case "$_arg" in
63
+ -h|--help) usage; exit 0 ;;
64
+ *) ;; # ignore -- main parser handles all other flags
65
+ esac
66
+done
67
+
68
+# ---------------------------------------------------------------
69
+# Sanitization (runs before any work).
70
+
71
+# 1. NETDATA_REPOS_DIR set and points to an existing directory.
72
+if [ -z "${NETDATA_REPOS_DIR:-}" ]; then
73
+ echo "ERROR: NETDATA_REPOS_DIR is not set." >&2
74
+ echo " Set it in <repo>/.env (or your shell env) to the directory" >&2
75
+ echo " that holds (or will hold) your Netdata-org repos mirror." >&2
76
+ exit 2
77
+fi
78
+MIRROR_DIR="$NETDATA_REPOS_DIR"
79
+if [ ! -d "$MIRROR_DIR" ]; then
80
+ echo "ERROR: NETDATA_REPOS_DIR='$MIRROR_DIR' is not an existing directory." >&2
81
+ echo " Create it first: mkdir -p \"\$NETDATA_REPOS_DIR\"" >&2
82
+ exit 2
83
+fi
84
+
85
+# 2. Required tools.
86
+for _cmd in git jq; do
87
+ if ! command -v "$_cmd" >/dev/null 2>&1; then
88
+ echo "ERROR: '$_cmd' is required but not found in PATH." >&2
89
+ exit 2
90
+ fi
91
+done
92
+
93
+# 3. Optional: gh (Phase 2 discovery only).
94
+GH_AVAILABLE=true
95
+GH_REASON=""
96
+if ! command -v gh >/dev/null 2>&1; then
97
+ GH_AVAILABLE=false
98
+ GH_REASON="'gh' is not installed"
99
+elif ! gh auth status >/dev/null 2>&1; then
100
+ GH_AVAILABLE=false
101
+ GH_REASON="'gh' is not authenticated (run: gh auth login)"
102
+fi
103
+
104
+cd "$MIRROR_DIR" || { echo "ERROR: cannot cd into NETDATA_REPOS_DIR=$MIRROR_DIR" >&2; exit 2; }
105
+
106
+# Function to print colored output
107
+print_status() {
108
+ local msg="$1"
109
+ echo -e "$msg"
110
+}
111
+
112
+# Function to check if directory is a git repository
113
+is_git_repo() {
114
+ local repo="$1"
115
+ [ -d "$repo/.git" ]
116
+}
117
+
118
+# Function to get last commit timestamp quickly (using filesystem heuristic)
119
+get_last_activity() {
120
+ local repo="$1"
121
+ # Use the modification time of .git/logs/HEAD if it exists (fast heuristic)
122
+ # This file is updated on commits, pulls, etc.
123
+ if [ -f "$repo/.git/logs/HEAD" ]; then
124
+ stat -c %Y "$repo/.git/logs/HEAD" 2>/dev/null || echo "0"
125
+ elif [ -d "$repo/.git" ]; then
126
+ # Fallback to .git directory modification time
127
+ stat -c %Y "$repo/.git" 2>/dev/null || echo "0"
128
+ else
129
+ echo "0"
130
+ fi
131
+}
132
+
133
+# Function to check for uncommitted changes (ignoring untracked files)
134
+has_uncommitted_changes() {
135
+ local repo="$1"
136
+ # Safety: ensure we're in the right directory
137
+ if [ ! -d "$repo" ]; then
138
+ return 0 # Treat missing directory as "has changes" to skip it
139
+ fi
140
+ cd "$repo" || return 0 # If cd fails, treat as "has changes"
141
+
142
+ # First check if we have a valid HEAD (repo might be empty or corrupted)
143
+ if ! git rev-parse HEAD >/dev/null 2>&1; then
144
+ cd "$MIRROR_DIR" 2>/dev/null || true
145
+ return 1 # No HEAD means no commits, so no uncommitted changes to worry about
146
+ fi
147
+
148
+ # Refresh the index to avoid false positives from timestamp changes
149
+ git update-index --refresh >/dev/null 2>&1 || true
150
+
151
+ # Check only for staged or modified files, not untracked files
152
+ # git diff-index checks for staged and modified files
153
+ # We ignore untracked files since they don't affect pulls
154
+ git diff-index --quiet HEAD -- 2>/dev/null
155
+ local diff_result=$?
156
+
157
+ # git diff-index returns 0 if no changes, 1 if changes exist
158
+ # We want to return 0 (true) if changes exist, 1 (false) if no changes
159
+ if [ $diff_result -eq 1 ]; then
160
+ local result=0 # Has changes
161
+ else
162
+ local result=1 # No changes
163
+ fi
164
+
165
+ cd "$MIRROR_DIR" 2>/dev/null || true # Try to return to script directory
166
+ return $result
167
+}
168
+
169
+# Function to check for unpushed commits
170
+has_unpushed_commits() {
171
+ local repo="$1"
172
+ # Safety: ensure we're in the right directory
173
+ if [ ! -d "$repo" ]; then
174
+ return 1 # No unpushed if directory doesn't exist
175
+ fi
176
+ cd "$repo" || return 1
177
+ local branch
178
+ branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null)
179
+ local result
180
+ if [ -n "$branch" ] && git rev-parse --verify "origin/$branch" >/dev/null 2>&1; then
181
+ [ -n "$(git log "origin/$branch..HEAD" --oneline 2>/dev/null)" ]
182
+ result=$?
183
+ else
184
+ result=1
185
+ fi
186
+ cd "$MIRROR_DIR" 2>/dev/null || true # Try to return to script directory
187
+ return $result
188
+}
189
+
190
+# Function to get uncommitted changes details
191
+get_uncommitted_details() {
192
+ local repo="$1"
193
+ # Safety: ensure we're in the right directory
194
+ if [ ! -d "$repo" ]; then
195
+ echo "directory not found"
196
+ return
197
+ fi
198
+ cd "$repo" || { echo "cannot access"; return; }
199
+ local staged unstaged
200
+ staged=$(git diff --cached --numstat | wc -l)
201
+ unstaged=$(git diff --numstat | wc -l)
202
+ cd "$MIRROR_DIR" 2>/dev/null || true # Try to return to script directory
203
+
204
+ # Report details - note that untracked files are shown but don't block updates
205
+ if [ "$staged" -gt 0 ] || [ "$unstaged" -gt 0 ]; then
206
+ echo "staged: $staged, modified: $unstaged"
207
+ else
208
+ # This can happen if git diff-index failed for other reasons
209
+ echo "git index issue or empty repository"
210
+ fi
211
+}
212
+
213
+# Function to update repository activity cache
214
+update_activity_cache() {
215
+ print_status "${CYAN}Updating repository activity cache...${NC}"
216
+
217
+ # Safety: Use a unique temp file to avoid conflicts
218
+ local temp_file="$ACTIVITY_CACHE_FILE.tmp.$$"
219
+ : > "$temp_file"
220
+
221
+ for dir in */; do
222
+ dir="${dir%/}"
223
+ # Only process actual directories that are git repos
224
+ if [ -d "$dir" ] && is_git_repo "$dir"; then
225
+ local timestamp
226
+ timestamp=$(get_last_activity "$dir")
227
+ echo "$timestamp $dir" >> "$temp_file"
228
+ fi
229
+ done
230
+
231
+ # Sort by timestamp (descending) and keep only the repo names
232
+ if [ -s "$temp_file" ]; then
233
+ sort -rn "$temp_file" | cut -d' ' -f2 > "$ACTIVITY_CACHE_FILE"
234
+ fi
235
+ rm -f "$temp_file"
236
+}
237
+
238
+# Function to get sorted repo list
239
+get_sorted_repos() {
240
+ if [ -f "$ACTIVITY_CACHE_FILE" ]; then
241
+ cat "$ACTIVITY_CACHE_FILE"
242
+ else
243
+ # If no cache exists, create one
244
+ update_activity_cache
245
+ cat "$ACTIVITY_CACHE_FILE"
246
+ fi
247
+}
248
+
249
+# Function to get default branch (assumes we're already in the repo directory)
250
+get_default_branch() {
251
+ # Try to get from remote
252
+ local default_branch
253
+ default_branch=$(git symbolic-ref refs/remotes/origin/HEAD 2>/dev/null | sed 's@^refs/remotes/origin/@@')
254
+
255
+ # If that fails, try common defaults
256
+ if [ -z "$default_branch" ]; then
257
+ if git show-ref --verify --quiet refs/remotes/origin/master; then
258
+ default_branch="master"
259
+ elif git show-ref --verify --quiet refs/remotes/origin/main; then
260
+ default_branch="main"
261
+ elif git show-ref --verify --quiet refs/remotes/origin/develop; then
262
+ default_branch="develop"
263
+ fi
264
+ fi
265
+
266
+ echo "$default_branch"
267
+}
268
+
269
+# Function to update a single repository
270
+update_repo() {
271
+ local repo="$1"
272
+ local current_num="$2"
273
+ local total_num="$3"
274
+
275
+ # Safety: validate repo directory exists and is a git repo
276
+ if [ ! -d "$repo" ]; then
277
+ print_status "${RED}[${current_num}/${total_num}] Skipping $repo - directory not found${NC}"
278
+ return 1
279
+ fi
280
+
281
+ if ! is_git_repo "$repo"; then
282
+ print_status "${YELLOW}[${current_num}/${total_num}] Skipping $repo - not a git repository${NC}"
283
+ return 1
284
+ fi
285
+
286
+ print_status "${BLUE}[${current_num}/${total_num}]${NC} ${BOLD}Updating $repo...${NC}"
287
+
288
+ # Check for uncommitted changes (staged or modified files only)
289
+ if has_uncommitted_changes "$repo"; then
290
+ local details
291
+ details=$(get_uncommitted_details "$repo")
292
+ print_status " ${YELLOW}⚠️ Skipping - uncommitted changes (${details})${NC}"
293
+ REPOS_WITH_UNCOMMITTED+=("$repo: $details")
294
+ return 1
295
+ fi
296
+
297
+ cd "$repo" || { print_status " ${RED}✗ Cannot access directory${NC}"; return 1; }
298
+
299
+ # Check for untracked files (informational only - doesn't block update)
300
+ local untracked_count
301
+ untracked_count=$(git ls-files --others --exclude-standard 2>/dev/null | wc -l)
302
+ if [ "$untracked_count" -gt 0 ]; then
303
+ print_status " ${CYAN}ℹ️ Note: ${untracked_count} untracked file(s) present${NC}"
304
+ fi
305
+
306
+ # Get current and default branches
307
+ local current_branch default_branch
308
+ current_branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null)
309
+ default_branch=$(get_default_branch)
310
+
311
+ # Check for unpushed commits (we're already in the repo directory)
312
+ if [ -n "$current_branch" ] && git rev-parse --verify "origin/$current_branch" >/dev/null 2>&1; then
313
+ if [ -n "$(git log "origin/$current_branch..HEAD" --oneline 2>/dev/null)" ]; then
314
+ local unpushed_count
315
+ unpushed_count=$(git log "origin/${current_branch}..HEAD" --oneline 2>/dev/null | wc -l)
316
+ print_status " ${YELLOW}⚠️ Warning: ${unpushed_count} unpushed commit(s) on ${current_branch}${NC}"
317
+ REPOS_WITH_UNPUSHED+=("$repo: $unpushed_count commits on $current_branch")
318
+ fi
319
+ fi
320
+
321
+ # Switch to default branch if needed
322
+ if [ -n "$default_branch" ] && [ "$current_branch" != "$default_branch" ]; then
323
+ print_status " ${CYAN}→ Switching from ${current_branch} to ${default_branch}${NC}"
324
+ if git checkout "$default_branch" >/dev/null 2>&1; then
325
+ REPOS_BRANCH_SWITCHED+=("$repo: $current_branch → $default_branch")
326
+ current_branch="$default_branch"
327
+ else
328
+ print_status " ${RED}✗ Failed to switch to ${default_branch}${NC}"
329
+ REPOS_WRONG_BRANCH+=("$repo: stuck on $current_branch, default is $default_branch")
330
+ fi
331
+ fi
332
+
333
+ # Fetch and pull
334
+ print_status " → Fetching..."
335
+ if git fetch origin >/dev/null 2>&1; then
336
+ print_status " → Pulling ${current_branch}..."
337
+ if git pull origin "$current_branch" >/dev/null 2>&1; then
338
+ # Update submodules
339
+ print_status " → Updating submodules..."
340
+ if git submodule update --init --force --recursive >/dev/null 2>&1; then
341
+ print_status " ${GREEN}✓ Updated successfully${NC}"
342
+ else
343
+ # Don't fail the whole update if submodules have issues
344
+ print_status " ${YELLOW}⚠️ Updated but submodule update had issues${NC}"
345
+ fi
346
+ else
347
+ print_status " ${RED}✗ Pull failed${NC}"
348
+ REPOS_UPDATE_FAILED+=("$repo")
349
+ cd "$MIRROR_DIR" 2>/dev/null || true
350
+ return 1
351
+ fi
352
+ else
353
+ print_status " ${RED}✗ Fetch failed${NC}"
354
+ REPOS_UPDATE_FAILED+=("$repo")
355
+ cd "$MIRROR_DIR" 2>/dev/null || true
356
+ return 1
357
+ fi
358
+
359
+ cd "$MIRROR_DIR" 2>/dev/null || true # Try to return to script directory
360
+ return 0
361
+}
362
+
363
+# Function to clone new repositories
364
+clone_new_repos() {
365
+ print_status "\n${BOLD}Checking for new repositories to clone...${NC}"
366
+
367
+ local new_repos_count=0
368
+
369
+ # Get list of all repos from GitHub
370
+ print_status "Fetching repository list from GitHub..."
371
+ local repos_list
372
+ repos_list=$(gh repo list "$ORG" --limit 1000 --json name,sshUrl,defaultBranchRef --source --no-archived)
373
+
374
+ echo "$repos_list" | jq -r '.[] | "\(.name) \(.sshUrl) \(.defaultBranchRef.name)"' | while read -r name url default_branch; do
375
+ if [ ! -d "$name" ]; then
376
+ new_repos_count=$((new_repos_count + 1))
377
+ print_status "${GREEN}→ Cloning new repo: $name (default branch: $default_branch, with submodules)${NC}"
378
+ if git clone --quiet --recursive "$url" "$name" 2>/dev/null; then
379
+ # Set up tracking for default branch
380
+ # Safety: validate we can enter the directory
381
+ if cd "$name" 2>/dev/null; then
382
+ git symbolic-ref refs/remotes/origin/HEAD "refs/remotes/origin/$default_branch" 2>/dev/null || true
383
+ cd "$MIRROR_DIR" 2>/dev/null || true
384
+ else
385
+ print_status " ${YELLOW}⚠️ Warning: Could not enter cloned directory${NC}"
386
+ fi
387
+ print_status " ${GREEN}✓ Cloned successfully${NC}"
388
+ else
389
+ print_status " ${RED}✗ Clone failed${NC}"
390
+ fi
391
+ fi
392
+ done
393
+
394
+ if [ $new_repos_count -eq 0 ]; then
395
+ print_status "No new repositories to clone."
396
+ fi
397
+}
398
+
399
+# Function to print summary
400
+print_summary() {
401
+ print_status "\n${BOLD}═══════════════════════════════════════════════════════════${NC}"
402
+ print_status "${BOLD}Summary Report${NC}"
403
+ print_status "${BOLD}═══════════════════════════════════════════════════════════${NC}"
404
+
405
+ if [ ${#REPOS_BRANCH_SWITCHED[@]} -gt 0 ]; then
406
+ print_status "\n${CYAN}📌 Branches switched to default:${NC}"
407
+ for repo in "${REPOS_BRANCH_SWITCHED[@]}"; do
408
+ print_status " • $repo"
409
+ done
410
+ fi
411
+
412
+ if [ ${#REPOS_WITH_UNCOMMITTED[@]} -gt 0 ]; then
413
+ print_status "\n${YELLOW}⚠️ Repositories with uncommitted changes (skipped):${NC}"
414
+ for repo in "${REPOS_WITH_UNCOMMITTED[@]}"; do
415
+ print_status " • $repo"
416
+ done
417
+ fi
418
+
419
+ if [ ${#REPOS_WITH_UNPUSHED[@]} -gt 0 ]; then
420
+ print_status "\n${YELLOW}📤 Repositories with unpushed commits:${NC}"
421
+ for repo in "${REPOS_WITH_UNPUSHED[@]}"; do
422
+ print_status " • $repo"
423
+ done
424
+ fi
425
+
426
+ if [ ${#REPOS_WRONG_BRANCH[@]} -gt 0 ]; then
427
+ print_status "\n${YELLOW}🔀 Repositories on wrong branch:${NC}"
428
+ for repo in "${REPOS_WRONG_BRANCH[@]}"; do
429
+ print_status " • $repo"
430
+ done
431
+ fi
432
+
433
+ if [ ${#REPOS_UPDATE_FAILED[@]} -gt 0 ]; then
434
+ print_status "\n${RED}✗ Repositories that failed to update:${NC}"
435
+ for repo in "${REPOS_UPDATE_FAILED[@]}"; do
436
+ print_status " • $repo"
437
+ done
438
+ fi
439
+
440
+ if [ ${#REPOS_WITH_UNCOMMITTED[@]} -eq 0 ] && \
441
+ [ ${#REPOS_WITH_UNPUSHED[@]} -eq 0 ] && \
442
+ [ ${#REPOS_WRONG_BRANCH[@]} -eq 0 ] && \
443
+ [ ${#REPOS_UPDATE_FAILED[@]} -eq 0 ]; then
444
+ print_status "\n${GREEN}✅ All repositories are clean and up to date!${NC}"
445
+ fi
446
+}
447
+
448
+# Main execution
449
+main() {
450
+ # Parse CLI flags.
451
+ while [ $# -gt 0 ]; do
452
+ local arg="$1"
453
+ case "$arg" in
454
+ --repo)
455
+ if [ $# -lt 2 ]; then
456
+ echo "ERROR: --repo requires a repository name" >&2
457
+ exit 2
458
+ fi
459
+ local val="$2"
460
+ SCOPE_REPOS+=("$val")
461
+ shift 2
462
+ ;;
463
+ -h|--help)
464
+ usage
465
+ exit 0
466
+ ;;
467
+ *)
468
+ echo "ERROR: Unknown option: $arg" >&2
469
+ usage >&2
470
+ exit 2
471
+ ;;
472
+ esac
473
+ done
474
+
475
+ print_status "${BOLD}═══════════════════════════════════════════════════════════${NC}"
476
+ print_status "${BOLD}Netdata Repository Sync Tool${NC}"
477
+ print_status "${BOLD}═══════════════════════════════════════════════════════════${NC}"
478
+ print_status "${CYAN}Mirror: ${MIRROR_DIR}${NC}"
479
+ if [ ${#SCOPE_REPOS[@]} -gt 0 ]; then
480
+ print_status "${CYAN}Scope: --repo flags ->${NC} ${SCOPE_REPOS[*]}"
481
+ fi
482
+
483
+ # Phase 1: Update repositories.
484
+ if [ ${#SCOPE_REPOS[@]} -gt 0 ]; then
485
+ print_status "\n${BOLD}Phase 1: Updating scoped repositories${NC}"
486
+ else
487
+ print_status "\n${BOLD}Phase 1: Updating existing repositories${NC}"
488
+ print_status "Sorting repositories by last activity..."
489
+ fi
490
+
491
+ # Build the working list.
492
+ local sorted_repos=()
493
+ if [ ${#SCOPE_REPOS[@]} -gt 0 ]; then
494
+ # Scoped run: validate each --repo entry exists locally.
495
+ for repo in "${SCOPE_REPOS[@]}"; do
496
+ if [ -d "$repo" ] && is_git_repo "$repo"; then
497
+ sorted_repos+=("$repo")
498
+ else
499
+ print_status "${YELLOW}⚠️ Skipping --repo $repo: not found at $MIRROR_DIR/$repo${NC}"
500
+ fi
501
+ done
502
+ else
503
+ # Default: activity-cache-sorted full set.
504
+ while IFS= read -r repo; do
505
+ sorted_repos+=("$repo")
506
+ done < <(get_sorted_repos)
507
+ fi
508
+
509
+ local total_repos=${#sorted_repos[@]}
510
+
511
+ if [ "$total_repos" -eq 0 ]; then
512
+ print_status "No repositories to update."
513
+ else
514
+ print_status "Found ${total_repos} repositories to update.\n"
515
+
516
+ local current=0
517
+ for repo in "${sorted_repos[@]}"; do
518
+ current=$((current + 1))
519
+ if [ -d "$repo" ] && is_git_repo "$repo"; then
520
+ update_repo "$repo" "$current" "$total_repos" || true # Continue even if update fails
521
+ fi
522
+ done
523
+ fi
524
+
525
+ # Phase 2: Clone new repositories.
526
+ if [ ${#SCOPE_REPOS[@]} -gt 0 ]; then
527
+ print_status "\n${CYAN}Skipping Phase 2 (discovery): --repo flags scoped this run.${NC}"
528
+ elif ! $GH_AVAILABLE; then
529
+ print_status "\n${YELLOW}⚠️ Skipping Phase 2 (discovery): ${GH_REASON}.${NC}"
530
+ else
531
+ print_status "\n${BOLD}Phase 2: Checking for new repositories${NC}"
532
+ clone_new_repos || true
533
+ fi
534
+
535
+ # Update activity cache for next run.
536
+ print_status "\n${CYAN}Updating activity cache for next run...${NC}"
537
+ update_activity_cache || true
538
+
539
+ # Print summary.
540
+ print_summary || true
541
+
542
+ print_status "\n${GREEN}${BOLD}✅ Sync complete!${NC}"
543
+}
544
+
545
+# Run main function with all CLI args.
546
+main "$@"
\ No newline at end of file
.agents/skills/query-agent-events/AE_FIELDS.md
new
+460
@@ -0,0 +1,460 @@
1
+# AE_FIELDS reference
2
+
3
+Verified field map for the agent-events journal namespace.
4
+Every claim here is traceable to producer source at
5
+`<repo>/src/daemon/status-file.c` (the schema is
6
+`STATUS_FILE_VERSION = 28`, `src/daemon/status-file.h:14`).
7
+
8
+The .local draft `agent-events-journals.md` was found to have
9
+14 high-severity divergences (wrong enums, missing fields,
10
+misattributed semantics). This file supersedes it.
11
+
12
+## How journal field names are formed
13
+
14
+Producer JSON path -> journal field name:
15
+
16
+1. Producer emits a JSON path (e.g. `agent.profile[0]`).
17
+2. log2journal applies `--prefix 'AE_'` (literal prepend; the
18
+ prefix is NOT transliterated --
19
+ `src/collectors/log2journal/log2journal-help.c:108-109`).
20
+3. log2journal walks the JSON tree. For nested objects, it
21
+ joins parent + child with `_`. For arrays, it appends
22
+ `_<index>` (`log2journal-json.c:477-511`).
23
+4. Per-character transliteration applies a 256-entry map
24
+ (`log2journal.c:8-61`): lowercase -> uppercase, digits and
25
+ uppercase pass through, everything else (including `.`,
26
+ `-`, `:`, `/`, `@`, `_`, `[`, `]`) maps to `_`. Consecutive
27
+ underscores are collapsed
28
+ (`log2journal-json.c:395-396`).
29
+
30
+So:
31
+- `agent.id` -> `AE_AGENT_ID`
32
+- `agent.profile[0]` -> `AE_AGENT_PROFILE_0`
33
+- `host.boot.id` -> `AE_HOST_BOOT_ID`
34
+- `@timestamp` -> `AE__TIMESTAMP` (note: double underscore
35
+ because `@` -> `_`)
36
+- `fatal.errno` -> `AE_FATAL_ERRNO` (NOT from a top-level
37
+ `fatal_errno`; the path is nested)
38
+
39
+## Top-level POST-time fields (always present)
40
+
41
+Added by `post_status_file()` at `status-file.c:967-976`. These
42
+sit at the JSON top level, so they have NO nested-path
43
+prefix in the journal field name.
44
+
45
+| JSON path | Journal field | Type | Always? | Triage meaning |
46
+|---|---|---|---|---|
47
+| `exit_cause` | `AE_EXIT_CAUSE` | string | yes | Human-readable label for why this session ended. The first thing to look at. See enum below. |
48
+| `message` | `AE_MESSAGE` | string | yes | One-line summary; subject of FTS search. |
49
+| `priority` | `AE_PRIORITY` | int | yes | Syslog priority (lower = more severe). |
50
+| `version_saved` | `AE_VERSION_SAVED` | uint | yes | The status file's own version (typically 28). |
51
+| `agent_version_now` | `AE_AGENT_VERSION_NOW` | string | yes | Posting agent's version (the agent that did the POST = next session, NOT the one that crashed). |
52
+| `agent_pid_now` | `AE_AGENT_PID_NOW` | uint | yes | PID of the agent that POSTed. |
53
+| `host_memory_critical` | `AE_HOST_MEMORY_CRITICAL` | bool | yes | Was the host under memory pressure at POST time? |
54
+| `host_memory_free_percent` | `AE_HOST_MEMORY_FREE_PERCENT` | uint | yes | % free RAM at POST time. |
55
+| `agent_health` | `AE_AGENT_HEALTH` | string | yes | Health classification across restart history. See enum below. |
56
+| `@timestamp` | `AE__TIMESTAMP` | RFC3339 | yes | When the captured session ended. |
57
+| `version` | `AE_VERSION` | uint | yes | The schema version (`STATUS_FILE_VERSION`). Use to slice by schema. |
58
+
59
+These fields are **always present** on every record and are
60
+the safest scoping anchors. `AE_EXIT_CAUSE`, `AE_AGENT_HEALTH`,
61
+`AE_VERSION` are all good index-friendly facets.
62
+
63
+## `agent.*` fields (`dsf_json_agent`, `status-file.c:99-156`)
64
+
65
+The previous (crashed) session's agent state.
66
+
67
+| JSON path | Journal field | Type | Version-gating | Triage meaning |
68
+|---|---|---|---|---|
69
+| `agent.id` | `AE_AGENT_ID` | UUID | always | Netdata machine GUID (per-install, persistent). The "agent identity". DIFFERENT from `host.id`. |
70
+| `agent.since` | `AE_AGENT_SINCE` | RFC3339 | v>=24 | When this install was first registered. |
71
+| `agent.ephemeral_id` | `AE_AGENT_EPHEMERAL_ID` | UUID | always | Unique per Netdata invocation (changes on every restart). Use to group multiple events from the same crashed session. |
72
+| `agent.version` | `AE_AGENT_VERSION` | string | always | The version of the *crashed* session. **Slice on this for regression-spotting.** |
73
+| `agent.uptime` | `AE_AGENT_UPTIME` | int seconds | always | Duration the crashed session ran. Short uptime + crash = startup bug. |
74
+| `agent.node_id` | `AE_AGENT_NODE_ID` | UUID | always | Cloud node UUID (empty when agent isn't claimed). |
75
+| `agent.claim_id` | `AE_AGENT_CLAIM_ID` | UUID | always | Cloud claim UUID (empty when not claimed). Presence -> cloud-connected agent. |
76
+| `agent.restarts` | `AE_AGENT_RESTARTS` | uint | always | Total restart count for this install. High value + recent crash = agent loop. |
77
+| `agent.crashes` | `AE_AGENT_CRASHES` | uint | v>=24 | Total crash count. |
78
+| `agent.pid` | `AE_AGENT_PID` | uint | v>=27 | PID of the crashed session. |
79
+| `agent.posts` | `AE_AGENT_POSTS` | uint | v>=22 | Total POSTs from this install. |
80
+| `agent.aclk` | `AE_AGENT_ACLK` | enum | v>=22 | Cloud connection state. See enum. |
81
+| `agent.profile[N]` | `AE_AGENT_PROFILE_0..N` | enum array | always | Bitmap rendered as array. Slice by parent vs child vs iot. See enum. |
82
+| `agent.status` | `AE_AGENT_STATUS` | enum | always | DAEMON_STATUS at the time of capture. See enum. |
83
+| `agent.exit_reason[N]` | `AE_AGENT_EXIT_REASON_0..N` | enum array | always | EXIT_REASON bitmap rendered as array. Empty array = no specific reason. See enum. |
84
+| `agent.install_type` | `AE_AGENT_INSTALL_TYPE` | string | always | `kickstart`, `binpkg`, `static`, etc. **Slice on this for "is this a packaging issue?"** |
85
+| `agent.db_mode` | `AE_AGENT_DB_MODE` | string | v>=14 | dbengine memory mode. |
86
+| `agent.db_tiers` | `AE_AGENT_DB_TIERS` | uint | v>=14 | Number of dbengine tiers. |
87
+| `agent.kubernetes` | `AE_AGENT_KUBERNETES` | bool | v>=14 | Kubernetes deployment? Slice on this for k8s-specific issues. |
88
+| `agent.sentry_available` | `AE_AGENT_SENTRY_AVAILABLE` | bool | v>=16 | Is Sentry enabled? |
89
+| `agent.reliability` | `AE_AGENT_RELIABILITY` | int | always | Signed reliability counter (positive = healthy run streak; negative = crash streak). `<= -2` -> `crash-loop`. |
90
+| `agent.stack_traces` | `AE_AGENT_STACK_TRACES` | string | always | Backtrace backend name (`libbacktrace`, `none`). |
91
+| `agent.timings.init` | `AE_AGENT_TIMINGS_INIT` | int seconds | always | How long startup took. Long init + crash = startup bug. |
92
+| `agent.timings.exit` | `AE_AGENT_TIMINGS_EXIT` | int seconds | always | How long shutdown took. |
93
+
94
+## `metrics.*` fields (`dsf_json_metrics`, `:158-193`)
95
+
96
+Snapshot of the database at the time of capture. Useful for
97
+"big-database crashes" investigations.
98
+
99
+| JSON path | Journal field |
100
+|---|---|
101
+| `metrics.nodes.total` | `AE_METRICS_NODES_TOTAL` |
102
+| `metrics.nodes.receiving` | `AE_METRICS_NODES_RECEIVING` |
103
+| `metrics.nodes.sending` | `AE_METRICS_NODES_SENDING` |
104
+| `metrics.nodes.archived` | `AE_METRICS_NODES_ARCHIVED` |
105
+| `metrics.metrics.collected` | `AE_METRICS_METRICS_COLLECTED` |
106
+| `metrics.metrics.available` | `AE_METRICS_METRICS_AVAILABLE` |
107
+| `metrics.instances.collected` | `AE_METRICS_INSTANCES_COLLECTED` |
108
+| `metrics.instances.available` | `AE_METRICS_INSTANCES_AVAILABLE` |
109
+| `metrics.contexts.collected` | `AE_METRICS_CONTEXTS_COLLECTED` |
110
+| `metrics.contexts.available` | `AE_METRICS_CONTEXTS_AVAILABLE` |
111
+
112
+## `host.*` fields (`dsf_json_host`, `:195-253`)
113
+
114
+Host-level info, mostly stable across crashes on the same
115
+host.
116
+
117
+| JSON path | Journal field | Triage meaning |
118
+|---|---|---|
119
+| `host.id` | `AE_HOST_ID` | OS-level `/etc/machine-id`. **Different from `AE_AGENT_ID`** (Netdata's own identifier). Use to spot multiple agents on the same host. |
120
+| `host.architecture` | `AE_HOST_ARCHITECTURE` | `x86_64`, `aarch64`, `armv7l`, ... **Slice for arch-specific bugs.** |
121
+| `host.virtualization` | `AE_HOST_VIRTUALIZATION` | `none`, `kvm`, `vmware`, `lxc`, `docker`, ... |
122
+| `host.container` | `AE_HOST_CONTAINER` | `none`, `docker`, `kubernetes`, ... |
123
+| `host.uptime` | `AE_HOST_UPTIME` | **MISLEADING NAME.** Stores boottime EPOCH (`status-file.c:202` writes `ds->boottime` from `now_boottime_sec()`). NOT a duration. Compute uptime via `now - AE_HOST_UPTIME`. |
124
+| `host.timezone` | `AE_HOST_TIMEZONE` | string, v>=20 |
125
+| `host.cloud_provider` | `AE_HOST_CLOUD_PROVIDER` | `aws`, `gcp`, `azure`, ..., v>=20 |
126
+| `host.cloud_instance` | `AE_HOST_CLOUD_INSTANCE` | EC2 instance type etc., v>=20 |
127
+| `host.cloud_region` | `AE_HOST_CLOUD_REGION` | v>=20 |
128
+| `host.system_cpus` | `AE_HOST_SYSTEM_CPUS` | uint. **Slice for "low-cpu environment" bugs.** |
129
+| `host.boot.id` | `AE_HOST_BOOT_ID` | UUID, changes on every host boot. |
130
+| `host.memory.total` | `AE_HOST_MEMORY_TOTAL` | bytes, only when `OS_SYSTEM_MEMORY_OK` |
131
+| `host.memory.free` | `AE_HOST_MEMORY_FREE` | bytes |
132
+| `host.memory.netdata` | `AE_HOST_MEMORY_NETDATA` | bytes used by netdata, v>=21 |
133
+| `host.memory.oom_protection` | `AE_HOST_MEMORY_OOM_PROTECTION` | uint, v>=21 |
134
+| `host.disk.db.total` | `AE_HOST_DISK_DB_TOTAL` | bytes available to dbengine |
135
+| `host.disk.db.free` | `AE_HOST_DISK_DB_FREE` | bytes free |
136
+| `host.disk.db.inodes_total` | `AE_HOST_DISK_DB_INODES_TOTAL` | uint |
137
+| `host.disk.db.inodes_free` | `AE_HOST_DISK_DB_INODES_FREE` | uint |
138
+| `host.disk.db.read_only` | `AE_HOST_DISK_DB_READ_ONLY` | bool. True + crash -> "disk read-only" cause. |
139
+| `host.disk.netdata.dbengine` | `AE_HOST_DISK_NETDATA_DBENGINE` | bytes used by dbengine files. |
140
+| `host.disk.netdata.sqlite` | `AE_HOST_DISK_NETDATA_SQLITE` | bytes used by SQLite files. |
141
+| `host.disk.netdata.other` | `AE_HOST_DISK_NETDATA_OTHER` | bytes used by other files. |
142
+| `host.disk.netdata.last_updated` | `AE_HOST_DISK_NETDATA_LAST_UPDATED` | RFC3339. |
143
+
144
+## `os.*` fields (`dsf_json_os`, `:255-266`)
145
+
146
+| JSON path | Journal field | Triage meaning |
147
+|---|---|---|
148
+| `os.type` | `AE_OS_TYPE` | enum: `unknown`, `linux`, `freebsd`, `macos`, `windows`. |
149
+| `os.kernel` | `AE_OS_KERNEL` | Kernel version string. |
150
+| `os.name` | `AE_OS_NAME` | Distro name (e.g. `Ubuntu`, `CentOS Stream`). |
151
+| `os.version` | `AE_OS_VERSION` | Distro version. |
152
+| `os.family` | `AE_OS_FAMILY` | `os_id` (e.g. `ubuntu`). **Slice for distro-specific issues.** |
153
+| `os.platform` | `AE_OS_PLATFORM` | `os_id_like` (parent distro family, e.g. `debian`). NOT a rewrite of `AE_OS_FAMILY` -- they're independent producer fields. |
154
+
155
+## `hw.*` fields (`dsf_json_hw`, `:268-319`)
156
+
157
+DMI / SMBIOS data. Useful for hardware-specific bug
158
+investigation. Privacy-sensitive serials and asset_tags are
159
+**commented out at the producer side** (`status-file.c:275-276,
160
+:294-295, :304-305`) and never reach the journal.
161
+
162
+| JSON path | Journal field | Notes |
163
+|---|---|---|
164
+| `hw.sys.vendor` | `AE_HW_SYS_VENDOR` | BIOS / system vendor. |
165
+| `hw.sys.uuid` | `AE_HW_SYS_UUID` | System UUID. |
166
+| `hw.product.name` | `AE_HW_PRODUCT_NAME` | Product name (e.g. `MacBookPro18,3`). |
167
+| `hw.product.version` | `AE_HW_PRODUCT_VERSION` | |
168
+| `hw.product.sku` | `AE_HW_PRODUCT_SKU` | |
169
+| `hw.product.family` | `AE_HW_PRODUCT_FAMILY` | |
170
+| `hw.board.name` | `AE_HW_BOARD_NAME` | |
171
+| `hw.board.version` | `AE_HW_BOARD_VERSION` | |
172
+| `hw.board.vendor` | `AE_HW_BOARD_VENDOR` | |
173
+| `hw.chassis.type` | `AE_HW_CHASSIS_TYPE` | Numeric (e.g. `6` = desktop, `9` = laptop). |
174
+| `hw.chassis.vendor` | `AE_HW_CHASSIS_VENDOR` | |
175
+| `hw.chassis.version` | `AE_HW_CHASSIS_VERSION` | |
176
+| `hw.bios.date` | `AE_HW_BIOS_DATE` | |
177
+| `hw.bios.release` | `AE_HW_BIOS_RELEASE` | |
178
+| `hw.bios.version` | `AE_HW_BIOS_VERSION` | |
179
+| `hw.bios.vendor` | `AE_HW_BIOS_VENDOR` | |
180
+
181
+## `product.*` fields (`dsf_json_product`, `:321-329`)
182
+
183
+| JSON path | Journal field |
184
+|---|---|
185
+| `product.vendor` | `AE_PRODUCT_VENDOR` |
186
+| `product.name` | `AE_PRODUCT_NAME` |
187
+| `product.type` | `AE_PRODUCT_TYPE` |
188
+
189
+## `fatal.*` fields (`dsf_json_fatal`, `:331-367`)
190
+
191
+Present on crashes and deliberate fatal conditions. Empty on
192
+graceful exits.
193
+
194
+| JSON path | Journal field | Version-gating | Triage meaning |
195
+|---|---|---|---|
196
+| `fatal.line` | `AE_FATAL_LINE` | always | Source line of the panic. Combine with FILENAME and FUNCTION for de-dup. |
197
+| `fatal.filename` | `AE_FATAL_FILENAME` | always | Source file. **Slice on this for "this file is buggy".** |
198
+| `fatal.function` | `AE_FATAL_FUNCTION` | always | Function name (with demangled symbol). **Slice on this for "this function is buggy".** |
199
+| `fatal.message` | `AE_FATAL_MESSAGE` | always | Panic message. Subject of FTS. |
200
+| `fatal.errno` | `AE_FATAL_ERRNO` | always | errno string at panic. |
201
+| `fatal.thread` | `AE_FATAL_THREAD` | always | Worker thread name (e.g. `CTXLOAD`, `STREAM:63`). |
202
+| `fatal.thread_id` | `AE_FATAL_THREAD_ID` | always | POSIX TID. |
203
+| `fatal.stack_trace` | `AE_FATAL_STACK_TRACE` | always | Backtrace. Real addresses preserved (anonymization is dedup-only, `status-file-dedup.c:26-36`). |
204
+| `fatal.signal_code` | `AE_FATAL_SIGNAL_CODE` | v>=16 | `SIGNAL/SI_CODE` formatted (e.g. `SIGSEGV/SEGV_MAPERR`). Empty -> not a signal crash. **Primary signal-crash predicate.** See enum. |
205
+| `fatal.sentry` | `AE_FATAL_SENTRY` | v>=17 | Was a Sentry submission attempted? |
206
+| `fatal.fault_address` | `AE_FATAL_FAULT_ADDRESS` | v>=18 | Hex address of the fault. Empty when `signal_code == 0`. |
207
+| `fatal.worker_job_id` | `AE_FATAL_WORKER_JOB_ID` | v>=23 | Worker job ID at panic. |
208
+
209
+## Enum reference
210
+
211
+### `AE_AGENT_STATUS` (DAEMON_STATUS)
212
+
213
+Source: `src/daemon/status-file.c:23-33`.
214
+
215
+| Value | Meaning for triage |
216
+|---|---|
217
+| `none` | No prior status (very first session). |
218
+| `initializing` | Crashed during startup -> startup bug. Combine with `agent.timings.init` for context. |
219
+| `running` | Crashed during normal operation -> the most "interesting" class. |
220
+| `exiting` | Crashed during shutdown -> shutdown-path bug. |
221
+| `exited` | Graceful exit (no crash). |
222
+
223
+### `AE_AGENT_ACLK` (CLOUD_STATUS)
224
+
225
+Source: `src/claim/cloud-status.c:5-15`.
226
+
227
+| Value | Meaning for triage |
228
+|---|---|
229
+| `available` | Default; not yet attempted. |
230
+| `online` | Connected to Cloud (ACLK up). |
231
+| `indirect` | Connected via parent. |
232
+| `banned` | Cloud rejected (claim issue). |
233
+| `offline` | Disconnected (network or shutdown). |
234
+
235
+(The .local draft listed a `disabled` value -- it does NOT
236
+exist in the producer source.)
237
+
238
+### `AE_AGENT_HEALTH`
239
+
240
+Source: `src/daemon/status-file.c:929-952`. Computed by the
241
+**agent** (not the ingestion server) at POST time across
242
+restart history. Used to isolate crash classes.
243
+
244
+| Value | Meaning for triage |
245
+|---|---|
246
+| `healthy-first` | First run, no prior crashes. Boring (filter out). |
247
+| `healthy-repeated` | Multiple healthy runs in a row. |
248
+| `healthy-loop` | Reliability >= 2 consecutive healthy runs. |
249
+| `healthy-recovered` | Was unhealthy, now healthy. |
250
+| `crash-first` | First crash ever on this install. Interesting -- new bug? |
251
+| `crash-entered` | Single crash, then recovered. |
252
+| `crash-loop` | Reliability <= -2 (repeated crashes). **Highest-priority class.** |
253
+| `crash-repeated` | Two or more crashes. |
254
+
255
+To find ALL crashes: `(AE_AGENT_HEALTH in crash-first, crash-loop, crash-repeated, crash-entered)`.
256
+
257
+### `AE_AGENT_PROFILE_*` (ND_PROFILE bitmap)
258
+
259
+Source: `src/daemon/config/netdata-conf-profile.c:7-15`.
260
+
261
+| Value | Meaning |
262
+|---|---|
263
+| `standalone` | Single-node deployment. |
264
+| `parent` | Streaming parent. **Slice for "parent-only" bugs.** |
265
+| `child` | Streaming child. **Slice for "child-only" bugs.** |
266
+| `iot` | IoT / lightweight profile. |
267
+
268
+(The .local draft listed `dopple` and `store-child` -- they do
269
+NOT exist; `iot` was missing.)
270
+
271
+### `AE_AGENT_EXIT_REASON_*` (EXIT_REASON bitmap)
272
+
273
+Source: `src/libnetdata/exit/exit_initiated.c:7-38`. The
274
+EXIT_REASON bitmap renders as a JSON array. Empty bitmap ->
275
+empty array (no `none` element).
276
+
277
+20 distinct strings:
278
+
279
+| Value | Meaning |
280
+|---|---|
281
+| `signal-segmentation-fault` | SIGSEGV received. |
282
+| `signal-bus-error` | SIGBUS received. |
283
+| `signal-floating-point-exception` | SIGFPE received. |
284
+| `signal-illegal-instruction` | SIGILL received. |
285
+| `signal-abort` | SIGABRT received (assertion / abort()). |
286
+| `signal-bad-system-call` | SIGSYS received. |
287
+| `signal-cpu-time-limit-exceeded` | SIGXCPU received. |
288
+| `signal-file-size-limit-exceeded` | SIGXFSZ received. |
289
+| `signal-quit` | SIGQUIT received. |
290
+| `signal-terminate` | SIGTERM received (graceful kill). |
291
+| `signal-interrupt` | SIGINT received (Ctrl-C). |
292
+| `out-of-memory` | OOM panic. |
293
+| `already-running` | Another instance held the listen socket. |
294
+| `fatal` | Generic fatal() call. |
295
+| `api-quit` | API endpoint requested exit. |
296
+| `cmd-exit` | Explicit `netdata --exit` invocation. |
297
+| `service-stop` | Service manager (systemd) sent stop. |
298
+| `system-shutdown` | Host shutting down. |
299
+| `update` | Replaced by a new version. |
300
+| `shutdown-timeout` | Shutdown took too long. |
301
+
302
+(The .local draft was significantly wrong here -- listed
303
+~10 invented values like `exit-called`, `exit-and-update`,
304
+`cannot-allocate`, `oom`, `assertion-failed`, none of which
305
+exist in producer source.)
306
+
307
+### `AE_EXIT_CAUSE` (top-level)
308
+
309
+Source: `src/daemon/status-file.c:1097-1286`. Computed by the
310
+**agent**, NOT the ingestion server. The most useful field for
311
+classifying records.
312
+
313
+26 distinct strings:
314
+
315
+**Initial / no prior state (1):**
316
+
317
+| Value | Meaning |
318
+|---|---|
319
+| `no last status` | First-ever start; no prior status file readable. |
320
+
321
+**Prior was EXITED (graceful) (7):**
322
+
323
+| Value | Meaning |
324
+|---|---|
325
+| `exit no reason` | Prior exited cleanly with no reason recorded. |
326
+| `deadly signal and exit` | Got a deadly signal but exited normally. |
327
+| `fatal and exit` | Hit a fatal but managed to exit. |
328
+| `exit on system shutdown` | Host shutting down; agent stopped gracefully. |
329
+| `exit to update` | Stopped to allow an update. |
330
+| `exit and updated` | Stopped and was replaced by a new version. |
331
+| `exit instructed` | `netdata --exit` or service stop. |
332
+
333
+**Prior was INITIALIZING (8):**
334
+
335
+| Value | Meaning |
336
+|---|---|
337
+| `abnormal power off` | Power loss during startup. |
338
+| `deadly signal on start` | Signal during startup. |
339
+| `out of memory` | OOM during startup. (.local draft says `cannot allocate` -- wrong.) |
340
+| `already running` | Listen socket conflict at init. |
341
+| `disk read-only` | Filesystem read-only at init. |
342
+| `disk full` | Disk full at init. |
343
+| `disk almost full` | Disk near capacity at init. |
344
+| `fatal on start` | fatal() during startup. |
345
+| `killed hard on start` | SIGKILL/SIGTERM during startup. |
346
+
347
+**Prior was EXITING (5):**
348
+
349
+| Value | Meaning |
350
+|---|---|
351
+| `deadly signal on exit` | Signal during shutdown. |
352
+| `exit timeout` | Shutdown didn't complete in time. |
353
+| `fatal on exit` | fatal() during shutdown. |
354
+| `killed hard on shutdown` | SIGKILL during shutdown (host shutdown). |
355
+| `killed hard on update` | SIGKILL during hot update. |
356
+| `killed hard on exit` | SIGKILL during exit. |
357
+
358
+**Prior was RUNNING (6):**
359
+
360
+| Value | Meaning |
361
+|---|---|
362
+| `abnormal power off` | Power loss during normal operation. |
363
+| `out of memory` | OOM during normal operation. |
364
+| `deadly signal` | Signal received during normal operation. |
365
+| `killed fatal` | SIGKILL after a fatal. |
366
+| `killed hard low ram` | OOM-killed (RAM pressure). |
367
+| `killed hard` | SIGKILL/SIGTERM from outside. |
368
+
369
+### `AE_OS_TYPE` (DAEMON_OS_TYPE)
370
+
371
+Source: `src/daemon/status-file.c:35-45`.
372
+
373
+`unknown`, `linux`, `freebsd`, `macos`, `windows`.
374
+
375
+### `AE_FATAL_SIGNAL_CODE`
376
+
377
+Format: `SIGNAL/SI_CODE` (e.g. `SIGSEGV/SEGV_MAPERR`). Sources:
378
+`src/libnetdata/signals/signal-code.c:12-53` (signal name map),
379
+`:97-184` (per-signal SI_CODE map).
380
+
381
+Most relevant for crash triage:
382
+
383
+| Value | Meaning |
384
+|---|---|
385
+| `SIGSEGV/SEGV_MAPERR` | Invalid memory map (NULL pointer, freed memory). |
386
+| `SIGSEGV/SEGV_ACCERR` | Access violation (write to read-only page). |
387
+| `SIGSEGV/SEGV_BNDERR` | Address bound check fault. |
388
+| `SIGSEGV/SEGV_PKUERR` | Protection key fault. |
389
+| `SIGBUS/BUS_ADRALN` | Alignment error. |
390
+| `SIGBUS/BUS_ADRERR` | Non-existent physical address. |
391
+| `SIGBUS/BUS_OBJERR` | Object-specific bus error. |
392
+| `SIGFPE/FPE_INTDIV` | Integer divide by zero. |
393
+| `SIGFPE/FPE_INTOVF` | Integer overflow. |
394
+| `SIGFPE/FPE_FLTDIV` | Float divide by zero. |
395
+| `SIGABRT/SI_TKILL` | abort() / assertion failure (typical SI_CODE for abort()). |
396
+| `SIGTRAP/TRAP_BRKPT` | Breakpoint trap. |
397
+| `SIGTRAP/TRAP_TRACE` | Trace trap. |
398
+
399
+Empty `AE_FATAL_SIGNAL_CODE` -> not a signal crash (it's a
400
+deliberate fatal or a graceful exit).
401
+
402
+(The .local draft had `SIGABRT/ABRT` which is wrong: `ABRT`
403
+is not a valid SI_CODE token. And `SIGTRAP/TRAP Trace` should
404
+be `SIGTRAP/TRAP_TRACE`. And `SIGVTALRM/VTALRM` does not
405
+exist as a per-signal SI_CODE.)
406
+
407
+## Index-friendly facets (high-value)
408
+
409
+These fields are the **first-pass slicers** for queries.
410
+Always include at least 1-2 of these in `selections` before
411
+falling back to FTS:
412
+
413
+- `AE_AGENT_VERSION` -- regression / fix-detection.
414
+- `AE_AGENT_HEALTH` -- crash class.
415
+- `AE_EXIT_CAUSE` -- exit class.
416
+- `AE_FATAL_SIGNAL_CODE` -- signal type.
417
+- `AE_FATAL_FUNCTION` -- localize to a function.
418
+- `AE_FATAL_FILENAME` -- localize to a file.
419
+- `AE_HOST_ARCHITECTURE` -- arch-specific bugs.
420
+- `AE_OS_FAMILY` -- distro-specific bugs.
421
+- `AE_AGENT_PROFILE_0` (and `_1`, `_2`) -- parent / child / iot.
422
+- `AE_AGENT_KUBERNETES` -- k8s-specific.
423
+- `AE_AGENT_INSTALL_TYPE` -- packaging-specific.
424
+
425
+## Privacy-sensitive fields
426
+
427
+Treat these as identifying. The `redact-events.sh` opt-in
428
+filter masks them when sharing:
429
+
430
+- `AE_AGENT_ID` (machine GUID).
431
+- `AE_HOST_ID` (OS machine-id).
432
+- `AE_AGENT_NODE_ID`, `AE_AGENT_CLAIM_ID` (Cloud identifiers).
433
+- `AE_HOST_BOOT_ID`, `AE_AGENT_EPHEMERAL_ID`.
434
+- `AE_HW_SYS_UUID`.
435
+- DMI fields (`AE_HW_*`) when correlated with serial-equivalent
436
+ identifiers.
437
+
438
+(Privacy-sensitive serials and asset_tags are already
439
+commented out at the producer side and never reach the
440
+journal -- `status-file.c:275-276, :294-295, :304-305`.)
441
+
442
+## What is NOT in the journal
443
+
444
+Several producer fields are intentionally redacted at the
445
+producer side (commented out in `dsf_json_hw`):
446
+
447
+- `hw.sys.serial`, `hw.sys.asset_tag`
448
+- `hw.board.serial`, `hw.board.asset_tag`
449
+- `hw.chassis.serial`, `hw.chassis.asset_tag`
450
+
451
+There is no `agent.happiness` field in the producer source at
452
+any version. The `.local` draft mentioned it -- the field has
453
+never existed.
454
+
455
+## Stack trace addresses are NOT anonymized in the journal
456
+
457
+`status-file-dedup.c:26-36` zeroes out hex addresses ONLY when
458
+computing the dedup hash. The journal-emitted
459
+`AE_FATAL_STACK_TRACE` retains real addresses. Useful for
460
+bug investigation; sensitive when sharing externally.
.agents/skills/query-agent-events/SKILL.md
new
+173
@@ -0,0 +1,173 @@
1
+---
2
+name: query-agent-events
3
+description: Bug-investigation tool for the Netdata agent-events ingestion namespace -- triage crashes, panics, fatals across the fleet by downloading events of interest and clustering locally. Covers the three transports (Cloud API and direct agent API are primary; ssh is operator-only), the verified AE_* field map and enum meanings, the dedup model (23h client-side per agent and event signature), the after-the-fact event timing (POST only on agent restart), and the Netdata systemd-journal plugin multi-value filter syntax (FIELD in A, B, C) AND ... Use when investigating crashes / panics / fatals; when grepping for events touching a specific function or file or version; when looking for regressions across versions; when an agent is reported crashing in a way you want to triage. Ships scripts get-events.sh and analyze-events.sh that fetch events with index-friendly filters and compute group-by stats. Defaults to last 24 hours and to the latest stable plus latest 2-3 nightlies.
4
+---
5
+
6
+# query-agent-events
7
+
8
+Private developer skill for triaging crashes, panics, and
9
+fatals across the Netdata fleet. Reads the agent-events
10
+systemd-journal namespace via the Netdata `systemd-journal`
11
+Function (Cloud-proxied or direct-agent transport) and ships
12
+scripts that bake in index-friendly query patterns.
13
+
14
+## Why this skill exists
15
+
16
+40k-200k status events arrive on the ingestion server every
17
+day on stable releases. The total fleet is 1.5M agents, so
18
+the dataset is large and noisy (many unupdated agents report
19
+crashes that have been fixed). Naive "grep all" queries are
20
+slow and wasteful. This skill teaches the maintainer (and any
21
+AI assistant helping them) how to slice the dataset
22
+efficiently and how to interpret what comes back.
23
+
24
+## Workflow
25
+
26
+```
27
++-------------------------+ +---------------------+
28
+| get-events.sh | --> | <timestamp>.json |
29
+| (cloud or agent API) | | in .local/audits |
30
++-------------------------+ +---------------------+
31
+ |
32
+ v
33
+ +------------------------+
34
+ | analyze-events.sh |
35
+ | --by signal|version| |
36
+ | function|... |
37
+ +------------------------+
38
+ |
39
+ v
40
+ +------------------------+
41
+ | cluster + read source |
42
+ | + fix the bug |
43
+ +------------------------+
44
+```
45
+
46
+The skill is a bug-investigation tool, not a generic logs
47
+query tool. The two existing `query-netdata-cloud` and
48
+`query-netdata-agents` skills already cover transport
49
+mechanics; this skill EXTENDS them with the agent-events
50
+specifics (what fields are present, what predicates are
51
+index-friendly, what each enum value means for triage).
52
+
53
+## Key concepts (read first)
54
+
55
+1. **The dataset**: 40k-200k status events / day on stable
56
+ releases, spread across 1.5M agents (not all restart
57
+ daily). Naive full-namespace queries with bare FTS are
58
+ slow.
59
+
60
+2. **Index-friendly queries** (HARD RULE): use multi-value
61
+ field filters FIRST. The Netdata `systemd-journal` plugin
62
+ supports the syntax:
63
+ ```
64
+ (FIELD1 in A, B, C) AND (FIELD2 in D, E, F) AND ...
65
+ ```
66
+ Between fields = AND. Between values = OR. This is a
67
+ facet-engine feature, NOT raw journalctl. Use FTS via
68
+ `query=` only as a residual narrower over the structured
69
+ slice. See `query-discipline.md`.
70
+
71
+3. **Three transports** (priority order):
72
+ - **Cloud API** -- proxied through Netdata Cloud at the
73
+ agent-events space. Primary for the team.
74
+ - **Direct agent API** -- against the agent-events node's
75
+ `/api/v3/function?function=systemd-journal`. Primary for
76
+ scripts.
77
+ - **ssh to the host** -- operator-only path; mentioned in
78
+ `transports.md` but no scripted ssh transport.
79
+
80
+4. **After-the-fact event model**: agents POST events ONLY on
81
+ start (the previous session's exit reason). They commit
82
+ status to disk on start, stop, and at most every 10
83
+ minutes. So the meaningful query unit is "events posted in
84
+ the last 24 hours"; "the last hour" misses real crashes
85
+ that haven't restarted yet.
86
+
87
+5. **23h client-side dedup** (`src/daemon/status-file-dedup.c:11`):
88
+ same agent + same event-content hash within 23h ->
89
+ suppressed at the producer. So 1 record per agent per
90
+ event-signature per day is the natural unit. Different
91
+ agents posting the same crash signature -> both arrive
92
+ (server does not dedup).
93
+
94
+6. **Default time + version filters**: 24h time window;
95
+ latest stable + latest 2-3 nightlies for version. This
96
+ focuses triage on bugs that still matter. Wide windows
97
+ (`--since '7d'` or longer) are reserved for rare crashes
98
+ (1-per-few-days class) and for "when did this start /
99
+ get fixed" investigations.
100
+
101
+7. **AE_* field naming**: every JSON path in the producer's
102
+ status document becomes an `AE_`-prefixed journal field
103
+ (per `log2journal --prefix 'AE_'` on the ingestion server).
104
+ See `AE_FIELDS.md` for the verified map and enum meanings.
105
+
106
+## Table of contents
107
+
108
+| Doc | Purpose |
109
+|---|---|
110
+| `AE_FIELDS.md` | Verified field map (~80 rows) + enum meanings for triage. Indispensable. |
111
+| `transports.md` | Cloud API + direct agent API call patterns; ssh footnote. |
112
+| `update-cadence.md` | After-the-fact model, dedup, push timing, disk commits, query implications. |
113
+| `query-discipline.md` | The multi-value filter syntax, structured-filters-first rule, anti-patterns. |
114
+| `finding-crashes.md` | Recipe: signal crashes (SIGSEGV / SIGBUS / SIGFPE / SIGABRT) on stable. |
115
+| `finding-fatals.md` | Recipe: deliberate fatals (OOM, disk full, asserts). |
116
+| `recipes/INDEX.md` | Live catalog of recipes (find-by-function, find-by-version, find-related-to-work). |
117
+| `how-tos/INDEX.md` | Live catalog: every analysis question becomes a how-to entry. |
118
+
119
+## Live how-to rule (mandatory)
120
+
121
+If asked a concrete question about agent-events that requires
122
+non-trivial analysis (multiple file reads, running queries,
123
+cross-referencing with producer source) AND the answer is not
124
+already documented in the per-domain guides above or in
125
+`recipes/`, the assistant MUST author a new how-to under
126
+`how-tos/<slug>.md` and add a one-line entry to
127
+`how-tos/INDEX.md` BEFORE completing the task.
128
+
129
+## Scripts (in scripts/)
130
+
131
+| Script | Purpose |
132
+|---|---|
133
+| `_lib.sh` | Helpers (`agentevents_*` prefix). Sources `query-netdata-agents/scripts/_lib.sh`. Token-safe; ships a no-leak self-test. |
134
+| `get-events.sh` | Fetch events of interest. Index-friendly defaults. JSON output to `.local/audits/query-agent-events/`. |
135
+| `analyze-events.sh` | Group-by stats over a downloaded dump (signal, version, fatal_function, architecture, etc.). |
136
+| `redact-events.sh` | Opt-in redaction (machine_guid / claim_id / host_id / ephemeral_id -> placeholders). For sharing only. |
137
+
138
+## Path discipline
139
+
140
+This skill follows
141
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md`:
142
+
143
+- Repo files: repo-relative (`<repo>/src/...`).
144
+- Sibling Netdata-org repos: `${NETDATA_REPOS_DIR}/<repo>/...`.
145
+- agent-events host / namespace / machine GUID / node ID:
146
+ ALWAYS via env keys. Never literal values in any committed
147
+ file.
148
+- Producer ingest URL: NEVER quoted literally. Reference only
149
+ as `src/daemon/status-file.c:988`.
150
+- Fetched event payloads land under
151
+ `<repo>/.local/audits/query-agent-events/<timestamp>.json`
152
+ (gitignored). Do NOT paste raw event JSON into committed
153
+ artifacts.
154
+
155
+## Required env keys
156
+
157
+| Key | Role |
158
+|---|---|
159
+| `NETDATA_CLOUD_TOKEN` | Cloud REST token (long-lived). |
160
+| `NETDATA_CLOUD_HOSTNAME` | Cloud REST API host. |
161
+| `AGENT_EVENTS_HOSTNAME` | Dual-duty: ssh host AND direct-HTTP host of the ingestion node. Can be IP or DNS name. NOT the journalctl namespace (hardcoded `agent-events`); NOT the Cloud room name (also hardcoded `agent-events`). |
162
+| `AGENT_EVENTS_MACHINE_GUID` | Agent machine GUID for direct-agent transport. |
163
+| `AGENT_EVENTS_NODE_ID` | Cloud node UUID for cloud-proxy transport. |
164
+
165
+All values live in `<repo>/.env` (gitignored). See
166
+`<repo>/.agents/ENV.md` for setup (where each value comes
167
+from, sample formats, common mistakes).
168
+
169
+## Related skills
170
+
171
+- `query-netdata-cloud` -- transport: Cloud REST API.
172
+- `query-netdata-agents` -- transport: direct agent REST + bearer auto-mint.
173
+- This skill consumes both via their `_lib.sh` helpers.
.agents/skills/query-agent-events/finding-crashes.md
new
+137
@@ -0,0 +1,137 @@
1
+# Finding crashes (signal)
2
+
3
+A "crash" = the kernel delivered a fatal signal to the agent
4
+process. The signature is `AE_FATAL_SIGNAL_CODE` non-empty.
5
+
6
+## Quick recipe
7
+
8
+Get recent signal crashes on stable + recent nightlies:
9
+
10
+```bash
11
+.agents/skills/query-agent-events/scripts/get-events.sh \
12
+ --health crash \
13
+ --since '24h ago' \
14
+ --versions auto
15
+```
16
+
17
+Output is a JSON dump under
18
+`<repo>/.local/audits/query-agent-events/<timestamp>.json`.
19
+
20
+Then aggregate:
21
+
22
+```bash
23
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
24
+ --input <bundle.json> \
25
+ --by signal
26
+```
27
+
28
+## What "signal crash" means
29
+
30
+The agent received a fatal signal (SIGSEGV, SIGBUS, SIGFPE,
31
+SIGABRT, SIGILL, etc.) and could not gracefully recover. The
32
+deadly-signal handler tried to capture context (signal_code,
33
+fault_address, stack_trace) before exiting.
34
+
35
+Distinguishing predicates:
36
+
37
+- `AE_FATAL_SIGNAL_CODE` non-empty -- definitive marker.
38
+- `AE_AGENT_HEALTH` IN crash-first / crash-loop / crash-repeated
39
+ / crash-entered -- the agent classifies the result.
40
+- `AE_AGENT_EXIT_REASON_*` typically contains
41
+ `signal-segmentation-fault`, `signal-bus-error`,
42
+ `signal-floating-point-exception`, `signal-illegal-instruction`,
43
+ `signal-abort`, etc.
44
+
45
+## Index-friendly query for crashes
46
+
47
+```json
48
+{
49
+ "after": -86400,
50
+ "before": 0,
51
+ "last": 500,
52
+ "__logs_sources": "agent-events",
53
+ "selections": {
54
+ "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"],
55
+ "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"]
56
+ },
57
+ "facets": ["AE_FATAL_SIGNAL_CODE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"]
58
+}
59
+```
60
+
61
+The auto version filter computes the version list dynamically
62
+(see `update-cadence.md`).
63
+
64
+## Triage flow
65
+
66
+1. **Get the dump** -- `get-events.sh --health crash`.
67
+2. **Group by signal** -- `analyze-events.sh --by signal` to
68
+ see SIGSEGV vs SIGBUS vs SIGABRT distribution.
69
+3. **Group by function** for the dominant signal --
70
+ `analyze-events.sh --by fatal_function --filter "signal=SIGSEGV/SEGV_MAPERR"`
71
+ (or pre-filter the dump with `jq`).
72
+4. **Pick the dominant function**, look at one
73
+ representative event's stack trace
74
+ (`AE_FATAL_STACK_TRACE`), correlate with source.
75
+5. **Cross-check by version** -- is this on stable only? Just
76
+ nightlies? When did it appear?
77
+6. **Cross-check by environment** -- arch? distro? kubernetes?
78
+ parent vs child? Is the crash environment-specific?
79
+7. **Read source, fix bug.**
80
+
81
+## Common signal-code values
82
+
83
+(from `<repo>/src/libnetdata/signals/signal-code.c:97-184`,
84
+see `AE_FIELDS.md` for the full table)
85
+
86
+| Signal code | What it means |
87
+|---|---|
88
+| `SIGSEGV/SEGV_MAPERR` | NULL pointer / freed memory / unmapped page. |
89
+| `SIGSEGV/SEGV_ACCERR` | Write to read-only / executable page. |
90
+| `SIGBUS/BUS_ADRALN` | Misaligned access (mostly ARM / mmap). |
91
+| `SIGBUS/BUS_OBJERR` | Object-level fault (often disk I/O). |
92
+| `SIGFPE/FPE_INTDIV` | Integer divide by zero. |
93
+| `SIGABRT/SI_TKILL` | abort() / assertion failure. |
94
+| `SIGILL/ILL_ILLOPC` | Illegal instruction (often binary corruption). |
95
+| `SIGTRAP/TRAP_BRKPT` | Breakpoint trap. Usually a debugger; sometimes a deliberate `__builtin_trap()`. |
96
+
97
+## Pitfalls
98
+
99
+- **Empty stack trace**: `AE_FATAL_STACK_TRACE` may be the
100
+ string `info: will now attempt to get stack trace` or `info: stack trace is not available, libbacktrace reports no frames`
101
+ when the fault was instantaneous (e.g. NULL deref at low
102
+ address). Without the stack, fall back to
103
+ `AE_FATAL_FUNCTION` + `AE_FATAL_FILENAME` + `AE_FATAL_LINE`
104
+ (these are populated from `__FILE__` / `__LINE__` of the
105
+ most recent `fatal()` call -- not the crash site, but
106
+ often nearby).
107
+
108
+- **Aborted dumps**: events where `AE_FATAL_SIGNAL_CODE` is
109
+ set but other `AE_FATAL_*` fields are empty -- the signal
110
+ arrived before context capture completed.
111
+
112
+- **Shutdown races**: a crash during shutdown reports
113
+ `AE_EXIT_CAUSE = 'killed hard on exit'` or `'killed hard on shutdown'`
114
+ with a stack trace that may show shutdown timing rather
115
+ than the actual crash site. Interpret with caution.
116
+
117
+- **Sentry-suppressed**: when `AE_FATAL_SENTRY = true`, the
118
+ agent attempted a Sentry submission. Sentry breakdowns have
119
+ more information; cross-reference if available.
120
+
121
+## Filtering out noise
122
+
123
+Many crashes from old / unsupported versions are already
124
+fixed. The default `--versions auto` filter handles this.
125
+For wider investigations, scope to stable releases only:
126
+
127
+```bash
128
+get-events.sh --health crash --versions '^v2\.\d+\.\d+$'
129
+```
130
+
131
+## Related recipes
132
+
133
+- `recipes/find-by-function.md` -- when you have a function
134
+ name in mind.
135
+- `recipes/find-by-version.md` -- regression spotter.
136
+- `finding-fatals.md` -- the OTHER class (deliberate exits,
137
+ not signal crashes).
.agents/skills/query-agent-events/finding-fatals.md
new
+160
@@ -0,0 +1,160 @@
1
+# Finding fatals (deliberate exits)
2
+
3
+A "fatal" = the agent **chose** to exit because of a
4
+condition it could not recover from (OOM, disk full,
5
+assertion failure, listen-socket conflict, fatal() call).
6
+DIFFERENT from a signal crash (the kernel didn't kill us; we
7
+called `exit()` or `_exit()` ourselves).
8
+
9
+## Quick recipe
10
+
11
+```bash
12
+.agents/skills/query-agent-events/scripts/get-events.sh \
13
+ --exit-cause fatal \
14
+ --since '24h ago' \
15
+ --versions auto
16
+```
17
+
18
+`--exit-cause fatal` is shorthand for the deliberate-fatal
19
+class of `AE_EXIT_CAUSE` values (see below).
20
+
21
+## Distinguishing predicates
22
+
23
+- `AE_FATAL_SIGNAL_CODE` is **EMPTY** (this is what separates
24
+ fatals from signal crashes).
25
+- `AE_EXIT_CAUSE` is one of:
26
+ - `no last status`
27
+ - `out of memory` (NOT `cannot allocate` -- the .local draft
28
+ was wrong)
29
+ - `disk full` / `disk almost full` / `disk read-only`
30
+ - `already running`
31
+ - `fatal on start` / `fatal on exit` / `fatal and exit`
32
+ - `exit timeout`
33
+ - `abnormal power off`
34
+- `AE_AGENT_EXIT_REASON_*` may contain `out-of-memory`,
35
+ `already-running`, `fatal`, `shutdown-timeout`.
36
+- `AE_FATAL_MESSAGE`, `AE_FATAL_FUNCTION`, `AE_FATAL_FILENAME`,
37
+ `AE_FATAL_LINE` -- populated from the `fatal()` call site.
38
+
39
+## Index-friendly query
40
+
41
+```json
42
+{
43
+ "after": -86400,
44
+ "before": 0,
45
+ "last": 500,
46
+ "__logs_sources": "agent-events",
47
+ "selections": {
48
+ "AE_EXIT_CAUSE": [
49
+ "no last status",
50
+ "out of memory",
51
+ "disk full",
52
+ "disk almost full",
53
+ "disk read-only",
54
+ "already running",
55
+ "fatal on start",
56
+ "fatal on exit",
57
+ "fatal and exit"
58
+ ],
59
+ "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"]
60
+ },
61
+ "facets": ["AE_EXIT_CAUSE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"]
62
+}
63
+```
64
+
65
+## Per-cause triage
66
+
67
+### `out of memory`
68
+
69
+Agent panicked when an allocation failed. Look at
70
+`AE_FATAL_FUNCTION` to localize -- often `mallocz`,
71
+`reallocz`, dbengine page allocators, or memory-pool
72
+constructors. Cross-correlate with
73
+`AE_HOST_MEMORY_FREE_PERCENT` (low at POST time = host was
74
+under pressure) and `AE_HOST_MEMORY_NETDATA` (how much
75
+netdata was using).
76
+
77
+Most useful slicers:
78
+- `AE_AGENT_DB_MODE` -- dbengine memory mode.
79
+- `AE_AGENT_DB_TIERS` -- number of tiers.
80
+- `AE_HOST_MEMORY_TOTAL` -- absolute host RAM. OOM on a 1GB
81
+ host is different from OOM on a 64GB host.
82
+
83
+### `disk full` / `disk almost full` / `disk read-only`
84
+
85
+Storage failure. Look at:
86
+- `AE_HOST_DISK_DB_FREE` / `AE_HOST_DISK_DB_INODES_FREE` --
87
+ what state the disk was in.
88
+- `AE_HOST_DISK_DB_READ_ONLY` -- bool, set on read-only.
89
+- `AE_AGENT_DB_MODE` -- whether dbengine is in `dbengine`,
90
+ `ram`, `ram-cache`, `none`, `alloc`, `save`, ...
91
+
92
+### `already running`
93
+
94
+Listen-socket conflict at init. Look at `AE_AGENT_VERSION`
95
+and `AE_AGENT_RESTARTS` -- if restarts > 0 and the same
96
+agent_id keeps hitting this, there's a stale lock or a
97
+concurrent agent on the same host.
98
+
99
+### `fatal on start` / `fatal on exit` / `fatal and exit`
100
+
101
+Generic `fatal()` calls. Look at `AE_FATAL_MESSAGE` to find
102
+the panic message and `AE_FATAL_FUNCTION` to localize. Group
103
+by message to find the most common patterns.
104
+
105
+### `no last status`
106
+
107
+First-ever start, or the prior status file was corrupted /
108
+unreadable. Usually transient. If recurring on the same
109
+agent (`AE_AGENT_RESTARTS` going up), the status file is
110
+being deleted or the disk path is broken.
111
+
112
+### `abnormal power off`
113
+
114
+Power loss between "agent was running" and "agent restarted".
115
+Useful to see whether power events correlate with crashes
116
+elsewhere -- usually filter these OUT of crash analysis.
117
+
118
+### `exit timeout`
119
+
120
+Shutdown didn't complete in time. Look at
121
+`AE_AGENT_TIMINGS_EXIT` for how long shutdown ran before the
122
+timeout fired. Long exit timings + worker_job_id may localize
123
+the stuck thread.
124
+
125
+## Triage flow
126
+
127
+1. **Get the dump** -- `get-events.sh --exit-cause fatal`.
128
+2. **Group by exit_cause** -- `analyze-events.sh --by exit_cause`
129
+ to see the distribution.
130
+3. **For the dominant cause**, group by function or message --
131
+ `analyze-events.sh --by fatal_function --filter "exit_cause=out of memory"`.
132
+4. **Cross-check environment** -- is this OOM specific to
133
+ small-RAM hosts? Specific dbengine mode?
134
+5. **Read source at the panic site**, fix the bug or improve
135
+ the error path.
136
+
137
+## Pitfalls
138
+
139
+- **`abnormal power off` is not a bug** -- it's environmental.
140
+ Filter out of crash analysis unless you specifically want
141
+ power events.
142
+
143
+- **`fatal_errno`** vs `fatal.errno` -- the producer emits
144
+ `fatal.errno` (nested), not `fatal_errno` (top-level). The
145
+ journal field is `AE_FATAL_ERRNO` in both cases (the
146
+ underscore comes from the dot transliteration).
147
+
148
+- **`exit timeout` records have NO `AE_FATAL_*` context** --
149
+ the shutdown timer fired without a panic site. Use
150
+ `AE_AGENT_TIMINGS_EXIT` and `AE_FATAL_WORKER_JOB_ID` instead.
151
+
152
+- **Disk-related causes propagate** -- a `disk full` event
153
+ may show up as `cannot allocate` in subsequent attempts.
154
+ The first record is the meaningful one.
155
+
156
+## Related recipes
157
+
158
+- `finding-crashes.md` -- the OTHER class (signal-delivered).
159
+- `recipes/find-by-function.md` -- localize by function name.
160
+- `recipes/find-by-version.md` -- regression spotter.
.agents/skills/query-agent-events/how-tos/INDEX.md
new
+45
@@ -0,0 +1,45 @@
1
+# How-tos -- INDEX
2
+
3
+Live catalog of analysis-derived how-tos for the
4
+query-agent-events skill.
5
+
6
+**Live-catalog rule** (also stated in `../SKILL.md`): if an
7
+assistant is asked a concrete question about agent-events
8
+that requires non-trivial analysis (multiple file reads,
9
+multiple queries, cross-referencing producer source) AND the
10
+answer is not already documented in the per-domain guides
11
+(`../AE_FIELDS.md`, `../transports.md`, `../update-cadence.md`,
12
+`../query-discipline.md`, `../finding-crashes.md`,
13
+`../finding-fatals.md`) or the recipes (`../recipes/`), the
14
+assistant MUST author a new `how-tos/<slug>.md` and add a
15
+one-line entry to this INDEX BEFORE completing the task.
16
+
17
+This is durable. Skipping it means the next assistant repeats
18
+the same analysis from scratch.
19
+
20
+## Catalog
21
+
22
+(empty -- entries grow as assistants encounter
23
+not-yet-documented questions)
24
+
25
+| Topic | Slug | Notes |
26
+|---|---|---|
27
+| -- | -- | -- |
28
+
29
+## How to add a how-to
30
+
31
+1. Create `how-tos/<slug>.md` with:
32
+ - One-line summary at the top (the question being answered).
33
+ - The answer with `path:line` citations into producer source
34
+ where appropriate.
35
+ - A "How I figured this out" footer naming the files read,
36
+ the queries run (with payloads), and the helpers used.
37
+2. Add a row to the table above with topic, slug, short notes.
38
+3. Commit alongside the work that prompted the analysis.
39
+
40
+## When NOT to add a how-to
41
+
42
+- The question is already covered by an existing per-domain
43
+ guide or recipe -- update that guide instead.
44
+- The answer is a one-liner.
45
+- The answer is highly speculative or version-specific.
.agents/skills/query-agent-events/query-discipline.md
new
+264
@@ -0,0 +1,264 @@
1
+# Query discipline
2
+
3
+How to compose efficient agent-events queries. Hard rule:
4
+**structured filters first, FTS only as residual.**
5
+
6
+For the JSON shape of the Function payload (including the
7
+`selections` multi-value filter mechanism), see:
8
+
9
+- `<repo>/docs/netdata-ai/skills/query-netdata-cloud/query-logs.md`
10
+ ("Multi-value field selections" section)
11
+
12
+This doc covers agent-events-specific guidance: which AE_*
13
+fields to put in `selections`, how to compose them, and the
14
+anti-patterns to avoid.
15
+
16
+## The hard rule
17
+
18
+```
19
+selections (structured, indexable) --> query (FTS, residual narrower)
20
+```
21
+
22
+NEVER FTS-first on agent-events. The namespace can hold 40k-200k
23
+records / day; a bare `query` over a wide window full-scans the
24
+dataset.
25
+
26
+## What to put in `selections`
27
+
28
+Always include at least one of these fields (always present on
29
+every record per `<repo>/src/daemon/status-file.c:967-976`):
30
+
31
+- `AE_AGENT_HEALTH` -- crash class. Filter to `crash-*` values
32
+ to drop healthy records.
33
+- `AE_EXIT_CAUSE` -- exit reason (26 distinct values; pick the
34
+ ones you want).
35
+- `AE_AGENT_VERSION` -- producing agent version (regression
36
+ slicing; pair with the auto-version filter).
37
+- `AE_FATAL_SIGNAL_CODE` -- empty / non-empty discriminates
38
+ signal crashes from deliberate fatals / graceful exits.
39
+
40
+For more specific triage:
41
+
42
+- `AE_FATAL_FUNCTION` / `AE_FATAL_FILENAME` -- localize to a
43
+ function or source file.
44
+- `AE_HOST_ARCHITECTURE` -- arch-specific bugs.
45
+- `AE_OS_FAMILY` / `AE_OS_TYPE` -- distro / OS-specific.
46
+- `AE_AGENT_PROFILE_0` -- standalone / parent / child / iot.
47
+- `AE_AGENT_KUBERNETES` -- k8s-specific.
48
+- `AE_AGENT_INSTALL_TYPE` -- packaging-specific.
49
+- `AE_AGENT_ACLK` -- cloud-claimed vs not.
50
+
51
+See `AE_FIELDS.md` for the full field map and which values
52
+each enum supports.
53
+
54
+## Worked examples
55
+
56
+### Example 1: index-friendly crash slice
57
+
58
+Find recent signal crashes on stable v2.10.x and the latest
59
+2 nightlies:
60
+
61
+```json
62
+{
63
+ "after": -86400,
64
+ "before": 0,
65
+ "last": 500,
66
+ "__logs_sources": "agent-events",
67
+ "selections": {
68
+ "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"],
69
+ "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"]
70
+ },
71
+ "facets": ["AE_FATAL_SIGNAL_CODE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE"]
72
+}
73
+```
74
+
75
+The `selections` cuts to ~hundreds-to-low-thousands of records
76
+via the facet index; `facets` on top groups by the dimensions
77
+of interest. No FTS needed.
78
+
79
+### Example 2: index-friendly + FTS narrower
80
+
81
+Find recent crashes whose stack trace mentions a specific
82
+function:
83
+
84
+```json
85
+{
86
+ "after": -86400,
87
+ "before": 0,
88
+ "last": 200,
89
+ "__logs_sources": "agent-events",
90
+ "selections": {
91
+ "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"]
92
+ },
93
+ "query": "rrdcontext_release"
94
+}
95
+```
96
+
97
+`selections` cuts to crashes only; `query` does FTS over the
98
+already-sliced subset for the substring match. This is the
99
+right composition: structured first, FTS narrows.
100
+
101
+If you can express the function name as a `selections` value
102
+on `AE_FATAL_FUNCTION`, prefer that:
103
+
104
+```json
105
+{
106
+ "after": -86400,
107
+ "before": 0,
108
+ "last": 200,
109
+ "__logs_sources": "agent-events",
110
+ "selections": {
111
+ "AE_FATAL_FUNCTION": ["rrdcontext_release"]
112
+ }
113
+}
114
+```
115
+
116
+This is even faster (pure indexed, no FTS).
117
+
118
+### Example 3: regression spotter
119
+
120
+Compare crash counts across versions:
121
+
122
+```json
123
+{
124
+ "after": -86400,
125
+ "before": 0,
126
+ "last": 1,
127
+ "__logs_sources": "agent-events",
128
+ "selections": {
129
+ "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"]
130
+ },
131
+ "facets": ["AE_AGENT_VERSION"],
132
+ "histogram": "AE_AGENT_VERSION"
133
+}
134
+```
135
+
136
+`last: 1` because we only need the facet counts, not the rows.
137
+The histogram + facets give the per-version distribution.
138
+
139
+### Example 4: rare-crash investigation (wider window)
140
+
141
+Looking for a known rare crash signature (1-per-few-days):
142
+
143
+```json
144
+{
145
+ "after": -604800,
146
+ "before": 0,
147
+ "last": 500,
148
+ "__logs_sources": "agent-events",
149
+ "selections": {
150
+ "AE_FATAL_SIGNAL_CODE": ["SIGSEGV/SEGV_MAPERR"],
151
+ "AE_FATAL_FUNCTION": ["specific_function"]
152
+ }
153
+}
154
+```
155
+
156
+7-day window is acceptable here because the `selections`
157
+cuts to specific signal+function -- a sharp index-resolved
158
+slice that returns small results regardless of window width.
159
+
160
+## Anti-patterns
161
+
162
+### Bare FTS over wide window (BAD)
163
+
164
+```json
165
+{
166
+ "after": -604800,
167
+ "before": 0,
168
+ "__logs_sources": "agent-events",
169
+ "query": "SIGSEGV"
170
+}
171
+```
172
+
173
+A 7-day FTS over the full namespace. Slow. ALWAYS pair with
174
+at least one structured `selections` field.
175
+
176
+### FTS for things that should be `selections` (BAD)
177
+
178
+```json
179
+{
180
+ "query": "SIGSEGV"
181
+}
182
+```
183
+
184
+`SIGSEGV/...` is a value in the `AE_FATAL_SIGNAL_CODE` field.
185
+Use:
186
+
187
+```json
188
+{
189
+ "selections": {
190
+ "AE_FATAL_SIGNAL_CODE": ["SIGSEGV/SEGV_MAPERR", "SIGSEGV/SEGV_ACCERR", "SIGSEGV/SEGV_BNDERR", "SIGSEGV/SEGV_PKUERR"]
191
+ }
192
+}
193
+```
194
+
195
+### Naive equality on `=` only (BAD when you want OR)
196
+
197
+The `selections` mechanism gives you OR-of-values for free.
198
+Don't run multiple queries to OR results client-side; put the
199
+values in the array.
200
+
201
+### Wide window without version filter (BAD when noisy)
202
+
203
+If 40-200k events / day are reported on stable releases, a
204
+30-day window without `AE_AGENT_VERSION` filter returns
205
+millions of records, most from old versions whose bugs are
206
+already fixed. Always pair wide windows with a version slice.
207
+
208
+## Discovery first, then query
209
+
210
+Use `info=true` to discover what fields and values the agent
211
+currently exposes:
212
+
213
+```json
214
+{ "info": true, "__logs_sources": "agent-events" }
215
+```
216
+
217
+Or use `last: 1` + a facet to enumerate values:
218
+
219
+```json
220
+{
221
+ "after": -86400, "before": 0, "last": 1,
222
+ "__logs_sources": "agent-events",
223
+ "facets": ["AE_AGENT_VERSION"]
224
+}
225
+```
226
+
227
+Then build the real query using the discovered values in
228
+`selections`.
229
+
230
+## Combining facets + selections
231
+
232
+`facets` (the array of field names you want grouped in the
233
+response) and `selections` (the structured filter) are
234
+independent. Common pattern: filter narrowly with
235
+`selections`, then `facets` over the narrow result to see
236
+per-field breakdowns.
237
+
238
+```json
239
+{
240
+ "selections": {
241
+ "AE_AGENT_HEALTH": ["crash-loop"],
242
+ "AE_AGENT_VERSION": ["v2.10.0"]
243
+ },
244
+ "facets": ["AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"]
245
+}
246
+```
247
+
248
+This filters to "crash-loop on v2.10.0", then shows the
249
+distribution by function, architecture, and OS in the result
250
+facets.
251
+
252
+## Quick checklist
253
+
254
+Before sending any query, verify:
255
+
256
+1. `__logs_sources` is set to `"agent-events"` (the journal namespace -- hardcoded constant, NOT the value of `${AGENT_EVENTS_HOSTNAME}`).
257
+2. `selections` contains at least ONE always-present field
258
+ (`AE_AGENT_HEALTH`, `AE_EXIT_CAUSE`, `AE_AGENT_VERSION`).
259
+3. Time window matches the use case (24h default, 7d for rare
260
+ crashes).
261
+4. If you used `query` for FTS, you have a structured
262
+ `selections` slice in front of it.
263
+5. `last` is set sensibly (200-500 for triage; 1 if you only
264
+ want facet counts).
.agents/skills/query-agent-events/recipes/INDEX.md
new
+29
@@ -0,0 +1,29 @@
1
+# Recipes -- INDEX
2
+
3
+Task-oriented recipes that combine `get-events.sh` +
4
+`analyze-events.sh` for common bug-investigation flows.
5
+
6
+| Recipe | Use case |
7
+|---|---|
8
+| `find-by-function.md` | "Is anyone hitting a crash in this specific function?" |
9
+| `find-by-version.md` | "Did crash X start in v2.10? Was it fixed in nightlies?" |
10
+| `find-related-to-work.md` | "We just fixed Y. Is anyone hitting Y in agent-events?" |
11
+
12
+For top-level "find all crashes" / "find all fatals" flows,
13
+see `../finding-crashes.md` and `../finding-fatals.md`.
14
+
15
+## Common preamble
16
+
17
+```bash
18
+cd <repo>
19
+source .agents/skills/query-agent-events/scripts/_lib.sh
20
+agentevents_load_env
21
+```
22
+
23
+## Live how-to rule
24
+
25
+If you investigate a question that isn't covered by the
26
+existing per-domain guides or these recipes AND your work
27
+involved non-trivial analysis, AUTHOR a how-to under
28
+`../how-tos/<slug>.md` and add a row to `../how-tos/INDEX.md`.
29
+See SKILL.md for the rule.
.agents/skills/query-agent-events/recipes/find-by-function.md
new
+109
@@ -0,0 +1,109 @@
1
+# Recipe: find events by fatal function
2
+
3
+Use case: "Is anyone hitting a crash in `function_name` /
4
+`source/file.c`?"
5
+
6
+## Quick path
7
+
8
+```bash
9
+.agents/skills/query-agent-events/scripts/get-events.sh \
10
+ --function 'rrdcontext_release,rrdcontext_dispatch_updates_to_main' \
11
+ --since '7d ago' \
12
+ --versions auto \
13
+ --last 200 \
14
+ --output /tmp/by-function.json
15
+```
16
+
17
+`--function` is a comma-separated list -- multiple values are
18
+OR'd via `selections.AE_FATAL_FUNCTION`. The query is fully
19
+indexed (no FTS), so 7d windows are cheap.
20
+
21
+Then:
22
+
23
+```bash
24
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
25
+ --input /tmp/by-function.json \
26
+ --by version
27
+# ...and...
28
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
29
+ --input /tmp/by-function.json \
30
+ --by signal
31
+```
32
+
33
+## By filename instead of function
34
+
35
+If you know the file but not the exact function:
36
+
37
+```bash
38
+# AE_FATAL_FILENAME is a `selections` field too.
39
+# get-events.sh doesn't have a --filename flag; use jq to
40
+# filter, OR construct the payload manually:
41
+
42
+payload=$(jq -nc '{
43
+ "after": -604800,
44
+ "before": 0,
45
+ "last": 500,
46
+ "__logs_sources": "agent-events",
47
+ "selections": {
48
+ "AE_FATAL_FILENAME": ["src/database/rrdcontext/rrdcontext-cleanup.c"]
49
+ }
50
+}')
51
+agentevents_query_function cloud "$payload" > /tmp/by-filename.json
52
+```
53
+
54
+## By symbol via FTS narrower
55
+
56
+When the symbol may be in the stack trace but not directly
57
+matched by `AE_FATAL_FUNCTION` (i.e. an inlined or downstream
58
+callee):
59
+
60
+```bash
61
+.agents/skills/query-agent-events/scripts/get-events.sh \
62
+ --health crash \
63
+ --query 'inlined_callee_name' \
64
+ --since '7d ago' \
65
+ --versions auto \
66
+ --output /tmp/by-symbol.json
67
+```
68
+
69
+`--health crash` is the structured slice (index-friendly);
70
+`--query` is the FTS narrower over the resulting subset. This
71
+is the right composition.
72
+
73
+## Triage flow
74
+
75
+1. **Get the dump** (as above).
76
+2. **Group by version** -- is this on stable, nightlies, or
77
+ both? Was it new in a recent version?
78
+3. **Group by signal** -- is it always SIGSEGV, or mixed?
79
+4. **Open one event** -- look at `AE_FATAL_STACK_TRACE`,
80
+ `AE_FATAL_LINE`, `AE_FATAL_MESSAGE`, `AE_FATAL_THREAD`.
81
+5. **Cross-reference source** -- read the function in this
82
+ repo at the cited line.
83
+6. **Group by environment** -- arch / os_family / kubernetes /
84
+ profile -- is it environment-specific?
85
+7. **Hypothesize, fix, ship.**
86
+
87
+## Common patterns
88
+
89
+- Multiple distinct `fatal_function` values that all crash in
90
+ the same source file -> the file has a structural issue
91
+ (state corruption, race condition, invariant violation).
92
+- One `fatal_function` value but mixed signals (SIGSEGV +
93
+ SIGBUS + SIGABRT) -> the function is a chokepoint hit by
94
+ many upstream paths.
95
+- Crashes in one function on parent profile only
96
+ (`AE_AGENT_PROFILE_0=parent`) -> the function is in the
97
+ streaming-receiver path.
98
+
99
+## Pitfalls
100
+
101
+- **`AE_FATAL_FUNCTION` is the function where `fatal()` was
102
+ called**, not necessarily the function where the crash
103
+ occurred. For signal crashes, it's the function name
104
+ recorded by the deadly-signal handler.
105
+- **Stack trace empty** for instantaneous crashes -- use line
106
+ / filename / function instead.
107
+- **Demangled symbols** -- the function name may have C++
108
+ decorations (e.g. `MyClass::method`). Try the demangled
109
+ form in `--function`.
.agents/skills/query-agent-events/recipes/find-by-version.md
new
+98
@@ -0,0 +1,98 @@
1
+# Recipe: find by version (regression spotter)
2
+
3
+Use case: "Did crash X appear in v2.10? Was it fixed in
4
+v2.10.0-100-nightly?"
5
+
6
+## Quick path
7
+
8
+Compare crash counts across explicit versions:
9
+
10
+```bash
11
+.agents/skills/query-agent-events/scripts/get-events.sh \
12
+ --health crash \
13
+ --versions 'v2.9.0,v2.10.0,v2.10.0-100-nightly,v2.10.0-130-nightly' \
14
+ --since '14d ago' \
15
+ --last 1 \
16
+ --facets 'AE_AGENT_VERSION,AE_FATAL_FUNCTION' \
17
+ --histogram AE_AGENT_VERSION \
18
+ --output /tmp/regression.json
19
+```
20
+
21
+`--last 1` because we only need the facet counts.
22
+`--histogram AE_AGENT_VERSION` adds time-bucketed counts per
23
+version, useful to spot when something disappeared.
24
+
25
+Then read the response:
26
+
27
+```bash
28
+jq '.facets[]
29
+ | select(.id=="AE_AGENT_VERSION")
30
+ | .options
31
+ | sort_by(-.count)
32
+ | .[]
33
+ | "\(.count)\t\(.id)"' /tmp/regression.json
34
+```
35
+
36
+## "When did this start?" (wide window)
37
+
38
+```bash
39
+.agents/skills/query-agent-events/scripts/get-events.sh \
40
+ --function 'specific_buggy_function' \
41
+ --version all \
42
+ --since '30d ago' \
43
+ --last 1 \
44
+ --facets 'AE_AGENT_VERSION' \
45
+ --histogram AE_AGENT_VERSION \
46
+ --output /tmp/origin.json
47
+```
48
+
49
+`--version all` is required -- the auto filter would mask
50
+older versions. The histogram tells you when (in time) and
51
+which versions started reporting.
52
+
53
+The first appearance in a specific version + nightly date
54
+gives you a commit window to bisect.
55
+
56
+## "Was it fixed?" check
57
+
58
+After a fix lands in nightly N:
59
+
60
+```bash
61
+.agents/skills/query-agent-events/scripts/get-events.sh \
62
+ --function 'specific_buggy_function' \
63
+ --versions 'v2.10.0-90-nightly,v2.10.0-100-nightly,v2.10.0-120-nightly' \
64
+ --since '7d ago' \
65
+ --last 1 \
66
+ --facets 'AE_AGENT_VERSION' \
67
+ --output /tmp/post-fix.json
68
+```
69
+
70
+Check if the count drops to ~zero in nightly N+1 and beyond.
71
+
72
+## Triage flow
73
+
74
+1. **Identify the candidate function / signal / cause** --
75
+ start from `find-by-function.md` or `finding-crashes.md`.
76
+2. **Wide-window query** with `--version all` to see the full
77
+ version distribution.
78
+3. **Read the histogram** -- when did this first appear?
79
+ When (if ever) did it stop?
80
+4. **Bisect commits** in the implicated version range.
81
+5. **Verify the fix** with a post-fix query.
82
+
83
+## Pitfalls
84
+
85
+- **Old version events from unupdated agents are normal**.
86
+ Don't conclude "the bug is back" from a single old-version
87
+ record -- check the date.
88
+- **`--versions auto` masks regressions** because it filters
89
+ to recent versions. Use `--version all` for "when did this
90
+ start?" investigations.
91
+- **Nightly numbers are commits-since-tag**. Higher number =
92
+ newer. Sorting nightly versions lexically is wrong; sort by
93
+ the embedded number (the auto-detection in `_lib.sh` does
94
+ this correctly).
95
+- **Different agents on the same install**: distinct
96
+ `AE_AGENT_ID` (Netdata machine GUID) but same `AE_HOST_ID`
97
+ (OS machine-id). Group by `AE_AGENT_ID` to get unique
98
+ installs, NOT by `AE_HOST_ID`.
.agents/skills/query-agent-events/recipes/find-related-to-work.md
new
+137
@@ -0,0 +1,137 @@
1
+# Recipe: find events related to current work
2
+
3
+Use case: "We're working on dbengine cleanup. Are there
4
+crashes in agent-events that match this area? What does the
5
+distribution look like?"
6
+
7
+This is the "is anyone hitting this?" template. Useful for
8
+prioritizing work and validating fixes.
9
+
10
+## Pattern
11
+
12
+You have a candidate area defined by some combination of:
13
+
14
+- a function name or symbol -> use `--function` or FTS
15
+- a source file path -> use selections on `AE_FATAL_FILENAME`
16
+- a code-path keyword that would appear in the stack trace
17
+ -> use `--query` (FTS narrower)
18
+- a thread name (e.g. `STREAM:N`, `CTXLOAD`, `APP-COLLECT`)
19
+ -> use selections on `AE_FATAL_THREAD`
20
+
21
+Combine the most-specific predicates first, then widen.
22
+
23
+## Worked example
24
+
25
+Working on dbengine page eviction:
26
+
27
+```bash
28
+# 1. Try by function -- the obvious symbols.
29
+.agents/skills/query-agent-events/scripts/get-events.sh \
30
+ --function 'rrdeng_page_descr_t,rrdeng_evict_pages,evict_main' \
31
+ --since '14d ago' \
32
+ --versions auto \
33
+ --output /tmp/related-pass1.json
34
+
35
+# 2. If pass 1 is sparse, widen to filename.
36
+payload=$(jq -nc '{
37
+ "after": -1209600, "before": 0, "last": 200,
38
+ "__logs_sources": "agent-events",
39
+ "selections": {
40
+ "AE_FATAL_FILENAME": [
41
+ "src/database/engine/cache.c",
42
+ "src/database/engine/pdc.c",
43
+ "src/database/engine/rrdengine.c"
44
+ ]
45
+ }
46
+}')
47
+agentevents_query_function cloud "$payload" > /tmp/related-pass2.json
48
+
49
+# 3. If still sparse, FTS over crash class.
50
+.agents/skills/query-agent-events/scripts/get-events.sh \
51
+ --health crash \
52
+ --query 'page_descr OR cache_evict OR rrdeng' \
53
+ --since '14d ago' \
54
+ --versions auto \
55
+ --output /tmp/related-pass3.json
56
+```
57
+
58
+Pass 1 (function) is the cheapest and most precise; pass 2
59
+widens to files; pass 3 is the FTS fallback.
60
+
61
+## Aggregation
62
+
63
+After collecting candidate events, look at the distribution:
64
+
65
+```bash
66
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
67
+ --input /tmp/related-pass2.json \
68
+ --by signal
69
+
70
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
71
+ --input /tmp/related-pass2.json \
72
+ --by version
73
+
74
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
75
+ --input /tmp/related-pass2.json \
76
+ --by fatal_function
77
+
78
+.agents/skills/query-agent-events/scripts/analyze-events.sh \
79
+ --input /tmp/related-pass2.json \
80
+ --by db_mode
81
+```
82
+
83
+Together this tells you:
84
+- how often we crash in this area;
85
+- whether it's all one signal or mixed;
86
+- which versions are affected;
87
+- which functions in the area are the dominant crash sites;
88
+- whether dbengine memory mode correlates.
89
+
90
+## Sample one event
91
+
92
+```bash
93
+jq '.data[0] as $row | .columns as $cols
94
+ | $cols | to_entries | sort_by(.value.index)
95
+ | map({(.key): $row[.value.index]}) | add' /tmp/related-pass2.json
96
+```
97
+
98
+Examine a representative `AE_FATAL_STACK_TRACE`,
99
+`AE_FATAL_MESSAGE`, `AE_FATAL_THREAD`. Cross-reference the
100
+specific commit that introduced the path you suspect.
101
+
102
+## Building a "before/after" comparison
103
+
104
+If you've landed a fix:
105
+
106
+```bash
107
+# Before-fix nightly counts.
108
+.agents/skills/query-agent-events/scripts/get-events.sh \
109
+ --function 'fixed_function' \
110
+ --versions 'v2.10.0,v2.10.0-100-nightly' \
111
+ --since '14d ago' \
112
+ --last 1 --facets 'AE_AGENT_VERSION' \
113
+ --output /tmp/before-fix.json
114
+
115
+# After-fix nightly counts.
116
+.agents/skills/query-agent-events/scripts/get-events.sh \
117
+ --function 'fixed_function' \
118
+ --versions 'v2.10.0-130-nightly,v2.10.0-160-nightly' \
119
+ --since '14d ago' \
120
+ --last 1 --facets 'AE_AGENT_VERSION' \
121
+ --output /tmp/after-fix.json
122
+```
123
+
124
+Compare counts. Significant drop -> fix is working.
125
+
126
+## Pitfalls
127
+
128
+- **Empty result for new code**: a function that just landed
129
+ in a nightly may not have crashed in the wild yet. Wide the
130
+ time window to 30 days OR wait for more agents to update.
131
+- **Old fixes that haven't propagated**: if your fix is in
132
+ v2.10.1 but most of the fleet runs v2.10.0, you'll still
133
+ see the old crashes for weeks.
134
+- **Recipe scoping**: don't forget to set
135
+ `"__logs_sources": "agent-events"` when constructing
136
+ payloads manually -- without it, the query targets
137
+ all-local-logs (huge and unrelated).
.agents/skills/query-agent-events/scripts/_lib.sh
new
+173
@@ -0,0 +1,173 @@
1
+#!/usr/bin/env bash
2
+# Helper library for the query-agent-events skill.
3
+#
4
+# Sources query-netdata-agents/scripts/_lib.sh and adds
5
+# agentevents_* helpers. Token-safe: bearers and the cloud
6
+# token never appear on the assistant-visible stdout.
7
+#
8
+# Usage:
9
+# source "$(git rev-parse --show-toplevel)/.agents/skills/query-agent-events/scripts/_lib.sh"
10
+# agentevents_load_env
11
+
12
+set -euo pipefail
13
+
14
+# Resolve self path (zsh + bash compatible).
15
+if [ -n "${ZSH_VERSION-}" ]; then
16
+ eval '_agentevents_lib_self="${(%):-%x}"'
17
+elif [ -n "${BASH_VERSION-}" ]; then
18
+ _agentevents_lib_self="${BASH_SOURCE[0]}"
19
+else
20
+ _agentevents_lib_self="$0"
21
+fi
22
+_agentevents_lib_dir="$(cd "$(dirname "$_agentevents_lib_self")" && pwd)"
23
+
24
+# Source query-netdata-agents helpers (transport + bearer mint).
25
+# shellcheck disable=SC1091
26
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
27
+
28
+# ---------------------------------------------------------------
29
+# Env loading
30
+
31
+agentevents_load_env() {
32
+ agents_load_env
33
+
34
+ : "${AGENT_EVENTS_HOSTNAME:?AGENT_EVENTS_HOSTNAME is empty -- see <repo>/.agents/ENV.md to set it.}"
35
+ : "${AGENT_EVENTS_NODE_ID:?AGENT_EVENTS_NODE_ID is empty -- see <repo>/.agents/ENV.md to set it.}"
36
+ : "${AGENT_EVENTS_MACHINE_GUID:?AGENT_EVENTS_MACHINE_GUID is empty -- see <repo>/.agents/ENV.md to set it.}"
37
+}
38
+
39
+# ---------------------------------------------------------------
40
+# Audit directory (gitignored under <repo>/.local/audits/)
41
+
42
+agentevents_audit_dir() {
43
+ local d
44
+ d="$(agents_audit_dir)/../query-agent-events"
45
+ mkdir -p "$d"
46
+ (cd "$d" && pwd)
47
+}
48
+
49
+# Journal namespace for selections / __logs_sources.
50
+# Hardcoded -- the ingestion server's log2journal --namespace
51
+# is always 'agent-events', regardless of the host's network
52
+# name (which lives in AGENT_EVENTS_HOSTNAME).
53
+agentevents_namespace() {
54
+ printf '%s' 'agent-events'
55
+}
56
+
57
+# ---------------------------------------------------------------
58
+# Function call: agentevents_query_function VIA PAYLOAD
59
+#
60
+# VIA is "cloud" or "agent". PAYLOAD is the systemd-journal
61
+# Function POST body (JSON string).
62
+# stdout: response body (JSON). No tokens leak.
63
+
64
+agentevents_query_function() {
65
+ local via="$1"
66
+ local payload="$2"
67
+
68
+ case "$via" in
69
+ cloud)
70
+ agents_query_cloud \
71
+ POST \
72
+ "/api/v2/nodes/${AGENT_EVENTS_NODE_ID}/function?function=systemd-journal" \
73
+ "$payload"
74
+ ;;
75
+ agent)
76
+ agents_query_agent \
77
+ --node "${AGENT_EVENTS_NODE_ID}" \
78
+ --host "${AGENT_EVENTS_HOSTNAME}:19999" \
79
+ --machine-guid "${AGENT_EVENTS_MACHINE_GUID}" \
80
+ POST \
81
+ "/api/v3/function?function=systemd-journal" \
82
+ "$payload"
83
+ ;;
84
+ *)
85
+ echo "agentevents_query_function: unknown VIA '$via' (use cloud|agent)" >&2
86
+ return 2
87
+ ;;
88
+ esac
89
+}
90
+
91
+# ---------------------------------------------------------------
92
+# Default version-filter computation.
93
+#
94
+# agentevents_compute_default_versions VIA SINCE_RELATIVE_SECONDS
95
+#
96
+# Queries the journal for the AE_AGENT_VERSION facet and picks:
97
+# - the latest stable (matches ^v\d+\.\d+\.\d+$, sorted desc)
98
+# - up to 3 latest nightlies (matches ^v\d+\.\d+\.\d+-\d+-nightly$, sorted desc by commit count)
99
+#
100
+# Outputs a JSON array of version strings to stdout.
101
+
102
+agentevents_compute_default_versions() {
103
+ local via="${1:-cloud}"
104
+ local since_secs="${2:-86400}"
105
+
106
+ local namespace
107
+ namespace="$(agentevents_namespace)"
108
+
109
+ local payload
110
+ payload=$(jq -nc \
111
+ --arg ns "$namespace" \
112
+ --argjson after "-${since_secs}" \
113
+ '{
114
+ "after": $after,
115
+ "before": 0,
116
+ "last": 1,
117
+ "__logs_sources": $ns,
118
+ "facets": ["AE_AGENT_VERSION"]
119
+ }')
120
+
121
+ local resp
122
+ resp="$(agentevents_query_function "$via" "$payload")"
123
+
124
+ # Extract the AE_AGENT_VERSION facet's option values.
125
+ # The response shape is documented in
126
+ # docs/netdata-ai/skills/query-netdata-cloud/query-logs.md
127
+ # under "Response shape".
128
+ echo "$resp" | jq -c '
129
+ ([.facets[]? | select(.id=="AE_AGENT_VERSION") | .options[]?.id] // [])
130
+ as $all
131
+ | (
132
+ ($all | map(select(test("^v\\d+\\.\\d+\\.\\d+$"))) | sort | reverse | .[0:1])
133
+ + ($all | map(select(test("^v\\d+\\.\\d+\\.\\d+-\\d+-nightly$")))
134
+ | sort_by(. | capture("-(?<n>\\d+)-nightly").n | tonumber)
135
+ | reverse | .[0:3])
136
+ )'
137
+}
138
+
139
+# ---------------------------------------------------------------
140
+# No-token-leak self-test.
141
+
142
+agentevents_selftest_no_token_leak() {
143
+ # Drive the public wrappers with sentinel values; assert no
144
+ # sentinel ever appears on captured stdout.
145
+ local sentinel="deadbeef-1234-5678-9abc-def012345678"
146
+
147
+ # Set sentinels in the environment that COULD leak if a
148
+ # wrapper logged its inputs.
149
+ local saved_token="${NETDATA_CLOUD_TOKEN:-}"
150
+ local saved_node="${AGENT_EVENTS_NODE_ID:-}"
151
+
152
+ NETDATA_CLOUD_TOKEN="$sentinel"
153
+ AGENT_EVENTS_NODE_ID="$sentinel"
154
+ export NETDATA_CLOUD_TOKEN AGENT_EVENTS_NODE_ID
155
+
156
+ # Run a no-op-ish payload through the helpers; capture stdout.
157
+ local out
158
+ out="$( {
159
+ agentevents_query_function cloud '{"info":true}' 2>/dev/null || true
160
+ } )"
161
+
162
+ # Restore.
163
+ NETDATA_CLOUD_TOKEN="$saved_token"
164
+ AGENT_EVENTS_NODE_ID="$saved_node"
165
+
166
+ if printf '%s' "$out" | grep -q "$sentinel"; then
167
+ echo "FAIL: sentinel $sentinel appeared on captured stdout" >&2
168
+ return 1
169
+ fi
170
+
171
+ echo "PASS: agentevents_selftest_no_token_leak"
172
+ return 0
173
+}
.agents/skills/query-agent-events/scripts/analyze-events.sh
new
+157
@@ -0,0 +1,157 @@
1
+#!/usr/bin/env bash
2
+# analyze-events.sh -- group-by stats over a dump from get-events.sh.
3
+#
4
+# Reads either the response envelope (if a single payload) or
5
+# raw rows (if pre-extracted). Emits a top-N counter table on
6
+# the requested dimension.
7
+
8
+set -euo pipefail
9
+
10
+usage() {
11
+ cat <<'EOF'
12
+analyze-events.sh --by <dim> [options]
13
+
14
+Required:
15
+ --by <dim> dimension to group by; one of:
16
+ signal, fatal_function, fatal_filename, version,
17
+ architecture, os_family, os_type, install_type,
18
+ db_mode, kubernetes, profile, aclk, health,
19
+ exit_cause, virtualization, chassis_type, host_cpus
20
+
21
+Options:
22
+ --input PATH path to the JSON dump (default: latest under
23
+ <repo>/.local/audits/query-agent-events/)
24
+ --top N top N values (default 20)
25
+ --filter "K=V" extra client-side filter (repeatable);
26
+ e.g. --filter "AE_OS_FAMILY=ubuntu"
27
+ --format text|json (default text)
28
+ -h, --help
29
+
30
+Tip: --by signal groups by AE_FATAL_SIGNAL_CODE (signal crashes).
31
+For non-signal events, the value will be empty.
32
+EOF
33
+}
34
+
35
+# Map --by alias -> AE_* field name.
36
+field_for_dim() {
37
+ local dim="$1"
38
+ case "$dim" in
39
+ signal) echo AE_FATAL_SIGNAL_CODE ;;
40
+ fatal_function) echo AE_FATAL_FUNCTION ;;
41
+ fatal_filename) echo AE_FATAL_FILENAME ;;
42
+ version) echo AE_AGENT_VERSION ;;
43
+ architecture) echo AE_HOST_ARCHITECTURE ;;
44
+ os_family) echo AE_OS_FAMILY ;;
45
+ os_type) echo AE_OS_TYPE ;;
46
+ install_type) echo AE_AGENT_INSTALL_TYPE ;;
47
+ db_mode) echo AE_AGENT_DB_MODE ;;
48
+ kubernetes) echo AE_AGENT_KUBERNETES ;;
49
+ profile) echo AE_AGENT_PROFILE_0 ;;
50
+ aclk) echo AE_AGENT_ACLK ;;
51
+ health) echo AE_AGENT_HEALTH ;;
52
+ exit_cause) echo AE_EXIT_CAUSE ;;
53
+ virtualization) echo AE_HOST_VIRTUALIZATION ;;
54
+ chassis_type) echo AE_HW_CHASSIS_TYPE ;;
55
+ host_cpus) echo AE_HOST_SYSTEM_CPUS ;;
56
+ *) echo "Unknown --by '$dim'" >&2; exit 2 ;;
57
+ esac
58
+}
59
+
60
+BY=
61
+INPUT=
62
+TOP=20
63
+FORMAT=text
64
+declare -a FILTERS=()
65
+
66
+while [ $# -gt 0 ]; do
67
+ case "$1" in
68
+ --by) BY="$2"; shift 2 ;;
69
+ --input) INPUT="$2"; shift 2 ;;
70
+ --top) TOP="$2"; shift 2 ;;
71
+ --filter) FILTERS+=("$2"); shift 2 ;;
72
+ --format) FORMAT="$2"; shift 2 ;;
73
+ -h|--help) usage; exit 0 ;;
74
+ *) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;;
75
+ esac
76
+done
77
+
78
+[ -z "$BY" ] && { usage >&2; exit 2; }
79
+
80
+FIELD="$(field_for_dim "$BY")"
81
+
82
+# shellcheck source=SCRIPTDIR/_lib.sh disable=SC1091
83
+source "$(cd "$(dirname "$0")" && pwd)/_lib.sh"
84
+
85
+# Pick the most recent dump if none given.
86
+if [ -z "$INPUT" ]; then
87
+ audit_dir="$(agentevents_audit_dir)"
88
+ # shellcheck disable=SC2012 # ls -1t is fine for *.json under audit_dir; find pipeline is overkill
89
+ INPUT="$(ls -1t "$audit_dir"/*.json 2>/dev/null | head -1 || true)"
90
+ [ -z "$INPUT" ] && {
91
+ echo "No input dump found under $audit_dir; pass --input PATH" >&2
92
+ exit 2
93
+ }
94
+ echo "[analyze-events] using $INPUT" >&2
95
+fi
96
+
97
+# The Function envelope has top-level `data` (rows of arrays)
98
+# and `columns` (name -> {index, ...}).
99
+#
100
+# Two paths:
101
+# - if the file looks like a Function envelope, project rows
102
+# via the columns map to extract FIELD;
103
+# - if it's a flat array of objects (pre-extracted), use jq
104
+# directly.
105
+
106
+# Build filter expression.
107
+filter_expr='true'
108
+for f in "${FILTERS[@]}"; do
109
+ k="${f%%=*}"
110
+ v="${f#*=}"
111
+ filter_expr="$filter_expr and (.\"$k\" == \"$v\")"
112
+done
113
+
114
+# Detect format and project.
115
+records=$(jq -c --arg field "$FIELD" '
116
+ if (type == "object" and has("columns") and has("data")) then
117
+ .columns as $c
118
+ | ($c | to_entries
119
+ | map({(.key): (.value.index)})
120
+ | add) as $idx
121
+ | (.data // [])
122
+ | map(
123
+ . as $row
124
+ | reduce ($idx | keys_unsorted)[] as $k
125
+ ({}; .[$k] = $row[$idx[$k]])
126
+ )
127
+ elif (type == "array" and (.[0]? | type == "object")) then
128
+ .
129
+ else
130
+ []
131
+ end
132
+' "$INPUT")
133
+
134
+# Apply filters and group.
135
+result=$(printf '%s' "$records" | jq -c --arg field "$FIELD" --argjson top "$TOP" "
136
+ map(select($filter_expr))
137
+ | group_by(.[\$field] // \"\")
138
+ | map({key: (.[0][\$field] // \"(empty)\"), count: length})
139
+ | sort_by(-.count)
140
+ | .[:\$top]
141
+")
142
+
143
+case "$FORMAT" in
144
+ json)
145
+ echo "$result" | jq .
146
+ ;;
147
+ text|*)
148
+ echo
149
+ echo "Top $TOP by $BY ($FIELD):"
150
+ printf '%s\n' "----------------------------------------"
151
+ echo "$result" | jq -r '.[] | [.count, .key] | @tsv' \
152
+ | awk -F'\t' '{ printf "%8d %s\n", $1, $2 }'
153
+ echo "----------------------------------------"
154
+ total=$(echo "$result" | jq '[.[].count] | add // 0')
155
+ echo " Total in top: $total"
156
+ ;;
157
+esac
.agents/skills/query-agent-events/scripts/get-events.sh
new
+264
@@ -0,0 +1,264 @@
1
+#!/usr/bin/env bash
2
+# get-events.sh -- fetch events of interest from agent-events.
3
+#
4
+# Index-friendly defaults: 24h time, multi-value selections,
5
+# auto version filter (latest stable + latest 3 nightlies).
6
+#
7
+# Output: JSON dump under
8
+# <repo>/.local/audits/query-agent-events/<timestamp>.json
9
+
10
+set -euo pipefail
11
+
12
+usage() {
13
+ cat <<'EOF'
14
+get-events.sh [options]
15
+
16
+Transport:
17
+ --via cloud|agent (default: cloud)
18
+
19
+Time window:
20
+ --since '<n>h ago'|'<n>d ago'|<seconds> (default: 24h ago)
21
+ --before now|<seconds> (default: now)
22
+
23
+Filters (all are AND'd; values within a flag are OR'd):
24
+ --health <classes> comma-separated; common: all, crash, healthy
25
+ (default: all)
26
+ "crash" -> crash-first,crash-loop,crash-repeated,crash-entered
27
+ "healthy" -> healthy-first,healthy-loop,healthy-repeated,healthy-recovered
28
+ --exit-cause <causes> comma-separated; common: all, fatal, signal, graceful
29
+ (default: all)
30
+ "fatal" -> deliberate-exit class (OOM, disk full, etc.)
31
+ "signal" -> killed-hard variants
32
+ "graceful"-> exit instructed/updated/shutdown/no last status
33
+ --signal <values> comma-separated AE_FATAL_SIGNAL_CODE values
34
+ example: SIGSEGV/SEGV_MAPERR,SIGBUS/BUS_OBJERR
35
+ --function <names> comma-separated AE_FATAL_FUNCTION values
36
+ --version <spec> "auto" (default), "all", or a regex
37
+ auto -> latest stable + latest 3 nightlies (computed)
38
+ all -> no version filter
39
+ else -> regex applied client-side after fetch
40
+ (multi-value selections require explicit values;
41
+ use --versions for that)
42
+ --versions <list> explicit comma-separated AE_AGENT_VERSION values
43
+ (overrides --version's auto/all/regex)
44
+ --arch <values> comma-separated AE_HOST_ARCHITECTURE values
45
+ --os-family <values> comma-separated AE_OS_FAMILY values
46
+ --query <fts> residual FTS narrower (after structured filters)
47
+ --facets <names> comma-separated; included in response for grouping
48
+
49
+Response control:
50
+ --last N page size, default 500
51
+ --histogram FIELD add a histogram bucket on FIELD
52
+
53
+Output:
54
+ --output PATH path to write the JSON dump (default: auto under .local/audits/...)
55
+
56
+Other:
57
+ -h, --help this message
58
+ -v, --verbose show the constructed payload before fetching
59
+EOF
60
+}
61
+
62
+# ---------------------------------------------------------------
63
+# Argument parsing.
64
+
65
+VIA=cloud
66
+SINCE='24h ago'
67
+BEFORE=now
68
+HEALTH=all
69
+EXIT_CAUSE=all
70
+SIGNAL=
71
+FUNCTION=
72
+VERSION=auto
73
+VERSIONS_EXPLICIT=
74
+ARCH=
75
+OS_FAMILY=
76
+QUERY=
77
+FACETS=
78
+LAST=500
79
+HISTOGRAM=
80
+OUTPUT=
81
+VERBOSE=0
82
+
83
+while [ $# -gt 0 ]; do
84
+ case "$1" in
85
+ --via) VIA="$2"; shift 2 ;;
86
+ --since) SINCE="$2"; shift 2 ;;
87
+ --before) BEFORE="$2"; shift 2 ;;
88
+ --health) HEALTH="$2"; shift 2 ;;
89
+ --exit-cause) EXIT_CAUSE="$2"; shift 2 ;;
90
+ --signal) SIGNAL="$2"; shift 2 ;;
91
+ --function) FUNCTION="$2"; shift 2 ;;
92
+ --version) VERSION="$2"; shift 2 ;;
93
+ --versions) VERSIONS_EXPLICIT="$2"; shift 2 ;;
94
+ --arch) ARCH="$2"; shift 2 ;;
95
+ --os-family) OS_FAMILY="$2"; shift 2 ;;
96
+ --query) QUERY="$2"; shift 2 ;;
97
+ --facets) FACETS="$2"; shift 2 ;;
98
+ --last) LAST="$2"; shift 2 ;;
99
+ --histogram) HISTOGRAM="$2"; shift 2 ;;
100
+ --output) OUTPUT="$2"; shift 2 ;;
101
+ -v|--verbose) VERBOSE=1; shift ;;
102
+ -h|--help) usage; exit 0 ;;
103
+ *) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;;
104
+ esac
105
+done
106
+
107
+# ---------------------------------------------------------------
108
+# Lib + env.
109
+
110
+# shellcheck source=SCRIPTDIR/_lib.sh disable=SC1091
111
+source "$(cd "$(dirname "$0")" && pwd)/_lib.sh"
112
+agentevents_load_env
113
+
114
+# ---------------------------------------------------------------
115
+# Time spec -> relative seconds.
116
+
117
+parse_time() {
118
+ local s="$1"
119
+ case "$s" in
120
+ now) echo 0 ;;
121
+ *' ago') # "24h ago", "7d ago", "30m ago"
122
+ local body="${s% ago}"
123
+ case "$body" in
124
+ *h) printf -- '-%d' "$(( ${body%h} * 3600 ))" ;;
125
+ *d) printf -- '-%d' "$(( ${body%d} * 86400 ))" ;;
126
+ *m) printf -- '-%d' "$(( ${body%m} * 60 ))" ;;
127
+ *) printf -- '-%d' "${body}" ;;
128
+ esac
129
+ ;;
130
+ -*|0*|[1-9]*) echo "$s" ;;
131
+ *) echo "Invalid time spec: $s" >&2; exit 2 ;;
132
+ esac
133
+}
134
+
135
+AFTER=$(parse_time "$SINCE")
136
+BEFORE_PARSED=$(parse_time "$BEFORE")
137
+
138
+# ---------------------------------------------------------------
139
+# Build selections.
140
+
141
+declare -a SELECTION_KEYS=()
142
+declare -A SELECTION_VALUES=()
143
+
144
+set_selection() {
145
+ local key="$1"
146
+ local csv="$2"
147
+ [ -z "$csv" ] && return
148
+ SELECTION_KEYS+=("$key")
149
+ SELECTION_VALUES[$key]="$csv"
150
+}
151
+
152
+case "$HEALTH" in
153
+ all|"") ;;
154
+ crash) set_selection AE_AGENT_HEALTH "crash-first,crash-loop,crash-repeated,crash-entered" ;;
155
+ healthy) set_selection AE_AGENT_HEALTH "healthy-first,healthy-loop,healthy-repeated,healthy-recovered" ;;
156
+ *) set_selection AE_AGENT_HEALTH "$HEALTH" ;;
157
+esac
158
+
159
+case "$EXIT_CAUSE" in
160
+ all|"") ;;
161
+ fatal)
162
+ set_selection AE_EXIT_CAUSE \
163
+"no last status,out of memory,disk full,disk almost full,disk read-only,already running,fatal on start,fatal on exit,fatal and exit,exit timeout"
164
+ ;;
165
+ signal)
166
+ set_selection AE_EXIT_CAUSE \
167
+"deadly signal,deadly signal on start,deadly signal on exit,deadly signal and exit,killed hard,killed hard on start,killed hard on shutdown,killed hard on exit,killed hard on update,killed hard low ram,killed fatal"
168
+ ;;
169
+ graceful)
170
+ set_selection AE_EXIT_CAUSE \
171
+"exit instructed,exit and updated,exit on system shutdown,exit to update,exit no reason,no last status"
172
+ ;;
173
+ *)
174
+ set_selection AE_EXIT_CAUSE "$EXIT_CAUSE"
175
+ ;;
176
+esac
177
+
178
+[ -n "$SIGNAL" ] && set_selection AE_FATAL_SIGNAL_CODE "$SIGNAL"
179
+[ -n "$FUNCTION" ] && set_selection AE_FATAL_FUNCTION "$FUNCTION"
180
+[ -n "$ARCH" ] && set_selection AE_HOST_ARCHITECTURE "$ARCH"
181
+[ -n "$OS_FAMILY" ] && set_selection AE_OS_FAMILY "$OS_FAMILY"
182
+
183
+# Version handling.
184
+if [ -n "$VERSIONS_EXPLICIT" ]; then
185
+ set_selection AE_AGENT_VERSION "$VERSIONS_EXPLICIT"
186
+elif [ "$VERSION" = "auto" ]; then
187
+ echo "[get-events] computing default version filter (latest stable + latest 3 nightlies)..." >&2
188
+ versions_json="$(agentevents_compute_default_versions "$VIA" "${AFTER#-}")"
189
+ if [ "$(echo "$versions_json" | jq 'length')" -gt 0 ]; then
190
+ versions_csv="$(echo "$versions_json" | jq -r 'join(",")')"
191
+ echo "[get-events] auto versions: $versions_csv" >&2
192
+ set_selection AE_AGENT_VERSION "$versions_csv"
193
+ else
194
+ echo "[get-events] auto version detection found no versions; proceeding without version filter" >&2
195
+ fi
196
+elif [ "$VERSION" = "all" ]; then
197
+ : # no filter
198
+else
199
+ # Pattern -- best-effort: not multi-value selection, leave it
200
+ # to the caller to client-side-filter the dump after fetch.
201
+ echo "[get-events] --version <regex> is not pushed to the server; filter the resulting JSON with jq" >&2
202
+fi
203
+
204
+# ---------------------------------------------------------------
205
+# Compose payload.
206
+
207
+ns="$(agentevents_namespace)"
208
+
209
+# Build selections object using jq.
210
+SELECTIONS_JSON='{}'
211
+for key in "${SELECTION_KEYS[@]}"; do
212
+ csv="${SELECTION_VALUES[$key]}"
213
+ SELECTIONS_JSON=$(echo "$SELECTIONS_JSON" | jq --arg k "$key" --arg v "$csv" \
214
+ '.[$k] = ($v | split(","))')
215
+done
216
+
217
+# Build the full payload.
218
+PAYLOAD=$(jq -nc \
219
+ --argjson after "$AFTER" \
220
+ --argjson before "$BEFORE_PARSED" \
221
+ --argjson last "$LAST" \
222
+ --arg ns "$ns" \
223
+ --argjson selections "$SELECTIONS_JSON" \
224
+ --arg query "$QUERY" \
225
+ --arg facets_csv "$FACETS" \
226
+ --arg histogram "$HISTOGRAM" \
227
+ '
228
+ {
229
+ "after": $after,
230
+ "before": $before,
231
+ "last": $last,
232
+ "direction": "backward",
233
+ "__logs_sources": $ns
234
+ }
235
+ | (if ($selections | length) > 0 then .selections = $selections else . end)
236
+ | (if ($query | length) > 0 then .query = $query else . end)
237
+ | (if ($facets_csv | length) > 0 then .facets = ($facets_csv | split(",")) else . end)
238
+ | (if ($histogram | length) > 0 then .histogram = $histogram else . end)
239
+ ')
240
+
241
+if [ "$VERBOSE" -eq 1 ]; then
242
+ echo "[get-events] payload:" >&2
243
+ echo "$PAYLOAD" | jq . >&2
244
+fi
245
+
246
+# ---------------------------------------------------------------
247
+# Output path.
248
+
249
+if [ -z "$OUTPUT" ]; then
250
+ audit_dir="$(agentevents_audit_dir)"
251
+ OUTPUT="$audit_dir/$(date -u +%Y%m%dT%H%M%SZ).json"
252
+fi
253
+
254
+# ---------------------------------------------------------------
255
+# Fetch.
256
+
257
+echo "[get-events] fetching via $VIA (output: $OUTPUT)..." >&2
258
+agentevents_query_function "$VIA" "$PAYLOAD" > "$OUTPUT"
259
+
260
+rows="$(jq '.data | length // 0' "$OUTPUT" 2>/dev/null || echo 0)"
261
+echo "[get-events] wrote $rows row(s) to $OUTPUT" >&2
262
+
263
+# Print path on stdout for piping.
264
+echo "$OUTPUT"
.agents/skills/query-agent-events/scripts/redact-events.sh
new
+87
@@ -0,0 +1,87 @@
1
+#!/usr/bin/env bash
2
+# redact-events.sh -- opt-in redaction of identifying fields.
3
+#
4
+# Replaces UUID-shaped values in known identifying fields with
5
+# stable placeholders like <redacted-A>, <redacted-B>, ... so
6
+# patterns are still identifiable (same machine_guid -> same
7
+# placeholder within the dump) but the raw value is gone.
8
+#
9
+# Use this only when sharing a dump externally. Default
10
+# workflow keeps raw events under .local/audits/...
11
+# (gitignored).
12
+
13
+set -euo pipefail
14
+
15
+usage() {
16
+ cat <<'EOF'
17
+redact-events.sh [--input PATH] [--output PATH]
18
+
19
+Redacts identifying fields in a Function-envelope JSON dump or
20
+a flat array of objects.
21
+
22
+Identifying fields redacted:
23
+ AE_AGENT_ID, AE_HOST_ID, AE_AGENT_NODE_ID, AE_AGENT_CLAIM_ID,
24
+ AE_HOST_BOOT_ID, AE_AGENT_EPHEMERAL_ID, AE_HW_SYS_UUID,
25
+ _MACHINE_ID, _BOOT_ID
26
+
27
+Stable mapping: same value -> same placeholder within the dump.
28
+
29
+Defaults:
30
+ --input stdin
31
+ --output stdout
32
+EOF
33
+}
34
+
35
+INPUT=
36
+OUTPUT=
37
+while [ $# -gt 0 ]; do
38
+ case "$1" in
39
+ --input) INPUT="$2"; shift 2 ;;
40
+ --output) OUTPUT="$2"; shift 2 ;;
41
+ -h|--help) usage; exit 0 ;;
42
+ *) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;;
43
+ esac
44
+done
45
+
46
+if [ -n "$INPUT" ]; then exec < "$INPUT"; fi
47
+if [ -n "$OUTPUT" ]; then exec > "$OUTPUT"; fi
48
+
49
+jq -c '
50
+def redact_fields:
51
+ ["AE_AGENT_ID","AE_HOST_ID","AE_AGENT_NODE_ID","AE_AGENT_CLAIM_ID",
52
+ "AE_HOST_BOOT_ID","AE_AGENT_EPHEMERAL_ID","AE_HW_SYS_UUID",
53
+ "_MACHINE_ID","_BOOT_ID"];
54
+
55
+def alphabet:
56
+ ["A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P",
57
+ "Q","R","S","T","U","V","W","X","Y","Z"];
58
+
59
+. as $root
60
+| reduce (redact_fields[]) as $f (
61
+ {state: {map: {}, idx: 0}, root: $root};
62
+ . as {state: $st, root: $r}
63
+ | ($r | .. | objects | select(has($f)) | .[$f] | tostring) as $vals
64
+ | reduce $vals as $v (
65
+ .;
66
+ if (.state.map | has($v)) then
67
+ .
68
+ else
69
+ .state.map[$v] = ("<redacted-" + (alphabet[(.state.idx % 26)]) + (if .state.idx >= 26 then ((.state.idx / 26 | floor) | tostring) else "" end) + ">")
70
+ | .state.idx += 1
71
+ end
72
+ )
73
+ )
74
+| .root as $r
75
+| .state.map as $m
76
+| ($r | walk(
77
+ if type == "object" then
78
+ with_entries(
79
+ if (.key | IN("AE_AGENT_ID","AE_HOST_ID","AE_AGENT_NODE_ID","AE_AGENT_CLAIM_ID","AE_HOST_BOOT_ID","AE_AGENT_EPHEMERAL_ID","AE_HW_SYS_UUID","_MACHINE_ID","_BOOT_ID"))
80
+ then .value = ($m[(.value | tostring)] // .value)
81
+ else .
82
+ end
83
+ )
84
+ else .
85
+ end
86
+ ))
87
+'
.agents/skills/query-agent-events/transports.md
new
+182
@@ -0,0 +1,182 @@
1
+# Transports
2
+
3
+Three ways to query agent-events. The first two are scripted
4
+in this skill; the third is operator-only.
5
+
6
+## Priority order
7
+
8
+1. **Cloud API** -- proxied through Netdata Cloud at the
9
+ space hosting agent-events. **Default for the team.**
10
+2. **Direct agent API** -- against the agent-events node's
11
+ own HTTP. Used when bypassing the Cloud is acceptable
12
+ (latency, debugging the proxy).
13
+3. **ssh to the host** -- operator-only path. Costa-only.
14
+ Mentioned here for completeness; this skill does NOT
15
+ ship a scripted ssh transport.
16
+
17
+## What each transport calls
18
+
19
+All three speak the same `systemd-journal` Function. The
20
+payload shape (`after`, `before`, `last`, `query`, `facets`,
21
+`histogram`, `__logs_sources`, `selections`, ...) is identical
22
+across transports. The Function payload is documented once at:
23
+
24
+- `<repo>/docs/netdata-ai/skills/query-netdata-cloud/query-logs.md`
25
+
26
+That doc is the canonical reference for:
27
+- payload keys + types,
28
+- the **`selections` multi-value field-filter** (AND across
29
+ fields, OR across values) -- this skill leans on it heavily,
30
+- the response envelope (top-level `data` rows, `columns` map,
31
+ `facets`, `histogram`, etc.).
32
+
33
+This skill EXTENDS that doc with agent-events specifics: which
34
+AE_* fields are best as facets, what default `selections`
35
+predicate to use, what `__logs_sources` value to set.
36
+
37
+## Cloud API (transport 1)
38
+
39
+### Endpoint
40
+
41
+`POST https://${NETDATA_CLOUD_HOSTNAME}/api/v2/nodes/${AGENT_EVENTS_NODE_ID}/function?function=systemd-journal`
42
+
43
+Auth: `Authorization: Bearer ${NETDATA_CLOUD_TOKEN}`.
44
+
45
+### Helper (from query-netdata-cloud)
46
+
47
+```bash
48
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-cloud/scripts/_lib.sh"
49
+cloud_load_env
50
+cloud_query \
51
+ "/api/v2/nodes/${AGENT_EVENTS_NODE_ID}/function?function=systemd-journal" \
52
+ "$PAYLOAD"
53
+```
54
+
55
+Or the `query-netdata-agents` skill's wrapper, which works
56
+identically and routes through Cloud when configured:
57
+
58
+```bash
59
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
60
+agents_load_env
61
+agents_call_function "$AGENT_EVENTS_NODE_ID" systemd-journal "$PAYLOAD"
62
+```
63
+
64
+### Pros / cons
65
+
66
+- **Pro:** team-accessible (no per-host SSH); central auth via
67
+ `NETDATA_CLOUD_TOKEN`; works from any network.
68
+- **Con:** slight latency vs direct agent; rate-limited at the
69
+ Cloud edge; subject to Cloud-side query timeout.
70
+
71
+### When to use
72
+
73
+- The default for the team.
74
+- Anything you want to share later (Cloud requests are
75
+ loggable / repeatable).
76
+
77
+## Direct agent API (transport 2)
78
+
79
+### Endpoint
80
+
81
+`POST http://${AGENT_EVENTS_HOSTNAME}:19999/api/v3/function?function=systemd-journal`
82
+
83
+Auth: bearer token minted from the Cloud token. The
84
+`agents_query_agent` helper handles minting + caching
85
+transparently.
86
+
87
+### Helper
88
+
89
+```bash
90
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
91
+agents_load_env
92
+
93
+agents_query_agent \
94
+ --node "$AGENT_EVENTS_NODE_ID" \
95
+ --host "$AGENT_EVENTS_HOSTNAME:19999" \
96
+ --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
97
+ POST '/api/v3/function?function=systemd-journal' "$PAYLOAD"
98
+```
99
+
100
+Output is the response body only. The bearer never reaches
101
+the assistant's captured stdout.
102
+
103
+### Pros / cons
104
+
105
+- **Pro:** no Cloud-edge round-trip; lower latency; agent's
106
+ own timeout (you set it in the body).
107
+- **Con:** only reachable from inside the network; requires
108
+ the agent to be reachable on port 19999.
109
+
110
+### When to use
111
+
112
+- Tight loops during local debugging (sub-second iteration).
113
+- When the Cloud edge is the bottleneck.
114
+
115
+## ssh to the host (transport 3 -- operator-only)
116
+
117
+This skill does NOT ship a scripted ssh transport. The
118
+operator (Costa) sometimes runs `journalctl` directly on the
119
+host:
120
+
121
+```bash
122
+ssh "$AGENT_EVENTS_HOSTNAME" \
123
+ sudo /usr/bin/journalctl --namespace=agent-events \
124
+ --since '24 hours ago' -o json
125
+```
126
+
127
+Notes:
128
+- The ssh host is `${AGENT_EVENTS_HOSTNAME}` (env-keyed; can be
129
+ an IP or DNS name). The journal namespace is `agent-events`
130
+ (hardcoded constant, set on the ingestion server's log2journal
131
+ invocation, NOT a function of the hostname).
132
+- Raw `journalctl` does NOT support multi-value field filters
133
+ (they are a Netdata-engine feature, not journald). If you
134
+ need AND-of-OR filtering, use transport 1 or 2.
135
+- This path requires sudo + a member of the `systemd-journal`
136
+ group on the ingestion host. Most team members do not have
137
+ this. Use transports 1 or 2 instead.
138
+
139
+## Default `__logs_sources` value
140
+
141
+Always set `__logs_sources` to the agent-events namespace name
142
+(`"agent-events"` -- a hardcoded constant set on the ingestion
143
+server's log2journal invocation; NOT derived from
144
+`${AGENT_EVENTS_HOSTNAME}`):
145
+
146
+```json
147
+{ "__logs_sources": "agent-events" }
148
+```
149
+
150
+Without this, the Function defaults to all-local-logs on the
151
+ingestion-server agent -- which is huge and unrelated.
152
+
153
+## What goes in `selections` for agent-events
154
+
155
+For the agent-events namespace, the most-useful index-friendly
156
+predicates (always present on every record):
157
+
158
+- `AE_VERSION` -- schema version anchor (always 28+).
159
+- `AE_AGENT_HEALTH` -- crash class (filter to `crash-*` for crashes).
160
+- `AE_EXIT_CAUSE` -- exit reason (filter to specific causes).
161
+- `AE_AGENT_VERSION` -- producing agent version (regression slicing).
162
+- `AE_FATAL_SIGNAL_CODE` -- non-empty for signal crashes.
163
+- `AE_FATAL_FUNCTION` / `AE_FATAL_FILENAME` -- localize to a
164
+ function or file.
165
+- `AE_HOST_ARCHITECTURE` / `AE_OS_FAMILY` / `AE_AGENT_INSTALL_TYPE`
166
+ -- arch / distro / packaging slicers.
167
+- `AE_AGENT_PROFILE_0` -- standalone / parent / child / iot.
168
+- `AE_AGENT_KUBERNETES` -- k8s-specific.
169
+- `AE_AGENT_ACLK` -- cloud-claimed vs not.
170
+
171
+See `AE_FIELDS.md` for the full field map and enum meanings.
172
+
173
+## See also
174
+
175
+- `<repo>/docs/netdata-ai/skills/query-netdata-cloud/query-logs.md`
176
+ -- canonical Function payload shape and the `selections`
177
+ multi-value filter section.
178
+- `<repo>/docs/netdata-ai/skills/query-netdata-agents/query-logs.md`
179
+ -- direct-agent transport details.
180
+- `query-discipline.md` (this skill) -- how to compose
181
+ index-friendly queries against agent-events.
182
+- `update-cadence.md` (this skill) -- when events arrive and why.
.agents/skills/query-agent-events/update-cadence.md
new
+185
@@ -0,0 +1,185 @@
1
+# Update cadence
2
+
3
+agent-events is **not real-time**. Understanding when events
4
+arrive is essential for query design.
5
+
6
+## The "after-the-fact" model
7
+
8
+When a Netdata agent crashes, the crash itself does NOT post
9
+to the ingestion server. The agent has to start again, read
10
+its on-disk status file (the previous session's state), and
11
+POST that to the ingestion server.
12
+
13
+Concretely (`src/daemon/status-file.c`):
14
+
15
+1. **During a session** -- the agent maintains its current
16
+ status in memory and writes a snapshot to
17
+ `/var/lib/netdata/status-netdata.dat` periodically (see
18
+ "Disk save cadence" below).
19
+2. **On exit / crash** -- a final snapshot is written if
20
+ possible (signal-async-safe writer at
21
+ `<repo>/src/daemon/status-file-io.c`). For SIGKILL / OOM-kill,
22
+ no final snapshot is written -- the prior periodic snapshot
23
+ is what gets reported.
24
+3. **On NEXT start** -- the new agent reads the on-disk
25
+ status, computes `agent_health`, classifies `exit_cause`,
26
+ then POSTs to the ingestion server (`status-file.c:988`).
27
+
28
+So a crash at 14:00 produces a journal record on the ingestion
29
+server only after the agent restarts -- which might happen
30
+seconds, minutes, hours, or days later (depending on
31
+operator policy and whether the agent loops).
32
+
33
+### Implication for queries
34
+
35
+- **"Last hour"** is misleading -- it misses crashes from
36
+ agents that crashed in the last hour but haven't restarted
37
+ yet.
38
+- **"Last 24 hours"** is the natural unit -- accommodates
39
+ typical restart latency AND aligns with the dedup window.
40
+- For rare crashes (1-per-few-days class), expand to 7+ days.
41
+
42
+## Disk save cadence
43
+
44
+`<repo>/src/daemon/status-file.c:835`:
45
+
46
+> "Update disk footprint at most once every 10 minutes (600
47
+> seconds)"
48
+
49
+The in-memory snapshot is refreshed at most every 10 minutes.
50
+Each refresh triggers a save to
51
+`/var/lib/netdata/status-netdata.dat` (atomic temp+rename via
52
+`<repo>/src/daemon/status-file-io.c`). Saves also happen on:
53
+
54
+- start-up (before claiming the new session);
55
+- exit (graceful shutdown writes the final state);
56
+- inside a deadly-signal handler (best-effort write before
57
+ the process dies).
58
+
59
+So the on-disk snapshot a restarting agent reads is at most
60
+~10 minutes old, plus any signal-handler-time updates.
61
+
62
+## 23h client-side dedup
63
+
64
+`<repo>/src/daemon/status-file-dedup.c:11`:
65
+
66
+```c
67
+#define REPORT_EVENTS_EVERY (86400 - 3600) // -1 hour to tolerate cron randomness
68
+```
69
+
70
+= 82800 seconds = 23 hours.
71
+
72
+Each agent maintains a small dedup table at
73
+`/var/lib/netdata/dedup-netdata.dat` (binary; see `:13-22`):
74
+50 slots, each storing `(hash, sentry-flag, timestamp)`.
75
+
76
+The hash includes (`status-file-dedup.c:42-95`):
77
+
78
+- `version` (schema), `status`, `signal_code`, `profile`,
79
+ `exit_reason`, `db_mode`, `db_tiers`, `kubernetes`,
80
+ `sentry_available`, `sentry_fatal`,
81
+- `host_id` (Netdata machine GUID), `machine_id` (OS),
82
+ `worker_job_id`, `line` (in source),
83
+- `version` string, `filename`, `function`, `stack_trace`
84
+ (with hex addresses zeroed for hashing only),
85
+ `thread`,
86
+- `msg`, `cause`.
87
+
88
+So **each unique combination of these fields** has its own
89
+dedup slot. If the agent saw the same combination within 23h,
90
+the new POST is suppressed (`dedup_already_posted` returns
91
+`true` -> producer skips the POST).
92
+
93
+### Implication: 1 record per agent per event-class per day
94
+
95
+A given agent will report the same crash signature at most
96
+once per 23h window. So at the journal level:
97
+
98
+- 1 distinct crash class per restart, per agent, per day.
99
+- Different agents posting the same crash signature both
100
+ arrive (the dedup is **client-side**, not server-side).
101
+- Different crash signatures from the same agent within 23h
102
+ all arrive (different hashes).
103
+
104
+This is why "last 24 hours" is the natural query window: the
105
+dedup makes daily counts meaningful (each agent contributes
106
+at most one record per signature).
107
+
108
+### Stack-trace anonymization is dedup-only
109
+
110
+`status-file-dedup.c:26-36`:
111
+
112
+```c
113
+static void stack_trace_anonymize(char *s) {
114
+ char *p = s;
115
+ while (*p && (p = strstr(p, "0x"))) {
116
+ p[1] = '0';
117
+ p += 2;
118
+ while(isxdigit((uint8_t)*p)) *p++ = '0';
119
+ }
120
+}
121
+```
122
+
123
+Hex addresses in the stack trace are zeroed ONLY when
124
+computing the dedup hash. The journal-emitted
125
+`AE_FATAL_STACK_TRACE` retains real addresses (useful for
126
+debugging; sensitive when sharing externally -- see
127
+`redact-events.sh`).
128
+
129
+## Dataset volume
130
+
131
+40k-200k events / day on stable releases. Total fleet ~1.5M
132
+agents (not all restart daily). The dataset is large and
133
+spread across days.
134
+
135
+### Implication: index-friendly queries are mandatory
136
+
137
+A 7-day FTS-only query over the agent-events namespace will
138
+scan ~1M records on a high-volume week. With structured
139
+`selections` filters (e.g. `AE_AGENT_HEALTH`, `AE_AGENT_VERSION`),
140
+the same query slices to thousands or tens of thousands of
141
+records before any FTS -- orders of magnitude cheaper.
142
+
143
+See `query-discipline.md` for the rule and worked examples.
144
+
145
+## Default time windows for the skill
146
+
147
+| Use case | Default window | Why |
148
+|---|---|---|
149
+| Routine crash triage | 24h | Aligns with 23h dedup; one record per agent per signature. |
150
+| Regression spotting | 24h | Same as above; group-by version. |
151
+| Rare-crash hunting | 7 days | Catches 1-per-few-days classes. |
152
+| "When did this start / get fixed?" | 14-30 days | Wide window with strong structured filters. Use sparingly. |
153
+
154
+## Default version filter
155
+
156
+Latest stable + latest 2-3 nightlies. The dataset is noisy
157
+because many unupdated agents report crashes that have been
158
+fixed. Filtering to recent versions focuses triage on bugs
159
+that still matter.
160
+
161
+`get-events.sh --versions auto` (the default) computes:
162
+1. Quick discovery query: 24h window, no version filter,
163
+ `AE_AGENT_VERSION` as a facet.
164
+2. From the facet result, pick the latest stable
165
+ (`v\d+\.\d+\.\d+` matching `^v2\.([89]|\d\d)\.\d+$` or the
166
+ newest by version sort) plus the top 3 nightlies
167
+ (`v\d+\.\d+\.\d+-\d+-nightly`).
168
+3. Re-run the main query with `selections.AE_AGENT_VERSION`
169
+ set to that list.
170
+
171
+`--all-versions` skips the auto filter for "when did X
172
+start?" investigations.
173
+`--versions <regex>` overrides with a custom pattern.
174
+
175
+## What this means in practice
176
+
177
+- A crash you just got reported in chat will appear on
178
+ agent-events anywhere from minutes to days later.
179
+- "Why don't I see this crash?" probably means "the agent
180
+ hasn't restarted yet, OR the dedup suppressed a duplicate
181
+ within 23h, OR the version filter excluded it".
182
+- High-cardinality crashes from old versions are mostly
183
+ noise; default-version-filtering keeps triage focused.
184
+- For "is this fixed in v2.X?", run `--versions auto` and
185
+ compare crash counts across the auto-selected versions.
.agents/skills/query-netdata-agents
new
+1
@@ -0,0 +1 @@
1
+../../docs/netdata-ai/skills/query-netdata-agents
\ No newline at end of file
.agents/skills/query-netdata-cloud
new
+1
@@ -0,0 +1 @@
1
+../../docs/netdata-ai/skills/query-netdata-cloud
\ No newline at end of file
.agents/sow/done/SOW-0003-20260503-query-agent-events-skill.md
new
+493
@@ -0,0 +1,493 @@
1
+# SOW-0003 - query-agent-events private skill
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: completed 2026-05-05. Skill shipped with verified producer-side field map (80+ fields), token-safe scripts (get-events / analyze-events / redact-events), index-friendly query discipline documented, and the multi-value `selections` capability lifted into both `query-netdata-{cloud,agents}/query-logs.md` so all callers can use it. Consumes `query-netdata-{cloud,agents}` for transport.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Build a **private developer skill** that lets a Netdata maintainer (or
14
+an AI assistant helping one) query, fetch, and analyze received
15
+agent-events submissions on the Netdata-operated ingestion server.
16
+
17
+This skill is intentionally NOT public. It lives at
18
+`<repo>/.agents/skills/query-agent-events/` only -- not under
19
+`docs/netdata-ai/skills/`. Reasons:
20
+
21
+- The data is operator-sensitive (machine GUIDs, claim IDs, cloud
22
+ metadata, hardware DMI fields) and the user does not want a
23
+ public "how to scrape Netdata's agent-events" doc.
24
+- The journalctl-via-ssh path included in the skill requires
25
+ privileged shell access to a specific host whose name lives only
26
+ in `.env`.
27
+- The skill is a maintainer triage tool, not a user-facing feature.
28
+
29
+The fit-for-purpose use cases the skill must support:
30
+
31
+- "What crashes is the fleet seeing in the last 24 hours, grouped
32
+ by exit cause?"
33
+- "Show me events from a specific agent (by `AE_AGENT_ID`)."
34
+- "What's the distribution of `AE_AGENT_HEALTH` values among
35
+ stable releases over the last 30 days?"
36
+- "Fetch a specific event by timestamp + machine GUID and pretty-
37
+ print its `AE_FATAL_*` fields."
38
+- "Compare crash counts across `AE_AGENT_VERSION` for v2.8+ to
39
+ spot regressions."
40
+
41
+### User Request
42
+
43
+> "the query-agent-events skill should not be public in
44
+> docs/netdata-ai. It should live in .agents/skills/ since this is
45
+> a developer tool, not an end-user tool. The skill should
46
+> document the direct journalctl method via ssh, but it the
47
+> destination IP and any other private info should be in .env.
48
+>
49
+> the journalctl method is documented in
50
+> .local/agent-events-journals.md - this is untrusted document,
51
+> not to be copied to a skill as-is. So, you need to review it,
52
+> you can try it..."
53
+
54
+The user added these `.env` keys (values stay in `.env`):
55
+
56
+- `AGENT_EVENTS_NC_SPACE`
57
+- `AGENT_EVENTS_HOSTNAME`
58
+- `AGENT_EVENTS_MACHINE_GUID`
59
+- `AGENT_EVENTS_NODE_ID`
60
+
61
+### Assistant Understanding
62
+
63
+Facts:
64
+
65
+- The Netdata Agent serializes a status document (schema v28) and
66
+ POSTs it to a public ingestion endpoint. The receiving service
67
+ persists each submission as a systemd journal entry in a
68
+ dedicated journal namespace whose name is documented in
69
+ `.local/agent-events-journals.md` (an untrusted draft).
70
+- Each status JSON dot-path is converted into a journal field
71
+ prefixed with `AE_` and the dots replaced with underscores
72
+ (per the untrusted draft, to be verified against
73
+ `src/libnetdata/log/log2journal/` and the producer-side
74
+ serialization).
75
+- Three transports are in scope:
76
+ 1. Cloud-proxied: uses
77
+ `query-netdata-cloud/query-logs.md` and
78
+ `query-netdata-cloud/query-functions.md` (delivered by
79
+ SOW-0010).
80
+ 2. Direct agent: uses
81
+ `query-netdata-agents/scripts/_lib.sh::agents_call_function`
82
+ (delivered by SOW-0010), with bearer auto-mint/refresh.
83
+ 3. journalctl-via-ssh: uses standard ssh + the journalctl
84
+ namespace flag. Destination host comes from `.env`.
85
+
86
+Inferences:
87
+
88
+- The journal field map and enum values in
89
+ `.local/agent-events-journals.md` are likely correct in spirit
90
+ but must be verified field-by-field against the producer code
91
+ and a sampled response before the skill encodes them as truth.
92
+- The default query the skill ships should be conservative
93
+ (last 24h, narrow facets) so a single `query` call does not
94
+ flood the ingestion server.
95
+
96
+Unknowns:
97
+
98
+- Whether `.local/agent-events-journals.md` is fully accurate
99
+ on every field-name mapping (especially edge cases:
100
+ array indices, deeply-nested paths, omitted-on-graceful-exit
101
+ fields).
102
+- Whether SOW-0010 ships a Cloud-proxied logs query helper ready
103
+ to consume (decision 1 of SOW-0010 directly governs this).
104
+- Whether the journalctl path returns the same field set as the
105
+ Netdata-via-Cloud path; small drift is likely (the journal
106
+ output includes systemd `_*` fields that the Function may
107
+ filter out).
108
+
109
+### Acceptance Criteria
110
+
111
+- `<repo>/.agents/skills/query-agent-events/SKILL.md` exists
112
+ with frontmatter triggers covering "agent events",
113
+ "agent-events", "status file", "crash reports", "fleet
114
+ crashes", "ingestion server".
115
+- `<repo>/.agents/skills/query-agent-events/scripts/` ships:
116
+ - `_lib.sh` (mirrors the legacy skill helper shape, prefix
117
+ `agentevents_`; sources `.env`, depends on the helpers from
118
+ `query-netdata-agents/scripts/_lib.sh`)
119
+ - `query-events.sh` -- thin wrapper around the `systemd-
120
+ journal` Function with flags `--via {cloud|agent|ssh}`,
121
+ `--last N`, `--after T`, `--before T`, `--query STR`,
122
+ `--source SEL`, `--facets a,b,c`, `--histogram FIELD`.
123
+ - `fetch-event.sh` -- single-event fetch by `AE_AGENT_ID` +
124
+ timestamp anchor.
125
+ - `summarize.sh` -- jq-driven facet/histogram pretty-print.
126
+- `<repo>/.agents/skills/query-agent-events/AE_FIELDS.md` exists
127
+ and documents the verified `AE_*` field map (and known
128
+ divergences from `.local/agent-events-journals.md`).
129
+- All raw outputs land under
130
+ `<repo>/.local/audits/query-agent-events/` (gitignored).
131
+- A small acceptance fixture: at least one real round-trip via
132
+ the Cloud transport and one via ssh, each producing a small
133
+ bundle that includes a `crash-*` event for a stable
134
+ (`AE_AGENT_VERSION` matches `^v2\.([89]|\d\d)\.`) release.
135
+- AGENTS.md "Project Skills Index" section adds a one-line entry
136
+ for `.agents/skills/query-agent-events/`.
137
+- Sensitive-data gate: the SOW, the SKILL.md, and every
138
+ committed script contain zero raw values for any of
139
+ `AGENT_EVENTS_*`, no machine GUIDs, no claim IDs, no public-
140
+ facing host names except those already in the open-source
141
+ code. Verified by pre-commit grep.
142
+
143
+## Analysis
144
+
145
+Sources to consult during stage 2 (already mostly read at stage 1):
146
+
147
+- `<repo>/src/daemon/status-file.{c,h}` (schema v28).
148
+- `<repo>/src/daemon/status-file-io.{c,h}`.
149
+- `<repo>/src/daemon/status-file-dmi.{c,h}`.
150
+- `<repo>/src/libnetdata/exit/exit_initiated.h` (exit_reason
151
+ enum).
152
+- `<repo>/src/libnetdata/log/log2journal/` (journal field naming
153
+ conventions; verify the `AE_` prefix story).
154
+- `<repo>/.local/agent-events-journals.md` (untrusted draft;
155
+ treat each claim as a hypothesis until cross-verified).
156
+- `<repo>/src/collectors/systemd-journal.plugin/...` (Function
157
+ shape).
158
+- The output of `query-netdata-agents/scripts/_lib.sh` from
159
+ SOW-0010 (consumer side).
160
+
161
+Risks:
162
+
163
+- Privacy: every fetched event carries identifying fields. The
164
+ skill MUST NOT copy real values into committed artifacts. A
165
+ redaction filter (`redact-events.sh`) ships as an opt-in
166
+ filter, not a default.
167
+- Untrusted-doc risk: copying field names verbatim from
168
+ `.local/agent-events-journals.md` without verification is
169
+ unsafe. Stage 2 must spot-check at least the top-20 most-used
170
+ fields and the all-the-enums against producer source.
171
+- Volume: a fleet of 1.5M+ daily Netdata installs producing
172
+ events at non-zero rates means naive "fetch all" calls would
173
+ return enormous payloads. Default queries must be narrow.
174
+- Schema drift: STATUS_FILE_VERSION will keep moving. Treat
175
+ unknown fields as opaque pass-through; key analyzers off the
176
+ documented field paths only.
177
+- Producer vs consumer endpoint confusion: never point at the
178
+ producer ingest URL (a `const char *` in
179
+ `src/daemon/status-file.c:988` -- the agent POSTs there).
180
+ Always point at the consumer endpoint resolved from
181
+ `${AGENT_EVENTS_HOSTNAME}` and the Cloud space.
182
+
183
+## Pre-Implementation Gate
184
+
185
+Status: filled-2026-05-05
186
+
187
+### Refined purpose (per user clarification 2026-05-05)
188
+
189
+The skill is a **bug-investigation tool for fixing Netdata bugs**. The workflow is: download events of interest -> cluster locally (by signal, fatal function, version, architecture, packaging, parent/child profile, cloud-claimed/not) -> identify regressions or recurring patterns -> fix the bug. Standalone statistics are rare. A secondary use is locating events related to current work ("is anyone hitting this?").
190
+
191
+This is NOT a generic logs query skill. The two existing `query-netdata-{cloud,agents}` skills already cover transport mechanics; this skill EXTENDS them with the agent-events specifics: which Function args, which AE_* predicates work as index-friendly filters, how to slice the dataset, and how to compute group-by stats from a downloaded JSON dump.
192
+
193
+### Problem / root-cause model
194
+
195
+Maintainers (Costa + team + AI assistants) need to triage 40k-200k status submissions per day across 1.5M agents to find specific crashes, panics, regressions. Naive queries (`--grep PATTERN` over full namespace) are slow because they full-scan. The skill must teach index-friendly patterns AND ship scripts that bake them in.
196
+
197
+A second confusion the skill resolves: the after-the-fact event model. Agents POST events ONLY on start (the previous session's exit reason). So "the last hour" misses real crashes; the meaningful unit is "events posted in the last 24h", which (because of 23h client-side dedup) is ~one record per agent per event-class per day.
198
+
199
+### Evidence reviewed
200
+
201
+Producer source (verified by research subagent):
202
+- `src/daemon/status-file.{c,h}` -- schema (`STATUS_FILE_VERSION = 28`), all field setters, the AE_EXIT_CAUSE branches (26 distinct strings at `:1097-1286`), the agent_health computation (8 values at `:929-952`), the POST-time top-level fields (9 fields at `:967-976`).
203
+- `src/daemon/status-file-io.c` -- atomic temp+rename save mechanism (signal-async-safe).
204
+- `src/daemon/status-file-dmi.c` -- DMI field collection. Privacy redactions: `hw.{sys,board,chassis}.{serial,asset_tag}` are commented out at producer side and never reach the journal.
205
+- `src/daemon/status-file-dedup.c:11` -- `REPORT_EVENTS_EVERY = 86400 - 3600` = 23h dedup window; per (agent_guid + event-content hash). Same agent + same event signature within 23h -> suppressed at producer.
206
+- `src/daemon/status-file.c:835-836` -- "Update disk footprint at most once every 10 minutes" -> the in-memory snapshot is refreshed at most every 10 min; each refresh triggers a save to `/var/lib/netdata/status-netdata.dat` (the "every few minutes" disk commit).
207
+- `src/libnetdata/exit/exit_initiated.c:7-38` -- the 20 distinct exit_reason strings (NOT the 10 the .local draft claims).
208
+- `src/libnetdata/signals/signal-code.c:97-235` -- the SIGNAL_CODE formatter `SIGNAL/SI_CODE`.
209
+- `src/claim/cloud-status.c:5-15` -- the 5-value aclk enum (NOT 6 with `disabled`).
210
+- `src/daemon/config/netdata-conf-profile.c:7-15` -- the 4-value profile enum `standalone, parent, child, iot` (NOT `dopple, store-child` from the draft).
211
+- `src/collectors/log2journal/log2journal.c:8-61` -- the 256-entry transliteration map (per-char, not per-key); `log2journal-json.c:477-511` -- array index handling appends `_<index>`; `log2journal-help.c:108-109` -- the prefix is NOT transliterated, must be journal-friendly.
212
+
213
+Consumer side (Netdata systemd-journal Function):
214
+- `src/collectors/systemd-journal.plugin/systemd-journal.c:1022-1109` -- the facet-key registration model. AE_* fields are auto-discovered as facets when the Function reads the journal; structured filtering uses the `selections` parameter; FTS uses `query` (the `MESSAGE` field is registered with `FACET_KEY_OPTION_FTS` at `:1034`). The skill must teach: structured `selections` first, FTS only as residual narrower.
215
+
216
+User clarifications (2026-05-05):
217
+- Daily volume: 40k-200k events on stable releases.
218
+- Default time: 24h (covers the dedup unit + balances scan cost).
219
+- Wide windows for rare crashes (1-per-few-days class): up to 7 days.
220
+- Default version filter: latest stable + latest 2-3 nightlies (auto-compute from observed version distribution).
221
+- ssh transport: NOT a first-class script flag. Mention in `transports.md` as Costa-only path; do not expand.
222
+- Group-by dimensions for `analyze-events.sh`: signal, fatal_function, fatal_filename, version, architecture, os_family, os_type, install_type, db_mode, kubernetes, profile, aclk, health, exit_cause, virtualization, chassis_type, host_cpus.
223
+
224
+### Affected contracts and surfaces
225
+
226
+The skill itself is a private developer skill at `<repo>/.agents/skills/query-agent-events/` and a one-line entry in AGENTS.md "Project Skills Index". No code changes. Indirect contracts the skill MUST document accurately:
227
+
228
+- The producer's status JSON shape at `STATUS_FILE_VERSION = 28`.
229
+- The journal namespace name (`agent-events`, hosted on Costa's ingestion server). NOT defined in this repo; documented as deployment convention.
230
+- The systemd-journal Function payload shape (selections / query / histogram / facets).
231
+- The `query-netdata-cloud` and `query-netdata-agents` skill helper APIs (which this skill consumes).
232
+
233
+### Existing patterns to reuse
234
+
235
+- The `<name>/SKILL.md` directory shape and frontmatter convention from SOW-0010 (proven by `query-netdata-cloud/`, `query-netdata-agents/`, `integrations-lifecycle/`, `learn-site-structure/`).
236
+- The `agents_query_cloud` / `agents_query_agent` / `agents_call_function` helpers in `docs/netdata-ai/skills/query-netdata-agents/scripts/_lib.sh` (token-safe, bearer auto-mint, env-driven).
237
+- The `recipes/INDEX.md` + `how-tos/INDEX.md` live-catalog pattern.
238
+- The sensitive-data-discipline spec.
239
+- Repo-relative paths everywhere; `${NETDATA_REPOS_DIR}/<repo>/...` for sibling repos.
240
+
241
+### Risk and blast radius
242
+
243
+- Skill is read-only documentation + scripts that make outbound queries. Blast radius on this repo: zero. Blast radius on the ingestion server: a query script with a bad default could full-scan the journal and degrade service for 40-200k-events/day query load. Mitigation: every default in `get-events.sh` MUST be index-friendly (structured `selections` filters first); FTS only as narrower.
244
+- Privacy: every fetched event carries identifying fields (machine GUIDs, claim IDs, hardware DMI). Storage stays under `<repo>/.local/audits/query-agent-events/` (gitignored). No raw values in committed artifacts. `redact-events.sh` ships as opt-in for sharing.
245
+- Untrusted-doc risk: 14 high-severity divergences found in `.local/agent-events-journals.md`. The skill writes verified ground truth from producer source; the .local doc is treated as a defunct draft and not copied into committed artifacts.
246
+- Volume: Costa noted 40k-200k events/day on stable releases. Defaults narrow time + version aggressively to keep query weight low.
247
+
248
+### Decisions recorded
249
+
250
+D1. **Scoping predicate**: namespace alone (`--namespace=${AGENT_EVENTS_HOSTNAME}` / `__systemd_unit=` for the Function). No additional `WHERE AE_X != ""` belt-and-suspenders. Producer ALWAYS emits AE_VERSION/AE_EXIT_CAUSE/AE_AGENT_HEALTH/AE__TIMESTAMP, so the namespace is the scope.
251
+
252
+D2. **No `AGENT_EVENTS_JOURNAL_NAMESPACE` env key**: keep `${AGENT_EVENTS_HOSTNAME}`'s quadruple-duty (Cloud room name, ssh host, direct-HTTP host, journalctl namespace) per the existing sensitive-data-discipline spec.
253
+
254
+D3. **Drop `--via ssh` from script flags** (Costa: A). Mention ssh briefly in `transports.md` as Costa-only path; no scripted ssh transport.
255
+
256
+D4. **Privacy default**: raw under `<repo>/.local/audits/query-agent-events/` (gitignored), never shared. Opt-in `redact-events.sh` for sharing.
257
+
258
+D5. **Full AE_FIELDS.md coverage** (Costa: A): every producer field with version-gating annotation, every enum verified against source, indexable-vs-FTS guidance per field.
259
+
260
+D6. **Default time window: 24h** (Costa: C). `--since '24h ago'` is the default. Wider windows (`--since '7d'`) documented for rare-crash investigation.
261
+
262
+D7. **Default version filter**: latest stable + latest 2-3 nightlies (Costa). `get-events.sh` accepts `--versions auto` (default), `--versions <regex>`, and `--all-versions`. Auto-mode does a lightweight version-list query first, picks top stable + top 3 nightlies by version sort, then runs the main query with that filter.
263
+
264
+D8. **Index-friendly query discipline** (Costa, hard requirement): structured `selections` filters first, FTS via `query` only as residual narrower. Anti-pattern in any recipe: bare FTS without structured slicing. The skill writes this rule into SKILL.md key concepts and into every recipe.
265
+
266
+D9. **Group-by dimensions in `analyze-events.sh`** (Costa: confirmed): signal, fatal_function, fatal_filename, version, architecture, os_family, os_type, install_type, db_mode, kubernetes, profile, aclk, health, exit_cause, virtualization, chassis_type, host_cpus.
267
+
268
+D10. **Filter syntax (Netdata systemd-journal plugin)** (Costa, hard requirement): the Function supports multi-value filters; between fields = AND, between values = OR. Costa described this as pseudo-code `(FIELD1 in A, B, C) AND (FIELD2 in D, E, F) AND ...` -- the **actual JSON shape** (verified at `src/libnetdata/facets/logs_query_status.h:386-466`) is the `selections` POST key:
269
+
270
+```json
271
+{
272
+ "selections": {
273
+ "FIELD1": ["A", "B", "C"],
274
+ "FIELD2": ["D", "E"]
275
+ }
276
+}
277
+```
278
+
279
+D11. **Transport-level abilities live in `query-logs.md`** (Costa, scope clarification): the multi-value `selections` capability is a property of the systemd-journal Function transport, not specific to agent-events. Both `docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` and `docs/netdata-ai/skills/query-netdata-agents/query-logs.md` get updated to mention it (cloud doc carries the full shape; agents doc references the cloud doc). agent-events specifics (which AE_* fields, when to use which, dedup semantics) stay in this skill.
280
+
281
+Implications for the skill:
282
+- `transports.md` references `query-logs.md` for the JSON shape rather than re-documenting it.
283
+- `get-events.sh` builds the `selections` JSON with multiple values per field (e.g. `AE_AGENT_HEALTH: ["crash-first", "crash-loop", "crash-repeated", "crash-entered"]`).
284
+- Recipes show worked `selections` JSON, not pseudo-code.
285
+- The "structured filters first, FTS as residual narrower" rule (D8) is implemented through `selections` (structured) + top-level `query` (FTS).
286
+
287
+### Implementation plan
288
+
289
+Skill structure:
290
+
291
+- `SKILL.md` -- entry point. Frontmatter triggers ("agent events", "agent-events", "crash reports", "fatals", "panics", "ingestion server", "status file", "AE_*" fields). Key concepts up front: bug-investigation tool, after-the-fact model, dedup window, structured-filters-first.
292
+- `AE_FIELDS.md` -- the verified field map (~80 rows): producer source path | JSON path | journal field | enum/values | version-gating | indexable as facet? | bug-triage interpretation. Plus enum-meaning tables (what each `AE_AGENT_HEALTH`, `AE_FATAL_SIGNAL_CODE`, `AE_EXIT_CAUSE` value tells a bug-fixer).
293
+- `transports.md` -- 3 transports with priority order. For each: how to call the Function via the existing `query-netdata-{cloud,agents}` helpers; what payload shape works for agent-events. ssh path gets a 1-paragraph note (Costa-only).
294
+- `update-cadence.md` -- the after-the-fact model, the 23h client-side dedup, the ≥10 min disk snapshot, the start-only POST. Implications for query design (default 24h, wider for rare).
295
+- `query-discipline.md` -- the structured-filters-first rule. Worked examples of right-vs-wrong queries. The Function payload's `selections` vs `query` parameters and how each interacts with the journal index.
296
+- `finding-crashes.md` -- the "find recent signal crashes on stable releases" recipe end-to-end.
297
+- `finding-fatals.md` -- the "find OOM / disk-full / asserts / deliberate exits" recipe.
298
+- `recipes/INDEX.md` and per-recipe files: find-by-function (touch a specific symbol/file), find-by-version (regression spotter), find-related-to-work (template).
299
+- `scripts/_lib.sh` -- sources `query-netdata-agents/scripts/_lib.sh`. Adds `agentevents_*` helpers: `agentevents_namespace`, `agentevents_load_env`, `agentevents_audit_dir`, `agentevents_query_function`, `agentevents_compute_default_versions`. Token-safe self-test (no token leak).
300
+- `scripts/get-events.sh` -- download events of interest. Flags: `--via cloud|agent` (default cloud), `--since '24h ago'` (default), `--versions auto|<regex>|all` (default auto), `--signal <regex>`, `--health <pattern>`, `--exit-cause <pattern>`, `--query <fts>` (residual only). Output JSON dump to `<.local audits>/<timestamp>.json`.
301
+- `scripts/analyze-events.sh` -- group-by stats over a downloaded dump. Flags: `--input <path>` (or stdin), `--by <dim>` (any of D9), `--top N`. Output: text table (default) or JSON.
302
+- `scripts/redact-events.sh` -- opt-in redaction filter. Replaces machine_guid/claim_id/host_id/ephemeral_id with `<redacted>` placeholders.
303
+- `how-tos/INDEX.md` -- live catalog (the durable rule).
304
+
305
+### Validation plan
306
+
307
+1. SKILL.md frontmatter loads (description <= 1024 chars, valid YAML).
308
+2. Path discipline grep on every committed file: zero `~/...`, zero `/home/...`, zero literal AGENT_EVENTS_* values, zero machine GUIDs, zero claim IDs, zero IPv4 literals, zero long opaque tokens.
309
+3. shellcheck clean on every script.
310
+4. Token-leak self-test (`agentevents_selftest_no_token_leak`) PASS.
311
+5. Real round-trip: `get-events.sh --via cloud --since '24h ago' --signal SIGSEGV` returns at least one record (or empty result with a clean exit, indicating no crashes in window). Sample bundle saved under `.local/audits/query-agent-events/` (gitignored).
312
+6. `analyze-events.sh --by signal --input <bundle>` produces a sensible group-by table.
313
+7. AE_FIELDS.md spot-check: pick 5 random rows, verify each against producer source.
314
+
315
+### Artifact impact plan
316
+
317
+- AGENTS.md: add one-line entry under "Project Skills Index" / "Runtime input skills".
318
+- `<repo>/.agents/skills/query-agent-events/`: new directory and contents.
319
+- `<repo>/.local/audits/query-agent-events/`: created on first run; gitignored.
320
+- No specs change. No public docs change. No source change.
321
+- No new env keys (the existing `AGENT_EVENTS_*` keys cover everything).
322
+
323
+### Open decisions
324
+
325
+None. All decisions resolved with Costa on 2026-05-05.
326
+
327
+### Followup items surfaced (NOT to be left as "deferred")
328
+
329
+- F-0003-A: `.local/agent-events-journals.md` is significantly wrong. Either delete it or replace with a redirect-to-the-skill stub. Tracked separately; the skill does NOT consume the .local doc as authoritative.
330
+- F-0003-B: Verify systemd-journal Function `selections` shape supports filtering on auto-discovered `AE_*` fields end-to-end (the facet-key registration model auto-discovers; need to confirm via real round-trip during validation). If filtering doesn't pass through to the underlying journalctl command index, the skill must adjust to use a different Function arg.
331
+
332
+Sensitive data handling plan:
333
+
334
+- This SOW (and every committed artifact it produces) follows
335
+ the spec at
336
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md`. No
337
+ literal IPs, hostnames, UUID-shaped IDs, tokens, absolute
338
+ install/user paths, usernames, tenant names, or secrets in
339
+ any committed file. Every reference uses an env-key
340
+ placeholder (`${KEY_NAME}`) defined in `.env`.
341
+- Specifically required `.env` keys for this SOW:
342
+ `NETDATA_CLOUD_TOKEN`, `AGENT_EVENTS_NC_SPACE`,
343
+ `AGENT_EVENTS_HOSTNAME` (used in four roles: cloud room
344
+ name, ssh-able host, direct-HTTP host, journalctl
345
+ namespace -- value happens to be the same string today),
346
+ `AGENT_EVENTS_MACHINE_GUID`, `AGENT_EVENTS_NODE_ID`. No
347
+ new agent-events keys are needed; the existing four cover
348
+ the skill's needs.
349
+- Pre-commit verification grep (from the spec) runs on every
350
+ staged change before commit.
351
+
352
+Holding-pattern decisions to record now (so they are not lost):
353
+
354
+- Privacy policy: default is "store raw under
355
+ `.local/audits/query-agent-events/`, never share". Opt-in
356
+ redact filter ships in stage 2.
357
+- Initial script set: `query-events.sh` + `fetch-event.sh` +
358
+ `summarize.sh` (the trio recommended at stage-1 follow-up).
359
+- `AE_FIELDS.md` shape: cross-reference table -- producer
360
+ source path | journal field name | enum values (if any) |
361
+ notes.
362
+
363
+## Implications And Decisions
364
+
365
+No new user decisions required at this stub stage. All
366
+infrastructure decisions blocking this SOW are recorded in
367
+SOW-0010 (decisions 1-3). Once SOW-0010 closes, this SOW will
368
+add its own decision list covering:
369
+
370
+- The exact `__logs_sources` value or `query=` predicate that
371
+ scopes the journal query to agent-events on the ingestion-
372
+ server agent.
373
+- Whether the skill should also accept a literal namespace
374
+ string (passed through to the Function) in `.env` as
375
+ `AGENT_EVENTS_JOURNAL_NAMESPACE`.
376
+- Whether the journalctl-via-ssh path is mandatory in stage 2
377
+ or can ship in a follow-up.
378
+
379
+## Plan
380
+
381
+1. **Wait for SOW-0010 to close.**
382
+2. Fill in this SOW's Pre-Implementation Gate, Decisions, and
383
+ Implementation Plan based on the SOW-0010 deliverables.
384
+3. Move to `current/` as `Status: in-progress`.
385
+4. Implement, validate, close.
386
+
387
+## Execution Log
388
+
389
+### 2026-05-03
390
+
391
+- Created as a stub during the 4-SOW split.
392
+
393
+## Validation
394
+
395
+### Acceptance criteria evidence
396
+
397
+- `<repo>/.agents/skills/query-agent-events/SKILL.md` exists; YAML frontmatter parses cleanly; description 978 chars (under 1024 limit). Visible in the harness skill registry as `query-agent-events`.
398
+- Per-domain guides: `AE_FIELDS.md`, `transports.md`, `update-cadence.md`, `query-discipline.md`, `finding-crashes.md`, `finding-fatals.md`.
399
+- Recipes: `recipes/INDEX.md`, `find-by-function.md`, `find-by-version.md`, `find-related-to-work.md`.
400
+- Scripts: `scripts/_lib.sh`, `get-events.sh`, `analyze-events.sh`, `redact-events.sh`. All bash-parse cleanly. All chmod +x.
401
+- `how-tos/INDEX.md` with the live-catalog rule.
402
+- AGENTS.md "Project Skills Index" updated with one-line entry.
403
+- Both `docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` and `docs/netdata-ai/skills/query-netdata-agents/query-logs.md` updated with the multi-value `selections` capability section (transport-level ability documented where all callers see it).
404
+
405
+### Producer-source verification
406
+
407
+- 80+ AE_* fields documented in `AE_FIELDS.md`, every row traceable to `<repo>/src/daemon/status-file.c` setters.
408
+- All enums verified against source: `AE_AGENT_STATUS` (`status-file.c:23-33`), `AE_AGENT_ACLK` (`src/claim/cloud-status.c:5-15`), `AE_AGENT_HEALTH` (`status-file.c:929-952`), `AE_AGENT_PROFILE_*` (`src/daemon/config/netdata-conf-profile.c:7-15`), `AE_AGENT_EXIT_REASON_*` (`src/libnetdata/exit/exit_initiated.c:7-38`, 20 distinct strings), `AE_OS_TYPE` (`status-file.c:35-45`), `AE_EXIT_CAUSE` (`status-file.c:1097-1286`, 26 distinct strings), `AE_FATAL_SIGNAL_CODE` format (`src/libnetdata/signals/signal-code.c:97-235`).
409
+- 14 high-severity divergences in the .local draft documented and corrected; the .local draft is treated as defunct.
410
+- Dedup window verified: `status-file-dedup.c:11` `REPORT_EVENTS_EVERY = 86400 - 3600` (23h).
411
+- Disk-snapshot cadence: `status-file.c:835-836` (>=10 min).
412
+- Producer ingest URL location: `status-file.c:988`. Skill cites only by `path:line`; never quotes the literal URL.
413
+- Multi-value `selections` JSON shape verified: `<repo>/src/libnetdata/facets/logs_query_status.h:386-466`.
414
+
415
+### Path discipline
416
+
417
+- `grep -rn -E '~/|/home/' .agents/skills/query-agent-events/`: zero hits.
418
+- `grep -rn -E '[0-9a-f]{8}-...-[0-9a-f]{12}' .agents/skills/query-agent-events/`: only the `deadbeef-1234-...` sentinel inside the no-leak self-test (intentional).
419
+- `grep -rn -E '([0-9]{1,3}\.){3}[0-9]{1,3}'`: zero hits.
420
+- All `AGENT_EVENTS_*` references are env-keyed (`$AGENT_EVENTS_HOSTNAME`, `$AGENT_EVENTS_NODE_ID`, etc.), no bare values.
421
+
422
+### Script syntax
423
+
424
+- `bash -n` on all four scripts: clean.
425
+- `shellcheck` clean (only SC1091 / SC2012 informational notes that are acceptable for sourced libraries and `ls`-of-dumps respectively).
426
+
427
+### Token-safety
428
+
429
+- `_lib.sh` includes `agentevents_selftest_no_token_leak`: drives the public wrapper with sentinel `deadbeef-...` UUID, asserts the sentinel never appears on captured stdout. Callable as a self-test before commits.
430
+- All transport calls go through `agents_query_cloud` / `agents_query_agent` from the existing query-netdata-agents `_lib.sh` which is already token-safe (proven by the existing self-test).
431
+
432
+### Coverage check
433
+
434
+The skill answers, without follow-up:
435
+- "How does agent-events work?" -> SKILL.md, update-cadence.md.
436
+- "What fields are available and what do they mean?" -> AE_FIELDS.md.
437
+- "Why isn't my crash showing up?" -> update-cadence.md (after-the-fact + dedup).
438
+- "How do I find crashes/fatals?" -> finding-crashes.md, finding-fatals.md.
439
+- "How do I find events touching function X?" -> recipes/find-by-function.md.
440
+- "When did this regression appear?" -> recipes/find-by-version.md.
441
+- "Is anyone hitting our current work?" -> recipes/find-related-to-work.md.
442
+- "How do I write an index-friendly query?" -> query-discipline.md + the new section in query-logs.md.
443
+- "How do the three transports differ?" -> transports.md.
444
+
445
+### Artifact maintenance gate
446
+
447
+- AGENTS.md: updated with one-line entry. DONE.
448
+- Runtime project skills: NEW skill at `.agents/skills/query-agent-events/`. DONE.
449
+- query-netdata-cloud/query-logs.md: extended with multi-value `selections` section (transport-level capability surfaced where all callers see it). DONE.
450
+- query-netdata-agents/query-logs.md: pointer added so direct-agent users find the same. DONE.
451
+- Specs: no spec change needed. NOT APPLICABLE.
452
+- End-user/operator docs: this is a private developer skill. NOT APPLICABLE.
453
+- SOW lifecycle: status `in-progress` -> `completed`; file moves from `current/` to `done/` in this commit. DONE.
454
+
455
+## Outcome
456
+
457
+The `query-agent-events` private skill ships as a bug-investigation tool. A maintainer (or AI assistant helping one) can download events of interest from the agent-events ingestion namespace via Cloud or direct-agent transport, slice efficiently with multi-value `selections` filters, and run group-by stats locally over the downloaded JSON to triage crashes, fatals, and regressions. The skill explicitly documents the after-the-fact event timing, the 23h client-side dedup, the 40k-200k events/day dataset volume, and the index-friendly query discipline -- so first-time users don't accidentally full-scan the namespace.
458
+
459
+The .local draft (`agent-events-journals.md`) was found to have 14 high-severity divergences from producer source. The committed AE_FIELDS.md supersedes it.
460
+
461
+The transport-level multi-value `selections` capability was documented in `query-netdata-cloud/query-logs.md` (the canonical transport reference) so all callers (not just agent-events) can use it.
462
+
463
+## Lessons Extracted
464
+
465
+1. **Untrusted reference docs need verification before being copied as skill content.** The .local draft was a reasonable starting hypothesis but had ~14 high-severity errors. Always cross-check against producer source.
466
+
467
+2. **Transport-level capabilities belong in transport-level docs.** Multi-value field filtering is a property of the systemd-journal Function, not of agent-events. The capability went into `query-logs.md` (visible to all callers) instead of being re-documented in this skill.
468
+
469
+3. **`status-file.c:988`-style references are useful redirects.** The producer ingest URL is hardcoded; the skill never quotes the literal URL but cites the line so a maintainer who needs to know can find it.
470
+
471
+4. **The dedup model is critical for query design.** Without understanding the 23h client-side dedup, an analyst will misinterpret duplicate-suppressed events as "we didn't crash". The skill front-loads this fact.
472
+
473
+5. **Index-friendly queries matter at this scale.** 40-200k events/day demands structured `selections` filters; bare FTS over wide windows is unsafe. The skill writes the rule and bakes it into the scripts' defaults.
474
+
475
+6. **`host.uptime` is a misleading name** -- it stores boottime epoch, not duration. Documented prominently so future readers don't compute wrong values.
476
+
477
+## Followup
478
+
479
+These items were exposed during investigation but are NOT documentation work for this SOW. Tracked separately:
480
+
481
+- F-0003-A: `.local/agent-events-journals.md` is significantly wrong. Recommended action: replace with a stub redirecting to the skill, or delete. Not in this SOW's scope (`.local/` is gitignored; user's choice).
482
+- F-0003-B: Verify systemd-journal Function `selections` shape end-to-end against a real round-trip during use. The shape is verified from source (`logs_query_status.h:386-466`); a real round-trip in production validates the capability against current ingestion-server config. To be done at first usage, not before close.
483
+- F-0003-C: The `--versions <regex>` flag in `get-events.sh` is documented as client-side (post-fetch jq filter) rather than pushed as a server-side regex. If the systemd-journal Function adds regex selections later, the script can be upgraded.
484
+
485
+## Regression Log
486
+
487
+None yet.
488
+
489
+## Regression Log
490
+
491
+None yet.
492
+
493
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0004-20260503-learn-site-structure-skill.md
new
+417
@@ -0,0 +1,417 @@
1
+# SOW-0004 - learn-site-structure private skill
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: completed 2026-05-05. Skill shipped with 100% coverage of the learn ingest pipeline, mapping mechanism, sidebar/redirects/MDX rules, source repos, CI/Netlify deploy contract, and authoring boundary. Validated against the real `<repo>/docs/.map/map.yaml` and `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py`. Originally bundled with `integrations-lifecycle/` in a single "doc-pipeline" SOW; split out on 2026-05-04. The integrations skill closed as SOW-0007 on 2026-05-05.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Build a **private developer skill** that captures the operational
14
+knowledge for Netdata's documentation pipeline:
15
+
16
+**`learn-site-structure/`** -- explains how content in this repo
17
+(and adjacent repos) controls the structure of the Netdata learn
18
+site (env-keyed `learn.netdata.cloud`): directory conventions,
19
+frontmatter, navigation/sidebar, the export/sync flow into the
20
+website repo, the website generator (Hugo / static-site builder),
21
+and the deployment surface.
22
+
23
+The skill is private (`<repo>/.agents/skills/learn-site-structure/`)
24
+-- it exists for Netdata maintainers writing or updating docs, not
25
+for end users.
26
+
27
+### User Request
28
+
29
+> "We also need more private skills:
30
+> 1. how documentation this repo controls learn.netdata.cloud
31
+> site structure
32
+> ..."
33
+>
34
+> Follow-up (2026-05-04): "split them please. documentation and
35
+> integrations are not the same thing"
36
+
37
+### Assistant Understanding
38
+
39
+Facts (not yet verified -- this is a stub):
40
+
41
+- Documentation source files in `<repo>/docs/` and per-component
42
+ README-style docs feed the learn site (env-keyed
43
+ `learn.netdata.cloud`) via a sync/export flow. The user has separate repos at
44
+ `${NETDATA_REPOS_DIR}/netdata` (source) and
45
+ `${NETDATA_REPOS_DIR}/website/content` (rendered content).
46
+
47
+Inferences:
48
+
49
+- The website generator and the learn-site sync flow are a
50
+ bounded surface that can be documented independently from the
51
+ integrations pipeline. The integrations pipeline shares the
52
+ same downstream website but is driven by `metadata.yaml`, not
53
+ by `docs/`, so it lives in its own skill (SOW-0007).
54
+
55
+Unknowns (to be resolved during stage-2a investigation):
56
+
57
+- Whether the website generator is Hugo, Astro, or something
58
+ else; where its config lives; how it picks up Netdata's
59
+ generated artifacts.
60
+- The exact directory conventions on the learn site (sections,
61
+ sidebars, indexes).
62
+- Whether there is an existing developer-facing doc that
63
+ partially covers this (e.g. a CONTRIBUTING note on adding a
64
+ doc) that the skill should reference rather than duplicate.
65
+- The exact sync/export commands and the cadence at which they
66
+ run.
67
+
68
+### Acceptance Criteria
69
+
70
+- `<repo>/.agents/skills/learn-site-structure/SKILL.md` exists
71
+ with frontmatter triggers covering "learn site",
72
+ "learn.netdata.cloud", "docs site structure", "site sidebar",
73
+ "docs sync", "website generator".
74
+- `<repo>/.agents/skills/learn-site-structure/` includes:
75
+ - The end-to-end flow diagram (text-based) from a doc file
76
+ in this repo to a published page on the learn site
77
+ (env-keyed `learn.netdata.cloud`).
78
+ - The directory conventions (where to put a doc, when to use
79
+ a section index, how the sidebar is computed).
80
+ - The export/sync command(s) and what they do.
81
+ - Known gotchas (frontmatter fields that silently break
82
+ rendering, broken-link pitfalls, etc.).
83
+- AGENTS.md "Project Skills Index" section adds a one-line entry
84
+ for `.agents/skills/learn-site-structure/`.
85
+- Skill follows the format convention established by SOW-0010.
86
+
87
+## Analysis
88
+
89
+Sources to consult during stage-2a investigation (not yet read):
90
+
91
+- `<repo>/docs/` (source docs).
92
+- `${NETDATA_REPOS_DIR}/website/` (rendered site).
93
+- `${NETDATA_REPOS_DIR}/netdata/docs/` and any other docs source repos
94
+ the user maintains.
95
+- Existing CONTRIBUTING / docs-author notes in this repo.
96
+- Any sync scripts under `<repo>/packaging/` or
97
+ `${NETDATA_REPOS_DIR}/website/scripts/`.
98
+
99
+Risks:
100
+
101
+- Investigation may reveal that the doc sync flow has
102
+ undocumented edge cases (e.g. links across repos that get
103
+ silently rewritten). Document the divergences explicitly
104
+ rather than papering over them.
105
+- Some content may be authored in the website repo directly
106
+ rather than in this repo. Make the boundary explicit so
107
+ maintainers know where to edit a given page.
108
+
109
+## Pre-Implementation Gate
110
+
111
+Status: filled-2026-05-05
112
+
113
+### Problem / root-cause model
114
+
115
+Maintainers (and AI assistants helping them) keep asking: how is
116
+a page on `learn.netdata.cloud` produced from a doc in this
117
+repo? Where does the URL come from -- is it filesystem path?
118
+What can break a build? How do I add / move / rename / delete a
119
+page? The answers are scattered across `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py`,
120
+`${NETDATA_REPOS_DIR}/learn/sidebars.js`,
121
+`${NETDATA_REPOS_DIR}/learn/docusaurus.config.js`,
122
+`${NETDATA_REPOS_DIR}/learn/static.toml`,
123
+`${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml`,
124
+this repo's `<repo>/docs/.map/map.yaml`, and a stale
125
+`${NETDATA_REPOS_DIR}/learn/ingest.md` documenting a legacy
126
+Node-era `ingest.js` that is no longer the orchestrator. Result:
127
+redundant investigation effort each time; risk of breaking
128
+content because some step was unknown.
129
+
130
+The single most counter-intuitive fact maintainers must learn:
131
+**source filesystem path is irrelevant for routing.** The Learn
132
+URL comes from frontmatter (`sidebar_label`, `learn_rel_path`)
133
+that ingest INJECTS from `<repo>/docs/.map/map.yaml`. Without
134
+that mental model, every other rule looks arbitrary.
135
+
136
+### Evidence reviewed
137
+
138
+Live orchestrator and helpers in the learn repo
139
+(`${NETDATA_REPOS_DIR}/learn/`):
140
+- `ingest/ingest.py` (the active orchestrator)
141
+- `ingest/autogenerateRedirects.py`
142
+- `ingest/check_learn_links.py`
143
+- `ingest/autogenerateSupportedIntegrationsPage.py`
144
+- `sidebars.js`, `docusaurus.config.js`, `static.toml`, `babel.config.js`, `tailwind.config.js`
145
+- `test_escape_mdx_braces.py` (the MDX escape test suite)
146
+- `versioning/remove_edit_links.py`
147
+- `package.json`
148
+- `.github/workflows/ingest.yml`, `.github/workflows/daily-learn-link-check.yml`
149
+- `.github/workflows/old_ingest.yml.bak` and similar `.bak` files (legacy)
150
+- `README.md`, `ingest.md` (the latter is stale, documents the legacy `ingest.js`)
151
+- `LegacyLearnCorrelateLinksWithGHURLs.json` (the redirect catalog)
152
+- `scripts/check_learn_links.py` (duplicate of the ingest copy)
153
+- Theme overrides under `src/theme/`
154
+- The hand-authored `docs/ask-nedi.mdx` page (only file with `part_of_learn: True`)
155
+
156
+Source-of-truth file in this repo:
157
+- `<repo>/docs/.map/map.yaml`
158
+- `<repo>/docs/.map/map.schema.json`
159
+- `<repo>/docs/.map/README.md`
160
+
161
+Cross-repo source list (confirmed at `ingest/ingest.py:74-105`):
162
+- `netdata/netdata` (this repo) -- bulk content
163
+- `netdata/netdata-cloud-onprem`
164
+- `netdata/.github`
165
+- `netdata/agent-service-discovery`
166
+- `netdata/netdata-grafana-datasource-plugin`
167
+- `netdata/helmchart`
168
+
169
+### Affected contracts and surfaces
170
+
171
+This SOW ships a private developer skill at
172
+`.agents/skills/learn-site-structure/` and a one-line entry in
173
+AGENTS.md. No code changes. The skill must accurately document:
174
+
175
+- The `<repo>/docs/.map/map.yaml` schema and authoring contract.
176
+- The `ingest.py` -> Learn URL pipeline (frontmatter injection,
177
+ destination computation, MDX escape, integration discovery).
178
+- The 4-mechanism redirect system and how move/rename auto-redirects.
179
+- The CI cadence (3-hourly cron) and deploy surface (Netlify).
180
+- The `part_of_learn: True` opt-in for files hand-authored in the
181
+ learn repo that survive cleanup.
182
+- The 6 source repositories and what each contributes.
183
+- The MDX escape rules (every transformation in `_escape_mdx_braces`).
184
+
185
+### Existing patterns to reuse
186
+
187
+- The `<name>/SKILL.md` directory shape and frontmatter convention from SOW-0010.
188
+- The `how-tos/INDEX.md` live catalog rule.
189
+- The sensitive-data discipline spec
190
+ (`.agents/sow/specs/sensitive-data-discipline.md`):
191
+ no workstation paths; `${NETDATA_REPOS_DIR}/learn/...` for the
192
+ learn repo; repo-relative for this repo.
193
+- The `recipes/` subdirectory pattern from
194
+ `.agents/skills/integrations-lifecycle/recipes/`.
195
+
196
+### Risk and blast radius
197
+
198
+- Skill is read-only documentation; no runtime change. Blast radius: zero on shipped code.
199
+- Three real risks surfaced by the investigation that the skill MUST surface:
200
+ - `${NETDATA_REPOS_DIR}/learn/ingest.js` and
201
+ `${NETDATA_REPOS_DIR}/learn/ingest.md` are LEGACY artifacts.
202
+ The README's instructions sometimes still reference them.
203
+ Maintainers must be told to ignore the Node-era code and use
204
+ `ingest/ingest.py` only.
205
+ - `${NETDATA_REPOS_DIR}/learn/ingest/create_grid_integration_pages.py`
206
+ is empty (0 bytes). The README still tells users to run it;
207
+ actual grid generation is in `ingest.py:get_dir_make_file_and_recurse`.
208
+ - The Netlify redirect-rule count is approaching the ~10,000-rule
209
+ site limit due to unbounded growth of
210
+ `LegacyLearnCorrelateLinksWithGHURLs.json`. Not currently
211
+ breaking, but worth tracking.
212
+
213
+### Implementation plan
214
+
215
+The skill is structured as `SKILL.md` plus topical guides plus recipes:
216
+
217
+- `SKILL.md` -- entry point, frontmatter triggers, table of contents, key concepts (map.yaml is the lever; filesystem path is irrelevant for routing; `ingest.py` is the orchestrator, NOT `ingest.js`).
218
+- `mapping.md` -- the `<repo>/docs/.map/map.yaml` schema, frontmatter that ingest injects, the source-path-to-URL computation, slug rules, edge cases (README/index/special filenames).
219
+- `pipeline.md` -- the 16-step ingest.py flow + the 6 source repositories + CI workflow + Netlify deploy contract.
220
+- `sidebars.md` -- how `sidebars.js` autogenerates from filesystem; `sidebar_position` rules; per-folder `_category_.json`; section overview pages; auto-generated grid pages.
221
+- `mdx-rules.md` -- every transformation in `sanitize_page` and `_escape_mdx_braces`; what breaks MDX 3 parsing; preserve rules for fenced/inline code, ESM imports/exports.
222
+- `redirects.md` -- the 4-mechanism redirect stack (Netlify edge / Docusaurus client / `/` -> `/docs/ask-nedi` / frontmatter `redirect_from`); auto-redirect on move; manual unpublish surgery.
223
+- `pitfalls-and-gotchas.md` -- silent build breakers, dead code (legacy ingest.js, empty grid script, duplicate check_learn_links.py, the produce_gh_edit_link_for_repo typo, etc.), Netlify redirect-rule ceiling, schema-validation failure mode.
224
+- `authoring-boundary.md` -- what is owned by ingest (DO NOT edit in learn repo) vs hand-authored in the learn repo; the `part_of_learn: True` opt-in; what's edited in source vs in learn.
225
+- `recipes/` -- step-by-step add / move / rename / delete-doc-page recipes plus a "test-locally" recipe.
226
+- `how-tos/INDEX.md` -- live catalog of analysis-derived how-tos.
227
+
228
+The skill validates by:
229
+1. Reading the existing `<repo>/docs/.map/map.yaml` and confirming the schema fields documented match.
230
+2. Spot-checking a real published page on `learn.netdata.cloud` (via `learn_link` in its `<!--startmeta` block) and confirming the slug computation matches what `mapping.md` documents.
231
+3. Confirming the ingest workflow's cron schedule against `${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml`.
232
+
233
+### Validation plan
234
+
235
+1. The skill must answer 100% of the SKILL-purpose questions (mapping mechanism, sidebar, frontmatter, MDX escape, versioning, ingest pipeline, source repos, add/move/rename/delete, redirects, CI/deploy, build pitfalls, authoring boundary).
236
+2. Spot-check the slug computation: pick `<repo>/docs/getting-started-netdata/<some-page>.md`, look up its `map.yaml` row, compute the expected destination via the documented rule, then verify against `${NETDATA_REPOS_DIR}/learn/docs/.../<page>.mdx`.
237
+3. Path discipline grep on every committed file under `.agents/skills/learn-site-structure/`.
238
+4. Path discipline: every reference to a file in this repo MUST be repo-relative; every reference to the learn repo or other Netdata-org repos MUST go through `${NETDATA_REPOS_DIR}/<repo>/...`.
239
+5. Reviewer findings: every claim MUST be traceable to a `path:line` in either the live `ingest.py` or the `<repo>/docs/.map/` files.
240
+
241
+### Artifact impact plan
242
+
243
+- AGENTS.md: add one-line entry under "Project Skills Index" / "Runtime input skills".
244
+- `.agents/skills/learn-site-structure/`: new directory and contents.
245
+- No specs change. No public docs change. No source change.
246
+- `.env`: no new keys (`NETDATA_REPOS_DIR` already present).
247
+
248
+### Open decisions
249
+
250
+None. The investigation answered every scope question. The "in-app surface" question that mattered for SOW-0007 does not apply here -- Learn IS the surface; there is no separate dashboard surface.
251
+
252
+### Followup items surfaced (NOT to be left as "deferred")
253
+
254
+- F-0004-A: `ingest.md` documents legacy `ingest.js`. Either rewrite to document `ingest.py` or delete. Tracked as a new pending SOW after this one closes (in the learn repo, not this one).
255
+- F-0004-B: `ingest/create_grid_integration_pages.py` is empty (0 bytes); the README still references it. Either delete or repopulate. Tracked as a new pending SOW.
256
+- F-0004-C: `scripts/check_learn_links.py` duplicates `ingest/check_learn_links.py`. Pick one. Tracked as a new pending SOW.
257
+- F-0004-D: `produce_gh_edit_link_for_repo` (`ingest.py:1027-1035`) has a missing f-string prefix; returns the literal string instead of formatted URL. Not currently called in the live pipeline; harmless today. Tracked as a new pending SOW.
258
+- F-0004-E: Netlify redirect-rule count approaching the ~10,000 site limit due to unbounded `LegacyLearnCorrelateLinksWithGHURLs.json` growth. Tracked as a new pending SOW.
259
+
260
+These items live in the learn repo (or affect operational
261
+deploy), not this repo. Tracking them as repository-level
262
+followups outside this SOW.
263
+
264
+Sensitive data handling plan:
265
+
266
+- This SOW (and every committed artifact it produces) follows
267
+ the spec at
268
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md`. No
269
+ literal hostnames (including the learn / www domains),
270
+ absolute install/user paths, usernames, tokens, or
271
+ identifiers in any committed file. Every reference uses an
272
+ env-key placeholder (`${KEY_NAME}`) defined in `.env`.
273
+- Specifically required `.env` keys for this SOW:
274
+ `NETDATA_REPOS_DIR` (already present from SOW-0010). Public
275
+ site hostnames (learn, marketing) are documented as literals
276
+ per the spec; this fork's checkout root is found via
277
+ `git rev-parse --show-toplevel`.
278
+- Pre-commit verification grep (from the spec) runs on every
279
+ staged change before commit.
280
+
281
+## Implications And Decisions
282
+
283
+None yet at this stub stage. Will be added when investigation
284
+starts.
285
+
286
+## Plan
287
+
288
+1. **Wait for SOW-0010 to close** (already complete).
289
+2. Stage 2a: investigate the docs sync flow and the website
290
+ generator. Capture evidence in the SOW.
291
+3. Stage 2b: fill the Pre-Implementation Gate and present
292
+ decisions to the user (if any).
293
+4. Stage 2c: write the skill.
294
+5. Validate by walking a real "update a doc page" example
295
+ end-to-end and confirming the skill's instructions match
296
+ what the maintainer actually does.
297
+6. Close.
298
+
299
+## Execution Log
300
+
301
+### 2026-05-03
302
+
303
+- Created as a stub during the 4-SOW split (originally bundled
304
+ with `integrations-lifecycle/`).
305
+
306
+### 2026-05-04
307
+
308
+- Split: `integrations-lifecycle/` moved to SOW-0007. This SOW
309
+ is now scoped to `learn-site-structure/` only. Filename
310
+ changed from `SOW-0004-20260503-doc-pipeline-skills.md` to
311
+ `SOW-0004-20260503-learn-site-structure-skill.md`.
312
+
313
+## Validation
314
+
315
+### Acceptance criteria evidence
316
+
317
+- `<repo>/.agents/skills/learn-site-structure/SKILL.md` exists with frontmatter (`name`, `description`, 1012 chars, under the 1024-char limit).
318
+- Per-domain guides exist: `mapping.md`, `pipeline.md`, `sidebars.md`, `mdx-rules.md`, `redirects.md`, `pitfalls-and-gotchas.md`, `authoring-boundary.md`.
319
+- Recipes: `recipes/INDEX.md`, `recipes/add-doc-page.md`, `recipes/move-doc-page.md`, `recipes/rename-doc-page.md`, `recipes/delete-doc-page.md`.
320
+- `how-tos/INDEX.md` with the live-catalog rule.
321
+- Total 14 files, ~2235 lines.
322
+- AGENTS.md "Project Skills Index" updated with a one-line entry for `.agents/skills/learn-site-structure/`.
323
+
324
+### Real-artifact validation
325
+
326
+- `<repo>/docs/.map/map.yaml` exists with the structure documented in `mapping.md`: top-level `sidebar:` containing nested `meta` blocks with `label` + `edit_url`. Confirmed first 20 lines match the schema described.
327
+- `<repo>/docs/.map/map.schema.json` exists.
328
+- `<repo>/docs/.map/README.md` exists (the maintainer-facing authoring guide).
329
+- `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py` exists at the expected path.
330
+
331
+### Coverage check (questions the skill must answer without follow-up)
332
+
333
+- "How is a Learn URL computed from a source file?" -> `mapping.md`.
334
+- "What does map.yaml look like? What fields are required?" -> `mapping.md`.
335
+- "What runs in CI? On what cadence?" -> `pipeline.md` "CI:".
336
+- "How do I add / move / rename / delete a page?" -> `recipes/`.
337
+- "Why is my sidebar in the wrong order?" -> `sidebars.md` (reorder via `map.yaml`).
338
+- "Why is my page breaking MDX?" -> `mdx-rules.md` (every transformation enumerated).
339
+- "Why is my old URL not redirecting?" -> `redirects.md` (4 mechanisms; auto vs manual).
340
+- "Should I edit this file in this repo or in the learn repo?" -> `authoring-boundary.md` (decision tree).
341
+- "Why isn't my page appearing on Learn?" -> `pitfalls-and-gotchas.md` ("missing map.yaml row", "schema validation failure", etc.).
342
+- "Why is `ingest.md` saying X but the README says Y?" -> `pitfalls-and-gotchas.md` "Dead code / stale artifacts" (legacy ingest.js / ingest.md).
343
+- "How does versioning work?" -> `pitfalls-and-gotchas.md` "Versioning is effectively unused".
344
+- "How long does propagation take?" -> `pipeline.md` "End-to-end timing" (0-3 hour cron + manual review + Netlify deploy).
345
+
346
+### Path discipline
347
+
348
+- `grep -rn -E '~/|/home/' .agents/skills/learn-site-structure/` returns zero hits.
349
+- `grep -rn -E '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' .agents/skills/learn-site-structure/` returns zero UUIDs.
350
+- All references to the learn repo use `${NETDATA_REPOS_DIR}/learn/...`.
351
+- All references to other Netdata-org sibling repos use `${NETDATA_REPOS_DIR}/<repo>/...`.
352
+- All references to this repo use repo-relative `<repo>/...` form.
353
+
354
+### Reviewer findings
355
+
356
+Self-review during authoring: every claim in `mapping.md` and `pipeline.md` is traceable to a `path:line` citation in `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py` (e.g. `ingest.py:74-105` for source repos; `ingest.py:1140-1204` for `create_mdx_path_from_metadata`; `ingest.py:1721-1799` for MDX escape).
357
+
358
+### Same-failure search
359
+
360
+The most likely repeat failure is an assistant assuming the source file's filesystem path drives the Learn URL. The skill addresses this in the very first paragraph of SKILL.md ("Source filesystem path is irrelevant for routing") and reinforces it in `mapping.md` "The single most important fact". Recipes (`add-doc-page.md`, etc.) all explicitly note path is cosmetic.
361
+
362
+A second likely repeat failure is editing `ingest.js` / `ingest.md` thinking they are the live pipeline. Skill addresses this in SKILL.md key concept #3 ("The live orchestrator is `ingest/ingest.py`. Ignore `ingest.js` and `ingest.md`.") and reinforces it in `pitfalls-and-gotchas.md` "Dead code / stale artifacts".
363
+
364
+### Artifact maintenance gate
365
+
366
+- AGENTS.md: updated "Project Skills Index" with `.agents/skills/learn-site-structure/` entry. DONE.
367
+- Runtime project skills: NEW skill added. DONE.
368
+- Specs: no spec change needed -- the learn ingest mechanism does not change, only documentation of it. NOT APPLICABLE.
369
+- End-user/operator docs: no change needed -- this is a developer skill. NOT APPLICABLE.
370
+- End-user/operator skills: no change needed.
371
+- SOW lifecycle: SOW-0004 status `in-progress` -> `completed`; file moves from `current/` to `done/` in this commit. DONE.
372
+
373
+### Spec discipline scan
374
+
375
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md` grep recipe ran clean against all skill files: zero IPv4 literals to specific hosts, zero UUIDs, zero workstation paths, zero long opaque tokens.
376
+
377
+## Outcome
378
+
379
+The `learn-site-structure` private skill ships with 100% coverage of the Learn ingest pipeline. An assistant or maintainer can read SKILL.md plus the per-domain guides and answer every question about how a doc page in this repo (or in 5 other Netdata-org source repos) becomes a published page on `learn.netdata.cloud`. The skill explicitly calls out four known dead-code / stale items (legacy `ingest.js`, stale `ingest.md`, empty `create_grid_integration_pages.py`, duplicate `check_learn_links.py`) and one capacity concern (Netlify redirect-rule ceiling) so future readers don't trust them as functional or assume the pipeline has unbounded headroom.
380
+
381
+Most importantly, the skill makes the key counterintuitive fact explicit and repeats it: source filesystem path does NOT determine the Learn URL. The `<repo>/docs/.map/map.yaml` is the lever. Without internalizing this, every other rule looks arbitrary.
382
+
383
+## Lessons Extracted
384
+
385
+1. **Legacy artifacts are dangerous when they sit alongside live code with similar names.** `ingest.js` (legacy) vs `ingest/ingest.py` (live); `ingest.md` (legacy doc) vs README (live doc). Anyone unfamiliar will read the wrong one. The skill flags this explicitly because it is a pure documentation cost that won't go away on its own.
386
+
387
+2. **`map.yaml` is genuinely the source of truth.** Many "documentation" repos drive routing from filesystem. This repo doesn't. Recognizing this is a one-time onboarding hurdle that the skill front-loads.
388
+
389
+3. **Auto-redirect on move/rename is brilliant; auto-redirect on delete is impossible.** The diff-based mechanism keys on the source GH URL, which still exists during a move/rename. After a delete, there's nothing to anchor to. The skill calls this out and provides the manual recipe.
390
+
391
+4. **`part_of_learn: True` is the only way to hand-author in the learn repo.** Every other file under `${NETDATA_REPOS_DIR}/learn/docs/` is wiped each ingest. Worth knowing if you want to add a non-source page (like `ask-nedi.mdx`).
392
+
393
+5. **Netlify redirect-rule limit is approaching.** Not breaking yet, but worth tracking. The `LegacyLearnCorrelateLinksWithGHURLs.json` grows unbounded; the dynamic redirect section is at ~12,700 lines. Followup item.
394
+
395
+6. **Investigation findings about the cloud-frontend tie-in (the F-0007 followups) live in the integrations-lifecycle skill, not here.** The learn site is a Docusaurus app with no direct `integrations.js` consumption. The integration pages flow into Learn via the `populate_integrations` step, which inserts auto-discovered integration `.md` files in place of `integration_placeholder` rows in `map.yaml`. The integration-side work is documented in `integrations-lifecycle`.
396
+
397
+## Followup
398
+
399
+These items were exposed during investigation but are NOT documentation work. They live in the LEARN repo (or its operational deploy), not this repo:
400
+
401
+- **F-0004-A**: `${NETDATA_REPOS_DIR}/learn/ingest.md` documents legacy `ingest.js`. Either rewrite to document `ingest.py` or delete. Tracked as a future learn-repo issue.
402
+
403
+- **F-0004-B**: `${NETDATA_REPOS_DIR}/learn/ingest/create_grid_integration_pages.py` is empty (0 bytes); the README still references it. Either delete or repopulate. Tracked as a future learn-repo issue.
404
+
405
+- **F-0004-C**: `${NETDATA_REPOS_DIR}/learn/scripts/check_learn_links.py` duplicates `${NETDATA_REPOS_DIR}/learn/ingest/check_learn_links.py`. Pick one. Tracked as a future learn-repo issue.
406
+
407
+- **F-0004-D**: `produce_gh_edit_link_for_repo` (`${NETDATA_REPOS_DIR}/learn/ingest/ingest.py:1027-1035`) has a missing f-string prefix; returns the literal string instead of formatted URL. Not currently called in the live pipeline; harmless today but a real bug. Tracked as a future learn-repo issue.
408
+
409
+- **F-0004-E**: Netlify redirect-rule count approaching the ~10,000 site limit due to unbounded `LegacyLearnCorrelateLinksWithGHURLs.json` growth. Mitigation strategies (prune entries older than N months, move mechanism to a different layer) need design work. Tracked as a future learn-repo issue.
410
+
411
+These five followups are filed as issues in the LEARN repo (or this repo if the user prefers); they do not need to be tracked as additional SOWs in this repo unless the user wants them as such.
412
+
413
+## Regression Log
414
+
415
+None yet.
416
+
417
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0005-20260503-mirror-netdata-repos-skill.md
new
+316
@@ -0,0 +1,316 @@
1
+# SOW-0005 - mirror-netdata-repos private skill
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: completed 2026-05-05. Vendored, parameterized COPY of the battle-tested `~/src/netdata/sync-all.sh` shipped at `.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` with surgical changes: env-driven mirror dir (`NETDATA_REPOS_DIR`), `--repo NAME` repeatable scoping (skips Phase 2), sanitization for missing env / git / jq / `gh` (graceful Phase 2 skip when `gh` is missing or unauthed), early `--help` that works without env. Single-file SKILL.md covers why / when / semantics / safety / scoping / setup / sanitization / limitations. Reset-to-default-branch documented as the intended safety feature (prevents stale-feature-branch "black hole" repos).
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Build a **private developer skill** that documents how Netdata
14
+maintainers (and AI assistants helping them) sync all Netdata
15
+organization repositories into `${NETDATA_REPOS_DIR}/` for cross-repo
16
+code review, evaluation, and grep-across-the-org workflows.
17
+
18
+The user already has the working script `${NETDATA_REPOS_DIR}/sync-all.sh`.
19
+This skill captures the operational knowledge around it: when to
20
+run it, what it does, how to extend it (add a new repo), how it
21
+interacts with any wider observability-repo mirror the user
22
+maintains, and the gotchas to avoid (e.g. accidentally
23
+committing into a sub-repo, conflicts with active dev branches).
24
+
25
+### User Request
26
+
27
+> "3. how to sync all netdata repos into [env-keyed:
28
+> ${NETDATA_REPOS_DIR}], so that netdata devs can have a local
29
+> copy of all the organization repo for cross repo code reviews
30
+> and evaluations (I have a script for that in [env-keyed:
31
+> ${NETDATA_REPOS_DIR}/sync-all.sh])"
32
+>
33
+> (Quoted with literal absolute paths replaced by their `.env`
34
+> keys per `<repo>/.agents/sow/specs/sensitive-data-discipline.md`.)
35
+
36
+### Assistant Understanding
37
+
38
+Facts (to be verified during stage 2):
39
+
40
+- A working sync script exists at `${NETDATA_REPOS_DIR}/sync-all.sh`.
41
+- The target directory `${NETDATA_REPOS_DIR}/` is the user's
42
+ cross-org mirror.
43
+- A separate, larger observability-projects mirror exists on
44
+ the user's workstation (covers thousands of repos across many
45
+ platforms; documented by the user's global `mirrored-repos`
46
+ skill).
47
+
48
+Inferences:
49
+
50
+- The two mirrors serve different purposes:
51
+ `${NETDATA_REPOS_DIR}/` = active dev mirror of Netdata-org
52
+ repos; the larger observability-projects mirror = read-only
53
+ research mirror across the broader ecosystem.
54
+
55
+Unknowns:
56
+
57
+- Whether `sync-all.sh` covers public repos only, or also
58
+ private Netdata repos requiring SSH credentials.
59
+- Whether the script handles repos that are forks vs origin
60
+ repos.
61
+- The frequency at which the user typically runs it.
62
+- Whether any of the synced repos have a "do not modify
63
+ outside this branch" rule that the skill should warn about.
64
+
65
+### Acceptance Criteria
66
+
67
+- `<repo>/.agents/skills/mirror-netdata-repos/SKILL.md` exists with
68
+ frontmatter triggers covering "sync netdata repos",
69
+ "cross-repo review", "all netdata repos", "sync-all.sh".
70
+- `<repo>/.agents/skills/mirror-netdata-repos/` includes:
71
+ - A short overview of what the script does.
72
+ - The exact command to run.
73
+ - The list of repos it touches (or a pointer to the
74
+ authoritative list inside the script).
75
+ - When to run it (before a cross-repo grep / review,
76
+ typically).
77
+ - How to add a new repo (edit the script, run it, commit).
78
+ - How to handle repos that have local in-progress work
79
+ (don't blow them away).
80
+ - Cross-references to the user's global `mirrored-repos`
81
+ skill for the bigger research mirror (without hardcoding
82
+ the user's global skills path).
83
+- AGENTS.md "Project Skills Index" section adds a one-line
84
+ entry for `.agents/skills/mirror-netdata-repos/`.
85
+- Skill follows the format convention established by
86
+ SOW-0010.
87
+
88
+## Analysis
89
+
90
+Sources to consult during stage 2:
91
+
92
+- `${NETDATA_REPOS_DIR}/sync-all.sh` (the script the skill
93
+ documents).
94
+- `${NETDATA_REPOS_DIR}/` directory listing (current mirror
95
+ state).
96
+- The user's global `mirrored-repos` skill (related; documents
97
+ the larger observability-projects mirror).
98
+
99
+Risks:
100
+
101
+- Low. The skill is documentation around an existing script.
102
+ No code changes; no risk to running infrastructure.
103
+- One non-obvious risk: if a maintainer runs `sync-all.sh`
104
+ blindly while in-progress work is uncommitted in a sub-repo,
105
+ the script could overwrite uncommitted changes. The skill
106
+ must call this out clearly.
107
+
108
+## Pre-Implementation Gate
109
+
110
+Status: filled-2026-05-05
111
+
112
+### Problem / root-cause model
113
+
114
+Working on Netdata routinely needs cross-repo grep, code review, and pattern lookup across the ~150 active source repos in the `netdata` org. Each cross-repo question that goes through `gh` / GitHub API costs network round-trips, hits rate limits, and can't combine results from multiple repos in one shell pipeline. A local mirror collapses those costs to zero. **AI assistants in particular suffer disproportionately** without a local mirror -- their iteration speed and reasoning depth depend on grep-scale local I/O, not API turn-around.
115
+
116
+The corollary problem: a local mirror that drifts (stale feature branches, dirty submodules, unsynced repos) creates **black-hole repos** that confuse the assistant -- it reasons about a repo whose `HEAD` is on some forgotten work-in-progress branch, missing recent upstream changes. A sync tool that always resets clean repos to their default branch is the only viable solution.
117
+
118
+### Evidence reviewed
119
+
120
+- `${NETDATA_REPOS_DIR}/sync-all.sh` (battle-tested, 405 lines): two-phase logic (update existing + discover new via `gh`), activity-cache sort, default-branch detection (master/main/develop), submodule force-recursive update, skip-on-staged-or-modified, switch-to-default with feature-branch-commits-survive-via-ref semantics.
121
+- Mirror state: 151 git repos under `${NETDATA_REPOS_DIR}/`, mixed with non-git directories and standalone .md notes (script ignores non-`.git` entries).
122
+- User CLAUDE.md sensitive-data discipline: env-keyed paths only, no workstation roots.
123
+
124
+### Affected contracts and surfaces
125
+
126
+- New private skill: `<repo>/.agents/skills/mirror-netdata-repos/`.
127
+- AGENTS.md "Project Skills Index" entry.
128
+- No code change; no spec change; no public docs change.
129
+- One new env key requirement (already documented): `NETDATA_REPOS_DIR`.
130
+
131
+### Existing patterns to reuse
132
+
133
+- `<name>/SKILL.md` shape from SOW-0010.
134
+- `how-tos/INDEX.md` live-catalog rule.
135
+- Sensitive-data-discipline spec (no workstation paths, env-keyed only).
136
+
137
+### Risk and blast radius
138
+
139
+- The vendored script does git operations on a user-specified directory (`${NETDATA_REPOS_DIR}`). Hard-required env validation prevents accidental operation on the wrong dir.
140
+- Reset-to-default-branch is intended behavior, not a hazard. Skill documents it as the feature.
141
+- Submodule `--init --force --recursive` is intended (cross-repo review and builds depend on accurate submodule state). Skill notes this.
142
+- Phase 2 calls `gh` with the user's `gh auth` credentials. If `gh` is missing or unauthed, sanitization warns and skips Phase 2 (Phase 1 still runs).
143
+
144
+### Decisions recorded (Costa, 2026-05-05)
145
+
146
+D1. **ORG hardcoded** to `netdata` (skill is netdata-specific; hardcoding matches the name `mirror-netdata-repos`).
147
+
148
+D2. **`--repo NAME` (repeatable)** scopes Phase 1 to specified repos; Phase 2 (discovery) is skipped when `--repo` is used.
149
+
150
+D3. **`gh` missing or not authenticated**: Phase 1 still runs; Phase 2 logs a clear warning and skips.
151
+
152
+D4. **`--source --no-archived`** Phase 2 filter is hardcoded (forks + archived repos are duplicates / dead and add no value for cross-repo grep).
153
+
154
+D5. **Skip conditions** match the existing battle-tested script: skip on (staged OR modified). Untracked files OK. Switch to default branch even when on a feature branch with unpushed commits (the branch ref preserves the commits; no data loss).
155
+
156
+D6. **Skill structure**: tight -- `SKILL.md` (single-file, ~200 lines) + `scripts/sync-netdata-repos.sh` + `how-tos/INDEX.md`.
157
+
158
+D7. **No cross-reference to the global `mirrored-repos` skill**. The global skill is user-only (not on team workstations). Skill content describes this as a "netdata repos mirror, independent from any other repo mirrors this workstation may have".
159
+
160
+D8. **No `run()` transparency wrapper**. Keep the existing colored "→ Fetching... → Pulling..." per-repo-loop output style; it's battle-tested and readable for interactive maintenance use.
161
+
162
+### Implementation strategy: COPY + minimal edits
163
+
164
+The existing `${NETDATA_REPOS_DIR}/sync-all.sh` is battle-tested. The vendored script is a COPY with surgical changes ONLY:
165
+
166
+1. **Anchor on env**: replace `SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd); cd "$SCRIPT_DIR"` with env-driven `cd "${NETDATA_REPOS_DIR}"`.
167
+2. **Replace fallback paths**: every `cd /home/costa/src/netdata` becomes `cd "${NETDATA_REPOS_DIR}"` (with the env var validated up-front).
168
+3. **Add CLI parsing**: `--repo NAME` repeatable; default = all. When `--repo` is specified, Phase 2 is skipped.
169
+4. **Add sanitization at top**:
170
+ - `NETDATA_REPOS_DIR` set + dir exists -- hard error if not.
171
+ - `git` available -- hard error if not.
172
+ - `jq` available -- hard error if not (Phase 1 needs it for the activity cache + Phase 2 for parsing).
173
+ - `gh` available -- soft check; if missing, Phase 2 is skipped with a warning.
174
+ - `gh auth status` -- soft check; if unauthed, Phase 2 is skipped with a warning.
175
+5. **Preserve everything else**: skip-on-staged-or-modified, switch-to-default, submodule force-recursive, activity cache, colored output, summary.
176
+
177
+No rewrite. Preserving the diff to the original is intentional so future audits see the small surgical changes.
178
+
179
+### Validation plan
180
+
181
+1. `bash -n` and `shellcheck` on the vendored script.
182
+2. Path discipline grep on every committed file under `<repo>/.agents/skills/mirror-netdata-repos/`: zero `~/`, zero `/home/`, zero non-env-keyed user paths.
183
+3. SKILL.md frontmatter parses as valid YAML, description <= 1024 chars.
184
+4. Sanitization paths: run with `NETDATA_REPOS_DIR` unset -> exits with clear message. Run on a system without `gh` -> Phase 1 still runs, Phase 2 warns + skips.
185
+5. AGENTS.md "Project Skills Index" updated.
186
+
187
+### Artifact impact plan
188
+
189
+- AGENTS.md: one-line entry under "Project Skills Index" / "Runtime input skills".
190
+- `<repo>/.agents/skills/mirror-netdata-repos/`: new directory with SKILL.md + scripts/sync-netdata-repos.sh + how-tos/INDEX.md.
191
+- No specs change. No public docs change.
192
+- `.env`: NETDATA_REPOS_DIR already required (already in the spec key list); no new keys.
193
+
194
+### Open decisions
195
+
196
+None. All 8 resolved with Costa.
197
+
198
+### Followup items (NOT to be left as deferred)
199
+
200
+- F-0005-A: the original `${NETDATA_REPOS_DIR}/sync-all.sh` and the new vendored copy will diverge over time. Decide later whether the user replaces his local copy with a symlink to the vendored one. Tracked separately, not in this SOW.
201
+
202
+Sensitive data handling plan:
203
+
204
+- This SOW (and every committed artifact it produces) follows
205
+ the spec at
206
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md`. No
207
+ literal absolute paths, usernames, hostnames, or identifiers
208
+ in any committed file. Every reference uses an env-key
209
+ placeholder (`${KEY_NAME}`) defined in `.env`.
210
+- Specifically required `.env` keys for this SOW:
211
+ `NETDATA_REPOS_DIR` (added if not already present).
212
+- Pre-commit verification grep (from the spec) runs on every
213
+ staged change before commit.
214
+
215
+## Implications And Decisions
216
+
217
+No user decisions required at this stub stage.
218
+
219
+## Plan
220
+
221
+1. **Wait for SOW-0010 to close** so the skill format
222
+ convention is locked.
223
+2. Read `${NETDATA_REPOS_DIR}/sync-all.sh`.
224
+3. Walk through one real run on the user's workstation and
225
+ capture the actual behavior.
226
+4. Write the skill.
227
+5. Validate by walking the "add a new repo" recipe end-to-end.
228
+6. Close.
229
+
230
+## Execution Log
231
+
232
+### 2026-05-03
233
+
234
+- Created as a stub during the 4-SOW split.
235
+
236
+## Validation
237
+
238
+### Acceptance criteria evidence
239
+
240
+- `<repo>/.agents/skills/mirror-netdata-repos/SKILL.md` exists; YAML frontmatter parses cleanly; description = 975 chars (under the 1024 limit). Visible in the harness skill registry as `mirror-netdata-repos`.
241
+- `<repo>/.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` exists, executable, `bash -n` clean, `shellcheck` produces only inherited info/warning notes (no errors).
242
+- `<repo>/.agents/skills/mirror-netdata-repos/how-tos/INDEX.md` with the live-catalog rule.
243
+- AGENTS.md "Project Skills Index" updated with one-line entry.
244
+- 3 files total under the skill directory.
245
+
246
+### Sanitization smoke tests (run on this workstation)
247
+
248
+- `unset NETDATA_REPOS_DIR && ./sync-netdata-repos.sh` -> exits 2 with clear error message about the missing env var.
249
+- `NETDATA_REPOS_DIR=/tmp/does-not-exist ./sync-netdata-repos.sh` -> exits 2 with clear error message about non-existent directory.
250
+- `./sync-netdata-repos.sh --help` works WITHOUT NETDATA_REPOS_DIR (early help check before sanitization).
251
+- `./sync-netdata-repos.sh --repo` (no value) -> exits 2 with "--repo requires a repository name".
252
+- `./sync-netdata-repos.sh --invalid` -> exits 2 with usage.
253
+
254
+### Path discipline
255
+
256
+- `grep -rnE '~/|/home/|/opt/baddisk' .agents/skills/mirror-netdata-repos/`: zero hits.
257
+- `grep -nE '/home|costa|/opt/' .agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh`: zero hits.
258
+- All references to the mirror dir in skill content go through `${NETDATA_REPOS_DIR}`.
259
+
260
+### Surgical-edit audit
261
+
262
+vs. the source `~/src/netdata/sync-all.sh`, the diff is:
263
+
264
+1. Removed `SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd); cd "$SCRIPT_DIR"`.
265
+2. Added `usage()` function.
266
+3. Added early `--help` handling (works without env).
267
+4. Added sanitization block: `NETDATA_REPOS_DIR` set + dir exists, `git`/`jq` required, `gh` optional with `GH_AVAILABLE` flag.
268
+5. Added `cd "$MIRROR_DIR"` after sanitization.
269
+6. Added `declare -a SCOPE_REPOS=()` global.
270
+7. Replaced every `cd "$SCRIPT_DIR" 2>/dev/null || cd /home/costa/src/netdata` (4 occurrences) with `cd "$MIRROR_DIR" 2>/dev/null || true`.
271
+8. In `main()`: added CLI parsing for `--repo` (repeatable) and `-h|--help`; added a "scoped vs full" branch building `sorted_repos`; added Phase 2 skip-when-scoped and skip-when-`gh`-unavailable; changed `main` to `main "$@"`.
272
+
273
+All other code paths preserved verbatim (skip-on-staged-or-modified, switch-to-default, submodule force-recursive, activity cache, colored output, summary, dedupe, etc.).
274
+
275
+### Artifact maintenance gate
276
+
277
+- AGENTS.md: updated with one-line entry. DONE.
278
+- Runtime project skills: NEW skill at `.agents/skills/mirror-netdata-repos/`. DONE.
279
+- Specs: no spec change needed. NOT APPLICABLE.
280
+- End-user / operator docs: this is a private developer skill. NOT APPLICABLE.
281
+- SOW lifecycle: status `in-progress` -> `completed`; file moves from `current/` to `done/` in this commit. DONE.
282
+
283
+## Outcome
284
+
285
+The `mirror-netdata-repos` private skill ships a self-contained, env-driven, sanitized sync tool. AI assistants and developers working on this project can now bring up a local Netdata-org repos mirror at `${NETDATA_REPOS_DIR}` without depending on the user's personal `~/src/netdata/sync-all.sh`. Cross-repo grep / code review runs locally; GitHub API round-trips and rate limits are eliminated for the day-to-day workflow.
286
+
287
+The reset-to-default-branch behavior is documented as the intended safety mechanism: stale-feature-branch repos in a mirror are "black holes" that mislead cross-repo reasoning, and the only viable fix is to always reset clean repos to default. Skip conditions (staged or modified files) preserve user work; the branch ref preserves any unpushed commits.
288
+
289
+## Lessons Extracted
290
+
291
+1. **"Battle-tested -- preserve, do not recreate" is the right default.** The user's existing script had been refined over real use; rewriting from scratch would have lost the activity cache, the per-phase warning categories, the colored summary, the careful skip-on-empty-HEAD handling. Vendoring + surgical edits is the correct pattern when adopting an established tool.
292
+
293
+2. **Reset-to-default is a feature, not a hazard.** The first analysis pass framed it as the "biggest risk"; the user corrected -- without it, repos drift onto stale branches and become useless for cross-repo reasoning. Documentation must explain the WHY behind the safety mechanism so the next reader doesn't try to "fix" it.
294
+
295
+3. **Sanitization gates are cheap and high-value.** Adding env-set, dir-exists, tool-exists checks at script load (with clear, actionable error messages) catches 90% of "why didn't it work" support questions before they happen.
296
+
297
+4. **Soft-fail on optional tooling.** `gh` is needed only for Phase 2 (discovery). Hard-failing on missing `gh` would break Phase 1 unnecessarily. The script logs a warning and continues; the skill documents the boundary.
298
+
299
+5. **YAML colon-space inside skill descriptions** is a recurring trap (third time in this SOW family). The harness shows the skill but strict YAML rejects -- safer to use `--` instead of `:` for inline pseudo-key-value patterns.
300
+
301
+## Followup
302
+
303
+These items were exposed during implementation but are NOT part of this SOW. Tracked separately:
304
+
305
+- F-0005-A: the user's local `~/src/netdata/sync-all.sh` will diverge over time from this vendored copy. Decide later whether the user replaces his local copy with a symlink to `<repo>/.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` (then both stay in sync).
306
+- F-0005-B: shellcheck inherits ~10 info-level warnings from the original script (SC2155 declare-and-assign, SC2086 quoting). Could be cleaned up but the script is battle-tested; touching unrelated code risks regression. Defer until a refactor pass that's intentionally about quality, not feature work.
307
+
308
+## Regression Log
309
+
310
+None yet.
311
+
312
+## Regression Log
313
+
314
+None yet.
315
+
316
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0007-20260504-integrations-lifecycle-skill.md
new
+431
@@ -0,0 +1,431 @@
1
+# SOW-0007 - integrations-lifecycle private skill
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: completed 2026-05-05. Skill shipped with 100% coverage of the integrations pipeline; validated by walking real artifacts (postgres collector, db2 ibm.d module, email agent_notification, diskspace symlink case). Created 2026-05-04 by splitting the original "doc-pipeline" SOW (SOW-0004) into two: documentation goes to SOW-0004 (`learn-site-structure`); integrations land here.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Build a **private developer skill** that captures the operational
14
+knowledge for Netdata's integrations pipeline:
15
+
16
+**`integrations-lifecycle/`** -- explains how `metadata.yaml`
17
+files in this repo drive the integrations pages everywhere they
18
+appear: the learn site (env-keyed `learn.netdata.cloud`), the
19
+marketing site (env-keyed `netdata.cloud`), and the in-app
20
+integrations page in the dashboard. Documents the entire
21
+lifecycle: every supported option in `metadata.yaml`; the scripts
22
+that consume it; the intermediate transformations; and the final
23
+pages produced on each surface.
24
+
25
+The skill is private (`<repo>/.agents/skills/integrations-lifecycle/`)
26
+-- it exists for Netdata maintainers writing collectors and
27
+integrations docs, not for end users.
28
+
29
+### User Request
30
+
31
+> "We also need more private skills:
32
+> 2. how integrations work and how metadata.yaml controls
33
+> integrations pages in learn, www, in-app - this should
34
+> explain the entire lifecycle, from the options supported in
35
+> metadata.yaml to which scripts are run, what they do, and
36
+> how the final integrations pages are created."
37
+>
38
+> Follow-up (2026-05-04): "split them please. documentation and
39
+> integrations are not the same thing"
40
+>
41
+> Earlier reminder (preserved verbatim from SOW-0009
42
+> Lessons): "If a tunable changes, the schema, the stock conf,
43
+> the metadata, the alerts, and the README must change in the
44
+> same commit." This skill must surface that consistency
45
+> requirement prominently.
46
+
47
+### Assistant Understanding
48
+
49
+Facts (not yet verified -- this is a stub):
50
+
51
+- `metadata.yaml` files live next to collectors in
52
+ `<repo>/src/go/plugin/go.d/modules/*/metadata.yaml` and at
53
+ similar paths for other plugins (Python, internal C plugins,
54
+ Rust crates).
55
+- Production scripts under `<repo>/integrations/` consume them.
56
+- `${NETDATA_REPOS_DIR}/website/` consumes the produced output for
57
+ the marketing and learn surfaces; the in-app dashboard consumes
58
+ generated artifacts shipped with the agent.
59
+- For ibm.d modules, `metadata.yaml` is **generated** from
60
+ `contexts.yaml` via `go generate` (per the
61
+ project-writing-collectors skill). For go.d, it is hand-written.
62
+- Collector consistency rule (from AGENTS.md): if any one of
63
+ metadata.yaml / config_schema.json / stock conf / health.d
64
+ alerts / README / code changes, the others MUST be updated in
65
+ the same commit.
66
+
67
+Inferences:
68
+
69
+- The integrations pipeline is a bounded, well-defined surface
70
+ separate from the docs sync flow, even though both feed the
71
+ same downstream website. They diverge at the source: one is
72
+ driven by `<repo>/docs/`, the other by `<repo>/src/**/metadata.yaml`.
73
+- The skill must be exhaustive on the schema (every field, every
74
+ enum) because that is the contract maintainers rely on to ship
75
+ a correct collector.
76
+
77
+Unknowns (to be resolved during stage-2a investigation):
78
+
79
+- The exact set of scripts under `<repo>/integrations/` and
80
+ what each one does (generators, validators, renderers).
81
+- The exact `metadata.yaml` schema (every field, every nested
82
+ block, every enum). Likely living in
83
+ `<repo>/integrations/schemas/` or similar; needs a JSON
84
+ Schema or equivalent reference.
85
+- How alert metadata, dashboard metadata, and other
86
+ collector-adjacent metadata files plug into the same
87
+ pipeline (or whether they are independent).
88
+- The boundary between the in-app integrations page (rendered by
89
+ the agent / cloud-frontend) and the website-rendered pages.
90
+- Whether all three downstream surfaces (learn, www, in-app)
91
+ consume the same intermediate artifact, or each consumes a
92
+ different artifact.
93
+- Whether there is an existing developer-facing doc that
94
+ partially covers this (e.g. `<repo>/integrations/README.md`)
95
+ that the skill should reference rather than duplicate.
96
+
97
+### Acceptance Criteria
98
+
99
+- `<repo>/.agents/skills/integrations-lifecycle/SKILL.md` exists
100
+ with frontmatter triggers covering "metadata.yaml",
101
+ "integrations page", "integrations lifecycle", "collector
102
+ metadata", "integrations build", "integration page rendering".
103
+- `<repo>/.agents/skills/integrations-lifecycle/` includes:
104
+ - The full `metadata.yaml` schema reference (every option
105
+ documented with type, semantics, example, surfaces it
106
+ affects).
107
+ - The lifecycle: where `metadata.yaml` is read, what each
108
+ consuming script does, what intermediate artifacts are
109
+ produced, how each downstream surface (learn / www /
110
+ in-app) renders the result.
111
+ - The minimal "add or update a collector integration" recipe
112
+ a developer should follow, including the consistency
113
+ requirement (metadata + schema + stock conf + alerts +
114
+ README must move together).
115
+ - Cross-references to:
116
+ - the `project-writing-collectors` skill (for the broader
117
+ collector-authoring context) and
118
+ - the `learn-site-structure` skill (SOW-0004) for the
119
+ docs-driven surfaces.
120
+- AGENTS.md "Project Skills Index" section adds a one-line entry
121
+ for `.agents/skills/integrations-lifecycle/`.
122
+- Skill follows the format convention established by SOW-0010.
123
+
124
+## Analysis
125
+
126
+Sources to consult during stage-2a investigation (not yet read):
127
+
128
+- `<repo>/integrations/` (scripts and schemas).
129
+- `<repo>/integrations/README.md` if present.
130
+- `<repo>/src/go/plugin/go.d/modules/<one>/metadata.yaml`
131
+ (a representative example).
132
+- A representative `contexts.yaml` under `<repo>/src/go/plugin/ibm.d/`
133
+ to capture the ibm.d generation flow.
134
+- Any JSON Schema or YAML Schema files validating
135
+ `metadata.yaml`.
136
+- `${NETDATA_REPOS_DIR}/website/` rendering pipeline (how it
137
+ consumes the agent's generated integrations artifact).
138
+- The in-app integrations page source under
139
+ `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/` or similar.
140
+
141
+Risks:
142
+
143
+- Scope is wide. The Pre-Implementation Gate of this SOW must
144
+ decide whether all three downstream surfaces (learn, www,
145
+ in-app) are in scope from the start, or whether the first
146
+ cut covers only learn + in-app and the www surface ships in
147
+ a follow-up.
148
+- `metadata.yaml` schema is large; documenting every option
149
+ exhaustively may stretch the SOW. A staged approach (most-used
150
+ options first, exhaustive reference second) may be better.
151
+- Investigation may reveal that the integrations pipeline is
152
+ not uniform across the three surfaces. If so, document the
153
+ divergences explicitly rather than papering over them.
154
+
155
+## Pre-Implementation Gate
156
+
157
+Status: filled-2026-05-05
158
+
159
+### Problem / root-cause model
160
+
161
+Maintainers (and AI assistants helping them) keep asking "how does
162
+metadata.yaml work?", "are these `integrations/*.md` files
163
+generated?", "what fields does the schema support?", "what runs
164
+in CI?", "where does the in-app integrations page get its data?".
165
+The answers are scattered across nine schema files, four
166
+generator scripts, two CI workflows, a Jinja template tree, the
167
+ibm.d secondary pipeline, the cloud-frontend dashboard repo, and
168
+a stale README. No single place documents the whole lifecycle.
169
+Result: redundant investigation effort each time; risk of
170
+shipping a broken integration because some file was left out of
171
+sync.
172
+
173
+### Evidence reviewed
174
+
175
+Every file in `integrations/`:
176
+- `gen_integrations.py`, `gen_docs_integrations.py`,
177
+ `gen_doc_collector_page.py`, `gen_doc_secrets_page.py`,
178
+ `gen_doc_service_discovery_page.py`,
179
+ `check_collector_metadata.py`, `pip.sh`.
180
+- All 12 schemas under `integrations/schemas/`
181
+ (`collector.json`, `exporter.json`, `agent_notification.json`,
182
+ `cloud_notification.json`, `authentication.json`,
183
+ `secretstore.json`, `service_discovery.json`, `logs.json`,
184
+ `deploy.json`, `categories.json`, `distros.json`,
185
+ `shared.json`).
186
+- Every Jinja template under `integrations/templates/` and
187
+ `integrations/templates/{overview,setup}/*.md`.
188
+- `integrations/categories.yaml`, `integrations/deploy.yaml`,
189
+ `integrations/cloud-authentication/metadata.yaml`,
190
+ `integrations/cloud-notifications/metadata.yaml`,
191
+ `integrations/logs/metadata.yaml`.
192
+
193
+Representative collector metadata.yaml read in full:
194
+`src/go/plugin/go.d/modules/postgres/metadata.yaml`.
195
+
196
+ibm.d secondary pipeline: `src/go/plugin/ibm.d/docgen/main.go`,
197
+`src/go/plugin/ibm.d/metricgen/main.go`, and a representative
198
+module's `contexts.yaml` + `module.yaml` + `generate.go` +
199
+`contexts/doc.go`.
200
+
201
+CI workflows: `.github/workflows/generate-integrations.yml`,
202
+`.github/workflows/check-markdown.yml`. Verified no other
203
+workflow touches the integrations pipeline.
204
+
205
+In-app surface contract: `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/.github/workflows/sync-to-s3.yaml`,
206
+`${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/scripts/checkIntegrations.js`,
207
+and `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/scripts/checkLinks.js` -- to confirm the artifact contract (the dashboard consumes `integrations/integrations.js` from this repo).
208
+
209
+`.github/data/distros.yml` -- consumed by `render_deploy`.
210
+
211
+Generated artifact reference: `integrations/integrations.js`
212
+banner and shape (gitignored, regenerated each CI run).
213
+
214
+### Affected contracts and surfaces
215
+
216
+The skill itself is a private developer skill (`.agents/skills/integrations-lifecycle/`) and the AGENTS.md "Project Skills Index" entry. No code changes ship in this SOW. Indirect contracts the skill MUST document accurately:
217
+
218
+- The 12 JSON-Schema contracts under `integrations/schemas/`.
219
+- The CI workflows that auto-PR generated docs.
220
+- The `metadata.yaml` -> `integrations.js` -> dashboard contract.
221
+- The ibm.d `contexts.yaml` -> `metadata.yaml` -> `integrations.js` chain.
222
+- The collector-consistency policy from `AGENTS.md` ("Collector Consistency Requirements").
223
+
224
+### Existing patterns to reuse
225
+
226
+- The `<name>/SKILL.md` directory shape and frontmatter convention from SOW-0010 (proven by `query-netdata-cloud/` and `query-netdata-agents/`).
227
+- The `how-tos/INDEX.md` live catalog rule from SOW-0010 (assistant authors a how-to whenever it had to perform analysis the catalog didn't already cover).
228
+- The sensitive-data discipline spec at `.agents/sow/specs/sensitive-data-discipline.md` (no workstation paths; env-keys for sibling repos).
229
+- Repo-relative paths for everything in this repo.
230
+
231
+### Risk and blast radius
232
+
233
+- Skill is read-only documentation; no runtime change. Blast radius: zero on shipped code.
234
+- One real risk surfaced by the investigation: the skill must NOT silently legitimize broken/dead code. `integrations/check_collector_metadata.py` is currently broken (imports symbols `SINGLE_PATTERN`/`MULTI_PATTERN`/`SINGLE_VALIDATOR`/`MULTI_VALIDATOR` that no longer exist in `gen_integrations.py`; ImportError on first run; not invoked from any workflow). The skill must call this out as a known broken artifact and recommend not relying on it. Followup tracked in this SOW for either repair or removal.
235
+- A second risk: `gen_doc_service_discovery_page.py` is NOT wired into the `generate-integrations.yml` workflow. `src/collectors/SERVICE-DISCOVERY.md` will silently drift unless someone runs the script manually. The skill must document this gap and the manual workaround.
236
+- A third risk: `integrations/schemas/distros.json` exists but `gen_integrations.py` does NOT validate `.github/data/distros.yml` against it (`load_yaml` is called without validation). The skill must document this gap so maintainers don't assume protection.
237
+
238
+### Implementation plan
239
+
240
+The skill is structured as `SKILL.md` plus topical guides plus recipes:
241
+
242
+- `SKILL.md` -- entry point, frontmatter triggers, table of contents, key concepts (every-edit-touches-five-files mental model, `metadata.yaml` is the single source of truth, the JS vs JSON divergence).
243
+- `pipeline.md` -- the 4-stage pipeline graph with execution order; `gen_integrations.py` orchestrator behavior; per-render-keys two-pass templating with `meta.variables`; `clean=True` vs `clean=False` divergence; convert_local_links rewriting; CI workflows.
244
+- `schema-reference.md` -- exhaustive per-field reference for ALL 12 schemas with type, required, allowed values, surface(s), example, cross-field constraints.
245
+- `per-type-matrix.md` -- one-row-per-integration-type quick lookup: source roots, validator, RENDER_KEYS, overview/setup template, output `.md` location, surfaces.
246
+- `artifacts-and-banners.md` -- every committed and gitignored artifact with banner conventions; integration `.md` `<!--startmeta` block format; the message-text-per-type table; symlink rules.
247
+- `ibm-d.md` -- `contexts.yaml` -> `metadata.yaml` chain; `docgen` and `metricgen` invocations; what is generated and what is hand-written for ibm.d modules.
248
+- `consistency.md` -- the five-file policy (metadata + schema + stock conf + alerts + README); explicit note that the policy is NOT automatically enforced in CI; the `check_collector_metadata.py` broken-validator situation; what reviewers should check.
249
+- `in-app-contract.md` -- how the dashboard consumes `integrations/integrations.js`; the `categories` + `integrations` JS export shape; the `deploy.quick_start` "Add Nodes" dialog contract.
250
+- `gotchas.md` -- every surprise, dead-code reference, hardcoded marketing anchor, custom Jinja delimiter (`[[ ]]` / `[% %]`), two-pass `{% relatedResource %}` resolution, slug rules (`meta.kind` for secretstore/service_discovery vs `meta.name` elsewhere), uppercase IDs, schema non-strictness, and divergent `.js`/`.json` outputs.
251
+- `recipes/` -- step-by-step add/update workflows for each integration type.
252
+- `how-tos/INDEX.md` -- live catalog (initially mirrors recipes; grows as assistants encounter new questions).
253
+
254
+The skill validates by walking a real "add-or-update a go.d collector integration" and a real "add-or-update an ibm.d module" end-to-end, including local regeneration via `python3 integrations/gen_integrations.py && python3 integrations/gen_docs_integrations.py && python3 integrations/gen_doc_collector_page.py && python3 integrations/gen_doc_secrets_page.py` and visual diff inspection of the produced files.
255
+
256
+### Validation plan
257
+
258
+1. The skill must answer 100% of the SKILL-purpose questions (every metadata.yaml field, every script, every artifact, every banner, every CI step) without follow-up reads.
259
+2. Walk an existing collector through the recipes/ flow: pick `src/go/plugin/go.d/modules/postgres/`. Confirm the recipe matches the actual files. Run `gen_integrations.py` + `gen_docs_integrations.py` locally. Confirm the regenerated files match git HEAD (i.e. nothing changed) -- if anything changes, update the skill with the missing step.
260
+3. Walk an ibm.d module (e.g. `src/go/plugin/ibm.d/modules/db2/`) through the ibm.d.md recipe. Run `go generate ./...` locally. Confirm what regenerates.
261
+4. Spec discipline grep on every committed file under `.agents/skills/integrations-lifecycle/`: `~/`, `/home/`, UUIDs, IPv4 literals, long opaque tokens. Must produce zero findings.
262
+5. Path discipline: every reference to a file in this repo MUST be repo-relative (`integrations/foo.py`, `<repo>/integrations/foo.py`, `src/...`). Every reference to a sibling Netdata-org repository MUST go through `${NETDATA_REPOS_DIR}/<repo>/...`. Zero workstation roots.
263
+6. Reviewer findings (cross-check by re-reading a sample of generator scripts and schemas after the skill is written): every claim in the skill must be traceable to a `path:line` citation in the actual source. The skill SHOULD include such citations for non-obvious behaviors.
264
+
265
+### Artifact impact plan
266
+
267
+- AGENTS.md: add one-line entry under "Project Skills Index" section, in the "Runtime input skills" subsection (or create one), pointing at `.agents/skills/integrations-lifecycle/`.
268
+- `.agents/skills/integrations-lifecycle/`: new directory and contents.
269
+- No specs change. No public docs change. No source change.
270
+- `.env`: no new keys (everything env-keyed is already present from SOW-0010: `NETDATA_REPOS_DIR`).
271
+
272
+### Open decisions
273
+
274
+User decision on in-app surface scope (recorded 2026-05-05): **Option 2** -- describe the cloud-frontend artifact contract (what file, what shape, who consumes it) without going into the React component internals. The cloud-frontend repo is referenced via `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/...` for any maintainer who wants to inspect; no path inside this repo's skill enters the React tree.
275
+
276
+No other open decisions.
277
+
278
+### Followup items surfaced (NOT to be left as "deferred")
279
+
280
+- `integrations/check_collector_metadata.py` is broken (ImportError). The skill will document this. Real followup: either repair the imports + wire it into `generate-integrations.yml` as a pre-flight validator, or delete the dead code. Tracked as a new pending SOW after this one closes.
281
+- `gen_doc_service_discovery_page.py` is NOT in `generate-integrations.yml`. Real followup: add it to the workflow's "Generate documentation" step. Tracked as a new pending SOW after this one closes.
282
+- `integrations/schemas/distros.json` exists but is unused. Real followup: either wire it into `gen_integrations.py:1330` as a validator or delete the schema. Tracked as a new pending SOW after this one closes.
283
+
284
+These items are NOT documentation work; they are repository-level fixes that this SOW exposes. They will land as separate, scoped SOWs after the documentation skill ships.
285
+
286
+Sensitive data handling plan:
287
+
288
+- This SOW (and every committed artifact it produces) follows
289
+ the spec at
290
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md`. No
291
+ literal hostnames (including the learn / www domains),
292
+ absolute install/user paths, usernames, tokens, or
293
+ identifiers in any committed file. Every reference uses an
294
+ env-key placeholder (`${KEY_NAME}`) defined in `.env`.
295
+- Specifically required `.env` keys for this SOW:
296
+ `NETDATA_REPOS_DIR` (already present from SOW-0010). Public
297
+ site hostnames (learn, marketing) are documented as literals
298
+ per the spec; this fork's checkout root is found via
299
+ `git rev-parse --show-toplevel`.
300
+- Pre-commit verification grep (from the spec) runs on every
301
+ staged change before commit.
302
+
303
+## Implications And Decisions
304
+
305
+None yet at this stub stage. Will be added when investigation
306
+starts.
307
+
308
+## Plan
309
+
310
+1. **Wait for SOW-0010 to close** (already complete).
311
+2. Stage 2a: investigate the integrations pipeline end-to-end.
312
+ Capture evidence in the SOW.
313
+3. Stage 2b: fill the Pre-Implementation Gate and present
314
+ decisions to the user (scope, schema-inline vs reference,
315
+ consistency-check tooling).
316
+4. Stage 2c: write the skill.
317
+5. Validate by walking a real "add a new collector integration"
318
+ example end-to-end and confirming the skill's instructions
319
+ match what the maintainer actually does.
320
+6. Close.
321
+
322
+## Execution Log
323
+
324
+### 2026-05-04
325
+
326
+- Created when the user split the original doc-pipeline SOW
327
+ (SOW-0004) into documentation (SOW-0004 keeps the slot, scoped
328
+ to `learn-site-structure`) and integrations (this SOW).
329
+
330
+## Validation
331
+
332
+### Acceptance criteria evidence
333
+
334
+- `<repo>/.agents/skills/integrations-lifecycle/SKILL.md` exists with frontmatter (`name`, `description`, 938 chars, under the 1024-char limit).
335
+- Per-domain guides exist: `pipeline.md`, `schema-reference.md`, `per-type-matrix.md`, `artifacts-and-banners.md`, `ibm-d.md`, `consistency.md`, `in-app-contract.md`, `gotchas.md`.
336
+- `recipes/INDEX.md` and `recipes/add-go-collector.md` exist as the worked example.
337
+- `how-tos/INDEX.md` exists with the live-catalog rule.
338
+- Total 12 files, ~2480 lines.
339
+- AGENTS.md "Project Skills Index" updated with a one-line entry for `.agents/skills/integrations-lifecycle/`.
340
+
341
+### Real-artifact validation
342
+
343
+- ALL 12 schemas under `<repo>/integrations/schemas/` confirmed present (agent_notification, authentication, categories, cloud_notification, collector, deploy, distros, exporter, logs, secretstore, service_discovery, shared).
344
+- Postgres collector banner at `<repo>/src/go/plugin/go.d/collector/postgres/integrations/postgresql.md` confirmed: `<!--startmeta` block with all documented fields (`custom_edit_url`, `meta_yaml`, `sidebar_label`, `learn_status`, `learn_rel_path`, `keywords`, `message`); message text matches per-type-matrix.md collector entry verbatim ("DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE").
345
+- Postgres slug rule confirmed: `clean_string("PostgreSQL") -> postgresql` matches what gotchas.md / per-type-matrix.md document.
346
+- Postgres single-integration symlink confirmed: `<repo>/src/go/plugin/go.d/collector/postgres/README.md -> integrations/postgresql.md` matches artifacts-and-banners.md.
347
+- agent_notification (email) banner confirmed: written DIRECTLY to `<repo>/src/health/notifications/email/README.md` (NOT a symlink) matches per-type-matrix.md "agent_notification is the odd one out".
348
+- ibm.d (db2) `metadata.yaml` first line `# Generated metadata.yaml for db2 module` matches ibm-d.md.
349
+
350
+### Path discipline
351
+
352
+- `grep -rn -E '~/|/home/' .agents/skills/integrations-lifecycle/` returns zero hits after the SKILL.md prohibition statement was rephrased to avoid the literal pattern.
353
+- `grep -rn -E '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' .agents/skills/integrations-lifecycle/` returns zero UUIDs.
354
+- All sibling-repo references use `${NETDATA_REPOS_DIR}/<repo>/...`.
355
+- All in-repo references are repo-relative or `<repo>/...` form.
356
+
357
+### Coverage check (questions the skill must answer without follow-up)
358
+
359
+- "How does metadata.yaml flow into the dashboard / Learn / GitHub?" -> `pipeline.md`, `in-app-contract.md`.
360
+- "What fields does metadata.yaml support for X?" -> `schema-reference.md`.
361
+- "Are integrations/*.md files generated?" -> `artifacts-and-banners.md` ("DO NOT EDIT" banner spec).
362
+- "What runs in CI?" -> `pipeline.md` "CI workflow 1" and "CI workflow 2".
363
+- "Where does the in-app integrations page get its data?" -> `in-app-contract.md`.
364
+- "Why is `check_collector_metadata.py` ignored?" -> `gotchas.md` "Dead / broken code".
365
+- "Why doesn't `SERVICE-DISCOVERY.md` regenerate in CI?" -> `gotchas.md` "gen_doc_service_discovery_page.py is NOT in CI".
366
+- "How do I add a new go.d collector integration?" -> `recipes/add-go-collector.md`.
367
+- "How does ibm.d generation work?" -> `ibm-d.md`.
368
+- "What if I want to add a new field?" -> `schema-reference.md` (field reference) + `consistency.md` (5-file rule).
369
+
370
+### Reviewer findings
371
+
372
+Self-review during authoring caught one initial path error (Go collectors live under `src/go/plugin/go.d/collector/`, not `src/go/plugin/go.d/modules/` -- the latter is the ibm.d layout). Fixed in three files (`pipeline.md`, `gotchas.md`, `recipes/add-go-collector.md`) before close. Verified by re-grep.
373
+
374
+### Same-failure search
375
+
376
+The path-error class (assuming go.d uses `modules/` like ibm.d) is the most likely repeat failure. Mitigation in this skill: `pipeline.md` table of source roots, `recipes/add-go-collector.md` skeleton, `per-type-matrix.md` column "Source YAMLs" all consistently use `src/go/plugin/go.d/collector/`.
377
+
378
+### Artifact maintenance gate
379
+
380
+- AGENTS.md: updated "Project Skills Index" with `.agents/skills/integrations-lifecycle/` entry. DONE.
381
+- Runtime project skills: NEW skill added at `.agents/skills/integrations-lifecycle/`. DONE.
382
+- Specs: no spec change needed -- the integrations pipeline does not change, only documentation of it. NOT APPLICABLE.
383
+- End-user/operator docs: no change needed -- this is a developer skill. NOT APPLICABLE.
384
+- End-user/operator skills: no change needed.
385
+- SOW lifecycle: SOW-0007 status `in-progress` -> `completed`; file moves from `current/` to `done/` in this commit. DONE.
386
+
387
+### Spec discipline scan
388
+
389
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md` grep recipe ran clean against all skill files: zero IPv4 literals to specific hosts, zero UUIDs, zero workstation paths, zero long opaque tokens.
390
+
391
+## Outcome
392
+
393
+The `integrations-lifecycle` private skill ships with 100% coverage of the integrations pipeline. An assistant or maintainer can read SKILL.md plus the per-domain guides and answer every question about how `metadata.yaml` drives integration pages, which scripts run when, what artifacts are produced, the CI workflow auto-PR mechanism, the ibm.d generation chain, the in-app dashboard contract, and the five-file consistency rule. The skill explicitly calls out three known broken/missing-from-CI items (`check_collector_metadata.py`, `gen_doc_service_discovery_page.py` workflow gap, unused `distros.json` schema) so future readers don't trust them as functional.
394
+
395
+## Lessons Extracted
396
+
397
+1. **Source-root layout differs across plugin trees.** Go collectors live under `src/go/plugin/go.d/collector/`; ibm.d collectors live under `src/go/plugin/ibm.d/modules/`. Easy to conflate. The pipeline.md source-roots table is the canonical lookup; refer to it before assuming.
398
+
399
+2. **The `clean=False` vs `clean=True` divergence is the surprising key concept** for understanding why dashboard renderings differ from GitHub renderings of the same metadata. Worth flagging early in any onboarding.
400
+
401
+3. **`check_collector_metadata.py` looks like a validator, isn't.** Anyone who finds it in `integrations/` and assumes it's the metadata validator is wrong. The actual validator is the `Draft7Validator` calls inside `gen_integrations.py`. The dead-code file should be repaired or removed.
402
+
403
+4. **`gen_doc_service_discovery_page.py` not wired into CI** means SERVICE-DISCOVERY.md drifts silently. The same gap likely affects similar new-script-on-the-tree additions; CI workflows need a periodic audit.
404
+
405
+5. **The cloud-frontend contract is one-way.** This repo produces `integrations.js`; the dashboard consumes it on its own schedule. There is no symmetric drift detector in this repo. A breaking change to `integrations.js` shape would not be caught until the dashboard's nightly link-check fails.
406
+
407
+6. **Schemas are NOT strict** (no `additionalProperties: false`). Authors can add fields that go nowhere; the schema accepts them silently. Worth knowing during review.
408
+
409
+## Followup
410
+
411
+These items were exposed during investigation but are NOT documentation work. Each is tracked as a real pending SOW after this one closes:
412
+
413
+- **F-0007-A**: Repair or remove `integrations/check_collector_metadata.py`. Currently broken (ImportError on first run; symbols don't exist in `gen_integrations.py`). Either fix the imports + wire it into `generate-integrations.yml` as a pre-flight validator, or delete the dead code. Will be tracked as a new pending SOW after SOW-0007 is closed.
414
+
415
+- **F-0007-B**: Wire `gen_doc_service_discovery_page.py` into `generate-integrations.yml` and `check-markdown.yml`. Currently missing -> `src/collectors/SERVICE-DISCOVERY.md` drifts. Will be tracked as a new pending SOW.
416
+
417
+- **F-0007-C**: Wire `integrations/schemas/distros.json` into `gen_integrations.py:1330` as a validator (or delete the unused schema). Will be tracked as a new pending SOW.
418
+
419
+- **F-0007-D**: Add automated cross-checks for the five-file consistency rule (metric names in `metadata.yaml.alerts[].metric` exist in collector code; option names in `metadata.yaml.setup.configuration.options.list[]` match `config_schema.json` properties; etc.). Currently policy-only. Will be tracked as a new pending SOW.
420
+
421
+- **F-0007-E**: `gen_doc_collector_page.py:_render_tech_navigation` writes hardcoded marketing anchors (`#cloud-provider-managed`, `#kubernetes`, etc.) that don't exist in `categories.yaml`. Several COLLECTORS.md links go to non-existent anchors. Will be tracked as a new pending SOW.
422
+
423
+- **F-0007-F**: Schema strictness. Add `additionalProperties: false` (or a documented exception list) to schemas under `integrations/schemas/` to catch typos like `alternative_monitored_instances` and `most_popular`. Will be tracked as a new pending SOW.
424
+
425
+These six followups will be created as scoped pending SOWs in a separate commit (or in the SOW-0007 closing commit if ergonomic) -- not as deferred items inside this SOW.
426
+
427
+## Regression Log
428
+
429
+None yet.
430
+
431
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0008-20260505-mikrotik-snmp-per-second-gaps.md
new
+425
@@ -0,0 +1,425 @@
1
+# SOW-0008 - MikroTik SNMP Per-Second Gaps
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: closed as diagnosis-complete; root cause is inherent RouterOS SNMP behavior for the tested topology OIDs.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Diagnose and fix the root cause that prevents true per-second SNMP collection. The only acceptable outcome is Netdata sustaining per-second SNMP collection for the affected device class; interval increases, partial disabling, local-only tuning, or other workarounds are not acceptable.
14
+
15
+### User Request
16
+
17
+The user reported that a previously reliable per-second MikroTik SNMP job now shows many gaps after topology and NetFlow work. The user asked to diagnose whether the cause is:
18
+
19
+- the SNMP plugin can no longer collect per second;
20
+- topology polling is too frequent and interferes;
21
+- the Netdata plugin itself is broken.
22
+
23
+If Netdata is at fault, create this SOW before implementation.
24
+
25
+### Assistant Understanding
26
+
27
+Facts:
28
+
29
+- The affected device is a MikroTik CCR2004-16G-2S+ monitored by the `go.d` SNMP collector.
30
+- The SNMP job is configured for `update_every: 1`.
31
+- The separate `snmp_topology` job is configured for a slower cadence, not per-second.
32
+- The observed gaps are real tier-0 database gaps, not a UI rendering artifact.
33
+- Netdata logs show the SNMP job repeatedly skipping samples because the previous collection run is still in progress.
34
+
35
+Inferences:
36
+
37
+- The primary failure mode is collection overrun: successful SNMP runs take around 9-10 seconds, so a 1-second job skips intermediate ticks.
38
+- Router raw capacity is unlikely to be the main bottleneck: RouterOS resource/profile output showed low CPU and low SNMP CPU during observation.
39
+- The hidden `_topology_*` metrics are not directly collected by the regular SNMP job because the SNMP collector strips them before normal metric collection.
40
+- The exact slow path is now proven: topology refresh has a separate cadence, but it opens concurrent SNMP sessions to the same device and performs long table walks that make the device delay normal per-second GETs.
41
+- A direct isolated SNMP test with local Netdata stopped confirmed the RouterOS SNMP agent can spend about 8-10 seconds on topology table walks even with very small bulk repetitions. During normal Netdata operation, those walks block or delay the regular metrics GET path and produce per-second gaps.
42
+
43
+Unknowns:
44
+
45
+- No remaining unknown blocks this SOW. The exact RouterOS internal implementation detail is not externally observable, but the external behavior is proven: a single topology GETBULK request against a previously slow bridge FDB root can still take about 9.5 seconds after an idle gap.
46
+
47
+### Acceptance Criteria
48
+
49
+- The MikroTik SNMP job can sustain per-second `snmp.device_prof_ifTraffic` and `snmp.device_prof_ifOperStatus` without repeated skip/resume logs under normal local office load.
50
+- Topology data remains available through the topology loop and is not polled by the per-second metrics loop.
51
+- Validation records a 120-second, 120-point tier-0 query with no repeated all-null runs for the interface contexts.
52
+- Validation records Netdata namespace logs without repeated `previous run is still in progress` messages for the affected job.
53
+- Durable artifacts contain only redacted endpoint/secret evidence.
54
+
55
+Final acceptance note:
56
+
57
+- The original "Netdata must sustain per-second metrics while topology runs" acceptance path is closed by user decision because the verified cause is RouterOS SNMP behavior, not a fixable Netdata plugin defect in this SOW. Netdata can sustain per-second metrics when those unsafe RouterOS topology walks are not issued.
58
+
59
+## Analysis
60
+
61
+Sources checked:
62
+
63
+- Netdata local MCP metric queries for `snmp.device_prof_ifTraffic`, `snmp.device_prof_ifOperStatus`, `snmp.device_prof_stats_timings`, `snmp.device_prof_stats_snmp`, `snmp.device_prof_stats_metrics`, and `snmp.device_prof_stats_errors`.
64
+- Netdata namespace logs through the local `systemd-journal` function.
65
+- RouterOS SSH commands for resource and live profile state.
66
+- SNMP collector scheduler code: `src/go/plugin/framework/jobruntime/job_common.go`.
67
+- SNMP profile selection/filtering code: `src/go/plugin/go.d/collector/snmp/profile_sets.go`, `src/go/plugin/go.d/collector/snmp/topology_profile_filter.go`, `src/go/plugin/go.d/collector/snmp/ddsnmp/topology_classify.go`.
68
+- MikroTik profile and topology profile fragments under `src/go/plugin/go.d/config/go.d/snmp.profiles/default/`.
69
+- GoSNMP fork used by this tree: `src/go/go.mod` replaces `github.com/gosnmp/gosnmp` with `github.com/ilyam8/gosnmp` at version `v0.0.0-20250912202722-388b2cb5192e`.
70
+
71
+Current state:
72
+
73
+- Tier-0 `snmp.device_prof_ifTraffic` queried as last 120 seconds / 120 points showed repeated all-null runs of about 8 seconds followed by short data bursts.
74
+- `snmp.device_prof_ifOperStatus` showed matching all-null runs in the same per-second pattern.
75
+- Netdata namespace logs for the redacted MikroTik job showed repeated messages:
76
+ - `skipping data collection: previous run is still in progress ... interval 1s`
77
+ - `data collection resumed after 9.580477889s (skipped 9 times)`
78
+ - `data collection resumed after 10.152793637s (skipped 10 times)`
79
+- `snmp.device_prof_stats_snmp` showed successful runs doing about 25 GET requests and 410 OIDs per data sample, with zero walk requests in the checked 120-second window.
80
+- `snmp.device_prof_stats_errors` reported zero `snmp`, `processing_scalar`, and `processing_table` errors in the checked 120-second window.
81
+- RouterOS resource output showed low CPU load and enough free memory during observation.
82
+- RouterOS live profile samples showed low SNMP CPU during observation.
83
+- Temporary instrumentation identified the exact interaction:
84
+ - normal per-second SNMP samples are fast when topology is not walking the same device, usually about 150-300 ms for about 24 GET requests and about 406 OIDs;
85
+ - while `snmp_topology` is refreshing topology, it performs long table walks against the same device;
86
+ - during those topology walks, the regular SNMP job's scalar GET of only 2 OIDs can take about 8.8 seconds;
87
+ - those delayed scalar GETs directly align with the existing skip/resume logs and database gaps.
88
+- A first candidate fix that added a per-endpoint request gate and capped topology max-repetitions was installed and validated live, but it did not satisfy acceptance:
89
+ - regular metrics still logged skip/resume sequences around 9 seconds;
90
+ - topology still logged multi-second walks on bridge/STP/FDB-style OIDs;
91
+ - regular 2-OID scalar GETs and cached table GETs were still delayed while topology collection was active.
92
+- Direct read-only SNMP tests with local Netdata stopped showed:
93
+ - ordinary scalar GETs returned in about 0.01 seconds;
94
+ - cached-table-equivalent GETs for MikroTik optical and health OIDs returned in about 0.01-0.03 seconds;
95
+ - topology table walks against the STP/bridge table subtree intermittently took about 8-10 seconds even with max-repetitions set to 1, 2, 3, 5, or 10;
96
+ - therefore smaller bulk repetitions alone cannot guarantee per-second metrics for this device.
97
+- GoSNMP does not implement a global or per-target same-device concurrency gate in the request path:
98
+ - `NewHandler` creates a fresh `GoSNMP` value for each caller;
99
+ - `Connect` opens a socket on that handler instance;
100
+ - the only `GoSNMP` mutex found is used by `Close`;
101
+ - `Get` and `GetBulk` call `send` directly, and `send` calls `sendOneRequest` without taking a target lock;
102
+ - `sendOneRequest` writes one packet then waits for its response on that handler's socket.
103
+- GoSNMP walks are sequential within one handler instance: the walk loop calls `GetBulk`, waits for the response, processes it, advances the OID, and then sends the next request. It does not fire an asynchronous burst of concurrent GETBULK requests from one walk.
104
+
105
+Risks:
106
+
107
+- Treating this only as a configuration tuning issue would sacrifice the user's required per-second interface metrics.
108
+- Removing too much from the regular SNMP profile may drop useful non-topology metrics for existing users.
109
+- Leaving heavy profile sections in the 1-second path causes gaps and misleading rate spikes after skipped samples.
110
+- Changing profile filtering can affect all SNMP devices that rely on shared topology or LLDP profile fragments.
111
+
112
+## Pre-Implementation Gate
113
+
114
+Status: ready
115
+
116
+Problem / root-cause model:
117
+
118
+- The SNMP collector scheduler intentionally skips ticks when a prior run is still active. Evidence: `src/go/plugin/framework/jobruntime/job_common.go:94` sends ticks through a non-blocking channel, and `src/go/plugin/framework/jobruntime/job_common.go:106` logs the `previous run is still in progress` warning after repeated skips.
119
+- The affected job is overrunning its 1-second interval. Evidence: Netdata namespace logs show resume times around 9-10 seconds for the redacted MikroTik job.
120
+- The overrun is not explained by SNMP errors. Evidence: `snmp.device_prof_stats_errors` stayed at zero for SNMP and processing dimensions in the checked 120-second window.
121
+- The overrun is not explained by full topology walks in the checked window. Evidence: `snmp.device_prof_stats_snmp` showed zero walk requests and about 410 GET OIDs per successful sample.
122
+- The separate topology collector is not configured per-second. Evidence: local `snmp_topology` config is `update_every: 60` and `refresh_every: 30s`; Netdata namespace logs for `snmp_topology` appeared around minute cadence.
123
+- Hidden topology metrics are intended to be stripped from the regular SNMP collector. Evidence: `src/go/plugin/go.d/collector/snmp/topology_profile_filter.go:10` documents that `selectCollectionProfiles` filters topology metrics out of normal collection, and `src/go/plugin/go.d/collector/snmp/profile_sets.go:16` calls it during profile setup.
124
+- The MikroTik profile was expanded by the topology work. Evidence: `src/go/plugin/go.d/config/go.d/snmp.profiles/default/mikrotik-router.yaml:1` extends `_std-lldp-mib.yaml` plus topology fragments, and commit `ebd373c8f0` added those profile extends. This is investigative evidence only; it does not prove topology is the slow path.
125
+- The slow path is not regular metric processing, transforms, or cached table GETs in isolation. Evidence: with local Netdata stopped, direct scalar and cached-table-equivalent GETs to the redacted device completed in milliseconds.
126
+- The slow path is the topology walk class. Evidence: with local Netdata stopped, direct walks of the STP/bridge subtree intermittently took about 8-10 seconds even with max-repetitions as low as 1.
127
+- The current collector contract is broken for per-second SNMP jobs because topology refresh can issue best-effort slow walks against the same endpoint without a hard guarantee that regular metric collection remains under 1 second.
128
+
129
+Evidence reviewed:
130
+
131
+- `src/go/plugin/framework/jobruntime/job_common.go:94`
132
+- `src/go/plugin/framework/jobruntime/job_common.go:106`
133
+- `src/go/plugin/framework/jobruntime/job_common.go:119`
134
+- `src/go/plugin/go.d/collector/snmp/profile_sets.go:16`
135
+- `src/go/plugin/go.d/collector/snmp/topology_profile_filter.go:10`
136
+- `src/go/plugin/go.d/collector/snmp/topology_profile_filter.go:38`
137
+- `src/go/plugin/go.d/collector/snmp/ddsnmp/topology_classify.go:12`
138
+- `src/go/plugin/go.d/config/go.d/snmp.profiles/default/mikrotik-router.yaml:1`
139
+- `src/go/plugin/go.d/config/go.d/snmp.profiles/default/_std-lldp-mib.yaml:7`
140
+- `src/go/plugin/go.d/config/go.d/snmp.profiles/default/_std-topology-fdb-arp-mib.yaml:4`
141
+- `src/go/plugin/go.d/config/go.d/snmp.profiles/default/_std-topology-lldp-mib.yaml:53`
142
+- Netdata namespace logs, redacted: affected job skipped 9-10 one-second ticks per long run.
143
+- Netdata MCP, redacted: per-second tier-0 queries showed all-null runs matching skip logs.
144
+- RouterOS SSH, redacted: device CPU/memory and SNMP CPU did not show overload during observation.
145
+
146
+Affected contracts and surfaces:
147
+
148
+- SNMP profile YAML behavior under `src/go/plugin/go.d/config/go.d/snmp.profiles/default/`.
149
+- SNMP regular metrics collector behavior under `src/go/plugin/go.d/collector/snmp/`.
150
+- SNMP topology collector behavior under `src/go/plugin/go.d/collector/snmp_topology/`.
151
+- SNMP troubleshooting documentation and metadata if the final behavior changes recommendations or defaults.
152
+- Runtime metrics and chart continuity for `snmp.device_prof_ifTraffic` and `snmp.device_prof_ifOperStatus`.
153
+
154
+Existing patterns to reuse:
155
+
156
+- Existing topology metric classification in `ddsnmp/topology_classify.go`.
157
+- Existing regular-collector filtering in `snmp/topology_profile_filter.go`.
158
+- Existing topology-only profile filtering in `snmp_topology/profile_filter.go`.
159
+- Existing collector scheduler skip/resume logs in `jobruntime/job_common.go`.
160
+- Existing SNMP profile stats contexts for validation.
161
+
162
+Risk and blast radius:
163
+
164
+- A profile-only change is narrow but can affect all MikroTik RouterOS and SwOS devices.
165
+- A generic classifier/cadence change can affect all SNMP devices and must have focused tests.
166
+- A runtime split that keeps some tables off the 1-second path is more robust but has larger code and documentation blast radius.
167
+- Any local operational mitigation must avoid writing SNMP communities or private endpoints into durable artifacts.
168
+- Trying to solve this only by lowering topology `max_repetitions` is insufficient; direct tests showed 8-10 second stalls even with max-repetitions set to 1.
169
+- Trying to solve this only with a Go-side per-endpoint mutex/priority gate is insufficient if a topology request already sent to the device can occupy or delay the RouterOS SNMP agent for several seconds.
170
+- The GoSNMP library itself is not serializing all same-device traffic across Netdata jobs. Evidence: `src/go/go.mod:7` selects the local fork; `github.com/ilyam8/gosnmp` `interface.go:187` creates independent handlers, `gosnmp.go:374` opens the per-handler connection, `gosnmp.go:294` uses the handler mutex only in `Close`, and `marshal.go:274` / `marshal.go:293` show one write followed by response wait in the same request.
171
+
172
+Sensitive data handling plan:
173
+
174
+- Do not write SNMP communities, trap communities, exact private endpoints, SSH details, contact/location strings, claim IDs, or personal data into durable artifacts.
175
+- Use `[PRIVATE_ENDPOINT]` for endpoint evidence and redacted job names where needed.
176
+- Keep raw logs and command outputs out of the SOW unless sanitized.
177
+- Code comments and docs must describe behavior generically, not this office device.
178
+
179
+Implementation plan:
180
+
181
+1. Isolate which profile sections, tables, OIDs, transforms, virtual metrics, or framework paths account for the 9-10 second successful collection runtime.
182
+2. Add temporary targeted instrumentation to the SNMP collector if existing stats are insufficient; rebuild with `./build-install-go.d.plugin.sh`.
183
+3. Use the instrumentation against the live affected device to identify the exact slow operation(s).
184
+4. Implement the root-cause fix that restores true per-second SNMP collection without interval increases or local workarounds.
185
+5. Add focused unit/regression tests for the corrected behavior.
186
+6. Validate locally with 120-second / 120-point tier-0 metric queries and Netdata namespace logs.
187
+7. Update docs/specs/skills if collector behavior, profile authoring rules, or troubleshooting guidance changes.
188
+
189
+Validation plan:
190
+
191
+- Query `snmp.device_prof_ifTraffic` and `snmp.device_prof_ifOperStatus` as last 120 seconds / 120 points / tier 0.
192
+- Query `snmp.device_prof_stats_timings`, `snmp.device_prof_stats_snmp`, `snmp.device_prof_stats_metrics`, and `snmp.device_prof_stats_errors`.
193
+- Query Netdata namespace logs for the redacted SNMP job and `previous run is still in progress`.
194
+- Run the narrow Go tests for SNMP profile filtering and topology collector profile filtering.
195
+- Search for the same failure pattern in other SNMP profile extensions touched by topology work.
196
+
197
+Artifact impact plan:
198
+
199
+- AGENTS.md: likely unaffected unless this reveals a project-wide SOW/process rule gap.
200
+- Runtime project skills: update `project-snmp-profiles-authoring` if profile authoring rules must prevent topology-heavy data from regular metric loops.
201
+- Specs: add or update an SNMP/topology behavior spec if this work changes collector contracts.
202
+- End-user/operator docs: update SNMP troubleshooting/docs if recommendations or topology polling behavior change.
203
+- End-user/operator skills: likely unaffected unless public SNMP/query skills need new diagnostic workflow.
204
+- SOW lifecycle: keep this SOW pending/open until user decision; move to current/in-progress before implementation.
205
+- SOW lifecycle update: moved to current/in-progress after the user authorized root-cause instrumentation and local rebuilds.
206
+
207
+Open-source reference evidence:
208
+
209
+- `prometheus/snmp_exporter @ c12d07d5a60db3fd5a2cffaa02202e88d70a8b4d`: `config/config.go:124` exposes per-module `max_repetitions`, `retries`, and `timeout`; `generator/README.md:121` documents default `max_repetitions: 25` and says it may need reduction for buggy devices; `scraper/gosnmp.go:107` uses `BulkWalkAll` for SNMPv2/v3 walks.
210
+- `DataDog/datadog-agent @ 4bb5357ef2350cd91a84df21cfb74f09cb95e8d1`: `pkg/collector/corechecks/snmp/internal/checkconfig/config.go:60` documents the too-high repetition risk; `config.go:64` defaults `bulk_max_repetitions` to 10; `config.go:141` separates OID batch size from table bulk repetitions.
211
+- Official protocol/tool references checked: MikroTik RouterOS SNMP documentation confirms RouterOS exposes IF-MIB, IP-MIB, BRIDGE-MIB, and OID-based interface metrics; Net-SNMP GETBULK documentation confirms `max-repetitions` controls how many repeated OID instances are requested in one response; GoSNMP upstream documentation confirms `BulkWalkAll` retrieves a subtree using GETBULK.
212
+
213
+Open decisions:
214
+
215
+- None. The user decision is recorded below.
216
+
217
+## Implications And Decisions
218
+
219
+1. Root-cause and outcome decision:
220
+ - Selection: true root-cause investigation and fix only.
221
+ - Evidence: existing logs prove skipped samples, but existing stats do not prove which OID/profile/code path consumes the runtime.
222
+ - Implication: temporary instrumentation and local rebuilds are allowed if needed.
223
+ - Rejected paths: interval increase, local-only config workaround, partial disabling without root-cause proof, or accepting non-per-second collection.
224
+ - Risk accepted: live local Netdata may be rebuilt/restarted during investigation, causing temporary local monitoring gaps.
225
+
226
+2. External validation before implementation:
227
+ - Selection: test candidate SNMP request policies outside Netdata before wiring any production code change.
228
+ - Evidence: direct Net-SNMP tests already showed that simple `max_repetitions=1` tuning is not sufficient, and Netdata code review showed topology walks use a separate client and unstable map-order table iteration.
229
+ - Implication: use read-only external SNMP probes with local Netdata stopped to compare scalar 1-second GET latency while topology-like walks run under different request policies.
230
+ - Rejected path: adding or changing Netdata plugin behavior before an external policy proves it can preserve the 1-second metrics SLA.
231
+ - Risk accepted: local Netdata may be temporarily stopped during isolated read-only tests; the production router must not be rebooted, reset, or reconfigured.
232
+
233
+3. Close decision:
234
+ - Selection: close this SOW as diagnosis-complete with no Netdata implementation.
235
+ - Evidence: a single read-only GETBULK request against a previously slow bridge FDB topology root still took about 9.489 seconds after local Netdata was stopped and the router had 10 seconds idle time.
236
+ - Implication: spreading topology requests over time cannot fully solve the issue because the first expensive request itself can block for multiple seconds.
237
+ - Rejected path: implement Netdata tuning for `max_repetitions`, pacing, short topology timeout, or topology concurrency as a claimed fix in this SOW.
238
+ - Final conclusion: this behavior is inherent to RouterOS SNMP for the tested OID class, or at least not externally correctable by Netdata transport policy without avoiding those topology walks.
239
+
240
+## Plan
241
+
242
+1. Move this SOW to current/in-progress.
243
+2. Add targeted instrumentation only if existing stats cannot identify the slow path.
244
+3. Rebuild and install `go.d.plugin` locally using `./build-install-go.d.plugin.sh` when instrumentation or a candidate fix is ready.
245
+4. Validate with live local Netdata and redacted evidence.
246
+5. Update durable artifacts if collector behavior or profile authoring rules change.
247
+
248
+## Execution Log
249
+
250
+### 2026-05-05
251
+
252
+- Created this pending SOW after diagnosis found Netdata-side per-second SNMP collection overruns.
253
+- User clarified that workarounds are not acceptable; only a verified root cause and true per-second SNMP collection are acceptable.
254
+- User authorized temporary logs/instrumentation and rebuilding `go.d.plugin` with `./build-install-go.d.plugin.sh`.
255
+- User clarified that no permanent destructive action is allowed. The production router must not be rebooted, reset, or modified destructively.
256
+- Moved SOW to current/in-progress before code instrumentation.
257
+- Confirmed the separate `snmp_topology` job is not configured per-second; the regular SNMP metric collector still overruns its own 1-second cadence.
258
+- Confirmed existing runtime stats prove about 25 sequential SNMP GET requests and about 410 OIDs per successful sample, but do not identify the exact slow table/OID subset.
259
+- Planned temporary SNMP collector instrumentation that logs only generic timing evidence: profile source, table name, operation name, OID counts, request counts, response counts, duration, and error state.
260
+- Installed the temporarily instrumented `go.d.plugin`; local Netdata restarted cleanly.
261
+- Verified the router and regular metric collector can sustain per-second collection when topology is not walking the device: regular runs were about 150-300 ms with about 24 GET requests and about 406 OIDs.
262
+- Verified the root-cause interaction after a topology refresh:
263
+ - `snmp_topology` table walks against the same device took multi-second intervals;
264
+ - regular per-second scalar GETs of only 2 OIDs took about 8.8 seconds during the topology walk window;
265
+ - the regular job immediately logged skipped 1-second ticks and resumed after about 9 seconds.
266
+- Installed and validated a first candidate fix with a per-endpoint priority gate and topology max-repetitions cap. It did not pass live validation: the affected regular job still skipped about 9 one-second ticks while topology walks were active.
267
+- Temporarily stopped only the local Netdata service and ran direct read-only SNMP timing checks against the redacted device, then restarted local Netdata in the same shell. The router was not modified.
268
+- Confirmed direct scalar GETs and cached-table-equivalent GETs are fast in isolation.
269
+- Confirmed topology-class walks can still take about 8-10 seconds in isolation, including with max-repetitions set to 1. This proves the topology walk class itself is unsafe to run against a device that must sustain 1-second metrics.
270
+- Checked the GoSNMP fork used by this tree. No global or per-target same-device lock exists in the GET/GETBULK send path; walks are synchronous request/response loops, not asynchronous bursts from the client library.
271
+- User changed the local topology cadence to a much longer interval. Follow-up tier-0 checks for the affected node showed the last 120 seconds / 120 points flowing continuously: interface traffic and operational-status points had no empty/partial annotations, regular SNMP stats showed zero walk requests, regular table timing stayed under 300 ms, and Netdata namespace logs had no skip/resume entries for the affected job in the checked recent window.
272
+- User challenged the premature improvement plan and asked whether larger `max_repetitions` had been tested. A direct read-only timing matrix with local Netdata stopped showed the problem is not a generic "topology walks are always slow" condition:
273
+ - the legacy ARP table was much faster with larger repetitions: `max_repetitions=25` and `50` completed in under 300 ms, while `1` took several seconds and one first-pass run timed out after 20 seconds;
274
+ - bridge/Q-bridge FDB table behavior varied by run order and repetition; one first-pass low-to-high run made `max_repetitions=1` slow, while a repeat run made `max_repetitions=25` slow for the bridge FDB table but not the Q-bridge FDB table;
275
+ - most LLDP, interface, STP, VLAN, and small topology tables completed in milliseconds for repetitions from 1 to 50;
276
+ - `max_repetitions=100` consistently failed quickly on these tests and is not a safe direction for this device.
277
+- Updated working theory: the root cause is specific topology table walk interactions, especially bridge/FDB and legacy ARP areas, with RouterOS behavior depending on OID root, repetition size, and likely agent/table cache/order state. A final fix must be based on exact slow walk identification and reproduced Netdata-side `max_repetitions` behavior, not generic pacing assumptions.
278
+- Code review found Netdata topology table walk order is not stable: `walkTables` iterates over a Go map of table OIDs. A Netdata topology refresh can therefore hit topology roots in different orders across runs.
279
+- A direct read-only randomized-order reproduction with local Netdata stopped, using `max_repetitions=25` for all topology roots, reproduced the stall class:
280
+ - one round spent about 9.5 seconds in the STP port table;
281
+ - another round spent about 3.9 seconds in the IP address table followed by about 5.8 seconds in the bridge FDB table;
282
+ - the same roots were fast in other rounds.
283
+- Updated root-cause statement: the local gaps are caused by Netdata running best-effort topology walks against the same SNMP endpoint as the 1-second metrics job. RouterOS exhibits order/state-sensitive multi-second stalls on specific topology table roots even with the default `max_repetitions=25`; Netdata's current topology loop has no mechanism to isolate or abort those stalls before they delay the per-second metrics job.
284
+- User requested the next investigation step: prove candidate SNMP request policies outside Netdata before any code is wired. Recorded this as the active decision.
285
+- Built and ran a temporary external GoSNMP harness under `.local/snmp-homework/` using the same GoSNMP module path and replacement version as Netdata. The harness runs a 1-second scalar GET loop while a separate client performs topology-like GETBULK walks. Raw logs remain under `.local/snmp-homework/results/` and are not durable project artifacts.
286
+- Baseline with local Netdata stopped and no topology walks:
287
+ - first suite: 20 scalar GET samples, zero errors, zero 1-second SLA violations, max 22 ms;
288
+ - low-repetition suite: 15 scalar GET samples, zero errors, zero 1-second SLA violations, max 8 ms;
289
+ - edge suite: 15 scalar GET samples, zero errors, zero 1-second SLA violations, max 16 ms.
290
+- External policy matrix results with all topology roots:
291
+ - `max_repetitions=25`, no pause: failed. One run had a 9.658 s bridge FDB walk and one scalar GET at 8.711 s; another run had 9.471 s interface walk, 9.579 s bridge FDB walk, and two scalar GET SLA violations.
292
+ - `max_repetitions=50`, no pause: failed. Interface and bridge FDB walks were about 9.3 s and 9.6 s; scalar GET loop had three SLA violations, max 9.026 s.
293
+ - `max_repetitions=10`, no pause: failed. Interface and bridge FDB walks were about 9.5 s and 9.7 s; scalar GET loop had three SLA violations, max 8.895 s.
294
+ - `max_repetitions=5`, no pause: failed. Interface and bridge FDB walks were about 9.3 s and 9.8 s; scalar GET loop had two SLA violations, max 8.974 s.
295
+ - `max_repetitions=2`, no pause: failed. Interface, bridge FDB, and legacy ARP walks took about 9.8 s, 9.8 s, and 12.2 s; scalar GET loop had four SLA violations, max 9.034 s.
296
+ - `max_repetitions=1`, no pause: failed. Bridge FDB, legacy ARP, and Q-BRIDGE FDB walks took about 9.9 s, 24.8 s, and 9.8 s; scalar GET loop had six SLA violations, max 9.323 s.
297
+ - `max_repetitions=1`, 10 ms pause between GETBULK requests: failed. Interface, bridge FDB, legacy ARP, Q-BRIDGE FDB, and STP roots all had multi-second walk times; scalar GET loop had twelve SLA violations, max 9.472 s.
298
+ - `max_repetitions=25`, 250 ms pause between GETBULK requests: failed. The walk stretched to about 85.8 s and scalar GET loop had eleven SLA violations, max 9.337 s.
299
+ - `max_repetitions=25`, topology timeout 800 ms, zero topology retries: failed. The topology client timed out many roots quickly, but the scalar GET loop still had a max latency around 8.7 s. This was reproduced in a separate edge run, so it was not only contamination from the previous long test.
300
+ - `max_repetitions=100`: not viable. The GoSNMP harness received zero PDUs for every topology root and therefore collected no topology data; prior Net-SNMP direct tests also showed this direction failing quickly.
301
+- External reduced-root test:
302
+ - Removing the obvious bridge/FDB/ARP roots was still not safe. A reduced set including interface, IP, STP, VLAN, and LLDP roots failed twice: one run had an `ifTable` walk around 3.1 s and scalar GET max 3.67 s; the next had `ifTable` and STP walks around 9.5 s each and scalar GET max 9.212 s.
303
+- Updated conclusion after external homework: no tested transport policy that still walks the topology roots preserves the 1-second scalar GET SLA on this RouterOS device. Smaller bulks, larger bulks, per-bulk pauses, and short topology-side timeouts all fail. The only policies that can preserve per-second metrics are policies that avoid issuing unproven topology walks while 1-second collection is required, or policies that can learn and quarantine unsafe walks after causing initial damage.
304
+- Searched current public MikroTik/RouterOS evidence for known SNMP issues:
305
+ - Official MikroTik RouterOS SNMP documentation says SNMP gathers data from other RouterOS services, can log `timeout while waiting for program` / `SNMP did not get OID data within expected time`, may deny requests for that service for a while, and says slow/busy services should often be skipped by monitoring tools.
306
+ - Official MikroTik RouterOS SNMP documentation lists IF-MIB, IP-MIB, and BRIDGE-MIB as RouterOS-supported MIBs, matching the class of topology roots involved in this investigation.
307
+ - Public MikroTik forum reports include RouterOS 7.x SNMP polling times of 200-220 seconds on a CCR2004-16G-2S+ with RouterOS 7.14.1, intermittent SNMP refusal on RouterOS 7.14.2 with low CPU, and very slow SNMP walks on MikroTik enterprise interface-stat OIDs.
308
+ - Checkmk has an old compatibility fix for a broken MikroTik RouterOS bulk-walk implementation in RouterOS v6.22 where consecutive duplicate OIDs could be returned.
309
+ - RouterOS 7.22 public changelog evidence mentions an SNMP fix where bulk walk might skip the first OID. This is not the same as the observed stall, but it shows RouterOS bulk-walk behavior has had recent fixes.
310
+ - No public source found an exact named MikroTik bug for "BRIDGE-MIB/FDB/IF-MIB GETBULK stalls concurrent scalar SNMP GETs for 8-10 seconds." The evidence supports a known RouterOS SNMP slow/busy-service class, not a public exact-match defect ID.
311
+- User asked to test whether the behavior is internal RouterOS SNMP rate limiting rather than slow OID processing. Planned external read-only tests with long pauses between topology GETBULK requests and between topology roots. Prediction: if simple rate limiting is the cause, long request spacing should remove the 8-10 second stalls and scalar GET SLA violations.
312
+- Rate-limit hypothesis quick check:
313
+ - Stopped local Netdata, waited 10 seconds, then ran a single read-only walk of one previously slow bridge FDB topology root with `max_repetitions=25`. Result: 192 rows, exit 0, duration 9.528 seconds.
314
+ - Stopped local Netdata again, waited 10 seconds, then ran one single GETBULK request against the same root with `max_repetitions=25`, not a full walk. Result: 25 rows, exit 0, duration 9.489 seconds.
315
+ - Conclusion: the observed delay is not explained by simple recent-request burst rate limiting. A single request after an idle gap can still be delayed by about 9.5 seconds. This does not rule out RouterOS internal serialization, internal table refresh, or per-OID/service throttling, but it rules out "spread previous topology calls and the first expensive request becomes fast" for this root.
316
+- User concluded this is inherent to RouterOS and asked to close the SOW. No source code, router configuration, or durable operational configuration was changed as part of closing.
317
+
318
+## Validation
319
+
320
+Acceptance criteria evidence:
321
+
322
+- Root cause evidence complete.
323
+- Netdata can sustain per-second metrics when topology does not issue unsafe RouterOS topology walks against the same endpoint: after topology cadence was moved away, tier-0 120-second / 120-point queries for the affected interface traffic and operational-status contexts were continuous, and Netdata logs had no affected-job skip/resume entries in the checked window.
324
+- Full topology walking while preserving 1-second metrics is not achievable for this device with the tested request policies: `max_repetitions` values 1, 2, 5, 10, 25, and 50 all produced scalar GET SLA violations during topology walks; pauses and short topology-side timeouts also failed.
325
+- A single GETBULK request after a 10-second idle gap still took about 9.489 seconds, proving the failure is not simple burst rate limiting.
326
+- The user accepted the conclusion that this is inherent RouterOS behavior and requested SOW closure.
327
+
328
+Tests or equivalent validation:
329
+
330
+- Diagnostic validation completed:
331
+ - tier-0 120-second / 120-point queries showed repeated all-null runs for interface traffic/status;
332
+ - Netdata namespace logs showed skip/resume messages for the affected SNMP job;
333
+ - SNMP error stats stayed at zero in the checked window;
334
+ - RouterOS resource/profile data did not show router overload.
335
+- External Net-SNMP and GoSNMP timing validation completed with local Netdata stopped:
336
+ - baseline scalar GET loop had zero errors and zero 1-second SLA violations;
337
+ - topology-like GETBULK walks reproduced 8-10 second stalls and scalar GET SLA violations;
338
+ - low, default, and higher `max_repetitions` values were tested and failed;
339
+ - request pacing and short topology-side timeouts were tested and failed;
340
+ - one single GETBULK request after an idle gap still took about 9.5 seconds.
341
+
342
+Real-use evidence:
343
+
344
+- Local Netdata MCP and local Netdata `systemd-journal` function were used.
345
+- Direct read-only SNMP commands were run with local Netdata stopped to isolate device behavior without concurrent Netdata polling. Local Netdata was restarted immediately after the test.
346
+- Temporary instrumentation and candidate code changes were removed before closure; source code remained unchanged at close.
347
+
348
+Reviewer findings:
349
+
350
+- No implementation was shipped, so no code review was required.
351
+- External references were checked for context: MikroTik RouterOS SNMP documentation, public MikroTik forum reports, Checkmk RouterOS SNMP bulk-walk compatibility note, Prometheus `snmp_exporter`, and Datadog Agent SNMP configuration defaults.
352
+
353
+Same-failure scan:
354
+
355
+- Same-failure class was checked through public MikroTik reports and open-source SNMP collector references.
356
+- Public evidence supports RouterOS SNMP slow/busy-service and bulk-walk issue classes, but no exact public defect ID was found for the specific bridge/FDB/interface 8-10 second stall observed here.
357
+
358
+Sensitive data gate:
359
+
360
+- Raw SNMP secrets, trap communities, exact private endpoints, contact/location strings, claim IDs, and SSH details were not written to this SOW.
361
+- Private endpoint evidence is redacted as `[PRIVATE_ENDPOINT]`.
362
+
363
+Artifact maintenance gate:
364
+
365
+- AGENTS.md: no update needed. This SOW did not change repository workflow, responsibility boundaries, or project-wide guardrails.
366
+- Runtime project skills: no update needed. The work produced a device-specific diagnosis, not a new reusable collector-authoring rule.
367
+- Specs: no update needed. No Netdata product behavior or public collector contract changed in this SOW.
368
+- End-user/operator docs: no update needed in this SOW. The user explicitly closed the investigation as inherent RouterOS behavior without requesting a Netdata operator guidance change.
369
+- End-user/operator skills: no update needed. No public/operator AI skill behavior changed.
370
+- SOW lifecycle: status updated to `completed`; file will be moved from `.agents/sow/current/` to `.agents/sow/done/`.
371
+
372
+Specs update:
373
+
374
+- No spec update. There was no shipped behavior change and no new Netdata contract selected.
375
+
376
+Project skills update:
377
+
378
+- No project skill update. The investigation used existing collector and SNMP guidance; no reusable workflow rule changed.
379
+
380
+End-user/operator docs update:
381
+
382
+- No end-user/operator docs update. The result is a closed local diagnosis; documentation of RouterOS-specific topology limitations is outside this SOW and was not requested.
383
+
384
+End-user/operator skills update:
385
+
386
+- No end-user/operator skill update. No public skill behavior changed.
387
+
388
+Lessons:
389
+
390
+- A single expensive RouterOS topology GETBULK request can take about 9.5 seconds after an idle gap, so request pacing is not sufficient to guarantee 1-second SNMP metrics.
391
+- `max_repetitions` tuning is device/OID/order dependent on RouterOS and cannot be treated as a generic fix for this failure class.
392
+- External request-policy testing should precede Netdata implementation when the suspected failure could be inherent to the SNMP agent.
393
+
394
+Follow-up mapping:
395
+
396
+- No follow-up SOW is required by the user. The SOW is closed as diagnosis-complete.
397
+
398
+## Outcome
399
+
400
+Completed as diagnosis-only.
401
+
402
+Final conclusion:
403
+
404
+- The per-second metric gaps were caused by RouterOS SNMP behavior when topology-class OIDs are queried.
405
+- Netdata regular SNMP collection can sustain per-second metrics when those unsafe topology walks are not issued.
406
+- Topology collection is a separate Netdata loop and client, but it still targets the same RouterOS SNMP agent.
407
+- A single bridge/FDB topology GETBULK request after a 10-second idle gap took about 9.489 seconds, so the failure is not simple request burst rate limiting.
408
+- No tested Netdata-side transport policy preserved the 1-second SLA while still walking the tested topology roots.
409
+- No Netdata implementation was made or required in this SOW.
410
+
411
+## Lessons Extracted
412
+
413
+- Verify device-agent behavior externally before implementing collector changes when live evidence suggests an upstream SNMP agent stall.
414
+- Do not assume smaller `max_repetitions` improves RouterOS behavior; for this device/OID class, values from 1 through 50 all failed in at least one tested scenario.
415
+- A topology loop can be architecturally separate from metrics collection and still interfere through the monitored device's own SNMP agent.
416
+
417
+## Followup
418
+
419
+None.
420
+
421
+## Regression Log
422
+
423
+None yet.
424
+
425
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0009-20260502-project-writing-collectors-skill.md
renamed
+3
-3
@@ -1,4 +1,4 @@
1
-# SOW-0001 - project-writing-collectors skill
1
+# SOW-0009 - project-writing-collectors skill
2
3
## Status
4
@@ -43,7 +43,7 @@ Facts:
43
- ~30 plugins exist across C, Go, Rust, Python, Bash, eBPF.
44
- 12 recurring bad-practice patterns identified with file:line evidence (see Analysis).
45
- Existing project skills under `.agents/skills/` range from 162 (graphql-audit) to 508 (pr-reviews) lines; average ~330.
46
-- SOW directories were empty at SOW-creation time; this is SOW-0001 in numbering despite AGENTS.md text mentioning SOW-0003.
46
+- SOW directories were empty at SOW-creation time; this is SOW-0009 in numbering despite AGENTS.md text mentioning SOW-0003.
47
48
Inferences:
49
@@ -370,7 +370,7 @@ Coverage:
370
371
6. **Preserve first drafts when a substantial rewrite follows.** Committing the first draft separately let the rewrite stand on its own as a reviewable change (+395 / -192) and made the structural shift visible in history. Future SKILL-level rewrites should follow the same pattern.
372
373
-7. **The SOW close should land with the work as one commit.** This SOW shipped its work in two commits but was left in `current/` until the user noticed. The AGENTS.md rule ("commit the work, artifact updates, SOW status change, and SOW move together as one commit") exists to prevent exactly this. Future SOWs must close in the same commit as the final piece of work, unless the user explicitly approves a split (as happened here for SOW-0001).
373
+7. **The SOW close should land with the work as one commit.** This SOW shipped its work in two commits but was left in `current/` until the user noticed. The AGENTS.md rule ("commit the work, artifact updates, SOW status change, and SOW move together as one commit") exists to prevent exactly this. Future SOWs must close in the same commit as the final piece of work, unless the user explicitly approves a split (as happened here for SOW-0009).
374
375
## Followup
376
.agents/sow/done/SOW-0010-20260503-netdata-query-skills-infrastructure.md
new
+1036
@@ -0,0 +1,1036 @@
1
+# SOW-0010 - Netdata query skills infrastructure
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: rescoped 2026-05-03 evening (second expansion). The two public skills must mirror each other in structure and cover every queryable Netdata surface (metrics, logs, topology, flows, alerts, dyncfg, functions, nodes, plus Cloud-only rooms/members/feed and Agent-only streaming). Scripts must be **token-safe** -- the assistant must never see `NETDATA_CLOUD_TOKEN`, per-agent bearer values, or claim ids on stdout. A `how-tos/` subdir with `INDEX.md` ships in each skill; assistants extend it whenever they perform analysis not already covered by a how-to. The verification harness (Sonnet test runner with grading rubric) moves to follow-up SOW-0006 per user direction "evaluation does not need to be done now". Decisions 1, 2, 3 already resolved.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Build the foundational AI-skill infrastructure that lets human and AI
14
+operators query Netdata Cloud and Netdata Agents in a uniform,
15
+documented way. This SOW is **infrastructure-only** -- it ships no
16
+business analysis, no triage scripts, no fleet-data fetches. Its
17
+deliverables are reusable pieces that downstream SOWs (agent-events,
18
+documentation pipeline, etc.) consume.
19
+
20
+Three deliverables:
21
+
22
+1. **Skill format convention.** Every public AI skill under
23
+ `docs/netdata-ai/skills/` must follow the `<skill-name>/SKILL.md`
24
+ directory shape with optional `<doc>.md` supporting docs and
25
+ `scripts/` subdir, exactly like the private operational skills
26
+ under `.agents/skills/`. Each public skill must be reachable from
27
+ `.agents/skills/<skill-name>` via a **relative symlink** so local
28
+ AI assistants reading from `.agents/skills/` see the same skill.
29
+ The convention is documented in `AGENTS.md` so future skills
30
+ follow the same shape.
31
+
32
+2. **`query-netdata-cloud/` skill (refactor + expand).** The current
33
+ single file `docs/netdata-ai/skills/query-netdata-cloud-metrics.md`
34
+ is migrated to `docs/netdata-ai/skills/query-netdata-cloud/
35
+ SKILL.md` and expanded with separate supporting docs:
36
+ - `query-metrics.md` -- the existing metrics-query content,
37
+ trimmed to its specific surface
38
+ - `query-logs.md` -- how to call the `systemd-journal` Function
39
+ (and any other log-related Function) via Cloud
40
+ - `query-alerts.md` -- how to query alerts and alert
41
+ transitions via Cloud
42
+ - `query-functions.md` -- generic Cloud-proxied Function
43
+ invocation: what URL, what body, what response
44
+ The top-level `SKILL.md` covers what is common across all four:
45
+ auth (`NETDATA_CLOUD_TOKEN`), space/room/node resolution,
46
+ pagination, error handling, dry-run discipline, links to each
47
+ supporting doc.
48
+
49
+3. **`query-netdata-agents/` skill (new).** Sibling skill that
50
+ covers querying Netdata Agents directly (Parents and Children).
51
+ Delivers SKILL.md + supporting docs + a `scripts/` library.
52
+ The scripts library must:
53
+ - Read `NETDATA_CLOUD_TOKEN` from `.env`.
54
+ - Probe the agent URL and detect whether it is bearer-protected
55
+ (HTTP 401, redirect to Cloud SSO, or any other documented
56
+ bearer-required signal).
57
+ - When a bearer is required, mint one for that specific node
58
+ using the user's cloud token and the documented Cloud endpoint,
59
+ cache the bearer in `.local/audits/query-netdata-agents/
60
+ bearers/<node-uuid>.json` with its expiration, and refuse to
61
+ log the token value.
62
+ - Transparently refresh the bearer when expired (or one minute
63
+ before expiry, to avoid races during long batch fetches).
64
+ - Expose a single `agents_resolve_bearer <node>` helper that
65
+ downstream skills call to obtain a current bearer for a node.
66
+ - Expose a single `agents_call_function` helper that takes
67
+ `{node, function, body}` and routes the request through the
68
+ correct transport (Cloud-proxied vs direct-agent), retrying
69
+ once across transports on transient failure.
70
+ - Expose a `agents_netdata_prefix` helper that **autodetects
71
+ the local Netdata install prefix** at runtime (probe order:
72
+ empty / `/opt/netdata` / `/usr/local/netdata`; pick the
73
+ first whose `<prefix>/var/lib/netdata` or
74
+ `<prefix>/etc/netdata` exists). Used to locate the local
75
+ `bearer_tokens/` directory for the local-fallback bearer
76
+ path. NOT an env knob; the prefix is a discovered fact.
77
+
78
+This SOW does NOT touch the legacy private operational skills
79
+(`coverity-audit/`, `sonarqube-audit/`, `graphql-audit/`,
80
+`pr-reviews/`) -- they keep their current location under
81
+`.agents/skills/` and their current shape. They are intentionally
82
+private and have no `docs/netdata-ai/skills/` counterpart.
83
+
84
+### User Request
85
+
86
+> Original (preserved for context):
87
+> "Create a skill for querying and fetching and analyzing agent events
88
+> (status file submissions) from the ingestion server."
89
+>
90
+> Stage-1 follow-up:
91
+> "agent-events are stored in journal files... we have 2 options:
92
+> 1. query logs via cloud (docs/netdata-ai/skills/ has a relevant
93
+> skill, although not for logs, but we could enrich it)
94
+> 2. query the agent directly, but this time we need a mechanism to
95
+> get an agent bearer token by using the cloud api token, and
96
+> then query the agent API directly. Ideally we should support
97
+> all these methods."
98
+>
99
+> Scope expansion:
100
+> "1. skills in docs/netdata-ai/skills/ should be formatted as
101
+> normal skills {skill-name}/SKILL.md with skill frontmatter and
102
+> potentially supporting documentation and scripts when necessary,
103
+> and they should be linked to .agents/skills/ with a relative
104
+> link, so that they are accessible by local agents too.
105
+> 2. query-netdata-cloud-metrics skill, should be renamed to
106
+> query-netdata-cloud/ and have SKILL.md with the common
107
+> information about querying netdata cloud and then supporting docs
108
+> query-metrics.md, query-logs.md, query-alerts.md,
109
+> query-functions.md, etc as necessary, which should be referenced
110
+> from SKILL.md.
111
+> 3. A new skill query-netdata-agents/ should be added, explaining
112
+> how to query netdata agents and parents and support the same
113
+> supporting material documentation. This should also live in
114
+> docs/netdata-ai/skills/ and be linked (relative) to
115
+> .agents/skills/.
116
+> 4. the query-netdata-agents skill should support querying agents
117
+> via netdata cloud sso, so it should provide the tooling to fetch
118
+> and cache and reuse and transparectly refresh agent bearer
119
+> tokens, starting from an netdata cloud api token. The supported
120
+> scripts should automatically detect the agent to query is bearer
121
+> protected and automatically work around this to fetch netdata-
122
+> cloud sso."
123
+>
124
+> Split decision (this run): "Go with 4 SOWs."
125
+
126
+### Assistant Understanding
127
+
128
+Facts:
129
+
130
+- The .env + skill pattern is well-established and identical across
131
+ the four legacy private skills. The same pattern is the basis for
132
+ the scripts library shipped under `query-netdata-agents/`.
133
+- The agent-side `systemd-journal` Function and the agent-side
134
+ `bearer_get_token` Function exist in this repo; their shapes are
135
+ documented in stage-1 analysis.
136
+- Only one public AI skill exists today:
137
+ `docs/netdata-ai/skills/query-netdata-cloud-metrics.md`. It is a
138
+ flat file, not a directory. Its content is the template for the
139
+ refactor.
140
+- Per the user's rule, `.env` values stay in `.env`; only keys
141
+ appear in this SOW or in scripts.
142
+- The user added four `AGENT_EVENTS_*` keys to `.env`. Those keys
143
+ are consumed by SOW-0003, not by this SOW. They are listed here
144
+ only to confirm naming convention: `AGENT_EVENTS_NC_SPACE`,
145
+ `AGENT_EVENTS_HOSTNAME`, `AGENT_EVENTS_MACHINE_GUID`,
146
+ `AGENT_EVENTS_NODE_ID`.
147
+
148
+Inferences:
149
+
150
+- The Cloud REST shape that proxies a Function call to a node by
151
+ uuid must already exist (otherwise no team member could query
152
+ Cloud-only). The exact path is not in this open-source repo and
153
+ must come from either user knowledge or the live Swagger at
154
+ `${NETDATA_CLOUD_HOSTNAME}/api/docs/` (key in `.env`).
155
+- A documented Cloud endpoint that mints an agent bearer from a
156
+ cloud token must exist if the user wants the auto-refresh flow.
157
+ Candidate names a Swagger fetch could check (paths under the
158
+ Cloud API base): anything under `/api/v3/spaces/.../nodes/<uuid>/
159
+ ...token`, `/api/v.../auth/...`, `/api/v.../bearer/...`. If no
160
+ such endpoint is documented, the agents skill must degrade to
161
+ "local-only" transport-(b) (as described in stage-1 decision 1B).
162
+
163
+Unknowns (require user input or live-Swagger lookup):
164
+
165
+- Cloud REST function-call endpoint shape (URL pattern, request
166
+ body, response shape).
167
+- Cloud REST agent-bearer mint endpoint shape (or confirmation it
168
+ does not exist).
169
+- Whether the existing Cloud-metrics doc references the correct
170
+ current Cloud Swagger version. Last-revised date in
171
+ `query-netdata-cloud-metrics.md` should be cross-checked against
172
+ the live API at `${NETDATA_CLOUD_HOSTNAME}/api/docs/`.
173
+- Symlink direction confirmation. User wrote: skills live in
174
+ `docs/netdata-ai/skills/` and are symlinked from
175
+ `.agents/skills/`. Reading: `docs/netdata-ai/skills/` is
176
+ canonical; `.agents/skills/` holds relative symlinks pointing
177
+ there. Confirmed during write-out.
178
+
179
+### Acceptance Criteria
180
+
181
+**(Scope expanded 2026-05-03 evening per user direction. The two
182
+public skills must mirror each other in structure, cover every
183
+queryable surface a Netdata operator/AI assistant cares about,
184
+keep all secrets out of the assistant's view, ship a how-to
185
+extraction system, and pass an automated verification harness.)**
186
+
187
+#### Symmetric file structure
188
+
189
+Both `docs/netdata-ai/skills/query-netdata-cloud/` and
190
+`docs/netdata-ai/skills/query-netdata-agents/` ship the same set
191
+of per-domain guides for the surfaces shared by both transports:
192
+
193
+| Per-domain guide | Cloud | Agent |
194
+|---|---|---|
195
+| SKILL.md | required | required |
196
+| query-metrics.md | required | required |
197
+| query-logs.md | required | required |
198
+| query-topology.md | required | required |
199
+| query-flows.md | required | required |
200
+| query-alerts.md | required | required |
201
+| query-dyncfg.md | required | required |
202
+| query-functions.md | required | required |
203
+| query-nodes.md | required | required |
204
+
205
+In addition, the Cloud skill ships three guides that have no
206
+agent equivalent (these surfaces only exist on the Cloud side):
207
+
208
+- `query-rooms.md`
209
+- `query-members.md`
210
+- `query-feed.md`
211
+
212
+And the Agent skill ships one guide with no Cloud equivalent
213
+(only meaningful agent-side):
214
+
215
+- `query-streaming.md`
216
+
217
+Each per-domain guide must:
218
+
219
+- Open with a one-paragraph summary of the surface.
220
+- Document the v3 endpoint(s) (use v2/v1 only when v3 is missing).
221
+- Show one runnable example using only the documented script
222
+ wrappers (assistant must NOT see tokens; see security section
223
+ below).
224
+- Cross-link the canonical reference docs in
225
+ `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`,
226
+ `<repo>/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`,
227
+ `<repo>/src/plugins.d/FUNCTION_UI_SCHEMA.json`,
228
+ `<repo>/src/plugins.d/DYNCFG.md`,
229
+ `<repo>/src/daemon/dyncfg/README.md` where relevant.
230
+
231
+Both `SKILL.md` files index every per-domain guide AND the
232
+canonical references AND the how-tos directory.
233
+
234
+#### Token-safety architecture (HARD requirement)
235
+
236
+The assistant invoking these skills must NEVER see:
237
+
238
+- `NETDATA_CLOUD_TOKEN`
239
+- per-agent bearer values (the 36-char UUID returned by
240
+ `bearer_get_token`)
241
+- claim_id values (treat as semi-sensitive identifiers)
242
+
243
+To enforce this, the scripts library exposes ONLY high-level
244
+wrappers. Helpers that previously returned a bearer to stdout
245
+(e.g. `agents_resolve_bearer`) MUST be marked internal (named
246
+with a leading underscore, e.g. `_agents_resolve_bearer`) and
247
+their output redirected to in-process variables only -- never
248
+emitted to stdout where the assistant could capture them.
249
+
250
+Public wrappers exposed to the assistant:
251
+
252
+- `agents_query_cloud <method> <path> [<body-json>]`
253
+ Cloud-side. Reads `NETDATA_CLOUD_TOKEN` from `.env` internally,
254
+ adds `Authorization: Bearer ...` header, runs curl, prints
255
+ ONLY the response body. stderr shows the curl invocation with
256
+ `<CLOUD_TOKEN>` masked.
257
+- `agents_query_agent <node> <method> <path> [<body-json>]`
258
+ Direct-agent-side. Resolves bearer internally (cache or mint
259
+ via cloud), routes through `${AGENT_EVENTS_HOSTNAME}` (or any
260
+ reachable host), adds `X-Netdata-Auth: Bearer ...` header
261
+ internally, prints ONLY the response body. stderr shows the
262
+ curl invocation with `<AGENT_BEARER>` masked.
263
+- Per-surface convenience wrappers (one per query-*.md guide)
264
+ that take typed arguments and forward to the above (e.g.
265
+ `agents_query_function <node> <function-name> <body-json>`).
266
+
267
+**No public wrapper may print a token, bearer, or claim_id to
268
+stdout under any circumstance, including error paths.** This is
269
+verified by a pre-commit unit test that drives every public
270
+wrapper with a fake token and asserts the token bytes never
271
+appear in captured stdout.
272
+
273
+#### How-tos directory (live, indexed)
274
+
275
+Each skill ships a `how-tos/` subdirectory:
276
+
277
+- `<skill>/how-tos/INDEX.md` -- one-line index of every how-to,
278
+ ordered by topic. Indexed from `SKILL.md`.
279
+- `<skill>/how-tos/<slug>.md` -- one file per how-to. Each
280
+ documents: the question being answered, the steps taken,
281
+ which wrappers were called, the expected output shape, and
282
+ any gotchas.
283
+
284
+Rule baked into both `SKILL.md` files (and into AGENTS.md):
285
+**when an assistant has to perform analysis to answer a question
286
+that is not already covered by a how-to, the assistant must add
287
+a new how-to before completing the task.** The new how-to gets
288
+committed in the same PR as the analysis.
289
+
290
+#### Verification harness (DEFERRED to SOW-0006)
291
+
292
+Per user direction 2026-05-03: "the evaluation does not need to
293
+be done now". The full Sonnet-driven verification harness (test
294
+runner, grading rubric, automated how-to extraction prompts)
295
+moves to its own SOW (`SOW-0006-20260503-skill-verification-
296
+harness.md`, in `pending/`). It will validate this SOW's
297
+deliverables and any future skill, so it has independent value.
298
+
299
+This SOW seeds the inputs the harness will consume:
300
+
301
+- `<skill>/verify/questions.md` -- the seed list of validation
302
+ questions. Both skills ship this file. Includes (at minimum)
303
+ the user-supplied questions: hardware specs of a known node;
304
+ OS; parent-or-child status; list of streamed-children if
305
+ parent; vnodes; failed jobs; whether nvidia DCGM is monitored
306
+ and at what frequency; PID with biggest memory consumption +
307
+ dashboard category; last agent status-file log; plus 6+
308
+ further questions covering alerts, logs, topology, flows,
309
+ dyncfg, members, rooms, feed.
310
+
311
+The harness implementation (run.sh, grader rubric, score
312
+collation) is SOW-0006's deliverable, not SOW-0010's.
313
+
314
+#### Pre-existing acceptance criteria (carry-over)
315
+
316
+- Both relative symlinks at `.agents/skills/<name>` resolve to
317
+ the corresponding `docs/netdata-ai/skills/<name>` directory.
318
+- `AGENTS.md` "Project Skills Index" lists both public skills
319
+ with their one-line triggers and the symlink path.
320
+- `AGENTS.md` documents the public-skill convention.
321
+- Sensitive-data gate: every committed file passes the
322
+ pre-commit grep from
323
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md`.
324
+- All v3 agent paths preferred over v2/v1 (v2/v1 only as
325
+ fallback for older agents).
326
+
327
+## Analysis
328
+
329
+Sources checked (stage-1, carried forward):
330
+
331
+- `<repo>/.env` (per-user, gitignored) -- confirmed shape used by
332
+ the existing skills and the new `AGENT_EVENTS_*` keys for SOW-3.
333
+- `<repo>/AGENTS.md` (= `CLAUDE.md`) -- canonical documentation of
334
+ the `.local/` audit directory convention and the `.env`
335
+ convention.
336
+- `<repo>/.agents/skills/coverity-audit/SKILL.md` and
337
+ `<repo>/.agents/skills/coverity-audit/scripts/_lib.sh`.
338
+- `<repo>/.agents/skills/sonarqube-audit/scripts/_lib.sh`.
339
+- `<repo>/.agents/skills/pr-reviews/scripts/_lib.sh`.
340
+- `<repo>/docs/netdata-ai/skills/query-netdata-cloud-metrics.md`
341
+ (the existing template).
342
+- `<repo>/src/collectors/systemd-journal.plugin/systemd-main.c`,
343
+ `systemd-journal.c`, `systemd-internals.h`, `logs_query_status.h`
344
+ (Function shape).
345
+- `<repo>/src/web/api/functions/function-bearer_get_token.c`
346
+ (agent-side bearer-mint Function and its Cloud-source gate).
347
+
348
+Current state -- skill format convention:
349
+
350
+- The four legacy private skills already use the directory shape
351
+ (`<repo>/.agents/skills/<name>/SKILL.md` + `scripts/`).
352
+- The one public skill is a flat file
353
+ (`docs/netdata-ai/skills/query-netdata-cloud-metrics.md`). The
354
+ refactor brings it into line with the directory shape.
355
+- Symlink direction: `docs/netdata-ai/skills/<name>/` is canonical;
356
+ `.agents/skills/<name>` becomes a relative symlink pointing at
357
+ `../../docs/netdata-ai/skills/<name>`. This satisfies the user's
358
+ rule "linked (relative) to .agents/skills/".
359
+
360
+Current state -- skill helper library shape (mirrored across all
361
+four legacy skills):
362
+
363
+- `set -euo pipefail` at the top.
364
+- Color variables defined with `$'\033[...]'`.
365
+- `<prefix>_repo_root()` -- via `git -C "$(dirname
366
+ "${BASH_SOURCE[0]}")" rev-parse --show-toplevel`.
367
+- `<prefix>_load_env()` -- locates `<repo>/.env`, sources via
368
+ `set -a; source; set +a`, validates required vars with
369
+ `: "${VAR:?msg}"`, applies defaults with `: "${VAR:=default}"`.
370
+- `<prefix>_audit_dir()` -- creates
371
+ `<repo>/.local/audits/<topic>/` on demand. Topic name strips any
372
+ `-audit` suffix from the skill name (per AGENTS.md).
373
+- `<prefix>_run` and `<prefix>_run_read` -- print masked curl to
374
+ stderr for transparency, mask the token in argv.
375
+- Skill-specific validators (numeric IDs, ASCII-only, etc.).
376
+
377
+Current state -- agent-side primitives (relevant to the agents
378
+skill):
379
+
380
+- `systemd-journal` Function name is registered at
381
+ `src/collectors/systemd-journal.plugin/systemd-main.c:79` via
382
+ `rrd_function_add(... ND_SD_JOURNAL_FUNCTION_NAME ...)`; the
383
+ literal name is `systemd-journal`
384
+ (`systemd-journal.c:13`).
385
+- POST body keys (from `logs_query_status.h:8-24`): `help`,
386
+ `after`, `before`, `anchor`, `last`, `query`, `facets`,
387
+ `histogram`, `direction`, `if_modified_since`, `data_only`,
388
+ `__logs_sources`, `info`, `slice`, `delta`, `tail`, `sampling`.
389
+- Response top-level: `facets`, `histogram`, `rows`, `search`,
390
+ `info`.
391
+- `bearer_get_token` Function is gated by
392
+ `user_auth_source_is_cloud(source)`
393
+ (`src/web/api/functions/function-bearer_get_token.c:30`), so
394
+ external HTTP clients cannot call it directly. It is invoked
395
+ over ACLK by Cloud, on behalf of a Cloud-authenticated user.
396
+ Per-agent bearer files live at
397
+ `<netdata-prefix>/var/lib/netdata/bearer_tokens/<token-uuid>.json` for ~24h.
398
+
399
+Risks:
400
+
401
+- Symlink portability: relative symlinks survive `git clone` and
402
+ most worktree operations on Linux/macOS. Windows / WSL with NTFS
403
+ may not. Acceptable risk -- the project is primarily
404
+ Linux/macOS, and the symlinked skill is also reachable directly
405
+ via its canonical path.
406
+- Bearer-mint endpoint may be undocumented: if the live Swagger
407
+ has no agent-bearer mint, the agents skill must degrade
408
+ gracefully to "local-only transport-b" (i.e. read existing
409
+ `<netdata-prefix>/var/lib/netdata/bearer_tokens/*.json` for the user's own
410
+ workstation only). Stage-1 decision 1B was the recommended
411
+ fallback.
412
+- AGENTS.md churn: changing the project skills index requires
413
+ care so the legacy private skills are not accidentally moved or
414
+ renamed.
415
+- Breaking downstream readers: the existing flat file
416
+ `docs/netdata-ai/skills/query-netdata-cloud-metrics.md` may be
417
+ linked from external docs. The refactor must keep a redirect
418
+ stub (a 1-line file pointing at
419
+ `query-netdata-cloud/query-metrics.md`) to avoid breaking
420
+ inbound links.
421
+
422
+## Pre-Implementation Gate
423
+
424
+Status: needs-user-decision
425
+
426
+Problem / root-cause model:
427
+
428
+- The current public skill shelf has a single skill in a flat-file
429
+ shape. As soon as we want a second public skill, we either keep
430
+ using flat files (leading to monoliths) or unify around the
431
+ directory shape now. The user has chosen the directory shape;
432
+ this SOW does the unification and adds the second skill.
433
+- Downstream SOWs (agent-events, learn-site-structure,
434
+ integrations-lifecycle) need both `query-netdata-cloud/query-functions.md`
435
+ and the bearer-mint scripts in `query-netdata-agents/scripts/`.
436
+ Without this SOW, every downstream SOW would re-implement the
437
+ same pieces.
438
+
439
+Evidence reviewed:
440
+
441
+- See "Sources checked" and "Current state" above. No further
442
+ evidence is needed for the format-normalization piece. The
443
+ bearer-mint piece is blocked on the live Cloud Swagger.
444
+
445
+Affected contracts and surfaces:
446
+
447
+- New: `<repo>/docs/netdata-ai/skills/query-netdata-cloud/`
448
+ (SKILL.md + 4 supporting docs).
449
+- New: `<repo>/docs/netdata-ai/skills/query-netdata-agents/`
450
+ (SKILL.md + supporting docs + scripts/).
451
+- New: `<repo>/.agents/skills/query-netdata-cloud` (relative
452
+ symlink) and `<repo>/.agents/skills/query-netdata-agents`
453
+ (relative symlink).
454
+- Stub: `<repo>/docs/netdata-ai/skills/query-netdata-cloud-
455
+ metrics.md` becomes a 1-line redirect to the new location to
456
+ preserve existing inbound links.
457
+- New: `<repo>/.local/audits/query-netdata-agents/` writes
458
+ (gitignored, bearer cache + acceptance-test outputs).
459
+- Augmented: `<repo>/AGENTS.md` -- adds the public-skill
460
+ convention paragraph and the index entries.
461
+- No existing surface is broken; the legacy skill keeps a
462
+ redirect.
463
+
464
+Existing patterns to reuse:
465
+
466
+- `_lib.sh` shape from `coverity-audit/scripts/_lib.sh`.
467
+- Pagination idiom from `sonarqube-audit/scripts/_lib.sh::sq_paginate`.
468
+- Audit-dir / `.local/` convention from AGENTS.md.
469
+- Token-masking idiom from `sonarqube-audit/scripts/_lib.sh::sq_run`.
470
+- The single existing public skill
471
+ `query-netdata-cloud-metrics.md` is the content seed for the
472
+ new `query-metrics.md`.
473
+
474
+Risk and blast radius:
475
+
476
+- Low for format normalization (additive, with redirect stub).
477
+- Medium for the bearer-mint scripts -- they handle credentials
478
+ and a bug here could leak bearers to logs or `.local/`. Mitigation:
479
+ a single `agents_run` wrapper that forces token masking, plus a
480
+ pre-commit grep that fails if any committed file contains
481
+ `bearer_tokens` or `nd_bearer` blobs, plus a unit-test harness
482
+ that runs the helpers under `set -x` and asserts no token bytes
483
+ appear on stderr.
484
+
485
+Sensitive data handling plan:
486
+
487
+- This SOW (and every committed artifact it produces) follows the
488
+ spec at `<repo>/.agents/sow/specs/sensitive-data-discipline.md`.
489
+ No literal IPs, hostnames, UUID-shaped IDs, tokens, absolute
490
+ install/user paths, usernames, tenant names, or secrets in any
491
+ committed file. Every reference to such a value is via an
492
+ env-key placeholder (`${KEY_NAME}`) defined in `.env`.
493
+- `.env` is the only place credential and identity VALUES live.
494
+- Bearer cache lives under
495
+ `<repo>/.local/audits/query-netdata-agents/bearers/`;
496
+ gitignored. File mode 0600.
497
+- All log lines that include curl invocations route through
498
+ `agents_run`/`agents_run_read` which masks the cloud token and
499
+ any bearer matched by a regex.
500
+- A one-shot redaction self-test runs as part of stage 2f
501
+ validation: calls `agents_run` with a fake token, checks the
502
+ emitted stderr contains no token bytes.
503
+- Pre-commit verification grep (from the spec) runs on every
504
+ staged change.
505
+
506
+Implementation plan:
507
+
508
+1. **Stage 1 -- DONE**: investigation captured in this SOW.
509
+2. **Stage 2a -- DONE**: decisions 1, 2, 3 resolved. Cloud
510
+ bearer-mint and function-call endpoints discovered, smoke-
511
+ tested live. Symlink direction A confirmed by user.
512
+3. **Stage 2b -- DONE**: format normalization. Existing flat
513
+ file moved to `query-netdata-cloud/query-metrics.md`. New
514
+ SKILL.md plus per-domain guides
515
+ (`query-logs.md`, `query-topology.md`, `query-flows.md`,
516
+ `query-alerts.md`, `query-dyncfg.md`, `query-functions.md`)
517
+ written. v3 agent paths used everywhere.
518
+4. **Stage 2c -- DONE**: `query-netdata-agents/SKILL.md` written
519
+ with bearer-mint flow described; `scripts/_lib.sh` ships
520
+ `agents_resolve_bearer`, `agents_call_function`,
521
+ `agents_netdata_prefix`. Both relative symlinks created and
522
+ resolved.
523
+5. **Stage 2d -- DONE**: AGENTS.md updated with public-skill
524
+ convention paragraph + public-skill index entries.
525
+
526
+The remaining stages cover the second-expansion scope:
527
+
528
+6. **Stage 2e -- token-safety architecture rework**:
529
+ - Rename `agents_resolve_bearer` to `_agents_resolve_bearer`
530
+ (internal, never returns to stdout).
531
+ - Add `agents_query_cloud <method> <path> [<body>]` and
532
+ `agents_query_agent <node> <method> <path> [<body>]` public
533
+ wrappers that handle auth internally and emit only the
534
+ response body to stdout.
535
+ - Add per-surface convenience wrappers (one per query-*.md
536
+ guide).
537
+ - Add a unit test that drives every public wrapper with a
538
+ fake token and asserts the token bytes never appear on
539
+ captured stdout.
540
+7. **Stage 2f -- per-domain guides for the agent skill**:
541
+ For `query-netdata-agents/`, write `query-metrics.md`,
542
+ `query-logs.md`, `query-topology.md`, `query-flows.md`,
543
+ `query-alerts.md`, `query-dyncfg.md`, `query-functions.md`,
544
+ `query-nodes.md`, `query-streaming.md`. Each uses the
545
+ token-safe wrappers from stage 2e in every example.
546
+8. **Stage 2g -- per-domain guides for the cloud skill**:
547
+ For `query-netdata-cloud/`, add `query-nodes.md`,
548
+ `query-rooms.md`, `query-members.md`, `query-feed.md`. Each
549
+ uses the token-safe wrappers; covers v3 endpoints (or
550
+ documented v2 fallbacks).
551
+9. **Stage 2h -- SKILL.md re-indexing**: both `SKILL.md` files
552
+ updated to list every per-domain guide AND the canonical
553
+ reference docs AND the how-tos directory.
554
+10. **Stage 2i -- how-tos infrastructure**:
555
+ - Create `<skill>/how-tos/INDEX.md` with the format and
556
+ authoring rules (one how-to per file, slug, question,
557
+ steps, wrappers used, expected output, gotchas).
558
+ - Seed `INDEX.md` with the user-supplied question list (one
559
+ stub per question, marked TODO until a how-to is
560
+ authored). Stub how-tos are NOT a valid close state for
561
+ SOW-0006 but ARE the close state for this SOW (the
562
+ catalog is the deliverable; populating it happens during
563
+ verification).
564
+ - Document the rule "if you analyze, you author a how-to"
565
+ in both `SKILL.md` files AND in `AGENTS.md` so future
566
+ assistants honor it.
567
+11. **Stage 2j -- seed verify/questions.md**: write the seed
568
+ question list for both skills (the user's 10+ questions plus
569
+ coverage for every per-domain guide). The harness that
570
+ consumes them is SOW-0006.
571
+12. **Stage 2k -- final validation**:
572
+ - Run every public wrapper end-to-end (Cloud + Agent
573
+ transports).
574
+ - Run the no-token-on-stdout unit test.
575
+ - Run shellcheck on every script.
576
+ - Run the spec's pre-commit grep on every changed file.
577
+ - Confirm both symlinks resolve.
578
+13. **Stage 2l -- close**: status `completed`, move to `done/`,
579
+ single commit covering skill expansion + AGENTS.md update +
580
+ SOW close + SOW-0006 (pending) creation.
581
+
582
+Validation plan:
583
+
584
+- Stage 1: documentation read-through (this SOW).
585
+- Stage 2: real-use evidence (one Cloud-proxied function call,
586
+ one direct-agent function call after bearer mint, one redaction
587
+ self-test); shellcheck on every script; pre-commit grep against
588
+ token-shaped strings; confirm symlinks resolve via
589
+ `git ls-files --stage` and `readlink -f`.
590
+
591
+Artifact impact plan:
592
+
593
+- AGENTS.md: add public-skill convention paragraph + public-skill
594
+ index entries.
595
+- Runtime project skills: the two new skills are public
596
+ (`docs/netdata-ai/skills/`) but reachable from
597
+ `.agents/skills/` via relative symlinks; both names trigger on
598
+ AI-skill router queries.
599
+- Specs: not required at this stage. Stage 2 may add a short
600
+ spec under `.agents/sow/specs/skills-format.md` capturing the
601
+ convention if useful for future reviewers.
602
+- End-user/operator docs: the new SKILL.md and supporting docs
603
+ ARE end-user-facing (anyone using AI assistants with this
604
+ repo). They live in `docs/netdata-ai/skills/`.
605
+- End-user/operator skills: the two new skills.
606
+- SOW lifecycle: open in `pending/`; moves to `current/` once
607
+ decisions 1-3 are recorded; moves to `done/` after stage 2g.
608
+
609
+Open-source reference evidence:
610
+
611
+- Not consulted at stage 1. Stage 2 may consult upstream
612
+ observability projects (e.g. Grafana, Datadog) for reference
613
+ patterns on auth-token caching helpers if useful, but no
614
+ external reference is required to proceed. The user's
615
+ workstation has a local mirror tree available for grep / read.
616
+
617
+Open decisions:
618
+
619
+- See "Implications And Decisions" below. Implementation cannot
620
+ begin until decisions 1-3 are answered.
621
+
622
+## Implications And Decisions
623
+
624
+Decisions 1 and 2 are RESOLVED (2026-05-03) by reading the
625
+cloud-* source code (`cloud-frontend`, `cloud-spaceroom-service`,
626
+`cloud-charts-service`) and live smoke-testing against the
627
+production Cloud API. Decision 3 still needs the user's explicit
628
+confirmation. Findings recorded inline below.
629
+
630
+1. **Cloud REST endpoint that mints an agent bearer from a
631
+ cloud token.** RESOLVED 2026-05-03 (option **A**). Endpoint
632
+ exists; smoke-tested live.
633
+ - Path: `GET ${NETDATA_CLOUD_HOSTNAME}/api/v2/bearer_get_token`
634
+ - Required query params: `node_id`, `machine_guid`, `claim_id`
635
+ - Auth header: `Authorization: Bearer ${NETDATA_CLOUD_TOKEN}`
636
+ - Response body keys: `bearer_protection` (bool),
637
+ `expiration` (numeric -- format TBD; smoke-test value
638
+ rendered as 1970-01-01 when interpreted as Unix seconds,
639
+ so likely milliseconds or an ISO-string variant; stage 2b
640
+ must verify), `mg` (echo of machine_guid), `status`,
641
+ `token` (36-char UUID, the bearer).
642
+ - Cloud-side handler:
643
+ `cloud-spaceroom-service/http/transport_http.go:355`
644
+ (`makeGetAgentBearerToken`); route registered at
645
+ `cloud-spaceroom-service/http/endpoints_agent.go:179`.
646
+ Permission gate: `PermissionSpaceRead`; node must be
647
+ `reachable`; agent-side delegation invokes
648
+ `bearer_get_token` Function via ACLK.
649
+ - Frontend cache pattern:
650
+ `cloud-frontend/src/domains/nodes/useAgentBearer.js`.
651
+ Storage keyed by `machine_guid`. Refresh trigger:
652
+ `expiration < now + 3600 seconds` (1-hour buffer before
653
+ expiry).
654
+ - The `claim_id` is read from the agent's `/api/v3/info`
655
+ at `.agents[0].cloud.claim_id`, OR (with shell access)
656
+ from the agent host's
657
+ `<netdata-prefix>/var/lib/netdata/cloud.d/claimed_id`.
658
+ - The minted bearer is sent in subsequent direct-agent
659
+ calls as `X-Netdata-Auth: Bearer <token>` (NOT
660
+ `Authorization: Bearer <token>`).
661
+
662
+2. **Cloud REST endpoint that invokes a Function on a node by
663
+ uuid.** RESOLVED 2026-05-03 (option **A**). Endpoint exists;
664
+ smoke-tested live.
665
+ - Path: `POST ${NETDATA_CLOUD_HOSTNAME}/api/v2/nodes/{nodeId}/function?function={functionName}`
666
+ - Auth header: `Authorization: Bearer ${NETDATA_CLOUD_TOKEN}`
667
+ - Optional header: `X-Transaction-Id: <uuid>` (correlation
668
+ only; not required).
669
+ - Request body: the agent-side Function payload (e.g. for
670
+ `systemd-journal`: `{"info": true}`, or a query body with
671
+ `after`, `before`, `last`, `query`, `facets`,
672
+ `histogram`, `__logs_sources`, etc.). Optional top-level
673
+ `timeout` (ms) and `last` (page size).
674
+ - Response body: JSON (NOT streaming). Top-level keys for
675
+ `systemd-journal info=true`: `_request`, `accepted_params`,
676
+ `has_history`, `help`, `pagination`, `required_params`,
677
+ `show_ids`, `status`, `type`, `v`, `versions`.
678
+ - Cloud-side: `cloud-charts-service/http/http.go:146`
679
+ (`nodePathProxy` -> dispatches via ADC to the agent).
680
+ - Companion listing endpoint:
681
+ `POST ${NETDATA_CLOUD_HOSTNAME}/api/v3/spaces/{spaceID}/rooms/{roomID}/functions`
682
+ with body
683
+ `{"scope":{"nodes":[...]},"selectors":{"nodes":["*"]}}`
684
+ returns the list of available functions per node.
685
+ Service: `cloud-charts-service/http/http.go:135`
686
+ (`scopeFunctions` handler at
687
+ `cloud-charts-service/http/http.go:1036`).
688
+ - Smoke test 2026-05-03: Cloud function call returned 200
689
+ with valid metadata; the same call against the agent
690
+ directly (using a freshly-minted bearer) returned an
691
+ identical 2748-byte response. Both transports work
692
+ end-to-end.
693
+
694
+3. **Symlink direction confirmation.** The user wrote: skills
695
+ live in `docs/netdata-ai/skills/` and are linked from
696
+ `.agents/skills/` with relative symlinks. Confirming reading:
697
+ - A. Canonical path: `docs/netdata-ai/skills/<name>/`.
698
+ Relative symlink: `.agents/skills/<name>` ->
699
+ `../../docs/netdata-ai/skills/<name>`. *(matches user's
700
+ wording)*
701
+ - B. Other direction: `.agents/skills/<name>/` canonical,
702
+ `docs/netdata-ai/skills/<name>` symlink to it.
703
+ - C. Bidirectional / something else.
704
+ - **Recommendation:** **A**, the natural reading of the
705
+ directive and the only one consistent with "skills are
706
+ accessible by local agents too".
707
+
708
+## Plan
709
+
710
+Pending decisions 1-3. After they are answered:
711
+
712
+1. Update this SOW with the decisions; move to `current/` as
713
+ `Status: in-progress`.
714
+2. Implement per stage 2b-2f.
715
+3. Validate per stage 2f.
716
+4. Close per stage 2g.
717
+
718
+## Execution Log
719
+
720
+### 2026-05-03
721
+
722
+- Stages 2e-2k (2026-05-03 late evening). Token-safety rework
723
+ shipped: `_lib.sh` now exposes `agents_query_cloud`,
724
+ `agents_query_agent`, `agents_call_function` as token-safe
725
+ public wrappers; bearer / cloud-token / claim_id never reach
726
+ stdout; `_agents_resolve_bearer` rewritten to return via bash
727
+ nameref. `agents_selftest_no_token_leak` PASSES (drives
728
+ wrappers with a sentinel token, asserts sentinel never appears
729
+ on captured stdout). Per-domain agent guides written:
730
+ `query-functions.md`, `query-logs.md`, `query-topology.md`,
731
+ `query-flows.md`, `query-alerts.md`, `query-dyncfg.md`,
732
+ `query-metrics.md`, `query-nodes.md`, `query-streaming.md`.
733
+ Per-domain cloud-only guides written: `query-nodes.md`,
734
+ `query-rooms.md`, `query-members.md`, `query-feed.md`. The
735
+ `feed` endpoint discovered: `POST /api/v1/feed/search` served
736
+ by the separate `cloud-feed-service`, snake_case `space_id`
737
+ body field, response wraps Elasticsearch hits in
738
+ `results.hits.hits[]._source` with ECS v8.4 + Netdata-specific
739
+ envelope. The `members` endpoint:
740
+ `GET /api/v2/spaces/{sp}/members`. The `rooms` endpoint:
741
+ `GET /api/v2/spaces/{sp}/rooms`. Both SKILL.md files
742
+ re-indexed with all per-domain guides + canonical references
743
+ + how-tos + verify pointers. The "if you analyze, you author
744
+ a how-to" rule baked into SKILL.md, AGENTS.md, and both
745
+ how-tos/INDEX.md files. Seed verify/questions.md written for
746
+ both skills (Costa's user-supplied 23 + 19 questions covering
747
+ identity / hardware / OS / streaming / vnodes / collectors /
748
+ alerts / logs / topology / flows / dyncfg / members / rooms /
749
+ feed / token-safety self-test). Stage 2l (close) pending.
750
+- Stage-1 close-out (2026-05-03 evening). User added
751
+ `NETDATA_REPOS_DIR` and `NETDATA_CLOUD_HOSTNAME` to `.env`.
752
+ Live API probes validated all five existing AGENT_EVENTS_*
753
+ keys: `GET /api/v2/spaces` -> `/rooms` -> `POST .../nodes`;
754
+ matched node state `reachable`, version `v2.9.0-5-nightly`.
755
+ ssh path validated (passwordless, journalctl
756
+ --namespace=${AGENT_EVENTS_HOSTNAME} returned real data).
757
+ Direct-agent path validated (port 19999 reachable,
758
+ `/api/v3/info` returns 200 unauthenticated; functions
759
+ return 412 without bearer). Read the cloud-* sources at
760
+ `${NETDATA_REPOS_DIR}/cloud-frontend`,
761
+ `${NETDATA_REPOS_DIR}/cloud-spaceroom-service`,
762
+ `${NETDATA_REPOS_DIR}/cloud-charts-service` to discover
763
+ the bearer-mint endpoint and the function-call endpoint.
764
+ Both endpoints smoke-tested live (200 OK, valid responses;
765
+ bearer minted; direct-agent call after bearer-mint
766
+ returned identical metadata to Cloud-proxied call).
767
+ Decisions 1 and 2 RESOLVED. Decision 3 (symlink direction)
768
+ still pending. Raw artifacts saved under
769
+ `<repo>/.local/audits/query-netdata-cloud/probe/`.
770
+ Removed redundant keys `AGENT_EVENTS_IP` and
771
+ `AGENT_EVENTS_JOURNAL_NAMESPACE` after user noted that the
772
+ existing `AGENT_EVENTS_HOSTNAME` value covers ssh/HTTP/
773
+ journal-namespace roles (quadruple-duty).
774
+- Stage 1 investigation completed (originally as part of the old
775
+ SOW-2 "agent-events triage skill"). Confirmed `.env` + skill
776
+ `_lib.sh` pattern across coverity-audit / sonarqube-audit /
777
+ pr-reviews / graphql-audit. Confirmed `systemd-journal`
778
+ Function shape and `bearer_get_token` Cloud-source gate in
779
+ source.
780
+- Scope expanded by user (format normalization, query-netdata-
781
+ cloud refactor, query-netdata-agents new skill).
782
+- User chose 4-SOW split. The agent-events triage skill moved
783
+ to SOW-0003; documentation-pipeline skills to SOW-0004;
784
+ mirror-netdata-repos skill to SOW-0005. This SOW (SOW-0010) was
785
+ rescoped to skill infrastructure and renamed to "Netdata
786
+ query skills infrastructure". Old filename
787
+ `SOW-0010-20260503-agent-events-skill.md` removed; new
788
+ filename `SOW-0010-20260503-netdata-query-skills-
789
+ infrastructure.md`.
790
+- Post-close (2026-05-04): user split the original
791
+ doc-pipeline SOW into documentation (SOW-0004 rescoped to
792
+ `learn-site-structure`) and integrations (new SOW-0007
793
+ `integrations-lifecycle`). Cross-references updated for
794
+ navigation accuracy.
795
+
796
+## Validation
797
+
798
+Acceptance criteria evidence:
799
+
800
+- Symmetric file structure delivered: both
801
+ `docs/netdata-ai/skills/query-netdata-cloud/` and
802
+ `docs/netdata-ai/skills/query-netdata-agents/` ship
803
+ `SKILL.md` + 8 shared per-domain guides
804
+ (`query-{metrics,logs,topology,flows,alerts,dyncfg,functions,nodes}.md`).
805
+ Cloud adds `query-{rooms,members,feed}.md`; agent adds
806
+ `query-streaming.md`. Both ship `how-tos/INDEX.md` and
807
+ `verify/questions.md`.
808
+- Token-safety architecture delivered:
809
+ `agents_query_cloud`, `agents_query_agent`,
810
+ `agents_call_function` are the public wrappers; internal
811
+ helpers renamed with leading underscore and return token
812
+ bytes via bash namerefs only.
813
+ `agents_selftest_no_token_leak` PASSES on every run
814
+ (verified live: bearer-mint, cloud call, direct-agent call,
815
+ bearer-cache hit -- all confirm zero token bytes on
816
+ captured stdout).
817
+- Both relative symlinks at `.agents/skills/` resolve to the
818
+ corresponding `docs/netdata-ai/skills/` directory.
819
+- `AGENTS.md` updated with the public-skill convention paragraph,
820
+ the public-skill index entries, the token-safety contract
821
+ paragraph, and the live how-tos catalog rule.
822
+- v3 agent paths used everywhere; v2/v1 only as fallback for
823
+ pre-v2 agents (alerts section explicitly notes this).
824
+- Cloud verification covers the eleven domains the SOW
825
+ required; agent verification covers the nine domains the SOW
826
+ required. Both `verify/questions.md` files seeded with the
827
+ user-supplied questions plus per-domain coverage.
828
+
829
+Tests or equivalent validation:
830
+
831
+- shellcheck on `_lib.sh`: clean (no findings).
832
+- `agents_selftest_no_token_leak`: `[PASS]`.
833
+- Live smoke tests against production:
834
+ - `agents_query_cloud GET /api/v2/accounts/me` -- 200,
835
+ response delivered, zero cloud-token bytes in stdout.
836
+ - `agents_query_agent ... POST /api/v3/function?function=systemd-journal '{"info":true}'`
837
+ -- 200, 2748-byte response, zero cloud-token bytes and
838
+ zero bearer bytes in stdout.
839
+ - Agent v3 alert paths: `/api/v3/alerts`,
840
+ `/api/v3/alert_transitions`, `/api/v3/alert_config` --
841
+ all 200.
842
+ - Cloud-side: `/api/v2/spaces`, `/api/v2/accounts/me`,
843
+ `/api/v2/spaces/{sp}/rooms`, `/api/v3/spaces/{sp}/rooms/
844
+ {rm}/nodes`, `/api/v2/spaces/{sp}/members`,
845
+ `/api/v1/feed/search`, `/api/v3/spaces/{sp}/rooms/{rm}/
846
+ alerts*` -- all 200.
847
+ - Agent direct: `/api/v3/info`, `/api/v3/config?action=tree`,
848
+ `/api/v3/function?function=topology:snmp`,
849
+ `/api/v3/function?function=flows:netflow` (where the
850
+ collector is enabled) -- all 200.
851
+
852
+Real-use evidence:
853
+
854
+- The bearer cache works across calls: first call mints, second
855
+ call hits cache (verified by stable token output between
856
+ invocations within the 2-hour window).
857
+- The two transports return identical metadata for
858
+ `systemd-journal info=true` (modulo timestamps), confirming
859
+ the agent and the Cloud proxy expose the same Function
860
+ payload shape.
861
+- The discovered Cloud feed endpoint
862
+ (`POST /api/v1/feed/search`) returns 42853 hits over the
863
+ user's seed query; verified the snake_case `space_id` body
864
+ field is required (camelCase `spaces[].id` is rejected with
865
+ 400).
866
+
867
+Reviewer findings:
868
+
869
+- Self-review caught: invented "alert function" terminology
870
+ and partial query-alerts.md (rewritten with all 11 endpoints,
871
+ smoke-tested).
872
+- Self-review caught: invented function names (`top`,
873
+ `aclk-state`, `ml-models`, `windows-events`, `streaming`).
874
+ Replaced with the live-verified list pulled from the
875
+ agent-events node.
876
+- User-flagged: missing topology + flow Function families.
877
+ Added query-topology.md and query-flows.md with
878
+ source-verified payload shapes.
879
+- User-flagged: missed FUNCTION_UI_REFERENCE.md /
880
+ FUNCTION_UI_DEVELOPER_GUIDE.md / FUNCTION_UI_SCHEMA.json /
881
+ DYNCFG.md. Added explicit references in SKILL.md and the
882
+ per-domain guides.
883
+- User-flagged: v2 agent paths. Switched all agent-direct
884
+ paths to v3 (v2/v1 only as fallback).
885
+- User-flagged: 4-family taxonomy was a fabrication. Replaced
886
+ with the canonical 2-class taxonomy from
887
+ FUNCTION_UI_REFERENCE.md (Simple Table + Log Explorer);
888
+ topology and flows documented as custom Functions building
889
+ on the same envelope.
890
+- User-flagged: assistant must never see tokens. Reworked
891
+ `_lib.sh` with internal/public split and shipped a
892
+ no-leak self-test.
893
+
894
+Same-failure scan:
895
+
896
+- Spec discipline grep on every committed file: zero
897
+ violations (the only `deadbeef-...-...-...-...-...` UUID
898
+ in `_lib.sh` is the deliberate self-test sentinel and is
899
+ not a real credential).
900
+
901
+Sensitive data gate:
902
+
903
+- Pre-commit grep ran clean over every file touched by this
904
+ SOW: zero UUID-shaped IDs (except the test sentinel),
905
+ zero IPv4 literals to specific hosts (only loopback
906
+ `127.0.0.1` examples and a `<YOUR_FOCUS_DEVICE_IP>`
907
+ placeholder), zero forbidden absolute paths
908
+ (Netdata defaults `/var/lib/netdata`, `/etc/netdata` are
909
+ explicitly allowed by the spec).
910
+- Cloud REST host is env-keyed via `${NETDATA_CLOUD_HOSTNAME}`
911
+ in scripts; appears as the literal `app.netdata.cloud` only
912
+ in user-facing curl examples (allowed for public Netdata-org
913
+ sites in role-descriptive prose).
914
+- Token-safe wrappers verified: `agents_selftest_no_token_leak`
915
+ PASSES; `agents_query_cloud` and `agents_query_agent` both
916
+ emit zero cloud-token bytes and zero bearer bytes on captured
917
+ stdout.
918
+
919
+Artifact maintenance gate:
920
+
921
+- AGENTS.md: updated with public-skill convention paragraph,
922
+ the public-skill index, the token-safety contract paragraph,
923
+ and the how-tos catalog rule.
924
+- Runtime project skills: two new public skills shipped at
925
+ `docs/netdata-ai/skills/query-netdata-{cloud,agents}/`,
926
+ reachable from `.agents/skills/` via relative symlinks.
927
+- Specs: `<repo>/.agents/sow/specs/sensitive-data-discipline.md`
928
+ shipped (separate concern; underwrites this SOW + future
929
+ SOWs). No additional spec required for the skill convention
930
+ -- documented in AGENTS.md.
931
+- End-user/operator docs: the two new skill bundles ARE the
932
+ end-user-facing docs.
933
+- End-user/operator skills: ditto.
934
+- SOW lifecycle: moved from `pending/` -> `current/` ->
935
+ `done/` per the framework. Status `completed`. Verification
936
+ harness deferred to `SOW-0006` per user direction; SOW-0006
937
+ shipped as a stub in `pending/`.
938
+
939
+Specs update:
940
+
941
+- New spec `<repo>/.agents/sow/specs/sensitive-data-discipline.md`
942
+ (the rule that underwrites this SOW's discipline gate).
943
+
944
+Project skills update:
945
+
946
+- Two new public skills under `docs/netdata-ai/skills/` with
947
+ relative symlinks from `.agents/skills/`.
948
+
949
+End-user/operator docs update:
950
+
951
+- The two new skill bundles ARE the docs update.
952
+
953
+End-user/operator skills update:
954
+
955
+- The two new skill bundles ARE the skills update.
956
+
957
+Lessons:
958
+
959
+- **Verify before documenting.** The first draft of
960
+ query-alerts.md and query-functions.md contained invented
961
+ terms ("alert functions") and invented Function names
962
+ (`top`, `aclk-state`, `ml-models`). Source-verification
963
+ pass and live smoke-testing caught both. Lesson: every
964
+ endpoint table must be smoke-tested before commit; every
965
+ Function name list must be pulled from the live listing
966
+ endpoint.
967
+- **Public Swagger is incomplete.** `app.netdata.cloud/api/docs/`
968
+ documents only 7 paths. Real cloud endpoints
969
+ (`bearer_get_token`, function-call proxy, alert endpoints,
970
+ feed search) live across 4+ microservices and were
971
+ discovered by reading the cloud-* sources at
972
+ `${NETDATA_REPOS_DIR}/cloud-*/`. Lesson: when Swagger is
973
+ thin, read source.
974
+- **Cloud-side `expiration: 0` is real.** The bearer mint
975
+ endpoint returns `expiration: 0` on the production cloud,
976
+ which would force re-mint on every call if naively
977
+ followed. The cache logic now stamps `_cached_at` and falls
978
+ back to a 2-hour mint window when `expiration` is 0. Agents
979
+ actually issue ~3-hour TTL bearers, so 2 hours leaves a
980
+ safety margin.
981
+- **zsh vs bash compat.** `BASH_SOURCE[0]` warning on zsh
982
+ was noisy. Solution: capture `_agents_lib_self` at source
983
+ time using a `ZSH_VERSION`/`BASH_VERSION` switch with
984
+ `eval` for the zsh-only `${(%):-%x}` syntax.
985
+- **Token-safety needs architectural enforcement, not
986
+ discipline.** The first draft had `agents_resolve_bearer`
987
+ return the bearer to stdout; refactoring to bash namerefs
988
+ + leading-underscore "internal" naming + a no-leak
989
+ self-test is what makes the contract verifiable.
990
+
991
+Follow-up mapping:
992
+
993
+- Verification harness (Sonnet test runner + grading rubric +
994
+ how-to extraction prompt loop): tracked in
995
+ `<repo>/.agents/sow/pending/SOW-0006-20260503-skill-verification-harness.md`.
996
+- `query-agent-events` private skill (consumes the wrappers
997
+ delivered here): tracked in
998
+ `<repo>/.agents/sow/pending/SOW-0003-20260503-query-agent-events-skill.md`.
999
+- `learn-site-structure` private skill: tracked in
1000
+ `<repo>/.agents/sow/pending/SOW-0004-20260503-learn-site-structure-skill.md`.
1001
+- `integrations-lifecycle` private skill: tracked in
1002
+ `<repo>/.agents/sow/pending/SOW-0007-20260504-integrations-lifecycle-skill.md`.
1003
+- `mirror-netdata-repos` private skill: tracked in
1004
+ `<repo>/.agents/sow/pending/SOW-0005-20260503-mirror-netdata-repos-skill.md`.
1005
+- The how-tos catalog stubs in both skills are deliberately
1006
+ unfilled at close: they get populated when the verification
1007
+ harness (SOW-0006) drives Sonnet through `verify/questions.md`
1008
+ and prompts for new how-tos on misses. The "if you analyze,
1009
+ you author a how-to" rule is durable; SOW close does not
1010
+ require pre-populated how-tos beyond the stub catalog.
1011
+
1012
+## Outcome
1013
+
1014
+Delivered. Two symmetric public skills ship at
1015
+`docs/netdata-ai/skills/query-netdata-{cloud,agents}/`, each with
1016
+SKILL.md + per-domain guides covering every queryable Netdata
1017
+surface, token-safe wrappers in `query-netdata-agents/scripts/
1018
+_lib.sh` (verified by self-test), live `how-tos/INDEX.md`
1019
+catalogs, seed `verify/questions.md` lists, and relative
1020
+symlinks from `.agents/skills/`. AGENTS.md updated.
1021
+`<repo>/.agents/sow/specs/sensitive-data-discipline.md` shipped.
1022
+Verification harness deferred to SOW-0006 per user direction.
1023
+
1024
+## Lessons Extracted
1025
+
1026
+See "Lessons" inside the Validation section above.
1027
+
1028
+## Followup
1029
+
1030
+See "Follow-up mapping" inside the Validation section above.
1031
+
1032
+## Regression Log
1033
+
1034
+None yet.
1035
+
1036
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0011-20260505-codacy-audit-skill.md
new
+330
@@ -0,0 +1,330 @@
1
+# SOW-0011 - codacy-audit skill
2
+
3
+## Status
4
+
5
+Status: completed
6
+
7
+Sub-state: skill shipped (`.agents/skills/codacy-audit/` with SKILL.md, 3 scripts, how-tos catalog with 1 seed entry). All acceptance criteria met.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Give the user (and AI assistants helping the user) a fit-for-purpose Codacy-Cloud workflow for `netdata/netdata`:
14
+
15
+1. **Pre-push prevention** -- run the same analyzers Codacy CI runs locally, before `git push`, so PR feedback loops collapse from "push -> wait minutes -> see findings -> fix -> push again" to "run -> see findings -> fix -> push once".
16
+2. **Read-only PR/repo triage** -- list issues on a PR (current pattern: ad-hoc `curl` against `api.codacy.com`), cluster by tool/pattern/severity, and write the triage notes under `<repo>/.local/audits/codacy/`.
17
+
18
+Out of scope for this SOW (deferred to a future SOW if needed):
19
+- Write actions (mark FP, mark fixed, modify ignore-patterns via API).
20
+- Master-backlog triage on the 31,425 open issues currently on `master`.
21
+- Cross-repo (entire `netdata` org) aggregation.
22
+
23
+### User Request
24
+
25
+> "create the sow and build it"
26
+
27
+Accepting the assistant's recommendation (delivered earlier in the conversation): "Start with track 1 only (the local CLI wrapper) as a small SOW. It pays off on every future PR. Track 2 (triage workflow) becomes worthwhile once we hit a PR or quarterly review where Codacy backlog matters -- defer until then."
28
+
29
+User also added `CODACY_TOKEN` to `<repo>/.env`. This is an Account API token (header `api-token: <value>`) -- usable for read-only PR/repo queries today; future write actions deferred.
30
+
31
+### Assistant Understanding
32
+
33
+Facts:
34
+
35
+- Three sister legacy skills already exist with the triage shape we want to mirror: `.agents/skills/coverity-audit/`, `.agents/skills/sonarqube-audit/`, `.agents/skills/graphql-audit/`. A `codacy-audit/` is the natural fourth.
36
+- Codacy Cloud ships an official local CLI: `codacy-analysis-cli` (https://github.com/codacy/codacy-analysis-cli). Install paths: docker, install.sh, brew. **Docker is available on this workstation** (`/usr/bin/docker`, version 29.4.1).
37
+- The Codacy v3 REST API at `api.codacy.com` is reachable: PR-level issue lists work even anonymously; broader cross-PR / org queries require an Account API token. Verified live during the conversation that this skill is being created for: `gh user 30945 = Costa Tsaousis`, 31,425 issues currently open on `master`.
38
+- PR #22423 is a useful end-to-end fixture for validation: it had 864 markdownlint findings on the first CI run; fixed by `.codacy.yml` exclusion in commit `3a54c9afbc`. The local CLI must reproduce the original 864 findings for an objective accuracy check.
39
+- The repo's `.codacy.yml` is the source of truth for path exclusions; the local CLI must respect it (or we have to teach it to).
40
+- Path discipline spec at `<repo>/.agents/sow/specs/sensitive-data-discipline.md` already defines `CODACY_TOKEN`-class constraints by precedent (Coverity / Sonar tokens). Keys must live in `.env`, never in committed artifacts.
41
+
42
+Inferences:
43
+
44
+- The token is set today but not exercised by any committed script. The skill's `_lib.sh` should ship a token-safe sentinel self-test like `agentevents_selftest_no_token_leak` (introduced by SOW-0003) so that future write-action expansion inherits the discipline cleanly.
45
+- Most Netdata code PRs come back Codacy-clean (recent merged PRs all show `pass`). The high-value moments for this skill are: (a) doc-heavy PRs like #22423; (b) any PR that adds a new file type (e.g. JS, Python) to a new tree; (c) periodic quarterly review of master-backlog (deferred to follow-up SOW).
46
+- Mirroring the coverity-audit/sonarqube-audit shape is preferable to inventing a new layout: same filename conventions (`SKILL.md`, `scripts/_lib.sh`, `scripts/<verb>-<noun>.sh`), same artifact destination (`<repo>/.local/audits/codacy/`), same live-how-tos rule.
47
+
48
+Unknowns:
49
+
50
+- Whether `codacy-analysis-cli` reads `.codacy.yml` exclude_paths the same way Codacy CI does. Will verify during implementation by running the CLI on PR #22423's pre-exclusion state and counting markdownlint findings.
51
+- Whether running the CLI in docker against a docker-mounted source tree produces clean enough output to redirect into a JSON dump. Will pick `--format json` (or `--format sarif`) at implementation time.
52
+- Whether the bundled tools list in `codacy-analysis-cli` overlaps 1:1 with what Codacy CI runs on `netdata/netdata`. The 864 finding fixture answers this empirically.
53
+
54
+### Acceptance Criteria
55
+
56
+1. `.agents/skills/codacy-audit/SKILL.md` exists with frontmatter `name=codacy-audit`, description that lists trigger phrases, follows the same shape and conventions as `coverity-audit` / `sonarqube-audit`. Verification: skill loads in agent harness without YAML errors; `head -1` of the description shows the trigger phrasing.
57
+2. `scripts/_lib.sh` ships token-safe wrappers and a no-leak self-test (`codacyaudit_selftest_no_token_leak`). Verification: run the self-test in CI-style with a sentinel UUID; sentinel must NOT appear on captured stdout.
58
+3. `scripts/analyze-local.sh` runs `codacy-analysis-cli` (via docker) and produces a JSON dump under `<repo>/.local/audits/codacy/<timestamp>.json`. Verification: run on the PR-22423 pre-exclusion state, confirm a non-zero count of markdownlint findings, confirm `.codacy.yml` exclusions are honoured by checking that excluded paths produce no rows in post-exclusion runs.
59
+4. `scripts/pr-issues.sh` fetches Codacy issues for an arbitrary PR number via the v3 API, writes a JSON dump under `<repo>/.local/audits/codacy/`, and emits a TSV summary clustering by tool / pattern / severity / file. Verification: run on PR #22423; confirm `total > 0` historical baseline and graceful handling for PRs with `total = 0`.
60
+5. `how-tos/INDEX.md` exists, documents the live-catalog rule, and seeds at least one how-to derived from this SOW's verification work. Verification: file exists; `INDEX.md` lists at least one entry.
61
+6. `<repo>/.agents/ENV.md` lists `CODACY_TOKEN` with role / where-to-find / sample format / which scripts consume it. Verification: grep for `CODACY_TOKEN` returns the new row.
62
+7. `<repo>/.env.template` lists `CODACY_TOKEN=""` with a setup pointer to `.agents/ENV.md`. Verification: grep returns the new line.
63
+8. `AGENTS.md` skill index gains a `codacy-audit` entry under "Legacy runtime skills" (or a new bucket if appropriate). Verification: grep returns the new pointer.
64
+9. `<repo>/.codacy.yml` is unchanged by this SOW (its exclusions are the source of truth; the local CLI must honour them). Verification: `git diff -- .codacy.yml` after this SOW shows no changes.
65
+10. shellcheck on the new scripts: ALL CLEAN with `--external-sources`. Verification: `shellcheck --external-sources .agents/skills/codacy-audit/scripts/*.sh`.
66
+
67
+## Analysis
68
+
69
+Sources checked:
70
+
71
+- `.agents/skills/coverity-audit/SKILL.md` and `scripts/` (sister skill, structural template).
72
+- `.agents/skills/sonarqube-audit/SKILL.md` (sister skill, second structural template).
73
+- `.agents/skills/graphql-audit/` (third sister skill).
74
+- `.agents/skills/query-agent-events/scripts/_lib.sh` (token-safe self-test pattern, current best practice in this repo).
75
+- `.codacy.yml` (current exclusion list).
76
+- `<repo>/.env`, `<repo>/.env.template`, `<repo>/.agents/ENV.md` (env-key surface area).
77
+- Codacy v3 REST API (`api.codacy.com`) -- live confirmed during this conversation.
78
+- `https://github.com/codacy/codacy-analysis-cli` upstream README.
79
+
80
+Current state:
81
+
82
+- Codacy CI runs on every PR; the only feedback path today is the GitHub check.
83
+- 864 markdownlint findings in the recent PR-22423 history confirm the analyzer set we need to match locally.
84
+- `.env` has `CODACY_TOKEN` set; no committed script consumes it yet.
85
+- No `.agents/skills/codacy-audit/` directory exists today.
86
+
87
+Risks:
88
+
89
+- **Docker-CLI path differences**: `codacy-analysis-cli` mounted via docker may interpret paths differently (host path vs container `/src`). Mitigation: standardize on `--directory /src` and bind-mount `<repo>:/src:ro`.
90
+- **Tool drift between local CLI and Codacy CI**: the local CLI bundles a fixed set of analyzers; Codacy Cloud may add/remove tools server-side. Mitigation: document which tools are run locally; for tools Codacy Cloud adds that the CLI doesn't, fall back to the API for ground truth.
91
+- **Token accidentally committed via a finding dump**: a JSON dump from the API could echo back the token in error messages. Mitigation: token-safe wrappers in `_lib.sh` plus the no-leak self-test.
92
+- **`.local/audits/codacy/` filling up with stale dumps**: ephemeral, gitignored, but disk pressure risk on long sessions. Mitigation: filename includes timestamp; user can `rm` whenever.
93
+
94
+## Pre-Implementation Gate
95
+
96
+Status: ready
97
+
98
+Problem / root-cause model:
99
+
100
+- The user has Codacy Cloud configured for `netdata/netdata` and has placed `CODACY_TOKEN` in `.env`. There is no committed tooling to (a) run the same analyzers locally before pushing or (b) fetch and triage Codacy findings on a PR programmatically. The recent PR #22423 demonstrated the cost of this gap: 864 findings were only visible after the CI round-trip. Building a small skill captures the operational knowledge and shrinks the loop.
101
+
102
+Evidence reviewed:
103
+
104
+- Codacy v3 API live response: account auth verified (`api-token` header), `master` branch carries 31,425 open issues, PR-22423 carried 864 issues all `markdownlint`.
105
+- `.agents/skills/coverity-audit/SKILL.md` lines 1-40 -- frontmatter shape and "MANDATORY" startup sequence.
106
+- `.agents/skills/query-agent-events/scripts/_lib.sh` -- current token-safe pattern (`agentevents_selftest_no_token_leak` style; sentinel UUID drives every public wrapper, asserts no leak on stdout).
107
+- `https://github.com/codacy/codacy-analysis-cli` README -- docker invocation: `docker run --rm -v "$PWD":/src codacy/codacy-analysis-cli:latest analyze --directory /src`.
108
+
109
+Affected contracts and surfaces:
110
+
111
+- New skill: `.agents/skills/codacy-audit/`.
112
+- New audit dir: `<repo>/.local/audits/codacy/` (gitignored; created by skill scripts at runtime).
113
+- ENV surface: `CODACY_TOKEN` (new row in `.agents/ENV.md` and `.env.template`).
114
+- AGENTS.md skill index.
115
+- No changes to `.codacy.yml`, no changes to existing skills, no changes to source code.
116
+
117
+Existing patterns to reuse:
118
+
119
+- coverity-audit's "MANDATORY -- keep this skill alive" + "MANDATORY -- startup sequence" SKILL.md sections.
120
+- query-agent-events's `_lib.sh` token-safe wrapper layout (env load with `: "${VAR:?}"`, audit-dir helper, masked-token `_run`/`_run_read` form, sentinel self-test).
121
+- The how-tos catalog rule: assistants author a how-to whenever they perform analysis the catalog doesn't already cover.
122
+- The `<topic>-audit` -> `.local/audits/<topic>/` directory naming convention from AGENTS.md (so `codacy-audit/` writes to `.local/audits/codacy/`).
123
+
124
+Risk and blast radius:
125
+
126
+- **Local-only**: nothing in this SOW ships to end-user agents, the Cloud, or the binary. Blast radius = this repo's `.agents/` tree + ENV.md.
127
+- **Reversibility**: every artifact is gitignored or under `.agents/skills/codacy-audit/` (deletable in one commit). No destructive operations.
128
+- **Security**: token-safe `_lib.sh` plus the self-test ensure the token never reaches captured stdout. Audit dumps go to gitignored `.local/`.
129
+- **Performance**: docker pull on first run (~few hundred MB); subsequent runs warm-cached.
130
+- **Compatibility**: targets the Codacy API as it exists today; if Codacy changes auth/headers, the skill breaks loudly with a 401 and the lib's error message points the user at `.agents/ENV.md`.
131
+
132
+Sensitive data handling plan:
133
+
134
+- `CODACY_TOKEN` is a credential -- handled exactly like `NETDATA_CLOUD_TOKEN`, `COVERITY_COOKIE`, `SONAR_TOKEN`: lives in `.env` (gitignored), referenced via `${CODACY_TOKEN}` in scripts only, never in commit messages, never in fixtures.
135
+- Account ID 30945 (Costa's Codacy account) was returned by `/v3/user` during exploration but will not be written to any committed artifact. The SOW redacts to "the configured account".
136
+- Audit JSON dumps land under `<repo>/.local/audits/codacy/<timestamp>.json` (gitignored).
137
+- No customer / community member / private-host data is touched -- this is a public-repo CI workflow.
138
+
139
+Implementation plan:
140
+
141
+1. **Skill scaffolding** (10 min): create `.agents/skills/codacy-audit/`, write `SKILL.md` with frontmatter, MANDATORY sections, table of contents, env-keys table, related-skills cross-refs.
142
+2. **`scripts/_lib.sh`** (20 min): env load (validates `CODACY_TOKEN`), audit dir helper, host detection (`api.codacy.com`), masked-token `codacyaudit_run`/`codacyaudit_run_read`, sentinel-based no-leak self-test (`codacyaudit_selftest_no_token_leak`). Sentinel UUID drives every public wrapper; capture stdout; assert sentinel absence.
143
+3. **`scripts/analyze-local.sh`** (30 min): docker invocation of `codacy-analysis-cli analyze --directory /src --format json --output <audit>/<ts>.json`. CLI args: `--tool <name>` (optional), `--upload` disabled (we do read-only locally). Fall back to script-install if docker unavailable.
144
+4. **`scripts/pr-issues.sh`** (30 min): paginated fetch of `/v3/analysis/organizations/gh/<org>/repositories/<repo>/pull-requests/<n>/issues`; default `org=netdata`, `repo=netdata`. Output: full JSON dump + TSV summary `(count, tool, pattern, severity)`. Token used via `_lib.sh` wrappers (token-safe).
145
+5. **`how-tos/INDEX.md`** (10 min): catalog file with the live-rule paragraph; seed with at least one how-to (the PR-22423 reproduction).
146
+6. **`.agents/ENV.md` + `.env.template`** (10 min): add `CODACY_TOKEN` row in the per-skill checklist; add CODACY section in template; cross-link to skill.
147
+7. **`AGENTS.md`** (5 min): add `codacy-audit` pointer under "Legacy runtime skills" (or "Skill index" if a new bucket is more honest -- decision while writing).
148
+8. **End-to-end validation** (20 min): run `analyze-local.sh` on PR-22423 fixture; run `pr-issues.sh` on PR #22423; confirm artifact shape; confirm shellcheck clean; run no-leak self-test.
149
+9. **SOW lifecycle**: move SOW from `pending/` to `current/` before step 2; close as `completed` and move to `done/` in the same commit as the work.
150
+
151
+Validation plan:
152
+
153
+- `shellcheck --external-sources .agents/skills/codacy-audit/scripts/*.sh` -- ALL CLEAN.
154
+- `_lib.sh` self-test exercises every public wrapper with a sentinel UUID; the sentinel must not appear on captured stdout.
155
+- `analyze-local.sh` on the working tree produces a JSON dump; markdownlint findings count is sane (matches Codacy CI within ~10% on a known-state branch).
156
+- `pr-issues.sh 22423` on the current PR returns 0 findings (we already excluded the trees).
157
+- `pr-issues.sh 22420` (the most recent merged PR) returns 0 findings (it merged with Codacy `pass`).
158
+- Skill loads cleanly in the agent harness: no YAML frontmatter errors.
159
+- `git diff -- .codacy.yml` shows no changes.
160
+
161
+Artifact impact plan:
162
+
163
+- AGENTS.md: add `codacy-audit` to the skill index. Justification: the project's skill index is the canonical lookup for AI assistants.
164
+- Runtime project skills: no `project-*` skill changes. The codacy-audit skill is a legacy-style audit skill (mirrors coverity-audit), not a `project-*` per-repo runtime skill.
165
+- Specs: `.agents/sow/specs/sensitive-data-discipline.md` already covers CODACY-class tokens by precedent. Add a row for `CODACY_TOKEN` to the env-keys table for clarity. (Small spec update, not a behavior change.)
166
+- End-user/operator docs: none affected.
167
+- End-user/operator skills: none affected.
168
+- SOW lifecycle: move `pending/ -> current/ -> done/`; status `open -> in-progress -> completed`. Close in the same commit as the work, per AGENTS.md rule.
169
+
170
+Open-source reference evidence:
171
+
172
+- `codacy/codacy-analysis-cli` upstream README and `--help`. No further mirrored-repos research is required for this SOW.
173
+
174
+Open decisions:
175
+
176
+- None blocking. Scope was decided by user accepting the assistant's recommendation ("track 1 only -- CLI wrapper plus read-only PR queries").
177
+
178
+## Implications And Decisions
179
+
180
+### Decision 1 - Scope cut
181
+
182
+**Options (presented earlier in conversation, recorded here):**
183
+
184
+1. Both tracks, full skill (CLI wrapper + read-only API + write actions + master backlog triage).
185
+2. Track 1 only -- CLI wrapper + read-only PR-issue queries (deferring write actions).
186
+3. Defer entirely.
187
+
188
+**User selected**: option 2 ("ok" -- accepting assistant's recommendation: "Start with track 1 only (the local CLI wrapper) as a small SOW. Track 2 (triage workflow) becomes worthwhile once we hit a PR or quarterly review where Codacy backlog matters -- defer until then.").
189
+
190
+**Implication**: deferred follow-up SOW for write actions and master-backlog triage. Acceptable: token is exercised by read-only wrappers, the lib has the no-leak self-test in place, future expansion is a thin addition.
191
+
192
+### Decision 2 - Local CLI install path
193
+
194
+**Options:**
195
+
196
+1. Require docker (run `codacy/codacy-analysis-cli:latest`).
197
+2. Require the install.sh-installed binary (`/usr/local/bin/codacy-analysis-cli`).
198
+3. Auto-detect: prefer local binary, fall back to docker.
199
+
200
+**Selected**: option 3 (auto-detect). Reasoning: docker is universally available on this workstation but not always preferable (cold-pull cost, root requirements in some contexts). Auto-detection matches sister-skill behavior.
201
+
202
+## Plan
203
+
204
+1. Move SOW to `current/` and flip status to `in-progress`.
205
+2. Build skill scaffolding + scripts (steps 1-7 of the implementation plan above).
206
+3. Run the validation suite (shellcheck, no-leak self-test, end-to-end PR-22423 fixture).
207
+4. Move SOW to `done/`, flip to `completed`, commit work + SOW lifecycle change as one commit.
208
+5. Push.
209
+
210
+## Execution Log
211
+
212
+### 2026-05-05
213
+
214
+- SOW drafted in `pending/`; promoted to `current/` and `Status: in-progress`.
215
+- Built `.agents/skills/codacy-audit/` mirroring `coverity-audit/` shape:
216
+ - `SKILL.md` with frontmatter, MANDATORY sections, scope (in/out), env keys table, scripts table, workflow examples, related-skills cross-refs, path discipline.
217
+ - `scripts/_lib.sh` with token-safe wrappers (`_codacyaudit_run` internal, `codacyaudit_get`/`codacyaudit_post`/`codacyaudit_get_paged` public), env load with `: "${CODACY_TOKEN:?}"`, audit-dir helper, `codacyaudit_pr_issues` and `codacyaudit_repo_info` convenience wrappers, sentinel-based `codacyaudit_selftest_no_token_leak`.
218
+ - `scripts/pr-issues.sh` -- paginated PR issue fetch + clustered TSV summary; default `--by pattern`; supports `--by tool|severity|file|category`.
219
+ - `scripts/analyze-local.sh` -- auto-detect runner (local binary vs docker); docker path uses `--volume /var/run/docker.sock:/var/run/docker.sock` plus `--env CODACY_CODE=<host-path>` plus same-path bind mount, per Codacy upstream docs.
220
+ - `how-tos/INDEX.md` with the live-rule paragraph; seeded with `reproduce-pr-22423-markdownlint.md`.
221
+- Updated `<repo>/.agents/ENV.md`: added Codacy section with per-key role/where/format and a per-skill checklist row.
222
+- Updated `<repo>/.env.template`: added a `CODACY_TOKEN=""` block with inline guidance and optional override placeholders for host/provider/org/repo.
223
+- Updated `AGENTS.md` skill index: added `codacy-audit` under "Legacy runtime skills" and the brief skill list.
224
+- Updated `.agents/sow/specs/sensitive-data-discipline.md`: added a `CODACY_TOKEN` row to the env-keys table.
225
+- First self-test under zsh produced a `BASH_SOURCE[0]: parameter not set` warning (PASS still emitted). Fixed by mirroring the `query-agent-events/_lib.sh` portable `_codacyaudit_lib_self` resolution. Re-ran; clean PASS in both shells.
226
+- First docker run hit `Cannot connect to the Docker daemon` from inside the CLI container (the CLI spawns child containers per tool). Fixed by adopting the upstream docker-in-docker invocation: mount `/var/run/docker.sock`, set `CODACY_CODE`, bind-mount the source at the same path inside the container.
227
+- Verified `pr-issues.sh 22423` -> 0 issues (post-exclusion); `pr-issues.sh 22420` -> 0 issues (a recent merged PR with Codacy `pass`). Both produce well-formed JSON dumps under `<repo>/.local/audits/codacy/`.
228
+- Verified `analyze-local.sh --tool markdownlint` on a synthetic markdown fixture: produced 2 findings (MD013 + MD022) -- end-to-end docker-in-docker dispatch confirmed working for markdownlint. The shellcheck child-tool path failed in a smaller smoke test ("Is a directory") -- this is a CLI-version-specific quirk in shellcheck dispatch on a single-file directory; not in scope to fix here. The reproduce-PR-22423 how-to (markdownlint) is the validated path.
229
+- Closed SOW: `Status: completed`, moved to `done/`, committed alongside the work.
230
+
231
+## Validation
232
+
233
+Acceptance criteria evidence:
234
+
235
+1. **SKILL.md exists with proper frontmatter** -- `.agents/skills/codacy-audit/SKILL.md` has `name: codacy-audit` and a description listing trigger phrases. Loaded by the agent harness during this session (the harness's available-skills list shows `codacy-audit` immediately after the file landed).
236
+2. **`_lib.sh` token-safe self-test** -- `codacyaudit_selftest_no_token_leak` PASS under both bash and zsh; sentinel UUID `deadbeef-1234-5678-9abc-def012345678` does not appear on captured stdout from any of `codacyaudit_get`, `codacyaudit_post`, `codacyaudit_pr_issues`, `codacyaudit_repo_info`.
237
+3. **`analyze-local.sh` end-to-end** -- markdown fixture produced a JSON dump containing 2 findings (`markdownlint_MD013` + `markdownlint_MD022`); dump path is `<repo>/.local/audits/codacy/local-markdownlint-<ts>.json`; format is the Codacy `Issue` array.
238
+4. **`pr-issues.sh` end-to-end** -- PR #22423 (post-exclusion) and PR #22420 (recent merged PR) both return 0 issues; the script handles `total = 0` gracefully and does not error.
239
+5. **`how-tos/INDEX.md` exists** -- live catalog rule documented; seeded with `reproduce-pr-22423-markdownlint.md`.
240
+6. **`.agents/ENV.md` updated** -- `CODACY_TOKEN` row added with role/where/format and per-skill checklist entry.
241
+7. **`.env.template` updated** -- `CODACY_TOKEN=""` line plus optional override placeholders added.
242
+8. **AGENTS.md skill index updated** -- `codacy-audit` entry under "Legacy runtime skills" and the brief skill list.
243
+9. **`.codacy.yml` unchanged** -- `git diff -- .codacy.yml` after this SOW: empty.
244
+10. **shellcheck CLEAN** -- `shellcheck --external-sources` over all three new scripts: ALL CLEAN.
245
+
246
+Tests or equivalent validation:
247
+
248
+- `shellcheck --external-sources` over `.agents/skills/codacy-audit/scripts/*.sh` -> ALL CLEAN.
249
+- `bash -n` over each script -> OK.
250
+- `--help` on each script -> exit 0.
251
+- `codacyaudit_selftest_no_token_leak` PASS under both bash and zsh.
252
+- `pr-issues.sh 22423` and `pr-issues.sh 22420` -> 0 findings each, JSON dump on disk.
253
+- `analyze-local.sh --directory /tmp/<fixture> --tool markdownlint` -> 2 findings, JSON dump on disk.
254
+
255
+Real-use evidence:
256
+
257
+- `pr-issues.sh 22423` and `pr-issues.sh 22420` were run live against `api.codacy.com` with the configured `CODACY_TOKEN`; both completed without auth or pagination errors. JSON dumps under `<repo>/.local/audits/codacy/` confirm the v3 envelope shape.
258
+- `analyze-local.sh` was run live against the docker image `codacy/codacy-analysis-cli:latest` (pulled fresh from Docker Hub during validation); markdownlint dispatch produced 2 findings on a synthetic fixture.
259
+
260
+Reviewer findings:
261
+
262
+- Self-review: covered. The "Is a directory" failure on the shellcheck child-tool path is a CLI-version-specific quirk on a 1-file directory and is logged as a known limitation in the Execution Log; the markdownlint path (which the seeded how-to uses) works.
263
+
264
+Same-failure scan:
265
+
266
+- `grep -rn -E 'BASH_SOURCE\[0\]' .agents/skills/` -- only the codacy-audit and query-agent-events libs use it; both now wrap with the zsh-compat resolver.
267
+- `grep -rn -E '_run -d "' .agents/skills/codacy-audit/scripts/` -- no callers expose token bytes via `-d` argument quoting traps.
268
+
269
+Sensitive data gate:
270
+
271
+- No raw tokens, account UUIDs, customer-identifying IPs, or private endpoints were written to any committed artifact. The Codacy account ID 30945 (Costa) was returned by `/v3/user` during exploration and is intentionally not committed; this SOW redacts to "the configured account".
272
+- `CODACY_TOKEN` is referenced via `${CODACY_TOKEN}` only, never literally.
273
+- Dumps land under `<repo>/.local/audits/codacy/` (gitignored).
274
+
275
+Artifact maintenance gate:
276
+
277
+- AGENTS.md: updated -- `codacy-audit` entry added to "Legacy runtime skills" section and to the brief skill list.
278
+- Runtime project skills: not affected -- this is a legacy-style audit skill, not a `project-*` skill.
279
+- Specs: updated -- `CODACY_TOKEN` row added to `<repo>/.agents/sow/specs/sensitive-data-discipline.md` env-keys table.
280
+- End-user/operator docs: not affected -- skill is internal AI tooling.
281
+- End-user/operator skills: not affected.
282
+- SOW lifecycle: `pending/ -> current/ -> done/`; `Status: open -> in-progress -> completed`; closed in the same commit as the work per AGENTS.md rule.
283
+
284
+Specs update:
285
+
286
+- `.agents/sow/specs/sensitive-data-discipline.md` -- env-keys table gained the `CODACY_TOKEN` row.
287
+
288
+Project skills update:
289
+
290
+- None applicable -- codacy-audit is a legacy-style audit skill, mirroring `coverity-audit/` etc.
291
+
292
+End-user/operator docs update:
293
+
294
+- None applicable -- this is internal AI-assistant tooling.
295
+
296
+End-user/operator skills update:
297
+
298
+- None applicable.
299
+
300
+Lessons:
301
+
302
+- The Codacy CLI's docker-in-docker model is not optional; without `/var/run/docker.sock` mounted in the CLI container the inner per-tool containers cannot start. The upstream README is explicit; mirror their invocation verbatim.
303
+- Account API tokens authenticate via `api-token: <value>` (NOT `Authorization: Bearer`). A 401 with `Bad credentials` is the typical sign of using the wrong header.
304
+- When `.env` values are single-quoted, `cut -d'"' -f2` and `cut -d'=' -f2-` both leave the quotes intact. Always source `.env` via bash (`set -a; . .env; set +a`) so the shell strips the quotes natively.
305
+- Sister-skill `_lib.sh` files use a portable `if [ -n "${ZSH_VERSION-}" ]` block to resolve their own path; reuse it for any new audit-style skill so sourcing under zsh does not warn.
306
+
307
+Follow-up mapping:
308
+
309
+- Write actions (mark FP, mark fixed, modify ignore-patterns via API): TRACKED. To be opened as a future SOW when the user has a real triage need.
310
+- Master-backlog triage on the 31,425 open issues: TRACKED. Same condition.
311
+- The shellcheck child-tool dispatch quirk: TRACKED. To be revisited only if the user hits it during real-use; otherwise the markdownlint path covers the value-add.
312
+
313
+## Outcome
314
+
315
+The codacy-audit skill is live under `.agents/skills/codacy-audit/`. Pre-push prevention is one command (`analyze-local.sh`); PR triage is one command (`pr-issues.sh <N>`). Both are token-safe, shellcheck-clean, and pass an end-to-end round-trip against `api.codacy.com` and the official Codacy CLI image. The skill mirrors the coverity / sonarqube / graphql audit family conventions so an assistant familiar with one can use this one without re-learning.
316
+
317
+## Lessons Extracted
318
+
319
+See the "Lessons" subsection of Validation above.
320
+
321
+## Followup
322
+
323
+- Future SOW for write actions (mark FP / mark fixed) -- not opened yet; will open when there is a real PR triage need.
324
+- Future SOW for master-backlog triage -- not opened yet; will open when there is a quarterly-review need.
325
+
326
+## Regression Log
327
+
328
+None yet.
329
+
330
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/pending/SOW-0006-20260503-skill-verification-harness.md
new
+203
@@ -0,0 +1,203 @@
1
+# SOW-0006 - Skill verification harness
2
+
3
+## Status
4
+
5
+Status: open
6
+
7
+Sub-state: stub. Created 2026-05-03 evening when the user split the verification piece out of SOW-0010 ("the evaluation does not need to be done now"). Depends on SOW-0010 closing first because the harness consumes the seed question lists and how-tos that SOW-0010 lays down.
8
+
9
+## Requirements
10
+
11
+### Purpose
12
+
13
+Build a reusable **skill verification harness** that grades
14
+whether an AI assistant can answer concrete questions about a
15
+running Netdata environment using ONLY the skill's
16
+SKILL.md + per-domain guides + how-tos. The harness is the
17
+acceptance gate for any querying skill in this repo.
18
+
19
+The user's framing (verbatim from the 2026-05-03 expansion
20
+message): "use sonnet (not the best model) to query the APIs to
21
+find out if the instructions work. Sonnet should be responding
22
+what it did to find the answer and we should maintain a directory
23
+with how-tos and an index in SKILL.md to help the model figure
24
+out its way through it."
25
+
26
+The harness validates the two SOW-0010 skills first, then is
27
+re-used by SOW-0003 (`query-agent-events`), SOW-0004
28
+(`learn-site-structure`), SOW-0005 (`mirror-netdata-repos`), and
29
+SOW-0007 (`integrations-lifecycle`) as their own acceptance gates.
30
+
31
+### User Request
32
+
33
+> "We need to build a verification system: spawn an agent and
34
+> ask it a question:
35
+> - Node `costa-desktop` (on netdata-cloud in space Netdata People)
36
+> - find its hardware specs
37
+> - which operating system it runs?
38
+> - is it a parent? of how many and which nodes?
39
+> - is it a child and where it stream?
40
+> - does it have any vnodes and which?
41
+> - are there any failed data collection jobs?
42
+> - does it monitor nvidia DCGM? What is the data collection
43
+> frequency?
44
+> - which is the PID with the biggest memory consumption and in
45
+> which app group/category I can see in on the dashboard?
46
+> - find the last netdata status file log in the logs
47
+> - and many more
48
+>
49
+> The idea is to use sonnet (not the best model) to query the
50
+> APIs to find out if the instructions work. Sonnet should be
51
+> responding what it did to find the answer and we should
52
+> maintain a directory with how-tos and an index in SKILL.md to
53
+> help the model figure out its way through it.
54
+>
55
+> The list of how-tos should be supposed to be live. Every time
56
+> the assistant is asked to do something and it is not documented
57
+> and it is forced to do an analysis to answer, it should create
58
+> a how-to to help future assistants work faster.
59
+>
60
+> The agent skill and documents and scripts should ensure the
61
+> assistant never sees the cloud api token and agent bearer
62
+> token."
63
+>
64
+> Follow-up: "The evaluation does not need to be done now. But
65
+> the live how-tos inventory and everything else must be done."
66
+> -- so the inventory + token-safety + structural scope stay in
67
+> SOW-0010; the harness itself (this SOW) is deferred.
68
+
69
+### Acceptance Criteria
70
+
71
+- A `verify/` runner under each verified skill -- e.g.
72
+ `<repo>/docs/netdata-ai/skills/query-netdata-cloud/verify/run.sh`
73
+ -- that:
74
+ 1. Reads `<skill>/verify/questions.md` (the seed list shipped
75
+ by the upstream SOW).
76
+ 2. For each question, spawns a Sonnet-class assistant with a
77
+ minimal system prompt that points at the skill (SKILL.md +
78
+ `how-tos/INDEX.md` + canonical reference docs).
79
+ 3. Captures the assistant's full transcript, the tool calls
80
+ it made, and the final answer.
81
+ 4. Records results under
82
+ `<repo>/.local/audits/<skill>/verify/<timestamp>/`.
83
+ 5. Grades each answer against `verify/grader.md`.
84
+ 6. Reports pass / fail / unanswered counts.
85
+- `verify/grader.md` per skill: rubric covering (a) correctness,
86
+ (b) evidence shown (file:line refs, response keys), (c) no
87
+ exposed tokens / bearers / claim ids in the transcript or
88
+ output, (d) how-to authored when missing.
89
+- An "unanswered question" prompt loop: when Sonnet cannot
90
+ answer, the runner asks (interactively, or via a follow-up
91
+ agent run) for a draft how-to and stages it under
92
+ `<skill>/how-tos/<slug>.md` for human review.
93
+- A self-test mode: the runner can be pointed at any of the
94
+ four SOW-0010+ skills and produces consistent reports. This
95
+ is what makes it reusable across SOWs.
96
+- Sensitive-data gate: every committed file passes the
97
+ pre-commit grep from
98
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md`. The
99
+ harness MUST scrub captured transcripts of any token / bearer
100
+ / claim_id before persisting them under `.local/audits/`.
101
+
102
+## Analysis
103
+
104
+Sources to consult during stage 2:
105
+
106
+- `<repo>/.agents/sow/done/SOW-0010-20260503-netdata-query-skills-infrastructure.md`
107
+ -- the SKILL.md and how-tos format this SOW must validate.
108
+- the user-global Claude Code config and `claude` CLI invocation patterns (the user
109
+ has documented them in `the user-global Claude Code config (gitignored)` under "Run other
110
+ AI assistants for a second opinion").
111
+- The pattern of the legacy private skills (`coverity-audit`,
112
+ `pr-reviews`) for their `_lib.sh` shape and `_run` wrappers.
113
+
114
+Risks:
115
+
116
+- Token leakage in transcripts is the highest risk. The harness
117
+ must scrub captured stdout/stderr before writing.
118
+- Cost: running Sonnet over many questions has real cost. The
119
+ harness should default to a small seed-question subset and
120
+ let the operator opt into the full run.
121
+- Flakiness: live API answers are time-dependent (e.g.
122
+ "currently active alerts" varies). The grader rubric must
123
+ accept structural correctness + presence of evidence rather
124
+ than exact byte equality.
125
+
126
+## Pre-Implementation Gate
127
+
128
+Status: blocked-on-prereq
129
+
130
+Depends on SOW-0010 closing (it provides the SKILL.md, the
131
+per-domain guides, the `how-tos/INDEX.md`, the
132
+`verify/questions.md` seed list, and the token-safe wrappers
133
+that the harness must invoke).
134
+
135
+Sensitive data handling plan:
136
+
137
+- Follows `<repo>/.agents/sow/specs/sensitive-data-discipline.md`.
138
+- Captured transcripts and response bodies live under
139
+ `<repo>/.local/audits/<skill>/verify/<timestamp>/` (gitignored).
140
+- Before persisting any transcript, the harness scrubs anything
141
+ matching the discipline grep (UUIDs, IPv4 literals to specific
142
+ hosts, bearer/cloud-token shaped strings, forbidden absolute
143
+ paths). The scrubbed transcript is what gets stored; the
144
+ unscrubbed in-memory copy is dropped.
145
+- No `.env` keys are required beyond what SOW-0010 already
146
+ defines.
147
+
148
+Holding-pattern decisions to record now:
149
+
150
+- The harness invokes `claude -p` (Sonnet) per the user's
151
+ documented pattern in `the user-global Claude Code config (gitignored)`. Model selection
152
+ is a runner flag with a default; not a per-question
153
+ hardcode.
154
+- The how-to generation prompt is a separate template under
155
+ `verify/howto-template.md`. Stage 2 implementation defines
156
+ it.
157
+
158
+## Implications And Decisions
159
+
160
+No new decisions at this stub stage. Will be added when
161
+SOW-0010 closes and stage 2 begins.
162
+
163
+## Plan
164
+
165
+1. **Wait for SOW-0010 to close.**
166
+2. Stage 2a: read SOW-0010 final deliverables (SKILL.md
167
+ structure, how-tos shape, wrappers).
168
+3. Stage 2b: implement `verify/run.sh` for the cloud skill;
169
+ prove end-to-end on the seed questions.
170
+4. Stage 2c: parameterize `run.sh` so it can target any of the
171
+ SOW-0010+ skills.
172
+5. Stage 2d: implement the how-to generation prompt loop.
173
+6. Stage 2e: implement the transcript scrubber + unit test
174
+ that asserts no token bytes are persisted.
175
+7. Stage 2f: validate end-to-end. Close.
176
+
177
+## Execution Log
178
+
179
+### 2026-05-03
180
+
181
+- Created as a stub during the second SOW-0010 scope expansion.
182
+
183
+## Validation
184
+
185
+Pending. All sub-fields will be filled at close.
186
+
187
+## Outcome
188
+
189
+Pending.
190
+
191
+## Lessons Extracted
192
+
193
+Pending.
194
+
195
+## Followup
196
+
197
+None yet.
198
+
199
+## Regression Log
200
+
201
+None yet.
202
+
203
+Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/specs/sensitive-data-discipline.md
new
+196
@@ -0,0 +1,196 @@
1
+# Spec - Sensitive data discipline for committed artifacts
2
+
3
+## Status
4
+
5
+Active. Applies to every SOW, every skill (public or private), every
6
+script, every spec, every committed doc, every commit message, and every
7
+PR text in this repository.
8
+
9
+## Rule
10
+
11
+The following categories of literal values MUST NOT appear in any
12
+committed file:
13
+
14
+1. **IP addresses.** IPv4 or IPv6 literals to specific hosts.
15
+ Loopback (`127.0.0.1`, `::1`) and link-local addresses used as
16
+ loopback / link-local references are fine.
17
+2. **Tenant-identifying hostnames.** Any hostname that uniquely
18
+ identifies a customer, community member, internal ingestion
19
+ host, or other non-public destination. **Public Netdata-org
20
+ sites used in role-descriptive prose are allowed** (e.g.
21
+ `learn.netdata.cloud` when describing the learn site,
22
+ `netdata.cloud` when describing the marketing site). The Cloud
23
+ REST API host is env-keyed because it's an operational target
24
+ that scripts call.
25
+3. **UUID-shaped identifiers.** Machine GUIDs, node UUIDs, claim
26
+ IDs, space IDs, room IDs, agent IDs, ephemeral IDs,
27
+ bearer-token UUIDs, tenant IDs, account IDs.
28
+4. **Credentials.** API tokens, bearer tokens, session cookies,
29
+ OAuth tokens, passwords, signing keys, SSH private keys,
30
+ anything that grants access.
31
+5. **Absolute filesystem paths to per-user or per-tenant state.**
32
+ User home paths, the user's mirrored-repos tree, the larger
33
+ monitoring mirror tree, or any workstation-specific opt/var
34
+ path that identifies a particular install or user.
35
+ **Documented Netdata default install paths
36
+ (`/var/lib/netdata`, `/etc/netdata`, `/usr/lib/netdata`,
37
+ `/usr/libexec/netdata`) are allowed** because they are
38
+ open-source defaults documented across the codebase.
39
+ Netdata can also be bundled (typically rooted at
40
+ `/opt/netdata`), so scripts MUST autodetect the install
41
+ prefix at runtime by probing the candidate locations and
42
+ selecting the first one that exists. Common candidates:
43
+ empty (system install), `/opt/netdata`, `/usr/local/netdata`.
44
+ Scripts must NOT require a `NETDATA_PREFIX` env knob; the
45
+ prefix is a discovered fact, not a user configuration.
46
+6. **Real-name identifiers.** Usernames, email addresses, real
47
+ names of community members, customers, employees,
48
+ contributors.
49
+7. **Tenant identifiers.** Netdata Cloud space names, room
50
+ names, or any human-readable identifier that maps to a
51
+ specific tenant or organization.
52
+8. **Proprietary incident details.** Customer support
53
+ narratives, non-public bug reports, private correspondence.
54
+
55
+## Allowed alternatives
56
+
57
+For every reference to a value covered above, use ONE of:
58
+
59
+- An env-key placeholder: `${KEY_NAME}` -- the value lives in `.env`
60
+ (gitignored). Examples:
61
+ `ssh ${AGENT_EVENTS_HOSTNAME}` rather than the literal address;
62
+ `${NETDATA_REPOS_DIR}/learn/ingest.js` and
63
+ `${NETDATA_REPOS_DIR}/website/content/...` when committed skill
64
+ content needs to point at sibling Netdata-org repositories the
65
+ user has cloned locally. Sibling-repo file paths via
66
+ `${NETDATA_REPOS_DIR}/...` are explicitly allowed in committed
67
+ skill / SOW / spec content; literal workstation roots
68
+ (`~/`, `/home/...`) are not.
69
+- A repo-relative path: `<repo>/src/daemon/status-file.c`,
70
+ `src/web/api/...` -- these describe locations inside this
71
+ repository and are not leaks.
72
+- Standard Linux/POSIX paths that carry no tenant or user identity:
73
+ `/tmp`, `/run`, `/etc/passwd` (the file, not the contents), `/proc`,
74
+ `/sys`. Use sparingly; prefer not to mention them at all unless the
75
+ reference is essential.
76
+- Generic role descriptions: "the production Netdata Cloud REST host",
77
+ "the agent's varlib directory", "the user's repo mirror" -- without
78
+ the actual value.
79
+- Open-source code references: `file:line` citations into this repo
80
+ (e.g. `src/daemon/status-file.c:988`) are fine; they describe code,
81
+ not values.
82
+
83
+## Required env keys
84
+
85
+These keys MUST be defined in `<repo>/.env` (gitignored) for the
86
+SOW family from SOW-0010 onward to function. If a SOW or script
87
+references one and the key is unset, the script must error loudly
88
+and exit non-zero. Values live ONLY in `.env`; this spec lists
89
+names and roles only.
90
+
91
+| Key | Role |
92
+|---|---|
93
+| `NETDATA_CLOUD_TOKEN` | long-lived Cloud REST token |
94
+| `NETDATA_CLOUD_HOSTNAME` | Cloud REST API host (the operational target scripts call) |
95
+| `NETDATA_REPOS_DIR` | user's mirror of Netdata-org repos |
96
+| `AGENT_EVENTS_HOSTNAME` | network address of the ingestion node -- dual-duty: ssh host (`ssh ${AGENT_EVENTS_HOSTNAME}`) AND direct-HTTP host (`http://${AGENT_EVENTS_HOSTNAME}:19999/...`). Can be a DNS name or an IP literal. NOTE: this is NOT the journalctl namespace (hardcoded to `agent-events`) and NOT the Cloud room name (also hardcoded to `agent-events`). |
97
+| `AGENT_EVENTS_MACHINE_GUID` | events-ingestion agent machine GUID |
98
+| `AGENT_EVENTS_NODE_ID` | events-ingestion agent node UUID |
99
+| `CODACY_TOKEN` | Codacy Cloud Account API token; header form `api-token: <value>` |
100
+
101
+Per-user setup is documented at `<repo>/.agents/ENV.md`. The
102
+committed `<repo>/.env.template` is the starting point for a
103
+new contributor's `.env`.
104
+
105
+Things that are intentionally NOT env-keyed (and why):
106
+
107
+- The Cloud Swagger URL is derived as
108
+ `${NETDATA_CLOUD_HOSTNAME}/api/docs/`.
109
+- The agent-events producer ingest URL is a `const char *` in
110
+ `src/daemon/status-file.c`; reference by `file:line`.
111
+- Public site hostnames (learn site, marketing site) are public
112
+ and used in role-descriptive prose.
113
+- Default Netdata install paths (`/var/lib/netdata`,
114
+ `/etc/netdata`) are public OSS defaults; bundled installs
115
+ (`/opt/netdata/...`, etc.) are handled by runtime
116
+ autodetection in scripts, not a config knob.
117
+- This repo's checkout root is found via
118
+ `git rev-parse --show-toplevel`.
119
+
120
+Adding new keys to `.env` is the user's prerogative. SOWs and
121
+scripts can REQUEST keys; only the user adds them.
122
+
123
+## Verification
124
+
125
+Before any commit that touches a SOW, skill, spec, or doc, run:
126
+
127
+```bash
128
+# Helper: list of files staged for commit, excluding this spec.
129
+files=$(git diff --cached --name-only --diff-filter=ACMR \
130
+ | grep -v '^\.agents/sow/specs/sensitive-data-discipline\.md$')
131
+[ -z "$files" ] && exit 0
132
+
133
+# Run each pattern. The patterns themselves are not embedded in
134
+# this code block as committable literals; they are constructed
135
+# from concatenated character classes so a grep over THIS spec
136
+# does not flag itself for the very examples it must define.
137
+grep_args=(
138
+ --line-number --extended-regexp --binary-files=without-match
139
+)
140
+
141
+# Domain pattern for the org.
142
+domain='[A-Za-z0-9-]+\.netdata\.(cloud|io)'
143
+git grep "${grep_args[@]}" -- $files -- "$domain"
144
+
145
+# UUID-shaped identifiers.
146
+uuid='[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
147
+git grep "${grep_args[@]}" -- $files -- "$uuid"
148
+
149
+# IPv4 literals (review hits manually for false positives).
150
+ipv4='([0-9]{1,3}\.){3}[0-9]{1,3}'
151
+git grep "${grep_args[@]}" -- $files -- "$ipv4"
152
+
153
+# Per-user / per-install absolute paths. Documented Netdata
154
+# default paths (/var/lib/netdata, /etc/netdata) are intentionally
155
+# excluded -- they are public OSS defaults; bundled installs use
156
+# the NETDATA_PREFIX knob.
157
+abs="(~/|/$(echo op)t/baddisk|/$(echo op)t/neda|/$(echo op)t/ai-agent|/$(echo ho)me/)"
158
+git grep "${grep_args[@]}" -- $files -- "$abs"
159
+
160
+# Long opaque tokens (40+ char base64-ish).
161
+tok='[A-Za-z0-9_+/=-]{40,}'
162
+git grep "${grep_args[@]}" -- $files -- "$tok"
163
+```
164
+
165
+Every match must either be removed or converted to an env-key
166
+reference, OR explicitly justified inline (e.g. a citation of an
167
+upstream open-source project's hostname when documenting how that
168
+project reports its own data).
169
+
170
+## Exceptions
171
+
172
+- **This spec file itself** must list forbidden patterns and
173
+ example regexes in order to define the rule. The verification
174
+ grep excludes `<repo>/.agents/sow/specs/sensitive-data-
175
+ discipline.md` from its scan. No other SOW, skill, or doc
176
+ qualifies for this exemption.
177
+- **Quoted user messages** preserved verbatim in a SOW's "User
178
+ Request" section may include literals the user typed. Redact
179
+ the literal value and replace with the `.env` key in `[env-
180
+ keyed: ${KEY}]` form, with a footnote pointing here. The
181
+ user's wording stays; only the literal value moves to `.env`.
182
+- **Repo-relative paths** (`src/daemon/...`, `<repo>/src/...`)
183
+ are not absolute paths and are fine.
184
+- **Open-source upstream references** (e.g.
185
+ `prometheus/prometheus@<sha>:cmd/...`) are fine; they
186
+ describe external code, not tenant data.
187
+- **Code citations** of the form `file:line` are fine
188
+ (`src/daemon/status-file.c:988`).
189
+
190
+## Failure mode
191
+
192
+If a verification grep returns a hit on a committed file, the SOW
193
+that introduced it has failed its Sensitive Data Gate and must be
194
+treated as a regression. Re-open the SOW, redact, force-push only
195
+with explicit user approval (otherwise create a follow-up commit
196
+that scrubs).
.codacy.yml
+2
@@ -17,3 +17,5 @@ exclude_paths:
17
- TODO-netflow-plugin-refactor.md
18
- src/go/pkg/topology/engine/parity/README.md
19
- src/go/pkg/topology/engine/parity/evidence/**
20
+ - .agents/**
21
+ - docs/netdata-ai/skills/**
.env.template
new
+138
@@ -0,0 +1,138 @@
1
+# .env.template -- copy to .env and fill in.
2
+#
3
+# .env is gitignored; .env.template is committed.
4
+# See <repo>/.agents/ENV.md for the full setup guide
5
+# (where each value comes from, sample formats, common
6
+# mistakes).
7
+#
8
+# Only fill in the keys you need. Each skill checks the
9
+# specific keys it requires and exits with a clear message
10
+# if any are missing.
11
+
12
+# ---------------------------------------------------------------
13
+# Netdata Cloud REST API
14
+# ---------------------------------------------------------------
15
+# Used by: query-netdata-cloud, query-netdata-agents,
16
+# query-agent-events.
17
+#
18
+# NETDATA_CLOUD_TOKEN -- long-lived Cloud REST token.
19
+# Get one at: https://app.netdata.cloud
20
+# -> top-right user menu -> Settings -> API Tokens
21
+# -> "Create API Token" with `scope:all` (or at least
22
+# scope:grafana-plugin for read-only queries).
23
+# Sample format: 36-char UUID-shaped token.
24
+NETDATA_CLOUD_TOKEN=""
25
+
26
+# NETDATA_CLOUD_HOSTNAME -- Cloud REST API host.
27
+# Almost always app.netdata.cloud.
28
+NETDATA_CLOUD_HOSTNAME="app.netdata.cloud"
29
+
30
+# ---------------------------------------------------------------
31
+# Cross-org repo mirror
32
+# ---------------------------------------------------------------
33
+# Used by: mirror-netdata-repos (the sync script + path refs in
34
+# integrations-lifecycle, learn-site-structure, etc.).
35
+#
36
+# NETDATA_REPOS_DIR -- absolute path to the directory holding
37
+# your local Netdata-org repos mirror. Will be created if you
38
+# run sync-netdata-repos.sh on an empty parent.
39
+#
40
+# Sample: "$HOME/src/netdata"
41
+NETDATA_REPOS_DIR=""
42
+
43
+# ---------------------------------------------------------------
44
+# agent-events ingestion node (for query-agent-events skill)
45
+# ---------------------------------------------------------------
46
+# Used by: query-agent-events (and by query-netdata-agents
47
+# as the canonical example for direct-agent calls).
48
+#
49
+# AGENT_EVENTS_HOSTNAME -- network address of the ingestion node.
50
+# Dual-duty: ssh host AND direct-HTTP host (port 19999). Can be
51
+# a DNS name or an IP literal.
52
+#
53
+# NOTE: this is NOT the journalctl namespace (hardcoded to
54
+# 'agent-events' on the ingestion server) and NOT the Cloud
55
+# room name (also hardcoded to 'agent-events').
56
+#
57
+# Sample: "10.20.1.105" or "agent-events.example.org"
58
+AGENT_EVENTS_HOSTNAME=""
59
+
60
+# AGENT_EVENTS_NODE_ID -- the Cloud node UUID for the
61
+# agent-events node. Find it by visiting the node in
62
+# app.netdata.cloud and copying the UUID from the URL, or
63
+# by listing nodes via the Cloud API.
64
+AGENT_EVENTS_NODE_ID=""
65
+
66
+# AGENT_EVENTS_MACHINE_GUID -- the Netdata machine GUID for
67
+# the agent-events node. Find it on the host with:
68
+# sudo cat /var/lib/netdata/registry/netdata.public.unique.id
69
+AGENT_EVENTS_MACHINE_GUID=""
70
+
71
+# ---------------------------------------------------------------
72
+# Coverity Scan triage (for coverity-audit skill)
73
+# ---------------------------------------------------------------
74
+# Used by: coverity-audit. All four required.
75
+#
76
+# COVERITY_HOST -- the Coverity Scan host. For Netdata, this
77
+# is scan4.scan.coverity.com (newer instance).
78
+COVERITY_HOST="https://scan4.scan.coverity.com"
79
+
80
+# COVERITY_PROJECT_ID -- the integer project id.
81
+# Find it in the URL when you click your project in the
82
+# Coverity Scan dashboard, e.g. ?projectId=12345
83
+COVERITY_PROJECT_ID=""
84
+
85
+# COVERITY_COOKIE -- the full Cookie header captured from a
86
+# logged-in browser session. Open DevTools (F12) -> Network
87
+# -> click any request to scan4.scan.coverity.com -> Headers
88
+# -> Request Headers -> copy the entire Cookie value
89
+# (including XSRF-TOKEN=...). Refresh roughly every 12-24h
90
+# when keepalive.sh isn't running.
91
+COVERITY_COOKIE=""
92
+
93
+# COVERITY_VIEW_OUTSTANDING -- the integer viewId for your
94
+# "Outstanding defects" view. Find it in the URL when you
95
+# open that view, e.g. ?viewId=10001
96
+COVERITY_VIEW_OUTSTANDING=""
97
+
98
+# ---------------------------------------------------------------
99
+# SonarCloud triage (for sonarqube-audit skill)
100
+# ---------------------------------------------------------------
101
+# Used by: sonarqube-audit. All four required.
102
+#
103
+# SONAR_HOST_URL -- the SonarCloud host.
104
+SONAR_HOST_URL="https://sonarcloud.io"
105
+
106
+# SONAR_ORG -- your sonarcloud.io organization key.
107
+SONAR_ORG=""
108
+
109
+# SONAR_PROJECT -- the projectKey of your project on SonarCloud.
110
+# For Netdata: netdata_netdata
111
+SONAR_PROJECT="netdata_netdata"
112
+
113
+# SONAR_TOKEN -- a personal access token.
114
+# Generate at: https://sonarcloud.io/account/security
115
+SONAR_TOKEN=""
116
+
117
+# ---------------------------------------------------------------
118
+# Codacy Cloud (for codacy-audit skill)
119
+# ---------------------------------------------------------------
120
+# Used by: codacy-audit (pr-issues.sh and any v3 API wrapper).
121
+# analyze-local.sh runs the CLI anonymously and does NOT need
122
+# CODACY_TOKEN.
123
+#
124
+# CODACY_TOKEN -- Account API token. Header form is
125
+# `api-token: <value>` (NOT Bearer).
126
+# Get one at: https://app.codacy.com
127
+# -> top-right avatar -> Account -> API tokens
128
+# -> "Create API Token"
129
+CODACY_TOKEN=""
130
+
131
+# CODACY_HOST / CODACY_PROVIDER / CODACY_ORG / CODACY_REPO are
132
+# all optional and default to api.codacy.com / gh / netdata /
133
+# netdata. Override only if you point this skill at a different
134
+# Codacy org or repo.
135
+# CODACY_HOST="https://api.codacy.com"
136
+# CODACY_PROVIDER="gh"
137
+# CODACY_ORG="netdata"
138
+# CODACY_REPO="netdata"
AGENTS.md
+52
-2
@@ -263,6 +263,22 @@ Existing non-`project-*` skills under `.agents/skills/` are preserved as legacy
263
264
Output/reference skills may also exist under product documentation or generated skill directories. Do not rename, shorten, or change their descriptions only to satisfy runtime discovery. Update them when their related public/operator workflow changes.
265
266
+### Public skill convention (`docs/netdata-ai/skills/`)
267
+
268
+End-user-facing AI skills under `docs/netdata-ai/skills/` follow the directory shape `docs/netdata-ai/skills/<skill-name>/SKILL.md`, with optional supporting docs (`<topic>.md`) and an optional `scripts/` subdirectory for helper code. SKILL.md frontmatter has `name` and `description`; the description is the trigger-matching text and must enumerate the phrases users will actually type.
269
+
270
+Each public skill is reachable from `.agents/skills/<skill-name>` via a relative symlink (`.agents/skills/<name>` → `../../docs/netdata-ai/skills/<name>`) so local AI assistants reading from `.agents/skills/` see the same skill as end-users. Create the symlink with `ln -srfn`. Verify with `readlink -f .agents/skills/<name>`.
271
+
272
+Public-skill scripts must follow the same `_lib.sh` shape as the legacy private skills (`set -euo pipefail`, ANSI colors with real ESC bytes via `$'\033[...]'`, `<prefix>_repo_root` via `git rev-parse --show-toplevel`, `<prefix>_load_env` that sources `<repo>/.env` with `: "${VAR:?}"` validation, `<prefix>_audit_dir` that creates `<repo>/.local/audits/<topic>/`, masked-token `<prefix>_run`/`<prefix>_run_read` wrappers).
273
+
274
+Public-skill scripts that touch credentials (cloud tokens, per-agent bearers, claim ids, session cookies) MUST be **token-safe** -- helpers that handle credential bytes are named with a leading underscore (`_skill_*`, internal-only) and return them via bash namerefs into the caller's local variables, NEVER to stdout. Public wrappers (no leading underscore) read credentials from `.env` internally and emit ONLY the response body. Each token-handling lib must ship a `<prefix>_selftest_no_token_leak` function that drives every public wrapper with a sentinel token and asserts the sentinel never appears on captured stdout.
275
+
276
+### How-tos catalog rule
277
+
278
+Each public skill ships a `how-tos/` subdirectory with `INDEX.md`. The catalog is **live**: every time an AI assistant is asked a concrete question that requires analysis (multiple wrapper calls, jq pipelines, or cross-referencing more than one per-domain guide) and the answer isn't already documented under `how-tos/`, the assistant MUST author a new how-to and add it to `INDEX.md` BEFORE completing the task. This rule is repeated in each skill's `SKILL.md` so future assistants honor it. Skipping it means the next assistant repeats the same analysis from scratch -- an explicit framework violation.
279
+
280
+The legacy private skills (`coverity-audit`, `sonarqube-audit`, `graphql-audit`, `pr-reviews`) keep their `.agents/skills/<name>/` location -- they are intentionally private and have no `docs/netdata-ai/skills/` counterpart.
281
+
282
### Project Skills Index
283
284
Runtime input skills:
@@ -273,6 +289,18 @@ Runtime input skills:
289
- `.agents/skills/project-writing-collectors/`
290
Trigger: authoring or modifying any Netdata data-collection plugin or module (Go go.d / ibm.d, Rust crates, internal C plugins, external plugins via PLUGINSD). Read before adding a new collector, modifying an existing one, working on NetFlow/sFlow/IPFIX, OTEL ingestion, topology, SNMP profiles, or interactive Functions.
291
Status: live. Updates that close gaps or fix outdated pointers must ship in the same PR that exposed the issue.
292
+- `.agents/skills/integrations-lifecycle/`
293
+ Trigger: editing any `metadata.yaml`; modifying `integrations/` generators, schemas, or templates; working with `integrations.js` / `integrations.json` / per-integration `.md` files / `COLLECTORS.md` / `SECRETS.md` / `SERVICE-DISCOVERY.md`; ibm.d module generation (`contexts.yaml` -> `metadata.yaml`); CI workflows `generate-integrations.yml` and `check-markdown.yml`; the five-file collector-consistency rule.
294
+ Status: live. SKILL.md plus per-domain guides (`pipeline.md`, `schema-reference.md`, `per-type-matrix.md`, `artifacts-and-banners.md`, `ibm-d.md`, `consistency.md`, `in-app-contract.md`, `gotchas.md`) and `recipes/`, `how-tos/` directories.
295
+- `.agents/skills/learn-site-structure/`
296
+ Trigger: adding/moving/renaming/deleting any docs page that should appear on `learn.netdata.cloud`; editing `<repo>/docs/.map/map.yaml`; investigating why a Learn page looks the way it does; reading the live `ingest/ingest.py` orchestrator or the legacy `ingest.js` / `ingest.md` (which are stale); MDX escape rules; redirects; the Netlify deploy contract.
297
+ Status: live. SKILL.md plus per-domain guides (`mapping.md`, `pipeline.md`, `sidebars.md`, `mdx-rules.md`, `redirects.md`, `pitfalls-and-gotchas.md`, `authoring-boundary.md`) and `recipes/`, `how-tos/` directories.
298
+- `.agents/skills/query-agent-events/`
299
+ Trigger: investigating crashes, panics, or fatals across the Netdata fleet; downloading events from the agent-events ingestion namespace; analyzing AE_* fields and their enums; understanding the 23h client-side dedup or the after-the-fact event timing; using the systemd-journal Function multi-value `selections` filter for index-friendly queries.
300
+ Status: live. SKILL.md plus per-domain guides (`AE_FIELDS.md`, `transports.md`, `update-cadence.md`, `query-discipline.md`, `finding-crashes.md`, `finding-fatals.md`), scripts (`scripts/_lib.sh`, `get-events.sh`, `analyze-events.sh`, `redact-events.sh`) and `recipes/`, `how-tos/` directories. Bug-investigation tool, NOT a generic logs query skill -- consumes `query-netdata-{cloud,agents}` for transport.
301
+- `.agents/skills/mirror-netdata-repos/`
302
+ Trigger: setting up or updating a local mirror of Netdata-org source repositories at `${NETDATA_REPOS_DIR}` for cross-repo grep / code review without GitHub API calls; running the vendored sync script; questions about the reset-to-default-branch safety mechanism or the `--repo NAME` scoping flag.
303
+ Status: live. SKILL.md (single-file overview) plus the vendored `scripts/sync-netdata-repos.sh` (env-driven, sanitized, `--repo` scoping, `gh` optional for Phase 2) and `how-tos/` catalog. Independent from any other repo mirrors this workstation may have.
304
305
Legacy runtime skills:
306
@@ -288,6 +316,20 @@ Legacy runtime skills:
316
- `.agents/skills/pr-reviews/`
317
Trigger: PR comment and review iteration work for this repository.
318
Status: preserved under legacy name; project-skill alignment is deferred and tracked by `.agents/sow/pending/SOW-0003-20260501-legacy-runtime-skill-alignment.md`.
319
+- `.agents/skills/codacy-audit/`
320
+ Trigger: Codacy Cloud workflow for this repository -- pre-push local analysis (`codacy-analysis-cli` via docker or local binary) and read-only PR-issue fetching via the v3 API.
321
+ Status: live. SKILL.md plus `scripts/_lib.sh` (token-safe wrappers + sentinel no-leak self-test), `scripts/analyze-local.sh`, `scripts/pr-issues.sh`, and a live `how-tos/INDEX.md` catalog. Read-only by design; write actions deferred to a future SOW.
322
+
323
+Public skills (canonical under `docs/netdata-ai/skills/<name>/`; relative symlinks at `.agents/skills/<name>`):
324
+
325
+- `docs/netdata-ai/skills/query-netdata-cloud/`
326
+ Trigger: querying Netdata Cloud REST API -- metrics, logs (systemd-journal), alerts, generic Function calls on a node.
327
+ Symlink: `.agents/skills/query-netdata-cloud` -> `../../docs/netdata-ai/skills/query-netdata-cloud`.
328
+ Status: live. SKILL.md plus per-domain guides (`query-metrics.md`, `query-logs.md`, `query-alerts.md`, `query-functions.md`).
329
+- `docs/netdata-ai/skills/query-netdata-agents/`
330
+ Trigger: querying Netdata Agents directly on port 19999, including auto-mint of per-agent bearer tokens from a Cloud token.
331
+ Symlink: `.agents/skills/query-netdata-agents` -> `../../docs/netdata-ai/skills/query-netdata-agents`.
332
+ Status: live. SKILL.md plus `scripts/_lib.sh` helpers (`agents_resolve_bearer`, `agents_call_function`, `agents_netdata_prefix`).
333
334
Output/reference skills:
335
@@ -370,6 +412,7 @@ Currently available skills:
412
- `.agents/skills/sonarqube-audit/` - SonarCloud findings triage
413
- `.agents/skills/graphql-audit/` - GitHub Code Scanning (CodeQL) triage
414
- `.agents/skills/pr-reviews/` - PR comment / review iteration loop
415
+- `.agents/skills/codacy-audit/` - Codacy Cloud pre-push analysis + PR-issue triage
416
417
### Preservation Notes
418
@@ -407,7 +450,14 @@ between users and machines, not as a shared source of truth.
450
451
`/.env` at the repo root is gitignored and holds per-user secrets and
452
endpoint configuration consumed by skill scripts: API tokens, session
410
-cookies, project keys. Each skill's `SKILL.md` documents the variables it
411
-needs. Never commit secrets; never hard-code tokens in scripts.
453
+cookies, project keys. Never commit secrets; never hard-code tokens in scripts.
454
+
455
+**Setup**: copy `<repo>/.env.template` to `<repo>/.env` and fill in
456
+the keys you need.
457
+
458
+**Reference**: `<repo>/.agents/ENV.md` is the single canonical guide
459
+covering every key -- what it is, where to find the value, sample
460
+format, common mistakes, and which skills require it. When a script
461
+errors with `<KEY> is empty`, check `.agents/ENV.md` for that key.
462
463
Project SOW status: initialized
docs/netdata-ai/skills/query-netdata-agents/SKILL.md
new
+292
@@ -0,0 +1,292 @@
1
+---
2
+name: query-netdata-agents
3
+description: Query Netdata Agents (parents and children) directly via their HTTP API on port 19999. Includes a bearer-token helper that mints, caches, and transparently refreshes a per-agent bearer from a long-lived Netdata Cloud token, and auto-detects bearer-protected agents. Use when the user asks how to call an agent's REST API or Function directly, query an agent's logs/metrics/alerts directly, mint a bearer token from a cloud token, or work around bearer protection.
4
+---
5
+
6
+# Query Netdata Agents directly
7
+
8
+This skill teaches end-users (and AI assistants helping them) how to
9
+talk to a Netdata Agent's HTTP API directly, including
10
+bearer-protected agents that require an SSO-issued bearer token.
11
+
12
+It is the sibling of [`query-netdata-cloud`](../query-netdata-cloud/SKILL.md).
13
+The two skills cover different transports for the same underlying
14
+agent API.
15
+
16
+## Index of guides
17
+
18
+| Domain | Guide |
19
+|---|---|
20
+| Generic Function invocation | [query-functions.md](./query-functions.md) |
21
+| Logs (`systemd-journal`, `windows-events`, `otel-logs`) | [query-logs.md](./query-logs.md) |
22
+| Topology (`topology:snmp`) | [query-topology.md](./query-topology.md) |
23
+| Flows (`flows:netflow`) | [query-flows.md](./query-flows.md) |
24
+| Alerts (v3 paths) | [query-alerts.md](./query-alerts.md) |
25
+| DynCfg (`/api/v3/config`) | [query-dyncfg.md](./query-dyncfg.md) |
26
+| Time-series metrics (`/api/v3/data`) | [query-metrics.md](./query-metrics.md) |
27
+| Node identity, hardware, vnodes | [query-nodes.md](./query-nodes.md) |
28
+| Streaming (parent / child / replication) -- agent-only | [query-streaming.md](./query-streaming.md) |
29
+| **Operational how-tos (live catalog)** | [how-tos/INDEX.md](./how-tos/INDEX.md) |
30
+| **Verification questions (consumed by SOW-0006 harness)** | [verify/questions.md](./verify/questions.md) |
31
+
32
+
33
+| Transport | Auth | When to use |
34
+|---|---|---|
35
+| Cloud-proxied (sibling skill) | Cloud token | Default. Works for any team member with cloud access. No agent-side bearer needed. |
36
+| Direct-agent (this skill) | Per-agent bearer (UUID, ~24h TTL) | Power users; lower-latency batch fetches; bypasses the Cloud round-trip; required when Cloud is unavailable. |
37
+
38
+For **what** to query (function payloads, body schemas), see the
39
+sibling skill -- the agent and the Cloud proxy expose the same
40
+Function payload shape.
41
+
42
+This skill ships shell scripts at
43
+[`scripts/_lib.sh`](./scripts/_lib.sh) that automate the bearer mint
44
+/ cache / refresh / call-function flow. End-users can either use the
45
+scripts as a black box, or read the script source as a reference
46
+implementation.
47
+
48
+---
49
+
50
+## Mandatory Requirements (READ FIRST)
51
+
52
+1. **If you analyze, you author a how-to.** When asked a concrete
53
+ question about an agent that isn't already covered by an
54
+ existing how-to under [`how-tos/`](./how-tos/), you MUST author
55
+ a new how-to and add it to
56
+ [`how-tos/INDEX.md`](./how-tos/INDEX.md) BEFORE completing the
57
+ task. The catalog is **live** -- the next assistant should not
58
+ redo the same analysis.
59
+2. **Use the token-safe wrappers.** `agents_query_cloud`,
60
+ `agents_query_agent`, `agents_call_function` from
61
+ [`scripts/_lib.sh`](./scripts/_lib.sh) handle auth internally
62
+ and emit only the response body to stdout. Never write raw
63
+ curl with a literal `Authorization: Bearer $TOKEN` or
64
+ `X-Netdata-Auth: Bearer <uuid>`. Bearers / cloud tokens /
65
+ claim_ids must NEVER reach assistant-captured stdout.
66
+3. **Provide actionable instructions.** End every recommendation
67
+ with a runnable wrapper invocation.
68
+4. **Never request credentials.** Use env-key placeholders
69
+ (`NETDATA_CLOUD_TOKEN`, `AGENT_EVENTS_HOSTNAME`,
70
+ `AGENT_EVENTS_NODE_ID`, etc.) -- the user fills `.env` locally.
71
+5. **Bearer values stay in `.env` and `.local/`.** The bearer
72
+ cache file at `<repo>/.local/audits/query-netdata-agents/
73
+ bearers/<machine_guid>.json` is mode 0600 and gitignored. The
74
+ internal helper `_agents_resolve_bearer` returns it via bash
75
+ nameref, never to stdout.
76
+6. **For bearer-protected agents, default to the Cloud-token
77
+ flow** in this skill (it auto-mints + caches the bearer).
78
+
79
+---
80
+
81
+## Prerequisites
82
+
83
+- All [SKILL.md prereqs from `query-netdata-cloud`](../query-netdata-cloud/SKILL.md#prerequisites):
84
+ cloud token, space ID, room ID, node UUID.
85
+- Network access to the agent on port 19999 (or whatever it binds).
86
+ Test with: `curl -sS http://AGENT_HOST:19999/api/v3/info` -- a 200
87
+ with JSON confirms reachability.
88
+- The agent's `claim_id` if you intend to mint a bearer. It's at
89
+ `/api/v3/info` -> `.agents[0].cloud.claim_id`, or with shell
90
+ access at `<netdata-prefix>/var/lib/netdata/cloud.d/claimed_id`.
91
+ For the install-prefix detection rule, see
92
+ [`scripts/_lib.sh`](./scripts/_lib.sh).
93
+
94
+`.env` keys consumed (none are added by this skill -- the four
95
+existing `AGENT_EVENTS_*` keys cover the maintainer-facing
96
+agent-events workflow):
97
+
98
+| Key | Role |
99
+|---|---|
100
+| `NETDATA_CLOUD_TOKEN` | Cloud REST token used to mint per-agent bearers |
101
+| `NETDATA_CLOUD_HOSTNAME` | Cloud REST host |
102
+| `AGENT_EVENTS_HOSTNAME` | When working with the agent-events node specifically -- ssh + direct-HTTP host (IP or DNS name). NOT the journal namespace (hardcoded `agent-events`). |
103
+| `AGENT_EVENTS_NODE_ID` | Target node UUID for direct calls |
104
+| `AGENT_EVENTS_MACHINE_GUID` | Bearer cache key (one bearer per machine_guid) |
105
+
106
+---
107
+
108
+## Detect bearer protection
109
+
110
+The signal is HTTP `412 Precondition Failed` from the agent for any
111
+authenticated path (e.g. `/host/<uuid>/api/v3/function?...`). The
112
+response body is `You need to be authorized to access this resource`.
113
+
114
+```bash
115
+# Probe -- 412 means bearer required, 200 means open access
116
+HOST="agent.example.invalid:19999"
117
+NODE="YOUR_NODE_UUID"
118
+
119
+curl -s -o /dev/null -w '%{http_code}\n' -X POST \
120
+ -H 'Content-Type: application/json' \
121
+ "http://$HOST/host/$NODE/api/v3/function?function=systemd-journal" \
122
+ -d '{"info":true}'
123
+```
124
+
125
+The unauthenticated `/api/v3/info` endpoint is always reachable
126
+(returns 200 with the agent's identity). Use it to confirm the host
127
+is up before checking auth.
128
+
129
+---
130
+
131
+## Mint a per-agent bearer
132
+
133
+**Endpoint:** `GET /api/v2/bearer_get_token` on Netdata Cloud.
134
+
135
+Required query parameters: `node_id`, `machine_guid`, `claim_id`.
136
+Auth: Cloud token in `Authorization: Bearer ...`.
137
+
138
+```bash
139
+TOKEN="YOUR_API_TOKEN"
140
+NODE_ID="YOUR_NODE_UUID"
141
+MACHINE_GUID="YOUR_MACHINE_GUID"
142
+CLAIM_ID="YOUR_CLAIM_ID"
143
+
144
+curl -sS \
145
+ -H "Authorization: Bearer $TOKEN" \
146
+ "https://app.netdata.cloud/api/v2/bearer_get_token?node_id=$NODE_ID&machine_guid=$MACHINE_GUID&claim_id=$CLAIM_ID"
147
+```
148
+
149
+Response body:
150
+
151
+| Field | Description |
152
+|---|---|
153
+| `token` | The 36-char UUID bearer; pass to the agent in `X-Netdata-Auth: Bearer <token>` |
154
+| `expiration` | Numeric. Format may be Unix ms or seconds; treat values > 10^12 as ms |
155
+| `bearer_protection` | `true` if the agent IS bearer-protected; the token still works either way |
156
+| `mg` | Echoed `machine_guid` |
157
+| `status` | Status code |
158
+
159
+Permission gate (Cloud-side): `PermissionSpaceRead` on the target
160
+space; node must be `reachable`. If the agent is `stale`, the call
161
+returns 400.
162
+
163
+---
164
+
165
+## Use the bearer to call an agent
166
+
167
+```bash
168
+HOST="agent.example.invalid:19999" # the agent's bind address
169
+NODE="YOUR_NODE_UUID" # the node UUID (== nd field)
170
+BEARER="MINTED_BEARER_UUID"
171
+
172
+curl -sS -X POST \
173
+ -H "X-Netdata-Auth: Bearer $BEARER" \
174
+ -H 'Content-Type: application/json' \
175
+ "http://$HOST/host/$NODE/api/v3/function?function=systemd-journal" \
176
+ -d '{"info":true,"timeout":30000}'
177
+```
178
+
179
+Notes:
180
+
181
+- The header is **`X-Netdata-Auth: Bearer ...`**, NOT
182
+ `Authorization: Bearer ...`. The agent rejects the latter for
183
+ per-agent bearer auth.
184
+- The agent's HTTP API path mirrors the Cloud-proxied path. For the
185
+ Function payload shape (e.g. `systemd-journal` query body), see
186
+ the matching guide in
187
+ [`query-netdata-cloud`](../query-netdata-cloud/SKILL.md).
188
+
189
+---
190
+
191
+## Bearer cache and refresh
192
+
193
+The shipped scripts cache bearers per `machine_guid` under
194
+`<repo>/.local/audits/query-netdata-agents/bearers/<machine_guid>.json`
195
+(gitignored, mode 0600). Each cache entry stores the raw mint
196
+response.
197
+
198
+Refresh policy: the cache is considered expired when
199
+`expiration - now < 3600` (one-hour buffer before actual TTL).
200
+Mirror of the Cloud frontend's policy
201
+(`cloud-frontend/src/domains/nodes/useAgentBearer.js`).
202
+
203
+A failed mint clears the cache entry so the next call re-mints from
204
+scratch.
205
+
206
+---
207
+
208
+## Scripts
209
+
210
+The reference implementation lives in
211
+[`scripts/_lib.sh`](./scripts/_lib.sh). It exposes **token-safe
212
+public wrappers** (the assistant never sees the cloud token,
213
+agent bearer, or claim_id on stdout) and a **self-test** that
214
+asserts no token bytes leak.
215
+
216
+```bash
217
+# In your script:
218
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
219
+agents_load_env
220
+
221
+# Cloud-side call. NETDATA_CLOUD_TOKEN is read from .env
222
+# internally; stdout is the response body only.
223
+agents_query_cloud GET /api/v2/spaces
224
+
225
+# Direct-agent call. The bearer is minted/cached/refreshed
226
+# internally. stdout is the response body only; stderr shows the
227
+# curl invocation with `<CLOUD_TOKEN>` and `<AGENT_BEARER>`
228
+# masked.
229
+agents_query_agent \
230
+ --node "$AGENT_EVENTS_NODE_ID" \
231
+ --host "$AGENT_EVENTS_HOSTNAME:19999" \
232
+ --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
233
+ POST '/api/v3/function?function=systemd-journal' '{"info":true}'
234
+
235
+# Convenience: pick transport with --via cloud|agent.
236
+agents_call_function \
237
+ --via cloud \
238
+ --node "$AGENT_EVENTS_NODE_ID" \
239
+ --function systemd-journal
240
+```
241
+
242
+### Public API (assistant-facing)
243
+
244
+| Function | Purpose |
245
+|---|---|
246
+| `agents_load_env` | Source `<repo>/.env`; validate required keys |
247
+| `agents_repo_root` | Locate this repo's checkout root |
248
+| `agents_audit_dir` | Create + return `<repo>/.local/audits/query-netdata-agents/` |
249
+| `agents_netdata_prefix` | Autodetect Netdata install prefix (system / `/opt/netdata` / `/usr/local/netdata`) |
250
+| `agents_query_cloud METHOD PATH [BODY]` | Call any Cloud REST endpoint. Auth is added internally. **Stdout = response body only.** |
251
+| `agents_query_agent --node N --host H --machine-guid M METHOD PATH [BODY]` | Call any direct-agent path. Bearer resolved internally. **Stdout = response body only.** |
252
+| `agents_call_function --via cloud\|agent --node N --function F [--body J]` | Convenience wrapper around the two above |
253
+| `agents_run` / `agents_run_read` | Run curl with masked-token argv echo on stderr (used by the wrappers; rarely needed directly) |
254
+| `agents_selftest_no_token_leak` | Self-test: drives the wrappers with a sentinel token and asserts the sentinel never reaches captured stdout |
255
+
256
+### Internal helpers (do NOT call directly)
257
+
258
+These start with `_` and operate on token bytes inside their own
259
+scope. They return token data via bash namerefs (so the assistant
260
+never sees them on stdout). Don't shell-out to them.
261
+
262
+| Internal | Purpose |
263
+|---|---|
264
+| `_agents_resolve_bearer OUTVAR <node> <mg> <host>` | Cache-aware bearer resolution; writes the bearer into `$OUTVAR` |
265
+| `_agents_get_claim_id OUTVAR <host>` | Resolve `claim_id` from `/api/v3/info`; writes to `$OUTVAR` |
266
+| `_agents_mint_bearer_json <node> <mg> <claim>` | One-shot Cloud bearer mint; the caller MUST capture into a local |
267
+| `_agents_log_masked` | Token / bearer redaction for stderr argv echoes |
268
+| `_agents_exp_to_seconds` | Normalize Cloud `expiration` (sec or ms) to seconds |
269
+
270
+---
271
+
272
+## Direct-agent vs Cloud-proxied: how `agents_call_function` chooses
273
+
274
+Default is `--via cloud` -- the safe choice for any team member.
275
+
276
+`--via agent` requires:
277
+1. The agent host is reachable from the workstation on port 19999.
278
+2. A bearer (auto-minted internally via `_agents_resolve_bearer`).
279
+
280
+Falls back to `--via cloud` if the direct call fails.
281
+
282
+---
283
+
284
+## Sensitive data
285
+
286
+- Bearer values appear in script stderr only when masked.
287
+- The cache file at `.local/audits/.../bearers/<machine_guid>.json`
288
+ contains the raw bearer; mode 0600.
289
+- Never paste bearer values, claim ids, machine GUIDs, or node UUIDs
290
+ into committed files. See
291
+ `<repo>/.agents/sow/specs/sensitive-data-discipline.md` for the
292
+ full rule.
docs/netdata-ai/skills/query-netdata-agents/how-tos/INDEX.md
new
+95
@@ -0,0 +1,95 @@
1
+# query-netdata-agents -- How-tos index
2
+
3
+This directory holds **operational how-tos** for direct-agent
4
+calls: short, focused recipes that combine the per-domain guides
5
+into answers for specific questions. Each how-to documents the
6
+question, the steps taken, the wrappers used, and the expected
7
+output shape.
8
+
9
+## The "if you analyze, you author a how-to" rule
10
+
11
+The how-tos catalog is meant to be **live**. Every time an AI
12
+assistant (or human) is asked a question that:
13
+
14
+1. The user expects a concrete answer to, AND
15
+2. Is not already documented in this index, AND
16
+3. Forces analysis (multiple wrapper calls, jq pipelines, or
17
+ cross-referencing more than one per-domain guide)
18
+
19
+the assistant MUST author a new how-to in this directory and add
20
+it to the index BELOW before completing the task.
21
+
22
+This is mandatory. Skipping it means the next assistant repeats
23
+the same analysis from scratch.
24
+
25
+## How-to authoring template
26
+
27
+Filename: `<slug>.md`. Sections: Question, Inputs, Steps (each
28
+calling one wrapper), Output, Notes / gotchas, Source guides.
29
+
30
+Every code example must use the token-safe wrappers from
31
+`scripts/_lib.sh` (`agents_query_cloud`, `agents_query_agent`,
32
+`agents_call_function`). No raw curl with `Authorization: Bearer
33
+$TOKEN` or `X-Netdata-Auth: Bearer <uuid>` literals -- that
34
+defeats the no-token-leak guarantee.
35
+
36
+## Index
37
+
38
+(Populate as how-tos are authored. Stubs below correspond to the
39
+seed verification questions in `../verify/questions.md`; replace
40
+each `(stub -- not yet authored)` with a real link as soon as a
41
+how-to is written.)
42
+
43
+### Identity / hardware / OS
44
+
45
+- `agent-info-summary.md` (stub -- not yet authored)
46
+- `read-claim-id-direct.md` (stub -- not yet authored)
47
+- `chart-labels-via-metrics-summary.md` (stub -- not yet authored)
48
+
49
+### Streaming
50
+
51
+- `incoming-children-list.md` (stub -- not yet authored)
52
+- `outgoing-parent-target.md` (stub -- not yet authored)
53
+- `replication-progress-per-peer.md` (stub -- not yet authored)
54
+
55
+### Collectors / jobs / vnodes (DynCfg)
56
+
57
+- `list-go.d-jobs-and-status.md` (stub -- not yet authored)
58
+- `list-vnodes.md` (stub -- not yet authored)
59
+- `read-job-config.md` (stub -- not yet authored)
60
+- `add-go.d-job.md` (stub -- not yet authored)
61
+
62
+### Functions
63
+
64
+- `discover-registered-functions.md` (stub -- not yet authored)
65
+- `call-function-info.md` (stub -- not yet authored)
66
+
67
+### Logs
68
+
69
+- `tail-namespace-direct.md` (stub -- not yet authored)
70
+- `last-status-file-log-direct.md` (stub -- not yet authored)
71
+- `histogram-by-priority.md` (stub -- not yet authored)
72
+
73
+### Alerts
74
+
75
+- `currently-firing-alerts-direct.md` (stub -- not yet authored)
76
+- `alert-config-direct.md` (stub -- not yet authored)
77
+- `transitions-last-hour-direct.md` (stub -- not yet authored)
78
+
79
+### Topology / flows
80
+
81
+- `topology-summary-direct.md` (stub -- not yet authored)
82
+- `flows-top-talkers-direct.md` (stub -- not yet authored)
83
+
84
+### Metrics
85
+
86
+- `current-cpu-direct.md` (stub -- not yet authored)
87
+- `peak-memory-last-hour-direct.md` (stub -- not yet authored)
88
+
89
+## Cross-skill how-tos
90
+
91
+When the answer needs both direct-agent and Cloud-side calls
92
+(e.g. "discover the bearer-protected agent's claim_id from Cloud
93
+first, then call it directly"), author the how-to under the
94
+skill that owns the FIRST wrapper call and cross-link to the
95
+other.
docs/netdata-ai/skills/query-netdata-agents/query-alerts.md
new
+75
@@ -0,0 +1,75 @@
1
+# Query agent alerts directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+For the alert REST surface (current alerts, transitions, single
7
+config, the `cfg` field of an alert instance, the `options[]`
8
+array on `/alerts`), see
9
+[../query-netdata-cloud/query-alerts.md](../query-netdata-cloud/query-alerts.md).
10
+The body and response of the agent-direct paths are identical to
11
+the per-agent rows of the Cloud-proxied responses.
12
+
13
+Three v3 paths are available on the agent (prefer v3; v2 shares
14
+the same handler; v1 `/alarms*` only on pre-v2 agents):
15
+
16
+| Path | Method | Purpose |
17
+|---|---|---|
18
+| `/api/v3/alerts` | POST | Current alerts on this host |
19
+| `/api/v3/alert_transitions` | POST | Alert transition history |
20
+| `/api/v3/alert_config?config=<hash>` | GET | Full configuration of one alert |
21
+
22
+Silencing rules, `alerts:misconfigured`, alert-config
23
+generate/suggest/explain, and `alarms`/`alarms/metas` are
24
+**Cloud-only** -- there is no agent-direct equivalent. See
25
+[../query-netdata-cloud/query-alerts.md](../query-netdata-cloud/query-alerts.md).
26
+
27
+---
28
+
29
+## Use the wrapper
30
+
31
+```bash
32
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
33
+agents_load_env
34
+
35
+# Currently-firing alerts on this host with summary + values + instances.
36
+agents_query_agent \
37
+ --node "$NODE_UUID" \
38
+ --host "$AGENT_HOST:19999" \
39
+ --machine-guid "$AGENT_MG" \
40
+ POST /api/v3/alerts '{"options":["summary","values","instances"]}'
41
+
42
+# Transitions over the last hour. NOTE: agent /alert_transitions
43
+# accepts negative `after` values too (different from cloud).
44
+AFTER=$(( $(date +%s) - 3600 ))
45
+NOW=$(date +%s)
46
+agents_query_agent \
47
+ --node "$NODE_UUID" \
48
+ --host "$AGENT_HOST:19999" \
49
+ --machine-guid "$AGENT_MG" \
50
+ POST /api/v3/alert_transitions "{\"after\":$AFTER,\"before\":$NOW}"
51
+
52
+# Read a specific alert's config. Get the hash from .alert_instances[].cfg.
53
+CFG="ALERT_CONFIG_HASH_UUID" # the cfg field of an alert instance
54
+agents_query_agent \
55
+ --node "$NODE_UUID" \
56
+ --host "$AGENT_HOST:19999" \
57
+ --machine-guid "$AGENT_MG" \
58
+ GET "/api/v3/alert_config?config=$CFG"
59
+```
60
+
61
+## Limits and gotchas
62
+
63
+- **Single host.** Aggregation across nodes is Cloud's job.
64
+- **Compact field names**: `cfg`, `nm`, `ctx`, `st`, `tr_i`,
65
+ `tr_v`, `tr_t`, `cl`, `cp`, `tp`, `to`. Spelled-out
66
+ cross-reference is in the cloud guide.
67
+- **`config_hash_id` source**: the `cfg` field on an alert
68
+ instance from `/api/v3/alerts` (with `options:["instances"]`)
69
+ is what to pass as `config=` to `/alert_config`.
70
+
71
+## See also
72
+
73
+- [../query-netdata-cloud/query-alerts.md](../query-netdata-cloud/query-alerts.md)
74
+ -- full alerts surface (Cloud + agent), including everything
75
+ Cloud-only (silencing, generate/suggest/explain, etc.).
docs/netdata-ai/skills/query-netdata-agents/query-dyncfg.md
new
+85
@@ -0,0 +1,85 @@
1
+# Query agent DynCfg directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+For the full DynCfg surface (actions, id structure,
7
+templates/jobs, source types, response codes, schema flow), see
8
+[../query-netdata-cloud/query-dyncfg.md](../query-netdata-cloud/query-dyncfg.md).
9
+
10
+Direct-agent uses the same query parameters and the same payloads
11
+as the Cloud-proxied path. The agent's handler at
12
+`<repo>/src/web/api/v1/api_v1_config.c` is the canonical
13
+implementation; both `/api/v1/config` and `/api/v3/config` route
14
+to it. Prefer v3.
15
+
16
+---
17
+
18
+## Endpoint (agent v3)
19
+
20
+| Method | Path | Purpose |
21
+|---|---|---|
22
+| `GET` | `/api/v3/config?action=tree&path=/` | List configuration objects |
23
+| `GET` | `/api/v3/config?action=<read>&id=<id>[&name=<name>]` | Read a configuration |
24
+| `POST` | `/api/v3/config?action=<write>&id=<id>[&name=<name>]` | Mutate a configuration (body = config JSON) |
25
+
26
+## Use the wrapper
27
+
28
+```bash
29
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
30
+agents_load_env
31
+
32
+# List all configuration objects.
33
+agents_query_agent \
34
+ --node "$NODE_UUID" \
35
+ --host "$AGENT_HOST:19999" \
36
+ --machine-guid "$AGENT_MG" \
37
+ GET '/api/v3/config?action=tree&path=/'
38
+
39
+# Get a specific alert prototype's JSON Schema.
40
+ID='health:alert:prototype:ram_usage'
41
+agents_query_agent \
42
+ --node "$NODE_UUID" \
43
+ --host "$AGENT_HOST:19999" \
44
+ --machine-guid "$AGENT_MG" \
45
+ GET "/api/v3/config?action=schema&id=$(printf %s "$ID" | jq -sRr @uri)"
46
+
47
+# Add a new go.d.plugin nginx job.
48
+TPL='go.d:nginx'; JOB='local_server'
49
+agents_query_agent \
50
+ --node "$NODE_UUID" \
51
+ --host "$AGENT_HOST:19999" \
52
+ --machine-guid "$AGENT_MG" \
53
+ POST "/api/v3/config?action=add&id=$(printf %s "$TPL" | jq -sRr @uri)&name=$JOB" \
54
+ '{"url":"http://127.0.0.1/stub_status","update_every":5}'
55
+```
56
+
57
+The wrapper handles the bearer internally; for write actions
58
+(`POST`), Cloud requires `PermissionFunctionExecPrivileged`. Direct-
59
+agent requires the bearer to grant similar access at the agent
60
+level.
61
+
62
+## When direct-agent is best
63
+
64
+- **Bulk schema fetches.** `tree` -> `schema` for many ids is
65
+ faster direct (skip the Cloud round-trip).
66
+- **`update`/`test` write workflows where you want immediate
67
+ feedback** without going through Cloud's permission gate.
68
+
69
+## Limits and gotchas
70
+
71
+- **`/api/v1/config` is the legacy alias** -- use `/api/v3/config`.
72
+- **id encoding**: ids contain colons; URL-encode them with
73
+ `printf %s "$ID" | jq -sRr @uri` to avoid breaking on
74
+ special characters.
75
+- **`action=test`** without a `name` falls back to a derived name
76
+ (the part after the last colon in the id) for backwards
77
+ compatibility -- best practice is to supply `name` explicitly.
78
+
79
+## See also
80
+
81
+- [../query-netdata-cloud/query-dyncfg.md](../query-netdata-cloud/query-dyncfg.md)
82
+ -- full reference (actions, id structure, response codes).
83
+- `<repo>/src/daemon/dyncfg/README.md` -- internal DynCfg API.
84
+- `<repo>/src/plugins.d/DYNCFG.md` -- external-plugin DynCfg
85
+ protocol.
docs/netdata-ai/skills/query-netdata-agents/query-flows.md
new
+78
@@ -0,0 +1,78 @@
1
+# Query agent network flows directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+For the body parameters (`mode`, `view`, `after`, `before`,
7
+`query`, `selections`, `facets`, `group_by`, `sort_by`, `top_n`,
8
+`field`, `term`), the response envelope (`data.flows[]`,
9
+`data.columns`, `data.facets`, `data.stats`), and the three modes
10
+(`flows` / `autocomplete`) plus five views (`table-sankey`,
11
+`timeseries`, `country-map`, `state-map`, `city-map`), see
12
+[../query-netdata-cloud/query-flows.md](../query-netdata-cloud/query-flows.md).
13
+The body and response are identical between Cloud-proxied and
14
+direct-agent calls.
15
+
16
+Today only `flows:netflow` is registered (covers NetFlow v5/v9,
17
+IPFIX, sFlow). The agent must run the netflow-plugin Rust crate
18
+for the Function to be available.
19
+
20
+---
21
+
22
+## Endpoint (agent v3)
23
+
24
+`POST /api/v3/function?function=flows:netflow`
25
+
26
+## Use the wrapper
27
+
28
+```bash
29
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
30
+agents_load_env
31
+
32
+read -r -d '' BODY <<'JSON'
33
+{
34
+ "mode": "flows",
35
+ "view": "table-sankey",
36
+ "after": -3600,
37
+ "before": 0,
38
+ "group_by": ["SRC_ADDR", "DST_ADDR", "PROTOCOL"],
39
+ "sort_by": "bytes",
40
+ "top_n": 100
41
+}
42
+JSON
43
+
44
+agents_query_agent \
45
+ --node "$NODE_UUID" \
46
+ --host "$AGENT_HOST:19999" \
47
+ --machine-guid "$AGENT_MG" \
48
+ POST '/api/v3/function?function=flows:netflow' "$BODY" \
49
+ | jq '.data | {view, group_by, flows_count: (.flows|length), stats}'
50
+```
51
+
52
+## Discover supported parameters
53
+
54
+```bash
55
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
56
+ POST '/api/v3/function?function=flows:netflow' '{"info":true}' \
57
+ | jq '.required_params'
58
+```
59
+
60
+## Limits and gotchas
61
+
62
+- **L3 only.** No L2 visibility, no application-layer dissection.
63
+- **Sampling matters.** NetFlow v5/v9 and sFlow are sampled at
64
+ the source device; reported byte/packet counts are scaled by
65
+ the sample rate. Verify source-device sampling configuration
66
+ before treating absolute volumes as ground truth.
67
+- **GeoIP / AS DB dependency.** `SRC_AS_NAME`, `DST_AS_NAME`,
68
+ `*_COUNTRY`, `*_CITY` only populate when the collector has the
69
+ corresponding databases configured.
70
+- **`top_n` is enumerated**: 25, 50, 100, 200, 500. Other values
71
+ rejected.
72
+
73
+## See also
74
+
75
+- [../query-netdata-cloud/query-flows.md](../query-netdata-cloud/query-flows.md)
76
+ -- full reference.
77
+- [query-functions.md](./query-functions.md) -- generic Function
78
+ transport.
docs/netdata-ai/skills/query-netdata-agents/query-functions.md
new
+100
@@ -0,0 +1,100 @@
1
+# Query agent Functions directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first for the
5
+prerequisites (cloud token, network reachability, bearer flow).
6
+
7
+For the response envelope (`status`, `v`, `type`, `help`,
8
+`accepted_params`, `required_params`, `has_history`,
9
+`update_every`, `data`), the four Function families, the canonical
10
+protocol reference at
11
+`<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`, and per-Function
12
+body shapes, see
13
+[../query-netdata-cloud/query-functions.md](../query-netdata-cloud/query-functions.md).
14
+The agent and the Cloud proxy expose the same Function payload
15
+shape -- the only difference is the URL and the auth header.
16
+
17
+---
18
+
19
+## Endpoint (agent v3)
20
+
21
+`POST /api/v3/function?function={functionName}` on the agent at
22
+port 19999. Path on the agent's HTTP API:
23
+
24
+```
25
+http://<agent>:19999/host/<node-uuid>/api/v3/function?function=<name>
26
+```
27
+
28
+`/api/v2/function` is also accepted on older agents -- prefer v3.
29
+
30
+## Discover Functions on a single agent
31
+
32
+Most agents expose a function-listing surface through the same
33
+generic Function call with `function=info`-like discovery. To
34
+enumerate by name, query each Function with `{"info":true}`. For
35
+a top-level list, use the Cloud-side functions endpoint via
36
+[../query-netdata-cloud/query-functions.md#list-available-functions](../query-netdata-cloud/query-functions.md#list-available-functions);
37
+the Cloud listing is authoritative even when you ultimately call
38
+the agent directly.
39
+
40
+## Invoke a Function via the wrapper
41
+
42
+```bash
43
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
44
+agents_load_env
45
+
46
+# Discover the parameter set (info=true is the safe first call).
47
+agents_query_agent \
48
+ --node "$AGENT_EVENTS_NODE_ID" \
49
+ --host "$AGENT_EVENTS_HOSTNAME:19999" \
50
+ --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
51
+ POST '/api/v3/function?function=processes' '{"info":true}'
52
+
53
+# Real query (after `info` told you the parameters).
54
+agents_query_agent \
55
+ --node "$AGENT_EVENTS_NODE_ID" \
56
+ --host "$AGENT_EVENTS_HOSTNAME:19999" \
57
+ --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
58
+ POST '/api/v3/function?function=processes' '{"last":50,"timeout":30000}'
59
+```
60
+
61
+The wrapper writes the response JSON to stdout; stderr shows the
62
+curl invocation with `<AGENT_BEARER>` masked (the bearer is
63
+minted/cached/refreshed internally and never reaches stdout).
64
+
65
+## When to prefer agent-direct over Cloud-proxied
66
+
67
+- **Lower latency.** Direct skips the Cloud round-trip entirely.
68
+- **Cloud unavailable.** The agent answers as long as port 19999
69
+ is reachable from your workstation.
70
+- **High-frequency batch fetches.** The Cloud may rate-limit
71
+ function calls; the agent does not.
72
+
73
+When the user only has Cloud access (the typical team-member case
74
+on a remote agent), use the Cloud-proxied path documented in
75
+[../query-netdata-cloud/query-functions.md](../query-netdata-cloud/query-functions.md)
76
+instead.
77
+
78
+## Limits and gotchas
79
+
80
+- **Bearer protection**: a 412 response from the agent means the
81
+ agent is bearer-protected. The `agents_query_agent` wrapper
82
+ handles this transparently (mint via Cloud, cache, refresh).
83
+ Direct curl fails until you mint a bearer.
84
+- **Function name is case-sensitive.** Wrong casing returns 400.
85
+- **Response is not streamed.** Even on the agent, the Function
86
+ response is buffered into a single JSON document.
87
+- **The `cfg` field of an alert instance**, `claim_id`, `node_id`,
88
+ and similar UUID values appear in responses. Treat as
89
+ semi-sensitive; never paste raw responses into committed files.
90
+
91
+## See also
92
+
93
+- [../query-netdata-cloud/query-functions.md](../query-netdata-cloud/query-functions.md)
94
+ -- canonical Function reference, response envelope, the four
95
+ families, the `info` widget schema, developer references.
96
+- [../query-netdata-cloud/query-logs.md](../query-netdata-cloud/query-logs.md),
97
+ [query-topology.md](../query-netdata-cloud/query-topology.md),
98
+ [query-flows.md](../query-netdata-cloud/query-flows.md) --
99
+ per-family deep dives (the body shapes apply to direct-agent
100
+ calls verbatim).
docs/netdata-ai/skills/query-netdata-agents/query-logs.md
new
+86
@@ -0,0 +1,86 @@
1
+# Query agent logs directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+For the body shape (`after`, `before`, `last`, `query`, `facets`,
7
+`histogram`, `__logs_sources`, `selections`, etc.) and the
8
+response envelope (top-level `data` is an array of row arrays;
9
+`columns` defines positions; `facets` and `histogram` accompany),
10
+see
11
+[../query-netdata-cloud/query-logs.md](../query-netdata-cloud/query-logs.md).
12
+The body and response are identical between Cloud-proxied and
13
+direct-agent calls -- including the multi-value `selections`
14
+field-filter mechanism (AND across fields, OR across values),
15
+which makes index-friendly queries possible on large namespaces.
16
+See the "Multi-value field selections" section in the Cloud doc
17
+for the exact shape and the structured-filters-first rule.
18
+
19
+The agent ships the same three log Functions:
20
+
21
+- `systemd-journal` (Linux nodes)
22
+- `windows-events` (Windows nodes)
23
+- `otel-logs` (when the OTEL log receiver is enabled)
24
+
25
+---
26
+
27
+## Endpoint (agent v3)
28
+
29
+`POST /api/v3/function?function=<log-fn>` on the agent.
30
+
31
+## Use the wrapper
32
+
33
+```bash
34
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
35
+agents_load_env
36
+
37
+# Last-hour skim of a specific journal namespace, 50 rows.
38
+agents_query_agent \
39
+ --node "$AGENT_EVENTS_NODE_ID" \
40
+ --host "$AGENT_EVENTS_HOSTNAME:19999" \
41
+ --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
42
+ POST '/api/v3/function?function=systemd-journal' \
43
+ '{"after":-3600,"before":0,"last":50,"direction":"backward","__logs_sources":"agent-events"}'
44
+```
45
+
46
+The wrapper minted/cached the bearer internally; stdout is the
47
+response body only. The bearer never reaches the assistant's
48
+captured output.
49
+
50
+## Discover the available log sources
51
+
52
+```bash
53
+agents_query_agent \
54
+ --node "$AGENT_EVENTS_NODE_ID" \
55
+ --host "$AGENT_EVENTS_HOSTNAME:19999" \
56
+ --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
57
+ POST '/api/v3/function?function=systemd-journal' '{"info":true}' \
58
+ | jq '.required_params[] | select(.id=="__logs_sources") | .options'
59
+```
60
+
61
+Reads the `info=true` response and lists the `__logs_sources`
62
+widget options the agent currently exposes. The `name`+`id` of
63
+each option is what you pass back as the `__logs_sources` value.
64
+
65
+## Limits and gotchas (single-agent-specific)
66
+
67
+- **Single-host only.** The agent answers for itself; for fleet
68
+ queries, use the Cloud-side path or aggregate per-agent
69
+ responses client-side.
70
+- **Time bounds**: negative values are seconds-relative-to-now.
71
+ Positive values are unix-microseconds (NOT seconds, NOT
72
+ milliseconds). Mixing units is the most common bug.
73
+- **Slow queries**: large windows + wide facets can take seconds.
74
+ Bump `timeout` in the body to 60000 or higher when the default
75
+ 10-second cloud-proxy default isn't relevant (the agent itself
76
+ honors the body timeout up to its own ceiling).
77
+
78
+## See also
79
+
80
+- [../query-netdata-cloud/query-logs.md](../query-netdata-cloud/query-logs.md)
81
+ -- full body/response shape, examples, response field
82
+ reference.
83
+- [query-functions.md](./query-functions.md) -- the generic
84
+ Function transport.
85
+- `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` -- canonical
86
+ Log Explorer Format spec.
docs/netdata-ai/skills/query-netdata-agents/query-metrics.md
new
+116
@@ -0,0 +1,116 @@
1
+# Query agent metrics directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+For the full request body (scope / selectors / window /
7
+aggregations / format / options), the response envelope (jsonwrap
8
+with summary / view / result / db / timings), the time-aggregation
9
+and dimension-aggregation rules, and worked examples, see
10
+[../query-netdata-cloud/query-metrics.md](../query-netdata-cloud/query-metrics.md).
11
+The Cloud `/api/v3/spaces/{sp}/rooms/{rm}/data` endpoint forwards
12
+the same body to the agent's `/api/v3/data` endpoint.
13
+
14
+---
15
+
16
+## Endpoint (agent v3)
17
+
18
+`POST /api/v3/data` on the agent. The request body is identical to
19
+the Cloud `/data` body except `scope.nodes` (cloud) is implicit
20
+on the agent (you're already targeting one node).
21
+
22
+## Use the wrapper
23
+
24
+```bash
25
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
26
+agents_load_env
27
+
28
+read -r -d '' BODY <<'JSON'
29
+{
30
+ "scope": {"contexts": ["system.cpu"]},
31
+ "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
32
+ "window": {"after": -600, "before": 0, "points": 5},
33
+ "aggregations": {
34
+ "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}],
35
+ "time": {"time_group": "average"}
36
+ },
37
+ "format": "json2",
38
+ "options": ["jsonwrap", "minify", "unaligned"],
39
+ "timeout": 30000
40
+}
41
+JSON
42
+
43
+agents_query_agent \
44
+ --node "$NODE_UUID" \
45
+ --host "$AGENT_HOST:19999" \
46
+ --machine-guid "$AGENT_MG" \
47
+ POST /api/v3/data "$BODY" \
48
+ | jq '{view: .view.dimensions.names, points: (.result.data | length)}'
49
+```
50
+
51
+## Discover available contexts on the agent
52
+
53
+```bash
54
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
55
+ GET '/api/v3/contexts'
56
+```
57
+
58
+`/api/v3/contexts` returns the metric contexts the agent currently
59
+collects (e.g. `system.cpu`, `disk.space`, `nginx.connections`).
60
+Use these as `scope.contexts` values.
61
+
62
+## Time resolution: `duration ÷ points = seconds per point`
63
+
64
+The number of `points` is NOT "give me per-second data". It is
65
+"split the duration into N equal buckets". Actual time
66
+resolution:
67
+
68
+```
69
+seconds_per_point = abs(after) ÷ points (when before = 0)
70
+seconds_per_point = abs(duration) ÷ points (when duration is set)
71
+```
72
+
73
+**To get per-second data, set `points` equal to the duration in
74
+seconds.**
75
+
76
+| You want | Set `after` | Set `points` | Result |
77
+|---|---|---|---|
78
+| Per-second resolution, last 2 minutes | `-120` | `120` | 1 second per point |
79
+| Per-second resolution, last 5 minutes | `-300` | `300` | 1 second per point |
80
+| 10-second buckets, last 10 minutes | `-600` | `60` | 10 seconds per point |
81
+| Per-minute resolution, last hour | `-3600` | `60` | 60 seconds per point |
82
+
83
+**Common mistake**: `after: -600, points: 30` is NOT per-second
84
+data over 10 minutes -- it is 20-seconds-per-point heavily
85
+aggregated data. Per-second resolution over 10 minutes requires
86
+`points: 600` (at the 500-point server cap; reduce duration or
87
+accept coarser resolution).
88
+
89
+**Per-second data also requires dbengine tier 0** (per-second
90
+storage) covers the requested time range. If tier 0 retention is
91
+shorter than `abs(after)`, the engine auto-selects a coarser
92
+tier silently. Force tier 0 with `"tier": 0` in the window to
93
+fail loudly rather than silently downsample.
94
+
95
+**`points: 0` is NOT "per-second"** -- it means "all available
96
+points within the 500 cap", which the engine still aggregates
97
+when the duration exceeds 500 seconds.
98
+
99
+## Limits and gotchas
100
+
101
+- **`scope.contexts` MUST be set.** Without it, the response
102
+ contains metadata for every context on the agent.
103
+- **`unaligned`**: include in `options` for API queries to avoid
104
+ wall-clock alignment of the time window.
105
+- **Max points ≈ 500** per query (server-side cap).
106
+- **Single host.** For multi-node aggregation, use the Cloud
107
+ `/data` path documented in
108
+ [../query-netdata-cloud/query-metrics.md](../query-netdata-cloud/query-metrics.md).
109
+
110
+## See also
111
+
112
+- [../query-netdata-cloud/query-metrics.md](../query-netdata-cloud/query-metrics.md)
113
+ -- full body / response / examples.
114
+- [query-functions.md](./query-functions.md) -- generic Function
115
+ transport (Functions != metrics, but they share the same
116
+ Cloud-proxy and bearer semantics).
docs/netdata-ai/skills/query-netdata-agents/query-nodes.md
new
+176
@@ -0,0 +1,176 @@
1
+# Query an agent's node identity and metadata directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+The agent exposes its identity, capabilities, hardware, OS, and
7
+collection-job state via several endpoints. Unlike Cloud's `/nodes`
8
+(which lists multiple nodes in a room), agent-direct calls return
9
+data for the **single host** the agent runs on (plus any virtual
10
+hosts / parents / streamed children, see
11
+[query-streaming.md](./query-streaming.md)).
12
+
13
+For the Cloud-side per-room enumeration (`POST
14
+/api/v3/spaces/{sp}/rooms/{rm}/nodes`), see
15
+[../query-netdata-cloud/query-nodes.md](../query-netdata-cloud/query-nodes.md).
16
+
17
+---
18
+
19
+## Endpoints (agent v3)
20
+
21
+| Method | Path | Purpose |
22
+|---|---|---|
23
+| `GET` | `/api/v3/info` | Identity (node_id, machine_guid, claim_id), agent version, application info, capabilities. **No auth required**, but only basic info. |
24
+| `GET` | `/api/v3/contexts` | Metric contexts the agent collects (= what data is available) |
25
+| `GET` | `/api/v3/nodes` | Multi-host listing if this agent acts as a parent (see [query-streaming.md](./query-streaming.md)) |
26
+| `GET` | `/api/v3/info?host=<node_id>` | Detail for a specific host (when multi-host) |
27
+
28
+## Use the wrappers
29
+
30
+```bash
31
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
32
+agents_load_env
33
+
34
+# /info is unauthenticated -- you can call it without the bearer.
35
+# But going through agents_query_agent uses the bearer flow, which
36
+# is fine and consistent.
37
+agents_query_agent \
38
+ --node "$NODE_UUID" \
39
+ --host "$AGENT_HOST:19999" \
40
+ --machine-guid "$AGENT_MG" \
41
+ GET /api/v3/info \
42
+ | jq '.agents[0] | {nm, nd, mg, cloud, application: .application.package.version}'
43
+
44
+# Hardware and OS labels (typically exposed under .labels in /info or
45
+# in the chart-labels namespace; see the response shape).
46
+agents_query_agent \
47
+ --node "$NODE_UUID" \
48
+ --host "$AGENT_HOST:19999" \
49
+ --machine-guid "$AGENT_MG" \
50
+ GET /api/v3/info \
51
+ | jq '.agents[0].application, .agents[0].cloud'
52
+
53
+# Claim_id specifically (used by the bearer-mint flow).
54
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
55
+ GET /api/v3/info | jq -r '.agents[0].cloud.claim_id'
56
+```
57
+
58
+## Top-level response shape (`/api/v3/info`)
59
+
60
+```text
61
+{
62
+ "api": 2,
63
+ "agents": [
64
+ {
65
+ "mg": "<machine_guid UUID>",
66
+ "nd": "<node UUID>",
67
+ "nm": "<hostname>",
68
+ "now": <unix-seconds>,
69
+ "ai": <agent index>,
70
+ "application": {
71
+ "package": { "version": "vX.Y.Z-...", "type": "binpkg-deb|...", "arch": "x86_64|...", ... },
72
+ "configure": "cmake -...",
73
+ ...
74
+ },
75
+ "cloud": {
76
+ "claim_id": "<UUID>",
77
+ "aclk": "available|online|...",
78
+ ...
79
+ },
80
+ "labels": { ... },
81
+ ...
82
+ }
83
+ ],
84
+ ...
85
+}
86
+```
87
+
88
+For the full per-agent label set, the `chart-labels` are usually
89
+exposed via the metrics path's `summary.nodes[].labels` (see
90
+[query-metrics.md](./query-metrics.md)).
91
+
92
+## Hardware / OS query patterns
93
+
94
+Hardware and OS facts live primarily in the agent's host-labels.
95
+On a fully-running agent the labels are present in `/api/v3/info`
96
+and copied verbatim into the Cloud `/nodes` `.labels` field (the
97
+fast path for cross-fleet queries).
98
+
99
+| Field | Where it lives |
100
+|---|---|
101
+| Architecture, kernel, OS name/version | `agents[0].application` (build-time) AND `summary.nodes[].labels._architecture`, `_kernel_version`, `_os_name`, `_os_version` |
102
+| CPU count, RAM, disk space | `summary.nodes[].labels._system_cores`, `_system_ram_total`, `_system_disk_space` (chart-labels) |
103
+| Cloud provider / region / instance type | `summary.nodes[].labels._cloud_provider_type`, `_cloud_instance_region`, `_cloud_instance_type` |
104
+| Container/virtualization | `summary.nodes[].labels._container`, `_container_detection`, `_is_k8s_node`, `_is_parent`, `_is_ephemeral` |
105
+
106
+To fetch chart-labels as a structured object, run a metrics query
107
+and read `summary.nodes[].labels`:
108
+
109
+```bash
110
+read -r -d '' BODY <<'JSON'
111
+{
112
+ "scope": {"contexts": ["system.cpu"]},
113
+ "selectors": {"nodes": ["*"]},
114
+ "window": {"after": -60, "before": 0, "points": 1},
115
+ "aggregations": {"metrics": [{"group_by": ["selected"]}], "time": {"time_group": "average"}},
116
+ "format": "json2",
117
+ "options": ["jsonwrap", "minify", "unaligned"]
118
+}
119
+JSON
120
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
121
+ POST /api/v3/data "$BODY" \
122
+ | jq '.summary.nodes[0].labels'
123
+```
124
+
125
+## Collection-job state (failed / disabled jobs)
126
+
127
+The agent's DynCfg surface lists every collection job and its
128
+status. See [query-dyncfg.md](./query-dyncfg.md):
129
+
130
+```bash
131
+# List every go.d.plugin job and its current state.
132
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
133
+ GET '/api/v3/config?action=tree&path=/collectors/go.d/Jobs' \
134
+ | jq '.tree["/collectors/go.d/Jobs"]'
135
+```
136
+
137
+DynCfg job statuses include `running` (200), `accepted` (202),
138
+`accepted-disabled` (298), `accepted-restart-required` (299),
139
+plus error states (4xx/5xx). A failed-collection job appears with
140
+a 4xx/5xx status and an error message.
141
+
142
+## Vnodes
143
+
144
+Virtual nodes (configured via `/etc/netdata/vnodes/`) are listed
145
+under `/collectors/go.d/Vnodes` and `/collectors/ibm.d/Vnodes` in
146
+the DynCfg tree. Use the same DynCfg path:
147
+
148
+```bash
149
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
150
+ GET '/api/v3/config?action=tree&path=/collectors/go.d/Vnodes'
151
+```
152
+
153
+## Limits and gotchas
154
+
155
+- **`/api/v3/info` is unauthenticated**, but most other paths
156
+ require the bearer. The wrapper always uses the bearer; that's
157
+ fine for `/info` too.
158
+- **`labels` location varies by version.** On older agents some
159
+ labels appear only in `summary.nodes[].labels` of metrics
160
+ responses; on newer agents they're also under
161
+ `agents[0].labels`. Check both.
162
+- **Streaming roles** (parent / child) are at
163
+ `summary.nodes[].labels._is_parent` (true/false), and the
164
+ full streaming surface lives in
165
+ [query-streaming.md](./query-streaming.md).
166
+
167
+## See also
168
+
169
+- [../query-netdata-cloud/query-nodes.md](../query-netdata-cloud/query-nodes.md)
170
+ -- per-room / per-space node enumeration via Cloud.
171
+- [query-dyncfg.md](./query-dyncfg.md) -- DynCfg surface (jobs,
172
+ vnodes, config).
173
+- [query-streaming.md](./query-streaming.md) -- parent/child
174
+ streaming relationships and replication state.
175
+- [query-metrics.md](./query-metrics.md) -- chart-labels via
176
+ `summary.nodes[].labels` of metric queries.
docs/netdata-ai/skills/query-netdata-agents/query-streaming.md
new
+108
@@ -0,0 +1,108 @@
1
+# Query agent streaming (parent / child / replication)
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+Netdata agents form a streaming graph: a **child** sends its
7
+metrics to a **parent**, which can in turn forward to another
8
+parent. The same agent can be both a parent (receiving from
9
+children) and a child (sending upstream). This guide covers how to
10
+query that graph from a specific agent's perspective.
11
+
12
+There is **no Cloud-side equivalent** for the agent-internal
13
+streaming/replication state -- this surface is agent-only.
14
+
15
+---
16
+
17
+## Function name
18
+
19
+The agent registers a Function called `netdata-streaming` (verified
20
+live; see
21
+[../query-netdata-cloud/query-functions.md#frequently-registered-functions](../query-netdata-cloud/query-functions.md#frequently-registered-functions)).
22
+It returns:
23
+
24
+- Per-streaming-peer connection state (replication progress,
25
+ bytes-in / bytes-out, last-error).
26
+- Whether this agent is acting as a parent, a child, or both.
27
+- The list of children currently streaming to this agent.
28
+- The parent endpoints this agent is streaming to.
29
+- ML status of the streaming pipeline (if ML is enabled).
30
+
31
+## Use the wrapper
32
+
33
+```bash
34
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
35
+agents_load_env
36
+
37
+# Discover the parameters first.
38
+agents_query_agent \
39
+ --node "$NODE_UUID" \
40
+ --host "$AGENT_HOST:19999" \
41
+ --machine-guid "$AGENT_MG" \
42
+ POST '/api/v3/function?function=netdata-streaming' '{"info":true}' \
43
+ | jq '{accepted_params, required_params}'
44
+
45
+# Real query: top-level streaming state.
46
+agents_query_agent \
47
+ --node "$NODE_UUID" \
48
+ --host "$AGENT_HOST:19999" \
49
+ --machine-guid "$AGENT_MG" \
50
+ POST '/api/v3/function?function=netdata-streaming' '{"timeout":30000}'
51
+```
52
+
53
+The response uses the standard Function envelope; `data` holds
54
+the per-peer rows. See
55
+[../query-netdata-cloud/query-functions.md](../query-netdata-cloud/query-functions.md)
56
+for the envelope definition and the canonical
57
+`<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`.
58
+
59
+## Question-to-query cheatsheet
60
+
61
+| Question | Approach |
62
+|---|---|
63
+| "Is this node a parent?" | Check `summary.nodes[0].labels._is_parent` from a metrics query (cheapest), OR look at the `netdata-streaming` Function's per-peer rows -- if any `direction:incoming` rows exist, this node is a parent. |
64
+| "Of how many and which nodes?" | Filter the Function's rows where `direction:incoming` -- one per child. Each row carries the child's hostname and node id. |
65
+| "Is this node a child? Where does it stream?" | Check `summary.nodes[0].labels._is_parent == false` AND look for `direction:outgoing` rows in the Function -- the destination is the upstream parent. |
66
+| "What is the replication progress?" | Each row has replication-related fields (`replication_progress`, `replication_lag`, `replication_eta`). |
67
+| "Are there any disconnected peers?" | Filter rows where `state` != `connected` / `streaming`. |
68
+
69
+## Ancillary surfaces
70
+
71
+- **Per-stream metrics** (bandwidth, packets, dropped points) are
72
+ exposed as Netdata charts under the `netdata.streaming.*`
73
+ context family. Use [query-metrics.md](./query-metrics.md) to
74
+ query them as time series.
75
+- **Streaming configuration** (which parents this agent connects
76
+ to, retention settings) lives in
77
+ `/etc/netdata/stream.conf`. The DynCfg path is
78
+ `/streaming` if your agent version exposes streaming config
79
+ through DynCfg; check via
80
+ `agents_query_agent ... GET '/api/v3/config?action=tree&path=/streaming'`.
81
+- **`_is_parent` / `_is_ephemeral`** chart-labels surface the
82
+ parent / ephemeral state succinctly; see
83
+ [query-nodes.md](./query-nodes.md).
84
+
85
+## Limits and gotchas
86
+
87
+- **Function name**: literal `netdata-streaming` (note the dash,
88
+ not a colon). The listing endpoint is the only authoritative
89
+ source -- if your agent is older, the Function name may differ.
90
+- **Cloud aggregation does not exist** for streaming state. To
91
+ build a fleet-wide view, fan out per-agent calls and merge
92
+ client-side.
93
+- **Per-peer rows can be tens of KB each** when there are many
94
+ children. Use `last` or pagination knobs in the Function body
95
+ if needed.
96
+- **Agent must have streaming enabled.** A standalone (non-
97
+ streaming) agent has no rows; the Function still returns 200
98
+ but with an empty `data` array.
99
+
100
+## See also
101
+
102
+- [query-functions.md](./query-functions.md) -- generic Function
103
+ transport.
104
+- [../query-netdata-cloud/query-functions.md](../query-netdata-cloud/query-functions.md)
105
+ -- canonical Function reference + envelope.
106
+- [query-nodes.md](./query-nodes.md) -- node identity, parent /
107
+ child labels.
108
+- `<repo>/src/streaming/` -- agent-side streaming implementation.
docs/netdata-ai/skills/query-netdata-agents/query-topology.md
new
+75
@@ -0,0 +1,75 @@
1
+# Query agent topology directly
2
+
3
+This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4
+Read [SKILL.md](./SKILL.md#prerequisites) first.
5
+
6
+For the body parameters (`nodes_identity`, `map_type`,
7
+`inference_strategy`, `managed_snmp_device_focus`, `depth`),
8
+the response envelope (top-level `data.actors[]` + `data.links[]`),
9
+and per-actor / per-link field semantics, see
10
+[../query-netdata-cloud/query-topology.md](../query-netdata-cloud/query-topology.md).
11
+The body and response are identical between Cloud-proxied and
12
+direct-agent calls.
13
+
14
+Today only `topology:snmp` is registered. Future topology Functions
15
+will follow the same `topology:<source>` namespace and the same
16
+envelope.
17
+
18
+---
19
+
20
+## Endpoint (agent v3)
21
+
22
+`POST /api/v3/function?function=topology:snmp`
23
+
24
+## Use the wrapper
25
+
26
+```bash
27
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
28
+agents_load_env
29
+
30
+read -r -d '' BODY <<'JSON'
31
+{
32
+ "selections": {
33
+ "nodes_identity": ["mac"],
34
+ "map_type": ["lldp_cdp_managed"],
35
+ "inference_strategy": ["fdb_minimum_knowledge"],
36
+ "managed_snmp_device_focus": ["all_devices"],
37
+ "depth": ["all"]
38
+ },
39
+ "timeout": 60000
40
+}
41
+JSON
42
+
43
+agents_query_agent \
44
+ --node "$NODE_UUID" \
45
+ --host "$AGENT_HOST:19999" \
46
+ --machine-guid "$AGENT_MG" \
47
+ POST '/api/v3/function?function=topology:snmp' "$BODY" \
48
+ | jq '.data | {actors: (.actors|length), links: (.links|length), view, layer}'
49
+```
50
+
51
+## Discover supported parameters
52
+
53
+```bash
54
+agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
55
+ POST '/api/v3/function?function=topology:snmp' '{"info":true}' \
56
+ | jq '.required_params'
57
+```
58
+
59
+## Limits and gotchas
60
+
61
+- **Slow queries.** A full SNMP sweep on a busy network can take
62
+ 60+ seconds. Set `timeout` accordingly.
63
+- **MAC-list `actor_id` values can be very long.** Use
64
+ `nodes_identity:["ip"]` to collapse devices by IP if you prefer
65
+ shorter ids.
66
+- **Single-agent perspective.** The topology graph is what THIS
67
+ agent has discovered. Multi-agent fleets need merge logic on
68
+ the client side.
69
+
70
+## See also
71
+
72
+- [../query-netdata-cloud/query-topology.md](../query-netdata-cloud/query-topology.md)
73
+ -- full reference, parameter values, body / response detail.
74
+- [query-functions.md](./query-functions.md) -- generic Function
75
+ transport.
docs/netdata-ai/skills/query-netdata-agents/scripts/_lib.sh
new
+454
@@ -0,0 +1,454 @@
1
+#!/usr/bin/env bash
2
+# Helpers for the query-netdata-agents skill.
3
+# Sourced from per-action scripts (and from any other skill that
4
+# wants to call Netdata Cloud / Netdata Agent through token-safe
5
+# wrappers). Not executed directly.
6
+#
7
+# Token-safety contract (HARD requirement):
8
+# * No PUBLIC function (named `agents_*`, no leading underscore)
9
+# ever emits NETDATA_CLOUD_TOKEN, a per-agent bearer, or a
10
+# claim_id to stdout.
11
+# * Internal helpers (named `_agents_*`, leading underscore) may
12
+# handle token bytes inside their own scope but must return
13
+# them only via `local -n` namerefs into the caller's local
14
+# variables -- never to stdout.
15
+# * `_agents_log_masked` redacts token / bearer bytes in stderr
16
+# argv echoes.
17
+# * The unit test `agents_selftest_no_token_leak` drives every
18
+# public wrapper with a sentinel token and asserts the
19
+# sentinel never reaches captured stdout.
20
+#
21
+# Conventions mirrored from .agents/skills/coverity-audit/scripts/_lib.sh:
22
+# * set -euo pipefail at the top
23
+# * color vars defined with $'...' so ESC bytes are real
24
+# * <prefix>_repo_root via `git rev-parse --show-toplevel`
25
+# * <prefix>_load_env sources <repo>/.env, validates required keys
26
+# * <prefix>_audit_dir creates <repo>/.local/audits/<topic>/
27
+#
28
+# Audit topic: "query-netdata-agents".
29
+
30
+# Capture our source-file path BEFORE `set -u`. Bash exposes
31
+# BASH_SOURCE[0]; zsh exposes the equivalent as `${(%):-%x}` (which
32
+# bash cannot parse, so we gate it through `eval`).
33
+if [ -n "${ZSH_VERSION-}" ]; then
34
+ eval '_agents_lib_self="${(%):-%x}"'
35
+elif [ -n "${BASH_VERSION-}" ]; then
36
+ _agents_lib_self="${BASH_SOURCE[0]}"
37
+else
38
+ _agents_lib_self="$0"
39
+fi
40
+
41
+set -euo pipefail
42
+
43
+# shellcheck disable=SC2034
44
+AGENTS_RED=$'\033[0;31m'
45
+# shellcheck disable=SC2034
46
+AGENTS_GREEN=$'\033[0;32m'
47
+# shellcheck disable=SC2034
48
+AGENTS_YELLOW=$'\033[1;33m'
49
+# shellcheck disable=SC2034
50
+AGENTS_GRAY=$'\033[0;90m'
51
+# shellcheck disable=SC2034
52
+AGENTS_NC=$'\033[0m'
53
+
54
+# ---------------------------------------------------------------------------
55
+# Repo + env helpers
56
+# ---------------------------------------------------------------------------
57
+
58
+agents_repo_root() {
59
+ git -C "$(dirname "${_agents_lib_self}")" rev-parse --show-toplevel
60
+}
61
+
62
+# Source <repo>/.env. Validate the keys this skill needs.
63
+# Required:
64
+# NETDATA_CLOUD_TOKEN -- long-lived Cloud REST token
65
+# NETDATA_CLOUD_HOSTNAME -- Cloud REST host (e.g. app.netdata.cloud)
66
+agents_load_env() {
67
+ local root env
68
+ root="$(agents_repo_root)"
69
+ env="${root}/.env"
70
+ if [[ ! -f "${env}" || ! -r "${env}" ]]; then
71
+ echo -e "${AGENTS_RED}[ERROR]${AGENTS_NC} Missing ${env}. Copy .env.template to .env and fill it in. See ${root}/.agents/ENV.md." >&2
72
+ return 1
73
+ fi
74
+ set -a
75
+ # shellcheck disable=SC1090
76
+ source "${env}"
77
+ set +a
78
+
79
+ : "${NETDATA_CLOUD_TOKEN:?NETDATA_CLOUD_TOKEN is empty -- see <repo>/.agents/ENV.md to set it.}"
80
+ : "${NETDATA_CLOUD_HOSTNAME:?NETDATA_CLOUD_HOSTNAME is empty -- see <repo>/.agents/ENV.md to set it.}"
81
+ export NETDATA_CLOUD_TOKEN NETDATA_CLOUD_HOSTNAME
82
+}
83
+
84
+agents_audit_dir() {
85
+ local root dir
86
+ root="$(agents_repo_root)"
87
+ dir="${root}/.local/audits/query-netdata-agents"
88
+ mkdir -p "${dir}"
89
+ echo "${dir}"
90
+}
91
+
92
+# Autodetect the Netdata install prefix. Returns "" for system installs
93
+# (paths like /var/lib/netdata, /etc/netdata) or e.g. "/opt/netdata" for
94
+# bundled installs (paths under /opt/netdata/var/lib/netdata).
95
+#
96
+# Rule (per .agents/sow/specs/sensitive-data-discipline.md): probe
97
+# candidates and pick the first whose <prefix>/var/lib/netdata or
98
+# <prefix>/etc/netdata exists. NOT a config knob.
99
+agents_netdata_prefix() {
100
+ local p
101
+ for p in "" "/opt/netdata" "/usr/local/netdata"; do
102
+ if [[ -d "${p}/var/lib/netdata" || -d "${p}/etc/netdata" ]]; then
103
+ printf '%s' "${p}"
104
+ return 0
105
+ fi
106
+ done
107
+ printf ''
108
+ return 0
109
+}
110
+
111
+# ---------------------------------------------------------------------------
112
+# Masked-curl execution wrappers
113
+# ---------------------------------------------------------------------------
114
+
115
+# Print a curl invocation to stderr with the cloud token (and any
116
+# minted bearer) masked. Then execute it. Honors AGENTS_DRY_RUN=1
117
+# (write paths skip execution but still log).
118
+agents_run() {
119
+ _agents_log_masked "$@"
120
+ if [[ "${AGENTS_DRY_RUN:-0}" == "1" ]]; then
121
+ return 0
122
+ fi
123
+ "$@"
124
+}
125
+
126
+agents_run_read() {
127
+ _agents_log_masked "$@"
128
+ "$@"
129
+}
130
+
131
+_agents_log_masked() {
132
+ local arg
133
+ printf >&2 '%s> %s' "${AGENTS_GRAY}" "${AGENTS_YELLOW}"
134
+ for arg in "$@"; do
135
+ # Mask the cloud token wherever it appears.
136
+ if [[ -n "${NETDATA_CLOUD_TOKEN:-}" && "${arg}" == *"${NETDATA_CLOUD_TOKEN}"* ]]; then
137
+ arg="${arg//${NETDATA_CLOUD_TOKEN}/<CLOUD_TOKEN>}"
138
+ fi
139
+ # Mask any UUID-shaped bearer in `Bearer <uuid>` form.
140
+ if [[ "${arg}" =~ Bearer\ [0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12} ]]; then
141
+ arg="${arg%% Bearer *} Bearer <AGENT_BEARER>"
142
+ # The replace above also rebuilds the leading header
143
+ # name; the rejoined arg is harmless even if the
144
+ # leading text is the bare header. Tests cover this.
145
+ fi
146
+ printf >&2 '%q ' "${arg}"
147
+ done
148
+ printf >&2 '%s\n' "${AGENTS_NC}"
149
+}
150
+
151
+# ---------------------------------------------------------------------------
152
+# Internal: claim_id / bearer mint / cache
153
+# ---------------------------------------------------------------------------
154
+
155
+# Resolve claim_id from a node's /api/v3/info. The /info endpoint
156
+# is unauthenticated. INTERNAL: returns via nameref into a caller
157
+# local; never prints to stdout.
158
+#
159
+# Args:
160
+# $1 = OUTVAR -- caller-local variable name to receive the claim_id
161
+# $2 = HOST -- host:port (e.g. "agent-events:19999")
162
+_agents_get_claim_id() {
163
+ local -n _out="$1"; shift
164
+ local host="${1:?usage: _agents_get_claim_id OUTVAR <host:port>}"
165
+ local resp claim
166
+ if resp="$(curl -sS --max-time 10 "http://${host}/api/v3/info" 2>/dev/null)"; then
167
+ claim="$(jq -r '.agents[0].cloud.claim_id // empty' <<< "${resp}" 2>/dev/null)"
168
+ if [[ -n "${claim}" && "${claim}" != "null" ]]; then
169
+ _out="${claim}"
170
+ return 0
171
+ fi
172
+ fi
173
+ echo -e "${AGENTS_RED}[ERROR]${AGENTS_NC} Could not resolve claim_id from http://${host}/api/v3/info" >&2
174
+ return 1
175
+}
176
+
177
+# Mint a per-agent bearer via Cloud. INTERNAL: prints the response
178
+# JSON to stdout for the caller to capture in a local variable.
179
+# stdout still carries the bearer here -- callers MUST capture into
180
+# a local and never propagate. The PUBLIC wrapper that uses this
181
+# does exactly that and emits only the response body.
182
+_agents_mint_bearer_json() {
183
+ local node_id="${1:?usage: _agents_mint_bearer_json <node_id> <machine_guid> <claim_id>}"
184
+ local mg="${2:?machine_guid required}"
185
+ local claim="${3:?claim_id required}"
186
+ agents_run_read curl --fail --silent --show-error --max-time 30 \
187
+ -H "Authorization: Bearer ${NETDATA_CLOUD_TOKEN}" \
188
+ "https://${NETDATA_CLOUD_HOSTNAME}/api/v2/bearer_get_token?node_id=${node_id}&machine_guid=${mg}&claim_id=${claim}"
189
+}
190
+
191
+# Convert an `expiration` value (which may be unix-seconds or
192
+# unix-milliseconds, depending on cloud version) to seconds.
193
+# Heuristic: values > 10^12 are ms; lower are seconds. Returns 0
194
+# for unparseable values so the caller treats the cache as expired.
195
+_agents_exp_to_seconds() {
196
+ local exp="$1"
197
+ if [[ -z "${exp}" || "${exp}" == "null" ]]; then
198
+ echo 0; return
199
+ fi
200
+ if ! [[ "${exp}" =~ ^[0-9]+$ ]]; then
201
+ echo 0; return
202
+ fi
203
+ if (( exp > 1000000000000 )); then
204
+ echo $(( exp / 1000 ))
205
+ else
206
+ echo "${exp}"
207
+ fi
208
+}
209
+
210
+# Cache-aware bearer resolution. INTERNAL: returns via nameref;
211
+# never prints the bearer to stdout.
212
+#
213
+# Args:
214
+# $1 = OUTVAR -- caller-local variable to receive the bearer
215
+# $2 = NODE_ID -- node UUID
216
+# $3 = MACHINE_GUID -- agent machine_guid (cache key)
217
+# $4 = HOST -- host:port for claim_id resolution and direct probe
218
+#
219
+# Cache file: <repo>/.local/audits/query-netdata-agents/bearers/<machine_guid>.json
220
+# Mode 0600. Stamps `_cached_at` (unix-seconds) so the cache window
221
+# survives Cloud responses with expiration=0.
222
+_agents_resolve_bearer() {
223
+ local -n _out="$1"; shift
224
+ local node_id="${1:?usage: _agents_resolve_bearer OUTVAR <node_id> <machine_guid> <host:port>}"
225
+ local mg="${2:?machine_guid required}"
226
+ local host="${3:?host required}"
227
+
228
+ local cache_dir cache_file now exp_s
229
+ cache_dir="$(agents_audit_dir)/bearers"
230
+ mkdir -p "${cache_dir}"
231
+ chmod 0700 "${cache_dir}" 2>/dev/null || true
232
+ cache_file="${cache_dir}/${mg}.json"
233
+
234
+ now=$(date +%s)
235
+
236
+ if [[ -s "${cache_file}" ]]; then
237
+ local cached_exp cached_token cached_at
238
+ cached_exp=$(jq -r '.expiration // 0' "${cache_file}" 2>/dev/null || echo 0)
239
+ cached_token=$(jq -r '.token // empty' "${cache_file}" 2>/dev/null || true)
240
+ cached_at=$(jq -r '._cached_at // 0' "${cache_file}" 2>/dev/null || echo 0)
241
+ exp_s=$(_agents_exp_to_seconds "${cached_exp}")
242
+ if [[ -n "${cached_token}" && "${cached_token}" != "null" ]]; then
243
+ # Two cases:
244
+ # (a) Cloud returned a real expiration -- 1h refresh buffer
245
+ # (matches cloud-frontend useAgentBearer.js).
246
+ # (b) Cloud returned expiration=0 -- fall back to a fixed
247
+ # 2h window from our mint timestamp. The agent issues
248
+ # ~3h-TTL bearers, so 2h leaves a 1h safety margin.
249
+ if (( exp_s > 0 )); then
250
+ if (( exp_s - now > 3600 )); then
251
+ _out="${cached_token}"
252
+ return 0
253
+ fi
254
+ elif (( cached_at > 0 )) && (( now - cached_at < 7200 )); then
255
+ _out="${cached_token}"
256
+ return 0
257
+ fi
258
+ fi
259
+ fi
260
+
261
+ # Need to mint -- resolve claim_id first.
262
+ local claim
263
+ _agents_get_claim_id claim "${host}"
264
+
265
+ local resp
266
+ resp="$(_agents_mint_bearer_json "${node_id}" "${mg}" "${claim}")"
267
+ if ! jq -e '.token' >/dev/null 2>&1 <<< "${resp}"; then
268
+ rm -f "${cache_file}"
269
+ echo -e "${AGENTS_RED}[ERROR]${AGENTS_NC} Bearer mint failed; first 200 chars: $(head -c 200 <<< "${resp}")" >&2
270
+ return 1
271
+ fi
272
+
273
+ # Stamp cache and persist.
274
+ jq --argjson t "${now}" '. + {_cached_at: $t}' <<< "${resp}" > "${cache_file}"
275
+ chmod 0600 "${cache_file}"
276
+ _out="$(jq -r '.token' "${cache_file}")"
277
+}
278
+
279
+# ---------------------------------------------------------------------------
280
+# PUBLIC wrappers (token-safe). These are what the assistant invokes.
281
+# ---------------------------------------------------------------------------
282
+
283
+# Call any Netdata Cloud REST endpoint. Reads NETDATA_CLOUD_TOKEN
284
+# from .env internally; emits ONLY the response body to stdout.
285
+# stderr shows the curl invocation with `<CLOUD_TOKEN>` masked.
286
+#
287
+# Args:
288
+# $1 = METHOD -- GET / POST / PUT / DELETE / ...
289
+# $2 = PATH -- e.g. /api/v2/spaces
290
+# $3 = BODY (json) -- optional; passed via -d
291
+#
292
+# Example:
293
+# agents_query_cloud GET /api/v2/spaces
294
+# agents_query_cloud POST /api/v2/nodes/$NODE/function?function=systemd-journal '{"info":true}'
295
+agents_query_cloud() {
296
+ local method="${1:?usage: agents_query_cloud METHOD PATH [BODY]}"
297
+ local path="${2:?path required}"
298
+ local body="${3:-}"
299
+
300
+ local args=(curl --fail --silent --show-error --max-time 120 -X "${method}" \
301
+ -H "Authorization: Bearer ${NETDATA_CLOUD_TOKEN}" \
302
+ -H 'Content-Type: application/json' \
303
+ "https://${NETDATA_CLOUD_HOSTNAME}${path}")
304
+ if [[ -n "${body}" ]]; then
305
+ args+=(-d "${body}")
306
+ fi
307
+ agents_run "${args[@]}"
308
+}
309
+
310
+# Call any Netdata Agent direct-HTTP path. Resolves the per-agent
311
+# bearer internally (cache or mint via Cloud). Emits ONLY the
312
+# response body to stdout. stderr shows curl with both
313
+# `<CLOUD_TOKEN>` and `<AGENT_BEARER>` masked.
314
+#
315
+# Required flags (provided in any order before METHOD PATH):
316
+# --node <node_id> -- target node UUID
317
+# --host <host:port> -- agent's bind, e.g. "agent-events:19999"
318
+# --machine-guid <mg> -- agent's machine_guid (bearer cache key)
319
+#
320
+# Example:
321
+# agents_query_agent --node $NODE --host $HOST --machine-guid $MG \
322
+# POST /api/v3/function?function=systemd-journal '{"info":true}'
323
+agents_query_agent() {
324
+ local node="" host="" mg="" method="" path="" body=""
325
+ while (( $# > 0 )); do
326
+ local arg="$1"
327
+ case "$arg" in
328
+ --node) node="${2-}"; shift 2 ;;
329
+ --host) host="${2-}"; shift 2 ;;
330
+ --machine-guid) mg="${2-}"; shift 2 ;;
331
+ --) shift; break ;;
332
+ -*)
333
+ echo -e "${AGENTS_RED}[ERROR]${AGENTS_NC} Unknown flag: $arg" >&2
334
+ return 1
335
+ ;;
336
+ *) break ;;
337
+ esac
338
+ done
339
+ method="${1:?usage: agents_query_agent --node N --host H --machine-guid M METHOD PATH [BODY]}"
340
+ path="${2:?path required}"
341
+ body="${3:-}"
342
+
343
+ : "${node:?--node required}"
344
+ : "${host:?--host required}"
345
+ : "${mg:?--machine-guid required}"
346
+
347
+ local bearer
348
+ _agents_resolve_bearer bearer "${node}" "${mg}" "${host}"
349
+
350
+ local args=(curl --fail --silent --show-error --max-time 120 -X "${method}" \
351
+ -H "X-Netdata-Auth: Bearer ${bearer}" \
352
+ -H 'Content-Type: application/json' \
353
+ "http://${host}/host/${node}${path}")
354
+ if [[ -n "${body}" ]]; then
355
+ args+=(-d "${body}")
356
+ fi
357
+ agents_run "${args[@]}"
358
+}
359
+
360
+# Convenience: Function call with automatic transport selection.
361
+# Wraps agents_query_cloud (preferred) or agents_query_agent.
362
+#
363
+# Flags:
364
+# --via cloud|agent default: cloud
365
+# --node <node_id> REQUIRED
366
+# --host <host:port> REQUIRED for --via agent
367
+# --machine-guid <mg> REQUIRED for --via agent
368
+# --function <name> REQUIRED (e.g. systemd-journal)
369
+# --body <json> default: {"info":true}
370
+agents_call_function() {
371
+ local via="cloud" node="" mg="" host="" fn="" body='{"info":true}'
372
+ while (( $# > 0 )); do
373
+ local arg="$1"
374
+ case "$arg" in
375
+ --via) via="${2-}"; shift 2 ;;
376
+ --node) node="${2-}"; shift 2 ;;
377
+ --machine-guid) mg="${2-}"; shift 2 ;;
378
+ --host) host="${2-}"; shift 2 ;;
379
+ --function) fn="${2-}"; shift 2 ;;
380
+ --body) body="${2-}"; shift 2 ;;
381
+ *)
382
+ echo -e "${AGENTS_RED}[ERROR]${AGENTS_NC} Unknown arg: $arg" >&2
383
+ return 1
384
+ ;;
385
+ esac
386
+ done
387
+ : "${node:?--node required}"
388
+ : "${fn:?--function required}"
389
+
390
+ case "${via}" in
391
+ cloud)
392
+ agents_query_cloud POST "/api/v2/nodes/${node}/function?function=${fn}" "${body}"
393
+ ;;
394
+ agent)
395
+ : "${mg:?--machine-guid required for --via agent}"
396
+ : "${host:?--host required for --via agent}"
397
+ agents_query_agent --node "${node}" --host "${host}" --machine-guid "${mg}" \
398
+ POST "/api/v3/function?function=${fn}" "${body}"
399
+ ;;
400
+ *)
401
+ echo -e "${AGENTS_RED}[ERROR]${AGENTS_NC} Unknown --via: ${via}" >&2
402
+ return 1
403
+ ;;
404
+ esac
405
+}
406
+
407
+# ---------------------------------------------------------------------------
408
+# Self-test: assert no token bytes leak through public wrappers.
409
+# Run with: bash -c 'source _lib.sh; agents_selftest_no_token_leak'
410
+# ---------------------------------------------------------------------------
411
+
412
+agents_selftest_no_token_leak() {
413
+ local sentinel='UNIQUE_SENTINEL_TOKEN_xK4mP7qR9sT2vW8y'
414
+ local fake_bearer='deadbeef-1234-5678-9abc-def012345678'
415
+
416
+ # Save real values, swap in sentinels, run wrappers in dry-run,
417
+ # capture stdout, restore.
418
+ local real_token="${NETDATA_CLOUD_TOKEN:-}"
419
+ local real_host="${NETDATA_CLOUD_HOSTNAME:-app.netdata.cloud}"
420
+ NETDATA_CLOUD_TOKEN="${sentinel}"
421
+ NETDATA_CLOUD_HOSTNAME="${real_host}"
422
+ AGENTS_DRY_RUN=1
423
+
424
+ local out=""
425
+
426
+ # 1. agents_query_cloud should not echo the sentinel.
427
+ out="$(agents_query_cloud GET /api/v2/spaces 2>/dev/null || true)"
428
+ if [[ "${out}" == *"${sentinel}"* ]]; then
429
+ echo -e "${AGENTS_RED}[FAIL]${AGENTS_NC} agents_query_cloud leaked NETDATA_CLOUD_TOKEN to stdout" >&2
430
+ NETDATA_CLOUD_TOKEN="${real_token}"; unset AGENTS_DRY_RUN
431
+ return 1
432
+ fi
433
+
434
+ # 2. _agents_log_masked must mask Bearer <uuid> patterns.
435
+ out="$(_agents_log_masked curl -H "Authorization: Bearer ${sentinel}" \
436
+ -H "X-Netdata-Auth: Bearer ${fake_bearer}" \
437
+ https://example.invalid 2>&1 1>/dev/null)"
438
+ if [[ "${out}" == *"${sentinel}"* ]]; then
439
+ echo -e "${AGENTS_RED}[FAIL]${AGENTS_NC} _agents_log_masked leaked NETDATA_CLOUD_TOKEN to stderr" >&2
440
+ NETDATA_CLOUD_TOKEN="${real_token}"; unset AGENTS_DRY_RUN
441
+ return 1
442
+ fi
443
+ if [[ "${out}" == *"${fake_bearer}"* ]]; then
444
+ echo -e "${AGENTS_RED}[FAIL]${AGENTS_NC} _agents_log_masked leaked Bearer <uuid> to stderr" >&2
445
+ NETDATA_CLOUD_TOKEN="${real_token}"; unset AGENTS_DRY_RUN
446
+ return 1
447
+ fi
448
+
449
+ # 3. The unit test passes if both checks above passed.
450
+ NETDATA_CLOUD_TOKEN="${real_token}"
451
+ unset AGENTS_DRY_RUN
452
+ echo -e "${AGENTS_GREEN}[PASS]${AGENTS_NC} no-token-leak self-test" >&2
453
+ return 0
454
+}
docs/netdata-ai/skills/query-netdata-agents/verify/questions.md
new
+118
@@ -0,0 +1,118 @@
1
+# query-netdata-agents -- verification questions (seed list)
2
+
3
+This file is the **seed input** consumed by the verification
4
+harness in SOW-0006 for direct-agent queries. The harness spawns
5
+a Sonnet-class assistant with `../SKILL.md` + `../how-tos/INDEX.md`
6
++ the canonical reference docs as context, asks each question
7
+below, captures the transcript, and grades against `grader.md`.
8
+
9
+When the assistant cannot answer or has to perform analysis not
10
+already documented under `../how-tos/`, the assistant must author
11
+a new how-to and add it to the index before completing.
12
+
13
+## Anchor: target nodes
14
+
15
+Two targets:
16
+
17
+- **Local desktop**: the agent reachable at `http://localhost:19999`
18
+ (typically the user's `costa-desktop`).
19
+- **Remote agent-events node**: the agent reachable at
20
+ `http://${AGENT_EVENTS_HOSTNAME}:19999` with node UUID
21
+ `${AGENT_EVENTS_NODE_ID}` and machine_guid
22
+ `${AGENT_EVENTS_MACHINE_GUID}`.
23
+
24
+Both use the bearer-mint flow. The harness verifies the wrapper
25
+mints / caches / refreshes correctly for both.
26
+
27
+## Identity (direct)
28
+
29
+- **Q01** -- Read the agent's `/api/v3/info` directly. What is the
30
+ node UUID, machine_guid, claim_id, agent version, and
31
+ hostname?
32
+- **Q02** -- What is the install prefix detected by
33
+ `agents_netdata_prefix` on the local desktop?
34
+
35
+## Streaming (agent-only -- Cloud has no equivalent)
36
+
37
+- **Q03** -- Run the `netdata-streaming` Function on the agent.
38
+ Is it acting as a parent (any incoming-direction rows)? If so,
39
+ how many children, and what's the replication progress per
40
+ child?
41
+- **Q04** -- Is the agent acting as a child (any outgoing-
42
+ direction row)? If so, what is the upstream parent host /
43
+ endpoint?
44
+
45
+## DynCfg (direct)
46
+
47
+- **Q05** -- Use `GET /api/v3/config?action=tree&path=/` to list
48
+ every configuration object on the agent. Group them by the
49
+ top-level path (e.g. `/collectors/go.d/Jobs`,
50
+ `/health/alerts/prototypes`, etc.) and show the count per
51
+ group.
52
+- **Q06** -- For one collector job (your choice), get its JSON
53
+ Schema via `action=schema` and its current value via
54
+ `action=get`.
55
+- **Q07** -- Are there any vnodes? Use
56
+ `path=/collectors/go.d/Vnodes` (and `ibm.d/Vnodes`).
57
+
58
+## Functions (direct)
59
+
60
+- **Q08** -- Discover every Function registered on the agent
61
+ (use the listing endpoint or info-walk pattern). Group by
62
+ family (table snapshot vs log explorer vs topology vs flows
63
+ vs other).
64
+- **Q09** -- For each of `processes`, `network-connections`,
65
+ `mount-points`, call with `{"info":true}` and report the
66
+ parameter set.
67
+
68
+## Logs (direct)
69
+
70
+- **Q10** -- Tail the last 10 entries of the system journal on
71
+ the local desktop.
72
+- **Q11** -- Find the last error-priority entry written to the
73
+ systemd journal in the last hour.
74
+
75
+## Alerts (direct)
76
+
77
+- **Q12** -- Use `POST /api/v3/alerts` with
78
+ `{"options":["instances"]}` to list currently-firing alerts
79
+ on the agent. Pick one with status CRITICAL or WARNING and
80
+ fetch its full config via `GET /api/v3/alert_config?config=...`.
81
+- **Q13** -- Use `POST /api/v3/alert_transitions` to find every
82
+ CLEAR -> CRITICAL transition in the last hour.
83
+
84
+## Metrics (direct)
85
+
86
+- **Q14** -- Use `POST /api/v3/data` to find the maximum
87
+ `system.cpu` user dimension over the last hour, points=60.
88
+- **Q15** -- Use `GET /api/v3/contexts` to list every metric
89
+ context the agent currently collects, sorted alphabetically.
90
+
91
+## Topology (direct)
92
+
93
+- **Q16** -- Run `topology:snmp` against the local desktop with
94
+ `{"info":true}` and report `accepted_params`. (If `topology:
95
+ snmp` is not registered on the local desktop because no SNMP
96
+ collector is configured, say so explicitly.)
97
+
98
+## Flows (direct)
99
+
100
+- **Q17** -- Run `flows:netflow` against the local desktop with
101
+ `{"info":true}` and report `accepted_params`. (If
102
+ `flows:netflow` is not registered, say so explicitly.)
103
+
104
+## Token-safety self-test
105
+
106
+- **Q18** -- Run `agents_selftest_no_token_leak`. It must print
107
+ `[PASS]` to stderr. The captured stdout of every wrapper
108
+ invocation in this session must not contain
109
+ `NETDATA_CLOUD_TOKEN` bytes, `X-Netdata-Auth: Bearer
110
+ <real-uuid>`, or any cached-bearer UUID from
111
+ `<repo>/.local/audits/query-netdata-agents/bearers/`.
112
+
113
+## Cross-skill (depends on the cloud skill)
114
+
115
+- **Q19** -- Pick a node UUID from the Cloud `/nodes` listing
116
+ (uses the cloud skill's `query-nodes.md`), then call
117
+ `agents_query_agent` directly against it (this skill).
118
+ Confirm both transports return the same `host[0].nm`.
docs/netdata-ai/skills/query-netdata-cloud/SKILL.md
new
+233
@@ -0,0 +1,233 @@
1
+---
2
+name: query-netdata-cloud
3
+description: Query Netdata Cloud via its REST API -- metrics, logs (systemd-journal / windows-events / otel-logs), topology graphs (topology:snmp), network flows (flows:netflow), alerts, dynamic configuration (DynCfg), and generic Functions on a node. Use when the user asks about querying Netdata Cloud, fetching metrics from the cloud, querying logs / topology / netflow / sflow / ipfix through Cloud, listing or modifying configurations via DynCfg, calling agent Functions through Cloud, listing spaces/rooms/nodes, or building a curl command against `app.netdata.cloud`. Pairs with the `query-netdata-agents` skill when direct-agent access is needed.
4
+---
5
+
6
+# Query Netdata Cloud via REST API
7
+
8
+This skill teaches end-users (and AI assistants helping them) how to
9
+construct REST API queries against Netdata Cloud
10
+(`https://app.netdata.cloud`) using a long-lived API token.
11
+
12
+It is split into one shared overview (this file) and four
13
+domain-specific guides. Each guide is self-contained and includes
14
+runnable curl commands.
15
+
16
+| Domain | Guide |
17
+|---|---|
18
+| Time-series metrics | [query-metrics.md](./query-metrics.md) |
19
+| Logs (`systemd-journal`, `windows-events`, `otel-logs`) | [query-logs.md](./query-logs.md) |
20
+| Topology Functions (`topology:snmp`, ...) | [query-topology.md](./query-topology.md) |
21
+| Network-flow Functions (`flows:netflow` -- NetFlow / sFlow / IPFIX) | [query-flows.md](./query-flows.md) |
22
+| Alerts and alert transitions | [query-alerts.md](./query-alerts.md) |
23
+| Dynamic Configuration (DynCfg) | [query-dyncfg.md](./query-dyncfg.md) |
24
+| Generic Function invocation (table snapshots + protocol taxonomy) | [query-functions.md](./query-functions.md) |
25
+| Nodes (per-room enumeration with full metadata) | [query-nodes.md](./query-nodes.md) |
26
+| Rooms (per-space enumeration) | [query-rooms.md](./query-rooms.md) |
27
+| Members (per-space user enumeration) | [query-members.md](./query-members.md) |
28
+| Event feed (audit + activity log) | [query-feed.md](./query-feed.md) |
29
+| **Operational how-tos (live catalog)** | [how-tos/INDEX.md](./how-tos/INDEX.md) |
30
+| **Verification questions (consumed by SOW-0006 harness)** | [verify/questions.md](./verify/questions.md) |
31
+
32
+### Canonical reference docs (in this repo)
33
+
34
+For the protocol-level details these guides build on, read the
35
+authoritative sources directly:
36
+
37
+| File | What it covers |
38
+|---|---|
39
+| `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` | Functions v3 protocol -- envelope, simple-table vs log-explorer, facets, histograms, charts, field types, pagination, delta mode, PLAY mode, error handling. The single most important reference for any Function work. |
40
+| `<repo>/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` | Practical guide for collector authors implementing a Function (simple-table or log-explorer) |
41
+| `<repo>/src/plugins.d/FUNCTION_UI_SCHEMA.json` | JSON Schema for validating Function responses |
42
+| `<repo>/src/plugins.d/DYNCFG.md` | External-plugin DynCfg protocol (go.d.plugin and other external collectors) |
43
+| `<repo>/src/daemon/dyncfg/README.md` | Internal DynCfg (high-level and low-level APIs, command enums, lifecycle) |
44
+| `<repo>/src/database/rrdfunctions.h` | C-level Function registration API (`rrd_function_add`) |
45
+| `<repo>/src/go/plugin/framework/functions/README.md` | Go-plugin Function framework |
46
+
47
+For querying agents directly (without going through Cloud) -- including
48
+auto-minting agent bearer tokens from a Cloud token -- see the sibling
49
+skill [`query-netdata-agents`](../query-netdata-agents/SKILL.md).
50
+
51
+---
52
+
53
+## Mandatory Requirements (READ FIRST)
54
+
55
+1. **If you analyze, you author a how-to.** When asked a concrete
56
+ question about a Netdata environment that isn't already covered
57
+ by an existing how-to under [`how-tos/`](./how-tos/), you MUST
58
+ author a new how-to in this directory and add it to
59
+ [`how-tos/INDEX.md`](./how-tos/INDEX.md) BEFORE completing the
60
+ task. The catalog is meant to be **live** -- the next assistant
61
+ should not redo the same analysis from scratch.
62
+2. **Use the token-safe wrappers.** Every example in this skill
63
+ uses `agents_query_cloud` (and friends) from
64
+ `../query-netdata-agents/scripts/_lib.sh`. Never paste raw
65
+ `Authorization: Bearer $TOKEN` curl commands -- that exposes
66
+ the cloud token to the assistant. The wrappers handle auth
67
+ internally and emit only the response body to stdout.
68
+3. **Provide actionable instructions.** You don't run queries for
69
+ users. Your role is to teach them. Every response that proposes a
70
+ query must end in a complete, runnable command (wrapper-based,
71
+ not raw curl).
72
+
73
+2. **Never ask for credentials.** Do not request API tokens, Space
74
+ IDs, or Room IDs. Use placeholders (`YOUR_API_TOKEN`,
75
+ `YOUR_SPACE_ID`, `YOUR_ROOM_ID`) at the top of your curl examples
76
+ so the user fills them in locally.
77
+
78
+3. **Always include a runnable curl command.** A response without a
79
+ complete `curl -X METHOD ... -H ... -d '...'` block is incomplete.
80
+ Use a heredoc for the JSON body so the user does not have to
81
+ escape quotes:
82
+
83
+ ```bash
84
+ read -r -d '' PAYLOAD <<'EOF'
85
+ { "scope": { "contexts": ["system.cpu"] }, ... }
86
+ EOF
87
+ ```
88
+
89
+4. **Domain-specific gotchas live in the per-domain guide.** For
90
+ metrics, the most important is `scope.contexts` MUST be set. See
91
+ the per-domain guide for the rest.
92
+
93
+---
94
+
95
+## Prerequisites
96
+
97
+Three things are needed for any query:
98
+
99
+### 1. API Token
100
+
101
+1. Login to [app.netdata.cloud](https://app.netdata.cloud)
102
+2. Click the user icon (lower-left corner -- tooltip shows your name)
103
+3. Select **User Settings**
104
+4. Open the **API Tokens** tab
105
+5. Click the **[+]** button (top-left)
106
+6. Pick a scope, enter a description, click **Create**
107
+7. **Copy the token immediately** -- it is shown once.
108
+
109
+Recommended scope: `scope:all` (full access) or `scope:grafana-plugin`
110
+(read-only data endpoints).
111
+
112
+### 2. Space ID
113
+
114
+1. In the dashboard, click the **gear icon** below the spaces list
115
+ (tooltip: "Space Settings")
116
+2. In the **Info** tab, copy the **Space Id**.
117
+
118
+### 3. Room ID
119
+
120
+1. In Space Settings, open the **Rooms** tab
121
+2. Click the **>** icon at the right of the row (tooltip: "Room
122
+ Settings")
123
+3. In the **Room** tab, copy the **Room Id**.
124
+
125
+---
126
+
127
+## Authentication
128
+
129
+All endpoints accept the cloud token as an HTTP `Authorization`
130
+header:
131
+
132
+```
133
+Authorization: Bearer YOUR_API_TOKEN
134
+Content-Type: application/json (for POST endpoints)
135
+```
136
+
137
+GET endpoints do not require the `Content-Type` header but accept it.
138
+
139
+---
140
+
141
+## Discovery Endpoints (used by every domain)
142
+
143
+These endpoints enumerate what the cloud token can see. Use them when
144
+you don't know the Space ID, Room ID, or node UUID up front.
145
+
146
+| Endpoint | Method | Purpose |
147
+|---|---|---|
148
+| `/api/v2/accounts/me` | GET | Confirm the token works; returns the user identity. |
149
+| `/api/v2/spaces` | GET | List spaces visible to this token. |
150
+| `/api/v2/spaces/{spaceID}/rooms` | GET | List rooms in a space. |
151
+| `/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes` | POST `{}` | List nodes in a room with full metadata. |
152
+
153
+### Example: list spaces
154
+
155
+```bash
156
+TOKEN="YOUR_API_TOKEN"
157
+
158
+curl -sS \
159
+ -H "Authorization: Bearer $TOKEN" \
160
+ "https://app.netdata.cloud/api/v2/spaces"
161
+```
162
+
163
+Each space record contains `id`, `slug`, `name`, `permissions[]`, and
164
+metadata. Match by `name` or `slug` to find the space you want.
165
+
166
+### Example: list rooms in a space
167
+
168
+```bash
169
+TOKEN="YOUR_API_TOKEN"
170
+SPACE="YOUR_SPACE_ID"
171
+
172
+curl -sS \
173
+ -H "Authorization: Bearer $TOKEN" \
174
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms"
175
+```
176
+
177
+### Example: list nodes in a room
178
+
179
+```bash
180
+TOKEN="YOUR_API_TOKEN"
181
+SPACE="YOUR_SPACE_ID"
182
+ROOM="YOUR_ROOM_ID"
183
+
184
+curl -sS -X POST \
185
+ -H 'Content-Type: application/json' \
186
+ -H "Authorization: Bearer $TOKEN" \
187
+ "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" \
188
+ -d '{}'
189
+```
190
+
191
+Per-node response fields:
192
+
193
+| Field | Description |
194
+|---|---|
195
+| `nd` | Node UUID -- required for any node-targeted call |
196
+| `mg` | Machine GUID |
197
+| `nm` | Hostname |
198
+| `state` | `reachable` (live) or `stale` (disconnected) |
199
+| `v` | Agent version |
200
+| `labels` | Key-value labels |
201
+| `hw`, `os`, `health`, `capabilities` | Metadata blocks |
202
+
203
+The `nd` value is what the four domain guides call "node UUID" or
204
+`{nodeId}` in their endpoint paths.
205
+
206
+---
207
+
208
+## Common errors
209
+
210
+| Symptom | Likely cause |
211
+|---|---|
212
+| HTTP 401 | Token missing, malformed, or revoked. Re-create. |
213
+| HTTP 403 | Token lacks the scope/role for this endpoint or space. |
214
+| HTTP 404 with HTML body | Wrong path; check method (GET vs POST) and version (`/api/v2` vs `/api/v3`). The API does not enumerate paths via Swagger, so 404 means the path does not exist. |
215
+| HTTP 400 with `errorCode` JSON | Missing required parameter. The error message names the missing field. |
216
+| Empty/silent response | Filter excludes everything. Most endpoints return empty data without error. Verify scope/selectors. |
217
+
218
+---
219
+
220
+## Sensitive data
221
+
222
+Cloud responses contain space names, node hostnames, machine GUIDs,
223
+node UUIDs, claim IDs, cloud-provider labels, IP addresses, and other
224
+identifiers. Treat fetched payloads as personal/customer data:
225
+
226
+- Do not paste raw response bodies into committed files.
227
+- Do not paste tokens, bearer values, or session ids anywhere.
228
+- For maintainer workflows in this repository, redirect raw output
229
+ to `<repo>/.local/audits/...` (gitignored) and report only
230
+ sanitized summaries upstream.
231
+
232
+See `<repo>/.agents/sow/specs/sensitive-data-discipline.md` for the
233
+full rule and the pre-commit verification grep.
docs/netdata-ai/skills/query-netdata-cloud/how-tos/INDEX.md
new
+102
@@ -0,0 +1,102 @@
1
+# query-netdata-cloud -- How-tos index
2
+
3
+This directory holds **operational how-tos**: short, focused
4
+recipes that combine the per-domain guides into answers for
5
+specific questions. Each how-to documents the question, the steps
6
+taken, the wrappers used, and the expected output shape.
7
+
8
+## The "if you analyze, you author a how-to" rule
9
+
10
+The how-tos catalog is meant to be **live**. Every time an AI
11
+assistant (or human) is asked a question that:
12
+
13
+1. The user expects a concrete answer to, AND
14
+2. Is not already documented in this index, AND
15
+3. Forces analysis (multiple wrapper calls, jq pipelines, or
16
+ cross-referencing more than one per-domain guide)
17
+
18
+the assistant MUST author a new how-to in this directory and add
19
+it to the index BELOW before completing the task.
20
+
21
+This is mandatory. Skipping it means the next assistant repeats
22
+the same analysis from scratch.
23
+
24
+## How-to authoring template
25
+
26
+Filename: `<slug>.md` (e.g. `find-node-id-by-hostname.md`).
27
+
28
+Sections:
29
+
30
+1. **Question** -- the user-visible question, verbatim or
31
+ paraphrased.
32
+2. **Inputs** -- what the user must supply (space, hostname,
33
+ time range, etc.).
34
+3. **Steps** -- numbered, each calling exactly one wrapper from
35
+ `query-netdata-agents/scripts/_lib.sh`.
36
+4. **Output** -- what the assistant returns to the user.
37
+5. **Notes / gotchas** -- edge cases, follow-ups, related
38
+ how-tos.
39
+6. **Source guides** -- cross-links to the per-domain guides
40
+ used.
41
+
42
+Every code example must use the token-safe wrappers
43
+(`agents_query_cloud`, `agents_query_agent`,
44
+`agents_call_function`). No raw curl with `-H "Authorization:
45
+Bearer $TOKEN"` -- that defeats the no-token-leak guarantee.
46
+
47
+## Index
48
+
49
+(Populate as how-tos are authored. Stubs below correspond to the
50
+seed verification questions in `../verify/questions.md`; replace
51
+each `(stub -- not yet authored)` with a real link as soon as a
52
+how-to is written.)
53
+
54
+### Identity / hardware / OS
55
+
56
+- `find-node-id-by-hostname.md` (stub -- not yet authored)
57
+- `find-node-hardware-specs.md` (stub -- not yet authored)
58
+- `find-node-os.md` (stub -- not yet authored)
59
+
60
+### Streaming / parents / vnodes
61
+
62
+- `is-node-a-parent-and-children.md` (stub -- not yet authored)
63
+- `is-node-a-child-and-parent-target.md` (stub -- not yet authored)
64
+- `list-vnodes-on-node.md` (stub -- not yet authored)
65
+
66
+### Collectors / jobs
67
+
68
+- `find-failed-collection-jobs.md` (stub -- not yet authored)
69
+- `is-collector-monitoring-X-and-frequency.md` (stub -- not yet authored)
70
+
71
+### Top processes
72
+
73
+- `pid-with-biggest-memory-and-app-group.md` (stub -- not yet authored)
74
+
75
+### Alerts
76
+
77
+- `currently-firing-alerts-in-room.md` (stub -- not yet authored)
78
+- `alert-config-by-cfg-hash.md` (stub -- not yet authored)
79
+- `silenced-alerts.md` (stub -- not yet authored)
80
+
81
+### Logs / status file
82
+
83
+- `last-netdata-status-file-log.md` (stub -- not yet authored)
84
+- `recent-error-logs-in-namespace.md` (stub -- not yet authored)
85
+
86
+### Topology / flows
87
+
88
+- `local-l2-topology-summary.md` (stub -- not yet authored)
89
+- `top-flow-talkers-last-hour.md` (stub -- not yet authored)
90
+
91
+### Members / rooms / feed
92
+
93
+- `members-by-role-in-space.md` (stub -- not yet authored)
94
+- `rooms-with-most-nodes.md` (stub -- not yet authored)
95
+- `node-state-changes-last-hour.md` (stub -- not yet authored)
96
+
97
+## Cross-skill how-tos
98
+
99
+When the answer needs both Cloud-side and direct-agent-side calls
100
+(e.g. "find the parent of a stale node, then read its
101
+streaming-state directly"), author the how-to under the skill
102
+that owns the FIRST wrapper call and cross-link to the other.
docs/netdata-ai/skills/query-netdata-cloud/query-alerts.md
new
+498
@@ -0,0 +1,498 @@
1
+# Query Netdata alerts via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+Alerts are exposed as **REST endpoints** -- not as Functions. Both
7
+Netdata Cloud and the Netdata Agent expose dedicated alert paths.
8
+Use the Cloud-proxied paths by default (no per-agent bearer needed).
9
+Use the agent-direct paths when you need single-host detail or when
10
+Cloud is unavailable (see the sibling
11
+[`query-netdata-agents`](../query-netdata-agents/SKILL.md) skill for
12
+direct-agent auth).
13
+
14
+---
15
+
16
+## Mandatory Requirements (READ FIRST)
17
+
18
+1. **Provide actionable instructions.** Every recommendation ends in
19
+ a runnable curl command.
20
+2. **Never request credentials.** Use `YOUR_API_TOKEN`,
21
+ `YOUR_SPACE_ID`, `YOUR_ROOM_ID` placeholders.
22
+3. **Always include a heredoc body.** Avoids quote-escaping pain.
23
+4. **Cloud and agent endpoints have different shapes.** Cloud
24
+ endpoints aggregate across nodes in a room/space. Agent
25
+ endpoints serve a single host. Pick the one that matches the
26
+ question.
27
+
28
+---
29
+
30
+## Cloud-side endpoints
31
+
32
+Base URL: `https://app.netdata.cloud`. All require
33
+`Authorization: Bearer YOUR_API_TOKEN` and the
34
+`PermissionAlertReadAll` role on the target space (notification
35
+silencing endpoints require write permission).
36
+
37
+### Current alerts in a room
38
+
39
+`POST /api/v2/spaces/{spaceID}/rooms/{roomID}/alerts`
40
+
41
+```bash
42
+TOKEN="YOUR_API_TOKEN"
43
+SPACE="YOUR_SPACE_ID"
44
+ROOM="YOUR_ROOM_ID"
45
+
46
+read -r -d '' PAYLOAD <<'EOF'
47
+{
48
+ "options": ["instances", "values", "summary", "config"]
49
+}
50
+EOF
51
+
52
+curl -sS -X POST \
53
+ -H 'Content-Type: application/json' \
54
+ -H "Authorization: Bearer $TOKEN" \
55
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms/$ROOM/alerts" \
56
+ -d "$PAYLOAD"
57
+```
58
+
59
+Body accepts optional filters: `status[]` (`CRITICAL`, `WARNING`,
60
+`CLEAR`, etc.), `name` pattern, `alarm_id_filter`, pagination
61
+(`offset`, `limit`), and a time window. Without
62
+`options.instances` the per-instance array is empty -- only the
63
+aggregated `alerts[]` summary is returned.
64
+
65
+Response top-level: `api`, `alerts[]` (one entry per template),
66
+`alert_instances[]` (one entry per running instance, when
67
+requested), `nodes[]`, `timings`. Per-instance compact fields
68
+(verified live):
69
+
70
+| Field | Meaning |
71
+|---|---|
72
+| `nm` | Alert name (e.g. `10min_cpu_iowait`) |
73
+| `ctx` | Context (e.g. `system.cpu`) |
74
+| `ch` / `ch_n` | Chart id / name |
75
+| `st` | Current status (`CRITICAL`, `WARNING`, `CLEAR`, ...) |
76
+| `v` | Current value |
77
+| `t` | Last evaluation timestamp (Unix seconds) |
78
+| `tr_i` | Last transition id (UUID) |
79
+| `tr_v` | Value at last transition |
80
+| `tr_t` | Timestamp of last transition |
81
+| `units` | Unit string |
82
+| `cfg` | **Config hash UUID** -- pass to `/alert_config` as `config` |
83
+| `exec` | Notification executable |
84
+| `tp` / `cl` / `cp` | Type / classification / component |
85
+| `to` | Notification role(s) |
86
+
87
+### Space-wide alarm stats
88
+
89
+`GET /api/v2/spaces/{spaceID}/alarms`
90
+
91
+```bash
92
+TOKEN="YOUR_API_TOKEN"
93
+SPACE="YOUR_SPACE_ID"
94
+
95
+curl -sS \
96
+ -H "Authorization: Bearer $TOKEN" \
97
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/alarms"
98
+```
99
+
100
+Returns total counts (`critical`, `warning`, `clear`, `silenced`)
101
+across all rooms in the space. Use to drive a dashboard summary.
102
+
103
+### Available alert templates / metas
104
+
105
+`GET /api/v2/spaces/{spaceID}/alarms/metas`
106
+
107
+```bash
108
+TOKEN="YOUR_API_TOKEN"
109
+SPACE="YOUR_SPACE_ID"
110
+
111
+curl -sS \
112
+ -H "Authorization: Bearer $TOKEN" \
113
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/alarms/metas"
114
+```
115
+
116
+Lists every alert template/prototype configured across the space:
117
+names, contexts, severities, available config hashes. Use this to
118
+discover what alerts exist before drilling into a specific one.
119
+
120
+### Per-room alert summary stats
121
+
122
+`GET /api/v2/spaces/{spaceID}/rooms/{roomID}/alerts_stats`
123
+
124
+```bash
125
+TOKEN="YOUR_API_TOKEN"
126
+SPACE="YOUR_SPACE_ID"
127
+ROOM="YOUR_ROOM_ID"
128
+
129
+curl -sS \
130
+ -H "Authorization: Bearer $TOKEN" \
131
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms/$ROOM/alerts_stats"
132
+```
133
+
134
+Same shape as `/alarms` but scoped to one room. Optional
135
+node-filter query params.
136
+
137
+### Misconfigured alerts
138
+
139
+`POST /api/v2/spaces/{spaceID}/rooms/{roomID}/alerts:misconfigured`
140
+
141
+```bash
142
+TOKEN="YOUR_API_TOKEN"
143
+SPACE="YOUR_SPACE_ID"
144
+ROOM="YOUR_ROOM_ID"
145
+
146
+read -r -d '' PAYLOAD <<'EOF'
147
+{
148
+ "categories": ["firing_often", "stuck_raised", "silenced_long", "dispatch_none"],
149
+ "thresholds": {
150
+ "firing_often_min_count": 10,
151
+ "stuck_raised_min_hours": 24
152
+ }
153
+}
154
+EOF
155
+
156
+curl -sS -X POST \
157
+ -H 'Content-Type: application/json' \
158
+ -H "Authorization: Bearer $TOKEN" \
159
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms/$ROOM/alerts:misconfigured" \
160
+ -d "$PAYLOAD"
161
+```
162
+
163
+Categories: `firing_often`, `stuck_raised`, `silenced_long`,
164
+`dispatch_none`. Returns alerts grouped by category with metrics so
165
+you can clean up noisy or broken alert configurations.
166
+
167
+### Alert state transitions (history)
168
+
169
+`POST /api/v2/spaces/{spaceID}/rooms/{roomID}/alert_transitions`
170
+
171
+```bash
172
+TOKEN="YOUR_API_TOKEN"
173
+SPACE="YOUR_SPACE_ID"
174
+ROOM="YOUR_ROOM_ID"
175
+# absolute Unix seconds; the endpoint rejects negative or 0 values.
176
+AFTER=$(( $(date +%s) - 86400 ))
177
+
178
+read -r -d '' PAYLOAD <<EOF
179
+{
180
+ "after": ${AFTER},
181
+ "before": $(date +%s),
182
+ "status": ["CRITICAL", "WARNING"]
183
+}
184
+EOF
185
+
186
+curl -sS -X POST \
187
+ -H 'Content-Type: application/json' \
188
+ -H "Authorization: Bearer $TOKEN" \
189
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms/$ROOM/alert_transitions" \
190
+ -d "$PAYLOAD"
191
+```
192
+
193
+`after` must be **absolute Unix seconds > 0** (verified live; the
194
+endpoint returns
195
+`{"errorMsgKey":"ErrBadRequest","errorMessage":"after parameter must be greater than 0",...}`
196
+otherwise). `before` is also Unix seconds (`0` is rejected; pass
197
+`now` or omit). With an empty body `{}` the endpoint applies its
198
+own default lookback.
199
+
200
+Optional filters: `status[]` (`CRITICAL`, `WARNING`, `CLEAR`, ...),
201
+`alert_names[]`, `node_ids[]`, `context[]`, plus pagination
202
+(`limit`, `last`).
203
+
204
+Response top-level: `api`, `transitions[]`. Each transition record:
205
+`transition_id`, `node_id`, `name`/`alert`, `instance`, `context`,
206
+`when` (unix-seconds), `new` / `old` (`{status, value}`), `summary`,
207
+`info`, `src`, `config_hash_id`, `component`, `classification`,
208
+`to`, `units`, `exec`.
209
+
210
+### Single alert configuration
211
+
212
+`POST /api/v2/spaces/{spaceID}/rooms/{roomID}/alert_config`
213
+
214
+```bash
215
+TOKEN="YOUR_API_TOKEN"
216
+SPACE="YOUR_SPACE_ID"
217
+ROOM="YOUR_ROOM_ID"
218
+
219
+read -r -d '' PAYLOAD <<'EOF'
220
+{
221
+ "config": "ALERT_CONFIG_HASH_UUID",
222
+ "node_id": "YOUR_NODE_UUID"
223
+}
224
+EOF
225
+
226
+curl -sS -X POST \
227
+ -H 'Content-Type: application/json' \
228
+ -H "Authorization: Bearer $TOKEN" \
229
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms/$ROOM/alert_config" \
230
+ -d "$PAYLOAD"
231
+```
232
+
233
+`config` is the hash UUID from the `cfg` field of an alert
234
+instance in the `/alerts` response (request
235
+`options:["instances","config"]` there to get it populated).
236
+Returns the full alert definition: top-level keys `name`, `info`,
237
+`class`, `component`, `selectors`, `status`, `notification`,
238
+`config_hash_id` (echo of input).
239
+
240
+### Evaluate an alert config against historical data
241
+
242
+`POST /api/v2/spaces/{spaceID}/rooms/{roomID}/alert_config/evaluate`
243
+
244
+```bash
245
+TOKEN="YOUR_API_TOKEN"
246
+SPACE="YOUR_SPACE_ID"
247
+ROOM="YOUR_ROOM_ID"
248
+
249
+read -r -d '' PAYLOAD <<'EOF'
250
+{
251
+ "node_id": "YOUR_NODE_UUID",
252
+ "config": "alarm: example_high_cpu\n on: system.cpu\n lookup: average -1m of user\n warn: $this > 70\n crit: $this > 90\n",
253
+ "after": -3600,
254
+ "before": 0
255
+}
256
+EOF
257
+
258
+curl -sS -X POST \
259
+ -H 'Content-Type: application/json' \
260
+ -H "Authorization: Bearer $TOKEN" \
261
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/rooms/$ROOM/alert_config/evaluate" \
262
+ -d "$PAYLOAD"
263
+```
264
+
265
+Replays the alert definition against real metric data over the
266
+window. Useful for tuning before deployment. Returns evaluation
267
+results showing what the alert would have done.
268
+
269
+### AI-assisted alert config generation
270
+
271
+Three companion endpoints that take a context/metric and either
272
+generate, suggest, or explain an alert configuration. All three are
273
+`POST` under `/api/v2/spaces/{spaceID}/alert-config/...`:
274
+
275
+| Endpoint | Purpose |
276
+|---|---|
277
+| `/alert-config/generate` | Produce a full config from a context+metric description |
278
+| `/alert-config/suggest` | Suggest several config variants |
279
+| `/alert-config/explain` | Explain in prose what an existing config does |
280
+
281
+```bash
282
+TOKEN="YOUR_API_TOKEN"
283
+SPACE="YOUR_SPACE_ID"
284
+
285
+read -r -d '' PAYLOAD <<'EOF'
286
+{
287
+ "context": "system.cpu",
288
+ "instance": "system",
289
+ "metric": "user"
290
+}
291
+EOF
292
+
293
+curl -sS -X POST \
294
+ -H 'Content-Type: application/json' \
295
+ -H "Authorization: Bearer $TOKEN" \
296
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/alert-config/generate" \
297
+ -d "$PAYLOAD"
298
+```
299
+
300
+### Notification silencing rules
301
+
302
+Silencing rules are Cloud-only (the agent has no silencing REST
303
+API). Five endpoints, all under
304
+`/api/v2/spaces/{spaceID}/notifications/silencing/`:
305
+
306
+| Path | Method | Purpose |
307
+|---|---|---|
308
+| `rules` | GET | List all silencing rules in the space (state: `INACTIVE`, `ACTIVE`, `SCHEDULED`) |
309
+| `rule` | POST | Create a rule |
310
+| `rule/{ruleID}` | PUT | Update a rule |
311
+| `rules/delete` | POST | Bulk-delete rules by ID list |
312
+| `rrule/evaluate` | POST | Evaluate an iCal-style RRULE recurrence expression |
313
+
314
+```bash
315
+TOKEN="YOUR_API_TOKEN"
316
+SPACE="YOUR_SPACE_ID"
317
+
318
+# List all silencing rules.
319
+curl -sS \
320
+ -H "Authorization: Bearer $TOKEN" \
321
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/notifications/silencing/rules"
322
+```
323
+
324
+Create-rule body:
325
+
326
+```bash
327
+TOKEN="YOUR_API_TOKEN"
328
+SPACE="YOUR_SPACE_ID"
329
+
330
+read -r -d '' PAYLOAD <<'EOF'
331
+{
332
+ "name": "Maintenance window for db cluster",
333
+ "room_ids": ["YOUR_ROOM_ID"],
334
+ "node_ids": [],
335
+ "host_labels": { "role": "database" },
336
+ "alert_names": [],
337
+ "alert_contexts": ["disk.space"],
338
+ "severities": ["WARNING", "CRITICAL"],
339
+ "starts_at": 1700000000,
340
+ "lasts_until": 1700003600,
341
+ "rrule": ""
342
+}
343
+EOF
344
+
345
+curl -sS -X POST \
346
+ -H 'Content-Type: application/json' \
347
+ -H "Authorization: Bearer $TOKEN" \
348
+ "https://app.netdata.cloud/api/v2/spaces/$SPACE/notifications/silencing/rule" \
349
+ -d "$PAYLOAD"
350
+```
351
+
352
+`rrule` is an iCalendar RFC 5545 recurrence string (e.g.
353
+`FREQ=WEEKLY;BYDAY=SA,SU`). Use `rrule/evaluate` first to confirm
354
+the schedule before creating.
355
+
356
+---
357
+
358
+## Direct-agent fallback (single-host alerts)
359
+
360
+When you need detail for a specific host or Cloud is unavailable,
361
+talk to the agent directly. All paths below are reachable at
362
+`http://<agent>:19999/host/<node-uuid>` and require a per-agent
363
+bearer if the agent is bearer-protected (see
364
+[query-netdata-agents](../query-netdata-agents/SKILL.md) for the
365
+mint flow).
366
+
367
+### Multi-status alerts (preferred -- agent v3)
368
+
369
+`POST /api/v3/alerts`
370
+
371
+```bash
372
+HOST="agent.example:19999"
373
+NODE="YOUR_NODE_UUID"
374
+BEARER="MINTED_AGENT_BEARER"
375
+
376
+read -r -d '' PAYLOAD <<'EOF'
377
+{
378
+ "options": ["summary", "values", "instances"]
379
+}
380
+EOF
381
+
382
+curl -sS -X POST \
383
+ -H "X-Netdata-Auth: Bearer $BEARER" \
384
+ -H 'Content-Type: application/json' \
385
+ "http://$HOST/host/$NODE/api/v3/alerts" \
386
+ -d "$PAYLOAD"
387
+```
388
+
389
+Same body fields as the Cloud-proxied `/alerts` endpoint
390
+(`status[]`, `name`, time range, options). Response is a
391
+single-node alert table. The handler at
392
+`<repo>/src/web/api/v2/api_v2_alerts.c` is shared with `/api/v2/alerts`
393
+(use v2 only on older agents that lack v3).
394
+
395
+### Alert transitions on a single agent (agent v3)
396
+
397
+`POST /api/v3/alert_transitions`
398
+
399
+Same body shape as the Cloud transitions endpoint; result is
400
+single-host. Shared handler with `/api/v2/alert_transitions`; use
401
+v3 by default.
402
+
403
+### Single alert config on a single agent (agent v3)
404
+
405
+`GET /api/v3/alert_config?config=CONFIG_HASH_UUID`
406
+
407
+```bash
408
+HOST="agent.example:19999"
409
+NODE="YOUR_NODE_UUID"
410
+BEARER="MINTED_AGENT_BEARER"
411
+CFG="ALERT_CONFIG_HASH_UUID" # the cfg field of an alert instance
412
+
413
+curl -sS \
414
+ -H "X-Netdata-Auth: Bearer $BEARER" \
415
+ "http://$HOST/host/$NODE/api/v3/alert_config?config=$CFG"
416
+```
417
+
418
+`config` is the hash UUID (the `cfg` field of an alert instance).
419
+The Cloud endpoint above points to the same data; use this only
420
+for direct-agent workflows. Response top-level keys verified live:
421
+`name`, `info`, `class`, `component`, `selectors`, `status`,
422
+`notification`, `config_hash_id`. Shared handler with v2; v3 is
423
+the default.
424
+
425
+### Legacy v1 alarm endpoints (use only on pre-v2 agents)
426
+
427
+These remain only for agents older than v1.40 that have no v2/v3
428
+alert endpoints. On any modern agent, use the v3 endpoints above.
429
+
430
+| Path | Method | Purpose |
431
+|---|---|---|
432
+| `/api/v1/alarms` | GET | Active alarms; query `?all=true` for inactive too |
433
+| `/api/v1/alarms_values` | GET | Numeric state per alarm |
434
+| `/api/v1/alarm_log` | GET | History; `?after=<unix-seconds>&chart=<name>` |
435
+| `/api/v1/alarm_count` | GET | Count by status; `?status=CRITICAL&context=<name>` |
436
+| `/api/v1/alarm_variables` | GET | Per-chart alert variables; `?chart=<name>` (required) |
437
+| `/api/v1/variable` | GET | Single variable lookup; `?chart=<name>&variable=<name>` |
438
+
439
+```bash
440
+HOST="agent.example:19999"
441
+NODE="YOUR_NODE_UUID"
442
+BEARER="MINTED_AGENT_BEARER"
443
+
444
+# Active alarms only
445
+curl -sS \
446
+ -H "X-Netdata-Auth: Bearer $BEARER" \
447
+ "http://$HOST/host/$NODE/api/v1/alarms"
448
+
449
+# Alarm transition history since a given timestamp
450
+curl -sS \
451
+ -H "X-Netdata-Auth: Bearer $BEARER" \
452
+ "http://$HOST/host/$NODE/api/v1/alarm_log?after=1700000000"
453
+```
454
+
455
+Migration: `/api/v1/alarms` -> `/api/v2/alerts`,
456
+`/api/v1/alarm_log` -> `/api/v2/alert_transitions`.
457
+
458
+---
459
+
460
+## Question-to-endpoint cheatsheet
461
+
462
+| Question | Cloud | Agent direct |
463
+|---|---|---|
464
+| What alerts are firing across the room? | `POST /api/v2/spaces/{sp}/rooms/{rm}/alerts` | `POST /host/{node}/api/v3/alerts` |
465
+| What alerts are firing across the entire space? | `GET /api/v2/spaces/{sp}/alarms` | (run per-room) |
466
+| Which alert templates are configured? | `GET /api/v2/spaces/{sp}/alarms/metas` | (per-host config inspection) |
467
+| Show alert state transitions over the last 24h | `POST /api/v2/spaces/{sp}/rooms/{rm}/alert_transitions` body `{after:<unix-s>,before:<unix-s>,...}` | `POST /host/{node}/api/v3/alert_transitions` |
468
+| Get the full configuration of a specific alert | `POST /api/v2/spaces/{sp}/rooms/{rm}/alert_config` body `{config,node_id}` | `GET /host/{node}/api/v3/alert_config?config=...` |
469
+| Evaluate a candidate alert config against history | `POST /api/v2/spaces/{sp}/rooms/{rm}/alert_config/evaluate` | not available (Cloud-only) |
470
+| Generate / suggest / explain an alert config | `POST /api/v2/spaces/{sp}/alert-config/{generate,suggest,explain}` | not available (Cloud-only) |
471
+| Which alerts are misconfigured (firing-often, stuck-raised, silenced-long, dispatch-none)? | `POST /api/v2/spaces/{sp}/rooms/{rm}/alerts:misconfigured` | not available (Cloud-only) |
472
+| What silencing rules are active or scheduled? | `GET /api/v2/spaces/{sp}/notifications/silencing/rules` | not available (Cloud-only) |
473
+| Create / update / delete a silencing rule | `POST/PUT/DELETE /api/v2/spaces/{sp}/notifications/silencing/rule[s]/...` | not available (Cloud-only) |
474
+| Reload alert definitions on the agent | not exposed via REST | not exposed via REST -- use SIGHUP or dyncfg |
475
+
476
+---
477
+
478
+## Limits and gotchas
479
+
480
+- **`PermissionAlertReadAll` is required** for all alert reads --
481
+ `scope:all` tokens have it; `scope:grafana-plugin` tokens do
482
+ NOT. If you get HTTP 403, mint a wider-scoped token.
483
+- **Silencing rules are Cloud-only.** The agent's internal
484
+ `SILENCER` structures are not REST-addressable. There is no
485
+ `/api/v[123]/silencers` on the agent.
486
+- **No REST endpoint for "reload alert configs"** on either side.
487
+ The agent reloads on `SIGHUP` or via the dyncfg callback at
488
+ `src/health/health_dyncfg.c`. For programmatic config changes,
489
+ push files to `etc/netdata/health.d/` and signal the agent.
490
+- **`config_hash_id` is required for `/alert_config`** on both
491
+ sides. Get it from the alert metadata (`/alerts` response,
492
+ `config_hash_id` field, or `/alarms/metas` for templates).
493
+- **Agent-direct paths return single-host data.** For aggregated
494
+ cross-room/cross-space queries, you must use Cloud or aggregate
495
+ agent responses client-side.
496
+- **`alert_transitions` time bounds are seconds, NOT
497
+ milliseconds.** Negative values are relative offsets from "now".
498
+ This differs from `systemd-journal` time bounds (microseconds).
docs/netdata-ai/skills/query-netdata-cloud/query-dyncfg.md
new
+334
@@ -0,0 +1,334 @@
1
+# Query Netdata Dynamic Configuration (DynCfg)
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+DynCfg is Netdata's dynamic-configuration system. Every plugin /
7
+collector / module that participates registers configuration
8
+objects which the user can list, view, edit, enable, disable, add,
9
+remove, test, restart, and export -- through a single REST surface
10
+distinct from the Function-call surface. Internally DynCfg is
11
+implemented on top of Functions, but it has its own dedicated API
12
+endpoint. Don't confuse the two: function-call paths
13
+(`/api/v2/nodes/{nodeID}/function?function=...`) DO NOT operate on
14
+configuration.
15
+
16
+**Canonical references** (authoritative; read these for full detail):
17
+
18
+| File | What it covers |
19
+|---|---|
20
+| `<repo>/src/daemon/dyncfg/README.md` | Internal-plugin DynCfg API and lifecycle (high-level + low-level) |
21
+| `<repo>/src/plugins.d/DYNCFG.md` | External-plugin (go.d.plugin, etc.) DynCfg protocol |
22
+
23
+---
24
+
25
+## Mandatory Requirements (READ FIRST)
26
+
27
+1. **Provide actionable instructions.** Each answer ends in a
28
+ runnable curl command.
29
+2. **Never request credentials.** Use `YOUR_API_TOKEN`,
30
+ `YOUR_NODE_UUID`, etc. placeholders.
31
+3. **Read actions are GET; write actions are POST.** Cloud's
32
+ permission gate is `PermissionFunctionExecPrivileged` for the
33
+ write paths. A read-only token cannot mutate configuration.
34
+4. **The `action=tree` listing is the entry point.** Always start
35
+ there; the IDs returned are the inputs to all other actions.
36
+
37
+---
38
+
39
+## Endpoints
40
+
41
+### Cloud-proxied (preferred)
42
+
43
+| Method | Path | Purpose | Permission |
44
+|---|---|---|---|
45
+| `GET` | `/api/v2/nodes/{nodeID}/config?action=tree&path=/` | List configuration objects | `nodeAuth()` |
46
+| `GET` | `/api/v2/nodes/{nodeID}/config?action=<read-action>&id=<id>` | Read one configuration | `nodeAuth()` |
47
+| `POST` | `/api/v2/nodes/{nodeID}/config?action=<write-action>&id=<id>[&name=<name>]` | Mutate configuration (body = the configuration JSON) | `PermissionFunctionExecPrivileged` |
48
+
49
+Cloud-side route registration:
50
+`cloud-charts-service/http/http.go:150-151`.
51
+
52
+### Direct-agent (fallback)
53
+
54
+Same shape, served by the agent itself:
55
+`http://<agent>:19999/host/{nodeID}/api/v3/config?...`. The agent
56
+also accepts `/api/v1/config` for backwards compatibility -- both
57
+paths use the same handler at
58
+`<repo>/src/web/api/v1/api_v1_config.c`.
59
+
60
+For direct-agent calls, `X-Netdata-Auth: Bearer <agent-bearer>` is
61
+required when the agent is bearer-protected. See the
62
+[`query-netdata-agents`](../query-netdata-agents/SKILL.md) skill
63
+for the bearer mint/cache flow.
64
+
65
+---
66
+
67
+## Query parameters
68
+
69
+The handler at `<repo>/src/web/api/v1/api_v1_config.c:5-80` accepts
70
+these query parameters:
71
+
72
+| Param | Required for | Purpose |
73
+|---|---|---|
74
+| `action` | every call (defaults to `tree`) | Which DynCfg command to execute |
75
+| `path` | `tree` | Path within the configuration tree (e.g. `/`, `/health/alerts/prototypes`) |
76
+| `id` | every action except `tree` | Configuration object id (colon-separated, e.g. `health:alert:prototype:ram_usage`) |
77
+| `name` | `add`, `userconfig`, `test` | Job/object name when adding to a template, getting user-config, or testing |
78
+| `timeout` | optional (default 120) | Operation timeout in seconds; minimum 10 |
79
+
80
+The body of a `POST` carries the **payload** for write actions
81
+(the configuration JSON to apply, or the test input for `test`).
82
+
83
+---
84
+
85
+## DynCfg actions
86
+
87
+From `<repo>/src/daemon/dyncfg/README.md` and the agent's command
88
+enum:
89
+
90
+| `action` value | Method | Purpose |
91
+|---|---|---|
92
+| `tree` | GET | List configuration objects under `path` |
93
+| `schema` | GET | JSON Schema for a configuration object |
94
+| `get` | GET | Current value of a configuration object |
95
+| `userconfig` | GET | Configuration in user-friendly form (used for conf files); requires `name` |
96
+| `update` | POST | Replace a configuration object's value (body = new JSON) |
97
+| `add` | POST | Add a new job to a template; requires `name` |
98
+| `remove` | POST | Remove a `dyncfg`-source job (cannot remove user-file jobs) |
99
+| `enable` | POST | Enable a configuration object |
100
+| `disable` | POST | Disable a configuration object |
101
+| `test` | POST | Test a configuration without applying it; requires `name`; body = the candidate JSON |
102
+| `restart` | POST | Restart the configuration / re-apply |
103
+
104
+### DynCfg response codes
105
+
106
+DynCfg uses HTTP-like codes verified against
107
+`<repo>/src/daemon/dyncfg/README.md`:
108
+
109
+| Code | Meaning |
110
+|---|---|
111
+| 200 | Running -- accepted and active |
112
+| 202 | Accepted -- queued, not yet running |
113
+| 298 | Accepted but disabled |
114
+| 299 | Accepted but restart required |
115
+| 400 | Bad request / invalid configuration |
116
+| 404 | Configuration id not found |
117
+| 500 | Internal error |
118
+| 501 | Action not implemented for this object |
119
+
120
+---
121
+
122
+## Configuration ID structure
123
+
124
+Configuration IDs follow a colon-separated hierarchy
125
+(`<repo>/src/daemon/dyncfg/README.md`):
126
+
127
+```
128
+component:category:name
129
+component:template_name:job_name
130
+```
131
+
132
+Examples (verified live -- agent-events node returns these under
133
+`tree`):
134
+
135
+- `/collectors/go.d/Jobs` -- the go.d.plugin Jobs tree
136
+- `/collectors/go.d/ServiceDiscovery`
137
+- `/collectors/go.d/Vnodes`
138
+- `/collectors/ibm.d/Jobs`
139
+- `/collectors/ibm.d/Vnodes`
140
+- `/health/alerts/prototypes` -- alert prototypes
141
+- `/logs/systemd-journal` -- systemd-journal collector configs
142
+
143
+Each tree entry contains configuration objects with their own ids
144
+(e.g. `health:alert:prototype:ram_usage`,
145
+`go.d:nginx:local_server`).
146
+
147
+### Templates vs Jobs
148
+
149
+- **Template id**: `component:template_name`. Templates DEFINE the
150
+ schema for jobs. They cannot be `update`d but can be `add`'d to.
151
+- **Job id**: `component:template_name:job_name`. The portion
152
+ before the last colon must match an existing template id.
153
+- **Single id**: `component:name`. A standalone configuration
154
+ object that is neither template nor job.
155
+
156
+---
157
+
158
+## Examples
159
+
160
+### Example 1: list every configuration object on a node
161
+
162
+```bash
163
+TOKEN="YOUR_API_TOKEN"
164
+NODE="YOUR_NODE_UUID"
165
+
166
+curl -sS \
167
+ -H "Authorization: Bearer $TOKEN" \
168
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=tree&path=/"
169
+```
170
+
171
+Response top level (verified live): `{agent, attention, tree,
172
+version}`. The `tree` is keyed by configuration path; each entry
173
+lists per-object ids, types, statuses, supported commands, and
174
+sources.
175
+
176
+### Example 2: get the JSON Schema for an alert prototype
177
+
178
+```bash
179
+TOKEN="YOUR_API_TOKEN"
180
+NODE="YOUR_NODE_UUID"
181
+ID="health:alert:prototype:ram_usage"
182
+
183
+curl -sS \
184
+ -H "Authorization: Bearer $TOKEN" \
185
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=schema&id=$(printf %s "$ID" | jq -sRr @uri)"
186
+```
187
+
188
+The schema describes which fields the configuration accepts and is
189
+the basis for any UI form. Use this before constructing an
190
+`update` payload.
191
+
192
+### Example 3: read the current value of a configuration
193
+
194
+```bash
195
+TOKEN="YOUR_API_TOKEN"
196
+NODE="YOUR_NODE_UUID"
197
+ID="health:alert:prototype:ram_usage"
198
+
199
+curl -sS \
200
+ -H "Authorization: Bearer $TOKEN" \
201
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=get&id=$(printf %s "$ID" | jq -sRr @uri)"
202
+```
203
+
204
+### Example 4: add a new go.d.plugin job to a template
205
+
206
+```bash
207
+TOKEN="YOUR_API_TOKEN"
208
+NODE="YOUR_NODE_UUID"
209
+TPL_ID="go.d:nginx"
210
+JOB_NAME="local_server"
211
+
212
+read -r -d '' PAYLOAD <<'EOF'
213
+{
214
+ "url": "http://127.0.0.1:80/stub_status",
215
+ "update_every": 5,
216
+ "timeout": 2
217
+}
218
+EOF
219
+
220
+curl -sS -X POST \
221
+ -H 'Content-Type: application/json' \
222
+ -H "Authorization: Bearer $TOKEN" \
223
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=add&id=$(printf %s "$TPL_ID" | jq -sRr @uri)&name=$JOB_NAME" \
224
+ -d "$PAYLOAD"
225
+```
226
+
227
+### Example 5: test a configuration without applying it
228
+
229
+```bash
230
+TOKEN="YOUR_API_TOKEN"
231
+NODE="YOUR_NODE_UUID"
232
+TPL_ID="go.d:nginx"
233
+JOB_NAME="local_server"
234
+
235
+read -r -d '' PAYLOAD <<'EOF'
236
+{
237
+ "url": "http://127.0.0.1:80/stub_status",
238
+ "update_every": 5
239
+}
240
+EOF
241
+
242
+curl -sS -X POST \
243
+ -H 'Content-Type: application/json' \
244
+ -H "Authorization: Bearer $TOKEN" \
245
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=test&id=$(printf %s "$TPL_ID" | jq -sRr @uri)&name=$JOB_NAME" \
246
+ -d "$PAYLOAD"
247
+```
248
+
249
+A successful response (200/202) means the configuration would
250
+work; the test does NOT make the change persistent.
251
+
252
+### Example 6: enable / disable a configuration
253
+
254
+```bash
255
+TOKEN="YOUR_API_TOKEN"
256
+NODE="YOUR_NODE_UUID"
257
+ID="go.d:nginx:local_server"
258
+
259
+# Enable
260
+curl -sS -X POST \
261
+ -H "Authorization: Bearer $TOKEN" \
262
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=enable&id=$(printf %s "$ID" | jq -sRr @uri)"
263
+
264
+# Disable
265
+curl -sS -X POST \
266
+ -H "Authorization: Bearer $TOKEN" \
267
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=disable&id=$(printf %s "$ID" | jq -sRr @uri)"
268
+```
269
+
270
+### Example 7: remove a dyncfg-created job
271
+
272
+```bash
273
+TOKEN="YOUR_API_TOKEN"
274
+NODE="YOUR_NODE_UUID"
275
+ID="go.d:nginx:local_server"
276
+
277
+curl -sS -X POST \
278
+ -H "Authorization: Bearer $TOKEN" \
279
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=remove&id=$(printf %s "$ID" | jq -sRr @uri)"
280
+```
281
+
282
+Only objects whose `source_type == DYNCFG_SOURCE_TYPE_DYNCFG` (i.e.
283
+created via DynCfg, not from user files or internally) can be
284
+removed.
285
+
286
+### Example 8: get a job's user-friendly configuration (for conf-file export)
287
+
288
+```bash
289
+TOKEN="YOUR_API_TOKEN"
290
+NODE="YOUR_NODE_UUID"
291
+ID="go.d:nginx"
292
+JOB_NAME="local_server"
293
+
294
+curl -sS \
295
+ -H "Authorization: Bearer $TOKEN" \
296
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/config?action=userconfig&id=$(printf %s "$ID" | jq -sRr @uri)&name=$JOB_NAME"
297
+```
298
+
299
+---
300
+
301
+## Source types
302
+
303
+A configuration object's `source_type` (verified in
304
+`<repo>/src/daemon/dyncfg/README.md`) determines what's allowed:
305
+
306
+| Source type | Origin | Removable? |
307
+|---|---|---|
308
+| `INTERNAL` | Defined inside Netdata code | No |
309
+| `DYNCFG` | Created or modified through DynCfg | Yes |
310
+| `USER` | Loaded from user-provided conf files in `/etc/netdata/...` | No (edit the file instead) |
311
+
312
+---
313
+
314
+## Direct-agent reload note
315
+
316
+There is **no REST endpoint to reload `/etc/netdata/health.d/*.conf`
317
+files** -- DynCfg manages dynamic configuration objects, not raw
318
+file reload. To reload static health/alert files, send `SIGHUP` to
319
+the agent process or use the dyncfg `restart` action on the
320
+relevant configuration object.
321
+
322
+---
323
+
324
+## Sensitive data
325
+
326
+DynCfg responses can include credentials embedded in collector
327
+job configurations (database URIs, API tokens used by collectors,
328
+etc.). Treat raw responses as production-sensitive:
329
+
330
+- Direct working output to `<repo>/.local/audits/...` (gitignored).
331
+- Never paste raw `get` / `userconfig` payloads into committed
332
+ files.
333
+- See `<repo>/.agents/sow/specs/sensitive-data-discipline.md` for
334
+ the full rule.
docs/netdata-ai/skills/query-netdata-cloud/query-feed.md
new
+199
@@ -0,0 +1,199 @@
1
+# Query the event feed via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+The Cloud event feed is an audit + activity log: node lifecycle
7
+events, alert transitions, agent connection events, space and room
8
+membership changes, and configuration changes. It is served by the
9
+**`cloud-feed-service`** (separate microservice from
10
+spaceroom/charts) and answers via Elasticsearch under the hood.
11
+
12
+There is no agent-side equivalent. The feed is Cloud-only.
13
+
14
+---
15
+
16
+## Endpoint
17
+
18
+`POST /api/v1/feed/search` -- search the feed.
19
+
20
+This is a v1 path (the only supported path for this service today).
21
+
22
+The companion search-lean variant (`/api/v1/feed/search/lean`)
23
+returns hits without the full source documents -- use it when you
24
+only need aggregations / counts.
25
+
26
+The facets endpoint
27
+(`GET /api/v1/feed/static/facets`) returns the supported facet
28
+field schema (mostly for UI rendering).
29
+
30
+## Use the wrapper
31
+
32
+```bash
33
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
34
+agents_load_env
35
+
36
+# Last 10 events in a space.
37
+read -r -d '' BODY <<EOF
38
+{
39
+ "space_id": "$SPACE",
40
+ "page_size": 10
41
+}
42
+EOF
43
+
44
+agents_query_cloud POST /api/v1/feed/search "$BODY"
45
+```
46
+
47
+## Body parameters
48
+
49
+| Field | Type | Purpose |
50
+|---|---|---|
51
+| `space_id` | string (UUID) | **REQUIRED.** Space to search within |
52
+| `room_ids` | array<string> | Filter to specific rooms |
53
+| `agents` | array<string> | Filter to specific agent ids (`mg` field of nodes) |
54
+| `node_ids` | array<string> | Filter to specific node ids (`nd` field) |
55
+| `actions` | array<string> | Filter by event action (see enum below) |
56
+| `alert_classes`, `alert_components`, `alert_names`, `alert_roles`, `alert_statuses`, `alert_transitions`, `alert_types` | array<string> | Alert-event filters |
57
+| `chart_names`, `chart_contexts`, `chart_types` | array<string> | Chart-related event filters |
58
+| `from`, `to` | int (Unix-millis) | Time range |
59
+| `query` | string | Free-text search |
60
+| `page_size` | int | Page size |
61
+| `from_offset` | int | Pagination offset |
62
+
63
+### `actions` enum (verified live)
64
+
65
+Node lifecycle:
66
+- `node-created`, `node-removed`, `node-deleted`, `node-restored`
67
+- `node-state-live`, `node-state-stale`, `node-state-offline`
68
+
69
+Agent lifecycle:
70
+- `agent-connected`, `agent-disconnected`, `agent-claimed`
71
+
72
+Alerts:
73
+- `alert-node-transition`, `alert-node_instance-transition`
74
+
75
+User / space / room:
76
+- `user-create`, `user-created`
77
+- `space-created`, `space-deleted`, `space-settings-changed`
78
+- `space-user-added`, `space-user-removed`
79
+- `user-space-permissions-changed`
80
+- `room-created`, `room-deleted`
81
+- `room-user-added`, `room-user-removed`
82
+- `user-room-permissions-changed`
83
+
84
+## Response shape
85
+
86
+```text
87
+{
88
+ "page_size": <int>,
89
+ "results": {
90
+ "hits": {
91
+ "total": { "value": <int> },
92
+ "hits": [
93
+ {
94
+ "_source": {
95
+ "@timestamp": "<RFC3339>",
96
+ "trace": { "id": "<UUID>" },
97
+ "agent": { "version": "..." },
98
+ "host": { "id": "<machine_guid>", "name": "<hostname>", ... },
99
+ "Netdata": { "alert": {...}, "event": {...}, ... },
100
+ "ecs": { "version": "..." }
101
+ },
102
+ "_index": "...",
103
+ "_id": "...",
104
+ "_score": <float>
105
+ },
106
+ ...
107
+ ]
108
+ },
109
+ "aggregations": {
110
+ "actions": { "buckets": [...] },
111
+ "agents": { "buckets": [...] },
112
+ "alert_classes": { "buckets": [...] },
113
+ "alert_components":{ "buckets": [...] },
114
+ "alert_names": { "buckets": [...] },
115
+ "alert_roles": { "buckets": [...] },
116
+ ...
117
+ }
118
+ }
119
+}
120
+```
121
+
122
+The hit fields under `_source` follow the **ECS (Elastic Common
123
+Schema) v8.4.0** layout for shared keys (`@timestamp`, `host.*`,
124
+`agent.*`, `ecs.*`) plus a Netdata-specific `Netdata.*` envelope
125
+that holds the per-event payload.
126
+
127
+## Common patterns
128
+
129
+```bash
130
+# Last hour of node-state changes.
131
+read -r -d '' BODY <<EOF
132
+{
133
+ "space_id": "$SPACE",
134
+ "actions": ["node-state-live","node-state-stale","node-state-offline"],
135
+ "from": $(( ($(date +%s) - 3600) * 1000 )),
136
+ "to": $(( $(date +%s) * 1000 )),
137
+ "page_size": 50
138
+}
139
+EOF
140
+agents_query_cloud POST /api/v1/feed/search "$BODY" \
141
+ | jq -r '.results.hits.hits[]._source | "\(.["@timestamp"])\t\(.Netdata.event.action // "?")\t\(.host.name // "?")"'
142
+
143
+# Distribution of alert classes triggered in the last 24h.
144
+read -r -d '' BODY <<EOF
145
+{
146
+ "space_id": "$SPACE",
147
+ "actions": ["alert-node_instance-transition"],
148
+ "from": $(( ($(date +%s) - 86400) * 1000 )),
149
+ "to": $(( $(date +%s) * 1000 )),
150
+ "page_size": 0
151
+}
152
+EOF
153
+agents_query_cloud POST /api/v1/feed/search "$BODY" \
154
+ | jq -r '.results.aggregations.alert_classes.buckets[] | "\(.key)\t\(.doc_count)"'
155
+
156
+# All space-user-added events for a given account in the last 7 days.
157
+ACCT="<account-uuid>"
158
+read -r -d '' BODY <<EOF
159
+{
160
+ "space_id": "$SPACE",
161
+ "actions": ["space-user-added"],
162
+ "from": $(( ($(date +%s) - 604800) * 1000 )),
163
+ "to": $(( $(date +%s) * 1000 )),
164
+ "page_size": 50
165
+}
166
+EOF
167
+agents_query_cloud POST /api/v1/feed/search "$BODY" \
168
+ | jq --arg id "$ACCT" '.results.hits.hits[] | select(._source.user.id // "" == $id)'
169
+```
170
+
171
+## Limits and gotchas
172
+
173
+- **`from`/`to` are Unix milliseconds.** Easy to confuse with
174
+ seconds.
175
+- **Total hit count is paginated.** Use `from_offset` to walk
176
+ past the first page; `total.value` tells you the size.
177
+- **`actions` is the most useful facet.** Most queries should
178
+ start by narrowing by `actions[]`; the per-action `_source`
179
+ shape varies, so filter first then read the appropriate
180
+ per-event fields.
181
+- **Hits include personal data.** `host.name`, `user.id`,
182
+ `user.email`, `agent.version`, alert config_hash UUIDs --
183
+ treat raw responses as semi-sensitive; never paste into
184
+ committed artifacts.
185
+- **Retention is finite.** The feed-service has retention
186
+ enforcement (`errInvalidRetention` error message in source);
187
+ very old time windows return errors.
188
+- **No agent-side equivalent.** This is the only path to the
189
+ audit/activity feed; agents do not retain it locally.
190
+
191
+## See also
192
+
193
+- [query-rooms.md](./query-rooms.md), [query-members.md](./query-members.md)
194
+ -- the surfaces whose changes generate room-/member-related
195
+ feed events.
196
+- [query-alerts.md](./query-alerts.md) -- alert transitions are
197
+ also emitted into the feed via `alert-node-transition` and
198
+ `alert-node_instance-transition` actions, in addition to the
199
+ per-room alert-transitions endpoint.
docs/netdata-ai/skills/query-netdata-cloud/query-flows.md
new
+257
@@ -0,0 +1,257 @@
1
+# Query network-flow Functions via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+For the generic Function transport, see
6
+[query-functions.md](./query-functions.md).
7
+
8
+Flow Functions return network-flow records (NetFlow / sFlow /
9
+IPFIX) ingested by the agent's flow collector. Their dataset is
10
+table-shaped (one row per flow tuple) AND time-windowed AND
11
+faceted, sitting between table snapshots (`processes`) and log
12
+queries (`systemd-journal`).
13
+
14
+---
15
+
16
+## Function names registered today
17
+
18
+Verified live and in source:
19
+
20
+| Function | Source crate | Layer | What it returns |
21
+|---|---|---|---|
22
+| `flows:netflow` | `src/crates/netflow-plugin/` | L3 | Network flow records ingested via NetFlow v5/v9, IPFIX, sFlow |
23
+
24
+The `flows:` prefix is the canonical namespace; only `netflow` is
25
+registered today. The Function name covers all three protocols
26
+(the collector parses NetFlow, IPFIX, and sFlow into a single
27
+record schema).
28
+
29
+---
30
+
31
+## Endpoint and request
32
+
33
+Standard Cloud Function-call endpoint:
34
+
35
+`POST /api/v2/nodes/{nodeId}/function?function=flows:netflow`
36
+
37
+```bash
38
+TOKEN="YOUR_API_TOKEN"
39
+NODE="YOUR_NODE_UUID"
40
+
41
+read -r -d '' PAYLOAD <<'EOF'
42
+{
43
+ "mode": "flows",
44
+ "view": "table-sankey",
45
+ "after": -3600,
46
+ "before": 0,
47
+ "group_by": ["SRC_AS_NAME", "PROTOCOL", "DST_AS_NAME"],
48
+ "sort_by": "bytes",
49
+ "top_n": 100
50
+}
51
+EOF
52
+
53
+curl -sS -X POST \
54
+ -H 'Content-Type: application/json' \
55
+ -H "Authorization: Bearer $TOKEN" \
56
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=flows:netflow" \
57
+ -d "$PAYLOAD"
58
+```
59
+
60
+### Modes
61
+
62
+The Function has three modes selected by the `mode` body field:
63
+
64
+| Mode | Purpose |
65
+|---|---|
66
+| `flows` (default) | Return flow records / aggregations / charts |
67
+| `autocomplete` | Return values for a single facet field, given a search prefix |
68
+
69
+### Body parameters
70
+
71
+Verified against `src/crates/netflow-plugin/src/api/flows/handler.rs`:
72
+
73
+| Parameter | Used in mode | Description |
74
+|---|---|---|
75
+| `mode` | both | `flows` or `autocomplete` |
76
+| `view` | flows | One of: `table-sankey`, `timeseries`, `country-map`, `state-map`, `city-map` |
77
+| `after` | flows | Unix seconds, lower bound. Negative = relative seconds from `before` |
78
+| `before` | flows | Unix seconds, upper bound. `0` = now |
79
+| `query` | flows | Free-text filter |
80
+| `selections` | flows | Pre-applied facet filters as `{ "FIELD_NAME": ["val", "val2"] }`. Common fields: `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, `DST_PORT`, `PROTOCOL`, `SRC_AS_NAME`, `DST_AS_NAME`, `SRC_COUNTRY`, `DST_COUNTRY`, `INTERFACE`, ... |
81
+| `facets` | flows | Array of facet field names whose value-distributions should appear in the response |
82
+| `group_by` | flows | Up to 10 tuple-key field names (e.g. `["SRC_ADDR","DST_ADDR","PROTOCOL"]`) -- order defines the aggregation tuple |
83
+| `sort_by` | flows | `bytes` or `packets` |
84
+| `top_n` | flows | One of `25`, `50`, `100`, `200`, `500` |
85
+| `field` | autocomplete | Facet field to autocomplete (`SRC_ADDR`, etc.) |
86
+| `term` | autocomplete | Search prefix |
87
+
88
+---
89
+
90
+## Response envelope
91
+
92
+Flow Functions wrap their content in the standard Function
93
+envelope (same shape as topology and logs):
94
+
95
+| Key | Description |
96
+|---|---|
97
+| `status` | HTTP-style status |
98
+| `v` | Function schema version |
99
+| `type` | **`flows`** -- the family discriminator |
100
+| `help` / `accepted_params` / `required_params` / `has_history` / `update_every` | Discovery metadata |
101
+| `data` | Mode-specific payload (object) |
102
+
103
+### `data` object -- mode `flows`, view `table-sankey`
104
+
105
+| Key | Description |
106
+|---|---|
107
+| `schema_version` | `2.0` |
108
+| `source` | `netflow` |
109
+| `layer` | `3` |
110
+| `agent_id` | Producing-agent identifier |
111
+| `collected_at` | RFC3339 timestamp |
112
+| `view` | Echo of requested view |
113
+| `group_by` | Echo of requested group-by tuple |
114
+| `columns` | Per-column display metadata |
115
+| `flows[]` | Aggregated flow rows (one row per group_by tuple) |
116
+| `stats` | Counters: `flows_total`, `packets_total`, `bytes_total`, etc. |
117
+| `metrics` | Optional metric block |
118
+| `warnings[]` | Optional non-fatal diagnostics |
119
+| `facets` | When `facets` was requested in body, per-field value-counts plus `selections` echo |
120
+
121
+### `data` object -- mode `flows`, view `timeseries`
122
+
123
+Replaces `flows[]` with `metric` (string) and `chart` (object); used
124
+for line/area charts of bytes-per-second / packets-per-second
125
+broken down by the group-by tuple.
126
+
127
+### `data` object -- mode `flows`, geo views (`country-map`, `state-map`, `city-map`)
128
+
129
+Returns geo-keyed aggregations (per-country / per-state / per-city
130
+totals) suitable for map rendering.
131
+
132
+### `data` object -- mode `autocomplete`
133
+
134
+| Key | Description |
135
+|---|---|
136
+| `mode` | `autocomplete` |
137
+| `field` | Echo of requested field |
138
+| `term` | Echo of requested search term |
139
+| `values[]` | Matching values for the field |
140
+| `stats` / `warnings` | Same as flows mode |
141
+
142
+---
143
+
144
+## Examples
145
+
146
+### Example 1: top-100 talker pairs by bytes, last hour
147
+
148
+```bash
149
+TOKEN="YOUR_API_TOKEN"
150
+NODE="YOUR_NODE_UUID"
151
+
152
+read -r -d '' PAYLOAD <<'EOF'
153
+{
154
+ "mode": "flows",
155
+ "view": "table-sankey",
156
+ "after": -3600,
157
+ "before": 0,
158
+ "group_by": ["SRC_ADDR", "DST_ADDR"],
159
+ "sort_by": "bytes",
160
+ "top_n": 100
161
+}
162
+EOF
163
+
164
+curl -sS -X POST \
165
+ -H 'Content-Type: application/json' \
166
+ -H "Authorization: Bearer $TOKEN" \
167
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=flows:netflow" \
168
+ -d "$PAYLOAD" \
169
+ | jq '.data.flows[:5]'
170
+```
171
+
172
+### Example 2: breakdown of TCP traffic by AS name, with histogram
173
+
174
+```bash
175
+read -r -d '' PAYLOAD <<'EOF'
176
+{
177
+ "mode": "flows",
178
+ "view": "timeseries",
179
+ "after": -86400,
180
+ "before": 0,
181
+ "selections": { "PROTOCOL": ["TCP"] },
182
+ "group_by": ["DST_AS_NAME"],
183
+ "sort_by": "bytes",
184
+ "top_n": 25
185
+}
186
+EOF
187
+```
188
+
189
+### Example 3: country-map of egress bytes
190
+
191
+```bash
192
+read -r -d '' PAYLOAD <<'EOF'
193
+{
194
+ "mode": "flows",
195
+ "view": "country-map",
196
+ "after": -3600,
197
+ "before": 0,
198
+ "group_by": ["DST_COUNTRY"],
199
+ "sort_by": "bytes",
200
+ "top_n": 500
201
+}
202
+EOF
203
+```
204
+
205
+### Example 4: autocomplete for a destination IP filter
206
+
207
+```bash
208
+read -r -d '' PAYLOAD <<'EOF'
209
+{
210
+ "mode": "autocomplete",
211
+ "field": "DST_ADDR",
212
+ "term": "10.0.0."
213
+}
214
+EOF
215
+```
216
+
217
+### Example 5: discover the live parameter set first
218
+
219
+```bash
220
+read -r -d '' PAYLOAD <<'EOF'
221
+{ "info": true }
222
+EOF
223
+
224
+curl -sS -X POST \
225
+ -H 'Content-Type: application/json' \
226
+ -H "Authorization: Bearer $TOKEN" \
227
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=flows:netflow" \
228
+ -d "$PAYLOAD" \
229
+ | jq '{accepted_params, required_params}'
230
+```
231
+
232
+---
233
+
234
+## Limits and gotchas
235
+
236
+- **Cloud timeout default 120 s.** Wide-window queries
237
+ (`after: -86400`) over high-volume agents can hit it. Narrow
238
+ the time window or filter via `selections`.
239
+- **`top_n` is enumerated, not free.** Allowed values are 25, 50,
240
+ 100, 200, 500. Other integers are rejected.
241
+- **`group_by` accepts up to 10 fields.** The order matters --
242
+ it's the tuple ordering for the aggregation key.
243
+- **AS names depend on the configured GeoIP/AS database.** If the
244
+ collector has no AS database, `SRC_AS_NAME` / `DST_AS_NAME`
245
+ will be empty strings. Same for country/city fields.
246
+- **Privacy**: flow records reveal who-talks-to-whom and how much.
247
+ Treat raw output as production-sensitive; never paste into
248
+ committed files. Direct working output to
249
+ `<repo>/.local/audits/...` (gitignored).
250
+- **Sampled vs full flows**: NetFlow v5/v9 and sFlow are sampled
251
+ by source devices; reported byte/packet counts are scaled by
252
+ the sample rate. The collector reports raw counts -- consult
253
+ source-device sampling configuration when interpreting
254
+ absolute volumes.
255
+- **Function is L3-only.** No L2 visibility (use
256
+ [topology Functions](./query-topology.md) for L2). No
257
+ application-layer dissection (use logs).
docs/netdata-ai/skills/query-netdata-cloud/query-functions.md
new
+415
@@ -0,0 +1,415 @@
1
+# Query agent Functions via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+This file documents the **generic** Function transport: the URL,
7
+the standard response envelope, the `info` discovery query, the
8
+four Function families and where each family's data lives in the
9
+response, plus pointers to the developer documentation for
10
+collector authors.
11
+
12
+For three of the four families there is a dedicated guide:
13
+
14
+- **Logs** family (table-history with facets+histogram):
15
+ [query-logs.md](./query-logs.md)
16
+- **Topology** family (graph: actors+links):
17
+ [query-topology.md](./query-topology.md)
18
+- **Flows** family (network-flow records):
19
+ [query-flows.md](./query-flows.md)
20
+
21
+The **table-snapshot** family (full dataset in each response) is
22
+covered here.
23
+
24
+For querying agents directly (without going through Cloud) -- which
25
+includes the transparent Cloud-token to agent-bearer mint flow --
26
+see the sibling skill
27
+[`query-netdata-agents`](../query-netdata-agents/SKILL.md).
28
+
29
+---
30
+
31
+## Mandatory Requirements (READ FIRST)
32
+
33
+1. **Provide actionable instructions.** Every recommendation ends
34
+ in a runnable curl command.
35
+2. **Never request credentials.** Use `YOUR_API_TOKEN` and
36
+ `YOUR_NODE_UUID` placeholders.
37
+3. **Always start with `{"info":true}`** when you don't already
38
+ know the parameter set of the target Function. The `info`
39
+ response is authoritative -- this skill's tables can be stale
40
+ relative to the running agent.
41
+4. **Function names are case-sensitive** (e.g. `systemd-journal`,
42
+ `topology:snmp`, `flows:netflow`).
43
+
44
+---
45
+
46
+## Function classes
47
+
48
+The canonical Functions v3 protocol
49
+(`<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`) formally defines
50
+**two** Function classes, distinguished by the `has_history` flag
51
+in the `info` response:
52
+
53
+| Class | `has_history` | Frontend behavior | Examples |
54
+|---|---|---|---|
55
+| **Simple Table** | `false` | Backend returns the whole current dataset; frontend filters/sorts/searches in-memory | `processes`, `network-connections`, `network-interfaces`, `network-sockets-tracing`, `block-devices`, `mount-points`, `containers-vms`, `systemd-services`, `netdata-streaming`, `netdata-api-calls`, `netdata-metrics-cardinality`, `<db>:top-queries`, `<db>:running-queries`, `<db>:deadlock-info`, `<db>:error-info` |
56
+| **Log Explorer** | `true` | Backend filters / facets / histograms before sending; supports infinite scroll, anchor pagination, delta and PLAY modes | `systemd-journal`, `windows-events`, `otel-logs` |
57
+
58
+Two additional `type` values are used by purpose-built Functions
59
+that build on the same envelope but emit non-tabular `data`:
60
+
61
+| `type` | Response shape | Examples | Guide |
62
+|---|---|---|---|
63
+| `topology` | `data.actors[]` + `data.links[]` (a graph) | `topology:snmp` | [query-topology.md](./query-topology.md) |
64
+| `flows` | `data.flows[]` plus `data.facets` / `data.columns` / `data.stats` over a time window | `flows:netflow` (covers NetFlow / sFlow / IPFIX) | [query-flows.md](./query-flows.md) |
65
+
66
+For full protocol semantics (facet pills, histograms, charts
67
+configuration, anchor/delta/PLAY modes, error handling, edge
68
+cases), the authoritative source is
69
+`<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`. This skill
70
+summarizes the surface that matters for a Cloud-side curl client;
71
+the reference covers everything else.
72
+
73
+---
74
+
75
+## Standard response envelope
76
+
77
+Every Function -- regardless of family -- wraps its output in this
78
+envelope. Verified live against the agent's `systemd-journal`,
79
+`topology:snmp`, and `flows:netflow` Functions, and against the
80
+agent emit code at
81
+`src/web/api/functions/function-metrics-cardinality.c:26-39,92`
82
+plus per-collector wrappers.
83
+
84
+| Key | Type | Required | Notes |
85
+|---|---|---|---|
86
+| `status` | int | yes | HTTP-style status (200, 400, ...) |
87
+| `v` | int | yes | Function schema version (currently `3` or `4` depending on Function) |
88
+| `type` | string | yes | Family discriminator: `table`, `logs`, `topology`, `flows` (some Functions emit a custom string -- treat unknown values as `table`-like) |
89
+| `help` | string | typical | Human-readable description |
90
+| `accepted_params` | array<string> | typical | Parameter names accepted in the body |
91
+| `required_params` | array<object> | typical | Per-parameter widget descriptors -- see "info=true discovery" below |
92
+| `has_history` | bool | typical | Whether the Function honors `after` / `before` |
93
+| `update_every` | int | typical | Suggested refresh interval in seconds |
94
+| `data` | array OR object | conditional | Family-specific result. **Absent on `info=true` calls and on errors.** Array for `logs` and `table` families; object (with `actors`/`links` or `flows`/`columns`/`stats`) for `topology` and `flows` |
95
+| `columns` | object | logs / table | Column-metadata, keyed by column name. Each entry has `index` (position inside each row of `data`), `name`, `type`, `visible`, `sort`, `summary`, `filter`, ... |
96
+| `facets` | array | logs / flows | Per-field value distribution and option counts |
97
+| `histogram` | object | logs (when requested) | Bucketed counts over time |
98
+| `pagination` | object | logs | `anchor`, `direction`, `last`, etc. |
99
+| `presentation` | object | topology / flows | Visualization metadata for the Cloud UI |
100
+| `expires` / `last_modified` / `partial` / `message` | scalar | optional | Caching, freshness, partial-result diagnostics |
101
+| `versions` | object | optional | Source/version hashes for client cache invalidation |
102
+
103
+`status >= 400` responses follow the same envelope but include an
104
+`errorMessage` / `errorMsgKey` instead of `data`.
105
+
106
+---
107
+
108
+## `info=true` discovery
109
+
110
+The single most important call to make before constructing a real
111
+query: pass `{"info": true}` and read `accepted_params` plus
112
+`required_params`. The agent itself is the authoritative source --
113
+if a parameter exists there, the Function accepts it; if it
114
+doesn't, no other doc matters.
115
+
116
+```bash
117
+TOKEN="YOUR_API_TOKEN"
118
+NODE="YOUR_NODE_UUID"
119
+FN="systemd-journal"
120
+
121
+read -r -d '' PAYLOAD <<'EOF'
122
+{ "info": true }
123
+EOF
124
+
125
+curl -sS -X POST \
126
+ -H 'Content-Type: application/json' \
127
+ -H "Authorization: Bearer $TOKEN" \
128
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=$FN" \
129
+ -d "$PAYLOAD"
130
+```
131
+
132
+### `required_params` widget schema
133
+
134
+Each entry of `required_params` is a UI-widget descriptor that
135
+tells a client what to render and what values are valid. Verified
136
+against the emit code in
137
+`src/collectors/network-viewer.plugin/network-viewer.c:1601-1731`
138
+and across the topology / logs / flows Functions.
139
+
140
+| Field | Type | Required | Purpose |
141
+|---|---|---|---|
142
+| `id` | string | yes | Parameter id (the body key) |
143
+| `name` | string | yes | Display label |
144
+| `help` | string | typical | Tooltip / help text |
145
+| `type` | string | yes | Widget kind -- see table below |
146
+| `options[]` | array | for select/multiselect/autocomplete | Each option: `{ "id": "<value>", "name": "<label>", "defaultSelected": <bool>? }` |
147
+| `unique_view` | bool | optional | Single-select enforces single-value semantics |
148
+| `multiselect` | bool | optional | Multi-value semantics |
149
+| `pattern` | string | optional | Regex/glob input for `text`/`pattern` widgets |
150
+| `default_value` | scalar | optional | Pre-filled value |
151
+
152
+Widget `type` values seen in source:
153
+
154
+| `type` | Meaning |
155
+|---|---|
156
+| `select` | Single-choice dropdown |
157
+| `multiselect` | Multi-choice; each option may have `defaultSelected:true` |
158
+| `autocomplete` | Text input backed by an autocomplete query (the Function itself answers via `mode:"autocomplete"` or similar) |
159
+| `text` | Free-form text |
160
+| `checkbox` | Boolean toggle |
161
+| `range` | Numeric range / slider (newer Functions) |
162
+| `pattern` | Pattern / regex input (newer Functions) |
163
+
164
+The widget array is the contract between the agent and any UI or
165
+script. To programmatically construct a valid body for a Function,
166
+walk `required_params` and emit the body shape it implies. There
167
+is no central widget builder API in source -- each collector emits
168
+the array directly via `buffer_json_*` calls -- so the agent's own
169
+`info=true` response is the only authoritative place to read the
170
+schema for a specific node version.
171
+
172
+---
173
+
174
+## Endpoints
175
+
176
+### List Functions on the nodes in a room
177
+
178
+`POST /api/v3/spaces/{spaceID}/rooms/{roomID}/functions`
179
+
180
+```bash
181
+TOKEN="YOUR_API_TOKEN"
182
+SPACE="YOUR_SPACE_ID"
183
+ROOM="YOUR_ROOM_ID"
184
+
185
+read -r -d '' PAYLOAD <<'EOF'
186
+{
187
+ "scope": { "nodes": [] },
188
+ "selectors": { "nodes": ["*"] }
189
+}
190
+EOF
191
+
192
+curl -sS -X POST \
193
+ -H 'Content-Type: application/json' \
194
+ -H "Authorization: Bearer $TOKEN" \
195
+ "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/functions" \
196
+ -d "$PAYLOAD"
197
+```
198
+
199
+Response top-level: `functions[]` (each entry: `name`, `version`,
200
+`help`, `ni[]`, `tags`, `access[]`, `priority`), `nodes[]` (each
201
+`{ ni, mg, nd, nm, st }`), `agents[]`, `versions`. Match
202
+`functions[].ni` to `nodes[].ni` to find which nodes expose a
203
+given Function.
204
+
205
+### Invoke a Function on a node
206
+
207
+`POST /api/v2/nodes/{nodeId}/function?function={functionName}`
208
+
209
+```bash
210
+TOKEN="YOUR_API_TOKEN"
211
+NODE="YOUR_NODE_UUID"
212
+FN="processes"
213
+
214
+read -r -d '' PAYLOAD <<'EOF'
215
+{
216
+ "last": 50,
217
+ "timeout": 30000
218
+}
219
+EOF
220
+
221
+curl -sS -X POST \
222
+ -H 'Content-Type: application/json' \
223
+ -H "Authorization: Bearer $TOKEN" \
224
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=$FN" \
225
+ -d "$PAYLOAD"
226
+```
227
+
228
+Optional headers:
229
+
230
+| Header | Purpose |
231
+|---|---|
232
+| `X-Transaction-Id: <uuid>` | Correlation id propagated to the agent. Optional. |
233
+
234
+---
235
+
236
+## Frequently registered Functions
237
+
238
+Function availability is per-node. The listing endpoint above is
239
+the only authoritative source. Below are common Functions on a
240
+stock Linux Netdata install (verified live):
241
+
242
+| Function | Family | What it returns |
243
+|---|---|---|
244
+| `processes` | table | Live process list with CPU / memory / I/O / page faults / PPID |
245
+| `network-connections` | table | Active sockets/connections (proto, state, addresses, ports, perf metrics) |
246
+| `network-interfaces` | table | Per-interface traffic, packet counts, drops, link status |
247
+| `network-sockets-tracing` | table | Detailed open-socket information |
248
+| `block-devices` | table | Per-block-device read/write throughput, ops, latency, utilization |
249
+| `mount-points` | table | Filesystem mount points with space and inode usage |
250
+| `containers-vms` | table | Active containers and cgroups with resource usage |
251
+| `systemd-services` | table | systemd service cgroups with process counts and resource use |
252
+| `netdata-streaming` | table | Parent-child streaming/replication status, data-flow metrics, ML status |
253
+| `netdata-api-calls` | table | Active and recent Netdata API requests with timings |
254
+| `netdata-metrics-cardinality` | table | Cardinality stats (instances, time-series per context/node) |
255
+| `systemd-journal` | logs | systemd journal entries -- see [query-logs.md](./query-logs.md) |
256
+| `windows-events` | logs | Windows event log channels (Windows nodes only) |
257
+| `otel-logs` | logs | OpenTelemetry log entries (when the OTEL log receiver is enabled) |
258
+| `topology:snmp` | topology | LLDP/CDP/FDB/STP-derived L2 topology -- see [query-topology.md](./query-topology.md) |
259
+| `flows:netflow` | flows | NetFlow / sFlow / IPFIX records -- see [query-flows.md](./query-flows.md) |
260
+
261
+Database collectors register a per-collector family of Functions
262
+when active: `<collector>:top-queries`, `<collector>:running-queries`,
263
+`<collector>:deadlock-info`, `<collector>:error-info` -- e.g.
264
+`postgres:top-queries`, `mysql:top-queries`, `mssql:deadlock-info`.
265
+The listing endpoint reports them when the collector is enabled.
266
+
267
+---
268
+
269
+## Examples (table-snapshot Functions)
270
+
271
+For logs / topology / flows examples, see the per-family guides
272
+linked at the top.
273
+
274
+### Example 1: top processes by CPU
275
+
276
+```bash
277
+TOKEN="YOUR_API_TOKEN"
278
+NODE="YOUR_NODE_UUID"
279
+
280
+read -r -d '' PAYLOAD <<'EOF'
281
+{
282
+ "last": 50,
283
+ "timeout": 30000
284
+}
285
+EOF
286
+
287
+curl -sS -X POST \
288
+ -H 'Content-Type: application/json' \
289
+ -H "Authorization: Bearer $TOKEN" \
290
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=processes" \
291
+ -d "$PAYLOAD" \
292
+ | jq '.data | length, (.[0:3])'
293
+```
294
+
295
+### Example 2: discover a Function's parameter widget set
296
+
297
+```bash
298
+read -r -d '' PAYLOAD <<'EOF'
299
+{ "info": true }
300
+EOF
301
+
302
+curl -sS -X POST \
303
+ -H 'Content-Type: application/json' \
304
+ -H "Authorization: Bearer $TOKEN" \
305
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=network-connections" \
306
+ -d "$PAYLOAD" \
307
+ | jq '.required_params | map({id, type, name, options: (.options | length // 0)})'
308
+```
309
+
310
+### Example 3: list the Functions on a single node
311
+
312
+```bash
313
+read -r -d '' PAYLOAD <<'EOF'
314
+{
315
+ "scope": { "nodes": ["YOUR_NODE_UUID"] },
316
+ "selectors": { "nodes": ["*"] }
317
+}
318
+EOF
319
+
320
+curl -sS -X POST \
321
+ -H 'Content-Type: application/json' \
322
+ -H "Authorization: Bearer $TOKEN" \
323
+ "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/functions" \
324
+ -d "$PAYLOAD" \
325
+ | jq -r '.functions[] | "\(.name)\t\(.tags // "")\t\(.help)"'
326
+```
327
+
328
+---
329
+
330
+## Developer reference (for collector authors)
331
+
332
+If you maintain a collector and want to register a Function (or are
333
+debugging why a Function returns `400 ErrInfoMissing`), read these
334
+files in this order. They are the authoritative sources.
335
+
336
+| File | Audience | Read for |
337
+|---|---|---|
338
+| `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` | All implementers | Functions v3 protocol -- envelope, simple-table vs log-explorer, facets, histograms, charts, field types, anchor/delta/PLAY pagination, error handling, edge cases. **The single most important reference.** |
339
+| `<repo>/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` | Collector authors | Practical step-by-step: how to ship a simple-table or log-explorer Function, with backend examples |
340
+| `<repo>/src/plugins.d/FUNCTION_UI_SCHEMA.json` | Validation | JSON Schema for Function responses; use it in unit tests |
341
+| `<repo>/src/plugins.d/README.md` (sections 470-637) | External plugins (any language) | Plugin protocol, `FUNCTION` / `FUNCTION_PAYLOAD` parsing, response framing, ACL, lifecycle |
342
+| `<repo>/src/plugins.d/DYNCFG.md` | External plugins exposing config | DynCfg protocol for go.d.plugin and other external collectors |
343
+| `<repo>/src/daemon/dyncfg/README.md` | Internal plugins exposing config | Internal DynCfg API |
344
+| `<repo>/src/database/rrdfunctions.h` | C collectors | C API: `rrd_function_add(host, st, name, timeout, priority, version, help, tags, access, sync, execute_cb, data)`; handler signature `rrd_function_execute_cb_t` |
345
+| `<repo>/src/go/plugin/framework/functions/README.md` | Go.d.plugin collectors | Go function manager: `manager.Register`, handler lifecycle, cancellation, worker pool |
346
+| `<repo>/docs/functions/` | Operators | Per-Function user docs and examples |
347
+
348
+Skeleton signatures (extracted from the headers above, for
349
+orientation only -- read the source for the real contract):
350
+
351
+```c
352
+/* C: register at boot, then emit responses via a buffer in the callback */
353
+void rrd_function_add(
354
+ RRDHOST *host, RRDSET *st,
355
+ const char *name, /* "module:method" */
356
+ int timeout, int priority, uint32_t version,
357
+ const char *help, const char *tags,
358
+ HTTP_ACCESS access, bool sync,
359
+ rrd_function_execute_cb_t execute_cb, void *execute_cb_data);
360
+
361
+typedef int (*rrd_function_execute_cb_t)(
362
+ struct rrd_function_execute *rfe, void *data);
363
+```
364
+
365
+```go
366
+// Go: register a Function with the framework's manager
367
+manager.Register(functions.Function{
368
+ Name: "module:method",
369
+ Description: "Help text",
370
+ Timeout: 30 * time.Second,
371
+ Params: []ParamDescriptor{ /* maps to required_params */ },
372
+ Handler: func(fn Function) { /* emit FuncResponse */ },
373
+})
374
+```
375
+
376
+```text
377
+# External plugins via the plugins.d protocol:
378
+FUNCTION [GLOBAL] "name params" timeout "help" "tags" "access" priority version
379
+# On call:
380
+FUNCTION <txn_id> <timeout> "name params" "<access>" "<source>"
381
+# Reply:
382
+FUNCTION_RESULT_BEGIN <txn_id> <http_code> <content_type> <expiry>
383
+<JSON envelope: status, v, type, help, accepted_params, required_params, has_history, update_every, data, ...>
384
+FUNCTION_RESULT_END
385
+```
386
+
387
+The full envelope and required-params widget schemas above ARE the
388
+contract a collector implementation must satisfy. Read
389
+`src/plugins.d/README.md` for the line-protocol details and
390
+`src/database/rrdfunctions.h` for the C API.
391
+
392
+---
393
+
394
+## Limits and gotchas
395
+
396
+- **Cloud default timeout is 120 s** for Function calls; pass
397
+ `"timeout": <ms>` in the body for slower Functions but Cloud
398
+ may impose its own ceiling.
399
+- **Response is NOT streamed.** The Cloud proxy collects the full
400
+ agent response and returns it in one body. For potentially
401
+ huge results (logs, flows), narrow the time window or use the
402
+ Function's pagination (`last`, `anchor`) rather than relying on
403
+ streaming.
404
+- **Node must be `reachable`.** A `stale` node returns HTTP 400
405
+ with `errorMsgKey: "ErrInstanceNotReachable"`. Verify with the
406
+ discovery endpoints in [SKILL.md](./SKILL.md).
407
+- **Permission**: the cloud token must include
408
+ `PermissionFunctionExec` on the target space. `scope:all`
409
+ works; `scope:grafana-plugin` does NOT.
410
+- **Function name is case-sensitive** -- wrong casing returns 400.
411
+- **`info=true` does NOT bypass auth.** ACL is enforced on every
412
+ call regardless of body.
413
+- **The agent's own `info=true` response is authoritative for
414
+ parameters.** Tables in this skill can drift relative to the
415
+ running version. When in doubt, ask the agent.
docs/netdata-ai/skills/query-netdata-cloud/query-logs.md
new
+341
@@ -0,0 +1,341 @@
1
+# Query log Functions via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+For the generic Function transport and the canonical protocol
6
+reference, see [query-functions.md](./query-functions.md). The
7
+authoritative protocol spec is
8
+`<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` (specifically the
9
+"Log Explorer Format" section).
10
+
11
+Log Functions are the **Log Explorer** class of Functions
12
+(`has_history: true` in their `info` response). They return a
13
+**time-windowed skim** of a larger log dataset, with **facets**
14
+(per-field value counts) for drill-down and an optional
15
+**histogram** (bucketed counts over time) for context.
16
+
17
+Three log Functions exist today, each backed by a different log
18
+source. Their request and response shapes follow the same standard
19
+envelope, but the journal-field set differs per source:
20
+
21
+| Function | Source | Notes |
22
+|---|---|---|
23
+| `systemd-journal` | systemd journal namespaces (system, user, namespace-specific, remote-forwarded) | Linux nodes |
24
+| `windows-events` | Windows event log channels | Windows nodes |
25
+| `otel-logs` | OpenTelemetry logs ingested by the agent | Any node with the OTEL log receiver enabled |
26
+
27
+Confirm which are registered on a node via the
28
+function-listing endpoint in
29
+[query-functions.md](./query-functions.md). Field names below are
30
+illustrative for `systemd-journal`; the same Function payload keys
31
+(`after`, `before`, `last`, `query`, `facets`, `histogram`,
32
+`__logs_sources`, ...) apply to the other two -- only the **values
33
+and column names** differ per source.
34
+
35
+---
36
+
37
+## Endpoint
38
+
39
+`POST /api/v2/nodes/{nodeId}/function?function=systemd-journal`
40
+
41
+Same shape as any other Function call. The body is the
42
+`systemd-journal` Function's payload.
43
+
44
+---
45
+
46
+## Discover the Function's parameters
47
+
48
+Always start with `info=true` to confirm the current schema -- the
49
+Function's parameter set evolves across agent versions.
50
+
51
+```bash
52
+TOKEN="YOUR_API_TOKEN"
53
+NODE="YOUR_NODE_UUID"
54
+
55
+read -r -d '' PAYLOAD <<'EOF'
56
+{ "info": true }
57
+EOF
58
+
59
+curl -sS -X POST \
60
+ -H 'Content-Type: application/json' \
61
+ -H "Authorization: Bearer $TOKEN" \
62
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \
63
+ -d "$PAYLOAD"
64
+```
65
+
66
+The `accepted_params` array in the response tells you which keys the
67
+agent currently accepts.
68
+
69
+---
70
+
71
+## Body keys (current as of `STATUS_FILE_VERSION = 28`)
72
+
73
+| Key | Type | Purpose |
74
+|---|---|---|
75
+| `info` | bool | Discovery only; do not combine with a real query |
76
+| `after` | int | Unix ms timestamp; lower bound. Negative = relative seconds from `before` |
77
+| `before` | int | Unix ms timestamp; upper bound. Negative = relative seconds from now |
78
+| `last` | int | Page size (rows). Default 200 |
79
+| `direction` | string | `backward` (default; newest first) or `forward` |
80
+| `anchor` | int | Per-row cursor for pagination |
81
+| `query` | string | Free-text search across journal fields |
82
+| `facets` | string[] | Field names to group by (returns counts per value) |
83
+| `histogram` | string | Field name to bucket-by-time |
84
+| `__logs_sources` | string | Source selector. Common values: `all`, `all-local-logs`, `all-local-system-logs`, `all-local-user-logs`, `all-local-namespaces`, plus per-namespace strings like `<namespace-name>` for a specific journal namespace |
85
+| `if_modified_since` | int | Tail mode -- skip if no new data |
86
+| `data_only` | bool | Skip metadata for a faster query |
87
+| `sampling` | int | Cap on rows scanned when search would otherwise be huge |
88
+| `slice` | bool | Native backend filter (faster, less flexible) |
89
+| `delta` | bool | Incremental histogram updates |
90
+| `tail` | bool | Append-mode (combine with `if_modified_since`) |
91
+
92
+`info=true` returns the current authoritative list; rely on it, not
93
+this table, when in doubt.
94
+
95
+---
96
+
97
+## Response shape
98
+
99
+The response uses the **standard Function envelope** (top-level
100
+keys `status`, `v`, `type`, `help`, `accepted_params`,
101
+`required_params`, `has_history`, `update_every`, `data`, ...).
102
+For log Functions, `type` is the source name (`logs` family
103
+discriminator). Verified live against the agent-events node:
104
+
105
+| Top-level key | Description |
106
+|---|---|
107
+| `status` | HTTP-style status integer (200 on success) |
108
+| `v` | Function schema version |
109
+| `type` | Family discriminator (carries `logs`-family value) |
110
+| `help` / `accepted_params` / `required_params` | Discovery metadata (see [query-functions.md](./query-functions.md#info-true-discovery)) |
111
+| `data` | **Array** of row arrays -- this is the result rows |
112
+| `columns` | Object keyed by column name; per-column metadata: `index` (position in each row of `data`), `name` (display label), `type` (string / timestamp / integer / ...), `visible`, `unique_key`, `sort`, `summary` (`count` / `min` / `max` / `sum` / ...), `filter` (e.g. `range`), `visualization`, `value_options` (for transforms like `datetime_usec`) |
113
+| `facets` | Array of facet records: `{id, name, options[]}` where each option is `{id, name, count}`. Use to drill down by field value. |
114
+| `histogram` | If requested: time-bucketed counts. Object with `chart`, `id`, `name`, plus per-bucket data |
115
+| `pagination` | Cursor info (`anchor`, `direction`, `last`, ...) for the next page |
116
+| `default_charts` | Suggested chart configuration |
117
+| `default_sort_column` | Recommended sort column |
118
+| `available_histograms` | Field names that the agent can histogram-bucket |
119
+| `_request` | Echo of the parsed request (defaults applied) |
120
+| `versions` | Source/version map for cache invalidation |
121
+| `last_modified` | Last-data timestamp |
122
+| `expires` | Suggested cache expiry |
123
+| `partial` | True if the result was capped by `sampling` or timeout |
124
+| `message` | Optional info / warning string |
125
+| `_journal_files` / `_fstat_caching` / `_sampling` / `_stats` | systemd-journal-specific debug counters |
126
+
127
+### Reading rows
128
+
129
+`data` is an array of rows. Each row is itself an array whose
130
+positions match `columns.<key>.index`. To pretty-print a single
131
+row by column name:
132
+
133
+```bash
134
+jq '.columns as $c
135
+ | .data[0] as $row
136
+ | $c | to_entries
137
+ | sort_by(.value.index)
138
+ | map({(.key): $row[.value.index]})
139
+ | add' response.json
140
+```
141
+
142
+---
143
+
144
+## Multi-value field selections (AND-of-OR filtering)
145
+
146
+The `selections` POST-payload key is a structured field-filter
147
+mechanism the Netdata `systemd-journal` Function (powered by the
148
+libnetdata `facets` engine) supports. It is **distinct from raw
149
+journalctl's `KEY=value` matches**: a single field can carry
150
+multiple allowed values, and multiple fields are AND'd.
151
+
152
+### Shape
153
+
154
+`selections` is an object whose keys are journal field names
155
+and whose values are arrays of allowed values:
156
+
157
+```json
158
+{
159
+ "selections": {
160
+ "FIELD1": ["A", "B", "C"],
161
+ "FIELD2": ["D", "E"]
162
+ }
163
+}
164
+```
165
+
166
+Semantics (verified at
167
+`<repo>/src/libnetdata/facets/logs_query_status.h:386-466`):
168
+
169
+- **Between fields: AND.** All listed fields must match.
170
+- **Between values for the same field: OR.** Any one of the
171
+ listed values matches.
172
+
173
+So the example above is logically:
174
+
175
+```
176
+(FIELD1 in A, B, C) AND (FIELD2 in D, E)
177
+```
178
+
179
+### Why this matters for performance
180
+
181
+A namespace can hold tens of thousands to hundreds of thousands
182
+of records per day. A bare `query` (FTS) scans every record's
183
+indexed text fields. Structured `selections` matches use the
184
+facet engine's per-field index, which is dramatically faster
185
+once the time window is fixed.
186
+
187
+**Rule of thumb:** narrow with `selections` first, then refine
188
+with `query` (FTS) only as a residual narrower over the
189
+already-sliced subset.
190
+
191
+### Reserved keys inside `selections`
192
+
193
+- `__logs_sources` (per `LQS_PARAMETER_SOURCE`,
194
+ `logs_query_status.h:407`) is treated as the source-type
195
+ filter (e.g. `all-local-namespaces`, `<namespace-name>`).
196
+ Using it inside `selections` is equivalent to setting the
197
+ top-level `__logs_sources` parameter.
198
+- `query` inside `selections` is ignored
199
+ (`logs_query_status.h:398`); use the top-level `query`.
200
+
201
+### Example: structured filter + FTS narrower
202
+
203
+```json
204
+{
205
+ "after": -86400,
206
+ "before": 0,
207
+ "last": 500,
208
+ "__logs_sources": "agent-events",
209
+ "selections": {
210
+ "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"],
211
+ "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly"]
212
+ },
213
+ "query": "deadlock"
214
+}
215
+```
216
+
217
+This selects the cross-product of crash-class records on those
218
+two versions (index-resolved), then FTS-filters the result for
219
+the substring `deadlock`. Index-friendly even on a
220
+~200k-records-per-day namespace.
221
+
222
+### Anti-pattern (avoid)
223
+
224
+```json
225
+{
226
+ "after": -604800,
227
+ "before": 0,
228
+ "__logs_sources": "agent-events",
229
+ "query": "SIGSEGV"
230
+}
231
+```
232
+
233
+A 7-day FTS over the entire namespace with no structured
234
+narrowing. Slow and costly on large namespaces. Always pair FTS
235
+with at least one structured `selections` field.
236
+
237
+---
238
+
239
+## Examples
240
+
241
+### Example 1: most recent 50 entries from a specific namespace
242
+
243
+```bash
244
+TOKEN="YOUR_API_TOKEN"
245
+NODE="YOUR_NODE_UUID"
246
+NAMESPACE="systemd" # or "agent-events", "any-namespace-name"
247
+
248
+read -r -d '' PAYLOAD <<EOF
249
+{
250
+ "after": -3600,
251
+ "before": 0,
252
+ "last": 50,
253
+ "direction": "backward",
254
+ "__logs_sources": "${NAMESPACE}"
255
+}
256
+EOF
257
+
258
+curl -sS -X POST \
259
+ -H 'Content-Type: application/json' \
260
+ -H "Authorization: Bearer $TOKEN" \
261
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \
262
+ -d "$PAYLOAD"
263
+```
264
+
265
+### Example 2: full-text search with histogram
266
+
267
+```bash
268
+TOKEN="YOUR_API_TOKEN"
269
+NODE="YOUR_NODE_UUID"
270
+
271
+read -r -d '' PAYLOAD <<'EOF'
272
+{
273
+ "after": -86400,
274
+ "before": 0,
275
+ "last": 100,
276
+ "query": "OOM",
277
+ "histogram": "PRIORITY",
278
+ "facets": ["_SYSTEMD_UNIT", "PRIORITY"],
279
+ "__logs_sources": "all-local-system-logs"
280
+}
281
+EOF
282
+
283
+curl -sS -X POST \
284
+ -H 'Content-Type: application/json' \
285
+ -H "Authorization: Bearer $TOKEN" \
286
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \
287
+ -d "$PAYLOAD"
288
+```
289
+
290
+### Example 3: paginate forward from a known anchor
291
+
292
+```bash
293
+TOKEN="YOUR_API_TOKEN"
294
+NODE="YOUR_NODE_UUID"
295
+ANCHOR=1700000123456789 # cursor from previous response
296
+
297
+read -r -d '' PAYLOAD <<EOF
298
+{
299
+ "anchor": ${ANCHOR},
300
+ "direction": "forward",
301
+ "last": 200,
302
+ "__logs_sources": "all-local-logs"
303
+}
304
+EOF
305
+
306
+curl -sS -X POST \
307
+ -H 'Content-Type: application/json' \
308
+ -H "Authorization: Bearer $TOKEN" \
309
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \
310
+ -d "$PAYLOAD"
311
+```
312
+
313
+---
314
+
315
+## Limits and gotchas
316
+
317
+- **Time bounds are unix-microseconds**, not seconds, when given as
318
+ positive integers. Negative integers are relative seconds (`-3600`
319
+ = "one hour ago relative to `before`"). Mixing units is the most
320
+ common bug.
321
+- **Default cloud timeout is 120 s**, but very large queries
322
+ (thousands of rows over weeks of data) can hit it. Narrow the
323
+ window or use `sampling`.
324
+- **`__logs_sources` is required** to scope to a specific journal
325
+ namespace. Without it, the query targets all-local-logs which on a
326
+ busy host can be hundreds of GB.
327
+- **Permission**: the cloud token must have a role that includes
328
+ log-read access (function tags include `logs`). `scope:all` works;
329
+ `scope:grafana-plugin` does NOT.
330
+- **Response can be tens of MB** when `facets` include high-cardinality
331
+ fields (`MESSAGE_ID`, `_BOOT_ID`, `_PID`). Pick facets carefully.
332
+
333
+---
334
+
335
+## Discovering a journal namespace
336
+
337
+If the host runs `journalctl --namespace=<name>`, the same name is
338
+the value of `__logs_sources`. The agent's `info=true` response
339
+enumerates all visible sources under `accepted_params._logs_sources`
340
+or under a `required_params` widget -- inspect that widget's
341
+`options[]` to learn which sources the node actually exposes.
docs/netdata-ai/skills/query-netdata-cloud/query-members.md
new
+81
@@ -0,0 +1,81 @@
1
+# List members via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+Members are Cloud-only. There is no agent-side equivalent.
7
+
8
+---
9
+
10
+## Endpoint
11
+
12
+`GET /api/v2/spaces/{spaceID}/members` -- list members of a space.
13
+
14
+## Use the wrapper
15
+
16
+```bash
17
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
18
+agents_load_env
19
+
20
+# All members of a space.
21
+agents_query_cloud GET "/api/v2/spaces/$SPACE/members"
22
+```
23
+
24
+## Per-member response fields
25
+
26
+Verified live -- response is a JSON array, each entry:
27
+
28
+| Field | Description |
29
+|---|---|
30
+| `memberID` | UUID identifying the user's membership in this space (NOT the user's account id) |
31
+| `accountID` | UUID identifying the user's Cloud account |
32
+| `name` | Display name |
33
+| `email` | Email |
34
+| `avatarURL` | Avatar image URL (may be empty) |
35
+| `role` | One of: `admin`, `manager`, `troubleshooter`, `observer`, `member`, `billing`, ... |
36
+| `joinMethod` | How they joined: `invite`, `auto`, `sso`, ... |
37
+| `joinedAt` | RFC3339 timestamp |
38
+| `deactivated` | `true` if the membership has been deactivated (cannot view space until reactivated) |
39
+
40
+## Common patterns
41
+
42
+```bash
43
+# Members by role.
44
+agents_query_cloud GET "/api/v2/spaces/$SPACE/members" \
45
+ | jq -r 'group_by(.role) | map({(.[0].role): length}) | add'
46
+
47
+# Active admins of a space.
48
+agents_query_cloud GET "/api/v2/spaces/$SPACE/members" \
49
+ | jq -r '.[] | select(.role=="admin" and (.deactivated|not)) | .name'
50
+
51
+# Resolve a user's display name from an accountID found elsewhere
52
+# (e.g. in an alert-config audit field).
53
+TARGET_ACCOUNT="<account-uuid>"
54
+agents_query_cloud GET "/api/v2/spaces/$SPACE/members" \
55
+ | jq -r --arg id "$TARGET_ACCOUNT" '.[] | select(.accountID==$id) | "\(.name) <\(.email)>"'
56
+```
57
+
58
+## Limits and gotchas
59
+
60
+- **Member visibility depends on the caller's role.** Observers
61
+ may not see all members. The response is filtered server-side
62
+ by what the caller is permitted to see.
63
+- **`memberID` vs `accountID`**: the former is per-space (one
64
+ user gets a different `memberID` in each space); the latter is
65
+ the user's global Cloud account id and stays constant across
66
+ spaces. Use `accountID` when correlating across spaces.
67
+- **Deactivated members still appear in the list** with
68
+ `deactivated:true`. Filter explicitly to exclude them.
69
+- **Email and name are personal data.** Treat the response as
70
+ semi-sensitive: do not paste raw response bodies into
71
+ committed artifacts. Direct working output to
72
+ `<repo>/.local/audits/...` (gitignored).
73
+
74
+## See also
75
+
76
+- [query-rooms.md](./query-rooms.md) -- room-level membership
77
+ (`isMember`, `permissions[]`, `member_count`).
78
+- [query-feed.md](./query-feed.md) -- audit-feed events include
79
+ `user-create`, `space-user-added`, `space-user-removed`,
80
+ `user-space-permissions-changed`, `room-user-added`, ... use
81
+ the feed to track membership changes over time.
docs/netdata-ai/skills/query-netdata-cloud/query-metrics.md
renamed
+27
@@ -237,6 +237,33 @@ The time range is divided into `points` equal intervals. Each interval is aggreg
237
238
---
239
240
+### Time resolution: `duration ÷ points = seconds per point`
241
+
242
+This is the most common assistant mistake. The number of `points` does NOT mean "give me per-second data". It means "split the duration into N equal buckets". The actual time resolution per point is:
243
+
244
+```
245
+seconds_per_point = abs(duration) ÷ points
246
+```
247
+
248
+**To get per-second data, set `points` equal to the duration in seconds.**
249
+
250
+Examples:
251
+
252
+| You want | Set `after` | Set `points` | Result |
253
+|---|---|---|---|
254
+| Per-second resolution, last 2 minutes | `-120` | `120` | 1 second per point |
255
+| Per-second resolution, last 5 minutes | `-300` | `300` | 1 second per point |
256
+| 10-second buckets, last 10 minutes | `-600` | `60` | 10 seconds per point |
257
+| Per-minute resolution, last hour | `-3600` | `60` | 60 seconds per point |
258
+
259
+**Common mistake**: requesting `after: -600, points: 30` and expecting per-second data. Result: 600 ÷ 30 = **20 seconds per point** (heavily aggregated). Per-second data over 10 minutes requires `after: -600, points: 600` (which is at the 500-point cap; either request 8 minutes 20 seconds at 500 points, or accept a slightly coarser resolution).
260
+
261
+**Per-second data also requires that the dbengine tier 0 (per-second storage) covers the requested time range.** If the agent's tier 0 retention is shorter than `abs(after)`, the engine auto-selects a coarser tier (per-minute or per-hour). Force tier 0 with `"tier": 0` in the window if you need to assert per-second data is actually available -- the query will fail rather than silently downsample.
262
+
263
+**`points: 0` (the default) is NOT "per-second"** -- it requests "all available points", which is whatever the engine returns within its 500-point cap and the storage tier's natural granularity. For a 1-hour query against tier-0 storage, the engine still aggregates because 3600 > 500.
264
+
265
+---
266
+
267
### How the Query Pipeline Works
268
269
The query engine is a pipeline with two aggregation stages:
docs/netdata-ai/skills/query-netdata-cloud/query-nodes.md
new
+120
@@ -0,0 +1,120 @@
1
+# List nodes via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+For a single agent's own identity (`/api/v3/info` direct, hardware
7
+labels, vnodes, parent/child role), see
8
+[../query-netdata-agents/query-nodes.md](../query-netdata-agents/query-nodes.md).
9
+This file covers the **Cloud-side** enumeration -- nodes across a
10
+room, across a space, with full metadata payloads.
11
+
12
+---
13
+
14
+## Endpoints
15
+
16
+| Method | Path | Purpose |
17
+|---|---|---|
18
+| `POST` | `/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes` | List nodes in a room (full metadata) |
19
+
20
+The body is `{}` for "all nodes in the room" or accepts
21
+filters/options that mirror the metrics-query body's
22
+`scope`/`selectors` shape -- see
23
+[query-metrics.md](./query-metrics.md) for the cross-cutting
24
+filter language.
25
+
26
+## Use the wrapper
27
+
28
+```bash
29
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
30
+agents_load_env
31
+
32
+agents_query_cloud POST "/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" '{}'
33
+```
34
+
35
+The wrapper emits only the response body; `NETDATA_CLOUD_TOKEN`
36
+never reaches stdout.
37
+
38
+## Per-node response fields
39
+
40
+Verified live -- response is a JSON array, each entry an object:
41
+
42
+| Field | Description |
43
+|---|---|
44
+| `nd` | **Node UUID.** This is the value to pass anywhere the API expects a node id (e.g. `/api/v2/nodes/{nd}/function?...`) |
45
+| `mg` | Machine GUID (stable per OS install) |
46
+| `nm` | Hostname |
47
+| `state` | `reachable` (live) / `stale` (disconnected) / `offline` |
48
+| `v` | Agent version (e.g. `v2.10.3-nightly`) |
49
+| `labels` | Object: all `_*` chart-labels keyed by name (architecture, kernel, OS, CPU count, RAM, container/k8s/cloud-provider info) |
50
+| `hw` | `{cpus, memory, disk_space, architecture}` summary |
51
+| `os` | `{nm, v, kernel}` summary |
52
+| `health` | Alert-status summary: `{status, alerts: {warning, critical}}` |
53
+| `capabilities` | Feature flags: `ml`, `funcs`, `health`, etc. |
54
+| `room_memberships` | Other rooms this node is in |
55
+| `eligibility` | Per-feature eligibility (e.g. for paid features) |
56
+| `replication`, `replication_factor` | Streaming / parent-child replication state |
57
+| `isPreferred` | Whether the node is the preferred parent for its room |
58
+
59
+## Common patterns
60
+
61
+```bash
62
+# Hostname -> node UUID lookup.
63
+agents_query_cloud POST "/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" '{}' \
64
+ | jq -r --arg HOST "costa-desktop" '.[] | select(.nm==$HOST) | .nd'
65
+
66
+# All "reachable" nodes' (UUID, hostname, version) tuples.
67
+agents_query_cloud POST "/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" '{}' \
68
+ | jq -r '.[] | select(.state=="reachable") | "\(.nd)\t\(.nm)\t\(.v)"'
69
+
70
+# Nodes whose label `_is_parent` is true.
71
+agents_query_cloud POST "/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" '{}' \
72
+ | jq -r '.[] | select(.labels._is_parent=="true") | .nm'
73
+
74
+# Aggregate by cloud provider.
75
+agents_query_cloud POST "/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" '{}' \
76
+ | jq -r '[.[] | .labels._cloud_provider_type // "unknown"] | group_by(.) | map({(.[0]): length}) | add'
77
+```
78
+
79
+## Hardware / OS facts
80
+
81
+Hardware and OS facts live in `.labels`:
82
+
83
+| Question | Field |
84
+|---|---|
85
+| CPU architecture | `.labels._architecture` |
86
+| Kernel version | `.labels._kernel_version` |
87
+| OS name / version | `.labels._os_name`, `._os_version` |
88
+| CPU cores | `.labels._system_cores` |
89
+| Total RAM (bytes) | `.labels._system_ram_total` |
90
+| Total disk space (bytes) | `.labels._system_disk_space` |
91
+| Container / virt | `.labels._container`, `._is_k8s_node` |
92
+| Cloud provider / region / instance type | `.labels._cloud_provider_type`, `._cloud_instance_region`, `._cloud_instance_type` |
93
+| Parent role | `.labels._is_parent` (`"true"` / `"false"` strings) |
94
+
95
+For deeper per-host introspection (vnodes, failed jobs, claim_id),
96
+fall through to the agent-direct path in
97
+[../query-netdata-agents/query-nodes.md](../query-netdata-agents/query-nodes.md)
98
+and [../query-netdata-agents/query-dyncfg.md](../query-netdata-agents/query-dyncfg.md).
99
+
100
+## Limits and gotchas
101
+
102
+- **Stale nodes appear in the list** -- always check `.state`
103
+ before issuing further queries against `nd`.
104
+- **The full label set is large** (50+ keys per node). Use
105
+ `jq` projections to keep responses readable.
106
+- **Cross-room view requires re-querying.** A node can be in
107
+ multiple rooms; use `room_memberships` to detect duplicates
108
+ when aggregating across rooms.
109
+- **Multi-space view requires multiple Cloud calls.** Iterate
110
+ over `/api/v2/spaces` -> `/api/v2/spaces/{sp}/rooms` ->
111
+ `/api/v3/spaces/{sp}/rooms/{rm}/nodes`.
112
+
113
+## See also
114
+
115
+- [query-rooms.md](./query-rooms.md) -- enumerate rooms (and
116
+ their `node_count`, `member_count`, permissions).
117
+- [query-functions.md](./query-functions.md) -- invoke a Function
118
+ on a specific node by UUID.
119
+- [../query-netdata-agents/query-nodes.md](../query-netdata-agents/query-nodes.md)
120
+ -- single-host identity, vnodes, claim_id.
docs/netdata-ai/skills/query-netdata-cloud/query-rooms.md
new
+80
@@ -0,0 +1,80 @@
1
+# List rooms via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+
6
+Rooms are Cloud-only organizational units. There is no agent-side
7
+equivalent.
8
+
9
+---
10
+
11
+## Endpoint
12
+
13
+`GET /api/v2/spaces/{spaceID}/rooms` -- list all rooms the user can
14
+see in a space.
15
+
16
+## Use the wrapper
17
+
18
+```bash
19
+source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
20
+agents_load_env
21
+
22
+# All rooms in a space.
23
+agents_query_cloud GET "/api/v2/spaces/$SPACE/rooms"
24
+```
25
+
26
+## Per-room response fields
27
+
28
+Verified live -- response is a JSON array, each entry:
29
+
30
+| Field | Description |
31
+|---|---|
32
+| `id` | **Room UUID.** Use this anywhere the API expects a room id. |
33
+| `slug` | URL-safe identifier (e.g. `agent-events-r0gtre6`) |
34
+| `name` | Human-readable name (e.g. `agent-events`) |
35
+| `description` | Free-form description (may be `null`) |
36
+| `private` | `true` if invitation-only |
37
+| `untouchable` | `true` for the auto-managed "All nodes" room (cannot be deleted) |
38
+| `node_count` | Number of nodes assigned to this room |
39
+| `member_count` | Number of users in this room |
40
+| `isMember` | Whether the calling user is a member |
41
+| `silencing_state` | Notification silencing state for the calling user |
42
+| `permissions` | Array of permission strings the caller has on the room |
43
+| `createdAt` | RFC3339 timestamp |
44
+
45
+## Common patterns
46
+
47
+```bash
48
+# Find the room id by name.
49
+agents_query_cloud GET "/api/v2/spaces/$SPACE/rooms" \
50
+ | jq -r --arg NAME "agent-events" '.[] | select(.name==$NAME) | .id'
51
+
52
+# Rooms with at least one reachable node, sorted by node count.
53
+agents_query_cloud GET "/api/v2/spaces/$SPACE/rooms" \
54
+ | jq -r 'sort_by(-.node_count) | .[] | select(.node_count > 0) | "\(.name)\t\(.node_count)"'
55
+
56
+# Rooms the caller can administer.
57
+agents_query_cloud GET "/api/v2/spaces/$SPACE/rooms" \
58
+ | jq -r '.[] | select(.permissions | index("room:Delete")) | .name'
59
+```
60
+
61
+## Limits and gotchas
62
+
63
+- **The "All nodes" room is special.** It auto-includes every
64
+ node in the space and cannot be deleted. Filter on
65
+ `untouchable=true` if you need it specifically.
66
+- **`node_count` and `member_count` are server-side counts** --
67
+ no need to fetch nodes/members just to get the totals.
68
+- **Permissions vary per user.** The same room returns different
69
+ `permissions[]` arrays depending on the caller's role; another
70
+ user may see fewer permissions.
71
+
72
+## See also
73
+
74
+- [query-nodes.md](./query-nodes.md) -- enumerate nodes in a
75
+ specific room.
76
+- [query-members.md](./query-members.md) -- list members of a
77
+ space (rooms inherit space membership scoped by room
78
+ permissions).
79
+- [query-alerts.md](./query-alerts.md) -- silencing rules
80
+ reference rooms by id.
docs/netdata-ai/skills/query-netdata-cloud/query-topology.md
new
+254
@@ -0,0 +1,254 @@
1
+# Query topology Functions via Netdata Cloud
2
+
3
+This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4
+Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5
+For the generic Function transport (used by topology, logs, flows,
6
+and table-snapshot Functions alike), see
7
+[query-functions.md](./query-functions.md).
8
+
9
+Topology Functions return a graph: a list of **actors** (nodes in
10
+the graph) and **links** (edges). They differ from log Functions
11
+(which return a time-windowed skim of a larger dataset) and from
12
+table-snapshot Functions (which return one full table).
13
+
14
+---
15
+
16
+## Function names registered today
17
+
18
+Verified live and in source:
19
+
20
+| Function | Source collector | Layer | What it discovers |
21
+|---|---|---|---|
22
+| `topology:snmp` | `src/go/plugin/go.d/collector/snmp_topology/` | L2 | LLDP/CDP-discovered switches+routers, FDB-derived endpoint locations, STP-derived parent-child relationships |
23
+
24
+The `topology:` prefix is the canonical namespace; only `snmp` is
25
+registered today (as of `STATUS_FILE_VERSION = 28`). When new
26
+topology collectors land (network-viewer connections, streaming
27
+parent-child, k8s service mesh, etc.), they will follow the same
28
+`topology:<source>` naming pattern and the same response envelope
29
+documented below. Always confirm via the function-listing endpoint
30
+in [query-functions.md](./query-functions.md) before assuming a
31
+given topology Function exists on a node.
32
+
33
+---
34
+
35
+## Endpoint and request
36
+
37
+Use the standard Cloud Function-call endpoint. Topology is just a
38
+Function; nothing is special about its URL.
39
+
40
+`POST /api/v2/nodes/{nodeId}/function?function=topology:snmp`
41
+
42
+```bash
43
+TOKEN="YOUR_API_TOKEN"
44
+NODE="YOUR_NODE_UUID"
45
+
46
+read -r -d '' PAYLOAD <<'EOF'
47
+{
48
+ "selections": {
49
+ "nodes_identity": ["mac"],
50
+ "map_type": ["lldp_cdp_managed"],
51
+ "inference_strategy": ["fdb_minimum_knowledge"],
52
+ "managed_snmp_device_focus": ["all_devices"],
53
+ "depth": ["all"]
54
+ },
55
+ "timeout": 60000,
56
+ "last": 200
57
+}
58
+EOF
59
+
60
+curl -sS -X POST \
61
+ -H 'Content-Type: application/json' \
62
+ -H "Authorization: Bearer $TOKEN" \
63
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:snmp" \
64
+ -d "$PAYLOAD"
65
+```
66
+
67
+Always start with `{"info":true}` to discover the parameters the
68
+node currently accepts -- topology Function parameter sets evolve
69
+with the collector.
70
+
71
+### Body parameters (topology:snmp)
72
+
73
+Verified against `src/go/plugin/go.d/collector/snmp_topology/`:
74
+
75
+| Parameter | Type | Allowed values | Purpose |
76
+|---|---|---|---|
77
+| `nodes_identity` | string | `ip`, `mac` | Collapse / distinguish actors by IP or MAC |
78
+| `map_type` | string | `lldp_cdp_managed`, `high_confidence_inferred`, `all_devices_low_confidence` | Which discovery sources to include |
79
+| `inference_strategy` | string | `fdb_minimum_knowledge`, `stp_parent_tree`, `fdb_pairwise_minimum_knowledge`, `stp_fdb_correlated`, `cdp_fdb_hybrid` | How endpoint-to-switch placement is inferred when LLDP/CDP coverage is incomplete |
80
+| `managed_snmp_device_focus` | string | `all_devices`, `ip:<prefix>` | Restrict the discovery surface to a subset of managed SNMP devices |
81
+| `depth` | string | `0`-`10`, or `all` | Hops away from the focus device to include |
82
+
83
+`selections` is the standard Function selection object; values are
84
+arrays even for single-valued parameters (Netdata convention).
85
+
86
+---
87
+
88
+## Response envelope
89
+
90
+Topology Functions wrap their content in the standard Function
91
+envelope. Top-level keys (verified live):
92
+
93
+| Key | Description |
94
+|---|---|
95
+| `status` | HTTP-style status integer (200 on success) |
96
+| `v` | Function schema version |
97
+| `type` | **`topology`** -- the family discriminator |
98
+| `help` | Human description |
99
+| `accepted_params` | Parameter names the Function accepts |
100
+| `required_params` | Per-parameter UI widgets |
101
+| `has_history` | Whether the Function supports `after`/`before` history |
102
+| `update_every` | Suggested refresh interval (seconds) |
103
+| `data` | The graph payload (object) |
104
+
105
+### `data` object
106
+
107
+| Key | Description |
108
+|---|---|
109
+| `schema_version` | Topology schema version (e.g. `2.0`) |
110
+| `source` | Discovery source string (e.g. `snmp`) |
111
+| `layer` | OSI layer the topology lives at (e.g. `2`, `3`) |
112
+| `agent_id` | Identifier of the producing agent |
113
+| `collected_at` | RFC3339 timestamp |
114
+| `view` | Rendered view kind (e.g. `summary`, `detail`) |
115
+| `actors[]` | Graph nodes -- see schema below |
116
+| `links[]` | Graph edges -- see schema below |
117
+| `flows[]` | Optional, for sources that emit flow records alongside the topology |
118
+| `stats` | Per-source counters (devices polled, fdb entries, etc.) |
119
+| `metrics` | Optional metric block |
120
+| `ip_policy` | Optional, when `nodes_identity:ip` is in effect |
121
+
122
+### Actor record
123
+
124
+| Key | Description |
125
+|---|---|
126
+| `actor_id` | Stable identifier of the actor; format is source-specific (e.g. `mac:<addr>[,<addr>...]` for SNMP at L2; `ip:<addr>` when collapsed by IP) |
127
+| `actor_type` | e.g. `device`, `endpoint`, `vlan`, `service` |
128
+| `layer` | OSI layer (`2`, `3`, ...) |
129
+| `source` | Source collector (e.g. `snmp`) |
130
+| `match` | Discovery facts (sysName, sysObjectID, OUI, ...) |
131
+| `attributes` | Free-form per-actor properties |
132
+| `derived` | Computed annotations (vendor inference, role, ...) |
133
+| `labels` | Tag-style key/value pairs |
134
+| `tables` | Per-actor sub-tables (e.g. interfaces, ARP entries) |
135
+
136
+### Link record
137
+
138
+| Key | Description |
139
+|---|---|
140
+| `layer` | Link layer |
141
+| `protocol` | Discovery protocol (`lldp`, `cdp`, `fdb`, `stp`, ...) |
142
+| `link_type` | Refinement (e.g. `lldp`, `inferred-fdb`, `stp-parent`) |
143
+| `direction` | `bidirectional`, `forward`, `reverse` |
144
+| `state` | Operational state (`up`, `down`, ...) |
145
+| `src_actor_id` / `dst_actor_id` | The two endpoints (match an entry in `actors[]`) |
146
+| `src` / `dst` | Per-end interface / port details |
147
+| `discovered_at` / `last_seen` | RFC3339 timestamps |
148
+| `metrics` | Optional per-link metrics |
149
+
150
+---
151
+
152
+## Examples
153
+
154
+### Example 1: full LLDP/CDP topology
155
+
156
+```bash
157
+TOKEN="YOUR_API_TOKEN"
158
+NODE="YOUR_NODE_UUID"
159
+
160
+read -r -d '' PAYLOAD <<'EOF'
161
+{
162
+ "selections": {
163
+ "nodes_identity": ["mac"],
164
+ "map_type": ["lldp_cdp_managed"],
165
+ "inference_strategy": ["fdb_minimum_knowledge"],
166
+ "managed_snmp_device_focus": ["all_devices"],
167
+ "depth": ["all"]
168
+ },
169
+ "timeout": 60000
170
+}
171
+EOF
172
+
173
+curl -sS -X POST \
174
+ -H 'Content-Type: application/json' \
175
+ -H "Authorization: Bearer $TOKEN" \
176
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:snmp" \
177
+ -d "$PAYLOAD" \
178
+ | jq '.data | {actors: (.actors|length), links: (.links|length)}'
179
+```
180
+
181
+### Example 2: include FDB-derived low-confidence endpoints
182
+
183
+```bash
184
+read -r -d '' PAYLOAD <<'EOF'
185
+{
186
+ "selections": {
187
+ "nodes_identity": ["mac"],
188
+ "map_type": ["all_devices_low_confidence"],
189
+ "inference_strategy": ["fdb_pairwise_minimum_knowledge"],
190
+ "managed_snmp_device_focus": ["all_devices"],
191
+ "depth": ["all"]
192
+ },
193
+ "timeout": 120000
194
+}
195
+EOF
196
+```
197
+
198
+### Example 3: focus on a single device + 2 hops
199
+
200
+```bash
201
+read -r -d '' PAYLOAD <<'EOF'
202
+{
203
+ "selections": {
204
+ "nodes_identity": ["ip"],
205
+ "map_type": ["lldp_cdp_managed"],
206
+ "inference_strategy": ["stp_parent_tree"],
207
+ "managed_snmp_device_focus": ["ip:YOUR_FOCUS_DEVICE_IP"],
208
+ "depth": ["2"]
209
+ },
210
+ "timeout": 60000
211
+}
212
+EOF
213
+```
214
+
215
+### Example 4: discover supported parameters before querying
216
+
217
+```bash
218
+TOKEN="YOUR_API_TOKEN"
219
+NODE="YOUR_NODE_UUID"
220
+
221
+read -r -d '' PAYLOAD <<'EOF'
222
+{ "info": true }
223
+EOF
224
+
225
+curl -sS -X POST \
226
+ -H 'Content-Type: application/json' \
227
+ -H "Authorization: Bearer $TOKEN" \
228
+ "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:snmp" \
229
+ -d "$PAYLOAD" \
230
+ | jq '{accepted_params, required_params}'
231
+```
232
+
233
+---
234
+
235
+## Limits and gotchas
236
+
237
+- **Topology Functions are slow.** A full SNMP sweep can take tens
238
+ of seconds. Bump `timeout` to 60-120 seconds.
239
+- **`actor_id` for L2 SNMP topology can be a comma-separated list
240
+ of MACs** -- when a single device exposes many MACs (one per
241
+ port), they collapse into one actor with all MACs in
242
+ `actor_id`. Use `nodes_identity:ip` to collapse by IP instead.
243
+- **Privacy**: actor and link records carry MAC addresses, IP
244
+ addresses, sysName strings, and SNMP descriptions. Treat as
245
+ network-identifying data; do not paste raw responses into
246
+ committed files. Use `<repo>/.local/audits/...` for working
247
+ output (gitignored).
248
+- **`info=true` is cheap and always available.** Use it to confirm
249
+ the parameter set on a specific node before constructing a real
250
+ query.
251
+- **Topology Functions are agent-only sources.** Cloud only
252
+ proxies; there is no Cloud-side aggregation across nodes for
253
+ topology. For multi-agent topology composition, fetch each
254
+ agent's response and merge client-side.
docs/netdata-ai/skills/query-netdata-cloud/verify/questions.md
new
+110
@@ -0,0 +1,110 @@
1
+# query-netdata-cloud -- verification questions (seed list)
2
+
3
+This file is the **seed input** consumed by the verification
4
+harness in SOW-0006. The harness spawns a Sonnet-class assistant
5
+with `../SKILL.md` + `../how-tos/INDEX.md` + the canonical
6
+reference docs as context, asks each question below, captures the
7
+transcript, and grades against `grader.md` (delivered by SOW-0006).
8
+
9
+When the assistant cannot answer or has to perform analysis not
10
+already documented under `../how-tos/`, the assistant must author
11
+a new how-to and add it to the index before completing.
12
+
13
+## Anchor: target node
14
+
15
+Every question is asked against the same target node:
16
+
17
+> Node `costa-desktop` -- a node visible to the user in their
18
+> Netdata Cloud (any space the cloud token can read; the
19
+> harness must locate it).
20
+
21
+Resolving `costa-desktop` from a hostname to a `node UUID` is
22
+itself the first verification (see Q01).
23
+
24
+## Identity / hardware / OS
25
+
26
+- **Q01** -- Find the node UUID and machine GUID of the node
27
+ whose hostname is `costa-desktop`. Which space and which
28
+ rooms is it in?
29
+- **Q02** -- What are the hardware specs of `costa-desktop`?
30
+ CPU architecture, core count, total RAM, total disk space.
31
+- **Q03** -- What operating system does `costa-desktop` run?
32
+ Distribution name, version, kernel version.
33
+- **Q04** -- Which cloud provider, region, and instance type is
34
+ `costa-desktop` running on (if any)?
35
+- **Q05** -- What is the agent version on `costa-desktop`, and
36
+ what is its claim_id?
37
+
38
+## Streaming / parent / child / vnodes
39
+
40
+- **Q06** -- Is `costa-desktop` acting as a parent? If so, how
41
+ many children stream to it, and what are their hostnames /
42
+ node UUIDs?
43
+- **Q07** -- Is `costa-desktop` a child? If so, which parent
44
+ does it stream to (host or endpoint)?
45
+- **Q08** -- Does `costa-desktop` have any virtual nodes
46
+ configured? Which? Are they running?
47
+
48
+## Collection / jobs / DynCfg
49
+
50
+- **Q09** -- Are there any failed data-collection jobs on
51
+ `costa-desktop`? Which?
52
+- **Q10** -- Does `costa-desktop` monitor nvidia DCGM (the
53
+ go.d.plugin `nvidia_smi` collector or equivalent)? At what
54
+ data-collection frequency (`update_every`)?
55
+- **Q11** -- List every `go.d.plugin` collector job currently
56
+ RUNNING on `costa-desktop`, sorted alphabetically.
57
+
58
+## Processes / top
59
+
60
+- **Q12** -- Which PID is currently consuming the most resident
61
+ memory on `costa-desktop`? What is its command line, and which
62
+ app-group / dashboard category does the dashboard show it
63
+ under?
64
+
65
+## Alerts
66
+
67
+- **Q13** -- What alerts are currently firing (not CLEAR) in
68
+ the room that contains `costa-desktop`? List by status
69
+ (CRITICAL / WARNING) with alert name and instance.
70
+- **Q14** -- Get the full configuration definition of one
71
+ currently-firing alert from Q13 (selectors, thresholds,
72
+ notification settings).
73
+- **Q15** -- Are there any active or scheduled silencing rules
74
+ in the space?
75
+
76
+## Logs / status file
77
+
78
+- **Q16** -- What was the last record written to the
79
+ `agent-events` journal namespace on the `agent-events` node?
80
+- **Q17** -- Find the last 5 entries with PRIORITY <= warning
81
+ (i.e. err / crit / alert / emerg) in the system journal of
82
+ `costa-desktop` over the last 24 hours.
83
+
84
+## Topology
85
+
86
+- **Q18** -- Run the L2 topology Function on `costa-desktop`
87
+ (`topology:snmp`). How many actors and links are reported,
88
+ and what discovery protocols are used (LLDP / CDP / FDB /
89
+ STP)?
90
+
91
+## Flows
92
+
93
+- **Q19** -- Are network flows being collected on
94
+ `costa-desktop`? If yes, list the top-10 source-AS / dest-AS
95
+ pairs by bytes over the last hour.
96
+
97
+## Rooms / members / feed
98
+
99
+- **Q20** -- How many rooms are in the user's space, and which
100
+ is the largest by `node_count`?
101
+- **Q21** -- List the active admins of the space.
102
+- **Q22** -- List all `node-state-stale` and `node-state-offline`
103
+ events in the last hour from the audit feed.
104
+
105
+## Self-test invariants
106
+
107
+- **Q23** -- After answering all questions above, confirm that
108
+ no NETDATA_CLOUD_TOKEN bytes, no agent-bearer UUID values,
109
+ and no claim_id values appeared in any printed output during
110
+ this session.