master
md 488 lines 19 KB
Rendered Raw
1 # Gotchas
2
3 Every surprise, dead-code reference, hardcoded marketing
4 anchor, custom Jinja delimiter, undocumented behavior, and
5 edge case the integrations pipeline carries today. Read this
6 before assuming the code does the obvious thing.
7
8 ## Taxonomy authoring gotchas
9
10 ### `grid.items` and `view_switch` branches do not accept string shorthand
11
12 - Wrong shape: `type: grid` with `items: ["mysql.queries"]`.
13 - Failure: schema validation rejects the string because grid children
14 are display-only objects.
15 - Correct shape: use `type: context` with `contexts:` and
16 `chart_library:` inside the grid; own the context elsewhere in a
17 structural item.
18
19 ### Dynamic selectors need explicit metadata opt-in
20
21 - Wrong shape: `context_prefix: [snmp.device_prof_]` without
22 `metrics.dynamic_context_prefixes:` in the sibling `metadata.yaml`.
23 - Failure: TAX031 fatal.
24 - Correct shape: declare the safe namespace in metadata, for example
25 `dynamic_context_prefixes: [{prefix: snmp., reason: ...}]`.
26
27 ### Taxonomy fields use snake_case, not legacy FE camelCase
28
29 - Wrong shape: `chartLibrary`, `groupByLabel`, `tableSortBy`.
30 - Failure: closed-schema `additionalProperties` rejection.
31 - Correct shape: `chart_library`, `group_by_label`, `table_sort_by`.
32
33 ### Structural containers need stable `id:` values
34
35 - Wrong shape: `type: group` with only `title:` and `items:`.
36 - Failure: schema validation rejects the missing `id`.
37 - Correct shape: choose a stable kebab-case `id` that does not change
38 when the display `title` is renamed.
39
40 ### `single_node:` is a sparse override, `view_switch` is replacement
41
42 - Wrong shape: putting `multi_node:` next to ordinary placement or item
43 fields.
44 - Failure: TAX022 fatal unless `multi_node` is inside
45 `type: view_switch`.
46 - Correct shape: use `single_node:` only for small same-kind field
47 deltas; use `type: view_switch` when the whole body differs.
48
49 ### `unresolved:` is only for staged literal widget references
50
51 - Wrong shape: a bare unknown literal context in widget `contexts:`.
52 - Failure: TAX003 fatal.
53 - Correct shape: either own/reference a real metadata context, use a
54 selector object, or use `{context, unresolved: {reason, owner,
55 expires}}` when the missing context is an intentional staged rollout.
56 `expires` must be `YYYY-MM-DD`.
57
58 ### Removed structural-only shapes stay rejected
59
60 - Wrong shape: top-level or placement `contexts:` / `subsections:`, or
61 old list-merge fields ending in `_extend`.
62 - Failure: TAX001/TAX023 fatal.
63 - Correct shape: use recursive `items:` with the v1 item kinds.
64
65 ## Dead / broken code in the pipeline
66
67 ### Metadata links may be `blob/master` or `edit/master`
68
69 - File path: `integrations/gen_docs_integrations.py`.
70 - `gen_integrations.py` can emit metadata links in GitHub
71 `blob/master` form, while older docs-generation code only
72 stripped `edit/master`.
73 - If `build_path()` does not normalize both forms, scoped
74 generation such as
75 `python3 integrations/gen_docs_integrations.py -c go.d.plugin/vsphere`
76 finds the collector in `integrations.js` but writes nothing
77 because the derived local path does not exist.
78 - Current contract: `build_path()` must strip both
79 `blob/master/` and `edit/master/` before removing
80 `/metadata.yaml`.
81
82 ### `integrations/check_collector_metadata.py` is broken
83
84 - File path: `integrations/check_collector_metadata.py`.
85 - Line 8 imports `SINGLE_PATTERN`, `MULTI_PATTERN`,
86 `SINGLE_VALIDATOR`, `MULTI_VALIDATOR` from `gen_integrations`.
87 - **None of those names exist in `gen_integrations.py`
88 today.** The current names are `METADATA_PATTERN` (single
89 pattern) and `COLLECTOR_VALIDATOR` (single validator).
90 - Therefore: any attempt to run
91 `python3 integrations/check_collector_metadata.py <path>`
92 exits with `ImportError`.
93 - It is referenced nowhere actionable: no workflow under
94 `.github/workflows/`, no script under `packaging/cmake/`.
95 Only mentioned in `integrations/README.md` style prose.
96 - Its body has a second bug: `'{ check_path } is a valid
97 collector metadata file.'` (line 84) -- the f-string `f`
98 prefix is missing, so the literal `{ check_path }` would be
99 printed even if the imports worked.
100 - **Treat this file as dead code.** Do NOT rely on it.
101 Follow-up must be tracked by a GitHub issue before implementation starts.
102
103 ### `gen_doc_service_discovery_page.py` is NOT in CI
104
105 - File exists, runs correctly when invoked manually.
106 - Workflow `.github/workflows/generate-integrations.yml`
107 runs only `gen_integrations.py`, `gen_docs_integrations.py`,
108 `gen_doc_collector_page.py`, `gen_doc_secrets_page.py`
109 (`generate-integrations.yml:48-63`).
110 - Workflow `.github/workflows/check-markdown.yml` has the
111 same gap.
112 - Consequence: `src/collectors/SERVICE-DISCOVERY.md` drifts
113 from source `metadata.yaml` until a developer manually
114 runs `python3 integrations/gen_doc_service_discovery_page.py`.
115 - Follow-up must be tracked by a GitHub issue before implementation starts.
116
117 ### `integrations/schemas/distros.json` is unused
118
119 - Schema declared and well-formed.
120 - `gen_integrations.py:1330` calls `load_yaml(DISTROS_FILE)`
121 WITHOUT validation.
122 - `DEPLOY_VALIDATOR` is for `deploy.yaml` only, NOT for
123 `distros.yml`.
124 - Garbage in `.github/data/distros.yml` produces broken
125 `platform_info` tables silently.
126 - Follow-up must be tracked by a GitHub issue before implementation starts.
127
128 ## Custom Jinja delimiters
129
130 `gen_integrations.py:233-238` configures Jinja with custom
131 delimiters:
132
133 | Default | This pipeline |
134 |---|---|
135 | `{{ ... }}` | `[[ ... ]]` |
136 | `{% ... %}` | `[% ... %]` |
137 | `{# ... #}` | `[# ... #]` |
138
139 Why: so that templates can pass `{% details %}`,
140 `{% relatedResource %}`, `{% if $showClaimingOptions %}`,
141 `{{ ... }}` markers through to the rendered output verbatim
142 (those markers are the cloud-frontend's renderer's syntax,
143 not Jinja's).
144
145 Documented in `integrations/templates/README.md:12-15` (which
146 itself is partly stale -- see below).
147
148 ## Two-pass templating with `meta.variables`
149
150 `gen_integrations.py:930-934`. When a metadata entry declares
151 `meta.monitored_instance.variables` (collectors) OR
152 `meta.variables` (other types), the FIRST pass renders the
153 section; if the rendered output still contains
154 `[[ variables.foo ]]` markers, a SECOND Jinja pass is run
155 with `variables=...` in context.
156
157 **Divergence**: collectors look up
158 `monitored_instance.variables`; exporters and notifications
159 look up `meta.variables`. Same goal, different lookup path.
160 A wart.
161
162 ## `{% relatedResource %}` two-pass resolution
163
164 Pass 1: templates emit literal
165 `{% relatedResource id="..." %}name{% /relatedResource %}`
166 markers. See `integrations/templates/overview/collector.md:42`
167 and `integrations/templates/related_resources.md:5`.
168
169 Pass 2: `gen_docs_integrations.py:resolve_related_links`
170 (`:56-78`) runs AFTER all per-integration `.md` files are
171 written. It replaces the markers with `[name](/path)` markdown
172 links using a global `id_to_path` map built from the
173 just-written files.
174
175 **Silent fallback**: if the marker's `id` is not found in the
176 map, the marker is replaced with bare `name` text (no link).
177 No warning.
178
179 ## `clean=False` vs `clean=True` divergence
180
181 Every render keys section is rendered TWICE
182 (`gen_integrations.py:805-947` for collectors, similar for
183 others):
184
185 - `clean=False` -- preserves `{% details %}` markers; goes
186 into `integrations.js` for the cloud-frontend dashboard.
187 - `clean=True` -- strips folding/details markers; goes into
188 `integrations.json` AND into the per-integration `.md`
189 files via `gen_docs_integrations.py:50-52`.
190
191 Consumers:
192
193 - Cloud-frontend reads `.js` -> rich variant with markers.
194 - Per-integration `.md` files (committed, viewed on Learn /
195 GitHub) -- clean variant.
196 - `gen_doc_collector_page.py`, `gen_doc_secrets_page.py`,
197 `gen_doc_service_discovery_page.py`,
198 `gen_docs_integrations.py` all parse `integrations.js`,
199 NOT `integrations.json` -- so they see the rich variant
200 but emit the clean variant downstream.
201
202 ## Slug rules diverge by integration type
203
204 - Most types: slug = `clean_string(meta.name)` ->
205 `<plugin-dir>/integrations/<slug>.md`.
206 - **Secretstore: slug = `clean_string(meta.kind)`**
207 (`gen_docs_integrations.py:640`). The kind matches the
208 runtime config filename `/etc/netdata/go.d/ss/<kind>.conf`.
209 - **Service-discovery: slug = `clean_string(meta.kind)`**
210 (`gen_docs_integrations.py:655`). Same reason -- the kind
211 is the discoverer registry name.
212 - **Collector custom_edit_url under `/integrations/functions/`**:
213 filename uses the function slug from the URL stem (with `-`
214 -> ` `) instead of `monitored_instance.name`. Avoids
215 collisions when many integrations share a "Top Queries"
216 label.
217
218 `clean_string` rules (`gen_docs_integrations.py:118-126`):
219 1. lowercase;
220 2. spaces -> `_`;
221 3. `/` -> `_`;
222 4. drop `(`, `)`, `,`, `'`, backtick, `:`.
223
224 So `Apache Kafka` -> `apache_kafka`. `Citrix/NetScaler` ->
225 `citrix_netscaler`. Note that these characters are stripped
226 silently; if two source names collide post-cleanup, one
227 overwrites the other (not warned about).
228
229 ## `make_id` allows uppercase
230
231 `gen_integrations.py:768`:
232 `monitored_instance.name.replace(' ', '_')`.
233
234 So `Apache Kafka` becomes id segment `Apache_Kafka` and full
235 id `go.d.plugin-kafka-Apache_Kafka`. Mixed case preserved;
236 only spaces translated. Not URL-safe in the strict sense
237 (uppercase).
238
239 ## Schemas are NOT strict
240
241 `additionalProperties: false` is NOT set on most schemas. Two
242 known undocumented fields that pass through silently:
243
244 - `alternative_monitored_instances` -- seen in
245 `src/go/plugin/go.d/collector/postgres/metadata.yaml:21`.
246 - `most_popular` -- seen in ibm.d-generated `metadata.yaml`
247 and elsewhere.
248
249 Neither is in `collector.json`. They appear in
250 `integrations.js` but no template renders them. They are
251 harmless but misleading -- maintainers may assume they do
252 something.
253
254 ## `global` scope renamed to `<instance> instance`
255
256 `gen_integrations.py:914-916`. Many `metadata.yaml` files
257 declare `metrics.scopes:` with `name: global`. The renderer
258 rewrites this in-place to `<monitored_instance.name> instance`
259 before templating. The original file stays as `global`.
260
261 ## Default-categories fallback
262
263 `gen_integrations.py:906-908`. If a collector's declared
264 categories are all bogus (none match `categories.yaml`), the
265 renderer falls back to ALL categories with
266 `collector_default: true` from `categories.yaml`. Currently
267 only `data-collection.applications` (`categories.yaml:50-52`)
268 is so flagged. So a typo in a collector's categories silently
269 parks the integration under "Applications".
270
271 ## `agent_notification` writes README directly
272
273 For every type EXCEPT `agent_notification`, the per-integration
274 `.md` lives under `<dir>/integrations/<slug>.md` and a
275 `README.md` symlink is made when there is exactly one
276 integration.
277
278 For `agent_notification`, the script writes the per-integration
279 file DIRECTLY to `<dir>/README.md` (`gen_docs_integrations.py:488-496`).
280 No `integrations/` subdirectory, no symlink. So
281 `src/health/notifications/email/README.md` is the generated
282 artifact, NOT a hand-written README. The
283 `<!--startmeta` banner is the giveaway.
284
285 ## Symlink only fires when exactly one integration
286
287 `gen_docs_integrations.py:466`:
288 `len(list(integrations_dir.iterdir())) == 1`. If a directory
289 has multiple integrations (rare), no top-level `README.md`
290 symlink is created -- the parent's existing README is left
291 alone (or absent).
292
293 ## Hardcoded marketing anchors that don't exist
294
295 `gen_doc_collector_page.py:_render_tech_navigation` (`:424-493`)
296 writes navigation links like:
297 `#cloud-provider-managed`, `#kubernetes`, `#search-engines`,
298 `#freebsd`, `#message-brokers`, `#linux-systems`,
299 `#windows-systems`, `#macos-systems`.
300
301 None of those category IDs exist in `categories.yaml`. The
302 real categories are `databases`, `web-servers-and-proxies`,
303 `containers-and-vms`, `operating-systems`, `networking`,
304 `cloud-and-devops`, `hardware-and-sensors`, `applications`,
305 `storage`, `synthetic-testing`. Several of the hardcoded
306 anchors point at non-existent sections in `COLLECTORS.md`.
307
308 The header text "850+ integrations" is also a baked literal.
309
310 ## Static prose baked into Python scripts
311
312 `gen_doc_secrets_page.py:20-203`: the `SECRETS_PAGE` dict
313 contains the bulk of `SECRETS.md`. Only the "Supported
314 Secretstore Backends" table is dynamic.
315
316 `gen_doc_service_discovery_page.py:21-257`: the `SD_PAGE` dict
317 contains the bulk of `SERVICE-DISCOVERY.md`. Only the
318 discoverer table is dynamic.
319
320 `gen_doc_collector_page.py`: the marketing header in
321 `_render_tech_navigation` is hardcoded.
322
323 To change the static prose on any of these umbrella pages,
324 edit the Python script and commit.
325
326 ## Scoped regen via `-c plugin/module`
327
328 `gen_docs_integrations.py:578-583`. Allows scoped cleanup +
329 regen:
330
331 ```bash
332 python3 integrations/gen_docs_integrations.py -c go.d.plugin/snmp
333 ```
334
335 NOT used by CI (CI always does full regen). Useful for fast
336 local iteration.
337
338 ## `templates/README.md` is partly stale
339
340 `integrations/templates/README.md:30-31` mentions
341 `setup-generic.md`, `setup-logs.md`, `setup-secretstore.md` as
342 the per-type setup templates, but `setup-service_discovery.md`
343 was added later and is not mentioned. Not pipeline-impacting,
344 just out-of-date docs.
345
346 ## Umbrella pages have NO DO-NOT-EDIT banner
347
348 `src/collectors/COLLECTORS.md`, `src/collectors/SECRETS.md`,
349 `src/collectors/SERVICE-DISCOVERY.md` have no generated-file
350 warning. `COLLECTORS.md` opens with
351 `<!-- markdownlint-disable-file -->` and then the marketing
352 header (`# Monitor anything with Netdata`); the other two open
353 with their marketing headers. None has a `<!--startmeta` block,
354 none has any DO-NOT-EDIT comment.
355
356 A maintainer who edits these files directly will have their
357 edits silently overwritten on the next CI run (for
358 COLLECTORS.md and SECRETS.md). For SERVICE-DISCOVERY.md the
359 absence of CI wiring means manual edits stick until someone
360 runs the script -- giving a false sense that hand-editing is
361 acceptable.
362
363 ## Edge case in `build_path`
364
365 `gen_docs_integrations.py:81-90` assumes `meta_yaml` URL
366 starts with `https://github.com/netdata/...`. For forks /
367 non-`netdata/netdata` sources, it would produce wrong paths.
368 The pipeline assumes `AGENT_REPO = 'netdata/netdata'`
369 everywhere (`gen_integrations.py:15`).
370
371 ## `convert_local_links` rewrites all `](/...)` links
372
373 `gen_integrations.py` runs `convert_local_links` on the
374 `integrations.js` output, rewriting any `](/...)` link in
375 rendered text to absolute
376 `https://github.com/netdata/netdata/blob/master/...`. This
377 applies to body links inside per-integration content. So
378 metadata authors writing `](/src/foo/bar.md)` get a GitHub
379 link in the dashboard, not a local-relative link.
380
381 ## `dependencies` in `collector.json:61-63`
382
383 Draft-7 JSON Schema `dependencies` keyword: when
384 `monitored_instance_name` is set on a
385 `related_resources.integrations.list[]` entry,
386 `module_name` becomes required. Correct semantics, but
387 non-obvious -- nothing else in the schemas uses
388 `dependencies`, and no commentary explains it.
389
390 ## `fail_on_warnings` makes ALL warnings fatal
391
392 `gen_integrations.py:150-160`. Any single validation warning
393 -- duplicate id, invalid category, missing related
394 integration -- causes `fail_on_warnings()` to return 1, which
395 fails CI. Even cosmetic issues block the regeneration PR.
396
397 Warnings are deduplicated by file path; the failure message
398 lists each warned file.
399
400 ## Cloud-notifications and authentication metadata are single-file arrays
401
402 Most types have one `metadata.yaml` per integration directory.
403 Two exceptions:
404
405 - `integrations/cloud-notifications/metadata.yaml` -- ONE
406 file containing an ARRAY of cloud-notification entries.
407 - `integrations/cloud-authentication/metadata.yaml` -- ONE
408 file containing an ARRAY of authentication-method entries.
409
410 The `_load_*_file` functions handle both shapes via
411 `if 'id' in data` branches (single entry vs array).
412
413 ## ibm.d websphere subdirectories
414
415 `gen_integrations.py:35` adds
416 `src/go/plugin/ibm.d/modules/websphere` to `COLLECTOR_SOURCES`
417 separately. That's because `websphere/{jmx,mp,pmi}/` are
418 sub-modules each with their own `metadata.yaml`,
419 `module.yaml`, `contexts/`, etc. The default
420 `src/go/plugin/ibm.d/modules` glob would not catch them at
421 the right depth.
422
423 ## `pip.sh` and the cmake module must stay in sync
424
425 `integrations/pip.sh` is a 2-line script:
426 `pip install jsonschema referencing jinja2 ruamel.yaml`. The
427 same four packages are listed at
428 `packaging/cmake/Modules/NetdataRenderDocs.cmake:21`. Both
429 must be updated together if the dep set changes (commented
430 inline in `pip.sh`).
431
432 ## Logo contrast analysis makes outbound HTTP calls
433
434 `gen_integrations.py:1647-1681` annotates `<img src="https://(www\.)?netdata\.cloud/img/...">`
435 tags with `data-integration-logo`, `data-logo-contrast-light`,
436 `data-logo-contrast-dark`, `data-logo-contrast-confidence`
437 after fetching each logo and analyzing its luminance. The
438 result is cached per-URL within a single run. CI runs may
439 trip over rate limits or transient network errors; per-request
440 timeout is hardcoded.
441
442 ## metadata.yaml prose lands in MDX -- author it MDX-safe
443
444 Anything you put in `description`, `setup`, `troubleshooting`,
445 related-resources blurbs, or any free-text field in
446 `metadata.yaml` flows through `gen_integrations.py` -> per-
447 integration `.md` -> learn ingest -> MDX 3 build on Netlify.
448 The escape battery in `learn/ingest/ingest.py:1721-1799` only
449 handles bare `{`, the three exact-substring operators
450 (`<=`, `%<`, `<->`), and `<details><summary>`. The following
451 break the MDX build silently in netdata land but loudly on
452 the next learn ingest preview deploy:
453
454 | Pattern in metadata.yaml | What MDX does | Fix |
455 |---|---|---|
456 | `<service-name>` placeholder | parses as JSX open, demands close | wrap in backticks: `` `<service-name>` `` or `<SERVICE_NAME>` in code |
457 | `<aws-region>`, `<scope>`, `<app>` | same | same |
458 | `Vec<u32>`, `HashMap<K,V>` (Rust/C++/Java generics in prose) | same | wrap in backticks |
459 | `<100 ms`, `<5 seconds`, `<10 connections` | `Unexpected character '1' before name` | rephrase as "under 100 ms" or escape with `&lt;` |
460 | Smart quotes (`"`, `"`, `'`, `'`) in YAML auto-converted by some editors | depends on context | use ASCII quotes |
461 | Unbalanced single backtick on a line | breaks code-fence detection | balance or remove |
462
463 Real-world hit: 2026-05-07 netflow-plugin metadata.yaml had
464 `description: Sets tenant=amazon, region=<aws-region>, role=<service-name>.`
465 for the AWS IP Ranges card and similar for GCP and phpIPAM.
466 Netdata CI passed (no MDX layer there), `gen_integrations.py`
467 generated the `.md` files cleanly, learn ingest produced the
468 `.mdx` files cleanly, the Netlify deploy preview failed with
469 `Expected a closing tag for \`<service-name>\` ...`. Fix
470 landed at the metadata.yaml source by wrapping placeholders
471 in backticks; gen_integrations.py was re-run to regenerate
472 the integration cards.
473
474 Cross-reference: `learn-site-structure/mdx-rules.md` ("Patterns
475 that the escape battery does NOT cover") and
476 `learn-site-structure/pitfalls-and-gotchas.md` document the
477 MDX side; this entry is the metadata-author-side mirror.
478
479 Practical recipe when authoring metadata.yaml prose:
480
481 - Wrap every angle-bracketed placeholder in backticks.
482 - Wrap every code-type expression in backticks.
483 - Avoid `<` immediately followed by a digit; use "under" or
484 `&lt;`.
485 - Avoid generics syntax in prose; if necessary, backtick it.
486
487 The integrations pipeline does not validate this on its own.
488 The next learn ingest deploy preview is what will catch you.