Network Flows: fix MDX deploy preview + capture MDX gotchas in skills (#22445)
* docs(network-flows): make MDX-safe for the learn deploy preview The Network Flows ingest produced four MDX 3 build failures on Netlify when learn rendered the new section. Three were `<placeholder>`-style patterns in metadata.yaml description fields: - aws_ip_ranges: `<aws-region>` and `<service-name>` - gcp_ip_ranges: `<service>` and `<scope>` - generic_json-over-http_ipam: `/api/<app>/subnets/` MDX 3 lexes `<word>` as a JSX open tag and demands a matching close. Inline code wrapping is the standard fix per the learn ingest escape battery (rule 4 preserves inline code). Wrapped each placeholder in backticks at the metadata.yaml source and re-ran gen_integrations.py to regenerate the three integration cards. The fourth was `<100 minutes` in retention-querying.md (`<` followed by a digit is parsed as JSX and rejected with `Unexpected character '1' before name`). Rephrased to "under 100 minutes". * skills: document MDX gotchas exposed by the netflow-plugin ingest The 2026-05-07 netflow-plugin learn ingest deploy preview surfaced three MDX 3 patterns that the existing escape battery does not cover. Capture them so the next author/agent does not repeat the round-trip: learn-site-structure/mdx-rules.md - New "Patterns that the escape battery does NOT cover" section under rule 5: `<word>` placeholders, `<` + digit, `Type<Param>` generics. - Spell out the three safe options in priority order (backticks, rephrase, backslash escape) with the real-world examples that triggered each. - Three new rows in the "What survives, what doesn't" table. learn-site-structure/pitfalls-and-gotchas.md - New entries below the existing `<= %< <->` near-variants warning, each with the literal MDX error message and the source-file/line context where it bit us. integrations-lifecycle/gotchas.md - New "metadata.yaml prose lands in MDX" entry: anything in description/setup/troubleshooting/related-resources flows through gen_integrations.py -> per-integration `.md` -> learn ingest -> MDX build, and netdata CI does NOT validate this; the next learn ingest deploy preview is what catches it. Lists the seven patterns that break, the fix for each, and a practical author-side recipe. - Cross-references the learn-site-structure entries so authors can see the consumer side too.
Costa Tsaousis committed
May 7, 2026 at 17:36 UTC
916eb8ee443d245b1c9bc7e467e22cedceadcd9f
8 files changed
+120
-7
.agents/skills/integrations-lifecycle/gotchas.md
+48
@@ -365,3 +365,51 @@ after fetching each logo and analyzing its luminance. The
365
result is cached per-URL within a single run. CI runs may
366
trip over rate limits or transient network errors; per-request
367
timeout is hardcoded.
368
+
369
+## metadata.yaml prose lands in MDX -- author it MDX-safe
370
+
371
+Anything you put in `description`, `setup`, `troubleshooting`,
372
+related-resources blurbs, or any free-text field in
373
+`metadata.yaml` flows through `gen_integrations.py` -> per-
374
+integration `.md` -> learn ingest -> MDX 3 build on Netlify.
375
+The escape battery in `learn/ingest/ingest.py:1721-1799` only
376
+handles bare `{`, the three exact-substring operators
377
+(`<=`, `%<`, `<->`), and `<details><summary>`. The following
378
+break the MDX build silently in netdata land but loudly on
379
+the next learn ingest preview deploy:
380
+
381
+| Pattern in metadata.yaml | What MDX does | Fix |
382
+|---|---|---|
383
+| `<service-name>` placeholder | parses as JSX open, demands close | wrap in backticks: `` `<service-name>` `` or `<SERVICE_NAME>` in code |
384
+| `<aws-region>`, `<scope>`, `<app>` | same | same |
385
+| `Vec<u32>`, `HashMap<K,V>` (Rust/C++/Java generics in prose) | same | wrap in backticks |
386
+| `<100 ms`, `<5 seconds`, `<10 connections` | `Unexpected character '1' before name` | rephrase as "under 100 ms" or escape with `<` |
387
+| Smart quotes (`"`, `"`, `'`, `'`) in YAML auto-converted by some editors | depends on context | use ASCII quotes |
388
+| Unbalanced single backtick on a line | breaks code-fence detection | balance or remove |
389
+
390
+Real-world hit: 2026-05-07 netflow-plugin metadata.yaml had
391
+`description: Sets tenant=amazon, region=<aws-region>, role=<service-name>.`
392
+for the AWS IP Ranges card and similar for GCP and phpIPAM.
393
+Netdata CI passed (no MDX layer there), `gen_integrations.py`
394
+generated the `.md` files cleanly, learn ingest produced the
395
+`.mdx` files cleanly, the Netlify deploy preview failed with
396
+`Expected a closing tag for \`<service-name>\` ...`. Fix
397
+landed at the metadata.yaml source by wrapping placeholders
398
+in backticks; gen_integrations.py was re-run to regenerate
399
+the integration cards.
400
+
401
+Cross-reference: `learn-site-structure/mdx-rules.md` ("Patterns
402
+that the escape battery does NOT cover") and
403
+`learn-site-structure/pitfalls-and-gotchas.md` document the
404
+MDX side; this entry is the metadata-author-side mirror.
405
+
406
+Practical recipe when authoring metadata.yaml prose:
407
+
408
+- Wrap every angle-bracketed placeholder in backticks.
409
+- Wrap every code-type expression in backticks.
410
+- Avoid `<` immediately followed by a digit; use "under" or
411
+ `<`.
412
+- Avoid generics syntax in prose; if necessary, backtick it.
413
+
414
+The integrations pipeline does not validate this on its own.
415
+The next learn ingest deploy preview is what will catch you.
.agents/skills/learn-site-structure/mdx-rules.md
+41
@@ -71,6 +71,44 @@ these as JSX tags. Note these are **exact-substring** rules;
71
near-variants like `< =` (with space) or `<---->` (multi-dash)
72
are NOT covered. See `pitfalls-and-gotchas.md`.
73
74
+### Patterns that the escape battery does NOT cover
75
+
76
+- **`<word>` placeholders in prose** (e.g. `<service-name>`,
77
+ `<scope>`, `<app>`). MDX 3 parses anything that lexically
78
+ looks like an open tag and then expects a matching close tag.
79
+ Without one, the build fails with
80
+ `Expected a closing tag for \`<word>\` ... before the end of \`paragraph\``.
81
+- **`<` followed by a digit** (e.g. `<100 minutes`,
82
+ `<5 seconds`). MDX rejects this with
83
+ `Unexpected character '1' (U+0031) before name, expected a
84
+ character that can start a name, such as a letter, $, or _`.
85
+ This pattern is common in lists describing thresholds.
86
+- **`Type<param>` Rust/C++/Java generic syntax** (e.g.
87
+ `Vec<u32>`, `HashMap<String, Vec<u8>>`, `unique_ptr<T>`).
88
+ Same JSX-tag issue.
89
+
90
+The safe options, in order of preference:
91
+
92
+1. **Wrap in inline code with backticks** -- rule 4
93
+ preserves inline code, so `\`<service-name>\``,
94
+ `\`Vec<u32>\``, `\`<APP>\`` all survive intact. This is
95
+ the standard fix for our Netdata integration content.
96
+2. **Rephrase the sentence** -- e.g. `< 100 minutes`
97
+ becomes `under 100 minutes`. Often clearer than the
98
+ original anyway.
99
+3. **Backslash-escape the `<`** -- `\<word>`. Works but
100
+ uglier than backticks. Use only when the `<` must remain
101
+ visibly a less-than operator, not a placeholder.
102
+
103
+These three were exercised in the netflow-plugin docs
104
+(2026-05-07 ingest preview deploy failure):
105
+`docs/network-flows/retention-querying.md` had `<100 minutes`,
106
+fixed by rephrasing; `aws_ip_ranges.md`, `gcp_ip_ranges.md`,
107
+and `generic_json-over-http_ipam.md` (generated from
108
+`metadata.yaml`) had `<service-name>`, `<scope>`, `<app>`,
109
+fixed by wrapping the placeholders in backticks at the
110
+`metadata.yaml` source.
111
+
112
## 6. Bare URL angle-bracket links
113
114
Converted to markdown links (`ingest.py:1797-1799`):
@@ -124,6 +162,9 @@ The escape rules cover the most common breakage patterns:
162
| `< =` (with space) | NOT covered | rule 5 is exact-substring |
163
| `<---->` (long arrow) | NOT covered | rule 5 is exact-substring |
164
| `<htmltag>` body content | NOT escaped | breaks MDX unless wrapped in code |
165
+| `<word>` placeholders in prose | NOT escaped | breaks MDX; wrap in backticks or rephrase |
166
+| `<` + digit (`<100`, `<5s`) | NOT escaped | breaks MDX; rephrase as "under N" or escape |
167
+| `Vec<u32>`, `HashMap<K,V>` generics | NOT escaped | breaks MDX; wrap in backticks |
168
| `<details>` inline summary | fixed | rule 3 |
169
| `<details class="x">` | NOT fixed | rule 3 only knows two forms |
170
| `}` closing brace alone | NOT escaped | rule 4 only escapes `{` |
.agents/skills/learn-site-structure/pitfalls-and-gotchas.md
+24
@@ -61,6 +61,30 @@ thing.
61
`sanitize_page` only as those exact substrings; near-variants
62
like `< =`, `<---->`, or `< -` aren't covered.
63
64
+- **`<word>` placeholders in prose break MDX silently
65
+ through ingest** -- patterns like `<service-name>`,
66
+ `<scope>`, `<app>`, `<aws-region>` look like JSX open
67
+ tags to MDX 3 and demand a closing tag. Build fails with
68
+ `Expected a closing tag for \`<word>\` ... before the end of
69
+ \`paragraph\``. The escape battery does NOT touch these.
70
+ Fix: wrap in backticks (`\`<service-name>\``) or rephrase.
71
+ Hit 2026-05-07 in netflow-plugin `metadata.yaml`
72
+ descriptions for AWS / GCP / phpIPAM IPAM sources.
73
+
74
+- **`<` immediately followed by a digit** -- patterns like
75
+ `<100 minutes`, `<5 seconds`, `<10ms` make MDX try to parse
76
+ a JSX tag and fail with `Unexpected character '1' (U+0031)
77
+ before name, expected a character that can start a name,
78
+ such as a letter, $, or _`. Common in threshold descriptions.
79
+ Fix: rephrase as `under 100 minutes` or `<100 minutes`.
80
+ Hit 2026-05-07 in
81
+ `docs/network-flows/retention-querying.md`.
82
+
83
+- **Generic-type syntax `Type<Param>`** -- Rust `Vec<u32>`,
84
+ C++ `unique_ptr<T>`, Java `List<String>`, etc. parse as
85
+ unbalanced JSX opens. Always wrap in backticks when
86
+ documenting code types.
87
+
88
- **`meta_yaml: "<url>"` rewrite** -- silently rewrites
89
`custom_edit_url` for any file containing `meta_yaml: "..."`,
90
even if the author didn't intend it
docs/network-flows/retention-querying.md
+1
-1
@@ -39,7 +39,7 @@ For every query the dashboard sends to the plugin, the planner makes a single de
39
2. **A non-empty full-text search → raw tier.** Full-text search runs as a regex against the raw journal payload, which only the raw tier carries.
40
3. **Otherwise, pick the coarsest tier that satisfies the time range and bucket-count requirement.**
41
- Time-Series view needs at least 100 buckets in the window. So:
42
- - <100 minutes → 1-minute tier
42
+ - under 100 minutes → 1-minute tier
43
- 100 minutes to 8h20m → 5-minute tier
44
- 8h20m and longer → 1-hour tier
45
- Table / Sankey / Maps don't have a bucket-count constraint, but the configured query-window guardrails (`query_1m_max_window` default 6h, `query_5m_max_window` default 24h) skip a tier when the window is too wide.
src/crates/netflow-plugin/integrations/aws_ip_ranges.md
+1
-1
@@ -112,7 +112,7 @@ sudo ./edit-config netflow.yaml
112
113
###### Tag all AWS prefixes by region and service
114
115
-Sets tenant=amazon, region=<aws-region>, role=<service-name>.
115
+Sets tenant=amazon, region=`aws-region`, role=`service-name`.
116
117
```yaml
118
enrichment:
src/crates/netflow-plugin/integrations/gcp_ip_ranges.md
+1
-1
@@ -104,7 +104,7 @@ sudo ./edit-config netflow.yaml
104
105
###### Tag all GCP prefixes by service and scope
106
107
-Sets tenant=gcp, role=<service>, region=<scope>.
107
+Sets tenant=gcp, role=`service`, region=`scope`.
108
109
```yaml
110
enrichment:
src/crates/netflow-plugin/integrations/generic_json-over-http_ipam.md
+1
-1
@@ -125,7 +125,7 @@ sudo ./edit-config netflow.yaml
125
126
###### phpIPAM with API token
127
128
-phpIPAM exposes /api/<app>/subnets/. Use the standard transform.
128
+phpIPAM exposes `/api/<APP>/subnets/`. Replace `<APP>` with your phpIPAM app name. Use the standard transform.
129
130
```yaml
131
enrichment:
src/crates/netflow-plugin/metadata.yaml
+3
-3
@@ -1511,7 +1511,7 @@ modules:
1511
- name: Tag all AWS prefixes by region and service
1512
folding:
1513
enabled: false
1514
- description: Sets tenant=amazon, region=<aws-region>, role=<service-name>.
1514
+ description: "Sets tenant=amazon, region=`aws-region`, role=`service-name`."
1515
config: |
1516
enrichment:
1517
network_sources:
@@ -1655,7 +1655,7 @@ modules:
1655
- name: Tag all GCP prefixes by service and scope
1656
folding:
1657
enabled: false
1658
- description: Sets tenant=gcp, role=<service>, region=<scope>.
1658
+ description: "Sets tenant=gcp, role=`service`, region=`scope`."
1659
config: |
1660
enrichment:
1661
network_sources:
@@ -2135,7 +2135,7 @@ modules:
2135
- name: phpIPAM with API token
2136
folding:
2137
enabled: false
2138
- description: phpIPAM exposes /api/<app>/subnets/. Use the standard transform.
2138
+ description: "phpIPAM exposes `/api/<APP>/subnets/`. Replace `<APP>` with your phpIPAM app name. Use the standard transform."
2139
config: |
2140
enrichment:
2141
network_sources: