| 1 | # Pitfalls and gotchas |
| 2 | |
| 3 | Every silent failure mode, dead artifact, undocumented |
| 4 | behavior, and edge case the Learn ingest pipeline carries |
| 5 | today. Read this BEFORE assuming the pipeline does the obvious |
| 6 | thing. |
| 7 | |
| 8 | ## Hard failures (exit non-zero) |
| 9 | |
| 10 | - **`map.yaml` schema validation** -- exit code 2 |
| 11 | (`ingest.py:2807, 2819`). |
| 12 | - **`--fail-links` plus broken internal links** -- exit 1 at |
| 13 | the very end of the run. |
| 14 | - **Kickstart checksum step in CI** -- exit 1 if `wget` or the |
| 15 | placeholder substitution fails. |
| 16 | |
| 17 | ## Soft failures / silent breakage |
| 18 | |
| 19 | - **Duplicate `(learn_rel_path, sidebar_label)`** -- only a |
| 20 | printed warning at `ingest.py:2891-2908`. On case-insensitive |
| 21 | filesystems (macOS/Windows), one file silently overwrites |
| 22 | the other. On Linux they coexist but collide in URL space. |
| 23 | |
| 24 | - **Missing `sidebar_label` / `learn_rel_path` in `map.yaml` |
| 25 | row** -- file is skipped with a printed `KeyError`; |
| 26 | downstream sidebar may have a hole and links that pointed to |
| 27 | it become broken anchors. |
| 28 | |
| 29 | - **`sidebar_label` containing characters that the slug |
| 30 | sanitizer collapses** (`'`, `:`, `/`, `(`, `)`, `,`, |
| 31 | backtick, repeated whitespace) -- destination filename can |
| 32 | collide with a sibling that differs only by punctuation. |
| 33 | |
| 34 | - **MDX 3 syntax bombs** -- bare `{word}` outside code is |
| 35 | escaped, but a single backtick line, smart quotes, or |
| 36 | non-paired `<...>` can still break the MDX parser. Symptoms: |
| 37 | "Unexpected character `}`", "Could not parse expression", or |
| 38 | a Docusaurus warning that becomes a blocking error in |
| 39 | `production` builds. The ingest only escapes `{`, not `}`, |
| 40 | not `<`. Bare `<word>` in body still breaks unless wrapped |
| 41 | in code. |
| 42 | |
| 43 | - **Broken anchor links** -- markdown header anchors are |
| 44 | recomputed by `extract_headers_from_file` |
| 45 | (`ingest.py:531-560`); custom anchors via `<a id="...">` may |
| 46 | not match this slugger and trigger anchor-mismatch warnings. |
| 47 | |
| 48 | - **Missing assets / images referenced by absolute path under |
| 49 | `/img`** -- Docusaurus `onBrokenLinks: 'warn'` |
| 50 | (`docusaurus.config.js:22`) and |
| 51 | `markdown.hooks.onBrokenMarkdownLinks: 'warn'` -- broken |
| 52 | links **warn but do not fail the build**, so they slip into |
| 53 | production. |
| 54 | |
| 55 | - **`<details><summary>` without newline** -- the source-side |
| 56 | fix at `ingest.py:1787-1788` only handles the two literal |
| 57 | forms it knows. Any other variant (e.g. `<details class="x">`) |
| 58 | breaks MDX rendering silently. |
| 59 | |
| 60 | - **`<= %< <->` not in code blocks** -- escaped by |
| 61 | `sanitize_page` only as those exact substrings; near-variants |
| 62 | like `< =`, `<---->`, or `< -` aren't covered. |
| 63 | |
| 64 | - **`<word>` placeholders in prose break MDX silently |
| 65 | through ingest** -- patterns like `<service-name>`, |
| 66 | `<scope>`, `<app>`, `<aws-region>` look like JSX open |
| 67 | tags to MDX 3 and demand a closing tag. Build fails with |
| 68 | `Expected a closing tag for \`<word>\` ... before the end of |
| 69 | \`paragraph\``. The escape battery does NOT touch these. |
| 70 | Fix: wrap in backticks (`\`<service-name>\``) or rephrase. |
| 71 | Hit 2026-05-07 in netflow-plugin `metadata.yaml` |
| 72 | descriptions for AWS / GCP / phpIPAM IPAM sources. |
| 73 | |
| 74 | - **`<` immediately followed by a digit** -- patterns like |
| 75 | `<100 minutes`, `<5 seconds`, `<10ms` make MDX try to parse |
| 76 | a JSX tag and fail with `Unexpected character '1' (U+0031) |
| 77 | before name, expected a character that can start a name, |
| 78 | such as a letter, $, or _`. Common in threshold descriptions. |
| 79 | Fix: rephrase as `under 100 minutes` or `<100 minutes`. |
| 80 | Hit 2026-05-07 in |
| 81 | `docs/network-flows/retention-querying.md`. |
| 82 | |
| 83 | - **Generic-type syntax `Type<Param>`** -- Rust `Vec<u32>`, |
| 84 | C++ `unique_ptr<T>`, Java `List<String>`, etc. parse as |
| 85 | unbalanced JSX opens. Always wrap in backticks when |
| 86 | documenting code types. |
| 87 | |
| 88 | - **`meta_yaml: "<url>"` rewrite** -- silently rewrites |
| 89 | `custom_edit_url` for any file containing `meta_yaml: "..."`, |
| 90 | even if the author didn't intend it |
| 91 | (`ingest.py:1801-1808`). |
| 92 | |
| 93 | - **Frontmatter `slug:` overrides** -- frontmatter `slug:` is |
| 94 | taken at face value (`ingest.py:1892`); a space in a slug |
| 95 | breaks every link to that page. Not validated. |
| 96 | |
| 97 | - **Auto-grid file overwrite risk**: |
| 98 | `get_dir_make_file_and_recurse` will NOT overwrite an |
| 99 | existing `<dir>/<dir>.mdx` (`ingest.py:2493`). However, if a |
| 100 | directory contains exactly one published integration plus |
| 101 | zero non-integrations, the script special-cases that as |
| 102 | content and skips the grid (`ingest.py:2429-2503`). |
| 103 | Adding/removing files can flip a directory between "grid" |
| 104 | and "leaf" presentations, surprising the maintainer. |
| 105 | |
| 106 | - **`safe_cleanup_learn_folders` deletes ALL `.json` files |
| 107 | unconditionally** (`ingest.py:1061-1063`) without checking |
| 108 | `part_of_learn`. Hand-authored `_category_.json` in |
| 109 | `${NETDATA_REPOS_DIR}/learn/docs/` is wiped each run unless |
| 110 | ingest itself re-creates it via |
| 111 | `ensure_category_json_for_dirs`. |
| 112 | |
| 113 | ## Dead code / stale artifacts |
| 114 | |
| 115 | ### `ingest.js` is LEGACY |
| 116 | |
| 117 | `${NETDATA_REPOS_DIR}/learn/ingest.js` is the original Node |
| 118 | orchestrator. It is no longer the live pipeline. The README, |
| 119 | the active workflow, and `<repo>/docs/.map/README.md` all run |
| 120 | `ingest/ingest.py`. **Do not edit `ingest.js`**; do not trust |
| 121 | its behavior as a description of the current pipeline. |
| 122 | |
| 123 | ### `ingest.md` is stale |
| 124 | |
| 125 | `${NETDATA_REPOS_DIR}/learn/ingest.md` documents `ingest.js`, |
| 126 | not `ingest.py`. The `README.md` describes the Python pipeline |
| 127 | (at lines 52-172). When in conflict, README + `ingest.py` win |
| 128 | over `ingest.md`. |
| 129 | |
| 130 | ### `ingest/create_grid_integration_pages.py` is empty |
| 131 | |
| 132 | `${NETDATA_REPOS_DIR}/learn/ingest/create_grid_integration_pages.py` |
| 133 | is 0 bytes. The README at lines 127-128 still tells users to |
| 134 | run it; the actual grid generation moved into |
| 135 | `ingest.py:get_dir_make_file_and_recurse` (`ingest.py:2333-2510`). |
| 136 | Running the empty script does nothing and produces no error. |
| 137 | |
| 138 | ### Duplicate link-checker |
| 139 | |
| 140 | `${NETDATA_REPOS_DIR}/learn/scripts/check_learn_links.py` |
| 141 | duplicates |
| 142 | `${NETDATA_REPOS_DIR}/learn/ingest/check_learn_links.py` |
| 143 | verbatim. Pick one as canonical; today both exist. |
| 144 | |
| 145 | ### `search-icons.js` is dead code |
| 146 | |
| 147 | `${NETDATA_REPOS_DIR}/learn/search-icons.js` is a |
| 148 | developer-only icon-picker shim that's never imported. Safe to |
| 149 | ignore. |
| 150 | |
| 151 | ### `.bak` workflows |
| 152 | |
| 153 | Three legacy workflows kept as `.bak`: |
| 154 | - `${NETDATA_REPOS_DIR}/learn/.github/workflows/old_ingest.yml.bak` |
| 155 | (Node `ingest.js` cron), |
| 156 | - `old_check-broken-links.yml.bak`, |
| 157 | - `old_check-broken-links-external.yml.bak`, |
| 158 | - `check-internal-links.yml.bak`. |
| 159 | |
| 160 | Not active. Useful as historical reference only. |
| 161 | |
| 162 | ### `produce_gh_edit_link_for_repo` typo |
| 163 | |
| 164 | `ingest.py:1027-1035`: the format string is |
| 165 | **single-quoted** (`"https://github.com/netdata/{repo}/edit/master/{file_path}"`) |
| 166 | -- the `f` prefix is missing -- so the function returns the |
| 167 | literal string with `{repo}` and `{file_path}` unsubstituted |
| 168 | for non-`.github` repos. Not currently called in the live |
| 169 | pipeline, but a real bug that would surface if the function |
| 170 | ever gets invoked. |
| 171 | |
| 172 | ## Versioning is effectively unused |
| 173 | |
| 174 | - `${NETDATA_REPOS_DIR}/learn/versioning/remove_edit_links.py` |
| 175 | is the only versioning helper. It is a manual prep step for |
| 176 | snapshotting a version: rewrite `custom_edit_url:` to |
| 177 | `null`. Not automated. |
| 178 | - No `versioned_docs/`, `versioned_sidebars/`, or |
| 179 | `versions.json` files exist in |
| 180 | `${NETDATA_REPOS_DIR}/learn/`. |
| 181 | - `package.json` has standard `docusaurus`, `start`, `build`, |
| 182 | `swizzle`, `deploy`, `clear`, `serve` scripts but no |
| 183 | version-tagging script. |
| 184 | - Conclusion: there is one live version. Freezing one would |
| 185 | require manual `versioning/remove_edit_links.py` on a |
| 186 | snapshot, then `docusaurus docs:version <name>` (no |
| 187 | automation exists for this). |
| 188 | |
| 189 | ## Netlify redirect-rule ceiling |
| 190 | |
| 191 | The dynamic redirect section keeps growing -- already ~12,700 |
| 192 | lines in `${NETDATA_REPOS_DIR}/learn/netlify.toml`. The |
| 193 | `LegacyLearnCorrelateLinksWithGHURLs.json` already has ~3,490 |
| 194 | entries. |
| 195 | |
| 196 | **Netlify's redirect-rule limit is ~10,000 rules per site.** |
| 197 | This repo is approaching/past that ceiling. Not noted anywhere |
| 198 | in the live code or docs. |
| 199 | |
| 200 | When the limit is exceeded, Netlify deploys still succeed but |
| 201 | some redirects stop working. Followup: either prune the |
| 202 | catalog (drop redirects older than N months) or move the |
| 203 | mechanism to a different layer. |
| 204 | |
| 205 | ## Inferring repo from a local `--repos /path` argument |
| 206 | |
| 207 | `ingest.py:2701-2738`: `ingest.py` first matches the basename |
| 208 | to a known repo key; if no exact match, it does a SUBSTRING |
| 209 | match in either direction (`netdata` vs `mynetdata-fork`). |
| 210 | This can pick the wrong repo silently. |
| 211 | |
| 212 | When testing locally with a fork that has an unusual name, use |
| 213 | `--local-repo netdata:/path/to/your/fork` to force the right |
| 214 | mapping. |
| 215 | |
| 216 | ## Schemas accept extras silently |
| 217 | |
| 218 | `<repo>/docs/.map/map.schema.json` sets |
| 219 | `additionalProperties: false`. Validation IS strict; unknown |
| 220 | keys fail the run. |
| 221 | |
| 222 | But the per-file metadata block format is NOT schema-validated |
| 223 | end-to-end; the script tolerates extra keys in the |
| 224 | `<!--startmeta...endmeta-->` block. Stray keys pass through to |
| 225 | the rendered frontmatter and may either be ignored by |
| 226 | Docusaurus or, in rare cases, trigger build warnings. |
| 227 | |
| 228 | ## `learn_link` is informational only |
| 229 | |
| 230 | `learn_link:` in frontmatter is NOT used by Docusaurus for |
| 231 | routing. It is checked daily by `check_learn_links.py` (HEAD |
| 232 | requests against `learn.netdata.cloud`) -- its purpose is to |
| 233 | catch the case where a slug in the source-controlled file no |
| 234 | longer matches what's actually deployed. |
| 235 | |
| 236 | ## Logo contrast analysis makes outbound HTTP calls |
| 237 | |
| 238 | `ingest.py:1647-1681` makes outbound HTTP calls to |
| 239 | `netdata.cloud/img/...` for every integration during ingest. |
| 240 | Result is cached per URL within a run. CI runs over hundreds |
| 241 | of logos; total network time is bounded by |
| 242 | `LOGO_ANALYSIS_TIMEOUT = 8 s` per request. Transient network |
| 243 | errors during ingest can cause subtle visual inconsistencies |
| 244 | (missing data attributes on logos). |
| 245 | |
| 246 | ## Sidebar position 0 reservation |
| 247 | |
| 248 | `sidebar_position: 0` is reserved for "Ask Nedi" |
| 249 | (`ingest.py:493-512`). Any other top-level entry assigned |
| 250 | position 0 gets re-stamped to >= 10. So you can't pin a |
| 251 | non-`docs/ask-nedi.mdx` page to the very top of the sidebar. |
| 252 | |
| 253 | ## Empty/unmapped directories vanish |
| 254 | |
| 255 | `ensure_category_json_for_dirs` (`ingest.py:357-360`) does NOT |
| 256 | create `_category_.json` when a directory has no `.mdx` files. |
| 257 | So if you remove all docs from a category but leave the empty |
| 258 | dir, it disappears from the sidebar (Docusaurus drops empty |
| 259 | dirs). To intentionally keep an empty section, you need at |
| 260 | least one `learn_status: Published` page in it. |
| 261 | |
| 262 | ## Slugs in source frontmatter are sticky |
| 263 | |
| 264 | Ingest writes `slug:` in the frontmatter for every published |
| 265 | file. Authors who want a stable URL across renames must set |
| 266 | `slug:` in the source `.md` (which becomes the override path |
| 267 | at `ingest.py:1890-1895`). |
| 268 | |
| 269 | But the source file's frontmatter is rewritten by ingest, so |
| 270 | `slug:` must be set BEFORE the metadata block, where it |
| 271 | survives the `<!-- ... -->` rewrite. In practice authors |
| 272 | rarely do this; they rely on `meta.label` and the redirect |
| 273 | machinery instead. |
| 274 | |
| 275 | ## Auto-redirect chain depth is unbounded |
| 276 | |
| 277 | Every ingest appends new entries to |
| 278 | `LegacyLearnCorrelateLinksWithGHURLs.json`. Old redirects are |
| 279 | re-resolved each run via `UpdateGHLinksBasedOnMap`, so a |
| 280 | multiple-times-moved page keeps working. But the catalog grows |
| 281 | indefinitely. |
| 282 | |
| 283 | ## Home page depends on `ask-nedi.mdx` |
| 284 | |
| 285 | `${NETDATA_REPOS_DIR}/learn/src/pages/index.js:1-6` redirects |
| 286 | `/` -> `/docs/ask-nedi`. Anyone removing or renaming |
| 287 | `ask-nedi.mdx` (which is `part_of_learn: True` and otherwise |
| 288 | survives ingest) breaks the site root. |
| 289 | |
| 290 | ## Search is local, not Algolia |
| 291 | |
| 292 | `docusaurus.config.js:34-39` uses |
| 293 | `@easyops-cn/docusaurus-search-local` -- client-side, hashed |
| 294 | indexes built at `yarn build` time. There is NO Algolia |
| 295 | DocSearch integration. Search results are not as polished, and |
| 296 | indexing is build-time (changes propagate only on next deploy). |
| 297 | |
| 298 | ## Posthog, GTM, gtag, Reo.dev, Nedi UI |
| 299 | |
| 300 | `docusaurus.config.js:179-264` loads Posthog, Google Tag |
| 301 | Manager, Google gtag, Reo.dev, and the Nedi UI as |
| 302 | plugins/scripts. The Nedi embed depends on |
| 303 | `nedi.netdata.cloud/ai-agent-public.js`; if that origin is |
| 304 | down, the chat widget fails silently but the rest of the site |
| 305 | loads. |
| 306 | |
| 307 | ## Custom anchor IDs |
| 308 | |
| 309 | `ingest.py:531-560` extracts header anchors via a slugger. |
| 310 | Custom anchors via `<a id="..."> </a>` may not match the |
| 311 | slugger and trigger anchor-mismatch warnings during link |
| 312 | resolution (step 12 of the pipeline). Use `## Heading` text |
| 313 | that produces the desired slug instead of injecting custom |
| 314 | anchors. |
| 315 | |
| 316 | ## OpenAPI / Swagger |
| 317 | |
| 318 | `${NETDATA_REPOS_DIR}/learn/static/api/` has Swagger UI files. |
| 319 | Not part of the docs/ ingest tree. Updated separately when the |
| 320 | agent's OpenAPI surface changes. |
| 321 | |
| 322 | ## Forks / non-`netdata/netdata` sources |
| 323 | |
| 324 | The pipeline assumes `AGENT_REPO = 'netdata/netdata'` |
| 325 | everywhere (`ingest.py:15`). Running `ingest.py` against a |
| 326 | fork (e.g. `--repos ktsaou/netdata:master`) works, but |
| 327 | `build_path` (`gen_docs_integrations.py:81-90`) assumes |
| 328 | `meta_yaml` URLs start with `https://github.com/netdata/...`, |
| 329 | so integration page URL rewriting may break. For local |
| 330 | testing use `--local-repo netdata:/path/to/your/fork` so the |
| 331 | edit URLs still reference `netdata/netdata` upstream. |