| 1 | # Mapping: source -> Learn URL |
| 2 | |
| 3 | The single most important fact to internalize about Learn: |
| 4 | **source filesystem path is irrelevant for routing**. A page's |
| 5 | URL on `learn.netdata.cloud` is computed from frontmatter that |
| 6 | `ingest.py` INJECTS from `<repo>/docs/.map/map.yaml`. Without |
| 7 | this mental model, every other rule seems arbitrary. |
| 8 | |
| 9 | ## The map.yaml source of truth |
| 10 | |
| 11 | `<repo>/docs/.map/map.yaml` is the canonical declaration of |
| 12 | which files are published on Learn and where they appear. It is |
| 13 | a hierarchical tree of nodes; each leaf node represents one |
| 14 | published page. |
| 15 | |
| 16 | Schema: `<repo>/docs/.map/map.schema.json`. Authoring guide: |
| 17 | `<repo>/docs/.map/README.md`. |
| 18 | |
| 19 | ### Per-node fields |
| 20 | |
| 21 | Each `meta` block declares one publication node: |
| 22 | |
| 23 | | Field | Type | Required | Notes | |
| 24 | |---|---|---|---| |
| 25 | | `meta.label` | string | yes | Display name. Becomes `sidebar_label` AND drives the destination filename. | |
| 26 | | `meta.edit_url` | string | yes (for doc-emitting nodes) | Must match `^https://github\.com/netdata/<repo>/edit/<branch>/.+\.(md\|mdx)$`. The unique key ingest uses to look up source files. Optional ONLY for category-only nodes that contain integration placeholders. | |
| 27 | | `meta.path` | string | no | Single-segment override of the URL path component if it should differ from the label. | |
| 28 | | `meta.description` | string | no | Frontmatter description; powers the page's meta description. | |
| 29 | | `meta.keywords` | array<string> | no | Frontmatter keywords; powers in-app search and Learn page metadata. | |
| 30 | |
| 31 | The schema sets `additionalProperties: false`. Unknown keys |
| 32 | fail validation and abort the run with exit code 2 |
| 33 | (`ingest/ingest.py:2815-2819`). |
| 34 | |
| 35 | ### Branches |
| 36 | |
| 37 | For all repos in the source list (see `pipeline.md`), the |
| 38 | `edit_url` branch is `master` -- with one exception: the |
| 39 | `netdata/.github` repo uses `main` branch instead of `master` |
| 40 | (`ingest.py:1232-1235`). |
| 41 | |
| 42 | ### Integration placeholders |
| 43 | |
| 44 | `integration_placeholder: true` rows in `map.yaml` are |
| 45 | substituted at ingest time by integration pages |
| 46 | auto-discovered from the source repos. The `populate_integrations` |
| 47 | step (`ingest.py:752-1003`) finds every `.md` carrying the |
| 48 | `DO NOT EDIT THIS FILE DIRECTLY` marker |
| 49 | (`INTEGRATION_MARKER`, `ingest.py:113`), parses its hidden |
| 50 | `<!--startmeta...endmeta-->` block, buckets by category |
| 51 | (collectors / exporters / secretstore / functions / |
| 52 | authentication / cloud-notifications / agent-notifications / |
| 53 | logs), sorts by `(learn_rel_path, sidebar_label)`, and |
| 54 | **inserts the rows in place of** the matching placeholder. |
| 55 | Result is written to |
| 56 | `${NETDATA_REPOS_DIR}/learn/ingest/generated_map.yaml`. |
| 57 | |
| 58 | So integration pages do NOT need explicit `map.yaml` rows; |
| 59 | they are pulled in via the placeholder mechanism. |
| 60 | |
| 61 | ## Frontmatter that ingest INJECTS |
| 62 | |
| 63 | When ingest matches a source file's `custom_edit_url` to a |
| 64 | `map.yaml` row, it writes a hidden HTML-comment metadata block |
| 65 | at the top of the source file. After `sanitize_page` runs, the |
| 66 | comment delimiters become real YAML frontmatter (`<!--` -> |
| 67 | `---`, `-->` -> `---`). |
| 68 | |
| 69 | Per `ingest.py:1221-1299` plus `ingest.py:1832-1897`: |
| 70 | |
| 71 | | Field | Source | Notes | |
| 72 | |---|---|---| |
| 73 | | `custom_edit_url` | `meta.edit_url` from `map.yaml` | Used by the "Edit this page" link. For autogenerated grid pages, this is `null`. | |
| 74 | | `sidebar_label` | `meta.label` | Drives sidebar AND destination filename. | |
| 75 | | `learn_status` | `"Published"` or `"AUTOGENERATED"` | Anything other than `Published` is excluded from `to_publish`. | |
| 76 | | `learn_rel_path` | reconstructed from `map.yaml` hierarchy | Drives the destination directory tree. `root` means write to `docs/` directly. | |
| 77 | | `keywords` | `meta.keywords` | Normalized to inline YAML array. CSV strings, brackets, and Python lists all accepted. | |
| 78 | | `description` | `meta.description` | Skipped when empty/None to minimize diffs. | |
| 79 | | `sidebar_position` | computed from map traversal order | Always written as quoted string `sidebar_position: "10"`. | |
| 80 | | `slug` | computed (`/<learn_rel_path>/<label>` lowercased) | Author-supplied `slug:` in source is preserved as override (`ingest.py:1890-1895`). | |
| 81 | | `learn_link` | `https://learn.netdata.cloud/docs<slug>` | Updated each ingest. Validated daily by `check_learn_links.py`. | |
| 82 | |
| 83 | Special case: paths containing `Collecting Metrics` get |
| 84 | `toc_max_heading_level: 6` and `toc_collapsible: true` injected |
| 85 | (`ingest.py:1251-1253`). |
| 86 | |
| 87 | ## Source-path-to-URL computation |
| 88 | |
| 89 | For files marked `learn_status: Published`, the destination is |
| 90 | computed by `create_mdx_path_from_metadata` |
| 91 | (`ingest.py:1140-1204`): |
| 92 | |
| 93 | ``` |
| 94 | docs/<learn_rel_path>/<sanitized_sidebar_label>.mdx |
| 95 | ``` |
| 96 | |
| 97 | ### Sanitization rules |
| 98 | |
| 99 | `sidebar_label` is sanitized for the filename (`ingest.py:1159-1175`): |
| 100 | 1. Strip `'`, `:`, `/`, `(`, `)`, `,`, backtick. |
| 101 | 2. Collapse repeated whitespace to single space. |
| 102 | |
| 103 | For the URL **slug** (different from filename), additionally: |
| 104 | - Lowercase everything. |
| 105 | - Spaces -> `-`. |
| 106 | - `//` -> `/`. |
| 107 | |
| 108 | Filename keeps original case; slug is lowercase. So |
| 109 | `docs/Welcome to Netdata/Welcome to Netdata.mdx` exists with |
| 110 | spaces and capitals. |
| 111 | |
| 112 | ### Special-cases |
| 113 | |
| 114 | 1. **Functions integrations** -- if the source `custom_edit_url` |
| 115 | is under `/integrations/functions/`, the filename uses the |
| 116 | function slug from the URL stem (with `-` -> ` `) instead of |
| 117 | `sidebar_label` (`ingest.py:1153-1157`). Avoids collisions |
| 118 | when many integrations share a "Top Queries" label. |
| 119 | |
| 120 | 2. **Category overview pages** -- if the last two segments of |
| 121 | the slug are equal (e.g. `/collecting-metrics/collecting-metrics`), |
| 122 | `create_mdx_path_from_metadata` returns the slug with the |
| 123 | duplicate trimmed (`ingest.py:1178-1191`). On disk the file |
| 124 | is still written as `docs/<X>/<X>.mdx`; Docusaurus then |
| 125 | routes it to the parent URL `/<X>` because it has the same |
| 126 | slug. |
| 127 | |
| 128 | 3. **Frontmatter `slug:` override** -- if the source file |
| 129 | already has `slug:` declared in frontmatter |
| 130 | (custom-authored, like `docs/ask-nedi.mdx`), that value wins |
| 131 | (`ingest.py:1890-1895`). For these files, the |
| 132 | author-supplied slug must be set BEFORE the |
| 133 | `<!-- ... -->` metadata block so it survives the rewrite. |
| 134 | |
| 135 | 4. **`id:` in mapDict** -- if the metadata in `mapDict` carries |
| 136 | an `id`, the last segment of the URL is replaced by the id |
| 137 | (`ingest.py:2120-2126`). |
| 138 | |
| 139 | ### Edge cases |
| 140 | |
| 141 | - `README.md`, `index.md`, `_index.md`, `index.mdx`, hidden |
| 142 | directories: NONE of these are special-cased in `ingest.py`. |
| 143 | The Python pipeline never publishes a file unless it appears |
| 144 | in `map.yaml` (or comes from an integration placeholder), so |
| 145 | filename has no routing effect. |
| 146 | - Files at repo root vs deep paths: same rule -- destination is |
| 147 | driven by `map.yaml.meta.label` + the node's position in the |
| 148 | hierarchy, not by source path depth. |
| 149 | - Special filenames: ignored. Only `map.yaml` matters. |
| 150 | |
| 151 | ## Examples |
| 152 | |
| 153 | ### Add a new top-level page |
| 154 | |
| 155 | 1. Create `<repo>/docs/getting-started-netdata/quick-tour.md`. |
| 156 | The exact path doesn't matter for routing -- use a |
| 157 | reasonable location. |
| 158 | 2. Add to `<repo>/docs/.map/map.yaml`: |
| 159 | ```yaml |
| 160 | - meta: |
| 161 | label: Quick Tour |
| 162 | edit_url: https://github.com/netdata/netdata/edit/master/docs/getting-started-netdata/quick-tour.md |
| 163 | description: A 5-minute tour of Netdata. |
| 164 | keywords: [tour, getting-started, intro] |
| 165 | ``` |
| 166 | under the appropriate parent. |
| 167 | 3. Resulting Learn URL: `https://learn.netdata.cloud/docs/<parent-slug>/quick-tour`. |
| 168 | 4. Resulting filename in learn repo: `docs/<parent>/Quick Tour.mdx` (with spaces and capitals). |
| 169 | |
| 170 | ### Move a page (different sidebar location, same source file) |
| 171 | |
| 172 | 1. In `map.yaml`, move the node to its new parent. |
| 173 | 2. Keep `meta.edit_url` unchanged -- still points to the same |
| 174 | source file. |
| 175 | 3. After ingest, the new `learn_rel_path` is computed from the |
| 176 | new tree position; the diff between previous and current |
| 177 | target produces an automatic Netlify redirect from the old |
| 178 | URL to the new (see `redirects.md`). |
| 179 | |
| 180 | ### Rename a page |
| 181 | |
| 182 | Same as a move -- the URL slug is derived from `sidebar_label`, |
| 183 | so changing `meta.label` changes the URL. Old URL is |
| 184 | auto-redirected. |
| 185 | |
| 186 | ### Delete a page |
| 187 | |
| 188 | The map.yaml side is just removing the node and the source |
| 189 | file. There is one manual step on the learn-repo side |
| 190 | (redirect surgery in |
| 191 | `LegacyLearnCorrelateLinksWithGHURLs.json`); see |
| 192 | `redirects.md` and `recipes/delete-doc-page.md`. |
| 193 | |
| 194 | ## Path-collision rule |
| 195 | |
| 196 | If two `to_publish` entries collide case-insensitively |
| 197 | (`<learn_rel_path>, <sidebar_label>` matches), ingest emits a |
| 198 | warning at `ingest.py:2891-2908`. On case-insensitive |
| 199 | filesystems (macOS / Windows), one file silently overwrites |
| 200 | the other. On Linux they coexist as separate files. |
| 201 | |
| 202 | ## What lookups happen |
| 203 | |
| 204 | The lookup key is the **canonical edit URL**: |
| 205 | |
| 206 | ``` |
| 207 | https://github.com/netdata/<repo>/edit/<branch>/<repo-rel-path> |
| 208 | ``` |
| 209 | |
| 210 | (`ingest.py:1232-1235`). Branch is `master` for everything |
| 211 | except `.github`, which uses `main`. This is the join key |
| 212 | between ingest's filesystem walk and `map.yaml`. |
| 213 | |
| 214 | If a source file's edit URL doesn't match any `map.yaml` row, |
| 215 | the file is skipped. No warning -- silent skip. |
| 216 | |
| 217 | ## Inter-page linking |
| 218 | |
| 219 | Markdown links between Learn pages must use the **GitHub-relative |
| 220 | path with `.md` extension** — NOT the learn.netdata.cloud URL, |
| 221 | and NOT a bare slug without extension. The ingest pipeline |
| 222 | rewrites these links to the correct Learn URLs during processing. |
| 223 | |
| 224 | **Correct:** |
| 225 | |
| 226 | ```markdown |
| 227 | See [Configuration](/docs/network-flows/configuration.md) for details. |
| 228 | ``` |
| 229 | |
| 230 | **Wrong (will NOT resolve on Learn or GitHub):** |
| 231 | |
| 232 | ```markdown |
| 233 | See [Configuration](/network-flows/configuration) for details. |
| 234 | ``` |
| 235 | |
| 236 | The link path is the repo-relative path to the source `.md` file, |
| 237 | prefixed with `/docs/`. The ingest pipeline matches it against the |
| 238 | `map.yaml` tree and rewrites it to the final Learn URL. |
| 239 | |
| 240 | This also means links work natively on GitHub — readers browsing |
| 241 | the source repo can click through to the linked file. |