@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 9.55 KB
View source Raw

Mapping: source -> Learn URL

The single most important fact to internalize about Learn: source filesystem path is irrelevant for routing. A page's URL on learn.netdata.cloud is computed from frontmatter that ingest.py INJECTS from <repo>/docs/.map/map.yaml. Without this mental model, every other rule seems arbitrary.

The map.yaml source of truth

<repo>/docs/.map/map.yaml is the canonical declaration of which files are published on Learn and where they appear. It is a hierarchical tree of nodes; each leaf node represents one published page.

Schema: <repo>/docs/.map/map.schema.json. Authoring guide: <repo>/docs/.map/README.md.

Per-node fields

Each meta block declares one publication node:

Field Type Required Notes
meta.label string yes Display name. Becomes sidebar_label AND drives the destination filename.
meta.edit_url string yes (for doc-emitting nodes) Must match `https://github\.com/netdata//edit//.+.(md\
meta.path string no Single-segment override of the URL path component if it should differ from the label.
meta.description string no Frontmatter description; powers the page's meta description.
meta.keywords array no Frontmatter keywords; powers in-app search and Learn page metadata.

The schema sets additionalProperties: false. Unknown keys fail validation and abort the run with exit code 2 (ingest/ingest.py:2815-2819).

Branches

For all repos in the source list (see pipeline.md), the edit_url branch is master -- with one exception: the netdata/.github repo uses main branch instead of master (ingest.py:1232-1235).

Integration placeholders

integration_placeholder: true rows in map.yaml are substituted at ingest time by integration pages auto-discovered from the source repos. The populate_integrations step (ingest.py:752-1003) finds every .md carrying the DO NOT EDIT THIS FILE DIRECTLY marker (INTEGRATION_MARKER, ingest.py:113), parses its hidden <!--startmeta...endmeta--> block, buckets by category (collectors / exporters / secretstore / functions / authentication / cloud-notifications / agent-notifications / logs), sorts by (learn_rel_path, sidebar_label), and inserts the rows in place of the matching placeholder. Result is written to ${NETDATA_REPOS_DIR}/learn/ingest/generated_map.yaml.

So integration pages do NOT need explicit map.yaml rows; they are pulled in via the placeholder mechanism.

Frontmatter that ingest INJECTS

When ingest matches a source file's custom_edit_url to a map.yaml row, it writes a hidden HTML-comment metadata block at the top of the source file. After sanitize_page runs, the comment delimiters become real YAML frontmatter (<!-- -> ---, --> -> ---).

Per ingest.py:1221-1299 plus ingest.py:1832-1897:

Field Source Notes
custom_edit_url meta.edit_url from map.yaml Used by the "Edit this page" link. For autogenerated grid pages, this is null.
sidebar_label meta.label Drives sidebar AND destination filename.
learn_status "Published" or "AUTOGENERATED" Anything other than Published is excluded from to_publish.
learn_rel_path reconstructed from map.yaml hierarchy Drives the destination directory tree. root means write to docs/ directly.
keywords meta.keywords Normalized to inline YAML array. CSV strings, brackets, and Python lists all accepted.
description meta.description Skipped when empty/None to minimize diffs.
sidebar_position computed from map traversal order Always written as quoted string sidebar_position: "10".
slug computed (/<learn_rel_path>/<label> lowercased) Author-supplied slug: in source is preserved as override (ingest.py:1890-1895).
learn_link https://learn.netdata.cloud/docs<slug> Updated each ingest. Validated daily by check_learn_links.py.

Special case: paths containing Collecting Metrics get toc_max_heading_level: 6 and toc_collapsible: true injected (ingest.py:1251-1253).

Source-path-to-URL computation

For files marked learn_status: Published, the destination is computed by create_mdx_path_from_metadata (ingest.py:1140-1204):

docs/<learn_rel_path>/<sanitized_sidebar_label>.mdx

Sanitization rules

sidebar_label is sanitized for the filename (ingest.py:1159-1175): 1. Strip ', :, /, (, ), ,, backtick. 2. Collapse repeated whitespace to single space.

For the URL slug (different from filename), additionally: - Lowercase everything. - Spaces -> -. - // -> /.

Filename keeps original case; slug is lowercase. So docs/Welcome to Netdata/Welcome to Netdata.mdx exists with spaces and capitals.

Special-cases

  1. Functions integrations -- if the source custom_edit_url is under /integrations/functions/, the filename uses the function slug from the URL stem (with - -> ) instead of sidebar_label (ingest.py:1153-1157). Avoids collisions when many integrations share a "Top Queries" label.

  2. Category overview pages -- if the last two segments of the slug are equal (e.g. /collecting-metrics/collecting-metrics), create_mdx_path_from_metadata returns the slug with the duplicate trimmed (ingest.py:1178-1191). On disk the file is still written as docs/<X>/<X>.mdx; Docusaurus then routes it to the parent URL /<X> because it has the same slug.

  3. Frontmatter slug: override -- if the source file already has slug: declared in frontmatter (custom-authored, like docs/ask-nedi.mdx), that value wins (ingest.py:1890-1895). For these files, the author-supplied slug must be set BEFORE the <!-- ... --> metadata block so it survives the rewrite.

  4. id: in mapDict -- if the metadata in mapDict carries an id, the last segment of the URL is replaced by the id (ingest.py:2120-2126).

Edge cases

  • README.md, index.md, _index.md, index.mdx, hidden directories: NONE of these are special-cased in ingest.py. The Python pipeline never publishes a file unless it appears in map.yaml (or comes from an integration placeholder), so filename has no routing effect.
  • Files at repo root vs deep paths: same rule -- destination is driven by map.yaml.meta.label + the node's position in the hierarchy, not by source path depth.
  • Special filenames: ignored. Only map.yaml matters.

Examples

Add a new top-level page

  1. Create <repo>/docs/getting-started-netdata/quick-tour.md. The exact path doesn't matter for routing -- use a reasonable location.
  2. Add to <repo>/docs/.map/map.yaml: yaml - meta: label: Quick Tour edit_url: https://github.com/netdata/netdata/edit/master/docs/getting-started-netdata/quick-tour.md description: A 5-minute tour of Netdata. keywords: [tour, getting-started, intro] under the appropriate parent.
  3. Resulting Learn URL: https://learn.netdata.cloud/docs/<parent-slug>/quick-tour.
  4. Resulting filename in learn repo: docs/<parent>/Quick Tour.mdx (with spaces and capitals).

Move a page (different sidebar location, same source file)

  1. In map.yaml, move the node to its new parent.
  2. Keep meta.edit_url unchanged -- still points to the same source file.
  3. After ingest, the new learn_rel_path is computed from the new tree position; the diff between previous and current target produces an automatic Netlify redirect from the old URL to the new (see redirects.md).

Rename a page

Same as a move -- the URL slug is derived from sidebar_label, so changing meta.label changes the URL. Old URL is auto-redirected.

Delete a page

The map.yaml side is just removing the node and the source file. There is one manual step on the learn-repo side (redirect surgery in LegacyLearnCorrelateLinksWithGHURLs.json); see redirects.md and recipes/delete-doc-page.md.

Path-collision rule

If two to_publish entries collide case-insensitively (<learn_rel_path>, <sidebar_label> matches), ingest emits a warning at ingest.py:2891-2908. On case-insensitive filesystems (macOS / Windows), one file silently overwrites the other. On Linux they coexist as separate files.

What lookups happen

The lookup key is the canonical edit URL:

https://github.com/netdata/<repo>/edit/<branch>/<repo-rel-path>

(ingest.py:1232-1235). Branch is master for everything except .github, which uses main. This is the join key between ingest's filesystem walk and map.yaml.

If a source file's edit URL doesn't match any map.yaml row, the file is skipped. No warning -- silent skip.

Inter-page linking

Markdown links between Learn pages must use the GitHub-relative path with .md extension — NOT the learn.netdata.cloud URL, and NOT a bare slug without extension. The ingest pipeline rewrites these links to the correct Learn URLs during processing.

Correct:

See [Configuration](/docs/network-flows/configuration.md) for details.

Wrong (will NOT resolve on Learn or GitHub):

See [Configuration](/network-flows/configuration) for details.

The link path is the repo-relative path to the source .md file, prefixed with /docs/. The ingest pipeline matches it against the map.yaml tree and rewrites it to the final Learn URL.

This also means links work natively on GitHub — readers browsing the source repo can click through to the linked file.