| 1 | # Pipeline: ingest, CI, deploy |
| 2 | |
| 3 | This document maps the Learn ingest pipeline end to end: every |
| 4 | step `ingest.py` performs, the source repositories it pulls |
| 5 | from, the CI workflow that triggers it, and how the result |
| 6 | gets to `learn.netdata.cloud`. |
| 7 | |
| 8 | ## The orchestrator: `ingest/ingest.py` |
| 9 | |
| 10 | Live entrypoint: `${NETDATA_REPOS_DIR}/learn/ingest/ingest.py`. |
| 11 | The legacy `ingest.js` and `ingest.md` at the learn-repo root |
| 12 | are NOT the orchestrator -- ignore them. |
| 13 | |
| 14 | ### Argument parsing (`ingest.py:2513-2762`) |
| 15 | |
| 16 | | Flag | Purpose | |
| 17 | |---|---| |
| 18 | | `--repos OWNER/REPO:BRANCH ...` | Override the default repo list. | |
| 19 | | `--local-repo NAME:/path ...` | Use a local copy via `shutil.copytree` instead of cloning. | |
| 20 | | `--dry-run` | Skip output writes. | |
| 21 | | `--debug` | Verbose logging. | |
| 22 | | `--docs-prefix DOCS` | Default `docs`; the target directory in learn for published files. | |
| 23 | | `--fail-links` | Exit 1 if broken internal links are detected at the end of the run. | |
| 24 | | `--fail-links-{netdata,helmchart,onprem,asd,grafana,github}` | Per-source-repo fail-on-broken-links. | |
| 25 | | `--gh-token` | GitHub token for clones. | |
| 26 | | `--use_plain_https` | Force HTTPS clone URLs (no SSH). | |
| 27 | | `--ignore-on-prem-repo` | Skip `netdata-cloud-onprem` clone and ignore broken-link references to it. | |
| 28 | |
| 29 | ### The 16-step flow (`ingest.py:2513-3087`, `__main__`) |
| 30 | |
| 31 | 1. **Argument parsing**. |
| 32 | |
| 33 | 2. **Cleanup** (`ingest.py:2766-2768`): |
| 34 | - `unsafe_cleanup_folders('ingest-temp-folder')` -- wipes |
| 35 | the work dir. |
| 36 | - `safe_cleanup_learn_folders('docs')` -- walks |
| 37 | `${NETDATA_REPOS_DIR}/learn/docs/` and removes every |
| 38 | `.md` / `.mdx` / `.json` that does NOT carry |
| 39 | `part_of_learn: True`. The flag is the **opt-in for |
| 40 | hand-authored files** that should survive between runs. |
| 41 | Currently only `docs/ask-nedi.mdx`. |
| 42 | |
| 43 | 3. **Clone all repos** (`ingest.py:2776-2800`) into |
| 44 | `ingest-temp-folder/<repo>/` with `--depth 1`. Each repo |
| 45 | tracks the branch declared in `default_repos` |
| 46 | (`ingest.py:74-105`). `--ignore-on-prem-repo` skips the |
| 47 | on-prem clone and ignores broken-link references to it. |
| 48 | `--local-repo X:/path` and `--repos /path/to/repo` use |
| 49 | `shutil.copytree` for developer ergonomics. |
| 50 | |
| 51 | 4. **Move + validate map** (`ingest.py:2802-2825`): |
| 52 | - `ingest-temp-folder/netdata/docs/.map/map.yaml` -> |
| 53 | `./map.yaml`. |
| 54 | - Validate against |
| 55 | `ingest-temp-folder/netdata/docs/.map/map.schema.json`. |
| 56 | - Build `MAP_SIDEBAR_ORDER` and `MAP_DOC_SCOPE` for sidebar |
| 57 | position assignment. |
| 58 | - Schema failure -> exit code 2. |
| 59 | |
| 60 | 5. **Enumerate markdowns** (`ingest.py:2828-2829`): |
| 61 | `glob('ingest-temp-folder/**/*.md*')` plus dot-directories |
| 62 | (`fetch_markdown_from_repo`, `ingest.py:1207-1211`). |
| 63 | |
| 64 | 6. **Populate integrations** (`ingest.py:2832`): scan all |
| 65 | markdowns for `INTEGRATION_MARKER`, parse hidden metadata, |
| 66 | bucket by category, replace each `integration_placeholder` |
| 67 | row in the map with sorted (by `learn_rel_path`, |
| 68 | `sidebar_label`) integration rows. Writes the resulting |
| 69 | tabular map to `ingest/generated_map.yaml`. |
| 70 | |
| 71 | 7. **Sidebar position assignment** (`ingest.py:2838-2840`): |
| 72 | `automate_sidebar_position` walks the (now expanded) map |
| 73 | dataframe and assigns sibling-relative positions in steps |
| 74 | of 10 within each parent scope, ordered by map traversal. |
| 75 | |
| 76 | 8. **Inject metadata into source files** (`ingest.py:2842-2889`): |
| 77 | for each markdown, look up its row by `custom_edit_url` and |
| 78 | write the hidden metadata block at the top of the file. If |
| 79 | `learn_status == Published`, compute the destination via |
| 80 | `create_mdx_path_from_metadata`, store the destination + |
| 81 | ingestedRepo in `to_publish`, also inject `slug:` and |
| 82 | `learn_link:` into the file. Otherwise file is dropped. |
| 83 | |
| 84 | 9. **Path collision check** (`ingest.py:2891-2908`): warn if |
| 85 | any two `to_publish` entries collide case-insensitively. |
| 86 | |
| 87 | 10. **Publish each file** (`ingest.py:2916-2919`): |
| 88 | - `local_to_absolute_links(md_file, to_publish)` -- |
| 89 | convert relative/abs-from-repo-root markdown links into |
| 90 | GitHub view URLs (rewritten in step 12). |
| 91 | - `copy_doc(md_file, to_publish[md_file]['learnPath'])` -- |
| 92 | copy temp file to `docs/<...>.mdx` (creating dirs). |
| 93 | - `sanitize_page(learnPath)` -- runs MDX-escape transforms |
| 94 | (see `mdx-rules.md`). |
| 95 | |
| 96 | 11. **Build cross-reference dictionary** (`ingest.py:2925-2927`): |
| 97 | `add_new_learn_path_key_to_dict` produces, for every |
| 98 | published file, a map: GitHub-view-link AND |
| 99 | GitHub-edit-link -> final `/docs/...` URL. Handles the |
| 100 | duplicate-segment slug trim and explicit `slug:` |
| 101 | overrides. |
| 102 | |
| 103 | 12. **Rewrite GitHub links to Learn URLs** (`ingest.py:2929-2930`): |
| 104 | `convert_github_links` walks every published file. Any |
| 105 | `https://github.com/netdata/<repo>/blob/.../*.md` in the |
| 106 | body that maps to a file in `to_publish` is rewritten to |
| 107 | its final Learn URL. Header anchors are validated; broken |
| 108 | anchors are accumulated. Links to integration md files |
| 109 | that aren't in the map fall back to the parent README's |
| 110 | URL (`ingest.py:2153-2209`). Links to GitHub files that |
| 111 | exist in the repos but aren't in the map stay as GitHub |
| 112 | links (intentional -- `file_exists_in_repos` at |
| 113 | `ingest.py:711-719`). Truly missing targets become |
| 114 | `UNCORRELATED_LINK_COUNTER` increments. |
| 115 | |
| 116 | 13. **Generate redirects** (`ingest.py:2932`): see |
| 117 | `redirects.md`. |
| 118 | |
| 119 | 14. **Broken-link reporting** (`ingest.py:2940-3022`): print |
| 120 | broken URLs grouped by repo, broken anchors grouped by |
| 121 | repo, decide whether to exit 1 based on `--fail-links` |
| 122 | flags. |
| 123 | |
| 124 | 15. **Post-processing** (`ingest.py:3030-3065`): |
| 125 | - Save current `(custom_edit_url -> new_learn_path)` |
| 126 | mapping to `ingest/one_commit_back_file-dict.yaml` for |
| 127 | next run's redirect diff. |
| 128 | - Cleanup `ingest-temp-folder` and `map.yaml`. |
| 129 | - `get_dir_make_file_and_recurse('./docs')` -- auto-create |
| 130 | grid pages for any integration directory that lacks an |
| 131 | overview. |
| 132 | - `ensure_category_json_for_dirs('docs')` -- write |
| 133 | `_category_.json` for any directory still without an |
| 134 | overview page. |
| 135 | - `normalize_sidebar_positions_by_parent('docs')` -- |
| 136 | assign sibling positions deterministically per parent |
| 137 | scope. |
| 138 | |
| 139 | 16. **No git push from the script.** The CI workflow handles |
| 140 | `git add` / commit / PR. |
| 141 | |
| 142 | ### How content gets fetched |
| 143 | |
| 144 | `ingest.py` does NOT use the GitHub REST API for content. It |
| 145 | clones full repos with `gitpython` (`ingest.py:1101-1137`), so |
| 146 | there's no API rate-limit consideration in the live pipeline. |
| 147 | (Legacy `ingest.js` did use the API; ignore it.) |
| 148 | |
| 149 | ## The 6 source repositories |
| 150 | |
| 151 | Default at `ingest/ingest.py:74-105`: |
| 152 | |
| 153 | | Logical repo | Owner | Branch | What it feeds | |
| 154 | |---|---|---|---| |
| 155 | | `netdata` | `netdata` | `master` | The bulk of Learn -- all docs declared in `<repo>/docs/.map/map.yaml`, plus integration metadata generated by the agent build (collectors, exporters, secretstore, alerts, functions). Hosts the canonical map files. | |
| 156 | | `netdata-cloud-onprem` | `netdata` | `master` | On-prem documentation (Netdata Cloud On-Prem category). Skippable via `--ignore-on-prem-repo`. | |
| 157 | | `.github` | `netdata` | **`main`** | `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md` -- only those whose edit URLs appear in `map.yaml`. | |
| 158 | | `agent-service-discovery` | `netdata` | `master` | Service-discovery docs for the agent. | |
| 159 | | `netdata-grafana-datasource-plugin` | `netdata` | `master` | Grafana datasource integration docs. | |
| 160 | | `helmchart` | `netdata` | `master` | Kubernetes Helm chart docs. | |
| 161 | |
| 162 | Override at runtime: `--repos OWNER/REPO:BRANCH ...` or |
| 163 | `--local-repo NAME:/path ...` (`ingest.py:2607-2613`). |
| 164 | |
| 165 | The legacy `ingest.js` listed only 4 repos (`netdata`, `.github`, |
| 166 | `go.d.plugin`, `agent-service-discovery`). `go.d.plugin` is no |
| 167 | longer pulled (its docs were absorbed into the netdata |
| 168 | monorepo); `netdata-cloud-onprem`, |
| 169 | `netdata-grafana-datasource-plugin`, `helmchart` are new in the |
| 170 | Python pipeline. |
| 171 | |
| 172 | ## CI: `${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml` |
| 173 | |
| 174 | ### Triggers (`ingest.yml:2-18`) |
| 175 | |
| 176 | - `workflow_dispatch` -- manual. |
| 177 | - `schedule: cron "10 8-23/3 * * *"` -- every 3 hours from |
| 178 | 08:10 to 23:10 UTC (6 runs/day). |
| 179 | - `push` to `master` when `plugins/**`, `src/**`, `static/**`, |
| 180 | `sidebar.js`, `package.json`, `yarn.lock`, |
| 181 | `tailwind.config.js`, or any `**/*.md` / `**/*.mdx` changes |
| 182 | in the LEARN repo (not in source repos -- those trigger |
| 183 | ingest indirectly via the cron). |
| 184 | |
| 185 | ### Steps (`ingest.yml:24-101`) |
| 186 | |
| 187 | 1. Checkout (full history, `fetch-depth: 0`) using |
| 188 | `secrets.GITHUB_TOKEN`. |
| 189 | 2. SSH agent for `secrets.NETDATABOT_SSH_PRIVATE_KEY` (used to |
| 190 | clone private repos like `.github`). |
| 191 | 3. Python 3.10, install |
| 192 | `${NETDATA_REPOS_DIR}/learn/.learn_environment/ingest-requirements.txt` |
| 193 | (`pip`, `requests==2.33.0`, `Pillow`, `PyGithub`, `gitpython`, |
| 194 | `mergedeep`, `pandas`, `numpy`, `retrypy`, `pyyaml`, |
| 195 | `jsonschema`). |
| 196 | 4. **Run `python ingest/ingest.py --fail-links 2>&1`** -- |
| 197 | captures output for issue creation, propagates exit code 1 |
| 198 | (broken links) into a workflow output but does NOT fail |
| 199 | the workflow on broken links (only on real errors, |
| 200 | `ingest.yml:65-77`). |
| 201 | 5. **Update kickstart checksum** -- fetches |
| 202 | `https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/kickstart.sh`, |
| 203 | MD5s it, replaces literal `@KICKSTART_CHECKSUM@` placeholder |
| 204 | in `docs/Netdata Agent/Installation/Linux/Linux.mdx`. |
| 205 | 6. **`peter-evans/create-pull-request@v6.0.1`** -- opens or |
| 206 | updates a branch named `ingest` with title |
| 207 | "Ingest New Documentation", labels `ingest, automation`. The |
| 208 | PR is reviewed and merged manually by the team. |
| 209 | 7. If broken links were found, create or update a single open |
| 210 | GitHub issue with label `broken-links`. |
| 211 | |
| 212 | ## CI: `${NETDATA_REPOS_DIR}/learn/.github/workflows/daily-learn-link-check.yml` |
| 213 | |
| 214 | Daily cron `00 04 * * *` -- runs |
| 215 | `python ingest/check_learn_links.py`, which extracts every |
| 216 | `learn_link:` URL from every `.md` / `.mdx` in `docs/`, |
| 217 | HEAD/GET checks them, exits 1 (failing the workflow) when any |
| 218 | 404 is observed. |
| 219 | |
| 220 | ## Deploy: Netlify |
| 221 | |
| 222 | There is **no GitHub-side build/deploy workflow**. Build & |
| 223 | deploy are handled by Netlify: |
| 224 | |
| 225 | - README confirms Netlify deploys from `master` automatically. |
| 226 | - Site is `netdata-docusaurus` on Netlify. |
| 227 | - Branches `master`, `staging`, `staging1` each get their own |
| 228 | deploy (preview URLs). |
| 229 | - Build env pinned by `static.toml:3-4` / |
| 230 | `netlify.toml [build]`: `NPM_VERSION=10.9.2`, |
| 231 | `NODE_VERSION=22.14.0`, `NETLIFY_USE_YARN=true`, |
| 232 | `NODE_OPTIONS=--max_old_space_size=4096`. |
| 233 | - Build command: `yarn build` (Docusaurus default). |
| 234 | - Output dir: `build/`. |
| 235 | - Propagation time: whatever Netlify takes after a master |
| 236 | commit; redirects ship in `netlify.toml` so they apply at |
| 237 | the edge immediately upon deploy. |
| 238 | |
| 239 | ## End-to-end timing |
| 240 | |
| 241 | 1. Maintainer pushes to `netdata/netdata` master (via merged |
| 242 | PR). |
| 243 | 2. Up to 3 hours later, `${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml` |
| 244 | cron fires. |
| 245 | 3. Ingest opens a PR in the learn repo titled "Ingest New |
| 246 | Documentation". |
| 247 | 4. A maintainer reviews and merges the PR. |
| 248 | 5. Netlify auto-deploys `master`. Within minutes, the new |
| 249 | content is live on `learn.netdata.cloud`. |
| 250 | |
| 251 | So end-to-end: 0-3 hours of cron lag + manual review/merge + |
| 252 | minutes of Netlify deploy. Plan accordingly when something |
| 253 | must be live by a deadline -- you can also trigger the ingest |
| 254 | manually via `workflow_dispatch` to skip the cron lag. |
| 255 | |
| 256 | ## Local testing |
| 257 | |
| 258 | From within the learn repo: |
| 259 | |
| 260 | ```bash |
| 261 | # Install deps once. |
| 262 | python3 -m venv venv && . venv/bin/activate |
| 263 | pip install -r .learn_environment/ingest-requirements.txt |
| 264 | |
| 265 | # Test against your local netdata clone. |
| 266 | python3 ingest/ingest.py --local-repo netdata:<repo> --ignore-on-prem-repo --fail-links-netdata |
| 267 | |
| 268 | # Then build the site locally. |
| 269 | yarn install |
| 270 | yarn start # dev server |
| 271 | # OR |
| 272 | yarn build # full build to ./build |
| 273 | ``` |