| 1 | # Streaming-path subsystem — maintenance reference |
| 2 | |
| 3 | ## 1. What `stream.path.array` represents |
| 4 | |
| 5 | For every `RRDHOST` known to the local agent — its own `localhost`, every |
| 6 | real child currently or previously connected, and every vnode registered |
| 7 | locally — the `host->stream.path` field stores a small ordered array of |
| 8 | [`STREAM_PATH`](stream-path.c) entries. Each entry describes one hop in the |
| 9 | chain that this host's data takes from its origin upstream toward whichever |
| 10 | agent currently serves as the apex of the chain. |
| 11 | |
| 12 | The chain is per-host. On any given agent, two hosts can have different |
| 13 | chains stored. |
| 14 | |
| 15 | ## 2. Slot semantics |
| 16 | |
| 17 | Entries are sorted ascending by the `hops` field (`compare_by_hops`). Slot 0 |
| 18 | has the lowest hops and is the **origin** — the agent where the data was |
| 19 | collected. Slots 1+ are the **upstream hops**: the agent at slot 1 is the |
| 20 | direct parent of slot 0, the agent at slot 2 is the parent of slot 1, etc. |
| 21 | |
| 22 | Helper for read access by index: `rrdhost_stream_path_get_host_ids(host, |
| 23 | from, host_ids, max)` (returns just the UUIDs starting at `from`). |
| 24 | |
| 25 | For full per-entry data with a callback: `rrdhost_stream_path_visit(host, |
| 26 | from, cb, userdata)` (added by PR #22432 / commit `cc50307bc`; mirrors the |
| 27 | get_host_ids locking pattern but exposes hostname, hops, since, capabilities, |
| 28 | flags, etc.). |
| 29 | |
| 30 | ## 3. Storage rule (received vs emitted) |
| 31 | |
| 32 | This is the single most important invariant to understand: |
| 33 | |
| 34 | - **Storage**: `stream_path_set_from_json(host, json, from_parent)` clears |
| 35 | the existing array and stores exactly the entries received in the JSON, |
| 36 | then sorts by hops. **Self is not added at storage time.** |
| 37 | - **Emission**: `rrdhost_stream_path_to_json(wb, host, key, add_version)` |
| 38 | iterates the stored array. For any entry whose `host_id == localhost-> |
| 39 | host_id`, it overlays the entry with a fresh `rrdhost_stream_path_self()` |
| 40 | to refresh the volatile fields (timing medians, retention boundaries). If |
| 41 | no stored entry matches localhost, it appends a fresh self at the end. |
| 42 | |
| 43 | Why the asymmetry exists: `rrdhost_stream_path_self()` reads `first_time_t` |
| 44 | from `rrdhost_retention()` and `start_time_ms` / `shutdown_time_ms` from |
| 45 | `get_agent_event_time_median()`. These change continuously. Storing them |
| 46 | would either require a watcher to keep storage current, or accept that |
| 47 | storage always lags the live values. Appending self at emit time avoids |
| 48 | both problems by always emitting current self while keeping the stored array |
| 49 | constant for everyone else's view of the host. |
| 50 | |
| 51 | **Consequence**: any consumer that reads `host->stream.path.array` directly |
| 52 | (or via `rrdhost_stream_path_get_host_ids` / the visitor) is reading |
| 53 | storage, which may lag the live wire format. See §7. |
| 54 | |
| 55 | ## 4. Propagation rules |
| 56 | |
| 57 | `stream_path_set_from_json(host, json, from_parent)` runs the |
| 58 | [propagation switch](stream-path.c) at the bottom: |
| 59 | |
| 60 | ```c |
| 61 | if(!XXH128_isEqual(old_hash, new_hash)) { |
| 62 | if(!from_parent) |
| 63 | stream_path_send_to_parent(host); |
| 64 | stream_path_send_to_child(host); |
| 65 | } |
| 66 | ``` |
| 67 | |
| 68 | | Trigger source | `from_parent` | Path emitted UP | Path emitted DOWN | |
| 69 | |---|---|---|---| |
| 70 | | Path message received from a **child** (via `pluginsd_parser`) | `false` | YES (only meaningful on a proxy where `host->sender` is set) | YES (echo back to the child with self appended) | |
| 71 | | Path message received from our **upstream parent** (via `stream-sender-execute`) | `true` | NO (cycle terminator) | YES (forward downstream to our children, only meaningful on a proxy) | |
| 72 | |
| 73 | Multi-hop convergence works because each forwarded host on a proxy gets its |
| 74 | own sender — see `stream-receiver-connection.c:188` where the receiver |
| 75 | creates the new host with `config.send.enabled` propagated. The same host |
| 76 | record on the proxy has both a receiver (from below) and a sender (to |
| 77 | above), so a child-originated update flows all the way to the apex. |
| 78 | |
| 79 | ## 5. The cycle terminator |
| 80 | |
| 81 | The `if(!from_parent)` guard at `stream-path.c:423` is **load-bearing**. |
| 82 | Without it, every parent-to-child echo would generate a child-to-parent |
| 83 | re-send, generating another echo, forever. |
| 84 | |
| 85 | Combined with the volatile self-overlay (see §3), even the hash check at |
| 86 | `stream-path.c:422` cannot stop oscillation: the live timing fields change |
| 87 | between emits, so the hash would differ on every round-trip even when the |
| 88 | structural content is identical. |
| 89 | |
| 90 | If a future protocol redesign needs to remove this guard (e.g., to make |
| 91 | storage authoritative on every node so consumers don't have to read stored |
| 92 | data with a "may be stale" caveat), it must FIRST replace the cycle |
| 93 | terminator with a different mechanism (e.g., a per-message generation |
| 94 | counter that gets incremented only by the originating node) AND make the |
| 95 | self-overlay deterministic for hash purposes (split volatile fields out of |
| 96 | the hash input). |
| 97 | |
| 98 | ## 6. Update triggers |
| 99 | |
| 100 | `stream_path_send_to_parent` and `stream_path_send_to_child` are called |
| 101 | from a small, sparse set of triggers — they do NOT fire on every metric |
| 102 | write: |
| 103 | |
| 104 | - **Connect** — `stream_sender_on_ready_to_dispatch` at |
| 105 | `stream-sender.c:157` calls `stream_path_send_to_parent(localhost)` when |
| 106 | the sender becomes ready. |
| 107 | - **Retention boundary movement** — `rrdcontext_post_process_updates` at |
| 108 | `rrdcontext-worker.c:99-104` calls `stream_path_retention_updated(host)` |
| 109 | only when `host->retention.first_time_s` changes. This is sparse — once |
| 110 | per DB rotation / archive event. |
| 111 | - **node_id update** — `sqlite_metadata.c:288` (when persisted) and |
| 112 | `command-nodeid.c:163` (when receiving a NODE_ID command from a child) |
| 113 | call `stream_path_node_id_updated(host)`. |
| 114 | - **Parent disconnect** — `stream-sender.c:171` calls |
| 115 | `stream_path_parent_disconnected(s->host)` when our upstream connection |
| 116 | drops. This truncates self's stored chain back to localhost. |
| 117 | |
| 118 | Storage convergence has lag. Right after a child connects, the parent's |
| 119 | storage of the child's path is `[child]` only. After the next sparse trigger |
| 120 | fires on the child (typically minutes-to-hours), the parent's storage |
| 121 | catches up to the full chain. |
| 122 | |
| 123 | ## 7. Consumer guidance |
| 124 | |
| 125 | For any feature that asks "is X a parent? what is X's chain?", read live |
| 126 | state where possible (`rrdhost_status()` exposes `s.ingest.type` and |
| 127 | `s.ingest.status` per host, no need to touch `host->receiver` or |
| 128 | `host->sender` directly). Fall back to stored paths only as a hint, with |
| 129 | the understanding that storage lags the live state. |
| 130 | |
| 131 | Known consumers of `host->stream.path.array` in this tree: |
| 132 | |
| 133 | - `src/web/api/functions/function-topology-streaming.c` — the |
| 134 | `topology:streaming` Function. Uses both `streaming_topology_get_path_ids` |
| 135 | (paths for actor/link emission) and `rrdhost_stream_path_visit` (Bug C |
| 136 | synthesis from PR #22432). Reads stored data, falls back to the |
| 137 | rrdhost-derived live data via `rrdhost_status()` for the localhost |
| 138 | classification (Bug A in PR #22432). |
| 139 | - `src/database/contexts/api_v2_contexts.c:425, :510` — emits |
| 140 | `streaming_path` per host via `rrdhost_stream_path_to_json` (so the |
| 141 | emit-time self-append fires). No direct array access. |
| 142 | - `src/streaming/stream-path.c:145-158` — `rrdhost_stream_path_total_reboot_time_ms` |
| 143 | walks the stored array looking for localhost's own entry. **This shares |
| 144 | Bug A's blind assumption**: on the apex parent, the localhost entry is |
| 145 | not in storage, so this function returns 0 silently. This is the same |
| 146 | failure class as the PR #22432 streaming topology fix and needs a separate |
| 147 | GitHub issue before implementation starts. |
| 148 | |
| 149 | ## 8. vnode special case |
| 150 | |
| 151 | Vnodes do not stream — they are virtual hosts collected directly by the |
| 152 | local agent (e.g., SNMP devices). Their stored `stream.path.array` is empty. |
| 153 | At JSON-emit time, `rrdhost_stream_path_to_json` appends a fresh self |
| 154 | (the collecting agent's identity) so the wire format always shows a vnode |
| 155 | hop into the collector at slot 0. |
| 156 | |
| 157 | For any consumer that synthesizes actors from path entries (e.g., the |
| 158 | topology function's Bug C synthesis), vnodes contribute nothing — their |
| 159 | visits via `rrdhost_stream_path_visit` return zero entries. |
| 160 | |
| 161 | ## 9. Topology-function classification logic |
| 162 | |
| 163 | The `topology:streaming` Function classifies each entry in |
| 164 | `rrdhost_root_index` as `parent`, `child`, `vnode`, or `stale`. The intended |
| 165 | rule was confirmed during PR #22432: |
| 166 | |
| 167 | 1. **Source of truth = `rrdhost_root_index`.** Every actor on the |
| 168 | topology graph corresponds to either an entry in this index OR a |
| 169 | synthesized "remote parent" derived from path data (Bug C synthesis). |
| 170 | 2. **For each non-vnode node, slot 0 of its `stream.path.array` is the |
| 171 | origin (a child role in this chain) and slots 1+ are upstream parents.** |
| 172 | 3. **A node X is classified `parent` if X appears at slot 1+ in any |
| 173 | node's path.** Otherwise `child`. |
| 174 | 4. **vnodes** are tagged `vnode` directly — they have no upstream chain |
| 175 | visible in their own stored data. |
| 176 | 5. **Stale** nodes (`s.ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED`) are |
| 177 | tagged `stale` directly and rendered as a stale link to localhost. |
| 178 | |
| 179 | For the localhost itself, classification reads live state via |
| 180 | `rrdhost_status()` (count of non-virtual children with `s.ingest.type == |
| 181 | CHILD` and `s.ingest.status` ∈ `{ONLINE, REPLICATING}`). This was Bug A in |
| 182 | PR #22432 — the path-based check was unreliable on apex agents because |
| 183 | storage hadn't converged yet. |
| 184 | |
| 185 | For the cross-agent merge (Cloud combines topology responses from multiple |
| 186 | parents into a single graph), the agent's response carries a stable |
| 187 | `agent_id` at the top level. The merge layer maps it to the canonical |
| 188 | actor_id form (`netdata-machine-guid:<agent_id>`) and uses that to identify |
| 189 | which actor in `actors[]` is the response's "self" — that response is the |
| 190 | authoritative source for that actor's attributes. |
| 191 | |
| 192 | For bug history, decisions, and design rationale, use PR #22432 / commit |
| 193 | `cc50307bc6ca180285a0e0cda16d73ab7a42cd86`. Completed SOW working files are |
| 194 | not retained on `master`; durable behavior belongs in this maintenance |
| 195 | reference. |