| 1 | # Spec - Topology Modes, Correlation, Aggregation, And Actor Identification |
| 2 | |
| 3 | ## Status |
| 4 | |
| 5 | Implementation contract introduced by topology v1 producer work in PR #22496 / |
| 6 | commit `8afe52d8b`. |
| 7 | |
| 8 | PR #22496 completed the cross-repo compatibility layer for modes, modal |
| 9 | identification, correlation classes, and table merge policy. The stronger |
| 10 | detailed network-connections loose-side graph model remains target behavior, |
| 11 | but requires a separate Agent/UI/aggregator execution pass before it becomes |
| 12 | current contract. |
| 13 | |
| 14 | ## Purpose |
| 15 | |
| 16 | Topology payloads must let operators inspect a topology at the right level: |
| 17 | |
| 18 | - exact evidence when they need detailed troubleshooting; |
| 19 | - compact relationships when they need an infrastructure-level map; |
| 20 | - consistent cross-node correlation when many independently produced payloads |
| 21 | are merged; |
| 22 | - useful actor modals without duplicating the same facts only for display. |
| 23 | |
| 24 | The contract must remain topology-agnostic. The UI and aggregator must not learn |
| 25 | domain words such as process, router, parent, child, endpoint, LLDP, socket, or |
| 26 | retention as hardcoded behavior. Producers describe identities, modes, merge |
| 27 | rules, table merge policies, and presentation recipes in the payload. |
| 28 | |
| 29 | ## Terms |
| 30 | |
| 31 | - **Producer**: the Agent Function that emits one `netdata.topology.v1` |
| 32 | payload, for example network-connections, SNMP/L2, streaming, or vSphere. |
| 33 | - **Aggregator**: Cloud service that fans out to many producers, decodes their |
| 34 | payloads, correlates them, optionally aggregates detail, and returns a normal |
| 35 | `netdata.topology.v1` payload. |
| 36 | - **UI**: Cloud frontend topology renderer and actor/link modal renderer. |
| 37 | - **Detailed mode**: payload or returned view that keeps the finest evidence |
| 38 | grain the producer exposes for troubleshooting. |
| 39 | - **Aggregated mode**: payload or returned view that groups detailed evidence |
| 40 | into compact relationships for map readability. |
| 41 | - **Known actor**: an entity the producer knows exists, such as a process, host, |
| 42 | SNMP device, interface, streaming node, vSphere object, or Kubernetes object. |
| 43 | - **Loose side**: one side of a relationship row that has endpoint facts but no |
| 44 | actor reference yet. |
| 45 | - **Materialized actor**: actor created from loose-side facts for presentation |
| 46 | or partial correlation, such as an endpoint grouped by IP. |
| 47 | - **Replacement**: aggregation action where a weaker actor is removed and all |
| 48 | incident relationships are rewired to a stronger actor. |
| 49 | - **Enrichment**: aggregation action where rows from multiple actors with the |
| 50 | same identity are merged into one actor, preserving all non-conflicting facts. |
| 51 | - **Evidence table**: lossless or near-lossless relationship facts, for example |
| 52 | sockets, L2 observations, or streaming relationships. |
| 53 | - **Relationship summary table**: compact relationship rows at a grain between |
| 54 | graph links and detailed evidence. |
| 55 | - **Actor labels**: actor-owned key/value rows for modal labels and display |
| 56 | selection. They are not a replacement for typed identity, matching, grouping, |
| 57 | sorting, filtering, or aggregation columns. |
| 58 | |
| 59 | ## Global Rules |
| 60 | |
| 61 | ### Mode Request |
| 62 | |
| 63 | The user-facing request key for topology mode is `__topology_mode`. |
| 64 | |
| 65 | Allowed values: |
| 66 | |
| 67 | - `detailed` |
| 68 | - `aggregated` |
| 69 | |
| 70 | If the key is absent, each Function uses its documented default. Producers that |
| 71 | do not have a meaningful detailed/aggregated difference should not expose a mode |
| 72 | selector only to return identical output. |
| 73 | |
| 74 | Mode-capable producers declare `data.view.supported_modes`. Consumers treat an |
| 75 | absent field or a single-value field as mode-invariant and must not show a |
| 76 | detailed/aggregated toggle for that payload. |
| 77 | |
| 78 | ### Aggregator Fanout |
| 79 | |
| 80 | The aggregator must consume detailed payloads whenever a producer supports |
| 81 | detail mode. |
| 82 | |
| 83 | When the user asks the aggregator for `__topology_mode=aggregated`, the |
| 84 | aggregator must rewrite fanout requests to producers as |
| 85 | `__topology_mode=detailed` before correlation and aggregation. This prevents |
| 86 | early information loss before cross-node matching. After correlation, the |
| 87 | aggregator returns either detailed or aggregated output according to the |
| 88 | original user request. |
| 89 | |
| 90 | If a producer does not expose `__topology_mode`, the aggregator must not invent |
| 91 | that parameter for it. SNMP/L2 and streaming are expected to be mode-invariant |
| 92 | unless a future producer change defines a real mode difference. |
| 93 | |
| 94 | ### Final Output |
| 95 | |
| 96 | Aggregator internal states must not appear in final payloads. Terms such as |
| 97 | absorbed, candidate, rewrite plan, partial class, or equivalence set may exist |
| 98 | inside the service, but final topology output contains only normal actors, |
| 99 | links, tables, labels, presentation, and diagnostics. |
| 100 | |
| 101 | ### No Duplicate Display Facts |
| 102 | |
| 103 | Do not copy high-cardinality evidence rows only to make modal tables easier. |
| 104 | Modal sections must select and project existing actors, links, evidence, |
| 105 | relationship tables, actor tables, and actor labels. |
| 106 | |
| 107 | Small scalar facts may appear in more than one plane when the grains differ. For |
| 108 | example, a graph link and a relationship-summary row may both carry |
| 109 | `socket_count`; the graph link is the renderable relationship, while the summary |
| 110 | row is the modal/drilldown grain. |
| 111 | |
| 112 | ### Actor Modal Identification |
| 113 | |
| 114 | The modal identification area is part of the schema contract. |
| 115 | |
| 116 | `types.actor_types.<id>.presentation.modal.labels` must be extended with an |
| 117 | ordered producer-selected identification list over the existing actor label |
| 118 | table. The UI renders those selected label keys near the actor title. The full |
| 119 | label table remains available in the Labels tab. |
| 120 | |
| 121 | Target shape: |
| 122 | |
| 123 | ```json |
| 124 | { |
| 125 | "labels": { |
| 126 | "enabled": true, |
| 127 | "table": "actor_labels", |
| 128 | "actor_column": "actor", |
| 129 | "key_column": "key", |
| 130 | "value_column": "value", |
| 131 | "identification": { |
| 132 | "enabled": true, |
| 133 | "fields": [ |
| 134 | { "key": "process", "label": "Process", "max_values": 1 }, |
| 135 | { "key": "username", "label": "User", "max_values": 1 }, |
| 136 | { "key": "cmdline", "label": "Command", "max_values": 1 } |
| 137 | ] |
| 138 | } |
| 139 | } |
| 140 | } |
| 141 | ``` |
| 142 | |
| 143 | Rules: |
| 144 | |
| 145 | - `identification.fields[]` selects rows from `actor_labels` by `key`. |
| 146 | - Selection is per selected modal actor through `actor_column`. |
| 147 | - Repeated values are ordered by `value_index` when present. |
| 148 | - Missing selected keys are skipped; they do not create empty labels. |
| 149 | - `max_values` limits displayed values for one key. The full Labels tab still |
| 150 | shows all values. |
| 151 | - The UI must not guess important labels from key names. |
| 152 | - Producers must not duplicate these values into a separate modal-only table. |
| 153 | |
| 154 | ## Schema Additions Required |
| 155 | |
| 156 | This spec requires these schema extensions beyond the currently deployed v1 |
| 157 | contract: |
| 158 | |
| 159 | 1. Modal label identification metadata: |
| 160 | `modal.labels.identification.enabled` and |
| 161 | `modal.labels.identification.fields[]`. |
| 162 | 2. Link-side materialization policy for tables that can carry loose sides: |
| 163 | producers must declare how a loose side can be grouped into presentation |
| 164 | actors when a detailed row has only one real actor. |
| 165 | 3. Aggregation/correlation rules must support three semantic outcomes: |
| 166 | - loose-side resolution; |
| 167 | - actor replacement; |
| 168 | - actor enrichment. |
| 169 | 4. Table merge policies must be explicit enough for the aggregator to merge |
| 170 | streaming path, retention, inbound, outbound, SNMP observation, and |
| 171 | relationship-summary rows without domain-specific code. |
| 172 | |
| 173 | The implementation may encode these additions in the most compact shape that |
| 174 | fits the existing JSON schema style. This spec defines semantics; exact field |
| 175 | names are accepted when they are schema-valid, documented, and used uniformly |
| 176 | by Agent, UI, and aggregator. |
| 177 | |
| 178 | ## Correlation And Aggregation Model |
| 179 | |
| 180 | ### Rule Classes |
| 181 | |
| 182 | Correlation rules are declarative. The aggregator builds keys from columns and |
| 183 | literals; it does not understand domain semantics. |
| 184 | |
| 185 | Required rule classes: |
| 186 | |
| 187 | - `resolve_loose_side`: matches a loose relationship side to a known actor or |
| 188 | to a materialized partial actor. |
| 189 | - `replace_actor`: replaces weaker actors with stronger actors and rewires |
| 190 | incident links/tables. |
| 191 | - `merge_enrich_actor`: merges actors with the same identity and combines their |
| 192 | labels, attributes, links, evidence, and detail tables according to declared |
| 193 | table policies. |
| 194 | |
| 195 | ### Priority |
| 196 | |
| 197 | Rules run by ascending priority number. Exact rules must run before broader or |
| 198 | partial rules. |
| 199 | |
| 200 | Example: |
| 201 | |
| 202 | ```json |
| 203 | { |
| 204 | "rules": { |
| 205 | "socket_exact": { |
| 206 | "class": "resolve_loose_side", |
| 207 | "priority": 10, |
| 208 | "key_space": "socket", |
| 209 | "key": [ |
| 210 | { "column": "protocol" }, |
| 211 | { "literal": "|" }, |
| 212 | { "column": "address_space" }, |
| 213 | { "literal": "|" }, |
| 214 | { "column": "ip" }, |
| 215 | { "literal": ":" }, |
| 216 | { "column": "port" } |
| 217 | ], |
| 218 | "output_link_type": "socket" |
| 219 | }, |
| 220 | "ip_partial": { |
| 221 | "class": "resolve_loose_side", |
| 222 | "priority": 100, |
| 223 | "key_space": "ip", |
| 224 | "key": [ |
| 225 | { "column": "address_space" }, |
| 226 | { "literal": "|" }, |
| 227 | { "column": "ip" } |
| 228 | ], |
| 229 | "output_link_type": "partial_endpoint" |
| 230 | } |
| 231 | } |
| 232 | } |
| 233 | ``` |
| 234 | |
| 235 | ### Ambiguity |
| 236 | |
| 237 | If one point matches exactly one claim, apply the rule. |
| 238 | |
| 239 | If one point matches multiple claims with the same priority, the aggregator must |
| 240 | not pick randomly. It keeps the point unresolved or materializes a partial actor |
| 241 | according to the rule, and records a diagnostic. |
| 242 | |
| 243 | If no match exists, the point or loose side remains unresolved and may be |
| 244 | materialized for UI display according to the declared materialization policy. |
| 245 | |
| 246 | ### Alias And NAT Evidence |
| 247 | |
| 248 | NAT, load balancer, or alias information is modeled as additional keys for the |
| 249 | same point or claim. Alias rows add match possibilities; they do not mutate or |
| 250 | delete the original observation. |
| 251 | |
| 252 | ## Table Merge Policies |
| 253 | |
| 254 | Every table type that can cross producer boundaries needs a merge policy. |
| 255 | |
| 256 | Required dimensions: |
| 257 | |
| 258 | - `key`: columns that identify equivalent rows. |
| 259 | - `action`: one of `deduplicate`, `append`, `set_union`, `merge_metrics`, |
| 260 | `latest`, or `preserve`. |
| 261 | - `metrics`: per numeric column merge operation, such as `sum`, `min`, `max`, |
| 262 | `avg_weighted`, or `latest`. |
| 263 | - `conflicts`: how non-key scalar conflicts are handled. Allowed policies are |
| 264 | `prefer_claim`, `prefer_newest_agent`, `preserve_all`, or `diagnostic`. |
| 265 | |
| 266 | General rules: |
| 267 | |
| 268 | - Relationship evidence normally uses `append` or `deduplicate`. |
| 269 | - Relationship summaries normally use `merge_metrics` on the declared key. |
| 270 | - Actor labels use `set_union` keyed by actor, key, value, source, kind, and |
| 271 | value_index. |
| 272 | - Streaming retention uses `preserve` or `append` when two parents retain data |
| 273 | for the same node because each retaining parent is meaningful. |
| 274 | - Raw JSON columns cannot participate in key construction unless a scalar JSON |
| 275 | path is explicitly declared in the table policy. Prefer typed scalar columns. |
| 276 | |
| 277 | ## Layer Responsibilities |
| 278 | |
| 279 | ### Agent Direct View |
| 280 | |
| 281 | The Agent returns the producer's local topology. |
| 282 | |
| 283 | - If detailed and aggregated modes are meaningful, the Agent may expose |
| 284 | `__topology_mode`. |
| 285 | - Detailed mode keeps the finest useful evidence. |
| 286 | - Aggregated mode returns a readable local graph and modal relationship |
| 287 | summaries. |
| 288 | - If modes are identical, the Agent should not expose a mode option. |
| 289 | |
| 290 | ### Aggregator View |
| 291 | |
| 292 | The aggregator always collects the highest-detail useful input it can get. |
| 293 | |
| 294 | - Requested detailed output: |
| 295 | - fan out detailed when supported; |
| 296 | - correlate and enrich; |
| 297 | - return detailed rows with resolved actors where exact matches exist; |
| 298 | - keep unresolved loose sides/materialized actors when no match exists. |
| 299 | - Requested aggregated output: |
| 300 | - fan out detailed when supported; |
| 301 | - correlate and enrich first; |
| 302 | - aggregate into compact actors, links, relationship summaries, and modal |
| 303 | tables according to schema policies. |
| 304 | |
| 305 | ### UI View |
| 306 | |
| 307 | The UI renders the payload it receives. |
| 308 | |
| 309 | - It does not run domain-specific correlation. |
| 310 | - It may materialize loose sides for direct-Agent detailed views only if the |
| 311 | payload declares the materialization policy. |
| 312 | - It renders actor modal identification from |
| 313 | `modal.labels.identification.fields[]`. |
| 314 | - It renders the full Labels tab from `actor_labels`. |
| 315 | - It renders modal tables from schema recipes and existing data planes. |
| 316 | |
| 317 | ## Network-Connections |
| 318 | |
| 319 | ### Domain Model |
| 320 | |
| 321 | A socket observation is associated with an observed process actor and two |
| 322 | dependency endpoint tuples: |
| 323 | |
| 324 | ```text |
| 325 | protocol, client_ip, client_port, server_ip, server_port, state |
| 326 | ``` |
| 327 | |
| 328 | For inbound sockets, the observed process owns the server tuple. |
| 329 | For outbound sockets, the observed process owns the client tuple. |
| 330 | For local/same-node sockets, both process actors may be known, but the emitted |
| 331 | topology direction remains client-to-server. |
| 332 | For listening sockets, there is no remote side. |
| 333 | |
| 334 | The exact remote tuple is required for cross-node correlation, but creating an |
| 335 | actor per `IP:PORT` in a single-node graph can explode actor count and force |
| 336 | layout noise. Therefore detailed mode must preserve the exact tuple without |
| 337 | requiring every tuple to be an actor. |
| 338 | |
| 339 | ### Agent Aggregated Mode |
| 340 | |
| 341 | Agent aggregated mode is a readable local process dependency map. |
| 342 | |
| 343 | Rules: |
| 344 | |
| 345 | - Every graph link has two actor references. |
| 346 | - The producer creates materialized endpoint actors for unknown peers using the |
| 347 | peer IP plus address space. |
| 348 | - Socket dependency link types use `direction_role: "dependency"` and graph |
| 349 | links point from client/dependant actor to server/dependency actor. |
| 350 | - Relationship-summary rows are collapsed by actor pair, protocol, and state. |
| 351 | They keep dependency endpoint IPs (`client_ip`, `server_ip`) and merged |
| 352 | metrics, but not per-socket ephemeral ports. |
| 353 | - Process actor size is driven by `socket_count`. |
| 354 | - Port bullets are driven by compact actor-owned port summaries, using a |
| 355 | numeric count column. |
| 356 | - Node-to-process ownership links are graph-coherence links, not networking |
| 357 | dependencies. |
| 358 | |
| 359 | Synthetic example: |
| 360 | |
| 361 | ```json |
| 362 | { |
| 363 | "view": { "mode": "aggregated" }, |
| 364 | "actors": [ |
| 365 | { "id": 1, "type": "node", "display_name": "node-a" }, |
| 366 | { "id": 2, "type": "process", "display_name": "api", "socket_count": 14 }, |
| 367 | { "id": 3, "type": "endpoint", "display_name": "198.51.100.20", "ip": "198.51.100.20" } |
| 368 | ], |
| 369 | "links": [ |
| 370 | { "src_actor": 1, "dst_actor": 2, "type": "ownership" }, |
| 371 | { "src_actor": 2, "dst_actor": 3, "type": "endpoint_socket", "protocol": "tcp", "socket_count": 12 } |
| 372 | ], |
| 373 | "tables": { |
| 374 | "relationship": { |
| 375 | "connections": [ |
| 376 | { "src_actor": 2, "dst_actor": 3, "protocol": "tcp", "client_ip": "192.0.2.10", "server_ip": "198.51.100.20", "socket_count": 12 } |
| 377 | ] |
| 378 | } |
| 379 | } |
| 380 | } |
| 381 | ``` |
| 382 | |
| 383 | The example uses row objects for readability. Production uses compact tables. |
| 384 | |
| 385 | ### Agent Detailed Mode |
| 386 | |
| 387 | Agent detailed mode preserves exact socket evidence. |
| 388 | |
| 389 | Rules: |
| 390 | |
| 391 | - Graph links still have two actor references. |
| 392 | - Unknown peers are still visible endpoint actors, grouped by peer IP plus |
| 393 | address space. |
| 394 | - Socket evidence preserves the exact client/server tuple, including ports, so |
| 395 | Cloud correlation has no tuple loss. |
| 396 | - Listening rows have no remote side and no fake remote actor. |
| 397 | - Local sockets with both processes known have two process actor refs. |
| 398 | |
| 399 | Synthetic example: |
| 400 | |
| 401 | ```json |
| 402 | { |
| 403 | "view": { "mode": "detailed" }, |
| 404 | "actors": [ |
| 405 | { "id": 1, "type": "node", "display_name": "node-a" }, |
| 406 | { "id": 2, "type": "process", "display_name": "api" }, |
| 407 | { "id": 3, "type": "endpoint", "display_name": "198.51.100.20", "ip": "198.51.100.20" } |
| 408 | ], |
| 409 | "evidence": { |
| 410 | "socket": [ |
| 411 | { |
| 412 | "src_actor": 2, |
| 413 | "dst_actor": 3, |
| 414 | "protocol": "tcp", |
| 415 | "client_ip": "192.0.2.10", |
| 416 | "client_port": 50120, |
| 417 | "server_ip": "198.51.100.20", |
| 418 | "server_port": 443 |
| 419 | } |
| 420 | ] |
| 421 | } |
| 422 | } |
| 423 | ``` |
| 424 | |
| 425 | ### Aggregator Network-Connections |
| 426 | |
| 427 | The aggregator receives detailed rows. |
| 428 | |
| 429 | Exact match: |
| 430 | |
| 431 | ```text |
| 432 | node-a api claims client tcp 192.0.2.10:50120, points at server tcp 198.51.100.20:443 |
| 433 | node-b nginx claims server tcp 198.51.100.20:443, points at client tcp 192.0.2.10:50120 |
| 434 | ``` |
| 435 | |
| 436 | Result: |
| 437 | |
| 438 | ```text |
| 439 | node-a/api -> node-b/nginx |
| 440 | ``` |
| 441 | |
| 442 | No endpoint actors remain for the exact match. The loose side was resolved to a |
| 443 | known actor and the final graph is a normal process-to-process dependency. |
| 444 | |
| 445 | Partial match: |
| 446 | |
| 447 | ```text |
| 448 | node-a api outbound -> 198.51.100.20:443 |
| 449 | node-b is known to own 198.51.100.20 |
| 450 | node-b has no matching process/socket row at collection time |
| 451 | ``` |
| 452 | |
| 453 | Result: |
| 454 | |
| 455 | ```text |
| 456 | node-a/api -> node-b/[materialized endpoint for 198.51.100.20] |
| 457 | ``` |
| 458 | |
| 459 | The graph remains truthful: the dependency points at node-b, but the exact |
| 460 | process could not be proven. |
| 461 | |
| 462 | No match: |
| 463 | |
| 464 | ```text |
| 465 | node-a/api -> materialized endpoint 198.51.100.20 |
| 466 | ``` |
| 467 | |
| 468 | The unresolved endpoint remains visible with presentation that clearly differs |
| 469 | from resolved process links. |
| 470 | |
| 471 | ### UI Network-Connections |
| 472 | |
| 473 | Direct Agent aggregated: |
| 474 | |
| 475 | - Show process and endpoint actors. |
| 476 | - Show two-sided graph links. |
| 477 | - Non-node actor modals show `Dependencies` from relationship-summary rows |
| 478 | where the selected actor is `src_actor`, and `Dependants` where it is |
| 479 | `dst_actor`. |
| 480 | |
| 481 | Direct Agent detailed: |
| 482 | |
| 483 | - Show known actors. |
| 484 | - Show visible endpoint actors for unknown peers, grouped by peer IP and address |
| 485 | space. |
| 486 | - Non-node actor modals show `Dependencies` and `Dependants` from exact socket |
| 487 | evidence using the same `src_actor` / `dst_actor` split. |
| 488 | |
| 489 | Aggregator aggregated: |
| 490 | |
| 491 | - Show the post-correlation compact dependency map. |
| 492 | - Exact cross-node matches are process-to-process. |
| 493 | - Unmatched loose sides are materialized according to policy. |
| 494 | |
| 495 | Aggregator detailed: |
| 496 | |
| 497 | - Show exact socket evidence with resolved actors where possible. |
| 498 | - Unresolved rows retain their loose-side facts. |
| 499 | |
| 500 | ## SNMP/L2 |
| 501 | |
| 502 | ### Domain Model |
| 503 | |
| 504 | SNMP/L2 topology observations are device, interface, neighbor, forwarding, |
| 505 | ARP, bridge, VLAN, and protocol facts. A graph link represents an observed or |
| 506 | inferred L2 relationship between two actors. |
| 507 | |
| 508 | SNMP is not a loose-side topology. Every link should have two actors in both |
| 509 | Agent and aggregator views. |
| 510 | |
| 511 | ### Mode Behavior |
| 512 | |
| 513 | SNMP/L2 detailed and aggregated modes are currently a no-op. The producer should |
| 514 | not expose `__topology_mode` until it has a real lower/higher-grain distinction. |
| 515 | |
| 516 | If a global Cloud topology request asks for aggregated output, the aggregator |
| 517 | still consumes the same SNMP payload and returns the same semantic grain after |
| 518 | correlation/replacement. |
| 519 | |
| 520 | ### Correlation Behavior |
| 521 | |
| 522 | SNMP mainly uses actor replacement. |
| 523 | |
| 524 | Examples: |
| 525 | |
| 526 | - A managed device actor is stronger than an LLDP remote placeholder that has |
| 527 | the same chassis id. |
| 528 | - A managed interface actor is stronger than an inferred endpoint that has the |
| 529 | same MAC/interface identity. |
| 530 | - A discovered management IP can help match a placeholder to a managed device, |
| 531 | but ambiguous matches must not be chosen randomly. |
| 532 | |
| 533 | Replacement example: |
| 534 | |
| 535 | ```text |
| 536 | payload-a: switch-a port 10 -> lldp-remote(chassis=aa:bb:cc) |
| 537 | payload-b: managed-switch-b(chassis=aa:bb:cc) |
| 538 | ``` |
| 539 | |
| 540 | Result: |
| 541 | |
| 542 | ```text |
| 543 | switch-a port 10 -> managed-switch-b |
| 544 | ``` |
| 545 | |
| 546 | The weaker LLDP remote actor is removed from the aggregated output and its |
| 547 | incident links/tables are rewired to the managed device actor. |
| 548 | |
| 549 | ### UI SNMP/L2 |
| 550 | |
| 551 | The UI should not expose a detailed/aggregated toggle for SNMP/L2 unless the |
| 552 | payload declares supported modes. |
| 553 | |
| 554 | Device modals should remain port-centric: |
| 555 | |
| 556 | - actor identification: device name, management IP, vendor, model, role, and |
| 557 | other selected labels; |
| 558 | - full labels tab: all labels; |
| 559 | - ports table: one row per known interface/port, with SNMP `if_index` as the |
| 560 | visible real numeric port ID when known, and the port name; |
| 561 | - expanded port rows: show a clickable neighbor actor and neighbor port name |
| 562 | when graph-link facts can align the port to a remote actor; |
| 563 | - links/neighbor information: derived from the same port rows or aligned |
| 564 | relationship rows so local port identity never contradicts the port table. |
| 565 | |
| 566 | ## Streaming |
| 567 | |
| 568 | ### Domain Model |
| 569 | |
| 570 | Streaming topology describes Netdata Agent streaming relationships: |
| 571 | |
| 572 | ```text |
| 573 | child -> parent |
| 574 | parent <-> parent |
| 575 | virtual/stale/remote nodes represented as actors |
| 576 | ``` |
| 577 | |
| 578 | All actors are real topology actors from the streaming view. Links always have |
| 579 | two actor refs. Streaming is not a loose-side topology. |
| 580 | |
| 581 | ### Mode Behavior |
| 582 | |
| 583 | Streaming detailed and aggregated modes are currently a no-op. The producer |
| 584 | should not expose `__topology_mode` until it has a real lower/higher-grain |
| 585 | distinction. |
| 586 | |
| 587 | The aggregator still consumes the same streaming payload for global aggregated |
| 588 | requests and returns merged/enriched streaming topology. |
| 589 | |
| 590 | ### Correlation Behavior |
| 591 | |
| 592 | Streaming uses actor enrichment and table merging by `machine_guid`. |
| 593 | |
| 594 | Example: |
| 595 | |
| 596 | ```text |
| 597 | child-1 -> parent-1 <-> parent-2 <- child-2 |
| 598 | ``` |
| 599 | |
| 600 | Both parents may report facts about the same node. The aggregator must not show |
| 601 | duplicate actors for the same `machine_guid`. It merges those actors and then |
| 602 | merges their tables according to table policy. |
| 603 | |
| 604 | Required merge behavior: |
| 605 | |
| 606 | - actor labels: set union; |
| 607 | - actor scalar facts: prefer non-empty, newest Agent version when explicitly |
| 608 | comparable, otherwise preserve conflicts in diagnostics or expanded labels; |
| 609 | - stream path rows: deduplicate identical path membership rows; |
| 610 | - retention rows: preserve each retaining parent/source row, because multiple |
| 611 | parents retaining the same child are meaningful; |
| 612 | - inbound stream rows: merge by parent actor, child actor, immediate source |
| 613 | actor when known, and relationship type, with numeric metrics merged by table |
| 614 | policy; |
| 615 | - outbound stream rows: merge by sending parent actor, streamed node actor, |
| 616 | destination actor when known, and stream state, with numeric metrics merged by |
| 617 | table policy; |
| 618 | - links: merge by source actor, destination actor, type, protocol, and state, |
| 619 | then merge metrics/evidence according to link type policy. |
| 620 | |
| 621 | Retention example: |
| 622 | |
| 623 | ```text |
| 624 | payload-parent-a: parent-a retains child-x for tier 0 |
| 625 | payload-parent-b: parent-b retains child-x for tier 0 |
| 626 | payload-child-x: child-x self retention tier 0 |
| 627 | ``` |
| 628 | |
| 629 | Result: |
| 630 | |
| 631 | ```text |
| 632 | actor child-x modal Retention table has 3 rows: |
| 633 | retaining actor parent-a |
| 634 | retaining actor parent-b |
| 635 | retaining actor child-x |
| 636 | ``` |
| 637 | |
| 638 | These rows must not be deduplicated away solely because the retained child is |
| 639 | the same. |
| 640 | |
| 641 | ### UI Streaming |
| 642 | |
| 643 | The UI should not expose a detailed/aggregated toggle for streaming unless the |
| 644 | payload declares supported modes. |
| 645 | |
| 646 | Actor modal identification should show selected labels such as role, hostname, |
| 647 | machine GUID when useful, and stream status. The full Labels tab remains |
| 648 | complete. |
| 649 | |
| 650 | Tables: |
| 651 | |
| 652 | - stream path: deduplicated path rows for the selected actor only. Timestamps |
| 653 | must be populated for every path row when the producer or aggregator can |
| 654 | derive them; synthetic rows used for highlighting are not allowed to drop |
| 655 | known timing facts; |
| 656 | - retained nodes: all nodes whose data is maintained by the selected actor, |
| 657 | using the same retention table filtered by `observer_actor`. This is the |
| 658 | default retention view in the current modal contract; |
| 659 | - received nodes: children, virtual nodes, stale nodes, and descendants |
| 660 | received or transiting through the selected parent. The immediate source must |
| 661 | be populated when known; direct local receipt should use the child/vnode actor |
| 662 | as the source instead of rendering an empty value; |
| 663 | - outbound streams: every node payload the selected parent sends upstream, |
| 664 | including self, virtual nodes, direct children, and transit descendants. Rows |
| 665 | are owned by the sending parent and must show the streamed node and the |
| 666 | destination. |
| 667 | |
| 668 | The current default modal contract does not show a separate `Retention for node` |
| 669 | section. The underlying retention table still preserves `actor` and |
| 670 | `observer_actor` so aggregated/cloud views can add an explicitly named |
| 671 | `Retained by` section later without changing the facts. |
| 672 | |
| 673 | Highlight path must use the deduplicated stream-path table, not direct sibling |
| 674 | selection only. |
| 675 | |
| 676 | ## Cross-Topology Examples |
| 677 | |
| 678 | ### Agent Detailed To Aggregator Aggregated |
| 679 | |
| 680 | Input request to Cloud: |
| 681 | |
| 682 | ```text |
| 683 | function=topology:network-connections __topology_mode=aggregated |
| 684 | ``` |
| 685 | |
| 686 | Aggregator fanout: |
| 687 | |
| 688 | ```text |
| 689 | node-a: topology:network-connections __topology_mode=detailed |
| 690 | node-b: topology:network-connections __topology_mode=detailed |
| 691 | ``` |
| 692 | |
| 693 | Aggregator work: |
| 694 | |
| 695 | 1. Decode detailed socket evidence from both nodes. |
| 696 | 2. Resolve exact loose-side socket keys. |
| 697 | 3. Materialize unresolved partial endpoints. |
| 698 | 4. Aggregate graph links and relationship summaries. |
| 699 | |
| 700 | Returned payload: |
| 701 | |
| 702 | ```json |
| 703 | { |
| 704 | "view": { "mode": "aggregated" }, |
| 705 | "actors": "... compact post-correlation actor table ...", |
| 706 | "links": "... compact post-correlation graph links ...", |
| 707 | "tables": { |
| 708 | "relationship": { |
| 709 | "connections": "... aggregated drilldown rows ..." |
| 710 | } |
| 711 | } |
| 712 | } |
| 713 | ``` |
| 714 | |
| 715 | ### Agent Detailed To UI Direct Detailed |
| 716 | |
| 717 | Input request to Agent: |
| 718 | |
| 719 | ```text |
| 720 | function=topology:network-connections __topology_mode=detailed |
| 721 | ``` |
| 722 | |
| 723 | UI work: |
| 724 | |
| 725 | 1. Decode known actors and exact socket evidence. |
| 726 | 2. Render known actors. |
| 727 | 3. Materialize loose endpoints only as declared by the payload, normally by IP. |
| 728 | 4. Show exact socket rows in process modals. |
| 729 | |
| 730 | ### SNMP Global Aggregated |
| 731 | |
| 732 | Input request to Cloud: |
| 733 | |
| 734 | ```text |
| 735 | function=topology:snmp __topology_mode=aggregated |
| 736 | ``` |
| 737 | |
| 738 | Aggregator fanout: |
| 739 | |
| 740 | ```text |
| 741 | node-a: topology:snmp |
| 742 | node-b: topology:snmp |
| 743 | ``` |
| 744 | |
| 745 | No mode parameter is sent unless the producer advertises one. Aggregator applies |
| 746 | replacement rules and returns a normal graph. |
| 747 | |
| 748 | ### Streaming Global Aggregated |
| 749 | |
| 750 | Input request to Cloud: |
| 751 | |
| 752 | ```text |
| 753 | function=topology:streaming __topology_mode=aggregated |
| 754 | ``` |
| 755 | |
| 756 | Aggregator fanout: |
| 757 | |
| 758 | ```text |
| 759 | node-a: topology:streaming |
| 760 | node-b: topology:streaming |
| 761 | ``` |
| 762 | |
| 763 | No mode parameter is sent unless the producer advertises one. Aggregator merges |
| 764 | actors by `machine_guid`, preserves retention rows by retaining source, and |
| 765 | deduplicates stream path rows by declared path identity. |
| 766 | |
| 767 | ## Open Edge Cases And Required Behavior |
| 768 | |
| 769 | - **Ambiguous network socket match**: keep unresolved or materialize partial; |
| 770 | record diagnostic; do not randomly choose a process. |
| 771 | - **Socket closed between node collections**: exact process match may fail; |
| 772 | partial IP/node match may still be valid if a node/endpoint claim exists. |
| 773 | - **NAT or load balancer aliases**: add alias keys; do not overwrite original |
| 774 | tuples. |
| 775 | - **SNMP duplicate weak actors**: replace all weak actors that match one strong |
| 776 | actor; preserve evidence and diagnostics. |
| 777 | - **SNMP weak actor matches multiple strong actors**: keep weak actor visible or |
| 778 | diagnostic; do not choose randomly. |
| 779 | - **Streaming stale node**: keep actor if it is meaningful to streaming status; |
| 780 | merge by `machine_guid` only when identities match. |
| 781 | - **Streaming retention from multiple parents**: preserve rows; do not collapse |
| 782 | them into one retained child row. |
| 783 | - **Actor label conflicts**: keep full label set; modal identification applies |
| 784 | display limits only, not data loss. |
| 785 | - **High-cardinality detailed network-connections**: detailed mode may be large; |
| 786 | aggregated mode must avoid actor-per-port explosion. |
| 787 | |
| 788 | ## Validation Requirements |
| 789 | |
| 790 | Agent: |
| 791 | |
| 792 | - Schema validation for all changed topology payloads. |
| 793 | - Network-connections fixtures for aggregated and detailed modes. |
| 794 | - SNMP fixture proving no mode selector is exposed unless behavior differs. |
| 795 | - Streaming fixture proving no mode selector is exposed unless behavior differs. |
| 796 | - Modal label identification metadata present for actor types with useful |
| 797 | labels. |
| 798 | |
| 799 | UI: |
| 800 | |
| 801 | - Decode modal label identification metadata. |
| 802 | - Render selected identification labels in actor modal header. |
| 803 | - Keep full Labels tab. |
| 804 | - Render loose-side materialized actors only from schema policy. |
| 805 | - Do not show SNMP/streaming mode toggles unless payload capability declares |
| 806 | them. |
| 807 | |
| 808 | Aggregator: |
| 809 | |
| 810 | - Rewrite `__topology_mode=aggregated` to `detailed` on fanout only when the |
| 811 | producer supports it. |
| 812 | - Consume detailed network-connections and return both detailed and aggregated |
| 813 | outputs. |
| 814 | - Resolve exact socket loose sides and preserve unresolved/partial cases. |
| 815 | - Replace SNMP weak actors with managed actors. |
| 816 | - Merge/enrich streaming actors and tables by `machine_guid` and table policy. |
| 817 | - Preserve schema-valid unknown future fields. |