| 1 | <!-- markdownlint-disable-file MD043 --> |
| 2 | |
| 3 | # Topology Schema Implementation Scope |
| 4 | |
| 5 | This document scopes the work needed to move Netdata topology producers, |
| 6 | Cloud aggregation, and the Cloud UI to the production topology schema defined |
| 7 | in [FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md](/src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md) |
| 8 | and [FUNCTION_TOPOLOGY_SCHEMA.json](/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json). |
| 9 | |
| 10 | It is not an implementation plan for one commit. It is the work map for the |
| 11 | backend, frontend, producer, and aggregator changes. |
| 12 | |
| 13 | ## Ground Rules |
| 14 | |
| 15 | - New topology producers emit only the new schema. |
| 16 | - Superseded topology schema support is removed from Agent/backend contracts |
| 17 | and docs. |
| 18 | - Temporary compatibility support may exist only as an isolated Cloud frontend |
| 19 | adapter during Agent rollout. |
| 20 | - Production payloads carry canonical topology facts, not reconstruction |
| 21 | instructions for compatibility payloads. |
| 22 | - Actor/link modals are composed from schema-declared recipes over existing |
| 23 | actors, links, evidence, detail tables, and actor labels. Production payloads |
| 24 | must not duplicate high-cardinality rows only for modal display. |
| 25 | - Test-only reconstruction/projection code may derive older shapes to prove |
| 26 | information parity, but that code must not affect production payloads. |
| 27 | - Raw payload captures from real systems stay under `.local/` and are never |
| 28 | committed. |
| 29 | |
| 30 | ## Shared Backend Work |
| 31 | |
| 32 | ### Function Contract |
| 33 | |
| 34 | Required changes: |
| 35 | |
| 36 | - add topology validation against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`; |
| 37 | - update Function validator tooling to recognize the new topology schema; |
| 38 | - make topology Function examples and tests use the new contract; |
| 39 | - remove superseded topology-schema references from Agent/backend docs once producer |
| 40 | migration lands. |
| 41 | |
| 42 | Likely files: |
| 43 | |
| 44 | - `src/go/tools/functions-validation/` |
| 45 | - `src/plugins.d/FUNCTION_UI_REFERENCE.md` |
| 46 | - `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` |
| 47 | - `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` |
| 48 | - `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` |
| 49 | |
| 50 | ### Shared Encoding Helpers |
| 51 | |
| 52 | The schema uses compact columnar tables. Producers should not hand-roll table |
| 53 | encoding repeatedly. |
| 54 | |
| 55 | Required helpers: |
| 56 | |
| 57 | - table builder for `rows` / `columns` / `values`; |
| 58 | - codecs for `const`, `values`, and `dict`; |
| 59 | - string dictionary builder; |
| 60 | - validation checks for column/value length; |
| 61 | - deterministic sorting helpers for actors, links, and evidence rows; |
| 62 | - size measurement hooks for tests. |
| 63 | |
| 64 | Likely homes: |
| 65 | |
| 66 | - Go: `src/go/pkg/topology/v1` or `src/go/pkg/funcapi/` |
| 67 | - C: small helper module for network-viewer, or a local builder until a shared |
| 68 | C helper is justified |
| 69 | - Rust: SDK helper if a Rust topology producer is added |
| 70 | |
| 71 | ## Current Migration Inventory |
| 72 | |
| 73 | ### Agent Producers |
| 74 | |
| 75 | `topology:network-connections`: |
| 76 | |
| 77 | - producer path: `src/collectors/network-viewer.plugin/network-viewer.c`; |
| 78 | - the Function now emits `netdata.topology.v1` at |
| 79 | `src/collectors/network-viewer.plugin/network-viewer.c:2535`; |
| 80 | - the Function parses `aggregated` / `mode:aggregated` and `detailed` / |
| 81 | `mode:detailed`, with aggregated as the default, at |
| 82 | `src/collectors/network-viewer.plugin/network-viewer.c:272`; |
| 83 | - response metadata exposes the `mode` selector at |
| 84 | `src/collectors/network-viewer.plugin/network-viewer.c:1451`; |
| 85 | - actors, graph links, and optional socket evidence rows are emitted as compact |
| 86 | columnar tables at `src/collectors/network-viewer.plugin/network-viewer.c:2568`; |
| 87 | - socket evidence is emitted only in detailed mode at |
| 88 | `src/collectors/network-viewer.plugin/network-viewer.c:2571`; |
| 89 | - repeated string columns use automatic dictionary encoding when it is smaller |
| 90 | than plain values at `src/collectors/network-viewer.plugin/network-viewer.c:2041`; |
| 91 | - old-schema presentation metadata and actor-nested socket tables have been |
| 92 | removed from the Agent producer. The v1 producer now emits compact |
| 93 | graph-presentation metadata inside type definitions plus `data.presentation`. |
| 94 | Actor modal socket lists must be derived from evidence by the Cloud |
| 95 | frontend/aggregator during rollout. |
| 96 | - modal-composition producer work now emits `actor_labels`, process |
| 97 | `username`, process `cmdline`, self `local_ip_count`, socket-port inventory, |
| 98 | and modal recipes. Remaining work is integrated UI/aggregator QA. |
| 99 | |
| 100 | `topology:streaming`: |
| 101 | |
| 102 | - producer paths: `src/web/api/functions/function-topology-streaming.c` and |
| 103 | `src/streaming/stream-path.c`; |
| 104 | - the Function now emits `netdata.topology.v1` directly at |
| 105 | `src/web/api/functions/function-topology-streaming.c:1870`; |
| 106 | - actors, graph links, link evidence, and actor-detail tables are emitted as |
| 107 | compact tables at `src/web/api/functions/function-topology-streaming.c:1912`; |
| 108 | - streaming path rows are preserved as an `actor_detail` table at |
| 109 | `src/web/api/functions/function-topology-streaming.c:1255`; |
| 110 | - inbound and outbound drilldown rows are declared as relationship summaries at |
| 111 | `src/web/api/functions/function-topology-streaming.c:1259`; |
| 112 | - streaming, virtual, and stale links have explicit directed link-type metadata |
| 113 | and separate evidence type ids at |
| 114 | `src/web/api/functions/function-topology-streaming.c:1226`; |
| 115 | - modal-composition producer work now emits `actor_labels`, complete host labels |
| 116 | where available, host/system metadata labels, OS/architecture/CPU fields, link |
| 117 | metric columns, and modal recipes. Remaining streaming work is parity/UX |
| 118 | validation with the Cloud frontend and Cloud aggregator once those parallel |
| 119 | workers are ready. |
| 120 | |
| 121 | `topology:snmp`: |
| 122 | |
| 123 | - producer paths: `src/go/plugin/go.d/collector/snmp_topology/` and |
| 124 | `src/go/pkg/l2topology/`; |
| 125 | - the L2 engine builds an internal, non-payload `l2topology.Graph` projection |
| 126 | from `l2topology.Result`; |
| 127 | - the Function handler adapts the current SNMP topology snapshot to |
| 128 | `netdata.topology.v1` through |
| 129 | `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`; |
| 130 | - the old method-level Go presentation adapter has been retired; presentation |
| 131 | metadata is emitted in the v1 payload type registry and `data.presentation`; |
| 132 | - current L2 emission uses directions such as `bidirectional` and |
| 133 | `unidirectional` at `src/go/pkg/l2topology/topology_adapter_segments_builder_emit.go:60` |
| 134 | and `src/go/pkg/l2topology/topology_adapter_projection_pairs.go:230`; |
| 135 | - migration target: use `observed_bidirectional` or unordered aggregation policy |
| 136 | where discovery direction is noise, preserve LLDP/CDP/FDB/ARP/STP evidence, |
| 137 | keep interface inventory as actor detail/inventory, and move metric query |
| 138 | definitions to overlay templates/refs. |
| 139 | |
| 140 | vSphere: |
| 141 | |
| 142 | - producer path: `src/go/plugin/go.d/collector/vsphere/`; |
| 143 | - the Function emits `netdata.topology.v1` directly from the Go collector; |
| 144 | - actor identity uses the vSphere managed-object type plus managed-object id; |
| 145 | - inventory containment is modeled as hierarchical ownership links; |
| 146 | - VM-to-host and host/VM-to-network relationships are graph links with typed |
| 147 | evidence. |
| 148 | |
| 149 | ### Cloud Frontend |
| 150 | |
| 151 | The Cloud frontend compatibility work is outside this repository, but the |
| 152 | schema rollout depends on it: |
| 153 | |
| 154 | - current topology fetch normalizer decodes every topology payload through |
| 155 | `normalizeTopologyPayload(response?.data || {})` and then computes render-time |
| 156 | aggregated links at `${CLOUD_FRONTEND_REPO}/src/domains/functions/useFetch/normalizers/topology/index.js:9`; |
| 157 | - current frontend graph aggregation groups by source, target, and link type, |
| 158 | canonicalizing reverse links if already seen, at |
| 159 | `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/graphAggregation.js:58`; |
| 160 | - current actor modal code still branches on presentation table `source` values, |
| 161 | including `source === "links"`, at |
| 162 | `${CLOUD_FRONTEND_REPO}/src/domains/functions/components/topology/actorModal/index.js:286`; |
| 163 | - migration target: add a new-schema decoder for compact tables, keep old-schema |
| 164 | support isolated in one temporary adapter, derive actor drilldown relationship |
| 165 | tables from evidence rows, render actor custom tables from typed actor-detail |
| 166 | tables, and use link-type direction metadata instead of guessing from raw link |
| 167 | direction strings. |
| 168 | - zero-heuristic v1 rendering target: read actor size scale, actor repulsion, |
| 169 | actor search policy, link semantic role, and closed icon tokens from the v1 |
| 170 | type registry. Keep `isSelfNode`, `isDerivedSegmentNode`, `isDeviceNode`, |
| 171 | LLDP/CDP protocol checks, capability icon inference, and hardcoded search |
| 172 | paths inside the temporary legacy adapter only. |
| 173 | |
| 174 | ## Producer Migration Scope |
| 175 | |
| 176 | ### `topology:network-connections` |
| 177 | |
| 178 | Producer path: |
| 179 | |
| 180 | - `src/collectors/network-viewer.plugin/network-viewer.c` |
| 181 | |
| 182 | Required behavior: |
| 183 | |
| 184 | - emit actors as compact actor table rows; |
| 185 | - emit graph links as three semantic families: |
| 186 | - node-to-process ownership links that keep each node cluster together; |
| 187 | - local process-to-process links when both process endpoints are known; |
| 188 | - process-to-correlation-endpoint links for unresolved or cross-node socket |
| 189 | endpoints; |
| 190 | - emit pure correlation endpoint actors plus `data.correlation.points` and |
| 191 | `data.correlation.claims` rows for socket tuple resolution; |
| 192 | - emit one socket evidence row per socket tuple needed for cross-node matching; |
| 193 | - default to aggregated graph projection while preserving detailed evidence; |
| 194 | - support aggregation scopes prepared for node, process name, PID, container, |
| 195 | and Kubernetes workload labels as enrichment becomes available; |
| 196 | - omit compatibility per-row display strings, duplicated labels, and actor |
| 197 | modal socket tables from production payload; |
| 198 | - keep current metrics optional and separate from topology identity. |
| 199 | |
| 200 | Validation: |
| 201 | |
| 202 | - compare against captured corpus under `.local/`; |
| 203 | - prove no truncation on large socket counts; |
| 204 | - assert payload size at corpus scale; |
| 205 | - assert exact reverse-tuple matching inputs remain present. |
| 206 | |
| 207 | Current state: |
| 208 | |
| 209 | - `src/collectors/network-viewer.plugin/network-viewer.c` now emits compact |
| 210 | actor rows, graph-link rows, and optional socket evidence rows directly in |
| 211 | `netdata.topology.v1`; |
| 212 | - aggregated mode is the default and omits socket evidence from the response; |
| 213 | - detailed mode keeps socket evidence as a shared relationship-evidence table, |
| 214 | not as actor-owned duplicated modal data; |
| 215 | - link and evidence string columns choose dictionary encoding only when it |
| 216 | reduces raw payload size; |
| 217 | - PR #22496 semantic-link split and correlation endpoint/point/claim emission |
| 218 | are implemented in the Agent producer; |
| 219 | - remaining network-connections work is corpus-scale validation with captured |
| 220 | Cloud payloads and Cloud/frontend integration once the parallel workers are |
| 221 | ready. |
| 222 | |
| 223 | ### `topology:streaming` |
| 224 | |
| 225 | Producer paths: |
| 226 | |
| 227 | - `src/web/api/functions/function-topology-streaming.c` |
| 228 | - `src/streaming/stream-path.c` |
| 229 | |
| 230 | Required behavior: |
| 231 | |
| 232 | - emit streaming agents as actors; |
| 233 | - emit parent/child streaming relationships as directed dependency links; |
| 234 | - classify `stream_path` as actor detail, not relationship evidence; |
| 235 | - keep retention and relationship summaries as typed detail tables; |
| 236 | - make direction semantics explicit through link type definitions. |
| 237 | |
| 238 | Validation: |
| 239 | |
| 240 | - preserve current actor modal data through new actor-detail tables; |
| 241 | - prove graph links and custom actor tables are not conflated; |
| 242 | - use fixtures from current streaming topology tests where possible. |
| 243 | |
| 244 | Current state: |
| 245 | |
| 246 | - `src/web/api/functions/function-topology-streaming.c` now emits |
| 247 | `netdata.topology.v1` directly from the C Function; |
| 248 | - actor rows, link rows, relationship evidence, `stream_path`, `retention`, |
| 249 | `inbound`, and `outbound` tables are compact columnar sections; |
| 250 | - stale stream-path hops remain signed values instead of being coerced to |
| 251 | unsigned values; |
| 252 | - streaming, virtual, and stale graph-link types have separate evidence types |
| 253 | so link-type metadata and evidence metadata agree. |
| 254 | - streaming graph-presentation metadata is emitted inside type definitions plus |
| 255 | `data.presentation`, including highlight-path selection, legend, link styles, |
| 256 | and graph port-bullet tokens. |
| 257 | |
| 258 | ### `topology:snmp` |
| 259 | |
| 260 | Producer paths: |
| 261 | |
| 262 | - `src/go/plugin/go.d/collector/snmp_topology/` |
| 263 | - `src/go/pkg/l2topology/` |
| 264 | |
| 265 | Required behavior: |
| 266 | |
| 267 | - emit devices, interfaces, bridge domains, VLANs, and endpoints as actor rows; |
| 268 | - emit L2 adjacencies with direction policy `canonicalize_unordered` when |
| 269 | direction is discovery noise; |
| 270 | - preserve LLDP/CDP/FDB/ARP/STP facts as evidence or actor inventory depending |
| 271 | on role; |
| 272 | - move interface traffic/errors/state metric pointers to overlay templates and |
| 273 | overlay refs; |
| 274 | - avoid copying metric query fragments on every link. |
| 275 | |
| 276 | Validation: |
| 277 | |
| 278 | - reuse existing SNMP topology golden fixtures; |
| 279 | - add schema-level golden fixtures for devices, interfaces, ports, and |
| 280 | bidirectional adjacency merge; |
| 281 | - verify overlay refs can query interface metrics without recomputing topology. |
| 282 | |
| 283 | Current state: |
| 284 | |
| 285 | - initial Function payload migration is implemented through a v1 adapter in |
| 286 | `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`; |
| 287 | - L2 graph synthesis is internal to `src/go/pkg/l2topology` and uses |
| 288 | `l2topology.Graph`, not the legacy Go topology payload package; |
| 289 | - the adapter emits compact actor, link, evidence, actor metadata, and |
| 290 | actor-detail tables and preserves nested custom actor cells with `json` |
| 291 | columns where needed; |
| 292 | - modal-composition producer work now emits `actor_labels`, promoted |
| 293 | scalar/count actor fields, stable `actor_ports` rows, structured endpoint |
| 294 | evidence, modal recipes, and payload-level presentation metadata. Remaining |
| 295 | SNMP work is to migrate metric lookup fragments into first-class overlay |
| 296 | templates/refs instead of only preserving them in actor/detail data, plus |
| 297 | integrated UI/aggregator QA. |
| 298 | |
| 299 | ### vSphere Topology |
| 300 | |
| 301 | Producer path: |
| 302 | |
| 303 | - `src/go/plugin/go.d/collector/vsphere/` |
| 304 | |
| 305 | Required behavior: |
| 306 | |
| 307 | - update the vSphere topology producer to the new schema in place; |
| 308 | - use stable vSphere managed object ids as actor identity where available; |
| 309 | - model inventory containment with hierarchical ownership links; |
| 310 | - represent VM-to-host, cluster-to-host, datastore, and network relationships |
| 311 | as graph links plus typed evidence where needed; |
| 312 | - use overlay templates for refreshable utilization/state metrics. |
| 313 | |
| 314 | Current state: |
| 315 | |
| 316 | - vSphere emits `netdata.topology.v1` directly from |
| 317 | `src/go/plugin/go.d/collector/vsphere/func_topology.go`; |
| 318 | - the producer builds compact actor, link, evidence, actor-detail, and |
| 319 | `actor_labels` tables with `src/go/pkg/topology/v1`; |
| 320 | - actor identity uses vSphere managed-object type plus managed-object id; |
| 321 | - containment, VM-to-host, and network relationships have explicit link types, |
| 322 | direction roles, and evidence types; |
| 323 | - the old method-level Go presentation adapter is retired. Presentation |
| 324 | metadata lives in the v1 type registry and `data.presentation`. |
| 325 | |
| 326 | ## Cloud Frontend Scope |
| 327 | |
| 328 | Required changes: |
| 329 | |
| 330 | - add a decoder for the compact table schema; |
| 331 | - build graph nodes from the actors table; |
| 332 | - build graph edges from the links table; |
| 333 | - derive actor drilldown relationship tables from evidence rows; |
| 334 | - render actor custom tables from typed actor-detail tables; |
| 335 | - decode and execute `presentation.modal` recipes for actor/link modals; |
| 336 | - render `actor_labels` as actor labels instead of raw metadata JSON; |
| 337 | - reuse existing topology modal/table components where practical, extending |
| 338 | them for v1 projections rather than building a separate v1 table stack; |
| 339 | - use link type direction metadata to decide whether links are directed, |
| 340 | undirected, hierarchical, or observation-only; |
| 341 | - use overlay templates and refs for metric refreshes; |
| 342 | - isolate compatibility support in one temporary adapter; |
| 343 | - delete the temporary adapter after Agent rollout. |
| 344 | |
| 345 | Likely frontend areas: |
| 346 | |
| 347 | - topology payload normalizer; |
| 348 | - graph aggregation layer; |
| 349 | - actor modal tables; |
| 350 | - link details; |
| 351 | - telemetry overlay query layer; |
| 352 | - Function response version detection. |
| 353 | |
| 354 | Frontend risks: |
| 355 | |
| 356 | - decoding large columnar sections synchronously can still block the main |
| 357 | thread; use streaming, workers, or chunked decode if needed; |
| 358 | - mixed Agent versions need clear adapter selection; |
| 359 | - actor modal tables must not duplicate evidence in memory unnecessarily. |
| 360 | - v1 actor modals can regress visually if they bypass the existing table, |
| 361 | port-table, labels, and navigation components. Component reuse is part of the |
| 362 | frontend migration, not just a cleanup preference. |
| 363 | |
| 364 | ## Cloud Aggregator Scope |
| 365 | |
| 366 | The aggregator should be implemented in Go as a separate Cloud component or |
| 367 | service, not inside charts-service request routing. |
| 368 | |
| 369 | The MVP aggregator must support all topology kinds covered by the production |
| 370 | schema contract. `topology:network-connections` remains the required |
| 371 | high-cardinality benchmark, but it is not an acceptable production boundary by |
| 372 | itself. The Cloud UI should not need separate aggregation paths for different |
| 373 | topology kinds. |
| 374 | |
| 375 | ### Inputs |
| 376 | |
| 377 | - one or more `netdata.topology.v1` payloads; |
| 378 | - requested aggregation scope, such as node, process name, container, |
| 379 | Kubernetes workload labels, vSphere object type, or SNMP device/interface; |
| 380 | - optional filters such as layer, link type, actor type, room, or node set. |
| 381 | |
| 382 | ### Outputs |
| 383 | |
| 384 | - a `netdata.topology.v1` payload with: |
| 385 | - merged actor rows; |
| 386 | - merged graph links; |
| 387 | - resolved correlation output as normal actors and links, with no exposed |
| 388 | aggregator internal states; |
| 389 | - preserved or counted evidence rows according to schema policy; |
| 390 | - merged detail tables according to table type policy; |
| 391 | - preserved and remapped modal/table presentation recipes; |
| 392 | - merged actor labels according to actor table policy; |
| 393 | - merged overlay refs according to overlay template policy; |
| 394 | - stats describing input rows, output rows, evidence rows, and drops/errors. |
| 395 | |
| 396 | ### Core Packages |
| 397 | |
| 398 | Suggested package split: |
| 399 | |
| 400 | - `schema`: generated or hand-written Go structs for the topology schema; |
| 401 | - `codec`: compact table decode/encode helpers; |
| 402 | - `model`: canonical in-memory actors, links, evidence, tables, overlays; |
| 403 | - `aggregate`: scope-based actor/link/evidence merge logic; |
| 404 | - `match`: declarative correlation-key normalization, priority handling, exact |
| 405 | and partial match resolution, and exact tuple matching; |
| 406 | - `validate`: schema and semantic validation; |
| 407 | - `fixtures`: sanitized corpus and synthetic scale fixtures. |
| 408 | |
| 409 | ### Aggregation Logic |
| 410 | |
| 411 | Required behavior: |
| 412 | |
| 413 | - merge actors by the requested scope and actor type identity; |
| 414 | - apply `data.correlation.rules` without hardcoding topology-kind-specific key |
| 415 | names in the aggregator; |
| 416 | - remove pure correlation actors only for exact unambiguous `absorb` matches, |
| 417 | rewiring incident correlation links to the matched actor with the rule's |
| 418 | `output_link_type`; |
| 419 | - keep correlation actors visible for no-match, ambiguous, and `link` partial |
| 420 | matches, emitting weak semantic correlation links for visible partial matches; |
| 421 | - preserve evidence rows when evidence policy is `preserve`; |
| 422 | - count evidence rows when evidence policy is `count`; |
| 423 | - preserve modal composition definitions and rewrite their type, table, |
| 424 | evidence, and column references after namespacing/deduplication; |
| 425 | - do not materialize modal rows during aggregation unless the underlying |
| 426 | canonical table is already being merged; |
| 427 | - merge `actor_labels` after actor reference remapping and preserve repeated |
| 428 | values as repeated rows; `string` and `string_ref` label columns are |
| 429 | equivalent logical strings and must be normalized before label deduplication; |
| 430 | - never silently truncate evidence; |
| 431 | - fail explicitly when a requested payload would exceed configured limits; |
| 432 | - canonicalize undirected links only when link type policy allows it; |
| 433 | - preserve directed links when direction is flow, dependency, or ownership; |
| 434 | - merge overlay refs with `set` or `append` semantics defined by templates. |
| 435 | |
| 436 | Network socket matching: |
| 437 | |
| 438 | - exact reverse-tuple matching should be expressed through the generic |
| 439 | correlation contract using process claims, endpoint points, correlation link |
| 440 | types, rule priorities, and output link types; |
| 441 | - NAT, load balancer, and proxy inference are out of scope for the first |
| 442 | aggregator, but later NAT evidence can add extra point/claim rows for the |
| 443 | same rule without changing the aggregator's key-building mechanism; |
| 444 | - unresolved endpoints can aggregate by visible endpoint identity, but the |
| 445 | evidence row must remain available when the requested mode preserves it. |
| 446 | |
| 447 | ### Limits And Failure Behavior |
| 448 | |
| 449 | The aggregator must have explicit limits: |
| 450 | |
| 451 | - maximum decoded bytes; |
| 452 | - maximum actor rows; |
| 453 | - maximum graph links; |
| 454 | - maximum evidence rows; |
| 455 | - maximum output bytes; |
| 456 | - maximum CPU time per request. |
| 457 | |
| 458 | If a limit is exceeded: |
| 459 | |
| 460 | - return a structured error; |
| 461 | - include stats showing which limit was exceeded; |
| 462 | - do not return a truncated topology as if it were complete. |
| 463 | |
| 464 | Paged or chunked evidence transport remains a phase-2 option. Phase 1 should |
| 465 | make payloads small enough that this is rarely needed. |
| 466 | |
| 467 | ### Tests |
| 468 | |
| 469 | Required test classes: |
| 470 | |
| 471 | - schema decode/encode round-trip; |
| 472 | - semantic validation failures; |
| 473 | - actor identity merge by scope; |
| 474 | - directed vs undirected link aggregation; |
| 475 | - relationship evidence preservation; |
| 476 | - actor-detail table aggregation; |
| 477 | - actor-label table aggregation; |
| 478 | - modal presentation recipe preservation and reference rewriting; |
| 479 | - overlay ref merge; |
| 480 | - network socket exact reverse-tuple matching; |
| 481 | - streaming hierarchy and actor-detail custom tables; |
| 482 | - SNMP/L2 unordered adjacency and observation evidence; |
| 483 | - vSphere ownership/dependency topology; |
| 484 | - generic schema-conformant custom topology passthrough; |
| 485 | - synthetic scale benchmark near and above current corpus scale; |
| 486 | - sanitized real-corpus replay from `.local/` promoted only as non-sensitive |
| 487 | fixtures when safe. |
| 488 | |
| 489 | ## Rollout Plan |
| 490 | |
| 491 | 1. Land schema docs, developer project skill, and implementation scope. |
| 492 | 2. Add validator support and compact-table helpers. |
| 493 | 3. Add Cloud frontend new-schema decoder and temporary compatibility adapter so |
| 494 | mixed Agent rollout is safe before producers emit the new schema broadly. |
| 495 | 4. Migrate producers behind tests. `topology:network-connections` should be the |
| 496 | first high-cardinality producer exercised internally, but it is not the |
| 497 | production boundary for Cloud aggregation. |
| 498 | 5. Migrate the streaming producer and complete SNMP overlay-template |
| 499 | refinement. |
| 500 | 6. Coordinate and migrate the vSphere topology producer. |
| 501 | 7. Build `cloud-topology-service` in parallel against fixtures and |
| 502 | new-schema payloads. Its MVP is complete only when all topology kinds covered |
| 503 | by this contract pass service-level aggregation tests. |
| 504 | 8. Hand final service ownership, environment-specific Helm values, deployment |
| 505 | targets, and production node-instance routing strategy to Cloud backend and |
| 506 | DevOps once the service is otherwise ready for operational integration. |
| 507 | 9. Remove compatibility support from Cloud frontend after supported Agent rollout. |
| 508 | |
| 509 | ## Resolved Phase-1 Defaults |
| 510 | |
| 511 | - Cloud aggregator service repository: `cloud-topology-service`. |
| 512 | - Cloud aggregated topology route: `POST /api/v3/spaces/{spaceID}/rooms/{roomID}/topology`. |
| 513 | - Cloud service contract: accepts and emits only `netdata.topology.v1`. |
| 514 | - Phase-1 topology service MVP: all topology kinds covered by this contract, |
| 515 | not only `topology:network-connections`. |
| 516 | - Network socket snapshot metrics such as RTT and retransmissions: opt-in, not |
| 517 | default core topology columns. |
| 518 | - Cloud-side topology payload cache: no payload cache in phase 1; aggregate on |
| 519 | demand and collect request-cost metrics first. |
| 520 | - Service-local validation package is sufficient for the Cloud service MVP; |
| 521 | producer CI may still add a separate validator binary later if needed. |
| 522 | |
| 523 | ## External Integration Gates |
| 524 | |
| 525 | These items cannot be safely invented from this repository and must be handed |
| 526 | to Cloud backend and DevOps when `cloud-topology-service` is otherwise ready for |
| 527 | operational integration: |
| 528 | |
| 529 | - final service owner and CODEOWNERS entries; |
| 530 | - environment-specific Helm values and deployment targets; |
| 531 | - approved production node-instance routing strategy. |