master
md 435 lines 23.4 KB
Rendered Raw
1 ---
2 name: project-create-topology
3 description: Developer workflow for creating or updating Netdata topology producers and topology Function payloads using the production netdata.topology.v1 schema. Use when adding or migrating topology:network-connections, topology:streaming, topology:snmp, vSphere topology, correlation rules, graph presentation, drilldowns, direction semantics, telemetry overlays, or Cloud topology aggregation fixtures.
4 type: project
5 ---
6
7 # Create Netdata Topologies
8
9 ## What This Skill Is
10
11 This is a developer skill for assistants working in this repository. It is not
12 an end-user/operator skill. Use it when changing topology producers, schema
13 fixtures, validation, topology developer documentation, or Cloud/frontend
14 handoff artifacts.
15
16 ## Required References
17
18 Read these before designing or changing topology payloads:
19
20 | File | Purpose |
21 |---|---|
22 | `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` | JSON Schema for production topology payloads |
23 | `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` | Human-readable topology schema contract and producer guidance |
24 | `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` | Backend/frontend/aggregator migration scope |
25 | `.agents/sow/specs/topology-function-schema.md` | Durable project spec for topology semantics |
26 | `.agents/sow/specs/topology-modes-correlation-aggregation.md` | Mode, correlation, aggregation, and actor modal identification contract |
27 | `src/go/pkg/topology/v1` | Go production topology payload builders and compact-table helpers |
28 | `.agents/skills/project-writing-collectors/SKILL.md` | Collector quality, Function, validation, and cardinality rules |
29
30 For transport-level Function behavior, also read:
31
32 - `src/plugins.d/FUNCTION_UI_REFERENCE.md`
33 - `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`
34
35 ## Developer How-Tos
36
37 The how-to catalog lives under [`how-tos/`](./how-tos/). These recipes are
38 developer-facing and must stay in this project skill, not under
39 `docs/netdata-ai/skills/`.
40
41 ## Core Rules
42
43 - Production payloads carry canonical topology facts for the aggregator and UI.
44 - Go producers MUST use `src/go/pkg/topology/v1`. The non-v1 root
45 `src/go/pkg/topology` payload model has been retired and MUST NOT be
46 reintroduced for production topology payloads.
47 - Test-only projection code may reconstruct compatibility payload shapes to
48 prove parity.
49 - Never add compatibility reconstruction fields, old-schema adapter names, or
50 duplicated display strings to production payloads.
51 - Keep display composition in type-level and graph-level presentation metadata,
52 not in high-cardinality rows.
53 - Keep raw sensitive payload captures under `.local/` only.
54
55 ## Workflow
56
57 1. Define the topology purpose and scale target.
58 - Identify the graph users need: nodes, processes, containers, L2 devices,
59 vSphere inventory, streaming parents, or another domain.
60 - Estimate actor count, graph-link count, evidence-row count, and payload
61 size on realistic data.
62
63 2. Pick actors.
64 - Use stable identities.
65 - Keep display names separate from identity.
66 - Declare `identity`, `merge_identity`, and `parent_identity` in actor types.
67 - Prepare aggregation scopes such as node, process name, PID, container,
68 Kubernetes workload, SNMP device/interface, or vSphere object.
69
70 3. Pick graph links.
71 - Graph links are renderable relationship groups.
72 - Keep graph links compact.
73 - Put one-to-many observation detail in evidence sections.
74 - Define direction semantics in link types.
75 - Use distinct semantic link types for ownership, local/resolved links,
76 correlation links, inferred links, and partial links when their meaning or
77 layout behavior differs.
78
79 4. Pick evidence rows.
80 - Evidence is the lossless relationship proof.
81 - For sockets, preserve the exact matching tuple.
82 - For SNMP/L2, preserve LLDP/CDP/FDB/ARP/STP facts according to role.
83 - For streaming, keep relationship facts separate from actor-owned path data.
84 - For vSphere, preserve inventory/relationship facts using stable object IDs.
85
86 5. Classify detail tables.
87 - `actor_detail`: custom actor state, not generally aggregatable.
88 - `actor_inventory`: actor-owned inventory data.
89 - `relationship_evidence`: exact relationship rows.
90 - `relationship_summary`: derived summaries.
91 - Use `json` columns only for custom actor/detail cells that must preserve
92 nested producer-owned values; avoid them for high-cardinality evidence.
93 - Use a compact actor-owned `actor_labels` table for modal labels:
94 `actor`, `key`, `value`, optional `source`, optional `kind`, and optional
95 `value_index`.
96 - Expose complete host/node labels when available.
97 - Expose useful non-node actor labels and metadata, while keeping identity,
98 correlation, grouping, sorting, filtering, and aggregation facts as typed
99 canonical columns.
100
101 6. Define telemetry overlays.
102 - Use overlay templates once per payload or type.
103 - Links and actors carry compact refs and parameters only.
104 - Do not put full metric query payloads on every row.
105 - Build Go producer refs with `topologyv1.NewActorOverlayRefsBuilder` or
106 `topologyv1.NewLinkOverlayRefsBuilder` instead of hand-assembling compact
107 tables.
108 - Overlay refs use a `template` column, exactly one convention owner column
109 (`actor` with type `actor_ref` or `link` with type `link_ref`), and one
110 column for each selector param required by the referenced template. Do not
111 add any other `actor_ref` or `link_ref` columns to overlay refs. Every row
112 must have a non-null owner value.
113 - The `template` column and required selector-param columns must be `string`
114 or `string_ref`; required selector-param row values must resolve to
115 non-empty strings.
116 - Do not use `template`, `actor`, or `link` as selector params; those names
117 are reserved refs-table convention columns.
118 - For `netdata.metrics`, `node_id` means node scope, `collect_job` maps to
119 chart label `_collect_job`, and other selector params map to same-named
120 chart labels. In go.d producers, pass `job.Name()` for `collect_job`; do
121 not use `job.FullName()`.
122
123 7. Define correlation semantics when actors can be resolved across payloads.
124 - Declare whether the topology needs loose-side resolution, actor
125 replacement, actor enrichment, or visible correlation actors.
126 - Do not hide correlation state as flags on real actors.
127 - Define `data.correlation.rules` with declarative key templates,
128 priorities, `class`, `absorb` or `link` actions, point actor types when
129 visible correlation actors exist, optional claim actor types, correlation
130 link types, and output link types.
131 - Emit compact `data.correlation.points` rows for visible correlation actors
132 when the input graph has them, and `data.correlation.claims` rows for real
133 actors that can satisfy keys.
134 - For high-cardinality exact observations, prefer loose relationship-side
135 facts plus declared materialization policy over creating one actor per
136 ephemeral endpoint.
137 - Use `absorb` only for exact matches that should remove correlation actors
138 or loose-side placeholders from the aggregated output.
139 - Use `link` for broader or partial matches that should keep the correlation
140 actor or materialized partial actor visible.
141 - Use `replace_actor` semantics for weaker placeholder actors that should be
142 replaced by stronger managed actors.
143 - Use `merge_enrich_actor` semantics when multiple payloads provide
144 complementary facts for the same actor identity.
145 - Keep NAT or alias information as additional point/claim rows, not as
146 mutation of the original observation.
147
148 8. Define graph presentation.
149 - Put actor presentation in `types.actor_types.<id>.presentation`.
150 - Put link presentation in `types.link_types.<id>.presentation`.
151 - Put graph port-bullet presentation in `types.port_types.<id>.presentation`.
152 - Put legend, actor-click highlight behavior, port fields, and scale keys in
153 `data.presentation`.
154 - Use `__topology_mode` for detailed vs aggregated topology requests when a
155 producer has a real mode difference. Do not expose a mode selector for
156 mode-invariant topologies.
157 - Use UI-owned color/icon/line/width/opacity/layout tokens only.
158 - Define `label_policy.columns` with safe scalar display columns; never let
159 canonical identity arrays become actor names.
160 - Define `search.columns[]` and/or `search.label_keys[]` for searchable
161 actors. Set `search.enabled: false` for helper actors that should not
162 appear in graph search. Do not rely on UI hardcoded `details`, `match`, or
163 `attributes` paths.
164 - Define `presentation.size.scale` when an actor type needs fixed visual
165 emphasis, and `presentation.layout.repulsion` when an actor type needs
166 relative force-graph separation. Do not emit raw force numbers.
167 - Define `link_types.<id>.semantic_role` when behavior depends on link
168 meaning, such as `discovery`, `ownership`, `traffic`, `correlation`, or
169 `control`. Do not make the UI infer this from link type names or protocol
170 strings.
171 - Keep `presentation.arrow` authoritative for arrows. Omitted or `auto`
172 derives no arrows for `undirected`, `observed_bidirectional`, `none`, or
173 `observation`; derives `forward` for directed `flow`/`dependency` and
174 hierarchical `ownership`. Use explicit `reverse` or `both` when needed.
175 `direction_role` is required; never rely on `orientation: directed` alone
176 to infer arrows.
177 - Define `ports.sources[]` whenever an actor type sets
178 `ports.show_bullets: true`.
179 - Use scalar display columns for `ports.sources[].name_column`; do not use
180 refs, arrays, or JSON as graph bullet labels.
181 - Use numeric `ports.sources[].value_column` when one compact row represents
182 multiple observations and the UI should size or count bullets by the sum.
183 - Use at most one variable visual channel per link type, keyed by
184 `variable.scale_key` and sourced from one raw numeric `value_column`.
185 - Use `presentation.layout.strength` tokens `weakest`, `weaker`, `normal`,
186 `stronger`, `strongest`, and `presentation.layout.distance` tokens
187 `closest`, `closer`, `normal`, `farther`, `farthest`; do not emit numeric
188 force values.
189 - Current producer tuning keeps `presentation.layout.strength` at `normal`
190 and varies only `presentation.layout.distance` where semantic separation is
191 needed. Do not emit non-normal strength tokens for graph polish unless a
192 later product decision explicitly re-enables force-strength tuning.
193 - Use only closed icon tokens. Do not emit raw SVG or depend on frontend
194 capability-string icon inference; add a schema/UI icon token first.
195 - Missing v1 `size.scale`, `layout.repulsion`, and `search` use neutral
196 defaults. Do not expect the UI to preserve legacy self/device/SNMP/
197 endpoint heuristics for v1.
198
199 9. Define modal/table composition.
200 - Put actor modal recipes in
201 `types.actor_types.<id>.presentation.modal`.
202 - Put link modal recipes in `types.link_types.<id>.presentation.modal`.
203 - Put reusable table defaults in `types.table_types.<id>.presentation`.
204 - Use `modal.labels.identification.fields[]` to choose the small set of
205 actor labels that should appear in the actor modal identification/header
206 area. The full `actor_labels` table remains the Labels tab.
207 - Modal sections must select from existing `actors`, `links`, `evidence`,
208 `actor_table`, or `relationship_table` sources.
209 - Do not duplicate evidence or actor metadata only to populate a modal.
210 - Use projections for display: direct column, actor-ref label, opposite
211 actor, formatted endpoint, selected-side endpoint, label lookup,
212 coalesce, const, or explicit scalar JSON path.
213 - For `selected_side_endpoint`, include source/destination actor-ref
214 columns and both endpoint sides in the projection so the UI can choose the
215 side from the selected actor without hardcoded table knowledge.
216 - For `label_lookup`, provide `label_key`; provide `actor_column` only when
217 the lookup should read labels for an actor referenced by the source row
218 instead of the selected modal actor.
219 - For `json_path`, provide both the JSON `column` and scalar `path`.
220 - Use cell types: text, number, badge, actor_link, timestamp, duration,
221 endpoint, array_count, or debug_json.
222 - Use visibility values: table, expanded, hidden, or debug.
223 - Raw `json` is debug-only unless a schema-declared scalar projection gives
224 the UI/aggregator semantics.
225 - Treat Function `info` responses as metadata only. Validate full topology
226 responses against `FUNCTION_TOPOLOGY_SCHEMA.json`; do not require
227 metadata-only `info` responses to carry `data`.
228
229 10. Encode large sections as compact tables.
230 - Use `const` for constant columns.
231 - Use `dict` for low/medium-cardinality repeated values.
232 - Use `values` only when values are high-cardinality.
233 - Prefer dictionary references for strings.
234 - For Go producers, use `src/go/pkg/topology/v1` compact-table helpers
235 instead of hand-building table JSON.
236
237 11. Validate and measure.
238 - Validate JSON with `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
239 - Add semantic validation fixtures.
240 - Measure raw and gzip size on realistic data.
241 - Fail explicitly on size/row limits; never silently truncate.
242 - For topology row limits, count rows as `max(actor rows, link rows)` so
243 valid actor-only payloads are not rejected.
244
245 ## Direction Rules
246
247 - `directed` + `flow`: sockets, traffic, request dependencies.
248 - `directed` + `dependency`: logical dependency direction.
249 - `hierarchical` + `ownership`: parent/child, host/VM, cluster/host.
250 - `undirected` + `none`: physical adjacency with no direction.
251 - `observed_bidirectional` + `observation`: discovery saw one or both sides,
252 but direction is not user-facing dependency.
253
254 If direction is noise, mark it so the aggregator can merge independently of
255 direction.
256
257 ## Network-Connections Correlation Shape
258
259 Network-connections uses three graph-link families:
260
261 - node-to-process ownership links;
262 - resolved process-to-process socket links;
263 - process-to-endpoint socket relationships for unresolved or cross-node
264 endpoint tuples.
265
266 Network-connections dependency direction is client-to-server. Use
267 `direction_role: "dependency"` for socket dependency link types. Emit
268 `src_actor` as the client/dependant and `dst_actor` as the server/dependency
269 target. Do not expose `local` as a topology socket direction; same-node sockets
270 still become inbound or outbound dependency rows based on which side is the
271 client.
272
273 Use distinct presentation for each family:
274
275 - `endpoint_socket`: solid, colored, thin, normal-strength, normal-distance unresolved
276 endpoint dependency links;
277 - `correlated_socket`: solid, colored, thin, normal-strength, farthest aggregator
278 output links after exact endpoint absorption;
279 - `socket`: gray, thin, normal-strength, normal-distance local process links, optionally
280 variable by `socket_count`;
281 - `ownership`: dotted, faded/dim, thin, normal-strength, normal-distance graph-coherence links.
282
283 In aggregated mode, do not enable process port bullets from detailed socket
284 evidence. Emit a compact actor inventory table such as `socket_ports` with
285 `actor`, `port`, and numeric `socket_count`, point the process actor
286 `ports.sources[]` at it with `value_column: "socket_count"`, and size process
287 actors with `size.mode: "metric"` over actor row `socket_count`.
288
289 For network-connections actor modals:
290
291 - self/node actors show a `Processes` section from `links` filtered to
292 `type == ownership`;
293 - non-node actors show `Dependencies` where the selected actor is `src_actor`
294 and `Dependants` where the selected actor is `dst_actor`;
295 - aggregated mode uses `tables.relationship.connections`;
296 - detailed mode uses `evidence.socket`;
297 - `socket_ports` stays an actor inventory for graph port bullets, not a normal
298 modal tab;
299 - secondary socket metrics belong in `visibility: "expanded"` columns instead
300 of separate duplicate sections.
301
302 For socket correlation:
303
304 - process actors emit claim rows for the socket tuple they own: client tuple
305 for outbound observations, server tuple for inbound observations;
306 - visible endpoint/correlation actors emit point rows when the producer
307 materializes them;
308 - the `socket_exact` rule uses `class: resolve_loose_side` and
309 `action: absorb`;
310 - the key is declarative, typically protocol + address space + IP + port;
311 - `endpoint_socket` links are normal-strength/normal-distance visible links before
312 aggregation;
313 - `correlated_socket` is the farthest output link type after exact absorption.
314
315 ## Streaming Modal Rules
316
317 For `topology:streaming` actor modals:
318
319 - Size parent actors from the actor row `retained_node_count` metric, not from
320 graph degree or direct child count. Emit `presentation.size.mode: "metric"`
321 and `presentation.size.metric_column: "retained_node_count"` for the parent
322 actor type. This count represents nodes for which the parent has retained
323 data, including self, virtual nodes, stale nodes, and transit descendants
324 when they have DB retention state.
325 - Attach parent graph bullets to the parent side of incoming streaming links.
326 For graph-link sources this means `ports.sources[].actor_column:
327 "dst_actor"` and a scalar child/node display `name_column`, such as
328 `port_name`.
329 - Keep `actor_labels`, `stream_path`, `retention`, `inbound`, and `outbound`
330 as the single source of truth. Do not duplicate rows only to populate modal
331 sections.
332 - Put important node identity/status facts in
333 `modal.labels.identification.fields[]`, backed by `actor_labels`. Typical
334 host-like keys are hostname, node type, health, stream, ingest, OS, OS
335 version, kernel, architecture, CPU, cores, RAM, virtualization, container,
336 cloud placement, and Agent version. Parent actors also include retained-node
337 count and direct child count.
338 Vnode actors should use inventory/device labels such as vnode type, vendor,
339 model, address, location, sys object id, LLDP name, and status. Keep long
340 stable identifiers such as machine GUID and node id in the full Labels tab by
341 default.
342 - Show `Stream path` from `stream_path` filtered by `actor`, ordered by
343 `path_index`. This is only the selected actor's own path; child and virtual
344 node paths belong to their own actors. Do not emit blank `since` or
345 `first_time` values for synthetic path rows when those timestamps can be
346 derived from adjacent path, ingest, or DB status.
347 - Show `Retained nodes` from the `retention` table filtered by
348 `observer_actor`; this answers which nodes' data the selected actor
349 maintains. Include self, virtual nodes, direct children, transit descendants,
350 and stale/archived hosts when present in the Agent root index. Preserve
351 `db_from` and `db_to` whenever the DB status knows the range.
352 - Show `Received nodes` from `inbound` filtered by `parent_actor`; this table
353 represents children, virtual nodes, stale nodes, and descendants received or
354 transiting through the selected parent. Populate `source_actor` whenever the
355 immediate sending actor is known; for direct local receipt, use the child or
356 virtual-node actor instead of leaving the cell empty.
357 - Show `Outbound streams` from `outbound` filtered by the sending parent actor.
358 This table must list every node payload the selected parent streams upstream,
359 including self, virtual nodes, direct children, and transit descendants. Rows
360 need at least streamed node actor, destination actor when known, status, age,
361 hops, TLS, compression, and useful counts/replication metrics when available.
362 - Do not show the old `Retention for node` default section in the current modal
363 contract. Keep `actor` and `observer_actor` in the canonical retention table
364 so Cloud aggregation can preserve multiple retaining parents and a future
365 explicitly named `Retained by` section can be added without changing facts.
366
367 ## SNMP/L2 Modal Rules
368
369 For SNMP/L2 managed device actor modals:
370
371 - Treat the device as a collection of ports. The primary section is `Ports`
372 over `actor_ports`.
373 - Put important device facts in `modal.labels.identification.fields[]`, backed
374 by `actor_labels`. Typical keys are display name, management IP, vendor,
375 model, port counts, and LLDP/CDP neighbor counts.
376 - Expose real port identity as typed `actor_ports` columns: SNMP `if_index` as
377 the visible numeric port ID when known, source `port_id`, display `name`,
378 `if_name`, `if_descr`, `if_alias`, MAC, speed, status, mode, role, VLAN, FDB,
379 link, and neighbor counts.
380 - Do not fabricate numeric port IDs. Do not derive port identity from row order
381 or any generated sequence; `if_index` must come from device/SNMP facts.
382 - Include compact expanded-row neighbor columns such as nullable
383 `neighbor_actor` and `neighbor_port_name` when graph-link facts can align the
384 port to a remote actor.
385 - Use an actor-owned `actor_port_links` modal index for `Port Neighbors` when
386 the device modal needs remote actor, remote port, link type, evidence count,
387 confidence, inference, attachment mode, or timestamps.
388 - `actor_port_links` may carry compact side-specific refs and scalar facts, but
389 must not duplicate raw LLDP/CDP/FDB/ARP/STP evidence JSON.
390 - Keep generic graph-link `Links` sections only for endpoint, segment, or
391 custom actors that do not own port inventory.
392 - Build link endpoint port labels only from real port fields: `port_name`,
393 `if_name`, `if_descr`, or source `port_id`. Never use actor labels such as
394 `display_name` or `sys_name` as port-name fallbacks.
395
396 ## Validation Checklist
397
398 - JSON validates against the topology schema.
399 - Semantic validation covers references, compact-table row counts, dictionaries,
400 correlation rules, layout tokens, and schema-token parity.
401 - Actor identities are documented and tested.
402 - Link direction policy is documented and tested.
403 - Correlation points, claims, rules, priorities, actions, and output link types
404 are documented and tested when cross-payload resolution applies.
405 - Evidence rows can reproduce required drilldown tables.
406 - Custom actor tables have correct roles and aggregation policy.
407 - Actor labels are emitted through `actor_labels` when the producer has labels
408 or actor metadata to show.
409 - `actor_labels.key`, `actor_labels.value`, `actor_labels.source`, and
410 `actor_labels.kind` are logical string fields. Accept `string` and
411 `string_ref` encodings as equivalent when validating, aggregating, or
412 rendering topology payloads.
413 - Treat `actor_labels` as sensitive topology Function data. Preserve the source
414 Function's access-control assumptions when forwarding, aggregating, testing,
415 or documenting labels.
416 - Modal sections are recipes over existing facts and do not duplicate
417 high-cardinality evidence rows.
418 - Raw JSON columns are hidden/debug-only unless a schema-declared projection
419 renders a scalar value.
420 - Overlay templates and refs validate global template references, required
421 selector-param columns, selector-param string types and values, reserved
422 selector-param names, exactly one convention actor/link owner column, no extra
423 actor/link ref owner columns, non-null owner row values, provider/merge enum
424 membership, and link-type `overlay_templates` references.
425 - Payload size is measured on realistic or captured data.
426 - Raw sensitive captures remain under `.local/`.
427
428 Before considering `cloud-topology-service` ready, verify service-level
429 fixtures for all topology kinds covered by the schema. `network-connections` is
430 the required high-cardinality benchmark, but it is not enough by itself.
431
432 ## vSphere Coordination
433
434 The vSphere topology producer lives in a separate PR worktree. Do not edit that
435 worktree before telling the user, because another agent may be working there.