@cryptotaxi247 / netdata / commits / 8afe52d8b

Add topology v1 payload contract and producers (#22496)

Costa Tsaousis committed May 22, 2026 at 11:38 UTC 8afe52d8b18245fcd55f93e531f0f0896adcadc7
77 files changed +28674 -2788
.agents/skill-verification/query-netdata-agents/questions.md renamed
.agents/skill-verification/query-netdata-cloud/questions.md renamed
.agents/skills/integrations-lifecycle/how-tos/adding-new-integration-type.md
+3 -3
@@ -126,9 +126,9 @@ If the type has hand-authored content alongside the catalogue entries (like the
126
127 Three downstream repos may need touching, in roughly decreasing likelihood:
128
129 -- **`netdata/website`** (`~/src/netdata/website`): the daily `update-integrations.yml` workflow renders marketing cards from `integrations.json`. Cards usually appear automatically. But pages that reference the type explicitly (FAQ entries, solution pages) may need rewriting if the new type changes the story (e.g., "we don't do flows" → "we do flows natively").
130 -- **`netdata/learn`** (`~/src/netdata/learn`): no PR usually needed. Learn ingest reads `map.yaml` + the `<!--startmeta-->` markers in the generated `.md` files. Sidebar regenerates automatically. ~3 hours after netdata-repo merge.
131 -- **`netdata/dashboard/cloud-frontend`** (`~/src/dashboard/cloud-frontend`): cards/categories are mostly data-driven, but content tabs and validation scripts still assume rendered markdown strings for standard section keys. Before merging a new type, inspect the generated `integrations.js` shape against `src/domains/integrations/components/content/integration/tabs.js`, `src/components/markdown/useRenderableTree.js`, and `scripts/checkIntegrations.js`. Special UI (a dedicated tab elsewhere in the dashboard) is a separate concern.
129 +- **`netdata/website`** (`<website-repo>`): the daily `update-integrations.yml` workflow renders marketing cards from `integrations.json`. Cards usually appear automatically. But pages that reference the type explicitly (FAQ entries, solution pages) may need rewriting if the new type changes the story (e.g., "we don't do flows" → "we do flows natively").
130 +- **`netdata/learn`** (`<learn-repo>`): no PR usually needed. Learn ingest reads `map.yaml` + the `<!--startmeta-->` markers in the generated `.md` files. Sidebar regenerates automatically. ~3 hours after netdata-repo merge.
131 +- **`netdata/dashboard/cloud-frontend`** (`<cloud-frontend-repo>`): cards/categories are mostly data-driven, but content tabs and validation scripts still assume rendered markdown strings for standard section keys. Before merging a new type, inspect the generated `integrations.js` shape against `src/domains/integrations/components/content/integration/tabs.js`, `src/components/markdown/useRenderableTree.js`, and `scripts/checkIntegrations.js`. Special UI (a dedicated tab elsewhere in the dashboard) is a separate concern.
132
133 ## Verification checklist
134
.agents/skills/mirror-netdata-repos/SKILL.md
+2 -2
@@ -109,8 +109,8 @@ output, end-of-run summary). Run it on demand.
109 4. Required tools: `git` and `jq`. Install via your package
110 manager.
111 5. For Phase 2 (auto-discovery): install `gh` (the GitHub CLI)
112 - and run `gh auth login`. SSH clone access to the Netdata GitHub
113 - organization must work for clones.
112 + and run `gh auth login`. SSH clone access to GitHub for the
113 + `netdata` organization must work for clones.
114
115 ### First sync
116
.agents/skills/project-create-topology/SKILL.md new
+409
@@ -0,0 +1,409 @@
1 +---
2 +name: project-create-topology
3 +description: Developer workflow for creating or updating Netdata topology producers and topology Function payloads using the production netdata.topology.v1 schema. Use when adding or migrating topology:network-connections, topology:streaming, topology:snmp, vSphere topology, correlation rules, graph presentation, drilldowns, direction semantics, telemetry overlays, or Cloud topology aggregation fixtures.
4 +type: project
5 +---
6 +
7 +# Create Netdata Topologies
8 +
9 +## What This Skill Is
10 +
11 +This is a developer skill for assistants working in this repository. It is not
12 +an end-user/operator skill. Use it when changing topology producers, schema
13 +fixtures, validation, topology developer documentation, or Cloud/frontend
14 +handoff artifacts.
15 +
16 +## Required References
17 +
18 +Read these before designing or changing topology payloads:
19 +
20 +| File | Purpose |
21 +|---|---|
22 +| `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` | JSON Schema for production topology payloads |
23 +| `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` | Human-readable topology schema contract and producer guidance |
24 +| `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` | Backend/frontend/aggregator migration scope |
25 +| `.agents/sow/specs/topology-function-schema.md` | Durable project spec for topology semantics |
26 +| `.agents/sow/specs/topology-modes-correlation-aggregation.md` | Mode, correlation, aggregation, and actor modal identification contract |
27 +| `.agents/skills/project-writing-collectors/SKILL.md` | Collector quality, Function, validation, and cardinality rules |
28 +
29 +For transport-level Function behavior, also read:
30 +
31 +- `src/plugins.d/FUNCTION_UI_REFERENCE.md`
32 +- `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`
33 +
34 +## Developer How-Tos
35 +
36 +The how-to catalog lives under [`how-tos/`](./how-tos/). These recipes are
37 +developer-facing and must stay in this project skill, not under
38 +`docs/netdata-ai/skills/`.
39 +
40 +## Core Rules
41 +
42 +- Production payloads carry canonical topology facts for the aggregator and UI.
43 +- Test-only projection code may reconstruct compatibility payload shapes to
44 + prove parity.
45 +- Never add compatibility reconstruction fields, old-schema adapter names, or
46 + duplicated display strings to production payloads.
47 +- Keep display composition in type-level and graph-level presentation metadata,
48 + not in high-cardinality rows.
49 +- Keep raw sensitive payload captures under `.local/` only.
50 +
51 +## Workflow
52 +
53 +1. Define the topology purpose and scale target.
54 + - Identify the graph users need: nodes, processes, containers, L2 devices,
55 + vSphere inventory, streaming parents, or another domain.
56 + - Estimate actor count, graph-link count, evidence-row count, and payload
57 + size on realistic data.
58 +
59 +2. Pick actors.
60 + - Use stable identities.
61 + - Keep display names separate from identity.
62 + - Declare `identity`, `merge_identity`, and `parent_identity` in actor types.
63 + - Prepare aggregation scopes such as node, process name, PID, container,
64 + Kubernetes workload, SNMP device/interface, or vSphere object.
65 +
66 +3. Pick graph links.
67 + - Graph links are renderable relationship groups.
68 + - Keep graph links compact.
69 + - Put one-to-many observation detail in evidence sections.
70 + - Define direction semantics in link types.
71 + - Use distinct semantic link types for ownership, local/resolved links,
72 + correlation links, inferred links, and partial links when their meaning or
73 + layout behavior differs.
74 +
75 +4. Pick evidence rows.
76 + - Evidence is the lossless relationship proof.
77 + - For sockets, preserve the exact matching tuple.
78 + - For SNMP/L2, preserve LLDP/CDP/FDB/ARP/STP facts according to role.
79 + - For streaming, keep relationship facts separate from actor-owned path data.
80 + - For vSphere, preserve inventory/relationship facts using stable object IDs.
81 +
82 +5. Classify detail tables.
83 + - `actor_detail`: custom actor state, not generally aggregatable.
84 + - `actor_inventory`: actor-owned inventory data.
85 + - `relationship_evidence`: exact relationship rows.
86 + - `relationship_summary`: derived summaries.
87 + - Use `json` columns only for custom actor/detail cells that must preserve
88 + nested producer-owned values; avoid them for high-cardinality evidence.
89 + - Use a compact actor-owned `actor_labels` table for modal labels:
90 + `actor`, `key`, `value`, optional `source`, optional `kind`, and optional
91 + `value_index`.
92 + - Expose complete host/node labels when available.
93 + - Expose useful non-node actor labels and metadata, while keeping identity,
94 + correlation, grouping, sorting, filtering, and aggregation facts as typed
95 + canonical columns.
96 +
97 +6. Define telemetry overlays.
98 + - Use overlay templates once per payload or type.
99 + - Links and actors carry compact refs and parameters only.
100 + - Do not put full metric query payloads on every row.
101 +
102 +7. Define correlation semantics when actors can be resolved across payloads.
103 + - Declare whether the topology needs loose-side resolution, actor
104 + replacement, actor enrichment, or visible correlation actors.
105 + - Do not hide correlation state as flags on real actors.
106 + - Define `data.correlation.rules` with declarative key templates,
107 + priorities, `class`, `absorb` or `link` actions, point actor types when
108 + visible correlation actors exist, optional claim actor types, correlation
109 + link types, and output link types.
110 + - Emit compact `data.correlation.points` rows for visible correlation actors
111 + when the input graph has them, and `data.correlation.claims` rows for real
112 + actors that can satisfy keys.
113 + - For high-cardinality exact observations, prefer loose relationship-side
114 + facts plus declared materialization policy over creating one actor per
115 + ephemeral endpoint.
116 + - Use `absorb` only for exact matches that should remove correlation actors
117 + or loose-side placeholders from the aggregated output.
118 + - Use `link` for broader or partial matches that should keep the correlation
119 + actor or materialized partial actor visible.
120 + - Use `replace_actor` semantics for weaker placeholder actors that should be
121 + replaced by stronger managed actors.
122 + - Use `merge_enrich_actor` semantics when multiple payloads provide
123 + complementary facts for the same actor identity.
124 + - Keep NAT or alias information as additional point/claim rows, not as
125 + mutation of the original observation.
126 +
127 +8. Define graph presentation.
128 + - Put actor presentation in `types.actor_types.<id>.presentation`.
129 + - Put link presentation in `types.link_types.<id>.presentation`.
130 + - Put graph port-bullet presentation in `types.port_types.<id>.presentation`.
131 + - Put legend, actor-click highlight behavior, port fields, and scale keys in
132 + `data.presentation`.
133 + - Use `__topology_mode` for detailed vs aggregated topology requests when a
134 + producer has a real mode difference. Do not expose a mode selector for
135 + mode-invariant topologies.
136 + - Use UI-owned color/icon/line/width/opacity/layout tokens only.
137 + - Define `label_policy.columns` with safe scalar display columns; never let
138 + canonical identity arrays become actor names.
139 + - Define `search.columns[]` and/or `search.label_keys[]` for searchable
140 + actors. Set `search.enabled: false` for helper actors that should not
141 + appear in graph search. Do not rely on UI hardcoded `details`, `match`, or
142 + `attributes` paths.
143 + - Define `presentation.size.scale` when an actor type needs fixed visual
144 + emphasis, and `presentation.layout.repulsion` when an actor type needs
145 + relative force-graph separation. Do not emit raw force numbers.
146 + - Define `link_types.<id>.semantic_role` when behavior depends on link
147 + meaning, such as `discovery`, `ownership`, `traffic`, `correlation`, or
148 + `control`. Do not make the UI infer this from link type names or protocol
149 + strings.
150 + - Keep `presentation.arrow` authoritative for arrows. Omitted or `auto`
151 + derives no arrows for `undirected`, `observed_bidirectional`, `none`, or
152 + `observation`; derives `forward` for directed `flow`/`dependency` and
153 + hierarchical `ownership`. Use explicit `reverse` or `both` when needed.
154 + `direction_role` is required; never rely on `orientation: directed` alone
155 + to infer arrows.
156 + - Define `ports.sources[]` whenever an actor type sets
157 + `ports.show_bullets: true`.
158 + - Use scalar display columns for `ports.sources[].name_column`; do not use
159 + refs, arrays, or JSON as graph bullet labels.
160 + - Use numeric `ports.sources[].value_column` when one compact row represents
161 + multiple observations and the UI should size or count bullets by the sum.
162 + - Use at most one variable visual channel per link type, keyed by
163 + `variable.scale_key` and sourced from one raw numeric `value_column`.
164 + - Use `presentation.layout.strength` tokens `weakest`, `weaker`, `normal`,
165 + `stronger`, `strongest`, and `presentation.layout.distance` tokens
166 + `closest`, `closer`, `normal`, `farther`, `farthest`; do not emit numeric
167 + force values.
168 + - Current producer tuning keeps `presentation.layout.strength` at `normal`
169 + and varies only `presentation.layout.distance` where semantic separation is
170 + needed. Do not emit non-normal strength tokens for graph polish unless a
171 + later product decision explicitly re-enables force-strength tuning.
172 + - Use only closed icon tokens. Do not emit raw SVG or depend on frontend
173 + capability-string icon inference; add a schema/UI icon token first.
174 + - Missing v1 `size.scale`, `layout.repulsion`, and `search` use neutral
175 + defaults. Do not expect the UI to preserve legacy self/device/SNMP/
176 + endpoint heuristics for v1.
177 +
178 +9. Define modal/table composition.
179 + - Put actor modal recipes in
180 + `types.actor_types.<id>.presentation.modal`.
181 + - Put link modal recipes in `types.link_types.<id>.presentation.modal`.
182 + - Put reusable table defaults in `types.table_types.<id>.presentation`.
183 + - Use `modal.labels.identification.fields[]` to choose the small set of
184 + actor labels that should appear in the actor modal identification/header
185 + area. The full `actor_labels` table remains the Labels tab.
186 + - Modal sections must select from existing `actors`, `links`, `evidence`,
187 + `actor_table`, or `relationship_table` sources.
188 + - Do not duplicate evidence or actor metadata only to populate a modal.
189 + - Use projections for display: direct column, actor-ref label, opposite
190 + actor, formatted endpoint, selected-side endpoint, label lookup,
191 + coalesce, const, or explicit scalar JSON path.
192 + - For `selected_side_endpoint`, include source/destination actor-ref
193 + columns and both endpoint sides in the projection so the UI can choose the
194 + side from the selected actor without hardcoded table knowledge.
195 + - For `label_lookup`, provide `label_key`; provide `actor_column` only when
196 + the lookup should read labels for an actor referenced by the source row
197 + instead of the selected modal actor.
198 + - For `json_path`, provide both the JSON `column` and scalar `path`.
199 + - Use cell types: text, number, badge, actor_link, timestamp, duration,
200 + endpoint, array_count, or debug_json.
201 + - Use visibility values: table, expanded, hidden, or debug.
202 + - Raw `json` is debug-only unless a schema-declared scalar projection gives
203 + the UI/aggregator semantics.
204 + - Treat Function `info` responses as metadata only. Validate full topology
205 + responses against `FUNCTION_TOPOLOGY_SCHEMA.json`; do not require
206 + metadata-only `info` responses to carry `data`.
207 +
208 +10. Encode large sections as compact tables.
209 + - Use `const` for constant columns.
210 + - Use `dict` for low/medium-cardinality repeated values.
211 + - Use `values` only when values are high-cardinality.
212 + - Prefer dictionary references for strings.
213 + - For Go producers, use `src/go/pkg/topology/v1` compact-table helpers
214 + instead of hand-building table JSON.
215 +
216 +11. Validate and measure.
217 + - Validate JSON with `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
218 + - Add semantic validation fixtures.
219 + - Measure raw and gzip size on realistic data.
220 + - Fail explicitly on size/row limits; never silently truncate.
221 + - For topology row limits, count rows as `max(actor rows, link rows)` so
222 + valid actor-only payloads are not rejected.
223 +
224 +## Direction Rules
225 +
226 +- `directed` + `flow`: sockets, traffic, request dependencies.
227 +- `directed` + `dependency`: logical dependency direction.
228 +- `hierarchical` + `ownership`: parent/child, host/VM, cluster/host.
229 +- `undirected` + `none`: physical adjacency with no direction.
230 +- `observed_bidirectional` + `observation`: discovery saw one or both sides,
231 + but direction is not user-facing dependency.
232 +
233 +If direction is noise, mark it so the aggregator can merge independently of
234 +direction.
235 +
236 +## Network-Connections Correlation Shape
237 +
238 +Network-connections uses three graph-link families:
239 +
240 +- node-to-process ownership links;
241 +- resolved process-to-process socket links;
242 +- process-to-endpoint socket relationships for unresolved or cross-node
243 + endpoint tuples.
244 +
245 +Network-connections dependency direction is client-to-server. Use
246 +`direction_role: "dependency"` for socket dependency link types. Emit
247 +`src_actor` as the client/dependant and `dst_actor` as the server/dependency
248 +target. Do not expose `local` as a topology socket direction; same-node sockets
249 +still become inbound or outbound dependency rows based on which side is the
250 +client.
251 +
252 +Use distinct presentation for each family:
253 +
254 +- `endpoint_socket`: solid, colored, thin, normal-strength, normal-distance unresolved
255 + endpoint dependency links;
256 +- `correlated_socket`: solid, colored, thin, normal-strength, farthest aggregator
257 + output links after exact endpoint absorption;
258 +- `socket`: gray, thin, normal-strength, normal-distance local process links, optionally
259 + variable by `socket_count`;
260 +- `ownership`: dotted, faded/dim, thin, normal-strength, normal-distance graph-coherence links.
261 +
262 +In aggregated mode, do not enable process port bullets from detailed socket
263 +evidence. Emit a compact actor inventory table such as `socket_ports` with
264 +`actor`, `port`, and numeric `socket_count`, point the process actor
265 +`ports.sources[]` at it with `value_column: "socket_count"`, and size process
266 +actors with `size.mode: "metric"` over actor row `socket_count`.
267 +
268 +For network-connections actor modals:
269 +
270 +- self/node actors show a `Processes` section from `links` filtered to
271 + `type == ownership`;
272 +- non-node actors show `Dependencies` where the selected actor is `src_actor`
273 + and `Dependants` where the selected actor is `dst_actor`;
274 +- aggregated mode uses `tables.relationship.connections`;
275 +- detailed mode uses `evidence.socket`;
276 +- `socket_ports` stays an actor inventory for graph port bullets, not a normal
277 + modal tab;
278 +- secondary socket metrics belong in `visibility: "expanded"` columns instead
279 + of separate duplicate sections.
280 +
281 +For socket correlation:
282 +
283 +- process actors emit claim rows for the socket tuple they own: client tuple
284 + for outbound observations, server tuple for inbound observations;
285 +- visible endpoint/correlation actors emit point rows when the producer
286 + materializes them;
287 +- the `socket_exact` rule uses `class: resolve_loose_side` and
288 + `action: absorb`;
289 +- the key is declarative, typically protocol + address space + IP + port;
290 +- `endpoint_socket` links are normal-strength/normal-distance visible links before
291 + aggregation;
292 +- `correlated_socket` is the farthest output link type after exact absorption.
293 +
294 +## Streaming Modal Rules
295 +
296 +For `topology:streaming` actor modals:
297 +
298 +- Size parent actors from the actor row `retained_node_count` metric, not from
299 + graph degree or direct child count. Emit `presentation.size.mode: "metric"`
300 + and `presentation.size.metric_column: "retained_node_count"` for the parent
301 + actor type. This count represents nodes for which the parent has retained
302 + data, including self, virtual nodes, stale nodes, and transit descendants
303 + when they have DB retention state.
304 +- Attach parent graph bullets to the parent side of incoming streaming links.
305 + For graph-link sources this means `ports.sources[].actor_column:
306 + "dst_actor"` and a scalar child/node display `name_column`, such as
307 + `port_name`.
308 +- Keep `actor_labels`, `stream_path`, `retention`, `inbound`, and `outbound`
309 + as the single source of truth. Do not duplicate rows only to populate modal
310 + sections.
311 +- Put important node identity/status facts in
312 + `modal.labels.identification.fields[]`, backed by `actor_labels`. Typical
313 + host-like keys are hostname, node type, health, stream, ingest, OS, OS
314 + version, kernel, architecture, CPU, cores, RAM, virtualization, container,
315 + cloud placement, and Agent version. Parent actors also include retained-node
316 + count and direct child count.
317 + Vnode actors should use inventory/device labels such as vnode type, vendor,
318 + model, address, location, sys object id, LLDP name, and status. Keep long
319 + stable identifiers such as machine GUID and node id in the full Labels tab by
320 + default.
321 +- Show `Stream path` from `stream_path` filtered by `actor`, ordered by
322 + `path_index`. This is only the selected actor's own path; child and virtual
323 + node paths belong to their own actors. Do not emit blank `since` or
324 + `first_time` values for synthetic path rows when those timestamps can be
325 + derived from adjacent path, ingest, or DB status.
326 +- Show `Retained nodes` from the `retention` table filtered by
327 + `observer_actor`; this answers which nodes' data the selected actor
328 + maintains. Include self, virtual nodes, direct children, transit descendants,
329 + and stale/archived hosts when present in the Agent root index. Preserve
330 + `db_from` and `db_to` whenever the DB status knows the range.
331 +- Show `Received nodes` from `inbound` filtered by `parent_actor`; this table
332 + represents children, virtual nodes, stale nodes, and descendants received or
333 + transiting through the selected parent. Populate `source_actor` whenever the
334 + immediate sending actor is known; for direct local receipt, use the child or
335 + virtual-node actor instead of leaving the cell empty.
336 +- Show `Outbound streams` from `outbound` filtered by the sending parent actor.
337 + This table must list every node payload the selected parent streams upstream,
338 + including self, virtual nodes, direct children, and transit descendants. Rows
339 + need at least streamed node actor, destination actor when known, status, age,
340 + hops, TLS, compression, and useful counts/replication metrics when available.
341 +- Do not show the old `Retention for node` default section in the current modal
342 + contract. Keep `actor` and `observer_actor` in the canonical retention table
343 + so Cloud aggregation can preserve multiple retaining parents and a future
344 + explicitly named `Retained by` section can be added without changing facts.
345 +
346 +## SNMP/L2 Modal Rules
347 +
348 +For SNMP/L2 managed device actor modals:
349 +
350 +- Treat the device as a collection of ports. The primary section is `Ports`
351 + over `actor_ports`.
352 +- Put important device facts in `modal.labels.identification.fields[]`, backed
353 + by `actor_labels`. Typical keys are display name, management IP, vendor,
354 + model, port counts, and LLDP/CDP neighbor counts.
355 +- Expose real port identity as typed `actor_ports` columns: SNMP `if_index` as
356 + the visible numeric port ID when known, source `port_id`, display `name`,
357 + `if_name`, `if_descr`, `if_alias`, MAC, speed, status, mode, role, VLAN, FDB,
358 + link, and neighbor counts.
359 +- Do not fabricate numeric port IDs. Do not derive port identity from row order
360 + or any generated sequence; `if_index` must come from device/SNMP facts.
361 +- Include compact expanded-row neighbor columns such as nullable
362 + `neighbor_actor` and `neighbor_port_name` when graph-link facts can align the
363 + port to a remote actor.
364 +- Use an actor-owned `actor_port_links` modal index for `Port Neighbors` when
365 + the device modal needs remote actor, remote port, link type, evidence count,
366 + confidence, inference, attachment mode, or timestamps.
367 +- `actor_port_links` may carry compact side-specific refs and scalar facts, but
368 + must not duplicate raw LLDP/CDP/FDB/ARP/STP evidence JSON.
369 +- Keep generic graph-link `Links` sections only for endpoint, segment, or
370 + custom actors that do not own port inventory.
371 +- Build link endpoint port labels only from real port fields: `port_name`,
372 + `if_name`, `if_descr`, or source `port_id`. Never use actor labels such as
373 + `display_name` or `sys_name` as port-name fallbacks.
374 +
375 +## Validation Checklist
376 +
377 +- JSON validates against the topology schema.
378 +- Semantic validation covers references, compact-table row counts, dictionaries,
379 + correlation rules, layout tokens, and schema-token parity.
380 +- Actor identities are documented and tested.
381 +- Link direction policy is documented and tested.
382 +- Correlation points, claims, rules, priorities, actions, and output link types
383 + are documented and tested when cross-payload resolution applies.
384 +- Evidence rows can reproduce required drilldown tables.
385 +- Custom actor tables have correct roles and aggregation policy.
386 +- Actor labels are emitted through `actor_labels` when the producer has labels
387 + or actor metadata to show.
388 +- `actor_labels.key`, `actor_labels.value`, `actor_labels.source`, and
389 + `actor_labels.kind` are logical string fields. Accept `string` and
390 + `string_ref` encodings as equivalent when validating, aggregating, or
391 + rendering topology payloads.
392 +- Treat `actor_labels` as sensitive topology Function data. Preserve the source
393 + Function's access-control assumptions when forwarding, aggregating, testing,
394 + or documenting labels.
395 +- Modal sections are recipes over existing facts and do not duplicate
396 + high-cardinality evidence rows.
397 +- Raw JSON columns are hidden/debug-only unless a schema-declared projection
398 + renders a scalar value.
399 +- Payload size is measured on realistic or captured data.
400 +- Raw sensitive captures remain under `.local/`.
401 +
402 +Before considering `cloud-topology-service` ready, verify service-level
403 +fixtures for all topology kinds covered by the schema. `network-connections` is
404 +the required high-cardinality benchmark, but it is not enough by itself.
405 +
406 +## vSphere Coordination
407 +
408 +The vSphere topology producer lives in a separate PR worktree. Do not edit that
409 +worktree before telling the user, because another agent may be working there.
.agents/skills/project-create-topology/how-tos/INDEX.md new
+18
@@ -0,0 +1,18 @@
1 +# project-create-topology how-tos
2 +
3 +This directory contains developer-facing recipes for validating and maintaining
4 +`netdata.topology.v1` producers, fixtures, schemas, and handoff artifacts.
5 +
6 +These are not operator workflows. Public/operator recipes for fetching Agent or
7 +Cloud data live under `docs/netdata-ai/skills/`.
8 +
9 +## Index
10 +
11 +- [add-graph-presentation.md](./add-graph-presentation.md) -- add backend-controlled graph presentation, safe labels, port-bullet sources, legends, highlight paths, and validation to a `netdata.topology.v1` producer.
12 +- [define-per-actor-highlight-paths.md](./define-per-actor-highlight-paths.md) -- encode per-actor ordered highlight paths without collapsing the clicked actor and path member into one column.
13 +- [preserve-semantic-link-types.md](./preserve-semantic-link-types.md) -- keep graph link types distinct when protocols, confidence, or inferred state need different visual treatment.
14 +- [verify-network-connections-layout-tokens.md](./verify-network-connections-layout-tokens.md) -- verify local `topology:network-connections` v1 link layout tokens and correlation rule wiring through a token-safe direct-agent call.
15 +- `migrate-network-connections.md` (stub -- not yet authored)
16 +- `migrate-streaming-topology.md` (stub -- not yet authored)
17 +- `migrate-snmp-l2-topology.md` (stub -- not yet authored)
18 +- `create-vsphere-topology.md` (stub -- coordinate before authoring in the separate worktree)
.agents/skills/project-create-topology/how-tos/add-graph-presentation.md new
+73
@@ -0,0 +1,73 @@
1 +# Add Graph Presentation To A Topology
2 +
3 +## Question
4 +
5 +How should a topology producer add polished graph presentation to
6 +`netdata.topology.v1` without making the UI domain-specific?
7 +
8 +## Inputs
9 +
10 +- A topology producer that emits `netdata.topology.v1`.
11 +- Actor and link compact tables with required `type` columns.
12 +- Type registry entries under `data.types`.
13 +- Optional evidence and actor-detail tables used by port bullets or
14 + highlight-path behavior.
15 +
16 +## Schema Choices
17 +
18 +- Put actor visuals in `data.types.actor_types.<id>.presentation`.
19 +- Put link visuals in `data.types.link_types.<id>.presentation`.
20 +- Put port bullet visuals in `data.types.port_types.<id>.presentation`.
21 +- Put cross-type graph behavior in `data.presentation`.
22 +- Use only schema-defined tokens for colors, icons, opacity, width, line style,
23 + curves, and arrows.
24 +- Use `label_policy.columns` for actor labels. Do not use canonical identity
25 + arrays as display text.
26 +- Use `ports.sources[]` when `ports.show_bullets` is true.
27 +- Use `selection.actor_click.mode: highlight_path` only with path table,
28 + path-member actor-column, and order-column references. Add an owner actor
29 + column when the same table stores different paths for different clicked
30 + actors.
31 +
32 +## Implementation Steps
33 +
34 +1. Define type-level presentation for every actor type that should have a
35 + domain-specific visual profile.
36 +2. Define link presentation for every renderable link type, including direction
37 + arrow and curve tokens.
38 +3. Define `port_types` and `ports.sources[]` together:
39 + - `source: links` reads bullets from the graph links table;
40 + - `source: evidence` reads bullets from a named evidence type;
41 + - `source: actor_table` reads bullets from a named actor detail table.
42 +4. Make `name_column` a scalar display column. Do not use `actor_ref`,
43 + `link_ref`, `evidence_ref`, `array`, or `json` as bullet labels.
44 +5. Add graph-level `data.presentation.legend`, `port_fields`, `scale_keys`, and
45 + `selection.actor_click`.
46 +6. Update producer tests or fixtures so the new presentation path is exercised.
47 +
48 +## Validation
49 +
50 +- Validate JSON against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
51 +- Run `topologyv1.ValidateDecodedResponse()` or the function-validation tool so
52 + semantic references are checked.
53 +- Add negative tests for:
54 + - missing label-policy columns;
55 + - non-display label columns;
56 + - missing port-bullet source tables;
57 + - bad highlight-path columns;
58 + - invalid token values.
59 +- Check C producers with the compile command from `build/compile_commands.json`
60 + using `-fsyntax-only` when a full local build is blocked.
61 +
62 +## Gotchas
63 +
64 +- Presentation is production payload data, not compatibility reconstruction
65 + data.
66 +- Type ids are producer-local until Cloud aggregation namespaces and
67 + canonicalizes them.
68 +- `profile_version` is diagnostic. Do not use it to drop facts or rows.
69 +- If a presentation source depends on optional runtime data, still declare a
70 + stable table/evidence type so validators can catch typos.
71 +- Sensitive identifiers may exist in topology detail tables. Do not reference
72 + them in `label_policy`, graph hover, port bullet labels, logs, docs, SOWs, or
73 + durable review artifacts.
.agents/skills/project-create-topology/how-tos/define-per-actor-highlight-paths.md new
+64
@@ -0,0 +1,64 @@
1 +# Define Per-Actor Highlight Paths
2 +
3 +## Question
4 +
5 +How should a `netdata.topology.v1` producer encode highlight paths when each
6 +clicked actor has its own ordered path?
7 +
8 +## Inputs
9 +
10 +- A producer that needs `data.presentation.selection.actor_click.mode:
11 + highlight_path`.
12 +- An actor detail table that stores path rows.
13 +- Actor row references for the clicked actor and every member of its path.
14 +
15 +## Schema Choices
16 +
17 +Use one actor detail table with:
18 +
19 +- `path_owner_column`: optional `actor_ref`; the actor whose click should use
20 + the row.
21 +- `path_actor_column`: required `actor_ref`; the actor that appears in the
22 + highlighted path.
23 +- `path_order_column`: required numeric column; the deterministic path order.
24 +
25 +Do not point `path_actor_column` at the owner column unless every actor shares
26 +one global path table. For producer-specific per-actor paths, owner and member
27 +must be separate columns.
28 +
29 +## Implementation Steps
30 +
31 +1. Define the table type with `role: actor_detail`, `owner: actor`, and
32 + `aggregation: append`.
33 +2. Add an owner actor-ref column such as `actor`.
34 +3. Add a path-member actor-ref column such as `path_actor`.
35 +4. Add a numeric order column such as `path_index`.
36 +5. Set `data.presentation.selection.actor_click` to:
37 +
38 +```json
39 +{
40 + "mode": "highlight_path",
41 + "path_table": "stream_path",
42 + "path_owner_column": "actor",
43 + "path_actor_column": "path_actor",
44 + "path_order_column": "path_index"
45 +}
46 +```
47 +
48 +## Validation
49 +
50 +- Validate the payload with `FUNCTION_TOPOLOGY_SCHEMA.json`.
51 +- Run the topology v1 validator. It checks that owner/member columns are
52 + `actor_ref` and the order column is numeric.
53 +- Add a frontend fixture where two actors have different rows in the same path
54 + table, then verify each click resolves only that actor's path.
55 +
56 +## Gotchas
57 +
58 +- The owner column is intentionally optional for backward compatibility and for
59 + a single shared global path.
60 +- Reusing the owner column as the path-member column causes each actor click to
61 + highlight only itself or direct graph neighbors, because the UI never receives
62 + the ordered path members.
63 +- Keep path rows as actor-owned detail data, not graph links. Graph links remain
64 + the compact renderable topology; path rows drive selection behavior.
.agents/skills/project-create-topology/how-tos/preserve-semantic-link-types.md new
+65
@@ -0,0 +1,65 @@
1 +# Preserve Semantic Link Types
2 +
3 +## Question
4 +
5 +How should a topology producer preserve different visual treatments for links
6 +that share the same general relationship family?
7 +
8 +## Inputs
9 +
10 +- A `netdata.topology.v1` producer.
11 +- A graph links table with a required `type` column.
12 +- Link protocols, states, or confidence markers that users must see
13 + differently, such as verified LLDP/CDP links versus inferred SNMP/L2 links.
14 +
15 +## Schema Choices
16 +
17 +- Use `links.type` for the renderable semantic link type, not only for the
18 + broad relationship family.
19 +- Define one `data.types.link_types.<id>.presentation` object for every link
20 + type that needs distinct color, width, line style, curve, arrow, hover, or
21 + variable-scaling behavior.
22 +- Keep raw producer facts, such as protocol, state, confidence, source port, and
23 + endpoint detail, in link columns or evidence rows.
24 +- If evidence rows are grouped by link type, define a matching
25 + `data.types.evidence_types.<id>.link_type` and a matching `data.evidence`
26 + section for each emitted type.
27 +
28 +## Implementation Steps
29 +
30 +1. Inventory the legacy or intended visual categories.
31 + - Example: SNMP/L2 uses `lldp`, `cdp`, `bridge`, `fdb`, `stp`, `arp`,
32 + `snmp`, and `probable`.
33 +2. Add all renderable categories to `data.types.link_types`.
34 +3. Put the visual contract in the matching link type presentation.
35 + - Example: verified LLDP/CDP can use an accent color and thicker width.
36 + - Example: probable/inferred links can use a dim color or dashed line.
37 +4. Emit the selected semantic type in `data.links.type` for every row.
38 +5. Keep the original protocol/state in separate columns, so drilldowns and
39 + evidence stay factual.
40 +6. Add legend entries for the user-facing categories that should be explained.
41 +7. Add a test that decodes `data.links.type` and checks presentation tokens and
42 + evidence type linkage.
43 +
44 +## Validation
45 +
46 +- Validate the full response against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
47 +- Run `topologyv1.ValidateDecodedResponse()` so type references are checked.
48 +- For Go producers, add a package test that verifies:
49 + - rows with verified protocols keep their protocol link type;
50 + - inferred/probable rows use the inferred/probable link type;
51 + - every emitted evidence section uses an evidence type whose `link_type`
52 + matches the graph link type;
53 + - legend entries include the visible categories.
54 +
55 +## Gotchas
56 +
57 +- Do not collapse distinct visual categories into one generic link type and
58 + expect the UI to infer producer-specific meaning from protocol or state.
59 +- Do not encode UI policy in frontend conditionals such as "if protocol is
60 + LLDP, make it green"; the producer owns type composition and the UI only maps
61 + schema tokens.
62 +- If two facts are both true, keep both: the row can have `type: "probable"`
63 + and `protocol: "bridge"` at the same time.
64 +- A fallback generic type such as `l2_observation` is useful for unknown
65 + protocols, but known user-visible categories should have explicit types.
.agents/skills/project-create-topology/how-tos/verify-network-connections-layout-tokens.md new
+185
@@ -0,0 +1,185 @@
1 +# Verify network-connections layout tokens
2 +
3 +## Question
4 +
5 +How can a developer verify that a local Cloud-connected Netdata Agent serves
6 +`topology:network-connections` with the expected v1 link layout tokens and
7 +correlation rule wiring, without exposing Cloud tokens, agent bearers, node ids,
8 +machine GUIDs, claim ids, cookies, or raw endpoint rows?
9 +
10 +This is a producer-contract validation recipe. It belongs to the developer
11 +topology skill, not to the public/operator query skills.
12 +
13 +## Inputs
14 +
15 +- Local Agent URL, usually `http://127.0.0.1:19999`; set `AGENT_URL` when
16 + using a non-default address.
17 +- `NETDATA_CLOUD_TOKEN` and `NETDATA_CLOUD_HOSTNAME` in `<repo>/.env`.
18 +- A local Agent that exposes `topology:network-connections`.
19 +
20 +## Steps
21 +
22 +1. Capture the local identity tuple in memory and print only presence
23 + checks:
24 +
25 + ```bash
26 + AGENT_URL="${AGENT_URL:-http://127.0.0.1:19999}"
27 + AGENT_URL="${AGENT_URL%/}"
28 + AGENT_HOST="${AGENT_URL#http://}"
29 + AGENT_HOST="${AGENT_HOST#https://}"
30 + AGENT_HOST="${AGENT_HOST%%/*}"
31 + AGENT_SCHEME="http"
32 + case "$AGENT_URL" in
33 + https://*) AGENT_SCHEME="https" ;;
34 + esac
35 + AGENT_ORIGIN="${AGENT_SCHEME}://${AGENT_HOST}"
36 +
37 + INFO_JSON="$(curl --fail -sS --max-time 10 "${AGENT_ORIGIN}/api/v3/info")"
38 +
39 + jq '{
40 + agent_count: (.agents | length),
41 + node_id_present: ((.agents[0].nd // "") | length > 0),
42 + machine_guid_present: ((.agents[0].mg // "") | length > 0),
43 + claim_id_present: ((.agents[0].cloud.claim_id // "") | length > 0),
44 + cloud_status: .agents[0].cloud.status
45 + }' <<<"$INFO_JSON"
46 + ```
47 +
48 +2. Load the token-safe direct-agent wrappers:
49 +
50 + ```bash
51 + source docs/netdata-ai/skills/query-netdata-agents/scripts/_lib.sh
52 + agents_load_env
53 + ```
54 +
55 +3. Query the topology Function through the direct-agent path. Store the
56 + raw response only under `.local/`:
57 +
58 + ```bash
59 + NODE_UUID="$(jq -r '.agents[0].nd' <<<"$INFO_JSON")"
60 + MACHINE_GUID="$(jq -r '.agents[0].mg' <<<"$INFO_JSON")"
61 + FUNCTION_NAME="topology:network-connections processes:by_name mode:aggregated sockets:inbound,outbound,listening,local protocols:ipv4_tcp,ipv6_tcp,ipv4_udp,ipv6_udp endpoints:by_ip"
62 + FUNCTION_ENCODED="$(printf '%s' "$FUNCTION_NAME" | jq -sRr @uri)"
63 + OUT="$(agents_audit_dir)/network-connections-aggregated-live.json"
64 +
65 + agents_query_agent \
66 + --node "$NODE_UUID" \
67 + --host "$AGENT_HOST" \
68 + --machine-guid "$MACHINE_GUID" \
69 + GET "/api/v3/function?function=${FUNCTION_ENCODED}&timeout=120000&last=200" \
70 + > "$OUT"
71 + ```
72 +
73 +4. Print a sanitized response summary:
74 +
75 + ```bash
76 + jq '{
77 + status,
78 + type,
79 + schema_version: .data.schema_version,
80 + actor_rows: .data.actors.rows,
81 + link_rows: .data.links.rows,
82 + correlation_point_rows: .data.correlation.points.rows,
83 + correlation_claim_rows: .data.correlation.claims.rows
84 + }' "$OUT"
85 + ```
86 +
87 +5. Verify the producer's link layout tokens:
88 +
89 + ```bash
90 + jq '.data.types.link_types
91 + | with_entries({
92 + key: .key,
93 + value: {
94 + label: .value.presentation.label,
95 + color_slot: .value.presentation.color_slot,
96 + line_style: .value.presentation.line_style,
97 + width: .value.presentation.width,
98 + variable: .value.presentation.variable,
99 + layout: .value.presentation.layout
100 + }
101 + })' "$OUT"
102 + ```
103 +
104 +6. Verify correlation rule wiring without printing endpoint rows:
105 +
106 + ```bash
107 + jq '.data.correlation.rules
108 + | with_entries({
109 + key: .key,
110 + value: {
111 + action: .value.action,
112 + priority: .value.priority,
113 + key_space: .value.key_space,
114 + point_actor_types: .value.point_actor_types,
115 + claim_actor_types: .value.claim_actor_types,
116 + correlation_link_types: .value.correlation_link_types,
117 + output_link_type: .value.output_link_type
118 + }
119 + })' "$OUT"
120 + ```
121 +
122 +7. Count graph links by type from the compact table:
123 +
124 + ```bash
125 + jq -r '
126 + .data as $d
127 + | def col($table; $name):
128 + ($table.columns | map(.id) | index($name)) as $i
129 + | $table.values[$i];
130 + def v($c; $i):
131 + if $c.codec == "const" then $c.value
132 + elif $c.codec == "values" then $c.values[$i]
133 + elif $c.codec == "dict" then $c.values[$c.indexes[$i]]
134 + else null end;
135 + (col($d.links; "type")) as $typeCol
136 + | (col($d.links; "socket_count")) as $socketCol
137 + | [range(0; $d.links.rows)
138 + | {type: v($typeCol; .), socket_count: (v($socketCol; .) // 0)}]
139 + | group_by(.type)
140 + | map({type: .[0].type, links: length, sockets: (map(.socket_count) | add)})
141 + | sort_by(.type)
142 + | .[]
143 + | [.type, .links, .sockets]
144 + | @tsv
145 + ' "$OUT"
146 + ```
147 +
148 +## Output
149 +
150 +For the current network-connections v1 contract, expect these link
151 +types:
152 +
153 +- `endpoint_socket`: visible unresolved endpoint links, weakest
154 + strength, normal distance.
155 +- `correlated_socket`: Cloud aggregator output after exact absorption,
156 + weakest strength, farthest distance.
157 +- `socket`: local/resolved process links, stronger strength, farther
158 + distance, variable width by `socket_count`.
159 +- `ownership`: graph-coherence node-to-process links, dotted/faded,
160 + normal strength, normal distance.
161 +
162 +The `socket_exact` correlation rule should consume `endpoint_socket`
163 +through `correlation_link_types` and emit `correlated_socket` through
164 +`output_link_type`.
165 +
166 +## Notes / gotchas
167 +
168 +- The wrapper logs masked curl commands on stderr. Cloud tokens,
169 + per-agent bearers, node ids, machine GUIDs, and claim ids must not
170 + appear in stdout or committed files.
171 +- Keep the raw Function response under `.local/`; it may contain
172 + hostnames, private addresses, and process names.
173 +- A rendered graph can still look stretched even when the payload uses
174 + `endpoint_socket` with normal distance. That is a frontend force-layout
175 + issue, not proof that the backend emitted `farthest`.
176 +- Compact topology tables require decoding the column codec before
177 + counting rows by type. Do not assume the high-cardinality rows are
178 + emitted as arrays of objects.
179 +
180 +## Source guides
181 +
182 +- [Topology producer skill](../SKILL.md)
183 +- [Direct-agent operator skill](../../../../docs/netdata-ai/skills/query-netdata-agents/SKILL.md)
184 +- [Direct Function calls](../../../../docs/netdata-ai/skills/query-netdata-agents/query-functions.md)
185 +- [Direct topology calls](../../../../docs/netdata-ai/skills/query-netdata-agents/query-topology.md)
.agents/skills/project-writing-collectors/SKILL.md
+3 -2
@@ -406,13 +406,13 @@ Interactive, on-demand tabular data: process lists, network connections, FDB tab
406
407 Build a Function when the answer is **interactive/tabular live data**. If the answer is a numeric time series, that's a metric.
408
409 -Response shape is one of `info_response`, `data_response`, `topology_response`, `flows_response`, `error_response`, `not_modified_response` (defined in `src/plugins.d/FUNCTION_UI_SCHEMA.json`). For Go, use builders in `src/go/pkg/funcapi/`. For Rust, implement the `FunctionHandler` trait from the SDK runtime (`src/crates/netdata-plugin/rt/`).
409 +Response shape is one of `info_response`, `data_response`, `topology_response`, `flows_response`, `error_response`, `not_modified_response` (defined in `src/plugins.d/FUNCTION_UI_SCHEMA.json`). New topology payloads use the dedicated production topology contract in `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`. For Go, use builders in `src/go/pkg/funcapi/`; Go topology producers should use `src/go/pkg/topology/v1` for the v1 response model and compact-table helpers. For Rust, implement the `FunctionHandler` trait from the SDK runtime (`src/crates/netdata-plugin/rt/`).
410
411 Functions run concurrently with the collection loop — they must not block it. Validate during development with `src/go/tools/functions-validation/`.
412
413 Reference implementations: `src/collectors/network-viewer.plugin/` (topology + connections), `src/collectors/systemd-journal.plugin/` (log explorer), `src/collectors/apps.plugin/` (processes).
414
415 -Backend docs: `src/go/plugin/framework/functions/README.md` (Go), `src/crates/netdata-plugin/rt/src/lib.rs` (Rust `FunctionHandler`). UI/protocol: `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_UI_REFERENCE.md`.
415 +Backend docs: `src/go/plugin/framework/functions/README.md` (Go), `src/crates/netdata-plugin/rt/src/lib.rs` (Rust `FunctionHandler`). UI/protocol: `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_UI_REFERENCE.md`. Topology contract: `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
416
417 ### 6.4 Topology / interconnections / links
418
@@ -504,6 +504,7 @@ Internal C plugins under `src/collectors/`. Reuse shared metric definitions from
504 | go.d V1 best practices / lifecycle | working in legacy V1 module | `src/go/BEST-PRACTICES.md`, `src/go/COLLECTOR-LIFECYCLE.md` |
505 | Functions backend (Go / Rust) | implementing a Function | `src/go/plugin/framework/functions/README.md`, `src/crates/netdata-plugin/rt/src/lib.rs` |
506 | Functions UI schema & guides | response shapes and patterns | `src/plugins.d/FUNCTION_UI_SCHEMA.json`, `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_UI_REFERENCE.md` |
507 +| Topology Function schema & guide | topology actors, links, evidence, overlays | `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`, `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` |
508 | Functions validator | E2E + schema validation | `src/go/tools/functions-validation/README.md` |
509 | ibm.d framework | starting `ibm.d` work | `src/go/plugin/ibm.d/AGENTS.md`, `src/go/plugin/ibm.d/framework/README.md` |
510 | Rust plugin SDK | new Rust plugin | `src/crates/netdata-plugin/` (`rt/`, `protocol/`, `bridge/`, `charts-derive/`, `schema/`, `types/`, `error/`) |
.agents/sow/current/SOW-0022-20260509-topology-table-composition.md new
+1007
@@ -0,0 +1,1007 @@
1 +# SOW-0022 - Topology table composition
2 +
3 +## Status
4 +
5 +Status: paused
6 +
7 +Sub-state: paused while the function-specific network-connections modal product
8 +composition work proceeds in SOW-0025. Agent producer implementation is in-tree,
9 +external read-only review found no remaining actionable issues, and narrow
10 +validation passed. Full integrated UI/aggregator validation is still pending
11 +before close.
12 +
13 +## Requirements
14 +
15 +### Purpose
16 +
17 +Make topology actor and link drilldowns useful, curated, compact, and domain-agnostic. The UI must not display raw producer JSON blobs, long nested arrays, unformatted endpoint structures, or internal matching identifiers as final modal content.
18 +
19 +### User Request
20 +
21 +The user observed that current topology actor modals and table cells can show raw JSON directly in the final UI, including large actor attribute objects, nested interface/status arrays, neighbor arrays, and endpoint objects. The user explicitly requested analysis of all actor modals and split this as the second remediation step after presentation:
22 +
23 +- SOW-0021: fix topology presentation.
24 +- SOW-0022: fix table composition.
25 +
26 +### Assistant Understanding
27 +
28 +Facts:
29 +
30 +- The new compact schema separates actor/link/evidence/detail tables, but modal composition is not yet sufficiently specified.
31 +- Some current UI paths render nested JSON structures directly instead of curated, typed fields.
32 +- Actor modal content includes both relationship evidence and actor-owned custom data, and those need different composition and aggregation semantics.
33 +- User-provided examples contain infrastructure-identifying values and must not be copied into durable artifacts.
34 +
35 +Inferences:
36 +
37 +- Modal composition needs an explicit schema/profile layer, not just raw table definitions.
38 +- Detail tables need typed column presentation, formatters, visibility defaults, source/purpose, aggregation policy, and safe rendering rules.
39 +- Relationship evidence should power drilldowns without duplicating every evidence row under every actor.
40 +
41 +Unknowns:
42 +
43 +- The full current cloud-frontend modal renderer shape and all topology-specific assumptions.
44 +- The full set of existing actor/link modal tables for network-connections, streaming, SNMP/L2, and vSphere.
45 +
46 +### Acceptance Criteria
47 +
48 +- Inventory every current actor/link modal and table source for old and new topology payloads.
49 +- Define table composition profiles for actor details, link details, relationship evidence, relationship summaries, inventory, endpoint summaries, custom actor data, and path tables.
50 +- Define safe scalar, enum, reference, array, and nested object rendering rules so raw JSON blobs do not leak into final UI unless explicitly marked as raw/debug.
51 +- Define actor label and table display-name behavior if not fully closed by SOW-0021.
52 +- Update schema/docs/skill/specs and backend producers as needed.
53 +- Create Cloud frontend and Cloud aggregator handoff requirements for modal/table composition.
54 +- Validate with sanitized fixtures covering SNMP/L2, streaming, and network-connections modal examples.
55 +- Ensure `data.correlation.points` and `data.correlation.claims` are not shown
56 + as raw actor modal tables unless a future schema explicitly exposes a curated
57 + debug/diagnostic view.
58 +
59 +## Analysis
60 +
61 +Sources checked:
62 +
63 +- `.agents/sow/done/SOW-0021-20260509-topology-presentation-contract.md`
64 +- `.agents/sow/done/SOW-0023-20260509-topology-cross-payload-matching.md`
65 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json:286-372`
66 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:1211-1260`
67 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md:456-462`
68 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md:541-584`
69 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation_schema.go:7-96`
70 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:635-713`
71 +
72 +Current state:
73 +
74 +- SOW-0021 restored graph-level presentation for actors, links, ports,
75 + legends, labels, and highlight paths.
76 +- SOW-0023 added the correlation plane, semantic correlation link types, and
77 + link layout tokens.
78 +- The old Function UI topology schema had actor-type `summary_fields`,
79 + `tables`, and `modal_tabs`, plus table column labels and cell types.
80 +- The old SNMP topology producer used those fields to describe device summary
81 + fields, Ports and Links tables, column labels, badge/number/actor-link cell
82 + hints, and an Info tab.
83 +- The v1 topology schema currently classifies table types by `role`, `owner`,
84 + `aggregation`, `source_evidence`, and raw `columns`, but does not define how
85 + actor/link modals compose summaries, tabs, table sections, visible columns,
86 + nested values, or relationship-derived rows.
87 +- The v1 developer guide explicitly says full modal/table composition, column
88 + hiding, nested JSON rendering, and richer formatting belong to this SOW.
89 +- SNMP v1 currently preserves actor attributes/labels as an `actor_metadata`
90 + actor-detail table with `json` cells, and converts actor-owned dynamic tables
91 + mechanically. This preserves facts but does not preserve curated modal
92 + behavior.
93 +
94 +Risks:
95 +
96 +- Uncurated modal tables can leak sensitive infrastructure details, overwhelm users, and make topology look unfinished.
97 +- Over-modeling table UI can couple backend producers to frontend component internals.
98 +- Under-modeling table UI forces the frontend to hardcode producer-specific modal logic.
99 +
100 +### Per-Function Modal/Table Inventory
101 +
102 +Design rule:
103 +
104 +- Modal/table definitions should describe how to select, filter, join, format,
105 + and order existing topology facts. They must not duplicate high-cardinality
106 + rows only for UI display.
107 +- If a value is a canonical actor, link, evidence, inventory, or relationship
108 + fact, it belongs in the relevant canonical row/table once. Modal composition
109 + may reference it many times.
110 +- If old modal behavior depended on a value that v1 no longer emits anywhere,
111 + the fix is to restore that canonical value once in the appropriate actor,
112 + link, evidence, or detail table, not to create a duplicate modal-only table.
113 +- Raw `json` columns are facts, not UI. They may be preserved for lossless
114 + debug or future structured expansion, but polished modals should only render
115 + curated scalar/array/reference projections from them when a schema declares
116 + that projection.
117 +
118 +#### topology:network-connections
119 +
120 +Legacy behavior reviewed:
121 +
122 +- Presentation came from `topology_write_presentation()` in the legacy
123 + network viewer function.
124 +- `self` actor summary showed hostname, local IP count, and observed sockets.
125 + Its `Connections` table used `source: links` and displayed the remote actor,
126 + protocol, and direction.
127 +- `process` actor summary showed process display name, command, sockets, local
128 + IP, and user. Its `Sockets` table used actor-owned socket rows and displayed
129 + remote endpoint, protocol, direction, and state. Its `Connections` table used
130 + `source: links` and displayed remote actor, protocol, direction, and state.
131 +- `endpoint` actor summary showed endpoint IP, socket count, and address
132 + space. Its `Connections` table used `source: links` and displayed remote
133 + actor, protocol, and direction.
134 +
135 +Current v1 facts reviewed:
136 +
137 +- Actor rows carry type, machine GUID, hostname, process, PID/PPID/UID when PID
138 + scope is selected, network namespace, local IP/address-space, endpoint
139 + IP/address-space, display name, and socket count.
140 +- Graph links carry actor refs, link type, protocol, direction, state,
141 + evidence count, socket count, retransmissions, and RTT maxima.
142 +- Detailed mode emits `socket` relationship evidence with actor refs, local and
143 + remote tuples, protocol family, direction, state, namespace, process, socket
144 + count, retransmissions, and RTT maxima.
145 +- Aggregated mode emits compact actor-owned `socket_ports` inventory with actor
146 + ref, port, protocol, direction, and socket count. This is enough for process
147 + port bullets without reading detailed socket evidence.
148 +- Correlation points and claims exist for matching, but must not become modal
149 + tables.
150 +
151 +No-duplication reconstruction strategy:
152 +
153 +- `self` and `endpoint` `Connections` tables should be generated by filtering
154 + graph links where `src_actor == actor` or `dst_actor == actor`, then
155 + deriving `remoteLabel` from the opposite actor ref. Protocol, direction, and
156 + state come from the link row.
157 +- `process` `Connections` should use the same graph-link projection and can
158 + optionally filter or group by semantic link types so ownership links do not
159 + appear as network dependencies unless explicitly requested.
160 +- `process` `Sockets` in detailed mode should be generated by filtering
161 + `evidence.socket` rows where `src_actor == actor` or `dst_actor == actor`.
162 + The old `remote` display is a formatted projection of the remote tuple, not
163 + a stored duplicate.
164 +- `process` `Sockets` in aggregated mode should use graph links plus
165 + `socket_count`, and `socket_ports` for port bullets. It cannot show every
166 + socket row because those rows were intentionally not emitted in aggregated
167 + mode.
168 +- Missing canonical fields: the current v1 implementation fills actor struct
169 + fields for `username` and `cmdline`, but the actor column list does not emit
170 + them. Old process summaries cannot be fully reconstructed until those values
171 + are restored as actor columns. Old `self.local_ip_count` is also not emitted
172 + as a v1 actor column; either restore it as a self actor metric or drop that
173 + legacy summary field deliberately.
174 +
175 +#### topology:streaming
176 +
177 +Legacy behavior reviewed:
178 +
179 +- Parent actor summary showed name, node type, agent version, OS, architecture,
180 + CPU count, child count, critical alerts, and warning alerts.
181 +- Child actor summary showed name, node type, agent version, OS, architecture,
182 + CPU count, critical alerts, and warning alerts.
183 +- Virtual node summary showed name, node type, and ephemerality.
184 +- Stale actor summary showed name, node type, agent version, OS, and
185 + architecture.
186 +- Parent `Inbound` table showed node actor link, received-from actor link, node
187 + type badge, ingest badge, hops, collected metrics/instances/contexts,
188 + replication completion, ingest age, SSL badge, and alert counts.
189 +- Parent `Outbound` table showed node actor link, streamed-to actor link, node
190 + type badge, stream status badge, hops, SSL badge, and compression badge.
191 +- `Streaming Path` table showed agent, hops, since, and flags.
192 +- `Retention` table showed actor link, database status, from/to timestamps,
193 + duration, metrics, instances, and contexts.
194 +
195 +Current v1 facts reviewed:
196 +
197 +- Actor rows carry type, machine GUID, node ID, hostname, display name,
198 + severity, ephemerality, ingest status, stream status, ML status, agent name,
199 + agent version, health status, child count, and health alert counts.
200 +- Link and evidence rows carry streaming/virtual/stale relationship refs,
201 + state, port name, timestamps, hops, connection and replication metrics, and
202 + collected metric/instance/context counts.
203 +- Detail tables already exist for `stream_path`, `retention`, `inbound`, and
204 + `outbound`, with actor refs instead of duplicated display strings.
205 +
206 +No-duplication reconstruction strategy:
207 +
208 +- Summary fields should read actor columns directly. `node_type` is the actor
209 + `type` column.
210 +- `Inbound.name` is a projection of `child_actor` rendered through actor label
211 + policy. `Inbound.received_from` is a projection of nullable `source_actor`.
212 + `node_type` is derived by joining `child_actor` to the actors table and
213 + reading its `type`.
214 +- `Outbound.name` is a projection of `actor`. `Outbound.streamed_to` is a
215 + projection of nullable `destination_actor`.
216 +- `Retention.name` is a projection of `actor`; the label can change per modal
217 + profile without changing row data.
218 +- `Streaming Path.Agent` is `path_actor` when present, otherwise `hostname`.
219 + This preserves streaming-path highlighting and avoids storing display strings
220 + twice.
221 +- Missing canonical fields: old summaries included OS, architecture, and CPU
222 + count. Current v1 actor rows do not emit those fields. Reconstructing the old
223 + summary requires restoring them as actor columns if the product still wants
224 + them in the modal.
225 +
226 +#### topology:snmp
227 +
228 +Legacy behavior reviewed:
229 +
230 +- Device summary showed type, vendor, model, description, location, contact,
231 + protocols, capabilities, total ports, VLAN count, FDB MAC count, LLDP/CDP
232 + neighbor counts, chart prefix, Netdata host, source, and layer.
233 +- Device `Ports` table showed port name, operational/admin status, port type,
234 + link mode, topology role, STP state, VLAN count/list, FDB MAC count, link
235 + count, and neighbor count.
236 +- Device `Links` table used `source: links` and showed local port, remote
237 + actor, remote port, protocol, and direction.
238 +- Segment summary showed type, discovery sources, ports total, endpoints total,
239 + source, and layer.
240 +- Endpoint summary showed type, vendor, discovery sources, source, and layer.
241 +
242 +Current v1 facts reviewed:
243 +
244 +- Actor rows carry identity and match-oriented fields: type, layer, source,
245 + display name, chassis IDs, MAC addresses, IP addresses, hostnames, DNS names,
246 + sysObjectID, sysName, and parent devices.
247 +- Link rows carry source/destination actor refs, semantic link type, protocol,
248 + direction, state, evidence count, and timestamps.
249 +- Evidence rows currently include `src_endpoint`, `dst_endpoint`, and `metrics`
250 + as `json` cells.
251 +- Actor detail tables are built mechanically from old actor tables. The old
252 + device `ports` table becomes `actor_ports`/`actor_ports`-like actor detail,
253 + while `actor_metadata` preserves raw attributes and labels as JSON.
254 +
255 +No-duplication reconstruction strategy:
256 +
257 +- Device, segment, and endpoint summaries should read scalar projections from
258 + actor rows and curated actor-detail columns. They should not display the
259 + entire `actor_metadata.attributes` or `actor_metadata.labels` objects.
260 +- Device `Ports` should be generated from the actor-owned port inventory table.
261 + Existing fields such as `name`, `oper_status`, `admin_status`, `port_type`,
262 + `link_mode`, `topology_role`, `stp_state`, `vlan_ids`, `fdb_mac_count`,
263 + `link_count`, and `neighbor_count` should be declared as visible columns
264 + where present. Nested `neighbors` must not render as raw JSON in the main
265 + ports grid; it needs either a compact count in the row or a nested explicit
266 + drilldown profile.
267 +- Device `Links` should be generated by filtering graph links for the selected
268 + actor and joining link/evidence endpoint columns. `remoteLabel` is the
269 + opposite actor ref; `protocol` and `direction` come from the link row.
270 + `localPort` and `remotePort` should be projections from structured endpoint
271 + fields in evidence, not raw endpoint JSON objects.
272 +- Missing canonical fields: many legacy SNMP summary fields live only inside
273 + `actor_metadata.attributes` today. To avoid raw JSON display, SOW-0022 must
274 + either move important scalar fields into typed actor columns/typed detail
275 + columns, or define a safe projection mechanism from JSON paths with strict
276 + scalar output and hidden-by-default raw JSON.
277 +
278 +#### vSphere topology
279 +
280 +- vSphere remains legacy in a separate PR worktree and is tracked as a later
281 + migration SOW. This SOW should define a generic modal/table contract that the
282 + vSphere migration can use, but it should not edit that worktree without user
283 + coordination.
284 +
285 +### Schema Direction For SOW-0022
286 +
287 +The schema needs compact composition definitions, not modal row duplication:
288 +
289 +- Actor and link types define modal profiles in presentation metadata.
290 +- Table types define display metadata for existing table rows: label, order,
291 + default visibility, column labels, column display types, sorting, grouping,
292 + empty-state behavior, and raw/debug visibility.
293 +- Modal sections reference existing sources:
294 + `actors`, `links`, `evidence.<type>`, `tables.actor.<table>`,
295 + `tables.relationship.<table>`, or future typed detail tables.
296 +- A section can declare owner filters such as `actor_ref == selected actor`,
297 + `link_ref == selected link`, or `src_actor/dst_actor contains selected actor`.
298 +- A section can declare safe projections:
299 + direct column, actor label from actor ref, opposite actor label from a link,
300 + formatted endpoint from IP/port columns, scalar JSON-path extraction, array
301 + length/count, badge/number/timestamp/duration formatting, and nullable
302 + fallback.
303 +- The UI must never infer producer domain names such as process, router,
304 + parent, child, client, server, or endpoint. It should execute schema-declared
305 + projections over topology tables.
306 +- The aggregator should preserve composition definitions and merge compatible
307 + table metadata by namespace/dedup rules from SOW-0021/SOW-0023. It should
308 + not materialize modal rows during aggregation unless it is already merging
309 + the underlying canonical table.
310 +
311 +User decisions recorded for this SOW:
312 +
313 +- Host labels must be exposed in full, without topology-specific filtering,
314 + when available. They belong on host/node-level actors and must be shown in
315 + actor modals as actor labels.
316 +- Non-node actors must also expose all known actor labels. For process actors
317 + this includes process metadata such as command line, user, group, namespace,
318 + and similar producer-known facts where available.
319 +- `json` columns should be used only when the UI or aggregator has declared
320 + semantics for them. If the value is only intended for display or filtering,
321 + prefer typed scalar/array columns or a key/value label table.
322 +- An actor modal has four top-level entities:
323 + actor name, actor labels, a depth-1 topology-map miniature, and tables.
324 +- Table composition must support actor-reference cells and row expansion.
325 + Actor references are table cell projections such as `actor_link`; expandable
326 + rows are presentation annotations over hidden/detail columns, not separate
327 + duplicated row data.
328 +
329 +Recommended representation for labels:
330 +
331 +- Use a compact actor-owned label table rather than one raw JSON map per actor:
332 + `actor_labels(actor, key, value, source?, kind?, value_index?)`.
333 +- For host/node actors, populate this table from the complete host label set.
334 +- For non-node actors, populate it from producer-known label and metadata
335 + facts. Keep facts that the aggregator must group or correlate on as
336 + canonical typed actor columns too; the label table is for display/filter/
337 + drilldown, not a replacement for canonical identity/grouping columns.
338 +- Repeated label values should use repeated rows with the same `actor` and
339 + `key`, ordered by `value_index`, rather than JSON arrays.
340 +- Sensitive-data note: topology Functions are sensitive-data surfaces with
341 + admin-controlled access. This permits exposing host labels, command lines,
342 + users, and topology metadata in Function responses. Implementation must still
343 + avoid copying raw label captures into durable repository artifacts or logs.
344 +
345 +### Mapping Validation Matrix
346 +
347 +This section maps old modal content to the new compact schema model. The goal
348 +is to prove whether information exists once in canonical tables and whether a
349 +table recipe can reconstruct the old polished UI without duplicating rows.
350 +
351 +#### Shared Actor Modal Model
352 +
353 +| Modal entity | Source in v1 | Needed schema/table recipe |
354 +|---|---|---|
355 +| Actor name | `actors` row via actor type `presentation.label_policy` | Existing SOW-0021 label policy is enough. |
356 +| Actor labels | New `tables.actor.actor_labels` table | Add table type and a default modal section that filters `actor_labels.actor == selected actor`. |
357 +| Depth-1 topology miniature | Existing `actors` and `links` tables | UI can build from incident links and opposite actors; schema may allow optional link-type filters. No duplicated payload. |
358 +| Tables | Existing `actors`, `links`, `evidence.*`, and `tables.actor.*` | Add modal/table composition recipes with source, owner filter, row filters, projections, cell types, visibility, sorting, and row expansion. |
359 +
360 +Required generic table-recipe primitives:
361 +
362 +- `source`: `actors`, `links`, `evidence.<type>`, `tables.actor.<table>`, or
363 + `tables.relationship.<table>`.
364 +- `owner_filter`: selected actor/link relationship, for example
365 + `actor_ref == selected_actor`, `src_actor == selected_actor`,
366 + `dst_actor == selected_actor`, or either endpoint.
367 +- `row_filters`: link type, evidence type, null/non-null, or value predicates.
368 +- `projection`: direct column, actor label from actor ref, opposite actor label,
369 + conditional local/remote endpoint field, formatted endpoint, label-table
370 + lookup, scalar JSON-path only when explicitly declared.
371 +- `cell`: `text`, `number`, `badge`, `actor_link`, `timestamp`, `duration`,
372 + `endpoint`, `array_count`, or `debug_json`.
373 +- `visibility`: `table`, `expanded`, `hidden`, or `debug`.
374 +
375 +#### topology:network-connections Mapping
376 +
377 +| Old modal/table item | Current/new canonical source | Recipe/status |
378 +|---|---|---|
379 +| Actor name for self/process/endpoint | `actors.display_name` with label policy fallback to hostname/process/IP | Covered by current actor columns. |
380 +| Self labels | New `actor_labels` rows from complete host labels plus topology facts such as hostname/local IP count/socket count | Add `actor_labels`; add `local_ip_count` as either actor metric column or actor label. |
381 +| Process labels | New `actor_labels` rows from process metadata: process name, PID/PPID/UID when available, user, command line, namespace, local IP/address space, socket count | Add `actor_labels`; emit `username` and `cmdline` because the v1 struct fills them but actor columns do not currently expose them. Group name is not currently collected by network-viewer; add it only if a canonical source is introduced. |
382 +| Endpoint labels | New `actor_labels` rows from IP, address space, socket counts, endpoint class | Add `actor_labels`; existing actor columns cover IP/address-space/socket count. |
383 +| Self `Connections` table | `links` incident to selected self actor | Recipe filters incident links, hides or de-emphasizes `ownership` unless a modal asks for graph-coherence links, projects opposite actor as `actor_link`, plus protocol/direction/state/socket metrics. |
384 +| Process `Connections` table | `links` incident to selected process actor | Recipe filters incident network links, projects opposite actor, protocol, direction, state, socket count, RTT/retransmit metrics. Ownership links should be a separate optional/expanded section. |
385 +| Process `Sockets` table, detailed mode | `evidence.socket` rows where selected actor is `src_actor` or `dst_actor` | Recipe projects formatted remote endpoint from local/remote tuple based on selected side, protocol, direction, state, counts, RTT/retransmit metrics; row expansion can expose PID/UID/netns/process/address-space fields. |
386 +| Process port bullets | `tables.actor.socket_ports` | Already canonical; recipe not needed for graph bullets, but modal can show the same table if useful. |
387 +| Process `Sockets` table, aggregated mode | `links` plus `socket_count`; `socket_ports` for port-level rollup | Covered with aggregated summary table. Exact per-socket rows are intentionally unavailable in aggregated mode. |
388 +| Endpoint `Connections` table | `links` incident to selected endpoint actor | Recipe projects opposite actor as `actor_link`, protocol, direction, state, socket count. |
389 +| Depth-1 miniature | Incident `links` and opposite actors | No new data. The mini graph should probably default to network link types and omit ownership unless explicitly enabled. |
390 +
391 +Network-connections validation result:
392 +
393 +- Information is mostly present once.
394 +- Required canonical additions: `actor_labels`, emitted `username`, emitted
395 + `cmdline`, and either emitted `local_ip_count` or a decision to drop that
396 + old self summary.
397 +- Required schema additions: modal/table recipes, conditional endpoint
398 + projection, actor-link cell, row expansion visibility.
399 +
400 +#### topology:streaming Mapping
401 +
402 +| Old modal/table item | Current/new canonical source | Recipe/status |
403 +|---|---|---|
404 +| Actor name | `actors.display_name` / `actors.hostname` via label policy | Covered. |
405 +| Host labels | New `actor_labels` rows from complete `host->rrdlabels` for every RRDHOST-backed actor | Add `actor_labels`; old code nested host labels under actor labels. |
406 +| Streaming actor labels | New `actor_labels` rows from actor columns and host/system metadata: node type, severity, ephemerality, ingest/stream/ML status, agent name/version, health status, child count, alert counts | Add `actor_labels`; do not duplicate graph identity fields outside canonical actor columns. |
407 +| Parent/child summaries: name, type, version, child count, health counts | `actors` columns and/or `actor_labels` | Covered for name/type/version/child/health. |
408 +| Parent/child/stale summaries: OS, architecture, CPU count | Old code emitted system info from `rrdhost_system_info_to_json_object_fields`; current v1 actor columns do not expose it | Add to `actor_labels` from host system info; add typed actor columns only if aggregator/UI needs grouping or sorting by these fields. |
409 +| Virtual node summary: ephemerality | `actors.ephemerality` | Covered. |
410 +| Parent `Inbound` table | `tables.actor.inbound` | Covered by current typed table: parent, child, source actor refs; status, hops, metrics, replication, age, SSL, alert counts. Recipe maps `child_actor` to old `name`, `source_actor` to old `received_from`, and child actor type to old `node_type`. |
411 +| Parent `Outbound` table | `tables.actor.outbound` | Covered by current typed table. Recipe maps `actor` to old `name`, nullable `destination_actor` to old `streamed_to`, actor type to old `node_type`, then status/hops/SSL/compression. |
412 +| `Streaming Path` table | `tables.actor.stream_path` | Covered by current typed table. Recipe maps `path_actor` to actor link when present, fallback `hostname`; shows hops/since/flags; expanded rows may show host/node/claim IDs and capabilities. |
413 +| `Retention` table | `tables.actor.retention` | Covered by current typed table. Recipe maps `actor` or `observer_actor` to actor link depending on selected actor type, then status/from/to/duration/metrics/instances/contexts. |
414 +| Highlight path | Existing SOW-0021 `data.presentation.selection.highlight_path` using `stream_path` | Covered outside modal table composition. |
415 +| Depth-1 miniature | Incident streaming/virtual/stale links and opposite actors | No new data. |
416 +
417 +Streaming validation result:
418 +
419 +- Relationship tables are already modeled correctly and compactly.
420 +- Required canonical additions: `actor_labels`, host labels, and host/system
421 + metadata labels for OS/architecture/CPU parity.
422 +- Required schema additions: recipes that project actor refs as old
423 + actor-link cells, fallback actor labels for path rows, and expanded-row
424 + visibility for host/node/claim/capability fields.
425 +
426 +#### topology:snmp Mapping
427 +
428 +| Old modal/table item | Current/new canonical source | Recipe/status |
429 +|---|---|---|
430 +| Actor name | `actors.display_name` / `actors.sys_name` via label policy | Covered by current actor columns and label policy. |
431 +| Actor labels | New `actor_labels` rows from actor labels and scalar/array metadata currently stored in `actor_metadata.attributes` | Add `actor_labels`; repeated array values should be repeated rows. Do not render raw `actor_metadata` as the user-facing label view. |
432 +| Device summary: type, source, layer | `actors.type`, `actors.source`, `actors.layer` | Covered. |
433 +| Device summary: vendor/model/sys description/location/contact/protocols/capabilities/ports/VLAN/FDB/LLDP/CDP/chart/netdata host | Currently mostly inside raw `actor_metadata.attributes`; some identity fields are in `actors` | Move important scalar/count fields into typed actor columns or typed actor-detail columns; also expose as `actor_labels`. This avoids raw JSON and supports sorting/filtering. |
434 +| Segment summary: type/source/layer | `actors` columns | Covered. |
435 +| Segment summary: learned sources, ports total, endpoints total | Currently attributes/labels, not typed actor columns | Add typed fields or `actor_labels` rows; use typed columns if used for sorting/filtering. |
436 +| Endpoint summary: type/source/layer | `actors` columns | Covered. |
437 +| Endpoint summary: vendor, learned sources | Currently attributes/labels, not typed actor columns | Add typed fields or `actor_labels` rows. |
438 +| Device `Ports` table | Actor-owned port table currently derived from old `ports` rows | Partly covered. Current required `actor_ports` type only declares `name`, `topology_role`, `oper_status`, and `link_mode`; dynamic actor tables can carry more, but schema/presentation is not curated. Need a stable `actor_ports` inventory/detail table with visible columns for name/status/admin/type/mode/role/STP/VLAN/FDB/link/neighbor counts and expanded columns for aliases, speeds, chart refs, and neighbor details. |
439 +| Device `Links` table | `links` plus evidence endpoint fields | Partly covered. Current SNMP evidence stores `src_endpoint`, `dst_endpoint`, and `metrics` as JSON. Need structured endpoint columns such as source/destination port ID/name/if_index/if_name/display name/management IP and structured metric fields needed by link modals. Recipe uses conditional local/remote projection based on whether selected actor is `src_actor` or `dst_actor`. |
440 +| Link protocol/direction/state | `links.protocol`, `links.direction`, `links.state` | Covered. |
441 +| Inferred vs verified link distinction | `links.type` and link presentation from SOW-0021 | Covered for graph; modal/legend recipes should expose link type/status. |
442 +| Raw neighbors/endpoint objects | Existing nested JSON values | Should not render raw in main tables. Map to counts in table rows, and expose detailed nested information only through explicit expanded sections or typed child tables. |
443 +| Depth-1 miniature | Incident L2 links and opposite actors | No new data. Mini graph can use existing link presentation, including inferred/verified link styles. |
444 +
445 +SNMP validation result:
446 +
447 +- Current schema preserves most facts, but too many user-facing facts are only
448 + reachable through JSON.
449 +- Required canonical additions: `actor_labels`; typed actor/detail columns for
450 + important summary fields; stable, richer `actor_ports` table; structured
451 + SNMP evidence endpoint/metric columns replacing user-facing dependence on
452 + `src_endpoint`, `dst_endpoint`, and `metrics` JSON.
453 +- Required schema additions: conditional local/remote endpoint projections,
454 + expanded-row visibility, and explicit debug-only handling for any remaining
455 + raw JSON.
456 +
457 +#### vSphere Mapping
458 +
459 +- vSphere is still legacy and tracked by a later SOW. The SOW-0022 contract
460 + must be generic enough for vSphere inventory actors and relationship tables:
461 + actor labels, mini depth-1 topology, actor/link tables, actor-link cells, and
462 + expandable rows.
463 +- No vSphere producer changes should happen in this worktree without user
464 + coordination.
465 +
466 +### Validation Outcome Before Implementation
467 +
468 +The mapping exercise shows that implementation can proceed after the schema
469 +adds the following generic primitives:
470 +
471 +1. `actor_labels` as a first-class actor-owned table type.
472 +2. Modal/table composition recipes on actor/link types or table types.
473 +3. Cell annotations including `actor_link`, badge, number, timestamp, duration,
474 + endpoint, and debug JSON.
475 +4. Projection primitives for direct columns, actor-ref labels, opposite actor,
476 + conditional local/remote endpoint columns, formatted endpoints, label-table
477 + lookup, and explicitly declared scalar JSON paths.
478 +5. Column/row visibility for table view, expanded row, hidden, and debug.
479 +6. Mini topology composition from existing incident links and actors, with
480 + optional link-type filters.
481 +
482 +The mapping also shows producer-specific canonical field work:
483 +
484 +1. `topology:network-connections`: emit process `username`, process `cmdline`,
485 + `actor_labels`, and self `local_ip_count` if retained.
486 +2. `topology:streaming`: emit `actor_labels`, complete host labels, and
487 + host/system metadata labels used by old summaries.
488 +3. `topology:snmp`: replace user-facing JSON dependence with `actor_labels`,
489 + typed summary fields, richer stable port rows, and structured link endpoint
490 + evidence columns.
491 +
492 +## Pre-Implementation Gate
493 +
494 +Status at implementation start: ready (historical snapshot; current SOW state is recorded in the top-level Status section).
495 +
496 +Problem / root-cause model:
497 +
498 +- Modal/table composition is underspecified. Producers can preserve useful facts, but the UI lacks enough schema-level guidance to turn those facts into polished modal content without raw JSON fallback or producer-specific hardcoding.
499 +
500 +Evidence reviewed:
501 +
502 +- User-provided examples of raw JSON leaking into final UI. Raw examples are intentionally not copied into this durable artifact.
503 +- Old schema support for modal composition is defined in
504 + `src/plugins.d/FUNCTION_UI_SCHEMA.json:286-372`.
505 +- Old SNMP modal composition is defined in
506 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation_schema.go:7-96`.
507 +- V1 table type metadata stops at structural table classification in
508 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:1211-1260`.
509 +- V1 SNMP currently emits raw actor metadata JSON in
510 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:687-713`.
511 +
512 +Affected contracts and surfaces:
513 +
514 +- `netdata.topology.v1` table/detail schema.
515 +- Cloud frontend actor/link modal renderer.
516 +- Cloud topology aggregator table merge behavior.
517 +- Backend topology producers.
518 +- Developer guide, topology spec, and `project-create-topology` skill.
519 +
520 +Existing patterns to reuse:
521 +
522 +- SOW-0021 presentation profiles.
523 +- SOW-0023 pure correlation actors, points, claims, and semantic correlation
524 + link types.
525 +- Existing compact table roles and column metadata.
526 +- Old topology presentation table and modal-tab metadata as inventory input.
527 +
528 +Risk and blast radius:
529 +
530 +- Applies to all topology producers and the Cloud UI.
531 +- Can affect payload size if table presentation metadata is repeated per row.
532 +- Raw data examples may contain sensitive information and must remain sanitized.
533 +
534 +Sensitive data handling plan:
535 +
536 +- Do not copy raw customer/infrastructure details into durable artifacts.
537 +- Use sanitized fixtures and placeholders only.
538 +- Keep any raw captures under `.local/`.
539 +
540 +Implementation plan:
541 +
542 +1. Record the chosen table/modal composition contract.
543 +2. Inventory current modal/table behavior across producers and UI.
544 +3. Define compact table-composition profiles without high-cardinality
545 + repetition.
546 +4. Update schema/docs/skill/specs and backend producers.
547 +5. Create Cloud frontend and Cloud aggregator handoff artifacts.
548 +6. Validate with sanitized fixtures.
549 +
550 +Validation plan:
551 +
552 +- Pending SOW-0021 and SOW-0023 output.
553 +
554 +Artifact impact plan:
555 +
556 +- AGENTS.md: no expected update unless workflow rules change.
557 +- Specs: likely update `.agents/sow/specs/topology-function-schema.md`.
558 +- End-user/operator docs: likely update `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`.
559 +- Runtime project skills: likely update `.agents/skills/project-create-topology/SKILL.md`.
560 +- End-user/operator skills: likely unaffected unless public operator workflows change.
561 +- SOW lifecycle: current/in-progress; close only after implementation,
562 + integrated validation, and commit.
563 +
564 +Open-source reference evidence:
565 +
566 +- No external OSS reference was used for this contract pass. The work is a
567 + Netdata-specific payload/UI contract derived from old Netdata topology modal
568 + behavior and current `netdata.topology.v1` producer facts.
569 +
570 +Open decisions:
571 +
572 +- Resolved: table/modal composition metadata lives in actor/link type
573 + `presentation.modal`, with reusable table defaults in
574 + `types.table_types.<id>.presentation`.
575 +- Resolved: actor and link modal composition belong to the same contract so
576 + table recipes and cell/projection tokens stay consistent.
577 +- Resolved: raw `json` cells are hidden/debug-only unless an explicit schema
578 + projection extracts a curated scalar value or a future structured child table
579 + is defined.
580 +
581 +## Implications And Decisions
582 +
583 +1. User decision: table and actor-modal composition is separate from topology presentation and should be handled as SOW-0022.
584 +2. User decision: document the target contract first, then implement. Before
585 + Agent implementation starts, create separate Cloud aggregator and Cloud UI
586 + handoff artifacts.
587 +3. Contract decision: modal sections are recipes over existing facts; they do
588 + not duplicate high-cardinality rows or raw actor metadata for display.
589 +4. Contract decision: actor labels use a compact actor-owned
590 + `actor_labels(actor, key, value, source?, kind?, value_index?)` table.
591 +5. Contract decision: the Cloud UI must reuse existing topology modal/table
592 + components where practical and must not reimplement the old table stack from
593 + scratch for v1.
594 +
595 +## Plan
596 +
597 +1. Update topology schema, developer guide, durable spec, and topology
598 + producer skill with the modal/table composition contract.
599 +2. Create a pending Cloud topology service SOW for aggregation behavior.
600 +3. Create a Cloud frontend TODO for UI behavior and component reuse.
601 +4. Implement Agent producer changes for network-connections, streaming, and
602 + SNMP/L2 after the contract/handoffs are accepted.
603 +5. Wait for Cloud UI and Cloud aggregator implementation slices, then run
604 + integrated QA before closing this SOW.
605 +
606 +## Execution Log
607 +
608 +### 2026-05-09
609 +
610 +- Opened as pending follow-up from user direction.
611 +
612 +### 2026-05-10
613 +
614 +- Updated dependencies after SOW-0023 added the correlation plane and link
615 + layout contract. Actor modals must not expose correlation point/claim tables
616 + as raw user-facing content.
617 +- Completed the per-function mapping exercise for network-connections,
618 + streaming, SNMP/L2, and future vSphere migration.
619 +- Began contract-first update of schema/spec/developer-guide/skill before
620 + Agent producer implementation.
621 +- Moved SOW-0022 to current/in-progress after the user approved Agent backend
622 + implementation. Cloud aggregator SOW-0009 remains pending in the service
623 + repo until the service worker finishes SOW-0008.
624 +- Implemented shared Go `netdata.topology.v1` modal/table composition structs
625 + and semantic validation for modal labels, mini topology link filters,
626 + section sources, owner filters, projections, cell visibility, table type
627 + presentation, and sort columns.
628 +- Implemented `topology:network-connections` producer additions:
629 + `actor_labels`, process `username`, process `cmdline`, self
630 + `local_ip_count`, modal recipes over graph links, socket evidence, and
631 + `socket_ports`.
632 +- Implemented `topology:streaming` producer additions: complete host-label
633 + export into `actor_labels`, system metadata labels/actor columns for OS,
634 + architecture, and CPU count, graph-link `port_name`, and modal recipes over
635 + existing `stream_path`, `retention`, `inbound`, and `outbound` tables.
636 +- Implemented `topology:snmp` producer additions: `actor_labels`, typed summary
637 + fields, stable `actor_ports`, structured evidence endpoint columns, and
638 + actor modal recipes that avoid raw JSON as the default user-facing view.
639 +- Added shared schema tests for v1 modal recipes and SNMP producer assertions
640 + for `actor_labels` and modal presence.
641 +- Ran read-only reviews with GLM, Kimi, MiMo, Qwen, and MiniMax against this
642 + SOW and the uncommitted implementation.
643 +- Addressed reviewer findings:
644 + - required `value` for `const` modal projections in schema and semantic
645 + validation;
646 + - required a local and remote side for `selected_side_endpoint` projections;
647 + - preserved explicit SNMP zero-valued counts/indexes instead of treating
648 + them as missing;
649 + - removed the misleading SNMP `actor_ports` table-type overwrite pattern;
650 + - added SNMP evidence-column and zero-preservation assertions.
651 +- Ran a second read-only review round after those fixes, then addressed the
652 + concrete findings:
653 + - normalized empty nullable SNMP `protocols` and `capabilities` arrays to
654 + `null`;
655 + - replaced fragile SNMP empty-array type assertions with a helper;
656 + - tightened `selected_side_endpoint` semantic validation so empty-string side
657 + columns do not satisfy the local/remote requirement;
658 + - made evidence-section validation return an error instead of panicking on a
659 + malformed section;
660 + - deduplicated the SNMP port modal column recipe shared by the device modal
661 + and `actor_ports` table presentation;
662 + - added tests for invalid evidence-section shape, empty selected-side column
663 + validation, and nullable SNMP array normalization.
664 +- Ran a third read-only review round with the same scope. Concrete fixes from
665 + that round:
666 + - aligned JSON schema and semantic validation for `label_lookup`,
667 + `json_path`, `coalesce`, and row-filter `value`/`values` requirements;
668 + - added schema and semantic negative tests for invalid modal projections and
669 + row filters;
670 + - documented that `actor_labels` logical string fields may be encoded as
671 + `string` or `string_ref`, and that aggregators/UI adapters must normalize
672 + both encodings;
673 + - removed a stale modal-column validator parameter;
674 + - documented and tested the SNMP `protocols` fallback from legacy
675 + `learned_sources`;
676 + - extended the SNMP test table decoder to support `dict` encodings.
677 +- Ran a fourth read-only review round with the same scope. Concrete fixes from
678 + that round:
679 + - required `formatted_endpoint` projections to name at least an IP or port
680 + column in JSON schema and Go semantic validation;
681 + - made semantic validation reject explicit non-array `modal.sections`;
682 + - made the modal projection switch fail closed if a future unsupported kind
683 + reaches the semantic validator;
684 + - documented that `actor_labels` inherits topology Function sensitive-data
685 + access-control assumptions;
686 + - added regression tests for empty `formatted_endpoint` and malformed
687 + `modal.sections`.
688 +
689 +## Validation
690 +
691 +Acceptance criteria evidence:
692 +
693 +- Contract artifacts now define `actor_labels`, modal sections, source kinds,
694 + owner filters, projections, cell types, visibility, table presentation, and
695 + raw JSON/debug rules in:
696 + - `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
697 + - `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
698 + - `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`
699 + - `.agents/sow/specs/topology-function-schema.md`
700 + - `.agents/skills/project-create-topology/SKILL.md`
701 +- Cloud frontend handoff created at
702 + `../../dashboard/cloud-frontend/TODO-topology-modal-composition-contract.md`.
703 +- Cloud aggregator handoff SOW created at
704 + `../../netdata/cloud-topology-service/.agents/sow/done/SOW-0009-20260510-modal-composition-and-actor-labels.md`.
705 +
706 +Tests or equivalent validation:
707 +
708 +- `python -m json.tool src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
709 +- `git diff --check`
710 +- `cd src/go && go test ./pkg/topology/v1 ./plugin/go.d/collector/snmp_topology`
711 +- C syntax/type checks were run manually with the exact include/define flags
712 + from `build/compile_commands.json`, replacing object output with
713 + `-fsyntax-only`, for:
714 + - `src/collectors/network-viewer.plugin/network-viewer.c`
715 + - `src/web/api/functions/function-topology-streaming.c`
716 +- `git -C ../../dashboard/cloud-frontend diff --check TODO-topology-modal-composition-contract.md`
717 +- `git -C ../../netdata/cloud-topology-service diff --check .agents/sow/done/SOW-0009-20260510-modal-composition-and-actor-labels.md`
718 +
719 +Real-use evidence:
720 +
721 +- Not run yet for the completed producer implementation. The local `build/`
722 + directory is root-owned, so targeted Ninja object rebuilds could not write
723 + `.ninja_lock` or `.ninja_log`; syntax/type checks were run instead with the
724 + exact compile flags. Full real-use evidence requires rebuilding/installing
725 + the Agent and validating with the updated Cloud UI/aggregator path.
726 +
727 +Reviewer findings:
728 +
729 +- GLM, Kimi, MiMo, Qwen, and MiniMax completed read-only reviews of
730 + `.agents/sow/current/SOW-0022-20260509-topology-table-composition.md` and
731 + the uncommitted implementation.
732 +- Findings accepted and fixed:
733 + - `const` projections were schema-valid without a `value`;
734 + - `selected_side_endpoint` projections were schema-valid with no usable
735 + endpoint side columns;
736 + - SNMP nullable integer handling collapsed explicit zero values into missing
737 + values;
738 + - SNMP `actor_ports` table-type construction had a confusing overwrite;
739 + - SNMP structured evidence columns needed direct test coverage.
740 +- Findings reviewed but not changed:
741 + - empty `actor_labels` tables are currently allowed for a stable table
742 + contract and tiny overhead;
743 + - raw/debug JSON remains available only through explicit debug visibility;
744 + - Cloud UI projection-engine coverage remains tracked in the Cloud frontend
745 + TODO, not in the Agent producer implementation;
746 + - modal recipes are intentionally type-level payload metadata; they are not
747 + repeated per high-cardinality row;
748 + - modal sources may reference declared table types even when a runtime table
749 + has zero rows or is omitted, allowing the UI to render stable empty
750 + sections.
751 +- Round-2 findings accepted and fixed:
752 + - empty nullable SNMP `protocols`/`capabilities` arrays normalized to `null`;
753 + - fragile empty-array type assertions replaced with `isEmptyArrayCell`;
754 + - `selected_side_endpoint` semantic validation now rejects empty-string side
755 + columns;
756 + - malformed evidence section objects now return validation errors;
757 + - duplicated SNMP port modal column recipe collapsed into one helper.
758 +- Round-3 findings accepted and fixed:
759 + - JSON schema now conditionally requires `label_key`, `path`, non-empty
760 + `columns`, and row-filter `value`/`values` in the same cases covered by
761 + semantic validation;
762 + - Go semantic validation now rejects row filters that omit required values;
763 + - `actor_labels` encoding compatibility is documented for direct strings and
764 + dictionary references;
765 + - SNMP `protocols` legacy fallback is explicit and covered by producer tests;
766 + - SNMP test helpers now decode `dict` compact-table columns.
767 +- Round-3 findings reviewed but not changed:
768 + - streaming may expose both curated system labels and complete host labels;
769 + complete host labels are an explicit product requirement and UI grouping can
770 + decide how to present duplicates;
771 + - streaming `actor_labels` table type does not need a table-presentation
772 + fallback because actor modal labels are driven by `presentation.modal.labels`;
773 + - C producer modal helper duplication is acceptable for now because the two C
774 + producers do not share a common topology-emitter module;
775 + - SNMP `sys_contact` and `sys_location` remain visible because topology
776 + Functions are already marked sensitive and access-controlled by the admin.
777 +- Round-4 findings accepted and fixed:
778 + - empty `formatted_endpoint` projections now fail schema and semantic
779 + validation;
780 + - explicit non-array `modal.sections` values now fail semantic validation;
781 + - semantic modal projection validation fails closed for unsupported kinds;
782 + - `actor_labels` sensitive-data inheritance is documented in the developer
783 + guide, durable spec, and topology producer skill.
784 +- Round-4 findings reviewed but not changed:
785 + - C producer test coverage remains an integrated-QA risk because current local
786 + C Function testing is manual/syntax-level only;
787 + - C modal helper duplication remains accepted for this SOW;
788 + - plain `string` actor-label columns in C producers are allowed by the
789 + documented contract and avoid adding new C dictionary infrastructure here;
790 + - payload-size growth remains a pre-close measurement/integrated-QA item.
791 +- Round-5 findings accepted and fixed:
792 + - `selected_side_endpoint` projections are now self-contained: schema,
793 + semantic validation, producer docs, durable spec, and topology producer
794 + skill require source/destination actor-ref columns plus both endpoint
795 + sides;
796 + - network-connections socket modal projections now emit those side actor
797 + columns;
798 + - SNMP device link modal columns now use selected-side projections for
799 + `Local Port` and `Remote Port` instead of source/destination labels;
800 + - semantic validation now rejects explicitly configured optional actor-label
801 + columns when the referenced column does not exist, while still allowing
802 + optional label columns to be omitted from a table type;
803 + - `actor_labels` column roles are aligned across producers by marking label
804 + key/value/source/kind/index as attributes and actor as the reference.
805 +- Round-5 findings reviewed but not changed:
806 + - C producer modal helper duplication remains accepted for this SOW;
807 + - C producer modal JSON has syntax/type validation but not dedicated C unit
808 + tests yet;
809 + - payload-size growth remains a pre-close measurement/integrated-QA item
810 + because realistic local/Cloud payloads are needed to measure the final
811 + producer output.
812 +- Round-6 findings accepted and fixed:
813 + - `ValidateDecodedResponse` now fails closed for malformed response
814 + envelopes, missing `data`, and wrong schema versions;
815 + - compact-table semantic validation now rejects empty column ids, duplicate
816 + column ids, missing column types, unsupported column types, and primitive
817 + values that do not match the declared column type;
818 + - modal section and modal column semantic validation now rejects missing or
819 + empty `id` and `label`;
820 + - JSON schema now mirrors modal source, owner-filter, and projection
821 + conditional requirements for `table`, `evidence`, actor/link side columns,
822 + and direct/actor/opposite projection columns;
823 + - SNMP actor-owned custom tables named `labels` or `metadata` can no longer
824 + overwrite the built-in `actor_labels` or `actor_metadata` tables;
825 + - streaming graph link rows now emit the metric columns declared by their
826 + link type aggregation policy;
827 + - streaming `info` requests now return metadata without building or emitting
828 + the full topology payload.
829 +- Round-6 findings reviewed but not changed:
830 + - C helper duplication remains accepted until/unless a shared C topology
831 + emitter module is introduced;
832 + - payload-size growth remains a pre-close measurement item;
833 + - dedicated C producer unit/snapshot tests remain an integrated-QA risk
834 + unless a small Function fixture harness is added before close.
835 +- Round-7 findings accepted and fixed:
836 + - JSON schema now mirrors semantic validation for `json_path` by requiring
837 + both `column` and `path`;
838 + - `modal_section.label`, `label_key`, and `json_path.path` now reject empty
839 + strings at schema level where applicable;
840 + - semantic validation now rejects modal sections with missing or empty
841 + `columns`, actor type presentation labels that are explicitly empty, and
842 + non-integer modal mini-topology `depth`;
843 + - semantic validation now reports empty `row_filters[].values` separately
844 + from wrong-type or missing `values`;
845 + - SNMP custom actor-detail table ids now reserve built-in replacement ids and
846 + generate unique ids, so `labels`, `metadata`, `custom_labels`, or similar
847 + table names cannot overwrite one another;
848 + - SNMP `actor_ports` now preserves unknown custom port fields in an `extra`
849 + debug JSON column instead of dropping them when normalizing the stable port
850 + table;
851 + - SNMP `vlan_ids` and similar array labels now stringify scalar typed arrays,
852 + including integer arrays, instead of accepting only `[]string`/`[]any`;
853 + - SNMP neighbor-count inference now runs only when the `neighbors` value is a
854 + valid array of objects, avoiding a misleading zero for malformed values;
855 + - docs and the topology producer skill now clarify that Function `info`
856 + responses are metadata-only and are not validated as full topology payloads;
857 + - docs now describe `empty_label`, `badge_map`, `align`, `sortable`, optional
858 + `label_lookup.actor_column`, and required `json_path.column`/`path`.
859 +- Round-7 findings reviewed but not changed:
860 + - `label_lookup.actor_column` remains optional by design. When omitted, the UI
861 + should look up labels for the selected modal actor; producers provide
862 + `actor_column` only for source-row actor references.
863 +- Round-8 findings accepted and fixed:
864 + - JSON schema now gives `modal_section.label`, `link_type_presentation.label`,
865 + `port_type_presentation.label`, and `table_type_presentation.label`
866 + `minLength: 1`, matching semantic validation for explicit empty labels;
867 + - semantic validation now rejects explicit empty labels in link type, port
868 + type, and table type presentation, in addition to actor type and modal
869 + section/column labels;
870 + - validation tests now cover explicit empty presentation labels for actor,
871 + link, port, and table type presentation;
872 + - the current SOW file was marked with git intent-to-add so `git diff` based
873 + reviewers see the pending-to-current SOW move before the final commit.
874 +- Round-8 findings reviewed but not changed:
875 + - `presentation.modal.sections: []` remains valid. A modal may provide labels
876 + and/or a mini-topology without table sections, and producers with no curated
877 + tables should not be forced to invent empty sections;
878 + - SNMP `anyStringSlice` keeps the small reflection fallback to preserve scalar
879 + typed arrays from non-JSON producers without adding a long type-switch;
880 + - C producer unit/snapshot tests and shared C modal emitter refactoring remain
881 + integrated-QA or follow-up risks already tracked in this SOW;
882 + - table type presentation validation intentionally validates type-registry
883 + table definitions. Runtime actor tables are validated separately and
884 + `presentation.modal.labels` already falls back to actor table columns when
885 + resolving label tables.
886 +- Round-9 findings accepted and fixed:
887 + - semantic validation now rejects duplicate modal section ids and duplicate
888 + modal column ids, matching the duplicate-column guard already used for
889 + compact tables;
890 + - the streaming inbound modal recipe now projects the nullable
891 + `source_actor` as `Received from`, matching the old inbound table behavior
892 + and the SOW mapping;
893 + - streaming inbound and outbound modal recipes now expose the already-existing
894 + node type, collected instance/context, ingest age, TLS, alert-count, and
895 + outbound node columns needed to preserve old visible table functionality
896 + without duplicating row data;
897 + - streaming link type aggregation now declares `replication_completion: avg`;
898 + - network-connections ownership links now declare `socket_count: sum`, and
899 + network-connections actor labels include the canonical actor `type`, aligned
900 + with streaming actor-label behavior;
901 + - the Go validator comment now states that `ValidateDecodedResponse` is for
902 + full topology responses, not metadata-only Function `info` responses;
903 + - validation tests now cover the non-object `presentation.modal` shape.
904 +- Round-10 findings reviewed but not changed:
905 + - C modal helper duplication remains accepted and tracked for a future shared
906 + emitter/refactor because the current C producers do not share a topology
907 + JSON helper module;
908 + - C producer unit/snapshot tests, payload-size measurement, and integrated
909 + Agent/UI/aggregator QA remain pre-close or follow-up items tracked in this
910 + SOW;
911 + - SNMP `anyStringSlice` reflection fallback remains intentional for typed
912 + scalar arrays from non-JSON producer data;
913 + - `formatted_endpoint` remains permissive by design: IP-only or port-only
914 + endpoints are valid when a producer has only partial endpoint facts.
915 +- Round-11 final external review:
916 + - GLM, Kimi, MiMo, MiniMax, and Qwen were rerun on the same full SOW-0022
917 + scope after the Round-10 fixes, with only short fix notes appended;
918 + - no reviewer reported a new actionable or blocking issue;
919 + - remaining reviewer notes were informational only and matched already
920 + tracked risks: C modal helper duplication, missing C producer
921 + unit/snapshot tests, payload-size measurement, integrated Agent/UI/
922 + aggregator QA, the intentional SNMP `anyStringSlice` reflection fallback,
923 + and intentionally permissive partial endpoint formatting;
924 + - the final Qwen review also verified that modal/table composition remains
925 + generic, avoids modal-only high-cardinality duplication, and keeps
926 + sensitive actor-label handling documented.
927 +
928 +Same-failure scan:
929 +
930 +- Searched updated contract artifacts for stale relationship-table naming,
931 + old public create-topology skill references, and unresolved
932 + pre-implementation decision markers.
933 +
934 +Sensitive data gate:
935 +
936 +- Raw user-provided examples are not copied into this SOW. This SOW uses sanitized summaries only.
937 +
938 +Artifact maintenance gate:
939 +
940 +- AGENTS.md: no workflow rule change in this step.
941 +- Runtime project skills: updated `.agents/skills/project-create-topology/SKILL.md`.
942 +- Specs: updated `.agents/sow/specs/topology-function-schema.md`.
943 +- End-user/operator docs: no operator workflow change. Updated developer-facing
944 + Function docs under `src/plugins.d/`.
945 +- End-user/operator skills: unaffected; this is developer topology work, not an
946 + operator skill change.
947 +- SOW lifecycle: remains in `current/` with `Status: paused`; do not close
948 + until integrated QA and commit are complete.
949 +
950 +Specs update:
951 +
952 +- Updated `.agents/sow/specs/topology-function-schema.md`.
953 +
954 +Project skills update:
955 +
956 +- Updated `.agents/skills/project-create-topology/SKILL.md`.
957 +
958 +End-user/operator docs update:
959 +
960 +- Not affected. Developer docs updated:
961 + `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` and
962 + `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`.
963 +
964 +End-user/operator skills update:
965 +
966 +- Not affected.
967 +
968 +Lessons:
969 +
970 +- The old modal behavior can be reconstructed without duplicating modal-only
971 + rows only if producers restore missing canonical fields first.
972 +
973 +Follow-up mapping:
974 +
975 +- Cloud frontend implementation is tracked by
976 + `../../dashboard/cloud-frontend/TODO-topology-modal-composition-contract.md`.
977 +- Cloud aggregator implementation is tracked by
978 + `../../netdata/cloud-topology-service/.agents/sow/done/SOW-0009-20260510-modal-composition-and-actor-labels.md`.
979 +- Function-specific modal product composition is intentionally split out and
980 + tracked separately:
981 + - `.agents/sow/done/SOW-0025-20260511-network-connections-modal-product-composition.md`;
982 + - `.agents/sow/done/SOW-0026-20260511-snmp-modal-product-composition.md`;
983 + - `.agents/sow/current/SOW-0027-20260511-streaming-modal-product-composition.md`.
984 +- Integrated Agent/UI/aggregator QA remains in this SOW after the other workers
985 + finish their implementation slices.
986 +
987 +## Outcome
988 +
989 +Contract documentation, Cloud handoff artifacts, shared Go schema validation,
990 +and Agent producer implementation are prepared. Full integrated QA is still
991 +pending before this SOW can close.
992 +
993 +## Lessons Extracted
994 +
995 +- See the Validation section's Lessons entry for the current extracted lesson.
996 +
997 +## Followup
998 +
999 +- See the Validation section's Follow-up mapping entry for the active tracker
1000 + list. This SOW is paused until integrated Agent/UI/aggregator QA can close
1001 + those mapped items.
1002 +
1003 +## Regression Log
1004 +
1005 +None yet.
1006 +
1007 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/current/SOW-0027-20260511-streaming-modal-product-composition.md new
+659
@@ -0,0 +1,659 @@
1 +# SOW-0027 - Streaming Modal Product Composition
2 +
3 +## Status
4 +
5 +Status: in-progress
6 +
7 +Sub-state: reopened for regression on 2026-05-11. Live modal review showed
8 +that the completed work still left streaming relationship sections with
9 +incorrect product semantics, especially retention direction, missing timestamps,
10 +missing received-from actors, and outbound stream ownership. Source repair is
11 +implemented and build-validated; live UI validation is pending install/restart
12 +of the rebuilt Agent. Reopened again on 2026-05-17 after live graph review
13 +showed streaming parents without child bullets and without child-count-based
14 +size emphasis.
15 +
16 +## Requirements
17 +
18 +### Purpose
19 +
20 +Make `topology:streaming` actor modals useful for operators of Netdata parent/child/vnode streaming trees, including path, retention, inbound, outbound, stale, and transit relationships.
21 +
22 +### User Request
23 +
24 +The user reported that streaming modals show wrong or incomplete table information. Example: a parent with many children/vnodes shows a `Retention` tab with one row and no clear node maintaining retention. The user expects the modal model to reflect stale nodes, children, parents, grandparents, great-grandparents, and parent transit responsibilities correctly.
25 +
26 +### Assistant Understanding
27 +
28 +Facts:
29 +
30 +- Current streaming actor modals use one generic modal recipe for all streaming actor types.
31 +- Current modal sections are `Stream path`, `Retention`, `Inbound children`, and `Outbound stream`.
32 +- Current retention rows have both `actor` and `observer_actor`.
33 +- Current retention modal filters by `actor`, so selecting a parent shows retention for that parent, not retention maintained by that parent for other nodes.
34 +- Current outbound modal filters by `actor`, so it shows the selected actor's own outbound stream row, not necessarily children/vnodes passing through that parent.
35 +
36 +Inferences:
37 +
38 +- The current table data has some right primitives but the modal recipes are not role-aware enough.
39 +- A parent actor needs at least two retention views:
40 + - retention for this actor;
41 + - retention this actor maintains for other actors.
42 +- A parent actor likely needs a transit/children view that includes children/vnodes that pass through it, not only the parent's own outbound stream.
43 +- Some rows may be missing for full cloud aggregation semantics, especially if querying several parents where many parents maintain retention for the same child.
44 +
45 +Unknowns:
46 +
47 +- Whether the current streaming Function sees enough local state to emit retention rows for every node whose data is retained by a parent, or only for actors known in the current topology.
48 +- Whether cloud aggregation will merge retention rows from multiple parents without losing `observer_actor`.
49 +- Whether parent transit relationships are fully represented by existing `inbound` rows, existing graph links/evidence, or require a new table/column.
50 +
51 +### Acceptance Criteria
52 +
53 +- A complete inventory exists for streaming modal facts: actors, actor labels, links/evidence, `stream_path`, `retention`, `inbound`, and `outbound`.
54 +- The SOW defines what each streaming actor role should show: self/local parent, parent, child, virtual node, stale node, and inferred path actors.
55 +- Retention tables distinguish "retention for this node" from "retention maintained by this node for others".
56 +- Inbound/outbound/transit tables show children and descendants passing through a parent where the data exists.
57 +- Actor identification/header labels expose important node identity/status fields, not only the generic `Labels` tab.
58 +- Missing data needed for correct modal semantics is identified as producer work, aggregator work, or frontend work.
59 +
60 +## Analysis
61 +
62 +Sources checked:
63 +
64 +- `src/web/api/functions/function-topology-streaming.c:320` stream path row type.
65 +- `src/web/api/functions/function-topology-streaming.c:337` retention row type.
66 +- `src/web/api/functions/function-topology-streaming.c:349` inbound row type.
67 +- `src/web/api/functions/function-topology-streaming.c:367` outbound row type.
68 +- `src/web/api/functions/function-topology-streaming.c:1130` retention row construction.
69 +- `src/web/api/functions/function-topology-streaming.c:1167` inbound row construction.
70 +- `src/web/api/functions/function-topology-streaming.c:1203` outbound row construction.
71 +- `src/web/api/functions/function-topology-streaming.c:1296` stream path columns.
72 +- `src/web/api/functions/function-topology-streaming.c:1313` retention columns.
73 +- `src/web/api/functions/function-topology-streaming.c:1325` inbound columns.
74 +- `src/web/api/functions/function-topology-streaming.c:1342` outbound columns.
75 +- `src/web/api/functions/function-topology-streaming.c:1443` current modal recipe.
76 +- `.agents/sow/specs/topology-function-schema.md:390` streaming modal composition notes.
77 +
78 +Current state:
79 +
80 +- The current modal recipe is generic across streaming actor types.
81 +- `Retention` section filters `retention` rows by `actor`, which answers "what retention exists for the selected node", not "what retention this selected parent maintains for other nodes".
82 +- The `Retention` section does not display `observer_actor`, even though the table has that column.
83 +- `Outbound stream` filters by `actor`, which answers "where this actor sends", not "which children/vnodes are sent through this parent".
84 +- `Inbound children` filters by `parent_actor`, which is closer to the parent view but may still not cover all transit/descendant questions.
85 +
86 +Available facts to inventory:
87 +
88 +- Actor labels:
89 + - display name, hostname, machine GUID, node ID, type, severity, ephemerality, ingest status, stream status, ML status, agent/version, health status, OS/architecture/CPU, child/alert counts, host labels.
90 +- Stream path rows:
91 + - selected actor, path actor, path index, hostname, host ID, node ID, claim ID, hops, since/first-time, capabilities/flags.
92 +- Retention rows:
93 + - actor whose data is retained, observer actor that maintains the data, DB status, time range, duration, metrics, instances, contexts.
94 +- Inbound rows:
95 + - parent actor, child actor, optional source actor, received type, ingest status, hops, collected metrics/instances/contexts, replication completion, ingest age, TLS, alert counts.
96 +- Outbound rows:
97 + - actor, destination actor, stream status, hops, TLS, compression.
98 +- Links/evidence:
99 + - directed streaming relationships with port name and collected/replication metrics.
100 +
101 +Target audience and questions:
102 +
103 +- Netdata operator looking at a child/vnode:
104 + - What is this node?
105 + - What is its path to cloud/parents?
106 + - Which parent receives it?
107 + - Who retains its data and over what time range?
108 + - Is it stale/virtual/healthy?
109 +- Netdata operator looking at a parent:
110 + - Which children/vnodes does this parent receive directly?
111 + - Which descendants pass through this parent?
112 + - Which nodes' data does this parent retain?
113 + - Where does this parent send data upstream?
114 + - What is the replication status and health of each stream?
115 +- Netdata operator looking at stale or virtual actors:
116 + - Why is this actor present?
117 + - When was it first/last observed?
118 + - Which path or retention state still references it?
119 +
120 +Risks:
121 +
122 +- A retention table that hides `observer_actor` is actively misleading in aggregated/cloud views where multiple parents may retain the same node.
123 +- A parent modal that only shows its own outbound stream misses its operational responsibility for children passing through it.
124 +- A generic recipe for all roles may be too simple; role-specific sections may be required.
125 +
126 +## Pre-Implementation Gate
127 +
128 +Status at implementation start: ready for implementation (historical snapshot;
129 +current SOW state is recorded in the top-level Status section).
130 +
131 +Problem / root-cause model:
132 +
133 +- The current streaming modal recipes expose local source tables but do not encode the operational roles of a selected actor.
134 +- The retention table has the key `observer_actor` fact but the current modal recipe neither filters by it nor displays it, so parent responsibility is hidden.
135 +- The inbound table already models descendants received through a parent; the current modal label `Inbound children` under-describes transit/descendant responsibility and makes the table look incomplete.
136 +
137 +Evidence reviewed:
138 +
139 +- Retention columns include `actor` and `observer_actor` in `src/web/api/functions/function-topology-streaming.c:1313-1323`.
140 +- The modal retention section currently filters by `actor` in `src/web/api/functions/function-topology-streaming.c:1484-1495`.
141 +- Inbound rows include `parent_actor`, `child_actor`, and `source_actor` in `src/web/api/functions/function-topology-streaming.c:1325-1340`.
142 +- Outbound rows include only `actor` and `destination_actor` plus stream attributes in `src/web/api/functions/function-topology-streaming.c:1342-1349`.
143 +- Descendant rows are populated into `parent_descendants` in `src/web/api/functions/function-topology-streaming.c:2546-2604` and then emitted as inbound rows in `src/web/api/functions/function-topology-streaming.c:1145-1192`.
144 +- Actor labels already include display name, hostname, machine GUID, node ID, type, stream/ingest/health status, agent fields, system fields, child count, alert counts, and full host labels where available in `src/web/api/functions/function-topology-streaming.c:549-589`.
145 +
146 +Affected contracts and surfaces:
147 +
148 +- Agent Function payload for `topology:streaming`.
149 +- Streaming C topology Function row construction and modal recipes.
150 +- Cloud aggregator retention/actor-table merge behavior.
151 +- Cloud frontend actor modal rendering and identity/header display.
152 +- Developer guide, topology spec, project topology skill.
153 +
154 +Existing patterns to reuse:
155 +
156 +- Actor-owned `stream_path`, `retention`, `inbound`, and `outbound` tables.
157 +- `actor_ref_label` and `label_lookup` projections.
158 +- Separate table sections filtered by different actor-ref columns.
159 +- Actor labels for identity/status facts.
160 +
161 +Risk and blast radius:
162 +
163 +- User-facing streaming modal behavior changes.
164 +- Aggregation semantics are important because a cloud topology can contain many parents reporting retention for the same node.
165 +- Sensitive data risk includes host labels, node IDs, claim IDs, machine GUIDs, and private hostnames; durable artifacts must use synthetic examples only.
166 +
167 +Sensitive data handling plan:
168 +
169 +- Do not copy raw host labels, machine GUIDs, claim IDs, hostnames, customer names, private endpoints, or production topology payloads into durable artifacts.
170 +- Store real payload captures only under `.local/`.
171 +- Use synthetic parent/child/vnode examples in docs/tests.
172 +
173 +Implementation plan:
174 +
175 +1. Keep one streaming modal recipe for all streaming actor types because the existing tables already use actor-ref owner filters and empty sections naturally disappear or show meaningful empty states per role.
176 +2. Make the recipe role-aware through section labels and owner filters, not by duplicating table data.
177 +3. Add or adjust modal sections for:
178 + - retention for selected node (`actor`);
179 + - retention maintained by selected node (`observer_actor`);
180 + - received/transit descendants (`parent_actor`);
181 + - upstream stream from selected node (`actor`).
182 +4. Display `observer_actor` in the selected-node retention view and `actor` in the maintained-retention view.
183 +5. Add important actor identification/header fields backed by `actor_labels`: health status and child count, while keeping full labels in the Labels tab.
184 +6. Add missing columns/rows only if current canonical tables cannot answer the required operational questions.
185 +7. Validate with local payload/schema checks and mark cloud/multi-parent validation as external if no aggregated streaming fixture is available in this repository.
186 +
187 +Validation plan:
188 +
189 +- C syntax check for `function-topology-streaming.c`.
190 +- Schema validation of generated streaming payloads.
191 +- Local Function call on a parent with children/vnodes.
192 +- Verify modal rows for:
193 + - selected child/vnode;
194 + - selected parent;
195 + - selected stale node if present.
196 +- Aggregated/cloud payload check that retention rows preserve both retained actor and observer actor.
197 +
198 +Artifact impact plan:
199 +
200 +- AGENTS.md: likely unaffected.
201 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md` if streaming modal guidance changes.
202 +- Specs: update `.agents/sow/specs/topology-function-schema.md`.
203 +- End-user/operator docs: likely unaffected unless Function examples are changed.
204 +- End-user/operator skills: unaffected.
205 +- SOW lifecycle: close only after local and, if available, cloud/aggregated streaming validation.
206 +
207 +Open-source reference evidence:
208 +
209 +- External open-source topology references were not used for implementation authority. This SOW changes Netdata-specific streaming semantics defined by local producer code and topology specs; local Netdata code is the authoritative source.
210 +
211 +Open decisions:
212 +
213 +- Resolved for this SOW: use one recipe with multiple well-labeled sections and owner filters. This avoids repeating modal metadata across actor types while preserving role-specific behavior through table filters.
214 +- Resolved for the regression repair: default streaming modals use concise
215 + operator-facing section names: `Stream path`, `Retained nodes`,
216 + `Received nodes`, and `Outbound streams`.
217 +
218 +## Implications And Decisions
219 +
220 +Decision recorded after the user asked to proceed to SOW-0027:
221 +
222 +- Keep the payload single-source-of-truth: reuse existing `actor_labels`, `stream_path`, `retention`, `inbound`, and `outbound` tables.
223 +- Do not create modal-only duplicate rows.
224 +- Keep one shared streaming actor modal recipe unless implementation proves role-specific recipes are necessary.
225 +- Rename/recompose sections so the modal answers operator questions directly:
226 + - `Retained nodes`: which nodes' data the selected actor maintains.
227 + - `Received nodes`: which children/vnodes/stale descendants are received through the selected parent.
228 + - `Outbound streams`: which node payloads the selected parent sends upstream and where they go.
229 +
230 +## Plan
231 +
232 +1. Inventory old/current streaming modal fields and role-specific facts.
233 +2. Design role-aware streaming actor modals.
234 +3. Identify missing canonical streaming rows/columns.
235 +4. Implement only streaming producer changes after design acceptance.
236 +5. Coordinate frontend/aggregator changes if required.
237 +6. Validate locally and with aggregated/cloud payloads when available.
238 +
239 +## Execution Log
240 +
241 +### 2026-05-11
242 +
243 +- Created SOW from user-reported streaming modal regressions and current code evidence.
244 +- Promoted SOW to current and recorded the implementation decision to keep one shared streaming actor modal recipe with role-aware sections over existing tables.
245 +- Updated `topology:streaming` modal identification fields to include health status and child count.
246 +- Split retention presentation using the existing `retention` table and different owner filters.
247 +- Renamed/recomposed relationship sections as `Received nodes` and `Outbound streams` so they match the underlying `inbound` and `outbound` table semantics.
248 +- Updated topology specs and the project topology skill with streaming modal rules.
249 +
250 +## Validation
251 +
252 +Acceptance criteria evidence:
253 +
254 +- Complete fact inventory recorded in `## Analysis` and `## Pre-Implementation Gate`.
255 +- Actor role expectations recorded under `Target audience and questions`.
256 +- `Retained nodes` now filters the same `retention` table by `observer_actor`, without duplicating retention rows.
257 +- `Received nodes` now explains the existing `inbound` rows as children, virtual nodes, stale nodes, and descendants received through a parent.
258 +- Actor modal identification now includes hostname, node type, stream status, ingest status, health status, retained-node count for parents, direct-child count for parents, OS/platform labels, and Agent version.
259 +- No missing producer rows were found for this SOW's modal fix. Aggregator preservation of multi-parent retention rows is already specified in `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
260 +
261 +Tests or equivalent validation:
262 +
263 +- `git diff --check` passed.
264 +- `(cd src/go && go test -count=1 ./pkg/topology/v1 ./tools/functions-validation/validate)` passed.
265 +- `sudo -n cmake --build build --target netdata -- -j2` passed. The build emitted unrelated protobuf/stringop warnings during link, not streaming topology compile errors.
266 +- `.agents/sow/audit.sh` passed with the pre-existing non-project skill classification warning.
267 +
268 +Real-use evidence:
269 +
270 +- Not performed against the live Agent in this SOW because the built binary was not installed/restarted. A live UI check before install would validate a different binary. The code path was validated by compiling the `netdata` target and by schema/fixture tests for the topology v1 contract.
271 +
272 +Reviewer findings:
273 +
274 +- No external reviewer run was requested for SOW-0027. The change is scoped to producer modal metadata and documentation/spec alignment.
275 +
276 +Same-failure scan:
277 +
278 +- `rg -n "Inbound children|Outbound stream|\"Retention\"|No inbound children|No outbound stream" src/web/api/functions .agents/sow/specs .agents/skills/project-create-topology/SKILL.md` found no remaining stale streaming modal labels.
279 +- `rg -n "retention|observer_actor|retained_nodes|Received nodes|Outbound streams|child_count" src/web/api/functions .agents/sow/specs .agents/skills/project-create-topology/SKILL.md` confirmed the new contract is present in the producer, specs, and project topology skill.
280 +
281 +Sensitive data gate:
282 +
283 +- This SOW uses only path/line evidence and synthetic descriptions. No raw sensitive payload data is included.
284 +
285 +Artifact maintenance gate:
286 +
287 +- AGENTS.md: unchanged. This SOW did not change project-wide workflow or guardrails.
288 +- Runtime project skills: updated `.agents/skills/project-create-topology/SKILL.md` with streaming modal composition rules.
289 +- Specs: updated `.agents/sow/specs/topology-function-schema.md` and `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
290 +- End-user/operator docs: unchanged. This is internal topology payload/modal composition, not a public user command or operator workflow.
291 +- End-user/operator skills: unchanged. No public/operator skill behavior changed.
292 +- SOW lifecycle: reopened from done to current and left paused until live UI validation is performed on an installed/restarted Agent.
293 +
294 +Specs update:
295 +
296 +- Updated `.agents/sow/specs/topology-function-schema.md` with required streaming modal sections and identification labels.
297 +- Updated `.agents/sow/specs/topology-modes-correlation-aggregation.md` with the streaming UI table semantics.
298 +
299 +Project skills update:
300 +
301 +- Updated `.agents/skills/project-create-topology/SKILL.md` with streaming modal rules for future topology producers.
302 +
303 +End-user/operator docs update:
304 +
305 +- Not needed. The Function remains sensitive/internal topology data; no public operator command, UI label documentation, or configuration guide changed.
306 +
307 +End-user/operator skills update:
308 +
309 +- Not needed. Public skills under `docs/netdata-ai/skills/` are for querying/operator workflows, not developer topology modal contracts.
310 +
311 +Lessons:
312 +
313 +- Streaming already had the canonical facts needed for the modal fix. The root problem was recipe composition and labels, not missing table data.
314 +
315 +Follow-up mapping:
316 +
317 +- No new follow-up SOW is required from this change. Live UI validation after installing the rebuilt Agent remains an execution check, not a new product requirement.
318 +
319 +## Outcome
320 +
321 +Implementation complete for the current producer changes; live UI validation
322 +after Agent install is still pending.
323 +
324 +- `topology:streaming` actor modals now expose operator-relevant identity in the modal header.
325 +- Retention is shown in both directions:
326 + - who maintains the selected node;
327 + - which nodes the selected actor maintains.
328 +- Parent responsibility is clearer through `Received nodes`, backed by existing descendant rows.
329 +- `Outbound streams` now lists node payloads sent by the selected parent, including the node and destination per row.
330 +
331 +## Lessons Extracted
332 +
333 +- Reusing the same canonical table through different owner filters is the right pattern for modal composition when the relationship has two actor-ref sides.
334 +- Section names must describe the selected actor's perspective; otherwise correct rows can still appear wrong to operators.
335 +
336 +## Followup
337 +
338 +None.
339 +
340 +## Regression Log
341 +
342 +Note: dated regression entries intentionally use `## Regression - YYYY-MM-DD`
343 +headings to match the repository SOW lifecycle contract.
344 +
345 +## Regression - 2026-05-11 - Streaming Modal Relationship Semantics
346 +
347 +What broke:
348 +
349 +- `Stream path` correctly scopes to the selected actor only, but timestamp
350 + columns may render empty because synthetic path rows do not always carry
351 + `since` and `first_time` values.
352 +- `Retention for node` is misleading in the direct parent modal. The useful
353 + operational view is the parent-owned list of nodes retained by the selected
354 + actor.
355 +- `Retained nodes` rows may render empty `from` and `to` values even though the
356 + retention table is expected to carry database time ranges.
357 +- `Received nodes` may render empty `Received from` values for locally received
358 + rows because the producer leaves `source_actor` empty when the source is
359 + considered local.
360 +- `Upstream stream` currently describes the selected actor's own upstream
361 + stream. For a parent actor, operators need every node payload this parent
362 + sends upstream, with the node and destination shown per row.
363 +
364 +Evidence:
365 +
366 +- `stream_path` rows contain `since_ut` and `first_time_ut` fields in
367 + `src/web/api/functions/function-topology-streaming.c`, but synthetic local
368 + append rows only set actor/path identity and do not set those timestamps.
369 +- `retention` rows contain both retained `actor` and retaining
370 + `observer_actor`, so the same table can answer parent-owned retained-node
371 + views without duplicate rows.
372 +- `inbound` rows contain nullable `source_actor`; local-source rows currently
373 + leave it empty.
374 +- `outbound` rows currently contain `actor` and nullable `destination_actor`
375 + only, so they cannot express "selected parent sends node X to destination Y"
376 + for all descendants.
377 +
378 +Why previous validation missed it:
379 +
380 +- Validation checked schema shape, compilation, and table presence, but did not
381 + inspect a live clustered-parent setup where one parent owns virtual nodes,
382 + retains children, receives descendants, and streams them to another parent.
383 +
384 +Repair plan:
385 +
386 +1. Update the durable topology specs and project topology skill first so future
387 + workers do not repeat the same interpretation mistake.
388 +2. Change the streaming producer modal contract so default visible sections are
389 + `Stream path`, `Retained nodes`, `Received nodes`, and `Outbound streams`.
390 +3. Keep the canonical `retention` table lossless, but remove the confusing
391 + default `Retention for node` modal section unless a future explicitly named
392 + `Retained by` view is designed for aggregated/cloud views.
393 +4. Add or repurpose outbound table columns so the table is owned by the sending
394 + parent and has at least `sender_actor`, `node_actor`, `destination_actor`,
395 + status, age, hops, TLS, compression, and useful counts where available.
396 +5. Populate `source_actor` for received rows whenever the immediate sending
397 + actor is known. For direct local receipt, use the received child/vnode actor
398 + rather than rendering an empty source.
399 +6. Ensure retention and stream-path timestamps are populated from the best
400 + available canonical source and remain nullable only when the Agent genuinely
401 + does not know the value.
402 +
403 +Validation required:
404 +
405 +- Local Function response for a clustered parent with self, virtual nodes,
406 + children, and an upstream clustered parent.
407 +- Verify the selected parent modal shows all retained nodes, all received
408 + nodes, and all outbound node transmissions.
409 +- Verify stale/archived hosts from the Agent root index are included in
410 + retention and received-node rows when present.
411 +- Schema validation and focused build/test commands for the streaming Function.
412 +
413 +Implementation evidence:
414 +
415 +- `src/web/api/functions/function-topology-streaming.c` now backfills stream
416 + path `since` and `first_time` timestamps from the best available host status
417 + source when path rows are missing those values.
418 +- `retention` rows still remain single-source canonical rows, but `db_from`
419 + and `db_to` now fall back to known DB/status timing when the raw retention
420 + range is incomplete.
421 +- Local-source `inbound` rows now set `source_actor` to the known child/vnode
422 + actor, so `Received from` does not render empty for direct local receipt.
423 +- The default modal no longer exposes the misleading `Retention for node`
424 + section. It exposes `Retained nodes`, `Received nodes`, and `Outbound streams`
425 + from canonical tables.
426 +- `outbound` rows now use `sender_actor`, `node_actor`, and
427 + `destination_actor`, so a parent modal can list every node payload that the
428 + selected parent currently sends upstream.
429 +
430 +Validation completed:
431 +
432 +- `git diff --check` passed.
433 +- `cmd=$(jq -r '.[] | select(.file|endswith("src/web/api/functions/function-topology-streaming.c")) | .command' build/compile_commands.json | sed 's# -o [^ ]*# -o /tmp/function-topology-streaming.c.o#'); eval "$cmd"` passed.
434 +- `sudo -n cmake --build build --target netdata -- -j2` passed. The build
435 + emitted unrelated generated protobuf/stringop warnings during link; the
436 + modified streaming topology translation unit compiled.
437 +- `(cd src/go && go test -count=1 ./pkg/topology/v1 ./tools/functions-validation/validate)`
438 + passed.
439 +- `(cd src/go && go test -count=1 ./tools/functions-validation/validate)`
440 + passed after adding the top-level Function envelope `v` schema acceptance
441 + test.
442 +- `rg -n "Inbound children|Outbound stream|Upstream stream|Retention for node|No inbound children|No outbound stream|No upstream stream|No retained nodes" src/web/api/functions .agents/sow/specs .agents/skills/project-create-topology/SKILL.md`
443 + found no stale producer modal labels; remaining `Retention for node`
444 + mentions are explicit spec/skill notes saying that section is not part of the
445 + current default modal.
446 +
447 +Validation still pending:
448 +
449 +- Live Function/UI validation against the running Agent after this rebuilt
450 + binary is installed/restarted. Validating before install would exercise the
451 + old binary.
452 +
453 +## Regression - 2026-05-17 - Streaming Parent Graph Bullets And Size
454 +
455 +What broke:
456 +
457 +- Parent actors in `topology:streaming` do not show child bullets.
458 +- Parent actors appear effectively the same size as ordinary nodes even when
459 + they retain data for many children, virtual nodes, stale nodes, or transit
460 + descendants.
461 +
462 +Evidence:
463 +
464 +- The parent actor type is configured with `show_port_bullets: true` and
465 + data-driven sizing in `src/web/api/functions/function-topology-streaming.c`.
466 +- Streaming graph links are emitted as child/source actor to parent/destination
467 + actor.
468 +- The parent bullet source incorrectly points to `src_actor`; therefore child
469 + bullets are attached to children, not parents.
470 +- The parent size policy uses `link_count`; the intended producer-owned size
471 + metric is the actor row `retained_node_count`.
472 +
473 +Decision:
474 +
475 +- Parent bullets must attach to the parent side of streaming links
476 + (`dst_actor`).
477 +- Parent actor size must use `presentation.size.mode: "metric"` with
478 + `metric_column: "retained_node_count"`.
479 +- `child_count` remains a direct-child count and can still explain immediate
480 + attachments in the actor header.
481 +
482 +Repair plan:
483 +
484 +1. Update the streaming actor-type emitter so actor types can declare metric
485 + sizing and the actor-ref side used by `ports.sources[]`.
486 +2. Configure the `parent` actor type with `metric(retained_node_count)` sizing.
487 +3. Configure parent port bullets to read streaming link rows where
488 + `dst_actor` is the selected actor.
489 +4. Keep child, virtual, and stale actor types fixed-size with no bullets.
490 +
491 +Validation required:
492 +
493 +- Compile `function-topology-streaming.c`.
494 +- Validate the emitted topology schema path with existing Function validation
495 + tests.
496 +- After install, verify a parent actor receives bullets and grows according to
497 + `retained_node_count`.
498 +
499 +Implementation evidence:
500 +
501 +- `src/web/api/functions/function-topology-streaming.c` now lets streaming actor
502 + types declare metric sizing and a port-bullet actor-ref side.
503 +- The `parent` actor type now declares `size.mode: "metric"`,
504 + `metric_column: "retained_node_count"`, and
505 + `ports.sources[].actor_column: "dst_actor"`.
506 +- The actor table now exposes `retained_node_count` as a metric column and
507 + parent actor labels expose it as `Retained Nodes`.
508 +- Child, virtual-node, and stale actor types remain fixed-size and do not show
509 + bullets.
510 +- `.agents/sow/specs/topology-function-schema.md`,
511 + `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, and
512 + `.agents/skills/project-create-topology/SKILL.md` now record that streaming
513 + parent size is `retained_node_count` and parent bullets attach on
514 + `dst_actor`.
515 +
516 +Validation completed:
517 +
518 +- `git diff --check` passed.
519 +- Compile command from `build/compile_commands.json` for
520 + `src/web/api/functions/function-topology-streaming.c` passed.
521 +- `(cd src/go && go test -count=1 ./pkg/topology/v1 ./tools/functions-validation/validate)`
522 + passed.
523 +
524 +Validation still pending:
525 +
526 +- Live Function/UI validation after the rebuilt Agent is installed/restarted.
527 +
528 +## Regression - 2026-05-18 - Streaming Parent Size Must Use Retained Nodes
529 +
530 +What changed:
531 +
532 +- Parent size must be relative to retained nodes, not direct children or graph
533 + degree.
534 +- Retained nodes means every node for which the parent has DB retention state,
535 + including self, virtual nodes, stale nodes, and transit descendants when the
536 + Agent has retained data for them.
537 +
538 +Evidence:
539 +
540 +- The producer already emits a `retention` table with `actor` and
541 + `observer_actor`, so retained-node ownership is already canonical in the v1
542 + payload.
543 +- The previous visual metric used `child_count`, which is a direct-child count
544 + and does not account for nodes received from another parent.
545 +- The existing schema already supports producer-defined actor metric columns and
546 + `presentation.size.metric_column`, so no schema change is required.
547 +
548 +Decision:
549 +
550 +- Add actor metric column `retained_node_count`.
551 +- Count retained nodes from the same DB-retention state that controls whether a
552 + retention row is emitted.
553 +- Size `parent` actors with
554 + `presentation.size.metric_column: "retained_node_count"`.
555 +- Keep `child_count` as a direct-child explanatory metric in the parent modal
556 + header.
557 +
558 +Implementation evidence:
559 +
560 +- `src/web/api/functions/function-topology-streaming.c` now emits
561 + `retained_node_count` in the actor table and actor labels.
562 +- `retained_node_count` uses aggregation `max` because it is an absolute
563 + retaining-parent property, not an additive duplicate-row metric.
564 +- The parent actor type now sizes by `retained_node_count`.
565 +- The Retained Nodes modal header value is backed by the same actor label.
566 +- `.agents/sow/specs/topology-function-schema.md`,
567 + `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, and
568 + `.agents/skills/project-create-topology/SKILL.md` now record retained-node
569 + sizing.
570 +
571 +Validation completed:
572 +
573 +- `git diff --check` passed.
574 +- Syntax-only compile from `build/compile_commands.json` for
575 + `src/web/api/functions/function-topology-streaming.c` passed. Direct object
576 + compile was not used because the local build object's output path is not
577 + writable in this worktree.
578 +- `(cd src/go && go test -count=1 ./pkg/topology/v1 ./tools/functions-validation/validate)`
579 + passed.
580 +- `git fetch upstream master && git rebase --autostash upstream/master` passed.
581 +- Post-rebase `git rev-list --left-right --count upstream/master...HEAD`
582 + reported `0 12`.
583 +
584 +Validation still pending:
585 +
586 +- Live Function/UI validation after the rebuilt Agent is installed/restarted.
587 +
588 +## Regression - 2026-05-18 - Streaming Actor Header Labels
589 +
590 +What broke:
591 +
592 +- The streaming actor modal header still exposes only generic status labels and
593 + long identity fields. It does not promote important host/inventory labels such
594 + as operating system, kernel, hardware model, CPU, RAM, virtualization, cloud
595 + placement, or vnode inventory fields.
596 +
597 +Evidence:
598 +
599 +- A live `topology:streaming` payload showed rich `actor_labels` for host-like
600 + actors, including OS, kernel, architecture, CPU, RAM, virtualization,
601 + container, cloud provider/type/region, and hardware vendor/product labels.
602 +- The same payload showed vnode/inventory labels including vendor, model,
603 + address, location, sys object id, vnode type, and LLDP identity fields.
604 +- The current modal identification recipe emits only hostname, node type,
605 + stream, ingest, health, children, machine GUID, and Agent version.
606 +
607 +Decision:
608 +
609 +- Host-like streaming actors (`parent`, `child`, `stale`) should expose concise
610 + operational identity plus OS/hardware/platform labels in the modal header.
611 +- Parent actors additionally expose `retained_node_count` and `child_count`.
612 +- Vnode actors should expose inventory/device identity labels instead of
613 + host-only OS/hardware labels.
614 +- `machine_guid` and `node_id` remain in the full Labels tab, not in the modal
615 + header, because they are long identifiers rather than human-scannable master
616 + labels.
617 +
618 +Repair plan:
619 +
620 +1. Make the streaming modal identification recipe role-specific by actor type.
621 +2. Use host-like labels for parent, child, and stale actor types.
622 +3. Use vnode/inventory labels for vnode actor types.
623 +4. Keep all labels in `actor_labels`; do not duplicate row data for the modal
624 + header.
625 +
626 +Validation required:
627 +
628 +- Compile `function-topology-streaming.c`.
629 +- Validate the topology schema tooling.
630 +- After install, verify the actor modal header shows the selected role-specific
631 + label set and hides missing labels cleanly.
632 +
633 +Implementation evidence:
634 +
635 +- `src/web/api/functions/function-topology-streaming.c` now emits role-specific
636 + modal identification recipes:
637 + - `parent`, `child`, and `stale` use host-like operational, OS, hardware, and
638 + platform labels;
639 + - `parent` additionally includes `retained_node_count` and `child_count`;
640 + - `vnode` uses inventory/device labels such as vnode type, vendor, model,
641 + address, location, sys object id, and LLDP name.
642 +- The modal header no longer promotes `machine_guid` or `node_id`; those remain
643 + available through the full Labels tab.
644 +- `.agents/sow/specs/topology-function-schema.md`,
645 + `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, and
646 + `.agents/skills/project-create-topology/SKILL.md` now describe the
647 + role-specific streaming identification policy.
648 +
649 +Validation completed:
650 +
651 +- `git diff --check` passed.
652 +- Compile command from `build/compile_commands.json` for
653 + `src/web/api/functions/function-topology-streaming.c` passed.
654 +- `(cd src/go && go test -count=1 ./pkg/topology/v1 ./tools/functions-validation/validate)`
655 + passed.
656 +
657 +Validation still pending:
658 +
659 +- Live Function/UI validation after the rebuilt Agent is installed/restarted.
.agents/sow/current/SOW-0030-20260517-network-connections-dependency-semantics.md new
+381
@@ -0,0 +1,381 @@
1 +# SOW-0030 - Network Connections Dependency Semantics
2 +
3 +## Status
4 +
5 +Status: paused
6 +
7 +Sub-state: selected by the user on 2026-05-17; implementation starting after
8 +pausing unrelated streaming modal validation work. Paused on 2026-05-17 while
9 +the user requested a streaming graph presentation regression fix under
10 +SOW-0027.
11 +
12 +## Requirements
13 +
14 +### Purpose
15 +
16 +Make `topology:network-connections` show real service dependency direction for
17 +sysadmins, DevOps engineers, and SREs: dependency arrows must point from client
18 +to server, actor modals must separate dependencies from dependants, and
19 +aggregated views must collapse remote endpoint noise without leaking
20 +observer-relative socket terminology.
21 +
22 +### User Request
23 +
24 +The user asked for these network-connections-only topology changes:
25 +
26 +- Direction is important and must be painted with an arrow.
27 +- Actor modals must split socket tables into `Dependencies` and `Dependants`.
28 +- Aggregated backend views should collapse inbound connections by remote IP and
29 + outbound connections by remote IP.
30 +- `local` / `remote` names are wrong for topology and must be removed from the
31 + topology contract.
32 +- Prefer `client` / `server`; if `src` / `dst` remains, then `src` must always
33 + mean client and `dst` must always mean server.
34 +- The topology `local` socket filter should be removed.
35 +- Changes must not affect other topology producers.
36 +
37 +### Assistant Understanding
38 +
39 +Facts:
40 +
41 +- The socket scanner already distinguishes local inbound and local outbound
42 + enum values before the network-viewer producer serializes them.
43 +- The current topology payload uses observer-relative `local_*` and `remote_*`
44 + columns in socket evidence and relationship-summary rows.
45 +- The current topology graph link `src_actor` is the process that owns the local
46 + socket row, not necessarily the client actor.
47 +- The current modal recipe has one `Connections`/`Sockets` section for process
48 + actors instead of separate dependencies and dependants.
49 +- The v1 schema already supports directed link types, actor-column owner
50 + filters, formatted endpoint projections, and arbitrary typed relationship
51 + table columns.
52 +
53 +Inferences:
54 +
55 +- No generic JSON Schema mechanism is needed. This is a network-connections
56 + producer contract change.
57 +- Correlation can remain generic if the producer emits correlation claims for
58 + client/server endpoint ownership and correlation points for visible endpoint
59 + actors using the existing declarative correlation tables.
60 +- Aggregated mode can collapse relationship rows by client actor, server actor,
61 + protocol, and state. The endpoint actor identity carries the remote IP
62 + grouping, while detailed evidence preserves exact client/server ports.
63 +
64 +Unknowns:
65 +
66 +- Live UI polish may still need separate cloud-frontend changes after this
67 + payload changes, but the schema already has enough modal primitives for the
68 + producer to declare the desired sections.
69 +
70 +### Acceptance Criteria
71 +
72 +- `topology:network-connections` type definitions declare socket link types as
73 + directed dependency links with forward arrows.
74 +- Topology graph links use `src_actor = client_actor` and
75 + `dst_actor = server_actor` for network dependency links.
76 +- Topology evidence and relationship-summary rows use `client_*` and
77 + `server_*` endpoint columns, not `local_*` / `remote_*`.
78 +- Topology modal recipes expose `Dependencies` and `Dependants` sections for
79 + process actors using actor-column owner filters.
80 +- The topology Function `info`/required params no longer expose a `local`
81 + socket filter. Local sockets are included through inbound/outbound
82 + dependency classification.
83 +- Aggregated topology relationship rows collapse by client actor, server actor,
84 + protocol, and state, with endpoint actor identity carrying remote IP grouping
85 + instead of local ephemeral port grouping.
86 +- Specs, developer guide, project topology skill, validation fixture, and
87 + focused validation tests are updated.
88 +- The change is limited to `topology:network-connections` and does not alter
89 + SNMP, streaming, vSphere, or legacy non-topology schemas except for the shared
90 + direction string mapping that stops rendering local sockets as `local`.
91 +
92 +## Analysis
93 +
94 +Sources checked:
95 +
96 +- `.agents/skills/project-create-topology/SKILL.md`
97 +- `.agents/skills/project-writing-collectors/SKILL.md`
98 +- `.agents/sow/done/SOW-0025-20260511-network-connections-modal-product-composition.md`
99 +- `.agents/sow/done/SOW-0028-20260511-topology-mode-correlation-aggregation.md`
100 +- `.agents/sow/pending/SOW-0029-20260511-network-connections-detailed-loose-sides.md`
101 +- `.agents/sow/specs/topology-function-schema.md`
102 +- `.agents/sow/specs/topology-modes-correlation-aggregation.md`
103 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
104 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
105 +- `src/collectors/network-viewer.plugin/network-viewer.c`
106 +- `src/libnetdata/local-sockets/local-sockets.h`
107 +- `src/go/tools/functions-validation/fixtures/topology-v1/network-connections.json`
108 +- `src/go/tools/functions-validation/validate/main_test.go`
109 +
110 +Current state:
111 +
112 +- Generic v1 link types already define `orientation`, `direction_role`, and
113 + aggregation direction policy.
114 +- Network-connections currently emits `socket`, `endpoint_socket`, and
115 + `correlated_socket` with `direction_role: flow`.
116 +- Network-connections currently emits observer-relative endpoint columns:
117 + `local_ip`, `local_port`, `remote_ip`, `remote_port`.
118 +- The shared `SOCKET_DIRECTION_2str()` map collapses local inbound and local
119 + outbound into the string `local`.
120 +- The topology required params expose a `local` socket filter.
121 +
122 +Risks:
123 +
124 +- Renaming endpoint columns is a contract-breaking change for any external UI
125 + or aggregator code that hardcoded network-connections table columns.
126 +- Getting client/server actor selection wrong would invert dependency arrows,
127 + which is worse than no arrow.
128 +- Aggregating by remote IP can hide exact port-level rows in aggregated mode;
129 + detailed mode must retain exact client/server tuples.
130 +- Existing dirty streaming/spec changes must not be reverted or mixed into this
131 + SOW outcome.
132 +
133 +## Pre-Implementation Gate
134 +
135 +Status at implementation start: ready (historical snapshot; current SOW state
136 +is recorded in the top-level Status section).
137 +
138 +Problem / root-cause model:
139 +
140 +- The current topology model exposes observer-relative socket facts as if they
141 + were topology semantics. This makes actor modals and aggregation confusing:
142 + `local` means "this socket row's local endpoint", not dependency direction.
143 +- The socket scanner has enough information to derive dependency direction:
144 + inbound/local-inbound maps remote endpoint to client and local endpoint to
145 + server; outbound/local-outbound maps local endpoint to client and remote
146 + endpoint to server; listen has a server only.
147 +- The producer must encode dependency direction directly through
148 + client/server columns and actor refs, so the UI and aggregator do not infer
149 + it from local/remote names.
150 +
151 +Evidence reviewed:
152 +
153 +- `src/libnetdata/local-sockets/local-sockets.h` has distinct
154 + `SOCKET_DIRECTION_LOCAL_INBOUND` and `SOCKET_DIRECTION_LOCAL_OUTBOUND` enum
155 + values and converts loopback/local peers after inbound/outbound detection.
156 +- `src/collectors/network-viewer.plugin/network-viewer.c` maps both local enum
157 + values to the string `local`.
158 +- `src/collectors/network-viewer.plugin/network-viewer.c` currently builds
159 + topology rows with `local_ip`, `remote_ip`, and row-owner `src_actor`.
160 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` already supports directed link
161 + types, relationship tables, actor-column modal owner filters, and formatted
162 + endpoint projections.
163 +
164 +Affected contracts and surfaces:
165 +
166 +- Agent Function payload for `topology:network-connections`.
167 +- Topology developer guide and specs.
168 +- Project topology skill guidance.
169 +- Function validation fixture and semantic tests.
170 +- Potential external consumers of network-connections v1 columns.
171 +
172 +Existing patterns to reuse:
173 +
174 +- Existing compact table encoder helpers in `network-viewer.c`.
175 +- Existing modal `actor_column` owner filters and `formatted_endpoint`
176 + projection.
177 +- Existing type-level link presentation and link aggregation metadata.
178 +- Existing correlation rows with declarative protocol/address-space/IP/port
179 + keys.
180 +
181 +Risk and blast radius:
182 +
183 +- Medium/high semantic risk inside network-connections only.
184 +- Low schema risk because no generic schema mechanism is added.
185 +- Medium compatibility risk for cloud-frontend and cloud-topology-service if
186 + they hardcoded old network-connections column names despite the v1 contract.
187 +- Performance risk is bounded by aggregating rows more aggressively in
188 + aggregated mode.
189 +
190 +Sensitive data handling plan:
191 +
192 +- Use synthetic examples in SOW/spec/docs.
193 +- Do not commit raw live Function payloads, process command lines, private
194 + endpoints, public customer-identifying endpoints, tokens, cookies, machine
195 + GUIDs, or node IDs.
196 +- Store any live captures under `.local/` only if needed.
197 +
198 +Implementation plan:
199 +
200 +1. Update the network-viewer topology producer internal row shape with derived
201 + client/server endpoints and dependency actor selection.
202 +2. Change topology output columns, match columns, correlation rows, and modal
203 + recipes to use client/server semantics.
204 +3. Remove the topology `local` socket filter while still collecting local
205 + sockets through inbound/outbound classification.
206 +4. Update specs, developer guide, project skill, fixtures, and focused tests.
207 +5. Validate with schema checks, semantic tests, and a network-viewer compile
208 + check or build.
209 +
210 +Validation plan:
211 +
212 +- `git diff --check`
213 +- `go test ./tools/functions-validation/validate`
214 +- Network-viewer plugin build target if the local build tree is available, or
215 + the equivalent compile command from `compile_commands.json` with output
216 + redirected to `/tmp`.
217 +- Fixture/schema checks proving network-connections uses client/server columns
218 + and no topology `local` filter.
219 +- Same-failure search for remaining network-connections topology uses of
220 + `local_ip`/`remote_ip` where they would leak topology semantics.
221 +
222 +Artifact impact plan:
223 +
224 +- AGENTS.md: not expected; workflow rules do not change.
225 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md`.
226 +- Specs: update `.agents/sow/specs/topology-function-schema.md` and
227 + `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
228 +- End-user/operator docs: network-connections integration metadata may mention
229 + `local`; update only if the user-facing non-topology Function output changes.
230 +- End-user/operator skills: not expected.
231 +- SOW lifecycle: create SOW-0030 as current/in-progress and pause SOW-0027.
232 +
233 +Open-source reference evidence:
234 +
235 +- Not checked. This is an internal topology payload contract correction, not an
236 + external protocol interpretation.
237 +
238 +Open decisions:
239 +
240 +- Resolved by user: use client/server dependency semantics, remove local/remote
241 + from network-connections topology, remove local topology filter, and keep the
242 + change scoped to network-connections.
243 +
244 +## Implications And Decisions
245 +
246 +1. User decision: `local` / `remote` are not meaningful topology concepts and
247 + must be removed from the network-connections topology contract.
248 +2. User decision: prefer `client` / `server`; if `src` / `dst` remains, then
249 + `src = client` and `dst = server` in all cases.
250 +3. User decision: graph arrows must show dependency direction.
251 +4. User decision: process actor modals must split dependency rows into
252 + `Dependencies` and `Dependants`.
253 +5. User decision: topology `local` socket filter must be removed.
254 +
255 +## Plan
256 +
257 +1. Patch network-viewer topology row model and option parsing.
258 +2. Patch network-viewer topology table schemas, values, modals, and correlation.
259 +3. Update docs/specs/project skill/fixture/tests.
260 +4. Validate narrow commands and record results.
261 +
262 +## Execution Log
263 +
264 +### 2026-05-17
265 +
266 +- Created SOW and paused SOW-0027 before implementation.
267 +- Changed network-connections topology row semantics from observer-relative
268 + `local_*` / `remote_*` columns to dependency-oriented `client_*` /
269 + `server_*` columns.
270 +- Reoriented socket graph links so `src_actor` is the client/dependant and
271 + `dst_actor` is the server/dependency target.
272 +- Removed the topology `local` socket selector from option parsing and Function
273 + metadata; local sockets are collected under inbound/outbound classification.
274 +- Split non-node network-connections modal recipes into `Dependencies` and
275 + `Dependants`.
276 +- Updated the network-connections topology validation fixture, developer guide,
277 + specs, project topology skill, and network-viewer integration text.
278 +
279 +## Validation
280 +
281 +Acceptance criteria evidence:
282 +
283 +- `src/collectors/network-viewer.plugin/network-viewer.c` emits
284 + `direction_role: "dependency"` for `socket`, `endpoint_socket`, and
285 + `correlated_socket`.
286 +- `src/collectors/network-viewer.plugin/network-viewer.c` resolves dependency
287 + actors as client/server and writes `client_*` / `server_*` columns for
288 + relationship and evidence tables.
289 +- `src/collectors/network-viewer.plugin/network-viewer.c` no longer exposes a
290 + topology `local` socket option and maps local inbound/outbound socket enums
291 + to `inbound` / `outbound`.
292 +- `src/go/tools/functions-validation/fixtures/topology-v1/network-connections.json`
293 + validates the updated client/server table contract.
294 +
295 +Tests or equivalent validation:
296 +
297 +- `cmd=$(jq -r '.[] | select(.file|endswith("src/collectors/network-viewer.plugin/network-viewer.c")) | .command' build/compile_commands.json | sed 's# -o [^ ]*# -o /tmp/network-viewer.c.o#'); eval "$cmd"` passed.
298 +- `(cd src/go && go test -count=1 ./tools/functions-validation/validate)` passed.
299 +- `git diff --check` passed.
300 +- `python3 integrations/gen_integrations.py` passed.
301 +- `python3 integrations/gen_docs_integrations.py --collector network-viewer.plugin/network-viewer.plugin` passed.
302 +
303 +Real-use evidence:
304 +
305 +- Pending user install/run validation. The local `build/` directory is owned by
306 + `root:root`, so `cmake --build build --target network-viewer.plugin` could
307 + not write `.ninja_lock`; compile validation used the same compile command
308 + from `build/compile_commands.json` with object output redirected to `/tmp`.
309 +
310 +Reviewer findings:
311 +
312 +- No external assistant review requested for this SOW.
313 +
314 +Same-failure scan:
315 +
316 +- Searched network-connections topology docs/fixtures/skill for stale
317 + `local_ip`, `local_port`, `remote_ip`, `remote_port`, and `sockets_local`
318 + contract references. Remaining matches are generic loose-side/correlation
319 + terminology or non-topology local-IP actor labels.
320 +
321 +Sensitive data gate:
322 +
323 +- No raw live payloads, cookies, tokens, machine GUIDs, or private endpoints
324 + were written. Fixture and docs use documentation-reserved example IP ranges.
325 +
326 +Artifact maintenance gate:
327 +
328 +- AGENTS.md: not expected.
329 +- Runtime project skills: updated `.agents/skills/project-create-topology/SKILL.md`.
330 +- Specs: updated topology schema/mode specs.
331 +- End-user/operator docs: updated `metadata.yaml` and generated integration
332 + markdown to remove the `local` direction wording.
333 +- End-user/operator skills: not expected.
334 +- SOW lifecycle: SOW-0030 current/in-progress; SOW-0027 paused.
335 +
336 +Specs update:
337 +
338 +- Updated `.agents/sow/specs/topology-function-schema.md` and
339 + `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
340 +
341 +Project skills update:
342 +
343 +- Updated `.agents/skills/project-create-topology/SKILL.md`.
344 +
345 +End-user/operator docs update:
346 +
347 +- Updated `src/collectors/network-viewer.plugin/metadata.yaml` and
348 + `src/collectors/network-viewer.plugin/integrations/network_connections.md`.
349 +
350 +End-user/operator skills update:
351 +
352 +- Not expected.
353 +
354 +Lessons:
355 +
356 +- The generic topology JSON Schema did not need a change; the contract change
357 + is a producer profile change using existing open table columns,
358 + actor-column modal filters, and formatted endpoint projections.
359 +
360 +Follow-up mapping:
361 +
362 +- UI and aggregator may still need compatibility updates if they hardcoded old
363 + network-connections `local_*` / `remote_*` columns; this SOW updates the Agent
364 + producer contract and fixture.
365 +- SOW-0029 remains the pending tracker for any future detailed loose-side model.
366 +
367 +## Outcome
368 +
369 +Pending.
370 +
371 +## Lessons Extracted
372 +
373 +Pending.
374 +
375 +## Followup
376 +
377 +None yet.
378 +
379 +## Regression Log
380 +
381 +None yet.
.agents/sow/done/SOW-0001-20260501-qbridge-fdb-mac-from-index.md
+1 -1
@@ -190,7 +190,7 @@ Cross-references:
190 - **VLAN fallback (D1)** could over-attribute if a device uses `fdbID != VLAN_ID` mapping but doesn't expose `dot1qVlanCurrentTable`. LibreNMS accepts this trade-off; the fallback is a documented best-effort.
191 - **Warn-on-drop (E4)** could be noisy. Implementation MUST be rate-limited to one log line per poll cycle with a count, not one per dropped row. This is in the acceptance criteria.
192
193 -## Pre-Implementation Gate
193 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
194
195 Gate state at implementation start: decisions locked; implementation authorized after validation.
196
.agents/sow/done/SOW-0003-20260503-query-agent-events-skill.md
+15 -15
@@ -180,7 +180,7 @@ Risks:
180 Always point at the consumer endpoint resolved from
181 `${AGENT_EVENTS_HOSTNAME}` and the Cloud space.
182
183 -## Pre-Implementation Gate
183 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
184
185 Status: filled-2026-05-05
186
@@ -192,7 +192,7 @@ This is NOT a generic logs query skill. The two existing `query-netdata-{cloud,a
192
193 ### Problem / root-cause model
194
195 -Maintainers (Costa + team + AI assistants) need to triage 40k-200k status submissions per day across 1.5M agents to find specific crashes, panics, regressions. Naive queries (`--grep PATTERN` over full namespace) are slow because they full-scan. The skill must teach index-friendly patterns AND ship scripts that bake them in.
195 +Maintainers need to triage 40k-200k status submissions per day across 1.5M agents to find specific crashes, panics, regressions. Naive queries (`--grep PATTERN` over full namespace) are slow because they full-scan. The skill must teach index-friendly patterns AND ship scripts that bake them in.
196
197 A second confusion the skill resolves: the after-the-fact event model. Agents POST events ONLY on start (the previous session's exit reason). So "the last hour" misses real crashes; the meaningful unit is "events posted in the last 24h", which (because of 23h client-side dedup) is ~one record per agent per event-class per day.
198
@@ -218,7 +218,7 @@ User clarifications (2026-05-05):
218 - Default time: 24h (covers the dedup unit + balances scan cost).
219 - Wide windows for rare crashes (1-per-few-days class): up to 7 days.
220 - Default version filter: latest stable + latest 2-3 nightlies (auto-compute from observed version distribution).
221 -- ssh transport: NOT a first-class script flag. Mention in `transports.md` as Costa-only path; do not expand.
221 +- ssh transport: NOT a first-class script flag. Mention in `transports.md` as operator-only path; do not expand.
222 - Group-by dimensions for `analyze-events.sh`: signal, fatal_function, fatal_filename, version, architecture, os_family, os_type, install_type, db_mode, kubernetes, profile, aclk, health, exit_cause, virtualization, chassis_type, host_cpus.
223
224 ### Affected contracts and surfaces
@@ -226,7 +226,7 @@ User clarifications (2026-05-05):
226 The skill itself is a private developer skill at `<repo>/.agents/skills/query-agent-events/` and a one-line entry in AGENTS.md "Project Skills Index". No code changes. Indirect contracts the skill MUST document accurately:
227
228 - The producer's status JSON shape at `STATUS_FILE_VERSION = 28`.
229 -- The journal namespace name (`agent-events`, hosted on Costa's ingestion server). NOT defined in this repo; documented as deployment convention.
229 +- The journal namespace name (`agent-events`, hosted on an operator-managed ingestion server). NOT defined in this repo; documented as deployment convention.
230 - The systemd-journal Function payload shape (selections / query / histogram / facets).
231 - The `query-netdata-cloud` and `query-netdata-agents` skill helper APIs (which this skill consumes).
232
@@ -243,7 +243,7 @@ The skill itself is a private developer skill at `<repo>/.agents/skills/query-ag
243 - Skill is read-only documentation + scripts that make outbound queries. Blast radius on this repo: zero. Blast radius on the ingestion server: a query script with a bad default could full-scan the journal and degrade service for 40-200k-events/day query load. Mitigation: every default in `get-events.sh` MUST be index-friendly (structured `selections` filters first); FTS only as narrower.
244 - Privacy: every fetched event carries identifying fields (machine GUIDs, claim IDs, hardware DMI). Storage stays under `<repo>/.local/audits/query-agent-events/` (gitignored). No raw values in committed artifacts. `redact-events.sh` ships as opt-in for sharing.
245 - Untrusted-doc risk: 14 high-severity divergences found in `.local/agent-events-journals.md`. The skill writes verified ground truth from producer source; the .local doc is treated as a defunct draft and not copied into committed artifacts.
246 -- Volume: Costa noted 40k-200k events/day on stable releases. Defaults narrow time + version aggressively to keep query weight low.
246 +- Volume: the user noted 40k-200k events/day on stable releases. Defaults narrow time + version aggressively to keep query weight low.
247
248 ### Decisions recorded
249
@@ -251,21 +251,21 @@ D1. **Scoping predicate**: namespace alone (`--namespace=${AGENT_EVENTS_HOSTNAME
251
252 D2. **No `AGENT_EVENTS_JOURNAL_NAMESPACE` env key**: keep `${AGENT_EVENTS_HOSTNAME}`'s quadruple-duty (Cloud room name, ssh host, direct-HTTP host, journalctl namespace) per the existing sensitive-data-discipline spec.
253
254 -D3. **Drop `--via ssh` from script flags** (Costa: A). Mention ssh briefly in `transports.md` as Costa-only path; no scripted ssh transport.
254 +D3. **Drop `--via ssh` from script flags** (the user: A). Mention ssh briefly in `transports.md` as operator-only path; no scripted ssh transport.
255
256 D4. **Privacy default**: raw under `<repo>/.local/audits/query-agent-events/` (gitignored), never shared. Opt-in `redact-events.sh` for sharing.
257
258 -D5. **Full AE_FIELDS.md coverage** (Costa: A): every producer field with version-gating annotation, every enum verified against source, indexable-vs-FTS guidance per field.
258 +D5. **Full AE_FIELDS.md coverage** (the user: A): every producer field with version-gating annotation, every enum verified against source, indexable-vs-FTS guidance per field.
259
260 -D6. **Default time window: 24h** (Costa: C). `--since '24h ago'` is the default. Wider windows (`--since '7d'`) documented for rare-crash investigation.
260 +D6. **Default time window: 24h** (the user: C). `--since '24h ago'` is the default. Wider windows (`--since '7d'`) documented for rare-crash investigation.
261
262 -D7. **Default version filter**: latest stable + latest 2-3 nightlies (Costa). `get-events.sh` accepts `--versions auto` (default), `--versions <regex>`, and `--all-versions`. Auto-mode does a lightweight version-list query first, picks top stable + top 3 nightlies by version sort, then runs the main query with that filter.
262 +D7. **Default version filter**: latest stable + latest 2-3 nightlies (the user). `get-events.sh` accepts `--versions auto` (default), `--versions <regex>`, and `--all-versions`. Auto-mode does a lightweight version-list query first, picks top stable + top 3 nightlies by version sort, then runs the main query with that filter.
263
264 -D8. **Index-friendly query discipline** (Costa, hard requirement): structured `selections` filters first, FTS via `query` only as residual narrower. Anti-pattern in any recipe: bare FTS without structured slicing. The skill writes this rule into SKILL.md key concepts and into every recipe.
264 +D8. **Index-friendly query discipline** (hard requirement): structured `selections` filters first, FTS via `query` only as residual narrower. Anti-pattern in any recipe: bare FTS without structured slicing. The skill writes this rule into SKILL.md key concepts and into every recipe.
265
266 -D9. **Group-by dimensions in `analyze-events.sh`** (Costa: confirmed): signal, fatal_function, fatal_filename, version, architecture, os_family, os_type, install_type, db_mode, kubernetes, profile, aclk, health, exit_cause, virtualization, chassis_type, host_cpus.
266 +D9. **Group-by dimensions in `analyze-events.sh`** (the user: confirmed): signal, fatal_function, fatal_filename, version, architecture, os_family, os_type, install_type, db_mode, kubernetes, profile, aclk, health, exit_cause, virtualization, chassis_type, host_cpus.
267
268 -D10. **Filter syntax (Netdata systemd-journal plugin)** (Costa, hard requirement): the Function supports multi-value filters; between fields = AND, between values = OR. Costa described this as pseudo-code `(FIELD1 in A, B, C) AND (FIELD2 in D, E, F) AND ...` -- the **actual JSON shape** (verified at `src/libnetdata/facets/logs_query_status.h:386-466`) is the `selections` POST key:
268 +D10. **Filter syntax (Netdata systemd-journal plugin)** (hard requirement): the Function supports multi-value filters; between fields = AND, between values = OR. The user described this as pseudo-code `(FIELD1 in A, B, C) AND (FIELD2 in D, E, F) AND ...` -- the **actual JSON shape** (verified at `src/libnetdata/facets/logs_query_status.h:386-466`) is the `selections` POST key:
269
270 ```json
271 {
@@ -276,7 +276,7 @@ D10. **Filter syntax (Netdata systemd-journal plugin)** (Costa, hard requirement
276 }
277 ```
278
279 -D11. **Transport-level abilities live in `query-logs.md`** (Costa, scope clarification): the multi-value `selections` capability is a property of the systemd-journal Function transport, not specific to agent-events. Both `docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` and `docs/netdata-ai/skills/query-netdata-agents/query-logs.md` get updated to mention it (cloud doc carries the full shape; agents doc references the cloud doc). agent-events specifics (which AE_* fields, when to use which, dedup semantics) stay in this skill.
279 +D11. **Transport-level abilities live in `query-logs.md`** (user scope clarification): the multi-value `selections` capability is a property of the systemd-journal Function transport, not specific to agent-events. Both `docs/netdata-ai/skills/query-netdata-cloud/query-logs.md` and `docs/netdata-ai/skills/query-netdata-agents/query-logs.md` get updated to mention it (cloud doc carries the full shape; agents doc references the cloud doc). agent-events specifics (which AE_* fields, when to use which, dedup semantics) stay in this skill.
280
281 Implications for the skill:
282 - `transports.md` references `query-logs.md` for the JSON shape rather than re-documenting it.
@@ -290,7 +290,7 @@ Skill structure:
290
291 - `SKILL.md` -- entry point. Frontmatter triggers ("agent events", "agent-events", "crash reports", "fatals", "panics", "ingestion server", "status file", "AE_*" fields). Key concepts up front: bug-investigation tool, after-the-fact model, dedup window, structured-filters-first.
292 - `AE_FIELDS.md` -- the verified field map (~80 rows): producer source path | JSON path | journal field | enum/values | version-gating | indexable as facet? | bug-triage interpretation. Plus enum-meaning tables (what each `AE_AGENT_HEALTH`, `AE_FATAL_SIGNAL_CODE`, `AE_EXIT_CAUSE` value tells a bug-fixer).
293 -- `transports.md` -- 3 transports with priority order. For each: how to call the Function via the existing `query-netdata-{cloud,agents}` helpers; what payload shape works for agent-events. ssh path gets a 1-paragraph note (Costa-only).
293 +- `transports.md` -- 3 transports with priority order. For each: how to call the Function via the existing `query-netdata-{cloud,agents}` helpers; what payload shape works for agent-events. ssh path gets a 1-paragraph operator-only note.
294 - `update-cadence.md` -- the after-the-fact model, the 23h client-side dedup, the ≥10 min disk snapshot, the start-only POST. Implications for query design (default 24h, wider for rare).
295 - `query-discipline.md` -- the structured-filters-first rule. Worked examples of right-vs-wrong queries. The Function payload's `selections` vs `query` parameters and how each interacts with the journal index.
296 - `finding-crashes.md` -- the "find recent signal crashes on stable releases" recipe end-to-end.
@@ -322,7 +322,7 @@ Skill structure:
322
323 ### Open decisions
324
325 -None. All decisions resolved with Costa on 2026-05-05.
325 +None. All decisions resolved with the user on 2026-05-05.
326
327 ### Followup items surfaced (NOT to be left as "deferred")
328
.agents/sow/done/SOW-0004-20260503-learn-site-structure-skill.md
+1 -1
@@ -106,7 +106,7 @@ Risks:
106 rather than in this repo. Make the boundary explicit so
107 maintainers know where to edit a given page.
108
109 -## Pre-Implementation Gate
109 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
110
111 Status: filled-2026-05-05
112
.agents/sow/done/SOW-0005-20260503-mirror-netdata-repos-skill.md
+10 -10
@@ -4,7 +4,7 @@
4
5 Status: completed
6
7 -Sub-state: completed 2026-05-05. Vendored, parameterized COPY of the battle-tested `~/src/netdata/sync-all.sh` shipped at `.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` with surgical changes: env-driven mirror dir (`NETDATA_REPOS_DIR`), `--repo NAME` repeatable scoping (skips Phase 2), sanitization for missing env / git / jq / `gh` (graceful Phase 2 skip when `gh` is missing or unauthed), early `--help` that works without env. Single-file SKILL.md covers why / when / semantics / safety / scoping / setup / sanitization / limitations. Reset-to-default-branch documented as the intended safety feature (prevents stale-feature-branch "black hole" repos).
7 +Sub-state: completed 2026-05-05. Vendored, parameterized COPY of the battle-tested `<local-netdata-repos>/sync-all.sh` shipped at `.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` with surgical changes: env-driven mirror dir (`NETDATA_REPOS_DIR`), `--repo NAME` repeatable scoping (skips Phase 2), sanitization for missing env / git / jq / `gh` (graceful Phase 2 skip when `gh` is missing or unauthed), early `--help` that works without env. Single-file SKILL.md covers why / when / semantics / safety / scoping / setup / sanitization / limitations. Reset-to-default-branch documented as the intended safety feature (prevents stale-feature-branch "black hole" repos).
8
9 ## Requirements
10
@@ -105,7 +105,7 @@ Risks:
105 the script could overwrite uncommitted changes. The skill
106 must call this out clearly.
107
108 -## Pre-Implementation Gate
108 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
109
110 Status: filled-2026-05-05
111
@@ -141,7 +141,7 @@ The corollary problem: a local mirror that drifts (stale feature branches, dirty
141 - Submodule `--init --force --recursive` is intended (cross-repo review and builds depend on accurate submodule state). Skill notes this.
142 - Phase 2 calls `gh` with the user's `gh auth` credentials. If `gh` is missing or unauthed, sanitization warns and skips Phase 2 (Phase 1 still runs).
143
144 -### Decisions recorded (Costa, 2026-05-05)
144 +### Decisions recorded (the user, 2026-05-05)
145
146 D1. **ORG hardcoded** to `netdata` (skill is netdata-specific; hardcoding matches the name `mirror-netdata-repos`).
147
@@ -164,7 +164,7 @@ D8. **No `run()` transparency wrapper**. Keep the existing colored "→ Fetching
164 The existing `${NETDATA_REPOS_DIR}/sync-all.sh` is battle-tested. The vendored script is a COPY with surgical changes ONLY:
165
166 1. **Anchor on env**: replace `SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd); cd "$SCRIPT_DIR"` with env-driven `cd "${NETDATA_REPOS_DIR}"`.
167 -2. **Replace fallback paths**: every `cd /home/costa/src/netdata` becomes `cd "${NETDATA_REPOS_DIR}"` (with the env var validated up-front).
167 +2. **Replace fallback paths**: every `cd <local-netdata-repos>` becomes `cd "${NETDATA_REPOS_DIR}"` (with the env var validated up-front).
168 3. **Add CLI parsing**: `--repo NAME` repeatable; default = all. When `--repo` is specified, Phase 2 is skipped.
169 4. **Add sanitization at top**:
170 - `NETDATA_REPOS_DIR` set + dir exists -- hard error if not.
@@ -193,7 +193,7 @@ No rewrite. Preserving the diff to the original is intentional so future audits
193
194 ### Open decisions
195
196 -None. All 8 resolved with Costa.
196 +None. All 8 resolved with the user.
197
198 ### Followup items (NOT to be left as deferred)
199
@@ -254,12 +254,12 @@ No user decisions required at this stub stage.
254 ### Path discipline
255
256 - `grep -rnE '~/|/home/|/opt/baddisk' .agents/skills/mirror-netdata-repos/`: zero hits.
257 -- `grep -nE '/home|costa|/opt/' .agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh`: zero hits.
257 +- `grep -nE '/home|user-name|/opt/' .agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh`: zero hits.
258 - All references to the mirror dir in skill content go through `${NETDATA_REPOS_DIR}`.
259
260 ### Surgical-edit audit
261
262 -vs. the source `~/src/netdata/sync-all.sh`, the diff is:
262 +vs. the source `<local-netdata-repos>/sync-all.sh`, the diff is:
263
264 1. Removed `SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd); cd "$SCRIPT_DIR"`.
265 2. Added `usage()` function.
@@ -267,7 +267,7 @@ vs. the source `~/src/netdata/sync-all.sh`, the diff is:
267 4. Added sanitization block: `NETDATA_REPOS_DIR` set + dir exists, `git`/`jq` required, `gh` optional with `GH_AVAILABLE` flag.
268 5. Added `cd "$MIRROR_DIR"` after sanitization.
269 6. Added `declare -a SCOPE_REPOS=()` global.
270 -7. Replaced every `cd "$SCRIPT_DIR" 2>/dev/null || cd /home/costa/src/netdata` (4 occurrences) with `cd "$MIRROR_DIR" 2>/dev/null || true`.
270 +7. Replaced every `cd "$SCRIPT_DIR" 2>/dev/null || cd <local-netdata-repos>` (4 occurrences) with `cd "$MIRROR_DIR" 2>/dev/null || true`.
271 8. In `main()`: added CLI parsing for `--repo` (repeatable) and `-h|--help`; added a "scoped vs full" branch building `sorted_repos`; added Phase 2 skip-when-scoped and skip-when-`gh`-unavailable; changed `main` to `main "$@"`.
272
273 All other code paths preserved verbatim (skip-on-staged-or-modified, switch-to-default, submodule force-recursive, activity cache, colored output, summary, dedupe, etc.).
@@ -282,7 +282,7 @@ All other code paths preserved verbatim (skip-on-staged-or-modified, switch-to-d
282
283 ## Outcome
284
285 -The `mirror-netdata-repos` private skill ships a self-contained, env-driven, sanitized sync tool. AI assistants and developers working on this project can now bring up a local Netdata-org repos mirror at `${NETDATA_REPOS_DIR}` without depending on the user's personal `~/src/netdata/sync-all.sh`. Cross-repo grep / code review runs locally; GitHub API round-trips and rate limits are eliminated for the day-to-day workflow.
285 +The `mirror-netdata-repos` private skill ships a self-contained, env-driven, sanitized sync tool. AI assistants and developers working on this project can now bring up a local Netdata-org repos mirror at `${NETDATA_REPOS_DIR}` without depending on the user's personal `<local-netdata-repos>/sync-all.sh`. Cross-repo grep / code review runs locally; GitHub API round-trips and rate limits are eliminated for the day-to-day workflow.
286
287 The reset-to-default-branch behavior is documented as the intended safety mechanism: stale-feature-branch repos in a mirror are "black holes" that mislead cross-repo reasoning, and the only viable fix is to always reset clean repos to default. Skip conditions (staged or modified files) preserve user work; the branch ref preserves any unpushed commits.
288
@@ -302,7 +302,7 @@ The reset-to-default-branch behavior is documented as the intended safety mechan
302
303 These items were exposed during implementation but are NOT part of this SOW. Tracked separately:
304
305 -- F-0005-A: the user's local `~/src/netdata/sync-all.sh` will diverge over time from this vendored copy. Decide later whether the user replaces his local copy with a symlink to `<repo>/.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` (then both stay in sync).
305 +- F-0005-A: the user's local `<local-netdata-repos>/sync-all.sh` will diverge over time from this vendored copy. Decide later whether the user replaces his local copy with a symlink to `<repo>/.agents/skills/mirror-netdata-repos/scripts/sync-netdata-repos.sh` (then both stay in sync).
306 - F-0005-B: shellcheck inherits ~10 info-level warnings from the original script (SC2155 declare-and-assign, SC2086 quoting). Could be cleaned up but the script is battle-tested; touching unrelated code risks regression. Defer until a refactor pass that's intentionally about quality, not feature work.
307
308 ## Regression Log
.agents/sow/done/SOW-0007-20260504-integrations-lifecycle-skill.md
+1 -1
@@ -152,7 +152,7 @@ Risks:
152 not uniform across the three surfaces. If so, document the
153 divergences explicitly rather than papering over them.
154
155 -## Pre-Implementation Gate
155 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
156
157 Status: filled-2026-05-05
158
.agents/sow/done/SOW-0008-20260505-mikrotik-snmp-per-second-gaps.md
+2 -2
@@ -109,9 +109,9 @@ Risks:
109 - Leaving heavy profile sections in the 1-second path causes gaps and misleading rate spikes after skipped samples.
110 - Changing profile filtering can affect all SNMP devices that rely on shared topology or LLDP profile fragments.
111
112 -## Pre-Implementation Gate
112 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
113
114 -Status: ready
114 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
115
116 Problem / root-cause model:
117
.agents/sow/done/SOW-0009-20260502-project-writing-collectors-skill.md
+1 -1
@@ -159,7 +159,7 @@ Topics where the repo lacks a canonical doc and the skill must either inline gui
159 | coverity-audit | 474 |
160 | pr-reviews | 508 |
161
162 -## Pre-Implementation Gate
162 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
163
164 Status: needs-user-decision
165
.agents/sow/done/SOW-0010-20260503-netdata-query-skills-infrastructure.md
+2 -2
@@ -419,7 +419,7 @@ Risks:
419 `query-netdata-cloud/query-metrics.md`) to avoid breaking
420 inbound links.
421
422 -## Pre-Implementation Gate
422 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
423
424 Status: needs-user-decision
425
@@ -743,7 +743,7 @@ Pending decisions 1-3. After they are answered:
743 + how-tos + verify pointers. The "if you analyze, you author
744 a how-to" rule baked into SKILL.md, AGENTS.md, and both
745 how-tos/INDEX.md files. Seed verify/questions.md written for
746 - both skills (Costa's user-supplied 23 + 19 questions covering
746 + both skills (the user-supplied 23 + 19 questions covering
747 identity / hardware / OS / streaming / vnodes / collectors /
748 alerts / logs / topology / flows / dyncfg / members / rooms /
749 feed / token-safety self-test). Stage 2l (close) pending.
.agents/sow/done/SOW-0011-20260505-codacy-audit-skill.md
+5 -5
@@ -34,7 +34,7 @@ Facts:
34
35 - Three sister legacy skills already exist with the triage shape we want to mirror: `.agents/skills/coverity-audit/`, `.agents/skills/sonarqube-audit/`, `.agents/skills/graphql-audit/`. A `codacy-audit/` is the natural fourth.
36 - Codacy Cloud ships an official local CLI: `codacy-analysis-cli` (https://github.com/codacy/codacy-analysis-cli). Install paths: docker, install.sh, brew. **Docker is available on this workstation** (`/usr/bin/docker`, version 29.4.1).
37 -- The Codacy v3 REST API at `api.codacy.com` is reachable: PR-level issue lists work even anonymously; broader cross-PR / org queries require an Account API token. Verified live during the conversation that this skill is being created for: `gh user 30945 = Costa Tsaousis`, 31,425 issues currently open on `master`.
37 +- The Codacy v3 REST API at `api.codacy.com` is reachable: PR-level issue lists work even anonymously; broader cross-PR / org queries require an Account API token. Verified live during the conversation that this skill is being created for the configured account; 31,425 issues were open on `master`.
38 - PR #22423 is a useful end-to-end fixture for validation: it had 864 markdownlint findings on the first CI run; fixed by `.codacy.yml` exclusion in commit `3a54c9afbc`. The local CLI must reproduce the original 864 findings for an objective accuracy check.
39 - The repo's `.codacy.yml` is the source of truth for path exclusions; the local CLI must respect it (or we have to teach it to).
40 - Path discipline spec at `<repo>/.agents/sow/specs/sensitive-data-discipline.md` already defines `CODACY_TOKEN`-class constraints by precedent (Coverity / Sonar tokens). Keys must live in `.env`, never in committed artifacts.
@@ -91,9 +91,9 @@ Risks:
91 - **Token accidentally committed via a finding dump**: a JSON dump from the API could echo back the token in error messages. Mitigation: token-safe wrappers in `_lib.sh` plus the no-leak self-test.
92 - **`.local/audits/codacy/` filling up with stale dumps**: ephemeral, gitignored, but disk pressure risk on long sessions. Mitigation: filename includes timestamp; user can `rm` whenever.
93
94 -## Pre-Implementation Gate
94 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
95
96 -Status: ready
96 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
97
98 Problem / root-cause model:
99
@@ -132,7 +132,7 @@ Risk and blast radius:
132 Sensitive data handling plan:
133
134 - `CODACY_TOKEN` is a credential -- handled exactly like `NETDATA_CLOUD_TOKEN`, `COVERITY_COOKIE`, `SONAR_TOKEN`: lives in `.env` (gitignored), referenced via `${CODACY_TOKEN}` in scripts only, never in commit messages, never in fixtures.
135 -- Account ID 30945 (Costa's Codacy account) was returned by `/v3/user` during exploration but will not be written to any committed artifact. The SOW redacts to "the configured account".
135 +- The configured Codacy account was returned by `/v3/user` during exploration but will not be written to any committed artifact. The SOW redacts to "the configured account".
136 - Audit JSON dumps land under `<repo>/.local/audits/codacy/<timestamp>.json` (gitignored).
137 - No customer / community member / private-host data is touched -- this is a public-repo CI workflow.
138
@@ -268,7 +268,7 @@ Same-failure scan:
268
269 Sensitive data gate:
270
271 -- No raw tokens, account UUIDs, customer-identifying IPs, or private endpoints were written to any committed artifact. The Codacy account ID 30945 (Costa) was returned by `/v3/user` during exploration and is intentionally not committed; this SOW redacts to "the configured account".
271 +- No raw tokens, account UUIDs, customer-identifying IPs, or private endpoints were written to any committed artifact. The configured Codacy account was returned by `/v3/user` during exploration and is intentionally not committed; this SOW redacts to "the configured account".
272 - `CODACY_TOKEN` is referenced via `${CODACY_TOKEN}` only, never literally.
273 - Dumps land under `<repo>/.local/audits/codacy/` (gitignored).
274
.agents/sow/done/SOW-0012-20260505-streaming-topology-classification-bugs.md
+3 -3
@@ -267,9 +267,9 @@ Risks:
267 - (Bug D fix) changing only the column type (Option 2) is a pure metadata change; it does not touch the data emission or the inter-agent wire format. Risk: very low.
268 - Same-failure search: `rrdhost_stream_path_total_reboot_time_ms` (stream-path.c:145-158) shares Bug A's blind assumption (that localhost is in localhost's own stored path). On a top parent it returns 0 silently. Add to the scan list during validation. The `topology:snmp` Function (separate) may have similar column-type issues; check `since`/timestamp columns there as part of validation, even if no fix is needed.
269
270 -## Pre-Implementation Gate
270 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
271
272 -Status: ready (user decisions recorded 2026-05-06: Decision 1 = Option 1, Decision 2 = Option 1, Decision 3 = Option 1 with no new actor type, Decision 4 = Option 2, hardening deferred to separate SOW)
272 +Status at implementation start: ready (user decisions recorded 2026-05-06: Decision 1 = Option 1, Decision 2 = Option 1, Decision 3 = Option 1 with no new actor type, Decision 4 = Option 2, hardening deferred to separate SOW; final closure evidence is in the Validation and Outcome sections).
273
274 Problem / root-cause model:
275
@@ -508,7 +508,7 @@ PR commit order: split → B → A → D → C → doc → tests → validation.
508 - 2026-05-06 — confirmed with project owner that the streaming-path subsystem must not be touched by this SOW. Decision 1 collapsed from three options (protocol fix / function-only / hybrid) to two options that are both contained in `function-streaming.c`: (1) localhost-only live-state classification — recommended; (2) all-host live-state classification — deferred to a follow-up if the residual surfaces. Removed all references to a protocol fix follow-up. The cycle terminator at stream-path.c:423 stays load-bearing and untouched.
509 - 2026-05-06 — added the project owner's intended classification logic (six rules) and intended actor set (per-agent count formula) to the Analysis section. Added the cross-agent mergeability requirements section (stable actor IDs, stable link tuples, timestamps for tie-breaking, no overreach). Synthesized upstream `parent` actors must preserve the canonical actor-id format and derive link timestamps from the `STREAM_PATH` struct so the merge layer can reconcile views from multiple parents.
510 - 2026-05-06 — added scope: split `function-streaming.{c,h}` into `function-netdata-streaming.{c,h}` and `function-topology-streaming.{c,h}`, with `function_streaming` renamed to `function_netdata_streaming`. The bug fixes land in the new topology file. Added scope: write `src/streaming/STREAM_PATH.md` as a co-located maintenance reference (not picked up by learn ingestion). Both additions confirmed with project owner.
511 -- 2026-05-06 — iteration-3 independent read-only review batch completed; outputs preserved under `<repo>/.local/audits/streaming-topology/`. Project owner's corrections applied: (1) **dropped `remote_parent` actor type** — synthesized upstream actors reuse the existing `actor_type:"parent"`, no new type, no presentation block changes, no FE coupling; (2) Decision 3 sub-spec adds multi-hop link emission for every consecutive non-localhost path slot, not only `slot[0] → slot[1]`; (3) Decision 3 sub-spec mandates synthesized link timestamps derive from STREAM_PATH `since`/`first_time_t`, not `now`; (4) Decision 1 wording corrected — use `rrdhost_status()` (`s.ingest.type == CHILD` and `s.ingest.status ∈ {ONLINE, REPLICATING}`), no `host->receiver` access; (5) Decision 1 sub-spec — skip slot-0+ append for localhost to avoid double-write to `parent_descendants[localhost]`; (6) **deferred all `stream-path.c` defensive hardening** (uint16_t array clamp + scalar range checks) to a separate hardening SOW — this SOW now touches zero shared subsystems beyond the topology function and a new doc; (7) replaced workstation paths (`~/src/...`) with repo-relative paths (`<repo>/...`) and dashboard-repo placeholder (`<dashboard-repo>/...`); (8) corrected STREAM_PATH.md framing — the protection from learn ingestion is "not referenced in `docs/.map/map.yaml`", not "not under `docs/`"; (9) pinned cloud-frontend evidence to commit `8d0258eb60aa32e3ee5fdd2144ef10b44f7995bc` next to the `dataTable.js:52-58` citation. STREAMING_FUNCTION_UPDATE_EVERY shared macro: project owner's call — duplicate per file, no shared header. Aggregator self-marker proposal dropped — existing `data.agent_id` is sufficient for cross-agent merge.
511 +- 2026-05-06 — iteration-3 independent read-only review batch completed; outputs preserved under `<repo>/.local/audits/streaming-topology/`. Project owner's corrections applied: (1) **dropped `remote_parent` actor type** — synthesized upstream actors reuse the existing `actor_type:"parent"`, no new type, no presentation block changes, no FE coupling; (2) Decision 3 sub-spec adds multi-hop link emission for every consecutive non-localhost path slot, not only `slot[0] → slot[1]`; (3) Decision 3 sub-spec mandates synthesized link timestamps derive from STREAM_PATH `since`/`first_time_t`, not `now`; (4) Decision 1 wording corrected — use `rrdhost_status()` (`s.ingest.type == CHILD` and `s.ingest.status ∈ {ONLINE, REPLICATING}`), no `host->receiver` access; (5) Decision 1 sub-spec — skip slot-0+ append for localhost to avoid double-write to `parent_descendants[localhost]`; (6) **deferred all `stream-path.c` defensive hardening** (uint16_t array clamp + scalar range checks) to a separate hardening SOW — this SOW now touches zero shared subsystems beyond the topology function and a new doc; (7) replaced workstation paths with repo-relative paths (`<repo>/...`) and dashboard-repo placeholder (`<dashboard-repo>/...`); (8) corrected STREAM_PATH.md framing — the protection from learn ingestion is "not referenced in `docs/.map/map.yaml`", not "not under `docs/`"; (9) pinned cloud-frontend evidence to commit `8d0258eb60aa32e3ee5fdd2144ef10b44f7995bc` next to the `dataTable.js:52-58` citation. STREAMING_FUNCTION_UPDATE_EVERY shared macro: project owner's call — duplicate per file, no shared header. Aggregator self-marker proposal dropped — existing `data.agent_id` is sufficient for cross-agent merge.
512 - 2026-05-06 — project-owner decision: remove backend topology graph filters because the UI does not expose them and the backend must return the full graph. Recorded as Decision 5. Implementation updated `function-topology-streaming.c` so `accepted_params` only advertises `info`; removed `node_type`, `ingest_status`, and `stream_status` parsing and all actor/link/synthetic filter checks.
513 - 2026-05-06 — revisited topology flow after removing filters. Added a separate `local_actor_ids` set for all `RRDHOST`-backed actors, keeping `emitted_actors` as only the actors actually written to JSON. Synthetic upstream parents are now skipped when a local `RRDHOST` actor exists, not because a filtered actor happened to be present or absent. `parent_descendants[localhost]` is now written by one live-state pass: vnodes as `virtual`, active/repl children as `streaming`, and disconnected/history-only local hosts as `stale`; the path walk no longer has a competing localhost stale branch. Phase 4 now emits vnode links as `link_type:"virtual"` to localhost regardless of stored path length, registers each emitted link in `emitted_links`, and the synthetic link pass relies on that shared dedup instead of the previous `phase4_emitted` heuristic.
514 - 2026-05-06 — validation so far: focused compiler syntax check of `src/web/api/functions/function-topology-streaming.c` passed using the existing `functions.c` compile flags from `compile_commands.json`. Full `cmake --build build-clion --target netdata -j4` did not reach code compilation because CMake reconfigured and failed while fetching the pre-existing Sentry/crashpad dependency (`mini_chromium` HTTP 400 / expected acknowledgments). This is an environment/dependency fetch failure, not a compiler error from the topology file.
.agents/sow/done/SOW-0012-20260506-snmp-profile-projection.md
+2 -2
@@ -109,9 +109,9 @@ Risks:
109 - Validation churn from adding `topology:` and consumer fields to the profile schema.
110 - Documentation drift if profile-format docs and project SNMP authoring skill are not updated with the new contract.
111
112 -## Pre-Implementation Gate
112 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
113
114 -Status: ready
114 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
115
116 Problem / root-cause model:
117
.agents/sow/done/SOW-0013-20260507-snmp-licensing-projection.md
+2 -2
@@ -128,9 +128,9 @@ Risks:
128 - Silent data loss if typed schema does not model scalar-only licensing rows and table licensing rows cleanly.
129 - Sensitive data/provenance risk from committing local MIB files, workstation paths, or unsanitized SNMP fixtures.
130
131 -## Pre-Implementation Gate
131 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
132
133 -Status: ready
133 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
134
135 Problem / root-cause model:
136
.agents/sow/done/SOW-0014-20260506-netflow-sflow-ipfix-documentation-guide.md
+7 -4
@@ -202,9 +202,9 @@ CPU semantics note (will appear in docs):
202 - At saturation on a multi-core host it is well above 100% (e.g. ~600-800% on this workstation when ingest threads + tier-batch threads are all busy)
203 - Documentation must call this out explicitly so capacity planners read the metric correctly
204
205 -## Pre-Implementation Gate
205 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
206
207 -Status: ready
207 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
208
209 Problem / root-cause model:
210
@@ -607,6 +607,9 @@ Open follow-ups, ordered by priority:
607
608 ## Regression Log
609
610 +Note: dated regression entries intentionally use `## Regression - YYYY-MM-DD`
611 +headings to match the repository SOW lifecycle contract.
612 +
613 ## Regression - 2026-05-07
614
615 ### What broke
@@ -3038,9 +3041,9 @@ but the actual Learn page is
3041 - The broken link returns a routing/publishing problem only when clicked or
3042 link-checked against Learn, not during metadata schema validation.
3043
3041 -### Pre-Implementation Gate
3044 +### Pre-Implementation Gate (Historical Snapshot at Implementation Start)
3045
3043 -Status: ready
3046 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
3047
3048 Problem / root-cause model:
3049
.agents/sow/done/SOW-0015-20260508-netflow-enrichment-verification.md
+6 -3
@@ -131,9 +131,9 @@ Risks:
131 - Installing and restarting the local plugin affects the user's running Netdata instance; use the provided script deliberately and record the command/result.
132 - Documentation is generated from metadata for integration cards; generated files must not be hand-edited.
133
134 -## Pre-Implementation Gate
134 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
135
136 -Status: ready
136 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
137
138 Problem / root-cause model:
139
@@ -650,7 +650,10 @@ Artifact maintenance for review pass:
650
651 ## Regression Log
652
653 -### Regression - 2026-05-08 - Missing Provider Integration Modules
653 +Note: dated regression entries intentionally use `## Regression - YYYY-MM-DD`
654 +headings to match the repository SOW lifecycle contract.
655 +
656 +## Regression - 2026-05-08 - Missing Provider Integration Modules
657
658 What broke:
659
.agents/sow/done/SOW-0015-20260508-snmp-bgp-typed-projection.md
+4 -4
@@ -2,9 +2,9 @@
2
3 ## Status
4
5 -Status: in-progress
5 +Status: completed
6
7 -Sub-state: reopened for final review-feedback fixes before merge; user decisions 1-23 resolved; all BGP-bearing stock profiles migrated to typed `bgp:` rows; legacy `bgp_public*` runtime deleted; metadata and generated SNMP integration docs regenerated; local `mibs/` reference files removed.
7 +Sub-state: completed after final review-feedback fixes; user decisions 1-23 resolved; all BGP-bearing stock profiles migrated to typed `bgp:` rows; legacy `bgp_public*` runtime deleted; metadata and generated SNMP integration docs regenerated; local `mibs/` reference files removed.
8
9 ## Requirements
10
@@ -124,9 +124,9 @@ Risks:
124 - Integration artifacts can drift unless metadata, generated docs, health alerts, config, and docs are closed together.
125 - Raw local MIBs and fixture provenance must not leak into committed artifacts.
126
127 -## Pre-Implementation Gate
127 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
128
129 -Status: ready
129 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Validation and Outcome sections).
130
131 Problem / root-cause model:
132
.agents/sow/done/SOW-0017-20260516-query-duration-below-collection-frequency.md
+1 -1
@@ -80,7 +80,7 @@ Risks:
80 - A strict no-widening interpretation preserves mathematical precision but makes sub-frequency historical inspection unreliable for real troubleshooting workflows.
81 - Tier-selection behavior may differ across tier 0, tier 1, and tier 2, so tests must not cover only tier 0.
82
83 -## Pre-Implementation Gate
83 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
84
85 Status: diagnosis complete for the verified local reproduction
86
.agents/sow/done/SOW-0020-20260505-network-connections-topology-cloud-errors.md new
+1358
@@ -0,0 +1,1358 @@
1 +# SOW-0020 - Network-connections topology Cloud errors
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +Sub-state: completed after the compact topology schema migration, producer migrations, validation fixtures, and the targeted network-connections graph-invariant repair for floating endpoint actors.
8 +
9 +## Requirements
10 +
11 +### Purpose
12 +
13 +Make `topology:network-connections` reliable enough for production Cloud topology use in the target Cloud space. The Function must not crash or disconnect mid-response, must report missing/unavailable cases clearly, and must not produce disconnected floating endpoint actors unless the graph semantics explicitly allow them.
14 +
15 +The expanded purpose is to define and implement a lossless compact `detailed` topology payload contract and an `aggregated` topology view path so large topologies can be used by the UI and by Cloud cross-node correlation without exceeding practical payload limits.
16 +
17 +### User Request
18 +
19 +The user reported three Cloud network-connections topology issues:
20 +
21 +- One node returns HTTP 503 while data appears to start arriving; browser console shows `AgentError` with message `The plugin that was servicing this request, exited before responding.`
22 +- Another node returns HTTP 404 for the same `topology:network-connections` Function request.
23 +- A previously observed UI state showed orphan endpoint actors floating in the topology, but the user cannot reproduce it now.
24 +
25 +The user then expanded the required solution:
26 +
27 +- Update the Function schema for the new topology payload contract.
28 +- Add/declare an `aggregated` vs `detailed` request mode for `topology:network-connections`; default behavior is `aggregated`.
29 +- Update all topology Functions to emit the new schema: `topology:network-connections`, `topology:streaming`, and `topology:snmp`.
30 +- Update the Cloud frontend to support deployed compatibility payloads and the new schema. Compatibility aggregation must stay isolated so it is easy to delete after supported Agents emit the new schema.
31 +- Add a separate Cloud topology aggregation microservice. It consumes detailed topology payloads, aggregates topologies, and returns the aggregation requested by the user.
32 +
33 +### Assistant Understanding
34 +
35 +Facts:
36 +
37 +- The failing Function name is `topology:network-connections`.
38 +- The first observed Cloud request returned HTTP 503 and the browser console reported an agent-side plugin exit before response completion.
39 +- The second observed Cloud request returned HTTP 404 for the same Function name and selections.
40 +- `topology:network-connections` is registered by the C network-viewer plugin at `src/collectors/network-viewer.plugin/network-viewer.c`.
41 +- Raw browser cookies and session tokens were present in the chat. They must not be copied into durable artifacts.
42 +
43 +Inferences:
44 +
45 +- The 503 is likely not a pure frontend cancellation because the console includes an explicit agent/plugin error. The early `Request was cancelled` stack may be a frontend request lifecycle side effect, but it is not the strongest failure signal.
46 +- The 404 may mean the target node does not expose the Function, the node is stale/unreachable through Cloud, the function list differs by agent/plugin version, or Cloud maps an agent-side function-missing condition to 404.
47 +- The orphan endpoint symptom may be a graph-construction invariant violation: actors emitted without at least one emitted link, or links filtered/dropped after actor creation.
48 +
49 +Unknowns:
50 +
51 +- Whether the network-viewer plugin process exits due to a crash, fatal(), timeout/cancellation path, memory pressure, malformed topology parameters, or Cloud transport disconnect.
52 +- Whether the 404 node lacks the Function, is stale, runs an older agent, or is behind a Cloud routing/state issue.
53 +- Whether the orphan endpoints were produced by agent data, Cloud/frontend filtering, or a race between partial data and final error handling.
54 +
55 +### Acceptance Criteria
56 +
57 +- The 503 path is reproduced or falsified through token-safe Cloud or direct-agent wrappers, with status code, response class, and sanitized failure evidence recorded.
58 +- The 404 path is explained with node state/function availability/version evidence.
59 +- The serving code path and any crash-prone branches for the reported selections are traced with file:line evidence.
60 +- If a code fix is needed, it has focused tests or an equivalent validation path, and the validation includes a same-failure search for orphan actors.
61 +- The new topology schema preserves the canonical information needed by Cloud aggregation and UI rendering. Compatibility projection for deployed payload parity is test/transition code only and is not part of the production payload.
62 +- The new aggregated topology schema/view is explicitly derived from detailed topology data and is not treated as a correlation source of truth.
63 +- `topology:network-connections`, `topology:streaming`, and `topology:snmp` all emit the new topology schema.
64 +- `topology:network-connections` exposes an `aggregated`/`detailed` request mode in the Function schema, with `aggregated` as the default.
65 +- The Cloud frontend supports deployed compatibility payloads and the new schema, with compatibility aggregation isolated in removable code.
66 +- The Cloud topology aggregation microservice has a defined API contract and validation showing it can consume detailed topology payloads and return requested aggregated views.
67 +- The detailed payload contract supports multiple actor scopes for Cloud aggregation, including node-level, process-name-level, PID-level, container/application-level, and Kubernetes label/workload-level grouping when the required enrichment evidence is available.
68 +- Topology actor drilldown modals must continue to support per-actor tables listing exact dependencies and supporting details. These tables must avoid duplicating the same socket/link evidence under multiple actors where compact references or derived views can provide the same drilldown.
69 +- Topology actor tables may also carry actor-owned custom data that is not relationship evidence and is not generally aggregatable, such as streaming topology `streaming_path` and retention/status tables. The schema must distinguish relationship/evidence tables from actor-detail/custom tables.
70 +- Link direction semantics must be explicit. Some topology link types are direction-significant and must aggregate/render as directed relationships, while others are undirected adjacencies where any `direction` field describes observation completeness or evidence state rather than graph direction.
71 +- Topology links must be able to expose refreshable traffic overlays without recomputing the topology. The schema must support compact link-level telemetry query templates, per-link template parameters, and merge rules for aggregated links.
72 +- Before freezing the schema or updating producers/UI, build an emulation and benchmarking harness that can model required topology use cases at multiple scales, compare alternative encodings, run a hypothetical aggregation service, and produce repeatable payload-size and correctness evidence.
73 +
74 +## Analysis
75 +
76 +Sources checked:
77 +
78 +- `docs/netdata-ai/skills/query-netdata-cloud/SKILL.md`
79 +- `docs/netdata-ai/skills/query-netdata-agents/SKILL.md`
80 +- `.agents/skills/query-agent-events/SKILL.md`
81 +- `.agents/skills/project-writing-collectors/SKILL.md`
82 +- `.agents/sow/pending/SOW-0002-20260501-unified-multi-layered-topology-schema.md`
83 +- `.agents/sow/current/SOW-0012-20260505-streaming-topology-classification-bugs.md`
84 +- `src/collectors/network-viewer.plugin/network-viewer.c`
85 +- `src/plugins.d/FUNCTION_UI_REFERENCE.md`
86 +
87 +Current state:
88 +
89 +- Existing SOW-0002 covers future unified topology merge semantics; it is broader and blocked on design decisions. This incident is narrower and should remain separate unless it turns into schema-level work.
90 +- This SOW has now turned into schema-level work. Before implementation, its relationship with SOW-0002 must be resolved by either merging scope, superseding the relevant part of SOW-0002, or narrowing this SOW back to `topology:network-connections` only. The current user direction points toward this SOW owning the concrete detailed/aggregated schema migration.
91 +- Existing SOW-0012 covers `topology:streaming`; it is also in `.agents/sow/current/` and its current branch state already has the streaming Function split into `src/web/api/functions/function-topology-streaming.c` and `src/web/api/functions/function-netdata-streaming.c`. On 2026-05-06 the user stated SOW-0012 is done. For SOW-0020 sequencing, this unblocks shared schema work against the streaming topology producer, but the SOW-0012 lifecycle file still needs separate close/move handling if that has not already happened elsewhere.
92 +- `project-writing-collectors` classifies topology Functions as live snapshots that must not block collection loops and should be validated with the Function protocol tooling.
93 +- Token-safe Cloud probes using the repository `.env` token path returned HTTP 200 for all checked aliases at investigation time. This means the reported 503 and 404 were not stable, always-reproducible failures from the same API surface.
94 +- Successful `topology:network-connections` responses were very large:
95 + - `node-503`: 134,180,370 bytes, 48 actors, 77,797 links, no orphan actors in the completed response.
96 + - `node-404-original`: 127,847,820 bytes, 47 actors, 73,989 links, no orphan actors in the completed response.
97 + - `node-404-latest`: 110,366,885 bytes, 47 actors, 63,895 links, no orphan actors in the completed response.
98 +- The graph has low visual actor-pair cardinality despite very high link cardinality. For the largest response, 77,797 links collapse to 65 actor pairs, with the hottest actor pair carrying 38,275 separate socket links.
99 +- `pluginsd` has a hard deferred Function response cap: `PLUGINSD_MAX_DEFERRED_SIZE` is `100 * 1024 * 1024` at `src/plugins.d/pluginsd_parser.h:18`, and the parser stops the plugin when a deferred response exceeds it at `src/plugins.d/pluginsd_parser.h:195-207`.
100 +- The exact browser console message `The plugin that was servicing this request, exited before responding.` is emitted when an in-flight Function is deleted with HTTP 503 and no response body at `src/plugins.d/pluginsd_functions.c:96-97`.
101 +- The `network-viewer` topology Function currently receives cancellation and timeout state but marks both unused at `src/collectors/network-viewer.plugin/network-viewer.c:2703-2705`; it builds the full response in memory and sends it only at the end at `src/collectors/network-viewer.plugin/network-viewer.c:2713-2727`.
102 +- The `processes:by_name` option collapses process actors by command name at `src/collectors/network-viewer.plugin/network-viewer.c:1242-1253`, but the link key still includes pid, uid, namespace, local IP, remote IP, protocol, direction, state, local port, and remote endpoint port at `src/collectors/network-viewer.plugin/network-viewer.c:1359-1370`. This is the immediate source of many parallel socket links between the same collapsed actor pairs.
103 +- The latest browser 404 maps to Cloud error key `ErrNodeInstanceNotFound` / message `could not find node instance`. At probe time, the same node alias was present in the room inventory, reachable, running the same nightly version as `node-503`, and advertised `topology:network-connections` through the room Functions endpoint. This points to a Cloud node-instance routing/state race or stale node-instance lookup, not a stable missing Function.
104 +- A sibling `cloud-charts-service` checkout shows the Cloud Function route (`/api/v2/nodes/{nodeID}/function`) is handled by `nodePathProxy` (`cloud-charts-service/http/http.go:145-147`), which calls `DirectNodeRequest` (`cloud-charts-service/http/http.go:264-270`). `DirectNodeRequest` selects a node instance before proxying the request (`cloud-charts-service/internal/service/agent_data.go:565-585`). `ErrNodeInstanceNotFound` is the 404 error key and message in `cloud-charts-service/internal/model/errors.go:15`, and it is returned when node-instance routing filters leave no candidate (`cloud-charts-service/internal/routing/node_instance_filter.go:35-37`).
105 +- A sibling `cloud-frontend` checkout has prior topology TODOs documenting the same class of payload issue: a previous 89 MB response with 42,609 links collapsed to 116 actor-pair tuples. Current source has render-time link aggregation, but the useFetch topology normalizer still synchronously normalizes every source link and then computes aggregated links for graph rendering: `src/domains/functions/useFetch/normalizers/topology/index.js:5-17`.
106 +- The frontend normalizer copies full actor tables into graph node details and stores raw actors/links in table rows: `src/domains/functions/topology/payload.js:276-311,436-460`. It also copies full link `labels`, `metrics`, `src`, and `dst` into every graph link detail object: `src/domains/functions/topology/payload.js:462-487`.
107 +- The topology actor modal uses actor-type table definitions from presentation metadata and renders either link-derived tables or data tables. Data tables read rows from `node.details.tables[tableKey]`, while link tables are derived from the actor's incident graph links: `cloud-frontend/src/domains/functions/components/topology/actorModal/index.js:122-130,284-309` and `cloud-frontend/src/domains/functions/components/topology/actorModal/dataTable.js:21-24`.
108 +- Current `topology:network-connections` presentation defines modal table metadata for actors, including `source: "data"` sockets tables and `source: "links"` connection tables: `src/collectors/network-viewer.plugin/network-viewer.c:1869-1938`.
109 +- Current `topology:network-connections` embeds per-process `tables.sockets` rows directly under each actor: `src/collectors/network-viewer.plugin/network-viewer.c:2209-2226`. This satisfies the modal drilldown UX but contributes to duplication and payload size.
110 +- Current `topology:streaming` defines `streaming_path`, `retention`, `inbound`, and `outbound` actor tables with `source: "data"` even though they have different semantics: some describe streaming relationships, while `streaming_path` is actor-specific path metadata. Evidence: `src/web/api/functions/function-topology-streaming.c:303-318,321-341,360-410`.
111 +- Current `topology:streaming` emits each actor's `streaming_path` table through `rrdhost_stream_path_to_json()`, whose rows contain path metadata such as hostname, host ID, node ID, claim ID, hops, timestamps, capabilities, and flags. Evidence: `src/web/api/functions/function-topology-streaming.c:1583-1584` and `src/streaming/stream-path.c:79-95`.
112 +- The current presentation table schema only distinguishes `source: "data"` from `source: "links"`: `src/plugins.d/FUNCTION_UI_SCHEMA.json:349-357` and `src/go/pkg/topology/types.go:72-78`. This is insufficient to communicate aggregation semantics.
113 +- Current shared `topology_link` has a free-form optional `direction` string, and `topology_presentation_link_type` has visual fields such as label/color/width/dash but no direction semantics or aggregation policy: `src/plugins.d/FUNCTION_UI_SCHEMA.json:304-315,479-497` and `src/go/pkg/topology/types.go:42-55,100-106`.
114 +- Current `topology:network-connections` includes socket direction in the link key and emits it as link `direction` and labels, so direction is part of socket identity and aggregation: `src/collectors/network-viewer.plugin/network-viewer.c:1359-1370,2532-2541,2630-2647`.
115 +- Current `topology:streaming` emits links from child/agent actor to parent/target actor based on stream path, so source/destination order is semantically directed even though the link object does not emit a separate `direction` field for those links: `src/web/api/functions/function-topology-streaming.c:1654-1738`.
116 +- Current SNMP/L2 topology projects discovery adjacencies as `unidirectional` until a reverse pair is merged into `bidirectional`; this is observation/completeness metadata for an L2 adjacency, not application traffic direction: `src/go/pkg/topology/engine/topology_adapter_projection_pairs.go:62-69,230-235,301-315`.
117 +- Current SNMP topology already attaches metric lookup fragments to actors and interface rows. Device actors include `chart_id_prefix`, `chart_context_prefix`, and `device_charts`, while port/status rows include `chart_id_suffix` and `available_metrics`: `src/go/plugin/go.d/collector/snmp_topology/topology_local_actor_attrs.go:61-68` and `src/go/plugin/go.d/collector/snmp_topology/topology_local_actor_charts.go:10-28,65-67,97-115`.
118 +- Current Cloud metric query guidance requires explicit `scope.contexts` to avoid metadata explosion and supports node/context/label/dimension filtering in request scope/selectors. This makes repeated full query payloads on every topology link a payload risk; topology should carry compact query references instead: `docs/netdata-ai/skills/query-netdata-cloud/query-metrics.md:9-13,182-219`.
119 +- Existing function validation tooling already validates Function output against `src/plugins.d/FUNCTION_UI_SCHEMA.json` and provides an E2E pattern that can be extended or mirrored for topology schema tests: `src/go/tools/functions-validation/README.md:1-42`.
120 +- Existing SNMP topology code already uses manifest/golden fixture tests for topology parity and real device scenarios, which is the right pattern for repeatable topology schema experiments: `src/go/pkg/topology/engine/parity/golden_fixture_test.go:1-40` and `src/go/pkg/topology/engine/parity/node_topology_parity_test.go:1-80`.
121 +- Frontend aggregation groups links after normalization by source, target, and link type: `src/domains/functions/topology/graphAggregation.js:58-125`. This reduces render complexity but does not reduce Cloud transfer size, JSON parse cost, or normalization memory cost.
122 +- Payload size analysis of the largest captured response shows the current response is minified but structurally verbose:
123 + - Full response: 134,180,370 bytes.
124 + - `data.links`: 126,858,432 bytes for 77,797 links.
125 + - `data.actors`: 7,316,909 bytes for 48 actors.
126 + - Link `src`/`dst` blocks alone account for 61,303,293 bytes.
127 + - Link `labels` blocks alone account for 32,556,526 bytes.
128 + - Link `metrics` blocks alone account for 11,191,014 bytes.
129 + - Link object key names repeat about 152 bytes per link, about 11,825,144 bytes total.
130 + - Actor `tables` account for 7,291,350 of 7,316,909 actor bytes; one process actor table accounts for 7,286,291 bytes.
131 +- Compact encoding estimates on the same captured response:
132 + - Current core link objects `{src_actor_id,dst_actor_id,link_type,direction,state,protocol,layer}`: 16,050,624 bytes.
133 + - The same core as arrays with string actor IDs: 9,749,067 bytes.
134 + - The same core as arrays with actor indexes: 4,006,272 bytes.
135 + - Actor indexes without repeating the constant layer: 3,617,287 bytes.
136 + - Actor indexes plus direction/protocol/state only: 2,917,072 bytes.
137 + - Backend graph aggregation by current UI dimensions collapses 77,797 raw links to 69 link groups; the grouped array representation is 3,794 bytes before actor data.
138 +- Link cardinality evidence: the largest response has only two protocol values, two state values, three direction values, two link type values, and one layer value, but `labels` and `metrics` objects are effectively per-link unique in the current contract. This means enum dictionary encoding is a clear win, while full per-socket details need a columnar/table form or a separate detail plane to reduce raw uncompressed size.
139 +- A 20,000 actor / 300,000 link target appears realistic under 100 MB only with an array-first, dictionary/columnar raw contract:
140 + - Measured on the largest captured response, core link rows encoded as arrays with enum indexes cost about 17 bytes/link before adjusting for larger actor-index digit width.
141 + - Link rows with core fields plus timestamps cost about 21 bytes/link before larger actor-index adjustment.
142 + - A full columnar/dictionary link row including endpoint, label, and metric fields cost about 222 bytes/link on the captured sample. Projected to 300,000 links this is about 66.7 MB, plus about 1.6 MB extra for wider actor indexes when actor indexes grow from two digits to up to five digits.
143 + - Sample actor rows without embedded tables cost about 180 bytes/actor with dictionary/columnar encoding, but actor richness is the main uncertainty. At 20,000 actors, 500 bytes/actor is about 10 MB, 1,000 bytes/actor is about 20 MB, and 1,500 bytes/actor is about 30 MB.
144 + - Therefore, a full-detail compact raw payload is expected to stay under 100 MB if average compact actor rows remain below roughly 1.5 KB and actor tables are not embedded in the graph actor list.
145 +- Array-first encoding alone is insufficient if it still carries object-shaped per-link endpoint/detail payloads. The captured response dropped from 126.9 MB links to 110.2 MB links when only top-level link keys were removed but `src`, `dst`, `labels`, and `metrics` remained object-shaped. The large reduction appears only when link details are represented as columnar arrays and string/enum dictionaries.
146 +- Socket-count scale evidence:
147 + - Linux sockets are file descriptors from the application point of view. `socket()` returns a file descriptor, and `accept()` creates a new connected socket with a new file descriptor. Therefore established TCP connection count is bounded in practice by per-process file descriptor limits, system-wide file-handle limits, TCP memory, socket buffers, and application architecture.
148 + - Linux kernel documentation defines `ip_local_port_range` as the automatic local port range for TCP/UDP. The documented default is `32768 60999`, which is 28,232 ports. A single client local IP talking to one server IP:port is therefore normally limited to about 28k simultaneous outbound TCP connections before local ephemeral-port exhaustion, unless the range, source IPs, namespaces, or explicit binding strategy differ.
149 + - A listening server is not bounded to 28k total inbound connections on one listening port. Each accepted TCP socket is identified by the full tuple. The practical upper bound becomes roughly client-source-IP fanout times client ephemeral-port availability, then file descriptors and memory. Many client IPs can therefore produce hundreds of thousands or millions of accepted sockets on one server.
150 + - Linux kernel documentation also exposes `tcp_max_tw_buckets`, `tcp_mem`, `tcp_max_orphans`, and TCP hash-bucket controls. These make TIME_WAIT/orphan/memory pressure part of worst-case socket inventory, not just established application-owned sockets.
151 + - The current workstation values at investigation time were `ip_local_port_range=32768 60999`, `tcp_max_tw_buckets=262144`, `somaxconn=4096`, `tcp_max_syn_backlog=4096`, `ulimit -n=524288`, `fs.nr_open=2147483584`, and effectively unlimited `fs.file-max`. These values are local evidence only, not product defaults.
152 + - `network-viewer` configures local-sockets collection with namespaces enabled, inbound/outbound enabled by default, all IPv4/IPv6 TCP/UDP protocols enabled by default, and local/listen graph output disabled by default. The helper reads `/proc/net/tcp`, `/proc/net/udp`, `/proc/net/tcp6`, and `/proc/net/udp6`, and the netlink path requests all socket states. Therefore the topology payload can reflect total socket inventory across host and container network namespaces, not only one process or one namespace.
153 + - Using the largest captured response as a sizing baseline, the old detailed payload costs about 1,725 bytes per emitted link including actor data and about 1,631 bytes per link for `data.links` alone. The 100 MiB Function cap is therefore crossed around 60k-64k links with the old format, matching the observed failure class.
154 + - The local lossless prototype costs about 522 bytes per socket-equivalent row on the captured response and would cross 100 MiB around 200k rows if the same actor/table shape repeats. The target columnar/dictionary estimate of about 222 bytes per full socket row supports about 236k rows in 50 MiB, 378k rows in 80 MiB, and 472k rows in 100 MiB before actor/dictionary overhead.
155 + - Phase 1 should therefore be engineered for hundreds of thousands of socket evidence rows per node in one compact response, while phase 2 chunking is required for honest million-socket scale.
156 +
157 +Risks:
158 +
159 +- Network-viewer is a C plugin; malformed topology building can cause a hard process exit if memory ownership, iterator invalidation, or JSON building is wrong.
160 +- Cloud-proxied Function errors may hide whether the agent, plugin, or Cloud bridge failed. Direct-agent validation may be needed, but only through token-safe bearer handling.
161 +- Raw topology payloads may contain process names, private IPs, hostnames, container identifiers, and other customer- or infrastructure-identifying data. They must stay in `.local/`.
162 +- Raising parser limits alone would move the failure boundary but would not fix the root problem: topology is shipping tens of thousands of near-duplicate graph links and 100+ MB JSON for fewer than 50 actors.
163 +
164 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
165 +
166 +Status at implementation start: open for schema documentation, lab work, validator/helper rails, and first Go producer migration; final closure evidence is in the Execution Log and Validation sections.
167 +
168 +Problem / root-cause model:
169 +
170 +- A Cloud-proxied `topology:network-connections` request can begin producing data but then fail with HTTP 503 and an agent-side `AgentError` saying the plugin exited before responding. Current evidence points to an oversized Function response operational cliff: the reported nodes can produce 110-134 MB topology JSON responses, while the Agent `pluginsd` parser has a 100 MiB deferred Function response cap. The completed responses contain few actors but tens of thousands of socket links because `processes:by_name` and `endpoints:by_ip` collapse actors but the backend still emits per-socket/per-port links. Root cause is not yet closed because the same token-safe API path returned HTTP 200 during investigation, so the failure is intermittent or path-dependent rather than deterministic.
171 +
172 +Evidence reviewed:
173 +
174 +- User-provided browser console evidence: HTTP 503 on `topology:network-connections` plus `AgentError` message `The plugin that was servicing this request, exited before responding.`
175 +- User-provided second request: HTTP 404 for the same Function on a different node.
176 +- Code search evidence: `src/collectors/network-viewer.plugin/network-viewer.c` defines `NETWORK_TOPOLOGY_VIEWER_FUNCTION` as `topology:network-connections`.
177 +- Project skill evidence: topology Functions are live Function snapshots and should not block collection loops.
178 +- Cloud probe evidence: raw bodies stored only under `.local/audits/network-connections-topology/`; durable artifacts record only aliases and aggregate counts.
179 +- Agent Function protocol evidence: `src/plugins.d/pluginsd_parser.h:18,195-207` enforces the 100 MiB deferred response cap, and `src/plugins.d/pluginsd_functions.c:96-97` emits the exact 503 error message observed by the browser when an in-flight Function has no body.
180 +- Network-viewer Function evidence: `src/collectors/network-viewer.plugin/network-viewer.c:1502-1557` prepares dictionaries and calls `local_sockets_process()`, `src/collectors/network-viewer.plugin/network-viewer.c:2703-2727` ignores cancellation state and sends the result only after full JSON generation, `src/collectors/network-viewer.plugin/network-viewer.c:1242-1253` collapses process actors by name, `src/collectors/network-viewer.plugin/network-viewer.c:1359-1370` keeps per-socket/per-port link keys, and the completed response stats show no orphan actors for captured successful payloads.
181 +- Cloud routing evidence: `cloud-charts-service/http/http.go:145-147,264-270`, `cloud-charts-service/internal/service/agent_data.go:565-585`, `cloud-charts-service/internal/model/errors.go:15`, and `cloud-charts-service/internal/routing/node_instance_filter.go:35-37` show the browser 404 is produced before or during Cloud node-instance selection, not by the network-viewer plugin itself.
182 +
183 +Affected contracts and surfaces:
184 +
185 +- `topology:network-connections` Function output.
186 +- `topology:streaming` Function output.
187 +- `topology:snmp` Function output.
188 +- Shared topology detailed/aggregated schema versioning and compatibility contract.
189 +- Network-viewer plugin process stability.
190 +- Cloud `/api/v2/nodes/{node}/function` proxy behavior as observed by users.
191 +- Cloud topology UI rendering and error handling.
192 +- Cloud topology aggregation microservice API and data model.
193 +- Function protocol compliance for topology responses.
194 +
195 +Existing patterns to reuse:
196 +
197 +- Network-viewer's existing `network-connections` table Function and `topology:network-connections` registration path.
198 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json` and Function UI reference for validating envelopes.
199 +- Token-safe query wrappers from `query-netdata-agents/scripts/_lib.sh`.
200 +- `.local/audits/` for raw, gitignored Cloud responses and sanitized summaries.
201 +
202 +Risk and blast radius:
203 +
204 +- Any fix in `network-viewer.c` can affect both topology and table `network-connections` Functions.
205 +- A fix that suppresses crashes by dropping data may hide real topology evidence and cause orphan or missing actors.
206 +- Direct production querying can expose sensitive data. Only sanitized aggregates and aliases are allowed in durable artifacts.
207 +
208 +Sensitive data handling plan:
209 +
210 +- Do not copy browser cookies, session tokens, Cloud tokens, bearer tokens, claim IDs, raw node UUIDs, raw machine GUIDs, raw hostnames, raw process command lines, raw IP addresses, or raw topology payloads into SOWs, specs, docs, skills, code comments, commits, or PR text.
211 +- Store raw API responses only under `.local/audits/network-connections-topology/`, which is gitignored.
212 +- Use aliases such as `node-503`, `node-404`, `endpoint-a`, and `process-a` in durable artifacts.
213 +- If a fixture is needed, derive a sanitized minimal reproducer with placeholder UUIDs and documentation that it is sanitized.
214 +
215 +Implementation plan:
216 +
217 +1. Keep SOW-0002 as the future unified merge/correlation SOW while this SOW owns the immediate topology Function payload migration.
218 +2. Define the new shared topology schema, including `detailed`, `aggregated`, compatibility versioning, extension handling, typed sentinels, actor/link/table columnar sections, actor-scope/grouping metadata, and canonical round-trip rules.
219 +3. Update Function schema/reference artifacts for the new topology payload contract.
220 +4. Add the `aggregated`/`detailed` request mode to `topology:network-connections`, with `aggregated` as the default. The exact parameter name remains to be finalized in the schema, but accepted mode values are fixed by user decision.
221 +5. Update `topology:network-connections`, `topology:streaming`, and `topology:snmp` to emit the new schema.
222 +6. Update the Cloud frontend to support both old and new schemas; isolate old-schema detailed normalization and frontend aggregation in compatibility modules so removal is straightforward.
223 +7. Design and implement a separate Cloud topology aggregation microservice that consumes detailed topology payloads and returns requested aggregated views.
224 +8. Preserve the incident-investigation validation for 503/404/orphan actors as regression coverage.
225 +
226 +Validation plan:
227 +
228 +- Token-safe Cloud Function checks for `node-503` and `node-404`.
229 +- Function discovery check for both nodes.
230 +- Source-level same-failure scan for topology actors emitted without corresponding links.
231 +- Compile or targeted test path for network-viewer/plugin Function code if a patch is made.
232 +- Schema validation of any captured successful topology response when feasible.
233 +- Golden old-detailed <-> new-detailed round-trip tests.
234 +- Synthetic edge-case detailed payload tests for sentinels, dynamic fields, sparse columns, duplicate-looking links, large indexes, and table-heavy actors.
235 +- Fuzz/property tests for lossless detailed conversion.
236 +- Cross-function schema tests for `topology:network-connections`, `topology:streaming`, and `topology:snmp`.
237 +- Cloud frontend compatibility tests for old schema path, new detailed path, and new aggregated path.
238 +- Cloud topology aggregation microservice tests proving requested aggregations are derived from detailed payloads and do not mutate the detailed source of truth.
239 +
240 +Artifact impact plan:
241 +
242 +- AGENTS.md: likely unaffected unless the project-wide topology schema migration workflow needs a new guardrail.
243 +- Runtime project skills: update `project-writing-collectors` with any reusable topology Function schema/validation rules exposed by the migration.
244 +- Specs: add/update a topology detailed/aggregated schema spec and record the lossless detailed invariant.
245 +- End-user/operator docs: update only if the public Function/query contract changes for users or downstream integrations.
246 +- End-user/operator skills: update `query-netdata-cloud` how-to coverage for `topology:network-connections` and the new detailed/aggregated request mode.
247 +- SOW lifecycle: SOW-0002 overlap is resolved by Decision 5; SOW-0012 implementation sequencing is unblocked by the user's 2026-05-06 statement that SOW-0012 is done. The SOW-0012 lifecycle file is still physically under `.agents/sow/current/` in this working tree and should be closed separately if needed.
248 +
249 +Open-source reference evidence:
250 +
251 +- Linux kernel documentation:
252 + - `https://docs.kernel.org/networking/ip-sysctl.html` documents `somaxconn`, `ip_local_port_range`, `tcp_max_tw_buckets`, `tcp_mem`, `tcp_max_orphans`, and TCP socket hash-bucket controls.
253 + - `https://docs.kernel.org/admin-guide/sysctl/fs.html` documents `file-max`, `file-nr`, and `nr_open`.
254 +- Linux man-pages:
255 + - `https://man7.org/linux/man-pages/man2/socket.2.html` documents that `socket()` returns a file descriptor.
256 + - `https://man7.org/linux/man-pages/man2/accept.2.html` documents that `accept()` creates a new connected socket and returns a new file descriptor, and that `EMFILE`, `ENFILE`, and `ENOBUFS`/`ENOMEM` are practical limit failures.
257 +- Apache Arrow official format docs (`https://arrow.apache.org/docs/format/Columnar.html`) describe a language-independent columnar format with data adjacency, random access, typed arrays, nested layouts, and dictionary encoding for repeated values. This supports the columnar/dictionary direction, but Netdata should keep JSON compatibility for Function transport unless a separate binary transport is explicitly chosen.
258 +- open-telemetry/otel-arrow @ 78856dcb2ecd93270265296c7c279cd9ab877e24:
259 + - `docs/otap_basics.md:53-68` splits one semantic telemetry signal across multiple normalized tables with foreign keys so it can be reconstructed.
260 + - `docs/otap_basics.md:136-149` records that dictionary encoding is an encoding choice and can vary by data characteristics.
261 +- grafana/grafana @ 1a416ef1a8724349c2c16b37f64bcf194a3d9b68:
262 + - `public/app/plugins/panel/nodeGraph/utils.ts:65-128` reads node and edge data as typed fields.
263 + - `public/app/plugins/panel/nodeGraph/utils.ts:134-185` transforms node/edge frames into layout objects for rendering.
264 + - `public/app/plugins/datasource/tempo/graphTransform.ts:76-137` builds separate node and edge data frames for a service-map graph.
265 +
266 +Open decisions:
267 +
268 +- Parameter naming for the `aggregated`/`detailed` request mode remains to be finalized.
269 +- Cloud topology aggregation microservice repository, ownership, API route shape, deployment model, and persistence/caching behavior remain to be finalized.
270 +
271 +## Implications And Decisions
272 +
273 +### Decision 1 - Detailed vs Aggregated Contract Terminology
274 +
275 +Date: 2026-05-06
276 +
277 +Decision:
278 +
279 +- Use the existing `network-connections` terminology: `aggregated` and `detailed`.
280 +- `aggregated` is a UI-oriented view only.
281 +- `detailed` is the canonical full-fidelity view for both UI drilldown and cross-node Cloud correlation.
282 +- Any new compact `detailed` topology payload must preserve every piece of information currently available in the old detailed payload.
283 +- A converter must be able to transform old detailed payloads to new detailed payloads and new detailed payloads back to old detailed payloads without information loss.
284 +
285 +Implications:
286 +
287 +- Compact detailed encoding may change representation, but not semantics.
288 +- Array/dictionary/columnar encoding is acceptable only if it preserves field names, field values, missing-vs-null-vs-empty distinctions where meaningful, row order where meaningful, dynamic fields, actor/link tables, endpoint details, labels, metrics, timestamps, and all data needed for future aggregation or correlation.
289 +- Aggregation must be derived from detailed data. It must not become the source of truth for correlation.
290 +- Any backend aggregation for UI must remain separate from detailed raw/correlation payloads.
291 +
292 +Risks:
293 +
294 +- A compact schema with fixed columns only would silently lose future or function-specific fields unless it includes an extension path.
295 +- Treating `aggregated` as suitable for Cloud correlation would destroy per-link/per-socket evidence and reduce future correlation options.
296 +
297 +Validation requirement:
298 +
299 +- Add round-trip compatibility tests when implementing the schema: old detailed -> new detailed -> old detailed must be canonically equivalent, and new detailed -> old detailed -> new detailed must preserve the same information model.
300 +
301 +### Decision 2 - Lossless Detailed Payload Validation Strategy
302 +
303 +Date: 2026-05-06
304 +
305 +Decision:
306 +
307 +- Losslessness must be validated through a canonical information model, not byte-for-byte JSON equality.
308 +- Tests must include both directions:
309 + - old detailed -> canonical -> new detailed -> canonical -> old detailed -> canonical
310 + - new detailed -> canonical -> old detailed -> canonical -> new detailed -> canonical
311 +- Every canonical representation in each chain must compare equal.
312 +
313 +Required test properties:
314 +
315 +- Preserve all known fields.
316 +- Preserve unknown/dynamic fields through extension storage.
317 +- Preserve missing, null, empty string, empty array, empty object, zero, and false distinctly where the old payload can distinguish them.
318 +- Preserve actor IDs, link IDs or reconstructable link identity, endpoints, labels, metrics, timestamps, actor tables, link tables, row ordering where semantically meaningful, and all cross-reference integrity.
319 +- Preserve enough information that all old aggregations and future cross-node correlation can be recomputed from the new detailed payload.
320 +
321 +Required test classes:
322 +
323 +- Golden fixture round-trip tests using sanitized old detailed payloads.
324 +- Synthetic edge-case fixtures for null/missing/empty, unknown fields, sparse columns, duplicate-looking links, large actor indexes, high cardinality labels, and table-heavy actors.
325 +- Property/fuzz tests that generate detailed payloads, round-trip them both ways, and assert canonical equality.
326 +- Field coverage tests that recursively enumerate old detailed JSON paths and fail when a path is not mapped, explicitly preserved as extension data, or intentionally rejected by a recorded decision.
327 +- Differential behavior tests that run the old aggregation/correlation logic on old detailed and on reconstructed old detailed from new detailed, then compare results.
328 +
329 +Implications:
330 +
331 +- The new compact detailed schema needs a typed extension path. Fixed arrays alone are not sufficient unless unknown fields have a lossless place to live.
332 +- The converter must be part of the contract and must be tested in CI alongside topology Functions and UI normalizers.
333 +- Aggregated payloads are not eligible for this lossless guarantee; only detailed payloads are.
334 +
335 +### Decision 3 - Topology Schema Migration Scope
336 +
337 +Date: 2026-05-06
338 +
339 +Decision:
340 +
341 +- The Function schema must be updated for the new topology payload contract.
342 +- `topology:network-connections` must expose an `aggregated`/`detailed` request mode using the existing `network-connections` terminology.
343 +- `aggregated` is the default mode for `topology:network-connections`.
344 +- `detailed` remains the full-fidelity lossless payload used by UI drilldown and Cloud cross-node correlation.
345 +- All topology Functions must emit the new schema:
346 + - `topology:network-connections`
347 + - `topology:streaming`
348 + - `topology:snmp`
349 +- The Cloud frontend must support both old and new schemas.
350 +- Frontend aggregation must be maintained only for the old-schema compatibility path.
351 +- Old-schema detailed normalization and aggregation code must be isolated so it is easy to remove after the compatibility window.
352 +- Cloud topology aggregation must be implemented as a separate microservice. It consumes detailed topology payloads, aggregates topologies, and outputs the aggregation requested by the user.
353 +
354 +Implications:
355 +
356 +- The new schema is not a `topology:network-connections`-only change. It becomes a shared topology contract across network connections, streaming, and SNMP.
357 +- The UI should not pay the old raw detailed payload cost for normal graph rendering once the new aggregated path is available.
358 +- Cloud correlation must use detailed payloads, not aggregated views.
359 +- The Cloud topology aggregation microservice must be able to recompute requested views from detailed topology data without mutating or weakening the detailed source of truth.
360 +- Backward compatibility in the Cloud frontend is required during rollout because old Agents and new Agents will coexist.
361 +
362 +Risks:
363 +
364 +- This is now a cross-repository migration touching Agent Functions, Function schemas, Cloud frontend, and a new Cloud aggregation service.
365 +- Supporting both old and new schemas can add long-lived compatibility code unless the old path is explicitly isolated and scheduled for removal.
366 +- If all topology Functions are not migrated consistently, Cloud aggregation will need per-function adapters and will become harder to reason about.
367 +
368 +Validation requirement:
369 +
370 +- Schema conformance tests must cover all topology Functions.
371 +- Frontend tests must prove old-schema aggregation remains only in the old compatibility path.
372 +- Cloud aggregation service tests must show requested aggregated outputs are reproducible from detailed inputs.
373 +- Rollout tests must include mixed old/new Agent responses.
374 +
375 +### Decision 4 - Cloud Aggregation Location
376 +
377 +Date: 2026-05-06
378 +
379 +Decision:
380 +
381 +- Cloud topology aggregation will be implemented as a separate Cloud microservice.
382 +- It will not be implemented inside charts-service.
383 +
384 +Context:
385 +
386 +- charts-service ownership was considered and recorded earlier on 2026-05-06.
387 +- The backend team changed direction and now prefers a separate service boundary for this capability.
388 +- charts-service still participates in Cloud Function routing for `/api/v2/nodes/{nodeID}/function` and node-instance selection, as recorded in the investigation evidence, so service integration with charts-service remains part of the design.
389 +
390 +Implications:
391 +
392 +- The SOW now includes a new Cloud service boundary, with repository, ownership, deployment, API, authentication/authorization, observability, and scaling decisions.
393 +- The aggregation service API, caching, request fan-out, and error handling must integrate cleanly with existing Cloud services, including charts-service where routing/proxy context is needed.
394 +- charts-service is no longer the implementation home for aggregation logic, but its integration path remains in scope.
395 +
396 +Risks:
397 +
398 +- A new service boundary adds deployment, operational, authentication, authorization, latency, retry, and observability work.
399 +- The service must bound CPU/memory use for detailed payload fetch, decoding, aggregation, and caching.
400 +- The integration path must avoid blocking or degrading existing charts/function proxy traffic.
401 +
402 +Validation requirement:
403 +
404 +- Add service-level tests for requested topology aggregations.
405 +- Add integration tests for charts-service/service interaction where charts-service participates in request routing.
406 +- Add resource-bound tests or benchmarks for large detailed inputs.
407 +- Validate that service errors preserve enough detail to distinguish node-instance routing failures, Agent Function failures, and aggregation failures.
408 +
409 +### Decision 5 - SOW-0002 / SOW-0020 Scope Boundary
410 +
411 +Date: 2026-05-06
412 +
413 +Decision:
414 +
415 +- SOW-0020 owns the immediate topology Function payload migration:
416 + - detailed and aggregated payload contract;
417 + - lossless compact detailed representation;
418 + - old/new schema compatibility;
419 + - `topology:network-connections`, `topology:streaming`, and `topology:snmp` emission requirements;
420 + - Cloud frontend compatibility requirements;
421 + - the separate Cloud topology aggregation microservice contract.
422 +- SOW-0002 remains pending and owns future unified topology merge semantics:
423 + - same-kind and cross-kind merge algorithms;
424 + - identity matching;
425 + - conflict resolution;
426 + - storage/indexing;
427 + - unified cross-layer view behavior.
428 +- SOW-0002 will consume SOW-0020 detailed payloads as source evidence when it eventually implements merge/correlation. It does not block SOW-0020 unless SOW-0020 would remove information needed for SOW-0002.
429 +
430 +Evidence:
431 +
432 +- SOW-0002 is still `open` and explicitly blocked on merge semantics, identity matching, conflict resolution, storage model, scale targets, and L7 process granularity.
433 +- SOW-0020 has concrete incident evidence that current topology payloads can exceed practical Function limits and needs a compact, lossless detailed contract now.
434 +- User decisions already recorded in this SOW require detailed payloads to remain full-fidelity and require aggregation to be derived from detailed data.
435 +
436 +Implications:
437 +
438 +- This SOW can proceed with the payload schema/spec and network-connections incident fixes without solving cross-layer merge semantics.
439 +- The detailed schema must keep enough extension capacity and source fidelity that SOW-0002 can later implement merge/correlation without another immediate payload rewrite.
440 +- SOW-0002 acceptance criteria that mention a unified schema depend on the SOW-0020 schema once this migration ships.
441 +
442 +Risks:
443 +
444 +- If SOW-0020 over-specializes the detailed schema for current UI rendering, SOW-0002 may need a schema vNext before merge work can proceed.
445 +- If SOW-0002 later needs correlation-specific provenance that SOW-0020 did not preserve, detailed payload compatibility tests will pass but future merge quality will suffer.
446 +
447 +Mitigation:
448 +
449 +- Treat `detailed` as the canonical evidence plane, not a render model.
450 +- Preserve unknown/dynamic fields and table data through typed extension storage.
451 +- Keep `aggregated` view-only and explicitly non-authoritative for correlation.
452 +
453 +### Decision 6 - Detailed Payload Socket Evidence And Footprint
454 +
455 +Date: 2026-05-08
456 +
457 +Decision:
458 +
459 +- The most important design goal for the new topology payload is minimizing footprint without losing evidence needed for Cloud-side cross-node matching.
460 +- `detailed` does not mean "one rendered graph edge per socket".
461 +- `detailed` may safely aggregate the graph projection when the aggregation is lossless with respect to reconstructing or correlating the underlying sockets.
462 +- `detailed` must preserve per-socket evidence one by one because Cloud needs socket tuples from multiple nodes to match both sides of a connection.
463 +- Per-socket evidence should move out of repeated object-shaped graph links and into a compact detail plane, such as a columnar `socket_rows` / `socket_evidence` table keyed back to the aggregated graph edge.
464 +- `aggregated` remains the view-oriented payload and does not carry the full per-socket evidence table.
465 +
466 +Evidence:
467 +
468 +- Current `network-viewer` topology link keys include process identity, local endpoint, remote endpoint, protocol, direction, state, and ports. This preserves correlation evidence but explodes link cardinality when actors are grouped.
469 +- Client-side outbound sockets can share the same server endpoint and can be graph-aggregated for rendering, but their local endpoint still matters for matching against server-side inbound observations.
470 +- Server-side inbound sockets see remote endpoint tuples; without the matching client-side local endpoint tuples, Cloud cannot prove which remote process owns each inbound socket.
471 +
472 +Implications:
473 +
474 +- The detailed payload should be structured as:
475 + - compact actor records grouped by selected actor scope;
476 + - compact graph links aggregated by actor pair / endpoint / protocol / direction / state as appropriate;
477 + - compact per-socket evidence rows that retain local endpoint, remote endpoint, protocol, direction, state, owner identity, namespace/container identity when available, and metric summaries needed for correlation.
478 +- The UI can render from the aggregated graph projection without inflating links to one row per socket.
479 +- Cloud correlation can use the per-socket evidence table to match observations from different nodes.
480 +- Future actor scopes such as container and Kubernetes labels require annotations/enrichment, but the payload contract should already allow those scope keys.
481 +
482 +Risks:
483 +
484 +- If `detailed` drops local client endpoint data during outbound aggregation, Cloud cannot reliably match those sockets to server-side inbound rows.
485 +- If per-socket evidence remains embedded in every graph link as repeated JSON objects, the payload-size problem remains.
486 +- If actor scope keys are not explicit, grouping by process name, PID, container, or Kubernetes labels will be hard to compare across agents and Cloud services.
487 +
488 +Validation requirement:
489 +
490 +- Add fixtures where many outbound sockets from one actor share one server endpoint and prove the rendered graph collapses while the socket evidence still preserves every local endpoint tuple.
491 +- Add fixtures where server inbound observations are matched to client outbound observations through the preserved socket evidence.
492 +- Add payload-size checks that fail if detailed output regresses toward one object-shaped graph link per socket.
493 +
494 +### Decision 7 - Phase 1 Payload Budget And Phase 2 Chunking
495 +
496 +Date: 2026-05-08
497 +
498 +Decision:
499 +
500 +- Paged/chunked socket evidence is the correct long-term answer when a lossless detailed topology response still cannot fit safely in one Function response.
501 +- Paged/chunked socket evidence is phase 2, not the immediate phase 1 implementation.
502 +- Phase 1 must do everything practical to prevent reaching the single-response limit:
503 + - keep `aggregated` as the default view;
504 + - keep graph links aggregated for rendering;
505 + - move per-socket evidence into compact columnar/dictionary tables;
506 + - avoid repeated object-shaped per-socket graph links;
507 + - avoid repeated strings where indexes/enums are sufficient;
508 + - keep actor tables and socket-evidence tables separate from the graph projection;
509 + - add payload-size regression tests and size budgets.
510 +- If phase 1 still cannot produce a complete lossless detailed payload within the safe response budget, it must fail explicitly rather than silently truncate.
511 +- Truncating per-socket evidence is not allowed because it breaks Cloud cross-node matching and can create false topology conclusions.
512 +
513 +Implications:
514 +
515 +- Phase 1 remains compatible with the current Function response model while reducing the probability of hitting the hard response cap.
516 +- The phase 1 schema must be designed so phase 2 chunking can add cursors/pages for socket evidence without another semantic rewrite.
517 +- The UI must not depend on receiving all per-socket rows for normal graph rendering; rendering should use the compact graph projection.
518 +- Cloud correlation can consume phase 1 detailed payloads where they fit, and phase 2 chunking will provide the scale-out path for larger estates.
519 +
520 +Risks:
521 +
522 +- Very large servers or very large Cloud selections may still exceed the safe response budget even after compact encoding.
523 +- If phase 1 schema couples graph rows and socket evidence too tightly, phase 2 chunking will require another disruptive schema migration.
524 +
525 +Validation requirement:
526 +
527 +- Add large synthetic payload tests that estimate serialized size for high socket counts and fail if the compact detailed format regresses.
528 +- Add an explicit over-budget behavior test proving the producer fails clearly and does not return partial socket evidence as complete data.
529 +- Add schema compatibility checks that reserve the phase 2 extension path for socket-evidence pagination/chunking.
530 +
531 +### Decision 8 - NAT/LB Matching Boundary
532 +
533 +Date: 2026-05-08
534 +
535 +Decision:
536 +
537 +- NAT, load-balancer, proxy, and masquerade correlation are out of scope for the current phase.
538 +- Cloud-side socket matching must use only exact observable socket evidence available from the endpoints.
539 +- When observable tuples cannot prove both sides of a connection, the topology must leave the edge unresolved rather than infer a process-to-process match.
540 +- NAT/LB/proxy-aware matching requires future evidence sources such as flow data, proxy telemetry, conntrack/NAT metadata, or explicit service annotations.
541 +
542 +Implications:
543 +
544 +- The current implementation should be truthful before it is complete.
545 +- Exact tuple matching can build reliable process-to-process edges where both sides expose compatible local/remote endpoint evidence.
546 +- NAT/LB/proxy paths may remain as endpoint or infrastructure edges until a future SOW adds the required evidence.
547 +
548 +Risks:
549 +
550 +- Users may see incomplete cross-node process maps behind NAT, load balancers, proxies, service meshes, or SNAT-heavy Kubernetes paths.
551 +- Heuristic matching would make the graph look more complete but would introduce false positives during incident analysis, so it is explicitly rejected for this phase.
552 +
553 +Validation requirement:
554 +
555 +- Add matching tests proving exact observable tuple matches are accepted.
556 +- Add NAT/LB-like fixtures proving non-proven edges remain unresolved and are not heuristically matched.
557 +
558 +### Decision 9 - Multi-Level Actor Scopes For Infrastructure Topologies
559 +
560 +Date: 2026-05-09
561 +
562 +Decision:
563 +
564 +- The detailed topology payload must separate evidence from presentation/grouping.
565 +- Cloud aggregation must be able to materialize different actor scopes from the same detailed evidence, without requiring a different Agent payload for each view.
566 +- Required actor scopes include:
567 + - node-level infrastructure dependency maps, for large pet-style fleets where users need to see which nodes depend on which nodes;
568 + - container/application-level dependency maps, for large Kubernetes clusters where users need to see which applications or container names depend on which other applications or container names, not individual container instances;
569 + - process-name-level dependency maps, for large HPC nodes where users need to see which process names depend on which other process names, not individual PIDs;
570 + - PID-level drilldown, where exact process-instance evidence is needed.
571 +- The schema must preserve raw evidence needed to derive those scopes:
572 + - Netdata node identity and host/network identity;
573 + - process name, PID, UID, command line where available, and network namespace identity;
574 + - container identity, container name, image, pod, namespace, workload, and Kubernetes labels when enrichment is available;
575 + - local/remote socket endpoint tuples and direction/state/protocol evidence.
576 +- Actor scopes must be explicit metadata in the payload or aggregation request. They must not be inferred only from actor ID string formats.
577 +- Container and Kubernetes label support is an enrichment requirement. The current network-viewer producer does not yet have enough evidence for those scopes, but the schema must be ready for them.
578 +
579 +Evidence:
580 +
581 +- Current `topology:network-connections` supports process grouping by process name by default and PID when requested.
582 +- Current network-viewer classifies socket namespace type as `system`, `unknown`, or `container` from network namespace identity, but does not yet expose container name, pod, workload, or Kubernetes labels.
583 +- Current shared topology `match` schema already has fields for node, container, pod, and namespace identities, and allows additional match properties.
584 +
585 +Implications:
586 +
587 +- The Agent detailed payload should behave like an evidence plane. It should expose compact rows with stable dimensions and enrichment columns, not only pre-rendered actor/link objects.
588 +- The Cloud aggregator should own materialized views such as `group_by=node`, `group_by=process.name`, `group_by=process.pid`, `group_by=container.name`, `group_by=k8s.workload`, or selected Kubernetes labels.
589 +- Aggregated payloads should state the actor scope used to produce them, so users and downstream systems know what each node in the graph represents.
590 +- The same per-socket evidence can support infrastructure maps, application maps, and process maps when the required identity columns are present.
591 +
592 +Risks:
593 +
594 +- If the schema bakes in only process actors, it will not serve infrastructure-level or Kubernetes-level topology without another migration.
595 +- If grouping scope is inferred from display labels, Cloud can merge unrelated actors that happen to share a name.
596 +- Container-name grouping can intentionally merge many instances, so the payload must preserve instance-level evidence for drilldown and cross-checking.
597 +
598 +Validation requirement:
599 +
600 +- Add schema fixtures proving the same detailed socket evidence can produce node-level, process-name-level, and PID-level aggregations.
601 +- Add synthetic enriched fixtures proving container-name and Kubernetes-label grouping can be derived without changing the detailed evidence contract.
602 +- Add collision tests where two different actor instances share the same display name but must remain distinguishable in detailed evidence.
603 +
604 +### Decision 10 - Actor Drilldown Tables Without Duplicating Evidence
605 +
606 +Date: 2026-05-09
607 +
608 +Decision:
609 +
610 +- Actor drilldown tables are a first-class topology requirement.
611 +- Every materialized actor scope must be able to expose per-actor drilldown tables that list exact dependencies and supporting evidence.
612 +- Drilldown tables must support both:
613 + - aggregated rows, such as one row per peer actor, peer node, peer container, endpoint, protocol, direction, or state with counts and summaries;
614 + - exact rows, such as socket evidence rows, when the user needs to inspect the underlying connections.
615 +- The schema must avoid duplicating the same detailed socket/link evidence inside every actor.
616 +- Actor tables should be materialized views over shared compact evidence, using table definitions, row references, row ranges, filters, or indexes where possible.
617 +- When an aggregated actor table row summarizes many sockets, it must carry enough references or drilldown keys to retrieve or reconstruct the exact rows from the detailed evidence plane.
618 +
619 +Evidence:
620 +
621 +- The current shared topology schema already allows `actor.tables`, so actor-specific drilldown data is part of the existing contract.
622 +- The Cloud frontend actor modal reads presentation table definitions and renders `source: "data"` tables from `node.details.tables[tableKey]` or `source: "links"` tables from incident graph links.
623 +- Current `topology:network-connections` emits per-process `tables.sockets` rows directly under each process actor, which satisfies the drilldown UX but duplicates information already present in link/socket evidence.
624 +
625 +Implications:
626 +
627 +- The new compact schema should keep the modal/table capability, but move repeated row payload out of actor objects.
628 +- Presentation metadata should define table columns and sources, while data sections should provide compact shared table storage and per-actor indexes into it.
629 +- Aggregated topology responses can expose aggregated drilldown rows, but detailed responses must preserve the exact evidence needed to expand them.
630 +- Frontend compatibility code can reconstruct old `actor.details.tables` for old UI paths, but new-schema UI should prefer shared table sections and references.
631 +
632 +Risks:
633 +
634 +- If actor tables remain embedded as full object arrays, large actors will recreate the same payload-size problem even after graph links are compacted.
635 +- If aggregated table rows do not reference the underlying evidence, users will lose trust because they cannot drill from summary to exact dependency rows.
636 +- If table definitions are tied to one actor scope, Cloud will need separate payload shapes for node, container, process, and PID views.
637 +
638 +Validation requirement:
639 +
640 +- Add fixtures where a high-cardinality actor has many socket rows and prove modal drilldown works without duplicating full rows under the actor.
641 +- Add tests proving aggregated table rows preserve counts/summaries and can drill down to exact socket evidence rows.
642 +- Add compatibility tests proving old-schema `actor.tables` can still be normalized while new-schema shared table references render the same user-visible rows.
643 +
644 +### Decision 11 - Actor Custom Tables vs Relationship Evidence Tables
645 +
646 +Date: 2026-05-09
647 +
648 +Decision:
649 +
650 +- The topology schema must differentiate table semantics, not only table rendering source.
651 +- `source: "data"` and `source: "links"` remain presentation/rendering hints, but they are not sufficient as aggregation semantics.
652 +- Every table definition or table section should declare a semantic role, such as:
653 + - `relationship_evidence`: exact rows that describe dependencies or links and can be grouped/rolled up, such as socket evidence;
654 + - `relationship_summary`: aggregated dependency rows derived from evidence, such as peer actor summaries with counts and drilldown keys;
655 + - `actor_detail`: actor-owned custom data that describes the actor and is not generally aggregatable, such as streaming path, retention, status, capabilities, or local inventory;
656 + - `actor_inventory`: actor-owned lists that may be searchable/displayable but should not be treated as dependency evidence unless explicitly referenced by a relationship table.
657 +- Every table should also declare an aggregation policy:
658 + - `none`: preserve per actor, do not merge across aggregated actors unless explicitly requested;
659 + - `derive`: recompute from detailed relationship evidence for the requested actor scope;
660 + - `rollup`: combine rows with declared group keys and measures;
661 + - `reference`: display rows by references/ranges/filters into a shared table section.
662 +- Custom actor tables may still be stored compactly in shared columnar sections with an owner actor column, but their semantic role remains actor-owned detail. Storing them compactly must not imply they are aggregatable relationship evidence.
663 +- When Cloud aggregates actors, relationship tables may be recomputed for the aggregated actor. Actor-detail/custom tables must either remain attached to their original member actors, be exposed as drilldown/member details, or use an explicit table-specific rollup policy.
664 +
665 +Evidence:
666 +
667 +- Streaming topology `streaming_path` is declared as a data table for the actor modal, but its rows describe the actor's path metadata, not an aggregatable dependency list.
668 +- Streaming topology retention/status tables are actor details or operational summaries, not always link evidence.
669 +- Network-connections socket rows are dependency evidence and can support exact drilldown plus aggregated summaries.
670 +- The current schema's `source` enum only says where the frontend reads rows from, not whether the rows are actor-owned custom data or relationship evidence.
671 +
672 +Implications:
673 +
674 +- The new schema needs table metadata beyond `source`, for example `role`, `aggregation_policy`, `owner_scope`, `evidence_ref`, `group_keys`, and `measures`.
675 +- The Cloud aggregator must not blindly merge custom actor tables when materializing node/container/process-level views.
676 +- UI can still render all table roles in the same modal, but Cloud services need the role/policy fields to avoid incorrect aggregation.
677 +- Compatibility adapters can map old `source: "data"` tables to conservative defaults:
678 + - network-connections `sockets` -> `relationship_evidence`;
679 + - streaming `streaming_path` -> `actor_detail`;
680 + - streaming `retention` -> `actor_detail` unless a future explicit rollup policy is added;
681 + - streaming `inbound` / `outbound` -> relationship or operational summary based on their declared policy.
682 +
683 +Risks:
684 +
685 +- If table semantics are not explicit, Cloud may aggregate custom actor metadata incorrectly and present false information.
686 +- If all custom data remains embedded as actor-local object arrays, very large custom tables can still inflate payloads.
687 +- If custom tables are treated as non-aggregatable without a drilldown/member path, users may lose useful details when viewing high-level aggregated actors.
688 +
689 +Validation requirement:
690 +
691 +- Add streaming topology fixtures with `streaming_path` and retention tables proving actor-detail tables survive schema conversion and are not merged as dependency evidence.
692 +- Add network-connections fixtures proving socket tables are relationship evidence and can produce both aggregated dependency summaries and exact drilldown rows.
693 +- Add mixed aggregation tests where multiple actors are grouped and custom actor-detail tables remain accessible as member details rather than incorrectly merged.
694 +
695 +### Decision 12 - Link Direction Semantics And Aggregation Policy
696 +
697 +Date: 2026-05-09
698 +
699 +Decision:
700 +
701 +- The schema must separate the link's endpoint order from the meaning of its `direction` value.
702 +- Each link type must declare direction semantics and aggregation policy. A free-form link-level `direction` string is not enough.
703 +- Required link-type metadata includes:
704 + - `orientation`: whether the graph relationship is `directed`, `undirected`, or `hierarchical`;
705 + - `direction_role`: what the link's direction value means, such as `flow`, `dependency`, `containment`, `observation_completeness`, or `none`;
706 + - `aggregation_direction_policy`: whether Cloud must `preserve` direction in aggregation keys, `ignore` direction and canonicalize endpoint pairs, or `retain_as_attribute` while aggregating independently of direction;
707 + - optional render hints such as whether arrows or curved parallel links are appropriate. The exact UI implementation is out of scope for this SOW, but the schema must carry the information.
708 +- Direction-significant link types, such as network socket flows and streaming parent/child paths, must preserve direction during aggregation.
709 +- Undirected adjacency link types, such as most L2 links, must allow Cloud to aggregate independently of direction while retaining observation metadata such as `unidirectional` or `bidirectional` when useful.
710 +- Backward compatibility may keep the existing `direction` field, but new producers and aggregators must interpret it through the declared link-type semantics.
711 +
712 +Examples:
713 +
714 +- Network socket link type:
715 + - `orientation: directed`
716 + - `direction_role: flow`
717 + - `aggregation_direction_policy: preserve`
718 + - Rationale: inbound and outbound sockets have different source/destination meaning and different cross-node matching behavior.
719 +- Streaming link type:
720 + - `orientation: directed`
721 + - `direction_role: dependency`
722 + - `aggregation_direction_policy: preserve`
723 + - Rationale: child-to-parent streaming direction is the topology relation.
724 +- Ownership/containment link type:
725 + - `orientation: hierarchical`
726 + - `direction_role: containment`
727 + - `aggregation_direction_policy: preserve`
728 + - Rationale: parent contains child; reversing endpoints changes meaning.
729 +- L2 discovery/adjacency link types:
730 + - `orientation: undirected`
731 + - `direction_role: observation_completeness`
732 + - `aggregation_direction_policy: retain_as_attribute`
733 + - Rationale: `unidirectional` and `bidirectional` describe whether one side or both sides reported evidence, not traffic direction. Cloud can aggregate endpoint pairs independently of direction, while preserving the observation status for details.
734 +
735 +Evidence:
736 +
737 +- Shared schema has an optional `direction` string on links but no declared semantics.
738 +- Network-viewer uses socket direction as part of link identity and emits it into links and labels.
739 +- Streaming links are direction-significant through source/destination order even without a separate `direction` field.
740 +- SNMP/L2 projection uses `unidirectional`/`bidirectional` to describe observation completeness and merges reverse evidence into one bidirectional link.
741 +
742 +Implications:
743 +
744 +- The Cloud aggregator must read link-type direction metadata before deciding its aggregation key.
745 +- UI rendering can use the same metadata to decide whether arrows/curves are meaningful, but rendering behavior will be implemented separately.
746 +- Tables and drilldown should expose observation completeness separately from graph direction when `direction_role` is not `flow` or `dependency`.
747 +
748 +Risks:
749 +
750 +- If direction semantics stay implicit, Cloud may incorrectly merge directional dependencies or incorrectly split undirected L2 adjacencies.
751 +- If the UI uses the raw `direction` field alone, it may draw arrows for L2 `unidirectional` evidence even though the user should read it as discovery completeness, not traffic direction.
752 +
753 +Validation requirement:
754 +
755 +- Add schema fixtures for directed socket links proving opposite directions do not collapse unless a requested aggregation explicitly allows it.
756 +- Add streaming fixtures proving child-to-parent direction survives aggregation.
757 +- Add L2 fixtures proving reverse discovery evidence can aggregate to one undirected adjacency while retaining `unidirectional`/`bidirectional` observation metadata.
758 +
759 +### Decision 13 - Refreshable Link Telemetry Overlays
760 +
761 +Date: 2026-05-09
762 +
763 +Decision:
764 +
765 +- The topology schema must support link-level telemetry overlays that can be refreshed without recomputing the topology graph.
766 +- Topology responses should define telemetry overlay templates once per response, view, or link type. Links should carry only a template identifier plus compact parameters.
767 +- Overlay templates must describe:
768 + - the provider kind, such as Cloud time-series metrics, direct Agent metrics, or Function-backed current snapshots;
769 + - the metric families exposed by the overlay, such as traffic, packets, errors, state, utilization, or future plugin-specific measurements;
770 + - required parameter names and their meaning, such as node IDs, host selectors, chart/context prefixes, chart suffixes, interface labels, socket tuple IDs, actor IDs, or compact dictionary references;
771 + - query construction rules, including required contexts, dimensions, labels/selectors, and target nodes;
772 + - merge rules for aggregated links, including whether parameters can be unioned, summed, counted, deduplicated, or must remain as separate query references;
773 + - coverage semantics, such as exact per-link, exact actor-scope, approximate actor-scope, unsupported, or snapshot-only.
774 +- Link records should reference overlay templates through compact telemetry refs rather than embedding full query definitions.
775 +- Aggregated links must merge telemetry refs according to template-defined merge policy. The aggregator must not infer merge behavior by string concatenating query fragments.
776 +- If multiple telemetry refs cannot be merged safely, the aggregated link may carry multiple refs under the same overlay metric, and the UI or Cloud overlay layer can query and combine them according to the template.
777 +- Topology payloads should distinguish static/snapshot metrics already present in the topology response from refreshable overlay definitions.
778 +
779 +Examples:
780 +
781 +- SNMP/L2:
782 + - template kind: time-series metric query;
783 + - parameters: monitored node or vnode, chart/context prefix, local interface chart suffix or interface labels;
784 + - metric families: traffic, packets, errors, operational state;
785 + - merge policy: traffic/packets/errors can usually sum across interfaces; state must use a state-specific rule such as worst-state, count-by-state, or separate member details, not sum;
786 + - coverage: exact for the selected interface rows when the SNMP collector emits the required chart/label references.
787 +- Network-viewer process aggregation:
788 + - template kind: none today, future Function-backed current snapshot;
789 + - parameters could be actor scope plus compact socket/link evidence IDs;
790 + - coverage must be declared as unsupported until the plugin can provide exact per-link current traffic snapshots.
791 +- Network-viewer container aggregation:
792 + - cgroup network metrics may provide container-interface time series, but they do not identify the remote peer link;
793 + - coverage must therefore be exact actor-scope or approximate actor-scope, not exact per-link, unless future evidence adds peer-aware counters.
794 +
795 +Evidence:
796 +
797 +- SNMP topology already exposes chart lookup fragments on devices and interface rows.
798 +- Cloud metric queries must be tightly scoped by context/selector to avoid metadata explosion.
799 +- Current topology links have only a generic `metrics` object and no refreshable query contract.
800 +- Network-viewer currently collects socket identity and TCP summaries, but not per-link traffic time series.
801 +- cgroup network metrics are container/interface scoped, not dependency-link scoped. The cgroups integration documents per-cgroup and per-k8s-cgroup network-device contexts such as `cgroup.net_net`, `cgroup.net_packets`, `cgroup.net_errors`, `k8s.cgroup.net_net`, and `k8s.cgroup.net_packets`: `src/collectors/cgroups.plugin/integrations/containers.md:145-172` and `src/collectors/cgroups.plugin/integrations/kubernetes_containers.md:160-194`.
802 +
803 +Implications:
804 +
805 +- The schema needs a dedicated overlay/telemetry contract, separate from actor tables, relationship evidence tables, and rendered graph links.
806 +- The UI can refresh traffic by issuing targeted metric or Function queries using stable overlay refs, without calling the topology Function again.
807 +- Cloud aggregation can merge telemetry refs at the same time it merges graph links, preserving enough information to calculate aggregate bandwidth for the aggregated edge.
808 +- Overlay refs must be compact and dictionary-friendly because they may appear on many links.
809 +- Overlay refs should carry coverage/confidence so the UI does not present container-level interface traffic as exact peer-to-peer link bandwidth.
810 +
811 +Risks:
812 +
813 +- If the schema stores full query JSON per link, the payload may grow as badly as today's repeated link objects.
814 +- If overlay coverage is not explicit, users may see bandwidth on a link and assume exact peer traffic when the source is actually actor- or interface-level.
815 +- If merge policies are not defined per metric family, Cloud may sum values that should use state-aware or non-additive aggregation.
816 +- If overlay refs are tied to unstable chart IDs without stable labels or repair metadata, topology links may refresh incorrectly after interface rename, container restart, or vnode reassignment.
817 +
818 +Validation requirement:
819 +
820 +- Add fixtures proving many SNMP/L2 links can reference one overlay template with only per-link parameters.
821 +- Add aggregation fixtures proving merged links union member telemetry refs and produce correct additive traffic queries.
822 +- Add state/error fixtures proving non-additive metrics do not use additive merge rules.
823 +- Add network-viewer fixtures proving unsupported overlays are represented explicitly and do not imply exact link traffic.
824 +
825 +### Decision 14 - Schema Emulation Lab Before Producer/UI Migration
826 +
827 +Date: 2026-05-09
828 +
829 +Decision:
830 +
831 +- Do not freeze the topology schema based only on hand-written examples.
832 +- Build a schema emulation and benchmarking harness before updating all topology Functions and the UI.
833 +- The harness must model required use cases at multiple scales, generate candidate payloads, run a prototype aggregator, and produce repeatable evidence for:
834 + - serialized payload size;
835 + - compressed payload size where relevant;
836 + - decode/encode CPU time;
837 + - peak memory where measurable;
838 + - actor/link/socket/relationship evidence counts;
839 + - lossless detailed round-trip correctness;
840 + - aggregation correctness for each required actor scope;
841 + - drilldown table correctness;
842 + - direction-preservation/canonicalization behavior;
843 + - telemetry overlay ref merging behavior.
844 +- The harness output must be durable enough for review and CI regression checks, but raw captured production payloads must stay under `.local/` and must not be committed.
845 +- Synthetic scenarios can be committed when they contain no sensitive data and no customer-identifying identifiers.
846 +- Use read-only captures from internal Netdata-owned Kubernetes infrastructure as a real-world corpus when explicitly authorized for a capture run. Raw payloads, endpoint details, node names, hostnames, process names, IPs, cluster labels, and IDs must stay under `.local/`; only sanitized shape statistics, generated synthetic fixtures, and redacted/canonicalized summaries may be committed.
847 +- For the current real-corpus capture, scope Cloud queries only to the user-authorized space named `Netdata Cloud`. Other spaces visible to the token are out of scope and must not be used for topology payload capture.
848 +
849 +Required modeled scenarios:
850 +
851 +- Real captured topology Function payloads from internal Netdata-owned Kubernetes infrastructure, used as local-only corpus inputs for size and schema-shape analysis.
852 +- Network-viewer process-name aggregation on one node with many sockets to common destination endpoints.
853 +- Network-viewer cross-node matching with client outbound and server inbound evidence preserved one by one.
854 +- Network-viewer node-level infrastructure map across many nodes.
855 +- Network-viewer enriched container/application and Kubernetes-label grouping, using synthetic enrichment until real enrichment exists.
856 +- SNMP/L2 adjacency with unidirectional and bidirectional evidence, many devices, many ports, and metric overlay refs for traffic/packets/errors/state.
857 +- Streaming topology with actor-owned custom tables such as `streaming_path`, plus relationship tables and directed child-to-parent links.
858 +- Telemetry overlay aggregation where additive metrics, state metrics, unsupported overlays, and incompatible refs all behave differently.
859 +
860 +Scale points:
861 +
862 +- Small: developer-readable fixtures for schema review and golden tests.
863 +- Medium: tens to hundreds of actors and thousands of links/evidence rows.
864 +- Large: thousands of actors and hundreds of thousands of socket/interface evidence rows.
865 +- Stress: one million socket evidence rows, expected to demonstrate whether phase 1 fits or must require phase 2 paging/chunking.
866 +
867 +Candidate strategies to compare:
868 +
869 +- Current object-shaped topology payload as the baseline.
870 +- Array-of-objects with dictionaries.
871 +- Columnar tables with dictionaries.
872 +- Split graph projection plus evidence planes.
873 +- Template-based telemetry overlay refs.
874 +- Optional compression measurements, while treating compression as transport relief rather than a schema substitute.
875 +
876 +Aggregator prototype requirements:
877 +
878 +- Consume the same detailed candidate payload the Agent would emit.
879 +- Produce requested views by actor scope: node, process name, PID, container/application, Kubernetes labels/workload.
880 +- Preserve socket evidence for cross-node matching and actor drilldown.
881 +- Apply link-type direction policy and overlay merge policy from schema metadata.
882 +- Produce deterministic output for golden tests.
883 +
884 +Evidence:
885 +
886 +- Existing payload evidence already shows object-shaped per-link JSON is the primary size problem.
887 +- Existing function validation and SNMP topology parity tests provide local patterns for schema validation, E2E checks, and golden fixtures.
888 +- External topology systems such as Kiali, Pixie, and Coroot compute traffic/dependency views from metric or event queries, but their implementation patterns also show that aggregation semantics are source-specific and cannot be inferred from a generic link alone.
889 +- The user has access to internal Netdata-owned Kubernetes infrastructure where existing topology Function payloads can be captured and analyzed locally. This gives the schema lab real-world payload shapes in addition to synthetic scale models, but it requires strict raw-data isolation and sanitization.
890 +
891 +Implications:
892 +
893 +- The schema design work becomes experiment-driven: candidate schema changes must be backed by size and correctness results before producer/UI migration starts.
894 +- The first implementation artifact should be a topology schema lab/test package and prototype aggregator, not changes to all producers.
895 +- The lab should support both synthetic scenario generation and importing raw local Function payloads from `.local/` so the same candidate schema and aggregator can be tested against real and synthetic data.
896 +- Once the schema wins against the scenarios, producer and UI changes can proceed with much lower risk.
897 +
898 +Risks:
899 +
900 +- If the lab becomes too detached from real producer payloads, it may optimize synthetic data and miss real-world fields.
901 +- If the lab only tests size and not semantics, it may choose a compact format that cannot support drilldown, cross-node matching, direction semantics, or overlays.
902 +- If raw captured payloads are committed, sensitive infrastructure data may leak.
903 +- If captures are taken from live infrastructure without clear read-only scope, rate limits, and local-only storage, the schema lab could create operational risk or expose internal topology details.
904 +
905 +Validation requirement:
906 +
907 +- CI-friendly tests must fail on payload-size regressions for representative scenarios.
908 +- Golden tests must prove aggregation output is stable for each modeled use case.
909 +- Round-trip tests must prove detailed schema alternatives preserve the canonical information model.
910 +- Real-corpus import tests must operate on `.local/` payloads when available, but CI must use sanitized synthetic fixtures only.
911 +- Stress tests that are too expensive for default CI may run behind an explicit build tag or local benchmark command, but their command and expected budget must be documented.
912 +
913 +### Decision 15 - Producer Encoding Helper Strategy
914 +
915 +Date: 2026-05-09
916 +
917 +Decision:
918 +
919 +- The user classified the exact helper split as an implementation detail and authorized using the safest path.
920 +- Implement the Go topology v1 model and compact-table helper first.
921 +- Defer the C helper API shape until the first C producer migration, so the C helper is informed by the actual `topology:network-connections` and `topology:streaming` write paths instead of guessed in advance.
922 +
923 +Evidence:
924 +
925 +- Go already has a central topology package, but it currently models the old object-shaped schema in `src/go/pkg/topology/types.go`.
926 +- The SNMP topology Function returns Go topology data directly from `src/go/plugin/go.d/collector/snmp_topology/func_topology_handler.go`.
927 +- The Network Viewer and Streaming producers write JSON manually with `BUFFER` helpers in C, and they have different table and link construction patterns.
928 +
929 +Implications:
930 +
931 +- Go producers and Go-side tooling can share typed compact-table construction and validation immediately.
932 +- The first C producer migration must include the C helper decision instead of duplicating compact-table encoding by hand.
933 +- This avoids freezing a C API before validating it against the high-cardinality Network Viewer socket evidence path and the Streaming custom-table path.
934 +
935 +Risks:
936 +
937 +- Until the C helper is added, C producer migration work remains blocked on a follow-up helper design step.
938 +- If Go helper types drift from the JSON Schema, tests must catch the mismatch before producers rely on the helper.
939 +
940 +Validation requirement:
941 +
942 +- Add Go tests for compact-table row counts, encoding length checks, dictionary index bounds, actor/link reference columns, and JSON round-trip behavior.
943 +- Validate committed topology fixtures against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` and the semantic checks in `src/go/tools/functions-validation/validate`.
944 +
945 +### Decision 16 - Nested Custom Detail Table Cells
946 +
947 +Date: 2026-05-09
948 +
949 +Decision:
950 +
951 +- Add an explicit `json` column type for actor/custom detail table cells that must preserve nested producer-owned data.
952 +- Keep high-cardinality relationship evidence typed as scalar/reference/array columns whenever possible.
953 +- Treat `json` cells as not generally aggregatable unless a table type explicitly defines a safe aggregation policy for that table.
954 +
955 +Evidence:
956 +
957 +- Current SNMP/L2 actor port details can include nested producer data. `src/go/pkg/topology/engine/topology_adapter_device_summary_render.go:35` emits `vlans` from `[]map[string]any`, and `src/go/pkg/topology/engine/topology_adapter_device_summary_render.go:44` builds `neighbors` as `[]map[string]any`.
958 +- The user requirement says actor custom tables, such as streaming paths and topology-specific per-actor data, must remain supported and must be differentiated from aggregatable relationship evidence.
959 +
960 +Implications:
961 +
962 +- The schema can preserve existing custom actor detail tables without flattening away topology-specific structure.
963 +- `json` columns are a compactness escape hatch for low-cardinality custom detail data, not the default representation for sockets, L2 observations, or other high-cardinality evidence.
964 +- Aggregators may append or retain `json` detail rows according to table policy, but should not infer generic merge semantics from arbitrary nested values.
965 +
966 +Risks:
967 +
968 +- Overusing `json` columns in evidence tables would recreate object-shaped payload bloat and weaken aggregation semantics.
969 +- Producers must still prefer typed scalar/reference columns when a value participates in identity, matching, grouping, or metric aggregation.
970 +
971 +Validation requirement:
972 +
973 +- Schema tests must include a typed topology v1 payload with a `json` custom detail column containing nested arrays/objects.
974 +- Producer docs and skills must warn against using `json` for high-cardinality evidence when typed columns are possible.
975 +
976 +## Plan
977 +
978 +1. Keep SOW-0002 as the future unified merge-semantics SOW while this SOW owns the immediate shared topology payload schema migration.
979 +2. Build the topology schema emulation lab and prototype aggregator with representative scenarios, scale profiles, and size/correctness reports.
980 +3. Use the lab to compare candidate schemas and freeze the detailed/aggregated topology contract with measured evidence.
981 +4. Draft the new detailed/aggregated topology schema, including compatibility versioning, typed sentinels, dictionaries, columnar sections, extension preservation, table handling, and refreshable telemetry overlay templates.
982 +5. Promote the local lossless converter prototype into production-grade tests and reusable conversion code.
983 +6. Update Function schema/reference artifacts for topology detailed/aggregated payloads.
984 +7. Update Agent topology Functions:
985 + - `topology:network-connections`
986 + - `topology:streaming`
987 + - `topology:snmp`
988 +8. Update the Cloud frontend compatibility architecture:
989 + - old schema adapter path
990 + - old frontend aggregation isolated in old path only
991 + - new detailed adapter path
992 + - new aggregated adapter path
993 +9. Design and implement the separate Cloud topology aggregation microservice.
994 +10. Validate losslessness, aggregation equivalence, frontend compatibility, mixed-version rollout, and the original 503/404/orphan-actor failure classes.
995 +
996 +## Execution Log
997 +
998 +### 2026-05-05
999 +
1000 +- Created the SOW.
1001 +- Loaded query and collector project skills.
1002 +- Confirmed the reported Function is served by `src/collectors/network-viewer.plugin/network-viewer.c`.
1003 +- Reproduced successful token-safe Cloud calls for all target aliases, saving raw responses under `.local/audits/network-connections-topology/`.
1004 +- Summarized response sizes and graph cardinality without storing sensitive values in the SOW.
1005 +- Traced the Agent-side 503 message and deferred response size cap.
1006 +- Confirmed the latest 404 alias was reachable and advertised `topology:network-connections` at probe time, so the browser 404 is not explained by stable Function absence.
1007 +- Traced the 404 error key through the sibling `cloud-charts-service` checkout and confirmed it is a Cloud node-instance routing/selection error.
1008 +- Measured payload-size waste in the largest captured response and confirmed the main waste is structural repetition: per-link `src`/`dst`, `labels`, `metrics`, repeated object keys, and actor `tables`.
1009 +- Built a local lossless detailed-payload prototype at `.local/audits/network-connections-topology/lossless-detailed-prototype.mjs`.
1010 +- Ran the prototype against the largest captured old detailed payload:
1011 + - old detailed size: 134,180,370 bytes.
1012 + - compact detailed size: 40,639,488 bytes.
1013 + - compact ratio: 30.29% of old detailed.
1014 + - actors: 48.
1015 + - links: 77,797.
1016 + - old canonical paths: 4,279,591.
1017 + - reconstructed canonical paths: 4,279,591.
1018 + - actor columns: 48.
1019 + - actor table sections: 14.
1020 + - maximum actor table columns: 4.
1021 + - link columns: 69.
1022 + - old detailed -> compact detailed -> old detailed canonical comparison: pass.
1023 + - compact detailed -> old detailed -> compact detailed canonical comparison: pass.
1024 +- The first generic prototype pass exposed a real validation bug: internal empty-array/empty-object markers collided with real numeric values. The encoder was corrected to use private internal markers before encoding cells. This reinforces the requirement for explicit sentinel tests in production CI.
1025 +- A second table-aware prototype pass split actor tables into owner-indexed table sections. This reduced the compact detailed size from 76,910,648 bytes to 40,639,488 bytes while preserving canonical equality.
1026 +
1027 +### 2026-05-06
1028 +
1029 +- Recorded the expanded schema migration requirements:
1030 + - Function schema update.
1031 + - `topology:network-connections` `aggregated`/`detailed` request mode with `aggregated` default.
1032 + - New schema for `topology:network-connections`, `topology:streaming`, and `topology:snmp`.
1033 + - Cloud frontend support for both schemas, with old-schema frontend aggregation isolated for easy removal.
1034 + - Separate Cloud topology aggregation microservice that consumes detailed payloads and returns requested aggregated views.
1035 +- Continuation pass: corrected the SOW-0012 relationship. SOW-0012 is also current and already owns active `topology:streaming` implementation work in this branch. SOW-0020 can continue with shared topology payload schema/spec work, but must not edit the streaming topology producer until SOW-0012 is closed or explicitly merged into this SOW.
1036 +- User update: the user stated SOW-0012 is done. For SOW-0020 sequencing, this unblocks shared schema work against the streaming topology producer. The SOW-0012 file still physically lives under `.agents/sow/current/` with `Status: in-progress` in this working tree, so closing or moving that SOW remains separate lifecycle work if needed.
1037 +
1038 +### 2026-05-08
1039 +
1040 +- Renumbered this SOW from `SOW-0013` to `SOW-0020` after rebasing onto `upstream/master`. Upstream already contains completed `SOW-0013` and `SOW-0014`; the user reserved `SOW-0015` and later reported that another worktree added more SOWs, so this work uses `SOW-0020`.
1041 +- Recorded the user's footprint decision: minimize detailed payload size first, while preserving per-socket evidence one by one for Cloud-side cross-node matching. Detailed graph links may be aggregated, but socket evidence must remain lossless in a compact detail plane.
1042 +- Recorded the user's payload-budget and matching decisions: paged/chunked socket evidence is phase 2, phase 1 must minimize footprint aggressively and fail explicitly rather than truncate if still over budget, and NAT/LB/proxy matching is out of scope for the current phase.
1043 +
1044 +### 2026-05-09
1045 +
1046 +- Rebased the worktree onto latest `upstream/master` at `79a23ebd9e`. The rebase completed with no conflicts, the autostash reapplied cleanly, and post-rebase checks `git diff --check` plus `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
1047 +- Recorded the user's multi-level topology requirement: the same detailed evidence must support Cloud aggregation at node, container/application, Kubernetes label/workload, process-name, and PID scopes. Current network-viewer evidence supports node/process/PID and network namespace classification, but container and Kubernetes scopes require enrichment that the schema must be ready to carry.
1048 +- Recorded the user's actor drilldown table requirement: topology actor modals must continue to list exact dependencies, but new schema tables should be compact materialized views over shared evidence rather than duplicated object arrays under every actor.
1049 +- Recorded the user's custom actor table requirement: actor-owned tables such as streaming topology `streaming_path` must remain supported and must be distinguished from aggregatable relationship/evidence tables through explicit table role and aggregation-policy metadata.
1050 +- Recorded the user's direction semantics requirement: schema metadata must tell Cloud and UI whether a link type is directed, undirected, or hierarchical, and whether `direction` is flow/dependency meaning or observation metadata that can be ignored for aggregation identity.
1051 +- Recorded the user's refreshable link-telemetry overlay requirement: topology links need compact template references for bandwidth/packets/errors/state overlays so the UI can refresh traffic without recomputing topology, and Cloud can merge overlay refs when aggregating links.
1052 +- Recorded the user's schema-emulation requirement: before freezing the schema and migrating producers/UI, build a topology schema lab with modeled use cases, scale profiles, candidate encoding comparisons, and a prototype aggregator that produces payload-size and correctness evidence.
1053 +- Recorded the user's real-corpus requirement: the schema lab should be able to import read-only local captures of existing topology Function payloads from internal Netdata-owned Kubernetes infrastructure, with raw payloads kept under `.local/` and only sanitized summaries or generated fixtures committed.
1054 +- Recorded the user's Cloud corpus scope correction: only the `Netdata Cloud` space is in scope for topology payload capture; other visible spaces are irrelevant and must be ignored.
1055 +- Captured the scoped Cloud topology corpus using token-safe wrappers. Raw payloads are local-only under `.local/audits/topology-schema-lab/20260509T053114Z/`. Sanitized shape summary:
1056 + - `topology:network-connections`: 12 successful payloads, 0 failures, 597,525,255 total raw bytes, 15,335,804 total gzip bytes, largest raw payload 151,865,348 bytes, largest graph 331 actors / 87,775 links / 87,894 sockets.
1057 + - `topology:streaming`: 14 successful payloads, 0 failures, 1,732,039 total raw bytes, 101,389 total gzip bytes, largest raw payload 1,202,302 bytes, largest graph 120 actors / 121 links.
1058 +- Built and ran a local schema-lab prototype under `.local/audits/topology-schema-lab/scripts/` against the scoped corpus. The lab compared current object-shaped payloads, recursive string dictionaries, columnar lossless encoding, table-split columnar lossless encoding, graph/evidence-split lossless encoding, table-split graph/evidence lossless encoding, and a lossy aggregated-view estimate. Lossless candidates reconstruct the original payload and are checked with deep equality.
1059 +- Schema-lab result for `topology:network-connections` corpus:
1060 + - Current baseline: 597,525,255 raw bytes / 15,335,804 gzip bytes.
1061 + - Recursive string dictionary only: 319,794,282 raw bytes (53.52%) / 12,728,643 gzip bytes (83.00%).
1062 + - Columnar lossless: 190,662,647 raw bytes (31.91%) / 10,004,844 gzip bytes (65.24%).
1063 + - Table-split columnar lossless: 188,163,449 raw bytes (31.49%) / 9,997,988 gzip bytes (65.19%).
1064 + - Graph/evidence split lossless: 171,733,199 raw bytes (28.74%) / 9,907,243 gzip bytes (64.60%).
1065 + - Table-split graph/evidence split lossless: 168,277,837 raw bytes (28.16%) / 9,888,322 gzip bytes (64.48%).
1066 + - Aggregated-view estimate without per-socket evidence: 25,258,107 raw bytes (4.23%) / 1,503,266 gzip bytes (9.80%).
1067 + - Largest lossless table-split graph/evidence payload: 44,996,216 raw bytes for the largest 87,775-link capture.
1068 + - Largest aggregated-view estimate: 6,284,950 raw bytes for the same largest capture class.
1069 +- Additional lossless scope measurement for `topology:network-connections` corpus:
1070 + - Per-node best raw encoding, choosing the smallest verified-lossless candidate per node: 168,277,025 raw bytes / 9,888,011 gzip bytes summed across 12 node responses.
1071 + - Largest per-node best raw response: 44,996,216 raw bytes / 2,748,857 gzip bytes for the largest 87,894-socket capture.
1072 + - Per-node best gzip encoding, choosing the smallest verified-lossless gzip candidate per node: 170,026,315 raw bytes / 9,878,994 gzip bytes summed across 12 node responses.
1073 + - Single combined current JSON-array response: 597,525,268 raw bytes / 15,342,691 gzip bytes.
1074 + - Best measured single combined raw response: `table-split-graph-evidence-shared-dict-by-node-lossless`, 168,807,003 raw bytes / 9,882,897 gzip bytes.
1075 + - Best measured single combined gzip response: `table-split-graph-evidence-global-table-lossless`, 176,133,862 raw bytes / 9,792,662 gzip bytes.
1076 + - Raw combined lossless encoding is not better than independent per-node lossless encoding in this prototype. The best single combined raw response is 529,978 bytes larger than the per-node best raw sum. The best single combined gzip response saves only 86,332 gzip bytes compared with the per-node best gzip sum, at a 6,107,547 raw-byte cost.
1077 +- Follow-up measurement rejected the earlier "best lossless" interpretation:
1078 + - The earlier 44,996,216-byte largest-node result was only the best among generic table transforms tested at that point; it was not proven best.
1079 + - A per-column codec variant that still reconstructs the current old payload exactly reduced the largest 87,894-socket node to 13,672,282 raw bytes / 2,362,223 gzip bytes.
1080 + - Across the 12-node `topology:network-connections` corpus, this exact-old-payload-lossless codec measured 52,398,087 raw bytes / 8,532,821 gzip bytes.
1081 + - Largest-node section split for the exact-old-payload-lossless codec: string dictionary 3,693,814 bytes, evidence table 8,909,152 bytes, actor/table section 1,062,652 bytes, graph table 1,788 bytes.
1082 + - The largest-node evidence table previously had 30,676,393 bytes of plain value arrays across 40 evidence columns; per-column codecs reduced those same evidence columns to 8,908,635 bytes. This proves the 44,996,216-byte result was not a schema lower bound.
1083 + - Largest-node exact-old-payload-lossless record-size split: 13,672,282 total raw bytes over 87,894 sockets, about 155.6 bytes/socket. Of that, the evidence table is 8,909,152 bytes (101.4 bytes/socket), the global string dictionary is 3,693,814 bytes (42.0 bytes/socket), the actor/table section is 1,062,652 bytes (12.1 bytes/socket), and graph/envelope/metadata are negligible.
1084 + - The largest string-dictionary consumers are legacy display/presentation strings: the shared `labels.display_name` / `metrics.display_name` value set has 72,526 unique strings and about 2,688,446 JSON string bytes before dictionary array overhead; remote `port_name` values add about 708,892 JSON string bytes; local/label port-name strings add about 157,331 JSON string bytes. These are not canonical topology identity requirements.
1085 + - The largest evidence-table column costs are legacy display/port-name fields and current snapshot metrics: `labels.display_name` 972,404 bytes, `dst.attributes.port_name` 706,476 bytes, `labels.port_name` 600,121 bytes, `metrics.rtt_ms_max` 578,083 bytes, `dst.attributes.port` 476,926 bytes, `src.attributes.port` 459,193 bytes, `metrics.recv_rtt_ms_max` 453,373 bytes, plus many low-cardinality legacy label columns that each still cost about 2 bytes/socket as JSON index arrays.
1086 + - A separate canonical socket-tuple experiment, not old-payload-lossless, preserved actor graph identity plus per-socket local bind IP/port, remote IP/port, namespace/address-space/family, direction/protocol/state, and ownership edges. It measured 2,015,773 raw bytes / 443,953 gzip bytes for the largest node, or 3,229,251 raw bytes / 897,960 gzip bytes when carrying current RTT/retransmission/socket-count snapshot metrics.
1087 + - The same canonical socket-tuple experiment across all 12 nodes measured 7,864,151 raw bytes / 1,599,786 gzip bytes without current socket metrics, or 11,861,840 raw bytes / 3,039,411 gzip bytes with current socket metrics.
1088 + - The canonical socket-tuple experiment is not yet a schema decision; it is evidence that the final schema should be purpose-built around socket evidence, not around exact reconstruction of the legacy object-shaped response.
1089 +- User correction on production payload vs test reconstruction:
1090 + - Production payload must be optimized for the Cloud aggregator and UI, not for reconstructing the legacy payload.
1091 + - Reconstruction details for old-payload parity are test harness code/fixtures only. They must not be shipped in production payloads.
1092 + - The new production payload must preserve canonical information needed by aggregator/UI; it should not carry legacy presentation/reconstruction paths, old field names, display-string derivations, or redundant data solely to make old JSON byte/object reconstruction easier.
1093 + - Losslessness for schema design should therefore be measured against a canonical information model, with separate test-side projection code proving that the old payload can be derived where compatibility/parity requires it.
1094 +- Production-only canonical socket payload rerun after separating reconstruction from payload:
1095 + - Corpus scale: 323,077 socket evidence rows, 324,177 reported sockets, 1,839 graph links, and 259 ownership links across 12 captured `topology:network-connections` responses.
1096 + - Current legacy corpus size: 597,525,255 raw bytes / 15,335,804 gzip bytes.
1097 + - Production core payload as independent per-node responses: 7,280,783 raw bytes / 1,568,720 gzip bytes, 22.536 raw bytes per socket evidence row.
1098 + - Production core payload as one combined response with shared string dictionary: 7,250,808 raw bytes / 1,556,744 gzip bytes, 22.443 raw bytes per socket evidence row.
1099 + - Production core plus current RTT/retransmission/socket-count metrics as independent per-node responses: 11,278,208 raw bytes / 3,007,786 gzip bytes, 34.909 raw bytes per socket evidence row.
1100 + - Production core plus current metrics as one combined response with shared string dictionary: 11,248,200 raw bytes / 2,992,573 gzip bytes, 34.816 raw bytes per socket evidence row.
1101 + - Largest captured node under production core: 1,996,265 raw bytes / 443,069 gzip bytes for 87,761 socket evidence rows and 52 graph links.
1102 + - Largest captured node under production core plus current metrics: 3,209,721 raw bytes / 896,764 gzip bytes.
1103 + - Production core column costs across the corpus: graph index 689,510 bytes (2.134 bytes/socket row), local IP 651,101 (2.015), local port 1,359,664 (4.208), remote IP 692,081 (2.142), remote port 1,799,512 (5.570), namespace 646,151 (2.000), protocol family 646,152 (2.000), local address space 96, remote address space 646,305 (2.000). The remaining bytes are graph rows, ownership rows, per-node metadata, actors, JSON separators, and the string dictionary.
1104 + - Current metrics add: `rtt_ms_max` 1,794,102 bytes (5.553 bytes/socket row), `recv_rtt_ms_max` 1,541,595 (4.772), retransmissions 661,560 (2.048), and `socket_count` 84 bytes because it is constant per node in this corpus.
1105 +- New documentation and implementation direction:
1106 + - Document the new topology schema in detail, including the full JSON schema/contract, developer documentation, and an AI skill for creating topology producers.
1107 + - Treat the superseded topology schema as removed from Agent/backend contracts and docs. The only temporary compatibility support should be isolated in Cloud frontend code until all supported Agents emit the new schema; that compatibility path is temporary and should be deleted later.
1108 + - The new schema must be generic across topology types, not network sockets only. It must support network-connections, streaming, SNMP/L2, vSphere topology, and future topology producers.
1109 + - Scope backend and frontend implementation changes after documenting the schema.
1110 + - Build a Cloud topology aggregator as a separate Go service/component after the schema contract is documented and implementation scope is clear.
1111 + - The vSphere topology work in the separate PR worktree must be updated in place, but no edits should be made there before telling the user because another agent is working in that directory.
1112 +- Drafted the production topology schema artifacts:
1113 + - Added `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` as the JSON Schema for `netdata.topology.v1` payloads.
1114 + - Added `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` documenting actors, links, evidence tables, actor/custom tables, direction semantics, aggregation policy, telemetry overlays, and producer examples for network-connections, streaming, SNMP/L2, and vSphere.
1115 + - Added `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` scoping backend producer migration, frontend decoder/compatibility migration, and the Go aggregation component.
1116 + - Added the public `docs/netdata-ai/skills/create-topology/` skill and `.agents/skills/create-topology` symlink so future assistants follow this topology contract.
1117 + - Updated Function reference/developer docs and topology query skills to point to the production schema instead of documenting compatibility payload details.
1118 +- Schema-lab result for `topology:streaming` corpus:
1119 + - Current baseline: 1,732,039 raw bytes / 101,389 gzip bytes.
1120 + - Table-split columnar lossless is the best measured lossless candidate: 511,821 raw bytes (29.55%) / 81,484 gzip bytes (80.37%).
1121 + - Graph/evidence split adds no value for streaming where each link already has one evidence row; table-splitting actor custom tables matters more.
1122 +- Prototype aggregator result for `topology:network-connections` corpus:
1123 + - Extracted 323,077 socket evidence rows: 214,546 inbound and 108,531 outbound.
1124 + - Exact reverse-tuple matching found 105,158 matched outbound rows, a 96.89% match ratio among outbound rows in this corpus.
1125 + - After exact matching and unresolved-endpoint aggregation by IP, node-level projection produced 1,093 graph edges from 217,919 dependency evidence rows; process-name projection produced 1,027 graph edges; actor-identity projection produced 1,762 graph edges.
1126 + - The largest node-level edge carried 43,196 evidence rows, proving that graph projection and evidence storage must be separate.
1127 +- User implementation clarification:
1128 + - The Go topology aggregator should be implemented as a Cloud-style microservice similar to `${NETDATA_REPOS_DIR}/cloud-charts-service`, not primarily as an Agent-side helper package.
1129 + - Before coding the aggregator, inspect the Cloud service layout, configuration, HTTP handler, test, and deployment conventions and mirror those patterns.
1130 + - The new microservice name/repository should be `cloud-topology-service`.
1131 + - Hard implementation requirement: `cloud-topology-service` must follow Cloud backend service coding, operational, configuration, testing, deployment, observability, and repository conventions exactly. Implementation must not begin until those conventions are extracted from existing Cloud service repositories with concrete file/line evidence and turned into a compliance checklist.
1132 +- Cloud backend service convention evidence collected so far:
1133 + - `${NETDATA_REPOS_DIR}/cloud-service-builder/README.md:1` describes the skeleton as a microservice based on Netdata standards, but the generated template must be cross-checked against newer services before use.
1134 + - `${NETDATA_REPOS_DIR}/cloud-service-builder/templates/cmd/app/main.go.tmpl:32`, `:41`, `:57`, `:129`, `:155`, and `:192` show the expected generated service-name constant, signal trap, config load, Prometheus/infra server, API server, and instance-name pattern.
1135 + - `${NETDATA_REPOS_DIR}/cloud-service-builder/templates/internal/config/config.go.tmpl:54`, `:63`, and `:136` show the flag groups, log subset, and env parser prefix/set-separator pattern.
1136 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/cmd/service/main.go:62`, `:100`, `:125`, `:177`, `:208`, `:214`, `:245`, `:258`, `:272`, `:280`, `:361`, and `:380` show the current production service-name, logger, config, env prefix, Prometheus, infra server, ADC instrumentation, Pulsar, spaceroom gRPC client, repositories/services, API server, and route registration pattern.
1137 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/http/http.go:86`, `:107`, and `:144` show HTTP route registration, node/room auth wrapping, and the existing Cloud Function execution route pattern.
1138 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/http/auth_middleware.go:64` and `:134` show the Cloud spaceroom authorization pattern for node and room scoped endpoints.
1139 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/http/utils.go:21` and `:57` show the response and error-response helpers, including request id handling and `ckerrors` mapping.
1140 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/internal/model/errors.go:13` shows the service error taxonomy style.
1141 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/internal/service/agent_data.go:565` and `:1233` show direct single-node Function passthrough and concurrent multi-node Function passthrough through ADC after node-instance routing.
1142 + - `${NETDATA_REPOS_DIR}/cloud-agent-data-ctrl-service/internal/config/config.go:21`, `:49`, and `:78` show an ADC-facing service configuration pattern with `ADC_` env prefix.
1143 + - `${NETDATA_REPOS_DIR}/cloud-agent-data-ctrl-service/cmd/service/main.go:138`, `:218`, `:225`, `:235`, and `:242` show errgroup lifecycle, Prometheus, infra server, API server, and OpenTelemetry handler wiring.
1144 + - `${NETDATA_REPOS_DIR}/cloud-agent-data-ctrl-service/transport/http.go:68`, `:95`, `:107`, `:132`, and `:196` show the internal agent API path map, handler construction, middleware, account validation, and internal proxy pattern.
1145 + - `${NETDATA_REPOS_DIR}/cloud-custom-dashboard-service/cmd/customdashboardsvc/main.go:39`, `:71`, `:82`, `:122`, `:136`, `:157`, `:179`, `:185`, and `:192` show another current HTTP service pattern with signal handling, flags, env parser, Prometheus, spaceroom gRPC client, infra/API servers, and OpenTelemetry middleware.
1146 + - `${NETDATA_REPOS_DIR}/cloud-custom-dashboard-service/internal/dashboard/transport_http.go:33` and `:54` show route construction and spaceroom authorization for room-scoped resources.
1147 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/Makefile:4`, `:18`, `:23`, `:28`, `:43`, and `:49` show expected tools, unit, integration, coverage, generate, and lint targets.
1148 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/Dockerfile:2`, `:4`, `:18`, `:29`, and `:42` show the Go base image, service env, build ldflags, Alpine production image, and entrypoint pattern.
1149 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/deployments/helm/values.yaml:2`, `:19`, `:30`, `:38`, `:44`, `:58`, `:64`, `:119`, and `:122` show the microservice anchor, Go memory limit env, probes, resource defaults, service env, security context, Prometheus annotations, service, and ingress-route shape.
1150 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/.github/workflows/main.yml:3`, `:40`, `:103`, `:128`, `:133`, `:137`, `:142`, and `:190` show PR/push triggers, permissions, Go setup, module verification, lint, integration test, coverage, and deployment workflow handoff.
1151 + - `${NETDATA_REPOS_DIR}/cloud-charts-service/.github/CODEOWNERS:1` and `:8` show ownership separation for deployment files and Go service code.
1152 +- Cloud backend service compliance plan:
1153 + - Treat existing Cloud service code as the source of truth. The service-builder template may bootstrap files, but every generated file must be reconciled against the current `cloud-charts-service`, `cloud-agent-data-ctrl-service`, `cloud-custom-dashboard-service`, and `cloud-spaceroom-service` patterns before implementation.
1154 + - Produce a pre-code compliance matrix covering repository layout, Go module/dependency versions, Makefile targets, Dockerfile, GitHub workflows, CODEOWNERS, Helm values, environment variable prefix, configuration flags, logging, signal handling, Prometheus/OpenTelemetry, infra health/readiness, HTTP routing, CORS, authorization, error responses, ADC access, spaceroom access, tests, generated mocks, and deployment annotations.
1155 + - Use `cloud-charts-service` as the primary behavioral reference for Function passthrough, node-instance routing, spaceroom authorization, ADC client instrumentation, request metadata, and Function-specific permissions.
1156 + - Use `cloud-agent-data-ctrl-service` as the primary reference for ADC proxy lifecycle, agent request timeout handling, internal path validation, and service shutdown behavior.
1157 + - Use `cloud-custom-dashboard-service` as the primary reference for compact room-scoped HTTP CRUD-style route construction and spaceroom auth middleware.
1158 + - Use `cloud-service-builder` only for baseline repository shape after checking whether any generated defaults are stale compared to current services.
1159 + - Implementation remains blocked until the compliance matrix identifies the exact file pattern to copy or adapt for each surface and records any gaps that need a user or Cloud-backend decision.
1160 +- Parallel microservice handoff:
1161 + - Created `${NETDATA_REPOS_DIR}/cloud-topology-service/REQUIREMENTS.md` as a standalone handoff contract for a parallel worker to build the Cloud microservice while this SOW continues with schema/frontend/producer tasks.
1162 + - The handoff requires the parallel worker to build the pre-code Cloud service compliance matrix before writing service behavior.
1163 + - The handoff points back to this SOW and the topology schema, developer guide, implementation scope, and `create-topology` skill as the topology contract sources.
1164 + - Reviewed the parallel worker's `${NETDATA_REPOS_DIR}/cloud-topology-service/QUESTIONS1.md` and answered it in `${NETDATA_REPOS_DIR}/cloud-topology-service/ANSWERS1.md`, setting phase-1 defaults for API ownership, route shape, new-schema-only behavior, source scope, partial/error semantics, caching, fixtures, optional metrics, validation location, deployment ownership handling, and compliance matrix location.
1165 + - Kept three true external decisions open in the answer handoff: final service ownership/CODEOWNERS, environment-specific Helm values/deployment targets, and the exact Cloud-approved node-instance routing implementation strategy.
1166 + - User clarified that these three open items are not user decisions; they should be handed to Cloud backend and DevOps once the service is otherwise ready for operational integration. Updated the microservice requirements and answers handoff so the parallel worker proceeds with compliance, schema/codec, aggregation, tests, HTTP scaffolding, and isolated fetcher interfaces without inventing ownership, environment values, or node-routing strategy.
1167 + - User clarified that the phase-1 microservice MVP must support all topology kinds covered by the schema contract. It is not acceptable for the UI to use the service for only some topologies while bypassing it for others. Updated the microservice requirements, answers handoff, and topology implementation scope so `network-connections` remains the high-cardinality benchmark but not the MVP boundary.
1168 + - Aligned the local topology implementation scope, topology schema spec, and `create-topology` skill with the all-topology MVP rule. Replaced stale Cloud aggregator open questions with resolved phase-1 defaults and Cloud backend/DevOps integration gates.
1169 + - Added a current migration inventory to `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`, covering network-viewer, streaming, SNMP/L2, vSphere, and Cloud frontend surfaces with concrete file/line evidence and target migration behavior.
1170 +- Parallel Cloud frontend handoff:
1171 + - Created sibling `cloud-frontend/TODO-topology-schema-new.md` as a standalone handoff contract for a parallel worker to implement frontend support for `netdata.topology.v1`.
1172 + - The handoff records current frontend file/line evidence, requires old-schema support and the old link dedupper to be isolated under a removable legacy adapter, and scopes the new v1 path around compact table decoding, actor/link/evidence indexes, actor-detail vs relationship table separation, explicit direction semantics, overlay refs, worker decoding, and all-topology fixtures.
1173 + - The handoff intentionally keeps Cloud service integration separate from the v1 decoder so per-node Function responses can migrate by schema version before the service route is ready.
1174 + - Removed a later appended open-questions section from the handoff. The useful items were resolved into implementation defaults and concrete decoder clarifications so the frontend worker is not blocked on questions that can be answered from the schema, current frontend code, or the Cloud service handoff.
1175 +- Agent-side validation and fixture rail:
1176 + - Extended `src/go/tools/functions-validation/validate` so the existing Function validation CLI now performs topology v1 semantic checks after JSON Schema validation. The checks validate compact table decoded lengths, inline dictionary indexes, global dictionary references, actor/link references, evidence references when a detail table declares `source_evidence`, and array column values.
1177 + - Added schema-level topology v1 fixtures under `src/go/tools/functions-validation/fixtures/topology-v1/` for `network-connections`, `streaming`, `snmp-l2`, and `vsphere`.
1178 + - The fixtures cover directed socket evidence, streaming actor-detail `stream_path`, SNMP/L2 unordered observation direction plus actor inventory and overlay refs, and vSphere hierarchy/dependency links plus actor detail.
1179 + - Updated `src/go/tools/functions-validation/README.md` with a topology v1 validation command and documented the additional compact-table semantic checks.
1180 +- Producer helper rail:
1181 + - Added `src/go/pkg/topology/v1` as the Go producer-side model and compact-table helper package for `netdata.topology.v1` payloads.
1182 + - The helper provides response/data/type/table structs, compact-table constructors, row-count validation, parallel column/value validation, dictionary-index validation, and a shared decoded-payload semantic validator.
1183 + - Refactored `src/go/tools/functions-validation/validate` to call the shared topology v1 semantic validator instead of keeping topology-specific validation private to the CLI.
1184 + - Updated the topology developer guide, `create-topology` skill, and `project-writing-collectors` skill so future Go topology producers use the helper instead of hand-building compact-table JSON.
1185 +- SNMP/L2 producer migration investigation:
1186 + - Investigated `topology:snmp` as the first Go producer candidate for migration to the helper.
1187 + - Found a schema gap before producer migration: existing SNMP actor port details may include nested custom data such as `vlans` and `neighbors`, which cannot be represented by scalar-only compact table cells without losing information or flattening producer-specific structure.
1188 + - Updated `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` to add a `json` column type for nested custom detail cells.
1189 + - Updated the topology developer guide and `create-topology` skill to restrict `json` columns to actor/custom detail data and warn against using them for high-cardinality relationship evidence.
1190 + - Extended the Go helper schema round-trip test with a nested `json` actor detail column so this requirement stays covered by tests.
1191 +- SNMP/L2 producer migration implementation:
1192 + - Updated `src/go/plugin/go.d/collector/snmp_topology/func_topology_handler.go` so `topology:snmp` returns `netdata.topology.v1` data through a dedicated adapter before sending the Function response.
1193 + - Added `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` to map the current SNMP topology snapshot into compact actor, link, L2 observation evidence, actor metadata, and actor-detail tables using `src/go/pkg/topology/v1`.
1194 + - Preserved nested SNMP custom actor detail cells such as `neighbors` and `vlans` with `json` columns while keeping graph links and L2 evidence typed.
1195 + - Updated SNMP topology Function tests to validate produced payloads against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` and the shared topology v1 semantic validator.
1196 + - Updated the topology implementation scope and spec to record that SNMP now emits v1 through an adapter, while interface metric overlay-template/ref migration remains a refinement.
1197 +- Streaming producer migration implementation:
1198 + - Replaced the superseded object-shaped `topology:streaming` payload in `src/web/api/functions/function-topology-streaming.c` with a direct `netdata.topology.v1` emitter.
1199 + - Preserved streaming agents as compact actor rows, directed streaming/virtual/stale graph links, link evidence rows, and actor/modal tables for `stream_path`, retention, inbound, and outbound data.
1200 + - Classified `stream_path` and retention as actor-detail tables and inbound/outbound rows as relationship summaries, so actor-owned custom data is not conflated with relationship evidence.
1201 + - Kept stale stream-path hops as signed values and split streaming, virtual, and stale link evidence into separate evidence type ids so link-type metadata and evidence metadata agree.
1202 + - Used a local C compact-table emitter for this low-cardinality producer. The shared C helper decision remains tied to the `topology:network-connections` migration, where high-cardinality socket evidence will determine the helper API.
1203 + - Updated the topology implementation scope and topology schema spec to record that streaming now emits v1 directly.
1204 +- Network-viewer producer migration implementation:
1205 + - Replaced the superseded object-shaped `topology:network-connections` payload in `src/collectors/network-viewer.plugin/network-viewer.c` with a direct `netdata.topology.v1` emitter.
1206 + - Added `aggregated` / `mode:aggregated` and `detailed` / `mode:detailed` request handling, with aggregated as the default mode.
1207 + - Preserved process grouping options and current socket filters while moving graph output to compact actor and graph-link tables.
1208 + - Detailed mode emits socket relationship evidence as a shared compact table for exact tuple matching and drilldowns; aggregated mode omits the evidence table.
1209 + - Removed superseded topology presentation emission and actor-nested socket tables from the Agent producer so the old schema is no longer present in this code path.
1210 + - Added automatic string-column encoding for link/evidence columns: the writer uses inline dictionaries only when estimated raw JSON size is smaller than plain values.
1211 + - Updated the topology implementation scope and topology schema spec to record that network-viewer now emits v1 directly.
1212 +- Orphan endpoint actor repair:
1213 + - Investigated the user's reproduced floating endpoint actor with a non-zero
1214 + socket count and no incident graph links.
1215 + - Root cause: `local_sockets_cb_to_topology()` can create a remote endpoint
1216 + actor while scanning sockets before all local IPs are known, but
1217 + `topology_v1_collect_links()` resolves socket destinations later using the
1218 + final local-IP set. If the same IP is learned as local later in the scan,
1219 + link resolution treats it as self while the earlier remote endpoint actor
1220 + remains in the actor table.
1221 + - Fixed `topology_v1_collect_actors()` so endpoint actor emission rechecks
1222 + `topology_ip_belongs_to_self()` against the final local-IP set. This makes
1223 + endpoint actor emission and link destination resolution use the same self
1224 + classification.
1225 +
1226 +## Validation
1227 +
1228 +Acceptance criteria evidence:
1229 +
1230 +- 503 path: not deterministically reproduced through the token-safe path; evidence points to oversized intermittent/path-dependent Function failure. Supporting evidence: 110-134 MB successful bodies, 100 MiB parser cap, and exact 503 message source.
1231 +- 404 path: latest browser 404 was not reproduced through the token-safe path; function discovery and room inventory showed the node alias reachable and exposing `topology:network-connections` at probe time. This supports a Cloud node-instance routing/state race or stale browser/request context rather than a stable missing Function.
1232 +- Orphan endpoint path: reproduced by the user after the initial completed
1233 + captures. Source analysis found an order-dependent actor/link classification
1234 + mismatch in the producer; the targeted repair now aligns endpoint actor
1235 + emission with final link destination classification.
1236 +
1237 +Tests or equivalent validation:
1238 +
1239 +- Schema-lab prototype transforms under `.local/audits/topology-schema-lab/scripts/` were run against the scoped Cloud corpus and produced the sanitized size/correctness measurements recorded in the execution log.
1240 +- `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
1241 +- A minimal `netdata.topology.v1` sample payload was validated with Ajv 2020 against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
1242 +- `git diff --check` passed.
1243 +- Stale-shape scan over the new topology docs/skills found no references to compatibility object fields such as `src_actor_id`, `dst_actor_id`, `data.actors[]`, `actors[]`, or `links[]`. The remaining `src_actor_id`/`dst_actor_id` references are in `src/plugins.d/FUNCTION_UI_SCHEMA.json`, which is the still-deployed Function UI schema and will be handled during implementation migration.
1244 +- Relative link target checks passed for the new query-topology references and the `.agents/skills/create-topology` symlink.
1245 +- Sensitive-path scan over the touched topology docs, skill, spec, and SOW found no per-user filesystem paths.
1246 +- Sensitive-path scan over the sibling Cloud frontend handoff found no per-user filesystem paths, personal names, or raw production identifiers.
1247 +- `jq empty` passed for all `src/go/tools/functions-validation/fixtures/topology-v1/*.json` fixtures.
1248 +- `go test ./tools/functions-validation/validate` passed from `src/go`.
1249 +- The validation CLI passed for all topology v1 fixtures with `--schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
1250 +- `go test ./pkg/topology/v1 ./tools/functions-validation/validate` passed from `src/go`.
1251 +- `go test ./pkg/topology/... ./tools/functions-validation/validate` passed from `src/go`.
1252 +- The topology v1 helper JSON round-trip test marshaled a typed `netdata.topology.v1` response with a nested `json` actor detail column, validated it against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`, and then ran the shared semantic validator.
1253 +- `go test ./plugin/go.d/collector/snmp_topology ./pkg/topology/v1 ./tools/functions-validation/validate` passed from `src/go`.
1254 +- `go test ./pkg/topology/... ./plugin/go.d/collector/snmp_topology ./tools/functions-validation/validate` passed from `src/go`.
1255 +- SNMP topology Function tests now validate actual handler responses against both the production topology JSON Schema and the shared semantic validator.
1256 +- `TestSNMPTopologyToV1_PreservesActorCustomTables` validates that nested SNMP actor detail cells are preserved as `json` columns without weakening the typed graph/evidence path.
1257 +- Local CMake validation build configured with `cmake -S . -B .local/build-topology -DCMAKE_BUILD_TYPE=Debug -DENABLE_PLUGIN_XENSTAT=OFF -DENABLE_PLUGIN_DEBUGFS=OFF`. `ENABLE_PLUGIN_XENSTAT=OFF` avoids a missing local `xenstat` package, and `ENABLE_PLUGIN_DEBUGFS=OFF` avoids an uninitialized optional libsensors subtree in this local worktree.
1258 +- Initialized only the declared `src/aclk/aclk-schemas` submodule so the normal Agent target can generate protobuf sources for the local validation build.
1259 +- `cmake --build .local/build-topology --target netdata -j2` passed after the streaming migration, including compilation of `src/web/api/functions/function-topology-streaming.c` and linking the `netdata` executable.
1260 +- After tightening signed hops and evidence type ids, the incremental `cmake --build .local/build-topology --target netdata -j2` passed again, rebuilding `function-topology-streaming.c` and linking `netdata`.
1261 +- `git diff --check -- src/web/api/functions/function-topology-streaming.c` passed.
1262 +- Stale-shape scan over `src/web/api/functions/function-topology-streaming.c` found no superseded presentation helpers, `schema_version: "2.0"`, or old `src_actor_id` / `dst_actor_id` payload fields.
1263 +- `git diff --check -- src/collectors/network-viewer.plugin/network-viewer.c` passed after the network-viewer migration.
1264 +- `cmake --build .local/build-topology --target network-viewer.plugin -j2` passed after the final network-viewer migration, rebuilding and linking `network-viewer.plugin`.
1265 +- Stale-shape scan over `src/collectors/network-viewer.plugin/network-viewer.c` found no superseded topology presentation helpers, `schema_version: "2.0"`, or old `l7` topology layer values.
1266 +- Local `network-viewer.plugin debug` aggregated sample was captured under `.local/audits/topology-network-viewer-v1/` and validated against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` with the Function validator. Sanitized shape: 74 actors, 86 graph links, 0 socket-evidence rows, 20,132 raw bytes.
1267 +- Local `network-viewer.plugin debug` detailed sample was captured under `.local/audits/topology-network-viewer-v1/` and validated against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` with the Function validator. Sanitized shape: 91 actors, 102 graph links, 173 socket-evidence rows, 39,865 raw bytes.
1268 +- Local network-viewer sample codec check confirmed graph-link string columns use `dict` where beneficial and detailed socket-evidence string columns use a mix of `dict` and `values` based on estimated raw JSON size.
1269 +- Final validation pass after the network-viewer migration: `cmake --build .local/build-topology --target netdata -j2` passed.
1270 +- Final validation pass after the network-viewer migration: `go test ./pkg/topology/... ./plugin/go.d/collector/snmp_topology ./tools/functions-validation/validate` passed from `src/go`.
1271 +- Final validation pass after the network-viewer migration: the Function validator passed for all `src/go/tools/functions-validation/fixtures/topology-v1/*.json` fixtures and for the local aggregated/detailed network-viewer samples.
1272 +- Re-ran `go test ./pkg/topology/... ./plugin/go.d/collector/snmp_topology ./tools/functions-validation/validate` from `src/go`; it passed.
1273 +- Re-ran the Function validator over all `src/go/tools/functions-validation/fixtures/topology-v1/*.json` fixtures with `--schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`; all fixtures passed.
1274 +- Sensitive-data scan over the new topology fixtures and validator changes found no personal paths, personal names, tokens, cookies, API keys, UUID-shaped identifiers, or pushback wording.
1275 +- After the orphan endpoint actor repair:
1276 + - C syntax validation passed for `src/collectors/network-viewer.plugin/network-viewer.c` using the compile command from `build/compile_commands.json` with `-fsyntax-only`.
1277 + - `cmake --build .local/build-topology --target network-viewer.plugin -j2` passed.
1278 + - A local `network-viewer.plugin debug` topology sample was captured under
1279 + `.local/audits/topology-network-viewer-v1/` and validated with
1280 + `go run ./tools/functions-validation/validate --schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
1281 + - The local sample had zero endpoint actors without incident links after
1282 + compact-table decoding.
1283 + - The specific reproduced self-classified endpoint address was not emitted
1284 + as an endpoint actor in the local sample after the repair.
1285 +
1286 +Real-use evidence:
1287 +
1288 +- Cloud API calls through the token-safe path returned HTTP 200 for all checked aliases during investigation. Browser evidence still shows real 503 and 404 failures, so the failures are intermittent/path-dependent.
1289 +
1290 +Reviewer findings:
1291 +
1292 +- No unresolved reviewer findings remain for this SOW. The final orphan-endpoint repair was narrow and was validated with source-path analysis, C syntax validation, a local plugin build, a runtime Function sample, schema validation, and a same-failure orphan scan. The broader topology presentation review was split into SOW-0021 and completed there with multi-reviewer findings and follow-up mapping.
1293 +
1294 +Same-failure scan:
1295 +
1296 +- Completed captured payloads were scanned for actors with no incident links and links referencing missing actors; no orphan actors or broken link references were found in those completed responses.
1297 +
1298 +Sensitive data gate:
1299 +
1300 +- Durable artifacts created so far use aliases and do not include browser cookies, raw tokens, raw node UUIDs, raw machine GUIDs, raw process details, or raw topology payloads.
1301 +
1302 +Artifact maintenance gate:
1303 +
1304 +- AGENTS.md: updated to register the public `create-topology` skill and symlink.
1305 +- Runtime project skills: `.agents/skills/project-writing-collectors/SKILL.md` updated to point topology producers at the new topology schema, guide, and implementation scope.
1306 +- Specs: `.agents/sow/specs/topology-function-schema.md` added as durable project memory for the production topology Function contract and updated with `json` detail-column, SNMP adapter migration notes, direct streaming v1 emission, signed stale-hop handling, and direct network-viewer v1 emission.
1307 +- End-user/operator docs: Function UI developer/reference docs updated to reference the topology schema; no external Learn docs were changed in this pass.
1308 +- End-user/operator skills: `docs/netdata-ai/skills/create-topology/`, `docs/netdata-ai/skills/query-netdata-cloud/`, and `docs/netdata-ai/skills/query-netdata-agents/` updated for topology schema/query guidance.
1309 +- SOW lifecycle: this SOW is marked `completed` and is moved to `.agents/sow/done/` with the implementation commit.
1310 +
1311 +Specs update:
1312 +
1313 +- `.agents/sow/specs/topology-function-schema.md` added and updated with `json` detail-column, SNMP adapter migration notes, direct streaming v1 migration notes, and direct network-viewer v1 migration notes.
1314 +
1315 +Project skills update:
1316 +
1317 +- `.agents/skills/project-writing-collectors/SKILL.md` updated.
1318 +
1319 +End-user/operator docs update:
1320 +
1321 +- `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` and `src/plugins.d/FUNCTION_UI_REFERENCE.md` updated.
1322 +
1323 +End-user/operator skills update:
1324 +
1325 +- `docs/netdata-ai/skills/create-topology/`, `docs/netdata-ai/skills/query-netdata-cloud/`, and `docs/netdata-ai/skills/query-netdata-agents/` updated.
1326 +
1327 +Lessons:
1328 +
1329 +- Actor emission and link emission must use the same final self/remote classification inputs. The network-viewer bug came from discovering remote endpoint actors during socket scanning before the final local-IP set was complete, while link emission later used the completed local-IP set.
1330 +- Compact topology validation must include graph invariants, not only schema validity. A payload can be valid JSON and still be unusable if it contains actors that no emitted link can reach.
1331 +- Large topology payload work needs both size tests and semantic round-trip tests. Size reduction alone is not enough if actor/link identity or drilldown evidence is weakened.
1332 +
1333 +Follow-up mapping:
1334 +
1335 +- SOW-0021 completed the topology presentation contract and backend producer presentation updates.
1336 +- SOW-0022 tracks actor modal and table composition.
1337 +- SOW-0023 tracks cross-payload actor identity, reconciliation, and matching strategies.
1338 +- SOW-0024 tracks vSphere topology migration to `netdata.topology.v1`.
1339 +- The Cloud topology service and Cloud frontend implementation handoffs are owned by their respective repositories and workers; this netdata commit does not include those repositories.
1340 +
1341 +## Outcome
1342 +
1343 +`topology:network-connections`, `topology:streaming`, and `topology:snmp` now have a compact `netdata.topology.v1` schema path, shared validation, producer guidance, and realistic fixtures. The network-connections orphan endpoint repair aligns remote actor emission with final link destination classification, so self-classified endpoints are no longer emitted as floating graph actors.
1344 +
1345 +## Lessons Extracted
1346 +
1347 +See `## Validation` lessons above. The durable schema, docs, and `create-topology` skill were updated so future topology producers preserve compactness, graph invariants, evidence, and presentation separation.
1348 +
1349 +## Followup
1350 +
1351 +- SOW-0021: topology presentation contract.
1352 +- SOW-0022: actor modal/table composition.
1353 +- SOW-0023: cross-payload actor matching and reconciliation.
1354 +- SOW-0024: vSphere topology v1 migration.
1355 +
1356 +## Regression Log
1357 +
1358 +None yet.
.agents/sow/done/SOW-0021-20260509-topology-presentation-contract.md new
+947
@@ -0,0 +1,947 @@
1 +# SOW-0021 - Topology presentation contract
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +Sub-state: completed after inventory, schema/docs/skill updates, producer presentation updates, handoff documents, review rounds, and runtime/user validation of the repaired presentation behavior.
8 +
9 +## Requirements
10 +
11 +### Purpose
12 +
13 +Restore polished, backend-controlled topology visualization while preserving the compact `netdata.topology.v1` payload model. The schema must let topology producers compose visual behavior from UI-supported tokens and profiles, without making the frontend know domain-specific concepts such as parent, child, router, client, server, process, switch, or endpoint.
14 +
15 +### User Request
16 +
17 +The user reported that the new compact topology schema lost important visual behavior from the prior topology payloads:
18 +
19 +- actor fill color, border color, annotation ring color, SVG/icon choice, and popover synthesis;
20 +- link color, line shape, curve direction, width, arrow direction, and popover synthesis;
21 +- streaming highlight-path behavior;
22 +- legend and color coding;
23 +- actor bullets/ports/sockets;
24 +- actor display-name synthesis, including a current failure where one actor label becomes a long concatenation of many MAC addresses;
25 +- Cloud aggregator behavior for conflicting presentation/type definitions when multiple topology payloads are merged;
26 +- Cloud aggregator matching strategies across payloads, for example socket endpoint matching, SNMP port/MAC matching, streaming machine identity matching, and future domain-specific matching.
27 +
28 +The user explicitly split the work:
29 +
30 +- SOW-0021: fix topology presentation.
31 +- SOW-0022: fix table and actor-modal composition.
32 +
33 +### Assistant Understanding
34 +
35 +Facts:
36 +
37 +- `netdata.topology.v1` reduced topology payload size by removing repeated row/object data and by separating graph links from evidence.
38 +- The old topology schema contained presentation metadata for actor types, link types, port types, legends, actor click behavior, actor modal tabs, and table hints.
39 +- The new topology schema currently has actor/link/evidence/table/overlay type registries, but no equivalent visual presentation contract.
40 +- The UI must remain domain-agnostic. It should provide rendering tokens/enums and rendering primitives, while the backend payload chooses how actor/link/table types use them.
41 +- Raw topology payloads and user examples may include hostnames, MAC addresses, IP addresses, interface aliases, private infrastructure details, usernames, masked passwords, and other sensitive or identifying data. Durable artifacts must only contain sanitized summaries.
42 +
43 +Inferences:
44 +
45 +- The current schema is strong for compact aggregation facts but weak for visual semantics.
46 +- Hardcoding domain names and producer-specific behavior in the UI would make the schema generic only on paper.
47 +- Reintroducing the old presentation object verbatim would preserve behavior quickly but would also preserve old schema ambiguity and bloat.
48 +- A compact, enum/token-based presentation plane attached to type registries is the likely correct replacement.
49 +
50 +Unknowns:
51 +
52 +- The full set of UI rendering tokens/enums available or required in `cloud-frontend`.
53 +- The exact conflict policy Cloud aggregation should use when payloads define the same actor/link/table type id with incompatible presentation profiles.
54 +- The complete inventory of old schema fields and frontend consumers that must be preserved, replaced, or deliberately dropped.
55 +
56 +### Acceptance Criteria
57 +
58 +- Inventory all old topology presentation fields, producer emissions, and frontend consumers, including legend, colors, icons, ports/bullets, highlight behavior, popovers, actor labels, link styles, and actor modal/table references.
59 +- Classify every old presentation capability as: preserve in v1, replace with a compact token/profile, move to SOW-0022, derive in UI from explicit backend tokens, or intentionally drop with evidence.
60 +- Run the requested external reviewers after the inventory: Claude, Codex, GLM, MiMo, Kimi, MiniMax, and Qwen. Prompts must be shown before execution, must be read-only, must include the SOW filename, and must ask reviewers to find missing presentation semantics, unwanted side effects, security/privacy issues, and aggregator conflict risks.
61 +- Extend `FUNCTION_TOPOLOGY_SCHEMA.json` with compact presentation/profile contracts that are enum/token based, not raw CSS/layout.
62 +- Extend `FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `.agents/sow/specs/topology-function-schema.md`, and `docs/netdata-ai/skills/create-topology/SKILL.md` so future topology producers know how to define presentation profiles.
63 +- Define Cloud aggregation merge/conflict policy for presentation profiles. Cross-payload actor reconciliation is tracked by SOW-0023.
64 +- Create worker handoff documentation for the Cloud frontend and Cloud topology aggregator.
65 +- Update backend producers enough to emit the new presentation contract for existing topology producers covered by this SOW.
66 +- Validate schema, docs, skill, and backend producer output with targeted tests or equivalent checks.
67 +
68 +## Analysis
69 +
70 +Sources checked:
71 +
72 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json`
73 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
74 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
75 +- `.agents/sow/specs/topology-function-schema.md`
76 +- `docs/netdata-ai/skills/create-topology/SKILL.md`
77 +- `.agents/skills/project-writing-collectors/SKILL.md`
78 +- `.agents/sow/current/SOW-0020-20260505-network-connections-topology-cloud-errors.md`
79 +
80 +Current state:
81 +
82 +- The old schema had `topology_presentation_actor_type` with `label`, `color_slot`, `opacity`, `border`, `role`, `size_by_links`, `show_port_bullets`, `icon_svg`, summary fields, tables, and modal tabs.
83 +- The old schema had `topology_presentation_link_type` with `label`, `color_slot`, `opacity`, `width`, and `dash`.
84 +- The old schema had `topology_presentation_port_type`, `topology_presentation_legend`, and `actor_click_behavior` with `highlight_connections` and `highlight_path`.
85 +- The new schema currently defines actors, links, evidence, detail tables, overlays, and aggregation metadata, but it has no presentation/profile plane.
86 +- The topology spec currently says the new network-connections producer no longer emits superseded presentation metadata. This SOW supersedes that statement by replacing old presentation metadata with compact v1 presentation profiles.
87 +
88 +### Inventory - 2026-05-09
89 +
90 +Old schema and shared Go model:
91 +
92 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json:277-324` defines actor, link, and port presentation types. Required actor/link/port fields include labels and `color_slot`; optional fields include opacity, border, role, size-by-links, port bullets, icon SVG, link width, and dashed links.
93 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json:349-363` defines presentation table metadata, including `source: "data" | "links"`, `bullet_source`, and display columns.
94 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json:376-424` defines legend entries, legend sections, and `actor_click_behavior: "highlight_connections" | "highlight_path"`.
95 +- `src/go/pkg/topology/types.go:57-136` mirrors the old presentation model in Go and states that it tells the UI how to render topology. This is useful as the old inventory source, but not the final v1 shape.
96 +- `src/go/pkg/topology/types.go:114-136` includes `port_fields` as Go-only presentation metadata. This is not represented in `FUNCTION_UI_SCHEMA.json`, but the UI consumes it for port bullet tooltips, so it is part of the real contract.
97 +
98 +Old producer emissions:
99 +
100 +- Network-connections old producer emitted:
101 + - actor profiles for `self`, `process`, and `endpoint`;
102 + - `self` and `process` used `size_by_links`;
103 + - `process` used `show_port_bullets`;
104 + - process socket table used `bullet_source`;
105 + - socket and ownership link types had color/width/dash settings;
106 + - port type `topology`;
107 + - actor/link/port legend;
108 + - `actor_click_behavior: "highlight_connections"`.
109 + Evidence: `git show HEAD:src/collectors/network-viewer.plugin/network-viewer.c`, lines 1734-2119 in the checked pre-v1 version.
110 +- Streaming old producer emitted:
111 + - actor profiles for `parent`, `child`, `vnode`, and `stale`;
112 + - parent used `show_port_bullets`, child/vnode/stale disabled it;
113 + - link profiles for `streaming`, `virtual`, and `stale`;
114 + - port profiles for `streaming`, `virtual`, and `stale`;
115 + - actor/link/port legend;
116 + - `actor_click_behavior: "highlight_path"`;
117 + - stream-path/retention/inbound/outbound table display metadata.
118 + Evidence: `git show HEAD:src/web/api/functions/function-topology-streaming.c`, lines 302-414 and 738-936 in the checked pre-v1 version.
119 +- SNMP old producer still has explicit presentation code:
120 + - device-like actor types map labels and color slots for router/switch/firewall/access point/server/storage/load balancer/printer/phone/UPS/camera;
121 + - device actor profiles use border, `size_by_links`, and `show_port_bullets`;
122 + - segment/endpoint profiles use their own colors/roles;
123 + - link types include LLDP/CDP/bridge/FDB/STP/ARP/SNMP/probable;
124 + - port fields and port type profiles drive port bullet popovers and legend.
125 + Evidence: `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation_types.go:7-134`.
126 +- SNMP emits `port_fields` with labels for `type`, `role`, `status`, `mode`, and `sources`, and the Function config attaches the presentation with `WithPresentation()`.
127 + Evidence: `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation_types.go:74-82` and `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation.go:16-50`.
128 +- SNMP old producer also defines curated summary fields and table columns for device, segment, endpoint, ports, and links.
129 + Evidence: `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation_schema.go:7-96`.
130 +- The vSphere worktree also has an old-schema topology producer that must be migrated after coordination with the worker in that directory. It defines actor profiles for datacenters, clusters, hosts, VMs, datastores, networks, datastore clusters, and resource pools; link profiles for `contains`, `connects`, and `runs`; a legend; and `actor_click_behavior: "highlight_connections"`.
131 + Evidence from the vSphere worktree: `src/go/plugin/go.d/collector/vsphere/func_topology_presentation.go:7-83`.
132 +- The vSphere producer attaches the old presentation to the Function method with `WithPresentation()` and emits inventory actors/links using the old topology package, so it is a real migration consumer, not only dead presentation code.
133 + Evidence from the vSphere worktree: `src/go/plugin/go.d/collector/vsphere/func_topology.go:33-42` and `src/go/plugin/go.d/collector/vsphere/func_topology.go:74-245`.
134 +
135 +Current v1 schema and producers:
136 +
137 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:220-243` has a type registry for actor/link/evidence/table/overlay/aggregation types, but no presentation/profile registry.
138 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:304-399` defines actor and link semantics for identity, layer, direction, and aggregation, but no labels, colors, icons, line styles, bullets, legend, or highlight behavior.
139 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:434-523` defines evidence and table type roles/columns/aggregation, but no modal or table display composition.
140 +- `src/go/pkg/topology/v1/types.go:72-116` mirrors that same v1 gap in producer-side Go types.
141 +- Current v1 network-connections emits a `display_name` actor column and socket evidence columns, but no presentation profile.
142 + Evidence: `src/collectors/network-viewer.plugin/network-viewer.c:2130-2172`.
143 +- Current v1 streaming emits display names, machine GUIDs, link/evidence/table types, and stream-path actor tables, but no presentation profile or highlight-path contract.
144 + Evidence: `src/web/api/functions/function-topology-streaming.c:1043-1272`.
145 +- Current v1 SNMP adapter emits actor metadata as compact rows and JSON table data, but no v1 presentation profile.
146 + Evidence: `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:386-407` and `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:451-483`.
147 +
148 +Current cloud-frontend behavior:
149 +
150 +- Legacy normalizer still has custom display-name synthesis from attributes, labels, and match fields; it avoids using raw match arrays as the first display choice.
151 + Evidence from the cloud-frontend worktree: `src/domains/functions/topology/legacy/normalizeLegacyTopology.js:96-121`.
152 +- V1 actor normalizer has a weaker `deriveLabel()` fallback. If `display_name`, name, hostname, address, or id are absent, it can fall back to actor type or generated identity; arrays are joined with commas by `safeString()`. This is consistent with the user-observed long aggregated MAC actor label class.
153 + Evidence from the cloud-frontend worktree: `src/domains/functions/topology/v1/buildActors.js:9-18` and `src/domains/functions/topology/v1/buildActors.js:46-56`.
154 +- V1 actor rows and nodes expose raw decoded row values directly as attributes.
155 + Evidence from the cloud-frontend worktree: `src/domains/functions/topology/v1/buildActors.js:106-129`.
156 +- The force graph still consumes old presentation keys for click behavior, port bullets, actor visuals, icon SVG, port colors, and legend.
157 + Evidence from the cloud-frontend worktree: `src/domains/functions/components/graph/forceGraph.js:601-645`, `src/domains/functions/components/graph/forceGraph.js:711-754`, and `src/domains/functions/components/graph/forceGraph.js:756-818`.
158 +- The graph legend is entirely driven by `presentation.legend` plus actor/link/port profile maps.
159 + Evidence from the cloud-frontend worktree: `src/domains/functions/components/graph/graphLegend.js:41-104`.
160 +- Color slots are already UI-owned tokens, not backend hex colors. The backend currently chooses slot names and the UI resolves them to theme colors/widths/opacities.
161 + Evidence from the cloud-frontend worktree: `src/domains/functions/topology/colorSlots.js:9-65`.
162 +- The current frontend color slot vocabulary is `primary`, `secondary`, `accent`, `self`, `neutral`, `muted`, `dim`, `derived`, `info`, `structural`, and `warning`. Old network-connections, streaming, and SNMP use these tokens, but the vSphere worktree uses hue names such as `blue`, `green`, `orange`, `purple`, `cyan`, `yellow`, `teal`, and `gray`, which are not present in the current frontend slot table.
163 + Evidence from the cloud-frontend worktree: `src/domains/functions/topology/colorSlots.js:9-65`. Evidence from the vSphere worktree: `src/go/plugin/go.d/collector/vsphere/func_topology_presentation.go:10-23`.
164 +- The canvas currently draws straight links only. It supports line color, opacity, width, and dash via presentation link types, but it has no generic curve/arrow enum yet.
165 + Evidence from the cloud-frontend worktree: `src/domains/functions/components/graph/forceGraphCanvas.js:63-82` and `src/domains/functions/components/graph/forceGraphCanvas.js:132-205`.
166 +- The current frontend icon token vocabulary is closed and UI-owned: `router`, `switch`, `firewall`, `access_point`, `server`, `storage`, `load_balancer`, `printer`, `phone`, `ups`, `camera`, `process`, `agent`, `netdata-agent`, `parent`, `remote-endpoint`, `local-endpoint`, `segment`, `self`, `ip`, `cloud`, `container`, `vm`, `database`, and `service`.
167 + Evidence from the cloud-frontend worktree: `src/domains/functions/topology/icons.js:5-260`.
168 +- Current v1 actor modal bypasses legacy presentation table tabs and shows a single `V1 data` tab.
169 + Evidence from the cloud-frontend worktree: `src/domains/functions/components/topology/actorModal/index.js:97-107`, `src/domains/functions/components/topology/actorModal/index.js:152-162`, and `src/domains/functions/components/topology/actorModal/index.js:319-364`.
170 +- Current v1 actor modal renderer stringifies objects and overlays directly, which explains raw JSON leaking into final UI.
171 + Evidence from the cloud-frontend worktree: `src/domains/functions/components/topology/actorModal/V1ActorPanel.js:26-30`, `src/domains/functions/components/topology/actorModal/V1ActorPanel.js:130-168`, and `src/domains/functions/components/topology/actorModal/V1ActorPanel.js:257-263`.
172 +- Legacy `summary_fields` are rendered by the frontend actor modal info panel, and legacy `port_fields` are rendered by the graph port bullet tooltip path. This means both are presentation-adjacent contracts even if full table/modal composition remains SOW-0022.
173 + Evidence from the cloud-frontend worktree: `src/domains/functions/components/topology/actorModal/index.js:277-282` and `src/domains/functions/components/graph/forceGraph.js:1015-1044`.
174 +
175 +Current cloud-topology-service behavior:
176 +
177 +- The service schema copy has no presentation fields in `Data`, `TypeRegistry`, `ActorType`, `LinkType`, `EvidenceType`, or `TableType`.
178 + Evidence from the cloud-topology-service repo: `internal/topology/schema/payload.go:29-110`.
179 +- The aggregation core merges type registry definitions by normalized deep equality and returns a hard error for conflicting definitions.
180 + Evidence from the cloud-topology-service repo: `internal/topology/aggregate/aggregate.go:984-1048`.
181 +- The aggregation spec already says type registry entries with the same id must be semantically compatible, and conflicting definitions are aggregation errors.
182 + Evidence from the cloud-topology-service repo: `.agents/sow/specs/cloud-topology-service-contract.md:121-123`.
183 +- The current aggregation core merges actors by actor type plus `merge_identity`, otherwise `identity`, and links by remapped endpoints plus link type and direction policy.
184 + Evidence from the cloud-topology-service repo: `internal/topology/aggregate/aggregate.go:196-246`, `internal/topology/aggregate/aggregate.go:249-300`, and `.agents/sow/specs/cloud-topology-service-contract.md:124-128`.
185 +- The service contract says evidence type match columns preserve exact relationship details, but the current model does not yet define a cross-payload matcher strategy that can replace one topology's endpoint with another topology's actor using domain-specific keys.
186 + Evidence from the cloud-topology-service repo: `.agents/sow/specs/cloud-topology-service-contract.md:103-116`.
187 +
188 +Gap classification:
189 +
190 +- Preserve in v1 as compact presentation/profile metadata:
191 + - actor type label;
192 + - actor fill color token;
193 + - actor border color/style token;
194 + - actor annotation ring token;
195 + - actor role/render role;
196 + - actor icon token;
197 + - actor size-by policy;
198 + - actor label/display-name synthesis policy;
199 + - link type label;
200 + - link color token;
201 + - link line shape token;
202 + - link curve token;
203 + - link width token;
204 + - link direction/arrow token;
205 + - port/bullet type label and color token;
206 + - legend entries/order;
207 + - graph selection/highlight behavior.
208 +- Replace old fields with safer tokens/profiles:
209 + - old `color_slot` stays as token semantics but should become explicit enum/profile vocabulary;
210 + - old raw `icon_svg` should become UI-owned `icon` token unless a controlled signed/allowlisted custom icon registry is explicitly approved;
211 + - old boolean `dash` should become line shape enum such as `solid`, `dotted`, `dashed`;
212 + - old numeric `width` should become width token such as `thin`, `normal`, `thick`, or bounded scalar if the UI team confirms safe limits;
213 + - old `actor_click_behavior` should become a selection/highlight profile with composition rules, not a single global string.
214 +- Move to SOW-0022:
215 + - summary field composition;
216 + - table columns, column formatters, and modal tab composition;
217 + - raw JSON/nested array rendering rules;
218 + - actor/link modal table grouping;
219 + - derived relationship evidence drilldowns;
220 + - safe display of endpoint objects, neighbors, port inventory, and overlays.
221 +- Add because old schema did not cover enough:
222 + - actor `label_policy` / display synthesis, to prevent canonical identity arrays from becoming actor labels;
223 + - explicit note that cross-payload actor/link reconciliation is structural and tracked by SOW-0023;
224 + - presentation conflict policy for Cloud aggregation;
225 + - link curve and arrow tokens for bidirectional/directed rendering;
226 + - popover profile references for actor/link hover summaries;
227 + - annotation ring semantics for status/classification overlays.
228 +- Add to the preserve/replace matrix because reviewers and local verification found missing old behavior:
229 + - `port_fields`, at least as UI label metadata for port bullet tooltip fields;
230 + - actor/link/port opacity semantics, preferably as closed opacity tokens rather than arbitrary floats;
231 + - old `topology_match` display-relevant fields as input to the new `label_policy`, not as a raw object to recreate. Cross-payload identity vocabulary moves to SOW-0023.
232 +- Intentionally keep out of the payload:
233 + - coordinates, force-layout physics, viewport, pan/zoom, z-index, CSS class names, raw theme colors, raw CSS, component names, and user runtime interaction state.
234 +
235 +Required design outputs before implementation:
236 +
237 +1. Presentation profile schema attached to type registry entries and/or a top-level presentation registry.
238 +2. UI token vocabulary for actor/link/port visuals that is stable and documented.
239 +3. Label synthesis policy that separates canonical identity from human display.
240 +4. Highlight profile schema for direct-neighbor, path, and future neighborhood behaviors.
241 +5. Cloud aggregation conflict policy for presentation profile disagreements.
242 +6. SOW-0022 handoff with the modal/table composition inventory above.
243 +7. SOW-0023 handoff for cross-payload actor reconciliation.
244 +
245 +Risks:
246 +
247 +- If presentation stays out of the payload, the UI will need producer/domain-specific hardcoding and future topologies will not render consistently.
248 +- If raw CSS, SVG, layout coordinates, or frontend component names enter the payload, the schema will couple backend producers to frontend implementation details.
249 +- If Cloud aggregation accepts conflicting profiles silently, merged topologies may show inconsistent colors, icons, arrows, legends, or highlight behavior.
250 +- Until SOW-0023 teaches Cloud cross-payload match strategies, Cloud cannot safely replace endpoints from one topology with actors from another.
251 +- If actor labels are synthesized from canonical identity without display rules, the UI can show unusable labels such as long concatenated identity lists.
252 +- If presentation fields allow raw untrusted HTML/SVG, the UI could get a security-sensitive rendering surface.
253 +- If the schema is designed only around the three current in-tree producers, the vSphere topology in the companion worktree will either need a special UI path or will regress during migration.
254 +
255 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
256 +
257 +Status at implementation start: ready-for-implementation (historical snapshot; final closure evidence is in the Validation and Outcome sections).
258 +
259 +Problem / root-cause model:
260 +
261 +- The compact schema removed the old presentation plane together with verbose compatibility data. This improved payload size but also removed backend-controlled visual semantics that the UI needs to render generic topologies without producer-specific hardcoding.
262 +- Actor display names and labels are currently not modeled as a first-class presentation contract. Producers may emit canonical or aggregated identities that are correct for matching but unusable as human labels.
263 +- Cloud aggregation cannot safely merge presentation profiles until the schema declares how profiles are identified, versioned, and resolved.
264 +
265 +Evidence reviewed:
266 +
267 +- `src/plugins.d/FUNCTION_UI_SCHEMA.json` defines the old topology presentation objects.
268 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` lacks equivalent presentation/profile definitions.
269 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` currently warns against visual layout hints but does not distinguish UI-owned layout from backend-owned visual semantics.
270 +- User-provided UI observation shows raw/aggregated identity and raw data currently leak into final topology presentation. Raw examples are intentionally not copied into this durable artifact.
271 +
272 +Affected contracts and surfaces:
273 +
274 +- `netdata.topology.v1` JSON Schema.
275 +- Topology producer output for network-connections, streaming, SNMP/L2, and future vSphere topology.
276 +- Cloud frontend topology renderer, graph legend, popovers, highlights, actor labels, and icon/color/line token mappings.
277 +- Cloud topology aggregation service type/profile merge logic.
278 +- Public `create-topology` skill and developer guide.
279 +- Topology schema spec under `.agents/sow/specs/`.
280 +
281 +Existing patterns to reuse:
282 +
283 +- Old `FUNCTION_UI_SCHEMA.json` presentation metadata as inventory input, not as a direct replacement.
284 +- New v1 type registry model in `FUNCTION_TOPOLOGY_SCHEMA.json`.
285 +- Existing compact-table and type-registry helpers in `src/go/pkg/topology/v1`.
286 +- Existing topology producer split between graph links, evidence, tables, and overlays.
287 +- UI-token model proposed by the user: UI provides enums/primitives; backend composes profiles using those enums.
288 +
289 +Risk and blast radius:
290 +
291 +- Schema changes affect every producer and both Cloud consumers.
292 +- The frontend must support old and new schemas during rollout.
293 +- Cloud aggregation must not conflate incompatible visual/type definitions.
294 +- Existing large-payload gains must not be lost by adding repeated per-row presentation data.
295 +- Raw topology captures can contain sensitive customer/infrastructure details and must remain out of durable artifacts.
296 +
297 +Sensitive data handling plan:
298 +
299 +- Do not copy raw topology examples, raw MAC/IP addresses, interface aliases, hostnames, usernames, passwords, tokens, node IDs, claim IDs, or customer-identifying data into SOWs, specs, docs, skills, code comments, commits, or PR text.
300 +- Use sanitized summaries and generic examples only.
301 +- Keep raw captured payloads under `.local/` only.
302 +- If fixtures are needed, generate sanitized fixtures with placeholder identifiers and no real infrastructure values.
303 +
304 +Implementation plan:
305 +
306 +1. Inventory old presentation contract, old producer emissions, current v1 schema gaps, and frontend consumption points.
307 +2. Run the requested external read-only reviewers against the inventory and this SOW.
308 +3. Resolve presentation schema decisions, especially profile shape, token sets, label synthesis, highlight composition, and Cloud aggregation conflict policy.
309 +4. Extend schema/docs/spec/skill with compact presentation profiles and SOW-0023 handoff notes for cross-payload matching.
310 +5. Create UI and Cloud aggregator worker handoff documents.
311 +6. Update backend producers to emit the presentation profiles and validate outputs.
312 +
313 +Validation plan:
314 +
315 +- JSON Schema validation for fixtures using presentation profiles.
316 +- Targeted tests for Go topology v1 helpers if helper structs/builders change.
317 +- Function validation for at least one producer output per topology kind touched.
318 +- Same-failure scan for old visual fields and current v1 gaps.
319 +- Reviewer pass before schema freeze and, if material changes are made after reviewer findings, repeat review with the same scope plus fix notes.
320 +
321 +Artifact impact plan:
322 +
323 +- AGENTS.md: no expected update unless workflow rules change.
324 +- Runtime project skills: likely no update except existing `project-writing-collectors` references if topology producer workflow changes.
325 +- Specs: update `.agents/sow/specs/topology-function-schema.md`.
326 +- End-user/operator docs: update `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`; no public user docs expected unless Function schema docs are published.
327 +- End-user/operator skills: update `docs/netdata-ai/skills/create-topology/SKILL.md`.
328 +- SOW lifecycle: SOW-0020 paused; SOW-0021 current; SOW-0022 pending for table/modal composition.
329 +
330 +Open-source reference evidence:
331 +
332 +- `kiali/kiali @ ad210d7fd2a4b819e6ceae5f9a744847c4dcc7b2`, `frontend/src/types/Graph.ts:384-451` models graph nodes and edges with domain-specific fields such as node type, namespace, traffic, health, and source/target. This is useful evidence that mature topology UIs often carry semantic graph data, but it is not a generic presentation-token contract Netdata can copy directly.
333 +- `apache/skywalking @ 4890024b6cc1c222838b5ebd16e10938762cd7f2`, `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/Node.java:27-58` and `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/Call.java:31-143` keep backend topology facts around nodes, calls, components, and detect points.
334 +- `apache/skywalking-booster-ui @ 0dfb65bad317fae75353dc4ff89fae663d5e1dc5`, `src/types/topology.ts:18-74` includes UI-side fields such as positions and lower-arc hints. This supports keeping coordinates, physics, and layout state out of Netdata producer payloads while still allowing backend-owned semantic style tokens.
335 +
336 +Open decisions:
337 +
338 +- Resolved in `## Implications And Decisions` on 2026-05-09. Remaining implementation details are schema design details unless new evidence exposes another product decision.
339 +
340 +## Implications And Decisions
341 +
342 +1. User decision: split the remediation into two SOWs.
343 + - SOW-0021 fixes topology presentation.
344 + - SOW-0022 fixes table and actor-modal composition.
345 + - Reason: presentation profiles and table composition are related but separable contracts, and separating them allows smaller reviewable changes.
346 +
347 +2. User decision: UI must be domain-agnostic.
348 + - The UI provides enums/tokens/primitives.
349 + - Backend payloads compose actor/link/highlight/modal behavior using those tokens.
350 + - The UI must not hardcode meanings such as parent, child, router, client, server, or process.
351 +
352 +3. User decision: new compact topology payloads are all-inclusive for presentation.
353 + - Production `netdata.topology.v1` payloads carry presentation definitions.
354 + - Function `info` must not be the production transport for the new presentation contract.
355 + - Function `info.presentation` may exist only for legacy schema compatibility during rollout.
356 +
357 +4. User decision: actor/link/etc. type definitions are the collision surface, not actor/link rows.
358 + - Payloads define actor types, link types, port types, and presentation profiles once.
359 + - Actor and link rows only reference those definitions.
360 + - Cloud aggregation conflicts are about contradictory type/profile definitions, for example two payloads defining the same `router` type differently.
361 +
362 +5. User decision: raw SVG/custom icon payloads are not allowed in the new compact schema.
363 + - Use UI-owned icon tokens only.
364 + - Raw SVG, raw CSS, and producer-owned rendering code are excluded from `netdata.topology.v1` presentation.
365 +
366 +6. Scope decision: keep SOW-0021 and SOW-0022 separate, but make the boundary explicit.
367 + - SOW-0021 owns graph presentation, safe actor labels, legend, highlight behavior, link style tokens, and port/socket bullets needed on the graph.
368 + - SOW-0022 owns full actor/link modal composition, custom tables, shown/hidden columns, formatters, nested JSON rendering, and curated drilldown views.
369 + - Rationale: merging the SOWs would increase the review surface and risk mixing graph rendering with table composition. The split is safe only if SOW-0021 provides the graph-facing hooks and SOW-0022 consumes them without redefining them.
370 +
371 +7. Scope decision: cross-payload actor reconciliation becomes SOW-0023.
372 + - SOW-0021 documents the requirement but does not implement the matcher.
373 + - SOW-0023 will define cross-payload identity vocabularies, matching strategies, ambiguity handling, endpoint replacement rules, and Cloud aggregator behavior.
374 +
375 +8. User decision: Cloud presentation conflicts should prefer the newest producer/schema definition.
376 + - Producers must expose enough version information for Cloud to compare presentation/type definitions deterministically.
377 + - Cloud should prefer the newer Netdata/function/schema definition when the same presentation/type id has conflicting visual definitions.
378 + - If version comparison cannot break the tie, Cloud may choose one deterministic fallback profile and record diagnostics.
379 + - Structural facts are not considered contradictions only because two producers do not correlate; they either match or remain separate valid observations.
380 + - Superseded by decision 9 for raw aggregation: newest-wins can be used only as a display-preference heuristic after canonicalization, never to drop facts, rows, or distinct definitions.
381 +
382 +9. User decision refinement: type/profile definition collisions should be namespaced, then deduplicated.
383 + - Cloud should treat producer-local type/profile ids as local names until aggregation resolves them.
384 + - Cloud should canonicalize definitions and deduplicate identical definitions by content.
385 + - If two local ids have different definitions, Cloud should keep both by assigning distinct canonical ids instead of failing aggregation or dropping one.
386 + - This creates no data loss and no fatal definition conflicts.
387 + - Version preference can still be used later to choose a preferred display profile when multiple variants are semantically equivalent, but raw aggregation must preserve all variants.
388 +
389 +10. User decision: link width/opacity can be data-scaled, but scaling is keyed per link type.
390 + - Link type definitions declare whether a visual channel is variable and the scale key it belongs to.
391 + - Link rows may carry one raw numeric weight for that declared key.
392 + - The raw number is producer-domain-specific, for example traffic, packet count, socket count, or another unit.
393 + - Producers do not pre-scale to pixels or opacity because each payload does not know the whole aggregated range.
394 + - Cloud/UI scale visual width or opacity per scale key across visible links that share that key.
395 + - Only one variable visual weight is allowed per link row in this SOW.
396 +
397 +11. Link scaling examples accepted for schema design:
398 + - A `bandwidth` link type can declare scale key `traffic`; link rows carry raw traffic such as KiB.
399 + - A `connections` link type can declare scale key `sockets`; link rows carry raw socket counts.
400 + - These two link types can coexist because Cloud/UI scale `traffic` links against other `traffic` links and `sockets` links against other `sockets` links.
401 +
402 +12. User clarification: link rows remain compact table rows, not per-link JSON objects.
403 + - Example object notation in discussion is illustrative only.
404 + - Production link data stays in the compact table shape: columns plus column encodings.
405 + - The link row carries a link type reference and, if the type declares a variable visual channel, the raw numeric value column declared by that link type.
406 +
407 +13. User clarification and schema decision: link type and scale key are separate concepts.
408 + - `type` identifies the semantic link type, such as dependency, bandwidth, connections, LLDP, ownership, or streaming.
409 + - `scale_key` groups raw numeric weights that can be scaled together, such as `traffic` or `sockets`.
410 + - Multiple link types may share one `scale_key`.
411 + - A link type may have no variable scale key.
412 + - Link rows should not repeat `scale_key`; Cloud/UI resolve `row.type -> link_type.presentation.variable.scale_key`.
413 + - Link type values in compact tables should use dictionary/string-ref encoding so repeated type names are stored as compact indexes in row data.
414 + - Scale-key definitions are small type-level metadata. They may use stable ids, but clarity is preferred over numeric-only ids unless measurement shows type-level definitions matter for payload size.
415 +
416 +14. User decision: presentation is attached inside actor, link, and port type definitions.
417 + - Each actor, link, and port type can carry a `presentation` object that defines backend-selected UI tokens for that type.
418 + - This keeps a type as one object that defines both its structural meaning and its visual profile.
419 + - Cloud aggregation still treats producer-local ids as local names, namespaces them, and deduplicates canonical definitions.
420 + - If two payloads use the same local id with different presentation, Cloud keeps both as distinct canonical ids instead of failing or dropping one.
421 + - Structural identity and display presentation remain conceptually separate even though they live in the same type object.
422 +
423 +15. User decision: execution order is SOW-0021, then SOW-0023, then SOW-0022.
424 + - SOW-0021 will finish graph presentation, safe labels, legend, highlighting, type-level presentation, and graph-facing port/socket bullets first.
425 + - SOW-0023 will then solve cross-payload identity, matching, correlation, ambiguity policy, and endpoint replacement before full modal/table composition.
426 + - SOW-0022 will then complete actor/link modal and table composition on top of the presentation and identity foundations.
427 + - Reason: actor identity and naming affect topology shape, but the immediate graph regression still needs the SOW-0021 presentation contract first.
428 +
429 +16. Identity guardrail for SOW-0021:
430 + - Actor identity is stable and canonical; it is never used directly as display text unless explicitly marked safe by the producer.
431 + - Display labels come from explicit presentation/label policy.
432 + - Type/profile ids are producer-local until Cloud namespaces and deduplicates them.
433 + - Cross-payload actor reconciliation is not implemented in SOW-0021, but SOW-0021 must not add presentation rules that block SOW-0023.
434 +
435 +17. vSphere migration posture:
436 + - The vSphere topology in the separate worktree is not migrated by SOW-0021.
437 + - SOW-0021 keeps the vSphere-required color/icon tokens in the schema so the later migration does not need another schema round.
438 + - The frontend handoff explicitly requires fallback and concrete mappings for these tokens before vSphere uses them.
439 +
440 +## Reviewer Findings - 2026-05-09
441 +
442 +Seven requested read-only reviewer agents were run in parallel. Their raw outputs are stored under `.local/audits/topology-presentation-contract/reviews/` and are intentionally not committed.
443 +
444 +Consolidated findings:
445 +
446 +1. Presentation transport is unresolved.
447 + - Old `topology.Presentation` is Function `info` metadata.
448 + - Current cloud-frontend reads `response.presentation` from the Function info response.
449 + - The Cloud topology service aggregates topology payload `data`, not Function info metadata.
450 + - Risk: putting v1 presentation in the wrong place can make either the UI or the aggregator blind to it.
451 +
452 +2. Presentation should not be mixed into structural type definitions without an explicit policy.
453 + - Cloud aggregation currently hard-errors on conflicting type definitions.
454 + - If visual fields are embedded in `actor_types` or `link_types`, a harmless color/label difference can become an aggregation failure.
455 + - A separate presentation registry allows structural semantics and visual semantics to have different merge rules.
456 +
457 +3. Port bullets are graph presentation, not only table composition.
458 + - Old producers used `show_port_bullets`, `port_types`, `port_fields`, and `bullet_source`.
459 + - The frontend uses `port_fields` and `port_types` in the force graph tooltip path.
460 + - SOW-0021 must preserve enough port-bullet metadata for graph polish; SOW-0022 can still own full modal/table composition.
461 +
462 +4. Actor labels need a first-class safe label policy.
463 + - Current v1 frontend fallback joins arrays and can expose long identity lists as labels.
464 + - The policy must define safe source columns, fallback order, max length, array rejection or summarization, and what the UI does when no label source is safe.
465 +
466 +5. Raw SVG must not remain an open-ended producer surface.
467 + - The old schema allowed `icon_svg`.
468 + - Current frontend has regex-based SVG stripping before rendering legacy icons, but this is not a sufficient long-term security boundary.
469 + - Local search found no current topology producer emission of `icon_svg` in network-connections, streaming, SNMP, or the vSphere worktree. This means a closed UI-token icon model can be adopted without preserving active producer SVG output.
470 +
471 +6. Token vocabularies must be explicit and versioned.
472 + - Current cloud-frontend supports color slots `primary`, `secondary`, `accent`, `self`, `neutral`, `muted`, `dim`, `derived`, `info`, `structural`, and `warning`.
473 + - The vSphere worktree uses color slot names not present in that vocabulary.
474 + - Current cloud-frontend supports a closed icon token map, but schema/docs do not list or version the tokens.
475 +
476 +7. Cross-payload matching is structural, not just visual.
477 + - `merge_identity` is per actor type and does not define shared identity classes across producers.
478 + - Evidence `match_columns` preserve exact relationship details but do not declare how to replace one topology's endpoint with another topology's actor.
479 + - Network-connections, SNMP/L2, streaming, and vSphere need different identity keys and ambiguity policies.
480 +
481 +8. Presentation conflict policy must be explicit.
482 + - Options include hard error, first-wins, priority-based merge, or separate profile IDs with deterministic fallback.
483 + - Silent merging is unsafe; hard errors on visual-only differences are operationally fragile.
484 +
485 +9. Existing docs and skills currently contradict the new direction.
486 + - The topology spec says the producer no longer emits superseded presentation metadata.
487 + - The create-topology skill does not describe presentation profiles.
488 + - This SOW must update both so future producers do not repeat the regression.
489 +
490 +10. Cloud service schema/validator drift was found outside pure presentation.
491 + - Agent-side v1 schema allows `json` columns.
492 + - SNMP v1 emits `json` columns for metadata tables.
493 + - The Cloud topology service validator currently does not allow `json` column type and scalar validation rejects objects.
494 + - This must be handed to the Cloud aggregator worker because otherwise valid Agent payloads can be rejected before presentation is considered.
495 +
496 +11. SNMP actor subtype information needs preservation analysis.
497 + - The old presentation registry has distinct profiles for router, switch, firewall, access point, server, storage, load balancer, printer, phone, UPS, camera, generic device, endpoint, and segment.
498 + - Current v1 SNMP type registry exposes only device, endpoint, segment, and custom actor types.
499 + - The backend must preserve enough subtype or classification data for presentation profiles to reproduce the old color/icon distinctions where the old producer actually emitted those actor types.
500 +
501 +12. Curve and arrow tokens require UI support but are still schema-relevant.
502 + - Current canvas draws straight lines and has no arrowhead rendering.
503 + - The user explicitly requires backend-controlled line shape, curve, width, and arrow direction.
504 + - The schema can define closed tokens now, but the UI worker must implement graceful fallback for unsupported tokens.
505 +
506 +## Decision Gate - 2026-05-09
507 +
508 +These decisions are blocking schema and backend implementation. They are written here before implementation so the user can answer by number and option letter.
509 +
510 +Resolution note: this gate is resolved by `## Implications And Decisions`. Decision 14 supersedes the original recommendation in Decision Gate item 2: presentation is attached inside actor, link, and port type definitions, with Cloud namespacing and deduplication preventing visual-only conflicts from becoming data loss or aggregation failure.
511 +
512 +### 1. Where does v1 presentation live?
513 +
514 +A. **Recommended: `data.presentation` in the topology payload, with legacy Function `info.presentation` kept only for old-schema compatibility during rollout.**
515 +
516 +- Pros: Cloud aggregator receives presentation; UI can render from the same production payload; avoids duplication after rollout.
517 +- Cons: cloud-frontend must learn to read v1 presentation from `data`.
518 +- Implications: Function info remains a compatibility path only; schema, docs, producer helpers, frontend, and cloud-topology-service all share one production presentation contract.
519 +- Risks: requires coordinated UI and Cloud service updates before end-to-end validation.
520 +
521 +B. Keep presentation only in Function `info`.
522 +
523 +- Pros: matches old UI transport.
524 +- Cons: Cloud aggregator cannot merge or validate presentation because it is not in topology `data`.
525 +- Implications: cloud-topology-service cannot be the single topology path for all topology kinds.
526 +- Risks: repeats the split-brain old/new contract.
527 +
528 +C. Duplicate presentation in both Function `info` and topology `data`.
529 +
530 +- Pros: easiest frontend transition.
531 +- Cons: larger payloads and possible divergence between two copies.
532 +- Implications: every producer must keep two presentation surfaces synchronized.
533 +- Risks: stale info/data presentation mismatches create hard-to-debug UI behavior.
534 +
535 +### 2. How should presentation attach to schema?
536 +
537 +A. **Recommended: separate top-level `presentation` registry parallel to `types`.**
538 +
539 +- Pros: structural type compatibility remains separate from visual profile compatibility; Cloud can merge profiles with a presentation-specific policy.
540 +- Cons: actor/link/port types need explicit profile references or same-id conventions.
541 +- Implications: schema can evolve visual profiles without redefining topology identity.
542 +- Risks: frontend and service need one more registry to resolve.
543 +
544 +B. Inline presentation fields inside `types.actor_types` and `types.link_types`.
545 +
546 +- Pros: simpler producer shape.
547 +- Cons: visual differences become type definition conflicts unless aggregation is weakened.
548 +- Implications: type registry stops being purely structural.
549 +- Risks: harmless visual changes can break Cloud aggregation.
550 +
551 +C. Put presentation under `extensions`.
552 +
553 +- Pros: fastest schema bypass.
554 +- Cons: no validation, no stable contract, no reliable aggregation policy.
555 +- Implications: future producers will invent incompatible shapes.
556 +- Risks: recreates the current regression with another untyped escape hatch.
557 +
558 +### 3. What is the icon/SVG policy?
559 +
560 +A. **Recommended: closed UI-owned icon tokens only; raw SVG and raw CSS are banned from v1 presentation.**
561 +
562 +- Pros: safest security boundary; compact; frontend can version tokens.
563 +- Cons: producers can only use icons the UI exposes.
564 +- Implications: add missing icon tokens to the UI token catalog as needed.
565 +- Risks: a producer needing a custom icon waits for a UI token addition.
566 +
567 +B. Allow raw SVG with sanitizer.
568 +
569 +- Pros: preserves maximum producer flexibility.
570 +- Cons: regex sanitization is not a strong security boundary; proper SVG sanitization is non-trivial.
571 +- Implications: every topology payload becomes a UI rendering security surface.
572 +- Risks: stored XSS or broken rendering through sanitizer bypasses.
573 +
574 +C. Design a signed/allowlisted custom icon registry now.
575 +
576 +- Pros: flexible and safer than raw inline SVG.
577 +- Cons: substantially larger security, distribution, signing, revocation, and on-prem/offline design.
578 +- Implications: delays this SOW.
579 +- Risks: scope creep and incomplete security model.
580 +
581 +### 4. What belongs to SOW-0021 vs SOW-0022?
582 +
583 +A. **Recommended: SOW-0021 owns graph presentation and safe labels; SOW-0022 owns modal/table composition.**
584 +
585 +- SOW-0021 includes actor/link/port profiles, legend, line shape, width, curve/arrow tokens, highlight profiles, `label_policy`, `show_port_bullets`, `port_types`, and `port_fields` needed for graph port tooltips.
586 +- SOW-0022 includes modal tabs, summary sections, custom tables, table columns, formatters, nested JSON rendering, and raw-data suppression inside actor/link modals.
587 +- Pros: fixes visible graph polish first while preserving the user's two-step split.
588 +- Cons: some legacy fields, especially `summary_fields`, sit near the boundary and need explicit handoff notes.
589 +- Implications: SOW-0021 must not attempt to fully solve actor modals; SOW-0022 must not redefine graph label/profile tokens.
590 +- Risks: port tooltip and hover summary details need careful ownership to avoid gaps.
591 +
592 +B. Move all summaries, popovers, port fields, and modal tabs to SOW-0022.
593 +
594 +- Pros: strict table/modal ownership.
595 +- Cons: SOW-0021 can restore colors but still leave bullets/popovers weak.
596 +- Implications: the graph may remain visibly incomplete until SOW-0022.
597 +- Risks: fails the "actor bullets/ports/sockets" part of the presentation complaint.
598 +
599 +C. Move modal tabs and summary fields into SOW-0021 too.
600 +
601 +- Pros: closer parity with the old `PresentationActorType` bundle.
602 +- Cons: expands SOW-0021 into table/modal composition and violates the requested two-step split.
603 +- Implications: SOW-0022 becomes smaller but SOW-0021 becomes much larger.
604 +- Risks: delays the graph presentation repair.
605 +
606 +### 5. Is cross-payload actor reconciliation in SOW-0021?
607 +
608 +A. **Recommended: SOW-0021 documents the requirement and creates/updates a separate SOW for cross-payload matching; it does not implement the matching algorithm.**
609 +
610 +- Pros: keeps SOW-0021 focused on presentation; acknowledges the issue honestly; avoids mixing visual schema work with structural graph reconciliation.
611 +- Cons: aggregated multi-producer topologies may still show duplicates until the matching SOW is implemented.
612 +- Implications: presentation schema can include only non-invasive identity labels needed by UI; matching strategy, ambiguity handling, and endpoint replacement live in the follow-up.
613 +- Risks: future schema changes may be needed if the matching SOW requires additional producer declarations.
614 +
615 +B. Add producer-declared match strategies to v1 now, but leave Cloud implementation to a later service change.
616 +
617 +- Pros: future-proofs the payload.
618 +- Cons: unimplemented schema fields can drift or be misused.
619 +- Implications: producers must emit strategy metadata before the aggregator consumes it.
620 +- Risks: false confidence that matching works because the payload has declarations.
621 +
622 +C. Include full cross-payload matching schema and Cloud aggregator algorithm in SOW-0021.
623 +
624 +- Pros: solves the structural duplicate-actor problem now.
625 +- Cons: significantly expands scope beyond presentation.
626 +- Implications: SOW-0021 must define typed identity vocabularies, normalization, ambiguity policy, match confidence, and tests across all topology kinds.
627 +- Risks: delays presentation repair and increases review surface.
628 +
629 +### 6. What is the presentation conflict policy for Cloud aggregation?
630 +
631 +A. **Recommended: structural conflicts stay hard errors; presentation conflicts use deterministic profile merge with diagnostics, not topology failure.**
632 +
633 +- Pros: preserves correctness for topology identity while avoiding visual-only aggregation outages.
634 +- Cons: a merged view may pick one visual profile when producers disagree.
635 +- Implications: Cloud service must record conflict counts/details and choose profiles by deterministic priority, such as explicit profile priority then producer/source ordering.
636 +- Risks: users may see a generic or first-selected style when producers disagree.
637 +
638 +B. Hard-error on any presentation conflict.
639 +
640 +- Pros: simplest and maximally strict.
641 +- Cons: visual-only differences can prevent topology rendering.
642 +- Implications: every producer version must agree exactly on profile definitions before Cloud aggregation works.
643 +- Risks: fragile during rolling upgrades.
644 +
645 +C. Last-writer-wins silently.
646 +
647 +- Pros: easy implementation.
648 +- Cons: nondeterministic unless merge order is guaranteed; hides real producer disagreements.
649 +- Implications: UI may change colors/icons depending on aggregation order.
650 +- Risks: confusing and hard to debug.
651 +
652 +### 7. Should link width/opacity be tokens or bounded numbers?
653 +
654 +A. **Recommended: closed tokens for width and opacity in v1.**
655 +
656 +- Pros: compact, theme-owned, predictable, easy to validate.
657 +- Cons: less granular than old numeric values.
658 +- Implications: map old width/opacity to `thin`/`normal`/`emphasis` and `normal`/`muted`/`faded`.
659 +- Risks: some old visual nuance may be approximated.
660 +
661 +B. Bounded numeric values.
662 +
663 +- Pros: closer to old schema and existing canvas math.
664 +- Cons: producers can still tune UI details too closely.
665 +- Implications: schema must enforce min/max and frontend must clamp.
666 +- Risks: style drift between producers.
667 +
668 +C. Keep old raw numeric semantics.
669 +
670 +- Pros: simplest migration from old schema.
671 +- Cons: keeps old ambiguity.
672 +- Implications: backend retains too much control over frontend look.
673 +- Risks: inconsistent and unreviewed visual scaling.
674 +
675 +## Plan
676 +
677 +1. Inventory old and new contracts.
678 +2. Produce a gap matrix with preserve/replace/drop/defer classification.
679 +3. Run requested external reviewers.
680 +4. Extend schema and docs.
681 +5. Write UI and Cloud aggregator handoff docs.
682 +6. Implement backend producer support.
683 +7. Validate and update SOW.
684 +8. Hand execution to SOW-0023 before starting SOW-0022.
685 +
686 +## Execution Log
687 +
688 +### 2026-05-09
689 +
690 +- Opened SOW-0021 from user direction after discovering the compact schema dropped required visual semantics.
691 +- Paused SOW-0020 so presentation repair can be handled as an explicit, reviewable step.
692 +- Created SOW-0022 as the follow-up for actor modal/table composition.
693 +- Ran all seven requested read-only external reviewers and consolidated their findings into `## Reviewer Findings - 2026-05-09`.
694 +- Added `## Decision Gate - 2026-05-09` because the reviewers found blocking design choices that must be answered before schema/backend implementation.
695 +- Recorded the final execution order decision: finish SOW-0021, then run SOW-0023, then run SOW-0022.
696 +- Recorded the SOW-0021 identity guardrail so presentation labels do not misuse canonical identity and do not block the cross-payload matcher.
697 +- Implemented compact presentation in `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`, `src/go/pkg/topology/v1`, network-connections, streaming, and SNMP v1 producers.
698 +- Created worker handoff docs:
699 + - `<cloud-frontend-repo>/TODO-topology-presentation-contract.md`
700 + - `<cloud-topology-service-repo>/REQUIREMENTS-topology-presentation.md`
701 +- Created the backend-worker SOW and expanded the frontend-worker TODO so the
702 + other workers can port the SOW-0021 changes into their codebases:
703 + - `<cloud-topology-service-repo>/.agents/sow/pending/SOW-0005-20260509-port-topology-presentation-contract.md`
704 + - `<cloud-topology-service-repo>/SOW-status.md`
705 + - `<cloud-frontend-repo>/TODO-topology-presentation-contract.md`
706 +- Answered the first Cloud frontend worker question batch by appending
707 + `## Answers - 2026-05-10` to
708 + `<cloud-frontend-repo>/TODO-topology-presentation-contract.md`.
709 + The answers explicitly state that the frontend remains topology-schema
710 + agnostic and must use representative v1 feature fixtures, not
711 + producer-specific UI branches.
712 +- Created `.agents/sow/pending/SOW-0024-20260510-vsphere-topology-v1-migration.md`
713 + to track the later vSphere migration to `netdata.topology.v1` after
714 + SOW-0021, SOW-0023, and SOW-0022 are finished.
715 +- Ran the second requested-style read-only reviewer pass after material implementation changes. Raw outputs are under `.local/audits/topology-presentation-contract/reviews-round2/`.
716 +- Gemini returned no review output in round 2; the output file is zero bytes. The other read-only reviewers completed and their findings were consolidated locally.
717 +- Fixed the real second-round findings: semantic presentation validation, `highlight_path` and metric-size schema conditionals, width/opacity token constraints for link variables, explicit port-bullet sources, token documentation, handoff merge rules, SNMP custom legend entry, and the network-connections ownership metrics leak.
718 +- Ran a third read-only reviewer pass after the second-round fixes, using the same scope plus fix notes. Raw outputs are under `.local/audits/topology-presentation-contract/reviews-round3/`.
719 +- Kimi timed out in the third pass after 30 minutes and produced only progress output, not a final review.
720 +- Fixed the concrete third-round findings:
721 + - streaming port-bullet labels now use `port_name`, not raw `actor_ref` row indexes;
722 + - network-connections only advertises socket-evidence port bullets in detailed mode, where socket evidence is present;
723 + - `label_policy` rejects non-display columns such as `array`, `json`, and raw refs;
724 + - hover fields validate against actor/link table columns;
725 + - highlight-path actor/order columns are type-checked;
726 + - actor-table port sources must reference declared table types or runtime actor tables;
727 + - SNMP always declares the `actor_ports` table type for its port-bullet source and adds `unknown` to the port legend;
728 + - Go `BorderPresentation.Enabled` is now optional so schema defaults are not contradicted by zero-value Go structs.
729 + - topology/v1 tests now verify Go token arrays stay in sync with the JSON Schema token enums.
730 +- Added the public `create-topology` how-to for graph presentation and linked it from the live how-to catalog:
731 + - `docs/netdata-ai/skills/create-topology/how-tos/add-graph-presentation.md`
732 + - `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`
733 +- Investigated runtime SNMP/L2 visual regression reported after frontend
734 + integration: inferred/probable links were rendered like LLDP/CDP links.
735 + Root cause: the SNMP v1 producer collapsed all graph rows into one
736 + `l2_observation` link type, so the frontend had no schema-level signal to
737 + apply the old `probable` presentation.
738 +- Fixed SNMP v1 link typing:
739 + - graph rows now preserve semantic link types for `lldp`, `cdp`, `bridge`,
740 + `fdb`, `stp`, `arp`, `snmp`, `probable`, and fallback `l2_observation`;
741 + - each link type has its own v1 presentation tokens;
742 + - evidence sections are split by matching evidence type so
743 + `evidence_types.<id>.link_type` stays coherent with the graph link type.
744 +- Added the public `create-topology` how-to for preserving semantic link types:
745 + - `docs/netdata-ai/skills/create-topology/how-tos/preserve-semantic-link-types.md`
746 + - `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`
747 +- Investigated runtime streaming highlight-path regression reported after the
748 + frontend integration: clicking a streaming actor highlighted only direct
749 + graph siblings instead of that actor's ordered streaming path.
750 + Root cause: the v1 streaming payload configured `path_actor_column: "actor"`
751 + while the `actor` column is the clicked/owner actor, not the path member; the
752 + frontend v1 adapter also did not materialize v1 path tables into the legacy
753 + `streamingPath` node field consumed by the graph click handler.
754 +- Extended the `highlight_path` contract with optional `path_owner_column`.
755 + `path_actor_column` now means path member; `path_owner_column` means the
756 + clicked actor that owns the path row. Existing global path-table payloads
757 + remain valid when `path_owner_column` is omitted.
758 +- Fixed streaming v1 path rows:
759 + - `stream_path` now carries both owner `actor` and member `path_actor`;
760 + - streaming presentation now points at `path_owner_column: "actor"` and
761 + `path_actor_column: "path_actor"`;
762 + - the v1 stream-path table mirrors the old highlight-path helper by appending
763 + the local agent to stored paths when storage does not already include it.
764 +- Fixed the frontend v1 adapter in the Cloud frontend worktree:
765 + - validates optional `path_owner_column`;
766 + - resolves per-owner highlight paths from compact actor tables;
767 + - materializes `streamingPath` arrays onto graph nodes before `ForceGraph`
768 + handles click selection.
769 +- Updated the Cloud frontend and Cloud topology service handoff documents so
770 + worker agents port `path_owner_column` semantics together with the backend
771 + schema change.
772 +- Added the public `create-topology` how-to for per-actor highlight paths:
773 + - `docs/netdata-ai/skills/create-topology/how-tos/define-per-actor-highlight-paths.md`
774 + - `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`
775 +
776 +## Validation
777 +
778 +Acceptance criteria evidence:
779 +
780 +- Old graph presentation semantics were inventoried from old schema, old producer emissions, and frontend consumers in `## Inventory - 2026-05-09`.
781 +- `netdata.topology.v1` now has:
782 + - type-level actor/link/port presentation;
783 + - graph-level `data.presentation`;
784 + - safe `label_policy`;
785 + - closed icon/color/opacity/width/line/curve/arrow tokens;
786 + - explicit port-bullet sources;
787 + - legend, highlight-path, scale-key, hover, annotation, and variable link scaling contract.
788 +- Backend producers covered by this SOW emit v1 presentation:
789 + - `src/collectors/network-viewer.plugin/network-viewer.c`
790 + - `src/web/api/functions/function-topology-streaming.c`
791 + - `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`
792 +- UI and Cloud aggregator worker handoff docs exist at the absolute paths recorded in the execution log.
793 +- Backend service handoff is now a real pending SOW in the service repository:
794 + `<cloud-topology-service-repo>/.agents/sow/pending/SOW-0005-20260509-port-topology-presentation-contract.md`.
795 +- Frontend handoff is now an expanded implementation TODO in the frontend
796 + repository: `<cloud-frontend-repo>/TODO-topology-presentation-contract.md`.
797 +
798 +Tests or equivalent validation:
799 +
800 +- `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
801 +- `go test ./pkg/topology/v1 ./plugin/go.d/collector/snmp_topology ./tools/functions-validation/validate` passed from `src/go`.
802 +- C syntax validation passed using the exact compile commands from `build/compile_commands.json` with `-fsyntax-only` for:
803 + - `src/collectors/network-viewer.plugin/network-viewer.c`
804 + - `src/web/api/functions/function-topology-streaming.c`
805 +- After third-round fixes, the same Go test command and both C `-fsyntax-only` checks passed again.
806 +- After the SNMP/L2 inferred-link fix,
807 + `go test ./plugin/go.d/collector/snmp_topology` passed from `src/go`.
808 +- After the streaming highlight-path fix:
809 + - `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
810 + - `go test ./pkg/topology/v1 ./plugin/go.d/collector/snmp_topology ./tools/functions-validation/validate` passed from `src/go`.
811 + - C syntax validation passed for `src/web/api/functions/function-topology-streaming.c` using the exact compile command from `build/compile_commands.json` with `-fsyntax-only`.
812 + - In `<cloud-frontend-repo>`, `yarn test src/domains/functions/topology/v1/normalizeTopologyV1.test.js src/domains/functions/topology/v1/buildRenderPresentation.test.js --runInBand` passed with 83 tests.
813 +- The Go test suite includes a schema-token parity test so future token additions must update the schema and Go validator together.
814 +- The SNMP Go tests now decode `data.links.type` and verify LLDP rows remain
815 + `lldp`, probable/inferred rows become `probable`, presentation tokens differ,
816 + evidence type link references match the graph link type, and legend entries
817 + include both categories.
818 +- Full CMake/Ninja build could not run because the local `build/` directory and Ninja files are owned by `root`; `cmake --build build --target network-viewer.plugin netdata -j 8` fails with `.ninja_lock` permission denied.
819 +- Backend service SOW audit passed after creating
820 + `cloud-topology-service/.agents/sow/pending/SOW-0005-20260509-port-topology-presentation-contract.md`.
821 +
822 +Real-use evidence:
823 +
824 +- The user confirmed the previous SNMP/L2 inferred-link visual fix works.
825 +- The streaming highlight-path repair is locally validated by backend schema,
826 + Go, C syntax, and frontend adapter tests. The user later confirmed the
827 + installed topology view works with the patched producer/UI path.
828 +- Direct Cloud aggregation runtime validation remains owned by the Cloud
829 + topology service worker handoff, not by this netdata repository commit.
830 +- The producer code was validated by Go tests, JSON Schema tests, function-validation fixtures, and C syntax checks. Full Cloud aggregation parity remains dependent on the Cloud topology service handoff.
831 +
832 +Reviewer findings:
833 +
834 +- First review round completed. Raw outputs are under `.local/audits/topology-presentation-contract/reviews/`; consolidated findings are recorded in `## Reviewer Findings - 2026-05-09`.
835 +- Second review round completed for Codex, Claude, Qwen, GLM, MiniMax, Kimi, and MiMo. Raw outputs are under `.local/audits/topology-presentation-contract/reviews-round2/`.
836 +- Second-round Gemini produced no output; `.local/audits/topology-presentation-contract/reviews-round2/gemini.txt` is zero bytes.
837 +- Real findings handled in code/docs/handoffs:
838 + - semantic presentation cross-reference validation in `src/go/pkg/topology/v1/validate.go`;
839 + - schema conditional requirements for `highlight_path` and metric actor sizing;
840 + - link variable `min`/`max` constrained to width/opacity tokens;
841 + - explicit `ports.sources[]` instead of implicit/misleading port type columns;
842 + - token vocabulary and fallback guidance;
843 + - Cloud merge rules for `label_policy`, scale keys, port sources, and column-name preservation;
844 + - sensitive identifier handling note for streaming detail tables;
845 + - SNMP `custom` actor added to legend;
846 + - network-connections ownership link no longer declares socket-only metrics.
847 +- Third review round completed for Codex, Claude, Qwen, GLM, MiniMax, and MiMo. Kimi timed out after 30 minutes without a final review. Raw outputs are under `.local/audits/topology-presentation-contract/reviews-round3/`.
848 +- Third-round concrete findings handled in code/docs:
849 + - streaming port-bullet `name_column` now references a scalar display column;
850 + - network-connections no longer advertises evidence-derived port bullets when evidence is omitted in aggregated mode;
851 + - actor-table port sources no longer silently pass missing table declarations;
852 + - label policy and hover fields now reject non-display columns;
853 + - highlight-path path columns now require `actor_ref` and numeric order types;
854 + - frontend/cloud handoffs now clarify scalar bullet labels, fallback defaults, and evidence id rewriting.
855 +- Third-round findings that remain outside SOW-0021 implementation are tracked by worker handoffs or follow-up SOWs:
856 + - Cloud topology service must accept Agent-valid `json` columns before end-to-end Cloud aggregation is ready;
857 + - frontend worker must implement token fallbacks and port-source rendering tests;
858 + - full CMake build and runtime Function samples require a writable build tree / installed Agent and the worker integrations.
859 +
860 +Same-failure scan:
861 +
862 +- `rg` scan for legacy-only presentation fields over the new v1 producers/docs found no remaining production use of `icon_svg`, `actor_click_behavior`, `summary_fields`, `bullet_source`, or `topology_match`.
863 +- Remaining `show_port_bullets` matches are local variable names in C emitters and the migration note that maps old `show_port_bullets` to v1 `ports.show_bullets`.
864 +- `rg` scan for `path_actor_column` still pointing at owner `actor` found no
865 + remaining production payload definition after the streaming fix; remaining
866 + matches are the SOW root-cause note, schema/spec prose, validator errors, and
867 + the intended `path_actor` emission.
868 +
869 +Sensitive data gate:
870 +
871 +- Raw user-provided examples are not copied into this SOW. This SOW uses sanitized summaries only.
872 +- No raw topology captures were committed. Raw reviewer outputs stay under `.local/`.
873 +
874 +Artifact maintenance gate:
875 +
876 +- AGENTS.md: no update needed; workflow rules did not change.
877 +- Runtime project skills: no generic `.agents/skills/project-*` update was needed; the relevant topology producer workflow is the public `create-topology` skill, updated below.
878 +- Specs: `.agents/sow/specs/topology-function-schema.md` was updated with the presentation contract, token vocabulary, port-bullet sources, and aggregation conflict policy.
879 +- End-user/operator docs: `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` and `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` were updated for the new v1 presentation model.
880 +- End-user/operator skills: `docs/netdata-ai/skills/create-topology/SKILL.md` was updated; `docs/netdata-ai/skills/create-topology/how-tos/add-graph-presentation.md` and `docs/netdata-ai/skills/create-topology/how-tos/preserve-semantic-link-types.md` were added and linked from `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`.
881 +- End-user/operator skills: `docs/netdata-ai/skills/create-topology/how-tos/define-per-actor-highlight-paths.md` was added and linked from `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`.
882 +- SOW audit hygiene: `.agents/skills/mirror-netdata-repos/SKILL.md` had an existing SSH clone example that matched the audit email-address heuristic. It was reworded without changing behavior so the sensitive-data audit passes cleanly.
883 +- SOW lifecycle: SOW-0021 is marked `completed` and is moved to `.agents/sow/done/` with the netdata implementation commit. SOW-0020 is closed in the same netdata commit because this work built on its compact topology schema foundation. SOW-0023 and SOW-0022 remain pending follow-ups.
884 +
885 +Specs update:
886 +
887 +- Updated `.agents/sow/specs/topology-function-schema.md`.
888 +
889 +Project skills update:
890 +
891 +- No generic `.agents/skills/project-*` update was needed.
892 +- Updated the public `create-topology` skill because it is the topology producer workflow reference:
893 + - `docs/netdata-ai/skills/create-topology/SKILL.md`
894 + - `docs/netdata-ai/skills/create-topology/how-tos/add-graph-presentation.md`
895 + - `docs/netdata-ai/skills/create-topology/how-tos/preserve-semantic-link-types.md`
896 + - `docs/netdata-ai/skills/create-topology/how-tos/define-per-actor-highlight-paths.md`
897 + - `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`
898 +
899 +End-user/operator docs update:
900 +
901 +- Updated `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`.
902 +- Updated `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`.
903 +
904 +End-user/operator skills update:
905 +
906 +- Updated `docs/netdata-ai/skills/create-topology/SKILL.md`.
907 +- Added `docs/netdata-ai/skills/create-topology/how-tos/add-graph-presentation.md`.
908 +- Added `docs/netdata-ai/skills/create-topology/how-tos/preserve-semantic-link-types.md`.
909 +- Added `docs/netdata-ai/skills/create-topology/how-tos/define-per-actor-highlight-paths.md`.
910 +- Updated `docs/netdata-ai/skills/create-topology/how-tos/INDEX.md`.
911 +
912 +Lessons:
913 +
914 +- Closed token schemas are not enough by themselves. Semantic validation must also check that labels, legends, path references, scale keys, and port-bullet sources point to real tables, columns, and type definitions.
915 +- Port bullets need explicit data-source declarations. A boolean `show_bullets` restores only visibility, not the data contract the UI needs.
916 +- Dynamic actor detail tables, especially SNMP tables, require careful validation: required source columns can be checked, but optional enrichment columns must be allowed to be absent when the table shape varies by device.
917 +- If a type-level presentation references optional runtime data, the type registry still needs a stable declaration for that source. Otherwise validation cannot distinguish an intentionally empty table from a typo.
918 +- Graph link type is the UI's presentation handle. If a producer collapses
919 + visually distinct facts into a generic link type, the frontend cannot remain
920 + topology-agnostic and still recover the old visual meaning.
921 +- Highlight paths that are actor-specific need two actor references: one for
922 + the owner/clicked actor and one for the path member. Reusing one column for
923 + both preserves table shape but loses the selection semantics.
924 +
925 +Follow-up mapping:
926 +
927 +- SOW-0022 tracks actor modal/table composition.
928 +- SOW-0023 tracks cross-payload actor reconciliation and must run before SOW-0022.
929 +
930 +## Outcome
931 +
932 +Backend/schema/docs/handoff work for SOW-0021 is implemented and locally validated in the netdata repository. The schema now carries backend-selected presentation profiles for actor, link, and port types; producer output preserves semantic link types; streaming can define per-actor highlight paths; and the Cloud frontend plus Cloud topology service handoffs are documented for their owning repositories.
933 +
934 +## Lessons Extracted
935 +
936 +See `## Validation` lessons above.
937 +
938 +## Followup
939 +
940 +- SOW-0022 handles modal/table composition.
941 +- SOW-0023 handles cross-payload actor reconciliation and is ordered before SOW-0022.
942 +
943 +## Regression Log
944 +
945 +None yet.
946 +
947 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0023-20260509-topology-cross-payload-matching.md new
+695
@@ -0,0 +1,695 @@
1 +# SOW-0023 - Topology cross-payload matching
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +Sub-state: netdata-side schema, spec, docs, developer skill, validators, fixtures,
8 +network-connections producer changes, and local frontend v1 rendering alignment
9 +are implemented and validated. Cloud service aggregation remains out of scope
10 +for the aggregated local UI fix because the Cloud aggregator receives detailed
11 +payloads.
12 +
13 +## Requirements
14 +
15 +### Purpose
16 +
17 +Define how Cloud reconciles actors and links that arrive from different topology producers and may describe the same real-world entity or relationship with different identity keys.
18 +
19 +### User Request
20 +
21 +The user identified that the Cloud aggregator must eventually know how to match across topology payloads:
22 +
23 +- network-connections by socket endpoint identity such as IP, port, protocol, direction, and address-space context;
24 +- SNMP/L2 by device, MAC, interface, port, chassis, and management identity;
25 +- streaming by machine or node identity;
26 +- vSphere by stable inventory object identity.
27 +
28 +The user accepted splitting this structural problem out of SOW-0021 so SOW-0021 can focus on presentation.
29 +
30 +### Assistant Understanding
31 +
32 +Facts:
33 +
34 +- `netdata.topology.v1` currently has per-actor-type `identity` and `merge_identity`.
35 +- Evidence types have `match_columns`, but those preserve relationship detail inside one payload and do not by themselves define cross-producer actor replacement.
36 +- Different topology producers use different identity vocabularies and may legitimately fail to correlate.
37 +- If two observations do not correlate, both remain valid; this is not a factual contradiction.
38 +
39 +Inferences:
40 +
41 +- Cross-payload matching needs a shared identity vocabulary or strategy registry, normalization rules, ambiguity policy, confidence, and tests.
42 +- This is structural graph reconciliation, not presentation.
43 +- The schema may need additional producer declarations, but those should be designed in this SOW, not hidden in presentation profiles.
44 +
45 +Unknowns:
46 +
47 +- Exact shared identity vocabulary and normalization rules.
48 +- Whether matching is pairwise between producer kinds or generic through typed identity facts.
49 +- How Cloud should handle ambiguous matches, partial matches, and conflicting confidence.
50 +- MVP scope is limited to the netdata-side generic contract plus
51 + network-connections producer emission. Cloud implementation details are
52 + tracked by the service worker SOW.
53 +- Exact producer migration order after the schema contract lands.
54 +
55 +### Acceptance Criteria
56 +
57 +- Inventory identity and match evidence emitted by network-connections, SNMP/L2, streaming, and vSphere.
58 +- Define a compact schema contract for cross-payload identity declarations if needed.
59 +- Define Cloud aggregator matching strategies, normalization rules, ambiguity policy, and diagnostics.
60 +- Define how endpoint actors are replaced, merged, or left separate.
61 +- Create the Cloud service handoff SOW that requires fixtures for successful
62 + match, no match, ambiguous match, partial link match, and conflicting
63 + presentation/type definitions.
64 +- Create the Cloud frontend handoff TODO for correlation rendering and link
65 + layout tokens.
66 +- Implement the network-connections producer changes needed to emit semantic
67 + ownership/resolved/correlation link types and socket correlation rows.
68 +- Update topology schema/spec/docs/skill if producer declarations are added.
69 +
70 +## Analysis
71 +
72 +Sources checked:
73 +
74 +- `.agents/sow/current/SOW-0021-20260509-topology-presentation-contract.md`
75 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
76 +- `src/collectors/network-viewer.plugin/network-viewer.c`
77 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`
78 +- `src/web/api/functions/function-topology-streaming.c`
79 +- Cloud topology service evidence recorded in SOW-0021.
80 +
81 +Current state:
82 +
83 +- SOW-0021 records that `merge_identity` is per actor type and does not define shared identity classes across producers.
84 +- SOW-0021 records that evidence `match_columns` preserve exact relationship details but do not declare endpoint replacement across topology payloads.
85 +- SOW-0021 records the user decision to split this into SOW-0023.
86 +
87 +Risks:
88 +
89 +- False positive matches could collapse unrelated actors.
90 +- False negative matches could duplicate actors that represent the same entity.
91 +- NAT, load balancers, address reuse, namespaces, and reused MAC/IP identities can make simple exact matching unsafe.
92 +- Matching strategies can leak sensitive infrastructure identities if durable artifacts include raw examples.
93 +
94 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
95 +
96 +Status: unblocked for netdata-side contract and network-connections producer
97 +implementation. Cloud and frontend implementation are delegated through the
98 +handoff artifacts created by this SOW.
99 +
100 +Problem / root-cause model:
101 +
102 +- The compact topology schema has enough identity to aggregate within producer-defined types, but it does not yet define how Cloud should reconcile actors across different producer domains.
103 +- The schema also lacked generic link-layout force/distance tokens, causing dense or weak semantic relationships to use the same graph forces as strong relationships.
104 +- Network-connections currently uses one graph link family for process-to-endpoint socket links, which prevents the UI and aggregator from distinguishing local resolved links from correlation links.
105 +
106 +Evidence reviewed:
107 +
108 +- SOW-0021 reviewer findings and user decision notes.
109 +- Current topology schema identity and evidence match-column fields.
110 +- Current topology schema link presentation fields: color, opacity, line style, width, curve, arrow, variable, and hover, with no layout strength/distance token.
111 +- Current network-viewer socket evidence and link type shape.
112 +- Current SNMP actor identity arrays for chassis, MAC, IP, and sys-name.
113 +- Current streaming actor identity through machine GUID and node id.
114 +
115 +Affected contracts and surfaces:
116 +
117 +- `netdata.topology.v1` schema.
118 +- Cloud topology service aggregation algorithm.
119 +- Topology producers for network-connections, SNMP/L2, streaming, vSphere, and future topology domains.
120 +- Cloud frontend behavior when merged actors replace endpoint actors.
121 +- Cloud frontend force-layout behavior for dense, weak, ownership, and partial-correlation links.
122 +
123 +Existing patterns to reuse:
124 +
125 +- Actor `identity`, `merge_identity`, and `parent_identity`.
126 +- Evidence `match_columns`.
127 +- Link direction and aggregation semantics.
128 +- Type-level presentation tokens and legend entries from SOW-0021.
129 +- Compact tables for high-cardinality rows.
130 +
131 +Risk and blast radius:
132 +
133 +- High: incorrect matching can materially change topology meaning.
134 +- High: Cloud aggregation behavior changes across all topology kinds.
135 +- Medium: producer schemas may need new typed identity declarations.
136 +- Medium: layout tokens influence graph readability but must not leak raw frontend physics into producer payloads.
137 +
138 +Sensitive data handling plan:
139 +
140 +- Do not copy raw IP addresses, MAC addresses, hostnames, machine GUIDs, node IDs, account IDs, customer identifiers, credentials, secrets, API tokens, bearer tokens, session cookies, SNMP communities, or private topology examples into this SOW, specs, docs, skills, code comments, commits, or PR text.
141 +- Use sanitized synthetic fixtures and placeholder identifiers.
142 +- Keep any raw captured payloads under `.local/` only.
143 +
144 +Implementation plan:
145 +
146 +1. Define the netdata-side schema contract for generic correlation rules, pure correlation actors, points, claims, actions, priorities, and output link types.
147 +2. Define generic link layout tokens for all link types.
148 +3. Update the Go topology v1 model and semantic validator.
149 +4. Update the topology spec, developer guide, implementation scope, and
150 + developer `project-create-topology` skill.
151 +5. Update fixtures so validators exercise correlation rules and link-layout tokens.
152 +6. Implement network-connections link taxonomy and correlation rows.
153 +7. Create Cloud aggregator and UI force-layout handoff artifacts for their
154 + owning workers.
155 +
156 +Validation plan:
157 +
158 +- Service-level fixtures for match, no match, ambiguous match, and unsafe match.
159 +- Schema validation for any new identity declarations.
160 +- Payload-size checks for added declarations.
161 +- Same-failure search across topology producers.
162 +- Netdata-side JSON Schema validation for correlation objects and link layout tokens.
163 +- Go semantic validation for correlation rule references, point/claim table shape, rule key columns, and link-layout token parity.
164 +- Go semantic validation must require only the key columns for rules actually
165 + referenced by each correlation table, not every column for every defined rule.
166 +
167 +Artifact impact plan:
168 +
169 +- AGENTS.md: likely unaffected.
170 +- Runtime project skills: likely unaffected unless topology workflow changes.
171 +- Specs: update topology function schema spec.
172 +- End-user/operator docs: update topology developer guide if producer contract changes.
173 +- Runtime project skills: update `project-create-topology` when producer
174 + authoring workflow changes.
175 +- End-user/operator skills: not affected unless public operator workflows
176 + change.
177 +- JSON Schema: update `FUNCTION_TOPOLOGY_SCHEMA.json`.
178 +- Go producer helper: update `src/go/pkg/topology/v1`.
179 +- Function validation fixtures: update topology v1 fixtures.
180 +- SOW lifecycle: this SOW tracks the SOW-0021 split decision.
181 +
182 +Open-source reference evidence:
183 +
184 +- No external open-source implementation was used as normative evidence for
185 + this schema contract. The contract is driven by Netdata producer semantics,
186 + existing topology payload requirements, and the Cloud aggregator handoff.
187 +
188 +Open decisions:
189 +
190 +- None for the Netdata-side SOW-0023 scope. Cross-kind identity policy remains
191 + tracked by SOW-0002, table/modal composition remains tracked by SOW-0022, and
192 + vSphere migration remains tracked by SOW-0024.
193 +
194 +## Correlation Contract
195 +
196 +The production schema must describe producer-visible correlation facts only. It
197 +must not expose aggregator internals. The aggregator may maintain any internal
198 +state needed, but the final emitted topology is a normal `netdata.topology.v1`
199 +payload with actors, links, evidence, tables, overlays, presentation, stats, and
200 +diagnostics as appropriate.
201 +
202 +Correlation points:
203 +
204 +- are pure topology actors;
205 +- have normal actor types, presentation, labels, legend entries, and links;
206 +- remain visible when unmatched, partial, or ambiguous;
207 +- disappear only when an exact unambiguous absorb rule resolves them.
208 +
209 +Correlation rules:
210 +
211 +- are declared by producers under `data.correlation.rules`;
212 +- are generic and topology-agnostic;
213 +- build exact keys from declarative column/literal templates;
214 +- have a `priority` so exact rules can run before broader/partial rules;
215 +- have a `key_space` to avoid accidental matches between unrelated domains;
216 +- have `action: absorb` or `action: link`;
217 +- list pure `point_actor_types`;
218 +- may list `claim_actor_types` that can satisfy a point;
219 +- may list `correlation_link_types` that connect real actors to correlation actors;
220 +- state an `output_link_type` for final rewritten or partial links.
221 +
222 +Correlation tables:
223 +
224 +- `data.correlation.points` contains actor refs for pure correlation actors plus
225 + rule id and key columns.
226 +- `data.correlation.claims` contains actor refs for real actors plus rule id and
227 + key columns.
228 +- A real actor can claim many keys without bloating the actor row itself.
229 +- A correlation actor can have several point rows for aliases such as NAT-derived
230 + additional keys.
231 +
232 +Actions:
233 +
234 +- `absorb`: exact, unambiguous matches remove all matched correlation actors from
235 + the final aggregated output and rewire incident correlation links to the
236 + matched real actors using the rule's `output_link_type`.
237 +- `link`: broader or partial matches keep the correlation actor visible and emit
238 + or preserve a weak semantic link to the matched actor using the rule's
239 + `output_link_type`.
240 +
241 +No match leaves the correlation actor visible. Ambiguous matches must not be
242 +guessed; they remain unresolved and the aggregator records diagnostics.
243 +
244 +The aggregator must stay agnostic. It should not need releases to learn new
245 +strategy names such as `ip_port`, `mac`, or `vsphere_moid`. It should build keys
246 +from declared columns and literals, normalize by column type, respect priority
247 +and key space, and apply the declared action.
248 +
249 +## Link Layout Contract
250 +
251 +Link layout is generic for all link types. It is not special to correlation.
252 +
253 +Every link type may define:
254 +
255 +- `types.link_types.<id>.presentation.layout.strength`
256 +- `types.link_types.<id>.presentation.layout.distance`
257 +
258 +Allowed strength tokens:
259 +
260 +- `weakest`
261 +- `weaker`
262 +- `normal`
263 +- `stronger`
264 +- `strongest`
265 +
266 +Allowed distance tokens:
267 +
268 +- `closest`
269 +- `closer`
270 +- `normal`
271 +- `farther`
272 +- `farthest`
273 +
274 +These are UI-owned relative tokens, not numeric force values. Producers use them
275 +to classify relationship strength and preferred separation:
276 +
277 +- ownership/containment: stronger + closer;
278 +- resolved normal dependency/flow: normal + normal;
279 +- local noise, dense mesh, inferred, or weak evidence: weaker/weakest +
280 + farther/farthest;
281 +- partial or cross-topology correlation links: weaker/weakest + farther.
282 +
283 +The legend must reflect visible semantic differences introduced by these link
284 +types. The UI must not infer forces from topology kind or actor names.
285 +
286 +## Network-Connections Required Shape
287 +
288 +Network-connections should use three graph-link families:
289 +
290 +1. Node-to-process ownership links that keep the graph clustered by node.
291 +2. Local process-to-process links when both endpoints are already resolved.
292 +3. Process-to-correlation-endpoint links for unresolved or cross-node remote
293 + socket endpoints.
294 +
295 +Socket tuple interpretation:
296 +
297 +- outbound: the process claims `protocol + local_ip + local_port`; the
298 + correlation endpoint points at `protocol + remote_ip + remote_port`;
299 +- inbound: the process claims the local destination tuple; the correlation
300 + endpoint points at the remote source tuple;
301 +- local: both process actors are already identified and should use a resolved
302 + local process-to-process link;
303 +- listen: no remote correlation point exists.
304 +
305 +Exact cross-node absorb example:
306 +
307 +1. Node A emits `process-a -> correlation-endpoint(server-ip:server-port)`.
308 +2. Node B emits `process-b` claiming `server-ip:server-port`.
309 +3. Aggregation removes the matched correlation endpoint and rewires the link to
310 + `process-a -> process-b`.
311 +
312 +Partial example:
313 +
314 +1. Node A emits `process-a -> correlation-endpoint(protocol:ip:port)`.
315 +2. Node B does not have a matching process claim, but emits a broader node/IP
316 + claim through a lower-priority `link` rule.
317 +3. Aggregation keeps the correlation endpoint visible and links it to Node B:
318 + `process-a -> correlation-endpoint(protocol:ip:port) -> node-b`.
319 +
320 +## Concrete Change Inventory
321 +
322 +Netdata repository changes required by this SOW:
323 +
324 +- Extend `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` with:
325 + - `data.correlation`;
326 + - `correlation.rules`;
327 + - compact `correlation.points` and `correlation.claims` tables;
328 + - `correlation_rule.action`, `priority`, `key_space`, `key`,
329 + `point_actor_types`, `claim_actor_types`, `correlation_link_types`, and
330 + `output_link_type`;
331 + - declarative `correlation_key_part` column/literal templates;
332 + - `link_type.presentation.layout.strength` and `.distance` tokens.
333 +- Extend `src/go/pkg/topology/v1` structs and semantic validation with:
334 + - correlation data types;
335 + - point/claim table validation;
336 + - rule reference validation against actor/link types;
337 + - rule key column validation;
338 + - link layout token validation and schema-token parity tests.
339 +- Update fixtures under `src/go/tools/functions-validation/fixtures/topology-v1/`
340 + with at least one correlation rule and link layout tokens.
341 +- Update `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`.
342 +- Update `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`.
343 +- Update `.agents/sow/specs/topology-function-schema.md`.
344 +- Update `.agents/skills/project-create-topology/SKILL.md` with topology
345 + correlation and layout authoring guidance.
346 +
347 +Producer changes implemented or scoped after the contract:
348 +
349 +- Network-viewer:
350 + - split link types into ownership, local/resolved socket, and correlation
351 + socket links;
352 + - emit correlation actor rows for remote endpoints;
353 + - emit correlation point rows for unresolved remote endpoint keys;
354 + - emit process claim rows for locally owned socket keys;
355 + - emit legend entries for ownership, resolved socket, and correlation socket;
356 + - assign layout tokens to each link type.
357 +- Streaming:
358 + - likely no correlation actors for the current topology; verify whether
359 + machine/node identity claims should be emitted for future cross-kind
360 + resolution.
361 +- SNMP/L2:
362 + - preserve semantic link types; evaluate whether inferred endpoints should
363 + emit MAC/IP/chassis correlation points or claims.
364 +- vSphere:
365 + - remains tracked by SOW-0024; when migrated, stable object identity can be
366 + expressed as claims or normal actor identity depending on topology scope.
367 +
368 +External repository handoffs required:
369 +
370 +- Cloud topology service:
371 + - implement rule parsing and generic key building;
372 + - apply absorb/link actions;
373 + - keep unmatched/ambiguous correlation actors visible;
374 + - add match/no-match/partial/ambiguous fixtures;
375 + - emit diagnostics without exposing internal states in the output graph.
376 +- Cloud frontend:
377 + - honor link layout strength/distance tokens for all link types;
378 + - show semantic correlation actor/link legend entries;
379 + - avoid topology-name-specific force/layout hardcoding.
380 +
381 +## Implications And Decisions
382 +
383 +1. User decision from SOW-0021: cross-payload actor reconciliation should be solved, but it can be split into SOW-0023.
384 +2. User decision from SOW-0021: execution order is SOW-0021, then SOW-0023, then SOW-0022.
385 +3. User decision from 2026-05-10: correlation points are pure topology actors, not flags on real actors. Exact resolved correlation may remove correlation actors from the aggregated output, but internal aggregator states such as absorbed/candidate/rewrite-plan must not be exposed in the production schema or final UI payload.
386 +4. User decision from 2026-05-10: the schema should define the producer-visible correlation contract for independently produced topology maps of the same kind. The aggregator may choose any internal indexing, matching, rewrite, and diagnostic implementation as long as the final output is a normal topology payload.
387 +5. User decision from 2026-05-10: visible partial or cross-topology correlation links should be weaker in the force layout so related topology clusters stay readable instead of blending into one dense actor soup.
388 +6. User decision from 2026-05-10: link layout force classification must be generic for all link types, not special-cased to correlation links. It should use five-step token scales with `normal` in the middle: strength `weakest`, `weaker`, `normal`, `stronger`, `strongest`; distance `closest`, `closer`, `normal`, `farther`, `farthest`.
389 +7. User decision from 2026-05-10: links between a real actor and a correlation actor must be marked with a distinct semantic link type even in a single unaggregated topology. The UI and aggregator must not infer correlation-link behavior from actor names or topology kind.
390 +8. User decision from 2026-05-10: network-connections graph links should be split into three semantic families: node-to-process ownership links that keep the graph together; local process-to-process links for already resolved local sockets; and process-to-correlation-endpoint links for unresolved/correlatable remote socket endpoints.
391 +
392 +## Plan
393 +
394 +1. Complete netdata-side contract artifacts: schema, Go types/validator, docs,
395 + spec, skill, and fixtures.
396 +2. Validate the contract with JSON Schema, Go tests, function-validation
397 + fixtures, and SOW audit.
398 +3. Implement network-viewer producer migration in SOW-0023.
399 +4. Create or update Cloud service and Cloud frontend handoff documents after the
400 + contract is validated.
401 +5. Implement Cloud aggregator and UI behavior in their owning repositories.
402 +
403 +## Execution Log
404 +
405 +### 2026-05-09
406 +
407 +- Created as follow-up from SOW-0021 decision discussion.
408 +
409 +### 2026-05-10
410 +
411 +- Recorded user decisions that correlation points are pure topology actors, that
412 + aggregator internals must not leak into the schema or final UI payload, and
413 + that the schema should describe producer-visible correlation rules only.
414 +- Recorded user decisions for generic declarative correlation keys, priorities,
415 + `absorb` and `link` actions, exact and partial correlation behavior, NAT/alias
416 + enrichment through additional keys, semantic correlation link types, and
417 + five-step link layout strength/distance tokens.
418 +- Added netdata-side schema/Go/docs/spec/skill work items to this SOW.
419 +- Implemented the network-viewer producer split into `ownership`, `socket`,
420 + `endpoint_socket`, and `correlated_socket` link types.
421 +- Implemented network-viewer `data.correlation.rules`, `points`, and `claims`
422 + emission for socket tuple correlation.
423 +- Created Cloud frontend and Cloud topology service handoff artifacts so their
424 + owning workers can port the correlation and layout contract.
425 +
426 +## Validation
427 +
428 +Acceptance criteria evidence:
429 +
430 +- Identity/evidence inventory is recorded in `## Correlation Contract`,
431 + `## Network-Connections Required Shape`, and `## Concrete Change Inventory`.
432 +- Netdata-side schema contract now defines generic correlation rules, compact
433 + points/claims tables, declarative key parts, absorb/link actions, priorities,
434 + point/claim actor types, correlation link types, output link type, and
435 + five-step link layout strength/distance tokens.
436 +- Cloud aggregator and Cloud frontend implementation requirements are scoped in
437 + `## Concrete Change Inventory` and handed off to their owning workers.
438 +- Network-connections now emits semantic ownership/resolved/correlation link
439 + types, correlation endpoint actors, `data.correlation.points`, and
440 + `data.correlation.claims`.
441 +
442 +Tests or equivalent validation:
443 +
444 +- `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
445 +- `jq empty src/go/tools/functions-validation/fixtures/topology-v1/network-connections.json` passed.
446 +- `go test ./pkg/topology/v1 ./tools/functions-validation/validate` passed from `src/go`.
447 +- Function validator passed for every fixture under
448 + `src/go/tools/functions-validation/fixtures/topology-v1/*.json`.
449 +- C syntax-only validation for
450 + `src/collectors/network-viewer.plugin/network-viewer.c` passed using the
451 + local `build/compile_commands.json` command with `-fsyntax-only`.
452 +- `ninja -C build network-viewer.plugin` could not run because the local build
453 + directory is root-owned and Ninja cannot create `.ninja_lock` or update the
454 + build log.
455 +- `git diff --check` passed.
456 +- `.agents/sow/audit.sh` exited successfully. It still reports the repository's
457 + existing non-project skill classification warning, which is outside this SOW;
458 + `project-create-topology` is classified as a runtime project skill.
459 +
460 +Real-use evidence:
461 +
462 +- A local bearer-protected Agent was queried through the token-safe direct-agent
463 + wrapper. The aggregated network-connections topology response returned
464 + `netdata.topology.v1` with 108 actors, 144 links, 96 correlation points, 179
465 + correlation claims, and 45 ownership links.
466 +- The live payload defined `endpoint_socket` as `strength: weakest` and
467 + `distance: normal`, `correlated_socket` as `strength: weakest` and
468 + `distance: farthest`, `socket` as `strength: stronger` and `distance:
469 + farther`, and `ownership` as dotted/faded normal-distance graph-coherence
470 + links.
471 +- The same live payload confirmed the `socket_exact` rule consumes
472 + `endpoint_socket` and emits `correlated_socket`, matching the intended
473 + single-node versus aggregated-layout split.
474 +- Full `ninja -C build network-viewer.plugin` validation remains blocked by the
475 + root-owned local build directory, but schema validation, Go tests, semantic
476 + fixture validation, C syntax validation, and live Function output validation
477 + all passed.
478 +
479 +Reviewer findings:
480 +
481 +- No external read-only reviewer was run for this close. User live testing found
482 + the remaining force-layout stretch for high-fanout endpoint leaves; analysis
483 + traced that to Cloud frontend physics, not to the Netdata producer contract,
484 + and it is mapped to the frontend polishing work.
485 +
486 +Same-failure scan:
487 +
488 +- Current same-failure class is schema ambiguity rather than a runtime crash.
489 + The netdata-side validator now rejects missing correlation key columns,
490 + unknown rule references, unknown actor/link type references, and invalid link
491 + layout tokens.
492 +- The validator also covers the multi-rule case where a table references only
493 + one of several defined rules, so unrelated rule key columns are not forced
494 + into every correlation table.
495 +
496 +Sensitive data gate:
497 +
498 +- This SOW currently contains only sanitized generic examples.
499 +
500 +Artifact maintenance gate:
501 +
502 +- AGENTS.md: updated the project skills index so topology authoring guidance is
503 + a runtime project skill, not a public operator skill.
504 +- Runtime project skills: `.agents/skills/project-create-topology/SKILL.md`
505 + added as the developer-facing topology authoring workflow.
506 +- Specs: `.agents/sow/specs/topology-function-schema.md` updated with
507 + correlation and link-layout contracts.
508 +- End-user/operator docs: `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`,
509 + `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`, and
510 + `src/plugins.d/FUNCTION_UI_REFERENCE.md` updated.
511 +- End-user/operator skills: `query-netdata-agents` and `query-netdata-cloud`
512 + were audited for developer-contract leakage. The misplaced public
513 + `create-topology` developer skill was removed from the public skill tree, and
514 + public query skills now state that developer validation recipes belong in
515 + project skills.
516 +- SOW lifecycle: SOW moved from pending to current for implementation and is
517 + marked `completed`; it is moved to `.agents/sow/done/` in the same commit as
518 + the implementation.
519 +
520 +Specs update:
521 +
522 +- `.agents/sow/specs/topology-function-schema.md` updated.
523 +
524 +Project skills update:
525 +
526 +- Added `.agents/skills/project-create-topology/SKILL.md`.
527 +- Moved topology developer how-tos from the public skill tree into
528 + `.agents/skills/project-create-topology/how-tos/`.
529 +- Added a developer how-to for verifying local network-connections layout
530 + tokens and correlation rule wiring with token-safe direct-agent wrappers.
531 +
532 +End-user/operator docs update:
533 +
534 +- Updated topology developer guide, topology implementation scope, and Function
535 + UI reference.
536 +
537 +End-user/operator skills update:
538 +
539 +- Removed the developer-facing `docs/netdata-ai/skills/create-topology/` skill
540 + and its `.agents/skills/create-topology` symlink.
541 +- Removed topology producer authoring links from public query-topology guides.
542 +- Removed the collector-author implementation section from the public Cloud
543 + query-functions guide.
544 +- Moved skill-verification seed question lists out of public skill directories
545 + into `.agents/skill-verification/` and updated the pending verification
546 + harness SOW.
547 +- Updated public query skills to keep future how-tos operator-facing.
548 +
549 +Lessons:
550 +
551 +- Correlation actors and correlation rules need a contract separate from actor
552 + identity. Actor `merge_identity` is not enough when one real actor owns many
553 + correlation keys.
554 +- Link force/layout must be type-level and generic. Correlation links exposed
555 + this need, but dense local meshes and ownership links need it too.
556 +
557 +Follow-up mapping:
558 +
559 +- Cloud service implementation is handed off through the service SOW created by
560 + this SOW.
561 +- Cloud frontend force-layout and correlation rendering implementation is handed
562 + off through the frontend TODO created by this SOW.
563 +- Cross-kind identity policy remains tracked by SOW-0002.
564 +- Table/modal composition remains tracked by SOW-0022.
565 +- vSphere migration remains tracked by SOW-0024.
566 +
567 +## Outcome
568 +
569 +Completed. Netdata now has a generic producer-visible topology correlation
570 +contract, generic link layout strength/distance tokens, semantic validation for
571 +the new contract, updated schema/spec/docs/developer skill artifacts, and a
572 +network-connections producer that emits ownership, local socket, unresolved
573 +endpoint socket, and post-correlation socket semantics separately.
574 +
575 +The local Agent runtime payload matches the intended split. Remaining visual
576 +layout stretch around high-fanout endpoint leaves is a Cloud frontend physics
577 +polish issue, not a backend payload issue.
578 +
579 +## 2026-05-10 Aggregated Network-Connections UI Alignment
580 +
581 +User-visible issue:
582 +
583 +- Aggregated network-connections process actors lost resize behavior and port
584 + bullets because the producer only enabled process bullets in detailed mode.
585 +- Endpoint links were rendered as dotted/secondary links even though they are
586 + the main unresolved network dependencies in a single-node view.
587 +- Node-to-process links rendered too prominently even though they only keep the
588 + graph coherent and do not represent network traffic.
589 +
590 +Root cause:
591 +
592 +- Aggregated mode intentionally omits detailed socket evidence, but process port
593 + bullets were still defined only from socket evidence rows.
594 +- The UI treated every bullet source row as one visible bullet and had no
595 + `value_column` to say that one compact row represents multiple sockets.
596 +- Process sizing still relied on graph degree instead of the producer's
597 + `socket_count` metric.
598 +
599 +Implemented contract and producer changes:
600 +
601 +- Added optional numeric `ports.sources[].value_column` to the schema, Go model,
602 + semantic validator, spec, and developer guidance.
603 +- Network-connections now emits an actor-owned `socket_ports` inventory table
604 + with `actor`, `port`, `protocol`, `direction`, and `socket_count`.
605 +- Process actor presentation now uses
606 + `size: {"mode": "metric", "metric_column": "socket_count"}` and
607 + `ports.sources[]` from `socket_ports` with `value_column: "socket_count"`.
608 +- Link presentation was aligned with the intended semantics:
609 + `endpoint_socket` is solid/colored/thin/weakest/normal-distance,
610 + `correlated_socket` is solid/colored/thin/weakest/farthest,
611 + `socket` is gray/thin/stronger/farther and variable by `socket_count`, and
612 + `ownership` is dotted/faded/dim/thin/normal/normal.
613 +
614 +Implemented frontend changes:
615 +
616 +- The v1 port-bullet decoder preserves `value_column` and sums duplicate bullet
617 + keys.
618 +- The v1 renderer adapter passes `size_metric_column` to the graph renderer.
619 +- The graph renderer uses actor metric sizing, weighted port capacity, weighted
620 + visible bullet count, and correct overflow without expanding unbounded data.
621 +- v1-derived socket bullets default to active topology bullets so process rings
622 + are visible even when there is no legacy SNMP-style port status.
623 +
624 +Validation added or rerun:
625 +
626 +- `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
627 +- `jq empty src/go/tools/functions-validation/fixtures/topology-v1/network-connections.json` passed.
628 +- `go test ./pkg/topology/v1 ./tools/functions-validation/validate` passed from
629 + `src/go`.
630 +- Function validator passed for
631 + `src/go/tools/functions-validation/fixtures/topology-v1/network-connections.json`.
632 +- C syntax-only validation passed for
633 + `src/collectors/network-viewer.plugin/network-viewer.c` using
634 + `-Ibuild -include build/config.h`.
635 +- Cloud frontend targeted Jest tests passed:
636 + `portBullets.test.js`, `buildRenderPresentation.test.js`,
637 + `portUtils.test.js`, and `useForceSimulation.test.js`.
638 +
639 +## 2026-05-10 Endpoint vs Correlated Socket Layout Split
640 +
641 +User-visible issue:
642 +
643 +- Unresolved endpoint links were set to `distance: farthest`, which makes
644 + single-node network-connections maps zoom out too much and shrink the useful
645 + process cluster.
646 +- The same link type was also planned as the aggregator's consumed correlation
647 + link, where `farthest` is appropriate after cross-payload absorption because
648 + it keeps independent topology clusters from blending.
649 +
650 +Implemented changes:
651 +
652 +- Split the overloaded network-connections link semantics:
653 + `endpoint_socket` is the visible process-to-endpoint unresolved link, while
654 + `correlated_socket` is the aggregator output link after exact absorption.
655 +- `endpoint_socket` uses solid/colored/thin, `strength: weakest`,
656 + `distance: normal`.
657 +- `correlated_socket` uses solid/colored/thin, `strength: weakest`,
658 + `distance: farthest`, and can vary by `socket_count`.
659 +- The `socket_exact` rule now lists `correlation_link_types:
660 + ["endpoint_socket"]` and `output_link_type: "correlated_socket"`.
661 +- Fixtures, developer docs, and the topology developer skill were updated to
662 + use the split names.
663 +
664 +## Lessons Extracted
665 +
666 +- Pure correlation actors keep the producer contract simple: the aggregator can
667 + absorb, link, or leave them visible without exposing internal matching state.
668 +- Link layout tokens are necessary but not sufficient for final graph polish:
669 + force-directed renderers may still need frontend-side handling for high-fanout
670 + leaves, collision radius, zoom-to-fit, and initial layout seeding.
671 +- Aggregated process port bullets need an actor-owned inventory table with a
672 + numeric value column. Counting compact rows is wrong when one row represents
673 + many sockets.
674 +- Distinguishing unresolved endpoint links from aggregated correlated links is
675 + required. Single-node endpoint leaves should not use the same farthest layout
676 + token as cross-payload correlation output.
677 +
678 +## Followup Mapping
679 +
680 +- Cloud topology service generic rule-based correlation is handed off through
681 + the Cloud service worker SOW.
682 +- Cloud frontend link layout token handling and correlation rendering are handed
683 + off through the Cloud frontend worker TODO.
684 +- The high-fanout endpoint leaf layout stretch observed during live testing is
685 + mapped to the Cloud frontend polishing work; the live Netdata payload already
686 + emits `endpoint_socket` as normal-distance and weakest-strength.
687 +- Cross-kind identity policy remains tracked by SOW-0002.
688 +- Table/modal composition remains tracked by SOW-0022.
689 +- vSphere topology v1 migration remains tracked by SOW-0024.
690 +
691 +## Regression Log
692 +
693 +None yet.
694 +
695 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0025-20260511-network-connections-modal-product-composition.md new
+500
@@ -0,0 +1,500 @@
1 +# SOW-0025 - Network Connections Modal Product Composition
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +Sub-state: completed after backend producer repair, schema/developer
8 +documentation updates, protocol validation, schema validation, and SOW audit.
9 +
10 +## Requirements
11 +
12 +### Purpose
13 +
14 +Make `topology:network-connections` actor modals useful for sysadmins, DevOps engineers, and SREs who need to understand process/socket dependencies quickly and accurately.
15 +
16 +### User Request
17 +
18 +The user reported that network-connections modals currently show `Connections`, `Ports`, and, in detailed mode, `Socket Evidence`, which creates three confusing views over the same socket facts. The user wants the work handled mechanically and separately from SNMP and streaming.
19 +
20 +### Assistant Understanding
21 +
22 +Facts:
23 +
24 +- The v1 schema can express actor labels, modal sections, projections, owner filters, and table columns without duplicating rows.
25 +- Current network-connections modal recipes are generated in `src/collectors/network-viewer.plugin/network-viewer.c`.
26 +- Current actor modals are declared generically for actor types and add:
27 + - `Connections` from `links`, excluding `ownership`;
28 + - `Ports` from actor table `socket_ports` when port bullets are enabled;
29 + - `Socket Evidence` from `evidence.socket` when detailed evidence exists.
30 +- `network-viewer.c` currently emits actor labels and typed actor columns such as process, username, command line, namespace, local IP, address space, socket count, PID, UID, and netns inode.
31 +
32 +Inferences:
33 +
34 +- The schema contract is not the main problem for this function. The issue is table product design: the current tabs reflect implementation sources instead of the troubleshooting questions users ask.
35 +- `Ports` and `Socket Evidence` are not independently useful as top-level tabs unless their relationship to the connection rows is obvious.
36 +
37 +Unknowns:
38 +
39 +- Whether the frontend can already promote selected actor labels into the modal identification/header area, or whether this requires a frontend contract extension in addition to producer recipe changes.
40 +- Whether all old-network-connections modal columns can be reconstructed from current v1 rows without adding new canonical columns.
41 +
42 +### Acceptance Criteria
43 +
44 +- A complete inventory exists for network-connections actor modal facts: actors, actor labels, links, `socket_ports`, and `evidence.socket`.
45 +- The SOW maps every useful old modal/table field to a v1 source, or records the missing canonical field to add.
46 +- The final modal design avoids three duplicate tabs for the same socket relationship.
47 +- Process, endpoint, and node/self actors each have an explicit useful modal shape.
48 +- The actor identification area has a declared source for important labels, not only the generic `Labels` tab.
49 +- Backend producer changes, schema/doc updates if needed, frontend TODO, and validation plan are completed before this SOW closes.
50 +
51 +## Analysis
52 +
53 +Sources checked:
54 +
55 +- `src/collectors/network-viewer.plugin/network-viewer.c:2505` modal column helper.
56 +- `src/collectors/network-viewer.plugin/network-viewer.c:2543` selected-side socket endpoint projection.
57 +- `src/collectors/network-viewer.plugin/network-viewer.c:2647` actor modal emission.
58 +- `src/collectors/network-viewer.plugin/network-viewer.c:2667` current `Connections` section.
59 +- `src/collectors/network-viewer.plugin/network-viewer.c:2710` current `Ports` section.
60 +- `src/collectors/network-viewer.plugin/network-viewer.c:2738` current `Socket Evidence` section.
61 +- `src/collectors/network-viewer.plugin/network-viewer.c:2867` process actor type enables port bullets from `socket_ports`.
62 +- `src/collectors/network-viewer.plugin/network-viewer.c:2937` `socket_ports` table type.
63 +- `src/collectors/network-viewer.plugin/network-viewer.c:2962` `actor_labels` table type.
64 +- `.agents/sow/specs/topology-function-schema.md:351` network-connections actor/link/socket-port schema notes.
65 +
66 +Current state:
67 +
68 +- The current modal exposes source tables directly:
69 + - graph links for high-level connections;
70 + - actor-owned socket-port inventory;
71 + - detailed socket evidence rows.
72 +- This is technically correct but product-confusing because a user sees multiple tabs that appear to describe the same thing with different missing columns.
73 +- Actor identification facts exist in actor labels and typed actor columns, but the modal header/identification area is not using selected important labels.
74 +
75 +Available facts to inventory:
76 +
77 +- Node/self actor:
78 + - `display_name`, `type`, `hostname`, `machine_guid`, `socket_count`, `local_ip_count`.
79 +- Process actor:
80 + - `display_name`, `type`, `process`, `username`, `cmdline`, `namespace_type`, `local_ip`, `local_address_space`, `socket_count`, optional `pid`, optional `uid`, optional `net_ns_inode`.
81 +- Endpoint actor:
82 + - `display_name`, `type`, `ip`, `address_space`, `socket_count`.
83 +- Graph links:
84 + - opposite actor, link type, protocol, direction, state, socket count, and link-level metrics.
85 +- `socket_ports`:
86 + - actor, port, protocol, direction, socket count.
87 +- `evidence.socket`:
88 + - exact source/destination actors and socket endpoint details, protocol, direction, state, RTT/retransmission fields when available.
89 +
90 +Field inventory and mapping:
91 +
92 +| Useful old/current fact | Existing v1 source | Gap / action |
93 +|---|---|---|
94 +| Actor display name | `actors.display_name`, `actor_labels.display_name` | Available. Use in actor label policy and modal header. |
95 +| Hostname | `actors.hostname`, `actor_labels.hostname` | Available for self actor. Promote to modal header. |
96 +| Machine GUID | `actors.machine_guid`, `actor_labels.machine_guid` | Available. Keep in labels; promote only if useful for debugging. |
97 +| Process name | `actors.process`, `actor_labels.process` | Available. Promote to process modal header. |
98 +| Process user | `actors.username`, `actor_labels.username` | Available. Promote to process modal header. |
99 +| Process command line | `actors.cmdline`, `actor_labels.cmdline` | Available. Promote to process modal header or expanded header; sensitive but allowed by Function classification. |
100 +| Namespace type | `actors.namespace_type`, `actor_labels.namespace_type` | Available. Promote to process modal header. |
101 +| Process local IP/address space | `actors.local_ip`, `actors.local_address_space`, labels | Available. Promote when present. |
102 +| PID/UID/netns inode | typed actor columns and labels when `processes:by_pid` | Available. Header for `by_pid`, labels for `by_name`. |
103 +| Endpoint IP/address space | `actors.ip`, `actors.address_space`, labels | Available. Promote to endpoint modal header. |
104 +| Socket count | actor/link/evidence metrics, labels | Available. Promote to header and tables. |
105 +| Direction/protocol/state | `links`, `evidence.socket`, internal `NV_TOPOLOGY_LINK` | Available. Use badges in connection rows. |
106 +| Local/remote IP and port | `evidence.socket` in detailed mode; internal `NV_TOPOLOGY_LINK` in all modes | Missing from aggregated v1 modal source. Add one relationship-summary table from existing internal rows. |
107 +| Server port/service name | internal `NV_TOPOLOGY_LINK.remote_port` or `local_port`, `port_name` | Missing from emitted v1 tables except exact socket evidence has ports but not `port_name`. Add to relationship summary; add `service_name` to evidence if needed. |
108 +| RTT / receiver RTT / retransmissions | `links` and `evidence.socket` metrics | Available. Relationship summary should expose the same metrics per summary row. |
109 +| Local process ownership / node keeps graph coherent | `ownership` links | Available. Self modal should use this as a process list; process modal should not show it as a network connection. |
110 +
111 +Important evidence:
112 +
113 +- Internal connection summary rows already contain process identity, local/remote IPs, local/remote ports, peer port, protocol, direction, state, address spaces, service name, command line, socket count, and latency/retransmission metrics in `NV_TOPOLOGY_LINK` (`src/collectors/network-viewer.plugin/network-viewer.c:80`).
114 +- Those internal rows are keyed by process, namespace, local IP, remote IP, protocol, direction, state, local port, and endpoint port (`src/collectors/network-viewer.plugin/network-viewer.c:1283`).
115 +- v1 graph links intentionally collapse those rows by source actor, destination actor, link type, protocol, direction, and state only (`src/collectors/network-viewer.plugin/network-viewer.c:2135`), so port pairing and service name cannot be reconstructed from graph links alone.
116 +- Current modal sections expose `links`, `socket_ports`, and `evidence.socket` as peer sections (`src/collectors/network-viewer.plugin/network-viewer.c:2647`). This mirrors storage tables, not the user's troubleshooting workflow.
117 +- The current schema supports `relationship_table` modal sources and `relationship_summary` table roles, so an aggregated connection-summary table fits the existing contract without inventing a network-connections-specific UI path.
118 +
119 +Target audience and questions:
120 +
121 +- Sysadmin/SRE process drilldown:
122 + - What is this process?
123 + - Which user/cmdline/container/namespace is it?
124 + - What is it listening on?
125 + - What remote endpoints or local processes is it connected to?
126 + - How many sockets are involved?
127 + - Which entries are unresolved correlation endpoints?
128 +- Sysadmin/SRE endpoint drilldown:
129 + - Which processes connect to this endpoint?
130 + - Is it inbound, outbound, listening, local, TCP, UDP?
131 + - Is the endpoint local/private/public?
132 +- Sysadmin/SRE node drilldown:
133 + - Which processes on this node participate in network activity?
134 + - Which local IPs are observed?
135 +
136 +Target product model:
137 +
138 +- Self/node actor modal:
139 + - Header: hostname, observed socket count, local IP count.
140 + - Primary table: `Processes` derived from `ownership` links, showing process actor and socket count.
141 + - No raw socket table; self is a scope/root actor, not a socket endpoint.
142 +- Process actor modal:
143 + - Header: process name/display name, user, namespace type, command line, local IP/address space, socket count, and PID/netns when available.
144 + - Primary table: `Connections`, sourced from one relationship-summary table in aggregated mode and from `evidence.socket` in detailed mode.
145 + - Columns should answer "where does this process connect/listen?": peer actor or endpoint, local endpoint, remote endpoint, service/port, protocol, direction, state, sockets, RTT/retransmissions.
146 + - Port bullets remain visual graph affordances. A separate `Ports` modal tab should not be shown unless it provides a distinct, clearly labeled local-port inventory.
147 +- Endpoint actor modal:
148 + - Header: IP, address space, socket count.
149 + - Primary table: `Processes`, sourced from the same connection rows, showing the local process, local endpoint, endpoint port/service, protocol, direction, state, sockets, RTT/retransmissions.
150 + - No `Ports` tab; the endpoint is already the port/IP drilldown target.
151 +
152 +Risks:
153 +
154 +- Leaving `Connections`, `Ports`, and `Socket Evidence` as peers makes users compare inconsistent row sets and assume data is wrong.
155 +- Hiding process identity under `Labels` makes the modal feel empty even when the payload contains the information.
156 +- Detailed evidence can be high-cardinality; it should be expandable or secondary, not a default duplicated top-level list unless the user explicitly asked for detailed sockets.
157 +- If aggregated mode continues to use only graph links plus `socket_ports`, it cannot show accurate local/remote port pairing. `socket_ports` is actor inventory, not a relationship table.
158 +- If the frontend guesses important labels from names such as `process`, `username`, or `cmdline`, the UI becomes topology-specific and violates the v1 contract that producers define presentation.
159 +
160 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
161 +
162 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Execution Log and Validation sections).
163 +
164 +Problem / root-cause model:
165 +
166 +- The schema migration preserved facts but lost product intent. The producer emits table recipes that mirror internal sources instead of user workflows.
167 +- The actor identification area is empty because important labels are only exposed through the `actor_labels` table and not selected for header display.
168 +
169 +Evidence reviewed:
170 +
171 +- Current network-connections modal sections are emitted in `src/collectors/network-viewer.plugin/network-viewer.c:2647-2760`.
172 +- Current network-connections actor labels are emitted in `src/collectors/network-viewer.plugin/network-viewer.c:2029-2102`.
173 +- Current network-connections table-type presentation for `socket_ports` and `actor_labels` is emitted in `src/collectors/network-viewer.plugin/network-viewer.c:2937-2979`.
174 +
175 +Affected contracts and surfaces:
176 +
177 +- Agent Function payload for `topology:network-connections`.
178 +- v1 topology schema only if the identification/header area requires a new contract.
179 +- Cloud frontend actor modal rendering.
180 +- Cloud aggregator table/label preservation.
181 +- Developer guide, durable topology spec, and project topology skill.
182 +
183 +Existing patterns to reuse:
184 +
185 +- `actor_labels` for labels and identity facts.
186 +- Modal sections over existing `links`, `evidence`, and `actor_table` sources.
187 +- `selected_side_endpoint` for socket endpoint rendering.
188 +- `socket_ports` with numeric `socket_count` for aggregated port bullets.
189 +
190 +Risk and blast radius:
191 +
192 +- User-facing modal behavior changes for network-connections only.
193 +- Payload size risk is moderate if detailed evidence is duplicated. This SOW must not duplicate socket rows only for modal display.
194 +- Sensitive data risk is high because process command lines, usernames, and local endpoints are intentionally exposed to authorized users.
195 +
196 +Sensitive data handling plan:
197 +
198 +- Do not copy raw command lines, usernames, bearer tokens, cookies, public IPs from user systems, or customer-identifying endpoint data into durable artifacts.
199 +- Use synthetic examples only.
200 +- Treat all `actor_labels` and socket evidence as topology Function sensitive data.
201 +
202 +Implementation plan:
203 +
204 +1. Inventory the old and current modal fields for node/self, process, and endpoint actors.
205 +2. Define the intended modal composition for each actor type:
206 + - selected identity/header labels;
207 + - one primary connection table;
208 + - optional listening/local-port summary;
209 + - optional detailed evidence expansion/section only when detailed evidence exists and adds information.
210 +3. Add or adjust canonical columns only when the useful field is missing from actors, links, `socket_ports`, or `evidence.socket`.
211 +4. Update `network-viewer.c` modal recipes without duplicating socket evidence.
212 +5. Update docs/spec/skill if the schema contract or recommended network-connections shape changes.
213 +6. Produce a frontend TODO if the header/identification area needs schema support or UI changes.
214 +
215 +Validation plan:
216 +
217 +- Validate generated payload JSON against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
218 +- Run C syntax check for `network-viewer.c`.
219 +- Use local Agent Function output in aggregated and detailed modes.
220 +- Confirm actor modals for self/node, process, and endpoint actors show non-empty identity and a non-duplicative useful connection view.
221 +- Check payload size before/after on a realistic network-connections payload.
222 +
223 +Artifact impact plan:
224 +
225 +- AGENTS.md: likely unaffected.
226 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md` if the recommended network-connections modal shape changes.
227 +- Specs: update `.agents/sow/specs/topology-function-schema.md`.
228 +- End-user/operator docs: likely unaffected unless Function output docs expose examples.
229 +- End-user/operator skills: unaffected.
230 +- SOW lifecycle: close only after integrated local Agent/UI validation or an explicit tracked follow-up SOW exists.
231 +
232 +Open-source reference evidence:
233 +
234 +- Not checked yet. This SOW is about Netdata-specific Function modal semantics; external references may be useful only for general socket table UX and should be recorded if used during analysis.
235 +
236 +Open decisions:
237 +
238 +- Resolved by SOW-0028 and the SOW-0025 decisions below:
239 + - modal identification uses `modal.labels.identification.fields[]`;
240 + - aggregated network-connections uses `data.tables.relationship.connections`;
241 + - detailed network-connections uses exact socket evidence as the primary
242 + modal table.
243 +
244 +## Implications And Decisions
245 +
246 +### Decision 1: Modal Identification/Header Contract
247 +
248 +Evidence:
249 +
250 +- The schema has `presentation.modal.labels` and `presentation.modal.sections`, but no field that selects important labels for the modal header (`src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:1073`).
251 +- The producer already emits the needed labels for self, process, and endpoint actors (`src/collectors/network-viewer.plugin/network-viewer.c:2029`, `src/collectors/network-viewer.plugin/network-viewer.c:2065`, `src/collectors/network-viewer.plugin/network-viewer.c:2101`).
252 +
253 +Options:
254 +
255 +- **A. Extend `presentation.modal.labels` with ordered `summary_keys` / `identity_keys`. Recommended.**
256 + - Pros: small schema change; keeps UI topology-agnostic; reuses `actor_labels`; no row duplication.
257 + - Cons: requires frontend and aggregator to preserve/read the new label presentation field.
258 + - Implication: each actor type can say which label keys appear in the header, while the full `Labels` tab remains complete.
259 +- **B. Add a more general `presentation.modal.identity.fields[]` using modal projections.**
260 + - Pros: most flexible; can pull from actor columns or labels.
261 + - Cons: larger schema/UI work; more ways for producers to create inconsistent header/table semantics.
262 + - Implication: useful later, but heavier than this SOW needs.
263 +- **C. Let the frontend guess from label keys.**
264 + - Pros: no schema change.
265 + - Cons: violates producer-driven presentation; the UI would learn network-connections-specific labels.
266 + - Risk: repeats the old problem in another form.
267 +
268 +Recommendation: **A**. The header is a curated subset of `actor_labels`, not a new data source.
269 +
270 +### Decision 2: Aggregated Mode Connection Detail Source
271 +
272 +Evidence:
273 +
274 +- Internal `NV_TOPOLOGY_LINK` rows preserve aggregated connection detail, including local/remote ports and service name (`src/collectors/network-viewer.plugin/network-viewer.c:80`).
275 +- v1 graph links intentionally remove port pairing from the graph-link key (`src/collectors/network-viewer.plugin/network-viewer.c:2135`).
276 +- `socket_ports` stores only actor-owned port summaries (`src/collectors/network-viewer.plugin/network-viewer.c:2235`), so it cannot answer "which remote endpoint used which port?".
277 +
278 +Options:
279 +
280 +- **A. Add `data.tables.relationship.connections` as a `relationship_summary` table. Recommended.**
281 + - Pros: uses existing internal rows once; restores accurate aggregated modal drilldown; no high-cardinality socket duplication in aggregated mode.
282 + - Cons: increases aggregated payload size by one compact row per internal connection summary.
283 + - Implication: graph links stay small, while modals use a relationship table for precise rows.
284 +- **B. Keep using graph links for aggregated modals.**
285 + - Pros: smallest payload.
286 + - Cons: cannot show local/remote port pairing or service names; leaves the current confusing modal mostly intact.
287 + - Risk: not fit for the requested SRE workflow.
288 +- **C. Use only `socket_ports` as the drilldown.**
289 + - Pros: compact and already emitted.
290 + - Cons: actor inventory is not relationship data; outbound local ephemeral ports and remote server ports get confused.
291 + - Risk: exactly the kind of misleading table the user reported.
292 +
293 +Recommendation: **A**. The summary table is not duplicated presentation data; it is the missing relationship fact plane for aggregated drilldowns.
294 +
295 +### Decision 3: Detailed Mode Modal Shape
296 +
297 +Evidence:
298 +
299 +- Detailed mode already emits `evidence.socket` rows with exact local/remote socket details (`src/collectors/network-viewer.plugin/network-viewer.c:2254`).
300 +- Current detailed modal shows both graph `Connections`, actor `Ports`, and `Socket Evidence` (`src/collectors/network-viewer.plugin/network-viewer.c:2664`, `src/collectors/network-viewer.plugin/network-viewer.c:2706`, `src/collectors/network-viewer.plugin/network-viewer.c:2735`).
301 +
302 +Options:
303 +
304 +- **A. In detailed mode, make exact socket evidence the single primary `Sockets` / `Connections` section. Recommended.**
305 + - Pros: no duplicated tabs; exact rows are shown when the user selected detailed mode; expanded columns can show less-common fields.
306 + - Cons: detailed mode tables can be large, as expected by the mode.
307 + - Implication: hide graph-link and port-summary sections from detailed actor modals unless they add a distinct summary later.
308 +- **B. Keep graph `Connections` and make socket evidence expandable below each row.**
309 + - Pros: best UX if frontend supports row-level grouping by link.
310 + - Cons: more frontend work; current schema cannot directly express nested row groups, only expanded columns.
311 + - Implication: good future improvement, but not the fastest reliable repair.
312 +- **C. Keep the three current tabs with clearer labels.**
313 + - Pros: smallest backend change.
314 + - Cons: still forces users to reconcile three views of the same fact.
315 + - Risk: fails the purpose of this SOW.
316 +
317 +Recommendation: **A**. Detailed mode should show the exact socket rows as the primary drilldown; aggregated mode should show the relationship summary rows.
318 +
319 +### Schema Support Assessment
320 +
321 +Supported by the current schema:
322 +
323 +- Actor modal recipes are already supported under `types.actor_types.<id>.presentation.modal`.
324 +- Modal sections can already read `links`, `evidence`, `actor_table`, and `relationship_table` sources.
325 +- `relationship_summary` table types are already valid.
326 +- Compact table columns can already use `actor_ref`, `link_ref`, `ip`, `uint`, metrics, strings, and dictionary encoding.
327 +- Owner filters can already bind a relationship table to the selected actor when the table has `src_actor` and `dst_actor` columns.
328 +- `selected_side_endpoint` already supports rendering a local/remote endpoint from table columns without topology-specific frontend code.
329 +
330 +Not supported yet:
331 +
332 +- There is no schema field for "show these actor label keys in the modal identification/header area". Current `modal.labels` only tells the UI how to render the full labels table.
333 +
334 +Duplication policy:
335 +
336 +- Do not duplicate raw detailed socket evidence only for modal display.
337 +- Aggregated mode needs one relationship-summary row per existing internal connection summary row because graph links intentionally collapse port pairing and service name. This is not duplicate evidence; it is a distinct compact drilldown grain between graph links and detailed socket evidence.
338 +- Some small scalar values such as protocol, direction, state, socket count, and actor refs will appear both on graph links and relationship-summary rows. This is acceptable because the rows have different grain, are dictionary/number encoded, and are needed for standalone modal filtering/sorting.
339 +- The producer should avoid copying fields that can be projected from actor labels or actor rows. Process identity stays in actors/labels; relationship rows carry relationship facts.
340 +
341 +## Plan
342 +
343 +1. Analyze field inventory and old/current modal parity for `topology:network-connections`.
344 +2. Propose the product-oriented modal tables and header labels.
345 +3. Implement only this function's backend changes after the design is accepted.
346 +4. Hand off required frontend/aggregator behavior if needed.
347 +5. Validate with real local Agent payloads in aggregated and detailed modes.
348 +
349 +## Execution Log
350 +
351 +### 2026-05-11
352 +
353 +- Created SOW from user-reported modal regressions and current code evidence.
354 +- Paused this SOW because SOW-0028 owns the broader cross-repo topology mode,
355 + correlation, aggregation, and actor-identification contract that this SOW now
356 + depends on.
357 +- Resumed after SOW-0028 completed and was committed. The next step is to
358 + validate the installed aggregated/detailed network-connections modal recipes
359 + and repair any remaining backend producer gaps.
360 +- Validated the rebuilt producer output and found one remaining mismatch: the
361 + self/root actor still needed a process summary instead of socket rows.
362 +- Updated `network-viewer.c` so self actors use a `Processes` section over
363 + `ownership` links, process actors use `Connections` or `Sockets` depending on
364 + mode, endpoint actors use `Processes`, and secondary socket metrics are
365 + expanded columns instead of separate duplicate sections.
366 +- Updated the durable topology spec, developer guide, and project topology skill
367 + so future topology producers do not reintroduce `socket_ports` as a normal
368 + network-connections modal tab.
369 +
370 +## Validation
371 +
372 +Acceptance criteria evidence:
373 +
374 +- Complete inventory and mapping are recorded above under `Field inventory and
375 + mapping`.
376 +- Self/node modal shape is explicit: `Processes` from `links`, filtered to
377 + `type == ownership`, with process actor, socket count, and expanded evidence
378 + count.
379 +- Process modal shape is explicit:
380 + - aggregated mode: `Connections` from `tables.relationship.connections`;
381 + - detailed mode: `Sockets` from `evidence.socket`.
382 +- Endpoint modal shape is explicit: `Processes` from the same mode-specific
383 + relationship/evidence source.
384 +- Actor identification source is `modal.labels.identification.fields[]` over
385 + `actor_labels`.
386 +- `socket_ports` remains graph port-bullet inventory only; it is not emitted as
387 + a normal network-connections modal section.
388 +
389 +Tests or equivalent validation:
390 +
391 +- `git diff --check`: passed.
392 +- `sudo -n cmake --build build --target network-viewer.plugin -- -j2`: passed.
393 +- Plugin protocol validation against the rebuilt binary:
394 + - aggregated: status 200, mode `aggregated`, 110 actors, 147 links, 244
395 + relationship rows, 730 actor labels, 192 port rows, 0 socket evidence rows;
396 + - detailed: status 200, mode `detailed`, 111 actors, 147 links, 0 relationship
397 + rows, 735 actor labels, 192 port rows, 244 socket evidence rows.
398 +- `go run ./tools/functions-validation/validate --schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json --input ../../.local/audits/topology-sow-0025/network-connections-aggregated-protocol.json --min-rows 1`: passed.
399 +- `go run ./tools/functions-validation/validate --schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json --input ../../.local/audits/topology-sow-0025/network-connections-detailed-protocol.json --min-rows 1`: passed.
400 +- `go test ./pkg/topology/v1 ./tools/functions-validation/validate`: passed.
401 +
402 +Real-use evidence:
403 +
404 +- Rebuilt and installed only `/usr/libexec/netdata/plugins.d/network-viewer.plugin`
405 + with the same owner/group/mode as the previous installed binary.
406 +- Direct HTTP validation through `localhost:19999` remained blocked by SSO
407 + authorization returning HTTP 412, including through the token-safe minted
408 + bearer helper. The rebuilt plugin was therefore validated through the normal
409 + plugins.d `FUNCTION` stdin protocol, which exercises the same producer code
410 + path without changing Agent authentication settings.
411 +- `.local/audits/topology-sow-0025/network-connections-aggregated-protocol.json`
412 + and `.local/audits/topology-sow-0025/network-connections-detailed-protocol.json`
413 + contain the validation payloads and are intentionally gitignored local
414 + artifacts.
415 +
416 +Reviewer findings:
417 +
418 +- No external AI reviewer was requested for this narrow backend repair. The
419 + broader topology modal/correlation contract was reviewed during the preceding
420 + SOW-0028 work; this SOW validated the concrete producer output with schema and
421 + semantic checks.
422 +
423 +Same-failure scan:
424 +
425 +- Searched the durable topology spec, developer guide, project topology skill,
426 + and this SOW for stale `graph links plus socket_ports`, duplicate modal, and
427 + `Socket Evidence` language. Historical problem statements remain in this SOW;
428 + durable current-contract docs were updated.
429 +
430 +Sensitive data gate:
431 +
432 +- This SOW uses only path/line evidence and synthetic descriptions. No raw sensitive payload data is included.
433 +
434 +Artifact maintenance gate:
435 +
436 +- `AGENTS.md`: unchanged; no workflow or guardrail changed.
437 +- Runtime project skills: updated
438 + `.agents/skills/project-create-topology/SKILL.md` with the
439 + network-connections modal recipe.
440 +- Specs: updated `.agents/sow/specs/topology-function-schema.md` with the
441 + current network-connections modal composition.
442 +- Developer docs: updated `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`.
443 +- End-user/operator docs: unchanged; this changes developer topology payload
444 + composition, not operator instructions.
445 +- End-user/operator skills: unchanged; no public operator workflow changed.
446 +- SOW lifecycle: this SOW is ready to move from `current/` to `done/` with
447 + `Status: completed`.
448 +
449 +Specs update:
450 +
451 +- Updated `.agents/sow/specs/topology-function-schema.md`.
452 +
453 +Project skills update:
454 +
455 +- Updated `.agents/skills/project-create-topology/SKILL.md`.
456 +
457 +End-user/operator docs update:
458 +
459 +- Not affected. The user-facing Function remains the same; only v1 topology
460 + modal presentation metadata changed.
461 +
462 +End-user/operator skills update:
463 +
464 +- Not affected. No public/operator skill behavior or examples changed.
465 +
466 +Lessons:
467 +
468 +- Network-connections modal tabs must describe user tasks, not storage tables.
469 + `socket_ports` is useful for graph bullets but misleading as a peer modal tab.
470 +- The plugin stdin protocol is useful for validating producer output when local
471 + Agent HTTP access is blocked by SSO.
472 +
473 +Follow-up mapping:
474 +
475 +- SOW-0029 tracks the separate detailed loose-side/network-connections model
476 + work. No additional SOW-0025 follow-up remains.
477 +
478 +## Outcome
479 +
480 +Completed. The network-connections producer emits task-oriented actor modal
481 +recipes for self, process, and endpoint actors in both aggregated and detailed
482 +modes, without adding duplicated modal-only socket rows.
483 +
484 +## Lessons Extracted
485 +
486 +- Keep modal composition tied to user troubleshooting questions. Internal tables
487 + can remain in the payload for graph affordances or aggregation, but they should
488 + not automatically become top-level modal tabs.
489 +- Validate topology producer changes through plugins.d protocol output when HTTP
490 + auth prevents local Function calls.
491 +
492 +## Followup
493 +
494 +- SOW-0029 covers the next network-connections detailed loose-side model work.
495 +
496 +## Regression Log
497 +
498 +None yet.
499 +
500 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/done/SOW-0026-20260511-snmp-modal-product-composition.md new
+642
@@ -0,0 +1,642 @@
1 +# SOW-0026 - SNMP Modal Product Composition
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +Sub-state: second regression repair, documentation updates, validation, and
8 +local build completed on 2026-05-11.
9 +
10 +## Requirements
11 +
12 +### Purpose
13 +
14 +Make `topology:snmp` actor modals useful for network engineers who need accurate port, neighbor, and L2 relationship information without contradictory tables.
15 +
16 +### User Request
17 +
18 +The user reported that SNMP modals show `Ports` and `Links`, but the distinction is confusing and appears incorrect. A device cannot have an L2 link that is not associated with a port. The user also reported that local port labels can be wrong and that `Ports` and `Links` must be 100% in sync or the modal is useless.
19 +
20 +### Assistant Understanding
21 +
22 +Facts:
23 +
24 +- Current SNMP device modals include a `Ports` section from actor table `actor_ports` and a `Links` section from graph links.
25 +- Current `actor_ports` modal columns show `name`, statuses, role, VLAN/FDB/link/neighbor counts, and debug `extra`.
26 +- Current `Links` modal columns show remote actor, local port, remote port, protocol, direction, state, and evidence count.
27 +- The current `Links` section derives local/remote ports from selected-side port projections over link endpoint columns.
28 +
29 +Inferences:
30 +
31 +- The issue is likely not just labels. The underlying product model is wrong if ports and links can disagree.
32 +- For a device actor, the primary table should probably be a port-centric table. Links/neighbors should be columns or expandable rows attached to each port, not an independent table with a different interpretation of "local port".
33 +- Inferred endpoint actors may still need a relationship/link modal, but managed SNMP devices need port inventory as the organizing principle.
34 +
35 +Confirmed by code review:
36 +
37 +- Link evidence already carries `src_if_index` and `dst_if_index`, and graph links carry `src_port_name` and `dst_port_name`.
38 +- The port inventory source rows commonly carry `if_index`, `if_name`, `if_descr`, `if_alias`, `mac`, and `speed`, but the current `actor_ports` table does not expose them as typed canonical columns.
39 +- Managed device actors should use a port-centric modal. Endpoint/segment actors, which do not have port inventory, can keep a relationship-oriented `Links` section.
40 +- `topologyV1EndpointPortName()` falls back to `display_name` and `sys_name`; that can turn an actor/device/IP label into a displayed local port name when no real port field is present.
41 +
42 +### Acceptance Criteria
43 +
44 +- A complete inventory exists for SNMP actor modal facts: actor labels, actor typed columns, `actor_ports`, graph links, and evidence sections.
45 +- The desired SNMP device modal is port-centric and shows actual numeric port IDs when known, plus port name.
46 +- `Ports` and link/neighbor information are derived from the same canonical endpoint facts or explicitly cross-checked.
47 +- Any remaining `Links` section has a precise purpose and cannot contradict the port table.
48 +- The SOW identifies every missing canonical field required to make port/link rows 100% aligned.
49 +- The actor identification area exposes important device labels, not only the generic `Labels` tab.
50 +
51 +## Analysis
52 +
53 +Sources checked:
54 +
55 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:158` current SNMP device modal.
56 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:163` current `Ports` section.
57 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:188` current `Links` section.
58 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:212` current selected-side local/remote port columns.
59 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:474` `actor_ports` table type.
60 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:493` current port modal columns.
61 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:534` current `actor_ports` columns.
62 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:699` link/evidence construction entry point.
63 +- `.agents/sow/specs/topology-function-schema.md:380` SNMP modal composition notes.
64 +- `.agents/sow/specs/topology-modes-correlation-aggregation.md:498-556` SNMP detailed/aggregated and port-centric modal requirements.
65 +- `librenms/librenms @ 1d096602a4ce1197faab084b2523c7dd50419427 app/Models/Port.php:90-138` shows port labels are built from `ifName`, `ifAlias`, `ifDescr`, and sometimes `ifIndex`.
66 +- `librenms/librenms @ 1d096602a4ce1197faab084b2523c7dd50419427 app/Models/Port.php:401-427` models links as port relationships (`local_port_id`, `remote_port_id`).
67 +- `librenms/librenms @ 1d096602a4ce1197faab084b2523c7dd50419427 app/Http/Controllers/Device/Tabs/PortsController.php:145-185` organizes neighbors from a port-first data set.
68 +- `netdisco/netdisco @ 7c1bc3e8290fe7b757c6a462de12e4613eca3899 share/views/ajax/device/ports.tt:394-406` renders neighbors from the device port row and remote port.
69 +
70 +Current state:
71 +
72 +- Device modal:
73 + - `Ports` comes from `actor_ports`.
74 + - `Links` comes from graph links.
75 +- `actor_ports` currently has stable columns such as `name`, topology role, admin/oper status, port type, link mode, STP state, VLAN IDs, FDB count, link count, neighbor count, neighbors JSON, VLANs JSON, and debug `extra`.
76 +- `Links` currently uses selected-side projections for `src_port_name` and `dst_port_name`, but the user observed cases where local port looks like a remote actor/IP. This points to a possible endpoint mapping or projection issue.
77 +- `topologyV1EndpointPortName()` currently accepts `display_name` and `sys_name` as final fallbacks. These are actor labels, not port labels, and explain how a remote actor/IP can appear as a port.
78 +
79 +Available facts to inventory:
80 +
81 +- Device actor:
82 + - display name, sysName, sysDescr, model, vendor, management IP, sys location/contact, protocols/capabilities, port counts, VLAN/FDB/LLDP/CDP counts.
83 +- Port table:
84 + - current normalized port name/status/role/VLAN/FDB/neighbor facts.
85 + - likely raw/custom fields in `extra` that may include actual `if_index`, `if_name`, `if_descr`, `if_alias`, MAC, speed, duplex, VLAN details, and neighbor objects.
86 +- Link/evidence:
87 + - remote actor, source/destination port names, source/destination ifIndex when present, protocol, direction/state, confidence/inference/attachment mode, evidence count.
88 +
89 +Target audience and questions:
90 +
91 +- Network engineers need:
92 + - Which physical/logical ports exist?
93 + - What is the actual numeric port ID, ifIndex, interface name, and description/alias?
94 + - Which ports are up/down/admin-down?
95 + - What is connected to each port and by which protocol/evidence?
96 + - Is the link verified by LLDP/CDP or inferred by FDB/ARP/STP?
97 + - Which VLANs, STP role/state, FDB MAC counts, speeds, and neighbors apply to each port?
98 +
99 +Reference pattern:
100 +
101 +- Network inventory tools treat the port as the organizing object. LibreNMS keeps port labels on `ifName`/`ifAlias`/`ifDescr`/`ifIndex`, and its link model joins ports through local and remote port IDs. Netdisco similarly renders neighbor details from the selected device port row and remote port.
102 +- For Netdata topology modals, this means a managed SNMP device should not present graph links as an equal peer of port inventory. It should present port inventory first and neighbor/link evidence as port-aligned details.
103 +
104 +Available backend facts:
105 +
106 +- Device actor labels and typed actor columns already include display name, sysName, vendor, model, management IP, sysDescr, sysLocation, sysContact, capabilities, protocol lists, port counts, VLAN counts, FDB counts, and LLDP/CDP neighbor counts.
107 +- Port actor-owned rows have raw fields such as `if_index`, `if_name`, `if_descr`, `if_alias`, `mac`, `speed`, status, type, mode, topology role, STP state, VLAN IDs/details, FDB count, link count, neighbor count, and neighbor objects.
108 +- Graph links have source/destination actors, link type, protocol, direction, state, source/destination port names, evidence count, discovered time, and last-seen time.
109 +- Evidence rows have source/destination `if_index`, source/destination management IP, confidence, inference, attachment mode, and raw endpoint/metrics JSON for debug.
110 +
111 +Missing or weak canonicalization:
112 +
113 +- `actor_ports` does not expose `if_index`, `if_name`, `if_descr`, `if_alias`, `port_id`, `mac`, or `speed` as typed columns.
114 +- Link modal selected-side port display relies on `src_port_name`/`dst_port_name`, and those values can currently be polluted by actor labels because of the endpoint-port fallback.
115 +- There is no port-aligned relationship table for managed devices. The only link view is generic graph-link-oriented, so it can disagree visually with the port inventory.
116 +
117 +Risks:
118 +
119 +- If port and link tables are inconsistent, users will distrust the topology.
120 +- Showing raw nested neighbor JSON in normal table cells would regress polish and usefulness.
121 +- If numeric port IDs are unavailable, fabricating autoincrement IDs would be worse than showing no numeric ID.
122 +
123 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
124 +
125 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Execution Log and Validation sections).
126 +
127 +Problem / root-cause model:
128 +
129 +- SNMP modals currently expose implementation tables separately rather than presenting a device as a collection of ports with attached neighbor/link evidence.
130 +- The current `Links` section can produce confusing local/remote port labels because it is graph-link-oriented, not port-oriented.
131 +
132 +Evidence reviewed:
133 +
134 +- SNMP device modal currently has both `Ports` and `Links` sections in `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:158-180`.
135 +- `Links` local/remote port columns are selected-side projections in `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:212-213`.
136 +- `actor_ports` modal currently does not expose a clear numeric port ID column in the visible section at `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:493-503`.
137 +- `actor_ports` table columns currently omit `if_index`, `if_name`, `if_descr`, `if_alias`, `port_id`, `mac`, and `speed` at `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:534-550`.
138 +- `topologyV1EndpointPortName()` falls back to `display_name` and `sys_name` at `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:1551-1559`.
139 +
140 +Affected contracts and surfaces:
141 +
142 +- Agent Function payload for `topology:snmp`.
143 +- SNMP topology Go v1 adapter.
144 +- SNMP topology tests and fixtures.
145 +- Cloud frontend modal rendering if the actor identification/header area needs a new selector.
146 +- Cloud aggregator if it merges device port rows and link/evidence rows.
147 +- Developer guide, topology spec, project topology skill.
148 +
149 +Existing patterns to reuse:
150 +
151 +- `actor_ports` actor-owned table for port inventory.
152 +- Structured evidence columns for L2 discovery details.
153 +- `selected_side_endpoint` only where side selection is truly needed.
154 +- Debug `extra` JSON for unknown custom port fields, not normal table display.
155 +- Actor-owned modal detail tables with `owner_filter: actor_column`.
156 +- `modal.labels.identification.fields[]` for modal header facts.
157 +
158 +Risk and blast radius:
159 +
160 +- User-facing SNMP device modal behavior changes.
161 +- Incorrect port/evidence joins can misrepresent physical topology.
162 +- Aggregated cloud topologies may need stricter table merge rules for port identity.
163 +- Sensitive data risk includes device sysContact/sysLocation and management IPs; do not write raw real values to durable artifacts.
164 +
165 +Sensitive data handling plan:
166 +
167 +- Use synthetic device/port/IP examples only.
168 +- Do not store raw SNMP sysContact/sysLocation, management IPs, customer names, credentials, or community strings in SOWs/docs/tests.
169 +- Keep real payload captures under `.local/` only.
170 +
171 +Implementation plan:
172 +
173 +1. Add SNMP modal identification fields:
174 + - managed devices: display name, management IP, vendor, model, port counts, LLDP/CDP counts;
175 + - endpoints/segments: display name and matching network identifiers.
176 +2. Expose typed port inventory columns in `actor_ports`: `if_index`, `port_id`, `if_name`, `if_descr`, `if_alias`, `mac`, and `speed`.
177 +3. Remove actor-label fallbacks from `topologyV1EndpointPortName()` so only real port fields can become link endpoint port labels.
178 +4. Add a compact actor-owned `actor_port_links` detail table:
179 + - one row per selected actor side of each graph link;
180 + - columns include local `if_index`, local port name, remote actor, remote port name, protocol, link type, state, evidence count, confidence, inference, attachment mode, discovered time, and last-seen time;
181 + - this table is derived from graph links/evidence facts and exists to align device modal relationships with port inventory, not to create new topology facts.
182 +5. Change managed device modals to show `Ports` then `Port Neighbors`; keep generic `Links` only for endpoint/segment/custom actors that do not own port inventory.
183 +6. Add tests proving:
184 + - device modal no longer uses the generic `Links` section;
185 + - `actor_ports` exposes typed port identity/status columns;
186 + - `actor_port_links` is side-owned and uses the same local `if_index`/port name as evidence;
187 + - endpoint port names do not fall back to device/IP display names.
188 +7. Update spec, developer guide, and project topology skill.
189 +
190 +Validation plan:
191 +
192 +- Go tests for SNMP topology v1 conversion.
193 +- Payload schema validation.
194 +- Fixture test with a device containing multiple ports, LLDP/CDP verified links, inferred FDB/ARP links, and ports without links.
195 +- Cross-check that every link shown under a device maps to the same local port row shown in the port table when a port identifier exists.
196 +- Manual UI validation on a real or sanitized SNMP topology payload.
197 +
198 +Artifact impact plan:
199 +
200 +- AGENTS.md: likely unaffected.
201 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md` if SNMP modal guidelines change.
202 +- Specs: update `.agents/sow/specs/topology-function-schema.md`.
203 +- End-user/operator docs: likely unaffected unless Function examples are changed.
204 +- End-user/operator skills: unaffected.
205 +- SOW lifecycle: close only after SNMP modal behavior is validated with realistic data.
206 +
207 +Open-source reference evidence:
208 +
209 +- LibreNMS and Netdisco both support the port-first model:
210 + - `librenms/librenms @ 1d096602a4ce1197faab084b2523c7dd50419427 app/Models/Port.php:90-138`
211 + - `librenms/librenms @ 1d096602a4ce1197faab084b2523c7dd50419427 app/Models/Port.php:401-427`
212 + - `librenms/librenms @ 1d096602a4ce1197faab084b2523c7dd50419427 app/Http/Controllers/Device/Tabs/PortsController.php:145-185`
213 + - `netdisco/netdisco @ 7c1bc3e8290fe7b757c6a462de12e4613eca3899 share/views/ajax/device/ports.tt:394-406`
214 +
215 +Open decisions:
216 +
217 +- Resolved by user approval to proceed after product analysis:
218 + - Managed SNMP device actors get a port-first modal with `Ports` and a port-aligned `Port Neighbors` section.
219 + - Endpoint/segment/custom actors keep the generic relationship-oriented `Links` section because they do not own port inventory.
220 + - No synthetic numeric port IDs may be generated. Numeric port ID is shown only when a real `if_index` or source port ID is known.
221 +
222 +## Implications And Decisions
223 +
224 +1. Device modal composition:
225 + - Decision: managed devices use `Ports` + `Port Neighbors`; generic graph `Links` is removed from managed device modals.
226 + - Implication: users inspect a device by physical/logical port first, which matches the network engineer workflow.
227 + - Risk: a link with missing local port identity will appear in `Port Neighbors` with an empty local port. That is preferable to fabricating a port or showing an actor label as a port.
228 +
229 +2. Endpoint/segment/custom modal composition:
230 + - Decision: actors without owned port inventory keep the existing `Links` section.
231 + - Implication: endpoint/segment actors can still explain how they connect to the graph.
232 + - Risk: selected-side projections remain there, but the endpoint port fallback fix prevents actor names from being shown as ports.
233 +
234 +3. Port identity:
235 + - Decision: expose actual `if_index` and source `port_id` as typed port columns; never autoincrement.
236 + - Implication: the UI can show numeric port IDs only when the SNMP backend knows them.
237 + - Risk: devices without `if_index` will show blank `Port ID`, which is truthful.
238 +
239 +4. Data duplication:
240 + - Decision: add `actor_port_links` as a compact actor-owned modal index over existing graph links.
241 + - Implication: the table adds small side-specific rows, but does not duplicate raw evidence or nested metadata.
242 + - Risk: Cloud aggregation must treat this as actor-owned modal detail and append/merge consistently.
243 +
244 +## Plan
245 +
246 +1. Move SOW to current and record product decisions.
247 +2. Implement typed port inventory columns and modal identification fields.
248 +3. Implement `actor_port_links` and switch device modal sections.
249 +4. Fix endpoint port fallback.
250 +5. Update tests, spec, developer guide, and project skill.
251 +6. Validate with Go tests and topology schema validation.
252 +
253 +## Execution Log
254 +
255 +### 2026-05-11
256 +
257 +- Created SOW from user-reported SNMP modal regressions and current code evidence.
258 +- Completed product analysis:
259 + - managed SNMP devices must be port-first;
260 + - endpoint/segment/custom actors keep generic links;
261 + - actor labels must not be used as port labels;
262 + - `actor_ports` needs typed port identity columns;
263 + - `actor_port_links` is needed as a compact modal index.
264 +- Implemented managed-device modal identification and port-first sections.
265 +- Added typed `actor_ports` columns for real port identity and status.
266 +- Added `actor_port_links` as a compact port-aligned modal index over graph links.
267 +- Removed actor-label fallbacks from SNMP endpoint port names.
268 +- Updated SNMP topology tests, topology spec, developer guide, and project topology skill.
269 +
270 +## Validation
271 +
272 +Acceptance criteria evidence:
273 +
274 +- Complete inventory: recorded in this SOW under `Analysis`, including actors,
275 + `actor_labels`, `actor_ports`, graph links, evidence, and missing canonical
276 + fields.
277 +- Port-centric device modal: managed device modal now uses `Ports` and
278 + `Port Neighbors`, with `actor_ports` first at
279 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:156-177`.
280 +- Actor identification: device modal label identification fields are declared
281 + at `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:189-204`.
282 +- Numeric port IDs: `actor_ports` exposes `if_index` and source `port_id` as
283 + typed columns at
284 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:658-682`.
285 +- Port/link alignment: `actor_port_links` table type and modal columns are
286 + declared at
287 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:597-628`,
288 + and row construction uses one row per incident actor side at
289 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:1447-1485`.
290 +- Generic graph `Links` remain available only through the endpoint/segment/custom
291 + modal path at
292 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:181-186`.
293 +- Actor labels no longer become port names:
294 + `topologyV1EndpointPortName()` now accepts only real port fields at
295 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go:1830-1837`.
296 +
297 +Tests or equivalent validation:
298 +
299 +- `cd src/go && go test ./plugin/go.d/collector/snmp_topology`
300 +- `cd src/go && go test ./plugin/go.d/collector/snmp_topology ./pkg/topology/v1 ./tools/functions-validation/validate`
301 +- `cd src/go && for fixture in tools/functions-validation/fixtures/topology-v1/*.json; do go run ./tools/functions-validation/validate -schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json -input "$fixture" -require-rows; done`
302 +- `git diff --check`
303 +- `sudo -n cmake --build build --target go.d.plugin -- -j2`
304 +- Unit-test evidence:
305 + - typed port columns and values:
306 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go:360-384`;
307 + - `actor_port_links` side-owned rows:
308 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go:386-398`;
309 + - managed device modal sections:
310 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go:420-430`;
311 + - endpoint actors keep generic links:
312 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go:432-437`;
313 + - actor label fallback regression:
314 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go:440-486`.
315 +
316 +Real-use evidence:
317 +
318 +- Built `go.d.plugin` through the local CMake/Ninja build path with the updated
319 + SNMP topology producer. The running local Agent was not modified by this SOW;
320 + no production or customer system was used.
321 +
322 +Reviewer findings:
323 +
324 +- External AI reviewers were not requested for this SOW. Review coverage came
325 + from code inspection, open-source reference comparison, schema validation,
326 + targeted tests, and same-failure search.
327 +
328 +Same-failure scan:
329 +
330 +- Searched for actor-label-to-port-name fallback and selected-side SNMP link
331 + modal risks with:
332 + `rg -n "display_name.*port|sys_name.*port|port.*display_name|port.*sys_name|topologyV1EndpointPortName|selected_side_endpoint" src/go/plugin/go.d/collector/snmp_topology src/go/plugin/go.d/collector src/collectors -S`
333 +- Remaining `selected_side_endpoint` usage in SNMP is intentionally limited to
334 + endpoint/segment/custom actor modals. The SNMP endpoint port-name helper no
335 + longer accepts actor labels.
336 +
337 +Sensitive data gate:
338 +
339 +- This SOW uses only path/line evidence and synthetic descriptions. No raw sensitive payload data is included.
340 +
341 +Artifact maintenance gate:
342 +
343 +- `AGENTS.md`: unchanged. No project-wide workflow rule changed.
344 +- Runtime project skills: updated
345 + `.agents/skills/project-create-topology/SKILL.md:260-283` with SNMP/L2 modal
346 + rules.
347 +- Specs: updated
348 + `.agents/sow/specs/topology-function-schema.md:418-444` with SNMP port-first
349 + modal contract.
350 +- End-user/operator docs: unchanged. This changes developer-facing topology
351 + payload composition, not an operator workflow or public querying procedure.
352 +- End-user/operator skills: unchanged. No operator-facing skill semantics
353 + changed.
354 +- SOW lifecycle: this SOW will move from `current/` to `done/` with status
355 + `completed` in the same commit as the implementation.
356 +
357 +Specs update:
358 +
359 +- Updated `.agents/sow/specs/topology-function-schema.md`.
360 +
361 +Project skills update:
362 +
363 +- Updated `.agents/skills/project-create-topology/SKILL.md`.
364 +
365 +End-user/operator docs update:
366 +
367 +- No update needed. The change is internal developer schema composition for
368 + topology producers.
369 +
370 +End-user/operator skills update:
371 +
372 +- No update needed. Public/operator skills do not describe developer modal
373 + composition.
374 +
375 +Lessons:
376 +
377 +- For SNMP/L2, a generic graph link table is not a good device modal. Port
378 + inventory has to be the organizing object, and relationship rows must align
379 + with that port identity.
380 +- Link endpoint display helpers must never mix actor identity labels with port
381 + labels.
382 +
383 +Follow-up mapping:
384 +
385 +- No deferred implementation work remains in this SOW. Interface overlay work
386 + for traffic/packets/errors/state is a separate topology overlay topic already
387 + documented in the spec and developer guide.
388 +
389 +## Outcome
390 +
391 +Completed. Managed SNMP device modals are now port-first, expose typed port
392 +identity, show port-aligned neighbor rows, and avoid using actor labels as port
393 +names. Endpoint/segment/custom actor modals keep generic graph-link drilldowns.
394 +
395 +## Lessons Extracted
396 +
397 +- SNMP topology presentation needs network-engineer semantics: port identity
398 + first, neighbor/link evidence attached to that identity, and raw JSON only for
399 + debug/expanded diagnostics.
400 +
401 +## Followup
402 +
403 +See regression entries below.
404 +
405 +## Regression Log
406 +
407 +See regression entries below.
408 +
409 +Note: dated regression entries intentionally use `## Regression - YYYY-MM-DD`
410 +headings to match the repository SOW lifecycle contract.
411 +
412 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
413 +
414 +## Regression - 2026-05-11
415 +
416 +### What Broke
417 +
418 +- The SNMP device `Ports` section labels `if_index` as `Port ID`, which looks like
419 + an invented/autoincrement port number. The user requirement is no synthetic
420 + numbering anywhere. Real port numbers may be shown only when sourced from
421 + canonical producer data.
422 +- The `Ports` expanded row does not show the neighboring actor as a clickable
423 + actor link. The separate `Port Neighbors` section has clickable remote actors,
424 + but the port-owned row must also expose the directly related neighbor.
425 +- Actor links in v1 modal tables were not clickable in the frontend because the
426 + producer emits `actor_ref_label` projections with `actor_link` cells, while
427 + the frontend projection engine returned labels instead of actor IDs for that
428 + cell type.
429 +
430 +### Evidence
431 +
432 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` currently
433 + emits the first `Ports` modal column as `if_index` with label `Port ID`.
434 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` currently
435 + defines `actor_ports` without neighbor actor-ref columns, so the expanded
436 + `Ports` view cannot link to the neighbor.
437 +- `cloud-frontend/src/domains/functions/topology/v1/buildModalSections.js`
438 + projected `actor_ref_label` to a label string even when the declared cell type
439 + was `actor_link`.
440 +
441 +### Root Cause
442 +
443 +- The SOW correctly made SNMP modals port-first, but it still reused SNMP
444 + `ifIndex` as the visible port identity. `ifIndex` is useful technical
445 + metadata, but it is not a physical/source port number and should not be
446 + presented as `Port ID`.
447 +- Port inventory and port-neighbor rows were aligned in separate tables, but the
448 + inventory table did not carry a compact derived neighbor actor reference for
449 + the expanded row.
450 +- The frontend projection engine treated projection kind as the only source of
451 + output shape. For `actor_link` cells, the renderer needs the actor ID, not the
452 + already formatted actor label.
453 +
454 +### Repair Plan
455 +
456 +- Add a nullable `port_number` column to `actor_ports`. Populate it only from an
457 + explicit numeric source field such as `port_number` or numeric `port_id`.
458 + Never derive it from row order or `if_index`.
459 +- Rename the expanded `if_index` modal column to `SNMP ifIndex` and remove it
460 + from the visible table columns.
461 +- Add compact nullable `neighbor_actor` and `neighbor_port_name` columns to
462 + `actor_ports`, derived from the same graph-link endpoint facts used by
463 + `actor_port_links`.
464 +- Show `Neighbor` as an expanded `actor_link` cell and `Neighbor Port` as an
465 + expanded text cell in the `Ports` section.
466 +- Fix the frontend modal projection engine so `actor_ref_label` returns actor IDs
467 + when the cell type is `actor_link`.
468 +
469 +### Validation Plan
470 +
471 +- Add/adjust SNMP topology tests proving:
472 + - `port_number` is present and populated only from explicit source data;
473 + - `if_index` remains available as expanded SNMP metadata;
474 + - `neighbor_actor` and `neighbor_port_name` are derived from graph-link facts;
475 + - the main `Ports` modal no longer labels `if_index` as `Port ID`.
476 +- Add frontend projection tests proving `actor_ref_label` + `actor_link` produces
477 + actor IDs.
478 +- Run targeted Go and frontend tests plus schema validation and `git diff --check`.
479 +
480 +### Implementation
481 +
482 +- Added nullable `actor_ports.port_number`, populated only from explicit
483 + `port_number` or numeric source `port_id`.
484 +- Kept SNMP `if_index` as expanded technical metadata labelled `SNMP ifIndex`,
485 + not as visible `Port ID`.
486 +- Added nullable `actor_ports.neighbor_actor` and `neighbor_port_name`, derived
487 + from graph-link endpoint facts so the expanded port row can link to the same
488 + neighbor shown in `actor_port_links` when the port has one unambiguous remote
489 + actor.
490 +- Fixed the frontend v1 modal projection engine so `actor_ref_label` returns
491 + actor IDs for `actor_link` cells while keeping text/badge cells label-based.
492 +- Updated the topology developer guide, topology specs, and project topology
493 + skill with the stricter SNMP port identity and expanded-row neighbor rules.
494 +
495 +### Validation
496 +
497 +Acceptance criteria evidence:
498 +
499 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` now emits
500 + `port_number` as the visible first Ports column and `if_index` as expanded
501 + `SNMP ifIndex`.
502 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` now emits
503 + `neighbor_actor` and `neighbor_port_name` in `actor_ports`.
504 +- `cloud-frontend/src/domains/functions/topology/v1/buildModalSections.js`
505 + now returns actor IDs for `actor_ref_label` projections when the cell is
506 + `actor_link`.
507 +
508 +Tests or equivalent validation:
509 +
510 +- `cd src/go && go test -count=1 ./plugin/go.d/collector/snmp_topology ./pkg/topology/v1 ./tools/functions-validation/validate`
511 +- `yarn test src/domains/functions/topology/v1/buildModalSections.test.js --runInBand`
512 +- `sudo -n cmake --build build --target go.d.plugin -- -j2`
513 +- `git diff --check` in the Agent repository.
514 +- `git diff --check` in the frontend repository.
515 +
516 +Real-use evidence:
517 +
518 +- Built the local `go.d.plugin` target successfully. Browser inspection can be
519 + repeated after the rebuilt Agent and UI are installed; no code change remains
520 + blocked on that check.
521 +
522 +Reviewer findings:
523 +
524 +- External AI reviewers were not requested for this regression repair. The fix
525 + is covered by targeted producer and frontend unit tests plus schema validation.
526 +
527 +Same-failure search:
528 +
529 +- Verified the repaired modal contract by searching for `Port ID`, `if_index`,
530 + `actor_ports`, `neighbor_actor`, and `actor_ref_label` in the affected producer,
531 + specs, skill, and frontend projection files.
532 +
533 +Artifact maintenance gate:
534 +
535 +- `AGENTS.md`: unchanged. No project-wide workflow rule changed.
536 +- Runtime project skills: updated `.agents/skills/project-create-topology/SKILL.md`
537 + with stricter SNMP `port_number`, `if_index`, and expanded neighbor rules.
538 +- Specs: updated `.agents/sow/specs/topology-function-schema.md` and
539 + `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
540 +- End-user/operator docs: unchanged. This is developer-facing topology payload
541 + composition, not an operator workflow.
542 +- End-user/operator skills: unchanged. No public/operator skill semantics
543 + changed.
544 +- SOW lifecycle: reopened regression SOW moved back to `current/`, repaired,
545 + then returned to `done/` with `Status: completed`.
546 +
547 +Follow-up mapping:
548 +
549 +- No new deferred implementation work remains in this SOW. Manual visual
550 + polishing of topology layout remains outside this SNMP modal regression.
551 +
552 +### Regression Outcome
553 +
554 +Completed. SNMP Ports no longer display SNMP `ifIndex` as a synthetic-looking
555 +port number, expanded port rows can expose a clickable neighbor actor when link
556 +facts allow it, and frontend v1 actor-link cells navigate again.
557 +
558 +## Regression - 2026-05-11 - SNMP Port Identity Alignment
559 +
560 +### What Broke
561 +
562 +- The previous regression repair introduced `actor_ports.port_number` and made
563 + it the visible `Ports` table identity.
564 +- The live SNMP payload has the real numeric device port identity in
565 + `actor_ports.if_index` and `actor_port_links.if_index`; it does not
566 + necessarily have a separate numeric `port_number` or numeric `port_id`.
567 +- Result: `Ports` showed empty port IDs while `Port Neighbors` showed correct
568 + non-empty port IDs.
569 +
570 +### Evidence
571 +
572 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` currently
573 + emits `port_number` as the first `Ports` modal column.
574 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` currently
575 + emits `if_index` as the first `Port Neighbors` modal column.
576 +- The user verified in the local UI that the `Ports` values are empty for the
577 + managed SNMP device, while `Port Neighbors` values are correct and non-empty.
578 +
579 +### Root Cause
580 +
581 +- The producer accidentally split one SNMP concept into two modal identities:
582 + `port_number` for `Ports`, and `if_index` for `Port Neighbors`.
583 +- For SNMP/L2, `ifIndex` is the device-provided numeric interface identifier
584 + and is the value the UI should show as the port ID. It is not a row-order
585 + autoincrement invented by Netdata.
586 +
587 +### Repair Plan
588 +
589 +- Remove the `actor_ports.port_number` column and helper logic.
590 +- Use `actor_ports.if_index` as the visible `Ports` `Port ID` column, matching
591 + `actor_port_links.if_index`.
592 +- Keep the no-synthetic-number rule: never derive port IDs from row order.
593 +- Update specs, developer guide, project skill, and tests to describe SNMP
594 + `if_index` as the visible real numeric port identity.
595 +
596 +### Implementation
597 +
598 +- Removed the `actor_ports.port_number` column and helper logic.
599 +- Restored `actor_ports.if_index` as the visible `Ports` `Port ID` column.
600 +- Kept `actor_port_links.if_index` unchanged, so `Ports` and `Port Neighbors`
601 + now use the same real SNMP numeric port identity.
602 +- Updated tests, topology specs, topology developer guide, and project topology
603 + skill to define `if_index` as the visible SNMP port ID when known.
604 +
605 +### Validation
606 +
607 +Acceptance criteria evidence:
608 +
609 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` emits
610 + `if_index` as the first `Ports` modal column and as the first
611 + `Port Neighbors` modal column.
612 +- `src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go` verifies
613 + `actor_ports` has no `port_number` column and still exposes `if_index`.
614 +
615 +Tests or equivalent validation:
616 +
617 +- `cd src/go && go test -count=1 ./plugin/go.d/collector/snmp_topology ./pkg/topology/v1 ./tools/functions-validation/validate`
618 +- `sudo -n cmake --build build --target go.d.plugin -- -j2`
619 +
620 +Artifact maintenance gate:
621 +
622 +- `AGENTS.md`: unchanged. No project-wide workflow rule changed.
623 +- Runtime project skills: updated `.agents/skills/project-create-topology/SKILL.md`
624 + with SNMP `if_index` as the visible port ID and the no-generated-sequence rule.
625 +- Specs: updated `.agents/sow/specs/topology-function-schema.md` and
626 + `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
627 +- End-user/operator docs: unchanged. This remains developer-facing topology
628 + payload composition, not an operator workflow.
629 +- End-user/operator skills: unchanged. No public/operator skill semantics
630 + changed.
631 +- SOW lifecycle: reopened from `done/`, repaired, then returned to `done/` with
632 + `Status: completed`.
633 +
634 +Follow-up mapping:
635 +
636 +- No new deferred implementation work remains in this SOW.
637 +
638 +### Regression Outcome
639 +
640 +Completed. SNMP `Ports` and `Port Neighbors` now use the same real
641 +device-provided `if_index` value for the visible port ID, without any synthetic
642 +numbering.
.agents/sow/done/SOW-0028-20260511-topology-mode-correlation-aggregation.md new
+531
@@ -0,0 +1,531 @@
1 +# SOW-0028 - Topology Mode Correlation Aggregation Contract
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +Sub-state: completed. The cross-repo compatibility contract is implemented and
8 +validated. True one-sided detailed network-connections graph rows are split to
9 +SOW-0029 because they require a separate Agent/UI/aggregator execution pass.
10 +
11 +## Requirements
12 +
13 +### Purpose
14 +
15 +Make topology maps and actor modals fit for SRE, DevOps, sysadmin, and network
16 +engineer workflows by defining one contract for detailed evidence, aggregated
17 +views, cross-payload correlation, table merging, and actor modal identification.
18 +
19 +### User Request
20 +
21 +The user asked to create the full specification first, then create SOW/TODO
22 +handoff artifacts for every affected repo, then implement the Agent, Cloud
23 +frontend, and Cloud topology aggregation service without relying on chat
24 +context.
25 +
26 +The user explicitly requested the spec to cover:
27 +
28 +- Agent detailed and aggregated views;
29 +- aggregator detailed and aggregated views while consuming detailed input;
30 +- UI detailed and aggregated views;
31 +- network-connections;
32 +- SNMP/L2;
33 +- streaming;
34 +- actor identification from selected labels in actor modals.
35 +
36 +### Assistant Understanding
37 +
38 +Facts:
39 +
40 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` already has
41 + `data.view.mode`, actor/link type presentation, modal recipes, actor labels,
42 + and a correlation section.
43 +- Current correlation documentation is centered on pure correlation actors and
44 + absorb/link actions.
45 +- The current network-connections modal analysis showed a broader contract gap:
46 + actor modal identification needs producer-selected labels, and aggregated
47 + network-connections needs relationship-summary rows for useful drilldowns.
48 +- SNMP and streaming do not currently have a meaningful detailed/aggregated
49 + mode split.
50 +
51 +Inferences:
52 +
53 +- The previous pure-correlation-actor model is not enough for
54 + network-connections detailed mode because exact remote `IP:PORT` tuples should
55 + often be loose relationship facts rather than actors.
56 +- SNMP/L2 needs replacement semantics, not loose-side socket resolution.
57 +- Streaming needs merge/enrichment semantics, not replacement.
58 +- The aggregator must consume detailed payloads before returning an aggregated
59 + view, otherwise cross-node correlation loses facts too early.
60 +
61 +Unknowns:
62 +
63 +- Exact final schema field names may need small adjustments during
64 + implementation to fit existing v1 schema style and frontend normalizer
65 + patterns.
66 +- The aggregator service may already implement a subset of SOW-0028 behavior;
67 + this must be verified against its current code before patching.
68 +
69 +### Acceptance Criteria
70 +
71 +- `.agents/sow/specs/topology-modes-correlation-aggregation.md` defines the
72 + full contract and examples for network-connections, SNMP/L2, and streaming.
73 +- This SOW records the cross-repo pending checklist for Agent, UI, and
74 + aggregator work.
75 +- A frontend TODO exists in the Cloud frontend repo and references the new spec.
76 +- A Cloud topology service SOW exists in the aggregator repo and references the
77 + new spec.
78 +- Agent schema and topology producers are updated to the new contract where the
79 + Agent owns the data.
80 +- Cloud frontend is updated to decode/render actor modal identification and the
81 + new mode/correlation semantics it must handle.
82 +- Cloud topology service is updated to aggregate according to the new contract.
83 +- Validation evidence records schema tests, relevant unit tests, and local
84 + Function/API checks that were possible.
85 +
86 +## Analysis
87 +
88 +Sources checked:
89 +
90 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
91 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
92 +- `.agents/sow/specs/topology-function-schema.md`
93 +- `.agents/skills/project-create-topology/SKILL.md`
94 +- `.agents/sow/done/SOW-0025-20260511-network-connections-modal-product-composition.md`
95 +- Cloud frontend `TODO-topology-modal-composition-contract.md`
96 +- Cloud topology service `AGENTS.md`
97 +- Cloud topology service `.agents/sow/specs/cloud-topology-service-contract.md`
98 +
99 +Current state:
100 +
101 +- `data.view.mode` can state `aggregated` or `detailed`, but the durable specs
102 + do not yet define layer-by-layer mode behavior.
103 +- `modal.labels` defines the label table shape, but cannot yet select important
104 + labels for the actor modal identification/header area.
105 +- Existing correlation docs assume pure correlation actors. The new model needs
106 + loose-side resolution for network-connections, replacement for SNMP/L2, and
107 + enrichment/table merging for streaming.
108 +- SOW-0025, SOW-0026, and SOW-0027 are narrower modal composition SOWs that
109 + depend on this broader contract.
110 +
111 +Risks:
112 +
113 +- A weak spec will recreate the same bug in three places: producer emits one
114 + meaning, aggregator merges another, UI renders a third.
115 +- Actor-per-`IP:PORT` detailed network-connections output can explode graph
116 + size and make the map unreadable.
117 +- Random conflict resolution in the aggregator can hide real dependencies.
118 +- Duplicating evidence rows for modal display can recreate the original payload
119 + size problem.
120 +- Actor labels may contain sensitive system metadata, users, command lines, or
121 + endpoint data. Durable artifacts must use synthetic examples only.
122 +
123 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
124 +
125 +Status at implementation start: ready (historical snapshot; final closure evidence is in the Execution Log and Validation sections).
126 +
127 +Problem / root-cause model:
128 +
129 +- The v1 topology contract optimized payload size and presentation tokens, but
130 + mode and correlation semantics were still incomplete. Network-connections,
131 + SNMP/L2, and streaming require different correlation outcomes: loose-side
132 + resolution, actor replacement, and actor enrichment.
133 +- Actor modals lost important identification because the schema exposes labels
134 + as a full table but does not let producers select the labels that belong in
135 + the modal header.
136 +
137 +Evidence reviewed:
138 +
139 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:179` defines `data.view.mode`.
140 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:469` defines current
141 + `data.correlation`.
142 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json:1097` defines current
143 + `modal_labels_presentation` without selected identification fields.
144 +- `.agents/sow/done/SOW-0025-20260511-network-connections-modal-product-composition.md`
145 + records the missing modal header label contract and the need for
146 + relationship-summary rows in aggregated network-connections.
147 +
148 +Affected contracts and surfaces:
149 +
150 +- Agent topology JSON schema and developer guide.
151 +- `topology:network-connections` producer.
152 +- `topology:snmp` producer/adapters, to ensure no false mode selector and to
153 + prepare replacement metadata.
154 +- `topology:streaming` producer, to ensure no false mode selector and to prepare
155 + merge/enrichment metadata.
156 +- Cloud frontend v1 normalizer and actor modal.
157 +- Cloud topology service aggregation core and request fanout.
158 +- Project topology skill and durable specs.
159 +
160 +Existing patterns to reuse:
161 +
162 +- Compact tables with `nullable` actor-ref columns.
163 +- Actor labels through `tables.actor.actor_labels`.
164 +- Type-level presentation under `types.actor_types.<id>.presentation` and
165 + `types.link_types.<id>.presentation`.
166 +- Modal sections over existing facts, not duplicate modal-only row stores.
167 +- Cloud topology service SOW framework and aggregator tests.
168 +- Cloud frontend v1 decoder/normalizer and modal table work.
169 +
170 +Risk and blast radius:
171 +
172 +- High semantic risk: wrong correlation can hide real dependencies or create
173 + false dependencies.
174 +- Medium payload risk: relationship-summary rows add payload size, but avoid
175 + much larger actor-per-socket graphs.
176 +- Medium UI risk: modal identification and loose-side materialization must not
177 + introduce topology-specific frontend code.
178 +- Low operational risk for Agent if schema validation and local Function output
179 + remain valid.
180 +
181 +Sensitive data handling plan:
182 +
183 +- Use only synthetic RFC 5737 IP ranges and synthetic host/process names in
184 + specs, SOWs, TODOs, tests, and examples.
185 +- Do not store raw Function captures, bearer tokens, cookies, usernames,
186 + command lines from local systems, SNMP communities, customer names, or
187 + customer-identifying public endpoints in durable artifacts.
188 +- Treat `actor_labels`, socket tuples, host labels, process command lines, and
189 + SNMP metadata as sensitive Function data.
190 +
191 +Implementation plan:
192 +
193 +1. Create the durable spec and cross-repo work items.
194 +2. Extend the Agent schema for modal label identification and mode/correlation
195 + metadata needed by the spec.
196 +3. Update topology developer docs and project topology skill.
197 +4. Update Agent producers:
198 + - network-connections mode and loose-side/relationship-summary semantics;
199 + - SNMP mode/correlation metadata where Agent owns it;
200 + - streaming merge/table metadata where Agent owns it.
201 +5. Update Cloud frontend to decode/render modal identification and supported
202 + mode/correlation metadata without domain-specific guesses.
203 +6. Update Cloud topology service to consume detailed inputs, correlate/merge by
204 + declared policies, and return detailed/aggregated outputs.
205 +7. Validate all touched repositories.
206 +
207 +Validation plan:
208 +
209 +- Agent: JSON schema validation, narrow C checks/build checks, and local
210 + Function payload checks for network-connections, SNMP, and streaming where
211 + available.
212 +- UI: unit tests for decoder/normalizer/modal identification and local build or
213 + focused test command where available.
214 +- Aggregator: Go tests for request fanout mode rewrite, loose-side resolution,
215 + SNMP replacement, streaming enrichment, and table merge policies.
216 +- Same-failure search for stale pure-correlation-only guidance.
217 +
218 +Artifact impact plan:
219 +
220 +- AGENTS.md: likely unchanged; existing SOW and public-skill boundary rules are
221 + already clear.
222 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md`
223 + so future topology work follows this spec.
224 +- Specs: add
225 + `.agents/sow/specs/topology-modes-correlation-aggregation.md` and update
226 + `.agents/sow/specs/topology-function-schema.md` if needed.
227 +- End-user/operator docs: likely unchanged because this is developer/internal
228 + topology schema work.
229 +- End-user/operator skills: unchanged unless a public querying skill documents
230 + developer validation, which must remain avoided.
231 +- SOW lifecycle: pause SOW-0025, keep SOW-0026 and SOW-0027 pending, create
232 + aggregator SOW and UI TODO, and map all remaining follow-ups before close.
233 +
234 +Open-source reference evidence:
235 +
236 +- Not checked for this gate. This contract is Netdata-specific and is defined
237 + by existing Agent, Cloud frontend, and Cloud topology service behavior.
238 +
239 +Open decisions:
240 +
241 +- No user decision is currently blocking the specification. The user already
242 + selected the key product direction: spec first, then durable repo work items,
243 + then implementation.
244 +
245 +## Implications And Decisions
246 +
247 +### Decision 1: Treat Actor Identification As Part Of This Contract
248 +
249 +Selection: implement in SOW-0028.
250 +
251 +Reasoning:
252 +
253 +- Actor identification affects schema, producer payloads, UI rendering, and
254 + aggregator preservation.
255 +- Keeping it separate would leave actor modals visually incomplete even if mode
256 + and correlation semantics are fixed.
257 +
258 +### Decision 2: Aggregator Consumes Detailed Input
259 +
260 +Selection: Cloud aggregator rewrites `__topology_mode=aggregated` fanout to
261 +`__topology_mode=detailed` only for producers that support the mode.
262 +
263 +Reasoning:
264 +
265 +- Exact cross-node socket correlation needs detailed tuples.
266 +- Aggregating before correlation loses facts that cannot be recovered.
267 +- SNMP/L2 and streaming currently do not have meaningful detailed/aggregated
268 + producer modes, so the aggregator should not send an invented parameter.
269 +
270 +### Decision 3: Network-Connections Detailed Uses Loose Sides
271 +
272 +Selection: known actors remain actors; unknown remote socket peers are
273 +loose-side facts until the aggregator or UI materializes them according to the
274 +schema.
275 +
276 +Reasoning:
277 +
278 +- Actor-per-`IP:PORT` detailed output can explode graph cardinality.
279 +- Exact tuples are still preserved for matching and drilldown.
280 +
281 +### Decision 4: SNMP Uses Replacement, Streaming Uses Enrichment
282 +
283 +Selection:
284 +
285 +- SNMP/L2 aggregation replaces weaker placeholder actors with stronger managed
286 + actors.
287 +- Streaming aggregation merges/enriches actors by `machine_guid` and table
288 + policy.
289 +
290 +Reasoning:
291 +
292 +- SNMP placeholder actors and managed devices represent the same physical
293 + entity at different confidence levels.
294 +- Streaming payloads from multiple parents may contain complementary facts for
295 + the same node; losing either side is wrong.
296 +
297 +## Cross-Repo Pending Checklist
298 +
299 +Agent repository:
300 +
301 +- Add/maintain the spec under `.agents/sow/specs/`.
302 +- Update `FUNCTION_TOPOLOGY_SCHEMA.json`.
303 +- Update `FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`.
304 +- Update `.agents/sow/specs/topology-function-schema.md` as needed.
305 +- Update `.agents/skills/project-create-topology/SKILL.md`.
306 +- Implement producer changes for network-connections, SNMP/L2, and streaming
307 + as far as the Agent owns the emitted payload.
308 +
309 +Cloud frontend repository:
310 +
311 +- Create `TODO-topology-mode-correlation-aggregation.md`.
312 +- Implement modal label identification rendering.
313 +- Implement mode capability behavior and hide no-op mode toggles.
314 +- Preserve and render loose-side/materialization metadata without
315 + domain-specific guesses.
316 +- Keep old-schema support isolated.
317 +
318 +Cloud topology service repository:
319 +
320 +- Create a new current SOW for topology mode, loose-side resolution,
321 + replacement, enrichment, and table merge policy.
322 +- Implement request fanout mode rewrite.
323 +- Implement generic rule classes and table merge policies.
324 +- Add fixtures/tests for network-connections, SNMP/L2, and streaming.
325 +
326 +## Plan
327 +
328 +1. Write the spec and create cross-repo work items.
329 +2. Patch Agent schema/docs/skill.
330 +3. Patch Agent producers and validate local payloads.
331 +4. Patch Cloud frontend and run focused tests.
332 +5. Patch Cloud topology service and run Go tests.
333 +6. Update SOW validation and follow-up mapping.
334 +
335 +## Execution Log
336 +
337 +### 2026-05-11
338 +
339 +- Created the cross-topology mode/correlation/aggregation spec.
340 +- Paused SOW-0025 because it depends on this broader contract.
341 +- Created Cloud frontend handoff TODO:
342 + `<cloud-frontend-repo>/TODO-topology-mode-correlation-aggregation.md`.
343 +- Created and completed Cloud topology service SOW:
344 + `<cloud-topology-service-repo>/.agents/sow/done/SOW-0011-20260511-topology-mode-correlation-aggregation.md`.
345 +- Updated Agent schema, developer guide, project topology skill, Go topology
346 + structs/validation, network-connections producer, SNMP topology tests, and
347 + streaming actor modal identification.
348 +- Implemented UI support for modal label identification and hiding
349 + `__topology_mode` controls when `data.view.supported_modes` does not
350 + advertise a real split.
351 +- Implemented Cloud topology service compatibility for supported modes, modal
352 + label identification, correlation rule classes, fanout mode rewrite helpers,
353 + and detail-table dedupe/merge policies.
354 +
355 +## Validation
356 +
357 +Acceptance criteria evidence:
358 +
359 +- Spec:
360 + - `.agents/sow/specs/topology-modes-correlation-aggregation.md` defines the
361 + layer responsibilities and examples for network-connections, SNMP/L2, and
362 + streaming.
363 +- Agent schema/docs:
364 + - `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` includes
365 + `data.view.supported_modes`, modal label identification, port source
366 + `value_column`, correlation rule `class`, and expanded table aggregation
367 + tokens.
368 + - `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md` documents the new
369 + fields.
370 + - `.agents/skills/project-create-topology/SKILL.md` was updated for the
371 + developer workflow.
372 +- Agent producers:
373 + - `src/collectors/network-viewer.plugin/network-viewer.c` accepts
374 + `__topology_mode`, advertises supported modes, emits selected actor modal
375 + identification labels, emits relationship-summary rows for aggregated
376 + connections, and marks socket correlation rules with
377 + `class: resolve_loose_side`.
378 + - `src/web/api/functions/function-topology-streaming.c` emits selected actor
379 + modal identification labels and remains mode-invariant.
380 + - `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go` remains
381 + mode-invariant and is covered by updated tests.
382 +- Cloud frontend:
383 + - `src/domains/functions/topology/v1/buildModalPresentation.js` decodes
384 + modal label identification.
385 + - `src/domains/functions/components/topology/actorModal/index.js` renders
386 + selected v1 identification labels in the actor modal header.
387 + - `src/domains/functions/useFetch/normalizers/topology/index.js` hides
388 + `__topology_mode` unless `supported_modes` advertises both modes.
389 +- Cloud topology service:
390 + - `internal/topology/schema/payload.go`, `internal/topology/validate/validate.go`,
391 + `internal/topology/mode/mode.go`, and
392 + `internal/topology/aggregate/aggregate.go` implement the service-side
393 + compatibility layer.
394 +
395 +Tests or equivalent validation:
396 +
397 +- Agent:
398 + - `git diff --check` passed.
399 + - `go test ./pkg/topology/v1 ./plugin/go.d/collector/snmp_topology` passed
400 + from `src/go`.
401 + - `sudo -n cmake --build build --target network-viewer.plugin -- -j2`
402 + passed after non-sudo build was blocked by existing build-directory
403 + permissions.
404 + - `.agents/sow/audit.sh` passed with existing non-project-skill
405 + classification warnings.
406 +- UI:
407 + - `git diff --check` passed.
408 + - `yarn test src/domains/functions/topology/v1/buildModalPresentation.test.js src/domains/functions/useFetch/normalizers/topology/index.test.js --runInBand`
409 + passed.
410 +- Cloud topology service:
411 + - `go test ./internal/topology/...` passed.
412 + - `go test ./...` passed.
413 + - `go vet ./internal/topology/...` passed.
414 + - `git diff --check` passed.
415 + - `.agents/sow/audit.sh` passed.
416 +
417 +Real-use evidence:
418 +
419 +- Installed Function checks through token-safe Cloud-proxied Agent calls passed
420 + on 2026-05-11:
421 + - `topology:network-connections ... mode:aggregated` returned status 200,
422 + `view.mode: aggregated`, `view.supported_modes: [aggregated, detailed]`,
423 + actor rows, link rows, `actor_labels`, `socket_ports`, relationship table
424 + `connections`, and `data.correlation`.
425 + - `topology:network-connections ... mode:detailed` returned status 200,
426 + `view.mode: detailed`, the same supported modes, graph rows, socket
427 + evidence rows, and `data.correlation`.
428 + - `topology:streaming` returned status 200, no `supported_modes`, and modal
429 + identification metadata for its actor types.
430 + - `topology:snmp` returned status 200 and no `supported_modes`, so the UI
431 + must not show a fake detailed/aggregated toggle. SNMP-specific modal
432 + identification polish remains tracked by SOW-0026.
433 +
434 +Reviewer findings:
435 +
436 +- No external reviewer loop was run for this SOW pass. The user asked for
437 + implementation continuity after the spec/SOW/TODO artifacts were created.
438 +
439 +Same-failure scan:
440 +
441 +- `rg` checks were used across Agent, UI, and Cloud topology service for the
442 + new contract fields: `supported_modes`, `identification`, `class`,
443 + `value_column`, `merge_metrics`, `set_union`, `__topology_mode`, and
444 + `RewriteFunctionCall`.
445 +
446 +Sensitive data gate:
447 +
448 +- Passed by inspection for this pass. Durable artifacts contain synthetic
449 + examples only; no raw Function captures, bearer tokens, cookies, usernames,
450 + command lines from local systems, SNMP communities, customer names,
451 + customer-identifying public endpoints, private endpoints, raw node IDs, or
452 + raw machine GUIDs were written.
453 +
454 +Artifact maintenance gate:
455 +
456 +- AGENTS.md: unchanged; existing SOW, public-skill boundary, and artifact
457 + maintenance rules already cover this workflow.
458 +- Runtime project skills: updated
459 + `.agents/skills/project-create-topology/SKILL.md`.
460 +- Specs: added
461 + `.agents/sow/specs/topology-modes-correlation-aggregation.md` and updated
462 + `.agents/sow/specs/topology-function-schema.md`.
463 +- End-user/operator docs: unaffected; this is an internal developer topology
464 + schema/producer/service/UI contract.
465 +- End-user/operator skills: unaffected; no operator querying skill was changed.
466 +- SOW lifecycle: SOW-0025 remains paused; SOW-0026 and SOW-0027 remain pending;
467 + SOW-0029 tracks true detailed loose-side graph rows; SOW-0028 is completed
468 + and moved to `done/`.
469 +
470 +Specs update:
471 +
472 +- Added `.agents/sow/specs/topology-modes-correlation-aggregation.md`.
473 +
474 +Project skills update:
475 +
476 +- Updated `.agents/skills/project-create-topology/SKILL.md`.
477 +
478 +End-user/operator docs update:
479 +
480 +- Not affected.
481 +
482 +End-user/operator skills update:
483 +
484 +- Not affected.
485 +
486 +Lessons and follow-up mapping:
487 +
488 +- Completed in the final Lessons Extracted and Followup sections below.
489 +
490 +## Outcome
491 +
492 +Implementation checkpoint is complete across Agent, Cloud frontend, and Cloud
493 +topology service for the documented schema compatibility layer:
494 +
495 +- schema/docs/specs/skill were updated first;
496 +- Agent producers and topology validation were updated;
497 +- UI modal identification and mode-control behavior were updated;
498 +- Cloud topology service schema/validation/mode/table-policy compatibility was
499 + updated;
500 +- focused validation passed in all three repositories.
501 +
502 +The SOW is complete for the implemented cross-repo compatibility contract.
503 +Remaining one-sided detailed network-connections graph rows are explicitly
504 +split to SOW-0029.
505 +
506 +## Lessons Extracted
507 +
508 +- The durable spec/SOW/TODO order worked: after compaction, the repo artifacts
509 + were enough to recover the intended cross-repo work without relying on chat
510 + memory.
511 +- Loose-side rows are the hardest part of the model because they affect schema
512 + shape, UI graph materialization, and aggregation execution together. The
513 + current checkpoint preserves exact socket facts and correlation metadata but
514 + does not yet switch the Agent producer to one-sided detailed graph rows.
515 +
516 +## Followup
517 +
518 +- True detailed one-sided network-connections graph rows and UI
519 + materialization are tracked by
520 + `.agents/sow/pending/SOW-0029-20260511-network-connections-detailed-loose-sides.md`.
521 + The current Agent producer still emits endpoint actors for local direct
522 + views while preserving exact socket evidence for correlation.
523 +- Cloud topology service production fanout/fetch integration remains tracked by
524 + its service handoff gates; SOW-0028 implemented the internal mode rewrite
525 + helper and aggregation compatibility layer.
526 +- SOW-0025, SOW-0026, and SOW-0027 remain the function-specific modal product
527 + polish work for network-connections, SNMP/L2, and streaming.
528 +
529 +## Regression Log
530 +
531 +None yet.
.agents/sow/done/SOW-0031-20260517-topology-v1-zero-heuristic-rendering-contract.md new
+428
@@ -0,0 +1,428 @@
1 +# SOW-0031 - Topology V1 Zero-Heuristic Rendering Contract
2 +
3 +## Status
4 +
5 +Status: completed
6 +
7 +`completed` is the successful terminal status. `done` is a directory name, not a status value. Do not use `Status: done` or `Status: complete`.
8 +
9 +Sub-state: implemented, validated, installed by the user, and committed across the Agent, Cloud frontend, and Cloud topology service repositories.
10 +
11 +## Requirements
12 +
13 +### Purpose
14 +
15 +Make `netdata.topology.v1` fit for polished, topology-agnostic graph rendering. The UI must not hardcode domain words such as self, segment, endpoint, device, SNMP, LLDP, CDP, parent, child, client, server, router, or switch when rendering v1 payloads.
16 +
17 +### User Request
18 +
19 +Create a SOW, analyze the remaining frontend heuristics, propose the missing schema fields, update local schema/spec/docs/skills, create Cloud aggregator and Cloud frontend handoff artifacts, then implement the approved zero-heuristic contract in the Agent producers/shared helpers, Cloud frontend, and Cloud topology service.
20 +
21 +### Assistant Understanding
22 +
23 +Facts:
24 +
25 +- The current v1 schema already carries actor/link/port presentation, modal recipes, legend, highlight behavior, port-bullet sources, and link layout distance/strength.
26 +- The Cloud frontend report identified remaining heuristics outside the v1 decoder/modal path: self detection, segment/endpoint/device/inferred detection, SNMP/LLDP/CDP detection, hardcoded search paths, capability-to-icon fallback, and v1-to-legacy `protocol`/port shims.
27 +- Raw SVG icons are explicitly not allowed by the current topology documentation and should remain disallowed.
28 +
29 +Inferences:
30 +
31 +- The missing contract is not more producer-specific fields. The missing contract is a small set of topology-agnostic type-level policies that let the UI treat v1 as data-driven.
32 +- Link behavior classification belongs on `link_types.<id>.semantic_role`, not under `presentation`, because discovery, ownership, traffic, correlation, and control are graph semantics. Visual appearance remains under `presentation`.
33 +
34 +Unknowns:
35 +
36 +- Exact numeric UI mappings for `size.scale` and `layout.repulsion` are UI-owned and must be tuned visually after implementation.
37 +
38 +### Acceptance Criteria
39 +
40 +- Agent repo has an active SOW with the zero-heuristic contract and implementation boundary.
41 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` defines the proposed optional schema fields without requiring producer migration immediately.
42 +- Local topology spec, developer guide, implementation scope, and project topology skill document the new contract.
43 +- Cloud frontend implements v1 renderer behavior from the TODO while preserving legacy behavior in the legacy path.
44 +- Cloud topology service decodes, preserves, namespaces/deduplicates, and emits the new fields.
45 +- Agent shared topology helpers and producers emit the new fields for network-connections, SNMP/L2, and streaming.
46 +- Validation covers Agent schema/producer fixtures, Cloud frontend tests, Cloud topology service tests, and real local topology payload smoke checks where practical.
47 +
48 +## Analysis
49 +
50 +Sources checked:
51 +
52 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
53 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
54 +- `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`
55 +- `.agents/sow/specs/topology-function-schema.md`
56 +- `.agents/sow/specs/topology-modes-correlation-aggregation.md`
57 +- `.agents/skills/project-create-topology/SKILL.md`
58 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/utils.js`
59 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/components/graph/forceGraph.js`
60 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/components/graph/useForceSimulation.js`
61 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/components/topology/actorModal/portTable.js`
62 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/v1/buildLinks.js`
63 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/v1/buildRenderableLinks.js`
64 +
65 +Current state:
66 +
67 +- Actor presentation exists, but actor size only has `mode` and optional `metric_column`; it lacks a type-level fixed scale token.
68 +- Link presentation has `layout.strength` and `layout.distance`; actor presentation lacks a corresponding repulsion token.
69 +- Link type has `direction_role`, but no generic semantic classification for discovery, ownership, traffic, correlation, or control behavior.
70 +- Actor type has no search contract, so the frontend still indexes hardcoded paths from legacy `details.match`, `details.attributes`, and `details.labels`.
71 +- Icon tokens are closed, which is correct, but producers cannot fully replace capability-based frontend icon inference unless every v1 actor type declares a suitable token.
72 +
73 +Risks:
74 +
75 +- If the UI keeps heuristics in v1, every new topology can regress visually when its actor/link names do not match existing frontend guesses.
76 +- If semantics are placed under `presentation`, the aggregator may need to parse a visual object to make graph-behavior decisions.
77 +- If raw SVG is allowed, topology payloads become a new script/rendering attack surface and every consumer must sanitize untrusted markup.
78 +- If repulsion reuses link strength without a separate field name, producers and UI developers will conflate two different force-graph quantities.
79 +
80 +## Proposed Schema
81 +
82 +### Actor Type Search
83 +
84 +```json
85 +{
86 + "types": {
87 + "actor_types": {
88 + "process": {
89 + "search": {
90 + "enabled": true,
91 + "columns": ["display_name", "process_name"],
92 + "label_keys": ["cmdline", "username"]
93 + }
94 + }
95 + }
96 + }
97 +}
98 +```
99 +
100 +Rules:
101 +
102 +- `search.columns[]` references actor-table scalar columns.
103 +- `search.label_keys[]` references `actor_labels.key` values.
104 +- `search.enabled: false` removes helper actors from graph search.
105 +- UI must not traverse producer-specific `details`, `match`, `attributes`, or label paths for v1 search.
106 +
107 +### Actor Presentation Size And Repulsion
108 +
109 +```json
110 +{
111 + "presentation": {
112 + "size": {
113 + "mode": "metric",
114 + "metric_column": "socket_count",
115 + "scale": "normal"
116 + },
117 + "layout": {
118 + "repulsion": "normal"
119 + }
120 + }
121 +}
122 +```
123 +
124 +Rules:
125 +
126 +- `size.scale` is `compact`, `normal`, or `emphasized`.
127 +- `layout.repulsion` is `weakest`, `weaker`, `normal`, `stronger`, or `strongest`.
128 +- Producers emit tokens only. The UI owns numeric radius and charge mappings.
129 +- Actor repulsion is separate from link strength.
130 +- `size.scale` composes with `size.mode`.
131 +- Missing `size.scale` and missing `layout.repulsion` use `normal`; the UI must
132 + not fall back to self/device/SNMP/endpoint heuristics for v1.
133 +- Initial UI-owned mappings are `compact=0.85`, `normal=1.0`,
134 + `emphasized=1.18` for size scale and `weakest=-200`, `weaker=-300`,
135 + `normal=-450`, `stronger=-700`, `strongest=-1000` for repulsion. These
136 + numbers are not schema.
137 +
138 +### Link Semantic Role
139 +
140 +```json
141 +{
142 + "types": {
143 + "link_types": {
144 + "lldp": {
145 + "orientation": "observed_bidirectional",
146 + "direction_role": "observation",
147 + "semantic_role": "discovery",
148 + "aggregation": {
149 + "direction": "canonicalize_unordered",
150 + "evidence": "append"
151 + }
152 + }
153 + }
154 + }
155 +}
156 +```
157 +
158 +Allowed roles:
159 +
160 +- `normal`
161 +- `discovery`
162 +- `ownership`
163 +- `traffic`
164 +- `correlation`
165 +- `control`
166 +
167 +Rules:
168 +
169 +- `semantic_role` drives behavior such as discovery filtering, ownership/coherence handling, traffic emphasis, and correlation treatment.
170 +- Link appearance still comes from `presentation`.
171 +- UI and aggregator must not infer role from `link.type`, `link.protocol`, LLDP/CDP string checks, or label names.
172 +- Day-1 UI behavior only requires a concrete behavior difference for
173 + `semantic_role: discovery`. Other semantic roles are preserved and render
174 + through presentation until future behavior is specified.
175 +
176 +### Arrow Auto
177 +
178 +`presentation.arrow` remains the authoritative visual signal. When it is
179 +`auto` or omitted, the UI derives arrows from `orientation` and
180 +`direction_role`:
181 +
182 +- `undirected` -> no arrow;
183 +- `observed_bidirectional` -> no arrow;
184 +- `direction_role: none` -> no arrow;
185 +- `direction_role: observation` -> no arrow;
186 +- `directed` with `flow` or `dependency` -> forward from `src_actor` to
187 + `dst_actor`;
188 +- `hierarchical` with `ownership` -> forward from `src_actor` to `dst_actor`;
189 +- all other combinations -> no arrow and a diagnostic if the combination is
190 + schema-valid but semantically unusual.
191 +
192 +`observed_bidirectional` does not mean draw arrows at both ends. Producers must
193 +set `presentation.arrow: "both"` or `"reverse"` explicitly when needed.
194 +
195 +`direction_role` is required by the v1 schema. Missing `direction_role` is
196 +invalid input and must not produce an inferred arrow from `orientation:
197 +"directed"` alone. The UI should render `auto` as no arrow and emit the normal
198 +missing/invalid-field diagnostic.
199 +
200 +For schema-valid values, the semantic diagnostic boundary is explicit:
201 +
202 +- no diagnostic for `directed+flow`, `directed+dependency`,
203 + `hierarchical+ownership`, `undirected+none`, `undirected+observation`,
204 + `observed_bidirectional+none`, or `observed_bidirectional+observation`;
205 +- diagnostic for `directed+none`, `directed+observation`,
206 + `directed+ownership`, `hierarchical+none`, `hierarchical+flow`,
207 + `hierarchical+dependency`, `hierarchical+observation`, `undirected+flow`,
208 + `undirected+dependency`, `undirected+ownership`,
209 + `observed_bidirectional+flow`, `observed_bidirectional+dependency`, or
210 + `observed_bidirectional+ownership`.
211 +
212 +### Closed Icon Tokens Only
213 +
214 +Allowed icons remain schema-owned tokens. This SOW adds generic tokens needed to remove capability inference:
215 +
216 +- `device`
217 +- `endpoint`
218 +- `correlation`
219 +- `interface`
220 +- `group`
221 +- `unknown`
222 +
223 +Rules:
224 +
225 +- Raw SVG remains disallowed.
226 +- Capability-to-icon inference moves to producers, where producer-specific capabilities are already known.
227 +- UI maps icon tokens to safe bundled icons only.
228 +
229 +## Pre-Implementation Gate (Historical Snapshot at Implementation Start)
230 +
231 +Status: approved
232 +
233 +Problem / root-cause model:
234 +
235 +- The v1 payload is mostly schema-driven, but renderer behavior still depends on legacy frontend heuristics. The root cause is missing v1 contract fields for actor search, actor fixed emphasis, actor repulsion, and link semantic behavior.
236 +
237 +Evidence reviewed:
238 +
239 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` already defines `actor_type.presentation`, `link_type.presentation`, and link `presentation.layout`.
240 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` had no actor `search`, no actor `presentation.layout`, no `size.scale`, and no link `semantic_role` before this SOW.
241 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/utils.js` contains actor/link kind heuristics such as self, segment, endpoint/derived, inferred, device, SNMP, LLDP, and CDP detection.
242 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/components/graph/forceGraph.js` contains hardcoded graph search path extraction and icon fallback from capability inference.
243 +- `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/v1/buildLinks.js` mirrors v1 row fields into legacy `protocol`, `sourcePort`, and `targetPort` properties only for legacy renderer paths.
244 +
245 +Affected contracts and surfaces:
246 +
247 +- Agent topology JSON schema.
248 +- Agent topology developer guide.
249 +- Agent topology durable spec.
250 +- Agent project topology skill.
251 +- Cloud frontend v1 decoder, renderer, search, force simulation, legend, icon mapping, port table, and legacy adapter.
252 +- Cloud topology service type registry decode, aggregation merge, type namespace/dedup, and returned schema preservation.
253 +- Producer migrations for network-connections, SNMP/L2, streaming, and future vSphere v1.
254 +
255 +Existing patterns to reuse:
256 +
257 +- Closed token pattern from color, opacity, width, icon, link layout strength, and link layout distance.
258 +- Existing type registry pattern for actor/link/port type presentation.
259 +- Existing frontend diagnostics pattern for unknown tokens.
260 +- Existing Cloud aggregator namespacing/dedup behavior for type definitions.
261 +
262 +Risk and blast radius:
263 +
264 +- UI renderer refactor has high visual regression risk because it touches the graph hot path.
265 +- Producer migration is broad but optional fields allow incremental rollout.
266 +- Aggregator should preserve and merge the new fields; it must not invent type semantics.
267 +- Raw SVG remains rejected to avoid new untrusted markup risk.
268 +
269 +Sensitive data handling plan:
270 +
271 +- This planning pass uses only schema/docs/code paths and sanitized descriptions. Durable artifacts must not include raw payload captures, credentials, cookies, bearer tokens, SNMP communities, customer names, personal data, non-private customer-identifying IPs, or private endpoints.
272 +
273 +Implementation plan:
274 +
275 +1. Update shared Go topology structs and validators for `actor.search`, `actor.presentation.size.scale`, `actor.presentation.layout.repulsion`, and `link.semantic_role`.
276 +2. Update each Agent producer to emit the new optional fields for every v1 actor/link type it owns.
277 +3. Implement Cloud frontend behavior using the TODO created by this SOW.
278 +4. Implement Cloud topology service behavior using the SOW created by this SOW.
279 +5. Validate with real topology payloads across network-connections, SNMP/L2, and streaming.
280 +
281 +Validation plan:
282 +
283 +- Validate JSON schema syntax with `jq`.
284 +- Validate SOW status/directory consistency with project audit.
285 +- During implementation, add schema validation fixtures and UI/aggregator tests that prove v1 rendering does not call legacy heuristics.
286 +
287 +Artifact impact plan:
288 +
289 +- AGENTS.md: no update expected; this does not change project-wide workflow.
290 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md`.
291 +- Specs: update `.agents/sow/specs/topology-function-schema.md`.
292 +- End-user/operator docs: no update expected; this is developer/schema contract work.
293 +- End-user/operator skills: no update expected; this is not an operator workflow.
294 +- SOW lifecycle: SOW-0031 is the active implementation ledger for this approved cross-repo work.
295 +
296 +Open-source reference evidence:
297 +
298 +- None checked. This is a local Netdata schema/frontend/aggregator contract gap, and the user requested local contract handoff rather than external product research.
299 +
300 +Open decisions:
301 +
302 +- None. The user approved implementation after the frontend TODO follow-up clarifications were recorded.
303 +
304 +## Implications And Decisions
305 +
306 +1. User accepted adding a schema contract to remove frontend v1 heuristics.
307 +2. User accepted keeping raw SVG out of the payload.
308 +3. User accepted adding a dedicated SOW/TODO handoff for Cloud frontend and Cloud aggregator before implementation.
309 +
310 +## Plan
311 +
312 +1. Record the zero-heuristic schema contract in local Agent schema/spec/docs/skill artifacts.
313 +2. Create a Cloud frontend TODO that asks the UI agent to remove v1 renderer heuristics only after schema support lands.
314 +3. Create a Cloud topology service SOW that asks the aggregator agent to decode, preserve, namespace, deduplicate, and emit the new optional fields without inventing producer semantics.
315 +4. Implement the approved contract in Agent, Cloud frontend, and Cloud topology service.
316 +
317 +## Execution Log
318 +
319 +### 2026-05-17
320 +
321 +- Created this SOW.
322 +- Added optional JSON schema fields for actor search, actor size scale, actor layout repulsion, link semantic role, and generic closed icon tokens.
323 +- Updated local topology spec, developer guide, implementation scope, and project topology skill.
324 +- Created Cloud frontend and Cloud topology service handoff artifacts.
325 +- Appended backend answers to frontend follow-up questions: arrow auto mapping,
326 + direction_role UI usage, initial UI-owned numeric mappings, transitional
327 + neutral defaults, and current aggregator unknown-field behavior.
328 +- Appended final `arrow: auto` clarifications covering required
329 + `direction_role`, current producer status, and the semantic diagnostic
330 + boundary.
331 +- User approved implementation after the Cloud frontend TODO review loop.
332 +
333 +## Validation
334 +
335 +Acceptance criteria evidence:
336 +
337 +- SOW completed at `.agents/sow/done/SOW-0031-20260517-topology-v1-zero-heuristic-rendering-contract.md`.
338 +- JSON schema updated at `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
339 +- Spec/docs/skill updates are present in `.agents/sow/specs/topology-function-schema.md`, `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md`, and `.agents/skills/project-create-topology/SKILL.md`.
340 +- Cloud frontend TODO created as `${CLOUD_FRONTEND_REPO}/TODO-topology-v1-zero-heuristic-rendering-contract.md`.
341 +- Cloud topology service SOW completed as `${CLOUD_TOPOLOGY_SERVICE_REPO}/.agents/sow/done/SOW-0017-20260517-topology-v1-zero-heuristic-contract.md`.
342 +- Agent topology structs and validators implement actor search, actor size scale, actor layout repulsion, link semantic role, and closed icon tokens in `src/go/pkg/topology/v1/types.go` and `src/go/pkg/topology/v1/validate.go`.
343 +- Agent producers emit the contract fields in `src/collectors/network-viewer.plugin/network-viewer.c`, `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`, and `src/web/api/functions/function-topology-streaming.c`.
344 +- Cloud frontend v1 rendering consumes schema-driven search, size, repulsion, semantic role, and port lookup fields in `${CLOUD_FRONTEND_REPO}`.
345 +- Cloud topology service decodes, validates, preserves, and namespaces conflicting definitions for the new fields in `${CLOUD_TOPOLOGY_SERVICE_REPO}`.
346 +
347 +Tests or equivalent validation:
348 +
349 +- `jq empty src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` passed.
350 +- Agent repo `.agents/sow/audit.sh` accepted SOW-0031 status/directory placement and reported no sensitive-data findings. The audit still reports unrelated pre-existing framework warnings: one older done SOW has a status/directory mismatch, and legacy non-project skill directories need classification.
351 +- Cloud topology service `.agents/sow/audit.sh` passed cleanly after creating SOW-0017.
352 +- `go test ./pkg/topology/v1 ./plugin/go.d/collector/snmp_topology` passed from `src/go`.
353 +- `git diff --check` passed in the Agent repo.
354 +- `git diff --check` passed in the Cloud frontend repo.
355 +- `git diff --check` passed in the Cloud topology service repo.
356 +- Cloud frontend focused tests passed for v1 normalization, presentation adapter, renderable links, color token handling, port utilities, and force simulation helpers.
357 +- Cloud topology service tests passed: `go test ./internal/topology/schema ./internal/topology/validate ./internal/topology/aggregate`.
358 +- Full Cloud frontend test suite passed: 269 suites passed, 2,139 tests passed, 6 skipped, 7 snapshots passed.
359 +- Agent C build validation is blocked by local filesystem permissions: `build`, `build/.ninja_log`, and `build/.ninja_deps` are owned by `root:root`; `ninja -C build` cannot create `build/.ninja_lock` in this worktree.
360 +- User later installed the Agent and UI successfully from the updated worktrees.
361 +
362 +Real-use evidence:
363 +
364 +- User reported the Agent and UI were installed and running after implementation.
365 +
366 +Reviewer findings:
367 +
368 +- Frontend agent report is the input to this SOW. Additional external review can be run after implementation validation if requested.
369 +
370 +Same-failure scan:
371 +
372 +- `rg` verified the new contract terms appear in the schema, developer guide, implementation scope, durable spec, project skill, and SOW.
373 +
374 +Sensitive data gate:
375 +
376 +- Durable artifacts use only schema names, file paths, sanitized repo placeholders, and generic examples. No raw secrets, credentials, bearer tokens, SNMP communities, personal names, customer identifiers, non-private customer-identifying IPs, private endpoints, or proprietary incidents are included.
377 +
378 +Artifact maintenance gate:
379 +
380 +- AGENTS.md: not changed; no workflow or project-wide guardrail changed.
381 +- Runtime project skills: `.agents/skills/project-create-topology/SKILL.md` updated.
382 +- Specs: `.agents/sow/specs/topology-function-schema.md` updated.
383 +- End-user/operator docs: not affected; this is developer schema work.
384 +- End-user/operator skills: not affected; no operator workflow changed.
385 +- SOW lifecycle: SOW is `completed` in `done/`; implementation and lifecycle closure are committed in follow-up commits because the first implementation commit accidentally left the SOWs in progress.
386 +
387 +Specs update:
388 +
389 +- `.agents/sow/specs/topology-function-schema.md` updated.
390 +
391 +Project skills update:
392 +
393 +- `.agents/skills/project-create-topology/SKILL.md` updated.
394 +
395 +End-user/operator docs update:
396 +
397 +- Not affected. The change is internal topology producer/UI/aggregator contract behavior.
398 +
399 +End-user/operator skills update:
400 +
401 +- Not affected. Public query skills do not teach topology producer development or frontend renderer internals.
402 +
403 +Lessons:
404 +
405 +- v1 schemas need behavior contracts for renderer decisions, not only visual tokens. Otherwise legacy UI heuristics leak into new topology types.
406 +
407 +Follow-up mapping:
408 +
409 +- Cloud frontend implementation is committed in `${CLOUD_FRONTEND_REPO}`.
410 +- Cloud topology service implementation is committed in `${CLOUD_TOPOLOGY_SERVICE_REPO}`.
411 +- No remaining SOW follow-up is open for this contract.
412 +
413 +## Outcome
414 +
415 +Completed. Core Agent, Cloud frontend, and Cloud topology service code paths are implemented, validated, installed by the user, and committed.
416 +
417 +## Lessons Extracted
418 +
419 +- The v1 renderer cannot become topology-agnostic from presentation colors alone. It needs producer-owned behavior tokens for search, discovery behavior, actor sizing, and graph-layout repulsion.
420 +- Aggregators must namespace contradictory type definitions instead of inventing precedence rules for producer-owned presentation or behavior metadata.
421 +
422 +## Followup
423 +
424 +- None.
425 +
426 +## Regression Log
427 +
428 +None yet.
.agents/sow/pending/SOW-0002-20260501-unified-multi-layered-topology-schema.md
+81 -17
@@ -4,7 +4,12 @@
4
5 Status: open
6
7 -Sub-state: scope captured, awaiting user decisions on merge semantics, identity-matching policy, conflict resolution, storage model, and scale targets before implementation planning advances.
7 +Sub-state: scope captured, awaiting user decisions on merge semantics,
8 +cross-layer identity-matching policy, conflict resolution, storage model, and
9 +scale targets before implementation planning advances. Immediate
10 +detailed/aggregated topology Function payload migration is owned by SOW-0020.
11 +Same-kind producer-visible correlation points/claims are owned by SOW-0023.
12 +This SOW remains the future unified cross-layer merge/correlation work.
13
14 ## Requirements
15
@@ -22,9 +27,17 @@ This SOW captures the problem space, references existing prior planning (`TODO-U
27
28 ### Assistant Understanding
29
25 -Facts (verified):
26 -
27 -- The current schema already exists at `src/go/pkg/topology/types.go`. Each `Actor` has a `Layer` field, a `Source` field, and a `Match` struct with extended identity fields: `ChassisIDs`, `MacAddresses`, `IPAddresses`, `Hostnames`, `DNSNames`, `SysObjectID`, `SysName`, `NetdataNodeID`, `NetdataMachineGUID`, `CloudInstanceID`, `CloudAccountID`, `ContainerIDs`, `PodNames`, `NamespaceIDs` (`types.go:7-22`). Each `Link` carries `Layer`, `Protocol`, `LinkType`, source/destination endpoints, direction, state, timestamps, and metrics (`types.go:42-55`).
30 +Facts (verified when this SOW was opened, updated 2026-05-10):
31 +
32 +- Historical note: this SOW originally referenced the legacy
33 + `src/go/pkg/topology/types.go` `Actor`/`Link`/`Match` model. That model is no
34 + longer the target contract for new topology Function payloads.
35 +- The current producer-facing contract is `netdata.topology.v1` in
36 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`, with durable semantics recorded
37 + in `.agents/sow/specs/topology-function-schema.md`.
38 +- SOW-0023 adds producer-visible `data.correlation` rules, points, and claims
39 + for same-kind correlation. Future unified cross-layer work must reuse that
40 + plane where it fits instead of resurrecting a parallel legacy `Match` surface.
41 - Topology data is produced today by four distinct sources (paths verified in repo):
42 1. **SNMP L2** — Go, in `src/go/pkg/topology/engine/` and `src/go/plugin/go.d/collector/snmp_topology/`. Produces actors of type `device` and `endpoint` with chassis IDs, MACs, IPs, sysName/sysObjectID. LLDP, CDP, FDB, ARP, STP.
43 2. **NetFlow / IPFIX / sFlow L3** — Rust, in `src/crates/netdata-netflow/`, plus Go plumbing in `src/go/`. Produces flow records with src/dst IP, ports, protocol, AS info, geolocation.
@@ -32,7 +45,11 @@ Facts (verified):
45 4. **Netdata streaming** — C, in `src/database/contexts/` and streaming subsystem. Produces parent ↔ child agent relationships with `NetdataNodeID`, `NetdataMachineGUID`.
46 - A 6797-line prior-planning artifact exists at `TODO-UNIFIED-TOPOLOGY-SCHEMA.md` with extensive notes on the design space, source coverage, IP tracking policy, ASN/geo enrichment options, and several decision items already taken or pending. This SOW supersedes the unstructured TODO once committed; the TODO stays in place as historical reference until the SOW closes.
47 - The codebase has helper code at `src/go/tools/topology-flow-merge/` and a per-poll merge implementation in `src/go/plugin/go.d/collector/snmp_topology/topology_output_merge.go` (sub-second scope: merging successive snapshots of the same SNMP topology). Cross-source / cross-layer merge does NOT exist today.
35 -- The `Match` struct is the right shape for cross-layer correlation in principle: each source contributes whatever identity fields it knows, and a downstream merge step can correlate any two actors that share at least one identity field. This is the "extended match fields, match on any overlapping field" principle stated in `TODO-UNIFIED-TOPOLOGY-SCHEMA.md:69-72`.
48 +- The underlying identity principle remains valid: each source contributes the
49 + identity facts it knows, and a downstream merge step can correlate actors when
50 + compatible identity facts overlap. In v1, those facts are represented through
51 + actor identity columns, evidence rows, and SOW-0023 correlation point/claim
52 + rows.
53
54 Inferences:
55
@@ -55,7 +72,10 @@ Unknowns (real, blocking design decisions):
72
73 (Final acceptance criteria are gated on user decisions in `## Implications And Decisions`. Until those decisions land, the criteria below are the high-level shape; they will be sharpened to specific verifiable outcomes once decisions are recorded.)
74
58 -- A single, documented unified topology schema is in production use by all four current sources (SNMP L2, NetFlow L3, network-viewer L7, Netdata streaming), with the existing `Match`/`Actor`/`Link` types as the canonical contract or a clearly evolved version.
75 +- A single, documented unified topology schema is in production use by all four
76 + current sources (SNMP L2, NetFlow L3, network-viewer L7, Netdata streaming),
77 + using `netdata.topology.v1` or a clearly evolved version of it as the
78 + canonical contract.
79 - A merge engine exists that takes N topology graphs (same-kind or cross-kind) and produces one merged graph following the chosen identity-match and conflict-resolution policies. Same-kind (L2 + L2) and cross-kind (L2 + L3, L2 + L3 + L7) cases each have explicit test coverage with expected merged output.
80 - Identity-matching and conflict-resolution behavior is unit-tested with targeted fixtures covering: actors that should merge, actors that should NOT merge despite an incidental field overlap, sources that legitimately disagree, and stale entries aged across sources with different freshness windows.
81 - A scale benchmark exists that exercises the merge engine at the chosen actors/links/sources targets and reports merge latency.
@@ -66,7 +86,9 @@ Unknowns (real, blocking design decisions):
86
87 Sources checked:
88
69 -- `src/go/pkg/topology/types.go` — current `Match`, `Actor`, `Link` schema.
89 +- `src/go/pkg/topology/types.go` — legacy `Match`, `Actor`, `Link` schema
90 + reviewed when this SOW was opened; superseded for new Function payload work
91 + by `netdata.topology.v1`.
92 - `src/go/pkg/topology/engine/` — current SNMP L2 merge logic (within-source).
93 - `src/go/plugin/go.d/collector/snmp_topology/topology_output_merge.go` — per-poll snapshot merge (within-source).
94 - `src/go/tools/topology-flow-merge/` — standalone helper, not currently wired into the runtime path (see TODO-UNIFIED-TOPOLOGY-SCHEMA.md branch-cleanup audit).
@@ -78,14 +100,25 @@ Sources checked:
100
101 Current state:
102
81 -- Schema shape is good. The extended-match principle is already encoded in `Match`. What's missing is the merge layer that uses these identity fields across sources.
103 +- The old extended-match principle remains useful, but the implementation path
104 + now needs to use v1 actor identity, evidence, and SOW-0023 correlation facts.
105 + What's missing is the merge layer that uses these identity facts across
106 + sources and layers.
107 - Each source today produces its own topology output as a separate function. There is no merged endpoint.
108 - Where merge does exist (same-source, per-poll snapshot fusion in the SNMP topology engine), it is implementation-internal — no shared abstraction, no reuse path for cross-source.
109 - The codebase has hooks for layered presentation in the topology UI (the `PresentationActorType`, `PresentationLinkType` and friends in `types.go`), suggesting prior thinking about layered views, but no live data wiring.
110 +- Scope boundary recorded 2026-05-06: SOW-0020 owns the immediate shared topology Function payload migration, including lossless detailed encoding and view-oriented aggregation contracts.
111 +- Scope boundary recorded 2026-05-10: SOW-0023 owns generic same-kind
112 + `netdata.topology.v1` correlation rules, pure correlation actors,
113 + point/claim tables, and link layout tokens. This SOW owns the later unified
114 + cross-layer merge/correlation layer that consumes those detailed payloads and
115 + SOW-0023 correlation facts.
116
117 Risks (cross-cutting):
118
88 -- **Schema lock-in**: any merge engine that consumes the current `Match` shape effectively freezes that shape for downstream Cloud consumers. A future schema change costs two migrations (sources + merge + Cloud).
119 +- **Schema lock-in**: any merge engine that consumes the wrong identity surface
120 + freezes that surface for downstream Cloud consumers. Future work must avoid
121 + reintroducing the legacy `Match` shape as a second public contract.
122 - **Identity correlation false positives**: matching on "any overlapping field" is dangerous if a field is non-unique (e.g. private RFC1918 IP that recurs across customer subnets, hostname `localhost`). Without canonicalization and field-quality weighting, merge can fuse unrelated actors.
123 - **Scale**: cross-layer merge expands actor count; L7 process granularity especially. If the design persists everything, storage grows; if it doesn't, historical queries degrade.
124 - **Conflict noise**: two sources legitimately disagreeing produces user-visible warnings unless conflict resolution is automatic. Policy choice affects perceived data quality.
@@ -98,7 +131,14 @@ Status: needs-user-decision
131
132 Problem / root-cause model:
133
101 -- Topology evidence is produced by four distinct sources at three distinct layers (L2/L3/L7) plus a streaming hierarchy. Each source has its own view, none is reconciled with the others, and there is no merge engine that takes evidence from multiple sources and produces a single coherent graph. The schema (`Match`/`Actor`/`Link`) is already shaped for cross-layer correlation, but the runtime layer that performs the correlation does not exist. Implementation cannot start until merge semantics, identity-matching policy, conflict resolution, and storage model are locked.
134 +- Topology evidence is produced by four distinct sources at three distinct
135 + layers (L2/L3/L7) plus a streaming hierarchy. Each source has its own view,
136 + none is reconciled with the others, and there is no merge engine that takes
137 + evidence from multiple sources and produces a single coherent graph.
138 + `netdata.topology.v1` now has actor identity and SOW-0023 same-kind
139 + correlation facts, but the cross-layer runtime layer that performs unified
140 + correlation does not exist. Implementation cannot start until merge semantics,
141 + identity-matching policy, conflict resolution, and storage model are locked.
142
143 Evidence reviewed:
144
@@ -106,7 +146,9 @@ Evidence reviewed:
146
147 Affected contracts and surfaces:
148
109 -- `src/go/pkg/topology/types.go` — schema may evolve.
149 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` — schema may evolve; SOW-0023
150 + correlation plane is the current v1 payload contract to reuse instead of
151 + inventing a parallel matching surface.
152 - All four source producers (SNMP topology Go path, netflow Rust+Go, network-viewer C, streaming C) — output schema may need conformance changes.
153 - Topology functions exposed to the UI — possibly a new "merged" function or the existing per-source functions extended.
154 - Cloud-side consumers — the merged output schema becomes a Cloud contract.
@@ -115,7 +157,8 @@ Affected contracts and surfaces:
157
158 Existing patterns to reuse:
159
118 -- The `Match` extended-fields model in `types.go:7-22` is the right contract for identity. Reuse, possibly extend.
160 +- Reuse the v1 identity and correlation contract rather than adding another
161 + public identity surface.
162 - Per-source within-source merge logic in `topology_output_merge.go` and the SNMP topology engine — the same shape (deduplicate by identity, union links) likely generalizes; a candidate to lift to a shared package.
163 - `src/go/tools/topology-flow-merge/` exists but is not wired in. Decide whether it's the seed of the cross-source merge engine or whether it gets retired.
164 - The Netdata streaming hierarchy already establishes a parent/child actor relation — useful as a precedent for the "lives-on" cross-layer relationship.
@@ -138,7 +181,8 @@ Implementation plan:
181
182 To be filled after the user decisions below are recorded. The plan will likely have these phases (illustrative, not committed):
183
141 -1. Lock the schema (extend `Match`/`Actor`/`Link` if needed; document version semantics).
184 +1. Reuse the SOW-0023 `data.correlation` contract where it fits cross-layer
185 + correlation; extend only when cross-layer semantics require additional facts.
186 2. Build a shared `topology/merge` package implementing the chosen identity-match + conflict-resolution policy. Unit-test thoroughly with same-kind and cross-kind fixtures.
187 3. Wire the merge engine behind a new topology function (`topology:unified` or similar). Keep per-source functions intact; the merged function is additive.
188 4. Add Cloud-side consumer integration plus migration notes for existing UI surfaces.
@@ -183,9 +227,12 @@ The decisions below are unresolved and block implementation. Each is presented w
227 ### Decision 2 — Identity-matching algorithm
228
229 **Options:**
186 -- **A. Strict any-overlap equality on raw `Match` fields.** Two actors merge if any one of their `Match` slices intersects.
230 +- **A. Strict any-overlap equality on raw identity facts.** Two actors merge if
231 + any one of their identity fact sets intersects.
232 - **B. Canonicalized equality.** MACs lowercased / colon-stripped, hostnames lowercased / FQDN-normalized, IPs canonicalized, before equality check.
188 -- **C. Field-quality-weighted scoring.** Each `Match` field has a discriminator weight (high for chassis ID, low for hostname `localhost`). Merge above a threshold.
233 +- **C. Field-quality-weighted scoring.** Each identity field has a
234 + discriminator weight (high for chassis ID, low for hostname `localhost`).
235 + Merge above a threshold.
236 - **D. Probabilistic / learned per deployment.** A model decides; tunable per environment.
237
238 **Implications/Risks:** A produces false positives on ambient identifiers (`localhost`, RFC1918 reuse). B handles 90% of false positives at low complexity. C handles ambiguity well but introduces a tuning knob users must understand. D is overkill for opening this work.
@@ -202,7 +249,10 @@ The decisions below are unresolved and block implementation. Each is presented w
249
250 **Implications/Risks:** A loses provenance; B requires getting the priority right and is hard to change later; C grows attribute payloads without bound but never silently drops information; D produces UX friction.
251
205 -**Recommendation: C for `Match` fields, B for `Attributes`/`Derived` fields.** Identity should be inclusive (preserve all evidence so future merges can still match); attributes should be resolvable (one displayed value at a time). Provenance in either case is preserved as a per-value `source` tag.
252 +**Recommendation: C for identity facts, B for `Attributes`/`Derived` fields.**
253 +Identity should be inclusive (preserve all evidence so future merges can still
254 +match); attributes should be resolvable (one displayed value at a time).
255 +Provenance in either case is preserved as a per-value `source` tag.
256
257 ### Decision 4 — Storage model
258
@@ -242,7 +292,10 @@ The decisions below are unresolved and block implementation. Each is presented w
292
293 Filled after Decisions 1-6 are recorded. Default skeleton (assuming the recommendations above):
294
245 -1. Lock and document the schema (extend `Match` if a decision requires it; otherwise leave as-is). Add a per-value `source` provenance tag mechanism to support Decision 3.
295 +1. Lock and document the v1-based unified schema. Extend
296 + `netdata.topology.v1` only if the recorded decisions require it. Add a
297 + per-value `source` provenance tag mechanism to support Decision 3 if still
298 + needed after reviewing existing v1 evidence/correlation provenance.
299 2. Implement a `topology/merge` package on the agent side with the chosen identity-match (Decision 2) and conflict-resolution (Decision 3) policies. Unit tests cover same-kind and cross-kind matrices.
300 3. Wire the merge engine behind a new topology function on the local agent. Per-source functions remain available unchanged.
301 4. Sanitized fixture set covering at least: a 2-switch L2 same-kind merge, an L2 + L3 cross-kind merge, an L2 + L3 + L7 cross-kind merge with a host-level L7 aggregation per Decision 6.
@@ -258,6 +311,17 @@ Filled after Decisions 1-6 are recorded. Default skeleton (assuming the recommen
311 - Existing prior planning artifact (`TODO-UNIFIED-TOPOLOGY-SCHEMA.md`, 6797 lines) noted as historical context; this SOW is the canonical replacement once user decisions are locked.
312 - Six open decisions recorded; no implementation work begins until they are resolved.
313
314 +### 2026-05-06
315 +
316 +- Scope boundary recorded from SOW-0020: detailed/aggregated topology Function payload migration is owned by SOW-0020. This SOW remains pending for unified merge semantics, identity matching, conflict resolution, storage/indexing, scale targets, and cross-layer view behavior.
317 +
318 +### 2026-05-10
319 +
320 +- Updated scope boundary after SOW-0023 added the v1 correlation plane. This
321 + SOW must not reintroduce the legacy `Match` schema as a second public
322 + contract; future work starts from `netdata.topology.v1`, actor/evidence
323 + identity facts, and SOW-0023 correlation points/claims.
324 +
325 ## Validation
326
327 Pending — gated on locked decisions and implementation.
.agents/sow/pending/SOW-0006-20260503-skill-verification-harness.md
+14 -11
@@ -68,11 +68,11 @@ SOW-0007 (`integrations-lifecycle`) as their own acceptance gates.
68
69 ### Acceptance Criteria
70
71 -- A `verify/` runner under each verified skill -- e.g.
72 - `<repo>/docs/netdata-ai/skills/query-netdata-cloud/verify/run.sh`
71 +- A verification runner under `.agents/skill-verification/<skill>/`, e.g.
72 + `<repo>/.agents/skill-verification/query-netdata-cloud/run.sh`
73 -- that:
74 - 1. Reads `<skill>/verify/questions.md` (the seed list shipped
75 - by the upstream SOW).
74 + 1. Reads `.agents/skill-verification/<skill>/questions.md` (the seed list
75 + shipped by the upstream SOW).
76 2. For each question, spawns a Sonnet-class assistant with a
77 minimal system prompt that points at the skill (SKILL.md +
78 `how-tos/INDEX.md` + canonical reference docs).
@@ -80,9 +80,9 @@ SOW-0007 (`integrations-lifecycle`) as their own acceptance gates.
80 it made, and the final answer.
81 4. Records results under
82 `<repo>/.local/audits/<skill>/verify/<timestamp>/`.
83 - 5. Grades each answer against `verify/grader.md`.
83 + 5. Grades each answer against `.agents/skill-verification/<skill>/grader.md`.
84 6. Reports pass / fail / unanswered counts.
85 -- `verify/grader.md` per skill: rubric covering (a) correctness,
85 +- `.agents/skill-verification/<skill>/grader.md` per skill: rubric covering (a) correctness,
86 (b) evidence shown (file:line refs, response keys), (c) no
87 exposed tokens / bearers / claim ids in the transcript or
88 output, (d) how-to authored when missing.
@@ -129,8 +129,8 @@ Status: blocked-on-prereq
129
130 Depends on SOW-0010 closing (it provides the SKILL.md, the
131 per-domain guides, the `how-tos/INDEX.md`, the
132 -`verify/questions.md` seed list, and the token-safe wrappers
133 -that the harness must invoke).
132 +`.agents/skill-verification/<skill>/questions.md` seed list, and the
133 +token-safe wrappers that the harness must invoke).
134
135 Sensitive data handling plan:
136
@@ -152,8 +152,11 @@ Holding-pattern decisions to record now:
152 is a runner flag with a default; not a per-question
153 hardcode.
154 - The how-to generation prompt is a separate template under
155 - `verify/howto-template.md`. Stage 2 implementation defines
156 - it.
155 + `.agents/skill-verification/<skill>/howto-template.md`. Stage 2
156 + implementation defines it.
157 +- Verification seed questions are harness inputs, not public/operator skill
158 + content. They live under `.agents/skill-verification/<skill>/questions.md`,
159 + not under `docs/netdata-ai/skills/`.
160
161 ## Implications And Decisions
162
@@ -165,7 +168,7 @@ SOW-0010 closes and stage 2 begins.
168 1. **Wait for SOW-0010 to close.**
169 2. Stage 2a: read SOW-0010 final deliverables (SKILL.md
170 structure, how-tos shape, wrappers).
168 -3. Stage 2b: implement `verify/run.sh` for the cloud skill;
171 +3. Stage 2b: implement `.agents/skill-verification/query-netdata-cloud/run.sh`;
172 prove end-to-end on the seed questions.
173 4. Stage 2c: parameterize `run.sh` so it can target any of the
174 SOW-0010+ skills.
.agents/sow/pending/SOW-0024-20260510-vsphere-topology-v1-migration.md new
+274
@@ -0,0 +1,274 @@
1 +# SOW-0024 - vSphere topology v1 migration
2 +
3 +## Status
4 +
5 +Status: open
6 +
7 +`completed` is the successful terminal status. `done` is a directory name, not a status value. Do not use `Status: done` or `Status: complete`.
8 +
9 +Sub-state: pending until SOW-0021 graph presentation, SOW-0023 cross-payload
10 +matching/link layout, and SOW-0022 table/modal composition are finished.
11 +
12 +## Requirements
13 +
14 +### Purpose
15 +
16 +Migrate the vSphere topology producer from the superseded topology schema to `netdata.topology.v1`, preserving vSphere inventory semantics, presentation, and drilldown data while keeping the Cloud frontend producer-agnostic.
17 +
18 +### User Request
19 +
20 +The user asked to add a later SOW to transform vSphere to the new schema because the vSphere topology is still legacy.
21 +
22 +### Assistant Understanding
23 +
24 +Facts:
25 +
26 +- The vSphere topology producer lives in the separate vSphere feature worktree
27 + `<vsphere-worktree>`.
28 +- The vSphere producer currently uses the legacy Go topology package and `WithPresentation()`.
29 +- Agent SOW-0021 explicitly did not migrate vSphere, but added color/icon tokens so the later vSphere migration should not need another graph-presentation schema redesign.
30 +- Agent SOW-0023 added generic correlation rules and link layout tokens that
31 + vSphere must respect when migrated.
32 +- The frontend must remain topology-schema agnostic; it must not branch on vSphere as a special UI domain.
33 +
34 +Inferences:
35 +
36 +- This migration should happen only after the schema, Cloud aggregator, and UI can consume the full v1 graph-presentation/table contract.
37 +- The migration should produce v1 topology facts and presentation profiles directly, not old-schema compatibility payloads.
38 +
39 +Unknowns:
40 +
41 +- Whether the vSphere worktree will still be clean and owned by the same worker when this SOW starts.
42 +- Whether SOW-0022 table/modal composition will require additional vSphere-specific actor detail table fields by the time this migration starts.
43 +
44 +### Acceptance Criteria
45 +
46 +- vSphere `topology:vsphere` emits `schema_version: "netdata.topology.v1"`.
47 +- The producer no longer uses the superseded topology package for the production payload.
48 +- Actor types cover datacenter, cluster, host, VM, datastore, network, datastore cluster, and resource pool.
49 +- Link types cover contains, connects, runs, and any other vSphere relationship emitted by the producer.
50 +- Actor identities use stable vSphere managed object identifiers where available, with display labels kept separate through `presentation.label_policy`.
51 +- vSphere graph presentation is expressed through `types.actor_types.<id>.presentation`, `types.link_types.<id>.presentation`, optional `types.port_types`, and `data.presentation`.
52 +- vSphere link types use SOW-0023 `presentation.layout.strength` and
53 + `presentation.layout.distance` tokens where ownership, dependency, inferred,
54 + or weak relationships need different graph forces.
55 +- vSphere stable object identifiers are evaluated for SOW-0023
56 + `data.correlation.claims` only when they can help cross-payload matching;
57 + producer output must not expose aggregator internal states.
58 +- vSphere actor detail/inventory tables use the SOW-0022 table/modal composition contract if that contract is complete before this SOW starts.
59 +- JSON Schema validation and Go semantic validation pass.
60 +- Payload size is measured on realistic or synthetic vSphere inventory shapes.
61 +- The vSphere worktree owner is informed before edits start.
62 +
63 +## Analysis
64 +
65 +Sources checked:
66 +
67 +- `.agents/skills/project-create-topology/SKILL.md`
68 +- `.agents/sow/current/SOW-0021-20260509-topology-presentation-contract.md`
69 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
70 +- `.agents/sow/specs/topology-function-schema.md`
71 +- `<vsphere-worktree>/src/go/plugin/go.d/collector/vsphere/func_topology.go`
72 +- `<vsphere-worktree>/src/go/plugin/go.d/collector/vsphere/func_topology_presentation.go`
73 +
74 +Current state:
75 +
76 +- `func_topology.go` in the vSphere worktree declares `vsphereTopologySchemaVersion = "2.0"` and returns `topology.Data`.
77 +- `vsphereTopologyMethodConfig()` attaches legacy presentation with `WithPresentation(vsphereTopologyPresentation())`.
78 +- `func_topology_presentation.go` defines legacy actor/link presentation for vSphere actor and link types.
79 +- The vSphere worktree was checked when this SOW was created and had no modified files under the vSphere collector.
80 +
81 +Risks:
82 +
83 +- Starting this before SOW-0021/SOW-0023/SOW-0022 integration settles may force duplicate migration work.
84 +- Editing the vSphere worktree without coordination may collide with another worker.
85 +- Mixing old-schema reconstruction metadata into the v1 payload would violate the topology schema contract.
86 +- Treating vSphere display names as identities would make aggregation and cross-payload matching fragile.
87 +
88 +## Pre-Implementation Gate
89 +
90 +Status: blocked
91 +
92 +Problem / root-cause model:
93 +
94 +- vSphere remains on the superseded topology schema while the rest of the topology work is moving to `netdata.topology.v1`.
95 +- SOW-0021 intentionally left vSphere unmigrated and only preserved the token vocabulary needed for a later migration.
96 +- This SOW is blocked until the core v1 contract, frontend rendering, Cloud aggregation, cross-payload matching, and table/modal composition are complete enough to avoid rework.
97 +
98 +Evidence reviewed:
99 +
100 +- SOW-0021 vSphere migration posture records that vSphere is not migrated by SOW-0021 and should be coordinated separately.
101 +- SOW-0023 records the final correlation and link layout contract that vSphere
102 + must use when migrated.
103 +- The vSphere worktree producer still uses the legacy topology package and `WithPresentation()`.
104 +- The project topology skill states the vSphere worktree must not be edited
105 + before telling the user because another agent may be working there.
106 +
107 +Affected contracts and surfaces:
108 +
109 +- vSphere Function output schema.
110 +- vSphere topology producer implementation in the separate worktree.
111 +- Topology JSON Schema and semantic validator compatibility.
112 +- Cloud frontend graph presentation and actor modal/table rendering.
113 +- Cloud topology service aggregation and cross-payload matching.
114 +- Tests/fixtures for vSphere inventory topology.
115 +
116 +Existing patterns to reuse:
117 +
118 +- `src/go/pkg/topology/v1` compact-table helpers from this worktree.
119 +- Network-connections, streaming, and SNMP v1 producers after SOW-0021.
120 +- SOW-0022 table/modal composition contract once complete.
121 +- SOW-0023 correlation rules, pure correlation actors, point/claim tables, and
122 + link layout tokens once complete.
123 +
124 +Risk and blast radius:
125 +
126 +- Medium: vSphere topology is a producer-specific migration, but it touches public Function payload shape.
127 +- Compatibility: Cloud frontend must continue old-schema support until the Agent rollout is complete.
128 +- Performance: large vSphere inventories may produce many actors/links and require compact tables.
129 +- Security: vSphere managed object ids, inventory paths, hostnames, datastores, networks, and labels can identify private infrastructure and must not be copied raw into durable artifacts.
130 +
131 +Sensitive data handling plan:
132 +
133 +- Do not commit raw vSphere payload captures.
134 +- Use synthetic or sanitized fixtures.
135 +- Redact private inventory names, managed object ids, hostnames, datastore names, network names, private endpoints, credentials, tokens, and customer data from SOWs, docs, skills, tests, and review artifacts.
136 +
137 +Implementation plan:
138 +
139 +1. Reconfirm with the user before editing `<vsphere-worktree>`.
140 +2. Read completed SOW-0021, SOW-0023, and SOW-0022 outcomes.
141 +3. Inventory current vSphere actors, links, attributes, labels, and presentation.
142 +4. Design the v1 actor/link/evidence/table types for vSphere.
143 +5. Implement v1 payload generation using compact tables.
144 +6. Preserve graph presentation through v1 type-level and graph-level presentation.
145 +7. Add fixtures/tests and validate against the schema and semantic validator,
146 + including link layout tokens and any vSphere correlation claims.
147 +8. Coordinate Cloud frontend and Cloud aggregator validation with generic topology fixtures.
148 +
149 +Validation plan:
150 +
151 +- Go unit tests for vSphere v1 payload shape.
152 +- JSON Schema validation against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.
153 +- Semantic validation using `src/go/pkg/topology/v1`.
154 +- Payload size measurement on synthetic or sanitized vSphere inventory shapes.
155 +- Frontend manual check through generic topology rendering after UI support lands.
156 +- Cloud aggregator fixture once the service supports the final v1 presentation/table/matching contracts.
157 +
158 +Artifact impact plan:
159 +
160 +- AGENTS.md: likely unaffected.
161 +- Runtime project skills: update `.agents/skills/project-create-topology/SKILL.md`
162 + only if migration reveals new reusable vSphere topology guidance.
163 +- Specs: update `.agents/sow/specs/topology-function-schema.md` if this migration changes durable topology semantics.
164 +- End-user/operator docs: update only if public vSphere topology behavior or Function docs exist and change.
165 +- End-user/operator skills: update only if public topology skill guidance changes.
166 +- SOW lifecycle: keep this SOW pending until SOW-0021, SOW-0023, and SOW-0022 are done.
167 +
168 +Open-source reference evidence:
169 +
170 +- None checked. This is an internal producer migration from the legacy Netdata topology contract to the new Netdata topology contract.
171 +
172 +Open decisions:
173 +
174 +- None for creating this pending SOW.
175 +- Before implementation starts, confirm the vSphere worktree ownership and whether SOW-0022 table/modal composition is mandatory for the first vSphere v1 payload.
176 +
177 +## Implications And Decisions
178 +
179 +- User decision: vSphere migration is tracked as a later SOW after the current topology schema/UI/Cloud work.
180 +- This SOW must not be started before the other topology SOWs are finished unless the user explicitly changes the order.
181 +
182 +## Plan
183 +
184 +1. Wait for SOW-0021, SOW-0023, and SOW-0022 completion.
185 +2. Confirm vSphere worktree ownership with the user.
186 +3. Move this SOW to current and fill any newly discovered implementation specifics.
187 +4. Implement and validate the vSphere migration to `netdata.topology.v1`.
188 +
189 +## Execution Log
190 +
191 +### 2026-05-10
192 +
193 +- Created pending SOW after the user noted that vSphere remains legacy and should be migrated after the other topology SOWs.
194 +
195 +- Updated prerequisites after SOW-0023 added declarative correlation rules and
196 + link layout tokens. vSphere migration must use the final v1 contract rather
197 + than the older presentation-only contract.
198 +
199 +## Validation
200 +
201 +Acceptance criteria evidence:
202 +
203 +- Pending; this SOW has not started implementation.
204 +
205 +Tests or equivalent validation:
206 +
207 +- Pending.
208 +
209 +Real-use evidence:
210 +
211 +- Pending.
212 +
213 +Reviewer findings:
214 +
215 +- Pending.
216 +
217 +Same-failure scan:
218 +
219 +- Pending.
220 +
221 +Sensitive data gate:
222 +
223 +- This SOW contains only sanitized file references and no raw vSphere inventory payloads or credentials.
224 +
225 +Artifact maintenance gate:
226 +
227 +- AGENTS.md: not updated; workflow rules did not change.
228 +- Runtime project skills: not updated; this SOW only records future work.
229 +- Specs: not updated; behavior has not changed yet.
230 +- End-user/operator docs: not updated; behavior has not changed yet.
231 +- End-user/operator skills: not updated; behavior has not changed yet.
232 +- SOW lifecycle: open in `.agents/sow/pending/` and blocked on earlier topology SOWs.
233 +
234 +Specs update:
235 +
236 +- No spec update yet because no behavior changed.
237 +
238 +Project skills update:
239 +
240 +- No project skill update yet because no workflow changed.
241 +
242 +End-user/operator docs update:
243 +
244 +- No docs update yet because no behavior changed.
245 +
246 +End-user/operator skills update:
247 +
248 +- No skill update yet because no behavior changed.
249 +
250 +Lessons:
251 +
252 +- vSphere must be tracked explicitly because SOW-0021 prepared the schema for it but did not migrate its producer.
253 +
254 +Follow-up mapping:
255 +
256 +- Implemented by this pending SOW after SOW-0021, SOW-0023, and SOW-0022 finish.
257 +
258 +## Outcome
259 +
260 +Pending.
261 +
262 +## Lessons Extracted
263 +
264 +Pending until implementation.
265 +
266 +## Followup
267 +
268 +- Start this after SOW-0021, SOW-0023, and SOW-0022 are complete.
269 +
270 +## Regression Log
271 +
272 +None yet.
273 +
274 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.
.agents/sow/pending/SOW-0029-20260511-network-connections-detailed-loose-sides.md new
+268
@@ -0,0 +1,268 @@
1 +# SOW-0029 - Network Connections Detailed Loose Sides
2 +
3 +## Status
4 +
5 +Status: open
6 +
7 +Sub-state: pending follow-up split from SOW-0028.
8 +
9 +## Requirements
10 +
11 +### Purpose
12 +
13 +Make detailed `topology:network-connections` preserve exact socket endpoints
14 +without forcing every remote `IP:PORT` tuple to become a graph actor, while
15 +keeping direct-Agent views readable and Cloud aggregation correlation-safe.
16 +
17 +### User Request
18 +
19 +The user asked to complete SOW-0028. The remaining true loose-side detailed
20 +graph work was split here because it requires a separate Agent/UI/aggregator
21 +execution pass.
22 +
23 +### Assistant Understanding
24 +
25 +Facts:
26 +
27 +- SOW-0028 implemented the schema compatibility layer, modal identification,
28 + mode metadata, correlation rule classes, and relationship-summary rows.
29 +- The current Agent producer still emits endpoint actors in detailed
30 + network-connections views.
31 +- The SOW-0028 spec describes a stronger target: detailed rows may have one
32 + known actor side and one loose endpoint side, with exact tuple facts preserved
33 + for correlation and UI materialization.
34 +
35 +Inferences:
36 +
37 +- Implementing true loose sides is not a small closure task. It affects compact
38 + table shape, schema validation, UI materialization, and aggregator
39 + correlation execution together.
40 +- Keeping it separate reduces risk of breaking current working direct-Agent
41 + topology views.
42 +
43 +Unknowns:
44 +
45 +- The exact compact schema shape for one-sided graph/evidence rows must be
46 + finalized against the existing `netdata.topology.v1` table model before code
47 + changes.
48 +
49 +### Acceptance Criteria
50 +
51 +- The schema/docs/specs define the exact compact representation for one-sided
52 + detailed network-connections rows and materialization policy.
53 +- The Agent detailed network-connections producer emits known actors plus
54 + one-sided loose endpoint facts where appropriate, without actor-per-ephemeral
55 + `IP:PORT` graph explosion.
56 +- The UI materializes loose sides only for direct-Agent detailed views and only
57 + according to producer-declared policy.
58 +- The Cloud topology service resolves exact loose-side matches before returning
59 + aggregated output and keeps unresolved/partial cases visible without exposing
60 + aggregator-internal state.
61 +- Installed Function checks and focused tests prove detailed and aggregated
62 + network-connections still render and correlate correctly.
63 +
64 +## Analysis
65 +
66 +Sources checked:
67 +
68 +- `.agents/sow/specs/topology-modes-correlation-aggregation.md`
69 +- `.agents/sow/done/SOW-0028-20260511-topology-mode-correlation-aggregation.md`
70 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
71 +- `src/collectors/network-viewer.plugin/network-viewer.c`
72 +- Cloud frontend `TODO-topology-mode-correlation-aggregation.md`
73 +- Cloud topology service SOW-0011
74 +
75 +Current state:
76 +
77 +- Detailed network-connections preserves exact socket evidence and correlation
78 + metadata.
79 +- Detailed network-connections still uses endpoint actors in the renderable
80 + graph.
81 +- UI loose-side materialization is documented in the frontend TODO but not yet
82 + implemented because producers do not emit one-sided detailed rows.
83 +
84 +Risks:
85 +
86 +- Actor-per-`IP:PORT` remains a graph cardinality risk if detailed views grow.
87 +- A careless loose-side implementation can become a hidden frontend aggregator
88 + or hide unresolved dependencies.
89 +- Changing detailed graph rows can break existing modal recipes if source
90 + columns and owner filters are not updated together.
91 +
92 +## Pre-Implementation Gate
93 +
94 +Status: blocked
95 +
96 +Problem / root-cause model:
97 +
98 +- The current v1 payload supports exact socket evidence, but graph links still
99 + require materialized endpoint actors for remote peers. The desired detailed
100 + model needs one-sided rows and a declared materialization policy, so exact
101 + evidence can be preserved without graph actor explosion.
102 +
103 +Evidence reviewed:
104 +
105 +- SOW-0028 installed checks show detailed network-connections returns socket
106 + evidence and correlation metadata, but still returns endpoint actors and
107 + normal graph links.
108 +- `src/collectors/network-viewer.plugin/network-viewer.c` currently emits
109 + non-null `src_actor` and `dst_actor` columns for graph links and socket
110 + evidence table declarations.
111 +
112 +Affected contracts and surfaces:
113 +
114 +- Agent topology JSON schema, developer guide, and network-viewer producer.
115 +- Cloud frontend v1 normalizer, graph model, actor modal source selection, and
116 + diagnostics.
117 +- Cloud topology service correlation and aggregation core.
118 +- Topology specs and project topology skill.
119 +
120 +Existing patterns to reuse:
121 +
122 +- Compact tables with nullable reference columns.
123 +- Existing modal `selected_side_endpoint` projection.
124 +- Existing correlation `resolve_loose_side` rule class.
125 +- Existing UI diagnostics for unsupported metadata.
126 +
127 +Risk and blast radius:
128 +
129 +- High semantic risk for network dependency correctness.
130 +- Medium UI risk because one-sided rows need deterministic render-only actors.
131 +- Medium aggregator risk because exact match, partial match, and no-match cases
132 + must remain visible and truthful.
133 +
134 +Sensitive data handling plan:
135 +
136 +- Use synthetic socket fixtures and summaries only.
137 +- Keep any live Function captures under `.local/`; do not commit raw process
138 + names, command lines, bearer tokens, private endpoints, raw node IDs, raw
139 + machine GUIDs, or customer-identifying public endpoints.
140 +
141 +Implementation plan:
142 +
143 +1. Finalize the schema shape and materialization policy with synthetic
144 + examples.
145 +2. Update Agent producer and schema validation.
146 +3. Update UI normalizer/materialization and modal behavior.
147 +4. Update Cloud topology service correlation handling for one-sided rows.
148 +5. Validate with synthetic tests and installed Function checks.
149 +
150 +Validation plan:
151 +
152 +- Agent schema validation and network-viewer build.
153 +- Focused UI tests for direct-Agent detailed loose-side materialization.
154 +- Cloud topology service tests for exact, partial, ambiguous, and no-match
155 + loose-side cases.
156 +- Installed `topology:network-connections` detailed/aggregated checks.
157 +
158 +Artifact impact plan:
159 +
160 +- AGENTS.md: not expected unless workflow changes.
161 +- Runtime project skills: update `project-create-topology` if the final
162 + materialization policy adds a recurring rule.
163 +- Specs: update topology schema specs.
164 +- End-user/operator docs: not expected.
165 +- End-user/operator skills: not expected.
166 +- SOW lifecycle: start only after SOW-0028 is completed and the user approves
167 + this follow-up priority.
168 +
169 +Open-source reference evidence:
170 +
171 +- Not checked. This is a Netdata topology schema contract, not an external
172 + protocol behavior.
173 +
174 +Open decisions:
175 +
176 +- Blocked until this SOW is selected as the next active topology task.
177 +
178 +## Implications And Decisions
179 +
180 +No user decision has been requested yet.
181 +
182 +## Plan
183 +
184 +1. Re-open the detailed network-connections schema section and decide the exact
185 + compact table representation.
186 +2. Patch Agent, UI, and Cloud topology service in one coordinated pass.
187 +3. Validate with installed Function output and focused tests.
188 +
189 +## Execution Log
190 +
191 +### 2026-05-11
192 +
193 +- Created as a pending follow-up split from SOW-0028 closure.
194 +
195 +## Validation
196 +
197 +Acceptance criteria evidence:
198 +
199 +- Pending.
200 +
201 +Tests or equivalent validation:
202 +
203 +- Pending.
204 +
205 +Real-use evidence:
206 +
207 +- Pending.
208 +
209 +Reviewer findings:
210 +
211 +- Pending.
212 +
213 +Same-failure scan:
214 +
215 +- Pending.
216 +
217 +Sensitive data gate:
218 +
219 +- Pending.
220 +
221 +Artifact maintenance gate:
222 +
223 +- AGENTS.md: not changed.
224 +- Runtime project skills: pending if materialization policy changes.
225 +- Specs: pending.
226 +- End-user/operator docs: not expected.
227 +- End-user/operator skills: not expected.
228 +- SOW lifecycle: pending/open.
229 +
230 +Specs update:
231 +
232 +- Pending.
233 +
234 +Project skills update:
235 +
236 +- Pending.
237 +
238 +End-user/operator docs update:
239 +
240 +- Not expected.
241 +
242 +End-user/operator skills update:
243 +
244 +- Not expected.
245 +
246 +Lessons:
247 +
248 +- Pending.
249 +
250 +Follow-up mapping:
251 +
252 +- Pending.
253 +
254 +## Outcome
255 +
256 +Pending.
257 +
258 +## Lessons Extracted
259 +
260 +Pending.
261 +
262 +## Followup
263 +
264 +None yet.
265 +
266 +## Regression Log
267 +
268 +None yet.
.agents/sow/specs/topology-function-schema.md new
+614
@@ -0,0 +1,614 @@
1 +# Spec - Topology Function Schema
2 +
3 +## Status
4 +
5 +Active for new topology Function work. Existing deployed topology producers are
6 +to be migrated to this contract.
7 +
8 +## Contract
9 +
10 +Topology Functions return normal Function envelopes with `type: "topology"` and
11 +`data.schema_version: "netdata.topology.v1"`.
12 +
13 +The production schema is defined by:
14 +
15 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
16 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
17 +
18 +The schema is generic across topology domains. It applies to network
19 +connections, streaming, SNMP/L2, vSphere, and future topology producers.
20 +
21 +## Planes
22 +
23 +Topology payloads separate these planes:
24 +
25 +- actors: observed entities such as nodes, processes, containers, ports,
26 + devices, streaming agents, or virtualization objects;
27 +- graph links: renderable relationship groups between actors;
28 +- evidence: canonical relationship facts behind graph links;
29 +- detail tables: actor-owned or relationship-owned drilldown data;
30 +- actor labels: actor-owned key/value rows for modal labels and filters, not a
31 + replacement for canonical identity or grouping columns;
32 +- presentation: backend-selected UI-token composition for labels, colors,
33 + icons, legends, highlighting, link styles, scale keys, and graph port
34 + bullets;
35 +- modal composition: actor/link modal recipes over existing actors, links,
36 + evidence, labels, and detail tables;
37 +- correlation: producer-visible rules, loose-side resolution, replacement,
38 + enrichment, visible correlation points, and claims used by an aggregator to
39 + correlate independently produced topology maps without exposing aggregator
40 + internal states;
41 +- overlay refs: compact references for refreshable metrics or Function-backed
42 + snapshots.
43 +
44 +Graph links are projections. Evidence rows are the facts used by Cloud
45 +aggregation, matching, and detailed drilldowns.
46 +
47 +## Mode Requests
48 +
49 +Mode requests use `__topology_mode` when a producer has a real detailed vs
50 +aggregated output difference. Valid values are `detailed` and `aggregated`.
51 +Mode-invariant topologies should not expose a selector only to return identical
52 +payloads. Mode-capable producers declare `data.view.supported_modes`; absent
53 +or single-value `supported_modes` means the topology is mode-invariant for UI
54 +control purposes. The Cloud topology aggregator consumes detailed payloads for
55 +mode-capable producers before returning an aggregated view, so producers must
56 +preserve correlation-grade evidence in detailed mode.
57 +
58 +## Compact Tables
59 +
60 +Large sections use compact columnar tables with:
61 +
62 +- `rows`
63 +- `columns`
64 +- `values`
65 +
66 +Supported column codecs are:
67 +
68 +- `const`
69 +- `values`
70 +- `dict`
71 +
72 +Every decoded column must produce exactly `rows` values. Producers should use
73 +shared helpers for table building, dictionary encoding, validation, and
74 +deterministic sorting rather than hand-rolling encoders in each plugin.
75 +
76 +Column type `json` is allowed only for actor-owned or custom detail cells that
77 +must preserve nested producer-owned values. It is not the default for
78 +relationship evidence, because high-cardinality evidence needs typed scalar,
79 +reference, array, or dictionary-backed columns for compactness and aggregation
80 +semantics.
81 +
82 +## Identity And Aggregation
83 +
84 +Actor types declare:
85 +
86 +- source-local `identity`;
87 +- cross-payload `merge_identity`;
88 +- optional `parent_identity`;
89 +- supported `aggregation_scopes`.
90 +- optional `search` policy over actor table columns and actor label keys.
91 +- optional `presentation` with UI-owned tokens, safe label policy, size policy,
92 + actor repulsion policy, and graph port-bullet policy.
93 +
94 +Link types declare:
95 +
96 +- `orientation`: `directed`, `undirected`, `hierarchical`, or
97 + `observed_bidirectional`;
98 +- `direction_role`: `none`, `flow`, `dependency`, `ownership`, or
99 + `observation`;
100 +- optional `semantic_role`: `normal`, `discovery`, `ownership`, `traffic`,
101 + `correlation`, or `control`;
102 +- aggregation policy for direction, evidence, metrics, tables, and overlays.
103 +- optional `presentation` with UI-owned color, line style, width, curve, arrow,
104 + tokenized layout strength/distance, and one variable visual channel.
105 +
106 +Port types are optional and declare graph port-bullet presentation only.
107 +Actor/link modal composition is declared separately under actor/link type
108 +presentation and table type presentation. Modal recipes are selectors and
109 +projections over existing facts; they are not duplicate row stores.
110 +
111 +Cloud aggregation may only canonicalize endpoint order when link type policy
112 +explicitly allows unordered aggregation. Direction-significant links must
113 +preserve direction.
114 +
115 +## Evidence And Tables
116 +
117 +Relationship evidence is not actor custom data.
118 +
119 +Evidence tables carry matchable relationship facts such as socket tuples,
120 +streaming hops, LLDP/CDP observations, or vSphere inventory edges. Actor-detail
121 +tables carry actor-owned data such as streaming path inventory or local status
122 +tables. Table metadata must state role and aggregation policy.
123 +
124 +Production payloads must not duplicate the same evidence under every actor only
125 +to populate drilldown modals. The UI and aggregator derive drilldown views from
126 +shared evidence and typed table references.
127 +
128 +Actor labels use a compact actor-owned table, normally
129 +`tables.actor.actor_labels`, with rows shaped as:
130 +
131 +```text
132 +actor, key, value, source?, kind?, value_index?
133 +```
134 +
135 +`key`, `value`, `source`, and `kind` are logical strings and may be encoded as
136 +`string` or `string_ref`. Producers should prefer `string_ref` when they already
137 +maintain a local dictionary; aggregators and UI adapters must normalize both
138 +encodings as equivalent label strings.
139 +
140 +Host/node actors should expose the complete host label set when available.
141 +Non-node actors should expose useful producer-known labels and metadata, such
142 +as process command line, user, group, namespace, interface role, or
143 +virtualization object properties. Repeated values are repeated rows ordered by
144 +`value_index`, not JSON arrays. Facts needed for identity, correlation,
145 +grouping, sorting, filtering, or aggregation must remain typed canonical
146 +columns too.
147 +
148 +`actor_labels` inherits the topology Function sensitive-data classification.
149 +Consumers must preserve the same access-control assumptions as the source
150 +Function because labels may include command lines, users, host labels, system
151 +contact/location fields, or other operator-controlled metadata.
152 +
153 +Actor modal identification is producer-selected through
154 +`presentation.modal.labels.identification.fields[]`. Each field references an
155 +existing `actor_labels.key`, provides the label to render near the actor title,
156 +and may limit the number of displayed values with `max_values`. The full Labels
157 +tab remains complete; identification is a curated header projection, not a
158 +second table.
159 +
160 +## Presentation
161 +
162 +Presentation belongs in the production `netdata.topology.v1` payload, not in
163 +Function `info`, except for old-schema compatibility during rollout.
164 +
165 +Function `info` responses are metadata responses. They may omit `data` and
166 +advertise only parameters, help text, and timing. The topology JSON schema
167 +applies to full topology responses, not metadata-only `info` responses.
168 +
169 +Type definitions carry type-local presentation:
170 +
171 +- `types.actor_types.<id>.presentation`
172 +- `types.link_types.<id>.presentation`
173 +- `types.port_types.<id>.presentation`
174 +
175 +Graph-level `data.presentation` carries legend order, actor-click highlight
176 +behavior, port tooltip field labels, and scale-key definitions.
177 +
178 +Actor and link type presentation may also carry modal composition:
179 +
180 +- `presentation.modal.labels` describes the actor label table, usually
181 + `actor_labels`;
182 +- `presentation.modal.mini_topology` describes a depth-1 modal graph preview
183 + built from incident links and opposite actors;
184 +- `presentation.modal.sections[]` describes table sections over existing
185 + actors, links, evidence, actor detail tables, or relationship tables.
186 +
187 +Table types may carry `presentation` defaults for table label, order, default
188 +visibility, and column display metadata. Table type presentation does not
189 +replace actor/link modal sections; it gives reusable defaults for existing
190 +table rows.
191 +
192 +Modal sections require non-empty `id`, non-empty `label`, a source, and at
193 +least one column. `empty_label` is section-only empty-state text. Modal columns
194 +require non-empty `id`, non-empty `label`, and a projection. `badge_map`,
195 +`align`, and `sortable` are presentation hints over projected values and must
196 +not introduce new facts.
197 +
198 +`label_lookup` uses `label_key` and optionally an `actor_column`; omitted
199 +`actor_column` means the selected modal actor. `json_path` always requires both
200 +the JSON `column` and the scalar `path` to extract.
201 +
202 +Presentation uses closed UI-owned tokens. Producers must not emit raw SVG, raw
203 +CSS, coordinates, component names, raw force-layout physics, or viewport state.
204 +
205 +Closed token values are part of the schema contract:
206 +
207 +- color slots: `primary`, `secondary`, `accent`, `self`, `neutral`, `muted`,
208 + `dim`, `derived`, `info`, `structural`, `warning`, `success`, `danger`,
209 + `blue`, `green`, `orange`, `purple`, `cyan`, `yellow`, `teal`, `gray`;
210 +- opacity tokens: `normal`, `muted`, `faded`;
211 +- width tokens: `thin`, `normal`, `thick`, `emphasis`;
212 +- link layout strength tokens: `weakest`, `weaker`, `normal`, `stronger`,
213 + `strongest`;
214 +- link layout distance tokens: `closest`, `closer`, `normal`, `farther`,
215 + `farthest`;
216 +- actor layout repulsion tokens: `weakest`, `weaker`, `normal`, `stronger`,
217 + `strongest`;
218 +- actor size scale tokens: `compact`, `normal`, `emphasized`;
219 +- link semantic roles: `normal`, `discovery`, `ownership`, `traffic`,
220 + `correlation`, `control`;
221 +- icons: `router`, `switch`, `firewall`, `access_point`, `server`, `storage`,
222 + `load_balancer`, `printer`, `phone`, `ups`, `camera`, `process`, `agent`,
223 + `netdata-agent`, `parent`, `remote-endpoint`, `local-endpoint`, `segment`,
224 + `self`, `ip`, `cloud`, `container`, `vm`, `database`, `service`,
225 + `datacenter`, `cluster`, `host`, `network`, `datastore`,
226 + `datastore_cluster`, `resource_pool`, `device`, `endpoint`, `correlation`,
227 + `interface`, `group`, `unknown`.
228 +
229 +The UI owns token rendering and must treat producer labels as plain text. If a
230 +new producer emits a schema-valid token that an older UI does not know, the UI
231 +must use a safe fallback and record diagnostics.
232 +
233 +Actor `label_policy` is the only approved way to choose display labels from
234 +actor rows. Canonical identity is not display text. The UI must reject array
235 +values by default so aggregated identities do not become long actor names.
236 +Label-policy columns must be safe scalar actor-table columns. Producers must
237 +not reference secrets, tokens, customer-identifying fields, or unbounded arrays
238 +from `label_policy.columns`.
239 +
240 +Actor `search` is the only approved way to choose graph search content for v1.
241 +`search.columns[]` references scalar actor-table columns. `search.label_keys[]`
242 +references values in the actor label table, normally `actor_labels.key`. Set
243 +`search.enabled: false` for helper actors that should not be searchable. The UI
244 +must not traverse producer-specific `details`, `match`, `attributes`, or label
245 +paths when rendering v1.
246 +
247 +Link types may define one variable visual channel using `variable.channel`,
248 +`variable.scale_key`, and `variable.value_column`. Producers emit raw domain
249 +values, such as socket counts or traffic bytes. Cloud or the UI scales values
250 +per `scale_key` across the visible graph. `variable.min` and `variable.max`
251 +are visual tokens for the chosen channel: width variables use width tokens,
252 +opacity variables use opacity tokens.
253 +
254 +Link types may also define tokenized force-layout hints using
255 +`presentation.layout.strength` and `presentation.layout.distance`. These are
256 +relative UI-owned tokens, not numeric physics. Current producer tuning keeps
257 +`strength` at `normal` and varies only `distance` where the topology needs
258 +semantic separation. Do not reintroduce non-normal `strength` tokens for graph
259 +polish unless a later product decision explicitly re-enables force-strength
260 +tuning.
261 +
262 +Actor types may define tokenized force-layout hints using
263 +`presentation.layout.repulsion`. Repulsion is separate from link strength:
264 +repulsion pushes actors apart, while link strength pulls endpoints together.
265 +Producers must not emit raw charge values. Actor size may define a type-level
266 +`size.scale` token for deliberate fixed emphasis, such as current/self actors;
267 +the UI must not infer this from actor type names or labels.
268 +
269 +`link_types.<id>.semantic_role` is behavior metadata, not visual styling. It
270 +drives UI behavior such as discovery-link filtering, ownership/coherence
271 +treatment, traffic emphasis, and correlation treatment without hardcoded
272 +protocol or type-name checks. Link appearance still comes from
273 +`presentation`.
274 +
275 +When link `presentation.arrow` is `auto` or omitted, the UI derives arrows from
276 +`orientation` and `direction_role`:
277 +
278 +- `undirected` -> no arrow;
279 +- `observed_bidirectional` -> no arrow;
280 +- `direction_role: none` -> no arrow;
281 +- `direction_role: observation` -> no arrow;
282 +- `directed` with `flow` or `dependency` -> forward from `src_actor` to
283 + `dst_actor`;
284 +- `hierarchical` with `ownership` -> forward from `src_actor` to `dst_actor`;
285 +- all other combinations -> no arrow and a diagnostic if the combination is
286 + schema-valid but semantically unusual.
287 +
288 +`observed_bidirectional` means observation completeness, not "draw both
289 +arrows". Producers that need reverse or both arrows must set
290 +`presentation.arrow` explicitly.
291 +
292 +`direction_role` is required by the v1 schema. Missing `direction_role` is
293 +schema-invalid and must not produce an inferred arrow from `orientation` alone.
294 +The UI should render `auto` as no arrow and emit the normal missing-field
295 +diagnostic for that invalid input.
296 +
297 +For schema-valid values, the `auto` semantic diagnostic boundary is:
298 +
299 +- no diagnostic for `directed+flow`, `directed+dependency`,
300 + `hierarchical+ownership`, `undirected+none`, `undirected+observation`,
301 + `observed_bidirectional+none`, or `observed_bidirectional+observation`;
302 +- diagnostic for `directed+none`, `directed+observation`,
303 + `directed+ownership`, `hierarchical+none`, `hierarchical+flow`,
304 + `hierarchical+dependency`, `hierarchical+observation`, `undirected+flow`,
305 + `undirected+dependency`, `undirected+ownership`,
306 + `observed_bidirectional+flow`, `observed_bidirectional+dependency`, or
307 + `observed_bidirectional+ownership`.
308 +
309 +Initial UI-owned mappings are:
310 +
311 +- `size.scale`: `compact=0.85`, `normal=1.0`, `emphasized=1.18`;
312 +- `layout.repulsion`: `weakest=-200`, `weaker=-300`, `normal=-450`,
313 + `stronger=-700`, `strongest=-1000`.
314 +
315 +These numeric values are not schema and may be tuned after visual QA.
316 +`size.scale` composes with `size.mode`; it does not override data-driven
317 +sizing. Missing optional fields use neutral defaults (`scale: normal`,
318 +`repulsion: normal`) and must not trigger v1 UI fallback heuristics.
319 +
320 +Actor port bullets require explicit `ports.sources[]` when
321 +`show_bullets: true`. The source may be `links`, `evidence`, or an
322 +`actor_table`. Source column names are table-local and must remain unchanged
323 +during Cloud aggregation. Type ids inside row values and `default_type` values
324 +are rewritten only when they refer to type registries. `ports.sources[].evidence`
325 +is an evidence type id. `name_column` must reference a scalar display column,
326 +not a raw actor/link/evidence reference, array, or JSON cell.
327 +`ports.sources[].value_column` is optional and must reference a numeric source
328 +column. When present, the UI sums values for matching bullet keys and uses the
329 +sum for bullet multiplicity, overflow, and sizing. Aggregated producers should
330 +use this instead of sending repeated rows only to drive presentation.
331 +
332 +Modal/table composition uses closed source, projection, cell, and visibility
333 +tokens. Supported source kinds are `actors`, `links`, `evidence`,
334 +`actor_table`, and `relationship_table`. Supported projections include direct
335 +column values, actor-ref labels, opposite actor labels, formatted endpoints,
336 +selected-side endpoints, label-table lookups, coalesced columns, constants,
337 +and explicitly declared scalar JSON paths. Supported cell types include text,
338 +number, badge, actor link, timestamp, duration, endpoint, array count, and
339 +debug JSON. Raw JSON belongs behind `debug` visibility or an explicit scalar
340 +projection; it must not be the default polished actor modal rendering.
341 +Selected-side endpoint projections must be self-contained: the projection
342 +names source and destination actor-ref columns, plus at least one source-side
343 +endpoint column and one destination-side endpoint column.
344 +
345 +`selection.actor_click.mode: highlight_path` requires `path_table`,
346 +`path_actor_column`, and `path_order_column`. `path_actor_column` identifies
347 +path members. When one table contains different paths for different clicked
348 +actors, `path_owner_column` identifies the actor that owns each path row.
349 +`highlight_connections` requires no path table.
350 +
351 +Presentation conflict policy:
352 +
353 +- producer-local type ids, port ids, scale keys, evidence ids, table type ids,
354 + and overlay template ids are namespaced before aggregation;
355 +- identical definitions are deduplicated after canonicalization;
356 +- conflicting local definitions are preserved as distinct canonical ids rather
357 + than hard-failing aggregation;
358 +- `label_policy` belongs to the actor type presentation and follows the same
359 + namespace/deduplicate rule;
360 +- `profile_version` is diagnostic. It may help choose a preferred display
361 + profile later, but it is not a comparable semantic-version contract and must
362 + not be used to drop facts or rows.
363 +
364 +## Correlation Contract
365 +
366 +Correlation is producer-visible graph semantics, not aggregator state.
367 +Producers must not encode correlation as hidden flags on real actors, and must
368 +not expose aggregator internal states such as absorbed, candidate, equivalence
369 +class, or rewrite plan. The final aggregated output is always a normal topology
370 +payload.
371 +
372 +Correlation can resolve several shapes:
373 +
374 +- loose relationship sides, where one side of a detailed row has endpoint facts
375 + but no known actor;
376 +- visible correlation actors, where the input graph intentionally materializes
377 + unresolved peers;
378 +- weaker placeholder actors that should be replaced by stronger managed actors;
379 +- equivalent actors that should be merged and enriched with facts from multiple
380 + payloads.
381 +
382 +`data.correlation.rules` defines how independent payloads of the same topology
383 +kind can be correlated. Each rule defines:
384 +
385 +- optional `class`: `resolve_loose_side`, `replace_actor`, or
386 + `merge_enrich_actor`;
387 +- `action`: `absorb` for exact matches that remove visible correlation actors
388 + or consume loose-side placeholders and rewrite incident correlation
389 + relationships, or `link` for partial/broader matches that keep the visible
390 + correlation/materialized actor and add a weak correlation link;
391 +- `priority`: lower numbers run first;
392 +- `key_space`: namespace for exact string-key matching;
393 +- `key`: a declarative template built from point/claim table columns and
394 + literals;
395 +- `point_actor_types`: actor types that are visible correlation points when the
396 + input graph materializes points;
397 +- optional `claim_actor_types`: actor types that may satisfy the point;
398 +- optional `correlation_link_types`: link types that connect real actors to
399 + correlation actors and may be consumed/replaced by the rule;
400 +- `output_link_type`: link type emitted for rewritten absorb links or visible
401 + partial correlation links.
402 +
403 +`data.correlation.points` is a compact table of visible correlation actors and keys.
404 +`data.correlation.claims` is a compact table of real actors and keys they can
405 +satisfy. Both tables require `actor`, `rule`, and the key columns referenced by
406 +their rules.
407 +
408 +The aggregator is intentionally agnostic. It builds normalized keys from
409 +declared columns and literals, applies rule priority, and handles ambiguity
410 +conservatively. It must not need new code to understand every future IP, port,
411 +MAC, chassis id, object id, label, or topology-domain key.
412 +
413 +No match keeps the visible correlation actor or loose-side materialization
414 +visible. Ambiguous matches remain unresolved and produce diagnostics. NAT or
415 +other alias evidence can be modeled by adding extra point/claim rows for the
416 +same actor and rule; aliases add facts without mutating the original
417 +observation.
418 +
419 +Correlation links must be semantic link types even for single-node payloads.
420 +The legend must include visible correlation actors and links when they are
421 +visible, so users can distinguish unresolved, partial, inferred, and resolved
422 +graph relationships.
423 +
424 +## Telemetry Overlays
425 +
426 +Refreshable traffic, state, error, packet, or utilization data is represented by
427 +overlay templates and per-actor/per-link refs.
428 +
429 +Templates define the query mechanism once. Refs provide only template ids and
430 +parameters. Aggregated links merge refs according to the template merge policy.
431 +
432 +## Compatibility
433 +
434 +Production payloads carry canonical topology facts, not compatibility
435 +reconstruction instructions. Projection code for parity with deployed
436 +compatibility consumers is test or rollout code only.
437 +
438 +Agent/backend contracts and docs should point new work to
439 +`netdata.topology.v1`. Temporary compatibility handling belongs in isolated
440 +Cloud frontend adapters during rollout.
441 +
442 +## Validation
443 +
444 +Topology producer changes must include:
445 +
446 +- JSON Schema validation against `FUNCTION_TOPOLOGY_SCHEMA.json`;
447 +- semantic validation for table column lengths and reference bounds;
448 +- fixture or corpus tests for payload size and evidence preservation;
449 +- tests proving direction and aggregation policy are honored;
450 +- checks that evidence is not silently truncated.
451 +
452 +Cloud topology aggregation service readiness also requires service-level
453 +fixtures for every topology kind covered by this contract. `network-connections`
454 +is the high-cardinality benchmark, but the service is not considered ready if
455 +the UI can use it for only some topology kinds while bypassing it for others.
456 +
457 +## Migration Notes
458 +
459 +`topology:network-connections` now emits `netdata.topology.v1` directly from
460 +the C network-viewer Function. Aggregated mode is the default and emits compact
461 +actor, graph-link, and actor-owned `socket_ports` tables. Detailed mode adds a
462 +shared socket relationship-evidence table for exact tuple matching and
463 +drilldowns. Process actor size uses the actor row `socket_count` metric, while
464 +process port bullets read the `socket_ports.socket_count` value column so an
465 +aggregated port row can represent several sockets. The producer no longer emits
466 +the superseded old-schema presentation object or duplicated actor-nested socket
467 +modal tables. It now emits compact graph presentation metadata in type
468 +definitions plus `data.presentation`, and repeated string columns use
469 +dictionary encoding when it reduces raw payload size.
470 +
471 +Network-connections distinguishes unresolved endpoint links from aggregator
472 +correlation output. `endpoint_socket` connects a process to a visible unresolved
473 +endpoint actor and must not use the farthest layout distance because that makes
474 +single-node maps zoom out unnecessarily. `correlated_socket` is the output link
475 +type after exact endpoint absorption by an aggregator and may use farthest
476 +distance to keep independent topology clusters from blending.
477 +
478 +Network-connections modal composition is producer-declared. Self/node actors use
479 +a `Processes` section over `ownership` graph links filtered by link type.
480 +Network-connections socket link types use `direction_role: "dependency"` and
481 +are client-to-server: `src_actor` is the client/dependant and `dst_actor` is the
482 +server/dependency target. Non-node actors therefore use two primary sections in
483 +both aggregated and detailed mode:
484 +`Dependencies`, filtered to rows where the selected actor is `src_actor`, and
485 +`Dependants`, filtered to rows where the selected actor is `dst_actor`.
486 +Aggregated mode reads these sections from `tables.relationship.connections`;
487 +detailed mode reads them from `evidence.socket`. `socket_ports` is an actor
488 +inventory for process port bullets only; it is not a standalone modal tab for
489 +network-connections.
490 +
491 +`topology:snmp` now emits `netdata.topology.v1` from the Function handler
492 +through an adapter over the existing SNMP topology engine output. The adapter
493 +preserves actors, links, L2 observation evidence, actor metadata, and actor
494 +custom detail tables. Remaining SNMP refinement is to promote interface metric
495 +lookup fragments into first-class overlay templates/refs.
496 +
497 +SNMP modal composition must be port-centric for managed device actors. A managed
498 +device modal uses actor-label identification for important device facts, a
499 +primary `Ports` section over `actor_ports`, and a `Port Neighbors` section over
500 +`actor_port_links`. Generic graph-link `Links` sections are reserved for
501 +endpoint, segment, or custom actors that do not own port inventory.
502 +
503 +SNMP `actor_ports` exposes real port identity and status as typed columns:
504 +SNMP `if_index` as the visible numeric port ID when known, source `port_id`,
505 +display `name`, `if_name`, `if_descr`, `if_alias`, MAC, speed, status, mode,
506 +role, VLAN, FDB, link, and neighbor counts. It must never fabricate numeric
507 +port IDs; row order and generated sequences must not be used. `if_index` must
508 +come from the device/SNMP facts and must align with `actor_port_links.if_index`.
509 +
510 +SNMP `actor_ports` may also carry compact expanded-row neighbor columns such as
511 +nullable `neighbor_actor` and `neighbor_port_name`, derived from graph-link
512 +endpoint facts. These columns make the port row clickable without duplicating
513 +raw LLDP/CDP/FDB/ARP/STP evidence.
514 +
515 +SNMP `actor_port_links` is a compact actor-owned modal index over existing graph
516 +links and evidence. It has one row per incident actor side and carries the local
517 +`if_index`/port name, remote actor, remote port facts, protocol, link type,
518 +state, evidence count, confidence, inference, attachment mode, and timestamps.
519 +It exists so device modals can align neighbor rows with the same port identity
520 +shown in `actor_ports`; it is not a second copy of raw evidence.
521 +
522 +SNMP polished UI must not depend on raw `actor_metadata` and endpoint JSON.
523 +Important scalar/count summary values live in typed actor or actor-detail
524 +columns and are also available through `actor_labels`. Nested neighbors, VLANs,
525 +unknown custom port attributes, and endpoint objects stay in expanded or debug
526 +sections unless a structured child table is defined. Link endpoint port labels
527 +must come only from real port fields such as `port_name`, `if_name`,
528 +`if_descr`, or source `port_id`; actor labels such as `display_name` or
529 +`sys_name` must not be used as port-name fallbacks.
530 +
531 +`topology:streaming` now emits `netdata.topology.v1` directly from the C
532 +Function. It models streaming agents as compact actor rows, streaming/virtual/
533 +stale relationships as directed graph links with matching evidence types, and
534 +keeps `stream_path`, retention, inbound, and outbound modal data as typed
535 +actor-detail or relationship-summary tables. Streaming also emits graph
536 +presentation metadata for highlight-path behavior, legend, link styles, and
537 +port bullets. Streaming hops remain signed so stale path values are not
538 +corrupted.
539 +
540 +Streaming parent actor size is data-driven by the actor row
541 +`retained_node_count` metric, not by graph degree or direct child count. This
542 +count represents nodes for which the parent has retained DB data, including
543 +self, virtual nodes, stale nodes, and transit descendants when they have
544 +retention state. The parent actor type must declare `presentation.size:
545 +{"mode":"metric","metric_column":"retained_node_count"}`. Parent port bullets
546 +represent child or vnode streams attached to the parent side of streaming graph
547 +links, so the parent `ports.sources[]` entry over `links` must use
548 +`actor_column: "dst_actor"` with a scalar child/node display column such as
549 +`port_name`.
550 +
551 +Streaming modal composition emits `actor_labels`, complete host labels where
552 +available, host/system metadata labels needed by old summaries, and typed
553 +OS/architecture/CPU columns. The actor modal header must select important
554 +identity/status labels from `actor_labels`, including hostname, node type,
555 +stream status, ingest status, health status, retained-node count where
556 +applicable, direct-child count where applicable, OS/platform labels, and Agent
557 +version. Existing `stream_path`, `retention`, and `inbound` tables have the
558 +right actor-ref shape for recipe-based table rendering. The `outbound` table
559 +must use the parent-owned shape described below; a table that only records the
560 +selected actor's own upstream destination is insufficient for parent operator
561 +workflows.
562 +
563 +Streaming actor modal identification is role-specific. Host-like actors
564 +(`parent`, `child`, and `stale`) should expose operational status plus
565 +OS/hardware/platform labels such as OS, OS version, kernel, architecture, CPU
566 +model, cores, RAM, virtualization, container, cloud provider/type/region, and
567 +Agent version. Parents additionally expose `retained_node_count` and
568 +`child_count` so the visual size and direct attachments are both explainable.
569 +Vnode actors should expose inventory/device labels such as vnode type, vendor,
570 +model, address, location, sys object id, LLDP system name, and status. Long
571 +stable identifiers such as `machine_guid` and `node_id` remain in the full
572 +Labels tab by default unless a future product decision explicitly promotes them.
573 +
574 +Streaming actor modals must keep those tables as the single source of truth and
575 +must not duplicate rows only for modal display. The default visible sections are:
576 +
577 +- `Stream path`: rows from `stream_path` filtered by `actor`, ordered by
578 + `path_index`. The table shows the selected actor's own path only; virtual
579 + nodes and children have their own actors and therefore their own path rows.
580 + `since` and `first_time` must be populated from the best canonical source
581 + available for every path row. Synthetic path rows added only for rendering or
582 + highlighting must still carry timestamps when the producer can derive them
583 + from the adjacent path edge, the selected actor's ingest status, or database
584 + first-time status. They may be null only when the Agent genuinely does not
585 + know the value.
586 +- `Retained nodes`: rows from the same `retention` table filtered by
587 + `observer_actor`; this view answers which nodes' data the selected actor
588 + maintains. The table must include self, virtual nodes, direct children,
589 + transit descendants, and stale/archived hosts when those hosts are present in
590 + the Agent root index and have retention state. It must show retained node,
591 + node type, retention status, from/to timestamps, duration, metrics, instances,
592 + and contexts. `db_from` and `db_to` may be null only when the database status
593 + genuinely has no time range.
594 +- `Received nodes`: rows from `inbound` filtered by `parent_actor`; this view
595 + represents children, virtual nodes, stale nodes, and descendants received or
596 + transiting through the selected parent. `source_actor` is the immediate actor
597 + from which the selected parent receives the row. For direct local receipt,
598 + `source_actor` should be the child/vnode actor itself; it should be null only
599 + when the immediate source is genuinely unknown.
600 +- `Outbound streams`: rows from `outbound` filtered by the sending parent
601 + actor, not by the streamed node actor. This view answers which node payloads
602 + the selected parent currently streams, and where it streams each one. Each row
603 + must include the streamed node actor, destination actor when known, status,
604 + age, hops, TLS, compression, and useful stream/replication/count metrics when
605 + available. In clustered-parent setups, the selected parent must list self,
606 + virtual nodes, direct children, and transit descendants that are sent to each
607 + upstream destination.
608 +
609 +The old `Retention for node` default section is not part of the current
610 +streaming modal contract. The canonical `retention` table still keeps both
611 +`actor` and `observer_actor` so Cloud aggregation can preserve multiple
612 +retaining parents for the same node. If a future modal needs a selected-node
613 +"who retains me" view, it must be explicitly named `Retained by` and must not
614 +replace the parent-owned `Retained nodes` view.
.agents/sow/specs/topology-modes-correlation-aggregation.md new
+816
@@ -0,0 +1,816 @@
1 +# Spec - Topology Modes, Correlation, Aggregation, And Actor Identification
2 +
3 +## Status
4 +
5 +Implementation contract introduced by SOW-0028.
6 +
7 +SOW-0028 completed the cross-repo compatibility layer for modes, modal
8 +identification, correlation classes, and table merge policy. The stronger
9 +detailed network-connections loose-side graph model remains the target
10 +behavior and is tracked by SOW-0029 because it requires a separate
11 +Agent/UI/aggregator execution pass.
12 +
13 +## Purpose
14 +
15 +Topology payloads must let operators inspect a topology at the right level:
16 +
17 +- exact evidence when they need detailed troubleshooting;
18 +- compact relationships when they need an infrastructure-level map;
19 +- consistent cross-node correlation when many independently produced payloads
20 + are merged;
21 +- useful actor modals without duplicating the same facts only for display.
22 +
23 +The contract must remain topology-agnostic. The UI and aggregator must not learn
24 +domain words such as process, router, parent, child, endpoint, LLDP, socket, or
25 +retention as hardcoded behavior. Producers describe identities, modes, merge
26 +rules, table merge policies, and presentation recipes in the payload.
27 +
28 +## Terms
29 +
30 +- **Producer**: the Agent Function that emits one `netdata.topology.v1`
31 + payload, for example network-connections, SNMP/L2, streaming, or vSphere.
32 +- **Aggregator**: Cloud service that fans out to many producers, decodes their
33 + payloads, correlates them, optionally aggregates detail, and returns a normal
34 + `netdata.topology.v1` payload.
35 +- **UI**: Cloud frontend topology renderer and actor/link modal renderer.
36 +- **Detailed mode**: payload or returned view that keeps the finest evidence
37 + grain the producer exposes for troubleshooting.
38 +- **Aggregated mode**: payload or returned view that groups detailed evidence
39 + into compact relationships for map readability.
40 +- **Known actor**: an entity the producer knows exists, such as a process, host,
41 + SNMP device, interface, streaming node, vSphere object, or Kubernetes object.
42 +- **Loose side**: one side of a relationship row that has endpoint facts but no
43 + actor reference yet.
44 +- **Materialized actor**: actor created from loose-side facts for presentation
45 + or partial correlation, such as an endpoint grouped by IP.
46 +- **Replacement**: aggregation action where a weaker actor is removed and all
47 + incident relationships are rewired to a stronger actor.
48 +- **Enrichment**: aggregation action where rows from multiple actors with the
49 + same identity are merged into one actor, preserving all non-conflicting facts.
50 +- **Evidence table**: lossless or near-lossless relationship facts, for example
51 + sockets, L2 observations, or streaming relationships.
52 +- **Relationship summary table**: compact relationship rows at a grain between
53 + graph links and detailed evidence.
54 +- **Actor labels**: actor-owned key/value rows for modal labels and display
55 + selection. They are not a replacement for typed identity, matching, grouping,
56 + sorting, filtering, or aggregation columns.
57 +
58 +## Global Rules
59 +
60 +### Mode Request
61 +
62 +The user-facing request key for topology mode is `__topology_mode`.
63 +
64 +Allowed values:
65 +
66 +- `detailed`
67 +- `aggregated`
68 +
69 +If the key is absent, each Function uses its documented default. Producers that
70 +do not have a meaningful detailed/aggregated difference should not expose a mode
71 +selector only to return identical output.
72 +
73 +Mode-capable producers declare `data.view.supported_modes`. Consumers treat an
74 +absent field or a single-value field as mode-invariant and must not show a
75 +detailed/aggregated toggle for that payload.
76 +
77 +### Aggregator Fanout
78 +
79 +The aggregator must consume detailed payloads whenever a producer supports
80 +detail mode.
81 +
82 +When the user asks the aggregator for `__topology_mode=aggregated`, the
83 +aggregator must rewrite fanout requests to producers as
84 +`__topology_mode=detailed` before correlation and aggregation. This prevents
85 +early information loss before cross-node matching. After correlation, the
86 +aggregator returns either detailed or aggregated output according to the
87 +original user request.
88 +
89 +If a producer does not expose `__topology_mode`, the aggregator must not invent
90 +that parameter for it. SNMP/L2 and streaming are expected to be mode-invariant
91 +unless a future producer change defines a real mode difference.
92 +
93 +### Final Output
94 +
95 +Aggregator internal states must not appear in final payloads. Terms such as
96 +absorbed, candidate, rewrite plan, partial class, or equivalence set may exist
97 +inside the service, but final topology output contains only normal actors,
98 +links, tables, labels, presentation, and diagnostics.
99 +
100 +### No Duplicate Display Facts
101 +
102 +Do not copy high-cardinality evidence rows only to make modal tables easier.
103 +Modal sections must select and project existing actors, links, evidence,
104 +relationship tables, actor tables, and actor labels.
105 +
106 +Small scalar facts may appear in more than one plane when the grains differ. For
107 +example, a graph link and a relationship-summary row may both carry
108 +`socket_count`; the graph link is the renderable relationship, while the summary
109 +row is the modal/drilldown grain.
110 +
111 +### Actor Modal Identification
112 +
113 +The modal identification area is part of the schema contract.
114 +
115 +`types.actor_types.<id>.presentation.modal.labels` must be extended with an
116 +ordered producer-selected identification list over the existing actor label
117 +table. The UI renders those selected label keys near the actor title. The full
118 +label table remains available in the Labels tab.
119 +
120 +Target shape:
121 +
122 +```json
123 +{
124 + "labels": {
125 + "enabled": true,
126 + "table": "actor_labels",
127 + "actor_column": "actor",
128 + "key_column": "key",
129 + "value_column": "value",
130 + "identification": {
131 + "enabled": true,
132 + "fields": [
133 + { "key": "process", "label": "Process", "max_values": 1 },
134 + { "key": "username", "label": "User", "max_values": 1 },
135 + { "key": "cmdline", "label": "Command", "max_values": 1 }
136 + ]
137 + }
138 + }
139 +}
140 +```
141 +
142 +Rules:
143 +
144 +- `identification.fields[]` selects rows from `actor_labels` by `key`.
145 +- Selection is per selected modal actor through `actor_column`.
146 +- Repeated values are ordered by `value_index` when present.
147 +- Missing selected keys are skipped; they do not create empty labels.
148 +- `max_values` limits displayed values for one key. The full Labels tab still
149 + shows all values.
150 +- The UI must not guess important labels from key names.
151 +- Producers must not duplicate these values into a separate modal-only table.
152 +
153 +## Schema Additions Required
154 +
155 +This spec requires these schema extensions beyond the currently deployed v1
156 +contract:
157 +
158 +1. Modal label identification metadata:
159 + `modal.labels.identification.enabled` and
160 + `modal.labels.identification.fields[]`.
161 +2. Link-side materialization policy for tables that can carry loose sides:
162 + producers must declare how a loose side can be grouped into presentation
163 + actors when a detailed row has only one real actor.
164 +3. Aggregation/correlation rules must support three semantic outcomes:
165 + - loose-side resolution;
166 + - actor replacement;
167 + - actor enrichment.
168 +4. Table merge policies must be explicit enough for the aggregator to merge
169 + streaming path, retention, inbound, outbound, SNMP observation, and
170 + relationship-summary rows without domain-specific code.
171 +
172 +The implementation may encode these additions in the most compact shape that
173 +fits the existing JSON schema style. This spec defines semantics; exact field
174 +names are accepted when they are schema-valid, documented, and used uniformly
175 +by Agent, UI, and aggregator.
176 +
177 +## Correlation And Aggregation Model
178 +
179 +### Rule Classes
180 +
181 +Correlation rules are declarative. The aggregator builds keys from columns and
182 +literals; it does not understand domain semantics.
183 +
184 +Required rule classes:
185 +
186 +- `resolve_loose_side`: matches a loose relationship side to a known actor or
187 + to a materialized partial actor.
188 +- `replace_actor`: replaces weaker actors with stronger actors and rewires
189 + incident links/tables.
190 +- `merge_enrich_actor`: merges actors with the same identity and combines their
191 + labels, attributes, links, evidence, and detail tables according to declared
192 + table policies.
193 +
194 +### Priority
195 +
196 +Rules run by ascending priority number. Exact rules must run before broader or
197 +partial rules.
198 +
199 +Example:
200 +
201 +```json
202 +{
203 + "rules": {
204 + "socket_exact": {
205 + "class": "resolve_loose_side",
206 + "priority": 10,
207 + "key_space": "socket",
208 + "key": [
209 + { "column": "protocol" },
210 + { "literal": "|" },
211 + { "column": "address_space" },
212 + { "literal": "|" },
213 + { "column": "ip" },
214 + { "literal": ":" },
215 + { "column": "port" }
216 + ],
217 + "output_link_type": "socket"
218 + },
219 + "ip_partial": {
220 + "class": "resolve_loose_side",
221 + "priority": 100,
222 + "key_space": "ip",
223 + "key": [
224 + { "column": "address_space" },
225 + { "literal": "|" },
226 + { "column": "ip" }
227 + ],
228 + "output_link_type": "partial_endpoint"
229 + }
230 + }
231 +}
232 +```
233 +
234 +### Ambiguity
235 +
236 +If one point matches exactly one claim, apply the rule.
237 +
238 +If one point matches multiple claims with the same priority, the aggregator must
239 +not pick randomly. It keeps the point unresolved or materializes a partial actor
240 +according to the rule, and records a diagnostic.
241 +
242 +If no match exists, the point or loose side remains unresolved and may be
243 +materialized for UI display according to the declared materialization policy.
244 +
245 +### Alias And NAT Evidence
246 +
247 +NAT, load balancer, or alias information is modeled as additional keys for the
248 +same point or claim. Alias rows add match possibilities; they do not mutate or
249 +delete the original observation.
250 +
251 +## Table Merge Policies
252 +
253 +Every table type that can cross producer boundaries needs a merge policy.
254 +
255 +Required dimensions:
256 +
257 +- `key`: columns that identify equivalent rows.
258 +- `action`: one of `deduplicate`, `append`, `set_union`, `merge_metrics`,
259 + `latest`, or `preserve`.
260 +- `metrics`: per numeric column merge operation, such as `sum`, `min`, `max`,
261 + `avg_weighted`, or `latest`.
262 +- `conflicts`: how non-key scalar conflicts are handled. Allowed policies are
263 + `prefer_claim`, `prefer_newest_agent`, `preserve_all`, or `diagnostic`.
264 +
265 +General rules:
266 +
267 +- Relationship evidence normally uses `append` or `deduplicate`.
268 +- Relationship summaries normally use `merge_metrics` on the declared key.
269 +- Actor labels use `set_union` keyed by actor, key, value, source, kind, and
270 + value_index.
271 +- Streaming retention uses `preserve` or `append` when two parents retain data
272 + for the same node because each retaining parent is meaningful.
273 +- Raw JSON columns cannot participate in key construction unless a scalar JSON
274 + path is explicitly declared in the table policy. Prefer typed scalar columns.
275 +
276 +## Layer Responsibilities
277 +
278 +### Agent Direct View
279 +
280 +The Agent returns the producer's local topology.
281 +
282 +- If detailed and aggregated modes are meaningful, the Agent may expose
283 + `__topology_mode`.
284 +- Detailed mode keeps the finest useful evidence.
285 +- Aggregated mode returns a readable local graph and modal relationship
286 + summaries.
287 +- If modes are identical, the Agent should not expose a mode option.
288 +
289 +### Aggregator View
290 +
291 +The aggregator always collects the highest-detail useful input it can get.
292 +
293 +- Requested detailed output:
294 + - fan out detailed when supported;
295 + - correlate and enrich;
296 + - return detailed rows with resolved actors where exact matches exist;
297 + - keep unresolved loose sides/materialized actors when no match exists.
298 +- Requested aggregated output:
299 + - fan out detailed when supported;
300 + - correlate and enrich first;
301 + - aggregate into compact actors, links, relationship summaries, and modal
302 + tables according to schema policies.
303 +
304 +### UI View
305 +
306 +The UI renders the payload it receives.
307 +
308 +- It does not run domain-specific correlation.
309 +- It may materialize loose sides for direct-Agent detailed views only if the
310 + payload declares the materialization policy.
311 +- It renders actor modal identification from
312 + `modal.labels.identification.fields[]`.
313 +- It renders the full Labels tab from `actor_labels`.
314 +- It renders modal tables from schema recipes and existing data planes.
315 +
316 +## Network-Connections
317 +
318 +### Domain Model
319 +
320 +A socket observation is associated with an observed process actor and two
321 +dependency endpoint tuples:
322 +
323 +```text
324 +protocol, client_ip, client_port, server_ip, server_port, state
325 +```
326 +
327 +For inbound sockets, the observed process owns the server tuple.
328 +For outbound sockets, the observed process owns the client tuple.
329 +For local/same-node sockets, both process actors may be known, but the emitted
330 +topology direction remains client-to-server.
331 +For listening sockets, there is no remote side.
332 +
333 +The exact remote tuple is required for cross-node correlation, but creating an
334 +actor per `IP:PORT` in a single-node graph can explode actor count and force
335 +layout noise. Therefore detailed mode must preserve the exact tuple without
336 +requiring every tuple to be an actor.
337 +
338 +### Agent Aggregated Mode
339 +
340 +Agent aggregated mode is a readable local process dependency map.
341 +
342 +Rules:
343 +
344 +- Every graph link has two actor references.
345 +- The producer creates materialized endpoint actors for unknown peers using the
346 + peer IP plus address space.
347 +- Socket dependency link types use `direction_role: "dependency"` and graph
348 + links point from client/dependant actor to server/dependency actor.
349 +- Relationship-summary rows are collapsed by actor pair, protocol, and state.
350 + They keep dependency endpoint IPs (`client_ip`, `server_ip`) and merged
351 + metrics, but not per-socket ephemeral ports.
352 +- Process actor size is driven by `socket_count`.
353 +- Port bullets are driven by compact actor-owned port summaries, using a
354 + numeric count column.
355 +- Node-to-process ownership links are graph-coherence links, not networking
356 + dependencies.
357 +
358 +Synthetic example:
359 +
360 +```json
361 +{
362 + "view": { "mode": "aggregated" },
363 + "actors": [
364 + { "id": 1, "type": "node", "display_name": "node-a" },
365 + { "id": 2, "type": "process", "display_name": "api", "socket_count": 14 },
366 + { "id": 3, "type": "endpoint", "display_name": "198.51.100.20", "ip": "198.51.100.20" }
367 + ],
368 + "links": [
369 + { "src_actor": 1, "dst_actor": 2, "type": "ownership" },
370 + { "src_actor": 2, "dst_actor": 3, "type": "endpoint_socket", "protocol": "tcp", "socket_count": 12 }
371 + ],
372 + "tables": {
373 + "relationship": {
374 + "connections": [
375 + { "src_actor": 2, "dst_actor": 3, "protocol": "tcp", "client_ip": "192.0.2.10", "server_ip": "198.51.100.20", "socket_count": 12 }
376 + ]
377 + }
378 + }
379 +}
380 +```
381 +
382 +The example uses row objects for readability. Production uses compact tables.
383 +
384 +### Agent Detailed Mode
385 +
386 +Agent detailed mode preserves exact socket evidence.
387 +
388 +Rules:
389 +
390 +- Graph links still have two actor references.
391 +- Unknown peers are still visible endpoint actors, grouped by peer IP plus
392 + address space.
393 +- Socket evidence preserves the exact client/server tuple, including ports, so
394 + Cloud correlation has no tuple loss.
395 +- Listening rows have no remote side and no fake remote actor.
396 +- Local sockets with both processes known have two process actor refs.
397 +
398 +Synthetic example:
399 +
400 +```json
401 +{
402 + "view": { "mode": "detailed" },
403 + "actors": [
404 + { "id": 1, "type": "node", "display_name": "node-a" },
405 + { "id": 2, "type": "process", "display_name": "api" },
406 + { "id": 3, "type": "endpoint", "display_name": "198.51.100.20", "ip": "198.51.100.20" }
407 + ],
408 + "evidence": {
409 + "socket": [
410 + {
411 + "src_actor": 2,
412 + "dst_actor": 3,
413 + "protocol": "tcp",
414 + "client_ip": "192.0.2.10",
415 + "client_port": 50120,
416 + "server_ip": "198.51.100.20",
417 + "server_port": 443
418 + }
419 + ]
420 + }
421 +}
422 +```
423 +
424 +### Aggregator Network-Connections
425 +
426 +The aggregator receives detailed rows.
427 +
428 +Exact match:
429 +
430 +```text
431 +node-a api claims client tcp 192.0.2.10:50120, points at server tcp 198.51.100.20:443
432 +node-b nginx claims server tcp 198.51.100.20:443, points at client tcp 192.0.2.10:50120
433 +```
434 +
435 +Result:
436 +
437 +```text
438 +node-a/api -> node-b/nginx
439 +```
440 +
441 +No endpoint actors remain for the exact match. The loose side was resolved to a
442 +known actor and the final graph is a normal process-to-process dependency.
443 +
444 +Partial match:
445 +
446 +```text
447 +node-a api outbound -> 198.51.100.20:443
448 +node-b is known to own 198.51.100.20
449 +node-b has no matching process/socket row at collection time
450 +```
451 +
452 +Result:
453 +
454 +```text
455 +node-a/api -> node-b/[materialized endpoint for 198.51.100.20]
456 +```
457 +
458 +The graph remains truthful: the dependency points at node-b, but the exact
459 +process could not be proven.
460 +
461 +No match:
462 +
463 +```text
464 +node-a/api -> materialized endpoint 198.51.100.20
465 +```
466 +
467 +The unresolved endpoint remains visible with presentation that clearly differs
468 +from resolved process links.
469 +
470 +### UI Network-Connections
471 +
472 +Direct Agent aggregated:
473 +
474 +- Show process and endpoint actors.
475 +- Show two-sided graph links.
476 +- Non-node actor modals show `Dependencies` from relationship-summary rows
477 + where the selected actor is `src_actor`, and `Dependants` where it is
478 + `dst_actor`.
479 +
480 +Direct Agent detailed:
481 +
482 +- Show known actors.
483 +- Show visible endpoint actors for unknown peers, grouped by peer IP and address
484 + space.
485 +- Non-node actor modals show `Dependencies` and `Dependants` from exact socket
486 + evidence using the same `src_actor` / `dst_actor` split.
487 +
488 +Aggregator aggregated:
489 +
490 +- Show the post-correlation compact dependency map.
491 +- Exact cross-node matches are process-to-process.
492 +- Unmatched loose sides are materialized according to policy.
493 +
494 +Aggregator detailed:
495 +
496 +- Show exact socket evidence with resolved actors where possible.
497 +- Unresolved rows retain their loose-side facts.
498 +
499 +## SNMP/L2
500 +
501 +### Domain Model
502 +
503 +SNMP/L2 topology observations are device, interface, neighbor, forwarding,
504 +ARP, bridge, VLAN, and protocol facts. A graph link represents an observed or
505 +inferred L2 relationship between two actors.
506 +
507 +SNMP is not a loose-side topology. Every link should have two actors in both
508 +Agent and aggregator views.
509 +
510 +### Mode Behavior
511 +
512 +SNMP/L2 detailed and aggregated modes are currently a no-op. The producer should
513 +not expose `__topology_mode` until it has a real lower/higher-grain distinction.
514 +
515 +If a global Cloud topology request asks for aggregated output, the aggregator
516 +still consumes the same SNMP payload and returns the same semantic grain after
517 +correlation/replacement.
518 +
519 +### Correlation Behavior
520 +
521 +SNMP mainly uses actor replacement.
522 +
523 +Examples:
524 +
525 +- A managed device actor is stronger than an LLDP remote placeholder that has
526 + the same chassis id.
527 +- A managed interface actor is stronger than an inferred endpoint that has the
528 + same MAC/interface identity.
529 +- A discovered management IP can help match a placeholder to a managed device,
530 + but ambiguous matches must not be chosen randomly.
531 +
532 +Replacement example:
533 +
534 +```text
535 +payload-a: switch-a port 10 -> lldp-remote(chassis=aa:bb:cc)
536 +payload-b: managed-switch-b(chassis=aa:bb:cc)
537 +```
538 +
539 +Result:
540 +
541 +```text
542 +switch-a port 10 -> managed-switch-b
543 +```
544 +
545 +The weaker LLDP remote actor is removed from the aggregated output and its
546 +incident links/tables are rewired to the managed device actor.
547 +
548 +### UI SNMP/L2
549 +
550 +The UI should not expose a detailed/aggregated toggle for SNMP/L2 unless the
551 +payload declares supported modes.
552 +
553 +Device modals should remain port-centric:
554 +
555 +- actor identification: device name, management IP, vendor, model, role, and
556 + other selected labels;
557 +- full labels tab: all labels;
558 +- ports table: one row per known interface/port, with SNMP `if_index` as the
559 + visible real numeric port ID when known, and the port name;
560 +- expanded port rows: show a clickable neighbor actor and neighbor port name
561 + when graph-link facts can align the port to a remote actor;
562 +- links/neighbor information: derived from the same port rows or aligned
563 + relationship rows so local port identity never contradicts the port table.
564 +
565 +## Streaming
566 +
567 +### Domain Model
568 +
569 +Streaming topology describes Netdata Agent streaming relationships:
570 +
571 +```text
572 +child -> parent
573 +parent <-> parent
574 +virtual/stale/remote nodes represented as actors
575 +```
576 +
577 +All actors are real topology actors from the streaming view. Links always have
578 +two actor refs. Streaming is not a loose-side topology.
579 +
580 +### Mode Behavior
581 +
582 +Streaming detailed and aggregated modes are currently a no-op. The producer
583 +should not expose `__topology_mode` until it has a real lower/higher-grain
584 +distinction.
585 +
586 +The aggregator still consumes the same streaming payload for global aggregated
587 +requests and returns merged/enriched streaming topology.
588 +
589 +### Correlation Behavior
590 +
591 +Streaming uses actor enrichment and table merging by `machine_guid`.
592 +
593 +Example:
594 +
595 +```text
596 +child-1 -> parent-1 <-> parent-2 <- child-2
597 +```
598 +
599 +Both parents may report facts about the same node. The aggregator must not show
600 +duplicate actors for the same `machine_guid`. It merges those actors and then
601 +merges their tables according to table policy.
602 +
603 +Required merge behavior:
604 +
605 +- actor labels: set union;
606 +- actor scalar facts: prefer non-empty, newest Agent version when explicitly
607 + comparable, otherwise preserve conflicts in diagnostics or expanded labels;
608 +- stream path rows: deduplicate identical path membership rows;
609 +- retention rows: preserve each retaining parent/source row, because multiple
610 + parents retaining the same child are meaningful;
611 +- inbound stream rows: merge by parent actor, child actor, immediate source
612 + actor when known, and relationship type, with numeric metrics merged by table
613 + policy;
614 +- outbound stream rows: merge by sending parent actor, streamed node actor,
615 + destination actor when known, and stream state, with numeric metrics merged by
616 + table policy;
617 +- links: merge by source actor, destination actor, type, protocol, and state,
618 + then merge metrics/evidence according to link type policy.
619 +
620 +Retention example:
621 +
622 +```text
623 +payload-parent-a: parent-a retains child-x for tier 0
624 +payload-parent-b: parent-b retains child-x for tier 0
625 +payload-child-x: child-x self retention tier 0
626 +```
627 +
628 +Result:
629 +
630 +```text
631 +actor child-x modal Retention table has 3 rows:
632 + retaining actor parent-a
633 + retaining actor parent-b
634 + retaining actor child-x
635 +```
636 +
637 +These rows must not be deduplicated away solely because the retained child is
638 +the same.
639 +
640 +### UI Streaming
641 +
642 +The UI should not expose a detailed/aggregated toggle for streaming unless the
643 +payload declares supported modes.
644 +
645 +Actor modal identification should show selected labels such as role, hostname,
646 +machine GUID when useful, and stream status. The full Labels tab remains
647 +complete.
648 +
649 +Tables:
650 +
651 +- stream path: deduplicated path rows for the selected actor only. Timestamps
652 + must be populated for every path row when the producer or aggregator can
653 + derive them; synthetic rows used for highlighting are not allowed to drop
654 + known timing facts;
655 +- retained nodes: all nodes whose data is maintained by the selected actor,
656 + using the same retention table filtered by `observer_actor`. This is the
657 + default retention view in the current modal contract;
658 +- received nodes: children, virtual nodes, stale nodes, and descendants
659 + received or transiting through the selected parent. The immediate source must
660 + be populated when known; direct local receipt should use the child/vnode actor
661 + as the source instead of rendering an empty value;
662 +- outbound streams: every node payload the selected parent sends upstream,
663 + including self, virtual nodes, direct children, and transit descendants. Rows
664 + are owned by the sending parent and must show the streamed node and the
665 + destination.
666 +
667 +The current default modal contract does not show a separate `Retention for node`
668 +section. The underlying retention table still preserves `actor` and
669 +`observer_actor` so aggregated/cloud views can add an explicitly named
670 +`Retained by` section later without changing the facts.
671 +
672 +Highlight path must use the deduplicated stream-path table, not direct sibling
673 +selection only.
674 +
675 +## Cross-Topology Examples
676 +
677 +### Agent Detailed To Aggregator Aggregated
678 +
679 +Input request to Cloud:
680 +
681 +```text
682 +function=topology:network-connections __topology_mode=aggregated
683 +```
684 +
685 +Aggregator fanout:
686 +
687 +```text
688 +node-a: topology:network-connections __topology_mode=detailed
689 +node-b: topology:network-connections __topology_mode=detailed
690 +```
691 +
692 +Aggregator work:
693 +
694 +1. Decode detailed socket evidence from both nodes.
695 +2. Resolve exact loose-side socket keys.
696 +3. Materialize unresolved partial endpoints.
697 +4. Aggregate graph links and relationship summaries.
698 +
699 +Returned payload:
700 +
701 +```json
702 +{
703 + "view": { "mode": "aggregated" },
704 + "actors": "... compact post-correlation actor table ...",
705 + "links": "... compact post-correlation graph links ...",
706 + "tables": {
707 + "relationship": {
708 + "connections": "... aggregated drilldown rows ..."
709 + }
710 + }
711 +}
712 +```
713 +
714 +### Agent Detailed To UI Direct Detailed
715 +
716 +Input request to Agent:
717 +
718 +```text
719 +function=topology:network-connections __topology_mode=detailed
720 +```
721 +
722 +UI work:
723 +
724 +1. Decode known actors and exact socket evidence.
725 +2. Render known actors.
726 +3. Materialize loose endpoints only as declared by the payload, normally by IP.
727 +4. Show exact socket rows in process modals.
728 +
729 +### SNMP Global Aggregated
730 +
731 +Input request to Cloud:
732 +
733 +```text
734 +function=topology:snmp __topology_mode=aggregated
735 +```
736 +
737 +Aggregator fanout:
738 +
739 +```text
740 +node-a: topology:snmp
741 +node-b: topology:snmp
742 +```
743 +
744 +No mode parameter is sent unless the producer advertises one. Aggregator applies
745 +replacement rules and returns a normal graph.
746 +
747 +### Streaming Global Aggregated
748 +
749 +Input request to Cloud:
750 +
751 +```text
752 +function=topology:streaming __topology_mode=aggregated
753 +```
754 +
755 +Aggregator fanout:
756 +
757 +```text
758 +node-a: topology:streaming
759 +node-b: topology:streaming
760 +```
761 +
762 +No mode parameter is sent unless the producer advertises one. Aggregator merges
763 +actors by `machine_guid`, preserves retention rows by retaining source, and
764 +deduplicates stream path rows by declared path identity.
765 +
766 +## Open Edge Cases And Required Behavior
767 +
768 +- **Ambiguous network socket match**: keep unresolved or materialize partial;
769 + record diagnostic; do not randomly choose a process.
770 +- **Socket closed between node collections**: exact process match may fail;
771 + partial IP/node match may still be valid if a node/endpoint claim exists.
772 +- **NAT or load balancer aliases**: add alias keys; do not overwrite original
773 + tuples.
774 +- **SNMP duplicate weak actors**: replace all weak actors that match one strong
775 + actor; preserve evidence and diagnostics.
776 +- **SNMP weak actor matches multiple strong actors**: keep weak actor visible or
777 + diagnostic; do not choose randomly.
778 +- **Streaming stale node**: keep actor if it is meaningful to streaming status;
779 + merge by `machine_guid` only when identities match.
780 +- **Streaming retention from multiple parents**: preserve rows; do not collapse
781 + them into one retained child row.
782 +- **Actor label conflicts**: keep full label set; modal identification applies
783 + display limits only, not data loss.
784 +- **High-cardinality detailed network-connections**: detailed mode may be large;
785 + aggregated mode must avoid actor-per-port explosion.
786 +
787 +## Validation Requirements
788 +
789 +Agent:
790 +
791 +- Schema validation for all changed topology payloads.
792 +- Network-connections fixtures for aggregated and detailed modes.
793 +- SNMP fixture proving no mode selector is exposed unless behavior differs.
794 +- Streaming fixture proving no mode selector is exposed unless behavior differs.
795 +- Modal label identification metadata present for actor types with useful
796 + labels.
797 +
798 +UI:
799 +
800 +- Decode modal label identification metadata.
801 +- Render selected identification labels in actor modal header.
802 +- Keep full Labels tab.
803 +- Render loose-side materialized actors only from schema policy.
804 +- Do not show SNMP/streaming mode toggles unless payload capability declares
805 + them.
806 +
807 +Aggregator:
808 +
809 +- Rewrite `__topology_mode=aggregated` to `detailed` on fanout only when the
810 + producer supports it.
811 +- Consume detailed network-connections and return both detailed and aggregated
812 + outputs.
813 +- Resolve exact socket loose sides and preserve unresolved/partial cases.
814 +- Replace SNMP weak actors with managed actors.
815 +- Merge/enrich streaming actors and tables by `machine_guid` and table policy.
816 +- Preserve schema-valid unknown future fields.
AGENTS.md
+28 -1
@@ -280,6 +280,24 @@ Output/reference skills may also exist under product documentation or generated
280
281 End-user-facing AI skills under `docs/netdata-ai/skills/` follow the directory shape `docs/netdata-ai/skills/<skill-name>/SKILL.md`, with optional supporting docs (`<topic>.md`) and an optional `scripts/` subdirectory for helper code. SKILL.md frontmatter has `name` and `description`; the description is the trigger-matching text and must enumerate the phrases users will actually type.
282
283 +Public skills are for operators and end-users. They may teach users how to
284 +query Netdata Cloud, query Agents, inspect metrics/logs/topology/alerts, or run
285 +safe operational commands. They must not contain developer-contract validation,
286 +schema migration plans, producer authoring workflows, UI adapter work,
287 +aggregator implementation notes, SOW handoff instructions, fixture maintenance,
288 +PR-review tasks, or codebase-internal implementation recipes.
289 +
290 +Developer-facing skills must live under `.agents/skills/`, preferably with a
291 +`project-` prefix when they are runtime input for repository work. If a workflow
292 +requires reading source files, updating schemas, validating fixtures, changing
293 +collectors/producers, or coordinating frontend/backend/aggregator code, it is a
294 +project developer skill, not a public skill.
295 +
296 +Skill verification harness inputs are not public skill content. Keep seed
297 +questions, grader rubrics, runner scripts, and transcript-generation prompts
298 +under `.agents/skill-verification/<skill>/`, not under
299 +`docs/netdata-ai/skills/<skill>/`.
300 +
301 Each public skill is reachable from `.agents/skills/<skill-name>` via a relative symlink (`.agents/skills/<name>` → `../../docs/netdata-ai/skills/<name>`) so local AI assistants reading from `.agents/skills/` see the same skill as end-users. Create the symlink with `ln -srfn`. Verify with `readlink -f .agents/skills/<name>`.
302
303 Public-skill scripts must follow the same `_lib.sh` shape as existing skills (`set -euo pipefail`, ANSI colors with real ESC bytes via `$'\033[...]'`, `<prefix>_repo_root` via `git rev-parse --show-toplevel`, `<prefix>_load_env` that sources `<repo>/.env` with `: "${VAR:?}"` validation, `<prefix>_audit_dir` that creates `<repo>/.local/audits/<topic>/`, masked-token `<prefix>_run`/`<prefix>_run_read` wrappers).
@@ -288,7 +306,12 @@ Public-skill scripts that touch credentials (cloud tokens, per-agent bearers, cl
306
307 ### How-tos catalog rule
308
291 -Each public skill ships a `how-tos/` subdirectory with `INDEX.md`. The catalog is **live**: every time an AI assistant is asked a concrete question that requires analysis (multiple wrapper calls, jq pipelines, or cross-referencing more than one per-domain guide) and the answer isn't already documented under `how-tos/`, the assistant MUST author a new how-to and add it to `INDEX.md` BEFORE completing the task. This rule is repeated in each skill's `SKILL.md` so future assistants honor it. Skipping it means the next assistant repeats the same analysis from scratch -- an explicit framework violation.
309 +Each public skill ships a `how-tos/` subdirectory with `INDEX.md`. The catalog is **live**: every time an AI assistant is asked a concrete operator/end-user question that requires analysis (multiple wrapper calls, jq pipelines, or cross-referencing more than one per-domain guide) and the answer isn't already documented under `how-tos/`, the assistant MUST author a new how-to and add it to `INDEX.md` BEFORE completing the task. This rule is repeated in each skill's `SKILL.md` so future assistants honor it. Skipping it means the next assistant repeats the same analysis from scratch -- an explicit framework violation.
310 +
311 +The how-to rule does not override audience boundaries. If the analysis produced
312 +a developer validation recipe, put it in the matching `.agents/skills/` project
313 +skill and update that skill's index instead of adding it under
314 +`docs/netdata-ai/skills/`.
315
316 The existing private skills (`coverity-audit`, `sonarqube-audit`, `graphql-audit`, `pr-reviews`) keep their `.agents/skills/<name>/` location -- they are intentionally private and have no `docs/netdata-ai/skills/` counterpart.
317
@@ -304,6 +327,10 @@ Runtime input skills:
327 Trigger: authoring or modifying any Netdata data-collection plugin or module (Go go.d / ibm.d, Rust crates, internal C plugins, external plugins via PLUGINSD). Read before adding a new collector, modifying an existing one, working on NetFlow/sFlow/IPFIX, OTEL ingestion, topology, SNMP profiles, or interactive Functions.
328 Status: live. Updates that close gaps or fix outdated pointers must ship in the same PR that exposed the issue.
329
330 +- `.agents/skills/project-create-topology/`
331 + Trigger: creating or updating Netdata topology producers, topology Function payloads, topology schema fixtures, graph presentation, correlation rules, direction semantics, topology drilldowns, telemetry overlays, or Cloud topology aggregation fixtures.
332 + Status: live. Developer-facing topology authoring workflow. End-user/operator-facing AI skills belong under `docs/netdata-ai/skills/`; this project skill is the runtime guidance for repository work.
333 +
334 - `.agents/skills/project-writing-go-modules-framework-v2/`
335 Trigger: creating or migrating a Go go.d collector to framework V2; touching `CollectorV2`, `metrix.CollectorStore`, `ChartTemplateYAML` / `charts.yaml`, `charttpl`, `chartengine`, V2 host scopes, or V2 collector tests.
336 Purpose: mirror maintainer-preferred framework V2 patterns from accepted collectors so new or migrated modules blend with repository style.
docs/netdata-ai/skills/query-netdata-agents/SKILL.md
+5 -2
@@ -27,7 +27,6 @@ agent API.
27 | Node identity, hardware, vnodes | [query-nodes.md](./query-nodes.md) |
28 | Streaming (parent / child / replication) -- agent-only | [query-streaming.md](./query-streaming.md) |
29 | **Operational how-tos (live catalog)** | [how-tos/INDEX.md](./how-tos/INDEX.md) |
30 -| **Verification questions (consumed by SOW-0006 harness)** | [verify/questions.md](./verify/questions.md) |
30
31
32 | Transport | Auth | When to use |
@@ -55,7 +54,11 @@ implementation.
54 a new how-to and add it to
55 [`how-tos/INDEX.md`](./how-tos/INDEX.md) BEFORE completing the
56 task. The catalog is **live** -- the next assistant should not
58 - redo the same analysis.
57 + redo the same analysis. Keep this catalog operator-facing:
58 + recipes here should explain how to fetch or use Agent data.
59 + Developer contract validation for topology producers, schemas,
60 + fixtures, UI adapters, or aggregator handoffs belongs in the
61 + relevant project developer skill, not in this public skill.
62 2. **Use the token-safe wrappers.** `agents_query_cloud`,
63 `agents_query_agent`, `agents_call_function` from
64 [`scripts/_lib.sh`](./scripts/_lib.sh) handle auth internally
docs/netdata-ai/skills/query-netdata-agents/how-tos/INDEX.md
+3 -4
@@ -35,10 +35,9 @@ defeats the no-token-leak guarantee.
35
36 ## Index
37
38 -(Populate as how-tos are authored. Stubs below correspond to the
39 -seed verification questions in `../verify/questions.md`; replace
40 -each `(stub -- not yet authored)` with a real link as soon as a
41 -how-to is written.)
38 +(Populate as how-tos are authored. Stubs below mirror the canonical
39 +skill-verification harness questions for `verify/questions.md`; replace each
40 +`(stub -- not yet authored)` with a real link as soon as a how-to is written.)
41
42 ### Identity / hardware / OS
43
docs/netdata-ai/skills/query-netdata-agents/query-streaming.md
-1
@@ -105,4 +105,3 @@ for the envelope definition and the canonical
105 -- canonical Function reference + envelope.
106 - [query-nodes.md](./query-nodes.md) -- node identity, parent /
107 child labels.
108 -- `<repo>/src/streaming/` -- agent-side streaming implementation.
docs/netdata-ai/skills/query-netdata-agents/query-topology.md
+35 -37
@@ -3,73 +3,71 @@
3 This guide is part of the [`query-netdata-agents`](./SKILL.md) skill.
4 Read [SKILL.md](./SKILL.md#prerequisites) first.
5
6 -For the body parameters (`nodes_identity`, `map_type`,
7 -`inference_strategy`, `managed_snmp_device_focus`, `depth`),
8 -the response envelope (top-level `data.actors[]` + `data.links[]`),
9 -and per-actor / per-link field semantics, see
6 +The topology body and response payload are the same as the Cloud-proxied
7 +transport. For the production topology schema, response fields, compact table
8 +format, and interpretation rules, see
9 [../query-netdata-cloud/query-topology.md](../query-netdata-cloud/query-topology.md).
11 -The body and response are identical between Cloud-proxied and
12 -direct-agent calls.
10
14 -Today only `topology:snmp` is registered. Future topology Functions
15 -will follow the same `topology:<source>` namespace and the same
16 -envelope.
11 +## Endpoint
12
18 ----
13 +`POST /api/v3/function?function=topology:<source>`
14
20 -## Endpoint (agent v3)
21 -
22 -`POST /api/v3/function?function=topology:snmp`
23 -
24 -## Use the wrapper
15 +Example:
16
17 ```bash
18 source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
19 agents_load_env
20 +AGENT_URL="${AGENT_URL:-http://${AGENT_HOST:-127.0.0.1}:${AGENT_PORT:-19999}}"
21 +AGENT_TARGET="${AGENT_URL#http://}"
22 +AGENT_TARGET="${AGENT_TARGET#https://}"
23 +AGENT_TARGET="${AGENT_TARGET%%/*}"
24
25 read -r -d '' BODY <<'JSON'
26 {
27 "selections": {
33 - "nodes_identity": ["mac"],
34 - "map_type": ["lldp_cdp_managed"],
35 - "inference_strategy": ["fdb_minimum_knowledge"],
36 - "managed_snmp_device_focus": ["all_devices"],
37 - "depth": ["all"]
28 + "mode": ["aggregated"]
29 },
30 "timeout": 60000
31 }
32 JSON
33
34 agents_query_agent \
44 - --node "$NODE_UUID" \
45 - --host "$AGENT_HOST:19999" \
35 + --node "$NODE_UUID" \
36 + --host "$AGENT_TARGET" \
37 --machine-guid "$AGENT_MG" \
47 - POST '/api/v3/function?function=topology:snmp' "$BODY" \
48 - | jq '.data | {actors: (.actors|length), links: (.links|length), view, layer}'
38 + POST '/api/v3/function?function=topology:network-connections' "$BODY" \
39 + | jq '.data | {
40 + schema: .schema_version,
41 + actors: .actors.rows,
42 + links: .links.rows,
43 + evidence_rows: ([.evidence[]?.table.rows] | add // 0)
44 + }'
45 ```
46
47 ## Discover supported parameters
48
49 ```bash
54 -agents_query_agent --node "$NODE_UUID" --host "$AGENT_HOST:19999" --machine-guid "$AGENT_MG" \
55 - POST '/api/v3/function?function=topology:snmp' '{"info":true}' \
50 +agents_query_agent \
51 + --node "$NODE_UUID" \
52 + --host "$AGENT_TARGET" \
53 + --machine-guid "$AGENT_MG" \
54 + POST '/api/v3/function?function=topology:network-connections' \
55 + '{"info":true,"timeout":30000}' \
56 | jq '.required_params'
57 ```
58
59 -## Limits and gotchas
59 +## Notes
60
61 -- **Slow queries.** A full SNMP sweep on a busy network can take
62 - 60+ seconds. Set `timeout` accordingly.
63 -- **MAC-list `actor_id` values can be very long.** Use
64 - `nodes_identity:["ip"]` to collapse devices by IP if you prefer
65 - shorter ids.
66 -- **Single-agent perspective.** The topology graph is what THIS
67 - agent has discovered. Multi-agent fleets need merge logic on
68 - the client side.
61 +- The graph is the perspective of the queried Agent or producer instance.
62 +- Fleet-wide views require Cloud aggregation over multiple topology payloads.
63 +- High-cardinality relationship facts live in evidence sections, not graph
64 + links.
65 +- Topology Functions should fail explicitly on size limits; they must not
66 + silently truncate evidence.
67
68 ## See also
69
70 - [../query-netdata-cloud/query-topology.md](../query-netdata-cloud/query-topology.md)
73 - -- full reference, parameter values, body / response detail.
74 -- [query-functions.md](./query-functions.md) -- generic Function
71 + -- full response reference.
72 +- [query-functions.md](./query-functions.md) -- generic direct-agent Function
73 transport.
docs/netdata-ai/skills/query-netdata-cloud/SKILL.md
+9 -7
@@ -27,22 +27,19 @@ runnable curl commands.
27 | Members (per-space user enumeration) | [query-members.md](./query-members.md) |
28 | Event feed (audit + activity log) | [query-feed.md](./query-feed.md) |
29 | **Operational how-tos (live catalog)** | [how-tos/INDEX.md](./how-tos/INDEX.md) |
30 -| **Verification questions (consumed by SOW-0006 harness)** | [verify/questions.md](./verify/questions.md) |
30
31 ### Canonical reference docs (in this repo)
32
34 -For the protocol-level details these guides build on, read the
35 -authoritative sources directly:
33 +For the query protocol details these guides build on, read the authoritative
34 +sources directly:
35
36 | File | What it covers |
37 |---|---|
38 | `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` | Functions v3 protocol -- envelope, simple-table vs log-explorer, facets, histograms, charts, field types, pagination, delta mode, PLAY mode, error handling. The single most important reference for any Function work. |
40 -| `<repo>/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` | Practical guide for collector authors implementing a Function (simple-table or log-explorer) |
39 | `<repo>/src/plugins.d/FUNCTION_UI_SCHEMA.json` | JSON Schema for validating Function responses |
40 +| `<repo>/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` | JSON Schema for validating production topology payloads |
41 | `<repo>/src/plugins.d/DYNCFG.md` | External-plugin DynCfg protocol (go.d.plugin and other external collectors) |
42 | `<repo>/src/daemon/dyncfg/README.md` | Internal DynCfg (high-level and low-level APIs, command enums, lifecycle) |
44 -| `<repo>/src/database/rrdfunctions.h` | C-level Function registration API (`rrd_function_add`) |
45 -| `<repo>/src/go/plugin/framework/functions/README.md` | Go-plugin Function framework |
43
44 For querying agents directly (without going through Cloud) -- including
45 auto-minting agent bearer tokens from a Cloud token -- see the sibling
@@ -58,7 +55,12 @@ skill [`query-netdata-agents`](../query-netdata-agents/SKILL.md).
55 author a new how-to in this directory and add it to
56 [`how-tos/INDEX.md`](./how-tos/INDEX.md) BEFORE completing the
57 task. The catalog is meant to be **live** -- the next assistant
61 - should not redo the same analysis from scratch.
58 + should not redo the same analysis from scratch. Keep this
59 + catalog operator-facing: recipes here should explain how to fetch
60 + or use Cloud data. Developer contract validation for collectors,
61 + topology producers, schemas, fixtures, UI adapters, or aggregator
62 + handoffs belongs in the relevant project developer skill, not in
63 + this public skill.
64 2. **Use the token-safe wrappers.** Every example in this skill
65 uses `agents_query_cloud` (and friends) from
66 `../query-netdata-agents/scripts/_lib.sh`. Never paste raw
docs/netdata-ai/skills/query-netdata-cloud/how-tos/INDEX.md
+3 -4
@@ -46,10 +46,9 @@ Bearer $TOKEN"` -- that defeats the no-token-leak guarantee.
46
47 ## Index
48
49 -(Populate as how-tos are authored. Stubs below correspond to the
50 -seed verification questions in `../verify/questions.md`; replace
51 -each `(stub -- not yet authored)` with a real link as soon as a
52 -how-to is written.)
49 +(Populate as how-tos are authored. Stubs below mirror the canonical
50 +skill-verification harness questions for `verify/questions.md`; replace each
51 +`(stub -- not yet authored)` with a real link as soon as a how-to is written.)
52
53 ### Identity / hardware / OS
54
docs/netdata-ai/skills/query-netdata-cloud/query-functions.md
+2 -67
@@ -6,8 +6,7 @@ Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
6 This file documents the **generic** Function transport: the URL,
7 the standard response envelope, the `info` discovery query, the
8 four Function families and where each family's data lives in the
9 -response, plus pointers to the developer documentation for
10 -collector authors.
9 +response.
10
11 For three of the four families there is a dedicated guide:
12
@@ -60,7 +59,7 @@ that build on the same envelope but emit non-tabular `data`:
59
60 | `type` | Response shape | Examples | Guide |
61 |---|---|---|---|
63 -| `topology` | `data.actors[]` + `data.links[]` (a graph) | `topology:snmp` | [query-topology.md](./query-topology.md) |
62 +| `topology` | `data.actors`/`data.links` graph plus compact-schema sections (`data.evidence`, `data.tables`, `data.overlays`) | `topology:network-connections`, `topology:streaming`, `topology:snmp` | [query-topology.md](./query-topology.md) |
63 | `flows` | `data.flows[]` plus `data.facets` / `data.columns` / `data.stats` over a time window | `flows:netflow` (covers NetFlow / sFlow / IPFIX) | [query-flows.md](./query-flows.md) |
64
65 For full protocol semantics (facet pills, histograms, charts
@@ -327,70 +326,6 @@ curl -sS -X POST \
326
327 ---
328
330 -## Developer reference (for collector authors)
331 -
332 -If you maintain a collector and want to register a Function (or are
333 -debugging why a Function returns `400 ErrInfoMissing`), read these
334 -files in this order. They are the authoritative sources.
335 -
336 -| File | Audience | Read for |
337 -|---|---|---|
338 -| `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` | All implementers | Functions v3 protocol -- envelope, simple-table vs log-explorer, facets, histograms, charts, field types, anchor/delta/PLAY pagination, error handling, edge cases. **The single most important reference.** |
339 -| `<repo>/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` | Collector authors | Practical step-by-step: how to ship a simple-table or log-explorer Function, with backend examples |
340 -| `<repo>/src/plugins.d/FUNCTION_UI_SCHEMA.json` | Validation | JSON Schema for Function responses; use it in unit tests |
341 -| `<repo>/src/plugins.d/README.md` (sections 470-637) | External plugins (any language) | Plugin protocol, `FUNCTION` / `FUNCTION_PAYLOAD` parsing, response framing, ACL, lifecycle |
342 -| `<repo>/src/plugins.d/DYNCFG.md` | External plugins exposing config | DynCfg protocol for go.d.plugin and other external collectors |
343 -| `<repo>/src/daemon/dyncfg/README.md` | Internal plugins exposing config | Internal DynCfg API |
344 -| `<repo>/src/database/rrdfunctions.h` | C collectors | C API: `rrd_function_add(host, st, name, timeout, priority, version, help, tags, access, sync, execute_cb, data)`; handler signature `rrd_function_execute_cb_t` |
345 -| `<repo>/src/go/plugin/framework/functions/README.md` | Go.d.plugin collectors | Go function manager: `manager.Register`, handler lifecycle, cancellation, worker pool |
346 -| `<repo>/docs/functions/` | Operators | Per-Function user docs and examples |
347 -
348 -Skeleton signatures (extracted from the headers above, for
349 -orientation only -- read the source for the real contract):
350 -
351 -```c
352 -/* C: register at boot, then emit responses via a buffer in the callback */
353 -void rrd_function_add(
354 - RRDHOST *host, RRDSET *st,
355 - const char *name, /* "module:method" */
356 - int timeout, int priority, uint32_t version,
357 - const char *help, const char *tags,
358 - HTTP_ACCESS access, bool sync,
359 - rrd_function_execute_cb_t execute_cb, void *execute_cb_data);
360 -
361 -typedef int (*rrd_function_execute_cb_t)(
362 - struct rrd_function_execute *rfe, void *data);
363 -```
364 -
365 -```go
366 -// Go: register a Function with the framework's manager
367 -manager.Register(functions.Function{
368 - Name: "module:method",
369 - Description: "Help text",
370 - Timeout: 30 * time.Second,
371 - Params: []ParamDescriptor{ /* maps to required_params */ },
372 - Handler: func(fn Function) { /* emit FuncResponse */ },
373 -})
374 -```
375 -
376 -```text
377 -# External plugins via the plugins.d protocol:
378 -FUNCTION [GLOBAL] "name params" timeout "help" "tags" "access" priority version
379 -# On call:
380 -FUNCTION <txn_id> <timeout> "name params" "<access>" "<source>"
381 -# Reply:
382 -FUNCTION_RESULT_BEGIN <txn_id> <http_code> <content_type> <expiry>
383 -<JSON envelope: status, v, type, help, accepted_params, required_params, has_history, update_every, data, ...>
384 -FUNCTION_RESULT_END
385 -```
386 -
387 -The full envelope and required-params widget schemas above ARE the
388 -contract a collector implementation must satisfy. Read
389 -`src/plugins.d/README.md` for the line-protocol details and
390 -`src/database/rrdfunctions.h` for the C API.
391 -
392 ----
393 -
329 ## Limits and gotchas
330
331 - **Cloud default timeout is 120 s** for Function calls; pass
docs/netdata-ai/skills/query-netdata-cloud/query-topology.md
+105 -199
@@ -2,156 +2,53 @@
2
3 This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill.
4 Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first.
5 -For the generic Function transport (used by topology, logs, flows,
6 -and table-snapshot Functions alike), see
7 -[query-functions.md](./query-functions.md).
5 +For the generic Function transport, see [query-functions.md](./query-functions.md).
6
9 -Topology Functions return a graph: a list of **actors** (nodes in
10 -the graph) and **links** (edges). They differ from log Functions
11 -(which return a time-windowed skim of a larger dataset) and from
12 -table-snapshot Functions (which return one full table).
7 +Topology Functions return compact graph payloads using the production topology
8 +schema:
9
14 ----
10 +- [FUNCTION_TOPOLOGY_SCHEMA.json](../../../../src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json)
11
16 -## Function names registered today
12 +The response contains actors, graph links, relationship evidence, optional
13 +actor detail tables, and optional telemetry overlay refs. Large sections use
14 +compact columnar tables.
15
18 -Verified live and in source:
16 +## Function namespace
17
20 -| Function | Source collector | Layer | What it discovers |
21 -|---|---|---|---|
22 -| `topology:snmp` | `src/go/plugin/go.d/collector/snmp_topology/` | L2 | LLDP/CDP-discovered switches+routers, FDB-derived endpoint locations, STP-derived parent-child relationships |
18 +Topology Functions use the `topology:<source>` namespace.
19
24 -The `topology:` prefix is the canonical namespace; only `snmp` is
25 -registered today (as of `STATUS_FILE_VERSION = 28`). When new
26 -topology collectors land (network-viewer connections, streaming
27 -parent-child, k8s service mesh, etc.), they will follow the same
28 -`topology:<source>` naming pattern and the same response envelope
29 -documented below. Always confirm via the function-listing endpoint
30 -in [query-functions.md](./query-functions.md) before assuming a
31 -given topology Function exists on a node.
20 +Known producer families:
21
33 ----
22 +| Function | Source | Typical topology |
23 +|---|---|---|
24 +| `topology:network-connections` | Network Viewer plugin | process, endpoint, socket evidence |
25 +| `topology:streaming` | Netdata streaming subsystem | parent/child streaming graph |
26 +| `topology:snmp` | SNMP topology collector | L2 devices, interfaces, endpoints, adjacencies |
27 +| `topology:vsphere` | vSphere collector (planned) | inventory and virtualization relationships |
28
35 -## Endpoint and request
29 +Always start with an info request to discover the parameters supported by the
30 +Agent version you are querying.
31
37 -Use the standard Cloud Function-call endpoint. Topology is just a
38 -Function; nothing is special about its URL.
32 +## Endpoint
33
40 -`POST /api/v2/nodes/{nodeId}/function?function=topology:snmp`
34 +Use the standard Cloud Function endpoint:
35 +
36 +`POST /api/v2/nodes/{nodeId}/function?function=topology:<source>`
37 +
38 +Example info request:
39
40 ```bash
41 TOKEN="YOUR_API_TOKEN"
42 NODE="YOUR_NODE_UUID"
43
46 -read -r -d '' PAYLOAD <<'EOF'
47 -{
48 - "selections": {
49 - "nodes_identity": ["mac"],
50 - "map_type": ["lldp_cdp_managed"],
51 - "inference_strategy": ["fdb_minimum_knowledge"],
52 - "managed_snmp_device_focus": ["all_devices"],
53 - "depth": ["all"]
54 - },
55 - "timeout": 60000,
56 - "last": 200
57 -}
58 -EOF
59 -
44 curl -sS -X POST \
45 -H 'Content-Type: application/json' \
46 -H "Authorization: Bearer $TOKEN" \
63 - "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:snmp" \
64 - -d "$PAYLOAD"
47 + "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:network-connections" \
48 + -d '{"info":true,"timeout":30000}'
49 ```
50
67 -Always start with `{"info":true}` to discover the parameters the
68 -node currently accepts -- topology Function parameter sets evolve
69 -with the collector.
70 -
71 -### Body parameters (topology:snmp)
72 -
73 -Verified against `src/go/plugin/go.d/collector/snmp_topology/`:
74 -
75 -| Parameter | Type | Allowed values | Purpose |
76 -|---|---|---|---|
77 -| `nodes_identity` | string | `ip`, `mac` | Collapse / distinguish actors by IP or MAC |
78 -| `map_type` | string | `lldp_cdp_managed`, `high_confidence_inferred`, `all_devices_low_confidence` | Which discovery sources to include |
79 -| `inference_strategy` | string | `fdb_minimum_knowledge`, `stp_parent_tree`, `fdb_pairwise_minimum_knowledge`, `stp_fdb_correlated`, `cdp_fdb_hybrid` | How endpoint-to-switch placement is inferred when LLDP/CDP coverage is incomplete |
80 -| `managed_snmp_device_focus` | string | `all_devices`, `ip:<prefix>` | Restrict the discovery surface to a subset of managed SNMP devices |
81 -| `depth` | string | `0`-`10`, or `all` | Hops away from the focus device to include |
82 -
83 -`selections` is the standard Function selection object; values are
84 -arrays even for single-valued parameters (Netdata convention).
85 -
86 ----
87 -
88 -## Response envelope
89 -
90 -Topology Functions wrap their content in the standard Function
91 -envelope. Top-level keys (verified live):
92 -
93 -| Key | Description |
94 -|---|---|
95 -| `status` | HTTP-style status integer (200 on success) |
96 -| `v` | Function schema version |
97 -| `type` | **`topology`** -- the family discriminator |
98 -| `help` | Human description |
99 -| `accepted_params` | Parameter names the Function accepts |
100 -| `required_params` | Per-parameter UI widgets |
101 -| `has_history` | Whether the Function supports `after`/`before` history |
102 -| `update_every` | Suggested refresh interval (seconds) |
103 -| `data` | The graph payload (object) |
104 -
105 -### `data` object
106 -
107 -| Key | Description |
108 -|---|---|
109 -| `schema_version` | Topology schema version (e.g. `2.0`) |
110 -| `source` | Discovery source string (e.g. `snmp`) |
111 -| `layer` | OSI layer the topology lives at (e.g. `2`, `3`) |
112 -| `agent_id` | Identifier of the producing agent |
113 -| `collected_at` | RFC3339 timestamp |
114 -| `view` | Rendered view kind (e.g. `summary`, `detail`) |
115 -| `actors[]` | Graph nodes -- see schema below |
116 -| `links[]` | Graph edges -- see schema below |
117 -| `flows[]` | Optional, for sources that emit flow records alongside the topology |
118 -| `stats` | Per-source counters (devices polled, fdb entries, etc.) |
119 -| `metrics` | Optional metric block |
120 -| `ip_policy` | Optional, when `nodes_identity:ip` is in effect |
121 -
122 -### Actor record
123 -
124 -| Key | Description |
125 -|---|---|
126 -| `actor_id` | Stable identifier of the actor; format is source-specific (e.g. `mac:<addr>[,<addr>...]` for SNMP at L2; `ip:<addr>` when collapsed by IP) |
127 -| `actor_type` | e.g. `device`, `endpoint`, `vlan`, `service` |
128 -| `layer` | OSI layer (`2`, `3`, ...) |
129 -| `source` | Source collector (e.g. `snmp`) |
130 -| `match` | Discovery facts (sysName, sysObjectID, OUI, ...) |
131 -| `attributes` | Free-form per-actor properties |
132 -| `derived` | Computed annotations (vendor inference, role, ...) |
133 -| `labels` | Tag-style key/value pairs |
134 -| `tables` | Per-actor sub-tables (e.g. interfaces, ARP entries) |
135 -
136 -### Link record
137 -
138 -| Key | Description |
139 -|---|---|
140 -| `layer` | Link layer |
141 -| `protocol` | Discovery protocol (`lldp`, `cdp`, `fdb`, `stp`, ...) |
142 -| `link_type` | Refinement (e.g. `lldp`, `inferred-fdb`, `stp-parent`) |
143 -| `direction` | `bidirectional`, `forward`, `reverse` |
144 -| `state` | Operational state (`up`, `down`, ...) |
145 -| `src_actor_id` / `dst_actor_id` | The two endpoints (match an entry in `actors[]`) |
146 -| `src` / `dst` | Per-end interface / port details |
147 -| `discovered_at` / `last_seen` | RFC3339 timestamps |
148 -| `metrics` | Optional per-link metrics |
149 -
150 ----
151 -
152 -## Examples
153 -
154 -### Example 1: full LLDP/CDP topology
51 +Example data request:
52
53 ```bash
54 TOKEN="YOUR_API_TOKEN"
@@ -160,11 +57,7 @@ NODE="YOUR_NODE_UUID"
57 read -r -d '' PAYLOAD <<'EOF'
58 {
59 "selections": {
163 - "nodes_identity": ["mac"],
164 - "map_type": ["lldp_cdp_managed"],
165 - "inference_strategy": ["fdb_minimum_knowledge"],
166 - "managed_snmp_device_focus": ["all_devices"],
167 - "depth": ["all"]
60 + "mode": ["aggregated"]
61 },
62 "timeout": 60000
63 }
@@ -173,82 +66,95 @@ EOF
66 curl -sS -X POST \
67 -H 'Content-Type: application/json' \
68 -H "Authorization: Bearer $TOKEN" \
176 - "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:snmp" \
177 - -d "$PAYLOAD" \
178 - | jq '.data | {actors: (.actors|length), links: (.links|length)}'
69 + "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:network-connections" \
70 + -d "$PAYLOAD"
71 ```
72
181 -### Example 2: include FDB-derived low-confidence endpoints
73 +## Response shape
74
183 -```bash
184 -read -r -d '' PAYLOAD <<'EOF'
75 +Top-level response:
76 +
77 +```json
78 {
186 - "selections": {
187 - "nodes_identity": ["mac"],
188 - "map_type": ["all_devices_low_confidence"],
189 - "inference_strategy": ["fdb_pairwise_minimum_knowledge"],
190 - "managed_snmp_device_focus": ["all_devices"],
191 - "depth": ["all"]
192 - },
193 - "timeout": 120000
79 + "status": 200,
80 + "type": "topology",
81 + "has_history": false,
82 + "data": {
83 + "schema_version": "netdata.topology.v1",
84 + "producer": {},
85 + "collected_at": "2026-05-09T10:00:00Z",
86 + "dictionaries": {},
87 + "types": {},
88 + "actors": {},
89 + "links": {},
90 + "evidence": {},
91 + "tables": {},
92 + "overlays": {},
93 + "stats": {}
94 + }
95 }
195 -EOF
96 ```
97
198 -### Example 3: focus on a single device + 2 hops
98 +The important fields:
99 +
100 +| Field | Description |
101 +|---|---|
102 +| `data.schema_version` | Topology contract version, currently `netdata.topology.v1` |
103 +| `data.producer` | Producer source, instance, node, plugin, and version metadata |
104 +| `data.dictionaries` | Shared dictionaries, especially `strings` |
105 +| `data.types.actor_types` | Actor identity and aggregation-scope metadata |
106 +| `data.types.link_types` | Link direction and aggregation policy |
107 +| `data.types.evidence_types` | Evidence role and exact match columns |
108 +| `data.actors` | Compact table of graph actors |
109 +| `data.links` | Compact table of renderable graph links |
110 +| `data.evidence` | Compact relationship evidence sections |
111 +| `data.tables` | Optional actor or relationship detail tables |
112 +| `data.overlays` | Optional metric/function overlay refs |
113 +| `data.stats` | Producer and payload counters |
114 +
115 +## Decode compact tables
116 +
117 +Every table has:
118 +
119 +- `rows`: number of rows;
120 +- `columns`: column definitions;
121 +- `values`: parallel array of column encodings.
122 +
123 +Supported codecs:
124 +
125 +| Codec | Meaning |
126 +|---|---|
127 +| `const` | one value repeated for all rows |
128 +| `values` | one value per row |
129 +| `dict` | per-column dictionary plus row indexes |
130 +
131 +Minimal jq-friendly counts:
132
133 ```bash
201 -read -r -d '' PAYLOAD <<'EOF'
202 -{
203 - "selections": {
204 - "nodes_identity": ["ip"],
205 - "map_type": ["lldp_cdp_managed"],
206 - "inference_strategy": ["stp_parent_tree"],
207 - "managed_snmp_device_focus": ["ip:YOUR_FOCUS_DEVICE_IP"],
208 - "depth": ["2"]
209 - },
210 - "timeout": 60000
211 -}
212 -EOF
134 +jq '.data | {
135 + schema: .schema_version,
136 + actors: .actors.rows,
137 + links: .links.rows,
138 + evidence_rows: ([.evidence[]?.table.rows] | add // 0),
139 + stats
140 +}'
141 ```
142
215 -### Example 4: discover supported parameters before querying
143 +## Interpretation rules
144
217 -```bash
218 -TOKEN="YOUR_API_TOKEN"
219 -NODE="YOUR_NODE_UUID"
145 +- Actors are entities.
146 +- Links are graph edges.
147 +- Evidence rows are the exact facts behind links.
148 +- Actor custom tables are separate from relationship evidence.
149 +- Direction semantics come from `data.types.link_types`.
150 +- Telemetry overlays come from `data.types.overlay_templates` plus
151 + `data.overlays.refs`.
152
221 -read -r -d '' PAYLOAD <<'EOF'
222 -{ "info": true }
223 -EOF
153 +Do not assume every evidence row is rendered as a graph edge. A single graph
154 +link may summarize many evidence rows.
155
225 -curl -sS -X POST \
226 - -H 'Content-Type: application/json' \
227 - -H "Authorization: Bearer $TOKEN" \
228 - "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=topology:snmp" \
229 - -d "$PAYLOAD" \
230 - | jq '{accepted_params, required_params}'
231 -```
156 +## See also
157
233 ----
234 -
235 -## Limits and gotchas
236 -
237 -- **Topology Functions are slow.** A full SNMP sweep can take tens
238 - of seconds. Bump `timeout` to 60-120 seconds.
239 -- **`actor_id` for L2 SNMP topology can be a comma-separated list
240 - of MACs** -- when a single device exposes many MACs (one per
241 - port), they collapse into one actor with all MACs in
242 - `actor_id`. Use `nodes_identity:ip` to collapse by IP instead.
243 -- **Privacy**: actor and link records carry MAC addresses, IP
244 - addresses, sysName strings, and SNMP descriptions. Treat as
245 - network-identifying data; do not paste raw responses into
246 - committed files. Use `<repo>/.local/audits/...` for working
247 - output (gitignored).
248 -- **`info=true` is cheap and always available.** Use it to confirm
249 - the parameter set on a specific node before constructing a real
250 - query.
251 -- **Topology Functions are agent-only sources.** Cloud only
252 - proxies; there is no Cloud-side aggregation across nodes for
253 - topology. For multi-agent topology composition, fetch each
254 - agent's response and merge client-side.
158 +- [query-functions.md](./query-functions.md) -- generic Function transport.
159 +- [../query-netdata-agents/query-topology.md](../query-netdata-agents/query-topology.md)
160 + -- direct-agent transport for the same topology payload.
src/collectors/network-viewer.plugin/integrations/network_connections.md
+1 -1
@@ -95,7 +95,7 @@ This plugin exposes a real-time function for viewing active network connections.
95 Shows active network connections with protocol details, states, addresses, ports, and performance metrics.
96
97 Provides both aggregated and detailed views of TCP and UDP connections for IPv4 and IPv6,
98 -including connection direction (listen, inbound, outbound, local), process information,
98 +including connection direction (listen, inbound, outbound), process information,
99 and TCP performance metrics (RTT, retransmissions).
100
101 Connections are classified as system or container based on network namespace.
src/collectors/network-viewer.plugin/metadata.yaml
+1 -1
@@ -82,7 +82,7 @@ modules:
82 Shows active network connections with protocol details, states, addresses, ports, and performance metrics.
83
84 Provides both aggregated and detailed views of TCP and UDP connections for IPv4 and IPv6,
85 - including connection direction (listen, inbound, outbound, local), process information,
85 + including connection direction (listen, inbound, outbound), process information,
86 and TCP performance metrics (RTT, retransmissions).
87
88 Connections are classified as system or container based on network namespace.
src/collectors/network-viewer.plugin/network-viewer.c
+2271 -1038
@@ -32,13 +32,15 @@ static SPAWN_SERVER *spawn_srv = NULL;
32 #define NETWORK_TOPOLOGY_VIEWER_HELP "Shows live network-connections topology with self/process/endpoint actors and ownership/socket links."
33 #define NETWORK_VIEWER_RESPONSE_UPDATE_EVERY 5
34 // Keep in sync with the topology schema contract used across topology producers.
35 -#define NETWORK_TOPOLOGY_SCHEMA_VERSION "2.0"
35 +#define NETWORK_TOPOLOGY_SCHEMA_VERSION "netdata.topology.v1"
36 #define NETWORK_TOPOLOGY_SOURCE "network-connections"
37 -#define NETWORK_TOPOLOGY_LAYER "l7"
37 +#define NETWORK_TOPOLOGY_LAYER "network"
38 #define NV_TOPOLOGY_MAX_PPID_DEPTH 64
39
40 #define NV_TOPOLOGY_USERNAME_MAX 128
41 #define NV_TOPOLOGY_CMDLINE_MAX 512
42 +#define NV_TOPOLOGY_LABEL_KEY_MAX 96
43 +#define NV_TOPOLOGY_LABEL_VALUE_MAX 512
44 #define NV_TOPOLOGY_KEY_MAX 1024
45
46 typedef struct {
@@ -75,6 +77,12 @@ typedef struct {
77 char process[TASK_COMM_LEN + 1];
78 } NV_ENDPOINT_OWNER;
79
80 +typedef enum {
81 + NV_TOPOLOGY_ENDPOINT_ROLE_NONE = 0,
82 + NV_TOPOLOGY_ENDPOINT_ROLE_CLIENT,
83 + NV_TOPOLOGY_ENDPOINT_ROLE_SERVER,
84 +} NV_TOPOLOGY_ENDPOINT_ROLE;
85 +
86 typedef struct {
87 uint64_t pid;
88 uint64_t ppid;
@@ -84,32 +92,29 @@ typedef struct {
92 uint64_t retransmissions;
93 uint32_t max_rtt_usec;
94 uint32_t max_rcv_rtt_usec;
87 - uint16_t local_port;
88 - uint16_t remote_port;
89 - uint16_t peer_port;
95 + uint16_t client_port;
96 + uint16_t server_port;
97 + uint16_t process_port;
98 uint16_t protocol_id;
91 - uint8_t direction_id;
99 + bool process_is_client;
100 char process[TASK_COMM_LEN + 1];
101 char username[NV_TOPOLOGY_USERNAME_MAX];
102 char namespace_type[16];
103 char protocol[8];
104 char protocol_family[8];
97 - char direction[16];
105 char state[32];
99 - char local_ip[INET6_ADDRSTRLEN];
100 - char remote_ip[INET6_ADDRSTRLEN];
101 - char peer_ip[INET6_ADDRSTRLEN];
102 - char local_address_space[16];
103 - char remote_address_space[16];
104 - char port_name[64];
106 + char client_ip[INET6_ADDRSTRLEN];
107 + char server_ip[INET6_ADDRSTRLEN];
108 + char client_address_space[16];
109 + char server_address_space[16];
110 char cmdline[NV_TOPOLOGY_CMDLINE_MAX];
111 } NV_TOPOLOGY_LINK;
112
113 typedef struct {
114 bool info_only;
115 + bool detailed;
116 bool processes_by_pid;
117 bool sockets_listening;
112 - bool sockets_local;
118 bool sockets_inbound;
119 bool sockets_outbound;
120 bool protocols_ipv4_tcp;
@@ -148,16 +153,6 @@ typedef struct {
153 char host_actor_id[NV_TOPOLOGY_KEY_MAX];
154 } NV_TOPOLOGY_RENDER_STATE;
155
151 -typedef struct nv_process_socket_row {
152 - const NV_TOPOLOGY_LINK *link;
153 - struct nv_process_socket_row *next;
154 -} NV_PROCESS_SOCKET_ROW;
155 -
156 -typedef struct {
157 - NV_PROCESS_SOCKET_ROW *head;
158 - NV_PROCESS_SOCKET_ROW *tail;
159 -} NV_PROCESS_SOCKET_ROWS;
160 -
156 #define SIMPLE_HASHTABLE_VALUE_TYPE LOCAL_SOCKET *
157 #define SIMPLE_HASHTABLE_NAME _AGGREGATED_SOCKETS
158 #include "libnetdata/simple_hashtable/simple_hashtable.h"
@@ -176,8 +171,8 @@ static SERVICENAMES_CACHE *sc;
171
172 ENUM_STR_MAP_DEFINE(SOCKET_DIRECTION) = {
173 { .id = SOCKET_DIRECTION_LISTEN, .name = "listen" },
179 - { .id = SOCKET_DIRECTION_LOCAL_INBOUND, .name = "local" },
180 - { .id = SOCKET_DIRECTION_LOCAL_OUTBOUND, .name = "local" },
174 + { .id = SOCKET_DIRECTION_LOCAL_INBOUND, .name = "inbound" },
175 + { .id = SOCKET_DIRECTION_LOCAL_OUTBOUND, .name = "outbound" },
176 { .id = SOCKET_DIRECTION_INBOUND, .name = "inbound" },
177 { .id = SOCKET_DIRECTION_OUTBOUND, .name = "outbound" },
178
@@ -225,9 +220,9 @@ static inline void topology_options_defaults(NV_TOPOLOGY_OPTIONS *opts) {
220 return;
221
222 memset(opts, 0, sizeof(*opts));
223 + opts->detailed = false; // default: aggregated graph view
224 opts->processes_by_pid = false; // default: by_name
225 opts->sockets_listening = false;
230 - opts->sockets_local = false;
226 opts->sockets_inbound = true;
227 opts->sockets_outbound = true;
228 opts->protocols_ipv4_tcp = true;
@@ -241,7 +236,6 @@ static inline bool topology_sockets_any_enabled(const NV_TOPOLOGY_OPTIONS *opts)
236 return false;
237
238 return (opts->sockets_listening ||
244 - opts->sockets_local ||
239 opts->sockets_inbound ||
240 opts->sockets_outbound);
241 }
@@ -277,6 +271,23 @@ static void topology_parse_options(const char *function, NV_TOPOLOGY_OPTIONS *op
271 continue;
272 }
273
274 + if(strcmp(param, "aggregated") == 0 ||
275 + strcmp(param, "mode:aggregated") == 0 ||
276 + strcmp(param, "__topology_mode:aggregated") == 0 ||
277 + strcmp(param, "__topology_mode=aggregated") == 0 ||
278 + strcmp(param, "view:aggregated") == 0) {
279 + opts->detailed = false;
280 + continue;
281 + }
282 + if(strcmp(param, "detailed") == 0 ||
283 + strcmp(param, "mode:detailed") == 0 ||
284 + strcmp(param, "__topology_mode:detailed") == 0 ||
285 + strcmp(param, "__topology_mode=detailed") == 0 ||
286 + strcmp(param, "view:detailed") == 0) {
287 + opts->detailed = true;
288 + continue;
289 + }
290 +
291 if(strcmp(param, "processes:by_name") == 0 || strcmp(param, "processes:by-name") == 0) {
292 opts->processes_by_pid = false;
293 continue;
@@ -292,7 +303,6 @@ static void topology_parse_options(const char *function, NV_TOPOLOGY_OPTIONS *op
303 if(strncmp(param, "sockets:", 8) == 0) {
304 if(!sockets_selected_explicitly) {
305 opts->sockets_listening = false;
295 - opts->sockets_local = false;
306 opts->sockets_inbound = false;
307 opts->sockets_outbound = false;
308 sockets_selected_explicitly = true;
@@ -309,8 +319,6 @@ static void topology_parse_options(const char *function, NV_TOPOLOGY_OPTIONS *op
319
320 if(strcmp(socket_kind, "listening") == 0)
321 opts->sockets_listening = true;
312 - else if(strcmp(socket_kind, "local") == 0)
313 - opts->sockets_local = true;
322 else if(strcmp(socket_kind, "inbound") == 0)
323 opts->sockets_inbound = true;
324 else if(strcmp(socket_kind, "outbound") == 0)
@@ -354,7 +362,6 @@ static void topology_parse_options(const char *function, NV_TOPOLOGY_OPTIONS *op
362
363 if(!topology_sockets_any_enabled(opts)) {
364 opts->sockets_listening = false;
357 - opts->sockets_local = false;
365 opts->sockets_inbound = true;
366 opts->sockets_outbound = true;
367 }
@@ -399,19 +406,6 @@ static bool socket_endpoint_to_ip_text(const struct socket_endpoint *ep, char *d
406 return false;
407 }
408
402 -static inline void topology_format_ip_port(const char *ip, uint16_t port, char *dst, size_t dst_size) {
403 - if(!dst || !dst_size)
404 - return;
405 -
406 - if(!ip)
407 - ip = "";
408 -
409 - if(strchr(ip, ':'))
410 - snprintf(dst, dst_size, "[%s]:%u", ip, port);
411 - else
412 - snprintf(dst, dst_size, "%s:%u", ip, port);
413 -}
414 -
409 static inline bool topology_ip_is_unspecified(const char *ip) {
410 if(!ip || !*ip)
411 return true;
@@ -453,46 +447,6 @@ static inline bool topology_ip_belongs_to_self(const NV_TOPOLOGY_CONTEXT *ctx, c
447 return false;
448 }
449
456 -static void topology_add_single_item_string_array(BUFFER *wb, const char *key, const char *value) {
457 - if(!value || !*value)
458 - return;
459 -
460 - if(strcmp(key, "ip_addresses") == 0 && strcmp(value, "*") == 0)
461 - return;
462 -
463 - buffer_json_member_add_array(wb, key);
464 - {
465 - buffer_json_add_array_item_string(wb, value);
466 - }
467 - buffer_json_array_close(wb);
468 -}
469 -
470 -static void topology_add_process_match(BUFFER *wb, const NV_TOPOLOGY_CONTEXT *ctx, const NV_PROCESS_ACTOR *pa) {
471 - buffer_json_member_add_object(wb, "match");
472 - {
473 - buffer_json_member_add_string(wb, "process_name", pa->process);
474 - if(ctx && ctx->options.processes_by_pid) {
475 - buffer_json_member_add_uint64(wb, "pid", pa->pid);
476 - buffer_json_member_add_uint64(wb, "uid", pa->uid);
477 - buffer_json_member_add_uint64(wb, "net_ns_inode", pa->net_ns_inode);
478 - }
479 - }
480 - buffer_json_object_close(wb);
481 -}
482 -
483 -static void topology_add_process_identity_match(BUFFER *wb, const NV_TOPOLOGY_CONTEXT *ctx, uint64_t pid, uint64_t uid, uint64_t net_ns_inode, const char *process_name) {
484 - buffer_json_member_add_object(wb, "match");
485 - {
486 - buffer_json_member_add_string(wb, "process_name", process_name && *process_name ? process_name : "[unknown]");
487 - if(ctx && ctx->options.processes_by_pid) {
488 - buffer_json_member_add_uint64(wb, "pid", pid);
489 - buffer_json_member_add_uint64(wb, "uid", uid);
490 - buffer_json_member_add_uint64(wb, "net_ns_inode", net_ns_inode);
491 - }
492 - }
493 - buffer_json_object_close(wb);
494 -}
495 -
450 static inline void topology_process_parent_lookup_key(
451 char *dst,
452 size_t dst_size,
@@ -725,36 +679,6 @@ static NV_ENDPOINT_OWNER *topology_lookup_endpoint_owner(
679 return NULL;
680 }
681
728 -static void topology_add_host_match(BUFFER *wb, const NV_TOPOLOGY_CONTEXT *ctx) {
729 - buffer_json_member_add_object(wb, "match");
730 - {
731 - if(ctx->machine_guid[0])
732 - buffer_json_member_add_string(wb, "netdata_machine_guid", ctx->machine_guid);
733 -
734 - topology_add_single_item_string_array(wb, "hostnames", ctx->hostname);
735 - buffer_json_member_add_array(wb, "ip_addresses");
736 - {
737 - NV_LOCAL_IP *lip;
738 - dfe_start_read(ctx->local_ips, lip) {
739 - if(!lip->ip[0]) continue;
740 - if(topology_ip_is_unspecified(lip->ip)) continue;
741 - buffer_json_add_array_item_string(wb, lip->ip);
742 - }
743 - dfe_done(lip);
744 - }
745 - buffer_json_array_close(wb);
746 - }
747 - buffer_json_object_close(wb);
748 -}
749 -
750 -static void topology_add_remote_match(BUFFER *wb, const char *ip) {
751 - buffer_json_member_add_object(wb, "match");
752 - {
753 - topology_add_single_item_string_array(wb, "ip_addresses", ip);
754 - }
755 - buffer_json_object_close(wb);
756 -}
757 -
682 static void topology_encode_identifier_component(char *dst, size_t dst_size, const char *src) {
683 if(!dst || !dst_size)
684 return;
@@ -1171,17 +1095,35 @@ static void local_sockets_cb_to_aggregation(LS_STATE *ls __maybe_unused, const L
1095 }
1096 }
1097
1098 +static bool topology_socket_direction_selected(const NV_TOPOLOGY_CONTEXT *ctx, SOCKET_DIRECTION direction) {
1099 + if(!ctx)
1100 + return false;
1101 +
1102 + switch(direction) {
1103 + case SOCKET_DIRECTION_LISTEN:
1104 + return ctx->options.sockets_listening;
1105 + case SOCKET_DIRECTION_INBOUND:
1106 + case SOCKET_DIRECTION_LOCAL_INBOUND:
1107 + return ctx->options.sockets_inbound;
1108 + case SOCKET_DIRECTION_OUTBOUND:
1109 + case SOCKET_DIRECTION_LOCAL_OUTBOUND:
1110 + return ctx->options.sockets_outbound;
1111 + default:
1112 + return false;
1113 + }
1114 +}
1115 +
1116 static void local_sockets_cb_to_topology(LS_STATE *ls, const LOCAL_SOCKET *n, void *data) {
1117 if(n->direction == SOCKET_DIRECTION_NONE)
1118 return;
1119
1120 NV_TOPOLOGY_CONTEXT *ctx = data;
1179 - bool hidden_listen_socket = (n->direction == SOCKET_DIRECTION_LISTEN && !ctx->options.sockets_listening);
1121 ctx->sockets_total++;
1122 + bool selected_socket = topology_socket_direction_selected(ctx, n->direction);
1123 + bool hidden_listen_socket = (n->direction == SOCKET_DIRECTION_LISTEN && !ctx->options.sockets_listening);
1124
1125 char local_ip[INET6_ADDRSTRLEN] = "";
1126 char remote_ip[INET6_ADDRSTRLEN] = "";
1184 - char remote_peer_ip[INET6_ADDRSTRLEN] = "";
1127
1128 if(is_local_socket_ipv46(n))
1129 strncpyz(local_ip, "*", sizeof(local_ip) - 1);
@@ -1190,7 +1132,6 @@ static void local_sockets_cb_to_topology(LS_STATE *ls, const LOCAL_SOCKET *n, vo
1132
1133 if(!local_sockets_is_zero_address(&n->remote)) {
1134 socket_endpoint_to_ip_text(&n->remote, remote_ip);
1193 - snprintf(remote_peer_ip, sizeof(remote_peer_ip), "%s", remote_ip);
1135 }
1136
1137 const char *namespace_type;
@@ -1239,6 +1180,11 @@ static void local_sockets_cb_to_topology(LS_STATE *ls, const LOCAL_SOCKET *n, vo
1180 return;
1181 }
1182
1183 + if(!selected_socket) {
1184 + ctx->skipped_sockets++;
1185 + return;
1186 + }
1187 +
1188 char process_key[NV_TOPOLOGY_KEY_MAX];
1189 if(ctx->options.processes_by_pid) {
1190 snprintf(process_key, sizeof(process_key), "pid=%d|uid=%u|ns=%llu",
@@ -1283,91 +1229,70 @@ static void local_sockets_cb_to_topology(LS_STATE *ls, const LOCAL_SOCKET *n, vo
1229 n->direction == SOCKET_DIRECTION_LOCAL_INBOUND);
1230 topology_register_endpoint_owner(ctx, n->net_ns_inode, n->local.protocol, local_ip, n->local.port, pa, service_candidate);
1231
1286 - if(!remote_ip[0]) {
1287 - if(n->direction == SOCKET_DIRECTION_LISTEN || n->direction == SOCKET_DIRECTION_LOCAL_INBOUND) {
1288 - if(strcmp(local_ip, "*") == 0 || local_sockets_is_zero_address(&n->local)) {
1289 - ctx->skipped_sockets++;
1290 - return;
1291 - }
1292 - snprintf(remote_ip, sizeof(remote_ip), "%s", local_ip);
1293 - remote_address_space = local_address_space;
1294 - }
1295 - else {
1296 - ctx->skipped_sockets++;
1297 - return;
1298 - }
1232 + if(n->direction == SOCKET_DIRECTION_LISTEN) {
1233 + ctx->skipped_sockets++;
1234 + return;
1235 }
1236
1301 - if(topology_ip_is_unspecified(remote_ip)) {
1237 + if(!remote_ip[0] || topology_ip_is_unspecified(remote_ip)) {
1238 ctx->skipped_sockets++;
1239 return;
1240 }
1241
1306 - bool remote_is_self = topology_ip_belongs_to_self(ctx, remote_ip, remote_address_space);
1307 - bool self_owned_listen_socket = (n->direction == SOCKET_DIRECTION_LISTEN && remote_is_self);
1308 - bool create_endpoint_actor = !remote_is_self;
1309 - if(create_endpoint_actor) {
1242 + bool process_is_client = (n->direction == SOCKET_DIRECTION_OUTBOUND ||
1243 + n->direction == SOCKET_DIRECTION_LOCAL_OUTBOUND);
1244 +
1245 + char client_ip[INET6_ADDRSTRLEN] = "";
1246 + char server_ip[INET6_ADDRSTRLEN] = "";
1247 + const char *client_address_space = NULL;
1248 + const char *server_address_space = NULL;
1249 + uint16_t client_port = 0;
1250 + uint16_t server_port = 0;
1251 +
1252 + if(process_is_client) {
1253 + snprintf(client_ip, sizeof(client_ip), "%s", local_ip);
1254 + snprintf(server_ip, sizeof(server_ip), "%s", remote_ip);
1255 + client_address_space = local_address_space;
1256 + server_address_space = remote_address_space;
1257 + client_port = n->local.port;
1258 + server_port = n->remote.port;
1259 + }
1260 + else {
1261 + snprintf(client_ip, sizeof(client_ip), "%s", remote_ip);
1262 + snprintf(server_ip, sizeof(server_ip), "%s", local_ip);
1263 + client_address_space = remote_address_space;
1264 + server_address_space = local_address_space;
1265 + client_port = n->remote.port;
1266 + server_port = n->local.port;
1267 + }
1268 +
1269 + const char *endpoint_ip = process_is_client ? server_ip : client_ip;
1270 + const char *endpoint_address_space = process_is_client ? server_address_space : client_address_space;
1271 + bool endpoint_is_self = topology_ip_belongs_to_self(ctx, endpoint_ip, endpoint_address_space);
1272 + if(!endpoint_is_self) {
1273 char endpoint_actor_key[NV_TOPOLOGY_KEY_MAX];
1311 - topology_actor_id_for_remote_endpoint(ctx, remote_ip, remote_address_space, endpoint_actor_key, sizeof(endpoint_actor_key));
1274 + topology_actor_id_for_remote_endpoint(ctx, endpoint_ip, endpoint_address_space, endpoint_actor_key, sizeof(endpoint_actor_key));
1275 NV_REMOTE_ACTOR *ra = dictionary_get(ctx->remote_actors, endpoint_actor_key);
1276 if(!ra) {
1277 NV_REMOTE_ACTOR tmp = { 0 };
1315 - snprintf(tmp.ip, sizeof(tmp.ip), "%s", remote_ip);
1316 - snprintf(tmp.address_space, sizeof(tmp.address_space), "%s", remote_address_space);
1278 + snprintf(tmp.ip, sizeof(tmp.ip), "%s", endpoint_ip);
1279 + snprintf(tmp.address_space, sizeof(tmp.address_space), "%s", endpoint_address_space);
1280 ra = dictionary_set(ctx->remote_actors, endpoint_actor_key, &tmp, sizeof(tmp));
1281 }
1282 ra->sockets++;
1283 }
1284
1322 - if(self_owned_listen_socket) {
1323 - ctx->skipped_sockets++;
1324 - return;
1325 - }
1326 -
1327 - const struct socket_endpoint *server_endpoint = NULL;
1328 - uint16_t endpoint_port = n->remote.port;
1329 - switch(n->direction) {
1330 - case SOCKET_DIRECTION_LISTEN:
1331 - server_endpoint = &n->local;
1332 - endpoint_port = n->local.port;
1333 - break;
1334 -
1335 - case SOCKET_DIRECTION_INBOUND:
1336 - case SOCKET_DIRECTION_LOCAL_INBOUND:
1337 - server_endpoint = &n->local;
1338 - endpoint_port = n->remote.port;
1339 - break;
1340 -
1341 - case SOCKET_DIRECTION_OUTBOUND:
1342 - case SOCKET_DIRECTION_LOCAL_OUTBOUND:
1343 - server_endpoint = &n->remote;
1344 - endpoint_port = n->remote.port;
1345 - break;
1346 -
1347 - default:
1348 - break;
1349 - }
1350 -
1351 - char port_name[64] = "[unknown]";
1352 - if(server_endpoint) {
1353 - STRING *serv = system_servicenames_cache_lookup(sc, server_endpoint->port, server_endpoint->protocol);
1354 - const char *tmp_name = string2str(serv);
1355 - if(tmp_name && *tmp_name)
1356 - snprintf(port_name, sizeof(port_name), "%s", tmp_name);
1357 - }
1358 -
1285 char link_key[NV_TOPOLOGY_KEY_MAX];
1360 - snprintf(link_key, sizeof(link_key), "pid=%d|uid=%u|ns=%llu|local=%s|remote=%s|proto=%u|dir=%u|state=%u|lport=%u|rport=%u",
1286 + snprintf(link_key, sizeof(link_key), "pid=%d|uid=%u|ns=%llu|client=%s:%u|server=%s:%u|proto=%u|state=%u",
1287 n->pid,
1288 (unsigned)n->uid,
1289 (unsigned long long)n->net_ns_inode,
1364 - local_ip,
1365 - remote_ip,
1290 + client_ip,
1291 + (unsigned)client_port,
1292 + server_ip,
1293 + (unsigned)server_port,
1294 (unsigned)n->local.protocol,
1367 - (unsigned)n->direction,
1368 - (unsigned)n->state,
1369 - n->local.port,
1370 - endpoint_port);
1295 + (unsigned)n->state);
1296
1297 NV_TOPOLOGY_LINK *link = dictionary_get(ctx->links, link_key);
1298 if(!link) {
@@ -1376,25 +1301,22 @@ static void local_sockets_cb_to_topology(LS_STATE *ls, const LOCAL_SOCKET *n, vo
1301 tmp.ppid = n->ppid;
1302 tmp.uid = n->uid;
1303 tmp.net_ns_inode = n->net_ns_inode;
1379 - tmp.local_port = n->local.port;
1380 - tmp.remote_port = endpoint_port;
1381 - tmp.peer_port = n->remote.port;
1304 + tmp.client_port = client_port;
1305 + tmp.server_port = server_port;
1306 + tmp.process_port = n->local.port;
1307 tmp.protocol_id = n->local.protocol;
1383 - tmp.direction_id = (uint8_t)n->direction;
1308 + tmp.process_is_client = process_is_client;
1309 snprintf(tmp.process, sizeof(tmp.process), "%s", process_name);
1310 snprintf(tmp.username, sizeof(tmp.username), "%s", username);
1311 snprintf(tmp.namespace_type, sizeof(tmp.namespace_type), "%s", namespace_type);
1312 snprintf(tmp.protocol, sizeof(tmp.protocol), "%s", socket_protocol_name(n->local.protocol));
1313 snprintf(tmp.protocol_family, sizeof(tmp.protocol_family), "%s", socket_protocol_family_name(n));
1389 - snprintf(tmp.direction, sizeof(tmp.direction), "%s", SOCKET_DIRECTION_2str(n->direction));
1314 snprintf(tmp.state, sizeof(tmp.state), "%s",
1315 n->local.protocol == IPPROTO_TCP ? TCP_STATE_2str(n->state) : "stateless");
1392 - snprintf(tmp.local_ip, sizeof(tmp.local_ip), "%s", local_ip);
1393 - snprintf(tmp.remote_ip, sizeof(tmp.remote_ip), "%s", remote_ip);
1394 - snprintf(tmp.peer_ip, sizeof(tmp.peer_ip), "%s", remote_peer_ip);
1395 - snprintf(tmp.local_address_space, sizeof(tmp.local_address_space), "%s", local_address_space);
1396 - snprintf(tmp.remote_address_space, sizeof(tmp.remote_address_space), "%s", remote_address_space);
1397 - snprintf(tmp.port_name, sizeof(tmp.port_name), "%s", port_name);
1316 + snprintf(tmp.client_ip, sizeof(tmp.client_ip), "%s", client_ip);
1317 + snprintf(tmp.server_ip, sizeof(tmp.server_ip), "%s", server_ip);
1318 + snprintf(tmp.client_address_space, sizeof(tmp.client_address_space), "%s", client_address_space);
1319 + snprintf(tmp.server_address_space, sizeof(tmp.server_address_space), "%s", server_address_space);
1320 if(cmdline && *cmdline)
1321 snprintf(tmp.cmdline, sizeof(tmp.cmdline), "%s", cmdline);
1322 link = dictionary_set(ctx->links, link_key, &tmp, sizeof(tmp));
@@ -1430,63 +1352,6 @@ static void topology_context_destroy(NV_TOPOLOGY_CONTEXT *ctx) {
1352 dictionary_destroy(ctx->process_actors);
1353 }
1354
1433 -static DICTIONARY *topology_build_process_socket_index(const NV_TOPOLOGY_CONTEXT *ctx) {
1434 - if(!ctx || !ctx->links)
1435 - return NULL;
1436 -
1437 - DICTIONARY *index = dictionary_create_advanced(
1438 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
1439 - NULL,
1440 - sizeof(NV_PROCESS_SOCKET_ROWS));
1441 - if(!index)
1442 - return NULL;
1443 -
1444 - NV_TOPOLOGY_LINK *link;
1445 - dfe_start_read(ctx->links, link) {
1446 - char process_actor_id[NV_TOPOLOGY_KEY_MAX];
1447 - topology_actor_id_for_process(ctx, link->pid, link->uid, link->net_ns_inode, link->process, process_actor_id, sizeof(process_actor_id));
1448 -
1449 - NV_PROCESS_SOCKET_ROWS *rows = dictionary_get(index, process_actor_id);
1450 - if(!rows) {
1451 - NV_PROCESS_SOCKET_ROWS tmp = { 0 };
1452 - rows = dictionary_set(index, process_actor_id, &tmp, sizeof(tmp));
1453 - }
1454 -
1455 - if(!rows)
1456 - continue;
1457 -
1458 - NV_PROCESS_SOCKET_ROW *row = callocz(1, sizeof(*row));
1459 - row->link = link;
1460 -
1461 - if(rows->tail)
1462 - rows->tail->next = row;
1463 - else
1464 - rows->head = row;
1465 - rows->tail = row;
1466 - }
1467 - dfe_done(link);
1468 -
1469 - return index;
1470 -}
1471 -
1472 -static void topology_destroy_process_socket_index(DICTIONARY *index) {
1473 - if(!index)
1474 - return;
1475 -
1476 - NV_PROCESS_SOCKET_ROWS *rows;
1477 - dfe_start_read(index, rows) {
1478 - NV_PROCESS_SOCKET_ROW *row = rows->head;
1479 - while(row) {
1480 - NV_PROCESS_SOCKET_ROW *next = row->next;
1481 - freez(row);
1482 - row = next;
1483 - }
1484 - }
1485 - dfe_done(rows);
1486 -
1487 - dictionary_destroy(index);
1488 -}
1489 -
1355 static bool topology_prepare_context(NV_TOPOLOGY_CONTEXT *ctx, usec_t now_ut, const NV_TOPOLOGY_OPTIONS *options) {
1356 if(!ctx)
1357 return false;
@@ -1526,7 +1391,7 @@ static bool topology_prepare_context(NV_TOPOLOGY_CONTEXT *ctx, usec_t now_ut, co
1391 // Always collect listeners so inbound/outbound socket classification
1392 // remains stable across topology socket filters.
1393 .listening = true,
1529 - .local = ctx->options.sockets_local,
1394 + .local = ctx->options.sockets_inbound || ctx->options.sockets_outbound,
1395 .inbound = ctx->options.sockets_inbound,
1396 .outbound = ctx->options.sockets_outbound,
1397 .tcp4 = ctx->options.protocols_ipv4_tcp,
@@ -1593,6 +1458,8 @@ static void topology_write_response_metadata(BUFFER *wb) {
1458 {
1459 buffer_json_add_array_item_string(wb, "info");
1460 buffer_json_add_array_item_string(wb, "processes");
1461 + buffer_json_add_array_item_string(wb, "__topology_mode");
1462 + buffer_json_add_array_item_string(wb, "mode");
1463 buffer_json_add_array_item_string(wb, "sockets");
1464 buffer_json_add_array_item_string(wb, "protocols");
1465 buffer_json_add_array_item_string(wb, "endpoints");
@@ -1627,6 +1494,33 @@ static void topology_write_response_metadata(BUFFER *wb) {
1494 }
1495 buffer_json_object_close(wb);
1496
1497 + buffer_json_add_array_item_object(wb);
1498 + {
1499 + buffer_json_member_add_string(wb, "id", "__topology_mode");
1500 + buffer_json_member_add_string(wb, "name", "Mode");
1501 + buffer_json_member_add_string(wb, "help", "Return an aggregated graph by default, or include lossless socket evidence for detailed correlation.");
1502 + buffer_json_member_add_boolean(wb, "unique_view", true);
1503 + buffer_json_member_add_string(wb, "type", "select");
1504 + buffer_json_member_add_array(wb, "options");
1505 + {
1506 + buffer_json_add_array_item_object(wb);
1507 + {
1508 + buffer_json_member_add_string(wb, "id", "aggregated");
1509 + buffer_json_member_add_string(wb, "name", "Aggregated");
1510 + buffer_json_member_add_boolean(wb, "defaultSelected", true);
1511 + }
1512 + buffer_json_object_close(wb);
1513 + buffer_json_add_array_item_object(wb);
1514 + {
1515 + buffer_json_member_add_string(wb, "id", "detailed");
1516 + buffer_json_member_add_string(wb, "name", "Detailed");
1517 + }
1518 + buffer_json_object_close(wb);
1519 + }
1520 + buffer_json_array_close(wb);
1521 + }
1522 + buffer_json_object_close(wb);
1523 +
1524 buffer_json_add_array_item_object(wb);
1525 {
1526 buffer_json_member_add_string(wb, "id", "sockets");
@@ -1642,12 +1536,6 @@ static void topology_write_response_metadata(BUFFER *wb) {
1536 }
1537 buffer_json_object_close(wb);
1538 buffer_json_add_array_item_object(wb);
1645 - {
1646 - buffer_json_member_add_string(wb, "id", "local");
1647 - buffer_json_member_add_string(wb, "name", "Local");
1648 - }
1649 - buffer_json_object_close(wb);
1650 - buffer_json_add_array_item_object(wb);
1539 {
1540 buffer_json_member_add_string(wb, "id", "inbound");
1541 buffer_json_member_add_string(wb, "name", "Inbound");
@@ -1731,338 +1619,1664 @@ static void topology_write_response_metadata(BUFFER *wb) {
1619 buffer_json_array_close(wb);
1620 }
1621
1734 -static void topology_write_presentation(BUFFER *wb) {
1735 - buffer_json_member_add_object(wb, "presentation");
1736 - {
1737 - buffer_json_member_add_object(wb, "actor_types");
1738 - {
1739 - buffer_json_member_add_object(wb, "self");
1740 - {
1741 - buffer_json_member_add_string(wb, "label", "This host");
1742 - buffer_json_member_add_string(wb, "color_slot", "self");
1743 - buffer_json_member_add_boolean(wb, "border", true);
1744 - buffer_json_member_add_string(wb, "role", "actor");
1745 - buffer_json_member_add_boolean(wb, "size_by_links", true);
1622 +typedef struct {
1623 + char id[NV_TOPOLOGY_KEY_MAX];
1624 + char type[16];
1625 + char machine_guid[128];
1626 + char hostname[256];
1627 + char process[TASK_COMM_LEN + 1];
1628 + char username[NV_TOPOLOGY_USERNAME_MAX];
1629 + char namespace_type[16];
1630 + char local_ip[INET6_ADDRSTRLEN];
1631 + char local_address_space[16];
1632 + char ip[INET6_ADDRSTRLEN];
1633 + char address_space[16];
1634 + char display_name[NV_TOPOLOGY_KEY_MAX];
1635 + char cmdline[NV_TOPOLOGY_CMDLINE_MAX];
1636 + uint64_t pid;
1637 + uint64_t ppid;
1638 + uint64_t uid;
1639 + uint64_t net_ns_inode;
1640 + uint64_t sockets;
1641 + uint64_t local_ip_count;
1642 + bool has_pid;
1643 + bool has_ppid;
1644 + bool has_uid;
1645 + bool has_net_ns_inode;
1646 + bool has_local_ip_count;
1647 +} NV_TOPOLOGY_V1_ACTOR;
1648
1747 - buffer_json_member_add_array(wb, "summary_fields");
1748 - {
1749 - buffer_json_add_array_item_object(wb);
1750 - buffer_json_member_add_string(wb, "key", "hostname");
1751 - buffer_json_member_add_string(wb, "label", "Hostname");
1752 - buffer_json_member_add_array(wb, "sources");
1753 - buffer_json_add_array_item_string(wb, "attributes.hostname");
1754 - buffer_json_array_close(wb);
1755 - buffer_json_object_close(wb);
1649 +typedef struct {
1650 + uint64_t src_actor;
1651 + uint64_t dst_actor;
1652 + char type[32];
1653 + char protocol[16];
1654 + char state[32];
1655 + uint64_t evidence_count;
1656 + uint64_t socket_count;
1657 + uint64_t retransmissions;
1658 + uint32_t max_rtt_usec;
1659 + uint32_t max_rcv_rtt_usec;
1660 +} NV_TOPOLOGY_V1_GRAPH_LINK;
1661
1757 - buffer_json_add_array_item_object(wb);
1758 - buffer_json_member_add_string(wb, "key", "local_ip_count");
1759 - buffer_json_member_add_string(wb, "label", "Local IPs");
1760 - buffer_json_member_add_array(wb, "sources");
1761 - buffer_json_add_array_item_string(wb, "attributes.local_ip_count");
1762 - buffer_json_array_close(wb);
1763 - buffer_json_object_close(wb);
1662 +typedef struct {
1663 + uint64_t link;
1664 + uint64_t src_actor;
1665 + uint64_t dst_actor;
1666 + const NV_TOPOLOGY_LINK *source;
1667 +} NV_TOPOLOGY_V1_SOCKET_EVIDENCE;
1668
1765 - buffer_json_add_array_item_object(wb);
1766 - buffer_json_member_add_string(wb, "key", "observed_sockets");
1767 - buffer_json_member_add_string(wb, "label", "Sockets");
1768 - buffer_json_member_add_array(wb, "sources");
1769 - buffer_json_add_array_item_string(wb, "attributes.observed_sockets");
1770 - buffer_json_array_close(wb);
1771 - buffer_json_object_close(wb);
1772 - }
1773 - buffer_json_array_close(wb);
1669 +typedef struct {
1670 + uint64_t src_actor;
1671 + uint64_t dst_actor;
1672 + uint64_t socket_count;
1673 + uint64_t retransmissions;
1674 + uint32_t max_rtt_usec;
1675 + uint32_t max_rcv_rtt_usec;
1676 + char client_ip[INET6_ADDRSTRLEN];
1677 + char server_ip[INET6_ADDRSTRLEN];
1678 + char protocol[16];
1679 + char state[32];
1680 +} NV_TOPOLOGY_V1_CONNECTION_ROW;
1681
1775 - buffer_json_member_add_object(wb, "tables");
1776 - {
1777 - buffer_json_member_add_object(wb, "links");
1778 - {
1779 - buffer_json_member_add_string(wb, "label", "Connections");
1780 - buffer_json_member_add_string(wb, "source", "links");
1781 - buffer_json_member_add_array(wb, "columns");
1782 - {
1783 - buffer_json_add_array_item_object(wb);
1784 - buffer_json_member_add_string(wb, "key", "remoteLabel");
1785 - buffer_json_member_add_string(wb, "label", "Remote");
1786 - buffer_json_member_add_string(wb, "type", "actor_link");
1787 - buffer_json_object_close(wb);
1682 +typedef struct {
1683 + uint64_t actor;
1684 + uint64_t port;
1685 + uint64_t socket_count;
1686 + char protocol[16];
1687 +} NV_TOPOLOGY_V1_PORT_ROW;
1688
1789 - buffer_json_add_array_item_object(wb);
1790 - buffer_json_member_add_string(wb, "key", "protocol");
1791 - buffer_json_member_add_string(wb, "label", "Protocol");
1792 - buffer_json_object_close(wb);
1689 +typedef struct {
1690 + uint64_t actor;
1691 + uint64_t port;
1692 + char protocol[16];
1693 + char address_space[16];
1694 + char ip[INET6_ADDRSTRLEN];
1695 +} NV_TOPOLOGY_V1_CORRELATION_ROW;
1696
1794 - buffer_json_add_array_item_object(wb);
1795 - buffer_json_member_add_string(wb, "key", "direction");
1796 - buffer_json_member_add_string(wb, "label", "Direction");
1797 - buffer_json_object_close(wb);
1798 - }
1799 - buffer_json_array_close(wb);
1800 - }
1801 - buffer_json_object_close(wb);
1802 - }
1803 - buffer_json_object_close(wb);
1697 +typedef struct {
1698 + uint64_t actor;
1699 + uint64_t value_index;
1700 + bool has_value_index;
1701 + char key[NV_TOPOLOGY_LABEL_KEY_MAX];
1702 + char value[NV_TOPOLOGY_LABEL_VALUE_MAX];
1703 + char source[32];
1704 + char kind[32];
1705 +} NV_TOPOLOGY_V1_ACTOR_LABEL;
1706
1805 - buffer_json_member_add_array(wb, "modal_tabs");
1806 - {
1807 - buffer_json_add_array_item_object(wb);
1808 - buffer_json_member_add_string(wb, "id", "info");
1809 - buffer_json_member_add_string(wb, "label", "Info");
1810 - buffer_json_object_close(wb);
1811 - }
1812 - buffer_json_array_close(wb);
1813 - }
1814 - buffer_json_object_close(wb);
1707 +typedef struct {
1708 + NV_TOPOLOGY_V1_ACTOR *actors;
1709 + size_t actors_used;
1710 + size_t actors_size;
1711 +
1712 + NV_TOPOLOGY_V1_GRAPH_LINK *links;
1713 + size_t links_used;
1714 + size_t links_size;
1715 +
1716 + NV_TOPOLOGY_V1_SOCKET_EVIDENCE *evidence;
1717 + size_t evidence_used;
1718 + size_t evidence_size;
1719 +
1720 + NV_TOPOLOGY_V1_CONNECTION_ROW *connections;
1721 + size_t connections_used;
1722 + size_t connections_size;
1723 +
1724 + NV_TOPOLOGY_V1_PORT_ROW *ports;
1725 + size_t ports_used;
1726 + size_t ports_size;
1727 +
1728 + NV_TOPOLOGY_V1_CORRELATION_ROW *correlation_points;
1729 + size_t correlation_points_used;
1730 + size_t correlation_points_size;
1731 +
1732 + NV_TOPOLOGY_V1_CORRELATION_ROW *correlation_claims;
1733 + size_t correlation_claims_used;
1734 + size_t correlation_claims_size;
1735 +
1736 + NV_TOPOLOGY_V1_ACTOR_LABEL *labels;
1737 + size_t labels_used;
1738 + size_t labels_size;
1739 +
1740 + DICTIONARY *actor_index;
1741 + DICTIONARY *graph_link_index;
1742 + DICTIONARY *connection_index;
1743 + DICTIONARY *port_index;
1744 + DICTIONARY *correlation_point_index;
1745 + DICTIONARY *correlation_claim_index;
1746 +} NV_TOPOLOGY_V1_PAYLOAD;
1747
1816 - buffer_json_member_add_object(wb, "process");
1817 - {
1818 - buffer_json_member_add_string(wb, "label", "Process");
1819 - buffer_json_member_add_string(wb, "color_slot", "primary");
1820 - buffer_json_member_add_boolean(wb, "border", true);
1821 - buffer_json_member_add_string(wb, "role", "actor");
1822 - buffer_json_member_add_boolean(wb, "size_by_links", true);
1823 - buffer_json_member_add_boolean(wb, "show_port_bullets", true);
1748 +typedef struct {
1749 + const char **values;
1750 + uint64_t *indexes;
1751 + size_t rows;
1752 + size_t rows_used;
1753 + size_t values_used;
1754 + size_t values_size;
1755 + size_t values_json_size;
1756 + size_t unique_json_size;
1757 + size_t indexes_json_size;
1758 + DICTIONARY *index;
1759 +} NV_TOPOLOGY_V1_STRING_COLUMN;
1760 +
1761 +static void topology_v1_strncpy(char *dst, size_t dst_size, const char *src) {
1762 + if(!dst || !dst_size)
1763 + return;
1764
1825 - buffer_json_member_add_array(wb, "summary_fields");
1826 - {
1827 - buffer_json_add_array_item_object(wb);
1828 - buffer_json_member_add_string(wb, "key", "display_name");
1829 - buffer_json_member_add_string(wb, "label", "Process");
1830 - buffer_json_member_add_array(wb, "sources");
1831 - buffer_json_add_array_item_string(wb, "attributes.display_name");
1832 - buffer_json_array_close(wb);
1833 - buffer_json_object_close(wb);
1765 + strncpyz(dst, src ? src : "", dst_size - 1);
1766 +}
1767
1835 - buffer_json_add_array_item_object(wb);
1836 - buffer_json_member_add_string(wb, "key", "cmdline");
1837 - buffer_json_member_add_string(wb, "label", "Command");
1838 - buffer_json_member_add_array(wb, "sources");
1839 - buffer_json_add_array_item_string(wb, "attributes.cmdline");
1840 - buffer_json_array_close(wb);
1841 - buffer_json_object_close(wb);
1768 +static void topology_v1_actor_index_set(NV_TOPOLOGY_V1_PAYLOAD *payload, const char *actor_id, uint64_t index) {
1769 + dictionary_set(payload->actor_index, actor_id, &index, sizeof(index));
1770 +}
1771
1843 - buffer_json_add_array_item_object(wb);
1844 - buffer_json_member_add_string(wb, "key", "socket_count");
1845 - buffer_json_member_add_string(wb, "label", "Sockets");
1846 - buffer_json_member_add_array(wb, "sources");
1847 - buffer_json_add_array_item_string(wb, "attributes.socket_count");
1848 - buffer_json_array_close(wb);
1849 - buffer_json_object_close(wb);
1772 +static bool topology_v1_actor_index_get(NV_TOPOLOGY_V1_PAYLOAD *payload, const char *actor_id, uint64_t *index) {
1773 + uint64_t *stored = dictionary_get(payload->actor_index, actor_id);
1774 + if(!stored)
1775 + return false;
1776
1851 - buffer_json_add_array_item_object(wb);
1852 - buffer_json_member_add_string(wb, "key", "local_ip");
1853 - buffer_json_member_add_string(wb, "label", "Local IP");
1854 - buffer_json_member_add_array(wb, "sources");
1855 - buffer_json_add_array_item_string(wb, "attributes.local_ip");
1856 - buffer_json_array_close(wb);
1857 - buffer_json_object_close(wb);
1777 + if(index)
1778 + *index = *stored;
1779
1859 - buffer_json_add_array_item_object(wb);
1860 - buffer_json_member_add_string(wb, "key", "user");
1861 - buffer_json_member_add_string(wb, "label", "User");
1780 + return true;
1781 +}
1782 +
1783 +static NV_TOPOLOGY_V1_ACTOR *topology_v1_add_actor(NV_TOPOLOGY_V1_PAYLOAD *payload, const char *actor_id) {
1784 + if(payload->actors_used == payload->actors_size) {
1785 + size_t new_size = payload->actors_size ? payload->actors_size * 2 : 32;
1786 + payload->actors = reallocz(payload->actors, new_size * sizeof(*payload->actors));
1787 + payload->actors_size = new_size;
1788 + }
1789 +
1790 + NV_TOPOLOGY_V1_ACTOR *actor = &payload->actors[payload->actors_used];
1791 + *actor = (NV_TOPOLOGY_V1_ACTOR){ 0 };
1792 + topology_v1_strncpy(actor->id, sizeof(actor->id), actor_id);
1793 + topology_v1_actor_index_set(payload, actor_id, payload->actors_used);
1794 + payload->actors_used++;
1795 + return actor;
1796 +}
1797 +
1798 +static NV_TOPOLOGY_V1_GRAPH_LINK *topology_v1_add_graph_link(NV_TOPOLOGY_V1_PAYLOAD *payload) {
1799 + if(payload->links_used == payload->links_size) {
1800 + size_t new_size = payload->links_size ? payload->links_size * 2 : 64;
1801 + payload->links = reallocz(payload->links, new_size * sizeof(*payload->links));
1802 + payload->links_size = new_size;
1803 + }
1804 +
1805 + NV_TOPOLOGY_V1_GRAPH_LINK *link = &payload->links[payload->links_used++];
1806 + *link = (NV_TOPOLOGY_V1_GRAPH_LINK){ 0 };
1807 + return link;
1808 +}
1809 +
1810 +static NV_TOPOLOGY_V1_SOCKET_EVIDENCE *topology_v1_add_socket_evidence(NV_TOPOLOGY_V1_PAYLOAD *payload) {
1811 + if(payload->evidence_used == payload->evidence_size) {
1812 + size_t new_size = payload->evidence_size ? payload->evidence_size * 2 : 256;
1813 + payload->evidence = reallocz(payload->evidence, new_size * sizeof(*payload->evidence));
1814 + payload->evidence_size = new_size;
1815 + }
1816 +
1817 + NV_TOPOLOGY_V1_SOCKET_EVIDENCE *row = &payload->evidence[payload->evidence_used++];
1818 + *row = (NV_TOPOLOGY_V1_SOCKET_EVIDENCE){ 0 };
1819 + return row;
1820 +}
1821 +
1822 +static void topology_v1_add_connection_row(
1823 + NV_TOPOLOGY_V1_PAYLOAD *payload,
1824 + uint64_t src_actor,
1825 + uint64_t dst_actor,
1826 + const NV_TOPOLOGY_LINK *source) {
1827 + if(!payload || !payload->connection_index || !source)
1828 + return;
1829 +
1830 + char key[NV_TOPOLOGY_KEY_MAX];
1831 + snprintfz(key, sizeof(key), "%"PRIu64"|%"PRIu64"|%s|%s",
1832 + src_actor,
1833 + dst_actor,
1834 + source->protocol,
1835 + source->state);
1836 +
1837 + uint64_t *stored = dictionary_get(payload->connection_index, key);
1838 + if(stored) {
1839 + NV_TOPOLOGY_V1_CONNECTION_ROW *row = &payload->connections[*stored];
1840 + row->socket_count += source->sockets;
1841 + row->retransmissions += source->retransmissions;
1842 + if(row->max_rtt_usec < source->max_rtt_usec)
1843 + row->max_rtt_usec = source->max_rtt_usec;
1844 + if(row->max_rcv_rtt_usec < source->max_rcv_rtt_usec)
1845 + row->max_rcv_rtt_usec = source->max_rcv_rtt_usec;
1846 + return;
1847 + }
1848 +
1849 + if(payload->connections_used == payload->connections_size) {
1850 + size_t new_size = payload->connections_size ? payload->connections_size * 2 : 256;
1851 + payload->connections = reallocz(payload->connections, new_size * sizeof(*payload->connections));
1852 + payload->connections_size = new_size;
1853 + }
1854 +
1855 + uint64_t index = payload->connections_used;
1856 + NV_TOPOLOGY_V1_CONNECTION_ROW *row = &payload->connections[payload->connections_used++];
1857 + *row = (NV_TOPOLOGY_V1_CONNECTION_ROW){ 0 };
1858 + row->src_actor = src_actor;
1859 + row->dst_actor = dst_actor;
1860 + row->socket_count = source->sockets;
1861 + row->retransmissions = source->retransmissions;
1862 + row->max_rtt_usec = source->max_rtt_usec;
1863 + row->max_rcv_rtt_usec = source->max_rcv_rtt_usec;
1864 + topology_v1_strncpy(row->client_ip, sizeof(row->client_ip), source->client_ip);
1865 + topology_v1_strncpy(row->server_ip, sizeof(row->server_ip), source->server_ip);
1866 + topology_v1_strncpy(row->protocol, sizeof(row->protocol), source->protocol);
1867 + topology_v1_strncpy(row->state, sizeof(row->state), source->state);
1868 + dictionary_set(payload->connection_index, key, &index, sizeof(index));
1869 +}
1870 +
1871 +static void topology_v1_add_port_row(
1872 + NV_TOPOLOGY_V1_PAYLOAD *payload,
1873 + uint64_t actor,
1874 + const char *protocol,
1875 + uint64_t port,
1876 + uint64_t socket_count) {
1877 + if(!payload || !payload->port_index || !port || !socket_count)
1878 + return;
1879 +
1880 + char key[NV_TOPOLOGY_KEY_MAX];
1881 + snprintfz(key, sizeof(key), "%"PRIu64"|%s|%"PRIu64,
1882 + actor,
1883 + protocol ? protocol : "",
1884 + port);
1885 +
1886 + uint64_t *stored = dictionary_get(payload->port_index, key);
1887 + if(stored) {
1888 + payload->ports[*stored].socket_count += socket_count;
1889 + return;
1890 + }
1891 +
1892 + if(payload->ports_used == payload->ports_size) {
1893 + size_t new_size = payload->ports_size ? payload->ports_size * 2 : 256;
1894 + payload->ports = reallocz(payload->ports, new_size * sizeof(*payload->ports));
1895 + payload->ports_size = new_size;
1896 + }
1897 +
1898 + uint64_t index = payload->ports_used;
1899 + NV_TOPOLOGY_V1_PORT_ROW *row = &payload->ports[payload->ports_used++];
1900 + *row = (NV_TOPOLOGY_V1_PORT_ROW){ 0 };
1901 + row->actor = actor;
1902 + row->port = port;
1903 + row->socket_count = socket_count;
1904 + topology_v1_strncpy(row->protocol, sizeof(row->protocol), protocol);
1905 +
1906 + dictionary_set(payload->port_index, key, &index, sizeof(index));
1907 +}
1908 +
1909 +static void topology_v1_add_actor_label_ex(
1910 + NV_TOPOLOGY_V1_PAYLOAD *payload,
1911 + uint64_t actor,
1912 + const char *key,
1913 + const char *value,
1914 + const char *source,
1915 + const char *kind,
1916 + bool has_value_index,
1917 + uint64_t value_index) {
1918 + if(!payload || !key || !*key || !value || !*value)
1919 + return;
1920 +
1921 + if(payload->labels_used == payload->labels_size) {
1922 + size_t new_size = payload->labels_size ? payload->labels_size * 2 : 128;
1923 + payload->labels = reallocz(payload->labels, new_size * sizeof(*payload->labels));
1924 + payload->labels_size = new_size;
1925 + }
1926 +
1927 + NV_TOPOLOGY_V1_ACTOR_LABEL *row = &payload->labels[payload->labels_used++];
1928 + *row = (NV_TOPOLOGY_V1_ACTOR_LABEL){ 0 };
1929 + row->actor = actor;
1930 + row->has_value_index = has_value_index;
1931 + row->value_index = value_index;
1932 + topology_v1_strncpy(row->key, sizeof(row->key), key);
1933 + topology_v1_strncpy(row->value, sizeof(row->value), value);
1934 + topology_v1_strncpy(row->source, sizeof(row->source), source ? source : NETWORK_TOPOLOGY_SOURCE);
1935 + topology_v1_strncpy(row->kind, sizeof(row->kind), kind ? kind : "attribute");
1936 +}
1937 +
1938 +static void topology_v1_add_actor_label(
1939 + NV_TOPOLOGY_V1_PAYLOAD *payload,
1940 + uint64_t actor,
1941 + const char *key,
1942 + const char *value,
1943 + const char *kind) {
1944 + topology_v1_add_actor_label_ex(payload, actor, key, value, NETWORK_TOPOLOGY_SOURCE, kind, false, 0);
1945 +}
1946 +
1947 +static void topology_v1_add_actor_label_uint(
1948 + NV_TOPOLOGY_V1_PAYLOAD *payload,
1949 + uint64_t actor,
1950 + const char *key,
1951 + uint64_t value,
1952 + const char *kind) {
1953 + char buffer[32];
1954 + snprintfz(buffer, sizeof(buffer), "%"PRIu64, value);
1955 + topology_v1_add_actor_label(payload, actor, key, buffer, kind);
1956 +}
1957 +
1958 +static void topology_v1_add_correlation_row(
1959 + NV_TOPOLOGY_V1_CORRELATION_ROW **rows,
1960 + size_t *rows_used,
1961 + size_t *rows_size,
1962 + DICTIONARY *index,
1963 + uint64_t actor,
1964 + const char *protocol,
1965 + const char *address_space,
1966 + const char *ip,
1967 + uint64_t port) {
1968 + if(!rows || !rows_used || !rows_size || !index || !port || topology_ip_is_unspecified(ip))
1969 + return;
1970 +
1971 + char key[NV_TOPOLOGY_KEY_MAX];
1972 + snprintfz(key, sizeof(key), "%"PRIu64"|%s|%s|%s|%"PRIu64,
1973 + actor,
1974 + protocol ? protocol : "",
1975 + address_space ? address_space : "",
1976 + ip ? ip : "",
1977 + port);
1978 +
1979 + if(dictionary_get(index, key))
1980 + return;
1981 +
1982 + if(*rows_used == *rows_size) {
1983 + size_t new_size = *rows_size ? *rows_size * 2 : 256;
1984 + *rows = reallocz(*rows, new_size * sizeof(**rows));
1985 + *rows_size = new_size;
1986 + }
1987 +
1988 + NV_TOPOLOGY_V1_CORRELATION_ROW *row = &(*rows)[(*rows_used)++];
1989 + *row = (NV_TOPOLOGY_V1_CORRELATION_ROW){ 0 };
1990 + row->actor = actor;
1991 + row->port = port;
1992 + topology_v1_strncpy(row->protocol, sizeof(row->protocol), protocol);
1993 + topology_v1_strncpy(row->address_space, sizeof(row->address_space), address_space);
1994 + topology_v1_strncpy(row->ip, sizeof(row->ip), ip);
1995 +
1996 + bool stored = true;
1997 + dictionary_set(index, key, &stored, sizeof(stored));
1998 +}
1999 +
2000 +static void topology_v1_add_correlation_claim_endpoint(
2001 + NV_TOPOLOGY_V1_PAYLOAD *payload,
2002 + uint64_t actor,
2003 + const char *protocol,
2004 + const char *address_space,
2005 + const char *ip,
2006 + uint64_t port) {
2007 + if(!payload)
2008 + return;
2009 +
2010 + topology_v1_add_correlation_row(
2011 + &payload->correlation_claims,
2012 + &payload->correlation_claims_used,
2013 + &payload->correlation_claims_size,
2014 + payload->correlation_claim_index,
2015 + actor,
2016 + protocol,
2017 + address_space,
2018 + ip,
2019 + port);
2020 +}
2021 +
2022 +static void topology_v1_add_correlation_point_endpoint(
2023 + NV_TOPOLOGY_V1_PAYLOAD *payload,
2024 + uint64_t actor,
2025 + const char *protocol,
2026 + const char *address_space,
2027 + const char *ip,
2028 + uint64_t port) {
2029 + if(!payload)
2030 + return;
2031 +
2032 + topology_v1_add_correlation_row(
2033 + &payload->correlation_points,
2034 + &payload->correlation_points_used,
2035 + &payload->correlation_points_size,
2036 + payload->correlation_point_index,
2037 + actor,
2038 + protocol,
2039 + address_space,
2040 + ip,
2041 + port);
2042 +}
2043 +
2044 +static void topology_v1_free(NV_TOPOLOGY_V1_PAYLOAD *payload) {
2045 + if(!payload)
2046 + return;
2047 +
2048 + freez(payload->actors);
2049 + freez(payload->links);
2050 + freez(payload->evidence);
2051 + freez(payload->connections);
2052 + freez(payload->ports);
2053 + freez(payload->correlation_points);
2054 + freez(payload->correlation_claims);
2055 + freez(payload->labels);
2056 + if(payload->actor_index)
2057 + dictionary_destroy(payload->actor_index);
2058 + if(payload->graph_link_index)
2059 + dictionary_destroy(payload->graph_link_index);
2060 + if(payload->connection_index)
2061 + dictionary_destroy(payload->connection_index);
2062 + if(payload->port_index)
2063 + dictionary_destroy(payload->port_index);
2064 + if(payload->correlation_point_index)
2065 + dictionary_destroy(payload->correlation_point_index);
2066 + if(payload->correlation_claim_index)
2067 + dictionary_destroy(payload->correlation_claim_index);
2068 + *payload = (NV_TOPOLOGY_V1_PAYLOAD){ 0 };
2069 +}
2070 +
2071 +static bool topology_v1_should_emit_endpoint_actor(
2072 + const NV_TOPOLOGY_CONTEXT *ctx,
2073 + const NV_REMOTE_ACTOR *actor) {
2074 + if(!actor || !actor->ip[0])
2075 + return false;
2076 +
2077 + return !topology_ip_belongs_to_self(ctx, actor->ip, actor->address_space);
2078 +}
2079 +
2080 +static void topology_v1_collect_actors(
2081 + const NV_TOPOLOGY_CONTEXT *ctx,
2082 + NV_TOPOLOGY_RENDER_STATE *state,
2083 + NV_TOPOLOGY_V1_PAYLOAD *payload) {
2084 + NV_TOPOLOGY_V1_ACTOR *self = topology_v1_add_actor(payload, state->host_actor_id);
2085 + topology_v1_strncpy(self->type, sizeof(self->type), "self");
2086 + topology_v1_strncpy(self->machine_guid, sizeof(self->machine_guid), ctx->machine_guid);
2087 + topology_v1_strncpy(self->hostname, sizeof(self->hostname), ctx->hostname);
2088 + topology_v1_strncpy(self->display_name, sizeof(self->display_name), ctx->hostname);
2089 + self->sockets = ctx->sockets_total;
2090 + self->local_ip_count = state->local_ip_count;
2091 + self->has_local_ip_count = true;
2092 + uint64_t self_index = payload->actors_used - 1;
2093 + topology_v1_add_actor_label(payload, self_index, "display_name", self->display_name, "attribute");
2094 + topology_v1_add_actor_label(payload, self_index, "type", self->type, "metadata");
2095 + topology_v1_add_actor_label(payload, self_index, "hostname", self->hostname, "identity");
2096 + topology_v1_add_actor_label(payload, self_index, "machine_guid", self->machine_guid, "identity");
2097 + topology_v1_add_actor_label_uint(payload, self_index, "socket_count", self->sockets, "metric");
2098 + topology_v1_add_actor_label_uint(payload, self_index, "local_ip_count", self->local_ip_count, "metric");
2099 +
2100 + NV_PROCESS_ACTOR *pa;
2101 + dfe_start_read(ctx->process_actors, pa) {
2102 + char actor_id[NV_TOPOLOGY_KEY_MAX];
2103 + char display_name[NV_TOPOLOGY_KEY_MAX];
2104 + topology_actor_id_for_process(ctx, pa->pid, pa->uid, pa->net_ns_inode, pa->process, actor_id, sizeof(actor_id));
2105 + topology_process_display_name(ctx, pa->process, pa->pid, display_name, sizeof(display_name));
2106 +
2107 + NV_TOPOLOGY_V1_ACTOR *actor = topology_v1_add_actor(payload, actor_id);
2108 + topology_v1_strncpy(actor->type, sizeof(actor->type), "process");
2109 + topology_v1_strncpy(actor->machine_guid, sizeof(actor->machine_guid), ctx->machine_guid);
2110 + topology_v1_strncpy(actor->hostname, sizeof(actor->hostname), ctx->hostname);
2111 + topology_v1_strncpy(actor->process, sizeof(actor->process), pa->process);
2112 + topology_v1_strncpy(actor->username, sizeof(actor->username), pa->username);
2113 + topology_v1_strncpy(actor->namespace_type, sizeof(actor->namespace_type), pa->namespace_type);
2114 + topology_v1_strncpy(actor->local_ip, sizeof(actor->local_ip), pa->local_ip);
2115 + topology_v1_strncpy(actor->local_address_space, sizeof(actor->local_address_space), pa->local_address_space);
2116 + topology_v1_strncpy(actor->display_name, sizeof(actor->display_name), display_name);
2117 + topology_v1_strncpy(actor->cmdline, sizeof(actor->cmdline), pa->cmdline);
2118 + actor->pid = (uint64_t)pa->pid;
2119 + actor->ppid = (uint64_t)pa->ppid;
2120 + actor->uid = (uint64_t)pa->uid;
2121 + actor->net_ns_inode = pa->net_ns_inode;
2122 + actor->sockets = pa->sockets;
2123 + actor->has_pid = ctx->options.processes_by_pid;
2124 + actor->has_ppid = ctx->options.processes_by_pid;
2125 + actor->has_uid = ctx->options.processes_by_pid;
2126 + actor->has_net_ns_inode = ctx->options.processes_by_pid;
2127 +
2128 + uint64_t actor_index = payload->actors_used - 1;
2129 + topology_v1_add_actor_label(payload, actor_index, "display_name", actor->display_name, "attribute");
2130 + topology_v1_add_actor_label(payload, actor_index, "type", actor->type, "metadata");
2131 + topology_v1_add_actor_label(payload, actor_index, "process", actor->process, "identity");
2132 + topology_v1_add_actor_label(payload, actor_index, "username", actor->username, "attribute");
2133 + topology_v1_add_actor_label(payload, actor_index, "cmdline", actor->cmdline, "attribute");
2134 + topology_v1_add_actor_label(payload, actor_index, "namespace_type", actor->namespace_type, "attribute");
2135 + topology_v1_add_actor_label(payload, actor_index, "local_ip", actor->local_ip, "attribute");
2136 + topology_v1_add_actor_label(payload, actor_index, "local_address_space", actor->local_address_space, "attribute");
2137 + topology_v1_add_actor_label_uint(payload, actor_index, "socket_count", actor->sockets, "metric");
2138 + if(actor->has_pid)
2139 + topology_v1_add_actor_label_uint(payload, actor_index, "pid", actor->pid, "identity");
2140 + if(actor->has_uid)
2141 + topology_v1_add_actor_label_uint(payload, actor_index, "uid", actor->uid, "attribute");
2142 + if(actor->has_net_ns_inode)
2143 + topology_v1_add_actor_label_uint(payload, actor_index, "net_ns_inode", actor->net_ns_inode, "identity");
2144 + }
2145 + dfe_done(pa);
2146 +
2147 + NV_REMOTE_ACTOR *ra;
2148 + dfe_start_read(ctx->remote_actors, ra) {
2149 + // Remote actors are discovered while sockets are scanned, before all
2150 + // local IPs may be known. Recheck with the final local-IP set so actor
2151 + // emission and link destination resolution use the same self test.
2152 + if(!topology_v1_should_emit_endpoint_actor(ctx, ra))
2153 + continue;
2154 +
2155 + char actor_id[NV_TOPOLOGY_KEY_MAX];
2156 + topology_actor_id_for_remote_endpoint(ctx, ra->ip, ra->address_space, actor_id, sizeof(actor_id));
2157 +
2158 + NV_TOPOLOGY_V1_ACTOR *actor = topology_v1_add_actor(payload, actor_id);
2159 + topology_v1_strncpy(actor->type, sizeof(actor->type), "endpoint");
2160 + topology_v1_strncpy(actor->ip, sizeof(actor->ip), ra->ip);
2161 + topology_v1_strncpy(actor->address_space, sizeof(actor->address_space), ra->address_space);
2162 + topology_v1_strncpy(actor->display_name, sizeof(actor->display_name), ra->ip);
2163 + actor->sockets = ra->sockets;
2164 + uint64_t actor_index = payload->actors_used - 1;
2165 + topology_v1_add_actor_label(payload, actor_index, "display_name", actor->display_name, "attribute");
2166 + topology_v1_add_actor_label(payload, actor_index, "type", actor->type, "metadata");
2167 + topology_v1_add_actor_label(payload, actor_index, "ip", actor->ip, "identity");
2168 + topology_v1_add_actor_label(payload, actor_index, "address_space", actor->address_space, "attribute");
2169 + topology_v1_add_actor_label_uint(payload, actor_index, "socket_count", actor->sockets, "metric");
2170 + state->endpoint_actor_count++;
2171 + }
2172 + dfe_done(ra);
2173 +}
2174 +
2175 +static bool topology_v1_process_actor_index(
2176 + const NV_TOPOLOGY_CONTEXT *ctx,
2177 + NV_TOPOLOGY_V1_PAYLOAD *payload,
2178 + uint64_t pid,
2179 + uint64_t uid,
2180 + uint64_t net_ns_inode,
2181 + const char *process,
2182 + uint64_t *index) {
2183 + char actor_id[NV_TOPOLOGY_KEY_MAX];
2184 + topology_actor_id_for_process(ctx, pid, uid, net_ns_inode, process, actor_id, sizeof(actor_id));
2185 + return topology_v1_actor_index_get(payload, actor_id, index);
2186 +}
2187 +
2188 +static bool topology_v1_endpoint_actor_index(
2189 + const NV_TOPOLOGY_CONTEXT *ctx,
2190 + NV_TOPOLOGY_V1_PAYLOAD *payload,
2191 + const char *ip,
2192 + const char *address_space,
2193 + uint64_t *index) {
2194 + char actor_id[NV_TOPOLOGY_KEY_MAX];
2195 + topology_actor_id_for_remote_endpoint(ctx, ip, address_space, actor_id, sizeof(actor_id));
2196 + return topology_v1_actor_index_get(payload, actor_id, index);
2197 +}
2198 +
2199 +static uint64_t topology_v1_graph_link_get_or_add(
2200 + NV_TOPOLOGY_V1_PAYLOAD *payload,
2201 + uint64_t src_actor,
2202 + uint64_t dst_actor,
2203 + const char *type,
2204 + const char *protocol,
2205 + const char *state) {
2206 + char key[NV_TOPOLOGY_KEY_MAX];
2207 + snprintfz(key, sizeof(key), "%"PRIu64"|%"PRIu64"|%s|%s|%s",
2208 + src_actor, dst_actor,
2209 + type ? type : "",
2210 + protocol ? protocol : "",
2211 + state ? state : "");
2212 +
2213 + uint64_t *stored = dictionary_get(payload->graph_link_index, key);
2214 + if(stored)
2215 + return *stored;
2216 +
2217 + uint64_t index = payload->links_used;
2218 + NV_TOPOLOGY_V1_GRAPH_LINK *link = topology_v1_add_graph_link(payload);
2219 + link->src_actor = src_actor;
2220 + link->dst_actor = dst_actor;
2221 + topology_v1_strncpy(link->type, sizeof(link->type), type);
2222 + topology_v1_strncpy(link->protocol, sizeof(link->protocol), protocol);
2223 + topology_v1_strncpy(link->state, sizeof(link->state), state);
2224 + dictionary_set(payload->graph_link_index, key, &index, sizeof(index));
2225 + return index;
2226 +}
2227 +
2228 +static bool topology_v1_socket_peer_actor_index(
2229 + const NV_TOPOLOGY_CONTEXT *ctx,
2230 + NV_TOPOLOGY_V1_PAYLOAD *payload,
2231 + const NV_TOPOLOGY_LINK *link,
2232 + NV_TOPOLOGY_ENDPOINT_ROLE role,
2233 + uint64_t *actor,
2234 + bool *is_correlation_point) {
2235 + if(is_correlation_point)
2236 + *is_correlation_point = false;
2237 +
2238 + const char *ip = NULL;
2239 + const char *address_space = NULL;
2240 + uint16_t port = 0;
2241 +
2242 + if(role == NV_TOPOLOGY_ENDPOINT_ROLE_CLIENT) {
2243 + ip = link->client_ip;
2244 + address_space = link->client_address_space;
2245 + port = link->client_port;
2246 + }
2247 + else if(role == NV_TOPOLOGY_ENDPOINT_ROLE_SERVER) {
2248 + ip = link->server_ip;
2249 + address_space = link->server_address_space;
2250 + port = link->server_port;
2251 + }
2252 + else
2253 + return false;
2254 +
2255 + bool endpoint_is_self = topology_ip_belongs_to_self(ctx, ip, address_space);
2256 + if(endpoint_is_self) {
2257 + NV_ENDPOINT_OWNER *owner = topology_lookup_endpoint_owner(ctx, link->net_ns_inode, link->protocol_id, ip, port, true);
2258 + if(owner) {
2259 + return topology_v1_process_actor_index(ctx, payload,
2260 + owner->pid, owner->uid, owner->net_ns_inode,
2261 + owner->process, actor);
2262 + }
2263 +
2264 + return false;
2265 + }
2266 +
2267 + if(is_correlation_point)
2268 + *is_correlation_point = true;
2269 + return topology_v1_endpoint_actor_index(ctx, payload, ip, address_space, actor);
2270 +}
2271 +
2272 +static void topology_v1_collect_links(
2273 + const NV_TOPOLOGY_CONTEXT *ctx,
2274 + NV_TOPOLOGY_RENDER_STATE *state,
2275 + NV_TOPOLOGY_V1_PAYLOAD *payload) {
2276 + uint64_t host_actor = 0;
2277 + topology_v1_actor_index_get(payload, state->host_actor_id, &host_actor);
2278 +
2279 + NV_PROCESS_ACTOR *pa;
2280 + dfe_start_read(ctx->process_actors, pa) {
2281 + uint64_t process_actor;
2282 + if(!topology_v1_process_actor_index(ctx, payload, pa->pid, pa->uid, pa->net_ns_inode, pa->process, &process_actor))
2283 + continue;
2284 +
2285 + uint64_t link_index = topology_v1_graph_link_get_or_add(
2286 + payload, host_actor, process_actor, "ownership", "ownership", "active");
2287 + NV_TOPOLOGY_V1_GRAPH_LINK *link = &payload->links[link_index];
2288 + link->socket_count += pa->sockets;
2289 + state->ownership_link_count++;
2290 + }
2291 + dfe_done(pa);
2292 +
2293 + NV_TOPOLOGY_LINK *source;
2294 + dfe_start_read(ctx->links, source) {
2295 + uint64_t src_actor;
2296 + if(!topology_v1_process_actor_index(ctx, payload,
2297 + source->pid, source->uid, source->net_ns_inode,
2298 + source->process, &src_actor))
2299 + continue;
2300 +
2301 + uint64_t client_actor = 0;
2302 + uint64_t server_actor = 0;
2303 + bool client_is_correlation_point = false;
2304 + bool server_is_correlation_point = false;
2305 +
2306 + if(source->process_is_client) {
2307 + client_actor = src_actor;
2308 + if(!topology_v1_socket_peer_actor_index(
2309 + ctx, payload, source, NV_TOPOLOGY_ENDPOINT_ROLE_SERVER, &server_actor, &server_is_correlation_point))
2310 + continue;
2311 + }
2312 + else {
2313 + server_actor = src_actor;
2314 + if(!topology_v1_socket_peer_actor_index(
2315 + ctx, payload, source, NV_TOPOLOGY_ENDPOINT_ROLE_CLIENT, &client_actor, &client_is_correlation_point))
2316 + continue;
2317 + }
2318 +
2319 + if(!client_actor || !server_actor)
2320 + continue;
2321 +
2322 + if(source->process_is_client) {
2323 + topology_v1_add_correlation_claim_endpoint(
2324 + payload, client_actor, source->protocol, source->client_address_space, source->client_ip, source->client_port);
2325 + if(server_is_correlation_point)
2326 + topology_v1_add_correlation_point_endpoint(
2327 + payload, server_actor, source->protocol, source->server_address_space, source->server_ip, source->server_port);
2328 + }
2329 + else {
2330 + topology_v1_add_correlation_claim_endpoint(
2331 + payload, server_actor, source->protocol, source->server_address_space, source->server_ip, source->server_port);
2332 + if(client_is_correlation_point)
2333 + topology_v1_add_correlation_point_endpoint(
2334 + payload, client_actor, source->protocol, source->client_address_space, source->client_ip, source->client_port);
2335 + }
2336 +
2337 + topology_v1_add_port_row(
2338 + payload, src_actor, source->protocol, source->process_port, source->sockets);
2339 + if(!source->process_is_client && !client_is_correlation_point)
2340 + topology_v1_add_port_row(
2341 + payload, client_actor, source->protocol, source->client_port, source->sockets);
2342 + else if(source->process_is_client && !server_is_correlation_point)
2343 + topology_v1_add_port_row(
2344 + payload, server_actor, source->protocol, source->server_port, source->sockets);
2345 +
2346 + uint64_t link_index = topology_v1_graph_link_get_or_add(
2347 + payload, client_actor, server_actor,
2348 + (client_is_correlation_point || server_is_correlation_point) ? "endpoint_socket" : "socket",
2349 + source->protocol, source->state);
2350 + NV_TOPOLOGY_V1_GRAPH_LINK *link = &payload->links[link_index];
2351 + link->evidence_count++;
2352 + link->socket_count += source->sockets;
2353 + link->retransmissions += source->retransmissions;
2354 + if(link->max_rtt_usec < source->max_rtt_usec)
2355 + link->max_rtt_usec = source->max_rtt_usec;
2356 + if(link->max_rcv_rtt_usec < source->max_rcv_rtt_usec)
2357 + link->max_rcv_rtt_usec = source->max_rcv_rtt_usec;
2358 +
2359 + if(ctx->options.detailed) {
2360 + NV_TOPOLOGY_V1_SOCKET_EVIDENCE *row = topology_v1_add_socket_evidence(payload);
2361 + row->link = link_index;
2362 + row->src_actor = client_actor;
2363 + row->dst_actor = server_actor;
2364 + row->source = source;
2365 + }
2366 + else
2367 + topology_v1_add_connection_row(payload, client_actor, server_actor, source);
2368 + }
2369 + dfe_done(source);
2370 +}
2371 +
2372 +static void topology_v1_emit_column(
2373 + BUFFER *wb,
2374 + const char *id,
2375 + const char *type,
2376 + const char *role,
2377 + bool nullable,
2378 + const char *aggregation) {
2379 + buffer_json_add_array_item_object(wb);
2380 + buffer_json_member_add_string(wb, "id", id);
2381 + buffer_json_member_add_string(wb, "type", type);
2382 + if(nullable)
2383 + buffer_json_member_add_boolean(wb, "nullable", true);
2384 + if(role)
2385 + buffer_json_member_add_string(wb, "role", role);
2386 + if(aggregation)
2387 + buffer_json_member_add_string(wb, "aggregation", aggregation);
2388 + buffer_json_object_close(wb);
2389 +}
2390 +
2391 +static void topology_v1_values_start(BUFFER *wb) {
2392 + buffer_json_add_array_item_object(wb);
2393 + buffer_json_member_add_string(wb, "codec", "values");
2394 + buffer_json_member_add_array(wb, "values");
2395 +}
2396 +
2397 +static void topology_v1_const_string(BUFFER *wb, const char *value) {
2398 + buffer_json_add_array_item_object(wb);
2399 + buffer_json_member_add_string(wb, "codec", "const");
2400 + buffer_json_member_add_string(wb, "value", value);
2401 + buffer_json_object_close(wb);
2402 +}
2403 +
2404 +static void topology_v1_values_end(BUFFER *wb) {
2405 + buffer_json_array_close(wb);
2406 + buffer_json_object_close(wb);
2407 +}
2408 +
2409 +static void topology_v1_add_nullable_uint(BUFFER *wb, bool has_value, uint64_t value) {
2410 + if(has_value)
2411 + buffer_json_add_array_item_uint64(wb, value);
2412 + else
2413 + buffer_json_add_array_item_string(wb, NULL);
2414 +}
2415 +
2416 +static size_t topology_v1_uint_json_size(uint64_t value) {
2417 + size_t size = 1;
2418 + while(value >= 10) {
2419 + value /= 10;
2420 + size++;
2421 + }
2422 +
2423 + return size;
2424 +}
2425 +
2426 +static size_t topology_v1_string_json_size(const char *value) {
2427 + if(!value || !*value)
2428 + return 4; // null
2429 +
2430 + size_t size = 2; // quotes
2431 + for(const unsigned char *s = (const unsigned char *)value; *s; s++) {
2432 + if(*s == '"' || *s == '\\')
2433 + size += 2;
2434 + else if(*s <= 0x1f)
2435 + size += 6;
2436 + else
2437 + size++;
2438 + }
2439 +
2440 + return size;
2441 +}
2442 +
2443 +static void topology_v1_string_column_init(NV_TOPOLOGY_V1_STRING_COLUMN *column, size_t rows) {
2444 + *column = (NV_TOPOLOGY_V1_STRING_COLUMN){
2445 + .rows = rows,
2446 + .indexes = rows ? callocz(rows, sizeof(*column->indexes)) : NULL,
2447 + .index = dictionary_create_advanced(
2448 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2449 + NULL,
2450 + sizeof(uint64_t)),
2451 + };
2452 +}
2453 +
2454 +static void topology_v1_string_column_free(NV_TOPOLOGY_V1_STRING_COLUMN *column) {
2455 + if(!column)
2456 + return;
2457 +
2458 + freez(column->values);
2459 + freez(column->indexes);
2460 + if(column->index)
2461 + dictionary_destroy(column->index);
2462 + *column = (NV_TOPOLOGY_V1_STRING_COLUMN){ 0 };
2463 +}
2464 +
2465 +static void topology_v1_string_column_add(NV_TOPOLOGY_V1_STRING_COLUMN *column, const char *value) {
2466 + if(!column || column->rows_used >= column->rows)
2467 + return;
2468 +
2469 + const char *stored_value = (value && *value) ? value : NULL;
2470 + const char *key = stored_value ? stored_value : "\001";
2471 + uint64_t index = 0;
2472 + uint64_t *stored = column->index ? dictionary_get(column->index, key) : NULL;
2473 +
2474 + if(stored) {
2475 + index = *stored;
2476 + }
2477 + else {
2478 + index = column->values_used;
2479 + if(column->values_used == column->values_size) {
2480 + size_t new_size = column->values_size ? column->values_size * 2 : 16;
2481 + column->values = reallocz(column->values, new_size * sizeof(*column->values));
2482 + column->values_size = new_size;
2483 + }
2484 +
2485 + column->values[column->values_used++] = stored_value;
2486 + if(column->index)
2487 + dictionary_set(column->index, key, &index, sizeof(index));
2488 +
2489 + column->unique_json_size += topology_v1_string_json_size(stored_value);
2490 + if(column->values_used > 1)
2491 + column->unique_json_size++;
2492 + }
2493 +
2494 + column->indexes[column->rows_used] = index;
2495 + column->indexes_json_size += topology_v1_uint_json_size(index);
2496 + column->values_json_size += topology_v1_string_json_size(stored_value);
2497 + if(column->rows_used)
2498 + column->indexes_json_size++, column->values_json_size++;
2499 + column->rows_used++;
2500 +}
2501 +
2502 +static void topology_v1_emit_auto_string_column(BUFFER *wb, const NV_TOPOLOGY_V1_STRING_COLUMN *column) {
2503 + size_t values_encoding_size = sizeof("{\"codec\":\"values\",\"values\":[]}") - 1 + column->values_json_size;
2504 + size_t dict_encoding_size =
2505 + sizeof("{\"codec\":\"dict\",\"values\":[],\"indexes\":[]}") - 1 +
2506 + column->unique_json_size + column->indexes_json_size;
2507 + bool use_dict = column->rows > 0 && column->values_used < column->rows && dict_encoding_size < values_encoding_size;
2508 +
2509 + buffer_json_add_array_item_object(wb);
2510 + if(use_dict) {
2511 + buffer_json_member_add_string(wb, "codec", "dict");
2512 + buffer_json_member_add_array(wb, "values");
2513 + for(size_t i = 0; i < column->values_used; i++)
2514 + buffer_json_add_array_item_string(wb, column->values[i]);
2515 + buffer_json_array_close(wb);
2516 +
2517 + buffer_json_member_add_array(wb, "indexes");
2518 + for(size_t i = 0; i < column->rows_used; i++)
2519 + buffer_json_add_array_item_uint64(wb, column->indexes[i]);
2520 + buffer_json_array_close(wb);
2521 + }
2522 + else {
2523 + buffer_json_member_add_string(wb, "codec", "values");
2524 + buffer_json_member_add_array(wb, "values");
2525 + for(size_t i = 0; i < column->rows_used; i++)
2526 + buffer_json_add_array_item_string(wb, column->values[column->indexes[i]]);
2527 + buffer_json_array_close(wb);
2528 + }
2529 + buffer_json_object_close(wb);
2530 +}
2531 +
2532 +static void topology_v1_emit_actor_columns(BUFFER *wb) {
2533 + topology_v1_emit_column(wb, "id", "string", "identity", false, NULL);
2534 + topology_v1_emit_column(wb, "type", "string", "group_key", false, NULL);
2535 + topology_v1_emit_column(wb, "layer", "string", "group_key", false, NULL);
2536 + topology_v1_emit_column(wb, "machine_guid", "string", "merge_identity", true, NULL);
2537 + topology_v1_emit_column(wb, "hostname", "string", "merge_identity", true, NULL);
2538 + topology_v1_emit_column(wb, "process", "string", "group_key", true, NULL);
2539 + topology_v1_emit_column(wb, "username", "string", "attribute", true, NULL);
2540 + topology_v1_emit_column(wb, "cmdline", "string", "attribute", true, NULL);
2541 + topology_v1_emit_column(wb, "pid", "uint", "identity", true, NULL);
2542 + topology_v1_emit_column(wb, "ppid", "uint", "attribute", true, NULL);
2543 + topology_v1_emit_column(wb, "uid", "uint", "attribute", true, NULL);
2544 + topology_v1_emit_column(wb, "net_ns_inode", "uint", "identity", true, NULL);
2545 + topology_v1_emit_column(wb, "namespace_type", "string", "group_key", true, NULL);
2546 + topology_v1_emit_column(wb, "local_ip", "ip", "attribute", true, NULL);
2547 + topology_v1_emit_column(wb, "local_address_space", "string", "attribute", true, NULL);
2548 + topology_v1_emit_column(wb, "ip", "ip", "identity", true, NULL);
2549 + topology_v1_emit_column(wb, "address_space", "string", "group_key", true, NULL);
2550 + topology_v1_emit_column(wb, "display_name", "string", "attribute", true, NULL);
2551 + topology_v1_emit_column(wb, "socket_count", "uint", "metric", false, "sum");
2552 + topology_v1_emit_column(wb, "local_ip_count", "uint", "metric", true, "sum");
2553 +}
2554 +
2555 +static void topology_v1_emit_link_columns(BUFFER *wb) {
2556 + topology_v1_emit_column(wb, "src_actor", "actor_ref", "reference", false, NULL);
2557 + topology_v1_emit_column(wb, "dst_actor", "actor_ref", "reference", false, NULL);
2558 + topology_v1_emit_column(wb, "type", "string", "group_key", false, NULL);
2559 + topology_v1_emit_column(wb, "protocol", "string", "group_key", true, NULL);
2560 + topology_v1_emit_column(wb, "state", "string", "group_key", true, NULL);
2561 + topology_v1_emit_column(wb, "evidence_count", "uint", "metric", false, "sum");
2562 + topology_v1_emit_column(wb, "socket_count", "uint", "metric", false, "sum");
2563 + topology_v1_emit_column(wb, "retransmissions", "uint", "metric", false, "sum");
2564 + topology_v1_emit_column(wb, "rtt_ms_max", "float", "metric", false, "max");
2565 + topology_v1_emit_column(wb, "recv_rtt_ms_max", "float", "metric", false, "max");
2566 +}
2567 +
2568 +static void topology_v1_emit_socket_evidence_columns(BUFFER *wb) {
2569 + topology_v1_emit_column(wb, "link", "link_ref", "reference", false, NULL);
2570 + topology_v1_emit_column(wb, "src_actor", "actor_ref", "reference", false, NULL);
2571 + topology_v1_emit_column(wb, "dst_actor", "actor_ref", "reference", false, NULL);
2572 + topology_v1_emit_column(wb, "client_ip", "ip", "group_key", false, NULL);
2573 + topology_v1_emit_column(wb, "client_port", "uint", "group_key", false, NULL);
2574 + topology_v1_emit_column(wb, "server_ip", "ip", "group_key", false, NULL);
2575 + topology_v1_emit_column(wb, "server_port", "uint", "group_key", false, NULL);
2576 + topology_v1_emit_column(wb, "protocol", "string", "group_key", false, NULL);
2577 + topology_v1_emit_column(wb, "protocol_family", "string", "group_key", false, NULL);
2578 + topology_v1_emit_column(wb, "state", "string", "group_key", false, NULL);
2579 + topology_v1_emit_column(wb, "namespace_type", "string", "group_key", true, NULL);
2580 + topology_v1_emit_column(wb, "client_address_space", "string", "group_key", true, NULL);
2581 + topology_v1_emit_column(wb, "server_address_space", "string", "group_key", true, NULL);
2582 + topology_v1_emit_column(wb, "pid", "uint", "attribute", true, NULL);
2583 + topology_v1_emit_column(wb, "uid", "uint", "attribute", true, NULL);
2584 + topology_v1_emit_column(wb, "net_ns_inode", "uint", "attribute", true, NULL);
2585 + topology_v1_emit_column(wb, "process", "string", "attribute", true, NULL);
2586 + topology_v1_emit_column(wb, "socket_count", "uint", "metric", false, "sum");
2587 + topology_v1_emit_column(wb, "retransmissions", "uint", "metric", false, "sum");
2588 + topology_v1_emit_column(wb, "rtt_ms_max", "float", "metric", false, "max");
2589 + topology_v1_emit_column(wb, "recv_rtt_ms_max", "float", "metric", false, "max");
2590 +}
2591 +
2592 +static void topology_v1_emit_connection_columns(BUFFER *wb) {
2593 + topology_v1_emit_column(wb, "src_actor", "actor_ref", "reference", false, NULL);
2594 + topology_v1_emit_column(wb, "dst_actor", "actor_ref", "reference", false, NULL);
2595 + topology_v1_emit_column(wb, "client_ip", "ip", "group_key", false, NULL);
2596 + topology_v1_emit_column(wb, "server_ip", "ip", "group_key", false, NULL);
2597 + topology_v1_emit_column(wb, "protocol", "string", "group_key", false, NULL);
2598 + topology_v1_emit_column(wb, "state", "string", "group_key", false, NULL);
2599 + topology_v1_emit_column(wb, "socket_count", "uint", "metric", false, "sum");
2600 + topology_v1_emit_column(wb, "retransmissions", "uint", "metric", false, "sum");
2601 + topology_v1_emit_column(wb, "rtt_ms_max", "float", "metric", false, "max");
2602 + topology_v1_emit_column(wb, "recv_rtt_ms_max", "float", "metric", false, "max");
2603 +}
2604 +
2605 +static void topology_v1_emit_socket_port_columns(BUFFER *wb) {
2606 + topology_v1_emit_column(wb, "actor", "actor_ref", "reference", false, NULL);
2607 + topology_v1_emit_column(wb, "port", "uint", "group_key", false, NULL);
2608 + topology_v1_emit_column(wb, "protocol", "string", "group_key", false, NULL);
2609 + topology_v1_emit_column(wb, "socket_count", "uint", "metric", false, "sum");
2610 +}
2611 +
2612 +static void topology_v1_emit_actor_label_columns(BUFFER *wb) {
2613 + topology_v1_emit_column(wb, "actor", "actor_ref", "reference", false, NULL);
2614 + topology_v1_emit_column(wb, "key", "string", "attribute", false, NULL);
2615 + topology_v1_emit_column(wb, "value", "string", "attribute", false, NULL);
2616 + topology_v1_emit_column(wb, "source", "string", "attribute", true, NULL);
2617 + topology_v1_emit_column(wb, "kind", "string", "attribute", true, NULL);
2618 + topology_v1_emit_column(wb, "value_index", "uint", "attribute", true, NULL);
2619 +}
2620 +
2621 +static void topology_v1_emit_modal_direct_column_with_visibility(
2622 + BUFFER *wb,
2623 + const char *id,
2624 + const char *label,
2625 + const char *column,
2626 + const char *cell,
2627 + const char *visibility);
2628 +
2629 +static void topology_v1_emit_modal_direct_column(
2630 + BUFFER *wb,
2631 + const char *id,
2632 + const char *label,
2633 + const char *column,
2634 + const char *cell) {
2635 + topology_v1_emit_modal_direct_column_with_visibility(wb, id, label, column, cell, NULL);
2636 +}
2637 +
2638 +static void topology_v1_emit_modal_direct_column_with_visibility(
2639 + BUFFER *wb,
2640 + const char *id,
2641 + const char *label,
2642 + const char *column,
2643 + const char *cell,
2644 + const char *visibility) {
2645 + buffer_json_add_array_item_object(wb);
2646 + {
2647 + buffer_json_member_add_string(wb, "id", id);
2648 + buffer_json_member_add_string(wb, "label", label);
2649 + buffer_json_member_add_object(wb, "projection");
2650 + {
2651 + buffer_json_member_add_string(wb, "kind", "direct");
2652 + buffer_json_member_add_string(wb, "column", column);
2653 + }
2654 + buffer_json_object_close(wb);
2655 + buffer_json_member_add_string(wb, "cell", cell);
2656 + if(visibility)
2657 + buffer_json_member_add_string(wb, "visibility", visibility);
2658 + }
2659 + buffer_json_object_close(wb);
2660 +}
2661 +
2662 +static void topology_v1_emit_modal_opposite_actor_column_labeled(BUFFER *wb, const char *id, const char *label) {
2663 + buffer_json_add_array_item_object(wb);
2664 + {
2665 + buffer_json_member_add_string(wb, "id", id);
2666 + buffer_json_member_add_string(wb, "label", label);
2667 + buffer_json_member_add_object(wb, "projection");
2668 + {
2669 + buffer_json_member_add_string(wb, "kind", "opposite_actor");
2670 + buffer_json_member_add_string(wb, "src_actor_column", "src_actor");
2671 + buffer_json_member_add_string(wb, "dst_actor_column", "dst_actor");
2672 + }
2673 + buffer_json_object_close(wb);
2674 + buffer_json_member_add_string(wb, "cell", "actor_link");
2675 + }
2676 + buffer_json_object_close(wb);
2677 +}
2678 +
2679 +static void topology_v1_emit_modal_formatted_endpoint_column(
2680 + BUFFER *wb,
2681 + const char *id,
2682 + const char *label,
2683 + const char *ip_column,
2684 + const char *port_column) {
2685 + buffer_json_add_array_item_object(wb);
2686 + {
2687 + buffer_json_member_add_string(wb, "id", id);
2688 + buffer_json_member_add_string(wb, "label", label);
2689 + buffer_json_member_add_object(wb, "projection");
2690 + {
2691 + buffer_json_member_add_string(wb, "kind", "formatted_endpoint");
2692 + if(ip_column)
2693 + buffer_json_member_add_string(wb, "ip_column", ip_column);
2694 + if(port_column)
2695 + buffer_json_member_add_string(wb, "port_column", port_column);
2696 + buffer_json_member_add_string(wb, "protocol_column", "protocol");
2697 + }
2698 + buffer_json_object_close(wb);
2699 + buffer_json_member_add_string(wb, "cell", "endpoint");
2700 + }
2701 + buffer_json_object_close(wb);
2702 +}
2703 +
2704 +static void topology_v1_emit_network_connection_modal_section(
2705 + BUFFER *wb,
2706 + const char *id,
2707 + const char *label,
2708 + const char *selected_actor_column,
2709 + const char *peer_label,
2710 + const char *endpoint_label,
2711 + const char *endpoint_ip_column,
2712 + const char *endpoint_port_column,
2713 + bool detailed,
2714 + uint64_t order) {
2715 + buffer_json_add_array_item_object(wb);
2716 + {
2717 + buffer_json_member_add_string(wb, "id", id);
2718 + buffer_json_member_add_string(wb, "label", label);
2719 + buffer_json_member_add_uint64(wb, "order", order);
2720 + buffer_json_member_add_object(wb, "source");
2721 + {
2722 + buffer_json_member_add_string(wb, "kind", detailed ? "evidence" : "relationship_table");
2723 + if(detailed)
2724 + buffer_json_member_add_string(wb, "evidence", "socket");
2725 + else
2726 + buffer_json_member_add_string(wb, "table", "connections");
2727 + }
2728 + buffer_json_object_close(wb);
2729 + buffer_json_member_add_object(wb, "owner_filter");
2730 + {
2731 + buffer_json_member_add_string(wb, "mode", "actor_column");
2732 + buffer_json_member_add_string(wb, "actor_column", selected_actor_column);
2733 + }
2734 + buffer_json_object_close(wb);
2735 + buffer_json_member_add_array(wb, "columns");
2736 + {
2737 + topology_v1_emit_modal_opposite_actor_column_labeled(wb, "actor", peer_label);
2738 + topology_v1_emit_modal_formatted_endpoint_column(
2739 + wb, "endpoint", endpoint_label, endpoint_ip_column, detailed ? endpoint_port_column : NULL);
2740 + topology_v1_emit_modal_direct_column(wb, "protocol", "Protocol", "protocol", "badge");
2741 + topology_v1_emit_modal_direct_column(wb, "state", "State", "state", "badge");
2742 + topology_v1_emit_modal_direct_column(wb, "sockets", "Sockets", "socket_count", "number");
2743 + topology_v1_emit_modal_direct_column_with_visibility(
2744 + wb, "retransmissions", "Retransmissions", "retransmissions", "number", "expanded");
2745 + topology_v1_emit_modal_direct_column(wb, "rtt", "RTT max", "rtt_ms_max", "number");
2746 + topology_v1_emit_modal_direct_column_with_visibility(
2747 + wb, "recv_rtt", "Receiver RTT max", "recv_rtt_ms_max", "number", "expanded");
2748 + }
2749 + buffer_json_array_close(wb);
2750 + }
2751 + buffer_json_object_close(wb);
2752 +}
2753 +
2754 +static void topology_v1_emit_modal_identification_field(BUFFER *wb, const char *key, const char *label) {
2755 + buffer_json_add_array_item_object(wb);
2756 + {
2757 + buffer_json_member_add_string(wb, "key", key);
2758 + buffer_json_member_add_string(wb, "label", label);
2759 + buffer_json_member_add_uint64(wb, "max_values", 1);
2760 + }
2761 + buffer_json_object_close(wb);
2762 +}
2763 +
2764 +static void topology_v1_emit_modal_label_identification(BUFFER *wb, const char *actor_type) {
2765 + buffer_json_member_add_object(wb, "identification");
2766 + {
2767 + buffer_json_member_add_boolean(wb, "enabled", true);
2768 + buffer_json_member_add_array(wb, "fields");
2769 + {
2770 + if(strcmp(actor_type, "self") == 0) {
2771 + topology_v1_emit_modal_identification_field(wb, "hostname", "Hostname");
2772 + topology_v1_emit_modal_identification_field(wb, "socket_count", "Sockets");
2773 + topology_v1_emit_modal_identification_field(wb, "local_ip_count", "Local IPs");
2774 + }
2775 + else if(strcmp(actor_type, "process") == 0) {
2776 + topology_v1_emit_modal_identification_field(wb, "process", "Process");
2777 + topology_v1_emit_modal_identification_field(wb, "username", "User");
2778 + topology_v1_emit_modal_identification_field(wb, "namespace_type", "Namespace");
2779 + topology_v1_emit_modal_identification_field(wb, "local_ip", "Local IP");
2780 + topology_v1_emit_modal_identification_field(wb, "cmdline", "Command");
2781 + topology_v1_emit_modal_identification_field(wb, "socket_count", "Sockets");
2782 + }
2783 + else if(strcmp(actor_type, "endpoint") == 0) {
2784 + topology_v1_emit_modal_identification_field(wb, "ip", "IP");
2785 + topology_v1_emit_modal_identification_field(wb, "address_space", "Address Space");
2786 + topology_v1_emit_modal_identification_field(wb, "socket_count", "Sockets");
2787 + }
2788 + }
2789 + buffer_json_array_close(wb);
2790 + }
2791 + buffer_json_object_close(wb);
2792 +}
2793 +
2794 +static void topology_v1_emit_actor_type(
2795 + BUFFER *wb,
2796 + const char *id,
2797 + const char *merge_a,
2798 + const char *merge_b,
2799 + const char *scope,
2800 + const char *label,
2801 + const char *color_slot,
2802 + const char *icon,
2803 + const char *role,
2804 + bool border,
2805 + const char *size_mode,
2806 + const char *size_metric_column,
2807 + const char *size_scale,
2808 + const char *layout_repulsion,
2809 + bool show_port_bullets,
2810 + const char *port_table,
2811 + bool detailed,
2812 + const char *label_column_a,
2813 + const char *label_column_b) {
2814 + bool is_self = strcmp(id, "self") == 0;
2815 +
2816 + buffer_json_member_add_object(wb, id);
2817 + {
2818 + buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2819 + buffer_json_member_add_array(wb, "identity");
2820 + buffer_json_add_array_item_string(wb, "id");
2821 + buffer_json_array_close(wb);
2822 + buffer_json_member_add_array(wb, "merge_identity");
2823 + if(merge_a)
2824 + buffer_json_add_array_item_string(wb, merge_a);
2825 + if(merge_b)
2826 + buffer_json_add_array_item_string(wb, merge_b);
2827 + buffer_json_array_close(wb);
2828 + buffer_json_member_add_array(wb, "aggregation_scopes");
2829 + buffer_json_add_array_item_string(wb, scope);
2830 + buffer_json_array_close(wb);
2831 + buffer_json_member_add_object(wb, "search");
2832 + {
2833 + buffer_json_member_add_array(wb, "columns");
2834 + if(strcmp(id, "self") == 0) {
2835 + buffer_json_add_array_item_string(wb, "display_name");
2836 + buffer_json_add_array_item_string(wb, "hostname");
2837 + }
2838 + else if(strcmp(id, "process") == 0) {
2839 + buffer_json_add_array_item_string(wb, "display_name");
2840 + buffer_json_add_array_item_string(wb, "process");
2841 + buffer_json_add_array_item_string(wb, "username");
2842 + buffer_json_add_array_item_string(wb, "cmdline");
2843 + buffer_json_add_array_item_string(wb, "local_ip");
2844 + }
2845 + else if(strcmp(id, "endpoint") == 0) {
2846 + buffer_json_add_array_item_string(wb, "display_name");
2847 + buffer_json_add_array_item_string(wb, "ip");
2848 + }
2849 + buffer_json_array_close(wb);
2850 + }
2851 + buffer_json_object_close(wb);
2852 + buffer_json_member_add_object(wb, "presentation");
2853 + {
2854 + buffer_json_member_add_string(wb, "label", label);
2855 + buffer_json_member_add_string(wb, "role", role);
2856 + buffer_json_member_add_string(wb, "icon", icon);
2857 + buffer_json_member_add_string(wb, "color_slot", color_slot);
2858 + buffer_json_member_add_object(wb, "border");
2859 + {
2860 + buffer_json_member_add_boolean(wb, "enabled", border);
2861 + }
2862 + buffer_json_object_close(wb);
2863 + buffer_json_member_add_object(wb, "size");
2864 + {
2865 + buffer_json_member_add_string(wb, "mode", size_mode ? size_mode : "fixed");
2866 + if(size_metric_column)
2867 + buffer_json_member_add_string(wb, "metric_column", size_metric_column);
2868 + if(size_scale)
2869 + buffer_json_member_add_string(wb, "scale", size_scale);
2870 + }
2871 + buffer_json_object_close(wb);
2872 + if(layout_repulsion) {
2873 + buffer_json_member_add_object(wb, "layout");
2874 + {
2875 + buffer_json_member_add_string(wb, "repulsion", layout_repulsion);
2876 + }
2877 + buffer_json_object_close(wb);
2878 + }
2879 + buffer_json_member_add_object(wb, "label_policy");
2880 + {
2881 + buffer_json_member_add_array(wb, "columns");
2882 + if(label_column_a)
2883 + buffer_json_add_array_item_string(wb, label_column_a);
2884 + if(label_column_b)
2885 + buffer_json_add_array_item_string(wb, label_column_b);
2886 + buffer_json_array_close(wb);
2887 + buffer_json_member_add_string(wb, "fallback", "type_label");
2888 + buffer_json_member_add_uint64(wb, "max_length", 80);
2889 + buffer_json_member_add_string(wb, "array", "reject");
2890 + }
2891 + buffer_json_object_close(wb);
2892 + buffer_json_member_add_object(wb, "ports");
2893 + {
2894 + buffer_json_member_add_boolean(wb, "show_bullets", show_port_bullets);
2895 + if(show_port_bullets) {
2896 buffer_json_member_add_array(wb, "sources");
1863 - buffer_json_add_array_item_string(wb, "labels.user");
2897 + {
2898 + buffer_json_add_array_item_object(wb);
2899 + buffer_json_member_add_string(wb, "source", "actor_table");
2900 + buffer_json_member_add_string(wb, "table", port_table ? port_table : "socket_ports");
2901 + buffer_json_member_add_string(wb, "actor_column", "actor");
2902 + buffer_json_member_add_string(wb, "name_column", "port");
2903 + buffer_json_member_add_string(wb, "value_column", "socket_count");
2904 + buffer_json_member_add_string(wb, "default_type", "topology");
2905 + buffer_json_object_close(wb);
2906 + }
2907 buffer_json_array_close(wb);
1865 - buffer_json_object_close(wb);
2908 }
1867 - buffer_json_array_close(wb);
1868 -
1869 - buffer_json_member_add_object(wb, "tables");
2909 + }
2910 + buffer_json_object_close(wb);
2911 + buffer_json_member_add_object(wb, "modal");
2912 + {
2913 + buffer_json_member_add_object(wb, "labels");
2914 {
1871 - buffer_json_member_add_object(wb, "sockets");
1872 - {
1873 - buffer_json_member_add_string(wb, "label", "Sockets");
1874 - buffer_json_member_add_string(wb, "source", "data");
1875 - buffer_json_member_add_boolean(wb, "bullet_source", true);
1876 - buffer_json_member_add_uint64(wb, "order", 0);
1877 - buffer_json_member_add_array(wb, "columns");
2915 + buffer_json_member_add_string(wb, "table", "actor_labels");
2916 + topology_v1_emit_modal_label_identification(wb, id);
2917 + }
2918 + buffer_json_object_close(wb);
2919 + buffer_json_member_add_object(wb, "mini_topology");
2920 + {
2921 + buffer_json_member_add_uint64(wb, "depth", 1);
2922 + buffer_json_member_add_array(wb, "exclude_link_types");
2923 + if(!is_self)
2924 + buffer_json_add_array_item_string(wb, "ownership");
2925 + buffer_json_array_close(wb);
2926 + }
2927 + buffer_json_object_close(wb);
2928 + buffer_json_member_add_array(wb, "sections");
2929 + {
2930 + if(is_self) {
2931 + buffer_json_add_array_item_object(wb);
2932 {
1879 - buffer_json_add_array_item_object(wb);
1880 - buffer_json_member_add_string(wb, "key", "remote");
1881 - buffer_json_member_add_string(wb, "label", "Remote");
1882 - buffer_json_object_close(wb);
1883 -
1884 - buffer_json_add_array_item_object(wb);
1885 - buffer_json_member_add_string(wb, "key", "protocol");
1886 - buffer_json_member_add_string(wb, "label", "Protocol");
1887 - buffer_json_member_add_string(wb, "type", "badge");
1888 - buffer_json_object_close(wb);
1889 -
1890 - buffer_json_add_array_item_object(wb);
1891 - buffer_json_member_add_string(wb, "key", "direction");
1892 - buffer_json_member_add_string(wb, "label", "Direction");
1893 - buffer_json_member_add_string(wb, "type", "badge");
2933 + buffer_json_member_add_string(wb, "id", "processes");
2934 + buffer_json_member_add_string(wb, "label", "Processes");
2935 + buffer_json_member_add_uint64(wb, "order", 1);
2936 + buffer_json_member_add_object(wb, "source");
2937 + {
2938 + buffer_json_member_add_string(wb, "kind", "links");
2939 + }
2940 buffer_json_object_close(wb);
1895 -
1896 - buffer_json_add_array_item_object(wb);
1897 - buffer_json_member_add_string(wb, "key", "state");
1898 - buffer_json_member_add_string(wb, "label", "State");
1899 - buffer_json_member_add_string(wb, "type", "badge");
2941 + buffer_json_member_add_object(wb, "owner_filter");
2942 + {
2943 + buffer_json_member_add_string(wb, "mode", "incident_link");
2944 + buffer_json_member_add_string(wb, "src_actor_column", "src_actor");
2945 + buffer_json_member_add_string(wb, "dst_actor_column", "dst_actor");
2946 + }
2947 buffer_json_object_close(wb);
2948 + buffer_json_member_add_array(wb, "row_filters");
2949 + {
2950 + buffer_json_add_array_item_object(wb);
2951 + {
2952 + buffer_json_member_add_string(wb, "column", "type");
2953 + buffer_json_member_add_string(wb, "op", "eq");
2954 + buffer_json_member_add_string(wb, "value", "ownership");
2955 + }
2956 + buffer_json_object_close(wb);
2957 + }
2958 + buffer_json_array_close(wb);
2959 + buffer_json_member_add_array(wb, "columns");
2960 + {
2961 + topology_v1_emit_modal_opposite_actor_column_labeled(wb, "process", "Process");
2962 + topology_v1_emit_modal_direct_column(wb, "sockets", "Sockets", "socket_count", "number");
2963 + topology_v1_emit_modal_direct_column_with_visibility(
2964 + wb, "evidence", "Evidence", "evidence_count", "number", "expanded");
2965 + }
2966 + buffer_json_array_close(wb);
2967 }
1902 - buffer_json_array_close(wb);
2968 + buffer_json_object_close(wb);
2969 }
1904 - buffer_json_object_close(wb);
1905 -
1906 - buffer_json_member_add_object(wb, "links");
1907 - {
1908 - buffer_json_member_add_string(wb, "label", "Connections");
1909 - buffer_json_member_add_string(wb, "source", "links");
1910 - buffer_json_member_add_uint64(wb, "order", 1);
1911 - buffer_json_member_add_array(wb, "columns");
1912 - {
1913 - buffer_json_add_array_item_object(wb);
1914 - buffer_json_member_add_string(wb, "key", "remoteLabel");
1915 - buffer_json_member_add_string(wb, "label", "Remote");
1916 - buffer_json_member_add_string(wb, "type", "actor_link");
1917 - buffer_json_object_close(wb);
2970 + else {
2971 + topology_v1_emit_network_connection_modal_section(
2972 + wb,
2973 + detailed ? "dependency_sockets" : "dependencies",
2974 + "Dependencies",
2975 + "src_actor",
2976 + "Service",
2977 + "Server",
2978 + "server_ip",
2979 + "server_port",
2980 + detailed,
2981 + 1);
2982 + topology_v1_emit_network_connection_modal_section(
2983 + wb,
2984 + detailed ? "dependant_sockets" : "dependants",
2985 + "Dependants",
2986 + "dst_actor",
2987 + "Client",
2988 + "Client",
2989 + "client_ip",
2990 + "client_port",
2991 + detailed,
2992 + 2);
2993 + }
2994 + }
2995 + buffer_json_array_close(wb);
2996 + }
2997 + buffer_json_object_close(wb);
2998
1919 - buffer_json_add_array_item_object(wb);
1920 - buffer_json_member_add_string(wb, "key", "protocol");
1921 - buffer_json_member_add_string(wb, "label", "Protocol");
1922 - buffer_json_object_close(wb);
2999 + }
3000 + buffer_json_object_close(wb);
3001
1924 - buffer_json_add_array_item_object(wb);
1925 - buffer_json_member_add_string(wb, "key", "direction");
1926 - buffer_json_member_add_string(wb, "label", "Direction");
1927 - buffer_json_object_close(wb);
3002 + }
3003 + buffer_json_object_close(wb);
3004 +}
3005
1929 - buffer_json_add_array_item_object(wb);
1930 - buffer_json_member_add_string(wb, "key", "state");
1931 - buffer_json_member_add_string(wb, "label", "State");
1932 - buffer_json_object_close(wb);
1933 - }
1934 - buffer_json_array_close(wb);
1935 - }
1936 - buffer_json_object_close(wb);
3006 +static void topology_v1_emit_link_type(
3007 + BUFFER *wb,
3008 + const char *id,
3009 + const char *orientation,
3010 + const char *direction_role,
3011 + const char *semantic_role,
3012 + const char *evidence_type,
3013 + const char *label,
3014 + const char *color_slot,
3015 + const char *line_style,
3016 + const char *width,
3017 + const char *arrow,
3018 + const char *opacity,
3019 + const char *scale_key,
3020 + const char *value_column,
3021 + const char *layout_strength,
3022 + const char *layout_distance) {
3023 + buffer_json_member_add_object(wb, id);
3024 + {
3025 + buffer_json_member_add_string(wb, "orientation", orientation);
3026 + buffer_json_member_add_string(wb, "direction_role", direction_role);
3027 + if(semantic_role)
3028 + buffer_json_member_add_string(wb, "semantic_role", semantic_role);
3029 + buffer_json_member_add_object(wb, "aggregation");
3030 + {
3031 + buffer_json_member_add_string(wb, "direction", "preserve");
3032 + buffer_json_member_add_string(wb, "evidence", evidence_type ? "append" : "count");
3033 + buffer_json_member_add_object(wb, "metrics");
3034 + {
3035 + buffer_json_member_add_string(wb, "evidence_count", "sum");
3036 + buffer_json_member_add_string(wb, "socket_count", "sum");
3037 + if(evidence_type) {
3038 + buffer_json_member_add_string(wb, "retransmissions", "sum");
3039 + buffer_json_member_add_string(wb, "rtt_ms_max", "max");
3040 + buffer_json_member_add_string(wb, "recv_rtt_ms_max", "max");
3041 + }
3042 + }
3043 + buffer_json_object_close(wb);
3044 + }
3045 + buffer_json_object_close(wb);
3046 + if(evidence_type) {
3047 + buffer_json_member_add_array(wb, "evidence_types");
3048 + buffer_json_add_array_item_string(wb, evidence_type);
3049 + buffer_json_array_close(wb);
3050 + }
3051 + buffer_json_member_add_object(wb, "presentation");
3052 + {
3053 + buffer_json_member_add_string(wb, "label", label);
3054 + buffer_json_member_add_string(wb, "color_slot", color_slot);
3055 + if(opacity)
3056 + buffer_json_member_add_string(wb, "opacity", opacity);
3057 + buffer_json_member_add_string(wb, "line_style", line_style);
3058 + buffer_json_member_add_string(wb, "width", width);
3059 + buffer_json_member_add_string(wb, "curve", "auto");
3060 + buffer_json_member_add_string(wb, "arrow", arrow);
3061 + if(layout_strength || layout_distance) {
3062 + buffer_json_member_add_object(wb, "layout");
3063 + {
3064 + if(layout_strength)
3065 + buffer_json_member_add_string(wb, "strength", layout_strength);
3066 + if(layout_distance)
3067 + buffer_json_member_add_string(wb, "distance", layout_distance);
3068 }
3069 buffer_json_object_close(wb);
3070 + }
3071 + if(scale_key && value_column) {
3072 + buffer_json_member_add_object(wb, "variable");
3073 + {
3074 + buffer_json_member_add_string(wb, "channel", "width");
3075 + buffer_json_member_add_string(wb, "scale_key", scale_key);
3076 + buffer_json_member_add_string(wb, "value_column", value_column);
3077 + buffer_json_member_add_string(wb, "min", width);
3078 + buffer_json_member_add_string(wb, "max", "emphasis");
3079 + }
3080 + buffer_json_object_close(wb);
3081 + }
3082 + }
3083 + buffer_json_object_close(wb);
3084 +
3085 + }
3086 + buffer_json_object_close(wb);
3087 +}
3088 +
3089 +static void topology_v1_emit_type_registry(BUFFER *wb, bool detailed __maybe_unused) {
3090 + buffer_json_member_add_object(wb, "types");
3091 + {
3092 + buffer_json_member_add_object(wb, "actor_types");
3093 + {
3094 + topology_v1_emit_actor_type(
3095 + wb, "self", "machine_guid", "hostname", "node",
3096 + "This host", "self", "self", "actor", true, "fixed", NULL, "emphasized", "strongest", false, NULL,
3097 + detailed,
3098 + "display_name", "hostname");
3099 + topology_v1_emit_actor_type(
3100 + wb, "process", "machine_guid", "process", "process_name",
3101 + "Process", "primary", "process", "actor", true, "metric", "socket_count", "normal", "normal", true, "socket_ports",
3102 + detailed,
3103 + "display_name", "process");
3104 + topology_v1_emit_actor_type(
3105 + wb, "endpoint", "ip", "address_space", "endpoint",
3106 + "Correlation endpoint", "derived", "remote-endpoint", "endpoint", true, "fixed", NULL, "compact", "weaker", false, NULL,
3107 + detailed,
3108 + "display_name", "ip");
3109 + }
3110 + buffer_json_object_close(wb);
3111 +
3112 + buffer_json_member_add_object(wb, "link_types");
3113 + {
3114 + topology_v1_emit_link_type(
3115 + wb, "socket", "directed", "dependency", "traffic", "socket",
3116 + "Local socket", "gray", "solid", "thin", "forward", NULL,
3117 + "sockets", "socket_count", "normal", "normal");
3118 + topology_v1_emit_link_type(
3119 + wb, "endpoint_socket", "directed", "dependency", "traffic", "socket",
3120 + "Endpoint connection", "primary", "solid", "thin", "forward", NULL,
3121 + NULL, NULL, "normal", "normal");
3122 + topology_v1_emit_link_type(
3123 + wb, "correlated_socket", "directed", "dependency", "traffic", "socket",
3124 + "Correlated socket", "primary", "solid", "thin", "forward", NULL,
3125 + "sockets", "socket_count", "normal", "farthest");
3126 + topology_v1_emit_link_type(
3127 + wb, "ownership", "hierarchical", "ownership", "ownership", NULL,
3128 + "Process ownership", "dim", "dotted", "thin", "none", "faded",
3129 + NULL, NULL, "normal", "normal");
3130 + }
3131 + buffer_json_object_close(wb);
3132
1940 - buffer_json_member_add_array(wb, "modal_tabs");
3133 + buffer_json_member_add_object(wb, "port_types");
3134 + {
3135 + buffer_json_member_add_object(wb, "topology");
3136 + {
3137 + buffer_json_member_add_object(wb, "presentation");
3138 {
1942 - buffer_json_add_array_item_object(wb);
1943 - buffer_json_member_add_string(wb, "id", "info");
1944 - buffer_json_member_add_string(wb, "label", "Info");
1945 - buffer_json_object_close(wb);
3139 + buffer_json_member_add_string(wb, "label", "Socket");
3140 + buffer_json_member_add_string(wb, "color_slot", "primary");
3141 + buffer_json_member_add_string(wb, "opacity", "normal");
3142 }
1947 - buffer_json_array_close(wb);
3143 + buffer_json_object_close(wb);
3144 }
3145 buffer_json_object_close(wb);
3146 + }
3147 + buffer_json_object_close(wb);
3148
1951 - buffer_json_member_add_object(wb, "endpoint");
3149 + buffer_json_member_add_object(wb, "evidence_types");
3150 + {
3151 + buffer_json_member_add_object(wb, "socket");
3152 {
1953 - buffer_json_member_add_string(wb, "label", "Endpoint");
1954 - buffer_json_member_add_string(wb, "color_slot", "derived");
1955 - buffer_json_member_add_boolean(wb, "border", true);
1956 - buffer_json_member_add_string(wb, "role", "endpoint");
3153 + buffer_json_member_add_string(wb, "link_type", "socket");
3154 + buffer_json_member_add_string(wb, "role", "relationship_evidence");
3155 + buffer_json_member_add_array(wb, "columns");
3156 + topology_v1_emit_socket_evidence_columns(wb);
3157 + buffer_json_array_close(wb);
3158 + buffer_json_member_add_array(wb, "match_columns");
3159 + buffer_json_add_array_item_string(wb, "client_ip");
3160 + buffer_json_add_array_item_string(wb, "client_port");
3161 + buffer_json_add_array_item_string(wb, "server_ip");
3162 + buffer_json_add_array_item_string(wb, "server_port");
3163 + buffer_json_add_array_item_string(wb, "protocol");
3164 + buffer_json_array_close(wb);
3165 + }
3166 + buffer_json_object_close(wb);
3167 + }
3168 + buffer_json_object_close(wb);
3169
1958 - buffer_json_member_add_array(wb, "summary_fields");
3170 + buffer_json_member_add_object(wb, "table_types");
3171 + {
3172 + buffer_json_member_add_object(wb, "socket_ports");
3173 + {
3174 + buffer_json_member_add_string(wb, "role", "actor_inventory");
3175 + buffer_json_member_add_string(wb, "owner", "actor");
3176 + buffer_json_member_add_string(wb, "aggregation", "sum");
3177 + buffer_json_member_add_array(wb, "columns");
3178 + topology_v1_emit_socket_port_columns(wb);
3179 + buffer_json_array_close(wb);
3180 + buffer_json_member_add_object(wb, "presentation");
3181 {
1960 - buffer_json_add_array_item_object(wb);
1961 - buffer_json_member_add_string(wb, "key", "display_name");
1962 - buffer_json_member_add_string(wb, "label", "IP Address");
1963 - buffer_json_member_add_array(wb, "sources");
1964 - buffer_json_add_array_item_string(wb, "attributes.display_name");
1965 - buffer_json_add_array_item_string(wb, "match.ip_addresses.0");
1966 - buffer_json_array_close(wb);
1967 - buffer_json_object_close(wb);
1968 -
1969 - buffer_json_add_array_item_object(wb);
1970 - buffer_json_member_add_string(wb, "key", "socket_count");
1971 - buffer_json_member_add_string(wb, "label", "Sockets");
1972 - buffer_json_member_add_array(wb, "sources");
1973 - buffer_json_add_array_item_string(wb, "attributes.socket_count");
1974 - buffer_json_array_close(wb);
1975 - buffer_json_object_close(wb);
1976 -
1977 - buffer_json_add_array_item_object(wb);
1978 - buffer_json_member_add_string(wb, "key", "address_space");
1979 - buffer_json_member_add_string(wb, "label", "Address Space");
1980 - buffer_json_member_add_array(wb, "sources");
1981 - buffer_json_add_array_item_string(wb, "labels.address_space");
3182 + buffer_json_member_add_string(wb, "label", "Ports");
3183 + buffer_json_member_add_uint64(wb, "order", 1);
3184 + buffer_json_member_add_array(wb, "columns");
3185 + {
3186 + topology_v1_emit_modal_direct_column(wb, "port", "Port", "port", "number");
3187 + topology_v1_emit_modal_direct_column(wb, "protocol", "Protocol", "protocol", "badge");
3188 + topology_v1_emit_modal_direct_column(wb, "sockets", "Sockets", "socket_count", "number");
3189 + }
3190 buffer_json_array_close(wb);
1983 - buffer_json_object_close(wb);
3191 }
3192 + buffer_json_object_close(wb);
3193 + }
3194 + buffer_json_object_close(wb);
3195 +
3196 + buffer_json_member_add_object(wb, "connections");
3197 + {
3198 + buffer_json_member_add_string(wb, "role", "relationship_summary");
3199 + buffer_json_member_add_string(wb, "owner", "link");
3200 + buffer_json_member_add_string(wb, "aggregation", "merge_metrics");
3201 + buffer_json_member_add_array(wb, "columns");
3202 + topology_v1_emit_connection_columns(wb);
3203 buffer_json_array_close(wb);
3204 + }
3205 + buffer_json_object_close(wb);
3206
1987 - buffer_json_member_add_object(wb, "tables");
3207 + buffer_json_member_add_object(wb, "actor_labels");
3208 + {
3209 + buffer_json_member_add_string(wb, "role", "actor_inventory");
3210 + buffer_json_member_add_string(wb, "owner", "actor");
3211 + buffer_json_member_add_string(wb, "aggregation", "set");
3212 + buffer_json_member_add_array(wb, "columns");
3213 + topology_v1_emit_actor_label_columns(wb);
3214 + buffer_json_array_close(wb);
3215 + buffer_json_member_add_object(wb, "presentation");
3216 {
1989 - buffer_json_member_add_object(wb, "links");
3217 + buffer_json_member_add_string(wb, "label", "Labels");
3218 + buffer_json_member_add_uint64(wb, "order", 0);
3219 + buffer_json_member_add_array(wb, "columns");
3220 {
1991 - buffer_json_member_add_string(wb, "label", "Connections");
1992 - buffer_json_member_add_string(wb, "source", "links");
1993 - buffer_json_member_add_array(wb, "columns");
1994 - {
1995 - buffer_json_add_array_item_object(wb);
1996 - buffer_json_member_add_string(wb, "key", "remoteLabel");
1997 - buffer_json_member_add_string(wb, "label", "Remote");
1998 - buffer_json_member_add_string(wb, "type", "actor_link");
1999 - buffer_json_object_close(wb);
2000 -
2001 - buffer_json_add_array_item_object(wb);
2002 - buffer_json_member_add_string(wb, "key", "protocol");
2003 - buffer_json_member_add_string(wb, "label", "Protocol");
2004 - buffer_json_object_close(wb);
2005 -
2006 - buffer_json_add_array_item_object(wb);
2007 - buffer_json_member_add_string(wb, "key", "direction");
2008 - buffer_json_member_add_string(wb, "label", "Direction");
2009 - buffer_json_object_close(wb);
2010 - }
2011 - buffer_json_array_close(wb);
3221 + topology_v1_emit_modal_direct_column(wb, "key", "Label", "key", "text");
3222 + topology_v1_emit_modal_direct_column(wb, "value", "Value", "value", "text");
3223 + topology_v1_emit_modal_direct_column(wb, "source", "Source", "source", "badge");
3224 + topology_v1_emit_modal_direct_column(wb, "kind", "Kind", "kind", "badge");
3225 }
2013 - buffer_json_object_close(wb);
3226 + buffer_json_array_close(wb);
3227 }
3228 buffer_json_object_close(wb);
3229 + }
3230 + buffer_json_object_close(wb);
3231 + }
3232 + buffer_json_object_close(wb);
3233
2017 - buffer_json_member_add_array(wb, "modal_tabs");
2018 - {
2019 - buffer_json_add_array_item_object(wb);
2020 - buffer_json_member_add_string(wb, "id", "info");
2021 - buffer_json_member_add_string(wb, "label", "Info");
2022 - buffer_json_object_close(wb);
2023 - }
3234 + buffer_json_member_add_object(wb, "aggregation_scopes");
3235 + {
3236 + buffer_json_member_add_object(wb, "node");
3237 + {
3238 + buffer_json_member_add_array(wb, "columns");
3239 + buffer_json_add_array_item_string(wb, "machine_guid");
3240 + buffer_json_add_array_item_string(wb, "hostname");
3241 buffer_json_array_close(wb);
3242 + buffer_json_member_add_string(wb, "evidence_policy", "preserve");
3243 }
3244 buffer_json_object_close(wb);
2027 - }
2028 - buffer_json_object_close(wb);
3245
2030 - buffer_json_member_add_object(wb, "link_types");
2031 - {
2032 - buffer_json_member_add_object(wb, "ownership");
3246 + buffer_json_member_add_object(wb, "process_name");
3247 {
2034 - buffer_json_member_add_string(wb, "label", "Ownership");
2035 - buffer_json_member_add_string(wb, "color_slot", "muted");
2036 - buffer_json_member_add_boolean(wb, "dash", true);
3248 + buffer_json_member_add_array(wb, "columns");
3249 + buffer_json_add_array_item_string(wb, "process");
3250 + buffer_json_array_close(wb);
3251 + buffer_json_member_add_string(wb, "evidence_policy", "preserve");
3252 }
3253 buffer_json_object_close(wb);
3254
2040 - buffer_json_member_add_object(wb, "socket");
3255 + buffer_json_member_add_object(wb, "pid");
3256 {
2042 - buffer_json_member_add_string(wb, "label", "Socket");
2043 - buffer_json_member_add_string(wb, "color_slot", "primary");
2044 - buffer_json_member_add_double(wb, "width", 1.5);
3257 + buffer_json_member_add_array(wb, "columns");
3258 + buffer_json_add_array_item_string(wb, "pid");
3259 + buffer_json_add_array_item_string(wb, "net_ns_inode");
3260 + buffer_json_array_close(wb);
3261 + buffer_json_member_add_string(wb, "evidence_policy", "preserve");
3262 }
3263 buffer_json_object_close(wb);
2047 - }
2048 - buffer_json_object_close(wb);
3264
2050 - buffer_json_member_add_array(wb, "port_fields");
2051 - {
2052 - buffer_json_add_array_item_object(wb);
2053 - buffer_json_member_add_string(wb, "key", "type");
2054 - buffer_json_member_add_string(wb, "label", "Type");
2055 - buffer_json_object_close(wb);
3265 }
2057 - buffer_json_array_close(wb);
3266 + buffer_json_object_close(wb);
3267 + }
3268 + buffer_json_object_close(wb);
3269 +}
3270
2059 - buffer_json_member_add_object(wb, "port_types");
3271 +static void topology_v1_emit_presentation(BUFFER *wb) {
3272 + buffer_json_member_add_object(wb, "presentation");
3273 + {
3274 + buffer_json_member_add_string(wb, "profile_version", "network-connections.v1");
3275 + buffer_json_member_add_object(wb, "selection");
3276 {
2061 - buffer_json_member_add_object(wb, "topology");
3277 + buffer_json_member_add_object(wb, "actor_click");
3278 {
2063 - buffer_json_member_add_string(wb, "label", "Socket");
2064 - buffer_json_member_add_string(wb, "color_slot", "primary");
2065 - buffer_json_member_add_double(wb, "opacity", 1.0);
3279 + buffer_json_member_add_string(wb, "mode", "highlight_connections");
3280 }
3281 buffer_json_object_close(wb);
3282 }
@@ -2084,7 +3298,7 @@ static void topology_write_presentation(BUFFER *wb) {
3298
3299 buffer_json_add_array_item_object(wb);
3300 buffer_json_member_add_string(wb, "type", "endpoint");
2087 - buffer_json_member_add_string(wb, "label", "Endpoint");
3301 + buffer_json_member_add_string(wb, "label", "Correlation endpoint");
3302 buffer_json_object_close(wb);
3303 }
3304 buffer_json_array_close(wb);
@@ -2093,12 +3307,22 @@ static void topology_write_presentation(BUFFER *wb) {
3307 {
3308 buffer_json_add_array_item_object(wb);
3309 buffer_json_member_add_string(wb, "type", "ownership");
2096 - buffer_json_member_add_string(wb, "label", "Ownership");
3310 + buffer_json_member_add_string(wb, "label", "Process ownership");
3311 buffer_json_object_close(wb);
3312
3313 buffer_json_add_array_item_object(wb);
3314 buffer_json_member_add_string(wb, "type", "socket");
2101 - buffer_json_member_add_string(wb, "label", "Socket");
3315 + buffer_json_member_add_string(wb, "label", "Local socket");
3316 + buffer_json_object_close(wb);
3317 +
3318 + buffer_json_add_array_item_object(wb);
3319 + buffer_json_member_add_string(wb, "type", "endpoint_socket");
3320 + buffer_json_member_add_string(wb, "label", "Endpoint connection");
3321 + buffer_json_object_close(wb);
3322 +
3323 + buffer_json_add_array_item_object(wb);
3324 + buffer_json_member_add_string(wb, "type", "correlated_socket");
3325 + buffer_json_member_add_string(wb, "label", "Correlated socket");
3326 buffer_json_object_close(wb);
3327 }
3328 buffer_json_array_close(wb);
@@ -2114,567 +3338,481 @@ static void topology_write_presentation(BUFFER *wb) {
3338 }
3339 buffer_json_object_close(wb);
3340
2117 - buffer_json_member_add_string(wb, "actor_click_behavior", "highlight_connections");
2118 - }
2119 - buffer_json_object_close(wb);
2120 -}
2121 -
2122 -static void topology_write_actors(BUFFER *wb, const NV_TOPOLOGY_CONTEXT *ctx, NV_TOPOLOGY_RENDER_STATE *state) {
2123 - DICTIONARY *process_socket_index = topology_build_process_socket_index(ctx);
2124 -
2125 - buffer_json_member_add_array(wb, "actors");
2126 - {
2127 - buffer_json_add_array_item_object(wb);
3341 + buffer_json_member_add_array(wb, "port_fields");
3342 {
2129 - buffer_json_member_add_string(wb, "actor_id", state->host_actor_id);
2130 - buffer_json_member_add_string(wb, "actor_type", "self");
2131 - buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2132 - buffer_json_member_add_string(wb, "source", NETWORK_TOPOLOGY_SOURCE);
2133 - topology_add_host_match(wb, ctx);
3343 + buffer_json_add_array_item_object(wb);
3344 + buffer_json_member_add_string(wb, "key", "type");
3345 + buffer_json_member_add_string(wb, "label", "Type");
3346 + buffer_json_object_close(wb);
3347
2135 - buffer_json_member_add_object(wb, "attributes");
2136 - {
2137 - buffer_json_member_add_string(wb, "hostname", ctx->hostname);
2138 - buffer_json_member_add_uint64(wb, "local_ip_count", state->local_ip_count);
2139 - buffer_json_member_add_uint64(wb, "observed_sockets", ctx->sockets_total);
2140 - buffer_json_member_add_string(wb, "display_name", ctx->hostname);
2141 - buffer_json_member_add_string(wb, "actor_class", "self");
2142 - }
3348 + buffer_json_add_array_item_object(wb);
3349 + buffer_json_member_add_string(wb, "key", "socket_count");
3350 + buffer_json_member_add_string(wb, "label", "Sockets");
3351 buffer_json_object_close(wb);
3352 + }
3353 + buffer_json_array_close(wb);
3354
2145 - buffer_json_member_add_object(wb, "labels");
3355 + buffer_json_member_add_object(wb, "scale_keys");
3356 + {
3357 + buffer_json_member_add_object(wb, "sockets");
3358 {
2147 - buffer_json_member_add_string(wb, "hostname", ctx->hostname);
2148 - if(ctx->machine_guid[0])
2149 - buffer_json_member_add_string(wb, "netdata_machine_guid", ctx->machine_guid);
2150 - buffer_json_member_add_string(wb, "source", NETWORK_TOPOLOGY_SOURCE);
2151 - buffer_json_member_add_string(wb, "display_name", ctx->hostname);
2152 - buffer_json_member_add_string(wb, "actor_class", "self");
3359 + buffer_json_member_add_string(wb, "label", "Sockets");
3360 + buffer_json_member_add_string(wb, "unit", "count");
3361 }
3362 buffer_json_object_close(wb);
3363 +
3364 }
3365 buffer_json_object_close(wb);
3366 + }
3367 + buffer_json_object_close(wb);
3368 +}
3369
2158 - NV_PROCESS_ACTOR *pa;
2159 - dfe_start_read(ctx->process_actors, pa) {
2160 - char process_actor_id[NV_TOPOLOGY_KEY_MAX];
2161 - char process_display_name[NV_TOPOLOGY_KEY_MAX];
2162 - topology_actor_id_for_process(ctx, pa->pid, pa->uid, pa->net_ns_inode, pa->process, process_actor_id, sizeof(process_actor_id));
2163 - topology_process_display_name(ctx, pa->process, pa->pid, process_display_name, sizeof(process_display_name));
2164 - buffer_json_add_array_item_object(wb);
2165 - {
2166 - buffer_json_member_add_string(wb, "actor_id", process_actor_id);
2167 - buffer_json_member_add_string(wb, "actor_type", "process");
2168 - buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2169 - buffer_json_member_add_string(wb, "source", NETWORK_TOPOLOGY_SOURCE);
2170 - topology_add_process_match(wb, ctx, pa);
3370 +static void topology_v1_emit_actor_table(BUFFER *wb, NV_TOPOLOGY_V1_PAYLOAD *payload) {
3371 + buffer_json_member_add_object(wb, "actors");
3372 + {
3373 + buffer_json_member_add_uint64(wb, "rows", payload->actors_used);
3374 + buffer_json_member_add_array(wb, "columns");
3375 + topology_v1_emit_actor_columns(wb);
3376 + buffer_json_array_close(wb);
3377 + buffer_json_member_add_array(wb, "values");
3378 +
3379 +#define NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(member) do { \
3380 + topology_v1_values_start(wb); \
3381 + for(size_t i = 0; i < payload->actors_used; i++) \
3382 + buffer_json_add_array_item_string(wb, payload->actors[i].member[0] ? payload->actors[i].member : NULL); \
3383 + topology_v1_values_end(wb); \
3384 + } while(0)
3385 +#define NV_TOPOLOGY_V1_ACTOR_UINT_VALUES(member, has_member) do { \
3386 + topology_v1_values_start(wb); \
3387 + for(size_t i = 0; i < payload->actors_used; i++) \
3388 + topology_v1_add_nullable_uint(wb, payload->actors[i].has_member, payload->actors[i].member); \
3389 + topology_v1_values_end(wb); \
3390 + } while(0)
3391 +
3392 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(id);
3393 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(type);
3394 + topology_v1_const_string(wb, NETWORK_TOPOLOGY_LAYER);
3395 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(machine_guid);
3396 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(hostname);
3397 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(process);
3398 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(username);
3399 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(cmdline);
3400 + NV_TOPOLOGY_V1_ACTOR_UINT_VALUES(pid, has_pid);
3401 + NV_TOPOLOGY_V1_ACTOR_UINT_VALUES(ppid, has_ppid);
3402 + NV_TOPOLOGY_V1_ACTOR_UINT_VALUES(uid, has_uid);
3403 + NV_TOPOLOGY_V1_ACTOR_UINT_VALUES(net_ns_inode, has_net_ns_inode);
3404 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(namespace_type);
3405 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(local_ip);
3406 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(local_address_space);
3407 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(ip);
3408 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(address_space);
3409 + NV_TOPOLOGY_V1_ACTOR_STRING_VALUES(display_name);
3410 +
3411 + topology_v1_values_start(wb);
3412 + for(size_t i = 0; i < payload->actors_used; i++)
3413 + buffer_json_add_array_item_uint64(wb, payload->actors[i].sockets);
3414 + topology_v1_values_end(wb);
3415 +
3416 + NV_TOPOLOGY_V1_ACTOR_UINT_VALUES(local_ip_count, has_local_ip_count);
3417 +
3418 +#undef NV_TOPOLOGY_V1_ACTOR_STRING_VALUES
3419 +#undef NV_TOPOLOGY_V1_ACTOR_UINT_VALUES
3420
2172 - buffer_json_member_add_object(wb, "parent_match");
2173 - {
2174 - if(ctx->machine_guid[0])
2175 - buffer_json_member_add_string(wb, "netdata_machine_guid", ctx->machine_guid);
2176 - topology_add_single_item_string_array(wb, "hostnames", ctx->hostname);
2177 - }
2178 - buffer_json_object_close(wb);
3421 + buffer_json_array_close(wb);
3422 + }
3423 + buffer_json_object_close(wb);
3424 +}
3425
2180 - buffer_json_member_add_object(wb, "attributes");
2181 - {
2182 - if(ctx->options.processes_by_pid) {
2183 - buffer_json_member_add_uint64(wb, "pid", pa->pid);
2184 - buffer_json_member_add_uint64(wb, "ppid", pa->ppid);
2185 - buffer_json_member_add_uint64(wb, "uid", pa->uid);
2186 - buffer_json_member_add_uint64(wb, "net_ns_inode", pa->net_ns_inode);
2187 - }
2188 - buffer_json_member_add_uint64(wb, "socket_count", pa->sockets);
2189 - buffer_json_member_add_string(wb, "local_ip", pa->local_ip);
2190 - buffer_json_member_add_string(wb, "local_address_space", pa->local_address_space);
2191 - buffer_json_member_add_string(wb, "display_name", process_display_name);
2192 - buffer_json_member_add_string(wb, "actor_class", "process");
2193 - if(pa->cmdline[0])
2194 - buffer_json_member_add_string(wb, "cmdline", pa->cmdline);
2195 - }
2196 - buffer_json_object_close(wb);
3426 +static void topology_v1_emit_link_table(BUFFER *wb, NV_TOPOLOGY_V1_PAYLOAD *payload) {
3427 + buffer_json_member_add_object(wb, "links");
3428 + {
3429 + buffer_json_member_add_uint64(wb, "rows", payload->links_used);
3430 + buffer_json_member_add_array(wb, "columns");
3431 + topology_v1_emit_link_columns(wb);
3432 + buffer_json_array_close(wb);
3433 + buffer_json_member_add_array(wb, "values");
3434 +
3435 +#define NV_TOPOLOGY_V1_LINK_UINT_VALUES(member) do { \
3436 + topology_v1_values_start(wb); \
3437 + for(size_t i = 0; i < payload->links_used; i++) \
3438 + buffer_json_add_array_item_uint64(wb, payload->links[i].member); \
3439 + topology_v1_values_end(wb); \
3440 + } while(0)
3441 +#define NV_TOPOLOGY_V1_LINK_STRING_VALUES(member) do { \
3442 + NV_TOPOLOGY_V1_STRING_COLUMN column; \
3443 + topology_v1_string_column_init(&column, payload->links_used); \
3444 + for(size_t i = 0; i < payload->links_used; i++) \
3445 + topology_v1_string_column_add(&column, payload->links[i].member); \
3446 + topology_v1_emit_auto_string_column(wb, &column); \
3447 + topology_v1_string_column_free(&column); \
3448 + } while(0)
3449 +
3450 + NV_TOPOLOGY_V1_LINK_UINT_VALUES(src_actor);
3451 + NV_TOPOLOGY_V1_LINK_UINT_VALUES(dst_actor);
3452 + NV_TOPOLOGY_V1_LINK_STRING_VALUES(type);
3453 + NV_TOPOLOGY_V1_LINK_STRING_VALUES(protocol);
3454 + NV_TOPOLOGY_V1_LINK_STRING_VALUES(state);
3455 + NV_TOPOLOGY_V1_LINK_UINT_VALUES(evidence_count);
3456 + NV_TOPOLOGY_V1_LINK_UINT_VALUES(socket_count);
3457 + NV_TOPOLOGY_V1_LINK_UINT_VALUES(retransmissions);
3458 +
3459 + topology_v1_values_start(wb);
3460 + for(size_t i = 0; i < payload->links_used; i++)
3461 + buffer_json_add_array_item_double(wb, (double)payload->links[i].max_rtt_usec / (double)USEC_PER_MS);
3462 + topology_v1_values_end(wb);
3463 +
3464 + topology_v1_values_start(wb);
3465 + for(size_t i = 0; i < payload->links_used; i++)
3466 + buffer_json_add_array_item_double(wb, (double)payload->links[i].max_rcv_rtt_usec / (double)USEC_PER_MS);
3467 + topology_v1_values_end(wb);
3468 +
3469 +#undef NV_TOPOLOGY_V1_LINK_UINT_VALUES
3470 +#undef NV_TOPOLOGY_V1_LINK_STRING_VALUES
3471
2198 - buffer_json_member_add_object(wb, "labels");
3472 + buffer_json_array_close(wb);
3473 + }
3474 + buffer_json_object_close(wb);
3475 +}
3476 +
3477 +static void topology_v1_emit_socket_port_table(BUFFER *wb, NV_TOPOLOGY_V1_PAYLOAD *payload) {
3478 + buffer_json_member_add_object(wb, "tables");
3479 + {
3480 + buffer_json_member_add_object(wb, "actor");
3481 + {
3482 + buffer_json_member_add_object(wb, "socket_ports");
3483 + {
3484 + buffer_json_member_add_string(wb, "type", "socket_ports");
3485 + buffer_json_member_add_object(wb, "table");
3486 {
2200 - buffer_json_member_add_string(wb, "process", pa->process);
2201 - buffer_json_member_add_string(wb, "user", pa->username);
2202 - buffer_json_member_add_string(wb, "namespace", pa->namespace_type);
2203 - buffer_json_member_add_string(wb, "local_address_space", pa->local_address_space);
2204 - buffer_json_member_add_string(wb, "display_name", process_display_name);
2205 - buffer_json_member_add_string(wb, "actor_class", "process");
3487 + buffer_json_member_add_uint64(wb, "rows", payload->ports_used);
3488 + buffer_json_member_add_array(wb, "columns");
3489 + topology_v1_emit_socket_port_columns(wb);
3490 + buffer_json_array_close(wb);
3491 + buffer_json_member_add_array(wb, "values");
3492 +
3493 +#define NV_TOPOLOGY_V1_PORT_UINT_VALUES(member) do { \
3494 + topology_v1_values_start(wb); \
3495 + for(size_t i = 0; i < payload->ports_used; i++) \
3496 + buffer_json_add_array_item_uint64(wb, payload->ports[i].member); \
3497 + topology_v1_values_end(wb); \
3498 + } while(0)
3499 +#define NV_TOPOLOGY_V1_PORT_STRING_VALUES(member) do { \
3500 + NV_TOPOLOGY_V1_STRING_COLUMN column; \
3501 + topology_v1_string_column_init(&column, payload->ports_used); \
3502 + for(size_t i = 0; i < payload->ports_used; i++) \
3503 + topology_v1_string_column_add(&column, payload->ports[i].member); \
3504 + topology_v1_emit_auto_string_column(wb, &column); \
3505 + topology_v1_string_column_free(&column); \
3506 + } while(0)
3507 +
3508 + NV_TOPOLOGY_V1_PORT_UINT_VALUES(actor);
3509 + NV_TOPOLOGY_V1_PORT_UINT_VALUES(port);
3510 + NV_TOPOLOGY_V1_PORT_STRING_VALUES(protocol);
3511 + NV_TOPOLOGY_V1_PORT_UINT_VALUES(socket_count);
3512 +
3513 +#undef NV_TOPOLOGY_V1_PORT_UINT_VALUES
3514 +#undef NV_TOPOLOGY_V1_PORT_STRING_VALUES
3515 +
3516 + buffer_json_array_close(wb);
3517 }
3518 buffer_json_object_close(wb);
3519 + }
3520 + buffer_json_object_close(wb);
3521
2209 - buffer_json_member_add_object(wb, "tables");
3522 + buffer_json_member_add_object(wb, "actor_labels");
3523 + {
3524 + buffer_json_member_add_string(wb, "type", "actor_labels");
3525 + buffer_json_member_add_object(wb, "table");
3526 {
2211 - buffer_json_member_add_array(wb, "sockets");
2212 - {
2213 - NV_PROCESS_SOCKET_ROWS *rows = process_socket_index ? dictionary_get(process_socket_index, process_actor_id) : NULL;
2214 - for(NV_PROCESS_SOCKET_ROW *row = rows ? rows->head : NULL; row; row = row->next) {
2215 - char remote_endpoint[128];
2216 - topology_format_ip_port(row->link->remote_ip, row->link->remote_port, remote_endpoint, sizeof(remote_endpoint));
2217 - buffer_json_add_array_item_object(wb);
2218 - buffer_json_member_add_string(wb, "remote", remote_endpoint);
2219 - buffer_json_member_add_string(wb, "protocol", row->link->protocol);
2220 - buffer_json_member_add_string(wb, "direction", row->link->direction);
2221 - buffer_json_member_add_string(wb, "state", row->link->state);
2222 - buffer_json_object_close(wb);
2223 - }
2224 - }
3527 + buffer_json_member_add_uint64(wb, "rows", payload->labels_used);
3528 + buffer_json_member_add_array(wb, "columns");
3529 + topology_v1_emit_actor_label_columns(wb);
3530 + buffer_json_array_close(wb);
3531 + buffer_json_member_add_array(wb, "values");
3532 +
3533 +#define NV_TOPOLOGY_V1_LABEL_UINT_VALUES(member) do { \
3534 + topology_v1_values_start(wb); \
3535 + for(size_t i = 0; i < payload->labels_used; i++) \
3536 + buffer_json_add_array_item_uint64(wb, payload->labels[i].member); \
3537 + topology_v1_values_end(wb); \
3538 + } while(0)
3539 +#define NV_TOPOLOGY_V1_LABEL_STRING_VALUES(member) do { \
3540 + NV_TOPOLOGY_V1_STRING_COLUMN column; \
3541 + topology_v1_string_column_init(&column, payload->labels_used); \
3542 + for(size_t i = 0; i < payload->labels_used; i++) \
3543 + topology_v1_string_column_add(&column, payload->labels[i].member); \
3544 + topology_v1_emit_auto_string_column(wb, &column); \
3545 + topology_v1_string_column_free(&column); \
3546 + } while(0)
3547 +
3548 + NV_TOPOLOGY_V1_LABEL_UINT_VALUES(actor);
3549 + NV_TOPOLOGY_V1_LABEL_STRING_VALUES(key);
3550 + NV_TOPOLOGY_V1_LABEL_STRING_VALUES(value);
3551 + NV_TOPOLOGY_V1_LABEL_STRING_VALUES(source);
3552 + NV_TOPOLOGY_V1_LABEL_STRING_VALUES(kind);
3553 +
3554 + topology_v1_values_start(wb);
3555 + for(size_t i = 0; i < payload->labels_used; i++)
3556 + topology_v1_add_nullable_uint(wb, payload->labels[i].has_value_index, payload->labels[i].value_index);
3557 + topology_v1_values_end(wb);
3558 +
3559 +#undef NV_TOPOLOGY_V1_LABEL_UINT_VALUES
3560 +#undef NV_TOPOLOGY_V1_LABEL_STRING_VALUES
3561 +
3562 buffer_json_array_close(wb);
3563 }
3564 buffer_json_object_close(wb);
3565 }
3566 buffer_json_object_close(wb);
3567 }
2231 - dfe_done(pa);
2232 -
2233 - NV_REMOTE_ACTOR *ra;
2234 - dfe_start_read(ctx->remote_actors, ra) {
2235 - bool endpoint_is_self = topology_ip_belongs_to_self(ctx, ra->ip, ra->address_space);
3568 + buffer_json_object_close(wb);
3569
2237 - char endpoint_actor_id[NV_TOPOLOGY_KEY_MAX];
2238 - topology_actor_id_for_remote_endpoint(ctx, ra->ip, ra->address_space, endpoint_actor_id, sizeof(endpoint_actor_id));
2239 - state->endpoint_actor_count++;
2240 - buffer_json_add_array_item_object(wb);
3570 + if(payload->connections_used) {
3571 + buffer_json_member_add_object(wb, "relationship");
3572 {
2242 - buffer_json_member_add_string(wb, "actor_id", endpoint_actor_id);
2243 - buffer_json_member_add_string(wb, "actor_type", "endpoint");
2244 - buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2245 - buffer_json_member_add_string(wb, "source", NETWORK_TOPOLOGY_SOURCE);
2246 - topology_add_remote_match(wb, ra->ip);
2247 -
2248 - if(endpoint_is_self) {
2249 - buffer_json_member_add_object(wb, "parent_match");
3573 + buffer_json_member_add_object(wb, "connections");
3574 + {
3575 + buffer_json_member_add_string(wb, "type", "connections");
3576 + buffer_json_member_add_object(wb, "table");
3577 {
2251 - if(ctx->machine_guid[0])
2252 - buffer_json_member_add_string(wb, "netdata_machine_guid", ctx->machine_guid);
2253 - topology_add_single_item_string_array(wb, "hostnames", ctx->hostname);
3578 + buffer_json_member_add_uint64(wb, "rows", payload->connections_used);
3579 + buffer_json_member_add_array(wb, "columns");
3580 + topology_v1_emit_connection_columns(wb);
3581 + buffer_json_array_close(wb);
3582 + buffer_json_member_add_array(wb, "values");
3583 +
3584 +#define NV_TOPOLOGY_V1_CONNECTION_UINT_VALUES(member) do { \
3585 + topology_v1_values_start(wb); \
3586 + for(size_t i = 0; i < payload->connections_used; i++) \
3587 + buffer_json_add_array_item_uint64(wb, payload->connections[i].member); \
3588 + topology_v1_values_end(wb); \
3589 + } while(0)
3590 +#define NV_TOPOLOGY_V1_CONNECTION_STRING_VALUES(member) do { \
3591 + NV_TOPOLOGY_V1_STRING_COLUMN column; \
3592 + topology_v1_string_column_init(&column, payload->connections_used); \
3593 + for(size_t i = 0; i < payload->connections_used; i++) \
3594 + topology_v1_string_column_add(&column, payload->connections[i].member); \
3595 + topology_v1_emit_auto_string_column(wb, &column); \
3596 + topology_v1_string_column_free(&column); \
3597 + } while(0)
3598 +
3599 + NV_TOPOLOGY_V1_CONNECTION_UINT_VALUES(src_actor);
3600 + NV_TOPOLOGY_V1_CONNECTION_UINT_VALUES(dst_actor);
3601 + NV_TOPOLOGY_V1_CONNECTION_STRING_VALUES(client_ip);
3602 + NV_TOPOLOGY_V1_CONNECTION_STRING_VALUES(server_ip);
3603 + NV_TOPOLOGY_V1_CONNECTION_STRING_VALUES(protocol);
3604 + NV_TOPOLOGY_V1_CONNECTION_STRING_VALUES(state);
3605 + NV_TOPOLOGY_V1_CONNECTION_UINT_VALUES(socket_count);
3606 + NV_TOPOLOGY_V1_CONNECTION_UINT_VALUES(retransmissions);
3607 +
3608 + topology_v1_values_start(wb);
3609 + for(size_t i = 0; i < payload->connections_used; i++)
3610 + buffer_json_add_array_item_double(wb, (double)payload->connections[i].max_rtt_usec / (double)USEC_PER_MS);
3611 + topology_v1_values_end(wb);
3612 +
3613 + topology_v1_values_start(wb);
3614 + for(size_t i = 0; i < payload->connections_used; i++)
3615 + buffer_json_add_array_item_double(wb, (double)payload->connections[i].max_rcv_rtt_usec / (double)USEC_PER_MS);
3616 + topology_v1_values_end(wb);
3617 +
3618 +#undef NV_TOPOLOGY_V1_CONNECTION_UINT_VALUES
3619 +#undef NV_TOPOLOGY_V1_CONNECTION_STRING_VALUES
3620 +
3621 + buffer_json_array_close(wb);
3622 }
3623 buffer_json_object_close(wb);
3624 }
2257 -
2258 - buffer_json_member_add_object(wb, "attributes");
2259 - {
2260 - buffer_json_member_add_uint64(wb, "socket_count", ra->sockets);
2261 - buffer_json_member_add_uint64(wb, "local_socket_count", endpoint_is_self ? ra->sockets : 0);
2262 - buffer_json_member_add_uint64(wb, "remote_socket_count", endpoint_is_self ? 0 : ra->sockets);
2263 - buffer_json_member_add_string(wb, "endpoint_scope", endpoint_is_self ? "self" : "remote");
2264 - buffer_json_member_add_string(wb, "display_name", ra->ip);
2265 - buffer_json_member_add_string(wb, "actor_class", "endpoint");
2266 - }
2267 - buffer_json_object_close(wb);
2268 -
2269 - buffer_json_member_add_object(wb, "labels");
2270 - {
2271 - buffer_json_member_add_string(wb, "address_space", ra->address_space);
2272 - buffer_json_member_add_string(wb, "endpoint_scope", endpoint_is_self ? "self" : "remote");
2273 - buffer_json_member_add_string(wb, "display_name", ra->ip);
2274 - buffer_json_member_add_string(wb, "actor_class", "endpoint");
2275 - }
3625 buffer_json_object_close(wb);
3626 }
3627 buffer_json_object_close(wb);
3628 }
2280 - dfe_done(ra);
3629 }
2282 - buffer_json_array_close(wb);
2283 -
2284 - topology_destroy_process_socket_index(process_socket_index);
3630 + buffer_json_object_close(wb);
3631 }
3632
2287 -static void topology_write_links_and_stats(BUFFER *wb, const NV_TOPOLOGY_CONTEXT *ctx, NV_TOPOLOGY_RENDER_STATE *state) {
2288 - buffer_json_member_add_array(wb, "links");
3633 +static void topology_v1_emit_socket_evidence_table(BUFFER *wb, NV_TOPOLOGY_V1_PAYLOAD *payload) {
3634 + buffer_json_member_add_object(wb, "evidence");
3635 {
2290 - DICTIONARY *process_parent_ns_lookup = NULL;
2291 - DICTIONARY *process_parent_any_lookup = NULL;
2292 - DICTIONARY *ppid_cache = NULL;
2293 - if(ctx->options.processes_by_pid) {
2294 - process_parent_ns_lookup = dictionary_create_advanced(
2295 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2296 - NULL, sizeof(NV_ENDPOINT_OWNER));
2297 - process_parent_any_lookup = dictionary_create_advanced(
2298 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2299 - NULL, sizeof(NV_ENDPOINT_OWNER));
2300 - ppid_cache = dictionary_create_advanced(
2301 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2302 - NULL, sizeof(NV_PPID_CACHE_ENTRY));
2303 -
2304 - if(process_parent_ns_lookup && process_parent_any_lookup) {
2305 - char parent_key_ns[NV_TOPOLOGY_KEY_MAX];
2306 - char parent_key_any[NV_TOPOLOGY_KEY_MAX];
2307 - char pid_key[64];
2308 - NV_PROCESS_ACTOR *pa_index;
2309 - dfe_start_read(ctx->process_actors, pa_index) {
2310 - if(!pa_index->pid)
2311 - continue;
2312 -
2313 - NV_ENDPOINT_OWNER owner = {
2314 - .pid = pa_index->pid,
2315 - .ppid = pa_index->ppid,
2316 - .uid = pa_index->uid,
2317 - .net_ns_inode = pa_index->net_ns_inode,
2318 - };
2319 - snprintf(owner.process, sizeof(owner.process), "%s", pa_index->process);
2320 -
2321 - topology_process_parent_lookup_key(parent_key_ns, sizeof(parent_key_ns),
2322 - (uint64_t)pa_index->pid, pa_index->net_ns_inode, true);
2323 - dictionary_set(process_parent_ns_lookup, parent_key_ns, &owner, sizeof(owner));
2324 -
2325 - topology_process_parent_lookup_key(parent_key_any, sizeof(parent_key_any),
2326 - (uint64_t)pa_index->pid, 0, false);
2327 - dictionary_set(process_parent_any_lookup, parent_key_any, &owner, sizeof(owner));
2328 -
2329 - if(ppid_cache) {
2330 - NV_PPID_CACHE_ENTRY ppid_entry = { .ppid = pa_index->ppid };
2331 - topology_pid_lookup_key(pid_key, sizeof(pid_key), (uint64_t)pa_index->pid);
2332 - dictionary_set(ppid_cache, pid_key, &ppid_entry, sizeof(ppid_entry));
2333 - }
2334 - }
2335 - dfe_done(pa_index);
2336 - }
2337 - }
2338 -
2339 - NV_PROCESS_ACTOR *pa;
2340 - dfe_start_read(ctx->process_actors, pa) {
2341 - bool src_is_process = false;
2342 - NV_ENDPOINT_OWNER *parent_pa = NULL;
2343 - NV_ENDPOINT_OWNER parent_resolved = { 0 };
2344 - const char *ownership_kind = "self_root";
2345 - char src_actor_id[NV_TOPOLOGY_KEY_MAX];
2346 - char src_display_name[NV_TOPOLOGY_KEY_MAX];
2347 - char process_actor_id[NV_TOPOLOGY_KEY_MAX];
2348 - char process_display_name[NV_TOPOLOGY_KEY_MAX];
2349 - char ownership_display_name[NV_TOPOLOGY_KEY_MAX * 2 + 7];
2350 - topology_actor_id_for_process(ctx, pa->pid, pa->uid, pa->net_ns_inode, pa->process, process_actor_id, sizeof(process_actor_id));
2351 - topology_process_display_name(ctx, pa->process, pa->pid, process_display_name, sizeof(process_display_name));
2352 -
2353 - if(ctx->options.processes_by_pid &&
2354 - process_parent_ns_lookup && process_parent_any_lookup &&
2355 - pa->ppid && pa->ppid != pa->pid) {
2356 - parent_pa = topology_find_process_parent_actor(pa->ppid, pa->net_ns_inode,
2357 - process_parent_ns_lookup,
2358 - process_parent_any_lookup,
2359 - ppid_cache);
2360 -
2361 - if(parent_pa) {
2362 - src_is_process = true;
2363 - ownership_kind = "process_parent";
2364 - parent_resolved = *parent_pa;
2365 - parent_pa = &parent_resolved;
2366 - topology_actor_id_for_process(ctx, parent_pa->pid, parent_pa->uid, parent_pa->net_ns_inode, parent_pa->process, src_actor_id, sizeof(src_actor_id));
2367 - topology_process_display_name(ctx, parent_pa->process, parent_pa->pid, src_display_name, sizeof(src_display_name));
2368 - }
2369 - }
3636 + buffer_json_member_add_object(wb, "socket");
3637 + {
3638 + buffer_json_member_add_string(wb, "type", "socket");
3639 + buffer_json_member_add_object(wb, "table");
3640 + {
3641 + buffer_json_member_add_uint64(wb, "rows", payload->evidence_used);
3642 + buffer_json_member_add_array(wb, "columns");
3643 + topology_v1_emit_socket_evidence_columns(wb);
3644 + buffer_json_array_close(wb);
3645 + buffer_json_member_add_array(wb, "values");
3646 +
3647 +#define NV_TOPOLOGY_V1_EVIDENCE_UINT_VALUES(member) do { \
3648 + topology_v1_values_start(wb); \
3649 + for(size_t i = 0; i < payload->evidence_used; i++) \
3650 + buffer_json_add_array_item_uint64(wb, payload->evidence[i].member); \
3651 + topology_v1_values_end(wb); \
3652 + } while(0)
3653 +#define NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(member) do { \
3654 + topology_v1_values_start(wb); \
3655 + for(size_t i = 0; i < payload->evidence_used; i++) \
3656 + buffer_json_add_array_item_uint64(wb, payload->evidence[i].source->member); \
3657 + topology_v1_values_end(wb); \
3658 + } while(0)
3659 +#define NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(member) do { \
3660 + NV_TOPOLOGY_V1_STRING_COLUMN column; \
3661 + topology_v1_string_column_init(&column, payload->evidence_used); \
3662 + for(size_t i = 0; i < payload->evidence_used; i++) \
3663 + topology_v1_string_column_add(&column, payload->evidence[i].source->member); \
3664 + topology_v1_emit_auto_string_column(wb, &column); \
3665 + topology_v1_string_column_free(&column); \
3666 + } while(0)
3667 +
3668 + NV_TOPOLOGY_V1_EVIDENCE_UINT_VALUES(link);
3669 + NV_TOPOLOGY_V1_EVIDENCE_UINT_VALUES(src_actor);
3670 + NV_TOPOLOGY_V1_EVIDENCE_UINT_VALUES(dst_actor);
3671 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(client_ip);
3672 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(client_port);
3673 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(server_ip);
3674 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(server_port);
3675 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(protocol);
3676 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(protocol_family);
3677 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(state);
3678 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(namespace_type);
3679 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(client_address_space);
3680 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(server_address_space);
3681 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(pid);
3682 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(uid);
3683 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(net_ns_inode);
3684 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES(process);
3685 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(sockets);
3686 + NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES(retransmissions);
3687 +
3688 + topology_v1_values_start(wb);
3689 + for(size_t i = 0; i < payload->evidence_used; i++)
3690 + buffer_json_add_array_item_double(wb, (double)payload->evidence[i].source->max_rtt_usec / (double)USEC_PER_MS);
3691 + topology_v1_values_end(wb);
3692 +
3693 + topology_v1_values_start(wb);
3694 + for(size_t i = 0; i < payload->evidence_used; i++)
3695 + buffer_json_add_array_item_double(wb, (double)payload->evidence[i].source->max_rcv_rtt_usec / (double)USEC_PER_MS);
3696 + topology_v1_values_end(wb);
3697 +
3698 +#undef NV_TOPOLOGY_V1_EVIDENCE_UINT_VALUES
3699 +#undef NV_TOPOLOGY_V1_EVIDENCE_SOURCE_UINT_VALUES
3700 +#undef NV_TOPOLOGY_V1_EVIDENCE_SOURCE_STRING_VALUES
3701
2371 - if(!src_is_process) {
2372 - snprintf(src_actor_id, sizeof(src_actor_id), "%s", state->host_actor_id);
2373 - snprintf(src_display_name, sizeof(src_display_name), "%s", ctx->hostname);
3702 + buffer_json_array_close(wb);
3703 }
3704 + buffer_json_object_close(wb);
3705 + }
3706 + buffer_json_object_close(wb);
3707 + }
3708 + buffer_json_object_close(wb);
3709 +}
3710
2376 - snprintf(ownership_display_name, sizeof(ownership_display_name), "%s owns %s", src_display_name, process_display_name);
2377 - state->ownership_link_count++;
2378 -
2379 - buffer_json_add_array_item_object(wb);
2380 - {
2381 - buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2382 - buffer_json_member_add_string(wb, "protocol", "ownership");
2383 - buffer_json_member_add_string(wb, "link_type", "ownership");
2384 - buffer_json_member_add_string(wb, "direction", "contains");
2385 - buffer_json_member_add_string(wb, "state", "active");
2386 - buffer_json_member_add_string(wb, "src_actor_id", src_actor_id);
2387 - buffer_json_member_add_string(wb, "dst_actor_id", process_actor_id);
2388 - buffer_json_member_add_datetime_rfc3339(wb, "discovered_at", ctx->now_ut, true);
2389 - buffer_json_member_add_datetime_rfc3339(wb, "last_seen", ctx->now_ut, true);
2390 -
2391 - buffer_json_member_add_object(wb, "src");
2392 - {
2393 - if(src_is_process) {
2394 - topology_add_process_identity_match(wb, ctx, parent_pa->pid, parent_pa->uid, parent_pa->net_ns_inode, parent_pa->process);
3711 +static void topology_v1_emit_correlation_table(
3712 + BUFFER *wb,
3713 + const NV_TOPOLOGY_V1_CORRELATION_ROW *rows,
3714 + size_t rows_used) {
3715 + buffer_json_member_add_uint64(wb, "rows", rows_used);
3716 +
3717 + buffer_json_member_add_array(wb, "columns");
3718 + topology_v1_emit_column(wb, "actor", "actor_ref", "reference", false, NULL);
3719 + topology_v1_emit_column(wb, "rule", "string", "group_key", false, NULL);
3720 + topology_v1_emit_column(wb, "protocol", "string", "group_key", false, NULL);
3721 + topology_v1_emit_column(wb, "address_space", "string", "group_key", false, NULL);
3722 + topology_v1_emit_column(wb, "ip", "ip", "group_key", false, NULL);
3723 + topology_v1_emit_column(wb, "port", "uint", "group_key", false, NULL);
3724 + buffer_json_array_close(wb);
3725
2396 - buffer_json_member_add_object(wb, "attributes");
2397 - {
2398 - buffer_json_member_add_string(wb, "actor_type", "process");
2399 - if(ctx->options.processes_by_pid) {
2400 - buffer_json_member_add_uint64(wb, "pid", parent_pa->pid);
2401 - buffer_json_member_add_uint64(wb, "ppid", parent_pa->ppid);
2402 - buffer_json_member_add_uint64(wb, "uid", parent_pa->uid);
2403 - buffer_json_member_add_uint64(wb, "net_ns_inode", parent_pa->net_ns_inode);
2404 - }
2405 - buffer_json_member_add_string(wb, "process", parent_pa->process);
2406 - buffer_json_member_add_string(wb, "display_name", src_display_name);
2407 - }
2408 - buffer_json_object_close(wb);
2409 - }
2410 - else {
2411 - topology_add_host_match(wb, ctx);
3726 + buffer_json_member_add_array(wb, "values");
3727
2413 - buffer_json_member_add_object(wb, "attributes");
2414 - {
2415 - buffer_json_member_add_string(wb, "actor_type", "self");
2416 - buffer_json_member_add_string(wb, "display_name", ctx->hostname);
2417 - }
2418 - buffer_json_object_close(wb);
2419 - }
2420 - }
2421 - buffer_json_object_close(wb);
3728 + topology_v1_values_start(wb);
3729 + for(size_t i = 0; i < rows_used; i++)
3730 + buffer_json_add_array_item_uint64(wb, rows[i].actor);
3731 + topology_v1_values_end(wb);
3732
2423 - buffer_json_member_add_object(wb, "dst");
2424 - {
2425 - topology_add_process_match(wb, ctx, pa);
3733 + topology_v1_const_string(wb, "socket_exact");
3734
2427 - buffer_json_member_add_object(wb, "attributes");
2428 - {
2429 - buffer_json_member_add_string(wb, "actor_type", "process");
2430 - if(ctx->options.processes_by_pid) {
2431 - buffer_json_member_add_uint64(wb, "pid", pa->pid);
2432 - buffer_json_member_add_uint64(wb, "ppid", pa->ppid);
2433 - buffer_json_member_add_uint64(wb, "uid", pa->uid);
2434 - buffer_json_member_add_uint64(wb, "net_ns_inode", pa->net_ns_inode);
2435 - }
2436 - buffer_json_member_add_string(wb, "process", pa->process);
2437 - buffer_json_member_add_string(wb, "display_name", process_display_name);
2438 - }
2439 - buffer_json_object_close(wb);
2440 - }
2441 - buffer_json_object_close(wb);
3735 +#define NV_TOPOLOGY_V1_CORRELATION_STRING_VALUES(member) do { \
3736 + NV_TOPOLOGY_V1_STRING_COLUMN column; \
3737 + topology_v1_string_column_init(&column, rows_used); \
3738 + for(size_t i = 0; i < rows_used; i++) \
3739 + topology_v1_string_column_add(&column, rows[i].member); \
3740 + topology_v1_emit_auto_string_column(wb, &column); \
3741 + topology_v1_string_column_free(&column); \
3742 + } while(0)
3743
2443 - buffer_json_member_add_object(wb, "metrics");
2444 - {
2445 - buffer_json_member_add_uint64(wb, "socket_count", pa->sockets);
2446 - buffer_json_member_add_string(wb, "display_name", ownership_display_name);
2447 - }
2448 - buffer_json_object_close(wb);
3744 + NV_TOPOLOGY_V1_CORRELATION_STRING_VALUES(protocol);
3745 + NV_TOPOLOGY_V1_CORRELATION_STRING_VALUES(address_space);
3746 + NV_TOPOLOGY_V1_CORRELATION_STRING_VALUES(ip);
3747
2450 - buffer_json_member_add_object(wb, "labels");
2451 - {
2452 - buffer_json_member_add_string(wb, "link_class", "ownership");
2453 - buffer_json_member_add_string(wb, "render_intent", "dark");
2454 - buffer_json_member_add_string(wb, "ownership_kind", ownership_kind);
2455 - buffer_json_member_add_string(wb, "display_name", ownership_display_name);
2456 - }
2457 - buffer_json_object_close(wb);
2458 - }
2459 - buffer_json_object_close(wb);
2460 - }
2461 - dfe_done(pa);
2462 -
2463 - if(process_parent_ns_lookup)
2464 - dictionary_destroy(process_parent_ns_lookup);
2465 - if(process_parent_any_lookup)
2466 - dictionary_destroy(process_parent_any_lookup);
2467 - if(ppid_cache)
2468 - dictionary_destroy(ppid_cache);
2469 -
2470 - NV_TOPOLOGY_LINK *link;
2471 - dfe_start_read(ctx->links, link) {
2472 - char process_actor_id[NV_TOPOLOGY_KEY_MAX];
2473 - char dst_actor_id[NV_TOPOLOGY_KEY_MAX];
2474 - char endpoint_actor_id[NV_TOPOLOGY_KEY_MAX];
2475 - char local_bind_port[128];
2476 - char remote_endpoint_port[128];
2477 - char process_display_name[NV_TOPOLOGY_KEY_MAX];
2478 - char dst_process_display_name[NV_TOPOLOGY_KEY_MAX];
2479 - char dst_process_port_name[64] = "";
2480 - char link_display_name[NV_TOPOLOGY_KEY_MAX * 2 + 256 + 7];
2481 - bool remote_is_self = topology_ip_belongs_to_self(ctx, link->remote_ip, link->remote_address_space);
2482 - bool dst_is_process = false;
2483 - bool dst_local_peer_unresolved = false;
2484 - uint64_t dst_pid = link->pid;
2485 - uint64_t dst_ppid = link->ppid;
2486 - uint64_t dst_uid = link->uid;
2487 - uint64_t dst_net_ns_inode = link->net_ns_inode;
2488 - const char *dst_process_name = link->process;
2489 - uint16_t dst_process_port = 0;
2490 -
2491 - topology_actor_id_for_process(ctx, link->pid, link->uid, link->net_ns_inode, link->process, process_actor_id, sizeof(process_actor_id));
2492 - topology_actor_id_for_remote_endpoint(ctx, link->remote_ip, link->remote_address_space, endpoint_actor_id, sizeof(endpoint_actor_id));
2493 - snprintf(local_bind_port, sizeof(local_bind_port), "%u", link->local_port);
2494 - topology_format_ip_port(link->remote_ip, link->remote_port, remote_endpoint_port, sizeof(remote_endpoint_port));
2495 - topology_process_display_name(ctx, link->process, link->pid, process_display_name, sizeof(process_display_name));
2496 -
2497 - if(remote_is_self && link->direction_id != SOCKET_DIRECTION_LISTEN) {
2498 - const char *peer_ip = link->peer_ip[0] ? link->peer_ip : link->remote_ip;
2499 - bool allow_service_fallback = true;
2500 - NV_ENDPOINT_OWNER *owner = topology_lookup_endpoint_owner(ctx, link->net_ns_inode, link->protocol_id, peer_ip, link->peer_port, allow_service_fallback);
2501 -
2502 - if(owner) {
2503 - dst_pid = owner->pid;
2504 - dst_ppid = owner->ppid;
2505 - dst_uid = owner->uid;
2506 - dst_net_ns_inode = owner->net_ns_inode;
2507 - dst_process_name = owner->process;
2508 - dst_process_port = link->peer_port;
2509 - }
2510 - else {
2511 - dst_local_peer_unresolved = (link->direction_id != SOCKET_DIRECTION_LISTEN);
2512 - dst_process_port = (link->direction_id == SOCKET_DIRECTION_LISTEN) ? link->local_port : (link->peer_port ? link->peer_port : link->remote_port);
2513 - }
3748 +#undef NV_TOPOLOGY_V1_CORRELATION_STRING_VALUES
3749
2515 - dst_is_process = true;
2516 - topology_actor_id_for_process(ctx, dst_pid, dst_uid, dst_net_ns_inode, dst_process_name, dst_actor_id, sizeof(dst_actor_id));
2517 - topology_process_display_name(ctx, dst_process_name, dst_pid, dst_process_display_name, sizeof(dst_process_display_name));
2518 - if(dst_process_port)
2519 - snprintf(dst_process_port_name, sizeof(dst_process_port_name), "%u", dst_process_port);
2520 - else
2521 - snprintf(dst_process_port_name, sizeof(dst_process_port_name), "unknown");
3750 + topology_v1_values_start(wb);
3751 + for(size_t i = 0; i < rows_used; i++)
3752 + buffer_json_add_array_item_uint64(wb, rows[i].port);
3753 + topology_v1_values_end(wb);
3754
2523 - snprintf(link_display_name, sizeof(link_display_name), "%s:%s -> %s:%s",
2524 - process_display_name, local_bind_port, dst_process_display_name, dst_process_port_name);
2525 - }
2526 - else {
2527 - snprintf(dst_actor_id, sizeof(dst_actor_id), "%s", endpoint_actor_id);
2528 - snprintf(link_display_name, sizeof(link_display_name), "%s:%s -> %s",
2529 - process_display_name, local_bind_port, remote_endpoint_port);
2530 - }
3755 + buffer_json_array_close(wb);
3756 +}
3757
2532 - buffer_json_add_array_item_object(wb);
3758 +static void topology_v1_emit_correlation(BUFFER *wb, NV_TOPOLOGY_V1_PAYLOAD *payload) {
3759 + buffer_json_member_add_object(wb, "correlation");
3760 + {
3761 + buffer_json_member_add_object(wb, "rules");
3762 + {
3763 + buffer_json_member_add_object(wb, "socket_exact");
3764 {
2534 - buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2535 - buffer_json_member_add_string(wb, "protocol", link->protocol);
2536 - buffer_json_member_add_string(wb, "link_type", "socket");
2537 - buffer_json_member_add_string(wb, "direction", link->direction);
2538 - buffer_json_member_add_string(wb, "state", link->state);
2539 - buffer_json_member_add_string(wb, "src_actor_id", process_actor_id);
2540 - buffer_json_member_add_string(wb, "dst_actor_id", dst_actor_id);
2541 - buffer_json_member_add_datetime_rfc3339(wb, "discovered_at", ctx->now_ut, true);
2542 - buffer_json_member_add_datetime_rfc3339(wb, "last_seen", ctx->now_ut, true);
2543 -
2544 - buffer_json_member_add_object(wb, "src");
3765 + buffer_json_member_add_string(wb, "action", "absorb");
3766 + buffer_json_member_add_string(wb, "class", "resolve_loose_side");
3767 + buffer_json_member_add_uint64(wb, "priority", 1);
3768 + buffer_json_member_add_string(wb, "key_space", "network_socket");
3769 + buffer_json_member_add_array(wb, "key");
3770 {
2546 - buffer_json_member_add_object(wb, "match");
2547 - {
2548 - if(ctx->machine_guid[0])
2549 - buffer_json_member_add_string(wb, "netdata_machine_guid", ctx->machine_guid);
2550 - topology_add_single_item_string_array(wb, "hostnames", ctx->hostname);
2551 - topology_add_single_item_string_array(wb, "ip_addresses", link->local_ip);
2552 - }
3771 + buffer_json_add_array_item_object(wb);
3772 + buffer_json_member_add_string(wb, "column", "protocol");
3773 buffer_json_object_close(wb);
2554 -
2555 - buffer_json_member_add_object(wb, "attributes");
2556 - {
2557 - buffer_json_member_add_string(wb, "actor_type", "process");
2558 - if(ctx->options.processes_by_pid) {
2559 - buffer_json_member_add_uint64(wb, "pid", link->pid);
2560 - buffer_json_member_add_uint64(wb, "ppid", link->ppid);
2561 - buffer_json_member_add_uint64(wb, "uid", link->uid);
2562 - buffer_json_member_add_uint64(wb, "net_ns_inode", link->net_ns_inode);
2563 - }
2564 - buffer_json_member_add_string(wb, "process", link->process);
2565 - buffer_json_member_add_string(wb, "user", link->username);
2566 - buffer_json_member_add_string(wb, "namespace", link->namespace_type);
2567 - buffer_json_member_add_string(wb, "address_space", link->local_address_space);
2568 - buffer_json_member_add_uint64(wb, "port", link->local_port);
2569 - buffer_json_member_add_string(wb, "port_name", local_bind_port);
2570 - buffer_json_member_add_string(wb, "bind_ip", link->local_ip);
2571 - buffer_json_member_add_string(wb, "service_name", link->port_name);
2572 - buffer_json_member_add_string(wb, "display_name", process_display_name);
2573 - buffer_json_member_add_string(wb, "protocol_family", link->protocol_family);
2574 - if(link->cmdline[0])
2575 - buffer_json_member_add_string(wb, "cmdline", link->cmdline);
2576 - }
3774 + buffer_json_add_array_item_object(wb);
3775 + buffer_json_member_add_string(wb, "literal", ":");
3776 buffer_json_object_close(wb);
2578 - }
2579 - buffer_json_object_close(wb);
2580 -
2581 - buffer_json_member_add_object(wb, "dst");
2582 - {
2583 - if(dst_is_process)
2584 - topology_add_process_identity_match(wb, ctx, dst_pid, dst_uid, dst_net_ns_inode, dst_process_name);
2585 - else
2586 - topology_add_remote_match(wb, link->remote_ip);
2587 -
2588 - buffer_json_member_add_object(wb, "attributes");
2589 - {
2590 - if(dst_is_process) {
2591 - buffer_json_member_add_string(wb, "actor_type", "process");
2592 - if(ctx->options.processes_by_pid) {
2593 - buffer_json_member_add_uint64(wb, "pid", dst_pid);
2594 - buffer_json_member_add_uint64(wb, "ppid", dst_ppid);
2595 - buffer_json_member_add_uint64(wb, "uid", dst_uid);
2596 - buffer_json_member_add_uint64(wb, "net_ns_inode", dst_net_ns_inode);
2597 - }
2598 - buffer_json_member_add_string(wb, "process", dst_process_name);
2599 - buffer_json_member_add_string(wb, "address_space", "self");
2600 - if(dst_process_port) {
2601 - buffer_json_member_add_uint64(wb, "port", dst_process_port);
2602 - buffer_json_member_add_string(wb, "port_name", dst_process_port_name);
2603 - }
2604 - buffer_json_member_add_string(wb, "display_name", dst_process_display_name);
2605 - if(dst_local_peer_unresolved)
2606 - buffer_json_member_add_boolean(wb, "unresolved_local_peer", true);
2607 - }
2608 - else {
2609 - buffer_json_member_add_string(wb, "actor_type", "endpoint");
2610 - buffer_json_member_add_string(wb, "address_space", link->remote_address_space);
2611 - buffer_json_member_add_uint64(wb, "port", link->remote_port);
2612 - buffer_json_member_add_string(wb, "port_name", remote_endpoint_port);
2613 - buffer_json_member_add_string(wb, "display_name", link->remote_ip);
2614 - }
2615 - }
3777 + buffer_json_add_array_item_object(wb);
3778 + buffer_json_member_add_string(wb, "column", "address_space");
3779 + buffer_json_object_close(wb);
3780 + buffer_json_add_array_item_object(wb);
3781 + buffer_json_member_add_string(wb, "literal", ":");
3782 + buffer_json_object_close(wb);
3783 + buffer_json_add_array_item_object(wb);
3784 + buffer_json_member_add_string(wb, "column", "ip");
3785 + buffer_json_object_close(wb);
3786 + buffer_json_add_array_item_object(wb);
3787 + buffer_json_member_add_string(wb, "literal", ":");
3788 + buffer_json_object_close(wb);
3789 + buffer_json_add_array_item_object(wb);
3790 + buffer_json_member_add_string(wb, "column", "port");
3791 buffer_json_object_close(wb);
3792 }
2618 - buffer_json_object_close(wb);
2619 -
2620 - buffer_json_member_add_object(wb, "metrics");
2621 - {
2622 - buffer_json_member_add_uint64(wb, "socket_count", link->sockets);
2623 - buffer_json_member_add_uint64(wb, "retransmissions", link->retransmissions);
2624 - buffer_json_member_add_double(wb, "rtt_ms_max", (double)link->max_rtt_usec / (double)USEC_PER_MS);
2625 - buffer_json_member_add_double(wb, "recv_rtt_ms_max", (double)link->max_rcv_rtt_usec / (double)USEC_PER_MS);
2626 - buffer_json_member_add_string(wb, "display_name", link_display_name);
2627 - }
2628 - buffer_json_object_close(wb);
2629 -
2630 - buffer_json_member_add_object(wb, "labels");
2631 - {
2632 - buffer_json_member_add_string(wb, "protocol", link->protocol);
2633 - buffer_json_member_add_string(wb, "direction", link->direction);
2634 - buffer_json_member_add_string(wb, "state", link->state);
2635 - buffer_json_member_add_string(wb, "process", link->process);
2636 - buffer_json_member_add_string(wb, "user", link->username);
2637 - buffer_json_member_add_string(wb, "namespace", link->namespace_type);
2638 - buffer_json_member_add_string(wb, "protocol_family", link->protocol_family);
2639 - buffer_json_member_add_string(wb, "local_address_space", link->local_address_space);
2640 - buffer_json_member_add_string(wb, "remote_address_space", link->remote_address_space);
2641 - buffer_json_member_add_string(wb, "port_name", local_bind_port);
2642 - buffer_json_member_add_string(wb, "bind_ip", link->local_ip);
2643 - buffer_json_member_add_string(wb, "service_name", link->port_name);
2644 - buffer_json_member_add_string(wb, "link_class", "socket");
2645 - buffer_json_member_add_string(wb, "socket_kind", link->direction);
2646 - buffer_json_member_add_string(wb, "render_intent", "socket");
2647 - buffer_json_member_add_string(wb, "display_name", link_display_name);
2648 - }
2649 - buffer_json_object_close(wb);
3793 + buffer_json_array_close(wb);
3794 + buffer_json_member_add_array(wb, "point_actor_types");
3795 + buffer_json_add_array_item_string(wb, "endpoint");
3796 + buffer_json_array_close(wb);
3797 + buffer_json_member_add_array(wb, "claim_actor_types");
3798 + buffer_json_add_array_item_string(wb, "process");
3799 + buffer_json_array_close(wb);
3800 + buffer_json_member_add_array(wb, "correlation_link_types");
3801 + buffer_json_add_array_item_string(wb, "endpoint_socket");
3802 + buffer_json_array_close(wb);
3803 + buffer_json_member_add_string(wb, "output_link_type", "correlated_socket");
3804 }
3805 buffer_json_object_close(wb);
3806 }
2653 - dfe_done(link);
2654 - }
2655 - buffer_json_array_close(wb);
3807 + buffer_json_object_close(wb);
3808
2657 - buffer_json_member_add_object(wb, "stats");
2658 - {
2659 - size_t links_total = state->socket_link_count + state->ownership_link_count;
2660 - buffer_json_member_add_string(wb, "processes_mode", ctx->options.processes_by_pid ? "by_pid" : "by_name");
2661 - buffer_json_member_add_boolean(wb, "sockets_listening", ctx->options.sockets_listening);
2662 - buffer_json_member_add_boolean(wb, "sockets_local", ctx->options.sockets_local);
2663 - buffer_json_member_add_boolean(wb, "sockets_inbound", ctx->options.sockets_inbound);
2664 - buffer_json_member_add_boolean(wb, "sockets_outbound", ctx->options.sockets_outbound);
2665 - buffer_json_member_add_string(wb, "endpoints_mode_selected", "by_ip");
2666 - buffer_json_member_add_string(wb, "endpoints_mode_effective", "by_ip");
2667 - buffer_json_member_add_boolean(wb, "protocol_ipv4_tcp", ctx->options.protocols_ipv4_tcp);
2668 - buffer_json_member_add_boolean(wb, "protocol_ipv6_tcp", ctx->options.protocols_ipv6_tcp);
2669 - buffer_json_member_add_boolean(wb, "protocol_ipv4_udp", ctx->options.protocols_ipv4_udp);
2670 - buffer_json_member_add_boolean(wb, "protocol_ipv6_udp", ctx->options.protocols_ipv6_udp);
2671 - buffer_json_member_add_uint64(wb, "sockets_total", ctx->sockets_total);
2672 - buffer_json_member_add_uint64(wb, "sockets_without_remote_endpoint", ctx->skipped_sockets);
2673 - buffer_json_member_add_uint64(wb, "local_process_actors", state->process_actor_count);
2674 - buffer_json_member_add_uint64(wb, "endpoint_actors", state->endpoint_actor_count);
2675 - buffer_json_member_add_uint64(wb, "socket_links", state->socket_link_count);
2676 - buffer_json_member_add_uint64(wb, "ownership_links", state->ownership_link_count);
2677 - buffer_json_member_add_uint64(wb, "links_total", links_total);
3809 + buffer_json_member_add_object(wb, "points");
3810 + topology_v1_emit_correlation_table(wb, payload->correlation_points, payload->correlation_points_used);
3811 + buffer_json_object_close(wb);
3812 +
3813 + buffer_json_member_add_object(wb, "claims");
3814 + topology_v1_emit_correlation_table(wb, payload->correlation_claims, payload->correlation_claims_used);
3815 + buffer_json_object_close(wb);
3816 }
3817 buffer_json_object_close(wb);
3818 }
@@ -2686,18 +3824,114 @@ static void topology_write_data(BUFFER *wb, const NV_TOPOLOGY_CONTEXT *ctx) {
3824 NV_TOPOLOGY_RENDER_STATE state;
3825 topology_render_state_init(&state, ctx);
3826
3827 + NV_TOPOLOGY_V1_PAYLOAD topology = {
3828 + .actor_index = dictionary_create_advanced(
3829 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
3830 + NULL, sizeof(uint64_t)),
3831 + .graph_link_index = dictionary_create_advanced(
3832 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
3833 + NULL, sizeof(uint64_t)),
3834 + .connection_index = dictionary_create_advanced(
3835 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
3836 + NULL, sizeof(uint64_t)),
3837 + .port_index = dictionary_create_advanced(
3838 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
3839 + NULL, sizeof(uint64_t)),
3840 + .correlation_point_index = dictionary_create_advanced(
3841 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
3842 + NULL, sizeof(bool)),
3843 + .correlation_claim_index = dictionary_create_advanced(
3844 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
3845 + NULL, sizeof(bool)),
3846 + };
3847 +
3848 + if(!topology.actor_index || !topology.graph_link_index || !topology.connection_index || !topology.port_index ||
3849 + !topology.correlation_point_index || !topology.correlation_claim_index) {
3850 + topology_v1_free(&topology);
3851 + return;
3852 + }
3853 +
3854 + topology_v1_collect_actors(ctx, &state, &topology);
3855 + topology_v1_collect_links(ctx, &state, &topology);
3856 +
3857 buffer_json_member_add_object(wb, "data");
3858 {
3859 buffer_json_member_add_string(wb, "schema_version", NETWORK_TOPOLOGY_SCHEMA_VERSION);
2692 - buffer_json_member_add_string(wb, "source", NETWORK_TOPOLOGY_SOURCE);
2693 - buffer_json_member_add_string(wb, "layer", NETWORK_TOPOLOGY_LAYER);
2694 - buffer_json_member_add_string(wb, "agent_id", ctx->machine_guid[0] ? ctx->machine_guid : ctx->hostname);
3860 + buffer_json_member_add_object(wb, "producer");
3861 + {
3862 + buffer_json_member_add_string(wb, "source", NETWORK_TOPOLOGY_SOURCE);
3863 + buffer_json_member_add_string(wb, "instance", ctx->machine_guid[0] ? ctx->machine_guid : ctx->hostname);
3864 + if(ctx->machine_guid[0])
3865 + buffer_json_member_add_string(wb, "machine_guid", ctx->machine_guid);
3866 + buffer_json_member_add_string(wb, "plugin", "network-viewer.plugin");
3867 + buffer_json_member_add_array(wb, "capabilities");
3868 + buffer_json_add_array_item_string(wb, "topology-v1");
3869 + buffer_json_array_close(wb);
3870 + }
3871 + buffer_json_object_close(wb);
3872 buffer_json_member_add_datetime_rfc3339(wb, "collected_at", ctx->now_ut, true);
3873
2697 - topology_write_actors(wb, ctx, &state);
2698 - topology_write_links_and_stats(wb, ctx, &state);
3874 + buffer_json_member_add_object(wb, "view");
3875 + {
3876 + buffer_json_member_add_string(wb, "id", "network-connections");
3877 + buffer_json_member_add_string(wb, "scope", ctx->options.processes_by_pid ? "pid" : "process_name");
3878 + buffer_json_member_add_string(wb, "mode", ctx->options.detailed ? "detailed" : "aggregated");
3879 + buffer_json_member_add_array(wb, "supported_modes");
3880 + buffer_json_add_array_item_string(wb, "aggregated");
3881 + buffer_json_add_array_item_string(wb, "detailed");
3882 + buffer_json_array_close(wb);
3883 + buffer_json_member_add_array(wb, "group_by");
3884 + buffer_json_add_array_item_string(wb, ctx->options.processes_by_pid ? "pid" : "process_name");
3885 + buffer_json_array_close(wb);
3886 + }
3887 + buffer_json_object_close(wb);
3888 +
3889 + buffer_json_member_add_object(wb, "dictionaries");
3890 + {
3891 + buffer_json_member_add_array(wb, "strings");
3892 + buffer_json_array_close(wb);
3893 + }
3894 + buffer_json_object_close(wb);
3895 +
3896 + topology_v1_emit_type_registry(wb, ctx->options.detailed);
3897 + topology_v1_emit_presentation(wb);
3898 + topology_v1_emit_correlation(wb, &topology);
3899 + topology_v1_emit_actor_table(wb, &topology);
3900 + topology_v1_emit_link_table(wb, &topology);
3901 + topology_v1_emit_socket_port_table(wb, &topology);
3902 + if(ctx->options.detailed)
3903 + topology_v1_emit_socket_evidence_table(wb, &topology);
3904 +
3905 + buffer_json_member_add_object(wb, "stats");
3906 + {
3907 + buffer_json_member_add_string(wb, "processes_mode", ctx->options.processes_by_pid ? "by_pid" : "by_name");
3908 + buffer_json_member_add_string(wb, "mode", ctx->options.detailed ? "detailed" : "aggregated");
3909 + buffer_json_member_add_boolean(wb, "sockets_listening", ctx->options.sockets_listening);
3910 + buffer_json_member_add_boolean(wb, "sockets_inbound", ctx->options.sockets_inbound);
3911 + buffer_json_member_add_boolean(wb, "sockets_outbound", ctx->options.sockets_outbound);
3912 + buffer_json_member_add_string(wb, "endpoints_mode_selected", "by_ip");
3913 + buffer_json_member_add_string(wb, "endpoints_mode_effective", "by_ip");
3914 + buffer_json_member_add_boolean(wb, "protocol_ipv4_tcp", ctx->options.protocols_ipv4_tcp);
3915 + buffer_json_member_add_boolean(wb, "protocol_ipv6_tcp", ctx->options.protocols_ipv6_tcp);
3916 + buffer_json_member_add_boolean(wb, "protocol_ipv4_udp", ctx->options.protocols_ipv4_udp);
3917 + buffer_json_member_add_boolean(wb, "protocol_ipv6_udp", ctx->options.protocols_ipv6_udp);
3918 + buffer_json_member_add_uint64(wb, "sockets_total", ctx->sockets_total);
3919 + buffer_json_member_add_uint64(wb, "sockets_without_remote_endpoint", ctx->skipped_sockets);
3920 + buffer_json_member_add_uint64(wb, "actors", topology.actors_used);
3921 + buffer_json_member_add_uint64(wb, "local_process_actors", state.process_actor_count);
3922 + buffer_json_member_add_uint64(wb, "endpoint_actors", state.endpoint_actor_count);
3923 + buffer_json_member_add_uint64(wb, "links", topology.links_used);
3924 + buffer_json_member_add_uint64(wb, "socket_evidence_rows", topology.evidence_used);
3925 + buffer_json_member_add_uint64(wb, "socket_port_rows", topology.ports_used);
3926 + buffer_json_member_add_uint64(wb, "correlation_points", topology.correlation_points_used);
3927 + buffer_json_member_add_uint64(wb, "correlation_claims", topology.correlation_claims_used);
3928 + buffer_json_member_add_uint64(wb, "ownership_links", state.ownership_link_count);
3929 + }
3930 + buffer_json_object_close(wb);
3931 }
3932 buffer_json_object_close(wb);
3933 +
3934 + topology_v1_free(&topology);
3935 }
3936
3937 static void network_viewer_topology_function(
@@ -2716,7 +3950,6 @@ static void network_viewer_topology_function(
3950 buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
3951
3952 topology_write_response_metadata(wb);
2719 - topology_write_presentation(wb);
3953
3954 NV_TOPOLOGY_CONTEXT ctx;
3955 bool ctx_ready = topology_prepare_context(&ctx, now_ut, &options);
src/go/pkg/topology/v1/dictionary.go new
+45
@@ -0,0 +1,45 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package topologyv1
4 +
5 +type StringDictionary struct {
6 + values []string
7 + index map[string]int
8 +}
9 +
10 +func NewStringDictionary(values ...string) *StringDictionary {
11 + dict := &StringDictionary{
12 + index: make(map[string]int, len(values)),
13 + }
14 + for _, value := range values {
15 + dict.Ref(value)
16 + }
17 + return dict
18 +}
19 +
20 +func (dict *StringDictionary) Ref(value string) int {
21 + if dict == nil {
22 + panic("topologyv1.StringDictionary.Ref called on nil dictionary")
23 + }
24 + if dict.index == nil {
25 + dict.index = make(map[string]int)
26 + }
27 + if index, ok := dict.index[value]; ok {
28 + return index
29 + }
30 + index := len(dict.values)
31 + dict.values = append(dict.values, value)
32 + dict.index[value] = index
33 + return index
34 +}
35 +
36 +func (dict *StringDictionary) Values() []any {
37 + if dict == nil || len(dict.values) == 0 {
38 + return []any{}
39 + }
40 + values := make([]any, len(dict.values))
41 + for index, value := range dict.values {
42 + values[index] = value
43 + }
44 + return values
45 +}
src/go/pkg/topology/v1/response_test.go new
+1642
@@ -0,0 +1,1642 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package topologyv1
4 +
5 +import (
6 + "encoding/json"
7 + "os"
8 + "path/filepath"
9 + "testing"
10 + "time"
11 +
12 + "github.com/santhosh-tekuri/jsonschema/v6"
13 + "github.com/stretchr/testify/assert"
14 + "github.com/stretchr/testify/require"
15 +)
16 +
17 +var testCollectedAt = time.Date(2026, 5, 9, 10, 0, 0, 0, time.UTC)
18 +
19 +func TestResponseValidatesAgainstSchemaAndSemanticChecks(t *testing.T) {
20 + payload := NewResponse(Data{
21 + Producer: Producer{
22 + Source: "test-topology",
23 + Instance: "test-instance",
24 + Plugin: "go-test",
25 + },
26 + CollectedAt: time.Date(2026, 5, 9, 10, 0, 0, 0, time.UTC),
27 + Dictionaries: Dictionaries{
28 + "strings": StringValues("node-a"),
29 + },
30 + Types: TypeRegistry{
31 + ActorTypes: map[string]ActorType{
32 + "node": {
33 + Layer: "node",
34 + Identity: []string{"id"},
35 + Presentation: &ActorPresentation{
36 + Label: "Node",
37 + Role: "actor",
38 + Icon: "server",
39 + ColorSlot: "primary",
40 + Annotation: &AnnotationPresentation{
41 + ColorSlot: "warning",
42 + Style: "ring",
43 + },
44 + Border: &BorderPresentation{
45 + Enabled: new(true),
46 + Style: "solid",
47 + },
48 + LabelPolicy: &LabelPolicy{
49 + Columns: []string{"name"},
50 + Fallback: "type_label",
51 + MaxLength: 80,
52 + Array: "reject",
53 + },
54 + Ports: &ActorPortsPresentation{
55 + ShowBullets: true,
56 + Sources: []PortSourcePresentation{
57 + {
58 + Source: "links",
59 + ActorColumn: "src",
60 + NameColumn: "port_name",
61 + DefaultType: "topology",
62 + },
63 + },
64 + },
65 + Hover: &HoverPresentation{
66 + Fields: []PresentationField{{Key: "name", Label: "Name"}},
67 + },
68 + Modal: &ModalPresentation{
69 + Enabled: new(true),
70 + Labels: &ModalLabelsPresentation{
71 + Table: "actor_labels",
72 + ActorColumn: "actor",
73 + KeyColumn: "key",
74 + ValueColumn: "value",
75 + },
76 + MiniTopology: &ModalMiniTopologyPresentation{
77 + Enabled: new(true),
78 + Depth: 1,
79 + },
80 + Sections: []ModalSection{
81 + {
82 + ID: "ports",
83 + Label: "Ports",
84 + Source: ModalSource{
85 + Kind: "actor_table",
86 + Table: "ports",
87 + },
88 + OwnerFilter: &ModalOwnerFilter{
89 + Mode: "actor_column",
90 + ActorColumn: "actor",
91 + },
92 + Columns: []ModalColumn{
93 + {
94 + ID: "neighbors",
95 + Label: "Neighbors",
96 + Projection: ModalProjection{
97 + Kind: "direct",
98 + Column: "neighbors",
99 + },
100 + Cell: "debug_json",
101 + Visibility: "debug",
102 + },
103 + },
104 + },
105 + {
106 + ID: "paths",
107 + Label: "Paths",
108 + Source: ModalSource{
109 + Kind: "actor_table",
110 + Table: "path",
111 + },
112 + OwnerFilter: &ModalOwnerFilter{
113 + Mode: "actor_column",
114 + ActorColumn: "actor",
115 + },
116 + Columns: []ModalColumn{
117 + {
118 + ID: "path_actor",
119 + Label: "Path actor",
120 + Projection: ModalProjection{
121 + Kind: "actor_ref_label",
122 + ActorColumn: "path_actor",
123 + },
124 + Cell: "actor_link",
125 + },
126 + {
127 + ID: "path_index",
128 + Label: "Hop",
129 + Projection: ModalProjection{
130 + Kind: "direct",
131 + Column: "path_index",
132 + },
133 + Cell: "number",
134 + },
135 + {
136 + ID: "source",
137 + Label: "Source",
138 + Projection: ModalProjection{
139 + Kind: "const",
140 + Value: "test",
141 + },
142 + Cell: "text",
143 + Visibility: "expanded",
144 + },
145 + },
146 + Sort: &ModalSort{Column: "path_index", Direction: "asc"},
147 + },
148 + },
149 + },
150 + },
151 + },
152 + },
153 + LinkTypes: map[string]LinkType{
154 + "dependency": {
155 + Orientation: "directed",
156 + DirectionRole: "dependency",
157 + Aggregation: LinkAggregation{
158 + Direction: "preserve",
159 + },
160 + Presentation: &LinkPresentation{
161 + Label: "Dependency",
162 + ColorSlot: "primary",
163 + LineStyle: "solid",
164 + Width: "normal",
165 + Curve: "auto",
166 + Arrow: "forward",
167 + Variable: &LinkVariablePresentation{
168 + Channel: "width",
169 + ScaleKey: "requests",
170 + ValueColumn: "weight",
171 + Min: "normal",
172 + Max: "emphasis",
173 + },
174 + Hover: &HoverPresentation{
175 + Fields: []PresentationField{{Key: "weight", Label: "Requests"}},
176 + },
177 + Layout: &LinkLayoutPresentation{
178 + Strength: "weaker",
179 + Distance: "farther",
180 + },
181 + },
182 + },
183 + },
184 + PortTypes: map[string]PortType{
185 + "topology": {
186 + Presentation: &PortPresentation{
187 + Label: "Topology",
188 + ColorSlot: "primary",
189 + },
190 + },
191 + },
192 + TableTypes: map[string]TableType{
193 + "actor_labels": {
194 + Role: "actor_inventory",
195 + Owner: "actor",
196 + Aggregation: "set",
197 + Columns: []Column{
198 + NewColumn("actor", "actor_ref"),
199 + NewColumn("key", "string"),
200 + NewColumn("value", "string"),
201 + NewColumn("source", "string", WithNullable()),
202 + NewColumn("kind", "string", WithNullable()),
203 + NewColumn("value_index", "uint", WithNullable()),
204 + },
205 + },
206 + "port_detail": {
207 + Role: "actor_detail",
208 + Owner: "actor",
209 + Aggregation: "append",
210 + Columns: []Column{
211 + NewColumn("actor", "actor_ref"),
212 + NewColumn("neighbors", "json", WithNullable()),
213 + },
214 + },
215 + "path_detail": {
216 + Role: "actor_detail",
217 + Owner: "actor",
218 + Aggregation: "append",
219 + Columns: []Column{
220 + NewColumn("actor", "actor_ref"),
221 + NewColumn("path_actor", "actor_ref"),
222 + NewColumn("path_index", "uint"),
223 + },
224 + },
225 + },
226 + },
227 + Presentation: &Presentation{
228 + ProfileVersion: "test",
229 + Selection: &SelectionPresentation{
230 + ActorClick: &ActorClickPresentation{
231 + Mode: "highlight_path",
232 + PathTable: "path",
233 + PathOwnerColumn: "actor",
234 + PathActorColumn: "path_actor",
235 + PathOrderColumn: "path_index",
236 + },
237 + },
238 + Legend: &PresentationLegend{
239 + Actors: []LegendEntry{{Type: "node", Label: "Node"}},
240 + Links: []LegendEntry{{Type: "dependency", Label: "Dependency"}},
241 + Ports: []LegendEntry{{Type: "topology", Label: "Topology"}},
242 + },
243 + PortFields: []PresentationField{{Key: "type", Label: "Type"}},
244 + ScaleKeys: map[string]ScaleKeyPresentation{
245 + "requests": {Label: "Requests", Unit: "count"},
246 + },
247 + },
248 + Correlation: &Correlation{
249 + Rules: map[string]CorrelationRule{
250 + "node_name": {
251 + Action: "link",
252 + Priority: 10,
253 + KeySpace: "node_name",
254 + Key: []CorrelationKeyPart{{Column: "name"}},
255 + PointActorTypes: []string{"node"},
256 + ClaimActorTypes: []string{"node"},
257 + OutputLinkType: "dependency",
258 + },
259 + },
260 + Points: &Table{
261 + Rows: 1,
262 + Columns: []Column{
263 + NewColumn("actor", "actor_ref"),
264 + NewColumn("rule", "string"),
265 + NewColumn("name", "string_ref", WithDictionary("strings")),
266 + },
267 + Values: []ColumnEncoding{
268 + Values(0),
269 + Const("node_name"),
270 + Values(0),
271 + },
272 + },
273 + Claims: &Table{
274 + Rows: 1,
275 + Columns: []Column{
276 + NewColumn("actor", "actor_ref"),
277 + NewColumn("rule", "string"),
278 + NewColumn("name", "string_ref", WithDictionary("strings")),
279 + },
280 + Values: []ColumnEncoding{
281 + Values(0),
282 + Const("node_name"),
283 + Values(0),
284 + },
285 + },
286 + },
287 + Actors: MustTable(1,
288 + []Column{
289 + NewColumn("id", "string", WithRole("identity")),
290 + NewColumn("type", "string"),
291 + NewColumn("name", "string_ref", WithDictionary("strings")),
292 + },
293 + []ColumnEncoding{
294 + Values("node-a"),
295 + Const("node"),
296 + Values(0),
297 + },
298 + ),
299 + Links: MustTable(1,
300 + []Column{
301 + NewColumn("src", "actor_ref"),
302 + NewColumn("dst", "actor_ref"),
303 + NewColumn("type", "string"),
304 + NewColumn("port_name", "string"),
305 + NewColumn("weight", "uint", WithRole("metric"), WithAggregation("sum")),
306 + },
307 + []ColumnEncoding{
308 + Values(0),
309 + Values(0),
310 + Const("dependency"),
311 + Const("node-a"),
312 + Values(7),
313 + },
314 + ),
315 + Tables: &DetailTables{
316 + Actor: map[string]DetailTable{
317 + "actor_labels": {
318 + Type: "actor_labels",
319 + Table: MustTable(1,
320 + []Column{
321 + NewColumn("actor", "actor_ref"),
322 + NewColumn("key", "string"),
323 + NewColumn("value", "string"),
324 + NewColumn("source", "string", WithNullable()),
325 + NewColumn("kind", "string", WithNullable()),
326 + NewColumn("value_index", "uint", WithNullable()),
327 + },
328 + []ColumnEncoding{
329 + Values(0),
330 + Const("hostname"),
331 + Values("node-a"),
332 + Const("producer"),
333 + Const("identity"),
334 + Values(nil),
335 + },
336 + ),
337 + },
338 + "ports": {
339 + Type: "port_detail",
340 + Table: MustTable(1,
341 + []Column{
342 + NewColumn("actor", "actor_ref"),
343 + NewColumn("neighbors", "json", WithNullable()),
344 + },
345 + []ColumnEncoding{
346 + Values(0),
347 + Values(map[string]any{
348 + "neighbors": []any{
349 + map[string]any{
350 + "protocol": "lldp",
351 + "remote_port": "eth0",
352 + },
353 + },
354 + }),
355 + },
356 + ),
357 + },
358 + "path": {
359 + Type: "path_detail",
360 + Table: MustTable(1,
361 + []Column{
362 + NewColumn("actor", "actor_ref"),
363 + NewColumn("path_actor", "actor_ref"),
364 + NewColumn("path_index", "uint"),
365 + },
366 + []ColumnEncoding{
367 + Values(0),
368 + Values(0),
369 + Values(0),
370 + },
371 + ),
372 + },
373 + },
374 + },
375 + })
376 +
377 + payloadBytes, err := json.Marshal(payload)
378 + require.NoError(t, err)
379 + assert.Contains(t, string(payloadBytes), `"schema_version":"netdata.topology.v1"`)
380 +
381 + validateAgainstTopologySchema(t, payloadBytes)
382 +
383 + var decoded any
384 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
385 + require.NoError(t, ValidateDecodedResponse(decoded))
386 +}
387 +
388 +func TestValidateDecodedResponseRejectsInvalidPresentationReferences(t *testing.T) {
389 + err := validateResponseData(t, minimalValidationData(&ActorPresentation{
390 + LabelPolicy: &LabelPolicy{Columns: []string{"missing"}},
391 + }))
392 +
393 + require.Error(t, err)
394 + assert.Contains(t, err.Error(), "label_policy.columns[0] references unknown actor column")
395 +}
396 +
397 +func TestValidateDecodedResponseRejectsNonDisplayLabelColumn(t *testing.T) {
398 + actors := minimalActorTable(
399 + []Column{NewColumn("metadata", "json", WithNullable())},
400 + []ColumnEncoding{Values(map[string]any{"labels": []any{"not-a-label"}})},
401 + )
402 + err := validateResponseData(t, minimalValidationData(
403 + &ActorPresentation{LabelPolicy: &LabelPolicy{Columns: []string{"metadata"}}},
404 + withActors(actors),
405 + ))
406 +
407 + require.Error(t, err)
408 + assert.Contains(t, err.Error(), "references non-display actor column")
409 +}
410 +
411 +func TestValidateDecodedResponseRejectsMissingPortSourceTable(t *testing.T) {
412 + err := validateResponseData(t, minimalValidationData(
413 + &ActorPresentation{
414 + Ports: &ActorPortsPresentation{
415 + ShowBullets: true,
416 + Sources: []PortSourcePresentation{
417 + {
418 + Source: "actor_table",
419 + Table: "missing_ports",
420 + ActorColumn: "actor",
421 + NameColumn: "name",
422 + DefaultType: "topology",
423 + },
424 + },
425 + },
426 + },
427 + withPortTypes(map[string]PortType{
428 + "topology": {Presentation: &PortPresentation{Label: "Topology"}},
429 + }),
430 + ))
431 +
432 + require.Error(t, err)
433 + assert.Contains(t, err.Error(), "references unknown actor table")
434 +}
435 +
436 +func TestValidateDecodedResponseRejectsMissingModalActorTable(t *testing.T) {
437 + err := validateResponseData(t, minimalValidationData(
438 + &ActorPresentation{
439 + Modal: &ModalPresentation{
440 + Sections: []ModalSection{
441 + {
442 + ID: "missing",
443 + Label: "Missing",
444 + Source: ModalSource{
445 + Kind: "actor_table",
446 + Table: "missing_table",
447 + },
448 + Columns: []ModalColumn{
449 + {
450 + ID: "name",
451 + Label: "Name",
452 + Projection: ModalProjection{
453 + Kind: "direct",
454 + Column: "name",
455 + },
456 + },
457 + },
458 + },
459 + },
460 + },
461 + },
462 + withActors(minimalActorTable(
463 + []Column{NewColumn("name", "string_ref", WithDictionary("strings"))},
464 + []ColumnEncoding{Values(0)},
465 + )),
466 + withLinks(dependencyLinkTable(0)),
467 + ))
468 +
469 + require.Error(t, err)
470 + assert.Contains(t, err.Error(), "modal.sections[0].source.table references unknown actor table")
471 +}
472 +
473 +func TestValidateDecodedResponseRejectsInvalidModalProjectionShapes(t *testing.T) {
474 + cases := map[string]struct {
475 + projection ModalProjection
476 + want string
477 + }{
478 + "direct missing column": {
479 + projection: ModalProjection{Kind: "direct"},
480 + want: "column is required",
481 + },
482 + "actor label missing actor column": {
483 + projection: ModalProjection{Kind: "actor_ref_label"},
484 + want: "actor_column is required",
485 + },
486 + "opposite actor missing actor columns": {
487 + projection: ModalProjection{Kind: "opposite_actor"},
488 + want: "src_actor_column is required",
489 + },
490 + "const missing value": {
491 + projection: ModalProjection{Kind: "const"},
492 + want: "value is required when kind is const",
493 + },
494 + "selected endpoint missing local side": {
495 + projection: ModalProjection{Kind: "selected_side_endpoint", SrcActorColumn: "src", DstActorColumn: "dst", RemoteIPColumn: "remote_ip"},
496 + want: "requires local_ip_column or local_port_column",
497 + },
498 + "selected endpoint missing remote side": {
499 + projection: ModalProjection{Kind: "selected_side_endpoint", SrcActorColumn: "src", DstActorColumn: "dst", LocalIPColumn: "local_ip"},
500 + want: "requires remote_ip_column or remote_port_column",
501 + },
502 + "selected endpoint empty local side": {
503 + projection: ModalProjection{Kind: "selected_side_endpoint", SrcActorColumn: "src", DstActorColumn: "dst", LocalIPColumn: "", RemoteIPColumn: "remote_ip"},
504 + want: "requires local_ip_column or local_port_column",
505 + },
506 + "selected endpoint missing side actor columns": {
507 + projection: ModalProjection{Kind: "selected_side_endpoint", LocalIPColumn: "local_ip", RemoteIPColumn: "remote_ip"},
508 + want: "src_actor_column is required",
509 + },
510 + "label lookup missing key": {
511 + projection: ModalProjection{Kind: "label_lookup", ActorColumn: "src"},
512 + want: "label_key is required when kind is label_lookup",
513 + },
514 + "json path missing path": {
515 + projection: ModalProjection{Kind: "json_path", Column: "metadata"},
516 + want: "path is required when kind is json_path",
517 + },
518 + "json path missing column": {
519 + projection: ModalProjection{Kind: "json_path", Path: "$.state"},
520 + want: "column is required",
521 + },
522 + "coalesce missing columns": {
523 + projection: ModalProjection{Kind: "coalesce"},
524 + want: "columns is required when kind is coalesce",
525 + },
526 + "formatted endpoint missing endpoint columns": {
527 + projection: ModalProjection{Kind: "formatted_endpoint", ProtocolColumn: "type"},
528 + want: "requires ip_column or port_column",
529 + },
530 + }
531 +
532 + for name, tc := range cases {
533 + t.Run(name, func(t *testing.T) {
534 + data := minimalValidationData(
535 + &ActorPresentation{
536 + Modal: &ModalPresentation{
537 + Sections: []ModalSection{
538 + {
539 + ID: "links",
540 + Label: "Links",
541 + Source: ModalSource{Kind: "links"},
542 + Columns: []ModalColumn{
543 + {
544 + ID: "endpoint",
545 + Label: "Endpoint",
546 + Projection: tc.projection,
547 + },
548 + },
549 + },
550 + },
551 + },
552 + },
553 + withLinks(dependencyLinkTableWith(1,
554 + []Column{
555 + NewColumn("local_ip", "string", WithNullable()),
556 + NewColumn("remote_ip", "string", WithNullable()),
557 + NewColumn("metadata", "json", WithNullable()),
558 + },
559 + []ColumnEncoding{
560 + Values("10.0.0.1"),
561 + Values("10.0.0.2"),
562 + Values(map[string]any{"state": "open"}),
563 + },
564 + )),
565 + )
566 +
567 + payloadBytes, err := json.Marshal(NewResponse(data))
568 + require.NoError(t, err)
569 + require.Error(t, topologySchemaValidationError(t, payloadBytes))
570 +
571 + var decoded any
572 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
573 +
574 + err = ValidateDecodedResponse(decoded)
575 + require.Error(t, err)
576 + assert.Contains(t, err.Error(), tc.want)
577 + })
578 + }
579 +}
580 +
581 +func TestValidateDecodedResponseRejectsInvalidModalPresentationSemantics(t *testing.T) {
582 + basePayload := func(actorPresentation map[string]any) map[string]any {
583 + return map[string]any{
584 + "status": float64(200),
585 + "type": "topology",
586 + "data": map[string]any{
587 + "schema_version": SchemaVersion,
588 + "dictionaries": map[string]any{},
589 + "types": map[string]any{
590 + "actor_types": map[string]any{
591 + "node": map[string]any{
592 + "layer": "node",
593 + "identity": []any{"id"},
594 + "presentation": actorPresentation,
595 + },
596 + },
597 + "link_types": map[string]any{
598 + "dependency": map[string]any{
599 + "orientation": "directed",
600 + "direction_role": "dependency",
601 + "aggregation": map[string]any{"direction": "preserve"},
602 + },
603 + },
604 + "port_types": map[string]any{},
605 + "table_types": map[string]any{},
606 + },
607 + "actors": map[string]any{
608 + "rows": float64(1),
609 + "columns": []any{
610 + map[string]any{"id": "id", "type": "string", "role": "identity"},
611 + map[string]any{"id": "type", "type": "string"},
612 + },
613 + "values": []any{
614 + map[string]any{"codec": "const", "value": "node-a"},
615 + map[string]any{"codec": "const", "value": "node"},
616 + },
617 + },
618 + "links": map[string]any{
619 + "rows": float64(1),
620 + "columns": []any{
621 + map[string]any{"id": "src", "type": "actor_ref"},
622 + map[string]any{"id": "dst", "type": "actor_ref"},
623 + map[string]any{"id": "type", "type": "string"},
624 + },
625 + "values": []any{
626 + map[string]any{"codec": "const", "value": float64(0)},
627 + map[string]any{"codec": "const", "value": float64(0)},
628 + map[string]any{"codec": "const", "value": "dependency"},
629 + },
630 + },
631 + },
632 + }
633 + }
634 + validSection := func() map[string]any {
635 + return map[string]any{
636 + "id": "links",
637 + "label": "Links",
638 + "source": map[string]any{"kind": "links"},
639 + "columns": []any{
640 + map[string]any{
641 + "id": "type",
642 + "label": "Type",
643 + "projection": map[string]any{"kind": "direct", "column": "type"},
644 + },
645 + },
646 + }
647 + }
648 +
649 + cases := map[string]struct {
650 + presentation map[string]any
651 + want string
652 + }{
653 + "empty actor type label": {
654 + presentation: map[string]any{"label": ""},
655 + want: "presentation.label is required",
656 + },
657 + "mini topology non-integer depth": {
658 + presentation: map[string]any{
659 + "modal": map[string]any{
660 + "mini_topology": map[string]any{"depth": "1"},
661 + },
662 + },
663 + want: "mini_topology.depth is not an integer",
664 + },
665 + "modal is not an object": {
666 + presentation: map[string]any{
667 + "modal": "invalid",
668 + },
669 + want: "modal is not an object",
670 + },
671 + "missing modal columns": {
672 + presentation: map[string]any{
673 + "modal": map[string]any{
674 + "sections": []any{
675 + map[string]any{
676 + "id": "links",
677 + "label": "Links",
678 + "source": map[string]any{"kind": "links"},
679 + },
680 + },
681 + },
682 + },
683 + want: "columns is not an array",
684 + },
685 + "empty modal columns": {
686 + presentation: map[string]any{
687 + "modal": map[string]any{
688 + "sections": []any{
689 + map[string]any{
690 + "id": "links",
691 + "label": "Links",
692 + "source": map[string]any{"kind": "links"},
693 + "columns": []any{},
694 + },
695 + },
696 + },
697 + },
698 + want: "columns must not be empty",
699 + },
700 + "duplicate modal section id": {
701 + presentation: map[string]any{
702 + "modal": map[string]any{
703 + "sections": []any{
704 + validSection(),
705 + validSection(),
706 + },
707 + },
708 + },
709 + want: "duplicates modal section id",
710 + },
711 + "duplicate modal column id": {
712 + presentation: map[string]any{
713 + "modal": map[string]any{
714 + "sections": []any{
715 + func() map[string]any {
716 + section := validSection()
717 + section["columns"] = []any{
718 + map[string]any{
719 + "id": "type",
720 + "label": "Type",
721 + "projection": map[string]any{"kind": "direct", "column": "type"},
722 + },
723 + map[string]any{
724 + "id": "type",
725 + "label": "Type again",
726 + "projection": map[string]any{"kind": "direct", "column": "type"},
727 + },
728 + }
729 + return section
730 + }(),
731 + },
732 + },
733 + },
734 + want: "duplicates modal column id",
735 + },
736 + "sort references unknown modal column": {
737 + presentation: map[string]any{
738 + "modal": map[string]any{
739 + "sections": []any{
740 + func() map[string]any {
741 + section := validSection()
742 + section["sort"] = map[string]any{"column": "missing"}
743 + return section
744 + }(),
745 + },
746 + },
747 + },
748 + want: "sort.column references unknown modal column",
749 + },
750 + }
751 +
752 + for name, tc := range cases {
753 + t.Run(name, func(t *testing.T) {
754 + err := ValidateDecodedResponse(basePayload(tc.presentation))
755 + require.Error(t, err)
756 + assert.Contains(t, err.Error(), tc.want)
757 + })
758 + }
759 +}
760 +
761 +func TestValidateDecodedResponseRejectsInvalidZeroHeuristicContract(t *testing.T) {
762 + basePayload := func() map[string]any {
763 + return map[string]any{
764 + "status": float64(200),
765 + "type": "topology",
766 + "data": map[string]any{
767 + "schema_version": SchemaVersion,
768 + "dictionaries": map[string]any{},
769 + "types": map[string]any{
770 + "actor_types": map[string]any{
771 + "node": map[string]any{
772 + "layer": "node",
773 + "identity": []any{"id"},
774 + "search": map[string]any{
775 + "enabled": true,
776 + "columns": []any{"name"},
777 + },
778 + "presentation": map[string]any{
779 + "label": "Node",
780 + "size": map[string]any{"mode": "fixed", "scale": "normal"},
781 + "layout": map[string]any{"repulsion": "normal"},
782 + },
783 + },
784 + },
785 + "link_types": map[string]any{
786 + "dependency": map[string]any{
787 + "orientation": "directed",
788 + "direction_role": "dependency",
789 + "semantic_role": "traffic",
790 + "aggregation": map[string]any{"direction": "preserve"},
791 + },
792 + },
793 + },
794 + "actors": map[string]any{
795 + "rows": float64(1),
796 + "columns": []any{
797 + map[string]any{"id": "id", "type": "string", "role": "identity"},
798 + map[string]any{"id": "type", "type": "string"},
799 + map[string]any{"id": "name", "type": "string"},
800 + },
801 + "values": []any{
802 + map[string]any{"codec": "const", "value": "node-a"},
803 + map[string]any{"codec": "const", "value": "node"},
804 + map[string]any{"codec": "const", "value": "Node A"},
805 + },
806 + },
807 + "links": map[string]any{
808 + "rows": float64(1),
809 + "columns": []any{
810 + map[string]any{"id": "src", "type": "actor_ref"},
811 + map[string]any{"id": "dst", "type": "actor_ref"},
812 + map[string]any{"id": "type", "type": "string"},
813 + },
814 + "values": []any{
815 + map[string]any{"codec": "const", "value": float64(0)},
816 + map[string]any{"codec": "const", "value": float64(0)},
817 + map[string]any{"codec": "const", "value": "dependency"},
818 + },
819 + },
820 + },
821 + }
822 + }
823 +
824 + cases := map[string]struct {
825 + mutate func(map[string]any)
826 + want string
827 + }{
828 + "invalid search column": {
829 + mutate: func(payload map[string]any) {
830 + actorType := payload["data"].(map[string]any)["types"].(map[string]any)["actor_types"].(map[string]any)["node"].(map[string]any)
831 + actorType["search"].(map[string]any)["columns"] = []any{"missing"}
832 + },
833 + want: "search.columns[0] references unknown actor column",
834 + },
835 + "invalid size scale": {
836 + mutate: func(payload map[string]any) {
837 + actorType := payload["data"].(map[string]any)["types"].(map[string]any)["actor_types"].(map[string]any)["node"].(map[string]any)
838 + presentation := actorType["presentation"].(map[string]any)
839 + presentation["size"].(map[string]any)["scale"] = "huge"
840 + },
841 + want: "presentation.size.scale has unsupported value",
842 + },
843 + "invalid layout repulsion": {
844 + mutate: func(payload map[string]any) {
845 + actorType := payload["data"].(map[string]any)["types"].(map[string]any)["actor_types"].(map[string]any)["node"].(map[string]any)
846 + presentation := actorType["presentation"].(map[string]any)
847 + presentation["layout"].(map[string]any)["repulsion"] = "huge"
848 + },
849 + want: "presentation.layout.repulsion has unsupported value",
850 + },
851 + "invalid semantic role": {
852 + mutate: func(payload map[string]any) {
853 + linkType := payload["data"].(map[string]any)["types"].(map[string]any)["link_types"].(map[string]any)["dependency"].(map[string]any)
854 + linkType["semantic_role"] = "protocol"
855 + },
856 + want: "semantic_role has unsupported value",
857 + },
858 + }
859 +
860 + for name, tc := range cases {
861 + t.Run(name, func(t *testing.T) {
862 + payload := basePayload()
863 + tc.mutate(payload)
864 + err := ValidateDecodedResponse(payload)
865 + require.Error(t, err)
866 + assert.Contains(t, err.Error(), tc.want)
867 + })
868 + }
869 +}
870 +
871 +func TestValidateDecodedResponseRejectsEmptyPresentationLabels(t *testing.T) {
872 + basePayload := func(types map[string]any) map[string]any {
873 + return map[string]any{
874 + "status": float64(200),
875 + "type": "topology",
876 + "data": map[string]any{
877 + "schema_version": SchemaVersion,
878 + "dictionaries": map[string]any{},
879 + "types": types,
880 + "actors": map[string]any{
881 + "rows": float64(1),
882 + "columns": []any{
883 + map[string]any{"id": "id", "type": "string", "role": "identity"},
884 + map[string]any{"id": "type", "type": "string"},
885 + },
886 + "values": []any{
887 + map[string]any{"codec": "const", "value": "node-a"},
888 + map[string]any{"codec": "const", "value": "node"},
889 + },
890 + },
891 + "links": map[string]any{
892 + "rows": float64(0),
893 + "columns": []any{
894 + map[string]any{"id": "type", "type": "string"},
895 + },
896 + "values": []any{
897 + map[string]any{"codec": "const", "value": "dependency"},
898 + },
899 + },
900 + },
901 + }
902 + }
903 + defaultTypes := func() map[string]any {
904 + return map[string]any{
905 + "actor_types": map[string]any{
906 + "node": map[string]any{"layer": "node", "identity": []any{"id"}},
907 + },
908 + "link_types": map[string]any{
909 + "dependency": map[string]any{
910 + "orientation": "directed",
911 + "direction_role": "dependency",
912 + "aggregation": map[string]any{"direction": "preserve"},
913 + },
914 + },
915 + "port_types": map[string]any{},
916 + "table_types": map[string]any{
917 + "actor_labels": map[string]any{
918 + "role": "actor_inventory",
919 + "owner": "actor",
920 + "aggregation": "set",
921 + "columns": []any{
922 + map[string]any{"id": "actor", "type": "actor_ref"},
923 + },
924 + },
925 + },
926 + }
927 + }
928 +
929 + cases := map[string]func(map[string]any){
930 + "actor type label": func(types map[string]any) {
931 + types["actor_types"].(map[string]any)["node"].(map[string]any)["presentation"] = map[string]any{"label": ""}
932 + },
933 + "link type label": func(types map[string]any) {
934 + types["link_types"].(map[string]any)["dependency"].(map[string]any)["presentation"] = map[string]any{"label": ""}
935 + },
936 + "port type label": func(types map[string]any) {
937 + types["port_types"].(map[string]any)["port"] = map[string]any{"presentation": map[string]any{"label": ""}}
938 + },
939 + "table type label": func(types map[string]any) {
940 + types["table_types"].(map[string]any)["actor_labels"].(map[string]any)["presentation"] = map[string]any{"label": ""}
941 + },
942 + }
943 +
944 + for name, mutate := range cases {
945 + t.Run(name, func(t *testing.T) {
946 + types := defaultTypes()
947 + mutate(types)
948 +
949 + err := ValidateDecodedResponse(basePayload(types))
950 + require.Error(t, err)
951 + assert.Contains(t, err.Error(), "label is required")
952 + })
953 + }
954 +}
955 +
956 +func TestValidateDecodedResponseRejectsMalformedResponseEnvelope(t *testing.T) {
957 + cases := map[string]struct {
958 + payload any
959 + want string
960 + }{
961 + "not object": {
962 + payload: []any{},
963 + want: "response is not an object",
964 + },
965 + "missing data": {
966 + payload: map[string]any{"status": float64(200), "type": "topology"},
967 + want: "response.data is not an object",
968 + },
969 + "wrong schema": {
970 + payload: map[string]any{
971 + "status": float64(200),
972 + "type": "topology",
973 + "data": map[string]any{"schema_version": "old"},
974 + },
975 + want: "response.data.schema_version is not",
976 + },
977 + }
978 +
979 + for name, tc := range cases {
980 + t.Run(name, func(t *testing.T) {
981 + err := ValidateDecodedResponse(tc.payload)
982 + require.Error(t, err)
983 + assert.Contains(t, err.Error(), tc.want)
984 + })
985 + }
986 +}
987 +
988 +func TestValidateDecodedResponseRejectsInvalidCompactTableColumns(t *testing.T) {
989 + basePayload := func(columns []any, values []any) map[string]any {
990 + return map[string]any{
991 + "status": float64(200),
992 + "type": "topology",
993 + "data": map[string]any{
994 + "schema_version": SchemaVersion,
995 + "dictionaries": map[string]any{},
996 + "types": map[string]any{
997 + "actor_types": map[string]any{
998 + "node": map[string]any{"layer": "node", "identity": []any{"id"}},
999 + },
1000 + "link_types": map[string]any{
1001 + "dependency": map[string]any{
1002 + "orientation": "directed",
1003 + "direction_role": "dependency",
1004 + "aggregation": map[string]any{"direction": "preserve"},
1005 + },
1006 + },
1007 + "port_types": map[string]any{},
1008 + "table_types": map[string]any{},
1009 + },
1010 + "actors": map[string]any{
1011 + "rows": float64(1),
1012 + "columns": []any{
1013 + map[string]any{"id": "id", "type": "string", "role": "identity"},
1014 + map[string]any{"id": "type", "type": "string"},
1015 + },
1016 + "values": []any{
1017 + map[string]any{"codec": "const", "value": "node-a"},
1018 + map[string]any{"codec": "const", "value": "node"},
1019 + },
1020 + },
1021 + "links": map[string]any{
1022 + "rows": float64(1),
1023 + "columns": columns,
1024 + "values": values,
1025 + },
1026 + },
1027 + }
1028 + }
1029 +
1030 + cases := map[string]struct {
1031 + columns []any
1032 + values []any
1033 + want string
1034 + }{
1035 + "empty id": {
1036 + columns: []any{
1037 + map[string]any{"id": "", "type": "string"},
1038 + },
1039 + values: []any{
1040 + map[string]any{"codec": "const", "value": "dependency"},
1041 + },
1042 + want: "columns[0].id is required",
1043 + },
1044 + "duplicate id": {
1045 + columns: []any{
1046 + map[string]any{"id": "type", "type": "string"},
1047 + map[string]any{"id": "type", "type": "string"},
1048 + },
1049 + values: []any{
1050 + map[string]any{"codec": "const", "value": "dependency"},
1051 + map[string]any{"codec": "const", "value": "dependency"},
1052 + },
1053 + want: "duplicates column",
1054 + },
1055 + "wrong uint value type": {
1056 + columns: []any{
1057 + map[string]any{"id": "type", "type": "string"},
1058 + map[string]any{"id": "socket_count", "type": "uint"},
1059 + },
1060 + values: []any{
1061 + map[string]any{"codec": "const", "value": "dependency"},
1062 + map[string]any{"codec": "const", "value": "not-a-number"},
1063 + },
1064 + want: "is not a non-negative integer",
1065 + },
1066 + }
1067 +
1068 + for name, tc := range cases {
1069 + t.Run(name, func(t *testing.T) {
1070 + err := ValidateDecodedResponse(basePayload(tc.columns, tc.values))
1071 + require.Error(t, err)
1072 + assert.Contains(t, err.Error(), tc.want)
1073 + })
1074 + }
1075 +}
1076 +
1077 +func TestValidateDecodedResponseValidatesExplicitModalLabelColumns(t *testing.T) {
1078 + baseData := func(labels *ModalLabelsPresentation) Data {
1079 + return Data{
1080 + Producer: Producer{Source: "test-topology", Instance: "test-instance"},
1081 + CollectedAt: time.Date(2026, 5, 9, 10, 0, 0, 0, time.UTC),
1082 + Dictionaries: Dictionaries{"strings": StringValues("node-a")},
1083 + Types: TypeRegistry{
1084 + ActorTypes: map[string]ActorType{
1085 + "node": {
1086 + Layer: "node",
1087 + Identity: []string{"id"},
1088 + Presentation: &ActorPresentation{
1089 + Modal: &ModalPresentation{
1090 + Labels: labels,
1091 + },
1092 + },
1093 + },
1094 + },
1095 + LinkTypes: map[string]LinkType{},
1096 + PortTypes: map[string]PortType{},
1097 + TableTypes: map[string]TableType{
1098 + "actor_labels": {
1099 + Role: "actor_inventory",
1100 + Owner: "actor",
1101 + Aggregation: "set",
1102 + Columns: []Column{
1103 + NewColumn("actor", "actor_ref"),
1104 + NewColumn("key", "string"),
1105 + NewColumn("value", "string"),
1106 + },
1107 + },
1108 + },
1109 + },
1110 + Actors: MustTable(1,
1111 + []Column{
1112 + NewColumn("id", "string", WithRole("identity")),
1113 + NewColumn("type", "string"),
1114 + },
1115 + []ColumnEncoding{
1116 + Values("node-a"),
1117 + Const("node"),
1118 + },
1119 + ),
1120 + Links: MustTable(0,
1121 + []Column{NewColumn("type", "string")},
1122 + []ColumnEncoding{Const("none")},
1123 + ),
1124 + }
1125 + }
1126 +
1127 + t.Run("omitted optional columns are allowed", func(t *testing.T) {
1128 + payload := NewResponse(baseData(&ModalLabelsPresentation{Table: "actor_labels"}))
1129 + payloadBytes, err := json.Marshal(payload)
1130 + require.NoError(t, err)
1131 + require.NoError(t, topologySchemaValidationError(t, payloadBytes))
1132 +
1133 + var decoded any
1134 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
1135 + require.NoError(t, ValidateDecodedResponse(decoded))
1136 + })
1137 +
1138 + t.Run("identification fields are accepted", func(t *testing.T) {
1139 + payload := NewResponse(baseData(&ModalLabelsPresentation{
1140 + Table: "actor_labels",
1141 + Identification: &ModalLabelIdentificationPresentation{
1142 + Fields: []ModalLabelIdentificationField{
1143 + {Key: "display_name", Label: "Name", MaxValues: 1},
1144 + {Key: "role", Label: "Role", MaxValues: 2},
1145 + },
1146 + },
1147 + }))
1148 + payloadBytes, err := json.Marshal(payload)
1149 + require.NoError(t, err)
1150 + require.NoError(t, topologySchemaValidationError(t, payloadBytes))
1151 +
1152 + var decoded any
1153 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
1154 + require.NoError(t, ValidateDecodedResponse(decoded))
1155 + })
1156 +
1157 + t.Run("invalid identification max values is rejected", func(t *testing.T) {
1158 + payload := NewResponse(baseData(&ModalLabelsPresentation{Table: "actor_labels"}))
1159 + payloadBytes, err := json.Marshal(payload)
1160 + require.NoError(t, err)
1161 +
1162 + var decoded any
1163 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
1164 + data := decoded.(map[string]any)["data"].(map[string]any)
1165 + actorType := data["types"].(map[string]any)["actor_types"].(map[string]any)["node"].(map[string]any)
1166 + modal := actorType["presentation"].(map[string]any)["modal"].(map[string]any)
1167 + labels := modal["labels"].(map[string]any)
1168 + labels["identification"] = map[string]any{
1169 + "fields": []any{
1170 + map[string]any{"key": "display_name", "label": "Name", "max_values": float64(0)},
1171 + },
1172 + }
1173 +
1174 + err = ValidateDecodedResponse(decoded)
1175 + require.Error(t, err)
1176 + assert.Contains(t, err.Error(), "labels.identification.fields[0].max_values must be a positive integer")
1177 + })
1178 +
1179 + t.Run("explicit missing optional column is rejected", func(t *testing.T) {
1180 + payload := NewResponse(baseData(&ModalLabelsPresentation{Table: "actor_labels", SourceColumn: "missing"}))
1181 + payloadBytes, err := json.Marshal(payload)
1182 + require.NoError(t, err)
1183 + require.NoError(t, topologySchemaValidationError(t, payloadBytes))
1184 +
1185 + var decoded any
1186 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
1187 +
1188 + err = ValidateDecodedResponse(decoded)
1189 + require.Error(t, err)
1190 + assert.Contains(t, err.Error(), "labels.source_column references unknown column \"missing\"")
1191 + })
1192 +}
1193 +
1194 +func TestValidateDecodedResponseRejectsInvalidModalSectionsShape(t *testing.T) {
1195 + payload := map[string]any{
1196 + "status": float64(200),
1197 + "type": "topology",
1198 + "data": map[string]any{
1199 + "schema_version": SchemaVersion,
1200 + "dictionaries": map[string]any{},
1201 + "types": map[string]any{
1202 + "actor_types": map[string]any{
1203 + "node": map[string]any{
1204 + "layer": "node",
1205 + "identity": []any{"id"},
1206 + "presentation": map[string]any{
1207 + "modal": map[string]any{
1208 + "sections": map[string]any{"not": "an-array"},
1209 + },
1210 + },
1211 + },
1212 + },
1213 + "link_types": map[string]any{},
1214 + "port_types": map[string]any{},
1215 + "table_types": map[string]any{},
1216 + },
1217 + "actors": map[string]any{
1218 + "rows": float64(1),
1219 + "columns": []any{
1220 + map[string]any{"id": "id", "type": "string", "role": "identity"},
1221 + map[string]any{"id": "type", "type": "string"},
1222 + },
1223 + "values": []any{
1224 + map[string]any{"codec": "values", "values": []any{"node-a"}},
1225 + map[string]any{"codec": "const", "value": "node"},
1226 + },
1227 + },
1228 + "links": map[string]any{
1229 + "rows": float64(0),
1230 + "columns": []any{
1231 + map[string]any{"id": "type", "type": "string"},
1232 + },
1233 + "values": []any{
1234 + map[string]any{"codec": "const", "value": "none"},
1235 + },
1236 + },
1237 + },
1238 + }
1239 +
1240 + err := ValidateDecodedResponse(payload)
1241 + require.Error(t, err)
1242 + assert.Contains(t, err.Error(), "modal.sections is not an array")
1243 +}
1244 +
1245 +func TestValidateDecodedResponseRejectsInvalidModalRowFilters(t *testing.T) {
1246 + cases := map[string]struct {
1247 + filter ModalRowFilter
1248 + want string
1249 + }{
1250 + "eq missing value": {
1251 + filter: ModalRowFilter{Column: "type", Op: "eq"},
1252 + want: "value is required when op is eq",
1253 + },
1254 + "in missing values": {
1255 + filter: ModalRowFilter{Column: "type", Op: "in"},
1256 + want: "values is required when op is in",
1257 + },
1258 + }
1259 +
1260 + for name, tc := range cases {
1261 + t.Run(name, func(t *testing.T) {
1262 + payload := NewResponse(Data{
1263 + Producer: Producer{Source: "test-topology", Instance: "test-instance"},
1264 + CollectedAt: time.Date(2026, 5, 9, 10, 0, 0, 0, time.UTC),
1265 + Dictionaries: Dictionaries{"strings": StringValues("node-a")},
1266 + Types: TypeRegistry{
1267 + ActorTypes: map[string]ActorType{
1268 + "node": {
1269 + Layer: "node",
1270 + Identity: []string{"id"},
1271 + Presentation: &ActorPresentation{
1272 + Modal: &ModalPresentation{
1273 + Sections: []ModalSection{
1274 + {
1275 + ID: "links",
1276 + Label: "Links",
1277 + Source: ModalSource{Kind: "links"},
1278 + RowFilters: []ModalRowFilter{tc.filter},
1279 + Columns: []ModalColumn{
1280 + {
1281 + ID: "type",
1282 + Label: "Type",
1283 + Projection: ModalProjection{
1284 + Kind: "direct",
1285 + Column: "type",
1286 + },
1287 + },
1288 + },
1289 + },
1290 + },
1291 + },
1292 + },
1293 + },
1294 + },
1295 + LinkTypes: map[string]LinkType{
1296 + "dependency": {
1297 + Orientation: "directed",
1298 + DirectionRole: "dependency",
1299 + Aggregation: LinkAggregation{Direction: "preserve"},
1300 + },
1301 + },
1302 + },
1303 + Actors: MustTable(1,
1304 + []Column{
1305 + NewColumn("id", "string", WithRole("identity")),
1306 + NewColumn("type", "string"),
1307 + },
1308 + []ColumnEncoding{
1309 + Values("node-a"),
1310 + Const("node"),
1311 + },
1312 + ),
1313 + Links: MustTable(1,
1314 + []Column{
1315 + NewColumn("src", "actor_ref"),
1316 + NewColumn("dst", "actor_ref"),
1317 + NewColumn("type", "string"),
1318 + },
1319 + []ColumnEncoding{
1320 + Const(0),
1321 + Const(0),
1322 + Const("dependency"),
1323 + },
1324 + ),
1325 + })
1326 +
1327 + payloadBytes, err := json.Marshal(payload)
1328 + require.NoError(t, err)
1329 + require.Error(t, topologySchemaValidationError(t, payloadBytes))
1330 +
1331 + var decoded any
1332 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
1333 +
1334 + err = ValidateDecodedResponse(decoded)
1335 + require.Error(t, err)
1336 + assert.Contains(t, err.Error(), tc.want)
1337 + })
1338 + }
1339 +}
1340 +
1341 +func TestValidateDecodedResponseRejectsCorrelationMissingKeyColumn(t *testing.T) {
1342 + correlation := &Correlation{
1343 + Rules: map[string]CorrelationRule{
1344 + "node_name": {
1345 + Action: "link",
1346 + Priority: 10,
1347 + KeySpace: "node_name",
1348 + Key: []CorrelationKeyPart{{Column: "name"}},
1349 + PointActorTypes: []string{"node"},
1350 + OutputLinkType: "dependency",
1351 + },
1352 + "node_owner": {
1353 + Action: "link",
1354 + Priority: 20,
1355 + KeySpace: "node_owner",
1356 + Key: []CorrelationKeyPart{{Column: "owner"}},
1357 + PointActorTypes: []string{"node"},
1358 + OutputLinkType: "dependency",
1359 + },
1360 + },
1361 + Points: &Table{
1362 + Rows: 1,
1363 + Columns: []Column{
1364 + NewColumn("actor", "actor_ref"),
1365 + NewColumn("rule", "string"),
1366 + },
1367 + Values: []ColumnEncoding{
1368 + Values(0),
1369 + Const("node_name"),
1370 + },
1371 + },
1372 + }
1373 + err := validateResponseData(t, minimalValidationData(nil,
1374 + withCorrelation(correlation),
1375 + withLinks(dependencyLinkTable(0)),
1376 + ))
1377 +
1378 + require.Error(t, err)
1379 + assert.Contains(t, err.Error(), "missing correlation key column")
1380 +}
1381 +
1382 +func TestValidateDecodedResponseAllowsUnusedCorrelationRuleColumns(t *testing.T) {
1383 + correlation := &Correlation{
1384 + Rules: map[string]CorrelationRule{
1385 + "node_name": {
1386 + Action: "link",
1387 + Priority: 10,
1388 + KeySpace: "node_name",
1389 + Key: []CorrelationKeyPart{{Column: "name"}},
1390 + PointActorTypes: []string{"node"},
1391 + OutputLinkType: "dependency",
1392 + },
1393 + },
1394 + Points: &Table{
1395 + Rows: 1,
1396 + Columns: []Column{
1397 + NewColumn("actor", "actor_ref"),
1398 + NewColumn("rule", "string"),
1399 + NewColumn("name", "string_ref", WithDictionary("strings")),
1400 + },
1401 + Values: []ColumnEncoding{
1402 + Values(0),
1403 + Const("node_name"),
1404 + Values(0),
1405 + },
1406 + },
1407 + }
1408 + err := validateResponseData(t, minimalValidationData(nil,
1409 + withCorrelation(correlation),
1410 + withLinks(dependencyLinkTable(0)),
1411 + ))
1412 +
1413 + require.NoError(t, err)
1414 +}
1415 +
1416 +func TestPresentationTokenEnumsMatchSchema(t *testing.T) {
1417 + schemaDoc := loadTopologySchema(t)
1418 +
1419 + assert.ElementsMatch(t, colorSlotTokens, schemaEnum(t, schemaDoc, "color_slot"))
1420 + assert.ElementsMatch(t, opacityTokens, schemaEnum(t, schemaDoc, "opacity_token"))
1421 + assert.ElementsMatch(t, widthTokens, schemaEnum(t, schemaDoc, "width_token"))
1422 + assert.ElementsMatch(t, layoutStrengthTokens, schemaEnum(t, schemaDoc, "layout_strength_token"))
1423 + assert.ElementsMatch(t, layoutDistanceTokens, schemaEnum(t, schemaDoc, "layout_distance_token"))
1424 + assert.ElementsMatch(t, actorSizeScaleTokens, schemaEnum(t, schemaDoc, "actor_size_scale_token"))
1425 + assert.ElementsMatch(t, linkSemanticRoleTokens, schemaEnum(t, schemaDoc, "link_semantic_role"))
1426 + assert.ElementsMatch(t, iconTokens, schemaEnum(t, schemaDoc, "icon_token"))
1427 +}
1428 +
1429 +func TestValidateDecodedResponseRejectsInvalidActorReference(t *testing.T) {
1430 + payload := map[string]any{
1431 + "status": float64(200),
1432 + "type": "topology",
1433 + "data": map[string]any{
1434 + "schema_version": SchemaVersion,
1435 + "dictionaries": map[string]any{
1436 + "strings": []any{"node-a"},
1437 + },
1438 + "types": map[string]any{},
1439 + "actors": map[string]any{
1440 + "rows": float64(1),
1441 + "columns": []any{map[string]any{"id": "id", "type": "string"}},
1442 + "values": []any{map[string]any{"codec": "values", "values": []any{"node-a"}}},
1443 + },
1444 + "links": map[string]any{
1445 + "rows": float64(1),
1446 + "columns": []any{
1447 + map[string]any{"id": "src", "type": "actor_ref"},
1448 + map[string]any{"id": "dst", "type": "actor_ref"},
1449 + },
1450 + "values": []any{
1451 + map[string]any{"codec": "values", "values": []any{float64(0)}},
1452 + map[string]any{"codec": "values", "values": []any{float64(3)}},
1453 + },
1454 + },
1455 + },
1456 + }
1457 +
1458 + err := ValidateDecodedResponse(payload)
1459 + require.Error(t, err)
1460 + assert.Contains(t, err.Error(), "actor reference out of bounds")
1461 +}
1462 +
1463 +func TestValidateDecodedResponseRejectsInvalidEvidenceSectionShape(t *testing.T) {
1464 + payload := map[string]any{
1465 + "status": float64(200),
1466 + "type": "topology",
1467 + "data": map[string]any{
1468 + "schema_version": SchemaVersion,
1469 + "dictionaries": map[string]any{},
1470 + "types": map[string]any{},
1471 + "actors": map[string]any{
1472 + "rows": float64(0),
1473 + "columns": []any{},
1474 + "values": []any{},
1475 + },
1476 + "links": map[string]any{
1477 + "rows": float64(0),
1478 + "columns": []any{},
1479 + "values": []any{},
1480 + },
1481 + "evidence": map[string]any{
1482 + "socket": "not-an-object",
1483 + },
1484 + },
1485 + }
1486 +
1487 + err := ValidateDecodedResponse(payload)
1488 + require.Error(t, err)
1489 + assert.Contains(t, err.Error(), "data.evidence.socket is not an object")
1490 +}
1491 +
1492 +func validateResponseData(t *testing.T, data Data) error {
1493 + t.Helper()
1494 +
1495 + payloadBytes, err := json.Marshal(NewResponse(data))
1496 + require.NoError(t, err)
1497 +
1498 + var decoded any
1499 + require.NoError(t, json.Unmarshal(payloadBytes, &decoded))
1500 + return ValidateDecodedResponse(decoded)
1501 +}
1502 +
1503 +func minimalValidationData(actorPresentation *ActorPresentation, opts ...func(*Data)) Data {
1504 + data := Data{
1505 + Producer: Producer{Source: "test-topology", Instance: "test-instance"},
1506 + CollectedAt: testCollectedAt,
1507 + Dictionaries: Dictionaries{"strings": StringValues("node-a")},
1508 + Types: TypeRegistry{
1509 + ActorTypes: map[string]ActorType{
1510 + "node": {
1511 + Layer: "node",
1512 + Identity: []string{"id"},
1513 + Presentation: actorPresentation,
1514 + },
1515 + },
1516 + LinkTypes: map[string]LinkType{
1517 + "dependency": {
1518 + Orientation: "directed",
1519 + DirectionRole: "dependency",
1520 + Aggregation: LinkAggregation{Direction: "preserve"},
1521 + },
1522 + },
1523 + },
1524 + Actors: minimalActorTable(nil, nil),
1525 + Links: dependencyLinkTable(1),
1526 + }
1527 +
1528 + for _, opt := range opts {
1529 + opt(&data)
1530 + }
1531 + return data
1532 +}
1533 +
1534 +func minimalActorTable(extraColumns []Column, extraValues []ColumnEncoding) Table {
1535 + columns := []Column{
1536 + NewColumn("id", "string", WithRole("identity")),
1537 + NewColumn("type", "string"),
1538 + }
1539 + values := []ColumnEncoding{
1540 + Values("node-a"),
1541 + Const("node"),
1542 + }
1543 + columns = append(columns, extraColumns...)
1544 + values = append(values, extraValues...)
1545 + return MustTable(1, columns, values)
1546 +}
1547 +
1548 +func dependencyLinkTable(rows int) Table {
1549 + return dependencyLinkTableWith(rows, nil, nil)
1550 +}
1551 +
1552 +func dependencyLinkTableWith(rows int, extraColumns []Column, extraValues []ColumnEncoding) Table {
1553 + columns := []Column{
1554 + NewColumn("src", "actor_ref"),
1555 + NewColumn("dst", "actor_ref"),
1556 + NewColumn("type", "string"),
1557 + }
1558 + values := []ColumnEncoding{
1559 + Const(0),
1560 + Const(0),
1561 + Const("dependency"),
1562 + }
1563 + columns = append(columns, extraColumns...)
1564 + values = append(values, extraValues...)
1565 + return MustTable(rows, columns, values)
1566 +}
1567 +
1568 +func withActors(actors Table) func(*Data) {
1569 + return func(data *Data) {
1570 + data.Actors = actors
1571 + }
1572 +}
1573 +
1574 +func withLinks(links Table) func(*Data) {
1575 + return func(data *Data) {
1576 + data.Links = links
1577 + }
1578 +}
1579 +
1580 +func withPortTypes(portTypes map[string]PortType) func(*Data) {
1581 + return func(data *Data) {
1582 + data.Types.PortTypes = portTypes
1583 + }
1584 +}
1585 +
1586 +func withCorrelation(correlation *Correlation) func(*Data) {
1587 + return func(data *Data) {
1588 + data.Correlation = correlation
1589 + }
1590 +}
1591 +
1592 +func validateAgainstTopologySchema(t *testing.T, payload []byte) {
1593 + t.Helper()
1594 +
1595 + require.NoError(t, topologySchemaValidationError(t, payload))
1596 +}
1597 +
1598 +func topologySchemaValidationError(t *testing.T, payload []byte) error {
1599 + t.Helper()
1600 +
1601 + schemaDoc := loadTopologySchema(t)
1602 +
1603 + compiler := jsonschema.NewCompiler()
1604 + require.NoError(t, compiler.AddResource("schema.json", schemaDoc))
1605 + schema, err := compiler.Compile("schema.json")
1606 + require.NoError(t, err)
1607 +
1608 + var decoded any
1609 + require.NoError(t, json.Unmarshal(payload, &decoded))
1610 + return schema.Validate(decoded)
1611 +}
1612 +
1613 +func loadTopologySchema(t *testing.T) map[string]any {
1614 + t.Helper()
1615 +
1616 + schemaPath := filepath.Clean(filepath.Join("..", "..", "..", "..", "plugins.d", "FUNCTION_TOPOLOGY_SCHEMA.json"))
1617 + schemaBytes, err := os.ReadFile(schemaPath)
1618 + require.NoError(t, err)
1619 +
1620 + var schemaDoc map[string]any
1621 + require.NoError(t, json.Unmarshal(schemaBytes, &schemaDoc))
1622 + return schemaDoc
1623 +}
1624 +
1625 +func schemaEnum(t *testing.T, schemaDoc map[string]any, defName string) []string {
1626 + t.Helper()
1627 +
1628 + defs, ok := schemaDoc["$defs"].(map[string]any)
1629 + require.True(t, ok)
1630 + definition, ok := defs[defName].(map[string]any)
1631 + require.True(t, ok)
1632 + rawEnum, ok := definition["enum"].([]any)
1633 + require.True(t, ok)
1634 +
1635 + values := make([]string, 0, len(rawEnum))
1636 + for _, raw := range rawEnum {
1637 + value, ok := raw.(string)
1638 + require.True(t, ok)
1639 + values = append(values, value)
1640 + }
1641 + return values
1642 +}
src/go/pkg/topology/v1/table.go new
+232
@@ -0,0 +1,232 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package topologyv1
4 +
5 +import "fmt"
6 +
7 +func NewTable(rows int, columns []Column, values []ColumnEncoding) (Table, error) {
8 + table := Table{
9 + Rows: rows,
10 + Columns: append([]Column(nil), columns...),
11 + Values: append([]ColumnEncoding(nil), values...),
12 + }
13 + if err := table.Validate(); err != nil {
14 + return Table{}, err
15 + }
16 + return table, nil
17 +}
18 +
19 +func MustTable(rows int, columns []Column, values []ColumnEncoding) Table {
20 + table, err := NewTable(rows, columns, values)
21 + if err != nil {
22 + panic(err)
23 + }
24 + return table
25 +}
26 +
27 +func EmptyTable() Table {
28 + return Table{
29 + Rows: 0,
30 + Columns: []Column{},
31 + Values: []ColumnEncoding{},
32 + }
33 +}
34 +
35 +func (table Table) Validate() error {
36 + if table.Rows < 0 {
37 + return fmt.Errorf("rows is negative: %d", table.Rows)
38 + }
39 + if len(table.Columns) != len(table.Values) {
40 + return fmt.Errorf("columns/values length mismatch: %d columns, %d values", len(table.Columns), len(table.Values))
41 + }
42 +
43 + seenColumns := make(map[string]struct{}, len(table.Columns))
44 + for i, column := range table.Columns {
45 + if column.ID == "" {
46 + return fmt.Errorf("columns[%d].id is empty", i)
47 + }
48 + if _, ok := seenColumns[column.ID]; ok {
49 + return fmt.Errorf("columns[%d].id duplicates column %q", i, column.ID)
50 + }
51 + seenColumns[column.ID] = struct{}{}
52 + if column.Type == "" {
53 + return fmt.Errorf("columns[%d].type is empty", i)
54 + }
55 + if err := validateColumnContract(i, column); err != nil {
56 + return err
57 + }
58 + if err := validateEncoding(table.Rows, i, column, table.Values[i]); err != nil {
59 + return err
60 + }
61 + }
62 +
63 + return nil
64 +}
65 +
66 +func validateColumnContract(columnIndex int, column Column) error {
67 + switch column.Type {
68 + case "string_ref", "ip_ref", "mac_ref":
69 + if column.Dictionary == "" {
70 + return fmt.Errorf("columns[%d] type %q requires dictionary", columnIndex, column.Type)
71 + }
72 + case "bool", "int", "uint", "float", "string", "timestamp", "duration", "ip", "mac", "actor_ref", "link_ref", "evidence_ref", "array", "json":
73 + if column.Dictionary != "" {
74 + return fmt.Errorf("columns[%d] uses dictionary with non-reference type %q", columnIndex, column.Type)
75 + }
76 + default:
77 + return fmt.Errorf("columns[%d] has unsupported type %q", columnIndex, column.Type)
78 + }
79 +
80 + return nil
81 +}
82 +
83 +func validateEncoding(rows, columnIndex int, column Column, encoding ColumnEncoding) error {
84 + values, err := decodeEncodingValues(rows, columnIndex, encoding)
85 + if err != nil {
86 + return err
87 + }
88 + for rowIndex, value := range values {
89 + if err := validateEncodedColumnValue(columnIndex, rowIndex, column, value); err != nil {
90 + return err
91 + }
92 + }
93 + return nil
94 +}
95 +
96 +func decodeEncodingValues(rows, columnIndex int, encoding ColumnEncoding) ([]any, error) {
97 + switch value := encoding.(type) {
98 + case nil:
99 + return nil, fmt.Errorf("values[%d] is nil", columnIndex)
100 + case ConstEncoding:
101 + if value.Codec != "const" {
102 + return nil, fmt.Errorf("values[%d] const encoding has invalid codec %q", columnIndex, value.Codec)
103 + }
104 + return repeatValue(rows, value.Value), nil
105 + case *ConstEncoding:
106 + if value == nil {
107 + return nil, fmt.Errorf("values[%d] is nil", columnIndex)
108 + }
109 + if value.Codec != "const" {
110 + return nil, fmt.Errorf("values[%d] const encoding has invalid codec %q", columnIndex, value.Codec)
111 + }
112 + return repeatValue(rows, value.Value), nil
113 + case ValuesEncoding:
114 + if value.Codec != "values" {
115 + return nil, fmt.Errorf("values[%d] values encoding has invalid codec %q", columnIndex, value.Codec)
116 + }
117 + if len(value.Values) != rows {
118 + return nil, fmt.Errorf("values[%d] decoded length mismatch: expected %d, got %d", columnIndex, rows, len(value.Values))
119 + }
120 + return append([]any(nil), value.Values...), nil
121 + case *ValuesEncoding:
122 + if value == nil {
123 + return nil, fmt.Errorf("values[%d] is nil", columnIndex)
124 + }
125 + if value.Codec != "values" {
126 + return nil, fmt.Errorf("values[%d] values encoding has invalid codec %q", columnIndex, value.Codec)
127 + }
128 + if len(value.Values) != rows {
129 + return nil, fmt.Errorf("values[%d] decoded length mismatch: expected %d, got %d", columnIndex, rows, len(value.Values))
130 + }
131 + return append([]any(nil), value.Values...), nil
132 + case DictEncoding:
133 + if value.Codec != "dict" {
134 + return nil, fmt.Errorf("values[%d] dict encoding has invalid codec %q", columnIndex, value.Codec)
135 + }
136 + if err := validateDictEncoding(rows, columnIndex, value.Values, value.Indexes); err != nil {
137 + return nil, err
138 + }
139 + return decodeDictValues(value.Values, value.Indexes), nil
140 + case *DictEncoding:
141 + if value == nil {
142 + return nil, fmt.Errorf("values[%d] is nil", columnIndex)
143 + }
144 + if value.Codec != "dict" {
145 + return nil, fmt.Errorf("values[%d] dict encoding has invalid codec %q", columnIndex, value.Codec)
146 + }
147 + if err := validateDictEncoding(rows, columnIndex, value.Values, value.Indexes); err != nil {
148 + return nil, err
149 + }
150 + return decodeDictValues(value.Values, value.Indexes), nil
151 + default:
152 + return nil, fmt.Errorf("values[%d] has unsupported encoding type %T", columnIndex, encoding)
153 + }
154 +}
155 +
156 +func repeatValue(rows int, value any) []any {
157 + values := make([]any, rows)
158 + for i := range values {
159 + values[i] = value
160 + }
161 + return values
162 +}
163 +
164 +func decodeDictValues(values []any, indexes []int) []any {
165 + decoded := make([]any, len(indexes))
166 + for i, index := range indexes {
167 + decoded[i] = values[index]
168 + }
169 + return decoded
170 +}
171 +
172 +func validateEncodedColumnValue(columnIndex, rowIndex int, column Column, value any) error {
173 + if value == nil {
174 + if column.Nullable {
175 + return nil
176 + }
177 + return fmt.Errorf("values[%d][%d] is null but column is not nullable", columnIndex, rowIndex)
178 + }
179 +
180 + switch column.Type {
181 + case "string_ref", "ip_ref", "mac_ref":
182 + if n, ok := integerValue(value); !ok || n < 0 {
183 + return fmt.Errorf("values[%d][%d] is not a non-negative dictionary reference", columnIndex, rowIndex)
184 + }
185 + case "actor_ref", "link_ref", "evidence_ref":
186 + if n, ok := integerValue(value); !ok || n < 0 {
187 + return fmt.Errorf("values[%d][%d] is not a non-negative %s reference", columnIndex, rowIndex, column.Type)
188 + }
189 + case "array":
190 + if _, ok := value.([]any); !ok {
191 + return fmt.Errorf("values[%d][%d] is not an array", columnIndex, rowIndex)
192 + }
193 + case "bool":
194 + if _, ok := value.(bool); !ok {
195 + return fmt.Errorf("values[%d][%d] is not a bool", columnIndex, rowIndex)
196 + }
197 + case "int":
198 + if _, ok := integerValue(value); !ok {
199 + return fmt.Errorf("values[%d][%d] is not an integer", columnIndex, rowIndex)
200 + }
201 + case "uint":
202 + if n, ok := integerValue(value); !ok || n < 0 {
203 + return fmt.Errorf("values[%d][%d] is not a non-negative integer", columnIndex, rowIndex)
204 + }
205 + case "float", "duration":
206 + if _, ok := numberValue(value); !ok {
207 + return fmt.Errorf("values[%d][%d] is not a number", columnIndex, rowIndex)
208 + }
209 + case "string", "ip", "mac", "timestamp":
210 + if _, ok := value.(string); !ok {
211 + return fmt.Errorf("values[%d][%d] is not a string", columnIndex, rowIndex)
212 + }
213 + case "json":
214 + return nil
215 + default:
216 + return fmt.Errorf("columns[%d] has unsupported type %q", columnIndex, column.Type)
217 + }
218 +
219 + return nil
220 +}
221 +
222 +func validateDictEncoding(rows, columnIndex int, values []any, indexes []int) error {
223 + if len(indexes) != rows {
224 + return fmt.Errorf("values[%d] decoded length mismatch: expected %d, got %d", columnIndex, rows, len(indexes))
225 + }
226 + for i, index := range indexes {
227 + if index < 0 || index >= len(values) {
228 + return fmt.Errorf("values[%d].indexes[%d] out of bounds: %d", columnIndex, i, index)
229 + }
230 + }
231 + return nil
232 +}
src/go/pkg/topology/v1/table_test.go new
+186
@@ -0,0 +1,186 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package topologyv1
4 +
5 +import (
6 + "testing"
7 +
8 + "github.com/stretchr/testify/assert"
9 + "github.com/stretchr/testify/require"
10 +)
11 +
12 +func TestNewTableValidatesEncodings(t *testing.T) {
13 + table, err := NewTable(3,
14 + []Column{
15 + NewColumn("kind", "string"),
16 + NewColumn("name", "string"),
17 + NewColumn("state", "string"),
18 + },
19 + []ColumnEncoding{
20 + Const("process"),
21 + Values("web", "db", "cache"),
22 + Dict(StringValues("running", "stopped"), 0, 0, 1),
23 + },
24 + )
25 +
26 + require.NoError(t, err)
27 + assert.Equal(t, 3, table.Rows)
28 + assert.Len(t, table.Columns, 3)
29 + assert.Len(t, table.Values, 3)
30 +}
31 +
32 +func TestNewTableRejectsColumnValueMismatch(t *testing.T) {
33 + _, err := NewTable(1,
34 + []Column{
35 + NewColumn("id", "string"),
36 + NewColumn("name", "string"),
37 + },
38 + []ColumnEncoding{
39 + Values("actor-1"),
40 + },
41 + )
42 +
43 + require.Error(t, err)
44 + assert.Contains(t, err.Error(), "columns/values length mismatch")
45 +}
46 +
47 +func TestNewTableRejectsDecodedLengthMismatch(t *testing.T) {
48 + _, err := NewTable(2,
49 + []Column{NewColumn("name", "string")},
50 + []ColumnEncoding{Values("only-one")},
51 + )
52 +
53 + require.Error(t, err)
54 + assert.Contains(t, err.Error(), "decoded length mismatch")
55 +}
56 +
57 +func TestNewTableRejectsDictIndexOutOfBounds(t *testing.T) {
58 + _, err := NewTable(2,
59 + []Column{NewColumn("state", "string")},
60 + []ColumnEncoding{Dict(StringValues("up"), 0, 1)},
61 + )
62 +
63 + require.Error(t, err)
64 + assert.Contains(t, err.Error(), "out of bounds")
65 +}
66 +
67 +func TestNewTableRejectsNilEncoding(t *testing.T) {
68 + _, err := NewTable(1,
69 + []Column{NewColumn("id", "string")},
70 + []ColumnEncoding{nil},
71 + )
72 +
73 + require.Error(t, err)
74 + assert.Contains(t, err.Error(), "is nil")
75 +}
76 +
77 +func TestNewTableRejectsDuplicateColumnIDs(t *testing.T) {
78 + _, err := NewTable(1,
79 + []Column{
80 + NewColumn("id", "string"),
81 + NewColumn("id", "string"),
82 + },
83 + []ColumnEncoding{
84 + Const("actor-1"),
85 + Const("actor-2"),
86 + },
87 + )
88 +
89 + require.Error(t, err)
90 + assert.Contains(t, err.Error(), "duplicates column")
91 +}
92 +
93 +func TestNewTableRejectsColumnValueContractMismatch(t *testing.T) {
94 + cases := map[string]struct {
95 + column Column
96 + value ColumnEncoding
97 + want string
98 + }{
99 + "non nullable null": {
100 + column: NewColumn("name", "string"),
101 + value: Const(nil),
102 + want: "not nullable",
103 + },
104 + "string ref without dictionary": {
105 + column: NewColumn("name", "string_ref"),
106 + value: Const(0),
107 + want: "requires dictionary",
108 + },
109 + "string ref value must be integer": {
110 + column: NewColumn("name", "string_ref", WithDictionary("strings")),
111 + value: Const("node-a"),
112 + want: "dictionary reference",
113 + },
114 + "actor ref must be non negative integer": {
115 + column: NewColumn("actor", "actor_ref"),
116 + value: Const(-1),
117 + want: "actor_ref reference",
118 + },
119 + "dictionary only belongs on reference columns": {
120 + column: Column{ID: "name", Type: "string", Dictionary: "strings"},
121 + value: Const("node-a"),
122 + want: "dictionary with non-reference type",
123 + },
124 + "unsupported type rejected even when nullable": {
125 + column: Column{ID: "name", Type: "unknown", Nullable: true},
126 + value: Const(nil),
127 + want: "unsupported type",
128 + },
129 + "uint must be non negative": {
130 + column: NewColumn("socket_count", "uint"),
131 + value: Const(-1),
132 + want: "non-negative integer",
133 + },
134 + }
135 +
136 + for name, tc := range cases {
137 + t.Run(name, func(t *testing.T) {
138 + _, err := NewTable(1, []Column{tc.column}, []ColumnEncoding{tc.value})
139 + require.Error(t, err)
140 + assert.Contains(t, err.Error(), tc.want)
141 + })
142 + }
143 +}
144 +
145 +func TestNewTableAllowsNullableNull(t *testing.T) {
146 + table, err := NewTable(1,
147 + []Column{NewColumn("name", "string", WithNullable())},
148 + []ColumnEncoding{Const(nil)},
149 + )
150 +
151 + require.NoError(t, err)
152 + assert.Equal(t, 1, table.Rows)
153 +}
154 +
155 +func TestStringDictionaryDeduplicatesValues(t *testing.T) {
156 + dict := NewStringDictionary()
157 +
158 + first := dict.Ref("alpha")
159 + second := dict.Ref("beta")
160 + again := dict.Ref("alpha")
161 +
162 + assert.Equal(t, 0, first)
163 + assert.Equal(t, 1, second)
164 + assert.Equal(t, first, again)
165 + assert.Equal(t, []any{"alpha", "beta"}, dict.Values())
166 +}
167 +
168 +func TestNilStringDictionaryReturnsEmptyValues(t *testing.T) {
169 + var dict *StringDictionary
170 +
171 + assert.Equal(t, []any{}, dict.Values())
172 +}
173 +
174 +func TestEmptyStringDictionaryReturnsEmptyValues(t *testing.T) {
175 + dict := NewStringDictionary()
176 +
177 + assert.Equal(t, []any{}, dict.Values())
178 +}
179 +
180 +func TestNilStringDictionaryRefPanics(t *testing.T) {
181 + var dict *StringDictionary
182 +
183 + assert.Panics(t, func() {
184 + dict.Ref("alpha")
185 + })
186 +}
src/go/pkg/topology/v1/types.go new
+581
@@ -0,0 +1,581 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +// Package topologyv1 contains producer-side types for netdata.topology.v1
4 +// Function payloads.
5 +package topologyv1
6 +
7 +import "time"
8 +
9 +const (
10 + SchemaVersion = "netdata.topology.v1"
11 + ResponseType = "topology"
12 +)
13 +
14 +type Response struct {
15 + Status int `json:"status"`
16 + Type string `json:"type"`
17 + HasHistory bool `json:"has_history,omitempty"`
18 + AcceptedParams []string `json:"accepted_params,omitempty"`
19 + RequiredParams []any `json:"required_params,omitempty"`
20 + Help string `json:"help,omitempty"`
21 + UpdateEvery int `json:"update_every,omitempty"`
22 + Expires int64 `json:"expires,omitempty"`
23 + Data Data `json:"data"`
24 +}
25 +
26 +func NewResponse(data Data) Response {
27 + data.SchemaVersion = SchemaVersion
28 + return Response{
29 + Status: 200,
30 + Type: ResponseType,
31 + Data: data,
32 + }
33 +}
34 +
35 +type Data struct {
36 + SchemaVersion string `json:"schema_version"`
37 + Producer Producer `json:"producer"`
38 + CollectedAt time.Time `json:"collected_at"`
39 + ValidAfter *time.Time `json:"valid_after,omitempty"`
40 + ValidUntil *time.Time `json:"valid_until,omitempty"`
41 + View *View `json:"view,omitempty"`
42 + Dictionaries Dictionaries `json:"dictionaries"`
43 + Types TypeRegistry `json:"types"`
44 + Presentation *Presentation `json:"presentation,omitempty"`
45 + Correlation *Correlation `json:"correlation,omitempty"`
46 + Actors Table `json:"actors"`
47 + Links Table `json:"links"`
48 + Evidence EvidenceMap `json:"evidence,omitempty"`
49 + Tables *DetailTables `json:"tables,omitempty"`
50 + Overlays *OverlayRefs `json:"overlays,omitempty"`
51 + Stats map[string]any `json:"stats,omitempty"`
52 + Extensions map[string]any `json:"extensions,omitempty"`
53 +}
54 +
55 +type Producer struct {
56 + Source string `json:"source"`
57 + Instance string `json:"instance"`
58 + NodeID string `json:"node_id,omitempty"`
59 + MachineGUID string `json:"machine_guid,omitempty"`
60 + AgentVersion string `json:"agent_version,omitempty"`
61 + Plugin string `json:"plugin,omitempty"`
62 + Capabilities []string `json:"capabilities,omitempty"`
63 +}
64 +
65 +type View struct {
66 + ID string `json:"id,omitempty"`
67 + Scope string `json:"scope,omitempty"`
68 + Mode string `json:"mode,omitempty"`
69 + SupportedModes []string `json:"supported_modes,omitempty"`
70 + GroupBy []string `json:"group_by,omitempty"`
71 +}
72 +
73 +type Dictionaries map[string][]any
74 +
75 +type TypeRegistry struct {
76 + ActorTypes map[string]ActorType `json:"actor_types"`
77 + LinkTypes map[string]LinkType `json:"link_types"`
78 + PortTypes map[string]PortType `json:"port_types,omitempty"`
79 + EvidenceTypes map[string]EvidenceType `json:"evidence_types,omitempty"`
80 + TableTypes map[string]TableType `json:"table_types,omitempty"`
81 + OverlayTemplates map[string]OverlayTemplate `json:"overlay_templates,omitempty"`
82 + AggregationScopes map[string]AggregationScope `json:"aggregation_scopes,omitempty"`
83 +}
84 +
85 +type ActorType struct {
86 + Layer string `json:"layer"`
87 + Identity []string `json:"identity"`
88 + MergeIdentity []string `json:"merge_identity,omitempty"`
89 + ParentIdentity []string `json:"parent_identity,omitempty"`
90 + AggregationScopes []string `json:"aggregation_scopes,omitempty"`
91 + Search *ActorSearchPolicy `json:"search,omitempty"`
92 + Presentation *ActorPresentation `json:"presentation,omitempty"`
93 +}
94 +
95 +type LinkType struct {
96 + Orientation string `json:"orientation"`
97 + DirectionRole string `json:"direction_role"`
98 + SemanticRole string `json:"semantic_role,omitempty"`
99 + Aggregation LinkAggregation `json:"aggregation"`
100 + EvidenceTypes []string `json:"evidence_types,omitempty"`
101 + OverlayTemplates []string `json:"overlay_templates,omitempty"`
102 + Presentation *LinkPresentation `json:"presentation,omitempty"`
103 +}
104 +
105 +type PortType struct {
106 + Presentation *PortPresentation `json:"presentation,omitempty"`
107 +}
108 +
109 +type Presentation struct {
110 + ProfileVersion string `json:"profile_version,omitempty"`
111 + Selection *SelectionPresentation `json:"selection,omitempty"`
112 + Legend *PresentationLegend `json:"legend,omitempty"`
113 + PortFields []PresentationField `json:"port_fields,omitempty"`
114 + ScaleKeys map[string]ScaleKeyPresentation `json:"scale_keys,omitempty"`
115 +}
116 +
117 +type SelectionPresentation struct {
118 + ActorClick *ActorClickPresentation `json:"actor_click,omitempty"`
119 +}
120 +
121 +type ActorClickPresentation struct {
122 + Mode string `json:"mode"`
123 + PathTable string `json:"path_table,omitempty"`
124 + PathOwnerColumn string `json:"path_owner_column,omitempty"`
125 + PathActorColumn string `json:"path_actor_column,omitempty"`
126 + PathOrderColumn string `json:"path_order_column,omitempty"`
127 +}
128 +
129 +type PresentationLegend struct {
130 + Actors []LegendEntry `json:"actors,omitempty"`
131 + Links []LegendEntry `json:"links,omitempty"`
132 + Ports []LegendEntry `json:"ports,omitempty"`
133 +}
134 +
135 +type LegendEntry struct {
136 + Type string `json:"type"`
137 + Label string `json:"label,omitempty"`
138 +}
139 +
140 +type PresentationField struct {
141 + Key string `json:"key"`
142 + Label string `json:"label"`
143 +}
144 +
145 +type ScaleKeyPresentation struct {
146 + Label string `json:"label"`
147 + Unit string `json:"unit,omitempty"`
148 +}
149 +
150 +type ActorPresentation struct {
151 + Label string `json:"label,omitempty"`
152 + Role string `json:"role,omitempty"`
153 + Icon string `json:"icon,omitempty"`
154 + ColorSlot string `json:"color_slot,omitempty"`
155 + Opacity string `json:"opacity,omitempty"`
156 + Border *BorderPresentation `json:"border,omitempty"`
157 + Annotation *AnnotationPresentation `json:"annotation,omitempty"`
158 + Size *ActorSizePresentation `json:"size,omitempty"`
159 + Layout *ActorLayoutPresentation `json:"layout,omitempty"`
160 + LabelPolicy *LabelPolicy `json:"label_policy,omitempty"`
161 + Ports *ActorPortsPresentation `json:"ports,omitempty"`
162 + Hover *HoverPresentation `json:"hover,omitempty"`
163 + Modal *ModalPresentation `json:"modal,omitempty"`
164 +}
165 +
166 +type LinkPresentation struct {
167 + Label string `json:"label,omitempty"`
168 + ColorSlot string `json:"color_slot,omitempty"`
169 + Opacity string `json:"opacity,omitempty"`
170 + LineStyle string `json:"line_style,omitempty"`
171 + Width string `json:"width,omitempty"`
172 + Curve string `json:"curve,omitempty"`
173 + Arrow string `json:"arrow,omitempty"`
174 + Variable *LinkVariablePresentation `json:"variable,omitempty"`
175 + Hover *HoverPresentation `json:"hover,omitempty"`
176 + Layout *LinkLayoutPresentation `json:"layout,omitempty"`
177 + Modal *ModalPresentation `json:"modal,omitempty"`
178 +}
179 +
180 +type LinkLayoutPresentation struct {
181 + Strength string `json:"strength,omitempty"`
182 + Distance string `json:"distance,omitempty"`
183 +}
184 +
185 +type PortPresentation struct {
186 + Label string `json:"label,omitempty"`
187 + ColorSlot string `json:"color_slot,omitempty"`
188 + Opacity string `json:"opacity,omitempty"`
189 +}
190 +
191 +type BorderPresentation struct {
192 + Enabled *bool `json:"enabled,omitempty"`
193 + ColorSlot string `json:"color_slot,omitempty"`
194 + Style string `json:"style,omitempty"`
195 +}
196 +
197 +type AnnotationPresentation struct {
198 + ColorSlot string `json:"color_slot,omitempty"`
199 + Style string `json:"style,omitempty"`
200 +}
201 +
202 +type ActorSizePresentation struct {
203 + Mode string `json:"mode"`
204 + MetricColumn string `json:"metric_column,omitempty"`
205 + Scale string `json:"scale,omitempty"`
206 +}
207 +
208 +type ActorLayoutPresentation struct {
209 + Repulsion string `json:"repulsion,omitempty"`
210 +}
211 +
212 +type ActorSearchPolicy struct {
213 + Enabled *bool `json:"enabled,omitempty"`
214 + Columns []string `json:"columns,omitempty"`
215 + LabelKeys []string `json:"label_keys,omitempty"`
216 +}
217 +
218 +type LabelPolicy struct {
219 + Columns []string `json:"columns,omitempty"`
220 + Fallback string `json:"fallback,omitempty"`
221 + MaxLength int `json:"max_length,omitempty"`
222 + Array string `json:"array,omitempty"`
223 +}
224 +
225 +type ActorPortsPresentation struct {
226 + ShowBullets bool `json:"show_bullets,omitempty"`
227 + Sources []PortSourcePresentation `json:"sources,omitempty"`
228 +}
229 +
230 +type HoverPresentation struct {
231 + Fields []PresentationField `json:"fields,omitempty"`
232 +}
233 +
234 +type PortSourcePresentation struct {
235 + Source string `json:"source"`
236 + Table string `json:"table,omitempty"`
237 + Evidence string `json:"evidence,omitempty"`
238 + ActorColumn string `json:"actor_column"`
239 + NameColumn string `json:"name_column"`
240 + ValueColumn string `json:"value_column,omitempty"`
241 + TypeColumn string `json:"type_column,omitempty"`
242 + DefaultType string `json:"default_type,omitempty"`
243 + StatusColumn string `json:"status_column,omitempty"`
244 + ModeColumn string `json:"mode_column,omitempty"`
245 + RoleColumn string `json:"role_column,omitempty"`
246 + SourcesColumn string `json:"sources_column,omitempty"`
247 +}
248 +
249 +type LinkVariablePresentation struct {
250 + Channel string `json:"channel"`
251 + ScaleKey string `json:"scale_key"`
252 + ValueColumn string `json:"value_column"`
253 + Min string `json:"min,omitempty"`
254 + Max string `json:"max,omitempty"`
255 +}
256 +
257 +type ModalPresentation struct {
258 + Enabled *bool `json:"enabled,omitempty"`
259 + Labels *ModalLabelsPresentation `json:"labels,omitempty"`
260 + MiniTopology *ModalMiniTopologyPresentation `json:"mini_topology,omitempty"`
261 + Sections []ModalSection `json:"sections,omitempty"`
262 +}
263 +
264 +type ModalLabelsPresentation struct {
265 + Enabled *bool `json:"enabled,omitempty"`
266 + Table string `json:"table,omitempty"`
267 + ActorColumn string `json:"actor_column,omitempty"`
268 + KeyColumn string `json:"key_column,omitempty"`
269 + ValueColumn string `json:"value_column,omitempty"`
270 + SourceColumn string `json:"source_column,omitempty"`
271 + KindColumn string `json:"kind_column,omitempty"`
272 + ValueIndexColumn string `json:"value_index_column,omitempty"`
273 + Identification *ModalLabelIdentificationPresentation `json:"identification,omitempty"`
274 +}
275 +
276 +type ModalLabelIdentificationPresentation struct {
277 + Enabled *bool `json:"enabled,omitempty"`
278 + Fields []ModalLabelIdentificationField `json:"fields,omitempty"`
279 +}
280 +
281 +type ModalLabelIdentificationField struct {
282 + Key string `json:"key"`
283 + Label string `json:"label"`
284 + MaxValues int `json:"max_values,omitempty"`
285 +}
286 +
287 +type ModalMiniTopologyPresentation struct {
288 + Enabled *bool `json:"enabled,omitempty"`
289 + Depth int `json:"depth,omitempty"`
290 + IncludeLinkTypes []string `json:"include_link_types,omitempty"`
291 + ExcludeLinkTypes []string `json:"exclude_link_types,omitempty"`
292 +}
293 +
294 +type ModalSection struct {
295 + ID string `json:"id"`
296 + Label string `json:"label"`
297 + Order int `json:"order,omitempty"`
298 + Source ModalSource `json:"source"`
299 + OwnerFilter *ModalOwnerFilter `json:"owner_filter,omitempty"`
300 + RowFilters []ModalRowFilter `json:"row_filters,omitempty"`
301 + Columns []ModalColumn `json:"columns"`
302 + Sort *ModalSort `json:"sort,omitempty"`
303 + EmptyLabel string `json:"empty_label,omitempty"`
304 +}
305 +
306 +type ModalSource struct {
307 + Kind string `json:"kind"`
308 + Table string `json:"table,omitempty"`
309 + Evidence string `json:"evidence,omitempty"`
310 +}
311 +
312 +type ModalOwnerFilter struct {
313 + Mode string `json:"mode"`
314 + ActorColumn string `json:"actor_column,omitempty"`
315 + LinkColumn string `json:"link_column,omitempty"`
316 + SrcActorColumn string `json:"src_actor_column,omitempty"`
317 + DstActorColumn string `json:"dst_actor_column,omitempty"`
318 +}
319 +
320 +type ModalRowFilter struct {
321 + Column string `json:"column"`
322 + Op string `json:"op"`
323 + Value any `json:"value,omitempty"`
324 + Values []any `json:"values,omitempty"`
325 +}
326 +
327 +type ModalColumn struct {
328 + ID string `json:"id"`
329 + Label string `json:"label"`
330 + Projection ModalProjection `json:"projection"`
331 + Cell string `json:"cell,omitempty"`
332 + Visibility string `json:"visibility,omitempty"`
333 + Align string `json:"align,omitempty"`
334 + Sortable *bool `json:"sortable,omitempty"`
335 + BadgeMap map[string]ModalBadgePresentation `json:"badge_map,omitempty"`
336 +}
337 +
338 +type ModalProjection struct {
339 + Kind string `json:"kind"`
340 + Column string `json:"column,omitempty"`
341 + Columns []string `json:"columns,omitempty"`
342 + Value any `json:"value,omitempty"`
343 + ActorColumn string `json:"actor_column,omitempty"`
344 + SrcActorColumn string `json:"src_actor_column,omitempty"`
345 + DstActorColumn string `json:"dst_actor_column,omitempty"`
346 + IPColumn string `json:"ip_column,omitempty"`
347 + PortColumn string `json:"port_column,omitempty"`
348 + ProtocolColumn string `json:"protocol_column,omitempty"`
349 + LocalIPColumn string `json:"local_ip_column,omitempty"`
350 + LocalPortColumn string `json:"local_port_column,omitempty"`
351 + RemoteIPColumn string `json:"remote_ip_column,omitempty"`
352 + RemotePortColumn string `json:"remote_port_column,omitempty"`
353 + LabelKey string `json:"label_key,omitempty"`
354 + Path string `json:"path,omitempty"`
355 + Fallback any `json:"fallback,omitempty"`
356 +}
357 +
358 +type ModalBadgePresentation struct {
359 + Label string `json:"label,omitempty"`
360 + ColorSlot string `json:"color_slot,omitempty"`
361 + Opacity string `json:"opacity,omitempty"`
362 +}
363 +
364 +type ModalSort struct {
365 + Column string `json:"column"`
366 + Direction string `json:"direction,omitempty"`
367 +}
368 +
369 +type LinkAggregation struct {
370 + Direction string `json:"direction"`
371 + Evidence string `json:"evidence,omitempty"`
372 + Metrics map[string]string `json:"metrics,omitempty"`
373 +}
374 +
375 +type Correlation struct {
376 + Rules map[string]CorrelationRule `json:"rules"`
377 + Points *Table `json:"points,omitempty"`
378 + Claims *Table `json:"claims,omitempty"`
379 +}
380 +
381 +type CorrelationRule struct {
382 + Action string `json:"action"`
383 + Class string `json:"class,omitempty"`
384 + Priority int `json:"priority"`
385 + KeySpace string `json:"key_space"`
386 + Key []CorrelationKeyPart `json:"key"`
387 + PointActorTypes []string `json:"point_actor_types"`
388 + ClaimActorTypes []string `json:"claim_actor_types,omitempty"`
389 + CorrelationLinkTypes []string `json:"correlation_link_types,omitempty"`
390 + OutputLinkType string `json:"output_link_type"`
391 +}
392 +
393 +type CorrelationKeyPart struct {
394 + Column string `json:"column,omitempty"`
395 + Literal string `json:"literal,omitempty"`
396 +}
397 +
398 +type EvidenceType struct {
399 + LinkType string `json:"link_type"`
400 + Role string `json:"role"`
401 + Columns []Column `json:"columns"`
402 + MatchColumns []string `json:"match_columns,omitempty"`
403 +}
404 +
405 +type TableType struct {
406 + Role string `json:"role"`
407 + Owner string `json:"owner"`
408 + Aggregation string `json:"aggregation"`
409 + SourceEvidence string `json:"source_evidence,omitempty"`
410 + Columns []Column `json:"columns"`
411 + Presentation *TableTypePresentation `json:"presentation,omitempty"`
412 +}
413 +
414 +type TableTypePresentation struct {
415 + Label string `json:"label,omitempty"`
416 + Order int `json:"order,omitempty"`
417 + DefaultVisibility string `json:"default_visibility,omitempty"`
418 + Columns []ModalColumn `json:"columns,omitempty"`
419 +}
420 +
421 +type OverlayTemplate struct {
422 + Provider string `json:"provider"`
423 + Contexts []string `json:"contexts,omitempty"`
424 + Dimensions []string `json:"dimensions,omitempty"`
425 + SelectorParams []string `json:"selector_params,omitempty"`
426 + Merge OverlayMerge `json:"merge"`
427 +}
428 +
429 +type OverlayMerge struct {
430 + Refs string `json:"refs"`
431 + Values string `json:"values"`
432 +}
433 +
434 +type AggregationScope struct {
435 + Columns []string `json:"columns"`
436 + EvidencePolicy string `json:"evidence_policy,omitempty"`
437 +}
438 +
439 +//go:fix inline
440 +func Bool(value bool) *bool {
441 + return new(value)
442 +}
443 +
444 +type EvidenceMap map[string]EvidenceSection
445 +
446 +type EvidenceSection struct {
447 + Type string `json:"type"`
448 + Table Table `json:"table"`
449 +}
450 +
451 +type DetailTables struct {
452 + Actor map[string]DetailTable `json:"actor,omitempty"`
453 + Relationship map[string]DetailTable `json:"relationship,omitempty"`
454 +}
455 +
456 +type DetailTable struct {
457 + Type string `json:"type"`
458 + Table Table `json:"table"`
459 +}
460 +
461 +type OverlayRefs struct {
462 + Refs *Table `json:"refs,omitempty"`
463 +}
464 +
465 +type Table struct {
466 + Rows int `json:"rows"`
467 + Columns []Column `json:"columns"`
468 + Values []ColumnEncoding `json:"values"`
469 +}
470 +
471 +type Column struct {
472 + ID string `json:"id"`
473 + Type string `json:"type"`
474 + Dictionary string `json:"dictionary,omitempty"`
475 + Nullable bool `json:"nullable,omitempty"`
476 + Unit string `json:"unit,omitempty"`
477 + Role string `json:"role,omitempty"`
478 + Aggregation string `json:"aggregation,omitempty"`
479 +}
480 +
481 +type ColumnOption func(*Column)
482 +
483 +func NewColumn(id, typ string, opts ...ColumnOption) Column {
484 + column := Column{ID: id, Type: typ}
485 + for _, opt := range opts {
486 + opt(&column)
487 + }
488 + return column
489 +}
490 +
491 +func WithDictionary(name string) ColumnOption {
492 + return func(column *Column) {
493 + column.Dictionary = name
494 + }
495 +}
496 +
497 +func WithNullable() ColumnOption {
498 + return func(column *Column) {
499 + column.Nullable = true
500 + }
501 +}
502 +
503 +func WithUnit(unit string) ColumnOption {
504 + return func(column *Column) {
505 + column.Unit = unit
506 + }
507 +}
508 +
509 +func WithRole(role string) ColumnOption {
510 + return func(column *Column) {
511 + column.Role = role
512 + }
513 +}
514 +
515 +func WithAggregation(rule string) ColumnOption {
516 + return func(column *Column) {
517 + column.Aggregation = rule
518 + }
519 +}
520 +
521 +type ColumnEncoding interface {
522 + isColumnEncoding()
523 +}
524 +
525 +type ConstEncoding struct {
526 + Codec string `json:"codec"`
527 + Value any `json:"value"`
528 +}
529 +
530 +func (ConstEncoding) isColumnEncoding() {
531 + // Marker method for the closed ColumnEncoding union.
532 +}
533 +
534 +func Const(value any) ConstEncoding {
535 + return ConstEncoding{
536 + Codec: "const",
537 + Value: value,
538 + }
539 +}
540 +
541 +type ValuesEncoding struct {
542 + Codec string `json:"codec"`
543 + Values []any `json:"values"`
544 +}
545 +
546 +func (ValuesEncoding) isColumnEncoding() {
547 + // Marker method for the closed ColumnEncoding union.
548 +}
549 +
550 +func Values(values ...any) ValuesEncoding {
551 + return ValuesEncoding{
552 + Codec: "values",
553 + Values: append([]any(nil), values...),
554 + }
555 +}
556 +
557 +type DictEncoding struct {
558 + Codec string `json:"codec"`
559 + Values []any `json:"values"`
560 + Indexes []int `json:"indexes"`
561 +}
562 +
563 +func (DictEncoding) isColumnEncoding() {
564 + // Marker method for the closed ColumnEncoding union.
565 +}
566 +
567 +func Dict(values []any, indexes ...int) DictEncoding {
568 + return DictEncoding{
569 + Codec: "dict",
570 + Values: append([]any(nil), values...),
571 + Indexes: append([]int(nil), indexes...),
572 + }
573 +}
574 +
575 +func StringValues(values ...string) []any {
576 + result := make([]any, len(values))
577 + for i, value := range values {
578 + result[i] = value
579 + }
580 + return result
581 +}
src/go/pkg/topology/v1/validate.go new
+2304
@@ -0,0 +1,2304 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package topologyv1
4 +
5 +import (
6 + "encoding/json"
7 + "fmt"
8 + "math"
9 + "slices"
10 +)
11 +
12 +type validationContext struct {
13 + dictionaries map[string]any
14 + actorRows int
15 + linkRows int
16 + evidenceRowsByType map[string]int
17 + evidenceRows int
18 +}
19 +
20 +type topologyShape struct {
21 + actorColumns map[string]string
22 + linkColumns map[string]string
23 + actorTypes map[string]struct{}
24 + linkTypes map[string]struct{}
25 + portTypes map[string]struct{}
26 + evidenceTypes map[string]map[string]string
27 + tableTypes map[string]map[string]string
28 + tableTypeOwners map[string]string
29 + actorTables map[string]map[string]string
30 + relationshipTables map[string]map[string]string
31 + scaleKeys map[string]struct{}
32 +}
33 +
34 +// ValidateDecodedResponse validates a full topology v1 Function response.
35 +// Metadata-only Function info responses intentionally omit data and must not be
36 +// passed here.
37 +func ValidateDecodedResponse(payload any) error {
38 + obj, ok := payload.(map[string]any)
39 + if !ok {
40 + return fmt.Errorf("response is not an object")
41 + }
42 + data, ok := obj["data"].(map[string]any)
43 + if !ok {
44 + return fmt.Errorf("response.data is not an object")
45 + }
46 + if data["schema_version"] != SchemaVersion {
47 + return fmt.Errorf("response.data.schema_version is not %q", SchemaVersion)
48 + }
49 +
50 + return ValidateDecodedData(data)
51 +}
52 +
53 +func ValidateDecodedData(data map[string]any) error {
54 + dictionaries, ok := data["dictionaries"].(map[string]any)
55 + if !ok {
56 + return fmt.Errorf("data.dictionaries is not an object")
57 + }
58 +
59 + actorRows, err := decodedTableRows(data["actors"])
60 + if err != nil {
61 + return fmt.Errorf("data.actors: %w", err)
62 + }
63 + linkRows, err := decodedTableRows(data["links"])
64 + if err != nil {
65 + return fmt.Errorf("data.links: %w", err)
66 + }
67 +
68 + evidenceRowsByType, err := collectEvidenceRows(data["evidence"])
69 + if err != nil {
70 + return err
71 + }
72 + tableEvidenceSources := collectTableEvidenceSources(data["types"])
73 +
74 + ctx := validationContext{
75 + dictionaries: dictionaries,
76 + actorRows: actorRows,
77 + linkRows: linkRows,
78 + evidenceRowsByType: evidenceRowsByType,
79 + evidenceRows: -1,
80 + }
81 +
82 + if _, err := validateCompactTable("data.actors", data["actors"], ctx); err != nil {
83 + return err
84 + }
85 + if _, err := validateCompactTable("data.links", data["links"], ctx); err != nil {
86 + return err
87 + }
88 + if err := validateEvidenceSections(data["evidence"], ctx); err != nil {
89 + return err
90 + }
91 + if err := validateDetailTables(data["tables"], ctx, tableEvidenceSources); err != nil {
92 + return err
93 + }
94 + if err := validateOverlayRefs(data["overlays"], ctx); err != nil {
95 + return err
96 + }
97 + shape, err := collectTopologyShape(data)
98 + if err != nil {
99 + return err
100 + }
101 + if err := validateTypeColumns(data, shape); err != nil {
102 + return err
103 + }
104 + if err := validatePresentation(data, shape); err != nil {
105 + return err
106 + }
107 + if err := validateCorrelation(data["correlation"], shape, ctx); err != nil {
108 + return err
109 + }
110 +
111 + return nil
112 +}
113 +
114 +func IsDecodedData(raw any) bool {
115 + data, ok := raw.(map[string]any)
116 + return ok && data["schema_version"] == SchemaVersion
117 +}
118 +
119 +func LinkRowsFromDecodedData(raw any) (int, error) {
120 + data, ok := raw.(map[string]any)
121 + if !ok {
122 + return 0, fmt.Errorf("data is not an object")
123 + }
124 + rows, err := decodedTableRows(data["links"])
125 + if err != nil {
126 + return 0, fmt.Errorf("data.links: %w", err)
127 + }
128 + return rows, nil
129 +}
130 +
131 +func GraphRowsFromDecodedData(raw any) (int, error) {
132 + data, ok := raw.(map[string]any)
133 + if !ok {
134 + return 0, fmt.Errorf("data is not an object")
135 + }
136 + actorRows, err := decodedTableRows(data["actors"])
137 + if err != nil {
138 + return 0, fmt.Errorf("data.actors: %w", err)
139 + }
140 + linkRows, err := decodedTableRows(data["links"])
141 + if err != nil {
142 + return 0, fmt.Errorf("data.links: %w", err)
143 + }
144 + return max(actorRows, linkRows), nil
145 +}
146 +
147 +func collectEvidenceRows(raw any) (map[string]int, error) {
148 + rowsByType := make(map[string]int)
149 + if raw == nil {
150 + return rowsByType, nil
151 + }
152 + sections, ok := raw.(map[string]any)
153 + if !ok {
154 + return nil, fmt.Errorf("data.evidence is not an object")
155 + }
156 + for name, rawSection := range sections {
157 + section, ok := rawSection.(map[string]any)
158 + if !ok {
159 + return nil, fmt.Errorf("data.evidence.%s is not an object", name)
160 + }
161 + typ, ok := section["type"].(string)
162 + if !ok || typ == "" {
163 + return nil, fmt.Errorf("data.evidence.%s.type is empty", name)
164 + }
165 + rows, err := decodedTableRows(section["table"])
166 + if err != nil {
167 + return nil, fmt.Errorf("data.evidence.%s.table: %w", name, err)
168 + }
169 + rowsByType[name] = rows
170 + rowsByType[typ] = rows
171 + }
172 + return rowsByType, nil
173 +}
174 +
175 +func collectTableEvidenceSources(rawTypes any) map[string]string {
176 + sources := make(map[string]string)
177 + types, ok := rawTypes.(map[string]any)
178 + if !ok {
179 + return sources
180 + }
181 + tableTypes, ok := types["table_types"].(map[string]any)
182 + if !ok {
183 + return sources
184 + }
185 + for name, rawTableType := range tableTypes {
186 + tableType, ok := rawTableType.(map[string]any)
187 + if !ok {
188 + continue
189 + }
190 + source, ok := tableType["source_evidence"].(string)
191 + if ok && source != "" {
192 + sources[name] = source
193 + }
194 + }
195 + return sources
196 +}
197 +
198 +func validateEvidenceSections(raw any, ctx validationContext) error {
199 + if raw == nil {
200 + return nil
201 + }
202 + sections, ok := raw.(map[string]any)
203 + if !ok {
204 + return fmt.Errorf("data.evidence is not an object")
205 + }
206 + for name, rawSection := range sections {
207 + section, ok := rawSection.(map[string]any)
208 + if !ok {
209 + return fmt.Errorf("data.evidence.%s is not an object", name)
210 + }
211 + if _, err := validateCompactTable("data.evidence."+name+".table", section["table"], ctx); err != nil {
212 + return err
213 + }
214 + }
215 + return nil
216 +}
217 +
218 +func validateDetailTables(raw any, ctx validationContext, tableEvidenceSources map[string]string) error {
219 + if raw == nil {
220 + return nil
221 + }
222 + tables, ok := raw.(map[string]any)
223 + if !ok {
224 + return fmt.Errorf("data.tables is not an object")
225 + }
226 + for _, groupName := range []string{"actor", "relationship"} {
227 + groupRaw, ok := tables[groupName]
228 + if !ok {
229 + continue
230 + }
231 + group, ok := groupRaw.(map[string]any)
232 + if !ok {
233 + return fmt.Errorf("data.tables.%s is not an object", groupName)
234 + }
235 + for name, rawTable := range group {
236 + detail, ok := rawTable.(map[string]any)
237 + if !ok {
238 + return fmt.Errorf("data.tables.%s.%s is not an object", groupName, name)
239 + }
240 + tableCtx := ctx
241 + tableType, _ := detail["type"].(string)
242 + if sourceEvidence := tableEvidenceSources[tableType]; sourceEvidence != "" {
243 + rows, ok := ctx.evidenceRowsByType[sourceEvidence]
244 + if !ok {
245 + return fmt.Errorf("data.tables.%s.%s references unknown source_evidence %q", groupName, name, sourceEvidence)
246 + }
247 + tableCtx.evidenceRows = rows
248 + }
249 + if _, err := validateCompactTable("data.tables."+groupName+"."+name+".table", detail["table"], tableCtx); err != nil {
250 + return err
251 + }
252 + }
253 + }
254 + return nil
255 +}
256 +
257 +func validateOverlayRefs(raw any, ctx validationContext) error {
258 + if raw == nil {
259 + return nil
260 + }
261 + overlays, ok := raw.(map[string]any)
262 + if !ok {
263 + return fmt.Errorf("data.overlays is not an object")
264 + }
265 + refs, ok := overlays["refs"]
266 + if !ok {
267 + return nil
268 + }
269 + if _, err := validateCompactTable("data.overlays.refs", refs, ctx); err != nil {
270 + return err
271 + }
272 + return nil
273 +}
274 +
275 +func validateCompactTable(path string, raw any, ctx validationContext) (int, error) {
276 + table, ok := raw.(map[string]any)
277 + if !ok {
278 + return 0, fmt.Errorf("%s is not an object", path)
279 + }
280 + rows, err := decodedTableRows(table)
281 + if err != nil {
282 + return 0, fmt.Errorf("%s: %w", path, err)
283 + }
284 + columns, ok := table["columns"].([]any)
285 + if !ok {
286 + return 0, fmt.Errorf("%s.columns is not an array", path)
287 + }
288 + values, ok := table["values"].([]any)
289 + if !ok {
290 + return 0, fmt.Errorf("%s.values is not an array", path)
291 + }
292 + if len(columns) != len(values) {
293 + return 0, fmt.Errorf("%s columns/values length mismatch: %d columns, %d values", path, len(columns), len(values))
294 + }
295 +
296 + seenColumns := make(map[string]struct{}, len(columns))
297 + for i := range columns {
298 + column, ok := columns[i].(map[string]any)
299 + if !ok {
300 + return 0, fmt.Errorf("%s.columns[%d] is not an object", path, i)
301 + }
302 + columnID, _ := column["id"].(string)
303 + if columnID == "" {
304 + return 0, fmt.Errorf("%s.columns[%d].id is required", path, i)
305 + }
306 + if _, ok := seenColumns[columnID]; ok {
307 + return 0, fmt.Errorf("%s.columns[%d].id duplicates column %q", path, i, columnID)
308 + }
309 + seenColumns[columnID] = struct{}{}
310 + columnType, _ := column["type"].(string)
311 + if columnType == "" {
312 + return 0, fmt.Errorf("%s.columns[%d].type is required", path, i)
313 + }
314 + decoded, err := decodeColumn(path, i, rows, values[i])
315 + if err != nil {
316 + return 0, err
317 + }
318 + if err := validateColumnValues(fmt.Sprintf("%s.%s", path, columnID), column, columnType, decoded, ctx); err != nil {
319 + return 0, err
320 + }
321 + }
322 +
323 + return rows, nil
324 +}
325 +
326 +func decodeColumn(path string, columnIndex int, rows int, raw any) ([]any, error) {
327 + encoding, ok := raw.(map[string]any)
328 + if !ok {
329 + return nil, fmt.Errorf("%s.values[%d] is not an object", path, columnIndex)
330 + }
331 + codec, _ := encoding["codec"].(string)
332 + switch codec {
333 + case "const":
334 + value, ok := encoding["value"]
335 + if !ok {
336 + return nil, fmt.Errorf("%s.values[%d] const encoding missing value", path, columnIndex)
337 + }
338 + values := make([]any, rows)
339 + for i := range values {
340 + values[i] = value
341 + }
342 + return values, nil
343 + case "values":
344 + values, ok := encoding["values"].([]any)
345 + if !ok {
346 + return nil, fmt.Errorf("%s.values[%d] values encoding missing values array", path, columnIndex)
347 + }
348 + if len(values) != rows {
349 + return nil, fmt.Errorf("%s.values[%d] decoded length mismatch: expected %d, got %d", path, columnIndex, rows, len(values))
350 + }
351 + return values, nil
352 + case "dict":
353 + dictValues, ok := encoding["values"].([]any)
354 + if !ok {
355 + return nil, fmt.Errorf("%s.values[%d] dict encoding missing values array", path, columnIndex)
356 + }
357 + indexes, ok := encoding["indexes"].([]any)
358 + if !ok {
359 + return nil, fmt.Errorf("%s.values[%d] dict encoding missing indexes array", path, columnIndex)
360 + }
361 + if len(indexes) != rows {
362 + return nil, fmt.Errorf("%s.values[%d] decoded length mismatch: expected %d, got %d", path, columnIndex, rows, len(indexes))
363 + }
364 + values := make([]any, rows)
365 + for i, rawIndex := range indexes {
366 + index, ok := integerValue(rawIndex)
367 + if !ok {
368 + return nil, fmt.Errorf("%s.values[%d].indexes[%d] is not an integer", path, columnIndex, i)
369 + }
370 + if index < 0 || index >= len(dictValues) {
371 + return nil, fmt.Errorf("%s.values[%d].indexes[%d] out of bounds: %d", path, columnIndex, i, index)
372 + }
373 + values[i] = dictValues[index]
374 + }
375 + return values, nil
376 + default:
377 + return nil, fmt.Errorf("%s.values[%d] unsupported codec %q", path, columnIndex, codec)
378 + }
379 +}
380 +
381 +func validateColumnValues(path string, column map[string]any, columnType string, values []any, ctx validationContext) error {
382 + nullable, _ := column["nullable"].(bool)
383 + for i, value := range values {
384 + if value == nil {
385 + if nullable {
386 + continue
387 + }
388 + return fmt.Errorf("%s[%d] is null but column is not nullable", path, i)
389 + }
390 +
391 + switch columnType {
392 + case "string_ref", "ip_ref", "mac_ref":
393 + dictName, _ := column["dictionary"].(string)
394 + if dictName == "" {
395 + return fmt.Errorf("%s is %s without dictionary", path, columnType)
396 + }
397 + dict, ok := ctx.dictionaries[dictName].([]any)
398 + if !ok {
399 + return fmt.Errorf("%s references missing dictionary %q", path, dictName)
400 + }
401 + index, ok := integerValue(value)
402 + if !ok {
403 + return fmt.Errorf("%s[%d] is not an integer dictionary reference", path, i)
404 + }
405 + if index < 0 || index >= len(dict) {
406 + return fmt.Errorf("%s[%d] dictionary index out of bounds: %d", path, i, index)
407 + }
408 + case "actor_ref":
409 + if err := validateReference(path, i, value, ctx.actorRows, "actor"); err != nil {
410 + return err
411 + }
412 + case "link_ref":
413 + if err := validateReference(path, i, value, ctx.linkRows, "link"); err != nil {
414 + return err
415 + }
416 + case "evidence_ref":
417 + index, ok := integerValue(value)
418 + if !ok || index < 0 {
419 + return fmt.Errorf("%s[%d] is not a non-negative evidence reference", path, i)
420 + }
421 + if ctx.evidenceRows >= 0 && index >= ctx.evidenceRows {
422 + return fmt.Errorf("%s[%d] evidence reference out of bounds: %d", path, i, index)
423 + }
424 + case "array":
425 + if _, ok := value.([]any); !ok {
426 + return fmt.Errorf("%s[%d] is not an array", path, i)
427 + }
428 + case "bool":
429 + if _, ok := value.(bool); !ok {
430 + return fmt.Errorf("%s[%d] is not a bool", path, i)
431 + }
432 + case "int":
433 + if _, ok := integerValue(value); !ok {
434 + return fmt.Errorf("%s[%d] is not an integer", path, i)
435 + }
436 + case "uint":
437 + n, ok := integerValue(value)
438 + if !ok || n < 0 {
439 + return fmt.Errorf("%s[%d] is not a non-negative integer", path, i)
440 + }
441 + case "float", "duration":
442 + if _, ok := numberValue(value); !ok {
443 + return fmt.Errorf("%s[%d] is not a number", path, i)
444 + }
445 + case "string", "ip", "mac", "timestamp":
446 + if _, ok := value.(string); !ok {
447 + return fmt.Errorf("%s[%d] is not a string", path, i)
448 + }
449 + case "json":
450 + // Any decoded JSON value is valid for a json column.
451 + default:
452 + return fmt.Errorf("%s has unsupported column type %q", path, columnType)
453 + }
454 + }
455 + return nil
456 +}
457 +
458 +func collectTopologyShape(data map[string]any) (topologyShape, error) {
459 + types, ok := data["types"].(map[string]any)
460 + if !ok {
461 + return topologyShape{}, fmt.Errorf("data.types is not an object")
462 + }
463 + actorColumns, err := columnTypesFromTable(data["actors"], "data.actors")
464 + if err != nil {
465 + return topologyShape{}, err
466 + }
467 + linkColumns, err := columnTypesFromTable(data["links"], "data.links")
468 + if err != nil {
469 + return topologyShape{}, err
470 + }
471 + actorTypes, err := objectKeySet(types["actor_types"], "data.types.actor_types")
472 + if err != nil {
473 + return topologyShape{}, err
474 + }
475 + linkTypes, err := objectKeySet(types["link_types"], "data.types.link_types")
476 + if err != nil {
477 + return topologyShape{}, err
478 + }
479 + portTypes, err := optionalObjectKeySet(types["port_types"], "data.types.port_types")
480 + if err != nil {
481 + return topologyShape{}, err
482 + }
483 +
484 + evidenceTypes, err := columnTypesByRegistryObject(types["evidence_types"], "data.types.evidence_types")
485 + if err != nil {
486 + return topologyShape{}, err
487 + }
488 + tableTypes, err := columnTypesByRegistryObject(types["table_types"], "data.types.table_types")
489 + if err != nil {
490 + return topologyShape{}, err
491 + }
492 + tableTypeOwners, err := tableTypeOwners(types["table_types"], "data.types.table_types")
493 + if err != nil {
494 + return topologyShape{}, err
495 + }
496 + actorTables, relationshipTables, err := collectDetailTableColumnTypes(data["tables"])
497 + if err != nil {
498 + return topologyShape{}, err
499 + }
500 + scaleKeys, err := collectScaleKeys(data["presentation"])
501 + if err != nil {
502 + return topologyShape{}, err
503 + }
504 +
505 + return topologyShape{
506 + actorColumns: actorColumns,
507 + linkColumns: linkColumns,
508 + actorTypes: actorTypes,
509 + linkTypes: linkTypes,
510 + portTypes: portTypes,
511 + evidenceTypes: evidenceTypes,
512 + tableTypes: tableTypes,
513 + tableTypeOwners: tableTypeOwners,
514 + actorTables: actorTables,
515 + relationshipTables: relationshipTables,
516 + scaleKeys: scaleKeys,
517 + }, nil
518 +}
519 +
520 +func validateTypeColumns(data map[string]any, shape topologyShape) error {
521 + if _, ok := shape.actorColumns["type"]; !ok {
522 + return fmt.Errorf("data.actors is missing required type column")
523 + }
524 + if _, ok := shape.linkColumns["type"]; !ok {
525 + return fmt.Errorf("data.links is missing required type column")
526 + }
527 + dictionaries, _ := data["dictionaries"].(map[string]any)
528 + if err := validateTypeColumnValues("data.actors", data["actors"], shape.actorTypes, "actor", dictionaries); err != nil {
529 + return err
530 + }
531 + if err := validateTypeColumnValues("data.links", data["links"], shape.linkTypes, "link", dictionaries); err != nil {
532 + return err
533 + }
534 + return nil
535 +}
536 +
537 +func validateTypeColumnValues(path string, raw any, known map[string]struct{}, typeName string, dictionaries map[string]any) error {
538 + table, ok := raw.(map[string]any)
539 + if !ok {
540 + return fmt.Errorf("%s is not an object", path)
541 + }
542 + rows, err := decodedTableRows(table)
543 + if err != nil {
544 + return fmt.Errorf("%s: %w", path, err)
545 + }
546 + columns, ok := table["columns"].([]any)
547 + if !ok {
548 + return fmt.Errorf("%s.columns is not an array", path)
549 + }
550 + values, ok := table["values"].([]any)
551 + if !ok {
552 + return fmt.Errorf("%s.values is not an array", path)
553 + }
554 + for i, rawColumn := range columns {
555 + column, ok := rawColumn.(map[string]any)
556 + if !ok {
557 + return fmt.Errorf("%s.columns[%d] is not an object", path, i)
558 + }
559 + if id, _ := column["id"].(string); id != "type" {
560 + continue
561 + }
562 + columnType, _ := column["type"].(string)
563 + dictionary, _ := column["dictionary"].(string)
564 + decoded, err := decodeColumn(path, i, rows, values[i])
565 + if err != nil {
566 + return err
567 + }
568 + for row, value := range decoded {
569 + id, ok := resolveStringValue(value, columnType, dictionary, dictionaries)
570 + if !ok || id == "" {
571 + return fmt.Errorf("%s.type[%d] is not a non-empty string", path, row)
572 + }
573 + if _, ok := known[id]; !ok {
574 + return fmt.Errorf("%s.type[%d] references unknown %s type %q", path, row, typeName, id)
575 + }
576 + }
577 + return nil
578 + }
579 + return fmt.Errorf("%s is missing required type column", path)
580 +}
581 +
582 +func validateStringColumnValuesInSet(path string, raw any, columnID string, known map[string]struct{}, typeName string, dictionaries map[string]any) error {
583 + _, err := collectStringColumnValuesInSet(path, raw, columnID, known, typeName, dictionaries)
584 + return err
585 +}
586 +
587 +func collectStringColumnValuesInSet(path string, raw any, columnID string, known map[string]struct{}, typeName string, dictionaries map[string]any) (map[string]struct{}, error) {
588 + table, ok := raw.(map[string]any)
589 + if !ok {
590 + return nil, fmt.Errorf("%s is not an object", path)
591 + }
592 + rows, err := decodedTableRows(table)
593 + if err != nil {
594 + return nil, fmt.Errorf("%s: %w", path, err)
595 + }
596 + columns, ok := table["columns"].([]any)
597 + if !ok {
598 + return nil, fmt.Errorf("%s.columns is not an array", path)
599 + }
600 + values, ok := table["values"].([]any)
601 + if !ok {
602 + return nil, fmt.Errorf("%s.values is not an array", path)
603 + }
604 + for i, rawColumn := range columns {
605 + column, ok := rawColumn.(map[string]any)
606 + if !ok {
607 + return nil, fmt.Errorf("%s.columns[%d] is not an object", path, i)
608 + }
609 + if id, _ := column["id"].(string); id != columnID {
610 + continue
611 + }
612 + columnType, _ := column["type"].(string)
613 + dictionary, _ := column["dictionary"].(string)
614 + decoded, err := decodeColumn(path, i, rows, values[i])
615 + if err != nil {
616 + return nil, err
617 + }
618 + found := make(map[string]struct{}, len(decoded))
619 + for row, value := range decoded {
620 + id, ok := resolveStringValue(value, columnType, dictionary, dictionaries)
621 + if !ok || id == "" {
622 + return nil, fmt.Errorf("%s.%s[%d] is not a non-empty string", path, columnID, row)
623 + }
624 + if _, ok := known[id]; !ok {
625 + return nil, fmt.Errorf("%s.%s[%d] references unknown %s %q", path, columnID, row, typeName, id)
626 + }
627 + found[id] = struct{}{}
628 + }
629 + return found, nil
630 + }
631 + return nil, fmt.Errorf("%s is missing required %s column", path, columnID)
632 +}
633 +
634 +func resolveStringValue(value any, columnType, dictionary string, dictionaries map[string]any) (string, bool) {
635 + if text, ok := value.(string); ok {
636 + return text, true
637 + }
638 + if columnType != "string_ref" && columnType != "ip_ref" && columnType != "mac_ref" {
639 + return "", false
640 + }
641 + index, ok := integerValue(value)
642 + if !ok || dictionary == "" {
643 + return "", false
644 + }
645 + values, ok := dictionaries[dictionary].([]any)
646 + if !ok || index < 0 || index >= len(values) {
647 + return "", false
648 + }
649 + text, ok := values[index].(string)
650 + return text, ok
651 +}
652 +
653 +func validatePresentation(data map[string]any, shape topologyShape) error {
654 + types, _ := data["types"].(map[string]any)
655 + if err := validateActorTypePresentation(types["actor_types"], shape); err != nil {
656 + return err
657 + }
658 + if err := validateLinkTypePresentation(types["link_types"], shape); err != nil {
659 + return err
660 + }
661 + if err := validatePortTypePresentation(types["port_types"]); err != nil {
662 + return err
663 + }
664 + if err := validateTableTypePresentation(types["table_types"], shape); err != nil {
665 + return err
666 + }
667 + if err := validateGraphPresentation(data["presentation"], shape); err != nil {
668 + return err
669 + }
670 + return nil
671 +}
672 +
673 +func validateActorTypePresentation(raw any, shape topologyShape) error {
674 + actorTypes, ok := raw.(map[string]any)
675 + if !ok {
676 + return fmt.Errorf("data.types.actor_types is not an object")
677 + }
678 + for typeID, rawType := range actorTypes {
679 + actorType, ok := rawType.(map[string]any)
680 + if !ok {
681 + return fmt.Errorf("data.types.actor_types.%s is not an object", typeID)
682 + }
683 + presentation, ok := actorType["presentation"].(map[string]any)
684 + if !ok {
685 + continue
686 + }
687 + path := "data.types.actor_types." + typeID + ".presentation"
688 + if _, ok := presentation["label"]; ok {
689 + if err := validateRequiredString(path+".label", presentation["label"]); err != nil {
690 + return err
691 + }
692 + }
693 + if err := optionalEnum(path+".role", presentation["role"], "actor", "endpoint", "group"); err != nil {
694 + return err
695 + }
696 + if err := optionalEnum(path+".icon", presentation["icon"], iconTokens...); err != nil {
697 + return err
698 + }
699 + if err := optionalEnum(path+".color_slot", presentation["color_slot"], colorSlotTokens...); err != nil {
700 + return err
701 + }
702 + if err := optionalEnum(path+".opacity", presentation["opacity"], opacityTokens...); err != nil {
703 + return err
704 + }
705 + if err := validateBorderPresentation(path+".border", presentation["border"]); err != nil {
706 + return err
707 + }
708 + if err := validateAnnotationPresentation(path+".annotation", presentation["annotation"]); err != nil {
709 + return err
710 + }
711 + if err := validateActorSizePresentation(path+".size", presentation["size"], shape.actorColumns); err != nil {
712 + return err
713 + }
714 + if err := validateActorLayoutPresentation(path+".layout", presentation["layout"]); err != nil {
715 + return err
716 + }
717 + if err := validateLabelPolicy(path+".label_policy", presentation["label_policy"], shape.actorColumns); err != nil {
718 + return err
719 + }
720 + if err := validateActorPortsPresentation(path+".ports", presentation["ports"], shape); err != nil {
721 + return err
722 + }
723 + if err := validateHoverPresentation(path+".hover", presentation["hover"], shape.actorColumns); err != nil {
724 + return err
725 + }
726 + if err := validateModalPresentation(path+".modal", presentation["modal"], shape); err != nil {
727 + return err
728 + }
729 + if err := validateActorSearchPolicy("data.types.actor_types."+typeID+".search", actorType["search"], shape.actorColumns); err != nil {
730 + return err
731 + }
732 + }
733 + return nil
734 +}
735 +
736 +func validateLinkTypePresentation(raw any, shape topologyShape) error {
737 + linkTypes, ok := raw.(map[string]any)
738 + if !ok {
739 + return fmt.Errorf("data.types.link_types is not an object")
740 + }
741 + for typeID, rawType := range linkTypes {
742 + linkType, ok := rawType.(map[string]any)
743 + if !ok {
744 + return fmt.Errorf("data.types.link_types.%s is not an object", typeID)
745 + }
746 + if err := optionalEnum("data.types.link_types."+typeID+".semantic_role", linkType["semantic_role"], linkSemanticRoleTokens...); err != nil {
747 + return err
748 + }
749 + presentation, ok := linkType["presentation"].(map[string]any)
750 + if !ok {
751 + continue
752 + }
753 + path := "data.types.link_types." + typeID + ".presentation"
754 + if _, ok := presentation["label"]; ok {
755 + if err := validateRequiredString(path+".label", presentation["label"]); err != nil {
756 + return err
757 + }
758 + }
759 + if err := optionalEnum(path+".color_slot", presentation["color_slot"], colorSlotTokens...); err != nil {
760 + return err
761 + }
762 + if err := optionalEnum(path+".opacity", presentation["opacity"], opacityTokens...); err != nil {
763 + return err
764 + }
765 + if err := optionalEnum(path+".line_style", presentation["line_style"], "solid", "dashed", "dotted"); err != nil {
766 + return err
767 + }
768 + if err := optionalEnum(path+".width", presentation["width"], widthTokens...); err != nil {
769 + return err
770 + }
771 + if err := optionalEnum(path+".curve", presentation["curve"], "straight", "clockwise", "counter_clockwise", "auto"); err != nil {
772 + return err
773 + }
774 + if err := optionalEnum(path+".arrow", presentation["arrow"], "none", "forward", "reverse", "both", "auto"); err != nil {
775 + return err
776 + }
777 + if err := validateLinkVariablePresentation(path+".variable", presentation["variable"], shape); err != nil {
778 + return err
779 + }
780 + if err := validateHoverPresentation(path+".hover", presentation["hover"], shape.linkColumns); err != nil {
781 + return err
782 + }
783 + if err := validateLinkLayoutPresentation(path+".layout", presentation["layout"]); err != nil {
784 + return err
785 + }
786 + if err := validateModalPresentation(path+".modal", presentation["modal"], shape); err != nil {
787 + return err
788 + }
789 + }
790 + return nil
791 +}
792 +
793 +func validateLinkLayoutPresentation(path string, raw any) error {
794 + if raw == nil {
795 + return nil
796 + }
797 + layout, ok := raw.(map[string]any)
798 + if !ok {
799 + return fmt.Errorf("%s is not an object", path)
800 + }
801 + if err := optionalEnum(path+".strength", layout["strength"], layoutStrengthTokens...); err != nil {
802 + return err
803 + }
804 + return optionalEnum(path+".distance", layout["distance"], layoutDistanceTokens...)
805 +}
806 +
807 +func validateActorLayoutPresentation(path string, raw any) error {
808 + if raw == nil {
809 + return nil
810 + }
811 + layout, ok := raw.(map[string]any)
812 + if !ok {
813 + return fmt.Errorf("%s is not an object", path)
814 + }
815 + return optionalEnum(path+".repulsion", layout["repulsion"], layoutStrengthTokens...)
816 +}
817 +
818 +func validatePortTypePresentation(raw any) error {
819 + if raw == nil {
820 + return nil
821 + }
822 + portTypes, ok := raw.(map[string]any)
823 + if !ok {
824 + return fmt.Errorf("data.types.port_types is not an object")
825 + }
826 + for typeID, rawType := range portTypes {
827 + portType, ok := rawType.(map[string]any)
828 + if !ok {
829 + return fmt.Errorf("data.types.port_types.%s is not an object", typeID)
830 + }
831 + presentation, ok := portType["presentation"].(map[string]any)
832 + if !ok {
833 + continue
834 + }
835 + path := "data.types.port_types." + typeID + ".presentation"
836 + if _, ok := presentation["label"]; ok {
837 + if err := validateRequiredString(path+".label", presentation["label"]); err != nil {
838 + return err
839 + }
840 + }
841 + if err := optionalEnum(path+".color_slot", presentation["color_slot"], colorSlotTokens...); err != nil {
842 + return err
843 + }
844 + if err := optionalEnum(path+".opacity", presentation["opacity"], opacityTokens...); err != nil {
845 + return err
846 + }
847 + }
848 + return nil
849 +}
850 +
851 +func validateTableTypePresentation(raw any, shape topologyShape) error {
852 + if raw == nil {
853 + return nil
854 + }
855 + tableTypes, ok := raw.(map[string]any)
856 + if !ok {
857 + return fmt.Errorf("data.types.table_types is not an object")
858 + }
859 + for typeID, rawType := range tableTypes {
860 + tableType, ok := rawType.(map[string]any)
861 + if !ok {
862 + return fmt.Errorf("data.types.table_types.%s is not an object", typeID)
863 + }
864 + presentation, ok := tableType["presentation"].(map[string]any)
865 + if !ok {
866 + continue
867 + }
868 + path := "data.types.table_types." + typeID + ".presentation"
869 + if _, ok := presentation["label"]; ok {
870 + if err := validateRequiredString(path+".label", presentation["label"]); err != nil {
871 + return err
872 + }
873 + }
874 + if err := optionalEnum(path+".default_visibility", presentation["default_visibility"], "table", "expanded", "hidden", "debug"); err != nil {
875 + return err
876 + }
877 + columns := shape.tableTypes[typeID]
878 + if err := validateModalColumns(path+".columns", presentation["columns"], columns); err != nil {
879 + return err
880 + }
881 + }
882 + return nil
883 +}
884 +
885 +func validateModalPresentation(path string, raw any, shape topologyShape) error {
886 + if raw == nil {
887 + return nil
888 + }
889 + modal, ok := raw.(map[string]any)
890 + if !ok {
891 + return fmt.Errorf("%s is not an object", path)
892 + }
893 + if err := validateModalLabelsPresentation(path+".labels", modal["labels"], shape); err != nil {
894 + return err
895 + }
896 + if err := validateModalMiniTopologyPresentation(path+".mini_topology", modal["mini_topology"], shape); err != nil {
897 + return err
898 + }
899 + rawSections, hasSections := modal["sections"]
900 + if !hasSections || rawSections == nil {
901 + return nil
902 + }
903 + sections, ok := rawSections.([]any)
904 + if !ok {
905 + return fmt.Errorf("%s.sections is not an array", path)
906 + }
907 + seenIDs := make(map[string]struct{}, len(sections))
908 + for i, rawSection := range sections {
909 + section, ok := rawSection.(map[string]any)
910 + if !ok {
911 + return fmt.Errorf("%s.sections[%d] is not an object", path, i)
912 + }
913 + id, _ := section["id"].(string)
914 + if id != "" {
915 + if _, ok := seenIDs[id]; ok {
916 + return fmt.Errorf("%s.sections[%d].id duplicates modal section id %q", path, i, id)
917 + }
918 + seenIDs[id] = struct{}{}
919 + }
920 + if err := validateModalSection(fmt.Sprintf("%s.sections[%d]", path, i), section, shape); err != nil {
921 + return err
922 + }
923 + }
924 + return nil
925 +}
926 +
927 +func validateModalLabelsPresentation(path string, raw any, shape topologyShape) error {
928 + if raw == nil {
929 + return nil
930 + }
931 + labels, ok := raw.(map[string]any)
932 + if !ok {
933 + return fmt.Errorf("%s is not an object", path)
934 + }
935 + table, _ := labels["table"].(string)
936 + if table == "" {
937 + table = "actor_labels"
938 + }
939 + columns := shape.actorTables[table]
940 + if columns == nil {
941 + columns = shape.tableTypes[table]
942 + }
943 + if columns == nil {
944 + return fmt.Errorf("%s.table references unknown actor labels table %q", path, table)
945 + }
946 + for field, defaultColumn := range map[string]string{
947 + "actor_column": "actor",
948 + "key_column": "key",
949 + "value_column": "value",
950 + "source_column": "source",
951 + "kind_column": "kind",
952 + "value_index_column": "value_index",
953 + } {
954 + _, hasField := labels[field]
955 + column, _ := labels[field].(string)
956 + if column == "" {
957 + column = defaultColumn
958 + }
959 + columnType, ok := columns[column]
960 + if !ok {
961 + if (field == "source_column" || field == "kind_column" || field == "value_index_column") && !hasField {
962 + continue
963 + }
964 + return fmt.Errorf("%s.%s references unknown column %q", path, field, column)
965 + }
966 + if field == "actor_column" && columnType != "actor_ref" {
967 + return fmt.Errorf("%s.%s references non-actor_ref column %q (%s)", path, field, column, columnType)
968 + }
969 + }
970 + if err := validateModalLabelIdentification(path+".identification", labels["identification"]); err != nil {
971 + return err
972 + }
973 + return nil
974 +}
975 +
976 +func validateModalLabelIdentification(path string, raw any) error {
977 + if raw == nil {
978 + return nil
979 + }
980 + identification, ok := raw.(map[string]any)
981 + if !ok {
982 + return fmt.Errorf("%s is not an object", path)
983 + }
984 + if rawFields := identification["fields"]; rawFields != nil {
985 + fields, ok := rawFields.([]any)
986 + if !ok {
987 + return fmt.Errorf("%s.fields is not an array", path)
988 + }
989 + for i, rawField := range fields {
990 + field, ok := rawField.(map[string]any)
991 + if !ok {
992 + return fmt.Errorf("%s.fields[%d] is not an object", path, i)
993 + }
994 + key, ok := field["key"].(string)
995 + if !ok || key == "" {
996 + return fmt.Errorf("%s.fields[%d].key is required", path, i)
997 + }
998 + label, ok := field["label"].(string)
999 + if !ok || label == "" {
1000 + return fmt.Errorf("%s.fields[%d].label is required", path, i)
1001 + }
1002 + if rawMaxValues, ok := field["max_values"]; ok {
1003 + maxValues, ok := integerValue(rawMaxValues)
1004 + if !ok || maxValues < 1 {
1005 + return fmt.Errorf("%s.fields[%d].max_values must be a positive integer", path, i)
1006 + }
1007 + }
1008 + }
1009 + }
1010 + return nil
1011 +}
1012 +
1013 +func validateModalMiniTopologyPresentation(path string, raw any, shape topologyShape) error {
1014 + if raw == nil {
1015 + return nil
1016 + }
1017 + mini, ok := raw.(map[string]any)
1018 + if !ok {
1019 + return fmt.Errorf("%s is not an object", path)
1020 + }
1021 + if _, present := mini["depth"]; present {
1022 + depth, ok := integerValue(mini["depth"])
1023 + if !ok {
1024 + return fmt.Errorf("%s.depth is not an integer", path)
1025 + }
1026 + if depth != 1 {
1027 + return fmt.Errorf("%s.depth must be 1", path)
1028 + }
1029 + }
1030 + if err := validateKnownLinkTypeList(path+".include_link_types", mini["include_link_types"], shape); err != nil {
1031 + return err
1032 + }
1033 + return validateKnownLinkTypeList(path+".exclude_link_types", mini["exclude_link_types"], shape)
1034 +}
1035 +
1036 +func validateKnownLinkTypeList(path string, raw any, shape topologyShape) error {
1037 + if raw == nil {
1038 + return nil
1039 + }
1040 + values, ok := raw.([]any)
1041 + if !ok {
1042 + return fmt.Errorf("%s is not an array", path)
1043 + }
1044 + for i, rawValue := range values {
1045 + value, ok := rawValue.(string)
1046 + if !ok || value == "" {
1047 + return fmt.Errorf("%s[%d] is not a non-empty string", path, i)
1048 + }
1049 + if _, ok := shape.linkTypes[value]; !ok {
1050 + return fmt.Errorf("%s[%d] references unknown link type %q", path, i, value)
1051 + }
1052 + }
1053 + return nil
1054 +}
1055 +
1056 +func validateModalSection(path string, section map[string]any, shape topologyShape) error {
1057 + if err := validateRequiredString(path+".id", section["id"]); err != nil {
1058 + return err
1059 + }
1060 + if err := validateRequiredString(path+".label", section["label"]); err != nil {
1061 + return err
1062 + }
1063 + columns, err := validateModalSource(path+".source", section["source"], shape)
1064 + if err != nil {
1065 + return err
1066 + }
1067 + if err := validateModalOwnerFilter(path+".owner_filter", section["owner_filter"], columns); err != nil {
1068 + return err
1069 + }
1070 + if err := validateModalRowFilters(path+".row_filters", section["row_filters"], columns); err != nil {
1071 + return err
1072 + }
1073 + rawColumns, ok := section["columns"].([]any)
1074 + if !ok {
1075 + return fmt.Errorf("%s.columns is not an array", path)
1076 + }
1077 + if len(rawColumns) == 0 {
1078 + return fmt.Errorf("%s.columns must not be empty", path)
1079 + }
1080 + if err := validateModalColumns(path+".columns", section["columns"], columns); err != nil {
1081 + return err
1082 + }
1083 + return validateModalSort(path+".sort", section["sort"], section["columns"])
1084 +}
1085 +
1086 +func validateModalSource(path string, raw any, shape topologyShape) (map[string]string, error) {
1087 + source, ok := raw.(map[string]any)
1088 + if !ok {
1089 + return nil, fmt.Errorf("%s is not an object", path)
1090 + }
1091 + kind, err := requiredEnum(path+".kind", source["kind"], "actors", "links", "evidence", "actor_table", "relationship_table")
1092 + if err != nil {
1093 + return nil, err
1094 + }
1095 + switch kind {
1096 + case "actors":
1097 + return shape.actorColumns, nil
1098 + case "links":
1099 + return shape.linkColumns, nil
1100 + case "evidence":
1101 + evidence, _ := source["evidence"].(string)
1102 + if evidence == "" {
1103 + return nil, fmt.Errorf("%s.evidence is required when kind is evidence", path)
1104 + }
1105 + columns := shape.evidenceTypes[evidence]
1106 + if columns == nil {
1107 + return nil, fmt.Errorf("%s.evidence references unknown evidence type %q", path, evidence)
1108 + }
1109 + return columns, nil
1110 + case "actor_table":
1111 + table, _ := source["table"].(string)
1112 + if table == "" {
1113 + return nil, fmt.Errorf("%s.table is required when kind is actor_table", path)
1114 + }
1115 + columns := shape.actorTables[table]
1116 + if columns == nil {
1117 + columns = shape.tableTypes[table]
1118 + }
1119 + if columns == nil {
1120 + return nil, fmt.Errorf("%s.table references unknown actor table %q", path, table)
1121 + }
1122 + return columns, nil
1123 + case "relationship_table":
1124 + table, _ := source["table"].(string)
1125 + if table == "" {
1126 + return nil, fmt.Errorf("%s.table is required when kind is relationship_table", path)
1127 + }
1128 + columns := shape.relationshipTables[table]
1129 + if columns == nil {
1130 + columns = shape.tableTypes[table]
1131 + }
1132 + if columns == nil {
1133 + return nil, fmt.Errorf("%s.table references unknown relationship table %q", path, table)
1134 + }
1135 + return columns, nil
1136 + default:
1137 + return nil, fmt.Errorf("%s.kind has unsupported value %q", path, kind)
1138 + }
1139 +}
1140 +
1141 +func validateModalOwnerFilter(path string, raw any, columns map[string]string) error {
1142 + if raw == nil {
1143 + return nil
1144 + }
1145 + filter, ok := raw.(map[string]any)
1146 + if !ok {
1147 + return fmt.Errorf("%s is not an object", path)
1148 + }
1149 + mode, err := requiredEnum(path+".mode", filter["mode"], "none", "actor_column", "link_column", "incident_link", "incident_evidence", "selected_link")
1150 + if err != nil {
1151 + return err
1152 + }
1153 + switch mode {
1154 + case "actor_column":
1155 + return validateModalColumnRef(path+".actor_column", filter["actor_column"], columns, "actor_ref", false)
1156 + case "link_column", "selected_link":
1157 + return validateModalColumnRef(path+".link_column", filter["link_column"], columns, "link_ref", false)
1158 + case "incident_link", "incident_evidence":
1159 + if err := validateModalColumnRef(path+".src_actor_column", filter["src_actor_column"], columns, "actor_ref", false); err != nil {
1160 + return err
1161 + }
1162 + return validateModalColumnRef(path+".dst_actor_column", filter["dst_actor_column"], columns, "actor_ref", false)
1163 + default:
1164 + return nil
1165 + }
1166 +}
1167 +
1168 +func validateModalRowFilters(path string, raw any, columns map[string]string) error {
1169 + if raw == nil {
1170 + return nil
1171 + }
1172 + filters, ok := raw.([]any)
1173 + if !ok {
1174 + return fmt.Errorf("%s is not an array", path)
1175 + }
1176 + for i, rawFilter := range filters {
1177 + filter, ok := rawFilter.(map[string]any)
1178 + if !ok {
1179 + return fmt.Errorf("%s[%d] is not an object", path, i)
1180 + }
1181 + filterPath := fmt.Sprintf("%s[%d]", path, i)
1182 + if err := validateModalColumnRef(filterPath+".column", filter["column"], columns, "", false); err != nil {
1183 + return err
1184 + }
1185 + op, err := requiredEnum(filterPath+".op", filter["op"], "eq", "ne", "in", "not_in", "exists", "missing")
1186 + if err != nil {
1187 + return err
1188 + }
1189 + switch op {
1190 + case "eq", "ne":
1191 + if _, ok := filter["value"]; !ok {
1192 + return fmt.Errorf("%s.value is required when op is %s", filterPath, op)
1193 + }
1194 + case "in", "not_in":
1195 + values, ok := filter["values"].([]any)
1196 + if !ok {
1197 + return fmt.Errorf("%s.values is required when op is %s", filterPath, op)
1198 + }
1199 + if len(values) == 0 {
1200 + return fmt.Errorf("%s.values must not be empty when op is %s", filterPath, op)
1201 + }
1202 + }
1203 + }
1204 + return nil
1205 +}
1206 +
1207 +func validateModalColumns(path string, raw any, sourceColumns map[string]string) error {
1208 + if raw == nil {
1209 + return nil
1210 + }
1211 + columns, ok := raw.([]any)
1212 + if !ok {
1213 + return fmt.Errorf("%s is not an array", path)
1214 + }
1215 + seenIDs := make(map[string]struct{}, len(columns))
1216 + for i, rawColumn := range columns {
1217 + column, ok := rawColumn.(map[string]any)
1218 + if !ok {
1219 + return fmt.Errorf("%s[%d] is not an object", path, i)
1220 + }
1221 + columnPath := fmt.Sprintf("%s[%d]", path, i)
1222 + if err := validateRequiredString(columnPath+".id", column["id"]); err != nil {
1223 + return err
1224 + }
1225 + id := column["id"].(string)
1226 + if _, ok := seenIDs[id]; ok {
1227 + return fmt.Errorf("%s[%d].id duplicates modal column id %q", path, i, id)
1228 + }
1229 + seenIDs[id] = struct{}{}
1230 + if err := validateRequiredString(columnPath+".label", column["label"]); err != nil {
1231 + return err
1232 + }
1233 + if err := optionalEnum(columnPath+".cell", column["cell"], "text", "number", "badge", "actor_link", "timestamp", "duration", "endpoint", "array_count", "debug_json"); err != nil {
1234 + return err
1235 + }
1236 + if err := optionalEnum(columnPath+".visibility", column["visibility"], "table", "expanded", "hidden", "debug"); err != nil {
1237 + return err
1238 + }
1239 + if err := optionalEnum(columnPath+".align", column["align"], "left", "center", "right"); err != nil {
1240 + return err
1241 + }
1242 + if err := validateModalProjection(columnPath+".projection", column["projection"], sourceColumns); err != nil {
1243 + return err
1244 + }
1245 + if err := validateModalBadgeMap(columnPath+".badge_map", column["badge_map"]); err != nil {
1246 + return err
1247 + }
1248 + }
1249 + return nil
1250 +}
1251 +
1252 +func validateModalProjection(path string, raw any, columns map[string]string) error {
1253 + projection, ok := raw.(map[string]any)
1254 + if !ok {
1255 + return fmt.Errorf("%s is not an object", path)
1256 + }
1257 + kind, err := requiredEnum(path+".kind", projection["kind"],
1258 + "direct", "actor_ref_label", "opposite_actor", "formatted_endpoint", "label_lookup",
1259 + "json_path", "const", "coalesce", "selected_side_endpoint")
1260 + if err != nil {
1261 + return err
1262 + }
1263 + switch kind {
1264 + case "direct":
1265 + return validateModalColumnRef(path+".column", projection["column"], columns, "", false)
1266 + case "actor_ref_label":
1267 + return validateModalColumnRef(path+".actor_column", projection["actor_column"], columns, "actor_ref", false)
1268 + case "opposite_actor":
1269 + if err := validateModalColumnRef(path+".src_actor_column", projection["src_actor_column"], columns, "actor_ref", false); err != nil {
1270 + return err
1271 + }
1272 + return validateModalColumnRef(path+".dst_actor_column", projection["dst_actor_column"], columns, "actor_ref", false)
1273 + case "formatted_endpoint":
1274 + if stringValue(projection["ip_column"]) == "" && stringValue(projection["port_column"]) == "" {
1275 + return fmt.Errorf("%s requires ip_column or port_column when kind is formatted_endpoint", path)
1276 + }
1277 + if err := validateModalColumnRef(path+".ip_column", projection["ip_column"], columns, "", true); err != nil {
1278 + return err
1279 + }
1280 + if err := validateModalColumnRef(path+".port_column", projection["port_column"], columns, "", true); err != nil {
1281 + return err
1282 + }
1283 + return validateModalColumnRef(path+".protocol_column", projection["protocol_column"], columns, "", true)
1284 + case "label_lookup":
1285 + if err := validateModalColumnRef(path+".actor_column", projection["actor_column"], columns, "actor_ref", true); err != nil {
1286 + return err
1287 + }
1288 + labelKey, _ := projection["label_key"].(string)
1289 + if labelKey == "" {
1290 + return fmt.Errorf("%s.label_key is required when kind is label_lookup", path)
1291 + }
1292 + return nil
1293 + case "json_path":
1294 + if err := validateModalColumnRef(path+".column", projection["column"], columns, "json", false); err != nil {
1295 + return err
1296 + }
1297 + pathValue, _ := projection["path"].(string)
1298 + if pathValue == "" {
1299 + return fmt.Errorf("%s.path is required when kind is json_path", path)
1300 + }
1301 + return nil
1302 + case "const":
1303 + if _, ok := projection["value"]; !ok {
1304 + return fmt.Errorf("%s.value is required when kind is const", path)
1305 + }
1306 + return nil
1307 + case "coalesce":
1308 + rawColumns, ok := projection["columns"].([]any)
1309 + if !ok || len(rawColumns) == 0 {
1310 + return fmt.Errorf("%s.columns is required when kind is coalesce", path)
1311 + }
1312 + for i, rawColumn := range rawColumns {
1313 + if err := validateModalColumnRef(fmt.Sprintf("%s.columns[%d]", path, i), rawColumn, columns, "", false); err != nil {
1314 + return err
1315 + }
1316 + }
1317 + return nil
1318 + case "selected_side_endpoint":
1319 + if err := validateModalColumnRef(path+".src_actor_column", projection["src_actor_column"], columns, "actor_ref", false); err != nil {
1320 + return err
1321 + }
1322 + if err := validateModalColumnRef(path+".dst_actor_column", projection["dst_actor_column"], columns, "actor_ref", false); err != nil {
1323 + return err
1324 + }
1325 + if stringValue(projection["local_ip_column"]) == "" && stringValue(projection["local_port_column"]) == "" {
1326 + return fmt.Errorf("%s requires local_ip_column or local_port_column when kind is selected_side_endpoint", path)
1327 + }
1328 + if stringValue(projection["remote_ip_column"]) == "" && stringValue(projection["remote_port_column"]) == "" {
1329 + return fmt.Errorf("%s requires remote_ip_column or remote_port_column when kind is selected_side_endpoint", path)
1330 + }
1331 + for _, field := range []string{"local_ip_column", "local_port_column", "remote_ip_column", "remote_port_column", "protocol_column"} {
1332 + if err := validateModalColumnRef(path+"."+field, projection[field], columns, "", true); err != nil {
1333 + return err
1334 + }
1335 + }
1336 + return nil
1337 + default:
1338 + return fmt.Errorf("%s.kind has unsupported value %q", path, kind)
1339 + }
1340 +}
1341 +
1342 +func validateModalBadgeMap(path string, raw any) error {
1343 + if raw == nil {
1344 + return nil
1345 + }
1346 + badgeMap, ok := raw.(map[string]any)
1347 + if !ok {
1348 + return fmt.Errorf("%s is not an object", path)
1349 + }
1350 + for key, rawBadge := range badgeMap {
1351 + badge, ok := rawBadge.(map[string]any)
1352 + if !ok {
1353 + return fmt.Errorf("%s.%s is not an object", path, key)
1354 + }
1355 + if err := optionalEnum(path+"."+key+".color_slot", badge["color_slot"], colorSlotTokens...); err != nil {
1356 + return err
1357 + }
1358 + if err := optionalEnum(path+"."+key+".opacity", badge["opacity"], opacityTokens...); err != nil {
1359 + return err
1360 + }
1361 + }
1362 + return nil
1363 +}
1364 +
1365 +func validateModalSort(path string, raw any, rawColumns any) error {
1366 + if raw == nil {
1367 + return nil
1368 + }
1369 + sortSpec, ok := raw.(map[string]any)
1370 + if !ok {
1371 + return fmt.Errorf("%s is not an object", path)
1372 + }
1373 + column, _ := sortSpec["column"].(string)
1374 + if column == "" {
1375 + return fmt.Errorf("%s.column is empty", path)
1376 + }
1377 + if err := optionalEnum(path+".direction", sortSpec["direction"], "asc", "desc"); err != nil {
1378 + return err
1379 + }
1380 + columns, _ := rawColumns.([]any)
1381 + for _, rawColumn := range columns {
1382 + columnSpec, ok := rawColumn.(map[string]any)
1383 + if !ok {
1384 + continue
1385 + }
1386 + id, _ := columnSpec["id"].(string)
1387 + if id == column {
1388 + return nil
1389 + }
1390 + }
1391 + return fmt.Errorf("%s.column references unknown modal column %q", path, column)
1392 +}
1393 +
1394 +func validateModalColumnRef(path string, raw any, columns map[string]string, expectedType string, optional bool) error {
1395 + column, _ := raw.(string)
1396 + if column == "" {
1397 + if optional {
1398 + return nil
1399 + }
1400 + return fmt.Errorf("%s is required", path)
1401 + }
1402 + columnType, ok := columns[column]
1403 + if !ok {
1404 + return fmt.Errorf("%s references unknown source column %q", path, column)
1405 + }
1406 + if expectedType != "" && columnType != expectedType {
1407 + return fmt.Errorf("%s references non-%s source column %q (%s)", path, expectedType, column, columnType)
1408 + }
1409 + return nil
1410 +}
1411 +
1412 +func validateGraphPresentation(raw any, shape topologyShape) error {
1413 + if raw == nil {
1414 + return nil
1415 + }
1416 + presentation, ok := raw.(map[string]any)
1417 + if !ok {
1418 + return fmt.Errorf("data.presentation is not an object")
1419 + }
1420 + if err := validateSelectionPresentation(presentation["selection"], shape); err != nil {
1421 + return err
1422 + }
1423 + if err := validateLegendPresentation(presentation["legend"], shape); err != nil {
1424 + return err
1425 + }
1426 + return nil
1427 +}
1428 +
1429 +func validateBorderPresentation(path string, raw any) error {
1430 + if raw == nil {
1431 + return nil
1432 + }
1433 + border, ok := raw.(map[string]any)
1434 + if !ok {
1435 + return fmt.Errorf("%s is not an object", path)
1436 + }
1437 + if err := optionalEnum(path+".color_slot", border["color_slot"], colorSlotTokens...); err != nil {
1438 + return err
1439 + }
1440 + return optionalEnum(path+".style", border["style"], "solid", "dashed", "dotted")
1441 +}
1442 +
1443 +func validateAnnotationPresentation(path string, raw any) error {
1444 + if raw == nil {
1445 + return nil
1446 + }
1447 + annotation, ok := raw.(map[string]any)
1448 + if !ok {
1449 + return fmt.Errorf("%s is not an object", path)
1450 + }
1451 + if err := optionalEnum(path+".color_slot", annotation["color_slot"], colorSlotTokens...); err != nil {
1452 + return err
1453 + }
1454 + return optionalEnum(path+".style", annotation["style"], "ring", "dot", "none")
1455 +}
1456 +
1457 +func validateActorSizePresentation(path string, raw any, actorColumns map[string]string) error {
1458 + if raw == nil {
1459 + return nil
1460 + }
1461 + size, ok := raw.(map[string]any)
1462 + if !ok {
1463 + return fmt.Errorf("%s is not an object", path)
1464 + }
1465 + mode, err := requiredEnum(path+".mode", size["mode"], "fixed", "link_count", "metric")
1466 + if err != nil {
1467 + return err
1468 + }
1469 + if mode == "metric" {
1470 + column, _ := size["metric_column"].(string)
1471 + if column == "" {
1472 + return fmt.Errorf("%s.metric_column is required when mode is metric", path)
1473 + }
1474 + columnType, ok := actorColumns[column]
1475 + if !ok {
1476 + return fmt.Errorf("%s.metric_column references unknown actor column %q", path, column)
1477 + }
1478 + if !isNumericColumnType(columnType) {
1479 + return fmt.Errorf("%s.metric_column references non-numeric actor column %q (%s)", path, column, columnType)
1480 + }
1481 + }
1482 + return optionalEnum(path+".scale", size["scale"], actorSizeScaleTokens...)
1483 +}
1484 +
1485 +func validateActorSearchPolicy(path string, raw any, actorColumns map[string]string) error {
1486 + if raw == nil {
1487 + return nil
1488 + }
1489 + search, ok := raw.(map[string]any)
1490 + if !ok {
1491 + return fmt.Errorf("%s is not an object", path)
1492 + }
1493 + if enabled, ok := search["enabled"]; ok {
1494 + if _, ok := enabled.(bool); !ok {
1495 + return fmt.Errorf("%s.enabled is not a boolean", path)
1496 + }
1497 + }
1498 + for _, field := range []string{"columns", "label_keys"} {
1499 + rawList, ok := search[field]
1500 + if !ok {
1501 + continue
1502 + }
1503 + values, ok := rawList.([]any)
1504 + if !ok {
1505 + return fmt.Errorf("%s.%s is not an array", path, field)
1506 + }
1507 + seen := make(map[string]struct{}, len(values))
1508 + for i, rawValue := range values {
1509 + value, ok := rawValue.(string)
1510 + if !ok || value == "" {
1511 + return fmt.Errorf("%s.%s[%d] is not a non-empty string", path, field, i)
1512 + }
1513 + if _, ok := seen[value]; ok {
1514 + return fmt.Errorf("%s.%s[%d] duplicates %q", path, field, i, value)
1515 + }
1516 + seen[value] = struct{}{}
1517 + if field != "columns" {
1518 + continue
1519 + }
1520 + columnType, ok := actorColumns[value]
1521 + if !ok {
1522 + return fmt.Errorf("%s.columns[%d] references unknown actor column %q", path, i, value)
1523 + }
1524 + if !isDisplayColumnType(columnType) {
1525 + return fmt.Errorf("%s.columns[%d] references non-display actor column %q (%s)", path, i, value, columnType)
1526 + }
1527 + }
1528 + }
1529 + return nil
1530 +}
1531 +
1532 +func validateLabelPolicy(path string, raw any, actorColumns map[string]string) error {
1533 + if raw == nil {
1534 + return nil
1535 + }
1536 + policy, ok := raw.(map[string]any)
1537 + if !ok {
1538 + return fmt.Errorf("%s is not an object", path)
1539 + }
1540 + if err := optionalEnum(path+".fallback", policy["fallback"], "type_label", "row_index", "none"); err != nil {
1541 + return err
1542 + }
1543 + if err := optionalEnum(path+".array", policy["array"], "reject", "first", "summarize"); err != nil {
1544 + return err
1545 + }
1546 + columns, ok := policy["columns"].([]any)
1547 + if !ok {
1548 + return nil
1549 + }
1550 + for i, rawColumn := range columns {
1551 + column, ok := rawColumn.(string)
1552 + if !ok || column == "" {
1553 + return fmt.Errorf("%s.columns[%d] is not a non-empty string", path, i)
1554 + }
1555 + columnType, ok := actorColumns[column]
1556 + if !ok {
1557 + return fmt.Errorf("%s.columns[%d] references unknown actor column %q", path, i, column)
1558 + }
1559 + if !isDisplayColumnType(columnType) {
1560 + return fmt.Errorf("%s.columns[%d] references non-display actor column %q (%s)", path, i, column, columnType)
1561 + }
1562 + }
1563 + return nil
1564 +}
1565 +
1566 +func validateActorPortsPresentation(path string, raw any, shape topologyShape) error {
1567 + if raw == nil {
1568 + return nil
1569 + }
1570 + ports, ok := raw.(map[string]any)
1571 + if !ok {
1572 + return fmt.Errorf("%s is not an object", path)
1573 + }
1574 + showBullets, _ := ports["show_bullets"].(bool)
1575 + sources, ok := ports["sources"].([]any)
1576 + if showBullets && (!ok || len(sources) == 0) {
1577 + return fmt.Errorf("%s.sources is required when show_bullets is true", path)
1578 + }
1579 + if !ok {
1580 + return nil
1581 + }
1582 + for i, rawSource := range sources {
1583 + source, ok := rawSource.(map[string]any)
1584 + if !ok {
1585 + return fmt.Errorf("%s.sources[%d] is not an object", path, i)
1586 + }
1587 + if err := validatePortSourcePresentation(fmt.Sprintf("%s.sources[%d]", path, i), source, shape); err != nil {
1588 + return err
1589 + }
1590 + }
1591 + return nil
1592 +}
1593 +
1594 +func validatePortSourcePresentation(path string, source map[string]any, shape topologyShape) error {
1595 + sourceKind, err := requiredEnum(path+".source", source["source"], "links", "evidence", "actor_table")
1596 + if err != nil {
1597 + return err
1598 + }
1599 + defaultType, _ := source["default_type"].(string)
1600 + if defaultType != "" {
1601 + if _, ok := shape.portTypes[defaultType]; !ok {
1602 + return fmt.Errorf("%s.default_type references unknown port type %q", path, defaultType)
1603 + }
1604 + }
1605 +
1606 + var columns map[string]string
1607 + switch sourceKind {
1608 + case "links":
1609 + columns = shape.linkColumns
1610 + case "evidence":
1611 + evidence, _ := source["evidence"].(string)
1612 + if evidence == "" {
1613 + return fmt.Errorf("%s.evidence is required when source is evidence", path)
1614 + }
1615 + columns = shape.evidenceTypes[evidence]
1616 + if columns == nil {
1617 + return fmt.Errorf("%s.evidence references unknown evidence type %q", path, evidence)
1618 + }
1619 + case "actor_table":
1620 + table, _ := source["table"].(string)
1621 + if table == "" {
1622 + return fmt.Errorf("%s.table is required when source is actor_table", path)
1623 + }
1624 + columns = shape.actorTables[table]
1625 + if columns == nil {
1626 + columns = shape.tableTypes[table]
1627 + }
1628 + if columns == nil {
1629 + return fmt.Errorf("%s.table references unknown actor table %q", path, table)
1630 + }
1631 + }
1632 +
1633 + for _, field := range []string{"actor_column", "name_column"} {
1634 + column, _ := source[field].(string)
1635 + if column == "" {
1636 + return fmt.Errorf("%s.%s is required", path, field)
1637 + }
1638 + if _, ok := columns[column]; !ok {
1639 + return fmt.Errorf("%s.%s references unknown source column %q", path, field, column)
1640 + }
1641 + }
1642 + actorColumn, _ := source["actor_column"].(string)
1643 + if columnType := columns[actorColumn]; columnType != "actor_ref" {
1644 + return fmt.Errorf("%s.actor_column references non-actor_ref source column %q (%s)", path, actorColumn, columnType)
1645 + }
1646 + nameColumn, _ := source["name_column"].(string)
1647 + if columnType := columns[nameColumn]; !isDisplayColumnType(columnType) {
1648 + return fmt.Errorf("%s.name_column references non-display source column %q (%s)", path, nameColumn, columnType)
1649 + }
1650 + valueColumn, _ := source["value_column"].(string)
1651 + if valueColumn != "" {
1652 + columnType, ok := columns[valueColumn]
1653 + if !ok {
1654 + return fmt.Errorf("%s.value_column references unknown source column %q", path, valueColumn)
1655 + }
1656 + if !isNumericColumnType(columnType) {
1657 + return fmt.Errorf("%s.value_column references non-numeric source column %q (%s)", path, valueColumn, columnType)
1658 + }
1659 + }
1660 + for _, field := range []string{"type_column", "status_column", "mode_column", "role_column", "sources_column"} {
1661 + column, _ := source[field].(string)
1662 + if column == "" {
1663 + continue
1664 + }
1665 + if _, ok := columns[column]; !ok {
1666 + if sourceKind == "actor_table" {
1667 + continue
1668 + }
1669 + return fmt.Errorf("%s.%s references unknown source column %q", path, field, column)
1670 + }
1671 + }
1672 + return nil
1673 +}
1674 +
1675 +func validateLinkVariablePresentation(path string, raw any, shape topologyShape) error {
1676 + if raw == nil {
1677 + return nil
1678 + }
1679 + variable, ok := raw.(map[string]any)
1680 + if !ok {
1681 + return fmt.Errorf("%s is not an object", path)
1682 + }
1683 + channel, err := requiredEnum(path+".channel", variable["channel"], "width", "opacity")
1684 + if err != nil {
1685 + return err
1686 + }
1687 + scaleKey, _ := variable["scale_key"].(string)
1688 + if scaleKey == "" {
1689 + return fmt.Errorf("%s.scale_key is required", path)
1690 + }
1691 + if _, ok := shape.scaleKeys[scaleKey]; !ok {
1692 + return fmt.Errorf("%s.scale_key references unknown presentation scale key %q", path, scaleKey)
1693 + }
1694 + valueColumn, _ := variable["value_column"].(string)
1695 + if valueColumn == "" {
1696 + return fmt.Errorf("%s.value_column is required", path)
1697 + }
1698 + columnType, ok := shape.linkColumns[valueColumn]
1699 + if !ok {
1700 + return fmt.Errorf("%s.value_column references unknown link column %q", path, valueColumn)
1701 + }
1702 + if !isNumericColumnType(columnType) {
1703 + return fmt.Errorf("%s.value_column references non-numeric link column %q (%s)", path, valueColumn, columnType)
1704 + }
1705 + allowed := widthTokens
1706 + if channel == "opacity" {
1707 + allowed = opacityTokens
1708 + }
1709 + if err := optionalEnum(path+".min", variable["min"], allowed...); err != nil {
1710 + return err
1711 + }
1712 + return optionalEnum(path+".max", variable["max"], allowed...)
1713 +}
1714 +
1715 +func validateHoverPresentation(path string, raw any, columns map[string]string) error {
1716 + if raw == nil {
1717 + return nil
1718 + }
1719 + hover, ok := raw.(map[string]any)
1720 + if !ok {
1721 + return fmt.Errorf("%s is not an object", path)
1722 + }
1723 + fields, ok := hover["fields"].([]any)
1724 + if !ok {
1725 + return nil
1726 + }
1727 + for i, rawField := range fields {
1728 + field, ok := rawField.(map[string]any)
1729 + if !ok {
1730 + return fmt.Errorf("%s.fields[%d] is not an object", path, i)
1731 + }
1732 + key, _ := field["key"].(string)
1733 + if key == "" {
1734 + return fmt.Errorf("%s.fields[%d].key is empty", path, i)
1735 + }
1736 + columnType, ok := columns[key]
1737 + if !ok {
1738 + return fmt.Errorf("%s.fields[%d].key references unknown column %q", path, i, key)
1739 + }
1740 + if !isDisplayColumnType(columnType) {
1741 + return fmt.Errorf("%s.fields[%d].key references non-display column %q (%s)", path, i, key, columnType)
1742 + }
1743 + }
1744 + return nil
1745 +}
1746 +
1747 +func validateCorrelation(raw any, shape topologyShape, ctx validationContext) error {
1748 + if raw == nil {
1749 + return nil
1750 + }
1751 + correlation, ok := raw.(map[string]any)
1752 + if !ok {
1753 + return fmt.Errorf("data.correlation is not an object")
1754 + }
1755 + rules, ok := correlation["rules"].(map[string]any)
1756 + if !ok || len(rules) == 0 {
1757 + return fmt.Errorf("data.correlation.rules is empty")
1758 + }
1759 +
1760 + ruleIDs := make(map[string]struct{}, len(rules))
1761 + requiredColumnsByRule := make(map[string]map[string]struct{}, len(rules))
1762 + for ruleID, rawRule := range rules {
1763 + ruleIDs[ruleID] = struct{}{}
1764 + requiredColumns := make(map[string]struct{})
1765 + requiredColumnsByRule[ruleID] = requiredColumns
1766 + rule, ok := rawRule.(map[string]any)
1767 + if !ok {
1768 + return fmt.Errorf("data.correlation.rules.%s is not an object", ruleID)
1769 + }
1770 + if _, err := requiredEnum("data.correlation.rules."+ruleID+".action", rule["action"], "absorb", "link"); err != nil {
1771 + return err
1772 + }
1773 + if err := optionalEnum("data.correlation.rules."+ruleID+".class", rule["class"], "resolve_loose_side", "replace_actor", "merge_enrich_actor"); err != nil {
1774 + return err
1775 + }
1776 + if _, ok := integerValue(rule["priority"]); !ok {
1777 + return fmt.Errorf("data.correlation.rules.%s.priority is not an integer", ruleID)
1778 + }
1779 + outputLinkType, ok := rule["output_link_type"].(string)
1780 + if !ok || outputLinkType == "" {
1781 + return fmt.Errorf("data.correlation.rules.%s.output_link_type is empty", ruleID)
1782 + }
1783 + if _, ok := shape.linkTypes[outputLinkType]; !ok {
1784 + return fmt.Errorf("data.correlation.rules.%s.output_link_type references unknown link type %q", ruleID, outputLinkType)
1785 + }
1786 + if err := validateIDArrayRefs("data.correlation.rules."+ruleID+".point_actor_types", rule["point_actor_types"], shape.actorTypes, "actor type"); err != nil {
1787 + return err
1788 + }
1789 + if err := validateOptionalIDArrayRefs("data.correlation.rules."+ruleID+".claim_actor_types", rule["claim_actor_types"], shape.actorTypes, "actor type"); err != nil {
1790 + return err
1791 + }
1792 + if err := validateOptionalIDArrayRefs("data.correlation.rules."+ruleID+".correlation_link_types", rule["correlation_link_types"], shape.linkTypes, "link type"); err != nil {
1793 + return err
1794 + }
1795 + key, ok := rule["key"].([]any)
1796 + if !ok || len(key) == 0 {
1797 + return fmt.Errorf("data.correlation.rules.%s.key is empty", ruleID)
1798 + }
1799 + for i, rawPart := range key {
1800 + part, ok := rawPart.(map[string]any)
1801 + if !ok {
1802 + return fmt.Errorf("data.correlation.rules.%s.key[%d] is not an object", ruleID, i)
1803 + }
1804 + if column, ok := part["column"].(string); ok && column != "" {
1805 + requiredColumns[column] = struct{}{}
1806 + continue
1807 + }
1808 + if literal, ok := part["literal"].(string); ok && literal != "" {
1809 + continue
1810 + }
1811 + return fmt.Errorf("data.correlation.rules.%s.key[%d] must define column or literal", ruleID, i)
1812 + }
1813 + }
1814 +
1815 + if err := validateCorrelationTable("data.correlation.points", correlation["points"], ctx, ruleIDs, requiredColumnsByRule); err != nil {
1816 + return err
1817 + }
1818 + return validateCorrelationTable("data.correlation.claims", correlation["claims"], ctx, ruleIDs, requiredColumnsByRule)
1819 +}
1820 +
1821 +func validateCorrelationTable(path string, raw any, ctx validationContext, ruleIDs map[string]struct{}, requiredColumnsByRule map[string]map[string]struct{}) error {
1822 + if raw == nil {
1823 + return nil
1824 + }
1825 + if _, err := validateCompactTable(path, raw, ctx); err != nil {
1826 + return err
1827 + }
1828 + columns, err := columnTypesFromTable(raw, path)
1829 + if err != nil {
1830 + return err
1831 + }
1832 + if columnType := columns["actor"]; columnType != "actor_ref" {
1833 + return fmt.Errorf("%s.actor must be actor_ref, got %q", path, columnType)
1834 + }
1835 + ruleColumnType := columns["rule"]
1836 + if ruleColumnType != "string" && ruleColumnType != "string_ref" {
1837 + return fmt.Errorf("%s.rule must be string or string_ref, got %q", path, ruleColumnType)
1838 + }
1839 + referencedRules, err := collectStringColumnValuesInSet(path, raw, "rule", ruleIDs, "correlation rule", ctx.dictionaries)
1840 + if err != nil {
1841 + return err
1842 + }
1843 + for ruleID := range referencedRules {
1844 + for column := range requiredColumnsByRule[ruleID] {
1845 + if _, ok := columns[column]; !ok {
1846 + return fmt.Errorf("%s is missing correlation key column %q for rule %q", path, column, ruleID)
1847 + }
1848 + }
1849 + }
1850 + return nil
1851 +}
1852 +
1853 +func validateIDArrayRefs(path string, raw any, known map[string]struct{}, kind string) error {
1854 + values, ok := raw.([]any)
1855 + if !ok || len(values) == 0 {
1856 + return fmt.Errorf("%s is empty", path)
1857 + }
1858 + for i, rawValue := range values {
1859 + value, ok := rawValue.(string)
1860 + if !ok || value == "" {
1861 + return fmt.Errorf("%s[%d] is not a non-empty string", path, i)
1862 + }
1863 + if _, ok := known[value]; !ok {
1864 + return fmt.Errorf("%s[%d] references unknown %s %q", path, i, kind, value)
1865 + }
1866 + }
1867 + return nil
1868 +}
1869 +
1870 +func validateOptionalIDArrayRefs(path string, raw any, known map[string]struct{}, kind string) error {
1871 + if raw == nil {
1872 + return nil
1873 + }
1874 + values, ok := raw.([]any)
1875 + if !ok {
1876 + return fmt.Errorf("%s is not an array", path)
1877 + }
1878 + for i, rawValue := range values {
1879 + value, ok := rawValue.(string)
1880 + if !ok || value == "" {
1881 + return fmt.Errorf("%s[%d] is not a non-empty string", path, i)
1882 + }
1883 + if _, ok := known[value]; !ok {
1884 + return fmt.Errorf("%s[%d] references unknown %s %q", path, i, kind, value)
1885 + }
1886 + }
1887 + return nil
1888 +}
1889 +
1890 +func validateSelectionPresentation(raw any, shape topologyShape) error {
1891 + if raw == nil {
1892 + return nil
1893 + }
1894 + selection, ok := raw.(map[string]any)
1895 + if !ok {
1896 + return fmt.Errorf("data.presentation.selection is not an object")
1897 + }
1898 + rawActorClick := selection["actor_click"]
1899 + if rawActorClick == nil {
1900 + return nil
1901 + }
1902 + actorClick, ok := rawActorClick.(map[string]any)
1903 + if !ok {
1904 + return fmt.Errorf("data.presentation.selection.actor_click is not an object")
1905 + }
1906 + mode, err := requiredEnum("data.presentation.selection.actor_click.mode", actorClick["mode"], "none", "highlight_connections", "highlight_path")
1907 + if err != nil {
1908 + return err
1909 + }
1910 + if mode != "highlight_path" {
1911 + return nil
1912 + }
1913 + table, _ := actorClick["path_table"].(string)
1914 + if table == "" {
1915 + return fmt.Errorf("data.presentation.selection.actor_click.path_table is required when mode is highlight_path")
1916 + }
1917 + columns := shape.actorTables[table]
1918 + if columns == nil {
1919 + columns = shape.tableTypes[table]
1920 + if columns != nil && shape.tableTypeOwners[table] != "actor" {
1921 + return fmt.Errorf("data.presentation.selection.actor_click.path_table references non-actor table %q", table)
1922 + }
1923 + }
1924 + if columns == nil {
1925 + return fmt.Errorf("data.presentation.selection.actor_click.path_table references unknown actor table %q", table)
1926 + }
1927 + for _, field := range []string{"path_actor_column", "path_order_column"} {
1928 + column, _ := actorClick[field].(string)
1929 + if column == "" {
1930 + return fmt.Errorf("data.presentation.selection.actor_click.%s is required when mode is highlight_path", field)
1931 + }
1932 + if _, ok := columns[column]; !ok {
1933 + return fmt.Errorf("data.presentation.selection.actor_click.%s references unknown path table column %q", field, column)
1934 + }
1935 + }
1936 + actorColumn, _ := actorClick["path_actor_column"].(string)
1937 + if columnType := columns[actorColumn]; columnType != "actor_ref" {
1938 + return fmt.Errorf("data.presentation.selection.actor_click.path_actor_column references non-actor_ref path table column %q (%s)", actorColumn, columnType)
1939 + }
1940 + ownerColumn, _ := actorClick["path_owner_column"].(string)
1941 + if ownerColumn != "" {
1942 + columnType, ok := columns[ownerColumn]
1943 + if !ok {
1944 + return fmt.Errorf("data.presentation.selection.actor_click.path_owner_column references unknown path table column %q", ownerColumn)
1945 + }
1946 + if columnType != "actor_ref" {
1947 + return fmt.Errorf("data.presentation.selection.actor_click.path_owner_column references non-actor_ref path table column %q (%s)", ownerColumn, columnType)
1948 + }
1949 + }
1950 + orderColumn, _ := actorClick["path_order_column"].(string)
1951 + if columnType := columns[orderColumn]; !isNumericColumnType(columnType) {
1952 + return fmt.Errorf("data.presentation.selection.actor_click.path_order_column references non-numeric path table column %q (%s)", orderColumn, columnType)
1953 + }
1954 + return nil
1955 +}
1956 +
1957 +func validateLegendPresentation(raw any, shape topologyShape) error {
1958 + if raw == nil {
1959 + return nil
1960 + }
1961 + legend, ok := raw.(map[string]any)
1962 + if !ok {
1963 + return fmt.Errorf("data.presentation.legend is not an object")
1964 + }
1965 + if err := validateLegendEntries("data.presentation.legend.actors", legend["actors"], shape.actorTypes); err != nil {
1966 + return err
1967 + }
1968 + if err := validateLegendEntries("data.presentation.legend.links", legend["links"], shape.linkTypes); err != nil {
1969 + return err
1970 + }
1971 + return validateLegendEntries("data.presentation.legend.ports", legend["ports"], shape.portTypes)
1972 +}
1973 +
1974 +func validateLegendEntries(path string, raw any, known map[string]struct{}) error {
1975 + if raw == nil {
1976 + return nil
1977 + }
1978 + entries, ok := raw.([]any)
1979 + if !ok {
1980 + return fmt.Errorf("%s is not an array", path)
1981 + }
1982 + for i, rawEntry := range entries {
1983 + entry, ok := rawEntry.(map[string]any)
1984 + if !ok {
1985 + return fmt.Errorf("%s[%d] is not an object", path, i)
1986 + }
1987 + typeID, _ := entry["type"].(string)
1988 + if typeID == "" {
1989 + return fmt.Errorf("%s[%d].type is empty", path, i)
1990 + }
1991 + if _, ok := known[typeID]; !ok {
1992 + return fmt.Errorf("%s[%d].type references unknown type %q", path, i, typeID)
1993 + }
1994 + }
1995 + return nil
1996 +}
1997 +
1998 +func validateReference(path string, row int, value any, maxRows int, name string) error {
1999 + index, ok := integerValue(value)
2000 + if !ok {
2001 + return fmt.Errorf("%s[%d] is not an integer %s reference", path, row, name)
2002 + }
2003 + if index < 0 || index >= maxRows {
2004 + return fmt.Errorf("%s[%d] %s reference out of bounds: %d", path, row, name, index)
2005 + }
2006 + return nil
2007 +}
2008 +
2009 +func columnTypesFromTable(raw any, path string) (map[string]string, error) {
2010 + table, ok := raw.(map[string]any)
2011 + if !ok {
2012 + return nil, fmt.Errorf("%s is not an object", path)
2013 + }
2014 + return columnTypesFromRawColumns(table["columns"], path+".columns")
2015 +}
2016 +
2017 +func columnTypesFromRawColumns(raw any, path string) (map[string]string, error) {
2018 + columns, ok := raw.([]any)
2019 + if !ok {
2020 + return nil, fmt.Errorf("%s is not an array", path)
2021 + }
2022 + types := make(map[string]string, len(columns))
2023 + for i, rawColumn := range columns {
2024 + column, ok := rawColumn.(map[string]any)
2025 + if !ok {
2026 + return nil, fmt.Errorf("%s[%d] is not an object", path, i)
2027 + }
2028 + id, _ := column["id"].(string)
2029 + columnType, _ := column["type"].(string)
2030 + if id == "" {
2031 + return nil, fmt.Errorf("%s[%d].id is empty", path, i)
2032 + }
2033 + if columnType == "" {
2034 + return nil, fmt.Errorf("%s[%d].type is empty", path, i)
2035 + }
2036 + types[id] = columnType
2037 + }
2038 + return types, nil
2039 +}
2040 +
2041 +func objectKeySet(raw any, path string) (map[string]struct{}, error) {
2042 + obj, ok := raw.(map[string]any)
2043 + if !ok {
2044 + return nil, fmt.Errorf("%s is not an object", path)
2045 + }
2046 + keys := make(map[string]struct{}, len(obj))
2047 + for key := range obj {
2048 + keys[key] = struct{}{}
2049 + }
2050 + return keys, nil
2051 +}
2052 +
2053 +func optionalObjectKeySet(raw any, path string) (map[string]struct{}, error) {
2054 + if raw == nil {
2055 + return map[string]struct{}{}, nil
2056 + }
2057 + return objectKeySet(raw, path)
2058 +}
2059 +
2060 +func columnTypesByRegistryObject(raw any, path string) (map[string]map[string]string, error) {
2061 + out := make(map[string]map[string]string)
2062 + if raw == nil {
2063 + return out, nil
2064 + }
2065 + obj, ok := raw.(map[string]any)
2066 + if !ok {
2067 + return nil, fmt.Errorf("%s is not an object", path)
2068 + }
2069 + for id, rawType := range obj {
2070 + typeObj, ok := rawType.(map[string]any)
2071 + if !ok {
2072 + return nil, fmt.Errorf("%s.%s is not an object", path, id)
2073 + }
2074 + columns, err := columnTypesFromRawColumns(typeObj["columns"], path+"."+id+".columns")
2075 + if err != nil {
2076 + return nil, err
2077 + }
2078 + out[id] = columns
2079 + }
2080 + return out, nil
2081 +}
2082 +
2083 +func tableTypeOwners(raw any, path string) (map[string]string, error) {
2084 + out := make(map[string]string)
2085 + if raw == nil {
2086 + return out, nil
2087 + }
2088 + obj, ok := raw.(map[string]any)
2089 + if !ok {
2090 + return nil, fmt.Errorf("%s is not an object", path)
2091 + }
2092 + for id, rawType := range obj {
2093 + typeObj, ok := rawType.(map[string]any)
2094 + if !ok {
2095 + return nil, fmt.Errorf("%s.%s is not an object", path, id)
2096 + }
2097 + owner, _ := typeObj["owner"].(string)
2098 + if owner != "" {
2099 + out[id] = owner
2100 + }
2101 + }
2102 + return out, nil
2103 +}
2104 +
2105 +func collectDetailTableColumnTypes(raw any) (map[string]map[string]string, map[string]map[string]string, error) {
2106 + actorTables := make(map[string]map[string]string)
2107 + relationshipTables := make(map[string]map[string]string)
2108 + if raw == nil {
2109 + return actorTables, relationshipTables, nil
2110 + }
2111 + tables, ok := raw.(map[string]any)
2112 + if !ok {
2113 + return nil, nil, fmt.Errorf("data.tables is not an object")
2114 + }
2115 + if err := collectDetailTableGroupColumnTypes(tables["actor"], "data.tables.actor", actorTables); err != nil {
2116 + return nil, nil, err
2117 + }
2118 + if err := collectDetailTableGroupColumnTypes(tables["relationship"], "data.tables.relationship", relationshipTables); err != nil {
2119 + return nil, nil, err
2120 + }
2121 + return actorTables, relationshipTables, nil
2122 +}
2123 +
2124 +func collectDetailTableGroupColumnTypes(raw any, path string, out map[string]map[string]string) error {
2125 + if raw == nil {
2126 + return nil
2127 + }
2128 + group, ok := raw.(map[string]any)
2129 + if !ok {
2130 + return fmt.Errorf("%s is not an object", path)
2131 + }
2132 + for name, rawDetail := range group {
2133 + detail, ok := rawDetail.(map[string]any)
2134 + if !ok {
2135 + return fmt.Errorf("%s.%s is not an object", path, name)
2136 + }
2137 + columns, err := columnTypesFromTable(detail["table"], path+"."+name+".table")
2138 + if err != nil {
2139 + return err
2140 + }
2141 + out[name] = columns
2142 + }
2143 + return nil
2144 +}
2145 +
2146 +func collectScaleKeys(raw any) (map[string]struct{}, error) {
2147 + keys := make(map[string]struct{})
2148 + if raw == nil {
2149 + return keys, nil
2150 + }
2151 + presentation, ok := raw.(map[string]any)
2152 + if !ok {
2153 + return nil, fmt.Errorf("data.presentation is not an object")
2154 + }
2155 + scaleKeys, ok := presentation["scale_keys"].(map[string]any)
2156 + if !ok {
2157 + return keys, nil
2158 + }
2159 + for key := range scaleKeys {
2160 + keys[key] = struct{}{}
2161 + }
2162 + return keys, nil
2163 +}
2164 +
2165 +func requiredEnum(path string, raw any, allowed ...string) (string, error) {
2166 + value, ok := raw.(string)
2167 + if !ok || value == "" {
2168 + return "", fmt.Errorf("%s is not a non-empty string", path)
2169 + }
2170 + if !stringInSet(value, allowed) {
2171 + return "", fmt.Errorf("%s has unsupported value %q", path, value)
2172 + }
2173 + return value, nil
2174 +}
2175 +
2176 +func optionalEnum(path string, raw any, allowed ...string) error {
2177 + if raw == nil {
2178 + return nil
2179 + }
2180 + value, ok := raw.(string)
2181 + if !ok || value == "" {
2182 + return fmt.Errorf("%s is not a non-empty string", path)
2183 + }
2184 + if !stringInSet(value, allowed) {
2185 + return fmt.Errorf("%s has unsupported value %q", path, value)
2186 + }
2187 + return nil
2188 +}
2189 +
2190 +func stringValue(raw any) string {
2191 + value, _ := raw.(string)
2192 + return value
2193 +}
2194 +
2195 +func validateRequiredString(path string, raw any) error {
2196 + value, ok := raw.(string)
2197 + if !ok || value == "" {
2198 + return fmt.Errorf("%s is required", path)
2199 + }
2200 + return nil
2201 +}
2202 +
2203 +func stringInSet(value string, allowed []string) bool {
2204 + return slices.Contains(allowed, value)
2205 +}
2206 +
2207 +func isNumericColumnType(columnType string) bool {
2208 + return columnType == "int" || columnType == "uint" || columnType == "float" || columnType == "duration"
2209 +}
2210 +
2211 +func isDisplayColumnType(columnType string) bool {
2212 + switch columnType {
2213 + case "bool", "int", "uint", "float", "string", "string_ref", "timestamp", "duration", "ip", "ip_ref", "mac", "mac_ref":
2214 + return true
2215 + default:
2216 + return false
2217 + }
2218 +}
2219 +
2220 +var (
2221 + colorSlotTokens = []string{
2222 + "primary", "secondary", "accent", "self", "neutral", "muted", "dim", "derived",
2223 + "info", "structural", "warning", "success", "danger", "blue", "green", "orange",
2224 + "purple", "cyan", "yellow", "teal", "gray",
2225 + }
2226 + opacityTokens = []string{"normal", "muted", "faded"}
2227 + widthTokens = []string{"thin", "normal", "thick", "emphasis"}
2228 + layoutStrengthTokens = []string{"weakest", "weaker", "normal", "stronger", "strongest"}
2229 + layoutDistanceTokens = []string{"closest", "closer", "normal", "farther", "farthest"}
2230 + actorSizeScaleTokens = []string{"compact", "normal", "emphasized"}
2231 + linkSemanticRoleTokens = []string{"normal", "discovery", "ownership", "traffic", "correlation", "control"}
2232 + iconTokens = []string{
2233 + "router", "switch", "firewall", "access_point", "server", "storage", "load_balancer",
2234 + "printer", "phone", "ups", "camera", "process", "agent", "netdata-agent", "parent",
2235 + "remote-endpoint", "local-endpoint", "segment", "self", "ip", "cloud", "container",
2236 + "vm", "database", "service", "datacenter", "cluster", "host", "network", "datastore",
2237 + "datastore_cluster", "resource_pool", "device", "endpoint", "correlation", "interface",
2238 + "group", "unknown",
2239 + }
2240 +)
2241 +
2242 +func decodedTableRows(raw any) (int, error) {
2243 + table, ok := raw.(map[string]any)
2244 + if !ok {
2245 + return 0, fmt.Errorf("table is not an object")
2246 + }
2247 + rows, ok := integerValue(table["rows"])
2248 + if !ok || rows < 0 {
2249 + return 0, fmt.Errorf("rows is not a non-negative integer")
2250 + }
2251 + return rows, nil
2252 +}
2253 +
2254 +func integerValue(raw any) (int, bool) {
2255 + switch value := raw.(type) {
2256 + case int:
2257 + return value, true
2258 + case int64:
2259 + return int(value), true
2260 + case uint64:
2261 + if value > uint64(maxInt()) {
2262 + return 0, false
2263 + }
2264 + return int(value), true
2265 + case float64:
2266 + if math.Trunc(value) != value {
2267 + return 0, false
2268 + }
2269 + return int(value), true
2270 + case json.Number:
2271 + n, err := value.Int64()
2272 + if err != nil {
2273 + return 0, false
2274 + }
2275 + return int(n), true
2276 + default:
2277 + return 0, false
2278 + }
2279 +}
2280 +
2281 +func numberValue(raw any) (float64, bool) {
2282 + switch value := raw.(type) {
2283 + case int:
2284 + return float64(value), true
2285 + case int64:
2286 + return float64(value), true
2287 + case uint64:
2288 + return float64(value), true
2289 + case float64:
2290 + return value, true
2291 + case json.Number:
2292 + n, err := value.Float64()
2293 + if err != nil {
2294 + return 0, false
2295 + }
2296 + return n, true
2297 + default:
2298 + return 0, false
2299 + }
2300 +}
2301 +
2302 +func maxInt() int {
2303 + return int(^uint(0) >> 1)
2304 +}
src/go/plugin/go.d/collector/snmp_topology/func_topology_handler.go
+5 -1
@@ -40,12 +40,16 @@ func (f *funcTopology) Handle(_ context.Context, method string, params funcapi.R
40 if !ok {
41 return funcapi.UnavailableResponse("topology data not available yet, please retry after topology refresh")
42 }
43 + payload, err := snmpTopologyToV1(data)
44 + if err != nil {
45 + return funcapi.InternalErrorResponse("failed to build topology response: %v", err)
46 + }
47
48 return &funcapi.FunctionResponse{
49 Status: 200,
50 Help: "SNMP topology and neighbor discovery data",
51 ResponseType: "topology",
48 - Data: data,
52 + Data: payload,
53 }
54 }
55
src/go/plugin/go.d/collector/snmp_topology/func_topology_test.go
+563 -9
@@ -4,10 +4,15 @@ package snmptopology
4
5 import (
6 "context"
7 + "encoding/json"
8 + "os"
9 + "path/filepath"
10 "testing"
11 "time"
12
13 "github.com/netdata/netdata/go/plugins/pkg/funcapi"
14 + topologyv1 "github.com/netdata/netdata/go/plugins/pkg/topology/v1"
15 + "github.com/santhosh-tekuri/jsonschema/v6"
16 "github.com/stretchr/testify/assert"
17 "github.com/stretchr/testify/require"
18 )
@@ -131,10 +136,17 @@ func TestFuncTopology_Handle_DefaultStrictL2(t *testing.T) {
136 assert.Equal(t, 200, resp.Status)
137 assert.Equal(t, "topology", resp.ResponseType)
138
134 - data, ok := resp.Data.(topologyData)
139 + data, ok := resp.Data.(topologyv1.Data)
140 require.True(t, ok)
136 - assert.Equal(t, "2", data.Layer)
137 - assert.Equal(t, "summary", data.View)
141 + require.NoError(t, validateTopologyV1Data(data))
142 + assert.Equal(t, topologyv1.SchemaVersion, data.SchemaVersion)
143 + assert.Equal(t, snmpTopologyV1ProducerSource, data.Producer.Source)
144 + require.NotNil(t, data.View)
145 + assert.Equal(t, "summary", data.View.ID)
146 + assert.Equal(t, "network", data.View.Scope)
147 + assert.Equal(t, "detailed", data.View.Mode)
148 + assert.Greater(t, data.Actors.Rows, 0)
149 + assert.Greater(t, data.Links.Rows, 0)
150 }
151
152 func TestFuncTopology_Handle_AcceptsSelectorParams(t *testing.T) {
@@ -177,9 +189,11 @@ func TestFuncTopology_Handle_AcceptsSelectorParams(t *testing.T) {
189 resp := f.Handle(context.Background(), topologyMethodID, params)
190 require.NotNil(t, resp)
191 assert.Equal(t, 200, resp.Status)
180 - data, ok := resp.Data.(topologyData)
192 + data, ok := resp.Data.(topologyv1.Data)
193 require.True(t, ok)
182 - assert.Equal(t, "2", data.Layer)
194 + require.NoError(t, validateTopologyV1Data(data))
195 + require.NotNil(t, data.View)
196 + assert.Equal(t, "network", data.View.Scope)
197 }
198
199 func TestFuncTopology_Handle_UnknownSelectorsFallbackToDefaults(t *testing.T) {
@@ -215,7 +229,7 @@ func TestFuncTopology_Handle_UnknownSelectorsFallbackToDefaults(t *testing.T) {
229 defaultResp := f.Handle(context.Background(), topologyMethodID, nil)
230 require.NotNil(t, defaultResp)
231 require.Equal(t, 200, defaultResp.Status)
218 - defaultData, ok := defaultResp.Data.(topologyData)
232 + defaultData, ok := defaultResp.Data.(topologyv1.Data)
233 require.True(t, ok)
234
235 invalidParams := funcapi.ResolveParams(cfg, map[string][]string{
@@ -228,11 +242,397 @@ func TestFuncTopology_Handle_UnknownSelectorsFallbackToDefaults(t *testing.T) {
242 invalidResp := f.Handle(context.Background(), topologyMethodID, invalidParams)
243 require.NotNil(t, invalidResp)
244 require.Equal(t, 200, invalidResp.Status)
231 - invalidData, ok := invalidResp.Data.(topologyData)
245 + invalidData, ok := invalidResp.Data.(topologyv1.Data)
246 require.True(t, ok)
247
234 - assert.Equal(t, defaultData.Layer, invalidData.Layer)
235 - assert.Equal(t, defaultData.View, invalidData.View)
248 + require.NoError(t, validateTopologyV1Data(invalidData))
249 + require.NotNil(t, defaultData.View)
250 + require.NotNil(t, invalidData.View)
251 + assert.Equal(t, defaultData.View.Scope, invalidData.View.Scope)
252 + assert.Equal(t, defaultData.View.ID, invalidData.View.ID)
253 +}
254 +
255 +func TestSNMPTopologyToV1_PreservesActorCustomTables(t *testing.T) {
256 + ts := time.Date(2026, 5, 9, 12, 0, 0, 0, time.UTC)
257 + data := topologyData{
258 + AgentID: "agent-test",
259 + CollectedAt: ts,
260 + View: "summary",
261 + Actors: []topologyActor{
262 + {
263 + ActorID: "device-a",
264 + ActorType: "device",
265 + Match: topologyMatch{
266 + ChassisIDs: []string{"00:11:22:33:44:55"},
267 + MacAddresses: []string{"00:11:22:33:44:55"},
268 + SysName: "sw-a",
269 + },
270 + Attributes: map[string]any{
271 + "ports_total": uint64(0),
272 + "lldp_neighbor_count": uint64(0),
273 + },
274 + Tables: map[string][]map[string]any{
275 + "ports": {
276 + {
277 + "if_index": uint64(1),
278 + "port_id": "1",
279 + "name": "Gi0/1",
280 + "if_name": "Gi0/1",
281 + "if_descr": "GigabitEthernet0/1",
282 + "if_alias": "uplink to sw-b",
283 + "mac": "00:11:22:33:44:56",
284 + "speed": uint64(1000000000),
285 + "neighbor_count": uint64(0),
286 + "vlan_ids": []int{10, 20},
287 + "vendor_note": "uplink",
288 + "neighbors": []map[string]any{
289 + {
290 + "protocol": "lldp",
291 + "remote_port": "Gi0/2",
292 + },
293 + },
294 + "vlans": []map[string]any{
295 + {
296 + "id": "10",
297 + "name": "users",
298 + },
299 + },
300 + },
301 + },
302 + "labels": {
303 + {"name": "custom-label-row"},
304 + },
305 + "custom_labels": {
306 + {"name": "secondary-custom-label-row"},
307 + },
308 + "metadata": {
309 + {"name": "custom-metadata-row"},
310 + },
311 + },
312 + },
313 + {
314 + ActorID: "device-b",
315 + ActorType: "device",
316 + Match: topologyMatch{
317 + ChassisIDs: []string{"aa:bb:cc:dd:ee:ff"},
318 + MacAddresses: []string{"aa:bb:cc:dd:ee:ff"},
319 + SysName: "sw-b",
320 + },
321 + Attributes: map[string]any{
322 + "learned_sources": []string{"arp"},
323 + },
324 + },
325 + },
326 + Links: []topologyLink{
327 + {
328 + Protocol: "lldp",
329 + LinkType: "lldp",
330 + Direction: "bidirectional",
331 + SrcActorID: "device-a",
332 + DstActorID: "device-b",
333 + Src: topologyLinkEndpoint{
334 + Attributes: map[string]any{"if_name": "Gi0/1", "if_index": uint64(1), "port_id": "1"},
335 + },
336 + Dst: topologyLinkEndpoint{
337 + Attributes: map[string]any{"if_name": "Gi0/2", "if_index": uint64(2), "port_id": "2"},
338 + },
339 + Metrics: map[string]any{
340 + "confidence": "high",
341 + "inference": "observed",
342 + "attachment_mode": "lldp",
343 + },
344 + },
345 + },
346 + }
347 +
348 + payload, err := snmpTopologyToV1(data)
349 + require.NoError(t, err)
350 + require.NoError(t, validateTopologyV1Data(payload))
351 + require.NotNil(t, payload.Tables)
352 + require.Contains(t, payload.Tables.Actor, "actor_ports")
353 + require.Contains(t, payload.Tables.Actor, "actor_port_links")
354 + require.Contains(t, payload.Tables.Actor, "actor_labels")
355 + require.Contains(t, payload.Tables.Actor, "actor_metadata")
356 + require.Contains(t, payload.Tables.Actor, "actor_custom_labels")
357 + require.Contains(t, payload.Tables.Actor, "actor_detail_custom_labels")
358 + require.Contains(t, payload.Tables.Actor, "actor_custom_metadata")
359 +
360 + portTable := payload.Tables.Actor["actor_ports"].Table
361 + assert.Equal(t, 1, portTable.Rows)
362 + assert.Empty(t, topologyV1ColumnType(portTable, "port_number"))
363 + assert.Equal(t, "uint", topologyV1ColumnType(portTable, "if_index"))
364 + assert.Equal(t, "string_ref", topologyV1ColumnType(portTable, "port_id"))
365 + assert.Equal(t, "string_ref", topologyV1ColumnType(portTable, "if_name"))
366 + assert.Equal(t, "string_ref", topologyV1ColumnType(portTable, "if_descr"))
367 + assert.Equal(t, "string_ref", topologyV1ColumnType(portTable, "if_alias"))
368 + assert.Equal(t, "string_ref", topologyV1ColumnType(portTable, "mac"))
369 + assert.Equal(t, "uint", topologyV1ColumnType(portTable, "speed"))
370 + assert.Equal(t, "actor_ref", topologyV1ColumnType(portTable, "neighbor_actor"))
371 + assert.Equal(t, "string_ref", topologyV1ColumnType(portTable, "neighbor_port_name"))
372 + assert.Equal(t, "json", topologyV1ColumnType(portTable, "neighbors"))
373 + assert.Equal(t, "json", topologyV1ColumnType(portTable, "vlans"))
374 + assert.Equal(t, "json", topologyV1ColumnType(portTable, "extra"))
375 + assert.Equal(t, []any{uint64(0), nil}, topologyV1ColumnValues(t, payload.Actors, "ports_total"))
376 + assert.Equal(t, []any{nil, []any{"arp"}}, topologyV1ColumnValues(t, payload.Actors, "protocols"))
377 + assert.Equal(t, []any{nil, nil}, topologyV1ColumnValues(t, payload.Actors, "capabilities"))
378 + assert.Equal(t, []any{uint64(1)}, topologyV1ColumnValues(t, portTable, "if_index"))
379 + assert.Equal(t, []string{"1"}, topologyV1StringColumnValues(t, payload, portTable, "port_id"))
380 + assert.Equal(t, []string{"Gi0/1"}, topologyV1StringColumnValues(t, payload, portTable, "if_name"))
381 + assert.Equal(t, []string{"GigabitEthernet0/1"}, topologyV1StringColumnValues(t, payload, portTable, "if_descr"))
382 + assert.Equal(t, []string{"uplink to sw-b"}, topologyV1StringColumnValues(t, payload, portTable, "if_alias"))
383 + assert.Equal(t, []string{"00:11:22:33:44:56"}, topologyV1StringColumnValues(t, payload, portTable, "mac"))
384 + assert.Equal(t, []any{uint64(1000000000)}, topologyV1ColumnValues(t, portTable, "speed"))
385 + assert.Equal(t, []any{[]any{"10", "20"}}, topologyV1ColumnValues(t, portTable, "vlan_ids"))
386 + assert.Equal(t, []any{uint64(0)}, topologyV1ColumnValues(t, portTable, "neighbor_count"))
387 + assert.Equal(t, []any{1}, topologyV1ColumnValues(t, portTable, "neighbor_actor"))
388 + assert.Equal(t, []string{"Gi0/2"}, topologyV1StringColumnValues(t, payload, portTable, "neighbor_port_name"))
389 + assert.Equal(t, []any{map[string]any{"vendor_note": "uplink"}}, topologyV1ColumnValues(t, portTable, "extra"))
390 +
391 + portLinksTable := payload.Tables.Actor["actor_port_links"].Table
392 + assert.Equal(t, 2, portLinksTable.Rows)
393 + assert.Equal(t, "actor_ref", topologyV1ColumnType(portLinksTable, "actor"))
394 + assert.Equal(t, "link_ref", topologyV1ColumnType(portLinksTable, "link"))
395 + assert.Equal(t, "actor_ref", topologyV1ColumnType(portLinksTable, "remote_actor"))
396 + assert.Equal(t, []any{0, 1}, topologyV1ColumnValues(t, portLinksTable, "actor"))
397 + assert.Equal(t, []any{0, 0}, topologyV1ColumnValues(t, portLinksTable, "link"))
398 + assert.Equal(t, []any{1, 0}, topologyV1ColumnValues(t, portLinksTable, "remote_actor"))
399 + assert.Equal(t, []any{uint64(1), uint64(2)}, topologyV1ColumnValues(t, portLinksTable, "if_index"))
400 + assert.Equal(t, []any{uint64(2), uint64(1)}, topologyV1ColumnValues(t, portLinksTable, "remote_if_index"))
401 + assert.Equal(t, []string{"Gi0/1", "Gi0/2"}, topologyV1StringColumnValues(t, payload, portLinksTable, "port_name"))
402 + assert.Equal(t, []string{"Gi0/2", "Gi0/1"}, topologyV1StringColumnValues(t, payload, portLinksTable, "remote_port_name"))
403 + assert.Equal(t, []string{"high", "high"}, topologyV1StringColumnValues(t, payload, portLinksTable, "confidence"))
404 +
405 + labelTable := payload.Tables.Actor["actor_labels"].Table
406 + assert.GreaterOrEqual(t, labelTable.Rows, 2)
407 + assert.Equal(t, "actor_ref", topologyV1ColumnType(labelTable, "actor"))
408 + assert.Equal(t, "string_ref", topologyV1ColumnType(labelTable, "key"))
409 + assert.Equal(t, "attribute", topologyV1ColumnRole(labelTable, "key"))
410 +
411 + require.Contains(t, payload.Evidence, "lldp")
412 + evidenceTable := payload.Evidence["lldp"].Table
413 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "src_port_name"))
414 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "dst_port_name"))
415 + assert.Equal(t, "uint", topologyV1ColumnType(evidenceTable, "src_if_index"))
416 + assert.Equal(t, "uint", topologyV1ColumnType(evidenceTable, "dst_if_index"))
417 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "src_management_ip"))
418 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "dst_management_ip"))
419 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "confidence"))
420 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "inference"))
421 + assert.Equal(t, "string_ref", topologyV1ColumnType(evidenceTable, "attachment_mode"))
422 + assert.Equal(t, []any{uint64(1)}, topologyV1ColumnValues(t, evidenceTable, "src_if_index"))
423 + assert.Equal(t, []any{uint64(2)}, topologyV1ColumnValues(t, evidenceTable, "dst_if_index"))
424 +
425 + deviceType := payload.Types.ActorTypes["device"]
426 + require.NotNil(t, deviceType.Presentation)
427 + require.NotNil(t, deviceType.Presentation.Modal)
428 + require.NotNil(t, deviceType.Presentation.Modal.Labels)
429 + require.NotNil(t, deviceType.Presentation.Modal.Labels.Identification)
430 + assert.Equal(t, "management_ip", deviceType.Presentation.Modal.Labels.Identification.Fields[1].Key)
431 + require.Len(t, deviceType.Presentation.Modal.Sections, 2)
432 + assert.Equal(t, "ports", deviceType.Presentation.Modal.Sections[0].ID)
433 + assert.Equal(t, "if_index", deviceType.Presentation.Modal.Sections[0].Columns[0].ID)
434 + assert.Equal(t, "Port ID", deviceType.Presentation.Modal.Sections[0].Columns[0].Label)
435 + assert.Equal(t, "neighbor_actor", deviceType.Presentation.Modal.Sections[0].Columns[11].ID)
436 + assert.Equal(t, "actor_link", deviceType.Presentation.Modal.Sections[0].Columns[11].Cell)
437 + assert.Equal(t, "expanded", deviceType.Presentation.Modal.Sections[0].Columns[11].Visibility)
438 + assert.Equal(t, "port_neighbors", deviceType.Presentation.Modal.Sections[1].ID)
439 + assert.Equal(t, "actor_table", deviceType.Presentation.Modal.Sections[1].Source.Kind)
440 + assert.Equal(t, "actor_port_links", deviceType.Presentation.Modal.Sections[1].Source.Table)
441 +
442 + endpointType := payload.Types.ActorTypes["endpoint"]
443 + require.NotNil(t, endpointType.Presentation)
444 + require.NotNil(t, endpointType.Presentation.Modal)
445 + require.Len(t, endpointType.Presentation.Modal.Sections, 1)
446 + assert.Equal(t, "links", endpointType.Presentation.Modal.Sections[0].ID)
447 + assert.Equal(t, "selected_side_endpoint", endpointType.Presentation.Modal.Sections[0].Columns[1].Projection.Kind)
448 +}
449 +
450 +func TestSNMPTopologyToV1_UsesIfIndexAsVisiblePortID(t *testing.T) {
451 + data := topologyData{
452 + AgentID: "agent-test",
453 + View: "summary",
454 + Actors: []topologyActor{
455 + {
456 + ActorID: "device-a",
457 + ActorType: "device",
458 + Match: topologyMatch{
459 + SysName: "sw-a",
460 + },
461 + Tables: map[string][]map[string]any{
462 + "ports": {
463 + {
464 + "if_index": uint64(42),
465 + "if_name": "Gi0/42",
466 + },
467 + },
468 + },
469 + },
470 + {
471 + ActorID: "device-b",
472 + ActorType: "device",
473 + Match: topologyMatch{
474 + SysName: "sw-b",
475 + },
476 + },
477 + },
478 + Links: []topologyLink{
479 + {
480 + Protocol: "lldp",
481 + LinkType: "lldp",
482 + SrcActorID: "device-a",
483 + DstActorID: "device-b",
484 + Src: topologyLinkEndpoint{
485 + Attributes: map[string]any{"if_index": uint64(42), "if_name": "Gi0/42"},
486 + },
487 + Dst: topologyLinkEndpoint{
488 + Attributes: map[string]any{"if_name": "Gi0/1"},
489 + },
490 + },
491 + },
492 + }
493 +
494 + payload, err := snmpTopologyToV1(data)
495 + require.NoError(t, err)
496 + require.NoError(t, validateTopologyV1Data(payload))
497 + require.NotNil(t, payload.Tables)
498 +
499 + portTable := payload.Tables.Actor["actor_ports"].Table
500 + assert.Empty(t, topologyV1ColumnType(portTable, "port_number"))
501 + assert.Equal(t, []any{uint64(42)}, topologyV1ColumnValues(t, portTable, "if_index"))
502 +
503 + deviceType := payload.Types.ActorTypes["device"]
504 + require.NotNil(t, deviceType.Presentation)
505 + require.NotNil(t, deviceType.Presentation.Modal)
506 + require.NotEmpty(t, deviceType.Presentation.Modal.Sections)
507 + assert.Equal(t, "if_index", deviceType.Presentation.Modal.Sections[0].Columns[0].ID)
508 + assert.Equal(t, "Port ID", deviceType.Presentation.Modal.Sections[0].Columns[0].Label)
509 +}
510 +
511 +func TestSNMPTopologyToV1_PortNamesOnlyUsePortFields(t *testing.T) {
512 + data := topologyData{
513 + AgentID: "agent-test",
514 + View: "summary",
515 + Actors: []topologyActor{
516 + {
517 + ActorID: "device-a",
518 + ActorType: "device",
519 + Match: topologyMatch{
520 + SysName: "sw-a",
521 + },
522 + },
523 + {
524 + ActorID: "device-b",
525 + ActorType: "device",
526 + Match: topologyMatch{
527 + SysName: "sw-b",
528 + },
529 + },
530 + },
531 + Links: []topologyLink{
532 + {
533 + Protocol: "lldp",
534 + LinkType: "lldp",
535 + SrcActorID: "device-a",
536 + DstActorID: "device-b",
537 + Src: topologyLinkEndpoint{
538 + Attributes: map[string]any{"display_name": "10.0.0.10"},
539 + },
540 + Dst: topologyLinkEndpoint{
541 + Attributes: map[string]any{"sys_name": "sw-b"},
542 + },
543 + },
544 + },
545 + }
546 +
547 + payload, err := snmpTopologyToV1(data)
548 + require.NoError(t, err)
549 + require.NoError(t, validateTopologyV1Data(payload))
550 +
551 + assert.Equal(t, []any{nil}, topologyV1ColumnValues(t, payload.Links, "src_port_name"))
552 + assert.Equal(t, []any{nil}, topologyV1ColumnValues(t, payload.Links, "dst_port_name"))
553 +
554 + require.NotNil(t, payload.Tables)
555 + portLinksTable := payload.Tables.Actor["actor_port_links"].Table
556 + assert.Equal(t, []any{nil, nil}, topologyV1ColumnValues(t, portLinksTable, "port_name"))
557 + assert.Equal(t, []any{nil, nil}, topologyV1ColumnValues(t, portLinksTable, "remote_port_name"))
558 +}
559 +
560 +func TestSNMPTopologyToV1_PreservesLinkPresentationTypes(t *testing.T) {
561 + ts := time.Date(2026, 5, 10, 12, 0, 0, 0, time.UTC)
562 + data := topologyData{
563 + AgentID: "agent-test",
564 + CollectedAt: ts,
565 + View: "summary",
566 + Actors: []topologyActor{
567 + {
568 + ActorID: "device-a",
569 + ActorType: "device",
570 + Match: topologyMatch{
571 + ChassisIDs: []string{"00:11:22:33:44:55"},
572 + MacAddresses: []string{"00:11:22:33:44:55"},
573 + SysName: "sw-a",
574 + },
575 + },
576 + {
577 + ActorID: "device-b",
578 + ActorType: "device",
579 + Match: topologyMatch{
580 + ChassisIDs: []string{"aa:bb:cc:dd:ee:ff"},
581 + MacAddresses: []string{"aa:bb:cc:dd:ee:ff"},
582 + SysName: "sw-b",
583 + },
584 + },
585 + },
586 + Links: []topologyLink{
587 + {
588 + Protocol: "lldp",
589 + LinkType: "lldp",
590 + Direction: "bidirectional",
591 + SrcActorID: "device-a",
592 + DstActorID: "device-b",
593 + },
594 + {
595 + Protocol: "bridge",
596 + LinkType: "segment",
597 + Direction: "bidirectional",
598 + State: "probable",
599 + SrcActorID: "device-a",
600 + DstActorID: "device-b",
601 + Metrics: map[string]any{
602 + "attachment_mode": "probable_bridge_anchor",
603 + "inference": "probable",
604 + },
605 + },
606 + },
607 + }
608 +
609 + payload, err := snmpTopologyToV1(data)
610 + require.NoError(t, err)
611 + require.NoError(t, validateTopologyV1Data(payload))
612 +
613 + assert.Equal(t, []string{"lldp", "probable"}, topologyV1StringColumnValues(t, payload, payload.Links, "type"))
614 +
615 + require.Contains(t, payload.Types.LinkTypes, "lldp")
616 + require.NotNil(t, payload.Types.LinkTypes["lldp"].Presentation)
617 + assert.Equal(t, "accent", payload.Types.LinkTypes["lldp"].Presentation.ColorSlot)
618 + assert.Equal(t, "thick", payload.Types.LinkTypes["lldp"].Presentation.Width)
619 +
620 + require.Contains(t, payload.Types.LinkTypes, "probable")
621 + require.NotNil(t, payload.Types.LinkTypes["probable"].Presentation)
622 + assert.Equal(t, "dim", payload.Types.LinkTypes["probable"].Presentation.ColorSlot)
623 + assert.Equal(t, "normal", payload.Types.LinkTypes["probable"].Presentation.Width)
624 +
625 + require.Contains(t, payload.Types.EvidenceTypes, "lldp")
626 + assert.Equal(t, "lldp", payload.Types.EvidenceTypes["lldp"].LinkType)
627 + require.Contains(t, payload.Types.EvidenceTypes, "probable")
628 + assert.Equal(t, "probable", payload.Types.EvidenceTypes["probable"].LinkType)
629 + require.Contains(t, payload.Evidence, "lldp")
630 + assert.Equal(t, 1, payload.Evidence["lldp"].Table.Rows)
631 + require.Contains(t, payload.Evidence, "probable")
632 + assert.Equal(t, 1, payload.Evidence["probable"].Table.Rows)
633 +
634 + assert.Contains(t, topologyV1LegendLinkTypes(payload), "lldp")
635 + assert.Contains(t, topologyV1LegendLinkTypes(payload), "probable")
636 }
637
638 func TestNormalizeTopologyInferenceStrategy(t *testing.T) {
@@ -322,3 +722,157 @@ func newTestTopologyCacheLLDP(
722 }
723 return cache
724 }
725 +
726 +func validateTopologyV1Data(data topologyv1.Data) error {
727 + bs, err := json.Marshal(data)
728 + if err != nil {
729 + return err
730 + }
731 + var decoded map[string]any
732 + if err := json.Unmarshal(bs, &decoded); err != nil {
733 + return err
734 + }
735 + if err := topologyv1.ValidateDecodedData(decoded); err != nil {
736 + return err
737 + }
738 +
739 + schemaPath := filepath.Clean(filepath.Join("..", "..", "..", "..", "..", "plugins.d", "FUNCTION_TOPOLOGY_SCHEMA.json"))
740 + schemaBytes, err := os.ReadFile(schemaPath)
741 + if err != nil {
742 + return err
743 + }
744 + var schemaDoc any
745 + if err := json.Unmarshal(schemaBytes, &schemaDoc); err != nil {
746 + return err
747 + }
748 + compiler := jsonschema.NewCompiler()
749 + if err := compiler.AddResource("schema.json", schemaDoc); err != nil {
750 + return err
751 + }
752 + schema, err := compiler.Compile("schema.json")
753 + if err != nil {
754 + return err
755 + }
756 + var response any
757 + if err := json.Unmarshal([]byte(`{"status":200,"type":"topology","data":`+string(bs)+`}`), &response); err != nil {
758 + return err
759 + }
760 + return schema.Validate(response)
761 +}
762 +
763 +func topologyV1ColumnType(table topologyv1.Table, columnID string) string {
764 + for _, column := range table.Columns {
765 + if column.ID == columnID {
766 + return column.Type
767 + }
768 + }
769 + return ""
770 +}
771 +
772 +func topologyV1ColumnRole(table topologyv1.Table, columnID string) string {
773 + for _, column := range table.Columns {
774 + if column.ID == columnID {
775 + return column.Role
776 + }
777 + }
778 + return ""
779 +}
780 +
781 +func topologyV1ColumnValues(t *testing.T, table topologyv1.Table, columnID string) []any {
782 + t.Helper()
783 +
784 + for columnIndex, column := range table.Columns {
785 + if column.ID == columnID {
786 + return topologyV1DecodeColumnValues(t, table, columnIndex)
787 + }
788 + }
789 +
790 + require.Failf(t, "missing column", "column %q not found", columnID)
791 + return nil
792 +}
793 +
794 +func topologyV1StringColumnValues(t *testing.T, data topologyv1.Data, table topologyv1.Table, columnID string) []string {
795 + t.Helper()
796 +
797 + for columnIndex, column := range table.Columns {
798 + if column.ID != columnID {
799 + continue
800 + }
801 + require.Equal(t, "string_ref", column.Type)
802 + require.NotEmpty(t, column.Dictionary)
803 + dict := data.Dictionaries[column.Dictionary]
804 + require.NotNil(t, dict)
805 +
806 + values := topologyV1DecodeColumnValues(t, table, columnIndex)
807 + out := make([]string, 0, len(values))
808 + for _, value := range values {
809 + ref, ok := value.(int)
810 + require.Truef(t, ok, "expected integer dictionary reference for %q, got %T", columnID, value)
811 + require.GreaterOrEqual(t, ref, 0)
812 + require.Less(t, ref, len(dict))
813 + text, ok := dict[ref].(string)
814 + require.Truef(t, ok, "expected string dictionary value for %q, got %T", columnID, dict[ref])
815 + out = append(out, text)
816 + }
817 + return out
818 + }
819 +
820 + require.Failf(t, "missing column", "column %q not found", columnID)
821 + return nil
822 +}
823 +
824 +func topologyV1DecodeColumnValues(t *testing.T, table topologyv1.Table, columnIndex int) []any {
825 + t.Helper()
826 +
827 + switch encoding := table.Values[columnIndex].(type) {
828 + case topologyv1.ValuesEncoding:
829 + return encoding.Values
830 + case *topologyv1.ValuesEncoding:
831 + require.NotNil(t, encoding)
832 + return encoding.Values
833 + case topologyv1.ConstEncoding:
834 + values := make([]any, table.Rows)
835 + for i := range values {
836 + values[i] = encoding.Value
837 + }
838 + return values
839 + case *topologyv1.ConstEncoding:
840 + require.NotNil(t, encoding)
841 + values := make([]any, table.Rows)
842 + for i := range values {
843 + values[i] = encoding.Value
844 + }
845 + return values
846 + case topologyv1.DictEncoding:
847 + values := make([]any, 0, len(encoding.Indexes))
848 + for _, index := range encoding.Indexes {
849 + require.GreaterOrEqual(t, index, 0)
850 + require.Less(t, index, len(encoding.Values))
851 + values = append(values, encoding.Values[index])
852 + }
853 + return values
854 + case *topologyv1.DictEncoding:
855 + require.NotNil(t, encoding)
856 + values := make([]any, 0, len(encoding.Indexes))
857 + for _, index := range encoding.Indexes {
858 + require.GreaterOrEqual(t, index, 0)
859 + require.Less(t, index, len(encoding.Values))
860 + values = append(values, encoding.Values[index])
861 + }
862 + return values
863 + default:
864 + require.Failf(t, "unsupported encoding", "column %d has unsupported encoding %T", columnIndex, encoding)
865 + return nil
866 + }
867 +}
868 +
869 +func topologyV1LegendLinkTypes(data topologyv1.Data) []string {
870 + if data.Presentation == nil || data.Presentation.Legend == nil {
871 + return nil
872 + }
873 + out := make([]string, 0, len(data.Presentation.Legend.Links))
874 + for _, entry := range data.Presentation.Legend.Links {
875 + out = append(out, entry.Type)
876 + }
877 + return out
878 +}
src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go new
+2138
@@ -0,0 +1,2138 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package snmptopology
4 +
5 +import (
6 + "fmt"
7 + "maps"
8 + "math"
9 + "reflect"
10 + "regexp"
11 + "sort"
12 + "strings"
13 + "time"
14 +
15 + topologyengine "github.com/netdata/netdata/go/plugins/pkg/topology/engine"
16 + topologyv1 "github.com/netdata/netdata/go/plugins/pkg/topology/v1"
17 +)
18 +
19 +const (
20 + snmpTopologyV1ProducerSource = "snmp-l2"
21 + snmpTopologyV1Instance = "local"
22 +
23 + snmpTopologyV1ActorDevice = "device"
24 + snmpTopologyV1ActorEndpoint = "endpoint"
25 + snmpTopologyV1ActorSegment = "segment"
26 +
27 + snmpTopologyV1LinkObservation = "l2_observation"
28 + snmpTopologyV1LinkLLDP = "lldp"
29 + snmpTopologyV1LinkCDP = "cdp"
30 + snmpTopologyV1LinkBridge = "bridge"
31 + snmpTopologyV1LinkFDB = "fdb"
32 + snmpTopologyV1LinkSTP = "stp"
33 + snmpTopologyV1LinkARP = "arp"
34 + snmpTopologyV1LinkSNMP = "snmp"
35 + snmpTopologyV1LinkProbable = "probable"
36 +)
37 +
38 +var topologyV1IDInvalidChars = regexp.MustCompile(`[^A-Za-z0-9_.:-]+`)
39 +
40 +func snmpTopologyV1ActorTypes() map[string]topologyv1.ActorType {
41 + types := make(map[string]topologyv1.ActorType)
42 + addDevice := func(id, label, colorSlot, icon string) {
43 + types[id] = topologyv1.ActorType{
44 + Layer: "network",
45 + Identity: []string{"id"},
46 + MergeIdentity: []string{"chassis_ids", "mac_addresses", "ip_addresses", "sys_name"},
47 + AggregationScopes: []string{"device", "network"},
48 + Search: &topologyv1.ActorSearchPolicy{
49 + Columns: []string{"display_name", "sys_name", "management_ip", "vendor", "model"},
50 + },
51 + Presentation: &topologyv1.ActorPresentation{
52 + Label: label,
53 + Role: "actor",
54 + Icon: icon,
55 + ColorSlot: colorSlot,
56 + Border: &topologyv1.BorderPresentation{Enabled: new(true)},
57 + Size: &topologyv1.ActorSizePresentation{Mode: "link_count", Scale: "emphasized"},
58 + Layout: &topologyv1.ActorLayoutPresentation{Repulsion: "stronger"},
59 + LabelPolicy: &topologyv1.LabelPolicy{
60 + Columns: []string{"display_name", "sys_name"},
61 + Fallback: "type_label",
62 + MaxLength: 80,
63 + Array: "reject",
64 + },
65 + Ports: &topologyv1.ActorPortsPresentation{
66 + ShowBullets: true,
67 + Sources: []topologyv1.PortSourcePresentation{
68 + {
69 + Source: "actor_table",
70 + Table: "actor_ports",
71 + ActorColumn: "actor",
72 + NameColumn: "name",
73 + TypeColumn: "topology_role",
74 + DefaultType: "topology",
75 + StatusColumn: "oper_status",
76 + ModeColumn: "link_mode",
77 + RoleColumn: "topology_role",
78 + },
79 + },
80 + },
81 + Modal: snmpTopologyV1DeviceModal(),
82 + },
83 + }
84 + }
85 +
86 + addDevice("device", "Device", "primary", "server")
87 + addDevice("router", "Router", "primary", "router")
88 + addDevice("switch", "Switch", "primary", "switch")
89 + addDevice("firewall", "Firewall", "warning", "firewall")
90 + addDevice("access_point", "Access Point", "info", "access_point")
91 + addDevice("server", "Server", "secondary", "server")
92 + addDevice("storage", "Storage", "secondary", "storage")
93 + addDevice("load_balancer", "Load Balancer", "info", "load_balancer")
94 + addDevice("printer", "Printer", "neutral", "printer")
95 + addDevice("phone", "Phone", "neutral", "phone")
96 + addDevice("ups", "UPS", "structural", "ups")
97 + addDevice("camera", "Camera", "neutral", "camera")
98 +
99 + types[snmpTopologyV1ActorEndpoint] = topologyv1.ActorType{
100 + Layer: "network",
101 + Identity: []string{"id"},
102 + MergeIdentity: []string{"mac_addresses", "ip_addresses"},
103 + AggregationScopes: []string{"endpoint", "network"},
104 + Search: &topologyv1.ActorSearchPolicy{Columns: []string{"display_name"}},
105 + Presentation: &topologyv1.ActorPresentation{
106 + Label: "Inferred endpoint",
107 + Role: "endpoint",
108 + Icon: "remote-endpoint",
109 + ColorSlot: "derived",
110 + Border: &topologyv1.BorderPresentation{Enabled: new(true)},
111 + Size: &topologyv1.ActorSizePresentation{Mode: "fixed", Scale: "compact"},
112 + Layout: &topologyv1.ActorLayoutPresentation{Repulsion: "weaker"},
113 + LabelPolicy: &topologyv1.LabelPolicy{
114 + Columns: []string{"display_name"},
115 + Fallback: "type_label",
116 + MaxLength: 80,
117 + Array: "reject",
118 + },
119 + Modal: snmpTopologyV1EndpointModal(),
120 + },
121 + }
122 + types[snmpTopologyV1ActorSegment] = topologyv1.ActorType{
123 + Layer: "network",
124 + Identity: []string{"id"},
125 + MergeIdentity: []string{"id"},
126 + ParentIdentity: []string{"parent_devices"},
127 + AggregationScopes: []string{"segment", "network"},
128 + Search: &topologyv1.ActorSearchPolicy{Enabled: new(false)},
129 + Presentation: &topologyv1.ActorPresentation{
130 + Label: "Network segment",
131 + Role: "group",
132 + Icon: "segment",
133 + ColorSlot: "dim",
134 + Size: &topologyv1.ActorSizePresentation{Mode: "fixed", Scale: "compact"},
135 + Layout: &topologyv1.ActorLayoutPresentation{Repulsion: "weakest"},
136 + LabelPolicy: &topologyv1.LabelPolicy{
137 + Columns: []string{"display_name"},
138 + Fallback: "type_label",
139 + MaxLength: 80,
140 + Array: "reject",
141 + },
142 + Modal: snmpTopologyV1EndpointModal(),
143 + },
144 + }
145 + types["custom"] = topologyv1.ActorType{
146 + Layer: "custom",
147 + Identity: []string{"id"},
148 + MergeIdentity: []string{"id"},
149 + AggregationScopes: []string{"network"},
150 + Search: &topologyv1.ActorSearchPolicy{Columns: []string{"display_name"}},
151 + Presentation: &topologyv1.ActorPresentation{
152 + Label: "Custom",
153 + Role: "actor",
154 + Icon: "service",
155 + ColorSlot: "neutral",
156 + LabelPolicy: &topologyv1.LabelPolicy{
157 + Columns: []string{"display_name"},
158 + Fallback: "type_label",
159 + MaxLength: 80,
160 + Array: "reject",
161 + },
162 + Modal: snmpTopologyV1EndpointModal(),
163 + },
164 + }
165 + return types
166 +}
167 +
168 +func snmpTopologyV1DeviceModal() *topologyv1.ModalPresentation {
169 + return &topologyv1.ModalPresentation{
170 + Labels: snmpTopologyV1DeviceModalLabels(),
171 + MiniTopology: &topologyv1.ModalMiniTopologyPresentation{Depth: 1},
172 + Sections: []topologyv1.ModalSection{
173 + {
174 + ID: "ports",
175 + Label: "Ports",
176 + Order: 1,
177 + Source: topologyv1.ModalSource{
178 + Kind: "actor_table",
179 + Table: "actor_ports",
180 + },
181 + OwnerFilter: &topologyv1.ModalOwnerFilter{
182 + Mode: "actor_column",
183 + ActorColumn: "actor",
184 + },
185 + Columns: snmpTopologyV1PortModalColumns(),
186 + Sort: &topologyv1.ModalSort{Column: "if_index", Direction: "asc"},
187 + },
188 + snmpTopologyV1PortLinksSection(2),
189 + },
190 + }
191 +}
192 +
193 +func snmpTopologyV1EndpointModal() *topologyv1.ModalPresentation {
194 + return &topologyv1.ModalPresentation{
195 + Labels: snmpTopologyV1EndpointModalLabels(),
196 + MiniTopology: &topologyv1.ModalMiniTopologyPresentation{Depth: 1},
197 + Sections: []topologyv1.ModalSection{snmpTopologyV1LinksSection(1)},
198 + }
199 +}
200 +
201 +func snmpTopologyV1DeviceModalLabels() *topologyv1.ModalLabelsPresentation {
202 + return &topologyv1.ModalLabelsPresentation{
203 + Table: "actor_labels",
204 + Identification: &topologyv1.ModalLabelIdentificationPresentation{
205 + Fields: []topologyv1.ModalLabelIdentificationField{
206 + {Key: "display_name", Label: "Name", MaxValues: 1},
207 + {Key: "management_ip", Label: "Management IP", MaxValues: 1},
208 + {Key: "vendor", Label: "Vendor", MaxValues: 1},
209 + {Key: "model", Label: "Model", MaxValues: 1},
210 + {Key: "ports_total", Label: "Ports", MaxValues: 1},
211 + {Key: "lldp_neighbor_count", Label: "LLDP", MaxValues: 1},
212 + {Key: "cdp_neighbor_count", Label: "CDP", MaxValues: 1},
213 + },
214 + },
215 + }
216 +}
217 +
218 +func snmpTopologyV1EndpointModalLabels() *topologyv1.ModalLabelsPresentation {
219 + return &topologyv1.ModalLabelsPresentation{
220 + Table: "actor_labels",
221 + Identification: &topologyv1.ModalLabelIdentificationPresentation{
222 + Fields: []topologyv1.ModalLabelIdentificationField{
223 + {Key: "display_name", Label: "Name", MaxValues: 1},
224 + {Key: "ip_address", Label: "IP", MaxValues: 2},
225 + {Key: "mac_address", Label: "MAC", MaxValues: 2},
226 + {Key: "hostname", Label: "Hostname", MaxValues: 2},
227 + },
228 + },
229 + }
230 +}
231 +
232 +func snmpTopologyV1LinksSection(order int) topologyv1.ModalSection {
233 + return topologyv1.ModalSection{
234 + ID: "links",
235 + Label: "Links",
236 + Order: order,
237 + Source: topologyv1.ModalSource{
238 + Kind: "links",
239 + },
240 + OwnerFilter: &topologyv1.ModalOwnerFilter{
241 + Mode: "incident_link",
242 + SrcActorColumn: "src_actor",
243 + DstActorColumn: "dst_actor",
244 + },
245 + Columns: []topologyv1.ModalColumn{
246 + {
247 + ID: "remote",
248 + Label: "Remote Actor",
249 + Projection: topologyv1.ModalProjection{
250 + Kind: "opposite_actor",
251 + SrcActorColumn: "src_actor",
252 + DstActorColumn: "dst_actor",
253 + },
254 + Cell: "actor_link",
255 + },
256 + modalSelectedSidePortColumn("local_port", "Local Port", "src_port_name", "dst_port_name"),
257 + modalSelectedSidePortColumn("remote_port", "Remote Port", "dst_port_name", "src_port_name"),
258 + modalDirectColumn("protocol", "Protocol", "protocol", "badge"),
259 + modalDirectColumn("direction", "Direction", "direction", "text"),
260 + modalDirectColumn("state", "State", "state", "badge"),
261 + modalDirectColumn("evidence_count", "Evidence", "evidence_count", "number"),
262 + },
263 + }
264 +}
265 +
266 +func snmpTopologyV1PortLinksSection(order int) topologyv1.ModalSection {
267 + return topologyv1.ModalSection{
268 + ID: "port_neighbors",
269 + Label: "Port Neighbors",
270 + Order: order,
271 + Source: topologyv1.ModalSource{
272 + Kind: "actor_table",
273 + Table: "actor_port_links",
274 + },
275 + OwnerFilter: &topologyv1.ModalOwnerFilter{
276 + Mode: "actor_column",
277 + ActorColumn: "actor",
278 + },
279 + Columns: snmpTopologyV1PortLinkModalColumns(),
280 + Sort: &topologyv1.ModalSort{Column: "if_index", Direction: "asc"},
281 + EmptyLabel: "No port neighbors",
282 + }
283 +}
284 +
285 +func modalSelectedSidePortColumn(id, label, selectedSrcPortColumn, selectedDstPortColumn string) topologyv1.ModalColumn {
286 + return topologyv1.ModalColumn{
287 + ID: id,
288 + Label: label,
289 + Projection: topologyv1.ModalProjection{
290 + Kind: "selected_side_endpoint",
291 + SrcActorColumn: "src_actor",
292 + DstActorColumn: "dst_actor",
293 + LocalPortColumn: selectedSrcPortColumn,
294 + RemotePortColumn: selectedDstPortColumn,
295 + },
296 + Cell: "text",
297 + }
298 +}
299 +
300 +func modalDirectColumn(id, label, sourceColumn, cell string) topologyv1.ModalColumn {
301 + return topologyv1.ModalColumn{
302 + ID: id,
303 + Label: label,
304 + Projection: topologyv1.ModalProjection{Kind: "direct", Column: sourceColumn},
305 + Cell: cell,
306 + }
307 +}
308 +
309 +func modalDirectColumnWithVisibility(id, label, sourceColumn, cell, visibility string) topologyv1.ModalColumn {
310 + column := modalDirectColumn(id, label, sourceColumn, cell)
311 + column.Visibility = visibility
312 + return column
313 +}
314 +
315 +func modalActorRefColumn(id, label, actorColumn string) topologyv1.ModalColumn {
316 + return topologyv1.ModalColumn{
317 + ID: id,
318 + Label: label,
319 + Projection: topologyv1.ModalProjection{
320 + Kind: "actor_ref_label",
321 + ActorColumn: actorColumn,
322 + },
323 + Cell: "actor_link",
324 + }
325 +}
326 +
327 +func modalActorRefColumnWithVisibility(id, label, actorColumn, visibility string) topologyv1.ModalColumn {
328 + column := modalActorRefColumn(id, label, actorColumn)
329 + column.Visibility = visibility
330 + return column
331 +}
332 +
333 +func snmpTopologyV1PortTypes() map[string]topologyv1.PortType {
334 + return map[string]topologyv1.PortType{
335 + "lldp": {Presentation: &topologyv1.PortPresentation{Label: "lldp/cdp", ColorSlot: "accent", Opacity: "normal"}},
336 + "switch_facing": {Presentation: &topologyv1.PortPresentation{Label: "switch-facing", ColorSlot: "primary", Opacity: "normal"}},
337 + "host_facing": {Presentation: &topologyv1.PortPresentation{Label: "host-facing", ColorSlot: "secondary", Opacity: "normal"}},
338 + "host_candidate": {Presentation: &topologyv1.PortPresentation{Label: "host-candidate", ColorSlot: "info", Opacity: "normal"}},
339 + "trunk": {Presentation: &topologyv1.PortPresentation{Label: "trunk", ColorSlot: "warning", Opacity: "normal"}},
340 + "access": {Presentation: &topologyv1.PortPresentation{Label: "access", ColorSlot: "derived", Opacity: "normal"}},
341 + "topology": {Presentation: &topologyv1.PortPresentation{Label: "unclassified", ColorSlot: "neutral", Opacity: "normal"}},
342 + "idle": {Presentation: &topologyv1.PortPresentation{Label: "idle", ColorSlot: "muted", Opacity: "muted"}},
343 + "unknown": {Presentation: &topologyv1.PortPresentation{Label: "unknown", ColorSlot: "dim", Opacity: "muted"}},
344 + }
345 +}
346 +
347 +type snmpTopologyV1LinkTypeSpec struct {
348 + id string
349 + label string
350 + colorSlot string
351 + lineStyle string
352 + width string
353 + semanticRole string
354 +}
355 +
356 +func snmpTopologyV1LinkTypeSpecs() []snmpTopologyV1LinkTypeSpec {
357 + return []snmpTopologyV1LinkTypeSpec{
358 + {id: snmpTopologyV1LinkLLDP, label: "LLDP", colorSlot: "accent", lineStyle: "solid", width: "thick", semanticRole: "discovery"},
359 + {id: snmpTopologyV1LinkCDP, label: "CDP", colorSlot: "accent", lineStyle: "solid", width: "thick", semanticRole: "discovery"},
360 + {id: snmpTopologyV1LinkBridge, label: "Bridge", colorSlot: "neutral", lineStyle: "solid", width: "normal", semanticRole: "normal"},
361 + {id: snmpTopologyV1LinkFDB, label: "FDB", colorSlot: "neutral", lineStyle: "solid", width: "normal", semanticRole: "normal"},
362 + {id: snmpTopologyV1LinkSTP, label: "STP", colorSlot: "muted", lineStyle: "solid", width: "normal", semanticRole: "normal"},
363 + {id: snmpTopologyV1LinkARP, label: "ARP", colorSlot: "muted", lineStyle: "solid", width: "normal", semanticRole: "normal"},
364 + {id: snmpTopologyV1LinkSNMP, label: "SNMP", colorSlot: "primary", lineStyle: "solid", width: "normal", semanticRole: "normal"},
365 + {id: snmpTopologyV1LinkProbable, label: "Probable", colorSlot: "dim", lineStyle: "solid", width: "normal", semanticRole: "normal"},
366 + {id: snmpTopologyV1LinkObservation, label: "L2 observation", colorSlot: "neutral", lineStyle: "solid", width: "normal", semanticRole: "normal"},
367 + }
368 +}
369 +
370 +func snmpTopologyV1LinkTypes() map[string]topologyv1.LinkType {
371 + types := make(map[string]topologyv1.LinkType)
372 + for _, spec := range snmpTopologyV1LinkTypeSpecs() {
373 + types[spec.id] = topologyv1.LinkType{
374 + Orientation: "observed_bidirectional",
375 + DirectionRole: "observation",
376 + SemanticRole: spec.semanticRole,
377 + Aggregation: topologyv1.LinkAggregation{
378 + Direction: "canonicalize_unordered",
379 + Evidence: "append",
380 + Metrics: map[string]string{
381 + "evidence_count": "sum",
382 + },
383 + },
384 + EvidenceTypes: []string{spec.id},
385 + Presentation: &topologyv1.LinkPresentation{
386 + Label: spec.label,
387 + ColorSlot: spec.colorSlot,
388 + LineStyle: spec.lineStyle,
389 + Width: spec.width,
390 + Curve: "straight",
391 + Arrow: "none",
392 + },
393 + }
394 + }
395 + return types
396 +}
397 +
398 +func snmpTopologyV1EvidenceTypes() map[string]topologyv1.EvidenceType {
399 + types := make(map[string]topologyv1.EvidenceType)
400 + for _, spec := range snmpTopologyV1LinkTypeSpecs() {
401 + types[spec.id] = topologyv1.EvidenceType{
402 + LinkType: spec.id,
403 + Role: "observation_evidence",
404 + Columns: snmpTopologyV1EvidenceColumns(),
405 + MatchColumns: []string{
406 + "src_actor",
407 + "dst_actor",
408 + "protocol",
409 + "src_endpoint",
410 + "dst_endpoint",
411 + },
412 + }
413 + }
414 + return types
415 +}
416 +
417 +func snmpTopologyV1Presentation() *topologyv1.Presentation {
418 + return &topologyv1.Presentation{
419 + ProfileVersion: "snmp-l2.v1",
420 + Selection: &topologyv1.SelectionPresentation{
421 + ActorClick: &topologyv1.ActorClickPresentation{Mode: "highlight_connections"},
422 + },
423 + Legend: &topologyv1.PresentationLegend{
424 + Actors: []topologyv1.LegendEntry{
425 + {Type: "router", Label: "Router"},
426 + {Type: "switch", Label: "Switch"},
427 + {Type: "firewall", Label: "Firewall"},
428 + {Type: "access_point", Label: "Access Point"},
429 + {Type: "server", Label: "Server"},
430 + {Type: "storage", Label: "Storage"},
431 + {Type: "load_balancer", Label: "Load Balancer"},
432 + {Type: "printer", Label: "Printer"},
433 + {Type: "phone", Label: "IP Phone"},
434 + {Type: "ups", Label: "UPS / PDU"},
435 + {Type: "camera", Label: "Camera / Media"},
436 + {Type: "device", Label: "Other device"},
437 + {Type: "custom", Label: "Other"},
438 + {Type: "endpoint", Label: "Inferred endpoint"},
439 + {Type: "segment", Label: "Network segment"},
440 + },
441 + Links: []topologyv1.LegendEntry{
442 + {Type: snmpTopologyV1LinkLLDP, Label: "LLDP"},
443 + {Type: snmpTopologyV1LinkCDP, Label: "CDP"},
444 + {Type: snmpTopologyV1LinkSNMP, Label: "SNMP"},
445 + {Type: snmpTopologyV1LinkBridge, Label: "Bridge"},
446 + {Type: snmpTopologyV1LinkProbable, Label: "Probable"},
447 + },
448 + Ports: []topologyv1.LegendEntry{
449 + {Type: "lldp", Label: "lldp/cdp"},
450 + {Type: "switch_facing", Label: "switch-facing"},
451 + {Type: "host_facing", Label: "host-facing"},
452 + {Type: "host_candidate", Label: "host-candidate"},
453 + {Type: "trunk", Label: "trunk"},
454 + {Type: "access", Label: "access"},
455 + {Type: "topology", Label: "unclassified"},
456 + {Type: "idle", Label: "idle"},
457 + {Type: "unknown", Label: "unknown"},
458 + },
459 + },
460 + PortFields: []topologyv1.PresentationField{
461 + {Key: "type", Label: "Type"},
462 + {Key: "role", Label: "Role"},
463 + {Key: "status", Label: "Status"},
464 + {Key: "mode", Label: "Mode"},
465 + {Key: "sources", Label: "Sources"},
466 + },
467 + }
468 +}
469 +
470 +func snmpTopologyToV1(data topologyData) (topologyv1.Data, error) {
471 + stringsDict := topologyv1.NewStringDictionary("")
472 + actorRows, actorIndex := buildSNMPTopologyV1Actors(data.Actors, stringsDict)
473 +
474 + linkRows, evidenceSections, err := buildSNMPTopologyV1Links(data.Links, actorIndex, stringsDict)
475 + if err != nil {
476 + return topologyv1.Data{}, err
477 + }
478 +
479 + portNeighborSummaries := buildSNMPTopologyV1PortNeighborSummaries(data.Links, actorIndex)
480 + actorDetails, tableTypes, err := buildSNMPTopologyV1ActorDetails(data.Actors, stringsDict, portNeighborSummaries)
481 + if err != nil {
482 + return topologyv1.Data{}, err
483 + }
484 + if tableTypes == nil {
485 + tableTypes = make(map[string]topologyv1.TableType)
486 + }
487 + if _, ok := tableTypes["actor_labels"]; !ok {
488 + tableTypes["actor_labels"] = snmpTopologyV1ActorLabelsTableType()
489 + }
490 + if _, ok := tableTypes["actor_ports"]; !ok {
491 + tableTypes["actor_ports"] = snmpTopologyV1ActorPortsTableType()
492 + }
493 + tableTypes["actor_port_links"] = snmpTopologyV1ActorPortLinksTableType()
494 + portLinksTable, err := buildSNMPTopologyV1ActorPortLinksTable(data.Links, actorIndex, stringsDict)
495 + if err != nil {
496 + return topologyv1.Data{}, err
497 + }
498 + if portLinksTable.Rows > 0 {
499 + if actorDetails == nil {
500 + actorDetails = make(map[string]topologyv1.DetailTable)
501 + }
502 + actorDetails["actor_port_links"] = topologyv1.DetailTable{
503 + Type: "actor_port_links",
504 + Table: portLinksTable,
505 + }
506 + }
507 +
508 + types := topologyv1.TypeRegistry{
509 + ActorTypes: snmpTopologyV1ActorTypes(),
510 + LinkTypes: snmpTopologyV1LinkTypes(),
511 + PortTypes: snmpTopologyV1PortTypes(),
512 + EvidenceTypes: snmpTopologyV1EvidenceTypes(),
513 + TableTypes: tableTypes,
514 + AggregationScopes: map[string]topologyv1.AggregationScope{
515 + "device": {
516 + Columns: []string{"id"},
517 + EvidencePolicy: "preserve",
518 + },
519 + "network": {
520 + Columns: []string{"type"},
521 + EvidencePolicy: "preserve",
522 + },
523 + "segment": {
524 + Columns: []string{"id"},
525 + EvidencePolicy: "preserve",
526 + },
527 + "endpoint": {
528 + Columns: []string{"id"},
529 + EvidencePolicy: "preserve",
530 + },
531 + },
532 + }
533 +
534 + if len(types.TableTypes) == 0 {
535 + types.TableTypes = nil
536 + }
537 +
538 + payload := topologyv1.Data{
539 + SchemaVersion: topologyv1.SchemaVersion,
540 + Producer: topologyv1.Producer{
541 + Source: snmpTopologyV1ProducerSource,
542 + Instance: firstNonEmptyString(data.AgentID, snmpTopologyV1Instance),
543 + Plugin: "go.d/snmp_topology",
544 + Capabilities: []string{
545 + "lldp",
546 + "cdp",
547 + "fdb",
548 + "stp",
549 + },
550 + },
551 + CollectedAt: data.CollectedAt,
552 + View: &topologyv1.View{
553 + ID: firstNonEmptyString(data.View, "summary"),
554 + Scope: "network",
555 + Mode: "detailed",
556 + },
557 + Dictionaries: topologyv1.Dictionaries{
558 + "strings": stringsDict.Values(),
559 + },
560 + Types: types,
561 + Presentation: snmpTopologyV1Presentation(),
562 + Actors: actorRows,
563 + Links: linkRows,
564 + Evidence: evidenceSections,
565 + Stats: cloneAnyMapForTopologyV1(data.Stats),
566 + }
567 + if payload.CollectedAt.IsZero() {
568 + payload.CollectedAt = time.Now().UTC()
569 + }
570 + if actorDetails != nil {
571 + payload.Tables = &topologyv1.DetailTables{
572 + Actor: actorDetails,
573 + }
574 + }
575 + return payload, nil
576 +}
577 +
578 +func snmpTopologyV1ActorPortsTableType() topologyv1.TableType {
579 + return topologyv1.TableType{
580 + Role: "actor_detail",
581 + Owner: "actor",
582 + Aggregation: "append",
583 + Columns: snmpTopologyV1ActorPortsColumns(),
584 + Presentation: &topologyv1.TableTypePresentation{
585 + Label: "Ports",
586 + Order: 1,
587 + Columns: snmpTopologyV1PortModalColumns(),
588 + },
589 + }
590 +}
591 +
592 +func snmpTopologyV1PortModalColumns() []topologyv1.ModalColumn {
593 + return []topologyv1.ModalColumn{
594 + modalDirectColumn("if_index", "Port ID", "if_index", "number"),
595 + modalDirectColumn("name", "Port", "name", "text"),
596 + modalDirectColumn("oper_status", "Status", "oper_status", "badge"),
597 + modalDirectColumn("admin_status", "Admin", "admin_status", "badge"),
598 + modalDirectColumn("port_type", "Type", "port_type", "badge"),
599 + modalDirectColumn("link_mode", "Mode", "link_mode", "badge"),
600 + modalDirectColumn("topology_role", "Role", "topology_role", "badge"),
601 + modalDirectColumn("vlan_ids", "VLANs", "vlan_ids", "array_count"),
602 + modalDirectColumn("fdb_mac_count", "FDB", "fdb_mac_count", "number"),
603 + modalDirectColumn("link_count", "Links", "link_count", "number"),
604 + modalDirectColumn("neighbor_count", "Neighbors", "neighbor_count", "number"),
605 + modalActorRefColumnWithVisibility("neighbor_actor", "Neighbor", "neighbor_actor", "expanded"),
606 + modalDirectColumnWithVisibility("neighbor_port_name", "Neighbor Port", "neighbor_port_name", "text", "expanded"),
607 + modalDirectColumnWithVisibility("if_name", "ifName", "if_name", "text", "expanded"),
608 + modalDirectColumnWithVisibility("if_descr", "ifDescr", "if_descr", "text", "expanded"),
609 + modalDirectColumnWithVisibility("if_alias", "Alias", "if_alias", "text", "expanded"),
610 + modalDirectColumnWithVisibility("port_id", "Source Port ID", "port_id", "text", "expanded"),
611 + modalDirectColumnWithVisibility("mac", "MAC", "mac", "text", "expanded"),
612 + modalDirectColumnWithVisibility("speed", "Speed", "speed", "number", "expanded"),
613 + modalDirectColumnWithVisibility("stp_state", "STP", "stp_state", "badge", "expanded"),
614 + modalDirectColumnWithVisibility("neighbors", "Neighbor Data", "neighbors", "debug_json", "debug"),
615 + modalDirectColumnWithVisibility("vlans", "VLAN Data", "vlans", "debug_json", "debug"),
616 + modalDirectColumnWithVisibility("extra", "Extra", "extra", "debug_json", "debug"),
617 + }
618 +}
619 +
620 +func snmpTopologyV1ActorPortLinksTableType() topologyv1.TableType {
621 + return topologyv1.TableType{
622 + Role: "actor_detail",
623 + Owner: "actor",
624 + Aggregation: "append",
625 + Columns: snmpTopologyV1ActorPortLinksColumns(),
626 + Presentation: &topologyv1.TableTypePresentation{
627 + Label: "Port Neighbors",
628 + Order: 2,
629 + Columns: snmpTopologyV1PortLinkModalColumns(),
630 + },
631 + }
632 +}
633 +
634 +func snmpTopologyV1PortLinkModalColumns() []topologyv1.ModalColumn {
635 + return []topologyv1.ModalColumn{
636 + modalDirectColumn("if_index", "Port ID", "if_index", "number"),
637 + modalDirectColumn("port_name", "Port", "port_name", "text"),
638 + modalActorRefColumn("remote_actor", "Remote Actor", "remote_actor"),
639 + modalDirectColumn("remote_port_name", "Remote Port", "remote_port_name", "text"),
640 + modalDirectColumn("type", "Type", "type", "badge"),
641 + modalDirectColumn("state", "State", "state", "badge"),
642 + modalDirectColumn("evidence_count", "Evidence", "evidence_count", "number"),
643 + modalDirectColumnWithVisibility("protocol", "Protocol", "protocol", "badge", "expanded"),
644 + modalDirectColumnWithVisibility("remote_if_index", "Remote Port ID", "remote_if_index", "number", "expanded"),
645 + modalDirectColumnWithVisibility("port_id", "Source Port ID", "port_id", "text", "expanded"),
646 + modalDirectColumnWithVisibility("remote_port_id", "Remote Source Port ID", "remote_port_id", "text", "expanded"),
647 + modalDirectColumnWithVisibility("confidence", "Confidence", "confidence", "badge", "expanded"),
648 + modalDirectColumnWithVisibility("inference", "Inference", "inference", "badge", "expanded"),
649 + modalDirectColumnWithVisibility("attachment_mode", "Attachment", "attachment_mode", "badge", "expanded"),
650 + modalDirectColumnWithVisibility("discovered_at", "Discovered", "discovered_at", "timestamp", "expanded"),
651 + modalDirectColumnWithVisibility("last_seen", "Last Seen", "last_seen", "timestamp", "expanded"),
652 + }
653 +}
654 +
655 +func snmpTopologyV1ActorLabelsTableType() topologyv1.TableType {
656 + return topologyv1.TableType{
657 + Role: "actor_inventory",
658 + Owner: "actor",
659 + Aggregation: "set",
660 + Columns: []topologyv1.Column{
661 + topologyv1.NewColumn("actor", "actor_ref", topologyv1.WithRole("reference")),
662 + topologyv1.NewColumn("key", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("attribute")),
663 + topologyv1.NewColumn("value", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("attribute")),
664 + topologyv1.NewColumn("source", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable(), topologyv1.WithRole("attribute")),
665 + topologyv1.NewColumn("kind", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable(), topologyv1.WithRole("attribute")),
666 + topologyv1.NewColumn("value_index", "uint", topologyv1.WithNullable(), topologyv1.WithRole("attribute")),
667 + },
668 + Presentation: &topologyv1.TableTypePresentation{
669 + Label: "Labels",
670 + Order: 0,
671 + Columns: []topologyv1.ModalColumn{
672 + modalDirectColumn("key", "Label", "key", "text"),
673 + modalDirectColumn("value", "Value", "value", "text"),
674 + modalDirectColumn("source", "Source", "source", "badge"),
675 + modalDirectColumn("kind", "Kind", "kind", "badge"),
676 + },
677 + },
678 + }
679 +}
680 +
681 +func snmpTopologyV1ActorPortsColumns() []topologyv1.Column {
682 + return []topologyv1.Column{
683 + topologyv1.NewColumn("actor", "actor_ref", topologyv1.WithRole("reference")),
684 + topologyv1.NewColumn("if_index", "uint", topologyv1.WithNullable()),
685 + topologyv1.NewColumn("port_id", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
686 + topologyv1.NewColumn("name", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
687 + topologyv1.NewColumn("if_name", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
688 + topologyv1.NewColumn("if_descr", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
689 + topologyv1.NewColumn("if_alias", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
690 + topologyv1.NewColumn("mac", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
691 + topologyv1.NewColumn("speed", "uint", topologyv1.WithNullable()),
692 + topologyv1.NewColumn("topology_role", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
693 + topologyv1.NewColumn("oper_status", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
694 + topologyv1.NewColumn("admin_status", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
695 + topologyv1.NewColumn("port_type", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
696 + topologyv1.NewColumn("link_mode", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
697 + topologyv1.NewColumn("stp_state", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
698 + topologyv1.NewColumn("vlan_ids", "array", topologyv1.WithNullable()),
699 + topologyv1.NewColumn("fdb_mac_count", "uint", topologyv1.WithNullable()),
700 + topologyv1.NewColumn("link_count", "uint", topologyv1.WithNullable()),
701 + topologyv1.NewColumn("neighbor_count", "uint", topologyv1.WithNullable()),
702 + topologyv1.NewColumn("neighbor_actor", "actor_ref", topologyv1.WithNullable(), topologyv1.WithRole("reference")),
703 + topologyv1.NewColumn("neighbor_port_name", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
704 + topologyv1.NewColumn("neighbors", "json", topologyv1.WithNullable()),
705 + topologyv1.NewColumn("vlans", "json", topologyv1.WithNullable()),
706 + topologyv1.NewColumn("extra", "json", topologyv1.WithNullable()),
707 + }
708 +}
709 +
710 +func snmpTopologyV1ActorPortLinksColumns() []topologyv1.Column {
711 + return []topologyv1.Column{
712 + topologyv1.NewColumn("actor", "actor_ref", topologyv1.WithRole("reference")),
713 + topologyv1.NewColumn("link", "link_ref", topologyv1.WithRole("reference")),
714 + topologyv1.NewColumn("remote_actor", "actor_ref", topologyv1.WithRole("reference")),
715 + topologyv1.NewColumn("if_index", "uint", topologyv1.WithNullable()),
716 + topologyv1.NewColumn("port_id", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
717 + topologyv1.NewColumn("port_name", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
718 + topologyv1.NewColumn("remote_if_index", "uint", topologyv1.WithNullable()),
719 + topologyv1.NewColumn("remote_port_id", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
720 + topologyv1.NewColumn("remote_port_name", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
721 + topologyv1.NewColumn("type", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
722 + topologyv1.NewColumn("protocol", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
723 + topologyv1.NewColumn("state", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
724 + topologyv1.NewColumn("evidence_count", "uint", topologyv1.WithAggregation("sum")),
725 + topologyv1.NewColumn("confidence", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
726 + topologyv1.NewColumn("inference", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
727 + topologyv1.NewColumn("attachment_mode", "string_ref", topologyv1.WithNullable(), topologyv1.WithDictionary("strings")),
728 + topologyv1.NewColumn("discovered_at", "timestamp", topologyv1.WithNullable(), topologyv1.WithRole("timestamp")),
729 + topologyv1.NewColumn("last_seen", "timestamp", topologyv1.WithNullable(), topologyv1.WithRole("timestamp")),
730 + }
731 +}
732 +
733 +func buildSNMPTopologyV1Actors(actors []topologyActor, stringsDict *topologyv1.StringDictionary) (topologyv1.Table, map[string]int) {
734 + actorIndex := make(map[string]int, len(actors))
735 + ids := make([]any, len(actors))
736 + types := make([]any, len(actors))
737 + layers := make([]any, len(actors))
738 + sources := make([]any, len(actors))
739 + displayNames := make([]any, len(actors))
740 + chassisIDs := make([]any, len(actors))
741 + macAddresses := make([]any, len(actors))
742 + ipAddresses := make([]any, len(actors))
743 + hostnames := make([]any, len(actors))
744 + dnsNames := make([]any, len(actors))
745 + sysObjectIDs := make([]any, len(actors))
746 + sysNames := make([]any, len(actors))
747 + parentDevices := make([]any, len(actors))
748 + vendors := make([]any, len(actors))
749 + models := make([]any, len(actors))
750 + sysDescrs := make([]any, len(actors))
751 + sysLocations := make([]any, len(actors))
752 + sysContacts := make([]any, len(actors))
753 + managementIPs := make([]any, len(actors))
754 + protocols := make([]any, len(actors))
755 + capabilities := make([]any, len(actors))
756 + portsTotal := make([]any, len(actors))
757 + vlanCounts := make([]any, len(actors))
758 + fdbTotalMACs := make([]any, len(actors))
759 + lldpNeighborCounts := make([]any, len(actors))
760 + cdpNeighborCounts := make([]any, len(actors))
761 + endpointsTotal := make([]any, len(actors))
762 + chartIDPrefixes := make([]any, len(actors))
763 + netdataHostIDs := make([]any, len(actors))
764 +
765 + for i, actor := range actors {
766 + actorID := strings.TrimSpace(actor.ActorID)
767 + if actorID == "" {
768 + actorID = fmt.Sprintf("generated:%d", i)
769 + }
770 + actorIndex[actorID] = i
771 + ids[i] = stringsDict.Ref(actorID)
772 + types[i] = stringsDict.Ref(snmpTopologyV1ActorType(actor.ActorType))
773 + layers[i] = stringsDict.Ref(snmpTopologyV1ActorLayer(actor))
774 + sources[i] = stringsDict.Ref(firstNonEmptyString(actor.Source, snmpTopologyV1ProducerSource))
775 + displayNames[i] = nullableStringRef(stringsDict, snmpTopologyV1DisplayName(actor))
776 + chassisIDs[i] = stringArrayCell(actor.Match.ChassisIDs)
777 + macAddresses[i] = stringArrayCell(actor.Match.MacAddresses)
778 + ipAddresses[i] = stringArrayCell(actor.Match.IPAddresses)
779 + hostnames[i] = stringArrayCell(actor.Match.Hostnames)
780 + dnsNames[i] = stringArrayCell(actor.Match.DNSNames)
781 + sysObjectIDs[i] = stringsDict.Ref(actor.Match.SysObjectID)
782 + sysNames[i] = stringsDict.Ref(actor.Match.SysName)
783 + parentDevices[i] = stringArrayCell(anyStringSlice(actor.Attributes["parent_devices"]))
784 + vendors[i] = nullableStringRef(stringsDict, firstNonEmptyString(anyStringValue(actor.Attributes["vendor"]), anyStringValue(actor.Attributes["vendor_derived"])))
785 + models[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["model"]))
786 + sysDescrs[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["sys_descr"]))
787 + sysLocations[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["sys_location"]))
788 + sysContacts[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["sys_contact"]))
789 + managementIPs[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["management_ip"]))
790 + protocols[i] = stringArrayCell(anyStringSlice(actor.Attributes["protocols"]))
791 + if isEmptyArrayCell(protocols[i]) {
792 + // Older SNMP topology payloads used learned_sources for discovered protocols.
793 + protocols[i] = stringArrayCell(anyStringSlice(actor.Attributes["learned_sources"]))
794 + }
795 + if isEmptyArrayCell(protocols[i]) {
796 + protocols[i] = nil
797 + }
798 + capabilities[i] = stringArrayCell(anyStringSlice(actor.Attributes["capabilities"]))
799 + if isEmptyArrayCell(capabilities[i]) {
800 + capabilities[i] = nil
801 + }
802 + portsTotal[i] = nullableUintValue(actor.Attributes["ports_total"])
803 + vlanCounts[i] = nullableUintValue(actor.Attributes["vlan_count"])
804 + fdbTotalMACs[i] = nullableUintValue(actor.Attributes["fdb_total_macs"])
805 + lldpNeighborCounts[i] = nullableUintValue(actor.Attributes["lldp_neighbor_count"])
806 + cdpNeighborCounts[i] = nullableUintValue(actor.Attributes["cdp_neighbor_count"])
807 + endpointsTotal[i] = nullableUintValue(actor.Attributes["endpoints_total"])
808 + chartIDPrefixes[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["chart_id_prefix"]))
809 + netdataHostIDs[i] = nullableStringRef(stringsDict, anyStringValue(actor.Attributes["netdata_host_id"]))
810 + }
811 +
812 + return topologyv1.MustTable(len(actors),
813 + []topologyv1.Column{
814 + topologyv1.NewColumn("id", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("identity")),
815 + topologyv1.NewColumn("type", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
816 + topologyv1.NewColumn("layer", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
817 + topologyv1.NewColumn("source", "string_ref", topologyv1.WithDictionary("strings")),
818 + topologyv1.NewColumn("display_name", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable(), topologyv1.WithRole("attribute")),
819 + topologyv1.NewColumn("chassis_ids", "array", topologyv1.WithRole("merge_identity")),
820 + topologyv1.NewColumn("mac_addresses", "array", topologyv1.WithRole("merge_identity")),
821 + topologyv1.NewColumn("ip_addresses", "array", topologyv1.WithRole("merge_identity")),
822 + topologyv1.NewColumn("hostnames", "array"),
823 + topologyv1.NewColumn("dns_names", "array"),
824 + topologyv1.NewColumn("sys_object_id", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("merge_identity")),
825 + topologyv1.NewColumn("sys_name", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("merge_identity")),
826 + topologyv1.NewColumn("parent_devices", "array", topologyv1.WithRole("parent_identity")),
827 + topologyv1.NewColumn("vendor", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
828 + topologyv1.NewColumn("model", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
829 + topologyv1.NewColumn("sys_descr", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
830 + topologyv1.NewColumn("sys_location", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
831 + topologyv1.NewColumn("sys_contact", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
832 + topologyv1.NewColumn("management_ip", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
833 + topologyv1.NewColumn("protocols", "array", topologyv1.WithNullable()),
834 + topologyv1.NewColumn("capabilities", "array", topologyv1.WithNullable()),
835 + topologyv1.NewColumn("ports_total", "uint", topologyv1.WithNullable()),
836 + topologyv1.NewColumn("vlan_count", "uint", topologyv1.WithNullable()),
837 + topologyv1.NewColumn("fdb_total_macs", "uint", topologyv1.WithNullable()),
838 + topologyv1.NewColumn("lldp_neighbor_count", "uint", topologyv1.WithNullable()),
839 + topologyv1.NewColumn("cdp_neighbor_count", "uint", topologyv1.WithNullable()),
840 + topologyv1.NewColumn("endpoints_total", "uint", topologyv1.WithNullable()),
841 + topologyv1.NewColumn("chart_id_prefix", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
842 + topologyv1.NewColumn("netdata_host_id", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
843 + },
844 + []topologyv1.ColumnEncoding{
845 + topologyv1.Values(ids...),
846 + topologyv1.Values(types...),
847 + topologyv1.Values(layers...),
848 + topologyv1.Values(sources...),
849 + topologyv1.Values(displayNames...),
850 + topologyv1.Values(chassisIDs...),
851 + topologyv1.Values(macAddresses...),
852 + topologyv1.Values(ipAddresses...),
853 + topologyv1.Values(hostnames...),
854 + topologyv1.Values(dnsNames...),
855 + topologyv1.Values(sysObjectIDs...),
856 + topologyv1.Values(sysNames...),
857 + topologyv1.Values(parentDevices...),
858 + topologyv1.Values(vendors...),
859 + topologyv1.Values(models...),
860 + topologyv1.Values(sysDescrs...),
861 + topologyv1.Values(sysLocations...),
862 + topologyv1.Values(sysContacts...),
863 + topologyv1.Values(managementIPs...),
864 + topologyv1.Values(protocols...),
865 + topologyv1.Values(capabilities...),
866 + topologyv1.Values(portsTotal...),
867 + topologyv1.Values(vlanCounts...),
868 + topologyv1.Values(fdbTotalMACs...),
869 + topologyv1.Values(lldpNeighborCounts...),
870 + topologyv1.Values(cdpNeighborCounts...),
871 + topologyv1.Values(endpointsTotal...),
872 + topologyv1.Values(chartIDPrefixes...),
873 + topologyv1.Values(netdataHostIDs...),
874 + },
875 + ), actorIndex
876 +}
877 +
878 +func buildSNMPTopologyV1Links(
879 + links []topologyLink,
880 + actorIndex map[string]int,
881 + stringsDict *topologyv1.StringDictionary,
882 +) (topologyv1.Table, topologyv1.EvidenceMap, error) {
883 + srcActors := make([]any, len(links))
884 + dstActors := make([]any, len(links))
885 + linkTypes := make([]any, len(links))
886 + protocols := make([]any, len(links))
887 + directions := make([]any, len(links))
888 + states := make([]any, len(links))
889 + srcPortNames := make([]any, len(links))
890 + dstPortNames := make([]any, len(links))
891 + evidenceCounts := make([]any, len(links))
892 + discoveredAt := make([]any, len(links))
893 + lastSeen := make([]any, len(links))
894 + evidenceRowsByType := make(map[string]*snmpTopologyV1EvidenceRows)
895 +
896 + for i, link := range links {
897 + src, ok := actorIndex[strings.TrimSpace(link.SrcActorID)]
898 + if !ok {
899 + return topologyv1.Table{}, nil, fmt.Errorf("link %d references unknown source actor %q", i, link.SrcActorID)
900 + }
901 + dst, ok := actorIndex[strings.TrimSpace(link.DstActorID)]
902 + if !ok {
903 + return topologyv1.Table{}, nil, fmt.Errorf("link %d references unknown destination actor %q", i, link.DstActorID)
904 + }
905 + protocol := firstNonEmptyString(link.Protocol, link.LinkType, "l2")
906 + linkType := snmpTopologyV1LinkType(link)
907 + srcActors[i] = src
908 + dstActors[i] = dst
909 + linkTypes[i] = stringsDict.Ref(linkType)
910 + protocols[i] = stringsDict.Ref(protocol)
911 + directions[i] = stringsDict.Ref(firstNonEmptyString(link.Direction, "observed"))
912 + states[i] = nullableStringRef(stringsDict, link.State)
913 + srcPortNames[i] = nullableStringRef(stringsDict, topologyV1EndpointPortName(link.Src))
914 + dstPortNames[i] = nullableStringRef(stringsDict, topologyV1EndpointPortName(link.Dst))
915 + evidenceCounts[i] = 1
916 + discoveredAt[i] = nullableTime(link.DiscoveredAt)
917 + lastSeen[i] = nullableTime(link.LastSeen)
918 + srcEndpoint := nullableJSON(link.Src.Attributes)
919 + dstEndpoint := nullableJSON(link.Dst.Attributes)
920 + metrics := nullableJSON(link.Metrics)
921 +
922 + evidenceRows := evidenceRowsByType[linkType]
923 + if evidenceRows == nil {
924 + evidenceRows = &snmpTopologyV1EvidenceRows{}
925 + evidenceRowsByType[linkType] = evidenceRows
926 + }
927 + evidenceRows.linkRefs = append(evidenceRows.linkRefs, i)
928 + evidenceRows.srcActors = append(evidenceRows.srcActors, src)
929 + evidenceRows.dstActors = append(evidenceRows.dstActors, dst)
930 + evidenceRows.protocols = append(evidenceRows.protocols, stringsDict.Ref(protocol))
931 + evidenceRows.directions = append(evidenceRows.directions, stringsDict.Ref(firstNonEmptyString(link.Direction, "observed")))
932 + evidenceRows.states = append(evidenceRows.states, nullableStringRef(stringsDict, link.State))
933 + evidenceRows.srcPortNames = append(evidenceRows.srcPortNames, nullableStringRef(stringsDict, topologyV1EndpointPortName(link.Src)))
934 + evidenceRows.dstPortNames = append(evidenceRows.dstPortNames, nullableStringRef(stringsDict, topologyV1EndpointPortName(link.Dst)))
935 + evidenceRows.srcIfIndexes = append(evidenceRows.srcIfIndexes, nullableUintValue(link.Src.Attributes["if_index"]))
936 + evidenceRows.dstIfIndexes = append(evidenceRows.dstIfIndexes, nullableUintValue(link.Dst.Attributes["if_index"]))
937 + evidenceRows.srcManagementIPs = append(evidenceRows.srcManagementIPs, nullableStringRef(stringsDict, topologyV1EndpointString(link.Src, "management_ip")))
938 + evidenceRows.dstManagementIPs = append(evidenceRows.dstManagementIPs, nullableStringRef(stringsDict, topologyV1EndpointString(link.Dst, "management_ip")))
939 + evidenceRows.confidences = append(evidenceRows.confidences, nullableStringRef(stringsDict, topologyMetricValueString(link.Metrics, "confidence")))
940 + evidenceRows.inferences = append(evidenceRows.inferences, nullableStringRef(stringsDict, topologyMetricValueString(link.Metrics, "inference")))
941 + evidenceRows.attachmentModes = append(evidenceRows.attachmentModes, nullableStringRef(stringsDict, topologyMetricValueString(link.Metrics, "attachment_mode")))
942 + evidenceRows.srcEndpoints = append(evidenceRows.srcEndpoints, srcEndpoint)
943 + evidenceRows.dstEndpoints = append(evidenceRows.dstEndpoints, dstEndpoint)
944 + evidenceRows.metrics = append(evidenceRows.metrics, metrics)
945 + }
946 +
947 + linkTable := topologyv1.MustTable(len(links),
948 + []topologyv1.Column{
949 + topologyv1.NewColumn("src_actor", "actor_ref", topologyv1.WithRole("reference")),
950 + topologyv1.NewColumn("dst_actor", "actor_ref", topologyv1.WithRole("reference")),
951 + topologyv1.NewColumn("type", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
952 + topologyv1.NewColumn("protocol", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
953 + topologyv1.NewColumn("direction", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
954 + topologyv1.NewColumn("state", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
955 + topologyv1.NewColumn("src_port_name", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
956 + topologyv1.NewColumn("dst_port_name", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
957 + topologyv1.NewColumn("evidence_count", "uint", topologyv1.WithAggregation("sum")),
958 + topologyv1.NewColumn("discovered_at", "timestamp", topologyv1.WithNullable(), topologyv1.WithRole("timestamp")),
959 + topologyv1.NewColumn("last_seen", "timestamp", topologyv1.WithNullable(), topologyv1.WithRole("timestamp")),
960 + },
961 + []topologyv1.ColumnEncoding{
962 + topologyv1.Values(srcActors...),
963 + topologyv1.Values(dstActors...),
964 + topologyv1.Values(linkTypes...),
965 + topologyv1.Values(protocols...),
966 + topologyv1.Values(directions...),
967 + topologyv1.Values(states...),
968 + topologyv1.Values(srcPortNames...),
969 + topologyv1.Values(dstPortNames...),
970 + topologyv1.Values(evidenceCounts...),
971 + topologyv1.Values(discoveredAt...),
972 + topologyv1.Values(lastSeen...),
973 + },
974 + )
975 +
976 + evidenceSections := make(topologyv1.EvidenceMap, len(evidenceRowsByType))
977 + for linkType, rows := range evidenceRowsByType {
978 + evidenceSections[linkType] = topologyv1.EvidenceSection{
979 + Type: linkType,
980 + Table: rows.table(),
981 + }
982 + }
983 +
984 + return linkTable, evidenceSections, nil
985 +}
986 +
987 +type snmpTopologyV1EvidenceRows struct {
988 + linkRefs []any
989 + srcActors []any
990 + dstActors []any
991 + protocols []any
992 + directions []any
993 + states []any
994 + srcPortNames []any
995 + dstPortNames []any
996 + srcIfIndexes []any
997 + dstIfIndexes []any
998 + srcManagementIPs []any
999 + dstManagementIPs []any
1000 + confidences []any
1001 + inferences []any
1002 + attachmentModes []any
1003 + srcEndpoints []any
1004 + dstEndpoints []any
1005 + metrics []any
1006 +}
1007 +
1008 +func (rows *snmpTopologyV1EvidenceRows) table() topologyv1.Table {
1009 + return topologyv1.MustTable(len(rows.linkRefs),
1010 + snmpTopologyV1EvidenceColumns(),
1011 + []topologyv1.ColumnEncoding{
1012 + topologyv1.Values(rows.linkRefs...),
1013 + topologyv1.Values(rows.srcActors...),
1014 + topologyv1.Values(rows.dstActors...),
1015 + topologyv1.Values(rows.protocols...),
1016 + topologyv1.Values(rows.directions...),
1017 + topologyv1.Values(rows.states...),
1018 + topologyv1.Values(rows.srcPortNames...),
1019 + topologyv1.Values(rows.dstPortNames...),
1020 + topologyv1.Values(rows.srcIfIndexes...),
1021 + topologyv1.Values(rows.dstIfIndexes...),
1022 + topologyv1.Values(rows.srcManagementIPs...),
1023 + topologyv1.Values(rows.dstManagementIPs...),
1024 + topologyv1.Values(rows.confidences...),
1025 + topologyv1.Values(rows.inferences...),
1026 + topologyv1.Values(rows.attachmentModes...),
1027 + topologyv1.Values(rows.srcEndpoints...),
1028 + topologyv1.Values(rows.dstEndpoints...),
1029 + topologyv1.Values(rows.metrics...),
1030 + },
1031 + )
1032 +}
1033 +
1034 +func snmpTopologyV1EvidenceColumns() []topologyv1.Column {
1035 + return []topologyv1.Column{
1036 + topologyv1.NewColumn("link", "link_ref", topologyv1.WithRole("reference")),
1037 + topologyv1.NewColumn("src_actor", "actor_ref", topologyv1.WithRole("reference")),
1038 + topologyv1.NewColumn("dst_actor", "actor_ref", topologyv1.WithRole("reference")),
1039 + topologyv1.NewColumn("protocol", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
1040 + topologyv1.NewColumn("direction", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithRole("group_key")),
1041 + topologyv1.NewColumn("state", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1042 + topologyv1.NewColumn("src_port_name", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1043 + topologyv1.NewColumn("dst_port_name", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1044 + topologyv1.NewColumn("src_if_index", "uint", topologyv1.WithNullable()),
1045 + topologyv1.NewColumn("dst_if_index", "uint", topologyv1.WithNullable()),
1046 + topologyv1.NewColumn("src_management_ip", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1047 + topologyv1.NewColumn("dst_management_ip", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1048 + topologyv1.NewColumn("confidence", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1049 + topologyv1.NewColumn("inference", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1050 + topologyv1.NewColumn("attachment_mode", "string_ref", topologyv1.WithDictionary("strings"), topologyv1.WithNullable()),
1051 + topologyv1.NewColumn("src_endpoint", "json", topologyv1.WithNullable()),
1052 + topologyv1.NewColumn("dst_endpoint", "json", topologyv1.WithNullable()),
1053 + topologyv1.NewColumn("metrics", "json", topologyv1.WithNullable()),
1054 + }
1055 +}
1056 +
1057 +func snmpTopologyV1LinkType(link topologyLink) string {
1058 + if snmpTopologyV1LinkIsProbable(link) {
1059 + return snmpTopologyV1LinkProbable
1060 + }
1061 +
1062 + switch strings.ToLower(strings.TrimSpace(firstNonEmptyString(link.Protocol, link.LinkType))) {
1063 + case snmpTopologyV1LinkLLDP:
1064 + return snmpTopologyV1LinkLLDP
1065 + case snmpTopologyV1LinkCDP:
1066 + return snmpTopologyV1LinkCDP
1067 + case snmpTopologyV1LinkBridge:
1068 + return snmpTopologyV1LinkBridge
1069 + case snmpTopologyV1LinkFDB:
1070 + return snmpTopologyV1LinkFDB
1071 + case snmpTopologyV1LinkSTP:
1072 + return snmpTopologyV1LinkSTP
1073 + case snmpTopologyV1LinkARP:
1074 + return snmpTopologyV1LinkARP
1075 + case snmpTopologyV1LinkSNMP:
1076 + return snmpTopologyV1LinkSNMP
1077 + default:
1078 + return snmpTopologyV1LinkObservation
1079 + }
1080 +}
1081 +
1082 +func snmpTopologyV1LinkIsProbable(link topologyLink) bool {
1083 + if strings.EqualFold(strings.TrimSpace(link.State), snmpTopologyV1LinkProbable) {
1084 + return true
1085 + }
1086 + if len(link.Metrics) == 0 {
1087 + return false
1088 + }
1089 + if strings.EqualFold(topologyMetricValueString(link.Metrics, "inference"), snmpTopologyV1LinkProbable) {
1090 + return true
1091 + }
1092 + return strings.HasPrefix(strings.ToLower(topologyMetricValueString(link.Metrics, "attachment_mode")), snmpTopologyV1LinkProbable+"_")
1093 +}
1094 +
1095 +func buildSNMPTopologyV1ActorDetails(
1096 + actors []topologyActor,
1097 + stringsDict *topologyv1.StringDictionary,
1098 + portNeighborSummaries map[snmpTopologyV1PortNeighborKey]snmpTopologyV1PortNeighborSummary,
1099 +) (map[string]topologyv1.DetailTable, map[string]topologyv1.TableType, error) {
1100 + details := make(map[string]topologyv1.DetailTable)
1101 + tableTypes := make(map[string]topologyv1.TableType)
1102 +
1103 + labelsTable := buildSNMPTopologyV1ActorLabelsTable(actors, stringsDict)
1104 + details["actor_labels"] = topologyv1.DetailTable{
1105 + Type: "actor_labels",
1106 + Table: labelsTable,
1107 + }
1108 + tableTypes["actor_labels"] = snmpTopologyV1ActorLabelsTableType()
1109 + usedTableIDs := map[string]struct{}{
1110 + "actor_labels": {},
1111 + "actor_port_links": {},
1112 + }
1113 +
1114 + metadataTable := buildSNMPTopologyV1ActorMetadataTable(actors)
1115 + if metadataTable.Rows > 0 {
1116 + tableID := "actor_metadata"
1117 + details[tableID] = topologyv1.DetailTable{
1118 + Type: tableID,
1119 + Table: metadataTable,
1120 + }
1121 + tableTypes[tableID] = topologyv1.TableType{
1122 + Role: "actor_detail",
1123 + Owner: "actor",
1124 + Aggregation: "append",
1125 + Columns: metadataTable.Columns,
1126 + Presentation: &topologyv1.TableTypePresentation{
1127 + Label: "Debug metadata",
1128 + DefaultVisibility: "debug",
1129 + Columns: []topologyv1.ModalColumn{
1130 + modalDirectColumn("attributes", "Attributes", "attributes", "debug_json"),
1131 + modalDirectColumn("labels", "Labels", "labels", "debug_json"),
1132 + },
1133 + },
1134 + }
1135 + usedTableIDs[tableID] = struct{}{}
1136 + }
1137 +
1138 + tableRowsByName := collectSNMPTopologyV1ActorTableRows(actors)
1139 + tableNames := sortedMapKeys(tableRowsByName)
1140 + reservedCustomTableIDs := snmpTopologyV1ReservedCustomTableIDs(tableNames)
1141 + for _, tableName := range tableNames {
1142 + rows := tableRowsByName[tableName]
1143 + if len(rows) == 0 {
1144 + continue
1145 + }
1146 + tableID := snmpTopologyV1ActorDetailTableID(tableName, usedTableIDs, reservedCustomTableIDs)
1147 + var table topologyv1.Table
1148 + var err error
1149 + if tableID == "actor_ports" {
1150 + table = buildSNMPTopologyV1ActorPortsTable(rows, stringsDict, portNeighborSummaries)
1151 + } else {
1152 + table, err = buildSNMPTopologyV1DynamicTable(rows, stringsDict)
1153 + }
1154 + if err != nil {
1155 + return nil, nil, fmt.Errorf("build actor detail table %q: %w", tableName, err)
1156 + }
1157 + details[tableID] = topologyv1.DetailTable{
1158 + Type: tableID,
1159 + Table: table,
1160 + }
1161 + if tableID == "actor_ports" {
1162 + tableTypes[tableID] = snmpTopologyV1ActorPortsTableType()
1163 + } else {
1164 + tableTypes[tableID] = topologyv1.TableType{
1165 + Role: "actor_detail",
1166 + Owner: "actor",
1167 + Aggregation: "append",
1168 + Columns: table.Columns,
1169 + }
1170 + }
1171 + usedTableIDs[tableID] = struct{}{}
1172 + }
1173 + return details, tableTypes, nil
1174 +}
1175 +
1176 +func snmpTopologyV1ReservedCustomTableIDs(tableNames []string) map[string]struct{} {
1177 + reserved := make(map[string]struct{})
1178 + for _, tableName := range tableNames {
1179 + tableID := topologyID("actor_"+tableName, "actor_detail")
1180 + switch tableID {
1181 + case "actor_labels", "actor_metadata", "actor_port_links":
1182 + reserved[topologyID("actor_custom_"+tableName, "actor_detail")] = struct{}{}
1183 + }
1184 + }
1185 + return reserved
1186 +}
1187 +
1188 +func snmpTopologyV1ActorDetailTableID(tableName string, usedTableIDs, reservedCustomTableIDs map[string]struct{}) string {
1189 + tableID := topologyID("actor_"+tableName, "actor_detail")
1190 + switch tableID {
1191 + case "actor_labels", "actor_metadata", "actor_port_links":
1192 + return snmpTopologyV1UniqueActorDetailTableID(topologyID("actor_custom_"+tableName, "actor_detail"), usedTableIDs)
1193 + default:
1194 + if _, reserved := reservedCustomTableIDs[tableID]; reserved {
1195 + return snmpTopologyV1UniqueActorDetailTableID(topologyID("actor_detail_"+tableName, "actor_detail"), usedTableIDs)
1196 + }
1197 + return snmpTopologyV1UniqueActorDetailTableID(tableID, usedTableIDs)
1198 + }
1199 +}
1200 +
1201 +func snmpTopologyV1UniqueActorDetailTableID(tableID string, usedTableIDs map[string]struct{}) string {
1202 + if _, ok := usedTableIDs[tableID]; !ok {
1203 + return tableID
1204 + }
1205 + for suffix := 2; ; suffix++ {
1206 + candidate := fmt.Sprintf("%s_%d", tableID, suffix)
1207 + if _, ok := usedTableIDs[candidate]; !ok {
1208 + return candidate
1209 + }
1210 + }
1211 +}
1212 +
1213 +func buildSNMPTopologyV1ActorMetadataTable(actors []topologyActor) topologyv1.Table {
1214 + actorRefs := make([]any, 0, len(actors))
1215 + attributes := make([]any, 0, len(actors))
1216 + labels := make([]any, 0, len(actors))
1217 + for i, actor := range actors {
1218 + if len(actor.Attributes) == 0 && len(actor.Labels) == 0 {
1219 + continue
1220 + }
1221 + actorRefs = append(actorRefs, i)
1222 + attributes = append(attributes, nullableJSON(actor.Attributes))
1223 + labels = append(labels, nullableJSON(actor.Labels))
1224 + }
1225 + if len(actorRefs) == 0 {
1226 + return topologyv1.EmptyTable()
1227 + }
1228 + return topologyv1.MustTable(len(actorRefs),
1229 + []topologyv1.Column{
1230 + topologyv1.NewColumn("actor", "actor_ref", topologyv1.WithRole("reference")),
1231 + topologyv1.NewColumn("attributes", "json", topologyv1.WithNullable()),
1232 + topologyv1.NewColumn("labels", "json", topologyv1.WithNullable()),
1233 + },
1234 + []topologyv1.ColumnEncoding{
1235 + topologyv1.Values(actorRefs...),
1236 + topologyv1.Values(attributes...),
1237 + topologyv1.Values(labels...),
1238 + },
1239 + )
1240 +}
1241 +
1242 +func buildSNMPTopologyV1ActorLabelsTable(
1243 + actors []topologyActor,
1244 + stringsDict *topologyv1.StringDictionary,
1245 +) topologyv1.Table {
1246 + type labelRow struct {
1247 + actor int
1248 + key string
1249 + value string
1250 + source string
1251 + kind string
1252 + valueIndex any
1253 + }
1254 +
1255 + rows := make([]labelRow, 0, len(actors)*8)
1256 + add := func(actor int, key, value, source, kind string, valueIndex any) {
1257 + key = strings.TrimSpace(key)
1258 + value = strings.TrimSpace(value)
1259 + if key == "" || value == "" {
1260 + return
1261 + }
1262 + rows = append(rows, labelRow{
1263 + actor: actor,
1264 + key: key,
1265 + value: value,
1266 + source: source,
1267 + kind: kind,
1268 + valueIndex: valueIndex,
1269 + })
1270 + }
1271 + addSlice := func(actor int, key string, values []string, source, kind string) {
1272 + index := 0
1273 + for _, value := range values {
1274 + value = strings.TrimSpace(value)
1275 + if value == "" {
1276 + continue
1277 + }
1278 + add(actor, key, value, source, kind, index)
1279 + index++
1280 + }
1281 + }
1282 +
1283 + scalarAttributeKeys := []string{
1284 + "vendor", "vendor_derived", "model", "sys_descr", "sys_location", "sys_contact",
1285 + "management_ip", "display_name", "display_source", "chart_id_prefix", "chart_context_prefix",
1286 + "netdata_host_id", "ports_total", "ports_up", "ports_down", "vlan_count", "fdb_total_macs",
1287 + "lldp_neighbor_count", "cdp_neighbor_count", "endpoints_total", "if_admin_status_counts",
1288 + "if_oper_status_counts", "if_link_mode_counts", "if_topology_role_counts",
1289 + }
1290 + arrayAttributeKeys := []string{
1291 + "protocols", "protocols_collected", "learned_sources", "capabilities",
1292 + "capabilities_supported", "capabilities_enabled", "if_names", "if_indexes",
1293 + }
1294 +
1295 + for actorIndex, actor := range actors {
1296 + add(actorIndex, "actor_type", snmpTopologyV1ActorType(actor.ActorType), snmpTopologyV1ProducerSource, "identity", nil)
1297 + add(actorIndex, "layer", snmpTopologyV1ActorLayer(actor), snmpTopologyV1ProducerSource, "identity", nil)
1298 + add(actorIndex, "source", firstNonEmptyString(actor.Source, snmpTopologyV1ProducerSource), snmpTopologyV1ProducerSource, "identity", nil)
1299 + add(actorIndex, "display_name", snmpTopologyV1DisplayName(actor), snmpTopologyV1ProducerSource, "attribute", nil)
1300 + add(actorIndex, "sys_name", actor.Match.SysName, snmpTopologyV1ProducerSource, "match", nil)
1301 + add(actorIndex, "sys_object_id", actor.Match.SysObjectID, snmpTopologyV1ProducerSource, "match", nil)
1302 + addSlice(actorIndex, "chassis_id", actor.Match.ChassisIDs, snmpTopologyV1ProducerSource, "match")
1303 + addSlice(actorIndex, "mac_address", actor.Match.MacAddresses, snmpTopologyV1ProducerSource, "match")
1304 + addSlice(actorIndex, "ip_address", actor.Match.IPAddresses, snmpTopologyV1ProducerSource, "match")
1305 + addSlice(actorIndex, "hostname", actor.Match.Hostnames, snmpTopologyV1ProducerSource, "match")
1306 + addSlice(actorIndex, "dns_name", actor.Match.DNSNames, snmpTopologyV1ProducerSource, "match")
1307 +
1308 + for key, value := range actor.Labels {
1309 + add(actorIndex, key, value, "producer_label", "label", nil)
1310 + }
1311 + for _, key := range scalarAttributeKeys {
1312 + if value := topologyV1ScalarLabelValue(actor.Attributes[key]); value != "" {
1313 + add(actorIndex, key, value, snmpTopologyV1ProducerSource, "attribute", nil)
1314 + }
1315 + }
1316 + for _, key := range arrayAttributeKeys {
1317 + addSlice(actorIndex, key, anyStringSlice(actor.Attributes[key]), snmpTopologyV1ProducerSource, "attribute")
1318 + }
1319 + }
1320 +
1321 + actorRefs := make([]any, len(rows))
1322 + keys := make([]any, len(rows))
1323 + values := make([]any, len(rows))
1324 + sources := make([]any, len(rows))
1325 + kinds := make([]any, len(rows))
1326 + valueIndexes := make([]any, len(rows))
1327 + for i, row := range rows {
1328 + actorRefs[i] = row.actor
1329 + keys[i] = stringsDict.Ref(row.key)
1330 + values[i] = stringsDict.Ref(row.value)
1331 + sources[i] = nullableStringRef(stringsDict, row.source)
1332 + kinds[i] = nullableStringRef(stringsDict, row.kind)
1333 + valueIndexes[i] = row.valueIndex
1334 + }
1335 +
1336 + return topologyv1.MustTable(len(rows), snmpTopologyV1ActorLabelsTableType().Columns, []topologyv1.ColumnEncoding{
1337 + topologyv1.Values(actorRefs...),
1338 + topologyv1.Values(keys...),
1339 + topologyv1.Values(values...),
1340 + topologyv1.Values(sources...),
1341 + topologyv1.Values(kinds...),
1342 + topologyv1.Values(valueIndexes...),
1343 + })
1344 +}
1345 +
1346 +type topologyV1DynamicRow struct {
1347 + actorRef int
1348 + values map[string]any
1349 +}
1350 +
1351 +type snmpTopologyV1PortNeighborKey struct {
1352 + actorRef int
1353 + ifIndex uint64
1354 + portName string
1355 +}
1356 +
1357 +type snmpTopologyV1PortNeighborSummary struct {
1358 + remoteActor any
1359 + remotePortName string
1360 + ambiguous bool
1361 +}
1362 +
1363 +func snmpTopologyV1PortNeighborKeyFor(actorRef int, ifIndex any, portName string) snmpTopologyV1PortNeighborKey {
1364 + if index, ok := uintValue(ifIndex); ok && index > 0 {
1365 + return snmpTopologyV1PortNeighborKey{actorRef: actorRef, ifIndex: index}
1366 + }
1367 + portName = strings.ToLower(strings.TrimSpace(portName))
1368 + if portName == "" {
1369 + return snmpTopologyV1PortNeighborKey{actorRef: -1}
1370 + }
1371 + return snmpTopologyV1PortNeighborKey{actorRef: actorRef, portName: portName}
1372 +}
1373 +
1374 +func buildSNMPTopologyV1PortNeighborSummaries(
1375 + links []topologyLink,
1376 + actorIndex map[string]int,
1377 +) map[snmpTopologyV1PortNeighborKey]snmpTopologyV1PortNeighborSummary {
1378 + summaries := make(map[snmpTopologyV1PortNeighborKey]snmpTopologyV1PortNeighborSummary)
1379 + appendSide := func(actorID, remoteActorID string, endpoint, remoteEndpoint topologyLinkEndpoint) {
1380 + actorRef, ok := actorIndex[strings.TrimSpace(actorID)]
1381 + if !ok {
1382 + return
1383 + }
1384 + remoteActorRef, ok := actorIndex[strings.TrimSpace(remoteActorID)]
1385 + if !ok {
1386 + return
1387 + }
1388 + key := snmpTopologyV1PortNeighborKeyFor(actorRef, endpoint.Attributes["if_index"], topologyV1EndpointPortName(endpoint))
1389 + if key.actorRef < 0 {
1390 + return
1391 + }
1392 + if existing, exists := summaries[key]; exists {
1393 + if existing.remoteActor != remoteActorRef || strings.TrimSpace(existing.remotePortName) != strings.TrimSpace(topologyV1EndpointPortName(remoteEndpoint)) {
1394 + existing.ambiguous = true
1395 + summaries[key] = existing
1396 + }
1397 + return
1398 + }
1399 + summaries[key] = snmpTopologyV1PortNeighborSummary{
1400 + remoteActor: remoteActorRef,
1401 + remotePortName: topologyV1EndpointPortName(remoteEndpoint),
1402 + }
1403 + }
1404 +
1405 + for _, link := range links {
1406 + appendSide(link.SrcActorID, link.DstActorID, link.Src, link.Dst)
1407 + appendSide(link.DstActorID, link.SrcActorID, link.Dst, link.Src)
1408 + }
1409 + return summaries
1410 +}
1411 +
1412 +func snmpTopologyV1PortNeighborSummaryFor(
1413 + row topologyV1DynamicRow,
1414 + portName string,
1415 + summaries map[snmpTopologyV1PortNeighborKey]snmpTopologyV1PortNeighborSummary,
1416 +) (snmpTopologyV1PortNeighborSummary, bool) {
1417 + candidates := []string{
1418 + portName,
1419 + topologyV1ScalarLabelValue(row.values["if_name"]),
1420 + topologyV1ScalarLabelValue(row.values["port_name"]),
1421 + topologyV1ScalarLabelValue(row.values["port_id"]),
1422 + }
1423 + seen := make(map[snmpTopologyV1PortNeighborKey]struct{}, len(candidates)+1)
1424 + keys := []snmpTopologyV1PortNeighborKey{
1425 + snmpTopologyV1PortNeighborKeyFor(row.actorRef, row.values["if_index"], ""),
1426 + }
1427 + for _, candidate := range candidates {
1428 + keys = append(keys, snmpTopologyV1PortNeighborKeyFor(row.actorRef, nil, candidate))
1429 + }
1430 + for _, key := range keys {
1431 + if key.actorRef < 0 {
1432 + continue
1433 + }
1434 + if _, ok := seen[key]; ok {
1435 + continue
1436 + }
1437 + seen[key] = struct{}{}
1438 + if summary, ok := summaries[key]; ok {
1439 + if summary.ambiguous {
1440 + return snmpTopologyV1PortNeighborSummary{}, false
1441 + }
1442 + return summary, true
1443 + }
1444 + }
1445 + return snmpTopologyV1PortNeighborSummary{}, false
1446 +}
1447 +
1448 +func collectSNMPTopologyV1ActorTableRows(actors []topologyActor) map[string][]topologyV1DynamicRow {
1449 + tables := make(map[string][]topologyV1DynamicRow)
1450 + for actorIndex, actor := range actors {
1451 + for tableName, rows := range actor.Tables {
1452 + tableName = strings.TrimSpace(tableName)
1453 + if tableName == "" || len(rows) == 0 {
1454 + continue
1455 + }
1456 + for _, row := range rows {
1457 + if len(row) == 0 {
1458 + continue
1459 + }
1460 + tables[tableName] = append(tables[tableName], topologyV1DynamicRow{
1461 + actorRef: actorIndex,
1462 + values: row,
1463 + })
1464 + }
1465 + }
1466 + }
1467 + return tables
1468 +}
1469 +
1470 +func buildSNMPTopologyV1ActorPortsTable(
1471 + rows []topologyV1DynamicRow,
1472 + stringsDict *topologyv1.StringDictionary,
1473 + portNeighborSummaries map[snmpTopologyV1PortNeighborKey]snmpTopologyV1PortNeighborSummary,
1474 +) topologyv1.Table {
1475 + actorRefs := make([]any, len(rows))
1476 + ifIndexes := make([]any, len(rows))
1477 + portIDs := make([]any, len(rows))
1478 + names := make([]any, len(rows))
1479 + ifNames := make([]any, len(rows))
1480 + ifDescrs := make([]any, len(rows))
1481 + ifAliases := make([]any, len(rows))
1482 + macs := make([]any, len(rows))
1483 + speeds := make([]any, len(rows))
1484 + topologyRoles := make([]any, len(rows))
1485 + operStatuses := make([]any, len(rows))
1486 + adminStatuses := make([]any, len(rows))
1487 + portTypes := make([]any, len(rows))
1488 + linkModes := make([]any, len(rows))
1489 + stpStates := make([]any, len(rows))
1490 + vlanIDs := make([]any, len(rows))
1491 + fdbMACCounts := make([]any, len(rows))
1492 + linkCounts := make([]any, len(rows))
1493 + neighborCounts := make([]any, len(rows))
1494 + neighborActors := make([]any, len(rows))
1495 + neighborPortNames := make([]any, len(rows))
1496 + neighbors := make([]any, len(rows))
1497 + vlans := make([]any, len(rows))
1498 + extra := make([]any, len(rows))
1499 +
1500 + for i, row := range rows {
1501 + actorRefs[i] = row.actorRef
1502 + ifIndexes[i] = nullableUintValue(row.values["if_index"])
1503 + portIDs[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["port_id"]))
1504 + portName := firstNonEmptyString(
1505 + topologyV1ScalarLabelValue(row.values["name"]),
1506 + topologyV1ScalarLabelValue(row.values["if_name"]),
1507 + topologyV1ScalarLabelValue(row.values["port_name"]),
1508 + topologyV1ScalarLabelValue(row.values["port_id"]),
1509 + )
1510 + names[i] = nullableStringRef(stringsDict, portName)
1511 + ifNames[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["if_name"]))
1512 + ifDescrs[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["if_descr"]))
1513 + ifAliases[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["if_alias"]))
1514 + macs[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["mac"]))
1515 + speeds[i] = nullableUintValue(row.values["speed"])
1516 + topologyRoles[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["topology_role"]))
1517 + operStatuses[i] = nullableStringRef(stringsDict, firstNonEmptyString(
1518 + topologyV1ScalarLabelValue(row.values["oper_status"]),
1519 + topologyV1ScalarLabelValue(row.values["if_oper_status"]),
1520 + ))
1521 + adminStatuses[i] = nullableStringRef(stringsDict, firstNonEmptyString(
1522 + topologyV1ScalarLabelValue(row.values["admin_status"]),
1523 + topologyV1ScalarLabelValue(row.values["if_admin_status"]),
1524 + ))
1525 + portTypes[i] = nullableStringRef(stringsDict, firstNonEmptyString(
1526 + topologyV1ScalarLabelValue(row.values["port_type"]),
1527 + topologyV1ScalarLabelValue(row.values["if_type"]),
1528 + ))
1529 + linkModes[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["link_mode"]))
1530 + stpStates[i] = nullableStringRef(stringsDict, topologyV1ScalarLabelValue(row.values["stp_state"]))
1531 + vlanIDs[i] = stringArrayCell(anyStringSlice(row.values["vlan_ids"]))
1532 + if isEmptyArrayCell(vlanIDs[i]) {
1533 + vlanIDs[i] = nil
1534 + }
1535 + fdbMACCounts[i] = nullableUintValue(row.values["fdb_mac_count"])
1536 + linkCounts[i] = nullableUintValue(row.values["link_count"])
1537 + neighborCounts[i] = nullableUintValue(row.values["neighbor_count"])
1538 + if neighborCounts[i] == nil {
1539 + if values, ok := anyMapSlice(row.values["neighbors"]); ok {
1540 + neighborCounts[i] = uint64(len(values))
1541 + }
1542 + }
1543 + if summary, ok := snmpTopologyV1PortNeighborSummaryFor(row, portName, portNeighborSummaries); ok {
1544 + neighborActors[i] = summary.remoteActor
1545 + neighborPortNames[i] = nullableStringRef(stringsDict, summary.remotePortName)
1546 + }
1547 + neighbors[i] = nullableJSON(row.values["neighbors"])
1548 + vlans[i] = nullableJSON(row.values["vlans"])
1549 + extra[i] = nullableJSON(snmpTopologyV1ExtraPortValues(row.values))
1550 + }
1551 +
1552 + return topologyv1.MustTable(len(rows), snmpTopologyV1ActorPortsColumns(), []topologyv1.ColumnEncoding{
1553 + topologyv1.Values(actorRefs...),
1554 + topologyv1.Values(ifIndexes...),
1555 + topologyv1.Values(portIDs...),
1556 + topologyv1.Values(names...),
1557 + topologyv1.Values(ifNames...),
1558 + topologyv1.Values(ifDescrs...),
1559 + topologyv1.Values(ifAliases...),
1560 + topologyv1.Values(macs...),
1561 + topologyv1.Values(speeds...),
1562 + topologyv1.Values(topologyRoles...),
1563 + topologyv1.Values(operStatuses...),
1564 + topologyv1.Values(adminStatuses...),
1565 + topologyv1.Values(portTypes...),
1566 + topologyv1.Values(linkModes...),
1567 + topologyv1.Values(stpStates...),
1568 + topologyv1.Values(vlanIDs...),
1569 + topologyv1.Values(fdbMACCounts...),
1570 + topologyv1.Values(linkCounts...),
1571 + topologyv1.Values(neighborCounts...),
1572 + topologyv1.Values(neighborActors...),
1573 + topologyv1.Values(neighborPortNames...),
1574 + topologyv1.Values(neighbors...),
1575 + topologyv1.Values(vlans...),
1576 + topologyv1.Values(extra...),
1577 + })
1578 +}
1579 +
1580 +func buildSNMPTopologyV1ActorPortLinksTable(
1581 + links []topologyLink,
1582 + actorIndex map[string]int,
1583 + stringsDict *topologyv1.StringDictionary,
1584 +) (topologyv1.Table, error) {
1585 + rows := &snmpTopologyV1ActorPortLinkRows{}
1586 + appendSide := func(linkIndex int, link topologyLink, actorID, remoteActorID string, endpoint, remoteEndpoint topologyLinkEndpoint) error {
1587 + actorRef, ok := actorIndex[strings.TrimSpace(actorID)]
1588 + if !ok {
1589 + return fmt.Errorf("link %d references unknown actor %q", linkIndex, actorID)
1590 + }
1591 + remoteActorRef, ok := actorIndex[strings.TrimSpace(remoteActorID)]
1592 + if !ok {
1593 + return fmt.Errorf("link %d references unknown remote actor %q", linkIndex, remoteActorID)
1594 + }
1595 + protocol := firstNonEmptyString(link.Protocol, link.LinkType, "l2")
1596 + linkType := snmpTopologyV1LinkType(link)
1597 +
1598 + rows.actors = append(rows.actors, actorRef)
1599 + rows.links = append(rows.links, linkIndex)
1600 + rows.remoteActors = append(rows.remoteActors, remoteActorRef)
1601 + rows.ifIndexes = append(rows.ifIndexes, nullableUintValue(endpoint.Attributes["if_index"]))
1602 + rows.portIDs = append(rows.portIDs, nullableStringRef(stringsDict, topologyV1EndpointString(endpoint, "port_id")))
1603 + rows.portNames = append(rows.portNames, nullableStringRef(stringsDict, topologyV1EndpointPortName(endpoint)))
1604 + rows.remoteIfIndexes = append(rows.remoteIfIndexes, nullableUintValue(remoteEndpoint.Attributes["if_index"]))
1605 + rows.remotePortIDs = append(rows.remotePortIDs, nullableStringRef(stringsDict, topologyV1EndpointString(remoteEndpoint, "port_id")))
1606 + rows.remotePortNames = append(rows.remotePortNames, nullableStringRef(stringsDict, topologyV1EndpointPortName(remoteEndpoint)))
1607 + rows.types = append(rows.types, stringsDict.Ref(linkType))
1608 + rows.protocols = append(rows.protocols, stringsDict.Ref(protocol))
1609 + rows.states = append(rows.states, nullableStringRef(stringsDict, link.State))
1610 + rows.evidenceCounts = append(rows.evidenceCounts, uint64(1))
1611 + rows.confidences = append(rows.confidences, nullableStringRef(stringsDict, topologyMetricValueString(link.Metrics, "confidence")))
1612 + rows.inferences = append(rows.inferences, nullableStringRef(stringsDict, topologyMetricValueString(link.Metrics, "inference")))
1613 + rows.attachmentModes = append(rows.attachmentModes, nullableStringRef(stringsDict, topologyMetricValueString(link.Metrics, "attachment_mode")))
1614 + rows.discoveredAt = append(rows.discoveredAt, nullableTime(link.DiscoveredAt))
1615 + rows.lastSeen = append(rows.lastSeen, nullableTime(link.LastSeen))
1616 + return nil
1617 + }
1618 +
1619 + for i, link := range links {
1620 + if err := appendSide(i, link, link.SrcActorID, link.DstActorID, link.Src, link.Dst); err != nil {
1621 + return topologyv1.Table{}, err
1622 + }
1623 + if err := appendSide(i, link, link.DstActorID, link.SrcActorID, link.Dst, link.Src); err != nil {
1624 + return topologyv1.Table{}, err
1625 + }
1626 + }
1627 +
1628 + return rows.table(), nil
1629 +}
1630 +
1631 +type snmpTopologyV1ActorPortLinkRows struct {
1632 + actors []any
1633 + links []any
1634 + remoteActors []any
1635 + ifIndexes []any
1636 + portIDs []any
1637 + portNames []any
1638 + remoteIfIndexes []any
1639 + remotePortIDs []any
1640 + remotePortNames []any
1641 + types []any
1642 + protocols []any
1643 + states []any
1644 + evidenceCounts []any
1645 + confidences []any
1646 + inferences []any
1647 + attachmentModes []any
1648 + discoveredAt []any
1649 + lastSeen []any
1650 +}
1651 +
1652 +func (rows *snmpTopologyV1ActorPortLinkRows) table() topologyv1.Table {
1653 + return topologyv1.MustTable(len(rows.actors),
1654 + snmpTopologyV1ActorPortLinksColumns(),
1655 + []topologyv1.ColumnEncoding{
1656 + topologyv1.Values(rows.actors...),
1657 + topologyv1.Values(rows.links...),
1658 + topologyv1.Values(rows.remoteActors...),
1659 + topologyv1.Values(rows.ifIndexes...),
1660 + topologyv1.Values(rows.portIDs...),
1661 + topologyv1.Values(rows.portNames...),
1662 + topologyv1.Values(rows.remoteIfIndexes...),
1663 + topologyv1.Values(rows.remotePortIDs...),
1664 + topologyv1.Values(rows.remotePortNames...),
1665 + topologyv1.Values(rows.types...),
1666 + topologyv1.Values(rows.protocols...),
1667 + topologyv1.Values(rows.states...),
1668 + topologyv1.Values(rows.evidenceCounts...),
1669 + topologyv1.Values(rows.confidences...),
1670 + topologyv1.Values(rows.inferences...),
1671 + topologyv1.Values(rows.attachmentModes...),
1672 + topologyv1.Values(rows.discoveredAt...),
1673 + topologyv1.Values(rows.lastSeen...),
1674 + },
1675 + )
1676 +}
1677 +
1678 +var snmpTopologyV1ActorPortCanonicalKeys = map[string]struct{}{
1679 + "admin_status": {},
1680 + "fdb_mac_count": {},
1681 + "if_admin_status": {},
1682 + "if_alias": {},
1683 + "if_descr": {},
1684 + "if_index": {},
1685 + "if_name": {},
1686 + "if_oper_status": {},
1687 + "if_type": {},
1688 + "link_count": {},
1689 + "link_mode": {},
1690 + "mac": {},
1691 + "name": {},
1692 + "neighbor_actor": {},
1693 + "neighbor_count": {},
1694 + "neighbor_port_name": {},
1695 + "neighbors": {},
1696 + "oper_status": {},
1697 + "port_id": {},
1698 + "port_name": {},
1699 + "port_type": {},
1700 + "speed": {},
1701 + "stp_state": {},
1702 + "topology_role": {},
1703 + "vlan_ids": {},
1704 + "vlans": {},
1705 +}
1706 +
1707 +func snmpTopologyV1ExtraPortValues(values map[string]any) map[string]any {
1708 + extra := make(map[string]any)
1709 + for key, value := range values {
1710 + key = strings.TrimSpace(key)
1711 + if key == "" {
1712 + continue
1713 + }
1714 + if _, ok := snmpTopologyV1ActorPortCanonicalKeys[key]; ok {
1715 + continue
1716 + }
1717 + extra[key] = value
1718 + }
1719 + if len(extra) == 0 {
1720 + return nil
1721 + }
1722 + return extra
1723 +}
1724 +
1725 +func buildSNMPTopologyV1DynamicTable(rows []topologyV1DynamicRow, stringsDict *topologyv1.StringDictionary) (topologyv1.Table, error) {
1726 + keysSet := make(map[string]struct{})
1727 + for _, row := range rows {
1728 + for key := range row.values {
1729 + key = strings.TrimSpace(key)
1730 + if key != "" {
1731 + keysSet[key] = struct{}{}
1732 + }
1733 + }
1734 + }
1735 + keys := sortedMapKeys(keysSet)
1736 +
1737 + columns := make([]topologyv1.Column, 0, len(keys)+1)
1738 + values := make([]topologyv1.ColumnEncoding, 0, len(keys)+1)
1739 + actorRefs := make([]any, len(rows))
1740 + for i, row := range rows {
1741 + actorRefs[i] = row.actorRef
1742 + }
1743 + columns = append(columns, topologyv1.NewColumn("actor", "actor_ref", topologyv1.WithRole("reference")))
1744 + values = append(values, topologyv1.Values(actorRefs...))
1745 +
1746 + for _, key := range keys {
1747 + columnValues := make([]any, len(rows))
1748 + for i, row := range rows {
1749 + if value, ok := row.values[key]; ok {
1750 + columnValues[i] = value
1751 + }
1752 + }
1753 + columnType := inferTopologyV1ColumnType(columnValues)
1754 + columnID := topologyID(key, "field")
1755 + column := topologyv1.NewColumn(columnID, columnType, topologyv1.WithNullable())
1756 + if columnType == "string_ref" {
1757 + column = topologyv1.NewColumn(columnID, columnType, topologyv1.WithNullable(), topologyv1.WithDictionary("strings"))
1758 + for i, value := range columnValues {
1759 + if value == nil {
1760 + continue
1761 + }
1762 + columnValues[i] = stringsDict.Ref(fmt.Sprint(value))
1763 + }
1764 + }
1765 + columns = append(columns, column)
1766 + values = append(values, topologyv1.Values(columnValues...))
1767 + }
1768 +
1769 + return topologyv1.NewTable(len(rows), columns, values)
1770 +}
1771 +
1772 +func inferTopologyV1ColumnType(values []any) string {
1773 + typ := ""
1774 + for _, value := range values {
1775 + if value == nil {
1776 + continue
1777 + }
1778 + valueType := topologyV1ValueType(value)
1779 + if typ == "" {
1780 + typ = valueType
1781 + continue
1782 + }
1783 + if typ != valueType {
1784 + return "json"
1785 + }
1786 + }
1787 + if typ == "" {
1788 + return "json"
1789 + }
1790 + return typ
1791 +}
1792 +
1793 +func topologyV1ValueType(value any) string {
1794 + switch typed := value.(type) {
1795 + case bool:
1796 + return "bool"
1797 + case int, int8, int16, int32, int64:
1798 + return "int"
1799 + case uint, uint8, uint16, uint32, uint64:
1800 + return "uint"
1801 + case float32:
1802 + if math.Trunc(float64(typed)) == float64(typed) {
1803 + return "int"
1804 + }
1805 + return "float"
1806 + case float64:
1807 + if math.Trunc(typed) == typed {
1808 + return "int"
1809 + }
1810 + return "float"
1811 + case string:
1812 + return "string_ref"
1813 + case []string, []int, []int64, []uint, []uint64, []float64, []bool:
1814 + return "array"
1815 + case []any:
1816 + if scalarArray(typed) {
1817 + return "array"
1818 + }
1819 + return "json"
1820 + default:
1821 + return "json"
1822 + }
1823 +}
1824 +
1825 +func scalarArray(values []any) bool {
1826 + for _, value := range values {
1827 + switch value.(type) {
1828 + case nil, bool, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64, string:
1829 + default:
1830 + return false
1831 + }
1832 + }
1833 + return true
1834 +}
1835 +
1836 +func snmpTopologyV1ActorType(actorType string) string {
1837 + normalized := strings.ToLower(strings.TrimSpace(actorType))
1838 + if topologyengine.IsDeviceActorType(normalized) {
1839 + return normalized
1840 + }
1841 + switch normalized {
1842 + case snmpTopologyV1ActorDevice:
1843 + return snmpTopologyV1ActorDevice
1844 + case snmpTopologyV1ActorEndpoint:
1845 + return snmpTopologyV1ActorEndpoint
1846 + case snmpTopologyV1ActorSegment:
1847 + return snmpTopologyV1ActorSegment
1848 + default:
1849 + return "custom"
1850 + }
1851 +}
1852 +
1853 +func snmpTopologyV1DisplayName(actor topologyActor) string {
1854 + return firstNonEmptyString(
1855 + anyStringValue(actor.Attributes["display_name"]),
1856 + anyStringValue(actor.Attributes["name"]),
1857 + actor.Labels["display_name"],
1858 + actor.Labels["name"],
1859 + actor.Match.SysName,
1860 + firstString(actor.Match.Hostnames),
1861 + firstString(actor.Match.DNSNames),
1862 + )
1863 +}
1864 +
1865 +func snmpTopologyV1ActorLayer(actor topologyActor) string {
1866 + switch snmpTopologyV1ActorType(actor.ActorType) {
1867 + case snmpTopologyV1ActorEndpoint, snmpTopologyV1ActorSegment:
1868 + return "network"
1869 + default:
1870 + if topologyengine.IsDeviceActorType(actor.ActorType) {
1871 + return "network"
1872 + }
1873 + return "custom"
1874 + }
1875 +}
1876 +
1877 +func anyStringValue(value any) string {
1878 + switch typed := value.(type) {
1879 + case string:
1880 + return strings.TrimSpace(typed)
1881 + case fmt.Stringer:
1882 + return strings.TrimSpace(typed.String())
1883 + default:
1884 + return ""
1885 + }
1886 +}
1887 +
1888 +func topologyV1ScalarLabelValue(value any) string {
1889 + switch typed := value.(type) {
1890 + case nil:
1891 + return ""
1892 + case string:
1893 + return strings.TrimSpace(typed)
1894 + case bool:
1895 + if typed {
1896 + return "true"
1897 + }
1898 + return "false"
1899 + case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64:
1900 + return strings.TrimSpace(fmt.Sprint(typed))
1901 + default:
1902 + return ""
1903 + }
1904 +}
1905 +
1906 +func nullableUintValue(value any) any {
1907 + out, ok := uintValue(value)
1908 + if !ok {
1909 + return nil
1910 + }
1911 + return out
1912 +}
1913 +
1914 +func uintValue(value any) (uint64, bool) {
1915 + switch typed := value.(type) {
1916 + case int:
1917 + if typed >= 0 {
1918 + return uint64(typed), true
1919 + }
1920 + case int8:
1921 + if typed >= 0 {
1922 + return uint64(typed), true
1923 + }
1924 + case int16:
1925 + if typed >= 0 {
1926 + return uint64(typed), true
1927 + }
1928 + case int32:
1929 + if typed >= 0 {
1930 + return uint64(typed), true
1931 + }
1932 + case int64:
1933 + if typed >= 0 {
1934 + return uint64(typed), true
1935 + }
1936 + case uint:
1937 + return uint64(typed), true
1938 + case uint8:
1939 + return uint64(typed), true
1940 + case uint16:
1941 + return uint64(typed), true
1942 + case uint32:
1943 + return uint64(typed), true
1944 + case uint64:
1945 + return typed, true
1946 + case float32:
1947 + if typed >= 0 && math.Trunc(float64(typed)) == float64(typed) {
1948 + return uint64(typed), true
1949 + }
1950 + case float64:
1951 + if typed >= 0 && math.Trunc(typed) == typed {
1952 + return uint64(typed), true
1953 + }
1954 + }
1955 + return 0, false
1956 +}
1957 +
1958 +func topologyV1EndpointString(endpoint topologyLinkEndpoint, key string) string {
1959 + return firstNonEmptyString(
1960 + anyStringValue(endpoint.Attributes[key]),
1961 + topologyV1MatchString(endpoint.Match, key),
1962 + )
1963 +}
1964 +
1965 +func topologyV1EndpointPortName(endpoint topologyLinkEndpoint) string {
1966 + return firstNonEmptyString(
1967 + topologyV1EndpointString(endpoint, "port_name"),
1968 + topologyV1EndpointString(endpoint, "if_name"),
1969 + topologyV1EndpointString(endpoint, "if_descr"),
1970 + topologyV1EndpointString(endpoint, "port_id"),
1971 + )
1972 +}
1973 +
1974 +func topologyV1MatchString(match topologyMatch, key string) string {
1975 + switch key {
1976 + case "sys_name":
1977 + return match.SysName
1978 + case "sys_object_id":
1979 + return match.SysObjectID
1980 + default:
1981 + return ""
1982 + }
1983 +}
1984 +
1985 +func firstString(values []string) string {
1986 + for _, value := range values {
1987 + if value = strings.TrimSpace(value); value != "" {
1988 + return value
1989 + }
1990 + }
1991 + return ""
1992 +}
1993 +
1994 +func nullableTime(value *time.Time) any {
1995 + if value == nil || value.IsZero() {
1996 + return nil
1997 + }
1998 + return value.UTC().Format(time.RFC3339Nano)
1999 +}
2000 +
2001 +func nullableStringRef(dict *topologyv1.StringDictionary, value string) any {
2002 + value = strings.TrimSpace(value)
2003 + if value == "" {
2004 + return nil
2005 + }
2006 + return dict.Ref(value)
2007 +}
2008 +
2009 +func nullableJSON(value any) any {
2010 + switch typed := value.(type) {
2011 + case nil:
2012 + return nil
2013 + case map[string]any:
2014 + if len(typed) == 0 {
2015 + return nil
2016 + }
2017 + case map[string]string:
2018 + if len(typed) == 0 {
2019 + return nil
2020 + }
2021 + case []any:
2022 + if len(typed) == 0 {
2023 + return nil
2024 + }
2025 + case []map[string]any:
2026 + if len(typed) == 0 {
2027 + return nil
2028 + }
2029 + }
2030 + return value
2031 +}
2032 +
2033 +func stringArrayCell(values []string) []any {
2034 + out := make([]any, 0, len(values))
2035 + for _, value := range values {
2036 + value = strings.TrimSpace(value)
2037 + if value != "" {
2038 + out = append(out, value)
2039 + }
2040 + }
2041 + return out
2042 +}
2043 +
2044 +func isEmptyArrayCell(value any) bool {
2045 + values, ok := value.([]any)
2046 + return ok && len(values) == 0
2047 +}
2048 +
2049 +func anyStringSlice(value any) []string {
2050 + switch typed := value.(type) {
2051 + case []string:
2052 + return typed
2053 + case []any:
2054 + out := make([]string, 0, len(typed))
2055 + for _, item := range typed {
2056 + if s := strings.TrimSpace(fmt.Sprint(item)); s != "" {
2057 + out = append(out, s)
2058 + }
2059 + }
2060 + return out
2061 + default:
2062 + rv := reflect.ValueOf(value)
2063 + if rv.Kind() != reflect.Slice && rv.Kind() != reflect.Array {
2064 + return nil
2065 + }
2066 + out := make([]string, 0, rv.Len())
2067 + for i := 0; i < rv.Len(); i++ {
2068 + if s := topologyV1ScalarLabelValue(rv.Index(i).Interface()); s != "" {
2069 + out = append(out, s)
2070 + }
2071 + }
2072 + return out
2073 + }
2074 +}
2075 +
2076 +func anyMapSlice(value any) ([]map[string]any, bool) {
2077 + switch typed := value.(type) {
2078 + case []map[string]any:
2079 + return typed, true
2080 + case []any:
2081 + out := make([]map[string]any, 0, len(typed))
2082 + for _, item := range typed {
2083 + row, ok := item.(map[string]any)
2084 + if !ok {
2085 + return nil, false
2086 + }
2087 + out = append(out, row)
2088 + }
2089 + return out, true
2090 + default:
2091 + return nil, false
2092 + }
2093 +}
2094 +
2095 +func sortedMapKeys[T any](m map[string]T) []string {
2096 + keys := make([]string, 0, len(m))
2097 + for key := range m {
2098 + keys = append(keys, key)
2099 + }
2100 + sort.Strings(keys)
2101 + return keys
2102 +}
2103 +
2104 +func topologyID(value, fallback string) string {
2105 + value = strings.TrimSpace(value)
2106 + if value == "" {
2107 + value = fallback
2108 + }
2109 + value = topologyV1IDInvalidChars.ReplaceAllString(value, "_")
2110 + value = strings.Trim(value, "_.:-")
2111 + if value == "" {
2112 + value = fallback
2113 + }
2114 + first := value[0]
2115 + if (first >= 'A' && first <= 'Z') || (first >= 'a' && first <= 'z') {
2116 + return value
2117 + }
2118 + return "x_" + value
2119 +}
2120 +
2121 +func firstNonEmptyString(values ...string) string {
2122 + for _, value := range values {
2123 + value = strings.TrimSpace(value)
2124 + if value != "" {
2125 + return value
2126 + }
2127 + }
2128 + return ""
2129 +}
2130 +
2131 +func cloneAnyMapForTopologyV1(in map[string]any) map[string]any {
2132 + if len(in) == 0 {
2133 + return nil
2134 + }
2135 + out := make(map[string]any, len(in))
2136 + maps.Copy(out, in)
2137 + return out
2138 +}
src/go/tools/functions-validation/README.md
+17
@@ -22,11 +22,27 @@ src/go/go.d.plugin \
22 ```
23
24 ## Validate output
25 +
26 ```
27 echo '{"status":200,"type":"table","columns":{},"data":[]}' | \
28 (cd src/go && go run ./tools/functions-validation/validate)
29 ```
30
31 +## Validate topology v1 fixtures
32 +
33 +```
34 +(cd src/go && \
35 + go run ./tools/functions-validation/validate \
36 + --schema ../plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json \
37 + --input tools/functions-validation/fixtures/topology-v1/network-connections.json)
38 +```
39 +
40 +Topology v1 validation uses the JSON Schema and additional compact-table
41 +semantic checks: decoded column lengths must match `rows`, dictionary indexes
42 +must be in range, actor/link references must point to existing rows, and
43 +correlation rules must reference existing actor/link types and point/claim key
44 +columns.
45 +
46 ## Validate output (require rows)
47 ```
48 src/go/go.d.plugin \
@@ -69,3 +85,4 @@ src/go/go.d.plugin \
85 - MSSQL uses an init container to enable Query Store and seed data.
86 - MongoDB enables the profiler to populate `system.profile`.
87 - The validator reads the canonical schema at `src/plugins.d/FUNCTION_UI_SCHEMA.json`.
88 +- Topology v1 fixtures live under `src/go/tools/functions-validation/fixtures/topology-v1/`.
src/go/tools/functions-validation/fixtures/topology-v1/network-connections.json new
+448
@@ -0,0 +1,448 @@
1 +{
2 + "status": 200,
3 + "type": "topology",
4 + "has_history": false,
5 + "update_every": 1,
6 + "data": {
7 + "schema_version": "netdata.topology.v1",
8 + "producer": {
9 + "source": "network-connections",
10 + "instance": "sample-node",
11 + "node_id": "node-a",
12 + "machine_guid": "machine-a",
13 + "agent_version": "test",
14 + "plugin": "network-viewer.plugin",
15 + "capabilities": ["topology-v1"]
16 + },
17 + "collected_at": "2026-05-09T00:00:00Z",
18 + "view": {
19 + "id": "process-connections",
20 + "scope": "process_name",
21 + "mode": "detailed",
22 + "group_by": ["process_name"]
23 + },
24 + "dictionaries": {
25 + "strings": [
26 + "self",
27 + "process",
28 + "endpoint",
29 + "node-a",
30 + "machine-a",
31 + "curl",
32 + "198.51.100.10",
33 + "socket",
34 + "192.0.2.10",
35 + "tcp",
36 + "established",
37 + "public",
38 + "ownership",
39 + "endpoint_socket",
40 + "socket_exact",
41 + "network_socket",
42 + "correlated_socket",
43 + "active"
44 + ]
45 + },
46 + "types": {
47 + "actor_types": {
48 + "self": {
49 + "layer": "node",
50 + "identity": ["id"],
51 + "merge_identity": ["machine_guid", "hostname"],
52 + "aggregation_scopes": ["node"]
53 + },
54 + "process": {
55 + "layer": "process",
56 + "identity": ["id"],
57 + "merge_identity": ["machine_guid", "process"],
58 + "aggregation_scopes": ["node", "process_name", "pid"],
59 + "presentation": {
60 + "label": "Process",
61 + "color_slot": "primary",
62 + "size": {"mode": "metric", "metric_column": "socket_count"},
63 + "ports": {
64 + "show_bullets": true,
65 + "sources": [
66 + {
67 + "source": "actor_table",
68 + "table": "socket_ports",
69 + "actor_column": "actor",
70 + "name_column": "port",
71 + "value_column": "socket_count",
72 + "default_type": "topology"
73 + }
74 + ]
75 + }
76 + }
77 + },
78 + "endpoint": {
79 + "layer": "network",
80 + "identity": ["id"],
81 + "merge_identity": ["ip", "address_space"],
82 + "aggregation_scopes": ["node"]
83 + }
84 + },
85 + "link_types": {
86 + "ownership": {
87 + "orientation": "hierarchical",
88 + "direction_role": "ownership",
89 + "aggregation": {"direction": "preserve", "evidence": "count"},
90 + "presentation": {
91 + "label": "Process ownership",
92 + "color_slot": "dim",
93 + "opacity": "faded",
94 + "line_style": "dotted",
95 + "width": "thin",
96 + "arrow": "none",
97 + "layout": {"strength": "normal", "distance": "normal"}
98 + }
99 + },
100 + "socket": {
101 + "orientation": "directed",
102 + "direction_role": "dependency",
103 + "aggregation": {
104 + "direction": "preserve",
105 + "evidence": "append",
106 + "metrics": {
107 + "socket_count": "sum",
108 + "evidence_count": "sum",
109 + "retransmissions": "sum",
110 + "rtt_ms_max": "max",
111 + "recv_rtt_ms_max": "max"
112 + }
113 + },
114 + "evidence_types": ["socket"],
115 + "presentation": {
116 + "label": "Local socket",
117 + "color_slot": "gray",
118 + "line_style": "solid",
119 + "width": "thin",
120 + "arrow": "forward",
121 + "layout": {"strength": "normal", "distance": "normal"}
122 + }
123 + },
124 + "endpoint_socket": {
125 + "orientation": "directed",
126 + "direction_role": "dependency",
127 + "aggregation": {
128 + "direction": "preserve",
129 + "evidence": "append",
130 + "metrics": {
131 + "socket_count": "sum",
132 + "evidence_count": "sum",
133 + "retransmissions": "sum",
134 + "rtt_ms_max": "max",
135 + "recv_rtt_ms_max": "max"
136 + }
137 + },
138 + "evidence_types": ["socket"],
139 + "presentation": {
140 + "label": "Endpoint connection",
141 + "color_slot": "primary",
142 + "line_style": "solid",
143 + "width": "thin",
144 + "arrow": "forward",
145 + "layout": {"strength": "normal", "distance": "normal"}
146 + }
147 + },
148 + "correlated_socket": {
149 + "orientation": "directed",
150 + "direction_role": "dependency",
151 + "aggregation": {
152 + "direction": "preserve",
153 + "evidence": "append",
154 + "metrics": {
155 + "socket_count": "sum",
156 + "evidence_count": "sum",
157 + "retransmissions": "sum",
158 + "rtt_ms_max": "max",
159 + "recv_rtt_ms_max": "max"
160 + }
161 + },
162 + "evidence_types": ["socket"],
163 + "presentation": {
164 + "label": "Correlated socket",
165 + "color_slot": "primary",
166 + "line_style": "solid",
167 + "width": "thin",
168 + "arrow": "forward",
169 + "layout": {"strength": "normal", "distance": "farthest"}
170 + }
171 + }
172 + },
173 + "port_types": {
174 + "topology": {
175 + "presentation": {
176 + "label": "Socket",
177 + "color_slot": "primary"
178 + }
179 + }
180 + },
181 + "evidence_types": {
182 + "socket": {
183 + "link_type": "socket",
184 + "role": "relationship_evidence",
185 + "match_columns": [
186 + "client_ip",
187 + "client_port",
188 + "server_ip",
189 + "server_port",
190 + "protocol"
191 + ],
192 + "columns": [
193 + {"id": "link", "type": "link_ref", "role": "reference"},
194 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
195 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
196 + {"id": "client_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
197 + {"id": "client_port", "type": "uint", "role": "group_key"},
198 + {"id": "server_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
199 + {"id": "server_port", "type": "uint", "role": "group_key"},
200 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
201 + {"id": "state", "type": "string_ref", "dictionary": "strings"}
202 + ]
203 + }
204 + },
205 + "table_types": {
206 + "socket_ports": {
207 + "role": "actor_inventory",
208 + "owner": "actor",
209 + "aggregation": "sum",
210 + "columns": [
211 + {"id": "actor", "type": "actor_ref", "role": "reference"},
212 + {"id": "port", "type": "uint", "role": "group_key"},
213 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
214 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"}
215 + ]
216 + },
217 + "connections": {
218 + "role": "relationship_summary",
219 + "owner": "link",
220 + "aggregation": "merge_metrics",
221 + "columns": [
222 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
223 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
224 + {"id": "client_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
225 + {"id": "server_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
226 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
227 + {"id": "state", "type": "string_ref", "dictionary": "strings"},
228 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"}
229 + ]
230 + }
231 + }
232 + },
233 + "presentation": {
234 + "legend": {
235 + "actors": [
236 + {"type": "self", "label": "Node"},
237 + {"type": "process", "label": "Process"},
238 + {"type": "endpoint", "label": "Correlation endpoint"}
239 + ],
240 + "links": [
241 + {"type": "ownership", "label": "Process ownership"},
242 + {"type": "endpoint_socket", "label": "Endpoint connection"},
243 + {"type": "correlated_socket", "label": "Correlated socket"}
244 + ]
245 + },
246 + "port_fields": [
247 + {"key": "type", "label": "Type"},
248 + {"key": "socket_count", "label": "Sockets"}
249 + ],
250 + "scale_keys": {
251 + "sockets": {"label": "Sockets", "unit": "count"}
252 + }
253 + },
254 + "correlation": {
255 + "rules": {
256 + "socket_exact": {
257 + "action": "absorb",
258 + "priority": 1,
259 + "key_space": "network_socket",
260 + "key": [
261 + {"column": "protocol"},
262 + {"literal": ":"},
263 + {"column": "address_space"},
264 + {"literal": ":"},
265 + {"column": "ip"},
266 + {"literal": ":"},
267 + {"column": "port"}
268 + ],
269 + "point_actor_types": ["endpoint"],
270 + "claim_actor_types": ["process"],
271 + "correlation_link_types": ["endpoint_socket"],
272 + "output_link_type": "correlated_socket"
273 + }
274 + },
275 + "points": {
276 + "rows": 1,
277 + "columns": [
278 + {"id": "actor", "type": "actor_ref", "role": "reference"},
279 + {"id": "rule", "type": "string_ref", "dictionary": "strings"},
280 + {"id": "protocol", "type": "string_ref", "dictionary": "strings"},
281 + {"id": "address_space", "type": "string_ref", "dictionary": "strings"},
282 + {"id": "ip", "type": "ip_ref", "dictionary": "strings"},
283 + {"id": "port", "type": "uint"}
284 + ],
285 + "values": [
286 + {"codec": "const", "value": 2},
287 + {"codec": "const", "value": 14},
288 + {"codec": "const", "value": 9},
289 + {"codec": "const", "value": 11},
290 + {"codec": "const", "value": 6},
291 + {"codec": "const", "value": 443}
292 + ]
293 + },
294 + "claims": {
295 + "rows": 1,
296 + "columns": [
297 + {"id": "actor", "type": "actor_ref", "role": "reference"},
298 + {"id": "rule", "type": "string_ref", "dictionary": "strings"},
299 + {"id": "protocol", "type": "string_ref", "dictionary": "strings"},
300 + {"id": "address_space", "type": "string_ref", "dictionary": "strings"},
301 + {"id": "ip", "type": "ip_ref", "dictionary": "strings"},
302 + {"id": "port", "type": "uint"}
303 + ],
304 + "values": [
305 + {"codec": "const", "value": 1},
306 + {"codec": "const", "value": 14},
307 + {"codec": "const", "value": 9},
308 + {"codec": "const", "value": 11},
309 + {"codec": "const", "value": 8},
310 + {"codec": "const", "value": 43000}
311 + ]
312 + }
313 + },
314 + "actors": {
315 + "rows": 3,
316 + "columns": [
317 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
318 + {"id": "id", "type": "string_ref", "dictionary": "strings", "role": "identity"},
319 + {"id": "machine_guid", "type": "string_ref", "dictionary": "strings", "nullable": true},
320 + {"id": "process", "type": "string_ref", "dictionary": "strings", "nullable": true},
321 + {"id": "ip", "type": "ip_ref", "dictionary": "strings", "nullable": true},
322 + {"id": "address_space", "type": "string_ref", "dictionary": "strings", "nullable": true},
323 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"}
324 + ],
325 + "values": [
326 + {"codec": "dict", "values": [0, 1, 2], "indexes": [0, 1, 2]},
327 + {"codec": "values", "values": [3, 5, 6]},
328 + {"codec": "values", "values": [4, 4, null]},
329 + {"codec": "values", "values": [null, 5, null]},
330 + {"codec": "values", "values": [null, null, 6]},
331 + {"codec": "values", "values": [null, null, 11]},
332 + {"codec": "values", "values": [2, 2, 2]}
333 + ]
334 + },
335 + "links": {
336 + "rows": 2,
337 + "columns": [
338 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
339 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
340 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
341 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "nullable": true},
342 + {"id": "state", "type": "string_ref", "dictionary": "strings", "nullable": true},
343 + {"id": "evidence_count", "type": "uint", "role": "metric", "aggregation": "sum"},
344 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"},
345 + {"id": "retransmissions", "type": "uint", "role": "metric", "aggregation": "sum"},
346 + {"id": "rtt_ms_max", "type": "float", "role": "metric", "aggregation": "max"},
347 + {"id": "recv_rtt_ms_max", "type": "float", "role": "metric", "aggregation": "max"}
348 + ],
349 + "values": [
350 + {"codec": "values", "values": [0, 1]},
351 + {"codec": "values", "values": [1, 2]},
352 + {"codec": "values", "values": [12, 13]},
353 + {"codec": "values", "values": [null, 9]},
354 + {"codec": "values", "values": [17, 10]},
355 + {"codec": "values", "values": [0, 2]},
356 + {"codec": "values", "values": [2, 2]},
357 + {"codec": "values", "values": [0, 0]},
358 + {"codec": "values", "values": [0, 5.5]},
359 + {"codec": "values", "values": [0, 4.2]}
360 + ]
361 + },
362 + "tables": {
363 + "actor": {
364 + "socket_ports": {
365 + "type": "socket_ports",
366 + "table": {
367 + "rows": 1,
368 + "columns": [
369 + {"id": "actor", "type": "actor_ref", "role": "reference"},
370 + {"id": "port", "type": "uint", "role": "group_key"},
371 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
372 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"}
373 + ],
374 + "values": [
375 + {"codec": "const", "value": 1},
376 + {"codec": "const", "value": 43000},
377 + {"codec": "const", "value": 9},
378 + {"codec": "const", "value": 2}
379 + ]
380 + }
381 + }
382 + },
383 + "relationship": {
384 + "connections": {
385 + "type": "connections",
386 + "table": {
387 + "rows": 1,
388 + "columns": [
389 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
390 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
391 + {"id": "client_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
392 + {"id": "server_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
393 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
394 + {"id": "state", "type": "string_ref", "dictionary": "strings"},
395 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"}
396 + ],
397 + "values": [
398 + {"codec": "const", "value": 1},
399 + {"codec": "const", "value": 2},
400 + {"codec": "const", "value": 8},
401 + {"codec": "const", "value": 6},
402 + {"codec": "const", "value": 9},
403 + {"codec": "const", "value": 10},
404 + {"codec": "const", "value": 2}
405 + ]
406 + }
407 + }
408 + }
409 + },
410 + "evidence": {
411 + "socket": {
412 + "type": "socket",
413 + "table": {
414 + "rows": 2,
415 + "columns": [
416 + {"id": "link", "type": "link_ref", "role": "reference"},
417 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
418 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
419 + {"id": "client_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
420 + {"id": "client_port", "type": "uint", "role": "group_key"},
421 + {"id": "server_ip", "type": "ip_ref", "dictionary": "strings", "role": "group_key"},
422 + {"id": "server_port", "type": "uint", "role": "group_key"},
423 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
424 + {"id": "state", "type": "string_ref", "dictionary": "strings"},
425 + {"id": "socket_count", "type": "uint", "role": "metric", "aggregation": "sum"}
426 + ],
427 + "values": [
428 + {"codec": "const", "value": 1},
429 + {"codec": "const", "value": 1},
430 + {"codec": "const", "value": 2},
431 + {"codec": "const", "value": 8},
432 + {"codec": "values", "values": [43000, 43001]},
433 + {"codec": "const", "value": 6},
434 + {"codec": "const", "value": 443},
435 + {"codec": "const", "value": 9},
436 + {"codec": "const", "value": 10},
437 + {"codec": "const", "value": 1}
438 + ]
439 + }
440 + }
441 + },
442 + "stats": {
443 + "actors": 3,
444 + "links": 2,
445 + "evidence_rows": 2
446 + }
447 + }
448 +}
src/go/tools/functions-validation/fixtures/topology-v1/snmp-l2.json new
+213
@@ -0,0 +1,213 @@
1 +{
2 + "status": 200,
3 + "type": "topology",
4 + "has_history": false,
5 + "update_every": 10,
6 + "data": {
7 + "schema_version": "netdata.topology.v1",
8 + "producer": {
9 + "source": "snmp-l2",
10 + "instance": "sample-fabric",
11 + "node_id": "poller-node",
12 + "machine_guid": "poller-machine",
13 + "agent_version": "test",
14 + "plugin": "go.d.plugin",
15 + "capabilities": ["topology-v1"]
16 + },
17 + "collected_at": "2026-05-09T00:00:00Z",
18 + "view": {
19 + "id": "l2-fabric",
20 + "scope": "snmp_device",
21 + "mode": "detailed",
22 + "group_by": ["snmp_device"]
23 + },
24 + "dictionaries": {
25 + "strings": [
26 + "snmp_device",
27 + "snmp_interface",
28 + "switch-a",
29 + "switch-b",
30 + "if-a-1",
31 + "if-b-1",
32 + "l2_adjacency",
33 + "lldp",
34 + "observed_bidirectional",
35 + "up",
36 + "if_inventory",
37 + "if_traffic"
38 + ]
39 + },
40 + "types": {
41 + "actor_types": {
42 + "snmp_device": {
43 + "layer": "network",
44 + "identity": ["device_id"],
45 + "merge_identity": ["device_id"],
46 + "aggregation_scopes": ["snmp_device"]
47 + },
48 + "snmp_interface": {
49 + "layer": "network",
50 + "identity": ["device_id", "if_index"],
51 + "merge_identity": ["device_id", "if_index"],
52 + "parent_identity": ["device_id"],
53 + "aggregation_scopes": ["snmp_interface"]
54 + }
55 + },
56 + "link_types": {
57 + "l2_adjacency": {
58 + "orientation": "observed_bidirectional",
59 + "direction_role": "observation",
60 + "aggregation": {
61 + "direction": "canonicalize_unordered",
62 + "evidence": "append"
63 + },
64 + "evidence_types": ["snmp_l2_observation"],
65 + "overlay_templates": ["if_traffic"]
66 + }
67 + },
68 + "evidence_types": {
69 + "snmp_l2_observation": {
70 + "link_type": "l2_adjacency",
71 + "role": "observation_evidence",
72 + "match_columns": ["local_if", "remote_if", "protocol"],
73 + "columns": [
74 + {"id": "link", "type": "link_ref", "role": "reference"},
75 + {"id": "local_if", "type": "actor_ref", "role": "reference"},
76 + {"id": "remote_if", "type": "actor_ref", "role": "reference"},
77 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
78 + {"id": "observation", "type": "string_ref", "dictionary": "strings"}
79 + ]
80 + }
81 + },
82 + "table_types": {
83 + "if_inventory": {
84 + "role": "actor_inventory",
85 + "owner": "actor",
86 + "aggregation": "set",
87 + "columns": [
88 + {"id": "actor", "type": "actor_ref", "role": "reference"},
89 + {"id": "if_name", "type": "string_ref", "dictionary": "strings"},
90 + {"id": "if_index", "type": "uint"},
91 + {"id": "oper_status", "type": "string_ref", "dictionary": "strings"},
92 + {"id": "speed_bps", "type": "uint", "unit": "bit/s", "aggregation": "max"}
93 + ]
94 + }
95 + },
96 + "overlay_templates": {
97 + "if_traffic": {
98 + "provider": "netdata.metrics",
99 + "contexts": ["net.net"],
100 + "dimensions": ["received", "sent"],
101 + "selector_params": ["node_id", "if_name"],
102 + "merge": {
103 + "refs": "set",
104 + "values": "sum"
105 + }
106 + }
107 + },
108 + "aggregation_scopes": {
109 + "snmp_device": {"columns": ["device_id"], "evidence_policy": "preserve"},
110 + "snmp_interface": {"columns": ["device_id", "if_index"], "evidence_policy": "preserve"}
111 + }
112 + },
113 + "actors": {
114 + "rows": 4,
115 + "columns": [
116 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
117 + {"id": "device_id", "type": "string_ref", "dictionary": "strings", "role": "identity"},
118 + {"id": "if_index", "type": "uint", "nullable": true, "role": "identity"},
119 + {"id": "if_name", "type": "string_ref", "dictionary": "strings", "nullable": true}
120 + ],
121 + "values": [
122 + {"codec": "dict", "values": [0, 1], "indexes": [0, 0, 1, 1]},
123 + {"codec": "values", "values": [2, 3, 2, 3]},
124 + {"codec": "values", "values": [null, null, 1, 1]},
125 + {"codec": "values", "values": [null, null, 4, 5]}
126 + ]
127 + },
128 + "links": {
129 + "rows": 1,
130 + "columns": [
131 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
132 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
133 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
134 + {"id": "evidence_count", "type": "uint", "role": "metric", "aggregation": "sum"}
135 + ],
136 + "values": [
137 + {"codec": "const", "value": 0},
138 + {"codec": "const", "value": 1},
139 + {"codec": "const", "value": 6},
140 + {"codec": "const", "value": 1}
141 + ]
142 + },
143 + "evidence": {
144 + "snmp_l2_observation": {
145 + "type": "snmp_l2_observation",
146 + "table": {
147 + "rows": 1,
148 + "columns": [
149 + {"id": "link", "type": "link_ref", "role": "reference"},
150 + {"id": "local_if", "type": "actor_ref", "role": "reference"},
151 + {"id": "remote_if", "type": "actor_ref", "role": "reference"},
152 + {"id": "protocol", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
153 + {"id": "observation", "type": "string_ref", "dictionary": "strings"}
154 + ],
155 + "values": [
156 + {"codec": "const", "value": 0},
157 + {"codec": "const", "value": 2},
158 + {"codec": "const", "value": 3},
159 + {"codec": "const", "value": 7},
160 + {"codec": "const", "value": 8}
161 + ]
162 + }
163 + }
164 + },
165 + "tables": {
166 + "actor": {
167 + "if_inventory": {
168 + "type": "if_inventory",
169 + "table": {
170 + "rows": 2,
171 + "columns": [
172 + {"id": "actor", "type": "actor_ref", "role": "reference"},
173 + {"id": "if_name", "type": "string_ref", "dictionary": "strings"},
174 + {"id": "if_index", "type": "uint"},
175 + {"id": "oper_status", "type": "string_ref", "dictionary": "strings"},
176 + {"id": "speed_bps", "type": "uint", "unit": "bit/s", "aggregation": "max"}
177 + ],
178 + "values": [
179 + {"codec": "values", "values": [2, 3]},
180 + {"codec": "values", "values": [4, 5]},
181 + {"codec": "const", "value": 1},
182 + {"codec": "const", "value": 9},
183 + {"codec": "const", "value": 1000000000}
184 + ]
185 + }
186 + }
187 + }
188 + },
189 + "overlays": {
190 + "refs": {
191 + "rows": 2,
192 + "columns": [
193 + {"id": "template", "type": "string_ref", "dictionary": "strings"},
194 + {"id": "link", "type": "link_ref", "role": "reference"},
195 + {"id": "node_id", "type": "string_ref", "dictionary": "strings"},
196 + {"id": "if_name", "type": "string_ref", "dictionary": "strings"}
197 + ],
198 + "values": [
199 + {"codec": "const", "value": 11},
200 + {"codec": "const", "value": 0},
201 + {"codec": "values", "values": [2, 3]},
202 + {"codec": "values", "values": [4, 5]}
203 + ]
204 + }
205 + },
206 + "stats": {
207 + "actors": 4,
208 + "links": 1,
209 + "evidence_rows": 1,
210 + "overlay_refs": 2
211 + }
212 + }
213 +}
src/go/tools/functions-validation/fixtures/topology-v1/streaming.json new
+137
@@ -0,0 +1,137 @@
1 +{
2 + "status": 200,
3 + "type": "topology",
4 + "has_history": false,
5 + "update_every": 1,
6 + "data": {
7 + "schema_version": "netdata.topology.v1",
8 + "producer": {
9 + "source": "streaming",
10 + "instance": "sample-stream",
11 + "node_id": "child-node",
12 + "machine_guid": "child-machine",
13 + "agent_version": "test",
14 + "plugin": "netdata",
15 + "capabilities": ["topology-v1"]
16 + },
17 + "collected_at": "2026-05-09T00:00:00Z",
18 + "view": {
19 + "id": "streaming-path",
20 + "scope": "node",
21 + "mode": "detailed",
22 + "group_by": ["node"]
23 + },
24 + "dictionaries": {
25 + "strings": [
26 + "stream_node",
27 + "child-node",
28 + "parent-node",
29 + "child-machine",
30 + "parent-machine",
31 + "stream_parent",
32 + "stream_path",
33 + "direct",
34 + "replicating"
35 + ]
36 + },
37 + "types": {
38 + "actor_types": {
39 + "stream_node": {
40 + "layer": "streaming",
41 + "identity": ["node_id"],
42 + "merge_identity": ["node_id"],
43 + "aggregation_scopes": ["node"]
44 + }
45 + },
46 + "link_types": {
47 + "stream_parent": {
48 + "orientation": "hierarchical",
49 + "direction_role": "ownership",
50 + "aggregation": {
51 + "direction": "preserve",
52 + "evidence": "append"
53 + }
54 + }
55 + },
56 + "table_types": {
57 + "stream_path": {
58 + "role": "actor_detail",
59 + "owner": "actor",
60 + "aggregation": "append",
61 + "columns": [
62 + {"id": "actor", "type": "actor_ref", "role": "reference"},
63 + {"id": "hop", "type": "uint"},
64 + {"id": "node_id", "type": "string_ref", "dictionary": "strings"},
65 + {"id": "parent_node_id", "type": "string_ref", "dictionary": "strings"},
66 + {"id": "path_kind", "type": "string_ref", "dictionary": "strings"},
67 + {"id": "since", "type": "timestamp"}
68 + ]
69 + }
70 + },
71 + "aggregation_scopes": {
72 + "node": {"columns": ["node_id"], "evidence_policy": "preserve"}
73 + }
74 + },
75 + "actors": {
76 + "rows": 2,
77 + "columns": [
78 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
79 + {"id": "node_id", "type": "string_ref", "dictionary": "strings", "role": "identity"},
80 + {"id": "machine_guid", "type": "string_ref", "dictionary": "strings"}
81 + ],
82 + "values": [
83 + {"codec": "const", "value": 0},
84 + {"codec": "values", "values": [1, 2]},
85 + {"codec": "values", "values": [3, 4]}
86 + ]
87 + },
88 + "links": {
89 + "rows": 1,
90 + "columns": [
91 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
92 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
93 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
94 + {"id": "state", "type": "string_ref", "dictionary": "strings"},
95 + {"id": "stream_count", "type": "uint", "role": "metric", "aggregation": "sum"}
96 + ],
97 + "values": [
98 + {"codec": "const", "value": 0},
99 + {"codec": "const", "value": 1},
100 + {"codec": "const", "value": 5},
101 + {"codec": "const", "value": 8},
102 + {"codec": "const", "value": 1}
103 + ]
104 + },
105 + "tables": {
106 + "actor": {
107 + "stream_path": {
108 + "type": "stream_path",
109 + "table": {
110 + "rows": 1,
111 + "columns": [
112 + {"id": "actor", "type": "actor_ref", "role": "reference"},
113 + {"id": "hop", "type": "uint"},
114 + {"id": "node_id", "type": "string_ref", "dictionary": "strings"},
115 + {"id": "parent_node_id", "type": "string_ref", "dictionary": "strings"},
116 + {"id": "path_kind", "type": "string_ref", "dictionary": "strings"},
117 + {"id": "since", "type": "timestamp"}
118 + ],
119 + "values": [
120 + {"codec": "const", "value": 0},
121 + {"codec": "const", "value": 1},
122 + {"codec": "const", "value": 1},
123 + {"codec": "const", "value": 2},
124 + {"codec": "const", "value": 7},
125 + {"codec": "const", "value": "2026-05-09T00:00:00Z"}
126 + ]
127 + }
128 + }
129 + }
130 + },
131 + "stats": {
132 + "actors": 2,
133 + "links": 1,
134 + "actor_detail_rows": 1
135 + }
136 + }
137 +}
src/go/tools/functions-validation/fixtures/topology-v1/vsphere.json new
+209
@@ -0,0 +1,209 @@
1 +{
2 + "status": 200,
3 + "type": "topology",
4 + "has_history": false,
5 + "update_every": 60,
6 + "data": {
7 + "schema_version": "netdata.topology.v1",
8 + "producer": {
9 + "source": "vsphere",
10 + "instance": "sample-vcenter",
11 + "node_id": "vcenter-node",
12 + "machine_guid": "vcenter-machine",
13 + "agent_version": "test",
14 + "plugin": "go.d.plugin",
15 + "capabilities": ["topology-v1"]
16 + },
17 + "collected_at": "2026-05-09T00:00:00Z",
18 + "view": {
19 + "id": "vsphere-inventory",
20 + "scope": "vsphere_object",
21 + "mode": "detailed",
22 + "group_by": ["vsphere_object"]
23 + },
24 + "dictionaries": {
25 + "strings": [
26 + "vsphere_datacenter",
27 + "vsphere_cluster",
28 + "vsphere_host",
29 + "vsphere_vm",
30 + "vsphere_datastore",
31 + "dc-1",
32 + "domain-c1",
33 + "host-101",
34 + "vm-501",
35 + "datastore-301",
36 + "vsphere_ownership",
37 + "vsphere_dependency",
38 + "vsphere_relationship",
39 + "vsphere_object_detail",
40 + "contains",
41 + "runs_on",
42 + "uses",
43 + "powered_on",
44 + "/dc-1/cluster-a/host-a/vm-a"
45 + ]
46 + },
47 + "types": {
48 + "actor_types": {
49 + "vsphere_datacenter": {
50 + "layer": "virtualization",
51 + "identity": ["vsphere_moid"],
52 + "merge_identity": ["vsphere_moid"],
53 + "aggregation_scopes": ["vsphere_object"]
54 + },
55 + "vsphere_cluster": {
56 + "layer": "virtualization",
57 + "identity": ["vsphere_moid"],
58 + "merge_identity": ["vsphere_moid"],
59 + "parent_identity": ["parent_moid"],
60 + "aggregation_scopes": ["vsphere_object"]
61 + },
62 + "vsphere_host": {
63 + "layer": "virtualization",
64 + "identity": ["vsphere_moid"],
65 + "merge_identity": ["vsphere_moid"],
66 + "parent_identity": ["parent_moid"],
67 + "aggregation_scopes": ["vsphere_object"]
68 + },
69 + "vsphere_vm": {
70 + "layer": "virtualization",
71 + "identity": ["vsphere_moid"],
72 + "merge_identity": ["vsphere_moid"],
73 + "parent_identity": ["parent_moid"],
74 + "aggregation_scopes": ["vsphere_object"]
75 + },
76 + "vsphere_datastore": {
77 + "layer": "storage",
78 + "identity": ["vsphere_moid"],
79 + "merge_identity": ["vsphere_moid"],
80 + "aggregation_scopes": ["vsphere_object"]
81 + }
82 + },
83 + "link_types": {
84 + "vsphere_ownership": {
85 + "orientation": "hierarchical",
86 + "direction_role": "ownership",
87 + "aggregation": {
88 + "direction": "preserve",
89 + "evidence": "append"
90 + },
91 + "evidence_types": ["vsphere_relationship"]
92 + },
93 + "vsphere_dependency": {
94 + "orientation": "directed",
95 + "direction_role": "dependency",
96 + "aggregation": {
97 + "direction": "preserve",
98 + "evidence": "append"
99 + },
100 + "evidence_types": ["vsphere_relationship"]
101 + }
102 + },
103 + "evidence_types": {
104 + "vsphere_relationship": {
105 + "link_type": "vsphere_ownership",
106 + "role": "relationship_evidence",
107 + "match_columns": ["source_moid", "target_moid", "relationship"],
108 + "columns": [
109 + {"id": "link", "type": "link_ref", "role": "reference"},
110 + {"id": "source_moid", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
111 + {"id": "target_moid", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
112 + {"id": "relationship", "type": "string_ref", "dictionary": "strings", "role": "group_key"}
113 + ]
114 + }
115 + },
116 + "table_types": {
117 + "vsphere_object_detail": {
118 + "role": "actor_detail",
119 + "owner": "actor",
120 + "aggregation": "last",
121 + "columns": [
122 + {"id": "actor", "type": "actor_ref", "role": "reference"},
123 + {"id": "power_state", "type": "string_ref", "dictionary": "strings"},
124 + {"id": "inventory_path", "type": "string_ref", "dictionary": "strings"}
125 + ]
126 + }
127 + },
128 + "aggregation_scopes": {
129 + "vsphere_object": {"columns": ["vsphere_moid"], "evidence_policy": "preserve"}
130 + }
131 + },
132 + "actors": {
133 + "rows": 5,
134 + "columns": [
135 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
136 + {"id": "vsphere_moid", "type": "string_ref", "dictionary": "strings", "role": "identity"},
137 + {"id": "parent_moid", "type": "string_ref", "dictionary": "strings", "nullable": true, "role": "parent_identity"}
138 + ],
139 + "values": [
140 + {"codec": "values", "values": [0, 1, 2, 3, 4]},
141 + {"codec": "values", "values": [5, 6, 7, 8, 9]},
142 + {"codec": "values", "values": [null, 5, 6, 7, null]}
143 + ]
144 + },
145 + "links": {
146 + "rows": 4,
147 + "columns": [
148 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
149 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
150 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
151 + {"id": "relationship", "type": "string_ref", "dictionary": "strings"},
152 + {"id": "evidence_count", "type": "uint", "role": "metric", "aggregation": "sum"}
153 + ],
154 + "values": [
155 + {"codec": "values", "values": [0, 1, 2, 3]},
156 + {"codec": "values", "values": [1, 2, 3, 4]},
157 + {"codec": "values", "values": [10, 10, 10, 11]},
158 + {"codec": "values", "values": [14, 14, 15, 16]},
159 + {"codec": "const", "value": 1}
160 + ]
161 + },
162 + "evidence": {
163 + "vsphere_relationship": {
164 + "type": "vsphere_relationship",
165 + "table": {
166 + "rows": 4,
167 + "columns": [
168 + {"id": "link", "type": "link_ref", "role": "reference"},
169 + {"id": "source_moid", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
170 + {"id": "target_moid", "type": "string_ref", "dictionary": "strings", "role": "group_key"},
171 + {"id": "relationship", "type": "string_ref", "dictionary": "strings", "role": "group_key"}
172 + ],
173 + "values": [
174 + {"codec": "values", "values": [0, 1, 2, 3]},
175 + {"codec": "values", "values": [5, 6, 7, 8]},
176 + {"codec": "values", "values": [6, 7, 8, 9]},
177 + {"codec": "values", "values": [14, 14, 15, 16]}
178 + ]
179 + }
180 + }
181 + },
182 + "tables": {
183 + "actor": {
184 + "vsphere_object_detail": {
185 + "type": "vsphere_object_detail",
186 + "table": {
187 + "rows": 1,
188 + "columns": [
189 + {"id": "actor", "type": "actor_ref", "role": "reference"},
190 + {"id": "power_state", "type": "string_ref", "dictionary": "strings"},
191 + {"id": "inventory_path", "type": "string_ref", "dictionary": "strings"}
192 + ],
193 + "values": [
194 + {"codec": "const", "value": 3},
195 + {"codec": "const", "value": 17},
196 + {"codec": "const", "value": 18}
197 + ]
198 + }
199 + }
200 + }
201 + },
202 + "stats": {
203 + "actors": 5,
204 + "links": 4,
205 + "evidence_rows": 4,
206 + "actor_detail_rows": 1
207 + }
208 + }
209 +}
src/go/tools/functions-validation/validate/main.go
+42 -15
@@ -10,6 +10,7 @@ import (
10 "os"
11 "path/filepath"
12
13 + topologyv1 "github.com/netdata/netdata/go/plugins/pkg/topology/v1"
14 "github.com/santhosh-tekuri/jsonschema/v6"
15 )
16
@@ -35,41 +36,58 @@ func main() {
36 exitErr("load input: %v", err)
37 }
38
39 + payload, err := validateJSON(schemaBytes, inputBytes)
40 + if err != nil {
41 + exitErr("%v", err)
42 + }
43 +
44 + if requireRows && minRows == 0 {
45 + minRows = 1
46 + }
47 + if minRows > 0 {
48 + rows, err := countRows(payload)
49 + if err != nil {
50 + exitErr("row check failed: %v", err)
51 + }
52 + if rows < minRows {
53 + exitErr("row check failed: expected at least %d rows, got %d", minRows, rows)
54 + }
55 + }
56 +}
57 +
58 +func validateJSON(schemaBytes, inputBytes []byte) (any, error) {
59 var payload any
60 if err := json.Unmarshal(inputBytes, &payload); err != nil {
40 - exitErr("parse input JSON: %v", err)
61 + return nil, fmt.Errorf("parse input JSON: %w", err)
62 }
63
64 var schemaDoc any
65 if err := json.Unmarshal(schemaBytes, &schemaDoc); err != nil {
45 - exitErr("parse schema JSON: %v", err)
66 + return nil, fmt.Errorf("parse schema JSON: %w", err)
67 }
68
69 compiler := jsonschema.NewCompiler()
70 if err := compiler.AddResource("schema.json", schemaDoc); err != nil {
50 - exitErr("add schema resource: %v", err)
71 + return nil, fmt.Errorf("add schema resource: %w", err)
72 }
73 schema, err := compiler.Compile("schema.json")
74 if err != nil {
54 - exitErr("compile schema: %v", err)
75 + return nil, fmt.Errorf("compile schema: %w", err)
76 }
77
78 if err := schema.Validate(payload); err != nil {
58 - exitErr("validation failed: %v", err)
79 + return nil, fmt.Errorf("validation failed: %w", err)
80 }
81
61 - if requireRows && minRows == 0 {
62 - minRows = 1
63 - }
64 - if minRows > 0 {
65 - rows, err := countRows(payload)
66 - if err != nil {
67 - exitErr("row check failed: %v", err)
68 - }
69 - if rows < minRows {
70 - exitErr("row check failed: expected at least %d rows, got %d", minRows, rows)
82 + if obj, ok := payload.(map[string]any); ok && isTopologyResponse(obj) {
83 + if data, ok := obj["data"]; ok && topologyv1.IsDecodedData(data) {
84 + if err := topologyv1.ValidateDecodedResponse(payload); err != nil {
85 + return nil, fmt.Errorf("topology validation failed: %w", err)
86 + }
87 }
88 }
89 +
90 + return payload, nil
91 }
92
93 func countRows(payload any) (int, error) {
@@ -90,6 +108,10 @@ func countRows(payload any) (int, error) {
108 return 0, fmt.Errorf("missing data field")
109 }
110
111 + if isTopologyResponse(obj) && topologyv1.IsDecodedData(data) {
112 + return topologyv1.GraphRowsFromDecodedData(data)
113 + }
114 +
115 rows, ok := data.([]any)
116 if !ok {
117 return 0, fmt.Errorf("data is not an array")
@@ -98,6 +120,11 @@ func countRows(payload any) (int, error) {
120 return len(rows), nil
121 }
122
123 +func isTopologyResponse(obj map[string]any) bool {
124 + responseType, ok := obj["type"].(string)
125 + return ok && responseType == "topology"
126 +}
127 +
128 func loadSchema(path string) ([]byte, error) {
129 if path == "" {
130 path = filepath.Clean(filepath.Join("..", "plugins.d", "FUNCTION_UI_SCHEMA.json"))
src/go/tools/functions-validation/validate/main_test.go new
+208
@@ -0,0 +1,208 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package main
4 +
5 +import (
6 + "encoding/json"
7 + "os"
8 + "path/filepath"
9 + "strings"
10 + "testing"
11 +
12 + topologyv1 "github.com/netdata/netdata/go/plugins/pkg/topology/v1"
13 +)
14 +
15 +func TestTopologyV1FixturesValidate(t *testing.T) {
16 + schemaBytes := readTestFile(t, filepath.Join("..", "..", "..", "..", "plugins.d", "FUNCTION_TOPOLOGY_SCHEMA.json"))
17 + fixtures, err := filepath.Glob(filepath.Join("..", "fixtures", "topology-v1", "*.json"))
18 + if err != nil {
19 + t.Fatalf("glob fixtures: %v", err)
20 + }
21 + if len(fixtures) != 4 {
22 + t.Fatalf("expected 4 topology fixtures, got %d: %v", len(fixtures), fixtures)
23 + }
24 +
25 + for _, fixture := range fixtures {
26 + t.Run(filepath.Base(fixture), func(t *testing.T) {
27 + inputBytes := readTestFile(t, fixture)
28 + if _, err := validateJSON(schemaBytes, inputBytes); err != nil {
29 + t.Fatalf("validate fixture: %v", err)
30 + }
31 + })
32 + }
33 +}
34 +
35 +func TestTopologyV1EnvelopeVersionValidates(t *testing.T) {
36 + schemaBytes := readTestFile(t, filepath.Join("..", "..", "..", "..", "plugins.d", "FUNCTION_TOPOLOGY_SCHEMA.json"))
37 + fixture := readTestFile(t, filepath.Join("..", "fixtures", "topology-v1", "snmp-l2.json"))
38 +
39 + var payload map[string]any
40 + if err := json.Unmarshal(fixture, &payload); err != nil {
41 + t.Fatalf("decode fixture: %v", err)
42 + }
43 + payload["v"] = float64(3)
44 +
45 + inputBytes, err := json.Marshal(payload)
46 + if err != nil {
47 + t.Fatalf("encode fixture: %v", err)
48 + }
49 + if _, err := validateJSON(schemaBytes, inputBytes); err != nil {
50 + t.Fatalf("validate fixture with envelope version: %v", err)
51 + }
52 +}
53 +
54 +func TestFunctionUISchemaValidationSkipsTopologySemanticsForTableResponses(t *testing.T) {
55 + schemaBytes := readTestFile(t, filepath.Join("..", "..", "..", "..", "plugins.d", "FUNCTION_UI_SCHEMA.json"))
56 + input := []byte(`{
57 + "status": 200,
58 + "type": "table",
59 + "columns": {
60 + "name": {"index": 0, "name": "Name", "type": "string", "visualization": "value"}
61 + },
62 + "data": [["row-a"]]
63 + }`)
64 +
65 + if _, err := validateJSON(schemaBytes, input); err != nil {
66 + t.Fatalf("validate table response: %v", err)
67 + }
68 +}
69 +
70 +func TestValidationSkipsTopologySemanticsForNonTopologyObjectData(t *testing.T) {
71 + schemaBytes := []byte(`{"type": "object"}`)
72 + input := []byte(`{
73 + "status": 200,
74 + "type": "custom",
75 + "data": {
76 + "schema_version": "netdata.topology.v1"
77 + }
78 + }`)
79 +
80 + if _, err := validateJSON(schemaBytes, input); err != nil {
81 + t.Fatalf("validate custom response: %v", err)
82 + }
83 +}
84 +
85 +func TestCountRowsUsesActorsWhenTopologyHasNoLinks(t *testing.T) {
86 + payload := decodeTestJSON(t, `{
87 + "status": 200,
88 + "type": "topology",
89 + "data": {
90 + "schema_version": "netdata.topology.v1",
91 + "dictionaries": {"strings": ["node"]},
92 + "actors": {
93 + "rows": 2,
94 + "columns": [{"id": "type", "type": "string_ref", "dictionary": "strings"}],
95 + "values": [{"codec": "const", "value": 0}]
96 + },
97 + "links": {"rows": 0, "columns": [], "values": []}
98 + }
99 + }`)
100 +
101 + rows, err := countRows(payload)
102 + if err != nil {
103 + t.Fatalf("count rows: %v", err)
104 + }
105 + if rows != 2 {
106 + t.Fatalf("expected actor rows when there are no links, got %d", rows)
107 + }
108 +}
109 +
110 +func TestCountRowsDoesNotTreatNonTopologyObjectDataAsGraph(t *testing.T) {
111 + payload := decodeTestJSON(t, `{
112 + "status": 200,
113 + "type": "custom",
114 + "data": {
115 + "schema_version": "netdata.topology.v1",
116 + "actors": {"rows": 4, "columns": [], "values": []},
117 + "links": {"rows": 1, "columns": [], "values": []}
118 + }
119 + }`)
120 +
121 + _, err := countRows(payload)
122 + if err == nil {
123 + t.Fatal("expected non-array data error")
124 + }
125 + if !strings.Contains(err.Error(), "data is not an array") {
126 + t.Fatalf("expected non-array data error, got %v", err)
127 + }
128 +}
129 +
130 +func TestTopologySemanticChecksRejectColumnLengthMismatch(t *testing.T) {
131 + payload := decodeTestJSON(t, `{
132 + "status": 200,
133 + "type": "topology",
134 + "data": {
135 + "schema_version": "netdata.topology.v1",
136 + "dictionaries": {"strings": ["node"]},
137 + "actors": {
138 + "rows": 2,
139 + "columns": [{"id": "type", "type": "string_ref", "dictionary": "strings"}],
140 + "values": [{"codec": "values", "values": [0]}]
141 + },
142 + "links": {"rows": 0, "columns": [], "values": []}
143 + }
144 + }`)
145 +
146 + err := topologyv1.ValidateDecodedResponse(payload)
147 + if err == nil {
148 + t.Fatal("expected validation error")
149 + }
150 + if !strings.Contains(err.Error(), "decoded length mismatch") {
151 + t.Fatalf("expected decoded length mismatch, got %v", err)
152 + }
153 +}
154 +
155 +func TestTopologySemanticChecksRejectOutOfBoundsActorReference(t *testing.T) {
156 + payload := decodeTestJSON(t, `{
157 + "status": 200,
158 + "type": "topology",
159 + "data": {
160 + "schema_version": "netdata.topology.v1",
161 + "dictionaries": {"strings": ["node", "depends_on"]},
162 + "actors": {
163 + "rows": 1,
164 + "columns": [{"id": "type", "type": "string_ref", "dictionary": "strings"}],
165 + "values": [{"codec": "const", "value": 0}]
166 + },
167 + "links": {
168 + "rows": 1,
169 + "columns": [
170 + {"id": "src_actor", "type": "actor_ref"},
171 + {"id": "dst_actor", "type": "actor_ref"},
172 + {"id": "type", "type": "string_ref", "dictionary": "strings"}
173 + ],
174 + "values": [
175 + {"codec": "const", "value": 0},
176 + {"codec": "const", "value": 2},
177 + {"codec": "const", "value": 1}
178 + ]
179 + }
180 + }
181 + }`)
182 +
183 + err := topologyv1.ValidateDecodedResponse(payload)
184 + if err == nil {
185 + t.Fatal("expected validation error")
186 + }
187 + if !strings.Contains(err.Error(), "actor reference out of bounds") {
188 + t.Fatalf("expected actor reference bounds error, got %v", err)
189 + }
190 +}
191 +
192 +func readTestFile(t *testing.T, path string) []byte {
193 + t.Helper()
194 + b, err := os.ReadFile(path)
195 + if err != nil {
196 + t.Fatalf("read %s: %v", path, err)
197 + }
198 + return b
199 +}
200 +
201 +func decodeTestJSON(t *testing.T, input string) any {
202 + t.Helper()
203 + var payload any
204 + if err := json.Unmarshal([]byte(input), &payload); err != nil {
205 + t.Fatalf("decode test JSON: %v", err)
206 + }
207 + return payload
208 +}
src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md new
+1322
@@ -0,0 +1,1322 @@
1 +<!-- markdownlint-disable-file MD043 -->
2 +
3 +# Netdata Topology Function Schema
4 +
5 +This document defines the production topology payload contract for Netdata
6 +Functions. It is the source of truth for new topology producers and for the
7 +Cloud topology aggregator.
8 +
9 +The JSON Schema is [FUNCTION_TOPOLOGY_SCHEMA.json](/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json).
10 +
11 +## Purpose
12 +
13 +Topology payloads describe relationships between observed entities:
14 +
15 +- infrastructure nodes and virtual machines;
16 +- containers, pods, namespaces, and workloads;
17 +- processes and sockets;
18 +- network devices, ports, VLANs, MACs, and IPs;
19 +- streaming parents and children;
20 +- storage, virtualization, and custom topology domains.
21 +
22 +The payload is optimized for two consumers:
23 +
24 +- the Cloud aggregator, which merges topology evidence across nodes and views;
25 +- the UI, which renders graph views and drilldown tables.
26 +
27 +It is not optimized for reconstructing compatibility payloads. Parity with
28 +compatibility consumers is a test concern, implemented by test-side projection
29 +code.
30 +
31 +## Non-Goals
32 +
33 +Do not put these in production topology payloads:
34 +
35 +- compatibility field names kept only for rollout adapters;
36 +- instructions for reconstructing compatibility payloads;
37 +- repeated per-row display strings or labels copied from actor/type metadata;
38 +- duplicated actor modal rows that repeat relationship evidence;
39 +- visual layout hints that the UI can own;
40 +- raw CSS, raw SVG, component names, coordinates, viewport state, or force-layout
41 + physics;
42 +- raw secrets or customer-identifying sample data in fixtures.
43 +
44 +Type-level labels, UI-owned visual tokens, safe label policies, and small
45 +column descriptions are acceptable when they help render generic topology UI.
46 +Define them once per type or column, not once per row.
47 +
48 +## Mental Model
49 +
50 +The schema has six planes:
51 +
52 +1. **Actors** are entities: nodes, processes, containers, ports, devices,
53 + vSphere objects, streaming agents, and custom objects.
54 +2. **Graph links** are the renderable relationship groups between actors.
55 +3. **Evidence** is the lossless relationship proof behind each graph link:
56 + sockets, LLDP observations, streaming path hops, vSphere inventory edges,
57 + or custom relationship facts.
58 +4. **Detail tables** are non-graph data shown in actor or relationship
59 + drilldowns. They are explicitly typed as actor-owned or relationship-owned.
60 +5. **Presentation** is backend-selected composition of UI-owned tokens for
61 + labels, colors, icons, legend, link styles, highlight behavior, and graph
62 + port bullets.
63 +6. **Overlay refs** describe how to query refreshable metrics for links or
64 + actors without recomputing topology.
65 +
66 +Graph links are intentionally smaller than evidence. A graph link may have
67 +one evidence row or tens of thousands of evidence rows.
68 +
69 +## Response Envelope
70 +
71 +Topology Functions return a normal Function envelope with `type: "topology"`:
72 +
73 +```json
74 +{
75 + "v": 3,
76 + "status": 200,
77 + "type": "topology",
78 + "has_history": false,
79 + "update_every": 5,
80 + "expires": 10,
81 + "data": {
82 + "schema_version": "netdata.topology.v1",
83 + "producer": {
84 + "source": "network-connections",
85 + "instance": "local",
86 + "node_id": "node-uuid",
87 + "machine_guid": "machine-guid",
88 + "plugin": "network-viewer.plugin"
89 + },
90 + "collected_at": "2026-05-09T10:00:00Z",
91 + "dictionaries": {
92 + "strings": ["process", "endpoint", "tcp", "inbound", "established"]
93 + },
94 + "types": {
95 + "actor_types": {},
96 + "link_types": {}
97 + },
98 + "presentation": {
99 + "selection": {
100 + "actor_click": {"mode": "highlight_connections"}
101 + },
102 + "legend": {
103 + "actors": [],
104 + "links": [],
105 + "ports": []
106 + }
107 + },
108 + "actors": {
109 + "rows": 0,
110 + "columns": [],
111 + "values": []
112 + },
113 + "links": {
114 + "rows": 0,
115 + "columns": [],
116 + "values": []
117 + }
118 + }
119 +}
120 +```
121 +
122 +`v` is the optional Function transport protocol version. It belongs to the
123 +response envelope, not to `data`, and is commonly `3` for Functions that accept
124 +POST JSON parameters.
125 +
126 +`schema_version` is the topology contract version. It is not the producer
127 +version. Producers may expose their own version in `producer.agent_version`,
128 +`producer.plugin`, or `producer.capabilities`.
129 +
130 +Function `info` responses are response metadata, not topology payloads. They
131 +may advertise `accepted_params`, `required_params`, and help text without a
132 +`data` object. Validate only full topology responses against
133 +`FUNCTION_TOPOLOGY_SCHEMA.json`.
134 +
135 +## Mode Requests
136 +
137 +Use the request parameter `__topology_mode` when a topology producer has a real
138 +detailed vs aggregated output difference. Supported values are `detailed` and
139 +`aggregated`.
140 +
141 +Do not expose a detailed/aggregated selector for mode-invariant topologies.
142 +SNMP/L2 and streaming currently emit the same topology grain for both use cases
143 +and should not advertise a mode option until a real difference exists.
144 +
145 +When a producer has a real mode split, set `data.view.supported_modes` to the
146 +available modes. When the field is absent or contains a single value, consumers
147 +must treat the topology as mode-invariant and avoid showing a mode toggle.
148 +
149 +The Cloud topology aggregator consumes detailed payloads when a producer
150 +supports the mode, even when the user asked Cloud for aggregated output. The
151 +aggregator correlates first, then aggregates the returned graph. Producers must
152 +therefore keep detailed mode lossless enough for cross-node matching.
153 +
154 +## Compact Tables
155 +
156 +All large arrays use the same compact table shape:
157 +
158 +```json
159 +{
160 + "rows": 3,
161 + "columns": [
162 + {"id": "type", "type": "string_ref", "dictionary": "strings"},
163 + {"id": "socket_count", "type": "uint", "aggregation": "sum"}
164 + ],
165 + "values": [
166 + {"codec": "dict", "values": [0, 1], "indexes": [0, 0, 1]},
167 + {"codec": "values", "values": [10, 4, 19]}
168 + ]
169 +}
170 +```
171 +
172 +`columns` and `values` are parallel arrays. The Nth value encoding belongs to
173 +the Nth column. Every decoded column must produce exactly `rows` values.
174 +
175 +Supported codecs:
176 +
177 +- `const`: one value repeated for all rows;
178 +- `values`: one value per row;
179 +- `dict`: a per-column value dictionary plus integer indexes.
180 +
181 +Column values may be plain scalars or references into a global dictionary.
182 +For string-heavy columns, prefer `string_ref` with `dictionary: "strings"` and
183 +integer values. This is how production socket evidence reached about 22 raw
184 +bytes per socket evidence row in the measured corpus.
185 +
186 +Use column type `json` only for actor/custom detail cells that must preserve
187 +nested producer-owned data, such as SNMP port `neighbors` or `vlans`. Do not
188 +use `json` for high-cardinality relationship evidence when a typed scalar,
189 +reference, or array column can carry the fact.
190 +
191 +Go topology producers should use `src/go/pkg/topology/v1` for the
192 +`netdata.topology.v1` response model and compact-table constructors. The helper
193 +validates row counts, parallel `columns`/`values` lengths, dictionary indexes,
194 +and JSON round-trip behavior before a producer response reaches the Function
195 +transport.
196 +
197 +## Required Actor Semantics
198 +
199 +The `actors` table must carry enough canonical identity for aggregation. Actor
200 +row indexes are used as link endpoints inside the same payload.
201 +
202 +Required actor columns:
203 +
204 +- a type column, normally `type`;
205 +- a layer column, normally `layer`;
206 +- every identity column declared by `types.actor_types.<type>.identity`.
207 +
208 +Common actor identity columns:
209 +
210 +- `node_id`
211 +- `machine_guid`
212 +- `hostname`
213 +- `ip`
214 +- `mac`
215 +- `process_name`
216 +- `pid`
217 +- `container_id`
218 +- `container_name`
219 +- `pod`
220 +- `namespace`
221 +- `k8s_label:<name>`
222 +- `vsphere_moid`
223 +- `vsphere_inventory_path`
224 +
225 +Actor types define source-local identity and cross-source merge identity:
226 +
227 +```json
228 +{
229 + "types": {
230 + "actor_types": {
231 + "process": {
232 + "layer": "process",
233 + "identity": ["node_id", "process_name"],
234 + "merge_identity": ["node_id", "process_name"],
235 + "parent_identity": ["node_id"],
236 + "aggregation_scopes": ["node", "process_name", "container", "k8s_workload"],
237 + "search": {
238 + "enabled": true,
239 + "columns": ["display_name", "process_name"],
240 + "label_keys": ["cmdline", "username"]
241 + },
242 + "presentation": {
243 + "label": "Process",
244 + "role": "actor",
245 + "icon": "process",
246 + "color_slot": "primary",
247 + "border": {"enabled": true},
248 + "size": {"mode": "link_count", "scale": "normal"},
249 + "layout": {"repulsion": "normal"},
250 + "label_policy": {
251 + "columns": ["display_name", "process_name"],
252 + "fallback": "type_label",
253 + "max_length": 80,
254 + "array": "reject"
255 + },
256 + "ports": {
257 + "show_bullets": true,
258 + "sources": [
259 + {
260 + "source": "actor_table",
261 + "table": "socket_ports",
262 + "actor_column": "actor",
263 + "name_column": "port",
264 + "value_column": "socket_count",
265 + "default_type": "topology"
266 + }
267 + ]
268 + }
269 + }
270 + }
271 + }
272 + }
273 +}
274 +```
275 +
276 +`identity` is what makes an actor unique in the producer payload.
277 +`merge_identity` is what the Cloud aggregator may use across payloads.
278 +`parent_identity` expresses containment or "lives on" relationships.
279 +
280 +Do not use display names as identity. Identity is canonical matching data, not
281 +UI text. If an identity column is not explicitly listed in the actor type
282 +`presentation.label_policy.columns`, the UI must not use it as a label
283 +fallback.
284 +
285 +Actor type `search` declares exactly what the graph search bar may index for
286 +that actor type. `search.columns[]` references actor-table scalar columns.
287 +`search.label_keys[]` references values in the actor label table, normally
288 +`actor_labels.key`. Set `search.enabled: false` for helper actors that should
289 +not appear in graph search, such as synthetic segment or grouping actors. The
290 +UI must not traverse producer-specific `details`, `match`, `attributes`, or
291 +labels paths when rendering a v1 payload.
292 +
293 +## Required Link Semantics
294 +
295 +The `links` table is the renderable graph projection. It must contain:
296 +
297 +- `src_actor`: actor row index;
298 +- `dst_actor`: actor row index;
299 +- `type`: link type id;
300 +- enough counters or summary metrics for a graph view, such as
301 + `evidence_count` or `socket_count`.
302 +
303 +Link rows are not required to be one-to-one with raw observations. They should
304 +usually be grouped by graph identity. Evidence rows preserve the details.
305 +
306 +Link types define direction and aggregation semantics:
307 +
308 +```json
309 +{
310 + "types": {
311 + "link_types": {
312 + "socket": {
313 + "orientation": "directed",
314 + "direction_role": "flow",
315 + "semantic_role": "traffic",
316 + "aggregation": {
317 + "direction": "preserve",
318 + "evidence": "append",
319 + "metrics": {
320 + "socket_count": "sum",
321 + "rtt_ms_max": "max"
322 + }
323 + },
324 + "evidence_types": ["socket"],
325 + "presentation": {
326 + "label": "Socket",
327 + "color_slot": "primary",
328 + "line_style": "solid",
329 + "width": "normal",
330 + "curve": "auto",
331 + "arrow": "forward",
332 + "layout": {
333 + "strength": "normal",
334 + "distance": "normal"
335 + },
336 + "variable": {
337 + "channel": "width",
338 + "scale_key": "sockets",
339 + "value_column": "socket_count",
340 + "min": "normal",
341 + "max": "emphasis"
342 + }
343 + }
344 + },
345 + "l2_adjacency": {
346 + "orientation": "undirected",
347 + "direction_role": "observation",
348 + "aggregation": {
349 + "direction": "canonicalize_unordered",
350 + "evidence": "append"
351 + },
352 + "evidence_types": ["snmp_l2_observation"]
353 + }
354 + }
355 + }
356 +}
357 +```
358 +
359 +Direction rules:
360 +
361 +- `directed` links preserve source and destination order;
362 +- `undirected` links may be canonicalized by sorting endpoints;
363 +- `hierarchical` links express ownership or containment;
364 +- `observed_bidirectional` links preserve observation completeness without
365 + implying traffic direction.
366 +
367 +`direction_role` tells the aggregator and UI what direction means:
368 +
369 +- `flow`: traffic or socket direction;
370 +- `dependency`: logical dependency direction;
371 +- `ownership`: parent to child or owner to owned;
372 +- `observation`: discovery direction only;
373 +- `none`: direction is not meaningful.
374 +
375 +When `types.link_types.<id>.presentation.arrow` is `auto` or omitted, the UI
376 +derives arrows from `orientation` and `direction_role`:
377 +
378 +- `undirected` -> no arrow;
379 +- `observed_bidirectional` -> no arrow;
380 +- `direction_role: none` -> no arrow;
381 +- `direction_role: observation` -> no arrow;
382 +- `directed` with `flow` or `dependency` -> forward from `src_actor` to
383 + `dst_actor`;
384 +- `hierarchical` with `ownership` -> forward from `src_actor` to `dst_actor`;
385 +- all other combinations -> no arrow and a diagnostic if the combination is
386 + schema-valid but semantically unusual.
387 +
388 +`observed_bidirectional` does not mean draw arrows at both ends. Producers that
389 +need reverse or both arrows must set `presentation.arrow` explicitly.
390 +
391 +`direction_role` is required. A link type with `orientation: "directed"` and no
392 +`direction_role` is invalid v1 and must not get an inferred arrow from `auto`.
393 +Declare `direction_role: "flow"` or `direction_role: "dependency"` for directed
394 +links that should infer a forward arrow, or set `presentation.arrow`
395 +explicitly.
396 +
397 +For valid values, the `auto` diagnostic boundary is:
398 +
399 +- normal: `directed+flow`, `directed+dependency`,
400 + `hierarchical+ownership`, `undirected+none`, `undirected+observation`,
401 + `observed_bidirectional+none`, `observed_bidirectional+observation`;
402 +- diagnostic: `directed+none`, `directed+observation`, `directed+ownership`,
403 + `hierarchical+none`, `hierarchical+flow`, `hierarchical+dependency`,
404 + `hierarchical+observation`, `undirected+flow`, `undirected+dependency`,
405 + `undirected+ownership`, `observed_bidirectional+flow`,
406 + `observed_bidirectional+dependency`, `observed_bidirectional+ownership`.
407 +
408 +`semantic_role` is an optional topology-agnostic link classification used by
409 +the UI and aggregator when behavior is not just visual:
410 +
411 +- `normal`: ordinary relationship;
412 +- `discovery`: discovery-protocol relationship such as LLDP/CDP;
413 +- `ownership`: graph-coherence or containment relationship;
414 +- `traffic`: traffic, socket, or data-path relationship;
415 +- `correlation`: visible correlation or partial-correlation relationship;
416 +- `control`: control-plane relationship.
417 +
418 +Do not infer this role from link type names, protocols, or labels. If a link
419 +needs discovery filtering, ownership treatment, traffic emphasis, or
420 +correlation treatment, the producer must declare the semantic role explicitly.
421 +
422 +## Presentation Plane
423 +
424 +Presentation is compact and backend-controlled, but it is not raw frontend
425 +layout. The UI owns token meanings; producers only choose from documented
426 +tokens.
427 +
428 +Type definitions carry their own presentation:
429 +
430 +- actor types choose label, role, icon token, fill color token, border,
431 + annotation ring, size policy, safe label policy, and graph port-bullet policy;
432 +- link types choose label, color token, line style, width token, curve token,
433 + arrow token, tokenized layout strength/distance, and one optional variable
434 + visual channel;
435 +- port types choose label, color token, and opacity token.
436 +
437 +The graph-level `data.presentation` object carries definitions that are not
438 +owned by one type: legend order, actor-click highlight behavior, port tooltip
439 +field labels, and scale-key labels.
440 +
441 +Graph-level and type-level presentation are complementary. If both exist, there
442 +is no precedence rule to apply: type-level presentation styles actor/link/port
443 +types, while `data.presentation` describes cross-type behavior.
444 +
445 +Example:
446 +
447 +```json
448 +{
449 + "types": {
450 + "port_types": {
451 + "topology": {
452 + "presentation": {
453 + "label": "Socket",
454 + "color_slot": "primary",
455 + "opacity": "normal"
456 + }
457 + }
458 + }
459 + },
460 + "presentation": {
461 + "profile_version": "network-connections.v1",
462 + "selection": {
463 + "actor_click": {
464 + "mode": "highlight_connections"
465 + }
466 + },
467 + "legend": {
468 + "actors": [
469 + {"type": "process", "label": "Process"}
470 + ],
471 + "links": [
472 + {"type": "socket", "label": "Socket"}
473 + ],
474 + "ports": [
475 + {"type": "topology", "label": "Socket"}
476 + ]
477 + },
478 + "port_fields": [
479 + {"key": "type", "label": "Type"}
480 + ],
481 + "scale_keys": {
482 + "sockets": {
483 + "label": "Sockets",
484 + "unit": "count"
485 + }
486 + }
487 + }
488 +}
489 +```
490 +
491 +Safe label policy is mandatory for polished aggregated graphs. It prevents
492 +canonical identity arrays such as many MAC addresses from becoming actor names.
493 +Use only human-safe scalar columns in `label_policy.columns`; set
494 +`fallback: "type_label"` unless showing row numbers is explicitly useful.
495 +
496 +Link variable scaling is intentionally raw. The producer gives one numeric
497 +`value_column` and one `scale_key`; Cloud or the UI scales visible links that
498 +share the same key. `min` and `max` are visual tokens for the selected channel:
499 +width scaling uses width tokens and opacity scaling uses opacity tokens. Do not
500 +pre-scale to pixels or opacity in the producer.
501 +
502 +Link layout is also tokenized. Producers may set
503 +`types.link_types.<id>.presentation.layout.strength` and `.distance` for any
504 +link type. These are relative hints for the UI force layout, not raw physics
505 +values. The allowed strength tokens are `weakest`, `weaker`, `normal`,
506 +`stronger`, and `strongest`. The allowed distance tokens are `closest`,
507 +`closer`, `normal`, `farther`, and `farthest`.
508 +
509 +Current Netdata topology tuning keeps `strength` at `normal` and varies only
510 +`distance` where a topology needs semantic separation. Do not use non-normal
511 +`strength` tokens for graph polish unless a later product decision explicitly
512 +re-enables force-strength tuning.
513 +
514 +Actor layout is tokenized separately from link layout. Producers may set
515 +`types.actor_types.<id>.presentation.layout.repulsion` to `weakest`, `weaker`,
516 +`normal`, `stronger`, or `strongest`. This controls the relative separation of
517 +actors of that type in the UI force graph. It is not interchangeable with link
518 +`strength`: actor repulsion pushes nodes apart, while link strength pulls link
519 +endpoints together. Producers must not emit raw charge or force numbers.
520 +Initial UI-owned mappings are `weakest=-200`, `weaker=-300`, `normal=-450`,
521 +`stronger=-700`, and `strongest=-1000`; these numbers are not schema and may be
522 +tuned after visual QA.
523 +
524 +Actor size supports:
525 +
526 +- `fixed`: no data-driven size changes;
527 +- `link_count`: size may reflect graph degree;
528 +- `metric`: size comes from a numeric actor row column named by
529 + `metric_column`.
530 +
531 +Actor size may also set `scale: "compact"`, `"normal"`, or `"emphasized"` for
532 +type-level fixed visual emphasis. Use this for deliberate distinctions such as
533 +self/current node emphasis. Do not force the UI to infer emphasis from actor
534 +type names or labels. Initial UI-owned mappings are `compact=0.85`,
535 +`normal=1.0`, and `emphasized=1.18`; these numbers are not schema and may be
536 +tuned after visual QA. `size.scale` composes with `size.mode`; it does not
537 +override data-driven sizing.
538 +
539 +When `selection.actor_click.mode` is `highlight_path`, the payload must also
540 +set `path_table`, `path_actor_column`, and `path_order_column`. The path table
541 +is an actor detail table. The actor column must contain path-member `actor_ref`
542 +values and the order column must be numeric so the UI can render the path
543 +deterministically. If the table carries paths for multiple clicked actors, also
544 +set `path_owner_column` to an `actor_ref` column that identifies the actor whose
545 +click should use that row.
546 +
547 +Port bullets are graph presentation, not modal/table composition. Producers
548 +must define `ports.sources[]` for actor types that set `show_bullets: true`.
549 +Each source says where bullets come from:
550 +
551 +- `links`: read from the graph links table;
552 +- `evidence`: read from a named evidence type or evidence section;
553 +- `actor_table`: read from a named actor detail table when present.
554 +
555 +Each source must define `actor_column` and `name_column`. Optional
556 +`value_column`, `type_column`, `status_column`, `mode_column`, `role_column`,
557 +and `sources_column` enrich bullet multiplicity, color, and tooltip fields.
558 +`value_column` must reference a numeric source-table column; the UI sums it per
559 +bullet key and uses the sum for visible bullet count, overflow, and sizing. Use
560 +it when an aggregated row represents several observations, such as several
561 +sockets on the same process port. `default_type` must reference
562 +`types.port_types` and is used when `type_column` is absent or empty.
563 +`name_column` must reference a scalar display column, not raw actor/link/
564 +evidence references, arrays, or JSON cells. For evidence sources, `evidence`
565 +names an evidence type id.
566 +
567 +`hover.fields` is intentionally lightweight graph hover metadata. Full modal
568 +and table composition lives in `types.actor_types.<id>.presentation.modal`,
569 +`types.link_types.<id>.presentation.modal`, and optional
570 +`types.table_types.<id>.presentation`.
571 +
572 +`annotation` is a small actor marker such as a ring or dot. It must use closed
573 +color slots and style tokens; it is not a free-form badge or CSS hook.
574 +
575 +Do not emit raw SVG icons. Use the closed UI icon token vocabulary and request
576 +a new UI token when a producer needs a new visual concept.
577 +
578 +### Closed Token Vocabulary
579 +
580 +Color slots:
581 +
582 +`primary`, `secondary`, `accent`, `self`, `neutral`, `muted`, `dim`, `derived`,
583 +`info`, `structural`, `warning`, `success`, `danger`, `blue`, `green`,
584 +`orange`, `purple`, `cyan`, `yellow`, `teal`, `gray`.
585 +
586 +Prefer semantic slots such as `primary`, `warning`, `derived`, or `structural`
587 +when they fit. Use hue slots only when a topology needs distinct categories that
588 +do not map to the semantic slots, such as the vSphere migration.
589 +
590 +Opacity tokens:
591 +
592 +`normal`, `muted`, `faded`.
593 +
594 +Width tokens:
595 +
596 +`thin`, `normal`, `thick`, `emphasis`.
597 +
598 +Icon tokens:
599 +
600 +`router`, `switch`, `firewall`, `access_point`, `server`, `storage`,
601 +`load_balancer`, `printer`, `phone`, `ups`, `camera`, `process`, `agent`,
602 +`netdata-agent`, `parent`, `remote-endpoint`, `local-endpoint`, `segment`,
603 +`self`, `ip`, `cloud`, `container`, `vm`, `database`, `service`, `datacenter`,
604 +`cluster`, `host`, `network`, `datastore`, `datastore_cluster`,
605 +`resource_pool`, `device`, `endpoint`, `correlation`, `interface`, `group`,
606 +`unknown`.
607 +
608 +Producers must not emit tokens outside the schema. The UI should render
609 +unsupported tokens with a safe fallback and record diagnostics for version skew.
610 +Producers must not emit raw SVG icons or ask the UI to infer icons from
611 +capability strings. If a topology needs a new visual concept, add a closed icon
612 +token to the schema and UI token map first.
613 +
614 +## Evidence Plane
615 +
616 +Evidence sections are keyed by evidence type. Each evidence row must reference
617 +the graph link it supports using a `link_ref` column.
618 +
619 +Example socket evidence type:
620 +
621 +```json
622 +{
623 + "types": {
624 + "evidence_types": {
625 + "socket": {
626 + "link_type": "socket",
627 + "role": "relationship_evidence",
628 + "match_columns": [
629 + "client_ip",
630 + "client_port",
631 + "server_ip",
632 + "server_port",
633 + "protocol"
634 + ],
635 + "columns": [
636 + {"id": "link", "type": "link_ref", "role": "reference"},
637 + {"id": "src_actor", "type": "actor_ref", "role": "reference"},
638 + {"id": "dst_actor", "type": "actor_ref", "role": "reference"},
639 + {
640 + "id": "client_ip",
641 + "type": "ip_ref",
642 + "dictionary": "strings",
643 + "role": "group_key"
644 + },
645 + {"id": "client_port", "type": "uint", "role": "group_key"},
646 + {
647 + "id": "server_ip",
648 + "type": "ip_ref",
649 + "dictionary": "strings",
650 + "role": "group_key"
651 + },
652 + {"id": "server_port", "type": "uint", "role": "group_key"},
653 + {
654 + "id": "protocol",
655 + "type": "string_ref",
656 + "dictionary": "strings",
657 + "role": "group_key"
658 + },
659 + {"id": "namespace", "type": "string_ref", "dictionary": "strings"},
660 + {"id": "rtt_ms_max", "type": "float", "unit": "ms", "aggregation": "max"}
661 + ]
662 + }
663 + }
664 + }
665 +}
666 +```
667 +
668 +Evidence rows are the lossless plane for topology relationships. If Cloud must
669 +cross-match sockets across nodes, every socket evidence row must remain
670 +available. The graph link can be highly aggregated while the evidence rows stay
671 +one-per-observation.
672 +
673 +## Detail Tables
674 +
675 +Detail tables support actor modals and drilldowns.
676 +
677 +There are four roles:
678 +
679 +- `relationship_evidence`: exact relationship rows, usually backed by an
680 + evidence section;
681 +- `relationship_summary`: aggregated relationship rows derived from evidence;
682 +- `actor_detail`: actor-owned custom data that is not generally aggregatable;
683 +- `actor_inventory`: actor-owned inventory data that may be appended or set
684 + merged.
685 +
686 +Streaming `stream_path` is actor detail. It must not be treated as a link
687 +evidence table unless each row is also a relationship proof.
688 +
689 +Some actor-detail data is intentionally producer-specific and not generally
690 +aggregatable. Use table role `actor_detail` with aggregation `append` or
691 +`none`; use column type `json` only for cells that need nested objects or
692 +arrays that cannot be represented as scalar columns.
693 +
694 +Example actor custom table type:
695 +
696 +```json
697 +{
698 + "types": {
699 + "table_types": {
700 + "stream_path": {
701 + "role": "actor_detail",
702 + "owner": "actor",
703 + "aggregation": "append",
704 + "columns": [
705 + {"id": "actor", "type": "actor_ref", "role": "reference"},
706 + {"id": "hop", "type": "uint"},
707 + {"id": "node_id", "type": "string_ref", "dictionary": "strings"},
708 + {"id": "since", "type": "timestamp"}
709 + ]
710 + }
711 + }
712 + }
713 +}
714 +```
715 +
716 +Do not duplicate evidence in actor-owned tables. If a modal needs a socket list
717 +for an actor, derive it from socket evidence by filtering evidence rows whose
718 +link touches that actor.
719 +
720 +## Actor Labels And Modal Composition
721 +
722 +Actor modals are composed from existing topology facts. They do not get a
723 +second copy of actor attributes, socket rows, SNMP endpoint objects, or
724 +relationship evidence only for display.
725 +
726 +Every actor modal has four top-level entities:
727 +
728 +- actor name from the actor row through `presentation.label_policy`;
729 +- actor labels from an actor-owned `actor_labels` table when labels exist;
730 +- a depth-1 topology miniature built from existing incident links and opposite
731 + actors;
732 +- one or more table sections built from actors, links, evidence, or detail
733 + tables.
734 +
735 +Use a compact actor-owned label table for display labels and metadata:
736 +
737 +```json
738 +{
739 + "types": {
740 + "table_types": {
741 + "actor_labels": {
742 + "role": "actor_inventory",
743 + "owner": "actor",
744 + "aggregation": "set",
745 + "columns": [
746 + {"id": "actor", "type": "actor_ref", "role": "reference"},
747 + {"id": "key", "type": "string_ref", "dictionary": "strings"},
748 + {"id": "value", "type": "string_ref", "dictionary": "strings"},
749 + {
750 + "id": "source",
751 + "type": "string_ref",
752 + "dictionary": "strings",
753 + "nullable": true
754 + },
755 + {
756 + "id": "kind",
757 + "type": "string_ref",
758 + "dictionary": "strings",
759 + "nullable": true
760 + },
761 + {"id": "value_index", "type": "uint", "nullable": true}
762 + ]
763 + }
764 + }
765 + },
766 + "tables": {
767 + "actor": {
768 + "actor_labels": {
769 + "type": "actor_labels",
770 + "table": {"rows": 0, "columns": [], "values": []}
771 + }
772 + }
773 + }
774 +}
775 +```
776 +
777 +The `key`, `value`, `source`, and `kind` columns may use either `string` or
778 +`string_ref` encoding. Producers should prefer `string_ref` when a local
779 +dictionary is already used, but aggregators and UI adapters must treat both
780 +encodings as the same logical label fields.
781 +
782 +Host/node actors should expose the complete host label set when available.
783 +Non-node actors should expose all useful producer-known labels and metadata,
784 +such as process command line, user, group, namespace, interface role, or
785 +virtualization object properties. If a fact is needed for identity,
786 +correlation, grouping, sorting, filtering, or aggregation, keep it as a typed
787 +canonical actor/evidence/detail column too; `actor_labels` is not a replacement
788 +for canonical data.
789 +
790 +Repeated label values are repeated rows with the same `actor` and `key`,
791 +ordered by `value_index`. Do not encode repeated labels as raw JSON arrays for
792 +normal modal display.
793 +
794 +`actor_labels` inherits the topology Function sensitive-data classification.
795 +Labels may include command lines, users, host labels, system contact/location
796 +fields, or other operator-controlled metadata. Cloud services, aggregators, and
797 +UI adapters that consume topology payloads must apply the same access-control
798 +assumptions as the source Function.
799 +
800 +Use `modal.labels.identification.fields[]` to select the small ordered subset
801 +of label keys that belongs in the actor modal identification/header area. This
802 +selection references the existing `actor_labels` table; it must not duplicate
803 +values into a separate modal-only table. Missing selected keys are skipped, and
804 +the full Labels tab remains complete.
805 +
806 +Modal sections are recipes over existing tables:
807 +
808 +```json
809 +{
810 + "types": {
811 + "actor_types": {
812 + "process": {
813 + "layer": "process",
814 + "identity": ["node_id", "process_name"],
815 + "presentation": {
816 + "label": "Process",
817 + "label_policy": {
818 + "columns": ["display_name", "process_name"],
819 + "fallback": "type_label",
820 + "array": "reject"
821 + },
822 + "modal": {
823 + "labels": {
824 + "enabled": true,
825 + "table": "actor_labels",
826 + "identification": {
827 + "fields": [
828 + {"key": "process", "label": "Process", "max_values": 1},
829 + {"key": "username", "label": "User", "max_values": 1}
830 + ]
831 + }
832 + },
833 + "mini_topology": {
834 + "enabled": true,
835 + "depth": 1,
836 + "exclude_link_types": ["ownership"]
837 + },
838 + "sections": [
839 + {
840 + "id": "connections",
841 + "label": "Connections",
842 + "source": {"kind": "links"},
843 + "owner_filter": {
844 + "mode": "incident_link",
845 + "src_actor_column": "src_actor",
846 + "dst_actor_column": "dst_actor"
847 + },
848 + "row_filters": [
849 + {"column": "type", "op": "not_in", "values": ["ownership"]}
850 + ],
851 + "columns": [
852 + {
853 + "id": "remote",
854 + "label": "Remote",
855 + "projection": {
856 + "kind": "opposite_actor",
857 + "src_actor_column": "src_actor",
858 + "dst_actor_column": "dst_actor"
859 + },
860 + "cell": "actor_link"
861 + },
862 + {
863 + "id": "protocol",
864 + "label": "Protocol",
865 + "projection": {"kind": "direct", "column": "protocol"},
866 + "cell": "badge"
867 + },
868 + {
869 + "id": "sockets",
870 + "label": "Sockets",
871 + "projection": {"kind": "direct", "column": "socket_count"},
872 + "cell": "number"
873 + }
874 + ]
875 + }
876 + ]
877 + }
878 + }
879 + }
880 + }
881 + }
882 +}
883 +```
884 +
885 +Table recipes must support these source kinds:
886 +
887 +- `actors`;
888 +- `links`;
889 +- `evidence` with an `evidence` id;
890 +- `actor_table` with a `table` id;
891 +- `relationship_table` with a `table` id.
892 +
893 +Use `owner_filter` to bind rows to the selected actor or link. Common filters
894 +are `actor_column`, `link_column`, `incident_link`, `incident_evidence`, and
895 +`selected_link`.
896 +
897 +Use projections instead of duplicated display fields:
898 +
899 +- `direct`: read a column from the source row;
900 +- `actor_ref_label`: render an actor-ref column through actor label policy;
901 +- `opposite_actor`: render the opposite endpoint of a link row;
902 +- `formatted_endpoint`: combine IP, port, and optional protocol columns;
903 +- `selected_side_endpoint`: choose local or remote endpoint fields based on
904 + whether the selected actor matches the source or destination actor columns;
905 + producers must provide `src_actor_column` and `dst_actor_column` plus at
906 + least one local endpoint column and one remote endpoint column;
907 +- `label_lookup`: read a value from `actor_labels` by `label_key`; omit
908 + `actor_column` to look up labels for the selected modal actor, or provide an
909 + actor-ref source column when the lookup belongs to another actor in the row;
910 +- `coalesce`: choose the first non-empty column;
911 +- `json_path`: extract a declared scalar `path` from a declared JSON `column`;
912 +- `const`: emit a fixed value.
913 +
914 +Use cell types to keep rendering generic and polished:
915 +
916 +- `text`;
917 +- `number`;
918 +- `badge`;
919 +- `actor_link`;
920 +- `timestamp`;
921 +- `duration`;
922 +- `endpoint`;
923 +- `array_count`;
924 +- `debug_json`.
925 +
926 +Use visibility annotations instead of duplicating rows:
927 +
928 +- `table`: shown in the normal table;
929 +- `expanded`: hidden from the main grid but shown when the row expands;
930 +- `hidden`: available for joins, sorting, or future use, not displayed;
931 +- `debug`: diagnostic-only. Raw JSON belongs here unless a curated scalar
932 + projection exists.
933 +
934 +`json` columns are allowed only when they preserve nested producer-owned facts
935 +that the UI or aggregator understands through declared projections, or when
936 +they are explicitly marked as `debug_json`. They are not acceptable as the
937 +normal user-facing rendering for labels, endpoint objects, neighbor arrays, or
938 +actor attributes.
939 +
940 +Use `empty_label` for the section-level empty-state label. Use column
941 +`badge_map` only for explicit value-to-token mapping, and keep `align` and
942 +`sortable` as presentation hints over already-projected columns. These fields
943 +must not add new data or embed UI components.
944 +
945 +The Cloud aggregator should preserve and merge modal/table definitions by the
946 +same namespace/deduplicate rules used for type presentation. It should not
947 +materialize modal rows during aggregation unless it is already merging the
948 +underlying canonical table.
949 +
950 +## Telemetry Overlays
951 +
952 +Topology should be refreshable without recomputing topology. Overlay templates
953 +define how the UI or Cloud can query metrics for an actor or link.
954 +
955 +Templates live once in the type registry. Overlay refs carry only template ids
956 +and parameters.
957 +
958 +Example:
959 +
960 +```json
961 +{
962 + "types": {
963 + "overlay_templates": {
964 + "snmp_interface_traffic": {
965 + "provider": "netdata.metrics",
966 + "contexts": ["snmp.interface_traffic"],
967 + "dimensions": ["received", "sent"],
968 + "selector_params": ["node_id", "interface_id"],
969 + "merge": {
970 + "refs": "set",
971 + "values": "sum"
972 + }
973 + }
974 + }
975 + },
976 + "overlays": {
977 + "refs": {
978 + "rows": 1,
979 + "columns": [
980 + {"id": "owner_kind", "type": "string_ref", "dictionary": "strings"},
981 + {"id": "owner", "type": "link_ref"},
982 + {"id": "template", "type": "string_ref", "dictionary": "strings"},
983 + {"id": "node_id", "type": "string_ref", "dictionary": "strings"},
984 + {"id": "interface_id", "type": "string_ref", "dictionary": "strings"}
985 + ],
986 + "values": []
987 + }
988 + }
989 +}
990 +```
991 +
992 +Overlay refs are optional. Do not fabricate per-link bandwidth if the producer
993 +does not have it. For network sockets, current evidence may include snapshot
994 +metrics such as RTT or retransmissions, but that is not a time-series overlay
995 +unless a query provider exists.
996 +
997 +## Aggregation Rules
998 +
999 +Aggregation must be schema-driven:
1000 +
1001 +- actor type identity defines what actors can merge;
1002 +- link type direction policy defines whether edge direction is preserved;
1003 +- evidence type match columns define which details must remain exact;
1004 +- column aggregation rules define how summaries are produced.
1005 +
1006 +Common column rules:
1007 +
1008 +- `set`: preserve unique values;
1009 +- `sum`: add counters;
1010 +- `min` / `max`: preserve bounds;
1011 +- `last` / `first`: pick by observation order;
1012 +- `count`: count rows;
1013 +- `none`: not aggregatable.
1014 +
1015 +If a table or column has no valid aggregation, mark it `none` and keep rows
1016 +attached to their owner. Do not silently drop rows.
1017 +
1018 +## Correlation Plane
1019 +
1020 +Correlation is producer-visible graph semantics, not aggregator state. Producers
1021 +declare how independently produced topology maps can be correlated by keys, but
1022 +they do not expose internal aggregator states such as candidate, absorbed,
1023 +rewrite plan, or equivalence class in final payloads.
1024 +
1025 +Correlation can resolve several shapes:
1026 +
1027 +- loose relationship sides, where one side of a detailed row has endpoint facts
1028 + but no known actor;
1029 +- visible correlation actors, where the input graph intentionally materializes
1030 + unresolved peers;
1031 +- weaker placeholder actors that should be replaced by stronger managed actors;
1032 +- equivalent actors that should be merged and enriched with facts from multiple
1033 + payloads.
1034 +
1035 +Use `data.correlation` when a topology can resolve correlation actors across
1036 +payloads:
1037 +
1038 +- `rules` defines named correlation rules, priority, key space, key template,
1039 + rule class, action, point actor types when visible points exist, optional
1040 + claim actor types, correlation link types, and the final output link type;
1041 +- `points` is a compact table of visible correlation actors and their keys;
1042 +- `claims` is a compact table of real actors and the keys they satisfy.
1043 +
1044 +Correlation keys are declarative. A key is built from table columns and string
1045 +literals. The aggregator normalizes values by column type and concatenates the
1046 +parts; it does not execute code and does not need to know what IP, port, MAC,
1047 +chassis id, vSphere MOID, or another domain value means.
1048 +
1049 +Supported rule classes:
1050 +
1051 +- `resolve_loose_side`: resolve a loose relationship side or visible
1052 + correlation actor to a claim actor when the key is unambiguous;
1053 +- `replace_actor`: remove weaker placeholder actors and rewire incident
1054 + relationships to stronger managed actors;
1055 +- `merge_enrich_actor`: merge actors with the same declared identity and merge
1056 + their labels, attributes, evidence, and detail tables by table policy.
1057 +
1058 +Supported rule actions remain the visible output behavior:
1059 +
1060 +- `absorb`: on an exact unambiguous match, matching correlation actors are
1061 + removed from the aggregated output, or loose-side placeholders are consumed,
1062 + and incident correlation relationships are rewired to the matched real actor
1063 + using `output_link_type`;
1064 +- `link`: on an unambiguous broader/partial match, the correlation actor remains
1065 + visible, or a materialized partial actor remains visible, and a weak
1066 + correlation link to the matched actor is emitted using `output_link_type`.
1067 +
1068 +No match keeps the correlation actor visible. Ambiguous matches must stay
1069 +unresolved and produce diagnostics in the aggregator; producers must not encode
1070 +guessing policy in the schema.
1071 +
1072 +NAT or other alias evidence can be represented by adding more point or claim
1073 +rows for the same actor and rule. The original key remains intact; aliases are
1074 +additional facts, not mutations of the source observation.
1075 +
1076 +Links between real actors and visible correlation actors must use semantic
1077 +correlation link types even in a single-node topology. The UI and aggregator
1078 +must not infer correlation behavior from actor type names or topology kind. The
1079 +legend should include visible correlation actor and link types so users can
1080 +distinguish unresolved, partial, inferred, and resolved relationships.
1081 +
1082 +## Network Connections Shape
1083 +
1084 +For network-connections, graph links must be split into semantic families:
1085 +
1086 +- node-to-process ownership links that keep the graph together;
1087 +- resolved process-to-process links where both process endpoints are already
1088 + known;
1089 +- process-to-correlation-endpoint links for unresolved or cross-node socket
1090 + endpoints.
1091 +
1092 +Socket evidence preserves exact tuples.
1093 +
1094 +Canonical socket evidence columns:
1095 +
1096 +- graph link reference;
1097 +- source actor reference;
1098 +- destination actor reference;
1099 +- client IP;
1100 +- client port;
1101 +- server IP;
1102 +- server port;
1103 +- protocol;
1104 +- TCP state when available;
1105 +- network namespace or address-space tags;
1106 +- optional snapshot metrics such as RTT, receive RTT, retransmissions, and
1107 + socket count.
1108 +
1109 +Do not emit these per socket:
1110 +
1111 +- repeated display strings or labels;
1112 +- `port_name` if it can be derived from port;
1113 +- duplicated endpoint objects;
1114 +- actor labels repeated as evidence labels;
1115 +- actor modal socket rows.
1116 +
1117 +In the measured Cloud corpus, this production-only socket evidence shape was
1118 +about 7.25 MB raw for 323,077 socket evidence rows, or about 11.25 MB raw when
1119 +including current RTT/retransmission metrics.
1120 +
1121 +Network-connections graph direction is dependency direction: link types use
1122 +`direction_role: "dependency"`, the client actor is always `src_actor`, and the
1123 +server actor is always `dst_actor`. The topology payload must not expose a
1124 +separate `local` direction. Local host or same-node sockets still resolve to
1125 +either inbound or outbound dependency direction, and same-node process pairs
1126 +should emit resolved process-to-process links when both actors are known.
1127 +
1128 +For outbound sockets, the process claims the client `protocol + client_ip +
1129 +client_port` tuple and the correlation endpoint points at the server `protocol +
1130 +server_ip + server_port` tuple. For inbound sockets, the process claims the
1131 +server tuple and the correlation endpoint points at the client tuple. Listening
1132 +sockets have no remote correlation point.
1133 +The current `socket_exact` rule key is `protocol + address_space + ip + port`;
1134 +`address_space` prevents private or otherwise scoped endpoint identities from
1135 +matching across unrelated address domains.
1136 +
1137 +Network-connections uses four link types with different graph semantics:
1138 +
1139 +- `endpoint_socket`: process-to-correlation-endpoint links. These are the
1140 + primary unresolved network dependencies in a single-node view. They should be
1141 + solid, colored, thin, normal-strength, and normal-distance so unresolved
1142 + endpoints do not force the graph to zoom out.
1143 +- `correlated_socket`: aggregator output after exact endpoint absorption. These
1144 + are cross-payload process-to-process dependencies and should be
1145 + normal-strength and farthest so independent topology clusters do not blend
1146 + into one dense layout.
1147 +- `socket`: resolved local process-to-process socket links. These are gray,
1148 + thin, normal-distance links whose width can vary by `socket_count`.
1149 +- `ownership`: node-to-process containment links. These are graph-coherence
1150 + links, not network traffic. They should be dotted, faded/dim, thin, normal
1151 + strength, and normal distance.
1152 +
1153 +Aggregated network-connections payloads still need process port bullets without
1154 +shipping detailed socket evidence. Emit an `actor_inventory` table such as
1155 +`socket_ports` with `actor`, `port`, and numeric `socket_count`, then point the
1156 +process actor type's `presentation.ports.sources[]` at that actor table with
1157 +`value_column: "socket_count"`. Size process actors with
1158 +`presentation.size: {"mode": "metric", "metric_column": "socket_count"}`.
1159 +
1160 +Network-connections actor modals should expose dependency semantics, not every
1161 +internal table as a peer tab:
1162 +
1163 +- self/node actors: `Processes` over `links`, filtered to `type == ownership`;
1164 +- non-node actors in aggregated mode: `Dependencies` and `Dependants` over
1165 + `tables.relationship.connections`;
1166 +- non-node actors in detailed mode: `Dependencies` and `Dependants` over
1167 + `evidence.socket`;
1168 +- `Dependencies` filters rows where the selected actor is `src_actor`;
1169 +- `Dependants` filters rows where the selected actor is `dst_actor`.
1170 +
1171 +Do not show `socket_ports` as a normal network-connections modal section. It is
1172 +only the graph port-bullet inventory. Put less common per-connection fields such
1173 +as retransmissions or receiver RTT behind `visibility: "expanded"` instead of
1174 +creating another duplicate tab.
1175 +
1176 +## Streaming Shape
1177 +
1178 +For streaming:
1179 +
1180 +- actors are Netdata agents or streaming endpoints;
1181 +- links are parent/child or replication relationships;
1182 +- direction is meaningful as hierarchy/ownership, not packet flow;
1183 +- `stream_path` is an actor-detail table, not relationship evidence unless a
1184 + row proves a specific relationship.
1185 +
1186 +Streaming custom tables are allowed because they carry actor-owned state that
1187 +cannot be derived from links.
1188 +
1189 +Streaming parent actor size should use the actor row `retained_node_count`
1190 +metric: `presentation.size:
1191 +{"mode":"metric","metric_column":"retained_node_count"}`. Do not use generic
1192 +graph degree or direct child count for parent sizing; the operational question
1193 +is how many nodes have data retained by the parent, including self, virtual
1194 +nodes, stale nodes, and transit descendants when they have DB retention state.
1195 +Parent graph bullets should be derived from incoming streaming links by using a
1196 +`ports.sources[]` entry over `links` with `actor_column: "dst_actor"` and a
1197 +scalar child/node display column such as `port_name`.
1198 +
1199 +Streaming modal identification should be role-specific. Host-like actors
1200 +(`parent`, `child`, and `stale`) should use compact status plus OS/hardware/
1201 +platform labels from `actor_labels`: hostname, node type, health, stream,
1202 +ingest, OS, OS version, kernel, architecture, CPU, cores, RAM, virtualization,
1203 +container, cloud placement, and Agent version. Parents also include retained
1204 +nodes and direct children. Vnode actors should use inventory labels such as
1205 +vnode type, vendor, model, address, location, sys object id, LLDP name, and
1206 +status. Keep long identifiers such as machine GUID and node id available in the
1207 +full Labels tab instead of promoting them into the header.
1208 +
1209 +Streaming detail tables can contain stable node or Cloud identifiers. These
1210 +fields must not be used as graph labels and must not be copied into logs,
1211 +diagnostics, docs, SOWs, or durable review artifacts without redaction.
1212 +
1213 +## SNMP/L2 Shape
1214 +
1215 +For SNMP/L2:
1216 +
1217 +- actors are devices, interfaces, VLANs, bridge domains, and endpoints;
1218 +- links are L2 adjacencies, containment, forwarding evidence, and ownership;
1219 +- adjacency links are usually `undirected` or `observed_bidirectional`;
1220 +- discovery direction is observation metadata and should not prevent
1221 + aggregation of the same physical adjacency;
1222 +- FDB/ARP/LLDP/CDP rows are evidence sections or actor inventory tables,
1223 + depending on whether each row proves a relationship.
1224 +
1225 +Managed SNMP device actor modals are port-centric:
1226 +
1227 +- use `modal.labels.identification.fields[]` to show key device labels such as
1228 + device name, management IP, vendor, model, port counts, and LLDP/CDP counts
1229 + in the modal identification area;
1230 +- use `actor_ports` as the primary `Ports` section;
1231 +- expose real port identity columns, including SNMP `if_index` as the visible
1232 + numeric port ID when known, source `port_id`, display `name`, `if_name`,
1233 + `if_descr`, `if_alias`, MAC, speed, status, mode, role, VLAN, FDB, link, and
1234 + neighbor counts;
1235 +- never invent numeric port IDs. Do not use row order or any generated sequence;
1236 + `if_index` must come from the device/SNMP facts;
1237 +- include compact expanded-row neighbor columns such as nullable
1238 + `neighbor_actor` and `neighbor_port_name` when graph-link facts can align the
1239 + port to a remote actor;
1240 +- use `actor_port_links` as the `Port Neighbors` section when device modal
1241 + rows need remote actor/port/evidence details;
1242 +- keep endpoint, segment, and custom actors on a generic graph-link `Links`
1243 + section when they do not own port inventory.
1244 +
1245 +`actor_port_links` is an actor-owned modal index over existing graph links and
1246 +evidence. It may duplicate compact references and side-specific port facts so
1247 +the UI can show a device's local port next to its remote actor, but it must not
1248 +duplicate raw LLDP/CDP/FDB/ARP/STP evidence JSON. Every row should include the
1249 +selected actor, link reference, remote actor, local `if_index`, local port name,
1250 +remote port facts, protocol, link type, state, evidence count, confidence,
1251 +inference, attachment mode, and timestamps when known.
1252 +
1253 +SNMP endpoint port names must come only from real port fields: `port_name`,
1254 +`if_name`, `if_descr`, or source `port_id`. Do not fall back to actor labels
1255 +such as `display_name` or `sys_name`; those belong in actor labels, not in
1256 +local/remote port cells.
1257 +
1258 +SNMP interface traffic, packets, errors, and state should use overlay
1259 +templates with compact refs to node/context/label selectors.
1260 +
1261 +## vSphere Shape
1262 +
1263 +For vSphere:
1264 +
1265 +- actors should use stable vSphere managed object ids when available;
1266 +- inventory path can be an additional merge identity but should not be the
1267 + only identity when a stable object id exists;
1268 +- containment links such as datacenter -> cluster -> host -> VM are
1269 + `hierarchical` with `direction_role: "ownership"`;
1270 +- VM-to-host relationships are topology facts, not metric overlays;
1271 +- metrics for datastore, host, or VM utilization should be overlay templates
1272 + when the UI needs refreshable values.
1273 +
1274 +The vSphere producer in the separate worktree must be updated in place only
1275 +after coordination with the user because another agent may be editing it.
1276 +
1277 +## Test Reconstruction
1278 +
1279 +Compatibility tests may project the new canonical payload into rollout adapter
1280 +shapes to prove that no information needed by compatibility consumers was lost.
1281 +
1282 +That projection code must live only in tests or local schema-lab tools. It may
1283 +hardcode adapter presentation fields, display strings, modal table shapes, and
1284 +compatibility object layouts. None of that belongs in production payloads.
1285 +
1286 +The acceptance check is:
1287 +
1288 +1. decode the new canonical payload;
1289 +2. derive the compatibility shape in test code;
1290 +3. compare against sanitized fixtures or expected compatibility behavior.
1291 +
1292 +Do not add fields to production payloads solely to make this projection easier.
1293 +
1294 +## Producer Checklist
1295 +
1296 +Before shipping a topology producer:
1297 +
1298 +- define actor, link, evidence, table, and overlay types;
1299 +- choose actor identities that survive aggregation;
1300 +- separate graph links from evidence rows;
1301 +- mark direction semantics explicitly in link types;
1302 +- classify custom tables as actor detail, actor inventory,
1303 + relationship evidence, or relationship summary;
1304 +- expose host/node labels in an actor-owned `actor_labels` table when
1305 + available;
1306 +- expose useful non-node actor labels and metadata in `actor_labels`, while
1307 + keeping identity, correlation, grouping, sorting, filtering, and aggregation
1308 + facts as typed canonical columns;
1309 +- define actor/link modal sections as recipes over existing actors, links,
1310 + evidence, and detail tables;
1311 +- migrate old `show_port_bullets` to
1312 + `types.actor_types.<id>.presentation.ports.show_bullets` and define
1313 + `ports.sources[]` so bullet data is not implicit;
1314 +- use compact tables for high-cardinality sections;
1315 +- use `src/go/pkg/topology/v1` helpers for Go producers;
1316 +- avoid per-row display strings and duplicate labels;
1317 +- keep raw JSON out of normal user-facing modal tables unless a section marks
1318 + it as explicit debug output;
1319 +- include only canonical facts and optional metrics the producer really has;
1320 +- validate with [FUNCTION_TOPOLOGY_SCHEMA.json](/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json);
1321 +- run payload-size measurements on realistic or captured fixtures;
1322 +- keep raw real-environment payloads under `.local/` only.
src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md new
+527
@@ -0,0 +1,527 @@
1 +<!-- markdownlint-disable-file MD043 -->
2 +
3 +# Topology Schema Implementation Scope
4 +
5 +This document scopes the work needed to move Netdata topology producers,
6 +Cloud aggregation, and the Cloud UI to the production topology schema defined
7 +in [FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md](/src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md)
8 +and [FUNCTION_TOPOLOGY_SCHEMA.json](/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json).
9 +
10 +It is not an implementation plan for one commit. It is the work map for the
11 +backend, frontend, producer, and aggregator changes.
12 +
13 +## Ground Rules
14 +
15 +- New topology producers emit only the new schema.
16 +- Superseded topology schema support is removed from Agent/backend contracts
17 + and docs.
18 +- Temporary compatibility support may exist only as an isolated Cloud frontend
19 + adapter during Agent rollout.
20 +- Production payloads carry canonical topology facts, not reconstruction
21 + instructions for compatibility payloads.
22 +- Actor/link modals are composed from schema-declared recipes over existing
23 + actors, links, evidence, detail tables, and actor labels. Production payloads
24 + must not duplicate high-cardinality rows only for modal display.
25 +- Test-only reconstruction/projection code may derive older shapes to prove
26 + information parity, but that code must not affect production payloads.
27 +- Raw payload captures from real systems stay under `.local/` and are never
28 + committed.
29 +
30 +## Shared Backend Work
31 +
32 +### Function Contract
33 +
34 +Required changes:
35 +
36 +- add topology validation against `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`;
37 +- update Function validator tooling to recognize the new topology schema;
38 +- make topology Function examples and tests use the new contract;
39 +- remove superseded topology-schema references from Agent/backend docs once producer
40 + migration lands.
41 +
42 +Likely files:
43 +
44 +- `src/go/tools/functions-validation/`
45 +- `src/plugins.d/FUNCTION_UI_REFERENCE.md`
46 +- `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`
47 +- `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`
48 +- `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`
49 +
50 +### Shared Encoding Helpers
51 +
52 +The schema uses compact columnar tables. Producers should not hand-roll table
53 +encoding repeatedly.
54 +
55 +Required helpers:
56 +
57 +- table builder for `rows` / `columns` / `values`;
58 +- codecs for `const`, `values`, and `dict`;
59 +- string dictionary builder;
60 +- validation checks for column/value length;
61 +- deterministic sorting helpers for actors, links, and evidence rows;
62 +- size measurement hooks for tests.
63 +
64 +Likely homes:
65 +
66 +- Go: `src/go/pkg/topology/` or `src/go/pkg/funcapi/`
67 +- C: small helper module for network-viewer, or a local builder until a shared
68 + C helper is justified
69 +- Rust: SDK helper if a Rust topology producer is added
70 +
71 +## Current Migration Inventory
72 +
73 +### Agent Producers
74 +
75 +`topology:network-connections`:
76 +
77 +- producer path: `src/collectors/network-viewer.plugin/network-viewer.c`;
78 +- the Function now emits `netdata.topology.v1` at
79 + `src/collectors/network-viewer.plugin/network-viewer.c:2535`;
80 +- the Function parses `aggregated` / `mode:aggregated` and `detailed` /
81 + `mode:detailed`, with aggregated as the default, at
82 + `src/collectors/network-viewer.plugin/network-viewer.c:272`;
83 +- response metadata exposes the `mode` selector at
84 + `src/collectors/network-viewer.plugin/network-viewer.c:1451`;
85 +- actors, graph links, and optional socket evidence rows are emitted as compact
86 + columnar tables at `src/collectors/network-viewer.plugin/network-viewer.c:2568`;
87 +- socket evidence is emitted only in detailed mode at
88 + `src/collectors/network-viewer.plugin/network-viewer.c:2571`;
89 +- repeated string columns use automatic dictionary encoding when it is smaller
90 + than plain values at `src/collectors/network-viewer.plugin/network-viewer.c:2041`;
91 +- old-schema presentation metadata and actor-nested socket tables have been
92 + removed from the Agent producer. The v1 producer now emits compact
93 + graph-presentation metadata inside type definitions plus `data.presentation`.
94 + Actor modal socket lists must be derived from evidence by the Cloud
95 + frontend/aggregator during rollout.
96 +- modal-composition producer work now emits `actor_labels`, process
97 + `username`, process `cmdline`, self `local_ip_count`, socket-port inventory,
98 + and modal recipes. Remaining work is integrated UI/aggregator QA.
99 +
100 +`topology:streaming`:
101 +
102 +- producer paths: `src/web/api/functions/function-topology-streaming.c` and
103 + `src/streaming/stream-path.c`;
104 +- the Function now emits `netdata.topology.v1` directly at
105 + `src/web/api/functions/function-topology-streaming.c:1870`;
106 +- actors, graph links, link evidence, and actor-detail tables are emitted as
107 + compact tables at `src/web/api/functions/function-topology-streaming.c:1912`;
108 +- streaming path rows are preserved as an `actor_detail` table at
109 + `src/web/api/functions/function-topology-streaming.c:1255`;
110 +- inbound and outbound drilldown rows are declared as relationship summaries at
111 + `src/web/api/functions/function-topology-streaming.c:1259`;
112 +- streaming, virtual, and stale links have explicit directed link-type metadata
113 + and separate evidence type ids at
114 + `src/web/api/functions/function-topology-streaming.c:1226`;
115 +- modal-composition producer work now emits `actor_labels`, complete host labels
116 + where available, host/system metadata labels, OS/architecture/CPU fields, link
117 + metric columns, and modal recipes. Remaining streaming work is parity/UX
118 + validation with the Cloud frontend and Cloud aggregator once those parallel
119 + workers are ready.
120 +
121 +`topology:snmp`:
122 +
123 +- producer paths: `src/go/plugin/go.d/collector/snmp_topology/` and
124 + `src/go/pkg/topology/`;
125 +- the Function handler now adapts the current SNMP topology snapshot to
126 + `netdata.topology.v1` through
127 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`, while the
128 + deeper engine still builds the older shared Go topology model;
129 +- the older shared Go topology data model stores top-level actor/link arrays at
130 + `src/go/pkg/topology/types.go:167`;
131 +- older actor tables are nested under actor rows at
132 + `src/go/pkg/topology/types.go:24`;
133 +- the older link model carries `Direction`, `Src`, `Dst`, and metrics at
134 + `src/go/pkg/topology/types.go:42`;
135 +- the older presentation table metadata still uses `Source` only at
136 + `src/go/pkg/topology/types.go:72`, with SNMP ports/links examples at
137 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_presentation_schema.go:29`;
138 +- current L2 emission uses directions such as `bidirectional` and
139 + `unidirectional` at `src/go/pkg/topology/engine/topology_adapter_segments_builder_emit.go:60`
140 + and `src/go/pkg/topology/engine/topology_adapter_projection_pairs.go:230`;
141 +- migration target: use `observed_bidirectional` or unordered aggregation policy
142 + where discovery direction is noise, preserve LLDP/CDP/FDB/ARP/STP evidence,
143 + keep interface inventory as actor detail/inventory, and move metric query
144 + definitions to overlay templates/refs.
145 +
146 +vSphere:
147 +
148 +- producer path is in a separate PR worktree and must be updated in place only
149 + after telling the user;
150 +- migration target: use stable managed object ids for actor identity, model
151 + containment as hierarchical/ownership links, model VM-host/network/datastore
152 + relationships as graph links plus evidence where needed, and expose
153 + utilization/state through overlay templates/refs.
154 +
155 +### Cloud Frontend
156 +
157 +The Cloud frontend compatibility work is outside this repository, but the
158 +schema rollout depends on it:
159 +
160 +- current topology fetch normalizer decodes every topology payload through
161 + `normalizeTopologyPayload(response?.data || {})` and then computes render-time
162 + aggregated links at `${CLOUD_FRONTEND_REPO}/src/domains/functions/useFetch/normalizers/topology/index.js:9`;
163 +- current frontend graph aggregation groups by source, target, and link type,
164 + canonicalizing reverse links if already seen, at
165 + `${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/graphAggregation.js:58`;
166 +- current actor modal code still branches on presentation table `source` values,
167 + including `source === "links"`, at
168 + `${CLOUD_FRONTEND_REPO}/src/domains/functions/components/topology/actorModal/index.js:286`;
169 +- migration target: add a new-schema decoder for compact tables, keep old-schema
170 + support isolated in one temporary adapter, derive actor drilldown relationship
171 + tables from evidence rows, render actor custom tables from typed actor-detail
172 + tables, and use link-type direction metadata instead of guessing from raw link
173 + direction strings.
174 +- zero-heuristic v1 rendering target: read actor size scale, actor repulsion,
175 + actor search policy, link semantic role, and closed icon tokens from the v1
176 + type registry. Keep `isSelfNode`, `isDerivedSegmentNode`, `isDeviceNode`,
177 + LLDP/CDP protocol checks, capability icon inference, and hardcoded search
178 + paths inside the temporary legacy adapter only.
179 +
180 +## Producer Migration Scope
181 +
182 +### `topology:network-connections`
183 +
184 +Producer path:
185 +
186 +- `src/collectors/network-viewer.plugin/network-viewer.c`
187 +
188 +Required behavior:
189 +
190 +- emit actors as compact actor table rows;
191 +- emit graph links as three semantic families:
192 + - node-to-process ownership links that keep each node cluster together;
193 + - local process-to-process links when both process endpoints are known;
194 + - process-to-correlation-endpoint links for unresolved or cross-node socket
195 + endpoints;
196 +- emit pure correlation endpoint actors plus `data.correlation.points` and
197 + `data.correlation.claims` rows for socket tuple resolution;
198 +- emit one socket evidence row per socket tuple needed for cross-node matching;
199 +- default to aggregated graph projection while preserving detailed evidence;
200 +- support aggregation scopes prepared for node, process name, PID, container,
201 + and Kubernetes workload labels as enrichment becomes available;
202 +- omit compatibility per-row display strings, duplicated labels, and actor
203 + modal socket tables from production payload;
204 +- keep current metrics optional and separate from topology identity.
205 +
206 +Validation:
207 +
208 +- compare against captured corpus under `.local/`;
209 +- prove no truncation on large socket counts;
210 +- assert payload size at corpus scale;
211 +- assert exact reverse-tuple matching inputs remain present.
212 +
213 +Current state:
214 +
215 +- `src/collectors/network-viewer.plugin/network-viewer.c` now emits compact
216 + actor rows, graph-link rows, and optional socket evidence rows directly in
217 + `netdata.topology.v1`;
218 +- aggregated mode is the default and omits socket evidence from the response;
219 +- detailed mode keeps socket evidence as a shared relationship-evidence table,
220 + not as actor-owned duplicated modal data;
221 +- link and evidence string columns choose dictionary encoding only when it
222 + reduces raw payload size;
223 +- SOW-0023 semantic-link split and correlation endpoint/point/claim emission
224 + are implemented in the Agent producer;
225 +- remaining network-connections work is corpus-scale validation with captured
226 + Cloud payloads and Cloud/frontend integration once the parallel workers are
227 + ready.
228 +
229 +### `topology:streaming`
230 +
231 +Producer paths:
232 +
233 +- `src/web/api/functions/function-topology-streaming.c`
234 +- `src/streaming/stream-path.c`
235 +
236 +Required behavior:
237 +
238 +- emit streaming agents as actors;
239 +- emit parent/child streaming relationships as directed dependency links;
240 +- classify `stream_path` as actor detail, not relationship evidence;
241 +- keep retention and relationship summaries as typed detail tables;
242 +- make direction semantics explicit through link type definitions.
243 +
244 +Validation:
245 +
246 +- preserve current actor modal data through new actor-detail tables;
247 +- prove graph links and custom actor tables are not conflated;
248 +- use fixtures from current streaming topology tests where possible.
249 +
250 +Current state:
251 +
252 +- `src/web/api/functions/function-topology-streaming.c` now emits
253 + `netdata.topology.v1` directly from the C Function;
254 +- actor rows, link rows, relationship evidence, `stream_path`, `retention`,
255 + `inbound`, and `outbound` tables are compact columnar sections;
256 +- stale stream-path hops remain signed values instead of being coerced to
257 + unsigned values;
258 +- streaming, virtual, and stale graph-link types have separate evidence types
259 + so link-type metadata and evidence metadata agree.
260 +- streaming graph-presentation metadata is emitted inside type definitions plus
261 + `data.presentation`, including highlight-path selection, legend, link styles,
262 + and graph port-bullet tokens.
263 +
264 +### `topology:snmp`
265 +
266 +Producer paths:
267 +
268 +- `src/go/plugin/go.d/collector/snmp_topology/`
269 +- `src/go/pkg/topology/engine/`
270 +
271 +Required behavior:
272 +
273 +- emit devices, interfaces, bridge domains, VLANs, and endpoints as actor rows;
274 +- emit L2 adjacencies with direction policy `canonicalize_unordered` when
275 + direction is discovery noise;
276 +- preserve LLDP/CDP/FDB/ARP/STP facts as evidence or actor inventory depending
277 + on role;
278 +- move interface traffic/errors/state metric pointers to overlay templates and
279 + overlay refs;
280 +- avoid copying metric query fragments on every link.
281 +
282 +Validation:
283 +
284 +- reuse existing SNMP topology golden fixtures;
285 +- add schema-level golden fixtures for devices, interfaces, ports, and
286 + bidirectional adjacency merge;
287 +- verify overlay refs can query interface metrics without recomputing topology.
288 +
289 +Current state:
290 +
291 +- initial Function payload migration is implemented through a v1 adapter in
292 + `src/go/plugin/go.d/collector/snmp_topology/func_topology_v1.go`;
293 +- the adapter emits compact actor, link, evidence, actor metadata, and
294 + actor-detail tables and preserves nested custom actor cells with `json`
295 + columns where needed;
296 +- modal-composition producer work now emits `actor_labels`, promoted
297 + scalar/count actor fields, stable `actor_ports` rows, structured endpoint
298 + evidence, and modal recipes. Remaining SNMP work is to migrate metric lookup
299 + fragments into first-class overlay templates/refs instead of only preserving
300 + them in actor/detail data, plus integrated UI/aggregator QA.
301 +
302 +### vSphere Topology
303 +
304 +Producer worktree:
305 +
306 +- the separate vSphere topology worktree used for that PR
307 +
308 +Required behavior:
309 +
310 +- update the vSphere topology producer to the new schema in place;
311 +- use stable vSphere managed object ids as actor identity where available;
312 +- model inventory containment with hierarchical ownership links;
313 +- represent VM-to-host, cluster-to-host, datastore, and network relationships
314 + as graph links plus typed evidence where needed;
315 +- use overlay templates for refreshable utilization/state metrics.
316 +
317 +Coordination constraint:
318 +
319 +- do not edit the vSphere worktree before telling the user, because another
320 + agent is working in that directory.
321 +
322 +## Cloud Frontend Scope
323 +
324 +Required changes:
325 +
326 +- add a decoder for the compact table schema;
327 +- build graph nodes from the actors table;
328 +- build graph edges from the links table;
329 +- derive actor drilldown relationship tables from evidence rows;
330 +- render actor custom tables from typed actor-detail tables;
331 +- decode and execute `presentation.modal` recipes for actor/link modals;
332 +- render `actor_labels` as actor labels instead of raw metadata JSON;
333 +- reuse existing topology modal/table components where practical, extending
334 + them for v1 projections rather than building a separate v1 table stack;
335 +- use link type direction metadata to decide whether links are directed,
336 + undirected, hierarchical, or observation-only;
337 +- use overlay templates and refs for metric refreshes;
338 +- isolate compatibility support in one temporary adapter;
339 +- delete the temporary adapter after Agent rollout.
340 +
341 +Likely frontend areas:
342 +
343 +- topology payload normalizer;
344 +- graph aggregation layer;
345 +- actor modal tables;
346 +- link details;
347 +- telemetry overlay query layer;
348 +- Function response version detection.
349 +
350 +Frontend risks:
351 +
352 +- decoding large columnar sections synchronously can still block the main
353 + thread; use streaming, workers, or chunked decode if needed;
354 +- mixed Agent versions need clear adapter selection;
355 +- actor modal tables must not duplicate evidence in memory unnecessarily.
356 +- v1 actor modals can regress visually if they bypass the existing table,
357 + port-table, labels, and navigation components. Component reuse is part of the
358 + frontend migration, not just a cleanup preference.
359 +
360 +## Cloud Aggregator Scope
361 +
362 +The aggregator should be implemented in Go as a separate Cloud component or
363 +service, not inside charts-service request routing.
364 +
365 +The MVP aggregator must support all topology kinds covered by the production
366 +schema contract. `topology:network-connections` remains the required
367 +high-cardinality benchmark, but it is not an acceptable production boundary by
368 +itself. The Cloud UI should not need separate aggregation paths for different
369 +topology kinds.
370 +
371 +### Inputs
372 +
373 +- one or more `netdata.topology.v1` payloads;
374 +- requested aggregation scope, such as node, process name, container,
375 + Kubernetes workload labels, vSphere object type, or SNMP device/interface;
376 +- optional filters such as layer, link type, actor type, room, or node set.
377 +
378 +### Outputs
379 +
380 +- a `netdata.topology.v1` payload with:
381 + - merged actor rows;
382 + - merged graph links;
383 + - resolved correlation output as normal actors and links, with no exposed
384 + aggregator internal states;
385 + - preserved or counted evidence rows according to schema policy;
386 + - merged detail tables according to table type policy;
387 + - preserved and remapped modal/table presentation recipes;
388 + - merged actor labels according to actor table policy;
389 + - merged overlay refs according to overlay template policy;
390 + - stats describing input rows, output rows, evidence rows, and drops/errors.
391 +
392 +### Core Packages
393 +
394 +Suggested package split:
395 +
396 +- `schema`: generated or hand-written Go structs for the topology schema;
397 +- `codec`: compact table decode/encode helpers;
398 +- `model`: canonical in-memory actors, links, evidence, tables, overlays;
399 +- `aggregate`: scope-based actor/link/evidence merge logic;
400 +- `match`: declarative correlation-key normalization, priority handling, exact
401 + and partial match resolution, and exact tuple matching;
402 +- `validate`: schema and semantic validation;
403 +- `fixtures`: sanitized corpus and synthetic scale fixtures.
404 +
405 +### Aggregation Logic
406 +
407 +Required behavior:
408 +
409 +- merge actors by the requested scope and actor type identity;
410 +- apply `data.correlation.rules` without hardcoding topology-kind-specific key
411 + names in the aggregator;
412 +- remove pure correlation actors only for exact unambiguous `absorb` matches,
413 + rewiring incident correlation links to the matched actor with the rule's
414 + `output_link_type`;
415 +- keep correlation actors visible for no-match, ambiguous, and `link` partial
416 + matches, emitting weak semantic correlation links for visible partial matches;
417 +- preserve evidence rows when evidence policy is `preserve`;
418 +- count evidence rows when evidence policy is `count`;
419 +- preserve modal composition definitions and rewrite their type, table,
420 + evidence, and column references after namespacing/deduplication;
421 +- do not materialize modal rows during aggregation unless the underlying
422 + canonical table is already being merged;
423 +- merge `actor_labels` after actor reference remapping and preserve repeated
424 + values as repeated rows; `string` and `string_ref` label columns are
425 + equivalent logical strings and must be normalized before label deduplication;
426 +- never silently truncate evidence;
427 +- fail explicitly when a requested payload would exceed configured limits;
428 +- canonicalize undirected links only when link type policy allows it;
429 +- preserve directed links when direction is flow, dependency, or ownership;
430 +- merge overlay refs with `set` or `append` semantics defined by templates.
431 +
432 +Network socket matching:
433 +
434 +- exact reverse-tuple matching should be expressed through the generic
435 + correlation contract using process claims, endpoint points, correlation link
436 + types, rule priorities, and output link types;
437 +- NAT, load balancer, and proxy inference are out of scope for the first
438 + aggregator, but later NAT evidence can add extra point/claim rows for the
439 + same rule without changing the aggregator's key-building mechanism;
440 +- unresolved endpoints can aggregate by visible endpoint identity, but the
441 + evidence row must remain available when the requested mode preserves it.
442 +
443 +### Limits And Failure Behavior
444 +
445 +The aggregator must have explicit limits:
446 +
447 +- maximum decoded bytes;
448 +- maximum actor rows;
449 +- maximum graph links;
450 +- maximum evidence rows;
451 +- maximum output bytes;
452 +- maximum CPU time per request.
453 +
454 +If a limit is exceeded:
455 +
456 +- return a structured error;
457 +- include stats showing which limit was exceeded;
458 +- do not return a truncated topology as if it were complete.
459 +
460 +Paged or chunked evidence transport remains a phase-2 option. Phase 1 should
461 +make payloads small enough that this is rarely needed.
462 +
463 +### Tests
464 +
465 +Required test classes:
466 +
467 +- schema decode/encode round-trip;
468 +- semantic validation failures;
469 +- actor identity merge by scope;
470 +- directed vs undirected link aggregation;
471 +- relationship evidence preservation;
472 +- actor-detail table aggregation;
473 +- actor-label table aggregation;
474 +- modal presentation recipe preservation and reference rewriting;
475 +- overlay ref merge;
476 +- network socket exact reverse-tuple matching;
477 +- streaming hierarchy and actor-detail custom tables;
478 +- SNMP/L2 unordered adjacency and observation evidence;
479 +- vSphere ownership/dependency topology;
480 +- generic schema-conformant custom topology passthrough;
481 +- synthetic scale benchmark near and above current corpus scale;
482 +- sanitized real-corpus replay from `.local/` promoted only as non-sensitive
483 + fixtures when safe.
484 +
485 +## Rollout Plan
486 +
487 +1. Land schema docs, developer project skill, and implementation scope.
488 +2. Add validator support and compact-table helpers.
489 +3. Add Cloud frontend new-schema decoder and temporary compatibility adapter so
490 + mixed Agent rollout is safe before producers emit the new schema broadly.
491 +4. Migrate producers behind tests. `topology:network-connections` should be the
492 + first high-cardinality producer exercised internally, but it is not the
493 + production boundary for Cloud aggregation.
494 +5. Migrate the streaming producer and complete SNMP overlay-template
495 + refinement.
496 +6. Coordinate and migrate the vSphere topology producer.
497 +7. Build `cloud-topology-service` in parallel against fixtures and
498 + new-schema payloads. Its MVP is complete only when all topology kinds covered
499 + by this contract pass service-level aggregation tests.
500 +8. Hand final service ownership, environment-specific Helm values, deployment
501 + targets, and production node-instance routing strategy to Cloud backend and
502 + DevOps once the service is otherwise ready for operational integration.
503 +9. Remove compatibility support from Cloud frontend after supported Agent rollout.
504 +
505 +## Resolved Phase-1 Defaults
506 +
507 +- Cloud aggregator service repository: `cloud-topology-service`.
508 +- Cloud aggregated topology route: `POST /api/v3/spaces/{spaceID}/rooms/{roomID}/topology`.
509 +- Cloud service contract: accepts and emits only `netdata.topology.v1`.
510 +- Phase-1 topology service MVP: all topology kinds covered by this contract,
511 + not only `topology:network-connections`.
512 +- Network socket snapshot metrics such as RTT and retransmissions: opt-in, not
513 + default core topology columns.
514 +- Cloud-side topology payload cache: no payload cache in phase 1; aggregate on
515 + demand and collect request-cost metrics first.
516 +- Service-local validation package is sufficient for the Cloud service MVP;
517 + producer CI may still add a separate validator binary later if needed.
518 +
519 +## External Integration Gates
520 +
521 +These items cannot be safely invented from this repository and must be handed
522 +to Cloud backend and DevOps when `cloud-topology-service` is otherwise ready for
523 +operational integration:
524 +
525 +- final service owner and CODEOWNERS entries;
526 +- environment-specific Helm values and deployment targets;
527 +- approved production node-instance routing strategy.
src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json new
+2758
@@ -0,0 +1,2758 @@
1 +{
2 + "$schema": "https://json-schema.org/draft/2020-12/schema",
3 + "$id": "https://schemas.netdata.cloud/functions/topology/v1.json",
4 + "title": "Netdata Topology Function Payload",
5 + "description": "Production topology Function payload schema. This schema carries canonical topology facts for aggregation and UI rendering. It intentionally does not carry rollout-adapter reconstruction instructions.",
6 + "type": "object",
7 + "additionalProperties": false,
8 + "required": [
9 + "status",
10 + "type",
11 + "data"
12 + ],
13 + "properties": {
14 + "status": {
15 + "type": "integer",
16 + "const": 200
17 + },
18 + "type": {
19 + "type": "string",
20 + "const": "topology"
21 + },
22 + "v": {
23 + "type": "integer",
24 + "minimum": 0,
25 + "description": "Optional Function transport protocol version. This is part of the Function response envelope, not topology data."
26 + },
27 + "has_history": {
28 + "type": "boolean",
29 + "const": false
30 + },
31 + "accepted_params": {
32 + "type": "array",
33 + "items": {
34 + "type": "string"
35 + },
36 + "uniqueItems": true
37 + },
38 + "required_params": {
39 + "type": "array",
40 + "items": {
41 + "type": "object"
42 + }
43 + },
44 + "help": {
45 + "type": "string"
46 + },
47 + "update_every": {
48 + "type": "integer",
49 + "minimum": 0
50 + },
51 + "expires": {
52 + "type": "integer",
53 + "minimum": 0
54 + },
55 + "data": {
56 + "$ref": "#/$defs/topology_data"
57 + }
58 + },
59 + "$defs": {
60 + "topology_data": {
61 + "type": "object",
62 + "additionalProperties": false,
63 + "required": [
64 + "schema_version",
65 + "producer",
66 + "collected_at",
67 + "dictionaries",
68 + "types",
69 + "actors",
70 + "links"
71 + ],
72 + "properties": {
73 + "schema_version": {
74 + "type": "string",
75 + "const": "netdata.topology.v1"
76 + },
77 + "producer": {
78 + "$ref": "#/$defs/producer"
79 + },
80 + "collected_at": {
81 + "type": "string",
82 + "format": "date-time"
83 + },
84 + "valid_after": {
85 + "type": "string",
86 + "format": "date-time"
87 + },
88 + "valid_until": {
89 + "type": "string",
90 + "format": "date-time"
91 + },
92 + "view": {
93 + "$ref": "#/$defs/view"
94 + },
95 + "dictionaries": {
96 + "$ref": "#/$defs/dictionaries"
97 + },
98 + "types": {
99 + "$ref": "#/$defs/type_registry"
100 + },
101 + "presentation": {
102 + "$ref": "#/$defs/topology_presentation"
103 + },
104 + "correlation": {
105 + "$ref": "#/$defs/topology_correlation"
106 + },
107 + "actors": {
108 + "$ref": "#/$defs/table"
109 + },
110 + "links": {
111 + "$ref": "#/$defs/table"
112 + },
113 + "evidence": {
114 + "type": "object",
115 + "propertyNames": {
116 + "$ref": "#/$defs/id"
117 + },
118 + "additionalProperties": {
119 + "$ref": "#/$defs/evidence_section"
120 + },
121 + "default": {}
122 + },
123 + "tables": {
124 + "$ref": "#/$defs/detail_tables"
125 + },
126 + "overlays": {
127 + "$ref": "#/$defs/overlay_refs"
128 + },
129 + "stats": {
130 + "type": "object",
131 + "additionalProperties": {
132 + "$ref": "#/$defs/scalar_or_array"
133 + },
134 + "default": {}
135 + },
136 + "extensions": {
137 + "type": "object",
138 + "description": "Producer-private metadata. Extensions must be small, non-sensitive, and must not be required for core aggregation.",
139 + "additionalProperties": {
140 + "$ref": "#/$defs/scalar_or_array"
141 + },
142 + "default": {}
143 + }
144 + }
145 + },
146 + "producer": {
147 + "type": "object",
148 + "additionalProperties": false,
149 + "required": [
150 + "source",
151 + "instance"
152 + ],
153 + "properties": {
154 + "source": {
155 + "$ref": "#/$defs/id",
156 + "description": "Stable producer id, for example network-connections, streaming, snmp-l2, vsphere."
157 + },
158 + "instance": {
159 + "type": "string",
160 + "description": "Source-local producer instance id."
161 + },
162 + "node_id": {
163 + "type": "string"
164 + },
165 + "machine_guid": {
166 + "type": "string"
167 + },
168 + "agent_version": {
169 + "type": "string"
170 + },
171 + "plugin": {
172 + "type": "string"
173 + },
174 + "capabilities": {
175 + "type": "array",
176 + "items": {
177 + "$ref": "#/$defs/id"
178 + },
179 + "uniqueItems": true,
180 + "default": []
181 + }
182 + }
183 + },
184 + "view": {
185 + "type": "object",
186 + "additionalProperties": false,
187 + "properties": {
188 + "id": {
189 + "$ref": "#/$defs/id"
190 + },
191 + "scope": {
192 + "$ref": "#/$defs/id"
193 + },
194 + "mode": {
195 + "type": "string",
196 + "enum": [
197 + "aggregated",
198 + "detailed"
199 + ]
200 + },
201 + "supported_modes": {
202 + "type": "array",
203 + "description": "Modes this producer can return with different semantics. Omit or provide one value for mode-invariant topologies so UIs do not show a misleading mode selector.",
204 + "items": {
205 + "type": "string",
206 + "enum": [
207 + "aggregated",
208 + "detailed"
209 + ]
210 + },
211 + "uniqueItems": true,
212 + "default": []
213 + },
214 + "group_by": {
215 + "type": "array",
216 + "items": {
217 + "$ref": "#/$defs/id"
218 + },
219 + "uniqueItems": true,
220 + "default": []
221 + }
222 + }
223 + },
224 + "dictionaries": {
225 + "type": "object",
226 + "additionalProperties": {
227 + "type": "array",
228 + "items": {
229 + "$ref": "#/$defs/scalar_or_array"
230 + }
231 + },
232 + "properties": {
233 + "strings": {
234 + "type": "array",
235 + "items": {
236 + "type": "string"
237 + },
238 + "default": []
239 + }
240 + }
241 + },
242 + "type_registry": {
243 + "type": "object",
244 + "additionalProperties": false,
245 + "required": [
246 + "actor_types",
247 + "link_types"
248 + ],
249 + "properties": {
250 + "actor_types": {
251 + "$ref": "#/$defs/actor_type_map"
252 + },
253 + "link_types": {
254 + "$ref": "#/$defs/link_type_map"
255 + },
256 + "port_types": {
257 + "$ref": "#/$defs/port_type_map"
258 + },
259 + "evidence_types": {
260 + "$ref": "#/$defs/evidence_type_map"
261 + },
262 + "table_types": {
263 + "$ref": "#/$defs/table_type_map"
264 + },
265 + "overlay_templates": {
266 + "$ref": "#/$defs/overlay_template_map"
267 + },
268 + "aggregation_scopes": {
269 + "$ref": "#/$defs/aggregation_scope_map"
270 + }
271 + }
272 + },
273 + "actor_type_map": {
274 + "type": "object",
275 + "propertyNames": {
276 + "$ref": "#/$defs/id"
277 + },
278 + "additionalProperties": {
279 + "$ref": "#/$defs/actor_type"
280 + }
281 + },
282 + "link_type_map": {
283 + "type": "object",
284 + "propertyNames": {
285 + "$ref": "#/$defs/id"
286 + },
287 + "additionalProperties": {
288 + "$ref": "#/$defs/link_type"
289 + }
290 + },
291 + "port_type_map": {
292 + "type": "object",
293 + "propertyNames": {
294 + "$ref": "#/$defs/id"
295 + },
296 + "additionalProperties": {
297 + "$ref": "#/$defs/port_type"
298 + },
299 + "default": {}
300 + },
301 + "evidence_type_map": {
302 + "type": "object",
303 + "propertyNames": {
304 + "$ref": "#/$defs/id"
305 + },
306 + "additionalProperties": {
307 + "$ref": "#/$defs/evidence_type"
308 + },
309 + "default": {}
310 + },
311 + "table_type_map": {
312 + "type": "object",
313 + "propertyNames": {
314 + "$ref": "#/$defs/id"
315 + },
316 + "additionalProperties": {
317 + "$ref": "#/$defs/table_type"
318 + },
319 + "default": {}
320 + },
321 + "overlay_template_map": {
322 + "type": "object",
323 + "propertyNames": {
324 + "$ref": "#/$defs/id"
325 + },
326 + "additionalProperties": {
327 + "$ref": "#/$defs/overlay_template"
328 + },
329 + "default": {}
330 + },
331 + "aggregation_scope_map": {
332 + "type": "object",
333 + "propertyNames": {
334 + "$ref": "#/$defs/id"
335 + },
336 + "additionalProperties": {
337 + "$ref": "#/$defs/aggregation_scope"
338 + },
339 + "default": {}
340 + },
341 + "actor_type": {
342 + "type": "object",
343 + "additionalProperties": false,
344 + "required": [
345 + "layer",
346 + "identity"
347 + ],
348 + "properties": {
349 + "layer": {
350 + "$ref": "#/$defs/layer"
351 + },
352 + "identity": {
353 + "type": "array",
354 + "description": "Actor-table column ids that form the source-local canonical identity.",
355 + "items": {
356 + "$ref": "#/$defs/id"
357 + },
358 + "minItems": 1,
359 + "uniqueItems": true
360 + },
361 + "merge_identity": {
362 + "type": "array",
363 + "description": "Actor-table column ids that may be used by a cross-node or cross-source aggregator.",
364 + "items": {
365 + "$ref": "#/$defs/id"
366 + },
367 + "uniqueItems": true,
368 + "default": []
369 + },
370 + "parent_identity": {
371 + "type": "array",
372 + "description": "Actor-table column ids that identify the actor this actor lives on or belongs to.",
373 + "items": {
374 + "$ref": "#/$defs/id"
375 + },
376 + "uniqueItems": true,
377 + "default": []
378 + },
379 + "aggregation_scopes": {
380 + "type": "array",
381 + "items": {
382 + "$ref": "#/$defs/id"
383 + },
384 + "uniqueItems": true,
385 + "default": []
386 + },
387 + "search": {
388 + "$ref": "#/$defs/actor_search_policy"
389 + },
390 + "presentation": {
391 + "$ref": "#/$defs/actor_type_presentation"
392 + }
393 + }
394 + },
395 + "link_type": {
396 + "type": "object",
397 + "additionalProperties": false,
398 + "required": [
399 + "orientation",
400 + "direction_role",
401 + "aggregation"
402 + ],
403 + "properties": {
404 + "orientation": {
405 + "type": "string",
406 + "enum": [
407 + "directed",
408 + "undirected",
409 + "hierarchical",
410 + "observed_bidirectional"
411 + ]
412 + },
413 + "direction_role": {
414 + "type": "string",
415 + "enum": [
416 + "none",
417 + "flow",
418 + "dependency",
419 + "ownership",
420 + "observation"
421 + ]
422 + },
423 + "semantic_role": {
424 + "$ref": "#/$defs/link_semantic_role"
425 + },
426 + "aggregation": {
427 + "$ref": "#/$defs/link_aggregation"
428 + },
429 + "evidence_types": {
430 + "type": "array",
431 + "items": {
432 + "$ref": "#/$defs/id"
433 + },
434 + "uniqueItems": true,
435 + "default": []
436 + },
437 + "overlay_templates": {
438 + "type": "array",
439 + "items": {
440 + "$ref": "#/$defs/id"
441 + },
442 + "uniqueItems": true,
443 + "default": []
444 + },
445 + "presentation": {
446 + "$ref": "#/$defs/link_type_presentation"
447 + }
448 + }
449 + },
450 + "port_type": {
451 + "type": "object",
452 + "additionalProperties": false,
453 + "properties": {
454 + "presentation": {
455 + "$ref": "#/$defs/port_type_presentation"
456 + }
457 + }
458 + },
459 + "topology_presentation": {
460 + "type": "object",
461 + "description": "Graph-level presentation policy. Type-specific visual tokens live inside actor, link, and port type definitions; this object carries only cross-type policies such as selection, legend, port field labels, and scale-key labels.",
462 + "additionalProperties": false,
463 + "properties": {
464 + "profile_version": {
465 + "type": "string",
466 + "description": "Producer presentation profile version for diagnostics. This does not replace data.schema_version."
467 + },
468 + "selection": {
469 + "$ref": "#/$defs/selection_presentation"
470 + },
471 + "legend": {
472 + "$ref": "#/$defs/presentation_legend"
473 + },
474 + "port_fields": {
475 + "type": "array",
476 + "items": {
477 + "$ref": "#/$defs/presentation_field"
478 + },
479 + "default": []
480 + },
481 + "scale_keys": {
482 + "type": "object",
483 + "propertyNames": {
484 + "$ref": "#/$defs/id"
485 + },
486 + "additionalProperties": {
487 + "$ref": "#/$defs/scale_key_presentation"
488 + },
489 + "default": {}
490 + }
491 + }
492 + },
493 + "topology_correlation": {
494 + "type": "object",
495 + "description": "Producer-visible correlation contract used by aggregators to resolve loose relationship sides, replace weaker actors, merge/enrich equivalent actors, or resolve visible correlation actors across independently produced topology maps. This is not an aggregator state dump; final aggregated payloads remain normal topology graphs.",
496 + "additionalProperties": false,
497 + "required": [
498 + "rules"
499 + ],
500 + "properties": {
501 + "rules": {
502 + "type": "object",
503 + "description": "Correlation rules keyed by producer-local rule id. Rules define how keys are built, the action to apply on a match, and the link type used for the final visible graph.",
504 + "propertyNames": {
505 + "$ref": "#/$defs/id"
506 + },
507 + "additionalProperties": {
508 + "$ref": "#/$defs/correlation_rule"
509 + },
510 + "minProperties": 1
511 + },
512 + "points": {
513 + "description": "Compact table of visible correlation actors and their match keys when the input graph materializes correlation points. Required columns are actor, rule, and every column referenced by each referenced rule key.",
514 + "$ref": "#/$defs/table"
515 + },
516 + "claims": {
517 + "description": "Compact table of real actors and the keys they can satisfy. Required columns are actor, rule, and every column referenced by each referenced rule key.",
518 + "$ref": "#/$defs/table"
519 + }
520 + }
521 + },
522 + "correlation_rule": {
523 + "type": "object",
524 + "additionalProperties": false,
525 + "required": [
526 + "action",
527 + "priority",
528 + "key_space",
529 + "key",
530 + "point_actor_types",
531 + "output_link_type"
532 + ],
533 + "properties": {
534 + "action": {
535 + "type": "string",
536 + "description": "Visible output semantics when a point matches a claim. absorb removes matching correlation actors and rewrites links. link keeps the correlation actor visible and links it to the matched actor.",
537 + "enum": [
538 + "absorb",
539 + "link"
540 + ]
541 + },
542 + "class": {
543 + "type": "string",
544 + "description": "Topology-agnostic correlation outcome class. Older payloads may omit this and rely on action only.",
545 + "enum": [
546 + "resolve_loose_side",
547 + "replace_actor",
548 + "merge_enrich_actor"
549 + ]
550 + },
551 + "priority": {
552 + "type": "integer",
553 + "description": "Lower numbers run first. Exact rules should use higher priority than partial/broader rules.",
554 + "minimum": 0
555 + },
556 + "key_space": {
557 + "$ref": "#/$defs/id"
558 + },
559 + "key": {
560 + "type": "array",
561 + "description": "Declarative key template. The aggregator concatenates normalized column values and literals; it does not execute code or know the semantic meaning of the columns.",
562 + "items": {
563 + "$ref": "#/$defs/correlation_key_part"
564 + },
565 + "minItems": 1
566 + },
567 + "point_actor_types": {
568 + "type": "array",
569 + "description": "Actor type ids that represent visible correlation points for this rule. Rules that operate only on loose relationship-side facts may leave this empty in a future schema revision, but current payloads must provide at least one id when points are emitted.",
570 + "items": {
571 + "$ref": "#/$defs/id"
572 + },
573 + "minItems": 1,
574 + "uniqueItems": true
575 + },
576 + "claim_actor_types": {
577 + "type": "array",
578 + "description": "Actor type ids that may satisfy points for this rule. Empty means any actor type that emits a claim row may satisfy the rule.",
579 + "items": {
580 + "$ref": "#/$defs/id"
581 + },
582 + "uniqueItems": true,
583 + "default": []
584 + },
585 + "correlation_link_types": {
586 + "type": "array",
587 + "description": "Link type ids that connect real actors to correlation actors and may be consumed or rewritten by this rule.",
588 + "items": {
589 + "$ref": "#/$defs/id"
590 + },
591 + "uniqueItems": true,
592 + "default": []
593 + },
594 + "output_link_type": {
595 + "description": "Link type id to emit for rewritten absorb links or visible partial/link correlations.",
596 + "$ref": "#/$defs/id"
597 + }
598 + }
599 + },
600 + "correlation_key_part": {
601 + "oneOf": [
602 + {
603 + "type": "object",
604 + "additionalProperties": false,
605 + "required": [
606 + "column"
607 + ],
608 + "properties": {
609 + "column": {
610 + "$ref": "#/$defs/id"
611 + }
612 + }
613 + },
614 + {
615 + "type": "object",
616 + "additionalProperties": false,
617 + "required": [
618 + "literal"
619 + ],
620 + "properties": {
621 + "literal": {
622 + "type": "string",
623 + "minLength": 1,
624 + "maxLength": 64
625 + }
626 + }
627 + }
628 + ]
629 + },
630 + "actor_type_presentation": {
631 + "type": "object",
632 + "additionalProperties": false,
633 + "properties": {
634 + "label": {
635 + "type": "string",
636 + "minLength": 1
637 + },
638 + "role": {
639 + "type": "string",
640 + "enum": [
641 + "actor",
642 + "endpoint",
643 + "group"
644 + ]
645 + },
646 + "icon": {
647 + "$ref": "#/$defs/icon_token"
648 + },
649 + "color_slot": {
650 + "$ref": "#/$defs/color_slot"
651 + },
652 + "opacity": {
653 + "$ref": "#/$defs/opacity_token"
654 + },
655 + "border": {
656 + "$ref": "#/$defs/border_presentation"
657 + },
658 + "annotation": {
659 + "$ref": "#/$defs/annotation_presentation"
660 + },
661 + "size": {
662 + "$ref": "#/$defs/actor_size_presentation"
663 + },
664 + "layout": {
665 + "$ref": "#/$defs/actor_layout_presentation"
666 + },
667 + "label_policy": {
668 + "$ref": "#/$defs/label_policy"
669 + },
670 + "ports": {
671 + "$ref": "#/$defs/actor_ports_presentation"
672 + },
673 + "hover": {
674 + "$ref": "#/$defs/hover_presentation"
675 + },
676 + "modal": {
677 + "$ref": "#/$defs/modal_presentation"
678 + }
679 + }
680 + },
681 + "link_type_presentation": {
682 + "type": "object",
683 + "additionalProperties": false,
684 + "properties": {
685 + "label": {
686 + "type": "string",
687 + "minLength": 1
688 + },
689 + "color_slot": {
690 + "$ref": "#/$defs/color_slot"
691 + },
692 + "opacity": {
693 + "$ref": "#/$defs/opacity_token"
694 + },
695 + "line_style": {
696 + "type": "string",
697 + "enum": [
698 + "solid",
699 + "dashed",
700 + "dotted"
701 + ]
702 + },
703 + "width": {
704 + "$ref": "#/$defs/width_token"
705 + },
706 + "curve": {
707 + "type": "string",
708 + "enum": [
709 + "straight",
710 + "clockwise",
711 + "counter_clockwise",
712 + "auto"
713 + ]
714 + },
715 + "arrow": {
716 + "type": "string",
717 + "enum": [
718 + "none",
719 + "forward",
720 + "reverse",
721 + "both",
722 + "auto"
723 + ]
724 + },
725 + "variable": {
726 + "$ref": "#/$defs/link_variable_presentation"
727 + },
728 + "hover": {
729 + "$ref": "#/$defs/hover_presentation"
730 + },
731 + "layout": {
732 + "$ref": "#/$defs/link_layout_presentation"
733 + },
734 + "modal": {
735 + "$ref": "#/$defs/modal_presentation"
736 + }
737 + }
738 + },
739 + "link_layout_presentation": {
740 + "type": "object",
741 + "description": "Tokenized layout hints for the UI force graph. Producers choose relative tokens; the UI owns numeric force and distance values.",
742 + "additionalProperties": false,
743 + "properties": {
744 + "strength": {
745 + "$ref": "#/$defs/layout_strength_token"
746 + },
747 + "distance": {
748 + "$ref": "#/$defs/layout_distance_token"
749 + }
750 + }
751 + },
752 + "actor_layout_presentation": {
753 + "type": "object",
754 + "description": "Tokenized actor layout hints for the UI force graph. Producers choose relative tokens; the UI owns numeric force values.",
755 + "additionalProperties": false,
756 + "properties": {
757 + "repulsion": {
758 + "$ref": "#/$defs/layout_strength_token"
759 + }
760 + }
761 + },
762 + "port_type_presentation": {
763 + "type": "object",
764 + "additionalProperties": false,
765 + "properties": {
766 + "label": {
767 + "type": "string",
768 + "minLength": 1
769 + },
770 + "color_slot": {
771 + "$ref": "#/$defs/color_slot"
772 + },
773 + "opacity": {
774 + "$ref": "#/$defs/opacity_token"
775 + }
776 + }
777 + },
778 + "selection_presentation": {
779 + "type": "object",
780 + "additionalProperties": false,
781 + "properties": {
782 + "actor_click": {
783 + "$ref": "#/$defs/actor_click_presentation"
784 + }
785 + }
786 + },
787 + "actor_click_presentation": {
788 + "type": "object",
789 + "additionalProperties": false,
790 + "required": [
791 + "mode"
792 + ],
793 + "properties": {
794 + "mode": {
795 + "type": "string",
796 + "enum": [
797 + "none",
798 + "highlight_connections",
799 + "highlight_path"
800 + ]
801 + },
802 + "path_table": {
803 + "$ref": "#/$defs/id"
804 + },
805 + "path_owner_column": {
806 + "$ref": "#/$defs/id"
807 + },
808 + "path_actor_column": {
809 + "$ref": "#/$defs/id"
810 + },
811 + "path_order_column": {
812 + "$ref": "#/$defs/id"
813 + }
814 + },
815 + "allOf": [
816 + {
817 + "if": {
818 + "properties": {
819 + "mode": {
820 + "const": "highlight_path"
821 + }
822 + },
823 + "required": [
824 + "mode"
825 + ]
826 + },
827 + "then": {
828 + "required": [
829 + "path_table",
830 + "path_actor_column",
831 + "path_order_column"
832 + ]
833 + }
834 + }
835 + ]
836 + },
837 + "presentation_legend": {
838 + "type": "object",
839 + "additionalProperties": false,
840 + "properties": {
841 + "actors": {
842 + "type": "array",
843 + "items": {
844 + "$ref": "#/$defs/legend_entry"
845 + },
846 + "default": []
847 + },
848 + "links": {
849 + "type": "array",
850 + "items": {
851 + "$ref": "#/$defs/legend_entry"
852 + },
853 + "default": []
854 + },
855 + "ports": {
856 + "type": "array",
857 + "items": {
858 + "$ref": "#/$defs/legend_entry"
859 + },
860 + "default": []
861 + }
862 + }
863 + },
864 + "legend_entry": {
865 + "type": "object",
866 + "additionalProperties": false,
867 + "required": [
868 + "type"
869 + ],
870 + "properties": {
871 + "type": {
872 + "$ref": "#/$defs/id"
873 + },
874 + "label": {
875 + "type": "string",
876 + "minLength": 1
877 + }
878 + }
879 + },
880 + "presentation_field": {
881 + "type": "object",
882 + "additionalProperties": false,
883 + "required": [
884 + "key",
885 + "label"
886 + ],
887 + "properties": {
888 + "key": {
889 + "$ref": "#/$defs/id"
890 + },
891 + "label": {
892 + "type": "string"
893 + }
894 + }
895 + },
896 + "scale_key_presentation": {
897 + "type": "object",
898 + "additionalProperties": false,
899 + "required": [
900 + "label"
901 + ],
902 + "properties": {
903 + "label": {
904 + "type": "string",
905 + "minLength": 1
906 + },
907 + "unit": {
908 + "type": "string"
909 + }
910 + }
911 + },
912 + "border_presentation": {
913 + "type": "object",
914 + "additionalProperties": false,
915 + "properties": {
916 + "enabled": {
917 + "type": "boolean",
918 + "default": true
919 + },
920 + "color_slot": {
921 + "$ref": "#/$defs/color_slot"
922 + },
923 + "style": {
924 + "type": "string",
925 + "enum": [
926 + "solid",
927 + "dashed",
928 + "dotted"
929 + ],
930 + "default": "solid"
931 + }
932 + }
933 + },
934 + "annotation_presentation": {
935 + "type": "object",
936 + "additionalProperties": false,
937 + "properties": {
938 + "color_slot": {
939 + "$ref": "#/$defs/color_slot"
940 + },
941 + "style": {
942 + "type": "string",
943 + "enum": [
944 + "ring",
945 + "dot",
946 + "none"
947 + ]
948 + }
949 + }
950 + },
951 + "actor_size_presentation": {
952 + "type": "object",
953 + "additionalProperties": false,
954 + "required": [
955 + "mode"
956 + ],
957 + "properties": {
958 + "mode": {
959 + "type": "string",
960 + "enum": [
961 + "fixed",
962 + "link_count",
963 + "metric"
964 + ]
965 + },
966 + "metric_column": {
967 + "$ref": "#/$defs/id"
968 + },
969 + "scale": {
970 + "$ref": "#/$defs/actor_size_scale_token"
971 + }
972 + },
973 + "allOf": [
974 + {
975 + "if": {
976 + "properties": {
977 + "mode": {
978 + "const": "metric"
979 + }
980 + },
981 + "required": [
982 + "mode"
983 + ]
984 + },
985 + "then": {
986 + "required": [
987 + "metric_column"
988 + ]
989 + }
990 + }
991 + ]
992 + },
993 + "actor_search_policy": {
994 + "type": "object",
995 + "description": "Producer-declared searchable fields for this actor type. This replaces UI hardcoded traversal of domain-specific actor attributes.",
996 + "additionalProperties": false,
997 + "properties": {
998 + "enabled": {
999 + "type": "boolean",
1000 + "default": true
1001 + },
1002 + "columns": {
1003 + "type": "array",
1004 + "description": "Actor-table column ids whose scalar values should be indexed for graph search.",
1005 + "items": {
1006 + "$ref": "#/$defs/id"
1007 + },
1008 + "uniqueItems": true,
1009 + "default": []
1010 + },
1011 + "label_keys": {
1012 + "type": "array",
1013 + "description": "actor_labels.key values whose label values should be indexed for graph search.",
1014 + "items": {
1015 + "$ref": "#/$defs/id"
1016 + },
1017 + "uniqueItems": true,
1018 + "default": []
1019 + }
1020 + }
1021 + },
1022 + "label_policy": {
1023 + "type": "object",
1024 + "additionalProperties": false,
1025 + "properties": {
1026 + "columns": {
1027 + "type": "array",
1028 + "description": "Actor-table columns that are safe to use as display labels, in priority order.",
1029 + "items": {
1030 + "$ref": "#/$defs/id"
1031 + },
1032 + "uniqueItems": true,
1033 + "default": []
1034 + },
1035 + "fallback": {
1036 + "type": "string",
1037 + "enum": [
1038 + "type_label",
1039 + "row_index",
1040 + "none"
1041 + ],
1042 + "default": "type_label"
1043 + },
1044 + "max_length": {
1045 + "type": "integer",
1046 + "minimum": 8,
1047 + "maximum": 256,
1048 + "default": 80
1049 + },
1050 + "array": {
1051 + "type": "string",
1052 + "enum": [
1053 + "reject",
1054 + "first",
1055 + "summarize"
1056 + ],
1057 + "default": "reject"
1058 + }
1059 + }
1060 + },
1061 + "actor_ports_presentation": {
1062 + "type": "object",
1063 + "additionalProperties": false,
1064 + "properties": {
1065 + "show_bullets": {
1066 + "type": "boolean",
1067 + "default": false
1068 + },
1069 + "sources": {
1070 + "type": "array",
1071 + "description": "Port-bullet sources. The UI derives bullets from these compact table references instead of using domain-specific hardcoding.",
1072 + "items": {
1073 + "$ref": "#/$defs/port_source_presentation"
1074 + },
1075 + "default": []
1076 + }
1077 + }
1078 + },
1079 + "port_source_presentation": {
1080 + "type": "object",
1081 + "additionalProperties": false,
1082 + "required": [
1083 + "source",
1084 + "actor_column",
1085 + "name_column"
1086 + ],
1087 + "properties": {
1088 + "source": {
1089 + "type": "string",
1090 + "enum": [
1091 + "links",
1092 + "evidence",
1093 + "actor_table"
1094 + ]
1095 + },
1096 + "table": {
1097 + "$ref": "#/$defs/id",
1098 + "description": "Actor detail table id when source is actor_table."
1099 + },
1100 + "evidence": {
1101 + "$ref": "#/$defs/id",
1102 + "description": "Evidence section or evidence type id when source is evidence."
1103 + },
1104 + "actor_column": {
1105 + "$ref": "#/$defs/id",
1106 + "description": "Column in the source table that references the actor receiving the bullet."
1107 + },
1108 + "name_column": {
1109 + "$ref": "#/$defs/id",
1110 + "description": "Column in the source table used as the bullet key or label."
1111 + },
1112 + "value_column": {
1113 + "$ref": "#/$defs/id",
1114 + "description": "Optional numeric source-table column whose values are summed for weighted bullet count and actor sizing."
1115 + },
1116 + "type_column": {
1117 + "$ref": "#/$defs/id",
1118 + "description": "Optional source-table column whose value selects a port_types entry."
1119 + },
1120 + "default_type": {
1121 + "$ref": "#/$defs/id",
1122 + "description": "Port type to use when type_column is absent or a row has no usable type value."
1123 + },
1124 + "status_column": {
1125 + "$ref": "#/$defs/id"
1126 + },
1127 + "mode_column": {
1128 + "$ref": "#/$defs/id"
1129 + },
1130 + "role_column": {
1131 + "$ref": "#/$defs/id"
1132 + },
1133 + "sources_column": {
1134 + "$ref": "#/$defs/id"
1135 + }
1136 + }
1137 + },
1138 + "hover_presentation": {
1139 + "type": "object",
1140 + "additionalProperties": false,
1141 + "properties": {
1142 + "fields": {
1143 + "type": "array",
1144 + "items": {
1145 + "$ref": "#/$defs/presentation_field"
1146 + },
1147 + "default": []
1148 + }
1149 + }
1150 + },
1151 + "modal_presentation": {
1152 + "type": "object",
1153 + "description": "Actor/link modal composition. Modal sections are recipes over existing topology tables; they must not duplicate high-cardinality facts only for UI display.",
1154 + "additionalProperties": false,
1155 + "properties": {
1156 + "enabled": {
1157 + "type": "boolean",
1158 + "default": true
1159 + },
1160 + "labels": {
1161 + "$ref": "#/$defs/modal_labels_presentation"
1162 + },
1163 + "mini_topology": {
1164 + "$ref": "#/$defs/modal_mini_topology_presentation"
1165 + },
1166 + "sections": {
1167 + "type": "array",
1168 + "items": {
1169 + "$ref": "#/$defs/modal_section"
1170 + },
1171 + "default": []
1172 + }
1173 + }
1174 + },
1175 + "modal_labels_presentation": {
1176 + "type": "object",
1177 + "description": "How to render actor labels in the actor modal. The default source is an actor-owned actor_labels table with one row per label value.",
1178 + "additionalProperties": false,
1179 + "properties": {
1180 + "enabled": {
1181 + "type": "boolean",
1182 + "default": true
1183 + },
1184 + "table": {
1185 + "$ref": "#/$defs/id",
1186 + "default": "actor_labels"
1187 + },
1188 + "actor_column": {
1189 + "$ref": "#/$defs/id",
1190 + "default": "actor"
1191 + },
1192 + "key_column": {
1193 + "$ref": "#/$defs/id",
1194 + "default": "key"
1195 + },
1196 + "value_column": {
1197 + "$ref": "#/$defs/id",
1198 + "default": "value"
1199 + },
1200 + "source_column": {
1201 + "$ref": "#/$defs/id",
1202 + "default": "source"
1203 + },
1204 + "kind_column": {
1205 + "$ref": "#/$defs/id",
1206 + "default": "kind"
1207 + },
1208 + "value_index_column": {
1209 + "$ref": "#/$defs/id",
1210 + "default": "value_index"
1211 + },
1212 + "identification": {
1213 + "$ref": "#/$defs/modal_label_identification_presentation"
1214 + }
1215 + }
1216 + },
1217 + "modal_label_identification_presentation": {
1218 + "type": "object",
1219 + "description": "Producer-selected actor label keys that the UI should render in the actor modal identification/header area. The full label table remains available separately.",
1220 + "additionalProperties": false,
1221 + "properties": {
1222 + "enabled": {
1223 + "type": "boolean",
1224 + "default": true
1225 + },
1226 + "fields": {
1227 + "type": "array",
1228 + "items": {
1229 + "$ref": "#/$defs/modal_label_identification_field"
1230 + },
1231 + "default": []
1232 + }
1233 + }
1234 + },
1235 + "modal_label_identification_field": {
1236 + "type": "object",
1237 + "additionalProperties": false,
1238 + "required": [
1239 + "key",
1240 + "label"
1241 + ],
1242 + "properties": {
1243 + "key": {
1244 + "type": "string",
1245 + "minLength": 1,
1246 + "maxLength": 128
1247 + },
1248 + "label": {
1249 + "type": "string",
1250 + "minLength": 1,
1251 + "maxLength": 128
1252 + },
1253 + "max_values": {
1254 + "type": "integer",
1255 + "minimum": 1,
1256 + "default": 1
1257 + }
1258 + }
1259 + },
1260 + "modal_mini_topology_presentation": {
1261 + "type": "object",
1262 + "description": "Depth-1 modal topology preview built from existing actors and graph links.",
1263 + "additionalProperties": false,
1264 + "properties": {
1265 + "enabled": {
1266 + "type": "boolean",
1267 + "default": true
1268 + },
1269 + "depth": {
1270 + "type": "integer",
1271 + "const": 1,
1272 + "default": 1
1273 + },
1274 + "include_link_types": {
1275 + "type": "array",
1276 + "items": {
1277 + "$ref": "#/$defs/id"
1278 + },
1279 + "uniqueItems": true,
1280 + "default": []
1281 + },
1282 + "exclude_link_types": {
1283 + "type": "array",
1284 + "items": {
1285 + "$ref": "#/$defs/id"
1286 + },
1287 + "uniqueItems": true,
1288 + "default": []
1289 + }
1290 + }
1291 + },
1292 + "modal_section": {
1293 + "type": "object",
1294 + "additionalProperties": false,
1295 + "required": [
1296 + "id",
1297 + "label",
1298 + "source",
1299 + "columns"
1300 + ],
1301 + "properties": {
1302 + "id": {
1303 + "$ref": "#/$defs/id"
1304 + },
1305 + "label": {
1306 + "type": "string",
1307 + "minLength": 1
1308 + },
1309 + "order": {
1310 + "type": "integer",
1311 + "default": 0
1312 + },
1313 + "source": {
1314 + "$ref": "#/$defs/modal_source"
1315 + },
1316 + "owner_filter": {
1317 + "$ref": "#/$defs/modal_owner_filter"
1318 + },
1319 + "row_filters": {
1320 + "type": "array",
1321 + "items": {
1322 + "$ref": "#/$defs/modal_row_filter"
1323 + },
1324 + "default": []
1325 + },
1326 + "columns": {
1327 + "type": "array",
1328 + "items": {
1329 + "$ref": "#/$defs/modal_column"
1330 + },
1331 + "minItems": 1
1332 + },
1333 + "sort": {
1334 + "$ref": "#/$defs/modal_sort"
1335 + },
1336 + "empty_label": {
1337 + "type": "string"
1338 + }
1339 + }
1340 + },
1341 + "modal_source": {
1342 + "type": "object",
1343 + "additionalProperties": false,
1344 + "required": [
1345 + "kind"
1346 + ],
1347 + "properties": {
1348 + "kind": {
1349 + "type": "string",
1350 + "enum": [
1351 + "actors",
1352 + "links",
1353 + "evidence",
1354 + "actor_table",
1355 + "relationship_table"
1356 + ]
1357 + },
1358 + "table": {
1359 + "$ref": "#/$defs/id",
1360 + "description": "Detail table id when kind is actor_table or relationship_table."
1361 + },
1362 + "evidence": {
1363 + "$ref": "#/$defs/id",
1364 + "description": "Evidence section id when kind is evidence."
1365 + }
1366 + },
1367 + "allOf": [
1368 + {
1369 + "if": {
1370 + "required": [
1371 + "kind"
1372 + ],
1373 + "properties": {
1374 + "kind": {
1375 + "const": "evidence"
1376 + }
1377 + }
1378 + },
1379 + "then": {
1380 + "required": [
1381 + "evidence"
1382 + ]
1383 + }
1384 + },
1385 + {
1386 + "if": {
1387 + "required": [
1388 + "kind"
1389 + ],
1390 + "properties": {
1391 + "kind": {
1392 + "enum": [
1393 + "actor_table",
1394 + "relationship_table"
1395 + ]
1396 + }
1397 + }
1398 + },
1399 + "then": {
1400 + "required": [
1401 + "table"
1402 + ]
1403 + }
1404 + }
1405 + ]
1406 + },
1407 + "modal_owner_filter": {
1408 + "type": "object",
1409 + "additionalProperties": false,
1410 + "required": [
1411 + "mode"
1412 + ],
1413 + "properties": {
1414 + "mode": {
1415 + "type": "string",
1416 + "enum": [
1417 + "none",
1418 + "actor_column",
1419 + "link_column",
1420 + "incident_link",
1421 + "incident_evidence",
1422 + "selected_link"
1423 + ]
1424 + },
1425 + "actor_column": {
1426 + "$ref": "#/$defs/id"
1427 + },
1428 + "link_column": {
1429 + "$ref": "#/$defs/id"
1430 + },
1431 + "src_actor_column": {
1432 + "$ref": "#/$defs/id"
1433 + },
1434 + "dst_actor_column": {
1435 + "$ref": "#/$defs/id"
1436 + }
1437 + },
1438 + "allOf": [
1439 + {
1440 + "if": {
1441 + "required": [
1442 + "mode"
1443 + ],
1444 + "properties": {
1445 + "mode": {
1446 + "const": "actor_column"
1447 + }
1448 + }
1449 + },
1450 + "then": {
1451 + "required": [
1452 + "actor_column"
1453 + ]
1454 + }
1455 + },
1456 + {
1457 + "if": {
1458 + "required": [
1459 + "mode"
1460 + ],
1461 + "properties": {
1462 + "mode": {
1463 + "enum": [
1464 + "link_column",
1465 + "selected_link"
1466 + ]
1467 + }
1468 + }
1469 + },
1470 + "then": {
1471 + "required": [
1472 + "link_column"
1473 + ]
1474 + }
1475 + },
1476 + {
1477 + "if": {
1478 + "required": [
1479 + "mode"
1480 + ],
1481 + "properties": {
1482 + "mode": {
1483 + "enum": [
1484 + "incident_link",
1485 + "incident_evidence"
1486 + ]
1487 + }
1488 + }
1489 + },
1490 + "then": {
1491 + "required": [
1492 + "src_actor_column",
1493 + "dst_actor_column"
1494 + ]
1495 + }
1496 + }
1497 + ]
1498 + },
1499 + "modal_row_filter": {
1500 + "type": "object",
1501 + "additionalProperties": false,
1502 + "required": [
1503 + "column",
1504 + "op"
1505 + ],
1506 + "properties": {
1507 + "column": {
1508 + "$ref": "#/$defs/id"
1509 + },
1510 + "op": {
1511 + "type": "string",
1512 + "enum": [
1513 + "eq",
1514 + "ne",
1515 + "in",
1516 + "not_in",
1517 + "exists",
1518 + "missing"
1519 + ]
1520 + },
1521 + "value": {
1522 + "$ref": "#/$defs/cell_value"
1523 + },
1524 + "values": {
1525 + "type": "array",
1526 + "items": {
1527 + "$ref": "#/$defs/cell_value"
1528 + }
1529 + }
1530 + },
1531 + "allOf": [
1532 + {
1533 + "if": {
1534 + "required": [
1535 + "op"
1536 + ],
1537 + "properties": {
1538 + "op": {
1539 + "enum": [
1540 + "eq",
1541 + "ne"
1542 + ]
1543 + }
1544 + }
1545 + },
1546 + "then": {
1547 + "required": [
1548 + "value"
1549 + ]
1550 + }
1551 + },
1552 + {
1553 + "if": {
1554 + "required": [
1555 + "op"
1556 + ],
1557 + "properties": {
1558 + "op": {
1559 + "enum": [
1560 + "in",
1561 + "not_in"
1562 + ]
1563 + }
1564 + }
1565 + },
1566 + "then": {
1567 + "required": [
1568 + "values"
1569 + ],
1570 + "properties": {
1571 + "values": {
1572 + "minItems": 1
1573 + }
1574 + }
1575 + }
1576 + }
1577 + ]
1578 + },
1579 + "modal_column": {
1580 + "type": "object",
1581 + "additionalProperties": false,
1582 + "required": [
1583 + "id",
1584 + "label",
1585 + "projection"
1586 + ],
1587 + "properties": {
1588 + "id": {
1589 + "$ref": "#/$defs/id"
1590 + },
1591 + "label": {
1592 + "type": "string",
1593 + "minLength": 1
1594 + },
1595 + "projection": {
1596 + "$ref": "#/$defs/modal_projection"
1597 + },
1598 + "cell": {
1599 + "$ref": "#/$defs/modal_cell_type"
1600 + },
1601 + "visibility": {
1602 + "$ref": "#/$defs/modal_visibility"
1603 + },
1604 + "align": {
1605 + "type": "string",
1606 + "enum": [
1607 + "left",
1608 + "center",
1609 + "right"
1610 + ],
1611 + "default": "left"
1612 + },
1613 + "sortable": {
1614 + "type": "boolean",
1615 + "default": true
1616 + },
1617 + "badge_map": {
1618 + "type": "object",
1619 + "additionalProperties": {
1620 + "$ref": "#/$defs/modal_badge_presentation"
1621 + },
1622 + "default": {}
1623 + }
1624 + }
1625 + },
1626 + "modal_projection": {
1627 + "type": "object",
1628 + "additionalProperties": false,
1629 + "required": [
1630 + "kind"
1631 + ],
1632 + "properties": {
1633 + "kind": {
1634 + "type": "string",
1635 + "enum": [
1636 + "direct",
1637 + "actor_ref_label",
1638 + "opposite_actor",
1639 + "formatted_endpoint",
1640 + "label_lookup",
1641 + "json_path",
1642 + "const",
1643 + "coalesce",
1644 + "selected_side_endpoint"
1645 + ]
1646 + },
1647 + "column": {
1648 + "$ref": "#/$defs/id"
1649 + },
1650 + "columns": {
1651 + "type": "array",
1652 + "items": {
1653 + "$ref": "#/$defs/id"
1654 + },
1655 + "default": []
1656 + },
1657 + "value": {
1658 + "$ref": "#/$defs/cell_value"
1659 + },
1660 + "actor_column": {
1661 + "$ref": "#/$defs/id"
1662 + },
1663 + "src_actor_column": {
1664 + "$ref": "#/$defs/id"
1665 + },
1666 + "dst_actor_column": {
1667 + "$ref": "#/$defs/id"
1668 + },
1669 + "ip_column": {
1670 + "$ref": "#/$defs/id"
1671 + },
1672 + "port_column": {
1673 + "$ref": "#/$defs/id"
1674 + },
1675 + "protocol_column": {
1676 + "$ref": "#/$defs/id"
1677 + },
1678 + "local_ip_column": {
1679 + "$ref": "#/$defs/id"
1680 + },
1681 + "local_port_column": {
1682 + "$ref": "#/$defs/id"
1683 + },
1684 + "remote_ip_column": {
1685 + "$ref": "#/$defs/id"
1686 + },
1687 + "remote_port_column": {
1688 + "$ref": "#/$defs/id"
1689 + },
1690 + "label_key": {
1691 + "type": "string",
1692 + "minLength": 1
1693 + },
1694 + "path": {
1695 + "type": "string",
1696 + "minLength": 1,
1697 + "description": "JSON path for explicit scalar extraction from a json column. Use only for curated compatibility with producer-owned nested facts."
1698 + },
1699 + "fallback": {
1700 + "$ref": "#/$defs/cell_value"
1701 + }
1702 + },
1703 + "allOf": [
1704 + {
1705 + "if": {
1706 + "required": [
1707 + "kind"
1708 + ],
1709 + "properties": {
1710 + "kind": {
1711 + "const": "direct"
1712 + }
1713 + }
1714 + },
1715 + "then": {
1716 + "required": [
1717 + "column"
1718 + ]
1719 + }
1720 + },
1721 + {
1722 + "if": {
1723 + "required": [
1724 + "kind"
1725 + ],
1726 + "properties": {
1727 + "kind": {
1728 + "const": "actor_ref_label"
1729 + }
1730 + }
1731 + },
1732 + "then": {
1733 + "required": [
1734 + "actor_column"
1735 + ]
1736 + }
1737 + },
1738 + {
1739 + "if": {
1740 + "required": [
1741 + "kind"
1742 + ],
1743 + "properties": {
1744 + "kind": {
1745 + "const": "opposite_actor"
1746 + }
1747 + }
1748 + },
1749 + "then": {
1750 + "required": [
1751 + "src_actor_column",
1752 + "dst_actor_column"
1753 + ]
1754 + }
1755 + },
1756 + {
1757 + "if": {
1758 + "required": [
1759 + "kind"
1760 + ],
1761 + "properties": {
1762 + "kind": {
1763 + "const": "const"
1764 + }
1765 + }
1766 + },
1767 + "then": {
1768 + "required": [
1769 + "value"
1770 + ]
1771 + }
1772 + },
1773 + {
1774 + "if": {
1775 + "required": [
1776 + "kind"
1777 + ],
1778 + "properties": {
1779 + "kind": {
1780 + "const": "formatted_endpoint"
1781 + }
1782 + }
1783 + },
1784 + "then": {
1785 + "anyOf": [
1786 + {
1787 + "required": [
1788 + "ip_column"
1789 + ]
1790 + },
1791 + {
1792 + "required": [
1793 + "port_column"
1794 + ]
1795 + }
1796 + ]
1797 + }
1798 + },
1799 + {
1800 + "if": {
1801 + "required": [
1802 + "kind"
1803 + ],
1804 + "properties": {
1805 + "kind": {
1806 + "const": "label_lookup"
1807 + }
1808 + }
1809 + },
1810 + "then": {
1811 + "required": [
1812 + "label_key"
1813 + ]
1814 + }
1815 + },
1816 + {
1817 + "if": {
1818 + "required": [
1819 + "kind"
1820 + ],
1821 + "properties": {
1822 + "kind": {
1823 + "const": "json_path"
1824 + }
1825 + }
1826 + },
1827 + "then": {
1828 + "required": [
1829 + "column",
1830 + "path"
1831 + ]
1832 + }
1833 + },
1834 + {
1835 + "if": {
1836 + "required": [
1837 + "kind"
1838 + ],
1839 + "properties": {
1840 + "kind": {
1841 + "const": "coalesce"
1842 + }
1843 + }
1844 + },
1845 + "then": {
1846 + "required": [
1847 + "columns"
1848 + ],
1849 + "properties": {
1850 + "columns": {
1851 + "minItems": 1
1852 + }
1853 + }
1854 + }
1855 + },
1856 + {
1857 + "if": {
1858 + "required": [
1859 + "kind"
1860 + ],
1861 + "properties": {
1862 + "kind": {
1863 + "const": "selected_side_endpoint"
1864 + }
1865 + }
1866 + },
1867 + "then": {
1868 + "allOf": [
1869 + {
1870 + "required": [
1871 + "src_actor_column",
1872 + "dst_actor_column"
1873 + ]
1874 + },
1875 + {
1876 + "anyOf": [
1877 + {
1878 + "required": [
1879 + "local_ip_column"
1880 + ]
1881 + },
1882 + {
1883 + "required": [
1884 + "local_port_column"
1885 + ]
1886 + }
1887 + ]
1888 + },
1889 + {
1890 + "anyOf": [
1891 + {
1892 + "required": [
1893 + "remote_ip_column"
1894 + ]
1895 + },
1896 + {
1897 + "required": [
1898 + "remote_port_column"
1899 + ]
1900 + }
1901 + ]
1902 + }
1903 + ]
1904 + }
1905 + }
1906 + ]
1907 + },
1908 + "modal_cell_type": {
1909 + "type": "string",
1910 + "enum": [
1911 + "text",
1912 + "number",
1913 + "badge",
1914 + "actor_link",
1915 + "timestamp",
1916 + "duration",
1917 + "endpoint",
1918 + "array_count",
1919 + "debug_json"
1920 + ]
1921 + },
1922 + "modal_visibility": {
1923 + "type": "string",
1924 + "enum": [
1925 + "table",
1926 + "expanded",
1927 + "hidden",
1928 + "debug"
1929 + ],
1930 + "default": "table"
1931 + },
1932 + "modal_badge_presentation": {
1933 + "type": "object",
1934 + "additionalProperties": false,
1935 + "properties": {
1936 + "label": {
1937 + "type": "string"
1938 + },
1939 + "color_slot": {
1940 + "$ref": "#/$defs/color_slot"
1941 + },
1942 + "opacity": {
1943 + "$ref": "#/$defs/opacity_token"
1944 + }
1945 + }
1946 + },
1947 + "modal_sort": {
1948 + "type": "object",
1949 + "additionalProperties": false,
1950 + "required": [
1951 + "column"
1952 + ],
1953 + "properties": {
1954 + "column": {
1955 + "$ref": "#/$defs/id"
1956 + },
1957 + "direction": {
1958 + "type": "string",
1959 + "enum": [
1960 + "asc",
1961 + "desc"
1962 + ],
1963 + "default": "asc"
1964 + }
1965 + }
1966 + },
1967 + "link_variable_presentation": {
1968 + "type": "object",
1969 + "additionalProperties": false,
1970 + "required": [
1971 + "channel",
1972 + "scale_key",
1973 + "value_column"
1974 + ],
1975 + "properties": {
1976 + "channel": {
1977 + "type": "string",
1978 + "enum": [
1979 + "width",
1980 + "opacity"
1981 + ]
1982 + },
1983 + "scale_key": {
1984 + "$ref": "#/$defs/id"
1985 + },
1986 + "value_column": {
1987 + "$ref": "#/$defs/id"
1988 + },
1989 + "min": {
1990 + "type": "string",
1991 + "description": "Visual token lower bound for the selected channel."
1992 + },
1993 + "max": {
1994 + "type": "string",
1995 + "description": "Visual token upper bound for the selected channel."
1996 + }
1997 + },
1998 + "allOf": [
1999 + {
2000 + "if": {
2001 + "properties": {
2002 + "channel": {
2003 + "const": "width"
2004 + }
2005 + },
2006 + "required": [
2007 + "channel"
2008 + ]
2009 + },
2010 + "then": {
2011 + "properties": {
2012 + "min": {
2013 + "$ref": "#/$defs/width_token"
2014 + },
2015 + "max": {
2016 + "$ref": "#/$defs/width_token"
2017 + }
2018 + }
2019 + }
2020 + },
2021 + {
2022 + "if": {
2023 + "properties": {
2024 + "channel": {
2025 + "const": "opacity"
2026 + }
2027 + },
2028 + "required": [
2029 + "channel"
2030 + ]
2031 + },
2032 + "then": {
2033 + "properties": {
2034 + "min": {
2035 + "$ref": "#/$defs/opacity_token"
2036 + },
2037 + "max": {
2038 + "$ref": "#/$defs/opacity_token"
2039 + }
2040 + }
2041 + }
2042 + }
2043 + ]
2044 + },
2045 + "link_aggregation": {
2046 + "type": "object",
2047 + "additionalProperties": false,
2048 + "required": [
2049 + "direction"
2050 + ],
2051 + "properties": {
2052 + "direction": {
2053 + "type": "string",
2054 + "enum": [
2055 + "preserve",
2056 + "ignore",
2057 + "canonicalize_unordered"
2058 + ]
2059 + },
2060 + "evidence": {
2061 + "type": "string",
2062 + "enum": [
2063 + "append",
2064 + "count",
2065 + "drop"
2066 + ],
2067 + "default": "append"
2068 + },
2069 + "metrics": {
2070 + "type": "object",
2071 + "additionalProperties": {
2072 + "$ref": "#/$defs/aggregation_rule"
2073 + },
2074 + "default": {}
2075 + }
2076 + }
2077 + },
2078 + "evidence_type": {
2079 + "type": "object",
2080 + "additionalProperties": false,
2081 + "required": [
2082 + "link_type",
2083 + "role",
2084 + "columns"
2085 + ],
2086 + "properties": {
2087 + "link_type": {
2088 + "$ref": "#/$defs/id"
2089 + },
2090 + "role": {
2091 + "type": "string",
2092 + "enum": [
2093 + "relationship_evidence",
2094 + "observation_evidence",
2095 + "metric_sample"
2096 + ]
2097 + },
2098 + "columns": {
2099 + "type": "array",
2100 + "items": {
2101 + "$ref": "#/$defs/column"
2102 + },
2103 + "minItems": 1
2104 + },
2105 + "match_columns": {
2106 + "type": "array",
2107 + "description": "Evidence column ids required to preserve exact relationship matching.",
2108 + "items": {
2109 + "$ref": "#/$defs/id"
2110 + },
2111 + "uniqueItems": true,
2112 + "default": []
2113 + }
2114 + }
2115 + },
2116 + "table_type": {
2117 + "type": "object",
2118 + "additionalProperties": false,
2119 + "required": [
2120 + "role",
2121 + "owner",
2122 + "aggregation",
2123 + "columns"
2124 + ],
2125 + "properties": {
2126 + "role": {
2127 + "type": "string",
2128 + "enum": [
2129 + "actor_detail",
2130 + "actor_inventory",
2131 + "relationship_evidence",
2132 + "relationship_summary"
2133 + ]
2134 + },
2135 + "owner": {
2136 + "type": "string",
2137 + "enum": [
2138 + "actor",
2139 + "link",
2140 + "evidence",
2141 + "topology"
2142 + ]
2143 + },
2144 + "aggregation": {
2145 + "type": "string",
2146 + "enum": [
2147 + "none",
2148 + "append",
2149 + "deduplicate",
2150 + "set",
2151 + "set_union",
2152 + "merge_metrics",
2153 + "preserve",
2154 + "sum",
2155 + "min",
2156 + "max",
2157 + "last",
2158 + "first",
2159 + "latest"
2160 + ]
2161 + },
2162 + "source_evidence": {
2163 + "$ref": "#/$defs/id"
2164 + },
2165 + "columns": {
2166 + "type": "array",
2167 + "items": {
2168 + "$ref": "#/$defs/column"
2169 + },
2170 + "minItems": 1
2171 + },
2172 + "presentation": {
2173 + "$ref": "#/$defs/table_type_presentation"
2174 + }
2175 + }
2176 + },
2177 + "table_type_presentation": {
2178 + "type": "object",
2179 + "description": "Default table display metadata. Full modal sections live on actor/link type presentation and select from these table rows.",
2180 + "additionalProperties": false,
2181 + "properties": {
2182 + "label": {
2183 + "type": "string",
2184 + "minLength": 1
2185 + },
2186 + "order": {
2187 + "type": "integer",
2188 + "default": 0
2189 + },
2190 + "default_visibility": {
2191 + "$ref": "#/$defs/modal_visibility"
2192 + },
2193 + "columns": {
2194 + "type": "array",
2195 + "items": {
2196 + "$ref": "#/$defs/modal_column"
2197 + },
2198 + "default": []
2199 + }
2200 + }
2201 + },
2202 + "overlay_template": {
2203 + "type": "object",
2204 + "additionalProperties": false,
2205 + "required": [
2206 + "provider",
2207 + "merge"
2208 + ],
2209 + "properties": {
2210 + "provider": {
2211 + "type": "string",
2212 + "enum": [
2213 + "netdata.metrics",
2214 + "netdata.function",
2215 + "external"
2216 + ]
2217 + },
2218 + "contexts": {
2219 + "type": "array",
2220 + "items": {
2221 + "type": "string"
2222 + },
2223 + "uniqueItems": true,
2224 + "default": []
2225 + },
2226 + "dimensions": {
2227 + "type": "array",
2228 + "items": {
2229 + "type": "string"
2230 + },
2231 + "uniqueItems": true,
2232 + "default": []
2233 + },
2234 + "selector_params": {
2235 + "type": "array",
2236 + "description": "Parameter names supplied by overlay refs.",
2237 + "items": {
2238 + "$ref": "#/$defs/id"
2239 + },
2240 + "uniqueItems": true,
2241 + "default": []
2242 + },
2243 + "merge": {
2244 + "$ref": "#/$defs/overlay_merge"
2245 + }
2246 + }
2247 + },
2248 + "overlay_merge": {
2249 + "type": "object",
2250 + "additionalProperties": false,
2251 + "required": [
2252 + "refs",
2253 + "values"
2254 + ],
2255 + "properties": {
2256 + "refs": {
2257 + "type": "string",
2258 + "enum": [
2259 + "append",
2260 + "set"
2261 + ]
2262 + },
2263 + "values": {
2264 + "type": "string",
2265 + "enum": [
2266 + "sum",
2267 + "min",
2268 + "max",
2269 + "avg",
2270 + "last",
2271 + "none"
2272 + ]
2273 + }
2274 + }
2275 + },
2276 + "aggregation_scope": {
2277 + "type": "object",
2278 + "additionalProperties": false,
2279 + "required": [
2280 + "columns"
2281 + ],
2282 + "properties": {
2283 + "columns": {
2284 + "type": "array",
2285 + "items": {
2286 + "$ref": "#/$defs/id"
2287 + },
2288 + "minItems": 1,
2289 + "uniqueItems": true
2290 + },
2291 + "evidence_policy": {
2292 + "type": "string",
2293 + "enum": [
2294 + "preserve",
2295 + "count"
2296 + ],
2297 + "default": "preserve"
2298 + }
2299 + }
2300 + },
2301 + "evidence_section": {
2302 + "type": "object",
2303 + "additionalProperties": false,
2304 + "required": [
2305 + "type",
2306 + "table"
2307 + ],
2308 + "properties": {
2309 + "type": {
2310 + "$ref": "#/$defs/id"
2311 + },
2312 + "table": {
2313 + "$ref": "#/$defs/table"
2314 + }
2315 + }
2316 + },
2317 + "detail_tables": {
2318 + "type": "object",
2319 + "additionalProperties": false,
2320 + "properties": {
2321 + "actor": {
2322 + "type": "object",
2323 + "propertyNames": {
2324 + "$ref": "#/$defs/id"
2325 + },
2326 + "additionalProperties": {
2327 + "$ref": "#/$defs/detail_table"
2328 + },
2329 + "default": {}
2330 + },
2331 + "relationship": {
2332 + "type": "object",
2333 + "propertyNames": {
2334 + "$ref": "#/$defs/id"
2335 + },
2336 + "additionalProperties": {
2337 + "$ref": "#/$defs/detail_table"
2338 + },
2339 + "default": {}
2340 + }
2341 + },
2342 + "default": {}
2343 + },
2344 + "detail_table": {
2345 + "type": "object",
2346 + "additionalProperties": false,
2347 + "required": [
2348 + "type",
2349 + "table"
2350 + ],
2351 + "properties": {
2352 + "type": {
2353 + "$ref": "#/$defs/id"
2354 + },
2355 + "table": {
2356 + "$ref": "#/$defs/table"
2357 + }
2358 + }
2359 + },
2360 + "overlay_refs": {
2361 + "type": "object",
2362 + "additionalProperties": false,
2363 + "properties": {
2364 + "refs": {
2365 + "$ref": "#/$defs/table"
2366 + }
2367 + },
2368 + "default": {}
2369 + },
2370 + "table": {
2371 + "type": "object",
2372 + "additionalProperties": false,
2373 + "required": [
2374 + "rows",
2375 + "columns",
2376 + "values"
2377 + ],
2378 + "properties": {
2379 + "rows": {
2380 + "type": "integer",
2381 + "minimum": 0
2382 + },
2383 + "columns": {
2384 + "type": "array",
2385 + "items": {
2386 + "$ref": "#/$defs/column"
2387 + }
2388 + },
2389 + "values": {
2390 + "type": "array",
2391 + "items": {
2392 + "$ref": "#/$defs/column_encoding"
2393 + }
2394 + }
2395 + }
2396 + },
2397 + "column": {
2398 + "type": "object",
2399 + "additionalProperties": false,
2400 + "required": [
2401 + "id",
2402 + "type"
2403 + ],
2404 + "properties": {
2405 + "id": {
2406 + "$ref": "#/$defs/id"
2407 + },
2408 + "type": {
2409 + "type": "string",
2410 + "enum": [
2411 + "bool",
2412 + "int",
2413 + "uint",
2414 + "float",
2415 + "string",
2416 + "string_ref",
2417 + "timestamp",
2418 + "duration",
2419 + "ip",
2420 + "ip_ref",
2421 + "mac",
2422 + "mac_ref",
2423 + "actor_ref",
2424 + "link_ref",
2425 + "evidence_ref",
2426 + "array",
2427 + "json"
2428 + ]
2429 + },
2430 + "dictionary": {
2431 + "$ref": "#/$defs/id"
2432 + },
2433 + "nullable": {
2434 + "type": "boolean",
2435 + "default": false
2436 + },
2437 + "unit": {
2438 + "type": "string"
2439 + },
2440 + "role": {
2441 + "type": "string",
2442 + "enum": [
2443 + "identity",
2444 + "merge_identity",
2445 + "parent_identity",
2446 + "group_key",
2447 + "metric",
2448 + "attribute",
2449 + "timestamp",
2450 + "reference"
2451 + ]
2452 + },
2453 + "aggregation": {
2454 + "$ref": "#/$defs/aggregation_rule"
2455 + }
2456 + }
2457 + },
2458 + "column_encoding": {
2459 + "oneOf": [
2460 + {
2461 + "type": "object",
2462 + "additionalProperties": false,
2463 + "required": [
2464 + "codec",
2465 + "value"
2466 + ],
2467 + "properties": {
2468 + "codec": {
2469 + "type": "string",
2470 + "const": "const"
2471 + },
2472 + "value": {
2473 + "$ref": "#/$defs/cell_value"
2474 + }
2475 + }
2476 + },
2477 + {
2478 + "type": "object",
2479 + "additionalProperties": false,
2480 + "required": [
2481 + "codec",
2482 + "values"
2483 + ],
2484 + "properties": {
2485 + "codec": {
2486 + "type": "string",
2487 + "const": "values"
2488 + },
2489 + "values": {
2490 + "type": "array",
2491 + "items": {
2492 + "$ref": "#/$defs/cell_value"
2493 + }
2494 + }
2495 + }
2496 + },
2497 + {
2498 + "type": "object",
2499 + "additionalProperties": false,
2500 + "required": [
2501 + "codec",
2502 + "values",
2503 + "indexes"
2504 + ],
2505 + "properties": {
2506 + "codec": {
2507 + "type": "string",
2508 + "const": "dict"
2509 + },
2510 + "values": {
2511 + "type": "array",
2512 + "items": {
2513 + "$ref": "#/$defs/cell_value"
2514 + }
2515 + },
2516 + "indexes": {
2517 + "type": "array",
2518 + "items": {
2519 + "type": "integer",
2520 + "minimum": 0
2521 + }
2522 + }
2523 + }
2524 + }
2525 + ]
2526 + },
2527 + "aggregation_rule": {
2528 + "type": "string",
2529 + "enum": [
2530 + "none",
2531 + "first",
2532 + "last",
2533 + "set",
2534 + "count",
2535 + "sum",
2536 + "min",
2537 + "max",
2538 + "avg"
2539 + ]
2540 + },
2541 + "layer": {
2542 + "type": "string",
2543 + "enum": [
2544 + "node",
2545 + "virtualization",
2546 + "container",
2547 + "kubernetes",
2548 + "process",
2549 + "network",
2550 + "storage",
2551 + "streaming",
2552 + "service",
2553 + "custom"
2554 + ]
2555 + },
2556 + "color_slot": {
2557 + "type": "string",
2558 + "enum": [
2559 + "primary",
2560 + "secondary",
2561 + "accent",
2562 + "self",
2563 + "neutral",
2564 + "muted",
2565 + "dim",
2566 + "derived",
2567 + "info",
2568 + "structural",
2569 + "warning",
2570 + "success",
2571 + "danger",
2572 + "blue",
2573 + "green",
2574 + "orange",
2575 + "purple",
2576 + "cyan",
2577 + "yellow",
2578 + "teal",
2579 + "gray"
2580 + ]
2581 + },
2582 + "opacity_token": {
2583 + "type": "string",
2584 + "enum": [
2585 + "normal",
2586 + "muted",
2587 + "faded"
2588 + ]
2589 + },
2590 + "width_token": {
2591 + "type": "string",
2592 + "enum": [
2593 + "thin",
2594 + "normal",
2595 + "thick",
2596 + "emphasis"
2597 + ]
2598 + },
2599 + "layout_strength_token": {
2600 + "type": "string",
2601 + "enum": [
2602 + "weakest",
2603 + "weaker",
2604 + "normal",
2605 + "stronger",
2606 + "strongest"
2607 + ]
2608 + },
2609 + "layout_distance_token": {
2610 + "type": "string",
2611 + "enum": [
2612 + "closest",
2613 + "closer",
2614 + "normal",
2615 + "farther",
2616 + "farthest"
2617 + ]
2618 + },
2619 + "actor_size_scale_token": {
2620 + "type": "string",
2621 + "enum": [
2622 + "compact",
2623 + "normal",
2624 + "emphasized"
2625 + ]
2626 + },
2627 + "link_semantic_role": {
2628 + "type": "string",
2629 + "enum": [
2630 + "normal",
2631 + "discovery",
2632 + "ownership",
2633 + "traffic",
2634 + "correlation",
2635 + "control"
2636 + ]
2637 + },
2638 + "icon_token": {
2639 + "type": "string",
2640 + "enum": [
2641 + "router",
2642 + "switch",
2643 + "firewall",
2644 + "access_point",
2645 + "server",
2646 + "storage",
2647 + "load_balancer",
2648 + "printer",
2649 + "phone",
2650 + "ups",
2651 + "camera",
2652 + "process",
2653 + "agent",
2654 + "netdata-agent",
2655 + "parent",
2656 + "remote-endpoint",
2657 + "local-endpoint",
2658 + "segment",
2659 + "self",
2660 + "ip",
2661 + "cloud",
2662 + "container",
2663 + "vm",
2664 + "database",
2665 + "service",
2666 + "datacenter",
2667 + "cluster",
2668 + "host",
2669 + "network",
2670 + "datastore",
2671 + "datastore_cluster",
2672 + "resource_pool",
2673 + "device",
2674 + "endpoint",
2675 + "correlation",
2676 + "interface",
2677 + "group",
2678 + "unknown"
2679 + ]
2680 + },
2681 + "id": {
2682 + "type": "string",
2683 + "pattern": "^[A-Za-z][A-Za-z0-9_.:-]*$"
2684 + },
2685 + "scalar_or_array": {
2686 + "anyOf": [
2687 + {
2688 + "type": "null"
2689 + },
2690 + {
2691 + "type": "boolean"
2692 + },
2693 + {
2694 + "type": "integer"
2695 + },
2696 + {
2697 + "type": "number"
2698 + },
2699 + {
2700 + "type": "string"
2701 + },
2702 + {
2703 + "type": "array",
2704 + "items": {
2705 + "anyOf": [
2706 + {
2707 + "type": "null"
2708 + },
2709 + {
2710 + "type": "boolean"
2711 + },
2712 + {
2713 + "type": "integer"
2714 + },
2715 + {
2716 + "type": "number"
2717 + },
2718 + {
2719 + "type": "string"
2720 + }
2721 + ]
2722 + }
2723 + }
2724 + ]
2725 + },
2726 + "cell_value": {
2727 + "anyOf": [
2728 + {
2729 + "type": "null"
2730 + },
2731 + {
2732 + "type": "boolean"
2733 + },
2734 + {
2735 + "type": "integer"
2736 + },
2737 + {
2738 + "type": "number"
2739 + },
2740 + {
2741 + "type": "string"
2742 + },
2743 + {
2744 + "type": "array",
2745 + "items": {
2746 + "$ref": "#/$defs/cell_value"
2747 + }
2748 + },
2749 + {
2750 + "type": "object",
2751 + "additionalProperties": {
2752 + "$ref": "#/$defs/cell_value"
2753 + }
2754 + }
2755 + ]
2756 + }
2757 + }
2758 +}
src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md
+2 -1
@@ -1,6 +1,6 @@
1 # Netdata Functions: Developer Guide
2
3 -> **Note**: This is the practical developer guide. For the complete technical specification, see [Functions v3 Protocol Reference](/src/plugins.d/FUNCTION_UI_REFERENCE.md).
3 +> **Note**: This is the practical developer guide. For the complete technical specification, see [Functions v3 Protocol Reference](/src/plugins.d/FUNCTION_UI_REFERENCE.md). For topology Functions, use the dedicated [Topology Function Schema](/src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md).
4
5 ## Overview
6
@@ -17,6 +17,7 @@ This guide teaches you how to create Netdata functions that provide interactive
17 - [Part 1: Simple Table Functions](#part-1-simple-table-functions) - Basic monitoring data
18 - [Part 2: Log Explorer Functions](#part-2-log-explorer-functions) - Historical data with search
19 - [Part 3: Complete Options Reference](#part-3-complete-options-reference) - Every option explained
20 +- [Topology Functions](/src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md) - Graph payloads, evidence rows, aggregation policy, and telemetry overlays
21
22 ---
23
src/plugins.d/FUNCTION_UI_REFERENCE.md
+6 -1
@@ -1,6 +1,6 @@
1 # Netdata Functions v3 Protocol - Technical Reference
2
3 -> **Note**: This is the technical specification. For a practical guide to implementing functions, see [Functions Developer Guide](/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md).
3 +> **Note**: This is the technical specification. For a practical guide to implementing functions, see [Functions Developer Guide](/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md). Topology Functions use the dedicated [Topology Function Schema](/src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md).
4
5 ## Overview
6
@@ -33,6 +33,11 @@ Netdata Functions allow collectors/plugins to expose interactive data through a
33 - Advanced table with backend-powered faceted search, histograms, and infinite scroll
34 - Examples: `systemd-journal`, `windows-events`
35
36 +3. **Topology Format** (`type: "topology"`)
37 + - Compact graph payload with actors, graph links, relationship evidence, custom actor detail tables, correlation rules, and overlay references
38 + - Uses [FUNCTION_TOPOLOGY_SCHEMA.json](/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json)
39 + - Examples: `topology:network-connections`, `topology:streaming`, `topology:snmp`
40 +
41 ### Critical Implementation Differences
42
43 | Aspect | Simple Tables | Log Explorers |
src/web/api/functions/function-topology-streaming.c
+2526 -1307
@@ -37,25 +37,6 @@ struct streaming_topology_options {
37 char *function_copy;
38 };
39
40 -static void streaming_topology_add_host_match(BUFFER *wb, RRDHOST *host) {
41 - buffer_json_member_add_object(wb, "match");
42 - {
43 - buffer_json_member_add_array(wb, "hostnames");
44 - {
45 - buffer_json_add_array_item_string(wb, rrdhost_hostname(host));
46 - }
47 - buffer_json_array_close(wb);
48 -
49 - char host_guid[UUID_STR_LEN];
50 - if(streaming_topology_host_guid(host, host_guid, sizeof(host_guid)))
51 - buffer_json_member_add_string(wb, "netdata_machine_guid", host_guid);
52 -
53 - if(!UUIDiszero(host->node_id))
54 - buffer_json_member_add_uuid(wb, "netdata_node_id", host->node_id.uuid);
55 - }
56 - buffer_json_object_close(wb);
57 -}
58 -
40 static bool streaming_topology_host_guid(RRDHOST *host, char *dst, size_t dst_size) {
41 if(!dst || dst_size < UUID_STR_LEN)
42 return false;
@@ -268,252 +249,616 @@ static int streaming_topology_return_error(BUFFER *wb, char *function_copy, int
249 return status;
250 }
251
271 -// helpers: emit one summary field and one table column
272 -static void streaming_topology_emit_sf(BUFFER *wb, const char *key, const char *label, const char *source) {
273 - buffer_json_add_array_item_object(wb);
274 - buffer_json_member_add_string(wb, "key", key);
275 - buffer_json_member_add_string(wb, "label", label);
276 - buffer_json_member_add_array(wb, "sources");
277 - buffer_json_add_array_item_string(wb, source);
252 +static void streaming_topology_v1_emit_response_metadata(BUFFER *wb) {
253 + buffer_json_member_add_uint64(wb, "status", HTTP_RESP_OK);
254 + buffer_json_member_add_string(wb, "type", "topology");
255 + buffer_json_member_add_time_t(wb, "update_every", STREAMING_FUNCTION_UPDATE_EVERY);
256 + buffer_json_member_add_boolean(wb, "has_history", false);
257 + buffer_json_member_add_string(wb, "help", RRDFUNCTIONS_STREAMING_TOPOLOGY_HELP);
258 + buffer_json_member_add_array(wb, "accepted_params");
259 + {
260 + buffer_json_add_array_item_string(wb, "info");
261 + }
262 + buffer_json_array_close(wb);
263 + buffer_json_member_add_array(wb, "required_params");
264 buffer_json_array_close(wb);
279 - buffer_json_object_close(wb);
265 }
266
282 -static void streaming_topology_emit_col(BUFFER *wb, const char *key, const char *label, const char *type) {
283 - buffer_json_add_array_item_object(wb);
284 - buffer_json_member_add_string(wb, "key", key);
285 - buffer_json_member_add_string(wb, "label", label);
286 - if(type)
287 - buffer_json_member_add_string(wb, "type", type);
288 - buffer_json_object_close(wb);
267 +typedef struct streaming_topology_v1_actor {
268 + char actor_id[256];
269 + char type[32];
270 + char machine_guid[UUID_STR_LEN];
271 + char node_id[UUID_STR_LEN];
272 + char hostname[256];
273 + char display_name[256];
274 + char severity[32];
275 + char ephemerality[32];
276 + char ingest_status[64];
277 + char stream_status[64];
278 + char ml_status[64];
279 + char agent_name[128];
280 + char agent_version[128];
281 + char health_status[64];
282 + char os_name[128];
283 + char architecture[64];
284 + char cpu_count[32];
285 + uint64_t child_count;
286 + uint64_t retained_node_count;
287 + uint64_t health_critical;
288 + uint64_t health_warning;
289 + uint64_t health_clear;
290 + RRDHOST *host;
291 + bool synthetic;
292 +} STREAMING_TOPOLOGY_V1_ACTOR;
293 +
294 +typedef struct streaming_topology_v1_actor_label {
295 + uint64_t actor;
296 + char key[RRDLABELS_MAX_NAME_LENGTH + 1];
297 + char value[RRDLABELS_MAX_VALUE_LENGTH + 1];
298 + char source[32];
299 + char kind[32];
300 + bool has_value_index;
301 + uint64_t value_index;
302 +} STREAMING_TOPOLOGY_V1_ACTOR_LABEL;
303 +
304 +typedef struct streaming_topology_v1_link {
305 + uint64_t src_actor;
306 + uint64_t dst_actor;
307 + char type[32];
308 + char state[64];
309 + char port_name[256];
310 + uint64_t discovered_at_ut;
311 + uint64_t last_seen_ut;
312 + int64_t hops;
313 + uint64_t connections;
314 + uint64_t replication_instances;
315 + NETDATA_DOUBLE replication_completion;
316 + uint64_t collected_metrics;
317 + uint64_t collected_instances;
318 + uint64_t collected_contexts;
319 +} STREAMING_TOPOLOGY_V1_LINK;
320 +
321 +typedef struct streaming_topology_v1_stream_path_row {
322 + uint64_t actor;
323 + uint64_t path_actor;
324 + uint64_t path_index;
325 + char hostname[256];
326 + char host_id[UUID_STR_LEN];
327 + char node_id[UUID_STR_LEN];
328 + char claim_id[UUID_STR_LEN];
329 + int64_t hops;
330 + uint64_t since_ut;
331 + uint64_t first_time_ut;
332 + uint64_t start_time_ms;
333 + uint64_t shutdown_time_ms;
334 + uint64_t capabilities;
335 + uint64_t flags;
336 +} STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW;
337 +
338 +typedef struct streaming_topology_v1_retention_row {
339 + uint64_t actor;
340 + uint64_t observer_actor;
341 + char db_status[64];
342 + uint64_t db_from_ut;
343 + uint64_t db_to_ut;
344 + uint64_t db_duration;
345 + uint64_t db_metrics;
346 + uint64_t db_instances;
347 + uint64_t db_contexts;
348 +} STREAMING_TOPOLOGY_V1_RETENTION_ROW;
349 +
350 +typedef struct streaming_topology_v1_inbound_row {
351 + uint64_t parent_actor;
352 + uint64_t child_actor;
353 + bool has_source_actor;
354 + uint64_t source_actor;
355 + char received_type[32];
356 + char ingest_status[64];
357 + int64_t hops;
358 + uint64_t collected_metrics;
359 + uint64_t collected_instances;
360 + uint64_t collected_contexts;
361 + NETDATA_DOUBLE replication_completion;
362 + uint64_t ingest_age;
363 + char ssl[16];
364 + uint64_t alerts_critical;
365 + uint64_t alerts_warning;
366 +} STREAMING_TOPOLOGY_V1_INBOUND_ROW;
367 +
368 +typedef struct streaming_topology_v1_outbound_row {
369 + uint64_t sender_actor;
370 + uint64_t node_actor;
371 + bool has_destination_actor;
372 + uint64_t destination_actor;
373 + char stream_status[64];
374 + uint64_t stream_age;
375 + int64_t hops;
376 + char ssl[16];
377 + char compression[24];
378 + uint64_t collected_metrics;
379 + uint64_t collected_instances;
380 + uint64_t collected_contexts;
381 + uint64_t replication_instances;
382 + NETDATA_DOUBLE replication_completion;
383 +} STREAMING_TOPOLOGY_V1_OUTBOUND_ROW;
384 +
385 +typedef struct streaming_topology_v1_payload {
386 + STREAMING_TOPOLOGY_V1_ACTOR *actors;
387 + size_t actors_used;
388 + size_t actors_size;
389 +
390 + STREAMING_TOPOLOGY_V1_LINK *links;
391 + size_t links_used;
392 + size_t links_size;
393 +
394 + STREAMING_TOPOLOGY_V1_ACTOR_LABEL *labels;
395 + size_t labels_used;
396 + size_t labels_size;
397 +
398 + STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *stream_path_rows;
399 + size_t stream_path_used;
400 + size_t stream_path_size;
401 +
402 + STREAMING_TOPOLOGY_V1_RETENTION_ROW *retention_rows;
403 + size_t retention_used;
404 + size_t retention_size;
405 +
406 + STREAMING_TOPOLOGY_V1_INBOUND_ROW *inbound_rows;
407 + size_t inbound_used;
408 + size_t inbound_size;
409 +
410 + STREAMING_TOPOLOGY_V1_OUTBOUND_ROW *outbound_rows;
411 + size_t outbound_used;
412 + size_t outbound_size;
413 +
414 + DICTIONARY *actor_index;
415 + DICTIONARY *emitted_links;
416 +} STREAMING_TOPOLOGY_V1_PAYLOAD;
417 +
418 +static void streaming_topology_v1_strncpy(char *dst, size_t dst_size, const char *src) {
419 + if(!dst || !dst_size)
420 + return;
421 +
422 + strncpyz(dst, src ? src : "", dst_size - 1);
423 }
424
291 -static void streaming_topology_info_tab(BUFFER *wb) {
292 - buffer_json_member_add_array(wb, "modal_tabs");
293 - {
294 - buffer_json_add_array_item_object(wb);
295 - buffer_json_member_add_string(wb, "id", "info");
296 - buffer_json_member_add_string(wb, "label", "Info");
297 - buffer_json_object_close(wb);
425 +static void streaming_topology_v1_uuid_str(ND_UUID uuid, char *dst, size_t dst_size) {
426 + if(!dst || !dst_size)
427 + return;
428 +
429 + if(!streaming_topology_uuid_guid(uuid, dst, dst_size))
430 + dst[0] = '\0';
431 +}
432 +
433 +static void streaming_topology_v1_actor_index_set(STREAMING_TOPOLOGY_V1_PAYLOAD *payload, const char *actor_id, uint64_t index) {
434 + dictionary_set(payload->actor_index, actor_id, &index, sizeof(index));
435 +}
436 +
437 +static bool streaming_topology_v1_actor_index_get(STREAMING_TOPOLOGY_V1_PAYLOAD *payload, const char *actor_id, uint64_t *index) {
438 + uint64_t *stored = dictionary_get(payload->actor_index, actor_id);
439 + if(!stored)
440 + return false;
441 +
442 + if(index)
443 + *index = *stored;
444 +
445 + return true;
446 +}
447 +
448 +static STREAMING_TOPOLOGY_V1_ACTOR *streaming_topology_v1_add_actor(STREAMING_TOPOLOGY_V1_PAYLOAD *payload, const char *actor_id) {
449 + if(payload->actors_used == payload->actors_size) {
450 + size_t new_size = payload->actors_size ? payload->actors_size * 2 : 16;
451 + payload->actors = reallocz(payload->actors, new_size * sizeof(*payload->actors));
452 + payload->actors_size = new_size;
453 }
299 - buffer_json_array_close(wb);
454 +
455 + STREAMING_TOPOLOGY_V1_ACTOR *actor = &payload->actors[payload->actors_used];
456 + *actor = (STREAMING_TOPOLOGY_V1_ACTOR){ 0 };
457 + streaming_topology_v1_strncpy(actor->actor_id, sizeof(actor->actor_id), actor_id);
458 + streaming_topology_v1_actor_index_set(payload, actor_id, payload->actors_used);
459 + payload->actors_used++;
460 + return actor;
461 }
462
302 -// streaming path table: shared by all non-parent types
303 -static void streaming_topology_emit_streaming_path_table(BUFFER *wb, uint64_t order) {
304 - buffer_json_member_add_object(wb, "streaming_path");
305 - {
306 - buffer_json_member_add_string(wb, "label", "Streaming Path");
307 - buffer_json_member_add_string(wb, "source", "data");
308 - buffer_json_member_add_uint64(wb, "order", order);
309 - buffer_json_member_add_array(wb, "columns");
310 - {
311 - streaming_topology_emit_col(wb, "hostname", "Agent", NULL);
312 - streaming_topology_emit_col(wb, "hops", "Hops", "number");
313 - streaming_topology_emit_col(wb, "since", "Since", "timestamp");
314 - streaming_topology_emit_col(wb, "flags", "Flags", NULL);
315 - }
316 - buffer_json_array_close(wb);
463 +static STREAMING_TOPOLOGY_V1_LINK *streaming_topology_v1_add_link(STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
464 + if(payload->links_used == payload->links_size) {
465 + size_t new_size = payload->links_size ? payload->links_size * 2 : 16;
466 + payload->links = reallocz(payload->links, new_size * sizeof(*payload->links));
467 + payload->links_size = new_size;
468 }
318 - buffer_json_object_close(wb);
469 +
470 + STREAMING_TOPOLOGY_V1_LINK *link = &payload->links[payload->links_used++];
471 + *link = (STREAMING_TOPOLOGY_V1_LINK){ 0 };
472 + return link;
473 }
474
321 -// retention table: shared by all types
322 -static void streaming_topology_emit_retention_table(BUFFER *wb, uint64_t order, const char *name_label) {
323 - buffer_json_member_add_object(wb, "retention");
324 - {
325 - buffer_json_member_add_string(wb, "label", "Retention");
326 - buffer_json_member_add_string(wb, "source", "data");
327 - buffer_json_member_add_uint64(wb, "order", order);
328 - buffer_json_member_add_array(wb, "columns");
329 - {
330 - streaming_topology_emit_col(wb, "name", name_label, "actor_link");
331 - streaming_topology_emit_col(wb, "db_status", "Status", "badge");
332 - streaming_topology_emit_col(wb, "db_from", "From", "timestamp");
333 - streaming_topology_emit_col(wb, "db_to", "To", "timestamp");
334 - streaming_topology_emit_col(wb, "db_duration", "Duration", "duration");
335 - streaming_topology_emit_col(wb, "db_metrics", "Metrics", "number");
336 - streaming_topology_emit_col(wb, "db_instances", "Instances", "number");
337 - streaming_topology_emit_col(wb, "db_contexts", "Contexts", "number");
338 - }
339 - buffer_json_array_close(wb);
475 +static const char *streaming_topology_v1_label_source(RRDLABEL_SRC source) {
476 + if(source & RRDLABEL_SRC_K8S)
477 + return "k8s";
478 +
479 + if(source & RRDLABEL_SRC_ACLK)
480 + return "aclk";
481 +
482 + if(source & RRDLABEL_SRC_CONFIG)
483 + return "config";
484 +
485 + if(source & RRDLABEL_SRC_AUTO)
486 + return "auto";
487 +
488 + return "unknown";
489 +}
490 +
491 +static void streaming_topology_v1_add_actor_label_ex(
492 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
493 + uint64_t actor,
494 + const char *key,
495 + const char *value,
496 + const char *source,
497 + const char *kind,
498 + bool has_value_index,
499 + uint64_t value_index) {
500 + if(!payload || !key || !*key || !value || !*value)
501 + return;
502 +
503 + if(payload->labels_used == payload->labels_size) {
504 + size_t new_size = payload->labels_size ? payload->labels_size * 2 : 64;
505 + payload->labels = reallocz(payload->labels, new_size * sizeof(*payload->labels));
506 + payload->labels_size = new_size;
507 }
341 - buffer_json_object_close(wb);
508 +
509 + STREAMING_TOPOLOGY_V1_ACTOR_LABEL *row = &payload->labels[payload->labels_used++];
510 + *row = (STREAMING_TOPOLOGY_V1_ACTOR_LABEL){ 0 };
511 + row->actor = actor;
512 + streaming_topology_v1_strncpy(row->key, sizeof(row->key), key);
513 + streaming_topology_v1_strncpy(row->value, sizeof(row->value), value);
514 + streaming_topology_v1_strncpy(row->source, sizeof(row->source), source ? source : "producer");
515 + streaming_topology_v1_strncpy(row->kind, sizeof(row->kind), kind ? kind : "metadata");
516 + row->has_value_index = has_value_index;
517 + row->value_index = value_index;
518 }
519
344 -// parent actor presentation: intrinsic summary + inbound/retention tables
345 -static void streaming_topology_parent_presentation(BUFFER *wb) {
346 - buffer_json_member_add_array(wb, "summary_fields");
347 - {
348 - streaming_topology_emit_sf(wb, "display_name", "Name", "attributes.display_name");
349 - streaming_topology_emit_sf(wb, "node_type", "Type", "attributes.node_type");
350 - streaming_topology_emit_sf(wb, "agent_version", "Version", "attributes.agent_version");
351 - streaming_topology_emit_sf(wb, "os_name", "OS", "attributes.os_name");
352 - streaming_topology_emit_sf(wb, "architecture", "Arch", "attributes.architecture");
353 - streaming_topology_emit_sf(wb, "cpu_cores", "CPUs", "attributes.cpu_cores");
354 - streaming_topology_emit_sf(wb, "child_count", "Children", "attributes.child_count");
355 - streaming_topology_emit_sf(wb, "health_critical", "Alerts Critical", "attributes.health_critical");
356 - streaming_topology_emit_sf(wb, "health_warning", "Alerts Warning", "attributes.health_warning");
520 +static void streaming_topology_v1_add_actor_label(
521 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
522 + uint64_t actor,
523 + const char *key,
524 + const char *value,
525 + const char *kind) {
526 + streaming_topology_v1_add_actor_label_ex(payload, actor, key, value, "producer", kind, false, 0);
527 +}
528 +
529 +static void streaming_topology_v1_add_actor_label_uint(
530 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
531 + uint64_t actor,
532 + const char *key,
533 + uint64_t value,
534 + const char *kind) {
535 + char text[32];
536 + snprintfz(text, sizeof(text), "%"PRIu64, value);
537 + streaming_topology_v1_add_actor_label(payload, actor, key, text, kind);
538 +}
539 +
540 +struct streaming_topology_v1_host_label_ctx {
541 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload;
542 + uint64_t actor;
543 +};
544 +
545 +static int streaming_topology_v1_collect_host_label(
546 + const char *name,
547 + const char *value,
548 + RRDLABEL_SRC source,
549 + void *data) {
550 + struct streaming_topology_v1_host_label_ctx *ctx = data;
551 + streaming_topology_v1_add_actor_label_ex(
552 + ctx->payload, ctx->actor, name, value,
553 + streaming_topology_v1_label_source(source), "host_label", false, 0);
554 + return 0;
555 +}
556 +
557 +static void streaming_topology_v1_collect_actor_labels(
558 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
559 + uint64_t actor_index,
560 + STREAMING_TOPOLOGY_V1_ACTOR *actor) {
561 + if(!payload || !actor)
562 + return;
563 +
564 + streaming_topology_v1_add_actor_label(payload, actor_index, "display_name", actor->display_name, "identity");
565 + streaming_topology_v1_add_actor_label(payload, actor_index, "hostname", actor->hostname, "identity");
566 + streaming_topology_v1_add_actor_label(payload, actor_index, "machine_guid", actor->machine_guid, "identity");
567 + streaming_topology_v1_add_actor_label(payload, actor_index, "node_id", actor->node_id, "identity");
568 + streaming_topology_v1_add_actor_label(payload, actor_index, "type", actor->type, "metadata");
569 + streaming_topology_v1_add_actor_label(payload, actor_index, "severity", actor->severity, "status");
570 + streaming_topology_v1_add_actor_label(payload, actor_index, "ephemerality", actor->ephemerality, "metadata");
571 + streaming_topology_v1_add_actor_label(payload, actor_index, "ingest_status", actor->ingest_status, "status");
572 + streaming_topology_v1_add_actor_label(payload, actor_index, "stream_status", actor->stream_status, "status");
573 + streaming_topology_v1_add_actor_label(payload, actor_index, "ml_status", actor->ml_status, "status");
574 + streaming_topology_v1_add_actor_label(payload, actor_index, "agent_name", actor->agent_name, "metadata");
575 + streaming_topology_v1_add_actor_label(payload, actor_index, "agent_version", actor->agent_version, "metadata");
576 + streaming_topology_v1_add_actor_label(payload, actor_index, "health_status", actor->health_status, "status");
577 + streaming_topology_v1_add_actor_label(payload, actor_index, "os_name", actor->os_name, "system");
578 + streaming_topology_v1_add_actor_label(payload, actor_index, "architecture", actor->architecture, "system");
579 + streaming_topology_v1_add_actor_label(payload, actor_index, "cpu_count", actor->cpu_count, "system");
580 + streaming_topology_v1_add_actor_label_uint(payload, actor_index, "child_count", actor->child_count, "metric");
581 + streaming_topology_v1_add_actor_label_uint(
582 + payload, actor_index, "retained_node_count", actor->retained_node_count, "metric");
583 + streaming_topology_v1_add_actor_label_uint(payload, actor_index, "health_critical", actor->health_critical, "metric");
584 + streaming_topology_v1_add_actor_label_uint(payload, actor_index, "health_warning", actor->health_warning, "metric");
585 + streaming_topology_v1_add_actor_label_uint(payload, actor_index, "health_clear", actor->health_clear, "metric");
586 +
587 + if(actor->host && actor->host->rrdlabels) {
588 + struct streaming_topology_v1_host_label_ctx ctx = {
589 + .payload = payload,
590 + .actor = actor_index,
591 + };
592 + rrdlabels_walkthrough_read(actor->host->rrdlabels, streaming_topology_v1_collect_host_label, &ctx);
593 }
358 - buffer_json_array_close(wb);
594 +}
595
360 - buffer_json_member_add_object(wb, "tables");
361 - {
362 - // inbound: all nodes this parent has data for (rows = all known hosts)
363 - buffer_json_member_add_object(wb, "inbound");
364 - {
365 - buffer_json_member_add_string(wb, "label", "Inbound");
366 - buffer_json_member_add_string(wb, "source", "data");
367 - buffer_json_member_add_boolean(wb, "bullet_source", true);
368 - buffer_json_member_add_uint64(wb, "order", 0);
369 - buffer_json_member_add_array(wb, "columns");
370 - {
371 - streaming_topology_emit_col(wb, "name", "Node", "actor_link");
372 - streaming_topology_emit_col(wb, "received_from", "Source", "actor_link");
373 - streaming_topology_emit_col(wb, "node_type", "Type", "badge");
374 - streaming_topology_emit_col(wb, "ingest_status", "Ingest", "badge");
375 - streaming_topology_emit_col(wb, "hops", "Hops", "number");
376 - streaming_topology_emit_col(wb, "collected_metrics", "Metrics", "number");
377 - streaming_topology_emit_col(wb, "collected_instances", "Instances", "number");
378 - streaming_topology_emit_col(wb, "collected_contexts", "Contexts", "number");
379 - streaming_topology_emit_col(wb, "repl_completion", "Replication", "number");
380 - streaming_topology_emit_col(wb, "ingest_age", "Age", "duration");
381 - streaming_topology_emit_col(wb, "ssl", "SSL", "badge");
382 - streaming_topology_emit_col(wb, "alerts_critical", "Critical", "number");
383 - streaming_topology_emit_col(wb, "alerts_warning", "Warning", "number");
384 - }
385 - buffer_json_array_close(wb);
386 - }
387 - buffer_json_object_close(wb);
596 +static STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *streaming_topology_v1_add_stream_path_row(
597 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
598 + if(payload->stream_path_used == payload->stream_path_size) {
599 + size_t new_size = payload->stream_path_size ? payload->stream_path_size * 2 : 32;
600 + payload->stream_path_rows = reallocz(payload->stream_path_rows, new_size * sizeof(*payload->stream_path_rows));
601 + payload->stream_path_size = new_size;
602 + }
603
389 - // outbound: all nodes this parent streams to its parent (rows = all known hosts)
390 - buffer_json_member_add_object(wb, "outbound");
391 - {
392 - buffer_json_member_add_string(wb, "label", "Outbound");
393 - buffer_json_member_add_string(wb, "source", "data");
394 - buffer_json_member_add_uint64(wb, "order", 1);
395 - buffer_json_member_add_array(wb, "columns");
396 - {
397 - streaming_topology_emit_col(wb, "name", "Node", "actor_link");
398 - streaming_topology_emit_col(wb, "streamed_to", "Destination", "actor_link");
399 - streaming_topology_emit_col(wb, "node_type", "Type", "badge");
400 - streaming_topology_emit_col(wb, "stream_status", "Status", "badge");
401 - streaming_topology_emit_col(wb, "hops", "Hops", "number");
402 - streaming_topology_emit_col(wb, "ssl", "SSL", "badge");
403 - streaming_topology_emit_col(wb, "compression", "Compression", "badge");
404 - }
405 - buffer_json_array_close(wb);
406 - }
407 - buffer_json_object_close(wb);
604 + STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *row = &payload->stream_path_rows[payload->stream_path_used++];
605 + *row = (STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW){ 0 };
606 + return row;
607 +}
608
409 - streaming_topology_emit_retention_table(wb, 2, "Node");
410 - streaming_topology_emit_streaming_path_table(wb, 3);
609 +static STREAMING_TOPOLOGY_V1_RETENTION_ROW *streaming_topology_v1_add_retention_row(
610 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
611 + if(payload->retention_used == payload->retention_size) {
612 + size_t new_size = payload->retention_size ? payload->retention_size * 2 : 16;
613 + payload->retention_rows = reallocz(payload->retention_rows, new_size * sizeof(*payload->retention_rows));
614 + payload->retention_size = new_size;
615 }
412 - buffer_json_object_close(wb);
616
414 - streaming_topology_info_tab(wb);
617 + STREAMING_TOPOLOGY_V1_RETENTION_ROW *row = &payload->retention_rows[payload->retention_used++];
618 + *row = (STREAMING_TOPOLOGY_V1_RETENTION_ROW){ 0 };
619 + return row;
620 }
621
417 -// child actor presentation: intrinsic summary + streaming path/retention tables
418 -static void streaming_topology_child_presentation(BUFFER *wb) {
419 - buffer_json_member_add_array(wb, "summary_fields");
420 - {
421 - streaming_topology_emit_sf(wb, "display_name", "Name", "attributes.display_name");
422 - streaming_topology_emit_sf(wb, "node_type", "Type", "attributes.node_type");
423 - streaming_topology_emit_sf(wb, "agent_version", "Version", "attributes.agent_version");
424 - streaming_topology_emit_sf(wb, "os_name", "OS", "attributes.os_name");
425 - streaming_topology_emit_sf(wb, "architecture", "Arch", "attributes.architecture");
426 - streaming_topology_emit_sf(wb, "cpu_cores", "CPUs", "attributes.cpu_cores");
427 - streaming_topology_emit_sf(wb, "health_critical", "Alerts Critical", "attributes.health_critical");
428 - streaming_topology_emit_sf(wb, "health_warning", "Alerts Warning", "attributes.health_warning");
622 +static STREAMING_TOPOLOGY_V1_INBOUND_ROW *streaming_topology_v1_add_inbound_row(
623 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
624 + if(payload->inbound_used == payload->inbound_size) {
625 + size_t new_size = payload->inbound_size ? payload->inbound_size * 2 : 32;
626 + payload->inbound_rows = reallocz(payload->inbound_rows, new_size * sizeof(*payload->inbound_rows));
627 + payload->inbound_size = new_size;
628 }
430 - buffer_json_array_close(wb);
629
432 - buffer_json_member_add_object(wb, "tables");
433 - {
434 - streaming_topology_emit_streaming_path_table(wb, 0);
435 - streaming_topology_emit_retention_table(wb, 1, "Parent");
630 + STREAMING_TOPOLOGY_V1_INBOUND_ROW *row = &payload->inbound_rows[payload->inbound_used++];
631 + *row = (STREAMING_TOPOLOGY_V1_INBOUND_ROW){ 0 };
632 + return row;
633 +}
634 +
635 +static STREAMING_TOPOLOGY_V1_OUTBOUND_ROW *streaming_topology_v1_add_outbound_row(
636 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
637 + if(payload->outbound_used == payload->outbound_size) {
638 + size_t new_size = payload->outbound_size ? payload->outbound_size * 2 : 32;
639 + payload->outbound_rows = reallocz(payload->outbound_rows, new_size * sizeof(*payload->outbound_rows));
640 + payload->outbound_size = new_size;
641 }
642 +
643 + STREAMING_TOPOLOGY_V1_OUTBOUND_ROW *row = &payload->outbound_rows[payload->outbound_used++];
644 + *row = (STREAMING_TOPOLOGY_V1_OUTBOUND_ROW){ 0 };
645 + return row;
646 +}
647 +
648 +static bool streaming_topology_v1_link_seen(STREAMING_TOPOLOGY_V1_PAYLOAD *payload, uint64_t src, uint64_t dst, const char *type) {
649 + char link_key[128];
650 + snprintfz(link_key, sizeof(link_key), "%"PRIu64"|%"PRIu64"|%s", src, dst, type ? type : "");
651 + if(dictionary_get(payload->emitted_links, link_key))
652 + return true;
653 +
654 + uint8_t one = 1;
655 + dictionary_set(payload->emitted_links, link_key, &one, sizeof(one));
656 + return false;
657 +}
658 +
659 +static void streaming_topology_v1_free(STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
660 + if(!payload)
661 + return;
662 +
663 + freez(payload->actors);
664 + freez(payload->links);
665 + freez(payload->labels);
666 + freez(payload->stream_path_rows);
667 + freez(payload->retention_rows);
668 + freez(payload->inbound_rows);
669 + freez(payload->outbound_rows);
670 +
671 + if(payload->actor_index)
672 + dictionary_destroy(payload->actor_index);
673 + if(payload->emitted_links)
674 + dictionary_destroy(payload->emitted_links);
675 +
676 + *payload = (STREAMING_TOPOLOGY_V1_PAYLOAD){ 0 };
677 +}
678 +
679 +static void streaming_topology_v1_emit_column(
680 + BUFFER *wb,
681 + const char *id,
682 + const char *type,
683 + const char *role,
684 + bool nullable,
685 + const char *aggregation) {
686 + buffer_json_add_array_item_object(wb);
687 + buffer_json_member_add_string(wb, "id", id);
688 + buffer_json_member_add_string(wb, "type", type);
689 + if(nullable)
690 + buffer_json_member_add_boolean(wb, "nullable", true);
691 + if(role)
692 + buffer_json_member_add_string(wb, "role", role);
693 + if(aggregation)
694 + buffer_json_member_add_string(wb, "aggregation", aggregation);
695 buffer_json_object_close(wb);
696 +}
697
439 - streaming_topology_info_tab(wb);
698 +static void streaming_topology_v1_emit_values_start(BUFFER *wb) {
699 + buffer_json_add_array_item_object(wb);
700 + buffer_json_member_add_string(wb, "codec", "values");
701 + buffer_json_member_add_array(wb, "values");
702 }
703
442 -// vnode actor presentation: minimal summary + streaming path/retention
443 -static void streaming_topology_vnode_presentation(BUFFER *wb) {
444 - buffer_json_member_add_array(wb, "summary_fields");
445 - {
446 - streaming_topology_emit_sf(wb, "display_name", "Name", "attributes.display_name");
447 - streaming_topology_emit_sf(wb, "node_type", "Type", "attributes.node_type");
448 - streaming_topology_emit_sf(wb, "ephemerality", "Ephemerality", "attributes.ephemerality");
449 - }
704 +static void streaming_topology_v1_emit_values_end(BUFFER *wb) {
705 buffer_json_array_close(wb);
451 -
452 - buffer_json_member_add_object(wb, "tables");
453 - {
454 - streaming_topology_emit_streaming_path_table(wb, 0);
455 - streaming_topology_emit_retention_table(wb, 1, "Parent");
456 - }
706 buffer_json_object_close(wb);
707 +}
708
459 - streaming_topology_info_tab(wb);
709 +static void streaming_topology_v1_add_nullable_uint(BUFFER *wb, bool has_value, uint64_t value) {
710 + if(has_value)
711 + buffer_json_add_array_item_uint64(wb, value);
712 + else
713 + buffer_json_add_array_item_string(wb, NULL);
714 }
715
462 -// stale actor presentation: identity summary + streaming path/retention
463 -static void streaming_topology_stale_presentation(BUFFER *wb) {
464 - buffer_json_member_add_array(wb, "summary_fields");
465 - {
466 - streaming_topology_emit_sf(wb, "display_name", "Name", "attributes.display_name");
467 - streaming_topology_emit_sf(wb, "node_type", "Type", "attributes.node_type");
468 - streaming_topology_emit_sf(wb, "agent_version", "Version", "attributes.agent_version");
469 - streaming_topology_emit_sf(wb, "os_name", "OS", "attributes.os_name");
470 - streaming_topology_emit_sf(wb, "architecture", "Arch", "attributes.architecture");
716 +static uint64_t streaming_topology_v1_time_ut(time_t timestamp) {
717 + return timestamp > 0 ? (uint64_t)timestamp * USEC_PER_SEC : 0;
718 +}
719 +
720 +static uint64_t streaming_topology_v1_best_since_ut(RRDHOST_STATUS *status) {
721 + if(!status)
722 + return 0;
723 +
724 + if(status->ingest.since)
725 + return streaming_topology_v1_time_ut(status->ingest.since);
726 +
727 + if(status->stream.since)
728 + return streaming_topology_v1_time_ut(status->stream.since);
729 +
730 + if(status->db.first_time_s)
731 + return streaming_topology_v1_time_ut(status->db.first_time_s);
732 +
733 + return streaming_topology_v1_time_ut(netdata_start_time);
734 +}
735 +
736 +static uint64_t streaming_topology_v1_best_first_time_ut(RRDHOST_STATUS *status) {
737 + if(!status)
738 + return 0;
739 +
740 + if(status->db.first_time_s)
741 + return streaming_topology_v1_time_ut(status->db.first_time_s);
742 +
743 + if(status->ingest.since)
744 + return streaming_topology_v1_time_ut(status->ingest.since);
745 +
746 + if(status->stream.since)
747 + return streaming_topology_v1_time_ut(status->stream.since);
748 +
749 + return streaming_topology_v1_time_ut(netdata_start_time);
750 +}
751 +
752 +static bool streaming_topology_v1_status_has_db_counts(RRDHOST_STATUS *status) {
753 + return status && (status->db.metrics || status->db.instances || status->db.contexts);
754 +}
755 +
756 +static bool streaming_topology_v1_status_has_retention(RRDHOST_STATUS *status) {
757 + return status &&
758 + (status->db.first_time_s || status->db.last_time_s || streaming_topology_v1_status_has_db_counts(status));
759 +}
760 +
761 +static uint64_t streaming_topology_v1_count_local_retained_nodes(time_t now) {
762 + uint64_t retained = 0;
763 +
764 + RRDHOST *host;
765 + dfe_start_read(rrdhost_root_index, host) {
766 + RRDHOST_STATUS status;
767 + rrdhost_status(host, now, &status, RRDHOST_STATUS_ALL);
768 + if(streaming_topology_v1_status_has_retention(&status))
769 + retained++;
770 }
472 - buffer_json_array_close(wb);
771 + dfe_done(host);
772
474 - buffer_json_member_add_object(wb, "tables");
475 - {
476 - streaming_topology_emit_streaming_path_table(wb, 0);
477 - streaming_topology_emit_retention_table(wb, 1, "Parent");
773 + return retained;
774 +}
775 +
776 +static uint64_t streaming_topology_v1_retention_from_ut(RRDHOST_STATUS *status) {
777 + if(!status)
778 + return 0;
779 +
780 + if(status->db.first_time_s)
781 + return streaming_topology_v1_time_ut(status->db.first_time_s);
782 +
783 + if(streaming_topology_v1_status_has_db_counts(status))
784 + return streaming_topology_v1_best_first_time_ut(status);
785 +
786 + return 0;
787 +}
788 +
789 +static uint64_t streaming_topology_v1_retention_to_ut(RRDHOST_STATUS *status) {
790 + if(!status)
791 + return 0;
792 +
793 + if(status->db.last_time_s)
794 + return streaming_topology_v1_time_ut(status->db.last_time_s);
795 +
796 + if(streaming_topology_v1_status_has_db_counts(status))
797 + return streaming_topology_v1_time_ut(status->now);
798 +
799 + return 0;
800 +}
801 +
802 +static void streaming_topology_v1_add_timestamp(BUFFER *wb, uint64_t timestamp_ut) {
803 + if(timestamp_ut)
804 + buffer_json_add_array_item_datetime_rfc3339(wb, timestamp_ut, true);
805 + else
806 + buffer_json_add_array_item_string(wb, NULL);
807 +}
808 +
809 +static const char *streaming_topology_v1_node_type(
810 + RRDHOST *host,
811 + RRDHOST_STATUS *status,
812 + DICTIONARY *parent_child_count) {
813 + if(rrdhost_is_virtual(host))
814 + return "vnode";
815 +
816 + if(host != localhost && status->ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
817 + return "stale";
818 +
819 + uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, host);
820 + return (cc && *cc > 0) ? "parent" : "child";
821 +}
822 +
823 +static const char *streaming_topology_v1_severity(RRDHOST *host, RRDHOST_STATUS *status) {
824 + if(rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST))
825 + return "normal";
826 +
827 + switch(status->ingest.status) {
828 + case RRDHOST_INGEST_STATUS_OFFLINE:
829 + case RRDHOST_INGEST_STATUS_ARCHIVED:
830 + return "critical";
831 + default:
832 + break;
833 }
479 - buffer_json_object_close(wb);
834
481 - streaming_topology_info_tab(wb);
835 + if(status->stream.status == RRDHOST_STREAM_STATUS_OFFLINE &&
836 + status->stream.reason != STREAM_HANDSHAKE_SP_NO_DESTINATION)
837 + return "warning";
838 +
839 + return "normal";
840 }
841
484 -// Bug C synthesis context — passed through rrdhost_stream_path_visit
485 -// to emit synthetic upstream actors and multi-hop links.
486 -struct streaming_topology_synth_ctx {
487 - BUFFER *wb;
488 - DICTIONARY *local_actor_ids; // actor_id -> sentinel; local RRDHOST-backed actors
489 - DICTIONARY *emitted_actors; // actor_id -> sentinel; emitted actor dedup
490 - DICTIONARY *emitted_links; // "src|dst" -> sentinel; emitted link dedup
491 - size_t *actors_total;
492 - size_t *links_total;
493 - // for the multi-hop link pass — previous slot info
494 - bool has_prev;
495 - char prev_actor_id[256];
496 - char prev_hostname[256];
497 - time_t prev_since;
498 - time_t prev_first_time_t;
842 +struct streaming_topology_v1_synth_actor_ctx {
843 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload;
844 };
845
501 -// Visitor: emit a synthetic "parent" actor for each upstream path entry that
502 -// has no local RRDHOST-backed actor. Local actors are authoritative and are
503 -// always emitted by Phase 3.
504 -static bool streaming_topology_synth_actor_visitor(
505 - void *userdata, uint16_t index __maybe_unused,
846 +static bool streaming_topology_v1_collect_synth_actor(
847 + void *userdata,
848 + uint16_t index __maybe_unused,
849 STRING *hostname,
507 - ND_UUID host_id, ND_UUID node_id, ND_UUID claim_id __maybe_unused,
508 - int16_t hops,
509 - time_t since, time_t first_time_t,
510 - uint32_t start_time_ms, uint32_t shutdown_time_ms,
511 - STREAM_CAPABILITIES capabilities,
850 + ND_UUID host_id,
851 + ND_UUID node_id,
852 + ND_UUID claim_id __maybe_unused,
853 + int16_t hops __maybe_unused,
854 + time_t since __maybe_unused,
855 + time_t first_time_t __maybe_unused,
856 + uint32_t start_time_ms __maybe_unused,
857 + uint32_t shutdown_time_ms __maybe_unused,
858 + STREAM_CAPABILITIES capabilities __maybe_unused,
859 uint32_t flags __maybe_unused) {
860 + struct streaming_topology_v1_synth_actor_ctx *ctx = userdata;
861
514 - struct streaming_topology_synth_ctx *ctx = userdata;
515 -
516 - // Skip the visiting host's own self entry — Phase 3 emits it if needed.
862 if(UUIDeq(host_id, localhost->host_id))
863 return true;
864
@@ -523,127 +868,164 @@ static bool streaming_topology_synth_actor_visitor(
868
869 char actor_id[256];
870 streaming_topology_actor_id_from_guid(guid, actor_id, sizeof(actor_id));
526 -
527 - // Skip local RRDHOST-backed actors and already synthesized actors.
528 - if(dictionary_get(ctx->local_actor_ids, actor_id) || dictionary_get(ctx->emitted_actors, actor_id))
871 + if(streaming_topology_v1_actor_index_get(ctx->payload, actor_id, NULL))
872 return true;
873
531 - {
532 - uint8_t one = 1;
533 - dictionary_set(ctx->emitted_actors, actor_id, &one, sizeof(one));
534 - }
535 - (*ctx->actors_total)++;
536 -
537 - BUFFER *wb = ctx->wb;
538 - buffer_json_add_array_item_object(wb);
539 - {
540 - buffer_json_member_add_string(wb, "actor_id", actor_id);
541 - buffer_json_member_add_string(wb, "actor_type", "parent");
542 - buffer_json_member_add_string(wb, "layer", "infra");
543 - buffer_json_member_add_string(wb, "source", "streaming");
544 -
545 - // Match block — same key names as streaming_topology_add_host_match
546 - // (function-topology-streaming.c:45) so the cloud-frontend resolves
547 - // hostnames/GUIDs identically for synthetic and real actors.
548 - buffer_json_member_add_object(wb, "match");
549 - {
550 - buffer_json_member_add_array(wb, "hostnames");
551 - buffer_json_add_array_item_string(wb, string2str(hostname));
552 - buffer_json_array_close(wb);
553 -
554 - buffer_json_member_add_string(wb, "netdata_machine_guid", guid);
874 + STREAMING_TOPOLOGY_V1_ACTOR *actor = streaming_topology_v1_add_actor(ctx->payload, actor_id);
875 + actor->synthetic = true;
876 + streaming_topology_v1_strncpy(actor->type, sizeof(actor->type), "parent");
877 + streaming_topology_v1_strncpy(actor->machine_guid, sizeof(actor->machine_guid), guid);
878 + streaming_topology_v1_uuid_str(node_id, actor->node_id, sizeof(actor->node_id));
879 + streaming_topology_v1_strncpy(actor->hostname, sizeof(actor->hostname), string2str(hostname));
880 + streaming_topology_v1_strncpy(actor->display_name, sizeof(actor->display_name), string2str(hostname));
881 + streaming_topology_v1_strncpy(actor->severity, sizeof(actor->severity), "normal");
882 + streaming_topology_v1_strncpy(actor->ephemerality, sizeof(actor->ephemerality), "permanent");
883 + streaming_topology_v1_strncpy(actor->ingest_status, sizeof(actor->ingest_status), "unknown");
884 + streaming_topology_v1_strncpy(actor->stream_status, sizeof(actor->stream_status), "unknown");
885 + streaming_topology_v1_strncpy(actor->ml_status, sizeof(actor->ml_status), "unknown");
886 + streaming_topology_v1_strncpy(actor->health_status, sizeof(actor->health_status), "unknown");
887 + streaming_topology_v1_collect_actor_labels(ctx->payload, ctx->payload->actors_used - 1, actor);
888 + return true;
889 +}
890
556 - if(!UUIDiszero(node_id))
557 - buffer_json_member_add_uuid(wb, "netdata_node_id", node_id.uuid);
558 - }
559 - buffer_json_object_close(wb); // match
560 -
561 - // Limited attributes — synthesized from path data only. No local
562 - // rrdhost record exists for this remote upstream, so the agent has
563 - // no direct view of the parent's child_count, retention, OS info,
564 - // alerts, etc. The merge layer in Cloud will fill these in from the
565 - // parent's own response (where this actor is the "self" — its
566 - // agent_id matches this actor_id).
567 - buffer_json_member_add_object(wb, "attributes");
568 - {
569 - buffer_json_member_add_string(wb, "display_name", string2str(hostname));
570 - buffer_json_member_add_string(wb, "node_type", "parent");
571 - buffer_json_member_add_string(wb, "severity", "normal");
572 - buffer_json_member_add_uint64(wb, "child_count", 0);
573 - buffer_json_member_add_string(wb, "ephemerality", "permanent");
574 - buffer_json_member_add_int64(wb, "hops", hops);
575 - buffer_json_member_add_uint64(wb, "since", (uint64_t)since);
576 - buffer_json_member_add_uint64(wb, "first_time_t", (uint64_t)first_time_t);
577 - buffer_json_member_add_uint64(wb, "start_time_ms", start_time_ms);
578 - buffer_json_member_add_uint64(wb, "shutdown_time_ms", shutdown_time_ms);
579 - stream_capabilities_to_json_array(wb, capabilities, "capabilities");
891 +static void streaming_topology_v1_collect_actors(
892 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
893 + DICTIONARY *parent_child_count,
894 + uint64_t local_retained_node_count,
895 + time_t now) {
896 + RRDHOST *host;
897 + dfe_start_read(rrdhost_root_index, host) {
898 + RRDHOST_STATUS status;
899 + rrdhost_status(host, now, &status, RRDHOST_STATUS_ALL);
900 +
901 + char actor_id[256];
902 + streaming_topology_actor_id_for_host(host, actor_id, sizeof(actor_id));
903 +
904 + STREAMING_TOPOLOGY_V1_ACTOR *actor = streaming_topology_v1_add_actor(payload, actor_id);
905 + actor->host = host;
906 + streaming_topology_v1_strncpy(actor->type, sizeof(actor->type),
907 + streaming_topology_v1_node_type(host, &status, parent_child_count));
908 + streaming_topology_host_guid(host, actor->machine_guid, sizeof(actor->machine_guid));
909 + streaming_topology_v1_uuid_str(host->node_id, actor->node_id, sizeof(actor->node_id));
910 + streaming_topology_v1_strncpy(actor->hostname, sizeof(actor->hostname), rrdhost_hostname(host));
911 + streaming_topology_v1_strncpy(actor->display_name, sizeof(actor->display_name), rrdhost_hostname(host));
912 + streaming_topology_v1_strncpy(actor->severity, sizeof(actor->severity),
913 + streaming_topology_v1_severity(host, &status));
914 + streaming_topology_v1_strncpy(actor->ephemerality, sizeof(actor->ephemerality),
915 + rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST) ? "ephemeral" : "permanent");
916 + streaming_topology_v1_strncpy(actor->ingest_status, sizeof(actor->ingest_status),
917 + rrdhost_ingest_status_to_string(status.ingest.status));
918 + streaming_topology_v1_strncpy(actor->stream_status, sizeof(actor->stream_status),
919 + rrdhost_streaming_status_to_string(status.stream.status));
920 + streaming_topology_v1_strncpy(actor->ml_status, sizeof(actor->ml_status),
921 + rrdhost_ml_status_to_string(status.ml.status));
922 + streaming_topology_v1_strncpy(actor->agent_name, sizeof(actor->agent_name), rrdhost_program_name(host));
923 + streaming_topology_v1_strncpy(actor->agent_version, sizeof(actor->agent_version), rrdhost_program_version(host));
924 + streaming_topology_v1_strncpy(actor->health_status, sizeof(actor->health_status),
925 + rrdhost_health_status_to_string(status.health.status));
926 + rrdlabels_get_value_strcpyz(host->rrdlabels, actor->os_name, sizeof(actor->os_name), "_os_name");
927 + rrdlabels_get_value_strcpyz(host->rrdlabels, actor->architecture, sizeof(actor->architecture), "_architecture");
928 + rrdlabels_get_value_strcpyz(host->rrdlabels, actor->cpu_count, sizeof(actor->cpu_count), "_system_cores");
929 +
930 + uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, host);
931 + actor->child_count = cc ? *cc : 0;
932 + actor->retained_node_count = host == localhost ? local_retained_node_count : 0;
933 + if(status.health.status == RRDHOST_HEALTH_STATUS_RUNNING) {
934 + actor->health_critical = status.health.alerts.critical;
935 + actor->health_warning = status.health.alerts.warning;
936 + actor->health_clear = status.health.alerts.clear;
937 }
581 - buffer_json_object_close(wb); // attributes
938
583 - // Labels block — mirror Phase 3's set so the FE facet renderer sees
584 - // the same field keys for synthetic and real actors. Status fields
585 - // use path-derived defaults because no local RRDHOST exists.
586 - buffer_json_member_add_object(wb, "labels");
587 - {
588 - buffer_json_member_add_string(wb, "hostname", string2str(hostname));
589 - buffer_json_member_add_string(wb, "node_type", "parent");
590 - buffer_json_member_add_string(wb, "severity", "normal");
591 - buffer_json_member_add_string(wb, "ephemerality", "permanent");
592 - buffer_json_member_add_string(wb, "ingest_status", "online");
593 - buffer_json_member_add_string(wb, "stream_status", "online");
594 - buffer_json_member_add_string(wb, "ml_status", "unknown");
595 - buffer_json_member_add_string(wb, "display_name", string2str(hostname));
596 - }
597 - buffer_json_object_close(wb); // labels
598 -
599 - // The streaming_path field (highlight_path) is left empty for
600 - // synthesized actors — we don't have the full chain visible from
601 - // this side. The FE highlight_path still works for cross-actor
602 - // references because every actor_id is the canonical
603 - // netdata-machine-guid:<guid> form.
604 - buffer_json_member_add_array(wb, "streaming_path");
605 - buffer_json_array_close(wb);
939 + streaming_topology_v1_collect_actor_labels(payload, payload->actors_used - 1, actor);
940 + }
941 + dfe_done(host);
942
607 - buffer_json_member_add_object(wb, "tables");
608 - {
609 - buffer_json_member_add_array(wb, "streaming_path");
610 - {
611 - buffer_json_add_array_item_object(wb);
612 - {
613 - buffer_json_member_add_string(wb, "hostname", string2str(hostname));
614 - buffer_json_member_add_int64(wb, "hops", hops);
615 - buffer_json_member_add_uint64(wb, "since", (uint64_t)since);
616 - // flags omitted: STREAM_PATH_FLAGS bitmap is opaque to
617 - // synthesizing code; the table column renders blank for
618 - // synthesized rows. Real actors include flags via
619 - // rrdhost_stream_path_to_json.
620 - }
621 - buffer_json_object_close(wb);
622 - }
623 - buffer_json_array_close(wb);
624 - }
625 - buffer_json_object_close(wb); // tables
943 + struct streaming_topology_v1_synth_actor_ctx ctx = { .payload = payload };
944 + dfe_start_read(rrdhost_root_index, host) {
945 + rrdhost_stream_path_visit(host, 1, streaming_topology_v1_collect_synth_actor, &ctx);
946 }
627 - buffer_json_object_close(wb); // actor
947 + dfe_done(host);
948 +}
949
629 - return true;
950 +static bool streaming_topology_v1_actor_index_for_host(
951 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
952 + RRDHOST *host,
953 + uint64_t *index) {
954 + char actor_id[256];
955 + streaming_topology_actor_id_for_host(host, actor_id, sizeof(actor_id));
956 + return streaming_topology_v1_actor_index_get(payload, actor_id, index);
957 }
958
632 -// Visitor: emit a streaming link between consecutive path slots. Phase 4
633 -// registers the direct local-host links first; this pass uses emitted_links
634 -// for uniform dedup and adds whatever remains (localhost's own upstream link
635 -// and deeper multi-hop links).
636 -static bool streaming_topology_synth_link_visitor(
637 - void *userdata, uint16_t index __maybe_unused,
959 +static bool streaming_topology_v1_actor_index_for_uuid(
960 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
961 + ND_UUID host_id,
962 + uint64_t *index) {
963 + char actor_id[256];
964 + streaming_topology_actor_id_for_uuid(host_id, actor_id, sizeof(actor_id));
965 + return streaming_topology_v1_actor_index_get(payload, actor_id, index);
966 +}
967 +
968 +static void streaming_topology_v1_add_link_if_new(
969 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
970 + uint64_t src_actor,
971 + uint64_t dst_actor,
972 + const char *type,
973 + const char *state,
974 + const char *port_name,
975 + uint64_t discovered_at_ut,
976 + uint64_t last_seen_ut,
977 + int64_t hops,
978 + uint64_t connections,
979 + uint64_t replication_instances,
980 + NETDATA_DOUBLE replication_completion,
981 + uint64_t collected_metrics,
982 + uint64_t collected_instances,
983 + uint64_t collected_contexts) {
984 + if(streaming_topology_v1_link_seen(payload, src_actor, dst_actor, type))
985 + return;
986 +
987 + STREAMING_TOPOLOGY_V1_LINK *link = streaming_topology_v1_add_link(payload);
988 + link->src_actor = src_actor;
989 + link->dst_actor = dst_actor;
990 + streaming_topology_v1_strncpy(link->type, sizeof(link->type), type);
991 + streaming_topology_v1_strncpy(link->state, sizeof(link->state), state);
992 + streaming_topology_v1_strncpy(link->port_name, sizeof(link->port_name), port_name);
993 + link->discovered_at_ut = discovered_at_ut;
994 + link->last_seen_ut = last_seen_ut;
995 + link->hops = hops;
996 + link->connections = connections;
997 + link->replication_instances = replication_instances;
998 + link->replication_completion = replication_completion;
999 + link->collected_metrics = collected_metrics;
1000 + link->collected_instances = collected_instances;
1001 + link->collected_contexts = collected_contexts;
1002 +}
1003 +
1004 +struct streaming_topology_v1_synth_link_ctx {
1005 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload;
1006 + bool has_prev;
1007 + uint64_t prev_actor;
1008 + char prev_actor_id[256];
1009 + char prev_hostname[256];
1010 + time_t prev_since;
1011 + time_t prev_first_time_t;
1012 +};
1013 +
1014 +static bool streaming_topology_v1_collect_synth_link(
1015 + void *userdata,
1016 + uint16_t index __maybe_unused,
1017 STRING *hostname,
639 - ND_UUID host_id, ND_UUID node_id __maybe_unused, ND_UUID claim_id __maybe_unused,
1018 + ND_UUID host_id,
1019 + ND_UUID node_id __maybe_unused,
1020 + ND_UUID claim_id __maybe_unused,
1021 int16_t hops __maybe_unused,
641 - time_t since, time_t first_time_t,
642 - uint32_t start_time_ms __maybe_unused, uint32_t shutdown_time_ms __maybe_unused,
1022 + time_t since,
1023 + time_t first_time_t,
1024 + uint32_t start_time_ms __maybe_unused,
1025 + uint32_t shutdown_time_ms __maybe_unused,
1026 STREAM_CAPABILITIES capabilities __maybe_unused,
1027 uint32_t flags __maybe_unused) {
645 -
646 - struct streaming_topology_synth_ctx *ctx = userdata;
1028 + struct streaming_topology_v1_synth_link_ctx *ctx = userdata;
1029
1030 char guid[UUID_STR_LEN];
1031 if(!streaming_topology_uuid_guid(host_id, guid, sizeof(guid))) {
@@ -654,216 +1036,1082 @@ static bool streaming_topology_synth_link_visitor(
1036 char cur_actor_id[256];
1037 streaming_topology_actor_id_from_guid(guid, cur_actor_id, sizeof(cur_actor_id));
1038
657 - if(ctx->has_prev) {
658 - if(dictionary_get(ctx->emitted_actors, ctx->prev_actor_id) &&
659 - dictionary_get(ctx->emitted_actors, cur_actor_id)) {
660 -
661 - char link_key[600];
662 - snprintfz(link_key, sizeof(link_key), "%s|%s", ctx->prev_actor_id, cur_actor_id);
663 -
664 - if(!dictionary_get(ctx->emitted_links, link_key)) {
665 - uint8_t one = 1;
666 - dictionary_set(ctx->emitted_links, link_key, &one, sizeof(one));
667 - (*ctx->links_total)++;
1039 + uint64_t cur_actor;
1040 + if(!streaming_topology_v1_actor_index_get(ctx->payload, cur_actor_id, &cur_actor)) {
1041 + ctx->has_prev = false;
1042 + return true;
1043 + }
1044
669 - BUFFER *wb = ctx->wb;
670 - buffer_json_add_array_item_object(wb);
671 - {
672 - buffer_json_member_add_string(wb, "layer", "infra");
673 - buffer_json_member_add_string(wb, "protocol", "streaming");
674 - buffer_json_member_add_string(wb, "link_type", "streaming");
675 - buffer_json_member_add_string(wb, "src_actor_id", ctx->prev_actor_id);
676 - buffer_json_member_add_string(wb, "dst_actor_id", cur_actor_id);
677 - buffer_json_member_add_string(wb, "state", "online");
678 - // discovered_at / last_seen derive from STREAM_PATH
679 - // timestamps so the merge layer can reconcile views from
680 - // multiple agents reporting the same upstream link.
681 - buffer_json_member_add_datetime_rfc3339(wb, "discovered_at",
682 - ((uint64_t)(ctx->prev_first_time_t ? ctx->prev_first_time_t
683 - : ctx->prev_since)) * USEC_PER_SEC, true);
684 - buffer_json_member_add_datetime_rfc3339(wb, "last_seen",
685 - ((uint64_t)(since ? since : ctx->prev_since)) * USEC_PER_SEC, true);
686 -
687 - // port_name mirrors Phase 4: it is the source hostname.
688 - // Here the source is the previous stream-path slot.
689 - buffer_json_member_add_object(wb, "dst");
690 - {
691 - buffer_json_member_add_object(wb, "attributes");
692 - {
693 - buffer_json_member_add_string(wb, "port_name", ctx->prev_hostname);
694 - }
695 - buffer_json_object_close(wb);
696 - }
697 - buffer_json_object_close(wb);
698 - }
699 - buffer_json_object_close(wb); // link
700 - }
701 - }
1045 + if(ctx->has_prev) {
1046 + streaming_topology_v1_add_link_if_new(
1047 + ctx->payload,
1048 + ctx->prev_actor,
1049 + cur_actor,
1050 + "streaming",
1051 + "online",
1052 + ctx->prev_hostname,
1053 + ((uint64_t)(ctx->prev_first_time_t ? ctx->prev_first_time_t : ctx->prev_since)) * USEC_PER_SEC,
1054 + ((uint64_t)(since ? since : ctx->prev_since)) * USEC_PER_SEC,
1055 + 0, 0, 0, 0, 0, 0, 0);
1056 }
1057
1058 ctx->has_prev = true;
705 - snprintfz(ctx->prev_actor_id, sizeof(ctx->prev_actor_id), "%s", cur_actor_id);
706 - snprintfz(ctx->prev_hostname, sizeof(ctx->prev_hostname), "%s", string2str(hostname));
1059 + ctx->prev_actor = cur_actor;
1060 + streaming_topology_v1_strncpy(ctx->prev_actor_id, sizeof(ctx->prev_actor_id), cur_actor_id);
1061 + streaming_topology_v1_strncpy(ctx->prev_hostname, sizeof(ctx->prev_hostname), string2str(hostname));
1062 ctx->prev_since = since;
1063 ctx->prev_first_time_t = first_time_t;
1064 return true;
1065 }
1066
712 -int function_streaming_topology(BUFFER *wb, const char *function, BUFFER *payload __maybe_unused, const char *source __maybe_unused) {
713 - time_t now = now_realtime_sec();
714 - usec_t now_ut = now_realtime_usec();
1067 +static void streaming_topology_v1_collect_links(
1068 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
1069 + time_t now,
1070 + usec_t now_ut) {
1071 + char localhost_actor_id[256];
1072 + streaming_topology_actor_id_for_host(localhost, localhost_actor_id, sizeof(localhost_actor_id));
1073 + uint64_t localhost_actor = 0;
1074 + streaming_topology_v1_actor_index_get(payload, localhost_actor_id, &localhost_actor);
1075
716 - struct streaming_topology_options options = { 0 };
717 - streaming_topology_parse_options(function, &options);
718 - bool info_only = options.info_only;
719 - char *function_copy = options.function_copy;
1076 + RRDHOST *host;
1077 + dfe_start_read(rrdhost_root_index, host) {
1078 + if(host == localhost)
1079 + continue;
1080
721 - buffer_flush(wb);
722 - wb->content_type = CT_APPLICATION_JSON;
723 - buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_DEFAULT);
1081 + RRDHOST_STATUS status;
1082 + rrdhost_status(host, now, &status, RRDHOST_STATUS_ALL);
1083
725 - buffer_json_member_add_uint64(wb, "status", HTTP_RESP_OK);
726 - buffer_json_member_add_string(wb, "type", "topology");
727 - buffer_json_member_add_time_t(wb, "update_every", STREAMING_FUNCTION_UPDATE_EVERY);
728 - buffer_json_member_add_boolean(wb, "has_history", false);
729 - buffer_json_member_add_string(wb, "help", RRDFUNCTIONS_STREAMING_TOPOLOGY_HELP);
730 - buffer_json_member_add_array(wb, "accepted_params");
1084 + uint64_t src_actor;
1085 + if(!streaming_topology_v1_actor_index_for_host(payload, host, &src_actor))
1086 + continue;
1087 +
1088 + char target_actor_id[256] = "";
1089 + const char *link_type = NULL;
1090 + bool is_vnode = rrdhost_is_virtual(host);
1091 + ND_UUID link_ids[2];
1092 + uint16_t link_n = streaming_topology_get_path_ids(host, 0, link_ids, 2);
1093 +
1094 + if(is_vnode) {
1095 + snprintfz(target_actor_id, sizeof(target_actor_id), "%s", localhost_actor_id);
1096 + link_type = "virtual";
1097 + }
1098 + else if(link_n >= 2) {
1099 + streaming_topology_actor_id_for_uuid(link_ids[1], target_actor_id, sizeof(target_actor_id));
1100 + link_type = "streaming";
1101 + }
1102 + else {
1103 + snprintfz(target_actor_id, sizeof(target_actor_id), "%s", localhost_actor_id);
1104 + link_type = "stale";
1105 + }
1106 +
1107 + uint64_t dst_actor;
1108 + if(!streaming_topology_v1_actor_index_get(payload, target_actor_id, &dst_actor))
1109 + continue;
1110 +
1111 + streaming_topology_v1_add_link_if_new(
1112 + payload,
1113 + src_actor,
1114 + dst_actor,
1115 + link_type,
1116 + rrdhost_ingest_status_to_string(status.ingest.status),
1117 + rrdhost_hostname(host),
1118 + ((uint64_t)(status.ingest.since ? status.ingest.since : now)) * USEC_PER_SEC,
1119 + now_ut,
1120 + status.ingest.hops,
1121 + strcmp(link_type, "virtual") != 0 ? status.host->stream.rcv.status.connections : 0,
1122 + strcmp(link_type, "virtual") != 0 ? status.ingest.replication.instances : 0,
1123 + strcmp(link_type, "virtual") != 0 ? status.ingest.replication.completion : 0,
1124 + strcmp(link_type, "virtual") != 0 ? status.ingest.collected.metrics : 0,
1125 + strcmp(link_type, "virtual") != 0 ? status.ingest.collected.instances : 0,
1126 + strcmp(link_type, "virtual") != 0 ? status.ingest.collected.contexts : 0);
1127 + }
1128 + dfe_done(host);
1129 +
1130 + dfe_start_read(rrdhost_root_index, host) {
1131 + struct streaming_topology_v1_synth_link_ctx ctx = {
1132 + .payload = payload,
1133 + .has_prev = false,
1134 + };
1135 + rrdhost_stream_path_visit(host, 0, streaming_topology_v1_collect_synth_link, &ctx);
1136 + }
1137 + dfe_done(host);
1138 +}
1139 +
1140 +struct streaming_topology_v1_stream_path_ctx {
1141 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload;
1142 + uint64_t actor;
1143 + bool seen_localhost;
1144 + bool emitted;
1145 + uint16_t next_index;
1146 +};
1147 +
1148 +static bool streaming_topology_v1_collect_stream_path_row(
1149 + void *userdata,
1150 + uint16_t index,
1151 + STRING *hostname,
1152 + ND_UUID host_id,
1153 + ND_UUID node_id,
1154 + ND_UUID claim_id,
1155 + int16_t hops,
1156 + time_t since,
1157 + time_t first_time_t,
1158 + uint32_t start_time_ms,
1159 + uint32_t shutdown_time_ms,
1160 + STREAM_CAPABILITIES capabilities,
1161 + uint32_t flags) {
1162 + struct streaming_topology_v1_stream_path_ctx *ctx = userdata;
1163 +
1164 + STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *row = streaming_topology_v1_add_stream_path_row(ctx->payload);
1165 + row->actor = ctx->actor;
1166 + row->path_actor = ctx->actor;
1167 + row->path_index = index;
1168 + char path_actor_guid[UUID_STR_LEN];
1169 + if(streaming_topology_uuid_guid(host_id, path_actor_guid, sizeof(path_actor_guid))) {
1170 + char path_actor_id[256];
1171 + streaming_topology_actor_id_from_guid(path_actor_guid, path_actor_id, sizeof(path_actor_id));
1172 + streaming_topology_v1_actor_index_get(ctx->payload, path_actor_id, &row->path_actor);
1173 + }
1174 + streaming_topology_v1_strncpy(row->hostname, sizeof(row->hostname), string2str(hostname));
1175 + streaming_topology_v1_uuid_str(host_id, row->host_id, sizeof(row->host_id));
1176 + streaming_topology_v1_uuid_str(node_id, row->node_id, sizeof(row->node_id));
1177 + streaming_topology_v1_uuid_str(claim_id, row->claim_id, sizeof(row->claim_id));
1178 + row->hops = hops;
1179 + row->since_ut = since > 0 ? (uint64_t)since * USEC_PER_SEC : 0;
1180 + row->first_time_ut = first_time_t > 0 ? (uint64_t)first_time_t * USEC_PER_SEC : 0;
1181 + row->start_time_ms = start_time_ms;
1182 + row->shutdown_time_ms = shutdown_time_ms;
1183 + row->capabilities = capabilities;
1184 + row->flags = flags;
1185 + if(UUIDeq(host_id, localhost->host_id))
1186 + ctx->seen_localhost = true;
1187 + ctx->emitted = true;
1188 + ctx->next_index = index + 1;
1189 + return true;
1190 +}
1191 +
1192 +static void streaming_topology_v1_collect_actor_detail_rows(
1193 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
1194 + DICTIONARY *parent_descendants,
1195 + time_t now) {
1196 + uint64_t localhost_actor = 0;
1197 + streaming_topology_v1_actor_index_for_host(payload, localhost, &localhost_actor);
1198 +
1199 + for(size_t i = 0; i < payload->actors_used; i++) {
1200 + STREAMING_TOPOLOGY_V1_ACTOR *actor = &payload->actors[i];
1201 + if(!actor->host)
1202 + continue;
1203 +
1204 + RRDHOST_STATUS status;
1205 + rrdhost_status(actor->host, now, &status, RRDHOST_STATUS_ALL);
1206 +
1207 + size_t path_rows_start = payload->stream_path_used;
1208 + struct streaming_topology_v1_stream_path_ctx sp_ctx = {
1209 + .payload = payload,
1210 + .actor = i,
1211 + .emitted = false,
1212 + };
1213 + rrdhost_stream_path_visit(actor->host, 0, streaming_topology_v1_collect_stream_path_row, &sp_ctx);
1214 + for(size_t pi = path_rows_start; pi < payload->stream_path_used; pi++) {
1215 + STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *path_row = &payload->stream_path_rows[pi];
1216 + if(!path_row->since_ut)
1217 + path_row->since_ut = streaming_topology_v1_best_since_ut(&status);
1218 + if(!path_row->first_time_ut)
1219 + path_row->first_time_ut = streaming_topology_v1_best_first_time_ut(&status);
1220 + }
1221 + if(sp_ctx.emitted && !sp_ctx.seen_localhost) {
1222 + // Match the legacy highlight path helper: stored paths do not
1223 + // always carry the local agent, but rendered paths need it.
1224 + STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *row = streaming_topology_v1_add_stream_path_row(payload);
1225 + row->actor = i;
1226 + row->path_actor = localhost_actor;
1227 + row->path_index = sp_ctx.next_index;
1228 + streaming_topology_v1_strncpy(row->hostname, sizeof(row->hostname), rrdhost_hostname(localhost));
1229 + streaming_topology_host_guid(localhost, row->host_id, sizeof(row->host_id));
1230 + streaming_topology_v1_uuid_str(localhost->node_id, row->node_id, sizeof(row->node_id));
1231 + row->hops = status.ingest.hops;
1232 + row->since_ut = streaming_topology_v1_best_since_ut(&status);
1233 + row->first_time_ut = streaming_topology_v1_best_first_time_ut(&status);
1234 + }
1235 + else if(!sp_ctx.emitted) {
1236 + STREAMING_TOPOLOGY_V1_STREAM_PATH_ROW *row = streaming_topology_v1_add_stream_path_row(payload);
1237 + row->actor = i;
1238 + row->path_actor = i;
1239 + row->path_index = 0;
1240 + streaming_topology_v1_strncpy(row->hostname, sizeof(row->hostname), actor->hostname);
1241 + streaming_topology_v1_strncpy(row->host_id, sizeof(row->host_id), actor->machine_guid);
1242 + streaming_topology_v1_strncpy(row->node_id, sizeof(row->node_id), actor->node_id);
1243 + row->hops = status.ingest.hops;
1244 + row->since_ut = streaming_topology_v1_best_since_ut(&status);
1245 + row->first_time_ut = streaming_topology_v1_best_first_time_ut(&status);
1246 + }
1247 +
1248 + if(streaming_topology_v1_status_has_retention(&status)) {
1249 + STREAMING_TOPOLOGY_V1_RETENTION_ROW *retention = streaming_topology_v1_add_retention_row(payload);
1250 + retention->actor = i;
1251 + retention->observer_actor = localhost_actor;
1252 + streaming_topology_v1_strncpy(retention->db_status, sizeof(retention->db_status),
1253 + rrdhost_db_status_to_string(status.db.status));
1254 + retention->db_from_ut = streaming_topology_v1_retention_from_ut(&status);
1255 + retention->db_to_ut = streaming_topology_v1_retention_to_ut(&status);
1256 + retention->db_duration =
1257 + retention->db_from_ut && retention->db_to_ut && retention->db_to_ut > retention->db_from_ut ?
1258 + (retention->db_to_ut - retention->db_from_ut) / USEC_PER_SEC : 0;
1259 + retention->db_metrics = status.db.metrics;
1260 + retention->db_instances = status.db.instances;
1261 + retention->db_contexts = status.db.contexts;
1262 + }
1263 + }
1264 +
1265 + for(size_t i = 0; i < payload->actors_used; i++) {
1266 + STREAMING_TOPOLOGY_V1_ACTOR *parent = &payload->actors[i];
1267 + if(!parent->host || parent->child_count == 0)
1268 + continue;
1269 +
1270 + struct streaming_topology_descendant_list *nodes =
1271 + streaming_topology_descendants_get(parent_descendants, parent->host);
1272 + if(!nodes)
1273 + continue;
1274 +
1275 + for(size_t j = 0; j < nodes->used; j++) {
1276 + struct streaming_topology_descendant *descendant = &nodes->items[j];
1277 + if(!descendant->host || descendant->host == parent->host)
1278 + continue;
1279 +
1280 + uint64_t child_actor;
1281 + if(!streaming_topology_v1_actor_index_for_host(payload, descendant->host, &child_actor))
1282 + continue;
1283 +
1284 + RRDHOST_STATUS status;
1285 + rrdhost_status(descendant->host, now, &status, RRDHOST_STATUS_ALL);
1286 +
1287 + STREAMING_TOPOLOGY_V1_INBOUND_ROW *row = streaming_topology_v1_add_inbound_row(payload);
1288 + row->parent_actor = i;
1289 + row->child_actor = child_actor;
1290 + streaming_topology_v1_strncpy(row->received_type, sizeof(row->received_type),
1291 + streaming_topology_received_type_to_string((enum streaming_topology_received_type)descendant->type));
1292 + streaming_topology_v1_strncpy(row->ingest_status, sizeof(row->ingest_status),
1293 + rrdhost_ingest_status_to_string(status.ingest.status));
1294 + row->hops = status.ingest.hops;
1295 + row->collected_metrics = status.ingest.collected.metrics;
1296 + row->collected_instances = status.ingest.collected.instances;
1297 + row->collected_contexts = status.ingest.collected.contexts;
1298 + row->replication_completion = status.ingest.replication.completion;
1299 + row->ingest_age = status.ingest.since ? (uint64_t)(status.now - status.ingest.since) : 0;
1300 + streaming_topology_v1_strncpy(row->ssl, sizeof(row->ssl), status.ingest.ssl ? "SSL" : "PLAIN");
1301 + row->alerts_critical =
1302 + status.health.status == RRDHOST_HEALTH_STATUS_RUNNING ? status.health.alerts.critical : 0;
1303 + row->alerts_warning =
1304 + status.health.status == RRDHOST_HEALTH_STATUS_RUNNING ? status.health.alerts.warning : 0;
1305 +
1306 + if(descendant->source_local) {
1307 + row->has_source_actor = true;
1308 + row->source_actor = child_actor;
1309 + }
1310 + else if(!UUIDiszero(descendant->source_uuid)) {
1311 + char source_actor_id[256];
1312 + streaming_topology_actor_id_for_uuid(descendant->source_uuid, source_actor_id, sizeof(source_actor_id));
1313 + row->has_source_actor =
1314 + streaming_topology_v1_actor_index_get(payload, source_actor_id, &row->source_actor);
1315 + }
1316 + }
1317 + }
1318 +
1319 + RRDHOST_STATUS sender_status;
1320 + rrdhost_status(localhost, now, &sender_status, RRDHOST_STATUS_ALL);
1321 +
1322 + for(size_t i = 0; i < payload->actors_used; i++) {
1323 + STREAMING_TOPOLOGY_V1_ACTOR *actor = &payload->actors[i];
1324 + if(!actor->host)
1325 + continue;
1326 +
1327 + RRDHOST *path_host = rrdhost_is_virtual(actor->host) ? localhost : actor->host;
1328 + ND_UUID path[128];
1329 + uint16_t path_n = streaming_topology_get_path_ids(path_host, 0, path, 128);
1330 +
1331 + uint64_t destination_actor = 0;
1332 + bool has_destination_actor = false;
1333 + for(uint16_t pi = 0; pi + 1 < path_n; pi++) {
1334 + if(UUIDeq(path[pi], localhost->host_id)) {
1335 + has_destination_actor =
1336 + streaming_topology_v1_actor_index_for_uuid(payload, path[pi + 1], &destination_actor);
1337 + break;
1338 + }
1339 + }
1340 +
1341 + if(!has_destination_actor)
1342 + continue;
1343 +
1344 + RRDHOST_STATUS node_status;
1345 + rrdhost_status(actor->host, now, &node_status, RRDHOST_STATUS_ALL);
1346 +
1347 + STREAMING_TOPOLOGY_V1_OUTBOUND_ROW *row = streaming_topology_v1_add_outbound_row(payload);
1348 + row->sender_actor = localhost_actor;
1349 + row->node_actor = i;
1350 + row->has_destination_actor = true;
1351 + row->destination_actor = destination_actor;
1352 + streaming_topology_v1_strncpy(row->stream_status, sizeof(row->stream_status),
1353 + rrdhost_streaming_status_to_string(sender_status.stream.status));
1354 + row->stream_age =
1355 + sender_status.stream.since && sender_status.now >= sender_status.stream.since ?
1356 + (uint64_t)(sender_status.now - sender_status.stream.since) : 0;
1357 + row->hops = sender_status.stream.hops;
1358 + streaming_topology_v1_strncpy(row->ssl, sizeof(row->ssl), sender_status.stream.ssl ? "SSL" : "PLAIN");
1359 + streaming_topology_v1_strncpy(row->compression, sizeof(row->compression),
1360 + sender_status.stream.compression ? "COMPRESSED" : "UNCOMPRESSED");
1361 + row->collected_metrics = node_status.ingest.collected.metrics ?
1362 + node_status.ingest.collected.metrics : node_status.db.metrics;
1363 + row->collected_instances = node_status.ingest.collected.instances ?
1364 + node_status.ingest.collected.instances : node_status.db.instances;
1365 + row->collected_contexts = node_status.ingest.collected.contexts ?
1366 + node_status.ingest.collected.contexts : node_status.db.contexts;
1367 + row->replication_instances = node_status.ingest.replication.instances;
1368 + row->replication_completion = node_status.ingest.replication.completion;
1369 + }
1370 +}
1371 +
1372 +static void streaming_topology_v1_emit_actor_columns(BUFFER *wb) {
1373 + streaming_topology_v1_emit_column(wb, "id", "string", "identity", false, NULL);
1374 + streaming_topology_v1_emit_column(wb, "type", "string", "group_key", false, NULL);
1375 + streaming_topology_v1_emit_column(wb, "layer", "string", "group_key", false, NULL);
1376 + streaming_topology_v1_emit_column(wb, "machine_guid", "string", "merge_identity", true, NULL);
1377 + streaming_topology_v1_emit_column(wb, "node_id", "string", "merge_identity", true, NULL);
1378 + streaming_topology_v1_emit_column(wb, "hostname", "string", "attribute", true, NULL);
1379 + streaming_topology_v1_emit_column(wb, "display_name", "string", "attribute", true, NULL);
1380 + streaming_topology_v1_emit_column(wb, "severity", "string", "attribute", true, NULL);
1381 + streaming_topology_v1_emit_column(wb, "ephemerality", "string", "attribute", true, NULL);
1382 + streaming_topology_v1_emit_column(wb, "ingest_status", "string", "attribute", true, NULL);
1383 + streaming_topology_v1_emit_column(wb, "stream_status", "string", "attribute", true, NULL);
1384 + streaming_topology_v1_emit_column(wb, "ml_status", "string", "attribute", true, NULL);
1385 + streaming_topology_v1_emit_column(wb, "agent_name", "string", "attribute", true, NULL);
1386 + streaming_topology_v1_emit_column(wb, "agent_version", "string", "attribute", true, NULL);
1387 + streaming_topology_v1_emit_column(wb, "health_status", "string", "attribute", true, NULL);
1388 + streaming_topology_v1_emit_column(wb, "os_name", "string", "attribute", true, NULL);
1389 + streaming_topology_v1_emit_column(wb, "architecture", "string", "attribute", true, NULL);
1390 + streaming_topology_v1_emit_column(wb, "cpu_count", "string", "attribute", true, NULL);
1391 + streaming_topology_v1_emit_column(wb, "child_count", "uint", "metric", false, "sum");
1392 + streaming_topology_v1_emit_column(wb, "retained_node_count", "uint", "metric", false, "max");
1393 + streaming_topology_v1_emit_column(wb, "health_critical", "uint", "metric", false, "sum");
1394 + streaming_topology_v1_emit_column(wb, "health_warning", "uint", "metric", false, "sum");
1395 + streaming_topology_v1_emit_column(wb, "health_clear", "uint", "metric", false, "sum");
1396 +}
1397 +
1398 +static void streaming_topology_v1_emit_link_columns(BUFFER *wb) {
1399 + streaming_topology_v1_emit_column(wb, "src_actor", "actor_ref", "reference", false, NULL);
1400 + streaming_topology_v1_emit_column(wb, "dst_actor", "actor_ref", "reference", false, NULL);
1401 + streaming_topology_v1_emit_column(wb, "type", "string", "group_key", false, NULL);
1402 + streaming_topology_v1_emit_column(wb, "state", "string", "attribute", true, NULL);
1403 + streaming_topology_v1_emit_column(wb, "port_name", "string", "attribute", true, NULL);
1404 + streaming_topology_v1_emit_column(wb, "discovered_at", "timestamp", "timestamp", true, NULL);
1405 + streaming_topology_v1_emit_column(wb, "last_seen", "timestamp", "timestamp", true, NULL);
1406 + streaming_topology_v1_emit_column(wb, "hops", "int", "metric", false, "max");
1407 + streaming_topology_v1_emit_column(wb, "evidence_count", "uint", "metric", false, "sum");
1408 + streaming_topology_v1_emit_column(wb, "connections", "uint", "metric", false, "sum");
1409 + streaming_topology_v1_emit_column(wb, "replication_instances", "uint", "metric", false, "sum");
1410 + streaming_topology_v1_emit_column(wb, "replication_completion", "float", "metric", false, "avg");
1411 + streaming_topology_v1_emit_column(wb, "collected_metrics", "uint", "metric", false, "sum");
1412 + streaming_topology_v1_emit_column(wb, "collected_instances", "uint", "metric", false, "sum");
1413 + streaming_topology_v1_emit_column(wb, "collected_contexts", "uint", "metric", false, "sum");
1414 +}
1415 +
1416 +static void streaming_topology_v1_emit_actor_label_columns(BUFFER *wb) {
1417 + streaming_topology_v1_emit_column(wb, "actor", "actor_ref", "reference", false, NULL);
1418 + streaming_topology_v1_emit_column(wb, "key", "string", "attribute", false, NULL);
1419 + streaming_topology_v1_emit_column(wb, "value", "string", "attribute", false, NULL);
1420 + streaming_topology_v1_emit_column(wb, "source", "string", "attribute", true, NULL);
1421 + streaming_topology_v1_emit_column(wb, "kind", "string", "attribute", true, NULL);
1422 + streaming_topology_v1_emit_column(wb, "value_index", "uint", "attribute", true, NULL);
1423 +}
1424 +
1425 +static void streaming_topology_v1_emit_evidence_columns(BUFFER *wb) {
1426 + streaming_topology_v1_emit_column(wb, "link", "link_ref", "reference", false, NULL);
1427 + streaming_topology_v1_emit_column(wb, "src_actor", "actor_ref", "reference", false, NULL);
1428 + streaming_topology_v1_emit_column(wb, "dst_actor", "actor_ref", "reference", false, NULL);
1429 + streaming_topology_v1_emit_column(wb, "type", "string", "group_key", false, NULL);
1430 + streaming_topology_v1_emit_column(wb, "state", "string", "attribute", true, NULL);
1431 + streaming_topology_v1_emit_column(wb, "port_name", "string", "attribute", true, NULL);
1432 + streaming_topology_v1_emit_column(wb, "discovered_at", "timestamp", "timestamp", true, NULL);
1433 + streaming_topology_v1_emit_column(wb, "last_seen", "timestamp", "timestamp", true, NULL);
1434 + streaming_topology_v1_emit_column(wb, "hops", "int", "metric", false, "max");
1435 + streaming_topology_v1_emit_column(wb, "connections", "uint", "metric", false, "sum");
1436 + streaming_topology_v1_emit_column(wb, "replication_instances", "uint", "metric", false, "sum");
1437 + streaming_topology_v1_emit_column(wb, "replication_completion", "float", "metric", false, "avg");
1438 + streaming_topology_v1_emit_column(wb, "collected_metrics", "uint", "metric", false, "sum");
1439 + streaming_topology_v1_emit_column(wb, "collected_instances", "uint", "metric", false, "sum");
1440 + streaming_topology_v1_emit_column(wb, "collected_contexts", "uint", "metric", false, "sum");
1441 +}
1442 +
1443 +static void streaming_topology_v1_emit_stream_path_columns(BUFFER *wb) {
1444 + streaming_topology_v1_emit_column(wb, "actor", "actor_ref", "reference", false, NULL);
1445 + streaming_topology_v1_emit_column(wb, "path_actor", "actor_ref", "reference", false, NULL);
1446 + streaming_topology_v1_emit_column(wb, "path_index", "uint", NULL, false, NULL);
1447 + streaming_topology_v1_emit_column(wb, "hostname", "string", "attribute", true, NULL);
1448 + streaming_topology_v1_emit_column(wb, "host_id", "string", "merge_identity", true, NULL);
1449 + streaming_topology_v1_emit_column(wb, "node_id", "string", "merge_identity", true, NULL);
1450 + streaming_topology_v1_emit_column(wb, "claim_id", "string", "attribute", true, NULL);
1451 + streaming_topology_v1_emit_column(wb, "hops", "int", "metric", false, "max");
1452 + streaming_topology_v1_emit_column(wb, "since", "timestamp", "timestamp", true, NULL);
1453 + streaming_topology_v1_emit_column(wb, "first_time", "timestamp", "timestamp", true, NULL);
1454 + streaming_topology_v1_emit_column(wb, "start_time_ms", "uint", "metric", false, "max");
1455 + streaming_topology_v1_emit_column(wb, "shutdown_time_ms", "uint", "metric", false, "max");
1456 + streaming_topology_v1_emit_column(wb, "capabilities", "uint", "attribute", false, NULL);
1457 + streaming_topology_v1_emit_column(wb, "flags", "uint", "attribute", false, NULL);
1458 +}
1459 +
1460 +static void streaming_topology_v1_emit_retention_columns(BUFFER *wb) {
1461 + streaming_topology_v1_emit_column(wb, "actor", "actor_ref", "reference", false, NULL);
1462 + streaming_topology_v1_emit_column(wb, "observer_actor", "actor_ref", "reference", false, NULL);
1463 + streaming_topology_v1_emit_column(wb, "db_status", "string", "attribute", true, NULL);
1464 + streaming_topology_v1_emit_column(wb, "db_from", "timestamp", "timestamp", true, NULL);
1465 + streaming_topology_v1_emit_column(wb, "db_to", "timestamp", "timestamp", true, NULL);
1466 + streaming_topology_v1_emit_column(wb, "db_duration", "duration", "metric", false, "max");
1467 + streaming_topology_v1_emit_column(wb, "db_metrics", "uint", "metric", false, "sum");
1468 + streaming_topology_v1_emit_column(wb, "db_instances", "uint", "metric", false, "sum");
1469 + streaming_topology_v1_emit_column(wb, "db_contexts", "uint", "metric", false, "sum");
1470 +}
1471 +
1472 +static void streaming_topology_v1_emit_inbound_columns(BUFFER *wb) {
1473 + streaming_topology_v1_emit_column(wb, "parent_actor", "actor_ref", "reference", false, NULL);
1474 + streaming_topology_v1_emit_column(wb, "child_actor", "actor_ref", "reference", false, NULL);
1475 + streaming_topology_v1_emit_column(wb, "source_actor", "actor_ref", "reference", true, NULL);
1476 + streaming_topology_v1_emit_column(wb, "received_type", "string", "attribute", true, NULL);
1477 + streaming_topology_v1_emit_column(wb, "ingest_status", "string", "attribute", true, NULL);
1478 + streaming_topology_v1_emit_column(wb, "hops", "int", "metric", false, "max");
1479 + streaming_topology_v1_emit_column(wb, "collected_metrics", "uint", "metric", false, "sum");
1480 + streaming_topology_v1_emit_column(wb, "collected_instances", "uint", "metric", false, "sum");
1481 + streaming_topology_v1_emit_column(wb, "collected_contexts", "uint", "metric", false, "sum");
1482 + streaming_topology_v1_emit_column(wb, "replication_completion", "float", "metric", false, "avg");
1483 + streaming_topology_v1_emit_column(wb, "ingest_age", "duration", "metric", false, "max");
1484 + streaming_topology_v1_emit_column(wb, "ssl", "string", "attribute", true, NULL);
1485 + streaming_topology_v1_emit_column(wb, "alerts_critical", "uint", "metric", false, "sum");
1486 + streaming_topology_v1_emit_column(wb, "alerts_warning", "uint", "metric", false, "sum");
1487 +}
1488 +
1489 +static void streaming_topology_v1_emit_outbound_columns(BUFFER *wb) {
1490 + streaming_topology_v1_emit_column(wb, "sender_actor", "actor_ref", "reference", false, NULL);
1491 + streaming_topology_v1_emit_column(wb, "node_actor", "actor_ref", "reference", false, NULL);
1492 + streaming_topology_v1_emit_column(wb, "destination_actor", "actor_ref", "reference", true, NULL);
1493 + streaming_topology_v1_emit_column(wb, "stream_status", "string", "attribute", true, NULL);
1494 + streaming_topology_v1_emit_column(wb, "stream_age", "duration", "metric", false, "max");
1495 + streaming_topology_v1_emit_column(wb, "hops", "int", "metric", false, "max");
1496 + streaming_topology_v1_emit_column(wb, "ssl", "string", "attribute", true, NULL);
1497 + streaming_topology_v1_emit_column(wb, "compression", "string", "attribute", true, NULL);
1498 + streaming_topology_v1_emit_column(wb, "collected_metrics", "uint", "metric", false, "sum");
1499 + streaming_topology_v1_emit_column(wb, "collected_instances", "uint", "metric", false, "sum");
1500 + streaming_topology_v1_emit_column(wb, "collected_contexts", "uint", "metric", false, "sum");
1501 + streaming_topology_v1_emit_column(wb, "replication_instances", "uint", "metric", false, "sum");
1502 + streaming_topology_v1_emit_column(wb, "replication_completion", "float", "metric", false, "avg");
1503 +}
1504 +
1505 +static void streaming_topology_v1_emit_modal_direct_column(
1506 + BUFFER *wb,
1507 + const char *id,
1508 + const char *label,
1509 + const char *column,
1510 + const char *cell) {
1511 + buffer_json_add_array_item_object(wb);
1512 {
732 - buffer_json_add_array_item_string(wb, "info");
1513 + buffer_json_member_add_string(wb, "id", id);
1514 + buffer_json_member_add_string(wb, "label", label);
1515 + buffer_json_member_add_object(wb, "projection");
1516 + {
1517 + buffer_json_member_add_string(wb, "kind", "direct");
1518 + buffer_json_member_add_string(wb, "column", column);
1519 + }
1520 + buffer_json_object_close(wb);
1521 + buffer_json_member_add_string(wb, "cell", cell);
1522 }
734 - buffer_json_array_close(wb);
735 - buffer_json_member_add_array(wb, "required_params");
736 - buffer_json_array_close(wb);
1523 + buffer_json_object_close(wb);
1524 +}
1525
738 - // --- presentation metadata ---
739 - buffer_json_member_add_object(wb, "presentation");
1526 +static void streaming_topology_v1_emit_modal_actor_ref_column(
1527 + BUFFER *wb,
1528 + const char *id,
1529 + const char *label,
1530 + const char *actor_column) {
1531 + buffer_json_add_array_item_object(wb);
1532 {
741 - buffer_json_member_add_object(wb, "actor_types");
1533 + buffer_json_member_add_string(wb, "id", id);
1534 + buffer_json_member_add_string(wb, "label", label);
1535 + buffer_json_member_add_object(wb, "projection");
1536 + {
1537 + buffer_json_member_add_string(wb, "kind", "actor_ref_label");
1538 + buffer_json_member_add_string(wb, "actor_column", actor_column);
1539 + }
1540 + buffer_json_object_close(wb);
1541 + buffer_json_member_add_string(wb, "cell", "actor_link");
1542 + }
1543 + buffer_json_object_close(wb);
1544 +}
1545 +
1546 +static void streaming_topology_v1_emit_modal_label_lookup_column(
1547 + BUFFER *wb,
1548 + const char *id,
1549 + const char *label,
1550 + const char *actor_column,
1551 + const char *label_key,
1552 + const char *cell) {
1553 + buffer_json_add_array_item_object(wb);
1554 + {
1555 + buffer_json_member_add_string(wb, "id", id);
1556 + buffer_json_member_add_string(wb, "label", label);
1557 + buffer_json_member_add_object(wb, "projection");
1558 + {
1559 + buffer_json_member_add_string(wb, "kind", "label_lookup");
1560 + if(actor_column)
1561 + buffer_json_member_add_string(wb, "actor_column", actor_column);
1562 + buffer_json_member_add_string(wb, "label_key", label_key);
1563 + }
1564 + buffer_json_object_close(wb);
1565 + buffer_json_member_add_string(wb, "cell", cell);
1566 + }
1567 + buffer_json_object_close(wb);
1568 +}
1569 +
1570 +static void streaming_topology_v1_emit_modal_actor_table_source(BUFFER *wb, const char *table) {
1571 + buffer_json_member_add_object(wb, "source");
1572 + {
1573 + buffer_json_member_add_string(wb, "kind", "actor_table");
1574 + buffer_json_member_add_string(wb, "table", table);
1575 + }
1576 + buffer_json_object_close(wb);
1577 +}
1578 +
1579 +static void streaming_topology_v1_emit_modal_actor_owner_filter(BUFFER *wb, const char *actor_column) {
1580 + buffer_json_member_add_object(wb, "owner_filter");
1581 + {
1582 + buffer_json_member_add_string(wb, "mode", "actor_column");
1583 + buffer_json_member_add_string(wb, "actor_column", actor_column);
1584 + }
1585 + buffer_json_object_close(wb);
1586 +}
1587 +
1588 +static void streaming_topology_v1_emit_modal_sort(BUFFER *wb, const char *column, const char *direction) {
1589 + buffer_json_member_add_object(wb, "sort");
1590 + {
1591 + buffer_json_member_add_string(wb, "column", column);
1592 + buffer_json_member_add_string(wb, "direction", direction);
1593 + }
1594 + buffer_json_object_close(wb);
1595 +}
1596 +
1597 +static void streaming_topology_v1_emit_modal_identification_field(BUFFER *wb, const char *key, const char *label) {
1598 + buffer_json_add_array_item_object(wb);
1599 + {
1600 + buffer_json_member_add_string(wb, "key", key);
1601 + buffer_json_member_add_string(wb, "label", label);
1602 + buffer_json_member_add_uint64(wb, "max_values", 1);
1603 + }
1604 + buffer_json_object_close(wb);
1605 +}
1606 +
1607 +static void streaming_topology_v1_emit_host_identification_fields(BUFFER *wb, const char *actor_type) {
1608 + streaming_topology_v1_emit_modal_identification_field(wb, "hostname", "Hostname");
1609 + streaming_topology_v1_emit_modal_identification_field(wb, "type", "Node Type");
1610 + streaming_topology_v1_emit_modal_identification_field(wb, "health_status", "Health");
1611 + streaming_topology_v1_emit_modal_identification_field(wb, "stream_status", "Stream");
1612 + streaming_topology_v1_emit_modal_identification_field(wb, "ingest_status", "Ingest");
1613 + if(actor_type && strcmp(actor_type, "parent") == 0) {
1614 + streaming_topology_v1_emit_modal_identification_field(wb, "retained_node_count", "Retained Nodes");
1615 + streaming_topology_v1_emit_modal_identification_field(wb, "child_count", "Direct Children");
1616 + }
1617 + streaming_topology_v1_emit_modal_identification_field(wb, "os_name", "OS");
1618 + streaming_topology_v1_emit_modal_identification_field(wb, "_os_version", "OS Version");
1619 + streaming_topology_v1_emit_modal_identification_field(wb, "_kernel_version", "Kernel");
1620 + streaming_topology_v1_emit_modal_identification_field(wb, "architecture", "Architecture");
1621 + streaming_topology_v1_emit_modal_identification_field(wb, "_system_cpu_model", "CPU");
1622 + streaming_topology_v1_emit_modal_identification_field(wb, "_system_cores", "Cores");
1623 + streaming_topology_v1_emit_modal_identification_field(wb, "_system_ram_total", "RAM");
1624 + streaming_topology_v1_emit_modal_identification_field(wb, "_virtualization", "Virtualization");
1625 + streaming_topology_v1_emit_modal_identification_field(wb, "_container", "Container");
1626 + streaming_topology_v1_emit_modal_identification_field(wb, "_cloud_provider_type", "Cloud");
1627 + streaming_topology_v1_emit_modal_identification_field(wb, "_cloud_instance_type", "Instance");
1628 + streaming_topology_v1_emit_modal_identification_field(wb, "_cloud_instance_region", "Region");
1629 + streaming_topology_v1_emit_modal_identification_field(wb, "agent_version", "Agent");
1630 +}
1631 +
1632 +static void streaming_topology_v1_emit_vnode_identification_fields(BUFFER *wb) {
1633 + streaming_topology_v1_emit_modal_identification_field(wb, "hostname", "Hostname");
1634 + streaming_topology_v1_emit_modal_identification_field(wb, "type", "Node Type");
1635 + streaming_topology_v1_emit_modal_identification_field(wb, "_vnode_type", "Vnode Type");
1636 + streaming_topology_v1_emit_modal_identification_field(wb, "vendor", "Vendor");
1637 + streaming_topology_v1_emit_modal_identification_field(wb, "model", "Model");
1638 + streaming_topology_v1_emit_modal_identification_field(wb, "address", "Address");
1639 + streaming_topology_v1_emit_modal_identification_field(wb, "location", "Location");
1640 + streaming_topology_v1_emit_modal_identification_field(wb, "sys_object_id", "Sys Object ID");
1641 + streaming_topology_v1_emit_modal_identification_field(wb, "lldp_loc_sys_name", "LLDP Name");
1642 + streaming_topology_v1_emit_modal_identification_field(wb, "health_status", "Health");
1643 + streaming_topology_v1_emit_modal_identification_field(wb, "stream_status", "Stream");
1644 + streaming_topology_v1_emit_modal_identification_field(wb, "ingest_status", "Ingest");
1645 + streaming_topology_v1_emit_modal_identification_field(wb, "agent_version", "Agent");
1646 +}
1647 +
1648 +static void streaming_topology_v1_emit_modal_label_identification(BUFFER *wb, const char *actor_type) {
1649 + buffer_json_member_add_object(wb, "identification");
1650 + {
1651 + buffer_json_member_add_boolean(wb, "enabled", true);
1652 + buffer_json_member_add_array(wb, "fields");
1653 + {
1654 + if(actor_type && strcmp(actor_type, "vnode") == 0)
1655 + streaming_topology_v1_emit_vnode_identification_fields(wb);
1656 + else
1657 + streaming_topology_v1_emit_host_identification_fields(wb, actor_type);
1658 + }
1659 + buffer_json_array_close(wb);
1660 + }
1661 + buffer_json_object_close(wb);
1662 +}
1663 +
1664 +static void streaming_topology_v1_emit_actor_modal(BUFFER *wb, const char *actor_type) {
1665 + buffer_json_member_add_object(wb, "modal");
1666 + {
1667 + buffer_json_member_add_boolean(wb, "enabled", true);
1668 + buffer_json_member_add_object(wb, "labels");
1669 + {
1670 + buffer_json_member_add_boolean(wb, "enabled", true);
1671 + buffer_json_member_add_string(wb, "table", "actor_labels");
1672 + streaming_topology_v1_emit_modal_label_identification(wb, actor_type);
1673 + }
1674 + buffer_json_object_close(wb);
1675 + buffer_json_member_add_object(wb, "mini_topology");
1676 + {
1677 + buffer_json_member_add_boolean(wb, "enabled", true);
1678 + buffer_json_member_add_uint64(wb, "depth", 1);
1679 + }
1680 + buffer_json_object_close(wb);
1681 + buffer_json_member_add_array(wb, "sections");
1682 {
743 - buffer_json_member_add_object(wb, "parent");
1683 + buffer_json_add_array_item_object(wb);
1684 {
745 - buffer_json_member_add_string(wb, "label", "Netdata Parent");
746 - buffer_json_member_add_string(wb, "color_slot", "primary");
747 - buffer_json_member_add_double(wb, "opacity", 1.0);
748 - buffer_json_member_add_boolean(wb, "border", true);
749 - buffer_json_member_add_string(wb, "role", "actor");
750 - buffer_json_member_add_boolean(wb, "size_by_links", true);
751 - buffer_json_member_add_boolean(wb, "show_port_bullets", true);
752 - streaming_topology_parent_presentation(wb);
1685 + buffer_json_member_add_string(wb, "id", "stream_path");
1686 + buffer_json_member_add_string(wb, "label", "Stream path");
1687 + buffer_json_member_add_uint64(wb, "order", 1);
1688 + streaming_topology_v1_emit_modal_actor_table_source(wb, "stream_path");
1689 + streaming_topology_v1_emit_modal_actor_owner_filter(wb, "actor");
1690 + buffer_json_member_add_array(wb, "columns");
1691 + {
1692 + streaming_topology_v1_emit_modal_direct_column(wb, "path_index", "Hop", "path_index", "number");
1693 + streaming_topology_v1_emit_modal_actor_ref_column(wb, "path_actor", "Node", "path_actor");
1694 + streaming_topology_v1_emit_modal_direct_column(wb, "hostname", "Hostname", "hostname", "text");
1695 + streaming_topology_v1_emit_modal_direct_column(wb, "hops", "Hops", "hops", "number");
1696 + streaming_topology_v1_emit_modal_direct_column(wb, "since", "Since", "since", "timestamp");
1697 + streaming_topology_v1_emit_modal_direct_column(wb, "first_time", "First seen", "first_time", "timestamp");
1698 + }
1699 + buffer_json_array_close(wb);
1700 + streaming_topology_v1_emit_modal_sort(wb, "path_index", "asc");
1701 + buffer_json_member_add_string(wb, "empty_label", "No stream path rows");
1702 }
1703 buffer_json_object_close(wb);
1704
756 - buffer_json_member_add_object(wb, "child");
1705 + buffer_json_add_array_item_object(wb);
1706 {
758 - buffer_json_member_add_string(wb, "label", "Netdata Child");
759 - buffer_json_member_add_string(wb, "color_slot", "primary");
760 - buffer_json_member_add_double(wb, "opacity", 1.0);
761 - buffer_json_member_add_boolean(wb, "border", false);
762 - buffer_json_member_add_string(wb, "role", "actor");
763 - buffer_json_member_add_boolean(wb, "size_by_links", false);
764 - buffer_json_member_add_boolean(wb, "show_port_bullets", false);
765 - streaming_topology_child_presentation(wb);
1707 + buffer_json_member_add_string(wb, "id", "retained_nodes");
1708 + buffer_json_member_add_string(wb, "label", "Retained nodes");
1709 + buffer_json_member_add_uint64(wb, "order", 2);
1710 + streaming_topology_v1_emit_modal_actor_table_source(wb, "retention");
1711 + streaming_topology_v1_emit_modal_actor_owner_filter(wb, "observer_actor");
1712 + buffer_json_member_add_array(wb, "columns");
1713 + {
1714 + streaming_topology_v1_emit_modal_actor_ref_column(wb, "actor", "Node", "actor");
1715 + streaming_topology_v1_emit_modal_label_lookup_column(wb, "actor_type", "Node type", "actor", "type", "badge");
1716 + streaming_topology_v1_emit_modal_direct_column(wb, "db_status", "Status", "db_status", "badge");
1717 + streaming_topology_v1_emit_modal_direct_column(wb, "db_from", "From", "db_from", "timestamp");
1718 + streaming_topology_v1_emit_modal_direct_column(wb, "db_to", "To", "db_to", "timestamp");
1719 + streaming_topology_v1_emit_modal_direct_column(wb, "db_duration", "Duration", "db_duration", "duration");
1720 + streaming_topology_v1_emit_modal_direct_column(wb, "db_metrics", "Metrics", "db_metrics", "number");
1721 + streaming_topology_v1_emit_modal_direct_column(wb, "db_instances", "Instances", "db_instances", "number");
1722 + streaming_topology_v1_emit_modal_direct_column(wb, "db_contexts", "Contexts", "db_contexts", "number");
1723 + }
1724 + buffer_json_array_close(wb);
1725 + streaming_topology_v1_emit_modal_sort(wb, "db_duration", "desc");
1726 + buffer_json_member_add_string(wb, "empty_label", "No retained nodes");
1727 }
1728 buffer_json_object_close(wb);
1729
769 - buffer_json_member_add_object(wb, "vnode");
1730 + buffer_json_add_array_item_object(wb);
1731 {
771 - buffer_json_member_add_string(wb, "label", "Virtual Node");
772 - buffer_json_member_add_string(wb, "color_slot", "warning");
773 - buffer_json_member_add_double(wb, "opacity", 1.0);
774 - buffer_json_member_add_boolean(wb, "border", false);
775 - buffer_json_member_add_string(wb, "role", "actor");
776 - buffer_json_member_add_boolean(wb, "size_by_links", false);
777 - buffer_json_member_add_boolean(wb, "show_port_bullets", false);
778 - streaming_topology_vnode_presentation(wb);
1732 + buffer_json_member_add_string(wb, "id", "inbound");
1733 + buffer_json_member_add_string(wb, "label", "Received nodes");
1734 + buffer_json_member_add_uint64(wb, "order", 3);
1735 + streaming_topology_v1_emit_modal_actor_table_source(wb, "inbound");
1736 + streaming_topology_v1_emit_modal_actor_owner_filter(wb, "parent_actor");
1737 + buffer_json_member_add_array(wb, "columns");
1738 + {
1739 + streaming_topology_v1_emit_modal_actor_ref_column(wb, "child", "Node", "child_actor");
1740 + streaming_topology_v1_emit_modal_actor_ref_column(wb, "source", "Received from", "source_actor");
1741 + streaming_topology_v1_emit_modal_label_lookup_column(wb, "child_type", "Node type", "child_actor", "type", "badge");
1742 + streaming_topology_v1_emit_modal_direct_column(wb, "received_type", "Received as", "received_type", "badge");
1743 + streaming_topology_v1_emit_modal_direct_column(wb, "ingest_status", "Ingest", "ingest_status", "badge");
1744 + streaming_topology_v1_emit_modal_direct_column(wb, "hops", "Hops", "hops", "number");
1745 + streaming_topology_v1_emit_modal_direct_column(wb, "collected_metrics", "Metrics", "collected_metrics", "number");
1746 + streaming_topology_v1_emit_modal_direct_column(wb, "collected_instances", "Instances", "collected_instances", "number");
1747 + streaming_topology_v1_emit_modal_direct_column(wb, "collected_contexts", "Contexts", "collected_contexts", "number");
1748 + streaming_topology_v1_emit_modal_direct_column(wb, "replication_completion", "Replication %", "replication_completion", "number");
1749 + streaming_topology_v1_emit_modal_direct_column(wb, "ingest_age", "Age", "ingest_age", "duration");
1750 + streaming_topology_v1_emit_modal_direct_column(wb, "ssl", "TLS", "ssl", "badge");
1751 + streaming_topology_v1_emit_modal_direct_column(wb, "alerts_critical", "Critical", "alerts_critical", "number");
1752 + streaming_topology_v1_emit_modal_direct_column(wb, "alerts_warning", "Warning", "alerts_warning", "number");
1753 + }
1754 + buffer_json_array_close(wb);
1755 + buffer_json_member_add_string(wb, "empty_label", "No received nodes");
1756 }
1757 buffer_json_object_close(wb);
1758
782 - buffer_json_member_add_object(wb, "stale");
1759 + buffer_json_add_array_item_object(wb);
1760 {
784 - buffer_json_member_add_string(wb, "label", "Stale Node");
785 - buffer_json_member_add_string(wb, "color_slot", "dim");
786 - buffer_json_member_add_double(wb, "opacity", 0.5);
787 - buffer_json_member_add_boolean(wb, "border", false);
788 - buffer_json_member_add_string(wb, "role", "actor");
789 - buffer_json_member_add_boolean(wb, "size_by_links", false);
790 - buffer_json_member_add_boolean(wb, "show_port_bullets", false);
791 - streaming_topology_stale_presentation(wb);
1761 + buffer_json_member_add_string(wb, "id", "outbound");
1762 + buffer_json_member_add_string(wb, "label", "Outbound streams");
1763 + buffer_json_member_add_uint64(wb, "order", 4);
1764 + streaming_topology_v1_emit_modal_actor_table_source(wb, "outbound");
1765 + streaming_topology_v1_emit_modal_actor_owner_filter(wb, "sender_actor");
1766 + buffer_json_member_add_array(wb, "columns");
1767 + {
1768 + streaming_topology_v1_emit_modal_actor_ref_column(wb, "node", "Node", "node_actor");
1769 + streaming_topology_v1_emit_modal_label_lookup_column(wb, "node_type", "Node type", "node_actor", "type", "badge");
1770 + streaming_topology_v1_emit_modal_actor_ref_column(wb, "destination", "Destination", "destination_actor");
1771 + streaming_topology_v1_emit_modal_direct_column(wb, "stream_status", "Status", "stream_status", "badge");
1772 + streaming_topology_v1_emit_modal_direct_column(wb, "stream_age", "Age", "stream_age", "duration");
1773 + streaming_topology_v1_emit_modal_direct_column(wb, "hops", "Hops", "hops", "number");
1774 + streaming_topology_v1_emit_modal_direct_column(wb, "collected_metrics", "Metrics", "collected_metrics", "number");
1775 + streaming_topology_v1_emit_modal_direct_column(wb, "collected_instances", "Instances", "collected_instances", "number");
1776 + streaming_topology_v1_emit_modal_direct_column(wb, "collected_contexts", "Contexts", "collected_contexts", "number");
1777 + streaming_topology_v1_emit_modal_direct_column(wb, "replication_completion", "Replication %", "replication_completion", "number");
1778 + streaming_topology_v1_emit_modal_direct_column(wb, "ssl", "TLS", "ssl", "badge");
1779 + streaming_topology_v1_emit_modal_direct_column(wb, "compression", "Compression", "compression", "badge");
1780 + }
1781 + buffer_json_array_close(wb);
1782 + buffer_json_member_add_string(wb, "empty_label", "No outbound streams");
1783 }
1784 buffer_json_object_close(wb);
1785 }
795 - buffer_json_object_close(wb); // actor_types
1786 + buffer_json_array_close(wb);
1787 + }
1788 + buffer_json_object_close(wb);
1789 +}
1790 +
1791 +static void streaming_topology_v1_emit_actor_type(
1792 + BUFFER *wb,
1793 + const char *id,
1794 + const char *label,
1795 + const char *color_slot,
1796 + const char *icon,
1797 + bool border,
1798 + const char *size_mode,
1799 + const char *size_metric_column,
1800 + const char *size_scale,
1801 + const char *layout_repulsion,
1802 + bool show_port_bullets,
1803 + const char *port_actor_column) {
1804 + buffer_json_member_add_object(wb, id);
1805 + {
1806 + buffer_json_member_add_string(wb, "layer", "streaming");
1807 + buffer_json_member_add_array(wb, "identity");
1808 + buffer_json_add_array_item_string(wb, "id");
1809 + buffer_json_array_close(wb);
1810 + buffer_json_member_add_array(wb, "merge_identity");
1811 + buffer_json_add_array_item_string(wb, "machine_guid");
1812 + buffer_json_add_array_item_string(wb, "node_id");
1813 + buffer_json_array_close(wb);
1814 + buffer_json_member_add_array(wb, "aggregation_scopes");
1815 + buffer_json_add_array_item_string(wb, "node");
1816 + buffer_json_array_close(wb);
1817 + buffer_json_member_add_object(wb, "search");
1818 + {
1819 + if(strcmp(id, "stale") == 0)
1820 + buffer_json_member_add_boolean(wb, "enabled", true);
1821 + buffer_json_member_add_array(wb, "columns");
1822 + buffer_json_add_array_item_string(wb, "display_name");
1823 + buffer_json_add_array_item_string(wb, "hostname");
1824 + buffer_json_add_array_item_string(wb, "machine_guid");
1825 + buffer_json_add_array_item_string(wb, "node_id");
1826 + buffer_json_add_array_item_string(wb, "agent_version");
1827 + buffer_json_array_close(wb);
1828 + }
1829 + buffer_json_object_close(wb);
1830 + buffer_json_member_add_object(wb, "presentation");
1831 + {
1832 + buffer_json_member_add_string(wb, "label", label);
1833 + buffer_json_member_add_string(wb, "role", "actor");
1834 + buffer_json_member_add_string(wb, "icon", icon);
1835 + buffer_json_member_add_string(wb, "color_slot", color_slot);
1836 + buffer_json_member_add_string(wb, "opacity", strcmp(id, "stale") == 0 ? "faded" : "normal");
1837 + buffer_json_member_add_object(wb, "border");
1838 + {
1839 + buffer_json_member_add_boolean(wb, "enabled", border);
1840 + }
1841 + buffer_json_object_close(wb);
1842 + buffer_json_member_add_object(wb, "size");
1843 + {
1844 + buffer_json_member_add_string(wb, "mode", size_mode ? size_mode : "fixed");
1845 + if(size_metric_column)
1846 + buffer_json_member_add_string(wb, "metric_column", size_metric_column);
1847 + if(size_scale)
1848 + buffer_json_member_add_string(wb, "scale", size_scale);
1849 + }
1850 + buffer_json_object_close(wb);
1851 + if(layout_repulsion) {
1852 + buffer_json_member_add_object(wb, "layout");
1853 + {
1854 + buffer_json_member_add_string(wb, "repulsion", layout_repulsion);
1855 + }
1856 + buffer_json_object_close(wb);
1857 + }
1858 + buffer_json_member_add_object(wb, "label_policy");
1859 + {
1860 + buffer_json_member_add_array(wb, "columns");
1861 + buffer_json_add_array_item_string(wb, "display_name");
1862 + buffer_json_add_array_item_string(wb, "hostname");
1863 + buffer_json_array_close(wb);
1864 + buffer_json_member_add_string(wb, "fallback", "type_label");
1865 + buffer_json_member_add_uint64(wb, "max_length", 80);
1866 + buffer_json_member_add_string(wb, "array", "reject");
1867 + }
1868 + buffer_json_object_close(wb);
1869 + buffer_json_member_add_object(wb, "ports");
1870 + {
1871 + buffer_json_member_add_boolean(wb, "show_bullets", show_port_bullets);
1872 + if(show_port_bullets) {
1873 + buffer_json_member_add_array(wb, "sources");
1874 + {
1875 + buffer_json_add_array_item_object(wb);
1876 + buffer_json_member_add_string(wb, "source", "links");
1877 + buffer_json_member_add_string(
1878 + wb, "actor_column", port_actor_column ? port_actor_column : "src_actor");
1879 + buffer_json_member_add_string(wb, "name_column", "port_name");
1880 + buffer_json_member_add_string(wb, "type_column", "type");
1881 + buffer_json_member_add_string(wb, "default_type", "streaming");
1882 + buffer_json_object_close(wb);
1883 + }
1884 + buffer_json_array_close(wb);
1885 + }
1886 + }
1887 + buffer_json_object_close(wb);
1888 + streaming_topology_v1_emit_actor_modal(wb, id);
1889 + }
1890 + buffer_json_object_close(wb);
1891 + }
1892 + buffer_json_object_close(wb);
1893 +}
1894 +
1895 +static void streaming_topology_v1_emit_link_type(
1896 + BUFFER *wb,
1897 + const char *id,
1898 + const char *direction_role,
1899 + const char *semantic_role,
1900 + const char *evidence_type,
1901 + const char *label,
1902 + const char *color_slot,
1903 + const char *line_style,
1904 + const char *width,
1905 + const char *opacity) {
1906 + buffer_json_member_add_object(wb, id);
1907 + {
1908 + buffer_json_member_add_string(wb, "orientation", "directed");
1909 + buffer_json_member_add_string(wb, "direction_role", direction_role);
1910 + if(semantic_role)
1911 + buffer_json_member_add_string(wb, "semantic_role", semantic_role);
1912 + buffer_json_member_add_object(wb, "aggregation");
1913 + {
1914 + buffer_json_member_add_string(wb, "direction", "preserve");
1915 + buffer_json_member_add_string(wb, "evidence", "append");
1916 + buffer_json_member_add_object(wb, "metrics");
1917 + {
1918 + buffer_json_member_add_string(wb, "hops", "max");
1919 + buffer_json_member_add_string(wb, "evidence_count", "sum");
1920 + buffer_json_member_add_string(wb, "connections", "sum");
1921 + buffer_json_member_add_string(wb, "replication_instances", "sum");
1922 + buffer_json_member_add_string(wb, "replication_completion", "avg");
1923 + buffer_json_member_add_string(wb, "collected_metrics", "sum");
1924 + buffer_json_member_add_string(wb, "collected_instances", "sum");
1925 + buffer_json_member_add_string(wb, "collected_contexts", "sum");
1926 + }
1927 + buffer_json_object_close(wb);
1928 + }
1929 + buffer_json_object_close(wb);
1930 + buffer_json_member_add_array(wb, "evidence_types");
1931 + buffer_json_add_array_item_string(wb, evidence_type);
1932 + buffer_json_array_close(wb);
1933 + buffer_json_member_add_object(wb, "presentation");
1934 + {
1935 + buffer_json_member_add_string(wb, "label", label);
1936 + buffer_json_member_add_string(wb, "color_slot", color_slot);
1937 + buffer_json_member_add_string(wb, "line_style", line_style);
1938 + buffer_json_member_add_string(wb, "width", width);
1939 + buffer_json_member_add_string(wb, "opacity", opacity);
1940 + buffer_json_member_add_string(wb, "curve", "auto");
1941 + buffer_json_member_add_string(wb, "arrow", "forward");
1942 + }
1943 + buffer_json_object_close(wb);
1944 + }
1945 + buffer_json_object_close(wb);
1946 +}
1947 +
1948 +static void streaming_topology_v1_emit_evidence_type(BUFFER *wb, const char *id, const char *link_type) {
1949 + buffer_json_member_add_object(wb, id);
1950 + {
1951 + buffer_json_member_add_string(wb, "link_type", link_type);
1952 + buffer_json_member_add_string(wb, "role", "relationship_evidence");
1953 + buffer_json_member_add_array(wb, "columns");
1954 + streaming_topology_v1_emit_evidence_columns(wb);
1955 + buffer_json_array_close(wb);
1956 + buffer_json_member_add_array(wb, "match_columns");
1957 + buffer_json_add_array_item_string(wb, "src_actor");
1958 + buffer_json_add_array_item_string(wb, "dst_actor");
1959 + buffer_json_add_array_item_string(wb, "type");
1960 + buffer_json_array_close(wb);
1961 + }
1962 + buffer_json_object_close(wb);
1963 +}
1964 +
1965 +static void streaming_topology_v1_emit_table_type(
1966 + BUFFER *wb,
1967 + const char *id,
1968 + const char *role,
1969 + const char *owner,
1970 + const char *aggregation,
1971 + void (*emit_columns)(BUFFER *)) {
1972 + buffer_json_member_add_object(wb, id);
1973 + {
1974 + buffer_json_member_add_string(wb, "role", role);
1975 + buffer_json_member_add_string(wb, "owner", owner);
1976 + buffer_json_member_add_string(wb, "aggregation", aggregation);
1977 + buffer_json_member_add_array(wb, "columns");
1978 + emit_columns(wb);
1979 + buffer_json_array_close(wb);
1980 + }
1981 + buffer_json_object_close(wb);
1982 +}
1983 +
1984 +static void streaming_topology_v1_emit_type_registry(BUFFER *wb) {
1985 + buffer_json_member_add_object(wb, "types");
1986 + {
1987 + buffer_json_member_add_object(wb, "actor_types");
1988 + {
1989 + streaming_topology_v1_emit_actor_type(
1990 + wb, "parent", "Netdata Parent", "primary", "parent", true,
1991 + "metric", "retained_node_count", "emphasized", "stronger", true, "dst_actor");
1992 + streaming_topology_v1_emit_actor_type(
1993 + wb, "child", "Netdata Child", "primary", "netdata-agent", false,
1994 + "fixed", NULL, "normal", "normal", false, NULL);
1995 + streaming_topology_v1_emit_actor_type(
1996 + wb, "vnode", "Virtual Node", "warning", "netdata-agent", false,
1997 + "fixed", NULL, "normal", "normal", false, NULL);
1998 + streaming_topology_v1_emit_actor_type(
1999 + wb, "stale", "Stale Node", "dim", "netdata-agent", false,
2000 + "fixed", NULL, "compact", "weaker", false, NULL);
2001 + }
2002 + buffer_json_object_close(wb);
2003
2004 buffer_json_member_add_object(wb, "link_types");
2005 + {
2006 + streaming_topology_v1_emit_link_type(
2007 + wb, "streaming", "dependency", "traffic", "streaming_link",
2008 + "Streaming", "primary", "solid", "thick", "normal");
2009 + streaming_topology_v1_emit_link_type(
2010 + wb, "virtual", "dependency", "ownership", "virtual_link",
2011 + "Virtual origin", "warning", "dashed", "thin", "muted");
2012 + streaming_topology_v1_emit_link_type(
2013 + wb, "stale", "observation", "normal", "stale_link",
2014 + "Stale data", "dim", "dashed", "thin", "faded");
2015 + }
2016 + buffer_json_object_close(wb);
2017 +
2018 + buffer_json_member_add_object(wb, "port_types");
2019 {
2020 buffer_json_member_add_object(wb, "streaming");
2021 {
801 - buffer_json_member_add_string(wb, "label", "Streaming");
802 - buffer_json_member_add_string(wb, "color_slot", "primary");
803 - buffer_json_member_add_double(wb, "width", 2);
804 - buffer_json_member_add_boolean(wb, "dash", false);
805 - buffer_json_member_add_double(wb, "opacity", 1.0);
2022 + buffer_json_member_add_object(wb, "presentation");
2023 + {
2024 + buffer_json_member_add_string(wb, "label", "Streaming child");
2025 + buffer_json_member_add_string(wb, "color_slot", "primary");
2026 + buffer_json_member_add_string(wb, "opacity", "normal");
2027 + }
2028 + buffer_json_object_close(wb);
2029 }
2030 buffer_json_object_close(wb);
2031
2032 buffer_json_member_add_object(wb, "virtual");
2033 {
811 - buffer_json_member_add_string(wb, "label", "Virtual origin");
812 - buffer_json_member_add_string(wb, "color_slot", "warning");
813 - buffer_json_member_add_double(wb, "width", 1);
814 - buffer_json_member_add_boolean(wb, "dash", true);
815 - buffer_json_member_add_double(wb, "opacity", 0.7);
2034 + buffer_json_member_add_object(wb, "presentation");
2035 + {
2036 + buffer_json_member_add_string(wb, "label", "Virtual node");
2037 + buffer_json_member_add_string(wb, "color_slot", "warning");
2038 + buffer_json_member_add_string(wb, "opacity", "normal");
2039 + }
2040 + buffer_json_object_close(wb);
2041 }
2042 buffer_json_object_close(wb);
2043
2044 buffer_json_member_add_object(wb, "stale");
2045 {
821 - buffer_json_member_add_string(wb, "label", "Stale data");
822 - buffer_json_member_add_string(wb, "color_slot", "dim");
823 - buffer_json_member_add_double(wb, "width", 1);
824 - buffer_json_member_add_boolean(wb, "dash", true);
825 - buffer_json_member_add_double(wb, "opacity", 0.4);
2046 + buffer_json_member_add_object(wb, "presentation");
2047 + {
2048 + buffer_json_member_add_string(wb, "label", "Stale node");
2049 + buffer_json_member_add_string(wb, "color_slot", "dim");
2050 + buffer_json_member_add_string(wb, "opacity", "faded");
2051 + }
2052 + buffer_json_object_close(wb);
2053 }
2054 buffer_json_object_close(wb);
2055 }
829 - buffer_json_object_close(wb); // link_types
2056 + buffer_json_object_close(wb);
2057 +
2058 + buffer_json_member_add_object(wb, "evidence_types");
2059 + {
2060 + streaming_topology_v1_emit_evidence_type(wb, "streaming_link", "streaming");
2061 + streaming_topology_v1_emit_evidence_type(wb, "virtual_link", "virtual");
2062 + streaming_topology_v1_emit_evidence_type(wb, "stale_link", "stale");
2063 + }
2064 + buffer_json_object_close(wb);
2065
831 - buffer_json_member_add_array(wb, "port_fields");
2066 + buffer_json_member_add_object(wb, "table_types");
2067 {
833 - buffer_json_add_array_item_object(wb);
834 - buffer_json_member_add_string(wb, "key", "type");
835 - buffer_json_member_add_string(wb, "label", "Type");
836 - buffer_json_object_close(wb);
2068 + streaming_topology_v1_emit_table_type(wb, "actor_labels", "actor_inventory", "actor", "set",
2069 + streaming_topology_v1_emit_actor_label_columns);
2070 + streaming_topology_v1_emit_table_type(wb, "stream_path", "actor_detail", "actor", "append",
2071 + streaming_topology_v1_emit_stream_path_columns);
2072 + streaming_topology_v1_emit_table_type(wb, "retention", "actor_detail", "actor", "append",
2073 + streaming_topology_v1_emit_retention_columns);
2074 + streaming_topology_v1_emit_table_type(wb, "inbound", "relationship_summary", "actor", "append",
2075 + streaming_topology_v1_emit_inbound_columns);
2076 + streaming_topology_v1_emit_table_type(wb, "outbound", "relationship_summary", "actor", "append",
2077 + streaming_topology_v1_emit_outbound_columns);
2078 }
838 - buffer_json_array_close(wb); // port_fields
2079 + buffer_json_object_close(wb);
2080
840 - buffer_json_member_add_object(wb, "port_types");
2081 + buffer_json_member_add_object(wb, "aggregation_scopes");
2082 {
842 - buffer_json_member_add_object(wb, "streaming");
2083 + buffer_json_member_add_object(wb, "node");
2084 {
844 - buffer_json_member_add_string(wb, "label", "Streaming child");
845 - buffer_json_member_add_string(wb, "color_slot", "primary");
846 - buffer_json_member_add_double(wb, "opacity", 1.0);
847 - }
848 - buffer_json_object_close(wb);
849 -
850 - buffer_json_member_add_object(wb, "virtual");
851 - {
852 - buffer_json_member_add_string(wb, "label", "Virtual node");
853 - buffer_json_member_add_string(wb, "color_slot", "warning");
854 - buffer_json_member_add_double(wb, "opacity", 1.0);
2085 + buffer_json_member_add_array(wb, "columns");
2086 + buffer_json_add_array_item_string(wb, "machine_guid");
2087 + buffer_json_add_array_item_string(wb, "node_id");
2088 + buffer_json_array_close(wb);
2089 + buffer_json_member_add_string(wb, "evidence_policy", "preserve");
2090 }
2091 buffer_json_object_close(wb);
2092 + }
2093 + buffer_json_object_close(wb);
2094 + }
2095 + buffer_json_object_close(wb);
2096 +}
2097
858 - buffer_json_member_add_object(wb, "stale");
2098 +static void streaming_topology_v1_emit_presentation(BUFFER *wb) {
2099 + buffer_json_member_add_object(wb, "presentation");
2100 + {
2101 + buffer_json_member_add_string(wb, "profile_version", "streaming.v1");
2102 + buffer_json_member_add_object(wb, "selection");
2103 + {
2104 + buffer_json_member_add_object(wb, "actor_click");
2105 {
860 - buffer_json_member_add_string(wb, "label", "Stale node");
861 - buffer_json_member_add_string(wb, "color_slot", "dim");
862 - buffer_json_member_add_double(wb, "opacity", 0.5);
2106 + buffer_json_member_add_string(wb, "mode", "highlight_path");
2107 + buffer_json_member_add_string(wb, "path_table", "stream_path");
2108 + buffer_json_member_add_string(wb, "path_owner_column", "actor");
2109 + buffer_json_member_add_string(wb, "path_actor_column", "path_actor");
2110 + buffer_json_member_add_string(wb, "path_order_column", "path_index");
2111 }
2112 buffer_json_object_close(wb);
2113 }
866 - buffer_json_object_close(wb); // port_types
2114 + buffer_json_object_close(wb);
2115
2116 buffer_json_member_add_object(wb, "legend");
2117 {
@@ -889,7 +2137,7 @@ int function_streaming_topology(BUFFER *wb, const char *function, BUFFER *payloa
2137 buffer_json_member_add_string(wb, "label", "Stale Node");
2138 buffer_json_object_close(wb);
2139 }
892 - buffer_json_array_close(wb); // actors
2140 + buffer_json_array_close(wb);
2141
2142 buffer_json_member_add_array(wb, "links");
2143 {
@@ -908,7 +2156,7 @@ int function_streaming_topology(BUFFER *wb, const char *function, BUFFER *payloa
2156 buffer_json_member_add_string(wb, "label", "Stale data");
2157 buffer_json_object_close(wb);
2158 }
911 - buffer_json_array_close(wb); // links
2159 + buffer_json_array_close(wb);
2160
2161 buffer_json_member_add_array(wb, "ports");
2162 {
@@ -927,884 +2175,855 @@ int function_streaming_topology(BUFFER *wb, const char *function, BUFFER *payloa
2175 buffer_json_member_add_string(wb, "label", "Stale node");
2176 buffer_json_object_close(wb);
2177 }
930 - buffer_json_array_close(wb); // ports
2178 + buffer_json_array_close(wb);
2179 }
932 - buffer_json_object_close(wb); // legend
2180 + buffer_json_object_close(wb);
2181
934 - buffer_json_member_add_string(wb, "actor_click_behavior", "highlight_path");
2182 + buffer_json_member_add_array(wb, "port_fields");
2183 + {
2184 + buffer_json_add_array_item_object(wb);
2185 + buffer_json_member_add_string(wb, "key", "type");
2186 + buffer_json_member_add_string(wb, "label", "Type");
2187 + buffer_json_object_close(wb);
2188 + }
2189 + buffer_json_array_close(wb);
2190 }
936 - buffer_json_object_close(wb); // presentation
2191 + buffer_json_object_close(wb);
2192 +}
2193
938 - if(!info_only) {
939 - // --- Phase 1: build parent_child_count dictionary from streaming_paths ---
940 - // A node is a parent if any other node's streaming_path contains it at position > 0
941 - DICTIONARY *parent_child_count = dictionary_create_advanced(
942 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
943 - NULL, sizeof(uint32_t));
944 - DICTIONARY *parent_descendants = dictionary_create_advanced(
945 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
946 - NULL, sizeof(struct streaming_topology_descendant_list));
947 - // local_actor_ids: every actor backed by a local RRDHOST.
948 - // emitted_actors / emitted_links: dedup sets for actors/links actually
949 - // written to the response.
950 - // The dictionary stores a 1-byte sentinel so dictionary_get() returns
951 - // a non-NULL value for present keys. With value_len=0 the stored value
952 - // is NULL and dictionary_get() can't distinguish present vs absent.
953 - DICTIONARY *local_actor_ids = dictionary_create_advanced(
954 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
955 - NULL, sizeof(uint8_t));
956 - DICTIONARY *emitted_actors = dictionary_create_advanced(
957 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
958 - NULL, sizeof(uint8_t));
959 - DICTIONARY *emitted_links = dictionary_create_advanced(
960 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
961 - NULL, sizeof(uint8_t));
962 -
963 - if(!parent_child_count || !parent_descendants || !local_actor_ids || !emitted_actors || !emitted_links) {
964 - if(emitted_links)
965 - dictionary_destroy(emitted_links);
966 - if(emitted_actors)
967 - dictionary_destroy(emitted_actors);
968 - if(local_actor_ids)
969 - dictionary_destroy(local_actor_ids);
970 - if(parent_descendants)
971 - dictionary_destroy(parent_descendants);
972 - if(parent_child_count)
973 - dictionary_destroy(parent_child_count);
974 -
975 - return streaming_topology_return_error(wb, function_copy,
976 - HTTP_RESP_INTERNAL_SERVER_ERROR,
977 - "failed to allocate streaming topology dictionaries");
978 - }
979 - else {
2194 +static void streaming_topology_v1_emit_actor_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2195 + buffer_json_member_add_object(wb, "actors");
2196 + {
2197 + buffer_json_member_add_uint64(wb, "rows", payload->actors_used);
2198 + buffer_json_member_add_array(wb, "columns");
2199 + streaming_topology_v1_emit_actor_columns(wb);
2200 + buffer_json_array_close(wb);
2201
981 - {
982 - RRDHOST *host;
983 - dfe_start_read(rrdhost_root_index, host) {
984 - char host_actor_id[256];
985 - streaming_topology_actor_id_for_host(host, host_actor_id, sizeof(host_actor_id));
986 - uint8_t one = 1;
987 - dictionary_set(local_actor_ids, host_actor_id, &one, sizeof(one));
988 - }
989 - dfe_done(host);
990 - }
2202 + buffer_json_member_add_array(wb, "values");
2203 +#define STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(member) do { \
2204 + streaming_topology_v1_emit_values_start(wb); \
2205 + for(size_t i = 0; i < payload->actors_used; i++) \
2206 + buffer_json_add_array_item_string(wb, payload->actors[i].member[0] ? payload->actors[i].member : NULL); \
2207 + streaming_topology_v1_emit_values_end(wb); \
2208 + } while(0)
2209 +
2210 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(actor_id);
2211 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(type);
2212 + streaming_topology_v1_emit_values_start(wb);
2213 + for(size_t i = 0; i < payload->actors_used; i++)
2214 + buffer_json_add_array_item_string(wb, "streaming");
2215 + streaming_topology_v1_emit_values_end(wb);
2216 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(machine_guid);
2217 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(node_id);
2218 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(hostname);
2219 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(display_name);
2220 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(severity);
2221 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(ephemerality);
2222 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(ingest_status);
2223 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(stream_status);
2224 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(ml_status);
2225 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(agent_name);
2226 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(agent_version);
2227 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(health_status);
2228 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(os_name);
2229 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(architecture);
2230 + STREAMING_TOPOLOGY_ACTOR_STRING_VALUES(cpu_count);
2231 +
2232 +#undef STREAMING_TOPOLOGY_ACTOR_STRING_VALUES
2233 +#define STREAMING_TOPOLOGY_ACTOR_UINT_VALUES(member) do { \
2234 + streaming_topology_v1_emit_values_start(wb); \
2235 + for(size_t i = 0; i < payload->actors_used; i++) \
2236 + buffer_json_add_array_item_uint64(wb, payload->actors[i].member); \
2237 + streaming_topology_v1_emit_values_end(wb); \
2238 + } while(0)
2239 +
2240 + STREAMING_TOPOLOGY_ACTOR_UINT_VALUES(child_count);
2241 + STREAMING_TOPOLOGY_ACTOR_UINT_VALUES(retained_node_count);
2242 + STREAMING_TOPOLOGY_ACTOR_UINT_VALUES(health_critical);
2243 + STREAMING_TOPOLOGY_ACTOR_UINT_VALUES(health_warning);
2244 + STREAMING_TOPOLOGY_ACTOR_UINT_VALUES(health_clear);
2245 +#undef STREAMING_TOPOLOGY_ACTOR_UINT_VALUES
2246
992 - {
993 - RRDHOST *host;
994 - dfe_start_read(rrdhost_root_index, host) {
995 - // get all path entries at position > 0 (parents in the chain)
996 - ND_UUID path_ids[128];
997 - uint16_t n = streaming_topology_get_path_ids(host, 1, path_ids, 128);
998 - for(uint16_t i = 0; i < n; i++) {
999 - char guid[UUID_STR_LEN];
1000 - if(!streaming_topology_uuid_guid(path_ids[i], guid, sizeof(guid)))
1001 - continue;
1002 -
1003 - uint32_t *count = dictionary_get(parent_child_count, guid);
1004 - if(count)
1005 - (*count)++;
1006 - else {
1007 - uint32_t one = 1;
1008 - dictionary_set(parent_child_count, guid, &one, sizeof(one));
1009 - }
1010 - }
2247 + buffer_json_array_close(wb);
2248 + }
2249 + buffer_json_object_close(wb);
2250 +}
2251
1012 - ND_UUID full_path_ids[128];
1013 - uint16_t full_path_n = streaming_topology_get_path_ids(host, 0, full_path_ids, 128);
1014 - ND_UUID empty_uuid = {};
1015 -
1016 - if(rrdhost_is_virtual(host))
1017 - continue;
1018 -
1019 - if(full_path_n > 0) {
1020 - for(uint16_t i = 0; i < full_path_n; i++) {
1021 - // Skip the localhost slot here — Bug A's
1022 - // live-state walk below populates
1023 - // parent_descendants[localhost] authoritatively
1024 - // from rrdhost_status(). If we appended here too
1025 - // we would either double-count or mis-tag offline
1026 - // children as STREAMING.
1027 - if(UUIDeq(full_path_ids[i], localhost->host_id))
1028 - continue;
1029 -
1030 - bool source_local = (i == 0);
1031 - ND_UUID source_uuid = source_local ? empty_uuid : full_path_ids[i - 1];
1032 - streaming_topology_descendants_append(parent_descendants,
1033 - full_path_ids[i], host, STREAMING_TOPOLOGY_RECEIVED_STREAMING, source_local, source_uuid);
1034 - }
1035 - }
1036 - }
1037 - dfe_done(host);
1038 - }
2252 +static void streaming_topology_v1_emit_link_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2253 + buffer_json_member_add_object(wb, "links");
2254 + {
2255 + buffer_json_member_add_uint64(wb, "rows", payload->links_used);
2256 + buffer_json_member_add_array(wb, "columns");
2257 + streaming_topology_v1_emit_link_columns(wb);
2258 + buffer_json_array_close(wb);
2259 + buffer_json_member_add_array(wb, "values");
2260 +
2261 + streaming_topology_v1_emit_values_start(wb);
2262 + for(size_t i = 0; i < payload->links_used; i++)
2263 + buffer_json_add_array_item_uint64(wb, payload->links[i].src_actor);
2264 + streaming_topology_v1_emit_values_end(wb);
2265 +
2266 + streaming_topology_v1_emit_values_start(wb);
2267 + for(size_t i = 0; i < payload->links_used; i++)
2268 + buffer_json_add_array_item_uint64(wb, payload->links[i].dst_actor);
2269 + streaming_topology_v1_emit_values_end(wb);
2270 +
2271 + streaming_topology_v1_emit_values_start(wb);
2272 + for(size_t i = 0; i < payload->links_used; i++)
2273 + buffer_json_add_array_item_string(wb, payload->links[i].type);
2274 + streaming_topology_v1_emit_values_end(wb);
2275 +
2276 + streaming_topology_v1_emit_values_start(wb);
2277 + for(size_t i = 0; i < payload->links_used; i++)
2278 + buffer_json_add_array_item_string(wb, payload->links[i].state[0] ? payload->links[i].state : NULL);
2279 + streaming_topology_v1_emit_values_end(wb);
2280 +
2281 + streaming_topology_v1_emit_values_start(wb);
2282 + for(size_t i = 0; i < payload->links_used; i++)
2283 + buffer_json_add_array_item_string(wb, payload->links[i].port_name[0] ? payload->links[i].port_name : NULL);
2284 + streaming_topology_v1_emit_values_end(wb);
2285 +
2286 + streaming_topology_v1_emit_values_start(wb);
2287 + for(size_t i = 0; i < payload->links_used; i++)
2288 + streaming_topology_v1_add_timestamp(wb, payload->links[i].discovered_at_ut);
2289 + streaming_topology_v1_emit_values_end(wb);
2290 +
2291 + streaming_topology_v1_emit_values_start(wb);
2292 + for(size_t i = 0; i < payload->links_used; i++)
2293 + streaming_topology_v1_add_timestamp(wb, payload->links[i].last_seen_ut);
2294 + streaming_topology_v1_emit_values_end(wb);
2295 +
2296 + streaming_topology_v1_emit_values_start(wb);
2297 + for(size_t i = 0; i < payload->links_used; i++)
2298 + buffer_json_add_array_item_int64(wb, payload->links[i].hops);
2299 + streaming_topology_v1_emit_values_end(wb);
2300 +
2301 + streaming_topology_v1_emit_values_start(wb);
2302 + for(size_t i = 0; i < payload->links_used; i++)
2303 + buffer_json_add_array_item_uint64(wb, 1);
2304 + streaming_topology_v1_emit_values_end(wb);
2305 +
2306 + streaming_topology_v1_emit_values_start(wb);
2307 + for(size_t i = 0; i < payload->links_used; i++)
2308 + buffer_json_add_array_item_uint64(wb, payload->links[i].connections);
2309 + streaming_topology_v1_emit_values_end(wb);
2310 +
2311 + streaming_topology_v1_emit_values_start(wb);
2312 + for(size_t i = 0; i < payload->links_used; i++)
2313 + buffer_json_add_array_item_uint64(wb, payload->links[i].replication_instances);
2314 + streaming_topology_v1_emit_values_end(wb);
2315 +
2316 + streaming_topology_v1_emit_values_start(wb);
2317 + for(size_t i = 0; i < payload->links_used; i++)
2318 + buffer_json_add_array_item_double(wb, payload->links[i].replication_completion);
2319 + streaming_topology_v1_emit_values_end(wb);
2320 +
2321 + streaming_topology_v1_emit_values_start(wb);
2322 + for(size_t i = 0; i < payload->links_used; i++)
2323 + buffer_json_add_array_item_uint64(wb, payload->links[i].collected_metrics);
2324 + streaming_topology_v1_emit_values_end(wb);
2325 +
2326 + streaming_topology_v1_emit_values_start(wb);
2327 + for(size_t i = 0; i < payload->links_used; i++)
2328 + buffer_json_add_array_item_uint64(wb, payload->links[i].collected_instances);
2329 + streaming_topology_v1_emit_values_end(wb);
2330 +
2331 + streaming_topology_v1_emit_values_start(wb);
2332 + for(size_t i = 0; i < payload->links_used; i++)
2333 + buffer_json_add_array_item_uint64(wb, payload->links[i].collected_contexts);
2334 + streaming_topology_v1_emit_values_end(wb);
2335
1040 - // Bug A fix: localhost live-state classification and descendants.
1041 - // On an apex agent, the path-based walk above can miss localhost
1042 - // as a parent because each child's stored path on us still only
1043 - // contains the child itself until a sparse trigger (retention
1044 - // boundary, node_id update) fires on that child. localhost is
1045 - // appended to the wire format only at JSON-emit time, not stored.
1046 - // We walk rrdhost_root_index once and authoritatively populate
1047 - // parent_child_count[localhost] and parent_descendants[localhost].
1048 - // The path walk above skips localhost so this is the single writer
1049 - // for localhost descendants.
1050 - {
1051 - char localhost_guid[UUID_STR_LEN];
1052 - if(streaming_topology_uuid_guid(localhost->host_id, localhost_guid, sizeof(localhost_guid))) {
1053 - uint32_t live_count = 0;
1054 - ND_UUID empty_uuid_for_live = {};
1055 - RRDHOST *h;
1056 - dfe_start_read(rrdhost_root_index, h) {
1057 - if(h == localhost)
1058 - continue;
1059 -
1060 - if(rrdhost_is_virtual(h)) {
1061 - streaming_topology_descendants_append(parent_descendants,
1062 - localhost->host_id, h,
1063 - STREAMING_TOPOLOGY_RECEIVED_VIRTUAL, true, empty_uuid_for_live);
1064 - continue;
1065 - }
1066 -
1067 - RRDHOST_STATUS hs;
1068 - rrdhost_status(h, now, &hs, RRDHOST_STATUS_ALL);
1069 -
1070 - if(hs.ingest.type == RRDHOST_INGEST_TYPE_CHILD &&
1071 - (hs.ingest.status == RRDHOST_INGEST_STATUS_ONLINE ||
1072 - hs.ingest.status == RRDHOST_INGEST_STATUS_REPLICATING)) {
1073 - live_count++;
1074 -
1075 - streaming_topology_descendants_append(parent_descendants,
1076 - localhost->host_id, h,
1077 - STREAMING_TOPOLOGY_RECEIVED_STREAMING, false, empty_uuid_for_live);
1078 - }
1079 - else {
1080 - streaming_topology_descendants_append(parent_descendants,
1081 - localhost->host_id, h,
1082 - STREAMING_TOPOLOGY_RECEIVED_STALE, false, empty_uuid_for_live);
1083 - }
1084 - }
1085 - dfe_done(h);
2336 + buffer_json_array_close(wb);
2337 + }
2338 + buffer_json_object_close(wb);
2339 +}
2340
1087 - uint32_t *existing = dictionary_get(parent_child_count, localhost_guid);
1088 - if(existing)
1089 - *existing = live_count;
1090 - else if(live_count > 0)
1091 - dictionary_set(parent_child_count, localhost_guid, &live_count, sizeof(live_count));
1092 - }
2341 +static bool streaming_topology_v1_link_is_type(STREAMING_TOPOLOGY_V1_LINK *link, const char *link_type) {
2342 + return link && link_type && strcmp(link->type, link_type) == 0;
2343 +}
2344 +
2345 +static size_t streaming_topology_v1_count_links_by_type(STREAMING_TOPOLOGY_V1_PAYLOAD *payload, const char *link_type) {
2346 + size_t count = 0;
2347 + for(size_t i = 0; i < payload->links_used; i++) {
2348 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2349 + count++;
2350 + }
2351 + return count;
2352 +}
2353 +
2354 +static void streaming_topology_v1_emit_evidence_section(
2355 + BUFFER *wb,
2356 + STREAMING_TOPOLOGY_V1_PAYLOAD *payload,
2357 + const char *evidence_type,
2358 + const char *link_type) {
2359 + buffer_json_member_add_object(wb, evidence_type);
2360 + {
2361 + buffer_json_member_add_string(wb, "type", evidence_type);
2362 + buffer_json_member_add_object(wb, "table");
2363 + {
2364 + buffer_json_member_add_uint64(wb, "rows", streaming_topology_v1_count_links_by_type(payload, link_type));
2365 + buffer_json_member_add_array(wb, "columns");
2366 + streaming_topology_v1_emit_evidence_columns(wb);
2367 + buffer_json_array_close(wb);
2368 + buffer_json_member_add_array(wb, "values");
2369 +
2370 + streaming_topology_v1_emit_values_start(wb);
2371 + for(size_t i = 0; i < payload->links_used; i++) {
2372 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2373 + buffer_json_add_array_item_uint64(wb, i);
2374 }
2375 + streaming_topology_v1_emit_values_end(wb);
2376
1095 - buffer_json_member_add_object(wb, "data");
1096 - {
1097 - size_t actors_total = 0;
1098 - size_t links_total = 0;
2377 + streaming_topology_v1_emit_values_start(wb);
2378 + for(size_t i = 0; i < payload->links_used; i++) {
2379 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2380 + buffer_json_add_array_item_uint64(wb, payload->links[i].src_actor);
2381 + }
2382 + streaming_topology_v1_emit_values_end(wb);
2383
1100 - buffer_json_member_add_string(wb, "schema_version", "2.0");
1101 - buffer_json_member_add_string(wb, "source", "streaming");
1102 - buffer_json_member_add_string(wb, "layer", "infra");
1103 - char localhost_agent_id[256];
1104 - streaming_topology_agent_id_for_host(localhost, localhost_agent_id, sizeof(localhost_agent_id));
1105 - buffer_json_member_add_string(wb, "agent_id", localhost_agent_id);
1106 - buffer_json_member_add_datetime_rfc3339(wb, "collected_at", now_ut, true);
2384 + streaming_topology_v1_emit_values_start(wb);
2385 + for(size_t i = 0; i < payload->links_used; i++) {
2386 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2387 + buffer_json_add_array_item_uint64(wb, payload->links[i].dst_actor);
2388 + }
2389 + streaming_topology_v1_emit_values_end(wb);
2390
1108 - // --- Phase 3: emit actors ---
1109 - buffer_json_member_add_array(wb, "actors");
1110 - {
1111 - RRDHOST *host;
1112 - dfe_start_read(rrdhost_root_index, host) {
1113 - RRDHOST_STATUS s;
1114 - rrdhost_status(host, now, &s, RRDHOST_STATUS_ALL);
1115 - const char *hostname = rrdhost_hostname(host);
1116 - char host_actor_id[256];
1117 - streaming_topology_actor_id_for_host(host, host_actor_id, sizeof(host_actor_id));
1118 -
1119 - // classify node type by role
1120 - // stale = no connections ever (ARCHIVED status with 0 connections)
1121 - // vnode, parent, child determined by role in topology
1122 - const char *node_type;
1123 - if(rrdhost_is_virtual(host))
1124 - node_type = "vnode";
1125 - else if(host != localhost && s.ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
1126 - node_type = "stale";
1127 - else {
1128 - uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, host);
1129 - node_type = (cc && *cc > 0) ? "parent" : "child";
1130 - }
2391 + streaming_topology_v1_emit_values_start(wb);
2392 + for(size_t i = 0; i < payload->links_used; i++) {
2393 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2394 + buffer_json_add_array_item_string(wb, payload->links[i].type);
2395 + }
2396 + streaming_topology_v1_emit_values_end(wb);
2397
1132 - uint32_t child_count = 0;
1133 - {
1134 - uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, host);
1135 - if(cc) child_count = *cc;
1136 - }
2398 + streaming_topology_v1_emit_values_start(wb);
2399 + for(size_t i = 0; i < payload->links_used; i++) {
2400 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2401 + buffer_json_add_array_item_string(wb, payload->links[i].state[0] ? payload->links[i].state : NULL);
2402 + }
2403 + streaming_topology_v1_emit_values_end(wb);
2404
1138 - // compute severity (same logic as table function)
1139 - const char *severity = "normal";
1140 - if(!rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST)) {
1141 - switch(s.ingest.status) {
1142 - case RRDHOST_INGEST_STATUS_OFFLINE:
1143 - case RRDHOST_INGEST_STATUS_ARCHIVED:
1144 - severity = "critical";
1145 - break;
1146 - default:
1147 - break;
1148 - }
1149 - if(strcmp(severity, "normal") == 0) {
1150 - switch(s.stream.status) {
1151 - case RRDHOST_STREAM_STATUS_OFFLINE:
1152 - if(s.stream.reason != STREAM_HANDSHAKE_SP_NO_DESTINATION)
1153 - severity = "warning";
1154 - break;
1155 - default:
1156 - break;
1157 - }
1158 - }
1159 - }
2405 + streaming_topology_v1_emit_values_start(wb);
2406 + for(size_t i = 0; i < payload->links_used; i++) {
2407 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2408 + buffer_json_add_array_item_string(wb, payload->links[i].port_name[0] ? payload->links[i].port_name : NULL);
2409 + }
2410 + streaming_topology_v1_emit_values_end(wb);
2411
1161 - actors_total++;
1162 - {
1163 - uint8_t one = 1;
1164 - dictionary_set(emitted_actors, host_actor_id, &one, sizeof(one));
1165 - }
1166 - buffer_json_add_array_item_object(wb);
1167 - {
1168 - buffer_json_member_add_string(wb, "actor_id", host_actor_id);
1169 - buffer_json_member_add_string(wb, "actor_type", node_type);
1170 - buffer_json_member_add_string(wb, "layer", "infra");
1171 - buffer_json_member_add_string(wb, "source", "streaming");
1172 - streaming_topology_add_host_match(wb, host);
1173 -
1174 - buffer_json_member_add_object(wb, "attributes");
1175 - {
1176 - // intrinsic identity
1177 - buffer_json_member_add_string(wb, "display_name", hostname);
1178 - buffer_json_member_add_string(wb, "node_type", node_type);
1179 - buffer_json_member_add_string(wb, "severity", severity);
1180 - buffer_json_member_add_uint64(wb, "child_count", child_count);
1181 - buffer_json_member_add_string(wb, "ephemerality", rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST) ? "ephemeral" : "permanent");
1182 - buffer_json_member_add_string(wb, "agent_name", rrdhost_program_name(host));
1183 - buffer_json_member_add_string(wb, "agent_version", rrdhost_program_version(host));
1184 -
1185 - // system info (intrinsic hardware/OS fields)
1186 - rrdhost_system_info_to_json_object_fields(wb, s.host->system_info);
1187 -
1188 - // health/alerts (intrinsic to the node running health)
1189 - buffer_json_member_add_string(wb, "health_status", rrdhost_health_status_to_string(s.health.status));
1190 - if(s.health.status == RRDHOST_HEALTH_STATUS_RUNNING) {
1191 - buffer_json_member_add_uint64(wb, "health_critical", s.health.alerts.critical);
1192 - buffer_json_member_add_uint64(wb, "health_warning", s.health.alerts.warning);
1193 - buffer_json_member_add_uint64(wb, "health_clear", s.health.alerts.clear);
1194 - }
1195 - else {
1196 - buffer_json_member_add_uint64(wb, "health_critical", 0);
1197 - buffer_json_member_add_uint64(wb, "health_warning", 0);
1198 - buffer_json_member_add_uint64(wb, "health_clear", 0);
1199 - }
1200 - }
1201 - buffer_json_object_close(wb); // attributes
1202 -
1203 - buffer_json_member_add_object(wb, "labels");
1204 - {
1205 - // topology-specific labels (used by facets and tables)
1206 - buffer_json_member_add_string(wb, "hostname", hostname);
1207 - buffer_json_member_add_string(wb, "node_type", node_type);
1208 - buffer_json_member_add_string(wb, "severity", severity);
1209 - buffer_json_member_add_string(wb, "ephemerality", rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST) ? "ephemeral" : "permanent");
1210 - buffer_json_member_add_string(wb, "ingest_status", rrdhost_ingest_status_to_string(s.ingest.status));
1211 - buffer_json_member_add_string(wb, "stream_status", rrdhost_streaming_status_to_string(s.stream.status));
1212 - buffer_json_member_add_string(wb, "ml_status", rrdhost_ml_status_to_string(s.ml.status));
1213 - buffer_json_member_add_string(wb, "display_name", hostname);
1214 -
1215 - // Host labels are nested to avoid collisions with reserved
1216 - // topology label keys such as hostname/node_type/severity.
1217 - buffer_json_member_add_object(wb, "host_labels");
1218 - {
1219 - rrdlabels_to_buffer_json_members(host->rrdlabels, wb);
1220 - }
1221 - buffer_json_object_close(wb); // host_labels
1222 - }
1223 - buffer_json_object_close(wb); // labels
1224 -
1225 - // streaming_path: array of actor_ids for highlight_path
1226 - {
1227 - ND_UUID path_ids[128];
1228 - uint16_t path_n = streaming_topology_get_path_ids(host, 0, path_ids, 128);
1229 - buffer_json_member_add_array(wb, "streaming_path");
1230 - for(uint16_t pi = 0; pi < path_n; pi++) {
1231 - char path_actor_id[256];
1232 - streaming_topology_actor_id_for_uuid(path_ids[pi], path_actor_id, sizeof(path_actor_id));
1233 - buffer_json_add_array_item_string(wb, path_actor_id);
1234 - }
1235 - buffer_json_array_close(wb);
1236 - }
1237 -
1238 - // received_nodes: all nodes this parent receives data for (drives bullet count + type)
1239 - if(child_count > 0) {
1240 - buffer_json_member_add_array(wb, "received_nodes");
1241 - struct streaming_topology_descendant_list *received_nodes =
1242 - streaming_topology_descendants_get(parent_descendants, host);
1243 - if(received_nodes) {
1244 - for(size_t i = 0; i < received_nodes->used; i++) {
1245 - struct streaming_topology_descendant *descendant = &received_nodes->items[i];
1246 - RRDHOST *rn_host = descendant->host;
1247 -
1248 - if(rn_host == host)
1249 - continue;
1250 -
1251 - buffer_json_add_array_item_object(wb);
1252 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(rn_host));
1253 - buffer_json_member_add_string(wb, "type",
1254 - streaming_topology_received_type_to_string((enum streaming_topology_received_type)descendant->type));
1255 - buffer_json_object_close(wb);
1256 - }
1257 - }
1258 - buffer_json_array_close(wb);
1259 - }
1260 -
1261 - // --- per-actor tables ---
1262 - buffer_json_member_add_object(wb, "tables");
1263 - {
1264 - bool is_observer = (host == localhost);
1265 -
1266 - // PARENT tables: inbound, outbound, retention
1267 - if(child_count > 0) {
1268 - // inbound table: ALL nodes this parent has data for
1269 - if(is_observer) {
1270 - buffer_json_member_add_array(wb, "inbound");
1271 - {
1272 - RRDHOST *ih;
1273 - dfe_start_read(rrdhost_root_index, ih) {
1274 - char ih_actor_id[256];
1275 - streaming_topology_actor_id_for_host(ih, ih_actor_id, sizeof(ih_actor_id));
1276 -
1277 - RRDHOST_STATUS ihs;
1278 - rrdhost_status(ih, now, &ihs, RRDHOST_STATUS_ALL);
1279 -
1280 - // determine node type
1281 - const char *ih_node_type;
1282 - if(rrdhost_is_virtual(ih))
1283 - ih_node_type = "vnode";
1284 - else if(ih != localhost && ihs.ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
1285 - ih_node_type = "stale";
1286 - else {
1287 - uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, ih);
1288 - ih_node_type = (cc && *cc > 0) ? "parent" : "child";
1289 - }
1290 -
1291 - // determine source: who sends this host's data to us
1292 - const char *src_hostname = NULL;
1293 - char src_actor_id[256] = "";
1294 - if(ih == host) {
1295 - src_hostname = "local";
1296 - }
1297 - else if(rrdhost_is_virtual(ih)) {
1298 - src_hostname = "local";
1299 - }
1300 - else {
1301 - ND_UUID ih_path[128];
1302 - uint16_t ih_pn = streaming_topology_get_path_ids(ih, 0, ih_path, 128);
1303 - char src_guid[UUID_STR_LEN] = "";
1304 - for(uint16_t pi = 0; pi < ih_pn; pi++) {
1305 - if(UUIDeq(ih_path[pi], host->host_id) && pi > 0) {
1306 - uuid_unparse_lower(ih_path[pi - 1].uuid, src_guid);
1307 - streaming_topology_actor_id_from_guid(src_guid, src_actor_id, sizeof(src_actor_id));
1308 - RRDHOST *src = rrdhost_find_by_guid(src_guid);
1309 - src_hostname = src ? rrdhost_hostname(src) : src_guid;
1310 - break;
1311 - }
1312 - }
1313 - if(!src_hostname) src_hostname = "unknown";
1314 - }
1315 -
1316 - buffer_json_add_array_item_object(wb);
1317 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(ih));
1318 - buffer_json_member_add_string(wb, "name_id", ih_actor_id);
1319 - buffer_json_member_add_string(wb, "received_from", src_hostname);
1320 - if(src_actor_id[0])
1321 - buffer_json_member_add_string(wb, "received_from_id", src_actor_id);
1322 - buffer_json_member_add_string(wb, "node_type", ih_node_type);
1323 - buffer_json_member_add_string(wb, "ingest_status", rrdhost_ingest_status_to_string(ihs.ingest.status));
1324 - buffer_json_member_add_int64(wb, "hops", ihs.ingest.hops);
1325 - buffer_json_member_add_uint64(wb, "collected_metrics", ihs.ingest.collected.metrics);
1326 - buffer_json_member_add_uint64(wb, "collected_instances", ihs.ingest.collected.instances);
1327 - buffer_json_member_add_uint64(wb, "collected_contexts", ihs.ingest.collected.contexts);
1328 - buffer_json_member_add_double(wb, "repl_completion", ihs.ingest.replication.completion);
1329 - buffer_json_member_add_time_t(wb, "ingest_age", ihs.ingest.since ? ihs.now - ihs.ingest.since : 0);
1330 - buffer_json_member_add_string(wb, "ssl", ihs.ingest.ssl ? "SSL" : "PLAIN");
1331 - buffer_json_member_add_uint64(wb, "alerts_critical",
1332 - ihs.health.status == RRDHOST_HEALTH_STATUS_RUNNING ? ihs.health.alerts.critical : 0);
1333 - buffer_json_member_add_uint64(wb, "alerts_warning",
1334 - ihs.health.status == RRDHOST_HEALTH_STATUS_RUNNING ? ihs.health.alerts.warning : 0);
1335 - buffer_json_object_close(wb);
1336 - }
1337 - dfe_done(ih);
1338 - }
1339 - buffer_json_array_close(wb); // inbound
1340 - }
1341 - else {
1342 - // non-observer parent: use precomputed descendants
1343 - buffer_json_member_add_array(wb, "inbound");
1344 - {
1345 - struct streaming_topology_descendant_list *inbound_nodes =
1346 - streaming_topology_descendants_get(parent_descendants, host);
1347 - if(inbound_nodes) {
1348 - for(size_t i = 0; i < inbound_nodes->used; i++) {
1349 - struct streaming_topology_descendant *descendant = &inbound_nodes->items[i];
1350 - RRDHOST *ih = descendant->host;
1351 - const char *src_hostname = NULL;
1352 - char src_actor_id[256] = "";
1353 - RRDHOST_STATUS ihs;
1354 -
1355 - if(descendant->source_local)
1356 - src_hostname = "local";
1357 - else if(!UUIDiszero(descendant->source_uuid)) {
1358 - char src_guid[UUID_STR_LEN];
1359 - uuid_unparse_lower(descendant->source_uuid.uuid, src_guid);
1360 - streaming_topology_actor_id_from_guid(src_guid, src_actor_id, sizeof(src_actor_id));
1361 - RRDHOST *src = rrdhost_find_by_guid(src_guid);
1362 - src_hostname = src ? rrdhost_hostname(src) : src_guid;
1363 - }
1364 - else
1365 - src_hostname = "unknown";
1366 -
1367 - char ih_actor_id[256];
1368 - streaming_topology_actor_id_for_host(ih, ih_actor_id, sizeof(ih_actor_id));
1369 -
1370 - rrdhost_status(ih, now, &ihs, RRDHOST_STATUS_ALL);
1371 -
1372 - const char *ih_node_type;
1373 - if(rrdhost_is_virtual(ih))
1374 - ih_node_type = "vnode";
1375 - else if(ih != localhost && ihs.ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
1376 - ih_node_type = "stale";
1377 - else {
1378 - uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, ih);
1379 - ih_node_type = (cc && *cc > 0) ? "parent" : "child";
1380 - }
1381 -
1382 - buffer_json_add_array_item_object(wb);
1383 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(ih));
1384 - buffer_json_member_add_string(wb, "name_id", ih_actor_id);
1385 - buffer_json_member_add_string(wb, "received_from", src_hostname);
1386 - if(src_actor_id[0])
1387 - buffer_json_member_add_string(wb, "received_from_id", src_actor_id);
1388 - buffer_json_member_add_string(wb, "node_type", ih_node_type);
1389 - buffer_json_member_add_string(wb, "ingest_status", rrdhost_ingest_status_to_string(ihs.ingest.status));
1390 - buffer_json_member_add_int64(wb, "hops", ihs.ingest.hops);
1391 - buffer_json_member_add_uint64(wb, "collected_metrics", ihs.ingest.collected.metrics);
1392 - buffer_json_member_add_uint64(wb, "collected_instances", ihs.ingest.collected.instances);
1393 - buffer_json_member_add_uint64(wb, "collected_contexts", ihs.ingest.collected.contexts);
1394 - buffer_json_member_add_double(wb, "repl_completion", ihs.ingest.replication.completion);
1395 - buffer_json_member_add_time_t(wb, "ingest_age", ihs.ingest.since ? ihs.now - ihs.ingest.since : 0);
1396 - buffer_json_member_add_string(wb, "ssl", ihs.ingest.ssl ? "SSL" : "PLAIN");
1397 - buffer_json_member_add_uint64(wb, "alerts_critical",
1398 - ihs.health.status == RRDHOST_HEALTH_STATUS_RUNNING ? ihs.health.alerts.critical : 0);
1399 - buffer_json_member_add_uint64(wb, "alerts_warning",
1400 - ihs.health.status == RRDHOST_HEALTH_STATUS_RUNNING ? ihs.health.alerts.warning : 0);
1401 - buffer_json_object_close(wb);
1402 - }
1403 - }
1404 - }
1405 - buffer_json_array_close(wb); // inbound
1406 - }
1407 -
1408 - // retention table: ALL nodes with DB retention (including localhost)
1409 - if(is_observer) {
1410 - buffer_json_member_add_array(wb, "retention");
1411 - {
1412 - RRDHOST *rh;
1413 - dfe_start_read(rrdhost_root_index, rh) {
1414 - RRDHOST_STATUS rs;
1415 - rrdhost_status(rh, now, &rs, RRDHOST_STATUS_ALL);
1416 -
1417 - if(!rs.db.first_time_s && !rs.db.last_time_s)
1418 - continue;
1419 -
1420 - char rh_actor_id[256];
1421 - streaming_topology_actor_id_for_host(rh, rh_actor_id, sizeof(rh_actor_id));
1422 -
1423 - buffer_json_add_array_item_object(wb);
1424 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(rh));
1425 - buffer_json_member_add_string(wb, "name_id", rh_actor_id);
1426 - buffer_json_member_add_string(wb, "db_status", rrdhost_db_status_to_string(rs.db.status));
1427 - buffer_json_member_add_uint64(wb, "db_from", rs.db.first_time_s * MSEC_PER_SEC);
1428 - buffer_json_member_add_uint64(wb, "db_to", rs.db.last_time_s * MSEC_PER_SEC);
1429 - if(rs.db.first_time_s && rs.db.last_time_s && rs.db.last_time_s > rs.db.first_time_s)
1430 - buffer_json_member_add_uint64(wb, "db_duration", rs.db.last_time_s - rs.db.first_time_s);
1431 - else
1432 - buffer_json_member_add_uint64(wb, "db_duration", 0);
1433 - buffer_json_member_add_uint64(wb, "db_metrics", rs.db.metrics);
1434 - buffer_json_member_add_uint64(wb, "db_instances", rs.db.instances);
1435 - buffer_json_member_add_uint64(wb, "db_contexts", rs.db.contexts);
1436 - buffer_json_object_close(wb);
1437 - }
1438 - dfe_done(rh);
1439 - }
1440 - buffer_json_array_close(wb); // retention
1441 - }
1442 - }
1443 -
1444 - // outbound table: ALL nodes this actor streams to its parent
1445 - if(is_observer && s.stream.status != RRDHOST_STREAM_STATUS_DISABLED) {
1446 - bool stream_connected = (s.stream.status == RRDHOST_STREAM_STATUS_ONLINE ||
1447 - s.stream.status == RRDHOST_STREAM_STATUS_REPLICATING);
1448 -
1449 - // find our streaming destination (only when connected)
1450 - const char *dst_hostname = NULL;
1451 - char dst_actor_id[256] = "";
1452 - if(stream_connected) {
1453 - ND_UUID path_ids[16];
1454 - uint16_t n_ids = rrdhost_stream_path_get_host_ids(host, 0, path_ids, 16);
1455 - for(uint16_t pi = 0; pi < n_ids; pi++) {
1456 - if(!UUIDeq(path_ids[pi], host->host_id)) {
1457 - char guid[UUID_STR_LEN];
1458 - uuid_unparse_lower(path_ids[pi].uuid, guid);
1459 - streaming_topology_actor_id_from_guid(guid, dst_actor_id, sizeof(dst_actor_id));
1460 - RRDHOST *dst_host = rrdhost_find_by_guid(guid);
1461 - if(dst_host)
1462 - dst_hostname = rrdhost_hostname(dst_host);
1463 - break;
1464 - }
1465 - }
1466 - if(!dst_hostname) dst_hostname = s.stream.peers.peer.ip;
1467 - }
1468 -
1469 - buffer_json_member_add_array(wb, "outbound");
1470 - {
1471 - RRDHOST *oh;
1472 - dfe_start_read(rrdhost_root_index, oh) {
1473 - char oh_actor_id[256];
1474 - streaming_topology_actor_id_for_host(oh, oh_actor_id, sizeof(oh_actor_id));
1475 -
1476 - RRDHOST_STATUS ohs;
1477 - rrdhost_status(oh, now, &ohs, RRDHOST_STATUS_ALL);
1478 -
1479 - // determine node type
1480 - const char *oh_node_type;
1481 - if(rrdhost_is_virtual(oh))
1482 - oh_node_type = "vnode";
1483 - else if(ohs.ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
1484 - oh_node_type = "stale";
1485 - else {
1486 - uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, oh);
1487 - oh_node_type = (cc && *cc > 0) ? "parent" : "child";
1488 - }
1489 -
1490 - RRDHOST_STREAMING_STATUS oh_ss = (oh == host) ? s.stream.status : ohs.stream.status;
1491 - bool oh_streaming = (oh_ss == RRDHOST_STREAM_STATUS_ONLINE || oh_ss == RRDHOST_STREAM_STATUS_REPLICATING);
1492 -
1493 - buffer_json_add_array_item_object(wb);
1494 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(oh));
1495 - buffer_json_member_add_string(wb, "name_id", oh_actor_id);
1496 - if(dst_hostname && oh_streaming) {
1497 - buffer_json_member_add_string(wb, "streamed_to", dst_hostname);
1498 - if(dst_actor_id[0])
1499 - buffer_json_member_add_string(wb, "streamed_to_id", dst_actor_id);
1500 - }
1501 - buffer_json_member_add_string(wb, "node_type", oh_node_type);
1502 - buffer_json_member_add_string(wb, "stream_status", rrdhost_streaming_status_to_string(oh_ss));
1503 -
1504 - if(oh == host && oh_streaming) {
1505 - buffer_json_member_add_uint64(wb, "hops", s.stream.hops);
1506 - buffer_json_member_add_string(wb, "ssl", s.stream.ssl ? "SSL" : "PLAIN");
1507 - buffer_json_member_add_string(wb, "compression", s.stream.compression ? "COMPRESSED" : "UNCOMPRESSED");
1508 - }
1509 -
1510 - buffer_json_object_close(wb);
1511 - }
1512 - dfe_done(oh);
1513 - }
1514 - buffer_json_array_close(wb); // outbound
1515 - }
1516 - else if(!is_observer && child_count > 0) {
1517 - // non-observer parent: use precomputed descendants
1518 - // find this parent's outbound destination (next hop in its own path)
1519 - const char *dst_hostname = NULL;
1520 -
1521 - buffer_json_member_add_array(wb, "outbound");
1522 - {
1523 - char dst_actor_id[256] = "";
1524 - char dst_hostname_fallback[UUID_STR_LEN] = "";
1525 - ND_UUID host_path[128];
1526 - uint16_t host_pn = streaming_topology_get_path_ids(host, 0, host_path, 128);
1527 - for(uint16_t pi = 0; pi < host_pn; pi++) {
1528 - if(UUIDeq(host_path[pi], host->host_id) && pi + 1 < host_pn) {
1529 - uuid_unparse_lower(host_path[pi + 1].uuid, dst_hostname_fallback);
1530 - streaming_topology_actor_id_from_guid(dst_hostname_fallback, dst_actor_id, sizeof(dst_actor_id));
1531 - RRDHOST *dst = rrdhost_find_by_guid(dst_hostname_fallback);
1532 - dst_hostname = dst ? rrdhost_hostname(dst) : dst_hostname_fallback;
1533 - break;
1534 - }
1535 - }
1536 -
1537 - struct streaming_topology_descendant_list *outbound_nodes =
1538 - streaming_topology_descendants_get(parent_descendants, host);
1539 - if(outbound_nodes) {
1540 - for(size_t i = 0; i < outbound_nodes->used; i++) {
1541 - RRDHOST *oh = outbound_nodes->items[i].host;
1542 - char oh_actor_id[256];
1543 - RRDHOST_STATUS ohs;
1544 - streaming_topology_actor_id_for_host(oh, oh_actor_id, sizeof(oh_actor_id));
1545 -
1546 - rrdhost_status(oh, now, &ohs, RRDHOST_STATUS_ALL);
1547 -
1548 - const char *oh_node_type;
1549 - if(rrdhost_is_virtual(oh))
1550 - oh_node_type = "vnode";
1551 - else if(ohs.ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
1552 - oh_node_type = "stale";
1553 - else {
1554 - uint32_t *cc = streaming_topology_parent_child_count_get(parent_child_count, oh);
1555 - oh_node_type = (cc && *cc > 0) ? "parent" : "child";
1556 - }
1557 -
1558 - RRDHOST_STREAMING_STATUS oh_ss = (oh == host) ? s.stream.status : ohs.stream.status;
1559 - bool oh_streaming = (oh_ss == RRDHOST_STREAM_STATUS_ONLINE || oh_ss == RRDHOST_STREAM_STATUS_REPLICATING);
1560 -
1561 - buffer_json_add_array_item_object(wb);
1562 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(oh));
1563 - buffer_json_member_add_string(wb, "name_id", oh_actor_id);
1564 - if(dst_hostname && oh_streaming) {
1565 - buffer_json_member_add_string(wb, "streamed_to", dst_hostname);
1566 - if(dst_actor_id[0])
1567 - buffer_json_member_add_string(wb, "streamed_to_id", dst_actor_id);
1568 - }
1569 - buffer_json_member_add_string(wb, "node_type", oh_node_type);
1570 - buffer_json_member_add_string(wb, "stream_status", rrdhost_streaming_status_to_string(oh_ss));
1571 - if(oh == host && oh_streaming) {
1572 - buffer_json_member_add_uint64(wb, "hops", s.stream.hops);
1573 - buffer_json_member_add_string(wb, "ssl", s.stream.ssl ? "SSL" : "PLAIN");
1574 - buffer_json_member_add_string(wb, "compression", s.stream.compression ? "COMPRESSED" : "UNCOMPRESSED");
1575 - }
1576 - buffer_json_object_close(wb);
1577 - }
1578 - }
1579 - }
1580 - buffer_json_array_close(wb); // outbound
1581 - }
1582 -
1583 - // streaming_path table: per-hop data (uses the public API)
1584 - rrdhost_stream_path_to_json(wb, host, "streaming_path", false);
1585 -
1586 - // retention table for non-parent actors: rows = observers (parents)
1587 - // in single-observer mode, 1 row from localhost
1588 - if(child_count == 0) {
1589 - buffer_json_member_add_array(wb, "retention");
1590 - {
1591 - char observer_actor_id[256];
1592 - streaming_topology_actor_id_for_host(localhost, observer_actor_id, sizeof(observer_actor_id));
1593 -
1594 - buffer_json_add_array_item_object(wb);
1595 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(localhost));
1596 - buffer_json_member_add_string(wb, "name_id", observer_actor_id);
1597 - buffer_json_member_add_string(wb, "db_status", rrdhost_db_status_to_string(s.db.status));
1598 - buffer_json_member_add_uint64(wb, "db_from", s.db.first_time_s * MSEC_PER_SEC);
1599 - buffer_json_member_add_uint64(wb, "db_to", s.db.last_time_s * MSEC_PER_SEC);
1600 - if(s.db.first_time_s && s.db.last_time_s && s.db.last_time_s > s.db.first_time_s)
1601 - buffer_json_member_add_uint64(wb, "db_duration", s.db.last_time_s - s.db.first_time_s);
1602 - else
1603 - buffer_json_member_add_uint64(wb, "db_duration", 0);
1604 - buffer_json_member_add_uint64(wb, "db_metrics", s.db.metrics);
1605 - buffer_json_member_add_uint64(wb, "db_instances", s.db.instances);
1606 - buffer_json_member_add_uint64(wb, "db_contexts", s.db.contexts);
1607 - buffer_json_object_close(wb);
1608 - }
1609 - buffer_json_array_close(wb); // retention
1610 - }
1611 - }
1612 - buffer_json_object_close(wb); // tables
1613 - }
1614 - buffer_json_object_close(wb); // actor
1615 - }
1616 - dfe_done(host);
2412 + streaming_topology_v1_emit_values_start(wb);
2413 + for(size_t i = 0; i < payload->links_used; i++) {
2414 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2415 + streaming_topology_v1_add_timestamp(wb, payload->links[i].discovered_at_ut);
2416 }
2417 + streaming_topology_v1_emit_values_end(wb);
2418
1619 - // --- Bug C synthesis: emit "parent" actors for upstream entries
1620 - // not represented by a host in rrdhost_root_index. The agent may
1621 - // have STREAM_PATH metadata (hostname, hops, since, capabilities)
1622 - // about upstream parents that are not locally registered as
1623 - // RRDHOSTs (e.g., on a child viewing its own topology, the parent
1624 - // is in localhost->stream.path.array but not in rrdhost_root_index).
1625 - // This pass walks every host's stored path and emits a "parent"
1626 - // actor for each unique upstream entry, with attributes sourced
1627 - // directly from the path data. The merge layer fills missing
1628 - // attributes from the parent's own response.
1629 - {
1630 - struct streaming_topology_synth_ctx synth = {
1631 - .wb = wb,
1632 - .local_actor_ids = local_actor_ids,
1633 - .emitted_actors = emitted_actors,
1634 - .emitted_links = emitted_links,
1635 - .actors_total = &actors_total,
1636 - .links_total = NULL,
1637 - .has_prev = false,
1638 - };
1639 -
1640 - RRDHOST *sh;
1641 - dfe_start_read(rrdhost_root_index, sh) {
1642 - // Walk from slot 1: slot 0 of any host's stored path is
1643 - // the host itself (origin) and is already emitted by
1644 - // Phase 3 if it is in rrdhost_root_index. Synthesizing
1645 - // parent actors only makes sense for upstream entries
1646 - // (slot >= 1).
1647 - rrdhost_stream_path_visit(sh, 1, streaming_topology_synth_actor_visitor, &synth);
1648 - }
1649 - dfe_done(sh);
2419 + streaming_topology_v1_emit_values_start(wb);
2420 + for(size_t i = 0; i < payload->links_used; i++) {
2421 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2422 + streaming_topology_v1_add_timestamp(wb, payload->links[i].last_seen_ut);
2423 }
2424 + streaming_topology_v1_emit_values_end(wb);
2425 +
2426 +#define STREAMING_TOPOLOGY_LINK_INT_VALUES(member) do { \
2427 + streaming_topology_v1_emit_values_start(wb); \
2428 + for(size_t i = 0; i < payload->links_used; i++) { \
2429 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type)) \
2430 + buffer_json_add_array_item_int64(wb, payload->links[i].member); \
2431 + } \
2432 + streaming_topology_v1_emit_values_end(wb); \
2433 + } while(0)
2434 +#define STREAMING_TOPOLOGY_LINK_UINT_VALUES(member) do { \
2435 + streaming_topology_v1_emit_values_start(wb); \
2436 + for(size_t i = 0; i < payload->links_used; i++) { \
2437 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type)) \
2438 + buffer_json_add_array_item_uint64(wb, payload->links[i].member); \
2439 + } \
2440 + streaming_topology_v1_emit_values_end(wb); \
2441 + } while(0)
2442 +
2443 + STREAMING_TOPOLOGY_LINK_INT_VALUES(hops);
2444 + STREAMING_TOPOLOGY_LINK_UINT_VALUES(connections);
2445 + STREAMING_TOPOLOGY_LINK_UINT_VALUES(replication_instances);
2446 +#undef STREAMING_TOPOLOGY_LINK_INT_VALUES
2447 +
2448 + streaming_topology_v1_emit_values_start(wb);
2449 + for(size_t i = 0; i < payload->links_used; i++) {
2450 + if(streaming_topology_v1_link_is_type(&payload->links[i], link_type))
2451 + buffer_json_add_array_item_double(wb, payload->links[i].replication_completion);
2452 + }
2453 + streaming_topology_v1_emit_values_end(wb);
2454
1652 - buffer_json_array_close(wb); // actors
2455 + STREAMING_TOPOLOGY_LINK_UINT_VALUES(collected_metrics);
2456 + STREAMING_TOPOLOGY_LINK_UINT_VALUES(collected_instances);
2457 + STREAMING_TOPOLOGY_LINK_UINT_VALUES(collected_contexts);
2458 +#undef STREAMING_TOPOLOGY_LINK_UINT_VALUES
2459
1654 - // --- Phase 4: emit links from streaming_path ---
1655 - // nodes with an active path get streaming/virtual links
1656 - // nodes without a path (stale/offline) get a stale link to localhost
1657 - char localhost_actor_id[256];
1658 - streaming_topology_actor_id_for_host(localhost, localhost_actor_id, sizeof(localhost_actor_id));
2460 + buffer_json_array_close(wb);
2461 + }
2462 + buffer_json_object_close(wb);
2463 + }
2464 + buffer_json_object_close(wb);
2465 +}
2466
1660 - buffer_json_member_add_array(wb, "links");
1661 - {
1662 - RRDHOST *host;
1663 - dfe_start_read(rrdhost_root_index, host) {
1664 - // skip localhost — it's the root of the tree
1665 - if(host == localhost)
1666 - continue;
2467 +static void streaming_topology_v1_emit_evidence_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2468 + buffer_json_member_add_object(wb, "evidence");
2469 + {
2470 + streaming_topology_v1_emit_evidence_section(wb, payload, "streaming_link", "streaming");
2471 + streaming_topology_v1_emit_evidence_section(wb, payload, "virtual_link", "virtual");
2472 + streaming_topology_v1_emit_evidence_section(wb, payload, "stale_link", "stale");
2473 + }
2474 + buffer_json_object_close(wb);
2475 +}
2476 +
2477 +static void streaming_topology_v1_emit_actor_labels_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2478 + buffer_json_member_add_object(wb, "actor_labels");
2479 + {
2480 + buffer_json_member_add_string(wb, "type", "actor_labels");
2481 + buffer_json_member_add_object(wb, "table");
2482 + {
2483 + buffer_json_member_add_uint64(wb, "rows", payload->labels_used);
2484 + buffer_json_member_add_array(wb, "columns");
2485 + streaming_topology_v1_emit_actor_label_columns(wb);
2486 + buffer_json_array_close(wb);
2487 + buffer_json_member_add_array(wb, "values");
2488 +
2489 +#define STREAMING_TOPOLOGY_LABEL_UINT_VALUES(member) do { \
2490 + streaming_topology_v1_emit_values_start(wb); \
2491 + for(size_t i = 0; i < payload->labels_used; i++) \
2492 + buffer_json_add_array_item_uint64(wb, payload->labels[i].member); \
2493 + streaming_topology_v1_emit_values_end(wb); \
2494 + } while(0)
2495 +#define STREAMING_TOPOLOGY_LABEL_STRING_VALUES(member) do { \
2496 + streaming_topology_v1_emit_values_start(wb); \
2497 + for(size_t i = 0; i < payload->labels_used; i++) \
2498 + buffer_json_add_array_item_string(wb, payload->labels[i].member[0] ? payload->labels[i].member : NULL); \
2499 + streaming_topology_v1_emit_values_end(wb); \
2500 + } while(0)
2501 +
2502 + STREAMING_TOPOLOGY_LABEL_UINT_VALUES(actor);
2503 + STREAMING_TOPOLOGY_LABEL_STRING_VALUES(key);
2504 + STREAMING_TOPOLOGY_LABEL_STRING_VALUES(value);
2505 + STREAMING_TOPOLOGY_LABEL_STRING_VALUES(source);
2506 + STREAMING_TOPOLOGY_LABEL_STRING_VALUES(kind);
2507 +
2508 + streaming_topology_v1_emit_values_start(wb);
2509 + for(size_t i = 0; i < payload->labels_used; i++)
2510 + streaming_topology_v1_add_nullable_uint(wb, payload->labels[i].has_value_index, payload->labels[i].value_index);
2511 + streaming_topology_v1_emit_values_end(wb);
2512 +
2513 +#undef STREAMING_TOPOLOGY_LABEL_UINT_VALUES
2514 +#undef STREAMING_TOPOLOGY_LABEL_STRING_VALUES
2515 +
2516 + buffer_json_array_close(wb);
2517 + }
2518 + buffer_json_object_close(wb);
2519 + }
2520 + buffer_json_object_close(wb);
2521 +}
2522
1668 - RRDHOST_STATUS s;
1669 - rrdhost_status(host, now, &s, RRDHOST_STATUS_ALL);
2523 +static void streaming_topology_v1_emit_stream_path_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2524 + buffer_json_member_add_object(wb, "stream_path");
2525 + {
2526 + buffer_json_member_add_string(wb, "type", "stream_path");
2527 + buffer_json_member_add_object(wb, "table");
2528 + {
2529 + buffer_json_member_add_uint64(wb, "rows", payload->stream_path_used);
2530 + buffer_json_member_add_array(wb, "columns");
2531 + streaming_topology_v1_emit_stream_path_columns(wb);
2532 + buffer_json_array_close(wb);
2533 + buffer_json_member_add_array(wb, "values");
2534 +
2535 +#define STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(member) do { \
2536 + streaming_topology_v1_emit_values_start(wb); \
2537 + for(size_t i = 0; i < payload->stream_path_used; i++) \
2538 + buffer_json_add_array_item_uint64(wb, payload->stream_path_rows[i].member); \
2539 + streaming_topology_v1_emit_values_end(wb); \
2540 + } while(0)
2541 +#define STREAMING_TOPOLOGY_STREAM_PATH_STRING_VALUES(member) do { \
2542 + streaming_topology_v1_emit_values_start(wb); \
2543 + for(size_t i = 0; i < payload->stream_path_used; i++) \
2544 + buffer_json_add_array_item_string(wb, payload->stream_path_rows[i].member[0] ? payload->stream_path_rows[i].member : NULL); \
2545 + streaming_topology_v1_emit_values_end(wb); \
2546 + } while(0)
2547 +
2548 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(actor);
2549 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(path_actor);
2550 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(path_index);
2551 + STREAMING_TOPOLOGY_STREAM_PATH_STRING_VALUES(hostname);
2552 + STREAMING_TOPOLOGY_STREAM_PATH_STRING_VALUES(host_id);
2553 + STREAMING_TOPOLOGY_STREAM_PATH_STRING_VALUES(node_id);
2554 + STREAMING_TOPOLOGY_STREAM_PATH_STRING_VALUES(claim_id);
2555 +
2556 + streaming_topology_v1_emit_values_start(wb);
2557 + for(size_t i = 0; i < payload->stream_path_used; i++)
2558 + buffer_json_add_array_item_int64(wb, payload->stream_path_rows[i].hops);
2559 + streaming_topology_v1_emit_values_end(wb);
2560 +
2561 + streaming_topology_v1_emit_values_start(wb);
2562 + for(size_t i = 0; i < payload->stream_path_used; i++)
2563 + streaming_topology_v1_add_timestamp(wb, payload->stream_path_rows[i].since_ut);
2564 + streaming_topology_v1_emit_values_end(wb);
2565 +
2566 + streaming_topology_v1_emit_values_start(wb);
2567 + for(size_t i = 0; i < payload->stream_path_used; i++)
2568 + streaming_topology_v1_add_timestamp(wb, payload->stream_path_rows[i].first_time_ut);
2569 + streaming_topology_v1_emit_values_end(wb);
2570 +
2571 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(start_time_ms);
2572 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(shutdown_time_ms);
2573 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(capabilities);
2574 + STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES(flags);
2575 +
2576 +#undef STREAMING_TOPOLOGY_STREAM_PATH_UINT_VALUES
2577 +#undef STREAMING_TOPOLOGY_STREAM_PATH_STRING_VALUES
2578 + buffer_json_array_close(wb);
2579 + }
2580 + buffer_json_object_close(wb);
2581 + }
2582 + buffer_json_object_close(wb);
2583 +}
2584
1671 - char host_actor_id[256];
1672 - streaming_topology_actor_id_for_host(host, host_actor_id, sizeof(host_actor_id));
2585 +static void streaming_topology_v1_emit_retention_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2586 + buffer_json_member_add_object(wb, "retention");
2587 + {
2588 + buffer_json_member_add_string(wb, "type", "retention");
2589 + buffer_json_member_add_object(wb, "table");
2590 + {
2591 + buffer_json_member_add_uint64(wb, "rows", payload->retention_used);
2592 + buffer_json_member_add_array(wb, "columns");
2593 + streaming_topology_v1_emit_retention_columns(wb);
2594 + buffer_json_array_close(wb);
2595 + buffer_json_member_add_array(wb, "values");
2596 +
2597 +#define STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(member) do { \
2598 + streaming_topology_v1_emit_values_start(wb); \
2599 + for(size_t i = 0; i < payload->retention_used; i++) \
2600 + buffer_json_add_array_item_uint64(wb, payload->retention_rows[i].member); \
2601 + streaming_topology_v1_emit_values_end(wb); \
2602 + } while(0)
2603 +
2604 + STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(actor);
2605 + STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(observer_actor);
2606 +
2607 + streaming_topology_v1_emit_values_start(wb);
2608 + for(size_t i = 0; i < payload->retention_used; i++)
2609 + buffer_json_add_array_item_string(wb, payload->retention_rows[i].db_status[0] ? payload->retention_rows[i].db_status : NULL);
2610 + streaming_topology_v1_emit_values_end(wb);
2611 +
2612 + streaming_topology_v1_emit_values_start(wb);
2613 + for(size_t i = 0; i < payload->retention_used; i++)
2614 + streaming_topology_v1_add_timestamp(wb, payload->retention_rows[i].db_from_ut);
2615 + streaming_topology_v1_emit_values_end(wb);
2616 +
2617 + streaming_topology_v1_emit_values_start(wb);
2618 + for(size_t i = 0; i < payload->retention_used; i++)
2619 + streaming_topology_v1_add_timestamp(wb, payload->retention_rows[i].db_to_ut);
2620 + streaming_topology_v1_emit_values_end(wb);
2621 +
2622 + STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(db_duration);
2623 + STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(db_metrics);
2624 + STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(db_instances);
2625 + STREAMING_TOPOLOGY_RETENTION_UINT_VALUES(db_contexts);
2626 +#undef STREAMING_TOPOLOGY_RETENTION_UINT_VALUES
2627 + buffer_json_array_close(wb);
2628 + }
2629 + buffer_json_object_close(wb);
2630 + }
2631 + buffer_json_object_close(wb);
2632 +}
2633
1674 - bool is_vnode = rrdhost_is_virtual(host);
2634 +static void streaming_topology_v1_emit_inbound_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2635 + buffer_json_member_add_object(wb, "inbound");
2636 + {
2637 + buffer_json_member_add_string(wb, "type", "inbound");
2638 + buffer_json_member_add_object(wb, "table");
2639 + {
2640 + buffer_json_member_add_uint64(wb, "rows", payload->inbound_used);
2641 + buffer_json_member_add_array(wb, "columns");
2642 + streaming_topology_v1_emit_inbound_columns(wb);
2643 + buffer_json_array_close(wb);
2644 + buffer_json_member_add_array(wb, "values");
2645 +
2646 + streaming_topology_v1_emit_values_start(wb);
2647 + for(size_t i = 0; i < payload->inbound_used; i++)
2648 + buffer_json_add_array_item_uint64(wb, payload->inbound_rows[i].parent_actor);
2649 + streaming_topology_v1_emit_values_end(wb);
2650 +
2651 + streaming_topology_v1_emit_values_start(wb);
2652 + for(size_t i = 0; i < payload->inbound_used; i++)
2653 + buffer_json_add_array_item_uint64(wb, payload->inbound_rows[i].child_actor);
2654 + streaming_topology_v1_emit_values_end(wb);
2655 +
2656 + streaming_topology_v1_emit_values_start(wb);
2657 + for(size_t i = 0; i < payload->inbound_used; i++)
2658 + streaming_topology_v1_add_nullable_uint(wb,
2659 + payload->inbound_rows[i].has_source_actor, payload->inbound_rows[i].source_actor);
2660 + streaming_topology_v1_emit_values_end(wb);
2661 +
2662 +#define STREAMING_TOPOLOGY_INBOUND_STRING_VALUES(member) do { \
2663 + streaming_topology_v1_emit_values_start(wb); \
2664 + for(size_t i = 0; i < payload->inbound_used; i++) \
2665 + buffer_json_add_array_item_string(wb, payload->inbound_rows[i].member[0] ? payload->inbound_rows[i].member : NULL); \
2666 + streaming_topology_v1_emit_values_end(wb); \
2667 + } while(0)
2668 +#define STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(member) do { \
2669 + streaming_topology_v1_emit_values_start(wb); \
2670 + for(size_t i = 0; i < payload->inbound_used; i++) \
2671 + buffer_json_add_array_item_uint64(wb, payload->inbound_rows[i].member); \
2672 + streaming_topology_v1_emit_values_end(wb); \
2673 + } while(0)
2674 +
2675 + STREAMING_TOPOLOGY_INBOUND_STRING_VALUES(received_type);
2676 + STREAMING_TOPOLOGY_INBOUND_STRING_VALUES(ingest_status);
2677 +
2678 + streaming_topology_v1_emit_values_start(wb);
2679 + for(size_t i = 0; i < payload->inbound_used; i++)
2680 + buffer_json_add_array_item_int64(wb, payload->inbound_rows[i].hops);
2681 + streaming_topology_v1_emit_values_end(wb);
2682 +
2683 + STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(collected_metrics);
2684 + STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(collected_instances);
2685 + STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(collected_contexts);
2686 +
2687 + streaming_topology_v1_emit_values_start(wb);
2688 + for(size_t i = 0; i < payload->inbound_used; i++)
2689 + buffer_json_add_array_item_double(wb, payload->inbound_rows[i].replication_completion);
2690 + streaming_topology_v1_emit_values_end(wb);
2691 +
2692 + STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(ingest_age);
2693 + STREAMING_TOPOLOGY_INBOUND_STRING_VALUES(ssl);
2694 + STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(alerts_critical);
2695 + STREAMING_TOPOLOGY_INBOUND_UINT_VALUES(alerts_warning);
2696 +#undef STREAMING_TOPOLOGY_INBOUND_STRING_VALUES
2697 +#undef STREAMING_TOPOLOGY_INBOUND_UINT_VALUES
2698 + buffer_json_array_close(wb);
2699 + }
2700 + buffer_json_object_close(wb);
2701 + }
2702 + buffer_json_object_close(wb);
2703 +}
2704
1676 - // determine link target and type from streaming_path
1677 - const char *link_type = NULL;
1678 - char target_actor_id[256] = "";
2705 +static void streaming_topology_v1_emit_outbound_table(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2706 + buffer_json_member_add_object(wb, "outbound");
2707 + {
2708 + buffer_json_member_add_string(wb, "type", "outbound");
2709 + buffer_json_member_add_object(wb, "table");
2710 + {
2711 + buffer_json_member_add_uint64(wb, "rows", payload->outbound_used);
2712 + buffer_json_member_add_array(wb, "columns");
2713 + streaming_topology_v1_emit_outbound_columns(wb);
2714 + buffer_json_array_close(wb);
2715 + buffer_json_member_add_array(wb, "values");
2716 +
2717 + streaming_topology_v1_emit_values_start(wb);
2718 + for(size_t i = 0; i < payload->outbound_used; i++)
2719 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].sender_actor);
2720 + streaming_topology_v1_emit_values_end(wb);
2721 +
2722 + streaming_topology_v1_emit_values_start(wb);
2723 + for(size_t i = 0; i < payload->outbound_used; i++)
2724 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].node_actor);
2725 + streaming_topology_v1_emit_values_end(wb);
2726 +
2727 + streaming_topology_v1_emit_values_start(wb);
2728 + for(size_t i = 0; i < payload->outbound_used; i++)
2729 + streaming_topology_v1_add_nullable_uint(wb,
2730 + payload->outbound_rows[i].has_destination_actor, payload->outbound_rows[i].destination_actor);
2731 + streaming_topology_v1_emit_values_end(wb);
2732 +
2733 + streaming_topology_v1_emit_values_start(wb);
2734 + for(size_t i = 0; i < payload->outbound_used; i++)
2735 + buffer_json_add_array_item_string(wb, payload->outbound_rows[i].stream_status[0] ? payload->outbound_rows[i].stream_status : NULL);
2736 + streaming_topology_v1_emit_values_end(wb);
2737 +
2738 + streaming_topology_v1_emit_values_start(wb);
2739 + for(size_t i = 0; i < payload->outbound_used; i++)
2740 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].stream_age);
2741 + streaming_topology_v1_emit_values_end(wb);
2742 +
2743 + streaming_topology_v1_emit_values_start(wb);
2744 + for(size_t i = 0; i < payload->outbound_used; i++)
2745 + buffer_json_add_array_item_int64(wb, payload->outbound_rows[i].hops);
2746 + streaming_topology_v1_emit_values_end(wb);
2747 +
2748 + streaming_topology_v1_emit_values_start(wb);
2749 + for(size_t i = 0; i < payload->outbound_used; i++)
2750 + buffer_json_add_array_item_string(wb, payload->outbound_rows[i].ssl[0] ? payload->outbound_rows[i].ssl : NULL);
2751 + streaming_topology_v1_emit_values_end(wb);
2752 +
2753 + streaming_topology_v1_emit_values_start(wb);
2754 + for(size_t i = 0; i < payload->outbound_used; i++)
2755 + buffer_json_add_array_item_string(wb, payload->outbound_rows[i].compression[0] ? payload->outbound_rows[i].compression : NULL);
2756 + streaming_topology_v1_emit_values_end(wb);
2757 +
2758 + streaming_topology_v1_emit_values_start(wb);
2759 + for(size_t i = 0; i < payload->outbound_used; i++)
2760 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].collected_metrics);
2761 + streaming_topology_v1_emit_values_end(wb);
2762 +
2763 + streaming_topology_v1_emit_values_start(wb);
2764 + for(size_t i = 0; i < payload->outbound_used; i++)
2765 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].collected_instances);
2766 + streaming_topology_v1_emit_values_end(wb);
2767 +
2768 + streaming_topology_v1_emit_values_start(wb);
2769 + for(size_t i = 0; i < payload->outbound_used; i++)
2770 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].collected_contexts);
2771 + streaming_topology_v1_emit_values_end(wb);
2772 +
2773 + streaming_topology_v1_emit_values_start(wb);
2774 + for(size_t i = 0; i < payload->outbound_used; i++)
2775 + buffer_json_add_array_item_uint64(wb, payload->outbound_rows[i].replication_instances);
2776 + streaming_topology_v1_emit_values_end(wb);
2777 +
2778 + streaming_topology_v1_emit_values_start(wb);
2779 + for(size_t i = 0; i < payload->outbound_used; i++)
2780 + buffer_json_add_array_item_double(wb, payload->outbound_rows[i].replication_completion);
2781 + streaming_topology_v1_emit_values_end(wb);
2782
1680 - ND_UUID link_ids[2];
1681 - uint16_t link_n = streaming_topology_get_path_ids(host, 0, link_ids, 2);
2783 + buffer_json_array_close(wb);
2784 + }
2785 + buffer_json_object_close(wb);
2786 + }
2787 + buffer_json_object_close(wb);
2788 +}
2789
1683 - if(is_vnode) {
1684 - // vnodes do not stream; they are collected by localhost.
1685 - snprintfz(target_actor_id, sizeof(target_actor_id), "%s", localhost_actor_id);
1686 - link_type = "virtual";
1687 - }
1688 - else if(link_n >= 2) {
1689 - // children/parents: path[1] is the direct parent
1690 - streaming_topology_actor_id_for_uuid(link_ids[1], target_actor_id, sizeof(target_actor_id));
1691 - link_type = "streaming";
1692 - }
1693 - else {
1694 - // no active path — stale link to localhost
1695 - snprintfz(target_actor_id, sizeof(target_actor_id), "%s", localhost_actor_id);
1696 - link_type = "stale";
1697 - }
2790 +static void streaming_topology_v1_emit_detail_tables(BUFFER *wb, STREAMING_TOPOLOGY_V1_PAYLOAD *payload) {
2791 + buffer_json_member_add_object(wb, "tables");
2792 + {
2793 + buffer_json_member_add_object(wb, "actor");
2794 + {
2795 + streaming_topology_v1_emit_actor_labels_table(wb, payload);
2796 + streaming_topology_v1_emit_stream_path_table(wb, payload);
2797 + streaming_topology_v1_emit_retention_table(wb, payload);
2798 + streaming_topology_v1_emit_inbound_table(wb, payload);
2799 + streaming_topology_v1_emit_outbound_table(wb, payload);
2800 + }
2801 + buffer_json_object_close(wb);
2802 + }
2803 + buffer_json_object_close(wb);
2804 +}
2805
1699 - const char *hostname = rrdhost_hostname(host);
2806 +int function_streaming_topology(BUFFER *wb, const char *function, BUFFER *payload __maybe_unused, const char *source __maybe_unused) {
2807 + time_t now = now_realtime_sec();
2808 + usec_t now_ut = now_realtime_usec();
2809
1701 - char link_key[600];
1702 - snprintfz(link_key, sizeof(link_key), "%s|%s", host_actor_id, target_actor_id);
1703 - if(dictionary_get(emitted_links, link_key))
1704 - continue;
2810 + struct streaming_topology_options options = { 0 };
2811 + streaming_topology_parse_options(function, &options);
2812 + char *function_copy = options.function_copy;
2813
1706 - {
1707 - uint8_t one = 1;
1708 - dictionary_set(emitted_links, link_key, &one, sizeof(one));
1709 - }
2814 + if(options.info_only) {
2815 + buffer_flush(wb);
2816 + wb->content_type = CT_APPLICATION_JSON;
2817 + buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_DEFAULT);
2818 + streaming_topology_v1_emit_response_metadata(wb);
2819 + buffer_json_member_add_time_t(wb, "expires", now_realtime_sec() + STREAMING_FUNCTION_UPDATE_EVERY);
2820 + buffer_json_finalize(wb);
2821 + freez(function_copy);
2822 + return HTTP_RESP_OK;
2823 + }
2824
1711 - links_total++;
1712 - buffer_json_add_array_item_object(wb);
1713 - {
1714 - buffer_json_member_add_string(wb, "layer", "infra");
1715 - buffer_json_member_add_string(wb, "protocol", "streaming");
1716 - buffer_json_member_add_string(wb, "link_type", link_type);
1717 - buffer_json_member_add_string(wb, "src_actor_id", host_actor_id);
1718 - buffer_json_member_add_string(wb, "dst_actor_id", target_actor_id);
1719 - buffer_json_member_add_string(wb, "state", rrdhost_ingest_status_to_string(s.ingest.status));
1720 - buffer_json_member_add_datetime_rfc3339(wb, "discovered_at",
1721 - ((uint64_t)(s.ingest.since ? s.ingest.since : now)) * USEC_PER_SEC, true);
1722 - buffer_json_member_add_datetime_rfc3339(wb, "last_seen", now_ut, true);
1723 -
1724 - buffer_json_member_add_object(wb, "dst");
1725 - {
1726 - buffer_json_member_add_object(wb, "attributes");
1727 - {
1728 - buffer_json_member_add_string(wb, "port_name", hostname);
1729 - }
1730 - buffer_json_object_close(wb);
1731 - }
1732 - buffer_json_object_close(wb);
2825 + DICTIONARY *parent_child_count = dictionary_create_advanced(
2826 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2827 + NULL, sizeof(uint32_t));
2828 + DICTIONARY *parent_descendants = dictionary_create_advanced(
2829 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2830 + NULL, sizeof(struct streaming_topology_descendant_list));
2831
1734 - buffer_json_member_add_object(wb, "metrics");
1735 - {
1736 - buffer_json_member_add_uint64(wb, "hops", s.ingest.hops);
1737 - if(strcmp(link_type, "virtual") != 0) {
1738 - buffer_json_member_add_uint64(wb, "connections", s.host->stream.rcv.status.connections);
1739 - buffer_json_member_add_uint64(wb, "replication_instances", s.ingest.replication.instances);
1740 - buffer_json_member_add_double(wb, "replication_completion", s.ingest.replication.completion);
1741 - buffer_json_member_add_uint64(wb, "collected_metrics", s.ingest.collected.metrics);
1742 - buffer_json_member_add_uint64(wb, "collected_instances", s.ingest.collected.instances);
1743 - buffer_json_member_add_uint64(wb, "collected_contexts", s.ingest.collected.contexts);
1744 - }
1745 - }
1746 - buffer_json_object_close(wb); // metrics
1747 - }
1748 - buffer_json_object_close(wb); // link
1749 - }
1750 - dfe_done(host);
1751 - }
2832 + STREAMING_TOPOLOGY_V1_PAYLOAD topology = {
2833 + .actor_index = dictionary_create_advanced(
2834 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2835 + NULL, sizeof(uint64_t)),
2836 + .emitted_links = dictionary_create_advanced(
2837 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
2838 + NULL, sizeof(uint8_t)),
2839 + };
2840
1753 - // --- Bug C link synthesis: emit streaming links for consecutive
1754 - // path slots not already emitted by Phase 4. Phase 4 writes one
1755 - // direct link per local non-localhost actor and registers it in
1756 - // emitted_links; this pass adds localhost's own upstream link and
1757 - // any deeper multi-hop links.
1758 - // discovered_at / last_seen derive from STREAM_PATH timestamps so
1759 - // the merge layer can reconcile views from multiple agents.
1760 - {
1761 - struct streaming_topology_synth_ctx synth = {
1762 - .wb = wb,
1763 - .local_actor_ids = local_actor_ids,
1764 - .emitted_actors = emitted_actors,
1765 - .emitted_links = emitted_links,
1766 - .actors_total = NULL,
1767 - .links_total = &links_total,
1768 - .has_prev = false,
1769 - };
1770 -
1771 - RRDHOST *lh;
1772 - dfe_start_read(rrdhost_root_index, lh) {
1773 - synth.has_prev = false;
1774 - rrdhost_stream_path_visit(lh, 0, streaming_topology_synth_link_visitor, &synth);
2841 + if(!parent_child_count || !parent_descendants || !topology.actor_index || !topology.emitted_links) {
2842 + streaming_topology_v1_free(&topology);
2843 + if(parent_descendants)
2844 + dictionary_destroy(parent_descendants);
2845 + if(parent_child_count)
2846 + dictionary_destroy(parent_child_count);
2847 +
2848 + return streaming_topology_return_error(wb, function_copy,
2849 + HTTP_RESP_INTERNAL_SERVER_ERROR,
2850 + "failed to allocate streaming topology dictionaries");
2851 + }
2852 +
2853 + {
2854 + RRDHOST *host;
2855 + dfe_start_read(rrdhost_root_index, host) {
2856 + ND_UUID path_ids[128];
2857 + uint16_t n = streaming_topology_get_path_ids(host, 1, path_ids, 128);
2858 + for(uint16_t i = 0; i < n; i++) {
2859 + char guid[UUID_STR_LEN];
2860 + if(!streaming_topology_uuid_guid(path_ids[i], guid, sizeof(guid)))
2861 + continue;
2862 +
2863 + uint32_t *count = dictionary_get(parent_child_count, guid);
2864 + if(count)
2865 + (*count)++;
2866 + else {
2867 + uint32_t one = 1;
2868 + dictionary_set(parent_child_count, guid, &one, sizeof(one));
2869 }
1776 - dfe_done(lh);
2870 }
2871
1779 - buffer_json_array_close(wb); // links
2872 + ND_UUID full_path_ids[128];
2873 + uint16_t full_path_n = streaming_topology_get_path_ids(host, 0, full_path_ids, 128);
2874 + ND_UUID empty_uuid = {};
2875
1781 - buffer_json_member_add_object(wb, "stats");
1782 - {
1783 - buffer_json_member_add_uint64(wb, "actors_total", actors_total);
1784 - buffer_json_member_add_uint64(wb, "links_total", links_total);
2876 + if(rrdhost_is_virtual(host))
2877 + continue;
2878 +
2879 + for(uint16_t i = 0; i < full_path_n; i++) {
2880 + if(UUIDeq(full_path_ids[i], localhost->host_id))
2881 + continue;
2882 +
2883 + bool source_local = (i == 0);
2884 + ND_UUID source_uuid = source_local ? empty_uuid : full_path_ids[i - 1];
2885 + streaming_topology_descendants_append(parent_descendants,
2886 + full_path_ids[i], host, STREAMING_TOPOLOGY_RECEIVED_STREAMING, source_local, source_uuid);
2887 }
1786 - buffer_json_object_close(wb); // stats
2888 }
1788 - buffer_json_object_close(wb); // data
1789 -
1790 - struct streaming_topology_descendant_list *descendants;
1791 - dfe_start_write(parent_descendants, descendants) {
1792 - freez(descendants->items);
1793 - descendants->items = NULL;
1794 - descendants->used = 0;
1795 - descendants->size = 0;
2889 + dfe_done(host);
2890 + }
2891 +
2892 + {
2893 + char localhost_guid[UUID_STR_LEN];
2894 + if(streaming_topology_uuid_guid(localhost->host_id, localhost_guid, sizeof(localhost_guid))) {
2895 + uint32_t live_count = 0;
2896 + ND_UUID empty_uuid_for_live = {};
2897 + RRDHOST *host;
2898 + dfe_start_read(rrdhost_root_index, host) {
2899 + if(host == localhost)
2900 + continue;
2901 +
2902 + if(rrdhost_is_virtual(host)) {
2903 + streaming_topology_descendants_append(parent_descendants,
2904 + localhost->host_id, host,
2905 + STREAMING_TOPOLOGY_RECEIVED_VIRTUAL, true, empty_uuid_for_live);
2906 + continue;
2907 + }
2908 +
2909 + RRDHOST_STATUS status;
2910 + rrdhost_status(host, now, &status, RRDHOST_STATUS_ALL);
2911 +
2912 + if(status.ingest.type == RRDHOST_INGEST_TYPE_CHILD &&
2913 + (status.ingest.status == RRDHOST_INGEST_STATUS_ONLINE ||
2914 + status.ingest.status == RRDHOST_INGEST_STATUS_REPLICATING)) {
2915 + live_count++;
2916 + streaming_topology_descendants_append(parent_descendants,
2917 + localhost->host_id, host,
2918 + STREAMING_TOPOLOGY_RECEIVED_STREAMING, true, empty_uuid_for_live);
2919 + }
2920 + else {
2921 + streaming_topology_descendants_append(parent_descendants,
2922 + localhost->host_id, host,
2923 + STREAMING_TOPOLOGY_RECEIVED_STALE, true, empty_uuid_for_live);
2924 + }
2925 }
1797 - dfe_done(descendants);
1798 - dictionary_destroy(parent_descendants);
1799 - dictionary_destroy(parent_child_count);
1800 - dictionary_destroy(local_actor_ids);
1801 - dictionary_destroy(emitted_actors);
1802 - dictionary_destroy(emitted_links);
2926 + dfe_done(host);
2927 +
2928 + uint32_t *existing = dictionary_get(parent_child_count, localhost_guid);
2929 + if(existing)
2930 + *existing = live_count;
2931 + else if(live_count > 0)
2932 + dictionary_set(parent_child_count, localhost_guid, &live_count, sizeof(live_count));
2933 + }
2934 + }
2935 +
2936 + uint64_t local_retained_node_count = streaming_topology_v1_count_local_retained_nodes(now);
2937 + streaming_topology_v1_collect_actors(&topology, parent_child_count, local_retained_node_count, now);
2938 + streaming_topology_v1_collect_links(&topology, now, now_ut);
2939 + streaming_topology_v1_collect_actor_detail_rows(&topology, parent_descendants, now);
2940 +
2941 + buffer_flush(wb);
2942 + wb->content_type = CT_APPLICATION_JSON;
2943 + buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_DEFAULT);
2944 +
2945 + streaming_topology_v1_emit_response_metadata(wb);
2946 +
2947 + buffer_json_member_add_object(wb, "data");
2948 + {
2949 + buffer_json_member_add_string(wb, "schema_version", "netdata.topology.v1");
2950 +
2951 + buffer_json_member_add_object(wb, "producer");
2952 + {
2953 + char localhost_agent_id[256];
2954 + char localhost_guid[UUID_STR_LEN];
2955 + streaming_topology_agent_id_for_host(localhost, localhost_agent_id, sizeof(localhost_agent_id));
2956 + streaming_topology_host_guid(localhost, localhost_guid, sizeof(localhost_guid));
2957 +
2958 + buffer_json_member_add_string(wb, "source", "streaming");
2959 + buffer_json_member_add_string(wb, "instance", localhost_agent_id);
2960 + if(!UUIDiszero(localhost->node_id))
2961 + buffer_json_member_add_uuid(wb, "node_id", localhost->node_id.uuid);
2962 + if(localhost_guid[0])
2963 + buffer_json_member_add_string(wb, "machine_guid", localhost_guid);
2964 + buffer_json_member_add_string(wb, "agent_version", rrdhost_program_version(localhost));
2965 + buffer_json_member_add_string(wb, "plugin", "netdata");
2966 + buffer_json_member_add_array(wb, "capabilities");
2967 + buffer_json_add_array_item_string(wb, "topology-v1");
2968 + buffer_json_array_close(wb);
2969 + }
2970 + buffer_json_object_close(wb);
2971 +
2972 + buffer_json_member_add_datetime_rfc3339(wb, "collected_at", now_ut, true);
2973 + buffer_json_member_add_object(wb, "view");
2974 + {
2975 + buffer_json_member_add_string(wb, "id", "streaming");
2976 + buffer_json_member_add_string(wb, "scope", "node");
2977 + buffer_json_member_add_string(wb, "mode", "detailed");
2978 + buffer_json_member_add_array(wb, "group_by");
2979 + buffer_json_add_array_item_string(wb, "node");
2980 + buffer_json_array_close(wb);
2981 + }
2982 + buffer_json_object_close(wb);
2983 +
2984 + buffer_json_member_add_object(wb, "dictionaries");
2985 + {
2986 + buffer_json_member_add_array(wb, "strings");
2987 + buffer_json_array_close(wb);
2988 + }
2989 + buffer_json_object_close(wb);
2990 +
2991 + streaming_topology_v1_emit_type_registry(wb);
2992 + streaming_topology_v1_emit_presentation(wb);
2993 + streaming_topology_v1_emit_actor_table(wb, &topology);
2994 + streaming_topology_v1_emit_link_table(wb, &topology);
2995 + streaming_topology_v1_emit_evidence_table(wb, &topology);
2996 + streaming_topology_v1_emit_detail_tables(wb, &topology);
2997 +
2998 + buffer_json_member_add_object(wb, "stats");
2999 + {
3000 + buffer_json_member_add_uint64(wb, "actors", topology.actors_used);
3001 + buffer_json_member_add_uint64(wb, "links", topology.links_used);
3002 + buffer_json_member_add_uint64(wb, "evidence_rows", topology.links_used);
3003 + buffer_json_member_add_uint64(wb, "stream_path_rows", topology.stream_path_used);
3004 + buffer_json_member_add_uint64(wb, "retention_rows", topology.retention_used);
3005 + buffer_json_member_add_uint64(wb, "inbound_rows", topology.inbound_used);
3006 + buffer_json_member_add_uint64(wb, "outbound_rows", topology.outbound_used);
3007 }
3008 + buffer_json_object_close(wb);
3009 }
3010 + buffer_json_object_close(wb);
3011
3012 buffer_json_member_add_time_t(wb, "expires", now_realtime_sec() + STREAMING_FUNCTION_UPDATE_EVERY);
3013 buffer_json_finalize(wb);
3014 +
3015 + struct streaming_topology_descendant_list *descendants;
3016 + dfe_start_write(parent_descendants, descendants) {
3017 + freez(descendants->items);
3018 + descendants->items = NULL;
3019 + descendants->used = 0;
3020 + descendants->size = 0;
3021 + }
3022 + dfe_done(descendants);
3023 +
3024 + dictionary_destroy(parent_descendants);
3025 + dictionary_destroy(parent_child_count);
3026 + streaming_topology_v1_free(&topology);
3027 freez(function_copy);
3028 return HTTP_RESP_OK;
3029 }