@cryptotaxi247 / netdata / commits / 151d78c63

feat(go.d): add vnode-scoped metrics for Azure Monitor workloads (#22402)

Ilya Mashchenko committed May 4, 2026 at 11:18 UTC 151d78c63cc350a94aedc1d9b2dea58215671adb
99 files changed +10449 -879
.agents/skills/project-writing-collectors/SKILL.md
+2
@@ -72,6 +72,8 @@ The repo holds 132 go.d modules and 24 internal C plugins. Maintainer patterns l
72
73 When one collector talks to N targets (SNMP devices, remote DBs, cloud APIs, IPMI hosts, vCenter clusters), each target is a **vnode** so its metrics, alerts, and RBAC behave as if it were a separate node in Netdata Cloud. Every remote-target collector wires vnodes from the start.
74
75 +For Go v2 collectors that route one job's samples to multiple virtual nodes, use first-class `metrix.HostScope` rather than adding vnode identity as normal metric labels. Write per-resource metrics through scoped meters or vecs such as `meter.WithHostScope(scope)`, and leave metrics unscoped when they should follow the default job vnode or global host path. Scope keys must be stable for the virtual node identity; unbounded scope cardinality has the same operational cost profile as unbounded chart/cardinality growth.
76 +
77 ### 1.10 Cardinality discipline
78
79 - A chart with thousands of dimensions, or an instance list with thousands of entries, is unusable on the dashboard. The user cannot read it.
.agents/sow/specs/go-v2-host-scope.md new
+86
@@ -0,0 +1,86 @@
1 +# Go V2 Host Scope And Virtual Node Emission
2 +
3 +## Scope
4 +
5 +This spec records the framework contract for Go collector v2 host-scope routing.
6 +It applies to `pkg/metrix`, `plugin/framework/jobruntime`,
7 +`plugin/framework/chartengine`, and v2 collectors that emit metrics for remote
8 +or virtual-node targets.
9 +
10 +## Metrix Host Scope
11 +
12 +- `metrix.HostScope{}` is the default host scope.
13 +- Unscoped writes are equivalent to writes in the default scope.
14 +- A non-default host scope carries:
15 + - `ScopeKey`: stable scope partition key;
16 + - `GUID`: Netdata host/vnode GUID;
17 + - `Hostname`: Netdata host/vnode hostname when defining the host;
18 + - `Labels`: deterministic host/vnode labels.
19 +- Series identity includes host scope. The same metric name and labels can exist
20 + in default scope and multiple non-default scopes without collision.
21 +- `Read()` without `ReadHostScope` returns default-scope series only.
22 +- `Read(ReadHostScope(key))` returns only that scope's series.
23 +- `Reader.HostScopes()` enumerates all scopes present in the snapshot and is not
24 + filtered by the reader's active host scope.
25 +- Flattened synthetic series preserve the source host scope.
26 +- Scope metadata conflicts in one collect cycle are data errors surfaced through
27 + `CommitCycleSuccess() error`, not panics.
28 +
29 +## Jobruntime V2
30 +
31 +- V2 jobruntime owns host/vnode orchestration. Chartengine remains host-agnostic.
32 +- One `chartengine.Engine` is used per host scope for a job.
33 +- Scope engines are created lazily.
34 +- Default-scope metrics continue to emit under the job-level vnode when one is
35 + configured, otherwise under the global host.
36 +- Explicit non-default scopes emit under their `metrix.HostScope` GUID and host
37 + metadata.
38 +- Collection and `metrix.CommitCycleSuccess()` are still all-or-nothing.
39 +- Post-collect plan/apply/commit is per-scope partial success. A failed scope
40 + rolls back its own registry changes and does not block unrelated scopes.
41 +- Disappeared scopes are retained and read with empty scoped readers until the
42 + per-scope chartengine emits lifecycle removals. After successful removal
43 + emission, jobruntime releases scoped registry owners and destroys the scope
44 + engine.
45 +- Job cleanup emits obsolete charts for each retained scope before releasing
46 + registry owners.
47 +
48 +## Vnode Registry
49 +
50 +- V2 vnode definitions go through the shared `framework/vnoderegistry` registry.
51 +- Registry entries are keyed by host GUID and owner.
52 +- Metadata is update-on-change. A new normalized metadata value for an existing
53 + GUID replaces retained metadata and causes another `HOST_DEFINE`.
54 +- Owner release removes an entry only after the last owner for that GUID leaves.
55 +- Job-level vnode owners and explicit scoped vnode owners use separate owner
56 + namespaces.
57 +
58 +## Chartengine Runtime Metrics
59 +
60 +- Per-scope engines do not register per-scope runtime components.
61 +- Jobruntime feeds chartengine runtime samples into one job-level
62 + `chartengine.RuntimeAggregator`.
63 +- Aggregated runtime metrics do not include host-scope/workload labels.
64 +- Counter-like runtime metrics are summed across samples.
65 +- Gauge-like size metrics represent the latest successful build rollup, summed
66 + across successful engines in that rollup.
67 +- `build_seq_violation_active` is `1` when any observed engine reports a
68 + sequence violation in the rollup, otherwise `0`.
69 +
70 +## Collector Contract
71 +
72 +- V2 collectors that need per-target virtual nodes should write target metrics
73 + through `meter.WithHostScope(scope)` or equivalent scoped vec/instrument
74 + bindings.
75 +- V2 collectors should leave metrics unscoped when those metrics belong to the
76 + default job vnode/global host.
77 +- Collector-generated scope keys must be deterministic and stable for the target
78 + host/vnode identity.
79 +- Collector-generated host labels should include `_vnode_type=<source>` when a
80 + collector creates virtual nodes from an internal mechanism rather than a
81 + user-defined vnode entry. The value must identify the mechanism or source
82 + without embedding high-cardinality target values.
83 +- Azure Monitor resource-tag virtual nodes use `_vnode_type=azure_workload`.
84 + Their GUID and `ScopeKey` are the deterministic SHA1 UUID of
85 + `azure_monitor:` plus the trimmed, case-preserved tag value.
86 +- Collectors are responsible for bounding or documenting scope cardinality risk.
src/go/pkg/metrix/backend.go
+15 -15
@@ -8,21 +8,21 @@ package metrix
8 type meterBackend interface {
9 compileLabelSet(labels ...Label) LabelSet
10 registerInstrument(name string, kind metricKind, mode metricMode, opts ...InstrumentOption) (*instrumentDescriptor, error)
11 - recordGaugeSet(desc *instrumentDescriptor, value SampleValue, sets []LabelSet)
12 - recordGaugeAdd(desc *instrumentDescriptor, delta SampleValue, sets []LabelSet)
13 - recordCounterObserveTotal(desc *instrumentDescriptor, value SampleValue, sets []LabelSet)
14 - recordCounterAdd(desc *instrumentDescriptor, delta SampleValue, sets []LabelSet)
15 - recordHistogramObservePoint(desc *instrumentDescriptor, point HistogramPoint, sets []LabelSet)
16 - recordHistogramObserve(desc *instrumentDescriptor, value SampleValue, sets []LabelSet)
17 - recordSummaryObservePoint(desc *instrumentDescriptor, point SummaryPoint, sets []LabelSet)
18 - recordSummaryObserve(desc *instrumentDescriptor, value SampleValue, sets []LabelSet)
19 - recordStateSetObserve(desc *instrumentDescriptor, point StateSetPoint, sets []LabelSet)
20 - recordMeasureSetGaugeObservePoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet)
21 - recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet)
22 - recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, delta MeasureSetPoint, sets []LabelSet)
23 - recordMeasureSetGaugeSetField(desc *instrumentDescriptor, field string, value SampleValue, sets []LabelSet)
24 - recordMeasureSetCounterObserveTotalPoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet)
25 - recordMeasureSetCounterAddPoint(desc *instrumentDescriptor, delta MeasureSetPoint, sets []LabelSet)
11 + recordGaugeSet(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet)
12 + recordGaugeAdd(desc *instrumentDescriptor, scope HostScope, delta SampleValue, sets []LabelSet)
13 + recordCounterObserveTotal(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet)
14 + recordCounterAdd(desc *instrumentDescriptor, scope HostScope, delta SampleValue, sets []LabelSet)
15 + recordHistogramObservePoint(desc *instrumentDescriptor, scope HostScope, point HistogramPoint, sets []LabelSet)
16 + recordHistogramObserve(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet)
17 + recordSummaryObservePoint(desc *instrumentDescriptor, scope HostScope, point SummaryPoint, sets []LabelSet)
18 + recordSummaryObserve(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet)
19 + recordStateSetObserve(desc *instrumentDescriptor, scope HostScope, point StateSetPoint, sets []LabelSet)
20 + recordMeasureSetGaugeObservePoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet)
21 + recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet)
22 + recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, scope HostScope, delta MeasureSetPoint, sets []LabelSet)
23 + recordMeasureSetGaugeSetField(desc *instrumentDescriptor, scope HostScope, field string, value SampleValue, sets []LabelSet)
24 + recordMeasureSetCounterObserveTotalPoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet)
25 + recordMeasureSetCounterAddPoint(desc *instrumentDescriptor, scope HostScope, delta MeasureSetPoint, sets []LabelSet)
26 }
27
28 var _ meterBackend = (*storeCore)(nil)
src/go/pkg/metrix/collector_store.go
+106 -24
@@ -3,6 +3,7 @@
3 package metrix
4
5 import (
6 + "errors"
7 "fmt"
8 "maps"
9 "math"
@@ -69,6 +70,9 @@ type committedSeries struct {
70 hash64 uint64
71 key string
72 name string
73 + // hostScope is immutable after publish and partitions otherwise-identical series.
74 + hostScopeKey string
75 + hostScope HostScope
76 // labels are immutable after series publish and can be safely shared across snapshots.
77 labels []Label
78 labelsKey string
@@ -122,6 +126,8 @@ type readSnapshot struct {
126
127 type cycleFrame struct {
128 seq uint64
129 + err error
130 + hostScopes map[string]HostScope
131 gauges map[string]*stagedGauge
132 counters map[string]*stagedCounter
133 histograms map[string]*stagedHistogram
@@ -205,7 +211,7 @@ func (s *storeView) Read(opts ...ReadOption) Reader {
211 if cfg.flatten {
212 snap = flattenSnapshot(snap)
213 }
208 - return &storeReader{snap: snap, raw: cfg.raw, flattened: cfg.flatten}
214 + return &storeReader{snap: snap, raw: cfg.raw, flattened: cfg.flatten, hostScopeKey: cfg.hostScopeKey}
215 }
216
217 func (s *storeView) Write() Writer {
@@ -236,6 +242,7 @@ func (c *storeCycleController) BeginCycle() {
242 c.core.sequence++
243 c.core.active = &cycleFrame{
244 seq: c.core.sequence,
245 + hostScopes: make(map[string]HostScope),
246 gauges: make(map[string]*stagedGauge),
247 counters: make(map[string]*stagedCounter),
248 histograms: make(map[string]*stagedHistogram),
@@ -247,7 +254,7 @@ func (c *storeCycleController) BeginCycle() {
254 }
255
256 // CommitCycleSuccess publishes staged writes into a new committed snapshot.
250 -func (c *storeCycleController) CommitCycleSuccess() {
257 +func (c *storeCycleController) CommitCycleSuccess() error {
258 c.core.mu.Lock()
259 defer c.core.mu.Unlock()
260
@@ -256,23 +263,39 @@ func (c *storeCycleController) CommitCycleSuccess() {
263 }
264
265 oldSnap := c.core.snapshot.Load()
266 + if c.core.active.err != nil {
267 + abortSnap := &readSnapshot{
268 + collectMeta: oldSnap.collectMeta,
269 + series: oldSnap.series,
270 + byName: oldSnap.byName,
271 + }
272 + abortSnap.collectMeta.LastAttemptSeq = c.core.active.seq
273 + abortSnap.collectMeta.LastAttemptStatus = CollectStatusFailed
274 + c.core.snapshot.Store(abortSnap)
275 + err := c.core.active.err
276 + c.core.active = nil
277 + return err
278 + }
279 successSeq := c.core.successSeq + 1
280 next := &readSnapshot{
281 collectMeta: oldSnap.collectMeta,
282 series: make(map[string]*committedSeries, len(oldSnap.series)),
283 byName: nil,
284 }
285 + commitHostScopes := make(map[string]HostScope)
286
287 maps.Copy(next.series, oldSnap.series)
288
289 for key, staged := range c.core.active.gauges {
269 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
290 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
291 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
292 series.value = staged.value
293 markSeriesSeen(series, c.core.active.seq, successSeq)
294 }
295
296 for key, staged := range c.core.active.counters {
275 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
297 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
298 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
299
300 hadCurrent := series.desc != nil && series.desc.kind == kindCounter && series.counterCurrentSeq > 0
301 if hadCurrent {
@@ -292,7 +315,8 @@ func (c *storeCycleController) CommitCycleSuccess() {
315 }
316
317 for key, staged := range c.core.active.histograms {
295 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
318 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
319 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
320
321 if series.desc == nil {
322 panic("metrix: missing histogram descriptor")
@@ -319,7 +343,8 @@ func (c *storeCycleController) CommitCycleSuccess() {
343 }
344
345 for key, staged := range c.core.active.summaries {
322 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
346 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
347 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
348
349 if staged.desc.mode == modeStateful && len(staged.desc.summaryQuantiles()) > 0 {
350 if staged.sketch != nil {
@@ -346,20 +371,23 @@ func (c *storeCycleController) CommitCycleSuccess() {
371 }
372
373 for key, staged := range c.core.active.stateSet {
349 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
374 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
375 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
376
377 series.stateSetValues = cloneStateMap(staged.states)
378 markSeriesSeen(series, c.core.active.seq, successSeq)
379 }
380
381 for key, staged := range c.core.active.measureSetGauges {
356 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
382 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
383 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
384 series.measureSetValues = append(series.measureSetValues[:0], staged.values...)
385 markSeriesSeen(series, c.core.active.seq, successSeq)
386 }
387
388 for key, staged := range c.core.active.measureSetCounters {
362 - series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.labels, staged.labelsKey, staged.desc)
389 + commitHostScopes[staged.hostScopeKey] = staged.hostScope
390 + series := getOrCreateCommitSeries(oldSnap, next, key, staged.name, staged.hostScopeKey, staged.hostScope, staged.labels, staged.labelsKey, staged.desc)
391
392 if series.desc != nil && series.desc.kind == kindMeasureSet && series.desc.measureSet != nil && series.desc.measureSet.semantics == MeasureSetSemanticsCounter && series.measureSetCurrentSeq > 0 {
393 series.measureSetPreviousValues = append(series.measureSetPreviousValues[:0], series.measureSetValues...)
@@ -376,6 +404,7 @@ func (c *storeCycleController) CommitCycleSuccess() {
404 markSeriesSeen(series, c.core.active.seq, successSeq)
405 }
406
407 + refreshCommittedHostScopes(oldSnap, next, commitHostScopes)
408 applyCollectorRetention(next.series, c.core.retention, successSeq)
409 next.collectMeta.LastAttemptSeq = c.core.active.seq
410 next.collectMeta.LastAttemptStatus = CollectStatusSuccess
@@ -384,18 +413,21 @@ func (c *storeCycleController) CommitCycleSuccess() {
413 c.core.snapshot.Store(next)
414 c.core.successSeq = successSeq
415 c.core.active = nil
416 + return nil
417 }
418
389 -func newCommittedSeries(key, name string, labels []Label, labelsKey string, desc *instrumentDescriptor) *committedSeries {
419 +func newCommittedSeries(key, name, hostScopeKey string, hostScope HostScope, labels []Label, labelsKey string, desc *instrumentDescriptor) *committedSeries {
420 return &committedSeries{
391 - id: SeriesID(key),
392 - hash64: seriesIDHash(SeriesID(key)),
393 - key: key,
394 - name: name,
395 - labels: append([]Label(nil), labels...),
396 - labelsKey: labelsKey,
397 - desc: desc,
398 - meta: baseSeriesMeta(desc),
421 + id: SeriesID(key),
422 + hash64: seriesIDHash(SeriesID(key)),
423 + key: key,
424 + name: name,
425 + hostScopeKey: hostScopeKey,
426 + hostScope: cloneHostScope(hostScope),
427 + labels: append([]Label(nil), labels...),
428 + labelsKey: labelsKey,
429 + desc: desc,
430 + meta: baseSeriesMeta(desc),
431 }
432 }
433
@@ -412,16 +444,33 @@ func ensureCommitSeriesMutable(old, next *readSnapshot, key string) *committedSe
444 return series
445 }
446
415 -func getOrCreateCommitSeries(old, next *readSnapshot, key, name string, labels []Label, labelsKey string, desc *instrumentDescriptor) *committedSeries {
447 +func getOrCreateCommitSeries(old, next *readSnapshot, key, name, hostScopeKey string, hostScope HostScope, labels []Label, labelsKey string, desc *instrumentDescriptor) *committedSeries {
448 series := ensureCommitSeriesMutable(old, next, key)
449 if series != nil {
450 return series
451 }
420 - series = newCommittedSeries(key, name, labels, labelsKey, desc)
452 + series = newCommittedSeries(key, name, hostScopeKey, hostScope, labels, labelsKey, desc)
453 next.series[key] = series
454 return series
455 }
456
457 +func refreshCommittedHostScopes(old, next *readSnapshot, scopes map[string]HostScope) {
458 + if len(scopes) == 0 {
459 + return
460 + }
461 + for key, series := range next.series {
462 + scope, ok := scopes[series.hostScopeKey]
463 + if !ok || hostScopeEqual(series.hostScope, scope) {
464 + continue
465 + }
466 + series = ensureCommitSeriesMutable(old, next, key)
467 + if series == nil {
468 + continue
469 + }
470 + series.hostScope = cloneHostScope(scope)
471 + }
472 +}
473 +
474 func markSeriesSeen(series *committedSeries, attemptSeq, successSeq uint64) {
475 series.meta.LastSeenSuccessSeq = attemptSeq
476 series.lastSeenSuccessCycle = successSeq
@@ -463,6 +512,9 @@ func buildByName(series map[string]*committedSeries) map[string][]*committedSeri
512 }
513 for _, lst := range byName {
514 sort.Slice(lst, func(i, j int) bool {
515 + if lst[i].hostScopeKey != lst[j].hostScopeKey {
516 + return lst[i].hostScopeKey < lst[j].hostScopeKey
517 + }
518 return lst[i].labelsKey < lst[j].labelsKey
519 })
520 }
@@ -648,17 +700,47 @@ func (c *storeCore) registerInstrument(name string, kind metricKind, mode metric
700 return d, nil
701 }
702
651 -// makeSeriesKey joins metric name and canonical label key into one stable identity key.
652 -func makeSeriesKey(name, labelsKey string) string {
703 +func (c *storeCore) prepareHostScopeForWriteLocked(scope HostScope) (HostScope, bool) {
704 + scope = mustNormalizeHostScope(scope)
705 + if c.active == nil {
706 + return scope, true
707 + }
708 + if existing, ok := c.active.hostScopes[scope.ScopeKey]; ok {
709 + if hostScopeEqual(existing, scope) {
710 + return existing, true
711 + }
712 + c.recordCycleErrorLocked(fmt.Errorf("%w: scope_key=%q", ErrHostScopeConflict, scope.ScopeKey))
713 + return scope, false
714 + }
715 + c.active.hostScopes[scope.ScopeKey] = cloneHostScope(scope)
716 + return scope, true
717 +}
718 +
719 +func (c *storeCore) recordCycleErrorLocked(err error) {
720 + if err == nil || c.active == nil {
721 + return
722 + }
723 + c.active.err = errors.Join(c.active.err, err)
724 +}
725 +
726 +// makeSeriesKey joins host scope, metric name, and canonical label key into one stable identity key.
727 +func makeSeriesKey(hostScopeKey, name, labelsKey string) string {
728 + base := name
729 if labelsKey == "" {
654 - return name
730 + base = name
731 + } else {
732 + base = name + "\xfe" + labelsKey
733 + }
734 + if hostScopeKey == "" {
735 + return base
736 }
656 - return name + "\xfe" + labelsKey
737 + return hostScopeKey + "\xff" + base
738 }
739
740 func cloneCommittedSeries(s *committedSeries) *committedSeries {
741 cp := *s
742 ensureSeriesMeta(cp.desc, &cp.meta)
743 + cp.hostScope = cloneHostScope(s.hostScope)
744 // cp.labels intentionally reuses the original immutable label slice.
745 // Label identity is part of the series key and is never mutated after publish.
746 if s.stateSetValues != nil {
src/go/pkg/metrix/counter.go
+41 -23
@@ -6,6 +6,7 @@ package metrix
6 type snapshotCounterInstrument struct {
7 backend meterBackend
8 desc *instrumentDescriptor
9 + scope HostScope
10 base []LabelSet
11 }
12
@@ -13,17 +14,20 @@ type snapshotCounterInstrument struct {
14 type statefulCounterInstrument struct {
15 backend meterBackend
16 desc *instrumentDescriptor
17 + scope HostScope
18 base []LabelSet
19 }
20
21 // stagedCounter holds one in-cycle counter current total for a series identity.
22 type stagedCounter struct {
21 - key string
22 - name string
23 - labels []Label
24 - labelsKey string
25 - desc *instrumentDescriptor
26 - current SampleValue
23 + key string
24 + name string
25 + hostScopeKey string
26 + hostScope HostScope
27 + labels []Label
28 + labelsKey string
29 + desc *instrumentDescriptor
30 + current SampleValue
31 }
32
33 // Counter declares or reuses a snapshot counter under this meter.
@@ -35,6 +39,7 @@ func (m *snapshotMeter) Counter(name string, opts ...InstrumentOption) SnapshotC
39 return &snapshotCounterInstrument{
40 backend: m.backend,
41 desc: desc,
42 + scope: m.scope,
43 base: appendLabelSets(m.sets, nil),
44 }
45 }
@@ -48,22 +53,23 @@ func (m *statefulMeter) Counter(name string, opts ...InstrumentOption) StatefulC
53 return &statefulCounterInstrument{
54 backend: m.backend,
55 desc: desc,
56 + scope: m.scope,
57 base: appendLabelSets(m.sets, nil),
58 }
59 }
60
61 // ObserveTotal writes one monotonic total sample for this collect cycle.
62 func (c *snapshotCounterInstrument) ObserveTotal(v SampleValue, labels ...LabelSet) {
57 - c.backend.recordCounterObserveTotal(c.desc, v, appendLabelSets(c.base, labels))
63 + c.backend.recordCounterObserveTotal(c.desc, c.scope, v, appendLabelSets(c.base, labels))
64 }
65
66 // Add accumulates a delta for this collect cycle.
67 func (c *statefulCounterInstrument) Add(delta SampleValue, labels ...LabelSet) {
62 - c.backend.recordCounterAdd(c.desc, delta, appendLabelSets(c.base, labels))
68 + c.backend.recordCounterAdd(c.desc, c.scope, delta, appendLabelSets(c.base, labels))
69 }
70
71 // recordCounterObserveTotal writes one sampled monotonic total for snapshot counters.
66 -func (c *storeCore) recordCounterObserveTotal(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
72 +func (c *storeCore) recordCounterObserveTotal(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
73 mustFiniteSample(value)
74
75 c.mu.Lock()
@@ -77,16 +83,22 @@ func (c *storeCore) recordCounterObserveTotal(desc *instrumentDescriptor, value
83 if err != nil {
84 panic(err)
85 }
86 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
87 + if !ok {
88 + return
89 + }
90
81 - key := makeSeriesKey(desc.name, labelsKey)
91 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
92 entry, ok := c.active.counters[key]
93 if !ok {
94 entry = &stagedCounter{
85 - key: key,
86 - name: desc.name,
87 - labels: labels,
88 - labelsKey: labelsKey,
89 - desc: desc,
95 + key: key,
96 + name: desc.name,
97 + hostScopeKey: scope.ScopeKey,
98 + hostScope: scope,
99 + labels: labels,
100 + labelsKey: labelsKey,
101 + desc: desc,
102 }
103 c.active.counters[key] = entry
104 }
@@ -94,7 +106,7 @@ func (c *storeCore) recordCounterObserveTotal(desc *instrumentDescriptor, value
106 }
107
108 // recordCounterAdd accumulates delta for stateful counters.
97 -func (c *storeCore) recordCounterAdd(desc *instrumentDescriptor, delta SampleValue, sets []LabelSet) {
109 +func (c *storeCore) recordCounterAdd(desc *instrumentDescriptor, scope HostScope, delta SampleValue, sets []LabelSet) {
110 mustFiniteSample(delta)
111
112 if delta < 0 {
@@ -112,8 +124,12 @@ func (c *storeCore) recordCounterAdd(desc *instrumentDescriptor, delta SampleVal
124 if err != nil {
125 panic(err)
126 }
127 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
128 + if !ok {
129 + return
130 + }
131
116 - key := makeSeriesKey(desc.name, labelsKey)
132 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
133 entry, ok := c.active.counters[key]
134 if !ok {
135 baseline := SampleValue(0)
@@ -121,12 +137,14 @@ func (c *storeCore) recordCounterAdd(desc *instrumentDescriptor, delta SampleVal
137 baseline = existing.counterCurrent
138 }
139 entry = &stagedCounter{
124 - key: key,
125 - name: desc.name,
126 - labels: labels,
127 - labelsKey: labelsKey,
128 - desc: desc,
129 - current: baseline,
140 + key: key,
141 + name: desc.name,
142 + hostScopeKey: scope.ScopeKey,
143 + hostScope: scope,
144 + labels: labels,
145 + labelsKey: labelsKey,
146 + desc: desc,
147 + current: baseline,
148 }
149 c.active.counters[key] = entry
150 }
src/go/pkg/metrix/errors.go
+3
@@ -33,4 +33,7 @@ var (
33 errRuntimeSnapshotWrite = errors.New("metrix: runtime store supports stateful writes only")
34 errRuntimeFreshness = errors.New("metrix: runtime store freshness is fixed to FreshnessCommitted")
35 errRuntimeWindowCycle = errors.New("metrix: runtime store does not support window=cycle")
36 +
37 + // ErrHostScopeConflict reports inconsistent metadata for the same non-default host scope key.
38 + ErrHostScopeConflict = errors.New("metrix: host scope metadata conflict")
39 )
src/go/pkg/metrix/gauge.go
+42 -24
@@ -6,6 +6,7 @@ package metrix
6 type snapshotGaugeInstrument struct {
7 backend meterBackend
8 desc *instrumentDescriptor
9 + scope HostScope
10 base []LabelSet
11 }
12
@@ -13,17 +14,20 @@ type snapshotGaugeInstrument struct {
14 type statefulGaugeInstrument struct {
15 backend meterBackend
16 desc *instrumentDescriptor
17 + scope HostScope
18 base []LabelSet
19 }
20
21 // stagedGauge holds one in-cycle gauge sample for a single series identity.
22 type stagedGauge struct {
21 - key string
22 - name string
23 - labels []Label
24 - labelsKey string
25 - desc *instrumentDescriptor
26 - value SampleValue
23 + key string
24 + name string
25 + hostScopeKey string
26 + hostScope HostScope
27 + labels []Label
28 + labelsKey string
29 + desc *instrumentDescriptor
30 + value SampleValue
31 }
32
33 // Gauge declares or reuses a snapshot gauge under this meter.
@@ -35,6 +39,7 @@ func (m *snapshotMeter) Gauge(name string, opts ...InstrumentOption) SnapshotGau
39 return &snapshotGaugeInstrument{
40 backend: m.backend,
41 desc: desc,
42 + scope: m.scope,
43 base: appendLabelSets(m.sets, nil),
44 }
45 }
@@ -48,27 +53,28 @@ func (m *statefulMeter) Gauge(name string, opts ...InstrumentOption) StatefulGau
53 return &statefulGaugeInstrument{
54 backend: m.backend,
55 desc: desc,
56 + scope: m.scope,
57 base: appendLabelSets(m.sets, nil),
58 }
59 }
60
61 // Observe writes one absolute gauge value for this collect cycle.
62 func (g *snapshotGaugeInstrument) Observe(v SampleValue, labels ...LabelSet) {
57 - g.backend.recordGaugeSet(g.desc, v, appendLabelSets(g.base, labels))
63 + g.backend.recordGaugeSet(g.desc, g.scope, v, appendLabelSets(g.base, labels))
64 }
65
66 // Set overwrites the staged gauge value for this collect cycle.
67 func (g *statefulGaugeInstrument) Set(v SampleValue, labels ...LabelSet) {
62 - g.backend.recordGaugeSet(g.desc, v, appendLabelSets(g.base, labels))
68 + g.backend.recordGaugeSet(g.desc, g.scope, v, appendLabelSets(g.base, labels))
69 }
70
71 // Add accumulates on top of the committed baseline for this collect cycle.
72 func (g *statefulGaugeInstrument) Add(delta SampleValue, labels ...LabelSet) {
67 - g.backend.recordGaugeAdd(g.desc, delta, appendLabelSets(g.base, labels))
73 + g.backend.recordGaugeAdd(g.desc, g.scope, delta, appendLabelSets(g.base, labels))
74 }
75
76 // recordGaugeSet writes one gauge sample into the active frame (last-write-wins in-cycle).
71 -func (c *storeCore) recordGaugeSet(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
77 +func (c *storeCore) recordGaugeSet(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
78 mustFiniteSample(value)
79
80 c.mu.Lock()
@@ -82,16 +88,22 @@ func (c *storeCore) recordGaugeSet(desc *instrumentDescriptor, value SampleValue
88 if err != nil {
89 panic(err)
90 }
91 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
92 + if !ok {
93 + return
94 + }
95
86 - key := makeSeriesKey(desc.name, labelsKey)
96 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
97 entry, ok := c.active.gauges[key]
98 if !ok {
99 entry = &stagedGauge{
90 - key: key,
91 - name: desc.name,
92 - labels: labels,
93 - labelsKey: labelsKey,
94 - desc: desc,
100 + key: key,
101 + name: desc.name,
102 + hostScopeKey: scope.ScopeKey,
103 + hostScope: scope,
104 + labels: labels,
105 + labelsKey: labelsKey,
106 + desc: desc,
107 }
108 c.active.gauges[key] = entry
109 }
@@ -99,7 +111,7 @@ func (c *storeCore) recordGaugeSet(desc *instrumentDescriptor, value SampleValue
111 }
112
113 // recordGaugeAdd accumulates delta into the active frame using committed baseline on first write.
102 -func (c *storeCore) recordGaugeAdd(desc *instrumentDescriptor, delta SampleValue, sets []LabelSet) {
114 +func (c *storeCore) recordGaugeAdd(desc *instrumentDescriptor, scope HostScope, delta SampleValue, sets []LabelSet) {
115 mustFiniteSample(delta)
116
117 c.mu.Lock()
@@ -113,8 +125,12 @@ func (c *storeCore) recordGaugeAdd(desc *instrumentDescriptor, delta SampleValue
125 if err != nil {
126 panic(err)
127 }
128 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
129 + if !ok {
130 + return
131 + }
132
117 - key := makeSeriesKey(desc.name, labelsKey)
133 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
134 entry, ok := c.active.gauges[key]
135 if !ok {
136 baseline := SampleValue(0)
@@ -122,12 +138,14 @@ func (c *storeCore) recordGaugeAdd(desc *instrumentDescriptor, delta SampleValue
138 baseline = existing.value
139 }
140 entry = &stagedGauge{
125 - key: key,
126 - name: desc.name,
127 - labels: labels,
128 - labelsKey: labelsKey,
129 - desc: desc,
130 - value: baseline,
141 + key: key,
142 + name: desc.name,
143 + hostScopeKey: scope.ScopeKey,
144 + hostScope: scope,
145 + labels: labels,
146 + labelsKey: labelsKey,
147 + desc: desc,
148 + value: baseline,
149 }
150 c.active.gauges[key] = entry
151 }
src/go/pkg/metrix/histogram.go
+45 -27
@@ -15,6 +15,7 @@ const HistogramBucketLabel = "le"
15 type snapshotHistogramInstrument struct {
16 backend meterBackend
17 desc *instrumentDescriptor
18 + scope HostScope
19 base []LabelSet
20 }
21
@@ -22,20 +23,23 @@ type snapshotHistogramInstrument struct {
23 type statefulHistogramInstrument struct {
24 backend meterBackend
25 desc *instrumentDescriptor
26 + scope HostScope
27 base []LabelSet
28 }
29
30 // stagedHistogram holds one in-cycle histogram sample for a single series identity.
31 type stagedHistogram struct {
30 - key string
31 - name string
32 - labels []Label
33 - labelsKey string
34 - desc *instrumentDescriptor
35 - bounds []float64
36 - count SampleValue
37 - sum SampleValue
38 - cumulative []SampleValue
32 + key string
33 + name string
34 + hostScopeKey string
35 + hostScope HostScope
36 + labels []Label
37 + labelsKey string
38 + desc *instrumentDescriptor
39 + bounds []float64
40 + count SampleValue
41 + sum SampleValue
42 + cumulative []SampleValue
43 }
44
45 // Histogram declares or reuses a snapshot histogram under this meter.
@@ -47,6 +51,7 @@ func (m *snapshotMeter) Histogram(name string, opts ...InstrumentOption) Snapsho
51 return &snapshotHistogramInstrument{
52 backend: m.backend,
53 desc: desc,
54 + scope: m.scope,
55 base: appendLabelSets(m.sets, nil),
56 }
57 }
@@ -60,22 +65,23 @@ func (m *statefulMeter) Histogram(name string, opts ...InstrumentOption) Statefu
65 return &statefulHistogramInstrument{
66 backend: m.backend,
67 desc: desc,
68 + scope: m.scope,
69 base: appendLabelSets(m.sets, nil),
70 }
71 }
72
73 // ObservePoint writes one full histogram point for this collect cycle.
74 func (h *snapshotHistogramInstrument) ObservePoint(p HistogramPoint, labels ...LabelSet) {
69 - h.backend.recordHistogramObservePoint(h.desc, p, appendLabelSets(h.base, labels))
75 + h.backend.recordHistogramObservePoint(h.desc, h.scope, p, appendLabelSets(h.base, labels))
76 }
77
78 // Observe adds one sample to a stateful histogram for this collect cycle.
79 func (h *statefulHistogramInstrument) Observe(v SampleValue, labels ...LabelSet) {
74 - h.backend.recordHistogramObserve(h.desc, v, appendLabelSets(h.base, labels))
80 + h.backend.recordHistogramObserve(h.desc, h.scope, v, appendLabelSets(h.base, labels))
81 }
82
83 // recordHistogramObservePoint writes one full histogram point into the active frame.
78 -func (c *storeCore) recordHistogramObservePoint(desc *instrumentDescriptor, point HistogramPoint, sets []LabelSet) {
84 +func (c *storeCore) recordHistogramObservePoint(desc *instrumentDescriptor, scope HostScope, point HistogramPoint, sets []LabelSet) {
85 c.mu.Lock()
86 defer c.mu.Unlock()
87
@@ -90,6 +96,10 @@ func (c *storeCore) recordHistogramObservePoint(desc *instrumentDescriptor, poin
96 if labelsContainKey(labels, HistogramBucketLabel) {
97 panic(errHistogramLabelKey)
98 }
99 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
100 + if !ok {
101 + return
102 + }
103
104 schema := desc.histogram
105 if schema == nil {
@@ -99,15 +109,17 @@ func (c *storeCore) recordHistogramObservePoint(desc *instrumentDescriptor, poin
109 }
110 bounds, count, sum, cumulative := normalizeHistogramPoint(point, schema)
111
102 - key := makeSeriesKey(desc.name, labelsKey)
112 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
113 entry, ok := c.active.histograms[key]
114 if !ok {
115 entry = &stagedHistogram{
106 - key: key,
107 - name: desc.name,
108 - labels: labels,
109 - labelsKey: labelsKey,
110 - desc: desc,
116 + key: key,
117 + name: desc.name,
118 + hostScopeKey: scope.ScopeKey,
119 + hostScope: scope,
120 + labels: labels,
121 + labelsKey: labelsKey,
122 + desc: desc,
123 }
124 c.active.histograms[key] = entry
125 }
@@ -121,7 +133,7 @@ func (c *storeCore) recordHistogramObservePoint(desc *instrumentDescriptor, poin
133 }
134
135 // recordHistogramObserve adds one sample to a stateful histogram in the active frame.
124 -func (c *storeCore) recordHistogramObserve(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
136 +func (c *storeCore) recordHistogramObserve(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
137 mustFiniteSample(value)
138
139 c.mu.Lock()
@@ -143,18 +155,24 @@ func (c *storeCore) recordHistogramObserve(desc *instrumentDescriptor, value Sam
155 if labelsContainKey(labels, HistogramBucketLabel) {
156 panic(errHistogramLabelKey)
157 }
158 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
159 + if !ok {
160 + return
161 + }
162
147 - key := makeSeriesKey(desc.name, labelsKey)
163 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
164 entry, ok := c.active.histograms[key]
165 if !ok {
166 entry = &stagedHistogram{
151 - key: key,
152 - name: desc.name,
153 - labels: labels,
154 - labelsKey: labelsKey,
155 - desc: desc,
156 - bounds: append([]float64(nil), schema.bounds...),
157 - cumulative: make([]SampleValue, len(schema.bounds)),
167 + key: key,
168 + name: desc.name,
169 + hostScopeKey: scope.ScopeKey,
170 + hostScope: scope,
171 + labels: labels,
172 + labelsKey: labelsKey,
173 + desc: desc,
174 + bounds: append([]float64(nil), schema.bounds...),
175 + cumulative: make([]SampleValue, len(schema.bounds)),
176 }
177 if desc.window == WindowCumulative {
178 if existing := c.snapshot.Load().series[key]; existing != nil && existing.desc != nil && existing.desc.kind == kindHistogram {
src/go/pkg/metrix/host_scope.go new
+107
@@ -0,0 +1,107 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package metrix
4 +
5 +import (
6 + "fmt"
7 + "maps"
8 + "sort"
9 + "strings"
10 +)
11 +
12 +// HostScope identifies the target host/vnode partition for metric series.
13 +//
14 +// The zero value is the default scope and is equivalent to unscoped writes.
15 +// Non-default scopes must have a stable ScopeKey plus host metadata.
16 +type HostScope struct {
17 + ScopeKey string
18 + GUID string
19 + Hostname string
20 + Labels map[string]string
21 +}
22 +
23 +// IsDefault reports whether this scope is the default unscoped partition.
24 +func (s HostScope) IsDefault() bool {
25 + return strings.TrimSpace(s.ScopeKey) == ""
26 +}
27 +
28 +func normalizeHostScope(scope HostScope) (HostScope, error) {
29 + out := HostScope{
30 + ScopeKey: strings.TrimSpace(scope.ScopeKey),
31 + GUID: strings.TrimSpace(scope.GUID),
32 + Hostname: strings.TrimSpace(scope.Hostname),
33 + }
34 + if strings.ContainsAny(out.ScopeKey, "\xfe\xff") {
35 + return HostScope{}, fmt.Errorf("metrix: host scope key contains reserved separator")
36 + }
37 + if out.ScopeKey == "" {
38 + if out.GUID != "" || out.Hostname != "" || len(scope.Labels) > 0 {
39 + return HostScope{}, fmt.Errorf("metrix: default host scope cannot carry vnode metadata")
40 + }
41 + return HostScope{}, nil
42 + }
43 + if out.GUID == "" {
44 + return HostScope{}, fmt.Errorf("metrix: host scope guid is required")
45 + }
46 + if out.Hostname == "" {
47 + return HostScope{}, fmt.Errorf("metrix: host scope hostname is required")
48 + }
49 + if len(scope.Labels) > 0 {
50 + out.Labels = make(map[string]string, len(scope.Labels))
51 + for key, value := range scope.Labels {
52 + k := strings.TrimSpace(key)
53 + if k == "" {
54 + return HostScope{}, fmt.Errorf("metrix: host scope label key is required")
55 + }
56 + if _, ok := out.Labels[k]; ok {
57 + return HostScope{}, fmt.Errorf("metrix: duplicate host scope label key %q", k)
58 + }
59 + out.Labels[k] = strings.TrimSpace(value)
60 + }
61 + }
62 + return out, nil
63 +}
64 +
65 +func mustNormalizeHostScope(scope HostScope) HostScope {
66 + out, err := normalizeHostScope(scope)
67 + if err != nil {
68 + panic(err)
69 + }
70 + return out
71 +}
72 +
73 +func cloneHostScope(scope HostScope) HostScope {
74 + if scope.Labels != nil {
75 + scope.Labels = maps.Clone(scope.Labels)
76 + }
77 + return scope
78 +}
79 +
80 +func hostScopeEqual(a, b HostScope) bool {
81 + if a.ScopeKey != b.ScopeKey || a.GUID != b.GUID || a.Hostname != b.Hostname {
82 + return false
83 + }
84 + return maps.Equal(a.Labels, b.Labels)
85 +}
86 +
87 +func sortedHostScopes(scopes map[string]HostScope) []HostScope {
88 + if len(scopes) == 0 {
89 + return nil
90 + }
91 + keys := make([]string, 0, len(scopes))
92 + for key := range scopes {
93 + keys = append(keys, key)
94 + }
95 + sort.Strings(keys)
96 + out := make([]HostScope, 0, len(keys))
97 + if scope, ok := scopes[""]; ok {
98 + out = append(out, cloneHostScope(scope))
99 + }
100 + for _, key := range keys {
101 + if key == "" {
102 + continue
103 + }
104 + out = append(out, cloneHostScope(scopes[key]))
105 + }
106 + return out
107 +}
src/go/pkg/metrix/host_scope_test.go new
+397
@@ -0,0 +1,397 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package metrix
4 +
5 +import (
6 + "errors"
7 + "sync"
8 + "testing"
9 +
10 + "github.com/stretchr/testify/require"
11 +)
12 +
13 +func TestHostScopePartitionsSeriesIdentity(t *testing.T) {
14 + store := NewCollectorStore()
15 + cc := cycleController(t, store)
16 +
17 + scope := HostScope{
18 + ScopeKey: "workload/api",
19 + GUID: "guid-api",
20 + Hostname: "api",
21 + Labels: map[string]string{"_vnode_type": "azure_workload"},
22 + }
23 +
24 + meter := store.Write().SnapshotMeter("azure")
25 + defaultGauge := meter.Gauge("requests")
26 + scopedGauge := meter.WithHostScope(scope).Gauge("requests")
27 + labels := meter.LabelSet(Label{Key: "resource", Value: "vm1"})
28 +
29 + cc.BeginCycle()
30 + defaultGauge.Observe(1, labels)
31 + scopedGauge.Observe(2, labels)
32 + require.NoError(t, cc.CommitCycleSuccess())
33 +
34 + mustValue(t, store.Read(), "azure.requests", Labels{"resource": "vm1"}, 1)
35 + mustValue(t, store.Read(ReadHostScope(scope.ScopeKey)), "azure.requests", Labels{"resource": "vm1"}, 2)
36 +
37 + _, ok := store.Read().Value("azure.requests", Labels{"resource": "vm2"})
38 + require.False(t, ok)
39 + _, ok = store.Read(ReadHostScope(scope.ScopeKey)).Value("azure.requests", Labels{"resource": "vm2"})
40 + require.False(t, ok)
41 +
42 + scopes := store.Read().HostScopes()
43 + require.Len(t, scopes, 2)
44 + require.True(t, scopes[0].IsDefault())
45 + require.Equal(t, scope, scopes[1])
46 +
47 + filteredScopes := store.Read(ReadHostScope(scope.ScopeKey)).HostScopes()
48 + require.Equal(t, scopes, filteredScopes)
49 +
50 + var defaultID, scopedID SeriesID
51 + store.Read().ForEachSeriesIdentity(func(identity SeriesIdentity, _ SeriesMeta, name string, _ LabelView, _ SampleValue) {
52 + if name == "azure.requests" {
53 + defaultID = identity.ID
54 + }
55 + })
56 + store.Read(ReadHostScope(scope.ScopeKey)).ForEachSeriesIdentity(func(identity SeriesIdentity, _ SeriesMeta, name string, _ LabelView, _ SampleValue) {
57 + if name == "azure.requests" {
58 + scopedID = identity.ID
59 + }
60 + })
61 + require.NotEmpty(t, defaultID)
62 + require.NotEmpty(t, scopedID)
63 + require.NotEqual(t, defaultID, scopedID)
64 +}
65 +
66 +func TestHostScopeMetadataRefreshesRetainedSeries(t *testing.T) {
67 + store := NewCollectorStore()
68 + cc := cycleController(t, store)
69 +
70 + scopeV1 := HostScope{
71 + ScopeKey: "workload/api",
72 + GUID: "guid-api",
73 + Hostname: "api",
74 + Labels: map[string]string{"_vnode_type": "azure_workload"},
75 + }
76 + scopeV2 := HostScope{
77 + ScopeKey: "workload/api",
78 + GUID: "guid-api",
79 + Hostname: "api-v2",
80 + Labels: map[string]string{"_vnode_type": "azure_workload", "region": "eastus"},
81 + }
82 + meter := store.Write().SnapshotMeter("azure")
83 + labels := meter.LabelSet(Label{Key: "resource", Value: "vm1"})
84 +
85 + cc.BeginCycle()
86 + meter.WithHostScope(scopeV1).Gauge("requests").Observe(1, labels)
87 + meter.WithHostScope(scopeV1).Gauge("errors").Observe(2, labels)
88 + require.NoError(t, cc.CommitCycleSuccess())
89 +
90 + cc.BeginCycle()
91 + meter.WithHostScope(scopeV2).Gauge("requests").Observe(3, labels)
92 + require.NoError(t, cc.CommitCycleSuccess())
93 +
94 + require.Equal(t, []HostScope{scopeV2}, store.Read().HostScopes())
95 +
96 + snapshot := store.(*storeView).core.snapshot.Load()
97 + seen := 0
98 + for _, series := range snapshot.series {
99 + if series.hostScopeKey != scopeV2.ScopeKey {
100 + continue
101 + }
102 + seen++
103 + require.Equal(t, scopeV2, series.hostScope)
104 + }
105 + require.Equal(t, 2, seen)
106 +}
107 +
108 +func TestHostScopeVecDerivationPartitionsSeries(t *testing.T) {
109 + store := NewCollectorStore()
110 + cc := cycleController(t, store)
111 +
112 + scopeA := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
113 + scopeB := HostScope{ScopeKey: "workload/db", GUID: "guid-db", Hostname: "db"}
114 + vec := store.Write().SnapshotMeter("azure").Vec("resource").Gauge("cpu")
115 +
116 + cc.BeginCycle()
117 + vec.WithLabelValues("vm1").Observe(10)
118 + vec.WithHostScope(scopeA).WithLabelValues("vm1").Observe(20)
119 + vec.WithHostScope(scopeB).WithLabelValues("vm1").Observe(30)
120 + require.NoError(t, cc.CommitCycleSuccess())
121 +
122 + mustValue(t, store.Read(), "azure.cpu", Labels{"resource": "vm1"}, 10)
123 + mustValue(t, store.Read(ReadHostScope(scopeA.ScopeKey)), "azure.cpu", Labels{"resource": "vm1"}, 20)
124 + mustValue(t, store.Read(ReadHostScope(scopeB.ScopeKey)), "azure.cpu", Labels{"resource": "vm1"}, 30)
125 +
126 + missingScopeReader := store.Read(ReadHostScope("workload/missing"))
127 + _, ok := missingScopeReader.Value("azure.cpu", Labels{"resource": "vm1"})
128 + require.False(t, ok)
129 + _, ok = missingScopeReader.Family("azure.cpu")
130 + require.False(t, ok)
131 +}
132 +
133 +func TestHostScopeFlattenPreservesScopeScenarios(t *testing.T) {
134 + cases := map[string]struct {
135 + run func(t *testing.T)
136 + }{
137 + "histogram": {
138 + run: func(t *testing.T) {
139 + store := NewCollectorStore()
140 + cc := cycleController(t, store)
141 +
142 + scope := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
143 + hist := store.Write().SnapshotMeter("svc").WithHostScope(scope).Histogram("latency")
144 + labels := store.Write().SnapshotMeter("").LabelSet(Label{Key: "route", Value: "/api"})
145 +
146 + cc.BeginCycle()
147 + hist.ObservePoint(HistogramPoint{
148 + Count: 3,
149 + Sum: 6,
150 + Buckets: []BucketPoint{
151 + {UpperBound: 1, CumulativeCount: 1},
152 + {UpperBound: 5, CumulativeCount: 3},
153 + },
154 + }, labels)
155 + require.NoError(t, cc.CommitCycleSuccess())
156 +
157 + defaultReader := store.Read(ReadFlatten())
158 + _, ok := defaultReader.Value("svc.latency_count", Labels{"route": "/api"})
159 + require.False(t, ok)
160 +
161 + scopedReader := store.Read(ReadFlatten(), ReadHostScope(scope.ScopeKey))
162 + mustValue(t, scopedReader, "svc.latency_count", Labels{"route": "/api"}, 3)
163 + mustValue(t, scopedReader, "svc.latency_sum", Labels{"route": "/api"}, 6)
164 + mustValue(t, scopedReader, "svc.latency_bucket", Labels{"route": "/api", HistogramBucketLabel: "1"}, 1)
165 + mustValue(t, scopedReader, "svc.latency_bucket", Labels{"route": "/api", HistogramBucketLabel: "5"}, 3)
166 + },
167 + },
168 + "composite instruments": {
169 + run: func(t *testing.T) {
170 + store := NewCollectorStore()
171 + cc := cycleController(t, store)
172 +
173 + scope := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
174 + meter := store.Write().SnapshotMeter("svc")
175 + labels := meter.LabelSet(Label{Key: "route", Value: "/api"})
176 + summary := meter.WithHostScope(scope).Summary("latency", WithSummaryQuantiles(0.5))
177 + stateSet := meter.WithHostScope(scope).StateSet("mode", WithStateSetStates("idle", "busy"))
178 + measureSet := meter.WithHostScope(scope).MeasureSetGauge(
179 + "usage",
180 + WithMeasureSetFields(
181 + MeasureFieldSpec{Name: "used"},
182 + MeasureFieldSpec{Name: "limit"},
183 + ),
184 + )
185 +
186 + cc.BeginCycle()
187 + summary.ObservePoint(SummaryPoint{
188 + Count: 2,
189 + Sum: 1,
190 + Quantiles: []QuantilePoint{
191 + {Quantile: 0.5, Value: 0.4},
192 + },
193 + }, labels)
194 + stateSet.ObserveStateSet(StateSetPoint{States: map[string]bool{"busy": true}}, labels)
195 + measureSet.ObserveFields(map[string]SampleValue{"used": 7, "limit": 10}, labels)
196 + require.NoError(t, cc.CommitCycleSuccess())
197 +
198 + defaultReader := store.Read(ReadFlatten())
199 + _, ok := defaultReader.Value("svc.latency_count", Labels{"route": "/api"})
200 + require.False(t, ok)
201 + _, ok = defaultReader.Value("svc.mode", Labels{"route": "/api", "svc.mode": "busy"})
202 + require.False(t, ok)
203 + _, ok = defaultReader.Value("svc.usage_used", Labels{"route": "/api", MeasureSetFieldLabel: "used"})
204 + require.False(t, ok)
205 +
206 + scopedReader := store.Read(ReadFlatten(), ReadHostScope(scope.ScopeKey))
207 + mustValue(t, scopedReader, "svc.latency_count", Labels{"route": "/api"}, 2)
208 + mustValue(t, scopedReader, "svc.latency_sum", Labels{"route": "/api"}, 1)
209 + mustValue(t, scopedReader, "svc.latency", Labels{"route": "/api", SummaryQuantileLabel: "0.5"}, 0.4)
210 + mustValue(t, scopedReader, "svc.mode", Labels{"route": "/api", "svc.mode": "idle"}, 0)
211 + mustValue(t, scopedReader, "svc.mode", Labels{"route": "/api", "svc.mode": "busy"}, 1)
212 + mustValue(t, scopedReader, "svc.usage_used", Labels{"route": "/api", MeasureSetFieldLabel: "used"}, 7)
213 + mustValue(t, scopedReader, "svc.usage_limit", Labels{"route": "/api", MeasureSetFieldLabel: "limit"}, 10)
214 + },
215 + },
216 + }
217 +
218 + for name, tc := range cases {
219 + t.Run(name, tc.run)
220 + }
221 +}
222 +
223 +func TestHostScopeStatefulBaselinesArePerScope(t *testing.T) {
224 + store := NewCollectorStore()
225 + cc := cycleController(t, store)
226 +
227 + scopeA := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
228 + scopeB := HostScope{ScopeKey: "workload/db", GUID: "guid-db", Hostname: "db"}
229 + meter := store.Write().StatefulMeter("svc")
230 + gaugeA := meter.WithHostScope(scopeA).Gauge("load")
231 + gaugeB := meter.WithHostScope(scopeB).Gauge("load")
232 + counterA := meter.WithHostScope(scopeA).Counter("requests")
233 + counterB := meter.WithHostScope(scopeB).Counter("requests")
234 + labels := meter.LabelSet(Label{Key: "resource", Value: "vm1"})
235 +
236 + cc.BeginCycle()
237 + gaugeA.Add(1, labels)
238 + gaugeB.Add(10, labels)
239 + counterA.Add(1, labels)
240 + counterB.Add(10, labels)
241 + require.NoError(t, cc.CommitCycleSuccess())
242 +
243 + cc.BeginCycle()
244 + gaugeA.Add(2, labels)
245 + gaugeB.Add(3, labels)
246 + counterA.Add(2, labels)
247 + counterB.Add(3, labels)
248 + require.NoError(t, cc.CommitCycleSuccess())
249 +
250 + mustValue(t, store.Read(ReadHostScope(scopeA.ScopeKey)), "svc.load", Labels{"resource": "vm1"}, 3)
251 + mustValue(t, store.Read(ReadHostScope(scopeB.ScopeKey)), "svc.load", Labels{"resource": "vm1"}, 13)
252 + mustValue(t, store.Read(ReadHostScope(scopeA.ScopeKey)), "svc.requests", Labels{"resource": "vm1"}, 3)
253 + mustValue(t, store.Read(ReadHostScope(scopeB.ScopeKey)), "svc.requests", Labels{"resource": "vm1"}, 13)
254 + mustDelta(t, store.Read(ReadHostScope(scopeA.ScopeKey)), "svc.requests", Labels{"resource": "vm1"}, 2)
255 + mustDelta(t, store.Read(ReadHostScope(scopeB.ScopeKey)), "svc.requests", Labels{"resource": "vm1"}, 3)
256 +
257 + _, ok := store.Read().Value("svc.load", Labels{"resource": "vm1"})
258 + require.False(t, ok)
259 +}
260 +
261 +func TestHostScopeStatefulHistogramCumulativeWindowIsPerScope(t *testing.T) {
262 + store := NewCollectorStore()
263 + cc := cycleController(t, store)
264 +
265 + scopeA := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
266 + scopeB := HostScope{ScopeKey: "workload/db", GUID: "guid-db", Hostname: "db"}
267 + meter := store.Write().StatefulMeter("svc")
268 + histA := meter.WithHostScope(scopeA).Histogram("latency", WithHistogramBounds(1, 2))
269 + histB := meter.WithHostScope(scopeB).Histogram("latency", WithHistogramBounds(1, 2))
270 + labels := meter.LabelSet(Label{Key: "resource", Value: "vm1"})
271 +
272 + cc.BeginCycle()
273 + histA.Observe(0.5, labels)
274 + histA.Observe(1.5, labels)
275 + histB.Observe(3, labels)
276 + require.NoError(t, cc.CommitCycleSuccess())
277 +
278 + cc.BeginCycle()
279 + histA.Observe(0.2, labels)
280 + histB.Observe(0.8, labels)
281 + require.NoError(t, cc.CommitCycleSuccess())
282 +
283 + mustHistogram(t, store.Read(ReadHostScope(scopeA.ScopeKey)), "svc.latency", Labels{"resource": "vm1"}, HistogramPoint{
284 + Count: 3,
285 + Sum: 2.2,
286 + Buckets: []BucketPoint{
287 + {UpperBound: 1, CumulativeCount: 2},
288 + {UpperBound: 2, CumulativeCount: 3},
289 + },
290 + })
291 + mustHistogram(t, store.Read(ReadHostScope(scopeB.ScopeKey)), "svc.latency", Labels{"resource": "vm1"}, HistogramPoint{
292 + Count: 2,
293 + Sum: 3.8,
294 + Buckets: []BucketPoint{
295 + {UpperBound: 1, CumulativeCount: 1},
296 + {UpperBound: 2, CumulativeCount: 1},
297 + },
298 + })
299 +}
300 +
301 +func TestHostScopeConflictScenarios(t *testing.T) {
302 + cases := map[string]struct {
303 + run func(t *testing.T)
304 + }{
305 + "commit fails without publishing": {
306 + run: func(t *testing.T) {
307 + store := NewCollectorStore()
308 + cc := cycleController(t, store)
309 +
310 + scopeA := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
311 + scopeB := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api-v2"}
312 + meter := store.Write().SnapshotMeter("azure")
313 + a := meter.WithHostScope(scopeA).Gauge("requests")
314 + b := meter.WithHostScope(scopeB).Gauge("requests")
315 +
316 + cc.BeginCycle()
317 + a.Observe(1)
318 + b.Observe(2)
319 + err := cc.CommitCycleSuccess()
320 + require.Error(t, err)
321 + require.True(t, errors.Is(err, ErrHostScopeConflict), "unexpected error: %v", err)
322 +
323 + meta := store.Read().CollectMeta()
324 + require.Equal(t, CollectStatusFailed, meta.LastAttemptStatus)
325 + require.Equal(t, uint64(1), meta.LastAttemptSeq)
326 + require.Equal(t, uint64(0), meta.LastSuccessSeq)
327 +
328 + _, ok := store.Read(ReadRaw(), ReadHostScope(scopeA.ScopeKey)).Value("azure.requests", nil)
329 + require.False(t, ok)
330 + },
331 + },
332 + "multiple conflicts join errors": {
333 + run: func(t *testing.T) {
334 + store := NewCollectorStore()
335 + cc := cycleController(t, store)
336 +
337 + scopeA := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"}
338 + scopeAConflict := HostScope{ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api-v2"}
339 + scopeB := HostScope{ScopeKey: "workload/db", GUID: "guid-db", Hostname: "db"}
340 + scopeBConflict := HostScope{ScopeKey: "workload/db", GUID: "guid-db-v2", Hostname: "db"}
341 + meter := store.Write().SnapshotMeter("azure")
342 +
343 + cc.BeginCycle()
344 + meter.WithHostScope(scopeA).Gauge("requests").Observe(1)
345 + meter.WithHostScope(scopeAConflict).Gauge("requests").Observe(2)
346 + meter.WithHostScope(scopeB).Gauge("requests").Observe(3)
347 + meter.WithHostScope(scopeBConflict).Gauge("requests").Observe(4)
348 + err := cc.CommitCycleSuccess()
349 + require.Error(t, err)
350 + require.True(t, errors.Is(err, ErrHostScopeConflict), "unexpected error: %v", err)
351 + require.Contains(t, err.Error(), "scope_key=\"workload/api\"")
352 + require.Contains(t, err.Error(), "scope_key=\"workload/db\"")
353 + },
354 + },
355 + }
356 +
357 + for name, tc := range cases {
358 + t.Run(name, tc.run)
359 + }
360 +}
361 +
362 +func TestHostScopeConcurrentWrites(t *testing.T) {
363 + store := NewCollectorStore()
364 + cc := cycleController(t, store)
365 +
366 + scopes := []HostScope{
367 + {ScopeKey: "workload/api", GUID: "guid-api", Hostname: "api"},
368 + {ScopeKey: "workload/db", GUID: "guid-db", Hostname: "db"},
369 + {ScopeKey: "workload/cache", GUID: "guid-cache", Hostname: "cache"},
370 + }
371 + meter := store.Write().StatefulMeter("svc")
372 + counters := make([]StatefulCounter, len(scopes))
373 + for i, scope := range scopes {
374 + counters[i] = meter.WithHostScope(scope).Counter("requests")
375 + }
376 +
377 + const writersPerScope = 8
378 + const writesPerWriter = 100
379 +
380 + var wg sync.WaitGroup
381 + cc.BeginCycle()
382 + for i := range scopes {
383 + for range writersPerScope {
384 + wg.Go(func() {
385 + for range writesPerWriter {
386 + counters[i].Add(1)
387 + }
388 + })
389 + }
390 + }
391 + wg.Wait()
392 + require.NoError(t, cc.CommitCycleSuccess())
393 +
394 + for _, scope := range scopes {
395 + mustValue(t, store.Read(ReadHostScope(scope.ScopeKey)), "svc.requests", nil, writersPerScope*writesPerWriter)
396 + }
397 +}
src/go/pkg/metrix/interfaces.go
+23 -1
@@ -20,7 +20,7 @@ type RuntimeStore interface {
20 // Collector code does not call these methods directly.
21 type CycleController interface {
22 BeginCycle()
23 - CommitCycleSuccess()
23 + CommitCycleSuccess() error
24 AbortCycle()
25 }
26
@@ -45,6 +45,10 @@ type Reader interface {
45 // Example: histogram families resolve via *_bucket/*_count/*_sum names.
46 MetricMeta(name string) (MetricMeta, bool)
47 CollectMeta() CollectMeta
48 + // HostScopes returns all host scopes present in the snapshot, including the
49 + // default scope when it has series. The result is not filtered by the
50 + // reader's active scope.
51 + HostScopes() []HostScope
52 // Family returns a scalar-only view. For non-scalar families use Histogram/Summary/StateSet,
53 // or use Read(ReadFlatten()) at reader acquisition time.
54 Family(name string) (FamilyView, bool)
@@ -86,6 +90,7 @@ type RuntimeWriter interface {
90
91 // SnapshotMeter declares snapshot-mode instruments under a metric-name prefix.
92 type SnapshotMeter interface {
93 + WithHostScope(scope HostScope) SnapshotMeter
94 WithLabels(labels ...Label) SnapshotMeter
95 WithLabelSet(labels ...LabelSet) SnapshotMeter
96 // Vec binds a reusable vec label-key schema for multiple vector instruments.
@@ -102,6 +107,7 @@ type SnapshotMeter interface {
107
108 // SnapshotVecMeter declares snapshot vec instruments sharing one label-key schema.
109 type SnapshotVecMeter interface {
110 + WithHostScope(scope HostScope) SnapshotVecMeter
111 Gauge(name string, opts ...InstrumentOption) SnapshotGaugeVec
112 Counter(name string, opts ...InstrumentOption) SnapshotCounterVec
113 Histogram(name string, opts ...InstrumentOption) SnapshotHistogramVec
@@ -113,6 +119,7 @@ type SnapshotVecMeter interface {
119
120 // StatefulMeter declares stateful-mode instruments under a metric-name prefix.
121 type StatefulMeter interface {
122 + WithHostScope(scope HostScope) StatefulMeter
123 WithLabels(labels ...Label) StatefulMeter
124 WithLabelSet(labels ...LabelSet) StatefulMeter
125 // Vec binds a reusable vec label-key schema for multiple vector instruments.
@@ -129,6 +136,7 @@ type StatefulMeter interface {
136
137 // StatefulVecMeter declares stateful vec instruments sharing one label-key schema.
138 type StatefulVecMeter interface {
139 + WithHostScope(scope HostScope) StatefulVecMeter
140 Gauge(name string, opts ...InstrumentOption) StatefulGaugeVec
141 Counter(name string, opts ...InstrumentOption) StatefulCounterVec
142 Histogram(name string, opts ...InstrumentOption) StatefulHistogramVec
@@ -148,6 +156,7 @@ type SnapshotGauge interface {
156 // - GetWithLabelValues returns (metric, error)
157 // - WithLabelValues panics on invalid label values
158 type SnapshotGaugeVec interface {
159 + WithHostScope(scope HostScope) SnapshotGaugeVec
160 GetWithLabelValues(labelValues ...string) (SnapshotGauge, error)
161 WithLabelValues(labelValues ...string) SnapshotGauge
162 }
@@ -161,6 +170,7 @@ type StatefulGauge interface {
170
171 // StatefulGaugeVec provides labeled series handles for stateful gauges.
172 type StatefulGaugeVec interface {
173 + WithHostScope(scope HostScope) StatefulGaugeVec
174 GetWithLabelValues(labelValues ...string) (StatefulGauge, error)
175 WithLabelValues(labelValues ...string) StatefulGauge
176 }
@@ -171,6 +181,7 @@ type SnapshotCounter interface {
181
182 // SnapshotCounterVec provides labeled series handles for snapshot counters.
183 type SnapshotCounterVec interface {
184 + WithHostScope(scope HostScope) SnapshotCounterVec
185 GetWithLabelValues(labelValues ...string) (SnapshotCounter, error)
186 WithLabelValues(labelValues ...string) SnapshotCounter
187 }
@@ -181,6 +192,7 @@ type StatefulCounter interface {
192
193 // StatefulCounterVec provides labeled series handles for stateful counters.
194 type StatefulCounterVec interface {
195 + WithHostScope(scope HostScope) StatefulCounterVec
196 GetWithLabelValues(labelValues ...string) (StatefulCounter, error)
197 WithLabelValues(labelValues ...string) StatefulCounter
198 }
@@ -191,6 +203,7 @@ type SnapshotHistogram interface {
203
204 // SnapshotHistogramVec provides labeled series handles for snapshot histograms.
205 type SnapshotHistogramVec interface {
206 + WithHostScope(scope HostScope) SnapshotHistogramVec
207 GetWithLabelValues(labelValues ...string) (SnapshotHistogram, error)
208 WithLabelValues(labelValues ...string) SnapshotHistogram
209 }
@@ -201,6 +214,7 @@ type StatefulHistogram interface {
214
215 // StatefulHistogramVec provides labeled series handles for stateful histograms.
216 type StatefulHistogramVec interface {
217 + WithHostScope(scope HostScope) StatefulHistogramVec
218 GetWithLabelValues(labelValues ...string) (StatefulHistogram, error)
219 WithLabelValues(labelValues ...string) StatefulHistogram
220 }
@@ -211,6 +225,7 @@ type SnapshotSummary interface {
225
226 // SnapshotSummaryVec provides labeled series handles for snapshot summaries.
227 type SnapshotSummaryVec interface {
228 + WithHostScope(scope HostScope) SnapshotSummaryVec
229 GetWithLabelValues(labelValues ...string) (SnapshotSummary, error)
230 WithLabelValues(labelValues ...string) SnapshotSummary
231 }
@@ -221,6 +236,7 @@ type StatefulSummary interface {
236
237 // StatefulSummaryVec provides labeled series handles for stateful summaries.
238 type StatefulSummaryVec interface {
239 + WithHostScope(scope HostScope) StatefulSummaryVec
240 GetWithLabelValues(labelValues ...string) (StatefulSummary, error)
241 WithLabelValues(labelValues ...string) StatefulSummary
242 }
@@ -236,6 +252,7 @@ type SnapshotMeasureSetGauge interface {
252 }
253
254 type SnapshotMeasureSetGaugeVec interface {
255 + WithHostScope(scope HostScope) SnapshotMeasureSetGaugeVec
256 GetWithLabelValues(labelValues ...string) (SnapshotMeasureSetGauge, error)
257 WithLabelValues(labelValues ...string) SnapshotMeasureSetGauge
258 }
@@ -246,6 +263,7 @@ type SnapshotMeasureSetCounter interface {
263 }
264
265 type SnapshotMeasureSetCounterVec interface {
266 + WithHostScope(scope HostScope) SnapshotMeasureSetCounterVec
267 GetWithLabelValues(labelValues ...string) (SnapshotMeasureSetCounter, error)
268 WithLabelValues(labelValues ...string) SnapshotMeasureSetCounter
269 }
@@ -260,6 +278,7 @@ type StatefulMeasureSetGauge interface {
278 }
279
280 type StatefulMeasureSetGaugeVec interface {
281 + WithHostScope(scope HostScope) StatefulMeasureSetGaugeVec
282 GetWithLabelValues(labelValues ...string) (StatefulMeasureSetGauge, error)
283 WithLabelValues(labelValues ...string) StatefulMeasureSetGauge
284 }
@@ -271,18 +290,21 @@ type StatefulMeasureSetCounter interface {
290 }
291
292 type StatefulMeasureSetCounterVec interface {
293 + WithHostScope(scope HostScope) StatefulMeasureSetCounterVec
294 GetWithLabelValues(labelValues ...string) (StatefulMeasureSetCounter, error)
295 WithLabelValues(labelValues ...string) StatefulMeasureSetCounter
296 }
297
298 // SnapshotStateSetVec provides labeled series handles for snapshot statesets.
299 type SnapshotStateSetVec interface {
300 + WithHostScope(scope HostScope) SnapshotStateSetVec
301 GetWithLabelValues(labelValues ...string) (StateSetInstrument, error)
302 WithLabelValues(labelValues ...string) StateSetInstrument
303 }
304
305 // StatefulStateSetVec provides labeled series handles for stateful statesets.
306 type StatefulStateSetVec interface {
307 + WithHostScope(scope HostScope) StatefulStateSetVec
308 GetWithLabelValues(labelValues ...string) (StateSetInstrument, error)
309 WithLabelValues(labelValues ...string) StateSetInstrument
310 }
src/go/pkg/metrix/measureset.go
+92 -52
@@ -6,18 +6,21 @@ const MeasureSetFieldLabel = "measure_field"
6
7 // stagedMeasureSet holds one in-cycle MeasureSet sample for a single series identity.
8 type stagedMeasureSet struct {
9 - key string
10 - name string
11 - labels []Label
12 - labelsKey string
13 - desc *instrumentDescriptor
14 - values []SampleValue
9 + key string
10 + name string
11 + hostScopeKey string
12 + hostScope HostScope
13 + labels []Label
14 + labelsKey string
15 + desc *instrumentDescriptor
16 + values []SampleValue
17 }
18
19 // snapshotMeasureSetGaugeInstrument writes sampled MeasureSet gauge points.
20 type snapshotMeasureSetGaugeInstrument struct {
21 backend meterBackend
22 desc *instrumentDescriptor
23 + scope HostScope
24 base []LabelSet
25 }
26
@@ -25,6 +28,7 @@ type snapshotMeasureSetGaugeInstrument struct {
28 type snapshotMeasureSetCounterInstrument struct {
29 backend meterBackend
30 desc *instrumentDescriptor
31 + scope HostScope
32 base []LabelSet
33 }
34
@@ -32,6 +36,7 @@ type snapshotMeasureSetCounterInstrument struct {
36 type statefulMeasureSetGaugeInstrument struct {
37 backend meterBackend
38 desc *instrumentDescriptor
39 + scope HostScope
40 base []LabelSet
41 }
42
@@ -39,6 +44,7 @@ type statefulMeasureSetGaugeInstrument struct {
44 type statefulMeasureSetCounterInstrument struct {
45 backend meterBackend
46 desc *instrumentDescriptor
47 + scope HostScope
48 base []LabelSet
49 }
50
@@ -58,6 +64,7 @@ func (m *snapshotMeter) MeasureSetGauge(name string, opts ...InstrumentOption) S
64 return &snapshotMeasureSetGaugeInstrument{
65 backend: m.backend,
66 desc: desc,
67 + scope: m.scope,
68 base: appendLabelSets(m.sets, nil),
69 }
70 }
@@ -71,6 +78,7 @@ func (m *snapshotMeter) MeasureSetCounter(name string, opts ...InstrumentOption)
78 return &snapshotMeasureSetCounterInstrument{
79 backend: m.backend,
80 desc: desc,
81 + scope: m.scope,
82 base: appendLabelSets(m.sets, nil),
83 }
84 }
@@ -84,6 +92,7 @@ func (m *statefulMeter) MeasureSetGauge(name string, opts ...InstrumentOption) S
92 return &statefulMeasureSetGaugeInstrument{
93 backend: m.backend,
94 desc: desc,
95 + scope: m.scope,
96 base: appendLabelSets(m.sets, nil),
97 }
98 }
@@ -97,12 +106,13 @@ func (m *statefulMeter) MeasureSetCounter(name string, opts ...InstrumentOption)
106 return &statefulMeasureSetCounterInstrument{
107 backend: m.backend,
108 desc: desc,
109 + scope: m.scope,
110 base: appendLabelSets(m.sets, nil),
111 }
112 }
113
114 func (m *snapshotMeasureSetGaugeInstrument) ObservePoint(p MeasureSetPoint, labels ...LabelSet) {
105 - m.backend.recordMeasureSetGaugeObservePoint(m.desc, p, appendLabelSets(m.base, labels))
115 + m.backend.recordMeasureSetGaugeObservePoint(m.desc, m.scope, p, appendLabelSets(m.base, labels))
116 }
117
118 func (m *snapshotMeasureSetGaugeInstrument) ObserveFields(fields map[string]SampleValue, labels ...LabelSet) {
@@ -110,7 +120,7 @@ func (m *snapshotMeasureSetGaugeInstrument) ObserveFields(fields map[string]Samp
120 }
121
122 func (m *snapshotMeasureSetCounterInstrument) ObserveTotalPoint(p MeasureSetPoint, labels ...LabelSet) {
113 - m.backend.recordMeasureSetCounterObserveTotalPoint(m.desc, p, appendLabelSets(m.base, labels))
123 + m.backend.recordMeasureSetCounterObserveTotalPoint(m.desc, m.scope, p, appendLabelSets(m.base, labels))
124 }
125
126 func (m *snapshotMeasureSetCounterInstrument) ObserveTotalFields(fields map[string]SampleValue, labels ...LabelSet) {
@@ -118,7 +128,7 @@ func (m *snapshotMeasureSetCounterInstrument) ObserveTotalFields(fields map[stri
128 }
129
130 func (m *statefulMeasureSetGaugeInstrument) SetPoint(p MeasureSetPoint, labels ...LabelSet) {
121 - m.backend.recordMeasureSetGaugeSetPoint(m.desc, p, appendLabelSets(m.base, labels))
131 + m.backend.recordMeasureSetGaugeSetPoint(m.desc, m.scope, p, appendLabelSets(m.base, labels))
132 }
133
134 func (m *statefulMeasureSetGaugeInstrument) SetFields(fields map[string]SampleValue, labels ...LabelSet) {
@@ -126,11 +136,11 @@ func (m *statefulMeasureSetGaugeInstrument) SetFields(fields map[string]SampleVa
136 }
137
138 func (m *statefulMeasureSetGaugeInstrument) SetField(field string, value SampleValue, labels ...LabelSet) {
129 - m.backend.recordMeasureSetGaugeSetField(m.desc, field, value, appendLabelSets(m.base, labels))
139 + m.backend.recordMeasureSetGaugeSetField(m.desc, m.scope, field, value, appendLabelSets(m.base, labels))
140 }
141
142 func (m *statefulMeasureSetGaugeInstrument) AddPoint(delta MeasureSetPoint, labels ...LabelSet) {
133 - m.backend.recordMeasureSetGaugeAddPoint(m.desc, delta, appendLabelSets(m.base, labels))
143 + m.backend.recordMeasureSetGaugeAddPoint(m.desc, m.scope, delta, appendLabelSets(m.base, labels))
144 }
145
146 func (m *statefulMeasureSetGaugeInstrument) AddFields(delta map[string]SampleValue, labels ...LabelSet) {
@@ -142,7 +152,7 @@ func (m *statefulMeasureSetGaugeInstrument) AddField(field string, delta SampleV
152 }
153
154 func (m *statefulMeasureSetCounterInstrument) AddPoint(delta MeasureSetPoint, labels ...LabelSet) {
145 - m.backend.recordMeasureSetCounterAddPoint(m.desc, delta, appendLabelSets(m.base, labels))
155 + m.backend.recordMeasureSetCounterAddPoint(m.desc, m.scope, delta, appendLabelSets(m.base, labels))
156 }
157
158 func (m *statefulMeasureSetCounterInstrument) AddFields(delta map[string]SampleValue, labels ...LabelSet) {
@@ -222,11 +232,11 @@ func normalizeMeasureSetCounterDelta(delta MeasureSetPoint, schema *measureSetSc
232 return values
233 }
234
225 -func (c *storeCore) recordMeasureSetGaugeObservePoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet) {
226 - c.recordMeasureSetGaugeSetPoint(desc, point, sets)
235 +func (c *storeCore) recordMeasureSetGaugeObservePoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet) {
236 + c.recordMeasureSetGaugeSetPoint(desc, scope, point, sets)
237 }
238
229 -func (c *storeCore) recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet) {
239 +func (c *storeCore) recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet) {
240 schema := desc.measureSet
241 if schema == nil || schema.semantics != MeasureSetSemanticsGauge {
242 panic(errMeasureSetSchema)
@@ -248,23 +258,29 @@ func (c *storeCore) recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, po
258 if labelsContainKey(labels, MeasureSetFieldLabel) {
259 panic(errMeasureSetLabelKey)
260 }
261 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
262 + if !ok {
263 + return
264 + }
265
252 - key := makeSeriesKey(desc.name, labelsKey)
266 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
267 entry, ok := c.active.measureSetGauges[key]
268 if !ok {
269 entry = &stagedMeasureSet{
256 - key: key,
257 - name: desc.name,
258 - labels: labels,
259 - labelsKey: labelsKey,
260 - desc: desc,
270 + key: key,
271 + name: desc.name,
272 + hostScopeKey: scope.ScopeKey,
273 + hostScope: scope,
274 + labels: labels,
275 + labelsKey: labelsKey,
276 + desc: desc,
277 }
278 c.active.measureSetGauges[key] = entry
279 }
280 entry.values = append(entry.values[:0], values...)
281 }
282
267 -func (c *storeCore) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, delta MeasureSetPoint, sets []LabelSet) {
283 +func (c *storeCore) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, scope HostScope, delta MeasureSetPoint, sets []LabelSet) {
284 schema := desc.measureSet
285 if schema == nil || schema.semantics != MeasureSetSemanticsGauge {
286 panic(errMeasureSetSchema)
@@ -286,8 +302,12 @@ func (c *storeCore) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, de
302 if labelsContainKey(labels, MeasureSetFieldLabel) {
303 panic(errMeasureSetLabelKey)
304 }
305 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
306 + if !ok {
307 + return
308 + }
309
290 - key := makeSeriesKey(desc.name, labelsKey)
310 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
311 entry, ok := c.active.measureSetGauges[key]
312 if !ok {
313 baseline := make([]SampleValue, len(schema.fields))
@@ -298,12 +318,14 @@ func (c *storeCore) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, de
318 }
319 }
320 entry = &stagedMeasureSet{
301 - key: key,
302 - name: desc.name,
303 - labels: labels,
304 - labelsKey: labelsKey,
305 - desc: desc,
306 - values: baseline,
321 + key: key,
322 + name: desc.name,
323 + hostScopeKey: scope.ScopeKey,
324 + hostScope: scope,
325 + labels: labels,
326 + labelsKey: labelsKey,
327 + desc: desc,
328 + values: baseline,
329 }
330 c.active.measureSetGauges[key] = entry
331 }
@@ -312,7 +334,7 @@ func (c *storeCore) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, de
334 }
335 }
336
315 -func (c *storeCore) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, field string, value SampleValue, sets []LabelSet) {
337 +func (c *storeCore) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, scope HostScope, field string, value SampleValue, sets []LabelSet) {
338 schema := desc.measureSet
339 if schema == nil || schema.semantics != MeasureSetSemanticsGauge {
340 panic(errMeasureSetSchema)
@@ -335,8 +357,12 @@ func (c *storeCore) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, fi
357 if labelsContainKey(labels, MeasureSetFieldLabel) {
358 panic(errMeasureSetLabelKey)
359 }
360 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
361 + if !ok {
362 + return
363 + }
364
339 - key := makeSeriesKey(desc.name, labelsKey)
365 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
366 entry, ok := c.active.measureSetGauges[key]
367 if !ok {
368 baseline := make([]SampleValue, len(schema.fields))
@@ -347,12 +373,14 @@ func (c *storeCore) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, fi
373 }
374 }
375 entry = &stagedMeasureSet{
350 - key: key,
351 - name: desc.name,
352 - labels: labels,
353 - labelsKey: labelsKey,
354 - desc: desc,
355 - values: baseline,
376 + key: key,
377 + name: desc.name,
378 + hostScopeKey: scope.ScopeKey,
379 + hostScope: scope,
380 + labels: labels,
381 + labelsKey: labelsKey,
382 + desc: desc,
383 + values: baseline,
384 }
385 c.active.measureSetGauges[key] = entry
386 } else if len(entry.values) == 0 {
@@ -361,7 +389,7 @@ func (c *storeCore) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, fi
389 entry.values[fieldIndex] = value
390 }
391
364 -func (c *storeCore) recordMeasureSetCounterObserveTotalPoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet) {
392 +func (c *storeCore) recordMeasureSetCounterObserveTotalPoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet) {
393 schema := desc.measureSet
394 if schema == nil || schema.semantics != MeasureSetSemanticsCounter {
395 panic(errMeasureSetSchema)
@@ -383,23 +411,29 @@ func (c *storeCore) recordMeasureSetCounterObserveTotalPoint(desc *instrumentDes
411 if labelsContainKey(labels, MeasureSetFieldLabel) {
412 panic(errMeasureSetLabelKey)
413 }
414 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
415 + if !ok {
416 + return
417 + }
418
387 - key := makeSeriesKey(desc.name, labelsKey)
419 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
420 entry, ok := c.active.measureSetCounters[key]
421 if !ok {
422 entry = &stagedMeasureSet{
391 - key: key,
392 - name: desc.name,
393 - labels: labels,
394 - labelsKey: labelsKey,
395 - desc: desc,
423 + key: key,
424 + name: desc.name,
425 + hostScopeKey: scope.ScopeKey,
426 + hostScope: scope,
427 + labels: labels,
428 + labelsKey: labelsKey,
429 + desc: desc,
430 }
431 c.active.measureSetCounters[key] = entry
432 }
433 entry.values = append(entry.values[:0], values...)
434 }
435
402 -func (c *storeCore) recordMeasureSetCounterAddPoint(desc *instrumentDescriptor, delta MeasureSetPoint, sets []LabelSet) {
436 +func (c *storeCore) recordMeasureSetCounterAddPoint(desc *instrumentDescriptor, scope HostScope, delta MeasureSetPoint, sets []LabelSet) {
437 schema := desc.measureSet
438 if schema == nil || schema.semantics != MeasureSetSemanticsCounter {
439 panic(errMeasureSetSchema)
@@ -421,8 +455,12 @@ func (c *storeCore) recordMeasureSetCounterAddPoint(desc *instrumentDescriptor,
455 if labelsContainKey(labels, MeasureSetFieldLabel) {
456 panic(errMeasureSetLabelKey)
457 }
458 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
459 + if !ok {
460 + return
461 + }
462
425 - key := makeSeriesKey(desc.name, labelsKey)
463 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
464 entry, ok := c.active.measureSetCounters[key]
465 if !ok {
466 baseline := make([]SampleValue, len(schema.fields))
@@ -433,12 +471,14 @@ func (c *storeCore) recordMeasureSetCounterAddPoint(desc *instrumentDescriptor,
471 }
472 }
473 entry = &stagedMeasureSet{
436 - key: key,
437 - name: desc.name,
438 - labels: labels,
439 - labelsKey: labelsKey,
440 - desc: desc,
441 - values: baseline,
474 + key: key,
475 + name: desc.name,
476 + hostScopeKey: scope.ScopeKey,
477 + hostScope: scope,
478 + labels: labels,
479 + labelsKey: labelsKey,
480 + desc: desc,
481 + values: baseline,
482 }
483 c.active.measureSetCounters[key] = entry
484 }
src/go/pkg/metrix/meter.go
+36 -2
@@ -9,6 +9,7 @@ type writeView struct {
9 type snapshotMeter struct {
10 backend meterBackend
11 prefix string
12 + scope HostScope
13 sets []LabelSet
14 }
15
@@ -20,6 +21,7 @@ type snapshotVecMeter struct {
21 type statefulMeter struct {
22 backend meterBackend
23 prefix string
24 + scope HostScope
25 sets []LabelSet
26 }
27
@@ -43,13 +45,22 @@ func (m *snapshotMeter) WithLabels(labels ...Label) SnapshotMeter {
45 return m.WithLabelSet(set)
46 }
47
48 +func (m *snapshotMeter) WithHostScope(scope HostScope) SnapshotMeter {
49 + return &snapshotMeter{
50 + backend: m.backend,
51 + prefix: m.prefix,
52 + scope: mustNormalizeHostScope(scope),
53 + sets: appendLabelSets(m.sets, nil),
54 + }
55 +}
56 +
57 func (m *snapshotMeter) WithLabelSet(labels ...LabelSet) SnapshotMeter {
58 for _, ls := range labels {
59 if ls.set == nil || ls.set.owner != m.backend {
60 panic(errForeignLabelSet)
61 }
62 }
52 - return &snapshotMeter{backend: m.backend, prefix: m.prefix, sets: appendLabelSets(m.sets, labels)}
63 + return &snapshotMeter{backend: m.backend, prefix: m.prefix, scope: m.scope, sets: appendLabelSets(m.sets, labels)}
64 }
65
66 func (m *snapshotMeter) Vec(labelKeys ...string) SnapshotVecMeter {
@@ -68,13 +79,22 @@ func (m *statefulMeter) WithLabels(labels ...Label) StatefulMeter {
79 return m.WithLabelSet(set)
80 }
81
82 +func (m *statefulMeter) WithHostScope(scope HostScope) StatefulMeter {
83 + return &statefulMeter{
84 + backend: m.backend,
85 + prefix: m.prefix,
86 + scope: mustNormalizeHostScope(scope),
87 + sets: appendLabelSets(m.sets, nil),
88 + }
89 +}
90 +
91 func (m *statefulMeter) WithLabelSet(labels ...LabelSet) StatefulMeter {
92 for _, ls := range labels {
93 if ls.set == nil || ls.set.owner != m.backend {
94 panic(errForeignLabelSet)
95 }
96 }
77 - return &statefulMeter{backend: m.backend, prefix: m.prefix, sets: appendLabelSets(m.sets, labels)}
97 + return &statefulMeter{backend: m.backend, prefix: m.prefix, scope: m.scope, sets: appendLabelSets(m.sets, labels)}
98 }
99
100 func (m *statefulMeter) Vec(labelKeys ...string) StatefulVecMeter {
@@ -92,6 +112,13 @@ func (m *snapshotVecMeter) Gauge(name string, opts ...InstrumentOption) Snapshot
112 return m.meter.GaugeVec(name, m.labelKeys, opts...)
113 }
114
115 +func (m *snapshotVecMeter) WithHostScope(scope HostScope) SnapshotVecMeter {
116 + return &snapshotVecMeter{
117 + meter: m.meter.WithHostScope(scope).(*snapshotMeter),
118 + labelKeys: append([]string(nil), m.labelKeys...),
119 + }
120 +}
121 +
122 func (m *snapshotVecMeter) Counter(name string, opts ...InstrumentOption) SnapshotCounterVec {
123 return m.meter.CounterVec(name, m.labelKeys, opts...)
124 }
@@ -120,6 +147,13 @@ func (m *statefulVecMeter) Gauge(name string, opts ...InstrumentOption) Stateful
147 return m.meter.GaugeVec(name, m.labelKeys, opts...)
148 }
149
150 +func (m *statefulVecMeter) WithHostScope(scope HostScope) StatefulVecMeter {
151 + return &statefulVecMeter{
152 + meter: m.meter.WithHostScope(scope).(*statefulMeter),
153 + labelKeys: append([]string(nil), m.labelKeys...),
154 + }
155 +}
156 +
157 func (m *statefulVecMeter) Counter(name string, opts ...InstrumentOption) StatefulCounterVec {
158 return m.meter.CounterVec(name, m.labelKeys, opts...)
159 }
src/go/pkg/metrix/read_options.go
+11 -2
@@ -14,8 +14,9 @@ func (f readOptionFunc) applyRead(cfg *readConfig) {
14 }
15
16 type readConfig struct {
17 - raw bool
18 - flatten bool
17 + raw bool
18 + flatten bool
19 + hostScopeKey string
20 }
21
22 func resolveReadConfig(opts ...ReadOption) readConfig {
@@ -43,3 +44,11 @@ func ReadFlatten() ReadOption {
44 cfg.flatten = true
45 })
46 }
47 +
48 +// ReadHostScope filters the reader to one host scope. The empty key is the
49 +// default scope and matches unscoped writes.
50 +func ReadHostScope(scopeKey string) ReadOption {
51 + return readOptionFunc(func(cfg *readConfig) {
52 + cfg.hostScopeKey = scopeKey
53 + })
54 +}
src/go/pkg/metrix/reader.go
+85 -54
@@ -5,18 +5,20 @@ package metrix
5 import (
6 "maps"
7 "math"
8 + "slices"
9 "sort"
10 "sync"
11 )
12
13 type storeReader struct {
13 - snap *readSnapshot
14 - raw bool // true => ReadRaw semantics (no freshness filtering)
15 - flattened bool
16 - seriesOnce sync.Once
17 - series map[string]*committedSeries
18 - indexOnce sync.Once
19 - index map[string][]*committedSeries
14 + snap *readSnapshot
15 + raw bool // true => ReadRaw semantics (no freshness filtering)
16 + flattened bool
17 + hostScopeKey string
18 + seriesOnce sync.Once
19 + series map[string]*committedSeries
20 + indexOnce sync.Once
21 + index map[string][]*committedSeries
22 }
23
24 type familyView struct {
@@ -188,6 +190,16 @@ func (r *storeReader) CollectMeta() CollectMeta {
190 return r.snap.collectMeta
191 }
192
193 +func (r *storeReader) HostScopes() []HostScope {
194 + // Scope discovery intentionally ignores this reader's active scope filter so
195 + // jobruntime can enumerate all host partitions from one snapshot.
196 + scopes := make(map[string]HostScope)
197 + for _, s := range r.seriesView() {
198 + scopes[s.hostScopeKey] = cloneHostScope(s.hostScope)
199 + }
200 + return sortedHostScopes(scopes)
201 +}
202 +
203 func flattenSnapshot(src *readSnapshot) *readSnapshot {
204 series := snapshotSeriesView(src)
205 dst := &readSnapshot{
@@ -242,14 +254,16 @@ func appendFlattenedHistogramSeries(dst *readSnapshot, src *committedSeries) {
254 }
255
256 name := src.name + "_bucket"
245 - key := makeSeriesKey(name, labelsKey)
257 + key := makeSeriesKey(src.hostScopeKey, name, labelsKey)
258 dst.series[key] = &committedSeries{
247 - id: SeriesID(key),
248 - hash64: seriesIDHash(SeriesID(key)),
249 - key: key,
250 - name: name,
251 - labels: labels,
252 - labelsKey: labelsKey,
259 + id: SeriesID(key),
260 + hash64: seriesIDHash(SeriesID(key)),
261 + key: key,
262 + name: name,
263 + hostScopeKey: src.hostScopeKey,
264 + hostScope: cloneHostScope(src.hostScope),
265 + labels: labels,
266 + labelsKey: labelsKey,
267 desc: &instrumentDescriptor{
268 name: name,
269 kind: kindCounter,
@@ -276,14 +290,16 @@ func appendFlattenedHistogramSeries(dst *readSnapshot, src *committedSeries) {
290 infLabels, infLabelsKey, err := canonicalizeLabels(infMap)
291 if err == nil {
292 infName := src.name + "_bucket"
279 - infKey := makeSeriesKey(infName, infLabelsKey)
293 + infKey := makeSeriesKey(src.hostScopeKey, infName, infLabelsKey)
294 dst.series[infKey] = &committedSeries{
281 - id: SeriesID(infKey),
282 - hash64: seriesIDHash(SeriesID(infKey)),
283 - key: infKey,
284 - name: infName,
285 - labels: infLabels,
286 - labelsKey: infLabelsKey,
295 + id: SeriesID(infKey),
296 + hash64: seriesIDHash(SeriesID(infKey)),
297 + key: infKey,
298 + name: infName,
299 + hostScopeKey: src.hostScopeKey,
300 + hostScope: cloneHostScope(src.hostScope),
301 + labels: infLabels,
302 + labelsKey: infLabelsKey,
303 desc: &instrumentDescriptor{
304 name: infName,
305 kind: kindCounter,
@@ -304,6 +320,7 @@ func appendFlattenedHistogramSeries(dst *readSnapshot, src *committedSeries) {
320
321 appendFlattenedHistogramScalar(
322 dst,
323 + src,
324 src.name+"_count",
325 src.labels,
326 src.histogramCount,
@@ -312,6 +329,7 @@ func appendFlattenedHistogramSeries(dst *readSnapshot, src *committedSeries) {
329 )
330 appendFlattenedHistogramScalar(
331 dst,
332 + src,
333 src.name+"_sum",
334 src.labels,
335 src.histogramSum,
@@ -320,7 +338,7 @@ func appendFlattenedHistogramSeries(dst *readSnapshot, src *committedSeries) {
338 )
339 }
340
323 -func appendFlattenedHistogramScalar(dst *readSnapshot, name string, labels []Label, value SampleValue, meta SeriesMeta, desc *instrumentDescriptor) {
341 +func appendFlattenedHistogramScalar(dst *readSnapshot, src *committedSeries, name string, labels []Label, value SampleValue, meta SeriesMeta, desc *instrumentDescriptor) {
342 labelsMap := make(map[string]string, len(labels))
343 for _, lbl := range labels {
344 labelsMap[lbl.Key] = lbl.Value
@@ -329,14 +347,16 @@ func appendFlattenedHistogramScalar(dst *readSnapshot, name string, labels []Lab
347 if err != nil {
348 return
349 }
332 - key := makeSeriesKey(name, labelsKey)
350 + key := makeSeriesKey(src.hostScopeKey, name, labelsKey)
351 dst.series[key] = &committedSeries{
334 - id: SeriesID(key),
335 - hash64: seriesIDHash(SeriesID(key)),
336 - key: key,
337 - name: name,
338 - labels: items,
339 - labelsKey: labelsKey,
352 + id: SeriesID(key),
353 + hash64: seriesIDHash(SeriesID(key)),
354 + key: key,
355 + name: name,
356 + hostScopeKey: src.hostScopeKey,
357 + hostScope: cloneHostScope(src.hostScope),
358 + labels: items,
359 + labelsKey: labelsKey,
360 desc: &instrumentDescriptor{
361 name: name,
362 kind: kindCounter,
@@ -353,6 +373,7 @@ func appendFlattenedHistogramScalar(dst *readSnapshot, name string, labels []Lab
373 func appendFlattenedSummarySeries(dst *readSnapshot, src *committedSeries) {
374 appendFlattenedHistogramScalar(
375 dst,
376 + src,
377 src.name+"_count",
378 src.labels,
379 src.summaryCount,
@@ -361,6 +382,7 @@ func appendFlattenedSummarySeries(dst *readSnapshot, src *committedSeries) {
382 )
383 appendFlattenedHistogramScalar(
384 dst,
385 + src,
386 src.name+"_sum",
387 src.labels,
388 src.summarySum,
@@ -387,14 +409,16 @@ func appendFlattenedSummarySeries(dst *readSnapshot, src *committedSeries) {
409 if err != nil {
410 continue
411 }
390 - key := makeSeriesKey(src.name, labelsKey)
412 + key := makeSeriesKey(src.hostScopeKey, src.name, labelsKey)
413 dst.series[key] = &committedSeries{
392 - id: SeriesID(key),
393 - hash64: seriesIDHash(SeriesID(key)),
394 - key: key,
395 - name: src.name,
396 - labels: labels,
397 - labelsKey: labelsKey,
414 + id: SeriesID(key),
415 + hash64: seriesIDHash(SeriesID(key)),
416 + key: key,
417 + name: src.name,
418 + hostScopeKey: src.hostScopeKey,
419 + hostScope: cloneHostScope(src.hostScope),
420 + labels: labels,
421 + labelsKey: labelsKey,
422 desc: &instrumentDescriptor{
423 name: src.name,
424 kind: kindGauge,
@@ -437,14 +461,16 @@ func appendFlattenedStateSetSeries(dst *readSnapshot, src *committedSeries) {
461 continue
462 }
463
440 - key := makeSeriesKey(src.name, labelsKey)
464 + key := makeSeriesKey(src.hostScopeKey, src.name, labelsKey)
465 dst.series[key] = &committedSeries{
442 - id: SeriesID(key),
443 - hash64: seriesIDHash(SeriesID(key)),
444 - key: key,
445 - name: src.name,
446 - labels: labels,
447 - labelsKey: labelsKey,
466 + id: SeriesID(key),
467 + hash64: seriesIDHash(SeriesID(key)),
468 + key: key,
469 + name: src.name,
470 + hostScopeKey: src.hostScopeKey,
471 + hostScope: cloneHostScope(src.hostScope),
472 + labels: labels,
473 + labelsKey: labelsKey,
474 desc: &instrumentDescriptor{
475 name: src.name,
476 kind: kindGauge,
@@ -490,17 +516,19 @@ func appendFlattenedMeasureSetSeries(dst *readSnapshot, src *committedSeries) {
516 }
517
518 name := src.name + "_" + field.Name
493 - key := makeSeriesKey(name, labelsKey)
519 + key := makeSeriesKey(src.hostScopeKey, name, labelsKey)
520 meta := src.desc.meta
521 meta.Float = field.Float
522
523 series := &committedSeries{
498 - id: SeriesID(key),
499 - hash64: seriesIDHash(SeriesID(key)),
500 - key: key,
501 - name: name,
502 - labels: labels,
503 - labelsKey: labelsKey,
524 + id: SeriesID(key),
525 + hash64: seriesIDHash(SeriesID(key)),
526 + key: key,
527 + name: name,
528 + hostScopeKey: src.hostScopeKey,
529 + hostScope: cloneHostScope(src.hostScope),
530 + labels: labels,
531 + labelsKey: labelsKey,
532 desc: &instrumentDescriptor{
533 name: name,
534 kind: descKind,
@@ -532,10 +560,10 @@ func appendFlattenedMeasureSetSeries(dst *readSnapshot, src *committedSeries) {
560
561 func (r *storeReader) Family(name string) (FamilyView, bool) {
562 index := r.byNameIndex()
535 - if len(index[name]) == 0 {
536 - return nil, false
563 + if slices.ContainsFunc(index[name], r.visible) {
564 + return familyView{name: name, reader: r}, true
565 }
538 - return familyView{name: name, reader: r}, true
566 + return nil, false
567 }
568
569 func (r *storeReader) ForEachByName(name string, fn func(labels LabelView, v SampleValue)) {
@@ -609,7 +637,7 @@ func (r *storeReader) lookup(name string, labels Labels) (*committedSeries, bool
637 _ = items
638 return nil, false
639 }
612 - key := makeSeriesKey(name, labelsKey)
640 + key := makeSeriesKey(r.hostScopeKey, name, labelsKey)
641 return lookupSnapshotSeries(r.snap, key)
642 }
643
@@ -667,6 +695,9 @@ func materializeRuntimeSeries(snap *readSnapshot) map[string]*committedSeries {
695
696 // visible applies freshness policy for Read(); Read(ReadRaw()) bypasses it.
697 func (r *storeReader) visible(s *committedSeries) bool {
698 + if s.hostScopeKey != r.hostScopeKey {
699 + return false
700 + }
701 if r.raw {
702 return true
703 }
src/go/pkg/metrix/runtime_store.go
+57 -45
@@ -75,7 +75,7 @@ func (s *runtimeStoreView) Read(opts ...ReadOption) Reader {
75 if cfg.flatten {
76 snap = flattenSnapshot(snap)
77 }
78 - return &storeReader{snap: snap, raw: cfg.raw, flattened: cfg.flatten}
78 + return &storeReader{snap: snap, raw: cfg.raw, flattened: cfg.flatten, hostScopeKey: cfg.hostScopeKey}
79 }
80
81 func (s *runtimeStoreView) Write() RuntimeWriter {
@@ -118,43 +118,45 @@ func (r *runtimeStoreBackend) registerInstrument(name string, kind metricKind, m
118 return desc, nil
119 }
120
121 -func (r *runtimeStoreBackend) recordGaugeSet(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
121 +func (r *runtimeStoreBackend) recordGaugeSet(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
122 mustFiniteSample(value)
123
124 labels, labelsKey, err := labelsFromSet(sets, r)
125 if err != nil {
126 panic(err)
127 }
128 - key := makeSeriesKey(desc.name, labelsKey)
128 + scope = mustNormalizeHostScope(scope)
129 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
130 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
130 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
131 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
132 series.value = value
133 series.meta.LastSeenSuccessSeq = seq
134 series.runtimeLastSeenUnixNano = nowUnixNano
135 })
136 }
137
137 -func (r *runtimeStoreBackend) recordGaugeAdd(desc *instrumentDescriptor, delta SampleValue, sets []LabelSet) {
138 +func (r *runtimeStoreBackend) recordGaugeAdd(desc *instrumentDescriptor, scope HostScope, delta SampleValue, sets []LabelSet) {
139 mustFiniteSample(delta)
140
141 labels, labelsKey, err := labelsFromSet(sets, r)
142 if err != nil {
143 panic(err)
144 }
144 - key := makeSeriesKey(desc.name, labelsKey)
145 + scope = mustNormalizeHostScope(scope)
146 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
147 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
146 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
148 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
149 series.value += delta
150 series.meta.LastSeenSuccessSeq = seq
151 series.runtimeLastSeenUnixNano = nowUnixNano
152 })
153 }
154
153 -func (r *runtimeStoreBackend) recordCounterObserveTotal(_ *instrumentDescriptor, _ SampleValue, _ []LabelSet) {
155 +func (r *runtimeStoreBackend) recordCounterObserveTotal(_ *instrumentDescriptor, _ HostScope, _ SampleValue, _ []LabelSet) {
156 panic(errRuntimeSnapshotWrite)
157 }
158
157 -func (r *runtimeStoreBackend) recordCounterAdd(desc *instrumentDescriptor, delta SampleValue, sets []LabelSet) {
159 +func (r *runtimeStoreBackend) recordCounterAdd(desc *instrumentDescriptor, scope HostScope, delta SampleValue, sets []LabelSet) {
160 mustFiniteSample(delta)
161
162 if delta < 0 {
@@ -165,9 +167,10 @@ func (r *runtimeStoreBackend) recordCounterAdd(desc *instrumentDescriptor, delta
167 if err != nil {
168 panic(err)
169 }
168 - key := makeSeriesKey(desc.name, labelsKey)
170 + scope = mustNormalizeHostScope(scope)
171 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
172 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
170 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
173 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
174
175 hadCurrent := series.desc != nil && series.desc.kind == kindCounter && series.counterCurrentSeq > 0
176 if hadCurrent {
@@ -189,11 +192,11 @@ func (r *runtimeStoreBackend) recordCounterAdd(desc *instrumentDescriptor, delta
192 })
193 }
194
192 -func (r *runtimeStoreBackend) recordHistogramObservePoint(_ *instrumentDescriptor, _ HistogramPoint, _ []LabelSet) {
195 +func (r *runtimeStoreBackend) recordHistogramObservePoint(_ *instrumentDescriptor, _ HostScope, _ HistogramPoint, _ []LabelSet) {
196 panic(errRuntimeSnapshotWrite)
197 }
198
196 -func (r *runtimeStoreBackend) recordHistogramObserve(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
199 +func (r *runtimeStoreBackend) recordHistogramObserve(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
200 mustFiniteSample(value)
201
202 schema := desc.histogram
@@ -208,10 +211,11 @@ func (r *runtimeStoreBackend) recordHistogramObserve(desc *instrumentDescriptor,
211 if labelsContainKey(labels, HistogramBucketLabel) {
212 panic(errHistogramLabelKey)
213 }
214 + scope = mustNormalizeHostScope(scope)
215
212 - key := makeSeriesKey(desc.name, labelsKey)
216 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
217 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
214 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
218 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
219
220 if series.desc.histogram == nil || !equalHistogramBounds(series.desc.histogram.bounds, schema.bounds) {
221 panic("metrix: histogram schema drift detected")
@@ -233,11 +237,11 @@ func (r *runtimeStoreBackend) recordHistogramObserve(desc *instrumentDescriptor,
237 })
238 }
239
236 -func (r *runtimeStoreBackend) recordSummaryObservePoint(_ *instrumentDescriptor, _ SummaryPoint, _ []LabelSet) {
240 +func (r *runtimeStoreBackend) recordSummaryObservePoint(_ *instrumentDescriptor, _ HostScope, _ SummaryPoint, _ []LabelSet) {
241 panic(errRuntimeSnapshotWrite)
242 }
243
240 -func (r *runtimeStoreBackend) recordSummaryObserve(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
244 +func (r *runtimeStoreBackend) recordSummaryObserve(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
245 mustFiniteSample(value)
246
247 labels, labelsKey, err := labelsFromSet(sets, r)
@@ -247,10 +251,11 @@ func (r *runtimeStoreBackend) recordSummaryObserve(desc *instrumentDescriptor, v
251 if labelsContainKey(labels, SummaryQuantileLabel) {
252 panic(errSummaryLabelKey)
253 }
254 + scope = mustNormalizeHostScope(scope)
255
251 - key := makeSeriesKey(desc.name, labelsKey)
256 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
257 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
253 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
258 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
259
260 series.summaryCount++
261 series.summarySum += value
@@ -273,7 +278,7 @@ func (r *runtimeStoreBackend) recordSummaryObserve(desc *instrumentDescriptor, v
278 })
279 }
280
276 -func (r *runtimeStoreBackend) recordStateSetObserve(desc *instrumentDescriptor, point StateSetPoint, sets []LabelSet) {
281 +func (r *runtimeStoreBackend) recordStateSetObserve(desc *instrumentDescriptor, scope HostScope, point StateSetPoint, sets []LabelSet) {
282 schema := desc.stateSet
283 if schema == nil {
284 panic(errStateSetSchema)
@@ -286,22 +291,23 @@ func (r *runtimeStoreBackend) recordStateSetObserve(desc *instrumentDescriptor,
291 if labelsContainKey(labels, desc.name) {
292 panic(errStateSetLabelKey)
293 }
294 + scope = mustNormalizeHostScope(scope)
295 states := normalizeStateSetPoint(point, schema)
296
291 - key := makeSeriesKey(desc.name, labelsKey)
297 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
298 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
293 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
299 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
300 series.stateSetValues = cloneStateMap(states)
301 series.meta.LastSeenSuccessSeq = seq
302 series.runtimeLastSeenUnixNano = nowUnixNano
303 })
304 }
305
300 -func (r *runtimeStoreBackend) recordMeasureSetGaugeObservePoint(_ *instrumentDescriptor, _ MeasureSetPoint, _ []LabelSet) {
306 +func (r *runtimeStoreBackend) recordMeasureSetGaugeObservePoint(_ *instrumentDescriptor, _ HostScope, _ MeasureSetPoint, _ []LabelSet) {
307 panic(errRuntimeSnapshotWrite)
308 }
309
304 -func (r *runtimeStoreBackend) recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, point MeasureSetPoint, sets []LabelSet) {
310 +func (r *runtimeStoreBackend) recordMeasureSetGaugeSetPoint(desc *instrumentDescriptor, scope HostScope, point MeasureSetPoint, sets []LabelSet) {
311 schema := desc.measureSet
312 if schema == nil || schema.semantics != MeasureSetSemanticsGauge {
313 panic(errMeasureSetSchema)
@@ -316,16 +322,17 @@ func (r *runtimeStoreBackend) recordMeasureSetGaugeSetPoint(desc *instrumentDesc
322 if labelsContainKey(labels, MeasureSetFieldLabel) {
323 panic(errMeasureSetLabelKey)
324 }
319 - key := makeSeriesKey(desc.name, labelsKey)
325 + scope = mustNormalizeHostScope(scope)
326 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
327 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
321 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
328 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
329 series.measureSetValues = append(series.measureSetValues[:0], values...)
330 series.meta.LastSeenSuccessSeq = seq
331 series.runtimeLastSeenUnixNano = nowUnixNano
332 })
333 }
334
328 -func (r *runtimeStoreBackend) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, delta MeasureSetPoint, sets []LabelSet) {
335 +func (r *runtimeStoreBackend) recordMeasureSetGaugeAddPoint(desc *instrumentDescriptor, scope HostScope, delta MeasureSetPoint, sets []LabelSet) {
336 schema := desc.measureSet
337 if schema == nil || schema.semantics != MeasureSetSemanticsGauge {
338 panic(errMeasureSetSchema)
@@ -340,9 +347,10 @@ func (r *runtimeStoreBackend) recordMeasureSetGaugeAddPoint(desc *instrumentDesc
347 if labelsContainKey(labels, MeasureSetFieldLabel) {
348 panic(errMeasureSetLabelKey)
349 }
343 - key := makeSeriesKey(desc.name, labelsKey)
350 + scope = mustNormalizeHostScope(scope)
351 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
352 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
345 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
353 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
354 if len(series.measureSetValues) == 0 {
355 series.measureSetValues = make([]SampleValue, len(schema.fields))
356 }
@@ -354,7 +362,7 @@ func (r *runtimeStoreBackend) recordMeasureSetGaugeAddPoint(desc *instrumentDesc
362 })
363 }
364
357 -func (r *runtimeStoreBackend) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, field string, value SampleValue, sets []LabelSet) {
365 +func (r *runtimeStoreBackend) recordMeasureSetGaugeSetField(desc *instrumentDescriptor, scope HostScope, field string, value SampleValue, sets []LabelSet) {
366 schema := desc.measureSet
367 if schema == nil || schema.semantics != MeasureSetSemanticsGauge {
368 panic(errMeasureSetSchema)
@@ -370,9 +378,10 @@ func (r *runtimeStoreBackend) recordMeasureSetGaugeSetField(desc *instrumentDesc
378 if labelsContainKey(labels, MeasureSetFieldLabel) {
379 panic(errMeasureSetLabelKey)
380 }
373 - key := makeSeriesKey(desc.name, labelsKey)
381 + scope = mustNormalizeHostScope(scope)
382 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
383 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
375 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
384 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
385 if len(series.measureSetValues) == 0 {
386 series.measureSetValues = make([]SampleValue, len(schema.fields))
387 }
@@ -382,11 +391,11 @@ func (r *runtimeStoreBackend) recordMeasureSetGaugeSetField(desc *instrumentDesc
391 })
392 }
393
385 -func (r *runtimeStoreBackend) recordMeasureSetCounterObserveTotalPoint(_ *instrumentDescriptor, _ MeasureSetPoint, _ []LabelSet) {
394 +func (r *runtimeStoreBackend) recordMeasureSetCounterObserveTotalPoint(_ *instrumentDescriptor, _ HostScope, _ MeasureSetPoint, _ []LabelSet) {
395 panic(errRuntimeSnapshotWrite)
396 }
397
389 -func (r *runtimeStoreBackend) recordMeasureSetCounterAddPoint(desc *instrumentDescriptor, delta MeasureSetPoint, sets []LabelSet) {
398 +func (r *runtimeStoreBackend) recordMeasureSetCounterAddPoint(desc *instrumentDescriptor, scope HostScope, delta MeasureSetPoint, sets []LabelSet) {
399 schema := desc.measureSet
400 if schema == nil || schema.semantics != MeasureSetSemanticsCounter {
401 panic(errMeasureSetSchema)
@@ -401,9 +410,10 @@ func (r *runtimeStoreBackend) recordMeasureSetCounterAddPoint(desc *instrumentDe
410 if labelsContainKey(labels, MeasureSetFieldLabel) {
411 panic(errMeasureSetLabelKey)
412 }
404 - key := makeSeriesKey(desc.name, labelsKey)
413 + scope = mustNormalizeHostScope(scope)
414 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
415 r.commitRuntimeWrite(func(old, next *readSnapshot, seq uint64, nowUnixNano int64) {
406 - series := runtimeEnsureSeriesMutable(old, next, key, desc.name, labels, labelsKey, desc)
416 + series := runtimeEnsureSeriesMutable(old, next, key, desc.name, scope.ScopeKey, scope, labels, labelsKey, desc)
417 if len(series.measureSetValues) == 0 {
418 series.measureSetValues = make([]SampleValue, len(schema.fields))
419 }
@@ -460,7 +470,7 @@ func (r *runtimeStoreBackend) commitRuntimeWrite(apply func(old, next *readSnaps
470 r.core.snapshot.Store(next)
471 }
472
463 -func runtimeEnsureSeriesMutable(old, next *readSnapshot, key, name string, labels []Label, labelsKey string, desc *instrumentDescriptor) *committedSeries {
473 +func runtimeEnsureSeriesMutable(old, next *readSnapshot, key, name, hostScopeKey string, hostScope HostScope, labels []Label, labelsKey string, desc *instrumentDescriptor) *committedSeries {
474 series := next.series[key]
475 if series != nil {
476 ensureSeriesMeta(series.desc, &series.meta)
@@ -473,14 +483,16 @@ func runtimeEnsureSeriesMutable(old, next *readSnapshot, key, name string, label
483 return series
484 }
485 series = &committedSeries{
476 - id: SeriesID(key),
477 - hash64: seriesIDHash(SeriesID(key)),
478 - key: key,
479 - name: name,
480 - labels: append([]Label(nil), labels...),
481 - labelsKey: labelsKey,
482 - desc: desc,
483 - meta: baseSeriesMeta(desc),
486 + id: SeriesID(key),
487 + hash64: seriesIDHash(SeriesID(key)),
488 + key: key,
489 + name: name,
490 + hostScopeKey: hostScopeKey,
491 + hostScope: cloneHostScope(hostScope),
492 + labels: append([]Label(nil), labels...),
493 + labelsKey: labelsKey,
494 + desc: desc,
495 + meta: baseSeriesMeta(desc),
496 }
497 next.series[key] = series
498 return series
src/go/pkg/metrix/stateset.go
+27 -15
@@ -6,6 +6,7 @@ package metrix
6 type snapshotStateSetInstrument struct {
7 backend meterBackend
8 desc *instrumentDescriptor
9 + scope HostScope
10 base []LabelSet
11 }
12
@@ -13,17 +14,20 @@ type snapshotStateSetInstrument struct {
14 type statefulStateSetInstrument struct {
15 backend meterBackend
16 desc *instrumentDescriptor
17 + scope HostScope
18 base []LabelSet
19 }
20
21 // stagedStateSet holds one in-cycle stateset sample for a single series identity.
22 type stagedStateSet struct {
21 - key string
22 - name string
23 - labels []Label
24 - labelsKey string
25 - desc *instrumentDescriptor
26 - states map[string]bool
23 + key string
24 + name string
25 + hostScopeKey string
26 + hostScope HostScope
27 + labels []Label
28 + labelsKey string
29 + desc *instrumentDescriptor
30 + states map[string]bool
31 }
32
33 // StateSet declares or reuses a snapshot stateset under this meter.
@@ -35,6 +39,7 @@ func (m *snapshotMeter) StateSet(name string, opts ...InstrumentOption) StateSet
39 return &snapshotStateSetInstrument{
40 backend: m.backend,
41 desc: desc,
42 + scope: m.scope,
43 base: appendLabelSets(m.sets, nil),
44 }
45 }
@@ -48,13 +53,14 @@ func (m *statefulMeter) StateSet(name string, opts ...InstrumentOption) StateSet
53 return &statefulStateSetInstrument{
54 backend: m.backend,
55 desc: desc,
56 + scope: m.scope,
57 base: appendLabelSets(m.sets, nil),
58 }
59 }
60
61 // ObserveStateSet writes a full-state sample for this collect cycle.
62 func (s *snapshotStateSetInstrument) ObserveStateSet(p StateSetPoint, labels ...LabelSet) {
57 - s.backend.recordStateSetObserve(s.desc, p, appendLabelSets(s.base, labels))
63 + s.backend.recordStateSetObserve(s.desc, s.scope, p, appendLabelSets(s.base, labels))
64 }
65
66 // Enable writes enum/bitset convenience sample with listed active states.
@@ -64,7 +70,7 @@ func (s *snapshotStateSetInstrument) Enable(actives ...string) {
70
71 // ObserveStateSet writes a full-state sample for this collect cycle.
72 func (s *statefulStateSetInstrument) ObserveStateSet(p StateSetPoint, labels ...LabelSet) {
67 - s.backend.recordStateSetObserve(s.desc, p, appendLabelSets(s.base, labels))
73 + s.backend.recordStateSetObserve(s.desc, s.scope, p, appendLabelSets(s.base, labels))
74 }
75
76 // Enable writes enum/bitset convenience sample with listed active states.
@@ -98,7 +104,7 @@ func stateSetPointFromActives(desc *instrumentDescriptor, actives ...string) Sta
104 }
105
106 // recordStateSetObserve writes one full-state stateset sample into the active frame.
101 -func (c *storeCore) recordStateSetObserve(desc *instrumentDescriptor, point StateSetPoint, sets []LabelSet) {
107 +func (c *storeCore) recordStateSetObserve(desc *instrumentDescriptor, scope HostScope, point StateSetPoint, sets []LabelSet) {
108 c.mu.Lock()
109 defer c.mu.Unlock()
110
@@ -118,18 +124,24 @@ func (c *storeCore) recordStateSetObserve(desc *instrumentDescriptor, point Stat
124 if labelsContainKey(labels, desc.name) {
125 panic(errStateSetLabelKey)
126 }
127 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
128 + if !ok {
129 + return
130 + }
131
132 states := normalizeStateSetPoint(point, schema)
133
124 - key := makeSeriesKey(desc.name, labelsKey)
134 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
135 entry, ok := c.active.stateSet[key]
136 if !ok {
137 entry = &stagedStateSet{
128 - key: key,
129 - name: desc.name,
130 - labels: labels,
131 - labelsKey: labelsKey,
132 - desc: desc,
138 + key: key,
139 + name: desc.name,
140 + hostScopeKey: scope.ScopeKey,
141 + hostScope: scope,
142 + labels: labels,
143 + labelsKey: labelsKey,
144 + desc: desc,
145 }
146 c.active.stateSet[key] = entry
147 }
src/go/pkg/metrix/summary.go
+40 -21
@@ -18,6 +18,7 @@ const initialSummaryReservoirCapacity = 64
18 type snapshotSummaryInstrument struct {
19 backend meterBackend
20 desc *instrumentDescriptor
21 + scope HostScope
22 base []LabelSet
23 }
24
@@ -25,16 +26,20 @@ type snapshotSummaryInstrument struct {
26 type statefulSummaryInstrument struct {
27 backend meterBackend
28 desc *instrumentDescriptor
29 + scope HostScope
30 base []LabelSet
31 }
32
33 // stagedSummary holds one in-cycle summary sample for a single series identity.
34 type stagedSummary struct {
33 - key string
34 - name string
35 - labels []Label
36 - labelsKey string
37 - desc *instrumentDescriptor
35 + key string
36 + name string
37 + hostScopeKey string
38 + hostScope HostScope
39 + labels []Label
40 + labelsKey string
41 + desc *instrumentDescriptor
42 +
43 count SampleValue
44 sum SampleValue
45 quantileValues []SampleValue
@@ -50,6 +55,7 @@ func (m *snapshotMeter) Summary(name string, opts ...InstrumentOption) SnapshotS
55 return &snapshotSummaryInstrument{
56 backend: m.backend,
57 desc: desc,
58 + scope: m.scope,
59 base: appendLabelSets(m.sets, nil),
60 }
61 }
@@ -63,22 +69,23 @@ func (m *statefulMeter) Summary(name string, opts ...InstrumentOption) StatefulS
69 return &statefulSummaryInstrument{
70 backend: m.backend,
71 desc: desc,
72 + scope: m.scope,
73 base: appendLabelSets(m.sets, nil),
74 }
75 }
76
77 // ObservePoint writes one full summary point for this collect cycle.
78 func (s *snapshotSummaryInstrument) ObservePoint(p SummaryPoint, labels ...LabelSet) {
72 - s.backend.recordSummaryObservePoint(s.desc, p, appendLabelSets(s.base, labels))
79 + s.backend.recordSummaryObservePoint(s.desc, s.scope, p, appendLabelSets(s.base, labels))
80 }
81
82 // Observe adds one sample to a stateful summary for this collect cycle.
83 func (s *statefulSummaryInstrument) Observe(v SampleValue, labels ...LabelSet) {
77 - s.backend.recordSummaryObserve(s.desc, v, appendLabelSets(s.base, labels))
84 + s.backend.recordSummaryObserve(s.desc, s.scope, v, appendLabelSets(s.base, labels))
85 }
86
87 // recordSummaryObservePoint writes one full summary point into the active frame.
81 -func (c *storeCore) recordSummaryObservePoint(desc *instrumentDescriptor, point SummaryPoint, sets []LabelSet) {
88 +func (c *storeCore) recordSummaryObservePoint(desc *instrumentDescriptor, scope HostScope, point SummaryPoint, sets []LabelSet) {
89 c.mu.Lock()
90 defer c.mu.Unlock()
91
@@ -93,18 +100,24 @@ func (c *storeCore) recordSummaryObservePoint(desc *instrumentDescriptor, point
100 if labelsContainKey(labels, SummaryQuantileLabel) {
101 panic(errSummaryLabelKey)
102 }
103 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
104 + if !ok {
105 + return
106 + }
107
108 count, sum, quantiles := normalizeSummaryPoint(point, desc.summary)
109
99 - key := makeSeriesKey(desc.name, labelsKey)
110 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
111 entry, ok := c.active.summaries[key]
112 if !ok {
113 entry = &stagedSummary{
103 - key: key,
104 - name: desc.name,
105 - labels: labels,
106 - labelsKey: labelsKey,
107 - desc: desc,
114 + key: key,
115 + name: desc.name,
116 + hostScopeKey: scope.ScopeKey,
117 + hostScope: scope,
118 + labels: labels,
119 + labelsKey: labelsKey,
120 + desc: desc,
121 }
122 c.active.summaries[key] = entry
123 }
@@ -116,7 +129,7 @@ func (c *storeCore) recordSummaryObservePoint(desc *instrumentDescriptor, point
129 }
130
131 // recordSummaryObserve adds one sample to a stateful summary in the active frame.
119 -func (c *storeCore) recordSummaryObserve(desc *instrumentDescriptor, value SampleValue, sets []LabelSet) {
132 +func (c *storeCore) recordSummaryObserve(desc *instrumentDescriptor, scope HostScope, value SampleValue, sets []LabelSet) {
133 mustFiniteSample(value)
134
135 c.mu.Lock()
@@ -133,16 +146,22 @@ func (c *storeCore) recordSummaryObserve(desc *instrumentDescriptor, value Sampl
146 if labelsContainKey(labels, SummaryQuantileLabel) {
147 panic(errSummaryLabelKey)
148 }
149 + scope, ok := c.prepareHostScopeForWriteLocked(scope)
150 + if !ok {
151 + return
152 + }
153
137 - key := makeSeriesKey(desc.name, labelsKey)
154 + key := makeSeriesKey(scope.ScopeKey, desc.name, labelsKey)
155 entry, ok := c.active.summaries[key]
156 if !ok {
157 entry = &stagedSummary{
141 - key: key,
142 - name: desc.name,
143 - labels: labels,
144 - labelsKey: labelsKey,
145 - desc: desc,
158 + key: key,
159 + name: desc.name,
160 + hostScopeKey: scope.ScopeKey,
161 + hostScope: scope,
162 + labels: labels,
163 + labelsKey: labelsKey,
164 + desc: desc,
165 }
166 if desc.window == WindowCumulative {
167 if existing := c.snapshot.Load().series[key]; existing != nil && existing.desc != nil && existing.desc.kind == kindSummary {
src/go/pkg/metrix/vec.go
+142 -32
@@ -21,14 +21,14 @@ type vecCache[T any] struct {
21 mu sync.RWMutex
22 cache map[string]T
23
24 - makeHandle func(base []LabelSet, vecSet LabelSet) T
24 + makeHandle func(scope HostScope, base []LabelSet, vecSet LabelSet) T
25 }
26
27 func newVecCache[T any](
28 backend meterBackend,
29 base []LabelSet,
30 keys []string,
31 - makeHandle func(base []LabelSet, vecSet LabelSet) T,
31 + makeHandle func(scope HostScope, base []LabelSet, vecSet LabelSet) T,
32 ) *vecCache[T] {
33 return &vecCache[T]{
34 backend: backend,
@@ -39,13 +39,15 @@ func newVecCache[T any](
39 }
40 }
41
42 -func (v *vecCache[T]) get(labelValues ...string) (T, error) {
42 +func (v *vecCache[T]) get(scope HostScope, labelValues ...string) (T, error) {
43 var zero T
44
45 cacheKey, err := vecSeriesCacheKey(v.keys, labelValues)
46 if err != nil {
47 return zero, err
48 }
49 + scope = mustNormalizeHostScope(scope)
50 + cacheKey = scopedVecCacheKey(scope.ScopeKey, cacheKey)
51
52 v.mu.RLock()
53 inst, ok := v.cache[cacheKey]
@@ -67,7 +69,7 @@ func (v *vecCache[T]) get(labelValues ...string) (T, error) {
69 return zero, err
70 }
71
70 - inst = v.makeHandle(v.base, vecSet)
72 + inst = v.makeHandle(scope, v.base, vecSet)
73 v.cache[cacheKey] = inst
74 return inst, nil
75 }
@@ -95,71 +97,85 @@ func mustNormalizeVecLabelKeys(labelKeys []string) []string {
97 // snapshotGaugeVec caches snapshot gauge series handles by vec label values.
98 type snapshotGaugeVec struct {
99 cache *vecCache[*snapshotGaugeInstrument]
100 + scope HostScope
101 }
102
103 // statefulGaugeVec caches stateful gauge series handles by vec label values.
104 type statefulGaugeVec struct {
105 cache *vecCache[*statefulGaugeInstrument]
106 + scope HostScope
107 }
108
109 // snapshotCounterVec caches snapshot counter series handles by vec label values.
110 type snapshotCounterVec struct {
111 cache *vecCache[*snapshotCounterInstrument]
112 + scope HostScope
113 }
114
115 // statefulCounterVec caches stateful counter series handles by vec label values.
116 type statefulCounterVec struct {
117 cache *vecCache[*statefulCounterInstrument]
118 + scope HostScope
119 }
120
121 // snapshotHistogramVec caches snapshot histogram series handles by vec label values.
122 type snapshotHistogramVec struct {
123 cache *vecCache[*snapshotHistogramInstrument]
124 + scope HostScope
125 }
126
127 // statefulHistogramVec caches stateful histogram series handles by vec label values.
128 type statefulHistogramVec struct {
129 cache *vecCache[*statefulHistogramInstrument]
130 + scope HostScope
131 }
132
133 // snapshotSummaryVec caches snapshot summary series handles by vec label values.
134 type snapshotSummaryVec struct {
135 cache *vecCache[*snapshotSummaryInstrument]
136 + scope HostScope
137 }
138
139 // statefulSummaryVec caches stateful summary series handles by vec label values.
140 type statefulSummaryVec struct {
141 cache *vecCache[*statefulSummaryInstrument]
142 + scope HostScope
143 }
144
145 // snapshotStateSetVec caches snapshot stateset series handles by vec label values.
146 type snapshotStateSetVec struct {
147 cache *vecCache[*snapshotStateSetInstrument]
148 + scope HostScope
149 }
150
151 // statefulStateSetVec caches stateful stateset series handles by vec label values.
152 type statefulStateSetVec struct {
153 cache *vecCache[*statefulStateSetInstrument]
154 + scope HostScope
155 }
156
157 // snapshotMeasureSetGaugeVec caches snapshot MeasureSet gauge series handles by vec label values.
158 type snapshotMeasureSetGaugeVec struct {
159 cache *vecCache[*snapshotMeasureSetGaugeInstrument]
160 + scope HostScope
161 }
162
163 // snapshotMeasureSetCounterVec caches snapshot MeasureSet counter series handles by vec label values.
164 type snapshotMeasureSetCounterVec struct {
165 cache *vecCache[*snapshotMeasureSetCounterInstrument]
166 + scope HostScope
167 }
168
169 // statefulMeasureSetGaugeVec caches stateful MeasureSet gauge series handles by vec label values.
170 type statefulMeasureSetGaugeVec struct {
171 cache *vecCache[*statefulMeasureSetGaugeInstrument]
172 + scope HostScope
173 }
174
175 // statefulMeasureSetCounterVec caches stateful MeasureSet counter series handles by vec label values.
176 type statefulMeasureSetCounterVec struct {
177 cache *vecCache[*statefulMeasureSetCounterInstrument]
178 + scope HostScope
179 }
180
181 // GaugeVec declares or reuses a snapshot gauge and exposes a label-values lookup API.
@@ -168,10 +184,12 @@ func (m *snapshotMeter) GaugeVec(name string, labelKeys []string, opts ...Instru
184 keys := mustNormalizeVecLabelKeys(labelKeys)
185 base := appendLabelSets(m.sets, nil)
186 return &snapshotGaugeVec{
171 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotGaugeInstrument {
187 + scope: m.scope,
188 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotGaugeInstrument {
189 return &snapshotGaugeInstrument{
190 backend: m.backend,
191 desc: desc,
192 + scope: scope,
193 base: appendVecSet(base, vecSet),
194 }
195 }),
@@ -184,10 +202,12 @@ func (m *statefulMeter) GaugeVec(name string, labelKeys []string, opts ...Instru
202 keys := mustNormalizeVecLabelKeys(labelKeys)
203 base := appendLabelSets(m.sets, nil)
204 return &statefulGaugeVec{
187 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulGaugeInstrument {
205 + scope: m.scope,
206 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulGaugeInstrument {
207 return &statefulGaugeInstrument{
208 backend: m.backend,
209 desc: desc,
210 + scope: scope,
211 base: appendVecSet(base, vecSet),
212 }
213 }),
@@ -200,10 +220,12 @@ func (m *snapshotMeter) CounterVec(name string, labelKeys []string, opts ...Inst
220 keys := mustNormalizeVecLabelKeys(labelKeys)
221 base := appendLabelSets(m.sets, nil)
222 return &snapshotCounterVec{
203 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotCounterInstrument {
223 + scope: m.scope,
224 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotCounterInstrument {
225 return &snapshotCounterInstrument{
226 backend: m.backend,
227 desc: desc,
228 + scope: scope,
229 base: appendVecSet(base, vecSet),
230 }
231 }),
@@ -216,10 +238,12 @@ func (m *statefulMeter) CounterVec(name string, labelKeys []string, opts ...Inst
238 keys := mustNormalizeVecLabelKeys(labelKeys)
239 base := appendLabelSets(m.sets, nil)
240 return &statefulCounterVec{
219 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulCounterInstrument {
241 + scope: m.scope,
242 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulCounterInstrument {
243 return &statefulCounterInstrument{
244 backend: m.backend,
245 desc: desc,
246 + scope: scope,
247 base: appendVecSet(base, vecSet),
248 }
249 }),
@@ -232,10 +256,12 @@ func (m *snapshotMeter) HistogramVec(name string, labelKeys []string, opts ...In
256 keys := mustNormalizeVecLabelKeys(labelKeys)
257 base := appendLabelSets(m.sets, nil)
258 return &snapshotHistogramVec{
235 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotHistogramInstrument {
259 + scope: m.scope,
260 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotHistogramInstrument {
261 return &snapshotHistogramInstrument{
262 backend: m.backend,
263 desc: desc,
264 + scope: scope,
265 base: appendVecSet(base, vecSet),
266 }
267 }),
@@ -248,10 +274,12 @@ func (m *statefulMeter) HistogramVec(name string, labelKeys []string, opts ...In
274 keys := mustNormalizeVecLabelKeys(labelKeys)
275 base := appendLabelSets(m.sets, nil)
276 return &statefulHistogramVec{
251 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulHistogramInstrument {
277 + scope: m.scope,
278 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulHistogramInstrument {
279 return &statefulHistogramInstrument{
280 backend: m.backend,
281 desc: desc,
282 + scope: scope,
283 base: appendVecSet(base, vecSet),
284 }
285 }),
@@ -264,10 +292,12 @@ func (m *snapshotMeter) SummaryVec(name string, labelKeys []string, opts ...Inst
292 keys := mustNormalizeVecLabelKeys(labelKeys)
293 base := appendLabelSets(m.sets, nil)
294 return &snapshotSummaryVec{
267 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotSummaryInstrument {
295 + scope: m.scope,
296 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotSummaryInstrument {
297 return &snapshotSummaryInstrument{
298 backend: m.backend,
299 desc: desc,
300 + scope: scope,
301 base: appendVecSet(base, vecSet),
302 }
303 }),
@@ -280,10 +310,12 @@ func (m *statefulMeter) SummaryVec(name string, labelKeys []string, opts ...Inst
310 keys := mustNormalizeVecLabelKeys(labelKeys)
311 base := appendLabelSets(m.sets, nil)
312 return &statefulSummaryVec{
283 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulSummaryInstrument {
313 + scope: m.scope,
314 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulSummaryInstrument {
315 return &statefulSummaryInstrument{
316 backend: m.backend,
317 desc: desc,
318 + scope: scope,
319 base: appendVecSet(base, vecSet),
320 }
321 }),
@@ -296,10 +328,12 @@ func (m *snapshotMeter) StateSetVec(name string, labelKeys []string, opts ...Ins
328 keys := mustNormalizeVecLabelKeys(labelKeys)
329 base := appendLabelSets(m.sets, nil)
330 return &snapshotStateSetVec{
299 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotStateSetInstrument {
331 + scope: m.scope,
332 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotStateSetInstrument {
333 return &snapshotStateSetInstrument{
334 backend: m.backend,
335 desc: desc,
336 + scope: scope,
337 base: appendVecSet(base, vecSet),
338 }
339 }),
@@ -312,10 +346,12 @@ func (m *statefulMeter) StateSetVec(name string, labelKeys []string, opts ...Ins
346 keys := mustNormalizeVecLabelKeys(labelKeys)
347 base := appendLabelSets(m.sets, nil)
348 return &statefulStateSetVec{
315 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulStateSetInstrument {
349 + scope: m.scope,
350 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulStateSetInstrument {
351 return &statefulStateSetInstrument{
352 backend: m.backend,
353 desc: desc,
354 + scope: scope,
355 base: appendVecSet(base, vecSet),
356 }
357 }),
@@ -328,10 +364,12 @@ func (m *snapshotMeter) MeasureSetGaugeVec(name string, labelKeys []string, opts
364 keys := mustNormalizeVecLabelKeys(labelKeys)
365 base := appendLabelSets(m.sets, nil)
366 return &snapshotMeasureSetGaugeVec{
331 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotMeasureSetGaugeInstrument {
367 + scope: m.scope,
368 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotMeasureSetGaugeInstrument {
369 return &snapshotMeasureSetGaugeInstrument{
370 backend: m.backend,
371 desc: desc,
372 + scope: scope,
373 base: appendVecSet(base, vecSet),
374 }
375 }),
@@ -344,10 +382,12 @@ func (m *snapshotMeter) MeasureSetCounterVec(name string, labelKeys []string, op
382 keys := mustNormalizeVecLabelKeys(labelKeys)
383 base := appendLabelSets(m.sets, nil)
384 return &snapshotMeasureSetCounterVec{
347 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *snapshotMeasureSetCounterInstrument {
385 + scope: m.scope,
386 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *snapshotMeasureSetCounterInstrument {
387 return &snapshotMeasureSetCounterInstrument{
388 backend: m.backend,
389 desc: desc,
390 + scope: scope,
391 base: appendVecSet(base, vecSet),
392 }
393 }),
@@ -360,10 +400,12 @@ func (m *statefulMeter) MeasureSetGaugeVec(name string, labelKeys []string, opts
400 keys := mustNormalizeVecLabelKeys(labelKeys)
401 base := appendLabelSets(m.sets, nil)
402 return &statefulMeasureSetGaugeVec{
363 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulMeasureSetGaugeInstrument {
403 + scope: m.scope,
404 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulMeasureSetGaugeInstrument {
405 return &statefulMeasureSetGaugeInstrument{
406 backend: m.backend,
407 desc: desc,
408 + scope: scope,
409 base: appendVecSet(base, vecSet),
410 }
411 }),
@@ -376,19 +418,77 @@ func (m *statefulMeter) MeasureSetCounterVec(name string, labelKeys []string, op
418 keys := mustNormalizeVecLabelKeys(labelKeys)
419 base := appendLabelSets(m.sets, nil)
420 return &statefulMeasureSetCounterVec{
379 - cache: newVecCache(m.backend, base, keys, func(base []LabelSet, vecSet LabelSet) *statefulMeasureSetCounterInstrument {
421 + scope: m.scope,
422 + cache: newVecCache(m.backend, base, keys, func(scope HostScope, base []LabelSet, vecSet LabelSet) *statefulMeasureSetCounterInstrument {
423 return &statefulMeasureSetCounterInstrument{
424 backend: m.backend,
425 desc: desc,
426 + scope: scope,
427 base: appendVecSet(base, vecSet),
428 }
429 }),
430 }
431 }
432
433 +func (v *snapshotGaugeVec) WithHostScope(scope HostScope) SnapshotGaugeVec {
434 + return &snapshotGaugeVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
435 +}
436 +
437 +func (v *statefulGaugeVec) WithHostScope(scope HostScope) StatefulGaugeVec {
438 + return &statefulGaugeVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
439 +}
440 +
441 +func (v *snapshotCounterVec) WithHostScope(scope HostScope) SnapshotCounterVec {
442 + return &snapshotCounterVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
443 +}
444 +
445 +func (v *statefulCounterVec) WithHostScope(scope HostScope) StatefulCounterVec {
446 + return &statefulCounterVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
447 +}
448 +
449 +func (v *snapshotHistogramVec) WithHostScope(scope HostScope) SnapshotHistogramVec {
450 + return &snapshotHistogramVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
451 +}
452 +
453 +func (v *statefulHistogramVec) WithHostScope(scope HostScope) StatefulHistogramVec {
454 + return &statefulHistogramVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
455 +}
456 +
457 +func (v *snapshotSummaryVec) WithHostScope(scope HostScope) SnapshotSummaryVec {
458 + return &snapshotSummaryVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
459 +}
460 +
461 +func (v *statefulSummaryVec) WithHostScope(scope HostScope) StatefulSummaryVec {
462 + return &statefulSummaryVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
463 +}
464 +
465 +func (v *snapshotStateSetVec) WithHostScope(scope HostScope) SnapshotStateSetVec {
466 + return &snapshotStateSetVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
467 +}
468 +
469 +func (v *statefulStateSetVec) WithHostScope(scope HostScope) StatefulStateSetVec {
470 + return &statefulStateSetVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
471 +}
472 +
473 +func (v *snapshotMeasureSetGaugeVec) WithHostScope(scope HostScope) SnapshotMeasureSetGaugeVec {
474 + return &snapshotMeasureSetGaugeVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
475 +}
476 +
477 +func (v *snapshotMeasureSetCounterVec) WithHostScope(scope HostScope) SnapshotMeasureSetCounterVec {
478 + return &snapshotMeasureSetCounterVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
479 +}
480 +
481 +func (v *statefulMeasureSetGaugeVec) WithHostScope(scope HostScope) StatefulMeasureSetGaugeVec {
482 + return &statefulMeasureSetGaugeVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
483 +}
484 +
485 +func (v *statefulMeasureSetCounterVec) WithHostScope(scope HostScope) StatefulMeasureSetCounterVec {
486 + return &statefulMeasureSetCounterVec{cache: v.cache, scope: mustNormalizeHostScope(scope)}
487 +}
488 +
489 // GetWithLabelValues returns a snapshot gauge handle for the provided vec label values.
490 func (v *snapshotGaugeVec) GetWithLabelValues(labelValues ...string) (SnapshotGauge, error) {
391 - inst, err := v.cache.get(labelValues...)
491 + inst, err := v.cache.get(v.scope, labelValues...)
492 if err != nil {
493 return nil, err
494 }
@@ -406,7 +506,7 @@ func (v *snapshotGaugeVec) WithLabelValues(labelValues ...string) SnapshotGauge
506
507 // GetWithLabelValues returns a stateful gauge handle for the provided vec label values.
508 func (v *statefulGaugeVec) GetWithLabelValues(labelValues ...string) (StatefulGauge, error) {
409 - inst, err := v.cache.get(labelValues...)
509 + inst, err := v.cache.get(v.scope, labelValues...)
510 if err != nil {
511 return nil, err
512 }
@@ -424,7 +524,7 @@ func (v *statefulGaugeVec) WithLabelValues(labelValues ...string) StatefulGauge
524
525 // GetWithLabelValues returns a snapshot counter handle for the provided vec label values.
526 func (v *snapshotCounterVec) GetWithLabelValues(labelValues ...string) (SnapshotCounter, error) {
427 - inst, err := v.cache.get(labelValues...)
527 + inst, err := v.cache.get(v.scope, labelValues...)
528 if err != nil {
529 return nil, err
530 }
@@ -442,7 +542,7 @@ func (v *snapshotCounterVec) WithLabelValues(labelValues ...string) SnapshotCoun
542
543 // GetWithLabelValues returns a stateful counter handle for the provided vec label values.
544 func (v *statefulCounterVec) GetWithLabelValues(labelValues ...string) (StatefulCounter, error) {
445 - inst, err := v.cache.get(labelValues...)
545 + inst, err := v.cache.get(v.scope, labelValues...)
546 if err != nil {
547 return nil, err
548 }
@@ -460,7 +560,7 @@ func (v *statefulCounterVec) WithLabelValues(labelValues ...string) StatefulCoun
560
561 // GetWithLabelValues returns a snapshot histogram handle for the provided vec label values.
562 func (v *snapshotHistogramVec) GetWithLabelValues(labelValues ...string) (SnapshotHistogram, error) {
463 - inst, err := v.cache.get(labelValues...)
563 + inst, err := v.cache.get(v.scope, labelValues...)
564 if err != nil {
565 return nil, err
566 }
@@ -478,7 +578,7 @@ func (v *snapshotHistogramVec) WithLabelValues(labelValues ...string) SnapshotHi
578
579 // GetWithLabelValues returns a stateful histogram handle for the provided vec label values.
580 func (v *statefulHistogramVec) GetWithLabelValues(labelValues ...string) (StatefulHistogram, error) {
481 - inst, err := v.cache.get(labelValues...)
581 + inst, err := v.cache.get(v.scope, labelValues...)
582 if err != nil {
583 return nil, err
584 }
@@ -496,7 +596,7 @@ func (v *statefulHistogramVec) WithLabelValues(labelValues ...string) StatefulHi
596
597 // GetWithLabelValues returns a snapshot summary handle for the provided vec label values.
598 func (v *snapshotSummaryVec) GetWithLabelValues(labelValues ...string) (SnapshotSummary, error) {
499 - inst, err := v.cache.get(labelValues...)
599 + inst, err := v.cache.get(v.scope, labelValues...)
600 if err != nil {
601 return nil, err
602 }
@@ -514,7 +614,7 @@ func (v *snapshotSummaryVec) WithLabelValues(labelValues ...string) SnapshotSumm
614
615 // GetWithLabelValues returns a stateful summary handle for the provided vec label values.
616 func (v *statefulSummaryVec) GetWithLabelValues(labelValues ...string) (StatefulSummary, error) {
517 - inst, err := v.cache.get(labelValues...)
617 + inst, err := v.cache.get(v.scope, labelValues...)
618 if err != nil {
619 return nil, err
620 }
@@ -532,7 +632,7 @@ func (v *statefulSummaryVec) WithLabelValues(labelValues ...string) StatefulSumm
632
633 // GetWithLabelValues returns a snapshot stateset handle for the provided vec label values.
634 func (v *snapshotStateSetVec) GetWithLabelValues(labelValues ...string) (StateSetInstrument, error) {
535 - inst, err := v.cache.get(labelValues...)
635 + inst, err := v.cache.get(v.scope, labelValues...)
636 if err != nil {
637 return nil, err
638 }
@@ -550,7 +650,7 @@ func (v *snapshotStateSetVec) WithLabelValues(labelValues ...string) StateSetIns
650
651 // GetWithLabelValues returns a stateful stateset handle for the provided vec label values.
652 func (v *statefulStateSetVec) GetWithLabelValues(labelValues ...string) (StateSetInstrument, error) {
553 - inst, err := v.cache.get(labelValues...)
653 + inst, err := v.cache.get(v.scope, labelValues...)
654 if err != nil {
655 return nil, err
656 }
@@ -568,7 +668,7 @@ func (v *statefulStateSetVec) WithLabelValues(labelValues ...string) StateSetIns
668
669 // GetWithLabelValues returns a snapshot MeasureSet gauge handle for the provided vec label values.
670 func (v *snapshotMeasureSetGaugeVec) GetWithLabelValues(labelValues ...string) (SnapshotMeasureSetGauge, error) {
571 - inst, err := v.cache.get(labelValues...)
671 + inst, err := v.cache.get(v.scope, labelValues...)
672 if err != nil {
673 return nil, err
674 }
@@ -586,7 +686,7 @@ func (v *snapshotMeasureSetGaugeVec) WithLabelValues(labelValues ...string) Snap
686
687 // GetWithLabelValues returns a snapshot MeasureSet counter handle for the provided vec label values.
688 func (v *snapshotMeasureSetCounterVec) GetWithLabelValues(labelValues ...string) (SnapshotMeasureSetCounter, error) {
589 - inst, err := v.cache.get(labelValues...)
689 + inst, err := v.cache.get(v.scope, labelValues...)
690 if err != nil {
691 return nil, err
692 }
@@ -604,7 +704,7 @@ func (v *snapshotMeasureSetCounterVec) WithLabelValues(labelValues ...string) Sn
704
705 // GetWithLabelValues returns a stateful MeasureSet gauge handle for the provided vec label values.
706 func (v *statefulMeasureSetGaugeVec) GetWithLabelValues(labelValues ...string) (StatefulMeasureSetGauge, error) {
607 - inst, err := v.cache.get(labelValues...)
707 + inst, err := v.cache.get(v.scope, labelValues...)
708 if err != nil {
709 return nil, err
710 }
@@ -622,7 +722,7 @@ func (v *statefulMeasureSetGaugeVec) WithLabelValues(labelValues ...string) Stat
722
723 // GetWithLabelValues returns a stateful MeasureSet counter handle for the provided vec label values.
724 func (v *statefulMeasureSetCounterVec) GetWithLabelValues(labelValues ...string) (StatefulMeasureSetCounter, error) {
625 - inst, err := v.cache.get(labelValues...)
725 + inst, err := v.cache.get(v.scope, labelValues...)
726 if err != nil {
727 return nil, err
728 }
@@ -697,3 +797,13 @@ func packVecLabelValues(values []string) string {
797 }
798 return b.String()
799 }
800 +
801 +func scopedVecCacheKey(scopeKey, labelsKey string) string {
802 + if scopeKey == "" {
803 + return labelsKey
804 + }
805 + if labelsKey == "" {
806 + return "\xfe" + scopeKey
807 + }
808 + return "\xfe" + scopeKey + "\xff" + labelsKey
809 +}
src/go/pkg/netipc/service/cgroups/cgroups_unix_test.go
+2 -2
@@ -112,7 +112,7 @@ func (ts *unixTestServer) stop() {
112
113 func connectReadyUnix(t *testing.T, client *Client) {
114 t.Helper()
115 - for i := 0; i < 200; i++ {
115 + for range 200 {
116 client.Refresh()
117 if client.Ready() {
118 return
@@ -162,7 +162,7 @@ func TestCacheRoundTripUnix(t *testing.T) {
162 defer cache.Close()
163
164 var updated bool
165 - for i := 0; i < 200; i++ {
165 + for range 200 {
166 if cache.Refresh() {
167 updated = true
168 break
src/go/pkg/netipc/service/raw/cache_test.go
+2 -2
@@ -275,7 +275,7 @@ func TestCacheLargeDataset(t *testing.T) {
275 }
276 builder.SetHeader(1, 100)
277
278 - for i := uint32(0); i < N; i++ {
278 + for i := range uint32(N) {
279 name := fmt.Sprintf("cgroup-%d", i)
280 path := fmt.Sprintf("/sys/fs/cgroup/test/%d", i)
281 enabled := uint32(1)
@@ -320,7 +320,7 @@ func TestCacheLargeDataset(t *testing.T) {
320 }
321
322 // Verify all lookups
323 - for i := uint32(0); i < N; i++ {
323 + for i := range uint32(N) {
324 name := fmt.Sprintf("cgroup-%d", i)
325 item, found := cache.Lookup(i+1000, name)
326 if !found {
src/go/pkg/netipc/service/raw/client.go
+3 -9
@@ -153,20 +153,14 @@ func (c *Client) sessionMaxResponsePayloadBytes() uint32 {
153 }
154
155 func (c *Client) noteRequestCapacity(payloadLen uint32) {
156 - grown := nextPowerOf2U32(payloadLen)
157 - if grown > protocol.MaxPayloadCap {
158 - grown = protocol.MaxPayloadCap
159 - }
156 + grown := min(nextPowerOf2U32(payloadLen), protocol.MaxPayloadCap)
157 if grown > c.config.MaxRequestPayloadBytes {
158 c.config.MaxRequestPayloadBytes = grown
159 }
160 }
161
162 func (c *Client) noteResponseCapacity(payloadLen uint32) {
166 - grown := nextPowerOf2U32(payloadLen)
167 - if grown > protocol.MaxPayloadCap {
168 - grown = protocol.MaxPayloadCap
169 - }
163 + grown := min(nextPowerOf2U32(payloadLen), protocol.MaxPayloadCap)
164 if grown > c.config.MaxResponsePayloadBytes {
165 c.config.MaxResponsePayloadBytes = grown
166 }
@@ -477,7 +471,7 @@ func (c *Client) CallIncrementBatch(values []uint64) ([]uint64, error) {
471
472 // Extract each response item
473 out := make([]uint64, itemCount)
480 - for i := uint32(0); i < itemCount; i++ {
474 + for i := range itemCount {
475 itemData, gerr := protocol.BatchItemGet(respPayload, itemCount, i)
476 if gerr != nil {
477 return gerr
src/go/pkg/netipc/service/raw/client_test.go
+5 -5
@@ -402,13 +402,13 @@ func TestConcurrentClients(t *testing.T) {
402
403 results := make(chan result, numClients)
404
405 - for i := 0; i < numClients; i++ {
405 + for range numClients {
406 go func() {
407 r := result{}
408 client := NewSnapshotClient(testRunDir, svc, testClientConfig())
409 defer client.Close()
410
411 - for retry := 0; retry < 100; retry++ {
411 + for range 100 {
412 client.Refresh()
413 if client.Ready() {
414 break
@@ -422,7 +422,7 @@ func TestConcurrentClients(t *testing.T) {
422 return
423 }
424
425 - for j := 0; j < requestsPerClient; j++ {
425 + for range requestsPerClient {
426 view, err := client.CallSnapshot()
427 if err != nil || view.ItemCount != 3 {
428 r.failures++
@@ -442,7 +442,7 @@ func TestConcurrentClients(t *testing.T) {
442
443 totalSuccess := 0
444 totalFailure := 0
445 - for i := 0; i < numClients; i++ {
445 + for range numClients {
446 r := <-results
447 totalSuccess += r.successes
448 totalFailure += r.failures
@@ -514,7 +514,7 @@ func TestStatusReporting(t *testing.T) {
514 }
515
516 // Make 3 successful calls
517 - for i := 0; i < 3; i++ {
517 + for i := range 3 {
518 _, err := client.CallSnapshot()
519 if err != nil {
520 t.Fatalf("call %d failed: %v", i, err)
src/go/pkg/netipc/service/raw/more_unix_test.go
+1 -1
@@ -150,7 +150,7 @@ func startRawPosixSessionServerN(
150 go func() {
151 defer listener.Close()
152
153 - for i := 0; i < accepts; i++ {
153 + for range accepts {
154 session, err := listener.Accept()
155 if err != nil {
156 srv.doneCh <- err
src/go/pkg/netipc/service/raw/ping_pong_test.go
+2 -2
@@ -52,7 +52,7 @@ func TestIncrementPingPong(t *testing.T) {
52 // 10 rounds: send 0 -> get 1 -> send 1 -> get 2 -> ... -> value == 10
53 var val uint64
54 responsesReceived := 0
55 - for i := 0; i < 10; i++ {
55 + for i := range 10 {
56 got, err := client.CallIncrement(val)
57 if err != nil {
58 t.Fatalf("round %d: CallIncrement(%d) failed: %v", i, val, err)
@@ -108,7 +108,7 @@ func TestStringReversePingPong(t *testing.T) {
108 // 6 rounds: feed each response back as the next request
109 responsesReceived := 0
110 current := original
111 - for i := 0; i < 6; i++ {
111 + for i := range 6 {
112 view, err := client.CallStringReverse(current)
113 if err != nil {
114 t.Fatalf("round %d: CallStringReverse(%q) failed: %v", i+1, current, err)
src/go/pkg/netipc/service/raw/stress_test.go
+21 -23
@@ -30,7 +30,7 @@ func largeHandler(n int) DispatchHandler {
30 }
31 builder.SetHeader(1, 42)
32
33 - for i := 0; i < n; i++ {
33 + for i := range n {
34 name := fmt.Sprintf("container-%04d", i)
35 path := fmt.Sprintf("/sys/fs/cgroup/docker/%04d", i)
36 hash := simpleHash(name)
@@ -181,7 +181,7 @@ func TestStress1000Items(t *testing.T) {
181 if int(view.ItemCount) != N {
182 t.Fatalf("expected %d items, got %d", N, view.ItemCount)
183 }
184 - for i := 0; i < N; i++ {
184 + for i := range N {
185 item, ierr := view.Item(uint32(i))
186 if ierr != nil {
187 t.Fatalf("item %d decode error: %v", i, ierr)
@@ -265,13 +265,13 @@ func TestStress50Clients(t *testing.T) {
265
266 start := time.Now()
267
268 - for i := 0; i < numClients; i++ {
268 + for i := range numClients {
269 go func(id int) {
270 r := result{clientID: id}
271 client := NewSnapshotClient(testRunDir, svc, testClientConfig())
272 defer client.Close()
273
274 - for retry := 0; retry < 200; retry++ {
274 + for range 200 {
275 client.Refresh()
276 if client.Ready() {
277 break
@@ -285,7 +285,7 @@ func TestStress50Clients(t *testing.T) {
285 return
286 }
287
288 - for j := 0; j < requestsPerClient; j++ {
288 + for range requestsPerClient {
289 view, err := client.CallSnapshot()
290 if err != nil || view.ItemCount != 3 {
291 r.failures++
@@ -312,7 +312,7 @@ func TestStress50Clients(t *testing.T) {
312
313 totalSuccess := 0
314 totalFailure := 0
315 - for i := 0; i < numClients; i++ {
315 + for range numClients {
316 r := <-results
317 totalSuccess += r.successes
318 totalFailure += r.failures
@@ -354,13 +354,13 @@ func TestStressConcurrentCacheClients(t *testing.T) {
354
355 start := time.Now()
356
357 - for i := 0; i < numClients; i++ {
357 + for range numClients {
358 go func() {
359 r := result{}
360 cache := NewCache(testRunDir, svc, testClientConfig())
361 defer cache.Close()
362
363 - for j := 0; j < requestsPerClient; j++ {
363 + for range requestsPerClient {
364 updated := cache.Refresh()
365 if updated || cache.Ready() {
366 status := cache.Status()
@@ -386,7 +386,7 @@ func TestStressConcurrentCacheClients(t *testing.T) {
386
387 totalSuccess := 0
388 totalFailure := 0
389 - for i := 0; i < numClients; i++ {
389 + for range numClients {
390 r := <-results
391 totalSuccess += r.successes
392 totalFailure += r.failures
@@ -421,10 +421,10 @@ func TestStressRapidConnectDisconnect(t *testing.T) {
421
422 start := time.Now()
423
424 - for i := 0; i < cycles; i++ {
424 + for range cycles {
425 client := NewSnapshotClient(testRunDir, svc, testClientConfig())
426
427 - for r := 0; r < 50; r++ {
427 + for range 50 {
428 client.Refresh()
429 if client.Ready() {
430 break
@@ -483,10 +483,8 @@ func TestStressLongRunning60s(t *testing.T) {
483 var wg sync.WaitGroup
484 stop := make(chan struct{})
485
486 - for i := 0; i < numClients; i++ {
487 - wg.Add(1)
488 - go func() {
489 - defer wg.Done()
486 + for range numClients {
487 + wg.Go(func() {
488 cache := NewCache(testRunDir, svc, testClientConfig())
489 defer cache.Close()
490
@@ -511,7 +509,7 @@ func TestStressLongRunning60s(t *testing.T) {
509
510 time.Sleep(time.Millisecond)
511 }
514 - }()
512 + })
513 }
514
515 time.Sleep(duration)
@@ -588,7 +586,7 @@ func TestStressMixedTransport(t *testing.T) {
586 client := NewSnapshotClient(testRunDir, svc, ccfg)
587 defer client.Close()
588
591 - for retry := 0; retry < 200; retry++ {
589 + for range 200 {
590 client.Refresh()
591 if client.Ready() {
592 break
@@ -596,7 +594,7 @@ func TestStressMixedTransport(t *testing.T) {
594 time.Sleep(5 * time.Millisecond)
595 }
596
599 - for i := 0; i < 10; i++ {
597 + for range 10 {
598 view, err := client.CallSnapshot()
599 if err == nil && view.ItemCount == 3 {
600 item0, ierr := view.Item(0)
@@ -620,7 +618,7 @@ func TestStressMixedTransport(t *testing.T) {
618 client := NewSnapshotClient(testRunDir, svc, ccfg)
619 defer client.Close()
620
623 - for retry := 0; retry < 200; retry++ {
621 + for range 200 {
622 client.Refresh()
623 if client.Ready() {
624 break
@@ -628,7 +626,7 @@ func TestStressMixedTransport(t *testing.T) {
626 time.Sleep(5 * time.Millisecond)
627 }
628
631 - for i := 0; i < 10; i++ {
629 + for range 10 {
630 view, err := client.CallSnapshot()
631 if err == nil && view.ItemCount == 3 {
632 r.success++
@@ -649,7 +647,7 @@ func TestStressMixedTransport(t *testing.T) {
647 client := NewSnapshotClient(testRunDir, svc, ccfg)
648 defer client.Close()
649
652 - for retry := 0; retry < 200; retry++ {
650 + for range 200 {
651 client.Refresh()
652 if client.Ready() {
653 break
@@ -657,7 +655,7 @@ func TestStressMixedTransport(t *testing.T) {
655 time.Sleep(5 * time.Millisecond)
656 }
657
660 - for i := 0; i < 10; i++ {
658 + for range 10 {
659 view, err := client.CallSnapshot()
660 if err == nil && view.ItemCount == 3 {
661 item0, ierr := view.Item(0)
@@ -673,7 +671,7 @@ func TestStressMixedTransport(t *testing.T) {
671
672 totalSuccess := 0
673 totalFailure := 0
676 - for i := 0; i < 3; i++ {
674 + for range 3 {
675 r := <-results
676 t.Logf("client %d (%s): %d ok, %d fail", r.clientID, r.profile, r.success, r.failure)
677 totalSuccess += r.success
src/go/plugin/agent/jobmgr/job_factory.go
+4
@@ -18,6 +18,7 @@ import (
18 "github.com/netdata/netdata/go/plugins/plugin/framework/jobruntime"
19 "github.com/netdata/netdata/go/plugins/plugin/framework/metricsaudit"
20 "github.com/netdata/netdata/go/plugins/plugin/framework/runtimecomp"
21 + "github.com/netdata/netdata/go/plugins/plugin/framework/vnoderegistry"
22 "github.com/netdata/netdata/go/plugins/plugin/framework/vnodes"
23 )
24
@@ -46,6 +47,7 @@ type jobFactory struct {
47 auditDataDir string
48
49 runtimeService runtimecomp.Service
50 + vnodeRegistry *vnoderegistry.Registry
51
52 secretResolver *secretresolver.Resolver
53 secretStoreSvc secretstore.Service
@@ -66,6 +68,7 @@ func newJobFactory(m *Manager) *jobFactory {
68 auditDataDir: m.auditDataDir,
69
70 runtimeService: m.runtimeService,
71 + vnodeRegistry: m.vnodeRegistry,
72 secretResolver: m.secretResolver,
73 secretStoreSvc: m.secretsCtl.Service(),
74 ctx: m.baseContext(),
@@ -143,6 +146,7 @@ func (f *jobFactory) createV2(cfg confgroup.Config, creator collectorapi.Creator
146 Module: mod,
147 FunctionOnly: functionOnly,
148 RuntimeService: f.runtimeService,
149 + VnodeRegistry: f.vnodeRegistry,
150 }
151 if vnode != nil {
152 jobCfg.Vnode = *vnode.Copy()
src/go/plugin/agent/jobmgr/manager.go
+8
@@ -27,6 +27,7 @@ import (
27 "github.com/netdata/netdata/go/plugins/plugin/framework/functions"
28 "github.com/netdata/netdata/go/plugins/plugin/framework/metricsaudit"
29 "github.com/netdata/netdata/go/plugins/plugin/framework/runtimecomp"
30 + "github.com/netdata/netdata/go/plugins/plugin/framework/vnoderegistry"
31 "github.com/netdata/netdata/go/plugins/plugin/framework/vnodes"
32 )
33
@@ -47,6 +48,7 @@ type Config struct {
48 AuditDataDir string
49 FunctionJSONWriter func(payload []byte, code int)
50 RuntimeService runtimecomp.Service
51 + VnodeRegistry *vnoderegistry.Registry
52 }
53
54 const (
@@ -82,6 +84,10 @@ func New(cfg Config) *Manager {
84 storeCreators := backends.Creators()
85 secretStoreSvc = secretstore.NewService(storeCreators...)
86 }
87 + vnodeRegistry := cfg.VnodeRegistry
88 + if vnodeRegistry == nil {
89 + vnodeRegistry = vnoderegistry.New()
90 + }
91
92 mgr := &Manager{
93 Logger: logger.New().With(
@@ -116,6 +122,7 @@ func New(cfg Config) *Manager {
122
123 dyncfgResponder: api,
124 runtimeService: cfg.RuntimeService,
125 + vnodeRegistry: vnodeRegistry,
126 secretResolver: secretresolver.New(),
127 }
128 mgr.funcCtl = funcctl.New(funcctl.Options{
@@ -233,6 +240,7 @@ type Manager struct {
240 // RuntimeService is an optional runtime/internal metrics registration seam.
241 // When set, V2 jobs may register per-job runtime components.
242 runtimeService runtimecomp.Service
243 + vnodeRegistry *vnoderegistry.Registry
244
245 secretResolver *secretresolver.Resolver
246 }
src/go/plugin/framework/chartemit/apply.go
+9 -9
@@ -77,22 +77,22 @@ func emitHostSelection(api *netdataapi.API, env EmitEnv) error {
77 return nil
78 }
79
80 - guid := sanitizeWireID(env.HostScope.GUID)
80 + guid := strings.TrimSpace(env.HostScope.GUID)
81 if guid == "" {
82 return fmt.Errorf("chartemit: emit env host scope guid is required")
83 }
84 + if sanitizeWireID(guid) != guid {
85 + return fmt.Errorf("chartemit: emit env host scope guid contains unsupported characters")
86 + }
87 if env.HostScope.Define != nil {
85 - defineGUID := sanitizeWireID(env.HostScope.Define.GUID)
86 - if defineGUID == "" {
87 - return fmt.Errorf("chartemit: host define guid is required")
88 + define, err := PrepareHostInfo(*env.HostScope.Define)
89 + if err != nil {
90 + return err
91 }
89 - if defineGUID != guid {
92 + if define.GUID != guid {
93 return fmt.Errorf("chartemit: host define guid %q does not match host scope guid %q", env.HostScope.Define.GUID, env.HostScope.GUID)
94 }
92 - if strings.TrimSpace(env.HostScope.Define.Hostname) == "" {
93 - return fmt.Errorf("chartemit: host define hostname is required")
94 - }
95 - api.HOSTINFO(*env.HostScope.Define)
95 + api.HOSTINFO(define)
96 }
97 api.HOST(guid)
98 return nil
src/go/plugin/framework/chartemit/apply_test.go
+53
@@ -538,6 +538,59 @@ func TestApplyPlanRejectsMismatchedHostDefine(t *testing.T) {
538 assert.Equal(t, "", buf.String())
539 }
540
541 +func TestPrepareHostInfoScenarios(t *testing.T) {
542 + cases := map[string]struct {
543 + info netdataapi.HostInfo
544 + want netdataapi.HostInfo
545 + wantErr string
546 + }{
547 + "rejects unsafe guid": {
548 + info: netdataapi.HostInfo{
549 + GUID: "node'\nguid",
550 + Hostname: "node-host",
551 + },
552 + wantErr: "unsupported characters",
553 + },
554 + "rejects unsafe hostname": {
555 + info: netdataapi.HostInfo{
556 + GUID: "node-guid",
557 + Hostname: "node'\nhost",
558 + },
559 + wantErr: "unsupported characters",
560 + },
561 + "normalizes labels": {
562 + info: netdataapi.HostInfo{
563 + GUID: "node-guid",
564 + Hostname: "node-host",
565 + Labels: map[string]string{
566 + "region'\n": "eu'\n",
567 + " ": "ignored",
568 + },
569 + },
570 + want: netdataapi.HostInfo{
571 + GUID: "node-guid",
572 + Hostname: "node-host",
573 + Labels: map[string]string{
574 + "_hostname": "node-host",
575 + "region": "eu ",
576 + },
577 + },
578 + },
579 + }
580 +
581 + for name, tc := range cases {
582 + t.Run(name, func(t *testing.T) {
583 + got, err := PrepareHostInfo(tc.info)
584 + if tc.wantErr != "" {
585 + require.ErrorContains(t, err, tc.wantErr)
586 + return
587 + }
588 + require.NoError(t, err)
589 + assert.Equal(t, tc.want, got)
590 + })
591 + }
592 +}
593 +
594 func TestApplyPlanRemoveOnlyBatchStillSelectsHost(t *testing.T) {
595 var buf bytes.Buffer
596 api := netdataapi.New(&buf)
src/go/plugin/framework/chartemit/host.go
+33 -9
@@ -5,6 +5,7 @@ package chartemit
5 import (
6 "fmt"
7 "maps"
8 + "sort"
9 "strings"
10
11 "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
@@ -13,29 +14,29 @@ import (
14 // PrepareHostInfo normalizes host-definition payloads before HOST_DEFINE.
15 //
16 // Current semantics intentionally match v1 vnode emission:
16 -// - GUID/hostname must be present,
17 +// - GUID/hostname must be present and wire-safe,
18 // - "_hostname" is injected when absent,
18 -// - label values are sanitized for Netdata wire output.
19 +// - label keys and values are normalized for Netdata wire output.
20 func PrepareHostInfo(info netdataapi.HostInfo) (netdataapi.HostInfo, error) {
21 guid := strings.TrimSpace(info.GUID)
22 if guid == "" {
23 return netdataapi.HostInfo{}, fmt.Errorf("chartemit: host guid is required")
24 }
25 + if sanitizeWireID(guid) != guid {
26 + return netdataapi.HostInfo{}, fmt.Errorf("chartemit: host guid contains unsupported characters")
27 + }
28 hostname := strings.TrimSpace(info.Hostname)
29 if hostname == "" {
30 return netdataapi.HostInfo{}, fmt.Errorf("chartemit: host hostname is required")
31 }
28 -
29 - labels := maps.Clone(info.Labels)
30 - if labels == nil {
31 - labels = make(map[string]string)
32 + if sanitizeWireValue(hostname) != hostname {
33 + return netdataapi.HostInfo{}, fmt.Errorf("chartemit: host hostname contains unsupported characters")
34 }
35 +
36 + labels := normalizeHostInfoLabels(info.Labels)
37 if _, ok := labels["_hostname"]; !ok {
38 labels["_hostname"] = hostname
39 }
36 - for key, value := range labels {
37 - labels[key] = sanitizeWireValue(value)
38 - }
40
41 return netdataapi.HostInfo{
42 GUID: guid,
@@ -43,3 +44,26 @@ func PrepareHostInfo(info netdataapi.HostInfo) (netdataapi.HostInfo, error) {
44 Labels: labels,
45 }, nil
46 }
47 +
48 +func normalizeHostInfoLabels(in map[string]string) map[string]string {
49 + labels := maps.Clone(in)
50 + if len(labels) == 0 {
51 + return make(map[string]string)
52 + }
53 +
54 + keys := make([]string, 0, len(labels))
55 + for key := range labels {
56 + keys = append(keys, key)
57 + }
58 + sort.Strings(keys)
59 +
60 + out := make(map[string]string, len(labels))
61 + for _, key := range keys {
62 + sKey := sanitizeWireID(key)
63 + if sKey == "" {
64 + continue
65 + }
66 + out[sKey] = sanitizeWireValue(labels[key])
67 + }
68 + return out
69 +}
src/go/plugin/framework/chartengine/engine_test.go
+3
@@ -262,6 +262,9 @@ func TestEnginePreparePlanLifecycleScenarios(t *testing.T) {
262 e.ResetMaterialized()
263
264 require.ErrorIs(t, attempt.Commit(), ErrStalePlanAttempt)
265 + nextAttempt, err := e.PreparePlan(store.Read())
266 + require.NoError(t, err)
267 + nextAttempt.Abort()
268 },
269 },
270 "repeated commit is rejected after successful commit": {
src/go/plugin/framework/chartengine/options.go
+17
@@ -19,6 +19,7 @@ type engineConfig struct {
19 selectorOverride policyOverride[metrixselector.Selector]
20 runtimeStore metrix.RuntimeStore
21 runtimeStoreSet bool
22 + runtimeObserver func(PlanRuntimeSample)
23 log *logger.Logger
24 seriesSelection seriesSelectionMode
25 runtimePlanner bool
@@ -116,6 +117,22 @@ func WithRuntimeStore(store metrix.RuntimeStore) Option {
117 }
118 }
119
120 +// WithRuntimeSampleObserver configures a callback for per-build runtime samples.
121 +//
122 +// The callback fires for successful builds, build errors, and collect-status
123 +// skips. It does not fire for pre-build contract errors such as an outstanding
124 +// plan attempt.
125 +//
126 +// The callback is in addition to WithRuntimeStore. Pass WithRuntimeStore(nil)
127 +// when samples are aggregated elsewhere and the engine should not write its own
128 +// runtime metrics.
129 +func WithRuntimeSampleObserver(fn func(PlanRuntimeSample)) Option {
130 + return func(cfg *engineConfig) error {
131 + cfg.runtimeObserver = fn
132 + return nil
133 + }
134 +}
135 +
136 // WithLogger configures chartengine logger.
137 func WithLogger(l *logger.Logger) Option {
138 return func(cfg *engineConfig) error {
src/go/plugin/framework/chartengine/planner.go
+2 -3
@@ -123,9 +123,6 @@ func (e *Engine) preparePlan(reader metrix.Reader) (Plan, materializedState, uin
123 if reader == nil {
124 return Plan{}, materializedState{}, 0, 0, 0, false, fmt.Errorf("chartengine: nil metrics reader")
125 }
126 - sample := planRuntimeSample{startedAt: time.Now()}
127 - defer func() { e.observeBuildSample(sample) }()
128 -
126 out := Plan{
127 Actions: make([]EngineAction, 0),
128 InferredDimensions: make([]InferredDimension, 0),
@@ -137,6 +134,8 @@ func (e *Engine) preparePlan(reader metrix.Reader) (Plan, materializedState, uin
134 if e.state.outstanding != 0 {
135 return Plan{}, materializedState{}, 0, 0, 0, false, ErrOutstandingPlanAttempt
136 }
137 + sample := PlanRuntimeSample{startedAt: time.Now()}
138 + defer func() { e.observeBuildSample(sample) }()
139 // Failed attempt must not trigger lifecycle transitions.
140 if collectMeta.LastAttemptStatus != metrix.CollectStatusSuccess {
141 sample.skippedFailed = true
src/go/plugin/framework/chartengine/runtime_metrics.go
+203 -8
@@ -3,6 +3,7 @@
3 package chartengine
4
5 import (
6 + "sync"
7 "time"
8
9 "github.com/netdata/netdata/go/plugins/pkg/metrix"
@@ -55,7 +56,11 @@ type runtimeMetrics struct {
56 lifecycleRemovedDimensionByExpiry metrix.StatefulCounter
57 }
58
58 -type planRuntimeSample struct {
59 +// PlanRuntimeSample is an opaque snapshot of one planner build used by
60 +// chartengine runtime observers and RuntimeAggregator. Its fields are
61 +// intentionally package-private so chartengine remains the owner of runtime
62 +// metric semantics while callers can pass samples between chartengine APIs.
63 +type PlanRuntimeSample struct {
64 startedAt time.Time
65
66 buildErr bool
@@ -192,7 +197,7 @@ func newRuntimeMetrics(store metrix.RuntimeStore) *runtimeMetrics {
197 ),
198 routeCacheEntries: metrix.SeededGauge(meter,
199 "route_cache_entries",
195 - metrix.WithDescription("Current number of route cache entries"),
200 + metrix.WithDescription("Route cache entries in the latest successful build rollup"),
201 metrix.WithChartFamily("ChartEngine/Route Cache"),
202 metrix.WithUnit("entries"),
203 ),
@@ -244,13 +249,13 @@ func newRuntimeMetrics(store metrix.RuntimeStore) *runtimeMetrics {
249
250 planChartInstances: metrix.SeededGauge(meter,
251 "plan_chart_instances",
247 - metrix.WithDescription("Chart instances in last successful build plan"),
252 + metrix.WithDescription("Chart instances in the latest successful build rollup"),
253 metrix.WithChartFamily("ChartEngine/Plan"),
254 metrix.WithUnit("charts"),
255 ),
256 planInferredDimensions: metrix.SeededGauge(meter,
257 "plan_inferred_dimensions",
253 - metrix.WithDescription("Inferred dimensions in last successful build plan"),
258 + metrix.WithDescription("Inferred dimensions in the latest successful build rollup"),
259 metrix.WithChartFamily("ChartEngine/Plan"),
260 metrix.WithUnit("dimensions"),
261 ),
@@ -268,7 +273,7 @@ func newRuntimeMetrics(store metrix.RuntimeStore) *runtimeMetrics {
273 }
274 }
275
271 -func (m *runtimeMetrics) observeBuild(sample planRuntimeSample) {
276 +func (m *runtimeMetrics) observeBuild(sample PlanRuntimeSample) {
277 if m == nil {
278 return
279 }
@@ -384,18 +389,208 @@ func (e *Engine) RuntimeStore() metrix.RuntimeStore {
389 return e.state.runtimeStore
390 }
391
387 -func (e *Engine) observeBuildSample(sample planRuntimeSample) {
392 +func (e *Engine) observeBuildSample(sample PlanRuntimeSample) {
393 if e == nil {
394 return
395 }
396 + if e.state.cfg.runtimeObserver != nil {
397 + e.state.cfg.runtimeObserver(sample)
398 + }
399 if e.state.runtimeStats == nil {
400 return
401 }
402 e.state.runtimeStats.observeBuild(sample)
403 }
404
397 -func actionKindCounts(actions []EngineAction) planRuntimeSample {
398 - out := planRuntimeSample{}
405 +// RuntimeAggregator records runtime samples from multiple engines and emits one
406 +// rolled-up chartengine runtime stream.
407 +type RuntimeAggregator struct {
408 + mu sync.Mutex
409 + metrics *runtimeMetrics
410 + samples []PlanRuntimeSample
411 +}
412 +
413 +// NewRuntimeAggregator creates a runtime sample aggregator backed by store.
414 +func NewRuntimeAggregator(store metrix.RuntimeStore) *RuntimeAggregator {
415 + return &RuntimeAggregator{metrics: newRuntimeMetrics(store)}
416 +}
417 +
418 +// Observe records one engine build sample.
419 +func (a *RuntimeAggregator) Observe(sample PlanRuntimeSample) {
420 + if a == nil {
421 + return
422 + }
423 + a.mu.Lock()
424 + a.samples = append(a.samples, sample)
425 + a.mu.Unlock()
426 +}
427 +
428 +// Reset drops accumulated samples without writing them.
429 +func (a *RuntimeAggregator) Reset() {
430 + if a == nil {
431 + return
432 + }
433 + a.mu.Lock()
434 + a.samples = nil
435 + a.mu.Unlock()
436 +}
437 +
438 +// Flush emits accumulated samples into the aggregate runtime store.
439 +func (a *RuntimeAggregator) Flush() {
440 + if a == nil {
441 + return
442 + }
443 + a.mu.Lock()
444 + samples := a.samples
445 + a.samples = nil
446 + a.mu.Unlock()
447 + if len(samples) == 0 || a.metrics == nil {
448 + return
449 + }
450 + a.metrics.observeBuildRollup(samples)
451 +}
452 +
453 +func (m *runtimeMetrics) observeBuildRollup(samples []PlanRuntimeSample) {
454 + if m == nil || len(samples) == 0 {
455 + return
456 + }
457 +
458 + var buildSuccess, skippedFailed, buildErr int
459 + var routeCacheHits, routeCacheMisses uint64
460 + var routeCacheRetained, routeCachePruned, routeCacheFullDrops int
461 + var seriesScanned, seriesMatched, seriesUnmatched, seriesAutogenMatched uint64
462 + var seriesFilteredBySeq, seriesFilteredBySel uint64
463 + var actionCreateChart, actionCreateDimension, actionUpdateChart, actionRemoveDimension, actionRemoveChart int
464 + var lifecycleRemovedChartByCap, lifecycleRemovedChartByExpiry int
465 + var lifecycleRemovedDimensionByCap, lifecycleRemovedDimensionByExpiry int
466 + var routeCacheEntries, planChartInstances, planInferredDimensions int
467 + var buildSeqBroken, buildSeqRecovered int
468 + var buildSeqViolation, buildSeqObserved bool
469 +
470 + for _, sample := range samples {
471 + if sample.startedAt.IsZero() {
472 + sample.startedAt = time.Now()
473 + }
474 + switch {
475 + case sample.buildSuccess:
476 + buildSuccess++
477 + case sample.skippedFailed:
478 + skippedFailed++
479 + case sample.buildErr:
480 + buildErr++
481 + }
482 +
483 + m.buildDurationSeconds.Observe(time.Since(sample.startedAt).Seconds())
484 + observeDurationSeconds(m.buildPhasePrepareSec, sample.phasePrepareSeconds)
485 + observeDurationSeconds(m.buildPhaseValidateSec, sample.phaseValidateSeconds)
486 + observeDurationSeconds(m.buildPhaseScanSec, sample.phaseScanSeconds)
487 + observeDurationSeconds(m.buildPhaseRetainSec, sample.phaseRetainSeconds)
488 + observeDurationSeconds(m.buildPhaseCapsSec, sample.phaseLifecycleCapsSec)
489 + observeDurationSeconds(m.buildPhaseMaterializeSec, sample.phaseMaterializeSeconds)
490 + observeDurationSeconds(m.buildPhaseExpirySec, sample.phaseExpirySeconds)
491 + observeDurationSeconds(m.buildPhaseSortSec, sample.phaseSortSeconds)
492 +
493 + if sample.buildSeqObserved {
494 + buildSeqObserved = true
495 + if sample.buildSeqBroken {
496 + buildSeqBroken++
497 + }
498 + if sample.buildSeqRecovered {
499 + buildSeqRecovered++
500 + }
501 + if sample.buildSeqViolation {
502 + buildSeqViolation = true
503 + }
504 + }
505 +
506 + routeCacheHits += sample.routeCacheHits
507 + routeCacheMisses += sample.routeCacheMisses
508 + routeCacheRetained += sample.routeCacheRetained
509 + routeCachePruned += sample.routeCachePruned
510 + if sample.routeCacheFullDrop {
511 + routeCacheFullDrops++
512 + }
513 + if sample.buildSuccess {
514 + routeCacheEntries += sample.routeCacheEntries
515 + planChartInstances += sample.planChartInstances
516 + planInferredDimensions += sample.planInferredDimensions
517 + }
518 +
519 + seriesScanned += sample.seriesScanned
520 + seriesMatched += sample.seriesMatched
521 + seriesUnmatched += sample.seriesUnmatched
522 + seriesAutogenMatched += sample.seriesAutogenMatched
523 + seriesFilteredBySeq += sample.seriesFilteredBySeq
524 + seriesFilteredBySel += sample.seriesFilteredBySel
525 +
526 + actionCreateChart += sample.actionCreateChart
527 + actionCreateDimension += sample.actionCreateDimension
528 + actionUpdateChart += sample.actionUpdateChart
529 + actionRemoveDimension += sample.actionRemoveDimension
530 + actionRemoveChart += sample.actionRemoveChart
531 +
532 + lifecycleRemovedChartByCap += sample.lifecycleRemovedChartByCap
533 + lifecycleRemovedChartByExpiry += sample.lifecycleRemovedChartByExpiry
534 + lifecycleRemovedDimensionByCap += sample.lifecycleRemovedDimensionByCap
535 + lifecycleRemovedDimensionByExpiry += sample.lifecycleRemovedDimensionByExpiry
536 + }
537 +
538 + addIfPositive(m.buildSuccessTotal, float64(buildSuccess))
539 + addIfPositive(m.buildSkippedFailedTotal, float64(skippedFailed))
540 + addIfPositive(m.buildErrorTotal, float64(buildErr))
541 + addIfPositive(m.buildSeqBrokenTotal, float64(buildSeqBroken))
542 + addIfPositive(m.buildSeqRecoveredTotal, float64(buildSeqRecovered))
543 + if buildSeqObserved {
544 + if buildSeqViolation {
545 + m.buildSeqViolation.Set(1)
546 + } else {
547 + m.buildSeqViolation.Set(0)
548 + }
549 + } else {
550 + m.buildSeqViolation.Set(0)
551 + }
552 +
553 + addIfPositive(m.routeCacheHitsTotal, float64(routeCacheHits))
554 + addIfPositive(m.routeCacheMissesTotal, float64(routeCacheMisses))
555 + addIfPositive(m.routeCacheRetainedTotal, float64(routeCacheRetained))
556 + addIfPositive(m.routeCachePrunedTotal, float64(routeCachePruned))
557 + addIfPositive(m.routeCacheFullDropsTotal, float64(routeCacheFullDrops))
558 + if buildSuccess > 0 {
559 + m.routeCacheEntries.Set(float64(routeCacheEntries))
560 + }
561 +
562 + addIfPositive(m.seriesScannedTotal, float64(seriesScanned))
563 + addIfPositive(m.seriesMatchedTotal, float64(seriesMatched))
564 + addIfPositive(m.seriesUnmatchedTotal, float64(seriesUnmatched))
565 + addIfPositive(m.seriesAutogenMatchedTotal, float64(seriesAutogenMatched))
566 + addIfPositive(m.seriesFilteredBySeq, float64(seriesFilteredBySeq))
567 + addIfPositive(m.seriesFilteredBySelector, float64(seriesFilteredBySel))
568 +
569 + addIfPositive(m.actionCreateChart, float64(actionCreateChart))
570 + addIfPositive(m.actionCreateDimension, float64(actionCreateDimension))
571 + addIfPositive(m.actionUpdateChart, float64(actionUpdateChart))
572 + addIfPositive(m.actionRemoveDimension, float64(actionRemoveDimension))
573 + addIfPositive(m.actionRemoveChart, float64(actionRemoveChart))
574 +
575 + addIfPositive(m.lifecycleRemovedChartByCap, float64(lifecycleRemovedChartByCap))
576 + addIfPositive(m.lifecycleRemovedChartByExpiry, float64(lifecycleRemovedChartByExpiry))
577 + addIfPositive(m.lifecycleRemovedDimensionByCap, float64(lifecycleRemovedDimensionByCap))
578 + addIfPositive(m.lifecycleRemovedDimensionByExpiry, float64(lifecycleRemovedDimensionByExpiry))
579 +
580 + if buildSuccess > 0 {
581 + m.planChartInstances.Set(float64(planChartInstances))
582 + m.planInferredDimensions.Set(float64(planInferredDimensions))
583 + }
584 +}
585 +
586 +func addIfPositive(metric metrix.StatefulCounter, value float64) {
587 + if value > 0 {
588 + metric.Add(value)
589 + }
590 +}
591 +
592 +func actionKindCounts(actions []EngineAction) PlanRuntimeSample {
593 + out := PlanRuntimeSample{}
594 for _, action := range actions {
595 switch action.Kind() {
596 case ActionCreateChart:
src/go/plugin/framework/chartengine/runtime_metrics_test.go
+143
@@ -4,6 +4,7 @@ package chartengine
4
5 import (
6 "testing"
7 + "time"
8
9 "github.com/stretchr/testify/assert"
10 "github.com/stretchr/testify/require"
@@ -244,6 +245,144 @@ func TestEngineRuntimeObservabilityScenarios(t *testing.T) {
245 assert.Equal(t, "seconds", durationSum.Meta.Units)
246 },
247 },
248 + "runtime aggregator rolls up multiple engine samples": {
249 + run: func(t *testing.T) {
250 + store := metrix.NewRuntimeStore()
251 + agg := NewRuntimeAggregator(store)
252 + now := testNow()
253 + agg.Observe(PlanRuntimeSample{
254 + startedAt: now,
255 + buildSuccess: true,
256 + planRouteStats: planRouteStats{routeCacheHits: 2, seriesScanned: 3},
257 + routeCacheEntries: 4,
258 + planChartInstances: 5,
259 + planInferredDimensions: 6,
260 + buildSeqObserved: true,
261 + buildSeqViolation: true,
262 + })
263 + agg.Observe(PlanRuntimeSample{
264 + startedAt: now,
265 + buildSuccess: true,
266 + planRouteStats: planRouteStats{routeCacheHits: 7, seriesScanned: 11},
267 + routeCacheEntries: 13,
268 + planChartInstances: 17,
269 + planInferredDimensions: 19,
270 + buildSeqObserved: true,
271 + })
272 +
273 + agg.Flush()
274 +
275 + r := store.Read(metrix.ReadRaw())
276 + assertMetricValueAtLeast(t, r, "netdata.go.plugin.framework.chartengine.build_success_total", nil, 2)
277 + assertMetricValueAtLeast(t, r, "netdata.go.plugin.framework.chartengine.route_cache_hits_total", nil, 9)
278 + assertMetricValueAtLeast(t, r, "netdata.go.plugin.framework.chartengine.series_scanned_total", nil, 14)
279 + routeCacheEntries, ok := r.Value("netdata.go.plugin.framework.chartengine.route_cache_entries", nil)
280 + require.True(t, ok)
281 + assert.Equal(t, float64(17), routeCacheEntries)
282 + planChartInstances, ok := r.Value("netdata.go.plugin.framework.chartengine.plan_chart_instances", nil)
283 + require.True(t, ok)
284 + assert.Equal(t, float64(22), planChartInstances)
285 + planInferredDimensions, ok := r.Value("netdata.go.plugin.framework.chartengine.plan_inferred_dimensions", nil)
286 + require.True(t, ok)
287 + assert.Equal(t, float64(25), planInferredDimensions)
288 + seqViolation, ok := r.Value("netdata.go.plugin.framework.chartengine.build_seq_violation_active", nil)
289 + require.True(t, ok)
290 + assert.Equal(t, float64(1), seqViolation)
291 + assertSummaryCountAtLeast(t, r, "netdata.go.plugin.framework.chartengine.build_duration_seconds", nil, 2)
292 + },
293 + },
294 + "runtime sample observer fires without direct runtime store": {
295 + run: func(t *testing.T) {
296 + observed := 0
297 + e, err := New(
298 + WithRuntimeStore(nil),
299 + WithRuntimeSampleObserver(func(PlanRuntimeSample) {
300 + observed++
301 + }),
302 + )
303 + require.NoError(t, err)
304 + require.NoError(t, e.LoadYAML([]byte(runtimeObservabilityTemplateYAML()), 1))
305 +
306 + store := metrix.NewCollectorStore()
307 + cc := mustCycleController(t, store)
308 + c := store.Write().SnapshotMeter("mysql").Counter("queries_total")
309 + cc.BeginCycle()
310 + c.ObserveTotal(10)
311 + cc.CommitCycleSuccess()
312 +
313 + _, err = buildPlan(e, store.Read(metrix.ReadFlatten()))
314 + require.NoError(t, err)
315 + assert.Equal(t, 1, observed)
316 + assert.Nil(t, e.RuntimeStore())
317 + },
318 + },
319 + "outstanding attempt does not fire runtime sample observer": {
320 + run: func(t *testing.T) {
321 + observed := 0
322 + e, err := New(
323 + WithRuntimeStore(nil),
324 + WithRuntimeSampleObserver(func(PlanRuntimeSample) {
325 + observed++
326 + }),
327 + )
328 + require.NoError(t, err)
329 + require.NoError(t, e.LoadYAML([]byte(runtimeObservabilityTemplateYAML()), 1))
330 +
331 + store := metrix.NewCollectorStore()
332 + cc := mustCycleController(t, store)
333 + c := store.Write().SnapshotMeter("mysql").Counter("queries_total")
334 + cc.BeginCycle()
335 + c.ObserveTotal(10)
336 + cc.CommitCycleSuccess()
337 + reader := store.Read(metrix.ReadFlatten())
338 +
339 + attempt, err := e.PreparePlan(reader)
340 + require.NoError(t, err)
341 + defer attempt.Abort()
342 + assert.Equal(t, 1, observed)
343 +
344 + _, err = e.PreparePlan(reader)
345 + require.ErrorIs(t, err, ErrOutstandingPlanAttempt)
346 + assert.Equal(t, 1, observed)
347 + },
348 + },
349 + "runtime aggregator nil store and empty flush are no-ops": {
350 + run: func(t *testing.T) {
351 + agg := NewRuntimeAggregator(nil)
352 + require.NotPanics(t, func() {
353 + agg.Flush()
354 + agg.Observe(PlanRuntimeSample{startedAt: testNow(), buildSuccess: true})
355 + agg.Reset()
356 + agg.Flush()
357 + })
358 + },
359 + },
360 + "runtime aggregator clears build sequence violation on skipped-only rollup": {
361 + run: func(t *testing.T) {
362 + store := metrix.NewRuntimeStore()
363 + agg := NewRuntimeAggregator(store)
364 +
365 + agg.Observe(PlanRuntimeSample{
366 + startedAt: testNow(),
367 + buildSuccess: true,
368 + buildSeqObserved: true,
369 + buildSeqViolation: true,
370 + })
371 + agg.Flush()
372 + r := store.Read(metrix.ReadRaw())
373 + seqViolation, ok := r.Value("netdata.go.plugin.framework.chartengine.build_seq_violation_active", nil)
374 + require.True(t, ok)
375 + assert.Equal(t, float64(1), seqViolation)
376 +
377 + agg.Observe(PlanRuntimeSample{startedAt: testNow(), skippedFailed: true})
378 + agg.Flush()
379 + r = store.Read(metrix.ReadRaw())
380 + seqViolation, ok = r.Value("netdata.go.plugin.framework.chartengine.build_seq_violation_active", nil)
381 + require.True(t, ok)
382 + assert.Equal(t, float64(0), seqViolation)
383 + assertMetricValueAtLeast(t, r, "netdata.go.plugin.framework.chartengine.build_skipped_failed_collect_total", nil, 1)
384 + },
385 + },
386 }
387
388 for name, tc := range tests {
@@ -251,6 +390,10 @@ func TestEngineRuntimeObservabilityScenarios(t *testing.T) {
390 }
391 }
392
393 +func testNow() time.Time {
394 + return time.Now().Add(-time.Second)
395 +}
396 +
397 func assertMetricValueAtLeast(t *testing.T, reader metrix.Reader, name string, labels metrix.Labels, min float64) {
398 t.Helper()
399 value, ok := reader.Value(name, labels)
src/go/plugin/framework/jobruntime/job_v2.go
+334 -62
@@ -5,6 +5,7 @@ package jobruntime
5 import (
6 "bytes"
7 "context"
8 + "errors"
9 "fmt"
10 "io"
11 "log/slog"
@@ -22,6 +23,7 @@ import (
23 "github.com/netdata/netdata/go/plugins/plugin/framework/collectorapi"
24 "github.com/netdata/netdata/go/plugins/plugin/framework/runtimecomp"
25 "github.com/netdata/netdata/go/plugins/plugin/framework/tickstate"
26 + "github.com/netdata/netdata/go/plugins/plugin/framework/vnoderegistry"
27 "github.com/netdata/netdata/go/plugins/plugin/framework/vnodes"
28 )
29
@@ -38,6 +40,7 @@ type JobV2Config struct {
40 AutoDetectEvery int
41 IsStock bool
42 Vnode vnodes.VirtualNode
43 + VnodeRegistry *vnoderegistry.Registry
44 FunctionOnly bool
45 RuntimeService runtimecomp.Service
46 }
@@ -47,6 +50,10 @@ func NewJobV2(cfg JobV2Config) *JobV2 {
50 if cfg.UpdateEvery <= 0 {
51 cfg.UpdateEvery = 1
52 }
53 + registry := cfg.VnodeRegistry
54 + if registry == nil {
55 + registry = vnoderegistry.New()
56 + }
57
58 j := &JobV2{
59 pluginName: cfg.PluginName,
@@ -67,6 +74,7 @@ func NewJobV2(cfg JobV2Config) *JobV2 {
74 buf: &buf,
75 api: netdataapi.New(&buf),
76 vnode: cfg.Vnode,
77 + vnodeRegistry: registry,
78 runtimeService: cfg.RuntimeService,
79 }
80 if j.out == nil {
@@ -105,9 +113,15 @@ type JobV2 struct {
113 initialized bool
114 panicked atomic.Bool
115
108 - store metrix.CollectorStore
109 - cycle metrix.CycleController
110 - engine *chartengine.Engine
116 + store metrix.CollectorStore
117 + cycle metrix.CycleController
118 +
119 + scopeStates map[string]*jobV2ScopeState
120 + chartTemplateYAML []byte
121 + chartTemplateRevision uint64
122 + engineOptions []chartengine.Option
123 + runtimeStore metrix.RuntimeStore
124 + runtimeAggregator *chartengine.RuntimeAggregator
125
126 prevRun time.Time
127 retries atomic.Int64
@@ -116,7 +130,7 @@ type JobV2 struct {
130 vnode vnodes.VirtualNode
131 updVnode chan *vnodes.VirtualNode
132
119 - hostState jobV2HostState
133 + vnodeRegistry *vnoderegistry.Registry
134
135 ctxMu sync.RWMutex
136 runCtx context.Context
@@ -137,9 +151,24 @@ type JobV2 struct {
151 }
152
153 type jobV2PreparedEmission struct {
154 + scopes []jobV2PreparedScopeEmission
155 + scopeFailure bool
156 +}
157 +
158 +type jobV2PreparedScopeEmission struct {
159 + scope *jobV2ScopeState
160 attempt chartengine.PlanAttempt
161 plan chartengine.Plan
162 decision jobV2EmissionDecision
163 + output []byte
164 + live bool
165 +}
166 +
167 +type jobV2ScopeState struct {
168 + scopeKey string
169 + scope metrix.HostScope
170 + engine *chartengine.Engine
171 + host jobV2HostState
172 }
173
174 func (j *JobV2) FullName() string { return j.fullName }
@@ -179,37 +208,44 @@ func (j *JobV2) UpdateVnode(vnode *vnodes.VirtualNode) {
208 }
209 func (j *JobV2) Cleanup() {
210 j.buf.Reset()
182 - snapshot := j.hostState.captureCleanupSnapshot(j.currentVnode())
211 + snapshots := j.captureScopeCleanupSnapshots()
212 j.unregisterRuntimeComponent()
213 if j.module != nil {
214 j.module.Cleanup(context.Background())
215 }
216 if !collectorapi.ShouldObsoleteCharts() {
188 - return
189 - }
190 - if snapshot.staleVnodeSuppressed || len(snapshot.charts) == 0 {
217 + j.releaseAllScopeRegistryOwners()
218 + j.clearAllScopeStateAfterCleanup()
219 return
220 }
221
194 - env := chartemit.EmitEnv{
195 - TypeID: j.fullName,
196 - UpdateEvery: j.updateEvery,
197 - Plugin: j.pluginName,
198 - Module: j.moduleName,
199 - JobName: j.name,
200 - JobLabels: j.labels,
201 - }
202 - if snapshot.host.isVnode() {
203 - env.HostScope = &chartemit.HostScope{GUID: snapshot.host.guid}
204 - }
205 - if err := chartemit.ApplyPlan(j.api, buildJobV2CleanupPlan(snapshot.charts), env); err != nil {
206 - j.Warningf("cleanup apply plan failed: %v", err)
222 + for _, snapshot := range snapshots {
223 + if snapshot.staleVnodeSuppressed || len(snapshot.charts) == 0 {
224 + continue
225 + }
226 +
227 + env := chartemit.EmitEnv{
228 + TypeID: j.fullName,
229 + UpdateEvery: j.updateEvery,
230 + Plugin: j.pluginName,
231 + Module: j.moduleName,
232 + JobName: j.name,
233 + JobLabels: j.labels,
234 + }
235 + if snapshot.host.isVnode() {
236 + env.HostScope = &chartemit.HostScope{GUID: snapshot.host.guid}
237 + }
238 + j.buf.Reset()
239 + if err := chartemit.ApplyPlan(j.api, buildJobV2CleanupPlan(snapshot.charts), env); err != nil {
240 + j.Warningf("cleanup apply plan failed for host scope %q: %v", snapshot.scopeKey, err)
241 + j.buf.Reset()
242 + continue
243 + }
244 + _, _ = io.Copy(j.out, j.buf)
245 j.buf.Reset()
208 - return
246 }
210 - _, _ = io.Copy(j.out, j.buf)
211 - j.buf.Reset()
212 - j.hostState.clearAfterCleanup()
247 + j.releaseAllScopeRegistryOwners()
248 + j.clearAllScopeStateAfterCleanup()
249 }
250
251 func (j *JobV2) AutoDetection() (err error) {
@@ -343,25 +379,38 @@ func (j *JobV2) postCheck() error {
379 opts = append(opts, chartengine.WithEnginePolicy(v.EnginePolicy()))
380 }
381
346 - engine, err := chartengine.New(opts...)
347 - if err != nil {
348 - return err
349 - }
350 - if err := engine.LoadYAML([]byte(j.module.ChartTemplateYAML()), 1); err != nil {
382 + templateYAML := []byte(j.module.ChartTemplateYAML())
383 + if err := validateJobV2ChartTemplate(templateYAML, opts); err != nil {
384 return err
385 }
386
387 j.store = store
388 j.cycle = managed.CycleController()
356 - j.engine = engine
389 + j.scopeStates = make(map[string]*jobV2ScopeState)
390 + j.chartTemplateYAML = templateYAML
391 + j.chartTemplateRevision = 1
392 + j.engineOptions = opts
393 + j.runtimeStore = metrix.NewRuntimeStore()
394 + j.runtimeAggregator = chartengine.NewRuntimeAggregator(j.runtimeStore)
395 if err := j.registerRuntimeComponent(); err != nil {
396 j.Warningf("runtime metrics registration failed: %v", err)
397 }
398 return nil
399 }
400
401 +func validateJobV2ChartTemplate(templateYAML []byte, opts []chartengine.Option) error {
402 + engineOpts := append([]chartengine.Option{}, opts...)
403 + engineOpts = append(engineOpts, chartengine.WithRuntimeStore(nil))
404 + engine, err := chartengine.New(engineOpts...)
405 + if err != nil {
406 + return err
407 + }
408 + return engine.LoadYAML(templateYAML, 1)
409 +}
410 +
411 func (j *JobV2) runOnce() {
412 defer j.ResetAllOnce()
413 + defer j.flushRuntimeAggregator()
414
415 j.applyPendingVnodeUpdate()
416
@@ -384,6 +433,12 @@ func (j *JobV2) runOnce() {
433 j.buf.Reset()
434 }
435
436 +func (j *JobV2) flushRuntimeAggregator() {
437 + if j != nil && j.runtimeAggregator != nil {
438 + j.runtimeAggregator.Flush()
439 + }
440 +}
441 +
442 func (j *JobV2) applyPendingVnodeUpdate() {
443 select {
444 case vnode := <-j.updVnode:
@@ -392,6 +447,7 @@ func (j *JobV2) applyPendingVnodeUpdate() {
447 }
448 if j.module != nil && j.module.VirtualNode() != nil {
449 // Match v1 ownership model: do not override module-owned vnode state.
450 + j.Debugf("ignoring vnode update for module-owned vnode")
451 return
452 }
453
@@ -400,7 +456,11 @@ func (j *JobV2) applyPendingVnodeUpdate() {
456 j.vnodeMu.Lock()
457 j.vnode = *next
458 j.vnodeMu.Unlock()
403 - j.hostState.invalidateDefine()
459 + // Registry owner release is intentionally tied to the next successful
460 + // emission or cleanup, so obsolete emission can still select the old host.
461 + if state := j.scopeStates[defaultHostScopeKey]; state != nil {
462 + state.host.invalidateDefine()
463 + }
464 default:
465 }
466 }
@@ -408,11 +468,14 @@ func (j *JobV2) applyPendingVnodeUpdate() {
468 func (j *JobV2) collectAndEmit(sinceLastRun int) (prepared jobV2PreparedEmission, ok bool) {
469 j.panicked.Store(false)
470 cycleOpen := false
411 - var attempt chartengine.PlanAttempt
412 - attemptPending := false
471
472 defer func() {
473 if r := recover(); r != nil {
474 + j.rollbackPreparedEmission(prepared)
475 + j.buf.Reset()
476 + if j.runtimeAggregator != nil {
477 + j.runtimeAggregator.Reset()
478 + }
479 if cycleOpen {
480 // Recover path must close staged frame to keep subsequent cycles valid.
481 func() {
@@ -420,9 +483,7 @@ func (j *JobV2) collectAndEmit(sinceLastRun int) (prepared jobV2PreparedEmission
483 j.cycle.AbortCycle()
484 }()
485 }
423 - if attemptPending {
424 - attempt.Abort()
425 - }
486 + j.abortPreparedEmission(prepared)
487 j.panicked.Store(true)
488 j.Errorf("PANIC: %v", r)
489 if logger.Level.Enabled(slog.LevelDebug) {
@@ -439,50 +500,175 @@ func (j *JobV2) collectAndEmit(sinceLastRun int) (prepared jobV2PreparedEmission
500 j.Warningf("collect failed: %v", err)
501 return jobV2PreparedEmission{}, false
502 }
442 - j.cycle.CommitCycleSuccess()
503 + if err := j.cycle.CommitCycleSuccess(); err != nil {
504 + cycleOpen = false
505 + j.Warningf("commit cycle failed: %v", err)
506 + return jobV2PreparedEmission{}, false
507 + }
508 cycleOpen = false
509
445 - vnode := j.currentVnode()
446 - decision, err := j.hostState.prepareEmission(vnode)
510 + liveSet := j.liveScopeSet()
511 + workSet := j.scopeWorkSet(liveSet)
512 + for _, scopeKey := range sortedScopeKeys(workSet) {
513 + scope := workSet[scopeKey]
514 + _, live := liveSet[scopeKey]
515 + if !live {
516 + if state := j.scopeStates[scopeKey]; state != nil {
517 + scope = state.scope
518 + }
519 + }
520 + scopePrepared, scopeOK := j.prepareScopeEmission(scope, live, sinceLastRun)
521 + if !scopeOK {
522 + prepared.scopeFailure = true
523 + continue
524 + }
525 + prepared.scopes = append(prepared.scopes, scopePrepared)
526 + }
527 + j.Debugf("v2 scope count: %d", len(j.scopeStates))
528 + if len(prepared.scopes) == 0 && prepared.scopeFailure {
529 + return prepared, false
530 + }
531 + return prepared, true
532 +}
533 +
534 +func (j *JobV2) finishPreparedEmission(prepared jobV2PreparedEmission) error {
535 + successes := 0
536 + failures := 0
537 + var finalErr error
538 + for _, scope := range prepared.scopes {
539 + if err := scope.attempt.Commit(); err != nil {
540 + j.rollbackVnodeRegistryEmission(scope.decision)
541 + failures++
542 + finalErr = errors.Join(finalErr, err)
543 + j.Warningf("finalize emission for host scope %q failed: %v", scope.scope.scopeKey, err)
544 + continue
545 + }
546 + if len(scope.output) > 0 {
547 + _, _ = j.out.Write(scope.output)
548 + }
549 + j.commitScopeEmission(scope)
550 + successes++
551 + }
552 + if prepared.scopeFailure {
553 + failures++
554 + }
555 + if successes == 0 && failures > 0 {
556 + if finalErr != nil {
557 + return finalErr
558 + }
559 + return fmt.Errorf("all host scope emissions failed")
560 + }
561 + return nil
562 +}
563 +
564 +func (j *JobV2) prepareScopeEmission(scope metrix.HostScope, live bool, sinceLastRun int) (prepared jobV2PreparedScopeEmission, ok bool) {
565 + var attempt chartengine.PlanAttempt
566 + var decision jobV2EmissionDecision
567 + defer func() {
568 + if r := recover(); r != nil {
569 + j.rollbackVnodeRegistryEmission(decision)
570 + attempt.Abort()
571 + j.buf.Reset()
572 + panic(r)
573 + }
574 + if !ok {
575 + j.rollbackVnodeRegistryEmission(decision)
576 + attempt.Abort()
577 + j.buf.Reset()
578 + }
579 + }()
580 +
581 + state, err := j.ensureScopeState(scope)
582 if err != nil {
448 - j.Warningf("prepare host state failed: %v", err)
449 - return jobV2PreparedEmission{}, false
583 + j.Warningf("prepare host scope %q failed: %v", scope.ScopeKey, err)
584 + return jobV2PreparedScopeEmission{}, false
585 }
451 - if decision.needEngineReload {
452 - j.engine.ResetMaterialized()
453 - j.hostState.onEngineReload(decision.targetHost)
586 +
587 + if state.scopeKey == defaultHostScopeKey {
588 + vnode := j.currentVnode()
589 + decision, err = state.host.prepareEmission(vnode)
590 + if err == nil && decision.needEngineReload {
591 + state.engine.ResetMaterialized()
592 + }
593 + if err != nil {
594 + j.Warningf("prepare default host scope failed: %v", err)
595 + return jobV2PreparedScopeEmission{}, false
596 + }
597 + } else {
598 + decision, err = state.host.prepareScopedEmission(state.scope)
599 + if err == nil && decision.needEngineReload {
600 + state.engine.ResetMaterialized()
601 + }
602 + if err != nil {
603 + j.Warningf("prepare host scope %q failed: %v", state.scopeKey, err)
604 + return jobV2PreparedScopeEmission{}, false
605 + }
606 }
455 - attempt, err = j.engine.PreparePlan(j.store.Read(metrix.ReadRaw(), metrix.ReadFlatten()))
607 +
608 + attempt, err = state.engine.PreparePlan(j.store.Read(metrix.ReadRaw(), metrix.ReadFlatten(), metrix.ReadHostScope(state.scopeKey)))
609 if err != nil {
457 - j.Warningf("build plan failed: %v", err)
458 - return jobV2PreparedEmission{}, false
610 + j.Warningf("build plan for host scope %q failed: %v", state.scopeKey, err)
611 + return jobV2PreparedScopeEmission{}, false
612 }
460 - attemptPending = true
613 plan := attempt.Plan()
614 + if err := j.prepareScopeVnodeRegistryEmission(state, &decision, plan); err != nil {
615 + j.Warningf("prepare vnode registry for host scope %q failed: %v", state.scopeKey, err)
616 + return jobV2PreparedScopeEmission{}, false
617 + }
618
619 + j.buf.Reset()
620 env := j.emitEnv(sinceLastRun, decision)
621 if err := chartemit.ApplyPlan(j.api, plan, env); err != nil {
465 - attempt.Abort()
466 - attemptPending = false
467 - j.Warningf("apply plan failed: %v", err)
468 - return jobV2PreparedEmission{}, false
622 + j.Warningf("apply plan for host scope %q failed: %v", state.scopeKey, err)
623 + return jobV2PreparedScopeEmission{}, false
624 }
470 - return jobV2PreparedEmission{
625 + output := append([]byte(nil), j.buf.Bytes()...)
626 + j.buf.Reset()
627 +
628 + prepared = jobV2PreparedScopeEmission{
629 + scope: state,
630 attempt: attempt,
631 plan: plan,
632 decision: decision,
474 - }, true
633 + output: output,
634 + live: live,
635 + }
636 + return prepared, true
637 }
638
477 -func (j *JobV2) finishPreparedEmission(prepared jobV2PreparedEmission) error {
478 - if j.buf.Len() > 0 {
479 - _, _ = io.Copy(j.out, j.buf)
639 +func (j *JobV2) commitScopeEmission(prepared jobV2PreparedScopeEmission) {
640 + if prepared.scope == nil {
641 + return
642 }
481 - if err := prepared.attempt.Commit(); err != nil {
482 - return err
643 + state := prepared.scope
644 + if state.scopeKey == defaultHostScopeKey || prepared.decision.registryOwner != "" {
645 + keep := make(map[vnoderegistry.Owner]struct{}, 1)
646 + if prepared.decision.registryOwner != "" {
647 + keep[prepared.decision.registryOwner] = struct{}{}
648 + }
649 + state.host.releaseSupersededRegistryOwnersExcept(
650 + j.vnodeRegistry,
651 + keep,
652 + j.vnodeRegistryOwnerNamespacePrefix(state.scopeKey),
653 + )
654 + }
655 + state.host.commitSuccessfulEmission(prepared.plan, prepared.decision)
656 + if !prepared.live && len(state.host.cleanupCharts) == 0 {
657 + state.host.releaseRegistryOwners(j.vnodeRegistry)
658 + delete(j.scopeStates, state.scopeKey)
659 + }
660 +}
661 +
662 +func (j *JobV2) rollbackPreparedEmission(prepared jobV2PreparedEmission) {
663 + for _, scope := range prepared.scopes {
664 + j.rollbackVnodeRegistryEmission(scope.decision)
665 + }
666 +}
667 +
668 +func (j *JobV2) abortPreparedEmission(prepared jobV2PreparedEmission) {
669 + for _, scope := range prepared.scopes {
670 + scope.attempt.Abort()
671 }
484 - j.hostState.commitSuccessfulEmission(prepared.plan, prepared.decision)
485 - return nil
672 }
673
674 func (j *JobV2) emitEnv(sinceLastRun int, decision jobV2EmissionDecision) chartemit.EmitEnv {
@@ -510,6 +696,92 @@ func (j *JobV2) currentVnode() vnodes.VirtualNode {
696 return *j.vnode.Copy()
697 }
698
699 +func (j *JobV2) prepareScopeVnodeRegistryEmission(state *jobV2ScopeState, decision *jobV2EmissionDecision, plan chartengine.Plan) error {
700 + if decision == nil || !decision.targetHost.isVnode() || len(plan.Actions) == 0 {
701 + return nil
702 + }
703 + if state == nil {
704 + return fmt.Errorf("nil host scope state")
705 + }
706 + if state.scopeKey == defaultHostScopeKey {
707 + vnode := j.currentVnode()
708 + return j.prepareVnodeRegistryEmission(decision, j.vnodeRegistryOwner(decision.targetHost), netdataapi.HostInfo{
709 + GUID: vnode.GUID,
710 + Hostname: vnode.Hostname,
711 + Labels: vnode.Labels,
712 + })
713 + }
714 + return j.prepareVnodeRegistryEmission(decision, j.vnodeRegistryScopedOwner(state.scopeKey, state.scope.GUID), metrixHostScopeInfo(state.scope))
715 +}
716 +
717 +func (j *JobV2) prepareVnodeRegistryEmission(decision *jobV2EmissionDecision, owner vnoderegistry.Owner, info netdataapi.HostInfo) error {
718 + registryInfo := netdataapi.HostInfo{
719 + GUID: info.GUID,
720 + Hostname: info.Hostname,
721 + Labels: maps.Clone(info.Labels),
722 + }
723 + result, err := j.vnodeRegistry.Register(owner, registryInfo)
724 + if err != nil {
725 + return err
726 + }
727 + if result.MetadataUpdated && result.UpdateFirstSeen {
728 + j.Warningf(
729 + "vnode registry metadata updated for guid %q: hostname %q replaced by %q",
730 + result.Info.GUID,
731 + result.Previous.Hostname,
732 + result.Info.Hostname,
733 + )
734 + }
735 +
736 + scope := &chartemit.HostScope{GUID: decision.targetHost.guid}
737 + if result.NeedDefine {
738 + scope.Define = &result.Info
739 + }
740 + decision.hostScope = scope
741 + decision.defineInfo = result.Info
742 + decision.registryOwner = owner
743 + decision.registryRegistration = result
744 + return nil
745 +}
746 +
747 +func (j *JobV2) rollbackVnodeRegistryEmission(decision jobV2EmissionDecision) {
748 + if decision.registryOwner != "" {
749 + j.vnodeRegistry.Rollback(decision.registryOwner, decision.registryRegistration)
750 + }
751 +}
752 +
753 +const vnodeRegistryOwnerSeparator = "\xff"
754 +
755 +func (j *JobV2) vnodeRegistryOwnerPrefix() string {
756 + // Keep the separator outside valid metrix scope keys and GUIDs so owner
757 + // strings remain unambiguous without allocating a structured key.
758 + return j.fullName + vnodeRegistryOwnerSeparator
759 +}
760 +
761 +func (j *JobV2) vnodeRegistryJobOwnerPrefix() string {
762 + // Keep job-level vnode owners separate from future per-scope owners.
763 + return j.vnodeRegistryOwnerPrefix() + "job" + vnodeRegistryOwnerSeparator
764 +}
765 +
766 +func (j *JobV2) vnodeRegistryScopedOwnerPrefix(scopeKey string) string {
767 + return j.vnodeRegistryOwnerPrefix() + "scope" + vnodeRegistryOwnerSeparator + scopeKey + vnodeRegistryOwnerSeparator
768 +}
769 +
770 +func (j *JobV2) vnodeRegistryOwnerNamespacePrefix(scopeKey string) string {
771 + if scopeKey == defaultHostScopeKey {
772 + return j.vnodeRegistryJobOwnerPrefix()
773 + }
774 + return j.vnodeRegistryScopedOwnerPrefix(scopeKey)
775 +}
776 +
777 +func (j *JobV2) vnodeRegistryOwner(target jobV2HostRef) vnoderegistry.Owner {
778 + return vnoderegistry.Owner(j.vnodeRegistryJobOwnerPrefix() + target.guid)
779 +}
780 +
781 +func (j *JobV2) vnodeRegistryScopedOwner(scopeKey, guid string) vnoderegistry.Owner {
782 + return vnoderegistry.Owner(j.vnodeRegistryScopedOwnerPrefix(scopeKey) + guid)
783 +}
784 +
785 func (j *JobV2) penalty() int {
786 return penaltyFromRetries(int(j.retries.Load()), j.updateEvery)
787 }
src/go/plugin/framework/jobruntime/job_v2_cleanup.go
+46 -2
@@ -12,12 +12,13 @@ import (
12 )
13
14 type jobV2CleanupSnapshot struct {
15 + scopeKey string
16 charts map[string]chartengine.ChartMeta
17 host jobV2HostRef
18 staleVnodeSuppressed bool
19 }
20
20 -func (s *jobV2HostState) captureCleanupSnapshot(vnode vnodes.VirtualNode) jobV2CleanupSnapshot {
21 +func (s *jobV2HostState) captureCleanupSnapshot(vnode vnodes.VirtualNode, allowStaleVnodeSuppression bool) jobV2CleanupSnapshot {
22 if s == nil {
23 return jobV2CleanupSnapshot{}
24 }
@@ -25,10 +26,53 @@ func (s *jobV2HostState) captureCleanupSnapshot(vnode vnodes.VirtualNode) jobV2C
26 return jobV2CleanupSnapshot{
27 charts: maps.Clone(s.cleanupCharts),
28 host: host,
28 - staleVnodeSuppressed: shouldSuppressCleanupForStaleVnode(host, vnode),
29 + staleVnodeSuppressed: allowStaleVnodeSuppression && shouldSuppressCleanupForStaleVnode(host, vnode),
30 }
31 }
32
33 +func (j *JobV2) captureScopeCleanupSnapshots() []jobV2CleanupSnapshot {
34 + if j == nil || len(j.scopeStates) == 0 {
35 + return nil
36 + }
37 + vnode := j.currentVnode()
38 + keys := sortedScopeStateKeys(j.scopeStates)
39 +
40 + snapshots := make([]jobV2CleanupSnapshot, 0, len(keys))
41 + for _, key := range keys {
42 + state := j.scopeStates[key]
43 + if state == nil {
44 + continue
45 + }
46 + snapshot := state.host.captureCleanupSnapshot(vnode, key == defaultHostScopeKey)
47 + snapshot.scopeKey = key
48 + snapshots = append(snapshots, snapshot)
49 + }
50 + return snapshots
51 +}
52 +
53 +func (j *JobV2) releaseAllScopeRegistryOwners() {
54 + if j == nil {
55 + return
56 + }
57 + for _, state := range j.scopeStates {
58 + if state != nil {
59 + state.host.releaseRegistryOwners(j.vnodeRegistry)
60 + }
61 + }
62 +}
63 +
64 +func (j *JobV2) clearAllScopeStateAfterCleanup() {
65 + if j == nil {
66 + return
67 + }
68 + for _, state := range j.scopeStates {
69 + if state != nil {
70 + state.host.clearAfterCleanup()
71 + }
72 + }
73 + clear(j.scopeStates)
74 +}
75 +
76 func (s *jobV2HostState) clearAfterCleanup() {
77 if s == nil {
78 return
src/go/plugin/framework/jobruntime/job_v2_host_state.go
+53 -37
@@ -5,10 +5,13 @@ package jobruntime
5 import (
6 "fmt"
7 "maps"
8 + "strings"
9
10 + "github.com/netdata/netdata/go/plugins/pkg/metrix"
11 "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
12 "github.com/netdata/netdata/go/plugins/plugin/framework/chartemit"
13 "github.com/netdata/netdata/go/plugins/plugin/framework/chartengine"
14 + "github.com/netdata/netdata/go/plugins/plugin/framework/vnoderegistry"
15 "github.com/netdata/netdata/go/plugins/plugin/framework/vnodes"
16 )
17
@@ -37,11 +40,12 @@ func (r jobV2HostRef) isGlobal() bool { return r.kind == jobV2HostGlobal }
40 func (r jobV2HostRef) isVnode() bool { return r.kind == jobV2HostVnode }
41
42 type jobV2EmissionDecision struct {
40 - targetHost jobV2HostRef
41 - needEngineReload bool
42 - hostScope *chartemit.HostScope
43 - defineEmitted bool
44 - defineInfo netdataapi.HostInfo
43 + targetHost jobV2HostRef
44 + needEngineReload bool
45 + hostScope *chartemit.HostScope
46 + defineInfo netdataapi.HostInfo
47 + registryOwner vnoderegistry.Owner
48 + registryRegistration vnoderegistry.Registration
49 }
50
51 type jobV2HostState struct {
@@ -50,6 +54,9 @@ type jobV2HostState struct {
54 engineHost jobV2HostRef
55 cleanupOwner jobV2HostRef
56 cleanupCharts map[string]chartengine.ChartMeta
57 + // registryOwners tracks successfully emitted vnode owners so cleanup can
58 + // release them after obsolete-chart emission.
59 + registryOwners map[vnoderegistry.Owner]string
60 }
61
62 func (s *jobV2HostState) invalidateDefine() {
@@ -70,51 +77,43 @@ func (s *jobV2HostState) prepareEmission(vnode vnodes.VirtualNode) (jobV2Emissio
77 return decision, nil
78 }
79
73 - info, needDefine, err := s.prepareDefine(vnode, target)
74 - if err != nil {
75 - return jobV2EmissionDecision{}, err
76 - }
80 scope := &chartemit.HostScope{GUID: target.guid}
78 - if needDefine {
79 - scope.Define = &info
80 - }
81 decision.hostScope = scope
82 - decision.defineEmitted = needDefine
83 - decision.defineInfo = info
82 return decision, nil
83 }
84
87 -func (s *jobV2HostState) prepareDefine(vnode vnodes.VirtualNode, target jobV2HostRef) (netdataapi.HostInfo, bool, error) {
88 - info, err := chartemit.PrepareHostInfo(netdataapi.HostInfo{
89 - GUID: vnode.GUID,
90 - Hostname: vnode.Hostname,
91 - Labels: vnode.Labels,
92 - })
93 - if err != nil {
94 - return netdataapi.HostInfo{}, false, err
95 - }
96 - if s != nil && s.definedHost == target && hostInfoEqual(s.definedInfo, info) {
97 - return netdataapi.HostInfo{}, false, nil
85 +func (s *jobV2HostState) prepareScopedEmission(scope metrix.HostScope) (jobV2EmissionDecision, error) {
86 + target := jobV2HostRef{kind: jobV2HostVnode, guid: scope.GUID}
87 + decision := jobV2EmissionDecision{
88 + targetHost: target,
89 + needEngineReload: s != nil && s.engineHost.isSet() && s.engineHost != target,
90 + hostScope: &chartemit.HostScope{GUID: target.guid},
91 }
99 - return info, true, nil
92 + return decision, nil
93 }
94
102 -func (s *jobV2HostState) onEngineReload(target jobV2HostRef) {
95 +func (s *jobV2HostState) commitSuccessfulEmission(plan chartengine.Plan, decision jobV2EmissionDecision) {
96 if s == nil {
97 return
98 }
106 - s.engineHost = target
107 -}
108 -
109 -func (s *jobV2HostState) commitSuccessfulEmission(plan chartengine.Plan, decision jobV2EmissionDecision) {
110 - if s == nil || len(plan.Actions) == 0 {
99 + if len(plan.Actions) == 0 {
100 + if decision.needEngineReload {
101 + // Quiet host switches still need to finish after the scope attempt commits.
102 + s.engineHost = decision.targetHost
103 + }
104 return
105 }
106 s.engineHost = decision.targetHost
114 - if decision.defineEmitted {
107 + if decision.hostScope != nil {
108 s.definedHost = decision.targetHost
109 s.definedInfo = decision.defineInfo
110 }
111 + if decision.registryOwner != "" {
112 + if s.registryOwners == nil {
113 + s.registryOwners = make(map[vnoderegistry.Owner]string)
114 + }
115 + s.registryOwners[decision.registryOwner] = decision.targetHost.guid
116 + }
117 if s.cleanupCharts == nil {
118 s.cleanupCharts = make(map[string]chartengine.ChartMeta)
119 }
@@ -153,10 +152,27 @@ func (s *jobV2HostState) commitSuccessfulEmission(plan chartengine.Plan, decisio
152 s.cleanupOwner = decision.targetHost
153 }
154
156 -func hostInfoEqual(left, right netdataapi.HostInfo) bool {
157 - return left.GUID == right.GUID &&
158 - left.Hostname == right.Hostname &&
159 - maps.Equal(left.Labels, right.Labels)
155 +func (s *jobV2HostState) releaseRegistryOwners(registry *vnoderegistry.Registry) {
156 + if s == nil || registry == nil || len(s.registryOwners) == 0 {
157 + return
158 + }
159 + for owner, guid := range s.registryOwners {
160 + registry.Release(owner, guid)
161 + delete(s.registryOwners, owner)
162 + }
163 +}
164 +
165 +func (s *jobV2HostState) releaseSupersededRegistryOwnersExcept(registry *vnoderegistry.Registry, current map[vnoderegistry.Owner]struct{}, ownerPrefix string) {
166 + if s == nil || registry == nil || len(s.registryOwners) == 0 {
167 + return
168 + }
169 + for owner, guid := range s.registryOwners {
170 + if _, ok := current[owner]; ok || !strings.HasPrefix(string(owner), ownerPrefix) {
171 + continue
172 + }
173 + registry.Release(owner, guid)
174 + delete(s.registryOwners, owner)
175 + }
176 }
177
178 func (r jobV2HostRef) String() string {
src/go/plugin/framework/jobruntime/job_v2_runtime.go
+1 -4
@@ -13,10 +13,7 @@ func (j *JobV2) registerRuntimeComponent() error {
13 if j == nil || j.runtimeService == nil || j.runtimeComponentRegistered {
14 return nil
15 }
16 - if j.engine == nil {
17 - return fmt.Errorf("nil engine")
18 - }
19 - store := j.engine.RuntimeStore()
16 + store := j.runtimeStore
17 if store == nil {
18 return fmt.Errorf("nil runtime store")
19 }
src/go/plugin/framework/jobruntime/job_v2_scope.go new
+133
@@ -0,0 +1,133 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package jobruntime
4 +
5 +import (
6 + "fmt"
7 + "maps"
8 + "sort"
9 +
10 + "github.com/netdata/netdata/go/plugins/pkg/metrix"
11 + "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
12 + "github.com/netdata/netdata/go/plugins/plugin/framework/chartengine"
13 +)
14 +
15 +const defaultHostScopeKey = ""
16 +
17 +func (j *JobV2) ensureScopeState(scope metrix.HostScope) (*jobV2ScopeState, error) {
18 + if j == nil {
19 + return nil, fmt.Errorf("nil job")
20 + }
21 + scopeKey := scope.ScopeKey
22 + if scope.IsDefault() {
23 + scope = metrix.HostScope{}
24 + scopeKey = defaultHostScopeKey
25 + }
26 + if j.scopeStates == nil {
27 + j.scopeStates = make(map[string]*jobV2ScopeState)
28 + }
29 + if state := j.scopeStates[scopeKey]; state != nil {
30 + state.scope = scope
31 + return state, nil
32 + }
33 +
34 + engine, err := j.newScopeEngine()
35 + if err != nil {
36 + return nil, err
37 + }
38 + state := &jobV2ScopeState{
39 + scopeKey: scopeKey,
40 + scope: scope,
41 + engine: engine,
42 + }
43 + j.scopeStates[scopeKey] = state
44 + return state, nil
45 +}
46 +
47 +func (j *JobV2) newScopeEngine() (*chartengine.Engine, error) {
48 + opts := append([]chartengine.Option{}, j.engineOptions...)
49 + opts = append(opts, chartengine.WithRuntimeStore(nil))
50 + if j.runtimeAggregator != nil {
51 + opts = append(opts, chartengine.WithRuntimeSampleObserver(j.runtimeAggregator.Observe))
52 + }
53 + engine, err := chartengine.New(opts...)
54 + if err != nil {
55 + return nil, err
56 + }
57 + if err := engine.LoadYAML(j.chartTemplateYAML, j.chartTemplateRevision); err != nil {
58 + return nil, err
59 + }
60 + return engine, nil
61 +}
62 +
63 +func (j *JobV2) liveScopeSet() map[string]metrix.HostScope {
64 + scopes := make(map[string]metrix.HostScope)
65 + reader := j.store.Read(metrix.ReadRaw(), metrix.ReadFlatten())
66 + for _, scope := range reader.HostScopes() {
67 + if j.scopeHasVisibleSeries(scope.ScopeKey) {
68 + scopes[scope.ScopeKey] = scope
69 + }
70 + }
71 + return scopes
72 +}
73 +
74 +func (j *JobV2) scopeWorkSet(liveScopes map[string]metrix.HostScope) map[string]metrix.HostScope {
75 + scopes := make(map[string]metrix.HostScope, len(liveScopes)+len(j.scopeStates))
76 + maps.Copy(scopes, liveScopes)
77 + // Retain previously emitted scopes until their engine emits lifecycle
78 + // removals. This includes default scope when unscoped series disappear.
79 + for key, state := range j.scopeStates {
80 + if _, ok := scopes[key]; ok {
81 + continue
82 + }
83 + scopes[key] = state.scope
84 + }
85 + return scopes
86 +}
87 +
88 +func (j *JobV2) scopeHasVisibleSeries(scopeKey string) bool {
89 + reader := j.store.Read(metrix.ReadFlatten(), metrix.ReadHostScope(scopeKey))
90 + found := false
91 + reader.ForEachSeries(func(string, metrix.LabelView, metrix.SampleValue) {
92 + found = true
93 + })
94 + return found
95 +}
96 +
97 +func sortedScopeKeys(scopes map[string]metrix.HostScope) []string {
98 + keys := make([]string, 0, len(scopes))
99 + for key := range scopes {
100 + keys = append(keys, key)
101 + }
102 + sortHostScopeKeys(keys)
103 + return keys
104 +}
105 +
106 +func sortedScopeStateKeys(scopes map[string]*jobV2ScopeState) []string {
107 + keys := make([]string, 0, len(scopes))
108 + for key := range scopes {
109 + keys = append(keys, key)
110 + }
111 + sortHostScopeKeys(keys)
112 + return keys
113 +}
114 +
115 +func sortHostScopeKeys(keys []string) {
116 + sort.Slice(keys, func(i, j int) bool {
117 + if keys[i] == defaultHostScopeKey {
118 + return true
119 + }
120 + if keys[j] == defaultHostScopeKey {
121 + return false
122 + }
123 + return keys[i] < keys[j]
124 + })
125 +}
126 +
127 +func metrixHostScopeInfo(scope metrix.HostScope) netdataapi.HostInfo {
128 + return netdataapi.HostInfo{
129 + GUID: scope.GUID,
130 + Hostname: scope.Hostname,
131 + Labels: scope.Labels,
132 + }
133 +}
src/go/plugin/framework/jobruntime/job_v2_test.go
+912 -44
@@ -7,6 +7,7 @@ import (
7 "context"
8 "errors"
9 "fmt"
10 + "strings"
11 "testing"
12 "time"
13
@@ -15,6 +16,7 @@ import (
16 "github.com/netdata/netdata/go/plugins/plugin/framework/chartengine"
17 "github.com/netdata/netdata/go/plugins/plugin/framework/collectorapi"
18 "github.com/netdata/netdata/go/plugins/plugin/framework/runtimecomp"
19 + "github.com/netdata/netdata/go/plugins/plugin/framework/vnoderegistry"
20 "github.com/netdata/netdata/go/plugins/plugin/framework/vnodes"
21 "github.com/stretchr/testify/assert"
22 "github.com/stretchr/testify/require"
@@ -43,6 +45,12 @@ type mockRuntimeComponentService struct {
45 unregistered []string
46 }
47
48 +type writeFunc func([]byte) (int, error)
49 +
50 +func (f writeFunc) Write(p []byte) (int, error) {
51 + return f(p)
52 +}
53 +
54 func (m *mockRuntimeComponentService) RegisterComponent(cfg runtimecomp.ComponentConfig) error {
55 if m.registerErr != nil {
56 return m.registerErr
@@ -128,6 +136,39 @@ func newTestJobV2WithVnode(mod collectorapi.CollectorV2, out *bytes.Buffer, vnod
136 })
137 }
138
139 +func newRegistryTestJobV2(t *testing.T, fullName string, registry *vnoderegistry.Registry, out *bytes.Buffer, vnode vnodes.VirtualNode) *JobV2 {
140 + t.Helper()
141 + store := metrix.NewCollectorStore()
142 + mod := &mockModuleV2{
143 + store: store,
144 + template: chartTemplateV2(),
145 + collectFunc: func(context.Context) error {
146 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(1)
147 + return nil
148 + },
149 + }
150 + job := NewJobV2(JobV2Config{
151 + PluginName: pluginName,
152 + Name: fullName,
153 + ModuleName: modName,
154 + FullName: fullName,
155 + Module: mod,
156 + Out: out,
157 + UpdateEvery: 1,
158 + Vnode: vnode,
159 + VnodeRegistry: registry,
160 + })
161 + require.NoError(t, job.AutoDetection())
162 + return job
163 +}
164 +
165 +func requireDefaultScopeState(t *testing.T, job *JobV2) *jobV2ScopeState {
166 + t.Helper()
167 + state := job.scopeStates[defaultHostScopeKey]
168 + require.NotNil(t, state)
169 + return state
170 +}
171 +
172 func chartTemplateV2() string {
173 return `
174 version: v1
@@ -146,6 +187,26 @@ groups:
187 `
188 }
189
190 +func chartTemplateV2ExpireAfterOne() string {
191 + return `
192 +version: v1
193 +groups:
194 + - family: Workers
195 + metrics:
196 + - apache.workers_busy
197 + charts:
198 + - id: workers_busy
199 + title: Workers Busy
200 + context: workers_busy
201 + units: workers
202 + lifecycle:
203 + expire_after_cycles: 1
204 + dimensions:
205 + - selector: apache.workers_busy
206 + name: busy
207 +`
208 +}
209 +
210 func chartTemplateV2Dynamic() string {
211 return `
212 version: v1
@@ -183,8 +244,10 @@ func TestJobV2Scenarios(t *testing.T) {
244 require.NoError(t, job.AutoDetection())
245 require.NotNil(t, job.store)
246 require.NotNil(t, job.cycle)
186 - require.NotNil(t, job.engine)
187 - attempt, err := job.engine.PreparePlan(job.store.Read(metrix.ReadFlatten()))
247 + state, err := job.ensureScopeState(metrix.HostScope{})
248 + require.NoError(t, err)
249 + require.NotNil(t, state.engine)
250 + attempt, err := state.engine.PreparePlan(job.store.Read(metrix.ReadFlatten()))
251 require.NoError(t, err)
252 defer attempt.Abort()
253 err = attempt.Commit()
@@ -370,7 +433,7 @@ END`, chartengine.Priority, chartengine.Priority))
433 assert.NotContains(t, cfg.JobLabels, "source")
434 assert.NotContains(t, cfg.JobLabels, "collector_module")
435 require.NotNil(t, cfg.Store)
373 - assert.Equal(t, job.engine.RuntimeStore(), cfg.Store)
436 + assert.Equal(t, job.runtimeStore, cfg.Store)
437 },
438 },
439 "module context carries runtime component service when available": {
@@ -922,7 +985,7 @@ BEGIN 'module_job.workers_busy'`,
985 require.NoError(t, err)
986
987 job.runOnce()
925 - require.Equal(t, initialInfo, job.hostState.definedInfo)
988 + require.Equal(t, initialInfo, requireDefaultScopeState(t, job).host.definedInfo)
989
990 out.Reset()
991 tc.mutate(modVnode)
@@ -941,40 +1004,797 @@ BEGIN 'module_job.workers_busy'`)
1004
1005 expectedInfo, err := chartemit.PrepareHostInfo(tc.wantInfo)
1006 require.NoError(t, err)
944 - assert.Equal(t, expectedInfo, job.hostState.definedInfo)
1007 + assert.Equal(t, expectedInfo, requireDefaultScopeState(t, job).host.definedInfo)
1008 })
1009 }
1010 }
1011
949 -func TestJobV2EmptyPlanDoesNotMarkVnodeDefined(t *testing.T) {
950 - store := metrix.NewCollectorStore()
951 - emitValue := false
952 - mod := &mockModuleV2{
953 - store: store,
954 - template: chartTemplateV2(),
955 - collectFunc: func(context.Context) error {
956 - if emitValue {
957 - store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(1)
958 - }
959 - return nil
1012 +func TestJobV2VnodeRegistryScenarios(t *testing.T) {
1013 + cases := map[string]struct {
1014 + run func(t *testing.T)
1015 + }{
1016 + "shared registry suppresses duplicate and updates changed metadata": {
1017 + run: func(t *testing.T) {
1018 + registry := vnoderegistry.New()
1019 + jobAOut := &bytes.Buffer{}
1020 + jobBOut := &bytes.Buffer{}
1021 +
1022 + jobA := newRegistryTestJobV2(t, "module_job_a", registry, jobAOut, vnodes.VirtualNode{
1023 + Hostname: "node-host-a",
1024 + GUID: "node-guid",
1025 + Labels: map[string]string{
1026 + "region": "eu",
1027 + },
1028 + })
1029 + jobB := newRegistryTestJobV2(t, "module_job_b", registry, jobBOut, vnodes.VirtualNode{
1030 + Hostname: "node-host-b",
1031 + GUID: "node-guid",
1032 + Labels: map[string]string{
1033 + "region": "us",
1034 + },
1035 + })
1036 +
1037 + jobA.runOnce()
1038 + assert.Contains(t, jobAOut.String(), `HOST_DEFINE 'node-guid' 'node-host-a'`)
1039 + assert.Contains(t, jobAOut.String(), `HOST 'node-guid'`)
1040 +
1041 + jobB.runOnce()
1042 + assert.Contains(t, jobBOut.String(), `HOST_DEFINE 'node-guid' 'node-host-b'`)
1043 + assert.Contains(t, jobBOut.String(), `HOST 'node-guid'`)
1044 +
1045 + info, ok := registry.Lookup("node-guid")
1046 + require.True(t, ok)
1047 + assert.Equal(t, "node-host-b", info.Hostname)
1048 +
1049 + jobAOut.Reset()
1050 + jobA.runOnce()
1051 + assert.Contains(t, jobAOut.String(), `HOST_DEFINE 'node-guid' 'node-host-a'`)
1052 + info, ok = registry.Lookup("node-guid")
1053 + require.True(t, ok)
1054 + assert.Equal(t, "node-host-a", info.Hostname)
1055 +
1056 + assert.Equal(t, []vnoderegistry.Owner{
1057 + vnoderegistry.Owner("module_job_a\xffjob\xffnode-guid"),
1058 + vnoderegistry.Owner("module_job_b\xffjob\xffnode-guid"),
1059 + }, registry.Owners("node-guid"))
1060 +
1061 + jobA.Cleanup()
1062 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job_b\xffjob\xffnode-guid")}, registry.Owners("node-guid"))
1063 + jobB.Cleanup()
1064 + assert.Equal(t, 0, registry.Len())
1065 + },
1066 + },
1067 + "rollback on apply failure": {
1068 + run: func(t *testing.T) {
1069 + registry := vnoderegistry.New()
1070 + _, err := registry.Register("other", netdataapi.HostInfo{
1071 + GUID: "node-guid",
1072 + Hostname: "node-host-a",
1073 + })
1074 + require.NoError(t, err)
1075 +
1076 + store := metrix.NewCollectorStore()
1077 + mod := &mockModuleV2{
1078 + store: store,
1079 + template: chartTemplateV2(),
1080 + collectFunc: func(context.Context) error {
1081 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(1)
1082 + return nil
1083 + },
1084 + }
1085 +
1086 + var out bytes.Buffer
1087 + job := NewJobV2(JobV2Config{
1088 + PluginName: pluginName,
1089 + Name: jobName,
1090 + ModuleName: modName,
1091 + FullName: strings.Repeat("a", 1200),
1092 + Module: mod,
1093 + Out: &out,
1094 + UpdateEvery: 1,
1095 + VnodeRegistry: registry,
1096 + Vnode: vnodes.VirtualNode{
1097 + Hostname: "node-host-b",
1098 + GUID: "node-guid",
1099 + },
1100 + })
1101 + require.NoError(t, job.AutoDetection())
1102 +
1103 + job.runOnce()
1104 +
1105 + assert.Empty(t, out.String())
1106 + info, ok := registry.Lookup("node-guid")
1107 + require.True(t, ok)
1108 + assert.Equal(t, "node-host-a", info.Hostname)
1109 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("other")}, registry.Owners("node-guid"))
1110 + },
1111 + },
1112 + "rollback on commit failure emits nothing and next cycle recovers": {
1113 + run: func(t *testing.T) {
1114 + registry := vnoderegistry.New()
1115 + store := metrix.NewCollectorStore()
1116 + current := 1.0
1117 + mod := &mockModuleV2{
1118 + store: store,
1119 + template: chartTemplateV2(),
1120 + collectFunc: func(context.Context) error {
1121 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(current)
1122 + return nil
1123 + },
1124 + }
1125 +
1126 + var out bytes.Buffer
1127 + job := NewJobV2(JobV2Config{
1128 + PluginName: pluginName,
1129 + Name: jobName,
1130 + ModuleName: modName,
1131 + FullName: modName + "_" + jobName,
1132 + Module: mod,
1133 + Out: &out,
1134 + UpdateEvery: 1,
1135 + VnodeRegistry: registry,
1136 + Vnode: vnodes.VirtualNode{
1137 + Hostname: "node-host",
1138 + GUID: "node-guid",
1139 + },
1140 + })
1141 + require.NoError(t, job.AutoDetection())
1142 +
1143 + prepared, ok := job.collectAndEmit(0)
1144 + require.True(t, ok)
1145 + require.Len(t, prepared.scopes, 1)
1146 + require.NotEmpty(t, prepared.scopes[0].output)
1147 + assert.NotEmpty(t, registry.Owners("node-guid"))
1148 +
1149 + requireDefaultScopeState(t, job).engine.ResetMaterialized()
1150 + require.ErrorIs(t, job.finishPreparedEmission(prepared), chartengine.ErrStalePlanAttempt)
1151 + assert.Empty(t, out.String())
1152 + assert.Empty(t, registry.Owners("node-guid"))
1153 +
1154 + current = 2
1155 + job.runOnce()
1156 + assert.Contains(t, out.String(), `HOST_DEFINE 'node-guid' 'node-host'`)
1157 + assert.Contains(t, out.String(), "SET 'busy' = 2")
1158 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffjob\xffnode-guid")}, registry.Owners("node-guid"))
1159 + },
1160 + },
1161 + "guid change releases superseded owner": {
1162 + run: func(t *testing.T) {
1163 + registry := vnoderegistry.New()
1164 + store := metrix.NewCollectorStore()
1165 + current := 1.0
1166 + mod := &mockModuleV2{
1167 + store: store,
1168 + template: chartTemplateV2(),
1169 + collectFunc: func(context.Context) error {
1170 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(current)
1171 + return nil
1172 + },
1173 + }
1174 +
1175 + var out bytes.Buffer
1176 + job := NewJobV2(JobV2Config{
1177 + PluginName: pluginName,
1178 + Name: jobName,
1179 + ModuleName: modName,
1180 + FullName: modName + "_" + jobName,
1181 + Module: mod,
1182 + Out: &out,
1183 + UpdateEvery: 1,
1184 + VnodeRegistry: registry,
1185 + Vnode: vnodes.VirtualNode{
1186 + Hostname: "node-host-a",
1187 + GUID: "node-guid-a",
1188 + },
1189 + })
1190 + require.NoError(t, job.AutoDetection())
1191 +
1192 + job.runOnce()
1193 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffjob\xffnode-guid-a")}, registry.Owners("node-guid-a"))
1194 +
1195 + out.Reset()
1196 + current = 2
1197 + job.UpdateVnode(&vnodes.VirtualNode{
1198 + Hostname: "node-host-b",
1199 + GUID: "node-guid-b",
1200 + })
1201 + job.runOnce()
1202 +
1203 + assert.Empty(t, registry.Owners("node-guid-a"))
1204 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffjob\xffnode-guid-b")}, registry.Owners("node-guid-b"))
1205 + },
1206 + },
1207 + "vnode to global switch releases superseded owner": {
1208 + run: func(t *testing.T) {
1209 + registry := vnoderegistry.New()
1210 + store := metrix.NewCollectorStore()
1211 + current := 1.0
1212 + mod := &mockModuleV2{
1213 + store: store,
1214 + template: chartTemplateV2(),
1215 + collectFunc: func(context.Context) error {
1216 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(current)
1217 + return nil
1218 + },
1219 + }
1220 +
1221 + var out bytes.Buffer
1222 + job := NewJobV2(JobV2Config{
1223 + PluginName: pluginName,
1224 + Name: jobName,
1225 + ModuleName: modName,
1226 + FullName: modName + "_" + jobName,
1227 + Module: mod,
1228 + Out: &out,
1229 + UpdateEvery: 1,
1230 + VnodeRegistry: registry,
1231 + Vnode: vnodes.VirtualNode{
1232 + Hostname: "node-host",
1233 + GUID: "node-guid",
1234 + },
1235 + })
1236 + require.NoError(t, job.AutoDetection())
1237 +
1238 + job.runOnce()
1239 + require.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffjob\xffnode-guid")}, registry.Owners("node-guid"))
1240 +
1241 + out.Reset()
1242 + current = 2
1243 + job.UpdateVnode(&vnodes.VirtualNode{})
1244 + job.runOnce()
1245 +
1246 + assert.Empty(t, registry.Owners("node-guid"))
1247 + assert.Contains(t, out.String(), `HOST ''`)
1248 + assert.Contains(t, out.String(), "SET 'busy' = 2")
1249 + },
1250 + },
1251 + "cleanup emits obsoletes before releasing owner": {
1252 + run: func(t *testing.T) {
1253 + registry := vnoderegistry.New()
1254 + var out bytes.Buffer
1255 + job := newRegistryTestJobV2(t, "module_job", registry, &out, vnodes.VirtualNode{
1256 + Hostname: "node-host",
1257 + GUID: "node-guid",
1258 + })
1259 +
1260 + job.runOnce()
1261 + require.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffjob\xffnode-guid")}, registry.Owners("node-guid"))
1262 +
1263 + ownerPresentDuringWrite := false
1264 + job.out = writeFunc(func(p []byte) (int, error) {
1265 + ownerPresentDuringWrite = assert.Contains(t, registry.Owners("node-guid"), vnoderegistry.Owner("module_job\xffjob\xffnode-guid"))
1266 + return len(p), nil
1267 + })
1268 +
1269 + job.Cleanup()
1270 +
1271 + assert.True(t, ownerPresentDuringWrite)
1272 + assert.Empty(t, registry.Owners("node-guid"))
1273 + },
1274 + },
1275 + "cleanup with obsolete disabled releases owners and clears state": {
1276 + run: func(t *testing.T) {
1277 + registry := vnoderegistry.New()
1278 + var out bytes.Buffer
1279 + job := newRegistryTestJobV2(t, "module_job", registry, &out, vnodes.VirtualNode{
1280 + Hostname: "node-host",
1281 + GUID: "node-guid",
1282 + })
1283 +
1284 + job.runOnce()
1285 + require.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffjob\xffnode-guid")}, registry.Owners("node-guid"))
1286 + require.NotEmpty(t, job.scopeStates)
1287 +
1288 + out.Reset()
1289 + collectorapi.ObsoleteCharts(false)
1290 + defer collectorapi.ObsoleteCharts(true)
1291 + job.Cleanup()
1292 +
1293 + assert.Empty(t, out.String())
1294 + assert.Empty(t, registry.Owners("node-guid"))
1295 + assert.Empty(t, job.scopeStates)
1296 + },
1297 + },
1298 + "bad hostname aborts cycle without owner leak": {
1299 + run: func(t *testing.T) {
1300 + registry := vnoderegistry.New()
1301 + store := metrix.NewCollectorStore()
1302 + mod := &mockModuleV2{
1303 + store: store,
1304 + template: chartTemplateV2(),
1305 + collectFunc: func(context.Context) error {
1306 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(1)
1307 + return nil
1308 + },
1309 + }
1310 +
1311 + var out bytes.Buffer
1312 + job := NewJobV2(JobV2Config{
1313 + PluginName: pluginName,
1314 + Name: jobName,
1315 + ModuleName: modName,
1316 + FullName: modName + "_" + jobName,
1317 + Module: mod,
1318 + Out: &out,
1319 + UpdateEvery: 1,
1320 + VnodeRegistry: registry,
1321 + Vnode: vnodes.VirtualNode{
1322 + Hostname: "bad\nhost",
1323 + GUID: "node-guid",
1324 + },
1325 + })
1326 + require.NoError(t, job.AutoDetection())
1327 +
1328 + job.runOnce()
1329 +
1330 + assert.Empty(t, out.String())
1331 + assert.Empty(t, registry.Owners("node-guid"))
1332 + },
1333 + },
1334 + "empty plan does not reserve registry": {
1335 + run: func(t *testing.T) {
1336 + store := metrix.NewCollectorStore()
1337 + registry := vnoderegistry.New()
1338 + emitValue := false
1339 + mod := &mockModuleV2{
1340 + store: store,
1341 + template: chartTemplateV2(),
1342 + collectFunc: func(context.Context) error {
1343 + if emitValue {
1344 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(1)
1345 + }
1346 + return nil
1347 + },
1348 + }
1349 +
1350 + var out bytes.Buffer
1351 + job := NewJobV2(JobV2Config{
1352 + PluginName: pluginName,
1353 + Name: jobName,
1354 + ModuleName: modName,
1355 + FullName: modName + "_" + jobName,
1356 + Module: mod,
1357 + Out: &out,
1358 + UpdateEvery: 1,
1359 + VnodeRegistry: registry,
1360 + Vnode: vnodes.VirtualNode{
1361 + Hostname: "node-host",
1362 + GUID: "node-guid",
1363 + },
1364 + })
1365 + require.NoError(t, job.AutoDetection())
1366 +
1367 + job.runOnce()
1368 + assert.Equal(t, "", out.String())
1369 + assert.Equal(t, 0, registry.Len())
1370 +
1371 + emitValue = true
1372 + job.runOnce()
1373 + assert.Contains(t, out.String(), `HOST_DEFINE 'node-guid' 'node-host'`)
1374 + assert.Equal(t, 1, registry.Len())
1375 + },
1376 + },
1377 + "empty plan does not mark vnode defined": {
1378 + run: func(t *testing.T) {
1379 + store := metrix.NewCollectorStore()
1380 + emitValue := false
1381 + mod := &mockModuleV2{
1382 + store: store,
1383 + template: chartTemplateV2(),
1384 + collectFunc: func(context.Context) error {
1385 + if emitValue {
1386 + store.Write().SnapshotMeter("apache").Gauge("workers_busy").Observe(1)
1387 + }
1388 + return nil
1389 + },
1390 + }
1391 +
1392 + var out bytes.Buffer
1393 + job := newTestJobV2WithVnode(mod, &out, vnodes.VirtualNode{
1394 + Hostname: "node-host",
1395 + GUID: "node-guid",
1396 + })
1397 + require.NoError(t, job.AutoDetection())
1398 +
1399 + job.runOnce()
1400 + assert.Equal(t, "", out.String())
1401 + assert.Nil(t, job.scopeStates[defaultHostScopeKey])
1402 +
1403 + emitValue = true
1404 + job.runOnce()
1405 + assert.Contains(t, out.String(), `HOST_DEFINE 'node-guid' 'node-host'`)
1406 + assert.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid"}, requireDefaultScopeState(t, job).host.definedHost)
1407 + },
1408 },
1409 }
1410
963 - var out bytes.Buffer
964 - job := newTestJobV2WithVnode(mod, &out, vnodes.VirtualNode{
965 - Hostname: "node-host",
966 - GUID: "node-guid",
967 - })
968 - require.NoError(t, job.AutoDetection())
1411 + for name, tc := range cases {
1412 + t.Run(name, tc.run)
1413 + }
1414 +}
1415
970 - job.runOnce()
971 - assert.Equal(t, "", out.String())
972 - assert.False(t, job.hostState.definedHost.isSet())
1416 +func TestJobV2HostScopeScenarios(t *testing.T) {
1417 + scopeA := metrix.HostScope{ScopeKey: "scope-a", GUID: "guid-a", Hostname: "host-a", Labels: map[string]string{"workload": "a"}}
1418 + scopeB := metrix.HostScope{ScopeKey: "scope-b", GUID: "guid-b", Hostname: "host-b", Labels: map[string]string{"workload": "b"}}
1419
974 - emitValue = true
975 - job.runOnce()
976 - assert.Contains(t, out.String(), `HOST_DEFINE 'node-guid' 'node-host'`)
977 - assert.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid"}, job.hostState.definedHost)
1420 + cases := map[string]struct {
1421 + run func(t *testing.T)
1422 + }{
1423 + "mixed default and explicit scopes emit deterministic host batches": {
1424 + run: func(t *testing.T) {
1425 + store := metrix.NewCollectorStore()
1426 + mod := &mockModuleV2{
1427 + store: store,
1428 + template: chartTemplateV2(),
1429 + collectFunc: func(context.Context) error {
1430 + meter := store.Write().SnapshotMeter("apache")
1431 + meter.Gauge("workers_busy").Observe(1)
1432 + meter.WithHostScope(scopeB).Gauge("workers_busy").Observe(2)
1433 + meter.WithHostScope(scopeA).Gauge("workers_busy").Observe(3)
1434 + return nil
1435 + },
1436 + }
1437 +
1438 + var out bytes.Buffer
1439 + registry := vnoderegistry.New()
1440 + job := NewJobV2(JobV2Config{
1441 + PluginName: pluginName,
1442 + Name: jobName,
1443 + ModuleName: modName,
1444 + FullName: modName + "_" + jobName,
1445 + Module: mod,
1446 + Out: &out,
1447 + UpdateEvery: 1,
1448 + VnodeRegistry: registry,
1449 + })
1450 + require.NoError(t, job.AutoDetection())
1451 +
1452 + job.runOnce()
1453 +
1454 + wire := out.String()
1455 + assert.Contains(t, wire, `HOST ''
1456 +
1457 +CHART 'module_job.workers_busy'`)
1458 + assert.Contains(t, wire, `HOST_DEFINE 'guid-a' 'host-a'`)
1459 + assert.Contains(t, wire, `HOST 'guid-a'
1460 +
1461 +CHART 'module_job.workers_busy'`)
1462 + assert.Contains(t, wire, `HOST_DEFINE 'guid-b' 'host-b'`)
1463 + assert.Contains(t, wire, `HOST 'guid-b'
1464 +
1465 +CHART 'module_job.workers_busy'`)
1466 + assertContainsInOrder(t, wire, "HOST ''", "HOST_DEFINE 'guid-a'", "HOST_DEFINE 'guid-b'")
1467 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffscope\xffscope-a\xffguid-a")}, registry.Owners("guid-a"))
1468 + assert.Equal(t, []vnoderegistry.Owner{vnoderegistry.Owner("module_job\xffscope\xffscope-b\xffguid-b")}, registry.Owners("guid-b"))
1469 + },
1470 + },
1471 + "bad explicit scope does not block default scope": {
1472 + run: func(t *testing.T) {
1473 + store := metrix.NewCollectorStore()
1474 + badScope := metrix.HostScope{ScopeKey: "bad", GUID: "bad-guid", Hostname: "bad\nhost"}
1475 + mod := &mockModuleV2{
1476 + store: store,
1477 + template: chartTemplateV2(),
1478 + collectFunc: func(context.Context) error {
1479 + meter := store.Write().SnapshotMeter("apache")
1480 + meter.Gauge("workers_busy").Observe(1)
1481 + meter.WithHostScope(badScope).Gauge("workers_busy").Observe(2)
1482 + return nil
1483 + },
1484 + }
1485 +
1486 + var out bytes.Buffer
1487 + registry := vnoderegistry.New()
1488 + job := NewJobV2(JobV2Config{
1489 + PluginName: pluginName,
1490 + Name: jobName,
1491 + ModuleName: modName,
1492 + FullName: modName + "_" + jobName,
1493 + Module: mod,
1494 + Out: &out,
1495 + UpdateEvery: 1,
1496 + VnodeRegistry: registry,
1497 + })
1498 + require.NoError(t, job.AutoDetection())
1499 +
1500 + job.runOnce()
1501 +
1502 + wire := out.String()
1503 + assert.Contains(t, wire, `HOST ''
1504 +
1505 +CHART 'module_job.workers_busy'`)
1506 + assert.Contains(t, wire, "SET 'busy' = 1")
1507 + assert.NotContains(t, wire, "bad-guid")
1508 + assert.Empty(t, registry.Owners("bad-guid"))
1509 + assert.Equal(t, int64(0), job.retries.Load())
1510 + },
1511 + },
1512 + "disappeared scope is removed through chartengine lifecycle and releases owner": {
1513 + run: func(t *testing.T) {
1514 + store := metrix.NewCollectorStore()
1515 + emitScope := true
1516 + mod := &mockModuleV2{
1517 + store: store,
1518 + template: chartTemplateV2ExpireAfterOne(),
1519 + collectFunc: func(context.Context) error {
1520 + if emitScope {
1521 + store.Write().SnapshotMeter("apache").WithHostScope(scopeA).Gauge("workers_busy").Observe(7)
1522 + }
1523 + return nil
1524 + },
1525 + }
1526 +
1527 + var out bytes.Buffer
1528 + registry := vnoderegistry.New()
1529 + job := NewJobV2(JobV2Config{
1530 + PluginName: pluginName,
1531 + Name: jobName,
1532 + ModuleName: modName,
1533 + FullName: modName + "_" + jobName,
1534 + Module: mod,
1535 + Out: &out,
1536 + UpdateEvery: 1,
1537 + VnodeRegistry: registry,
1538 + })
1539 + require.NoError(t, job.AutoDetection())
1540 +
1541 + job.runOnce()
1542 + require.Contains(t, out.String(), `HOST_DEFINE 'guid-a' 'host-a'`)
1543 + require.NotEmpty(t, registry.Owners("guid-a"))
1544 + require.NotNil(t, job.scopeStates["scope-a"])
1545 +
1546 + out.Reset()
1547 + emitScope = false
1548 + job.runOnce()
1549 +
1550 + wire := out.String()
1551 + assert.Contains(t, wire, `HOST 'guid-a'`)
1552 + assert.Contains(t, wire, "obsolete")
1553 + assert.Empty(t, registry.Owners("guid-a"))
1554 + assert.Nil(t, job.scopeStates["scope-a"])
1555 + },
1556 + },
1557 + "disappeared zero-action scope is removed without registry owner": {
1558 + run: func(t *testing.T) {
1559 + store := metrix.NewCollectorStore()
1560 + emitScope := true
1561 + mod := &mockModuleV2{
1562 + store: store,
1563 + template: chartTemplateV2(),
1564 + collectFunc: func(context.Context) error {
1565 + if emitScope {
1566 + store.Write().SnapshotMeter("apache").WithHostScope(scopeA).Gauge("workers_idle").Observe(7)
1567 + }
1568 + return nil
1569 + },
1570 + }
1571 +
1572 + var out bytes.Buffer
1573 + registry := vnoderegistry.New()
1574 + job := NewJobV2(JobV2Config{
1575 + PluginName: pluginName,
1576 + Name: jobName,
1577 + ModuleName: modName,
1578 + FullName: modName + "_" + jobName,
1579 + Module: mod,
1580 + Out: &out,
1581 + UpdateEvery: 1,
1582 + VnodeRegistry: registry,
1583 + })
1584 + require.NoError(t, job.AutoDetection())
1585 +
1586 + job.runOnce()
1587 + assert.Empty(t, out.String())
1588 + assert.Empty(t, registry.Owners("guid-a"))
1589 + require.NotNil(t, job.scopeStates["scope-a"])
1590 + assert.Empty(t, job.scopeStates["scope-a"].host.cleanupCharts)
1591 +
1592 + emitScope = false
1593 + job.runOnce()
1594 +
1595 + assert.Empty(t, out.String())
1596 + assert.Empty(t, registry.Owners("guid-a"))
1597 + assert.Nil(t, job.scopeStates["scope-a"])
1598 + },
1599 + },
1600 + "default scope removal runs while explicit scope remains": {
1601 + run: func(t *testing.T) {
1602 + store := metrix.NewCollectorStore()
1603 + emitDefault := true
1604 + mod := &mockModuleV2{
1605 + store: store,
1606 + template: chartTemplateV2ExpireAfterOne(),
1607 + collectFunc: func(context.Context) error {
1608 + meter := store.Write().SnapshotMeter("apache")
1609 + if emitDefault {
1610 + meter.Gauge("workers_busy").Observe(1)
1611 + }
1612 + meter.WithHostScope(scopeA).Gauge("workers_busy").Observe(7)
1613 + return nil
1614 + },
1615 + }
1616 +
1617 + var out bytes.Buffer
1618 + job := newTestJobV2(mod, &out)
1619 + require.NoError(t, job.AutoDetection())
1620 +
1621 + job.runOnce()
1622 + require.NotNil(t, job.scopeStates[defaultHostScopeKey])
1623 + require.NotNil(t, job.scopeStates["scope-a"])
1624 +
1625 + out.Reset()
1626 + emitDefault = false
1627 + job.runOnce()
1628 +
1629 + wire := out.String()
1630 + assert.Contains(t, wire, `HOST ''`)
1631 + assert.Contains(t, wire, "obsolete")
1632 + assert.Nil(t, job.scopeStates[defaultHostScopeKey])
1633 + assert.NotNil(t, job.scopeStates["scope-a"])
1634 + },
1635 + },
1636 + "per-scope commit failure does not block peer scope": {
1637 + run: func(t *testing.T) {
1638 + store := metrix.NewCollectorStore()
1639 + mod := &mockModuleV2{
1640 + store: store,
1641 + template: chartTemplateV2(),
1642 + collectFunc: func(context.Context) error {
1643 + meter := store.Write().SnapshotMeter("apache")
1644 + meter.Gauge("workers_busy").Observe(1)
1645 + meter.WithHostScope(scopeA).Gauge("workers_busy").Observe(7)
1646 + return nil
1647 + },
1648 + }
1649 +
1650 + var out bytes.Buffer
1651 + registry := vnoderegistry.New()
1652 + job := NewJobV2(JobV2Config{
1653 + PluginName: pluginName,
1654 + Name: jobName,
1655 + ModuleName: modName,
1656 + FullName: modName + "_" + jobName,
1657 + Module: mod,
1658 + Out: &out,
1659 + UpdateEvery: 1,
1660 + VnodeRegistry: registry,
1661 + })
1662 + require.NoError(t, job.AutoDetection())
1663 +
1664 + prepared, ok := job.collectAndEmit(0)
1665 + require.True(t, ok)
1666 + require.Len(t, prepared.scopes, 2)
1667 + for _, scope := range prepared.scopes {
1668 + if scope.scope.scopeKey == "scope-a" {
1669 + scope.scope.engine.ResetMaterialized()
1670 + }
1671 + }
1672 +
1673 + require.NoError(t, job.finishPreparedEmission(prepared))
1674 +
1675 + wire := out.String()
1676 + assert.Contains(t, wire, `HOST ''`)
1677 + assert.Contains(t, wire, "SET 'busy' = 1")
1678 + assert.NotContains(t, wire, "guid-a")
1679 + assert.Empty(t, registry.Owners("guid-a"))
1680 + assert.NotNil(t, job.scopeStates[defaultHostScopeKey])
1681 + },
1682 + },
1683 + "cleanup apply failure on one scope does not block peer cleanup": {
1684 + run: func(t *testing.T) {
1685 + store := metrix.NewCollectorStore()
1686 + mod := &mockModuleV2{store: store, template: chartTemplateV2()}
1687 + var out bytes.Buffer
1688 + job := newTestJobV2(mod, &out)
1689 + require.NoError(t, job.AutoDetection())
1690 +
1691 + okMeta := chartengine.ChartMeta{
1692 + Title: "OK",
1693 + Family: "Workers",
1694 + Context: "workers_ok",
1695 + Units: "workers",
1696 + Type: chartengine.ChartTypeLine,
1697 + Priority: chartengine.Priority,
1698 + }
1699 + badMeta := okMeta
1700 + badMeta.Title = "Bad"
1701 + job.scopeStates = map[string]*jobV2ScopeState{
1702 + defaultHostScopeKey: {
1703 + scopeKey: defaultHostScopeKey,
1704 + host: jobV2HostState{
1705 + cleanupOwner: jobV2HostRef{kind: jobV2HostGlobal},
1706 + cleanupCharts: map[string]chartengine.ChartMeta{
1707 + "workers_ok": okMeta,
1708 + },
1709 + },
1710 + },
1711 + "bad": {
1712 + scopeKey: "bad",
1713 + scope: scopeA,
1714 + host: jobV2HostState{
1715 + cleanupOwner: jobV2HostRef{kind: jobV2HostGlobal},
1716 + cleanupCharts: map[string]chartengine.ChartMeta{
1717 + strings.Repeat("x", 1300): badMeta,
1718 + },
1719 + },
1720 + },
1721 + }
1722 +
1723 + job.Cleanup()
1724 +
1725 + assert.Contains(t, out.String(), "workers_ok")
1726 + assert.Contains(t, out.String(), "obsolete")
1727 + assert.Empty(t, job.scopeStates)
1728 + },
1729 + },
1730 + "panic after scoped runtime samples resets aggregator and in-flight scope": {
1731 + run: func(t *testing.T) {
1732 + store := metrix.NewCollectorStore()
1733 + mod := &mockModuleV2{
1734 + store: store,
1735 + template: chartTemplateV2(),
1736 + collectFunc: func(context.Context) error {
1737 + meter := store.Write().SnapshotMeter("apache")
1738 + meter.Gauge("workers_busy").Observe(1)
1739 + meter.WithHostScope(scopeA).Gauge("workers_busy").Observe(7)
1740 + return nil
1741 + },
1742 + }
1743 +
1744 + var out bytes.Buffer
1745 + registry := vnoderegistry.New()
1746 + job := NewJobV2(JobV2Config{
1747 + PluginName: pluginName,
1748 + Name: jobName,
1749 + ModuleName: modName,
1750 + FullName: modName + "_" + jobName,
1751 + Module: mod,
1752 + Out: &out,
1753 + UpdateEvery: 1,
1754 + VnodeRegistry: registry,
1755 + })
1756 + require.NoError(t, job.AutoDetection())
1757 + job.api = netdataapi.New(writeFunc(func(p []byte) (int, error) {
1758 + if bytes.Contains(p, []byte("HOST_DEFINE 'guid-a'")) {
1759 + panic("boom")
1760 + }
1761 + return job.buf.Write(p)
1762 + }))
1763 +
1764 + job.runOnce()
1765 +
1766 + assert.True(t, job.Panicked())
1767 + assert.Empty(t, out.String())
1768 + assert.Empty(t, registry.Owners(scopeA.GUID))
1769 + value, ok := job.runtimeStore.Read(metrix.ReadRaw()).Value("netdata.go.plugin.framework.chartengine.build_success_total", nil)
1770 + if ok {
1771 + assert.Zero(t, value)
1772 + }
1773 +
1774 + job.api = netdataapi.New(job.buf)
1775 + out.Reset()
1776 + job.runOnce()
1777 +
1778 + assert.False(t, job.Panicked())
1779 + assert.Contains(t, out.String(), `HOST_DEFINE 'guid-a' 'host-a'`)
1780 + assert.NotEmpty(t, registry.Owners(scopeA.GUID))
1781 + },
1782 + },
1783 + }
1784 +
1785 + for name, tc := range cases {
1786 + t.Run(name, tc.run)
1787 + }
1788 +}
1789 +
1790 +func assertContainsInOrder(t *testing.T, s string, parts ...string) {
1791 + t.Helper()
1792 + offset := 0
1793 + for _, part := range parts {
1794 + idx := strings.Index(s[offset:], part)
1795 + require.NotEqualf(t, -1, idx, "expected %q after offset %d", part, offset)
1796 + offset += idx + len(part)
1797 + }
1798 }
1799
1800 func TestJobV2CleanupUsesLastSuccessfulHostAfterFailedHostSwitch(t *testing.T) {
@@ -1018,8 +1838,7 @@ func TestJobV2CleanupUsesLastSuccessfulHostAfterFailedHostSwitch(t *testing.T) {
1838
1839 CHART 'module_job.workers_busy' '' 'Workers Busy' 'workers' 'Workers' 'workers_busy' 'line' '%d' '1' 'obsolete' 'plugin' 'module'`, chartengine.Priority))
1840 assert.NotContains(t, wire, "HOST 'node-guid-b'")
1021 - assert.Empty(t, job.hostState.cleanupCharts)
1022 - assert.False(t, job.hostState.cleanupOwner.isSet())
1841 + assert.Empty(t, job.scopeStates)
1842 }
1843
1844 func TestJobV2EmptyHostSwitchDoesNotKeepReloadingEngine(t *testing.T) {
@@ -1046,8 +1865,8 @@ func TestJobV2EmptyHostSwitchDoesNotKeepReloadingEngine(t *testing.T) {
1865
1866 job.runOnce()
1867 require.Equal(t, 1, mod.templateCalls)
1049 - require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, job.hostState.engineHost)
1050 - require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, job.hostState.cleanupOwner)
1868 + require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, requireDefaultScopeState(t, job).host.engineHost)
1869 + require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, requireDefaultScopeState(t, job).host.cleanupOwner)
1870
1871 out.Reset()
1872 emitValue = false
@@ -1058,15 +1877,15 @@ func TestJobV2EmptyHostSwitchDoesNotKeepReloadingEngine(t *testing.T) {
1877 job.runOnce()
1878 assert.Equal(t, "", out.String())
1879 require.Equal(t, 1, mod.templateCalls)
1061 - require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-b"}, job.hostState.engineHost)
1062 - require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, job.hostState.cleanupOwner)
1880 + require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-b"}, requireDefaultScopeState(t, job).host.engineHost)
1881 + require.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, requireDefaultScopeState(t, job).host.cleanupOwner)
1882
1883 out.Reset()
1884 job.runOnce()
1885 assert.Equal(t, "", out.String())
1886 assert.Equal(t, 1, mod.templateCalls)
1068 - assert.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-b"}, job.hostState.engineHost)
1069 - assert.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, job.hostState.cleanupOwner)
1887 + assert.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-b"}, requireDefaultScopeState(t, job).host.engineHost)
1888 + assert.Equal(t, jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid-a"}, requireDefaultScopeState(t, job).host.cleanupOwner)
1889 }
1890
1891 func TestJobV2CleanupDoesNotSuppressGlobalCleanupForDifferentStaleVnode(t *testing.T) {
@@ -1150,6 +1969,52 @@ func TestJobV2CleanupUsesPreModuleCleanupSnapshotForStaleSuppression(t *testing.
1969 assert.True(t, mod.cleaned)
1970 }
1971
1972 +func TestJobV2CleanupDoesNotSuppressExplicitScopeForStaleJobVnode(t *testing.T) {
1973 + mod := &mockModuleV2{
1974 + store: metrix.NewCollectorStore(),
1975 + template: chartTemplateV2(),
1976 + }
1977 +
1978 + var out bytes.Buffer
1979 + job := newTestJobV2WithVnode(mod, &out, vnodes.VirtualNode{
1980 + Hostname: "node-host",
1981 + GUID: "node-guid",
1982 + Labels: map[string]string{
1983 + "_node_stale_after_seconds": "60",
1984 + },
1985 + })
1986 + require.NoError(t, job.AutoDetection())
1987 +
1988 + job.scopeStates = map[string]*jobV2ScopeState{
1989 + "scope-a": {
1990 + scopeKey: "scope-a",
1991 + scope: metrix.HostScope{
1992 + ScopeKey: "scope-a",
1993 + GUID: "node-guid",
1994 + Hostname: "scoped-host",
1995 + },
1996 + host: jobV2HostState{
1997 + cleanupOwner: jobV2HostRef{kind: jobV2HostVnode, guid: "node-guid"},
1998 + cleanupCharts: map[string]chartengine.ChartMeta{
1999 + "workers_busy": {
2000 + Title: "Workers Busy",
2001 + Family: "Workers",
2002 + Context: "workers_busy",
2003 + Units: "workers",
2004 + Type: chartengine.ChartTypeLine,
2005 + Priority: chartengine.Priority,
2006 + },
2007 + },
2008 + },
2009 + },
2010 + }
2011 +
2012 + job.Cleanup()
2013 +
2014 + assert.Contains(t, out.String(), `HOST 'node-guid'`)
2015 + assert.Contains(t, out.String(), "obsolete")
2016 +}
2017 +
2018 func TestJobV2CleanupNoSuccessfulEmissionsIsNoOp(t *testing.T) {
2019 mod := &mockModuleV2{
2020 store: metrix.NewCollectorStore(),
@@ -1175,9 +2040,12 @@ func TestJobV2CleanupTrackerUsesEffectiveEmittedChartSet(t *testing.T) {
2040 Type: chartengine.ChartTypeLine,
2041 }
2042
1178 - job := &JobV2{}
2043 + job := &JobV2{scopeStates: map[string]*jobV2ScopeState{
2044 + defaultHostScopeKey: {scopeKey: defaultHostScopeKey},
2045 + }}
2046 + state := requireDefaultScopeState(t, job)
2047 decision := jobV2EmissionDecision{targetHost: jobV2HostRef{kind: jobV2HostGlobal}}
1180 - job.hostState.commitSuccessfulEmission(chartengine.Plan{
2048 + state.host.commitSuccessfulEmission(chartengine.Plan{
2049 Actions: []chartengine.EngineAction{
2050 chartengine.CreateDimensionAction{
2051 ChartID: "workers_busy",
@@ -1187,11 +2055,11 @@ func TestJobV2CleanupTrackerUsesEffectiveEmittedChartSet(t *testing.T) {
2055 },
2056 }, decision)
2057
1190 - require.Len(t, job.hostState.cleanupCharts, 1)
1191 - assert.Equal(t, meta, job.hostState.cleanupCharts["workers_busy"])
1192 - assert.Equal(t, jobV2HostRef{kind: jobV2HostGlobal}, job.hostState.cleanupOwner)
2058 + require.Len(t, state.host.cleanupCharts, 1)
2059 + assert.Equal(t, meta, state.host.cleanupCharts["workers_busy"])
2060 + assert.Equal(t, jobV2HostRef{kind: jobV2HostGlobal}, state.host.cleanupOwner)
2061
1194 - job.hostState.commitSuccessfulEmission(chartengine.Plan{
2062 + state.host.commitSuccessfulEmission(chartengine.Plan{
2063 Actions: []chartengine.EngineAction{
2064 chartengine.RemoveChartAction{
2065 ChartID: "workers_busy",
@@ -1200,7 +2068,7 @@ func TestJobV2CleanupTrackerUsesEffectiveEmittedChartSet(t *testing.T) {
2068 },
2069 }, decision)
2070
1203 - assert.Empty(t, job.hostState.cleanupCharts)
2071 + assert.Empty(t, state.host.cleanupCharts)
2072 }
2073
2074 func TestJobV2StopBeforeStartDoesNotBlock(t *testing.T) {
src/go/plugin/framework/vnoderegistry/registry.go new
+315
@@ -0,0 +1,315 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +// Package vnoderegistry tracks v2 vnode HOST_DEFINE metadata shared across jobs.
4 +package vnoderegistry
5 +
6 +import (
7 + "fmt"
8 + "maps"
9 + "slices"
10 + "sort"
11 + "strconv"
12 + "strings"
13 + "sync"
14 +
15 + "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
16 + "github.com/netdata/netdata/go/plugins/plugin/framework/chartemit"
17 +)
18 +
19 +const maxReportedMetadataStatesPerGUID = 64
20 +
21 +// Owner identifies one runtime owner of a vnode GUID.
22 +//
23 +// Jobruntime v2 uses stable per-job/per-scope owner IDs so a job can release
24 +// registry ownership after it has emitted cleanup for the corresponding scope.
25 +type Owner string
26 +
27 +// Registration reports the result of registering vnode metadata for an owner.
28 +type Registration struct {
29 + // Info is the metadata retained by the registry after registration.
30 + Info netdataapi.HostInfo
31 +
32 + // Previous is set when an existing GUID's metadata was updated.
33 + Previous netdataapi.HostInfo
34 +
35 + // NeedDefine is true when this registration created or updated the registry entry.
36 + NeedDefine bool
37 +
38 + // OwnerAdded is true when this call added a new owner record.
39 + OwnerAdded bool
40 +
41 + // MetadataUpdated is true when Info replaced previously retained metadata.
42 + MetadataUpdated bool
43 +
44 + // UpdateFirstSeen is true only for the first occurrence of a distinct metadata
45 + // transition. Callers should use this to avoid log spam.
46 + UpdateFirstSeen bool
47 +
48 + revision uint64
49 + previousRevision uint64
50 +}
51 +
52 +type Registry struct {
53 + mu sync.Mutex
54 + entries map[string]*entry
55 +}
56 +
57 +type entry struct {
58 + info netdataapi.HostInfo
59 + revision uint64
60 + owners map[Owner]struct{}
61 + reportedStates map[string]struct{}
62 + reportedOrder []string
63 +}
64 +
65 +// New returns an empty concurrency-safe vnode registry.
66 +func New() *Registry {
67 + return &Registry{entries: make(map[string]*entry)}
68 +}
69 +
70 +// Register records that owner emits metrics under info.GUID.
71 +//
72 +// New metadata for an existing GUID replaces the retained metadata. This keeps
73 +// runtime vnode updates simple: callers should log MetadataUpdated as a warning
74 +// because repeated conflicting writers can still cause metadata flip-flop.
75 +func (r *Registry) Register(owner Owner, info netdataapi.HostInfo) (Registration, error) {
76 + if r == nil {
77 + return Registration{}, fmt.Errorf("vnoderegistry: nil registry")
78 + }
79 + owner = Owner(strings.TrimSpace(string(owner)))
80 + if owner == "" {
81 + return Registration{}, fmt.Errorf("vnoderegistry: owner is required")
82 + }
83 + info, err := chartemit.PrepareHostInfo(info)
84 + if err != nil {
85 + return Registration{}, fmt.Errorf("vnoderegistry: %w", err)
86 + }
87 +
88 + r.mu.Lock()
89 + defer r.mu.Unlock()
90 +
91 + if r.entries == nil {
92 + r.entries = make(map[string]*entry)
93 + }
94 +
95 + ent, ok := r.entries[info.GUID]
96 + if !ok {
97 + ent = &entry{
98 + info: cloneHostInfo(info),
99 + revision: 1,
100 + owners: map[Owner]struct{}{owner: {}},
101 + reportedStates: make(map[string]struct{}),
102 + }
103 + r.entries[info.GUID] = ent
104 + return Registration{
105 + Info: cloneHostInfo(ent.info),
106 + NeedDefine: true,
107 + OwnerAdded: true,
108 + revision: ent.revision,
109 + }, nil
110 + }
111 +
112 + _, hadOwner := ent.owners[owner]
113 + ent.owners[owner] = struct{}{}
114 +
115 + result := Registration{
116 + Info: cloneHostInfo(ent.info),
117 + OwnerAdded: !hadOwner,
118 + revision: ent.revision,
119 + }
120 + if hostInfoEqual(ent.info, info) {
121 + return result, nil
122 + }
123 +
124 + previous := cloneHostInfo(ent.info)
125 + previousRevision := ent.revision
126 + ent.info = cloneHostInfo(info)
127 + ent.revision++
128 + result.Info = cloneHostInfo(ent.info)
129 + result.Previous = previous
130 + result.revision = ent.revision
131 + result.previousRevision = previousRevision
132 + result.NeedDefine = true
133 + result.MetadataUpdated = true
134 +
135 + updateKey := hostInfoFingerprint(info)
136 + if markReportedState(ent, updateKey) {
137 + result.UpdateFirstSeen = true
138 + }
139 + return result, nil
140 +}
141 +
142 +// Rollback undoes a registration that has not been emitted successfully.
143 +//
144 +// Rollback is best-effort: if another registration changed the same GUID after
145 +// reg, the metadata restore is skipped to avoid undoing a later writer.
146 +func (r *Registry) Rollback(owner Owner, reg Registration) {
147 + if r == nil {
148 + return
149 + }
150 + owner = Owner(strings.TrimSpace(string(owner)))
151 + guid := strings.TrimSpace(reg.Info.GUID)
152 + if owner == "" || guid == "" {
153 + return
154 + }
155 +
156 + r.mu.Lock()
157 + defer r.mu.Unlock()
158 +
159 + ent, ok := r.entries[guid]
160 + if !ok {
161 + return
162 + }
163 + if reg.OwnerAdded {
164 + delete(ent.owners, owner)
165 + }
166 + if reg.MetadataUpdated && ent.revision == reg.revision && hostInfoEqual(ent.info, reg.Info) {
167 + ent.info = cloneHostInfo(reg.Previous)
168 + ent.revision = reg.previousRevision
169 + }
170 + if len(ent.owners) == 0 {
171 + delete(r.entries, guid)
172 + }
173 +}
174 +
175 +// Release removes one owner record for guid. It returns true when the GUID entry
176 +// was removed because no owners remain.
177 +func (r *Registry) Release(owner Owner, guid string) bool {
178 + if r == nil {
179 + return false
180 + }
181 + owner = Owner(strings.TrimSpace(string(owner)))
182 + guid = strings.TrimSpace(guid)
183 + if owner == "" || guid == "" {
184 + return false
185 + }
186 +
187 + r.mu.Lock()
188 + defer r.mu.Unlock()
189 +
190 + ent, ok := r.entries[guid]
191 + if !ok {
192 + return false
193 + }
194 + delete(ent.owners, owner)
195 + if len(ent.owners) > 0 {
196 + return false
197 + }
198 + delete(r.entries, guid)
199 + return true
200 +}
201 +
202 +// Lookup returns the retained metadata for guid.
203 +func (r *Registry) Lookup(guid string) (netdataapi.HostInfo, bool) {
204 + if r == nil {
205 + return netdataapi.HostInfo{}, false
206 + }
207 + guid = strings.TrimSpace(guid)
208 + if guid == "" {
209 + return netdataapi.HostInfo{}, false
210 + }
211 +
212 + r.mu.Lock()
213 + defer r.mu.Unlock()
214 +
215 + ent, ok := r.entries[guid]
216 + if !ok {
217 + return netdataapi.HostInfo{}, false
218 + }
219 + return cloneHostInfo(ent.info), true
220 +}
221 +
222 +// Owners returns the sorted owner IDs currently registered for guid.
223 +func (r *Registry) Owners(guid string) []Owner {
224 + if r == nil {
225 + return nil
226 + }
227 + guid = strings.TrimSpace(guid)
228 + if guid == "" {
229 + return nil
230 + }
231 +
232 + r.mu.Lock()
233 + defer r.mu.Unlock()
234 +
235 + ent, ok := r.entries[guid]
236 + if !ok {
237 + return nil
238 + }
239 + owners := make([]Owner, 0, len(ent.owners))
240 + for owner := range ent.owners {
241 + owners = append(owners, owner)
242 + }
243 + slices.Sort(owners)
244 + return owners
245 +}
246 +
247 +// Len returns the number of retained GUID entries.
248 +func (r *Registry) Len() int {
249 + if r == nil {
250 + return 0
251 + }
252 + r.mu.Lock()
253 + defer r.mu.Unlock()
254 + return len(r.entries)
255 +}
256 +
257 +func cloneHostInfo(info netdataapi.HostInfo) netdataapi.HostInfo {
258 + return netdataapi.HostInfo{
259 + GUID: info.GUID,
260 + Hostname: info.Hostname,
261 + Labels: maps.Clone(info.Labels),
262 + }
263 +}
264 +
265 +func hostInfoEqual(left, right netdataapi.HostInfo) bool {
266 + return left.GUID == right.GUID &&
267 + left.Hostname == right.Hostname &&
268 + maps.Equal(left.Labels, right.Labels)
269 +}
270 +
271 +func hostInfoFingerprint(info netdataapi.HostInfo) string {
272 + var b strings.Builder
273 + writeHostInfoFingerprint(&b, info)
274 + return b.String()
275 +}
276 +
277 +func writeHostInfoFingerprint(b *strings.Builder, info netdataapi.HostInfo) {
278 + writeFingerprintPart(b, info.GUID)
279 + writeFingerprintPart(b, info.Hostname)
280 +
281 + keys := make([]string, 0, len(info.Labels))
282 + for key := range info.Labels {
283 + keys = append(keys, key)
284 + }
285 + sort.Strings(keys)
286 + for _, key := range keys {
287 + writeFingerprintPart(b, key)
288 + writeFingerprintPart(b, info.Labels[key])
289 + }
290 +}
291 +
292 +func writeFingerprintPart(b *strings.Builder, value string) {
293 + b.WriteString(strconv.Itoa(len(value)))
294 + b.WriteByte(':')
295 + b.WriteString(value)
296 + b.WriteByte('\xff')
297 +}
298 +
299 +func markReportedState(ent *entry, key string) bool {
300 + if ent.reportedStates == nil {
301 + ent.reportedStates = make(map[string]struct{})
302 + }
303 + if _, seen := ent.reportedStates[key]; seen {
304 + return false
305 + }
306 + if len(ent.reportedOrder) >= maxReportedMetadataStatesPerGUID {
307 + evicted := ent.reportedOrder[0]
308 + copy(ent.reportedOrder, ent.reportedOrder[1:])
309 + ent.reportedOrder = ent.reportedOrder[:len(ent.reportedOrder)-1]
310 + delete(ent.reportedStates, evicted)
311 + }
312 + ent.reportedStates[key] = struct{}{}
313 + ent.reportedOrder = append(ent.reportedOrder, key)
314 + return true
315 +}
src/go/plugin/framework/vnoderegistry/registry_test.go new
+285
@@ -0,0 +1,285 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package vnoderegistry
4 +
5 +import (
6 + "fmt"
7 + "sync"
8 + "testing"
9 +
10 + "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
11 + "github.com/stretchr/testify/assert"
12 + "github.com/stretchr/testify/require"
13 +)
14 +
15 +func TestRegistryScenarios(t *testing.T) {
16 + cases := map[string]struct {
17 + run func(t *testing.T)
18 + }{
19 + "updates metadata and tracks owners": {
20 + run: func(t *testing.T) {
21 + reg := New()
22 + first := netdataapi.HostInfo{
23 + GUID: "node-guid",
24 + Hostname: "node-a",
25 + Labels: map[string]string{"_hostname": "node-a", "region": "eu"},
26 + }
27 + same := netdataapi.HostInfo{
28 + GUID: "node-guid",
29 + Hostname: "node-a",
30 + Labels: map[string]string{"_hostname": "node-a", "region": "eu"},
31 + }
32 + conflict := netdataapi.HostInfo{
33 + GUID: "node-guid",
34 + Hostname: "node-b",
35 + Labels: map[string]string{"_hostname": "node-b", "region": "us"},
36 + }
37 +
38 + got, err := reg.Register("job-a", first)
39 + require.NoError(t, err)
40 + assert.True(t, got.NeedDefine)
41 + assert.True(t, got.OwnerAdded)
42 + assert.False(t, got.MetadataUpdated)
43 + assert.Equal(t, first, got.Info)
44 +
45 + got, err = reg.Register("job-b", same)
46 + require.NoError(t, err)
47 + assert.False(t, got.NeedDefine)
48 + assert.True(t, got.OwnerAdded)
49 + assert.False(t, got.MetadataUpdated)
50 + assert.Equal(t, first, got.Info)
51 +
52 + got, err = reg.Register("job-c", conflict)
53 + require.NoError(t, err)
54 + assert.True(t, got.NeedDefine)
55 + assert.True(t, got.OwnerAdded)
56 + assert.True(t, got.MetadataUpdated)
57 + assert.True(t, got.UpdateFirstSeen)
58 + assert.Equal(t, conflict, got.Info)
59 + assert.Equal(t, first, got.Previous)
60 +
61 + got, err = reg.Register("job-c", conflict)
62 + require.NoError(t, err)
63 + assert.False(t, got.NeedDefine)
64 + assert.False(t, got.OwnerAdded)
65 + assert.False(t, got.MetadataUpdated)
66 + assert.False(t, got.UpdateFirstSeen)
67 +
68 + assert.Equal(t, []Owner{"job-a", "job-b", "job-c"}, reg.Owners("node-guid"))
69 + assert.False(t, reg.Release("job-a", "node-guid"))
70 + assert.Equal(t, 1, reg.Len())
71 + assert.Equal(t, []Owner{"job-b", "job-c"}, reg.Owners("node-guid"))
72 + assert.False(t, reg.Release("job-b", "node-guid"))
73 + assert.True(t, reg.Release("job-c", "node-guid"))
74 + assert.Equal(t, 0, reg.Len())
75 + },
76 + },
77 + "normalizes metadata before compare": {
78 + run: func(t *testing.T) {
79 + reg := New()
80 + first := netdataapi.HostInfo{
81 + GUID: "node-guid",
82 + Hostname: "node-a",
83 + }
84 + sameWireInfo := netdataapi.HostInfo{
85 + GUID: " node-guid ",
86 + Hostname: " node-a ",
87 + Labels: map[string]string{
88 + "_hostname": "node-a",
89 + },
90 + }
91 +
92 + got, err := reg.Register("job-a", first)
93 + require.NoError(t, err)
94 + assert.True(t, got.NeedDefine)
95 +
96 + got, err = reg.Register("job-b", sameWireInfo)
97 + require.NoError(t, err)
98 + assert.False(t, got.NeedDefine)
99 + assert.Equal(t, map[string]string{"_hostname": "node-a"}, got.Info.Labels)
100 + },
101 + },
102 + "rollback restores metadata update": {
103 + run: func(t *testing.T) {
104 + reg := New()
105 + first := netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-a"}
106 + next := netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-b"}
107 +
108 + _, err := reg.Register("job-a", first)
109 + require.NoError(t, err)
110 + got, err := reg.Register("job-b", next)
111 + require.NoError(t, err)
112 + require.True(t, got.MetadataUpdated)
113 + require.True(t, got.OwnerAdded)
114 +
115 + reg.Rollback("job-b", got)
116 +
117 + info, ok := reg.Lookup("node-guid")
118 + require.True(t, ok)
119 + assert.Equal(t, "node-a", info.Hostname)
120 + assert.Equal(t, []Owner{"job-a"}, reg.Owners("node-guid"))
121 + },
122 + },
123 + "rollback does not undo newer metadata update": {
124 + run: func(t *testing.T) {
125 + reg := New()
126 + _, err := reg.Register("job-a", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-a"})
127 + require.NoError(t, err)
128 + rollback, err := reg.Register("job-b", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-b"})
129 + require.NoError(t, err)
130 + _, err = reg.Register("job-c", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-c"})
131 + require.NoError(t, err)
132 +
133 + reg.Rollback("job-b", rollback)
134 +
135 + info, ok := reg.Lookup("node-guid")
136 + require.True(t, ok)
137 + assert.Equal(t, "node-c", info.Hostname)
138 + assert.Equal(t, []Owner{"job-a", "job-c"}, reg.Owners("node-guid"))
139 + },
140 + },
141 + "update warnings are per state and bounded": {
142 + run: func(t *testing.T) {
143 + reg := New()
144 + _, err := reg.Register("job", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-a"})
145 + require.NoError(t, err)
146 +
147 + got, err := reg.Register("job", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-b"})
148 + require.NoError(t, err)
149 + assert.True(t, got.UpdateFirstSeen)
150 +
151 + got, err = reg.Register("job", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-a"})
152 + require.NoError(t, err)
153 + assert.True(t, got.UpdateFirstSeen)
154 +
155 + got, err = reg.Register("job", netdataapi.HostInfo{GUID: "node-guid", Hostname: "node-b"})
156 + require.NoError(t, err)
157 + assert.False(t, got.UpdateFirstSeen)
158 +
159 + for i := range maxReportedMetadataStatesPerGUID + 1 {
160 + _, err = reg.Register("job", netdataapi.HostInfo{
161 + GUID: "node-guid",
162 + Hostname: fmt.Sprintf("node-%d", i),
163 + })
164 + require.NoError(t, err)
165 + }
166 + reg.mu.Lock()
167 + defer reg.mu.Unlock()
168 + require.Len(t, reg.entries["node-guid"].reportedOrder, maxReportedMetadataStatesPerGUID)
169 + require.Len(t, reg.entries["node-guid"].reportedStates, maxReportedMetadataStatesPerGUID)
170 + },
171 + },
172 + }
173 +
174 + for name, tc := range cases {
175 + t.Run(name, tc.run)
176 + }
177 +}
178 +
179 +func TestRegistryValidation(t *testing.T) {
180 + cases := map[string]struct {
181 + owner Owner
182 + info netdataapi.HostInfo
183 + wantErr string
184 + }{
185 + "missing owner": {
186 + info: netdataapi.HostInfo{GUID: "guid", Hostname: "host"},
187 + wantErr: "owner is required",
188 + },
189 + "missing guid": {
190 + owner: "job",
191 + info: netdataapi.HostInfo{Hostname: "host"},
192 + wantErr: "host guid is required",
193 + },
194 + "missing hostname": {
195 + owner: "job",
196 + info: netdataapi.HostInfo{GUID: "guid"},
197 + wantErr: "host hostname is required",
198 + },
199 + "unsafe hostname": {
200 + owner: "job",
201 + info: netdataapi.HostInfo{GUID: "guid", Hostname: "host\nname"},
202 + wantErr: "unsupported characters",
203 + },
204 + }
205 +
206 + for name, tc := range cases {
207 + t.Run(name, func(t *testing.T) {
208 + reg := New()
209 + _, err := reg.Register(tc.owner, tc.info)
210 + require.ErrorContains(t, err, tc.wantErr)
211 + })
212 + }
213 +}
214 +
215 +func TestRegistryConcurrentScenarios(t *testing.T) {
216 + cases := map[string]struct {
217 + run func(t *testing.T)
218 + }{
219 + "registration": {
220 + run: func(t *testing.T) {
221 + reg := New()
222 + first := netdataapi.HostInfo{
223 + GUID: "node-guid",
224 + Hostname: "node",
225 + }
226 +
227 + const owners = 64
228 + var wg sync.WaitGroup
229 + errs := make(chan error, owners)
230 + for i := range owners {
231 + wg.Go(func() {
232 + info := first
233 + if i%2 == 1 {
234 + info.Hostname = "node-conflict"
235 + }
236 + _, err := reg.Register(Owner(fmt.Sprintf("job-%d", i)), info)
237 + errs <- err
238 + })
239 + }
240 + wg.Wait()
241 + close(errs)
242 + for err := range errs {
243 + require.NoError(t, err)
244 + }
245 +
246 + info, ok := reg.Lookup("node-guid")
247 + require.True(t, ok)
248 + require.NotEmpty(t, info.Hostname)
249 + require.Len(t, reg.Owners("node-guid"), owners)
250 + },
251 + },
252 + "register and release": {
253 + run: func(t *testing.T) {
254 + reg := New()
255 + const owners = 64
256 +
257 + var wg sync.WaitGroup
258 + errs := make(chan error, owners)
259 + for i := range owners {
260 + wg.Go(func() {
261 + owner := Owner(fmt.Sprintf("job-%d", i))
262 + _, err := reg.Register(owner, netdataapi.HostInfo{
263 + GUID: "node-guid",
264 + Hostname: "node",
265 + })
266 + errs <- err
267 + reg.Release(owner, "node-guid")
268 + })
269 + }
270 + wg.Wait()
271 + close(errs)
272 + for err := range errs {
273 + require.NoError(t, err)
274 + }
275 +
276 + assert.Empty(t, reg.Owners("node-guid"))
277 + assert.Equal(t, 0, reg.Len())
278 + },
279 + },
280 + }
281 +
282 + for name, tc := range cases {
283 + t.Run(name, tc.run)
284 + }
285 +}
src/go/plugin/go.d/collector/azure_monitor/collector.go
+9
@@ -78,6 +78,11 @@ type Collector struct {
78
79 discovery discoveryState
80
81 + tagsColumnMissingWarned bool
82 + tagsColumnMissingWarnedAt uint64
83 + tagsWrongShapeWarned bool
84 + tagsWrongShapeWarnedAt uint64
85 +
86 now func() time.Time
87
88 newResourceGraph func(subscriptionID string, cred azcore.TokenCredential, cloud azcloud.Configuration) (resourceGraphClient, error)
@@ -101,6 +106,10 @@ func (c *Collector) Init(ctx context.Context) error {
106 c.runtime = nil
107 c.observations = nil
108 c.discovery = discoveryState{}
109 + c.tagsColumnMissingWarned = false
110 + c.tagsColumnMissingWarnedAt = 0
111 + c.tagsWrongShapeWarned = false
112 + c.tagsWrongShapeWarnedAt = 0
113
114 return nil
115 }
src/go/plugin/go.d/collector/azure_monitor/collector_runtime.go
+19 -12
@@ -11,9 +11,10 @@ import (
11 )
12
13 type collectorRuntime struct {
14 - Profiles []*profileRuntime
15 - ChartTemplateYAML string
16 - Instruments map[string]*instrumentRuntime
14 + Profiles []*profileRuntime
15 + ChartTemplateYAML string
16 + Instruments map[string]*instrumentRuntime
17 + WorkloadResourceTagKey string
18 }
19
20 type profileRuntime struct {
@@ -46,25 +47,28 @@ type instrumentRuntime struct {
47 Counter metrix.SnapshotCounterVec
48 }
49
49 -func (i *instrumentRuntime) observe(labelValues []string, value float64) {
50 +func (i *instrumentRuntime) observe(scope metrix.HostScope, labelValues []string, value float64) {
51 if i == nil {
52 return
53 }
54 switch i.Kind {
55 case azureprofiles.SeriesKindCounter:
55 - i.Counter.WithLabelValues(labelValues...).ObserveTotal(value)
56 + i.Counter.WithHostScope(scope).WithLabelValues(labelValues...).ObserveTotal(value)
57 default:
57 - i.Gauge.WithLabelValues(labelValues...).Observe(value)
58 + i.Gauge.WithHostScope(scope).WithLabelValues(labelValues...).Observe(value)
59 }
60 }
61
62 type discoveryState struct {
62 - Resources []resourceInfo
63 - ByType map[string][]resourceInfo
64 - ByProfile map[string][]resourceInfo
65 - ExpiresAt time.Time
66 - FetchedAt time.Time
67 - FetchCounter uint64
63 + Resources []resourceInfo
64 + ByType map[string][]resourceInfo
65 + ByProfile map[string][]resourceInfo
66 + ExpiresAt time.Time
67 + FetchedAt time.Time
68 + FetchCounter uint64
69 + QueryTagsColumnMissing bool
70 + QueryTagsWrongShape bool
71 + UnsafeWorkloadValues map[string]int
72 }
73
74 type resourceTag struct {
@@ -81,6 +85,7 @@ type resourceInfo struct {
85 ResourceGroup string
86 Region string
87 Tags []resourceTag
88 + HostScope metrix.HostScope
89 }
90
91 func (r resourceInfo) String() string {
@@ -103,6 +108,7 @@ type metricSample struct {
108 Instrument string
109 Kind string
110 Labels metrix.Labels
111 + Scope metrix.HostScope
112 Value float64
113 }
114
@@ -113,6 +119,7 @@ type queryBatchResult struct {
119
120 type lastObservation struct {
121 instrument string
122 + scope metrix.HostScope
123 labelValues []string
124 value float64
125 }
src/go/plugin/go.d/collector/azure_monitor/collector_test.go
+916 -11
@@ -6,8 +6,11 @@ import (
6 "context"
7 "encoding/json"
8 "errors"
9 + "maps"
10 "os"
11 "path/filepath"
12 + "sort"
13 + "strconv"
14 "strings"
15 "sync"
16 "testing"
@@ -17,6 +20,7 @@ import (
20 azcloud "github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud"
21 "github.com/Azure/azure-sdk-for-go/sdk/monitor/query/azmetrics"
22 "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resourcegraph/armresourcegraph"
23 + "github.com/google/uuid"
24 "github.com/netdata/netdata/go/plugins/pkg/confopt"
25 "github.com/netdata/netdata/go/plugins/pkg/metrix"
26 "github.com/netdata/netdata/go/plugins/plugin/framework/chartengine"
@@ -29,14 +33,18 @@ import (
33 )
34
35 var (
32 - dataConfigJSON, _ = os.ReadFile("testdata/config.json")
33 - dataConfigYAML, _ = os.ReadFile("testdata/config.yaml")
36 + dataConfigJSON, _ = os.ReadFile("testdata/config.json")
37 + dataConfigYAML, _ = os.ReadFile("testdata/config.yaml")
38 + dataConfigWorkloadJSON, _ = os.ReadFile("testdata/config_workload.json")
39 + dataConfigWorkloadYAML, _ = os.ReadFile("testdata/config_workload.yaml")
40 )
41
42 func Test_testDataIsValid(t *testing.T) {
43 for name, data := range map[string][]byte{
38 - "dataConfigJSON": dataConfigJSON,
39 - "dataConfigYAML": dataConfigYAML,
44 + "dataConfigJSON": dataConfigJSON,
45 + "dataConfigYAML": dataConfigYAML,
46 + "dataConfigWorkloadJSON": dataConfigWorkloadJSON,
47 + "dataConfigWorkloadYAML": dataConfigWorkloadYAML,
48 } {
49 require.NotNil(t, data, name)
50 }
@@ -69,6 +77,12 @@ func TestConfigSchema_RuntimeContract(t *testing.T) {
77 assert.NotContains(t, profileProps, "mode_exact")
78 assert.NotContains(t, profileProps, "mode_combined")
79
80 + virtualNodes := requireMapField(t, properties, "virtual_nodes")
81 + assert.NotContains(t, virtualNodes, "required")
82 + virtualNodeProps := requireMapField(t, virtualNodes, "properties")
83 + byResourceTag := requireMapField(t, virtualNodeProps, "by_resource_tag")
84 + assert.Equal(t, "string", byResourceTag["type"])
85 +
86 uiSchema := requireMapField(t, doc, "uiSchema")
87 uiProfiles := requireMapField(t, uiSchema, "profiles")
88 _, hasIDs := uiProfiles["ids"]
@@ -81,6 +95,22 @@ func TestConfigSchema_RuntimeContract(t *testing.T) {
95 assert.True(t, hasModeExact)
96 _, hasModeCombined := uiProfiles["mode_combined"]
97 assert.True(t, hasModeCombined)
98 +
99 + tabs := requireArrayField(t, requireMapField(t, uiSchema, "ui:options"), "tabs")
100 + var baseFields, vnodeFields []string
101 + for _, item := range tabs {
102 + tab, ok := item.(map[string]any)
103 + require.True(t, ok)
104 + switch tab["title"] {
105 + case "Base":
106 + baseFields = requireStringSliceField(t, tab, "fields")
107 + case "Virtual Node":
108 + vnodeFields = requireStringSliceField(t, tab, "fields")
109 + }
110 + }
111 + assert.NotContains(t, baseFields, "vnode")
112 + assert.NotContains(t, baseFields, "virtual_nodes")
113 + assert.ElementsMatch(t, []string{"vnode", "virtual_nodes"}, vnodeFields)
114 }
115
116 func TestConfigSchema_ProfileTagHelpText(t *testing.T) {
@@ -94,6 +124,13 @@ func TestConfigSchema_ProfileTagHelpText(t *testing.T) {
124 assert.NotContains(t, schema, "allOf")
125
126 uiSchema := requireMapField(t, doc, "uiSchema")
127 + discovery := requireMapField(t, uiSchema, "discovery")
128 + modeQuery := requireMapField(t, discovery, "mode_query")
129 + kql := requireMapField(t, modeQuery, "kql")
130 + help, ok := kql["ui:help"].(string)
131 + require.True(t, ok)
132 + assert.Contains(t, help, "project `tags`")
133 +
134 uiProfiles := requireMapField(t, uiSchema, "profiles")
135 for _, mode := range []string{"mode_auto", "mode_exact", "mode_combined"} {
136 modeUI := requireMapField(t, uiProfiles, mode)
@@ -107,6 +144,252 @@ func TestConfigSchema_ProfileTagHelpText(t *testing.T) {
144 }
145 }
146
147 +func TestAzureMonitorStaticArtifacts_WorkloadVirtualNodes(t *testing.T) {
148 + tests := map[string]struct {
149 + path string
150 + contains []string
151 + notContain []string
152 + }{
153 + "metadata": {
154 + path: "metadata.yaml",
155 + contains: []string{
156 + "virtual_nodes.by_resource_tag",
157 + "_vnode_type=azure_workload",
158 + "azure_monitor:",
159 + "More alerts appear after enabling workload virtual nodes",
160 + },
161 + },
162 + "stock config": {
163 + path: "../../config/go.d/azure_monitor.conf",
164 + contains: []string{
165 + "subscription_ids:",
166 + "virtual_nodes:",
167 + "by_resource_tag: workload",
168 + "mode_exact:",
169 + "mode_combined:",
170 + },
171 + notContain: []string{
172 + "subscription_id:",
173 + "profile_selection_mode",
174 + },
175 + },
176 + }
177 +
178 + for name, tc := range tests {
179 + t.Run(name, func(t *testing.T) {
180 + raw, err := os.ReadFile(tc.path)
181 + require.NoError(t, err)
182 + text := string(raw)
183 + for _, want := range tc.contains {
184 + assert.Contains(t, text, want)
185 + }
186 + for _, unwanted := range tc.notContain {
187 + assert.NotContains(t, text, unwanted)
188 + }
189 + })
190 + }
191 +
192 + docs, err := filepath.Glob("integrations/*.md")
193 + require.NoError(t, err)
194 + require.Greater(t, len(docs), 1)
195 + for _, path := range docs {
196 + raw, err := os.ReadFile(path)
197 + require.NoError(t, err, path)
198 + text := string(raw)
199 + assert.Contains(t, text, "virtual_nodes.by_resource_tag", path)
200 + assert.Contains(t, text, "_vnode_type=azure_workload", path)
201 + }
202 +}
203 +
204 +func TestWorkloadScopeScenarios(t *testing.T) {
205 + tests := map[string]struct {
206 + tags []resourceTag
207 + tagKey string
208 + wantDefault bool
209 + wantUnsafe string
210 + wantHostname string
211 + }{
212 + "matching tag creates namespaced scope": {
213 + tags: []resourceTag{{Key: "workload", Value: " Api "}},
214 + tagKey: "WORKLOAD",
215 + wantHostname: "Api",
216 + },
217 + "missing tag uses default": {
218 + tags: []resourceTag{{Key: "env", Value: "prod"}},
219 + tagKey: "workload",
220 + wantDefault: true,
221 + },
222 + "empty tag value uses default": {
223 + tags: []resourceTag{{Key: "workload", Value: " "}},
224 + tagKey: "workload",
225 + wantDefault: true,
226 + },
227 + "unsafe hostname uses default and reports value": {
228 + tags: []resourceTag{{Key: "workload", Value: "api\nprod"}},
229 + tagKey: "workload",
230 + wantDefault: true,
231 + wantUnsafe: "api\nprod",
232 + },
233 + }
234 +
235 + for name, tc := range tests {
236 + t.Run(name, func(t *testing.T) {
237 + scope, unsafeValue := workloadHostScope(resourceInfo{Tags: tc.tags}, tc.tagKey)
238 +
239 + assert.Equal(t, tc.wantUnsafe, unsafeValue)
240 + if tc.wantDefault {
241 + assert.True(t, scope.IsDefault())
242 + return
243 + }
244 +
245 + wantGUID := uuid.NewSHA1(uuid.NameSpaceDNS, []byte(azureWorkloadGUIDPrefix+tc.wantHostname)).String()
246 + assert.Equal(t, wantGUID, scope.GUID)
247 + assert.Equal(t, wantGUID, scope.ScopeKey)
248 + assert.Equal(t, tc.wantHostname, scope.Hostname)
249 + assert.Equal(t, map[string]string{azureWorkloadScopeLabelKey: azureWorkloadScopeLabelValue}, scope.Labels)
250 + assert.NotEqual(t, uuid.NewSHA1(uuid.NameSpaceDNS, []byte(tc.wantHostname)).String(), scope.GUID)
251 + })
252 + }
253 +}
254 +
255 +func TestConfig_WorkloadResourceTagKey(t *testing.T) {
256 + tests := map[string]struct {
257 + virtualNodes *VirtualNodesConfig
258 + wantKey string
259 + wantNil bool
260 + }{
261 + "unset disables workload scoping": {
262 + wantNil: true,
263 + },
264 + "whitespace disables workload scoping": {
265 + virtualNodes: &VirtualNodesConfig{ByResourceTag: " \t "},
266 + wantNil: true,
267 + },
268 + "normalizes configured tag key": {
269 + virtualNodes: &VirtualNodesConfig{ByResourceTag: " Workload "},
270 + wantKey: "workload",
271 + },
272 + }
273 +
274 + for name, tc := range tests {
275 + t.Run(name, func(t *testing.T) {
276 + cfg := testConfig()
277 + cfg.VirtualNodes = tc.virtualNodes
278 + cfg.applyDefaults()
279 +
280 + assert.Equal(t, tc.wantKey, cfg.workloadResourceTagKey())
281 + if tc.wantNil {
282 + assert.Nil(t, cfg.VirtualNodes)
283 + return
284 + }
285 + require.NotNil(t, cfg.VirtualNodes)
286 + assert.Equal(t, tc.wantKey, cfg.VirtualNodes.ByResourceTag)
287 + })
288 + }
289 +}
290 +
291 +func TestDiscoveryState_WorkloadScopes(t *testing.T) {
292 + runtime := &collectorRuntime{
293 + WorkloadResourceTagKey: "workload",
294 + Profiles: []*profileRuntime{{
295 + Name: "sql_database",
296 + ResourceType: "Microsoft.Sql/servers/databases",
297 + }},
298 + }
299 + resources := []resourceInfo{
300 + {
301 + SubscriptionID: "sub-1",
302 + ID: "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.Sql/servers/sql-a/databases/db-a",
303 + UID: "uid-a",
304 + Name: "db-a",
305 + Type: "Microsoft.Sql/servers/databases",
306 + ResourceGroup: "rg-a",
307 + Region: "eastus",
308 + Tags: []resourceTag{{Key: "workload", Value: "api"}},
309 + },
310 + }
311 +
312 + state := buildDiscoveryState(resources, runtime, time.Unix(0, 0), 300, 7, discoveryFetchResult{})
313 + require.Len(t, state.Resources, 1)
314 + scope := state.Resources[0].HostScope
315 + require.False(t, scope.IsDefault())
316 + assert.Equal(t, "api", scope.Hostname)
317 + assert.Equal(t, scope.GUID, scope.ScopeKey)
318 + assert.Equal(t, state.Resources, state.ByProfile["sql_database"])
319 +
320 + changed := state.Resources[0]
321 + changed.HostScope = metrix.HostScope{}
322 + assert.False(t, equalResourceInfo(state.Resources[0], changed))
323 +}
324 +
325 +func TestParseStrictQueryDiscoveryRow_OptionalTags(t *testing.T) {
326 + base := map[string]any{
327 + "id": "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.Sql/servers/sql-a/databases/db-a",
328 + "name": "db-a",
329 + "type": "Microsoft.Sql/servers/databases",
330 + "resourceGroup": "rg-a",
331 + "location": "eastus",
332 + }
333 +
334 + tests := map[string]struct {
335 + mutate func(map[string]any)
336 + wantShape queryTagsShape
337 + wantTags []resourceTag
338 + wantErrSubstr string
339 + }{
340 + "absent tags": {
341 + wantShape: queryTagsShapeAbsent,
342 + },
343 + "map tags": {
344 + mutate: func(row map[string]any) {
345 + row["tags"] = map[string]any{"Workload": "api"}
346 + },
347 + wantShape: queryTagsShapePresentMap,
348 + wantTags: []resourceTag{{Key: "workload", Value: "api"}},
349 + },
350 + "null tags are projected empty tags": {
351 + mutate: func(row map[string]any) {
352 + row["tags"] = nil
353 + },
354 + wantShape: queryTagsShapePresentMap,
355 + },
356 + "wrong shaped tags do not fail required parsing": {
357 + mutate: func(row map[string]any) {
358 + row["tags"] = []any{"not", "a", "map"}
359 + },
360 + wantShape: queryTagsShapeWrong,
361 + },
362 + "missing required column still fails": {
363 + mutate: func(row map[string]any) {
364 + delete(row, "location")
365 + },
366 + wantShape: queryTagsShapeAbsent,
367 + wantErrSubstr: `missing required column "location"`,
368 + },
369 + }
370 +
371 + for name, tc := range tests {
372 + t.Run(name, func(t *testing.T) {
373 + row := make(map[string]any, len(base)+1)
374 + maps.Copy(row, base)
375 + if tc.mutate != nil {
376 + tc.mutate(row)
377 + }
378 +
379 + resource, shape, err := parseStrictQueryDiscoveryRow(row)
380 + assert.Equal(t, tc.wantShape, shape)
381 + if tc.wantErrSubstr != "" {
382 + require.Error(t, err)
383 + assert.ErrorContains(t, err, tc.wantErrSubstr)
384 + return
385 + }
386 +
387 + require.NoError(t, err)
388 + assert.Equal(t, tc.wantTags, resource.Tags)
389 + })
390 + }
391 +}
392 +
393 func requireMapField(t *testing.T, m map[string]any, key string) map[string]any {
394 t.Helper()
395
@@ -148,6 +431,10 @@ func TestCollector_ConfigurationSerialize(t *testing.T) {
431 collecttest.TestConfigurationSerialize(t, &Collector{}, dataConfigJSON, dataConfigYAML)
432 }
433
434 +func TestCollector_ConfigurationSerialize_WorkloadVirtualNodes(t *testing.T) {
435 + collecttest.TestConfigurationSerialize(t, &Collector{}, dataConfigWorkloadJSON, dataConfigWorkloadYAML)
436 +}
437 +
438 func TestCollector_Init(t *testing.T) {
439 tests := map[string]struct {
440 cfg Config
@@ -709,6 +996,71 @@ func TestCollector_CollectScenarios(t *testing.T) {
996 }
997 }
998
999 +func TestCollector_WorkloadMode_MixedTaggedAndUntaggedResources_RouteToCorrectScopes(t *testing.T) {
1000 + now := time.Date(2026, 3, 7, 12, 0, 0, 0, time.UTC)
1001 + taggedID := "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-a"
1002 + untaggedID := "/subscriptions/sub-1/resourceGroups/rg-b/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-b"
1003 +
1004 + rg := &mockResourceGraph{
1005 + resources: []map[string]any{
1006 + {
1007 + "id": taggedID,
1008 + "name": "pg-a",
1009 + "type": "Microsoft.DBforPostgreSQL/flexibleServers",
1010 + "resourceGroup": "rg-a",
1011 + "location": "eastus",
1012 + "tags": map[string]any{"workload": "api"},
1013 + },
1014 + {
1015 + "id": untaggedID,
1016 + "name": "pg-b",
1017 + "type": "Microsoft.DBforPostgreSQL/flexibleServers",
1018 + "resourceGroup": "rg-b",
1019 + "location": "eastus",
1020 + "tags": map[string]any{"env": "prod"},
1021 + },
1022 + },
1023 + }
1024 + mx := &mockMetricsClient{
1025 + queryResponse: azmetrics.QueryResourcesResponse{MetricResults: azmetrics.MetricResults{Values: []azmetrics.MetricData{
1026 + {
1027 + ResourceID: ptrString(strings.ToLower(taggedID)),
1028 + Values: []azmetrics.Metric{
1029 + metricWithAvg("cpu_percent", now, 21.5),
1030 + },
1031 + },
1032 + {
1033 + ResourceID: ptrString(strings.ToLower(untaggedID)),
1034 + Values: []azmetrics.Metric{
1035 + metricWithAvg("cpu_percent", now, 33.1),
1036 + },
1037 + },
1038 + }}},
1039 + }
1040 + c := newTestCollectorWithMocks(rg, mx)
1041 + c.Config = testConfig()
1042 + c.Config.VirtualNodes = &VirtualNodesConfig{ByResourceTag: "workload"}
1043 + c.now = func() time.Time { return now }
1044 +
1045 + require.NoError(t, c.Init(context.Background()))
1046 + defaultSeries, err := collecttest.CollectScalarSeries(c, metrix.ReadRaw())
1047 + require.NoError(t, err)
1048 + assert.Contains(t, strings.Join(keysFromSeries(defaultSeries), "\n"), `resource_name="pg-b"`)
1049 + assert.NotContains(t, strings.Join(keysFromSeries(defaultSeries), "\n"), `resource_name="pg-a"`)
1050 +
1051 + var workloadScope metrix.HostScope
1052 + for _, resource := range c.discovery.Resources {
1053 + if resource.Name == "pg-a" {
1054 + workloadScope = resource.HostScope
1055 + }
1056 + }
1057 + require.False(t, workloadScope.IsDefault())
1058 +
1059 + scopedSeries := scalarSeriesFromReader(c.store.Read(metrix.ReadRaw(), metrix.ReadHostScope(workloadScope.ScopeKey)))
1060 + assert.Contains(t, strings.Join(keysFromSeries(scopedSeries), "\n"), `resource_name="pg-a"`)
1061 + assert.NotContains(t, strings.Join(keysFromSeries(scopedSeries), "\n"), `resource_name="pg-b"`)
1062 +}
1063 +
1064 func TestCollector_RefreshDiscovery_PushesModeFiltersIntoQuery(t *testing.T) {
1065 tests := map[string]struct {
1066 resources []map[string]any
@@ -985,6 +1337,63 @@ template:
1337 assert.Equal(t, []string{"UsedCapacity"}, byInterval["PT5M"].MetricNames)
1338 }
1339
1340 +func TestSamplesFromQueryResponse_UsesResourceHostScope(t *testing.T) {
1341 + now := time.Date(2026, 3, 7, 12, 0, 0, 0, time.UTC)
1342 + scope, unsafeValue := workloadHostScope(resourceInfo{Tags: []resourceTag{{Key: "workload", Value: "api"}}}, "workload")
1343 + require.Empty(t, unsafeValue)
1344 +
1345 + tests := map[string]struct {
1346 + scope metrix.HostScope
1347 + }{
1348 + "default scope": {},
1349 + "workload scope": {
1350 + scope: scope,
1351 + },
1352 + }
1353 +
1354 + for name, tc := range tests {
1355 + t.Run(name, func(t *testing.T) {
1356 + resourceID := "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-a"
1357 + samples := samplesFromQueryResponse(
1358 + []azmetrics.MetricData{{
1359 + ResourceID: ptrString(strings.ToLower(resourceID)),
1360 + Values: []azmetrics.Metric{
1361 + metricWithAvg("cpu_percent", now, 10),
1362 + },
1363 + }},
1364 + "postgres_flexible",
1365 + map[string]*metricRuntime{
1366 + "cpu_percent": {
1367 + Series: []*seriesRuntime{{
1368 + Aggregation: "average",
1369 + Kind: azureprofiles.SeriesKindGauge,
1370 + Instrument: "postgres_flexible.cpu_percent.average",
1371 + }},
1372 + },
1373 + },
1374 + map[string]resourceInfo{
1375 + strings.ToLower(resourceID): {
1376 + SubscriptionID: "sub-1",
1377 + ID: resourceID,
1378 + UID: "uid-a",
1379 + Name: "pg-a",
1380 + Type: "Microsoft.DBforPostgreSQL/flexibleServers",
1381 + ResourceGroup: "rg-a",
1382 + Region: "eastus",
1383 + HostScope: tc.scope,
1384 + },
1385 + },
1386 + )
1387 +
1388 + require.Len(t, samples, 1)
1389 + assert.Equal(t, tc.scope, samples[0].Scope)
1390 + if tc.scope.IsDefault() {
1391 + assert.True(t, samples[0].Scope.IsDefault())
1392 + }
1393 + })
1394 + }
1395 +}
1396 +
1397 func TestCollector_CheckBootstrapProfileScenarios(t *testing.T) {
1398 tests := map[string]struct {
1399 resources []map[string]any
@@ -1183,6 +1592,281 @@ func TestCollector_CheckBootstrapQueryModeScenarios(t *testing.T) {
1592 }
1593 }
1594
1595 +func TestCollector_WorkloadMode_QueryModeTagFallbackState(t *testing.T) {
1596 + const kql = "resources | project id, name, type, resourceGroup, location"
1597 +
1598 + tests := map[string]struct {
1599 + tags any
1600 + tagsPresent bool
1601 + wantMissingWarning bool
1602 + wantWrongShapeWarning bool
1603 + wantDefaultScope bool
1604 + wantWorkloadScopeHostname string
1605 + }{
1606 + "tags column missing falls back and records warning": {
1607 + wantMissingWarning: true,
1608 + wantDefaultScope: true,
1609 + },
1610 + "tags wrong shape falls back and records warning": {
1611 + tagsPresent: true,
1612 + tags: "not-a-map",
1613 + wantWrongShapeWarning: true,
1614 + wantDefaultScope: true,
1615 + },
1616 + "configured tag missing in tags map falls back silently": {
1617 + tagsPresent: true,
1618 + tags: map[string]any{"env": "prod"},
1619 + wantDefaultScope: true,
1620 + },
1621 + "tags map derives workload scope": {
1622 + tagsPresent: true,
1623 + tags: map[string]any{"Workload": "api"},
1624 + wantWorkloadScopeHostname: "api",
1625 + },
1626 + }
1627 +
1628 + for name, tc := range tests {
1629 + t.Run(name, func(t *testing.T) {
1630 + row := map[string]any{
1631 + "id": "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-a",
1632 + "name": "pg-a",
1633 + "type": "Microsoft.DBforPostgreSQL/flexibleServers",
1634 + "resourceGroup": "rg-a",
1635 + "location": "eastus",
1636 + }
1637 + if tc.tagsPresent {
1638 + row["tags"] = tc.tags
1639 + }
1640 +
1641 + rg := &mockResourceGraph{resources: []map[string]any{row}}
1642 + c := newTestCollectorWithMocks(rg, &mockMetricsClient{})
1643 + c.Config = testConfig()
1644 + c.Config.VirtualNodes = &VirtualNodesConfig{ByResourceTag: "workload"}
1645 + c.Config.Profiles.Mode = profilesModeAuto
1646 + c.Config.Profiles.ModeExact = nil
1647 + c.Config.Profiles.ModeCombined = nil
1648 + c.Config.Discovery.Mode = discoveryModeQuery
1649 + c.Config.Discovery.ModeFilters = nil
1650 + c.Config.Discovery.ModeQuery = &DiscoveryQueryConfig{KQL: kql}
1651 +
1652 + require.NoError(t, c.Init(context.Background()))
1653 + require.NoError(t, c.Check(context.Background()))
1654 +
1655 + require.Len(t, c.discovery.Resources, 1)
1656 + assert.Equal(t, tc.wantMissingWarning, c.tagsColumnMissingWarned)
1657 + assert.Equal(t, tc.wantWrongShapeWarning, c.tagsWrongShapeWarned)
1658 + if tc.wantDefaultScope {
1659 + assert.True(t, c.discovery.Resources[0].HostScope.IsDefault())
1660 + return
1661 + }
1662 + require.False(t, c.discovery.Resources[0].HostScope.IsDefault())
1663 + assert.Equal(t, tc.wantWorkloadScopeHostname, c.discovery.Resources[0].HostScope.Hostname)
1664 + })
1665 + }
1666 +}
1667 +
1668 +func TestCollector_WorkloadMode_QueryModeWarningWatermarks(t *testing.T) {
1669 + c := New()
1670 + runtime := &collectorRuntime{WorkloadResourceTagKey: "workload"}
1671 +
1672 + state := discoveryState{
1673 + FetchCounter: 1,
1674 + QueryTagsColumnMissing: true,
1675 + QueryTagsWrongShape: true,
1676 + UnsafeWorkloadValues: map[string]int{
1677 + "bad\nvalue": 2,
1678 + },
1679 + }
1680 +
1681 + c.warnDiscoveryScopeFallbacks(state, runtime)
1682 + c.warnDiscoveryScopeFallbacks(state, runtime)
1683 + assert.True(t, c.tagsColumnMissingWarned)
1684 + assert.True(t, c.tagsWrongShapeWarned)
1685 + assert.Equal(t, uint64(1), c.tagsColumnMissingWarnedAt)
1686 + assert.Equal(t, uint64(1), c.tagsWrongShapeWarnedAt)
1687 +
1688 + state.FetchCounter = 2
1689 + c.warnDiscoveryScopeFallbacks(state, runtime)
1690 + assert.Equal(t, uint64(2), c.tagsColumnMissingWarnedAt)
1691 + assert.Equal(t, uint64(2), c.tagsWrongShapeWarnedAt)
1692 +}
1693 +
1694 +func TestCollector_WorkloadMode_QueryModeWarningDedupAcrossRefreshes(t *testing.T) {
1695 + const kql = "resources | project id, name, type, resourceGroup, location"
1696 +
1697 + tests := map[string]struct {
1698 + tagsPresent bool
1699 + tags any
1700 + wantMissingAt func(*Collector) uint64
1701 + wantWrongAt func(*Collector) uint64
1702 + }{
1703 + "missing tags column": {
1704 + wantMissingAt: func(c *Collector) uint64 { return c.tagsColumnMissingWarnedAt },
1705 + },
1706 + "wrong shaped tags column": {
1707 + tagsPresent: true,
1708 + tags: "not-a-map",
1709 + wantWrongAt: func(c *Collector) uint64 { return c.tagsWrongShapeWarnedAt },
1710 + },
1711 + }
1712 +
1713 + for name, tc := range tests {
1714 + t.Run(name, func(t *testing.T) {
1715 + row := map[string]any{
1716 + "id": "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-a",
1717 + "name": "pg-a",
1718 + "type": "Microsoft.DBforPostgreSQL/flexibleServers",
1719 + "resourceGroup": "rg-a",
1720 + "location": "eastus",
1721 + }
1722 + if tc.tagsPresent {
1723 + row["tags"] = tc.tags
1724 + }
1725 +
1726 + rg := &mockResourceGraph{resources: []map[string]any{row}}
1727 + c := newTestCollectorWithMocks(rg, &mockMetricsClient{})
1728 + c.Config = testConfig()
1729 + c.Config.VirtualNodes = &VirtualNodesConfig{ByResourceTag: "workload"}
1730 + c.Config.Profiles.Mode = profilesModeAuto
1731 + c.Config.Profiles.ModeExact = nil
1732 + c.Config.Profiles.ModeCombined = nil
1733 + c.Config.Discovery.Mode = discoveryModeQuery
1734 + c.Config.Discovery.ModeFilters = nil
1735 + c.Config.Discovery.ModeQuery = &DiscoveryQueryConfig{KQL: kql}
1736 +
1737 + require.NoError(t, c.Init(context.Background()))
1738 + require.NoError(t, c.Check(context.Background()))
1739 + require.Equal(t, uint64(1), c.discovery.FetchCounter)
1740 + if tc.wantMissingAt != nil {
1741 + assert.Equal(t, uint64(1), tc.wantMissingAt(c))
1742 + }
1743 + if tc.wantWrongAt != nil {
1744 + assert.Equal(t, uint64(1), tc.wantWrongAt(c))
1745 + }
1746 +
1747 + c.warnDiscoveryScopeFallbacks(c.discovery, c.runtime)
1748 + if tc.wantMissingAt != nil {
1749 + assert.Equal(t, uint64(1), tc.wantMissingAt(c))
1750 + }
1751 + if tc.wantWrongAt != nil {
1752 + assert.Equal(t, uint64(1), tc.wantWrongAt(c))
1753 + }
1754 +
1755 + _, err := c.refreshDiscovery(context.Background(), true)
1756 + require.NoError(t, err)
1757 + require.Equal(t, uint64(2), c.discovery.FetchCounter)
1758 + if tc.wantMissingAt != nil {
1759 + assert.Equal(t, uint64(2), tc.wantMissingAt(c))
1760 + }
1761 + if tc.wantWrongAt != nil {
1762 + assert.Equal(t, uint64(2), tc.wantWrongAt(c))
1763 + }
1764 + })
1765 + }
1766 +}
1767 +
1768 +func TestCollector_WorkloadMode_UnsafeValueReportResetsPerRefresh(t *testing.T) {
1769 + runtime := &collectorRuntime{
1770 + WorkloadResourceTagKey: "workload",
1771 + Profiles: []*profileRuntime{{
1772 + Name: "postgres_flexible",
1773 + ResourceType: "Microsoft.DBforPostgreSQL/flexibleServers",
1774 + }},
1775 + }
1776 + base := resourceInfo{
1777 + SubscriptionID: "sub-1",
1778 + ID: "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-a",
1779 + UID: "uid-a",
1780 + Name: "pg-a",
1781 + Type: "Microsoft.DBforPostgreSQL/flexibleServers",
1782 + ResourceGroup: "rg-a",
1783 + Region: "eastus",
1784 + }
1785 + tests := map[string]struct {
1786 + refreshes []struct {
1787 + tags []resourceTag
1788 + wantUnsafe map[string]int
1789 + }
1790 + }{
1791 + "unsafe report is scoped to the current refresh": {
1792 + refreshes: []struct {
1793 + tags []resourceTag
1794 + wantUnsafe map[string]int
1795 + }{
1796 + {
1797 + tags: []resourceTag{{Key: "workload", Value: "bad\none"}},
1798 + wantUnsafe: map[string]int{"bad\none": 1},
1799 + },
1800 + {
1801 + tags: []resourceTag{{Key: "workload", Value: "bad\ntwo"}},
1802 + wantUnsafe: map[string]int{"bad\ntwo": 1},
1803 + },
1804 + {
1805 + tags: []resourceTag{{Key: "workload", Value: "api"}},
1806 + },
1807 + },
1808 + },
1809 + }
1810 +
1811 + for name, tc := range tests {
1812 + t.Run(name, func(t *testing.T) {
1813 + for i, refresh := range tc.refreshes {
1814 + resource := base
1815 + resource.Tags = refresh.tags
1816 + fetchCounter := uint64(i + 1)
1817 +
1818 + state := buildDiscoveryState([]resourceInfo{resource}, runtime, time.Unix(int64(fetchCounter), 0), 300, fetchCounter, discoveryFetchResult{})
1819 +
1820 + assert.Equal(t, refresh.wantUnsafe, state.UnsafeWorkloadValues)
1821 + }
1822 + })
1823 + }
1824 +}
1825 +
1826 +func TestCollector_WorkloadMode_DisabledLeavesQueryAndScopesDefault(t *testing.T) {
1827 + const kql = "resources | project id, name, type, resourceGroup, location"
1828 +
1829 + tests := map[string]struct {
1830 + virtualNodes *VirtualNodesConfig
1831 + }{
1832 + "unset": {},
1833 + "whitespace": {
1834 + virtualNodes: &VirtualNodesConfig{ByResourceTag: " \t "},
1835 + },
1836 + }
1837 +
1838 + for name, tc := range tests {
1839 + t.Run(name, func(t *testing.T) {
1840 + rg := &mockResourceGraph{resources: []map[string]any{{
1841 + "id": "/subscriptions/sub-1/resourceGroups/rg-a/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-a",
1842 + "name": "pg-a",
1843 + "type": "Microsoft.DBforPostgreSQL/flexibleServers",
1844 + "resourceGroup": "rg-a",
1845 + "location": "eastus",
1846 + }}}
1847 + c := newTestCollectorWithMocks(rg, &mockMetricsClient{})
1848 + c.Config = testConfig()
1849 + c.Config.VirtualNodes = tc.virtualNodes
1850 + c.Config.Profiles.Mode = profilesModeAuto
1851 + c.Config.Profiles.ModeExact = nil
1852 + c.Config.Profiles.ModeCombined = nil
1853 + c.Config.Discovery.Mode = discoveryModeQuery
1854 + c.Config.Discovery.ModeFilters = nil
1855 + c.Config.Discovery.ModeQuery = &DiscoveryQueryConfig{KQL: kql}
1856 +
1857 + require.NoError(t, c.Init(context.Background()))
1858 + require.NoError(t, c.Check(context.Background()))
1859 +
1860 + assert.Equal(t, kql, rg.lastQuery())
1861 + assert.Empty(t, c.runtime.WorkloadResourceTagKey)
1862 + require.Len(t, c.discovery.Resources, 1)
1863 + assert.True(t, c.discovery.Resources[0].HostScope.IsDefault())
1864 + assert.False(t, c.tagsColumnMissingWarned)
1865 + assert.False(t, c.tagsWrongShapeWarned)
1866 + })
1867 + }
1868 +}
1869 +
1870 func TestCollector_InitQueryModeRejectsMalformedRows(t *testing.T) {
1871 const kql = "resources | project id, name, type, resourceGroup, location"
1872
@@ -1454,13 +2138,13 @@ func TestObservationState_PruneStaleResources_RemovesOldProfileMembership(t *tes
2138 Type: "Microsoft.Sql/servers/databases",
2139 }
2140 labels := labelValues(resourceLabels(resource, "sql_database"))
1457 - key := sampleObservationKey("sql.cpu", labels)
2141 + key := sampleObservationKey("sql.cpu", metrix.HostScope{}, labels)
2142
2143 state := &observationState{
1460 - accumulators: map[string]float64{
2144 + accumulators: map[observationKey]float64{
2145 key: 10,
2146 },
1463 - lastObserved: map[string]lastObservation{
2147 + lastObserved: map[observationKey]lastObservation{
2148 key: {
2149 instrument: "sql.cpu",
2150 labelValues: append([]string(nil), labels...),
@@ -1490,13 +2174,13 @@ func TestObservationState_PruneStaleResources_RemovesLabelChurnForSameResource(t
2174 newResource.Name = "db-a-renamed"
2175
2176 labels := labelValues(resourceLabels(oldResource, "sql_database"))
1493 - key := sampleObservationKey("sql.cpu", labels)
2177 + key := sampleObservationKey("sql.cpu", metrix.HostScope{}, labels)
2178
2179 state := &observationState{
1496 - accumulators: map[string]float64{
2180 + accumulators: map[observationKey]float64{
2181 key: 10,
2182 },
1499 - lastObserved: map[string]lastObservation{
2183 + lastObserved: map[observationKey]lastObservation{
2184 key: {
2185 instrument: "sql.cpu",
2186 labelValues: append([]string(nil), labels...),
@@ -1513,6 +2197,192 @@ func TestObservationState_PruneStaleResources_RemovesLabelChurnForSameResource(t
2197 assert.Empty(t, state.accumulators)
2198 }
2199
2200 +func TestObservationState_PruneStaleResources_RespectsScopeBoundary(t *testing.T) {
2201 + scopeA, unsafeValue := workloadHostScope(resourceInfo{Tags: []resourceTag{{Key: "workload", Value: "api"}}}, "workload")
2202 + require.Empty(t, unsafeValue)
2203 + scopeB, unsafeValue := workloadHostScope(resourceInfo{Tags: []resourceTag{{Key: "workload", Value: "worker"}}}, "workload")
2204 + require.Empty(t, unsafeValue)
2205 +
2206 + resource := resourceInfo{
2207 + SubscriptionID: "sub-1",
2208 + UID: "uid-a",
2209 + Name: "db-a",
2210 + ResourceGroup: "rg-a",
2211 + Region: "eastus",
2212 + Type: "Microsoft.Sql/servers/databases",
2213 + HostScope: scopeA,
2214 + }
2215 + labels := labelValues(resourceLabels(resource, "sql_database"))
2216 + keyA := sampleObservationKey("sql.cpu", scopeA, labels)
2217 + keyB := sampleObservationKey("sql.cpu", scopeB, labels)
2218 +
2219 + state := &observationState{
2220 + accumulators: map[observationKey]float64{
2221 + keyA: 10,
2222 + keyB: 20,
2223 + },
2224 + lastObserved: map[observationKey]lastObservation{
2225 + keyA: {
2226 + instrument: "sql.cpu",
2227 + scope: scopeA,
2228 + labelValues: append([]string(nil), labels...),
2229 + value: 10,
2230 + },
2231 + keyB: {
2232 + instrument: "sql.cpu",
2233 + scope: scopeB,
2234 + labelValues: append([]string(nil), labels...),
2235 + value: 20,
2236 + },
2237 + },
2238 + }
2239 +
2240 + state.pruneStaleResources(map[string][]resourceInfo{
2241 + "sql_database": {resource},
2242 + })
2243 +
2244 + assert.Contains(t, state.lastObserved, keyA)
2245 + assert.NotContains(t, state.lastObserved, keyB)
2246 + assert.Contains(t, state.accumulators, keyA)
2247 + assert.NotContains(t, state.accumulators, keyB)
2248 +}
2249 +
2250 +func TestObservationState_CounterAccumulatorFollowsScopeMobility(t *testing.T) {
2251 + scopes := map[string]metrix.HostScope{}
2252 + for _, workload := range []string{"api", "worker", "batch"} {
2253 + scope, unsafeValue := workloadHostScope(resourceInfo{Tags: []resourceTag{{Key: "workload", Value: workload}}}, "workload")
2254 + require.Empty(t, unsafeValue)
2255 + scopes[workload] = scope
2256 + }
2257 +
2258 + store := metrix.NewCollectorStore()
2259 + managed, ok := metrix.AsCycleManagedStore(store)
2260 + require.True(t, ok)
2261 + cycle := managed.CycleController()
2262 + vec := store.Write().SnapshotMeter("").Vec("resource_uid", "subscription_id", "resource_name", "resource_group", "region", "resource_type", "profile")
2263 + state := &observationState{
2264 + instruments: map[string]*instrumentRuntime{
2265 + "sql.transactions": {
2266 + Kind: azureprofiles.SeriesKindCounter,
2267 + Counter: vec.Counter("sql.transactions"),
2268 + },
2269 + },
2270 + accumulators: make(map[observationKey]float64),
2271 + lastObserved: make(map[observationKey]lastObservation),
2272 + }
2273 +
2274 + baseResource := resourceInfo{
2275 + SubscriptionID: "sub-1",
2276 + UID: "uid-a",
2277 + Name: "db-a",
2278 + ResourceGroup: "rg-a",
2279 + Region: "eastus",
2280 + Type: "Microsoft.Sql/servers/databases",
2281 + }
2282 + steps := map[string]struct {
2283 + path []struct {
2284 + workload string
2285 + value float64
2286 + }
2287 + }{
2288 + "a to b to c to a": {
2289 + path: []struct {
2290 + workload string
2291 + value float64
2292 + }{
2293 + {workload: "api", value: 5},
2294 + {workload: "worker", value: 7},
2295 + {workload: "batch", value: 11},
2296 + {workload: "api", value: 13},
2297 + },
2298 + },
2299 + }
2300 +
2301 + for name, tc := range steps {
2302 + t.Run(name, func(t *testing.T) {
2303 + for _, step := range tc.path {
2304 + scope := scopes[step.workload]
2305 + resource := baseResource
2306 + resource.HostScope = scope
2307 + labels := labelValues(resourceLabels(resource, "sql_database"))
2308 +
2309 + state.pruneStaleResources(map[string][]resourceInfo{"sql_database": {resource}})
2310 + cycle.BeginCycle()
2311 + observed := state.observeSamples([]metricSample{{
2312 + Instrument: "sql.transactions",
2313 + Kind: azureprofiles.SeriesKindCounter,
2314 + Scope: scope,
2315 + Labels: resourceLabels(resource, "sql_database"),
2316 + Value: step.value,
2317 + }})
2318 + state.reobserveCachedObservations(map[string]bool{}, observed)
2319 + require.NoError(t, cycle.CommitCycleSuccess())
2320 +
2321 + key := sampleObservationKey("sql.transactions", scope, labels)
2322 + require.Len(t, state.accumulators, 1)
2323 + assert.Equal(t, step.value, state.accumulators[key])
2324 + assert.Contains(t, state.lastObserved, key)
2325 + }
2326 + })
2327 + }
2328 +}
2329 +
2330 +func TestObservationState_ReobserveUsesStoredScope(t *testing.T) {
2331 + scope, unsafeValue := workloadHostScope(resourceInfo{Tags: []resourceTag{{Key: "workload", Value: "api"}}}, "workload")
2332 + require.Empty(t, unsafeValue)
2333 +
2334 + store := metrix.NewCollectorStore()
2335 + managed, ok := metrix.AsCycleManagedStore(store)
2336 + require.True(t, ok)
2337 + cycle := managed.CycleController()
2338 + vec := store.Write().SnapshotMeter("").Vec("resource_uid", "subscription_id", "resource_name", "resource_group", "region", "resource_type", "profile")
2339 + labels := []string{"uid-a", "sub-1", "db-a", "rg-a", "eastus", "Microsoft.Sql/servers/databases", "sql_database"}
2340 +
2341 + state := &observationState{
2342 + instruments: map[string]*instrumentRuntime{
2343 + "sql.cpu": {
2344 + Kind: azureprofiles.SeriesKindGauge,
2345 + Gauge: vec.Gauge("sql.cpu"),
2346 + },
2347 + },
2348 + accumulators: make(map[observationKey]float64),
2349 + lastObserved: map[observationKey]lastObservation{
2350 + sampleObservationKey("sql.cpu", scope, labels): {
2351 + instrument: "sql.cpu",
2352 + scope: scope,
2353 + labelValues: append([]string(nil), labels...),
2354 + value: 42,
2355 + },
2356 + },
2357 + }
2358 +
2359 + cycle.BeginCycle()
2360 + state.reobserveCachedObservations(map[string]bool{}, map[observationKey]bool{})
2361 + require.NoError(t, cycle.CommitCycleSuccess())
2362 +
2363 + _, ok = store.Read(metrix.ReadRaw()).Value("sql.cpu", metrix.Labels{
2364 + "resource_uid": "uid-a",
2365 + "subscription_id": "sub-1",
2366 + "resource_name": "db-a",
2367 + "resource_group": "rg-a",
2368 + "region": "eastus",
2369 + "resource_type": "Microsoft.Sql/servers/databases",
2370 + "profile": "sql_database",
2371 + })
2372 + assert.False(t, ok)
2373 +
2374 + _, ok = store.Read(metrix.ReadRaw(), metrix.ReadHostScope(scope.ScopeKey)).Value("sql.cpu", metrix.Labels{
2375 + "resource_uid": "uid-a",
2376 + "subscription_id": "sub-1",
2377 + "resource_name": "db-a",
2378 + "resource_group": "rg-a",
2379 + "region": "eastus",
2380 + "resource_type": "Microsoft.Sql/servers/databases",
2381 + "profile": "sql_database",
2382 + })
2383 + assert.True(t, ok)
2384 +}
2385 +
2386 func TestConfig_ValidateDiscoveryContracts(t *testing.T) {
2387 tests := map[string]struct {
2388 cfg Config
@@ -1774,7 +2644,7 @@ template:
2644 `,
2645 })
2646
1777 - _, err := buildCollectorRuntimeFromConfig(profileNames, nil, catalog)
2647 + _, err := buildCollectorRuntimeFromConfig(profileNames, nil, catalog, "")
2648 require.Error(t, err)
2649 }
2650
@@ -2043,3 +2913,38 @@ func keysFromSeries(series map[string]metrix.SampleValue) []string {
2913 }
2914 return out
2915 }
2916 +
2917 +func scalarSeriesFromReader(reader metrix.Reader) map[string]metrix.SampleValue {
2918 + out := make(map[string]metrix.SampleValue)
2919 + reader.ForEachSeries(func(name string, labels metrix.LabelView, value metrix.SampleValue) {
2920 + out[scalarKeyFromLabelView(name, labels)] = value
2921 + })
2922 + return out
2923 +}
2924 +
2925 +func scalarKeyFromLabelView(name string, labels metrix.LabelView) string {
2926 + if labels == nil || labels.Len() == 0 {
2927 + return strings.TrimSpace(name)
2928 + }
2929 +
2930 + labelsMap := labels.CloneMap()
2931 + keys := make([]string, 0, len(labelsMap))
2932 + for key := range labelsMap {
2933 + keys = append(keys, key)
2934 + }
2935 + sort.Strings(keys)
2936 +
2937 + var b strings.Builder
2938 + b.WriteString(strings.TrimSpace(name))
2939 + b.WriteByte('{')
2940 + for i, key := range keys {
2941 + if i > 0 {
2942 + b.WriteByte(',')
2943 + }
2944 + b.WriteString(key)
2945 + b.WriteByte('=')
2946 + b.WriteString(strconv.Quote(labelsMap[key]))
2947 + }
2948 + b.WriteByte('}')
2949 + return b.String()
2950 +}
src/go/plugin/go.d/collector/azure_monitor/config.go
+20
@@ -45,6 +45,7 @@ const (
45
46 type Config struct {
47 Vnode string `yaml:"vnode,omitempty" json:"vnode,omitempty"`
48 + VirtualNodes *VirtualNodesConfig `yaml:"virtual_nodes,omitempty" json:"virtual_nodes,omitempty"`
49 UpdateEvery int `yaml:"update_every,omitempty" json:"update_every,omitempty"`
50 AutoDetectionRetry int `yaml:"autodetection_retry,omitempty" json:"autodetection_retry,omitempty"`
51 SubscriptionIDs []string `yaml:"subscription_ids" json:"subscription_ids"`
@@ -57,6 +58,10 @@ type Config struct {
58 Auth cloudauth.AzureADAuthConfig `yaml:"auth" json:"auth"`
59 }
60
61 +type VirtualNodesConfig struct {
62 + ByResourceTag string `yaml:"by_resource_tag,omitempty" json:"by_resource_tag,omitempty"`
63 +}
64 +
65 type DiscoveryConfig struct {
66 RefreshEvery int `yaml:"refresh_every,omitempty" json:"refresh_every"`
67 Mode string `yaml:"mode,omitempty" json:"mode"`
@@ -130,6 +135,14 @@ func (c *Config) applyDefaults() {
135 if c.Limits.MaxMetricsPerQuery <= 0 {
136 c.Limits.MaxMetricsPerQuery = defaultMaxMetricsQuery
137 }
138 + if c.VirtualNodes != nil {
139 + tagKey := stringsLowerTrim(c.VirtualNodes.ByResourceTag)
140 + if tagKey == "" {
141 + c.VirtualNodes = nil
142 + } else {
143 + c.VirtualNodes = &VirtualNodesConfig{ByResourceTag: tagKey}
144 + }
145 + }
146 }
147
148 func (c Config) validate() error {
@@ -411,3 +424,10 @@ func (c Config) subscriptionIDs() []string {
424 }
425 return out
426 }
427 +
428 +func (c Config) workloadResourceTagKey() string {
429 + if c.VirtualNodes == nil {
430 + return ""
431 + }
432 + return stringsLowerTrim(c.VirtualNodes.ByResourceTag)
433 +}
src/go/plugin/go.d/collector/azure_monitor/config_schema.json
+28 -4
@@ -513,6 +513,18 @@
513 "title": "Virtual Node",
514 "description": "Associates this data collection job with a Virtual Node.",
515 "type": "string"
516 + },
517 + "virtual_nodes": {
518 + "title": "Virtual Nodes",
519 + "description": "Creates Azure workload virtual nodes from Azure resource tags.",
520 + "type": "object",
521 + "properties": {
522 + "by_resource_tag": {
523 + "title": "By resource tag",
524 + "description": "Optional Azure resource tag key. When set, resources with this tag are emitted under a virtual node whose hostname is the tag value. In custom query mode, project `tags` in the KQL to enable tag-based virtual nodes; missing or non-object tags fall back to the default host scope.",
525 + "type": "string"
526 + }
527 + }
528 }
529 },
530 "required": [
@@ -578,8 +590,8 @@
590 },
591 "mode_query": {
592 "kql": {
581 - "ui:help": "Provide a custom Azure Resource Graph KQL query. The result must include resource `id`, `name`, `type`, `resourceGroup`, and `location`.",
582 - "ui:placeholder": "resources | where tags.env == 'prod' | project id, name, type, resourceGroup, location",
593 + "ui:help": "Provide a custom Azure Resource Graph KQL query. The result must include resource `id`, `name`, `type`, `resourceGroup`, and `location`. If `virtual_nodes.by_resource_tag` is set, project `tags` too; rows without object-shaped `tags` use the default host scope.",
594 + "ui:placeholder": "resources | where tags.env == 'prod' | project id, name, type, resourceGroup, location, tags",
595 "ui:widget": "textarea"
596 }
597 }
@@ -709,6 +721,12 @@
721 "ui:help": "Optional virtual node name used to group charts from this job under a specific Netdata virtual node.",
722 "ui:placeholder": "To use this option, first create a Virtual Node and then reference its name here."
723 },
724 + "virtual_nodes": {
725 + "by_resource_tag": {
726 + "ui:help": "Optional Azure resource tag key used to create per-tag-value virtual nodes. Example: `workload` routes resources with `tags.workload=api` to a virtual node named `api`.",
727 + "ui:placeholder": "workload"
728 + }
729 + },
730 "ui:flavour": "tabs",
731 "ui:options": {
732 "tabs": [
@@ -720,8 +738,7 @@
738 "update_every",
739 "autodetection_retry",
740 "query_offset",
723 - "timeout",
724 - "vnode"
741 + "timeout"
742 ]
743 },
744 {
@@ -747,6 +764,13 @@
764 "fields": [
765 "limits"
766 ]
767 + },
768 + {
769 + "title": "Virtual Node",
770 + "fields": [
771 + "vnode",
772 + "virtual_nodes"
773 + ]
774 }
775 ]
776 }
src/go/plugin/go.d/collector/azure_monitor/discover.go
+68 -25
@@ -15,9 +15,11 @@ import (
15 )
16
17 type discoveryFetchResult struct {
18 - Resources []resourceInfo
19 - ByType map[string][]resourceInfo
20 - UnsupportedTypes []string
18 + Resources []resourceInfo
19 + ByType map[string][]resourceInfo
20 + UnsupportedTypes []string
21 + QueryTagsColumnMissing bool
22 + QueryTagsWrongShape bool
23 }
24
25 type normalizedTagFilter struct {
@@ -49,24 +51,29 @@ func (c *Collector) refreshDiscovery(ctx context.Context, force bool) ([]resourc
51 c.Warningf("ignoring unsupported discovered resource types: %v", fetched.UnsupportedTypes)
52 }
53
52 - state := buildDiscoveryState(fetched.Resources, c.runtime, now, c.Discovery.RefreshEvery, c.discovery.FetchCounter+1)
54 + state := buildDiscoveryState(fetched.Resources, c.runtime, now, c.Discovery.RefreshEvery, c.discovery.FetchCounter+1, fetched)
55 if !equalResourceSlices(state.Resources, c.discovery.Resources) {
56 c.Infof("discovered %d resources: %v", len(state.Resources), state.Resources)
57 }
58
59 c.discovery = state
60 + c.warnDiscoveryScopeFallbacks(state, c.runtime)
61 return state.Resources, nil
62 }
63
61 -func buildDiscoveryState(resources []resourceInfo, runtime *collectorRuntime, now time.Time, refreshEvery int, fetchCounter uint64) discoveryState {
62 - filteredResources, byType := filterDiscoveryResourcesByTypes(resources, runtimeResourceTypes(runtime))
64 +func buildDiscoveryState(resources []resourceInfo, runtime *collectorRuntime, now time.Time, refreshEvery int, fetchCounter uint64, fetched discoveryFetchResult) discoveryState {
65 + filteredResources, _ := filterDiscoveryResourcesByTypes(resources, runtimeResourceTypes(runtime))
66 + scopeReport := applyWorkloadHostScopes(filteredResources, runtime)
67 return discoveryState{
64 - Resources: filteredResources,
65 - ByType: byType,
66 - ByProfile: filterDiscoveryResourcesByProfiles(filteredResources, runtime),
67 - FetchedAt: now,
68 - ExpiresAt: discoveryExpiresAt(now, refreshEvery),
69 - FetchCounter: fetchCounter,
68 + Resources: filteredResources,
69 + ByType: indexResourcesByType(filteredResources),
70 + ByProfile: filterDiscoveryResourcesByProfiles(filteredResources, runtime),
71 + FetchedAt: now,
72 + ExpiresAt: discoveryExpiresAt(now, refreshEvery),
73 + FetchCounter: fetchCounter,
74 + QueryTagsColumnMissing: fetched.QueryTagsColumnMissing,
75 + QueryTagsWrongShape: fetched.QueryTagsWrongShape,
76 + UnsafeWorkloadValues: scopeReport.unsafeValues,
77 }
78 }
79
@@ -227,6 +234,7 @@ func discoverResourcesFromQuery(ctx context.Context, subscriptionIDs []string, t
234 result := make([]resourceInfo, 0, 256)
235 unsupported := make(map[string]struct{})
236 seenIDs := make(map[string]struct{})
237 + var resultMissingTagsColumn, resultWrongTagsShape bool
238
239 var skipToken *string
240 for {
@@ -244,10 +252,16 @@ func discoverResourcesFromQuery(ctx context.Context, subscriptionIDs []string, t
252 }
253
254 for i, row := range rows {
247 - resource, err := parseStrictQueryDiscoveryRow(row)
255 + resource, tagsShape, err := parseStrictQueryDiscoveryRow(row)
256 if err != nil {
257 return discoveryFetchResult{}, fmt.Errorf("query result row %d: %w", i, err)
258 }
259 + switch tagsShape {
260 + case queryTagsShapeAbsent:
261 + resultMissingTagsColumn = true
262 + case queryTagsShapeWrong:
263 + resultWrongTagsShape = true
264 + }
265
266 idKey := stringsLowerTrim(resource.ID)
267 if _, ok := seenIDs[idKey]; ok {
@@ -278,9 +292,11 @@ func discoverResourcesFromQuery(ctx context.Context, subscriptionIDs []string, t
292 slices.Sort(unsupportedTypes)
293
294 return discoveryFetchResult{
281 - Resources: result,
282 - ByType: indexResourcesByType(result),
283 - UnsupportedTypes: unsupportedTypes,
295 + Resources: result,
296 + ByType: indexResourcesByType(result),
297 + UnsupportedTypes: unsupportedTypes,
298 + QueryTagsColumnMissing: resultMissingTagsColumn,
299 + QueryTagsWrongShape: resultWrongTagsShape,
300 }, nil
301 }
302
@@ -498,40 +514,49 @@ func normalizedFilterSet(values []string) map[string]struct{} {
514 return set
515 }
516
501 -func parseStrictQueryDiscoveryRow(row map[string]any) (resourceInfo, error) {
517 +type queryTagsShape int
518 +
519 +const (
520 + queryTagsShapeAbsent queryTagsShape = iota
521 + queryTagsShapePresentMap
522 + queryTagsShapeWrong
523 +)
524 +
525 +func parseStrictQueryDiscoveryRow(row map[string]any) (resourceInfo, queryTagsShape, error) {
526 id, err := strictQueryStringColumn(row, "id")
527 if err != nil {
504 - return resourceInfo{}, err
528 + return resourceInfo{}, queryTagsShapeAbsent, err
529 }
530 subscriptionID, ok := parseARMResourceID(id)
531 if !ok {
508 - return resourceInfo{}, fmt.Errorf("invalid ARM resource id %q", id)
532 + return resourceInfo{}, queryTagsShapeAbsent, fmt.Errorf("invalid ARM resource id %q", id)
533 }
534
535 name, err := strictQueryStringColumn(row, "name")
536 if err != nil {
513 - return resourceInfo{}, err
537 + return resourceInfo{}, queryTagsShapeAbsent, err
538 }
539 resourceType, err := strictQueryStringColumn(row, "type")
540 if err != nil {
517 - return resourceInfo{}, err
541 + return resourceInfo{}, queryTagsShapeAbsent, err
542 }
543 if resourceType == "" {
520 - return resourceInfo{}, errors.New("column 'type' must not be empty")
544 + return resourceInfo{}, queryTagsShapeAbsent, errors.New("column 'type' must not be empty")
545 }
546
547 resourceGroup, err := strictQueryStringColumn(row, "resourceGroup")
548 if err != nil {
525 - return resourceInfo{}, err
549 + return resourceInfo{}, queryTagsShapeAbsent, err
550 }
551 location, err := strictQueryStringColumn(row, "location")
552 if err != nil {
529 - return resourceInfo{}, err
553 + return resourceInfo{}, queryTagsShapeAbsent, err
554 }
555 region := stringsLowerTrim(location)
556 if region == "" {
557 region = "global"
558 }
559 + tags, tagsShape := optionalQueryTagsColumn(row)
560
561 return resourceInfo{
562 SubscriptionID: subscriptionID,
@@ -541,7 +566,24 @@ func parseStrictQueryDiscoveryRow(row map[string]any) (resourceInfo, error) {
566 Type: resourceType,
567 ResourceGroup: resourceGroup,
568 Region: region,
544 - }, nil
569 + Tags: tags,
570 + }, tagsShape, nil
571 +}
572 +
573 +func optionalQueryTagsColumn(row map[string]any) ([]resourceTag, queryTagsShape) {
574 + value, ok := row["tags"]
575 + if !ok {
576 + return nil, queryTagsShapeAbsent
577 + }
578 + if value == nil {
579 + return nil, queryTagsShapePresentMap
580 + }
581 + switch value.(type) {
582 + case map[string]any, map[string]string:
583 + return normalizeResourceTags(value), queryTagsShapePresentMap
584 + default:
585 + return nil, queryTagsShapeWrong
586 + }
587 }
588
589 func strictQueryStringColumn(row map[string]any, column string) (string, error) {
@@ -783,6 +825,7 @@ func equalResourceInfo(a, b resourceInfo) bool {
825 a.Type == b.Type &&
826 a.ResourceGroup == b.ResourceGroup &&
827 a.Region == b.Region &&
828 + a.HostScope.ScopeKey == b.HostScope.ScopeKey &&
829 equalResourceTags(a.Tags, b.Tags)
830 }
831
src/go/plugin/go.d/collector/azure_monitor/init.go
+3 -2
@@ -117,7 +117,7 @@ func (c *Collector) ensureBootstrapped(ctx context.Context) error {
117 // lookup errors, and prune unsupported metrics/aggregations/time grains
118 // before final runtime build. Because runtime is currently global per job,
119 // multi-subscription capability differences need an explicit merge rule first.
120 - runtime, err := buildCollectorRuntimeFromConfig(selection.Names, selection.Entries, c.profileCatalog)
120 + runtime, err := buildCollectorRuntimeFromConfig(selection.Names, selection.Entries, c.profileCatalog, c.Config.workloadResourceTagKey())
121 if err != nil {
122 return fmt.Errorf("build collector runtime: %w", err)
123 }
@@ -128,7 +128,8 @@ func (c *Collector) ensureBootstrapped(ctx context.Context) error {
128
129 c.runtime = runtime
130 c.observations = newObservationState(runtime.Instruments)
131 - c.discovery = buildDiscoveryState(fetched.Resources, runtime, now, c.Discovery.RefreshEvery, 1)
131 + c.discovery = buildDiscoveryState(fetched.Resources, runtime, now, c.Discovery.RefreshEvery, 1, fetched)
132 + c.warnDiscoveryScopeFallbacks(c.discovery, runtime)
133
134 return nil
135 }
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_api_management.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -688,6 +791,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
791 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
792
793
794 +### Workload virtual nodes are not created
795 +
796 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
797 +
798 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
799 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
800 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
801 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
802 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
803 +
804 +
805 +### More alerts appear after enabling workload virtual nodes
806 +
807 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
808 +
809 +
810 ### Authentication errors in sovereign clouds
811
812 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_app_service.md
+124 -5
@@ -192,7 +192,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
192 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
195 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
195 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
196 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
197 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
198 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -200,7 +200,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
200 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
201 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
202 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
203 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
205
206 <a id="option-collection-query-offset"></a>
207 ##### query_offset
@@ -235,7 +236,7 @@ Controls how the collector finds candidate Azure resources.
236 | Mode | Behavior |
237 |:-----|:---------|
238 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
238 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
239 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
240
241
242 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -254,6 +255,8 @@ The query **must** project these five columns:
255 | `resourceGroup` | Resource group name |
256 | `location` | Azure region |
257
258 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
259 +
260 :::info
261
262 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -267,7 +270,7 @@ Example:
270 ```
271 resources
272 | where tags.env =~ "prod"
270 -| project id, name, type, resourceGroup, location
273 +| project id, name, type, resourceGroup, location, tags
274 ```
275
276
@@ -341,6 +344,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
344 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
345
346
347 +<a id="option-virtual-node-vnode"></a>
348 +##### vnode
349 +
350 +This job-level virtual node is used for metrics written to the default host scope.
351 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
352 +
353 +
354 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
355 +##### virtual_nodes.by_resource_tag
356 +
357 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
358 +
359 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
360 +- Empty or missing tag values use the default job host scope.
361 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
362 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
363 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
364 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
365 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
366 +
367 +
368
369 </details>
370
@@ -412,6 +436,85 @@ jobs:
436 client_secret: "your-client-secret"
437
438 ```
439 +###### Workload virtual nodes from resource tags
440 +
441 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
442 +
443 +<details open><summary>Config</summary>
444 +
445 +```yaml
446 +jobs:
447 + - name: prod-workloads
448 + subscription_ids:
449 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
450 + virtual_nodes:
451 + by_resource_tag: workload
452 + discovery:
453 + mode: filters
454 + profiles:
455 + mode: auto
456 + auth:
457 + mode: service_principal
458 + mode_service_principal:
459 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_secret: "your-client-secret"
462 +
463 +```
464 +</details>
465 +
466 +###### Custom KQL with workload virtual nodes
467 +
468 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
469 +
470 +<details open><summary>Config</summary>
471 +
472 +```yaml
473 +jobs:
474 + - name: prod-query-workloads
475 + subscription_ids:
476 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
477 + virtual_nodes:
478 + by_resource_tag: workload
479 + discovery:
480 + mode: query
481 + mode_query:
482 + kql: |
483 + resources
484 + | where tags.env =~ "prod"
485 + | project id, name, type, resourceGroup, location, tags
486 + profiles:
487 + mode: auto
488 + auth:
489 + mode: default
490 +
491 +```
492 +</details>
493 +
494 +###### Job virtual node plus workload virtual nodes
495 +
496 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
497 +
498 +<details open><summary>Config</summary>
499 +
500 +```yaml
501 +jobs:
502 + - name: prod-with-fallback-node
503 + vnode: azure-fallback-node
504 + subscription_ids:
505 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
506 + virtual_nodes:
507 + by_resource_tag: workload
508 + discovery:
509 + mode: filters
510 + profiles:
511 + mode: auto
512 + auth:
513 + mode: managed_identity
514 +
515 +```
516 +</details>
517 +
518 ###### Managed identity with exact profiles
519
520 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -491,7 +594,7 @@ jobs:
594 kql: |
595 resources
596 | where tags.env =~ "prod"
494 - | project id, name, type, resourceGroup, location
597 + | project id, name, type, resourceGroup, location, tags
598 profiles:
599 mode: auto
600 auth:
@@ -703,6 +806,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
806 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
807
808
809 +### Workload virtual nodes are not created
810 +
811 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
812 +
813 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
814 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
815 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
816 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
817 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
818 +
819 +
820 +### More alerts appear after enabling workload virtual nodes
821 +
822 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
823 +
824 +
825 ### Authentication errors in sovereign clouds
826
827 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_application_gateway.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -694,6 +797,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
797 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
798
799
800 +### Workload virtual nodes are not created
801 +
802 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
803 +
804 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
805 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
806 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
807 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
808 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
809 +
810 +
811 +### More alerts appear after enabling workload virtual nodes
812 +
813 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
814 +
815 +
816 ### Authentication errors in sovereign clouds
817
818 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_application_insights.md
+124 -5
@@ -191,7 +191,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
194 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
194 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
195 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
196 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
197 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -199,7 +199,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
199 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
200 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
201 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
202 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
204
205 <a id="option-collection-query-offset"></a>
206 ##### query_offset
@@ -234,7 +235,7 @@ Controls how the collector finds candidate Azure resources.
235 | Mode | Behavior |
236 |:-----|:---------|
237 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
237 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
238 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
239
240
241 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -253,6 +254,8 @@ The query **must** project these five columns:
254 | `resourceGroup` | Resource group name |
255 | `location` | Azure region |
256
257 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
258 +
259 :::info
260
261 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -266,7 +269,7 @@ Example:
269 ```
270 resources
271 | where tags.env =~ "prod"
269 -| project id, name, type, resourceGroup, location
272 +| project id, name, type, resourceGroup, location, tags
273 ```
274
275
@@ -340,6 +343,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
343 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
344
345
346 +<a id="option-virtual-node-vnode"></a>
347 +##### vnode
348 +
349 +This job-level virtual node is used for metrics written to the default host scope.
350 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
351 +
352 +
353 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
354 +##### virtual_nodes.by_resource_tag
355 +
356 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
357 +
358 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
359 +- Empty or missing tag values use the default job host scope.
360 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
361 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
362 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
363 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
364 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
365 +
366 +
367
368 </details>
369
@@ -411,6 +435,85 @@ jobs:
435 client_secret: "your-client-secret"
436
437 ```
438 +###### Workload virtual nodes from resource tags
439 +
440 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
441 +
442 +<details open><summary>Config</summary>
443 +
444 +```yaml
445 +jobs:
446 + - name: prod-workloads
447 + subscription_ids:
448 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
449 + virtual_nodes:
450 + by_resource_tag: workload
451 + discovery:
452 + mode: filters
453 + profiles:
454 + mode: auto
455 + auth:
456 + mode: service_principal
457 + mode_service_principal:
458 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_secret: "your-client-secret"
461 +
462 +```
463 +</details>
464 +
465 +###### Custom KQL with workload virtual nodes
466 +
467 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
468 +
469 +<details open><summary>Config</summary>
470 +
471 +```yaml
472 +jobs:
473 + - name: prod-query-workloads
474 + subscription_ids:
475 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
476 + virtual_nodes:
477 + by_resource_tag: workload
478 + discovery:
479 + mode: query
480 + mode_query:
481 + kql: |
482 + resources
483 + | where tags.env =~ "prod"
484 + | project id, name, type, resourceGroup, location, tags
485 + profiles:
486 + mode: auto
487 + auth:
488 + mode: default
489 +
490 +```
491 +</details>
492 +
493 +###### Job virtual node plus workload virtual nodes
494 +
495 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
496 +
497 +<details open><summary>Config</summary>
498 +
499 +```yaml
500 +jobs:
501 + - name: prod-with-fallback-node
502 + vnode: azure-fallback-node
503 + subscription_ids:
504 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
505 + virtual_nodes:
506 + by_resource_tag: workload
507 + discovery:
508 + mode: filters
509 + profiles:
510 + mode: auto
511 + auth:
512 + mode: managed_identity
513 +
514 +```
515 +</details>
516 +
517 ###### Managed identity with exact profiles
518
519 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -490,7 +593,7 @@ jobs:
593 kql: |
594 resources
595 | where tags.env =~ "prod"
493 - | project id, name, type, resourceGroup, location
596 + | project id, name, type, resourceGroup, location, tags
597 profiles:
598 mode: auto
599 auth:
@@ -703,6 +806,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
806 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
807
808
809 +### Workload virtual nodes are not created
810 +
811 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
812 +
813 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
814 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
815 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
816 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
817 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
818 +
819 +
820 +### More alerts appear after enabling workload virtual nodes
821 +
822 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
823 +
824 +
825 ### Authentication errors in sovereign clouds
826
827 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_cache_for_redis.md
+124 -5
@@ -192,7 +192,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
192 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
195 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
195 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
196 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
197 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
198 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -200,7 +200,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
200 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
201 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
202 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
203 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
205
206 <a id="option-collection-query-offset"></a>
207 ##### query_offset
@@ -235,7 +236,7 @@ Controls how the collector finds candidate Azure resources.
236 | Mode | Behavior |
237 |:-----|:---------|
238 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
238 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
239 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
240
241
242 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -254,6 +255,8 @@ The query **must** project these five columns:
255 | `resourceGroup` | Resource group name |
256 | `location` | Azure region |
257
258 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
259 +
260 :::info
261
262 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -267,7 +270,7 @@ Example:
270 ```
271 resources
272 | where tags.env =~ "prod"
270 -| project id, name, type, resourceGroup, location
273 +| project id, name, type, resourceGroup, location, tags
274 ```
275
276
@@ -341,6 +344,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
344 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
345
346
347 +<a id="option-virtual-node-vnode"></a>
348 +##### vnode
349 +
350 +This job-level virtual node is used for metrics written to the default host scope.
351 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
352 +
353 +
354 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
355 +##### virtual_nodes.by_resource_tag
356 +
357 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
358 +
359 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
360 +- Empty or missing tag values use the default job host scope.
361 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
362 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
363 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
364 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
365 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
366 +
367 +
368
369 </details>
370
@@ -412,6 +436,85 @@ jobs:
436 client_secret: "your-client-secret"
437
438 ```
439 +###### Workload virtual nodes from resource tags
440 +
441 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
442 +
443 +<details open><summary>Config</summary>
444 +
445 +```yaml
446 +jobs:
447 + - name: prod-workloads
448 + subscription_ids:
449 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
450 + virtual_nodes:
451 + by_resource_tag: workload
452 + discovery:
453 + mode: filters
454 + profiles:
455 + mode: auto
456 + auth:
457 + mode: service_principal
458 + mode_service_principal:
459 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_secret: "your-client-secret"
462 +
463 +```
464 +</details>
465 +
466 +###### Custom KQL with workload virtual nodes
467 +
468 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
469 +
470 +<details open><summary>Config</summary>
471 +
472 +```yaml
473 +jobs:
474 + - name: prod-query-workloads
475 + subscription_ids:
476 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
477 + virtual_nodes:
478 + by_resource_tag: workload
479 + discovery:
480 + mode: query
481 + mode_query:
482 + kql: |
483 + resources
484 + | where tags.env =~ "prod"
485 + | project id, name, type, resourceGroup, location, tags
486 + profiles:
487 + mode: auto
488 + auth:
489 + mode: default
490 +
491 +```
492 +</details>
493 +
494 +###### Job virtual node plus workload virtual nodes
495 +
496 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
497 +
498 +<details open><summary>Config</summary>
499 +
500 +```yaml
501 +jobs:
502 + - name: prod-with-fallback-node
503 + vnode: azure-fallback-node
504 + subscription_ids:
505 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
506 + virtual_nodes:
507 + by_resource_tag: workload
508 + discovery:
509 + mode: filters
510 + profiles:
511 + mode: auto
512 + auth:
513 + mode: managed_identity
514 +
515 +```
516 +</details>
517 +
518 ###### Managed identity with exact profiles
519
520 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -491,7 +594,7 @@ jobs:
594 kql: |
595 resources
596 | where tags.env =~ "prod"
494 - | project id, name, type, resourceGroup, location
597 + | project id, name, type, resourceGroup, location, tags
598 profiles:
599 mode: auto
600 auth:
@@ -712,6 +815,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
815 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
816
817
818 +### Workload virtual nodes are not created
819 +
820 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
821 +
822 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
823 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
824 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
825 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
826 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
827 +
828 +
829 +### More alerts appear after enabling workload virtual nodes
830 +
831 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
832 +
833 +
834 ### Authentication errors in sovereign clouds
835
836 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_cognitive_services.md
+124 -5
@@ -195,7 +195,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
195 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
196 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
197 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
198 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
198 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
199 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
200 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
201 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -203,7 +203,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
203 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
204 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
205 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
206 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
206 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
207 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
208
209 <a id="option-collection-query-offset"></a>
210 ##### query_offset
@@ -238,7 +239,7 @@ Controls how the collector finds candidate Azure resources.
239 | Mode | Behavior |
240 |:-----|:---------|
241 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
241 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
242 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
243
244
245 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -257,6 +258,8 @@ The query **must** project these five columns:
258 | `resourceGroup` | Resource group name |
259 | `location` | Azure region |
260
261 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
262 +
263 :::info
264
265 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -270,7 +273,7 @@ Example:
273 ```
274 resources
275 | where tags.env =~ "prod"
273 -| project id, name, type, resourceGroup, location
276 +| project id, name, type, resourceGroup, location, tags
277 ```
278
279
@@ -344,6 +347,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
347 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
348
349
350 +<a id="option-virtual-node-vnode"></a>
351 +##### vnode
352 +
353 +This job-level virtual node is used for metrics written to the default host scope.
354 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
355 +
356 +
357 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
358 +##### virtual_nodes.by_resource_tag
359 +
360 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
361 +
362 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
363 +- Empty or missing tag values use the default job host scope.
364 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
365 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
366 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
367 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
368 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
369 +
370 +
371
372 </details>
373
@@ -415,6 +439,85 @@ jobs:
439 client_secret: "your-client-secret"
440
441 ```
442 +###### Workload virtual nodes from resource tags
443 +
444 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
445 +
446 +<details open><summary>Config</summary>
447 +
448 +```yaml
449 +jobs:
450 + - name: prod-workloads
451 + subscription_ids:
452 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
453 + virtual_nodes:
454 + by_resource_tag: workload
455 + discovery:
456 + mode: filters
457 + profiles:
458 + mode: auto
459 + auth:
460 + mode: service_principal
461 + mode_service_principal:
462 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
463 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
464 + client_secret: "your-client-secret"
465 +
466 +```
467 +</details>
468 +
469 +###### Custom KQL with workload virtual nodes
470 +
471 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
472 +
473 +<details open><summary>Config</summary>
474 +
475 +```yaml
476 +jobs:
477 + - name: prod-query-workloads
478 + subscription_ids:
479 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
480 + virtual_nodes:
481 + by_resource_tag: workload
482 + discovery:
483 + mode: query
484 + mode_query:
485 + kql: |
486 + resources
487 + | where tags.env =~ "prod"
488 + | project id, name, type, resourceGroup, location, tags
489 + profiles:
490 + mode: auto
491 + auth:
492 + mode: default
493 +
494 +```
495 +</details>
496 +
497 +###### Job virtual node plus workload virtual nodes
498 +
499 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
500 +
501 +<details open><summary>Config</summary>
502 +
503 +```yaml
504 +jobs:
505 + - name: prod-with-fallback-node
506 + vnode: azure-fallback-node
507 + subscription_ids:
508 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
509 + virtual_nodes:
510 + by_resource_tag: workload
511 + discovery:
512 + mode: filters
513 + profiles:
514 + mode: auto
515 + auth:
516 + mode: managed_identity
517 +
518 +```
519 +</details>
520 +
521 ###### Managed identity with exact profiles
522
523 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -494,7 +597,7 @@ jobs:
597 kql: |
598 resources
599 | where tags.env =~ "prod"
497 - | project id, name, type, resourceGroup, location
600 + | project id, name, type, resourceGroup, location, tags
601 profiles:
602 mode: auto
603 auth:
@@ -762,6 +865,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
865 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
866
867
868 +### Workload virtual nodes are not created
869 +
870 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
871 +
872 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
873 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
874 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
875 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
876 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
877 +
878 +
879 +### More alerts appear after enabling workload virtual nodes
880 +
881 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
882 +
883 +
884 ### Authentication errors in sovereign clouds
885
886 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_container_apps.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -700,6 +803,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
803 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
804
805
806 +### Workload virtual nodes are not created
807 +
808 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
809 +
810 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
811 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
812 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
813 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
814 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
815 +
816 +
817 +### More alerts appear after enabling workload virtual nodes
818 +
819 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
820 +
821 +
822 ### Authentication errors in sovereign clouds
823
824 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_container_instances.md
+124 -5
@@ -186,7 +186,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
186 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
187 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
189 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
189 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
190 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
191 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
192 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -194,7 +194,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
194 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
195 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
196 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
197 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
197 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
199
200 <a id="option-collection-query-offset"></a>
201 ##### query_offset
@@ -229,7 +230,7 @@ Controls how the collector finds candidate Azure resources.
230 | Mode | Behavior |
231 |:-----|:---------|
232 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
232 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
233 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
234
235
236 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -248,6 +249,8 @@ The query **must** project these five columns:
249 | `resourceGroup` | Resource group name |
250 | `location` | Azure region |
251
252 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
253 +
254 :::info
255
256 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -261,7 +264,7 @@ Example:
264 ```
265 resources
266 | where tags.env =~ "prod"
264 -| project id, name, type, resourceGroup, location
267 +| project id, name, type, resourceGroup, location, tags
268 ```
269
270
@@ -335,6 +338,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
338 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
339
340
341 +<a id="option-virtual-node-vnode"></a>
342 +##### vnode
343 +
344 +This job-level virtual node is used for metrics written to the default host scope.
345 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
346 +
347 +
348 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
349 +##### virtual_nodes.by_resource_tag
350 +
351 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
352 +
353 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
354 +- Empty or missing tag values use the default job host scope.
355 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
356 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
357 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
358 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
359 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
360 +
361 +
362
363 </details>
364
@@ -406,6 +430,85 @@ jobs:
430 client_secret: "your-client-secret"
431
432 ```
433 +###### Workload virtual nodes from resource tags
434 +
435 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
436 +
437 +<details open><summary>Config</summary>
438 +
439 +```yaml
440 +jobs:
441 + - name: prod-workloads
442 + subscription_ids:
443 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
444 + virtual_nodes:
445 + by_resource_tag: workload
446 + discovery:
447 + mode: filters
448 + profiles:
449 + mode: auto
450 + auth:
451 + mode: service_principal
452 + mode_service_principal:
453 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
454 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_secret: "your-client-secret"
456 +
457 +```
458 +</details>
459 +
460 +###### Custom KQL with workload virtual nodes
461 +
462 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
463 +
464 +<details open><summary>Config</summary>
465 +
466 +```yaml
467 +jobs:
468 + - name: prod-query-workloads
469 + subscription_ids:
470 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
471 + virtual_nodes:
472 + by_resource_tag: workload
473 + discovery:
474 + mode: query
475 + mode_query:
476 + kql: |
477 + resources
478 + | where tags.env =~ "prod"
479 + | project id, name, type, resourceGroup, location, tags
480 + profiles:
481 + mode: auto
482 + auth:
483 + mode: default
484 +
485 +```
486 +</details>
487 +
488 +###### Job virtual node plus workload virtual nodes
489 +
490 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
491 +
492 +<details open><summary>Config</summary>
493 +
494 +```yaml
495 +jobs:
496 + - name: prod-with-fallback-node
497 + vnode: azure-fallback-node
498 + subscription_ids:
499 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
500 + virtual_nodes:
501 + by_resource_tag: workload
502 + discovery:
503 + mode: filters
504 + profiles:
505 + mode: auto
506 + auth:
507 + mode: managed_identity
508 +
509 +```
510 +</details>
511 +
512 ###### Managed identity with exact profiles
513
514 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -485,7 +588,7 @@ jobs:
588 kql: |
589 resources
590 | where tags.env =~ "prod"
488 - | project id, name, type, resourceGroup, location
591 + | project id, name, type, resourceGroup, location, tags
592 profiles:
593 mode: auto
594 auth:
@@ -670,6 +773,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
773 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
774
775
776 +### Workload virtual nodes are not created
777 +
778 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
779 +
780 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
781 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
782 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
783 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
784 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
785 +
786 +
787 +### More alerts appear after enabling workload virtual nodes
788 +
789 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
790 +
791 +
792 ### Authentication errors in sovereign clouds
793
794 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_container_registry.md
+124 -5
@@ -186,7 +186,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
186 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
187 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
189 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
189 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
190 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
191 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
192 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -194,7 +194,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
194 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
195 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
196 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
197 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
197 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
199
200 <a id="option-collection-query-offset"></a>
201 ##### query_offset
@@ -229,7 +230,7 @@ Controls how the collector finds candidate Azure resources.
230 | Mode | Behavior |
231 |:-----|:---------|
232 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
232 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
233 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
234
235
236 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -248,6 +249,8 @@ The query **must** project these five columns:
249 | `resourceGroup` | Resource group name |
250 | `location` | Azure region |
251
252 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
253 +
254 :::info
255
256 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -261,7 +264,7 @@ Example:
264 ```
265 resources
266 | where tags.env =~ "prod"
264 -| project id, name, type, resourceGroup, location
267 +| project id, name, type, resourceGroup, location, tags
268 ```
269
270
@@ -335,6 +338,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
338 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
339
340
341 +<a id="option-virtual-node-vnode"></a>
342 +##### vnode
343 +
344 +This job-level virtual node is used for metrics written to the default host scope.
345 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
346 +
347 +
348 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
349 +##### virtual_nodes.by_resource_tag
350 +
351 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
352 +
353 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
354 +- Empty or missing tag values use the default job host scope.
355 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
356 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
357 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
358 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
359 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
360 +
361 +
362
363 </details>
364
@@ -406,6 +430,85 @@ jobs:
430 client_secret: "your-client-secret"
431
432 ```
433 +###### Workload virtual nodes from resource tags
434 +
435 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
436 +
437 +<details open><summary>Config</summary>
438 +
439 +```yaml
440 +jobs:
441 + - name: prod-workloads
442 + subscription_ids:
443 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
444 + virtual_nodes:
445 + by_resource_tag: workload
446 + discovery:
447 + mode: filters
448 + profiles:
449 + mode: auto
450 + auth:
451 + mode: service_principal
452 + mode_service_principal:
453 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
454 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_secret: "your-client-secret"
456 +
457 +```
458 +</details>
459 +
460 +###### Custom KQL with workload virtual nodes
461 +
462 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
463 +
464 +<details open><summary>Config</summary>
465 +
466 +```yaml
467 +jobs:
468 + - name: prod-query-workloads
469 + subscription_ids:
470 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
471 + virtual_nodes:
472 + by_resource_tag: workload
473 + discovery:
474 + mode: query
475 + mode_query:
476 + kql: |
477 + resources
478 + | where tags.env =~ "prod"
479 + | project id, name, type, resourceGroup, location, tags
480 + profiles:
481 + mode: auto
482 + auth:
483 + mode: default
484 +
485 +```
486 +</details>
487 +
488 +###### Job virtual node plus workload virtual nodes
489 +
490 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
491 +
492 +<details open><summary>Config</summary>
493 +
494 +```yaml
495 +jobs:
496 + - name: prod-with-fallback-node
497 + vnode: azure-fallback-node
498 + subscription_ids:
499 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
500 + virtual_nodes:
501 + by_resource_tag: workload
502 + discovery:
503 + mode: filters
504 + profiles:
505 + mode: auto
506 + auth:
507 + mode: managed_identity
508 +
509 +```
510 +</details>
511 +
512 ###### Managed identity with exact profiles
513
514 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -485,7 +588,7 @@ jobs:
588 kql: |
589 resources
590 | where tags.env =~ "prod"
488 - | project id, name, type, resourceGroup, location
591 + | project id, name, type, resourceGroup, location, tags
592 profiles:
593 mode: auto
594 auth:
@@ -670,6 +773,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
773 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
774
775
776 +### Workload virtual nodes are not created
777 +
778 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
779 +
780 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
781 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
782 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
783 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
784 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
785 +
786 +
787 +### More alerts appear after enabling workload virtual nodes
788 +
789 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
790 +
791 +
792 ### Authentication errors in sovereign clouds
793
794 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_cosmos_db_account.md
+124 -5
@@ -189,7 +189,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
189 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
192 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
192 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
193 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
194 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
195 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -197,7 +197,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
197 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
198 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
199 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
200 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
202
203 <a id="option-collection-query-offset"></a>
204 ##### query_offset
@@ -232,7 +233,7 @@ Controls how the collector finds candidate Azure resources.
233 | Mode | Behavior |
234 |:-----|:---------|
235 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
235 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
236 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
237
238
239 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -251,6 +252,8 @@ The query **must** project these five columns:
252 | `resourceGroup` | Resource group name |
253 | `location` | Azure region |
254
255 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
256 +
257 :::info
258
259 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -264,7 +267,7 @@ Example:
267 ```
268 resources
269 | where tags.env =~ "prod"
267 -| project id, name, type, resourceGroup, location
270 +| project id, name, type, resourceGroup, location, tags
271 ```
272
273
@@ -338,6 +341,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
341 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
342
343
344 +<a id="option-virtual-node-vnode"></a>
345 +##### vnode
346 +
347 +This job-level virtual node is used for metrics written to the default host scope.
348 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
349 +
350 +
351 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
352 +##### virtual_nodes.by_resource_tag
353 +
354 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
355 +
356 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
357 +- Empty or missing tag values use the default job host scope.
358 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
359 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
360 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
361 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
362 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
363 +
364 +
365
366 </details>
367
@@ -409,6 +433,85 @@ jobs:
433 client_secret: "your-client-secret"
434
435 ```
436 +###### Workload virtual nodes from resource tags
437 +
438 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
439 +
440 +<details open><summary>Config</summary>
441 +
442 +```yaml
443 +jobs:
444 + - name: prod-workloads
445 + subscription_ids:
446 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
447 + virtual_nodes:
448 + by_resource_tag: workload
449 + discovery:
450 + mode: filters
451 + profiles:
452 + mode: auto
453 + auth:
454 + mode: service_principal
455 + mode_service_principal:
456 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_secret: "your-client-secret"
459 +
460 +```
461 +</details>
462 +
463 +###### Custom KQL with workload virtual nodes
464 +
465 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
466 +
467 +<details open><summary>Config</summary>
468 +
469 +```yaml
470 +jobs:
471 + - name: prod-query-workloads
472 + subscription_ids:
473 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
474 + virtual_nodes:
475 + by_resource_tag: workload
476 + discovery:
477 + mode: query
478 + mode_query:
479 + kql: |
480 + resources
481 + | where tags.env =~ "prod"
482 + | project id, name, type, resourceGroup, location, tags
483 + profiles:
484 + mode: auto
485 + auth:
486 + mode: default
487 +
488 +```
489 +</details>
490 +
491 +###### Job virtual node plus workload virtual nodes
492 +
493 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
494 +
495 +<details open><summary>Config</summary>
496 +
497 +```yaml
498 +jobs:
499 + - name: prod-with-fallback-node
500 + vnode: azure-fallback-node
501 + subscription_ids:
502 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
503 + virtual_nodes:
504 + by_resource_tag: workload
505 + discovery:
506 + mode: filters
507 + profiles:
508 + mode: auto
509 + auth:
510 + mode: managed_identity
511 +
512 +```
513 +</details>
514 +
515 ###### Managed identity with exact profiles
516
517 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -488,7 +591,7 @@ jobs:
591 kql: |
592 resources
593 | where tags.env =~ "prod"
491 - | project id, name, type, resourceGroup, location
594 + | project id, name, type, resourceGroup, location, tags
595 profiles:
596 mode: auto
597 auth:
@@ -696,6 +799,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
799 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
800
801
802 +### Workload virtual nodes are not created
803 +
804 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
805 +
806 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
807 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
808 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
809 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
810 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
811 +
812 +
813 +### More alerts appear after enabling workload virtual nodes
814 +
815 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
816 +
817 +
818 ### Authentication errors in sovereign clouds
819
820 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_data_explorer_cluster.md
+124 -5
@@ -193,7 +193,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
193 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
195 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
196 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
196 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
197 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
198 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
199 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -201,7 +201,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
201 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
202 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
203 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
204 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
205 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
206
207 <a id="option-collection-query-offset"></a>
208 ##### query_offset
@@ -236,7 +237,7 @@ Controls how the collector finds candidate Azure resources.
237 | Mode | Behavior |
238 |:-----|:---------|
239 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
239 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
240 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
241
242
243 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -255,6 +256,8 @@ The query **must** project these five columns:
256 | `resourceGroup` | Resource group name |
257 | `location` | Azure region |
258
259 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
260 +
261 :::info
262
263 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -268,7 +271,7 @@ Example:
271 ```
272 resources
273 | where tags.env =~ "prod"
271 -| project id, name, type, resourceGroup, location
274 +| project id, name, type, resourceGroup, location, tags
275 ```
276
277
@@ -342,6 +345,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
345 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
346
347
348 +<a id="option-virtual-node-vnode"></a>
349 +##### vnode
350 +
351 +This job-level virtual node is used for metrics written to the default host scope.
352 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
353 +
354 +
355 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
356 +##### virtual_nodes.by_resource_tag
357 +
358 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
359 +
360 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
361 +- Empty or missing tag values use the default job host scope.
362 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
363 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
364 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
365 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
366 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
367 +
368 +
369
370 </details>
371
@@ -413,6 +437,85 @@ jobs:
437 client_secret: "your-client-secret"
438
439 ```
440 +###### Workload virtual nodes from resource tags
441 +
442 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
443 +
444 +<details open><summary>Config</summary>
445 +
446 +```yaml
447 +jobs:
448 + - name: prod-workloads
449 + subscription_ids:
450 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
451 + virtual_nodes:
452 + by_resource_tag: workload
453 + discovery:
454 + mode: filters
455 + profiles:
456 + mode: auto
457 + auth:
458 + mode: service_principal
459 + mode_service_principal:
460 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
462 + client_secret: "your-client-secret"
463 +
464 +```
465 +</details>
466 +
467 +###### Custom KQL with workload virtual nodes
468 +
469 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
470 +
471 +<details open><summary>Config</summary>
472 +
473 +```yaml
474 +jobs:
475 + - name: prod-query-workloads
476 + subscription_ids:
477 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
478 + virtual_nodes:
479 + by_resource_tag: workload
480 + discovery:
481 + mode: query
482 + mode_query:
483 + kql: |
484 + resources
485 + | where tags.env =~ "prod"
486 + | project id, name, type, resourceGroup, location, tags
487 + profiles:
488 + mode: auto
489 + auth:
490 + mode: default
491 +
492 +```
493 +</details>
494 +
495 +###### Job virtual node plus workload virtual nodes
496 +
497 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
498 +
499 +<details open><summary>Config</summary>
500 +
501 +```yaml
502 +jobs:
503 + - name: prod-with-fallback-node
504 + vnode: azure-fallback-node
505 + subscription_ids:
506 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
507 + virtual_nodes:
508 + by_resource_tag: workload
509 + discovery:
510 + mode: filters
511 + profiles:
512 + mode: auto
513 + auth:
514 + mode: managed_identity
515 +
516 +```
517 +</details>
518 +
519 ###### Managed identity with exact profiles
520
521 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -492,7 +595,7 @@ jobs:
595 kql: |
596 resources
597 | where tags.env =~ "prod"
495 - | project id, name, type, resourceGroup, location
598 + | project id, name, type, resourceGroup, location, tags
599 profiles:
600 mode: auto
601 auth:
@@ -735,6 +838,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
838 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
839
840
841 +### Workload virtual nodes are not created
842 +
843 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
844 +
845 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
846 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
847 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
848 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
849 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
850 +
851 +
852 +### More alerts appear after enabling workload virtual nodes
853 +
854 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
855 +
856 +
857 ### Authentication errors in sovereign clouds
858
859 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_data_factory.md
+124 -5
@@ -191,7 +191,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
194 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
194 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
195 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
196 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
197 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -199,7 +199,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
199 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
200 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
201 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
202 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
204
205 <a id="option-collection-query-offset"></a>
206 ##### query_offset
@@ -234,7 +235,7 @@ Controls how the collector finds candidate Azure resources.
235 | Mode | Behavior |
236 |:-----|:---------|
237 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
237 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
238 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
239
240
241 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -253,6 +254,8 @@ The query **must** project these five columns:
254 | `resourceGroup` | Resource group name |
255 | `location` | Azure region |
256
257 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
258 +
259 :::info
260
261 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -266,7 +269,7 @@ Example:
269 ```
270 resources
271 | where tags.env =~ "prod"
269 -| project id, name, type, resourceGroup, location
272 +| project id, name, type, resourceGroup, location, tags
273 ```
274
275
@@ -340,6 +343,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
343 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
344
345
346 +<a id="option-virtual-node-vnode"></a>
347 +##### vnode
348 +
349 +This job-level virtual node is used for metrics written to the default host scope.
350 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
351 +
352 +
353 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
354 +##### virtual_nodes.by_resource_tag
355 +
356 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
357 +
358 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
359 +- Empty or missing tag values use the default job host scope.
360 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
361 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
362 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
363 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
364 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
365 +
366 +
367
368 </details>
369
@@ -411,6 +435,85 @@ jobs:
435 client_secret: "your-client-secret"
436
437 ```
438 +###### Workload virtual nodes from resource tags
439 +
440 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
441 +
442 +<details open><summary>Config</summary>
443 +
444 +```yaml
445 +jobs:
446 + - name: prod-workloads
447 + subscription_ids:
448 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
449 + virtual_nodes:
450 + by_resource_tag: workload
451 + discovery:
452 + mode: filters
453 + profiles:
454 + mode: auto
455 + auth:
456 + mode: service_principal
457 + mode_service_principal:
458 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_secret: "your-client-secret"
461 +
462 +```
463 +</details>
464 +
465 +###### Custom KQL with workload virtual nodes
466 +
467 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
468 +
469 +<details open><summary>Config</summary>
470 +
471 +```yaml
472 +jobs:
473 + - name: prod-query-workloads
474 + subscription_ids:
475 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
476 + virtual_nodes:
477 + by_resource_tag: workload
478 + discovery:
479 + mode: query
480 + mode_query:
481 + kql: |
482 + resources
483 + | where tags.env =~ "prod"
484 + | project id, name, type, resourceGroup, location, tags
485 + profiles:
486 + mode: auto
487 + auth:
488 + mode: default
489 +
490 +```
491 +</details>
492 +
493 +###### Job virtual node plus workload virtual nodes
494 +
495 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
496 +
497 +<details open><summary>Config</summary>
498 +
499 +```yaml
500 +jobs:
501 + - name: prod-with-fallback-node
502 + vnode: azure-fallback-node
503 + subscription_ids:
504 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
505 + virtual_nodes:
506 + by_resource_tag: workload
507 + discovery:
508 + mode: filters
509 + profiles:
510 + mode: auto
511 + auth:
512 + mode: managed_identity
513 +
514 +```
515 +</details>
516 +
517 ###### Managed identity with exact profiles
518
519 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -490,7 +593,7 @@ jobs:
593 kql: |
594 resources
595 | where tags.env =~ "prod"
493 - | project id, name, type, resourceGroup, location
596 + | project id, name, type, resourceGroup, location, tags
597 profiles:
598 mode: auto
599 auth:
@@ -742,6 +845,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
845 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
846
847
848 +### Workload virtual nodes are not created
849 +
850 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
851 +
852 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
853 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
854 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
855 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
856 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
857 +
858 +
859 +### More alerts appear after enabling workload virtual nodes
860 +
861 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
862 +
863 +
864 ### Authentication errors in sovereign clouds
865
866 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_event_grid_topic.md
+124 -5
@@ -187,7 +187,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
187 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
190 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
190 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
191 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
192 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
193 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -195,7 +195,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
195 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
196 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
197 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
198 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
200
201 <a id="option-collection-query-offset"></a>
202 ##### query_offset
@@ -230,7 +231,7 @@ Controls how the collector finds candidate Azure resources.
231 | Mode | Behavior |
232 |:-----|:---------|
233 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
233 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
234 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
235
236
237 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -249,6 +250,8 @@ The query **must** project these five columns:
250 | `resourceGroup` | Resource group name |
251 | `location` | Azure region |
252
253 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
254 +
255 :::info
256
257 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -262,7 +265,7 @@ Example:
265 ```
266 resources
267 | where tags.env =~ "prod"
265 -| project id, name, type, resourceGroup, location
268 +| project id, name, type, resourceGroup, location, tags
269 ```
270
271
@@ -336,6 +339,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
339 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
340
341
342 +<a id="option-virtual-node-vnode"></a>
343 +##### vnode
344 +
345 +This job-level virtual node is used for metrics written to the default host scope.
346 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
347 +
348 +
349 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
350 +##### virtual_nodes.by_resource_tag
351 +
352 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
353 +
354 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
355 +- Empty or missing tag values use the default job host scope.
356 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
357 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
358 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
359 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
360 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
361 +
362 +
363
364 </details>
365
@@ -407,6 +431,85 @@ jobs:
431 client_secret: "your-client-secret"
432
433 ```
434 +###### Workload virtual nodes from resource tags
435 +
436 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
437 +
438 +<details open><summary>Config</summary>
439 +
440 +```yaml
441 +jobs:
442 + - name: prod-workloads
443 + subscription_ids:
444 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
445 + virtual_nodes:
446 + by_resource_tag: workload
447 + discovery:
448 + mode: filters
449 + profiles:
450 + mode: auto
451 + auth:
452 + mode: service_principal
453 + mode_service_principal:
454 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_secret: "your-client-secret"
457 +
458 +```
459 +</details>
460 +
461 +###### Custom KQL with workload virtual nodes
462 +
463 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
464 +
465 +<details open><summary>Config</summary>
466 +
467 +```yaml
468 +jobs:
469 + - name: prod-query-workloads
470 + subscription_ids:
471 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
472 + virtual_nodes:
473 + by_resource_tag: workload
474 + discovery:
475 + mode: query
476 + mode_query:
477 + kql: |
478 + resources
479 + | where tags.env =~ "prod"
480 + | project id, name, type, resourceGroup, location, tags
481 + profiles:
482 + mode: auto
483 + auth:
484 + mode: default
485 +
486 +```
487 +</details>
488 +
489 +###### Job virtual node plus workload virtual nodes
490 +
491 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
492 +
493 +<details open><summary>Config</summary>
494 +
495 +```yaml
496 +jobs:
497 + - name: prod-with-fallback-node
498 + vnode: azure-fallback-node
499 + subscription_ids:
500 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
501 + virtual_nodes:
502 + by_resource_tag: workload
503 + discovery:
504 + mode: filters
505 + profiles:
506 + mode: auto
507 + auth:
508 + mode: managed_identity
509 +
510 +```
511 +</details>
512 +
513 ###### Managed identity with exact profiles
514
515 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -486,7 +589,7 @@ jobs:
589 kql: |
590 resources
591 | where tags.env =~ "prod"
489 - | project id, name, type, resourceGroup, location
592 + | project id, name, type, resourceGroup, location, tags
593 profiles:
594 mode: auto
595 auth:
@@ -676,6 +779,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
779 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
780
781
782 +### Workload virtual nodes are not created
783 +
784 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
785 +
786 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
787 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
788 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
789 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
790 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
791 +
792 +
793 +### More alerts appear after enabling workload virtual nodes
794 +
795 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
796 +
797 +
798 ### Authentication errors in sovereign clouds
799
800 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_event_hubs_namespace.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -690,6 +793,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
793 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
794
795
796 +### Workload virtual nodes are not created
797 +
798 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
799 +
800 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
801 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
802 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
803 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
804 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
805 +
806 +
807 +### More alerts appear after enabling workload virtual nodes
808 +
809 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
810 +
811 +
812 ### Authentication errors in sovereign clouds
813
814 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_expressroute_circuit.md
+124 -5
@@ -188,7 +188,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
188 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
191 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
191 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
192 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
193 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
194 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -196,7 +196,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
196 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
197 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
198 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
199 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
201
202 <a id="option-collection-query-offset"></a>
203 ##### query_offset
@@ -231,7 +232,7 @@ Controls how the collector finds candidate Azure resources.
232 | Mode | Behavior |
233 |:-----|:---------|
234 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
234 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
235 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
236
237
238 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -250,6 +251,8 @@ The query **must** project these five columns:
251 | `resourceGroup` | Resource group name |
252 | `location` | Azure region |
253
254 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
255 +
256 :::info
257
258 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -263,7 +266,7 @@ Example:
266 ```
267 resources
268 | where tags.env =~ "prod"
266 -| project id, name, type, resourceGroup, location
269 +| project id, name, type, resourceGroup, location, tags
270 ```
271
272
@@ -337,6 +340,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
340 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
341
342
343 +<a id="option-virtual-node-vnode"></a>
344 +##### vnode
345 +
346 +This job-level virtual node is used for metrics written to the default host scope.
347 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
348 +
349 +
350 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
351 +##### virtual_nodes.by_resource_tag
352 +
353 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
354 +
355 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
356 +- Empty or missing tag values use the default job host scope.
357 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
358 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
359 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
360 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
361 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
362 +
363 +
364
365 </details>
366
@@ -408,6 +432,85 @@ jobs:
432 client_secret: "your-client-secret"
433
434 ```
435 +###### Workload virtual nodes from resource tags
436 +
437 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
438 +
439 +<details open><summary>Config</summary>
440 +
441 +```yaml
442 +jobs:
443 + - name: prod-workloads
444 + subscription_ids:
445 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
446 + virtual_nodes:
447 + by_resource_tag: workload
448 + discovery:
449 + mode: filters
450 + profiles:
451 + mode: auto
452 + auth:
453 + mode: service_principal
454 + mode_service_principal:
455 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_secret: "your-client-secret"
458 +
459 +```
460 +</details>
461 +
462 +###### Custom KQL with workload virtual nodes
463 +
464 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
465 +
466 +<details open><summary>Config</summary>
467 +
468 +```yaml
469 +jobs:
470 + - name: prod-query-workloads
471 + subscription_ids:
472 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
473 + virtual_nodes:
474 + by_resource_tag: workload
475 + discovery:
476 + mode: query
477 + mode_query:
478 + kql: |
479 + resources
480 + | where tags.env =~ "prod"
481 + | project id, name, type, resourceGroup, location, tags
482 + profiles:
483 + mode: auto
484 + auth:
485 + mode: default
486 +
487 +```
488 +</details>
489 +
490 +###### Job virtual node plus workload virtual nodes
491 +
492 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
493 +
494 +<details open><summary>Config</summary>
495 +
496 +```yaml
497 +jobs:
498 + - name: prod-with-fallback-node
499 + vnode: azure-fallback-node
500 + subscription_ids:
501 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
502 + virtual_nodes:
503 + by_resource_tag: workload
504 + discovery:
505 + mode: filters
506 + profiles:
507 + mode: auto
508 + auth:
509 + mode: managed_identity
510 +
511 +```
512 +</details>
513 +
514 ###### Managed identity with exact profiles
515
516 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -487,7 +590,7 @@ jobs:
590 kql: |
591 resources
592 | where tags.env =~ "prod"
490 - | project id, name, type, resourceGroup, location
593 + | project id, name, type, resourceGroup, location, tags
594 profiles:
595 mode: auto
596 auth:
@@ -678,6 +781,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
781 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
782
783
784 +### Workload virtual nodes are not created
785 +
786 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
787 +
788 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
789 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
790 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
791 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
792 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
793 +
794 +
795 +### More alerts appear after enabling workload virtual nodes
796 +
797 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
798 +
799 +
800 ### Authentication errors in sovereign clouds
801
802 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_expressroute_gateway.md
+124 -5
@@ -188,7 +188,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
188 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
191 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
191 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
192 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
193 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
194 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -196,7 +196,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
196 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
197 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
198 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
199 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
201
202 <a id="option-collection-query-offset"></a>
203 ##### query_offset
@@ -231,7 +232,7 @@ Controls how the collector finds candidate Azure resources.
232 | Mode | Behavior |
233 |:-----|:---------|
234 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
234 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
235 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
236
237
238 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -250,6 +251,8 @@ The query **must** project these five columns:
251 | `resourceGroup` | Resource group name |
252 | `location` | Azure region |
253
254 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
255 +
256 :::info
257
258 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -263,7 +266,7 @@ Example:
266 ```
267 resources
268 | where tags.env =~ "prod"
266 -| project id, name, type, resourceGroup, location
269 +| project id, name, type, resourceGroup, location, tags
270 ```
271
272
@@ -337,6 +340,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
340 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
341
342
343 +<a id="option-virtual-node-vnode"></a>
344 +##### vnode
345 +
346 +This job-level virtual node is used for metrics written to the default host scope.
347 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
348 +
349 +
350 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
351 +##### virtual_nodes.by_resource_tag
352 +
353 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
354 +
355 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
356 +- Empty or missing tag values use the default job host scope.
357 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
358 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
359 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
360 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
361 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
362 +
363 +
364
365 </details>
366
@@ -408,6 +432,85 @@ jobs:
432 client_secret: "your-client-secret"
433
434 ```
435 +###### Workload virtual nodes from resource tags
436 +
437 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
438 +
439 +<details open><summary>Config</summary>
440 +
441 +```yaml
442 +jobs:
443 + - name: prod-workloads
444 + subscription_ids:
445 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
446 + virtual_nodes:
447 + by_resource_tag: workload
448 + discovery:
449 + mode: filters
450 + profiles:
451 + mode: auto
452 + auth:
453 + mode: service_principal
454 + mode_service_principal:
455 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_secret: "your-client-secret"
458 +
459 +```
460 +</details>
461 +
462 +###### Custom KQL with workload virtual nodes
463 +
464 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
465 +
466 +<details open><summary>Config</summary>
467 +
468 +```yaml
469 +jobs:
470 + - name: prod-query-workloads
471 + subscription_ids:
472 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
473 + virtual_nodes:
474 + by_resource_tag: workload
475 + discovery:
476 + mode: query
477 + mode_query:
478 + kql: |
479 + resources
480 + | where tags.env =~ "prod"
481 + | project id, name, type, resourceGroup, location, tags
482 + profiles:
483 + mode: auto
484 + auth:
485 + mode: default
486 +
487 +```
488 +</details>
489 +
490 +###### Job virtual node plus workload virtual nodes
491 +
492 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
493 +
494 +<details open><summary>Config</summary>
495 +
496 +```yaml
497 +jobs:
498 + - name: prod-with-fallback-node
499 + vnode: azure-fallback-node
500 + subscription_ids:
501 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
502 + virtual_nodes:
503 + by_resource_tag: workload
504 + discovery:
505 + mode: filters
506 + profiles:
507 + mode: auto
508 + auth:
509 + mode: managed_identity
510 +
511 +```
512 +</details>
513 +
514 ###### Managed identity with exact profiles
515
516 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -487,7 +590,7 @@ jobs:
590 kql: |
591 resources
592 | where tags.env =~ "prod"
490 - | project id, name, type, resourceGroup, location
593 + | project id, name, type, resourceGroup, location, tags
594 profiles:
595 mode: auto
596 auth:
@@ -680,6 +783,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
783 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
784
785
786 +### Workload virtual nodes are not created
787 +
788 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
789 +
790 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
791 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
792 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
793 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
794 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
795 +
796 +
797 +### More alerts appear after enabling workload virtual nodes
798 +
799 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
800 +
801 +
802 ### Authentication errors in sovereign clouds
803
804 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_firewall.md
+124 -5
@@ -189,7 +189,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
189 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
192 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
192 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
193 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
194 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
195 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -197,7 +197,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
197 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
198 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
199 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
200 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
202
203 <a id="option-collection-query-offset"></a>
204 ##### query_offset
@@ -232,7 +233,7 @@ Controls how the collector finds candidate Azure resources.
233 | Mode | Behavior |
234 |:-----|:---------|
235 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
235 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
236 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
237
238
239 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -251,6 +252,8 @@ The query **must** project these five columns:
252 | `resourceGroup` | Resource group name |
253 | `location` | Azure region |
254
255 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
256 +
257 :::info
258
259 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -264,7 +267,7 @@ Example:
267 ```
268 resources
269 | where tags.env =~ "prod"
267 -| project id, name, type, resourceGroup, location
270 +| project id, name, type, resourceGroup, location, tags
271 ```
272
273
@@ -338,6 +341,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
341 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
342
343
344 +<a id="option-virtual-node-vnode"></a>
345 +##### vnode
346 +
347 +This job-level virtual node is used for metrics written to the default host scope.
348 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
349 +
350 +
351 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
352 +##### virtual_nodes.by_resource_tag
353 +
354 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
355 +
356 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
357 +- Empty or missing tag values use the default job host scope.
358 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
359 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
360 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
361 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
362 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
363 +
364 +
365
366 </details>
367
@@ -409,6 +433,85 @@ jobs:
433 client_secret: "your-client-secret"
434
435 ```
436 +###### Workload virtual nodes from resource tags
437 +
438 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
439 +
440 +<details open><summary>Config</summary>
441 +
442 +```yaml
443 +jobs:
444 + - name: prod-workloads
445 + subscription_ids:
446 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
447 + virtual_nodes:
448 + by_resource_tag: workload
449 + discovery:
450 + mode: filters
451 + profiles:
452 + mode: auto
453 + auth:
454 + mode: service_principal
455 + mode_service_principal:
456 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_secret: "your-client-secret"
459 +
460 +```
461 +</details>
462 +
463 +###### Custom KQL with workload virtual nodes
464 +
465 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
466 +
467 +<details open><summary>Config</summary>
468 +
469 +```yaml
470 +jobs:
471 + - name: prod-query-workloads
472 + subscription_ids:
473 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
474 + virtual_nodes:
475 + by_resource_tag: workload
476 + discovery:
477 + mode: query
478 + mode_query:
479 + kql: |
480 + resources
481 + | where tags.env =~ "prod"
482 + | project id, name, type, resourceGroup, location, tags
483 + profiles:
484 + mode: auto
485 + auth:
486 + mode: default
487 +
488 +```
489 +</details>
490 +
491 +###### Job virtual node plus workload virtual nodes
492 +
493 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
494 +
495 +<details open><summary>Config</summary>
496 +
497 +```yaml
498 +jobs:
499 + - name: prod-with-fallback-node
500 + vnode: azure-fallback-node
501 + subscription_ids:
502 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
503 + virtual_nodes:
504 + by_resource_tag: workload
505 + discovery:
506 + mode: filters
507 + profiles:
508 + mode: auto
509 + auth:
510 + mode: managed_identity
511 +
512 +```
513 +</details>
514 +
515 ###### Managed identity with exact profiles
516
517 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -488,7 +591,7 @@ jobs:
591 kql: |
592 resources
593 | where tags.env =~ "prod"
491 - | project id, name, type, resourceGroup, location
594 + | project id, name, type, resourceGroup, location, tags
595 profiles:
596 mode: auto
597 auth:
@@ -676,6 +779,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
779 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
780
781
782 +### Workload virtual nodes are not created
783 +
784 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
785 +
786 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
787 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
788 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
789 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
790 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
791 +
792 +
793 +### More alerts appear after enabling workload virtual nodes
794 +
795 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
796 +
797 +
798 ### Authentication errors in sovereign clouds
799
800 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_front_door.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -686,6 +789,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
789 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
790
791
792 +### Workload virtual nodes are not created
793 +
794 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
795 +
796 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
797 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
798 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
799 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
800 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
801 +
802 +
803 +### More alerts appear after enabling workload virtual nodes
804 +
805 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
806 +
807 +
808 ### Authentication errors in sovereign clouds
809
810 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_functions.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -689,6 +792,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
792 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
793
794
795 +### Workload virtual nodes are not created
796 +
797 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
798 +
799 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
800 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
801 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
802 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
803 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
804 +
805 +
806 +### More alerts appear after enabling workload virtual nodes
807 +
808 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
809 +
810 +
811 ### Authentication errors in sovereign clouds
812
813 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_iot_hub.md
+124 -5
@@ -192,7 +192,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
192 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
195 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
195 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
196 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
197 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
198 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -200,7 +200,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
200 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
201 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
202 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
203 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
205
206 <a id="option-collection-query-offset"></a>
207 ##### query_offset
@@ -235,7 +236,7 @@ Controls how the collector finds candidate Azure resources.
236 | Mode | Behavior |
237 |:-----|:---------|
238 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
238 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
239 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
240
241
242 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -254,6 +255,8 @@ The query **must** project these five columns:
255 | `resourceGroup` | Resource group name |
256 | `location` | Azure region |
257
258 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
259 +
260 :::info
261
262 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -267,7 +270,7 @@ Example:
270 ```
271 resources
272 | where tags.env =~ "prod"
270 -| project id, name, type, resourceGroup, location
273 +| project id, name, type, resourceGroup, location, tags
274 ```
275
276
@@ -341,6 +344,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
344 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
345
346
347 +<a id="option-virtual-node-vnode"></a>
348 +##### vnode
349 +
350 +This job-level virtual node is used for metrics written to the default host scope.
351 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
352 +
353 +
354 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
355 +##### virtual_nodes.by_resource_tag
356 +
357 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
358 +
359 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
360 +- Empty or missing tag values use the default job host scope.
361 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
362 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
363 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
364 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
365 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
366 +
367 +
368
369 </details>
370
@@ -412,6 +436,85 @@ jobs:
436 client_secret: "your-client-secret"
437
438 ```
439 +###### Workload virtual nodes from resource tags
440 +
441 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
442 +
443 +<details open><summary>Config</summary>
444 +
445 +```yaml
446 +jobs:
447 + - name: prod-workloads
448 + subscription_ids:
449 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
450 + virtual_nodes:
451 + by_resource_tag: workload
452 + discovery:
453 + mode: filters
454 + profiles:
455 + mode: auto
456 + auth:
457 + mode: service_principal
458 + mode_service_principal:
459 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_secret: "your-client-secret"
462 +
463 +```
464 +</details>
465 +
466 +###### Custom KQL with workload virtual nodes
467 +
468 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
469 +
470 +<details open><summary>Config</summary>
471 +
472 +```yaml
473 +jobs:
474 + - name: prod-query-workloads
475 + subscription_ids:
476 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
477 + virtual_nodes:
478 + by_resource_tag: workload
479 + discovery:
480 + mode: query
481 + mode_query:
482 + kql: |
483 + resources
484 + | where tags.env =~ "prod"
485 + | project id, name, type, resourceGroup, location, tags
486 + profiles:
487 + mode: auto
488 + auth:
489 + mode: default
490 +
491 +```
492 +</details>
493 +
494 +###### Job virtual node plus workload virtual nodes
495 +
496 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
497 +
498 +<details open><summary>Config</summary>
499 +
500 +```yaml
501 +jobs:
502 + - name: prod-with-fallback-node
503 + vnode: azure-fallback-node
504 + subscription_ids:
505 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
506 + virtual_nodes:
507 + by_resource_tag: workload
508 + discovery:
509 + mode: filters
510 + profiles:
511 + mode: auto
512 + auth:
513 + mode: managed_identity
514 +
515 +```
516 +</details>
517 +
518 ###### Managed identity with exact profiles
519
520 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -491,7 +594,7 @@ jobs:
594 kql: |
595 resources
596 | where tags.env =~ "prod"
494 - | project id, name, type, resourceGroup, location
597 + | project id, name, type, resourceGroup, location, tags
598 profiles:
599 mode: auto
600 auth:
@@ -725,6 +828,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
828 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
829
830
831 +### Workload virtual nodes are not created
832 +
833 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
834 +
835 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
836 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
837 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
838 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
839 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
840 +
841 +
842 +### More alerts appear after enabling workload virtual nodes
843 +
844 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
845 +
846 +
847 ### Authentication errors in sovereign clouds
848
849 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_key_vault.md
+124 -5
@@ -186,7 +186,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
186 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
187 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
189 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
189 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
190 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
191 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
192 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -194,7 +194,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
194 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
195 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
196 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
197 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
197 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
199
200 <a id="option-collection-query-offset"></a>
201 ##### query_offset
@@ -229,7 +230,7 @@ Controls how the collector finds candidate Azure resources.
230 | Mode | Behavior |
231 |:-----|:---------|
232 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
232 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
233 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
234
235
236 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -248,6 +249,8 @@ The query **must** project these five columns:
249 | `resourceGroup` | Resource group name |
250 | `location` | Azure region |
251
252 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
253 +
254 :::info
255
256 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -261,7 +264,7 @@ Example:
264 ```
265 resources
266 | where tags.env =~ "prod"
264 -| project id, name, type, resourceGroup, location
267 +| project id, name, type, resourceGroup, location, tags
268 ```
269
270
@@ -335,6 +338,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
338 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
339
340
341 +<a id="option-virtual-node-vnode"></a>
342 +##### vnode
343 +
344 +This job-level virtual node is used for metrics written to the default host scope.
345 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
346 +
347 +
348 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
349 +##### virtual_nodes.by_resource_tag
350 +
351 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
352 +
353 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
354 +- Empty or missing tag values use the default job host scope.
355 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
356 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
357 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
358 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
359 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
360 +
361 +
362
363 </details>
364
@@ -406,6 +430,85 @@ jobs:
430 client_secret: "your-client-secret"
431
432 ```
433 +###### Workload virtual nodes from resource tags
434 +
435 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
436 +
437 +<details open><summary>Config</summary>
438 +
439 +```yaml
440 +jobs:
441 + - name: prod-workloads
442 + subscription_ids:
443 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
444 + virtual_nodes:
445 + by_resource_tag: workload
446 + discovery:
447 + mode: filters
448 + profiles:
449 + mode: auto
450 + auth:
451 + mode: service_principal
452 + mode_service_principal:
453 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
454 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_secret: "your-client-secret"
456 +
457 +```
458 +</details>
459 +
460 +###### Custom KQL with workload virtual nodes
461 +
462 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
463 +
464 +<details open><summary>Config</summary>
465 +
466 +```yaml
467 +jobs:
468 + - name: prod-query-workloads
469 + subscription_ids:
470 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
471 + virtual_nodes:
472 + by_resource_tag: workload
473 + discovery:
474 + mode: query
475 + mode_query:
476 + kql: |
477 + resources
478 + | where tags.env =~ "prod"
479 + | project id, name, type, resourceGroup, location, tags
480 + profiles:
481 + mode: auto
482 + auth:
483 + mode: default
484 +
485 +```
486 +</details>
487 +
488 +###### Job virtual node plus workload virtual nodes
489 +
490 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
491 +
492 +<details open><summary>Config</summary>
493 +
494 +```yaml
495 +jobs:
496 + - name: prod-with-fallback-node
497 + vnode: azure-fallback-node
498 + subscription_ids:
499 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
500 + virtual_nodes:
501 + by_resource_tag: workload
502 + discovery:
503 + mode: filters
504 + profiles:
505 + mode: auto
506 + auth:
507 + mode: managed_identity
508 +
509 +```
510 +</details>
511 +
512 ###### Managed identity with exact profiles
513
514 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -485,7 +588,7 @@ jobs:
588 kql: |
589 resources
590 | where tags.env =~ "prod"
488 - | project id, name, type, resourceGroup, location
591 + | project id, name, type, resourceGroup, location, tags
592 profiles:
593 mode: auto
594 auth:
@@ -670,6 +773,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
773 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
774
775
776 +### Workload virtual nodes are not created
777 +
778 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
779 +
780 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
781 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
782 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
783 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
784 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
785 +
786 +
787 +### More alerts appear after enabling workload virtual nodes
788 +
789 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
790 +
791 +
792 ### Authentication errors in sovereign clouds
793
794 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_kubernetes_service_cluster.md
+124 -5
@@ -187,7 +187,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
187 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
190 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
190 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
191 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
192 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
193 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -195,7 +195,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
195 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
196 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
197 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
198 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
200
201 <a id="option-collection-query-offset"></a>
202 ##### query_offset
@@ -230,7 +231,7 @@ Controls how the collector finds candidate Azure resources.
231 | Mode | Behavior |
232 |:-----|:---------|
233 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
233 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
234 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
235
236
237 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -249,6 +250,8 @@ The query **must** project these five columns:
250 | `resourceGroup` | Resource group name |
251 | `location` | Azure region |
252
253 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
254 +
255 :::info
256
257 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -262,7 +265,7 @@ Example:
265 ```
266 resources
267 | where tags.env =~ "prod"
265 -| project id, name, type, resourceGroup, location
268 +| project id, name, type, resourceGroup, location, tags
269 ```
270
271
@@ -336,6 +339,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
339 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
340
341
342 +<a id="option-virtual-node-vnode"></a>
343 +##### vnode
344 +
345 +This job-level virtual node is used for metrics written to the default host scope.
346 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
347 +
348 +
349 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
350 +##### virtual_nodes.by_resource_tag
351 +
352 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
353 +
354 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
355 +- Empty or missing tag values use the default job host scope.
356 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
357 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
358 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
359 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
360 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
361 +
362 +
363
364 </details>
365
@@ -407,6 +431,85 @@ jobs:
431 client_secret: "your-client-secret"
432
433 ```
434 +###### Workload virtual nodes from resource tags
435 +
436 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
437 +
438 +<details open><summary>Config</summary>
439 +
440 +```yaml
441 +jobs:
442 + - name: prod-workloads
443 + subscription_ids:
444 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
445 + virtual_nodes:
446 + by_resource_tag: workload
447 + discovery:
448 + mode: filters
449 + profiles:
450 + mode: auto
451 + auth:
452 + mode: service_principal
453 + mode_service_principal:
454 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_secret: "your-client-secret"
457 +
458 +```
459 +</details>
460 +
461 +###### Custom KQL with workload virtual nodes
462 +
463 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
464 +
465 +<details open><summary>Config</summary>
466 +
467 +```yaml
468 +jobs:
469 + - name: prod-query-workloads
470 + subscription_ids:
471 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
472 + virtual_nodes:
473 + by_resource_tag: workload
474 + discovery:
475 + mode: query
476 + mode_query:
477 + kql: |
478 + resources
479 + | where tags.env =~ "prod"
480 + | project id, name, type, resourceGroup, location, tags
481 + profiles:
482 + mode: auto
483 + auth:
484 + mode: default
485 +
486 +```
487 +</details>
488 +
489 +###### Job virtual node plus workload virtual nodes
490 +
491 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
492 +
493 +<details open><summary>Config</summary>
494 +
495 +```yaml
496 +jobs:
497 + - name: prod-with-fallback-node
498 + vnode: azure-fallback-node
499 + subscription_ids:
500 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
501 + virtual_nodes:
502 + by_resource_tag: workload
503 + discovery:
504 + mode: filters
505 + profiles:
506 + mode: auto
507 + auth:
508 + mode: managed_identity
509 +
510 +```
511 +</details>
512 +
513 ###### Managed identity with exact profiles
514
515 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -486,7 +589,7 @@ jobs:
589 kql: |
590 resources
591 | where tags.env =~ "prod"
489 - | project id, name, type, resourceGroup, location
592 + | project id, name, type, resourceGroup, location, tags
593 profiles:
594 mode: auto
595 auth:
@@ -699,6 +802,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
802 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
803
804
805 +### Workload virtual nodes are not created
806 +
807 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
808 +
809 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
810 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
811 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
812 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
813 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
814 +
815 +
816 +### More alerts appear after enabling workload virtual nodes
817 +
818 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
819 +
820 +
821 ### Authentication errors in sovereign clouds
822
823 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_load_balancer.md
+124 -5
@@ -187,7 +187,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
187 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
190 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
190 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
191 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
192 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
193 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -195,7 +195,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
195 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
196 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
197 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
198 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
200
201 <a id="option-collection-query-offset"></a>
202 ##### query_offset
@@ -230,7 +231,7 @@ Controls how the collector finds candidate Azure resources.
231 | Mode | Behavior |
232 |:-----|:---------|
233 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
233 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
234 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
235
236
237 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -249,6 +250,8 @@ The query **must** project these five columns:
250 | `resourceGroup` | Resource group name |
251 | `location` | Azure region |
252
253 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
254 +
255 :::info
256
257 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -262,7 +265,7 @@ Example:
265 ```
266 resources
267 | where tags.env =~ "prod"
265 -| project id, name, type, resourceGroup, location
268 +| project id, name, type, resourceGroup, location, tags
269 ```
270
271
@@ -336,6 +339,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
339 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
340
341
342 +<a id="option-virtual-node-vnode"></a>
343 +##### vnode
344 +
345 +This job-level virtual node is used for metrics written to the default host scope.
346 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
347 +
348 +
349 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
350 +##### virtual_nodes.by_resource_tag
351 +
352 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
353 +
354 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
355 +- Empty or missing tag values use the default job host scope.
356 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
357 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
358 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
359 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
360 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
361 +
362 +
363
364 </details>
365
@@ -407,6 +431,85 @@ jobs:
431 client_secret: "your-client-secret"
432
433 ```
434 +###### Workload virtual nodes from resource tags
435 +
436 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
437 +
438 +<details open><summary>Config</summary>
439 +
440 +```yaml
441 +jobs:
442 + - name: prod-workloads
443 + subscription_ids:
444 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
445 + virtual_nodes:
446 + by_resource_tag: workload
447 + discovery:
448 + mode: filters
449 + profiles:
450 + mode: auto
451 + auth:
452 + mode: service_principal
453 + mode_service_principal:
454 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_secret: "your-client-secret"
457 +
458 +```
459 +</details>
460 +
461 +###### Custom KQL with workload virtual nodes
462 +
463 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
464 +
465 +<details open><summary>Config</summary>
466 +
467 +```yaml
468 +jobs:
469 + - name: prod-query-workloads
470 + subscription_ids:
471 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
472 + virtual_nodes:
473 + by_resource_tag: workload
474 + discovery:
475 + mode: query
476 + mode_query:
477 + kql: |
478 + resources
479 + | where tags.env =~ "prod"
480 + | project id, name, type, resourceGroup, location, tags
481 + profiles:
482 + mode: auto
483 + auth:
484 + mode: default
485 +
486 +```
487 +</details>
488 +
489 +###### Job virtual node plus workload virtual nodes
490 +
491 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
492 +
493 +<details open><summary>Config</summary>
494 +
495 +```yaml
496 +jobs:
497 + - name: prod-with-fallback-node
498 + vnode: azure-fallback-node
499 + subscription_ids:
500 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
501 + virtual_nodes:
502 + by_resource_tag: workload
503 + discovery:
504 + mode: filters
505 + profiles:
506 + mode: auto
507 + auth:
508 + mode: managed_identity
509 +
510 +```
511 +</details>
512 +
513 ###### Managed identity with exact profiles
514
515 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -486,7 +589,7 @@ jobs:
589 kql: |
590 resources
591 | where tags.env =~ "prod"
489 - | project id, name, type, resourceGroup, location
592 + | project id, name, type, resourceGroup, location, tags
593 profiles:
594 mode: auto
595 auth:
@@ -676,6 +779,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
779 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
780
781
782 +### Workload virtual nodes are not created
783 +
784 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
785 +
786 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
787 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
788 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
789 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
790 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
791 +
792 +
793 +### More alerts appear after enabling workload virtual nodes
794 +
795 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
796 +
797 +
798 ### Authentication errors in sovereign clouds
799
800 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_log_analytics_workspace.md
+124 -5
@@ -187,7 +187,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
187 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
190 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
190 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
191 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
192 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
193 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -195,7 +195,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
195 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
196 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
197 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
198 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
200
201 <a id="option-collection-query-offset"></a>
202 ##### query_offset
@@ -230,7 +231,7 @@ Controls how the collector finds candidate Azure resources.
231 | Mode | Behavior |
232 |:-----|:---------|
233 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
233 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
234 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
235
236
237 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -249,6 +250,8 @@ The query **must** project these five columns:
250 | `resourceGroup` | Resource group name |
251 | `location` | Azure region |
252
253 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
254 +
255 :::info
256
257 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -262,7 +265,7 @@ Example:
265 ```
266 resources
267 | where tags.env =~ "prod"
265 -| project id, name, type, resourceGroup, location
268 +| project id, name, type, resourceGroup, location, tags
269 ```
270
271
@@ -336,6 +339,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
339 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
340
341
342 +<a id="option-virtual-node-vnode"></a>
343 +##### vnode
344 +
345 +This job-level virtual node is used for metrics written to the default host scope.
346 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
347 +
348 +
349 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
350 +##### virtual_nodes.by_resource_tag
351 +
352 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
353 +
354 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
355 +- Empty or missing tag values use the default job host scope.
356 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
357 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
358 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
359 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
360 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
361 +
362 +
363
364 </details>
365
@@ -407,6 +431,85 @@ jobs:
431 client_secret: "your-client-secret"
432
433 ```
434 +###### Workload virtual nodes from resource tags
435 +
436 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
437 +
438 +<details open><summary>Config</summary>
439 +
440 +```yaml
441 +jobs:
442 + - name: prod-workloads
443 + subscription_ids:
444 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
445 + virtual_nodes:
446 + by_resource_tag: workload
447 + discovery:
448 + mode: filters
449 + profiles:
450 + mode: auto
451 + auth:
452 + mode: service_principal
453 + mode_service_principal:
454 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_secret: "your-client-secret"
457 +
458 +```
459 +</details>
460 +
461 +###### Custom KQL with workload virtual nodes
462 +
463 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
464 +
465 +<details open><summary>Config</summary>
466 +
467 +```yaml
468 +jobs:
469 + - name: prod-query-workloads
470 + subscription_ids:
471 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
472 + virtual_nodes:
473 + by_resource_tag: workload
474 + discovery:
475 + mode: query
476 + mode_query:
477 + kql: |
478 + resources
479 + | where tags.env =~ "prod"
480 + | project id, name, type, resourceGroup, location, tags
481 + profiles:
482 + mode: auto
483 + auth:
484 + mode: default
485 +
486 +```
487 +</details>
488 +
489 +###### Job virtual node plus workload virtual nodes
490 +
491 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
492 +
493 +<details open><summary>Config</summary>
494 +
495 +```yaml
496 +jobs:
497 + - name: prod-with-fallback-node
498 + vnode: azure-fallback-node
499 + subscription_ids:
500 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
501 + virtual_nodes:
502 + by_resource_tag: workload
503 + discovery:
504 + mode: filters
505 + profiles:
506 + mode: auto
507 + auth:
508 + mode: managed_identity
509 +
510 +```
511 +</details>
512 +
513 ###### Managed identity with exact profiles
514
515 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -486,7 +589,7 @@ jobs:
589 kql: |
590 resources
591 | where tags.env =~ "prod"
489 - | project id, name, type, resourceGroup, location
592 + | project id, name, type, resourceGroup, location, tags
593 profiles:
594 mode: auto
595 auth:
@@ -715,6 +818,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
818 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
819
820
821 +### Workload virtual nodes are not created
822 +
823 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
824 +
825 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
826 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
827 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
828 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
829 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
830 +
831 +
832 +### More alerts appear after enabling workload virtual nodes
833 +
834 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
835 +
836 +
837 ### Authentication errors in sovereign clouds
838
839 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_logic_apps_workflow.md
+124 -5
@@ -189,7 +189,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
189 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
192 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
192 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
193 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
194 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
195 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -197,7 +197,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
197 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
198 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
199 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
200 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
202
203 <a id="option-collection-query-offset"></a>
204 ##### query_offset
@@ -232,7 +233,7 @@ Controls how the collector finds candidate Azure resources.
233 | Mode | Behavior |
234 |:-----|:---------|
235 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
235 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
236 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
237
238
239 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -251,6 +252,8 @@ The query **must** project these five columns:
252 | `resourceGroup` | Resource group name |
253 | `location` | Azure region |
254
255 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
256 +
257 :::info
258
259 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -264,7 +267,7 @@ Example:
267 ```
268 resources
269 | where tags.env =~ "prod"
267 -| project id, name, type, resourceGroup, location
270 +| project id, name, type, resourceGroup, location, tags
271 ```
272
273
@@ -338,6 +341,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
341 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
342
343
344 +<a id="option-virtual-node-vnode"></a>
345 +##### vnode
346 +
347 +This job-level virtual node is used for metrics written to the default host scope.
348 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
349 +
350 +
351 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
352 +##### virtual_nodes.by_resource_tag
353 +
354 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
355 +
356 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
357 +- Empty or missing tag values use the default job host scope.
358 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
359 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
360 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
361 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
362 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
363 +
364 +
365
366 </details>
367
@@ -409,6 +433,85 @@ jobs:
433 client_secret: "your-client-secret"
434
435 ```
436 +###### Workload virtual nodes from resource tags
437 +
438 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
439 +
440 +<details open><summary>Config</summary>
441 +
442 +```yaml
443 +jobs:
444 + - name: prod-workloads
445 + subscription_ids:
446 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
447 + virtual_nodes:
448 + by_resource_tag: workload
449 + discovery:
450 + mode: filters
451 + profiles:
452 + mode: auto
453 + auth:
454 + mode: service_principal
455 + mode_service_principal:
456 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_secret: "your-client-secret"
459 +
460 +```
461 +</details>
462 +
463 +###### Custom KQL with workload virtual nodes
464 +
465 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
466 +
467 +<details open><summary>Config</summary>
468 +
469 +```yaml
470 +jobs:
471 + - name: prod-query-workloads
472 + subscription_ids:
473 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
474 + virtual_nodes:
475 + by_resource_tag: workload
476 + discovery:
477 + mode: query
478 + mode_query:
479 + kql: |
480 + resources
481 + | where tags.env =~ "prod"
482 + | project id, name, type, resourceGroup, location, tags
483 + profiles:
484 + mode: auto
485 + auth:
486 + mode: default
487 +
488 +```
489 +</details>
490 +
491 +###### Job virtual node plus workload virtual nodes
492 +
493 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
494 +
495 +<details open><summary>Config</summary>
496 +
497 +```yaml
498 +jobs:
499 + - name: prod-with-fallback-node
500 + vnode: azure-fallback-node
501 + subscription_ids:
502 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
503 + virtual_nodes:
504 + by_resource_tag: workload
505 + discovery:
506 + mode: filters
507 + profiles:
508 + mode: auto
509 + auth:
510 + mode: managed_identity
511 +
512 +```
513 +</details>
514 +
515 ###### Managed identity with exact profiles
516
517 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -488,7 +591,7 @@ jobs:
591 kql: |
592 resources
593 | where tags.env =~ "prod"
491 - | project id, name, type, resourceGroup, location
594 + | project id, name, type, resourceGroup, location, tags
595 profiles:
596 mode: auto
597 auth:
@@ -691,6 +794,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
794 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
795
796
797 +### Workload virtual nodes are not created
798 +
799 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
800 +
801 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
802 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
803 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
804 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
805 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
806 +
807 +
808 +### More alerts appear after enabling workload virtual nodes
809 +
810 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
811 +
812 +
813 ### Authentication errors in sovereign clouds
814
815 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_machine_learning_workspace.md
+124 -5
@@ -192,7 +192,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
192 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
195 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
195 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
196 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
197 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
198 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -200,7 +200,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
200 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
201 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
202 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
203 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
205
206 <a id="option-collection-query-offset"></a>
207 ##### query_offset
@@ -235,7 +236,7 @@ Controls how the collector finds candidate Azure resources.
236 | Mode | Behavior |
237 |:-----|:---------|
238 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
238 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
239 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
240
241
242 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -254,6 +255,8 @@ The query **must** project these five columns:
255 | `resourceGroup` | Resource group name |
256 | `location` | Azure region |
257
258 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
259 +
260 :::info
261
262 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -267,7 +270,7 @@ Example:
270 ```
271 resources
272 | where tags.env =~ "prod"
270 -| project id, name, type, resourceGroup, location
273 +| project id, name, type, resourceGroup, location, tags
274 ```
275
276
@@ -341,6 +344,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
344 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
345
346
347 +<a id="option-virtual-node-vnode"></a>
348 +##### vnode
349 +
350 +This job-level virtual node is used for metrics written to the default host scope.
351 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
352 +
353 +
354 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
355 +##### virtual_nodes.by_resource_tag
356 +
357 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
358 +
359 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
360 +- Empty or missing tag values use the default job host scope.
361 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
362 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
363 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
364 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
365 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
366 +
367 +
368
369 </details>
370
@@ -412,6 +436,85 @@ jobs:
436 client_secret: "your-client-secret"
437
438 ```
439 +###### Workload virtual nodes from resource tags
440 +
441 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
442 +
443 +<details open><summary>Config</summary>
444 +
445 +```yaml
446 +jobs:
447 + - name: prod-workloads
448 + subscription_ids:
449 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
450 + virtual_nodes:
451 + by_resource_tag: workload
452 + discovery:
453 + mode: filters
454 + profiles:
455 + mode: auto
456 + auth:
457 + mode: service_principal
458 + mode_service_principal:
459 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_secret: "your-client-secret"
462 +
463 +```
464 +</details>
465 +
466 +###### Custom KQL with workload virtual nodes
467 +
468 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
469 +
470 +<details open><summary>Config</summary>
471 +
472 +```yaml
473 +jobs:
474 + - name: prod-query-workloads
475 + subscription_ids:
476 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
477 + virtual_nodes:
478 + by_resource_tag: workload
479 + discovery:
480 + mode: query
481 + mode_query:
482 + kql: |
483 + resources
484 + | where tags.env =~ "prod"
485 + | project id, name, type, resourceGroup, location, tags
486 + profiles:
487 + mode: auto
488 + auth:
489 + mode: default
490 +
491 +```
492 +</details>
493 +
494 +###### Job virtual node plus workload virtual nodes
495 +
496 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
497 +
498 +<details open><summary>Config</summary>
499 +
500 +```yaml
501 +jobs:
502 + - name: prod-with-fallback-node
503 + vnode: azure-fallback-node
504 + subscription_ids:
505 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
506 + virtual_nodes:
507 + by_resource_tag: workload
508 + discovery:
509 + mode: filters
510 + profiles:
511 + mode: auto
512 + auth:
513 + mode: managed_identity
514 +
515 +```
516 +</details>
517 +
518 ###### Managed identity with exact profiles
519
520 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -491,7 +594,7 @@ jobs:
594 kql: |
595 resources
596 | where tags.env =~ "prod"
494 - | project id, name, type, resourceGroup, location
597 + | project id, name, type, resourceGroup, location, tags
598 profiles:
599 mode: auto
600 auth:
@@ -714,6 +817,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
817 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
818
819
820 +### Workload virtual nodes are not created
821 +
822 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
823 +
824 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
825 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
826 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
827 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
828 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
829 +
830 +
831 +### More alerts appear after enabling workload virtual nodes
832 +
833 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
834 +
835 +
836 ### Authentication errors in sovereign clouds
837
838 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_monitor.md
+124 -5
@@ -183,7 +183,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
183 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
184 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
185 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
186 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
186 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
187 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
188 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
189 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -191,7 +191,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
192 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
193 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
194 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
194 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
195 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
196
197 <a id="option-collection-query-offset"></a>
198 ##### query_offset
@@ -226,7 +227,7 @@ Controls how the collector finds candidate Azure resources.
227 | Mode | Behavior |
228 |:-----|:---------|
229 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
229 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
230 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
231
232
233 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -245,6 +246,8 @@ The query **must** project these five columns:
246 | `resourceGroup` | Resource group name |
247 | `location` | Azure region |
248
249 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
250 +
251 :::info
252
253 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -258,7 +261,7 @@ Example:
261 ```
262 resources
263 | where tags.env =~ "prod"
261 -| project id, name, type, resourceGroup, location
264 +| project id, name, type, resourceGroup, location, tags
265 ```
266
267
@@ -332,6 +335,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
335 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
336
337
338 +<a id="option-virtual-node-vnode"></a>
339 +##### vnode
340 +
341 +This job-level virtual node is used for metrics written to the default host scope.
342 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
343 +
344 +
345 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
346 +##### virtual_nodes.by_resource_tag
347 +
348 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
349 +
350 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
351 +- Empty or missing tag values use the default job host scope.
352 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
353 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
354 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
355 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
356 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
357 +
358 +
359
360 </details>
361
@@ -403,6 +427,85 @@ jobs:
427 client_secret: "your-client-secret"
428
429 ```
430 +###### Workload virtual nodes from resource tags
431 +
432 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
433 +
434 +<details open><summary>Config</summary>
435 +
436 +```yaml
437 +jobs:
438 + - name: prod-workloads
439 + subscription_ids:
440 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
441 + virtual_nodes:
442 + by_resource_tag: workload
443 + discovery:
444 + mode: filters
445 + profiles:
446 + mode: auto
447 + auth:
448 + mode: service_principal
449 + mode_service_principal:
450 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
451 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
452 + client_secret: "your-client-secret"
453 +
454 +```
455 +</details>
456 +
457 +###### Custom KQL with workload virtual nodes
458 +
459 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
460 +
461 +<details open><summary>Config</summary>
462 +
463 +```yaml
464 +jobs:
465 + - name: prod-query-workloads
466 + subscription_ids:
467 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
468 + virtual_nodes:
469 + by_resource_tag: workload
470 + discovery:
471 + mode: query
472 + mode_query:
473 + kql: |
474 + resources
475 + | where tags.env =~ "prod"
476 + | project id, name, type, resourceGroup, location, tags
477 + profiles:
478 + mode: auto
479 + auth:
480 + mode: default
481 +
482 +```
483 +</details>
484 +
485 +###### Job virtual node plus workload virtual nodes
486 +
487 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
488 +
489 +<details open><summary>Config</summary>
490 +
491 +```yaml
492 +jobs:
493 + - name: prod-with-fallback-node
494 + vnode: azure-fallback-node
495 + subscription_ids:
496 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
497 + virtual_nodes:
498 + by_resource_tag: workload
499 + discovery:
500 + mode: filters
501 + profiles:
502 + mode: auto
503 + auth:
504 + mode: managed_identity
505 +
506 +```
507 +</details>
508 +
509 ###### Managed identity with exact profiles
510
511 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -482,7 +585,7 @@ jobs:
585 kql: |
586 resources
587 | where tags.env =~ "prod"
485 - | project id, name, type, resourceGroup, location
588 + | project id, name, type, resourceGroup, location, tags
589 profiles:
590 mode: auto
591 auth:
@@ -635,6 +738,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
738 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
739
740
741 +### Workload virtual nodes are not created
742 +
743 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
744 +
745 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
746 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
747 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
748 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
749 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
750 +
751 +
752 +### More alerts appear after enabling workload virtual nodes
753 +
754 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
755 +
756 +
757 ### Authentication errors in sovereign clouds
758
759 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_mysql_flexible_server.md
+124 -5
@@ -192,7 +192,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
192 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
195 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
195 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
196 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
197 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
198 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -200,7 +200,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
200 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
201 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
202 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
203 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
205
206 <a id="option-collection-query-offset"></a>
207 ##### query_offset
@@ -235,7 +236,7 @@ Controls how the collector finds candidate Azure resources.
236 | Mode | Behavior |
237 |:-----|:---------|
238 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
238 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
239 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
240
241
242 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -254,6 +255,8 @@ The query **must** project these five columns:
255 | `resourceGroup` | Resource group name |
256 | `location` | Azure region |
257
258 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
259 +
260 :::info
261
262 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -267,7 +270,7 @@ Example:
270 ```
271 resources
272 | where tags.env =~ "prod"
270 -| project id, name, type, resourceGroup, location
273 +| project id, name, type, resourceGroup, location, tags
274 ```
275
276
@@ -341,6 +344,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
344 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
345
346
347 +<a id="option-virtual-node-vnode"></a>
348 +##### vnode
349 +
350 +This job-level virtual node is used for metrics written to the default host scope.
351 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
352 +
353 +
354 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
355 +##### virtual_nodes.by_resource_tag
356 +
357 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
358 +
359 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
360 +- Empty or missing tag values use the default job host scope.
361 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
362 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
363 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
364 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
365 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
366 +
367 +
368
369 </details>
370
@@ -412,6 +436,85 @@ jobs:
436 client_secret: "your-client-secret"
437
438 ```
439 +###### Workload virtual nodes from resource tags
440 +
441 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
442 +
443 +<details open><summary>Config</summary>
444 +
445 +```yaml
446 +jobs:
447 + - name: prod-workloads
448 + subscription_ids:
449 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
450 + virtual_nodes:
451 + by_resource_tag: workload
452 + discovery:
453 + mode: filters
454 + profiles:
455 + mode: auto
456 + auth:
457 + mode: service_principal
458 + mode_service_principal:
459 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_secret: "your-client-secret"
462 +
463 +```
464 +</details>
465 +
466 +###### Custom KQL with workload virtual nodes
467 +
468 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
469 +
470 +<details open><summary>Config</summary>
471 +
472 +```yaml
473 +jobs:
474 + - name: prod-query-workloads
475 + subscription_ids:
476 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
477 + virtual_nodes:
478 + by_resource_tag: workload
479 + discovery:
480 + mode: query
481 + mode_query:
482 + kql: |
483 + resources
484 + | where tags.env =~ "prod"
485 + | project id, name, type, resourceGroup, location, tags
486 + profiles:
487 + mode: auto
488 + auth:
489 + mode: default
490 +
491 +```
492 +</details>
493 +
494 +###### Job virtual node plus workload virtual nodes
495 +
496 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
497 +
498 +<details open><summary>Config</summary>
499 +
500 +```yaml
501 +jobs:
502 + - name: prod-with-fallback-node
503 + vnode: azure-fallback-node
504 + subscription_ids:
505 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
506 + virtual_nodes:
507 + by_resource_tag: workload
508 + discovery:
509 + mode: filters
510 + profiles:
511 + mode: auto
512 + auth:
513 + mode: managed_identity
514 +
515 +```
516 +</details>
517 +
518 ###### Managed identity with exact profiles
519
520 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -491,7 +594,7 @@ jobs:
594 kql: |
595 resources
596 | where tags.env =~ "prod"
494 - | project id, name, type, resourceGroup, location
597 + | project id, name, type, resourceGroup, location, tags
598 profiles:
599 mode: auto
600 auth:
@@ -721,6 +824,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
824 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
825
826
827 +### Workload virtual nodes are not created
828 +
829 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
830 +
831 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
832 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
833 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
834 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
835 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
836 +
837 +
838 +### More alerts appear after enabling workload virtual nodes
839 +
840 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
841 +
842 +
843 ### Authentication errors in sovereign clouds
844
845 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_nat_gateway.md
+124 -5
@@ -187,7 +187,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
187 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
190 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
190 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
191 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
192 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
193 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -195,7 +195,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
195 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
196 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
197 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
198 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
200
201 <a id="option-collection-query-offset"></a>
202 ##### query_offset
@@ -230,7 +231,7 @@ Controls how the collector finds candidate Azure resources.
231 | Mode | Behavior |
232 |:-----|:---------|
233 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
233 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
234 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
235
236
237 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -249,6 +250,8 @@ The query **must** project these five columns:
250 | `resourceGroup` | Resource group name |
251 | `location` | Azure region |
252
253 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
254 +
255 :::info
256
257 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -262,7 +265,7 @@ Example:
265 ```
266 resources
267 | where tags.env =~ "prod"
265 -| project id, name, type, resourceGroup, location
268 +| project id, name, type, resourceGroup, location, tags
269 ```
270
271
@@ -336,6 +339,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
339 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
340
341
342 +<a id="option-virtual-node-vnode"></a>
343 +##### vnode
344 +
345 +This job-level virtual node is used for metrics written to the default host scope.
346 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
347 +
348 +
349 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
350 +##### virtual_nodes.by_resource_tag
351 +
352 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
353 +
354 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
355 +- Empty or missing tag values use the default job host scope.
356 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
357 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
358 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
359 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
360 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
361 +
362 +
363
364 </details>
365
@@ -407,6 +431,85 @@ jobs:
431 client_secret: "your-client-secret"
432
433 ```
434 +###### Workload virtual nodes from resource tags
435 +
436 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
437 +
438 +<details open><summary>Config</summary>
439 +
440 +```yaml
441 +jobs:
442 + - name: prod-workloads
443 + subscription_ids:
444 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
445 + virtual_nodes:
446 + by_resource_tag: workload
447 + discovery:
448 + mode: filters
449 + profiles:
450 + mode: auto
451 + auth:
452 + mode: service_principal
453 + mode_service_principal:
454 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_secret: "your-client-secret"
457 +
458 +```
459 +</details>
460 +
461 +###### Custom KQL with workload virtual nodes
462 +
463 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
464 +
465 +<details open><summary>Config</summary>
466 +
467 +```yaml
468 +jobs:
469 + - name: prod-query-workloads
470 + subscription_ids:
471 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
472 + virtual_nodes:
473 + by_resource_tag: workload
474 + discovery:
475 + mode: query
476 + mode_query:
477 + kql: |
478 + resources
479 + | where tags.env =~ "prod"
480 + | project id, name, type, resourceGroup, location, tags
481 + profiles:
482 + mode: auto
483 + auth:
484 + mode: default
485 +
486 +```
487 +</details>
488 +
489 +###### Job virtual node plus workload virtual nodes
490 +
491 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
492 +
493 +<details open><summary>Config</summary>
494 +
495 +```yaml
496 +jobs:
497 + - name: prod-with-fallback-node
498 + vnode: azure-fallback-node
499 + subscription_ids:
500 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
501 + virtual_nodes:
502 + by_resource_tag: workload
503 + discovery:
504 + mode: filters
505 + profiles:
506 + mode: auto
507 + auth:
508 + mode: managed_identity
509 +
510 +```
511 +</details>
512 +
513 ###### Managed identity with exact profiles
514
515 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -486,7 +589,7 @@ jobs:
589 kql: |
590 resources
591 | where tags.env =~ "prod"
489 - | project id, name, type, resourceGroup, location
592 + | project id, name, type, resourceGroup, location, tags
593 profiles:
594 mode: auto
595 auth:
@@ -674,6 +777,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
777 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
778
779
780 +### Workload virtual nodes are not created
781 +
782 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
783 +
784 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
785 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
786 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
787 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
788 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
789 +
790 +
791 +### More alerts appear after enabling workload virtual nodes
792 +
793 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
794 +
795 +
796 ### Authentication errors in sovereign clouds
797
798 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_postgresql_flexible_server.md
+124 -5
@@ -192,7 +192,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
192 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
195 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
195 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
196 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
197 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
198 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -200,7 +200,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
200 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
201 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
202 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
203 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
205
206 <a id="option-collection-query-offset"></a>
207 ##### query_offset
@@ -235,7 +236,7 @@ Controls how the collector finds candidate Azure resources.
236 | Mode | Behavior |
237 |:-----|:---------|
238 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
238 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
239 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
240
241
242 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -254,6 +255,8 @@ The query **must** project these five columns:
255 | `resourceGroup` | Resource group name |
256 | `location` | Azure region |
257
258 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
259 +
260 :::info
261
262 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -267,7 +270,7 @@ Example:
270 ```
271 resources
272 | where tags.env =~ "prod"
270 -| project id, name, type, resourceGroup, location
273 +| project id, name, type, resourceGroup, location, tags
274 ```
275
276
@@ -341,6 +344,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
344 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
345
346
347 +<a id="option-virtual-node-vnode"></a>
348 +##### vnode
349 +
350 +This job-level virtual node is used for metrics written to the default host scope.
351 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
352 +
353 +
354 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
355 +##### virtual_nodes.by_resource_tag
356 +
357 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
358 +
359 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
360 +- Empty or missing tag values use the default job host scope.
361 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
362 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
363 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
364 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
365 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
366 +
367 +
368
369 </details>
370
@@ -412,6 +436,85 @@ jobs:
436 client_secret: "your-client-secret"
437
438 ```
439 +###### Workload virtual nodes from resource tags
440 +
441 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
442 +
443 +<details open><summary>Config</summary>
444 +
445 +```yaml
446 +jobs:
447 + - name: prod-workloads
448 + subscription_ids:
449 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
450 + virtual_nodes:
451 + by_resource_tag: workload
452 + discovery:
453 + mode: filters
454 + profiles:
455 + mode: auto
456 + auth:
457 + mode: service_principal
458 + mode_service_principal:
459 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_secret: "your-client-secret"
462 +
463 +```
464 +</details>
465 +
466 +###### Custom KQL with workload virtual nodes
467 +
468 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
469 +
470 +<details open><summary>Config</summary>
471 +
472 +```yaml
473 +jobs:
474 + - name: prod-query-workloads
475 + subscription_ids:
476 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
477 + virtual_nodes:
478 + by_resource_tag: workload
479 + discovery:
480 + mode: query
481 + mode_query:
482 + kql: |
483 + resources
484 + | where tags.env =~ "prod"
485 + | project id, name, type, resourceGroup, location, tags
486 + profiles:
487 + mode: auto
488 + auth:
489 + mode: default
490 +
491 +```
492 +</details>
493 +
494 +###### Job virtual node plus workload virtual nodes
495 +
496 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
497 +
498 +<details open><summary>Config</summary>
499 +
500 +```yaml
501 +jobs:
502 + - name: prod-with-fallback-node
503 + vnode: azure-fallback-node
504 + subscription_ids:
505 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
506 + virtual_nodes:
507 + by_resource_tag: workload
508 + discovery:
509 + mode: filters
510 + profiles:
511 + mode: auto
512 + auth:
513 + mode: managed_identity
514 +
515 +```
516 +</details>
517 +
518 ###### Managed identity with exact profiles
519
520 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -491,7 +594,7 @@ jobs:
594 kql: |
595 resources
596 | where tags.env =~ "prod"
494 - | project id, name, type, resourceGroup, location
597 + | project id, name, type, resourceGroup, location, tags
598 profiles:
599 mode: auto
600 auth:
@@ -731,6 +834,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
834 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
835
836
837 +### Workload virtual nodes are not created
838 +
839 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
840 +
841 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
842 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
843 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
844 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
845 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
846 +
847 +
848 +### More alerts appear after enabling workload virtual nodes
849 +
850 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
851 +
852 +
853 ### Authentication errors in sovereign clouds
854
855 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_service_bus_namespace.md
+124 -5
@@ -191,7 +191,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
194 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
194 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
195 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
196 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
197 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -199,7 +199,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
199 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
200 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
201 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
202 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
204
205 <a id="option-collection-query-offset"></a>
206 ##### query_offset
@@ -234,7 +235,7 @@ Controls how the collector finds candidate Azure resources.
235 | Mode | Behavior |
236 |:-----|:---------|
237 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
237 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
238 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
239
240
241 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -253,6 +254,8 @@ The query **must** project these five columns:
254 | `resourceGroup` | Resource group name |
255 | `location` | Azure region |
256
257 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
258 +
259 :::info
260
261 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -266,7 +269,7 @@ Example:
269 ```
270 resources
271 | where tags.env =~ "prod"
269 -| project id, name, type, resourceGroup, location
272 +| project id, name, type, resourceGroup, location, tags
273 ```
274
275
@@ -340,6 +343,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
343 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
344
345
346 +<a id="option-virtual-node-vnode"></a>
347 +##### vnode
348 +
349 +This job-level virtual node is used for metrics written to the default host scope.
350 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
351 +
352 +
353 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
354 +##### virtual_nodes.by_resource_tag
355 +
356 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
357 +
358 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
359 +- Empty or missing tag values use the default job host scope.
360 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
361 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
362 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
363 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
364 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
365 +
366 +
367
368 </details>
369
@@ -411,6 +435,85 @@ jobs:
435 client_secret: "your-client-secret"
436
437 ```
438 +###### Workload virtual nodes from resource tags
439 +
440 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
441 +
442 +<details open><summary>Config</summary>
443 +
444 +```yaml
445 +jobs:
446 + - name: prod-workloads
447 + subscription_ids:
448 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
449 + virtual_nodes:
450 + by_resource_tag: workload
451 + discovery:
452 + mode: filters
453 + profiles:
454 + mode: auto
455 + auth:
456 + mode: service_principal
457 + mode_service_principal:
458 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_secret: "your-client-secret"
461 +
462 +```
463 +</details>
464 +
465 +###### Custom KQL with workload virtual nodes
466 +
467 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
468 +
469 +<details open><summary>Config</summary>
470 +
471 +```yaml
472 +jobs:
473 + - name: prod-query-workloads
474 + subscription_ids:
475 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
476 + virtual_nodes:
477 + by_resource_tag: workload
478 + discovery:
479 + mode: query
480 + mode_query:
481 + kql: |
482 + resources
483 + | where tags.env =~ "prod"
484 + | project id, name, type, resourceGroup, location, tags
485 + profiles:
486 + mode: auto
487 + auth:
488 + mode: default
489 +
490 +```
491 +</details>
492 +
493 +###### Job virtual node plus workload virtual nodes
494 +
495 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
496 +
497 +<details open><summary>Config</summary>
498 +
499 +```yaml
500 +jobs:
501 + - name: prod-with-fallback-node
502 + vnode: azure-fallback-node
503 + subscription_ids:
504 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
505 + virtual_nodes:
506 + by_resource_tag: workload
507 + discovery:
508 + mode: filters
509 + profiles:
510 + mode: auto
511 + auth:
512 + mode: managed_identity
513 +
514 +```
515 +</details>
516 +
517 ###### Managed identity with exact profiles
518
519 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -490,7 +593,7 @@ jobs:
593 kql: |
594 resources
595 | where tags.env =~ "prod"
493 - | project id, name, type, resourceGroup, location
596 + | project id, name, type, resourceGroup, location, tags
597 profiles:
598 mode: auto
599 auth:
@@ -696,6 +799,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
799 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
800
801
802 +### Workload virtual nodes are not created
803 +
804 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
805 +
806 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
807 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
808 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
809 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
810 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
811 +
812 +
813 +### More alerts appear after enabling workload virtual nodes
814 +
815 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
816 +
817 +
818 ### Authentication errors in sovereign clouds
819
820 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_sql_database.md
+124 -5
@@ -190,7 +190,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
190 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
193 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
193 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
194 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
195 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
196 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -198,7 +198,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
198 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
199 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
200 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
201 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
203
204 <a id="option-collection-query-offset"></a>
205 ##### query_offset
@@ -233,7 +234,7 @@ Controls how the collector finds candidate Azure resources.
234 | Mode | Behavior |
235 |:-----|:---------|
236 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
236 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
237 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
238
239
240 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -252,6 +253,8 @@ The query **must** project these five columns:
253 | `resourceGroup` | Resource group name |
254 | `location` | Azure region |
255
256 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
257 +
258 :::info
259
260 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -265,7 +268,7 @@ Example:
268 ```
269 resources
270 | where tags.env =~ "prod"
268 -| project id, name, type, resourceGroup, location
271 +| project id, name, type, resourceGroup, location, tags
272 ```
273
274
@@ -339,6 +342,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
342 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
343
344
345 +<a id="option-virtual-node-vnode"></a>
346 +##### vnode
347 +
348 +This job-level virtual node is used for metrics written to the default host scope.
349 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
350 +
351 +
352 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
353 +##### virtual_nodes.by_resource_tag
354 +
355 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
356 +
357 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
358 +- Empty or missing tag values use the default job host scope.
359 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
360 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
361 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
362 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
363 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
364 +
365 +
366
367 </details>
368
@@ -410,6 +434,85 @@ jobs:
434 client_secret: "your-client-secret"
435
436 ```
437 +###### Workload virtual nodes from resource tags
438 +
439 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
440 +
441 +<details open><summary>Config</summary>
442 +
443 +```yaml
444 +jobs:
445 + - name: prod-workloads
446 + subscription_ids:
447 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
448 + virtual_nodes:
449 + by_resource_tag: workload
450 + discovery:
451 + mode: filters
452 + profiles:
453 + mode: auto
454 + auth:
455 + mode: service_principal
456 + mode_service_principal:
457 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_secret: "your-client-secret"
460 +
461 +```
462 +</details>
463 +
464 +###### Custom KQL with workload virtual nodes
465 +
466 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
467 +
468 +<details open><summary>Config</summary>
469 +
470 +```yaml
471 +jobs:
472 + - name: prod-query-workloads
473 + subscription_ids:
474 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
475 + virtual_nodes:
476 + by_resource_tag: workload
477 + discovery:
478 + mode: query
479 + mode_query:
480 + kql: |
481 + resources
482 + | where tags.env =~ "prod"
483 + | project id, name, type, resourceGroup, location, tags
484 + profiles:
485 + mode: auto
486 + auth:
487 + mode: default
488 +
489 +```
490 +</details>
491 +
492 +###### Job virtual node plus workload virtual nodes
493 +
494 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
495 +
496 +<details open><summary>Config</summary>
497 +
498 +```yaml
499 +jobs:
500 + - name: prod-with-fallback-node
501 + vnode: azure-fallback-node
502 + subscription_ids:
503 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
504 + virtual_nodes:
505 + by_resource_tag: workload
506 + discovery:
507 + mode: filters
508 + profiles:
509 + mode: auto
510 + auth:
511 + mode: managed_identity
512 +
513 +```
514 +</details>
515 +
516 ###### Managed identity with exact profiles
517
518 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -489,7 +592,7 @@ jobs:
592 kql: |
593 resources
594 | where tags.env =~ "prod"
492 - | project id, name, type, resourceGroup, location
595 + | project id, name, type, resourceGroup, location, tags
596 profiles:
597 mode: auto
598 auth:
@@ -708,6 +811,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
811 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
812
813
814 +### Workload virtual nodes are not created
815 +
816 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
817 +
818 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
819 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
820 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
821 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
822 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
823 +
824 +
825 +### More alerts appear after enabling workload virtual nodes
826 +
827 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
828 +
829 +
830 ### Authentication errors in sovereign clouds
831
832 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_sql_elastic_pool.md
+124 -5
@@ -191,7 +191,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
194 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
194 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
195 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
196 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
197 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -199,7 +199,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
199 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
200 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
201 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
202 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
204
205 <a id="option-collection-query-offset"></a>
206 ##### query_offset
@@ -234,7 +235,7 @@ Controls how the collector finds candidate Azure resources.
235 | Mode | Behavior |
236 |:-----|:---------|
237 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
237 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
238 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
239
240
241 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -253,6 +254,8 @@ The query **must** project these five columns:
254 | `resourceGroup` | Resource group name |
255 | `location` | Azure region |
256
257 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
258 +
259 :::info
260
261 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -266,7 +269,7 @@ Example:
269 ```
270 resources
271 | where tags.env =~ "prod"
269 -| project id, name, type, resourceGroup, location
272 +| project id, name, type, resourceGroup, location, tags
273 ```
274
275
@@ -340,6 +343,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
343 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
344
345
346 +<a id="option-virtual-node-vnode"></a>
347 +##### vnode
348 +
349 +This job-level virtual node is used for metrics written to the default host scope.
350 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
351 +
352 +
353 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
354 +##### virtual_nodes.by_resource_tag
355 +
356 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
357 +
358 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
359 +- Empty or missing tag values use the default job host scope.
360 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
361 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
362 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
363 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
364 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
365 +
366 +
367
368 </details>
369
@@ -411,6 +435,85 @@ jobs:
435 client_secret: "your-client-secret"
436
437 ```
438 +###### Workload virtual nodes from resource tags
439 +
440 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
441 +
442 +<details open><summary>Config</summary>
443 +
444 +```yaml
445 +jobs:
446 + - name: prod-workloads
447 + subscription_ids:
448 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
449 + virtual_nodes:
450 + by_resource_tag: workload
451 + discovery:
452 + mode: filters
453 + profiles:
454 + mode: auto
455 + auth:
456 + mode: service_principal
457 + mode_service_principal:
458 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_secret: "your-client-secret"
461 +
462 +```
463 +</details>
464 +
465 +###### Custom KQL with workload virtual nodes
466 +
467 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
468 +
469 +<details open><summary>Config</summary>
470 +
471 +```yaml
472 +jobs:
473 + - name: prod-query-workloads
474 + subscription_ids:
475 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
476 + virtual_nodes:
477 + by_resource_tag: workload
478 + discovery:
479 + mode: query
480 + mode_query:
481 + kql: |
482 + resources
483 + | where tags.env =~ "prod"
484 + | project id, name, type, resourceGroup, location, tags
485 + profiles:
486 + mode: auto
487 + auth:
488 + mode: default
489 +
490 +```
491 +</details>
492 +
493 +###### Job virtual node plus workload virtual nodes
494 +
495 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
496 +
497 +<details open><summary>Config</summary>
498 +
499 +```yaml
500 +jobs:
501 + - name: prod-with-fallback-node
502 + vnode: azure-fallback-node
503 + subscription_ids:
504 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
505 + virtual_nodes:
506 + by_resource_tag: workload
507 + discovery:
508 + mode: filters
509 + profiles:
510 + mode: auto
511 + auth:
512 + mode: managed_identity
513 +
514 +```
515 +</details>
516 +
517 ###### Managed identity with exact profiles
518
519 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -490,7 +593,7 @@ jobs:
593 kql: |
594 resources
595 | where tags.env =~ "prod"
493 - | project id, name, type, resourceGroup, location
596 + | project id, name, type, resourceGroup, location, tags
597 profiles:
598 mode: auto
599 auth:
@@ -698,6 +801,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
801 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
802
803
804 +### Workload virtual nodes are not created
805 +
806 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
807 +
808 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
809 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
810 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
811 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
812 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
813 +
814 +
815 +### More alerts appear after enabling workload virtual nodes
816 +
817 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
818 +
819 +
820 ### Authentication errors in sovereign clouds
821
822 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_sql_managed_instance.md
+124 -5
@@ -186,7 +186,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
186 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
187 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
188 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
189 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
189 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
190 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
191 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
192 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -194,7 +194,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
194 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
195 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
196 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
197 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
197 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
198 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
199
200 <a id="option-collection-query-offset"></a>
201 ##### query_offset
@@ -229,7 +230,7 @@ Controls how the collector finds candidate Azure resources.
230 | Mode | Behavior |
231 |:-----|:---------|
232 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
232 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
233 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
234
235
236 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -248,6 +249,8 @@ The query **must** project these five columns:
249 | `resourceGroup` | Resource group name |
250 | `location` | Azure region |
251
252 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
253 +
254 :::info
255
256 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -261,7 +264,7 @@ Example:
264 ```
265 resources
266 | where tags.env =~ "prod"
264 -| project id, name, type, resourceGroup, location
267 +| project id, name, type, resourceGroup, location, tags
268 ```
269
270
@@ -335,6 +338,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
338 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
339
340
341 +<a id="option-virtual-node-vnode"></a>
342 +##### vnode
343 +
344 +This job-level virtual node is used for metrics written to the default host scope.
345 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
346 +
347 +
348 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
349 +##### virtual_nodes.by_resource_tag
350 +
351 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
352 +
353 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
354 +- Empty or missing tag values use the default job host scope.
355 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
356 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
357 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
358 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
359 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
360 +
361 +
362
363 </details>
364
@@ -406,6 +430,85 @@ jobs:
430 client_secret: "your-client-secret"
431
432 ```
433 +###### Workload virtual nodes from resource tags
434 +
435 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
436 +
437 +<details open><summary>Config</summary>
438 +
439 +```yaml
440 +jobs:
441 + - name: prod-workloads
442 + subscription_ids:
443 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
444 + virtual_nodes:
445 + by_resource_tag: workload
446 + discovery:
447 + mode: filters
448 + profiles:
449 + mode: auto
450 + auth:
451 + mode: service_principal
452 + mode_service_principal:
453 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
454 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
455 + client_secret: "your-client-secret"
456 +
457 +```
458 +</details>
459 +
460 +###### Custom KQL with workload virtual nodes
461 +
462 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
463 +
464 +<details open><summary>Config</summary>
465 +
466 +```yaml
467 +jobs:
468 + - name: prod-query-workloads
469 + subscription_ids:
470 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
471 + virtual_nodes:
472 + by_resource_tag: workload
473 + discovery:
474 + mode: query
475 + mode_query:
476 + kql: |
477 + resources
478 + | where tags.env =~ "prod"
479 + | project id, name, type, resourceGroup, location, tags
480 + profiles:
481 + mode: auto
482 + auth:
483 + mode: default
484 +
485 +```
486 +</details>
487 +
488 +###### Job virtual node plus workload virtual nodes
489 +
490 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
491 +
492 +<details open><summary>Config</summary>
493 +
494 +```yaml
495 +jobs:
496 + - name: prod-with-fallback-node
497 + vnode: azure-fallback-node
498 + subscription_ids:
499 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
500 + virtual_nodes:
501 + by_resource_tag: workload
502 + discovery:
503 + mode: filters
504 + profiles:
505 + mode: auto
506 + auth:
507 + mode: managed_identity
508 +
509 +```
510 +</details>
511 +
512 ###### Managed identity with exact profiles
513
514 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -485,7 +588,7 @@ jobs:
588 kql: |
589 resources
590 | where tags.env =~ "prod"
488 - | project id, name, type, resourceGroup, location
591 + | project id, name, type, resourceGroup, location, tags
592 profiles:
593 mode: auto
594 auth:
@@ -671,6 +774,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
774 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
775
776
777 +### Workload virtual nodes are not created
778 +
779 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
780 +
781 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
782 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
783 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
784 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
785 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
786 +
787 +
788 +### More alerts appear after enabling workload virtual nodes
789 +
790 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
791 +
792 +
793 ### Authentication errors in sovereign clouds
794
795 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_storage_account.md
+124 -5
@@ -188,7 +188,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
188 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
191 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
191 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
192 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
193 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
194 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -196,7 +196,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
196 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
197 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
198 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
199 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
201
202 <a id="option-collection-query-offset"></a>
203 ##### query_offset
@@ -231,7 +232,7 @@ Controls how the collector finds candidate Azure resources.
232 | Mode | Behavior |
233 |:-----|:---------|
234 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
234 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
235 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
236
237
238 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -250,6 +251,8 @@ The query **must** project these five columns:
251 | `resourceGroup` | Resource group name |
252 | `location` | Azure region |
253
254 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
255 +
256 :::info
257
258 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -263,7 +266,7 @@ Example:
266 ```
267 resources
268 | where tags.env =~ "prod"
266 -| project id, name, type, resourceGroup, location
269 +| project id, name, type, resourceGroup, location, tags
270 ```
271
272
@@ -337,6 +340,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
340 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
341
342
343 +<a id="option-virtual-node-vnode"></a>
344 +##### vnode
345 +
346 +This job-level virtual node is used for metrics written to the default host scope.
347 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
348 +
349 +
350 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
351 +##### virtual_nodes.by_resource_tag
352 +
353 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
354 +
355 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
356 +- Empty or missing tag values use the default job host scope.
357 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
358 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
359 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
360 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
361 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
362 +
363 +
364
365 </details>
366
@@ -408,6 +432,85 @@ jobs:
432 client_secret: "your-client-secret"
433
434 ```
435 +###### Workload virtual nodes from resource tags
436 +
437 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
438 +
439 +<details open><summary>Config</summary>
440 +
441 +```yaml
442 +jobs:
443 + - name: prod-workloads
444 + subscription_ids:
445 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
446 + virtual_nodes:
447 + by_resource_tag: workload
448 + discovery:
449 + mode: filters
450 + profiles:
451 + mode: auto
452 + auth:
453 + mode: service_principal
454 + mode_service_principal:
455 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_secret: "your-client-secret"
458 +
459 +```
460 +</details>
461 +
462 +###### Custom KQL with workload virtual nodes
463 +
464 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
465 +
466 +<details open><summary>Config</summary>
467 +
468 +```yaml
469 +jobs:
470 + - name: prod-query-workloads
471 + subscription_ids:
472 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
473 + virtual_nodes:
474 + by_resource_tag: workload
475 + discovery:
476 + mode: query
477 + mode_query:
478 + kql: |
479 + resources
480 + | where tags.env =~ "prod"
481 + | project id, name, type, resourceGroup, location, tags
482 + profiles:
483 + mode: auto
484 + auth:
485 + mode: default
486 +
487 +```
488 +</details>
489 +
490 +###### Job virtual node plus workload virtual nodes
491 +
492 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
493 +
494 +<details open><summary>Config</summary>
495 +
496 +```yaml
497 +jobs:
498 + - name: prod-with-fallback-node
499 + vnode: azure-fallback-node
500 + subscription_ids:
501 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
502 + virtual_nodes:
503 + by_resource_tag: workload
504 + discovery:
505 + mode: filters
506 + profiles:
507 + mode: auto
508 + auth:
509 + mode: managed_identity
510 +
511 +```
512 +</details>
513 +
514 ###### Managed identity with exact profiles
515
516 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -487,7 +590,7 @@ jobs:
590 kql: |
591 resources
592 | where tags.env =~ "prod"
490 - | project id, name, type, resourceGroup, location
593 + | project id, name, type, resourceGroup, location, tags
594 profiles:
595 mode: auto
596 auth:
@@ -676,6 +779,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
779 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
780
781
782 +### Workload virtual nodes are not created
783 +
784 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
785 +
786 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
787 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
788 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
789 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
790 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
791 +
792 +
793 +### More alerts appear after enabling workload virtual nodes
794 +
795 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
796 +
797 +
798 ### Authentication errors in sovereign clouds
799
800 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_stream_analytics_job.md
+124 -5
@@ -189,7 +189,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
189 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
191 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
192 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
192 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
193 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
194 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
195 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -197,7 +197,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
197 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
198 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
199 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
200 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
201 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
202
203 <a id="option-collection-query-offset"></a>
204 ##### query_offset
@@ -232,7 +233,7 @@ Controls how the collector finds candidate Azure resources.
233 | Mode | Behavior |
234 |:-----|:---------|
235 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
235 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
236 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
237
238
239 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -251,6 +252,8 @@ The query **must** project these five columns:
252 | `resourceGroup` | Resource group name |
253 | `location` | Azure region |
254
255 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
256 +
257 :::info
258
259 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -264,7 +267,7 @@ Example:
267 ```
268 resources
269 | where tags.env =~ "prod"
267 -| project id, name, type, resourceGroup, location
270 +| project id, name, type, resourceGroup, location, tags
271 ```
272
273
@@ -338,6 +341,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
341 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
342
343
344 +<a id="option-virtual-node-vnode"></a>
345 +##### vnode
346 +
347 +This job-level virtual node is used for metrics written to the default host scope.
348 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
349 +
350 +
351 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
352 +##### virtual_nodes.by_resource_tag
353 +
354 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
355 +
356 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
357 +- Empty or missing tag values use the default job host scope.
358 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
359 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
360 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
361 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
362 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
363 +
364 +
365
366 </details>
367
@@ -409,6 +433,85 @@ jobs:
433 client_secret: "your-client-secret"
434
435 ```
436 +###### Workload virtual nodes from resource tags
437 +
438 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
439 +
440 +<details open><summary>Config</summary>
441 +
442 +```yaml
443 +jobs:
444 + - name: prod-workloads
445 + subscription_ids:
446 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
447 + virtual_nodes:
448 + by_resource_tag: workload
449 + discovery:
450 + mode: filters
451 + profiles:
452 + mode: auto
453 + auth:
454 + mode: service_principal
455 + mode_service_principal:
456 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
458 + client_secret: "your-client-secret"
459 +
460 +```
461 +</details>
462 +
463 +###### Custom KQL with workload virtual nodes
464 +
465 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
466 +
467 +<details open><summary>Config</summary>
468 +
469 +```yaml
470 +jobs:
471 + - name: prod-query-workloads
472 + subscription_ids:
473 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
474 + virtual_nodes:
475 + by_resource_tag: workload
476 + discovery:
477 + mode: query
478 + mode_query:
479 + kql: |
480 + resources
481 + | where tags.env =~ "prod"
482 + | project id, name, type, resourceGroup, location, tags
483 + profiles:
484 + mode: auto
485 + auth:
486 + mode: default
487 +
488 +```
489 +</details>
490 +
491 +###### Job virtual node plus workload virtual nodes
492 +
493 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
494 +
495 +<details open><summary>Config</summary>
496 +
497 +```yaml
498 +jobs:
499 + - name: prod-with-fallback-node
500 + vnode: azure-fallback-node
501 + subscription_ids:
502 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
503 + virtual_nodes:
504 + by_resource_tag: workload
505 + discovery:
506 + mode: filters
507 + profiles:
508 + mode: auto
509 + auth:
510 + mode: managed_identity
511 +
512 +```
513 +</details>
514 +
515 ###### Managed identity with exact profiles
516
517 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -488,7 +591,7 @@ jobs:
591 kql: |
592 resources
593 | where tags.env =~ "prod"
491 - | project id, name, type, resourceGroup, location
594 + | project id, name, type, resourceGroup, location, tags
595 profiles:
596 mode: auto
597 auth:
@@ -686,6 +789,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
789 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
790
791
792 +### Workload virtual nodes are not created
793 +
794 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
795 +
796 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
797 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
798 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
799 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
800 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
801 +
802 +
803 +### More alerts appear after enabling workload virtual nodes
804 +
805 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
806 +
807 +
808 ### Authentication errors in sovereign clouds
809
810 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_synapse_analytics_workspace.md
+124 -5
@@ -188,7 +188,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
188 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
189 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
190 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
191 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
191 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
192 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
193 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
194 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -196,7 +196,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
196 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
197 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
198 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
199 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
199 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
200 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
201
202 <a id="option-collection-query-offset"></a>
203 ##### query_offset
@@ -231,7 +232,7 @@ Controls how the collector finds candidate Azure resources.
232 | Mode | Behavior |
233 |:-----|:---------|
234 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
234 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
235 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
236
237
238 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -250,6 +251,8 @@ The query **must** project these five columns:
251 | `resourceGroup` | Resource group name |
252 | `location` | Azure region |
253
254 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
255 +
256 :::info
257
258 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -263,7 +266,7 @@ Example:
266 ```
267 resources
268 | where tags.env =~ "prod"
266 -| project id, name, type, resourceGroup, location
269 +| project id, name, type, resourceGroup, location, tags
270 ```
271
272
@@ -337,6 +340,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
340 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
341
342
343 +<a id="option-virtual-node-vnode"></a>
344 +##### vnode
345 +
346 +This job-level virtual node is used for metrics written to the default host scope.
347 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
348 +
349 +
350 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
351 +##### virtual_nodes.by_resource_tag
352 +
353 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
354 +
355 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
356 +- Empty or missing tag values use the default job host scope.
357 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
358 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
359 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
360 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
361 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
362 +
363 +
364
365 </details>
366
@@ -408,6 +432,85 @@ jobs:
432 client_secret: "your-client-secret"
433
434 ```
435 +###### Workload virtual nodes from resource tags
436 +
437 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
438 +
439 +<details open><summary>Config</summary>
440 +
441 +```yaml
442 +jobs:
443 + - name: prod-workloads
444 + subscription_ids:
445 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
446 + virtual_nodes:
447 + by_resource_tag: workload
448 + discovery:
449 + mode: filters
450 + profiles:
451 + mode: auto
452 + auth:
453 + mode: service_principal
454 + mode_service_principal:
455 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
456 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
457 + client_secret: "your-client-secret"
458 +
459 +```
460 +</details>
461 +
462 +###### Custom KQL with workload virtual nodes
463 +
464 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
465 +
466 +<details open><summary>Config</summary>
467 +
468 +```yaml
469 +jobs:
470 + - name: prod-query-workloads
471 + subscription_ids:
472 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
473 + virtual_nodes:
474 + by_resource_tag: workload
475 + discovery:
476 + mode: query
477 + mode_query:
478 + kql: |
479 + resources
480 + | where tags.env =~ "prod"
481 + | project id, name, type, resourceGroup, location, tags
482 + profiles:
483 + mode: auto
484 + auth:
485 + mode: default
486 +
487 +```
488 +</details>
489 +
490 +###### Job virtual node plus workload virtual nodes
491 +
492 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
493 +
494 +<details open><summary>Config</summary>
495 +
496 +```yaml
497 +jobs:
498 + - name: prod-with-fallback-node
499 + vnode: azure-fallback-node
500 + subscription_ids:
501 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
502 + virtual_nodes:
503 + by_resource_tag: workload
504 + discovery:
505 + mode: filters
506 + profiles:
507 + mode: auto
508 + auth:
509 + mode: managed_identity
510 +
511 +```
512 +</details>
513 +
514 ###### Managed identity with exact profiles
515
516 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -487,7 +590,7 @@ jobs:
590 kql: |
591 resources
592 | where tags.env =~ "prod"
490 - | project id, name, type, resourceGroup, location
593 + | project id, name, type, resourceGroup, location, tags
594 profiles:
595 mode: auto
596 auth:
@@ -691,6 +794,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
794 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
795
796
797 +### Workload virtual nodes are not created
798 +
799 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
800 +
801 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
802 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
803 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
804 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
805 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
806 +
807 +
808 +### More alerts appear after enabling workload virtual nodes
809 +
810 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
811 +
812 +
813 ### Authentication errors in sovereign clouds
814
815 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_virtual_machine.md
+124 -5
@@ -191,7 +191,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
194 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
194 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
195 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
196 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
197 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -199,7 +199,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
199 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
200 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
201 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
202 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
204
205 <a id="option-collection-query-offset"></a>
206 ##### query_offset
@@ -234,7 +235,7 @@ Controls how the collector finds candidate Azure resources.
235 | Mode | Behavior |
236 |:-----|:---------|
237 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
237 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
238 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
239
240
241 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -253,6 +254,8 @@ The query **must** project these five columns:
254 | `resourceGroup` | Resource group name |
255 | `location` | Azure region |
256
257 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
258 +
259 :::info
260
261 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -266,7 +269,7 @@ Example:
269 ```
270 resources
271 | where tags.env =~ "prod"
269 -| project id, name, type, resourceGroup, location
272 +| project id, name, type, resourceGroup, location, tags
273 ```
274
275
@@ -340,6 +343,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
343 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
344
345
346 +<a id="option-virtual-node-vnode"></a>
347 +##### vnode
348 +
349 +This job-level virtual node is used for metrics written to the default host scope.
350 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
351 +
352 +
353 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
354 +##### virtual_nodes.by_resource_tag
355 +
356 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
357 +
358 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
359 +- Empty or missing tag values use the default job host scope.
360 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
361 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
362 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
363 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
364 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
365 +
366 +
367
368 </details>
369
@@ -411,6 +435,85 @@ jobs:
435 client_secret: "your-client-secret"
436
437 ```
438 +###### Workload virtual nodes from resource tags
439 +
440 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
441 +
442 +<details open><summary>Config</summary>
443 +
444 +```yaml
445 +jobs:
446 + - name: prod-workloads
447 + subscription_ids:
448 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
449 + virtual_nodes:
450 + by_resource_tag: workload
451 + discovery:
452 + mode: filters
453 + profiles:
454 + mode: auto
455 + auth:
456 + mode: service_principal
457 + mode_service_principal:
458 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_secret: "your-client-secret"
461 +
462 +```
463 +</details>
464 +
465 +###### Custom KQL with workload virtual nodes
466 +
467 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
468 +
469 +<details open><summary>Config</summary>
470 +
471 +```yaml
472 +jobs:
473 + - name: prod-query-workloads
474 + subscription_ids:
475 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
476 + virtual_nodes:
477 + by_resource_tag: workload
478 + discovery:
479 + mode: query
480 + mode_query:
481 + kql: |
482 + resources
483 + | where tags.env =~ "prod"
484 + | project id, name, type, resourceGroup, location, tags
485 + profiles:
486 + mode: auto
487 + auth:
488 + mode: default
489 +
490 +```
491 +</details>
492 +
493 +###### Job virtual node plus workload virtual nodes
494 +
495 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
496 +
497 +<details open><summary>Config</summary>
498 +
499 +```yaml
500 +jobs:
501 + - name: prod-with-fallback-node
502 + vnode: azure-fallback-node
503 + subscription_ids:
504 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
505 + virtual_nodes:
506 + by_resource_tag: workload
507 + discovery:
508 + mode: filters
509 + profiles:
510 + mode: auto
511 + auth:
512 + mode: managed_identity
513 +
514 +```
515 +</details>
516 +
517 ###### Managed identity with exact profiles
518
519 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -490,7 +593,7 @@ jobs:
593 kql: |
594 resources
595 | where tags.env =~ "prod"
493 - | project id, name, type, resourceGroup, location
596 + | project id, name, type, resourceGroup, location, tags
597 profiles:
598 mode: auto
599 auth:
@@ -728,6 +831,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
831 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
832
833
834 +### Workload virtual nodes are not created
835 +
836 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
837 +
838 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
839 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
840 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
841 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
842 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
843 +
844 +
845 +### More alerts appear after enabling workload virtual nodes
846 +
847 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
848 +
849 +
850 ### Authentication errors in sovereign clouds
851
852 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_virtual_machine_scale_set.md
+124 -5
@@ -193,7 +193,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
193 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
194 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
195 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
196 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
196 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
197 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
198 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
199 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -201,7 +201,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
201 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
202 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
203 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
204 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
204 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
205 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
206
207 <a id="option-collection-query-offset"></a>
208 ##### query_offset
@@ -236,7 +237,7 @@ Controls how the collector finds candidate Azure resources.
237 | Mode | Behavior |
238 |:-----|:---------|
239 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
239 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
240 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
241
242
243 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -255,6 +256,8 @@ The query **must** project these five columns:
256 | `resourceGroup` | Resource group name |
257 | `location` | Azure region |
258
259 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
260 +
261 :::info
262
263 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -268,7 +271,7 @@ Example:
271 ```
272 resources
273 | where tags.env =~ "prod"
271 -| project id, name, type, resourceGroup, location
274 +| project id, name, type, resourceGroup, location, tags
275 ```
276
277
@@ -342,6 +345,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
345 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
346
347
348 +<a id="option-virtual-node-vnode"></a>
349 +##### vnode
350 +
351 +This job-level virtual node is used for metrics written to the default host scope.
352 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
353 +
354 +
355 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
356 +##### virtual_nodes.by_resource_tag
357 +
358 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
359 +
360 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
361 +- Empty or missing tag values use the default job host scope.
362 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
363 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
364 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
365 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
366 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
367 +
368 +
369
370 </details>
371
@@ -413,6 +437,85 @@ jobs:
437 client_secret: "your-client-secret"
438
439 ```
440 +###### Workload virtual nodes from resource tags
441 +
442 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
443 +
444 +<details open><summary>Config</summary>
445 +
446 +```yaml
447 +jobs:
448 + - name: prod-workloads
449 + subscription_ids:
450 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
451 + virtual_nodes:
452 + by_resource_tag: workload
453 + discovery:
454 + mode: filters
455 + profiles:
456 + mode: auto
457 + auth:
458 + mode: service_principal
459 + mode_service_principal:
460 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
461 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
462 + client_secret: "your-client-secret"
463 +
464 +```
465 +</details>
466 +
467 +###### Custom KQL with workload virtual nodes
468 +
469 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
470 +
471 +<details open><summary>Config</summary>
472 +
473 +```yaml
474 +jobs:
475 + - name: prod-query-workloads
476 + subscription_ids:
477 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
478 + virtual_nodes:
479 + by_resource_tag: workload
480 + discovery:
481 + mode: query
482 + mode_query:
483 + kql: |
484 + resources
485 + | where tags.env =~ "prod"
486 + | project id, name, type, resourceGroup, location, tags
487 + profiles:
488 + mode: auto
489 + auth:
490 + mode: default
491 +
492 +```
493 +</details>
494 +
495 +###### Job virtual node plus workload virtual nodes
496 +
497 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
498 +
499 +<details open><summary>Config</summary>
500 +
501 +```yaml
502 +jobs:
503 + - name: prod-with-fallback-node
504 + vnode: azure-fallback-node
505 + subscription_ids:
506 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
507 + virtual_nodes:
508 + by_resource_tag: workload
509 + discovery:
510 + mode: filters
511 + profiles:
512 + mode: auto
513 + auth:
514 + mode: managed_identity
515 +
516 +```
517 +</details>
518 +
519 ###### Managed identity with exact profiles
520
521 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -492,7 +595,7 @@ jobs:
595 kql: |
596 resources
597 | where tags.env =~ "prod"
495 - | project id, name, type, resourceGroup, location
598 + | project id, name, type, resourceGroup, location, tags
599 profiles:
600 mode: auto
601 auth:
@@ -732,6 +835,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
835 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
836
837
838 +### Workload virtual nodes are not created
839 +
840 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
841 +
842 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
843 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
844 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
845 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
846 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
847 +
848 +
849 +### More alerts appear after enabling workload virtual nodes
850 +
851 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
852 +
853 +
854 ### Authentication errors in sovereign clouds
855
856 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/integrations/azure_vpn_gateway.md
+124 -5
@@ -191,7 +191,7 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
191 | | discovery.mode_filters.resource_groups | Optional list of Azure resource groups to include in `filters` mode. | [] | no |
192 | | discovery.mode_filters.regions | Optional list of Azure regions to include in `filters` mode. | [] | no |
193 | | discovery.mode_filters.tags | Optional exact-match tag filters for `filters` mode. Keys are matched case-insensitively and values case-sensitively. | {} | no |
194 -| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`. | | no |
194 +| | [discovery.mode_query.kql](#option-discovery-discovery-mode-query-kql) | Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`. | | no |
195 | **Profiles** | [profiles.mode](#option-profiles-profiles-mode) | How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both). | auto | no |
196 | | [profiles.mode_auto.entries](#option-profiles-profiles-mode-auto-entries) | Optional per-profile overrides applied only to profiles that auto-activate at bootstrap. | [] | no |
197 | | [profiles.mode_exact.entries](#option-profiles-profiles-mode-exact-entries) | Explicit profile entries used by `exact` mode. | [] | no |
@@ -199,7 +199,8 @@ Custom profiles extend the collector's catalog -- they do not replace the discov
199 | **Limits** | limits.max_concurrency | Maximum concurrent batch queries to Azure Monitor. | 4 | no |
200 | | limits.max_batch_resources | Maximum resources per Azure Monitor batch request. | 50 | no |
201 | | limits.max_metrics_per_query | Maximum metrics per Azure Monitor batch request. | 20 | no |
202 -| **Virtual Node** | vnode | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
202 +| **Virtual Node** | [vnode](#option-virtual-node-vnode) | Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes). | | no |
203 +| | [virtual_nodes.by_resource_tag](#option-virtual-node-virtual-nodes-by-resource-tag) | Creates workload virtual nodes from an Azure resource tag value. | | no |
204
205 <a id="option-collection-query-offset"></a>
206 ##### query_offset
@@ -234,7 +235,7 @@ Controls how the collector finds candidate Azure resources.
235 | Mode | Behavior |
236 |:-----|:---------|
237 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
237 -| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
238 +| `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
239
240
241 <a id="option-discovery-discovery-mode-query-kql"></a>
@@ -253,6 +254,8 @@ The query **must** project these five columns:
254 | `resourceGroup` | Resource group name |
255 | `location` | Azure region |
256
257 +If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
258 +
259 :::info
260
261 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -266,7 +269,7 @@ Example:
269 ```
270 resources
271 | where tags.env =~ "prod"
269 -| project id, name, type, resourceGroup, location
272 +| project id, name, type, resourceGroup, location, tags
273 ```
274
275
@@ -340,6 +343,27 @@ If an explicit `combined` entry matches an auto-selected profile, the collector
343 In `discovery.mode: query`, only `filters.resource_groups` and `filters.regions` apply.
344
345
346 +<a id="option-virtual-node-vnode"></a>
347 +##### vnode
348 +
349 +This job-level virtual node is used for metrics written to the default host scope.
350 +If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
351 +
352 +
353 +<a id="option-virtual-node-virtual-nodes-by-resource-tag"></a>
354 +##### virtual_nodes.by_resource_tag
355 +
356 +Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
357 +
358 +- Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
359 +- Empty or missing tag values use the default job host scope.
360 +- Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
361 +- The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
362 +- Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
363 +- Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
364 +- The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
365 +
366 +
367
368 </details>
369
@@ -411,6 +435,85 @@ jobs:
435 client_secret: "your-client-secret"
436
437 ```
438 +###### Workload virtual nodes from resource tags
439 +
440 +Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
441 +
442 +<details open><summary>Config</summary>
443 +
444 +```yaml
445 +jobs:
446 + - name: prod-workloads
447 + subscription_ids:
448 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
449 + virtual_nodes:
450 + by_resource_tag: workload
451 + discovery:
452 + mode: filters
453 + profiles:
454 + mode: auto
455 + auth:
456 + mode: service_principal
457 + mode_service_principal:
458 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
459 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
460 + client_secret: "your-client-secret"
461 +
462 +```
463 +</details>
464 +
465 +###### Custom KQL with workload virtual nodes
466 +
467 +In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
468 +
469 +<details open><summary>Config</summary>
470 +
471 +```yaml
472 +jobs:
473 + - name: prod-query-workloads
474 + subscription_ids:
475 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
476 + virtual_nodes:
477 + by_resource_tag: workload
478 + discovery:
479 + mode: query
480 + mode_query:
481 + kql: |
482 + resources
483 + | where tags.env =~ "prod"
484 + | project id, name, type, resourceGroup, location, tags
485 + profiles:
486 + mode: auto
487 + auth:
488 + mode: default
489 +
490 +```
491 +</details>
492 +
493 +###### Job virtual node plus workload virtual nodes
494 +
495 +Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
496 +
497 +<details open><summary>Config</summary>
498 +
499 +```yaml
500 +jobs:
501 + - name: prod-with-fallback-node
502 + vnode: azure-fallback-node
503 + subscription_ids:
504 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
505 + virtual_nodes:
506 + by_resource_tag: workload
507 + discovery:
508 + mode: filters
509 + profiles:
510 + mode: auto
511 + auth:
512 + mode: managed_identity
513 +
514 +```
515 +</details>
516 +
517 ###### Managed identity with exact profiles
518
519 Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
@@ -490,7 +593,7 @@ jobs:
593 kql: |
594 resources
595 | where tags.env =~ "prod"
493 - | project id, name, type, resourceGroup, location
596 + | project id, name, type, resourceGroup, location, tags
597 profiles:
598 mode: auto
599 auth:
@@ -721,6 +824,22 @@ Azure Monitor metrics have a built-in reporting delay of **1-3 minutes**.
824 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
825
826
827 +### Workload virtual nodes are not created
828 +
829 +When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
830 +
831 +- The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
832 +- In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
833 +- Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
834 +- The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
835 +- Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
836 +
837 +
838 +### More alerts appear after enabling workload virtual nodes
839 +
840 +Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
841 +
842 +
843 ### Authentication errors in sovereign clouds
844
845 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/metadata.yaml
+91 -4
@@ -225,7 +225,7 @@ modules:
225 | Mode | Behavior |
226 |:-----|:---------|
227 | `filters` | Builds an Azure Resource Graph query from the structured `mode_filters.*` options (resource groups, regions, tags). This is the default. |
228 - | `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. |
228 + | `query` | Uses the raw KQL you provide in `discovery.mode_query.kql`. The query must project `id`, `name`, `type`, `resourceGroup`, and `location`. Project `tags` too when using `virtual_nodes.by_resource_tag`. |
229 - name: discovery.mode_filters.resource_groups
230 description: Optional list of Azure resource groups to include in `filters` mode.
231 default_value: "[]"
@@ -242,7 +242,7 @@ modules:
242 required: false
243 group: Discovery
244 - name: discovery.mode_query.kql
245 - description: Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`.
245 + description: Custom Azure Resource Graph KQL for `query` mode. Must project `id`, `name`, `type`, `resourceGroup`, `location`; project `tags` too when using `virtual_nodes.by_resource_tag`.
246 default_value: ""
247 required: false
248 group: Discovery
@@ -260,6 +260,8 @@ modules:
260 | `resourceGroup` | Resource group name |
261 | `location` | Azure region |
262
263 + If `virtual_nodes.by_resource_tag` is set, also project `tags`. Missing `tags` or non-object-shaped `tags` do not fail discovery; affected resources fall back to the default job host scope and the collector logs a warning once per successful discovery refresh.
264 +
265 :::info
266
267 - Returned resource `type` values should match the Azure resource types expected by the active profiles.
@@ -273,7 +275,7 @@ modules:
275 ```
276 resources
277 | where tags.env =~ "prod"
276 - | project id, name, type, resourceGroup, location
278 + | project id, name, type, resourceGroup, location, tags
279 ```
280 - name: profiles.mode
281 description: "How profiles are selected: `auto` (discover from resources), `exact` (explicit list), or `combined` (both)."
@@ -369,6 +371,24 @@ modules:
371 default_value: ""
372 required: false
373 group: Virtual Node
374 + detailed_description: |
375 + This job-level virtual node is used for metrics written to the default host scope.
376 + If `virtual_nodes.by_resource_tag` is also set, resources that have a safe value for the configured tag are routed to workload virtual nodes instead. Resources without that tag, with an empty value, or with an unsafe hostname value continue to use the job-level `vnode` when configured.
377 + - name: virtual_nodes.by_resource_tag
378 + description: Creates workload virtual nodes from an Azure resource tag value.
379 + default_value: ""
380 + required: false
381 + group: Virtual Node
382 + detailed_description: |
383 + Set this to an Azure resource tag key, for example `workload`, to route resource metrics under virtual nodes named after that tag value.
384 +
385 + - Tag keys are matched case-insensitively; tag values keep their Azure case and are trimmed.
386 + - Empty or missing tag values use the default job host scope.
387 + - Unsafe hostname values use the default job host scope and are summarized in a warning once per successful discovery refresh.
388 + - The virtual node GUID is deterministic from `azure_monitor:` plus the trimmed tag value, so the same Azure workload value maps to the same Azure Monitor virtual node across jobs and subscriptions.
389 + - Virtual node metadata includes `_vnode_type=azure_workload`. Netdata also adds `_hostname` when emitting the virtual node definition.
390 + - Workload virtual nodes increase host and alert cardinality. Health alerts defined for Azure Monitor charts attach per workload virtual node.
391 + - The metrix scoped vec cache grows with observed workload scope and label cardinality during the collector runtime; keep the configured tag's cardinality bounded by operator policy.
392 examples:
393 folding:
394 title: Config
@@ -402,6 +422,61 @@ modules:
422 tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
423 client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
424 client_secret: "your-client-secret"
425 + - name: Workload virtual nodes from resource tags
426 + description: Route resources with `tags.workload` to virtual nodes named after the workload value. Resources without the tag stay on the default job host scope.
427 + config: |
428 + jobs:
429 + - name: prod-workloads
430 + subscription_ids:
431 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
432 + virtual_nodes:
433 + by_resource_tag: workload
434 + discovery:
435 + mode: filters
436 + profiles:
437 + mode: auto
438 + auth:
439 + mode: service_principal
440 + mode_service_principal:
441 + tenant_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
442 + client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
443 + client_secret: "your-client-secret"
444 + - name: Custom KQL with workload virtual nodes
445 + description: In `query` discovery mode, project `tags` so `virtual_nodes.by_resource_tag` can derive workload virtual nodes.
446 + config: |
447 + jobs:
448 + - name: prod-query-workloads
449 + subscription_ids:
450 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
451 + virtual_nodes:
452 + by_resource_tag: workload
453 + discovery:
454 + mode: query
455 + mode_query:
456 + kql: |
457 + resources
458 + | where tags.env =~ "prod"
459 + | project id, name, type, resourceGroup, location, tags
460 + profiles:
461 + mode: auto
462 + auth:
463 + mode: default
464 + - name: Job virtual node plus workload virtual nodes
465 + description: Use a job-level `vnode` as the default host scope while tagged resources route to workload virtual nodes.
466 + config: |
467 + jobs:
468 + - name: prod-with-fallback-node
469 + vnode: azure-fallback-node
470 + subscription_ids:
471 + - "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
472 + virtual_nodes:
473 + by_resource_tag: workload
474 + discovery:
475 + mode: filters
476 + profiles:
477 + mode: auto
478 + auth:
479 + mode: managed_identity
480 - name: Managed identity with exact profiles
481 description: Use a managed identity (on an Azure VM, VMSS, or AKS) and monitor only SQL Database and PostgreSQL Flexible Server resources -- skip auto-discovery of other services.
482 config: |
@@ -461,7 +536,7 @@ modules:
536 kql: |
537 resources
538 | where tags.env =~ "prod"
464 - | project id, name, type, resourceGroup, location
539 + | project id, name, type, resourceGroup, location, tags
540 profiles:
541 mode: auto
542 auth:
@@ -515,6 +590,18 @@ modules:
590 - The collector uses `query_offset` (default: **180 seconds**) as the minimum offset for metric query windows.
591 - Slower time-grain batches automatically use a larger effective offset when needed.
592 - If metrics are still missing or incomplete, increase `query_offset` to **240** or **300** seconds.
593 + - name: Workload virtual nodes are not created
594 + description: |
595 + When `virtual_nodes.by_resource_tag` is set, check the discovered resource tags:
596 +
597 + - The configured tag key is matched case-insensitively, but the value must be non-empty after trimming.
598 + - In `discovery.mode: query`, the KQL must project `tags`. If the `tags` column is missing or not an object, the collector logs a warning and affected resources use the default job host scope.
599 + - Tag values must be safe Netdata hostnames. Unsafe values are summarized in a warning once per successful discovery refresh and affected resources use the default job host scope.
600 + - The same Azure workload value intentionally maps to the same Azure Monitor virtual node across subscriptions and jobs. Use distinct tag values when tenants should remain separate.
601 + - Azure Monitor namespaces workload GUID input with `azure_monitor:` to avoid accidental collisions with other collectors that derive virtual node GUIDs from hostnames.
602 + - name: More alerts appear after enabling workload virtual nodes
603 + description: |
604 + Workload virtual nodes multiply the host scope of Azure Monitor charts. Health alerts attached to those charts evaluate per workload virtual node, so alert volume can increase with the number of distinct workload tag values.
605 - name: Authentication errors in sovereign clouds
606 description: |
607 For Azure Government or Azure China clouds, set the `cloud` parameter:
src/go/plugin/go.d/collector/azure_monitor/observation_state.go
+45 -32
@@ -3,29 +3,40 @@
3 package azure_monitor
4
5 import (
6 - "strings"
7 -
6 "github.com/netdata/netdata/go/plugins/pkg/metrix"
7 "github.com/netdata/netdata/go/plugins/plugin/go.d/collector/azure_monitor/azureprofiles"
8 )
9
10 +type observationLabelIdentity [7]string
11 +
12 +type scopedLabelIdentity struct {
13 + ScopeKey string
14 + Labels observationLabelIdentity
15 +}
16 +
17 +type observationKey struct {
18 + Instrument string
19 + ScopeKey string
20 + Labels observationLabelIdentity
21 +}
22 +
23 type observationState struct {
24 instruments map[string]*instrumentRuntime
14 - accumulators map[string]float64
15 - lastObserved map[string]lastObservation
25 + accumulators map[observationKey]float64
26 + lastObserved map[observationKey]lastObservation
27 }
28
29 func newObservationState(instruments map[string]*instrumentRuntime) *observationState {
30 return &observationState{
31 instruments: instruments,
21 - accumulators: make(map[string]float64),
22 - lastObserved: make(map[string]lastObservation),
32 + accumulators: make(map[observationKey]float64),
33 + lastObserved: make(map[observationKey]lastObservation),
34 }
35 }
36
37 func (s *observationState) reset() {
27 - s.accumulators = make(map[string]float64)
28 - s.lastObserved = make(map[string]lastObservation)
38 + s.accumulators = make(map[observationKey]float64)
39 + s.lastObserved = make(map[observationKey]lastObservation)
40 }
41
42 func dueInstrumentsForBatches(batches []queryBatch) map[string]bool {
@@ -40,8 +51,8 @@ func dueInstrumentsForBatches(batches []queryBatch) map[string]bool {
51 return dueInstruments
52 }
53
43 -func (s *observationState) observeSamples(samples []metricSample) map[string]bool {
44 - observedThisCycle := make(map[string]bool, len(s.lastObserved))
54 +func (s *observationState) observeSamples(samples []metricSample) map[observationKey]bool {
55 + observedThisCycle := make(map[observationKey]bool, len(s.lastObserved))
56 for _, sample := range samples {
57 key, ok := s.observeSample(sample)
58 if !ok {
@@ -52,14 +63,14 @@ func (s *observationState) observeSamples(samples []metricSample) map[string]boo
63 return observedThisCycle
64 }
65
55 -func (s *observationState) observeSample(sample metricSample) (string, bool) {
66 +func (s *observationState) observeSample(sample metricSample) (observationKey, bool) {
67 inst, ok := s.instruments[sample.Instrument]
68 if !ok {
58 - return "", false
69 + return observationKey{}, false
70 }
71
72 values := labelValues(sample.Labels)
62 - key := sampleObservationKey(sample.Instrument, values)
73 + key := sampleObservationKey(sample.Instrument, sample.Scope, values)
74 value := sample.Value
75
76 if sample.Kind == azureprofiles.SeriesKindCounter {
@@ -67,9 +78,10 @@ func (s *observationState) observeSample(sample metricSample) (string, bool) {
78 value = s.accumulators[key]
79 }
80
70 - inst.observe(values, value)
81 + inst.observe(sample.Scope, values, value)
82 s.lastObserved[key] = lastObservation{
83 instrument: sample.Instrument,
84 + scope: sample.Scope,
85 labelValues: append([]string(nil), values...),
86 value: value,
87 }
@@ -77,7 +89,7 @@ func (s *observationState) observeSample(sample metricSample) (string, bool) {
89 return key, true
90 }
91
80 -func (s *observationState) reobserveCachedObservations(dueInstruments, observedThisCycle map[string]bool) {
92 +func (s *observationState) reobserveCachedObservations(dueInstruments map[string]bool, observedThisCycle map[observationKey]bool) {
93 for key, obs := range s.lastObserved {
94 if observedThisCycle[key] {
95 continue
@@ -89,17 +101,20 @@ func (s *observationState) reobserveCachedObservations(dueInstruments, observedT
101 if !ok {
102 continue
103 }
92 - inst.observe(obs.labelValues, obs.value)
104 + inst.observe(obs.scope, obs.labelValues, obs.value)
105 }
106 }
107
108 // pruneStaleResources removes cache entries for resources that are no longer
109 // active for a specific profile/label identity.
110 func (s *observationState) pruneStaleResources(current map[string][]resourceInfo) {
99 - activeLabels := make(map[string]struct{})
111 + activeLabels := make(map[scopedLabelIdentity]struct{})
112 for profileName, resources := range current {
113 for _, resource := range resources {
102 - activeLabels[labelIdentity(labelValues(resourceLabels(resource, profileName)))] = struct{}{}
114 + activeLabels[scopedLabelIdentity{
115 + ScopeKey: resource.HostScope.ScopeKey,
116 + Labels: labelIdentity(labelValues(resourceLabels(resource, profileName))),
117 + }] = struct{}{}
118 }
119 }
120
@@ -107,34 +122,32 @@ func (s *observationState) pruneStaleResources(current map[string][]resourceInfo
122 if len(obs.labelValues) == 0 {
123 continue
124 }
110 - if _, ok := activeLabels[labelIdentity(obs.labelValues)]; ok {
125 + if _, ok := activeLabels[scopedLabelIdentity{ScopeKey: obs.scope.ScopeKey, Labels: labelIdentity(obs.labelValues)}]; ok {
126 continue
127 }
128 delete(s.lastObserved, key)
129 }
130
131 for key := range s.accumulators {
117 - if _, ok := activeLabels[labelIdentityFromObservationKey(key)]; ok {
132 + if _, ok := activeLabels[scopedLabelIdentity{ScopeKey: key.ScopeKey, Labels: key.Labels}]; ok {
133 continue
134 }
135 delete(s.accumulators, key)
136 }
137 }
138
124 -func sampleObservationKey(instrument string, values []string) string {
125 - return instrument + "\x00" + strings.Join(values, "\x00")
126 -}
127 -
128 -func labelIdentity(values []string) string {
129 - return strings.Join(values, "\x00")
139 +func sampleObservationKey(instrument string, scope metrix.HostScope, values []string) observationKey {
140 + return observationKey{
141 + Instrument: instrument,
142 + ScopeKey: scope.ScopeKey,
143 + Labels: labelIdentity(values),
144 + }
145 }
146
132 -func labelIdentityFromObservationKey(key string) string {
133 - parts := strings.SplitN(key, "\x00", 2)
134 - if len(parts) < 2 {
135 - return ""
136 - }
137 - return parts[1]
147 +func labelIdentity(values []string) observationLabelIdentity {
148 + var out observationLabelIdentity
149 + copy(out[:], values)
150 + return out
151 }
152
153 func labelValues(labels metrix.Labels) []string {
src/go/plugin/go.d/collector/azure_monitor/plan.go
+3 -2
@@ -12,14 +12,15 @@ import (
12 "gopkg.in/yaml.v3"
13 )
14
15 -func buildCollectorRuntimeFromConfig(profileNames []string, profileEntries map[string]ProfileEntryConfig, catalog azureprofiles.Catalog) (*collectorRuntime, error) {
15 +func buildCollectorRuntimeFromConfig(profileNames []string, profileEntries map[string]ProfileEntryConfig, catalog azureprofiles.Catalog, workloadResourceTagKey string) (*collectorRuntime, error) {
16 profiles, err := catalog.Resolve(profileNames)
17 if err != nil {
18 return nil, err
19 }
20
21 runtime := &collectorRuntime{
22 - Profiles: make([]*profileRuntime, 0, len(profiles)),
22 + Profiles: make([]*profileRuntime, 0, len(profiles)),
23 + WorkloadResourceTagKey: stringsLowerTrim(workloadResourceTagKey),
24 }
25
26 seenProfileNames := make(map[string]struct{}, len(profiles))
src/go/plugin/go.d/collector/azure_monitor/query_executor.go
+3 -2
@@ -155,12 +155,12 @@ func samplesFromQueryResponse(metricData []azmetrics.MetricData, profileName str
155 if !ok {
156 continue
157 }
158 - samples = append(samples, samplesFromMetricValues(data.Values, resourceLabels(resource, profileName), metricToRuntime)...)
158 + samples = append(samples, samplesFromMetricValues(data.Values, resource.HostScope, resourceLabels(resource, profileName), metricToRuntime)...)
159 }
160 return samples
161 }
162
163 -func samplesFromMetricValues(metrics []azmetrics.Metric, labels metrix.Labels, metricToRuntime map[string]*metricRuntime) []metricSample {
163 +func samplesFromMetricValues(metrics []azmetrics.Metric, scope metrix.HostScope, labels metrix.Labels, metricToRuntime map[string]*metricRuntime) []metricSample {
164 samples := make([]metricSample, 0, len(metrics))
165 for _, metric := range metrics {
166 runtimeMetric, ok := metricToRuntime[stringsLowerTrim(derefOrZero(metric.Name.Value))]
@@ -176,6 +176,7 @@ func samplesFromMetricValues(metrics []azmetrics.Metric, labels metrix.Labels, m
176 Instrument: series.Instrument,
177 Kind: series.Kind,
178 Labels: labels,
179 + Scope: scope,
180 Value: value,
181 })
182 }
src/go/plugin/go.d/collector/azure_monitor/testdata/config_workload.json new
+44
@@ -0,0 +1,44 @@
1 +{
2 + "update_every": 60,
3 + "subscription_ids": [
4 + "sub-1"
5 + ],
6 + "cloud": "public",
7 + "discovery": {
8 + "refresh_every": 300,
9 + "mode": "filters",
10 + "mode_filters": {
11 + "resource_groups": [
12 + "rg-a"
13 + ]
14 + }
15 + },
16 + "profiles": {
17 + "mode": "exact",
18 + "mode_exact": {
19 + "entries": [
20 + {
21 + "name": "sql_managed_instance"
22 + }
23 + ]
24 + }
25 + },
26 + "virtual_nodes": {
27 + "by_resource_tag": "workload"
28 + },
29 + "query_offset": 180,
30 + "timeout": 30,
31 + "limits": {
32 + "max_concurrency": 4,
33 + "max_batch_resources": 50,
34 + "max_metrics_per_query": 20
35 + },
36 + "auth": {
37 + "mode": "service_principal",
38 + "mode_service_principal": {
39 + "tenant_id": "11111111-1111-1111-1111-111111111111",
40 + "client_id": "22222222-2222-2222-2222-222222222222",
41 + "client_secret": "secret"
42 + }
43 + }
44 +}
src/go/plugin/go.d/collector/azure_monitor/testdata/config_workload.yaml new
+29
@@ -0,0 +1,29 @@
1 +update_every: 60
2 +subscription_ids:
3 + - sub-1
4 +cloud: public
5 +discovery:
6 + refresh_every: 300
7 + mode: filters
8 + mode_filters:
9 + resource_groups:
10 + - rg-a
11 +profiles:
12 + mode: exact
13 + mode_exact:
14 + entries:
15 + - name: sql_managed_instance
16 +virtual_nodes:
17 + by_resource_tag: workload
18 +query_offset: 180
19 +timeout: 30
20 +limits:
21 + max_concurrency: 4
22 + max_batch_resources: 50
23 + max_metrics_per_query: 20
24 +auth:
25 + mode: service_principal
26 + mode_service_principal:
27 + tenant_id: 11111111-1111-1111-1111-111111111111
28 + client_id: 22222222-2222-2222-2222-222222222222
29 + client_secret: secret
src/go/plugin/go.d/collector/azure_monitor/workload_scope.go new
+129
@@ -0,0 +1,129 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package azure_monitor
4 +
5 +import (
6 + "fmt"
7 + "slices"
8 + "strconv"
9 + "strings"
10 +
11 + "github.com/google/uuid"
12 + "github.com/netdata/netdata/go/plugins/pkg/metrix"
13 + "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
14 + "github.com/netdata/netdata/go/plugins/plugin/framework/chartemit"
15 +)
16 +
17 +const (
18 + azureWorkloadScopeLabelKey = "_vnode_type"
19 + azureWorkloadScopeLabelValue = "azure_workload"
20 + azureWorkloadGUIDPrefix = "azure_monitor:"
21 +)
22 +
23 +type workloadScopeReport struct {
24 + unsafeValues map[string]int
25 +}
26 +
27 +func applyWorkloadHostScopes(resources []resourceInfo, runtime *collectorRuntime) workloadScopeReport {
28 + if runtime == nil || runtime.WorkloadResourceTagKey == "" {
29 + return workloadScopeReport{}
30 + }
31 +
32 + var report workloadScopeReport
33 + for i := range resources {
34 + scope, unsafeValue := workloadHostScope(resources[i], runtime.WorkloadResourceTagKey)
35 + resources[i].HostScope = scope
36 + if unsafeValue == "" {
37 + continue
38 + }
39 + if report.unsafeValues == nil {
40 + report.unsafeValues = make(map[string]int)
41 + }
42 + report.unsafeValues[unsafeValue]++
43 + }
44 + return report
45 +}
46 +
47 +func workloadHostScope(resource resourceInfo, tagKey string) (metrix.HostScope, string) {
48 + value := workloadTagValue(resource.Tags, tagKey)
49 + if value == "" {
50 + return metrix.HostScope{}, ""
51 + }
52 +
53 + guid := uuid.NewSHA1(uuid.NameSpaceDNS, []byte(azureWorkloadGUIDPrefix+value)).String()
54 + scope := metrix.HostScope{
55 + ScopeKey: guid,
56 + GUID: guid,
57 + Hostname: value,
58 + Labels: map[string]string{
59 + azureWorkloadScopeLabelKey: azureWorkloadScopeLabelValue,
60 + },
61 + }
62 + if _, err := chartemit.PrepareHostInfo(netdataapi.HostInfo{
63 + GUID: scope.GUID,
64 + Hostname: scope.Hostname,
65 + Labels: scope.Labels,
66 + }); err != nil {
67 + return metrix.HostScope{}, value
68 + }
69 + return scope, ""
70 +}
71 +
72 +func workloadTagValue(tags []resourceTag, tagKey string) string {
73 + tagKey = stringsLowerTrim(tagKey)
74 + if tagKey == "" {
75 + return ""
76 + }
77 + for _, tag := range tags {
78 + if tag.Key != tagKey {
79 + continue
80 + }
81 + return stringsTrim(tag.Value)
82 + }
83 + return ""
84 +}
85 +
86 +func (c *Collector) warnDiscoveryScopeFallbacks(state discoveryState, runtime *collectorRuntime) {
87 + if runtime == nil || runtime.WorkloadResourceTagKey == "" || state.FetchCounter == 0 {
88 + return
89 + }
90 +
91 + if state.QueryTagsColumnMissing && (!c.tagsColumnMissingWarned || c.tagsColumnMissingWarnedAt != state.FetchCounter) {
92 + c.tagsColumnMissingWarned = true
93 + c.tagsColumnMissingWarnedAt = state.FetchCounter
94 + c.Warningf(
95 + "virtual_nodes.by_resource_tag %q is enabled, but the custom discovery query did not return a tags column; resources from that query will use the default host scope",
96 + runtime.WorkloadResourceTagKey,
97 + )
98 + }
99 + if state.QueryTagsWrongShape && (!c.tagsWrongShapeWarned || c.tagsWrongShapeWarnedAt != state.FetchCounter) {
100 + c.tagsWrongShapeWarned = true
101 + c.tagsWrongShapeWarnedAt = state.FetchCounter
102 + c.Warningf(
103 + "virtual_nodes.by_resource_tag %q is enabled, but the custom discovery query returned non-object tags; affected resources will use the default host scope",
104 + runtime.WorkloadResourceTagKey,
105 + )
106 + }
107 + if len(state.UnsafeWorkloadValues) == 0 {
108 + return
109 + }
110 + c.Warningf(
111 + "virtual_nodes.by_resource_tag %q ignored unsafe tag values; affected resources will use the default host scope: %s",
112 + runtime.WorkloadResourceTagKey,
113 + formatUnsafeWorkloadValues(state.UnsafeWorkloadValues),
114 + )
115 +}
116 +
117 +func formatUnsafeWorkloadValues(values map[string]int) string {
118 + keys := make([]string, 0, len(values))
119 + for value := range values {
120 + keys = append(keys, value)
121 + }
122 + slices.Sort(keys)
123 +
124 + parts := make([]string, 0, len(keys))
125 + for _, value := range keys {
126 + parts = append(parts, fmt.Sprintf("%s (%d resources)", strconv.Quote(value), values[value]))
127 + }
128 + return strings.Join(parts, ", ")
129 +}
src/go/plugin/go.d/config/go.d/azure_monitor.conf
+135 -26
@@ -4,21 +4,117 @@
4 ## Profile files:
5 ## stock: /usr/lib/netdata/conf.d/go.d/azure_monitor.profiles/default/
6 ## user: /etc/netdata/go.d/azure_monitor.profiles/
7 -## User profile files override stock files when profile `id` matches.
8 -## Filenames are packaging only; matching filename == profile id is recommended.
9 -## The default profile catalog is loaded once per go.d process and cached after the first successful load.
10 -## Changes to profile files under those default dirs require a go.d process restart to take effect.
7 +## User profile files override stock files when the profile basename matches.
8 +## The default profile catalog is loaded once per go.d process and cached after
9 +## the first successful load. Changes under the default profile directories
10 +## require a go.d process restart.
11 +##
12 +## Discovery modes:
13 +## filters - Netdata builds an Azure Resource Graph query from structured
14 +## filters and always projects tags.
15 +## query - you provide Azure Resource Graph KQL. The result must project
16 +## id, name, type, resourceGroup, and location. Project tags too
17 +## when using virtual_nodes.by_resource_tag.
18 ##
19 ## Profile selection modes:
13 -## auto - discovers resource types via Azure Resource Graph and enables matching profiles (default).
14 -## exact - uses only the explicitly listed profile ids.
15 -## combined - merges explicitly listed profile ids with auto-discovered profiles.
20 +## auto - discovers resource types and enables matching profiles.
21 +## exact - uses only explicitly listed profile entries.
22 +## combined - merges explicitly listed profile entries with auto-discovered
23 +## profiles.
24
25 #jobs:
18 -# - name: example_auto
19 -# subscription_id: "<subscription-id>"
20 -# timeout: 30
21 -# profile_selection_mode: auto
26 +# - name: service_principal_filters_auto
27 +# subscription_ids:
28 +# - "<subscription-id>"
29 +# discovery:
30 +# mode: filters
31 +# mode_filters:
32 +# resource_groups:
33 +# - production-rg
34 +# regions:
35 +# - eastus
36 +# tags:
37 +# env:
38 +# - prod
39 +# profiles:
40 +# mode: auto
41 +# auth:
42 +# mode: service_principal
43 +# mode_service_principal:
44 +# tenant_id: "<tenant-id>"
45 +# client_id: "<client-id>"
46 +# client_secret: "<client-secret>"
47 +#
48 +# - name: service_principal_workload_vnodes
49 +# subscription_ids:
50 +# - "<subscription-id>"
51 +# virtual_nodes:
52 +# by_resource_tag: workload
53 +# discovery:
54 +# mode: filters
55 +# profiles:
56 +# mode: auto
57 +# auth:
58 +# mode: service_principal
59 +# mode_service_principal:
60 +# tenant_id: "<tenant-id>"
61 +# client_id: "<client-id>"
62 +# client_secret: "<client-secret>"
63 +#
64 +# - name: service_principal_query_workload_vnodes
65 +# subscription_ids:
66 +# - "<subscription-id>"
67 +# virtual_nodes:
68 +# by_resource_tag: workload
69 +# discovery:
70 +# mode: query
71 +# mode_query:
72 +# kql: |
73 +# resources
74 +# | where tags.env == 'prod'
75 +# | project id, name, type, resourceGroup, location, tags
76 +# profiles:
77 +# mode: auto
78 +# auth:
79 +# mode: service_principal
80 +# mode_service_principal:
81 +# tenant_id: "<tenant-id>"
82 +# client_id: "<client-id>"
83 +# client_secret: "<client-secret>"
84 +#
85 +# - name: managed_identity_exact
86 +# subscription_ids:
87 +# - "<subscription-id>"
88 +# profiles:
89 +# mode: exact
90 +# mode_exact:
91 +# entries:
92 +# - name: postgres_flexible
93 +# - name: cosmos_db
94 +# auth:
95 +# mode: managed_identity
96 +# mode_managed_identity:
97 +# client_id: "<optional-user-assigned-managed-identity-client-id>"
98 +#
99 +# - name: default_credential_combined
100 +# subscription_ids:
101 +# - "<subscription-id>"
102 +# profiles:
103 +# mode: combined
104 +# mode_combined:
105 +# entries:
106 +# - name: sql_database
107 +# auth:
108 +# mode: default
109 +#
110 +# - name: multiple_subscriptions
111 +# subscription_ids:
112 +# - "<subscription-id-a>"
113 +# - "<subscription-id-b>"
114 +# discovery:
115 +# mode: filters
116 +# profiles:
117 +# mode: auto
118 # auth:
119 # mode: service_principal
120 # mode_service_principal:
@@ -26,14 +122,24 @@
122 # client_id: "<client-id>"
123 # client_secret: "<client-secret>"
124 #
29 -# - name: example_exact
30 -# subscription_id: "<subscription-id>"
31 -# timeout: 30
32 -# profile_selection_mode: exact
33 -# profile_selection_mode_exact:
34 -# profiles:
35 -# - postgres_flexible
36 -# - cosmos_db
125 +# - name: profile_level_filters
126 +# subscription_ids:
127 +# - "<subscription-id>"
128 +# discovery:
129 +# mode: filters
130 +# profiles:
131 +# mode: exact
132 +# mode_exact:
133 +# entries:
134 +# - name: postgres_flexible
135 +# filters:
136 +# resource_groups:
137 +# - database-rg
138 +# regions:
139 +# - eastus
140 +# tags:
141 +# role:
142 +# - primary
143 # auth:
144 # mode: service_principal
145 # mode_service_principal:
@@ -41,13 +147,16 @@
147 # client_id: "<client-id>"
148 # client_secret: "<client-secret>"
149 #
44 -# - name: example_combined
45 -# subscription_id: "<subscription-id>"
46 -# timeout: 30
47 -# profile_selection_mode: combined
48 -# profile_selection_mode_combined:
49 -# profiles:
50 -# - custom_profile
150 +# - name: job_vnode_with_workload_vnodes
151 +# vnode: azure-fallback-node
152 +# subscription_ids:
153 +# - "<subscription-id>"
154 +# virtual_nodes:
155 +# by_resource_tag: workload
156 +# discovery:
157 +# mode: filters
158 +# profiles:
159 +# mode: auto
160 # auth:
161 # mode: service_principal
162 # mode_service_principal: