7b860459d · cryptotaxi247/netdata-1

@@ -1,1019 +0,0 @@

1 -## TL;DR

2 -

3 -- Purpose: reduce `netflow.plugin` memory footprint drastically under sustained ingest, with a target of at least 30x reduction versus the currently observed catastrophic baseline on office traffic.

4 -- Costa reported: the running plugin uses about 2.4 GB RAM for roughly 20 flows/s. That is operationally unacceptable and likely scales into failure at higher flow rates.

5 -- This task includes: building a reproducible stress environment, generating/replaying flow traffic, measuring memory with evidence, optimizing the code, exposing Netdata charts for plugin memory accounting/breakdown, and re-measuring the improvement.

6 -- Follow-up scope added after install: investigate the remaining 4 failing `cargo test -p netflow-plugin` tests, find their real root cause, and fix them if they are regressions or latent bugs in this worktree.

7 -- Follow-up scope added after live validation on `2026-04-11`: the running plugin now shows about `700 MB` RSS in the local environment, and the current Netdata memory charts leave about `98%` of that RSS in `unaccounted`. This phase is to identify where that live heap goes, add subsystem-level attribution for the dominant owners, and produce a concrete runtime breakdown instead of an opaque `unaccounted` bucket.

8 -- Follow-up scope added after commit `d6e4ff75c0` on `2026-04-11`: rerun the full relevant test suites on the committed memory/rebuild changes, fix any regressions immediately, and then continue with the unresolved live anonymous `mmap_in_use` owner until it is concretely attributed and reduced.

9 -- Follow-up scope added after the retention-policy install on `2026-04-12`: review every PR change that touches shared components outside `netflow-plugin`, verify whether each one is truly necessary for the netflow work, identify any shared-layer risk or overreach, and then define a credible performance-testing plan for NetFlow/sFlow/IPFIX ingestion.

10 -

11 -## Analysis

12 -

13 -### Verified code paths

14 -

15 -- The plugin process instantiates all major in-memory subsystems at startup in [`src/crates/netdata-netflow/netflow-plugin/src/main.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/main.rs):

16 - - `IngestMetrics`

17 - - `OpenTierState`

18 - - `TierFlowIndexStore`

19 - - `FacetRuntime`

20 - - `FlowQueryService`

21 -- The plugin now exposes internal memory charts in [`src/crates/netdata-netflow/netflow-plugin/src/charts/metrics.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/charts/metrics.rs), [`src/crates/netdata-netflow/netflow-plugin/src/charts/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/charts/runtime.rs), and [`src/crates/netdata-netflow/netflow-plugin/src/charts/snapshot.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/charts/snapshot.rs):

22 - - `netflow.memory_resident_bytes`

23 - - `netflow.memory_accounted_bytes`

24 - - `netflow.memory_tier_index_bytes`

25 -- Verified charting gap on `2026-04-11`:

26 - - the runtime sampler in [`src/crates/netdata-netflow/netflow-plugin/src/charts/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/charts/runtime.rs) only reads `/proc/self/status` and `/proc/self/smaps_rollup`

27 - - this means it can capture totals like `rss`, `rss_anon`, `rss_file`, and `anon_huge_pages`, but it cannot classify which mappings own those bytes

28 - - as a result, the live `memory_accounted_bytes` chart still reports about `97.6%` of RSS as `unaccounted`, even though `/proc/<pid>/maps` and `/proc/<pid>/smaps` clearly show bounded raw/1m journal mmaps, heap, and large anonymous mappings as separate owners

29 -- Verified live state after installing the mmap-backed GeoIP build on `2026-04-11` late evening:

30 - - the currently running plugin (`PID 1874329`) is now:

31 - - `VmRSS = 86,204 kB`

32 - - `RssAnon = 27,672 kB`

33 - - `RssFile = 58,532 kB`

34 - - `Threads = 8`

35 - - `AnonHugePages = 0 kB`

36 - - `/proc/1874329/maps` now shows the GeoIP databases as shared read-only file mappings instead of copied anonymous memory:

37 - - `/usr/share/netdata/topology-ip-intel/topology-ip-geo.mmdb`

38 - - `/var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb`

39 - - `/proc/1874329/smaps` shows current resident usage for those mappings at:

40 - - `topology-ip-geo.mmdb`: `29,116 kB RSS`

41 - - `topology-ip-asn.mmdb`: `5,960 kB RSS`

42 - - direct implication:

43 - - the old large anonymous `~138.5 MB` owner is gone in the live process

44 - - the GeoIP optimization is now proven live, but as a file-backed/shared mapping reduction, not as an anonymous-memory reduction inside the old binary

45 - - current top resident classes from live `pmap` / `smaps` are now:

46 - - main anonymous heap/arena mapping: about `26,024 kB RSS`

47 - - GeoIP geo MMDB: about `29,116 kB RSS`

48 - - GeoIP ASN MMDB: about `5,960 kB RSS`

49 - - the rest split across smaller journal and file-backed mappings

50 -- Verified runtime/chart startup attribution on `2026-04-11`:

51 - - the startup profiling harness now exercises the real plugin-runtime path on in-memory streams:

52 - - constructs `PluginRuntime`

53 - - registers the `flows` handler

54 - - registers charts

55 - - starts runtime/chart emission long enough to initialize and emit

56 - - release measurement on the live dataset snapshot:

57 - - `plugin_runtime_configured delta_rss = 37,027,840`

58 - - `plugin_runtime_started delta_rss = 37,359,616`

59 - - incremental runtime+chart cost over the already settled rebuild state: about `331,776 bytes`

60 - - implication:

61 - - the plugin framework path is not the missing large memory owner

62 - - the remaining steady-state memory is explained by the already loaded runtime state plus file-backed MMDB/journal residency, not by a hidden `PluginRuntime` explosion

63 -- Verified dominant live anonymous-memory owner on `2026-04-11`:

64 - - the running plugin shows one large anonymous mapping at about `138.5 MB` RSS, and the allocator chart reports `mmap_in_use ≈ 143.2 MB`

65 - - the stock runtime config auto-detects GeoIP MMDB files even when paths are omitted in YAML:

66 - - [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs)

67 - - [`/usr/lib/netdata/conf.d/netflow.yaml`](/usr/lib/netdata/conf.d/netflow.yaml)

68 - - the plugin currently loads GeoIP databases with `Reader<Vec<u8>>`, which copies the full files into anonymous RAM:

69 - - [`src/crates/netdata-netflow/netflow-plugin/src/enrichment/data/geoip/files.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/enrichment/data/geoip/files.rs)

70 - - [`src/crates/netdata-netflow/netflow-plugin/src/enrichment/data/geoip/types.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/enrichment/data/geoip/types.rs)

71 - - the detected database sizes on this workstation are:

72 - - `/var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb = 10,402,916 bytes`

73 - - `/usr/share/netdata/topology-ip-intel/topology-ip-geo.mmdb = 131,453,438 bytes`

74 - - combined = `141,856,354 bytes`

75 - - this nearly matches the live anonymous `mmap_in_use` / `anon_other` footprint, making GeoIP DB loading the strongest verified remaining owner

76 - - the local `maxminddb 0.25.0` crate already supports `Reader::open_mmap()` behind its `mmap` feature, so this is optimizable without changing user-facing enrichment behavior

77 - - superseded by the later live result above:

78 - - after the mmap-backed build was actually installed and running, the large anonymous owner disappeared and the MMDBs moved to file-backed residency as expected

79 -- Ingestion writes every decoded flow to the raw journal and also updates:

80 - - facet state on each raw write in [`src/crates/netdata-netflow/netflow-plugin/src/ingest/service/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/service/runtime.rs)

81 - - materialized tier accumulators and tier flow indexes in [`src/crates/netdata-netflow/netflow-plugin/src/ingest/service/tiers.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/service/tiers.rs)

82 -- `FacetRuntime` stores global facet vocabularies and active-file contributions in memory in [`src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs).

83 -- Facet storage includes high-cardinality fields such as `SRC_ADDR`, `DST_ADDR`, `EXPORTER_IP`, ports, ASNs, and text labels, via [`src/crates/netdata-netflow/netflow-plugin/src/facet_catalog.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_catalog.rs).

84 -- The facet store keeps unique values in memory using:

85 - - `TextValueStore`

86 - - `DenseBitSet`

87 - - `RoaringTreemap`

88 - - `IpValueStore`

89 - in [`src/crates/netdata-netflow/netflow-plugin/src/facet_runtime/store.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_runtime/store.rs).

90 -- Active facet contributions are stored per active journal file as `BTreeMap<String, BTreeSet<String>>` in [`src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs).

91 -- Materialized tier indexes keep one `FlowIndex` per active hour bucket in [`src/crates/netdata-netflow/netflow-plugin/src/tiering/index/store.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/tiering/index/store.rs).

92 -- Open tier rows snapshot all currently open aggregate rows into vectors in [`src/crates/netdata-netflow/netflow-plugin/src/tiering/model.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/tiering/model.rs) and [`src/crates/netdata-netflow/netflow-plugin/src/ingest/service/tiers.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/service/tiers.rs).

93 -

94 -### Verified existing test assets

95 -

96 -- There are existing NetFlow/IPFIX/sFlow PCAP fixtures under [`src/crates/netdata-netflow/netflow-plugin/testdata/flows`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/testdata/flows).

97 -- There is already end-to-end UDP replay test scaffolding in [`src/crates/netdata-netflow/netflow-plugin/src/main_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/main_tests.rs), including:

98 - - UDP listener reservation

99 - - PCAP payload extraction

100 - - fixture replay over UDP

101 -

102 -### Measured hotspot evidence

103 -

104 -- A focused synthetic facet stress test now exists in [`src/crates/netdata-netflow/netflow-plugin/src/memory_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/memory_tests.rs) and was executed with:

105 - - `cargo test -p netflow-plugin stress_profile_high_cardinality_facet_runtime_memory -- --ignored --nocapture`

106 - - Baseline result before optimization: RSS grew from `9,715,712` to `30,953,472` bytes.

107 - - Baseline delta: `21,237,760` bytes for `50,000` synthetic high-cardinality flows.

108 - - Current result after the facet/runtime storage refactors: RSS grew from `10,420,224` to `18,903,040` bytes.

109 - - Current delta: `8,482,816` bytes for the same `50,000` synthetic high-cardinality flows.

110 - - Verified focused improvement for this hotspot: about `2.50x` lower RSS delta (`21.24 MB -> 8.48 MB`), still far from the requested overall `30x` target.

111 -- The same file contains a focused tier index stress test, executed with:

112 - - `cargo test -p netflow-plugin stress_profile_high_cardinality_tier_index_memory -- --ignored --nocapture`

113 - - Previous measured hotspot before the sparse row/index work: RSS grew from `6,823,936` to `29,032,448` bytes.

114 - - Previous delta: `22,208,512` bytes for `50,000` synthetic high-cardinality flows.

115 - - Current result after sparse/default-aware rollup rows and IPv4-specific IP storage: RSS grew from `6,967,296` to `16,265,216` bytes.

116 - - Current delta: `9,297,920` bytes for `50,000` synthetic high-cardinality flows.

117 - - Internal accounted heap for this subsystem is now `7,481,034` bytes with this measured breakdown:

118 - - row storage: `5,505,308`

119 - - field stores: `1,685,818`

120 - - flow lookup: `286,720`

121 - - schema: `2,668`

122 - - hour-key index metadata: `8`

123 - - scratch field ids: `512`

124 - - Verified focused improvement for this hotspot: about `2.39x` lower RSS delta (`22.21 MB -> 9.30 MB`).

125 -- The end-to-end synthetic ingest stress harness in the same file was executed with:

126 - - `cargo test -p netflow-plugin stress_profile_high_cardinality_netflow_memory -- --ignored --nocapture`

127 - - Current result: RSS peaked at `19,611,648` bytes from a `11,640,832` byte baseline.

128 - - Current peak delta: `7,970,816` bytes on the `5,000` flow synthetic ingest harness.

129 -- A focused decoder source-port churn stress test now exists in [`src/crates/netdata-netflow/netflow-plugin/src/memory_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/memory_tests.rs) and was executed before and after normalizing decoder parser scope away from raw UDP source ports:

130 - - `cargo test -p netflow-plugin stress_profile_decoder_source_port_churn_memory -- --ignored --nocapture`

131 - - Before normalization:

132 - - RSS grew from `7,118,848` to `875,028,480` bytes.

133 - - Delta: `867,909,632` bytes.

134 - - Parser scopes: `v9=20,000`.

135 - - After normalization:

136 - - RSS grew from `7,401,472` to `9,420,800` bytes.

137 - - Delta: `2,019,328` bytes.

138 - - Parser scopes: `v9=1`.

139 - - Verified focused improvement for this hotspot: about `430x` lower RSS delta on the same workload, caused by collapsing parser/template scope for one exporter IP across source-port churn.

140 -- The top facet cardinalities observed from the facet stress test were:

141 - - `DST_ADDR`: `50,000`

142 - - `SRC_ADDR`: `50,000`

143 - - `SRC_PORT`: `50,000`

144 - - `EXPORTER_IP`: `49,805`

145 - - `DST_PORT`: `13,107`

146 - - `DST_AS`: `4,096`

147 - - `DST_AS_NAME`: `4,096`

148 - - `IN_IF`: `4,096`

149 - - `OUT_IF`: `4,096`

150 - - `SRC_AS`: `4,096`

151 - - `SRC_AS_NAME`: `4,096`

152 - - `EXPORTER_NAME`: `256`

153 -

154 -### Live allocator and runtime evidence on 2026-04-11

155 -

156 -- The installed plugin now exports allocator-specific charts in addition to the existing resident/accounted charts:

157 - - `netflow.memory_allocator_bytes`

158 - - `netflow.memory_accounted_bytes`

159 - - `netflow.memory_resident_bytes`

160 -- Before the latest allocator-trim hook, the running plugin reported:

161 - - `rss = 636,342,300`

162 - - `unaccounted = 623,447,500`

163 - - allocator view:

164 - - `heap_arena = 488,673,300`

165 - - `heap_free = 421,649,500`

166 - - `heap_in_use = 67,023,820`

167 - - `mmap_in_use = 165,027,950`

168 - - `releasable = 15,486,688`

169 -- After adding a post-facet-reconcile `malloc_trim(0)` hook in [`src/crates/netdata-netflow/netflow-plugin/src/query/service.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/query/service.rs), the running plugin dropped to about half the live RSS:

170 - - `rss = 309,284,900`

171 - - `unaccounted = 296,349,500`

172 - - allocator view:

173 - - `heap_arena = 478,384,100`

174 - - `heap_free = 411,117,500`

175 - - `heap_in_use = 67,266,620`

176 - - `mmap_in_use = 165,027,950`

177 - - `releasable = 26,359,620`

178 -- Kernel/process evidence for the running post-trim process (`PID 1267128`) shows the remaining memory is still dominated by anonymous mappings:

179 - - `/proc/1267128/smaps_rollup`:

180 - - `Rss: 303,888 kB`

181 - - `Private_Dirty: 288,944 kB`

182 - - `AnonHugePages: 188,416 kB`

183 - - top resident mappings:

184 - - unlabeled anonymous mapping: `128,376 kB RSS`, `124,928 kB AnonHugePages`

185 - - `[heap]`: `85,460 kB RSS`, `26,624 kB AnonHugePages`

186 - - anonymous mapping at `7f0120000000-7f01221e5000`: `33,972 kB RSS`

187 - - anonymous mapping at `7f00f8000000-7f00fc000000`: `20,604 kB RSS`

188 - - anonymous mapping at `7f011c000000-7f0120000000`: `14,096 kB RSS`

189 -- These large anonymous mappings are aligned and sized like glibc secondary arena mappings, not like flow journals.

190 -- The running process has far more Tokio threads than the plugin workload appears to justify:

191 - - `/proc/1267128/status`: `Threads: 69`

192 - - CPU allowance for the process: `Cpus_allowed_list: 0-23`

193 - - `comm` entries show almost all threads are `tokio-runtime-w`

194 - - this is strong evidence that allocator arenas and thread-pool sizing are a major remaining memory driver

195 -- A release-profile startup harness was run against the live journal directory with:

196 - - `env MALLOC_ARENA_MAX=1 cargo test --release -p netflow-plugin stress_profile_live_startup_memory -- --ignored --nocapture`

197 - - `env MALLOC_ARENA_MAX=4 cargo test --release -p netflow-plugin stress_profile_live_startup_memory -- --ignored --nocapture`

198 -- Both release harness runs were effectively identical because the harness uses `#[tokio::test(flavor = "current_thread")]` and does not reproduce the live multithreaded runtime:

199 - - `MALLOC_ARENA_MAX=1`: `49,917,952` byte final RSS delta

200 - - `MALLOC_ARENA_MAX=4`: `49,885,184` byte final RSS delta

201 -- Conclusion from this evidence:

202 - - the on-disk journal/facet restore path explains about `50 MB`

203 - - the remaining `~250 MB` live gap is now most likely runtime-thread/allocator behavior, not just retained flow/journal structures

204 -

205 -### Rebuild-path root cause verified on 2026-04-11

206 -

207 -- The earlier startup harness was incomplete because it stopped at `IngestService::new_with_facet_runtime()` and never ran the production startup rebuild path:

208 - - [`src/crates/netdata-netflow/netflow-plugin/src/ingest/service/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/service/runtime.rs) calls `self.rebuild_materialized_from_raw().await?` before the UDP receive loop starts.

209 -- A new multithreaded release harness was added in [`src/crates/netdata-netflow/netflow-plugin/src/startup_memory_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/startup_memory_tests.rs) to reproduce the real startup path, thread pools, allocator counters, and rebuild step.

210 -- Verified pre-fix rebuild behavior with the multithread harness:

211 - - `env NETFLOW_PROFILE_WORKER_THREADS=24 NETFLOW_PROFILE_MAX_BLOCKING_THREADS=64 cargo test --release -p netflow-plugin stress_profile_live_startup_memory_multithreaded -- --ignored --nocapture`

212 - - rebuild phase before the latest fixes:

213 - - RSS: about `637 MB`

214 - - threads: `116`

215 - - `heap_in_use`: about `56 MB`

216 - - `heap_free`: about `603 MB`

217 - - This proved the catastrophic resident growth was primarily allocator retention created during rebuild, not live in-memory NetFlow state.

218 -- Verified that trimming only after facet reconcile was insufficient:

219 - - rebuild still settled around `272-286 MB` RSS even after the first rebuild trim

220 - - a second trim after a `12s` settle window recovered only about `0-5 MB`

221 -- The remaining long-lived owner was identified as the global Rayon pool used by journal indexing:

222 - - [`src/crates/journal-engine/src/indexing.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-engine/src/indexing.rs) used `into_par_iter()` on Rayon’s global pool inside `batch_compute_file_indexes()`

223 - - multithread harness after rebuild settled at `31` threads with `workers=4`, which matches `7` Tokio threads plus a persistent `24`-thread Rayon global pool

224 -- Fix implemented:

225 - - [`src/crates/netdata-netflow/netflow-plugin/src/ingest/rebuild.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/rebuild.rs)

226 - - trim glibc heap after raw rebuild completes

227 - - keep rebuild-only heavy objects scoped so they are eligible for release before the post-rebuild trim

228 - - [`src/crates/journal-engine/src/indexing.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-engine/src/indexing.rs)

229 - - replace Rayon global-pool indexing with a bounded local Rayon pool capped at `4` threads

230 -- Verified post-fix rebuild behavior:

231 - - `workers=4`, `max_blocking=8`

232 - - rebuild RSS: about `137 MB`

233 - - rebuild settle RSS: about `137 MB`

234 - - threads after settle: `7`

235 - - `workers=24`, `max_blocking=64`

236 - - rebuild RSS: about `152 MB`

237 - - rebuild settle RSS: about `151 MB`

238 - - threads after settle: `27`

239 -- Net effect of the verified rebuild-path fixes on the same local journal directory:

240 - - from about `637 MB` rebuild RSS down to about `152 MB` with the runtime shape closest to the real service on this host

241 - - about `4.2x` lower resident memory for the verified catastrophic startup path

242 - - from `116` threads down to `27` after rebuild settles

243 -

244 -### Verified structural causes

245 -

246 -- `FacetRuntime` used to keep the same logical vocabulary in multiple resident forms in [`src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs):

247 - - before the refactor it kept `archived_fields`, `fields`, and `active_contributions`

248 - - after the refactor it keeps `archived_fields`, `active_fields`, `active_contributions`, and a lightweight published snapshot

249 -- `active_contributions` no longer stores per-file values as `BTreeSet<String>`. It now uses typed `FacetStore` containers, so IPs, ports, and ASNs stay in compact representations instead of always becoming heap strings.

250 -- The full duplicate combined facet store (`fields`) has been removed from runtime memory. The plugin now keeps archived stores, active stores, per-file active contributions, and lightweight published metadata instead.

251 -- The tier index hot path in [`src/crates/netdata-netflow/netflow-plugin/src/tiering/index/store.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/tiering/index/store.rs) uses the custom `FlowIndex` in [`src/crates/netdata-netflow/flow-index/src/lib.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/flow-index/src/lib.rs), which interns field values but still stores a full `u32` field-id tuple per unique rollup flow.

252 -- The tier index now uses sparse/default-aware rollup rows instead of a fully dense `u32` tuple per stored flow, and it uses a smaller IPv4-specific representation for IP field dictionaries in the `FlowIndex` crate.

253 -- The facet IP store now uses bitmap-backed IPv4 sets and explicit IPv6 storage instead of a generic 17-byte packed representation for all IP addresses.

254 -- The local frontend gap notes already document that high-cardinality facets should not be fully enumerated inline and instead need text-input or autocomplete behavior in [`/home/costa/src/dashboard/cloud-frontend/TODO-flows-gaps.md`](/home/costa/src/dashboard/cloud-frontend/TODO-flows-gaps.md).

255 -- The upstream `netflow_parser::scoped_parser::AutoScopedParser` keys parser/template caches by full `SocketAddr`, not exporter identity:

256 - - `ipfix_parsers: HashMap<IpfixSourceKey, NetflowParser>`

257 - - `v9_parsers: HashMap<V9SourceKey, NetflowParser>`

258 - - `legacy_parsers: HashMap<SocketAddr, NetflowParser>`

259 - in [`/home/costa/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/netflow_parser-0.9.0/src/scoped_parser.rs`](/home/costa/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/netflow_parser-0.9.0/src/scoped_parser.rs).

260 -- The plugin also tracks hydrated decoder namespaces per full `SocketAddr` in [`src/crates/netdata-netflow/netflow-plugin/src/decoder/state/runtime/init.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/decoder/state/runtime/init.rs) and [`src/crates/netdata-netflow/netflow-plugin/src/ingest/persistence.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/persistence.rs).

261 -

262 -### Shared-component review on 2026-04-12

263 -

264 -- Reviewed shared crates changed by the PR against `origin/master`:

265 - - `journal-core`

266 - - `journal-engine`

267 - - `journal-log-writer`

268 - - `netdata-plugin/rt`

269 - - `journal-common`

270 - - `jf/journal_file`

271 -- Shared changes that are justified by the netflow work:

272 - - [`src/crates/journal-core/src/file/mmap.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/mmap.rs)

273 - - required to stop append-heavy writer windows from growing from file start toward file tail, which matched the live journal RSS explosion

274 - - [`src/crates/journal-engine/src/indexing.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-engine/src/indexing.rs)

275 - - `without_disk_cache()` is required by [`src/crates/netdata-netflow/netflow-plugin/src/ingest/rebuild.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/rebuild.rs) to avoid rebuild-time cache files and resident disk-cache overhead

276 - - the bounded local Rayon pool is required to eliminate the persistent global indexing pool that previously inflated rebuild threads and allocator arenas

277 - - [`src/crates/journal-engine/src/logs/query.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-engine/src/logs/query.rs)

278 - - the `merge_log_entries()` allocation fix is required because netflow rebuild executes an unlimited `LogQuery::execute()` path in [`src/crates/netdata-netflow/netflow-plugin/src/ingest/rebuild.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/rebuild.rs)

279 - - [`src/crates/journal-core/src/file/reader.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/reader.rs)

280 - - `build_filter()` is directly consumed by netflow direct journal scans in [`src/crates/netdata-netflow/netflow-plugin/src/query/scan/direct.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/query/scan/direct.rs)

281 - - [`src/crates/journal-log-writer/src/log/mod.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/mod.rs), [`src/crates/journal-log-writer/src/log/chain.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/chain.rs), and [`src/crates/journal-common/src/time.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-common/src/time.rs)

282 - - required for event-time journal writes (`EntryTimestamps` with realtime override), active-file visibility, and lifecycle notifications used by the facet runtime

283 - - [`src/crates/netdata-plugin/rt/src/netdata_env.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-plugin/rt/src/netdata_env.rs)

284 - - `stock_data_dir` is required by netflow GeoIP auto-detection in [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs)

285 -- Shared changes that look like scope expansion rather than netflow necessity:

286 - - [`src/crates/netdata-plugin/rt/src/lib.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-plugin/rt/src/lib.rs)

287 - - the `flows:*` GET-args-to-JSON shim is netflow-specific behavior inside the shared runtime

288 - - `ProgressState::snapshot()`, cancellation `499` behavior, and their new tests are not required by the production netflow path; current references are only test-side

289 - - [`src/crates/journal-engine/src/logs/query.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-engine/src/logs/query.rs)

290 - - `with_output_fields()` is not used by netflow production code; repository search found only self-tests as callers

291 - - [`src/crates/journal-core/src/file/file.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/file.rs) and [`src/crates/jf/journal_file/src/file.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/jf/journal_file/src/file.rs)

292 - - the zero-offset iterator hardening is a valid defensive fix, but it is not tied to a reproduced netflow requirement

293 - - the `jf/journal_file` copy is especially outside the netflow scope

294 -- Netflow-local extraction that is acceptable and should stay:

295 - - [`src/crates/netdata-netflow/flow-index/src/lib.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/flow-index/src/lib.rs)

296 - - this is a netflow-owned crate, not a general shared runtime/library change

297 - - it cleanly isolates the compact rollup/grouping index logic and memory accounting used by the plugin

298 -- This creates a verified memory-risk pattern: if the same exporter IP changes UDP source port over time, parser/template caches can grow by source-port churn even when exporter identity is operationally unchanged.

299 -- The plugin entrypoint in [`src/crates/netdata-netflow/netflow-plugin/src/main.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/main.rs) uses `#[tokio::main]`, which means:

300 - - Tokio chooses the multithread runtime defaults automatically

301 - - worker thread count is not explicitly bounded for this plugin

302 - - blocking-thread pool limits are not explicitly bounded for this plugin

303 -- This matters because the live process currently shows `69` threads, mostly `tokio-runtime-w`, on a machine where `nproc` and `getconf _NPROCESSORS_ONLN` both report `24`.

304 -- The current startup harness therefore underestimates live memory by construction, because it uses `current_thread` runtime flavor and never reproduces the production thread-pool shape.

305 -

306 -### Current accounted facet breakdown

307 -

308 -- The focused facet profile now exposes an internal accounted breakdown from `FacetRuntime::estimated_memory_breakdown()`:

309 - - archived facet stores: `972,497` bytes

310 - - active facet stores: `572,521` bytes

311 - - active per-file contributions: `505,107` bytes

312 - - published snapshot metadata: `3,005` bytes

313 - - archived path tracking: `152` bytes

314 -- This shows the duplicate combined store is gone and high-cardinality IP storage is materially smaller, but the total measured reduction is still nowhere near the requested `30x`.

315 -- Re-validated on `2026-04-11` after the latest live-memory fixes:

316 - - `cargo test -p netflow-plugin stress_profile_high_cardinality_facet_runtime_memory -- --ignored --nocapture`

317 - - Measured RSS growth: `8,966,144` bytes

318 - - Current facet-accounted total: `2,053,282` bytes

319 - - Verified undercount factor on this focused harness: about `4.37x`

320 - - This matters because the live process currently reports only about `12 MB` of facet memory in charts, while the startup harness and this focused profile strongly suggest the real facet footprint is materially larger.

321 -- Re-validated on `2026-04-11` with the live-startup multithread harness after disabling rebuild disk cache:

322 - - `cargo test -p netflow-plugin --release stress_profile_live_startup_memory_multithreaded -- --ignored --nocapture`

323 - - `facet_runtime_new` raised RSS by `48,795,648` bytes from baseline

324 - - `query_service_new` added only `856,064` bytes more

325 - - `ingest_service_new` added only `684,032` bytes more

326 - - `rebuild_materialized_from_raw` still settles around `151.8 MB` RSS

327 - - This confirms the remaining live `~315 MB` process is not just startup rebuild; a large steady-state owner remains, and facet memory accounting is one verified blind spot.

328 -- Re-validated on `2026-04-11` after removing the duplicate active store and making `DenseBitSet` lazy:

329 - - `cargo test -p netflow-plugin stress_profile_high_cardinality_facet_runtime_memory -- --ignored --nocapture`

330 - - focused facet RSS delta improved from `8,966,144` bytes to `8,302,592` bytes

331 - - a later compact persisted-text experiment improved this focused harness only marginally again (`8,302,592 -> 8,228,864` bytes), which is too small to justify a schema change by itself

332 - - `NETFLOW_PROFILE_SETTLE_SECS=20 cargo test -p netflow-plugin --release stress_profile_live_startup_memory_multithreaded -- --ignored --nocapture`

333 - - startup `rebuild_settle_trim` improved materially with the validated active-store removal and lazy bitset work: about `145.36 MB -> 124.69 MB`

334 -- Re-validated on `2026-04-11` after removing internal timestamp facets from the user-facing catalog and request allowlists:

335 - - `env MALLOC_ARENA_MAX=1 NETFLOW_PROFILE_SETTLE_SECS=5 NETFLOW_PROFILE_DISABLE_THP=1 cargo test -p netflow-plugin --release stress_profile_live_startup_memory_multithreaded -- --ignored --nocapture`

336 - - measured startup phases on the retained live dataset:

337 - - `facet_runtime_new delta_rss = 10,301,440`

338 - - `query_service_new delta_rss = 11,055,104`

339 - - `ingest_service_new delta_rss = 11,673,600`

340 - - `rebuild_materialized_from_raw delta_rss = 15,089,664`

341 - - `rebuild_settle_trim delta_rss = 15,052,800`

342 - - measured rebuild-time facet breakdown:

343 - - `archived = 3,598,723`

344 - - `published = 13,812`

345 - - `archived_paths = 55,050`

346 - - implication:

347 - - the archived timestamp-field removal is visible not only in deep allocative accounting but also in the startup harness; archived facet state is now a much smaller contributor during rebuild/load

348 -- Verified profiling confounder on `2026-04-11`:

349 - - the installed live plugin at [`/usr/libexec/netdata/plugins.d/netflow-plugin`](/usr/libexec/netdata/plugins.d/netflow-plugin) is still older code and continues writing `/var/cache/netdata/flows/facet-state.bin`

350 - - direct decode of the live `facet-state.bin` under the experimental compact persisted-text schema failed with `UnexpectedEof`

351 - - raw bytes inspection showed the file header still begins with `04 00 00 00`, consistent with the older on-disk schema, not the experimental bumped version

352 - - conclusion: startup profiling against `/var/cache/netdata/flows` is contaminated for persisted-state comparisons until the running service is updated or profiling is moved to an isolated snapshot

353 - - action taken: reverted the unproven compact persisted-text schema change and kept only the validated in-memory facet optimizations

354 -

355 -## Feasibility Assessment

356 -

357 -### Assumptions checked

358 -

359 -1. Assumption: the plugin can be run in a local isolated environment without touching production services.

360 - - Verified: `PluginConfig::new()` supports Netdata-style config discovery from environment variables in [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs) and [`src/crates/netdata-plugin/rt/src/netdata_env.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-plugin/rt/src/netdata_env.rs).

361 -2. Assumption: we can generate realistic ingest load without office routers.

362 - - Verified: existing PCAP fixtures and replay helpers already provide a base path for UDP ingestion tests in [`src/crates/netdata-netflow/netflow-plugin/src/main_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/main_tests.rs).

363 -3. Assumption: we can profile memory locally.

364 - - Verified: `/usr/bin/time`, `perf`, and `valgrind` are available on this machine.

365 -4. Assumption: the current worktree is clean enough to make isolated changes.

366 - - Verified false: `git status --short` now shows tracked changes in the netflow plugin crate plus untracked local helper files (`TODO-netflow-memory-footprint.md`, `build-install-netflow-plugin.sh`).

367 - - This is not a blocker because the modified files are scoped to the current task and the untracked files are intentionally left out of commits.

368 -

369 -### Feasibility verdict

370 -

371 -FEASIBLE AS SPECIFIED

372 -

373 -The codebase already contains the pieces needed to build a reproducible stress/profiling harness and to validate memory reductions locally before reporting results.

374 -

375 -## Decisions

376 -

377 -- Decision made by Costa on `2026-04-12`: clean up unnecessary shared-component changes before proceeding with flow-ingestion performance work.

378 - - Scope to clean first:

379 - - netflow-specific `flows:*` request parsing currently living in shared `rt`

380 - - shared-library additions that are not required by the netflow production path

381 - - duplicated/shared defensive fixes that do not benefit the netflow path and only expand review surface

382 - - Purpose:

383 - - keep the PR reviewable

384 - - keep shared crates generic

385 - - ensure the later performance-testing work is built on the code we actually intend to upstream

386 - - Implemented on `2026-04-12`:

387 - - moved the `flows:*` GET-argument compatibility shim out of shared [`src/crates/netdata-plugin/rt/src/lib.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-plugin/rt/src/lib.rs) and into netflow-local request parsing in [`src/crates/netdata-netflow/netflow-plugin/src/api/flows/handler.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/api/flows/handler.rs)

388 - - removed unused shared `journal-engine` API surface `LogQuery::with_output_fields()` from [`src/crates/journal-engine/src/logs/query.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-engine/src/logs/query.rs)

389 - - removed the duplicated zero-offset iterator hardening and test from the out-of-path [`src/crates/jf/journal_file/src/file.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/jf/journal_file/src/file.rs)

390 - - Verification on `2026-04-12`:

391 - - `cargo test -p rt --manifest-path src/crates/Cargo.toml` passed

392 - - `cargo test -p journal-engine --manifest-path src/crates/Cargo.toml` passed

393 - - `cargo test -p journal-core --manifest-path src/crates/Cargo.toml` passed

394 - - `cargo test -p journal-log-writer --manifest-path src/crates/Cargo.toml` passed

395 - - `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml` passed with `388 passed, 0 failed, 14 ignored`

396 - - standalone duplicate crate check:

397 - - `cargo test --manifest-path src/crates/jf/journal_file/Cargo.toml writer::tests::test_write_and_read_journal_entries -- --nocapture`

398 - - failed both on the cleanup worktree and on baseline checkpoint commit `973f200dd5`

399 - - panic is in [`src/crates/jf/journal_file/src/writer.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/jf/journal_file/src/writer.rs) at line `637`, not in the reverted iterator code

400 - - implication: this `jf` failure is pre-existing noise for the duplicated crate, not a regression introduced by the cleanup

401 -- Decision made autonomously on `2026-04-12` for the first ingestion-performance pass:

402 - - implement the benchmark as ignored `cargo test` harnesses inside `netflow-plugin`, not as Criterion benches yet

403 - - Evidence:

404 - - the repo already uses ignored profiling tests for startup, memory, and query profiling

405 - - the existing ingest throughput harness was already in [`src/crates/netdata-netflow/netflow-plugin/src/ingest_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest_tests.rs)

406 - - nearby flow-parser projects such as the mirrored NetGauze parser benchmark by protocol and packet shape, which this implementation now mirrors operationally without introducing new bench dependencies

407 - - Implication:

408 - - the first pass is easy to run in CI-like environments with `cargo test --release ... --ignored --nocapture`

409 - - if we later need statistical benchmarking or regression thresholds, we can still add Criterion on top of the same scenarios

410 -- Decision made by Costa on `2026-04-11`: proceed autonomously with diagnosis and optimization of the remaining live memory footprint until the dominant owners are concretely attributed and reduced.

411 -- Decision pending on `2026-04-11`: whether to change the shared journal window-manager logic in [`src/crates/journal-core/src/file/mmap.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/mmap.rs) as part of the netflow memory work.

412 - - Context:

413 - - this logic is shared infrastructure, not netflow-only code

414 - - live evidence currently shows one large `rw-s` mapping per active tier journal file, which matches the current remap behavior in the shared window manager

415 - - changing shared journal infrastructure could reduce resident memory for netflow and any other current/future users of the same writer path, but it must be treated as an infrastructure fix, not a plugin-local tweak

416 -- If implementation reveals that the only viable 30x reduction requires changing query/facet behavior visible to users, I must stop and present options with concrete impact before proceeding.

417 -- Decision made by Costa on `2026-04-11`: proceed autonomously with the recommended attribution path for the live `~700 MB` RSS case:

418 - - inspect the live allocator/process footprint first

419 - - expand the plugin's internal accounting for major in-memory subsystems

420 - - use stress-harness profiling to map large allocations back to code paths if the live allocator/process view is not sufficient

421 -- Decision made by Costa on `2026-04-11`: choose the implementation and tooling path autonomously using best practices. Do not escalate Rust/allocator implementation details back to Costa unless there is a real product-level tradeoff. Allocator replacement is allowed if evidence shows it materially improves accounting or production memory behavior, but it must not be used blindly to hide ownership bugs.

422 -- Operational decision made by Costa on `2026-04-11`: when Netdata service restarts are needed on this workstation, use `sudo` so the authentication flow is explicit and does not rely on a background privilege escalation path.

423 -- Decision made autonomously on `2026-04-11` after direct benchmark evidence: keep the plugin on glibc and do not switch to mimalloc, jemalloc, or tcmalloc for this workload.

424 - - Evidence from the release startup harness on the retained live dataset:

425 - - glibc with `MALLOC_ARENA_MAX=1`: `rebuild_settle_trim delta_rss = 45,494,272`

426 - - mimalloc (`LD_PRELOAD=/usr/lib/libmimalloc.so`): `99,500,032`

427 - - jemalloc (`LD_PRELOAD=/usr/lib/libjemalloc.so`): `303,493,120`

428 - - tcmalloc (`LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so`): `278,237,184`

429 - - Implication:

430 - - allocator replacement is not the right optimization path here; the right path is reducing transient allocations and capping glibc arena growth inside the process.

431 -- Decision made by Costa on `2026-04-11`: remove exact timestamp-microsecond fields such as `FLOW_START_USEC`, `FLOW_END_USEC`, and `OBSERVATION_TIME_MILLIS` from the facet/query catalog entirely.

432 - - Context:

433 - - the field is currently configured as `SparseU64`, `supports_autocomplete = true`, and `uses_sidecar = true` in [`src/crates/netdata-netflow/netflow-plugin/src/facet_catalog.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_catalog.rs)

434 - - archived facet autocomplete/search uses sidecars when a field is marked this way in [`src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs)

435 - - a targeted allocative diagnostic on the live dataset showed archived facet memory of `40,524,572` bytes total, of which `FLOW_END_USEC` alone consumed `32,263,968` bytes

436 - - Implication:

437 - - these fields will no longer be available as facet fields, autocomplete targets, or selection/filter fields for user requests

438 - - internal chart logic that reads `FLOW_END_USEC` directly from records remains unaffected

439 - - Implemented on `2026-04-11`:

440 - - removed from facet catalog and from query request allowlists in [`src/crates/netdata-netflow/netflow-plugin/src/facet_catalog.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_catalog.rs) and [`src/crates/netdata-netflow/netflow-plugin/src/query/request/constants.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/query/request/constants.rs)

441 - - added regressions for unsupported `group_by`, `selection`, and autocomplete requests in [`src/crates/netdata-netflow/netflow-plugin/src/query/tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/query/tests.rs)

442 - - verified with `cargo test -p netflow-plugin`: `378 passed, 0 failed, 14 ignored`

443 -- Decision made by Costa on `2026-04-12`: base per-tier journal rotation size on the per-tier total byte-retention budget, not on a fixed global file size.

444 - - Rule:

445 - - for per-tier `size_of_journal_files >= 100MB`, derive `size_of_journal_file = clamp(size_of_journal_files / 20, 5MB, 200MB)`

446 - - per-tier total byte-retention budgets below `100MB` are considered invalid for `netflow-plugin`

447 - - Intended operational effect:

448 - - once a tier is at steady-state retention, one full-file deletion should represent at most about `1/20` of the tier's byte history until the `200MB` cap is reached

449 - - above `4GB` per tier, the file-size cap keeps the rotation unit at `200MB`, so the fraction of history replaced on each steady-state rotation becomes even smaller than `1/20`

450 - - Critical implementation constraint verified in code:

451 - - the current plugin still enforces `number_of_journal_files` as an active retention limit in [`src/crates/netdata-netflow/netflow-plugin/src/ingest/service/init.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/service/init.rs) and [`src/crates/journal-log-writer/src/log/chain.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/chain.rs)

452 - - the current default of `64` files in [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/journal.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/journal.rs) would conflict with this rule for larger byte budgets unless that file-count limit is also derived, raised, or disabled by default

453 -- Decision made by Costa on `2026-04-12`: users must not configure `number_of_journal_files` directly.

454 - - Desired user-facing contract:

455 - - users configure per-tier retention by `size_of_journal_files` and/or `duration_of_journal_files`

456 - - the plugin derives any internal file-count limit needed to satisfy that contract

457 - - if the journal layer already supports size/time retention directly, the plugin should not keep an independent user-visible file-count knob

458 - - Verified library capability:

459 - - `journal-log-writer` retention already supports optional file-count, total-size, and age limits independently in [`src/crates/journal-log-writer/src/log/config.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/config.rs)

460 - - because those retention limits are optional, the plugin can satisfy size-only, time-only, or size+time retention without exposing `number_of_journal_files` to users

461 - - Implication:

462 - - the current plugin config model and validation in [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/journal.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/journal.rs) and [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/validation/journal.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/validation/journal.rs) need redesign before implementation, because they currently require `number_of_journal_files`, `size_of_journal_files`, and `duration_of_journal_files` all to be present and non-zero

463 -- Decision made by Costa on `2026-04-12`: for tiers that use time-only retention and do not define `size_of_journal_files`, start with a fixed internal rotation size of `100MB`.

464 - - Context:

465 - - rotation uses `size_of_journal_file` and/or `duration_of_journal_file`, and the active file rotates when any configured rotation limit is exceeded

466 - - retention deletion itself can already be time-only in [`src/crates/journal-log-writer/src/log/config.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/config.rs)

467 - - the current plugin default still hardcodes `size_of_journal_file = 256MB` and `duration_of_journal_file = 1h` in [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/journal.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/journal.rs)

468 - - because active journal residency is sensitive to file size, time-only retention still needs a bounded default rotation size even without a byte-retention budget

469 - - Additional feasibility constraint verified on `2026-04-12`:

470 - - true adaptive per-file sizing would require a small shared-writer change, because `journal-log-writer::Log` snapshots `size_of_journal_file` into `RotationState` at construction time in [`src/crates/journal-log-writer/src/log/mod.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/mod.rs)

471 - - there is currently no public API to update `rotation_policy.size_of_journal_file` or rebuild `RotationState` on an existing `Log`

472 - - Decision details:

473 - - use a fixed internal `100MB` rotation size for time-only tiers now

474 - - do not change `journal-log-writer` for adaptive rotation sizing in this pass

475 - - revisit adaptive sizing later only if real users ask for it

476 - - Implementation constraint found and fixed during validation on `2026-04-12`:

477 - - per-tier overrides need a tri-state model to distinguish:

478 - - omitted field => inherit global limit

479 - - explicit `null` => disable inherited limit

480 - - concrete value => override global limit

481 - - the first draft used plain `Option`, which could not represent explicit `null` for tier overrides correctly

482 - - the config model was updated to preserve this contract before claiming the refactor complete

483 -

484 -## Plan

485 -

486 -- Review the PR diff against `origin/master` and isolate every changed file outside `src/crates/netdata-netflow/netflow-plugin`.

487 -- For each shared-component change:

488 - - verify what behavior it changes in code

489 - - verify why it was introduced

490 - - decide whether it is necessary, optional, or should be reverted from the PR

491 - - record risks, especially shared-runtime or journal-layer blast radius

492 -- Present Costa with a review-style summary focused on findings first:

493 - - unnecessary shared changes

494 - - risky shared changes

495 - - shared changes that are justified and should remain

496 -- Cleanup implementation for the clearly unnecessary shared changes before performance work:

497 - - keep shared runtime generic by moving netflow request translation into the netflow handler

498 - - remove unused shared `journal-engine` builder surface that has no netflow production caller

499 - - revert duplicated `jf`-crate changes that are outside the netflow path

500 - - rerun the affected shared and plugin test suites before any performance work begins

501 -- After the shared-component review, design a performance test strategy for flow ingestion:

502 - - synthetic benchmark coverage for NetFlow, sFlow, and IPFIX

503 - - real fixture replay coverage where available

504 - - throughput and memory measurement dimensions

505 - - cardinality and field-variability scenarios

506 -- Implemented on `2026-04-12`:

507 - - split ingest test support out of oversized [`src/crates/netdata-netflow/netflow-plugin/src/ingest_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest_tests.rs) into dedicated support and benchmark modules

508 - - added protocol-specific release benchmarks in [`src/crates/netdata-netflow/netflow-plugin/src/ingest_bench_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest_bench_tests.rs):

509 - - NetFlow v5

510 - - NetFlow v9

511 - - IPFIX

512 - - sFlow

513 - - added a second benchmark matrix for post-decode ingest sensitivity to field variability/cardinality:

514 - - low-cardinality

515 - - medium-cardinality (`256` buckets)

516 - - high-cardinality (`50,000` unique buckets)

517 - - extracted a single-record ingest helper from [`src/crates/netdata-netflow/netflow-plugin/src/ingest/service/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest/service/runtime.rs) so the benchmark exercises the same raw-write/facet/tier path as the production packet handler

518 -- Verified on `2026-04-12`:

519 - - `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml` passed with `388 passed, 0 failed, 15 ignored`

520 - - `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release bench_ingestion_protocol_matrix -- --ignored --nocapture` passed

521 - - `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release bench_ingestion_cardinality_matrix -- --ignored --nocapture` passed

522 -- First measured throughput on this workstation with the current release harness:

523 - - protocol matrix:

524 - - NetFlow v5 full ingest: about `49.6k flows/s`

525 - - NetFlow v9 full ingest: about `47.6k flows/s`

526 - - IPFIX full ingest: about `62.7k flows/s`

527 - - sFlow full ingest: about `45.7k flows/s`

528 - - decode-only throughput is much higher (`0.75M - 5.2M flows/s` depending on protocol), so the dominant steady-state cost is post-decode ingest, not wire parsing

529 - - post-decode cardinality matrix:

530 - - low-cardinality: about `53.4k flows/s`

531 - - medium-cardinality (`256` buckets): about `47.6k flows/s`

532 - - high-cardinality (`50,000` unique buckets): about `29.3k flows/s`

533 - - operational implication:

534 - - with the current single-threaded ingest path on this host, a reasonable expectation is roughly `45k - 63k flows/s` on warmed real fixtures at low-to-medium cardinality

535 - - high-cardinality post-decode ingest drops to about `29k flows/s`, which is the more realistic planning number when users have highly variable endpoint/exporter/interface dimensions

536 -- Re-run the full affected test suites after each memory change:

537 - - `cargo test -p journal-core -p journal-log-writer -p journal-engine -p netflow-plugin`

538 -- Improve facet runtime accounting first, because it is now a proven blind spot:

539 - - replace rough `HashTable` byte heuristics with real allocation reporting where the container exposes it

540 - - use a better size estimate for `RoaringTreemap`

541 - - include missing container-structure overhead for the facet `BTreeMap`/published snapshot bookkeeping

542 -- Re-run the focused facet harness after the estimator changes and compare:

543 - - actual RSS growth

544 - - facet-accounted bytes

545 - - remaining gap

546 -- Use the improved live charts to decide the next optimization target instead of guessing.

547 -- New verified optimization target from Massif on `2026-04-11`:

548 - - `TextValueStore::insert()` appears twice from `apply_active_contribution()`, once for the per-file contribution store and once for the global `active_fields` union.

549 - - This means new active text values are stored twice in long-lived runtime memory.

550 - - `DenseBitSet::new()` also shows measurable startup allocation through `empty_field_stores()`, even before those dense stores hold any values.

551 -- Next implementation step:

552 - - remove the persistent `active_fields` duplicate store and derive active unions from `active_contributions` when needed

553 - - make dense facet bitsets allocate lazily on first insert instead of at empty-store construction

554 - - then rerun the full `netflow-plugin` suite and the focused facet-memory harness

555 -- Additional verified next target:

556 - - archived facet startup load still costs about `48.3 MB` RSS even after `active=0` and `active_contrib=0`

557 - - the current persisted text facet format is `Vec<String>`, which means startup deserializes many heap strings and then rebuilds compact text stores from them

558 - - the next optimization pass should persist/load text stores in compact arena form so startup can rebuild the lookup index without first materializing `Vec<String>`

559 -- New verified next target on `2026-04-11`:

560 - - the remaining live resident gap is dominated by allocator-retained free heap, not by GeoIP/MMDB data or by a better general-purpose allocator existing on this host

561 - - the next optimization pass should therefore focus on reducing transient startup allocations further and on improving charts so allocator-retained heap is shown explicitly instead of being hidden inside `unaccounted`

562 -- New verified next target on `2026-04-11` after allocative archived-facet profiling:

563 - - the current archived facet estimate is materially wrong for `SparseU64` timestamp-like fields

564 - - `FLOW_END_USEC` is the dominant verified archived-facet memory owner and should be treated as the next major optimization decision

565 -- New verified next target on `2026-04-11` after the latest live chart review:

566 - - add a true process-side resident mapping breakdown chart from `/proc/self/smaps`

567 - - classify resident bytes into disjoint buckets such as:

568 - - heap

569 - - anonymous non-heap mappings

570 - - raw journal mmaps

571 - - `1m` journal mmaps

572 - - `5m` journal mmaps

573 - - `1h` journal mmaps

574 - - other file-backed mappings

575 - - shmem

576 - - this will not replace internal heap ownership accounting, but it will stop forcing most of RSS into an opaque `unaccounted` bucket when the kernel already exposes the owning mapping classes

577 -- New verified next target on `2026-04-11` after correlating live mappings with enrichment assets:

578 - - switch GeoIP database loading from eager copied `Reader<Vec<u8>>` to mmap-backed readers

579 - - extend the resident mapping chart to break GeoIP MMDB residency out separately from generic `other_file`

580 - - this should remove about `142 MB` of anonymous copied database memory while preserving enrichment behavior

581 - - status update after install:

582 - - implemented and verified live

583 - - current remaining blind spot is no longer a huge anonymous GeoIP owner; it is the smaller steady-state heap / runtime path that the existing startup harness still does not exercise

584 -

585 -1. Build a local stress environment around the real `netflow-plugin` binary/crate using isolated cache/config dirs and existing UDP replay helpers as the starting point.

586 -2. Add a reproducible load generator that can sustain much higher cardinality and volume than the office feed, so memory growth can be observed quickly and deterministically.

587 -3. Capture a baseline:

588 - - RSS / max RSS

589 - - ingest counters

590 - - journal growth

591 - - memory snapshots over time

592 - - allocator evidence where possible

593 -4. Attribute memory to concrete subsystems by selectively measuring with feature/code-path instrumentation and targeted profiling.

594 -5. Add internal memory accounting for the major in-memory subsystems and expose it via Netdata charts.

595 -6. Implement the highest-impact reductions first:

596 - - first, replace per-file facet string sets with compact typed stores and stop reparsing/recloning compact values into the resident path

597 - - then remove the duplicate combined facet store from runtime memory while preserving published facet behavior on restart

598 - - then add byte accounting for resident facet/tier/open-row structures and expose them as charts

599 - - then switch tier rollup rows to sparse/default-aware storage and compress IPv4-heavy IP dictionaries

600 - - then normalize decoder template scope away from raw UDP source-port churn if the parser path proves to be retaining duplicate template state per exporter IP

601 - - then re-measure and decide whether more aggressive high-cardinality facet residency reduction is still required

602 -7. Add tests/benchmarks so the regression is pinned in CI or at least in local automated verification.

603 -8. Re-run the same stress scenario and compare against baseline with exact numbers.

604 -9. Attribute the current live `~700 MB` RSS case by combining:

605 - - kernel/process evidence from `/proc/<pid>/status`, `/proc/<pid>/smaps`, and `/proc/<pid>/smaps_rollup`

606 - - allocator/process evidence from the live glibc-backed binary

607 - - plugin-side accounting for the largest resident runtime structures

608 -10. Replace the coarse `unaccounted` bucket with named subsystem buckets where the codebase permits reliable accounting, so field diagnosis in customer environments points to concrete owners instead of just total RSS.

609 -11. Prove the live writable-journal mmap growth in the shared `journal-core` window manager with a focused regression test, then fix that shared behavior if the test confirms full-file growth from sequential append access.

610 -12. Verify or reject the current GeoIP/MMDB hypothesis with direct evidence from:

611 - - the auto-detection code path in the plugin

612 - - the actual MMDB files present on this machine

613 - - the live allocator/process footprint

614 -13. If GeoIP/MMDB is a dominant owner:

615 - - add explicit accounting for it in the memory charts

616 - - decide whether eager `Vec<u8>` loading is acceptable or should be replaced with a lower-RSS access pattern

617 -14. Extend the startup profiling harness so it also exercises the real plugin-runtime startup path:

618 - - construct `PluginRuntime`

619 - - register the `flows` handler

620 - - register charts

621 - - start the runtime on in-memory streams long enough for chart/runtime initialization to happen

622 - - measure whether that path adds any meaningful steady-state heap beyond the already measured ingest/query/rebuild path

623 -15. Re-run the full relevant test suites after the runtime-harness change:

624 - - `cargo test -p journal-core -p journal-log-writer -p journal-engine -p netflow-plugin`

625 -16. Re-check the live installed process after the harness and chart work:

626 - - validate RSS / anonymous / file-backed split

627 - - validate `memory_resident_mapping_bytes`

628 - - validate whether `memory_accounted_bytes.unaccounted` is now small enough to be operationally useful

629 - - current reality:

630 - - the live mapping chart is now operationally useful and explains the majority of RSS in concrete buckets (`heap`, `geoip_geo`, `geoip_asn`, journal tiers, `other_file`)

631 - - the logical `memory_accounted_bytes` chart still keeps a smaller `unaccounted` bucket because it intentionally tracks named plugin subsystems, not every resident mapping class

632 -

633 -## Implied decisions

634 -

635 -- All stress work will stay inside this worktree and temp directories created under it or safe local temp space.

636 -- No production routers, remote servers, or non-project directories will be touched beyond local profiling tools and temporary test data.

637 -- The stress harness should be reusable for future memory regressions, not a one-off manual sequence.

638 -

639 -## Testing requirements

640 -

641 -- Baseline and post-fix stress runs on the same workload and same binary mode.

642 -- Automated test coverage for the specific memory-driving structure(s) changed.

643 -- Regression-oriented assertions where practical:

644 - - bounded facet growth

645 - - bounded per-hour rollup index growth

646 - - no loss of ingest/query correctness for representative flows

647 -- Verification that new memory accounting/charts track the intended subsystems and update during load.

648 -- Re-run relevant `netflow-plugin` tests after code changes.

649 -

650 -### Verification completed so far

651 -

652 -- `cargo test -p netflow-plugin charts::tests`

653 -- `cargo test -p netflow-plugin facet_runtime::tests`

654 -- `cargo test -p netflow-plugin tiering::tests`

655 -- `cargo test -p netflow-plugin query::tests`

656 -- `cargo test -p netdata-flow-index`

657 -- `cargo test -p netflow-plugin stress_profile_high_cardinality_facet_runtime_memory -- --ignored --nocapture`

658 -- `cargo test -p netflow-plugin stress_profile_high_cardinality_tier_index_memory -- --ignored --nocapture`

659 -- `cargo test -p netflow-plugin stress_profile_high_cardinality_netflow_memory -- --ignored --nocapture`

660 -- `cargo test -p netflow-plugin stress_profile_decoder_source_port_churn_memory -- --ignored --nocapture`

661 -- `cargo test -p netflow-plugin decoder::tests::v9_parser_scope_reuses_templates_across_source_port_churn -- --nocapture`

662 -- `cargo test -p netflow-plugin decoder::tests::persisted_decoder_state_reuses_loaded_namespace_for_new_source_port -- --nocapture`

663 -- `cargo test -p journal-engine`

664 -- `cargo test -p netflow-plugin`

665 -

666 -### Broader suite status

667 -

668 -- The earlier 4 broader-suite failures were root-caused and fixed in this worktree:

669 - - `decoder::tests::akvorado_sflow_vxlan_fixture_matches_expected_inner_projection`

670 - - `decoder::tests::akvorado_sflow_1140_fixture_matches_expected_sample_projection`

671 - - `enrichment::tests::enricher_is_disabled_when_configuration_is_empty`

672 - - `ingest::tests::ingest_service_with_decap_vxlan_extracts_inner_header_view`

673 -- Verified before the latest memory/rebuild commit:

674 - - `cargo test -p netflow-plugin`

675 - - result: `375 passed, 0 failed, 13 ignored`

676 -- New requirement on `2026-04-11`: re-run the relevant suites after the latest pushed commit and treat any new failure as a blocker before continuing the unresolved live memory investigation.

677 -

678 -### Verified root causes for the remaining test failures

679 -

680 -- Empty enrichment config is not actually empty:

681 - - [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/enrichment/root.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/types/enrichment/root.rs) makes `EnrichmentConfig::default()` include non-empty `asn_providers` and `net_providers`.

682 - - [`src/crates/netdata-netflow/netflow-plugin/src/enrichment/init.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/enrichment/init.rs) treats any non-empty provider list as sufficient to enable enrichment.

683 - - Result: [`src/crates/netdata-netflow/netflow-plugin/src/enrichment/tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/enrichment/tests.rs) `enricher_is_disabled_when_configuration_is_empty()` fails because `EnrichmentConfig::default()` can no longer represent a disabled configuration.

684 -- IPv4/IPv6 packet-section length accounting is clamping to captured bytes instead of using the on-wire L3 length encoded in the packet headers:

685 - - [`src/crates/netdata-netflow/netflow-plugin/src/decoder/protocol/packet/ip.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/decoder/protocol/packet/ip.rs) returns `captured_length` / `payload_end`.

686 - - [`src/crates/netdata-netflow/netflow-plugin/src/decoder/record/packet/parse/ip.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/decoder/record/packet/parse/ip.rs) mirrors the same clamp in the `FlowRecord` path.

687 - - [`src/crates/netdata-netflow/netflow-plugin/src/decoder/protocol/sflow/record.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/decoder/protocol/sflow/record.rs) then uses that returned length as `rec.bytes`.

688 - - Reproduced failures:

689 - - `data-encap-vxlan.pcap`: expected `BYTES=104`, actual `BYTES=64`

690 - - `data-1140.pcap`: expected visible `BYTES=1536000`, actual `BYTES=116736`, which means raw `114 * 1024` instead of `1500 * 1024`

691 - - Upstream Akvorado evidence:

692 - - [`/opt/baddisk/monitoring/akvorado/akvorado__akvorado/outlet/flow/decoder/sflow/root_test.go`](/opt/baddisk/monitoring/akvorado/akvorado__akvorado/outlet/flow/decoder/sflow/root_test.go) expects `1500` for `data-1140.pcap` and `104` for `data-encap-vxlan.pcap`.

693 - - [`/opt/baddisk/monitoring/akvorado/akvorado__akvorado/outlet/flow/decoder/helpers.go`](/opt/baddisk/monitoring/akvorado/akvorado__akvorado/outlet/flow/decoder/helpers.go) returns the IPv4/IPv6 header-declared L3 length, not the truncated capture size.

694 -

695 -### Open reality check

696 -

697 -- The work is not yet at the requested end state.

698 -- The facet hotspot is materially better and now observable via charts/accounting.

699 -- The tier index hotspot is materially better and now exposes an internal breakdown via charts/accounting.

700 -- The decoder template/parser churn case is now both observable and fixed for same-exporter source-port churn, with a measured `~430x` reduction on the synthetic churn harness.

701 -- The measured focused reductions are real:

702 - - facet hotspot RSS delta: `21.24 MB -> 8.77 MB`

703 - - tier hotspot RSS delta: `22.21 MB -> 9.90 MB`

704 - - decoder source-port churn hotspot RSS delta: `867.91 MB -> 2.02 MB`

705 -- These measured results support a `>30x` reduction only for the verified decoder source-port churn failure mode. They still do not justify claiming an overall end-to-end `30x` reduction for all workloads.

706 -- Live measurement on `2026-04-11` shows the currently installed plugin at about `674 MiB` RSS (`706,560,000` bytes from the chart; `~690-708 MB` from `/proc` depending on sampling time).

707 -- The live process is dominated by private anonymous memory:

708 - - `RssAnon`: about `660 MB`

709 - - `RssFile`: about `30-48 MB`

710 - - largest mappings: `[heap]` about `361 MB`, `[anon]` about `299 MB`

711 -- The current charted accounting does not explain the live footprint yet:

712 - - `netdata.netflow.memory_accounted_bytes.unaccounted`: `692,828,350` bytes

713 - - all explicitly named buckets combined: about `13.1 MB`

714 - - conclusion: the chart framework works, but it still lacks attribution for the dominant live heap owners in this case

715 -- Live allocator introspection through `gdb` has a verified safety boundary:

716 - - attaching and detaching from the running process is safe on this machine

717 - - resolving glibc allocator symbols (`malloc_info`, `mallinfo2`) is also possible

718 - - calling `malloc_info()` from inside the live process is not safe in this environment: it faulted inside glibc `fputs()` while writing the allocator report and the plugin PID disappeared afterwards

719 - - this means "live allocator dump via injected `gdb` call" is no longer an acceptable primary method for this process

720 -

721 -## New obstacle

722 -

723 -- `gdb`-injected allocator reporting is not safe enough for the running plugin.

724 -- Verified evidence:

725 - - `gdb` attach/detach works and shared-library symbol resolution can find `malloc_info` / `mallinfo2`.

726 - - forcing `malloc_info()` inside the live process caused a fault in glibc while executing `fputs()`.

727 - - the inspected plugin PID (`1164472`) disappeared after that attempt.

728 -- Implication:

729 - - allocator attribution must move to one of these safer paths:

730 - - an in-process diagnostic endpoint/chart implemented in the plugin itself

731 - - an isolated stress-harness build with allocator tracing/profiling enabled

732 - - offline process/core analysis instead of injected live function calls

733 -- The broader `netflow-plugin` suite was green before commit `d6e4ff75c0`, but it must be re-validated after that commit before more optimization work continues.

734 -- Re-validated after commit `d6e4ff75c0`:

735 - - `cargo test -p journal-engine`: passed

736 - - `cargo test -p netflow-plugin`: passed

737 - - `netflow-plugin` result: `375 passed, 0 failed, 13 ignored`

738 -

739 -### New live-memory findings after test revalidation on 2026-04-11

740 -

741 -- The current running plugin (`/usr/libexec/netdata/plugins.d/netflow-plugin`, PID observed as `1356956`) is now dominated by both anonymous and file-backed resident memory:

742 - - `VmRSS`: about `623,908 kB`

743 - - `RssAnon`: about `355,928 kB`

744 - - `RssFile`: about `267,980 kB`

745 - - `Threads`: `14`

746 -- The live process currently holds four very large writable shared journal mappings:

747 - - raw journal mapping: about `111,668 kB RSS`

748 - - `1m` journal mapping: about `77,344 kB RSS`

749 - - `5m` journal mapping: about `53,480 kB RSS`

750 - - `1h` journal mapping: about `19,552 kB RSS`

751 - - Evidence: `/proc/<pid>/maps` and `pmap -x` show large `rw-s` mappings for the active journal files in `raw`, `1m`, `5m`, and `1h`.

752 -- The same live process also still holds dozens of open file descriptors to the rebuild index cache directory:

753 - - `/var/cache/netdata/flows/.rebuild-index-cache/foyer-storage-direct-fs-*`

754 - - Evidence: `/proc/<pid>/fd`

755 - - This means the rebuild-scoped Foyer cache is not being closed cleanly after startup.

756 -- The release startup harness with the current code and the same local journal base directory does not reproduce the full live file-backed shape:

757 - - `mmap_in_use` peaks at about `33.6 MB`

758 - - rebuild settle RSS is about `156.9 MB`

759 - - conclusion: the remaining live RSS is not just startup rebuild anymore

760 -- The journal window manager implementation in [`src/crates/journal-core/src/file/mmap.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/mmap.rs) has a likely structural bug for sequential appends:

761 - - when a request starts inside an existing window but extends beyond its end, `get_window()` remaps from the existing window's original start instead of aligning a fresh bounded tail window to the current position

762 - - this can grow one mapping from the beginning of the file toward the current write position over time

763 - - this behavior matches the observed one-large-map-per-active-tier pattern in the live process

764 -

765 -### Verified diagnosis of the large live journal mmaps on 2026-04-11

766 -

767 -- The suspicious shared remap logic in [`src/crates/journal-core/src/file/mmap.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/mmap.rs) was **not** introduced by this PR:

768 - - `git blame` on the remap branch (`window_start = window.offset`) points to commit `adc4d66ea03` from `2025-05-28`

769 - - the only newer changes around that code are the later consistency guards for mmap failure handling

770 -- The `journal-log-writer` active writer window size of `8 MiB` was also not changed by this PR:

771 - - `git blame` on [`src/crates/journal-log-writer/src/log/mod.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-log-writer/src/log/mod.rs) shows `.with_window_size(8 * 1024 * 1024)` coming from commit `d0905d9b99`

772 -- The shared journal code explicitly documents the intended design:

773 - - [`src/crates/journal-core/src/file/file.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/file.rs) says the implementation should maintain a **small set of memory-mapped windows** instead of mapping the entire file

774 -- Live process evidence contradicts that intended design for the active writable journals:

775 - - current active file sizes:

776 - - raw: `152 MiB`

777 - - `1m`: `128 MiB`

778 - - `5m`: `72 MiB`

779 - - `1h`: `24 MiB`

780 - - current main `rw-s` mapping sizes for those same files:

781 - - raw: `152 MiB`

782 - - `1m`: `128 MiB`

783 - - `5m`: `72 MiB`

784 - - `1h`: `24 MiB`

785 - - conclusion: the main writable mapping size matches the full current file size, not the configured `8 MiB` writer window

786 -- The always-mapped journal hash tables are **not** the explanation:

787 - - for a `256 MiB` max file size, the persistent data-hash table is about `1 MiB`

788 - - the field-hash table is negligible (`~2 KiB`)

789 - - therefore the large `24-152 MiB` mappings are not expected fixed metadata maps

790 -- The most likely overall explanation is now:

791 - - the shared journal window-manager behavior is pre-existing

792 - - this PR exposed it operationally by creating a workload with four continuously written active journals (`raw`, `1m`, `5m`, `1h`) and large active file sizes

793 - - the recent PR changes in `journal-log-writer` themselves are timestamp/lifecycle additions, not mmap-window behavior changes

794 -

795 -### Live revalidation after fixing the shared journal window manager on 2026-04-11

796 -

797 -- After installing the bounded-window fix from [`src/crates/journal-core/src/file/mmap.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/mmap.rs), the live process no longer maps whole active journal files:

798 - - live `raw` writer map: `8 MiB`

799 - - live `1m` writer map: `8 MiB`

800 - - the extra `~1 MiB` maps per file remain, which matches expected fixed per-file metadata mappings

801 -- Live resident memory after restart materially improved:

802 - - `/proc/<pid>/status` showed `RssFile` collapse from about `268 MiB` before the fix to about `19-28 MiB` after the fix

803 - - the same live process stabilized around `VmRSS ≈ 350-408 MiB`, `RssAnon ≈ 331-384 MiB`, `Threads = 14`

804 -- Netdata's own live charts now agree with `/proc`:

805 - - `netdata.netflow.memory_resident_bytes` current samples show `rss ≈ 406-408 MiB`, `rss_anon ≈ 383.6 MiB`, `rss_file ≈ 22.7-25.0 MiB`

806 - - `netdata.netflow.memory_accounted_bytes` current samples still show `unaccounted ≈ 393-395 MiB`

807 - - `netdata.netflow.memory_allocator_bytes` current samples show:

808 - - `heap_in_use ≈ 66.97 MiB`

809 -

810 -### Verified allocator-retention diagnosis and post-fix live state on 2026-04-11

811 -

812 -- GeoIP/MMDB data is not a dominant owner in this environment:

813 - - auto-detection uses [`src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/plugin_config/runtime.rs) and eager loading uses [`src/crates/netdata-netflow/netflow-plugin/src/enrichment/data/geoip/files.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/enrichment/data/geoip/files.rs)

814 - - actual detected files on this machine:

815 - - `/var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb`: `10,402,916` bytes

816 - - `/var/cache/netdata/topology-ip-intel/topology-ip-country.mmdb`: `5,875,413` bytes

817 - - combined resident upper bound is only about `16.3 MB`, so MMDB files cannot explain the remaining `~200+ MB` anonymous RSS

818 -- The allocator chart's `mmap_in_use` value was re-checked and must not be treated as resident RAM by itself:

819 - - [`src/crates/netdata-netflow/netflow-plugin/src/memory_allocator.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/memory_allocator.rs) reports `mallinfo2().hblkhd`

820 - - `/proc/<pid>/smaps` still shows the dominant live resident memory in anonymous heap/arena mappings, not in a separate resident mapping equal to `mmap_in_use`

821 - - implication: `mmap_in_use` is allocator state, not a direct resident-memory answer

822 -- Correct live process before the latest allocator-focused fixes:

823 - - observed plugin PID: `1699732`

824 - - `/proc/<pid>/status`:

825 - - `VmRSS: 362,188 kB`

826 - - `RssAnon: 320,984 kB`

827 - - `RssFile: 41,204 kB`

828 - - `Threads: 9`

829 - - `/proc/<pid>/smaps_rollup`:

830 - - `Private_Dirty: 320,984 kB`

831 - - `AnonHugePages: 0 kB`

832 - - top resident mappings:

833 - - anonymous: `146,000 kB`

834 - - `[heap]`: `82,912 kB`

835 - - anonymous: `39,032 kB`

836 - - anonymous: `33,164 kB`

837 - - anonymous: `18,784 kB`

838 -- The dominant remaining mechanism was verified as allocator-retained free heap:

839 - - live allocator chart before the latest code change showed approximately:

840 - - `heap_arena ≈ 500,064,300`

841 - - `heap_free ≈ 433,473,010`

842 - - `heap_in_use ≈ 66,591,236`

843 - - conclusion:

844 - - most anonymous heap footprint was not live payload; it was free heap retained by glibc after earlier startup/runtime allocation spikes

845 -- The service environment already had `MALLOC_ARENA_MAX=4`, but reducing it further to `1` still helped on the real startup workload:

846 - - release multithread harness on the retained live dataset:

847 - - `MALLOC_ARENA_MAX=4`: `rebuild_settle_trim delta_rss = 80,465,920`

848 - - `MALLOC_ARENA_MAX=1`: `rebuild_settle_trim delta_rss = 47,730,688`

849 - - verified reduction: about `32.7 MB`

850 -- Fix implemented in [`src/crates/netdata-netflow/netflow-plugin/src/memory_allocator.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/memory_allocator.rs) and wired from [`src/crates/netdata-netflow/netflow-plugin/src/main.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/main.rs):

851 - - call `mallopt(M_ARENA_MAX, 1)` inside the plugin process before Tokio runtime creation

852 - - keep THP disabled for the process

853 -- Additional startup-allocation reduction implemented in [`src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/facet_runtime.rs):

854 - - load persisted facet state via `BufReader + bincode::deserialize_from()` instead of `fs::read()`

855 - - consume persisted archived stores directly instead of cloning every saved store during hydration

856 -- Verified live state after installing the arena-cap build:

857 - - observed plugin PID: `1728342`

858 - - `/proc/<pid>/status`:

859 - - `VmRSS: 227,880 kB`

860 - - `RssAnon: 213,344 kB`

861 - - `RssFile: 14,536 kB`

862 - - `Threads: 9`

863 - - allocator chart:

864 - - `heap_arena ≈ 326,135,800`

865 - - `heap_free ≈ 261,295,660`

866 - - `heap_in_use ≈ 64,840,148`

867 - - `mmap_in_use ≈ 131,457,140`

868 - - `unaccounted ≈ 225,923,660`

869 - - verified improvement versus the earlier `362,188 kB` live RSS: about `134 MB` lower

870 -- Verified live state after also installing the persisted-load cleanup:

871 - - observed plugin PID: `1743390`

872 - - `/proc/<pid>/status`:

873 - - `VmRSS: 227,840 kB`

874 - - `RssAnon: 211,636 kB`

875 - - `RssFile: 16,204 kB`

876 - - `Threads: 9`

877 - - allocator chart:

878 - - `heap_arena ≈ 313,335,800`

879 - - `heap_free ≈ 260,037,540`

880 - - `heap_in_use ≈ 53,298,250`

881 - - `mmap_in_use ≈ 143,175,840`

882 - - `unaccounted ≈ 222,713,120`

883 - - conclusion:

884 - - the persisted-load cleanup helped slightly, but the major win came from capping glibc arena growth inside the process

885 -

886 -### Verified archived facet deep-size diagnosis on 2026-04-11

887 -

888 -- A targeted allocative diagnostic was added and executed against the live dataset:

889 - - `cargo test -p netflow-plugin stress_profile_live_archived_facet_allocative_breakdown -- --ignored --nocapture`

890 -- Results on the current `facet-state.bin`:

891 - - archived facet memory by deep allocative traversal: `40,524,572` bytes

892 - - archived path tracking: `38,026` bytes

893 - - published snapshot: `5,844` bytes

894 -- Top archived facet fields by measured deep allocation:

895 - - `FLOW_END_USEC`: `32,263,968` bytes

896 - - `DST_ADDR_NAT`: `1,564,840`

897 - - `DST_ADDR`: `1,564,768`

898 - - `SRC_ADDR`: `1,560,312`

899 - - `SRC_ADDR_NAT`: `1,560,304`

900 - - `DST_AS_NAME`: `589,824`

901 - - `SRC_AS_NAME`: `589,824`

902 -- This diagnostic also proved the current estimator is materially wrong for that dominant field:

903 - - `FLOW_END_USEC` current estimated bytes: `5,456,148`

904 - - `FLOW_END_USEC` measured deep bytes: `32,263,968`

905 - - undercount factor: about `5.9x`

906 -- Interpretation:

907 - - the current facet chart under-accounting is not generic hand-waving anymore; it now has a named dominant owner

908 - - the dominant owner is a timestamp-microsecond facet that is currently being treated as an archived/autocomplete/searchable dimension like a normal categorical facet

909 -- Follow-up verification after Costa approved removal of those fields:

910 - - `cargo test -p netflow-plugin stress_profile_live_archived_facet_allocative_breakdown -- --ignored --nocapture`

911 - - current archived facet memory by deep allocative traversal: `8,260,553` bytes

912 - - current top archived fields are now IP and ASN/name dimensions:

913 - - `DST_ADDR_NAT`: `1,564,840`

914 - - `DST_ADDR`: `1,564,768`

915 - - `SRC_ADDR`: `1,560,312`

916 - - `SRC_ADDR_NAT`: `1,560,304`

917 - - `DST_AS_NAME`: `589,824`

918 - - `SRC_AS_NAME`: `589,824`

919 - - conclusion:

920 - - removing the internal timestamp fields produced a measured `~4.9x` reduction in archived facet deep size on the same dataset (`40.5 MB -> 8.3 MB`)

921 - - the next archived facet owners are IP dictionaries, not hidden timestamp sidecars

922 - - `heap_free ≈ 604.41 MiB`

923 - - `heap_arena ≈ 671.38 MiB`

924 - - `mmap_in_use ≈ 165.03 MiB`

925 - - `releasable ≈ 4 KiB`

926 -- This is a strong allocator-fragmentation/retention signal:

927 - - the live process currently has far more free heap retained inside glibc arenas than live heap objects

928 - - `malloc_trim()` by itself is unlikely to solve the remaining problem because the allocator reports only `~4 KiB` releasable despite `~604 MiB` free inside arenas

929 -- A release startup harness run against the same live journal corpus no longer reproduces the current live RSS:

930 - - `cargo test -p netflow-plugin --release stress_profile_live_startup_memory_multithreaded -- --ignored --nocapture`

931 - - result after settle+trim: `rss ≈ 97.8 MiB`, `heap_free ≈ 324.9 MiB`, `heap_arena ≈ 379.5 MiB`, `threads = 7`

932 - - implication: pure startup rebuild is no longer sufficient to explain the live `~400 MiB` steady-state footprint; the remaining growth is tied to runtime behavior while the service is active

933 -- A new verified runtime-lifetime bug remains:

934 - - the live process still keeps `64` open file descriptors under `/var/cache/netdata/flows/.rebuild-index-cache/foyer-storage-direct-fs-*` long after rebuild should have finished

935 - - this indicates the Foyer rebuild index cache must be closed explicitly instead of relying on drop timing

936 - - source review of `foyer-storage` shows why `close()` alone is insufficient: the block-engine `close()` path stops writers and waits for reclaim, but the direct-fs partitions keep their `File` handles in `FsPartition.file` until the storage device itself is dropped

937 - - because the rebuild cache is only scratch state for one rebuild run, the safer memory-first fix is to stop using disk-backed Foyer storage for this path and keep the cache in memory only

938 -- Allocator-substitution diagnostics were run against the same startup profile and rejected:

939 - - glibc baseline settle+trim: `~97.8 MiB RSS`

940 - - `mimalloc` preload settle: `~252.3 MiB RSS`

941 - - `jemalloc` preload settle: `~446.4 MiB RSS`

942 - - glibc with transparent huge pages disabled settle+trim: `~138.2 MiB RSS`

943 - - conclusion: allocator replacement and THP disablement are not the right first-line fix for this workload

944 -

945 -### Verified shared journal mmap growth diagnosis on 2026-04-11

946 -

947 -- The shared remap branch in `origin/master` was inspected with:

948 - - `git show origin/master:src/crates/journal-core/src/file/mmap.rs | sed -n '200,260p'`

949 - - verified old logic:

950 - - remove current window

951 - - keep `window_start = window.offset`

952 - - extend only `window_end` to cover the new request

953 -- That means append-heavy accesses crossing the current window boundary can grow a single writable mapping from the beginning of the file toward the tail instead of sliding a bounded window.

954 -- The current worktree contains a focused regression in [`src/crates/journal-core/src/file/mmap.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/journal-core/src/file/mmap.rs#L476):

955 - - `sequential_boundary_crossing_slides_window_instead_of_growing_from_start`

956 - - it verifies a `max_windows=1` manager with `4 KiB` chunks:

957 - - first cross-boundary remap produces a `2`-chunk window at offset `0`

958 - - the next cross-boundary remap slides to offset `4096` instead of growing to `12288` bytes from offset `0`

959 - - the next remap slides again to offset `8192`

960 -- Verified test evidence:

961 - - `cargo test -p journal-core sequential_boundary_crossing_slides_window_instead_of_growing_from_start -- --nocapture`

962 - - `cargo test -p journal-core`

963 - - `cargo test -p journal-log-writer`

964 - - `cargo test -p journal-engine`

965 - - `cargo test -p netflow-plugin`

966 - - all passed on the current worktree

967 -- Conclusion:

968 - - the large writable journal mmaps observed live are consistent with a real pre-existing shared `journal-core` remap-growth bug/design failure, not with new mmap logic introduced by this PR

969 - - `netflow-plugin` exposed it because it keeps several active tier writers growing concurrently

970 - - this shared fix is still uncommitted at the time of this TODO update

971 -- Live post-install snapshot after installing the current worktree and restarting Netdata:

972 - - observed plugin PID: `1808192`

973 - - `/proc/<pid>/status`:

974 - - `VmRSS: 181,204 kB` initially, `185,940 kB` after another minute

975 - - `RssAnon: 165,896 kB` initially, `166,268 kB` after another minute

976 - - `RssFile: 15,308 kB` initially, `19,672 kB` after another minute

977 - - `Threads: 9`

978 - - `/proc/<pid>/smaps_rollup` after the extra minute:

979 - - `Rss: 185,940 kB`

980 - - `Private_Dirty: 168,508 kB`

981 - - `Private_Clean: 14,944 kB`

982 - - `AnonHugePages: 0 kB`

983 - - active journal mappings in `/proc/<pid>/maps`:

984 - - raw active journal mapped bytes: `9,449,472`

985 - - `1m` active journal mapped bytes: `9,449,472`

986 - - current active file sizes: `8,388,608` each

987 - - caveat:

988 - - this is a fresh-restart snapshot, so it is not a perfect long-runtime comparison to the earlier `24-152 MB` active full-file mappings

989 - - however it is consistent with the bounded-window fix and materially lower than the earlier live `227-362 MB` snapshots on this workstation

990 -

991 -### Performance-testing feasibility on 2026-04-12

992 -

993 -- `FEASIBLE AS SPECIFIED`

994 -- Existing benchmark and profiling surface already covers most of the path we need:

995 - - end-to-end ingest hot path replay already exists in [`src/crates/netdata-netflow/netflow-plugin/src/ingest_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/ingest_tests.rs) via `bench_full_hot_path()`

996 - - fixture replay helpers already exist in [`src/crates/netdata-netflow/netflow-plugin/src/main_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/main_tests.rs) via `ingest_fixture()` and `replay_fixture_udp()`

997 - - raw query stage profiling already exists in [`src/crates/netdata-netflow/netflow-plugin/src/query/scan/bench.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/query/scan/bench.rs)

998 - - memory stress harnesses already exist in [`src/crates/netdata-netflow/netflow-plugin/src/memory_tests.rs`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/src/memory_tests.rs)

999 -- Current gaps:

1000 - - the existing ingest hot-path benchmark mixes protocols together instead of reporting per-protocol throughput

1001 - - it currently covers NetFlow v5, NetFlow v9, and IPFIX fixture replay, but not sFlow in the same throughput harness

1002 - - it does not yet vary field variability or cardinality in a controlled way

1003 - - it does not emit a stable summary format suitable for repeated comparison

1004 -- First implementation path:

1005 - - keep the current ignored-test harness model

1006 - - extend `bench_full_hot_path()` or split it into protocol-specific ignored tests

1007 - - add protocol-isolated replay groups for NetFlow v5, NetFlow v9, IPFIX, and sFlow

1008 - - add synthetic high-cardinality flow generation using the existing stress-helper patterns so we can measure the effect of field variability/cardinality separately from decoder cost

1009 - - report at minimum:

1010 - - flows per second

1011 - - packets per second

1012 - - bytes per second

1013 - - microseconds per flow

1014 - - peak RSS / resident mapping snapshot when practical

1015 -

1016 -## Documentation updates required

1017 -

1018 -- Completed:

1019 - - updated [`src/crates/netdata-netflow/netflow-plugin/README.md`](/home/costa/src/PRs/topology-netflow-22111/src/crates/netdata-netflow/netflow-plugin/README.md) with the new memory chart names and their operational meaning, including decoder scope diagnostics for source-port churn.

Delete TODO-netflow-memory-footprint.md (#22252)