master
md 529 lines 21 KB
Rendered Raw
1 # IBM i (AS/400) collector
2
3 ## Overview
4
5 Monitors IBM i (AS/400) systems using SQL services and CL commands to
6 expose CPU, memory, storage, job, and subsystem activity.
7
8 **Dependencies:**
9 - unixODBC 2.3+ with IBM i Access ODBC driver
10 - IBM i 7.2 or later with SQL services enabled
11
12 **Required Libraries:**
13 - libodbc.so (provided by unixODBC)
14 - IBM i Access Client Solutions
15
16 **Collection paths**
17
18 The collector executes queries in multiple tracks:
19
20 - **Fast path (5s)**: lightweight system status queries remain sequential on the main plugin thread.
21 - **Slow path (10s beat)**: heavier queries (per-queue metrics, subsystems, plan cache, etc.) run in a background worker with bounded concurrency.
22 - **Batch path (≥60s beat)**: optional long-period worker used for expensive aggregate queries such as queue totals. Disabled by default unless queue totals are explicitly enabled.
23
24 **CPU Collection Methods:**
25
26 The collector uses a hybrid approach for CPU utilization metrics to handle IBM i 7.4+ where
27 `AVERAGE_CPU_*` columns were deprecated:
28
29 1. **Primary Method - TOTAL_CPU_TIME**: Uses the monotonic `TOTAL_CPU_TIME` counter from
30 `QSYS2.SYSTEM_STATUS()` to calculate CPU utilization via delta-based calculation. This is
31 the most accurate method but requires `*JOBCTL` special authority. TOTAL_CPU_TIME is a
32 cumulative counter in nanoseconds representing CPU-seconds consumed, naturally in per-core
33 scale.
34
35 2. **Fallback Method - ELAPSED_CPU_USED**: If `*JOBCTL` authority is not available, falls back
36 to `ELAPSED_CPU_USED` with automatic reset detection. This method tracks when IBM i statistics
37 are reset (either manually or via `reset_statistics` configuration) and re-establishes a
38 baseline after detecting resets. The values are already in per-core scale.
39
40 3. **Legacy Method - AVERAGE_CPU_UTILIZATION**: For IBM i versions before 7.4, uses the now-
41 deprecated `AVERAGE_CPU_UTILIZATION` column, which IBM reports in the same per-core scale.
42
43 The collector automatically selects the appropriate method based on available permissions and
44 logs which method is being used.
45
46 **CPU Metric Scale:**
47
48 CPU utilization is reported using the "100% = 1 CPU core" semantic. This means:
49 - 100% indicates one CPU core is fully utilized
50 - 400% indicates four CPU cores are fully utilized
51 - Values are limited to 100% × ConfiguredCPUs, matching the partition's configured capacity
52
53 For shared LPARs, the metrics show absolute CPU consumption in per-core scale, not relative to
54 entitled capacity. For example, a shared LPAR entitled to 0.20 cores can show 150% utilization
55 when bursting above entitlement.
56
57 **Statistics Reset Behavior:**
58
59 The `reset_statistics` configuration option controls whether the collector resets IBM i system
60 statistics on each query via `SYSTEM_STATUS(RESET_STATISTICS=>'YES')`. When enabled:
61
62 - System-level statistics (CPU, memory pools, etc.) are reset after each collection cycle
63 - Matches legacy behavior but clears global statistics that other tools may rely on
64 - The ELAPSED_CPU_USED fallback method will detect and handle these resets automatically
65 - **Caution**: Enabling this affects all users and applications on the IBM i system
66
67 Default: `false` (statistics are not reset, using `RESET_STATISTICS=>'NO'`)
68
69 **Chart Gaps During Baseline Resets:**
70
71 The `as400.system_activity_cpu_rate` and `as400.system_activity_cpu_utilization` charts rely on
72 delta calculations. When the collector detects that IBM i reset these statistics—or when it is
73 still establishing the initial baseline—it intentionally skips a sample instead of emitting a zero
74 or spike. Netdata renders those skipped samples as small gaps, which is expected behaviour.
75
76 **Cardinality Management:**
77
78 To prevent performance issues from excessive metric creation, the collector enforces cardinality
79 limits on per-instance metrics (disks, subsystems, job queues, message queues, output queues,
80 active jobs, network interfaces, HTTP servers).
81
82 **How Limits Work:**
83 - The collector counts instances before collecting metrics
84 - If count exceeds the configured `max_*` limit, **collection is skipped entirely** for that category
85 - The collector logs a warning: `"[category] count (X) exceeds limit (Y), skipping collection"`
86 - No metrics are collected for that category until you adjust the configuration
87
88 **Configuration Options:**
89
90 Use **both** limit and selector options together to manage high-cardinality environments:
91
92 | Option | Purpose | Default |
93 |--------|---------|---------|
94 | `max_disks` | Maximum disk units to monitor | 100 |
95 | `max_subsystems` | Maximum subsystems to monitor | 100 |
96 | `max_job_queues` | Maximum job queues to monitor | 100 |
97 | `max_message_queues` | Maximum message queues to monitor | 100 |
98 | `max_output_queues` | Maximum output queues to monitor | 100 |
99 | `active_jobs` | Fully qualified active jobs to monitor (`JOB_NUMBER/USER/JOB_NAME`) | `[]` |
100 | `collect_disks_matching` | Glob pattern to filter disks (e.g., `"001* 002*"`) | `""` (match all) |
101 | `collect_subsystems_matching` | Glob pattern to filter subsystems (e.g., `"QINTER QBATCH"`) | `""` (match all) |
102 | `collect_job_queues_matching` | Glob pattern to filter job queues (e.g., `"QSYS/*"`) | `""` (match all) |
103
104 Optional batch-path controls:
105
106 | Option | Purpose | Default |
107 |--------|---------|---------|
108 | `batch_path` | Enables the long-period batch worker for aggregate queries | `false` |
109 | `batch_path_update_every` | Batch worker cadence (minimum 60s, recommend ≥600s in production) | `60s` |
110 | `batch_path_max_connections` | Maximum concurrent connections for batch queries | `1` |
111 | `collect_message_queue_totals` | Enables full-scan counting of all message queues and messages | `auto` (off) |
112 | `collect_job_queue_totals` | Enables aggregate counting of job queues and queued jobs | `auto` (off) |
113 | `collect_output_queue_totals` | Enables aggregate counting of output queues and spooled files | `auto` (off) |
114
115 > **Warning:** queue totals require scanning IBM i catalog views and can be very expensive on large systems. Leave these options disabled unless aggregate counts are absolutely necessary.
116
117
118 **Example Workflow:**
119
120 1. System has 500 disks, collector skips disk metrics (exceeds default limit of 100)
121 2. Check logs: `"disk count (500) exceeds limit (100), skipping per-disk metrics"`
122 3. Two options:
123 - **Option A**: Increase limit: `max_disks: 500` (collects all 500 disks)
124 - **Option B**: Use selector: `collect_disks_matching: "00[1-5]*"` (cherry-pick specific disks)
125
126 **Best Practices:**
127 - Use selectors to monitor only business-critical objects in large environments
128 - Set limits based on your Netdata server's capacity (each instance = multiple charts)
129 - Start with defaults and adjust based on actual usage patterns
130
131 **IBM i 7.2–7.3 Behavior Note (Message Queues):**
132
133 IBM i 7.4 introduced a message-queue table function that returns only the live backlog. On
134 7.2–7.3 systems we fall back to the `QSYS2.MESSAGE_QUEUE_INFO` view, which includes *all*
135 recorded messages (even those already processed/cleared from the queue). Aggregations—especially
136 `MAX(SEVERITY)`—therefore reflect the historical log, not just the outstanding backlog. This
137 behaviour is inherent to the IBM SQL service and can lead to higher-than-expected max severity
138 values on pre-7.4 systems.
139
140 Network interface metrics have a fixed internal limit of 50 instances, and HTTP server metrics are capped at 200 instances; these limits are currently not configurable.
141
142
143 This collector is part of the [Netdata](https://github.com/netdata/netdata) monitoring solution.
144
145 ## Collected metrics
146
147 Metrics grouped by scope.
148
149 The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
150
151 ### Per IBM i (AS/400) instance
152
153
154 These metrics refer to the entire monitored IBM i (AS/400) instance.
155
156 This scope has no labels.
157
158 Metrics:
159
160 | Metric | Dimensions | Unit |
161 |:-------|:-----------|:-----|
162 | netdata.plugin_ibm.as400_query_latency_fast | count_disks, count_http_servers, count_network_interfaces, detect_ibmi_version_primary, detect_ibmi_version_fallback, disk_instances, disk_instances_enhanced, disk_status, http_server_info, job_info, memory_pools, network_connections, network_interfaces, serial_number, system_name, system_activity, system_model, system_status, temp_storage_named, temp_storage_total, technology_refresh_level, active_job | ms |
163 | netdata.plugin_ibm.as400_query_latency_slow | analyze_plan_cache, count_subsystems, subsystems, message_queue_aggregates, job_queues, output_queue_info, plan_cache_summary | ms |
164 | netdata.plugin_ibm.as400_query_latency_batch | message_queue_totals, job_queue_totals, output_queue_totals | ms |
165
166 These metrics refer to the entire monitored IBM i (AS/400) instance.
167
168 This scope has no labels.
169
170 Metrics:
171
172 | Metric | Dimensions | Unit |
173 |:-------|:-----------|:-----|
174 | as400.cpu_utilization | utilization | percentage |
175 | as400.cpu_utilization_entitled | utilization | percentage |
176 | as400.cpu_configuration | configured | cpus |
177 | as400.cpu_capacity | capacity | percentage |
178 | as400.total_jobs | total | jobs |
179 | as400.active_jobs_by_type | batch, interactive, active | jobs |
180 | as400.job_queue_length | waiting | jobs |
181 | as400.main_storage_size | total | bytes |
182 | as400.temporary_storage | current, maximum | MiB |
183 | as400.memory_pool_usage | machine, base, interactive, spool | bytes |
184 | as400.memory_pool_defined | machine, base | bytes |
185 | as400.memory_pool_reserved | machine, base | bytes |
186 | as400.memory_pool_threads | machine, base | threads |
187 | as400.memory_pool_max_threads | machine, base | threads |
188 | as400.disk_busy_average | busy | percentage |
189 | as400.system_asp_usage | used | percentage |
190 | as400.system_asp_storage | total | MiB |
191 | as400.total_auxiliary_storage | total | MiB |
192 | as400.system_threads | active, per_processor | threads |
193 | as400.network_connections | remote, total | connections |
194 | as400.network_connection_states | listen, close_wait | connections |
195 | as400.temp_storage_total | current, peak | bytes |
196 | as400.system_activity_cpu_rate | average | percentage |
197 | as400.system_activity_cpu_utilization | average, minimum, maximum | percentage |
198
199
200
201 ### Per activejob
202
203 These metrics refer to individual activejob instances.
204
205 Labels:
206
207 | Label | Description |
208 |:------|:------------|
209 | job_name | Job_name identifier |
210 | job_status | Job_status identifier |
211 | subsystem | Subsystem identifier |
212 | job_type | Job_type identifier |
213
214 Metrics:
215
216 | Metric | Dimensions | Unit |
217 |:-------|:-----------|:-----|
218 | as400.activejob_cpu | cpu | percentage |
219 | as400.activejob_resources | temp_storage | MiB |
220 | as400.activejob_time | cpu_time, total_time | seconds |
221 | as400.activejob_activity | disk_io, interactive_transactions | operations/s |
222 | as400.activejob_threads | threads | threads |
223
224 ### Per disk
225
226 These metrics refer to individual disk instances.
227
228 Labels:
229
230 | Label | Description |
231 |:------|:------------|
232 | disk_unit | Disk_unit identifier |
233 | disk_type | Disk_type identifier |
234 | disk_model | Disk_model identifier |
235 | hardware_status | Hardware_status identifier |
236 | disk_serial_number | Disk_serial_number identifier |
237
238 Metrics:
239
240 | Metric | Dimensions | Unit |
241 |:-------|:-----------|:-----|
242 | as400.disk_busy | busy | percentage |
243 | as400.disk_io_requests | read, write | requests/s |
244 | as400.disk_space_usage | used | percentage |
245 | as400.disk_capacity | available, used | gigabytes |
246 | as400.disk_blocks | read, write | blocks/s |
247 | as400.disk_ssd_health | life_remaining | percentage |
248 | as400.disk_ssd_age | power_on_days | days |
249
250 ### Per httpserver
251
252 These metrics refer to individual httpserver instances.
253
254 Labels:
255
256 | Label | Description |
257 |:------|:------------|
258 | server | Server identifier |
259 | function | Function identifier |
260
261 Metrics:
262
263 | Metric | Dimensions | Unit |
264 |:-------|:-----------|:-----|
265 | as400.http_server_connections | normal, ssl | connections |
266 | as400.http_server_threads | active, idle | threads |
267 | as400.http_server_requests | requests, responses, rejected | requests/s |
268 | as400.http_server_bytes | received, sent | bytes/s |
269
270 ### Per jobqueue
271
272 These metrics refer to individual jobqueue instances.
273
274 Labels:
275
276 | Label | Description |
277 |:------|:------------|
278 | job_queue | Job_queue identifier |
279 | library | Library identifier |
280 | status | Status identifier |
281
282 Metrics:
283
284 | Metric | Dimensions | Unit |
285 |:-------|:-----------|:-----|
286 | as400.jobqueue_length | jobs | jobs |
287
288 ### Per messagequeue
289
290 These metrics refer to individual messagequeue instances.
291
292 Labels:
293
294 | Label | Description |
295 |:------|:------------|
296 | library | Library identifier |
297 | queue | Queue identifier |
298
299 Metrics:
300
301 | Metric | Dimensions | Unit |
302 |:-------|:-----------|:-----|
303 | as400.message_queue_messages | total, informational, inquiry, diagnostic, escape, notify, sender_copy | messages |
304 | as400.message_queue_severity | max | severity |
305
306 ### Per networkinterface
307
308 These metrics refer to individual networkinterface instances.
309
310 Labels:
311
312 | Label | Description |
313 |:------|:------------|
314 | interface | Interface identifier |
315 | interface_type | Interface_type identifier |
316 | connection_type | Connection_type identifier |
317 | internet_address | Internet_address identifier |
318 | network_address | Network_address identifier |
319 | subnet_mask | Subnet_mask identifier |
320
321 Metrics:
322
323 | Metric | Dimensions | Unit |
324 |:-------|:-----------|:-----|
325 | as400.network_interface_status | active | status |
326 | as400.network_interface_mtu | mtu | bytes |
327
328 ### Per outputqueue
329
330 These metrics refer to individual outputqueue instances.
331
332 Labels:
333
334 | Label | Description |
335 |:------|:------------|
336 | library | Library identifier |
337 | queue | Queue identifier |
338 | status | Status identifier |
339
340 Metrics:
341
342 | Metric | Dimensions | Unit |
343 |:-------|:-----------|:-----|
344 | as400.output_queue_files | files | files |
345 | as400.output_queue_writers | writers | writers |
346 | as400.output_queue_status | released | state |
347
348 ### Per plancache
349
350 These metrics refer to individual plancache instances.
351
352 Labels:
353
354 | Label | Description |
355 |:------|:------------|
356 | metric | Metric identifier |
357
358 Metrics:
359
360 | Metric | Dimensions | Unit |
361 |:-------|:-----------|:-----|
362 | as400.plan_cache_summary | value | value |
363
364 ### Per queueoverview
365
366 These metrics refer to individual queueoverview instances.
367
368 Labels:
369
370 | Label | Description |
371 |:------|:------------|
372 | queue_type | Queue_type identifier |
373 | item_type | Item_type identifier |
374
375 Metrics:
376
377 | Metric | Dimensions | Unit |
378 |:-------|:-----------|:-----|
379 | as400.queues_count | queues | queues |
380 | as400.queued_items | items | items |
381
382 ### Per subsystem
383
384 These metrics refer to individual subsystem instances.
385
386 Labels:
387
388 | Label | Description |
389 |:------|:------------|
390 | subsystem | Subsystem identifier |
391 | library | Library identifier |
392 | status | Status identifier |
393
394 Metrics:
395
396 | Metric | Dimensions | Unit |
397 |:-------|:-----------|:-----|
398 | as400.subsystem_jobs | active, maximum | jobs |
399
400 ### Per tempstoragebucket
401
402 These metrics refer to individual tempstoragebucket instances.
403
404 Labels:
405
406 | Label | Description |
407 |:------|:------------|
408 | bucket | Bucket identifier |
409
410 Metrics:
411
412 | Metric | Dimensions | Unit |
413 |:-------|:-----------|:-----|
414 | as400.temp_storage_bucket | current, peak | bytes |
415
416
417 ## Configuration
418
419 ### File
420
421 The configuration file name for this integration is `ibm.d/as400.conf`.
422
423 You can edit the configuration file using the `edit-config` script from the
424 Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).
425
426 ```bash
427 cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
428 sudo ./edit-config ibm.d/as400.conf
429 ```
430
431 ### Options
432
433 The following options can be defined globally or per job.
434
435 | Name | Description | Default | Required | Min | Max |
436 |:-----|:------------|:--------|:---------|:----|:----|
437 | update_every | Data collection frequency | `5` | no | 1 | - |
438 | Vnode | Vnode allows binding the collector to a virtual node. | `` | no | - | - |
439 | DSN | DSN provides a full IBM i ODBC connection string if manual override is needed. | `` | no | - | - |
440 | Timeout | Timeout controls how long to wait for SQL statements and RPCs. | `2000000000` | no | - | - |
441 | Hostname | Hostname is the remote IBM i host to monitor. | `` | no | - | - |
442 | Port | Port is the TCP port for the IBM i Access ODBC server. | `8471` | no | 1 | 65535 |
443 | Username | Username supplies the credentials used for authentication. | `` | no | - | - |
444 | Password | Password supplies the password used for authentication. | `` | no | - | - |
445 | Database | Database selects the IBM i database (library) to use when building the DSN. | `*SYSBAS` | no | - | - |
446 | ConnectionType | ConnectionType selects how the collector connects (currently only "odbc"). | `odbc` | no | - | - |
447 | ODBCDriver | ODBCDriver specifies the driver name registered on the host. | `IBM i Access ODBC Driver` | no | - | - |
448 | UseSSL | UseSSL enables TLS for the ODBC connection when supported by the driver. | `false` | no | - | - |
449 | ResetStatistics | ResetStatistics toggles destructive SQL services that reset system statistics on each query. | `false` | no | - | - |
450 | CollectDiskMetrics | CollectDiskMetrics toggles collection of disk unit statistics. | `auto` | no | - | - |
451 | CollectSubsystemMetrics | CollectSubsystemMetrics toggles collection of subsystem activity metrics. | `auto` | no | - | - |
452 | CollectActiveJobs | CollectActiveJobs toggles collection of detailed per-job metrics. | `auto` | no | - | - |
453 | CollectHTTPServerMetrics | CollectHTTPServerMetrics toggles collection of IBM HTTP Server statistics. | `auto` | no | - | - |
454 | CollectPlanCacheMetrics | CollectPlanCacheMetrics toggles collection of plan cache analysis metrics. | `auto` | no | - | - |
455 | CollectMessageQueueTotals | CollectMessageQueueTotals enables expensive aggregate counting across all message queues. | `auto` | no | - | - |
456 | CollectJobQueueTotals | CollectJobQueueTotals enables expensive aggregate counting across all job queues. | `auto` | no | - | - |
457 | CollectOutputQueueTotals | CollectOutputQueueTotals enables expensive aggregate counting across all output queues. | `auto` | no | - | - |
458 | SlowPath | SlowPath enables the asynchronous slow-path worker for heavy queries. | `true` | no | - | - |
459 | SlowPathUpdateEvery | SlowPathUpdateEvery controls the beat interval for the slow-path worker. | `10000000000` | no | - | - |
460 | SlowPathMaxConnections | SlowPathMaxConnections caps the number of concurrent queries the slow-path worker may run. | `1` | no | - | - |
461 | BatchPath | BatchPath enables the long-period batch worker for expensive queue aggregates. | `false` | no | - | - |
462 | BatchPathUpdateEvery | BatchPathUpdateEvery controls the beat interval for the batch worker. | `60000000000` | no | - | - |
463 | BatchPathMaxConnections | BatchPathMaxConnections caps concurrent queries for the batch worker. | `1` | no | - | - |
464 | MaxDisks | MaxDisks caps how many disk units may be charted. | `100` | no | - | - |
465 | MaxSubsystems | MaxSubsystems caps how many subsystems may be charted. | `100` | no | - | - |
466 | DiskSelector | DiskSelector filters disk units by name using glob-style patterns. | `` | no | - | - |
467 | SubsystemSelector | SubsystemSelector filters subsystems by name using glob-style patterns. | `` | no | - | - |
468 | ActiveJobs | ActiveJobs lists active jobs to monitor, using fully-qualified job identifiers (JOB_NUMBER/USER/JOB_NAME). When empty, active job collection is disabled. | `nil` | no | - | - |
469 | MessageQueues | MessageQueues lists message queues to collect, formatted as LIBRARY/QUEUE strings. When empty, message queue collection is disabled. The default configuration monitors QSYS/QSYSOPR, QSYS/QSYSMSG, and QSYS/QHST. | `[QSYS/QSYSOPR QSYS/QSYSMSG QSYS/QHST]` | no | - | - |
470 | JobQueues | JobQueues lists job queues to collect, formatted as LIBRARY/QUEUE strings. When empty, job queue collection is disabled. | `nil` | no | - | - |
471 | OutputQueues | OutputQueues lists output queues to collect, formatted as LIBRARY/QUEUE strings. When empty, output queue collection is disabled. | `nil` | no | - | - |
472
473 ### Examples
474
475 #### Basic configuration
476
477 IBM i (AS/400) monitoring with default settings.
478
479 <details>
480 <summary>Config</summary>
481
482 ```yaml
483 jobs:
484 - name: local
485 endpoint: dummy://localhost
486 ```
487
488 </details>
489
490 ## Troubleshooting
491
492 ### Debug Mode
493
494 To troubleshoot issues with the `as400` collector, run the `ibm.d.plugin` with the debug option enabled.
495 The output should give you clues as to why the collector isn't working.
496
497 - Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`
498 - Switch to the `netdata` user
499 - Run the `ibm.d.plugin` to debug the collector:
500
501 ```bash
502 sudo -u netdata ./ibm.d.plugin -d -m as400
503 ```
504
505 ## Getting Logs
506
507 If you're encountering problems with the `as400` collector, follow these steps to retrieve logs and identify potential issues:
508
509 - **Run the command** specific to your system (systemd, non-systemd, or Docker container).
510 - **Examine the output** for any warnings or error messages that might indicate issues. These messages will typically provide clues about the root cause of the problem.
511
512 ### For systemd systems (most Linux distributions)
513
514 ```bash
515 sudo journalctl -u netdata --reverse | grep as400
516 ```
517
518 ### For non-systemd systems
519
520 ```bash
521 sudo grep as400 /var/log/netdata/error.log
522 sudo grep as400 /var/log/netdata/collector.log
523 ```
524
525 ### For Docker containers
526
527 ```bash
528 sudo docker logs netdata 2>&1 | grep as400
529 ```