@cryptotaxi247 / netdata-1 / commits / 9bd7528db

chore(python.d): remove python.d/* announced in v1.36.0 deprecation notice (#13503)

Ilya Mashchenko committed Aug 11, 2022 at 19:23 UTC 9bd7528dbc3a9462899b1abf4549c31ba541a67f

10 files changed +8 -1883

collectors/python.d.plugin/Makefile.am

-1

@@ -73,7 +73,6 @@ include ntpd/Makefile.inc

73 include openldap/Makefile.inc

74 include oracledb/Makefile.inc

75 include postfix/Makefile.inc

76 -include postgres/Makefile.inc

76 include proxysql/Makefile.inc

77 include puppet/Makefile.inc

78 include rabbitmq/Makefile.inc

collectors/python.d.plugin/postgres/Makefile.inc deleted

-13

@@ -1,13 +0,0 @@

1 -# SPDX-License-Identifier: GPL-3.0-or-later

2 -

3 -# THIS IS NOT A COMPLETE Makefile

4 -# IT IS INCLUDED BY ITS PARENT'S Makefile.am

5 -# IT IS REQUIRED TO REFERENCE ALL FILES RELATIVE TO THE PARENT

6 -

7 -# install these files

8 -dist_python_DATA += postgres/postgres.chart.py

9 -dist_pythonconfig_DATA += postgres/postgres.conf

10 -

11 -# do not install these files, but include them in the distribution

12 -dist_noinst_DATA += postgres/README.md postgres/Makefile.inc

13 -

collectors/python.d.plugin/postgres/README.md deleted

-145

@@ -1,145 +0,0 @@

1 -<!--

2 -title: "PostgreSQL monitoring with Netdata"

3 -custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/python.d.plugin/postgres/README.md

4 -sidebar_label: "PostgreSQL"

5 --->

6 -

7 -# PostgreSQL monitoring with Netdata

8 -

9 -> **Warning**: This module is deprecated and will be deleted in v1.37.0.

10 -> Use [go.d/postgres](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/postgres).

11 -

12 -Collects database health and performance metrics.

13 -

14 -## Requirements

15 -

16 -- `python-psycopg2` package. You have to install it manually and make sure that it is available to the `netdata` user, either using `pip`, the package manager of your Linux distribution, or any other method you prefer.

17 -

18 -- PostgreSQL v9.4+

19 -

20 -Following charts are drawn:

21 -

22 -1. **Database size** MB

23 -

24 - - size

25 -

26 -2. **Current Backend Processes** processes

27 -

28 - - active

29 -

30 -3. **Current Backend Process Usage** percentage

31 -

32 - - used

33 - - available

34 -

35 -4. **Write-Ahead Logging Statistics** files/s

36 -

37 - - total

38 - - ready

39 - - done

40 -

41 -5. **Checkpoints** writes/s

42 -

43 - - scheduled

44 - - requested

45 -

46 -6. **Current connections to db** count

47 -

48 - - connections

49 -

50 -7. **Tuples returned from db** tuples/s

51 -

52 - - sequential

53 - - bitmap

54 -

55 -8. **Tuple reads from db** reads/s

56 -

57 - - disk

58 - - cache

59 -

60 -9. **Transactions on db** transactions/s

61 -

62 - - committed

63 - - rolled back

64 -

65 -10. **Tuples written to db** writes/s

66 -

67 - - inserted

68 - - updated

69 - - deleted

70 - - conflicts

71 -

72 -11. **Locks on db** count per type

73 -

74 - - locks

75 -

76 -12. **Standby delta** KB

77 -

78 - - sent delta

79 - - write delta

80 - - flush delta

81 - - replay delta

82 -

83 -13. **Standby lag** seconds

84 -

85 - - write lag

86 - - flush lag

87 - - replay lag

88 -

89 -14. **Average number of blocking transactions in db** processes

90 -

91 - - blocking

92 -

93 -## Configuration

94 -

95 -Edit the `python.d/postgres.conf` configuration file using `edit-config` from the Netdata [config

96 -directory](/docs/configure/nodes.md), which is typically at `/etc/netdata`.

97 -

98 -```bash

99 -cd /etc/netdata # Replace this path with your Netdata config directory, if different

100 -sudo ./edit-config python.d/postgres.conf

101 -```

102 -

103 -When no configuration file is found, the module tries to connect to TCP/IP socket: `localhost:5432` with the

104 -following collection jobs.

105 -

106 -```yaml

107 -socket:

108 - name : 'socket'

109 - user : 'postgres'

110 - database : 'postgres'

111 -

112 -tcp:

113 - name : 'tcp'

114 - user : 'postgres'

115 - database : 'postgres'

116 - host : 'localhost'

117 - port : 5432

118 -```

119 -

120 -**Note**: Every job collection must have a unique identifier. In cases that you monitor multiple DBs, every

121 -job must have it's own name. Use a mnemonic of your preference (e.g us_east_db, us_east_tcp)

122 -

123 -## Troubleshooting

124 -

125 -To troubleshoot issues with the `postgres` collector, run the `python.d.plugin` with the debug option enabled. The output

126 -should give you clues as to why the collector isn't working.

127 -

128 -First, navigate to your plugins directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on your

129 -system, open `netdata.conf` and look for the setting `plugins directory`. Once you're in the plugin's directory, switch

130 -to the `netdata` user.

131 -

132 -```bash

133 -cd /usr/libexec/netdata/plugins.d/

134 -sudo su -s /bin/bash netdata

135 -```

136 -

137 -You can now run the `python.d.plugin` to debug the collector:

138 -

139 -```bash

140 -./python.d.plugin postgres debug trace

141 -```

142 -

143 ----

144 -

145 -

collectors/python.d.plugin/postgres/postgres.chart.py deleted

-1436

@@ -1,1436 +0,0 @@

1 -# -*- coding: utf-8 -*-

2 -# Description: example netdata python.d module

3 -# Authors: facetoe, dangtranhoang

4 -# SPDX-License-Identifier: GPL-3.0-or-later

5 -

6 -from copy import deepcopy

7 -

8 -try:

9 - import psycopg2

10 - from psycopg2 import extensions

11 - from psycopg2.extras import DictCursor

12 - from psycopg2 import OperationalError

13 -

14 - PSYCOPG2 = True

15 -except ImportError:

16 - PSYCOPG2 = False

17 -

18 -from bases.FrameworkServices.SimpleService import SimpleService

19 -

20 -DEFAULT_PORT = 5432

21 -DEFAULT_USER = 'postgres'

22 -DEFAULT_CONNECT_TIMEOUT = 2 # seconds

23 -DEFAULT_STATEMENT_TIMEOUT = 5000 # ms

24 -

25 -CONN_PARAM_DSN = 'dsn'

26 -CONN_PARAM_HOST = 'host'

27 -CONN_PARAM_PORT = 'port'

28 -CONN_PARAM_DATABASE = 'database'

29 -CONN_PARAM_USER = 'user'

30 -CONN_PARAM_PASSWORD = 'password'

31 -CONN_PARAM_CONN_TIMEOUT = 'connect_timeout'

32 -CONN_PARAM_STATEMENT_TIMEOUT = 'statement_timeout'

33 -CONN_PARAM_SSL_MODE = 'sslmode'

34 -CONN_PARAM_SSL_ROOT_CERT = 'sslrootcert'

35 -CONN_PARAM_SSL_CRL = 'sslcrl'

36 -CONN_PARAM_SSL_CERT = 'sslcert'

37 -CONN_PARAM_SSL_KEY = 'sslkey'

38 -

39 -QUERY_NAME_WAL = 'WAL'

40 -QUERY_NAME_ARCHIVE = 'ARCHIVE'

41 -QUERY_NAME_BACKENDS = 'BACKENDS'

42 -QUERY_NAME_BACKEND_USAGE = 'BACKEND_USAGE'

43 -QUERY_NAME_TABLE_STATS = 'TABLE_STATS'

44 -QUERY_NAME_INDEX_STATS = 'INDEX_STATS'

45 -QUERY_NAME_DATABASE = 'DATABASE'

46 -QUERY_NAME_BGWRITER = 'BGWRITER'

47 -QUERY_NAME_LOCKS = 'LOCKS'

48 -QUERY_NAME_BLOCKERS = 'BLOCKERS'

49 -QUERY_NAME_DATABASES = 'DATABASES'

50 -QUERY_NAME_STANDBY = 'STANDBY'

51 -QUERY_NAME_REPLICATION_SLOT = 'REPLICATION_SLOT'

52 -QUERY_NAME_STANDBY_DELTA = 'STANDBY_DELTA'

53 -QUERY_NAME_STANDBY_LAG = 'STANDBY_LAG'

54 -QUERY_NAME_REPSLOT_FILES = 'REPSLOT_FILES'

55 -QUERY_NAME_IF_SUPERUSER = 'IF_SUPERUSER'

56 -QUERY_NAME_SERVER_VERSION = 'SERVER_VERSION'

57 -QUERY_NAME_AUTOVACUUM = 'AUTOVACUUM'

58 -QUERY_NAME_FORCED_AUTOVACUUM = 'FORCED_AUTOVACUUM'

59 -QUERY_NAME_TX_WRAPAROUND = 'TX_WRAPAROUND'

60 -QUERY_NAME_DIFF_LSN = 'DIFF_LSN'

61 -QUERY_NAME_WAL_WRITES = 'WAL_WRITES'

62 -

63 -METRICS = {

64 - QUERY_NAME_DATABASE: [

65 - 'connections',

66 - 'xact_commit',

67 - 'xact_rollback',

68 - 'blks_read',

69 - 'blks_hit',

70 - 'tup_returned',

71 - 'tup_fetched',

72 - 'tup_inserted',

73 - 'tup_updated',

74 - 'tup_deleted',

75 - 'conflicts',

76 - 'temp_files',

77 - 'temp_bytes',

78 - 'size'

79 - ],

80 - QUERY_NAME_BACKENDS: [

81 - 'backends_active',

82 - 'backends_idle'

83 - ],

84 - QUERY_NAME_BACKEND_USAGE: [

85 - 'available',

86 - 'used'

87 - ],

88 - QUERY_NAME_INDEX_STATS: [

89 - 'index_count',

90 - 'index_size'

91 - ],

92 - QUERY_NAME_TABLE_STATS: [

93 - 'table_size',

94 - 'table_count'

95 - ],

96 - QUERY_NAME_WAL: [

97 - 'written_wal',

98 - 'recycled_wal',

99 - 'total_wal'

100 - ],

101 - QUERY_NAME_WAL_WRITES: [

102 - 'wal_writes'

103 - ],

104 - QUERY_NAME_ARCHIVE: [

105 - 'ready_count',

106 - 'done_count',

107 - 'file_count'

108 - ],

109 - QUERY_NAME_BGWRITER: [

110 - 'checkpoint_scheduled',

111 - 'checkpoint_requested',

112 - 'buffers_checkpoint',

113 - 'buffers_clean',

114 - 'maxwritten_clean',

115 - 'buffers_backend',

116 - 'buffers_alloc',

117 - 'buffers_backend_fsync'

118 - ],

119 - QUERY_NAME_LOCKS: [

120 - 'ExclusiveLock',

121 - 'RowShareLock',

122 - 'SIReadLock',

123 - 'ShareUpdateExclusiveLock',

124 - 'AccessExclusiveLock',

125 - 'AccessShareLock',

126 - 'ShareRowExclusiveLock',

127 - 'ShareLock',

128 - 'RowExclusiveLock'

129 - ],

130 - QUERY_NAME_BLOCKERS: [

131 - 'blocking_pids_avg'

132 - ],

133 - QUERY_NAME_AUTOVACUUM: [

134 - 'analyze',

135 - 'vacuum_analyze',

136 - 'vacuum',

137 - 'vacuum_freeze',

138 - 'brin_summarize'

139 - ],

140 - QUERY_NAME_FORCED_AUTOVACUUM: [

141 - 'percent_towards_forced_vacuum'

142 - ],

143 - QUERY_NAME_TX_WRAPAROUND: [

144 - 'oldest_current_xid',

145 - 'percent_towards_wraparound'

146 - ],

147 - QUERY_NAME_STANDBY_DELTA: [

148 - 'sent_delta',

149 - 'write_delta',

150 - 'flush_delta',

151 - 'replay_delta'

152 - ],

153 - QUERY_NAME_STANDBY_LAG: [

154 - 'write_lag',

155 - 'flush_lag',

156 - 'replay_lag'

157 - ],

158 - QUERY_NAME_REPSLOT_FILES: [

159 - 'replslot_wal_keep',

160 - 'replslot_files'

161 - ]

162 -}

163 -

164 -NO_VERSION = 0

165 -DEFAULT = 'DEFAULT'

166 -V72 = 'V72'

167 -V82 = 'V82'

168 -V91 = 'V91'

169 -V92 = 'V92'

170 -V96 = 'V96'

171 -V10 = 'V10'

172 -V11 = 'V11'

173 -

174 -QUERY_WAL = {

175 - DEFAULT: """

176 -SELECT

177 - count(*) as total_wal,

178 - count(*) FILTER (WHERE type = 'recycled') AS recycled_wal,

179 - count(*) FILTER (WHERE type = 'written') AS written_wal

180 -FROM

181 - (SELECT

182 - wal.name,

183 - pg_walfile_name(

184 - CASE pg_is_in_recovery()

185 - WHEN true THEN NULL

186 - ELSE pg_current_wal_lsn()

187 - END ),

188 - CASE

189 - WHEN wal.name > pg_walfile_name(

190 - CASE pg_is_in_recovery()

191 - WHEN true THEN NULL

192 - ELSE pg_current_wal_lsn()

193 - END ) THEN 'recycled'

194 - ELSE 'written'

195 - END AS type

196 - FROM pg_catalog.pg_ls_dir('pg_wal') AS wal(name)

197 - WHERE name ~ '^[0-9A-F]{24}$'

198 - ORDER BY

199 - (pg_stat_file('pg_wal/'||name, true)).modification,

200 - wal.name DESC) sub;

201 -""",

202 - V96: """

203 -SELECT

204 - count(*) as total_wal,

205 - count(*) FILTER (WHERE type = 'recycled') AS recycled_wal,

206 - count(*) FILTER (WHERE type = 'written') AS written_wal

207 -FROM

208 - (SELECT

209 - wal.name,

210 - pg_xlogfile_name(

211 - CASE pg_is_in_recovery()

212 - WHEN true THEN NULL

213 - ELSE pg_current_xlog_location()

214 - END ),

215 - CASE

216 - WHEN wal.name > pg_xlogfile_name(

217 - CASE pg_is_in_recovery()

218 - WHEN true THEN NULL

219 - ELSE pg_current_xlog_location()

220 - END ) THEN 'recycled'

221 - ELSE 'written'

222 - END AS type

223 - FROM pg_catalog.pg_ls_dir('pg_xlog') AS wal(name)

224 - WHERE name ~ '^[0-9A-F]{24}$'

225 - ORDER BY

226 - (pg_stat_file('pg_xlog/'||name, true)).modification,

227 - wal.name DESC) sub;

228 -""",

229 -}

230 -

231 -QUERY_ARCHIVE = {

232 - DEFAULT: """

233 -SELECT

234 - CAST(COUNT(*) AS INT) AS file_count,

235 - CAST(COALESCE(SUM(CAST(archive_file ~ $r$\.ready$$r$ as INT)),0) AS INT) AS ready_count,

236 - CAST(COALESCE(SUM(CAST(archive_file ~ $r$\.done$$r$ AS INT)),0) AS INT) AS done_count

237 -FROM

238 - pg_catalog.pg_ls_dir('pg_wal/archive_status') AS archive_files (archive_file);

239 -""",

240 - V96: """

241 -SELECT

242 - CAST(COUNT(*) AS INT) AS file_count,

243 - CAST(COALESCE(SUM(CAST(archive_file ~ $r$\.ready$$r$ as INT)),0) AS INT) AS ready_count,

244 - CAST(COALESCE(SUM(CAST(archive_file ~ $r$\.done$$r$ AS INT)),0) AS INT) AS done_count

245 -FROM

246 - pg_catalog.pg_ls_dir('pg_xlog/archive_status') AS archive_files (archive_file);

247 -

248 -""",

249 -}

250 -

251 -QUERY_BACKEND = {

252 - DEFAULT: """

253 -SELECT

254 - count(*) - (SELECT count(*)

255 - FROM pg_stat_activity

256 - WHERE state = 'idle')

257 - AS backends_active,

258 - (SELECT count(*)

259 - FROM pg_stat_activity

260 - WHERE state = 'idle')

261 - AS backends_idle

262 -FROM pg_stat_activity;

263 -""",

264 -}

265 -

266 -QUERY_BACKEND_USAGE = {

267 - DEFAULT: """

268 -SELECT

269 - COUNT(1) as used,

270 - current_setting('max_connections')::int - current_setting('superuser_reserved_connections')::int

271 - - COUNT(1) AS available

272 -FROM pg_catalog.pg_stat_activity

273 -WHERE backend_type IN ('client backend', 'background worker');

274 -""",

275 - V10: """

276 -SELECT

277 - SUM(s.conn) as used,

278 - current_setting('max_connections')::int - current_setting('superuser_reserved_connections')::int

279 - - SUM(s.conn) AS available

280 -FROM (

281 - SELECT 's' as type, COUNT(1) as conn

282 - FROM pg_catalog.pg_stat_activity

283 - WHERE backend_type IN ('client backend', 'background worker')

284 - UNION ALL

285 - SELECT 'r', COUNT(1)

286 - FROM pg_catalog.pg_stat_replication

287 -) as s;

288 -""",

289 - V92: """

290 -SELECT

291 - SUM(s.conn) as used,

292 - current_setting('max_connections')::int - current_setting('superuser_reserved_connections')::int

293 - - SUM(s.conn) AS available

294 -FROM (

295 - SELECT 's' as type, COUNT(1) as conn

296 - FROM pg_catalog.pg_stat_activity

297 - WHERE query NOT LIKE 'autovacuum: %%'

298 - UNION ALL

299 - SELECT 'r', COUNT(1)

300 - FROM pg_catalog.pg_stat_replication

301 -) as s;

302 -""",

303 - V91: """

304 -SELECT

305 - SUM(s.conn) as used,

306 - current_setting('max_connections')::int - current_setting('superuser_reserved_connections')::int

307 - - SUM(s.conn) AS available

308 -FROM (

309 - SELECT 's' as type, COUNT(1) as conn

310 - FROM pg_catalog.pg_stat_activity

311 - WHERE current_query NOT LIKE 'autovacuum: %%'

312 - UNION ALL

313 - SELECT 'r', COUNT(1)

314 - FROM pg_catalog.pg_stat_replication

315 -) as s;

316 -""",

317 - V82: """

318 -SELECT

319 - COUNT(1) as used,

320 - current_setting('max_connections')::int - current_setting('superuser_reserved_connections')::int

321 - - COUNT(1) AS available

322 -FROM pg_catalog.pg_stat_activity

323 -WHERE current_query NOT LIKE 'autovacuum: %%';

324 -""",

325 - V72: """

326 -SELECT

327 - COUNT(1) as used,

328 - current_setting('max_connections')::int - current_setting('superuser_reserved_connections')::int

329 - - COUNT(1) AS available

330 -FROM pg_catalog.pg_stat_activity s

331 -JOIN pg_catalog.pg_database d ON d.oid = s.datid

332 -WHERE d.datallowconn;

333 -""",

334 -}

335 -

336 -QUERY_TABLE_STATS = {

337 - DEFAULT: """

338 -SELECT

339 - sum(relpages) * current_setting('block_size')::numeric AS table_size,

340 - count(1) AS table_count

341 -FROM pg_class

342 -WHERE relkind IN ('r', 't', 'm');

343 -""",

344 -}

345 -

346 -QUERY_INDEX_STATS = {

347 - DEFAULT: """

348 -SELECT

349 - sum(relpages) * current_setting('block_size')::numeric AS index_size,

350 - count(1) AS index_count

351 -FROM pg_class

352 -WHERE relkind = 'i';

353 -""",

354 -}

355 -

356 -QUERY_DATABASE = {

357 - DEFAULT: """

358 -SELECT

359 - datname AS database_name,

360 - numbackends AS connections,

361 - xact_commit AS xact_commit,

362 - xact_rollback AS xact_rollback,

363 - blks_read AS blks_read,

364 - blks_hit AS blks_hit,

365 - tup_returned AS tup_returned,

366 - tup_fetched AS tup_fetched,

367 - tup_inserted AS tup_inserted,

368 - tup_updated AS tup_updated,

369 - tup_deleted AS tup_deleted,

370 - conflicts AS conflicts,

371 - pg_database_size(datname) AS size,

372 - temp_files AS temp_files,

373 - temp_bytes AS temp_bytes

374 -FROM pg_stat_database

375 -WHERE datname IN %(databases)s ;

376 -""",

377 -}

378 -

379 -QUERY_BGWRITER = {

380 - DEFAULT: """

381 -SELECT

382 - checkpoints_timed AS checkpoint_scheduled,

383 - checkpoints_req AS checkpoint_requested,

384 - buffers_checkpoint * current_setting('block_size')::numeric buffers_checkpoint,

385 - buffers_clean * current_setting('block_size')::numeric buffers_clean,

386 - maxwritten_clean,

387 - buffers_backend * current_setting('block_size')::numeric buffers_backend,

388 - buffers_alloc * current_setting('block_size')::numeric buffers_alloc,

389 - buffers_backend_fsync

390 -FROM pg_stat_bgwriter;

391 -""",

392 -}

393 -

394 -QUERY_LOCKS = {

395 - DEFAULT: """

396 -SELECT

397 - pg_database.datname as database_name,

398 - mode,

399 - count(mode) AS locks_count

400 -FROM pg_locks

401 -INNER JOIN pg_database

402 - ON pg_database.oid = pg_locks.database

403 -GROUP BY datname, mode

404 -ORDER BY datname, mode;

405 -""",

406 -}

407 -

408 -QUERY_BLOCKERS = {

409 - DEFAULT: """

410 -WITH B AS (

411 -SELECT DISTINCT

412 - pg_database.datname as database_name,

413 - pg_locks.pid,

414 - cardinality(pg_blocking_pids(pg_locks.pid)) AS blocking_pids

415 -FROM pg_locks

416 -INNER JOIN pg_database ON pg_database.oid = pg_locks.database

417 -WHERE NOT pg_locks.granted)

418 -SELECT database_name, AVG(blocking_pids) AS blocking_pids_avg

419 -FROM B

420 -GROUP BY database_name

421 -""",

422 - V96: """

423 -WITH B AS (

424 -SELECT DISTINCT

425 - pg_database.datname as database_name,

426 - blocked_locks.pid AS blocked_pid,

427 - COUNT(blocking_locks.pid) AS blocking_pids

428 -FROM pg_catalog.pg_locks blocked_locks

429 -INNER JOIN pg_database ON pg_database.oid = blocked_locks.database

430 -JOIN pg_catalog.pg_locks blocking_locks

431 - ON blocking_locks.locktype = blocked_locks.locktype

432 - AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database

433 - AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation

434 - AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page

435 - AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple

436 - AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid

437 - AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid

438 - AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid

439 - AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid

440 - AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid

441 - AND blocking_locks.pid != blocked_locks.pid

442 -WHERE NOT blocked_locks.GRANTED

443 -GROUP BY database_name, blocked_pid)

444 -SELECT database_name, AVG(blocking_pids) AS blocking_pids_avg

445 -FROM B

446 -GROUP BY database_name

447 -"""

448 -}

449 -

450 -QUERY_DATABASES = {

451 - DEFAULT: """

452 -SELECT

453 - datname

454 -FROM pg_stat_database

455 -WHERE

456 - has_database_privilege(

457 - (SELECT current_user), datname, 'connect')

458 - AND NOT datname ~* '^template\d'

459 -ORDER BY datname;

460 -""",

461 -}

462 -

463 -QUERY_STANDBY = {

464 - DEFAULT: """

465 -SELECT

466 - COALESCE(prs.slot_name, psr.application_name) application_name

467 -FROM pg_stat_replication psr

468 -LEFT OUTER JOIN pg_replication_slots prs on psr.pid = prs.active_pid

469 -WHERE application_name IS NOT NULL;

470 -""",

471 -}

472 -

473 -QUERY_REPLICATION_SLOT = {

474 - DEFAULT: """

475 -SELECT slot_name

476 -FROM pg_replication_slots;

477 -"""

478 -}

479 -

480 -QUERY_STANDBY_DELTA = {

481 - DEFAULT: """

482 -SELECT

483 - COALESCE(prs.slot_name, psr.application_name) application_name,

484 - pg_wal_lsn_diff(

485 - CASE pg_is_in_recovery()

486 - WHEN true THEN pg_last_wal_receive_lsn()

487 - ELSE pg_current_wal_lsn()

488 - END,

489 - sent_lsn) AS sent_delta,

490 - pg_wal_lsn_diff(

491 - CASE pg_is_in_recovery()

492 - WHEN true THEN pg_last_wal_receive_lsn()

493 - ELSE pg_current_wal_lsn()

494 - END,

495 - write_lsn) AS write_delta,

496 - pg_wal_lsn_diff(

497 - CASE pg_is_in_recovery()

498 - WHEN true THEN pg_last_wal_receive_lsn()

499 - ELSE pg_current_wal_lsn()

500 - END,

501 - flush_lsn) AS flush_delta,

502 - pg_wal_lsn_diff(

503 - CASE pg_is_in_recovery()

504 - WHEN true THEN pg_last_wal_receive_lsn()

505 - ELSE pg_current_wal_lsn()

506 - END,

507 - replay_lsn) AS replay_delta

508 -FROM pg_stat_replication psr

509 -LEFT OUTER JOIN pg_replication_slots prs on psr.pid = prs.active_pid

510 -WHERE application_name IS NOT NULL;

511 -""",

512 - V96: """

513 -SELECT

514 - COALESCE(prs.slot_name, psr.application_name) application_name,

515 - pg_xlog_location_diff(

516 - CASE pg_is_in_recovery()

517 - WHEN true THEN pg_last_xlog_receive_location()

518 - ELSE pg_current_xlog_location()

519 - END,

520 - sent_location) AS sent_delta,

521 - pg_xlog_location_diff(

522 - CASE pg_is_in_recovery()

523 - WHEN true THEN pg_last_xlog_receive_location()

524 - ELSE pg_current_xlog_location()

525 - END,

526 - write_location) AS write_delta,

527 - pg_xlog_location_diff(

528 - CASE pg_is_in_recovery()

529 - WHEN true THEN pg_last_xlog_receive_location()

530 - ELSE pg_current_xlog_location()

531 - END,

532 - flush_location) AS flush_delta,

533 - pg_xlog_location_diff(

534 - CASE pg_is_in_recovery()

535 - WHEN true THEN pg_last_xlog_receive_location()

536 - ELSE pg_current_xlog_location()

537 - END,

538 - replay_location) AS replay_delta

539 -FROM pg_stat_replication psr

540 -LEFT OUTER JOIN pg_replication_slots prs on psr.pid = prs.active_pid

541 -WHERE application_name IS NOT NULL;

542 -""",

543 -}

544 -

545 -QUERY_STANDBY_LAG = {

546 - DEFAULT: """

547 -SELECT

548 - COALESCE(prs.slot_name, psr.application_name) application_name,

549 - COALESCE(EXTRACT(EPOCH FROM write_lag)::bigint, 0) AS write_lag,

550 - COALESCE(EXTRACT(EPOCH FROM flush_lag)::bigint, 0) AS flush_lag,

551 - COALESCE(EXTRACT(EPOCH FROM replay_lag)::bigint, 0) AS replay_lag

552 -FROM pg_stat_replication psr

553 -LEFT OUTER JOIN pg_replication_slots prs on psr.pid = prs.active_pid

554 -WHERE application_name IS NOT NULL;

555 -"""

556 -}

557 -

558 -QUERY_REPSLOT_FILES = {

559 - DEFAULT: """

560 -WITH wal_size AS (

561 - SELECT

562 - setting::int AS val

563 - FROM pg_settings

564 - WHERE name = 'wal_segment_size'

565 - )

566 -SELECT

567 - slot_name,

568 - slot_type,

569 - replslot_wal_keep,

570 - count(slot_file) AS replslot_files

571 -FROM

572 - (SELECT

573 - slot.slot_name,

574 - CASE

575 - WHEN slot_file <> 'state' THEN 1

576 - END AS slot_file ,

577 - slot_type,

578 - COALESCE (

579 - floor(

580 - CASE WHEN pg_is_in_recovery()

581 - THEN (

582 - pg_wal_lsn_diff(pg_last_wal_receive_lsn(), slot.restart_lsn)

583 - -- this is needed to account for whole WAL retention and

584 - -- not only size retention

585 - + (pg_wal_lsn_diff(restart_lsn, '0/0') %% s.val)

586 - ) / s.val

587 - ELSE (

588 - pg_wal_lsn_diff(pg_current_wal_lsn(), slot.restart_lsn)

589 - -- this is needed to account for whole WAL retention and

590 - -- not only size retention

591 - + (pg_walfile_name_offset(restart_lsn)).file_offset

592 - ) / s.val

593 - END

594 - ),0) AS replslot_wal_keep

595 - FROM pg_replication_slots slot

596 - LEFT JOIN (

597 - SELECT

598 - slot2.slot_name,

599 - pg_ls_dir('pg_replslot/' || slot2.slot_name) AS slot_file

600 - FROM pg_replication_slots slot2

601 - ) files (slot_name, slot_file)

602 - ON slot.slot_name = files.slot_name

603 - CROSS JOIN wal_size s

604 - ) AS d

605 -GROUP BY

606 - slot_name,

607 - slot_type,

608 - replslot_wal_keep;

609 -""",

610 - V10: """

611 -WITH wal_size AS (

612 - SELECT

613 - current_setting('wal_block_size')::INT * setting::INT AS val

614 - FROM pg_settings

615 - WHERE name = 'wal_segment_size'

616 - )

617 -SELECT

618 - slot_name,

619 - slot_type,

620 - replslot_wal_keep,

621 - count(slot_file) AS replslot_files

622 -FROM

623 - (SELECT

624 - slot.slot_name,

625 - CASE

626 - WHEN slot_file <> 'state' THEN 1

627 - END AS slot_file ,

628 - slot_type,

629 - COALESCE (

630 - floor(

631 - CASE WHEN pg_is_in_recovery()

632 - THEN (

633 - pg_wal_lsn_diff(pg_last_wal_receive_lsn(), slot.restart_lsn)

634 - -- this is needed to account for whole WAL retention and

635 - -- not only size retention

636 - + (pg_wal_lsn_diff(restart_lsn, '0/0') %% s.val)

637 - ) / s.val

638 - ELSE (

639 - pg_wal_lsn_diff(pg_current_wal_lsn(), slot.restart_lsn)

640 - -- this is needed to account for whole WAL retention and

641 - -- not only size retention

642 - + (pg_walfile_name_offset(restart_lsn)).file_offset

643 - ) / s.val

644 - END

645 - ),0) AS replslot_wal_keep

646 - FROM pg_replication_slots slot

647 - LEFT JOIN (

648 - SELECT

649 - slot2.slot_name,

650 - pg_ls_dir('pg_replslot/' || slot2.slot_name) AS slot_file

651 - FROM pg_replication_slots slot2

652 - ) files (slot_name, slot_file)

653 - ON slot.slot_name = files.slot_name

654 - CROSS JOIN wal_size s

655 - ) AS d

656 -GROUP BY

657 - slot_name,

658 - slot_type,

659 - replslot_wal_keep;

660 -""",

661 -}

662 -

663 -QUERY_SUPERUSER = {

664 - DEFAULT: """

665 -SELECT current_setting('is_superuser') = 'on' AS is_superuser;

666 -""",

667 -}

668 -

669 -QUERY_SHOW_VERSION = {

670 - DEFAULT: """

671 -SHOW server_version_num;

672 -""",

673 -}

674 -

675 -QUERY_AUTOVACUUM = {

676 - DEFAULT: """

677 -SELECT

678 - count(*) FILTER (WHERE query LIKE 'autovacuum: ANALYZE%%') AS analyze,

679 - count(*) FILTER (WHERE query LIKE 'autovacuum: VACUUM ANALYZE%%') AS vacuum_analyze,

680 - count(*) FILTER (WHERE query LIKE 'autovacuum: VACUUM%%'

681 - AND query NOT LIKE 'autovacuum: VACUUM ANALYZE%%'

682 - AND query NOT LIKE '%%to prevent wraparound%%') AS vacuum,

683 - count(*) FILTER (WHERE query LIKE '%%to prevent wraparound%%') AS vacuum_freeze,

684 - count(*) FILTER (WHERE query LIKE 'autovacuum: BRIN summarize%%') AS brin_summarize

685 -FROM pg_stat_activity

686 -WHERE query NOT LIKE '%%pg_stat_activity%%';

687 -""",

688 -}

689 -

690 -QUERY_FORCED_AUTOVACUUM = {

691 - DEFAULT: """

692 -WITH max_age AS (

693 - SELECT setting AS autovacuum_freeze_max_age

694 - FROM pg_catalog.pg_settings

695 - WHERE name = 'autovacuum_freeze_max_age' )

696 -, per_database_stats AS (

697 - SELECT datname

698 - , m.autovacuum_freeze_max_age::int

699 - , age(d.datfrozenxid) AS oldest_current_xid

700 - FROM pg_catalog.pg_database d

701 - JOIN max_age m ON (true)

702 - WHERE d.datallowconn )

703 -SELECT max(ROUND(100*(oldest_current_xid/autovacuum_freeze_max_age::float))) AS percent_towards_forced_autovacuum

704 -FROM per_database_stats;

705 -""",

706 -}

707 -

708 -QUERY_TX_WRAPAROUND = {

709 - DEFAULT: """

710 -WITH max_age AS (

711 - SELECT 2000000000 as max_old_xid

712 - FROM pg_catalog.pg_settings

713 - WHERE name = 'autovacuum_freeze_max_age' )

714 -, per_database_stats AS (

715 - SELECT datname

716 - , m.max_old_xid::int

717 - , age(d.datfrozenxid) AS oldest_current_xid

718 - FROM pg_catalog.pg_database d

719 - JOIN max_age m ON (true)

720 - WHERE d.datallowconn )

721 -SELECT max(oldest_current_xid) AS oldest_current_xid

722 - , max(ROUND(100*(oldest_current_xid/max_old_xid::float))) AS percent_towards_wraparound

723 -FROM per_database_stats;

724 -""",

725 -}

726 -

727 -QUERY_DIFF_LSN = {

728 - DEFAULT: """

729 -SELECT

730 - pg_wal_lsn_diff(

731 - CASE pg_is_in_recovery()

732 - WHEN true THEN pg_last_wal_receive_lsn()

733 - ELSE pg_current_wal_lsn()

734 - END,

735 - '0/0') as wal_writes ;

736 -""",

737 - V96: """

738 -SELECT

739 - pg_xlog_location_diff(

740 - CASE pg_is_in_recovery()

741 - WHEN true THEN pg_last_xlog_receive_location()

742 - ELSE pg_current_xlog_location()

743 - END,

744 - '0/0') as wal_writes ;

745 -""",

746 -}

747 -

748 -def query_factory(name, version=NO_VERSION):

749 - if name == QUERY_NAME_BACKENDS:

750 - return QUERY_BACKEND[DEFAULT]

751 - elif name == QUERY_NAME_BACKEND_USAGE:

752 - if version < 80200:

753 - return QUERY_BACKEND_USAGE[V72]

754 - if version < 90100:

755 - return QUERY_BACKEND_USAGE[V82]

756 - if version < 90200:

757 - return QUERY_BACKEND_USAGE[V91]

758 - if version < 100000:

759 - return QUERY_BACKEND_USAGE[V92]

760 - elif version < 120000:

761 - return QUERY_BACKEND_USAGE[V10]

762 - return QUERY_BACKEND_USAGE[DEFAULT]

763 - elif name == QUERY_NAME_TABLE_STATS:

764 - return QUERY_TABLE_STATS[DEFAULT]

765 - elif name == QUERY_NAME_INDEX_STATS:

766 - return QUERY_INDEX_STATS[DEFAULT]

767 - elif name == QUERY_NAME_DATABASE:

768 - return QUERY_DATABASE[DEFAULT]

769 - elif name == QUERY_NAME_BGWRITER:

770 - return QUERY_BGWRITER[DEFAULT]

771 - elif name == QUERY_NAME_LOCKS:

772 - return QUERY_LOCKS[DEFAULT]

773 - elif name == QUERY_NAME_BLOCKERS:

774 - if version < 90600:

775 - return QUERY_BLOCKERS[V96]

776 - return QUERY_BLOCKERS[DEFAULT]

777 - elif name == QUERY_NAME_DATABASES:

778 - return QUERY_DATABASES[DEFAULT]

779 - elif name == QUERY_NAME_STANDBY:

780 - return QUERY_STANDBY[DEFAULT]

781 - elif name == QUERY_NAME_REPLICATION_SLOT:

782 - return QUERY_REPLICATION_SLOT[DEFAULT]

783 - elif name == QUERY_NAME_IF_SUPERUSER:

784 - return QUERY_SUPERUSER[DEFAULT]

785 - elif name == QUERY_NAME_SERVER_VERSION:

786 - return QUERY_SHOW_VERSION[DEFAULT]

787 - elif name == QUERY_NAME_AUTOVACUUM:

788 - return QUERY_AUTOVACUUM[DEFAULT]

789 - elif name == QUERY_NAME_FORCED_AUTOVACUUM:

790 - return QUERY_FORCED_AUTOVACUUM[DEFAULT]

791 - elif name == QUERY_NAME_TX_WRAPAROUND:

792 - return QUERY_TX_WRAPAROUND[DEFAULT]

793 - elif name == QUERY_NAME_WAL:

794 - if version < 100000:

795 - return QUERY_WAL[V96]

796 - return QUERY_WAL[DEFAULT]

797 - elif name == QUERY_NAME_ARCHIVE:

798 - if version < 100000:

799 - return QUERY_ARCHIVE[V96]

800 - return QUERY_ARCHIVE[DEFAULT]

801 - elif name == QUERY_NAME_STANDBY_DELTA:

802 - if version < 100000:

803 - return QUERY_STANDBY_DELTA[V96]

804 - return QUERY_STANDBY_DELTA[DEFAULT]

805 - elif name == QUERY_NAME_STANDBY_LAG:

806 - return QUERY_STANDBY_LAG[DEFAULT]

807 - elif name == QUERY_NAME_REPSLOT_FILES:

808 - if version < 110000:

809 - return QUERY_REPSLOT_FILES[V10]

810 - return QUERY_REPSLOT_FILES[DEFAULT]

811 - elif name == QUERY_NAME_DIFF_LSN:

812 - if version < 100000:

813 - return QUERY_DIFF_LSN[V96]

814 - return QUERY_DIFF_LSN[DEFAULT]

815 -

816 - raise ValueError('unknown query')

817 -

818 -

819 -ORDER = [

820 - 'db_stat_temp_files',

821 - 'db_stat_temp_bytes',

822 - 'db_stat_blks',

823 - 'db_stat_tuple_returned',

824 - 'db_stat_tuple_write',

825 - 'db_stat_transactions',

826 - 'db_stat_connections',

827 - 'db_stat_blocking_pids_avg',

828 - 'database_size',

829 - 'backend_process',

830 - 'backend_usage',

831 - 'index_count',

832 - 'index_size',

833 - 'table_count',

834 - 'table_size',

835 - 'wal',

836 - 'wal_writes',

837 - 'archive_wal',

838 - 'checkpointer',

839 - 'stat_bgwriter_alloc',

840 - 'stat_bgwriter_checkpoint',

841 - 'stat_bgwriter_backend',

842 - 'stat_bgwriter_backend_fsync',

843 - 'stat_bgwriter_bgwriter',

844 - 'stat_bgwriter_maxwritten',

845 - 'replication_slot',

846 - 'standby_delta',

847 - 'standby_lag',

848 - 'autovacuum',

849 - 'forced_autovacuum',

850 - 'tx_wraparound_oldest_current_xid',

851 - 'tx_wraparound_percent_towards_wraparound'

852 -]

853 -

854 -CHARTS = {

855 - 'db_stat_transactions': {

856 - 'options': [None, 'Transactions on db', 'transactions/s', 'db statistics', 'postgres.db_stat_transactions',

857 - 'line'],

858 - 'lines': [

859 - ['xact_commit', 'committed', 'incremental'],

860 - ['xact_rollback', 'rolled back', 'incremental']

861 - ]

862 - },

863 - 'db_stat_connections': {

864 - 'options': [None, 'Current connections to db', 'count', 'db statistics', 'postgres.db_stat_connections',

865 - 'line'],

866 - 'lines': [

867 - ['connections', 'connections', 'absolute']

868 - ]

869 - },

870 - 'db_stat_blks': {

871 - 'options': [None, 'Disk blocks reads from db', 'reads/s', 'db statistics', 'postgres.db_stat_blks', 'line'],

872 - 'lines': [

873 - ['blks_read', 'disk', 'incremental'],

874 - ['blks_hit', 'cache', 'incremental']

875 - ]

876 - },

877 - 'db_stat_tuple_returned': {

878 - 'options': [None, 'Tuples returned from db', 'tuples/s', 'db statistics', 'postgres.db_stat_tuple_returned',

879 - 'line'],

880 - 'lines': [

881 - ['tup_returned', 'sequential', 'incremental'],

882 - ['tup_fetched', 'bitmap', 'incremental']

883 - ]

884 - },

885 - 'db_stat_tuple_write': {

886 - 'options': [None, 'Tuples written to db', 'writes/s', 'db statistics', 'postgres.db_stat_tuple_write', 'line'],

887 - 'lines': [

888 - ['tup_inserted', 'inserted', 'incremental'],

889 - ['tup_updated', 'updated', 'incremental'],

890 - ['tup_deleted', 'deleted', 'incremental'],

891 - ['conflicts', 'conflicts', 'incremental']

892 - ]

893 - },

894 - 'db_stat_temp_bytes': {

895 - 'options': [None, 'Temp files written to disk', 'KiB/s', 'db statistics', 'postgres.db_stat_temp_bytes',

896 - 'line'],

897 - 'lines': [

898 - ['temp_bytes', 'size', 'incremental', 1, 1024]

899 - ]

900 - },

901 - 'db_stat_temp_files': {

902 - 'options': [None, 'Temp files written to disk', 'files', 'db statistics', 'postgres.db_stat_temp_files',

903 - 'line'],

904 - 'lines': [

905 - ['temp_files', 'files', 'incremental']

906 - ]

907 - },

908 - 'db_stat_blocking_pids_avg': {

909 - 'options': [None, 'Average number of blocking transactions in db', 'processes', 'db statistics',

910 - 'postgres.db_stat_blocking_pids_avg', 'line'],

911 - 'lines': [

912 - ['blocking_pids_avg', 'blocking', 'absolute']

913 - ]

914 - },

915 - 'database_size': {

916 - 'options': [None, 'Database size', 'MiB', 'database size', 'postgres.db_size', 'stacked'],

917 - 'lines': [

918 - ]

919 - },

920 - 'backend_process': {

921 - 'options': [None, 'Current Backend Processes', 'processes', 'backend processes', 'postgres.backend_process',

922 - 'line'],

923 - 'lines': [

924 - ['backends_active', 'active', 'absolute'],

925 - ['backends_idle', 'idle', 'absolute']

926 - ]

927 - },

928 - 'backend_usage': {

929 - 'options': [None, '% of Connections in use', 'percentage', 'backend processes', 'postgres.backend_usage', 'stacked'],

930 - 'lines': [

931 - ['available', 'available', 'percentage-of-absolute-row'],

932 - ['used', 'used', 'percentage-of-absolute-row']

933 - ]

934 - },

935 - 'index_count': {

936 - 'options': [None, 'Total indexes', 'index', 'indexes', 'postgres.index_count', 'line'],

937 - 'lines': [

938 - ['index_count', 'total', 'absolute']

939 - ]

940 - },

941 - 'index_size': {

942 - 'options': [None, 'Indexes size', 'MiB', 'indexes', 'postgres.index_size', 'line'],

943 - 'lines': [

944 - ['index_size', 'size', 'absolute', 1, 1024 * 1024]

945 - ]

946 - },

947 - 'table_count': {

948 - 'options': [None, 'Total Tables', 'tables', 'tables', 'postgres.table_count', 'line'],

949 - 'lines': [

950 - ['table_count', 'total', 'absolute']

951 - ]

952 - },

953 - 'table_size': {

954 - 'options': [None, 'Tables size', 'MiB', 'tables', 'postgres.table_size', 'line'],

955 - 'lines': [

956 - ['table_size', 'size', 'absolute', 1, 1024 * 1024]

957 - ]

958 - },

959 - 'wal': {

960 - 'options': [None, 'Write-Ahead Logs', 'files', 'wal', 'postgres.wal', 'line'],

961 - 'lines': [

962 - ['written_wal', 'written', 'absolute'],

963 - ['recycled_wal', 'recycled', 'absolute'],

964 - ['total_wal', 'total', 'absolute']

965 - ]

966 - },

967 - 'wal_writes': {

968 - 'options': [None, 'Write-Ahead Logs', 'KiB/s', 'wal_writes', 'postgres.wal_writes', 'line'],

969 - 'lines': [

970 - ['wal_writes', 'writes', 'incremental', 1, 1024]

971 - ]

972 - },

973 - 'archive_wal': {

974 - 'options': [None, 'Archive Write-Ahead Logs', 'files/s', 'archive wal', 'postgres.archive_wal', 'line'],

975 - 'lines': [

976 - ['file_count', 'total', 'incremental'],

977 - ['ready_count', 'ready', 'incremental'],

978 - ['done_count', 'done', 'incremental']

979 - ]

980 - },

981 - 'checkpointer': {

982 - 'options': [None, 'Checkpoints', 'writes', 'checkpointer', 'postgres.checkpointer', 'line'],

983 - 'lines': [

984 - ['checkpoint_scheduled', 'scheduled', 'incremental'],

985 - ['checkpoint_requested', 'requested', 'incremental']

986 - ]

987 - },

988 - 'stat_bgwriter_alloc': {

989 - 'options': [None, 'Buffers allocated', 'KiB/s', 'bgwriter', 'postgres.stat_bgwriter_alloc', 'line'],

990 - 'lines': [

991 - ['buffers_alloc', 'alloc', 'incremental', 1, 1024]

992 - ]

993 - },

994 - 'stat_bgwriter_checkpoint': {

995 - 'options': [None, 'Buffers written during checkpoints', 'KiB/s', 'bgwriter',

996 - 'postgres.stat_bgwriter_checkpoint', 'line'],

997 - 'lines': [

998 - ['buffers_checkpoint', 'checkpoint', 'incremental', 1, 1024]

999 - ]

1000 - },

1001 - 'stat_bgwriter_backend': {

1002 - 'options': [None, 'Buffers written directly by a backend', 'KiB/s', 'bgwriter',

1003 - 'postgres.stat_bgwriter_backend', 'line'],

1004 - 'lines': [

1005 - ['buffers_backend', 'backend', 'incremental', 1, 1024]

1006 - ]

1007 - },

1008 - 'stat_bgwriter_backend_fsync': {

1009 - 'options': [None, 'Fsync by backend', 'times', 'bgwriter', 'postgres.stat_bgwriter_backend_fsync', 'line'],

1010 - 'lines': [

1011 - ['buffers_backend_fsync', 'backend fsync', 'incremental']

1012 - ]

1013 - },

1014 - 'stat_bgwriter_bgwriter': {

1015 - 'options': [None, 'Buffers written by the background writer', 'KiB/s', 'bgwriter',

1016 - 'postgres.bgwriter_bgwriter', 'line'],

1017 - 'lines': [

1018 - ['buffers_clean', 'clean', 'incremental', 1, 1024]

1019 - ]

1020 - },

1021 - 'stat_bgwriter_maxwritten': {

1022 - 'options': [None, 'Too many buffers written', 'times', 'bgwriter', 'postgres.stat_bgwriter_maxwritten',

1023 - 'line'],

1024 - 'lines': [

1025 - ['maxwritten_clean', 'maxwritten', 'incremental']

1026 - ]

1027 - },

1028 - 'autovacuum': {

1029 - 'options': [None, 'Autovacuum workers', 'workers', 'autovacuum', 'postgres.autovacuum', 'line'],

1030 - 'lines': [

1031 - ['analyze', 'analyze', 'absolute'],

1032 - ['vacuum', 'vacuum', 'absolute'],

1033 - ['vacuum_analyze', 'vacuum analyze', 'absolute'],

1034 - ['vacuum_freeze', 'vacuum freeze', 'absolute'],

1035 - ['brin_summarize', 'brin summarize', 'absolute']

1036 - ]

1037 - },

1038 - 'forced_autovacuum': {

1039 - 'options': [None, 'Percent towards forced autovacuum', 'percent', 'autovacuum', 'postgres.forced_autovacuum', 'line'],

1040 - 'lines': [

1041 - ['percent_towards_forced_autovacuum', 'percent', 'absolute']

1042 - ]

1043 - },

1044 - 'tx_wraparound_oldest_current_xid': {

1045 - 'options': [None, 'Oldest current XID', 'xid', 'tx_wraparound', 'postgres.tx_wraparound_oldest_current_xid', 'line'],

1046 - 'lines': [

1047 - ['oldest_current_xid', 'xid', 'absolute']

1048 - ]

1049 - },

1050 - 'tx_wraparound_percent_towards_wraparound': {

1051 - 'options': [None, 'Percent towards wraparound', 'percent', 'tx_wraparound', 'postgres.percent_towards_wraparound', 'line'],

1052 - 'lines': [

1053 - ['percent_towards_wraparound', 'percent', 'absolute']

1054 - ]

1055 - },

1056 - 'standby_delta': {

1057 - 'options': [None, 'Standby delta', 'KiB', 'replication delta', 'postgres.standby_delta', 'line'],

1058 - 'lines': [

1059 - ['sent_delta', 'sent delta', 'absolute', 1, 1024],

1060 - ['write_delta', 'write delta', 'absolute', 1, 1024],

1061 - ['flush_delta', 'flush delta', 'absolute', 1, 1024],

1062 - ['replay_delta', 'replay delta', 'absolute', 1, 1024]

1063 - ]

1064 - },

1065 - 'standby_lag': {

1066 - 'options': [None, 'Standby lag', 'seconds', 'replication lag', 'postgres.standby_lag', 'line'],

1067 - 'lines': [

1068 - ['write_lag', 'write lag', 'absolute'],

1069 - ['flush_lag', 'flush lag', 'absolute'],

1070 - ['replay_lag', 'replay lag', 'absolute']

1071 - ]

1072 - },

1073 - 'replication_slot': {

1074 - 'options': [None, 'Replication slot files', 'files', 'replication slot', 'postgres.replication_slot', 'line'],

1075 - 'lines': [

1076 - ['replslot_wal_keep', 'wal keeped', 'absolute'],

1077 - ['replslot_files', 'pg_replslot files', 'absolute']

1078 - ]

1079 - }

1080 -}

1081 -

1082 -

1083 -class Service(SimpleService):

1084 - def __init__(self, configuration=None, name=None):

1085 - SimpleService.__init__(self, configuration=configuration, name=name)

1086 - self.order = list(ORDER)

1087 - self.definitions = deepcopy(CHARTS)

1088 - self.do_table_stats = configuration.pop('table_stats', False)

1089 - self.do_index_stats = configuration.pop('index_stats', False)

1090 - self.databases_to_poll = configuration.pop('database_poll', None)

1091 - self.configuration = configuration

1092 - self.conn = None

1093 - self.conn_params = dict()

1094 - self.server_version = None

1095 - self.is_superuser = False

1096 - self.alive = False

1097 - self.databases = list()

1098 - self.secondaries = list()

1099 - self.replication_slots = list()

1100 - self.queries = dict()

1101 - self.data = dict()

1102 -

1103 - def reconnect(self):

1104 - return self.connect()

1105 -

1106 - def build_conn_params(self):

1107 - conf = self.configuration

1108 -

1109 - # connection URIs: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING

1110 - if conf.get(CONN_PARAM_DSN):

1111 - return {'dsn': conf[CONN_PARAM_DSN]}

1112 -

1113 - params = {

1114 - CONN_PARAM_HOST: conf.get(CONN_PARAM_HOST),

1115 - CONN_PARAM_PORT: conf.get(CONN_PARAM_PORT, DEFAULT_PORT),

1116 - CONN_PARAM_DATABASE: conf.get(CONN_PARAM_DATABASE),

1117 - CONN_PARAM_USER: conf.get(CONN_PARAM_USER, DEFAULT_USER),

1118 - CONN_PARAM_PASSWORD: conf.get(CONN_PARAM_PASSWORD),

1119 - CONN_PARAM_CONN_TIMEOUT: conf.get(CONN_PARAM_CONN_TIMEOUT, DEFAULT_CONNECT_TIMEOUT),

1120 - 'options': '-c statement_timeout={0}'.format(

1121 - conf.get(CONN_PARAM_STATEMENT_TIMEOUT, DEFAULT_STATEMENT_TIMEOUT)),

1122 - }

1123 -

1124 - # https://www.postgresql.org/docs/current/libpq-ssl.html

1125 - ssl_params = dict(

1126 - (k, v) for k, v in {

1127 - CONN_PARAM_SSL_MODE: conf.get(CONN_PARAM_SSL_MODE),

1128 - CONN_PARAM_SSL_ROOT_CERT: conf.get(CONN_PARAM_SSL_ROOT_CERT),

1129 - CONN_PARAM_SSL_CRL: conf.get(CONN_PARAM_SSL_CRL),

1130 - CONN_PARAM_SSL_CERT: conf.get(CONN_PARAM_SSL_CERT),

1131 - CONN_PARAM_SSL_KEY: conf.get(CONN_PARAM_SSL_KEY),

1132 - }.items() if v)

1133 -

1134 - if CONN_PARAM_SSL_MODE not in ssl_params and len(ssl_params) > 0:

1135 - raise ValueError("mandatory 'sslmode' param is missing, please set")

1136 -

1137 - params.update(ssl_params)

1138 -

1139 - return params

1140 -

1141 - def connect(self):

1142 - if self.conn:

1143 - self.conn.close()

1144 - self.conn = None

1145 -

1146 - try:

1147 - self.conn = psycopg2.connect(**self.conn_params)

1148 - self.conn.set_isolation_level(extensions.ISOLATION_LEVEL_AUTOCOMMIT)

1149 - self.conn.set_session(readonly=True)

1150 - except OperationalError as error:

1151 - self.error(error)

1152 - self.alive = False

1153 - else:

1154 - self.alive = True

1155 -

1156 - return self.alive

1157 -

1158 - def check(self):

1159 - if not PSYCOPG2:

1160 - self.error("'python-psycopg2' package is needed to use postgres module")

1161 - return False

1162 -

1163 - try:

1164 - self.conn_params = self.build_conn_params()

1165 - except ValueError as error:

1166 - self.error('error on creating connection params : {0}', error)

1167 - return False

1168 -

1169 - if not self.connect():

1170 - self.error('failed to connect to {0}'.format(hide_password(self.conn_params)))

1171 - return False

1172 -

1173 - try:

1174 - self.check_queries()

1175 - except Exception as error:

1176 - self.error(error)

1177 - return False

1178 -

1179 - self.populate_queries()

1180 - self.create_dynamic_charts()

1181 -

1182 - return True

1183 -

1184 - def get_data(self):

1185 - if not self.alive and not self.reconnect():

1186 - return None

1187 -

1188 - self.data = dict()

1189 - try:

1190 - cursor = self.conn.cursor(cursor_factory=DictCursor)

1191 -

1192 - self.data.update(zero_lock_types(self.databases))

1193 -

1194 - for query, metrics in self.queries.items():

1195 - self.query_stats(cursor, query, metrics)

1196 -

1197 - except OperationalError:

1198 - self.alive = False

1199 - return None

1200 -

1201 - cursor.close()

1202 -

1203 - return self.data

1204 -

1205 - def query_stats(self, cursor, query, metrics):

1206 - cursor.execute(query, dict(databases=tuple(self.databases)))

1207 -

1208 - for row in cursor:

1209 - for metric in metrics:

1210 - # databases

1211 - if 'database_name' in row:

1212 - dimension_id = '_'.join([row['database_name'], metric])

1213 - # secondaries

1214 - elif 'application_name' in row:

1215 - dimension_id = '_'.join([row['application_name'], metric])

1216 - # replication slots

1217 - elif 'slot_name' in row:

1218 - dimension_id = '_'.join([row['slot_name'], metric])

1219 - # other

1220 - else:

1221 - dimension_id = metric

1222 -

1223 - if metric in row:

1224 - if row[metric] is not None:

1225 - self.data[dimension_id] = int(row[metric])

1226 - elif 'locks_count' in row:

1227 - if metric == row['mode']:

1228 - self.data[dimension_id] = row['locks_count']

1229 -

1230 - def check_queries(self):

1231 - cursor = self.conn.cursor()

1232 -

1233 - self.server_version = detect_server_version(cursor, query_factory(QUERY_NAME_SERVER_VERSION))

1234 - self.debug('server version: {0}'.format(self.server_version))

1235 -

1236 - self.is_superuser = check_if_superuser(cursor, query_factory(QUERY_NAME_IF_SUPERUSER))

1237 - self.debug('superuser: {0}'.format(self.is_superuser))

1238 -

1239 - self.databases = discover(cursor, query_factory(QUERY_NAME_DATABASES))

1240 - self.debug('discovered databases {0}'.format(self.databases))

1241 - if self.databases_to_poll:

1242 - to_poll = self.databases_to_poll.split()

1243 - self.databases = [db for db in self.databases if db in to_poll] or self.databases

1244 -

1245 - self.secondaries = discover(cursor, query_factory(QUERY_NAME_STANDBY))

1246 - self.debug('discovered secondaries: {0}'.format(self.secondaries))

1247 -

1248 - if self.server_version >= 94000:

1249 - self.replication_slots = discover(cursor, query_factory(QUERY_NAME_REPLICATION_SLOT))

1250 - self.debug('discovered replication slots: {0}'.format(self.replication_slots))

1251 -

1252 - cursor.close()

1253 -

1254 - def populate_queries(self):

1255 - self.queries[query_factory(QUERY_NAME_DATABASE)] = METRICS[QUERY_NAME_DATABASE]

1256 - self.queries[query_factory(QUERY_NAME_BACKENDS)] = METRICS[QUERY_NAME_BACKENDS]

1257 - self.queries[query_factory(QUERY_NAME_BACKEND_USAGE, self.server_version)] = METRICS[QUERY_NAME_BACKEND_USAGE]

1258 - self.queries[query_factory(QUERY_NAME_LOCKS)] = METRICS[QUERY_NAME_LOCKS]

1259 - self.queries[query_factory(QUERY_NAME_BGWRITER)] = METRICS[QUERY_NAME_BGWRITER]

1260 - self.queries[query_factory(QUERY_NAME_DIFF_LSN, self.server_version)] = METRICS[QUERY_NAME_WAL_WRITES]

1261 - self.queries[query_factory(QUERY_NAME_STANDBY_DELTA, self.server_version)] = METRICS[QUERY_NAME_STANDBY_DELTA]

1262 - self.queries[query_factory(QUERY_NAME_BLOCKERS, self.server_version)] = METRICS[QUERY_NAME_BLOCKERS]

1263 -

1264 - if self.do_index_stats:

1265 - self.queries[query_factory(QUERY_NAME_INDEX_STATS)] = METRICS[QUERY_NAME_INDEX_STATS]

1266 - if self.do_table_stats:

1267 - self.queries[query_factory(QUERY_NAME_TABLE_STATS)] = METRICS[QUERY_NAME_TABLE_STATS]

1268 -

1269 - if self.is_superuser:

1270 - self.queries[query_factory(QUERY_NAME_ARCHIVE, self.server_version)] = METRICS[QUERY_NAME_ARCHIVE]

1271 -

1272 - if self.server_version >= 90400:

1273 - self.queries[query_factory(QUERY_NAME_WAL, self.server_version)] = METRICS[QUERY_NAME_WAL]

1274 -

1275 - if self.server_version >= 100000:

1276 - v = METRICS[QUERY_NAME_REPSLOT_FILES]

1277 - self.queries[query_factory(QUERY_NAME_REPSLOT_FILES, self.server_version)] = v

1278 -

1279 - if self.server_version >= 90400:

1280 - self.queries[query_factory(QUERY_NAME_AUTOVACUUM)] = METRICS[QUERY_NAME_AUTOVACUUM]

1281 -

1282 - self.queries[query_factory(QUERY_NAME_FORCED_AUTOVACUUM)] = METRICS[QUERY_NAME_FORCED_AUTOVACUUM]

1283 - self.queries[query_factory(QUERY_NAME_TX_WRAPAROUND)] = METRICS[QUERY_NAME_TX_WRAPAROUND]

1284 -

1285 - if self.server_version >= 100000:

1286 - self.queries[query_factory(QUERY_NAME_STANDBY_LAG)] = METRICS[QUERY_NAME_STANDBY_LAG]

1287 -

1288 - def create_dynamic_charts(self):

1289 - for database_name in self.databases[::-1]:

1290 - dim = [

1291 - database_name + '_size',

1292 - database_name,

1293 - 'absolute',

1294 - 1,

1295 - 1024 * 1024,

1296 - ]

1297 - self.definitions['database_size']['lines'].append(dim)

1298 - for chart_name in [name for name in self.order if name.startswith('db_stat')]:

1299 - add_database_stat_chart(

1300 - order=self.order,

1301 - definitions=self.definitions,

1302 - name=chart_name,

1303 - database_name=database_name,

1304 - )

1305 - add_database_lock_chart(

1306 - order=self.order,

1307 - definitions=self.definitions,

1308 - database_name=database_name,

1309 - )

1310 -

1311 - for application_name in self.secondaries[::-1]:

1312 - add_replication_standby_chart(

1313 - order=self.order,

1314 - definitions=self.definitions,

1315 - name='standby_delta',

1316 - application_name=application_name,

1317 - chart_family='replication delta',

1318 - )

1319 - add_replication_standby_chart(

1320 - order=self.order,

1321 - definitions=self.definitions,

1322 - name='standby_lag',

1323 - application_name=application_name,

1324 - chart_family='replication lag',

1325 - )

1326 -

1327 - for slot_name in self.replication_slots[::-1]:

1328 - add_replication_slot_chart(

1329 - order=self.order,

1330 - definitions=self.definitions,

1331 - name='replication_slot',

1332 - slot_name=slot_name,

1333 - )

1334 -

1335 -

1336 -def discover(cursor, query):

1337 - cursor.execute(query)

1338 - result = list()

1339 - for v in [value[0] for value in cursor]:

1340 - if v not in result:

1341 - result.append(v)

1342 - return result

1343 -

1344 -

1345 -def check_if_superuser(cursor, query):

1346 - cursor.execute(query)

1347 - return cursor.fetchone()[0]

1348 -

1349 -

1350 -def detect_server_version(cursor, query):

1351 - cursor.execute(query)

1352 - return int(cursor.fetchone()[0])

1353 -

1354 -

1355 -def zero_lock_types(databases):

1356 - result = dict()

1357 - for database in databases:

1358 - for lock_type in METRICS['LOCKS']:

1359 - key = '_'.join([database, lock_type])

1360 - result[key] = 0

1361 -

1362 - return result

1363 -

1364 -

1365 -def hide_password(config):

1366 - return dict((k, v if k != 'password' or not v else '*****') for k, v in config.items())

1367 -

1368 -

1369 -def add_database_lock_chart(order, definitions, database_name):

1370 - def create_lines(database):

1371 - result = list()

1372 - for lock_type in METRICS['LOCKS']:

1373 - dimension_id = '_'.join([database, lock_type])

1374 - result.append([dimension_id, lock_type, 'absolute'])

1375 - return result

1376 -

1377 - chart_name = database_name + '_locks'

1378 - order.insert(-1, chart_name)

1379 - definitions[chart_name] = {

1380 - 'options':

1381 - [None, 'Locks on db: ' + database_name, 'locks', 'db ' + database_name, 'postgres.db_locks', 'line'],

1382 - 'lines': create_lines(database_name)

1383 - }

1384 -

1385 -

1386 -def add_database_stat_chart(order, definitions, name, database_name):

1387 - def create_lines(database, lines):

1388 - result = list()

1389 - for line in lines:

1390 - new_line = ['_'.join([database, line[0]])] + line[1:]

1391 - result.append(new_line)

1392 - return result

1393 -

1394 - chart_template = CHARTS[name]

1395 - chart_name = '_'.join([database_name, name])

1396 - order.insert(0, chart_name)

1397 - name, title, units, _, context, chart_type = chart_template['options']

1398 - definitions[chart_name] = {

1399 - 'options': [name, title + ': ' + database_name, units, 'db ' + database_name, context, chart_type],

1400 - 'lines': create_lines(database_name, chart_template['lines'])}

1401 -

1402 -

1403 -def add_replication_standby_chart(order, definitions, name, application_name, chart_family):

1404 - def create_lines(standby, lines):

1405 - result = list()

1406 - for line in lines:

1407 - new_line = ['_'.join([standby, line[0]])] + line[1:]

1408 - result.append(new_line)

1409 - return result

1410 -

1411 - chart_template = CHARTS[name]

1412 - chart_name = '_'.join([application_name, name])

1413 - position = order.index('database_size')

1414 - order.insert(position, chart_name)

1415 - name, title, units, _, context, chart_type = chart_template['options']

1416 - definitions[chart_name] = {

1417 - 'options': [name, title + ': ' + application_name, units, chart_family, context, chart_type],

1418 - 'lines': create_lines(application_name, chart_template['lines'])}

1419 -

1420 -

1421 -def add_replication_slot_chart(order, definitions, name, slot_name):

1422 - def create_lines(slot, lines):

1423 - result = list()

1424 - for line in lines:

1425 - new_line = ['_'.join([slot, line[0]])] + line[1:]

1426 - result.append(new_line)

1427 - return result

1428 -

1429 - chart_template = CHARTS[name]

1430 - chart_name = '_'.join([slot_name, name])

1431 - position = order.index('database_size')

1432 - order.insert(position, chart_name)

1433 - name, title, units, _, context, chart_type = chart_template['options']

1434 - definitions[chart_name] = {

1435 - 'options': [name, title + ': ' + slot_name, units, 'replication slot files', context, chart_type],

1436 - 'lines': create_lines(slot_name, chart_template['lines'])}

collectors/python.d.plugin/postgres/postgres.conf deleted

-134

@@ -1,134 +0,0 @@

1 -# netdata python.d.plugin configuration for postgresql

2 -#

3 -# This file is in YaML format. Generally the format is:

4 -#

5 -# name: value

6 -#

7 -# There are 2 sections:

8 -# - global variables

9 -# - one or more JOBS

10 -#

11 -# JOBS allow you to collect values from multiple sources.

12 -# Each source will have its own set of charts.

13 -#

14 -# JOB parameters have to be indented (using spaces only, example below).

15 -

16 -# ----------------------------------------------------------------------

17 -# Global Variables

18 -# These variables set the defaults for all JOBs, however each JOB

19 -# may define its own, overriding the defaults.

20 -

21 -# update_every sets the default data collection frequency.

22 -# If unset, the python.d.plugin default is used.

23 -# update_every: 1

24 -

25 -# priority controls the order of charts at the netdata dashboard.

26 -# Lower numbers move the charts towards the top of the page.

27 -# If unset, the default for python.d.plugin is used.

28 -# priority: 60000

29 -

30 -# penalty indicates whether to apply penalty to update_every in case of failures.

31 -# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes.

32 -# penalty: yes

33 -

34 -# autodetection_retry sets the job re-check interval in seconds.

35 -# The job is not deleted if check fails.

36 -# Attempts to start the job are made once every autodetection_retry.

37 -# This feature is disabled by default.

38 -# autodetection_retry: 0

39 -

40 -# ----------------------------------------------------------------------

41 -# JOBS (data collection sources)

42 -#

43 -# The default JOBS share the same *name*. JOBS with the same name

44 -# are mutually exclusive. Only one of them will be allowed running at

45 -# any time. This allows autodetection to try several alternatives and

46 -# pick the one that works.

47 -#

48 -# Any number of jobs is supported.

49 -#

50 -# All python.d.plugin JOBS (for all its modules) support a set of

51 -# predefined parameters. These are:

52 -#

53 -# job_name:

54 -# name: myname # the JOB's name as it will appear at the

55 -# # dashboard (by default is the job_name)

56 -# # JOBs sharing a name are mutually exclusive

57 -# update_every: 1 # the JOB's data collection frequency

58 -# priority: 60000 # the JOB's order on the dashboard

59 -# penalty: yes # the JOB's penalty

60 -# autodetection_retry: 0 # the JOB's re-check interval in seconds

61 -#

62 -# A single connection is required in order to pull statistics.

63 -#

64 -# Connections can be configured with the following options:

65 -#

66 -# dsn : 'connection URI' # see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING

67 -#

68 -# OR

69 -#

70 -# database : 'example_db_name'

71 -# user : 'example_user'

72 -# password : 'example_pass'

73 -# host : 'localhost'

74 -# port : 5432

75 -# connect_timeout : 2 # in seconds, default is 2

76 -# statement_timeout : 2000 # in ms, default is 2000

77 -# sslmode : mode # one of [disable, allow, prefer, require, verify-ca, verify-full]

78 -# sslrootcert : path/to/rootcert # the location of the root certificate file

79 -# sslcrl : path/to/crl # the location of the CRL file

80 -# sslcert : path/to/cert # the location of the client certificate file

81 -# sslkey : path/to/key # the location of the client key file

82 -#

83 -# SSL connection parameters description: https://www.postgresql.org/docs/current/libpq-ssl.html

84 -#

85 -# Additionally, the following options allow selective disabling of charts

86 -#

87 -# table_stats : false

88 -# index_stats : false

89 -# database_poll : 'dbase_name1 dbase_name2' # poll only specified databases (all other will be excluded from charts)

90 -#

91 -# Postgres permissions are configured at its pg_hba.conf file. You can

92 -# "trust" local clients to allow netdata to connect, or you can create

93 -# a postgres user for netdata and add its password below to allow

94 -# netdata connect.

95 -#

96 -# Please note that when running Postgres from inside the container,

97 -# the client (Netdata) is not considered local, unless it runs from inside

98 -# the same container.

99 -#

100 -# Superuser access is needed for these charts:

101 -# Write-Ahead Logs

102 -# Archive Write-Ahead Logs

103 -#

104 -# Autovacuum charts is allowed since Postgres 9.4

105 -# ----------------------------------------------------------------------

106 -

107 -socket:

108 - name : 'local'

109 - user : 'postgres'

110 - database : 'postgres'

111 -

112 -tcp:

113 - name : 'local'

114 - database : 'postgres'

115 - user : 'postgres'

116 - password : 'postgres'

117 - host : 'localhost'

118 - port : 5432

119 -

120 -tcpipv4:

121 - name : 'local'

122 - database : 'postgres'

123 - user : 'postgres'

124 - password : 'postgres'

125 - host : '127.0.0.1'

126 - port : 5432

127 -

128 -tcpipv6:

129 - name : 'local'

130 - database : 'postgres'

131 - user : 'postgres'

132 - password : 'postgres'

133 - host : '::1'

134 - port : 5432

collectors/python.d.plugin/python.d.conf

-1

@@ -63,7 +63,6 @@ logind: no

63 # openldap: yes

64 # oracledb: yes

65 # postfix: yes

66 -# postgres: yes

66 # proxysql: yes

67 # puppet: yes

68 # rabbitmq: yes

docs/guides/python-collector.md

+5 -5

@@ -424,8 +424,8 @@ configuration in [YAML](https://www.tutorialspoint.com/yaml/yaml_basics.htm) for

424 run. This enables you to define different "ways" to fetch data from a particular data source so that the collector has

425 more chances to work out-of-the-box. For example, if the data source supports both `HTTP` and `linux socket`, you can

426 define 2 jobs named `local`, with each using a different method.

427 -- Check the `postgresql` collector configuration file on

428 - [GitHub](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/postgres/postgres.conf) to get a

427 +- Check the `example` collector configuration file on

428 + [GitHub](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/example/example.conf) to get a

429 sense of the structure.

430

431 ```yaml

@@ -455,8 +455,8 @@ function takes 2 arguments, one with the name of the configuration field and one

455 find the configuration field. This allows you to define sane defaults for your collector.

456

457 Moreover, when creating the configuration file, create a large comment section that describes the configuration

458 -variables and inform the user about the defaults. For example, take a look at the `postgresql` collector on

459 -[GitHub](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/postgres/postgres.conf).

458 +variables and inform the user about the defaults. For example, take a look at the `example` collector on

459 +[GitHub](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/example/example.conf).

460

461 You can read more about the configuration file on the [`python.d.plugin`

462 documentation](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin).

@@ -465,7 +465,7 @@ documentation](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin

465

466 Find the source code for the above examples on [GitHub](https://github.com/papajohn-uop/netdata).

467

468 -Now we you ready to start developing our Netdata python Collector and share it with the rest of the Netdata community.

468 +Now you are ready to start developing our Netdata python Collector and share it with the rest of the Netdata community.

469

470 - If you need help while developing your collector, join our [Netdata

471 Community](https://community.netdata.cloud/c/agent-development/9) to chat about it.

packaging/installer/install-required-packages.sh

+2 -62

@@ -20,7 +20,6 @@ fi

20 PACKAGES_NETDATA=${PACKAGES_NETDATA-1}

21 PACKAGES_NETDATA_PYTHON=${PACKAGES_NETDATA_PYTHON-0}

22 PACKAGES_NETDATA_PYTHON3=${PACKAGES_NETDATA_PYTHON3-1}

23 -PACKAGES_NETDATA_PYTHON_POSTGRES=${PACKAGES_NETDATA_PYTHON_POSTGRES-0}

23 PACKAGES_NETDATA_PYTHON_MONGO=${PACKAGES_NETDATA_PYTHON_MONGO-0}

24 PACKAGES_DEBUG=${PACKAGES_DEBUG-0}

25 PACKAGES_IPRANGE=${PACKAGES_IPRANGE-0}

@@ -97,8 +96,7 @@ Supported installers (IN):

96 Supported packages (you can append many of them):

98 - netdata-all all packages required to install netdata

100 - including postgres client,

101 - node.js, python, sensors, etc

99 + including python, sensors, etc

100

101 - netdata minimum packages required to install netdata

102 (includes python)

@@ -107,10 +105,6 @@ Supported packages (you can append many of them):

105

106 - python3 install python3

107

110 - - python-postgres install psycopg2

111 - (for monitoring postgres, will install python3 version

112 - if python3 is enabled or detected)

113 -

108 - python-pymongo install python-pymongo (or python3-pymongo for python3)

109

110 - sensors install lm_sensors for monitoring h/w sensors

@@ -941,42 +935,6 @@ declare -A pkg_python=(

935 ['centos-8']="python2"

936 )

937

944 -declare -A pkg_python_psycopg2=(

945 - ['alpine']="py-psycopg2"

946 - ['arch']="python2-psycopg2"

947 - ['centos']="python-psycopg2"

948 - ['debian']="python-psycopg2"

949 - ['gentoo']="dev-python/psycopg"

950 - ['sabayon']="dev-python/psycopg:2"

951 - ['rhel']="python-psycopg2"

952 - ['ol']="python-psycopg2"

953 - ['suse']="python-psycopg2"

954 - ['clearlinux']="WARNING|"

955 - ['macos']="WARNING|"

956 - ['default']="python-psycopg2"

957 -)

958 -

959 -declare -A pkg_python3_psycopg2=(

960 - ['alpine']="py3-psycopg2"

961 - ['arch']="python-psycopg2"

962 - ['centos']="WARNING|"

963 - ['debian']="WARNING|"

964 - ['gentoo']="dev-python/psycopg"

965 - ['sabayon']="dev-python/psycopg:2"

966 - ['rhel']="WARNING|"

967 - ['ol']="WARNING|"

968 - ['suse']="WARNING|"

969 - ['clearlinux']="WARNING|"

970 - ['macos']="WARNING|"

971 - ['default']="WARNING|"

972 -

973 - ['centos-7']="python3-psycopg2"

974 - ['centos-8']="python38-psycopg2"

975 - ['rhel-7']="python3-psycopg2"

976 - ['rhel-8']="python38-psycopg2"

977 - ['ol-8']="python3-psycopg2"

978 -)

979 -

938 declare -A pkg_python_pip=(

939 ['alpine']="py-pip"

940 ['gentoo']="dev-python/pip"

@@ -1359,8 +1317,6 @@ packages() {

1317 [ "${PACKAGES_NETDATA_PYTHON_MONGO}" -ne 0 ] && suitable_package python-pymongo

1318 # suitable_package python-requests

1319 # suitable_package python-pip

1362 -

1363 - [ "${PACKAGES_NETDATA_PYTHON_POSTGRES}" -ne 0 ] && suitable_package python-psycopg2

1320 fi

1321

1322 # -------------------------------------------------------------------------

@@ -1372,8 +1328,6 @@ packages() {

1328 [ "${PACKAGES_NETDATA_PYTHON_MONGO}" -ne 0 ] && suitable_package python3-pymongo

1329 # suitable_package python3-requests

1330 # suitable_package python3-pip

1375 -

1376 - [ "${PACKAGES_NETDATA_PYTHON_POSTGRES}" -ne 0 ] && suitable_package python3-psycopg2

1331 fi

1332

1333 # -------------------------------------------------------------------------

@@ -1913,7 +1867,7 @@ EOF

1867 remote_log() {

1868 # log success or failure on our system

1869 # to help us solve installation issues

1916 - curl > /dev/null 2>&1 -Ss --max-time 3 "https://registry.my-netdata.io/log/installer?status=${1}&error=${2}&distribution=${distribution}&version=${version}&installer=${package_installer}&tree=${tree}&detection=${detection}&netdata=${PACKAGES_NETDATA}&python=${PACKAGES_NETDATA_PYTHON}&python3=${PACKAGES_NETDATA_PYTHON3}&postgres=${PACKAGES_NETDATA_PYTHON_POSTGRES}&pymongo=${PACKAGES_NETDATA_PYTHON_MONGO}&sensors=${PACKAGES_NETDATA_SENSORS}&database=${PACKAGES_NETDATA_DATABASE}&ebpf=${PACKAGES_NETDATA_EBPF}&firehol=${PACKAGES_FIREHOL}&fireqos=${PACKAGES_FIREQOS}&iprange=${PACKAGES_IPRANGE}&update_ipsets=${PACKAGES_UPDATE_IPSETS}&demo=${PACKAGES_NETDATA_DEMO_SITE}"

1870 + curl > /dev/null 2>&1 -Ss --max-time 3 "https://registry.my-netdata.io/log/installer?status=${1}&error=${2}&distribution=${distribution}&version=${version}&installer=${package_installer}&tree=${tree}&detection=${detection}&netdata=${PACKAGES_NETDATA}&python=${PACKAGES_NETDATA_PYTHON}&python3=${PACKAGES_NETDATA_PYTHON3}&pymongo=${PACKAGES_NETDATA_PYTHON_MONGO}&sensors=${PACKAGES_NETDATA_SENSORS}&database=${PACKAGES_NETDATA_DATABASE}&ebpf=${PACKAGES_NETDATA_EBPF}&firehol=${PACKAGES_FIREHOL}&fireqos=${PACKAGES_FIREQOS}&iprange=${PACKAGES_IPRANGE}&update_ipsets=${PACKAGES_UPDATE_IPSETS}&demo=${PACKAGES_NETDATA_DEMO_SITE}"

1871 }

1872

1873 if [ -z "${1}" ]; then

@@ -1976,11 +1930,9 @@ while [ -n "${1}" ]; do

1930 PACKAGES_NETDATA=1

1931 if [ "${pv}" -eq 2 ]; then

1932 PACKAGES_NETDATA_PYTHON=1

1979 - PACKAGES_NETDATA_PYTHON_POSTGRES=1

1933 PACKAGES_NETDATA_PYTHON_MONGO=1

1934 else

1935 PACKAGES_NETDATA_PYTHON3=1

1983 - PACKAGES_NETDATA_PYTHON3_POSTGRES=1

1936 PACKAGES_NETDATA_PYTHON3_MONGO=1

1937 fi

1938 PACKAGES_NETDATA_SENSORS=1

@@ -2003,16 +1955,6 @@ while [ -n "${1}" ]; do

1955 PACKAGES_NETDATA_PYTHON3=1

1956 ;;

1957

2006 - python-postgres | postgres-python | psycopg2 | netdata-postgres)

2007 - if [ "${pv}" -eq 2 ]; then

2008 - PACKAGES_NETDATA_PYTHON=1

2009 - PACKAGES_NETDATA_PYTHON_POSTGRES=1

2010 - else

2011 - PACKAGES_NETDATA_PYTHON3=1

2012 - PACKAGES_NETDATA_PYTHON3_POSTGRES=1

2013 - fi

2014 - ;;

2015 -

1958 python-pymongo)

1959 if [ "${pv}" -eq 2 ]; then

1960 PACKAGES_NETDATA_PYTHON=1

@@ -2042,11 +1984,9 @@ while [ -n "${1}" ]; do

1984 PACKAGES_NETDATA=1

1985 if [ "${pv}" -eq 2 ]; then

1986 PACKAGES_NETDATA_PYTHON=1

2045 - PACKAGES_NETDATA_PYTHON_POSTGRES=1

1987 PACKAGES_NETDATA_PYTHON_MONGO=1

1988 else

1989 PACKAGES_NETDATA_PYTHON3=1

2049 - PACKAGES_NETDATA_PYTHON3_POSTGRES=1

1990 PACKAGES_NETDATA_PYTHON3_MONGO=1

1991 fi

1992 PACKAGES_DEBUG=1

packaging/installer/methods/manual.md

+1 -2

@@ -41,7 +41,7 @@ and other operating systems and is regularly tested. You can find this tool [her

41 - **SLE12** Must have your system registered with SUSE Customer Center or have the DVD. See

42 [#1162](https://github.com/netdata/netdata/issues/1162)

44 -Install the packages for having a **basic Netdata installation** (system monitoring and many applications, without `mysql` / `mariadb`, `postgres`, `named`, hardware sensors and `SNMP`):

44 +Install the packages for having a **basic Netdata installation** (system monitoring and many applications, without `mysql` / `mariadb`, `named`, hardware sensors and `SNMP`):

46 ```sh

47 curl -Ss 'https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/install-required-packages.sh' >/tmp/install-required-packages.sh && bash /tmp/install-required-packages.sh -i netdata

@@ -99,7 +99,6 @@ Netdata plugins and various aspects of Netdata can be enabled or benefit when th

99 | `python-dnspython`|used for monitoring DNS query time|

100 | `python-ipaddress`|used for monitoring **DHCPd** this package is required only if the system has python v2. python v3 has this functionality embedded|

101 | `python-mysqldb` or `python-pymysql`|used for monitoring **mysql** or **mariadb** databases `python-mysqldb` is a lot faster and thus preferred|

102 -| `python-psycopg2`|used for monitoring **postgresql** databases|

102 | `python-pymongo`|used for monitoring **mongodb** databases|

103 | `nodejs`|used for `node.js` plugins for monitoring **named** and **SNMP** devices|

104 | `lm-sensors`|for monitoring **hardware sensors**|

web/gui/dashboard_info.js

-84

@@ -3802,90 +3802,6 @@ netdataDashboard.context = {

3802

3803 // ------------------------------------------------------------------------

3804 // POSTGRESQL

3805 -

3806 - // python version start

3807 - 'postgres.db_stat_blks': {

3808 - info: 'Blocks reads from disk or cache.<ul>' +

3809 - '<li>blks_read: number of disk blocks read in this database.</li>' +

3810 - '<li>blks_hit: number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache)</li>' +

3811 - '</ul>'

3812 - },

3813 - 'postgres.db_stat_tuple_write': {

3814 - info: '<ul><li>Number of rows inserted/updated/deleted.</li>' +

3815 - '<li>conflicts: number of queries canceled due to conflicts with recovery in this database. (Conflicts occur only on standby servers; see <a href="https://www.postgresql.org/docs/10/static/monitoring-stats.html#PG-STAT-DATABASE-CONFLICTS-VIEW" target="_blank">pg_stat_database_conflicts</a> for details.)</li>' +

3816 - '</ul>'

3817 - },

3818 - 'postgres.db_stat_temp_bytes': {

3819 - info: 'Temporary files can be created on disk for sorts, hashes, and temporary query results.'

3820 - },

3821 - 'postgres.db_stat_temp_files': {

3822 - info: '<ul>' +

3823 - '<li>files: number of temporary files created by queries. All temporary files are counted, regardless of why the temporary file was created (e.g., sorting or hashing).</li>' +

3824 - '</ul>'

3825 - },

3826 - 'postgres.archive_wal': {

3827 - info: 'WAL archiving.<ul>' +

3828 - '<li>total: total files.</li>' +

3829 - '<li>ready: WAL waiting to be archived.</li>' +

3830 - '<li>done: WAL successfully archived. ' +

3831 - 'Ready WAL can indicate archive_command is in error, see <a href="https://www.postgresql.org/docs/current/static/continuous-archiving.html" target="_blank">Continuous Archiving and Point-in-Time Recovery</a>.</li>' +

3832 - '</ul>'

3833 - },

3834 - 'postgres.checkpointer': {

3835 - info: 'Number of checkpoints.<ul>' +

3836 - '<li>scheduled: when checkpoint_timeout is reached.</li>' +

3837 - '<li>requested: when max_wal_size is reached.</li>' +

3838 - '</ul>' +

3839 - 'For more information see <a href="https://www.postgresql.org/docs/current/static/wal-configuration.html" target="_blank">WAL Configuration</a>.'

3840 - },

3841 - 'postgres.autovacuum': {

3842 - info: 'PostgreSQL databases require periodic maintenance known as vacuuming. For many installations, it is sufficient to let vacuuming be performed by the autovacuum daemon. ' +

3843 - 'For more information see <a href="https://www.postgresql.org/docs/current/static/routine-vacuuming.html#AUTOVACUUM" target="_blank">The Autovacuum Daemon</a>.'

3844 - },

3845 - 'postgres.standby_delta': {

3846 - info: 'Streaming replication delta.<ul>' +

3847 - '<li>sent_delta: replication delta sent to standby.</li>' +

3848 - '<li>write_delta: replication delta written to disk by this standby.</li>' +

3849 - '<li>flush_delta: replication delta flushed to disk by this standby server.</li>' +

3850 - '<li>replay_delta: replication delta replayed into the database on this standby server.</li>' +

3851 - '</ul>' +

3852 - 'For more information see <a href="https://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION" target="_blank">Synchronous Replication</a>.'

3853 - },

3854 - 'postgres.replication_slot': {

3855 - info: 'Replication slot files.<ul>' +

3856 - '<li>wal_keeped: WAL files retained by each replication slots.</li>' +

3857 - '<li>pg_replslot_files: files present in pg_replslot.</li>' +

3858 - '</ul>' +

3859 - 'For more information see <a href="https://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION-SLOTS" target="_blank">Replication Slots</a>.'

3860 - },

3861 - 'postgres.backend_usage': {

3862 - info: 'Connections usage against maximum connections allowed, as defined in the max_connections setting.<ul>' +

3863 - '<li>available: maximum new connections allowed.</li>' +

3864 - '<li>used: connections currently in use.</li>' +

3865 - '</ul>' +

3866 - 'Assuming non-superuser accounts are being used to connect to Postgres (so superuser_reserved_connections are subtracted from max_connections). ' +

3867 - 'For more information see <a href="https://www.postgresql.org/docs/current/runtime-config-connection.html" target="_blank">Connections and Authentication</a>.'

3868 - },

3869 - 'postgres.forced_autovacuum': {

3870 - info: 'Percent towards forced autovacuum for one or more tables.<ul>' +

3871 - '<li>percent_towards_forced_autovacuum: a forced autovacuum will run once this value reaches 100.</li>' +

3872 - '</ul>' +

3873 - 'For more information see <a href="https://www.postgresql.org/docs/current/routine-vacuuming.html" target="_blank">Preventing Transaction ID Wraparound Failures</a>.'

3874 - },

3875 - 'postgres.tx_wraparound_oldest_current_xid': {

3876 - info: 'The oldest current transaction id (xid).<ul>' +

3877 - '<li>oldest_current_xid: oldest current transaction id.</li>' +

3878 - '</ul>' +

3879 - 'If for some reason autovacuum fails to clear old XIDs from a table, the system will begin to emit warning messages when the database\'s oldest XIDs reach eleven million transactions from the wraparound point. ' +

3880 - 'For more information see <a href="https://www.postgresql.org/docs/current/routine-vacuuming.html" target="_blank">Preventing Transaction ID Wraparound Failures</a>.'

3881 - },

3882 - 'postgres.percent_towards_wraparound': {

3883 - info: 'Percent towards transaction wraparound.<ul>' +

3884 - '<li>percent_towards_wraparound: transaction wraparound may occur when this value reaches 100.</li>' +

3885 - '</ul>' +

3886 - 'For more information see <a href="https://www.postgresql.org/docs/current/routine-vacuuming.html" target="_blank">Preventing Transaction ID Wraparound Failures</a>.'

3887 - },

3888 - // python version end

3805 'postgres.connections_utilization': {

3806 info: 'Connections in use as percentage of max_connections. Connection "slots" that are reserved for superusers (superuser_reserved_connections) are subtracted from the limit. If the utilization is 100% new connections will be accepted only for superusers, and no new replication connections will be accepted.'

3807 },