Alert if zrepl replication hasn't succeeded recently
zrepl is currently configured to take snapshots every 30 minutes, so if we go a full day without a successful replication, I'd say something is pretty wrong.
Jeremy Fleischman committed
Apr 13, 2026 at 11:20 UTC
fcca2101a434907f3ec9526f5fd0cf95f790bca2
1 file changed
+23
-2
build/pluto/prometheus/exporters/zrepl.nix
+23
-2
@@ -1,4 +1,4 @@
1
-{ ... }:
1
+{ pkgs, ... }:
2
3
{
4
services.prometheus = {
@@ -16,6 +16,27 @@
16
}
17
];
18
19
- # TODO: alert on `zrepl_replication_last_successful` being too long ago.
19
+ ruleFiles = [
20
+ (pkgs.writeText "zrepl.rules" (
21
+ builtins.toJSON {
22
+ groups = [
23
+ {
24
+ name = "zrepl";
25
+ rules = [
26
+ {
27
+ alert = "ZreplLongTimeNoSuccess";
28
+ expr = ''
29
+ time() - zrepl_replication_last_successful > ${toString (6 * 60 * 60)}
30
+ '';
31
+ for = "6h";
32
+ labels.severity = "warning";
33
+ annotations.summary = "zrepl job {{ $labels.zrepl_job }} has not succeeded recently.";
34
+ }
35
+ ];
36
+ }
37
+ ];
38
+ }
39
+ ))
40
+ ];
41
};
42
}