The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 205 lines 6.19 KB
Raw
1 ---
2 title: "Alerting: monitors & notifiers"
3 description: Turn APL queries into alerts — Threshold, MatchEvent, and AnomalyDetection monitors wired to Slack, email, PagerDuty, and webhook notifiers, all as resources in your Stack.
4 ---
5
6 An Axiom `Monitor` is a saved APL query that runs on a fixed cadence
7 and fires when its condition is met. A `Notifier` is the destination
8 it fires to — Slack, email, PagerDuty, a webhook. In alchemy both are
9 resources in your Stack, so a threshold change or a new on-call
10 channel is a reviewable diff, not a console edit.
11
12 Everything below runs inside your Stack's `Effect.gen` — see
13 [Setup](/axiom/setup) for credentials and the
14 [Axiom overview](/axiom) for datasets like `app-traces` and
15 `app-logs` used in the queries.
16
17 ## Create a notifier
18
19 ```typescript
20 import * as Axiom from "@alchemy.run/axiom";
21
22 const slack = yield* Axiom.Notifier("ops-slack", {
23 name: "ops-channel",
24 properties: {
25 slack: { slackUrl: process.env.SLACK_WEBHOOK_URL! },
26 },
27 });
28 ```
29
30 Exactly one channel goes under `properties` — here `slack` with an
31 incoming webhook URL. The resource's `id` output is what monitors
32 target.
33
34 ## Threshold: alert when a value crosses a line
35
36 ```typescript
37 yield* Axiom.Monitor("error-rate", {
38 name: "High error rate",
39 description: "Fires when error count exceeds 100/5m",
40 type: "Threshold",
41 aplQuery: `
42 ['app-traces']
43 | where status >= 500
44 | summarize count() by bin_auto(_time)
45 `,
46 operator: "Above",
47 threshold: 100,
48 intervalMinutes: 5,
49 rangeMinutes: 5,
50 alertOnNoData: false,
51 resolvable: true,
52 notifierIds: [slack.id],
53 });
54 ```
55
56 Every `intervalMinutes`, the monitor runs `aplQuery` over the last
57 `rangeMinutes` and compares the aggregate against `threshold` using
58 `operator` (`Above`, `AboveOrEqual`, `Below`, `BelowOrEqual`, or
59 `AboveOrBelow`). `resolvable: true` sends a resolution notice when
60 the value drops back under the line; `alertOnNoData` controls whether
61 an empty result counts as a problem.
62
63 `notifierIds: [slack.id]` is the wiring: `slack.id` is an output
64 attribute of the notifier flowing into the monitor's inputs, so
65 alchemy deploys the notifier first and fills in the real id.
66
67 ## MatchEvent: alert on every matching event
68
69 ```typescript
70 yield* Axiom.Monitor("panics", {
71 name: "Service panic",
72 type: "MatchEvent",
73 aplQuery: `['app-logs'] | where message contains "panic:"`,
74 intervalMinutes: 1,
75 rangeMinutes: 1,
76 notifierIds: [slack.id],
77 });
78 ```
79
80 A `MatchEvent` monitor has no threshold — it fires for every event
81 the query matches. Use it for rare, individually-actionable events
82 like panics or failed deploys, where one occurrence is already worth
83 a page.
84
85 ## AnomalyDetection: alert on deviation from baseline
86
87 ```typescript
88 yield* Axiom.Monitor("traffic-anomaly", {
89 name: "Traffic anomaly",
90 type: "AnomalyDetection",
91 aplQuery: `['app-traces'] | summarize count() by bin_auto(_time)`,
92 compareDays: 7,
93 tolerance: 25, // %
94 intervalMinutes: 15,
95 rangeMinutes: 15,
96 notifierIds: [slack.id],
97 });
98 ```
99
100 Instead of a static threshold, the monitor learns a baseline from the
101 same window over the previous `compareDays` days and fires when the
102 current result deviates by more than `tolerance` percent. Good for
103 "traffic looks wrong" signals where the right absolute number changes
104 by hour and weekday.
105
106 Changing a monitor's `type` triggers a replacement; every other prop
107 updates in place.
108
109 ## Fan out to multiple notifiers
110
111 ```typescript
112 const pagerduty = yield* Axiom.Notifier("pagerduty", {
113 name: "primary-oncall",
114 properties: {
115 pagerduty: { routingKey: process.env.PAGERDUTY_ROUTING_KEY!, token: "" },
116 },
117 });
118
119 yield* Axiom.Monitor("error-rate", {
120 // ...same props as above
121 notifierIds: [slack.id, pagerduty.id],
122 });
123 ```
124
125 `notifierIds` takes any number of ids, so one monitor can post to the
126 team channel and page on-call at the same time. Notifier channels
127 cover Slack, email (`email: { emails: [...] }`), PagerDuty, Opsgenie,
128 Discord, Microsoft Teams, and generic webhooks.
129
130 ## Custom webhook with a templated body
131
132 ```typescript
133 yield* Axiom.Notifier("incident-webhook", {
134 name: "incident.io",
135 properties: {
136 customWebhook: {
137 url: "https://api.incident.io/v2/alert_events",
138 headers: { "Content-Type": "application/json" },
139 secretHeaders: { Authorization: `Bearer ${process.env.INCIDENT_TOKEN}` },
140 body: '{"title": "{{.Monitor.Name}}", "status": "firing"}',
141 },
142 },
143 });
144 ```
145
146 For anything without a built-in channel, `customWebhook` gives you
147 full control over the request: static `headers`, `secretHeaders` for
148 credentials, and a `body` template with access to monitor fields like
149 `{{.Monitor.Name}}`.
150
151 ## Derived alert fields with VirtualField
152
153 ```typescript
154 yield* Axiom.VirtualField("status-class", {
155 dataset: "app-traces",
156 name: "status_class",
157 description: "HTTP response class bucket",
158 expression: 'strcat(tostring(toint(status / 100)), "xx")',
159 type: "string",
160 });
161 ```
162
163 A `VirtualField` is a saved APL expression that appears as a derived
164 column on its `dataset` at query time. Define common computations —
165 status classes, latency buckets, parsed JSON paths — once, instead of
166 repeating the expression in every monitor and dashboard. A virtual
167 field is bound to one `dataset`; changing the dataset triggers a
168 replacement.
169
170 Monitors on that dataset can then query the derived column directly:
171
172 ```typescript
173 yield* Axiom.Monitor("5xx-rate", {
174 name: "5xx rate",
175 type: "Threshold",
176 aplQuery: `
177 ['app-traces']
178 | where status_class == "5xx"
179 | summarize count() by bin_auto(_time)
180 `,
181 operator: "Above",
182 threshold: 50,
183 intervalMinutes: 5,
184 rangeMinutes: 5,
185 notifierIds: [slack.id],
186 });
187 ```
188
189 If the definition of "5xx" ever changes, you update the virtual
190 field's `expression` — every monitor and dashboard built on
191 `status_class` follows.
192
193 ## Where next
194
195 - [Ingest](/axiom/data/ingest) — datasets, ingest tokens, and wiring OTLP
196 endpoints into your Workers and Functions.
197 - [Dashboards](/axiom/guides/dashboards) — chart the same queries
198 your monitors run.
199 - [Axiom overview](/axiom) — all Axiom resources at a glance.
200
201 Reference:
202
203 - [Monitor API reference](/providers/axiom/monitor)
204 - [Notifier API reference](/providers/axiom/notifier)
205 - [VirtualField API reference](/providers/axiom/virtualfield)