@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 300 lines 9.29 KB
Raw
1 ---
2 title: Event Sources
3 description: An event source is a binding that runs your Function when something happens on a resource — one call wires the event-source mapping, the permissions, and a typed handler.
4 ---
5
6 An **Event Source** is a [Binding](/infrastructure-as-effects/binding)
7 that runs your Function when something happens on a resource. A message
8 lands on a queue, an object lands in a bucket, a row changes in a table.
9 You name the resource and a handler, and the handler runs once per
10 batch:
11
12 ```typescript
13 import * as AWS from "@alchemy.run/aws";
14 import * as Effect from "effect/Effect";
15
16 export const Orders = AWS.SQS.Queue("Orders");
17
18 export default AWS.Lambda.Function(
19 "OrderProcessor",
20 { main: import.meta.url },
21 Effect.gen(function* () {
22 yield* AWS.SQS.consumeQueueMessages(
23 Orders,
24 Effect.fn(function* (event) {
25 for (const record of event.Records) {
26 yield* Effect.log(`order: ${record.body}`);
27 }
28 }),
29 );
30 }).pipe(Effect.provide(AWS.Lambda.QueueEventSource)),
31 );
32 ```
33
34 A pure consumer returns nothing from its constructor. It only declares
35 what it listens to.
36
37 ## What one call wires
38
39 In traditional IaC, running code when a message lands takes three
40 pieces: permission to receive and delete messages, a mapping that tells
41 the queue to invoke the Function, and a handler that unpacks the raw
42 event:
43
44 ```typescript
45 // infrastructure: grant access and wire the trigger
46 policy: { Action: ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"], Resource: [queue.arn] },
47 eventSourceMapping: { eventSourceArn: queue.arn, functionName: fn.name },
48 ```
49
50 ```typescript
51 // handler: unpack the batch by hand
52 export const handler = async (event) => {
53 for (const record of event.Records) {
54 console.log(record.body);
55 }
56 };
57 ```
58
59 `consumeQueueMessages` does all three. At deploy time it attaches that
60 statement to the Function's role, scoped to the queue's ARN, and
61 creates the event source mapping that points the queue at this
62 Function. At runtime it registers the listener that hands each
63 invocation's batch to your handler, inside a `process Orders` consumer
64 span.
65
66 `consumeQueueMessages` is the contract and `AWS.Lambda.QueueEventSource`
67 is the Layer that satisfies it, the same split every binding has
68 ([a contract and a Layer](/infrastructure-as-effects/binding#a-contract-and-a-layer)).
69
70 ## Report the messages that failed
71
72 The handler takes the batch Lambda delivers and returns Lambda's batch
73 response. Name the messages that failed in `batchItemFailures`, and
74 only those are retried:
75
76 ```typescript
77 yield* AWS.SQS.consumeQueueMessages(
78 Orders,
79 Effect.fn(function* (event) {
80 const failed = yield* Effect.filter(
81 event.Records,
82 (record) =>
83 processOrder(record.body).pipe(
84 Effect.as(false),
85 Effect.catchCause(() => Effect.succeed(true)),
86 ),
87 { concurrency: "unbounded" },
88 );
89 return {
90 batchItemFailures: failed.map((record) => ({ itemIdentifier: record.messageId })),
91 };
92 }),
93 );
94 ```
95
96 Return nothing when every message succeeded. If the handler fails,
97 the whole batch is retried. Kinesis and DynamoDB Streams take the same
98 response, naming the first record that failed so Lambda checkpoints
99 before it.
100
101 ## Compose with a Stream
102
103 The batch is plain data, so any Effect works on it. Map table changes
104 to JSON and run them straight into a queue:
105
106 ```typescript
107 const sink = yield* AWS.SQS.QueueSink(Outbound);
108
109 yield* AWS.DynamoDB.consumeTableChanges(
110 Jobs,
111 { streamViewType: "NEW_AND_OLD_IMAGES", startingPosition: "TRIM_HORIZON" },
112 Effect.fn(function* (event) {
113 yield* Stream.fromIterable(event.Records).pipe(
114 Stream.map((record) =>
115 JSON.stringify({
116 eventName: record.eventName,
117 keys: record.dynamodb.Keys,
118 }),
119 ),
120 Stream.run(sink),
121 );
122 }),
123 );
124 ```
125
126 That `sink` is a [Sink](/infrastructure-as-effects/sinks), the
127 write-side dual of an event source. Its permissions are generated the
128 same way.
129
130 ## The same shape on Cloudflare
131
132 Consume a Cloudflare queue the same way. The consumer settings go on
133 the call, and the handler receives a `Stream` of typed messages:
134
135 ```typescript
136 import * as Cloudflare from "@alchemy.run/cloudflare";
137 import * as Effect from "effect/Effect";
138 import * as Stream from "effect/Stream";
139 import * as HttpServerResponse from "effect/http/HttpServerResponse";
140
141 export const Orders = Cloudflare.Queues.Queue("Orders");
142
143 export default Cloudflare.Worker(
144 "OrderProcessor",
145 { main: import.meta.url },
146 Effect.gen(function* () {
147 yield* Cloudflare.Queues.consumeQueueMessages<{ text: string }>(
148 Orders,
149 { batchSize: 10, maxRetries: 3, retryDelay: "1 second" },
150 (stream) =>
151 Stream.runForEach(stream, (msg) => Effect.log(msg.body.text)),
152 );
153
154 return {
155 fetch: Effect.gen(function* () {
156 return HttpServerResponse.text("ok");
157 }),
158 };
159 }).pipe(Effect.provide(Cloudflare.Queues.EventSourceLive)),
160 );
161 ```
162
163 There is no IAM here. At deploy time the call creates the queue
164 consumer that dispatches batches to this Worker. At runtime it
165 registers the `queue` listener.
166
167 ## Batching and retries
168
169 Set `batchSize` and `maximumBatchingWindow` on the call to shape each
170 invocation's batch:
171
172 ```typescript
173 yield* AWS.SQS.consumeQueueMessages(
174 Orders,
175 { batchSize: 10, maximumBatchingWindow: "5 seconds" },
176 Effect.fn(function* (event) {
177 for (const record of event.Records) {
178 yield* Effect.log(record.body);
179 }
180 }),
181 );
182 ```
183
184 If your handler fails, the queue redelivers the batch. Stream sources
185 like Kinesis and DynamoDB add settings that bound a poison record's
186 blast radius:
187
188 ```typescript
189 yield* AWS.DynamoDB.consumeTableChanges(
190 Jobs,
191 {
192 streamViewType: "NEW_AND_OLD_IMAGES",
193 startingPosition: "TRIM_HORIZON",
194 bisectBatchOnFunctionError: true,
195 maximumRetryAttempts: 3,
196 },
197 Effect.fn(function* (event) {
198 for (const record of event.Records) {
199 yield* Effect.log(record.eventName);
200 }
201 }),
202 );
203 ```
204
205 On Cloudflare, set the retry policy on the call. A batch is acked when
206 your handler succeeds, retried `maxRetries` times when it fails, then
207 dead-lettered:
208
209 ```typescript
210 const deadLetters = yield* Cloudflare.Queues.Queue("OrdersDeadLetters");
211
212 yield* Cloudflare.Queues.consumeQueueMessages<{ text: string }>(
213 Orders,
214 { maxRetries: 3, retryDelay: "1 second", deadLetterQueue: deadLetters },
215 (stream) => Stream.runForEach(stream, (msg) => Effect.log(msg.body.text)),
216 );
217 ```
218
219 Nothing reads the dead-letter queue unless you consume it, so a failed
220 message expires there silently. Subscribe to it the same way
221 ([example](/cloudflare/messaging/queues#consume-the-dead-letter-queue)).
222
223 Call `msg.ack()` or `msg.retry()` inside the handler to decide per
224 message.
225
226 ## Sources that deliver one event at a time
227
228 A source that delivers one event at a time hands your handler the event
229 directly. A cron trigger passes one `controller` per fire:
230
231 ```typescript
232 export default Cloudflare.Worker(
233 "Nightly",
234 { main: import.meta.url },
235 Effect.gen(function* () {
236 yield* Cloudflare.Workers.cron("0 12 * * *", (controller) =>
237 Effect.log(`scheduled at ${controller.scheduledTime}`),
238 );
239
240 return {
241 fetch: Effect.gen(function* () {
242 return HttpServerResponse.text("ok");
243 }),
244 };
245 }).pipe(Effect.provide(Cloudflare.Workers.CronEventSourceLive)),
246 );
247 ```
248
249 At deploy time the cron expression is attached to the Worker. At runtime
250 the `scheduled` listener is registered. GitHub works the same way:
251 `consumeRepositoryEvents` creates a repository webhook pointing at the
252 Worker's URL, and the runtime listener verifies each delivery's HMAC
253 signature before calling your handler.
254
255 ## Every event source
256
257 The shape never changes. Only the `consume*` callable, the record type,
258 and the Layer you provide differ:
259
260 ```typescript
261 // AWS: handlers receive each batch and return Lambda's batch response
262 yield* AWS.SQS.consumeQueueMessages(queue, fn);
263 yield* AWS.Kinesis.consumeStreamRecords(stream, props, fn);
264 yield* AWS.DynamoDB.consumeTableChanges(table, props, fn);
265
266 // AWS: handlers receive a Stream of notifications
267 yield* AWS.S3.consumeBucketEvents(bucket, fn);
268 yield* AWS.SNS.consumeTopicNotifications(topic, fn);
269 yield* AWS.EventBridge.consumeBusEvents(bus, pattern, fn);
270
271 // Cloudflare
272 yield* Cloudflare.Queues.consumeQueueMessages(queue, fn);
273 yield* Cloudflare.Workers.cron(expression, fn);
274 yield* GitHub.consumeRepositoryEvents(props, fn);
275 ```
276
277 EventBridge can also route matching events to another resource instead
278 of consuming them locally:
279
280 ```typescript
281 yield* AWS.EventBridge.events(bus, { source: ["my.app"] }).toQueue(queue);
282 ```
283
284 Each source has its own guide. On AWS: [SQS](/aws/messaging/sqs),
285 [Kinesis](/aws/messaging/kinesis),
286 [DynamoDB Streams](/aws/messaging/dynamodb-streams),
287 [S3 Events](/aws/messaging/s3-events), [SNS](/aws/messaging/sns), and
288 [EventBridge](/aws/messaging/eventbridge). On Cloudflare:
289 [Queues](/cloudflare/messaging/queues), [Cron](/cloudflare/messaging/cron),
290 and [GitHub Events](/cloudflare/messaging/github-events).
291
292 ## Where next
293
294 - [Sinks](/infrastructure-as-effects/sinks) — the write-side dual:
295 resources as Effect `Sink`s, with batching and IAM generated the
296 same way.
297 - [Bindings](/infrastructure-as-effects/binding) — the deploy-time
298 mechanics every event source is built on.
299 - [Runtime](/infrastructure-as-effects/runtime) —
300 the Effectful Constructor these examples live inside.