The best way to control Claude Code, Codex and more from any computer or phone
| 1 | --- |
| 2 | title: Event Sources |
| 3 | description: An event source is a binding that runs your Function when something happens on a resource — one call wires the event-source mapping, the permissions, and a typed handler. |
| 4 | --- |
| 5 | |
| 6 | An **Event Source** is a [Binding](/infrastructure-as-effects/binding) |
| 7 | that runs your Function when something happens on a resource. A message |
| 8 | lands on a queue, an object lands in a bucket, a row changes in a table. |
| 9 | You name the resource and a handler, and the handler runs once per |
| 10 | batch: |
| 11 | |
| 12 | ```typescript |
| 13 | import * as AWS from "@alchemy.run/aws"; |
| 14 | import * as Effect from "effect/Effect"; |
| 15 | |
| 16 | export const Orders = AWS.SQS.Queue("Orders"); |
| 17 | |
| 18 | export default AWS.Lambda.Function( |
| 19 | "OrderProcessor", |
| 20 | { main: import.meta.url }, |
| 21 | Effect.gen(function* () { |
| 22 | yield* AWS.SQS.consumeQueueMessages( |
| 23 | Orders, |
| 24 | Effect.fn(function* (event) { |
| 25 | for (const record of event.Records) { |
| 26 | yield* Effect.log(`order: ${record.body}`); |
| 27 | } |
| 28 | }), |
| 29 | ); |
| 30 | }).pipe(Effect.provide(AWS.Lambda.QueueEventSource)), |
| 31 | ); |
| 32 | ``` |
| 33 | |
| 34 | A pure consumer returns nothing from its constructor. It only declares |
| 35 | what it listens to. |
| 36 | |
| 37 | ## What one call wires |
| 38 | |
| 39 | In traditional IaC, running code when a message lands takes three |
| 40 | pieces: permission to receive and delete messages, a mapping that tells |
| 41 | the queue to invoke the Function, and a handler that unpacks the raw |
| 42 | event: |
| 43 | |
| 44 | ```typescript |
| 45 | // infrastructure: grant access and wire the trigger |
| 46 | policy: { Action: ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"], Resource: [queue.arn] }, |
| 47 | eventSourceMapping: { eventSourceArn: queue.arn, functionName: fn.name }, |
| 48 | ``` |
| 49 | |
| 50 | ```typescript |
| 51 | // handler: unpack the batch by hand |
| 52 | export const handler = async (event) => { |
| 53 | for (const record of event.Records) { |
| 54 | console.log(record.body); |
| 55 | } |
| 56 | }; |
| 57 | ``` |
| 58 | |
| 59 | `consumeQueueMessages` does all three. At deploy time it attaches that |
| 60 | statement to the Function's role, scoped to the queue's ARN, and |
| 61 | creates the event source mapping that points the queue at this |
| 62 | Function. At runtime it registers the listener that hands each |
| 63 | invocation's batch to your handler, inside a `process Orders` consumer |
| 64 | span. |
| 65 | |
| 66 | `consumeQueueMessages` is the contract and `AWS.Lambda.QueueEventSource` |
| 67 | is the Layer that satisfies it, the same split every binding has |
| 68 | ([a contract and a Layer](/infrastructure-as-effects/binding#a-contract-and-a-layer)). |
| 69 | |
| 70 | ## Report the messages that failed |
| 71 | |
| 72 | The handler takes the batch Lambda delivers and returns Lambda's batch |
| 73 | response. Name the messages that failed in `batchItemFailures`, and |
| 74 | only those are retried: |
| 75 | |
| 76 | ```typescript |
| 77 | yield* AWS.SQS.consumeQueueMessages( |
| 78 | Orders, |
| 79 | Effect.fn(function* (event) { |
| 80 | const failed = yield* Effect.filter( |
| 81 | event.Records, |
| 82 | (record) => |
| 83 | processOrder(record.body).pipe( |
| 84 | Effect.as(false), |
| 85 | Effect.catchCause(() => Effect.succeed(true)), |
| 86 | ), |
| 87 | { concurrency: "unbounded" }, |
| 88 | ); |
| 89 | return { |
| 90 | batchItemFailures: failed.map((record) => ({ itemIdentifier: record.messageId })), |
| 91 | }; |
| 92 | }), |
| 93 | ); |
| 94 | ``` |
| 95 | |
| 96 | Return nothing when every message succeeded. If the handler fails, |
| 97 | the whole batch is retried. Kinesis and DynamoDB Streams take the same |
| 98 | response, naming the first record that failed so Lambda checkpoints |
| 99 | before it. |
| 100 | |
| 101 | ## Compose with a Stream |
| 102 | |
| 103 | The batch is plain data, so any Effect works on it. Map table changes |
| 104 | to JSON and run them straight into a queue: |
| 105 | |
| 106 | ```typescript |
| 107 | const sink = yield* AWS.SQS.QueueSink(Outbound); |
| 108 | |
| 109 | yield* AWS.DynamoDB.consumeTableChanges( |
| 110 | Jobs, |
| 111 | { streamViewType: "NEW_AND_OLD_IMAGES", startingPosition: "TRIM_HORIZON" }, |
| 112 | Effect.fn(function* (event) { |
| 113 | yield* Stream.fromIterable(event.Records).pipe( |
| 114 | Stream.map((record) => |
| 115 | JSON.stringify({ |
| 116 | eventName: record.eventName, |
| 117 | keys: record.dynamodb.Keys, |
| 118 | }), |
| 119 | ), |
| 120 | Stream.run(sink), |
| 121 | ); |
| 122 | }), |
| 123 | ); |
| 124 | ``` |
| 125 | |
| 126 | That `sink` is a [Sink](/infrastructure-as-effects/sinks), the |
| 127 | write-side dual of an event source. Its permissions are generated the |
| 128 | same way. |
| 129 | |
| 130 | ## The same shape on Cloudflare |
| 131 | |
| 132 | Consume a Cloudflare queue the same way. The consumer settings go on |
| 133 | the call, and the handler receives a `Stream` of typed messages: |
| 134 | |
| 135 | ```typescript |
| 136 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 137 | import * as Effect from "effect/Effect"; |
| 138 | import * as Stream from "effect/Stream"; |
| 139 | import * as HttpServerResponse from "effect/http/HttpServerResponse"; |
| 140 | |
| 141 | export const Orders = Cloudflare.Queues.Queue("Orders"); |
| 142 | |
| 143 | export default Cloudflare.Worker( |
| 144 | "OrderProcessor", |
| 145 | { main: import.meta.url }, |
| 146 | Effect.gen(function* () { |
| 147 | yield* Cloudflare.Queues.consumeQueueMessages<{ text: string }>( |
| 148 | Orders, |
| 149 | { batchSize: 10, maxRetries: 3, retryDelay: "1 second" }, |
| 150 | (stream) => |
| 151 | Stream.runForEach(stream, (msg) => Effect.log(msg.body.text)), |
| 152 | ); |
| 153 | |
| 154 | return { |
| 155 | fetch: Effect.gen(function* () { |
| 156 | return HttpServerResponse.text("ok"); |
| 157 | }), |
| 158 | }; |
| 159 | }).pipe(Effect.provide(Cloudflare.Queues.EventSourceLive)), |
| 160 | ); |
| 161 | ``` |
| 162 | |
| 163 | There is no IAM here. At deploy time the call creates the queue |
| 164 | consumer that dispatches batches to this Worker. At runtime it |
| 165 | registers the `queue` listener. |
| 166 | |
| 167 | ## Batching and retries |
| 168 | |
| 169 | Set `batchSize` and `maximumBatchingWindow` on the call to shape each |
| 170 | invocation's batch: |
| 171 | |
| 172 | ```typescript |
| 173 | yield* AWS.SQS.consumeQueueMessages( |
| 174 | Orders, |
| 175 | { batchSize: 10, maximumBatchingWindow: "5 seconds" }, |
| 176 | Effect.fn(function* (event) { |
| 177 | for (const record of event.Records) { |
| 178 | yield* Effect.log(record.body); |
| 179 | } |
| 180 | }), |
| 181 | ); |
| 182 | ``` |
| 183 | |
| 184 | If your handler fails, the queue redelivers the batch. Stream sources |
| 185 | like Kinesis and DynamoDB add settings that bound a poison record's |
| 186 | blast radius: |
| 187 | |
| 188 | ```typescript |
| 189 | yield* AWS.DynamoDB.consumeTableChanges( |
| 190 | Jobs, |
| 191 | { |
| 192 | streamViewType: "NEW_AND_OLD_IMAGES", |
| 193 | startingPosition: "TRIM_HORIZON", |
| 194 | bisectBatchOnFunctionError: true, |
| 195 | maximumRetryAttempts: 3, |
| 196 | }, |
| 197 | Effect.fn(function* (event) { |
| 198 | for (const record of event.Records) { |
| 199 | yield* Effect.log(record.eventName); |
| 200 | } |
| 201 | }), |
| 202 | ); |
| 203 | ``` |
| 204 | |
| 205 | On Cloudflare, set the retry policy on the call. A batch is acked when |
| 206 | your handler succeeds, retried `maxRetries` times when it fails, then |
| 207 | dead-lettered: |
| 208 | |
| 209 | ```typescript |
| 210 | const deadLetters = yield* Cloudflare.Queues.Queue("OrdersDeadLetters"); |
| 211 | |
| 212 | yield* Cloudflare.Queues.consumeQueueMessages<{ text: string }>( |
| 213 | Orders, |
| 214 | { maxRetries: 3, retryDelay: "1 second", deadLetterQueue: deadLetters }, |
| 215 | (stream) => Stream.runForEach(stream, (msg) => Effect.log(msg.body.text)), |
| 216 | ); |
| 217 | ``` |
| 218 | |
| 219 | Nothing reads the dead-letter queue unless you consume it, so a failed |
| 220 | message expires there silently. Subscribe to it the same way |
| 221 | ([example](/cloudflare/messaging/queues#consume-the-dead-letter-queue)). |
| 222 | |
| 223 | Call `msg.ack()` or `msg.retry()` inside the handler to decide per |
| 224 | message. |
| 225 | |
| 226 | ## Sources that deliver one event at a time |
| 227 | |
| 228 | A source that delivers one event at a time hands your handler the event |
| 229 | directly. A cron trigger passes one `controller` per fire: |
| 230 | |
| 231 | ```typescript |
| 232 | export default Cloudflare.Worker( |
| 233 | "Nightly", |
| 234 | { main: import.meta.url }, |
| 235 | Effect.gen(function* () { |
| 236 | yield* Cloudflare.Workers.cron("0 12 * * *", (controller) => |
| 237 | Effect.log(`scheduled at ${controller.scheduledTime}`), |
| 238 | ); |
| 239 | |
| 240 | return { |
| 241 | fetch: Effect.gen(function* () { |
| 242 | return HttpServerResponse.text("ok"); |
| 243 | }), |
| 244 | }; |
| 245 | }).pipe(Effect.provide(Cloudflare.Workers.CronEventSourceLive)), |
| 246 | ); |
| 247 | ``` |
| 248 | |
| 249 | At deploy time the cron expression is attached to the Worker. At runtime |
| 250 | the `scheduled` listener is registered. GitHub works the same way: |
| 251 | `consumeRepositoryEvents` creates a repository webhook pointing at the |
| 252 | Worker's URL, and the runtime listener verifies each delivery's HMAC |
| 253 | signature before calling your handler. |
| 254 | |
| 255 | ## Every event source |
| 256 | |
| 257 | The shape never changes. Only the `consume*` callable, the record type, |
| 258 | and the Layer you provide differ: |
| 259 | |
| 260 | ```typescript |
| 261 | // AWS: handlers receive each batch and return Lambda's batch response |
| 262 | yield* AWS.SQS.consumeQueueMessages(queue, fn); |
| 263 | yield* AWS.Kinesis.consumeStreamRecords(stream, props, fn); |
| 264 | yield* AWS.DynamoDB.consumeTableChanges(table, props, fn); |
| 265 | |
| 266 | // AWS: handlers receive a Stream of notifications |
| 267 | yield* AWS.S3.consumeBucketEvents(bucket, fn); |
| 268 | yield* AWS.SNS.consumeTopicNotifications(topic, fn); |
| 269 | yield* AWS.EventBridge.consumeBusEvents(bus, pattern, fn); |
| 270 | |
| 271 | // Cloudflare |
| 272 | yield* Cloudflare.Queues.consumeQueueMessages(queue, fn); |
| 273 | yield* Cloudflare.Workers.cron(expression, fn); |
| 274 | yield* GitHub.consumeRepositoryEvents(props, fn); |
| 275 | ``` |
| 276 | |
| 277 | EventBridge can also route matching events to another resource instead |
| 278 | of consuming them locally: |
| 279 | |
| 280 | ```typescript |
| 281 | yield* AWS.EventBridge.events(bus, { source: ["my.app"] }).toQueue(queue); |
| 282 | ``` |
| 283 | |
| 284 | Each source has its own guide. On AWS: [SQS](/aws/messaging/sqs), |
| 285 | [Kinesis](/aws/messaging/kinesis), |
| 286 | [DynamoDB Streams](/aws/messaging/dynamodb-streams), |
| 287 | [S3 Events](/aws/messaging/s3-events), [SNS](/aws/messaging/sns), and |
| 288 | [EventBridge](/aws/messaging/eventbridge). On Cloudflare: |
| 289 | [Queues](/cloudflare/messaging/queues), [Cron](/cloudflare/messaging/cron), |
| 290 | and [GitHub Events](/cloudflare/messaging/github-events). |
| 291 | |
| 292 | ## Where next |
| 293 | |
| 294 | - [Sinks](/infrastructure-as-effects/sinks) — the write-side dual: |
| 295 | resources as Effect `Sink`s, with batching and IAM generated the |
| 296 | same way. |
| 297 | - [Bindings](/infrastructure-as-effects/binding) — the deploy-time |
| 298 | mechanics every event source is built on. |
| 299 | - [Runtime](/infrastructure-as-effects/runtime) — |
| 300 | the Effectful Constructor these examples live inside. |