@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 391 lines 10.7 KB
Raw
1 ---
2 title: SQS
3 description: Add an SQS Queue, publish messages from your Lambda, and consume them from a second consumer Lambda — all wired through Alchemy bindings.
4 ---
5
6 **SQS** is AWS's general-purpose message queue — anything you can
7 serialize, dropped into a mailbox and processed asynchronously
8 somewhere else. In Alchemy it's the `Queue` resource plus
9 producer bindings (`SendMessage`, `SendMessageBatch`, and the
10 `Stream`-sink-shaped `QueueSink`) and a consumer side:
11 `SQS.consumeQueueMessages` subscribes a Lambda to the queue and hands
12 each batch of messages to your Effect, creating the event source
13 mapping and granting the receive/delete permissions for you.
14
15 The walkthrough below builds on the [Lambda](/aws/compute/lambda) and
16 [DynamoDB](/aws/data/dynamodb) pages: it adds a **Queue**, sends
17 messages to it from the existing function, and stands up a
18 **second Lambda** that consumes them.
19
20 ## Add the queue
21
22 ```diff lang="typescript"
23 // src/api.ts
24 import * as AWS from "@alchemy.run/aws";
25 import * as DynamoDB from "@alchemy.run/aws/DynamoDB";
26 +import * as SQS from "@alchemy.run/aws/SQS";
27 // ...
28
29 Effect.gen(function* () {
30 const bucket = yield* S3.Bucket("Blobs");
31 const table = yield* DynamoDB.Table("Items", { /* ... */ });
32 + const queue = yield* SQS.Queue("Jobs");
33
34 // ... existing bindings ...
35 });
36 ```
37
38 Calling `SQS.Queue("Jobs")` provisions a standard queue with the
39 defaults (4-day retention, 30-second visibility timeout, no
40 delivery delay). FIFO queues, dead-letter targets, and the rest
41 of the props can be supplied as a second argument when you need
42 them.
43
44 ## Bind SendMessage on the producer
45
46 `SQS.SendMessage(queue)` returns a callable Effect — same
47 shape as `S3.PutObject` or `DynamoDB.PutItem` — and quietly
48 attaches `sqs:SendMessage` scoped to the queue ARN:
49
50 ```diff lang="typescript"
51 const putItem = yield* DynamoDB.PutItem(table);
52 +const sendMessage = yield* SQS.SendMessage(queue);
53 ```
54
55 Let's hook it into the existing `PUT /items/:id` route so every
56 new record also drops a `job.created` message onto the queue:
57
58 ```diff lang="typescript"
59 if (request.method === "PUT") {
60 const content = yield* request.text;
61 yield* putItem({
62 Item: {
63 id: { S: id },
64 content: { S: content },
65 },
66 });
67 + yield* sendMessage({
68 + MessageBody: JSON.stringify({
69 + type: "job.created",
70 + id,
71 + content,
72 + }),
73 + });
74 return HttpServerResponse.empty({ status: 204 });
75 }
76 ```
77
78 The producer also needs the runtime layer for `SendMessage` —
79 add it to the merged layer alongside the others:
80
81 ```diff lang="typescript"
82 }).pipe(
83 Effect.provide(
84 Layer.mergeAll(
85 AWS.Lambda.BucketEventSource,
86 AWS.Lambda.TableEventSource,
87 DynamoDB.GetItemHttp,
88 DynamoDB.PutItemHttp,
89 S3.PutObjectHttp,
90 S3.GetObjectHttp,
91 + SQS.SendMessageHttp,
92 ),
93 ),
94 ),
95 ```
96
97 ## Stand up a consumer Lambda
98
99 Consumers live as their own Lambda Function. Create
100 `src/worker.ts` — bare shell first, no event source yet:
101
102 ```typescript
103 // src/worker.ts
104 import * as AWS from "@alchemy.run/aws";
105 import * as Effect from "effect/Effect";
106
107 export default class Worker extends AWS.Lambda.Function<Worker>()(
108 "JobsWorker",
109 { main: import.meta.url, functionUrl: false },
110 Effect.gen(function* () {
111 return {};
112 }),
113 ) {}
114 ```
115
116 `functionUrl: false` turns off the public Function URL every Lambda
117 gets by default. This consumer's only entrypoint is the SQS message
118 poller AWS will configure for it.
119
120 ## Share the queue between two functions
121
122 Both producer and consumer need to reference the same queue —
123 the cleanest way to do that is to lift the declaration out of
124 `api.ts` into a small shared module:
125
126 ```typescript
127 // src/queue.ts
128 import * as SQS from "@alchemy.run/aws/SQS";
129
130 export const Jobs = SQS.Queue("Jobs");
131 ```
132
133 Now both functions can import `Jobs`. Update the producer to use
134 the shared handle instead of declaring the queue inline:
135
136 ```diff lang="typescript"
137 // src/api.ts
138 +import { Jobs } from "./queue.ts";
139 // ...
140 -const queue = yield* SQS.Queue("Jobs");
141 +const queue = yield* Jobs;
142 ```
143
144 ## Resolve the queue in the consumer
145
146 Yield the same `Jobs` handle in the consumer's constructor to get
147 a typed `Queue` resource:
148
149 ```diff lang="typescript"
150 // src/worker.ts
151 import * as AWS from "@alchemy.run/aws";
152 import * as Effect from "effect/Effect";
153 +import { Jobs } from "./queue.ts";
154
155 export default class Worker extends AWS.Lambda.Function<Worker>()(
156 "JobsWorker",
157 { main: import.meta.url, functionUrl: false },
158 Effect.gen(function* () {
159 + const queue = yield* Jobs;
160 return {};
161 }),
162 ) {}
163 ```
164
165 ## Subscribe to messages
166
167 `SQS.consumeQueueMessages(queue, ...)` calls your handler with each
168 batch Lambda delivers — the `SQSEvent` — and takes back Lambda's
169 batch response. Add the smallest possible subscription — log each
170 message body:
171
172 ```diff lang="typescript"
173 // src/worker.ts
174 import * as AWS from "@alchemy.run/aws";
175 +import * as SQS from "@alchemy.run/aws/SQS";
176 +import * as Console from "effect/Console";
177 import * as Effect from "effect/Effect";
178 import { Jobs } from "./queue.ts";
179
180 export default class Worker extends AWS.Lambda.Function<Worker>()(
181 "JobsWorker",
182 { main: import.meta.url, functionUrl: false },
183 Effect.gen(function* () {
184 const queue = yield* Jobs;
185 +
186 + yield* SQS.consumeQueueMessages(
187 + queue,
188 + { batchSize: 10 },
189 + Effect.fn(function* (event) {
190 + for (const record of event.Records) {
191 + yield* Console.log(`received: ${record.body}`);
192 + }
193 + }),
194 + );
195
196 return {};
197 }),
198 ) {}
199 ```
200
201 That single `consumeQueueMessages(...)` call creates the Lambda event
202 source mapping pointing at the queue and grants the function
203 `sqs:ReceiveMessage`, `DeleteMessage`, and `GetQueueAttributes`
204 on the queue ARN. Returning nothing means every message succeeded,
205 so Lambda deletes the whole batch; if the handler fails, Lambda
206 retries the whole batch.
207
208 ## Provide the runtime layer
209
210 The consumer-side machinery for queue subscriptions ships in
211 `Lambda.QueueEventSource`. Provide it at the bottom of the
212 function:
213
214 ```diff lang="typescript"
215 +import * as Layer from "effect/Layer";
216 // ...
217 return {};
218 - }),
219 + }).pipe(Effect.provide(AWS.Lambda.QueueEventSource)),
220 ) {}
221 ```
222
223 ## Wire the worker into the Stack
224
225 The Worker needs to be yielded so Alchemy actually deploys it:
226
227 ```diff lang="typescript"
228 // alchemy.run.ts
229 import Api from "./src/api.ts";
230 +import Worker from "./src/worker.ts";
231
232 export default Alchemy.Stack(
233 "MyApp",
234 { providers: AWS.providers(), state: AWS.state() },
235 Effect.gen(function* () {
236 const api = yield* Api;
237 + yield* Worker;
238 return { url: api.functionUrl };
239 }),
240 );
241 ```
242
243 ## Deploy
244
245 ```sh
246 bun alchemy deploy
247 ```
248
249 Two functions deploy in parallel — `Api` (with `SendMessage`
250 permissions) and `JobsWorker` (with `ReceiveMessage` permissions
251 plus an event source mapping pointing at `Jobs`).
252
253 ## Verify
254
255 Trigger a write, then watch the consumer's logs:
256
257 ```sh
258 curl -X PUT --data 'hello queue' "$URL/items/q1"
259 bun alchemy logs JobsWorker --tail
260 ```
261
262 Within a couple of seconds you'll see:
263
264 ```
265 received: {"type":"job.created","id":"q1","content":"hello queue"}
266 ```
267
268 ## Bonus: retry only the failed messages
269
270 Return Lambda's `SQSBatchResponse` to report the messages that
271 failed. Lambda deletes the rest and redelivers only those once their
272 visibility timeout lapses:
273
274 ```typescript
275 yield* SQS.consumeQueueMessages(
276 queue,
277 { batchSize: 10 },
278 Effect.fn(function* (event) {
279 const failed = yield* Effect.filter(
280 event.Records,
281 (record) =>
282 doSomethingWith(record.body).pipe(
283 Effect.as(false),
284 Effect.catchCause(() => Effect.succeed(true)),
285 ),
286 { concurrency: 4 },
287 );
288 return {
289 batchItemFailures: failed.map((record) => ({ itemIdentifier: record.messageId })),
290 };
291 }),
292 );
293 ```
294
295 Up to 4 messages process concurrently, and a message that fails
296 never drags the rest of its batch back onto the queue.
297
298 ## Bonus: forward a stream to SQS with `QueueSink`
299
300 `SendMessage(queue)` is perfect for one-off sends from a
301 request handler, but if you've got a `Stream` of payloads —
302 DynamoDB change records, S3 notifications, anything — you don't
303 want to call `sendMessage` per item. **`SQS.QueueSink`** is the
304 sink-shaped counterpart: it consumes a `Stream<string>`, batches
305 items into chunks of up to 10, and ships each chunk through
306 `sqs:SendMessageBatch` for you.
307
308 Skip the per-write `sendMessage` in `PUT /items/:id` and forward
309 the existing DynamoDB stream straight into the queue instead:
310
311 ```diff lang="typescript"
312 // src/api.ts
313 import * as Stream from "effect/Stream";
314 import { Jobs } from "./queue.ts";
315 // ...
316
317 const queue = yield* Jobs;
318 const putItem = yield* DynamoDB.PutItem(table);
319 -const sendMessage = yield* SQS.SendMessage(queue);
320 +const sink = yield* SQS.QueueSink(queue);
321
322 yield* DynamoDB.consumeTableChanges(
323 table,
324 { streamViewType: "NEW_AND_OLD_IMAGES" },
325 Effect.fn(function* (event) {
326 + yield* Stream.fromIterable(event.Records).pipe(
327 + Stream.map((record) =>
328 + JSON.stringify({
329 + eventName: record.eventName,
330 + keys: record.dynamodb.Keys,
331 + newImage: record.dynamodb.NewImage,
332 + }),
333 + ),
334 + Stream.run(sink),
335 + );
336 }),
337 );
338 ```
339
340 `QueueSink(queue)` grants `sqs:SendMessage` and
341 `sqs:SendMessageBatch` on the queue ARN, then returns a
342 `Sink<void, string, readonly string[]>` you compose with
343 `Stream.run`. `QueueSinkHttp` is itself implemented on top of
344 `SendMessageBatch`, so split the merged layer into two tiers
345 with `Layer.provideMerge` — sink + event sources on top,
346 the underlying request layers + policy on the bottom:
347
348 ```diff lang="typescript"
349 }).pipe(
350 Effect.provide(
351 - Layer.mergeAll(
352 - AWS.Lambda.BucketEventSource,
353 - AWS.Lambda.TableEventSource,
354 - DynamoDB.GetItemHttp,
355 - DynamoDB.PutItemHttp,
356 - S3.PutObjectHttp,
357 - S3.GetObjectHttp,
358 - SQS.SendMessageHttp,
359 - ),
360 + Layer.mergeAll(
361 + AWS.Lambda.BucketEventSource,
362 + AWS.Lambda.TableEventSource,
363 + SQS.QueueSinkHttp,
364 + ).pipe(
365 + Layer.provideMerge(
366 + Layer.mergeAll(
367 + DynamoDB.GetItemHttp,
368 + DynamoDB.PutItemHttp,
369 + S3.PutObjectHttp,
370 + S3.GetObjectHttp,
371 + SQS.SendMessageBatchHttp,
372 + ),
373 + ),
374 + ),
375 ),
376 ),
377 ```
378
379 The consumer doesn't change — `JobsWorker` still subscribes via
380 `SQS.consumeQueueMessages(queue)` and now sees one record per DynamoDB
381 change, batched ten-at-a-time on the producer side.
382
383 ## Where next
384
385 - [Kinesis](/aws/messaging/kinesis) — stream higher-volume records:
386 ordered, sharded, and with the same batch handler.
387 - [Process DynamoDB Streams](/aws/messaging/dynamodb-streams) — the
388 change-data-capture pipeline the `QueueSink` bonus above
389 builds on.
390 - [`Queue` reference](/providers/aws/sqs/queue) — FIFO queues,
391 dead-letter targets, and every other prop.