The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 215 lines 6.14 KB
Raw
1 ---
2 title: Process DynamoDB Streams
3 description: Enable a DynamoDB Stream on your table and consume each batch of change records from the same Lambda.
4 sidebar:
5 order: 2
6 ---
7
8 The DynamoDB table from the previous part stores records, but
9 writes happen in the dark — there's no way for downstream systems
10 to react. **DynamoDB Streams** fix that: every change to the
11 table emits a change record (insert / modify / remove) you can
12 stream into a Lambda. In Alchemy this is one helper call.
13
14 ## Subscribe to the stream
15
16 Start with the smallest possible subscription: log just the
17 event name (`INSERT` / `MODIFY` / `REMOVE`) for each change.
18 Add it alongside the existing bindings:
19
20 ```diff lang="typescript"
21 // src/api.ts
22 import * as AWS from "@alchemy.run/aws";
23 import * as DynamoDB from "@alchemy.run/aws/DynamoDB";
24 +import * as Console from "effect/Console";
25 import * as Effect from "effect/Effect";
26 import * as Layer from "effect/Layer";
27 // ...
28
29 const table = yield* DynamoDB.Table("Items", { /* ... */ });
30 const putItem = yield* DynamoDB.PutItem(table);
31 const getItem = yield* DynamoDB.GetItem(table);
32
33 +yield* DynamoDB.consumeTableChanges(
34 + table,
35 + Effect.fn(function* (event) {
36 + for (const record of event.Records) {
37 + yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
38 + }
39 + }),
40 +);
41 ```
42
43 That single `consumeTableChanges(...)` call enables a stream on the table,
44 creates a Lambda `EventSourceMapping` that polls it, and adds
45 the IAM grants for `dynamodb:GetRecords`, `GetShardIterator`,
46 `DescribeStream`, and `ListStreams` — all from the call site. The
47 handler gets each batch Lambda delivers; when it succeeds, Lambda
48 checkpoints past the batch.
49
50 ## Pick the view type
51
52 By default the stream emits keys only. Most handlers want the
53 full row before and after the change — set `streamViewType` to
54 `NEW_AND_OLD_IMAGES`:
55
56 ```diff lang="typescript"
57 yield* DynamoDB.consumeTableChanges(
58 table,
59 + { streamViewType: "NEW_AND_OLD_IMAGES" },
60 Effect.fn(function* (event) {
61 for (const record of event.Records) {
62 yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
63 }
64 }),
65 );
66 ```
67
68 The other options (`KEYS_ONLY`, `NEW_IMAGE`, `OLD_IMAGE`) trade
69 payload size for less detail — pick the smallest one your
70 handler actually needs.
71
72 ## Control where consumption starts
73
74 `startingPosition: "LATEST"` is the default and ignores history;
75 the consumer only sees records that arrive after the
76 event-source mapping is enabled. Switch to `"TRIM_HORIZON"` to
77 replay everything currently in the stream's 24-hour retention
78 window:
79
80 ```diff lang="typescript"
81 yield* DynamoDB.consumeTableChanges(
82 table,
83 {
84 streamViewType: "NEW_AND_OLD_IMAGES",
85 + startingPosition: "TRIM_HORIZON",
86 + batchSize: 10,
87 },
88 Effect.fn(function* (event) {
89 for (const record of event.Records) {
90 yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
91 }
92 }),
93 );
94 ```
95
96 `batchSize` caps the number of records the function sees per
97 invocation. Larger batches are more efficient; smaller batches
98 fail in smaller blast radii.
99
100 ## Type the row payload
101
102 DynamoDB stream records use the same attribute-value shape as
103 `GetItem`/`PutItem`. Annotate the handler's `event` with the row
104 shape and TypeScript will help you destructure
105 `NewImage`/`OldImage`:
106
107 ```diff lang="typescript"
108 yield* DynamoDB.consumeTableChanges(
109 table,
110 {
111 streamViewType: "NEW_AND_OLD_IMAGES",
112 startingPosition: "TRIM_HORIZON",
113 batchSize: 10,
114 },
115 - Effect.fn(function* (event) {
116 + Effect.fn(function* (
117 + event: DynamoDB.StreamEvent<{ id: { S: string }; content: { S: string } }>,
118 + ) {
119 for (const record of event.Records) {
120 - yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`);
121 + yield* Console.log(
122 + `${record.eventName}: ${record.dynamodb.Keys?.id?.S} -> ` +
123 + `${record.dynamodb.NewImage?.content.S ?? "(deleted)"}`,
124 + );
125 }
126 }),
127 );
128 ```
129
130 ## Provide the runtime layer
131
132 The Lambda-specific implementation of `TableEventSource` is in
133 `Lambda.TableEventSource`. Add it to the merged layer:
134
135 ```diff lang="typescript"
136 }).pipe(
137 Effect.provide(
138 Layer.mergeAll(
139 AWS.Lambda.BucketEventSource,
140 + AWS.Lambda.TableEventSource,
141 DynamoDB.GetItemHttp,
142 DynamoDB.PutItemHttp,
143 S3.PutObjectHttp,
144 S3.GetObjectHttp,
145 ),
146 ),
147 ),
148 ```
149
150 `TableEventSource` is the binding that mutates the table's stream
151 configuration during deploy and creates the event source mapping.
152 You don't have to think about either — they're both side effects
153 of `consumeTableChanges(...)`.
154
155 ## Deploy
156
157 ```sh
158 bun alchemy deploy
159 ```
160
161 Alchemy enables the stream on the table (takes a few seconds for
162 the table to settle) then creates the event source mapping. Once
163 the mapping is `Enabled`, every `PutItem`/`UpdateItem`/`DeleteItem`
164 shows up in your function's CloudWatch logs.
165
166 ## Verify
167
168 Reuse the `PUT /items/:id` route to publish changes, then tail
169 the function logs to confirm they arrive:
170
171 ```sh
172 bun alchemy logs Api --tail
173 ```
174
175 In another terminal:
176
177 ```sh
178 curl -X PUT --data 'first' "$URL/items/one"
179 curl -X PUT --data 'second' "$URL/items/two"
180 curl -X PUT --data 'updated' "$URL/items/one"
181 ```
182
183 You'll see three `MODIFY`/`INSERT` log lines stream past — one
184 per write. Stream records typically arrive within 1–2 seconds of
185 the originating change.
186
187 ## Checkpoint at the first failure
188
189 Change records in a shard are ordered, so when one fails, report
190 it. Lambda checkpoints every record before it and retries from
191 there, instead of replaying the whole batch:
192
193 ```typescript
194 yield* DynamoDB.consumeTableChanges(
195 table,
196 { streamViewType: "NEW_AND_OLD_IMAGES" },
197 Effect.fn(function* (event) {
198 for (const record of event.Records) {
199 const ok = yield* writeToWarehouse(record).pipe(
200 Effect.as(true),
201 Effect.catchCause(() => Effect.succeed(false)),
202 );
203 if (!ok) {
204 return { batchItemFailures: [{ itemIdentifier: record.dynamodb.SequenceNumber! }] };
205 }
206 }
207 }),
208 );
209 ```
210
211 If the handler fails instead, Lambda retries the whole batch.
212
213 Next we'll add an [SQS Queue](/aws/messaging/sqs) so the Lambda
214 can fan changes (or anything else) out to a different consumer
215 process and pick them up downstream.