The best way to control Claude Code, Codex and more from any computer or phone
| 1 | --- |
| 2 | title: Process DynamoDB Streams |
| 3 | description: Enable a DynamoDB Stream on your table and consume each batch of change records from the same Lambda. |
| 4 | sidebar: |
| 5 | order: 2 |
| 6 | --- |
| 7 | |
| 8 | The DynamoDB table from the previous part stores records, but |
| 9 | writes happen in the dark — there's no way for downstream systems |
| 10 | to react. **DynamoDB Streams** fix that: every change to the |
| 11 | table emits a change record (insert / modify / remove) you can |
| 12 | stream into a Lambda. In Alchemy this is one helper call. |
| 13 | |
| 14 | ## Subscribe to the stream |
| 15 | |
| 16 | Start with the smallest possible subscription: log just the |
| 17 | event name (`INSERT` / `MODIFY` / `REMOVE`) for each change. |
| 18 | Add it alongside the existing bindings: |
| 19 | |
| 20 | ```diff lang="typescript" |
| 21 | // src/api.ts |
| 22 | import * as AWS from "@alchemy.run/aws"; |
| 23 | import * as DynamoDB from "@alchemy.run/aws/DynamoDB"; |
| 24 | +import * as Console from "effect/Console"; |
| 25 | import * as Effect from "effect/Effect"; |
| 26 | import * as Layer from "effect/Layer"; |
| 27 | // ... |
| 28 | |
| 29 | const table = yield* DynamoDB.Table("Items", { /* ... */ }); |
| 30 | const putItem = yield* DynamoDB.PutItem(table); |
| 31 | const getItem = yield* DynamoDB.GetItem(table); |
| 32 | |
| 33 | +yield* DynamoDB.consumeTableChanges( |
| 34 | + table, |
| 35 | + Effect.fn(function* (event) { |
| 36 | + for (const record of event.Records) { |
| 37 | + yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`); |
| 38 | + } |
| 39 | + }), |
| 40 | +); |
| 41 | ``` |
| 42 | |
| 43 | That single `consumeTableChanges(...)` call enables a stream on the table, |
| 44 | creates a Lambda `EventSourceMapping` that polls it, and adds |
| 45 | the IAM grants for `dynamodb:GetRecords`, `GetShardIterator`, |
| 46 | `DescribeStream`, and `ListStreams` — all from the call site. The |
| 47 | handler gets each batch Lambda delivers; when it succeeds, Lambda |
| 48 | checkpoints past the batch. |
| 49 | |
| 50 | ## Pick the view type |
| 51 | |
| 52 | By default the stream emits keys only. Most handlers want the |
| 53 | full row before and after the change — set `streamViewType` to |
| 54 | `NEW_AND_OLD_IMAGES`: |
| 55 | |
| 56 | ```diff lang="typescript" |
| 57 | yield* DynamoDB.consumeTableChanges( |
| 58 | table, |
| 59 | + { streamViewType: "NEW_AND_OLD_IMAGES" }, |
| 60 | Effect.fn(function* (event) { |
| 61 | for (const record of event.Records) { |
| 62 | yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`); |
| 63 | } |
| 64 | }), |
| 65 | ); |
| 66 | ``` |
| 67 | |
| 68 | The other options (`KEYS_ONLY`, `NEW_IMAGE`, `OLD_IMAGE`) trade |
| 69 | payload size for less detail — pick the smallest one your |
| 70 | handler actually needs. |
| 71 | |
| 72 | ## Control where consumption starts |
| 73 | |
| 74 | `startingPosition: "LATEST"` is the default and ignores history; |
| 75 | the consumer only sees records that arrive after the |
| 76 | event-source mapping is enabled. Switch to `"TRIM_HORIZON"` to |
| 77 | replay everything currently in the stream's 24-hour retention |
| 78 | window: |
| 79 | |
| 80 | ```diff lang="typescript" |
| 81 | yield* DynamoDB.consumeTableChanges( |
| 82 | table, |
| 83 | { |
| 84 | streamViewType: "NEW_AND_OLD_IMAGES", |
| 85 | + startingPosition: "TRIM_HORIZON", |
| 86 | + batchSize: 10, |
| 87 | }, |
| 88 | Effect.fn(function* (event) { |
| 89 | for (const record of event.Records) { |
| 90 | yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`); |
| 91 | } |
| 92 | }), |
| 93 | ); |
| 94 | ``` |
| 95 | |
| 96 | `batchSize` caps the number of records the function sees per |
| 97 | invocation. Larger batches are more efficient; smaller batches |
| 98 | fail in smaller blast radii. |
| 99 | |
| 100 | ## Type the row payload |
| 101 | |
| 102 | DynamoDB stream records use the same attribute-value shape as |
| 103 | `GetItem`/`PutItem`. Annotate the handler's `event` with the row |
| 104 | shape and TypeScript will help you destructure |
| 105 | `NewImage`/`OldImage`: |
| 106 | |
| 107 | ```diff lang="typescript" |
| 108 | yield* DynamoDB.consumeTableChanges( |
| 109 | table, |
| 110 | { |
| 111 | streamViewType: "NEW_AND_OLD_IMAGES", |
| 112 | startingPosition: "TRIM_HORIZON", |
| 113 | batchSize: 10, |
| 114 | }, |
| 115 | - Effect.fn(function* (event) { |
| 116 | + Effect.fn(function* ( |
| 117 | + event: DynamoDB.StreamEvent<{ id: { S: string }; content: { S: string } }>, |
| 118 | + ) { |
| 119 | for (const record of event.Records) { |
| 120 | - yield* Console.log(`${record.eventName}: ${record.dynamodb.Keys?.id?.S}`); |
| 121 | + yield* Console.log( |
| 122 | + `${record.eventName}: ${record.dynamodb.Keys?.id?.S} -> ` + |
| 123 | + `${record.dynamodb.NewImage?.content.S ?? "(deleted)"}`, |
| 124 | + ); |
| 125 | } |
| 126 | }), |
| 127 | ); |
| 128 | ``` |
| 129 | |
| 130 | ## Provide the runtime layer |
| 131 | |
| 132 | The Lambda-specific implementation of `TableEventSource` is in |
| 133 | `Lambda.TableEventSource`. Add it to the merged layer: |
| 134 | |
| 135 | ```diff lang="typescript" |
| 136 | }).pipe( |
| 137 | Effect.provide( |
| 138 | Layer.mergeAll( |
| 139 | AWS.Lambda.BucketEventSource, |
| 140 | + AWS.Lambda.TableEventSource, |
| 141 | DynamoDB.GetItemHttp, |
| 142 | DynamoDB.PutItemHttp, |
| 143 | S3.PutObjectHttp, |
| 144 | S3.GetObjectHttp, |
| 145 | ), |
| 146 | ), |
| 147 | ), |
| 148 | ``` |
| 149 | |
| 150 | `TableEventSource` is the binding that mutates the table's stream |
| 151 | configuration during deploy and creates the event source mapping. |
| 152 | You don't have to think about either — they're both side effects |
| 153 | of `consumeTableChanges(...)`. |
| 154 | |
| 155 | ## Deploy |
| 156 | |
| 157 | ```sh |
| 158 | bun alchemy deploy |
| 159 | ``` |
| 160 | |
| 161 | Alchemy enables the stream on the table (takes a few seconds for |
| 162 | the table to settle) then creates the event source mapping. Once |
| 163 | the mapping is `Enabled`, every `PutItem`/`UpdateItem`/`DeleteItem` |
| 164 | shows up in your function's CloudWatch logs. |
| 165 | |
| 166 | ## Verify |
| 167 | |
| 168 | Reuse the `PUT /items/:id` route to publish changes, then tail |
| 169 | the function logs to confirm they arrive: |
| 170 | |
| 171 | ```sh |
| 172 | bun alchemy logs Api --tail |
| 173 | ``` |
| 174 | |
| 175 | In another terminal: |
| 176 | |
| 177 | ```sh |
| 178 | curl -X PUT --data 'first' "$URL/items/one" |
| 179 | curl -X PUT --data 'second' "$URL/items/two" |
| 180 | curl -X PUT --data 'updated' "$URL/items/one" |
| 181 | ``` |
| 182 | |
| 183 | You'll see three `MODIFY`/`INSERT` log lines stream past — one |
| 184 | per write. Stream records typically arrive within 1–2 seconds of |
| 185 | the originating change. |
| 186 | |
| 187 | ## Checkpoint at the first failure |
| 188 | |
| 189 | Change records in a shard are ordered, so when one fails, report |
| 190 | it. Lambda checkpoints every record before it and retries from |
| 191 | there, instead of replaying the whole batch: |
| 192 | |
| 193 | ```typescript |
| 194 | yield* DynamoDB.consumeTableChanges( |
| 195 | table, |
| 196 | { streamViewType: "NEW_AND_OLD_IMAGES" }, |
| 197 | Effect.fn(function* (event) { |
| 198 | for (const record of event.Records) { |
| 199 | const ok = yield* writeToWarehouse(record).pipe( |
| 200 | Effect.as(true), |
| 201 | Effect.catchCause(() => Effect.succeed(false)), |
| 202 | ); |
| 203 | if (!ok) { |
| 204 | return { batchItemFailures: [{ itemIdentifier: record.dynamodb.SequenceNumber! }] }; |
| 205 | } |
| 206 | } |
| 207 | }), |
| 208 | ); |
| 209 | ``` |
| 210 | |
| 211 | If the handler fails instead, Lambda retries the whole batch. |
| 212 | |
| 213 | Next we'll add an [SQS Queue](/aws/messaging/sqs) so the Lambda |
| 214 | can fan changes (or anything else) out to a different consumer |
| 215 | process and pick them up downstream. |