@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 294 lines 8.69 KB
Raw
1 ---
2 title: Kinesis
3 description: Add a Kinesis Data Stream, publish records from one Lambda, and consume them in order from another — wired through the same batch event source as SQS.
4 ---
5
6 **Kinesis Data Streams** are the tool for **ordered**, sharded,
7 high-throughput data: delivery in order within a partition,
8 replay from arbitrary points in time, and multiple independent
9 consumers reading the same records. In Alchemy it's the `Stream`
10 resource plus producer bindings (`PutRecord`, `PutRecords`, and
11 the sink-shaped `StreamSink`) and a consumer side:
12 `Kinesis.consumeStreamRecords` subscribes a Lambda to the stream
13 and hands each batch of records to your Effect, creating the event
14 source mapping and granting the read permissions for you. Reach for
15 [SQS](/aws/messaging/sqs) instead when the work is unordered.
16
17 The walkthrough below builds on the [Lambda](/aws/compute/lambda),
18 [DynamoDB](/aws/data/dynamodb), and [SQS](/aws/messaging/sqs) pages: it adds a
19 Kinesis stream, publishes to it from the producer Lambda, and
20 consumes it from a new worker.
21
22 ## Add the stream
23
24 Lift the Kinesis stream into a shared module the same way you
25 did with the SQS queue, so both producer and consumer can
26 import the same handle:
27
28 ```typescript
29 // src/stream.ts
30 import * as Kinesis from "@alchemy.run/aws/Kinesis";
31
32 export const Events = Kinesis.Stream("Events", {
33 streamMode: "ON_DEMAND",
34 });
35 ```
36
37 `"ON_DEMAND"` lets AWS auto-scale the shards based on load —
38 fine for the example. Switch to `"PROVISIONED"` with an explicit
39 `shardCount` when you know your throughput.
40
41 ## Bind PutRecord on the producer
42
43 ```diff lang="typescript"
44 // src/api.ts
45 import * as DynamoDB from "@alchemy.run/aws/DynamoDB";
46 +import * as Kinesis from "@alchemy.run/aws/Kinesis";
47 import * as S3 from "@alchemy.run/aws/S3";
48 import * as SQS from "@alchemy.run/aws/SQS";
49 import { Jobs } from "./queue.ts";
50 +import { Events } from "./stream.ts";
51 // ...
52
53 const queue = yield* Jobs;
54 +const events = yield* Events;
55 const sendMessage = yield* SQS.SendMessage(queue);
56 +const putRecord = yield* Kinesis.PutRecord(events);
57 ```
58
59 The shape mirrors every other binding you've seen: `PutRecord(stream)`
60 returns a callable Effect, plus an IAM policy with
61 `kinesis:PutRecord` scoped to this stream's ARN.
62
63 ## Publish a record
64
65 Use `putRecord` inside the `PUT /items/:id` route — alongside
66 the SQS publish — to send an ordered audit record for each
67 write:
68
69 ```diff lang="typescript"
70 yield* sendMessage({
71 MessageBody: JSON.stringify({ type: "job.created", id, content }),
72 });
73 +yield* putRecord({
74 + PartitionKey: id,
75 + Data: new TextEncoder().encode(
76 + JSON.stringify({ type: "job.created", id, content, at: Date.now() }),
77 + ),
78 +});
79 ```
80
81 `PartitionKey` controls which shard a record lands in. All
82 records with the same partition key are delivered in order to
83 the same consumer instance — the entity id is exactly what you
84 want here.
85
86 `Data` is an arbitrary `Uint8Array`, not a string. JSON-encoded
87 UTF-8 bytes are the most common payload but you can ship Avro,
88 protobuf, or raw blobs without changing the binding.
89
90 ## Provide the runtime layer
91
92 ```diff lang="typescript"
93 }).pipe(
94 Effect.provide(
95 Layer.mergeAll(
96 // ... existing layers ...
97 + Kinesis.PutRecordHttp,
98 SQS.SendMessageHttp,
99 ),
100 ),
101 ),
102 ```
103
104 ## Stand up a consumer Lambda
105
106 Create `src/audit.ts` with a bare consumer shell — no event
107 source yet, just the empty class importing the shared stream
108 handle:
109
110 ```typescript
111 // src/audit.ts
112 import * as AWS from "@alchemy.run/aws";
113 import * as Effect from "effect/Effect";
114 import { Events } from "./stream.ts";
115
116 export default class Audit extends AWS.Lambda.Function<Audit>()(
117 "Audit",
118 { main: import.meta.url, functionUrl: false },
119 Effect.gen(function* () {
120 const events = yield* Events;
121 return {};
122 }),
123 ) {}
124 ```
125
126 ## Subscribe to records
127
128 `Kinesis.consumeStreamRecords(stream, ...)` mirrors the
129 [SQS `consumeQueueMessages(...)`](/aws/messaging/sqs) shape.
130 Add the smallest possible subscription — log each record's
131 partition key:
132
133 ```diff lang="typescript"
134 // src/audit.ts
135 import * as AWS from "@alchemy.run/aws";
136 +import * as Kinesis from "@alchemy.run/aws/Kinesis";
137 +import * as Console from "effect/Console";
138 import * as Effect from "effect/Effect";
139 import { Events } from "./stream.ts";
140
141 export default class Audit extends AWS.Lambda.Function<Audit>()(
142 "Audit",
143 { main: import.meta.url, functionUrl: false },
144 Effect.gen(function* () {
145 const events = yield* Events;
146 +
147 + yield* Kinesis.consumeStreamRecords(
148 + events,
149 + { startingPosition: "LATEST", batchSize: 100 },
150 + Effect.fn(function* (event) {
151 + for (const record of event.Records) {
152 + yield* Console.log(`[${record.kinesis.partitionKey}]`);
153 + }
154 + }),
155 + );
156
157 return {};
158 }),
159 ) {}
160 ```
161
162 `batchSize: 100` lets Lambda gather up to 100 records per
163 invocation; the whole batch arrives as one `KinesisStreamEvent`, and
164 Lambda checkpoints once the handler succeeds.
165 `startingPosition: "LATEST"` skips history — switch to
166 `"TRIM_HORIZON"` to replay the 24-hour retention window.
167
168 ## Decode the binary payload
169
170 Kinesis record data arrives base64-encoded inside the Lambda
171 envelope (an artifact of the event shape, not the stream
172 itself). Decode it back to bytes and on to a string, and run
173 log it alongside the partition key:
174
175 ```diff lang="typescript"
176 yield* Kinesis.consumeStreamRecords(
177 events,
178 { startingPosition: "LATEST", batchSize: 100 },
179 Effect.fn(function* (event) {
180 for (const record of event.Records) {
181 - yield* Console.log(`[${record.kinesis.partitionKey}]`);
182 + const text = new TextDecoder().decode(Buffer.from(record.kinesis.data, "base64"));
183 + yield* Console.log(`[${record.kinesis.partitionKey}] ${text}`);
184 }
185 }),
186 );
187 ```
188
189 ## Checkpoint at the first failure
190
191 Records in a shard are ordered, so report the first record that
192 failed. Lambda checkpoints everything before it and retries from
193 there:
194
195 ```diff lang="typescript"
196 yield* Kinesis.consumeStreamRecords(
197 events,
198 { startingPosition: "LATEST", batchSize: 100 },
199 Effect.fn(function* (event) {
200 for (const record of event.Records) {
201 const text = new TextDecoder().decode(Buffer.from(record.kinesis.data, "base64"));
202 - yield* Console.log(`[${record.kinesis.partitionKey}] ${text}`);
203 + const ok = yield* doSomethingWith(text).pipe(
204 + Effect.as(true),
205 + Effect.catchCause(() => Effect.succeed(false)),
206 + );
207 + if (!ok) {
208 + return { batchItemFailures: [{ itemIdentifier: record.kinesis.sequenceNumber }] };
209 + }
210 }
211 }),
212 );
213 ```
214
215 ## Provide the runtime layer
216
217 `Lambda.StreamEventSource` is the binding that enables the event
218 source mapping and grants `kinesis:GetRecords`,
219 `GetShardIterator`, `DescribeStream`, and `ListShards` on the
220 stream ARN. Provide it at the bottom of the function:
221
222 ```diff lang="typescript"
223 return {};
224 - }),
225 + }).pipe(Effect.provide(AWS.Lambda.StreamEventSource)),
226 ) {}
227 ```
228
229 ## Wire the consumer into the Stack
230
231 ```diff lang="typescript"
232 // alchemy.run.ts
233 import Api from "./src/api.ts";
234 +import Audit from "./src/audit.ts";
235 import Worker from "./src/worker.ts";
236
237 Effect.gen(function* () {
238 const api = yield* Api;
239 yield* Worker;
240 + yield* Audit;
241 return { url: api.functionUrl };
242 }),
243 ```
244
245 ## Deploy
246
247 ```sh
248 bun alchemy deploy
249 ```
250
251 Three functions now coexist: `Api` produces records, `Worker`
252 consumes the SQS queue, and `Audit` consumes the Kinesis stream.
253 Each one gets a least-privilege policy that names exactly the
254 resources its capability and `consume*` calls touched — no
255 `"*"`, no IAM JSON to maintain.
256
257 ## Verify
258
259 ```sh
260 curl -X PUT --data 'audit me' "$URL/items/k1"
261 bun alchemy logs Audit --tail
262 ```
263
264 You'll see a line like:
265
266 ```
267 [k1] {"type":"job.created","id":"k1","content":"audit me","at":1718000000000}
268 ```
269
270 Subsequent writes against the same `id` (`k1`) all land in the
271 same shard and arrive in order at the same `Audit` invocation
272 context — that's the ordering guarantee Kinesis provides per
273 partition key.
274
275 ## What you've built
276
277 If you followed the whole section, you've gone from a single
278 Lambda to a small event-driven system: a public HTTP API backed
279 by S3 blobs and a DynamoDB table, table change events on a
280 stream, an SQS work queue, and a high-throughput Kinesis
281 pipeline — all wired through the same binding pattern, with IAM
282 policies that match exactly the operations you call. Adding
283 another event source is a matter of declaring the resource and
284 pointing the matching `consume*` helper at it.
285
286 ## Where next
287
288 - [Process DynamoDB Streams](/aws/messaging/dynamodb-streams) —
289 change data capture with the same consumer surface.
290 - [React to S3 events](/aws/messaging/s3-events) — bucket
291 notifications as an Effect `Stream`.
292 - [`Stream` reference](/providers/aws/kinesis/stream) —
293 provisioned vs on-demand mode, retention, and every other
294 prop.