The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 255 lines 9.36 KB
Raw
1 ---
2 title: Sinks
3 description: A Sink is the write-side dual of an Event Source — a Binding that exposes a resource as an Effect Sink, batching writes into the batch API with least-privilege IAM.
4 ---
5
6 A **Sink** is the dual of an [Event Source](/infrastructure-as-effects/event-sources):
7 a [Binding](/infrastructure-as-effects/binding) for *writing* that
8 exposes a resource as an Effect
9 [`Sink`](https://effect.website/docs/sink/introduction/). Where an
10 Event Source hands you records, a Sink accepts a
11 `Stream` and drains it into the resource's batch API
12 (`SendMessageBatch`, `PublishBatch`, `PutRecords`) — emitting the
13 minimal IAM to go with it.
14
15 ## A Sink in a constructor
16
17 Like every Binding, a Sink is declared in the
18 [Effectful Constructor](/infrastructure-as-effects/runtime#the-effectful-constructor)
19 and used in the interface it returns:
20
21 ```typescript
22 import * as AWS from "@alchemy.run/aws";
23 import * as Effect from "effect/Effect";
24 import * as Layer from "effect/Layer";
25 import * as Stream from "effect/Stream";
26 import { HttpServerRequest } from "effect/http/HttpServerRequest";
27 import * as HttpServerResponse from "effect/http/HttpServerResponse";
28 import { OutboundQueue } from "./queue.ts";
29
30 export default class Publisher extends AWS.Lambda.Function<Publisher>()(
31 "Publisher",
32 { main: import.meta.url, functionUrl: true },
33 Effect.gen(function* () {
34 const sink = yield* AWS.SQS.QueueSink(OutboundQueue);
35 // → Sink<void, string, readonly string[], never>
36 // → policy: sqs:SendMessage + sqs:SendMessageBatch on OutboundQueue's ARN
37
38 return {
39 fetch: Effect.gen(function* () {
40 const request = yield* HttpServerRequest;
41 const body = (yield* request.json) as { messages: string[] };
42
43 yield* Stream.fromIterable(body.messages).pipe(Stream.run(sink));
44
45 return yield* HttpServerResponse.json({
46 ok: true,
47 count: body.messages.length,
48 });
49 }),
50 };
51 }).pipe(
52 Effect.provide(
53 AWS.SQS.QueueSinkHttp.pipe(
54 Layer.provideMerge(AWS.SQS.SendMessageBatchHttp),
55 ),
56 ),
57 ),
58 ) {}
59 ```
60
61 `QueueSink` is the contract; the provided `AWS.SQS.QueueSinkHttp`
62 Layer implements it over `SendMessageBatch` — itself a contract,
63 satisfied by `SendMessageBatchHttp`. That is why the provide chains a
64 `Layer.provideMerge`: implementation Layers can depend on other
65 contracts. The Layer attaches the least-privilege policy statements
66 to the Function's role; the `yield*` returns a plain Effect `Sink` —
67 no `QUEUE_URL` env var to read, no `SendMessageBatchCommand`
68 chunking by hand. The contract-and-Layer split is the same rule
69 every Binding follows — see
70 [Bindings › A contract and a Layer](/infrastructure-as-effects/binding#a-contract-and-a-layer).
71
72 ## Element types
73
74 Sinks are typed by the element they accept, so the type system stops
75 you from feeding the wrong shape in:
76
77 | Sink | Element | Batch API | Grants |
78 | ------------------------------------ | ----------------------------- | ------------------ | ----------------------------------------- |
79 | `SQS.QueueSink(queue)` | `string` | `SendMessageBatch` | `sqs:SendMessage`, `sqs:SendMessageBatch` |
80 | `SNS.TopicSink(topic)` | `string` | `PublishBatch` | `sns:Publish` |
81 | `Kinesis.StreamSink(stream)` | `PutRecordsRequestEntry` | `PutRecords` | `kinesis:PutRecords` |
82 | `Cloudflare.Queues.QueueSink(queue)` | message body (any JSON value) | `sendBatch` | `queue` binding on the Worker |
83
84 Every AWS policy is scoped to the exact resource ARN; the Cloudflare
85 sink binds only the one queue into the Worker.
86
87 `QueueSink` and `TopicSink` take message bodies as `string`.
88 `StreamSink` takes raw `PutRecordsRequestEntry` values, so you keep
89 control of `PartitionKey`:
90
91 ```typescript
92 export default class Ingest extends AWS.Lambda.Function<Ingest>()(
93 "Ingest",
94 { main: import.meta.url, functionUrl: true },
95 Effect.gen(function* () {
96 const sink = yield* AWS.Kinesis.StreamSink(ClickStream);
97
98 return {
99 fetch: Effect.gen(function* () {
100 const request = yield* HttpServerRequest;
101 const body = (yield* request.json) as {
102 records: Array<{ partitionKey: string; data: string }>;
103 };
104
105 yield* Stream.fromIterable(
106 body.records.map((record) => ({
107 PartitionKey: record.partitionKey,
108 Data: new TextEncoder().encode(record.data),
109 })),
110 ).pipe(Stream.run(sink));
111
112 return yield* HttpServerResponse.json({ ok: true });
113 }),
114 };
115 }),
116 // (layers elided — same shape as Publisher above:
117 // AWS.Kinesis.StreamSinkHttp over AWS.Kinesis.PutRecordsHttp)
118 ) {}
119 ```
120
121 ## Batching
122
123 A Sink drains its input chunk by chunk — each chunk of the stream
124 becomes **one** call to the underlying batch API. To control batch
125 size, rechunk the stream before running it into the sink:
126
127 ```typescript
128 yield* Stream.fromIterable(messages).pipe(
129 Stream.rechunk(10), // one SendMessageBatch call per 10 messages
130 Stream.run(sink),
131 );
132 ```
133
134 ## Source → transform → sink
135
136 Where the model pays off is when you compose a Sink with an
137 [Event Source](/infrastructure-as-effects/event-sources). Lift the
138 batch into a `Stream` and the whole pipe-and-transform pipeline is one
139 expression inside the constructor:
140
141 ```typescript
142 export default class OrderRelay extends AWS.Lambda.Function<OrderRelay>()(
143 "OrderRelay",
144 { main: import.meta.url },
145 Effect.gen(function* () {
146 const sink = yield* AWS.SQS.QueueSink(OutboundQueue);
147
148 yield* AWS.DynamoDB.consumeTableChanges(
149 OrdersTable,
150 {
151 streamViewType: "NEW_AND_OLD_IMAGES",
152 startingPosition: "LATEST",
153 },
154 Effect.fn(function* (event: AWS.DynamoDB.StreamEvent<Order>) {
155 yield* Stream.fromIterable(event.Records).pipe(
156 Stream.filterMap((r) => Option.fromNullable(r.dynamodb.NewImage)),
157 Stream.filter((order) => order.status === "PAID"),
158 Stream.map((order) => JSON.stringify({ orderId: order.id })),
159 Stream.run(sink),
160 );
161 }),
162 );
163 }).pipe(
164 Effect.provide(
165 Layer.mergeAll(AWS.Lambda.TableEventSource, AWS.SQS.QueueSinkHttp).pipe(
166 Layer.provideMerge(AWS.SQS.SendMessageBatchHttp),
167 ),
168 ),
169 ),
170 ) {}
171 ```
172
173 Two binding calls, one pipeline. The provided Layers attach **both**
174 sides of the policy to the Function's role —
175 `dynamodb:DescribeStream` / `GetRecords` / `GetShardIterator` on
176 `OrdersTable`'s stream, `sqs:SendMessage` / `SendMessageBatch` on
177 `OutboundQueue` — and `AWS.Lambda.TableEventSource` wires the
178 event-source mapping.
179
180 Drop in `Effect.retry`, `Stream.throttle`, `Stream.groupedWithin`,
181 or `Stream.mapEffect` anywhere along the chain — they're all the
182 same `Stream` you'd write in a plain Effect program.
183
184 ## Cloudflare Queues
185
186 `Cloudflare.Queues.QueueSink(queue)` is the same shape on a Worker.
187 Each element is one message body, and each stream chunk becomes one
188 `sendBatch` call. Chunks over Cloudflare's batch limits (100
189 messages / 256 KB) are split into consecutive batches, in order:
190
191 ```typescript
192 import * as Cloudflare from "@alchemy.run/cloudflare";
193 import * as Effect from "effect/Effect";
194 import * as Stream from "effect/Stream";
195 import * as HttpServerResponse from "effect/http/HttpServerResponse";
196 import { Clicks } from "./queues.ts";
197
198 export default Cloudflare.Worker(
199 "Api",
200 { main: import.meta.url },
201 Effect.gen(function* () {
202 const sink = yield* Cloudflare.Queues.QueueSink(Clicks);
203 // → Sink<void, unknown, never, SendError, RuntimeContext>
204
205 return {
206 fetch: Effect.gen(function* () {
207 yield* Stream.fromIterable(events).pipe(Stream.run(sink));
208 return HttpServerResponse.empty({ status: 202 });
209 }),
210 };
211 }).pipe(Effect.provide(Cloudflare.Queues.QueueSinkBinding)),
212 );
213 ```
214
215 `QueueSinkBinding` layers the sink over the native producer binding
216 (`WriteQueueBinding`), so there is nothing else to provide. It also
217 works under `alchemy dev`, where the binding targets the local queue
218 broker. `QueueSinkHttp` (scoped API token) and `QueueSinkLocal`
219 (current credentials, for Actions) layer it over the matching
220 `WriteQueue` implementations.
221
222 Paired with `consumeQueueMessages`, a queue-to-queue pipeline is one
223 expression:
224
225 ```typescript
226 Effect.gen(function* () {
227 const enriched = yield* Cloudflare.Queues.QueueSink(EnrichedClicks);
228
229 yield* Cloudflare.Queues.consumeQueueMessages<Click>(Clicks, (messages) =>
230 messages.pipe(
231 Stream.map((message) => message.body),
232 Stream.filter((click) => click.button === "buy"),
233 Stream.map((click) => ({ ...click, receivedAt: Date.now() })),
234 Stream.run(enriched),
235 ),
236 );
237 }).pipe(
238 Effect.provide(
239 Layer.mergeAll(
240 Cloudflare.Queues.EventSourceLive,
241 Cloudflare.Queues.QueueSinkBinding,
242 ),
243 ),
244 );
245 ```
246
247 A failed `sendBatch` fails the handler, so Cloudflare retries the
248 whole consumed batch. Messages already forwarded before the failure
249 are sent again, which fits the queue's at-least-once delivery.
250
251 ## Where next
252
253 - [Event Sources](/infrastructure-as-effects/event-sources) — the read-side dual: resources as `Stream`s.
254 - [Bindings](/infrastructure-as-effects/binding) — the deploy-time mechanics every Sink shares.
255 - [Layers](/infrastructure-as-effects/layers) — hide Sinks behind a service interface.