The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 299 lines 10.3 KB
Raw
1 ---
2 title: K2 streams
3 description: Cloudflare K2 is a durable, replayable log. Produce records from Workers with the WriteStream binding, then consume them from a long-running host with consumeStreamRecords.
4 ---
5
6 A K2 **stream** is a durable, ordered log of records stored on R2. Producers
7 append records; readers consume them through **subscriptions**. Each
8 subscription tracks its own position, so many readers can read the same
9 records independently, and records stay in the stream for its retention
10 period whether or not anyone has read them.
11
12 Reach for K2 when several consumers need the same events, when you need to
13 replay history, or when you want a log you can read at your own pace. Reach
14 for [Queues](/cloudflare/messaging/queues) when each message is a unit of work
15 that one consumer handles once, with retries and a dead-letter queue.
16
17 | | Queues | K2 |
18 | --- | --- | --- |
19 | Model | Work queue: a message is removed once it is acknowledged | Log: records stay for the retention period |
20 | Readers | One consumer | Any number of subscriptions, each reading every record |
21 | Delivery | Pushed to a consumer Worker | Pulled in leased batches over HTTP |
22 | Consume from a Worker | Yes | Not yet |
23 | Replay | No | From the start of retention (`startAt: "earliest"`) |
24
25 :::note
26 K2 cannot push records to a Worker yet; push consumers are on Cloudflare's
27 roadmap. Produce from Workers, and consume from a long-running host such as
28 a Fly, ECS, EC2, Railway, Hetzner, or Docker service. Alchemy will add a
29 Worker consumer once Cloudflare supports one.
30 :::
31
32 ## Create a stream
33
34 ```typescript
35 // src/Orders.ts
36 import * as Cloudflare from "@alchemy.run/cloudflare";
37
38 export const Orders = Cloudflare.K2.Stream("Orders", {
39 retention: "7 days",
40 });
41 ```
42
43 The stream name is generated from the stack, stage, and logical ID. Changing
44 `retention` updates the stream in place. It accepts any `Duration.Input`
45 between one hour and 30 days, and defaults to seven days.
46
47 A stream accepts records from Workers by default. Its HTTP input is off unless
48 you turn it on:
49
50 ```typescript
51 Cloudflare.K2.Stream("Orders", { http: true }); // authenticated
52 Cloudflare.K2.Stream("Orders", { http: { authentication: false } }); // public
53 Cloudflare.K2.Stream("Orders", { http: { cors: ["https://shop.dev"] } });
54 ```
55
56 Turn on the HTTP input when a host outside Cloudflare produces to the stream,
57 or when you produce with `WriteStreamHttp`.
58
59 ## Produce from a Worker
60
61 In the Worker's init phase, ask `Cloudflare.K2.WriteStream` for a client. Its
62 `send` appends a batch of records atomically.
63
64 ```typescript
65 // src/Api.ts
66 import * as Cloudflare from "@alchemy.run/cloudflare";
67 import * as Effect from "effect/Effect";
68 import { HttpServerRequest } from "effect/http/HttpServerRequest";
69 import * as HttpServerResponse from "effect/http/HttpServerResponse";
70 import { Orders } from "./Orders.ts";
71
72 export default class Api extends Cloudflare.Worker<Api>()(
73 "Api",
74 { main: import.meta.url },
75 Effect.gen(function* () {
76 const orders = yield* Cloudflare.K2.WriteStream(Orders);
77
78 return {
79 fetch: Effect.gen(function* () {
80 const request = yield* HttpServerRequest;
81 const body = yield* request.text;
82 yield* orders.send([{ content: body, headers: { source: "api" } }]);
83 return HttpServerResponse.empty({ status: 202 });
84 }),
85 };
86 }).pipe(Effect.provide(Cloudflare.K2.WriteStreamBinding)),
87 ) {}
88 ```
89
90 A record's `content` is a string (sent as UTF-8), a `Uint8Array`, or an
91 `ArrayBuffer`. `headers` is an optional map of up to 32 string pairs. A batch
92 can be up to 5 MB, and each record up to 1 MB.
93
94 `WriteStreamBinding` sends through the Worker's native `k2` binding, so it
95 needs no API token.
96
97 ## Type records with a schema
98
99 Declare the stream with an Effect Schema, and every binding on it is typed by
100 that schema:
101
102 ```diff lang="typescript"
103 // src/Orders.ts
104 import * as Cloudflare from "@alchemy.run/cloudflare";
105 +import * as Schema from "effect/Schema";
106 +
107 +export const Order = Schema.Struct({
108 + orderId: Schema.String,
109 + amountCents: Schema.Number,
110 +});
111
112 export const Orders = Cloudflare.K2.Stream("Orders", {
113 retention: "7 days",
114 + schema: Order,
115 });
116 ```
117
118 Now `orders.send` takes `Order` values. The client encodes each one as JSON
119 and sets a `content-type: application/json` header:
120
121 ```typescript
122 yield* orders.send([{ orderId: "o_1", amountCents: 4200 }]);
123 ```
124
125 K2 itself stores bytes, so the schema is a client-side codec. Changing it
126 never updates or replaces the stream. Producers and consumers read the schema
127 from the same stream, so they can't disagree about the record type.
128
129 ## Handle produce errors
130
131 Failures are typed. Two of them tell you whether the batch was stored:
132
133 ```typescript
134 yield* orders.send(batch).pipe(
135 // The batch was not stored. Retrying it is safe.
136 Effect.catchTag("K2Unavailable", () => retryLater(batch)),
137 // The batch may or may not have been stored, and K2 does not deduplicate.
138 // Alchemy never retries this one for you.
139 Effect.catchTag("K2AppendOutcomeUnknown", () => reconcile(batch)),
140 );
141 ```
142
143 The native binding and the HTTP path report the same error tags, so this code
144 works with either layer.
145
146 ## Drain a Stream into K2
147
148 `Cloudflare.K2.StreamSink` exposes the stream as an Effect
149 [Sink](/infrastructure-as-effects/sinks). It packs records into requests of
150 up to 5 MB and retries only the errors that mean the batch was not stored.
151
152 ```typescript
153 const sink = yield* Cloudflare.K2.StreamSink(Orders);
154 yield* Stream.fromIterable(pending).pipe(Stream.run(sink));
155 ```
156
157 Provide `Cloudflare.K2.StreamSinkBinding` alongside `WriteStreamBinding`.
158
159 ## Produce from outside a Worker
160
161 The producer code stays the same on every host. Only the layer changes:
162
163 - `WriteStreamBinding` and `StreamSinkBinding` run in Cloudflare Workers and
164 send through the native `k2` binding.
165 - `WriteStreamHttp` and `StreamSinkHttp` run on any host, such as Lambda or a
166 container. They send through the stream's HTTP input with a scoped "K2
167 Produce" token that Alchemy mints at deploy.
168 - `WriteStreamLocal` and `StreamSinkLocal` run in scripts and Actions. They
169 send through the stream's HTTP input with your current Cloudflare
170 credentials.
171
172 The HTTP and local layers need the stream's HTTP input turned on
173 (`http: true`).
174
175 ## Consume from a long-running host
176
177 `Cloudflare.K2.consumeStreamRecords(stream, handler)` subscribes a host to a
178 stream. The handler receives each batch as an Effect `Stream`:
179
180 ```typescript
181 // src/Fraud.ts
182 import * as Cloudflare from "@alchemy.run/cloudflare";
183 import * as Fly from "@alchemy.run/fly";
184 import * as Effect from "effect/Effect";
185 import * as Layer from "effect/Layer";
186 import * as Stream from "effect/Stream";
187 import { Orders } from "./Orders.ts";
188
189 export default class Fraud extends Fly.Service<Fraud>()(
190 "Fraud",
191 { main: import.meta.url, region: "iad" },
192 Effect.gen(function* () {
193 yield* Cloudflare.K2.consumeStreamRecords(
194 Orders,
195 { startAt: "earliest", concurrency: 4 },
196 (records) =>
197 records.pipe(
198 Stream.map((record) => record.value),
199 Stream.filter((order) => order.amountCents > 100_000),
200 Stream.runForEach((order) => Effect.log(`review ${order.orderId}`)),
201 ),
202 );
203
204 return {};
205 }).pipe(
206 Effect.provide(
207 Layer.provideMerge(
208 Cloudflare.K2.StreamEventSourcePolling,
209 Cloudflare.K2.ReadSubscriptionHttp,
210 ),
211 ),
212 ),
213 ) {}
214 ```
215
216 At deploy time, `consumeStreamRecords` creates a subscription owned by the
217 host. At runtime, `StreamEventSourcePolling` runs `concurrency` pollers. Each
218 poller leases a batch of up to `maxRecords` (default 100) and runs the
219 handler:
220
221 - When the handler succeeds, the batch is acknowledged.
222 - When it fails, the failure is logged and the batch is released for
223 redelivery.
224 - While the handler runs, the five-minute lease is extended every two
225 minutes.
226 - When a batch comes back empty, the poller waits `idleBackoff` (default one
227 second) before it polls again.
228
229 `ReadSubscriptionHttp` reads with a scoped "K2 Consume" token that Alchemy
230 mints at deploy. `StreamEventSourcePolling` needs a host with a long-lived
231 process (Fly, ECS, EC2, Railway, Hetzner, Docker), so it doesn't work in a
232 Worker.
233
234 For a stream with a schema, each record's JSON content is decoded into
235 `record.value`. For a stream without one, each record is
236 `{ content: Uint8Array, headers, timestamp }`.
237
238 ## Share a subscription between hosts
239
240 By default every consuming host gets its own subscription, so each host sees
241 every record. To split records between hosts instead, declare a
242 `Subscription` and pass it to each one:
243
244 ```typescript
245 export const FraudWorkers = Cloudflare.K2.Subscription("FraudWorkers", {
246 stream: Orders,
247 startAt: "latest",
248 });
249 ```
250
251 ```diff lang="typescript"
252 yield* Cloudflare.K2.consumeStreamRecords(
253 Orders,
254 - { startAt: "earliest", concurrency: 4 },
255 + { subscription: FraudWorkers, concurrency: 4 },
256 ```
257
258 Hosts sharing a subscription compete for batches. A subscription allows up
259 to 128 leases at a time. Subscriptions can't be changed after they are
260 created, so changing `startAt` replaces the subscription.
261
262 ## Control leases yourself
263
264 `Cloudflare.K2.ReadSubscription` is the client `consumeStreamRecords` uses
265 internally. Use it when you need your own loop:
266
267 ```typescript
268 const inbox = yield* Cloudflare.K2.ReadSubscription(FraudWorkers);
269
270 const batch = yield* inbox.consume({ workerId: "worker-1", maxRecords: 100 });
271 if (Option.isSome(batch)) {
272 yield* handle(batch.value.records).pipe(
273 Effect.andThen(inbox.ack(batch.value)),
274 Effect.catchCause(() => inbox.nack(batch.value)),
275 );
276 }
277 ```
278
279 `consume` returns `Option.none()` when there are no records. A worker that
280 already holds a lease gets the same batch back. `extend(batch)` renews a
281 lease for another five minutes, and fails with `K2LeaseLost` once the lease
282 is gone.
283
284 ## Where next
285
286 Related:
287
288 - [Queues](/cloudflare/messaging/queues): a work queue with push delivery to
289 Workers, retries, and dead-letter queues.
290 - [Pipelines](/cloudflare/data/pipelines): land events as files or Iceberg
291 tables instead of consuming them yourself.
292 - [Workers](/cloudflare/compute/workers): the runtime that produces records.
293 - [Sinks](/infrastructure-as-effects/sinks): the Effect `Sink` model behind
294 `StreamSink`.
295
296 Reference:
297
298 - [K2 API reference](/providers/cloudflare/k2): `Stream`, `Subscription`,
299 `WriteStream`, `StreamSink`, `ReadSubscription`, and the polling layer.