The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 04:45 UTC
| 1 | --- |
| 2 | title: K2 streams |
| 3 | description: Cloudflare K2 is a durable, replayable log. Produce records from Workers with the WriteStream binding, then consume them from a long-running host with consumeStreamRecords. |
| 4 | --- |
| 5 | |
| 6 | A K2 **stream** is a durable, ordered log of records stored on R2. Producers |
| 7 | append records; readers consume them through **subscriptions**. Each |
| 8 | subscription tracks its own position, so many readers can read the same |
| 9 | records independently, and records stay in the stream for its retention |
| 10 | period whether or not anyone has read them. |
| 11 | |
| 12 | Reach for K2 when several consumers need the same events, when you need to |
| 13 | replay history, or when you want a log you can read at your own pace. Reach |
| 14 | for [Queues](/cloudflare/messaging/queues) when each message is a unit of work |
| 15 | that one consumer handles once, with retries and a dead-letter queue. |
| 16 | |
| 17 | | | Queues | K2 | |
| 18 | | --- | --- | --- | |
| 19 | | Model | Work queue: a message is removed once it is acknowledged | Log: records stay for the retention period | |
| 20 | | Readers | One consumer | Any number of subscriptions, each reading every record | |
| 21 | | Delivery | Pushed to a consumer Worker | Pulled in leased batches over HTTP | |
| 22 | | Consume from a Worker | Yes | Not yet | |
| 23 | | Replay | No | From the start of retention (`startAt: "earliest"`) | |
| 24 | |
| 25 | :::note |
| 26 | K2 cannot push records to a Worker yet; push consumers are on Cloudflare's |
| 27 | roadmap. Produce from Workers, and consume from a long-running host such as |
| 28 | a Fly, ECS, EC2, Railway, Hetzner, or Docker service. Alchemy will add a |
| 29 | Worker consumer once Cloudflare supports one. |
| 30 | ::: |
| 31 | |
| 32 | ## Create a stream |
| 33 | |
| 34 | ```typescript |
| 35 | // src/Orders.ts |
| 36 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 37 | |
| 38 | export const Orders = Cloudflare.K2.Stream("Orders", { |
| 39 | retention: "7 days", |
| 40 | }); |
| 41 | ``` |
| 42 | |
| 43 | The stream name is generated from the stack, stage, and logical ID. Changing |
| 44 | `retention` updates the stream in place. It accepts any `Duration.Input` |
| 45 | between one hour and 30 days, and defaults to seven days. |
| 46 | |
| 47 | A stream accepts records from Workers by default. Its HTTP input is off unless |
| 48 | you turn it on: |
| 49 | |
| 50 | ```typescript |
| 51 | Cloudflare.K2.Stream("Orders", { http: true }); // authenticated |
| 52 | Cloudflare.K2.Stream("Orders", { http: { authentication: false } }); // public |
| 53 | Cloudflare.K2.Stream("Orders", { http: { cors: ["https://shop.dev"] } }); |
| 54 | ``` |
| 55 | |
| 56 | Turn on the HTTP input when a host outside Cloudflare produces to the stream, |
| 57 | or when you produce with `WriteStreamHttp`. |
| 58 | |
| 59 | ## Produce from a Worker |
| 60 | |
| 61 | In the Worker's init phase, ask `Cloudflare.K2.WriteStream` for a client. Its |
| 62 | `send` appends a batch of records atomically. |
| 63 | |
| 64 | ```typescript |
| 65 | // src/Api.ts |
| 66 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 67 | import * as Effect from "effect/Effect"; |
| 68 | import { HttpServerRequest } from "effect/http/HttpServerRequest"; |
| 69 | import * as HttpServerResponse from "effect/http/HttpServerResponse"; |
| 70 | import { Orders } from "./Orders.ts"; |
| 71 | |
| 72 | export default class Api extends Cloudflare.Worker<Api>()( |
| 73 | "Api", |
| 74 | { main: import.meta.url }, |
| 75 | Effect.gen(function* () { |
| 76 | const orders = yield* Cloudflare.K2.WriteStream(Orders); |
| 77 | |
| 78 | return { |
| 79 | fetch: Effect.gen(function* () { |
| 80 | const request = yield* HttpServerRequest; |
| 81 | const body = yield* request.text; |
| 82 | yield* orders.send([{ content: body, headers: { source: "api" } }]); |
| 83 | return HttpServerResponse.empty({ status: 202 }); |
| 84 | }), |
| 85 | }; |
| 86 | }).pipe(Effect.provide(Cloudflare.K2.WriteStreamBinding)), |
| 87 | ) {} |
| 88 | ``` |
| 89 | |
| 90 | A record's `content` is a string (sent as UTF-8), a `Uint8Array`, or an |
| 91 | `ArrayBuffer`. `headers` is an optional map of up to 32 string pairs. A batch |
| 92 | can be up to 5 MB, and each record up to 1 MB. |
| 93 | |
| 94 | `WriteStreamBinding` sends through the Worker's native `k2` binding, so it |
| 95 | needs no API token. |
| 96 | |
| 97 | ## Type records with a schema |
| 98 | |
| 99 | Declare the stream with an Effect Schema, and every binding on it is typed by |
| 100 | that schema: |
| 101 | |
| 102 | ```diff lang="typescript" |
| 103 | // src/Orders.ts |
| 104 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 105 | +import * as Schema from "effect/Schema"; |
| 106 | + |
| 107 | +export const Order = Schema.Struct({ |
| 108 | + orderId: Schema.String, |
| 109 | + amountCents: Schema.Number, |
| 110 | +}); |
| 111 | |
| 112 | export const Orders = Cloudflare.K2.Stream("Orders", { |
| 113 | retention: "7 days", |
| 114 | + schema: Order, |
| 115 | }); |
| 116 | ``` |
| 117 | |
| 118 | Now `orders.send` takes `Order` values. The client encodes each one as JSON |
| 119 | and sets a `content-type: application/json` header: |
| 120 | |
| 121 | ```typescript |
| 122 | yield* orders.send([{ orderId: "o_1", amountCents: 4200 }]); |
| 123 | ``` |
| 124 | |
| 125 | K2 itself stores bytes, so the schema is a client-side codec. Changing it |
| 126 | never updates or replaces the stream. Producers and consumers read the schema |
| 127 | from the same stream, so they can't disagree about the record type. |
| 128 | |
| 129 | ## Handle produce errors |
| 130 | |
| 131 | Failures are typed. Two of them tell you whether the batch was stored: |
| 132 | |
| 133 | ```typescript |
| 134 | yield* orders.send(batch).pipe( |
| 135 | // The batch was not stored. Retrying it is safe. |
| 136 | Effect.catchTag("K2Unavailable", () => retryLater(batch)), |
| 137 | // The batch may or may not have been stored, and K2 does not deduplicate. |
| 138 | // Alchemy never retries this one for you. |
| 139 | Effect.catchTag("K2AppendOutcomeUnknown", () => reconcile(batch)), |
| 140 | ); |
| 141 | ``` |
| 142 | |
| 143 | The native binding and the HTTP path report the same error tags, so this code |
| 144 | works with either layer. |
| 145 | |
| 146 | ## Drain a Stream into K2 |
| 147 | |
| 148 | `Cloudflare.K2.StreamSink` exposes the stream as an Effect |
| 149 | [Sink](/infrastructure-as-effects/sinks). It packs records into requests of |
| 150 | up to 5 MB and retries only the errors that mean the batch was not stored. |
| 151 | |
| 152 | ```typescript |
| 153 | const sink = yield* Cloudflare.K2.StreamSink(Orders); |
| 154 | yield* Stream.fromIterable(pending).pipe(Stream.run(sink)); |
| 155 | ``` |
| 156 | |
| 157 | Provide `Cloudflare.K2.StreamSinkBinding` alongside `WriteStreamBinding`. |
| 158 | |
| 159 | ## Produce from outside a Worker |
| 160 | |
| 161 | The producer code stays the same on every host. Only the layer changes: |
| 162 | |
| 163 | - `WriteStreamBinding` and `StreamSinkBinding` run in Cloudflare Workers and |
| 164 | send through the native `k2` binding. |
| 165 | - `WriteStreamHttp` and `StreamSinkHttp` run on any host, such as Lambda or a |
| 166 | container. They send through the stream's HTTP input with a scoped "K2 |
| 167 | Produce" token that Alchemy mints at deploy. |
| 168 | - `WriteStreamLocal` and `StreamSinkLocal` run in scripts and Actions. They |
| 169 | send through the stream's HTTP input with your current Cloudflare |
| 170 | credentials. |
| 171 | |
| 172 | The HTTP and local layers need the stream's HTTP input turned on |
| 173 | (`http: true`). |
| 174 | |
| 175 | ## Consume from a long-running host |
| 176 | |
| 177 | `Cloudflare.K2.consumeStreamRecords(stream, handler)` subscribes a host to a |
| 178 | stream. The handler receives each batch as an Effect `Stream`: |
| 179 | |
| 180 | ```typescript |
| 181 | // src/Fraud.ts |
| 182 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 183 | import * as Fly from "@alchemy.run/fly"; |
| 184 | import * as Effect from "effect/Effect"; |
| 185 | import * as Layer from "effect/Layer"; |
| 186 | import * as Stream from "effect/Stream"; |
| 187 | import { Orders } from "./Orders.ts"; |
| 188 | |
| 189 | export default class Fraud extends Fly.Service<Fraud>()( |
| 190 | "Fraud", |
| 191 | { main: import.meta.url, region: "iad" }, |
| 192 | Effect.gen(function* () { |
| 193 | yield* Cloudflare.K2.consumeStreamRecords( |
| 194 | Orders, |
| 195 | { startAt: "earliest", concurrency: 4 }, |
| 196 | (records) => |
| 197 | records.pipe( |
| 198 | Stream.map((record) => record.value), |
| 199 | Stream.filter((order) => order.amountCents > 100_000), |
| 200 | Stream.runForEach((order) => Effect.log(`review ${order.orderId}`)), |
| 201 | ), |
| 202 | ); |
| 203 | |
| 204 | return {}; |
| 205 | }).pipe( |
| 206 | Effect.provide( |
| 207 | Layer.provideMerge( |
| 208 | Cloudflare.K2.StreamEventSourcePolling, |
| 209 | Cloudflare.K2.ReadSubscriptionHttp, |
| 210 | ), |
| 211 | ), |
| 212 | ), |
| 213 | ) {} |
| 214 | ``` |
| 215 | |
| 216 | At deploy time, `consumeStreamRecords` creates a subscription owned by the |
| 217 | host. At runtime, `StreamEventSourcePolling` runs `concurrency` pollers. Each |
| 218 | poller leases a batch of up to `maxRecords` (default 100) and runs the |
| 219 | handler: |
| 220 | |
| 221 | - When the handler succeeds, the batch is acknowledged. |
| 222 | - When it fails, the failure is logged and the batch is released for |
| 223 | redelivery. |
| 224 | - While the handler runs, the five-minute lease is extended every two |
| 225 | minutes. |
| 226 | - When a batch comes back empty, the poller waits `idleBackoff` (default one |
| 227 | second) before it polls again. |
| 228 | |
| 229 | `ReadSubscriptionHttp` reads with a scoped "K2 Consume" token that Alchemy |
| 230 | mints at deploy. `StreamEventSourcePolling` needs a host with a long-lived |
| 231 | process (Fly, ECS, EC2, Railway, Hetzner, Docker), so it doesn't work in a |
| 232 | Worker. |
| 233 | |
| 234 | For a stream with a schema, each record's JSON content is decoded into |
| 235 | `record.value`. For a stream without one, each record is |
| 236 | `{ content: Uint8Array, headers, timestamp }`. |
| 237 | |
| 238 | ## Share a subscription between hosts |
| 239 | |
| 240 | By default every consuming host gets its own subscription, so each host sees |
| 241 | every record. To split records between hosts instead, declare a |
| 242 | `Subscription` and pass it to each one: |
| 243 | |
| 244 | ```typescript |
| 245 | export const FraudWorkers = Cloudflare.K2.Subscription("FraudWorkers", { |
| 246 | stream: Orders, |
| 247 | startAt: "latest", |
| 248 | }); |
| 249 | ``` |
| 250 | |
| 251 | ```diff lang="typescript" |
| 252 | yield* Cloudflare.K2.consumeStreamRecords( |
| 253 | Orders, |
| 254 | - { startAt: "earliest", concurrency: 4 }, |
| 255 | + { subscription: FraudWorkers, concurrency: 4 }, |
| 256 | ``` |
| 257 | |
| 258 | Hosts sharing a subscription compete for batches. A subscription allows up |
| 259 | to 128 leases at a time. Subscriptions can't be changed after they are |
| 260 | created, so changing `startAt` replaces the subscription. |
| 261 | |
| 262 | ## Control leases yourself |
| 263 | |
| 264 | `Cloudflare.K2.ReadSubscription` is the client `consumeStreamRecords` uses |
| 265 | internally. Use it when you need your own loop: |
| 266 | |
| 267 | ```typescript |
| 268 | const inbox = yield* Cloudflare.K2.ReadSubscription(FraudWorkers); |
| 269 | |
| 270 | const batch = yield* inbox.consume({ workerId: "worker-1", maxRecords: 100 }); |
| 271 | if (Option.isSome(batch)) { |
| 272 | yield* handle(batch.value.records).pipe( |
| 273 | Effect.andThen(inbox.ack(batch.value)), |
| 274 | Effect.catchCause(() => inbox.nack(batch.value)), |
| 275 | ); |
| 276 | } |
| 277 | ``` |
| 278 | |
| 279 | `consume` returns `Option.none()` when there are no records. A worker that |
| 280 | already holds a lease gets the same batch back. `extend(batch)` renews a |
| 281 | lease for another five minutes, and fails with `K2LeaseLost` once the lease |
| 282 | is gone. |
| 283 | |
| 284 | ## Where next |
| 285 | |
| 286 | Related: |
| 287 | |
| 288 | - [Queues](/cloudflare/messaging/queues): a work queue with push delivery to |
| 289 | Workers, retries, and dead-letter queues. |
| 290 | - [Pipelines](/cloudflare/data/pipelines): land events as files or Iceberg |
| 291 | tables instead of consuming them yourself. |
| 292 | - [Workers](/cloudflare/compute/workers): the runtime that produces records. |
| 293 | - [Sinks](/infrastructure-as-effects/sinks): the Effect `Sink` model behind |
| 294 | `StreamSink`. |
| 295 | |
| 296 | Reference: |
| 297 | |
| 298 | - [K2 API reference](/providers/cloudflare/k2): `Stream`, `Subscription`, |
| 299 | `WriteStream`, `StreamSink`, `ReadSubscription`, and the polling layer. |