The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 540 lines 19.3 KB
Raw
1 ---
2 title: Queues
3 description: Cloudflare Queues give you reliable, at-least-once message delivery between Workers — a WriteQueue producer binding on one side and an Effect-style consumeQueueMessages handler with automatic ack/retry on the other.
4 ---
5
6 A Cloudflare Queue decouples work from the request that triggered
7 it: a Worker *produces* messages onto the queue, Cloudflare buffers
8 them, and a consumer Worker receives them in batches with
9 **at-least-once** delivery — failed batches are retried and
10 eventually dead-lettered.
11
12 Reach for a Queue whenever work shouldn't block a response: sending
13 emails, processing uploads, fanning out webhooks, or smoothing
14 bursty traffic. Both halves live in the same alchemy program: the
15 producer side is the `Cloudflare.Queues.WriteQueue(...)` binding
16 ([reference](/providers/cloudflare/queues/queue)), and the consumer
17 side is the Effect-style
18 `Cloudflare.Queues.consumeQueueMessages(queue, handler)` API this
19 page walks through.
20
21 `consumeQueueMessages(...)` does both halves in one call: it registers a
22 runtime queue listener on the Worker, and it auto-creates the
23 `Cloudflare.Queues.Consumer` resource that tells Cloudflare to
24 dispatch messages from the queue to this Worker. No separate
25 deploy-time wiring is needed.
26
27 By the end you'll have a Worker that:
28
29 - Sends a JSON message via `POST /queue/send`.
30 - Receives the message in a queue handler registered on the same
31 Worker, persists the body to R2, and acks the batch.
32 - Reads the persisted body via `GET /queue/result/:id`.
33
34 ## Create the Queue and Bucket
35
36 Both resources are plain `yield*` calls — no special config.
37
38 ```typescript
39 // alchemy.run.ts
40 import * as Alchemy from "alchemy";
41 import * as Cloudflare from "@alchemy.run/cloudflare";
42 import * as Effect from "effect/Effect";
43
44 export const Queue = Cloudflare.Queues.Queue("Queue");
45 export const Bucket = Cloudflare.R2.Bucket("Bucket");
46 ```
47
48 The Queue's name is generated from the stack/stage/id. The Bucket
49 will store each consumed message at `/queue/<id>` so the integ
50 test can read it back.
51
52 ## Bind the Queue producer
53
54 In the Worker's Construction phase, yield the queue resource and ask
55 `Cloudflare.Queues.WriteQueue` for a typed sender. The sender exposes `send` /
56 `sendBatch` that round-trip through Cloudflare's runtime.
57
58 ```typescript
59 // src/Api.ts
60 import * as Cloudflare from "@alchemy.run/cloudflare";
61 import * as Effect from "effect/Effect";
62 import { Bucket } from "./Bucket.ts";
63 import { Queue } from "./Queue.ts";
64
65 export default Cloudflare.Worker(
66 "Api",
67 { main: import.meta.url },
68 Effect.gen(function* () {
69 const bucket = yield* Cloudflare.R2.ReadWriteBucket(Bucket);
70 const queueResource = yield* Queue;
71 const queue = yield* Cloudflare.Queues.WriteQueue(queueResource);
72
73 return { fetch: Effect.gen(function* () { return new Response("ok"); }) };
74 }).pipe(
75 Effect.provide(Cloudflare.Queues.WriteQueueBinding),
76 Effect.provide(Cloudflare.R2.ReadWriteBucketBinding),
77 ),
78 );
79 ```
80
81 `queueResource` (the resolved `Queue` resource) is what you pass to
82 `consumeQueueMessages(...)` next — it's the same handle, not a new one.
83
84 ## Subscribe to incoming messages
85
86 `Cloudflare.Queues.consumeQueueMessages(queue, handler)` registers a queue
87 event listener on the Worker. The handler receives a
88 `Stream.Stream<Message<Body>>` — one stream per batch — and is
89 expected to return `Effect.Effect<void>`.
90
91 ```diff lang="typescript"
92 +import * as Stream from "effect/Stream";
93 +
94 +interface QueueMessageBody {
95 + id: string;
96 + text: string;
97 + sentAt: number;
98 +}
99
100 Effect.gen(function* () {
101 const bucket = yield* Cloudflare.R2.ReadWriteBucket(Bucket);
102 const queueResource = yield* Queue;
103 const queue = yield* Cloudflare.Queues.WriteQueue(queueResource);
104
105 + yield* Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(
106 + queueResource,
107 + (stream) =>
108 + Stream.runForEach(stream, (msg) =>
109 + bucket
110 + .put(`/queue/${msg.body.id}`, JSON.stringify(msg.body), {
111 + httpMetadata: { contentType: "application/json" },
112 + })
113 + .pipe(Effect.asVoid),
114 + ),
115 + );
116 ```
117
118 Acking is automatic: if the handler succeeds, every message in the
119 batch is `ack()`ed; if it fails, every message is `retry()`ed and
120 Cloudflare applies the consumer's `maxRetries` and `retryDelay`
121 before dead-lettering. For finer control, call `msg.ack()` /
122 `msg.retry()` per message inside the handler.
123
124 ## Tune batching and backoff
125
126 `consumeQueueMessages(queue, props)` accepts a settings object. Time fields are
127 `Duration.Input` — any of these are accepted:
128
129 ```typescript
130 maxWaitTime: Duration.seconds(5) // a Duration value
131 maxWaitTime: 5_000 // a number of milliseconds
132 maxWaitTime: "5 seconds" // a "<number> <unit>" string
133 retryDelay: "1 second" // singular and plural units both work
134 ```
135
136 ```diff lang="typescript"
137 +import * as Duration from "effect/Duration";
138
139 - Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(queueResource, (stream) =>
140 + Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(
141 + queueResource,
142 + {
143 + batchSize: 25,
144 + maxRetries: 3,
145 + maxWaitTime: "5 seconds",
146 + retryDelay: Duration.seconds(30),
147 + },
148 + (stream) =>
149 ```
150
151 `maxWaitTime` is rounded up to whole milliseconds and `retryDelay`
152 to whole seconds when forwarded to Cloudflare.
153
154 ## Consume the dead-letter queue
155
156 A message that fails `maxRetries` times moves to the `deadLetterQueue`.
157 Cloudflare does not drain or alert on that queue. Without a consumer, the
158 message waits there until the queue's retention period expires, and
159 nothing reports the failure.
160
161 Create the dead-letter queue as its own resource and pass it to the
162 source queue's consumer. Then consume it too:
163
164 ```typescript
165 export const DeadLetters = Cloudflare.Queues.Queue("DeadLetters");
166
167 Effect.gen(function* () {
168 const bucket = yield* Cloudflare.R2.ReadWriteBucket(Bucket);
169 const queueResource = yield* Queue;
170 const deadLetters = yield* DeadLetters;
171
172 yield* Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(
173 queueResource,
174 { maxRetries: 3, deadLetterQueue: deadLetters },
175 (stream) =>
176 Stream.runForEach(stream, (msg) =>
177 bucket.put(`/queue/${msg.body.id}`, JSON.stringify(msg.body)).pipe(Effect.asVoid),
178 ),
179 );
180
181 yield* Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(
182 deadLetters,
183 (stream) =>
184 Stream.runForEach(stream, (msg) =>
185 Effect.logError(`dead-lettered ${msg.body.id} after ${msg.attempts} attempts`),
186 ),
187 );
188 });
189 ```
190
191 One Worker can consume both queues. Replace the log with whatever should
192 happen to a failed message: persist it for replay, page someone, or send
193 it back to the source queue after a fix.
194
195 ### Dead-letter to a queue in another stack
196
197 If another stack owns the dead-letter queue, pass a reference to it:
198
199 ```typescript
200 yield* Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(
201 queueResource,
202 {
203 maxRetries: 3,
204 deadLetterQueue: yield* Cloudflare.Queues.Queue.ref("DeadLetters", {
205 stack: "shared",
206 stage: "production",
207 }),
208 },
209 (stream) => Stream.runForEach(stream, (msg) => Effect.log(msg.body.id)),
210 );
211 ```
212
213 Deploy the `shared` stack first, because a reference reads its persisted
214 state. That stack's Worker should consume the dead-letter queue.
215
216 ### Dead-letter to a queue by name
217
218 For a queue Alchemy doesn't manage, pass its name:
219
220 ```typescript
221 yield* Cloudflare.Queues.consumeQueueMessages<QueueMessageBody>(
222 queueResource,
223 { maxRetries: 3, deadLetterQueue: "orders-dead-letters" },
224 (stream) => Stream.runForEach(stream, (msg) => Effect.log(msg.body.id)),
225 );
226 ```
227
228 A name adds no deploy dependency on that queue, and nothing in this stack
229 consumes it. Make sure something else does.
230
231 ## Provide the runtime layer
232
233 `consumeQueueMessages(...)` is a `Context.Service` call —
234 `EventSourceLive` is the layer that registers the listener
235 with the Worker's runtime context. Add it to the layer stack
236 alongside the other binding lives.
237
238 ```diff lang="typescript"
239 }).pipe(
240 + Effect.provide(Cloudflare.Queues.EventSourceLive),
241 Effect.provide(Cloudflare.Queues.WriteQueueBinding),
242 Effect.provide(Cloudflare.R2.ReadWriteBucketBinding),
243 ),
244 ```
245
246 Without the live layer, the `consumeQueueMessages` call fails at deploy with
247 `Service not found: Cloudflare.Queues.EventSource`.
248
249 ## Add the producer route
250
251 `POST /queue/send` enqueues a message and returns the generated id.
252 The integ test uses the id to poll for the consumed result.
253
254 ```typescript
255 return {
256 fetch: Effect.gen(function* () {
257 const request = yield* HttpServerRequest;
258 if (request.url === "/queue/send" && request.method === "POST") {
259 const text = yield* request.text;
260 const msg: QueueMessageBody = {
261 id: crypto.randomUUID(),
262 text,
263 sentAt: Date.now(),
264 };
265 yield* queue.send(msg).pipe(Effect.orDie);
266 return yield* HttpServerResponse.json({ sent: msg }, { status: 202 });
267 }
268 return HttpServerResponse.text("Not Found", { status: 404 });
269 }),
270 };
271 ```
272
273 ## Add the result-read route
274
275 `GET /queue/result/:id` reads `/queue/<id>` from the bucket. The
276 consumer runs asynchronously, so the test polls this route with a
277 short backoff until the object appears.
278
279 ```typescript
280 if (request.url.startsWith("/queue/result/") && request.method === "GET") {
281 const id = request.url.split("/queue/result/")[1];
282 return yield* bucket.get(`/queue/${id}`).pipe(
283 Effect.flatMap((object) =>
284 object === null
285 ? Effect.succeed(HttpServerResponse.text("not yet", { status: 404 }))
286 : object.text().pipe(
287 Effect.map((body) =>
288 HttpServerResponse.text(body, {
289 headers: { "content-type": "application/json" },
290 }),
291 ),
292 ),
293 ),
294 Effect.catchTag("R2Error", (e) =>
295 Effect.succeed(HttpServerResponse.text(e.message, { status: 500 })),
296 ),
297 );
298 }
299 ```
300
301 ## Drain a Stream into the queue
302
303 `Cloudflare.Queues.QueueSink(queue)` exposes the queue as an Effect
304 [Sink](/infrastructure-as-effects/sinks). Run any `Stream` of message
305 bodies into it; each stream chunk becomes one `sendBatch` call, split
306 at Cloudflare's 100-message / 256 KB batch limits.
307
308 ```typescript
309 const sink = yield* Cloudflare.Queues.QueueSink(Clicks);
310 yield* Stream.fromIterable(events).pipe(Stream.run(sink));
311 ```
312
313 Combined with `consumeQueueMessages`, a Worker can consume one queue,
314 transform each batch, and drain it into another:
315
316 ```typescript
317 Effect.gen(function* () {
318 const enriched = yield* Cloudflare.Queues.QueueSink(EnrichedClicks);
319
320 yield* Cloudflare.Queues.consumeQueueMessages<Click>(Clicks, (messages) =>
321 messages.pipe(
322 Stream.map((message) => message.body),
323 Stream.map((click) => ({ ...click, receivedAt: Date.now() })),
324 Stream.run(enriched),
325 ),
326 );
327 }).pipe(
328 Effect.provide(Cloudflare.Queues.EventSourceLive),
329 Effect.provide(Cloudflare.Queues.QueueSinkBinding),
330 );
331 ```
332
333 `QueueSinkBinding` builds on the native `WriteQueueBinding`, so it is
334 the only layer the sink needs. It also works under `alchemy dev`.
335
336 ## What's the difference vs. a native `queue()` handler?
337
338 Cloudflare's runtime delivers queue events to a `queue(batch, env)`
339 export on the worker module. You can write that directly — see
340 [`examples/cloudflare-worker-async`](https://github.com/alchemy-run/alchemy/blob/main/examples/cloudflare-worker-async)
341 for the plain async-handler shape:
342
343 ```typescript
344 export default {
345 async queue(batch, env) {
346 for (const msg of batch.messages) {
347 await env.Bucket.put(`/queue/${msg.body.id}`, ...);
348 msg.ack();
349 }
350 },
351 };
352 ```
353
354 `Cloudflare.Queues.consumeQueueMessages(queue, handler)` is the same primitive
355 on the Effect side — the Worker bundle's runtime context routes
356 the dispatch to the registered listener — but you get
357 `Effect.gen` composition, typed errors, automatic batch
358 ack/retry, and the same surface as `AWS.SQS.consumeQueueMessages(queue, handler)`.
359
360 ## Receive Workflow lifecycle events
361
362 A subscription delivers platform events into a Queue. For a Workflow
363 bound to an async Worker, pass the binding directly as the source:
364
365 ```typescript
366 const worker = yield* Cloudflare.Worker("IngestionWorker", {
367 main: "./src/worker.ts",
368 env: {
369 INGESTION: Cloudflare.Workflow("Ingestion", {
370 className: "IngestionWorkflow",
371 }),
372 },
373 });
374 const queue = yield* Cloudflare.Queues.Queue("WorkflowEventsQueue");
375
376 yield* Cloudflare.Queues.Subscription("WorkflowEvents", {
377 source: worker.env.INGESTION,
378 events: ["instance.completed", "instance.errored"],
379 queueId: queue.queueId,
380 });
381 ```
382
383 The subscription uses the binding's deferred physical name, including on
384 the first deployment. Renaming the Workflow replaces the subscription.
385 The Queue still needs a consumer to handle these messages, as above.
386
387 A `WorkflowResource` or a reference to a deployed Workflow is also a source:
388
389 ```typescript
390 yield* Cloudflare.Queues.Subscription("WorkflowEvents", {
391 source: yield* Cloudflare.Workflow.ref("Ingestion", {
392 stack: "workflow-host",
393 stage: "production",
394 }),
395 events: ["instance.completed", "instance.errored"],
396 queueId: queue.queueId,
397 });
398 ```
399
400 Use the Workflow's logical ID, including any namespace. Omit the options for
401 the current stack and stage. References read persisted state, so deploy the
402 host first; deleting the consumer stack does not delete the referenced Workflow.
403
404 Explicit sources remain supported, including
405 `{ type: "workflows.workflow", workflowName: "existing-ingestion" }` for a
406 Workflow referenced by name and `{ type: "r2" }` for R2 events. Cloudflare
407 allows at most one subscription per source per account.
408
409 ## Use resources as event sources
410
411 A subscription also accepts these resource values, or their yielded `.ref(...)` references:
412
413 | Resource | Cloudflare source | Event scope |
414 | --- | --- | --- |
415 | `Cloudflare.Images.Variant` | `images` | All Images events in the account |
416 | `Cloudflare.KV.Namespace` | `kv` | All namespace events in the account |
417 | `Cloudflare.R2.Bucket` | `r2` | All bucket events in the account |
418 | `Cloudflare.R2.SuperSlurperJob` | `superSlurper` | All migration job events in the account |
419 | `Cloudflare.Vectorize.Index` | `vectorize` | All index events in the account |
420 | `Cloudflare.AI.Model` | `workersAi.model` | Batch events for the selected model |
421 | `Cloudflare.Worker` | `workersBuilds.worker` | Builds for the selected Worker |
422 | `Cloudflare.Workflows.WorkflowResource` | `workflows.workflow` | Instances of the selected Workflow |
423
424 The resource's account must match the subscription's Cloudflare account. Account metadata remains in Alchemy state and is not sent as an extra Cloudflare source field. A collision with an existing subscription fails rather than changing another stack's destination Queue.
425
426 Event delivery can lag subscription creation or replacement even after the destination Queue accepts messages. Deployment confirms configuration, not delivery readiness. Verify delivery before emitting events that must be observed.
427
428 During Vectorize subscription replacement or a destination Queue update, Cloudflare can still route new events to the previous Queue. Replacement events can carry the deleted subscription's ID; updates retain the same subscription ID. A single early event does not prove that routing has fully propagated. Keep the previous destination available during the transition and verify the receiving Queue, `metadata.eventSubscriptionId`, and the event's resource identity.
429
430 ### Account-wide resource events
431
432 ```typescript
433 const cache = yield* Cloudflare.KV.Namespace("Cache");
434
435 yield* Cloudflare.Queues.Subscription("NamespaceEvents", {
436 source: cache,
437 events: ["namespace.created", "namespace.deleted"],
438 queueId: queue.queueId,
439 });
440 ```
441
442 This receives namespace events for the **entire account**, not just `Cache`. R2, Vectorize, Images, and Super Slurper have the same account-wide behavior. Passing a resource creates a deployment dependency, so its initial creation can precede the subscription and its final deletion can follow subscription removal. Use the explicit `{ type: "kv" }` form if the subscription must be deployed before a namespace is created.
443
444 ### References from another stack
445
446 ```typescript
447 yield* Cloudflare.Queues.Subscription("BucketEvents", {
448 source: yield* Cloudflare.R2.Bucket.ref("Uploads", {
449 stack: "storage",
450 stage: "production",
451 }),
452 events: ["bucket.created", "bucket.deleted"],
453 queueId: queue.queueId,
454 });
455 ```
456
457 All the resource types above support `.ref`. The source must already exist in Alchemy state. Removing this subscription preserves the referenced resource and its source stack.
458
459 ### Images upload events
460
461 ```typescript
462 yield* Cloudflare.Queues.Subscription("ImageEvents", {
463 source: yield* Cloudflare.Images.Variant.ref("Thumbnail"),
464 events: ["image.uploaded"],
465 queueId: queue.queueId,
466 });
467 ```
468
469 The variant selects its Images account. The subscription receives uploads across that account, not only images served with `Thumbnail`.
470
471 ### Vectorize events
472
473 ```typescript
474 yield* Cloudflare.Queues.Subscription("IndexEvents", {
475 source: yield* Cloudflare.Vectorize.Index.ref("Search"),
476 events: ["index.created", "index.deleted"],
477 queueId: queue.queueId,
478 });
479 ```
480
481 These events describe index lifecycle changes, not individual vector writes.
482
483 ### Super Slurper events
484
485 ```typescript
486 yield* Cloudflare.Queues.Subscription("MigrationEvents", {
487 source: yield* Cloudflare.R2.SuperSlurperJob.ref("Migration"),
488 events: ["job.started", "job.completed", "job.aborted"],
489 queueId: queue.queueId,
490 });
491 ```
492
493 `R2.SuperSlurperJob` manages a one-off migration. Destroying an active job cancels it; it does not delete migrated objects or terminal job history. This subscription selects all jobs in the account, not object-level events from one job.
494
495 ### Workers AI batch events
496
497 ```typescript
498 const model = yield* Cloudflare.AI.Model("Embeddings", {
499 modelName: "@cf/baai/bge-m3",
500 });
501
502 yield* Cloudflare.Queues.Subscription("BatchEvents", {
503 source: model,
504 events: ["batch.queued", "batch.succeeded", "batch.failed"],
505 queueId: queue.queueId,
506 });
507 ```
508
509 `AI.Model` validates a Cloudflare-managed catalog model and persists a non-owning handle. It neither deploys nor invokes the model. `yield* Cloudflare.AI.Model.ref("Embeddings")` can be used in place of `model` after deployment. Batch events require asynchronous batch inference; ordinary synchronous inference does not emit them.
510
511 ### Workers Builds events
512
513 ```typescript
514 yield* Cloudflare.Queues.Subscription("BuildEvents", {
515 source: yield* Cloudflare.Worker.ref("Website"),
516 events: ["build.started", "build.succeeded", "build.failed"],
517 queueId: queue.queueId,
518 });
519 ```
520
521 A direct Worker resource is also accepted. The source uses the Worker's physical name and follows replacements. The Worker needs a Workers Builds integration to emit these events; an ordinary Alchemy upload is not a Workers Builds run.
522
523 ## Where next
524
525 Related:
526
527 - [Workers](/cloudflare/compute/workers) — the runtime that produces and
528 consumes queue messages.
529 - [Durable Objects](/cloudflare/compute/durable-objects) — for stateful
530 coordination instead of fire-and-forget work.
531 - [Workflows](/cloudflare/compute/workflows) — multi-step orchestration with
532 checkpointed steps when fire-and-forget isn't enough.
533
534 Reference:
535
536 - [Queue API reference](/providers/cloudflare/queues/queue)
537 - [Consumer API reference](/providers/cloudflare/queues/consumer)
538 - [Subscription API reference](/providers/cloudflare/queues/subscription)
539 - [Workers AI model handle API reference](/providers/cloudflare/ai/model)
540 - [Super Slurper migration API reference](/providers/cloudflare/r2/superslurperjob)