The best way to control Claude Code, Codex and more from any computer or phone
| 1 | --- |
| 2 | title: Pub/Sub |
| 3 | description: Publish to Pub/Sub topics from Cloud Run, consume them with push delivery or a pull loop, filter by attribute, and dead-letter failures. |
| 4 | --- |
| 5 | |
| 6 | Pub/Sub moves messages between services. A producer publishes to a |
| 7 | topic. Each consumer gets its own subscription and sees every message |
| 8 | it subscribes to, independent of the others. |
| 9 | |
| 10 | ## Create a topic |
| 11 | |
| 12 | ```typescript |
| 13 | // src/resources.ts |
| 14 | import * as GCP from "@alchemy.run/gcp"; |
| 15 | |
| 16 | export const Orders = GCP.PubSub.Topic("Orders", {}); |
| 17 | ``` |
| 18 | |
| 19 | ## Publish |
| 20 | |
| 21 | `WriteTopic` returns a client with `publish`: |
| 22 | |
| 23 | ```typescript |
| 24 | // src/Api.ts |
| 25 | import * as GCP from "@alchemy.run/gcp"; |
| 26 | import * as Effect from "effect/Effect"; |
| 27 | import { HttpServerRequest } from "effect/http/HttpServerRequest"; |
| 28 | import * as HttpServerResponse from "effect/http/HttpServerResponse"; |
| 29 | import { Orders } from "./resources.ts"; |
| 30 | |
| 31 | export default class Api extends GCP.Function<Api>()( |
| 32 | "Api", |
| 33 | { main: import.meta.url, invokerIamDisabled: true }, |
| 34 | Effect.gen(function* () { |
| 35 | const orders = yield* GCP.PubSub.WriteTopic(Orders); |
| 36 | |
| 37 | return { |
| 38 | fetch: Effect.gen(function* () { |
| 39 | const request = yield* HttpServerRequest; |
| 40 | const order = yield* request.json; |
| 41 | const messageId = yield* orders.publish({ |
| 42 | data: JSON.stringify(order), |
| 43 | attributes: { type: "order.created" }, |
| 44 | }); |
| 45 | return yield* HttpServerResponse.json({ messageId }, { status: 202 }); |
| 46 | }).pipe(Effect.orDie), |
| 47 | }; |
| 48 | }).pipe(Effect.provide(GCP.PubSub.WriteTopicHttp)), |
| 49 | ) {} |
| 50 | ``` |
| 51 | |
| 52 | `publish` takes a string or bytes and returns the message id. |
| 53 | Attributes can be filtered on without reading the body. The binding |
| 54 | grants `roles/pubsub.publisher` on this topic only. |
| 55 | |
| 56 | ## Consume with push |
| 57 | |
| 58 | A Cloud Run service receives messages over HTTP: |
| 59 | |
| 60 | ```typescript |
| 61 | // src/Email.ts |
| 62 | import * as GCP from "@alchemy.run/gcp"; |
| 63 | import * as Effect from "effect/Effect"; |
| 64 | import * as HttpServerResponse from "effect/http/HttpServerResponse"; |
| 65 | import * as Stream from "effect/Stream"; |
| 66 | import { Orders } from "./resources.ts"; |
| 67 | |
| 68 | export default class Email extends GCP.Function<Email>()( |
| 69 | "Email", |
| 70 | { main: import.meta.url }, |
| 71 | Effect.gen(function* () { |
| 72 | yield* GCP.PubSub.consumeTopicMessages(Orders, (messages) => |
| 73 | messages.pipe( |
| 74 | Stream.runForEach(({ message }) => |
| 75 | Effect.gen(function* () { |
| 76 | const order = JSON.parse(atob(message.data ?? "")); |
| 77 | yield* Effect.log(`confirm order ${order.id}`); |
| 78 | }), |
| 79 | ), |
| 80 | Effect.orDie, |
| 81 | ), |
| 82 | ); |
| 83 | |
| 84 | return { fetch: Effect.succeed(HttpServerResponse.text("ok")) }; |
| 85 | }).pipe(Effect.provide(GCP.Run.TopicEventSource)), |
| 86 | ) {} |
| 87 | ``` |
| 88 | |
| 89 | Alchemy creates a push subscription to a private path on the service, |
| 90 | signed with the service's own identity. The service stays private: |
| 91 | only Pub/Sub's signed requests get through. A success acks the |
| 92 | message. A failure answers 500 and Pub/Sub delivers it again. |
| 93 | |
| 94 | ## Filter by attribute |
| 95 | |
| 96 | Subscribe to a subset of messages. Pub/Sub applies the filter, so |
| 97 | other messages never reach the service: |
| 98 | |
| 99 | ```typescript |
| 100 | yield* GCP.PubSub.consumeTopicMessages( |
| 101 | Orders, |
| 102 | { filter: 'attributes.type = "order.created"' }, |
| 103 | (messages) => messages.pipe(Stream.runForEach(handle), Effect.orDie), |
| 104 | ); |
| 105 | ``` |
| 106 | |
| 107 | ## Dead-letter failures |
| 108 | |
| 109 | Stop retrying a message after a few tries and park it on another |
| 110 | topic: |
| 111 | |
| 112 | ```typescript |
| 113 | yield* GCP.PubSub.consumeTopicMessages( |
| 114 | Orders, |
| 115 | { |
| 116 | deadLetter: { topic: yield* FailedOrders, maxDeliveryAttempts: 5 }, |
| 117 | }, |
| 118 | (messages) => messages.pipe(Stream.runForEach(handle), Effect.orDie), |
| 119 | ); |
| 120 | ``` |
| 121 | |
| 122 | A topic with no subscription drops what is published to it. Give the |
| 123 | dead-letter topic a subscription so failed messages are kept: |
| 124 | |
| 125 | ```typescript |
| 126 | export const FailedOrdersInbox = Effect.gen(function* () { |
| 127 | const topic = yield* FailedOrders; |
| 128 | return yield* GCP.PubSub.Subscription("FailedOrdersInbox", { |
| 129 | topic: topic.name, |
| 130 | messageRetentionDuration: "604800s", |
| 131 | }); |
| 132 | }); |
| 133 | ``` |
| 134 | |
| 135 | ## Consume with pull |
| 136 | |
| 137 | A [worker pool](/gcp/compute/worker-pools) or a |
| 138 | [job](/gcp/compute/jobs) has no URL. Pull instead: |
| 139 | |
| 140 | ```typescript |
| 141 | yield* GCP.PubSub.consumeTopicMessages( |
| 142 | Orders, |
| 143 | { maxMessages: 50 }, |
| 144 | (messages) => messages.pipe(Stream.runForEach(handle), Effect.orDie), |
| 145 | ); |
| 146 | // …provided with Effect.provide(GCP.Run.TopicPullEventSource) |
| 147 | ``` |
| 148 | |
| 149 | A batch is acked after your handler succeeds. |
| 150 | |
| 151 | ## Drain a subscription in batches |
| 152 | |
| 153 | To control the batch yourself, bind a pull subscription with |
| 154 | `ReadSubscription`. A [job](/gcp/compute/jobs) can drain it into |
| 155 | BigQuery and exit: |
| 156 | |
| 157 | ```typescript |
| 158 | const subscription = yield* GCP.PubSub.ReadSubscription(Inbox); |
| 159 | const warehouse = yield* GCP.BigQuery.WriteTable(table); |
| 160 | |
| 161 | return { |
| 162 | run: Effect.gen(function* () { |
| 163 | const received = yield* subscription |
| 164 | .pull({ maxMessages: 500 }) |
| 165 | .pipe(Effect.timeoutOption("10 seconds")); |
| 166 | const messages = Option.getOrElse(received, () => []); |
| 167 | if (messages.length === 0) return 0; |
| 168 | |
| 169 | const rows = messages.map((message) => JSON.parse(message.text)); |
| 170 | yield* warehouse.insert(rows, { insertIds: rows.map((row) => row.id) }); |
| 171 | yield* subscription.acknowledge(messages.map((message) => message.ackId)); |
| 172 | return messages.length; |
| 173 | }).pipe( |
| 174 | Effect.repeat({ until: (count) => count === 0 }), |
| 175 | Effect.asVoid, |
| 176 | Effect.orDie, |
| 177 | ), |
| 178 | }; |
| 179 | ``` |
| 180 | |
| 181 | Insert before you acknowledge. A crash in between delivers the batch |
| 182 | again, and `insertIds` drops the duplicates. Acknowledging first would |
| 183 | lose the events. |
| 184 | |
| 185 | ## Handle repeats |
| 186 | |
| 187 | Pub/Sub delivers at least once. Make handlers safe to run twice: write |
| 188 | with `set` keyed by an id, or use `insertIds` in BigQuery. |
| 189 | |
| 190 | ## Examples |
| 191 | |
| 192 | - [`gcp-pubsub-fanout`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-pubsub-fanout) |
| 193 | — one topic, two push consumers, an attribute filter, and a |
| 194 | dead-letter topic. |
| 195 | - [`gcp-pubsub-worker`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-pubsub-worker) |
| 196 | — a job queue on a worker pool. |
| 197 | - [`gcp-event-pipeline`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-event-pipeline) |
| 198 | — a service publishes, a job drains into BigQuery. |
| 199 | |
| 200 | ## Where next |
| 201 | |
| 202 | - [Cloud Scheduler](/gcp/messaging/scheduler) — run the drain on a |
| 203 | timer. |
| 204 | - [PubSub reference](/providers/gcp/pubsub#topic) — every prop and |
| 205 | attribute. |