@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
mdx 205 lines 6.15 KB
Raw
1 ---
2 title: Pub/Sub
3 description: Publish to Pub/Sub topics from Cloud Run, consume them with push delivery or a pull loop, filter by attribute, and dead-letter failures.
4 ---
5
6 Pub/Sub moves messages between services. A producer publishes to a
7 topic. Each consumer gets its own subscription and sees every message
8 it subscribes to, independent of the others.
9
10 ## Create a topic
11
12 ```typescript
13 // src/resources.ts
14 import * as GCP from "@alchemy.run/gcp";
15
16 export const Orders = GCP.PubSub.Topic("Orders", {});
17 ```
18
19 ## Publish
20
21 `WriteTopic` returns a client with `publish`:
22
23 ```typescript
24 // src/Api.ts
25 import * as GCP from "@alchemy.run/gcp";
26 import * as Effect from "effect/Effect";
27 import { HttpServerRequest } from "effect/http/HttpServerRequest";
28 import * as HttpServerResponse from "effect/http/HttpServerResponse";
29 import { Orders } from "./resources.ts";
30
31 export default class Api extends GCP.Function<Api>()(
32 "Api",
33 { main: import.meta.url, invokerIamDisabled: true },
34 Effect.gen(function* () {
35 const orders = yield* GCP.PubSub.WriteTopic(Orders);
36
37 return {
38 fetch: Effect.gen(function* () {
39 const request = yield* HttpServerRequest;
40 const order = yield* request.json;
41 const messageId = yield* orders.publish({
42 data: JSON.stringify(order),
43 attributes: { type: "order.created" },
44 });
45 return yield* HttpServerResponse.json({ messageId }, { status: 202 });
46 }).pipe(Effect.orDie),
47 };
48 }).pipe(Effect.provide(GCP.PubSub.WriteTopicHttp)),
49 ) {}
50 ```
51
52 `publish` takes a string or bytes and returns the message id.
53 Attributes can be filtered on without reading the body. The binding
54 grants `roles/pubsub.publisher` on this topic only.
55
56 ## Consume with push
57
58 A Cloud Run service receives messages over HTTP:
59
60 ```typescript
61 // src/Email.ts
62 import * as GCP from "@alchemy.run/gcp";
63 import * as Effect from "effect/Effect";
64 import * as HttpServerResponse from "effect/http/HttpServerResponse";
65 import * as Stream from "effect/Stream";
66 import { Orders } from "./resources.ts";
67
68 export default class Email extends GCP.Function<Email>()(
69 "Email",
70 { main: import.meta.url },
71 Effect.gen(function* () {
72 yield* GCP.PubSub.consumeTopicMessages(Orders, (messages) =>
73 messages.pipe(
74 Stream.runForEach(({ message }) =>
75 Effect.gen(function* () {
76 const order = JSON.parse(atob(message.data ?? ""));
77 yield* Effect.log(`confirm order ${order.id}`);
78 }),
79 ),
80 Effect.orDie,
81 ),
82 );
83
84 return { fetch: Effect.succeed(HttpServerResponse.text("ok")) };
85 }).pipe(Effect.provide(GCP.Run.TopicEventSource)),
86 ) {}
87 ```
88
89 Alchemy creates a push subscription to a private path on the service,
90 signed with the service's own identity. The service stays private:
91 only Pub/Sub's signed requests get through. A success acks the
92 message. A failure answers 500 and Pub/Sub delivers it again.
93
94 ## Filter by attribute
95
96 Subscribe to a subset of messages. Pub/Sub applies the filter, so
97 other messages never reach the service:
98
99 ```typescript
100 yield* GCP.PubSub.consumeTopicMessages(
101 Orders,
102 { filter: 'attributes.type = "order.created"' },
103 (messages) => messages.pipe(Stream.runForEach(handle), Effect.orDie),
104 );
105 ```
106
107 ## Dead-letter failures
108
109 Stop retrying a message after a few tries and park it on another
110 topic:
111
112 ```typescript
113 yield* GCP.PubSub.consumeTopicMessages(
114 Orders,
115 {
116 deadLetter: { topic: yield* FailedOrders, maxDeliveryAttempts: 5 },
117 },
118 (messages) => messages.pipe(Stream.runForEach(handle), Effect.orDie),
119 );
120 ```
121
122 A topic with no subscription drops what is published to it. Give the
123 dead-letter topic a subscription so failed messages are kept:
124
125 ```typescript
126 export const FailedOrdersInbox = Effect.gen(function* () {
127 const topic = yield* FailedOrders;
128 return yield* GCP.PubSub.Subscription("FailedOrdersInbox", {
129 topic: topic.name,
130 messageRetentionDuration: "604800s",
131 });
132 });
133 ```
134
135 ## Consume with pull
136
137 A [worker pool](/gcp/compute/worker-pools) or a
138 [job](/gcp/compute/jobs) has no URL. Pull instead:
139
140 ```typescript
141 yield* GCP.PubSub.consumeTopicMessages(
142 Orders,
143 { maxMessages: 50 },
144 (messages) => messages.pipe(Stream.runForEach(handle), Effect.orDie),
145 );
146 // …provided with Effect.provide(GCP.Run.TopicPullEventSource)
147 ```
148
149 A batch is acked after your handler succeeds.
150
151 ## Drain a subscription in batches
152
153 To control the batch yourself, bind a pull subscription with
154 `ReadSubscription`. A [job](/gcp/compute/jobs) can drain it into
155 BigQuery and exit:
156
157 ```typescript
158 const subscription = yield* GCP.PubSub.ReadSubscription(Inbox);
159 const warehouse = yield* GCP.BigQuery.WriteTable(table);
160
161 return {
162 run: Effect.gen(function* () {
163 const received = yield* subscription
164 .pull({ maxMessages: 500 })
165 .pipe(Effect.timeoutOption("10 seconds"));
166 const messages = Option.getOrElse(received, () => []);
167 if (messages.length === 0) return 0;
168
169 const rows = messages.map((message) => JSON.parse(message.text));
170 yield* warehouse.insert(rows, { insertIds: rows.map((row) => row.id) });
171 yield* subscription.acknowledge(messages.map((message) => message.ackId));
172 return messages.length;
173 }).pipe(
174 Effect.repeat({ until: (count) => count === 0 }),
175 Effect.asVoid,
176 Effect.orDie,
177 ),
178 };
179 ```
180
181 Insert before you acknowledge. A crash in between delivers the batch
182 again, and `insertIds` drops the duplicates. Acknowledging first would
183 lose the events.
184
185 ## Handle repeats
186
187 Pub/Sub delivers at least once. Make handlers safe to run twice: write
188 with `set` keyed by an id, or use `insertIds` in BigQuery.
189
190 ## Examples
191
192 - [`gcp-pubsub-fanout`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-pubsub-fanout)
193 — one topic, two push consumers, an attribute filter, and a
194 dead-letter topic.
195 - [`gcp-pubsub-worker`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-pubsub-worker)
196 — a job queue on a worker pool.
197 - [`gcp-event-pipeline`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-event-pipeline)
198 — a service publishes, a job drains into BigQuery.
199
200 ## Where next
201
202 - [Cloud Scheduler](/gcp/messaging/scheduler) — run the drain on a
203 timer.
204 - [PubSub reference](/providers/gcp/pubsub#topic) — every prop and
205 attribute.