The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 78 lines 2.26 KB
Raw
1 ---
2 title: Worker pools
3 description: Run always-on queue consumers with GCP.Run.WorkerPool — no URL, a fixed instance count, and a Pub/Sub pull loop.
4 ---
5
6 `GCP.Run.WorkerPool` runs containers that have no URL and keep
7 running. Use it for a consumer that pulls from Pub/Sub all day,
8 whether or not anyone sends HTTP traffic.
9
10 ## Declare a pool
11
12 Set how many instances run with `scaling.manualInstanceCount`:
13
14 ```typescript
15 // src/Worker.ts
16 import * as GCP from "@alchemy.run/gcp";
17 import * as Effect from "effect/Effect";
18 import * as Stream from "effect/Stream";
19 import { Jobs, Results } from "./resources.ts";
20
21 export default class Worker extends GCP.Run.WorkerPool<Worker>()(
22 "Worker",
23 { main: import.meta.url, scaling: { manualInstanceCount: 1 } },
24 Effect.gen(function* () {
25 const results = yield* GCP.Firestore.WriteDatabase(Results);
26
27 yield* GCP.PubSub.consumeTopicMessages(
28 Jobs,
29 { maxMessages: 10, ackDeadlineSeconds: 60 },
30 (messages) =>
31 messages.pipe(
32 Stream.runForEach(({ message }) =>
33 results.set(`jobs/${message.messageId}`, { done: true }),
34 ),
35 Effect.orDie,
36 ),
37 );
38 }).pipe(
39 Effect.provide([GCP.Run.TopicPullEventSource, GCP.Firestore.WriteDatabaseHttp]),
40 ),
41 ) {}
42 ```
43
44 `TopicPullEventSource` creates a pull subscription for the pool and
45 runs the pull loop. A batch is acked only after your handler succeeds.
46 If it fails, Pub/Sub delivers the batch again.
47
48 ## Make processing idempotent
49
50 Pub/Sub delivers at least once. Write results with `set` keyed by the
51 job id, so a second delivery overwrites the same document instead of
52 adding another.
53
54 ## Run a loop of your own
55
56 Return `run` instead to own the loop:
57
58 ```typescript
59 return {
60 run: Effect.gen(function* () {
61 yield* Effect.log("tick");
62 yield* Effect.sleep("30 seconds");
63 }).pipe(Effect.forever),
64 };
65 ```
66
67 ## Example
68
69 [`examples/gcp-pubsub-worker`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-pubsub-worker)
70 is a job queue: a public service publishes jobs, a worker pool
71 processes them, and the service reads results from Firestore.
72
73 ## Where next
74
75 - [Pub/Sub](/gcp/messaging/pubsub) — topics, push vs pull, filters, and
76 dead letters.
77 - [Run reference](/providers/gcp/run#workerpool) — every prop and
78 attribute.