The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 06:00 UTC
| 1 | --- |
| 2 | title: Worker pools |
| 3 | description: Run always-on queue consumers with GCP.Run.WorkerPool — no URL, a fixed instance count, and a Pub/Sub pull loop. |
| 4 | --- |
| 5 | |
| 6 | `GCP.Run.WorkerPool` runs containers that have no URL and keep |
| 7 | running. Use it for a consumer that pulls from Pub/Sub all day, |
| 8 | whether or not anyone sends HTTP traffic. |
| 9 | |
| 10 | ## Declare a pool |
| 11 | |
| 12 | Set how many instances run with `scaling.manualInstanceCount`: |
| 13 | |
| 14 | ```typescript |
| 15 | // src/Worker.ts |
| 16 | import * as GCP from "@alchemy.run/gcp"; |
| 17 | import * as Effect from "effect/Effect"; |
| 18 | import * as Stream from "effect/Stream"; |
| 19 | import { Jobs, Results } from "./resources.ts"; |
| 20 | |
| 21 | export default class Worker extends GCP.Run.WorkerPool<Worker>()( |
| 22 | "Worker", |
| 23 | { main: import.meta.url, scaling: { manualInstanceCount: 1 } }, |
| 24 | Effect.gen(function* () { |
| 25 | const results = yield* GCP.Firestore.WriteDatabase(Results); |
| 26 | |
| 27 | yield* GCP.PubSub.consumeTopicMessages( |
| 28 | Jobs, |
| 29 | { maxMessages: 10, ackDeadlineSeconds: 60 }, |
| 30 | (messages) => |
| 31 | messages.pipe( |
| 32 | Stream.runForEach(({ message }) => |
| 33 | results.set(`jobs/${message.messageId}`, { done: true }), |
| 34 | ), |
| 35 | Effect.orDie, |
| 36 | ), |
| 37 | ); |
| 38 | }).pipe( |
| 39 | Effect.provide([GCP.Run.TopicPullEventSource, GCP.Firestore.WriteDatabaseHttp]), |
| 40 | ), |
| 41 | ) {} |
| 42 | ``` |
| 43 | |
| 44 | `TopicPullEventSource` creates a pull subscription for the pool and |
| 45 | runs the pull loop. A batch is acked only after your handler succeeds. |
| 46 | If it fails, Pub/Sub delivers the batch again. |
| 47 | |
| 48 | ## Make processing idempotent |
| 49 | |
| 50 | Pub/Sub delivers at least once. Write results with `set` keyed by the |
| 51 | job id, so a second delivery overwrites the same document instead of |
| 52 | adding another. |
| 53 | |
| 54 | ## Run a loop of your own |
| 55 | |
| 56 | Return `run` instead to own the loop: |
| 57 | |
| 58 | ```typescript |
| 59 | return { |
| 60 | run: Effect.gen(function* () { |
| 61 | yield* Effect.log("tick"); |
| 62 | yield* Effect.sleep("30 seconds"); |
| 63 | }).pipe(Effect.forever), |
| 64 | }; |
| 65 | ``` |
| 66 | |
| 67 | ## Example |
| 68 | |
| 69 | [`examples/gcp-pubsub-worker`](https://github.com/alchemy-run/alchemy/tree/main/examples/gcp-pubsub-worker) |
| 70 | is a job queue: a public service publishes jobs, a worker pool |
| 71 | processes them, and the service reads results from Firestore. |
| 72 | |
| 73 | ## Where next |
| 74 | |
| 75 | - [Pub/Sub](/gcp/messaging/pubsub) — topics, push vs pull, filters, and |
| 76 | dead letters. |
| 77 | - [Run reference](/providers/gcp/run#workerpool) — every prop and |
| 78 | attribute. |