The best way to control Claude Code, Codex and more from any computer or phone

main
md 87 lines 4.12 KB
Rendered Raw
1 # gcp-storage-uploads
2
3 An upload pipeline on Cloud Run. A public API stores files in Cloud
4 Storage; a private indexer picks each upload up from the bucket's
5 notification, hashes it, and records its metadata in Firestore.
6
7 ```
8 PUT /files/:name ──> Api ──> gs://…/uploads/:name
9 │ OBJECT_FINALIZE
10 v
11 Pub/Sub push ──> Indexer ──> Firestore files/:name
12 GET /files ─────────> Api <────────────────────────── (size, type, sha256)
13 ```
14
15 - `src/resources.ts` — the `Uploads` bucket and the `Files` Firestore
16 database (named, `FIRESTORE_NATIVE`).
17 - `src/Api.ts` — public `GCP.Function`:
18 - `PUT /files/:name` stores the body at `uploads/:name` and returns `202`.
19 - `GET /files/:name` returns the stored bytes.
20 - `GET /files` lists the indexed metadata.
21 - `DELETE /files/:name` deletes the object and its metadata.
22 - `src/Indexer.ts` — private `GCP.Function` consuming `OBJECT_FINALIZE`
23 events for `uploads/`. It reads the exact object generation the event
24 names, computes its sha256, and writes `files/:name`. Uploads never wait
25 on the hash.
26
27 ## Bindings and event sources
28
29 | Host | Binding / event source | IAM granted to the host's service account |
30 | ------- | ----------------------------------------- | -------------------------------------------------------------------- |
31 | Api | `GCP.Storage.ReadBucket(bucket)` | `roles/storage.objectViewer` on the bucket |
32 | Api | `GCP.Storage.WriteBucket(bucket)` | `roles/storage.objectUser` on the bucket |
33 | Api | `GCP.Firestore.ReadWriteDatabase(files)` | `roles/datastore.user` on the project, conditioned on this database |
34 | Indexer | `GCP.Storage.ReadBucket(bucket)` | `roles/storage.objectViewer` on the bucket |
35 | Indexer | `GCP.Firestore.WriteDatabase(files)` | `roles/datastore.user` on the project, conditioned on this database |
36 | Indexer | `GCP.Storage.consumeBucketEvents(bucket)` | `roles/run.invoker` on the Indexer itself (for the push OIDC token) |
37
38 `consumeBucketEvents` is provided by `GCP.Storage.BucketEventSourceLive`
39 with `GCP.Run.TopicEventSource` for delivery. Together they create a
40 Pub/Sub topic, a `JSON_API_V1` bucket notification filtered to
41 `OBJECT_FINALIZE` under `uploads/` (granting the Cloud Storage service
42 agent `roles/pubsub.publisher` on the topic), and a push subscription that
43 calls the Indexer with an OIDC token for its own service account.
44
45 Firestore databases have no resource-level IAM policy, so the Firestore
46 roles are granted on the project under an IAM Condition that names the
47 `Files` database.
48
49 ## Deploy
50
51 Requires Docker (the services are built from `main`). Credentials come from your alchemy profile: run `alchemy profile` once and pick GCP (*Service account JSON* for a key file, or *Stored* for an access token or key kept in `~/.alchemy/credentials`, plus a default region), then deploy with `--profile <name>`.
52
53 ```sh
54 pnpm deploy --profile <name>
55 ```
56
57 ```sh
58 curl -X PUT --data-binary @photo.jpg -H 'content-type: image/jpeg' "$url/files/photo.jpg"
59 curl "$url/files" # metadata appears once the indexer has run
60 curl -o out.jpg "$url/files/photo.jpg"
61 curl -X DELETE "$url/files/photo.jpg"
62 ```
63
64 On a fresh deploy the project-level Firestore grants can take a few
65 minutes to propagate. Until then the indexer's writes fail and Pub/Sub
66 redelivers them, so metadata shows up late rather than never.
67
68 ## Test
69
70 ```sh
71 ALCHEMY_PROFILE=<name> bun test
72 ```
73
74 Deploys the stack, uploads a text file and a random binary, reads both
75 back, waits for the indexer's records, checks each sha256 against the
76 uploaded bytes and each record against the live Firestore document and
77 object generation, deletes a file, then destroys the stack and checks the
78 bucket, database, and services are gone.
79
80 ## Destroy
81
82 ```sh
83 pnpm destroy --profile <name>
84 ```
85
86 The bucket uses `forceDestroy: true`, so destroy removes its objects too.
87 Drop it for real user data.