The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 04:45 UTC
| 1 | --- |
| 2 | title: Bytes in S3, hashing on Lambda |
| 3 | description: Compute on Cloudflare, bytes in S3, and large pushes hashed one Lambda per chunk. One stack carries both provider sets, and Alchemy mints the cross-cloud identity. |
| 4 | --- |
| 5 | |
| 6 | The examples use `Authentication` from |
| 7 | [Getting Started](/git/getting-started): the application's request middleware. |
| 8 | |
| 9 | import PackageManagers from "../../../../components/PackageManagers.astro"; |
| 10 | |
| 11 | The Worker and the Durable Objects stay on Cloudflare. Packs, bundles, |
| 12 | and spilled pushes live in S3, and every 4 MiB chunk of a push is |
| 13 | hashed on its own Lambda: |
| 14 | |
| 15 | ```typescript |
| 16 | // src/git.ts |
| 17 | import * as AWS from "@alchemy.run/aws"; |
| 18 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 19 | import * as Git from "@alchemy.run/git"; |
| 20 | import * as GitHasher from "@alchemy.run/git/Hasher"; |
| 21 | import * as Effect from "effect/Effect"; |
| 22 | import * as Layer from "effect/Layer"; |
| 23 | import * as HttpRouter from "effect/http/HttpRouter"; |
| 24 | import * as Http from "alchemy/Http"; |
| 25 | |
| 26 | const GitObjects = AWS.S3.Bucket("GitObjects"); |
| 27 | |
| 28 | const GitLive = Git.ApiLive.pipe( |
| 29 | Layer.provide(Git.ApiHandlersLive), |
| 30 | Layer.provide(Authentication.layer), |
| 31 | Layer.provide(Git.ReposDurableObject), |
| 32 | Layer.provide(Git.RegistryDurableObject), |
| 33 | Layer.provide(Git.BlobStoreS3(GitObjects)), |
| 34 | Layer.provide(GitHasher.HasherLambda(GitHasher.HasherFunction)), |
| 35 | Layer.provide(AWS.Lambda.InvokeFunctionHttp), |
| 36 | ); |
| 37 | |
| 38 | export default class GitHost extends Cloudflare.Worker<GitHost>()( |
| 39 | "Git", |
| 40 | { main: import.meta.url, ...Git.GIT_WORKER_OPTIONS }, |
| 41 | Effect.gen(function* () { |
| 42 | const fetch = yield* HttpRouter.toHttpEffect(GitLive.pipe(Layer.provide(Http.Platform))); |
| 43 | return { fetch }; |
| 44 | }), |
| 45 | ) {} |
| 46 | ``` |
| 47 | |
| 48 | ```typescript |
| 49 | // alchemy.run.ts |
| 50 | import * as Alchemy from "alchemy"; |
| 51 | import * as AWS from "@alchemy.run/aws"; |
| 52 | import * as Cloudflare from "@alchemy.run/cloudflare"; |
| 53 | import * as Effect from "effect/Effect"; |
| 54 | import * as Layer from "effect/Layer"; |
| 55 | import GitHost from "./src/git.ts"; |
| 56 | |
| 57 | export default Alchemy.Stack( |
| 58 | "GitService", |
| 59 | { |
| 60 | providers: Layer.mergeAll(Cloudflare.providers(), AWS.providers()), |
| 61 | state: Cloudflare.state(), |
| 62 | }, |
| 63 | Effect.gen(function* () { |
| 64 | const git = yield* GitHost; |
| 65 | return { url: git.url.as<string>() }; |
| 66 | }), |
| 67 | ); |
| 68 | ``` |
| 69 | |
| 70 | <PackageManagers run="alchemy deploy" /> |
| 71 | |
| 72 | ## What gets created |
| 73 | |
| 74 | On Cloudflare, the same Worker and Durable Object namespaces as |
| 75 | [All on Cloudflare](/git/recipes/cloudflare), minus the bucket. |
| 76 | |
| 77 | On AWS: |
| 78 | |
| 79 | - `GitObjects`, an S3 bucket with an engine-generated name in the |
| 80 | deploying profile's region. |
| 81 | - `HasherFunction`, a 3 GB Lambda whose only handler answers hash |
| 82 | events, with its execution role. |
| 83 | - One IAM user, one access key, and one least-privilege role the Worker |
| 84 | assumes. `BlobStoreS3` and `InvokeFunctionHttp` share that identity: |
| 85 | the key and the role's ARN are bound onto the Worker, each request is |
| 86 | signed with credentials assumed at runtime, and the role is granted |
| 87 | the S3 actions on that one bucket and `lambda:InvokeFunction` on that |
| 88 | one function. |
| 89 | |
| 90 | ## What crosses the internet |
| 91 | |
| 92 | A push's chunks go to Lambda and their scan results come back. Pack |
| 93 | reads for a fetch, bundle streams for a clone, and pack writes from |
| 94 | compaction all go to S3. Reads are coalesced into few large ranges, |
| 95 | and the Worker streams bundle bytes straight through, but every one of |
| 96 | those is a hop the R2 stack does not make. |
| 97 | |
| 98 | ## When it is worth it |
| 99 | |
| 100 | Take this stack when the bytes have to be in AWS. Take the Lambda |
| 101 | hasher on its own when pushes are regularly wider than four chunks. |
| 102 | Lambda hashes every chunk at once. The loaded Workers hash four at a |
| 103 | time: |
| 104 | |
| 105 | ```diff lang="typescript" |
| 106 | const GitLive = Git.ApiLive.pipe( |
| 107 | Layer.provide(Git.ApiHandlersLive), |
| 108 | Layer.provide(Authentication.layer), |
| 109 | Layer.provide(Git.ReposDurableObject), |
| 110 | Layer.provide(Git.RegistryDurableObject), |
| 111 | - Layer.provide(GitHasher.HasherWorkerLoader()), |
| 112 | + Layer.provide(GitHasher.HasherLambda(GitHasher.HasherFunction)), |
| 113 | Layer.provide(Git.BlobStoreR2(GitObjects)), |
| 114 | + Layer.provide(AWS.Lambda.InvokeFunctionHttp), |
| 115 | ); |
| 116 | ``` |
| 117 | |
| 118 | Measured on the same 40 MiB delta-heavy push, the two hashers land |
| 119 | within a second of each other. [Hasher](/git/blocks/hasher) has the |
| 120 | table. |