The best way to control Claude Code, Codex and more from any computer or phone

main
mdx 120 lines 4.08 KB
Raw
1 ---
2 title: Bytes in S3, hashing on Lambda
3 description: Compute on Cloudflare, bytes in S3, and large pushes hashed one Lambda per chunk. One stack carries both provider sets, and Alchemy mints the cross-cloud identity.
4 ---
5
6 The examples use `Authentication` from
7 [Getting Started](/git/getting-started): the application's request middleware.
8
9 import PackageManagers from "../../../../components/PackageManagers.astro";
10
11 The Worker and the Durable Objects stay on Cloudflare. Packs, bundles,
12 and spilled pushes live in S3, and every 4 MiB chunk of a push is
13 hashed on its own Lambda:
14
15 ```typescript
16 // src/git.ts
17 import * as AWS from "@alchemy.run/aws";
18 import * as Cloudflare from "@alchemy.run/cloudflare";
19 import * as Git from "@alchemy.run/git";
20 import * as GitHasher from "@alchemy.run/git/Hasher";
21 import * as Effect from "effect/Effect";
22 import * as Layer from "effect/Layer";
23 import * as HttpRouter from "effect/http/HttpRouter";
24 import * as Http from "alchemy/Http";
25
26 const GitObjects = AWS.S3.Bucket("GitObjects");
27
28 const GitLive = Git.ApiLive.pipe(
29 Layer.provide(Git.ApiHandlersLive),
30 Layer.provide(Authentication.layer),
31 Layer.provide(Git.ReposDurableObject),
32 Layer.provide(Git.RegistryDurableObject),
33 Layer.provide(Git.BlobStoreS3(GitObjects)),
34 Layer.provide(GitHasher.HasherLambda(GitHasher.HasherFunction)),
35 Layer.provide(AWS.Lambda.InvokeFunctionHttp),
36 );
37
38 export default class GitHost extends Cloudflare.Worker<GitHost>()(
39 "Git",
40 { main: import.meta.url, ...Git.GIT_WORKER_OPTIONS },
41 Effect.gen(function* () {
42 const fetch = yield* HttpRouter.toHttpEffect(GitLive.pipe(Layer.provide(Http.Platform)));
43 return { fetch };
44 }),
45 ) {}
46 ```
47
48 ```typescript
49 // alchemy.run.ts
50 import * as Alchemy from "alchemy";
51 import * as AWS from "@alchemy.run/aws";
52 import * as Cloudflare from "@alchemy.run/cloudflare";
53 import * as Effect from "effect/Effect";
54 import * as Layer from "effect/Layer";
55 import GitHost from "./src/git.ts";
56
57 export default Alchemy.Stack(
58 "GitService",
59 {
60 providers: Layer.mergeAll(Cloudflare.providers(), AWS.providers()),
61 state: Cloudflare.state(),
62 },
63 Effect.gen(function* () {
64 const git = yield* GitHost;
65 return { url: git.url.as<string>() };
66 }),
67 );
68 ```
69
70 <PackageManagers run="alchemy deploy" />
71
72 ## What gets created
73
74 On Cloudflare, the same Worker and Durable Object namespaces as
75 [All on Cloudflare](/git/recipes/cloudflare), minus the bucket.
76
77 On AWS:
78
79 - `GitObjects`, an S3 bucket with an engine-generated name in the
80 deploying profile's region.
81 - `HasherFunction`, a 3 GB Lambda whose only handler answers hash
82 events, with its execution role.
83 - One IAM user, one access key, and one least-privilege role the Worker
84 assumes. `BlobStoreS3` and `InvokeFunctionHttp` share that identity:
85 the key and the role's ARN are bound onto the Worker, each request is
86 signed with credentials assumed at runtime, and the role is granted
87 the S3 actions on that one bucket and `lambda:InvokeFunction` on that
88 one function.
89
90 ## What crosses the internet
91
92 A push's chunks go to Lambda and their scan results come back. Pack
93 reads for a fetch, bundle streams for a clone, and pack writes from
94 compaction all go to S3. Reads are coalesced into few large ranges,
95 and the Worker streams bundle bytes straight through, but every one of
96 those is a hop the R2 stack does not make.
97
98 ## When it is worth it
99
100 Take this stack when the bytes have to be in AWS. Take the Lambda
101 hasher on its own when pushes are regularly wider than four chunks.
102 Lambda hashes every chunk at once. The loaded Workers hash four at a
103 time:
104
105 ```diff lang="typescript"
106 const GitLive = Git.ApiLive.pipe(
107 Layer.provide(Git.ApiHandlersLive),
108 Layer.provide(Authentication.layer),
109 Layer.provide(Git.ReposDurableObject),
110 Layer.provide(Git.RegistryDurableObject),
111 - Layer.provide(GitHasher.HasherWorkerLoader()),
112 + Layer.provide(GitHasher.HasherLambda(GitHasher.HasherFunction)),
113 Layer.provide(Git.BlobStoreR2(GitObjects)),
114 + Layer.provide(AWS.Lambda.InvokeFunctionHttp),
115 );
116 ```
117
118 Measured on the same 40 MiB delta-heavy push, the two hashers land
119 within a second of each other. [Hasher](/git/blocks/hasher) has the
120 table.