@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 04:45 UTC

main
mdx 121 lines 4.86 KB
Raw
1 ---
2 title: Hasher
3 description: Every object a push sends is inflated and hashed before a ref moves. That is CPU, and where it runs is a Layer, in the Worker, in dynamically loaded Workers, or on Lambda.
4 ---
5
6 A push arrives as a pack. Before any ref moves, every object in it is
7 inflated, hashed, and checked against the objects the repository
8 already has. `Git.Hasher` is where that work runs:
9
10 ```typescript
11 Layer.provide(Git.HasherInline)
12 ```
13
14 The Worker receiving the push never holds it whole:
15
16 1. The first 1 MiB is read for the ref commands, and
17 [Auth](/git/blocks/auth) rules on them before any pack byte is
18 inspected.
19 2. The body streams through in 8 MiB parts. Each part is cut into
20 4 MiB chunks for the hasher as it lands and is parked in the
21 [blob store](/git/blocks/blob-store) as a multipart upload.
22 3. Hashed objects reach the repository's Durable Object as batches of
23 rows over RPC: coordinates into the parked pack for large objects,
24 inline bytes for small ones.
25 4. The Durable Object checks connectivity over the staged rows,
26 updates the commit graph, moves the refs in one transaction, and
27 schedules compaction.
28
29 The pack never enters the Durable Object. Its memory, CPU, and egress
30 are untouched by push size, and its share of a push is SQL.
31
32 The Worker retains a 16 MiB window of the body for objects that span
33 parts, so peak memory is that window plus the parts in flight,
34 whatever the push size. Pushes share a 64 MiB budget per isolate, and
35 one that does not fit waits for an earlier one to drain.
36
37 An object that straddles two chunks is hashed from its own header the
38 moment its first chunk settles, so a 10 MB blob spanning three parts
39 does not wait for the last one. Deltas whose base is in another chunk
40 are resolved as a second round of hasher jobs, and the resolved blobs
41 are written out as a pack of their own so a fetch never re-resolves
42 them.
43
44 Where a chunk is hashed is the only thing the Layers below disagree
45 on.
46
47 ## In the Worker
48
49 `HasherInline` hashes chunks on the thread that received them. It costs nothing
50 extra, needs nothing else deployed, and is the right answer for the
51 kilobyte pushes that make up almost all of a repository's life.
52
53 ## In dynamically loaded Workers
54
55 ```typescript
56 import * as GitHasher from "@alchemy.run/git/Hasher";
57
58 Layer.provide(GitHasher.HasherWorkerLoader())
59 ```
60
61 Four Workers, loaded on demand from a module bundled into your Worker,
62 each in its own isolate with its own 128 MB, hash four chunks at once.
63 Nothing leaves Cloudflare and nothing needs credentials. The runtime
64 allows four loaded Workers per request, so a push wider than that is
65 hashed in waves.
66
67 ```typescript
68 GitHasher.HasherWorkerLoader({ concurrency: 2 })
69 ```
70
71 ## On Lambda
72
73 ```typescript
74 import * as AWS from "@alchemy.run/aws";
75 import * as GitHasher from "@alchemy.run/git/Hasher";
76
77 Layer.provide(GitHasher.HasherLambda(GitHasher.HasherFunction)),
78 Layer.provide(AWS.Lambda.InvokeFunctionHttp),
79 ```
80
81 One Lambda invocation per chunk, all of them at once, each on a 3 GB
82 function. `HasherFunction` is an Effect-native Lambda the package
83 declares. Your Worker binds it cross-cloud: Alchemy creates an IAM user
84 and a least-privilege role for the Worker, binds the key onto it, and
85 signs each invoke with credentials assumed at runtime. The stack that
86 deploys this carries both provider sets.
87
88 If a chunk cannot be hashed there, because of a throttle or a cold
89 start that runs out of time, the Worker hashes it inline. A push never
90 fails because its hasher did.
91
92 ## Choosing
93
94 One client, the same edge. A 40 MiB delta-heavy pack of 15.6k
95 objects, a 44 MiB pack of whole blobs, and GitHub on a fresh
96 repository for scale:
97
98 | Layer | Parallelism | Memory per hasher | Leaves Cloudflare | Extra cost | 40 MiB deltas | 44 MiB blobs |
99 | --- | --- | --- | --- | --- | --- | --- |
100 | `HasherInline` | none | the Worker's | no | none | 9.9–13.5 s | 8.8 s |
101 | `HasherWorkerLoader` | 4 isolates | 128 MB | no | none | 5.9–7.2 s | 4.8–6.4 s |
102 | `HasherLambda` | one Lambda per chunk | 3 GB | yes | Lambda time, egress | 6.8–7.9 s | 6.3–6.7 s |
103 | GitHub | | | | | 3.3 s | 4.6 s |
104
105 Incremental pushes take 0.2–0.4 s on every Layer, against 1.2 s on
106 GitHub. The choice only matters for pushes of tens of megabytes.
107
108 Start with `HasherInline`. Move to `HasherWorkerLoader` when large
109 pushes matter and you want to stay on one cloud. Reach for
110 `HasherLambda` when a push is wider than four chunks and you already
111 run AWS. [Scaling](/git/recipes/scaling) walks a large push through
112 the swap.
113
114 ## Your own
115
116 The contract is three operations over bytes: scan a chunk and return
117 its entries, resolve a batch of deltas against their bases, and hash
118 the bounds of an entry that straddles two chunks. `Git.HasherSelf` is
119 the smallest implementation to read. It fans out through the Worker's
120 own service binding, which on Cloudflare runs on the caller's thread
121 and buys nothing, but it is the shape a hasher on another host takes.